Package: z3-tap-mongodb
Tap-Mongodbtap-mongodbis a Singer tap for MongoDB.This tap differentiates itself from existing taps in a few ways. First, rather than expose a very specific set of configuration options for the underlying pymongo driver, we expose all possible arguments by accepting an object underneath themongokey which pass all kwargs straight through to the driver. There are over 40 configurable kwargs available as seenhere. This gives it more flexibility in contrast to a constrained interface. Secondly, this tap has three output modes configurable viastrategy: raw | envelope | infer.Strategy 1(strategy: raw) merely outputs data as-is with anadditonalProperties: trueschema. This is ideal for loading to unstructured sources such as blob storage where it may be preferable to keep documents exactly as they are.Strategy 2(strategy: envelope) will wrap the document in adocumentkey and output a fixed schema. The schema will usetype: objectand the target should be able to handle unstructured data, ie. via a VARIANT/JSON column.Strategy 3(strategy: infer) infers the schema of each collection from a configurable sample size of records. This allows the tap to work with strongly typed destinations. This is an attractive option. Particularly when we don't expect the documents to vary dramatically.The last differentiator is the minimal code footprint in comparison to existing Mongo taps. I hope that this tap exemplifies how we can use as little code as possible with the existing plumbing in the SDK. The SDK means the tap supports theBATCHspecification out of the box and is able to receive ongoing updates and improvements as the SDK continues to mature.InstallationThe package on pypi is namedz3-tap-mongodbbut the executable it ships with is simplytap-mongodb. This allows me to release work without concerns of naming conflicts on the package index.# Use pipx or pippipxinstallz3-tap-mongodb# Verify it is installedtap-mongodb--versionIncremental SyncsWe support incremental syncs on a collection by collection basis. All this requires is the developer adding areplication_keyto the catalog for a stream. After dumping thetap-mongodb --config ... --discover > catalog.json, you can modify the catalog and version control it for ongoing use. If using meltano, you can use themetadatakey to update the catalog dynamically achieving the same affect. One caveat of our incrementality is that this relies on an alphanumerically sortable replication key. This accounts for a majority of use cases but there are some edge cases. Integer based replication keys (such as epochs) work fine obviously, ISO formatted dates work fine, but any other format may not work as expected.SettingsSettingRequiredDefaultDescriptionmongoTrueNoneThese props are passed directly to pymongo MongoClient allowing the tap user full flexibility not provided in any other Mongo tap since every kwarg can be tuned.stream_prefixFalseOptionally add a prefix for all streams, useful if ingesting from multiple shards/clusters via independent tap-mongodb configs. This is applied during catalog generation. Regenerate the catalog to apply a new stream prefix.optional_replication_keyFalse0This setting allows the tap to continue processing if a document is missing the replication key. Useful if a very small percentage of documents are missing the property.database_includesFalseNoneA list of databases to include. If this list is empty, all databases will be included.database_excludesFalseNoneA list of databases to exclude. If this list is empty, no databases will be excluded.strategyFalse"raw"The strategy to use for schema resolution. Defaults to 'raw'. The 'raw' strategy uses a relaxed schema using additionalProperties: true to accept the document as-is leaving the target to respect it. Useful for blob or jsonl. The 'envelope' strategy will envelope the document under a key nameddocument. The target should use a variant type for this key. The 'infer' strategy will infer the schema from the data based on a configurable number of documents.infer_schema_max_docsFalse2000The maximum number of documents to sample when inferring the schema. This is only used whenstrategyis set toinfer.batch_configFalseNoneBatch configuration as definedherestream_mapsFalseNonestream_map_configFalseNoneUser-defined config values to be used within map expressions.flattening_enabledFalseNone'True' to enable schema flattening and automatically expand nested properties.flattening_max_depthFalseNoneThe max depth to flatten schemas.A full list of supported settings and capabilities is available by running:tap-mongodb --aboutConfigure using environment variablesThis Singer tap will automatically import any environment variables within the working directory's.envif the--config=ENVis provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the.envfile.Capabilitiesbatchcatalogstatediscoveraboutstream-mapsschema-flatteningUsageYou can easily runtap-mongodbby itself or in a pipeline usingMeltano.Executing the Tap Directlytap-mongodb--version
tap-mongodb--help
tap-mongodb--configCONFIG--discover>./catalog.jsonDeveloper ResourcesInitialize your Development Environmentpipxinstallpoetry
poetryinstallCreate and Run TestsCreate tests within thetap_mongodb/testssubfolder and
then run:poetryrunpytestYou can also test thetap-mongodbCLI interface directly usingpoetry run:poetryruntap-mongodb--helpTesting withMeltanoNote:This tap will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios.Your project comes with a custommeltano.ymlproject file already created. Open themeltano.ymland follow any"TODO"items listed in
the file.Next, install Meltano (if you haven't already) and any needed plugins:# Install meltanopipxinstallmeltano# Initialize meltano within this directorycdtap-mongodb
meltanoinstallNow you can test and orchestrate using Meltano:# Test invocation:meltanoinvoketap-mongodb--version# OR run a test `elt` pipeline:meltanoelttap-mongodbtarget-jsonlSDK Dev GuideSee thedev guidefor more instructions on how to use the SDK to
develop your own taps and targets.Built with theMeltano Tap SDKfor Singer Taps.