Package: pipelinewise-tap-mongodb
pipelinewise-tap-mongodbThis is aSingertap that produces JSON-formatted data following theSinger specfrom a MongoDB source.Set up local dev environment:makesetupActivate virtual environment.venv/bin/activateSet up Config fileCreate json file calledconfig.json, with the following contents:{"password":"<password>","user":"<username>","host":"<host ip address>","auth_database":"<database name to authenticate on>","database":"<database name to sync from>"}The following parameters are optional for your config file:NameTypeDefault valueDescriptionsrvBooleanfalseuses amongodb+srvprotocol to connect. Disables the usage ofportargument if set toTrueportIntegerfalseConnection port. Required if a non-srv connection is being used.replica_setstringnullname of replica setsslBooleanfalsecan be set to true to connect using sslverify_modeBooleantrueDefault SSL verify modeinclude_schemas_in_destination_stream_nameBooleanfalseforces the stream names to take the form<database_name>-<collection_name>instead of<collection_name>update_buffer_sizeint1[LOG_BASED] The size of the buffer that holds detected update operations in memory, the buffer is flushed once the size is reachedawait_time_msint1000[LOG_BASED] The maximum amount of time in milliseconds the loge_base method waits for new data changes before exiting.All of the above attributes are required by the tap to connect to your mongo instance.
here is asample configuration file.Run in discovery modeRun the following command and redirect the output into the catalog filetap-mongodb--config~/config.json--discover>~/catalog.jsonYour catalog file should now look like this:{"streams":[{"table_name":"<table name>","tap_stream_id":"<tap_stream_id>","metadata":[{"breadcrumb":[],"metadata":{"row-count":<int>,"is-view":<bool>,"database-name":"<database name>","table-key-properties":["_id"],"valid-replication-keys":["_id"]}}],"stream":"<stream name>","schema":{"type":"object"}}]}Edit Catalog fileUsing valid json, edit the config.json fileTo select a stream, enter the following to the stream's metadata:"selected":true,"replication-method":"<replication method>",<replication-method>must be eitherFULL_TABLE,INCREMENTALorLOG_BASED, if it'sINCREMENTAL, make sure to add a"replication-key".For example, if you were to edit the example stream to select the stream as well as add a projection, config.json should look this:{"streams":[{"table_name":"<table name>","tap_stream_id":"<tap_stream_id>","metadata":[{"breadcrumb":[],"metadata":{"row-count":<int>,"is-view":<bool>,"database-name":"<database name>","table-key-properties":["_id"],"valid-replication-keys":["_id"],"selected":true,"replication-method":"<replication method>"}}],"stream":"<stream name>","schema":{"type":"object"}}]}Run in sync mode:tap-mongodb--config~/config.json--catalog~/catalog.jsonThe tap will write bookmarks to stdout which can be captured and passed as an optional--state state.jsonparameter to the tap for the next sync.Logging configurationThe tap uses a predefined logging config if none is provided, however, you can set your own config by setting the environment variableLOGGING_CONFIG_FILEas the path to the logging config.
A sample config is availablehere.Copyright © 2020 TransferWise