Package: c8-source-postgres
c8-source-postgresSingertap that extracts data from aPostgreSQLdatabase and
produces JSON-formatted data following
theSinger spec.How to use itThe recommended method of running this tap is to use it with Macrometa data connectors.
If you want to run thisSinger Tapindependently please read further.Install and RunFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallc8-source-postgresormakevenvCreate a config.json{
  "host": "localhost",
  "port": 5432,
  "user": "postgres",
  "password": "secret",
  "dbname": "db"
}These are the same basic configuration properties used by the PostgreSQL command-line client (psql).Full list of options inconfig.json:PropertyTypeRequired?DescriptionhostStringYesPostgreSQL hostportIntegerYesPostgreSQL portuserStringYesPostgreSQL userpasswordStringYesPostgreSQL passworddbnameStringYesPostgreSQL database namefilter_schemasStringNoComma separated schema names to scan only the required schemas to improve the performance of data extraction. (Default: None)sslStringNoIf set to"true"then use SSL via postgres sslmoderequireoption. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail. (Default: None)logical_poll_total_secondsIntegerNoStop running the tap when no data received from wal after certain number of seconds. (Default: 10800)break_at_end_lsnBooleanNoStop running the tap if the newly received lsn is after the max lsn that was detected when the tap started. (Default: true)max_run_secondsIntegerNoStop running the tap after certain number of seconds. (Default: 43200)debug_lsnStringNoIf set to"true"then add_sdc_lsnproperty to the singer messages to debug postgres LSN position in the WAL stream. (Default: None)tap_idStringNoID of the pipeline/tap (Default: None)itersizeIntegerNoSize of PG cursor iterator when doing INCREMENTAL or FULL_TABLE  (Default: 20000)default_replication_methodStringNoDefault replication method to use when no one is provided in the catalog (Values:LOG_BASED,INCREMENTALorFULL_TABLE)  (Default: None)use_secondaryBooleanNoUse a database replica forINCREMENTALandFULL_TABLEreplication (Default : False)secondary_hostStringNoPostgreSQL Replica host (required ifuse_secondaryisTrue)secondary_portIntegerNoPostgreSQL Replica port (required ifuse_secondaryisTrue)Run the tap in Discovery Modec8-source-postgres --config config.json --discover                # Should dump a Catalog to stdout
c8-source-postgres --config config.json --discover > catalog.json # Capture the CatalogAdd Metadata to the CatalogEach entry under the Catalog's "stream" key will need the following metadata:{
  "streams": [
    {
      "stream_name": "my_topic"
      "metadata": [{
        "breadcrumb": [],
        "metadata": {
          "selected": true,
          "replication-method": "LOG_BASED",
        }
      }]
    }
  ]
}The replication method can be one ofFULL_TABLE,INCREMENTALorLOG_BASED.Note: Log based replication requires a few adjustments in the source postgres database, please read further
for more information.Run the tap in Sync Modec8-source-postgres --config config.json --catalog catalog.jsonThe tap will write bookmarks to stdout which can be captured and passed as an optional--state state.jsonparameter
to the tap for the next sync.Log Based replication requirementsPostgreSQL database's runningPostgreSQL versions 9.4.x or greater. To avoid a critical PostgreSQL bug,
use at least one of the following minor versions:PostgreSQL 12.0PostgreSQL 11.2PostgreSQL 10.7PostgreSQL 9.6.12PostgreSQL 9.5.16PostgreSQL 9.4.21A connection to the master instance. Log-based replication will only work by connecting to the master instance.wal2json plugin: To use Log Based for your PostgreSQL integration, you must install the wal2json plugin.
The wal2json plugin outputs JSON objects for logical decoding, which the tap then uses to perform Log-based
Replication.
Steps for installing the plugin vary depending on your operating system. Instructions for each operating system type
are in the wal2json’s GitHub repository:Unix-based operating systemsWindowspostgres config file: Locate the database configuration file (usuallypostgresql.conf) and define
the parameters as follows:wal_level=logical
max_replication_slots=5
max_wal_senders=5Restart your PostgreSQL service to ensure the changes take effect.Note: Formax_replication_slotsandmax_wal_senders, we’re defaulting to a value of 5.
This should be sufficient unless you have a large number of read replicas connected to the master instance.Existing replication slot: Log based replication requires a dedicated logical replication slot.
In PostgreSQL, a logical replication slot represents a stream of database changes that can then be replayed to a
client in the order they were made on the original server. Each slot streams a sequence of changes from a single
database.Login to the master instance as a superuser and using thewal2jsonplugin, create a logical replication slot:SELECT *
  FROM pg_create_logical_replication_slot('macrometa_<database_name>', 'wal2json');Note: Replication slots are specific to a given database in a cluster. If you want to connect multiple
databases - whether in one integration or several - you’ll need to create a replication slot for each database.To run tests:Install python test dependencies in a virtual env:make venvYou need to have a postgres database to run the tests and export its credentials:export C8_SOURCE_POSTGRES_HOST=<postgres-host>
  export C8_SOURCE_POSTGRES_PORT=<postgres-port>
  export C8_SOURCE_POSTGRES_SECONDARY_HOST=<postgres-replica-host>
  export C8_SOURCE_POSTGRES_SECONDARY_PORT=<postgres-replica-port>
  export C8_SOURCE_POSTGRES_USER=<postgres-user>
  export C8_SOURCE_POSTGRES_PASSWORD=<postgres-password>You can make use of the local docker-compose to spin up a test database by runningmake start_dbTest objects will be created in thepostgresdatabase.To run the tests:make testTo run pylint:Install python dependencies and run python lintermake venv
  make pylint