Package: pypg-cdc
pypgoutputPython package to read, parse and convert PostgreSQL logical decoding messages to change data capture messages. Built using psycopg2's logical replication support objects, PostgreSQL's pgoutput plugin and Pydantic.Uses python >= 3.8Developed on PostgreSQL 12 for now.Warning: this is a prototype and not production testedInstallation$pipinstallpypgoutputHow it worksReplication messages are consumed via psycopg2's replication connection.https://www.psycopg.org/docs/extras.html#replication-support-objectsThe binary messages from pgoutput logical decoding are parsed in thedecoders.pymodule.Parsed messages are converted to change events and yieled from theLogicalReplicationReaderChange events are nested Pydantic models where the tuple data (before/after) schema is dynamically generated depending on the table being processed.ExampleFirst, setup a publication and a logical replication slot in the source database.CREATEPUBLICATIONtest_pubFORALLTABLES;SELECT*FROMpg_create_logical_replication_slot('test_slot','pgoutput');Second, run the script to collect the changes:importosimportpypgoutputHOST=os.environ.get("PGHOST")PORT=os.environ.get("PGPORT")DATABASE_NAME=os.environ.get("PGDATABASE")USER=os.environ.get("PGUSER")PASSWORD=os.environ.get("PGPASSWORD")cdc_reader=pypgoutput.LogicalReplicationReader(publication_name="test_pub",slot_name="test_slot",host=HOST,database=DATABASE_NAME,port=PORT,user=USER,password=PASSWORD,)formessageincdc_reader:print(message.json(indent=2))cdc_reader.stop()Generate some change messagesCREATETABLEpublic.readme(idintegerprimarykey,created_attimestamptzdefaultnow());INSERTINTOpublic.readme(id)SELECTdataFROMgenerate_series(1,3)ASdata;Output:{"op":"I","message_id":"4606b12b-ab41-41e6-9717-7ce92f8a9857","lsn":23530912,"transaction":{"tx_id":499,"begin_lsn":23531416,"commit_ts":"2022-01-14T17:22:10.298334+00:00"},"table_schema":{"column_definitions":[{"name":"id","part_of_pkey":true,"type_id":23,"type_name":"integer","optional":false},{"name":"created_at","part_of_pkey":false,"type_id":1184,"type_name":"timestamp with time zone","optional":true}],"db":"test_db","schema_name":"public","table":"readme","relation_id":16403},"before":null,"after":{"id":1,"created_at":"2022-01-14T17:22:10.296740+00:00"}}{"op":"I","message_id":"1ede0643-42b6-4bb1-8a98-8e4ca10c7915","lsn":23531144,"transaction":{"tx_id":499,"begin_lsn":23531416,"commit_ts":"2022-01-14T17:22:10.298334+00:00"},"table_schema":{"column_definitions":[{"name":"id","part_of_pkey":true,"type_id":23,"type_name":"integer","optional":false},{"name":"created_at","part_of_pkey":false,"type_id":1184,"type_name":"timestamp with time zone","optional":true}],"db":"test_db","schema_name":"public","table":"readme","relation_id":16403},"before":null,"after":{"id":2,"created_at":"2022-01-14T17:22:10.296740+00:00"}}{"op":"I","message_id":"fb477de5-8281-4102-96ee-649a838d38f2","lsn":23531280,"transaction":{"tx_id":499,"begin_lsn":23531416,"commit_ts":"2022-01-14T17:22:10.298334+00:00"},"table_schema":{"column_definitions":[{"name":"id","part_of_pkey":true,"type_id":23,"type_name":"integer","optional":false},{"name":"created_at","part_of_pkey":false,"type_id":1184,"type_name":"timestamp with time zone","optional":true}],"db":"test_db","schema_name":"public","table":"readme","relation_id":16403},"before":null,"after":{"id":3,"created_at":"2022-01-14T17:22:10.296740+00:00"}}Why use this package?Preference to use the built in pgoutput plugin. Some plugins are not available in managed database services such as RDS or cannot be updated to a new version.The pgoutput plugin includes useful metadata such as relid.Useful linkslogical decoding:https://www.postgresql.org/docs/12/logicaldecoding.htmlmessage specification:https://www.postgresql.org/docs/12/protocol-logicalrep-message-formats.htmlpostgres publication:https://www.postgresql.org/docs/12/sql-createpublication.htmlpsycopg2 replication support:https://www.psycopg.org/docs/extras.html#replication-support-objects