Project: ClickHouse (App Definition and Development / Database)
File: materialized-postgresql.md

slug: /en/engines/database-engines/materialized-postgresql sidebar_label: MaterializedPostgreSQL sidebar_position: 60 Creates a ClickHouse database with tables from PostgreSQL database. Firstly, database with engine `MaterializedPostgreSQL` creates a snapshot of PostgreSQL database and loads required tables. Required tables can include any subset of tables from any subset of schemas from specified database. Along with the snapshot database engine acquires LSN and once initial dump of tables is performed - it starts pulling updates from WAL. After database is created, newly added tables to PostgreSQL database are not automatically added to replication. They have to be added manually with `ATTACH TABLE db.table` query. Replication is implemented with PostgreSQL Logical Replication Protocol, which does not allow to replicate DDL, but allows to know whether replication breaking changes happened (column type changes, adding/removing columns). Such changes are detected and according tables stop receiving updates. In this case you should use `ATTACH`/ `DETACH PERMANENTLY` queries to reload table completely. If DDL does not break replication (for example, renaming a column) table will still receive updates (insertion is done by position). :::note This database engine is experimental. To use it, set `allowexperimentaldatabasematerializedpostgresql` to 1 in your configuration files or by using the `SET` command: ```sql SET allowexperimentaldatabasematerializedpostgresql=1 ``` ::: ``` sql CREATE DATABASE [IF NOT EXISTS] db_name [ON CLUSTER cluster] ENGINE = MaterializedPostgreSQL('host:port', 'database', 'user', 'password') [SETTINGS ...] ``` Engine Parameters `host:port` PostgreSQL server endpoint. `database` PostgreSQL database name. `user` PostgreSQL user. `password` User password. ``` sql CREATE DATABASE postgres_db ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresuser', 'postgres_password'); SHOW TABLES FROM postgres_db; name table1 SELECT * FROM postgresqldb.postgrestable; ``` After `MaterializedPostgreSQL` database is created, it does not automatically detect new tables in according PostgreSQL database. Such tables can be added manually: ``` sql ATTACH TABLE postgresdatabase.newtable; ``` :::warning Before version 22.1, adding a table to replication left a non-removed temporary replication slot (named `{dbname}chreplicationslottmp`). If attaching tables in ClickHouse version before 22.1, make sure to delete it manually (`SELECT pgdropreplicationslot('{dbname}chreplicationslot_tmp')`). Otherwise disk usage will grow. This issue is fixed in 22.1. ::: It is possible to remove specific tables from replication: ``` sql DETACH TABLE postgresdatabase.tableto_remove PERMANENTLY; ``` PostgreSQL can be configured in 3 ways (starting from version 21.12). One schema for one `MaterializedPostgreSQL` database engine. Requires to use setting `materializedpostgresqlschema`. Tables are accessed via table name only: ``` sql CREATE DATABASE postgres_database ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresuser', 'postgres_password') SETTINGS materializedpostgresqlschema = 'postgres_schema'; SELECT * FROM

``` Any number of schemas with specified set of tables for one `MaterializedPostgreSQL` database engine. Requires to use setting `materializedpostgresqltables_list`. Each table is written along with its schema. Tables are accessed via schema name and table name at the same time: ``` sql CREATE DATABASE database1 ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresuser', 'postgres_password') SETTINGS materializedpostgresqltables_list = 'schema1.table1,schema2.table2,schema1.table3', materializedpostgresqltableslistwith_schema = 1; SELECT * FROM database1.`schema1.table1`; SELECT * FROM database1.`schema2.table2`; ``` But in this case all tables in `materializedpostgresqltables_list` must be written with its schema name. Requires `materializedpostgresqltableslistwith_schema = 1`. Warning: for this case dots in table name are not allowed. Any number of schemas with full set of tables for one `MaterializedPostgreSQL` database engine. Requires to use setting `materializedpostgresqlschema_list`. ``` sql CREATE DATABASE database1 ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresuser', 'postgres_password') SETTINGS materializedpostgresqlschema_list = 'schema1,schema2,schema3'; SELECT * FROM database1.`schema1.table1`; SELECT * FROM database1.`schema1.table2`; SELECT * FROM database1.`schema2.table2`; ``` Warning: for this case dots in table name are not allowed. The setting must have a value `logical` and `maxreplicationslots` parameter must have a value at least `2` in the PostgreSQL config file. Each replicated table must have one of the following : primary key (by default) index ``` bash postgres# CREATE TABLE postgres_table (a Integer NOT NULL, b Integer, c Integer NOT NULL, d Integer, e Integer NOT NULL); postgres# CREATE unique INDEX postgrestableindex on postgres_table(a, c, e); postgres# ALTER TABLE postgrestable REPLICA IDENTITY USING INDEX postgrestable_index; ``` The primary key is always checked first. If it is absent, then the index, defined as replica identity index, is checked. If the index is used as a replica identity, there has to be only one such index in a table. You can check what type is used for a specific table with the following command: ``` bash postgres# SELECT CASE relreplident WHEN 'd' THEN 'default' WHEN 'n' THEN 'nothing' WHEN 'f' THEN 'full' WHEN 'i' THEN 'index' END AS replica_identity FROM pg_class WHERE oid = 'postgres_table'::regclass; ``` :::note Replication of values is not supported. The default value for the data type will be used. ::: Sets a comma-separated list of PostgreSQL database tables, which will be replicated via database engine. Default value: empty list means whole PostgreSQL database will be replicated. Default value: empty string. (Default schema is used) Default value: empty list. (Default schema is used) Sets the number of rows collected in memory before flushing data into PostgreSQL database table. Possible values: Positive integer. Default value: `65536`. A user-created replication slot. Must be used together with

A text string identifying a snapshot, from which will be performed. Must be used together with `materializedpostgresqlreplication_slot`. ``` sql CREATE DATABASE database1 ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresuser', 'postgres_password') SETTINGS materializedpostgresqltables_list = 'table1,table2,table3'; SELECT * FROM database1.table1; ``` The settings can be changed, if necessary, using a DDL query. But it is impossible to change the setting `materializedpostgresqltables_list`. To update the list of tables in this setting use the `ATTACH TABLE` query. ``` sql ALTER DATABASE postgresdatabase MODIFY SETTING materializedpostgresqlmaxblocksize = <newsize>; ``` Use a unique replication consumer identifier for replication. Default: `0`. If set to `1`, allows to setup several `MaterializedPostgreSQL` tables pointing to the same `PostgreSQL` table. Logical Replication Slots which exist on the primary are not available on standby replicas. So if there is a failover, new primary (the old physical standby) wont be aware of any slots which were existing with old primary. This will lead to a broken replication from PostgreSQL. A solution to this is to manage replication slots yourself and define a permanent replication slot (some information can be found ). You'll need to pass slot name via `materializedpostgresqlreplicationslot` setting, and it has to be exported with `EXPORT SNAPSHOT` option. The snapshot identifier needs to be passed via `materializedpostgresql_snapshot` setting. Please note that this should be used only if it is actually needed. If there is no real need for that or full understanding why, then it is better to allow the table engine to create and manage its own replication slot. Example (from ) Configure replication slot in PostgreSQL. ```yaml apiVersion: "acid.zalan.do/v1" kind: postgresql metadata: name: acid-demo-cluster spec: numberOfInstances: 2 postgresql: parameters: wal_level: logical patroni: slots: clickhouse_sync: type: logical database: demodb plugin: pgoutput ``` Wait for replication slot to be ready, then begin a transaction and export the transaction snapshot identifier: ```sql BEGIN; SELECT pgexportsnapshot(); ``` In ClickHouse create database: ```sql CREATE DATABASE demodb ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresuser', 'postgres_password') SETTINGS materializedpostgresqlreplicationslot = 'clickhousesync', materializedpostgresqlsnapshot = '0000000A-0000023F-3', materializedpostgresqltables_list = 'table1,table2,table3'; ``` End the PostgreSQL transaction once replication to ClickHouse DB is confirmed. Verify that replication continues after failover: ```bash kubectl exec acid-demo-cluster-0 -c postgres -- su postgres -c 'patronictl failover --candidate acid-demo-cluster-1 --force' ``` -- create query privilege. -- replication privilege. -- replication privilege or superuser. -- owner of publication (`username` in MaterializedPostgreSQL engine itself). It is possible to avoid executing `2` and `3` commands and having those permissions. Use settings `materializedpostgresqlreplicationslot` and `materializedpostgresql_snapshot`. But with much care. Access to tables: pg_publication pgreplicationslots pgpublicationtables