Project: YugabyteDB (App Definition and Development / Database)
File: async-deployment.md

title: Deploy to two universes with xCluster replication headerTitle: xCluster deployment linkTitle: xCluster description: Enable deployment using unidirectional (master-follower) or bidirectional (multi-master) replication between universes headContent: Unidirectional (master-follower) and bidirectional (multi-master) replication menu: stable: parent: async-replication identifier: async-deployment weight: 10 type: docs You can create source and target universes as follows: Create the source universe by following the procedure from . Create tables for the APIs being used by the source universe. Create the target universe by following the procedure from . Create tables for the APIs being used by the target universe. These should be the same tables as you created for the source universe. Proceed to setting up or replication. If you already have existing data in your tables, follow the bootstrap process described in . After you created the required tables, you can set up unidirectional replication as follows: Look up the source universe UUID and the table IDs for the two tables and the index table: To find a universe's UUID, check `http://yb-master-ip:7000/varz` for `--cluster_uuid`. If it is not available in this location, check the same field in the universe configuration. To find a table ID, execute the following command as an admin user: ```sh ./bin/yb-admin -masteraddresses <sourceuniversemasteraddresses> listtables includetable_id ``` The preceding command lists all the tables, including system tables. To locate a specific table, you can add `grep`, as follows: ```sh ./bin/yb-admin -masteraddresses <sourceuniversemasteraddresses> listtables includetableid | grep tablename ``` Run the following `yb-admin` command from the YugabyteDB home directory in the source universe: ```sh ./bin/yb-admin \ -masteraddresses <targetuniversemasteraddresses> \ setupuniversereplication <sourceuniverseUUID><replicationstream_name> \ <sourceuniversemaster_addresses> \ <tableid>,[<tableid>..] ``` For example: ```sh ./bin/yb-admin \ -master_addresses 127.0.0.11:7100,127.0.0.12:7100,127.0.0.13:7100 \ setupuniversereplication e260b8b6-e89f-4505-bb8e-b31f74aa29f3_xClusterSetup1 \ 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 \ 000030a5000030008000000000004000,000030a5000030008000000000004005,dfef757c415c4b2cacc9315b8acb539a ``` The preceding command contains three table IDs: the first two are YSQL for the base table and index, and the third is the YCQL table. Make sure that all master addresses for source and target universes are specified in the command. If you need to set up bidirectional replication, see instructions provided in . Otherwise, proceed to . To set up bidirectional replication, repeat the procedure described in applying the steps to the target universe. You need to set up each source to consume data from target. When completed, proceed to . After you have set up replication, load data into the source universe, as follows: Download the YugabyteDB workload generator JAR file `yb-sample-apps.jar` from . Start loading data into source by following examples for YSQL or YCQL: YSQL: ```sh java -jar yb-sample-apps.jar --workload SqlSecondaryIndex --nodes 127.0.0.1:5433 ``` YCQL: ```sh java -jar yb-sample-apps.jar --workload CassandraBatchKeyValue --nodes 127.0.0.1:9042 ``` Note that the IP address needs to correspond to the IP of any YB-TServers in the universe. For bidirectional replication, repeat the preceding step in the target universe. When completed, proceed to . You can verify replication by stopping the workload and then using the `COUNT(*)` function on the target to source match. For unidirectional replication, connect to the target universe using the YSQL shell (`ysqlsh`) or the YCQL shell (`ycqlsh`), and confirm that you can see the expected records. For bidirectional replication, repeat the procedure described in , but reverse the source and destination information, as follows: Run `yb-admin setupuniversereplication` on the target universe, pointing to source. Use the workload generator to start loading data into the target

Verify replication from target to source. To avoid primary key conflict errors, keep the key ranges for the two universes separate. This is done automatically by the applications included in the `yb-sample-apps.jar`. Replication lag is computed at the tablet level as follows: `replication lag = hybridclocktime - lastreadhybrid_time` hybrid_clock_time is the hybrid clock timestamp on the source's tablet server, and last_read_hybrid_time is the hybrid clock timestamp of the latest record pulled from the source. To obtain information about the overall maximum lag, you should check `/metrics` or `/prometheus-metrics` for `asyncreplicationsentlagmicros` or `asyncreplicationcommittedlagmicros` and take the maximum of these values across each source's YB-TServer. For information on how to set up the Node Exporter and Prometheus manually, see . You can use `yb-admin` to return the current replication status. The `getreplicationstatus` command returns the replication status for all consumer-side replication streams. An empty `errors` field means the replication stream is healthy. ```sh ./bin/yb-admin \ -master_addresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 \ getreplicationstatus ``` ```output.yaml statuses { table_id: "03ee1455f2134d5b914dd499ccad4377" stream_id: "53441ad2dd9f4e44a76dccab74d0a2ac" errors { error: REPLICATIONMISSINGOP_ID error_detail: "Unable to find expected op id on the producer" } } ``` The setup process depends on whether the source and target universes have the same certificates. If both universes use the same certificates, run `yb-admin setupuniversereplication` and include the flag. Setting that to the target universe's certificate directory will make replication use those certificates for connecting to both universes. Consider the following example: ```sh ./bin/yb-admin -master_addresses 127.0.0.11:7100,127.0.0.12:7100,127.0.0.13:7100 \ -certsdirname /home/yugabyte/yugabyte-tls-config \ setupuniversereplication e260b8b6-e89f-4505-bb8e-b31f74aa29f3_xClusterSetup1 \ 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 \ 000030a5000030008000000000004000,000030a5000030008000000000004005,dfef757c415c4b2cacc9315b8acb539a ``` When universes use different certificates, you need to store the certificates for the source universe on the target universe, as follows: Ensure that `usenodetonodeencryption` is set to `true` on all and on both the source and target. For each YB-Master and YB-TServer on the target universe, set the flag `certsforcdc_dir` to the parent directory where you want to store all the source universe's certificates for replication. Find the certificate authority file used by the source universe (`ca.crt`). This should be stored in the . Copy this file to each node on the target. It needs to be copied to a directory named`<certsforcdcdir>/<sourceuniverse_uuid>`. For example, if you previously set `certsforcdcdir=/home/yugabyte/yugabyteproducercerts`, and the source universe's ID is `00000000-1111-2222-3333-444444444444`, then you would need to copy the certificate file to `/home/yugabyte/yugabyteproducer_certs/00000000-1111-2222-3333-444444444444/ca.crt`. Set up replication using `yb-admin setupuniversereplication`, making sure to also set the `-certsdirname` flag to the directory with the target universe's certificates (this should be different from the directory used in the previous steps). For example, if you have the target universe's certificates in `/home/yugabyte/yugabyte-tls-config`, then you would run the following: ```sh ./bin/yb-admin -master_addresses 127.0.0.11:7100,127.0.0.12:7100,127.0.0.13:7100 \ -certsdirname /home/yugabyte/yugabyte-tls-config \ setupuniversereplication 00000000-1111-2222-3333-444444444444_xClusterSetup1 \ 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 \ 000030a5000030008000000000004000,000030a5000030008000000000004005,dfef757c415c4b2cacc9315b8acb539a ``` You start by creating a source and a target universe with the same configurations (the same regions and zones), as follows: Regions: EU(Paris), Asia Pacific(Mumbai), US West(Oregon) Zones: eu-west-3a, ap-south-1a, us-west-2a ```sh ./bin/yb-ctl --rf 3 create --placement_info "cloud1.region1.zone1,cloud2.region2.zone2,cloud3.region3.zone3" ``` Consider the following example: ```sh ./bin/yb-ctl --rf 3 create --placement_info

``` Create tables, tablespaces, and partition tables at both the source and target universes, as per the following example: Main table: transactions Tablespaces: euts, apts, us_ts Partition tables: transactionseu, transactionsin, transactions_us ```sql CREATE TABLE transactions ( user_id INTEGER NOT NULL, account_id INTEGER NOT NULL, geo_partition VARCHAR, amount NUMERIC NOT NULL, created_at TIMESTAMP DEFAULT NOW() ) PARTITION BY LIST (geo_partition); CREATE TABLESPACE eu_ts WITH( replicaplacement='{"numreplicas": 1, "placement_blocks": [{"cloud": "aws", "region": "eu-west-3","zone":"eu-west-3a", "minnumreplicas":1}]}'); CREATE TABLESPACE us_ts WITH( replicaplacement='{"numreplicas": 1, "placement_blocks": [{"cloud": "aws", "region": "us-west-2","zone":"us-west-2a", "minnumreplicas":1}]}'); CREATE TABLESPACE ap_ts WITH( replicaplacement='{"numreplicas": 1, "placement_blocks": [{"cloud": "aws", "region": "ap-south-1","zone":"ap-south-1a", "minnumreplicas":1}]}'); CREATE TABLE transactions_eu PARTITION OF transactions (userid, accountid, geopartition, amount, createdat, PRIMARY KEY (userid HASH, accountid, geo_partition)) FOR VALUES IN ('EU') TABLESPACE eu_ts; CREATE TABLE transactions_in PARTITION OF transactions (userid, accountid, geopartition, amount, createdat, PRIMARY KEY (userid HASH, accountid, geo_partition)) FOR VALUES IN ('IN') TABLESPACE ap_ts; CREATE TABLE transactions_us PARTITION OF transactions (userid, accountid, geopartition, amount, createdat, PRIMARY KEY (userid HASH, accountid, geo_partition)) DEFAULT TABLESPACE us_ts; ``` To create unidirectional replication, perform the following: Collect partition table UUIDs from the source universe (partition tables, transactionseu, transactionsin, transactions_us) by navigating to Tables in the Admin UI available at 127.0.0.1:7000. These UUIDs are to be used while setting up replication. Run the replication setup command for the source universe, as follows: ```sh ./bin/yb-admin -masteraddresses <targetmaster_addresses> \ setupuniversereplication <sourceuniverseUUID><replicationstream_name> \ <sourcemasteraddresses> <commaseparatedtable_ids> ``` Consider the following example: ```sh ./bin/yb-admin -master_addresses 127.0.0.11:7100,127.0.0.12:7100,127.0.0.13:7100 \ setupuniversereplication 00000000-1111-2222-3333-444444444444_xClusterSetup1 \ 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 \ 000033e1000030008000000000004007,000033e100003000800000000000400d,000033e1000030008000000000004013 ``` Optionally, if you have access to YugabyteDB Anywhere, you can observe the replication setup (`xClusterSetup1`) by navigating to Replication on the source and target universe. In the Kubernetes environment, you can set up a pod to pod connectivity, as follows: Create a source and a target universe. Create tables in both universes, as follows: Execute the following commands for the source universe: ```sh kubectl exec -it -n <sourceuniversenamespace> -t <sourceuniversemasterleader> -c <sourceuniverse_container> -- bash /home/yugabyte/bin/ysqlsh -h <sourceuniverseyqlserver> create table query ``` Consider the following example: ```sh kubectl exec -it -n xcluster-source -t yb-master-2 -c yb-master -- bash /home/yugabyte/bin/ysqlsh -h yb-tserver-1.yb-tservers.xcluster-source create table employees(id int primary key, name text); ``` Execute the following commands for the target universe: ```sh kubectl exec -it -n <targetuniversenamespace> -t <targetuniversemasterleader> -c <targetuniverse_container> -- bash /home/yugabyte/bin/ysqlsh -h <targetuniverseyqlserver> create table query ``` Consider the following example: ```sh kubectl exec -it -n xcluster-target -t yb-master-2 -c yb-master -- bash /home/yugabyte/bin/ysqlsh -h yb-tserver-1.yb-tservers.xcluster-target create table employees(id int primary key, name text); ``` Collect table UUIDs by navigating to Tables in the Admin UI available at 127.0.0.1:7000. These UUIDs are to be used while setting up replication. Set up replication from the source universe by executing the following command on the source universe: ```sh kubectl exec -it -n <sourceuniversenamespace> -t <sourceuniversemaster_leader> -c \ <sourceuniversecontainer> -- bash -c "/home/yugabyte/bin/yb-admin -master_addresses \ <targetuniversemasteraddresses> setupuniverse_replication \ <sourceuniverseUUID><replicationstreamname> <sourceuniversemasteraddresses> \ <commaseparatedtable_ids>" ``` Consider the following example: ```sh kubectl exec -it -n xcluster-source -t yb-master-2 -c yb-master -- bash -c \ "/home/yugabyte/bin/yb-admin -master_addresses yb-master-2.yb-masters.xcluster-target.svc.cluster.local, \ yb-master-1.yb-masters.xcluster-target.svc.cluster.local,yb-master-0.yb-masters.xcluster-target.svc.cluster.local \ setupuniversereplication ac39666d-c183-45d3-945a-475452deac9fxCluster1 \ yb-master-2.yb-masters.xcluster-source.svc.cluster.local,yb-master-1.yb-masters.xcluster-source.svc.cluster.local, \ yb-master-0.yb-masters.xcluster-source.svc.cluster.local 00004000000030008000000000004001" ``` Perform the following on the source universe and then observe replication on the target universe: ```sh kubectl exec -it -n <sourceuniversenamespace> -t <sourceuniversemasterleader> -c <sourceuniverse_container> -- bash /home/yugabyte/bin/ysqlsh -h <sourceuniverseyqlserver> insert query select query ``` Consider the following example: ```sh kubectl exec -it -n xcluster-source -t yb-master-2 -c yb-master -- bash /home/yugabyte/bin/ysqlsh -h

INSERT INTO employees VALUES(1, 'name'); SELECT * FROM employees; ``` Perform the following on the target universe: ```sh kubectl exec -it -n <targetuniversenamespace> -t <targetuniversemasterleader> -c <targetuniverse_container> -- bash /home/yugabyte/bin/ysqlsh -h <targetuniverseyqlserver> select query ``` Consider the following example: ```sh kubectl exec -it -n xcluster-target -t yb-master-2 -c yb-master -- bash /home/yugabyte/bin/ysqlsh -h yb-tserver-1.yb-tservers.xcluster-target SELECT * FROM employees; ``` You can set up xCluster replication for the following purposes: Enabling replication on a table that has existing data. Catching up an existing stream where the target has fallen too far behind. To ensure that the WALs are still available, you need to perform the following steps in the flag window. If the process is going to take more time than the value defined by this flag, you should increase the value. Proceed as follows: Create a checkpoint on the source universe for all the tables you want to replicate by executing the following command: ```sh ./bin/yb-admin -masteraddresses <sourceuniversemasteraddresses> \ bootstrapcdcproducer <commaseparatedsourceuniversetable_ids> ``` Consider the following example: ```sh ./bin/yb-admin -master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 \ bootstrapcdcproducer 000033e1000030008000000000004000,000033e1000030008000000000004003,000033e1000030008000000000004006 ``` The following output is a list of bootstrap IDs, one per table ID: ```output table id: 000033e1000030008000000000004000, CDC bootstrap id: fb156717174941008e54fa958e613c10 table id: 000033e1000030008000000000004003, CDC bootstrap id: a2a46f5cbf8446a3a5099b5ceeaac28b table id: 000033e1000030008000000000004006, CDC bootstrap id: c967967523eb4e03bcc201bb464e0679 ``` Take the backup of the tables on the source universe and restore at the target universe by following instructions from. Execute the following command to set up the replication stream using the bootstrap IDs generated in step 1. Ensure that the bootstrap IDs are in the same order as their corresponding table IDs. ```sh ./bin/yb-admin -masteraddresses <targetuniversemasteraddresses> setupuniversereplication \ <sourceuniverseuuid><replicationstreamname> <sourceuniversemasteraddresses> \ <commaseparatedsourceuniversetableids> <commaseparatedbootstrapids> ``` Consider the following example: ```sh ./bin/yb-admin -masteraddresses 127.0.0.11:7100,127.0.0.12:7100,127.0.0.13:7100 setupuniverse_replication \ 00000000-1111-2222-3333-444444444444_xCluster1 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 \ 000033e1000030008000000000004000,000033e1000030008000000000004003,000033e1000030008000000000004006 \ fb156717174941008e54fa958e613c10,a2a46f5cbf8446a3a5099b5ceeaac28b,c967967523eb4e03bcc201bb464e0679 ``` You can modify the bootstrap as follows: To wipe the test setup, use the `deleteuniversereplication` command. After running the `bootstrapcdcproducer` command on the source universe, you can verify that it work as expected by running the `listcdcstreams` command to view the associated entries: the bootstrap IDs generated by the `bootstrapcdcproducer` command should match the `streamid` values you see after executing the `listcdc_streams` command. You can also perform the following modifications: To add a table to the source and target universes, use the `alteruniversereplication add_table` command. See . To remove an existing table from the source and target universes, use the `alteruniversereplication remove_table` command. See . To change master nodes on the source universe, execute the `alteruniversereplication setmasteraddresses` command. You can verify changes via the `getuniverseconfig` command. You can execute DDL operations after replication has been already been configured. Depending on the type of DDL operations, additional considerations are required. When new tables (or partitions) are created, to ensure that all changes from the time of object creation are replicated, writes should start on the new objects only after they are added to replication. If tables (or partitions) already have existing data before they are added to replication, then follow the bootstrap process described in . Create a table (with partitions) on both the source and target universes as follows: ```sql CREATE TABLE order_changes ( order_id int, change_date date, type text, description text) PARTITION BY RANGE (change_date); CREATE TABLE orderchangesdefault PARTITION OF order_changes DEFAULT; --Create a new partition CREATE TABLE orderchanges202301 PARTITION OF orderchanges FOR VALUES FROM ('2023-01-01') TO ('2023-03-30'); ``` Assume the parent table and default partition are included in the replication

Get table IDs of the new partition from the source as follows: ```sql yb-admin -masteraddresses <sourcemaster_ips> \ -certsdirname <cert_dir> \ listtables includetableid|grep 'orderchanges202301' ``` You should see output similar to the following: ```output yugabyte.orderchanges2021_01 000033e8000030008000000000004106 ``` Add the new table (or partition) to replication. ```sql yb-admin -masteraddresses <targetmaster_ips> \ -certsdirname <cert_dir> \ alteruniversereplication <replicationgroupname> \ add_table 000033e800003000800000000000410b ``` You should see output similar to the following: ```output Replication altered successfully ``` To add a new index to an empty table, follow the same steps as described in . However, to add a new index to a table that already has data, the following additional steps are required to ensure that the index has all the updates: Create an - for example, `my_new index` on the source. Wait for index backfill to finish. For more details, refer to YugabyteDB tips on . Determine the table ID for `my_new index`. ```sql yb-admin -masteraddresses <sourcemaster_ips> \ -certsdirname <cert_dir> \ listtables includetableid|grep 'mynew_index' ``` You should see output similar to the following: ```output 000033e8000030008000000000004028 ``` Bootstrap the replication stream on the source using the `bootstrapcdcproducer` API and provide the table ID of the new index as follows: ```sql yb-admin -masteraddresses <sourcemaster_ips> \ -certsdirname <cert_dir> \ bootstrapcdcproducer 000033e8000030008000000000004028 ``` You should see output similar to the following: ```output table id: 000033e8000030008000000000004028, CDC bootstrap id: c8cba563e39c43feb66689514488591c ``` Wait for replication to be 0 on the main table using the replication lag metrics described in . Create an on the target. Wait for index backfill to finish. For more details, refer to YugabyteDB tips on . Add the index to replication with the bootstrap ID from Step 4. ```sql yb-admin -masteraddresses <targetmaster_ips> \ -certsdirname <cert_dir> \ alteruniversereplication 59e58153-eec6-4cb5-a858-bf685df52316_east-west \ add_table 000033e8000030008000000000004028 c8cba563e39c43feb66689514488591c ``` You should see output similar to the following: ```output Replication altered successfully ``` Objects (tables, indexes, partitions) need to be removed from replication before they can be dropped as follows: Get the table ID for the object to be removed from the source. ```sql yb-admin -masteraddresses <sourcemaster_ips> \ -certsdirname <cert_dir> \ listtables includetableid |grep '<partitionname>' ``` Remove the table from replication on the target. ```sql yb-admin -masteraddresses <targetmaster_ips> \ -certsdirname <cert_dir> \ alteruniversereplication <replicationgroupname> \ remove_table 000033e800003000800000000000410b ``` Alters involving adding/removing columns or modifying data types require replication to be paused before applying schema changes as follows: Pause replication on both sides. ```sql yb-admin -masteraddresses <targetmaster_ips> -certsdirname <cert_dir> \ setuniversereplicationenabled <replicationgroup_name> 0 ``` You should see output similar to the following: ```output Replication disabled successfully ``` Perform the schema modification. Resume replication as follows: ```sql yb-admin -masteraddresses <targetmaster_ips> -certsdirname <cert_dir> \ setuniversereplicationenabled <replicationgroup_name> 0 ``` ```output Replication enabled successfully ``` When adding a new column with a (non-volatile) default expression, make sure to perform the schema modification on the target with the computed default value. For example, say you have a replicated table `test_table`. Pause replication on both sides. Execute add column command on the source: ```sql ALTER TABLE testtable ADD COLUMN testcolumn TIMESTAMP DEFAULT NOW() ``` Run the preceding `ALTER TABLE` command with the computed default value on the target as follows: The computed default value can be retrieved from the `attmissingval` column in the `pg_attribute` catalog table. Example: ```sql SELECT attmissingval FROM pgattribute WHERE attrelid='test'::regclass AND attname='testcolumn'; ``` ```output attmissingval {"2024-01-09 12:29:11.88894"} (1 row) ``` Execute the `ADD COLUMN` command on the target with the computed default value. ```sql ALTER TABLE test ADD COLUMN test_column TIMESTAMP DEFAULT "2024-01-09 12:29:11.88894" ```