Project: ShardingSphere (App Definition and Development / Database)
File: usage.en.md

+++ title = "Manual" weight = 2 +++ Supported MySQL versions: 5.1.15 to 8.0.x. Enable `binlog` in source MySQL 5.7 `my.cnf` configuration sample: ``` [mysqld] server-id=1 log-bin=mysql-bin binlog-format=row binlog-row-image=full max_connections=600 ``` Run the following command and check whether `binlog` is enabled. ``` show variables like '%log_bin%'; show variables like '%binlog%'; ``` If the following information is displayed, binlog is enabled. ``` +--++ | Variable_name | Value | +--++ | log_bin | ON | | binlog_format | ROW | | binlogrowimage | FULL | +--++ ``` Grant Replication-related permissions for source MySQL account. Run the following command to check whether the user has migration permission. ``` SHOW GRANTS FOR 'migration_user'; ``` Result sample: ``` ++ |Grants for ${username}@${host} | ++ |GRANT REPLICATION SLAVE, REPLICATION CLIENT ON . TO ${username}@${host} | |....... | ++ ``` Grant DDL DML permissions for MySQL account Source MySQL account needs SELECT permission. Example: ```sql GRANT SELECT ON migrationds0.* TO `migration_user`@`%`; ``` Target MySQL account needs part of DDL and all DML permissions. Example: ```sql GRANT CREATE, DROP, INDEX, SELECT, INSERT, UPDATE, DELETE ON . TO `migration_user`@`%`; ``` Please refer to Prepare the source database, table, and data in MySQL. ```sql DROP DATABASE IF EXISTS migrationds0; CREATE DATABASE migrationds0 DEFAULT CHARSET utf8; USE migrationds0 CREATE TABLE torder (orderid INT NOT NULL, userid INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (orderid)); INSERT INTO torder (orderid, user_id, status) VALUES (1,2,'ok'),(2,4,'ok'),(3,6,'ok'),(4,1,'ok'),(5,3,'ok'),(6,5,'ok'); ``` Prepare the target database in MySQL. ```sql DROP DATABASE IF EXISTS migrationds10; CREATE DATABASE migrationds10 DEFAULT CHARSET utf8; DROP DATABASE IF EXISTS migrationds11; CREATE DATABASE migrationds11 DEFAULT CHARSET utf8; DROP DATABASE IF EXISTS migrationds12; CREATE DATABASE migrationds12 DEFAULT CHARSET utf8; ``` Create a new logical database in proxy and configure storage units and rules. ```sql CREATE DATABASE sharding_db; USE sharding_db REGISTER STORAGE UNIT ds_2 ( URL="jdbc:mysql://127.0.0.1:3306/migrationds10?serverTimezone=UTC&useSSL=false", USER="root", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ), ds_3 ( URL="jdbc:mysql://127.0.0.1:3306/migrationds11?serverTimezone=UTC&useSSL=false", USER="root", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ), ds_4 ( URL="jdbc:mysql://127.0.0.1:3306/migrationds12?serverTimezone=UTC&useSSL=false", USER="root", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ); CREATE SHARDING TABLE RULE t_order( STORAGEUNITS(ds2,ds3,ds4), SHARDINGCOLUMN=orderid, TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")), KEYGENERATESTRATEGY(COLUMN=order_id,TYPE(NAME="snowflake")) ); ``` If you are migrating to a heterogeneous database, you need to execute the table-creation statements in proxy. Configure the source storage units in proxy. ```sql REGISTER MIGRATION SOURCE STORAGE UNIT ds_0 ( URL="jdbc:mysql://127.0.0.1:3306/migrationds0?serverTimezone=UTC&useSSL=false", USER="root", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ); ``` Start data migration. ```sql MIGRATE TABLE ds0.torder INTO t_order; ``` Or you can specify a target logical database. ```sql MIGRATE TABLE ds0.torder INTO shardingdb.torder; ``` Check the data migration job list. ```sql SHOW MIGRATION LIST; ``` Result example: ``` +--+--+-+--++--+ | id | tables | jobitemcount | active | createtime | stoptime | +--+--+-+--++--+ | j0102p00002333dcb3d9db141cef14bed6fbf1ab54 | ds0.torder | 1 | true | 2023-09-20 14:41:32 | NULL | +--+--+-+--++--+ ``` View the data migration details. ```sql SHOW MIGRATION STATUS 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Result example: ``` ++-+--+--+--+-+-+--++ | item | datasource | tables | status | active | processedrecordscount | inventoryfinishedpercentage | incrementalidleseconds | errormessage | ++-+--+--+--+-+-+--++ | 0 | ds0 |

| EXECUTEINCREMENTAL_TASK | true | 6 | 100 | | | ++-+--+--+--+-+-+--++ ``` Verify data consistency. ```sql CHECK MIGRATION 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54' BY TYPE (NAME='DATA_MATCH'); ``` Data consistency check algorithm list: ```sql SHOW MIGRATION CHECK ALGORITHMS; ``` Result example: ``` +-+--+--+-+ | type | typealiases | supporteddatabase_types | description | +-+--+--+-+ | CRC32_MATCH | | MySQL,MariaDB,H2 | Match CRC32 of records. | | DATA_MATCH | | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2 | Match raw data of records. | +-+--+--+-+ ``` If encrypt rule is configured in target proxy, then `DATA_MATCH` could be used. If you are migrating to a heterogeneous database, then `DATA_MATCH` could be used. Query data consistency check progress: ```sql SHOW MIGRATION CHECK STATUS 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Result example: ``` +--+--++--+-+--+--+-+-++-+--++ | tables | result | checkfailedtables | active | inventoryfinishedpercentage | inventoryremainingseconds | incrementalidleseconds | checkbegintime | checkendtime | durationseconds | algorithmtype | algorithmprops | errormessage | +--+--++--+-+--+--+-+-++-+--++ | ds0.torder | true | | false | 100 | 0 | | 2023-09-20 14:45:31.992 | 2023-09-20 14:45:33.519 | 1 | DATA_MATCH | | | +--+--++--+-+--+--+-+-++-+--++ ``` Commit the job. ```sql COMMIT MIGRATION 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Please refer to for more details. Supported PostgreSQL version: 9.4 or later. Enable in source. Modify WAL configuration in source. `postgresql.conf` configuration sample: ``` wal_level = logical maxwalsenders = 10 maxreplicationslots = 10 walsendertimeout = 0 max_connections = 600 ``` Please refer to and for details. Grant replication permission for source PostgreSQL account. `pg_hba.conf` instance configuration: ``` host replication repl_acct 0.0.0.0/0 md5 ``` Please refer to for details. Grant DDL DML permissions for PostgreSQL account. If you are using a non-super admin account for migration, you need to GRANT CREATE and CONNECT privileges on the database used for migration. ```sql GRANT CREATE, CONNECT ON DATABASE migrationds0 TO migration_user; ``` The account also needs to have access to the migrated tables and schema. Take the t_order table under test schema as an example. ```sql \c migrationds0 GRANT USAGE ON SCHEMA test TO GROUP migration_user; GRANT SELECT ON TABLE test.torder TO migrationuser; ``` PostgreSQL has the concept of OWNER, and if the account is the OWNER of a database, SCHEMA, or table, the relevant steps can be omitted. Please refer to Prepare the source database, table, and data in PostgreSQL. ```sql DROP DATABASE IF EXISTS migrationds0; CREATE DATABASE migrationds0; \c migrationds0 CREATE TABLE torder (orderid INT NOT NULL, userid INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (orderid)); INSERT INTO torder (orderid, user_id, status) VALUES (1,2,'ok'),(2,4,'ok'),(3,6,'ok'),(4,1,'ok'),(5,3,'ok'),(6,5,'ok'); ``` Prepare the target database in PostgreSQL. ```sql DROP DATABASE IF EXISTS migrationds10; CREATE DATABASE migrationds10; DROP DATABASE IF EXISTS migrationds11; CREATE DATABASE migrationds11; DROP DATABASE IF EXISTS migrationds12; CREATE DATABASE migrationds12; ``` Create a new logical database in proxy and configure storage units and rules. ```sql CREATE DATABASE sharding_db; \c sharding_db REGISTER STORAGE UNIT ds_2 ( URL="jdbc:postgresql://127.0.0.1:5432/migrationds10", USER="postgres", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ), ds_3 ( URL="jdbc:postgresql://127.0.0.1:5432/migrationds11", USER="postgres", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ), ds_4 (

USER="postgres", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ); CREATE SHARDING TABLE RULE t_order( STORAGEUNITS(ds2,ds3,ds4), SHARDINGCOLUMN=orderid, TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")), KEYGENERATESTRATEGY(COLUMN=order_id,TYPE(NAME="snowflake")) ); ``` If you are migrating to a heterogeneous database, you need to execute the table-creation statements in proxy. Configure the source storage units in proxy. ```sql REGISTER MIGRATION SOURCE STORAGE UNIT ds_0 ( URL="jdbc:postgresql://127.0.0.1:5432/migrationds0", USER="postgres", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ); ``` Enable data migration. ```sql MIGRATE TABLE ds0.torder INTO t_order; ``` Or you can specify a target logical database. ```sql MIGRATE TABLE ds0.torder INTO shardingdb.torder; ``` Or you can specify a source schema name. ```sql MIGRATE TABLE ds0.public.torder INTO shardingdb.torder; ``` Check the data migration job list. ```sql SHOW MIGRATION LIST; ``` Result example: ``` +--+--+-+--++--+ | id | tables | jobitemcount | active | createtime | stoptime | +--+--+-+--++--+ | j0102p00002333dcb3d9db141cef14bed6fbf1ab54 | ds0.torder | 1 | true | 2023-09-20 14:41:32 | NULL | +--+--+-+--++--+ ``` View the data migration details. ```sql SHOW MIGRATION STATUS 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Result example: ``` ++-+--+--+--+-+-+--++ | item | datasource | tables | status | active | processedrecordscount | inventoryfinishedpercentage | incrementalidleseconds | errormessage | ++-+--+--+--+-+-+--++ | 0 | ds0 | ds0.torder | EXECUTEINCREMENTAL_TASK | true | 6 | 100 | | | ++-+--+--+--+-+-+--++ ``` Verify data consistency. ``` CHECK MIGRATION 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; Query OK, 0 rows affected (0.09 sec) ``` Query data consistency check progress: ```sql SHOW MIGRATION CHECK STATUS 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Result example: ``` +--+--++--+-+--+--+-+-++-+--++ | tables | result | checkfailedtables | active | inventoryfinishedpercentage | inventoryremainingseconds | incrementalidleseconds | checkbegintime | checkendtime | durationseconds | algorithmtype | algorithmprops | errormessage | +--+--++--+-+--+--+-+-++-+--++ | ds0.torder | true | | false | 100 | 0 | | 2023-09-20 14:45:31.992 | 2023-09-20 14:45:33.519 | 1 | DATA_MATCH | | | +--+--++--+-+--+--+-+-++-+--++ ``` Commit the job. ```sql COMMIT MIGRATION 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Please refer to for more details. Supported openGauss version: 2.0.1 to 3.0.0. Modify WAL configuration in source. `postgresql.conf` configuration sample: ``` wal_level = logical maxwalsenders = 10 maxreplicationslots = 10 walsendertimeout = 0 max_connections = 600 ``` Please refer to and for details. Grant replication permission for source openGauss account. `pg_hba.conf` instance configuration: ``` host replication repl_acct 0.0.0.0/0 md5 ``` Please refer to and for details. Grant DDL DML permissions for openGauss account. If you are using a non-super admin account for migration, you need to GRANT CREATE and CONNECT privileges on the database used for migration. ```sql GRANT CREATE, CONNECT ON DATABASE migrationds0 TO migration_user; ``` The account also needs to have access to the migrated tables and schema. Take the t_order table under test schema as an example. ```sql \c migrationds0 GRANT USAGE ON SCHEMA test TO GROUP migration_user; GRANT SELECT ON TABLE test.torder TO migrationuser; ``` openGauss has the concept of OWNER, and if the account is the OWNER of a database, SCHEMA, or table, the relevant steps can be omitted. openGauss does not allow normal accounts to operate in public schema, so if the migrated table is in public schema, you need to authorize additional. Please refer to ```sql GRANT ALL PRIVILEGES TO migration_user; ``` Prepare the source database, table, and data. 1.1. Isomorphic

```sql DROP DATABASE IF EXISTS migrationds0; CREATE DATABASE migrationds0; \c migrationds0 CREATE TABLE torder (orderid INT NOT NULL, userid INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (orderid)); INSERT INTO torder (orderid, user_id, status) VALUES (1,2,'ok'),(2,4,'ok'),(3,6,'ok'),(4,1,'ok'),(5,3,'ok'),(6,5,'ok'); ``` 1.2. Heterogeneous database. MySQL example: ```sql DROP DATABASE IF EXISTS migrationds0; CREATE DATABASE migrationds0 DEFAULT CHARSET utf8; USE migrationds0 CREATE TABLE torder (orderid INT NOT NULL, userid INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (orderid)); INSERT INTO torder (orderid, user_id, status) VALUES (1,2,'ok'),(2,4,'ok'),(3,6,'ok'),(4,1,'ok'),(5,3,'ok'),(6,5,'ok'); ``` Prepare the target database in openGauss. ```sql DROP DATABASE IF EXISTS migrationds10; CREATE DATABASE migrationds10; DROP DATABASE IF EXISTS migrationds11; CREATE DATABASE migrationds11; DROP DATABASE IF EXISTS migrationds12; CREATE DATABASE migrationds12; ``` Create a new logical database and configure storage units and rules. 1.1. Create logic database. ```sql CREATE DATABASE sharding_db; \c sharding_db ``` 1.2. Register storage units. ```sql REGISTER STORAGE UNIT ds_2 ( URL="jdbc:opengauss://127.0.0.1:5432/migrationds10", USER="gaussdb", PASSWORD="Root@123", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ), ds_3 ( URL="jdbc:opengauss://127.0.0.1:5432/migrationds11", USER="gaussdb", PASSWORD="Root@123", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ), ds_4 ( URL="jdbc:opengauss://127.0.0.1:5432/migrationds12", USER="gaussdb", PASSWORD="Root@123", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ); ``` 1.3. Create sharding table rule. ```sql CREATE SHARDING TABLE RULE t_order( STORAGEUNITS(ds2,ds3,ds4), SHARDINGCOLUMN=orderid, TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")), KEYGENERATESTRATEGY(COLUMN=order_id,TYPE(NAME="snowflake")) ); ``` 1.4. Create target table. If you are migrating to a heterogeneous database, you need to execute the table-creation statements in proxy. MySQL example: ```sql CREATE TABLE torder (orderid INT NOT NULL, userid INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (orderid)); ``` Configure the source storage units in proxy. 2.1. Isomorphic database. ```sql REGISTER MIGRATION SOURCE STORAGE UNIT ds_0 ( URL="jdbc:opengauss://127.0.0.1:5432/migrationds0", USER="gaussdb", PASSWORD="Root@123", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ); ``` 2.2. Heterogeneous database. MySQL example: ```sql REGISTER MIGRATION SOURCE STORAGE UNIT ds_0 ( URL="jdbc:mysql://127.0.0.1:3306/migrationds0?serverTimezone=UTC&useSSL=false", USER="root", PASSWORD="root", PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000") ); ``` Enable data migration. ```sql MIGRATE TABLE ds0.torder INTO t_order; ``` Or you can specify a target logical database. ```sql MIGRATE TABLE ds0.torder INTO shardingdb.torder; ``` Or you can specify a source schema name. ```sql MIGRATE TABLE ds0.public.torder INTO shardingdb.torder; ``` Check the data migration job list. ```sql SHOW MIGRATION LIST; ``` Result example: ``` +--+--+-+--++--+ | id | tables | jobitemcount | active | createtime | stoptime | +--+--+-+--++--+ | j0102p00002333dcb3d9db141cef14bed6fbf1ab54 | ds0.torder | 1 | true | 2023-09-20 14:41:32 | NULL | +--+--+-+--++--+ ``` View the data migration details. ```sql SHOW MIGRATION STATUS 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Result example: ``` ++-+--+--+--+-+-+--++ | item | datasource | tables | status | active | processedrecordscount | inventoryfinishedpercentage | incrementalidleseconds | errormessage | ++-+--+--+--+-+-+--++ | 0 | ds0 | ds0.torder | EXECUTEINCREMENTAL_TASK | true | 6 | 100 | | | ++-+--+--+--+-+-+--++ ``` Verify data consistency. ``` CHECK MIGRATION 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; Query OK, 0 rows affected (0.09 sec) ``` Query data consistency check progress: ```sql SHOW MIGRATION CHECK STATUS 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Result example: ``` +--+--++--+-+--+--+-+-++-+--++ | tables | result | checkfailedtables | active | inventoryfinishedpercentage | inventoryremainingseconds | incrementalidleseconds | checkbegintime | checkendtime | durationseconds | algorithmtype | algorithmprops | errormessage | +--+--++--+-+--+--+-+-++-+--++ | ds0.torder | true | | false | 100 | 0 | | 2023-09-20 14:45:31.992 | 2023-09-20 14:45:33.519 | 1 | DATA_MATCH | | | +--+--++--+-+--+--+-+-++-+--++ ``` Commit the job. ```sql COMMIT MIGRATION 'j0102p00002333dcb3d9db141cef14bed6fbf1ab54'; ``` Please refer to for more details.