Project: StarRocks (App Definition and Development / Database)
File: SMT.md

displayed_sidebar: English StarRocks Migration Tool (SMT) is a data migration tool provided by StarRocks to load data from source databases through Flink into StarRocks. SMT mainly can: Generate statements to create tables in StarRocks based on information of the source database and the target StarRocks cluster. Generate SQL statements that can be executed in Flink's SQL client to submit Flink jobs for synchronizing data, which simplifies full or incremental data synchronization in the pipeline. Currently, SMT supports the following source databases: | Source database | Generate statement to create table in StarRocks | Full data synchronization | Incremental data synchronization | | | -- | - | -- | | MySQL | Support | Support | Support | | PostgreSQL | Support | Support | Support | | Oracle | Support | Support | Support | | Hive | Support | Support | Not support | | ClickHouse | Support | Support | Not support | | SQL Server | Support | Support | Support | | TiDB | Support | Support | Support | Download link: https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2 The generally involved steps are as follows: Configure the conf/config_prod.conf file. Execute starrocks-migration-tool. After execution, the SQL scripts are generated in the result directory by default. You can then use the SQL scripts in the result directory for metadata or data synchronization. `[db]`: information to connect the data source. Configure the information to connect the data source corresponding to the database type specified in the `type` patameter. `[other]`: additional configurations. It is recommended to specify the actual number of BE nodes in the `be_num` patameter. `flink.starrocks.sink.*`: configurations of flink-connector-starrocks. For detailed configurations and description, see . `[table-rule.1]`: rule to match tables in the data source. The CREATE TABLE statement is generated based on the regular expressions configured in the rule to match the names of databases and tables in data source. Multiple rules can be configured, and each rule generates a corresponding result file, for example: `[table-rule.1]` -> `result/starrocks-create.1.sql` `[table-rule.2]` -> `result/starrocks-create.2.sql` Each rule needs to contain the configurations of database, table, and flink-connector-starrocks. ```Bash [table-rule.1] database = ^ database1.*$ table = ^.*$ schema = ^.*$ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.stripouterarray=true [table-rule.2] database = ^database2.*$ table = ^.*$ schema = ^.*$ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.stripouterarray=true ``` A separate rule can be configured for a large table that is split into shards in databases. For example, suppose that the two databases `edudb1` and `edudb2` contain the tables `course1` and `course2` respectively, and these two tables have the same structure. You can use the following rule to load data from these two tables into one StarRocks table for analysis. ```Bash [table-rule.3] database = ^edudb[0-9]*$ table = ^course_[0-9]*$ schema = ^.*$ ``` This rule will automatically form a many-to-one loading relationship. The default name of the table that will be generated in StarRocks is `courseauto_shard`, and you can also modify the table name in the related SOL script, such as `result/starrocks-create.3.sql`. Flink CDC connector and SMT can synchronize data from MySQL within subsecond. As shown in the image, SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of MySQL and

Flink CDC connector reads the MySQL Binlog and Flink-connector-starrocks writes data to StarRocks . Download . The version of Flink is supported to be 1.11 or later. Download . Make sure that you download the `flink-sql-connector-mysql-cdc-xxx.jar` corresponding to the version of Flink. Download . Copy flink-sql-connector-mysql-cdc-xxx.jar and flink-connector-starrocks-xxx.jar to flink-xxx/lib/. Download . Extract and modify the configuration file of SMT. ```Bash [db] host = 192.168.1.1 port = 3306 user = root password = type = mysql [other] be_num = 3 usedecimalv3 = false output_dir = ./result [table-rule.1] database = ^db$ table = ^table$ schema = ^.*$ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.stripouterarray=true ``` Execute starrocks-migrate-tool. All SQL scripts are generated in the result directory. ```Bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ``` Use a SQL script whose prefix is starrocks-create to generate the table in StarRocks. ```Bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` Use a SQL script whose prefix is flink-create to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```Bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. Observe the status of the Flink job. ```Bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file conf/flink-conf.yaml, such as memory and slot. How to enable MySQL binlog? Modify /etc/my.cnf: ```plaintext log-bin=/var/lib/mysql/mysql-bin server-id=1 binlog_format = row ``` Restart mysqld. You can check whether MySQL binlog is enabled by executing `SHOW VARIABLES LIKE 'log_bin';`. Flink CDC connector and SMT can synchronize data from PostgreSQL within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of PostgreSQL and StarRocks. Flink CDC connector reads the WAL of PostgreSQL and Flink-connector-starrocks writes data to StarRocks . Download . The version of Flink is supported to be 1.11 or later. Download . Make sure that you download the flink-sql-connector-postgres-cdc-xxx.jar corresponding to the version of Flink. Download . Copy flink-sql-connector-postgres-cdc-xxx.jar and flink-connector-starrocks-xxx.jar to flink-xxx/lib/. Download . Extract and modify the configuration file of SMT. ```Bash [db] host = 192.168.1.1 port = 5432 user = xxx password = xxx type = pgsql [other] be_num = 3 usedecimalv3 = false output_dir = ./result [table-rule.1] database = ^db$ table = ^table$ schema = ^.*$ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.stripouterarray=true ``` Execute starrocks-migrate-tool. All SQL scripts are generated in the result directory. ```Bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` Use a SQL script whose prefix is starrocks-create to generate the table in StarRocks. ```Bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` Use a SQL script whose prefix is flink-create to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```Bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. Observe the status of the Flink job. ```Bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file conf/flink-conf.yaml, such as memory and

For PostgreSQL `v9.`, a special flink-cdc configuration as shown below is required (It is recommended to use PostgreSQL `v10.` or later. Otherwise, you need to install WAL decoding plugins): ```Bash ``` How to enable PostgreSQL WAL? ```Bash echo "host all all 0.0.0.0/32 trust" >> pg_hba.conf echo "host replication all 0.0.0.0/32 trust" >> pg_hba.conf echo "wal_level = logical" >> postgresql.conf echo "maxwalsenders = 2" >> postgresql.conf echo "maxreplicationslots = 8" >> postgresql.conf ``` Specify replica identity FULL for the tables that need to be synchronized. ```SQL ALTER TABLE schemaname.tablename REPLICA IDENTITY FULL ``` After making these changes, restart PostgreSQL . Flink CDC connector and SMT can synchronize data from Oracle within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of Oracle and StarRocks. Flink CDC connector reads the logminer of Oracle and Flink-connector-starrocks writes data to StarRocks. Download . The version of Flink is supported to be 1.11 or later. Download . Make sure that you download the flink-sql-connector-oracle-cdc-xxx.jar corresponding to the Flink version. Download . Copy `flink-sql-connector-oracle-cdc-xxx.jar` and `flink-connector-starrocks-xxx.jar` to `flink-xxx/lib/`. Download . Extract and modify the configuration file of SMT. ```Bash [db] host = 192.168.1.1 port = 1521 user = xxx password = xxx type = oracle [other] be_num = 3 usedecimalv3 = false output_dir = ./result [table-rule.1] database = ^db$ table = ^table$ schema = ^.*$ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.stripouterarray=true ``` Execute starrocks-migrate-tool. All SQL scripts are generated in the result directory. ```Bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` Use a SQL script whose prefix is starrocks-create to generate the table in StarRocks. ```Bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` Use a SQL script whose prefix is flink-create to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```Bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. Observe the status of the Flink job. ```Bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file conf/flink-conf.yaml, such as memory and slot. Synchronize Oracle using logminer: ```SQL alter system set dbrecoveryfile_dest = '/home/oracle/data' scope=spfile; alter system set dbrecoveryfiledestsize = 10G; shutdown immediate; startup mount; alter database archivelog; alter database open; ALTER TABLE

ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; GRANT CREATE SESSION TO flinkuser; GRANT SET CONTAINER TO flinkuser; GRANT SELECT ON V_$DATABASE TO flinkuser; GRANT FLASHBACK ANY TABLE TO flinkuser; GRANT SELECT ANY TABLE TO flinkuser; GRANT SELECTCATALOGROLE TO flinkuser; GRANT EXECUTECATALOGROLE TO flinkuser; GRANT SELECT ANY TRANSACTION TO flinkuser; GRANT LOGMINING TO flinkuser; GRANT CREATE TABLE TO flinkuser; GRANT LOCK ANY TABLE TO flinkuser; GRANT ALTER ANY TABLE TO flinkuser; GRANT CREATE SEQUENCE TO flinkuser; GRANT EXECUTE ON DBMS_LOGMNR TO flinkuser; GRANT EXECUTE ON DBMSLOGMNRD TO flinkuser; GRANT SELECT ON V_$LOG TO flinkuser; GRANT SELECT ON V$LOGHISTORY TO flinkuser; GRANT SELECT ON V$LOGMNRLOGS TO flinkuser; GRANT SELECT ON V$LOGMNRCONTENTS TO flinkuser; GRANT SELECT ON V$LOGMNRPARAMETERS TO flinkuser; GRANT SELECT ON V_$LOGFILE TO flinkuser; GRANT SELECT ON V$ARCHIVEDLOG TO flinkuser; GRANT SELECT ON V$ARCHIVEDEST_STATUS TO flinkuser; ``` The database configurations in the [table-rule.1] does not support regular expression, so the complete database names need to be specified. Because Oracle12c supports CDB mode, SMT internally automatically determines whether CDB is enabled and modifies the flink-cdc configuration correspondingly. However, users need to pay attention to whether the c## prefix needs to be added to the configuration of `[db].user` to avoid issues of insufficient permission. This guide explains how to use SMT to synchronize Hive data to StarRocks. During synchronization, a Duplicate table in StarRocks is created and a Flink job keeps running to synchronize data. ```SQL [db] host = 127.0.0.1 port = 10000 user = hive/emr-header-1.cluster-49148 password = type = hive authentication = kerberos ``` The supported authentication methods are as follows: nosasl, zk: do not need to specify the `user` and `password`. none, none_http, ldap: specify the `user` and `password`. kerberos, kerberos_http: perform the following steps: Execute `kadmin.local` on the Hive cluster and check `list_principals` to find the corresponding principal name. For example when the principal name is `hive/emr-header-1.cluster-49148@EMR.49148.COM`, the user needs to be set as `hive/emr-header-1.cluster-49148`, and the password is left empty. Execute `kinit -kt /path/to/keytab principal` on the machine where SMT is executed and execute `klist` to see if the correct token is generated. Execute starrocks-migrate-tool. Use a SQL script whose prefix is starrocks-create to generate the table in StarRocks. ```Bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` In flink/conf/, create and edit the file sql-client-defaults.yaml: ```YAML execution: planner: blink type: batch current-catalog: hive-starrocks catalogs: name: hive-starrocks type: hive hive-conf-dir: /path/to/apache-hive-xxxx-bin/conf ``` Download the (flink-sql-connector-hive-xxxx) from the Hive page of the corresponding version of Flink and place it in the `flink/lib` directory. Start the Flink cluster and execute `flink/bin/sql-client.sh embedded < result/flink-create.all.sql` to start data synchronization. Flink CDC connector and SMT can synchronize data from SQL Server within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of SQL Server and StarRocks. Flink CDC connector captures and records row-level changes that occur in the SQL Server database server. The principle is to use the CDC feature provided by SQL Server itself. The CDC capability of SQL Server itself can archive specified changes in the database to specified change tables. The SQL Server CDC connector first reads historical data from the table using JDBC, and then fetches incremental changes from the change tables, thereby achieving full incremental synchronization. Then, Flink-connector-starrocks writes data to StarRocks. Download . The version of Flink is supported to be 1.11 or later. Download . Make sure that you download the flink-sql-connector-sqlserver-cdc-xxx.jar corresponding to the Flink version. Download . Copy flink-sql-connector-sqlserver-cdc-xxx.jar, flink-connector-starrocks-xxx.jar to flink-xxx/lib/. Download . Extract and modify the configuration file of SMT. ```Bash [db] host = 127.0.0.1 port = 1433 user = xxx password = xxx type = sqlserver [other] be_num = 3 usedecimalv3 = false output_dir = ./result [table-rule.1] database = ^db$ table = ^table$ schema = ^.*$ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.stripouterarray=true ``` Execute starrocks-migrate-tool*. All SQL scripts are generated in the `result` directory. ```Bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ``` Use a SQL script whose prefix is `starrocks-create` to generate the table in StarRocks. ```Bash mysql

-P9030 -uroot -p < starrocks-create.all.sql ``` Use a SQL script whose prefix is `flink-create` to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```Bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. Observe the status of the Flink job. ```Bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file conf/flink-conf.yaml, such as memory and slot. Make sure that the Server Agent Service is enabled. Check if the Server Agent Service is running normally. ```sql EXEC master.dbo.xp_servicecontrol N'QUERYSTATE', N'SQLSERVERAGENT' GO ``` Enable the Server Agent Service. ```Bash /opt/mssql/bin/mssql-conf set sqlagent.enabled true ``` Make sure that CDC for the corresponding database is enabled. Check if CDC for the corresponding database is enabled. ```sql select iscdcenabled, name from sys.databases where name = 'XXX_databases' GO ``` Enable CDC. :::note When executing this command, make sure that the user `serverRole` is `sysadmin`. ::: ```SQL USE XXX_databases GO EXEC sys.spcdcenable_db GO ``` Make sure that CDC for the corresponding table is enabled. ```SQL EXEC sys.spcdcenable_table @sourceschema = 'XXXschema', @sourcename = 'XXXtable', @role_name = NULL, @supportsnetchanges = 0; GO ``` Flink CDC connector and SMT can synchronize data from TiDB within subsecond. SMT can automatically generate DDL statements for the Flink's source tables and sink tables based on the cluster information and table structure of TiDB and StarRocks. The Flink CDC connector captures data by directly reading full and incremental data from the underlying TiKV storage. The full data is obtained from ranges partitioned based on keys, and the incremental data is obtained by using the CDC Client provided by TiDB. Subsequently, the data is written to StarRocks through Flink-connector-starrocks. Download . The version of Flink is supported to be 1.11 or later. Download . Make sure that you download the flink-sql-connector-tidb-cdc-xxx.jar corresponding to the Flink version. Download . Copy flink-sql-connector-tidb-cdc-xxx.jar, flink-connector-starrocks-xxx.jar to flink-xxx/lib/. Download . Extract and modify the configuration file of SMT. ```Bash [db] host = 127.0.0.1 port = 4000 user = root password = type = tidb [other] be_num = 3 usedecimalv3 = false output_dir = ./result [table-rule.1] database = ^db$ table = ^table$ schema = ^.*$ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.stripouterarray=true ``` Execute starrocks-migrate-tool*. All SQL scripts are generated in the `result` directory. ```Bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ``` Use a SQL script whose prefix is `starrocks-create` to generate the table in StarRocks. ```Bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` Use a SQL script whose prefix is `flink-create` to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```Bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. Observe the status of the Flink job. ```Bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file conf/flink-conf.yaml, such as memory and slot. For TiDB whose version is before v4.0.0, additional configuration of `flink.cdc.pd-addresses` is required. ```` ```Bash ``` ````