Project: StarRocks (App Definition and Development / Database)
File: Json_loading.md

displayed_sidebar: "English" You can import semi-structured data (for example, JSON) by using stream load or routine load. Stream Load: For JSON data stored in text files, use stream load to import. Routine Load: For JSON data in Kafka, use routine load to import. Sample data: ~~~json { "id": 123, "city" : "beijing"}, { "id": 456, "city" : "shanghai"}, ... ~~~ Example: ~~~shell curl -v --location-trusted -u <username>:<password> \ -H "format: json" -H "jsonpaths: [\"$.id\", \"$.city\"]" \ -T example.json \ http://FEHOST:HTTPPORT/api/DATABASE/TABLE/streamload ~~~ The `format: json` parameter allows you to execute the format of the imported data. `jsonpaths` is used to execute the corresponding data import path. Related parameters: jsonpaths: Select the JSON path for each column json\_root: Select the column where the JSON starts to be parsed strip\outer\array: Crop the outermost array field strict\_mode: Strictly filter for column type conversion during import When the JSON data schema and StarRocks data schema are not exactly the same, modify the `Jsonpath`. Sample data: ~~~json {"k1": 1, "k2": 2} ~~~ Import example: ~~~bash curl -v --location-trusted -u <username>:<password> \ -H "format: json" -H "jsonpaths: [\"$.k2\", \"$.k1\"]" \ -H "columns: k2, tmpk1, k1 = tmpk1 * 100" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/streamload ~~~ The ETL operation of multiplying k1 by 100 is performed during the import, and the column is matched with the original data by `Jsonpath`. The import results are as follows: ~~~plain text +++ | k1 | k2 | +++ | 100 | 2 | +++ ~~~ For missing columns, if the column definition is nullable, then `NULL` will be added, or the default value can be added by `ifnull`. Sample data: ~~~json [ {"k1": 1, "k2": "a"}, {"k1": 2}, {"k1": 3, "k2": "c"}, ] ~~~ Import Example-1: ~~~shell curl -v --location-trusted -u <username>:<password> \ -H "format: json" -H "stripouterarray: true" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/streamload ~~~ The import results are as follows: ~~~plain text +++ | k1 | k2 | +++ | 1 | a | +++ | 2 | NULL | +++ | 3 | c | +++ ~~~ Import Example-2: ~~~shell curl -v --location-trusted -u <username>:<password> \ -H "format: json" -H "stripouterarray: true" \ -H "jsonpaths: [\"$.k1\", \"$.k2\"]" \ -H "columns: k1, tmpk2, k2 = ifnull(tmpk2, 'x')" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/streamload ~~~ The import results are as follows: ~~~plain text +++ | k1 | k2 | +++ | 1 | a | +++ | 2 | x | +++ | 3 | c | +++ ~~~ Similar to stream load, the message content of Kafka data sources is treated as a complete JSON data. If a message contains multiple rows of data in array format, all rows will be imported and Kafka's offset will only be incremented by 1. If a JSON in Array format represents multiple rows of data, but the parsing of the JSON fails due to a JSON format error, the error row will only be incremented by 1 (given that the parsing fails, StarRocks cannot actually determine how many rows of data it contains, and can only record the error data as one row). is an open-source MySQL binlog synchronization tool from Alibaba, through which we can synchronize MySQL data to Kafka. The data is generated in JSON format in

Here is a demonstration of how to use routine load to synchronize data in Kafka for incremental data synchronization with MySQL. In MySQL we have a data table with the following table creation statement. ~~~sql CREATE TABLE `query_record` ( `query_id` varchar(64) NOT NULL, `conn_id` int(11) DEFAULT NULL, `fe_host` varchar(32) DEFAULT NULL, `user` varchar(32) DEFAULT NULL, `start_time` datetime NOT NULL, `end_time` datetime DEFAULT NULL, `time_used` double DEFAULT NULL, `state` varchar(16) NOT NULL, `error_message` text, `sql` text NOT NULL, `database` varchar(128) NOT NULL, `profile` longtext, `plan` longtext, PRIMARY KEY (`query_id`), KEY `idxstarttime` (`start_time`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ~~~ Prerequisite: Make sure MySQL has binlog enabled and the format is ROW. ~~~bash [mysqld] log-bin=mysql-bin # Enable binlog binlog-format=ROW # Select ROW mode server_id=1 # MySQL replication need to be defined, and do not duplicate canal's slaveId ~~~ Create an account and grant privileges to the secondary MySQL server: ~~~sql CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON . TO 'canal'@'%'; -- GRANT ALL PRIVILEGES ON . TO 'canal'@'%'; FLUSH PRIVILEGES; ~~~ Then download and install Canal. ~~~bash wget https://github.com/alibaba/canal/releases/download/canal-1.0.17/canal.deployer-1.0.17.tar.gz mkdir /tmp/canal tar zxvf canal.deployer-$version.tar.gz -C /tmp/canal ~~~ Modify the configuration (MySQL related). `$ vi conf/example/instance.properties` ~~~bash canal.instance.mysql.slaveId = 1234 canal.instance.master.address = 127.0.0.1:3306 canal.instance.master.journal.name = canal.instance.master.position = canal.instance.master.timestamp = canal.instance.dbUsername = canal canal.instance.dbPassword = canal canal.instance.defaultDatabaseName = canal.instance.connectionCharset = UTF-8 canal.instance.filter.regex = .\\\\\..\ canal.mq.dynamicTopic=databasename.query_record canal.mq.partitionHash= databasename.queryrecord:queryid ~~~ Modify the configuration (Kafka related). `$ vi /usr/local/canal/conf/canal.properties` ~~~bash canal.serverMode = kafka canal.mq.servers = 127.0.0.1:6667 canal.mq.retries = 0 canal.mq.batchSize = 16384 canal.mq.maxRequestSize = 1048576 canal.mq.lingerMs = 1 canal.mq.bufferMemory = 33554432 canal.mq.canalBatchSize = 50 canal.mq.canalGetTimeout = 100 canal.mq.flatMessage = false canal.mq.compressionType = none canal.mq.acks = all canal.mq.transaction = false ~~~ Initiation `bin/startup.sh` The corresponding synchronization log is shown in `logs/example/example.log` and in Kafka, with the following format: ~~~json { "data": [{ "query_id": "3c7ebee321e94773-b4d79cc3f08ca2ac", "conn_id": "34434", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.578", "end_time": "2020-10-19 20:40:10", "time_used": "1.0", "state": "FINISHED", "error_message": "", "sql": "COMMIT", "database": "", "profile": "", "plan": "" }, { "query_id": "7ff2df7551d64f8e-804004341bfa63ad", "conn_id": "34432", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.566", "end_time": "2020-10-19 20:40:10", "time_used": "0.0", "state": "FINISHED", "error_message": "", "sql": "COMMIT", "database": "", "profile": "", "plan": "" }, { "query_id": "3a4b35d1c1914748-be385f5067759134", "conn_id": "34440", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.601", "end_time": "1970-01-01 08:00:00", "time_used": "-1.0", "state": "RUNNING", "error_message": "", "sql": " SELECT SUM(length(locustkey)), SUM(length(ccustkey)) FROM lineorderstr INNER JOIN customerstr ON locustkey=ccustkey;", "database": "ssb", "profile": "", "plan": "" }], "database": "centerservicelihailei", "es": 1603111211000, "id": 122, "isDdl": false, "mysqlType": { "query_id": "varchar(64)", "conn_id": "int(11)", "fe_host": "varchar(32)", "user": "varchar(32)", "start_time": "datetime(3)", "end_time": "datetime", "time_used": "double", "state": "varchar(16)", "error_message": "text", "sql": "text", "database": "varchar(128)", "profile": "longtext", "plan": "longtext" }, "old": null, "pkNames": ["query_id"], "sql": "", "sqlType": { "query_id": 12, "conn_id": 4, "fe_host": 12, "user": 12, "start_time": 93, "end_time": 93, "time_used": 8, "state": 12, "error_message": 2005, "sql": 2005, "database": 12, "profile": 2005, "plan": 2005 }, "table": "query_record", "ts": 1603111212015, "type": "INSERT" } ~~~ Add `jsonroot` and `stripouter_array = true` to import data from `data`. ~~~sql create routine load manual.queryjob on queryrecord columns (queryid,connid,fehost,user,starttime,endtime,timeused,state,error_message,`sql`,`database`,profile,plan) PROPERTIES ( "format"="json", "json_root"="$.data", "desiredconcurrentnumber"="1", "stripouterarray" ="true", "maxerrornumber"="1000" ) FROM KAFKA ( "kafkabrokerlist"= "172.26.92.141:9092", "kafkatopic" = "databasename.queryrecord" ); ~~~ This completes the near real-time synchronization of data from MySQL to StarRocks. View status and error messages of the import job by `show routine load`.