Project: StarRocks (App Definition and Development / Database)
File: RoutineLoad.md

displayed_sidebar: "English" import InsertPrivNote from '../assets/commonMarkdown/insertPrivNote.md' This topic introduces how to create a Routine Load job to stream Kafka messages (events) into StarRocks, and familiarizes you with some basic concepts about Routine Load. To continuously load messages of a stream into StarRocks, you can store the message stream in a Kafka topic, and create a Routine Load job to consume the messages. The Routine Load job persists in StarRocks, generates a series of load tasks to consume the messages in all or part of the partitions in the topic, and loads the messages into StarRocks. A Routine Load job supports exactly-once delivery semantics to guarantee the data loaded into StarRocks is neither lost nor duplicated. Routine Load supports data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see and . <InsertPrivNote /> Routine Load now supports consuming CSV, JSON, and Avro (supported since v3.0.1) formatted data from a Kafka cluster. NOTE For CSV data, take note of the following points: You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. Load job A Routine Load job is a long-running job. As long as its status is RUNNING, a load job continuously generates one or multiple concurrent load tasks which consume the messages in a topic of a Kafka cluster and load the data into StarRocks. Load task A load job is split into multiple load tasks by certain rules. A load task is the basic unit of data loading. As an individual event, a load task implements the load mechanism based on . Multiple load tasks concurrently consume the messages from different partitions of a topic, and load the data into StarRocks. Create a Routine Load job. To load data from Kafka, you need to create a Routine Load job by running the statement. The FE parses the statement, and creates the job according to the properties you have specified. The FE splits the job into multiple load tasks. The FE split the job into multiple load tasks based on certain rules. Each load task is an individual transaction. The splitting rules are as follows: The FE calculates the actual concurrent number of the load tasks according to the desired concurrent number `desiredconcurrentnumber`, the partition number in the Kafka topic, and the number of the BE nodes that are alive. The FE splits the job into load tasks based on the actual concurrent number calculated, and arranges the tasks in the task queue. Each Kafka topic consists of multiple

The relation between the topic partition and the load task is as follows: A partition is uniquely assigned to a load task, and all messages from the partition are consumed by the load task. A load task can consume messages from one or more partitions. All partitions are distributed evenly among load tasks. Multiple load tasks run concurrently to consume the messages from multiple Kafka topic partitions, and load the data into StarRocks The FE schedules and submits load tasks: the FE schedules the load tasks in the queue on a timely basis, and assigns them to selected Coordinator BE nodes. The interval between load tasks is defined by the configuration item `maxbatchinterval`. The FE distributes the load tasks evenly to all BE nodes. See for more information about `maxbatchinterval`. The Coordinator BE starts the load task, consumes messages in partitions, parses and filters the data. A load task lasts until the pre-defined amount of messages are consumed or the pre-defined time limit is reached. The message batch size and time limit are defined in the FE configurations `maxroutineloadbatchsize` and `routineloadtaskconsumesecond`. For detailed information, see . The Coordinator BE then distributes the messages to the Executor BEs. The Executor BEs write the messages to disks. > NOTE > > StarRocks supports access to Kafka via security protocols including SASLSSL, SASPLAINTEXT, SSL, and PLAINTEXT. This topic uses connecting to Kafka via PLAINTEXT as an example. If you need to connect to Kafka via other security protocols, see . The FE generates new load tasks to load data continuously. After the Executor BEs has written the data to disks, the Coordinator BE reports the result of the load task to the FE. Based on the result, the FE then generates new load tasks to load the data continuously. Or the FE retries the failed tasks to make sure the data loaded into StarRocks is neither lost nor duplicated. The following three examples describe how to consume CSV-format, JSON-format and Avro-format data in Kafka, and load the data into StarRocks by creating a Routine Load job. For detailed syntax and parameter descriptions, see . This section describes how to create a Routine Load job to consume CSV-format data in a Kafka cluster, and load the data into StarRocks. Suppose there is a CSV-format dataset in the topic `ordertest1` in a Kafka cluster. Every message in the dataset includes six fields: order ID, payment date, customer name, nationality, gender, and price. ```Plain 2020050802,2020-05-08,Johann Georg Faust,Deutschland,male,895 2020050802,2020-05-08,Julien Sorel,France,male,893 2020050803,2020-05-08,Dorian Grey,UK,male,1262 2020050901,2020-05-09,Anna Karenina",Russia,female,175 2020051001,2020-05-10,Tess Durbeyfield,US,female,986 2020051101,2020-05-11,Edogawa Conan,japan,male,8924 ``` According to the fields of CSV-format data, create the table `exampletbl1` in the database `exampledb`. The following example creates a table with 5 fields excluding the field of customer gender in the CSV-format data. ```SQL CREATE TABLE exampledb.exampletbl1 ( `order_id` bigint NOT NULL COMMENT "Order ID", `pay_dt` date NOT NULL COMMENT "Payment date", `customer_name` varchar(26) NULL COMMENT "Customer name", `nationality` varchar(26) NULL COMMENT "Nationality", `price`double NULL COMMENT "Price" ) ENGINE=OLAP DUPLICATE KEY (orderid,paydt) DISTRIBUTED BY HASH(`order_id`); ``` NOTICE Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see

Execute the following statement to submit a Routine Load job named `exampletbl1ordertest1` to consume the messages in the topic `ordertest1` and load the data into the table `example_tbl1`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```SQL CREATE ROUTINE LOAD exampledb.exampletbl1ordertest1 ON exampletbl1 COLUMNS TERMINATED BY ",", COLUMNS (orderid, paydt, customername, nationality, tempgender, price) PROPERTIES ( "desiredconcurrentnumber" = "5" ) FROM KAFKA ( "kafkabrokerlist" = "<kafkabroker1ip>:<kafkabroker1port>,<kafkabroker2ip>:<kafkabroker2port>", "kafka_topic" = "ordertest1", "kafka_partitions" = "0,1,2,3,4", "property.kafkadefaultoffsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the statement to check the status of the load job. load job name There could be multiple load job on a table. Therefore, we recommend you name a load job with the corresponding Kafka topic and the time when the load job is submitted. It helps you distinguish the load job on each table. Column separator The property `COLUMN TERMINATED BY` defines the column separator of the CSV-format data. The default is `\t`. Kafka topic partition and offset You can specify the properties `kafkapartitions` and `kafkaoffsets` to specify the partitions and offsets to consume the messages. For example, if you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"` of the topic `ordertest1` all with the initial offsets, you can specify the properties as follows: If you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"`and you need to specify a separate starting offset for each partition, you can configure as follows: ```SQL "kafka_partitions" ="0,1,2,3,4", "kafkaoffsets" = "OFFSETBEGINNING, OFFSET_END, 1000, 2000, 3000" ``` You can also set the default offsets of all partitions with the property `property.kafkadefaultoffsets`. ```SQL "kafka_partitions" ="0,1,2,3,4", "property.kafkadefaultoffsets" = "OFFSET_BEGINNING" ``` For detailed information, see . Data mapping and transformation To specify the mapping and transformation relationship between the CSV-format data, and the StarRocks table, you need to use the `COLUMNS` parameter. Data mapping: StarRocks extracts the columns in the CSV-format data and maps them in sequence onto the fields declared in the `COLUMNS` parameter. StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them by name onto the columns of StarRocks table. Data transformation: And because the example excludes the column of customer gender from the CSV-format data, the field `tempgender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `exampletbl1` directly. For more information about data transformation, see . > NOTE > > You do not need to specify the `COLUMNS` parameter if the names, number, and order of the columns in the CSV-format data completely correspond to those of the StarRocks table. Task concurrency When there are many Kafka topic partitions and enough BE nodes, you can accelerate the loading by increasing the task concurrency. To increase the actual load task concurrency, you can increase the desired load task concurrency `desiredconcurrentnumber` when you create a routine load job. You can also set the dynamic configuration item of FE `maxroutineloadtaskconcurrentnum` ( default maximum load task currency ) to a larger value. For more information about `maxroutineloadtaskconcurrentnum`, please see

The actual task concurrency is defined by the minimum value among the number of BE nodes that are alive, the number of the pre-specified Kafka topic partitions, and the values of `desiredconcurrentnumber` and `maxroutineloadtaskconcurrent_num`. In the example, the number of BE nodes that are alive is `5`, the number of the pre-specified Kafka topic partitions is `5`, and the value of `maxroutineloadtaskconcurrentnum` is `5`. To increase the actual load task concurrency, you can increase the `desiredconcurrent_number` from the default value `3` to `5`. For more about the properties, see . For detailed instructions on accelerating the loading, see . This section describes how to create a Routine Load job to consume JSON-format data in a Kafka cluster, and load the data into StarRocks. Suppose there is a JSON-format dataset in the topic `ordertest2` in a Kafka cluster. The dataset includes six keys: commodity ID, customer name, nationality, payment time, and price. Besides, you want to transform the payment time column into the DATE type, and load it into the `pay_dt` column in the StarRocks table. ```JSON {"commodityid": "1", "customername": "Mark Twain", "country": "US","pay_time": 1589191487,"price": 875} {"commodityid": "2", "customername": "Oscar Wilde", "country": "UK","pay_time": 1589191487,"price": 895} {"commodityid": "3", "customername": "Antoine de Saint-Exupry","country": "France","pay_time": 1589191487,"price": 895} ``` CAUTION Each JSON object in a row must be in one Kafka message, otherwise a JSON parsing error is returned. According to the keys of the JSON-format data, create the table `exampletbl2` in the database `exampledb`. ```SQL CREATE TABLE `example_tbl2` ( `commodity_id` varchar(26) NULL COMMENT "Commodity ID", `customer_name` varchar(26) NULL COMMENT "Customer name", `country` varchar(26) NULL COMMENT "Country", `pay_time` bigint(20) NULL COMMENT "Payment time", `pay_dt` date NULL COMMENT "Payment date", `price`double SUM NULL COMMENT "Price" ) ENGINE=OLAP AGGREGATE KEY(`commodityid`,`customername`,`country`,`paytime`,`paydt`) DISTRIBUTED BY HASH(`commodity_id`); ``` NOTICE Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see . Execute the following statement to submit a Routine Load job named `exampletbl2ordertest2` to consume the messages in the topic `ordertest2` and load the data into the table `example_tbl2`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```SQL CREATE ROUTINE LOAD exampledb.exampletbl2ordertest2 ON exampletbl2 COLUMNS(commodityid, customername, country, paytime, price, paydt=fromunixtime(paytime, '%Y%m%d')) PROPERTIES ( "desiredconcurrentnumber" = "5", "format" = "json", "jsonpaths" = "[\"$.commodityid\",\"$.customername\",\"$.country\",\"$.pay_time\",\"$.price\"]" ) FROM KAFKA ( "kafkabrokerlist" ="<kafkabroker1ip>:<kafkabroker1port>,<kafkabroker2ip>:<kafkabroker2port>", "kafka_topic" = "ordertest2", "kafka_partitions" ="0,1,2,3,4", "property.kafkadefaultoffsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the statement to check the status of the load job. Data format You need to specify `"format" = "json"` in the clause `PROPERTIES` to define that the data format is JSON. Data mapping and transformation To specify the mapping and transformation relationship between the JSON-format data, and the StarRocks table, you need to specify the parameter `COLUMNS` and property`jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the JSON-format data, and the name of fields must match that of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the JSON data. These fields are then named by the property

Because the example needs to transform the payment time field to the DATE data type, and load the data into the `paydt` column in the StarRocks table, you need to use the fromunixtime function. The other fields are mapped to fields of the table `example_tbl2` directly. Data mapping: StarRocks extracts the `name` and `code` keys of JSON-format data and maps them onto the keys declared in the `jsonpaths` property. StarRocks extracts the keys declared in the `jsonpaths` property and maps them in sequence onto the fields declared in the `COLUMNS` parameter. StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them by name onto the columns of StarRocks table. Data transformation: Because the example needs to transform the key `paytime` to the DATE data type, and load the data into the `paydt` column in the StarRocks table, you need to use the fromunixtime function in `COLUMNS` parameter. The other fields are mapped to fields of the table `exampletbl2` directly. And because the example excludes the column of customer gender from the JSON-format data, the field `tempgender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `exampletbl1` directly. For more information about data transformation, see . > NOTE > > You do not need to specify the `COLUMNS` parameter if the names and number of the keys in the JSON object completely match those of fields in the StarRocks table. Since v3.0.1, StarRocks supports loading Avro data by using Routine Load. Create the following Avro schema file `avro_schema.avsc`: ```JSON { "type": "record", "name": "sensor_log", "fields" : [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"}, {"name": "checked", "type" : "boolean"}, {"name": "data", "type": "double"}, {"name": "sensortype", "type": {"type": "enum", "name": "sensortype_enum", "symbols" : ["TEMPERATURE", "HUMIDITY", "AIR-PRESSURE"]}} ] } ``` Register the Avro schema in the . Prepare the Avro data and send it to the Kafka topic `topic_0`. According to the fields of Avro data, create a table `sensorlog` in the target database `exampledb` in the StarRocks cluster. The column names of the table must match the field names in the Avro data. For the data type mapping between the table columns and the Avro data fields, see . ```SQL CREATE TABLE exampledb.sensorlog ( `id` bigint NOT NULL COMMENT "sensor id", `name` varchar(26) NOT NULL COMMENT "sensor name", `checked` boolean NOT NULL COMMENT "checked", `data` double NULL COMMENT "sensor data", `sensor_type` varchar(26) NOT NULL COMMENT "sensor type" ) ENGINE=OLAP DUPLICATE KEY (id) DISTRIBUTED BY HASH(`id`); ``` NOTICE Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see . Execute the following statement to submit a Routine Load job named `sensorlogloadjob` to consume the Avro messages in the Kafka topic `topic0` and load the data into the table `sensor_log` in the database `sensor`. The load job consumes the messages from the initial offset in the specified partitions of the topic. ```SQL CREATE ROUTINE LOAD exampledb.sensorlogloadjob ON sensor_log PROPERTIES ( "format" = "avro" ) FROM KAFKA ( "kafkabrokerlist" = "<kafkabroker1ip>:<kafkabroker1port>,<kafkabroker2ip>:<kafkabroker2port>,...", "confluent.schema.registry.url" = "http://172.xx.xxx.xxx:8081", "kafkatopic" = "topic0", "kafka_partitions" = "0,1,2,3,4,5",

= "OFFSET_BEGINNING" ); ``` Data Format You need to specify `"format = "avro"` in the clause `PROPERTIES` to define that the data format is Avro. Schema Registry You need to configure `confluent.schema.registry.url` to specify the URL of the Schema Registry where the Avro schema is registered. StarRocks retrieves the Avro schema by using this URL. The format is as follows: ```Plaintext confluent.schema.registry.url = http[s]://[<schema-registry-api-key>:<schema-registry-api-secret>@]<hostname|ip address>[:<port>] ``` Data mapping and transformation To specify the mapping and transformation relationship between the Avro-format data and the StarRocks table, you need to specify the parameter `COLUMNS` and property `jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the fields in the property `jsonpaths`, and the names of fields must match these of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the Avro data. These fields are then named by the property `COLUMNS`. For more information about data transformation, see . > NOTE > > You do not need to specify the `COLUMNS` parameter if the names and number of the fields in the Avro record completely match those of columns in the StarRocks table. After submitting the load job, you can execute the statement to check the status of the load job. The data type mapping between the Avro data fields you want to load and the StarRocks table columns is as follows: | Avro | StarRocks | | - | | | nul | NULL | | boolean | BOOLEAN | | int | INT | | long | BIGINT | | float | FLOAT | | double | DOUBLE | | bytes | STRING | | string | STRING | | Avro | StarRocks | | -- | | | record | Load the entire RECORD or its subfields into StarRocks as JSON. | | enums | STRING | | arrays | ARRAY | | maps | JSON | | union(T, null) | NULLABLE(T) | | fixed | STRING | Currently, StarRocks does not support schema evolution. Each Kafka message must only contain a single Avro data record. Execute the statement to check the status of the load job `exampletbl2ordertest2`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. If the state of the load job is automatically changed to PAUSED, it is possibly because the number of error rows has exceeded the threshold. For detailed instructions on setting this threshold, see . You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. Having fixed the problem, you can then execute the statement to resume the PAUSED load job. If the state of the load job is CANCELLED, it is possibly because the load job encounters an exception (such as the table has been dropped). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. However, you cannot resume a CANCELLED load job. ```SQL MySQL [exampledb]> SHOW ROUTINE LOAD FOR exampletbl2_ordertest2 \G row * Id: 63013 Name: exampletbl2ordertest2 CreateTime: 2022-08-10 17:09:00 PauseTime: NULL EndTime: NULL DbName: defaultcluster:exampledb TableName: example_tbl2 State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 3 JobProperties: {"partitions":"","partial_update":"false","columnToColumnExpr":"commodity_id,customer_name,country,pay_time,pay_dt=from_unixtime(`pay_time`,

DataSourceProperties: {"topic":"ordertest2","currentKafkaPartitions":"0,1,2,3,4","brokerList":"<kafkabroker1ip>:<kafkabroker1port>,<kafkabroker2ip>:<kafkabroker2port>"} CustomProperties: {"kafkadefaultoffsets":"OFFSET_BEGINNING"} Statistic: {"receivedBytes":230,"errorRows":0,"committedTaskNum":1,"loadedRows":2,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":2,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":522} Progress: {"0":"1","1":"OFFSETZERO","2":"OFFSETZERO","3":"OFFSETZERO","4":"OFFSETZERO"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: ``` CAUTION You cannot check a load job that has stopped or has not yet started. Execute the statement to check the load tasks of the load job `exampletbl2ordertest2`, such as how many tasks are currently running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```SQL MySQL [exampledb]> SHOW ROUTINE LOAD TASK WHERE JobName = "exampletbl2_ordertest2" \G row * TaskId: 18c3a823-d73e-4a64-b9cb-b9eced026753 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"1":0,"4":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again row * TaskId: f76c97ac-26aa-4b41-8194-a8ba2063eb00 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:26 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"2":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again row * TaskId: 1a327a34-99f4-4f8d-8014-3cd38db99ec6 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:26 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"0":2,"3":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again ``` You can execute the statement to pause a load job. The state of the load job will be PAUSED after the statement is executed. However, it has not stopped. You can execute the statement to resume it. You can also check its status with the statement. The following example pauses the load job `exampletbl2ordertest2`: ```SQL PAUSE ROUTINE LOAD FOR exampletbl2ordertest2; ``` You can execute the statement to resume a paused load job. The state of the load job will be NEED_SCHEDULE temporarily (because the load job is being re-scheduled), and then become RUNNING. You can check its status with the statement. The following example resumes the paused load job `exampletbl2ordertest2`: ```SQL RESUME ROUTINE LOAD FOR exampletbl2ordertest2; ``` Before altering a load job, you must pause it with the statement. Then you can execute the . After altering it, you can execute the statement to resume it, and check its status with the statement. Suppose the number of the BE nodes that are alive increases to `6` and the Kafka topic partitions to be consumed is `"0,1,2,3,4,5,6,7"`. If you want to increase the actual load task concurrency, you can execute the following statement to increase the number of desired task concurrency `desiredconcurrentnumber` to `6` (greater than or equal to the number of BE nodes that are alive), and specify the Kafka topic partitions and initial offsets. NOTE Because the actual task concurrency is determined by the minimum value of multiple parameters, you must make sure that the value of the FE dynamic parameter `maxroutineloadtaskconcurrent_num` is greater than or equal to `6`. ```SQL ALTER ROUTINE LOAD FOR exampletbl2ordertest2 PROPERTIES ( "desiredconcurrentnumber" = "6" ) FROM kafka ( "kafka_partitions" = "0,1,2,3,4,5,6,7", "kafkaoffsets" = "OFFSETBEGINNING,OFFSETBEGINNING,OFFSETBEGINNING,OFFSETBEGINNING,OFFSETEND,OFFSETEND,OFFSETEND,OFFSET_END" ); ``` You can execute the statement to stop a load job. The state of the load job will be STOPPED after the statement is executed, and you cannot resume a stopped load job. You cannot check the status of a stopped load job with the statement. The following example stops the load job `exampletbl2ordertest2`: ```SQL STOP ROUTINE LOAD FOR exampletbl2ordertest2; ``` Please see .