Project: StarRocks (App Definition and Development / Database)
File: load_from_pulsar.md

displayed_sidebar: "English" As of StarRocks version 2.5, Routine Load supports continuously loading data from Apache Pulsar. Pulsar is distributed, open source pub-sub messaging and streaming platform with a store-compute separation architecture. Loading data from Pulsar via Routine Load is similar to loading data from Apache Kafka. This topic uses CSV-formatted data as an example to introduce how to load data from Apache Pulsar via Routine Load. Routine Load supports consuming CSV and JSON formatted data from a Pulsar cluster. NOTE As for data in CSV format, StarRocks supports UTF-8 encoded strings within 50 bytes as column separators. Commonly used column separators include comma (,), tab and pipe (|). Topics in Pulsar are named channels for transmitting messages from producers to consumers. Topics in Pulsar are divided into partitioned topics and non-partitioned topics. are a special type of topic that are handled by multiple brokers, thus allowing for higher throughput. A partitioned topic is actually implemented as N internal topics, where N is the number of partitions. Non-partitioned topics are a normal type of topic that are served only by a single broker, which limits the maximum throughput of the topic. The message ID of a message is assigned by as soon as the message is persistently stored. Message ID indicates a message' s specific position in a ledger and is unique within a Pulsar cluster. Pulsar supports consumers specifying the initial position through consumer.seek(messageId). But compared to the Kafka consumer offset which is a long integer value, the message ID consists of four parts: `ledgerId:entryID:partition-index:batch-index`. Therefore, you can't get the Message ID directly from the message. As a result, at present, Routine Load does not support specifying initial position when loading data from Pulsar, but only supports consuming data from the beginning or end of a partition. A subscription is a named configuration rule that determines how messages are delivered to consumers. Pulsar also supports consumers simultaneously subscribing to multiple topics. A topic can have multiple subscriptions. The type of a subscription is defined when a consumer connects to it, and the type can be changed by restarting all consumers with a different configuration. Four subscription types are available in Pulsar: `exclusive` (default): Only a single consumer is allowed to attach to the subscription. Only one customer is allowed to consume messages. `shared`: Multiple consumers can attach to the same subscription. Messages are delivered in a round robin distribution across consumers, and any given message is delivered to only one

`failover`: Multiple consumers can attach to the same subscription. A master consumer is picked for a non-partitioned topic or each partition of a partitioned topic and receives messages. When the master consumer disconnects, all (non-acknowledged and subsequent) messages are delivered to the next consumer in line. `key_shared`: Multiple consumers can attach to the same subscription. Messages are delivered in a distribution across consumers and message with same key or same ordering key are delivered to only one consumer. Note: Currently Routine Load uses exclusive type. The following examples describe how to consume CSV-formatted messages in Pulsar, and load the data into StarRocks by creating a Routine Load job. For detailed instruction and reference, see . ```SQL CREATE ROUTINE LOAD loadtest.routinewikiedit1 ON routinewikiedit COLUMNS TERMINATED BY ",", ROWS TERMINATED BY "\n", COLUMNS (orderid, paydt, customername, nationality, tempgender, price) WHERE event_time > "2022-01-01 00:00:00", PROPERTIES ( "desiredconcurrentnumber" = "1", "maxbatchinterval" = "15000", "maxerrornumber" = "1000" ) FROM PULSAR ( "pulsarserviceurl" = "pulsar://localhost:6650", "pulsar_topic" = "persistent://tenant/namespace/topic-name", "pulsar_subscription" = "load-test", "pulsar_partitions" = "load-partition-0,load-partition-1", "pulsarinitialpositions" = "POSITIONEARLIEST,POSITIONLATEST", "property.auth.token" = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD5Y" ); ``` When Routine Load is created to consume data from Pulsar, most input parameters except for `datasourceproperties` are the same as consuming data from Kafka . For descriptions about parameters except datasourceproperties `datasourceproperties` , see . The parameters related to `datasourceproperties` and their descriptions are as follows: | Parameter | Required | Description | | - | | | | pulsarserviceurl | Yes | The URL that is used to connect to the Pulsar cluster. Format: `"pulsar://ip:port"` or `"pulsar://service:port"`.Example: `"pulsarserviceurl" = "pulsar://``localhost:6650``"` | | pulsartopic | Yes | Subscribed topic. Example: "pulsartopic" = "persistent://tenant/namespace/topic-name" | | pulsarsubscription | Yes | Subscription configured for the topic.Example: `"pulsarsubscription" = "my_subscription"` | | pulsarpartitions, pulsarinitialpositions | No | `pulsarpartitions` : Subscribed partitions in the topic.`pulsarinitialpositions`: initial positions of partitions specified by `pulsarpartitions`. The initial positions must correspond to the partitions in `pulsarpartitions`. Valid values:`POSITIONEARLIEST` (Default value): Subscription starts from the earliest available message in the partition. `POSITIONLATEST`: Subscription starts from the latest available message in the partition.Note:If `pulsarpartitions` is not specified, the topic's all partitions are subscribed.If both `pulsarpartitions` and `property.pulsardefaultinitialposition` are specified, the `pulsarpartitions` value overrides `property.pulsardefaultinitialposition` value.If neither `pulsarpartitions` nor `property.pulsardefaultinitialposition` is specified, subscription starts from the latest available message in the partition.Example:`"pulsarpartitions" = "my-partition-0,my-partition-1,my-partition-2,my-partition-3", "pulsarinitialpositions" = "POSITIONEARLIEST,POSITIONEARLIEST,POSITIONLATEST,POSITIONLATEST"` | Routine Load supports the following custom parameters for Pulsar. | Parameter | Required | Description | | - | -- | | |

| No | The default initial positions when the topic's partitions are subscribed. The parameter takes effect when `pulsarinitialpositions` is not specified. Its valid values are the same as the valid values of `pulsarinitialpositions`.Example: `"``property.pulsardefaultinitialposition" = "POSITION_EARLIEST"` | | property.auth.token | No | If Pulsar enables authenticating clients using security tokens, you need the token string to verify your identity.Example: `"p``roperty.auth.token" = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD"` | Execute the statement to check the status of the load job `routinewikiedit_1`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. When you check a Routine Load job that consumes data from Pulsar, most returned parameters except for `progress` are the same as consuming data from Kafka. `progress` refers to backlog, that is the number of unacked messages in a partition. ```Plaintext MySQL [loadtest] > SHOW ROUTINE LOAD for routinewikiedit1 \G row * Id: 10142 Name: routinewikiedit_1 CreateTime: 2022-06-29 14:52:55 PauseTime: 2022-06-29 17:33:53 EndTime: NULL DbName: defaultcluster:testpulsar TableName: test1 State: PAUSED DataSourceType: PULSAR CurrentTaskNum: 0 JobProperties: {"partitions":"","rowDelimiter":"'\n'","partial_update":"false","columnToColumnExpr":"","maxBatchIntervalS":"10","whereExpr":"*","timezone":"Asia/Shanghai","format":"csv","columnSeparator":"','","jsonroot":"","strictmode":"false","jsonpaths":"","desireTaskConcurrentNum":"3","maxErrorNum":"10","stripouterarray":"false","currentTaskConcurrentNum":"0","maxBatchRows":"200000"} DataSourceProperties: {"serviceUrl":"pulsar://localhost:6650","currentPulsarPartitions":"my-partition-0,my-partition-1","topic":"persistent://tenant/namespace/topic-name","subscription":"load-test"} CustomProperties: {"auth.token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD"} Statistic: {"receivedBytes":5480943882,"errorRows":0,"committedTaskNum":696,"loadedRows":66243440,"loadRowsRate":29000,"abortedTaskNum":0,"totalRows":66243440,"unselectedRows":0,"receivedBytesRate":2400000,"taskExecuteTimeMs":2283166} Progress: {"my-partition-0(backlog): 100","my-partition-1(backlog): 0"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: 1 row in set (0.00 sec) ``` Execute the statement to check the load tasks of the load job `routinewikiedit_1`, such as how many tasks are running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```SQL MySQL [exampledb]> SHOW ROUTINE LOAD TASK WHERE JobName = "routinewikiedit1" \G ``` Before altering a load job, you must pause it by using the statement. Then you can execute the . After altering it, you can execute the statement to resume it, and check its status by using the statement. When Routine Load is used to consume data from Pulsar, most returned parameters except for `datasourceproperties` are the same as consuming data from Kafka. Take note of the following points: Among the `datasourceproperties` related parameters, only `pulsarpartitions`, `pulsarinitialpositions`, and custom Pulsar parameters `property.pulsardefaultinitialposition` and `property.auth.token` are currently supported to be modified. The parameters `pulsarserviceurl`, `pulsartopic`, and `pulsarsubscription` cannot be modified. If you need to modify the partition to be consumed and the matched initilal position, you need to make sure that you specify the partition using `pulsarpartitions` when you create the Routine Load job, and only the intial position `pulsarinitial_positions` of the specified partition can be modified. If you specify only Topic `pulsartopic` when creating a Routine Load job, but not partitions `pulsarpartitions`, you can modify the starting position of all partitions under topic via `pulsardefaultinitial_position`.