Project: SeaTunnel (App Definition and Development / Streaming & Messaging)
File: StarRocks.md

StarRocks source connector Read external data source data through StarRocks. The internal implementation of StarRocks source connector is obtains the query plan from the frontend (FE), delivers the query plan as a parameter to BE nodes, and then obtains data results from BE nodes. | name | type | required | default value | |-|--|-|-| | node_urls | list | yes | - | | username | string | yes | - | | password | string | yes | - | | database | string | yes | - | | table | string | yes | - | | scan_filter | string | no | - | | schema | config | yes | - | | requesttabletsize | int | no | Integer.MAX_VALUE | | scanconnecttimeout_ms | int | no | 30000 | | scanquerytimeout_sec | int | no | 3600 | | scankeepalive_min | int | no | 10 | | scanbatchrows | int | no | 1024 | | scanmemlimit | long | no | 2147483648 | | max_retries | int | no | 3 | | scan.params.* | string | no | - | `StarRocks` cluster address, the format is `["feip:fehttp_port", ...]` `StarRocks` user username `StarRocks` user password The name of StarRocks database The name of StarRocks table Filter expression of the query, which is transparently transmitted to StarRocks. StarRocks uses this expression to complete source-side data filtering. e.g. ``` "tinyint_1 = 100" ``` The schema of the starRocks that you want to generate e.g. ``` schema { fields { name = string age = int } } ``` The number of StarRocks Tablets corresponding to an Partition. The smaller this value is set, the more partitions will be generated. This will increase the parallelism on the engine side, but at the same time will cause greater pressure on

The following is an example to explain how to use requesttabletsize to controls the generation of partitions ``` the tablet distribution of StarRocks table in cluster as follower benode1 tablet[1, 2, 3, 4, 5] benode2 tablet[6, 7, 8, 9, 10] benode3 tablet[11, 12, 13, 14, 15] 1.If not set requesttabletsize, there will no limit on the number of tablets in a single partition. The partitions will be generated as follows partition[0] read data of tablet[1, 2, 3, 4, 5] from benode1 partition[1] read data of tablet[6, 7, 8, 9, 10] from benode2 partition[2] read data of tablet[11, 12, 13, 14, 15] from benode3 2.if set requesttabletsize=3, the limit on the number of tablets in a single partition is 3. The partitions will be generated as follows partition[0] read data of tablet[1, 2, 3] from benode1 partition[1] read data of tablet[4, 5] from benode1 partition[2] read data of tablet[6, 7, 8] from benode2 partition[3] read data of tablet[9, 10] from benode2 partition[4] read data of tablet[11, 12, 13] from benode3 partition[5] read data of tablet[14, 15] from benode3 ``` requests connection timeout sent to StarRocks Query the timeout time of StarRocks, the default value is 1 hour, -1 means no timeout limit The keep-alive duration of the query task, in minutes. The default value is 10. we recommend that you set this parameter to a value greater than or equal to 5. The maximum number of data rows to read from BE at a time. Increasing this value reduces the number of connections established between engine and StarRocks and therefore mitigates overhead caused by network latency. The maximum memory space allowed for a single query in the BE node, in bytes. The default value is 2147483648 (2 GB). number of retry requests sent to StarRocks The parameter of the scan data from be ``` source { StarRocks { nodeUrls = ["starrocks_e2e:8030"] username = root password = "" database = "test" table = "e2etablesource" scanbatchrows = 10 max_retries = 3 fields { BIGINT_COL = BIGINT LARGEINT_COL = STRING SMALLINT_COL = SMALLINT TINYINT_COL = TINYINT BOOLEAN_COL = BOOLEAN DECIMAL_COL = "DECIMAL(20, 1)" DOUBLE_COL = DOUBLE FLOAT_COL = FLOAT INT_COL = INT CHAR_COL = STRING VARCHAR11COL = STRING STRING_COL = STRING DATETIME_COL = TIMESTAMP DATE_COL = DATE } scan.params.scannerthreadpoolthreadnum = "3" } } ``` Add StarRocks Source Connector