Project: StarRocks (App Definition and Development / Database)
File: automq-routine-load.md

displayed_sidebar: "English" description: Cloud based Kafka from AutoMQ import Replicanum from '../assets/commonMarkdown/replicanum.md' is a cloud-native version of Kafka redesigned for cloud environments. AutoMQ Kafka is and fully compatible with the Kafka protocol, fully leveraging cloud benefits. Compared to self-managed Apache Kafka, AutoMQ Kafka, with its cloud-native architecture, offers features like capacity auto scaling, self-balancing of network traffic, move partition in seconds. These features contribute to a significantly lower Total Cost of Ownership (TCO) for users. This article will guide you through importing data into AutoMQ Kafka using StarRocks Routine Load. For an understanding of the basic principles of Routine Load, refer to the section on Routine Load Fundamentals. Ensure you have a running StarRocks cluster. For demonstration purposes, this article follow the to install a StarRocks cluster on a Linux machine via Docker. Creating a database and a Primary Key table for testing: ```sql create database automq_db; create table users ( id bigint NOT NULL, name string NOT NULL, timestamp string NULL, status string NULL ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id) PROPERTIES ( "enablepersistentindex" = "true" ); ``` <Replicanum /> To prepare your AutoMQ Kafka environment and test data, follow the AutoMQ guide to deploy your AutoMQ Kafka cluster. Ensure that StarRocks can directly connect to your AutoMQ Kafka server. To quickly create a topic named `example_topic` in AutoMQ Kafka and write a test JSON data into it, follow these steps: Use Kafkas command-line tools to create a topic. Ensure you have access to the Kafka environment and the Kafka service is running. Here is the command to create a topic: ```shell ./kafka-topics.sh --create --topic example_topic --bootstrap-server 10.0.96.4:9092 --partitions 1 --replication-factor 1 ``` Note: Replace `topic` and `bootstrap-server` with your Kafka server address. To check the result of the topic creation, use this command: ```shell ./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092 ``` Generate a simple JSON format test data ```json { "id": 1, "name": "testuser", "timestamp": "2023-11-10T12:00:00", "status": "active" } ``` Use Kafkas command-line tools or programming methods to write test data into

Here is an example using command-line tools: ```shell echo '{"id": 1, "name": "testuser", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic ``` Note: Replace `topic` and `bootstrap-server` with your Kafka server address. To view the recently written topic data, use the following command: ```shell sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning ``` In the StarRocks command line, create a Routine Load task to continuously import data from the AutoMQ Kafka topic: ```sql CREATE ROUTINE LOAD automqexampleload ON users COLUMNS(id, name, timestamp, status) PROPERTIES ( "desiredconcurrentnumber" = "5", "format" = "json", "jsonpaths" = "[\"$.id\",\"$.name\",\"$.timestamp\",\"$.status\"]" ) FROM KAFKA ( "kafkabrokerlist" = "10.0.96.4:9092", "kafkatopic" = "exampletopic", "kafka_partitions" = "0", "property.kafkadefaultoffsets" = "OFFSET_BEGINNING" ); ``` Note: Replace `kafkabrokerlist` with your Kafka server address. Specify the data format as JSON in the "format" = "json" of the PROPERTIES clause. To specify the mapping and transformation relationship between the source data and the target table, configure the COLUMNS and jsonpaths parameters. The column names in COLUMNS correspond to the column names of the target table, and their order corresponds to the column order in the source data. The jsonpaths parameter is used to extract the required field data from JSON data, similar to newly generated CSV data. Then the COLUMNS parameter temporarily names the fields in jsonpaths in order. For more explanations on data transformation, please see . Note: If each JSON object per line has key names and quantities (order is not required) that correspond to the columns of the target table, there is no need to configure COLUMNS. First, we check the Routine Load import job and confirm the Routine Load import task status is in RUNNING status. ```sql show routine load\G ``` Then, querying the corresponding table in the StarRocks database, we can observe that the data has been successfully imported. ```sql StarRocks > select * from users; ++--++--+ | id | name | timestamp | status | ++--++--+ | 1 | testuser | 2023-11-10T12:00:00 | active | | 2 | testuser | 2023-11-10T12:00:00 | active | ++--++--+ 2 rows in set (0.01 sec) ```