Project: StarRocks (App Definition and Development / Database)
File: shared-data.md

displayed_sidebar: "English" sidebar_position: 2 description: Separate compute and storage import DDL from '../assets/quick-start/_DDL.mdx' import Clients from '../assets/quick-start/_clientsCompose.mdx' import SQL from '../assets/quick-start/_SQL.mdx' import Curl from '../assets/quick-start/_curl.mdx' In systems that separate storage from compute data is stored in low-cost reliable remote storage systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage like MinIO. Hot data is cached locally and When the cache is hit, the query performance is comparable to that of storage-compute coupled architecture. Compute nodes (CN) can be added or removed on demand within seconds. This architecture reduces storage cost, ensures better resource isolation, and provides elasticity and scalability. This tutorial covers: Running StarRocks in Docker containers Using MinIO for Object Storage Configuring StarRocks for shared-data Loading two public datasets Analyzing the data with SELECT and JOIN Basic data transformation (the T in ETL) The data used is provided by NYC OpenData and the National Centers for Environmental Information at NOAA. Both of these datasets are very large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run the Docker image and load this data on a machine with 4 GB RAM assigned to Docker. For larger fault-tolerant and scalable deployments we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with the step by step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: Allow the reader to load data in a shared-data deployment and analyze that data. Provide the configuration details for shared-data deployments. Explain the basics of data transformation during loading. 4GB RAM assigned to Docker 10 GB free disk space assigned to Docker You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work, and this guide covers the configuration of DBeaver and MySQL WorkBench. `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, . Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. Compute Nodes are responsible for executing query plans in shared-data deployments. Backend nodes are responsible for both data storage and executing query plans in shared-nothing deployments. :::note This guide does not use BEs, this information is included here so that you understand the difference between BEs and CNs. ::: To run StarRocks with shared-data using Object Storage we need: A frontend engine (FE) A compute node (CN) Object Storage This guide uses MinIO, which is S3 compatible Object Storage provided under the GNU Affero General Public

In order to provide an environment with the three necessary containers StarRocks provides a Docker compose file. ```bash mkdir quickstart cd quickstart curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/docker-compose.yml ``` ```bash docker compose up -d ``` Check the progress of the services. It should take around 30 seconds for the FE and CN to become healthy. The MinIO container will not show a health indicator, but you will be using the MinIO web UI and that will verify its health. Run `docker compose ps` until MinIO, the FE, and the CN services show a status of `healthy`: ```bash docker compose ps ``` ```plaintext SERVICE CREATED STATUS PORTS minio 32 seconds ago Up 32 seconds (healthy) 0.0.0.0:9000-9001->9000-9001/tcp starrocks-cn 32 seconds ago Up 31 seconds (healthy) 0.0.0.0:8040->8040/tcp starrocks-fe 32 seconds ago Up 31 seconds (healthy) 0.0.0.0:8030->8030/tcp, 0.0.0.0:9020->9020/tcp, 0.0.0.0:9030->9030/tcp ``` In order to use MinIO for Object Storage with StarRocks, StarRocks needs a MinIO access key. The access key was generated during the startup of the Docker services. To help you better understand the way that StarRocks connects to MinIO you should verify that the key exists. Browse to http://localhost:9001/access-keys The username and password are specified in the Docker compose file, and are `miniouser` and `miniopassword`. You should see that there is one access key. The Key is `AAAAAAAAAAAAAAAAAAAA`, you cannot see the secret in the MinIO Console, but it is in the Docker compose file and is `BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`: <Clients /> Download these two datasets to your FE container. Open a shell and create a directory for the downloaded files: ```bash docker compose exec starrocks-fe bash ``` ```bash mkdir quickstart cd quickstart ``` ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPDCrashData.csv ``` ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` At this point you have StarRocks running, and you have MinIO running. The MinIO access key is used to connect StarRocks and Minio. When StarRocks started up it established the connection with MinIO and created the default storage volume in MinIO. This is the configuration used to set the default storage volume to use MinIO (this is also in the Docker compose file). The configuration will be described in detail at the end of this guide, for now just note that the `awss3accesskey` is set to the string that you saw in the MinIO Console and that the `runmode` is set to `shared_data`: ```plaintext runmode = shareddata cloudnativestorage_type = S3 awss3endpoint = minio:9000 awss3path = starrocks awss3access_key = AAAAAAAAAAAAAAAAAAAA awss3secret_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB awss3useinstanceprofile = false awss3useawssdkdefaultbehavior = false enableloadvolumefromconf = true ``` :::tip Run this command from the directory containing the `docker-compose.yml` file. If you are using a client other than the mysql CLI, open that now. ::: ```sql docker compose exec starrocks-fe \ mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > " ``` ```sql SHOW STORAGE VOLUMES; ``` ```plaintext ++ | Storage Volume | ++ | builtinstoragevolume | ++ 1 row in set (0.00 sec) ``` ```sql DESC STORAGE VOLUME builtinstoragevolume\G ``` :::tip Some of the SQL in this document, and many other documents in the StarRocks documentation, and with `\G` instead of a semicolon. The `\G` causes the mysql CLI to render the query results

Many SQL clients do not interpret vertical formatting output, so you should replace `\G` with `;`. ::: ```plaintext row * Name: builtinstoragevolume Type: S3 IsDefault: true Location: s3://starrocks Params: {"aws.s3.accesskey":"******","aws.s3.secretkey":"","aws.s3.endpoint":"minio:9000","aws.s3.region":"","aws.s3.useinstanceprofile":"false","aws.s3.useawssdkdefaultbehavior":"false"} Enabled: true Comment: 1 row in set (0.03 sec) ``` Verify that the parameters match the configuration. :::note The folder `builtinstoragevolume` will not be visible in the MinIO object list until data is written to the bucket. ::: <DDL /> There are many ways to load data into StarRocks. For this tutorial the simplest way is to use curl and StarRocks Stream Load. :::tip Run these curl commands from the FE shell in the directory where you downloaded the dataset. You will be prompted for a password. You probably have not assigned a password to the MySQL `root` user, so just hit enter. ::: The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ```bash curl --location-trusted -u root \ -T ./NYPDCrashData.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "maxfilterratio:1" \ -H "columns:tmpCRASHDATE, tmpCRASHTIME, CRASHDATE=strtodate(concatws(' ', tmpCRASHDATE, tmpCRASHTIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIPCODE,LATITUDE,LONGITUDE,LOCATION,ONSTREETNAME,CROSSSTREETNAME,OFFSTREETNAME,NUMBEROFPERSONSINJURED,NUMBEROFPERSONSKILLED,NUMBEROFPEDESTRIANSINJURED,NUMBEROFPEDESTRIANSKILLED,NUMBEROFCYCLISTINJURED,NUMBEROFCYCLISTKILLED,NUMBEROFMOTORISTINJURED,NUMBEROFMOTORISTKILLED,CONTRIBUTINGFACTORVEHICLE1,CONTRIBUTINGFACTORVEHICLE2,CONTRIBUTINGFACTORVEHICLE3,CONTRIBUTINGFACTORVEHICLE4,CONTRIBUTINGFACTORVEHICLE5,COLLISIONID,VEHICLETYPECODE1,VEHICLETYPECODE2,VEHICLETYPECODE3,VEHICLETYPECODE4,VEHICLETYPECODE5" \ -XPUT http://localhost:8030/api/quickstart/crashdata/streamload ``` Here is the output of the above command. The first highlighted section shown what you should expect to see (OK and all but one row inserted). One row was filtered out because it does not contain the correct number of columns. ```bash Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 1013, "BeginTxnTimeMs": 21, "StreamLoadPlanTimeMs": 63, "ReadDataTimeMs": 563, "WriteDataTimeMs": 870, "CommitAndPublishTimeMs": 57, "ErrorURL": "http://10.5.0.3:8040/api/loaderrorlog?file=errorlogda41dd88276a7bfc739087c94262ae9f" }% ``` If there was an error the output provides a URL to see the error messages. Because the container has a private IP address you will have to view it by running curl from the container. ```bash curl http://10.5.0.3:8040/api/loaderror_log<details from ErrorURL> ``` Expand the summary for the content seen while developing this tutorial: <details> <summary>Reading error messages in the browser</summary> ```bash Error: Value count does not match column count. Expect 29, but got 32. Column delimiter: 44,Row delimiter: 10.. Row: 09/06/2015,14:15,,,40.6722269,-74.0110059,"(40.6722269, -74.0110059)",,,"R/O 1 BEARD ST. ( IKEA'S 09/14/2015,5:30,BRONX,10473,40.814551,-73.8490955,"(40.814551, -73.8490955)",TORRY AVENUE ,NORTON AVENUE ,,0,0,0,0,0,0,0,0,Driver Inattention/Distraction,Unspecified,,,,3297457,PASSENGER VEHICLE,PASSENGER VEHICLE,,, ``` </details> Load the weather dataset in the same manner as you loaded the crash data. ```bash curl --location-trusted -u root \ -T

\ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "maxfilterratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://localhost:8030/api/quickstart/weatherdata/streamload ``` Open MinIO and verify that you have `data`, `metadata`, and `schema` entries in each of the directories under `starrocks/shared/` :::tip The folder names below `starrocks/shared/` are generated when you load the data. You should see a single directory below `shared`, and then two more below that. Inside each of those directories you will find the data, metadata, and schema entries. ::: <SQL /> Now that you have experienced using StarRocks with shared-data it is important to understand the configuration. The CN configuration used here is the default, as the CN is designed for shared-data use. The default configuration is shown below. You do not need to make any changes. ```bash sysloglevel = INFO be_port = 9060 behttpport = 8040 heartbeatserviceport = 9050 brpc_port = 8060 starlet_port = 9070 ``` The FE configuration is slightly different from the default as the FE must be configured to expect that data is stored in Object Storage rather than on local disks on BE nodes. The `docker-compose.yml` file generates the FE configuration in the `command`. ```plaintext runmode = shareddata cloudnativestorage_type = S3 awss3endpoint = minio:9000 awss3path = starrocks awss3access_key = AAAAAAAAAAAAAAAAAAAA awss3secret_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB awss3useinstanceprofile = false awss3useawssdkdefaultbehavior = false enableloadvolumefromconf = true ``` :::note This config file does not contain the default entries for an FE, only the shared-data configuration is shown. ::: The non-default FE configuration settings: :::note Many configuration parameters are prefixed with `s3`. This prefix is used for all Amazon S3 compatible storage types (for example: S3, GCS, and MinIO). When using Azure Blob Storage the prefix is `azure`. ::: This enables shared-data use. This specifies whether S3 compatible storage or Azure Blob Storage is used. For MinIO this is always S3. The MinIO endpoint, including port number. The bucket name. The MinIO access key. The MinIO access key secret. When using MinIO an access key is used, and so instance profiles are not used with MinIO. When using MinIO this parameter is always set to false. When this is true, a StarRocks storage volume named `builtinstoragevolume` is created using MinIO object storage, and it is set to be the default storage volume for the tables that you create. In this tutorial you: Deployed StarRocks and Minio in Docker Created a MinIO access key Configured a StarRocks Storage Volume that uses MinIO Loaded crash data provided by New York City and weather data provided by NOAA Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transform done during the Stream Load. The details on that are in the notes on the curl commands below. <Curl /> The dataset is provided by New