Project: StarRocks (App Definition and Development / Database)
File: shared-nothing.md

displayed_sidebar: "English" sidebar_position: 1 description: "StarRocks in Docker: Query real data with JOINs" import DDL from '../assets/quick-start/_DDL.mdx' import Clients from '../assets/quick-start/_clientsAllin1.mdx' import SQL from '../assets/quick-start/_SQL.mdx' import Curl from '../assets/quick-start/_curl.mdx' This tutorial covers: Running StarRocks in a single Docker container Loading two public datasets including basic transformation of the data Analyzing the data with SELECT and JOIN Basic data transformation (the T in ETL) The data used is provided by NYC OpenData and the National Centers for Environmental Information. Both of these datasets are very large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run the Docker image and load this data on a machine with 4GB RAM assigned to Docker. For larger fault-tolerant and scalable deployments we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with the step by step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: Allow the reader to load data in StarRocks and analyze that data. Explain the basics of data transformation during loading. 4GB RAM assigned to Docker 10GB free disk space assigned to Docker You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work, and this guide covers the configuration of DBeaver and MySQL WorkBench. `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, . Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. Backend nodes are responsible for both data storage and executing query plans. ```bash docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd \ --name quickstart starrocks/allin1-ubuntu ``` <Clients /> Download these two datasets to your machine. You can download them to the host machine where you are running Docker, they do not need to be downloaded inside the container. ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPDCrashData.csv ``` ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` :::tip If you are using a client other than the mysql CLI, open that now. ::: This command will run the `mysql` command in the Docker container: ```sql docker exec -it quickstart \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` <DDL /> There are many ways to load data into StarRocks. For this tutorial the simplest way is to use curl and StarRocks Stream Load. :::tip Open a new shell as these curl commands are run at the operating system prompt, not in the `mysql` client. The commands refer to the datasets that you downloaded, so run them from the directory where you downloaded the files. You will be prompted for a

You probably have not assigned a password to the MySQL `root` user, so just hit enter. ::: The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ```bash curl --location-trusted -u root \ -T ./NYPDCrashData.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "maxfilterratio:1" \ -H "columns:tmpCRASHDATE, tmpCRASHTIME, CRASHDATE=strtodate(concatws(' ', tmpCRASHDATE, tmpCRASHTIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIPCODE,LATITUDE,LONGITUDE,LOCATION,ONSTREETNAME,CROSSSTREETNAME,OFFSTREETNAME,NUMBEROFPERSONSINJURED,NUMBEROFPERSONSKILLED,NUMBEROFPEDESTRIANSINJURED,NUMBEROFPEDESTRIANSKILLED,NUMBEROFCYCLISTINJURED,NUMBEROFCYCLISTKILLED,NUMBEROFMOTORISTINJURED,NUMBEROFMOTORISTKILLED,CONTRIBUTINGFACTORVEHICLE1,CONTRIBUTINGFACTORVEHICLE2,CONTRIBUTINGFACTORVEHICLE3,CONTRIBUTINGFACTORVEHICLE4,CONTRIBUTINGFACTORVEHICLE5,COLLISIONID,VEHICLETYPECODE1,VEHICLETYPECODE2,VEHICLETYPECODE3,VEHICLETYPECODE4,VEHICLETYPECODE5" \ -XPUT http://localhost:8030/api/quickstart/crashdata/streamload ``` Here is the output of the preceding command. The first highlighted section shows what you should expect to see (OK and all but one row inserted). One row was filtered out because it does not contain the correct number of columns. ```bash Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 1013, "BeginTxnTimeMs": 21, "StreamLoadPlanTimeMs": 63, "ReadDataTimeMs": 563, "WriteDataTimeMs": 870, "CommitAndPublishTimeMs": 57, "ErrorURL": "http://127.0.0.1:8040/api/loaderrorlog?file=errorlogda41dd88276a7bfc739087c94262ae9f" }% ``` If there was an error the output provides a URL to see the error messages. Open this in a browser to find out what happened. Expand the detail to see the error message: <details> <summary>Reading error messages in the browser</summary> ```bash Error: Value count does not match column count. Expect 29, but got 32. Column delimiter: 44,Row delimiter: 10.. Row: 09/06/2015,14:15,,,40.6722269,-74.0110059,"(40.6722269, -74.0110059)",,,"R/O 1 BEARD ST. ( IKEA'S 09/14/2015,5:30,BRONX,10473,40.814551,-73.8490955,"(40.814551, -73.8490955)",TORRY AVENUE ,NORTON AVENUE ,,0,0,0,0,0,0,0,0,Driver Inattention/Distraction,Unspecified,,,,3297457,PASSENGER VEHICLE,PASSENGER VEHICLE,,, ``` </details> Load the weather dataset in the same manner as you loaded the crash data. ```bash curl --location-trusted -u root \ -T ./72505394728.csv \ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "maxfilterratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://localhost:8030/api/quickstart/weatherdata/streamload ``` <SQL /> In this tutorial you: Deployed StarRocks in Docker Loaded crash data provided by New York City and weather data provided by NOAA Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transformation done during the Stream Load. The details on that are in the notes on the curl commands below. <Curl /> The dataset is provided by New York City subject to these and . The (LCD) is provided by NOAA with this and this .