Project: Druid (App Definition and Development / Database)
File: README.md

<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, ~ software distributed under the License is distributed on an ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY ~ KIND, either express or implied. See the License for the ~ specific language governing permissions and limitations ~ under the License. --> The simulation framework allows developers to recreate arbitrary cluster setups and verify coordinator behaviour. Tests written using the framework can also help identify performance bottlenecks or potential bugs in the system and even compare different balancing strategies. As opposed to unit tests, simulations are meant to test the coordinator as a whole and verify the interactions of all the underlying parts. In that regard, these simulations resemble integration tests more closely. The primary test target is the `DruidCoordinator` itself. The behaviour of the following entities can also be verified using simulations: `HttpLoadQueuePeon`, `LoadQueueTaskMaster` All coordinator duties, e.g. `BalanceSegments`, `RunRules` All retention rules The framework provides control over the following aspects of the setup: | Input | Details | Actions | |-||| |cluster | server name, type, tier, size | add a server, remove a server| |segment |datasource, interval, version, partition num, size | add/remove from server, mark used/unused, publish new segments| |rules | type (foreverLoad, drop, etc), replica count per tier | set rules for a datasource| |configs |coordinator period, load queue type, load queue size, max segments to balance | set or update a config | The above actions can be performed at any point after building the simulation. So, you could even recreate scenarios where during a coordinator run, a server crashes or the retention rules of a datasource change, and verify the behaviour of the coordinator in these situations. Execution: A tight dependency on time durations such as the period of a repeating task or the delay before a scheduled task makes it difficult to reliably reproduce a test scenario. As a result, the tests become flaky. Thus, all the executors required for coordinator operations have been allowed only two possible modes of execution: immediate: Execute tasks on the calling thread itself. blocked: Keep tasks in a queue until explicitly invoked. Internal dependencies: In order to allow realistic reproductions of the coordinator behaviour, none of the internal parts of the coordinator have been mocked in the framework and new tests need not mock anything at

External dependencies: Since these tests are meant to verify the behaviour of only the coordinator, the interfaces to communicate with external dependencies have been provided as simple in-memory implementations: communication with metadata store: `SegmentMetadataManager`, `MetadataRuleManager` communication with historicals: `HttpClient`, `ServerInventoryView` `CuratorFramework`: provided as a mock as simulations of `CuratorLoadQueuePeon` are not supported yet Inventory: The coordinator maintains an inventory view of the cluster state. Simulations can choose from two modes of inventory update - auto and manual. In auto update mode, any change made to the cluster is immediately reflected in the inventory view. In manual update mode, the inventory must be explicitly synchronized with the cluster state. The framework does not expose the coordinator HTTP endpoints. It should not be used to verify the absolute values of execution latencies, e.g. the time taken to compute the balancing cost of a segment. But the relative values can still be a good indicator while doing comparisons between, say two balancing strategies. It does not support simulation of the zk-based `CuratorLoadQueuePeon`. Writing a test class: Extend `CoordinatorSimulationBaseTest`. This base test exposes methods to get or set the state of the cluster and coordinator during a simulation. Build a simulation using `CoordinatorSimulation.builder()` with specified segments, servers, rules and configs. Start the simulation with `startSimulation(simulation)`. Invoke coordinator runs with `runCoordinatorCycle()` Verify emitted metrics and current cluster state Example: ```java public class SimpleSimulationTest extends CoordinatorSimulationBaseTest { @Test public void testShiftSegmentsToDifferentTier() { // Create segments List<DataSegment> segments = CreateDataSegments.ofDatasource("wiki") .forIntervals(30, Granularities.DAY) .startingAt("2022-01-01") .withNumPartitions(10) .eachOfSizeInMb(500); // Create servers DruidServer historicalTier1 = createHistoricalTier(1, "tier_1", 10000); DruidServer historicalTier2 = createHistoricalTier(1, "tier_2", 20000); // Build simulation CoordinatorSimulation sim = CoordinatorSimulation.builder() .withServers(historicalTier1, historicalTier2) .withSegments(segments) .withRules("wiki".Load.on("tier_2", 1).forever()) .build(); // Start the simulation with all segments loaded on tier_1 segments.forEach(historicalTier1::addSegment); startSimulation(sim); // Run a few coordinator cycles int totalLoadedOnT2 = 0; int totalDroppedFromT1 = 0; for (int i = 0; i < 10; ++i) { runCoordinatorCycle(); loadQueuedSegments(); totalLoadedOnT2 += getValue("segment/assigned/count", filter("tier", "tier_2")); totalDroppedFromT1 += getValue("segment/dropped/count", filter("tier", "tier_1")); } // Verify that some segments have been loaded/dropped Assert.assertTrue(totalLoadedOnT2 > 0 && totalLoadedOnT2 <= segments.size()); Assert.assertTrue(totalDroppedFromT1 > 0 && totalDroppedFromT1 <= segments.size()); Assert.assertTrue(totalDroppedFromT1 <= totalLoadedOnT2); } } ``` See `org.apache.druid.server.coordinator.simulate.SegmentLoadingTest` configurations to be the following Vm options: ``` -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.test.config.dockerIp=localhost -Ddruid.zk.service.host=localhost -Ddruid.client.https.trustStorePath=client_tls/truststore.jks -Ddruid.client.https.trustStorePassword=druid123 -Ddruid.client.https.keyStorePath=client_tls/client.jks -Ddruid.client.https.certAlias=druid -Ddruid.client.https.keyManagerPassword=druid123 -Ddruid.client.https.keyStorePassword=druid123 ``` Run tests from the test configuration often found in the top right corner of the IntelliJ IDE. The values shown above are for the default docker compose cluster. For other clusters the values will need to be changed. docker-compose.base.yml Base file that defines all containers for integration testing docker-compose.yml Defines a Druid cluster with default configuration that is used for running integration tests. ```bash docker compose -f docker-compose.yml up DRUIDINTEGRATIONTEST_GROUP=batch-index docker compose -f docker-compose.yml up ``` You can change the default configuration using a custom configuration file. The settings in the file will override the default settings if they conflict. They will be appended to the default configuration otherwise. ```bash OVERRIDEENV=./environment-configs/test-groups/prepopulated-data DRUIDINTEGRATIONTESTGROUP=query docker compose -f docker-compose.yml up ``` docker-compose.security.yml Defines three additional Druid router services with permissive tls, no client auth tls, and custom check tls respectively. This is meant to be used together with docker-compose.yml and is only needed for the "security" group integration test. ```bash docker compose -f docker-compose.yml -f docker-compose.security.yml up ``` docker-compose.druid-hadoop.yml For starting Apache Hadoop

cluster with the same setup as the Druid tutorial. ```bash docker compose -f docker-compose.druid-hadoop.yml up ``` | Flag | Description | |:|| | -Ddocker.build.skip=true | Skip building the containers. <br/><br/>If you do not apply any change to Druid then you skip rebuilding the containers. This can save ~4 minutes. You need to build druid containers only once, after you can skip docker build step. | | -Ddocker.run.skip=true | Skip starting docker containers.<br/><br/> This can save ~3 minutes by skipping building and bringing up the docker containers (Druid, Kafka, Hadoop, MYSQL, zookeeper, etc).<br/> Please make sure that you actually do have these containers already running if using this flag.<br/><br/> Additionally, please make sure that the running containers are in the same state that the setup script (run_cluster.sh) would have brought it up in. | | -Ddocker.build.hadoop=true | Build the hadoop image when either running integration tests or when building the integration test docker images without running the tests. | | -Dstart.hadoop.docker=true | Start hadoop container when you need to run IT tests that utilize local hadoop docker. | For your convenience, Druid processes running inside Docker have been debugging enabled at following debugging ports: | Process | Remote Debugging Port | | | :: | | Router with permissive tls | 5001 | | Router with no client auth tls | 5002 | | Router with custom check tls | 5003 | | Router| 5004 | | Broker| 5005 | | Coordinator | 5006 | | Historical | 5007 | | Middlemanager | 5008 | | Overlord | 5009 | | Peons (Workers on Middlemanager) | Ephemeral port assigned by debugger (check task log for port assigned to each task) | You can use remote debugger(such as via IntelliJ IDEA's Remote Configuration) to debug the corresponding Druid process at above port. Running Tests Using A Quickstart Cluster When writing integration tests, it can be helpful to test against a quickstart cluster so that you can set up remote debugging with in your developer environment. This section walks you through setting up the integration tests so that it can run against a running on your development machine. NOTE: Not all features run by default on a quickstart cluster, so it may not make sense to run the entire test suite against this configuration. Quickstart does not run with ssl, so to trick the integration tests we specify the `*tlsurl` in the config to be the same as the http url. Make sure you have at least 6GiB of memory available before you run the tests. The tests rely on files in the test/resources folder to exist under the path /resources, so create a symlink to make them available: ```bash ln -s ${DRUID_HOME}/integration-tests/src/test/resources /resources ``` Set the cluster config file environment variable to the quickstart config: ```bash export CONFIGFILE=${DRUIDHOME}/integration-tests/quickstart-it.json ``` The test group `quickstart-compatible` has tests that have been verified to work against the quickstart cluster. There may be more tests that work, if you find that they do, please mark it as quickstart-compatible (TestNGGroup#QUICKSTART_COMPATIBLE) and open a

If you find some integration tests do not work, look at the docker files to see what setup they do. You may need to do similar steps to get the test to work. Then run the tests using a command similar to: ```bash mvn verify -P int-tests-config-file -Dit.test=<test_name> mvn verify -P int-tests-config-file -Dgroups=quickstart-compatible ``` Running Tests Using A Configuration File for Any Cluster Make sure that you have at least 6GiB of memory available before you run the tests. To run tests on any druid cluster that is already running, create a configuration file: { "brokerhost": "<brokerip>", "brokerport": "<brokerport>", "routerhost": "<routerip>", "routerport": "<routerport>", "indexerhost": "<indexerip>", "indexerport": "<indexerport>", "coordinatorhost": "<coordinatorip>", "coordinatorport": "<coordinatorport>", "middlemanagerhost": "<middlemanager_ip>", "zookeeperhosts": "<comma-separated list of zookeeperip:zookeeper_port>", "cloudbucket": "<(optional) cloudbucket for test data if running cloud integration test>", "cloudpath": "<(optional) cloudpath for test data if running cloud integration test>" } Set the environment variable `CONFIG_FILE` to the name of the configuration file: ```bash export CONFIG_FILE=<config file name> ``` To run all tests from a test group using mvn run the following command: (list of test groups can be found at integration-tests/src/test/java/org/apache/druid/tests/TestNGGroup.java) ```bash mvn verify -P int-tests-config-file -Dgroups=<test_group> ``` To run only a single test using mvn run the following command: ```bash mvn verify -P int-tests-config-file -Dit.test=<test_name> ``` Running a Test That Uses Cloud The integration test that indexes from Cloud or uses Cloud as deep storage is not run as part of the integration test run discussed above. Running these tests requires the user to provide their own Cloud. Currently, the integration test supports Amazon Kinesis, Google Cloud Storage, Amazon S3, and Microsoft Azure. These can be run by providing "kinesis-index", "kinesis-data-format", "gcs-deep-storage", "s3-deep-storage", or "azure-deep-storage" to -Dgroups for Amazon Kinesis, Google Cloud Storage, Amazon S3, and Microsoft Azure respectively. Note that only one group should be run per mvn command. For all the Cloud Integration tests, the following will also need to be provided: 1) Provide -Doverride.config.path=<PATHTOFILE> with your Cloud credentials/configs set. See integration-tests/docker/environment-configs/override-examples/ directory for env vars to provide for each Cloud. For Amazon Kinesis, the following will also need to be provided: 1) Provide -Ddruid.test.config.streamEndpoint=<STREAM_ENDPOINT> with the endpoint of your stream set. For example, kinesis.us-east-1.amazonaws.com For Google Cloud Storage, Amazon S3, and Microsoft Azure, the following will also need to be provided: 1) Set the bucket and path for your test data. This can be done by setting -Ddruid.test.config.cloudBucket and -Ddruid.test.config.cloudPath in the mvn command or setting "cloudbucket" and "cloudpath" in the config file. 2) Copy wikipediaindexdata1.json, wikipediaindexdata2.json, and wikipediaindexdata3.json located in integration-tests/src/test/resources/data/batch_index/json to your Cloud storage at the location set in step 1. For Google Cloud Storage, in addition to the above, you will also have to: 1) Provide -Dresource.file.dir.path=<PATHTOFOLDER> with folder that contains GOOGLEAPPLICATIONCREDENTIALS file For example, to run integration test for Google Cloud Storage: ```bash mvn verify -P integration-tests -Dgroups=gcs-deep-storage -Doverride.config.path=<PATHTOFILE> \ -Dresource.file.dir.path=<PATHTOFOLDER> -Ddruid.test.config.cloudBucket=test-bucket \ -Ddruid.test.config.cloudPath=test-data-folder/ ``` Running a Test That Uses Hadoop The integration test that indexes from hadoop is not run as part of the integration test run discussed above. This is because druid test clusters might not, in general, have access to

This also applies to integration test that uses Hadoop HDFS as an inputSource or as a deep storage. To run integration test that uses Hadoop, you will have to run a Hadoop cluster. This can be done in two ways: 1) Run Druid Docker test clusters with Hadoop container by passing -Dstart.hadoop.docker=true to the mvn command. If you have not already built the hadoop image, you will also need to add -Ddocker.build.hadoop=true to the mvn command. 2) Run your own Druid + Hadoop cluster and specified Hadoop configs in the configuration file (CONFIG_FILE). Currently, hdfs-deep-storage and other <cloud>-deep-storage integration test groups can only be run with Druid Docker test clusters by passing -Dstart.hadoop.docker=true to start Hadoop container. You will also have to provide -Doverride.config.path=<PATHTOFILE> with your Druid's Hadoop configs set. See integration-tests/docker/environment-configs/override-examples/hdfs directory for example. Note that if the integration test you are running also uses other cloud extension (S3, Azure, GCS), additional credentials/configs may need to be set in the same file as your Druid's Hadoop configs set. If you are running ITHadoopIndexTest with your own Druid + Hadoop cluster, please follow the below steps: Copy wikipediaindexdata1.json, wikipediaindexdata2.json, and wikipediaindexdata3.json located in integration-tests/src/test/resources/data/batchindex/json to your HDFS at /batchindex/json/ Copy batchhadoop.data located in integration-tests/src/test/resources/data/batchindex/hadoop_tsv to your HDFS at /batchindex/hadooptsv/ If using the Docker-based Hadoop container, the steps above are automatically done by the integration tests. When running the Hadoop tests, you must set `-Dextra.datasource.name.suffix=''`, due to https://github.com/apache/druid/issues/9788. Option 1: Run the test using mvn (using the bundled Docker-based Hadoop cluster and building docker images at runtime): ```bash mvn verify -P integration-tests -Dit.test=ITHadoopIndexTest -Dstart.hadoop.docker=true -Ddocker.build.hadoop=true \ -Doverride.config.path=docker/environment-configs/override-examples/hdfs -Dextra.datasource.name.suffix='' ``` Option 2: Run the test using mvn (using the bundled Docker-based hadoop cluster and not building images at runtime): ```bash mvn verify -P integration-tests -Dit.test=ITHadoopIndexTest -Dstart.hadoop.docker=true -Ddocker.build.skip=true \ -Doverride.config.path=docker/environment-configs/override-examples/hdfs -Dextra.datasource.name.suffix='' ``` Option 3: Run the test using mvn (using the bundled Docker-based hadoop cluster and when you have already started all containers) ```bash mvn verify -P integration-tests -Dit.test=ITHadoopIndexTest -Ddocker.run.skip=true -Ddocker.build.skip=true \ -Doverride.config.path=docker/environment-configs/override-examples/hdfs -Dextra.datasource.name.suffix='' ``` Option 4: Run the test using mvn (using config file for existing Hadoop cluster): ```bash mvn verify -P int-tests-config-file -Dit.test=ITHadoopIndexTest -Dextra.datasource.name.suffix='' ``` In some test environments, the machine where the tests need to be executed cannot access the outside internet, so mvn cannot be run. In that case, do the following instead of running the tests using mvn: On a machine that can do mvn builds: ```bash cd druid mvn clean package cd integration_tests mvn dependency:copy-dependencies package ``` Copy the integration-tests directory to the test cluster. ```bash TDIR=<directory containing integration-tests>/target VER=<version of druid you built> export CLASSPATH=$TDIR/dependency/*:$TDIR/druid-integration-tests-$VER.jar:$TDIR/druid-integration-tests-$VER-tests.jar ``` ```bash java -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.test.config.type=configFile \ -Ddruid.test.config.configFile=<pathname of configuration file> org.testng.TestNG \ -testrunfactory org.testng.DruidTestRunnerFactory -testclass org.apache.druid.tests.hadoop.ITHadoopIndexTest ``` Writing a New Test For every end-user functionality provided by druid we should have an integration-test verifying the

1) Name of the test must start with a prefix "IT" 2) A test should be independent of other tests 3) Tests are to be written in TestNG style () 4) If a test loads some data it is the responsibility of the test to clean up the data from the cluster A test can access different helper and utility classes provided by test-framework in order to access Coordinator,Broker etc.. To mark a test be able to use Guice Dependency Injection - Annotate the test class with the below annotation ```java @Guice(moduleFactory = DruidTestModuleFactory.class) ``` This will tell the test framework that the test class needs to be constructed using guice. 1) IntegrationTestingConfig - configuration of the test 2) CoordinatorResourceTestClient - httpclient for coordinator endpoints 3) OverlordResourceTestClient - httpclient for indexer endpoints 4) QueryResourceTestClient - httpclient for broker endpoints 1) RetryUtil - provides methods to retry an operation until it succeeds for configurable no. of times 2) FromFileTestQueryHelper - reads queries with expected results from file and executes them and verifies the results using ResultVerifier Refer ITIndexerTest as an example on how to use dependency Injection By default, test methods in a test class will be run in sequential order one at a time. Test methods for a given test class can be set to run in parallel (multiple test methods of each class running at the same time) by excluding the given class/package from the "AllSerializedTests" test tag section and including it in the "AllParallelizedTests" test tag section in integration-tests/src/test/resources/testng.xml. TestNG uses two parameters, i.e., `thread-count` and `data-provider-thread-count`, for parallel test execution, which are both set to 2 for Druid integration tests. For test using parallel execution with data provider, you will also need to set `@DataProvider(parallel = true)` on your data provider method in your test class. Note that for test using parallel execution with data provider, the test class does not need to be in the "AllParallelizedTests" test tag section and if it is in the "AllParallelizedTests" test tag section it will actually be run with `thread-count` times `data-provider-thread-count` threads. You may want to modify those values for faster execution. See https://testng.org/doc/documentation-main.html#parallel-running and https://testng.org/doc/documentation-main.html#parameters-dataproviders for details. Please be mindful when adding tests to the "AllParallelizedTests" test tag that the tests can run in parallel with other tests from the same class at the same time. i.e. test does not modify/restart/stop the druid cluster or other dependency containers, test does not use excessive memory starving other concurent task, test does not modify and/or use other task, supervisor, datasource it did not create. By default, integration tests are run in Travis environment on commits made to open PR. These integration test jobs are required to pass for a PR to be elligible to be merged. Here are known issues and limitations to the Druid docker cluster running in Travis machine that may cause the tests to fail: Number of concurrent running tasks. Although the default Druid cluster config sets the maximum number of tasks (druid.worker.capacity) to 10, the actual maximum can be lowered depending on the type of the tasks. For example, running 2 range partitioning compaction tasks with 2 subtasks each (for a total of 6 tasks) concurrently can cause the cluster to intermittently fail. This can cause the Travis job to become stuck until it timeouts (50 minutes) and/or terminates after 10 mins of not receiving new output.