Issue 1:
Apache Druid 25.0.0 contains over 300 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 51 contributors.

See the complete set of changes for additional details.

# Highlights

# MSQ task engine now production ready

The multi-stage query (MSQ) task engine used for SQL-based ingestion is now production ready. Use it for any supported workloads. For more information, see the following pages:

Ingestion

SQL-based ingestion

# Simplified Druid deployments

The new start-druid script greatly simplifies deploying any combination of Druid services on a single-server. It comes pre-packaged with the required configs and can be used to launch a fully functional Druid cluster simply by invoking ./start-druid. For experienced Druids, it also gives complete control over the runtime properties and JVM arguments to have a cluster that exactly fits your needs.

The start-druid script deprecates the existing profiles such as start-micro-quickstart and start-nano-quickstart. These profiles may be removed in future releases. For more information, see Single server deployment.

# String dictionary compression (experimental)

Added support for front coded string dictionaries for smaller string columns, leading to reduced segment sizes with only minor performance penalties for most Druid queries.

This can be enabled by setting IndexSpec.stringDictionaryEncoding to {"type":"frontCoded", "bucketSize": 4} , where bucketSize is any power of 2 less than or equal to 128. Setting this property instructs indexing tasks to write segments using compressed dictionaries of the specified bucket size.

Any segment written using string dictionary compression is not readable by older versions of Druid.

For more information, see Front coding.

#12277

# Kubernetes-native tasks

Druid can now use Kubernetes to launch and manage tasks, eliminating the need for middle managers.

To use this feature, enable the druid-kubernetes-overlord-extensions in the extensions load list for your Overlord process.

#13156

# Hadoop-3 compatible binary

Druid now comes packaged as a dedicated binary for Hadoop-3 users, which contains Hadoop-3 compatible jars. If you do not use Hadoop-3 with your Druid cluster, you may continue using the classic binary.

# Multi-stage query (MSQ) task engine

# MSQ enabled for Docker

MSQ task query engine is now enabled for Docker by default.

#13069

# Query history

Multi-stage queries no longer show up in the Query history dialog. They are still available in the Recent query tasks panel.

# Limit on CLUSTERED BY columns

When using the MSQ task engine to ingest data, the number of columns that can be passed in the CLUSTERED BY clause is now limited to 1500.

#13352

# Support for string dictionary compression

The MSQ task engine supports the front-coding of String dictionaries for better compression. This can be enabled for INSERT or REPLACE statements by setting indexSpec to a valid json string in the query context.

#13275

# Sketch merging mode

Workers can now gather key statistics, used to generate partition boundaries, either sequentially or in parallel. Set clusterStatisticsMergeMode to PARALLEL, SEQUENTIAL or AUTO in the query context to use the corresponding sketch merging mode. For more information, see Sketch merging mode.

#13205

# Performance and operational improvements

Error messages: For disallowed MSQ warnings of certain types, the warning is now surfaced as the error. #13198

Secrets: For tasks containing SQL with sensitive keys, Druid now masks the keys while logging with the help regular expressions. #13231

Downsampling accuracy: MSQ task engine now uses the number of bytes instead of number of keys when downsampling data. #12998

Memory usage: When determining partition boundaries, the heap footprint of internal sketches used by MSQ is now capped at 10% of available memory or 300 MB, whichever is lower. Previously, the cap was strictly 300 MB. #13274

Task reports: Added fields pendingTasks and runningTasks to the worker report. See Query task status information for related web console changes. #13263

# Querying

# Async reads for JDBC

Prevented JDBC timeouts on long queries by returning empty batches when a batch fetch takes too long. Uses an async model to run the result fetch concurrently with JDBC requests.

#13196

# Improved algorithm to check values of an IN filter

To accommodate large value sets arising from large IN filters or from joins pushed down as IN filters, Druid now uses a sorted merge algorithm for merging the set and dictionary for larger values.

#13133

# Enhanced query context security

Added the following configuration properties that refine the query context security model controlled by druid.auth.authorizeQueryContextParams:

druid.auth.unsecuredContextKeys: A JSON list of query context keys that do not require a security check.

druid.auth.securedContextKeys: A JSON list of query context keys that do require a security check.

If both are set, unsecuredContextKeys acts as exceptions to securedContextKeys.

#13071

# HTTP response headers

The HTTP response for a SQL query now correctly sets response headers, same as a native query.

#13052

# Metrics

# New metrics

The following metrics have been newly added. For more details, see the complete list of Druid metrics.

# Batched segment allocation

These metrics pertain to batched segment allocation.

Metric
Description
Dimensions

task/action/batch/runTime
Milliseconds taken to execute a batch of task actions. Currently only being emitted for batched segmentAllocate actions
dataSource, taskActionType=segmentAllocate

task/action/batch/queueTime
Milliseconds spent by a batch of task actions in queue. Currently only being emitted for batched segmentAllocate actions
dataSource, taskActionType=segmentAllocate

task/action/batch/size
Number of task actions in a batch that was executed during the emission period. Currently only being emitted for batched segmentAllocate actions
dataSource, taskActionType=segmentAllocate

task/action/batch/attempts
Number of execution attempts for a single batch of task actions. Currently only being emitted for batched segmentAllocate actions
dataSource, taskActionType=segmentAllocate

task/action/success/count
Number of task actions that were executed successfully during the emission period. Currently only being emitted for batched segmentAllocate actions
dataSource, taskId, taskType, taskActionType=segmentAllocate

task/action/failed/count
Number of task actions that failed during the emission period. Currently only being emitted for batched segmentAllocate actions
dataSource, taskId, taskType, taskActionType=segmentAllocate

# Streaming ingestion

Metric
Description
Dimensions

ingest/kafka/partitionLag
Partition-wise lag between the offsets consumed by the Kafka indexing tasks and latest offsets in Kafka brokers. Minimum emission period for this metric is a minute.
dataSource, stream, partition

ingest/kinesis/partitionLag/time
Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. Minimum emission period for this metric is a minute.
dataSource, stream, partition

ingest/pause/time
Milliseconds spent by a task in a paused state without ingesting.
dataSource, taskId, taskType

ingest/handoff/time
Total time taken in milliseconds for handing off a given set of published segments.
dataSource, taskId, taskType

#13238

#13331

#13313

# Other improvements

New dimension taskActionType which may take values such as segmentAllocate, segmentTransactionalInsert, etc. This dimension is reported for task/action/run/time and the new batched segment allocation metrics. #13333

Metric namespace/cache/heapSizeInBytes for global cached lookups now accounts for the String object overhead of 40 bytes. #13219

jvm/gc/cpu has been fixed to report nanoseconds instead of milliseconds. #13383

# Nested columns

# Nested columns performance improvement

Improved NestedDataColumnSerializer to no longer explicitly write null values to the field writers for the missing values of every row. Instead, passing the row counter is moved to the field writers so that they can backfill null values in bulk.

#13101

# Support for more formats

Druid nested columns and the associated JSON transform functions now support Avro, ORC, and Parquet.

#13325

#13375

# Refactored a datasource before unnest

When data requires "flattening" during processing, the operator now takes in an array and then flattens the array into N (N=number of elements in the array) rows where each row has one of the values from the array.

#13085

# Ingestion

# Improved filtering for cloud objects

You can now stop at arbitrary subfolders using glob syntax in the ioConfig.inputSource.filter field for native batch ingestion from cloud storage, such as S3.

#13027

# Async task client for streaming ingestion

You can now enable asynchronous communication between the stream supervisor and indexing tasks by setting chatAsync to true in the tuningConfig. The async task client uses its own internal thread pool and thus ignrores the chatThreads property.

#13354

# Improved handling of JSON data with streaming ingestion

You can now better control how Druid reads JSON data for streaming ingestion by setting the following fields in the input format specification:

assumedNewlineDelimited to parse lines of JSON independently.

useJsonNodeReader to retain valid JSON events when parsing multi-line JSON events when a parsing exception occurs.

The web console has been updated to include these options.

#13089

# Ingesting from an idle Kafka stream

When a Kafka stream becomes inactive, the supervisor ingesting from it can be configured to stop creating new indexing tasks. The supervisor automatically resumes creation of new indexing tasks once the stream becomes active again. Set the property dataSchema.ioConfig.idleConfig.enabled to true in the respective supervisor spec or set druid.supervisor.idleConfig.enabled on the overlord to enable this behaviour. Please see the following for details:

Overlord configuration

Supervisor spec

#13144

# Kafka Consumer improvement

You can now configure the Kafka Consumer's custom deserializer after its instantiation.

#13097

# Kafka supervisor logging

Kafka supervisor logs are now less noisy. The supervisors now log events at the DEBUG level instead of INFO.

#13392

# Fixed Overlord leader election

Fixed a problem where Overlord leader election failed due to lock reacquisition issues. Druid now fails these tasks and clears all locks so that the Overlord leader election isn't blocked.

#13172

# Support for inline protobuf descriptor

Added a new inline type protoBytesDecoder that allows a user to pass inline the contents of a Protobuf descriptor file, encoded as a Base64 string.

#13192

# Duplicate notices

For streaming ingestion, notices that are the same as one already in queue won't be enqueued. This will help reduce notice queue size.

#13334

# Sampling from stream input now respects the configured timeout

Fixed a problem where sampling from a stream input, such as Kafka or Kinesis, failed to respect the configured timeout when the stream had no records available. You can now set the maximum amount of time in which the entry iterator will return results.

#13296

# Streaming tasks resume on Overlord switch

Fixed a problem where streaming ingestion tasks continued to run until their duration elapsed after the Overlord leader had issued a pause to the tasks. Now, when the Overlord switch occurs right after it has issued a pause to the task, the task remains in a paused state even after the Overlord re-election.

#13223

# Fixed Parquet list conversion

Fixed an issue with Parquet list conversion, where lists of complex objects could unexpectedly be wrapped in an extra object, appearing as [{"element":<actual_list_element>},{"element":<another_one>}...] instead of the direct list. This changes the behavior of the parquet reader for lists of structured objects to be consistent with other parquet logical list conversions. The data is now fetched directly, more closely matching its expected structure.

#13294

# Introduced a tree type to flattenSpec

Introduced a tree type to flattenSpec. In the event that a simple hierarchical lookup is required, the tree type allows for faster JSON parsing than jq and path parsing types.

#12177

# Operations

# Compaction

Compaction behavior has changed to improve the amount of time it takes and disk space it takes:

When segments need to be fetched, download them one at a time and delete them when Druid is done with them. This still takes time but minimizes the required disk space.

Don't fetch segments on the main compact task when they aren't needed. If the user provides a full granularitySpec, dimensionsSpec, and metricsSpec, Druid skips fetching segments.

For more information, see the documentation on Compaction and Automatic compaction.

#13280

# Idle configs for the Supervisor

You can now set the Supervisor to idle, which is useful in cases where freeing up slots so that autoscaling can be more effective.

To configure the idle behavior, use the following properties:

Property
Description
Default

druid.supervisor.idleConfig.enabled
(Cluster wide) If true, supervisor can become idle if there is no data on input stream/topic for some time.
false

druid.supervisor.idleConfig.inactiveAfterMillis
(Cluster wide) Supervisor is marked as idle if all existing data has been read from input topic and no new data has been published for inactiveAfterMillis milliseconds.
600_000

inactiveAfterMillis
(Individual Supervisor) Supervisor is marked as idle if all existing data has been read from input topic and no new data has been published for inactiveAfterMillis milliseconds.
no (default == 600_000)

#13311

# Improved supervisor termination

Fixed issues with delayed supervisor termination during certain transient states.

#13072

# Backoff for HttpPostEmitter

The HttpPostEmitter option now has a backoff. This means that there should be less noise in the logs and lower CPU usage if you use this option for logging.

#12102

# DumpSegment tool for nested columns

The DumpSegment tool can now be used on nested columns with the --dump nested option.

For more information, see dump-segment tool.

#13356

# Segment loading and balancing

# Batched segment allocation

Segment allocation on the Overlord can take some time to finish, which can cause ingestion lag while a task waits for segments to be allocated. Performing segment allocation in batches can help improve performance.

There are two new properties that affect how Druid performs segment allocation:

Property
Description
Default

druid.indexer.tasklock.batchSegmentAllocation
If set to true, Druid performs segment allocate actions in batches to improve throughput and reduce the average task/action/run/time. See batching segmentAllocate actions for details.
false

druid.indexer.tasklock.batchAllocationWaitTime
Number of milliseconds after Druid adds the first segment allocate action to a batch, until it executes the batch. Allows the batch to add more requests and improve the average segment allocation run time. This configuration takes effect only if batchSegmentAllocation is enabled.
500

In addition to these properties, there are new metrics to track batch segment allocation. For more information, see New metrics for segment allocation.

For more information, see the following:

Overlord operations

Task actions and Batching segmentAllocate actions

#13369

#13503

# Improved cachingCost balancer strategy

The cachingCost balancer strategy now behaves more similarly to cost strategy. When computing the cost of moving a segment to a server, the following calculations are performed:

Subtract the self cost of a segment if it is being served by the target server

Subtract the cost of segments that are marked to be dropped

#13321

# Faster segment assignment

You can now use a round-robin segment strategy to speed up initial segment assignments. Set useRoundRobinSegmentAssigment to true in the Coordinator dynamic config to enable this feature.

#13367

# Default to batch sampling for balancing segments

Batch sampling is now the default method for sampling segments during balancing as it performs significantly better than the alternative when there is a large number of used segments in the cluster.

As part of this change, the following have been deprecated and will be removed in future releases:

coordinator dynamic config useBatchedSegmentSampler

coordinator dynamic config percentOfSegmentsToConsiderPerMove

old non-batch method of sampling segments

# Remove unused property

The unused coordinator property druid.coordinator.loadqueuepeon.repeatDelay has been removed. Use only druid.coordinator.loadqueuepeon.http.repeatDelay to configure repeat delay for the HTTP-based segment loading queue.

#13391

# Avoid segment over-replication

Improved the process of checking server inventory to prevent over-replication of segments during segment balancing.

#13114

# Provided service specific log4j overrides in containerized deployments

Provided an option to override log4j configs setup at the service level directories so that it works with Druid-operator based deployments.

#13020

# Various Docker improvements

Updated Docker to run with JRE 11 by default.

Updated Docker to use gcr.io/distroless/java11-debian11 image as base by default.

Enabled Docker buildkit cache to speed up building.

Downloaded bash-static to the Docker image so that scripts that require bash can be executed.

Bumped builder image from 3.8.4-jdk-11-slim to 3.8.6-jdk-11-slim.

Switched busybox from amd64/busybox:1.30.0-glibc to busybox:1.35.0-glibc.

Added support to build arm64-based image.

#13059

# Enabled cleaner JSON for various input sources and formats

Added JsonInclude to various properties, to avoid population of default values in serialized JSON.

#13064

# Improved direct memory check on startup

Improved direct memory check on startup by providing better support for Java 9+ in RuntimeInfo, and clearer log messages where validation fails.

#13207

# Improved the run time of the MarkAsUnusedOvershadowedSegments duty

Improved the run time of the MarkAsUnusedOvershadowedSegments duty by iterating over all overshadowed segments and marking segments as unused in batches.

#13287

# Web console

# Delete an interval

You can now pick an interval to delete from a dropdown in the kill task dialog.

#13431

# Removed the old query view

The old query view is removed. Use the new query view with tabs.

For more information, see Web console.

#13169

# Filter column values in query results

The web console now allows you to add to existing filters for a selected column.

#13169

# Support for Kafka lookups in the web-console

Added support for Kafka-based lookups rendering and input in the web console.

#13098

# Query task status information

The web console now exposes a textual indication about running and pending tasks when a query is stuck due to lack of task slots.

#13291

# Extensions

# Extension optimization

Optimized the compareTo function in CompressedBigDecimal.

#13086

# CompressedBigDecimal cleanup and extension

Removed unnecessary generic type from CompressedBigDecimal, added support for number input types, added support for reading aggregator input types directly (uningested data), and fixed scaling bug in buffer aggregator.

#13048

# Support for Kubernetes discovery

Added POD_NAME and POD_NAMESPACE env variables to all Kubernetes Deployments and StatefulSets.

Helm deployment is now compatible with druid-kubernetes-extension.

#13262

# Docs

# Jupyter Notebook tutorials

We released our first Jupyter Notebook-based tutorial to learn the basics of the Druid API. Download the notebook and follow along with the tutorial to learn how to get basic cluster information, ingest data, and query data.

For more information, see Jupyter Notebook tutorials.

#13342

#13345

# Dependency updates

# Updated Kafka version

Updated the Apache Kafka core dependency to version 3.3.1.

#13176

# Docker improvements

Updated dependencies for the Druid image for Docker, including JRE 11. Docker BuildKit cache is enabled to speed up building.

#13059

# Upgrading to 25.0.0

Consider the following changes and updates when upgrading from Druid 24.0.x to 25.0.0. If you're updating from an earlier version, see the release notes of the relevant intermediate versions.

# Default HTTP-based segment discovery and task management

The default segment discovery method now uses HTTP instead of ZooKeeper.

This update changes the defaults for the following properties:

Property
New default
Previous default

druid.serverview.type for segment management
http
batch

druid.coordinator.loadqueuepeon.type for segment management
http
curator

druid.indexer.runner.type for the Overlord
httpRemote
local

To use ZooKeeper instead of HTTP, change the values for the properties back to the previous defaults. ZooKeeper-based implementations for these properties are deprecated and will be removed in a subsequent release.

#13092

# Finalizing HLL and quantiles sketch aggregates

The aggregation functions for HLL and quantiles sketches returned sketches or numbers when they are finalized depending on where they were in the native query plan.

Druid no longer finalizes aggregators in the following two cases:

aggregators appear in the outer level of a query

aggregators are used as input to an expression or finalizing-field-access post-aggregator

This change aligns the behavior of HLL and quantiles sketches with theta sketches.

To restore old behaviour, you can set sqlFinalizeOuterSketches=true in the query context.

#13247

# Kill tasks mark segments as unused only if specified

When you issue a kill task, Druid marks the underlying segments as unused only if explicitly specified. For more information, see the API reference

#13104

# Incompatible changes

# Upgrade curator to 5.3.0

Apache Curator upgraded to the latest version, 5.3.0. This version drops support for ZooKeeper 3.4 but Druid has already officially dropped support in 0.22. In 5.3.0, Curator has removed support for Exhibitor so all related configurations and tests have been removed.

#12939

# Fixed Parquet list conversion

The behavior of the parquet reader for lists of structured objects has been changed to be consistent with other parquet logical list conversions. The data is now fetched directly, more closely matching its expected structure. See parquet list conversion for more details.

#13294

# Credits

Thanks to everyone who contributed to this release!

@317brian

@599166320

@a2l007

@abhagraw

@abhishekagarwal87

@adarshsanjeev

@adelcast

@AlexanderSaydakov

@amaechler

@AmatyaAvadhanula

@ApoorvGuptaAi

@arvindanugula

@asdf2014

@churromorales

@clintropolis

@cloventt

@cristian-popa

@cryptoe

@dampcake

@dependabot[bot]

@didip

@ektravel

@eshengit

@findingrish

@FrankChen021

@gianm

@hnakamor

@hosswald

@imply-cheddar

@jasonk000

@jon-wei

@Junge-401

@kfaraz

@LakshSingla

@mcbrewster

@paul-rogers

@petermarshallio

@rash67

@rohangarg

@sachidananda007

@santosh-d3vpl3x

@senthilkv

@somu-imply

@techdocsmith

@tejaswini-imply

@vogievetsky

@vtlim

@wcc526

@writer-jill

@xvrl

@zachjsh

Issue 2:
Apache Druid 24.0.0 contains over 300 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 67 contributors. See the complete set of changes for additional details.

# New Features

# Multi-stage query task engine

SQL-based ingestion for Apache Druid uses a distributed multi-stage query architecture, which includes a query engine called the multi-stage query task engine (MSQ task engine). The MSQ task engine extends Druid's query capabilities, so you can write queries that reference external data as well as perform ingestion with SQL INSERT and REPLACE. Essentially, you can perform SQL-based ingestion instead of using JSON ingestion specs that Druid's native ingestion uses.

SQL-based ingestion using the multi-stage query task engine is the recommended solution starting in Druid 24.0.0. Alternative ingestion solutions such as native batch and Hadoop-based ingestion systems will still be supported. We recommend you read all known issues and test the feature in a development environment before rolling out in production. Using the multi-stage query task engine with SELECT statements that do not write to a datasource is experimental.

The extension for it (druid-multi-stage-query) is loaded by default. If you're upgrading from an earlier version of Druid, you'll need to add the extension to druid.extensions.loadlist in your common.runtime.properties file.

For more information, see the overview for the multi-stage query architecture.

#12524

#12386

#12523

#12589

# Nested columns

Druid now supports directly storing nested data structures in a newly added COMPLEX<json> column type. COMPLEX<json> columns store a copy of the structured data in JSON format as well as specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized virtual column allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns.

Newly added Druid SQL, native JSON functions, and virtual column allow you to extract, transform, and create COMPLEX<json> values in at query time. You can also use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a transformSpec in native ingestion as an alternative to using a flattenSpec object to "flatten" nested data for ingestion.

See SQL JSON functions, native JSON functions, Nested columns, virtual columns, and the feature summary for more detail.

#12753

#12714

#12753

#12920

# Updated Java support

Java 11 is fully supported is no longer experimental. Java 17 support is improved.

#12839

# Query engine updates

# Updated column indexes and query processing of filters

Reworked column indexes to be extraordinarily flexible, which will eventually allow us to model a wide range of index types. Added machinery to build the filters that use the updated indexes, while also allowing for other column implementations to implement the built-in index types to provide adapters to make use indexing in the current set filters that Druid provides.

#12388

# Time filter operator

You can now use the Druid SQL operator TIME_IN_INTERVAL to filter query results based on time. Prefer TIME_IN_INTERVAL over the SQL BETWEEN operator to filter on time. For more information, see Date and time functions.

#12662

# Null values and the "in" filter

If a values array contains null, the "in" filter matches null values. This differs from the SQL IN filter, which does not match null values.

For more information, see Query filters and SQL data types.

#12863

# Virtual columns in search queries

Previously, a search query could only search on dimensions that existed in the data source. Search queries now support virtual columns as a parameter in the query.

#12720

# Optimize simple MIN / MAX SQL queries on __time

Simple queries like select max(__time) from ds now run as a timeBoundary queries to take advantage of the time dimension sorting in a segment. You can set a feature flag to enable this feature.

#12472

#12491

# String aggregation results

The first/last string aggregator now only compares based on values. Previously, the first/last string aggregator’s values were compared based on the _time column first and then on values.

If you have existing queries and want to continue using both the _time column and values, update your queries to use ORDER BY MAX(timeCol).

#12773

# Reduced allocations due to Jackson serialization

Introduced and implemented new helper functions in JacksonUtils to enable reuse of

SerializerProvider objects.

Additionally, disabled backwards compatibility for map-based rows in the GroupByQueryToolChest by default, which eliminates the need to copy the heavyweight ObjectMapper. Introduced a configuration option to allow administrators to explicitly enable backwards compatibility.

#12468

# Updated IPAddress Java library

Added a new IPAddress Java library dependency to handle IP addresses. The library includes IPv6 support. Additionally, migrated IPv4 functions to use the new library.

#11634

# Query performance improvements

Optimized SQL operations and functions as follows:

Vectorized numeric latest aggregators (#12439)

Optimized isEmpty() and equals() on RangeSets (#12477)

Optimized reuse of Yielder objects (#12475)

Operations on numeric columns with indexes are now faster (#12830)

Optimized GroupBy by reducing allocations. Reduced allocations by reusing entry and key holders (#12474)

Added a vectorized version of string last aggregator (#12493)

Added Direct UTF-8 access for IN filters (#12517)

Enabled virtual columns to cache their outputs in case Druid calls them multiple times on the same underlying row (#12577)

Druid now rewrites a join as a filter when possible in IN joins (#12225)

Added automatic sizing for GroupBy dictionaries (#12763)

Druid now distributes JDBC connections more evenly amongst brokers (#12817)

# Streaming ingestion

# Kafka consumers

Previously, consumers that were registered and used for ingestion persisted until Kafka deleted them. They were only used to make sure that an entire topic was consumed. There are no longer consumer groups that linger.

#12842

# Kinesis ingestion

You can now perform Kinesis ingestion even if there are empty shards. Previously, all shards had to have at least one record.

#12792

# Batch ingestion

# Batch ingestion from S3

You can now ingest data from endpoints that are different from your default S3 endpoint and signing region.

For more information, see S3 config.

#11798

# Improvements to ingestion in general

This release includes the following improvements for ingestion in general.

# Increased robustness for task management

Added setNumProcessorsPerTask to prevent various automatically-sized thread pools from becoming unreasonably large. It isn't ideal for each task to size its pools as if it is the only process on the entire machine. On large machines, this solves a common cause of OutOfMemoryError due to "unable to create native thread".

#12592

# Avatica JDBC driver

The JDBC driver now follows the JDBC standard and uses two kinds of statements, Statement and PreparedStatement.

#12709

# Eight hour granularity

Druid now accepts the EIGHT_HOUR granularity. You can segment incoming data to EIGHT_HOUR buckets as well as group query results by eight hour granularity.

#12717

# Ingestion general

# Updated Avro extension

The previous Avro extension leaked objects from the parser. If these objects leaked into your ingestion, you had objects being stored as a string column with the value as the .toString(). This string column will remain after you upgrade but will return Map.toString() instead of GenericRecord.toString. If you relied on the previous behavior, you can use the Avro extension from an earlier release.

#12828

# Sampler API

The sampler API has additional limits: maxBytesInMemory and maxClientResponseBytes. These options augment the existing options numRows and timeoutMs. maxBytesInMemory can be used to control the memory usage on the Overlord while sampling. maxClientResponseBytes can be used by clients to specify the maximum size of response they would prefer to handle.

#12947

# SQL

# Column order

The DruidSchema and SegmentMetadataQuery properties now preserve column order instead of ordering columns alphabetically. This means that query order better matches ingestion order.

#12754

# Converting JOINs to filter

You can improve performance by pushing JOINs partially or fully to the base table as a filter at runtime by setting the enableRewriteJoinToFilter context parameter to true for a query.

Druid now pushes down join filters in case the query computing join references any columns from the right side.

#12749

#12868

# Add is_active to sys.segments

Added is_active as shorthand for (is_published = 1 AND is_overshadowed = 0) OR is_realtime = 1). This represents "all the segments that should be queryable, whether or not they actually are right now".

#11550

# useNativeQueryExplain now defaults to true

The useNativeQueryExplain property now defaults to true. This means that EXPLAIN PLAN FOR returns the explain plan as a JSON representation of equivalent native query(s) by default. For more information, see Broker Generated Query Configuration Supplementation.

#12936

# Running queries with inline data using druid query engine

Some queries that do not refer to any table, such as select 1, are now always translated to a native Druid query with InlineDataSource before execution. If translation is not possible, for queries such as SELECT (1, 2), then an error occurs. In earlier versions, this query would still run.

#12897

# Coordinator/Overlord

# You can configure the Coordinator to kill segments in the future

You can now set druid.coordinator.kill.durationToRetain to a negative period to configure the Druid cluster to kill segments whose interval_end is a date in the future. For example, PT-24H would allow segments to be killed if their interval_end date was 24 hours or less into the future at the time that the kill task is generated by the system.

A cluster operator can also disregard the druid.coordinator.kill.durationToRetain entirely by setting a new configuration, druid.coordinator.kill.ignoreDurationToRetain=true. This ignores interval_end date when looking for segments to kill, and can instead kill any segment marked unused. This new configuration is turned off by default, and a cluster operator should fully understand and accept the risks before enabling it.

# Improved Overlord stability

Reduced contention between the management thread and the reception of status updates from the cluster. This improves the stability of Overlord and all tasks in a cluster when there are large (1000+) task counts.

#12099

# Improved Coordinator segment logging

Updated Coordinator load rule logging to include current replication levels. Added missing segment ID and tier information from some of the log messages.

#12511

# Optimized overlord GET tasks memory usage

Addressed the significant memory overhead caused by the web-console indirectly calling the Overlord’s GET tasks API. This could cause unresponsiveness or Overlord failure when the ingestion tab was opened multiple times.

#12404

# Reduced time to create intervals

In order to optimize segment cost computation time by reducing time taken for interval creation, store segment interval instead of creating it each time from primitives and reduce memory overhead of storing intervals by interning them. The set of intervals for segments is low in cardinality.

#12670

# Brokers/Overlord

Brokers now have a default of 25MB maximum queued per query. Previously, there was no default limit. Depending on your use case, you may need to increase the value, especially if you have large result sets or large amounts of intermediate data. To adjust the maximum memory available, use the druid.broker.http.maxQueuedBytes property.

For more information, see Configuration reference.

# Web console

Prepare to have your Web Console experience elevated! - @vogievetsky

# New query view (WorkbenchView) with tabs and long running query support

You can use the new query view to execute multi-stage, task based, queries with the /druid/v2/sql/task and /druid/indexer/v1/task/* APIs as well as native and sql-native queries just like the old Query view. A key point of the sql-msq-task based queries is that they may run for a long time. This inspired / necessitated many UX changes including, but not limited to the following:

# Tabs

You can now have many queries stored and running at the same time, significantly improving the query view UX.

You can open several tabs, duplicate them, and copy them as text to paste into any console and reopen there.

# Progress reports (counter reports)

Queries run with the multi-stage query task engine have detailed progress reports shown in the summary progress bar and the in detail execution table that provides summaries of the counters for every step.

# Error and warning reports

Queries run with the multi-stage query task engine present user friendly warnings and errors should anything go wrong.

The new query view has components to visualize these with their full detail including a stack-trace.

# Recent query tasks panel

Queries run with the multi-stage query task engine are tasks. This makes it possible to show queries that are executing currently and that have executed in the recent past.

For any query in the Recent query tasks panel you can view the execution details for it and you can also attach it as a new tab and continue iterating on the query. It is also possible to download the "query detail archive", a JSON file containing all the important details for a given query to use for troubleshooting.

# Connect external data flow

Connect external data flow lets you use the sampler to sample your source data to, determine its schema and generate a fully formed SQL query that you can edit to fit your use case before you launch your ingestion job. This point-and-click flow will save you much typing.

# Preview button

The Preview button appears when you type in an INSERT or REPLACE SQL query. Click the button to remove the INSERT or REPLACE clause and execute your query as an "inline" query with a limi). This gives you a sense of the shape of your data after Druid applies all your transformations from your SQL query.

# Results table

The query results table has been improved in style and function. It now shows you type icons for the column types and supports the ability to manipulate nested columns with ease.

# Helper queries

The Web Console now has some UI affordances for notebook and CTE users. You can reference helper queries, collapsable elements that hold a query, from the main query just like they were defined with a WITH statement. When you are composing a complicated query, it is helpful to break it down into multiple queries to preview the parts individually.

# Additional Web Console tools

More tools are available from the ... menu:

Explain query - show the query plan for sql-native and multi-stage query task engine queries.

Convert ingestion spec to SQL - Helps you migrate your native batch and Hadoop based specs to the SQL-based format.

Open query detail archive - lets you open a query detail archive downloaded earlier.

Load demo queries - lets you load a set of pre-made queries to play around with multi-stage query task engine functionality.

# New SQL-based data loader

The data loader exists as a GUI wizard to help users craft a JSON ingestion spec using point and click and quick previews. The SQL data loader is the SQL-based ingestion analog of that.

Like the native based data loader, the SQL-based data loader stores all the state in the SQL query itself. You can opt to manipulate the query directly at any stage. See (#12919) for more information about how the data loader differs from the Connect external data workflow.

# Other changes and improvements

The query view has so much new functionality that it has moved to the far left as the first view available in the header.

You can now click on a datasource or segment to see a preview of the data within.

The task table now explicitly shows if a task has been canceled in a different color than a failed task.

The user experience when you view a JSON payload in the Druid console has been improved. There’s now syntax highlighting and a search.

The Druid console can now use the column order returned by a scan query to determine the column order for reindexing data.

The way errors are displayed in the Druid console has been improved. Errors no longer appear as a single long line.

See (#12919) for more details and other improvements

# Metrics

# Sysmonitor stats for Peons

Sysmonitor stats, like memory or swap, are no longer reported since Peons always run on the same host as MiddleManagerse. This means that duplicate stats will no longer be reported.

#12802

# Prometheus

You can now include the host and service as labels for Prometheus by setting the following properties to true:

druid.emitter.prometheus.addHostAsLabel

druid.emitter.prometheus.addServiceAsLabel

#12769

# Rows per segment

(Experimental) You can now see the average number of rows in a segment and the distribution of segments in predefined buckets with the following metrics: segment/rowCount/avg and segment/rowCount/range/count.

Enable the metrics with the following property: org.apache.druid.server.metrics.SegmentStatsMonitor

#12730

# New sqlQuery/planningTimeMs metric

There’s a new sqlQuery/planningTimeMs metric for SQL queries that computes the time it takes to build a native query from a SQL query.

#12923

# StatsD metrics reporter

The StatsD metrics reporter extension now includes the following metrics:

coordinator/time

coordinator/global/time

tier/required/capacity

tier/total/capacity

tier/replication/factor

tier/historical/count

compact/task/count

compactTask/maxSlot/count

compactTask/availableSlot/count

segment/waitCompact/bytes

segment/waitCompact/count

interval/waitCompact/count

segment/skipCompact/bytes

segment/skipCompact/count

interval/skipCompact/count

segment/compacted/bytes

segment/compacted/count

interval/compacted/count

#12762

# New worker level task metrics

Added a new monitor, WorkerTaskCountStatsMonitor, that allows each middle manage worker to report metrics for successful / failed tasks, and task slot usage.

#12446

# Improvements to the JvmMonitor

The JvmMonitor can now handle more generation and collector scenarios. The monitor is more robust and works properly for ZGC on both Java 11 and 15.

#12469

# Garbage collection

Garbage collection metrics now use MXBeans.

#12481

# Metric for task duration in the pending queue

Introduced the metric task/pending/time to measure how long a task stays in the pending queue.

#12492

# Emit metrics object for Scan, Timeseries, and GroupBy queries during cursor creation

Adds vectorized metric for scan, timeseries and groupby queries.

#12484

# Emit state of replace and append for native batch tasks

Druid now emits metrics so you can monitor and assess the use of different types of batch ingestion, in particular replace and tombstone creation.

#12488

#12840

# KafkaEmitter emits queryType

The KafkaEmitter now properly emits the queryType property for native queries.

#12915

# Security

You can now hide properties that are sensitive in the API response from /status/properties, such as S3 access keys. Use the druid.server.hiddenProperties property in common.runtime.properties to specify the properties (case insensitive) you want to hide.

#12950

# Other changes

You can now configure the retention period for request logs stored on disk with the druid.request.logging.durationToRetain property. Set the retention period to be longer than P1D (#12559)

You can now specify liveness and readiness probe delays for the historical StatefulSet in your values.yaml file. The default is 60 seconds (#12805)

Improved exception message for native binary operators (#12335)

​​Improved error messages when URI points to a file that doesn't exist (#12490)

​​Improved build performance of modules (#12486)

Improved lookups made using the druid-kafka-extraction-namespace extension to handle records that have been deleted from a kafka topic (#12819)

Updated core Apache Kafka dependencies to 3.2.0 (#12538)

Updated ORC to 1.7.5 (#12667)

Updated Jetty to 9.4.41.v20210516 (#12629)

Added Zstandard compression library to CompressionStrategy (#12408)

Updated the default gzip buffer size to 8 KB to for improved performance (#12579)

Updated the default inputSegmentSizeBytes in Compaction configuration to 100,000,000,000,000 (~100TB)

# Bug fixes

Druid 24.0 contains over 68 bug fixes. You can find the complete list here

# Upgrading to 24.0

# Permissions for multi-stage query engine

To read external data using the multi-stage query task engine, you must have READ permissions for the EXTERNAL resource type. Users without the correct permission encounter a 403 error when trying to run SQL queries that include EXTERN.

The way you assign the permission depends on your authorizer. For example, with [basic security]((/docs/development/extensions-core/druid-basic-security.md) in Druid, add the EXTERNAL READ permission by sending a POST request to the roles API.

The example adds permissions for users with the admin role using a basic authorizer named MyBasicMetadataAuthorizer. The following permissions are granted:

DATASOURCE READ

DATASOURCE WRITE

CONFIG READ

CONFIG WRITE

STATE READ

STATE WRITE

EXTERNAL READ

curl --location --request POST 'http://localhost:8081/druid-ext/basic-security/authorization/db/MyBasicMetadataAuthorizer/roles/admin/permissions' \
--header 'Content-Type: application/json' \
--data-raw '[
{
"resource": {
"name": ".*",
"type": "DATASOURCE"
},
"action": "READ"
},
{
"resource": {
"name": ".*",
"type": "DATASOURCE"
},
"action": "WRITE"
},
{
"resource": {
"name": ".*",
"type": "CONFIG"
},
"action": "READ"
},
{
"resource": {
"name": ".*",
"type": "CONFIG"
},
"action": "WRITE"
},
{
"resource": {
"name": ".*",
"type": "STATE"
},
"action": "READ"
},
{
"resource": {
"name": ".*",
"type": "STATE"
},
"action": "WRITE"
},
{
"resource": {
"name": "EXTERNAL",
"type": "EXTERNAL"
},
"action": "READ"
}
]'

# Behavior for unused segments

Druid automatically retains any segments marked as unused. Previously, Druid permanently deleted unused segments from metadata store and deep storage after their duration to retain passed. This behavior was reverted from 0.23.0.

#12693

# Default for druid.processing.fifo

The default for druid.processing.fifo is now true. This means that tasks of equal priority are treated in a FIFO manner. For most use cases, this change can improve performance on heavily loaded clusters.

#12571

# Update to JDBC statement closure

In previous releases, Druid automatically closed the JDBC Statement when the ResultSet was closed. Druid closed the ResultSet on EOF. Druid closed the statement on any exception. This behavior is, however, non-standard.

In this release, Druid's JDBC driver follows the JDBC standards more closely:

The ResultSet closes automatically on EOF, but does not close the Statement or PreparedStatement. Your code must close these statements, perhaps by using a try-with-resources block.

The PreparedStatement can now be used multiple times with different parameters. (Previously this was not true since closing the ResultSet closed the PreparedStatement.)

If any call to a Statement or PreparedStatement raises an error, the client code must still explicitly close the statement. According to the JDBC standards, statements are not closed automatically on errors. This allows you to obtain information about a failed statement before closing it.

If you have code that depended on the old behavior, you may have to change your code to add the required close statement.

#12709

# Known issues

# Credits

@2bethere

@317brian

@a2l007

@abhagraw

@abhishekagarwal87

@abhishekrb19

@adarshsanjeev

@aggarwalakshay

@AmatyaAvadhanula

@BartMiki

@capistrant

@chenrui333

@churromorales

@clintropolis

@cloventt

@CodingParsley

@cryptoe

@dampcake

@dependabot[bot]

@dherg

@didip

@dongjoon-hyun

@ektravel

@EsoragotoSpirit

@exherb

@FrankChen021

@gianm

@hellmarbecker

@hwball

@iandr413

@imply-cheddar

@jarnoux

@jasonk000

@jihoonson

@jon-wei

@kfaraz

@LakshSingla

@liujianhuanzz

@liuxiaohui1221

@lmsurpre

@loquisgon

@machine424

@maytasm

@MC-JY

@Mihaylov93

@nishantmonu51

@paul-rogers

@petermarshallio

@pjfanning

@rockc2020

@rohangarg

@somu-imply

@suneet-s

@superivaj

@techdocsmith

@tejaswini-imply

@TSFenwick

@vimil-saju

@vogievetsky

@vtlim

@williamhyun

@wiquan

@writer-jill

@xvrl

@yuanlihan

@zachjsh

@zemin-piao