Project: Druid (App Definition and Development / Database)
File: radstack.pdf

The RADStack: Open Source Lambda Architecture for
Interactive Analytics
Fangjin Y ang, Gian Merlino, Nelson Ray, Xavier Léauté, Himanshu Gupta, Eric Tschetter
{fangjinyang, gianmerlino, ncray86, xavier.leaute, g.himanshu, echeddar}@gmail.com
ABSTRACT
The Real-time Analytics Data Stack, colloquially referred to
as the RADStack, is an open-source data analytics stack de-
signed to provide fast, ﬂexible queries over up-to-the-second
data. It is designed to overcome the limitations of either
a purely batch processing system (it takes too long to sur-
face new events) or a purely real-time system (it’s diﬃcult
to ensure that no data is left behind and there is often no
way to correct data after initial processing). It will seam-
lessly return best-eﬀort results on very recent data combined
with guaranteed-correct results on older data. In this paper,
we introduce the architecture of the RADStack and discuss
our methods of providing interactive analytics and a ﬂexible
data processing environment to handle a variety of real-world
workloads.
1. INTRODUCTION
The rapid growth of the Hadoop[16] ecosystem has en-
abled many organizations to ﬂexibly process and gain in-
sights from large quantities of data. These insights are typ-
ically generated from business intelligence, or OnLine Ana-
lyticalProcessing(OLAP)queries. Hadoophasproventobe
an extremely eﬀective framework capable of providing many
analytical insights and is able to solve a wide range of dis-
tributed computing problems. However, as much as Hadoop
is lauded for its wide range of use cases, it is derided for its
high latency in processing and returning results. A common
approach to surface data insights is to run MapReduce jobs
that may take several hours to complete.
Data analysis and data-driven applications are becoming
increasingly important in industry, and the long query times
encountered with using batch frameworks such as Hadoop
are becoming increasingly intolerable. User facing appli-
cations are replacing traditional reporting interfaces as the
preferred means for organizations to derive value from their
datasets. In order to provide an interactive user experi-
ence with data applications, queries must complete in an
order of milliseconds. Because most of these interactions
revolve around data exploration and computation, organi-
zations quickly realized that in order to support low latency
queries, dedicated serving layers were necessary. Today,
most of these serving layers are Relational Database Man-
agement Systems (RDBMS) or NoSQL key/value stores.
Neither RDBMS nor NoSQL key/value stores are partic-
ularly designed for analytics [19], but these technologies are
still frequently selected as serving layers. Solutions that in-
volve these broad-focus technologies can be inﬂexible once
tailored to the analytics use case, or suﬀer from architecture
drawbacks that prevent them from returning queries fast
enough to power interactive, user-facing applications [20].An ideal data serving layer alone is often not suﬃcient
as a complete analytics solution. In most real-world use
cases, rawdatacannotbedirectlystoredintheservinglayer.
Raw data suﬀers from many imperfections and must ﬁrst
be processed (transformed, or cleaned) before it is usable
[17]. The drawback of this requirement is that loading and
processing batch data is slow, and insights on events cannot
be obtained until hours after the events have occurred.
To address the delays in data freshness caused by batch
processing frameworks, numerous open-source stream pro-
cessingframeworkssuchasApacheStorm[12], ApacheSpark
Streaming[25], and Apache Samza[1] have gained popular-
ity for oﬀering a low-latency model to ingest and process
event streams at near real-time speeds. The drawback of
almost all stream processors is that they do not necessarily
provide the same correctness guarantees as batch processing
frameworks. Events can come in days late, and may need
to be corrected after the fact. Large batches of data may
also need to be reprocessed when new columns are added or
removed.
Combining batch processing, streaming processing, and
a serving layer in a single technology stack is known as a
lambda architecture[9]. In lambda architectures, data en-
tering the system is concurrently fed to both the batch and
streaming processing layer. The streaming layer is responsi-
ble for immediately processing incoming data, however, the
processed data may suﬀer from duplicated events and other
imperfections in data accuracy. The batch layer processes
incoming data much slower than the streaming layer, but is
able to provide accurate views of data. The serving layer
merges the results from the batch and streaming layers and
provides an interface for queries. Although each individual
component in a lambda architecture has their own limita-
tions, the pieces complement each other extremely well and
the overall stack is robust enough to handle a wide array of
data processing and querying challenges at scale.
The RADStack is an open source lambda architecture im-
plementation meant to oﬀer ﬂexible, low-latency analytic
queries on near real-time data. The solution combines the
low latency guarantees of stream processors and the correct-
ness and ﬂexibility guarantees of batch processors. It also
introduces a serving layer speciﬁcally designed for interac-
tive analytics. The stack’s main building blocks are Apache
Kafka[11], Apache Samza, Apache Hadoop, and Druid [23],
and we have found that the combination of technologies is
ﬂexible enough to handle a wide variety of processing re-
quirements and query loads. Each piece of the stack is de-
signed to do a speciﬁc set of things very well. This paper
will cover the details and design principles of the RADStack.
Our contributions are around the architecture of the stack
itself, the introduction of Druid as a serving layer, and our
1Figure 1: The components of the RADStack. Kafka
acts as the event delivery endpoints. Samza and
Hadoop process data to load data into Druid. Druid
acts as the endpoint for queries.
model for unifying real-time and historical workﬂows.
The structure of the paper is as follows: Section 2 de-
scribes the problems and use cases that led to the creation
of the RADStack. Section 3 describes Druid, the serving
layer of the stack, and how Druid is built for real-time and
batch data ingestion, as well as exploratory analytics. Sec-
tion 4 covers the role of Samza and Hadoop for data pro-
cessing, and Section 5 describes the role of Kafka for event
delivery. In Section 6, we present our production metrics.
Section 7 presents our experiences with running the RAD-
Stack in production, and in Section 8 we discuss the related
solutions.
2. BACKGROUND
The RADStack was ﬁrst developed to address problems
in the online advertising. In online advertising, automated
systems from diﬀerent organizations will place bids against
one another to display users ads in the milliseconds before a
webpageloads. Theseactionsgenerateatremendousvolume
of data. The data shown in Table 1 is an example of such
data. Each event is comprised of three components: a times-
tamp indicating when the event occurred; a set of dimen-
sions indicating various attributes about the event; and a set
of metrics concerning the event. Organizations frequently
serve this insights to this data to ad publishers through vi-
sualizations and data applications. These applications must
rapidly compute drill-down and aggregates with this data,
and answer questions such as “How many clicks occurred
over the span of one week for publisher google.com?” or
“How many impressions were seen over the last quarter in
San Francisco?”. Queries over any arbitrary number of di-
mensions should return in a few hundred milliseconds.
As an additional requirement, user-facing applications of-
ten face highly concurrent workloads and good applications
needtoproviderelativelyconsistentperformancetoallusers.
Of course, backend infrastructure also needs to be highly
available. Downtime is costly and many businesses cannot
aﬀord to wait if a system is unavailable in the face of soft-
ware upgrades or network failure.
To address these requirements of scale, stability, and per-
formance, we created Druid. Druid was designed from the
ground up to provide arbitrary data exploration, low la-
tency aggregations, and fast data ingestion. Druid was also
designed to accept fully denormalized data, and moves away
from the traditional relational model. Since most raw data
is not denormalized, it must be processed before it can be
ingested and queried. Multiple streams of data had to bejoined, cleaned up, and transformed before it was usable in
Druid, but that was the trade-oﬀ we were willing to make
in order to get the performance necessary to power an in-
teractive data application. We introduced stream process-
ing to our stack to provide the processing required before
raw data could be loaded into Druid. Our stream process-
ing jobs range from simple data transformations, such as id
to name lookups, up to complex operations such as multi-
stream joins. Pairing Druid with a stream processor enabled
ﬂexible data processing and querying, but we still had prob-
lems with event delivery. Our events were delivered from
many diﬀerent locations and sources, and peaked at several
million events per second. We required a high throughput
message bus that could hold these events for consumpation
by our stream processor. To simplify data transmission for
our clients, we wanted the message bus to be the single de-
livery endpoint for events entering our cluster.
Our stack would be complete here if real-time processing
were perfect, but the open source stream processing space
is still young. Processing jobs can go down for extended
periods of time and events may be delivered more than
once. These are realities of any production data pipeline.
To overcome these issues, we included Hadoop in our stack
to periodically clean up any data generated by the real-time
pipeline. We stored a copy of the raw events we received in
a distributed ﬁle system, and periodically ran batch process-
ing jobs over this data. The high level architecture of our
setup is shown in Figure 1. Each component is designed
to do a speciﬁc set of things well, and there is isolation in
terms of functionality. Individual components can entirely
fail without impacting the services of the other components.
3. THE SERVING LAYER
Druid is a column-oriented data store designed for ex-
ploratory analytics and is the serving layer in the RAD-
Stack. A Druid cluster consists of diﬀerent types of nodes
and, similar to the overall design of the RADStack, each
node type is instrumented to perform a speciﬁc set of things
well. We believe this design separates concerns and simpli-
ﬁes the complexity of the overall system. To solve complex
data analysis problems, the diﬀerent node types come to-
gether to form a fully working system. The composition of
and ﬂow of data in a Druid cluster are shown in Figure 2.
3.1 Segments
Data tables in Druid (called ”data sources”) are collec-
tions of timestamped events and partitioned into a set of
segments, where each segment is typically 5–10 million rows.
Segments represent the fundamental storage unit in Druid
and Druid queries only understand how to scan segments.
Druid always requires a timestamp column as a method
of simplifying data distribution policies, data retention poli-
cies, and ﬁrst level query pruning. Druid partitions its data
sources into well deﬁned time intervals, typically an hour
or a day, and may further partition on values from other
columns to achieve the desired segment size. The time gran-
ularity to partition segments is a function of data volume
and time range. A data set with timestamps spread over a
year is better partitioned by day, and a data set with times-
tamps spread over a day is better partitioned by hour.
Segments are uniquely identiﬁed by a data source iden-
tiﬁer, the time interval of the data, and a version string
2Timestamp Publisher Advertiser Gender City Click Price
2011-01-01T01:01:35Z bieberfever.com google.com Male San Francisco 0 0.65
2011-01-01T01:03:63Z bieberfever.com google.com Male Waterloo 0 0.62
2011-01-01T01:04:51Z bieberfever.com google.com Male Calgary 1 0.45
2011-01-01T01:00:00Z ultratrimfast.com google.com Female Taiyuan 0 0.87
2011-01-01T02:00:00Z ultratrimfast.com google.com Female New York 0 0.99
2011-01-01T02:00:00Z ultratrimfast.com google.com Female Vancouver 1 1.53
Table 1: Sample ad data. These events are created when users views or clicks on ads.
Real-time 
Nodes 
Coordinator 
Nodes Broker Nodes 
Historical 
Nodes Metadata 
Storage 
Distributed 
Coordination 
Deep 
Storage Streaming 
Data 
Batch 
Data Client 
Queries 
Queries 
Metadata 
Data/Segments Druid Nodes 
External Dependencies 
Figure 2: An overview of a Druid cluster and the ﬂow of data through the cluster.
that increases whenever a new segment is created. The ver-
sion string indicates the freshness of segment data; segments
with later versions have newer views of data (over some
time range) than segments with older versions. This seg-
ment metadata is used by the system for concurrency con-
trol; read operations always access data in a particular time
range from the segments with the latest version identiﬁers
for that time range.
Druid segments are stored in a column orientation. Given
that Druid is best used for aggregating event streams, the
advantagesofstoringaggregateinformationascolumnsrather
than rows are well documented [2]. Column storage allows
for more eﬃcient CPU usage as only what is needed is ac-
tually loaded and scanned. In a row oriented data store,
all columns associated with a row must be scanned as part
of an aggregation. The additional scan time can introduce
signiﬁcant performance degradations [2].
Druid nodes use one thread to scan one segment at a time,
and the amount of data that can be scanned in parallel is
directly correlated to the number of available cores in the
cluster. Segments are immutable, and hence, this no con-
tention between reads and writes in a segment.
A single query may scan thousands of segments concur-
rently, and many queries may run at the same time. We
want to ensure that the entire cluster is not starved out
while a single expensive query is executing. Thus, segments
have an upper limit in how much data they can hold, and
are sized to be scanned in a few milliseconds. By keeping
segment computation very fast, cores and other resources
are constantly being yielded. This ensures segments from
diﬀerent queries are always being scanned.
Druid segments are very self-contained for the time inter-
val of data that they hold. Column data is stored directlyin the segment. Druid has multiple column types to repre-
sent various data formats. Timestamps are stored in long
columns, dimensions are stored in string columns, and met-
rics are stored in int, ﬂoat, long or double columns. Depend-
ing on the column type, diﬀerent compression methods may
be used. Metric columns are compressed using LZ4[3] com-
pression. String columns are dictionary encoded, similar to
other data stores such as PowerDrill[8]. Additional indexes
may be created for particular columns. For example, Druid
will by default create inverted indexes for string columns.
3.2 Streaming Data Ingestion
Druid real-time nodes encapsulate the functionality to in-
gest, query, and create segments from event streams. Events
indexed via these nodes are immediately available for query-
ing. The nodes are only concerned with events for a rela-
tively small time range (e.g. hours) and periodically hand
oﬀ immutable batches of events they have collected over
this small time range to other nodes in the Druid cluster
that are specialized in dealing with batches of immutable
events. The nodes announce their online state and the data
they serve using a distributed coordination service (this is
currently Zookeeper[10]).
Real-time nodes employ a log structured merge tree[14]
for recently ingested data. Incoming events are ﬁrst stored
in an in-memory buﬀer. The in-memory buﬀer is directly
queryable and Druid behaves as a key/value store for queries
on events that exist in this JVM heap-based store. The in-
memory buﬀer is heavily write optimized, and given that
Druid is really designed for heavy concurrent reads, events
do not remain in the in-memory buﬀer for very long. Real-
timenodespersisttheirin-memoryindexes todiskeitherpe-
riodically or after some maximum row limit is reached. This
persist process converts data stored in the in-memory buﬀer
3event_23312
event_23481
event_23593
...
event_1234
event_2345
...event_3456
event_4567
...
event_5678
event_6789
...event_7890
event_8901
...Disk and persisted indexesHeap and in-memory index
Persistevent_34982
event_35789
event_36791
...
event_1234
event_2345
...event_3456
event_4567
...
event_5678
event_6789
...event_7890
event_8901
...Off-heap memory and 
persisted indexes
LoadQueries
Figure 3: Real-time nodes write events to a write
optimized in-memory index. Periodically, events are
persisted to disk, converting the write optimized for-
mat to a read optimized one. On a periodic basis,
persisted indexes are then merged together and the
ﬁnal segment is handed oﬀ. Queries will hit both
the in-memory and persisted indexes.
to the column oriented segment storage format described
in Section 3.1. Persisted segments are memory mapped
and loaded to oﬀ-heap memory such that they can still be
queried. This is illustrated in Figure 4. Data is continuously
queryable during the persist process.
Real-time ingestion in Druid is self-throttling. If a signiﬁ-
cant spike occurs in event volume from the upstream event
producer, there are a few safety mechanisms built in. Re-
call that events are ﬁrst stored in an in-memory buﬀer and
persists can occur when a maximum conﬁgurable row limit
is reached. Spikes in event volume should cause persists
to occur more often and not overﬂow the in-memory buﬀer.
However, theprocessofbuildingasegmentdoesrequiretime
and resources. If too many concurrent persists occur, and if
events are added to the in-memory buﬀer faster than they
can be removed through the persist process, problems can
still arise. Druid sets a limit on the maximum number of
persists that can occur at a time, and if this limit is reached,
Druid will begin to throttle event ingestion. In this case, the
onus is on the upstream consumer to be resilient in the face
of increasing backlog.
Real-timenodesstorerecentdataforaconﬁgurableperiod
of time, typically an hour. This period is referred to as the
segment granularity period. The nodes employ a sliding
window to accept and reject events and use the wall-clock
time as the basis of the window. Events within a range of
the node’s wall-clock time are accepted, and events outside
this window are dropped. This period is referred to as the
window period and typical window periods are 10 minutes
in length. At the end of the segment granularity period plus
the window period, a real-time node will hand oﬀ the data
it has collected during the segment granularity period. The
use of the window period means that delayed events may
be dropped. In practice, we see that these occurrences are
rare, but they do occur. Druid’s real-time logic does not
guarantee exactly once processing and is instead best eﬀort.
The lack of exactly once processing in Druid is one of the
motivations for requiring batch ﬁxup in the RADStack.For further clariﬁcation, consider Figure 4. Figure 4 illus-
trates the operations of a real-time node. The node starts
at 13:37 and, with a 10 minute window period, will only
accept events for a window between 13:27 and 13:47. When
the ﬁrst events are ingested, the node announces that it is
serving a segment for an interval from 13:00 to 14:00. Every
10 minutes (the persist period is conﬁgurable), the node will
ﬂush and persist its in-memory buﬀer to disk. Near the end
of the hour, the node will likely see events for 14:00 to 15:00.
When this occurs, the node prepares to serve data for the
next hour and creates a new in-memory buﬀer. The node
then announces that it is also serving a segment from 14:00
to 15:00. At 13:10, which is the end of the hour plus the
window period, the node begins the hand oﬀ process.
3.3 Hand off
Real-time nodes are designed to deal with a small win-
dow of recent data and need periodically hand oﬀ segments
they’ve built. The hand-oﬀ process ﬁrst involves a com-
paction step. The compaction process ﬁnds all the segments
that were created for a speciﬁc interval of time (for example,
all the segments that were created by intermediate persists
over the period of an hour). These segments are merged
together to form a ﬁnal immutable segment for handoﬀ.
Handoﬀ occurs in a few steps. First, the ﬁnalized segment
is uploaded to a permanent backup storage, typically a dis-
tributedﬁlesystemsuchasS3[5]orHDFS[16], whichDruid
refers to as “deep storage”. Next, an entry is created in the
metadata store (typically a RDBMS such as MySQL) to in-
dicate that a new segment has been created. This entry in
the metadata store will eventually cause other nodes in the
Druid cluster to download and serve the segment. The real-
timenodecontinuestoservethesegmentuntilitnoticesthat
the segment is available on Druid historical nodes, which are
nodes that are dedicated to serving historical data. At this
point, the segment is dropped and unannounced from the
real-time node. The entire handoﬀ process is ﬂuid; data re-
mains continuously queryable throughout the entire handoﬀ
process. Segments created by real-time processing are ver-
sioned by the start of the segment granularity interval.
3.4 Batch Data Ingestion
The core component used by real-time ingestion is a hash
map that can be incrementally populated and ﬁnalized to
createanimmutablesegment. Thiscorecomponentisshared
across both real-time and batch ingestion. Druid has built
in support for creating segments by leveraging Hadoop and
running MapReduce jobs to partition data for segments.
Events can be read in one at a time directly from static
ﬁles in a ”streaming” fashion.
Similar to the real-time ingestion logic, segments created
through batch ingestion are directly uploaded to deep stor-
age. Druid’s Hadoop-based batch indexer will also create an
entry in the metadata storage once all segments have been
created. The version of the segments created by batch inges-
tion are based on the time the batch processing job started
at.
3.5 Unifying Views
When new entries are created in the metadata storage,
they will eventually be noticed by Druid coordinator nodes.
Druid coordinator nodes poll the metadata storage for what
segments should be loaded on Druid historical nodes, and
413:00 14:00 15:00
13:37
- node starts
- announce segment 
for data 13:00-14:0013:47
persist data for 13:00-14:00
~14:00
- announce segment 
for data 14:00-15:0014:10
- merge and handoff for data 13:00-14:00
- persist data for 14:00-15:00~14:11
- unannounce segment 
for data 13:00-14:00
13:57
persist data for 13:00-14:0014:07
persist data for 13:00-14:00Figure 4: The node starts, ingests data, persists, and periodically hands data oﬀ. This process repeats
indeﬁnitely. The time periods between diﬀerent real-time node operations are conﬁgurable.
compare the result with what is actually loaded on those
nodes. Coordinator nodes will tell historical nodes to load
new segments, drop outdated segments, and move segments
across nodes.
Druid historical nodes are very simple in operation. They
know how to load, drop, and respond to queries to scan
segments. Historical nodes typically store all the data that
is older than an hour (recent data lives on the real-time
node). The real-time handoﬀ process requires that a histor-
ical must ﬁrst load and begin serving queries for a segment
beforethatsegmentcanbedroppedfromthereal-timenode.
Since segments are immutable, the same copy of a segment
can exist on multiple historical nodes and real-time nodes.
Most nodes in typical production Druid clusters are histor-
ical nodes.
To consolidate results from historical and real-time nodes,
Druid has a set of broker nodes which act as the client query
endpoint. Broker nodes in part function as query routers to
historical and real-time nodes. Broker nodes understand
the metadata published in distributed coordination service
(Zookeeper) about what segments are queryable and where
those segments are located. Broker nodes route incoming
queries such that the queries hit the right historical or real-
time nodes. Broker nodes also merge partial results from
historical and real-time nodes before returning a ﬁnal con-
solidated result to the caller.
Broker nodes maintain a segment timeline containing in-
formation about what segments exist in the cluster and the
version of those segments. Druid uses multi-version concun-
currency control to manage how data is extracted from seg-
ments. Segments with higher version identiﬁers have prece-
dence over segments with lower version identiﬁers. If two
segments exactly overlap for an interval, Druid only consid-
ers the data from the segment with the higher version. This
is illustrated in Figure 5
Segments are inserted into the timeline as they are an-
nounced. The timeline sorts the segment based on their
data interval in a data structure similar to an interval tree.
Lookups in the timeline will return all segments with in-
Segment_v4 
Segment_v3 
Segment_v2 
Segment_v1 
Segment_v4 Segment_v3 Segment_v1 Day 1 Day 2 Day 3 
Results Figure 5: Druid utilizes multi-version concurrency
control and reads data from segments with the latest
version for a given interval. Segments that are that
completely overshadowed are ignored and eventually
automatically dropped from the cluster.
tervals that overlap the lookup interval, along with interval
ranges for which the data in a segment is valid.
Brokers extract the interval of a query and use it for
lookups into the timeline. The result of the timeline is used
to remap the original query into a set of speciﬁc queries
for the actual historical and real-time nodes that hold the
pertinent query data. The results from the historical and
real-time nodes are ﬁnally merged by the broker, which re-
turns the ﬁnal result to the caller.
The coordinator node also builds a segment timeline for
segments in the cluster. If a segment is completely over-
shadowed by one or more segments, it will be ﬂagged in this
timeline. When the coordinator notices overshadowed seg-
ments, it tells historical nodes to drop these segments from
the cluster.
4. THE PROCESSING LAYER
Although Druid can ingest events that are streamed in
one at a time, data must be denormalized beforehand as
Druid cannot yet support join queries. Furthermore, real
world data must often be transformed before it is usable by
an application.
5Figure 6: Ad impressions and clicks are recorded
in two separate streams. An event we want to join
is located in two diﬀerent Kafka partitions on two
diﬀerent topics.
4.1 Stream Processing
Stream processors provide infrastructure to develop pro-
cessing logic for unbounded sequences of messages. We use
Apache Samza as our stream processor, although other tech-
nologiesareviablealternatives(weinitiallychoseStorm, but
have since switched to Samza). Samza provides an API to
write jobs that run over a sequence of tuples and perform
operations over those tuples in a user-deﬁned way. The in-
put to each job is provided by Kafka, which can also act as
a sink for the output of the job. Samza jobs are executed
in a resource management and task execution framework
such as YARN[21]. It is beyond the scope of this paper to
go into the full details of Kafka/YARN/Samza interactions,
but more information is available in other literature[1]. We
will instead focus on how we leverage this framework for
processing data for analytic use cases.
On top of Samza infrastructure, we introduce the idea of
a “pipeline”, which is a grouping for a series of related pro-
cessing stages where “upstream” stages produce messages
that are consumed by “downstream” stages. Some of our
jobs involve operations such as renaming data, inserting de-
fault values for nulls and empty strings, and ﬁltering data.
One pipeline may write to many data sources in Druid.
To understand a real-world pipeline, let’s consider an ex-
ample from online advertising. In online advertising, events
are generated by impressions (views) of an ad and clicks
of an ad. Many advertisers are interested in knowing how
many impressions of an ad converted into clicks. Impression
streams and click streams are almost always generated as
separate streams by ad servers. Recall that Druid does not
support joins at query time, so the events must be generated
at processing time. An example of data generated by these
two event streams is shown in Figure 6. Every event has a
unique impression id that identiﬁes the ad served. We use
this id as our join key.
Theﬁrst stage of the Samza processing pipeline is a shuﬄe
step. Events are written to a keyed Kafka topic based on
the hash of an event’s impression id. This ensures that the
events that need to be joined are written to the same Kafka
topic. YARNcontainersrunningSamzatasksmayreadfrom
one or more Kafka topics, so it is important Samza task for
joins actually has both events that need to be joined. This
is shuﬄe stage is shown in Figure 7.
The next stage in the data pipeline is to actually join
the impression and click events. This is done by another
Samza task that creates a new event in the data with a new
ﬁeld called ”is_clicked”. This ﬁeld is marked as ”true” if an
impression event and a click event with the same impression
Figure 7: A shuﬄe operation ensures events to be
joined at stored in the same Kafka partition.
Figure 8: The join operation adds a new ﬁeld,
”is_clicked”.
id are both present. The original events are discarded, and
the new event is send further downstream. This join stage
shown in Figure 8
The ﬁnal stage of our data processing is to enhance the
data. This stage cleans up faults in data, and performs
lookups and transforms of events. Once data is cleaned,
it is ready to be delivered to Druid for queries. The total
streaming data processing pipeline is shown in Figure 9.
The system we have designed is not perfect. Because
we are doing windowed joins and because events cannot be
buﬀered indeﬁnitely, not all joins are guaranteed to com-
plete. If events are substantially delayed and do not arrive
in the allocated window period, they will not be joined. In
practice, thisgenerallyleadstoone“primary”eventcontinu-
ingthroughthepipelineandothersecondaryeventswiththe
same join key getting dropped. This means that our stream
processing layer is not guaranteed to deliver 100% accurate
results. Furthermore, even without this restriction, Samza
does not oﬀer exactly-once processing semantics. Problems
in network connectivity or node failure can lead to dupli-
cated events. For these reasons, we run a separate batch
pipeline that generates a more accurate transformation of
the ingested data.
Figure 9: The streaming processing data pipeline.
6The ﬁnal job of our processing pipeline is to deliver data
to Druid. For high availability, processed events from Samza
are transmitted concurrently to two real-time nodes. Both
nodes receive the same copy of data, and eﬀectively act as
replicas of each other. The Druid broker can query for either
copyofthedata. Whenhandoﬀoccurs, bothreal-timenodes
race to hand oﬀ the segments they’ve created. The segment
that is pushed into deep storage ﬁrst will be the one that is
used for historical querying, and once that segment is loaded
on the historical nodes, both real-time nodes will drop their
versions of the same segment.
4.2 Batch Processing
Ourbatchprocessingpipelineiscomposedofamulti-stage
MapReduce[4] pipeline. The ﬁrst set of jobs mirrors our
stream processing pipeline in that it transforms data and
joins relevant events in preparation for the data to be loaded
into Druid. The second set of jobs is responsible for directly
creating immutable Druid segments. The indexing code for
both streaming and batch ingestion in Druid is shared be-
tween the two modes of ingestion. These segments are then
uploaded to deep storage and registered with the metadata
store. Druid will proceed to load the batch generated seg-
ments.
Thebatchprocesstypicallyrunsmuchlessfrequentlythan
the real-time process, and may run many hours or even days
after raw events have been delivered. The wait is necessary
for severely delayed events, and to ensure that the raw data
is indeed complete.
Segments generated by the batch process are versioned by
the start time of the process. Hence, segments created by
batch processing will have a version identiﬁer that is greater
than segments created by real-time processing. When these
batch created segments are loaded in the cluster, they atom-
ically replace segments created by real-time processing for
their processed interval. Hence, soon after batch processing
completes, Druid queries begin reﬂecting batch-originated
data rather than real-time-originated data.
WeusethestreamingdatapipelinedescribedinSection4.1
to deliver immediate insights on events as they are occur-
ring, and the batch data pipeline described in this section to
provideanaccuratecopyofthedata. Thebatchprocesstyp-
ically runs much less frequently than the real-time process,
and may run many hours or even days after raw events have
been delivered. The wait is necessary for severely delayed
events, and to ensure that the raw data is indeed complete.
5. THE DELIVERY LAYER
In our stack, events are delivered over HTTP to Kafka.
Events are transmitted via POST requests to a receiver that
acts as a front for a Kafka producer. Kafka is a distributed
messaging system with a publish and subscribe model. At
a high level, Kafka maintains events or messages in cate-
gories called topics. A distributed Kafka cluster consists of
numerous brokers, which store messages in a replicated com-
mit log. Kafka consumers subscribe to topics and process
feeds of published messages.
Kafka provides functionality isolation between producers
ofdataandconsumersofdata. Thepublish/subscribemodel
works well for our use case as multiple consumers can sub-
scribe to the same topic and process the same set of events.
We have two primary Kafka consumers. The ﬁrst is a Samza
job that reads messages from Kafka for stream processing asData Source Dimensions Metrics
a 25 21
b 30 26
c 71 35
d 60 19
e 29 8
f 30 16
g 26 18
h 78 14
Table 2: Characteristics of production data sources.
described in Section 4.1. Topics in Kafka map to pipelines
in Samza, and pipelines in Samza map to data sources in
Druid. The second consumer reads messages from Kafka
and stores them in a distributed ﬁle system. This ﬁle sys-
temisthesameastheoneusedforDruid’sdeepstorage, and
also acts as a backup for raw events. The purpose of storing
raw events in deep storage is so that we can run batch pro-
cessing jobs over them at any given time. For example, our
stream processing layer may choose to not include certain
columns when it ﬁrst processes a new pipeline. If we want
to include these columns in the future, we can reprocess the
raw data to generate new Druid segments.
Kafka is the single point of delivery for events entering our
system, and must have the highest availability. We repli-
cate our Kafka producers across multiple datacenters. In
the event that Kafka brokers and consumers become unre-
sponsive, as long as our HTTP endpoints are still available,
we can buﬀer events on the producer side while recovering
the system. Similarily, if our processing and serving lay-
ers completely fail, we can recover by replaying events from
Kafka.
6. PERFORMANCE
Druid runs in production at several organizations, and to
brieﬂydemonstrateitsperformance, wehavechosentoshare
some real world numbers for one of the larger production
clusters. We also include results from synthetic workloads
on TPC-H data.
6.1 Query Performance in Production
Druid query performance can vary signﬁcantly depending
on the query being issued. For example, sorting the values
of a high cardinality dimension based on a given metric is
much more expensive than a simple count over a time range.
To showcase the average query latencies in a production
Druid cluster, we selected 8 frequently queried data sources,
described in Table 2.
Thequeriesvaryfromstandardaggregatesinvolvingdiﬀer-
ent types of metrics and ﬁlters, ordered group bys over one
or more dimensions with aggregates, and search queries and
metadata retrieval queries. Queries involving a single col-
umn are very frequent, and queries involving all columns are
very rare.
There were approximately 50 total data sources in this
particularclusterandseveralhundredusersissuingqueries.
Therewasapproximately10.5TBofRAMavailableinthis
cluster and approximately 10TB of segments loaded. Col-
lectively, there are about 50 billion Druid rows in this
cluster. Results for every data source is not shown.
70.00.51.0
Feb 03 Feb 10 Feb 17 Feb 24
timequery time (s)datasource
a
b
c
d
e
f
g
hMean query latency
0.00.51.01.5
01234
0510152090%ile 95%ile 99%ile
Feb 03 Feb 10 Feb 17 Feb 24
timequery time (seconds)datasource
a
b
c
d
e
f
g
hQuery latency percentilesFigure 10: Query latencies of production data
sources.
This cluster uses Intel®Xeon®E5-2670 processors and
consists of 1302 processing threads and 672 total cores
(hyperthreaded).
A memory-mapped storage engine was used (the machine
wasconﬁguredtomemorymapthedatainsteadofloading
it into the Java heap.)
Query latencies are shown in Figure 10. Across all the
various data sources, average query latency is approximately
550 milliseconds, with 90% of queries returning in less than
1 second, 95% in under 2 seconds, and 99% of queries re-
turning in less than 10 seconds.
6.2 Query Benchmarks on TPC-H Data
We also present Druid benchmarks on TPC-H data. We
selected queries more typical of Druid’s workload to demon-
strate query performance. In Figure 11, we present our
results compared again Google BigQuery, which is Google
Dremel[13]. The Druid results were ran on a 24 vCPU, 156
GB Google Compute Engine instance (2.2 GHz Intel Xeon
E5v4Broadwell)andtheBigQueryresultswererunthrough
Google’swebinterface. Inourresults, Druidperformedfrom
2-20x faster than BigQuery.
Our Druid setup used Amazon EC2 m3.2xlarge instance
types (Intel®Xeon®E5-2680 v2 @ 2.80GHz) for historical
nodes and c3.2xlarge instances (Intel®Xeon®E5-2670 v2
@ 2.50GHz) for broker nodes.
WebenchmarkedDruid’sscanrateat53,539,211rows/sec-
ond/core for select count(*) equivalent query over a given
Figure 11: Druid benchmarked against Google Big-
Query – 100GB TPC-H data.
Data Source Dimensions Metrics Peak Events/s
d1 34 24 218123
d2 36 24 172902
d3 46 21 170264
d4 40 17 94064
d5 41 23 68104
d6 31 31 64222
d7 29 8 30048
Table 3: Characteristics of production data sources.
time interval and 36,246,530 rows/second/core for a select
sum(float) type query.
6.3 Data Ingestion Performance
To showcase the ingestion latency of the RADStack, we
selected the top seven production datasources in terms of
peak events ingested per second for early 2015. These data-
sources are described in Table 3. Our production ingestion
setup used over 40 nodes, each with 60GB of RAM and 32
cores (12 x Intel®Xeon®E5-2670). Each pipeline for each
datasource involved transforms and joins.
Ingestion latency is heavily dependent on the complexity
of the data set being ingested. The data complexity is deter-
mined by the number of dimensions in each event, the num-
ber of metrics in each event, and the types of aggregations
we want to perform on those metrics. With the most basic
data set (one that only has a timestamp column), our setup
can ingest data at a rate of 800,000 events/second/core,
which is really just a measurement of how fast we can de-
serialize events. At peak, a single node was able to process
62259 events/second. The total peak events per second was
840500. The median events per second was 590100. The
ﬁrst and third quantiles were 487000 events/s and 663200
events/s respectively.
7. PRODUCTION EXPERIENCES
7.1 Experiences with Druid
7.1.1 Query Patterns
Druid is often used for exploratory analytics and report-
ing, which are two very distinct use cases. Exploratory ana-
lytic workﬂows provide insights by progressively narrowing
8down a view of data until an interesting observation is made.
Users tend to explore short intervals of recent data. In the
reporting use case, users query for much longer data inter-
vals, and the volume of queries is generally much less. The
insights that users are looking for are often pre-determined.
7.1.2 Multitenancy
Expensiveconcurrentqueriescanbeproblematicinamul-
titenant environment. Queries for large data sources may
end up hitting every historical node in a cluster and con-
sume all cluster resources. Smaller, cheaper queries may be
blocked from executing in such cases. We introduced query
prioritization to address these issues. Each historical node
is able to prioritize which segments it needs to scan. Proper
query planning is critical for production workloads. Thank-
fully, queries for a signiﬁcant amount of data tend to be for
reporting use cases and can be de-prioritized. Users do not
expect the same level of interactivity in this use case as they
do when they are exploring data.
7.1.3 Node Failures
Single node failures are common in distributed environ-
ments, but many nodes failing at once are not. If historical
nodes completely fail and do not recover, their segments
need to be reassigned, which means we need excess cluster
capacity to load this data. The amount of additional capac-
ity to have at any time contributes to the cost of running
a cluster. From our experiences, it is extremely rare to see
more than 2 nodes completely fail at once and hence, we
leave enough capacity in our cluster to completely reassign
the data from 2 historical nodes.
7.2 Experiences with Ingestion
7.2.1 Multitenancy
Before moving our streaming pipeline to Samza, we exper-
imented with other stream processors. One of the biggest
pains we felt was around multi-tenancy. Multiple pipelines
may contend for resources, and it is often unclear how vari-
ous jobs impact one another when running in the same en-
vironment. Given that each of our pipelines is composed
of diﬀerent tasks, Samza was able to provide per task re-
source isolation, which was far easier to manage than per
application resource isolation.
7.3 Operational Monitoring
Proper monitoring is critical to run a large scale dis-
tributed cluster, especially with many diﬀerent technologies.
Each Druid node is designed to periodically emit a set of op-
erational metrics. These metrics may include system level
data such as CPU usage, available memory, and disk capac-
ity, JVM statistics such as garbage collection time, and heap
usage, or node speciﬁc metrics such as segment scan time,
cache hit rates, and data ingestion latencies. Druid also
emits per query metrics so we can examine why a particular
query may be slow. We’ve also added functionality to peri-
odically emit metrics from Samza, Kafka, and Hadoop. We
emit metrics from our production RADStack and load them
into a dedicated metrics RADstack. The metrics cluster is
used to explore the performance and stability of the pro-
duction cluster. This dedicated metrics cluster has allowed
us to ﬁnd numerous production problems, such as gradualquery speed degradations, less than optimally tuned hard-
ware, and various other system bottlenecks.
8. RELATED WORK
8.1 Hybrid Batch/Streaming Workﬂows
Spark[24] is a cluster computing framework optimized for
iterative workﬂows. Spark Streaming is a separate project
that converts sequences of tuples into immutable micro-
batches. Each micro-batch can be processed using the un-
derlying Spark framework. Spark SQL is a query optimiza-
tion layer that can sit on top of Spark and issue SQL queries,
along with Spark’s native API. Druid’s approach to query-
ing is quite diﬀerent and Druid insteads builds immutable
indexed data structures optimized for low latency OLAP
queries, and does not leverage lineage in its architecture.
The RADStack can theoretically be composed of Spark and
Spark Streaming for processing, Kafka for event delivery,
and Druid to serve queries.
8.2 Druid and Other Data Stores
Druid builds on many of the same principles as other dis-
tributed columnar data stores[7], and in-memory databasesi
such as SAP’s HANA[6] and VoltDB[22]. These data stores
lackDruid’slowlatencyingestioncharacteristics. Druidalso
has native analytical features baked in, similar to ParAc-
cel[15], however, Druid allows system wide rolling software
updates with no downtime.
Druid is similar to C-Store[18] in that it has two subsys-
tems, a read-optimized subsystem in the historical nodes
and a write-optimized subsystem in real-time nodes. Real-
time nodes are designed to ingest a high volume of ap-
pend heavy data, and do not support data updates. Unlike
the two aforementioned systems, Druid is meant for OLAP
transactions and not OLTP transactions.
9. CONCLUSIONS AND FUTURE WORK
In this paper we presented the RADStack, a collection
of complementary technologies that can be used together
to power interactive analytic applications. The key pieces
of the stack are Kafka, Samza, Hadoop, and Druid. Druid
is designed for exploratory analytics and is optimized for
low latency data exploration, aggregation, and ingestion,
and is well suited for OLAP workﬂows. Samza and Hadoop
complement Druid and add data processing functionality,
and Kafka enables high throughput event delivery problem.
We believe that in later iterations of this work, batch pro-
cessing may not be necessary. As open source technologies
mature, the existing problems around exactly-once process-
ing will eventually be solved. The Druid, Samza and Kafka
communities are working on exactly once, lossless processing
for their respective systems, and in the near future, the same
guarantees that the RADStack provides right now should be
available using only these technologies.
10. ACKNOWLEDGMENTS
Druid, Samza, Kafka, and Hadoop could not have been
built the assistance of their respective communities. We
want to thank everyone that contributes to open source for
their invaluable support.
911. REFERENCES
[1] Apache samza. http://samza.apache.org/ , 2014.
[2] D. J. Abadi, S. R. Madden, and N. Hachem.
Column-stores vs. row-stores: How diﬀerent are they
really? In Proceedings of the 2008 ACM SIGMOD
international conference on Management of data ,
pages 967–980. ACM, 2008.
[3] Y. Collet. Lz4: Extremely fast compression algorithm.
code. google. com , 2013.
[4] J. Dean and S. Ghemawat. Mapreduce: simpliﬁed
data processing on large clusters. Communications of
the ACM , 51(1):107–113, 2008.
[5] G. DeCandia, D. Hastorun, M. Jampani,
G. Kakulapati, A. Lakshman, A. Pilchin,
S. Sivasubramanian, P. Vosshall, and W. Vogels.
Dynamo: amazon’s highly available key-value store. In
ACM SIGOPS Operating Systems Review , volume 41,
pages 205–220. ACM, 2007.
[6] F. Färber, S. K. Cha, J. Primsch, C. Bornhövd,
S. Sigg, and W. Lehner. Sap hana database: data
management for modern business applications. ACM
Sigmod Record , 40(4):45–51, 2012.
[7] B. Fink. Distributed computation on dynamo-style
distributed storage: riak pipe. In Proceedings of the
eleventh ACM SIGPLAN workshop on Erlang
workshop , pages 43–50. ACM, 2012.
[8] A. Hall, O. Bachmann, R. Büssow, S. Gănceanu, and
M. Nunkesser. Processing a trillion cells per mouse
click.Proceedings of the VLDB Endowment ,
5(11):1436–1446, 2012.
[9] M. Hausenblas and N. Bijnens. Lambda architecture.
URL: http://lambda-architecture. net/. Luettu , 6:2015,
2014.
[10] P. Hunt, M. Konar, F. P. Junqueira, and B. Reed.
Zookeeper: Wait-free coordination for internet-scale
systems. In USENIX ATC , volume 10, 2010.
[11] J. Kreps, N. Narkhede, and J. Rao. Kafka: A
distributed messaging system for log processing. In
Proceedings of 6th International Workshop on
Networking Meets Databases (NetDB), Athens, Greece ,
2011.
[12] N. Marz. Storm: Distributed and fault-tolerant
realtime computation. http://storm-project.net/ ,
February 2013.
[13] S. Melnik, A. Gubarev, J. J. Long, G. Romer,
S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel:
interactive analysis of web-scale datasets. Proceedings
of the VLDB Endowment , 3(1-2):330–339, 2010.
[14] P. O’Neil, E. Cheng, D. Gawlick, and E. O’Neil. The
log-structured merge-tree (lsm-tree). Acta
Informatica , 33(4):351–385, 1996.[15] Paraccel analytic database.
http://www.paraccel.com/resources/Datasheets/
ParAccel-Core-Analytic-Database.pdf , March
2013.
[16] K. Shvachko, H. Kuang, S. Radia, and R. Chansler.
The hadoop distributed ﬁle system. In Mass Storage
Systems and Technologies (MSST), 2010 IEEE 26th
Symposium on , pages 1–10. IEEE, 2010.
[17] M. Stonebraker, D. Abadi, D. J. DeWitt, S. Madden,
E. Paulson, A. Pavlo, and A. Rasin. Mapreduce and
parallel dbmss: friends or foes? Communications of
the ACM , 53(1):64–71, 2010.
[18] M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen,
M. Cherniack, M. Ferreira, E. Lau, A. Lin,
S. Madden, E. O’Neil, et al. C-store: a
column-oriented dbms. In Proceedings of the 31st
international conference on Very large data bases ,
pages 553–564. VLDB Endowment, 2005.
[19] M. Stonebraker, J. Becla, D. J. DeWitt, K.-T. Lim,
D. Maier, O. Ratzesberger, and S. B. Zdonik.
Requirements for science data bases and scidb. In
CIDR, volume 7, pages 173–184, 2009.
[20] E. Tschetter. Introducing druid: Real-time analytics
at a billion rows per second. http://druid.io/blog/
2011/04/30/introducing-druid.html , April 2011.
[21] V. K. Vavilapalli, A. C. Murthy, C. Douglas,
S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe,
H. Shah, S. Seth, et al. Apache hadoop yarn: Yet
another resource negotiator. In Proceedings of the 4th
annual Symposium on Cloud Computing , page 5.
ACM, 2013.
[22] L. VoltDB. Voltdb technical overview.
https://voltdb.com/ , 2010.
[23] F. Yang, E. Tschetter, X. Léauté, N. Ray, G. Merlino,
and D. Ganguli. Druid: a real-time analytical data
store. In Proceedings of the 2014 ACM SIGMOD
international conference on Management of data ,
pages 157–168. ACM, 2014.
[24] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma,
M. McCauley, M. J. Franklin, S. Shenker, and
I. Stoica. Resilient distributed datasets: A
fault-tolerant abstraction for in-memory cluster
computing. In Proceedings of the 9th USENIX
conference on Networked Systems Design and
Implementation , pages 2–2. USENIX Association,
2012.
[25] M. Zaharia, T. Das, H. Li, S. Shenker, and I. Stoica.
Discretized streams: an eﬃcient and fault-tolerant
model for stream processing on large clusters. In
Proceedings of the 4th USENIX conference on Hot
Topics in Cloud Computing , pages 10–10. USENIX
Association, 2012.
10