Project: Druid (App Definition and Development / Database)
File: druid.pdf

Druid
A Real-time Analytical Data Store
Fangjin Y ang
Metamarkets Group, Inc.
fangjin@metamarkets.comEric Tschetter
echeddar@gmail.comXavier Léauté
Metamarkets Group, Inc.
xavier@metamarkets.com
Nelson Ray
ncray86@gmail.comGian Merlino
Metamarkets Group, Inc.
gian@metamarkets.comDeep Ganguli
Metamarkets Group, Inc.
deep@metamarkets.com
ABSTRACT
Druidisanopensource1datastoredesignedforreal-timeexploratory
analyticsonlargedatasets. Thesystemcombinesacolumn-oriented
storage layout, a distributed, shared-nothing architecture, and an
advanced indexing structure to allow for the arbitrary exploration
of billion-row tables with sub-second latencies. In this paper, we
describeDruid’sarchitecture,anddetailhowitsupportsfastaggre-
gations, ﬂexible ﬁlters, and low latency data ingestion.
Categories and Subject Descriptors
H.2.4[DatabaseManagement ]: Systems— Distributeddatabases
Keywords
distributed;real-time;fault-tolerant;highlyavailable;opensource;
analytics; column-oriented; OLAP
1. INTRODUCTION
In recent years, the proliferation of internet technology has cre-
atedasurgeinmachine-generatedevents. Individually,theseevents
containminimalusefulinformationandareoflowvalue. Giventhe
time and resources required to extract meaning from large collec-
tionsofevents,manycompanieswerewillingtodiscardthisdatain-
stead. Althoughinfrastructurehasbeenbuilttohandleevent-based
data (e.g. IBM’s Netezza[37], HP’s Vertica[5], and EMC’s Green-
plum[29]), they are largely sold at high price points and are only
targetedtowards those companies who can affordthe offering.
A few years ago, Google introduced MapReduce [11] as their
mechanism of leveraging commodity hardware to index the inter-
net and analyze logs. The Hadoop [36] project soon followed and
waslargelypatternedaftertheinsightsthatcameoutoftheoriginal
MapReduce paper. Hadoop is currently deployed in many orga-
nizations to store and analyze large amounts of log data. Hadoop
hascontributedmuchtohelpingcompaniesconverttheirlow-value
1http://druid.io/ https://github.com/metamx/druid
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior speciﬁc permission
and/or a fee. Request permissions from permissions@acm.org.
SIGMOD’14, June 22–27, 2014, Snowbird, UT, USA.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-2376-5/14/06 ...$15.00.
http://dx.doi.org/10.1145/2588555.2595631.event streams into high-value aggregates for a variety of applica-
tions such as business intelligence and A-B testing.
As with many great systems, Hadoop has opened our eyes to
a new space of problems. Speciﬁcally, Hadoop excels at storing
andprovidingaccesstolargeamountsofdata,however,itdoesnot
makeanyperformanceguaranteesaroundhowquicklythatdatacan
be accessed. Furthermore, although Hadoop is a highly available
system,performancedegradesunderheavyconcurrentload. Lastly,
whileHadoopworkswellforstoringdata,itisnotoptimizedforin-
gesting data and making that data immediately readable.
EarlyoninthedevelopmentoftheMetamarketsproduct,weran
intoeachoftheseissuesandcametotherealizationthatHadoopis
a great back-ofﬁce, batch processing, and data warehousing sys-
tem. However, as a company that has product-level guarantees
aroundqueryperformanceanddataavailabilityinahighlyconcur-
rent environment (1000+ users), Hadoop wasn’t going to meet our
needs. Weexploreddifferentsolutionsinthespace,andaftertrying
bothRelationalDatabaseManagementSystemsandNoSQLarchi-
tectures, we came to the conclusion that there was nothing in the
open source world that could be fully leveraged for our require-
ments. We ended up creating Druid, an open source, distributed,
column-oriented, real-time analytical data store. In many ways,
DruidsharessimilaritieswithotherOLAPsystems[30,35,22],in-
teractivequerysystems[28],main-memorydatabases[14],aswell
aswidelyknowndistributeddatastores[7,12,23]. Thedistribution
and query model also borrow ideas from current generation search
infrastructure [25, 3, 4].
ThispaperdescribesthearchitectureofDruid,exploresthevari-
ousdesigndecisionsmadeincreatinganalways-onproductionsys-
tem that powers a hosted service, and attempts to help inform any-
onewhofacesasimilarproblemaboutapotentialmethodofsolving
it. Druid is deployed in production at several technology compa-
nies2. The structure of the paper is as follows: we ﬁrst describe
theprobleminSection2. Next,wedetailsystemarchitecturefrom
the point of view of how data ﬂows through the system in Section
3. We then discuss how and why data gets converted into a binary
formatinSection4. WebrieﬂydescribethequeryAPIinSection5
and present performance results in Section 6. Lastly, we leave off
withourlessonsfromrunningDruidinproductioninSection7,and
related work in Section 8.
2. PROBLEM DEFINITION
Druid was originally designed to solve problems around ingest-
ingandexploringlargequantitiesoftransactionalevents(logdata).
This form of timeseries data is commonly found in OLAP work-
2http://druid.io/druid.htmlTimestamp Page Username Gender City Characters Added Characters Removed
2011-01-01T01:00:00Z Justin Bieber Boxer Male San Francisco 1800 25
2011-01-01T01:00:00Z Justin Bieber Reach Male Waterloo 2912 42
2011-01-01T02:00:00Z Ke$ha Helz Male Calgary 1953 17
2011-01-01T02:00:00Z Ke$ha Xeno Male Taiyuan 3194 170
Table1: Sample Druid data for edits that have occurredon Wikipedia.
ﬂowsandthenatureofthedatatendstobeveryappendheavy. For
example,considerthedatashowninTable1. Table1containsdata
for edits that have occurred on Wikipedia. Each time a user edits
a page in Wikipedia, an event is generated that contains metadata
about the edit. This metadata is comprised of 3 distinct compo-
nents. First, there is a timestamp column indicating when the edit
wasmade. Next,thereareasetdimensioncolumnsindicatingvar-
ious attributes about the edit such as the page that was edited, the
user who made the edit, and the location of the user. Finally, there
are a set of metric columns that contain values (usually numeric)
that can be aggregated, such as the number of characters added or
removed in an edit.
Our goal is to rapidly compute drill-downs and aggregates over
thisdata. Wewanttoanswerquestionslike“Howmanyeditswere
madeonthepageJustinBieberfrommalesinSanFrancisco?”and
“Whatistheaveragenumberofcharactersthatwereaddedbypeo-
plefromCalgaryoverthespanofamonth?”. Wealsowantqueries
over any arbitrary combination of dimensions to return with sub-
second latencies.
The need for Druid was facilitated by the fact that existing open
source Relational Database Management Systems (RDBMS) and
NoSQLkey/valuestoreswereunabletoprovidealowlatencydata
ingestionandqueryplatformforinteractiveapplications[40]. Inthe
early days of Metamarkets, we were focused on building a hosted
dashboardthatwouldallowuserstoarbitrarilyexploreandvisualize
event streams. The data store powering the dashboard needed to
return queries fast enough that the data visualizations built on top
of it could provide users with an interactive experience.
Inadditiontothequerylatencyneeds,thesystemhadtobemulti-
tenant and highly available. The Metamarkets product is used in a
highlyconcurrentenvironment. Downtimeiscostlyandmanybusi-
nessescannotaffordtowaitifasystemisunavailableinthefaceof
software upgrades or network failure. Downtime for startups, who
often lack proper internal operations management, can determine
business success or failure.
Finally, another challenge that Metamarkets faced in its early
days was to allow users and alerting systems to be able to make
business decisions in “real-time”. The time from when an event is
created to when that event is queryable determines how fast inter-
estedpartiesareabletoreacttopotentiallycatastrophicsituationsin
theirsystems. Popularopensourcedatawarehousingsystemssuch
as Hadoop were unable to provide the sub-second data ingestion
latencies we required.
Theproblemsofdataexploration,ingestion,andavailabilityspan
multipleindustries. SinceDruidwasopensourcedinOctober2012,
it been deployed as a video, network monitoring, operations mon-
itoring, and online advertising analytics platform at multiple com-
panies.
3. ARCHITECTURE
ADruidclusterconsistsofdifferenttypesofnodesandeachnode
type is designed to perform a speciﬁc set of things. We believe
this design separates concerns and simpliﬁes the complexity of the
overallsystem. Thedifferentnodetypesoperatefairlyindependentofeachotherandthereisminimalinteractionamongthem. Hence,
intra-cluster communication failures have minimal impact on data
availability.
Tosolvecomplexdataanalysisproblems,thedifferentnodetypes
come together to form a fully working system. The name Druid
comes from the Druid class in many role-playing games: it is a
shape-shifter, capable of taking on many different forms to fulﬁll
various different roles in a group. The composition of and ﬂow of
data in a Druid cluster are shown in Figure 1.
3.1 Real-time Nodes
Real-timenodesencapsulatethefunctionalitytoingestandquery
event streams. Events indexed via these nodes are immediately
available for querying. The nodes are only concerned with events
for some small time range and periodically hand off immutable
batches of events they have collected over this small time range to
othernodesintheDruidclusterthatarespecializedindealingwith
batchesofimmutableevents. Real-timenodesleverageZooKeeper
[19] for coordination with the rest of the Druid cluster. The nodes
announce their online state and the data they serve in ZooKeeper.
Real-time nodes maintain an in-memory index buffer for all in-
comingevents. Theseindexesareincrementallypopulatedasevents
are ingested and the indexes are also directly queryable. Druid be-
haves as a row store for queries on events that exist in this JVM
heap-based buffer. To avoid heap overﬂow problems, real-time
nodes persist their in-memory indexes to disk either periodically
or after some maximum row limit is reached. This persist process
converts data stored in the in-memory buffer to a column oriented
storage format described in Section 4. Each persisted index is im-
mutable and real-time nodes load persisted indexes into off-heap
memory such that they can still be queried. This process is de-
scribed in detail in [33] and is illustrated in Figure 2.
On a periodic basis, each real-time node will schedule a back-
groundtaskthatsearchesforalllocallypersistedindexes. Thetask
merges these indexes together and builds an immutable block of
data that contains all the events that have been ingested by a real-
time node for some span of time. We refer to this block of data as
a “segment”. During the handoff stage, a real-time node uploads
thissegmenttoapermanentbackupstorage,typicallyadistributed
ﬁle system such as S3 [12] or HDFS [36], which Druid refers to as
“deep storage”. The ingest, persist, merge, and handoff steps are
ﬂuid; there is no data loss during any of the processes.
Figure 3 illustrates the operations of a real-time node. The node
startsat13:37andwillonlyaccepteventsforthecurrenthourorthe
next hour. When events are ingested, the node announces that it is
servingasegmentofdataforanintervalfrom13:00to14:00. Every
10 minutes (the persist period is conﬁgurable), the node will ﬂush
and persist its in-memory buffer to disk. Near the end of the hour,
thenodewilllikelyseeeventsfor14:00to15:00. Whenthisoccurs,
the node prepares to serve data for the next hour and creates a new
in-memory index. The node then announces that it is also serving
a segment from 14:00 to 15:00. The node does not immediately
merge persisted indexes from 13:00 to 14:00, instead it waits for
a conﬁgurable window period for straggling events from 13:00 toReal-time 
Nodes
Coordinator 
NodesBroker Nodes
Historical 
NodesMySQL
Zookeeper
Deep 
Storage
Streaming 
Data
Batch
DataClient 
Queries
Queries
Metadata
Data/SegmentsDruid Nodes
External DependenciesFigure1: An overviewof a Druid cluster andthe ﬂow of data throughthe cluster.
event_23312
event_23481
event_23593
...
event_1234
event_2345
...event_3456
event_4567
...
event_5678
event_6789
...event_7890
event_8901
...Disk and persisted indexesHeap and in-memory index
Persistevent_34982
event_35789
event_36791
...
event_1234
event_2345
...event_3456
event_4567
...
event_5678
event_6789
...event_7890
event_8901
...Off-heap memory and 
persisted indexes
LoadQueries
Figure2: Real-timenodesbuffereventstoanin-memoryindex,
which is regularly persisted to disk. On a periodic basis, per-
sisted indexes are then merged together before getting handed
off. Queries will hit both the in-memory and persisted indexes.
14:00toarrive. Thiswindowperiodminimizestheriskofdataloss
fromdelaysineventdelivery. Attheendofthewindowperiod,the
nodemergesallpersistedindexesfrom13:00to14:00intoasingle
immutable segment and hands the segment off. Once this segment
is loaded and queryable somewhere else in the Druid cluster, the
real-timenodeﬂushesallinformationaboutthedataitcollectedfor
13:00 to 14:00 and unannounces it is serving this data.
3.1.1 Availability and Scalability
Real-timenodesareaconsumerofdataandrequireacorrespond-
ingproducertoprovidethedatastream. Commonly,fordatadura-
bility purposes, a message bus such as Kafka [21] sits between the
producer and the real-time node as shown in Figure 4. Real-time
nodes ingest data by reading events from the message bus. The
time from event creation to event consumption is ordinarily on the
order of hundreds of milliseconds.
ThepurposeofthemessagebusinFigure4istwo-fold. First,the
message bus acts as a buffer for incoming events. A message bus
suchasKafkamaintainspositionaloffsetsindicatinghowfaracon-
sumer (a real-time node) has read in an event stream. Consumers
canprogrammaticallyupdatetheseoffsets. Real-timenodesupdatethisoffseteachtimetheypersisttheirin-memorybufferstodisk. In
afailandrecoverscenario,ifanodehasnotlostdisk,itcanreload
all persisted indexes from disk and continue reading events from
the last offset it committed. Ingesting events from a recently com-
mitted offset greatly reduces a node’s recovery time. In practice,
weseenodesrecoverfromsuchfailurescenariosinafewseconds.
The second purpose of the message bus is to act as a single end-
point from which multiple real-time nodes can read events. Multi-
ple real-time nodes can ingest the same set of events from the bus,
creating a replication of events. In a scenario where a node com-
pletely fails and loses disk, replicated streams ensure that no data
is lost. A single ingestion endpoint also allows for data streams
to be partitioned such that multiple real-time nodes each ingest a
portion of a stream. This allows additional real-time nodes to be
seamlessly added. In practice, this model has allowed one of the
largestproductionDruidclusterstobeabletoconsumerawdataat
approximately 500 MB/s (150,000 events/s or 2 TB/hour).
3.2 Historical Nodes
Historical nodes encapsulate the functionality to load and serve
theimmutableblocksofdata(segments)createdbyreal-timenodes.
In many real-world workﬂows, most of the data loaded in a Druid
cluster is immutable and hence, historical nodes are typically the
main workers of a Druid cluster. Historical nodes follow a shared-
nothingarchitectureandthereisnosinglepointofcontentionamong
the nodes. The nodes have no knowledge of one another and are
operationally simple; they only know how to load, drop, and serve
immutable segments.
Similartoreal-timenodes,historicalnodesannouncetheironline
state and the data they are serving in ZooKeeper. Instructions to
loadanddropsegmentsaresentoverZooKeeperandcontaininfor-
mationaboutwherethesegmentislocatedindeepstorageandhow
to decompress and process the segment. Before a historical node
downloadsaparticularsegmentfromdeepstorage,itﬁrstchecksa
localcachethatmaintainsinformationaboutwhatsegmentsalready
exist on the node. If information about a segment is not present in
thecache,thehistoricalnodewillproceedtodownloadthesegment
from deep storage. This process is shown in Figure 5. Once pro-
cessing is complete, the segment is announced in ZooKeeper. At
this point, the segment is queryable. The local cache also allows
forhistoricalnodestobequicklyupdatedandrestarted. Onstartup,
thenodeexaminesitscacheandimmediatelyserveswhateverdata
it ﬁnds.13:00 14:00 15:00
13:37
- node starts
- announce segment 
for data 13:00-14:0013:47
persist data for 13:00-14:00
~14:00
- announce segment 
for data 14:00-15:0014:10
- merge and handoff for data 13:00-14:00
- persist data for 14:00-15:00~14:11
- unannounce segment 
for data 13:00-14:00
13:57
persist data for 13:00-14:0014:07
persist data for 13:00-14:00Figure 3: The node starts, ingests data, persists, and periodically hands data off. This process repeats indeﬁnitely. The time periods
between differentreal-time node operations areconﬁgurable.
event_12345
event_23456
event_34567
event_35582
event_37193
event_78901
event_79902
event_79932
event_89012event_2849219
event_120202
…
event_90192
Real-time
Node 1
Real-time
Node 2
offset 1
offset 2
eventseventseventsKafka
Streaming events
Figure4: Multiplereal-timenodescanreadfromthesamemes-
sage bus. Each node maintains its own offset.
Deep Storage
SegmentMemory
Disk
Cache 
EntriesSegmentSegment
download
create keyLoad
Figure5: Historicalnodesdownloadimmutablesegmentsfrom
deep storage. Segments must be loaded in memory before they
can be queried.Historicalnodescansupportreadconsistencybecausetheyonly
dealwithimmutabledata. Immutabledatablocksalsoenableasim-
ple parallelization model: historical nodes can concurrently scan
and aggregate immutable blocks without blocking.
3.2.1 Tiers
Historicalnodescanbegroupedindifferenttiers,whereallnodes
inagiventierareidenticallyconﬁgured. Differentperformanceand
fault-tolerance parameters can be set for each tier. The purpose of
tierednodesistoenablehigherorlowerprioritysegmentstobedis-
tributed according to their importance. For example, it is possible
to spin up a “hot” tier of historical nodes that have a high num-
ber of cores and large memory capacity. The “hot” cluster can be
conﬁgured to download more frequently accessed data. A parallel
“cold”clustercanalsobecreatedwithmuchlesspowerfulbacking
hardware. The “cold” cluster would only contain less frequently
accessed segments.
3.2.2 Availability
HistoricalnodesdependonZooKeeperforsegmentloadandun-
load instructions. Should ZooKeeper become unavailable, histor-
ical nodes are no longer able to serve new data or drop outdated
data, however, because the queries are served over HTTP, histori-
calnodesarestillabletorespondtoqueryrequestsforthedatathey
are currently serving. This means that ZooKeeper outages do not
impact current data availability on historical nodes.
3.3 Broker Nodes
Brokernodesactasqueryrouterstohistoricalandreal-timenodes.
They understand the metadata published in ZooKeeper about what
segmentsarequeryableandwherethosesegmentsarelocated. Bro-
kernodesrouteincomingqueriessuchthatthequerieshittheright
historical or real-time nodes. Broker nodes also merge partial re-
sults from historical and real-time nodes before returning a ﬁnal
consolidated result to the caller.
3.3.1 Caching
Broker nodes contain a cache with a LRU [31, 20] invalidation
strategy. The cache can use local heap memory or an external dis-tributedkey/valuestoresuchasMemcached[16]. Eachtimeabro-
ker node receives a query, it ﬁrst maps the query to a set of seg-
ments. Results for certain segments may already exist in the cache
and there is no need to recompute them. For any results that do
notexistinthecache,thebrokernodewillforwardthequerytothe
correcthistoricalandreal-timenodes. Oncehistoricalnodesreturn
theirresults,thebrokerwillcachetheseresultsonapersegmentba-
sis for future use. This process is illustrated in Figure 6. Real-time
data is never cached and hence requests for real-time data will al-
waysbeforwardedtoreal-timenodes. Real-timedataisperpetually
changing and caching the results is unreliable.
The cache also acts as an additional level of data durability. In
the event that all historical nodes fail, it is still possible to query
results if those results already exist in the cache.
3.3.2 Availability
In the event of a total ZooKeeper outage, data is still queryable.
IfbrokernodesareunabletocommunicatetoZooKeeper,theyuse
theirlastknownviewoftheclusterandcontinuetoforwardqueries
to real-time and historical nodes. Broker nodes make the assump-
tion that the structure of the cluster is the same as it was before the
outage. In practice, this availability model has allowed our Druid
cluster to continue serving queries for a signiﬁcant period of time
while we diagnosed ZooKeeper outages.
3.4 Coordinator Nodes
Druidcoordinatornodesareprimarilyinchargeofdatamanage-
ment and distribution on historical nodes. The coordinator nodes
tell historical nodes to load new data, drop outdated data, replicate
data, and move data to load balance. Druid uses a multi-version
concurrency control swapping protocol for managing immutable
segments in order to maintain stable views. If any immutable seg-
mentcontainsdatathatiswhollyobsoletedbynewersegments,the
outdated segment is dropped from the cluster. Coordinator nodes
undergoaleader-electionprocessthatdeterminesasinglenodethat
runsthecoordinatorfunctionality. Theremainingcoordinatornodes
act as redundant backups.
A coordinator node runs periodically to determine the current
state of the cluster. It makes decisions by comparing the expected
state of the cluster with the actual state of the cluster at the time
of the run. As with all Druid nodes, coordinator nodes maintain
a ZooKeeper connection for current cluster information. Coordi-
nator nodes also maintain a connection to a MySQL database that
containsadditionaloperationalparametersandconﬁgurations. One
of the key pieces of information located in the MySQL database is
a table that contains a list of all segments that should be served by
historicalnodes. Thistablecanbeupdatedbyanyservicethatcre-
atessegments,forexample,real-timenodes. TheMySQLdatabase
also contains a rule table that governs how segments are created,
destroyed, and replicated in the cluster.
3.4.1 Rules
Rules govern how historical segments are loaded and dropped
fromthecluster. Rulesindicatehowsegmentsshouldbeassignedto
differenthistoricalnodetiersandhowmanyreplicatesofasegment
should exist in each tier. Rules may also indicate when segments
should be dropped entirely from the cluster. Rules are usually set
for a period of time. For example, a user may use rules to load the
mostrecentonemonth’sworthofsegmentsintoa“hot”cluster,the
mostrecentoneyear’sworthofsegmentsintoa“cold”cluster,and
drop any segments that are older.
Thecoordinatornodesloadasetofrulesfromaruletableinthe
MySQL database. Rules may be speciﬁc to a certain data sourceand/or a default set of rules may be conﬁgured. The coordinator
nodewillcyclethroughallavailablesegmentsandmatcheachseg-
ment with the ﬁrst rule that applies to it.
3.4.2 Load Balancing
In a typical production environment, queries often hit dozens or
even hundreds of segments. Since each historical node has limited
resources, segments must be distributed among the cluster to en-
sure that the cluster load is not too imbalanced. Determining opti-
malloaddistributionrequiressomeknowledgeaboutquerypatterns
andspeeds. Typically,queriescoverrecentsegmentsspanningcon-
tiguoustimeintervalsforasingledatasource. Onaverage,queries
that access smaller segments are faster.
These query patterns suggest replicating recent historical seg-
ments at a higher rate, spreading out large segments that are close
intimetodifferenthistoricalnodes,andco-locatingsegmentsfrom
different data sources. To optimally distribute and balance seg-
ments among the cluster, we developed a cost-based optimization
procedurethattakesintoaccountthesegmentdatasource,recency,
andsize. Theexactdetailsofthealgorithmarebeyondthescopeof
this paper and may be discussed in future literature.
3.4.3 Replication
Coordinator nodes may tell different historical nodes to load a
copy of the same segment. The number of replicates in each tier
of the historical compute cluster is fully conﬁgurable. Setups that
require high levels of fault tolerance can be conﬁgured to have a
high number of replicas. Replicated segments are treated the same
astheoriginalsandfollowthesameloaddistributionalgorithm. By
replicatingsegments,singlehistoricalnodefailuresaretransparent
in the Druid cluster. We use this property for software upgrades.
We can seamlessly take a historical node ofﬂine, update it, bring it
backup,andrepeattheprocessforeveryhistoricalnodeinacluster.
Overthelasttwoyears,wehavenevertakendowntimeinourDruid
cluster for software upgrades.
3.4.4 Availability
Druid coordinator nodes have ZooKeeper and MySQL as exter-
nal dependencies. Coordinator nodes rely on ZooKeeper to deter-
minewhathistoricalnodesalreadyexistinthecluster. IfZooKeeper
becomesunavailable,thecoordinatorwillnolongerbeabletosend
instructionstoassign,balance,anddropsegments. However,these
operations do not affectdata availability at all.
The design principle for responding to MySQL and ZooKeeper
failures is the same: if an external dependency responsible for co-
ordination fails, the cluster maintains the status quo. Druid uses
MySQLtostoreoperationalmanagementinformationandsegment
metadatainformationaboutwhatsegmentsshouldexistintheclus-
ter. IfMySQLgoesdown,thisinformationbecomesunavailableto
coordinator nodes. However, this does not mean data itself is un-
available. If coordinator nodes cannot communicate to MySQL,
they will cease to assign new segments and drop outdated ones.
Broker, historical, and real-time nodes are still queryable during
MySQLoutages.
4. STORAGE FORMAT
DatatablesinDruid(called datasources )arecollectionsoftimes-
tamped events and partitioned into a set of segments, where each
segmentistypically5–10millionrows. Formally,wedeﬁneaseg-
ment as a collection of rows of data that span some period of time.
SegmentsrepresentthefundamentalstorageunitinDruidandrepli-
cation and distribution are done at a segment level.Query for data 
from 2013-01-01 
to 2013-01-08results for segment 2013-01-01/2013-01-02
results for segment 2013-01-02/2013-01-03
results for segment 2013-01-07/2013-01-08Cache (on broker nodes)
segment for data 2013-01-03/2013-01-04
segment for data 2013-01-04/2013-01-05
segment for data 2013-01-05/2013-01-06
segment for data 2013-01-06/2013-01-07Historical and real-time nodes
Query for data 
not in cacheFigure6: Results arecached per segment. Queriescombine cached resultswith resultscomputed on historical and real-timenodes.
Druid always requires a timestamp column as a method of sim-
plifyingdatadistributionpolicies,dataretentionpolicies,andﬁrst-
level query pruning. Druid partitions its data sources into well-
deﬁned time intervals, typically an hour or a day, and may further
partition on values from other columns to achieve the desired seg-
ment size. The time granularity to partition segments is a function
of data volume and time range. A data set with timestamps spread
over a year is better partitioned by day, and a data set with times-
tamps spread over a day is better partitioned by hour.
Segments are uniquely identiﬁed by a data source identifer, the
time interval of the data, and a version string that increases when-
ever a new segment is created. The version string indicates the
freshnessofsegmentdata;segmentswithlaterversionshavenewer
viewsofdata(oversometimerange)thansegmentswitholderver-
sions. This segment metadata is used by the system for concur-
rency control; read operations always access data in a particular
time range from the segments with the latest version identiﬁers for
that time range.
Druid segments are stored in a column orientation. Given that
Druidisbestusedforaggregatingeventstreams(alldatagoinginto
Druidmusthaveatimestamp),theadvantagesofstoringaggregate
information as columns rather than rows are well documented [1].
Column storage allows for more efﬁcient CPU usage as only what
is needed is actually loaded and scanned. In a row oriented data
store,allcolumnsassociatedwitharowmustbescannedaspartof
an aggregation. The additional scan time can introduce signﬁcant
performance degradations [1].
Druid has multiple column types to represent various data for-
mats. Depending on the column type, different compression meth-
ods are used to reduce the cost of storing a column in memory and
on disk. In the example given in Table 1, the page, user, gender,
and city columns only contain strings. Storing strings directly is
unnecessarily costly and string columns can be dictionary encoded
instead. Dictionaryencodingisacommonmethodtocompressdata
and has been used in other data stores such as PowerDrill [17]. In
the example in Table 1, we can map each page to a unique integer
identiﬁer.
Justin Bieber -> 0
Ke$ha -> 1
This mapping allows us to represent the page column as an in-
teger array where the array indices correspond to the rows of the
originaldataset. Forthepagecolumn,wecanrepresenttheunique
pages as follows:
[0, 0, 1, 1]
The resulting integer array lends itself very well to compression
methods. Generic compression algorithms on top of encodings are
extremelycommonincolumn-stores. DruidusestheLZF[24]com-
pression algorithm.
Similarcompressionmethodscanbeappliedtonumericcolumns.
Forexample,thecharactersaddedandcharactersremovedcolumns
in Table1 can also be expressed as individual arrays.
Characters Added -> [1800, 2912, 1953, 3194]
Characters Removed -> [25, 42, 17, 170]
Inthiscase,wecompresstherawvaluesasopposedtotheirdic-
tionary representations.
Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)Integer array size (bytes)
1e+041e+06
1e+02 1e+05 1e+08
CardinalityConcise compressed size (bytes)sorted
sorted
unsortedFigure7: Integer array size versus Concise set size.
4.1 Indices for Filtering Data
In many real world OLAP workﬂows, queries are issued for the
aggregated results of some set of metrics where some set of di-
mension speciﬁcations are met. An example query is: “How many
Wikipedia edits were done by users in San Francisco who are also
male?”ThisqueryisﬁlteringtheWikipediadatasetinTable1based
on a Boolean expression of dimension values. In many real world
data sets, dimension columns contain strings and metric columns
containnumericvalues. Druidcreatesadditionallookupindicesfor
stringcolumnssuchthatonlythoserowsthatpertaintoaparticular
query ﬁlter are ever scanned.
LetusconsiderthepagecolumninTable1. Foreachuniquepage
in Table 1, we can form some representation indicating in which
table rows a particular page is seen. We can store this information
in a binary array where the array indices represent our rows. If a
particular page is seen in a certain row, that array index is marked
as1. For example:
Justin Bieber -> rows [0, 1] -> [1][1][0][0]
Ke$ha -> rows [2, 3] -> [0][0][1][1]
Justin Bieber is seen in rows 0and1. This mapping of col-
umn values to row indices forms an inverted index [39]. To know
whichrowscontain Justin Bieber orKe$ha,wecanORtogether
the two arrays.
[0][1][0][1] OR [1][0][1][0] = [1][1][1][1]
ThisapproachofperformingBooleanoperationsonlargebitmap
setsiscommonlyusedinsearchengines. BitmapindicesforOLAP
workloads is described in detail in [32]. Bitmap compression al-
gorithms are a well-deﬁned area of research [2, 44, 42] and often
utilize run-length encoding. Druid opted to use the Concise algo-
rithm [10]. Figure 7 illustrates the number of bytes using Concise
compression versus using an integer array. The results were gen-
erated on a cc2.8xlarge system with a single thread, 2G heap,
512m young gen, and a forced GC between each run. The data set
is a single day’s worth of data collected from the Twitter garden
hose [41] data stream. The data set contains 2,272,295 rows and12dimensionsofvaryingcardinality. Asanadditionalcomparison,
we also resorted the data set rows to maximize compression.
Intheunsortedcase,thetotalConcisesizewas53,451,144bytes
and the total integer array size was 127,248,520 bytes. Overall,
Concise compressed sets are about 42% smaller than integer ar-
rays. In the sorted case, the total Concise compressed size was
43,832,884 bytes and the total integer array size was 127,248,520
bytes. What is interesting to note is that after sorting, global com-
pression only increased minimally.
4.2 Storage Engine
Druid’s persistence components allows for different storage en-
gines to be plugged in, similar to Dynamo [12]. These storage en-
ginesmaystoredatainanentirelyin-memorystructuresuchasthe
JVM heap or in memory-mapped structures. The ability to swap
storage engines allows for Druid to be conﬁgured depending on a
particular application’s speciﬁcations. An in-memory storage en-
ginemaybeoperationallymoreexpensivethanamemory-mapped
storage engine but could be a better alternative if performance is
critical. By default, a memory-mapped storage engine is used.
When using a memory-mapped storage engine, Druid relies on
theoperatingsystemtopagesegmentsinandoutofmemory. Given
that segments can only be scanned if they are loaded in memory, a
memory-mapped storage engine allows recent segments to retain
inmemorywhereassegmentsthatareneverqueriedarepagedout.
Themaindrawbackwithusingthememory-mappedstorageengine
is when a query requires more segments to be paged into memory
thanagivennodehascapacityfor. Inthiscase,queryperformance
willsufferfromthecostofpagingsegmentsinandoutofmemory.
5. QUERY API
Druid has its own query language and accepts queries as POST
requests. Broker, historical, and real-time nodes all share the same
query API.
The body of the POST request is a JSON object containing key-
value pairs specifying various query parameters. A typical query
willcontainthedatasourcename,thegranularityoftheresultdata,
time range of interest, the type of request, and the metrics to ag-
gregate over. The result will also be a JSON object containing the
aggregated metrics over the time period.
Most query types will also support a ﬁlter set. A ﬁlter set is a
Booleanexpressionofdimensionnameandvaluepairs. Anynum-
ber and combination of dimensions and values may be speciﬁed.
When a ﬁlter set is provided, only the subset of the data that per-
tainstotheﬁltersetwillbescanned. Theabilitytohandlecomplex
nestedﬁltersetsiswhatenablesDruidtodrillintodataatanydepth.
Theexactquerysyntaxdependsonthequerytypeandtheinfor-
mation requested. A sample count query over a week of data is as
follows:
{
"queryType" : "timeseries",
"dataSource" : "wikipedia",
"intervals" : "2013-01-01/2013-01-08",
"filter" : {
"type" : "selector",
"dimension" : "page",
"value" : "Ke$ha"
},
"granularity" : "day",
"aggregations" : [{"type":"count", "name":"rows"}]
}
Thequeryshownabovewillreturnacountofthenumberofrows
in the Wikipedia data source from 2013-01-01 to 2013-01-08, ﬁl-
tered for only those rows where the value of the “page” dimension
is equal to “Ke$ha”. The results will be bucketed by day and will
be a JSON array of the following form:[ {
"timestamp": "2012-01-01T00:00:00.000Z",
"result": {"rows":393298}
},
{
"timestamp": "2012-01-02T00:00:00.000Z",
"result": {"rows":382932}
},
...
{
"timestamp": "2012-01-07T00:00:00.000Z",
"result": {"rows": 1337}
} ]
Druid supports many types of aggregations including sums on
ﬂoating-point and integer types, minimums, maximums, and com-
plex aggregations such as cardinality estimation and approximate
quantile estimation. The results of aggregations can be combined
in mathematical expressions to form other aggregations. It is be-
yond the scope of this paper to fully describe the query API but
more information can be found online3.
Asofthiswriting,ajoinqueryforDruidisnotyetimplemented.
Thishasbeenafunctionofengineeringresourceallocationanduse
case decisions more than a decision driven by technical merit. In-
deed, Druid’s storage format would allow for the implementation
of joins (there is no loss of ﬁdelity for columns included as dimen-
sions)andtheimplementationofthemhasbeenaconversationthat
we have every few months. To date, we have made the choice that
theimplementationcostisnotworththeinvestmentforourorgani-
zation. The reasons for this decision are generally two-fold.
1. Scalingjoinquerieshasbeen,inourprofessionalexperience,
a constant bottleneck of working with distributed databases.
2. The incremental gains in functionality are perceived to be
of less value than the anticipated problems with managing
highly concurrent, join-heavy workloads.
Ajoinqueryisessentiallythemergingoftwoormorestreamsof
data based on a shared set of keys. The primary high-level strate-
gies for join queries we are aware of are a hash-based strategy or a
sorted-mergestrategy. Thehash-basedstrategyrequiresthatallbut
one data set be available as something that looks like a hash table,
a lookup operation is then performed on this hash table for every
row in the “primary” stream. The sorted-merge strategy assumes
thateachstreamissortedbythejoinkeyandthusallowsforthein-
crementaljoiningofthestreams. Eachofthesestrategies,however,
requiresthematerializationofsomenumberofthestreamseitherin
sorted order or in a hash table form.
When all sides of the join are signiﬁcantly large tables (> 1 bil-
lion records), materializing the pre-join streams requires complex
distributed memory management. The complexity of the memory
managementisonlyampliﬁedbythefactthatwearetargetinghighly
concurrent, multitenant workloads. This is, as far as we are aware,
an active academic research problem that we would be willing to
help resolve in a scalable manner.
6. PERFORMANCE
Druidrunsinproductionatseveralorganizations,andtodemon-
strate its performance, we have chosen to share some real world
numbersforthemainproductionclusterrunningatMetamarketsas
ofearly2014. Forcomparisonwithotherdatabaseswealsoinclude
results from synthetic workloads on TPC-H data.
3http://druid.io/docs/latest/Querying.htmlDataSource Dimensions Metrics
a 25 21
b 30 26
c 71 35
d 60 19
e 29 8
f 30 16
g 26 18
h 78 14
Table2: Characteristics of productiondata sources.
6.1 Query Performance in Production
Druidqueryperformancecanvarysignﬁcantlydependingonthe
querybeingissued. Forexample,sortingthevaluesofahighcardi-
nality dimension based on a given metric is much more expensive
than a simple count over a time range. To showcase the average
query latencies in a production Druid cluster, we selected 8 of our
most queried data sources, described in Table2.
Approximately30%ofqueriesarestandardaggregatesinvolving
different types of metrics and ﬁlters, 60% of queries are ordered
group bys over one or more dimensions with aggregates, and 10%
of queries are search queries and metadata retrieval queries. The
number of columns scanned in aggregate queries roughly follows
an exponential distribution. Queries involving a single column are
very frequent, and queries involving all columns are very rare.
Afewnotes about our results:
Theresultsarefroma“hot”tierinourproductioncluster. There
were approximately 50 data sources in the tier and several hun-
dred users issuing queries.
Therewasapproximately10.5TBofRAMavailableinthe“hot”
tier and approximately 10TB of segments loaded. Collectively,
thereareabout50billionDruidrowsinthistier. Resultsforevery
data source are not shown.
The hot tier uses Intel®Xeon®E5-2670 processors and consists
of 1302 processing threads and 672 total cores (hyperthreaded).
A memory-mapped storage engine was used (the machine was
conﬁguredtomemorymapthedatainsteadofloadingitintothe
Java heap.)
QuerylatenciesareshowninFigure8andthequeriesperminute
are shown in Figure 9. Across all the various data sources, aver-
age query latency is approximately 550 milliseconds, with 90% of
queries returning in less than 1 second, 95% in under 2 seconds,
and99%ofqueriesreturninginlessthan10seconds. Occasionally
we observe spikes in latency, as observed on February 19, where
network issues on the Memcached instances were compounded by
very high query load on one of our largestdata sources.
6.2 Query Benchmarks on TPC-H Data
WealsopresentDruidbenchmarksonTPC-Hdata. MostTPC-H
queriesdonotdirectlyapplytoDruid,soweselectedqueriesmore
typicalofDruid’sworkloadtodemonstratequeryperformance. As
a comparison, we also provide the results of the same queries us-
ingMySQLusingtheMyISAMengine(InnoDBwasslowerinour
experiments).
WeselectedMySQLtobenchmarkagainstbecauseofitsuniver-
sal popularity. We chose not to select another open source column
store because we were not conﬁdent we could correctly tune it for
optimal performance.
Our Druid setup used Amazon EC2 m3.2xlarge instance types
(Intel®Xeon®E5-2680 v2 @ 2.80GHz) for historical nodes and
c3.2xlarge instances(Intel®Xeon®E5-2670v2@2.50GHz)for
0.00.51.0
Feb 03 Feb 10 Feb 17 Feb 24
timequery time (s)datasource
a
b
c
d
e
f
g
hMean query latency
0.00.51.01.5
01234
0510152090%ile 95%ile 99%ile
Feb 03 Feb 10 Feb 17 Feb 24
timequery time (seconds)datasource
a
b
c
d
e
f
g
hQuery latency percentilesFigure8: Query latencies of productiondata sources.
050010001500
Feb 03 Feb 10 Feb 17 Feb 24
timequeries / minutedatasource
a
b
c
d
e
f
g
hQueries per minute
Figure9: Queriesper minute of productiondata sources.01234
count_star_interval
sum_all
sum_all_filter
sum_all_year
sum_price
top_100_commitdate
top_100_parts
top_100_parts_details
top_100_parts_filter
QueryTime (seconds)engine
Druid
MySQLMedian query time (100 runs) − 1GB data − single nodeFigure10: Druid & MySQL benchmarks – 1GB TPC-H data.
aggregation top−n
0200400600
02500500075001000012500count_star_interval
sum_all
sum_all_filter
sum_all_year
sum_price
top_100_commitdate
top_100_parts
top_100_parts_details
top_100_parts_filter
QueryTime (seconds)engine
Druid
MySQLMedian Query Time (3+ runs) − 100GB data − single node
Figure11: Druid&MySQLbenchmarks–100GBTPC-Hdata.
broker nodes. Our MySQL setup was an Amazon RDS instance
that ran on the same m3.2xlarge instance type.
The results for the 1 GB TPC-H data set are shown in Figure 10
and the results of the 100 GB data set are shown in Figure 11.
WebenchmarkedDruid’sscanrateat53,539,211rows/second/core
forselect count(*) equivalentqueryoveragiventimeinterval
and36,246,530rows/second/corefora select sum(float) type
query.
Finally,wepresentourresultsofscalingDruidtomeetincreasing
data volumes with the TPC-H 100 GB data set. We observe that
whenweincreasedthenumberofcoresfrom8to48,notalltypesof
queries achieve linear scaling, but the simpler aggregation queries
do, as shown in Figure 12.
Theincreaseinspeedofaparallelcomputingsystemisoftenlim-
itedbythetimeneededforthesequentialoperationsofthesystem.
In this case, queries requiring a substantial amount of work at the
broker level do not parallelize as well.
6.3 Data Ingestion Performance
To showcase Druid’s data ingestion latency, we selected several
production datasources of varying dimensions, metrics, and event
volumes. Our production ingestion setup consists of 6 nodes, to-
talling 360GB of RAM and 96 cores (12 x Intel®Xeon®E5-2670).
count_star_intervalsum_allsum_all_filtersum_all_yearsum_pricetop_100_commitdatetop_100_partstop_100_parts_detailstop_100_parts_filter
0
50
100 150Time (seconds)QueryDruid Scaling − 100GB
count_star_intervalsum_allsum_all_filtersum_all_yearsum_pricetop_100_commitdatetop_100_partstop_100_parts_detailstop_100_parts_filter
1 2 3 4 5 6Speedup FactorQuery
Cores 8 (1 node) 48 (6 nodes)Druid Scaling ... 100GBFigure12: Druid scalingbenchmarks – 100GB TPC-H data.
DataSource Dimensions Metrics Peak events/s
s 7 228334.60
t 10 768808.70
u 5 149933.93
v 30 1022240.45
w 35 14135763.17
x 28 646525.85
y 33 24162462.41
z 33 2495747.74
Table3: Ingestion characteristics of various data sources.
Note that in this setup, several other data sources were being in-
gested and many other Druid related ingestion tasks were running
concurrently on the machines.
Druid’s data ingestion latency is heavily dependent on the com-
plexity of the data set being ingested. The data complexity is de-
termined by the number of dimensions in each event, the number
of metrics in each event, and the types of aggregations we want to
perform on those metrics. With the most basic data set (one that
onlyhasatimestampcolumn),oursetupcaningestdataatarateof
800,000 events/second/core, which is really just a measurement of
how fast we can deserialize events. Real world data sets are never
this simple. Table 3 shows a selection of data sources and their
characteristics.
We can see that, based on the descriptions in Table 3, latencies
varysigniﬁcantlyandtheingestionlatencyisnotalwaysafactorof
thenumberofdimensionsandmetrics. Weseesomelowerlatencies
onsimpledatasetsbecausethatwastheratethatthedataproducer
was delivering data. The results are shown in Figure 13.
We deﬁne throughput as the number of events a real-time node
can ingest and also make queryable. If too many events are sent
to the real-time node, those events are blocked until the real-time
node has capacity to accept them. The peak ingestion latency we
measuredinproductionwas22914.43events/second/coreonadata-
source with 30 dimensions and 19 metrics, running an Amazon
cc2.8xlarge instance.050,000100,000150,000200,000250,000
Dec 15 Jan 01 Jan 15 Feb 01 Feb 15 Mar 01
timeevents / sdatasource
s
t
u
v
w
x
y
zEvents per second ... 24h moving averageFigure13: Combined cluster ingestion rates.
Thelatencymeasurementswepresentedaresufﬁcienttoaddress
thestatedproblemsofinteractivity. Wewouldpreferthevariability
in the latencies to be less. It is still possible to decrease latencies
by adding additional hardware, but we have not chosen to do so
because infrastructure costs are still a consideration for us.
7. DRUID IN PRODUCTION
Over the last few years, we have gained tremendous knowledge
about handling production workloads with Druid and have made a
couple of interesting observations.
Query Patterns.
Druid is often used to explore data and generate reports on data.
In the explore use case, the number of queries issued by a single
user are much higher than in the reporting use case. Exploratory
queriesofteninvolveprogressivelyaddingﬁltersforthesametime
range to narrow down results. Users tend to explore short time in-
tervals of recent data. In the generate report use case, users query
for much longer data intervals, but those queries are generally few
and pre-determined.
Multitenancy.
Expensiveconcurrentqueriescanbeproblematicinamultitenant
environment. Queriesforlargedatasourcesmayenduphittingev-
ery historical node in a cluster and consume all cluster resources.
Smaller, cheaper queries may be blocked from executing in such
cases. We introduced query prioritization to address these issues.
Each historical node is able to prioritize which segments it needs
toscan. Properqueryplanningiscriticalforproductionworkloads.
Thankfully, queries for a signiﬁcant amount of data tend to be for
reporting use cases and can be deprioritized. Users do not expect
the same level of interactivity in this use case as when they are ex-
ploring data.
Node failures.
Singlenodefailuresarecommonindistributedenvironments,but
many nodes failing at once are not. If historical nodes completely
failanddonotrecover,theirsegmentsneedtobereassigned,which
meansweneedexcessclustercapacitytoloadthisdata. Theamount
of additional capacity to have at any time contributes to the cost
of running a cluster. From our experiences, it is extremely rare to
see more than 2 nodes completely fail at once and hence, we leave
enoughcapacityinourclustertocompletelyreassignthedatafrom
2 historical nodes.Data Center Outages.
Complete cluster failures are possible, but extremely rare. If
Druid is only deployed in a single data center, it is possible for
the entire data center to fail. In such cases, new machines need
to be provisioned. As long as deep storage is still available, clus-
terrecoverytimeisnetworkbound,ashistoricalnodessimplyneed
to redownload every segment from deep storage. We have experi-
enced such failures in the past, and the recovery time was several
hoursintheAmazonAWSecosystemforseveralterabytesofdata.
7.1 Operational Monitoring
Propermonitoringiscriticaltorunalargescaledistributedclus-
ter. Each Druid node is designed to periodically emit a set of oper-
ational metrics. These metrics may include system level data such
asCPUusage,availablememory,anddiskcapacity,JVMstatistics
such as garbage collection time, and heap usage, or node speciﬁc
metrics such as segment scan time, cache hit rates, and data inges-
tion latencies. Druid also emits per query metrics.
We emit metrics from a production Druid cluster and load them
into a dedicated metrics Druid cluster. The metrics Druid cluster
is used to explore the performance and stability of the production
cluster. This dedicated metrics cluster has allowed us to ﬁnd nu-
merousproductionproblems,suchasgradualqueryspeeddegrega-
tions,lessthanoptimallytunedhardware,andvariousothersystem
bottlenecks. We also use a metrics cluster to analyze what queries
aremadeinproductionandwhataspectsofthedatausersaremost
interested in.
7.2 Pairing Druid with a Stream Processor
Currently, Druid can only understand fully denormalized data
streams. Inordertoprovidefullbusinesslogicinproduction,Druid
can be paired with a stream processor such as Apache Storm [27].
A Storm topology consumes events from a data stream, retains
only those that are “on-time”, and applies any relevant business
logic. This could range from simple transformations, such as id
tonamelookups,tocomplexoperationssuchasmulti-streamjoins.
The Storm topology forwards the processed event stream to Druid
in real-time. Storm handles the streaming data processing work,
and Druid is used for responding to queries for both real-time and
historical data.
7.3 Multiple Data Center Distribution
Largescaleproductionoutagesmaynotonlyaffectsinglenodes,
but entire data centers as well. The tier conﬁguration in Druid co-
ordinatornodesallowforsegmentstobereplicatedacrossmultiple
tiers. Hence, segments can be exactly replicated across historical
nodes in multiple data centers. Similarily, query preference can be
assigned to different tiers. It is possible to have nodes in one data
center act as a primary cluster (and receive all queries) and have a
redundant cluster in another data center. Such a setup may be de-
sired if one data center is situated much closer to users.
8. RELATED WORK
Cattell [6] maintains a great summary about existing Scalable
SQL and NoSQL data stores. Hu [18] contributed another great
summary for streaming databases. Druid, feature-wise, sits some-
where between Google’s Dremel [28] and PowerDrill [17]. Druid
has most of the features implemented in Dremel (Dremel handles
arbitrarynesteddatastructureswhileDruidonlyallowsforasingle
level of array-based nesting) and many of the interesting compres-
sion algorithms mentioned in PowerDrill.
Although Druid builds on many of the same principles as other
distributedcolumnardatastores[15],manyofthesedatastoresaredesigned to be more generic key-value stores [23] and do not sup-
port computation directly in the storage layer. There are also other
data stores designed for some of the same data warehousing issues
that Druid is meant to solve. These systems include in-memory
databases such as SAP’s HANA [14] and VoltDB [43]. These data
storeslackDruid’slowlatencyingestioncharacteristics. Druidalso
has native analytical features baked in, similar to ParAccel [34],
however, Druid allows system wide rolling software updates with
no downtime.
Druid is similiar to C-Store [38] and LazyBase [8] in that it has
twosubsystems,aread-optimizedsubsysteminthehistoricalnodes
and a write-optimized subsystem in the real-time nodes. Real-time
nodes are designed to ingest a high volume of append heavy data,
and do not support data updates. Unlike the two aforementioned
systems,DruidismeantforOLAPtransactionsandnotOLTPtrans-
actions.
Druid’s low latency data ingestion features share some similar-
ities with Trident/Storm [27] and Spark Streaming [45], however,
both systems are focused on stream processing whereas Druid is
focused on ingestion and aggregation. Stream processors are great
complementstoDruidasameansofpre-processingthedatabefore
the data enters Druid.
There are a class of systems that specialize in queries on top of
cluster computing frameworks. Shark [13] is such a system for
queriesontopofSpark,andCloudera’sImpala[9]isanothersystem
focused on optimizing query performance on top of HDFS. Druid
historical nodes download data locally and only work with native
Druid indexes. We believe this setup allows for faster query laten-
cies.
Druidleveragesauniquecombinationofalgorithmsinitsarchi-
tecture. Although we believe no other data store has the same set
of functionality as Druid, some of Druid’s optimization techniques
suchasusinginvertedindicestoperformfastﬁltersarealsousedin
other data stores [26].
9. CONCLUSIONS
InthispaperwepresentedDruid,adistributed,column-oriented,
real-time analytical data store. Druid is designed to power high
performanceapplicationsandisoptimizedforlowquerylatencies.
Druid supports streaming data ingestion and is fault-tolerant. We
discussed Druid benchmarks and summarized key architecture as-
pects such as the storage format, query language, and general exe-
cution.
10. ACKNOWLEDGEMENTS
Druid could not have been built without the help of many great
engineersatMetamarketsandinthecommunity. Wewanttothank
everyone that has contributed to the Druid codebase for their in-
valuable support.
11. REFERENCES
[1] D. J. Abadi, S. R. Madden, and N. Hachem. Column-stores
vs.row-stores: Howdifferentaretheyreally? In Proceedings
of the 2008 ACM SIGMOD international conferenceon
Management of data , pages 967–980. ACM,2008.
[2] G. Antoshenkov.Byte-aligned bitmap compression. In Data
CompressionConference,1995. DCC’95. Proceedings , page
476. IEEE, 1995.
[3] Apache. Apache solr.
http://lucene.apache.org/solr/ , February 2013.
[4] S. Banon. Elasticsearch.
http://www.elasticseach.com/ , July 2013.[5] C. Bear,A. Lamb, and N. Tran. The vertica database: Sql
rdbms for managing big data. In Proceedingsof the 2012
workshop on Management of big data systems , pages 37–38.
ACM,2012.
[6] R. Cattell. Scalable sql and nosql data stores. ACM SIGMOD
Record, 39(4):12–27, 2011.
[7] F.Chang, J. Dean, S. Ghemawat, W.C. Hsieh, D. A.
Wallach,M. Burrows, T.Chandra, A. Fikes,and R. E.
Gruber.Bigtable: A distributed storage system for structured
data.ACM Transactionson Computer Systems (TOCS) ,
26(2):4, 2008.
[8] J. Cipar,G. Ganger, K. Keeton, C. B. Morrey III, C. A.
Soules, and A. Veitch.Lazybase: trading freshness for
performanceinascalabledatabase.In Proceedingsofthe7th
ACM europeanconference on Computer Systems , pages
169–182. ACM,2012.
[9] Cloudera impala. http://blog.cloudera.com/blog ,
March 2013.
[10] A. Colantonio and R. Di Pietro. Concise: Compressed
‘n’composable integer set. Information ProcessingLetters ,
110(16):644–650,2010.
[11] J. Dean and S. Ghemawat. Mapreduce: simpliﬁed data
processing on largeclusters. Communications of the ACM ,
51(1):107–113,2008.
[12] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati,
A. Lakshman, A. Pilchin, S. Sivasubramanian, P.Vosshall,
and W.Vogels.Dynamo: amazon’s highly available
key-value store. In ACM SIGOPS Operating Systems
Review, volume 41, pages 205–220. ACM, 2007.
[13] C. Engle, A. Lupher,R. Xin, M. Zaharia, M. J. Franklin,
S. Shenker,and I. Stoica. Shark: fast data analysis using
coarse-grained distributed memory.In Proceedingsof the
2012 international conferenceon Management of Data ,
pages 689–692. ACM, 2012.
[14] F.Färber,S. K. Cha, J. Primsch, C. Bornhövd, S. Sigg, and
W.Lehner.Sap hana database: data management for modern
business applications. ACM Sigmod Record , 40(4):45–51,
2012.
[15] B. Fink. Distributed computation on dynamo-style
distributed storage: riak pipe. In Proceedingsof the eleventh
ACM SIGPLAN workshop on Erlang workshop , pages
43–50. ACM,2012.
[16] B. Fitzpatrick. Distributed caching with memcached. Linux
journal, (124):72–74, 2004.
[17] A. Hall, O. Bachmann, R. Büssow,S. Gănceanu, and
M. Nunkesser.Processing a trillion cells per mouse click.
Proceedingsof the VLDB Endowment , 5(11):1436–1446,
2012.
[18] B. Hu. Stream database survey.2011.
[19] P.Hunt, M. Konar,F.P. Junqueira, and B. Reed. Zookeeper:
Wait-freecoordinationforinternet-scalesystems.In USENIX
ATC, volume 10, 2010.
[20] C. S. Kim. Lrfu: A spectrum of policies that subsumes the
least recently used and least frequently used policies. IEEE
Transactionson Computers , 50(12), 2001.
[21] J. Kreps, N.Narkhede, and J. Rao. Kafka: A distributed
messaging system for log processing. In Proceedingsof 6th
International Workshopon Networking Meets Databases
(NetDB), Athens, Greece , 2011.
[22] T.Lachev. Applied MicrosoftAnalysis Services 2005: And
MicrosoftBusiness Intelligence Platform . Prologika Press,
2005.[23] A. Lakshman and P.Malik. Cassandra—a decentralized
structured storage system. Operating systems review ,
44(2):35, 2010.
[24] Liblzf. http://freecode.com/projects/liblzf , March
2013.
[25] LinkedIn. Senseidb. http://www.senseidb.com/ , July
2013.
[26] R. MacNicol and B. French. Sybase iq multiplex-designed
for analytics. In Proceedingsof the Thirtieth international
conferenceon Verylargedata bases-Volume30 , pages
1227–1230.VLDBEndowment, 2004.
[27] N. Marz. Storm: Distributed and fault-tolerant realtime
computation. http://storm-project.net/ , February
2013.
[28] S. Melnik, A. Gubarev,J. J. Long, G. Romer,S. Shivakumar,
M.Tolton,and T.Vassilakis.Dremel: interactive analysis of
web-scale datasets. Proceedingsof the VLDB Endowment ,
3(1-2):330–339, 2010.
[29] D. Miner.Uniﬁed analytics platform for big data. In
Proceedingsof the WICSA/ECSA 2012 Companion Volume ,
pages 176–176. ACM,2012.
[30] K.Oehler,J.Gruenes,C.Ilacqua,andM.Perez. IBMCognos
TM1: The Ofﬁcial Guide . McGraw-Hill, 2012.
[31] E. J. O’neil, P.E. O’neil, and G. Weikum.The lru-k page
replacement algorithm for database disk buffering.In ACM
SIGMODRecord , volume 22, pages 297–306. ACM, 1993.
[32] P.O’Neil and D. Quass. Improved query performance with
variant indexes. In ACM Sigmod Record , volume 26, pages
38–49. ACM,1997.
[33] P.O’Neil, E. Cheng, D. Gawlick, and E. O’Neil. The
log-structured merge-tree(lsm-tree). Acta Informatica ,
33(4):351–385, 1996.
[34] Paraccel analytic database.
http://www.paraccel.com/resources/Datasheets/
ParAccel-Core-Analytic-Database.pdf , March 2013.
[35] M. Schrader,D. Vlamis, M. Nader,C. Claterbos, D. Collins,
M.Campbell, and F.Conrad. Oracle Essbase & Oracle
OLAP.McGraw-Hill, Inc., 2009.[36] K. Shvachko, H. Kuang, S. Radia, and R. Chansler.The
hadoop distributed ﬁle system. In Mass Storage Systems and
Technologies(MSST), 2010 IEEE 26th Symposium on , pages
1–10. IEEE, 2010.
[37] M. Singh and B. Leonhardi. Introduction to the ibm netezza
warehouse appliance. In Proceedingsof the 2011Conference
of the Center for Advanced Studies on Collaborative
Research, pages 385–386. IBM Corp., 2011.
[38] M. Stonebraker,D. J. Abadi, A. Batkin, X. Chen,
M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden,
E. O’Neil, et al. C-store: a column-oriented dbms. In
Proceedingsof the 31st international conferenceon Very
largedata bases , pages 553–564. VLDB Endowment, 2005.
[39] A. Tomasicand H. Garcia-Molina. Performance of inverted
indices in shared-nothing distributed text document
information retrieval systems. In Parallel and Distributed
Information Systems, 1993., Proceedingsof the Second
International Conferenceon ,pages 8–17. IEEE, 1993.
[40] E. Tschetter.Introducing druid: Real-time analytics at a
billion rows per second. http://druid.io/blog/2011/
04/30/introducing-druid.html , April 2011.
[41] Twitterpublic streams. https://dev.twitter.com/
docs/streaming-apis/streams/public , March 2013.
[42] S. J.van Schaik and O. de Moor.A memory efﬁcient
reachability data structure through bit vector compression.In
Proceedingsof the 2011international conferenceon
Management of data , pages 913–924. ACM, 2011.
[43] L. VoltDB.Voltdbtechnical overview.
https://voltdb.com/ , 2010.
[44] K. Wu,E. J. Otoo, and A. Shoshani. Optimizing bitmap
indices with efﬁcient compression. ACM Transactionson
Database Systems (TODS) , 31(1):1–38, 2006.
[45] M. Zaharia, T.Das, H. Li, S. Shenker, and I. Stoica.
Discretized streams: an efﬁcient and fault-tolerant model for
streamprocessingonlargeclusters.In Proceedingsofthe4th
USENIX conferenceon Hot Topicsin Cloud Computing ,
pages 10–10. USENIX Association, 2012.