Amazon Aurora: Design Considerations for HighThroughput Cloud-Native Relational Databases
Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta,
Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, Xiaofeng Bao
Amazon Web Services

ABSTRACTThe I/O bottleneck faced by traditional database systems changes
in this environment. Since I/Os can be spread across many nodes
and many disks in a multi-tenant fleet, the individual disks andnodes are no longer hot. Instead, the bottleneck moves to the
network between the database tier requesting I/Os and the storagetier that performs these I/Os. Beyond the basic bottlenecks ofpackets per second (PPS) and bandwidth, there is amplification of
traffic since a performant database will issue writes out to thestorage fleet in parallel. The performance of the outlier storage
node, disk or network path can dominate response time.

Amazon Aurora is a relational database service for OLTP
workloads offered as part of Amazon Web Services (AWS). In
this paper, we describe the architecture of Aurora and the design
considerations leading to that architecture. We believe the central
constraint in high throughput data processing has moved fromcompute and storage to the network. Aurora brings a novel
architecture to the relational database to address this constraint,
most notably by pushing redo processing to a multi-tenant scaleout storage service, purpose-built for Aurora. We describe how
doing so not only reduces network traffic, but also allows for fastcrash recovery, failovers to replicas without loss of data, andfault-tolerant, self-healing storage. We then describe how Aurora
achieves consensus on durable state across numerous storage
nodes using an efficient asynchronous scheme, avoiding
expensive and chatty recovery protocols. Finally, having operated
Aurora as a production service for over 18 months, we sharelessons we have learned from our customers on what modern
cloud applications expect from their database tier.

Although most operations in a database can overlap with eachother, there are several situations that require synchronous
operations. These result in stalls and context switches. One suchsituation is a disk read due to a miss in the database buffer cache.
A reading thread cannot continue until its read completes. A cache
miss may also incur the extra penalty of evicting and flushing adirty cache page to accommodate the new page. Backgroundprocessing such as checkpointing and dirty page writing canreduce the occurrence of this penalty, but can also cause stalls,
context switches and resource contention.

Keywords

Transaction commits are another source of interference; a stall incommitting one transaction can inhibit others from progressing.
Handling commits with multi-phase synchronization protocols
such as 2-phase commit (2PC) [3][4][5] is challenging in a cloudscale distributed system. These protocols are intolerant of failureand high-scale distributed systems have a continual “backgroundnoise” of hard and soft failures. They are also high latency, as
high scale systems are distributed across multiple data centers.

Databases; Distributed Systems; Log Processing; Quorum
Models; Replication; Recovery; Performance; OLTP

1. INTRODUCTION
IT workloads are increasingly moving to public cloud providers.
Significant reasons for this industry-wide transition include theability to provision capacity on a flexible on-demand basis and topay for this capacity using an operational expense as opposed tocapital expense model. Many IT workloads require a relational
OLTP database; providing equivalent or superior capabilities toon-premise databases is critical to support this secular transition.

Data Plane

In modern distributed cloud services, resilience and scalability are
increasingly achieved by decoupling compute from storage[10][24][36][38][39] and by replicating storage across multiple
nodes. Doing so lets us handle operations such as replacing
misbehaving or unreachable hosts, adding replicas, failing overfrom a writer to a replica, scaling the size of a database instanceup or down, etc.

Control Plane

SQL
Transactions
CachingAmazonDynamoDB

Logging + Storage

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for components of this work owned
by others than the author(s) must be honored. Abstracting with credit is permitted.
To copy otherwise, or republish, to post on servers or to redistribute to lists, requireprior specific permission and/or a fee. Request permissions from
Permissions@acm.org
SIGMOD’17, May 14 – 19, 2017, Chicago, IL, USA.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-4197-4/17/05…$15.00
DOI: http://dx.doi.org/10.1145/3035918.3056101

Amazon SWF

Amazon S3

Figure 1: Move logging and storage off the database engine

1041

In this paper, we describe Amazon Aurora, a new database service
that addresses the above issues by more aggressively leveraging
the redo log across a highly-distributed cloud environment. We
use a novel service-oriented architecture (see Figure 1) with amulti-tenant scale-out storage service that abstracts a virtualized
segmented redo log and is loosely coupled to a fleet of databaseinstances. Although each instance still includes most of thecomponents of a traditional kernel (query processor, transactions,locking, buffer cache, access methods and undo management)
several functions (redo logging, durable storage, crash recovery,
and backup/restore) are off-loaded to the storage service.

low level background noise of node, disk and network pathfailures. Each failure can have a different duration and a different
blast radius. For example, one can have a transient lack of
network availability to a node, temporary downtime on a reboot,or a permanent failure of a disk, a node, a rack, a leaf or a spine
network switch, or even a data center.
One approach to tolerate failures in a replicated system is to use aquorum-based voting protocol as described in [6]. If each of the V
copies of a replicated data item is assigned a vote, a read or write
operation must respectively obtain a read quorum of Vr votes or awrite quorum of Vw votes. To achieve consistency, the quorums
must obey two rules. First, each read must be aware of the mostrecent write, formulated as Vr + Vw > V. This rule ensures the setof nodes used for a read intersects with the set of nodes used for awrite and the read quorum contains at least one location with thenewest version. Second, each write must be aware of the mostrecent write to avoid conflicting writes, formulated as Vw > V/2.

Our architecture has three significant advantages over traditional
approaches. First, by building storage as an independent faulttolerant and self-healing service across multiple data-centers, we
protect the database from performance variance and transient orpermanent failures at either the networking or storage tiers. We
observe that a failure in durability can be modeled as a longlasting availability event, and an availability event can be modeled
as a long-lasting performance variation – a well-designed system
can treat each of these uniformly [42]. Second, by only writing
redo log records to storage, we are able to reduce network IOPS
by an order of magnitude. Once we removed this bottleneck, wewere able to aggressively optimize numerous other points ofcontention, obtaining significant throughput improvements overthe base MySQL code base from which we started. Third, wemove some of the most complex and critical functions (backup
and redo recovery) from one-time expensive operations in the
database engine to continuous asynchronous operations amortized
across a large distributed fleet. This yields near-instant crashrecovery without checkpointing as well as inexpensive backups
that do not interfere with foreground processing.

A common approach to tolerate the loss of a single node is to
replicate data to (V = 3) nodes and rely on a write quorum of 2/3(Vw = 2) and a read quorum of 2/3 (Vr = 2).
We believe 2/3 quorums are inadequate. To understand why, let’s
first understand the concept of an Availability Zone (AZ) in
AWS. An AZ is a subset of a Region that is connected to otherAZs in the region through low latency links but is isolated formost faults, including power, networking, software deployments,
flooding, etc. Distributing data replicas across AZs ensures thattypical failure modalities at scale only impact one data replica.
This implies that one can simply place each of the three replicas ina different AZ, and be tolerant to large-scale events in addition to
the smaller individual failures.
However, in a large storage fleet, the background noise of failures
implies that, at any given moment in time, some subset of disks or
nodes may have failed and are being repaired. These failures may
be spread independently across nodes in each of AZ A, B and C.
However, the failure of AZ C, due to a fire, roof failure, flood,etc, will break quorum for any of the replicas that concurrently
have failures in AZ A or AZ B. At that point, in a 2/3 readquorum model, we will have lost two copies and will be unable todetermine if the third is up to date. In other words, while theindividual failures of replicas in each of the AZs are uncorrelated,the failure of an AZ is a correlated failure of all disks and nodes inthat AZ. Quorums need to tolerate an AZ failure as well asconcurrently occuring background noise failures.

In this paper, we describe three contributions:
1.
2.
3.

How to reason about durability at cloud scale and how
to design quorum systems that are resilient to correlated
failures. (Section 2).
How to leverage smart storage by offloading the lower
quarter of a traditional database to this tier. (Section 3).
How to eliminate multi-phase synchronization, crashrecovery and checkpointing in distributed storage(Section 4).

We then show how we bring these three ideas together to design
the overall architecture of Aurora in Section 5, followed by areview of our performance results in Section 6 and the lessons wehave learned in Section 7. Finally, we briefly survey related workin Section 8 and present concluding remarks in Section 9.

In Aurora, we have chosen a design point of tolerating (a) losing
an entire AZ and one additional node (AZ+1) without losing data,and (b) losing an entire AZ without impacting the ability to write
data. We achieve this by replicating each data item 6 ways across
3 AZs with 2 copies of each item in each AZ. We use a quorum
model with 6 votes (V = 6), a write quorum of 4/6 (Vw = 4), and aread quorum of 3/6 (Vr = 3). With such a model, we can (a) lose asingle AZ and one additional node (a failure of 3 nodes) without
losing read availability, and (b) lose any two nodes, including asingle AZ failure and maintain write availability. Ensuring readquorum enables us to rebuild write quorum by adding additional
replica copies.

2. DURABILITY AT SCALE
If a database system does nothing else, it must satisfy the contractthat data, once written, can be read. Not all systems do. In thissection, we discuss the rationale behind our quorum model, why
we segment storage, and how the two, in combination, provide not
only durability, availability and reduction of jitter, but also help us
solve the operational issues of managing a storage fleet at scale.

2.1 Replication and Correlated Failures
Instance lifetime does not correlate well with storage lifetime.
Instances fail. Customers shut them down. They resize them up
and down based on load. For these reasons, it helps to decouple
the storage tier from the compute tier.

2.2 Segmented Storage
Let’s consider the question of whether AZ+1 provides sufficientdurability. To provide sufficient durability in this model, one mustensure the probability of a double fault on uncorrelated failures(Mean Time to Failure – MTTF) is sufficiently low over the time

Once you do so, those storage nodes and disks can also fail. They
therefore must be replicated in some form to provide resiliency to
failure. In a large-scale cloud environment, there is a continuous1042

it takes to repair one of these failures (Mean Time to Repair –MTTR). If the probability of a double fault is sufficiently high, we
may see these on an AZ failure, breaking quorum. It is difficult,past a point, to reduce the probability of MTTF on independent
failures. We instead focus on reducing MTTR to shrink thewindow of vulnerability to a double fault. We do so by
partitioning the database volume into small fixed size segments,
currently 10GB in size. These are each replicated 6 ways intoProtection Groups (PGs) so that each PG consists of six 10GB
segments, organized across three AZs, with two segments in each
AZ. A storage volume is a concatenated set of PGs, physically
implemented using a large fleet of storage nodes that areprovisioned as virtual hosts with attached SSDs using Amazon
Elastic Compute Cloud (EC2). The PGs that constitute a volume
are allocated as the volume grows. We currently support volumes
that can grow up to 64 TB on an unreplicated basis.

that stall pipelines and dilate latencies. While chain replication [8]and its alternatives can reduce network cost, they still suffer fromsynchronous stalls and additive latencies.
Let’s examine how writes work in a traditional database. A
system like MySQL writes data pages to objects it exposes (e.g.,heap files, b-trees etc.) as well as redo log records to a write-aheadlog (WAL). Each redo log record consists of the difference
between the after-image and the before-image of the page that was
modified. A log record can be applied to the before-image of thepage to produce its after-image.
In practice, other data must also be written. For instance, consider
a synchronous mirrored MySQL configuration that achieves highavailability across data-centers and operates in an active-standby
configuration as shown in Figure 2. There is an active MySQL
instance in AZ1 with networked storage on Amazon Elastic BlockStore (EBS). There is also a standby MySQL instance in AZ2,
also with networked storage on EBS. The writes made to theprimary EBS volume are synchronized with the standby EBS
volume using software mirroring.

Segments are now our unit of independent background noise
failure and repair. We monitor and automatically repair faults aspart of our service. A 10GB segment can be repaired in 10
seconds on a 10Gbps network link. We would need to see twosuch failures in the same 10 second window plus a failure of an
AZ not containing either of these two independent failures to losequorum. At our observed failure rates, that’s sufficiently unlikely,
even for the number of databases we manage for our customers.

2.3 Operational Advantages of Resilience
Once one has designed a system that is naturally resilient to long
failures, it is naturally also resilient to shorter ones. A storage
system that can handle the long-term loss of an AZ can also
handle a brief outage due to a power event or bad software
deployment requiring rollback. One that can handle a multisecond loss of availability of a member of a quorum can handle abrief period of network congestion or load on a storage node.
Since our system has a high tolerance to failures, we can leverage
this for maintenance operations that cause segment unavailability.
For example, heat management is straightforward. We can markone of the segments on a hot disk or node as bad, and the quorum
will be quickly repaired by migration to some other colder node in
the fleet. OS and security patching is a brief unavailability eventfor that storage node as it is being patched. Even software
upgrades to our storage fleet are managed this way. We execute
them one AZ at a time and ensure no more than one member of a
PG is being patched simultaneously. This allows us to use agile
methodologies and rapid deployments in our storage service.

Figure 2: Network IO in mirrored MySQL

3. THE LOG IS THE DATABASE

Figure 2 shows the various types of data that the engine needs towrite: the redo log, the binary (statement) log that is archived to
Amazon Simple Storage Service (S3) in order to support point-intime restores, the modified data pages, a second temporary writeof the data page (double-write) to prevent torn pages, and finallythe metadata (FRM) files. The figure also shows the order of theactual IO flow as follows. In Steps 1 and 2, writes are issued to
EBS, which in turn issues it to an AZ-local mirror, and theacknowledgement is received when both are done. Next, in Step
3, the write is staged to the standby instance using synchronous
block-level software mirroring. Finally, in steps 4 and 5, writes
are written to the standby EBS volume and associated mirror.

In this section, we explain why using a traditional database on asegmented replicated storage system as described in Section 2
imposes an untenable performance burden in terms of network
IOs and synchronous stalls. We then explain our approach where
we offload log processing to the storage service andexperimentally demonstrate how our approach can dramaticallyreduce network IOs. Finally, we describe various techniques we
use in the storage service to minimize synchronous stalls andunnecessary writes.

3.1 The Burden of Amplified Writes
Our model of segmenting a storage volume and replicating eachsegment 6 ways with a 4/6 write quorum gives us high resilience.
Unfortunately, this model results in untenable performance for a
traditional database like MySQL that generates many different
actual I/Os for each application write. The high I/O volume isamplified by replication, imposing a heavy packets per second(PPS) burden. Also, the I/Os result in points of synchronization

The mirrored MySQL model described above is undesirable not
only because of how data is written but also because of what datais written. First, steps 1, 3, and 5 are sequential and synchronous.
Latency is additive because many writes are sequential. Jitter isamplified because, even on asynchronous writes, one must wait
for the slowest operation, leaving the system at the mercy of1043

outliers. From a distributed system perspective, this model can beviewed as having a 4/4 write quorum, and is vulnerable to failuresand outlier performance. Second, user operations that are a resultof OLTP applications cause many different types of writes often
representing the same information in multiple ways – for example,
the writes to the double write buffer in order to prevent torn pages
in the storage infrastructure.

of 6 replicas in order to satisfy the write quorum and consider thelog records in question durable or hardened. The replicas use theredo log records to apply changes to their buffer caches.
To measure network I/O, we ran a test using the SysBench [9]write-only workload with a 100GB data set for bothconfigurations described above: one with a synchronous mirrored
MySQL configuration across multiple AZs and the other withRDS Aurora (with replicas across multiple AZs). In bothinstances, the test ran for 30 minutes against database engines
running on an r3.8xlarge EC2 instance.

3.2 Offloading Redo Processing to Storage
When a traditional database modifies a data page, it generates aredo log record and invokes a log applicator that applies the redo
log record to the in-memory before-image of the page to produce
its after-image. Transaction commit requires the log to be written,
but the data page write may be deferred.
AZ 1

AZ 2

AZ 3

Primary
Instance

Replica
Instance

Replica
Instance

Table 1: Network IOs for Aurora vs MySQL
Configuration

Transactions

IOs/Transaction

Mirrored MySQL

780,000

7.4

Aurora with Replicas

27,378,000

0.95

The results of our experiment are summarized in Table 1. Overthe 30-minute period, Aurora was able to sustain 35 times moretransactions than mirrored MySQL. The number of I/Os pertransaction on the database node in Aurora was 7.7 times fewer
than in mirrored MySQL despite amplifying writes six times with
Aurora and not counting the chained replication within EBS nor
the cross-AZ writes in MySQL. Each storage node sees
unamplified writes, since it is only one of the six copies, resultingin 46 times fewer I/Os requiring processing at this tier. The
savings we obtain by writing less data to the network allow us toaggressively replicate data for durability and availability and issue
requests in parallel to minimize the impact of jitter.

ASYNC
4/6 QUORUM

DISTRIBUTEDWRITES

Amazon S3

Moving processing to a storage service also improves availability
by minimizing crash recovery time and eliminates jitter caused bybackground processes such as checkpointing, background data
page writing and backups.

Figure 3: Network IO in Amazon Aurora
In Aurora, the only writes that cross the network are redo log
records. No pages are ever written from the database tier, not forbackground writes, not for checkpointing, and not for cache
eviction. Instead, the log applicator is pushed to the storage tierwhere it can be used to generate database pages in background or
on demand. Of course, generating each page from the complete
chain of its modifications from the beginning of time isprohibitively expensive. We therefore continually materialize
database pages in the background to avoid regenerating them fromscratch on demand every time. Note that backgroundmaterialization is entirely optional from the perspective ofcorrectness: as far as the engine is concerned, the log is the
database, and any pages that the storage system materializes are
simply a cache of log applications. Note also that, unlike
checkpointing, only pages with a long chain of modifications need
to be rematerialized. Checkpointing is governed by the length ofthe entire redo log chain. Aurora page materialization is governedby the length of the chain for a given page.

Let’s examine crash recovery. In a traditional database, after acrash the system must start from the most recent checkpoint andreplay the log to ensure that all persisted redo records have been
applied. In Aurora, durable redo record application happens at thestorage tier, continuously, asynchronously, and distributed acrossthe fleet. Any read request for a data page may require some redo
records to be applied if the page is not current. As a result, theprocess of crash recovery is spread across all normal foregroundprocessing. Nothing is required at database startup.

3.3 Storage Service Design Points
A core design tenet for our storage service is to minimize thelatency of the foreground write request. We move the majority ofstorage processing to the background. Given the naturalvariability between peak to average foreground requests from thestorage tier, we have ample time to perform these tasks outside theforeground path. We also have the opportunity to trade CPU for
disk. For example, it isn’t necessary to run garbage collection(GC) of old page versions when the storage node is busy
processing foreground write requests unless the disk isapproaching capacity. In Aurora, background processing has
negative correlation with foreground processing. This is unlike a
traditional database, where background writes of pages andcheckpointing have positive correlation with the foreground loadon the system. If we build up a backlog on the system, we will
throttle foreground activity to prevent a long queue buildup. Since
segments are placed with high entropy across the various storage
nodes in our system, throttling at one storage node is readily
handled by our 4/6 quorum writes, appearing as a slow node.

Our approach dramatically reduces network load despite
amplifying writes for replication and provides performance aswell as durability. The storage service can scale out I/Os in an
embarrassingly parallel fashion without impacting write
throughput of the database engine. For instance, Figure 3 showsan Aurora cluster with one primary instance and multiple replicas
instances deployed across multiple AZs. In this model, theprimary only writes log records to the storage service and streams
those log records as well as metadata updates to the replica
instances. The IO flow batches fully ordered log records based on
a common destination (a logical segment, i.e., a PG) and delivers
each batch to all 6 replicas where the batch is persisted on disk
and the database engine waits for acknowledgements from 4 out

1044

each of these individual transactions. The logic for tracking
partially completed transactions and undoing them is kept in the
database engine, just as if it were writing to simple disks.
However, upon restart, before the database is allowed to accessthe storage volume, the storage service does its own recovery
which is focused not on user-level transactions, but on making
sure that the database sees a uniform view of storage despite its
distributed nature.

STORAGE NODELOG RECORDS

Primary
Instance

INCOMING QUEUE

ACK

GC

2

UPDATE
QUEUE

SORT
GROUP

Peer
Storage
Nodes

7

1

PEER TO PEER GOSSIP

43COALESCEDATAPAGES SCRUB

5

8The storage service determines the highest LSN for which it canguarantee availability of all prior log records (this is known as the
VCL or Volume Complete LSN). During storage recovery, every
log record with an LSN larger than the VCL must be truncated.
The database can, however, further constrain a subset of points
that are allowable for truncation by tagging log records andidentifying them as CPLs or Consistency Point LSNs. We
therefore define VDL or the Volume Durable LSN as the highest
CPL that is smaller than or equal to VCL and truncate all log
records with LSN greater than the VDL. For example, even if wehave the complete data up to LSN 1007, the database may havedeclared that only 900, 1000, and 1100 are CPLs, in which case,
we must truncate at 1000. We are complete to 1007, but only
durable to 1000.

HOT
LOG
POINT IN TIME
SNAPSHOT

6
S3 BACKUP

Figure 4: IO Traffic in Aurora Storage Nodes
Let’s examine the various activities on the storage node in moredetail. As seen in Figure 4, it involves the following steps: (1)receive log record and add to an in-memory queue, (2) persist
record on disk and acknowledge, (3) organize records and identify
gaps in the log since some batches may be lost, (4) gossip withpeers to fill in gaps, (5) coalesce log records into new data pages,(6) periodically stage log and new pages to S3, (7) periodicallygarbage collect old versions, and finally (8) periodically validate
CRC codes on pages.

Completeness and durability are therefore different and a CPL canbe thought of as delineating some limited form of storage systemtransaction that must be accepted in order. If the client has no use
for such distinctions, it can simply mark every log record as a
CPL. In practice, the database and storage interact as follows:

Note that not only are each of the steps above asynchronous, only
steps (1) and (2) are in the foreground path potentially impacting
latency.

1.

4. THE LOG MARCHES FORWARD

2.

In this section, we describe how the log is generated from thedatabase engine so that the durable state, the runtime state, and thereplica state are always consistent. In particular, we will describe
how consistency is implemented efficiently without an expensive
2PC protocol. First, we show how we avoid expensive redo
processing on crash recovery. Next, we explain normal operationand how we maintain runtime and replica state. Finally, we
provide details of our recovery process.

3.

Each database-level transaction is broken up into
multiple mini-transactions (MTRs) that are ordered andmust be performed atomically.
Each mini-transaction is composed of multiple
contiguous log records (as many as needed).
The final log record in a mini-transaction is a CPL.

On recovery, the database talks to the storage service to establish
the durable point of each PG and uses that to establish the VDL
and then issue commands to truncate the log records above VDL.

4.2 Normal Operation
We now describe the “normal operation” of the database engine
and focus in turn on writes, reads, commits, and replicas.

4.1 Solution sketch: Asynchronous Processing

4.2.1 Writes

Since we model the database as a redo log stream (as described in
Section 3), we can exploit the fact that the log advances as anordered sequence of changes. In practice, each log record has an
associated Log Sequence Number (LSN) that is a monotonicallyincreasing value generated by the database.

In Aurora, the database continuously interacts with the storage
service and maintains state to establish quorum, advance volume
durability, and register transactions as committed. For instance, inthe normal/forward path, as the database receives
acknowledgements to establish the write quorum for each batch oflog records, it advances the current VDL. At any given moment,
there can be a large number of concurrent transactions active inthe database, each generating their own redo log records. The
database allocates a unique ordered LSN for each log record
subject to a constraint that no LSN is allocated with a value that isgreater than the sum of the current VDL and a constant called the
LSN Allocation Limit (LAL) (currently set to 10 million). This
limit ensures that the database does not get too far ahead of thestorage system and introduces back-pressure that can throttle theincoming writes if the storage or network cannot keep up.

This lets us simplify a consensus protocol for maintaining state byapproaching the problem in an asynchronous fashion instead ofusing a protocol like 2PC which is chatty and intolerant of
failures. At a high level, we maintain points of consistency anddurability, and continually advance these points as we receive
acknowledgements for outstanding storage requests. Since any
individual storage node might have missed one or more log
records, they gossip with the other members of their PG, looking
for gaps and fill in the holes. The runtime state maintained by the
database lets us use single segment reads rather than quorum reads
except on recovery when the state is lost and has to be rebuilt.

Note that each segment of each PG only sees a subset of log
records in the volume that affect the pages residing on thatsegment. Each log record contains a backlink that identifies theprevious log record for that PG. These backlinks can be used to
track the point of completeness of the log records that havereached each segment to establish a Segment Complete LSN

The database may have multiple outstanding isolated transactions,which can complete (reach a finished and durable state) in adifferent order than initiated. Supposing the database crashes or
reboots, the determination of whether to roll back is separate for

1045

(SCL) that identifies the greatest LSN below which all log recordsof the PG have been received. The SCL is used by the storage
nodes when they gossip with each other in order to find andexchange log records that they are missing.

The actual concurrency control protocols are executed in the
database engine exactly as though the database pages and undo
segments are organized in local storage as with traditional
MySQL.

4.2.2 Commits

4.2.4 Replicas

In Aurora, transaction commits are completed asynchronously.
When a client commits a transaction, the thread handling thecommit request sets the transaction aside by recording its “commit
LSN” as part of a separate list of transactions waiting on commit
and moves on to perform other work. The equivalent to the WAL
protocol is based on completing a commit, if and only if, the latest
VDL is greater than or equal to the transaction’s commit LSN. As
the VDL advances, the database identifies qualifying transactions
that are waiting to be committed and uses a dedicated thread tosend commit acknowledgements to waiting clients. Worker
threads do not pause for commits, they simply pull other pending
requests and continue processing.

In Aurora, a single writer and up to 15 read replicas can all mounta single shared storage volume. As a result, read replicas add no
additional costs in terms of consumed storage or disk write
operations. To minimize lag, the log stream generated by thewriter and sent to the storage nodes is also sent to all read replicas.
In the reader, the database consumes this log stream byconsidering each log record in turn. If the log record refers to apage in the reader's buffer cache, it uses the log applicator toapply the specified redo operation to the page in the cache.
Otherwise it simply discards the log record. Note that the replicas
consume log records asynchronously from the perspective of thewriter, which acknowledges user commits independent of thereplica. The replica obeys the following two important rules while
applying log records: (a) the only log records that will be applied
are those whose LSN is less than or equal to the VDL, and (b) thelog records that are part of a single mini-transaction are applied
atomically in the replica's cache to ensure that the replica sees aconsistent view of all database objects. In practice, each replica
typically lags behind the writer by a short interval (20 ms or less).

4.2.3 Reads
In Aurora, as with most databases, pages are served from thebuffer cache and only result in a storage IO request if the page inquestion is not present in the cache.
If the buffer cache is full, the system finds a victim page to evictfrom the cache. In a traditional system, if the victim is a “dirty
page” then it is flushed to disk before replacement. This is toensure that a subsequent fetch of the page always results in thelatest data. While the Aurora database does not write out pages oneviction (or anywhere else), it enforces a similar guarantee: a pagein the buffer cache must always be of the latest version. The
guarantee is implemented by evicting a page from the cache only
if its “page LSN” (identifying the log record associated with thelatest change to the page) is greater than or equal to the VDL. This
protocol ensures that: (a) all changes in the page have been
hardened in the log, and (b) on a cache miss, it is sufficient torequest a version of the page as of the current VDL to get its latest
durable version.

4.3 Recovery
Most traditional databases use a recovery protocol such as ARIES[7] that depends on the presence of a write-ahead log (WAL) thatcan represent the precise contents of all committed transactions.
These systems also periodically checkpoint the database toestablish points of durability in a coarse-grained fashion byflushing dirty pages to disk and writing a checkpoint record to thelog. On restart, any given page can either miss some committed
data or contain uncommitted data. Therefore, on crash recovery
the system processes the redo log records since the last checkpointby using the log applicator to apply each log record to the relevant
database page. This process brings the database pages to aconsistent state at the point of failure after which the in-flighttransactions during the crash can be rolled back by executing therelevant undo log records. Crash recovery can be an expensive
operation. Reducing the checkpoint interval helps, but at theexpense of interference with foreground transactions. No suchtradeoff is required with Aurora.

The database does not need to establish consensus using a readquorum under normal circumstances. When reading a page fromdisk, the database establishes a read-point, representing the VDLat the time the request was issued. The database can then select astorage node that is complete with respect to the read point,knowing that it will therefore receive an up to date version. A
page that is returned by the storage node must be consistent withthe expected semantics of a mini-transaction (MTR) in the
database. Since the database directly manages feeding log records
to storage nodes and tracking progress (i.e., the SCL of eachsegment), it normally knows which segment is capable ofsatisfying a read (the segments whose SCL is greater than theread-point) and thus can issue a read request directly to a segmentthat has sufficient data.

A great simplifying principle of a traditional database is that thesame redo log applicator is used in the forward processing path aswell as on recovery where it operates synchronously and in theforeground while the database is offline. We rely on the same
principle in Aurora as well, except that the redo log applicator isdecoupled from the database and operates on storage nodes, inparallel, and all the time in the background. Once the databasestarts up it performs volume recovery in collaboration with thestorage service and as a result, an Aurora database can recoververy quickly (generally under 10 seconds) even if it crashed while
processing over 100,000 write statements per second.

Given that the database is aware of all outstanding reads, it cancompute at any time the Minimum Read Point LSN on a per-PG
basis. If there are read replicas the writer gossips with them toestablish the per-PG Minimum Read Point LSN across all nodes.
This value is called the Protection Group Min Read Point LSN(PGMRPL) and represents the “low water mark” below which all
the log records of the PG are unnecessary. In other words, astorage node segment is guaranteed that there will be no read page
requests with a read-point that is lower than the PGMRPL. Eachstorage node is aware of the PGMRPL from the database and can,therefore, advance the materialized pages on disk by coalescing
the older log records and then safely garbage collecting them.

The database does need to reestablish its runtime state after acrash. In this case, it contacts for each PG, a read quorum ofsegments which is sufficient to guarantee discovery of any datathat could have reached a write quorum. Once the database has
established a read quorum for every PG it can recalculate the
VDL above which data is truncated by generating a truncation
range that annuls every log record after the new VDL, up to andincluding an end LSN which the database can prove is at least as
high as the highest possible outstanding log record that could ever1046

have been seen. The database infers this upper bound because it
allocates LSNs, and limits how far allocation can occur above
VDL (the 10 million limit described earlier). The truncation
ranges are versioned with epoch numbers, and written durably tothe storage service so that there is no confusion over the durabilityof truncations in case recovery is interrupted and restarted.

commits in the writer and use this information to support snapshot
isolation for local transactions that are of course read-only. Notethat concurrency control is implemented entirely in the databaseengine without impacting the storage service. The storage service
presents a unified view of the underlying data that is logically
identical to what you would get by writing the data to local
storage in community InnoDB.

The database still needs to perform undo recovery to unwind theoperations of in-flight transactions at the time of the crash.
However, undo recovery can happen when the database is online
after the system builds the list of these in-flight transactions fromthe undo segments.

Aurora leverages Amazon Relational Database Service (RDS) for
its control plane. RDS includes an agent on the database instancecalled the Host Manager (HM) that monitors a cluster’s health anddetermines if it needs to fail over, or if an instance needs to be
replaced. Each database instance is part of a cluster that consistsof a single writer and zero or more read replicas. The instances ofa cluster are in a single geographical region (e.g., us-east-1, uswest-1 etc.), are typically placed in different AZs, and connect toa storage fleet in the same region. For security, we isolate thecommunication between the database, applications and storage. In
practice, each database instance can communicate on three
Amazon Virtual Private Cloud (VPC) networks: the customer
VPC through which customer applications interact with theengine, the RDS VPC through which the database engine andcontrol plane interact with each other, and the Storage VPC
through which the database interacts with storage services.

5. PUTTING IT ALL TOGETHER
In this section, we describe the building blocks of Aurora as
shown with a bird’s eye view in Figure 5.

The storage service is deployed on a cluster of EC2 VMs that areprovisioned across at least 3 AZs in each region and iscollectively responsible for provisioning multiple customer
storage volumes, reading and writing data to and from thosevolumes, and backing up and restoring data from and to thosevolumes. The storage nodes manipulate local SSDs and interact
with database engine instances, other peer storage nodes, and the
backup/restore services that continuously backup changed data to
S3 and restore data from S3 as needed. The storage control plane
uses the Amazon DynamoDB database service for persistentstorage of cluster and storage volume configuration, volume
metadata, and a detailed description of data backed up to S3. For
orchestrating long-running operations, e.g. a database volume
restore operation or a repair (re-replication) operation following astorage node failure, the storage control plane uses the Amazon
Simple Workflow Service. Maintaining a high level of availability
requires pro-active, automated, and early detection of real andpotential problems, before end users are impacted. All criticalaspects of storage operations are constantly monitored using
metric collection services that raise alarms if key performance oravailability metrics indicate a cause for concern.

Figure 5: Aurora Architecture: A Bird's Eye View
The database engine is a fork of “community” MySQL/InnoDB
and diverges primarily in how InnoDB reads and writes data to
disk. In community InnoDB, a write operation results in data
being modified in buffer pages, and the associated redo log
records written to buffers of the WAL in LSN order. On
transaction commit, the WAL protocol requires only that the redo
log records of the transaction are durably written to disk. The
actual modified buffer pages are also written to disk eventually
through a double-write technique to avoid partial page writes.
These page writes take place in the background, or during eviction
from the cache, or while taking a checkpoint. In addition to the IO
Subsystem, InnoDB also includes the transaction subsystem, thelock manager, a B+-Tree implementation and the associatednotion of a “mini transaction” (MTR). An MTR is a construct
only used inside InnoDB and models groups of operations thatmust be executed atomically (e.g., split/merge of B+-Tree pages).

6. PERFORMANCE RESULTS
In this section, we will share our experiences in running Aurora as
a production service that was made “Generally Available” in July
2015. We begin with a summary of results running industry
standard benchmarks and then present some performance resultsfrom our customers.

6.1 Results with Standard Benchmarks

In the Aurora InnoDB variant, the redo log records representing
the changes that must be executed atomically in each MTR areorganized into batches that are sharded by the PGs each log record
belongs to, and these batches are written to the storage service.
The final log record of each MTR is tagged as a consistency point.
Aurora supports exactly the same isolation levels that aresupported by community MySQL in the writer (the standard
ANSI levels and Snapshot Isolation or consistent reads). Aurora
read replicas get continuous information on transaction starts andHere we present results of different experiments that compare theperformance of Aurora and MySQL using industry standard
benchmarks such as SysBench and TPC-C variants. We ran
MySQL on instances that are attached to an EBS volume with
30K provisioned IOPS. Except when stated otherwise, these are
r3.8xlarge EC2 instances with 32 vCPUs and 244GB of memory1047

and features the Intel Xeon E5-2670 v2 (Ivy Bridge) processors.
The buffer cache on the r3.8xlarge is set to 170GB.

shows that for the SysBench write-only workload, Aurora can be
up to 67x faster than MySQL with a database size of 100GB.
Even for a database size of 1TB with an out-of-cache workload,
Aurora is still 34x faster than MySQL.

6.1.1 Scaling with instance sizes
In this experiment, we report that throughput in Aurora can scalelinearly with instance sizes, and with the highest instance size canbe 5x that of MySQL 5.6 and MySQL 5.7. Note that Aurora iscurrently based on the MySQL 5.6 code base. We ran the
SysBench read-only and write-only benchmarks for a 1GB dataset (250 tables) on 5 EC2 instances of the r3 family (large, xlarge,
2xlarge, 4xlarge, 8xlarge). Each instance size has exactly half thevCPUs and memory of the immediately larger instance.

Table 2: SysBench Write-Only (writes/sec)
DB Size

Amazon Aurora

MySQL

1 GB

107,000

8,400

10 GB

107,000

2,400

100 GB

101,000

1,500

1 TB

41,0001,200

6.1.3 Scaling with user connections
In this experiment, we report that throughput in Aurora can scalewith the number of client connections. Table 3 shows the resultsof running the SysBench OLTP benchmark in terms of writes/sec
as the number of connections grows from 50 to 500 to 5000.
While Aurora scales from 40,000 writes/sec to 110,000 writes/sec,
the throughput in MySQL peaks at around 500 connections andthen drops sharply as the number of connections grows to 5000.
Table 3: SysBench OLTP (writes/sec)

Figure 7: Aurora scales linearly for write-only workload

Connections

Amazon Aurora

MySQL

50

40,00010,000500

71,00021,0005,000

110,000

13,0006.1.4 Scaling with Replicas
In this experiment, we report that the lag in an Aurora read replica
is significantly lower than that of a MySQL replica even withmore intense workloads. Table 4 shows that as the workload
varies from 1,000 to 10,000 writes/second, the replica lag in
Aurora grows from 2.62 milliseconds to 5.38 milliseconds. In
contrast, the replica lag in MySQL grows from under a second to
300 seconds. At 10,000 writes/second Aurora has a replica lagthat is several orders of magnitude smaller than that of MySQL.
Replica lag is measured in terms of the time it takes for acommitted transaction to be visible in the replica.
Table 4: Replica Lag for SysBench Write-Only (msec)
Writes/sec

Amazon Aurora

MySQL

1,000

2.62

< 1000

2,000

3.42

1000

Figure 6: Aurora scales linearly for read-only workload

5,000

3.94

60,000

The results are shown in Figure 7 and Figure 6, and measure theperformance in terms of write and read statements per second
respectively. Aurora’s performance doubles for each higher
instance size and for the r3.8xlarge achieves 121,000 writes/sec
and 600,000 reads/sec which is 5x that of MySQL 5.7 which topsout at 20,000 reads/sec and 125,000 writes/sec.

10,0005.38

300,000

6.1.5 Throughput with hot row contention
In this experiment, we report that Aurora performs very wellrelative to MySQL on workloads with hot row contention, such as
those based on the TPC-C benchmark. We ran the Percona TPC-C
variant [37] against Amazon Aurora and MySQL 5.6 and 5.7 onan r3.8xlarge where MySQL uses an EBS volume with 30K
provisioned IOPS. Table 5 shows that Aurora can sustain between
2.3x to 16.3x the throughput of MySQL 5.7 as the workload
varies from 500 connections and a 10GB data size to 5000
connections and a 100GB data size.

6.1.2 Throughput with varying data sizes
In this experiment, we report that throughput in Aurora
significantly exceeds that of MySQL even with larger data sizes
including workloads with out-of-cache working sets. Table 21048

Table 5: Percona TPC-C Variant (tpmC)
Connections/Size/
Warehouses

AmazonAurora

MySQL
5.6

MySQL
5.7

500/10GB/100

73,955

6,093

25,2895000/10GB/100

42,181

1,671

2,592

500/100GB/1000

70,663

3,231

11,8685000/100GB/1000

30,221

5,575

13,005Figure 10: INSERT per-record latency (P50 vs P95)

6.2 Results with Real Customer Workloads

6.2.3 Replica Lag with Multiple Replicas

In this section, we share results reported by some of our customers
who migrated production workloads from MySQL to Aurora.

MySQL replicas often lag significantly behind their writers andcan “can cause strange bugs” as reported by Weiner at Pinterest[40]. For the education technology company described earlier, thereplica lag often spiked to 12 minutes and impacted application
correctness and so the replica was only useful as a stand by. In
contrast, after migrating to Aurora, the maximum replica lag
across 4 replicas never exceeded 20ms as shown in Figure 11. The
improved replica lag provided by Aurora let the company divert asignificant portion of their application load to the replicas saving
costs and increasing availability.

6.2.1 Application response time with Aurora
An internet gaming company migrated their production service
from MySQL to Aurora on an r3.4xlarge instance. The average
response time that their web transactions experienced prior to themigration was 15 ms. In contrast, after the migration the average
response time 5.5 ms, a 3x improvement as shown in Figure 8.

Figure 8: Web application response time

6.2.2 Statement Latencies with Aurora
Figure 11: Maximum Replica Lag (averaged hourly)

An education technology company whose service helps schools
manage student laptops migrated their production workload fromMySQL to Aurora. The median (P50) and 95th percentile (P99)
latencies for select and per-record insert operations before andafter the migration (at 14:00 hours) are shown in Figure 9 and
Figure 10.

7. LESSONS LEARNED
We have now seen a large variety of applications run bycustomers ranging from small internet companies all the way tohighly sophisticated organizations operating large numbers of
Aurora clusters. While many of their use cases are standard, we
focus on scenarios and expectations that are common in the cloud
and are leading us to new directions.

7.1 Multi-tenancy and database consolidation
Many of our customers operate Software-as-a-Service (SaaS)
businesses, either exclusively or with some residual on-premisecustomers they are trying to move to their SaaS model. We findthat these customers often rely on an application they cannot
easily change. Therefore, they typically consolidate their different
customers on a single instance by using a schema/database as aunit of tenancy. This idiom reduces costs: they avoid paying for a
dedicated instance per customer when it is unlikely that all of theircustomers active at once. For instance, some of our SaaS
customers report ha ving more than 50,000 customers of theirown.

Figure 9: SELECT latency (P50 vs P95)
Before the migration, the P95 latencies ranged between 40ms to80ms and were much worse than the P50 latencies of about 1ms.
The application was experiencing the kinds of poor outlier
performance that we described earlier in this paper. After the
migration, however, the P95 latencies for both operations
improved dramatically and approximated the P50 latencies.

1049

This model is markedly different from well-known multi-tenant
applications like Salesforce.com [14] which use a multi-tenantdata model and pack the data of multiple customers into unified
tables of a single schema with tenancy identified on a per-row
basis. As a result, we see many customers with consolidated
databases containing a large number of tables. Production
instances of over 150,000 tables for small database are quite
common. This puts pressure on components that manage metadata
like the dictionary cache. More importantly, such customers need(a) to sustain a high level of throughput and many concurrent user
connections, (b) a model where data is only provisioned and paid
for as it is used since it is hard to anticipate in advance how muchstorage space is needed, and (c) reduced jitter so that spikes for asingle tenant have minimal impact on other tenants. Aurora
supports these attributes and fits such SaaS applications very well.

Figure 12: Zero-Downtime Patching

8. RELATED WORK

7.2 Highly concurrent auto-scaling workloads

In this section, we discuss other contributions and how they relate
to the approaches taken in Aurora.

Internet workloads often need to deal with spikes in traffic basedon sudden unexpected events. One of our major customers had a
special appearance in a highly popular nationally televised show
and experienced one such spike that greatly surpassed theirnormal peak throughput without stressing the database. To
support such spikes, it is important for a database to handle many
concurrent connections. This approach is feasible in Aurora since
the underlying storage system scales so well. We have several
customers that run at over 8000 connections per second.

Decoupling storage from compute. Although traditional systems
have usually been built as monolithic daemons [27], there has
been recent work on databases that decompose the kernel into
different components. For instance, Deuteronomy [10] is one such
system that separates a Transaction Component (TC) that provides
concurrency control and recovery from a Data Component (DC)that provides access methods on top of LLAMA [34], a latch-freelog-structured cache and storage manager. Sinfonia [39] andHyder [38] are systems that abstract transactional access methods
over a scale out service and database systems can be implemented
using these abstractions. The Yesquel [36] system implements a
multi-version distributed balanced tree and separates concurrency
control from the query processor. Aurora decouples storage at alevel lower than that of Deuteronomy, Hyder, Sinfonia, andYesquel. In Aurora, query processing, transactions, concurrency,
buffer cache, and access methods are decoupled from logging,storage, and recovery that are implemented as a scale out service.

7.3 Schema evolution
Modern web application frameworks such as Ruby on Rails
deeply integrate object-relational mapping tools. As a result, it iseasy for application developers to make many schema changes totheir database making it challenging for DBAs to manage how theschema evolves. In Rails applications, these are called “DBMigrations” and we have heard first-hand accounts of DBAs thathave to either deal with a “few dozen migrations a week”, or put
in place hedging strategies to ensure that future migrations takeplace without pain. The situation is exacerbated with MySQL
offering liberal schema evolution semantics and implementing
most changes using a full table copy. Since frequent DDL is apragmatic reality, we have implemented an efficient online DDL
implementation that (a) versions schemas on a per-page basis anddecodes individual pages on demand using their schema history,and (b) lazily upgrades individual pages to the latest schema usinga modify-on-write primitive.

Distributed Systems. The trade-offs between correctness andavailability in the face of partitions have long been known withthe major result that one-copy serializability is not possible in theface of network partitions [15]. More recently Brewer’s CAP
Theorem as proved in [16] stated that a highly available systemcannot provide “strong” consistency guarantees in the presence of
network partitions. These results and our experience with cloudscale complex and correlated failures motivated our consistency
goals even in the presence of partitions caused by an AZ failure.
Bailis et al [12] study the problem of providing Highly Available
Transactions (HATs) that neither suffer unavailability during
partitions nor incur high network latency. They show that
Serializability, Snapshot Isolation and Repeatable Read isolation
are not HAT-compliant, while most other isolation levels areachievable with high availability. Aurora provides all these
isolation levels by making a simplifying assumption that at any
time there is only a single writer generating log updates with
LSNs allocated from a single ordered domain.

7.4 Availability and Software Upgrades
Our customers have demanding expectations of cloud-native
databases that can conflict with how we operate the fleet and how
often we patch servers. Since our customers use Aurora primarily
as an OLTP service backing production applications, any
disruption can be traumatic. As a result, many of our customers
have a very low tolerance to our updates of database software,even if this amounts to a planned downtime of 30 seconds every 6
weeks or so. Therefore, we recently released a new ZeroDowntime Patch (ZDP) feature that allows us to patch a customer
while in-flight database connections are unaffected.

Google’s Spanner [24] provides externally consistent [25] reads
and writes, and globally-consistent reads across the database at atimestamp. These features enable Spanner to support consistent
backups, consistent distributed query processing [26], and atomic
schema updates, all at global scale, and even in the presence ofongoing transactions. As explained by Bailis [12], Spanner ishighly specialized for Google’s read-heavy workload and relies on
two-phase commit and two-phase locking for read/write
transactions.

As shown in Figure 12, ZDP works by looking for an instantwhere there are no active transactions, and in that instant spoolingthe application state to local ephemeral storage, patching theengine and then reloading the application state. In the process,user sessions remain active and oblivious that the engine changed
under the covers.

1050

11. REFERENCES

Concurrency Control. Weaker consistency (PACELC [17]) andisolation models [18][20] are well known in distributed databases
and have led to optimistic replication techniques [19] as well as
eventually consistent systems [21][22][23]. Other approaches incentralized systems range from classic pessimistic schemes basedon locking [28], optimistic schemes like multi-versioned
concurrency control in Hekaton [29], sharded approaches such as
VoltDB [30] and Timestamp ordering in HyPer [31][32] andDeuteronomy. Aurora’s storage service provides the databaseengine the abstraction of a local disk that is durably persisted, andallows the engine to determine isolation and concurrency control.

[1] B. Calder, J. Wang, et al. Windows Azure storage: A highly
available cloud storage service with strong consistency. In
SOSP 2011.
[2] O. Khan, R. Burns, J. Plank, W. Pierce, and C. Huang.
Rethinking erasure codes for cloud file systems: Minimizing
I/O for recovery and degraded reads. In FAST 2012.
[3] P.A. Bernstein, V. Hadzilacos, and N. Goodman.
Concurrency control and recovery in database systems,
Chapter 7, Addison Wesley Publishing Company, ISBN 0201-10715-5, 1997.

Log-structured storage. Log-structured storage systems wereintroduced by LFS [33] in 1992. More recently Deuteronomy andthe associated work in LLAMA [34] and Bw-Tree [35] use logstructured techniques in multiple ways across the storage engine
stack and, like Aurora, reduce write amplification by writing
deltas instead of whole pages. Both Deuteronomy and Aurora
implement pure redo logging, and keep track of the highest stable
LSN for acknowledging commits.

[4] C. Mohan, B. Lindsay, and R. Obermarck. Transaction
management in the R* distributed database management
system”. ACM TODS, 11(4):378-396, 1986.
[5] C. Mohan and B. Lindsay. Efficient commit protocols for thetree of processes model of distributed transactions. ACM
SIGOPS Operating Systems Review, 19(2):40-52, 1985.

Recovery. While traditional databases rely on a recovery protocol
based on ARIES [5], some recent systems have chosen other paths
for performance. For example, Hekaton and VoltDB rebuild theirin-memory state after a crash using some form of an update log.
Systems like Sinfonia [39] avoid recovery by using techniques
like process pairs and state machine replication. Graefe [41]describes a system with per-page log record chains that enables
on-demand page-by-page redo that can make recovery fast. Like
Aurora, Deuteronomy does not require redo recovery. This isbecause Deuteronomy delays transactions so that only committed
updates are posted to durable storage. As a result, unlike Aurora,
the size of transactions can be constrained in Deuteronomy.

[6] D.K. Gifford. Weighted voting for replicated data. In SOSP
1979.
[7] C. Mohan, D.L. Haderle, B. Lindsay, H. Pirahesh, and P.
Schwarz. ARIES: A transaction recovery method supporting
fine-granularity locking and partial rollbacks using writeahead logging. ACM TODS, 17 (1): 94–162, 1992.
[8] R. van Renesse and F. Schneider. Chain replication forsupporting high throughput and availability. In OSDI 2004.
[9] A. Kopytov. Sysbench Manual. Available at
http://imysql.com/wp-content/uploads/2014/10/sysbenchmanual.pdf

9. CONCLUSION[10] J. Levandoski, D. Lomet, S. Sengupta, R. Stutsman, and R.
Wang. High performance transactions in deuteronomy. In
CIDR 2015.

We designed Aurora as a high throughput OLTP database thatcompromises neither availability nor durability in a cloud-scale
environment. The big idea was to move away from the monolithic
architecture of traditional databases and decouple storage fromcompute. In particular, we moved the lower quarter of thedatabase kernel to an independent scalable and distributed service
that managed logging and storage. With all I/Os written over the
network, our fundamental constraint is now the network. As aresult we need to focus on techniques that relieve the network andimprove throughput. We rely on quorum models that can handle
the complex and correlated failures that occur in large-scale cloudenvironments and avoid outlier performance penalties, log
processing to reduce the aggregate I/O burden, and asynchronous
consensus to eliminate chatty and expensive multi-phasesynchronization protocols, offline crash recovery, andcheckpointing in distributed storage. Our approach has led to asimplified architecture with reduced complexity that is easy toscale as well as a foundation for future advances.

[11] P. Bailis, A. Fekete, A. Ghodsi, J.M. Hellerstein, and I.
Stoica. Scalable atomic visibility with RAMP Transactions.
In SIGMOD 2014.
[12] P. Bailis, A. Davidson, A. Fekete, A. Ghodsi, J.M.
Hellerstein, and I. Stoica. Highly available transactions:virtues and limitations. In VLDB 2014.
[13] R. Taft, E. Mansour, M. Serafini, J. Duggan, A.J. Elmore, A.
Aboulnaga, A. Pavlo, and M. Stonebraker. E-Store: finegrained elastic partitioning for distributed transaction
processing systems. In VLDB 2015.
[14] R. Woollen. The internal design of salesforce.com’s multitenant architecture. In SoCC 2010.
[15] S. Davidson, H. Garcia-Molina, and D. Skeen. Consistency
in partitioned networks. ACM CSUR, 17(3):341–370, 1985.

10. ACKNOWLEDGMENTS[16] S. Gilbert and N. Lynch. Brewer’s conjecture and thefeasibility of consistent, available, partition-tolerant web
services. SIGACT News, 33(2):51–59, 2002.

We thank the entire Aurora development team for their efforts on
the project including our current members as well as ourdistinguished alumni (James Corey, Sam McKelvie, Yan
Leshinsky, Lon Lundgren, Pradeep Madhavarapu, and Stefano
Stefani). We are particularly grateful to our customers who
operate production workloads using our service and have been
generous in sharing their experiences and expectations with us.
We also thank the shepherds for their invaluable comments inshaping this paper.

[17] D.J. Abadi. Consistency tradeoffs in modern distributed
database system design: CAP is only part of the story. IEEE
Computer, 45(2), 2012.
[18] A. Adya. Weak consistency: a generalized theory andoptimistic implementations for distributed transactions. PhD
Thesis, MIT, 1999.

1051

[19] Y. Saito and M. Shapiro. Optimistic replication. ACM
Comput. Surv., 37(1), Mar. 2005.

[31] V. Leis, A. Kemper, et al. Exploiting hardware transactional
memory in main-memory databases. In ICDE 2014.

[20] H. Berenson, P. Bernstein, J. Gray, J. Melton, E. O’Neil, andP. O’Neil. A critique of ANSI SQL isolation levels. In
SIGMOD 1995.

[32] H. Mühe, S. Wolf, A. Kemper, and T. Neumann: An
evaluation of strict timestamp ordering concurrency control
for main-memory database systems. In IMDM 2013.

[21] P. Bailis and A. Ghodsi. Eventual consistency today:
limitations, extensions, and beyond. ACM Queue, 11(3),March 2013.

[33] M. Rosenblum and J. Ousterhout. The design andimplementation of a log-structured file system. ACM TOCS
10(1): 26–52, 1992.

[22] P. Bernstein and S. Das. Rethinking eventual consistency. In
SIGMOD, 2013.

[34] J. Levandoski, D. Lomet, S. Sengupta. LLAMA: A
cache/storage subsystem for modern hardware. PVLDB
6(10): 877-888, 2013.

[23] B. Cooper et al. PNUTS: Yahoo!’s hosted data serving
platform. In VLDB 2008.

[35] J. Levandoski, D. Lomet, and S. Sengupta. The Bw-Tree: A
B-tree for new hardware platforms. In ICDE 2013.

[24] J. C. Corbett, J. Dean, et al. Spanner: Google’s globallydistributed database. In OSDI 2012.

[36] M. Aguilera, J. Leners, and M. Walfish. Yesquel: scalable
SQL storage for web applications. In SOSP 2015.

[25] David K. Gifford. Information Storage in a Decentralized
Computer System. Tech. rep. CSL-81-8. PhD dissertation.
Xerox PARC, July 1982.

[37] Percona Lab. TPC-C Benchmark over MySQL. Available athttps://github.com/Percona-Lab/tpcc-mysql[26] Jeffrey Dean and Sanjay Ghemawat. MapReduce: a flexible
data processing tool”. CACM 53 (1):72-77, 2010.

[38] P. Bernstein, C. Reid, and S. Das. Hyder – A transactional
record manager for shared flash. In CIDR 2011.

[27] J. M. Hellerstein, M. Stonebraker, and J. R. Hamilton.
Architecture of a database system. Foundations and Trends
in Databases. 1(2) pp. 141-259, 2007.

[39] M. Aguilera, A. Merchant, M. Shah, A. Veitch, and C.
Karamanolis. Sinfonia: A new paradigm for building scalable
distributed systems. ACM Trans. Comput. Syst. 27(3): 2009.

[28] J. Gray, R. A. Lorie, G. R. Putzolu, I. L. Traiger. Granularityof locks in a shared data base. In VLDB 1975.

[40] M. Weiner. Sharding Pinterest: How we scaled our MySQL
fleet. Pinterest Engineering Blog. Available at:https://engineering.pinterest.com/blog/sharding-pinteresthow-we-scaled-our-mysql-fleet[29] P-A Larson, et al. High-Performance Concurrency control
mechanisms for main-memory databases. PVLDB, 5(4): 298309, 2011.

[41] G. Graefe. Instant recovery for data center savings. ACM
SIGMOD Record. 44(2):29-34, 2015.

[30] M. Stonebraker and A. Weisberg. The VoltDB main memory
DBMS. IEEE Data Eng. Bull., 36(2): 21-27, 2013.

[42] J. Dean and L. Barroso. The tail at scale. CACM 56(2):7480, 2013.

1052