Binary Star: Coordinated Reliability in Heterogeneous Memory
Systems for High Performance and Scalability
Xiao Liu1

David Roberts

1 UC San Diego

Rachata Ausavarungnirun2

Onur Mutlu3

2 King Mongkut’s University of Technology North Bangkok

Jishen Zhao1
3 ETH Zürich

ABSTRACT1

As memory capacity scales, traditional cache and memory hierarchy designs are facing increasingly difficult challenges in ensuringhigh reliability with low storage and performance cost. Recent
developments in 3D die-stacked DRAM caches and nonvolatilememories (NVRAMs) introduce promising opportunities in tackling the reliability, performance, and capacity challenges, due tothe diverse reliability characteristics of the technologies. However,
simply replacing DRAM with NVRAM does not solve the reliability issues of the memory system, as conventional memory systemdesigns maintain separate reliability schemes across caches andmain memory. Our goal in this paper is to enable a reliable andhigh-performance memory hierarchy design, as memory capacity scales. To this end, we propose Binary Star, which coordinates
the reliability schemes and consistent cache writeback between
3D-stacked DRAM last-level cache and NVRAM main memory tomaintain the reliability of the cache and the memory hierarchy.
Binary Star significantly reduces the performance and storage overhead of consistent cache writeback by coordinating it with NVRAM
wear leveling. As a result, Binary Star is much more reliable andoffers better performance than state-of-the-art memory systems
with error correction. On a set of memory-intensive workloads, weshow that Binary Star reduces memory failures in time (FIT) by
92.9% compared to state-of-the-art error correction schemes, while
retaining 99% of the performance of a conventional DRAM designthat provides no error correction.

Modern cloud servers adopt increasingly larger main memory andlast-level caches (LLC) to accommodate the large working set of various in-memory computing [83, 85], big-data analytics [4, 73], deeplearning [82], and server virtualization [4] applications. The memory capacity demand requires further process technology scaling oftraditional DRAMs, e.g., to sub-20nm technology nodes [8, 53]. Yet,such aggressive technology scaling imposes substantial challenges
in maintaining reliable and high-performance memory system operation due to two main reasons. First, resilience schemes to maintainmemory reliability can impose high-performance overhead in future memory systems. Future sub-20nm DRAMs require stronger
resilience techniques, such as in-DRAM error correction codes(IECC) [8, 34, 58, 62, 64, 67] and rank-level error correction codes(RECC) [46, 48, 75], compared to current commodity DRAMs. Figure 1 shows that such error correction code (ECC) schemes impose
significant performance overhead to the memory hierarchy that
employs sub-20nm DRAM as main memory. Second, even withstrong resilience schemes, future memory systems do not appear
to be as reliable as the state-of-the-art. In fact, due to increased
bit error rates (BERs), even when we combine RECC and IECC,
sub-20nm-DRAM-based memory systems with 3D-stacked DRAM
LLC can have lower reliability than a 28nm-DRAM-based memory
system that employs only RECC [8], as seen in Figure 1 (whencomparing the triangle and the circle).

Normalized Reliability

CCS CONCEPTS• Computer systems organization → Reliability; Processors
and memory architectures; • Hardware → Emerging technologies.

KEYWORDS
reliability, nonvolatile memory, hybrid memory, memory systems

INTRODUCTION

1028nm RECC DRAM
with 3D DRAM Cache

1

Sub-20nm
RECC+IECC DRAM
Sub-20nm
RECC+IECC DRAM
with 3D DRAM Cache0

Binary Star

0.9

ACM Reference Format:
Liu, et al.. 2019. Binary Star: Coordinated Reliability in Heterogeneous
Memory Systems for High Performance and Scalability. In Proceedings of
The 52nd Annual IEEE/ACM International Symposium on Microarchitecture,
Columbus, OH, USA, October 12–16, 2019 (MICRO-52), 14 pages.

28nm RECC DRAM

Better performanceand reliability

0.95

1

1.05

1.1

Normalized Throughput

Figure 1: Reliability and throughput of various LLC and main
memory hierarchy configurations normalized to 28nm DRAM with
RECC. We define “reliability” as the reciprocal of device failures intime (FIT) [8, 79], i.e., 1/FIT. Reliability data is based on a recent
study [8]. Throughput is measured using the benchmarks and system configurations described in Section 5. SRAM caches, which areless critical to scalability are kept constant across systems.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
MICRO-52, October 12–16, 2019, Columbus, OH, USA© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6938-1/19/10. . . $15.00
https://doi.org/10.1145/3352460.3358262

To address the DRAM technology scaling challenges, nextgeneration servers will adopt various new memory technologies.
For example, Intel’s next-generation Xeon servers and Lenovo’s
ThinkSystem SD650 servers support Optane DC PM [27, 29], which
1

MICRO-52, October 12–16, 2019, Columbus, OH, USA

Liu, et al.

demonstrates the practical use of large-capacity nonvolatile memories (NVRAMs) in servers. Intel’s Knights Landing Xeon Phi
server processors integrate up to 16GB multi-channel DRAM (MCDRAM) [26], which is a type of 3D-stacked DRAM,1 used as the
LLC. 3D DRAM and NVRAM promise much larger capacity thancommodity SRAM-based caches and DRAM-based main memory,
respectively.
However, reliability and performance issues remain. For example, compared with a 28nm DRAM (represented by the circle in
Figure 1), a sub-20nm DRAM (the cross in the figure) has bothworse performance and reliability. When 3D DRAM LLC is used,
the sub-20nm 3D-DRAM-cached DRAM (represented by the triangle in Figure 1) outperforms the sub-20nm DRAM (the cross in the
figure), but suffers from degraded reliability. We make similar observations when we compare the 28nm 3D-DRAM-cached DRAM
(the diamond in Figure 1) and the 28nm DRAM (the circle in the
figure).
Most NVRAM technologies are less vulnerable to soft errors than
DRAM [2, 76]. However, many of these technologies have much
lower endurance compared to DRAM [42, 74]. As a result, most
NVRAM-based main memory needs to adopt hard error protection techniques, imposing both performance and storage overheads
compared to DRAM-based main memory [3, 74, 81]. NVRAM alsoenables persistent memory techniques [30, 66], which allows data
structures stored in NVRAM to be recovered after system failures.
However, the performance of persistent memory systems is degraded by the high performance and storage overheads of multiversioning and write-order control mechanisms (e.g., cache flushes
and memory barriers) [30, 59, 84].
Our goal in this paper is to achieve both high reliability and highperformance, when we scale the capacity of LLC and main memory using new 3D DRAM and NVRAM technologies. To achieve
our goal, we propose Binary Star,2 a coordinated memory hierarchy reliability scheme, which consists of 1) coordinated reliability
mechanisms between 3D DRAM LLC and NVRAM (Section 3.1),
and 2) NVRAM wear leveling with consistent cache writeback (Section 3.2). The key insight of our design is that traditional memory
hierarchy designs typically strive to separately optimize the reliability of LLCs and main memory with decoupled reliability schemes,
yet coordinating reliability schemes across the LLC and main memory enhances the reliability of the overall memory hierarchy, whileat the same time reducing unnecessary ECC, multiversioning, andordering control overheads that degrade performance. As illustrated
in Figure 1, Binary Star (represented by a double star) achieves muchbetter reliability than state-of-the-art resilience schemes with various LLC and main memory configurations. Even with a slower PCM
as the main memory, Binary Star’s performance is comparable tothe combination of 28nm DRAM main memory and 3D DRAM LLC.
We show that Binary Star provides benefits across various types ofworkloads, including in-memory databases, in-memory key-value
stores, online transaction processing, data mining, scientific computation, image processing, and video encoding (Section 6).

This paper makes the following key contributions:• We identify the inefficiency of traditional reliability schemes thatare decoupled and uncoordinated across the memory hierarchy(i.e., between the LLC and main memory).
• We propose “Binary Star”, the first coordinated reliability scheme
across 3D DRAM LLC and NVRAM main memory. Our design
leverages consistent cache writeback in NVRAM to recover errors in 3D DRAM LLC. The LLC requires only error detection to
perform consistent cache writeback. This significantly reduces
the performance and storage overhead of consistent cache writeback.
• We develop a new technique that coordinates our consistentcache writeback technique with NVRAM wear leveling to reduce the performance, storage, and hardware implementation
overheads of memory reliability schemes.
• We develop a set of new software-hardware cooperative mechanisms to enable our design. Binary Star is transparent to theapplications and thus requires no application code modification.

2

BACKGROUND AND MOTIVATION

Existing memory hierarchy designs typically strive to optimize thereliability of individual components (either caches or main memory) in the memory hierarchy [8, 53, 62]. These designs lead todecoupled and uncoordinated reliability across different memory
hierarchy levels. As a result, existing designs suffer from high performance overhead and high hardware implementation cost. In thissection, we motivate our coordinated reliability mechanism by first
discussing the inefficiency of existing reliability schemes for DRAM
in Section 2.1, 3D DRAM LLC in Section 2.2, and NVRAM in Section 2.3. Then, we illustrate issues with decoupled, uncoordinatedreliability schemes in Section 2.4.

2.1

DRAM Main Memory

DRAM has been used as the major technology for implementing
main memory. However, DRAM scaling to sub-20nm technology
nodes introduces substantial reliability challenges [8, 37, 53, 54].
DRAM errors. DRAM errors can be roughly classified into transient errors and hard errors. Transient errors are caused by alphaparticles and cosmic rays [24], as well as retention time fluctuations [34, 44, 65, 67, 86]. These transient errors are less likely
to repeat and can be avoided after rewriting the corresponding
erroneous bits. Hard errors are caused by physical defects or failures [48, 55, 79]. These hard errors repeatedly occur in the same
memory cells due to permanent faults.
Challenges with process scaling. Process technology scaling
can compromise DRAM reliability. Randomly distributed singlecell failure (SCF) is the primary source of device failures [79]. The
frequency of SCF increases as DRAM cell size reduces, because as
smaller transistors and capacitors are more vulnerable to process
variation and manufacturing imperfections [8, 44]. Redundant rows
and columns can be used to fix SCFs and keep DRAM device yieldhigh [8]. Recent studies show that in-DRAM ECC (IECC), which
integrates ECC engines inside the DRAM device, is a promising
method to address DRAM reliability issues [8, 33, 64]. However,
IECC leads to significant storage overhead in DRAM chips [8, 58,
62, 64]. Furthermore, IECC needs to be combined with RECC [8],1 We use 3D DRAM and 3D-stacked DRAM interchangeably in this paper.

2 A “binary star” is a star system consisting of two stars orbiting around their common
barycenter. Our coordinated memory hierarchy reliability design appears like a starsystem that consists of two “stars” – a 3D DRAM LLC and an NVRAM main memory.

2

Binary StarMICRO-52, October 12–16, 2019, Columbus, OH, USA

to achieve reliability close to commodity DRAM [79], imposing
substantial performance overhead (See Section 6).

Ac

ECC

Error

On-chip 3D-stacked DRAM LLC(a)Using large 3D DRAM as an LLC can accommodate the increasing
working set size of server applications and reduce the performance
overhead of capacity scaling. However, the Through Silicon Vias
(TSVs) connecting the layers of 3D DRAM can increase the BER
by introducing additional defects [32]. Previous studies adopt twolevel error correction/detection schemes to improve 3D DRAM
reliability [9, 47, 57, 77]. However, these usually require heavy
hardware modifications and significant storage overhead.

2.3

(b)

log

Off-chip Main Memory(Persistent)

100%
93%
86%

0

5

10

Time (Seconds)

YCSB

TPCC

Redis

Memcached

mysql

facesimfluidanimate

freqmine

vips

x264

Figure 2: (a) Multiple reliability mechanisms for a cache line across
LLC and main memory. (b) Fraction of clean LLC lines during application execution. We collected data for ten seconds after the benchmark finishes the warm up stage.

NVRAM Main Memory

To tackle the scalability issues with DRAM technologies, NVRAM(e.g., PCM [42, 78], STT-RAM [41, 93], and RRAM [6]) introduces
a new tier in the memory hierarchy, due to its promising scalable
density and cost potential [16, 51, 80]. Many data center server software and hardware suppliers are adopting NVRAMs in their nextgeneration designs. Examples include Intel’s Optane DC PM [29],Microsoft’s storage class memory support in Windows OS and inmemory databases [11, 18], Red Hat’s persistent memory support
in the Linux kernel [52], and Mellanox’s persistent memory support
over the network fabric [15].
NVRAM errors. Similar to DRAM, NVRAM is also susceptible totransient and hard errors. However, the sources of these errors aredifferent from DRAM. Alpha particles and cosmic rays are less of
an issue with NVRAM. Instead, NVRAM transient errors are largely
caused by resistance drift [76]. In fact, resistance drift can dominate
bit errors in MLC NVRAMs (e.g., MLC PCM [2]). Resistance drift is
a less critical issue with SLC NVRAMs, as it can be addressed byinfrequent scrubbing (e.g., every several seconds) [2, 76]. NVRAM
hard errors are often caused by limited endurance. For example,
PCM cells can wear out after 107 -109 write cycles [42, 91]. Certain
NVRAM technologies, such as certain types of PCM, may havemuch lower transient BER compared to DRAM before the cells
wear out [10, 91].
To mitigate transient and hard errors, previous works propose
techniques to reduce MLC NVRAM BER at low performance andstorage overheads [2, 76, 90]. SLC NVRAM main memory can beeffectively protected using existing ECC mechanisms, as done instate-of-the-art DRAM [10, 76]. In addition to ECC, scrubbing canbe used to address NVRAM transient errors (resistance drift) [2, 76].
To mitigate hard errors caused by endurance issues, previous studies
adopt wear leveling [42, 69, 94], which scatters NVRAM accesses
across the whole device to ensure that all bits in the device wear out
in a balanced manner. Hard errors (bits that are already worn out)
can be remapped [3, 81], which disables the faulty bits, redirecting
accesses to alternative physical locations in the memory.

2.4

Ap

Am

ECC

3D DRAM Last Level Cache

Fraction of
Clean LLC Lines

2.2

hierarchy: the same piece of data can be repeatedly (and unnecessarily) protected multiple times. Figure 2 (a) shows an example. A
clean cache line-A is protected by ECCs in both 3D DRAM LLC andmain memory (represented by Ac and Am , respectively). In fact,
as shown in Figure 2 (b) (our methodology, system, and workload
configurations are described in Section 5), a large portion of the
3D DRAM LLC lines are clean throughout execution time (afterwarm-up) with an identical copy sitting in main memory, and bothcopies are protected with ECCs. In this case, carefully enforcing thereliability of only one of the data copies is sufficient. For instance,
without ECC in LLC, we can simply invalidate a clean LLC line Ac
in case it has a transient error; the error will be naturally recoveredby servicing a cache miss to A with an ECC-protected copy Am
from main memory. As such, the LLC only needs to detect whether
or not Ac has an error. However, it is challenging to fully exploit
the coordination between cache and main memory. For example,
what if cache line Ac is dirty? Once Ac has errors, we lose the newvalue of that cache line because the copy in memory (Am ) does
not have the up-to-date value. Even if Ac is not dirty, Ac with Am
can lead to inconsistent data structures in the program: Ac may
belong to a data structure comprising multiple cache lines. If weload the old value Am to recover the LLC line, the data structure
will become inconsistent: the old value of Am will be inconsistent
with new values of other dirty LLC lines that belong to the samedata structure. Section 3 discusses our solution to the reliability andconsistency problems of both clean and dirty LLC lines.
While there are several prior works that handle some SRAM
cache and DRAM main memory reliability issues together [36, 87,88], these are not desirable for the combination of 3D DRAM and
NVRAM. On the one hand, these works are hard to scale to the
3D DRAM size. On the other hand, because these designs require
frequently clearing dirty cache lines [36, 87] or off-loading SRAM
cache ECC to the main memory [87, 88], these mechanisms further
consume NVRAM’s short endurance and low bandwidth.
Instead of coordinating the reliability schemes of caches andmain memory, persistent memory designs enhance the reliabilityof the main memory when a system completely loses data in caches
due to a system crash or power failure [38, 92]. Persistent memory
allows data in NVRAM to be accessed via load/store instructions liketraditional main memory, yet recoverable across system reboots [12,Issues with Decoupled Reliability

A modern system that employs a large 3D DRAM as LLC and ahigh-capacity NVRAM as main memory typically utilizes two separate uncoordinated reliability schemes for DRAM and NVRAM [3,21, 77, 81]. The key issue with such decoupled reliability across the
LLC and main memory is redundant protection across the memory
3

MICRO-52, October 12–16, 2019, Columbus, OH, USA

Liu, et al.

49]. As shown in Figure 2 (a), persistent memory systems typicallymaintain an additional copy of data (e.g., Ap that can be a log
entry or a checkpoint), which is used to recover original data (Am )in NVRAM main memory if needed after a system crash. This
process can be done through controlled data versioning (e.g., bylogging [84] or shadow paging [12]), or via write-ordering (e.g.,
by cache flushes and memory barriers [12]). However, relying on
persistent memory mechanisms for error recovery purposes has atleast two major disadvantages: first, all these techniques require
code modifications, which require significant software engineering
effort [70]; second, applications with no persistence requirements
suffer from substantial performance and storage overheads in main
memory access [30, 70, 84].
In summary, we need to address the overhead and scalability
issues with (i) the decoupled reliability schemes across caches andmain memory in state-of-the-art memory systems that combine 3D
DRAM and NVRAM, and (ii) the challenges of exploiting persistentmemory techniques in order to efficiently maintain the reliability of memory hierarchy. Our goal in this paper is to design acomprehensive coordinated memory hierarchy reliability schemethat is efficient and effective for memory hierarchies that combine
multiple technologies.

3

3.1

Our coordinated reliability scheme strives to optimize the reliabilityof the 3D DRAM LLC and NVRAM hierarchy as a whole, insteadof separately optimizing the reliability of individual components
at different hierarchy levels, at low performance and storage cost.
Our scheme is enabled by the following key observation:
Observation 1: Errors in the LLC can be corrected by consistent data
copies in NVRAM main memory.
The data stored in the LLC is a subset of the data in main memory.
Therefore, as long as the main memory maintains a consistent
data copy at a known point in time, errors in the corresponding
LLC line can be corrected (or ignored) using the consistent data
copy in memory. This observation makes error correction codes inthe LLC unnecessary. Instead, Binary Star coordinates 3D DRAM
LLC and NVRAM reliability in the following manner: the LLC
adopts only cyclic redundancy check (CRC) codes [39] to ensure
errors can be detected; NVRAM maintains consistent data copies
that can be used to correct (or avoid) the detected errors in the
LLC. While it is straightforward to implement CRC in the LLC,
maintaining consistent data in NVRAM for error recovery purposes
is complicated for two reasons. First, storing and updating multiple
versions of data can incur significant storage and performance
overhead [70, 92]. Second, LLC writebacks can corrupt consistentversions of data in NVRAM: For example, a data structure updatethat spans multiple cache lines can result in multiple dirty cachelines; yet only one or several of the dirty cache lines may be written
back to NVRAM main memory at a given time, leaving the state ofthe data structure in main memory inconsistent.
We need to avoid such inconsistency of data in NVRAM main
memory, in order to use the main memory contents to correct errorsthat happen in LLC lines. To this end, we propose periodic forced
writeback and consistent cache writeback (depicted in Figure 4 (a)and (b), respectively).

BINARY STAR DESIGN

To address the aforementioned reliability challenges, we propose
Binary Star. Binary Star 1) coordinates reliability schemes across
3D DRAM LLC and NVRAM main memory, and 2) uses an efficient
hardware-based 3D DRAM LLC error correction and a softwarebased error recovery mechanism.
3D DRAM LLC and NVRAM technologies allow the memory
hierarchy to continue to scale in a cost-efficient manner. Figure 3
shows our basic architectural configuration. We assume that higherlevel caches are SRAM. The LLC in the processor is a 3D DRAM.
We use SLC PCM, which is a well-understood NVRAM technology,
as a representative for NVRAM main memory to make our designand evaluation concrete. Our design principles can be applied tovarious NVRAM technologies. In this section, we describe our design principles and mechanisms, using Figure 4 to illustrate them.
We leave the description of the implementation of our mechanisms
to Section 4.

Core
L1 Cache

Core

…

Core

L1 Cache…

L1 Cache

Periodic forced writeback. During the execution of an application, portions of updated data might remain in the dirty cache lines,
causing inconsistent data to be present in the NVRAM. Because
dirty cache lines contain all the most recent updates, Binary Star
periodically forces the writeback of dirty cache lines from all cachelevels into the NVRAM main memory (e.g., cache lines a and b in
Figure 4 (a)). These written-back data along with previously writtenback dirty cache lines (e.g, cache line c in the figure.) generate a
set of consistent data, i.e., a checkpoint of data, in the NVRAM by
ensuring that all updates to data are propagated to the NVRAM.
When taking the checkpoint, Binary Star marks all updated NVRAM
blocks as consistent (i.e., the striped squares in Figure 4 (a) turninto solid squares after the checkpoint is taken). To recover to thisconsistent checkpointed state in the presence of LLC errors, Binary
Star maintains a single checkpoint and saves the state of an affected
process to ensure that the process is able to recover to the point
immediately after the periodic forced writeback. The checkpointed
consistent data in NVRAM stays intact in NVRAM until the endof the next forced writeback period. This checkpointed consistent
data in NVRAM can be used to recover from LLC errors detected by
the CRC between two consecutive forced writeback periods. In ourevaluation, we perform detailed sensitivity studies (Section 6.3) and

Shared SRAM Caches
On-chip 3D-stacked DRAM LLC (Gigabytes)

Off-chip NVRAM Main Memory(Hundreds of Gigabytes ~ Terabytes)

On-chip Components

Coordinated Reliability Scheme

Off-chip Components

Figure 3: Basic architecture configuration.
4

Binary StarMICRO-52, October 12–16, 2019, Columbus, OH, USA
Periodic forced
writeback
All SRAM caches
CRC

On-chip 3D DRAM
LLC

Naturalcache writebacks

CRCc

LLC controller reads thedirty bit of the cache line

On-chip 3D DRAM
LLC

cConsistent
NVRAM
block

a’’

a’

a
b

a

a’’

A

A’

b

Inconsistent
NVRAM
block

b

a’

Off-chip NVRAM(a)

LLC CRC detects error in acache line

Consistent cache writebackbaData remapping
via wear leveling

Cache line

Off-chip NVRAM

(b)

N
Error
Correction
LLC controller invalidates
the cache line

Dirty?

Y

Error
RecoveryMemory controller retrieves
the consistent data in NVRAM

Application rolls back to
execute from the lastcheckpoint

Naturally service the
LLC miss, which reloads theline from NVRAM into LLC(c)Figure 4: Binary Star design overview. (a) Periodic forced writeback and (b) Consistent cache writeback of our CoordinatedReliability scheme. (c) Binary Star error correction and error recovery mechanisms. A solid filled square represents a checkpointed consistent data block, while a striped filled square represents a remapped inconsistent data block.
demonstrate that 30 minutes is the best periodic forced writeback
interval for the Binary Star system with our benchmarks.

3.2

Consistent cache writeback. To maintain consistency of thecheckpointed data between two forced writeback periods, we propose
a consistent cache writeback mechanism as shown in Figure 4 (b).
This mechanism redirects natural LLC writebacks to NVRAM locations that are different from the locations of the checkpointed data,so that checkpointed data stays consistent.
Application Transparency. Maintaining data consistency
through persistent memory typically requires support for (i) aprogramming interface (e.g., transactional interface), (ii) multiversioning (e.g., logging or copy-on-write), and (iii) write-ordercontrol (e.g., cache flushes and memory barriers). These can impose
non-trivial performance and implementation overheads on top ofa traditional memory hierarchy [59, 92]. Furthermore, persistentmemory systems need to redirect data upon each persistent data
commit [59, 66, 84]. Our design does not impose large overheads
due to three major reasons. First, the granularity of our consistentcache writeback mechanism is only one cache line. Therefore,
we do not require a transactional interface to the application todetermine the commit granularity. Second, our consistent cache
writeback mechanism is used to update the application’s working
data, rather than the checkpointed consistent data. Therefore, we
do not need to enforce write-order control of consistent cache
writeback. Finally, our design only needs to ensure that natural
LLC writebacks do not overwrite the original, i.e., checkpointed
copy of consistent data, which is reserved to be used to correct
errors in LLC. As such, we do not redirect each natural LLC
writeback to a new NVRAM location. Rather, in our technique, the
LLC writebacks to the same data block can repeatedly overwrite thesame NVRAM block3 that is outside of the checkpointed consistentdata. As shown in Figure 4 (b), the up-to-date version (e.g., cachelines a ′′ and b) in NVRAM is visible to applications, while thecheckpointed consistent version (e.g., the original data a) is hiddenfrom applications.

Our consistent cache writeback mechanism requires redirecting anatural LLC writeback to an NVRAM location that is different fromthe location of the checkpointed data. However, doing so with a
traditional memory system design would introduce extra effort to(i) maintain address remapping with metadata and free data block
management and (ii) manage alternative memory regions to store
redirected natural LLC writebacks. These can impose substantial
performance and NVRAM storage overheads. Instead of explicitly
maintaining data consistency for each cache writeback, Binary Star
leverages the remapping techniques that are already employed byexisting NVRAM wear leveling mechanisms. Our method redirects
the natural LLC writebacks to memory locations different from thecheckpointed data locations during consistent cache writeback. We
make the following key observation:
Observation 2: NVRAM wear leveling naturally redirects and maintains the remapping of data updates to alternative memory locations.

Coordinating Wear Leveling andConsistent Cache Writeback

Note that it is difficult to leverage NVRAM wear leveling with regular persistent data commits on a system with persistent memory.
To provide crash consistency, persistent memory systems typicallyrequire each data commit to be redirected to a new location, e.g.,a new log entry [84], a newly allocated shadow copy [12], a new
checkpoint [70], or a new version managed by hardware [70, 92].
However, NVRAM wear leveling redirects data updates only when
the original physical memory location is repeatedly overwritten
for a given number of times [94]. Therefore, the remapping of datatypically happens infrequently. If we would like to exploit wear
leveling to provide consistency in persistent memory, wear levelingneeds to be synchronized with each persistent data commit. This
likely imposes a substantial performance overhead because it leads
to prohibitively frequent data remapping and metadata updates.
In contrast, Binary Star’s consistent cache writeback mechanism
can effectively take advantage of existing NVRAM wear levelingmechanisms. During the interval between two consecutive occurrences of periodic forced writeback, Binary Star’s consistent cache
writeback mechanism needs to redirect NVRAM updates of eachdata block only once. After the first update to the consistent data
block is redirected due to a natural LLC writeback (e.g., the update ofcache line a ′ , which is overwritten by the later update a ′′ , as shown

3 A block is the minimal granularity of remapping and data consistency ensured in the

NVRAM in our design. The block size is larger than a cache line and depends on the
granularity of wear leveling mechanism used.
5

MICRO-52, October 12–16, 2019, Columbus, OH, USA

Liu, et al.

in Figure 4 (b)), subsequent natural LLC writebacks of the same data
block can repeatedly overwrite the same redirected data location(e.g., the updated version a ′′ , as shown in Figure 4 (b)). Therefore,
Binary Star does not need to synchronize wear leveling with eachconsistent cache writeback. Instead, Binary Star must trigger one
single extra data remapping via the wear leveling mechanism atonly the first update to a consistent data block; wear leveling canoperate as is during the rest of the execution.
By coordinating wear leveling with consistent data writeback,Binary Star enables better utilization of wear leveling. Traditional
NVRAM wear leveling remaps data of frequently accessed NVRAM
blocks to infrequently accessed blocks [94], without being explicitly
aware of the consistency of data. Our design exploits this remapping
capability by exposing data consistency to the wear leveling, suchthat we can guarantee data consistency by using the data remapping capability of wear leveling (as illustrated in Figure 4 (b)). In
particular, we coordinate NVRAM wear leveling with our consistentcache writeback mechanism in the following manner:• Binary Star-triggered wear leveling: When the memory controller
receives an LLC writeback (either a natural LLC writeback or
a periodic forced writeback) to a checkpointed consistent data
block, Binary Star actively triggers wear leveling to remap the
LLC writeback to another NVRAM block. Such remapped updates, along with the original blocks in the current checkpointed
consistent data, form the next checkpoint of consistent data.
As a result, Binary Star ensures that the current version of thecheckpointed consistent data block remains intact.
• Natural wear leveling: During the rest of the time, wear leveling can operate as is. Note that Binary Star does not prevent
natural wear leveling from remapping the checkpointed consistent data blocks. In case a checkpointed consistent data block isremapped, Binary Star simply updates the metadata to reflect
the new location of the block (Section 4).
• NVRAM updates without remapping: NVRAM wear leveling is
performed periodically [94]. During the interval when wear leveling is not performed, natural LLC writebacks to the blocks
outside of the checkpointed consistent data blocks can directly
overwrite the same location.

3.3

is serviced as a natural LLC miss without any hardware modifications. The LLC miss generates an access to the corresponding mostrecently updated data block in the NVRAM. The corresponding
data block could reside in either the checkpointed consistent data
block or the remapped data block.
If the LLC line is dirty, Binary Star determines that it is an uncorrectable error. If such an error is detected, Binary Star triggers error
recovery through the system software by 1) reverting the memory
state to the checkpointed consistent version of application data,
and 2) rolling back the application to execute from the checkpoint.

3.4

Putting it All Together: An Example

With these design components, we demonstrate a simple example
to show the detailed operation of Binary Star when running an
unmodified application. Figure 5 depicts an example of four cases( a to d ) that a running application may encounter when our
proposed mechanisms are employed. These four cases are softwareand hardware operations that conduct consistent cache writeback( a ) and periodic forced writeback ( b ) during normal execution,
as well as handling of correctable errors ( c ) and handling of uncorrectable errors ( d ), when an error is detected.
• Binary Star hardware conducts consistent cache writeback during
normal application execution ( a ). Consistent cache writeback
is transparent and asynchronous to the application. When the
LLC has to evict a cache line, the memory controller identifies if
the cache line belongs to a consistent block ( 1 ). The cache line
is directly written into NVRAM if it belongs to an inconsistent
block, otherwise the wear leveling remaps the consistent block
to a new block and writes the cache line into it ( 2 ). Section 4.2
provides the details.
• Binary Star daemon conducts periodic forced writeback during
normal execution ( b ). The daemon stalls the application ( 1 ),saves the process state, and executes the Binary Star cache line
writeback (bclwb) instruction, which enables the memory controller to write back all dirty cache lines to main memory without
invalidation and mark the corresponding NVRAM data blocks
as consistent ( 2 ). This procedure creates a new checkpoint andinvalidates the previous one. The daemon resumes application
execution once the periodic forced writeback finishes ( 3 ).
• A correctable error ( c ), i.e., an error in a clean LLC line, istransparent to the software. Hardware detects and corrects theseerrors: memory controller detects the error using CRC ( 1 ),reads the dirty bit and identifies that this is a correctable error( 2 ), invalidates the LLC line ( 3 ), and issues a cache miss toretrieve the correct copy of the cache line from NVRAM ( 4 ).
• An uncorrectable error, i.e., an error in a dirty LLC line, requires
software and hardware operations to recover the LLC line ( d ).
LLC controller detects the error using CRC ( 1 ), reads the dirty
bit and identifies that this is an uncorrectable error ( 2 ), andsends a signal to the Binary Star daemon ( 3 ). The daemon stalls
the application and triggers the rollback procedure ( 4 ). The
daemon uses the Binary Star data reset (drst) instruction toreclaim the previously checkpointed consistent data blocks (i.e.,the checkpoint) in NVRAM ( 5 ). The whole procedure rolls thememory hierarchy back to the consistent checkpointed state andresumes normal application execution ( 6 ).

3D DRAM LLC Error Correction and Error
RecoveryOur 3D DRAM LLC uses CRC codes that can only detect errors.
Binary Star recovers from them using the copies of cache linesthat are in the NVRAM main memory. Based on our coordinatedreliability scheme, there can be two types of data copies in NVRAM
for a given block: 1) the checkpointed consistent copy, which is
generated by periodic forced writeback and which contains theconsistent data, and 2) the latest updated copy that is managed by
Binary Star-triggered wear leveling. Once CRC detects an error in agiven 3D DRAM LLC cache line during an access to the 3D DRAM
LLC, Binary Star performs error correction and recovery via thefollowing steps that we also illustrate in Figure 4 (c): First, the LLC
controller reads the dirty bit of the erroneous cache line.
If the LLC line is clean, Binary Star performs error correctionin hardware as follows: 1) the LLC controller invalidates the cacheline, 2) the corresponding LLC access turns into an LLC miss, which
6Binary StarMICRO-52, October 12–16, 2019, Columbus, OH, USA
6 Rollback

Application

Time

1
3
End of periodic forced
writeback interval
On-chip 3D DRAM
Binary Star
CRC
Software(daemon) Periodic forced2
writebackbclwb 2
Process state
MemoryConsistentcontroller
Data

Correctable error

Stallprocess

On-chip 3D DRAM
CRC

Binary Star
Software(daemon)Consistentcache writeback
Consistent
Data

Memorycontroller
12Off-chip NVRAMaNormal runtime operations

Uncorrectableerror detected5
On-chip 3D DRAM
2 3
CRC
Retrieve
Erroneouscache line
cache line 4
1

Binary Star
Software(daemon)Consistent
Data

Memorycontroller

Off-chip NVRAM

Off-chip NVRAM

On-chip 3D DRAM
Binary Star
2
CRC
SoftwareErroneous(daemon)cache line
Mem 3 drst 1
4
Process state
error
MemoryConsistentcontroller
Data
4Off-chip NVRAM

cbd
Error handling operations

Control signal

Datapath

Figure 5: An example of running an application on Binary Star.

Segment… Segment

Segment
numberA

A

1

Free Segment
Consistent Segment
Inconsistent Segment

3D DRAM LLC Modification

A’

B

A

2

A’

B’

3

Consistent Cache
Writeback

Periodic forced
writebackOff-chip NVRAM

We replace the ECC in 3D DRAM LLC with a 32-bit CRC (CRC32) [39] for each 64B LLC line. This design allows Binary Star todetect up to eight-bit errors as well as a portion of nine or more bit
errors, and all burst errors up to 32 bits long [39]. Binary Star also
modifies the finite state machine in the LLC controller to support
the steps required to handle LLC error detection, correction, androllback as listed in Section 3.3.

4.2

Free list

Metadata…

This section discusses our Binary Star implementation, including
modifications we make to the 3D DRAM LLC, the memory controller and the system software.

4.1

Segment number

Free bit
Consistency bit

IMPLEMENTATION

…

4

B

B

4

B’

Figure 6: Modifications to NVRAM wear leveling.
(segment B 3 ) to it, and marking it as an inconsistent segment(segment B’ 4 ).
Modification to NVRAM write control. We modify the memory controller’s wear leveling control logic to enable our consistentcache writeback mechanism (Section 3.2). When the memory controller performs a cache writeback, the wear leveling control logic
reads the consistent bit in the metadata space of NVRAM to determine the consistency state of a segment that the cache line belongs
to. If the cache line belongs to an inconsistent segment, the controller directly writes data into NVRAM. If the cache line belongs
to a consistent segment, the segment is remapped via the wear leveling mechanism. In this case, the controller selects a free segmentfrom the free list. The free list keeps track of all the free segments’
numbers and is stored in the memory controller. The writeback
data, along with the original data in the old segment, is moved to
the new free segment. The memory controller also changes theentry in the mapping table to the new segment [94].

Memory Controller Modifications

Binary Star modifies the state-of-the-art area- and performanceefficient Segment Swap design [94] for NVRAM wear leveling. Segment Swap counts the number of writes to each segment, andperiodically swaps a hot segment with a cold segment.4
Modifications to the wear leveling mechanism. To coordinate
the NVRAM wear leveling mechanism with our consistent cache
writeback mechanism, we modify Segment Swap [94] design asshown in Figure 6. In the original design, the mapping table maps
a “virtual” segment to a “physical” segment. In our design, we
maintain a set of metadata with each segment that includes 1) a
consistent bit that indicates whether the segment belongs to theconsistent checkpoint, 2) a free bit that indicates whether the segment is free, and 3) a segment number that identifies the segmentthat stores the corresponding checkpointed consistent data. The
metadata is stored in a reserved NVRAM space similar to previous
works [69, 94]. During periodic forced writeback, the memory controller uses the segment numbers to release each segment belonging
to the previous checkpointed consistent data (e.g., segment A 1
in Figure 6) by resetting the free bit and adding them to the free list.
After all dirty cache lines are written back, the memory controller
also marks the current inconsistent segment as consistent (segment
A’ 2 ) and deletes the stored segment number. During consistentcache writeback, NVRAM wear leveling performs remapping byselecting a free segment, copying data from the consistent segment

New instructions. To coordinate between the memory controller
and the system software, we add two instructions to 1) support periodic forced writeback, and 2) retrieve and roll back to checkpointed
consistent data.
The first instruction is called the Binary Star cache line writeback
instruction (bclwb). Binary Star leverages the bclwb instruction
to write back all dirty cache lines. Similar to an existing cacheline writeback instruction (clwb) [38], bclwb allows Binary Star towrite back each dirty cache line to NVRAM without invalidating
the cache line. The bclwb instruction not only performs the same
cache line writeback functionality as clwb, but also triggers thememory controller to modify wear leveling metadata to guarantee
consistency. During the execution of a bclwb instruction, after data
writeback, the memory controller marks the current segment as
consistent. The memory controller also tracks the old segment with

4 Segment is the granularity of wear leveling. In this paper, we set the segment size to

be one Mbyte, similarly to the original design [94].

7MICRO-52, October 12–16, 2019, Columbus, OH, USA

Liu, et al.

the segment number, frees it by setting the free bit, and adding itto the free list.
The second instruction is called the Binary Star data reset instruction (drst) and is used to retrieve checkpointed consistent data. drst
has only one operand: memory address. drst is used for handlingerror-triggered rollback. When the memory controller receives adrst command, the memory controller checks the consistent bit of
the associated segment’s memory address to see whether or not thesegment is consistent. The memory controller does nothing if thesegment is consistent. Otherwise, the memory controller finds thecorresponding consistent segment using the stored consistent segment number. The memory controller then invalidates the current
segment by setting the free bit, and adding the segment number tothe free list.

The Binary Star daemon operates in normal mode until either 1)
the periodic forced writeback interval is reached, or 2) a memory error signal is received. To determine if the periodic forced writeback
interval is reached, we use a simple timer, which consumes minimal
CPU resources ( 1 ). When the timer reaches a pre-defined timeinterval, the daemon sends interrupt signals to stall all the running
processes and starts the periodic forced writeback procedure.
The periodic forced writeback procedure consists of five steps.
First, the daemon sends an interrupt to each of the executing processes and stalls them ( 2 ). Second, a copy of each process control
block (PCB) is flushed into the memory, so that the process canbe restored ( 3 ). Each PCB contains process-related state, suchas process ID, stack, and page table. Instead of introducing extra
implementation cost, Binary Star leverages the PCB state to rollback a process to the previously written-back checkpoint in thesame way as existing designs [60]. Third, the daemon flushes all
the dirty caches lines (including SRAM and 3D DRAM caches) tothe memory, by using the bclwb instruction ( 4 ). Fourth, when allcache line writebacks are completed, the daemon resumes all thestalled processes ( 5 ).
The second key task of the Binary Star daemon is to manage
error-triggered rollback. When the daemon receives an unrecoverable memory error signal from the memory controller, it starts
the rollback procedure. First, the daemon sends an interrupt tostall all executing processes ( 6 ). Second, the daemon identifies
the error-affected processes, inspects page tables of each process
and identifies the process that triggered the error ( 7 ). Third, thedaemon retrieves all the PCB data, such that the affected processes’
PCBs are rolled back to the checkpointed PCBs ( 8 ). Fourth, thedaemon retrieves the data of all error-affected processes ( 9 ). The
daemon rolls back memory data by retrieving the old page tablefrom the PCB and using the drst instruction to fetch previously
checkpointed consistent data. The daemon also employs drst to free
all the remapped inconsistent segments to avoid memory leakage.
Finally, the daemon resumes the execution of all the processes ( 5 ).

Reducing latency by controlling NVRAM writebacks. Because the latency of Binary Star-triggered wear leveling can degrade
the performance of a latency-critical application, Binary Star provides two optional optimizations. First, when the NVRAM bus isidle, Binary Star allows the memory controller to issue preemptive
consistent cache writeback to remap a consistent segment in advance. Second, when read requests are delayed by the writebacks,
Binary Star allows the memory controller to postpone LLC writebacks, which has been shown to provide performance benefits in aprevious study [68].
NVRAM over-provisioning. NVRAM preserves a certain amountof physical space specifically for wear leveling purposes [28]. Overprovisioned space guarantees that wear leveling can always find afree segment when the logical space is full. To ensure that Binary
Star-triggered wear leveling always finds a free segment, this overprovisioned space has to be large enough to provide a sufficientnumber of free segments. Our experiments (Section 3.2) show that
10% extra space for over-provisioning is sufficient to accommodate a 30-minute periodic forced writeback interval with all of ourbenchmarks.

4.3

4.4

System Software Modifications

Overhead in 3D DRAM Cache. The CRC-32 code requires 32 bits
per 64-byte cache line, i.e., 6.25% storage overhead, which is 50%
lower than the storage overhead of traditional ECC. The area andperformance overheads of CRC encoding and decoding engines are
similar to the existing ECC encoding and decoding engines.
Overhead in NVRAM. Our modification to NVRAM wear levelingrequires three bytes per segment. With a 512GB NVRAM and a
1MB segment size, this introduces 0.0003% storage overhead. The
free segment list takes 1.1875MB additional space at most. Because
NVRAM typically over-provisions its capacity by roughly 10% tosupport wear leveling [28], our design reuses the over-provisioning
space and, as such, avoids additional storage overhead.
Software overhead. For software support, Binary Star only requires a loadable module to the operating system. An application
requires no modification to utilize Binary Star.
Overhead of periodic forced writeback and uncorrectableerror-triggered rollback. We test periodic forced writeback androllback using several workloads on an emulated NVRAM. Typically, the periodic forced writeback takes 50µs to 3.6 seconds. Because Binary Star rollback reuses data hidden by wear leveling and

We implement software support as part of the operating system toleverage the hardware support.
Binary Star daemon. We develop a Binary Star daemon, which isin charge of periodic forced writeback and rollback. Because theperiodic forced writeback and rollback are infrequent operations,
we assume a single daemon can manage all processes on a Binary
Star enabled machine. Figure 7 shows the state machine of the daemon. Binary Star daemon can be implemented as a kernel loadable
module.

2

Stall allprocesses

Periodic forced
writebackinterval reached1

Memory error
signalNormalapplication
execution

6

7

Stall allprocesses

Identify
erroneous
processes

3

4

5

9

8

Save allprocesses’
PCB

Flush
all cacheswith bclwb

Resumeall theprocesses

Retrievedatawith drst

Retrievethe ckpted
PCBs

Periodic forced writeback

Rollback

Figure 7: State machine of the Binary Star software daemon.

Binary Star Overheads

8

Binary StarMICRO-52, October 12–16, 2019, Columbus, OH, USA
Table 1: Evaluated processor and memory configurations.

avoids large amounts of data movement, the rollback takes only
dozens of microseconds. The Binary Star daemon has negligible
performance impact on the operating system and applications because of the low frequency of periodic forced writeback and thevery small likelihood of error-triggered rollback (See Section 6.3).

5

Processor

EXPERIMENTAL METHODOLOGY

We evaluate both the performance and reliability of Binary Star bycomparing it with four baseline memory configurations: 1) DRAM
memory with 3D DRAM LLC (3D DRAM+DRAM), 2) DRAM memory without 3D DRAM LLC (DRAM), 3) PCM as main memory with
3D DRAM LLC (3D DRAM+PCM), and 4) PCM as main memory
without 3D DRAM LLC (PCM).
For all four memory configurations, we use various state-of-theart ECC mechanisms, which we describe below. With DRAM, weevaluate both 28nm and sub-20nm process technologies. We use
Failures In Time (FIT) as our main reliability metric.

Cores

4 cores, 3.2 GHz, 2 threads/core, 22nm

IL1 Cache

32KB per core, 8-way, 64B cache lines, 1.6ns latency

DL1 Cache

32KB per core, 8-way, 64B cache lines, 1.6ns latency

L2 Cache

32MB shared, 16-way, 64B cache lines, 4.4ns latency

LLC

3D DRAM, 16 GB, CRC-32 (for Binary Star),tCAS-tRCD-tRP: 8-8-8, tRAS-tRC: 26-34 [50, 77]

DRAM

512 GB DDR4-2133 timing, 2 128-bit channels,
ECC schemes: non-ECC, RECC [75], combined RECC
and IECC [8], combined Chipkill and IECC [8, 58]

PCM

512 GB and 52GB over-provisioning space, 36ns rowbuffer hit, 100/300ns read/write row-buffer conflict [42],2 channels, 128 bits per channel, Single error correction,
double error detection (SECDED) + NVRAM remap [81]Main Memory

Performance simulation. We evaluate the performance of Binary
Star modifications using McSimA+ [1], which is a Pin-based [45]cycle-level multi-core simulator. Our infrastructure models the row
buffer as well as row buffer hits and row buffer misses. We use the
FR-FCFS memory scheduling policy [71, 95] for Binary Star and all
our baselines. We configure the simulator to model a multi-coreout-of-order processor with 3D DRAM LLC and NVRAM DIMM
(for Binary Star) as described in Table 1. The higher-level L1 and L2
caches are SRAM-based. We model the 3D DRAM LLC based on the
HMC 2.1 specification [50]. We use PCM timing parameters [42] inour basic configuration for the NVRAM DIMM. We adopt the stateof-the-art NVRAM error protection design [81] in our evaluations.
Reliability simulation and modeling. We evaluate the reliability of our design using a combination of reliability simulation andtheoretical calculation. We employ FaultSim [56], a configurable
memory resilience simulator, to compare the reliability and storage
overhead of various memory technologies and reliability schemes.
To evaluate the error correction and recovery ability of Binary
Star, we calculate the device failure rates using a method proposed
in previous work [8]. A device failure occurs when one or more erroneous bits within any 64-bit line of a ×8 device are uncorrectable
after the corresponding ECC or correction/recovery mechanisms
are applied. We calculate the device failure rate based on the error
correction/recovery capability (the number of bits that can be corrected) of each mechanism given the number and distribution of
single cell errors [8]. We assume that single cell errors are randomly
distributed. We omit column, row, and connection faults for thesame reasons discussed in previous studies [8]. We only consider
errors that are found after device shipment time [8]. We evaluatevarious reliability schemes, including no-ECC, RECC only [75],combined RECC (rank-level ECC) and IECC (in-DRAM ECC) [8],combined Chipkill and IECC [8, 58], and Binary Star (which combines CRC-32 and our error correction and recovery techniques
described in Section 4). For both reliability simulation and theoretical analysis, we adopt PCM and sub-20nm DRAM device reliability
parameters published in previous works [8, 10, 56, 79]. We implement 3D DRAM Chipkill with multiple DRAM cubes. For a faircomparison, we use multiple DRAM cubes across all the reliability
experiments. However, we note that Binary Star is applicable toboth single-cube and multiple-cube 3D DRAM. Because 3D DRAM
does not have ranks, RECC on 3D DRAM is the same as ECC on asingle-cube as in various previous works [31, 57].
Benchmarks. We evaluate our design with various data-intensive
workloads. Redis [7] and Memcached [19] are in-memory key-value
store systems widely used in web applications. Our Redis benchmark has one million keys in total and simulates the case when
Redis is used as an LRU cache; Our Memcached benchmark has
100K ops (read/write), with 5% write/update operations. TPC-C [14]is a popular on-line transaction processing (OLTP) benchmark; werun 400K transactions, mix of reads and updates (40%). YCSB [13]is an open-source specification and program suite for evaluating
retrieval and maintenance capabilities of computer programs; werun eight million transactions, randomly performing inserts, updates, reads, and scans. MySQL [63] is one of the most widelyused relational database management systems (RDBMSs), which
is typically used in online transaction processing; we use a tableof 10 million tuples and conduct complex OLTP operations (usingsysbench [40]), including point, range, update, delete, and insert
queries. We exercise these server workloads using four clients andscale the memory footprint. We also evaluate Binary Star withseveral memory-intensive benchmarks from the PARSEC 3.0 benchmark suite [5], including facesim, freqmine, fluidanimate, vips, andx264. We configure the dataset to be the same as the capacity of ourmain memory configuration (512GB) on all the benchmarks.
We evaluate application performance using instruction throughput (instructions per cycle) for PARSEC 3.0 [5] benchmarks andoperational throughput (the number of executed application-level
operations, e.g., inserting or deleting data structure nodes, per second, not including logging) for all other benchmarks.

6

RESULTSThis section presents the results of our Binary Star evaluation.
Section 6.1 shows Binary Star improves reliability over various
error correction baselines. Section 6.2 shows the performance of Binary Star compared to multiple memory configurations. Section 6.3
presents a study of the impact of Binary Star’s periodic forced
writeback interval on various metrics.
9

MICRO-52, October 12–16, 2019, Columbus, OH, USA

6.1

Liu, et al.
Table 2: Comparison of the evaluated resilience schemes.

Reliability

Figure 8 shows the projected 3D DRAM device failure rates ofdifferent resilience schemes as the single cell bit error rate varies
from 10−14 to 10−4 . The x-axis represents the bit error rate (BER) ofa single cell and the y-axis is the corresponding device failure rate.
We make three major observations. First, for sub-20nm DRAM with
10−5 single cell bit error rate [8], Binary Star reduces the device
failure rate by 1012 times compared to IECC with Chipkill [8] andby 1016 times compared to IECC with RECC [8]. Second, evenwhen the sub-20nm single cell error rate is lower (i.e., 10−6 ), Binary
Star still provides better reliability than other schemes: Binary Star
reduces the device failure rate by 1016 times compared to IECC
with Chipkill, and by 1022 times compared to IECC with RECC.
These results indicate that Binary Star is effective for technologies
that exhibit lower error rates than those projected in [8]. Third,
compared to the commonly-employed RECC DRAM, Binary Star
greatly lowers the device failure rate on both 28nm and sub-20nm
technology nodes (i.e., at 10−10 and 10−15 single cell BERs). We
conclude that Binary Star provides strong error protection on 3D
DRAM LLCs, regardless of technology node and single cell error
rate.
RECC

RECC+IECC

Chipkill+IECC

Binary Star

Device Failure Rate

1.E-7
10-7

1016
1012

Sub-20 nm

28 nm

1.E-2710-27
1.E-3110-31
1.E-3510-35 -14
10
1.E-14-1210
1.E-12-1010
1.E-10

-8

10
1.E-8

Single Cell BER

-6

10
1.E-6

44032-66150

N/A

0%

RECC [72] 28nm DRAM

8806-13230

N/A

12.5%

IECC+RECC [8]sub-20nm 3D DRAM + DRAM 78211-117912

18.75%

18.75%

IECC+Chipkill [8, 58]
sub-20nm 3D DRAM + DRAM

37949-59518

18.75%

18.75%

RECC [72, 81]
sub-20nm 3D DRAM + PCM

6352-9963

12.5%

12.79%

Binary Star
sub-20nm 3D DRAM + PCM

2637-3968

6.25%

12.79%

main
memoryPerformance

-4

10
1.E-4
Normalized Throughput

1.E-23
10-23

No-ECC [79] 28nm DRAM

DRAM
LLC

Figure 9 shows the performance of three baseline memory configurations, all of which have no ECC, and Binary Star normalized
to the baseline with 3D DRAM cache and DRAM main memory.6
We make two observations based on this data. First, Binary Star’s
performance is very close to the baseline with 3D DRAM. Binary
Star performs within 0.2% of a baseline with 3D DRAM and PCM
Binary Star also performs within 1% of the performance of thebaseline with 3D DRAM cache and DRAM main memory. Second,
we observe that using 3D DRAM as the LLC to PCM main memory improves performance by 10% over the PCM-only baseline.
We observe that the performance improvements are significant on
workloads that exhibit high LLC locality (e.g., memcached, facesim,
vips and x264). In summary, we find that Binary Star provides comparable performance to the baseline with 3D DRAM cache and
DRAM main memory but at significantly higher reliability.

1.E-1110-11

1.E-19
10-19

FIT

6.2

1.E-3
10-3

1.E-15
10-15

Storage cost

SystemFigure 8: Device failure rates of 3D DRAM for traditional
RECC [81], RECC with IECC [8], Binary Star, and Chipkill with
IECC [8, 58] vs. single cell bit error rate. Vertical lines represent 3D
DRAM technology nodes (28nm and sub-20nm).5

Table 2 summarizes the FIT and storage cost of five state-of-theart baselines and Binary Star. We show results when the device
single cell BER is 10−6 (left) and 10−5 (right). We make three observations. First, Binary Star provides a clear advantage against
all state-of-the-art baselines. Compared to IECC+Chipkill, whichis the best IECC based scheme, Binary Star reduces FIT by 14.4×.
Compared to the IECC+RECC protected sub-20nm DRAM system,
Binary Star reduces FIT by 29.7×. Second, compared to the NVRAM
based main memory system that uses RECC 3D DRAM cache andPCM, Binary Star experiences 58.5% fewer failures. Third, Binary
Star incurs lower space overhead compared to all other state-ofthe-art designs, with only 6.25% additional space in 3D DRAM andonly 12.79% additional space in PCM, respectively. We conclude
that Binary Star is the most efficient reliability mechanism for hybrid emerging technologies of 3D DRAM cache and NVRAM main
memory.

1

DRAM

PCM

3D DRAM+PCM

Binary Star

0.9
0.8
0.7
0.6

Figure 9: System performance of different memory configurations
with no ECC and Binary Star. We normalize the throughput of eachconfiguration to the throughput of 3D DRAM LLC with DRAM.
Effect of NVRAM wear-out. To understand the effect of NVRAM
wear-out-induced latency on performance, Figure 10 provides theperformance of each workload with different NVRAM access latencies. In this experiment, we vary the latency of PCM to 0.5×,
1×, 2× and 4× of the default PCM latency provided in Table 1. All
throughput values are normalized to the highest performance memory configuration (3D DRAM+DRAM). As Figure 10 shows, Binary
Star retains 99% of the performance of the baseline with a 0.5×
6 We include the CRC encoding and decoding performance overheads based on [8]. We

normalize the throughput of each configuration to the throughput of 3D DRAM LLC
with DRAM. We find that the error detection and recovery overheads are negligible
compared to the CRC encoding and decoding overheads, because errors happen rarely(See Section 6.1).

5 The sub-20nm single bit error rate is based on [8]. The single cell error rates of futuresub-20nm products can vary and depend on the specific technologies.

10

MICRO-52, October 12–16, 2019, Columbus, OH, USA

1
0.8
0.6
0.4
0.2
0

0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x
0.5x
1x
2x
4x

Normalized Throughput

Binary Star

YCSB

TPCC

Redis

Memcached

mysql

facesimfluidanimate

freqmine

vips

x264

Gmean

Figure 10: Performance of Binary Star normalized to the performance of the 3D DRAM+DRAM configuration, as PCM latency is varied as amultiple of its value in Table 1.

PCM latency and its average performance degradation is only 3%
with a 2× PCM access latency. In the unrealistic case where the
PCM access latency is quadrupled (4×), the average performancedegradation of Binary Star is only 5% lower than the baseline.

(a)Normalized
Throughput

1
0.8
0.6
0.4

YCSB
mysql
vips

0.2
0

1

TPCC
facesim
x264

60

300

Redis
fluidanimate
Gmean
600

Memcachedfreqmine

1800

3600

1800

3600

Redis
fluidanimate
Gmean

Memcachedfreqmine

Periodic Forced Writeback Interval (s)Impact of the Periodic Forced Writeback
Interval

Rollback rate

8E-09
8×10-9
6E-09
6×10-9(b)

4E-09
4×10-92E-09
2×10-9Binary Star’s periodic forced writeback interval is critical tothroughput, number of rollbacks triggered by uncorrectable errors,
NVRAM lifetime, and capacity.

00

1

60

300

600

Periodic Forced Writeback Interval (s)Throughput. Figure 11 (a) shows the normalized system throughput of different workloads for six different periodic forced writeback intervals, ranging from one second to one hour. We normalize
the throughput of each workload to the throughput of the same
workload under the 3D DRAM LLC with DRAM configuration. We
make two observations. First, throughput decreases when the interval decreases. As the periodic forced writeback interval decreases,
both periodic forced writeback and consistent cache writeback happen more often and occupy more NVRAM bandwidth, leading to
reduction in throughput. Second, the periodic forced writeback
interval impacts the efficiency of consistent cache writeback. As
the interval decreases, the locality of NVRAM access reduces suchthat Binary Star-triggered wear leveling happens more frequently.
Therefore, Binary Star triggered wear leveling overlaps much lesswith endurance-triggered wear leveling, leading to throughput loss.

(c)

Normalized writes

6.3

16x
14x
12x
10x
8x
6x
4x
2x
0x

YCSB
mysql
vips

TPCC
facesimx264

6.3x

3D DRAM+PCM

2.3x
17.2x

1

60

1.9x

300

600

1.2x

1800

1.1x

3600

Periodic Forced Writeback Interval (s)Figure 11: Effect of varying the periodic forced writeback intervalon (a) throughput, (b) probability of rollbacks (i.e., error rate on adirty line), and (c) NVRAM writes.

likelihood of rollback, but can cause up to 94% faster NVRAM wearout due to the larger number of writes. Second, when the interval
gets larger, the number of NVRAM writes gradually becomes close
to that in the 3D DRAM+PCM baseline. Third, the increase in theinterval length has diminishing benefit. We find that increasing theinterval from 30 minutes to one hour only leads to a 10% reductionin the number of NVRAM writes, but forces applications to rollback consistent data from earlier in the past (up to one hour insteadof up to 30 minutes) when an uncorrectable error happens, leading
to high re-execution overheads.

Rollback Triggered by Uncorrectable Errors. Figure 11 (b)shows the rollback rate for six different intervals. Rollback is triggered by uncorrectable errors that occur when accessing dirty cachelines in 3D DRAM. As such, the results also reflect the error rate on
dirty cache lines. As the forced writeback interval decreases, dirty
cache lines remain in LLC for less time. Therefore, the possibility of
uncorrectable-error-triggered rollback reduces. When the interval
decreases to one second, the likelihood of error-triggered rollbackis < 10−11 . However, the throughput loss is too much in this case,as shown in in Figure 11 (a). At a 1800-second (30-minute) periodic
forced writeback interval, the rollback rate is 10−9 , which is close
to the error rate of current DRAM without ECC.

Lifetime impact of coordinated wear leveling. We use thesame methodology as previous work [90] to evaluate the NVRAM
lifetime impact of our design. We determine the end of NVRAM
lifetime as the time when NVRAM capacity drops below 90% of theoriginal capacity (i.e., when the over-provisioned space is saturated).
We make two observations when the periodic forced writeback interval is 30 minutes (1800 seconds). First, Binary Star triggers 20%
additional NVRAM writes, compared to the 3D DRAM+PCM configuration (Figure 11 (c)). This leads to an 8.4% NVRAM lifetime reduction compared with the baseline wear leveling mechanism [94]. As
the baseline wear leveling mechanism already offers 10× NVRAM
lifetime improvement on native NVRAM without wear leveling [94],our design provides 9.2× better lifetime than native NVRAM, andour coordinated wear leveling mechanism has minimal impact onEndurance. Figure 11 (c) shows the normalized number of NVRAM
writes for six different periodic forced writeback intervals, normalized to the NVRAM writes in the 3D DRAM+PCM baseline foreach benchmark. We make three observations. First, the number ofwrites increases drastically to 17.2× of the PCM-only baseline when
the interval is only one second. Such a short interval leads to a low
11

MICRO-52, October 12–16, 2019, Columbus, OH, USA

Liu, et al.

NVRAM lifetime. Second, using 3D DRAM as LLC significantly prolongs the lifetime, because it reduces writes to NVRAM by 90.1% in
the NVRAM baseline and 87.8% in Binary Star, respectively.

dirty cache lines without periodic cache flush. However, MME
stores error correction codes in main memory, which generates
additional writes to NVRAM upon cache writeback. This is notdesirable for NVRAM as writes reduce both NVRAM performanceand endurance. ECC FIFO [87] stores error correction codes in a
FIFO in main memory. However, ECC FIFO still requires cache
flushes as frequently as [36].

Capacity. One disadvantage of consistent cache writeback on
NVRAM is that Binary Star gradually consumes more space overtime because the consistent cache writeback triggered block remapping consumes free blocks and holds consistent blocks. To free up
space for incoming remapping operations, periodic forced writeback releases space for previous checkpointed consistent data (asdiscussed in Section 4.2). For the above experiments, we guaranteethat all the workloads have sufficient memory space while keeping
10% of the NVRAM space for wear leveling.
Overall, the periodic forced writeback interval length causes
a trade off between performance, reliability, NVRAM endurance
and the effective NVRAM capacity. In this work, we choose 30
minutes (1800s) as the interval, such that periodic forced writeback
only incurs less than 1% average throughput loss, with only a 10−9chance of rollback and 5 to 7.5 FIT per GB of NVRAM.

7

Coordinating reliability in heterogeneous architectures.
Gupta et al. [21] investigate the coordination of reliability in asystem with heterogeneous main memory that consists of a 3D
DRAM and a DIMM-based DRAM. The study focuses on the placement of hot data to improve system performance and reduce theenergy consumption of reliability mechanisms in main memory. In
contrast, Binary Star focuses on providing reliability by coordinating multiple reliability mechanisms across the memory hierarchy.
Persistent memory. Persistent memory combines a fast load/store
based memory interface with data recovery support [12, 30, 38, 49,
59, 66, 70, 84, 92]. Though not explored in previous works, persistent memory may also be leveraged to recover data from memory
errors via multiversioning (e.g., logging and copy-on-write) andwrite-order control (e.g., memory barriers and forced cache writebacks) [59, 61, 66]. However, these operations are costly. Instead,
Binary Star maintains memory system reliability by borrowing only
the notion of persistent data commits from persistent memory techniques. As such, Binary Star is orthogonal to previous proposals
on persistent memory and can be combined with them to further
improve the performance and reliability of persistent memory.

RELATED WORK

To our knowledge, Binary Star is the first memory hierarchy reliability scheme that coordinates the reliability mechanisms across thelast-level cache and nonvolatile memory. In this section, we discuss
related works that provide reliability to different components inthe memory hierarchy.
Optimizing DRAM main memory reliability. Redundancy and
ECC are commonly used to tackle DRAM errors. Redundancy isused to address hard errors by incorporating redundant rows andcolumns. Rows and columns with hard errors are remapped using
the address decoder and/or the operating system page offlining
policy [48, 55]. ECC can be used to recover both transient andhard errors using the parity data generated from the original data.
Bose-Chaudhuri-Hocquenghem (BCH) codes can detect and correct errors [43]. CRC codes can only detect errors but provide nocorrection capability [39]. Single error correction, double error detection (SECDED) is one of the most popular ECC codes used in
ECC DRAM [75], which adds an additional DRAM chip in a DIMM
to store parity bits [23]. Traditional ECC engines in ECC DRAMs
are deployed in a memory controller – we refer to such reliability mechanisms as rank-level ECC, e.g., rank-level SECDED [8].
Server memory systems can also adopt Chipkill to protect against
both memory bit errors and chip errors [17, 22, 25]. One of thecommercial Chipkill methods is Single Device Data Correction(SDDC) [22, 25], which remedies a single DRAM device failure using a combination of CRC and parity check codes. Several previous
works address the storage overhead on traditional ECC [35, 55, 89].
Recent works employ in-DRAM ECC [8, 20, 33, 58], which integrates the ECC engine inside the DRAM chip. Existing LPDDR4
chips have in-DRAM ECC in them [64]. In-DRAM ECC requires
changes inside a DRAM chip, introducing extra area, logic, andmanufacturing costs.
SRAM cache reliability. Kim [36] proposes a design that uses
error detection codes for clean cache lines and error correction
codes for dirty cache lines. However, this requires flushing the entire
cache every 4M cycles, which is unacceptable for a large 3D DRAM.
Memory Mapped ECC (MME) [88] uses ECC codes to protect the8

CONCLUSION

We propose Binary Star, the first coordinated memory hierarchy
reliability scheme that achieves both high reliability and high performance with a 3D DRAM last-level cache and nonvolatile main
memory. Binary Star coordinates 1) the reliability mechanisms
between the last-level cache and main memory, leveraging consistent cache writeback and periodic forced writeback to the main
memory to allow the elimination of ECC in the last-level cache,and 2) the wear leveling scheme in main memory with consistentcache writeback to significantly reduce the performance and storage overhead of consistent cache writeback. As a result, we caneliminate the error correction codes in the last-level cache, whileachieving higher reliability than using costly sophisticated ECC
mechanisms. Our work rethinks the reliability support in the memory hierarchy in light of the next-generation computing systems
that may adopt two key new memory technologies, i.e., 3D DRAM
caches and byte-addressable nonvolatile memories. We hopefully
provide a critical step in reaping the full reliability advantages ofthese emerging technologies beyond simply replacing the existing
memory technologies.

ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable feedback.
This paper is supported in part by NSF grants 1829524, 1829525,
1817077, the SRC System Level Design program, and SRC/DARPA
Center for Research on Intelligent Storage and Processing-inmemory.
12

Binary StarMICRO-52, October 12–16, 2019, Columbus, OH, USA

REFERENCES[33] U. Kang, H.-s. Yu, C. Park, H. Zheng, J. Halbert, K. Bains, S. Jang, and J. S. Choi,“Co-architecting controllers and DRAM to enhance DRAM process scaling,” inThe Memory Forum, 2014.
[34] S. Khan, D. Lee, Y. Kim, A. R. Alameldeen, C. Wilkerson, and O. Mutlu, “The
Efficacy of Error Mitigation Techniques for DRAM Retention Failures: A Comparative Experimental Study,” in SIGMETRICS, 2014.
[35] J. Kim, M. Sullivan, and M. Erez, “Bamboo ECC: Strong, safe, and flexible codes
for reliable computer memory,” in HPCA, 2015.
[36] S. Kim, “Area-efficient error protection for caches,” in DATE, 2006.
[37] Y. Kim, R. Daly, J. Kim, C. Fallin, J. H. Lee, D. Lee, C. Wilkerson, K. Lai, andO. Mutlu, “Flipping Bits in Memory Without Accessing Them: An Experimental
Study of DRAM Disturbance Errors,” in ISCA, 2014.
[38] A. Kolli, J. Rosen, S. Diestelhorst, A. Saidi, S. Pelley, S. Liu, P. M. Chen, and T. F.
Wenisch, “Delegated Persist Ordering,” in MICRO, 2016.
[39] P. Koopman and T. Chakravarty, “Cyclic redundancy code (CRC) polynomial
selection for embedded networks,” in DSN, 2004.
[40] A. Kopytov, “Sysbench manual,” MySQL AB, 2012.
[41] E. Kültürsay, M. Kandemir, A. Sivasubramaniam, and O. Mutlu, “Evaluating
STT-RAM as an energy-efficient main memory alternative,” in ISPASS, 2013.
[42] B. C. Lee, E. Ipek, O. Mutlu, and D. Burger, “Architecting phase change memory
as a scalable DRAM alternative,” in ISCA, 2009.
[43] S. Lin and D. J. Costello, Error control coding. Pearson Education India, 2004.
[44] J. Liu, B. Jaiyen, Y. Kim, C. Wilkerson, and O. Mutlu, “An Experimental Study ofData Retention Behavior in Modern DRAM Devices: Implications for Retention
Time Profiling Mechanisms,” in ISCA, 2013.
[45] C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J.
Reddi, and K. Hazelwood, “Pin: Building customized program analysis tools withdynamic instrumentation,” in PLDI, 2005.
[46] Y. Luo, S. Govindan, B. Sharma, M. Santaniello, J. Meza, A. Kansal, J. Liu,
B. Khessib, K. Vaid, and O. Mutlu, “Characterizing application memory error
vulnerability to optimize datacenter cost via heterogeneous-reliability memory,”in DSN, 2014.
[47] G. Mappouras, A. Vahid, R. Calderbank, D. R. Hower, and D. J. Sorin, “Jenga:
Efficient Fault Tolerance for Stacked DRAM,” in ICCD, 2017.
[48] J. Meza, Q. Wu, S. Kumar, and O. Mutlu, “Revisiting memory errors in large-scale
production data centers: Analysis and modeling of new trends from the field,” in
DSN, 2015.
[49] J. Meza, Y. Luo, S. Khan, J. Zhao, Y. Xie, and O. Mutlu, “A case for efficient
hardware/software cooperative management of storage and memory,” in WEED,
2013.
[50] Micron, “Hybrid memory cube specification 2.1,” 2014, http://http://www.
hybridmemorycube.org/.
[51] Micron, “3D XPoint Technology: Breakthrough Nonvolatile Memory Technology,”
2016,
https://www.micron.com/products/advanced-solutions/
3d-xpoint-technology.
[52] J. Moyer, “Persistent memory in Linux,” in SNIA NVM Summit, 2016.
[53] O. Mutlu, “Memory scaling: A systems architecture perspective,” in IMW, 2013.
[54] O. Mutlu and J. S. Kim, “Rowhammer: A retrospective,” IEEE TCAD, 2019.
[55] P. J. Nair, D.-H. Kim, and M. K. Qureshi, “Archshield: Architectural framework
for assisting DRAM scaling by tolerating high error rates,” in ISCA, 2013.
[56] P. J. Nair, D. A. Roberts, and M. K. Qureshi, “FaultSim: A fast, configurable
memory-reliability simulator for conventional and 3D-Stacked systems,” ACM
TACO, 2015.
[57] P. J. Nair, D. A. Roberts, and M. K. Qureshi, “Citadel: Efficiently protecting stacked
memory from TSV and large granularity failures,” ACM TACO, 2016.
[58] P. J. Nair, V. Sridharan, and M. K. Qureshi, “XED: Exposing on-die error detection
information for strong memory reliability,” in ISCA, 2016.
[59] S. Nalli, S. Haria, M. D. Hill, M. M. Swift, H. Volos, and K. Keeton, “An analysis of
persistent memory use with WHISPER,” in ASPLOS, 2017.
[60] D. Narayanan and O. Hodson, “Whole-system persistence,” in ASPLOS, 2012.
[61] M. A. Ogleari, E. L. Miller, and J. Zhao, “Steal but no force: Efficient Hardwarebased Undo+Redo Logging for Persistent Memory Systems,” in HPCA, 2018.
[62] T. Oh, H. Chung, J. Park, K. Lee, S. Oh, S. Doo, H. Kim, C. Lee, H. Kim, J. Lee,
J. Lee, K. Ha, Y. Choi, Y. Cho, Y. Bae, T. Jang, C. Park, K. Park, S. Jang, and J. S.
Choi, “A 3.2 Gbps/pin 8 Gbit 1.0 V LPDDR4 SDRAM With Integrated ECC Engine
for Sub-1 V DRAM Core Operation,” IEEE Journal of Solid-State Circuits, 2015.
[63] Oracle, “MySQL,” 2019, https://www.mysql.com/.
[64] M. Patel, J. S. Kim, H. Hassan, and O. Mutlu, “Understanding and Modeling
On-Die Error Correction in Modern DRAM: An Experimental Study Using Real
Devices,” in DSN, 2019.
[65] M. Patel, J. S. Kim, and O. Mutlu, “The Reach Profiler (REAPER): Enabling the
Mitigation of DRAM Retention Failures via Profiling at Aggressive Conditions,”in ISCA, 2017.
[66] S. Pelley, P. M. Chen, and T. F. Wenisch, “Memory persistency,” in ISCA, 2014.
[67] M. K. Qureshi, D. Kim, S. Khan, P. J. Nair, and O. Mutlu, “AVATAR: A VariableRetention-Time (VRT) Aware Refresh for DRAM Systems,” in DSN, 2015.
[68] M. K. Qureshi, M. M. Franceschini, and L. A. Lastras-Montano, “Improving read
performance of phase change memories via write cancellation and write pausing,”[1] J. H. Ahn, S. Li, S. O, and N. P. Jouppi, “McSimA+: A manycore simulator withapplication-level+ simulation and detailed microarchitecture modeling,” in ISPASS,
2013.
[2] M. Awasthi, M. Shevgoor, K. Sudan, B. Rajendran, R. Balasubramonian, andV. Srinivasan, “Efficient scrub mechanisms for error-prone emerging memories,”in HPCA, 2012.
[3] R. Azevedo, J. D. Davis, K. Strauss, P. Gopalan, M. Manasse, and S. Yekhanin,“Zombie memory: Extending memory lifetime by reviving dead blocks,” in ISCA,
2013.
[4] J. Barr, “EC2 in-memory processing update: Instances with 4 to 16 TB of memoryand scale-out SAP HANA to 34 TB,” 2017.
[5] C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC benchmark suite: Characterization and architectural implications,” in PACT, 2008.
[6] C. Cagli, “Characterization and modelling of electrode impact in HfO2-based
RRAM,” in Proceedings of the Memory Workshop, 2012.
[7] J. L. Carlson, Redis in action. Manning Publications Co., 2013.
[8] S. Cha, S. O, H. Shin, S. Hwang, K. Park, S. J. Jang, J. S. Choi, G. Y. Jin, Y. H. Son,
H. Cho, J. H. Ahn, and N. S. Kim, “Defect Analysis and Cost-Effective Resilience
Architecture for Future DRAM Devices,” in HPCA, 2017.
[9] H.-M. Chen, C.-J. Wu, T. Mudge, and C. Chakrabarti, “RATT-ECC: Rate Adaptive
Two-Tiered Error Correction Codes for Reliable 3D Die-Stacked Memory,” ACM
TACO, 2016.
[10] W. C. Chien, H. Y. Cheng, M. BrightSky, A. Ray, C. W. Yeh, W. Kim, R. Bruce,
Y. Zhu, H. Y. Ho, H. L. Lung, and C. Lam, “Reliability study of a 128Mb phasechange memory chip implemented with doped Ga-Sb-Ge with extraordinary
thermal stability,” in IEDM, 2016.
[11] N. Christiansen, “Storage class memory support in the Windows OS,” in SNIANVM Summit, 2016.
[12] J. Condit, E. B. Nightingale, C. Frost, E. Ipek, B. Lee, D. Burger, and D. Coetzee,“Better I/O Through Byte-addressable, Persistent Memory,” in SOSP, 2009.
[13] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, “Benchmarking cloud serving systems with YCSB,” in SoCC, 2010.
[14] T. Council, “tpc-c benchmark, revision 5.11,” 2010.
[15] K. Deierling, “Persistent memory over fabric,” in SNIA NVM Summit, 2016.
[16] K. Deierling, “Persistent memory over fabric: let it out of the box!” in Open Server
Summit, 2016.
[17] T. J. Dell, “A white paper on the benefits of chipkill-correct ECC for PC server
main memory,” IBM Microelectronics Division, 1997.
[18] C. Diaconu, “Microsoft SQL Hekaton – Towards Large Scale Use of PM for
In-memory Databases,” in SNIA NVM Summit, 2016.
[19] B. Fitzpatrick, “Distributed caching with memcached,” Linux journal, 2004.
[20] S. Gong, J. Kim, S. Lym, M. Sullivan, H. David, and M. Erez, “DUO: Exposing
On-Chip Redundancy to Rank-Level ECC for High Reliability,” in HPCA, 2018.
[21] M. Gupta, V. Sridharan, D. Roberts, A. Prodromou, A. Venkat, D. Tullsen, and
R. Gupta, “Reliability-aware data placement for heterogeneous memory architecture,” in HPCA, 2018.
[22] HPE, “Memory RAS technologies in HPE ProLiant/Synergy/Blade servers,” 2019,https://psnow.ext.hpe.com/doc?id=4aa4-3490enw.pdf.
[23] M.-Y. Hsiao, “A class of optimal minimum odd-weight-column SEC-DED codes,”IBM Journal of Research and Development, 1970.
[24] A. A. Hwang, I. A. Stefanovici, and B. Schroeder, “Cosmic rays don’t strike twice:
understanding the nature of DRAM errors and the implications for system design,”
in ASPLOS, 2012.
[25] Intel, “Intel Xeon Processor E7 Family: Reliability, Availability, and Serviceability,” 2011, https://www.intel.com/content/dam/www/public/us/en/documents/
white-papers/xeon-e7-family-ras-server-paper.pdf.
[26] Intel, “An intro to MCDRAM (high bandwidth memory) on Knights
Landing,”2016,
https://software.intel.com/en-us/blogs/2016/01/20/
an-intro-to-mcdram-high-bandwidth-memory-on-knights-landing.
[27] Intel, “Intel launches Optane DIMMs up to 512GB: ApachePassishere!”
2018,https://www.anandtech.com/show/12828/
intel-launches-optane-dimms-up-to-512gb-apache-pass-is-here.
[28] Intel, “Over-provisioning nand-based intel SSDs for better endurance,” 2018, https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/
over-provisioning-nand-based-ssds-better-endurance-whitepaper.pdf.
[29] “Data-centric innovation spotlight series: Big memory breakthrough for yourbiggest data challenges,” 2019, https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html.
[30] Intel, “PMDK: Persistent Memory Development Kit,” 2019, https://github.com/
pmem/pmdk/.
[31] H. Jeon, G. H. Loh, and M. Annavaram, “Efficient RAS support for die-stacked
DRAM,” in ITC, 2014.
[32] U. Kang, H. Chung, S. Heo, D. Park, H. Lee, J. H. Kim, S. Ahn, S. Cha, J. Ahn,
D. Kwon, J. Lee, H. Joo, W. Kim, D. H. Jang, N. S. Kim, J. Choi, T. Chung, J. Yoo,
J. S. Choi, C. Kim, and Y. Jun, “8 Gb 3-D DDR3 DRAM Using Through-Silicon-Via
Technology,” IEEE Journal of Solid-State Circuits, 2010.
13

MICRO-52, October 12–16, 2019, Columbus, OH, USA

Liu, et al.

in HPCA, 2010.
[69] M. K. Qureshi, J. Karidis, M. Franceschini, V. Srinivasan, L. Lastras, and B. Abali,“Enhancing lifetime and security of PCM-based main memory with Start-gap
wear leveling,” in MICRO, 2009.
[70] J. Ren, J. Zhao, S. Khan, J. Choi, Y. Wu, and O. Mutlu, “ThyNVM: Enabling
software-transparent crash consistency in persistent memory systems,” in MICRO,
2015.
[71] S. Rixner, W. J. Dally, U. J. Kapasi, P. Mattson, and J. D. Owens, “Memory accessscheduling,” in ISCA, 2000.
[72] D. Roberts and P. Nair, “FAULTSIM: A fast, configurable memory-resiliencesimulator,” in The Memory Forum: In conjunction with ISCA, 2014.
[73] SAP, “SAP HANA: an in-memory, column-oriented, relational database management system,” 2014, http://www.saphana.com/.
[74] S. Schechter, G. H. Loh, K. Strauss, and D. Burger, “Use ECP, not ECC, for hard
failures in resistive memories,” in ISCA, 2010.
[75] B. Schroeder, E. Pinheiro, and W.-D. Weber, “DRAM errors in the wild: a largescale field study,” in SIGMETRICS, 2009.
[76] N. H. Seong, S. Yeo, and H.-H. S. Lee, “Tri-level-cell phase change memory:
Toward an efficient and reliable memory system,” ISCA, 2013.
[77] J. Sim, G. H. Loh, V. Sridharan, and M. O’Connor, “Resilient die-stacked DRAM
caches,” in ISCA, 2013.
[78] V. Sousa, “Phase change materials engineering for RESET current reduction,” in
IMW, 2012.
[79] V. Sridharan, N. DeBardeleben, S. Blanchard, K. B. Ferreira, J. Stearley, J. Shalf,
and S. Gurumurthi, “Memory errors in modern systems: The good, the bad, andthe ugly,” in ASPLOS, 2015.
[80] K. Suzuki and S. Swanson, “The Non-Volatile Memory Technology Database(NVMDB),” 2015, http://nvmdb.ucsd.edu.
[81] S. Swami, P. M. Palangappa, and K. Mohanram, “ECS: Error-correcting strings
for lifetime improvements in nonvolatile memories,” ACM TACO, 2017.
[82] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in CVPR,
2015.
[83] The Apache Software Foundation, “Spark,” 2014, http://spark.incubator.apache.
org/.
[84] H. Volos, A. J. Tack, and M. M. Swift, “Mnemosyne: Lightweight persistentmemory,” in ASPLOS, 2011.
[85] VoltDB, “VoltDB: Smart data fast,” 2014, http://voltdb.com/.
[86] D. S. Yaney, C. Y. Lu, R. A. Kohler, M. J. Kelly, and J. T. Nelson, “A meta-stableleakage phenomenon in DRAM charge storage - Variable hold time,” in IEDM,
1987.
[87] D. H. Yoon and M. Erez, “Flexible cache error protection using an ECC FIFO,” inSC, 2009.
[88] D. H. Yoon and M. Erez, “Memory mapped ECC: low-cost error protection forlast level caches,” in ISCA, 2009.
[89] D. H. Yoon and M. Erez, “Virtualized ECC: Flexible reliability in main memory,”
IEEE micro, 2011.
[90] D. H. Yoon, N. Muralimanohar, J. Chang, P. Ranganathan, N. P. Jouppi, and
M. Erez, “FREE-p: Protecting non-volatile memory against both hard and soft
errors,” in HPCA, 2011.
[91] Z. Zhang, W. Xiao, N. Park, and D. J. Lilja, “Memory module-level testing anderror behaviors for phase change memory,” in ICCD, 2012.
[92] J. Zhao, S. Li, D. H. Yoon, Y. Xie, and N. P. Jouppi, “Kiln: Closing the performancegap between systems with and without persistence support,” in MICRO, 2013.
[93] W. Zhao, E. Belhaire, Q. Mistral, C. Chappert, V. Javerliac, B. Dieny, and E. Nicolle,“Macro-model of spin-transfer torque based magnetic tunnel junction device forhybrid magnetic-CMOS design,” in IEEE International Behavioral Modeling and
Simulation Workshop, 2006.
[94] P. Zhou, B. Zhao, J. Yang, and Y. Zhang, “A durable and energy efficient main
memory using phase change memory technology,” in ISCA, 2009.
[95] W. K. Zuravleff and T. Robinson, “Controller for a Synchronous DRAM That
Maximizes Throughput by Allowing Memory Requests and Commands to Be
Issued Out of Order,” 1997.

14