Grant-­‐in-­‐Aid	
  for	
  Scientific	
  Research	
  S	
  (Kiban	  S)	
  「Fault	  Tolerant	  Infrastructure	  	
  Toward	  Billion	
  of	
  Parallelization	  and	
  Exa-­‐scale	
  Supercomputer」(Adopted	  FY2011)	
  
	
  
1)	
  Motivations	
  	
  
	
  	
  	
  The	
   growing	
   computational	
   power	
   of	
   high	
   performance	
   computing	
   (HPC)	
   systems	
  enables	  increasingly	
  larger	
  simulations.	
  However,	
   as	
   the	   number	
   of	
   system	
   components	
  increase,	
  the	
  overall	  failure	
  rate	  of	
  systems	
  increases.	
  Further,	
  the	
  mean	
  time	  between	
  failures	
  (MTBF)	
  of	
  future	
  systems	
  is	
  projected	
  to	
  be	
  on	
  the	
  order	
  of	
  tens	  of	
  minutes	
  or	
  hours	
  [1,	
  2,	
  3]	
  at	
  exascale.	
  	
  The	
  main	
  objectives	
  are	  to	
  identify	
  bottlenecks	
  in	
  the	
  current	
  design	
   approach	
   from	
   the	   highest	
   level,	
   algorithms,	
   to	
   the	   lowest	
   level,	
   system	
   design,	  which	
   prevents	
   applications	
   to	
   run	   at	
   exascale.	
   Our	   works	
   focus	
   on	
   the	   indirect	
  performance/scalability	
   improvements	
   achievable	
   with	
   fault	
   tolerant	
   techniques	
   and	  an	
  enhanced	
   resilient	
   network	
   layer,	
   rather	
   than	
   the	   direct	
   improvements	
   achievable	
   via	  source	  code	  redesign.	
  	  	
  Checkpointing	
   is	
   one	   of	
   indispensable	
   fault	
   tolerance	
   techniques,	
   commonly	
   used	   by	  HPC	
   applications	   that	
   run	
   continuously	
   for	   hours	
   or	
   days	   at	   a	   time.	
   A	
   checkpoint	
   is	   a	  snapshot	
   of	
   application	
   state	
   that	
   can	   be	
   used	   to	
   restart	
   execution	
   if	   a	   failure	
   occurs.	
  However,	
  when	  checkpointing	
  large-­‐scale	
  systems,	
  tens	  of	
  thousands	
  of	
  compute	
  nodes	  write	
   checkpoints	
   to	   a	   parallel	
   file	   system	
   (PFS)	
   concurrently,	
   and	   the	   low	
   I/O	  throughput	
  becomes	
  a	
  bottleneck.	
  Although	
  simple,	
  this	  straightforward	
  checkpointing	
  scheme	  can	
  impose	
  huge	  overheads	
  on	
  application	
  run	
  times.	
  2)	
  Researches	
  (Our	  way	  of	
  addressing	
  the	
  problem)	
  

	
  	
  Asynchronous	
  Checkpointing	  System:	
  We	
  developed	
  an	
  asynchronous	
  checkpointing	  system	   to	
   minimize	
   checkpointing	
   overhead/workload	
   to	
   PFS	   (Figure	   1).	
   The	  asynchronous	
   checkpointing	
   system	
   solves	
   the	   problem	
   through	   agents	
   running	
   on	  additional	
   nodes	   that	
  asynchronously	
   transfer	
   checkpoints	
  from	  the	
  compute	
  nodes	
  to	
  
the	
  PFS.	
  Our	
  asynchronous	
  checkpointing	
  model	  optimize	
  the	
  checkpointing	
  frequency	  (Figure	   2).	
   Our	   approach	
   has	   two	   key	   advantages.	
   It	
   lowers	
   application	
   checkpoint	
  overhead	   by	
   overlapping	
   computation	
   and	   writing	
   checkpoints	
   to	
   the	   PFS.	
   Also,	
   it	  reduces	
   PFS	   load	   by	
   using	
   fewer	
   concurrent	
   writers	
   and	   moderating	
   the	   rate	   of	
   PFS	   I/O	  operations.	
  	
  Process 1

Compute node
C. Blocking and Non-blocking
Checkpointing Staging node

Process 2
Process 3

Applica'on

kSuccessful
Computation

L2##

	
  	
  

Successful

L2## Computation

1L2##

1

1

11

2
transfer.info

RDMA Buffers

Buﬀer-1

5

RDMA client

3

6
7

8

10

RDMA server

4

Parallel file systems

Process 4L2##	  Figure 5 shows Checkpoint
sthe difference between blocking and nonSCR-library
Level ! kSuccessful
Failuresduringblocking checkpointing. To clarify the differences,
Level-k
Datawewritecharacthreads
Process 1
computation or
recoveryLevel < kcheckpointing
terize both schemes with two metrics, checkpoint overhead andProcess 2
Failures duringcomputation
orProcess 3
checkpoint latency. Checkpoint overhead (C) is the increased
checkpointingk
Process 4Level ! k
Level
<
k
RDMA-Read- of checkpointing.
execution time of an application because
Failures during
Failures during
L2#via#staging#nodes##recovery
Staging process
recovery
Staging server
Checkpoint latency (L)isclient
the time to complete a checkpoint.
Staging
During blocking checkpointing, each process writes its
tatbtFigure	
  
	
  :	
  	
  Asynchronous	  Checkpointing	
  Figure	
  2	
  :	
  	
  The	
  basic	
  structure	
  of	
  the	
  c
4: 1
Non-blocking
checkpointing
client/serverusing RDMA
checkpoint dataclient/server	
  toFig.thePFS, uandblocksuntil
thecheckpointsing	
  RDMA	  asynchronous	
  checkpointing	
  model	  operation completes. Thus, the checkpoint overhead is identi- Fig. 6: The basic structure of the non-blocking checkpointing model
L1

L1

L1

9

L1

Time line: Application computation: L1 checkpoint: L2 checkpoint initialization: L2 checkpoint to PFS

	
  
	
  
	
  
	
  
Exploration	
   of	
   Multi-­‐tier	
   Storage	
   Design	
   with	   Burst	
   Buffer:	
   We	
   explored	
   multi-­‐tier	
  storage	
   design	
   for	   resilient	
   architecture	
   using	
   burst	
   buffers.	
   Burst	
   buffers	
   have	
   been	
  proposed	
   to	   alleviate	
   the	   problems	
   of	
   I/O	
   operations	
   to	   a	   shared	
   PFS	   [4,	
   5].	
   A	
   Burst	
  buffer	  is	
  a	
  new	
  tier	  in	
  the	
  storage	
  hierarchy	
  to	
  fill	  the	  performance	
  gap	
  between	
  node-­‐local	
   storage	
   and	   the	   PFS,	
   and	   is	   shared	
   by	   a	   subset	
   of	
   compute	
   nodes.	
   The	   new	
   tire	
   can	  absorb	  the	
  bursty	
  I/O	
  requests	
  from	  applications	  and	
  thus	  can	
  reduce	  the	
  effective	
  load	  on	
   the	   PFS.	
   We	
   considered	
   using	
   burst	
   buffers	
   from	
   different	
   viewpoint,	
   and	   tried	
   to	
  improve	
   system	
   resiliency	
   with	   burst	
   buffer	
   storage	
   design.	
   With	   burst	
   buffers,	
   an	
  application	
  can	
  store	
  checkpoints	
  on	
  a	
  smaller	
  number	
  of	
  dedicated	
  burst	  buffer	
  nodes,	
  so	
  the	
  probability	
  of	
  lost	  checkpoints	
  is	
  decreased.	
  We	
  explored	
  how	
  burst	  buffers	
  can	
  improve	
  efficiency	
  and	  resiliency	
  compared	
  to	
  using	  node-­‐local	
  storage	
  instead	
  of	
  burst	  buffers	
   based	
   on	   a	   performance	   model	
   combined	
   with	   our	   multi-­‐level	   asynchronous	
  checkpoint/restart	
  model.	
  Compute node 1

Compute node 2

Compute node 3

Compute node 4

IBIOclientIBIO
clientIBIO
clientIBIO
client1!

5! chunk
2!

Compute node 1

Compute node 2

Compute node 3

Compute node 4

IBIOclientIBIO
clientIBIO
clientIBIO
client1

and 4 for node 3) unlike a flat
buffersystem. This capability
1!
5!
IBIO server C/R.
threadIBIO server
threadaccelerates
the partial restart of uncoordinated
4!

3!

!!!

!!
!!

fd1

Writer thread

file1

fd2

Writer thread

file2

fd3

Writer thread

file3

fd4

Writer thread

file4

file4

Writer threads

Storage

Storage

Chunk buffers

file1

TABLE I

2!

file2

4!

Hi-1

3!

fd1

Reader thread

fd2

Reader thread

Compute(
node(

Hi

fd3

fd4

Hi-1
Si

N ODE SPECIFICATION Reader thread
file3

mi

2

Hi-1

Reader thread

i =0

Chunk buffers

Reader threads

i >0

CPU
Intel Core i7-3770K CPU (3.50 GHz x 4 cores)
Memory(16
GB)
Fig. 3. IBIO Write: four IBIO clients andone IBIO server Cetus DDR3-1600
Fig. 4. IBIO
Read:
four IBIO clients and one IBIO server
Figure	
  3	
  :	
  	
  IBIO	
  Write/Read	
  :	
  fGIGABYTE
our	
  IBIO	
  GA-Z77X-UD5H
clients	
  and	
  one	
  IBIO	  Fig.
Recursive structured
storagemodel
Figure	
  4
	
  :	
  	
  5.Recursive	  structured	
  storage	  M/B
SSD
Crucialm4
CT256M4SSD3(256GB, mSATA)
server	  model	
  represented
as H2 {1, 4}. It has 2 levels of storage: the nodethe
encoding
overheadandlatency.
Wedifferentiate
between(Peak read: 500 MB/s, Peak write: 260 MB/s)
local storage is not shared, so m1 = 1; however, the PFS checkpoint overhead and latency to show the differences beSATA converter
KOUTECH IO-ASS110 mSATA to 2.5’ SATA
is shared across all compute nodes, so m2 = 4. In the tween synchronous and asynchronous checkpointing. Duringcommunications from the client to the server using a lowDevice
Converterwith Metalcheckpoint
Frame overhead and latencylevel, user-space InfiniBand API called ibverbs. Once the IBIO
same manner, the burst buffer system in Figure 2 (b) can besynchronous
checkpointing,
RAID
Card nodes
Adaptec
ASR-7805Q
represented as H2 {2, 2}. The total numberof computeare equal,i.e., Oi = LSingle
i , because each process is blocked until

	
  

Q2

server thread receives a write request from a client, the IBIO

i=1 mi = 4 nodes.

the checkpoint is completed. Asynchronous checkpointing,meanwhile, incurs only initialization overhead, so checkpointserver reads the chunk into the selected buffer according tooverhead is equal or smaller than checkpoint latency, i.e.,the fd by using an RDMA read (Step 2). Then, the IBIO
T IER i STORAGE (Si ) PERFORMANCEPARAMETERS
To build a reliable Oburst
buffer system at extreme scale,i < Li .
server creates a writer thread to asynchronously write theWesystemmodel leveli checkpoint overhead
and latencyasminimizing
thenumberofcomponents
is criticalwhileri
Sequential read throughput from compute nodes (Hi=0 )⇢Ci +under
Ei (synchronous
checkpointing)wiSequential write throughput fromcompute nodes (Hhighchunk to the file (Step 3) so that the IBIO server thread can
i=0 )maximizing
I/O
throughput
minimalbudget.
ToO=imi
The number of a upper hierarchical entities (Hi 1 ) sharing Si
Ii(asynchronous checkpointing)can be calculated as

FMI	
  (Fault	
  Tolerant	
  Messaging	  Interface):	
  We	
  proposed	
  FMI	
  for	
  fast	  and	  transparent	
  recovery.	
  FMI	  is	
  a	
  survivable	
  messaging	
  interface	  that	  uses	  fast,	  transparent	
  in-­‐memory	
  checkpoint/restart	  and	
   dynamic	
  node	
   allocation.	
   With	   FMI,	
   a	
   developer	
   writes	
   an	
  explore the bandwidth we canachieve by a single node with receive subsequent chunks from IBIO clients while writing
L =C +E
chunk. When
all chunks for(MPI).	
  the write callare written,
application	  using	  semantics	
  similar	  to	
   Message	
  Passing	
  Interface	
  
The	
  
FMI	
   theIn thismodel, we do not distinguishbetweennode-local
only commodity devices,
we developed an mSATA-based
SSD thewhere C denotes actual checkpointing time, E denotes
storage and network-attached storage. Instead, we differentiate
test
system.
The
detailed
specification
is
shown
in
Table
encoding
time, and I denotes
initialization
time forI.asyn-writer threads inform the IBIO server (Step 4), and the IBIO
the storagelevels using performanceparameters.
We consider
runtime	  ensures	
  
that	
  
the	
  chronous
application	
  runs	
   through	
   failures	   by	
   handling	
   the	   activities	
  checkpointing. If the level i checkpointing does notserver informs the IBIO client that the write is complete (Step
Thebecause
theoretical
only sequential read/write bandwidth
typically peakthe I/O throughputs of sequential read and write
encode checkpoints, E becomes 0; otherwise we model thepatternneeded	
  for	
  fault	
  tolerance,	
  such	  as	
  checkpointing,	
  failure	
  detections,	
  and	  recoveries.	
  All	
  of C/R is sequential. Note that the read and write
operation of the mSATA-basedisD/e
500whereMB/sec260 size5).
encoding timeSSD
as E =
D is theandcheckpoint
bandwidth values are not the peak performance of the storage
percomputenode andthee is eight
encodingSSDs
throughput.
Theaactual Since communications from server to client are required
MB/sec,respectively.
Weaggregate
into
but theof	
  effective
throughput
betweencompute
nodes
and
the
this	
   motivates	
  
the	
  
need	
  
for	
   a	
  
messaging	
  runtime	
   system.	
   Such	
   a	
   system	
  is calculated
checkpointing
timesurvivable	  C , i.e., sequential write time,storage location. For example,RAID
if tier i card,storage andhas a connect
read
for IBIO read operations, we extended the RDMA transfer
as the RAID cards via PCIe 3.0 (x8).
8bandwidthof r, but the network-based file system delivers
should	  be	
  able	  to	
  maintain	
  processes	
  and	  connections	
  that	  are	
  unaffected	
  by	
  the	
  failure	
  API from prior work to support bi-directional communications.
(i = N ) is 4>
ThemodeltheoreticalofM/w
this configuration< D⇥
a bandwidth of r̂ < r, then theparameter is peak
set as performance=l write min total. To use
When
IBIO client reads a file from remote storage via
r = r̂. Using these performanceGB/sec
parameters,for
we estimate
C/R 2.08 C
read
andGB/secfor>while	  starting	
  
a
nd	
  
i
ntegrating	  replacement	  processes	  as	
  naneeded.	
  
:
D
⇥
/w
(otherwise)time. However, as we show in Section VI, We can minimize
IBIO read (Figure 4), the IBIO client sends the read request
TABLE II

i

i

i

i

i

i

i

i

i

i

i

i

i

i

QN M

the performance gap between thelocal storage
and remotesystems
read/write as burst buffers, the mSTA-based SSDs
where M denotesQthe total number of checkpointing computecontaining the fd to the IBIO server (Step 1). Once theaccesses with IBIO.
N
must be accessed via anodes,
high-speed(e.g., InfiniBand)i.e., M = network
i=1 mi . With uncoordinated checkpointing,
B. Modeling of C/R Strategies with a network-based weFMI
rank(virtualrank)IBIO server receives the read request, the IBIO server thread
the checkpointing
timesimple
is identicalto coordinatedfileassume
system.
However,
methods
synchronizationidentifies the file according to the fd, and it creates a reader
Given the storage performance parameters of each tier, we checkpointing time because of indirect globalQ
cannotexploitthebandwidth.
For
example,
if
we
use
NFS
with
N
model level i checkpoint overhead (Oi ), checkpoint latency as described in Section II-C. Because
k=i+1
mthread to handle the read request (Step 2). Then the IBIOl mk is thefor thenetwork-based
filestorage
system,theSusefulbandwidth(Li ), and restart overhead (Ri )IPoIB
in a multilevel
checkpointing
QN M
number oflocations
i , the quantity
k=i+1 mk
library [4]. For simplicity, if multiple
computenodes
conUser’sview
is only 1 GB/s for bothrepresents
read and(details
in Section
VI).
the write
max numberof computenodes perstorageserver thread waits for the next request. The reader thread
currently access a single storage location, we assume the location S .
FMIi
FMI’s view file system
reads a chunk of the file with file descriptor fd into its buffer
A newisrequiredto
exploittheread/write throughput scales down
linearlynetwork-basedwith the number
When restarting with uncoordinated checkpointing, theof concurrent accesses.
PCIe-attached high bandwidthstorage.
restart overhead
is different from coordinated checkpointing.(Step 3). The reader thread requests that the IBIO server thread
Checkpoint overhead Oi is the increased execution time of We model the restart overhead R , i.e., sequential read time,theto the Join
IBIO client (Step 4), and then reads
P9
P6 sends
P8 chunk
Leave
P0
P2
P7
P1
P3 i
P4
P5
an application because of checkpointing. Checkpoint
latency as:
8- BASED F ILE S YSTEM FOR
the nextNodechunk.
When all chunks for the read call have been
- LEVELLi and restart overhead Ri are theIV.
timesU
toSER
completea check-I NFINI BAND
4
(i
= N2)
Node 0 (failed)1
Node
Node 3> D ⇥ K/rNodei<
point and restart respectively. If a checkpoint strategy conducts B URST B UFFERS
read, the reader thread informs the IBIO server, and the IBIORi =lmerasure encoding, such as XOR [7] and Reed-Solomon en>: D ⇥ QN K
/ri (otherwise)
server informs the IBIO client that the read is complete (Stepcoding [8], the checkpoint overhead and latency also includem
k=i+1 mk

0

1

2

i

3

4

5

6

7

	
  

1: int main(int *argc,
2:
FMI_Init(&argc, &3:
while ((n = FMI_L
4:/*user program
5:}
6:
FMI_Finalize();
7: }To exploit the bandwidth of burst buffers, we developed
5).
Fig. 3: FMa user-level InfiniBand-based file systemandAPI called
Fig.
of oFMI
Figure	  52:	
  :I/O	  	
  Overview
Overview	
  f	
  FMI	  IBIO. Our earlier work showed that I/O operations with
V. M ODELING
concurrent multiple threads can effectively exploit the highAs described in Sections II and III, each checkpoint
P0
P1 strategy
P2
P3
P4
P5
bandwidth of PCIe-attached storage [23] as well as the PFS and storage architecture have advantages and disadvantages.
FMI_Init(…)[4]. Thus, we parallelize the operations of IBIO with multiple Here we discuss the model we developed to identify the best
reader andthreads.estimating
The current API
includesis writercorrectly
theof IBIO
numberof sparenodes
byconfiguration.
checkpoint
strategyneeded
for a given
4
5
0
2
1
3
open, write, read and close. The interfaces are identical
a
job,
which
requiresknowledgeofthefailurecharacteristics
0=
FMI_Loop(…):
checkpoin
to POSIX except that of open. The IBIO open requires a A. Recursive Structured Storage Model
ofas themachineof the We
application.
Anotherhostnamewell asa pathname andso thatthe
IBIO behavior
clients can accessintroduce a recursive
structured storage model to gener4
5
0
2
1
3
any files on any IBIO servers. IBIO open sends a query to the alize storage architectures to describe both flat and burst buffer
solution is to request compute nodes from the resource mank=i+1

kFail-­‐in-­‐Place	
  Large	  scale	
  Network	
  Design:	
  As	
  a	
  first	
  stage,	
  we	
  built	
  a	
  tool	  chain	
  (Figure	  6),	
  which	  allows	
  us	
  to	
  simulate	
  the	  performance	
  of	
  different	
  routing	  algorithms	
  on	
  state-­‐of-­‐the-­‐art	
  network	
  topologies.	
  This	  was	
  combined	
  with	  an	
  injection	
  of	
  network	
  failures	
  to	
  make	  design	
  decisions	
  for	  a	
  future	
  fail-­‐in-­‐place	
  network.	
  Fail-­‐in-­‐place	
  networks	
  will	  enable	   high	
   communication	
   performance,	
   a	
   crucial	
   component	
   for	   application	  scalability,	
   while	
   being	
   extremely	
   resilient	
   for	   non-­‐critical	
   network	
   failures,	
   which	
  increases	
   the	   system	
   availability	
   for	   scientific	
   simulations	
   and	   decreases	
   the	   need	   for	  application	
  level	  resiliency.	
  -level Lossy Compression for Fast Checkpoint/Restart	
  ki, Kento Sato, ToshioFigure	
  Endo6	  :	
  and
SatoshiMatsuoka
Toolchain	
  to	
  evaluate	
  the	
  network	
  throughput	
  of	
  a	
  fail-­‐in-­‐place	
  network	  p. of Mathematical and Computing Science	
  Tokyo Institute of Technology
Lossy	
  compression	
  for	
  fast	  checkpint/restart	
  :	
  To	
  reduce	  checkpoint	
  and	  restart	
  2-12-1-W8-33,
Ohokayama,
time,	  we	
  e
xplored	  application-­‐level	
  lossy	
  compression	
  approach	
  based	  a	
  wavelat	  Meguro-ku, Tokyo 152-8552 Japantransformation	
  [6]	  .	
  	
  Although	  lossy	  compression	
  can	
  introduce	
  errors	
  to	
  the	
  simulation	
  h.ac.jp, kent@matsulab.is.titech.ac.jp,{endo, matsu}@is.titech.ac.jp
data	
  after	
  a	
  restart,	
  applications	
  can	
  proceed	
  and	
  produce	
  approximate	
  results	  even	  with	
  high	  failure	
  rate	  (Figure	
  7).	
  wavelettransformation
encoding
High Performancequantization
L
H
L
H’’
L
H’
puters is growing
entific simulations.
an also increase as
nt/Restart is one oflossy
entific applications
sort
transformation
kpoint and restartad due to the highd restart time, we	  based on a wavelet
Figure	  7	
  :	
  Application-­‐level	
  lossy	  compression	
  using	  a	
  wavelet	
  transformation	  how that our lossy
Figure 1. Application-level lossy compression using a wavelat transformulation data of amation	  09% of an average3)	
  Results	
  (what	  did	  we	
  learn,	
  what	  did	
  you	
  produce?)	
  	
  
	
  	
  Asynchronous	
   Checkpointing	
   System:	
   Our	   asynchronous	
   (or	   Non-­‐blocking)	
  Restart; CompresApplica'on*data*(double)

(double)(double)(double)(double)(double)(double)Sort*&**
Lossy*transforma'on*

with system	
  high failure
rate.with	  Our preliminary
show that ourtechnique	
  checkpointing	
  coupled	  a	
   multi-­‐level	  studiescheckpoint/restart	  maintains	
  a	
  given	
  application	
  efficiency	
  with	  significantly	
  lower	  PFS	
  requirements	
  than	  lossy compression approach can reduce size of simulation
simple	  synchronous	
  (or	
  Blocking)	
  checkpointing,	
  which	  write	  checkpoints	
  such	  that	  all	  data of a real climate application by 86-87% with 0.09% ofprocesses	
   write	   their	
   own	
   checkpoints	
   concurrently,	
   and	   are	   blocked	
   until	
   the	  an average error.
ormance Computcheckpoint	
   operation	
   completes.	
   	
   Especially,	
   our	   results	
   show	   that	
   combining	
  asynchronous	
   and	   multi-­‐level	   checkpointing	
   results	
   in	
   highly	
   efficient	
   application	
   runs	  is growing expoII. A PPLICATION - LEVEL L OSSY C OMPRESSION
c simulations.with	
  Thelow	
  PFS	
  bandwidth	
  requirements.	
  s to conduct finhe overall systemm size grows, andof future exascale
an order of hours.

To reduce checkpoint and restart time, we explore
application-level lossy compression approach based a wavelat transformation. In our approach, we apply three-step
lossy compression to only arrays of simulation data, such aspressures, temperatures, and velocities, to avoid applications

50"

0.1"

0"

2"

4"

8"

16"

32"

64"

0"

128"

0"

XOR#group#size#

Fig. 14: Effi

Fig. 12: XOR encoding performance per node

100"
90"
80"
70"
60"
50"
40"
30"
20"
10"
0"

	
  

we
and 8.32
8.32 GB/s
GB/sforforwritewrite
we useuse 16
16 GB/s
GB/s forfor read throughput,
throughput, 9and	
  throughput
system.
throughput inin thethe burst buffer system.
we useuse 16
16 nodesnodes forforthetheFor uncoordinateduncoordinated checkpointing,checkpointing, we	
  For
clustershowed thatthat thetheoptimal
optimalcluster
clusterclustersize
size (K).
(K). Earlier
Earlier studies showed	  sizeisisfrom
from 32
32 to
to 128 processes, i.e.,
size
i.e., 44 to
to 16
16 nodesnodesforforaa8-core
8-core	
  Coastalcompute node, to provide
Coastalcomputeprovide aa good
good trade-off
trade-off between
betweenthesize of the clusters and the amounttheamount ofof messages
messagestotolog
logforfor	
   size of the clusters
most applications
applications [11], [24]. Because
mostBecause thethe cluster
cluster size
sizeisissmall
small
tomassign
enoughtoto aa single
single burst
burstbuffer
buffernode,
node,lmassign a compute node tolenoughK
Q2K
is computed as 1 compute
Q
compute node
node forfor uncoordinateduncoordinated2
mkk is computed
k=2m
k=2
restart.
restart.
We useuse asynchronous
asynchronous checkpointingWecheckpointing forfor PFS,PFS, andand synsyn-

Efficiency
Efficiency

Eﬃciency"

Efficiency
Efficiency

Read/Write throughput (GB/seconds)
Read/Write throughput (GB/seconds)

Required(PFS(throughput((GB/sec)(level 1" checkpointing system and optimized the checkpoint
0.9"
no checkpoint
frequency
based on collected failure rates and checkpointing
Non/Blocking(80%)"0.8" We extend their model in this work. Bautista-Gomez etcosts.
time computed
Non/Blocking(70%)"
PFS"cost"x1"/"NonAblocking"0.7" proposed multi-level checkpointing using local
al. [5]
SSDs
Blocking"(90%)"
and the existin
PFS"cost"x1"/"Blocking"Blocking"(80%)"
and 0.6"
a PFS. They use Reed-Solomon (RS) encoding for highly
PFS"cost"x2"/"NonAblocking"compute effici
Blocing(70%)"
0.5"
resilientcached checkpoints to reduce PFS usage. Generally,
PFS"cost"x2"/"Blocking"Level 2 check
PFS0.4"usage is costly when compared to local storage,
andPFS"cost"x10"/"NonAblocking"the PFSis accessed less often in multi-level checkpointing.
0.3"
given failure r
PFS"cost"x10"/"Blocking"
However,increasing failure rates require checkpoints to a PFS
0.2"
maximal efficie
more0.1"frequently. Thus, even with multi-level checkpointing,higher efficien
checkpointing
to a PFS is crucial for future systems.
0"
0"
2"
4"
6"
8"
10"
12"
14"
16"
gap become m
Failure"rate"x1""
Failure"rate"x2"
Failure"rate"x10"Asynchronous
I/O haslong been used
to hide I/O bottleScale(factor(of(failure(rate((
higher checkpo
necks [11], [15]–[17]. These techniques enable applications to
Figure	
  
9
	
  
:
	
  
	
  
E
fficiency	
  
o
f	
  
s
ynchronous	
  
a
nd	
  
a
synchronous	
  parallelize
I/Owith
computation
toincreaseCPU
utilization
on a blocking c
Figure	  	
  :	
  Required	  PFS	
  throughput	  at	
   ratesFig. 13: Efficiency of blocking and non-blocking checkpointing
Fig. 15: 8
Required
PFS throughput
at different failure
andto enhance I/O performance. Patrick et al. [15] precheckpointing	
  different	
  failure	
  rates	  during the PFS
sented a comprehensive study of different techniques of overwhile a non-blo
lapping I/O, communication, and computation, and showedin 	
  future	
   systems. As for different systems, we observed that XOR
checkpoint. Fu
the performanceof asynchronous
I/O. Nawabcheckpoint,benefits
SCR computes
the parityof eachet block as
three clusters at LLNL show similar failure rates [4], whereis identical toal.
[16]
asynchronouslytransfer
multiple
stripedTCP
data
B
system failures can be recovered from level 1 checkpoints in RAID-5 [8], [9], which creates S = B + N −1 bytes of
streams
to increase
I/O
performance
in k
Grid
environments.
Exploration	
  
o
f	
  
M
ulti-­‐tier	  Storage	  Design	
  
w
ith	
  
B
urst	
  
B
uffer:	
  
T
he	
  
ey	
  
c
ontributions	
  
o
f	
  
application run
and frequent level 2 checkpoints are not required. Thus, non- encoded checkpoint data from B bytes of original checkpoint
An asynchronous staging service using RDMA proposed bythan with nonthis	  work	  include	
  an	
  InfiniBand-­‐based	
  file	  system	
  (IBIO)	
  that	  exploits	  the	
  bandwidth	
  of	
  blocking
checkpointing would benefit other systems.
data
withinof research
an XOR
group.
Hasan
et al. N[11]members
is the closest
to ours.
The Since
authorsencoding
Since staging
achieved
high I/O
throughput
additional
nodes. As size, theC. burst	
  buffers,	
  and	
  exploration	
  showing	
  how	
  system	
  resiliency	
  improves	
  from	  the	
  use	  of	
  Building an Efficient and Resilient Systemtimeincreaselinearlywith bytheusing
encodedcheckpoint
dentlyof comp
we observed,
optimizing
requiresdeterminationlarge
XOR groupsize, performance
N S, saturates
the
XOR encoding rate.
Whenbuilding
a reliableand	
  data centeror supercomputer, twoburst	  buffers,	
  uncoordinated	
  checkpinting.	
  Especially,	
  these	
  contributions	
  
can	
  
of
the
propernumberofstaging
nodes
for
a
given
numberrate
without
redmajor concerns are cost of the PFS and the PFS throughput As Section III showed, most failures affect just one node.
benefit	
  system	
  designers	
  in	
  making	
  the	
  tof
rade-­‐offs	
  in	
  pHowever,
erformance	
  of	
  components	
  compute nodes.
the comprehensive
study on the so	
   that	
   we could readrequired to maintain high efficiency. Generally, we want to Thus,
weisuseXOR
onlytheir
toscale	  handlefailurecatnot shown
do theypresent
solution.
Tothey	
  cost,can	
  butcreate	  efficient	
  and	
  Using
cost-­‐effective	  machines	
  
for	
  nor
fcheckpoints
uture	
  extreme	  systems.	
  minimize
not sacrifice
performance.
our model, problem
maximum band
egory
1
in
Table
I,
and
we
handletheotherfailurecategories
we can predict the required PFS bandwidth to achieve high deal with bursty I/O requests, Liu et al. [17] proposed a storage
buffers onThus,
I/O nodes.
Thethe XOR 90% efficiency
k system= 2, 3,design
4 . . . 5thatbyintegrates
a PFS SSD
checkpoint.
we setsystem	
   efficiency when using our checkpointing system.
system achieved
bandwidth.rate,
As Figure14
Because a n
Figure 15 shows the PFS bandwidth required to maintain encoding
rate ashightheaggregate
saturatedI/Omaximal
400MB/s.
plication comp
90%, 80%, and 70% efficiency under increasing failure rates. showed, if we apply asynchronous I/O to checkpointing, we
considerthe interferencewith the runningapplication,
CoordinatedFlat Buffer
UncoordinatedFlat Buffer
B.mustEfficiency
Comparison
Coordinated
The failure
to 16×rates.
Read rates- Peak are scaled
Read -from
Local 1× up
Read- IBIO today’s
Read
- NFS
the application
Coordinated
Flat
Flat
Bufferfactor,to Bufferacheivesystemefficiency.
Coordinated
Fl
Coordinated
Burst
Bufferhigh UncoordinatedUncoordinated
Burst
BufferRead
Peak checkpointing
Read - Local
Read--IBIORead- -NFS-- Peak
Write
Localrequires
Write
IBIO
Write
NFS i.e., the overhead
BecauseWrite
blocking
extremely
high
PFS
inCoordinateddifferentAs
futuresystemsBurst
willBufferbelargerand will
haveBuffer
larger α, Coordinated
CoordinatedUncoordinated
Burst
Optimizationintervalis critical,
becauseBu
Peak
Writeefficiency,- LocalWrite- IBIOWrite- NFS
1 of checkpoint1
bandwidth
to - achieve
90%
we
omit
that linefrom
4.5 Write
sizes,
ratesoperation.
and checkpoint
size are expected enough,1 is failure
checkpointing
an expensive
Several optimization
10.9 our no
4.5 Alternatively, our non-blocking checkpointing sys- memory
the figure.
0.9
4
have
studied.
[18]weproposeda methodblocking c
increase.
To beenexploretheYoung
effects,increasefailure rates than
0.9
0.90.8tem achieves
90% efficiency with a mere 10GB/s bandwidth totechniques
4
0.8
3.5
to
determinetheoptimalcheckpointintervalforsingle-level,
0.8
0.8increasingoverand checkpoint
for failure
0.7 costs by factors of 1, 2, and 10, and compare
3.5 rates up to 4× today’s rates.
0.7
blocking checkpoints.
Vaidya extended the model to support
3
Overall,our checkpointing system outperforms blocking efficiency
0.7
0.7
cost
betweenblocking
and non-blocking checkpointing. checkpoint
0.6
0.6
3
checkpoints, called fork checkpoint [19] , andcheckpointing.
However, at 90% efficiency, the bandwidth non-blocking
2.5
0.6GB for the checkpoint size per compute node,
0.6
checkpoint
costWe
use 290.5
0.5
2.5
two-level
checkpoints[20].
Vaidyaalso
combinedboth
methrequirement
rises sharply with failure rates larger than 5×
0.5
0.5 than non
2
half
the memoryof acheckpoint
TSUBAME2.0
0.4 aof0.4
ods to is
supporttwo-level
non-blocking
system to thin. As ficientbecause 2the time for L1 checkpoints begins to dominate appli- which
0.4
0.4
1.5
achieve12higher[21].encoding
Vaidya’s rate
modelisassumes
thatregardless rent
0.3 efficiency
failurerate
shows,an XOR
constant
0.3
cation 1.5
run time due to shortened optimal checkpoint intervals. Figure
0.3
atthe
most
one0.2fast-level
checkpoint
is taken
betweeneachXOR
slow-encoding the0.3
1
0.2
Here, increased
PFS
bandwidth
cannotincrease
efficiency.
failurerates
ofnumberofcomputenodes,
which
means
1
0.2
0.2
level checkpoint.
However, in current multi-level checkpoint0.5 we found that current levels of PFS throughput are scales
However,
0.1
be effecti
with0.1
systemsize. Thus, when we increase checkpoint can0.10.1
0.5
ing systems,
the fastest checkpoints, which are often saved
adequateformaintaining
80%
and
70%
efficiency.
0
0
0
large
costs,
weincrease
only
PFS
checkpoint
cost.
0 failure ra
to node-local0 storage, are orders of magnitude faster than theWith 0blocking
require0
2 checkpointing,4
6 systems
810
12higher
14 PFS16
1
1
2
10
50
100
0
2
4
6
8
10
12
14
16
1
1
2
10
50
100
Figure13
showstheefficiencyofbothcheckpointing
methinterval
decreaTo factor
account
forxL2)
this, wethroughput to minimize the riskof failure during a PFS slowest checkpoints saved to the PFS.
Scale
(xF,
# of Processes
Scale andfactor(xF,# of Processes
extendpriordifferent
work to modelan arbitrary
numberofxL2)
node-local
underfailurerates
checkpoint
costs. We the overall run
checkpoint. Further, blocking checkpointing uses the PFS ods
	
   when taking a PFS checkpointing, which means the define
ideal
time
checkpoints
between consecutive
PFS
checkpoints.
only
the efficiency
as expected
.
The
ideal
time
is the blocking check
time
Fig.
Efficiency
ofmultilevel
coordinatedFig.
6.6. Sequential
andwrite throughput
throughput
PFS
underutilized
during mostreadof theapplicationrun.
With run time
Fig.
Effi
Fig.
7.7. application
Efficiency
ofmultilevel
coordinatedFig.
Sequential
read
and
write
if theencountersnofailuresandtake Fig.
plicationrun
tim
VIII. C ONCLUSION8. 8.Efficie
oursystem,we notonly IBIO
hide PFS
checkpoint
anduncoordinated
C/R
on
a
flat
buffersystemofofcheckpointing
local
I/O,
and
I/O
with
and
NFS
via
formanceinanduncoordinated
C/R
on
a
flat
buffersystem
local
I/O,
and
I/O
with
IBIO
and
NFS
via
formance in x1
Figure	  	
  :	
  Sequencial	
  
read	
  
and	
  write	  throughput	
  of	
   We have
Figure	  10	
  :	
  Eandfficiency	
  ultilevel	
  ccheckpointoordinated	
  and	
  designedmodeledof	  amnon-blocking
overhead,but9usePFS throughout
application
execution.
andthatburstbuffer
buffersystemFDR
InfiniBandfactoring system
extends
anexisting
multi-level checkpointing
local	
  I/O,	
  a nd	  I/O	  networks
with	
  IBIO	
  and	
  NFS	
  via	
  FDR	
  uncoordinated	
  checkpoint/restart	
  ona	
  flat	  buffer	  andaa burst
systemFDR
InfiniBand
networks
8 factor
R ELATED W ORK
system.system	
  Our non-blocking
checkpointing
system enables apInfiniBand	
  nVII.
etworks	
  
and	
  a	
  burst	  buffer	  system	
  Multi-level checkpointing [4], [5] is a promising technique plications to save checkpoints to fast, scalable storage locatedfor fault-tolerant execution. Moody et al. [4] modeled a multi- on the compute nodes and then continue with their execution
Non/Blocking(90%)"

thetwotwolevellevelC/R
C/Rcasecasewhere
where
the
l
withfailure
failurerates
ratesofof{F{Fwith
1 ,1 ,F2F}(SeeTable
TableIV).
IV).
(Seeburstbuffer
buffersystem,
system,wew
InIna aburst
assumeassumethethefailure
failurerate
rateofofa abaacomputecomputenode.
node.OnOna acompucom
does
doesnot
notloselosecheckpoint
checkpointdatadanotnotinincomputecomputenodes.
nodes.Howev
Howcheckpoint
checkpointdata
dataononthe
thefailedfailed
we
also
use
two
level
C/R
we also use two level C/Rwhew
PFS.
PFS.Because
Becausethe
thetotal
totalnumber
numbrequiringinrequiringlevel
level1 1checkpoint
checkpointof burst buffer nodes. For 34

300

Checkpoint (XOR encoding)
Restart (XOR decoding)Global failure notification time
(Seconds)

C/R Throughput (GB/seconds)

350

0.4
0.35

250

2000

Probability to run for 24 hours

0.3states) on
FMI establishes the log-ring overlay network (H20.25
therecovery.
The
initialization
must
be
fast
and
Nodes
1,856
compute nodes(1,944 nodes inInterface)	
  total)FMI	
  (Fault	  Tolerant	
  Messaging	  :	
   The	   key	   contributions	
   include	   a	
   simplified	
  scalable for200
fast recovery. In Figure 14, we show the initialization
time
CPU
2.8 GHz Intel Xeon EP X5660 × 2 (12 cores in total)
programming	
  model	  to	
   bandwidth:enable	  32 GB/s)
fast,	
   150transparent	
  fault	
   tolerance	
   based	
   on	
  0.2
Memory24 GB (PeakCPU memory
for MVAPICH2 and FMI. For FMI, this is time0.15
spent in theInterconnect
QLogic InfiniBand
QDRcheckpoint/restart;	
  implementation	
  
of	
   a	
   runtime	
  failures	
  H1that	  and withstands	  H2 states. We process	
  compare thetime inand	
  
FMI Init with
100
0.1
allocates	
   spare	
   resources;	
   a	
   new	
   overlay	
   network	
  called	  log-­‐ring	
  
for	
  
thatstructure	
  in MVAPICH2’s
MPIInit. We
seescalable	  that the time to build50
0.05
the log-ring of	  (H2 the	  state)fault	
  is smalland scalable,
becauseeach
failure	
   detection	
   and	   notification;	
   and	   demonstration	  tolerance	
  
and	
  
0
0
process
onlyconnects
to
log
n
other
processes.
The
FMI
FMI
with
an
MPI
implementation.
For
those
experiments,
we2
scalability	
  of	
  FMI	
  even	
  with	  a	
  MTBF	
  of	
  1	
  minute.	
  Especially,	
  our	
  implementation	
  of	
  FMI	
  0 bootsrapping
500 time (H11000state) is1500
48
about
two
times
fasterthan 96
used
MVAPICH2
version
1.2
running
on
top
of
SLURM
[17].
has	
  failure-­‐free	
  performance	
  that	  is	
  comparable	
  with	  MPI,	
  and	
  our	
  experiments	
  with	  a	  # of Processes
that of MVAPICH2.
The current prototype of FMI has limited
equation	
   solver	
   show	   that	
   running	
   with	   FcapabilitiesMI	
   incurs	
  
only	
   a	
   2
overhead	
  
with	
   a	
  of messages
A. Poisson	
  
FMI Performancecompared
to8%	
  
MPI.
smaller number
Fig. 12: Checkpoint/Restartscalability
with A
6 GB/nodeFig. 13: Failure notificatvery	
  high	  mean	
  time	  to	
  failure	
  of	
  1	
  minute.	
  in FMI initialization than in network
MVAPICH2, whichcheckpoints,are
12exchanged
processes/node
results
infasterbootstrapping.
However,
we
expect that if FMI
TABLE III: Ping-Poing Performance of MPI and FMI	
  evolves to support more capabilities, it will also exchange
2500
1
1-byte Latency
Bandwidth (8MB)
moreMPImessages and its initialization time will approach that
0.9
MPI
3.555 usec
3.227 GB/s	  of MVAPICH2.
FMI
FMI
3.573 usec
3.211 GB/s
TABLE II: Sierra Cluster Specification

	
  
	
  
	
  

Performance (GFlops)

0.8
MPI + C
0.7
B. Application
Performance with FMI
FMI + C50.4
0.6
1500 Bootstrapping
4.5
+ C/R
We measured
ToFMI
investigate
the impact of FMI on the performanceof0.35 the point-to-point communication perforLog-ring overlay
0.5
4(MVAPICH2)
mance on Sierra,anactualapplication
run,
we
used
a
Poissonequationsolver,
0.3 and compare FMI to MVAPICH2. Table SLURM
0.4
3.5 1000
III shows the 0.25
ping-pong communication latency for 1-byte
the Himeno benchmark [18]. Himeno is a stencil
3
0.3application
2.5
0.2
messages, and bandwidth
for a message size of 8 MB. Because
in which each grid point is iteratively updated
0.2using only
2 500
0.15 MPI calls, we compiled the same ping-pong
FMI can intercept
neighbor
points.
The
computational
patternfrequently
appears0.1
1.5
source for both0.1MPI and FMI. The results show that 1FMI in numerical simulation codes for solving partial0 differential
0
Elapsed time (Seconds)(XOR decoding)	
  Global failure notification time
(Seconds)

oint (XOR encoding)	  Efficiency

Probability to run for 24 hours

0.05
0.5 thehas	
   very similar
performance compared to MPI for both
Himeno uses
communications
0 5 and
10 15
0equations.500
1000 point-to-point
1500
0
0
	
  
latency
andthebandwidth.
The
overheadforprovidingfault48 one
Allreducetheendof each iteration.
500
1000
1500
Scale factor
48
96
192
384
768
1536
96 #
384at (12768processes/node)
1536
of192
Processes# of Processes
Processes# of Processes
tolerance in FMI is negligibly# ofsmall
for messaging.
Figure15 shows the performance of Himeno compared withFig.
15:
Himenobenchmark(Checkpoint
size:metric
821 is computed based
Figure	
  
12	
  
:MPI
	
  Himeno	
  benchmark	  (Checkpoint	  	  Becausefailurerates
areexpectedtoincreaseatextremeusing
SCR [4].
The
FLOPS
Restart scalability with 6 GB/node Fig.
13: Failure
notification
time with log-ring
Figure	  11	
  :Failure	
  notification	
  time	
  overlay
Fig. 16: Probability to c
MB/node,MTBF:
minute)size	
  :	
  821MB/node,	  M1	
  minute)	
  Fig.
14: MPI1Init
vs.TBF	  
FMI :	
  Initsses/nodescale, C/Rnetworkforfailurerecovery
must
be
fast
and
scalable.
Toon time spent in application code making useful progress.
with	
  log-­‐ring	
  overlay	
  network	  	
  evaluate
the scalability of C/R in FMI, we ran a benchmark For example, if an application fails at time t1 , and rolls
11
which writes 0.9
checkpoints (6 GB/node),back to time t0 , the FLOPS metric does not include the lost
Coastal (w/andFMI) then recovers
0.9
Coastal (w/o FMI)
0.8 	
  In	
  conclusion	
  0.8
Fail-­‐in-­‐Place	
  
L
arge	
  
s
cale	
  
N
etwork	  Design:	  odonef	
  our	
  srestore
imulations,	
  the	
   backusing
the checkpoints.
Figure12 shows
the C/R
throughput
computation withtoor
the application
the state
run0.7 continuously
without
FMI,
we tosimulateda+C
0.7
including
XOR
encoding
anddecoding.
The
checkpoint
time
at
t
.
We
configured
SCR
to
write
checkpoints
to
tmpfs
and
+C
change	  of	
  0.6the	  routing	  algorithm	
  from	  the	
  c0.6
urrently	
  u1sed	
  Up*/Down*	
  routing	  to	
  DFSSSP	
  application
running checkpoint
at extreme scale.
If SCR
we andassumefailure+ C/Rof routing	  FMI is fairly
scalable because
the ncheckpointing
both
FMI with
0.5 TSUBAME2.5	  0.5 andon	
  would	  ot	
  only	  lead	  to	
  a	
  optimize
higher	  ptheerformance	
  intervalof	
  the	  of
MPI_Alltoall	
  0.4
encoding times0.4 are constant regardless of the totaloccur
numberaccordingto Poisson’s
the probability
Vaidya’s model[13] baseddistribution,
on configured MTBFof 1 minute, tha- 1 GB/nodeon	
  the	
  fault-­‐free	
  nFMI
etwork,	  aands	
  shown	  in	
  Fig.	
  0.3
2,	
  btout	
  L1aand
lso	
  
will	
  increase	  the	
  fail-­‐in-­‐place	  of nodes. Also,0.3
becausewritesreads
checkpoints
measuredcheckpointingtime.
L1 - 10 GB/node
0.2
anapplication
runs for time T continuously is e−λ·T where
0.2
L1& 2e-fforts	  1 GB/nodeto	
  achieve	
  exascale	
  scientific	
  of	
  throughputsthe	
  network.	
  
Both	
  
will	
  
support	
  
t
he	
  
andcharacteristic	  from memory,theare high.
FMI
achieves
2.4
Becausethe point-to-point communication performance of
0.1
0.1
L1 & 2 - 10 GB/node
theunrecoverable
failure arerate.
0
simulations.	
  	
   throughput per node, and 1.3is GB/sec
0
GB/seccheckpointing
FMI and MVAPICH2
nearly the same (Table III), the0 5 10 15 20 25 30 35 40 45 50
0
5 10 15 20 25 30 35 40 45 50
500
1000
1500
restartthroughput
per
node.
On
a
restart,newly
launched
performanceofHimenoisnearlyFMI and MPI if
Scale factor (Current failure rate = 1)
Scale factor
rocesses (12 processes/node)
16 shows
the probability theto samerun forcontinuouslyfor 2	
   the restored checkpoint chunks from theFigure
processes gather
other we do not write any checkpoints during the execution (FMI &enchmark (Checkpoint size: 821
Fig.
17:
Efficiency
ofmultilevel
C/R
under
increasing
failure
hours
using
rates
from
the LLNLSCR
failureProbability
continuously
run for 24 hours
processes inFig.
the16:
XOR
groupto after
the decoding
as in Figure
11,
MPIfailurein Figure15). For
checkpointing,writesanalysis
to memoryof th
minute)rate and L2
C/R timeso 	
  the restart throughput is lower than that of checkpointing.
via
a
file
system
(MPI
+
C),
while
FMI
writescheckpoints
Coastal cluster [4], with a level-1 failure rate of 2.13−6 (MTB
Fast and scalable failure detection time and reinitialization directly to memory using memcpy (FMI + C). Thus, FMI
=with
130
hours)(recoverable
by byXOR
encoding), and level	   (H1y with or without
FMI,
Here,
we predict theefficiency
of usinghigher
multilevel
C/R withtimeandwe
H2simulated
states) are an
criticalin environments
high exhibits
performance
10.3 % with the same memory−7
failurerate
ofof 4.27(MTBF= 650 hours)(unrecoverabl
ing at extremefailure
scale. rates.
If weFigure
assume13failures
efficiency
is be
the
ratio
time asspentin
usefulshows theFMI,
time where
for all processes
toconsumption
MPI
when
checkpointing
is enabled.
We also	
  notifiedof failure
with the log-ringoverlay. In thisto Poisson’s distribution,
the probability
that computation
onlyexperiment,
versuscomputation,
C/R
activities,
andinjected
failures
into
Himenotoseetheimpactofkilling
failures). We increase the failure rates from the observedaleve
we inject a failure
sendingsignal to killafter
a processns for time T continuouslyis e−λ·Tbywhere
λ arecomputation
future systems
becomeoflarger,
processwith a MTBF
1 minute during the execution. Even
1recovery.
and in2Asvaluesbyscale
factorsof 1 (observed values) to 5
two checkpoints to measure
averagedperformance.
	  withtotal
the veryhigh failurerate, we found that Himeno incurredable failure rate.between
we expecthigherfailure rates andaggregatecheckpoint
evaluate
performancelargerthescales.
For example, if we write checkpoints after 10, 20, 30toseconds,
onlyFMI’s
a 28% overhead
with FMI.atBecause
FMI C/RWith
time is FM
ws the probability to run continuously for 24 sizes. Thus, to predict application efficiency at larger scales,we	
   inject failures after 15, 25, 35 we
seconds.
later.failure
As shown,
constant
regardless
of to
the
total
nodesevenaccordingincrease
ratestheandexecutions
checkpoint
costs
50×,
80%
of
can uprunfor24numberhoursofwith
6× tohighe
re rates from the LLNL
failure analysis of the
global detection is scalable
becausethe log-ringpropagates
theperformancemodellevel-1
in Section
V-B, we expect FMI to scale
using
the
Coastalsystem
as
a
base
line.
BecauseC/R−64], with a level-1 failure rate of 2.13 (MTBF
failure rates.
At larger
failurerates nodes.
of 10× higher than today’s
notification
in logarithmic time. time
Whenis aconstantprocess regardlessterminates,of the
to atotal
muchnumbernumber ofnodes,ofwe	
   encoding),
Figure	
  13	
  :	
  MPI_Alltoall	
  runtime	
  simulation	
  for	
  TSUABME2.5	
  ecoverable by XOR
and level-2
70%
of
FMI
executions
run continuously for 24 hours
ibverbs waits approximatelyusing	  0.2only
beforeclosing
increaselevel-2
C/Rthetime.
Fornetwork	
  level-2
wecan
assumedseconds
ifferent	
  routing	  algorithms	  while	
  fC/R,ailures	  .27−7 (MTBF = 650 hours) (unrecoverable
C.
Resiliency
with
FMI
connection
to the terminatedprocess.
Therefore
there
is aonly 10%
wewrite
checkpoints
asynchronously
to theusing theexecutions can do the same	
   from thewhile
of PFS
non-FMI
have	
  
b
een	
  
injected	  rease the failure rates
observed levelconstant overhead of 0.2 secondsframework
before the andnotification
startsFMI
applications
canto run as long as all failuresmodel
developed
in
our
prior
work
[16].
Executions
without FMI are continue
terminatedby any failures, whil
by scale factors of 1 (observed values) to 50
to propagate in the log-ring.
are recoverable. To investigate how long an application can’s performance at larger scales. With FMI,FMI executions are terminated by only level-2 failure. Thus
ns can run for 24 hours with even 6× higher
using FMI effectively decreases unrecoverable failure rate, λfailure rates of 10× higher than today’s,
Figure 17 shows the efficiency of multilevel C/R in FMI.
and thus
the probability of long continuous runs is higher wit
ecutions can run continuously for 24 hours, Because we are uncertain
as to whether level-2 failure rates% of non-FMI executions can do the same. will increase at extreme
FMI,scale,eveninatourvery
high wefailurerates. At a scale factor 50, th
evaluation
increaseout FMI are terminated by any failures, while only the level-1 failure rate (L1) or both the level-1 and 2

level-1 MTBF becomes 2.6 hours (= 130 hours / 50), whic

Compressi

Figurepoint size
creasing dLossy	
  compression	
  for	
  fast	  checkpint/restart	
  :	
  Our	  preliminary	  studies	
  show	  that	  our	
  that we c
2. Ratio of compressed checkpoint size to original checkpoint size
lossy	
  compression	
  approach	
  can	
  reduce	
  size	  Figureof	
  simulation	
  
d
ata	
  
o
f	
  
a
	
  
r
eal	
  
c
limate	  14 %. M
under increasing division #checkpoinapplication,	
  NICAM,	  	  to	
  12-­‐13%	
  with	  0.2	  to	
  0.8	  of	
  errors	
  on	
  average	
  error.	
  without lo
Table I
only reduc
C OMPRESSION RATIO BY SIMPLE TECHNIQUES
cannot co
Figurecompressi
gzip
gzip
three-step
difference+ wavlet transformation
compression
Compression ratio
86.57 %
78.70%
12-14%
3, our loserrors. In
error is re#, we canFigure 2 shows compression ratio to the original checkpoint size (compressed size/original size x100) with0.3
in-of erro	
  creasing division #. As shown in the figure, we confirmed

Figure	  that
15	
  :	
  Awe
verage	  rrors	  of	
  theuncompressed	  heckpoint	  to	
   12 tocan ereduce
checkpoint cdata
to about
Figure	  14	
  :Ratio	
  of	
  compressed	
  checkpoint	
  size	  to	
  To redFigure 2. Ratio of compressed checkpoint size to original checkpointFiguresize
3. Average
errorsuncompressed
checkpoint
original value
original	
  value	
  
u
nder	
  of
increasing	  division	  #	
   togzip
original	
  checkpoint	
  
s
ize	
  
u
nder	
  
i
ncreasing	  division	
  
#
	
  
14
%.
Meanwhile,
directly
applying
totheoriginal
under increasing division #applicatio
checkpoint only reduce the size to 86.57%, and our approachlate transf	
  without lossy-compression, i.e., quantization and encoding,proach fir
only reduce to 78.70%. The simple compression techniques
transform
reducecheckpoint
cannotconsiderably
reduce size.
the checkpoint size.
4)	
  Meeting	
  (other	
  than	
  the	  monthly	
  conf	
  cconsiderablyalls	
  and	
  
the	
  
G8	
  the
ECS	
  
workshops)	
  to reduce
Finally, we
apply
encoding
to each values
after
the quanFigure
3nd	
  presents
averageerrorsproducedby our lossy• Meeting	
  with	  Franck	
  Cappello	
  at	
  Titech,	  ANL-­‐UIUC	  a
I
NRIA,	  2th-­‐4th	
  
J
une	
  
2
k
k 012	
  checkpointization. compression
If division # iswith
2 , wecan differentiate
valuesincreasing
division 2#. ofTheerror is thecompressi• Meeting	
  with	  John	  Dennis	
  at	
  Titech,	
  3by
1th	
  
M
arch	
  
2
014	
  
k bits.difference
We reducetosize
each value
in in
theKelvinhigh-frequency
the oforiginal
values(K). As in Figurea real clim
band
representing
thearch	  valueswithk bits.
Then,fromwe store• Meeting	
  with	  Leonard	
  Bautista	
  Gomez	
  
at	
  by
Titech,	
  
1th	
  M
2014	
  
3,
our 3lossy
compression
only
produces
0.2 to 0.8 of
on averagthe float,
doublevalueschar variables.
Thus, wecanerrors.
In terms
of totemperatureof a climate
simulation,theand comp
store othevalueswithGfour
times
less
loat values,5)	
  Visits	
  (visiting	
  partners	
  or	
  hosting	
  partners	
  
ther	
  
tishan	  8	
  Esmall.
CS	
  workshops)	
  error
relatively
If wethancan fchoose
13 for andthe divisioneight times
lesscanfordouble
#,	
  	
  Jwe
reducethecheckpoint
size to about 13% with only
• Ana	
  Gainaru	  (Ph.D.	
  at	
  University	
  of	
  Illinois),	
  
une	
  
2012	
  
to	
  
Avalues.
ugust	  2013	
  For 2-Dand
3-Darrays,theseriesof the transformation is
0.3
of
error
on
averagein
NICAM.
This w	
  applied separately in the X and Y directions for 2-D arrays,
Scientific
IV. Cfor
ONCLUSION
and in the X, Y and Z directions
3-D arrays.
ware Tech6)	
  I
mpact	  	
  To reduce checkpoint and restart time, we explored
Figure 3. Average errors of uncompressed checkpoint to original value
in Post-pe
III. EVALUATION
under
increasing division #Publications:	
  application-level
lossy
compression approach based a wavelate transformation.
three-step lossy approachcompression apTo explore
the effect of ourOur
lossy-compression[SMK12]	
  Kento	  Sato,	  Adam	  Moody,	  Kathryn	
  on
Mohror,	
  
Todd	
  
Gamblin,	
  
R.	
  de	
   todata
firsttransforms
theusing a wavelet
a realproach
application,
we applyBronis	
  our simulation
approachsimulation[1] A. Gra
transformation,
and
then
apply
quantization
and
data
of
a
climatesimulation
application,
NICAM
[2],
thenencodingence E
Supinski,	
  
N
aoya	
  
M
aruyama	
  
a
nd	
  
S
atoshi	  Matsuoka,	  "Design	
  
a
nd	
  
M
odeling	
  
o
f	
  
a
	
  
N
on-­‐
considerably reduce the checkpoint size.
reducethe simulation
data, which
we measured
compressed
size ofConference	
  checkpiont
data,
errors to be1995.
Finally, weapply encoding toeach values
the quanblocking	
  Checkpointing	  System",	
  In	
  after
Proceedings	
  of	
  tothe	
  International	  on	
  isand
Hnecessary
igh	
  
k
k
checkpointed.
Ourpreliminary
studies
show
that
to
the
original
valueswithdifferent
division#.
In
thisour lossy
tization.
If division# is 2 , we candifferentiate 2Storage	
  of valuesand	
  Analysis	
  2012	
  (SC12),	
  Salt	  Lake,	
  USA,	
  Performance	
  Computing,	
  Networking,	
  [2] M. Sa
compression
approachcanreducesizeofsimulation
dataofevaluation,
we
also
apply
gzip
to
the
compressed
data
after
by
k bits.
We reduce size of each value in the high-frequency
and S.
Nov,	
  2012.	  arealclimateapplication
to12-13%
with
0.2
to
0.8
of
errorsour
lossy-compression
intheprevious
section.
The
NICAM
(NICA
band by representing the values with k bits. Then, we store
on average
In future
work, such
we willthe errors,of Co
code
writesseveral error.
physicalquantities,
as reducepressure,
thefloat,
doublevaluestocharvariables.
Thus,
we
can
[SSS13]	
  Takafumi	
  Saito,	  Kento	  Sato,	  Hitoshi	  Sato	  and	  atoshi	  Matsuoka,	  "Energy-­‐aware	
  3514,
andSandcompression
time.
velocity, asa checkpoint. We target an array
store the values with four times less than f loat values, temperature
andcom/scI/O	
  O
ptimization	
  
f
or	
  
C
heckpoint	
  
a
nd	
  
R
estart	
  
o
n	
  
a
	
  
N
AND	
  
F
lash	
  
M
emory	  System",	
  
I
n	
  
t
he	
  
of
temperature
valuesinthisevaluation.
eight times less for double values.
ACKNOWLEDGMENT
Workshop	
  n	
  Farrays,
ault-­‐Tolerance	
  or	
  transformation
HPC	
  at	
  Extreme	  Scale	  2013	
  (FTXS2013)	
  in	
  conjunction	
  For 2-D ando3-D
the series of ftheisThis work was partially supported by Grant-in-Aid for
with	
  
t
he	
  
I
nternational	
  S
ymposium	
  
o
n	
  
H
igh	
  
P
erformance	
  Parallel	
  and	  Distributed	
  applied separately in the X and Y directions for 2-D arrays,
Scientific Research (S) 23220003, and JST-CREST ”Softandin the X, Y(HPDC13),	
  and Z directions
3-D arrays.
Computing	  New	
  forYork,	
  USA,	
  June,	
  2013.	
   ware Technology that Deals with Deeper Memory Hierarchy
in Post-petascale Era” (2012-2018).
E VALUATION[SMM13]	
  Kento	  III.
Sato,	
  Satoshi	  Matsuoka,	  Adam	  Moody,	  Kathryn	  Mohror,	
  Todd	  Gamblin,	  under increasing division #R EFERENCES
To explore
effect of ourlossy-compression
approachBronis	
  R.	
  dthee	
  Supinski	
  
and	
  
Naoya	
  Maruyama,	  "Burst	  SSD	
  Buffer:	
  Checkpoint	
  Strategy	
  at	
  
on
a
real
application,
we
apply
our
approachtosimulation[1]
A.
Graps,“An
introduction
towavelets,” Computational SciExtreme	  Scale",	
  IPSJ	
  SIG	
  Technical	
  Reports	
  2013-­‐HPC-­‐141,	
  Okinawa,	
  Sep,	
  2013.	
  data of a climate simulation application, NICAM [2], thenence Engineering, IEEE, vol. 2, no. 2, pp. 50–61, Summer
wemeasured
compressed
sizeofcheckpiont
data,
and
errors[SMM14]	
  Kento	  Sato,	  Adam	  Moody,	  Kathryn	  Mohror,	
  T1995.
odd	
  Gamblin,	
  Bronis	
  R.	
  de	
  
to the original values with different division #. In this[2]
M. F
Satoh,
Matsuno, M
H.essaging	
  Tomita, H. Miura, T. Nasuno,Supinski,	
  
N
aoya	
  
M
aruyama	
  
a
nd	
  
S
atoshi	  Matsuoka,	
  
"
FMI:	
  
ault	
  T.
Tolerant	
  evaluation, we also apply gzip to the compressed data afterand S. Iga, “Nonhydrostatic icosahedral atmospheric modelInterface	  for	
  Fast	  ainnd	  ransparent	  Recovery",	
  of	
  the	  International	
  ourlossy-compression
theTprevious
section.
The NICAMIn	
  Proceedings	
  (NICAM) for global cloud resolving simulations,” Journal
code writes several physical quantities, such as pressure,of Computational Physics, vol. 227, no. 7, pp. 3486 –
3514, 2008. [Online]. Available: http://www.sciencedirect.
temperature and velocity, as a checkpoint. We target an arraycom/science/article/pii/S0021999107000654
of temperature values in this evaluation.

Conference	
  on	
  Parallel	
  and	  Distributed	
  Processing	  Symposium	
  2014	
  (IPDPS2014),	
  Phoenix,	
  USA,	
  May,	
  2014.	
  [SMM14-­‐2]	
  Kento	  Sato,	
  Kathryn	  Mohror,	
  Adam	  Moody,	
  Todd	  Gamblin,	
  Bronis	
  R.	
  de	
  
Supinski,	
  Naoya	
  Maruyama	
  and	
  Satoshi	  Matsuoka,	
  "A	
  User-­‐level	  InfiniBand-­‐based	
  File	  System	  and	
  Checkpoint	
  Strategy	
  for	
  Burst	  Buffers",	
  In	
  Proceedings	
  of	
  the	
  14th	  IEEE/ACM	
  International	
  Symposium	
  on	  Cluster,	
  Cloud	
  and	
  Grid	  Computing	  (CCGrid2014),	
  Chicago,	
  USA,	
  May,	
  2014.	
  (Best	  Paper	  Finalist)	
  [SSE14]	
  Naoto	  Sasaki,	
  Kento	  Sato,	  Toshio	
  Endo	  and	
  Satoshi	  Matsuoka,	  "Exploration	
  of	
  Application-­‐level	  Lossy	  Compression	
  for	
  Fast	  Checkpoint/Restart",	
  In	
  HPC	  in	
  Asia	  Workshop	  in	
  conjunction	
  with	  the	  International	
  Supercomputing	
  Conference	
  (ISC'14),	
  Leipzig,	
  Germany,	
  June,	
  2014.	
  7)	
  Future	
  plans	
  for	  the	
  next	
  18	
  months	
  (please	
  look	  carefully	
  at	
  the	
  tasks	
  and	  deliverable)	
  Our	
  objectives	
  in	
  the	
  next	
  18	
  months	
  are	
  related	
  to	
  part	  of	  task	  3.3	
  and	
  D3.3:	  Asynchronous	
   Checkpointing	
   System:	
   We	
   will	   integrate	
   the	   asynchronous	
  checkpointing	  system	
  with	  SCR	  (Scalable	
  Checkpoint/Restart),	
  which	  was	
  developed	  by	  LLNL.	
  Exploration	
   of	
   Multi-­‐tier	
   Storage	
   Design	
   with	   Burst	
   Buffer:	
   	
   We	
   will	   integrate	
   the	  asynchronous	
  checkpointing	  system	
  with	  SCR	  (Scalable	
  Checkpoint	  FMI	
   (Fault	
   Tolerant	
   Messaging	
   Interface):	   	
   Although	   the	   current	
   FMI	
   prototype	
   has	  demonstrated	
  promising	  results,	
  it	
  not	
  yet	
  complete	
  enough	
  to	
  support	
  a	
  broad	  range	  of	
  applications.	
   Future	
   FMI	
   will	
   support	
   I/O	
   interfaces	
   like	
   MPI	
   I/O,	
   restoring	
  communicators	
  along	
  with	  checkpoints,	
  exploit	
  multi-­‐level	  checkpointing.	
  Fail-­‐in-­‐Place	
   Large	
   scale	
   Network	
   Design:	
   	
   A	
   next	
   step	
   will	
   be	   a	   detailed	
   analysis	
   of	  the	
  communication	
  layer	
  of	
  the	
  applications	  and	  a	
  co-­‐design	
  phase	  to	
  match	  the	
  needs	
  of	
  
the	
   applications	
   with	
   the	   underlying	
   network	
   infrastructure,	
   i.e.,	   topology	
   and	   routing	  algorithm.	
  Lossy	
  compression	
  for	
  fast	  checkpint/restart	
  :	
  	
  We	
  will	  reduce	  the	
  errors,	
  and	
  compression	
  time.	
  8)	
  References	
  [1]	
   Bianca	
   Schroeder	
   and	   Garth	
   A.	
   Gibson.	
   Understanding	
   Failures	
   in	
   Petascale	
   Com-­‐	
  puters.	
  Journal	
  of	
  Physics:	
  Conference	  Series,	
  78(1):012022+,	
  July	  2007	
  [2]	
   Al	
   Geist	
   and	   Christian	
   Engelmann.	
   Development	
   of	
   Naturally	
   Fault	
   Tolerant	
   Al-­‐	  gorithms	
  for	  Computing	  on	
  100,000	
  Processors,	
  2002.	
  [3]	
   John	
   Daly	
   et	   al.	
   Inter-­‐Agency	
   Workshop	   on	
   HPC	
   Resilience	
   at	
   Extreme	
   Scale,	  February	
  2012.	
  [4]	
  Ning	  Liu,	
  Jason	  Cope,	  Philip	
  H.	
  Carns,	
  Christopher	
  D.	
  Carothers,	
  Robert	
  B.	
  Ross,	
  Gary	  Grider,	
  Adam	  Crume,	  and	  Carlos	
  Maltzahn.	
  On	
  the	
  Role	  of	
  Burst	  Buffers	
  in	
  Leadership-­‐
Class	
   Storage	
   Systems.	
   In	
   Symposium	
   on	
   Mass	
   Storage	
   Systems	
   and	   Technologies,	
   MSST	  2012,	
  April	
  2012.	
  [5]	
   Dries	
   Kimpe,	
   Kathryn	
   Mohror,	
   Adam	
   Moody,	
   Brian	
   Van	   Essen,	
   Maya	
   Gokhale,	
   Rob	  Ross,	
   and	   Bronis	
   R.	
   de	
   Supinski.	
   Integrated	
   In-­‐System	
   Storage	
   Architecture	
   for	   High	  Performance	
  Computing.	
  In	
  Proceedings	
  of	
  the	
  2nd	  International	
  Workshop	
  on	
  Runtime	
  and	
  Operating	  Systems	
  for	  Supercomputers,	
  ROSS	  ’12,	
  2012.	
  
[6]	
  A.	
  Graps,	
  “An	
  introduction	
  to	
  wavelets,”	
  Computational	
  Sci-­‐	  ence	  Engineering,	
  IEEE,	
  
vol.	
  2,	
  no.	  2,	  pp.	
  50–61,	
  Summer	  1995.