DATA CENTRIC ARCHITECTURE AT
CEAJLUG 2014, Tokyo | Aurélien Degrémont
CEA | October
16 octobre
2014 2014| PAGE 1

AGENDA
WHAT IS CEA?
DATA PRODUCTION INCREASESDATA CENTRIC ARCHITECTURE EVOLUTIONSFUTURE APPROACHES WITH TERA-1000

16 octobre 2014
JLUG 2014, Tokyo

CEA | October 2014 | PAGE 2

WHAT IS CEA?

CEA, Commissariat à l'énergie atomique et aux énergies alternatives
French agency dedicated to research in energy, physics, biology, electronics fields, ...

CEA is running supercomputers for decades for science simulations.

1963
16 octobre 2014JLUG 2014, Tokyo2012
CEA | October 2014 | PAGE 3

EVOLUTION OF CEA COMPUTING COMPLEXES2012

SecurityDeterrence

2011
CCRT-C2010

Biology

CURIE

Astrophysics

2007
TERA-100

2005
2003
CCRT-B

Aeronautics

TERA-10CCRTClimate

16 octobre 2014
JLUG 2014, Tokyo

Nuclear Energy

CEA | October 2014 | PAGE 4

DATA PRODUCTION INCREASES| PAGE 5
16 octobre 2014

DEUS Grand Challenge
Massive computation on TGCC
76 000 cores
300 TB of memory
50 PB of data produced
Reduced to 500TB

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 6

Experimental Data Analysis
Bioinformatics community
New generation of DNA sequencer produces 4 TB / instrument / month
Continuous data stream
No data removal
Data analysis produces 3 times initial volume
CEA is involved in France Génomique project

U.S. DOE Joint Genome Institute (JGI)
A 10-Year Strategic Vision (September, 2012)

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 7

ARCHITECTURE EVOLUTION| PAGE 8
16 octobre 2014

Last Century
Compute Systems
Few Cray Supercomputers (vectors and MPP)
Few front-end machines

Storage Systems
Directly connected to the front-end or to the super-computer
Data managed through HSM

YMP

T3E
T90
JLUG 2014, Tokyo

CEA | October 2014 | PAGE 9

Early 2000 and later
Cluster Age
Tera 15 Tflops
Dsk: 50 TB, 7 GB/s
PFS + HPSS
Tapes: 1PB

TERA1Tera 1060 Tflops
Dsk: 2PB, 100 GB/s
Lustre + HPSS
Tapes: 10 PB
Curie or Tera 100
Over 1 Petaflops
Dsk: 20 PB, 500 GB/s
Lustre + HPSS
Tapes: 30 PB

JLUG 2014, TokyoTERA10TERA100
Curie

CEA | October 2014 | PAGE 10

Data Management for Cluster
Data Centric Architecture
Around a storage network
Dedicated to data movement
Independent of computing center
backbone
Hierarchical levels of storage
Fast storage (L1)
Large storage (L2)

Initially based on the archival system: HPSS (T1 + T10)
T1: L1 = striped fast tapes
T10: L1 = striped disks
NFS access to HPSS name space (Ganesha)
CEA parallel tool to move data between HPSS and compute cluster
Now based on parallel file system (T100 + TGCC)
Lustre based
Lustre transparently migrated to HPSSJLUG 2014, Tokyo

CEA | October 2014 | PAGE 11

Lustre centric architecture
TERA architecture
Storage and Lustre at the center of the computing complex

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 12

Evolution
Data management is at the heart of our concerns
For many codes, IO is the next barrier
User data set size explodes
Data set management takes more and more time
We have to help users to increase their productivity

Ubiquitous components in CEA computing complex
Lustre for the frontend
Performance
Scalability
HPSS for the archive
Co-development with HPSS community (US Labs)
Lustre-HSM brings archive independence
CEA storage softwareGanesha, Robinhood, Shine, Hercule

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 13FUTURE : TERA-1000| PAGE 14
16 octobre 2014

TERA-100: Current architecture

Compute Nodes~4000 machines

Petaflopic File SystemClients, Posix access

Cluster fabricNetwork~300 GB/s

I/O Servers~200 machines
Data + Meta-Data

RAID Controllers~20 000 disks
JLUG 2014, Tokyo

CEA | October 2014 | PAGE 15

Computing Center Evolution

Exaflop, 25 TB/s
100000x 1000
10000
10000

1000

PetaFlop, 500 GB/s

x 50

GB/s

TFlop/s

1000

1001001010Teraflop, 10 GB/s
12001 2002

2003 2004

20052006 2007 2008

12009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

TflopsJLUG 2014, Tokyo

GB/sCEA | October 2014 | PAGE 16

Roadmap to get an Exaflop Class System
2015
30 Petaflops
3.5 To/s
10 000 nodes> 20 000 disks

2020~1 Exaflops
25 To/s> 50 000 nodes> 30 000 disks

Impossible to keep the same ratio for storage bandwidth andfor compute power
New type of compute nodes
ManyCore architecture (ratio GB/Thread decreases)
Memory starvation for operating system

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 17

How to get filesystem performances?
Using I/O Forwarding
It ships I/O calls from compute nodes to dedicated I/O nodes
I/O proxies perform operations on behalf of compute nodesThis limits impact of I/O on compute nodes
They reduce file system traffic and recovery time
They allow : aggregation, rescheduling and caching of I/O requests

Lowmemory
per core

Computenodes

I/O
proxies

Compute cluster
JLUG 2014, Tokyo

Lustreservers

Storage cluster
CEA | October 2014 | PAGE 18

I/O Delegation
Codes implement dedicated I/O tasks
Use I/O Proxy architecture
Code knows their I/O needs.
They can write dedicated and optimized I/O delegators
I/O task running on I/O Proxies has efficient communication with compute task andoptimized access to storage.

customUser
compute task
GaneshaI/O task
protocolLustre client
application
LustreLustreservers

But this could not be done by all codes

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 19

I/O Forwarding server with Ganesha
Ganesha
CEA project originally developed as a user-space NFS Server.
Supports optimized Lustre NFS export.
The projects evolved to export Lustre with 9P protocol
9p export supports RDMA (mooshika)
http://nfs-ganesha.github.io/

9P
Lightweight filesystem protocol supported by Linux kernel: v9fs
Synchronous I/O
No caching
Supports RDMAv9fs

9p

GaneshaLustre client

JLUG 2014, Tokyo

LustreLustreservers

CEA | October 2014 | PAGE 20

I/O workflow

Efficient and scalable I/Os
with proxy and Ganesha

9p/
customComputenodes

LustreHPSSI/O
proxies

Storage clusterwith Lustre/HSM
Automatic data migration
With Lustre/HSM binding

Compute cluster

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 21

Conclusion
TERA: a generic data centric architecture
Data was always at center of CEA computing centers
Computing requirements are bigger and biggerStorage architecture needs to scale accordingly
More complex architecture is needed as data production increases faster than storage
hardware.
To be ready for exascale, tiered architecture is needed to future I/O workload.

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 22

Questions ?

JLUG 2014, Tokyo

CEA | October 2014 | PAGE 23

| PAGE 24
CEA | October 201416 octobre 2014

Commissariat à l’énergie atomique et aux énergies alternatives
Centre DAM-Ile de France | 91297 Bruyères-le-Châtel Cedex
T. +33 (0)1 69 26 40 00 | F. +33 (0)1 69 26 70 86
Etablissement public à caractère industriel et commercial | RCS Paris B 775 685 019

Direction des applications militaires
Département sciences de la simulation et de l'information
Service informatique scientifique et réseaux