# Presentation: 319175

- CEA/DAM's Site Report
- (site of Bruyères le Châtel - France)

- Thomas LEIBOVICI
- thomas.leibovici@cea.fr

## Scope of this presentation

-  What is CEA
-  The TERA project	Goals, architecture, evolutions
-  The storage system	Current system, next generation
-  Our use of HPSS	Current architecture, future
-  Around HPSS

## Who are we ?

-  CEA is a research institute which handlesscientific research in many domains:
-  Energy
-  Biotechnologies
-  Micro/nano-electronics
-  Geology, climatology, ...
-  Within CEA, DAM (Direction des Applications Militaires) is a subdivision focused on military applications
-  DAM's Compute Center is located in Bruyères-Le-Châtel, south of Paris

## TERA: project overview

- Production codes simulate physics within complex and critical systems
- TERA-1 (5,35 TFlops) is in full production since 2H2002
- TERA-10 (>60 TFlops) is about to replace TERA-1 in 1H2006
- Compute power x10 every 4 years(TERA-100 in 2009)

## TERA: architecture overview

- _Supercomputers_
-  Simulation computations
-  Local data caches

- _Storage systems_
-  Simulation results
-  Experimental data

- _Dedicated servers_
-  Visualization
-  Users data

- _Local Workstations_

- _Other__CEA/DAM__centers_

- Storage level 2 (Tapes)

- Storage level 1 (Disks)

- Networks

## Supercomputers: From TERA-1 to TERA-10

- _TERA-10 :_ (under deployment)
- Linux Cluster550 nodes x 16 processors (8 Montecito)
- Lustre Filesystem
- Compute power x12  (1 -> 12 Tflops)
- Memory size  x12  (2,5 -> 30 TB)
- Disk space  x20  (50 TB -> 1 PB)
- Disk bandwidth  x15  (7,5 GB/s -> 100 GB/s)Peak performance > 60 TFlops

- _TERA-1:_SMP Cluster
- 640 nodes x 4 processorsLinPack: 3,98 Tflops
- Sustained performance: 1 TFlops
- Peak performance: 5,35 Tflops

## Storage: current system

- HPSS Production system:
-  1 WinterHawk Node: core server
- 	- 4 procs / 2 GB
-  5 NightHawk-I Nodes: Disk/Tapes Movers
- 	- 5 TB of disks (RAID 5+1)
- 	- 6 interfaces / node dedicated to data transfers
- 	- 8 HBAs for SAN access to tapes drive / node
- 	- 8 procs / 4GB RAM
- 	- Interconnection via Colony Switch
-  1 silver node : nfs server
- 	- 2 procs / 512 MB
-  Sun Blade 100: DCE CDS Server
-  7 PowderHorn Silos with 48 x 9840C
- 	- 1 PB = 25000 x 40GB tapes
-  1 PowderHorn silo with 10 x LTO2
- 	- 1 PB = 5000 x 200GB tapes

## Storage system evolutions

-  Production system is about to be upgraded  from HPSS 4.3.0.2 to HPSS 5.1.1 (next week !)
-  Data path to supercomputers:
-  Migration from Hippi to Gigabit Ethernet
- RFP for a new storage system:
-  Linux cluster to run HPSS 6.2
-  Up to 6 PB of disks for the first storage level
-  Data path: Infiniband

## Our specific use of HPSS

- Computing nodes

- Local disks(data cache)

- Storage level 2(Tapes)

- Storage level 1(Disks)

- Data path
- (mvr protocol)via local tool hpss_rcp

- NFS server

- NFS clients

- Metadata path
- (NFS)

-  Access to file’s metadatas through NFS
-  Synchronization between local data cache and HPSS is done by SHERPA (CEA’s local software suit)
-  Data transfers use HPSS mover protocol

## SHERPA

- Compute node

- User’s program

- Client library

- 1: sherpa_open

- Data path

- Sherpa server

- 2: call to sherpa

- NFS Server

- HPSS

- Mover slave

- HPSS movers

- local disks

- 3: compare

- 4: launch  transfert

- 6: R / W operations

- 5: handle to local file

## Long term architecture

- _Supercomputers_

- _Storage systems_
-  Lustre handles concurrent access to files via a distributed & shared filesystem.
- => Interface between lustre and HPSS is required

- _Dedicated servers_
-  Visualization
-  Users data

- Storage level 2(Tapes)

- Storage level 1(Disks)

## Current work around HPSS

-  Preparing the transition to HPSS 6.2 on linux platforms
-  Currently implementing an NFSv4 serverover HPSS client API
-  Enhanced caching capabilities
-  See  Ph. Deniel’s talk
-  Adaptation to TERA-10 architecture
-  Adding a new subsystem to HPSS to enhance scalability and to provide access to more users

## Wish list

-  Full Linux support
-  Full 64 bits support
-  Interfacing Lustre with HPSS

- Questions ?