# Presentation: 553369

## HPSS - National Center for Computational Sciences

**Mitchell Griffith**
**June 7 - 9, 2005**
**griffithmr1@ornl.gov**
- The submitted manuscript has been authored by a contractor of the U.S. Government under Contract No. DE-AC05-00OR22725. Accordingly, the U.S. Government retains a non-exclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others to do so, for U.S. Government purposes.

## HPSS at NCCS

**NCCS/HPSS Mission:**
**	****Provide Stable, Ample,**
**   ****and Nimble storage to **
**   ****users and systems**
**Currently using HPSS 4.5, upgrade to **
**   ****5.1.1 (3Q2005)**
**HSI main transfer utility, ftp ****(limited support)**

## NCCS HPSS Storage Environment

**STK 9310**
- (4) 9840-SCSI
- (4) 9840

**STK 9310**
- (8) 9940B

**STK 9310**
- (6) 9840
- (2) 9940A

**STK 9310**
- (4) 9840
- (8) 9940B

**IBM 44P**
- beagle
- 4 - SCSI 9840

**DataDirect**
**S2A8000**
- 5 TB

**IBM**
**FAStT700**
- 1 TB

**IBM p630**
- saturn
- HPSS ndapi
- STK ACSLS
- 1 TB DataDirect
- 4 - 9840
- 8 - 9940

**IBM p630**
- jupiter
- 1 TB DataDirect
- 8 - 9940
- DCE cds/secd rep.

**IBM p630**
- arm18
- 1 TB FAStT 700
- HPSS Core Services
- DCE cds/secd master rep.

**IBM p660**
- Mooch
- Disk: 1 TB DataDirect

**IBM H70**
- arm17
- .5 TB SSA
- 6 - 9840

**Brocade Switches**
- 1 - SilkWorm 2800
- 2 - SilkWorm 3800
- 1 - SilkWorm 3900

**IBM SSA**
- 1 TB

**IBM p660**
- Fozzie
- 278 GB SSA
- 4 - 9840

**Notes:**
-. p5 (P550) core server, ftp and ndapi server (internal and external)
 -. New brocade switch to replace the 2800
 -. a couple of new silos (one each fire zone)
 -. metadata mirrored on each fire zone

## NCCS Networking

**Internal networks are GigE, being upgraded to 10GigE this year.**
**External connections to:**
**ESnet – OC192**
**Internet2 – OC192**
**National Lambda Rail – 2 x OC192**
**UltraScienceNet Testbed – up to 16 x OC192**

## NCCS is responsible for the software to manage HPSS – the Storage System Management (SSM) components. 
Several elements
System Manager (SM)
Data Server (DS) through 5.1
GUI and command-line interfaces (for operators and admins)
Developers – Deryl Steinert, Vicky White and Tom Barron with Kathleen Tinch, IBM and Debbie Morford, LLNL
Huge effort
310 Java source files	265,000 source lines
170 C source files		230,000 source lines
272 auto-generated files	173,000 generated lines
            121,000 executable statements
100+ screens, 1000+ variables

- Several elements
- System Manager (SM)
- Data Server (DS) through 5.1
- GUI and command-line interfaces (for operators and admins)
**Developers – Deryl Steinert, Vicky White and Tom Barron with Kathleen Tinch, IBM and Debbie Morford, LLNL**
**Huge effort**
- 310 Java source files	265,000 source lines
- 170 C source files		230,000 source lines
- 272 auto-generated files	173,000 generated lines
-             121,000 executable statements
**100+ screens, 1000+ variables**

- NCCS Development Activities

## NCCS HPSS Statistics

**Production System**
**Over 6.8 Million files**
**~576 TB stored **
**Growing ~ 7TB/week**
**Moving 20 TB/day**

**Notes:**
6.7 million files
570 TB stored
Double overall capacity annually

## NCCS HPSS Statistics

**Notes:**
6.7 million files
570 TB stored
Double overall capacity annually

## Testing Shared File Systems

**HPSS**

- Likely to begin testing
- Lustre
- as a shared
- global File System
- 4Q - 2005

_**Where we **_
_**are today**_

_**Where we need to go...**_

**Large Shared**
**Global Filesystem**

_**For more information on:**_
- Lustre Filesystem see http://www.clusterfs.com

## NDAPI Statistics 

- Total Logins -----> 30
- Total Puts -------> 606
- Total Put Errors -> 1
- Total Gets -------> 128
- Total Get Errors -> 3

| Username | Login | Sent | #Puts | #PErr | Recv | #Gets | #GErr | Avg FS | DCE | KRB | KYT |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| User1 | 1 | 0 B | 0 | 0 | 0 B | 0 | 2 | 0 B | 0  | 1 | 0 |
| User2 | 19 | 146.623 MB | 435 | 1 | 0 B | 0 | 0 | 345.153 KB | 0 | 0 | 19 |
| User3 | 3 | 129.628 MB | 171 | 0 | 66.060 MB | 110 | 0 | 776.253 KB | 2 | 1 | 0 |
| User4 | 7 | 0 B | 0 | 0 | 20.649 GB | 18 | 1 | 0 B | 0 | 7 | 0 |

| COS | Xfer Avg | Xfer Min | Xfer Max | Files | FS Avg | FS Min | FS Max |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 3003 | 359.16 KB/s | 0 B/s | 18.48 MB/s | 1208 | 2.66 MB | 257 B | 652.38 MB |
| 6003 | 47.71 MB/s | 9.02 MB/s | 88.08 MB/s | 218 | 94.05 MB | 4.06 MB | 168.74 MB |
| 6005 | 62.42 MB/s | 14.08 MB/s | 112.91 MB/s | 210 | 319.17 MB | 221.04 MB | 781.76 MB |

- Host(s) Storing Files
- ------------------
- arm18.ccs.NCCS.gov
- cheetah48.ccs.NCCS.gov
- taurus.ccs.NCCS.gov

- Host(s) Retrieving Files
- ------------------
- cheetah48.ccs.NCCS.gov
- ram1.ccs.NCCS.gov

## FTP Statistics 

- Total Puts -------> 606
- Total Gets -------> 128

| Username | Write | Read | Par_Open | Par_Get | Par_Put | GetSize | PutSize |
| --- | --- | --- | --- | --- | --- | --- | --- |
| User1 | 0 | 0 | 2 | 0 | 2 | 0 B | 1002.94 MB |
| User2 | 0 | 0 | 1 | 1 | 0 | 134.90 MB | 0 B |
| User3 | 1 | 4 | 8 | 3 | 5 | 366.16 KB | 313.85 KB |

- Host(s) Storing Files
- -----------------------
- cheetah48.ccs.NCCS.gov
- taurus.ccs.NCCS.gov

- Host(s) Retrieving Files
- ------------------------
- cheetah48.ccs.NCCS.gov
- ram1.ccs.NCCS.gov

## NCCS supports research in a number of scientific disciplines 
with the HPCC and Storage resources located in the National
Center for Computational Sciences (NCCS).

Astrophysics -- http://www.ccs.NCCS.gov/astro/
Atmospheric Radiation Measurement -- http://www.arm.gov/docs/
Climate and Carbon research -- http://www.ccs.NCCS.gov/CCR/
Computational Biology -- http://www.ccs.NCCS.gov/cbi/
Computational Materials -- http://www.ccs.NCCS.gov/mri/
Computational Sciences -- http://www.csm.NCCS.gov/
Fusion Simulation -- http://www.ccs.NCCS.gov/fsi/
Nanoscience -- http://www.cnms.NCCS.gov/
Neutron Sciences -- http://www.sns.gov/
Transportation -- http://www.NCCS.gov/info/NCCSreview/v33_3_00/features.htm

**with the HPCC and Storage resources located in the National**
**Center for Computational Sciences (NCCS).**
**Astrophysics **-- http://www.ccs.NCCS.gov/astro/
**Atmospheric Radiation Measurement **-- http://www.arm.gov/docs/
**Climate and Carbon research **-- http://www.ccs.NCCS.gov/CCR/
**Computational Biology **-- http://www.ccs.NCCS.gov/cbi/
**Computational Materials **-- http://www.ccs.NCCS.gov/mri/
**Computational Sciences **-- http://www.csm.NCCS.gov/
**Fusion Simulation **-- http://www.ccs.NCCS.gov/fsi/
**Nanoscience **-- http://www.cnms.NCCS.gov/
**Neutron Sciences **-- http://www.sns.gov/
**Transportation **-- http://www.NCCS.gov/info/NCCSreview/v33_3_00/features.htm

- NCCS’s Science Support

**Notes:**
Center for Computational Sciences (CCS) has been changed to NCCS (National Center for Computational Sciences)

## 	The ARM Program is the largest global change research program supported by the U.S. Department of Energy (DOE). 

The ARM Program’s primary objective is to develop and test parameterizations of clouds and their effect on the radiative energy balance, with the ultimate goal of improving general circulation models used for climate research and prediction.

To meet this goal, data is gathered at a number of                                         sites around the globe to measure key aspects of the                              radiation field under vastly different 	climate conditions.

ARM data stored in HPSS at NCCS accounts
	for ~25% of the data stored, and ~half of the
	total number of files.

- The ARM Program’s primary objective is to develop and test parameterizations of clouds and their effect on the radiative energy balance, with the ultimate goal of improving general circulation models used for climate research and prediction.
- To meet this goal, data is gathered at a number of                                         sites around the globe to measure key aspects of the                              radiation field under vastly different 	climate conditions.
- ARM data stored in HPSS at NCCS accounts
- 	for ~25% of the data stored, and ~half of the
- 	total number of files.

- Atmospheric Radiation Measurement (ARM) Program

**For more information see:**
-  http://www.arm.gov/docs/
-  http://www.arm.gov/docs/data

*The ARM Program's award-winning Millimeter Wave Cloud Radar (MMCR) is an active sensor that can measure cloud boundaries and microphysical properties with high space and time resolution. This data sample shows a thunderstorm passing above the Central Facility at ARM's Southern Great Plains site*

## Computational Platforms @ NCCS

**...****optimized for Applications**

**Extremely low latency, high bandwidth, interconnect**
**Efficient scalar processors, balanced interconnect**
**Shares interconnect technology with Cray X2**
**Front end-hosted environment - system calls and I/O**

**Proven architecture for performance and reliability**
**Most-powerful processors and interconnect**
**Scalable, globally addressable memory and bandwidth**
**Leverages commodity where possible**
**Offers capability computing for key applications**

**Notes:**
Phoenix:
 -. 512 multi-streaming vector processors (MSPs)
 -. modified 2D torus topology
 -. liquid fluorinent cooled
 -. 12.8 GF per processor

Cheetah:
 -. Federation Switch

## Cray X1 - Phoenix

**Largest X1 in the world today at 512 MSPs (6.4 TF)**
**4 GB of memory per MSP (2 TB total)**
**32 TB of directly attached disks**

- System will be upgraded to X1E over the summer
- 1,024 MSPs
- 18.5 TF

## Cray XT3 - Jaguar

**Cray XT3 (based on Red Storm @ Sandia)**
**5000+ processors **
**	****(Opteron)**
**10+ TB memory**
**120 TB disk**
**25 TeraFlops**

|  | Cabinets | Processors | Memory | TFLOPS |
| --- | --- | --- | --- | --- |
| Today | 40 | 3,748 | 7.5 TB | 18 |
| June ‘05 | 56 | 5,212 | 10.5 TB | 25.1 |
| 2006 | 120 | 22,000+ | 44 TB | 100+ |

## Wish List

**Real Relational Database ****(HPSS 5.1)**
**Real-time job monitor ****(Mike’s Monitor)**
**Better small file support**
**BBCP-like HSI**
**HPSS/Lustre integration**
**Tuning Document (best practice)**

## Questions?THE END

## Potential Breakthroughs

| Application | Simulation Need | Computing Need (TF) | Significance |
| --- | --- | --- | --- |
| Climate Science | Reliable predictions of climate change at the regional (e.g., state) scale, including biogeochemical feedbacks. | > 50 | Provides U.S. policymakers with data to support policy decisions; predicts extreme weather conditions arising from changing climate |
| Magnetic Fusion Energy | Understand and predict plasma turbulence, including self-heating of plasma and heat leakage caused by turbulence. | > 50 | Crucial for understanding burning plasmas and quantifying prospects for commercial fusion; underpins U.S. decisions about future international fusion collaborations. |
| Computational Biology | Understand the molecular processes that provide the mechanism for cells and organisms to adjust to changing conditions and to take advantage of available energy sources | > 50 | Allows the design of innovative new approaches to bioremediation, energy production, and climate management |
| Computational Chemistry | Reliable prediction of the structures, energetics, and reactions of molecules, especially the large molecules involved in nanoscience, catalysis and energy and environmental science. | > 100
 | Control of chemical processes responsible for energy produced and pollutants released by engines; prediction and mitigation of spread of pollutants in underground plumes and long-term effects of greenhouse gases and stratospheric ozone depletion; control of processing of high-level radioactive wastes. |
| Computational Material Science | Develop systematic approaches, based on sound theoretical models, that integrate across many length and time scales, to understand materials properties | > 100 | Foundation for the design of materials with specified properties from first principles |
| Astrophysics | Realistically simulate the explosion of a core collapse supernova for the first time | >> 100 | Understand the origin of the elements |

## The 170,000 square foot Eugene P. Wigner Center for Computational Sciences Building is home to the National Center for Computational Sciences (NCCS) 40,000 sq. ft. computing facility for unclassified scientific computing.

A unique feature of the building is that it was built entirely using private money and is owned by a private developer who leases it to UT-Battelle Development Corp. who then subleases it to the DOE.  Construction was completed in 15 months from breaking ground to move-in.  

- The NCCS Facility

**The 170,000 square foot Eugene P. Wigner Center for Computational Sciences Building is home to the National Center for Computational Sciences (NCCS) 40,000 sq. ft. computing facility for unclassified scientific computing.**
**A unique feature of the building is that it was built entirely using private money and is owned by a private developer who leases it to UT-Battelle Development Corp. who then subleases it to the DOE.  Construction was completed in 15 months from breaking ground to move-in.  **

## NCCS intends to bring neutron science to the TeraGrid through the Southeastern TeraGrid Extension for Neutron Science (SETENS).

Provide access to active experimental tools
High Flux Isotope Reactor (HFIR)
Spallation Neutron Source (SNS)

Access to live and archived neutron scattering data
Real-time access to neutron science data
Archived data stores through HPSS

**Provide access to active experimental tools**
- High Flux Isotope Reactor (HFIR)
- Spallation Neutron Source (SNS)
**Access to live and archived neutron scattering data**
- Real-time access to neutron science data
- Archived data stores through HPSS

- Neutron Science & The TeraGrid

**For more information see:**
-  http://www.csm.NCCS.gov/
-  http://www.sns.gov/
-  http://www.teragrid.org

**Notes:**
was slide 7.  Taken out because of no relation of teragrid to hpss at NCCS
I do not think this has relevance anymore.