# Presentation: 223911

## SDSC

## SDSC TeraGrid Presentation

- Fran Berman
- Director, SDSC and NPACI

## SDSC “node” optimized for data-oriented computing

- Archive
**6 PB**

**NCSA**
**10 TF**
**230 TB disk**

**SDSC**
**4 TFLOP IA-64 (DTF)**
**1.1 TFLOP Power 4 (ETF)**
**500 TB SAN disk (DTF+ETF)**
**SUN & IBM DB Engines**
**Oracle and DB2**
**6 PB Archive**

**Caltech**
**0.4 TF **
**80 TB disk**

**Argonne**
**1.25 TF**
**20 TB disk**

- Myrinet Clos Spine

**TeraGrid Backbone (40 Gbps)**

- Blue Horizon
- IBM SP
**1.7 TFLOPs**

- Abilene
- Calren
- ESnet

- Sun
- F15K

***SDSC “node” optimized for data-oriented computing***

**PSC**
**6.3 TF**
**221 TB disk**

- IBM Power 4
**1.1 TFLOPs**

- SAN Switch Fabric

- 2-IBM Power 4
**Database Engines**

- every node
- is a server

## SDSC TeraGrid Data Architecture 

- Philosophy:  enable SDSC config to serve the grid as data center

- Blue Horizon

- HPSS

- LAN (multiple GbE, TCP/IP)

- SAN (2 Gb/s, SCSI)

- Linux Cluster, 4TF

- Sun F15K

- WAN    (30 Gb/s)

- SCSI/IP or FC/IP

- FC Disk Cache (400 TB)

- FC GPFS Disk (100TB)

- 200 MB/s per controller

- Silos and Tape, 6 PB,
-  1+ GB/sec disk to tape
- 52 tape drives

- 30 MB/s per drive

- Database Engine

- Data Miner

- Vis Engine

- Local Disk (50TB)

**design leveraged ****at other TG sites**

- Power 4

- Power 4 DB

- .5 PB disk
- 6 PB archive
- 1 GB/s disk-to-tape
- Optimized support for DB2 /Oracle
- Data working group lead

## GigE to everything
Fibrechannel to everything
Myrinet to everything

- Switch

- Switch

- Switch

- Switch

- Switch

- 256 x 2Gb FCS

- 4 x 10Gb Ethernet

- 256 x 1Gb Ethernet

- 128 x 2Gb FCS

- FC Disk Arrays (500 TB total)

**SDSC DTF Cluster: *****Every Node is a Server***

- 3 x 10 Gbps *lambdas*

- Los Angeles

- 3 x 10 Gbps

- Router

- 1.0 TFLOP

- 1.0 TFLOP

- 1.0 TFLOP

- 1.0 TFLOP

**2x 2Gb Myrinet Interconnect**

- Multi-port 2 Gb mesh

- Brocade
- Silkworm
- 12000

- Brocade
- Silkworm
- 12000

- Brocade
- Silkworm
- 12000

- Brocade
- Silkworm
- 12000

- GigE to everything
- Fibrechannel to everything
- Myrinet to everything

## TG Working Groups

- Data (Chaitan Baru)
- Defining a data management strategy for the TeraGrid
- HDF, filesystems (PVFS, GPFS), global name space
- Currently merging several documents to produce cohesive plan
- Operations (Tom Sherwin)
- Distributed TeraGrid Operations Center
- Global ticket system
- System monitoring
- Distributed accounting – allocations used on any resource
- User Services – co-lead (Nancy Wilkins-Diehr)
- Allocations white paper
- Draft User Guide
- Friendly Users starting now on IA-32 systems

## Data Computing: NVO and TeraGrid

- TeraGrid should enable NVO researchers to
**improve performance and enable scaling to multiple sky surveys**
- improved bandwidth
- increased object server performance
- access to sufficient on-line storage
**transform usage paradigm **
- by providing easy, on-line access to all data
**enable distributed joins across distributed survey catalogs**
- due to high-speed interconnects and large, local disk caches
- to find possible correspondences or new information (objects)

## NVO and TeraGrid

**0.1 sec/file**

**250+ MB/s**

**1. Filter**

**Raw**
**Data**

**3. Retrieve,**
**Analyze data**

**4. Output**
**image**

**MCAT (update indexes)**

** ****Significant reduction in time to ingest data; less than 5 days**
** ****Web-based query interface for on-line access to **_**all**_** data**
** ****Support for multiple surveys; distributed join of data across surveys**
** ****Storage and Network works together**

**With TeraGrid:**

**1. Filter**

**SRB Containers (store **
**sorted, processed data into HPSS)**

**Raw**
**Data**

**3. Retrieve,**
**Analyze data**

**4. Output**
**image**

**9.5TB raw data**
**5 million files**

**MCAT (update indexes)**

**16MB/s**

**1 sec/file**

** ****60 days to ingest (sort, index, store) data**
** ****Restricted web-based access to a maximum of 100 MB file (1 SRB container)**

**Today:**

**Store data via SRB**

**Store products**

**1. Filter**

**Raw**
**Data**

**1. Filter**

**Raw**
**Data**

**1. Filter**

**Data from other surveys**

**10’s of TB **
**of data**

**Blue Horizon**

** ****DTF Cluster**

**Raw**
**Data**

**2. Sort/**
**Index  **
**Data   **

**2. Sort/**
**Index  **
**Data   **

## Points-of-Contact

- PI:  Fran Berman 				(_[berman@sdsc.](mailto:berman@sdsc.edu)__[edu](mailto:berman@sdsc.edu)_)
- Site Lead:  Phil Andrews 			(_[andrews@sdsc.edu](mailto:andrews@sdsc.edu)_)
- Project Manager:  Mark Sheddon 		(_[sheddon@sdsc.](mailto:sheddon@sdsc.edu)__[edu](mailto:sheddon@sdsc.edu)_)
- Data:  Chaitan Baru 				(baru@sdsc.edu)
- Operations:  Tom Sherwin 			(sherwin@sdsc.edu)
- User Services:  Nancy Wilkins-Diehr 		(wilkinsn@sdsc.edu)
- Grids:  Phil Papadopoulos 			(phil@sdsc.edu)
- Ext. Rel.:  Dave Hart 				(dhart@sdsc.edu)
- Clusters:  Patricia Kovatch 			(pkovatch@sdsc.edu)
- Applications:  Stuart Johnson 			(sjohnson@sdsc.edu)
- Networks:  Jay Dombrowski 			(dombrowh@sdsc.edu)
- Perf. Eval.:  Wayne Pfeiffer 			(pfeiffer@sdsc.edu)
- Visualization:  Steve Cutchin 			(_cutchin@sdsc.edu_)