# Presentation: 378429

## SOS 11

**Challenges of Sustained Petascale Computation**
**June 12, 2007**
**Dave Fellinger, CTO**
**dfellinger@datadirectnet.com**

**Architecting High Performance, Scalable & Highly Available Cluster Storage**

## Cluster Storage Requirements
Drive Roadmap
Parallel Storage Architecture
S2A 9900
Overview; 9500 vs. 9900 Comparison
Performance Highlights

- Drive Roadmap
- Parallel Storage Architecture
- S2A 9900
- Overview; 9500 vs. 9900 Comparison
- Performance Highlights

**Agenda**

## Cluster Storage Requirements

**“****Scratch” storage on simulation clusters has specific requirements**
**Write cycles must be fast and consistant**
**Disk I/O errors and retries cannot affect the performance of writes to the system**
**I/O rates must scale well across threads and transfer size**
**Storage for visualization clusters has specific requirements**
**Read cycles must be fast and consistant**
**Disk I/O must be checked for errors if SATA is employed**
**Disk I/O errors and retries cannot affect the performance of reads from the system**

**Notes:**
The discussion will also include a look ahead to models that allow aggregation of very large pools of disks to optimize reliability, cost, and bandwidth.  The discussion will cover future innovations to lower latencies and deliver higher bandwidths to maximize the computational capabilities of these clusters.  Moreover, he will discuss the need to provide integrated solutions that enable ease of installation, management and support.

## Slide 4

## Slide 5

## Cluster Storage Requirements
Drive Roadmap
Parallel Storage Architecture
S2A 9900
Overview; 9500 vs. 9900 Comparison
Performance Highlights

- Drive Roadmap
- Parallel Storage Architecture
- S2A 9900
- Overview; 9500 vs. 9900 Comparison
- Performance Highlights

**Agenda**

## Storage Trends

- Moore’s law is still valid for CPU processing power
- 2X performance every 18 months =  CAGR of 45% per annum
- How about storage at the individual disk drive level?
- Disk areal density = CAGR of 60%/year (capacity is doing fine)
- Seek time improvement = CAGR of <15%/year (performance is NOT doing fine!)
- Disk IO/second increase = CAGR of <15%/year (performance is NOT doing fine!)
- Sustained bandwidth = CAGR of <25%/year (performance is NOT doing fine!)
- What does this mean?
- The more cost effective the system, the lower its performance!!!
- The ratio of Performance to Capacity has become a significant factor in managing storage subsystem performance
**THE PERFORMANCE OF DISK DRIVES IS NOT KEEPING UP WITH OTHER TECHNOLOGICAL TRENDS IN THE CLUSTER!**

**Notes:**
Disk Densities are growing but not nearly as fast as CPU performance
Seek, latency and transfer times are not going to change much over the next few years
Transfer Rates Lag Capacity Significantly
I/O performance is comprised of a combination of raw media transfer rates, access times (seek + latency), file system efficiency
Different applications and architectures are more or less sensitive to different of the three areas
You are not going to spin disk 5.76 billion RPM/second
You are not going to sustain 4.7 TB/sec. to a single disk
You are not going to have a disk with 215TB of capacity (soon)

## I/O and Storage Challenges

- Source: David Koester, Ph.D. and Henry Newman @ HPCS I/O Workshop, July 12, 2005

**Notes:**
The limitations and bottlenecks with today’s solutions in particular the growing imbalance between computational performance and disk drive performance.  While great strides have been made on the CPU side in both power management and bandwidth, disk drives have followed a path of increased capacity and lower cost while maintaining the same random performance levels for the past 10 years.  The discussion will include what is needed to bridge this gap and enable low cost, easy to use high performance storage for compute clusters. 

## Drive Roadmap

- S2A

- Disk Drives,
- FC

- Disk Drives,
- SATA

- SMI-s, R1

- Today

- Q2 ‘07

- Disk Drives,
- SAS

- 73GB 15k FC 4Gb

- RAID 6 Enhanced,
- R3.1

- 750GB SATA

- 1TB SATA 3Gb

- Sleep Mode
- Drives

- 300GB 15k FC 4Gb

- S2A9550

- 146GB 15k FC 4Gb

- S2A9900

- 146GB 15K FC 4Gb

- 450GB 15K FC 4Gb

- 300GB 15K FC 4Gb

- Q3 ‘07

- 300GB 15K SAS 3Gb

- 450GB 15K SAS 3Gb

- 146GB 15K SAS 3Gb

**Drive Roadmap**

- Q1 ‘08

- Q4 ‘07

## Cluster Storage Requirements
Drive Roadmap
Parallel Storage Architecture
S2A 9900
Overview; 9500 vs. 9900 Comparison
Performance Highlights

- Drive Roadmap
- Parallel Storage Architecture
- S2A 9900
- Overview; 9500 vs. 9900 Comparison
- Performance Highlights

**Agenda**

## Large Scale Parallel Storage

**Low Latency High Performance Silicon Based Storage Controller with RDMA**
- Parallel Access For Hosts
- Parallel Access To A Large Number Of Disk Drives
- True Performance Aggregation
- Reliability From A Parallel Pool
- Quality Of Service
- Scalability
- Drive Error Recovery In Real Time
- True State Machine Control
- 10 Virtex 4 FPGAs, 16 Intel embedded processors, 8 Data FPGAs

**Notes:**
The discussion will also include a look ahead to models that allow aggregation of very large pools of disks to optimize reliability, cost, and bandwidth.  The discussion will cover future innovations to lower latencies and deliver higher bandwidths to maximize the computational capabilities of these clusters.  Moreover, he will discuss the need to provide integrated solutions that enable ease of installation, management and support.

## An Implementation of Parallelism w/ Double Parity RAID Protection

**Loops to Disks**

**8 FC-4 and/or****4 IB 4X**
**Parallel Host Ports**

**An Implementation of Parallelism w/ Double Parity RAID Protection**

**Double Disk Failure Protection**
**LUNs can span tiers**
**All ports access all storage**
**Implemented in Hardware State Machine**
**Equivalent 8+1 & 8+2 performance**
**Parity Computed On Writes AND Reads**
**No loss of performance on any failure**
**Multi-Tier Storage Support, Fibre Channel   or SATA Disks**
**Up to 1250 disks total**
**1000 formattable disks**

## Data Flow, To Disk

**D1**

**D2**

**D3**

**D4**

**D5**

**D6**

**D7**

**D8**

**P1**

**P2***

**D1**

**D2**

**D3**

**D4**

**D5**

**D6**

**D7**

**D8**

**P1**

**P2***

**Serial IB/FC**
**Data Streams**

**512Byte Parallel**
**Data Segments**

**Data Flow, To Disk**

**FC-4 and/or IB Host Interface**

**Parallel Processing Parity Engine**

**FPGA PCI Bridge**
**Generates 512B Parallel Segments**

**FPGA Parity Engine**
**Generates One or Optionally Two Parity Segments Synchronously**

**Disk Controller Engines**
**Queue Command Re-Ordering in Queue Cache**
**Vertical Striping**
**Disk Interfaces**

## Data Flow, From Disk

**D1**

**D2**

**D3**

**D4**

**D5**

**D6**

**D7**

**D8**

**P1**

**P2***

**D1**

**D2**

**D3**

**D4**

**D5**

**D6**

**D7**

**D8**

**P1**

**P2***

**Serial IB/FC**
**Data Streams**

**512Byte Parallel**
**Data Segments**

**Data Flow, From Disk**

**FC-4 and/or IB Host Interface**

**Parallel Processing Parity Engine**

**FPGA PCI Bridge**
**Saturate Multiple Host Ports w/ High Speed Read Data**

**FPGA Parity Engine**
**Real-time Parity Checking and Data Correction for each Read I/O Synchronously**

**Disk Controller Engines**
**Data Staging with Level One Cache**
**Shared Data Access**

## Reliability and Performance Solution

**Implementation of RAID 6**
**Added parity drives effect double redundancy**
**Reed Solomon coding in Real Time**
**Continuous parity checking in Reads and real time generation in Writes**
**Bad block recovery in real time**
**Drive error recovery in real time**
**Partial rebuilds without affecting host side access**

**Notes:**
The discussion will also include a look ahead to models that allow aggregation of very large pools of disks to optimize reliability, cost, and bandwidth.  The discussion will cover future innovations to lower latencies and deliver higher bandwidths to maximize the computational capabilities of these clusters.  Moreover, he will discuss the need to provide integrated solutions that enable ease of installation, management and support.

## Increase Data Availability

- SATA technology has enabled great cost economies but can significantly jeopardize data integrity without proper controls
- DDN has the experience (_a recognized leader in SATA_)
- DDN has the understanding (multi-faceted SATA protections)
**The Challenge**: to maintain QOS regardless of drive retry, reset, and internal recovery issues.
**The Solution: **All devices will be constantly monitored through HW and SW for excessive errors or defect growth and system software can begin rebuilds to spares ***before*** a failure occurs.

## Increased Data Availability

- The Hardware Solution
- Check parity for every read and correct it in real time.
- Use RAID 6 to identify individual drives that have read corrupt data through Reed-Solomon data recovery algorithms.
- Exercise total control over the array including the ability to power cycle each drive.

## Increased Data Availability

- The Software Solution
- Take a questionable drive offline immediately.
- Begin a journal of all writes that have been made to the array since the moment that a specific element was taken offline.
- Utilize a series of recovery techniques including command retries, drive resets, and finally power cycling to confirm the status of the specific device.
- If the device cannot be revived it can be replaced.
- If the device can be revived it can be rebuilt from the journal in a short time.

## Cluster Storage Requirements
Drive Roadmap
Parallel Storage Architecture
S2A 9900
Overview; 9500 vs. 9900 Comparison
Performance Highlights

- Drive Roadmap
- Parallel Storage Architecture
- S2A 9900
- Overview; 9500 vs. 9900 Comparison
- Performance Highlights

**Agenda**

## S2A 9900 Hardware Specifications

| Supported Disk Technology | SAS & SATA | FibreChannel & SATA |
| --- | --- | --- |
| RAID Parity Protection | RAID6 8+2 Only | RAID3 (8+1+1),  RAID6 8+2 |
| Sustained Throughput | 5.6GB/s – 6.0GB/s | 2.4 GB/s – 2.8GB/s |
| Maximum Cache | 5.0 GB ECC Protected | 2.5GB RAID Protected |
| Minimum Cache | 2.5 GB ECC Protected | 2.5GB RAID Protected |
| Disk Side Ports | 20 x SAS 4 Lane | 20 x FC-2 |
| Host Side FC Ports | 8 x IB 4x DDR or 8 x FC-8 | 8 x FC-4 or 8 x IB 4x |
| Dimensions | 7 x 19 x 28 in.   (4U) | 7 x 19 x 25 in.   (4U) |
| Certifications | UL,CE,CUL,C-Tick,FCC | UL,CE,CUL,C-Tick,FCC |
| Release Date | 1Q/2008 | September 2005 |

**Specification                  ****	**** S2A9900 Couplet    ****	****S2A9550 Couplet**

**S2A 9900 Hardware Specifications**

## Performance & Capacity Scalability

**Performance & Capacity Scalability**

## S2A 9900 Capacity

**S2A 9900 Capacity**

| Five 60-Slot JBODs
Two Dual Loop per JBOD: 300 Disks
 300TB SATA using 1TB Drives
 135TB SAS using 450GB Drives | Ten 60-Slot JBODs
Two Dual Loop per JBOD: 600 Disks
 600TB SATA using 1TB Drives
 270TB SAS using 450GB Drives | Twenty 60-Slot JBODs
Two Dual Loop per JBOD: 1200 Disks
 1.2PB SATA using 1TB Drives
 540TB SAS using 450GB Drives |
| --- | --- | --- |

## Improvements

- Faster Intel Main CPU
- Faster Interface
- SDR IB -> DDR IB
- FC4 -> FC8
- PCI *Express* Bus Architecture
- Faster Intel Host Processors
- Doubled Cache Size & Cache Rate
- Faster Backend
- FC2 -> SAS
- Optimized Drive Health Management
- Increased Component Reliability
- Cooling
- Connection

**Notes:**
Gartner, midrange storage, Raw Terabytes shipped for independent storage company

## Additional Enhancements

- Expanded log capability
- Rebuild write journaling
- Power Down Archiving of writeback data (coupled with UPS)
- Power Consumption Reduction
- Sleep Mode Drives (SATA)
- DC Power

**Notes:**
Gartner, midrange storage, Raw Terabytes shipped for independent storage company

## Cluster Storage Requirements
Drive Roadmap
Parallel Storage Architecture
S2A 9900
Overview; 9500 vs. 9900 Comparison
Performance Highlights

- Drive Roadmap
- Parallel Storage Architecture
- S2A 9900
- Overview; 9500 vs. 9900 Comparison
- Performance Highlights

**Agenda**

## Backend Throughput

**12GB/s** potential backend bandwidth
- 10 x 4-lane SAS Channels per Singlet
- Disk Channel Controller
- Provides Cache to SAS Connectivity
- Provides 2.5GB/5GB Cache Memory Segment via DCC FPGA
- Cache Controller Interface
- Interfaces to Main CPU via Dual Port SRAM

## Front-end Throughput

- Maximum **4GB/s** Singlet Front-end Bandwidth
- 4 x 8-lane PCI Express Ports per Singlet
- Host Interface
- Dual Protocol
- Fibre Channel (FC8 when available)
- Infiniband (DDR x4 IB SRP target (iSER tbd))
- DMA Capable
- Enables Zero-Copy Interfacing

## Increased IOPS

- Target: 2-3X 9550 Performance
- Robust Processors:
- Intel Chevelon Host CPU
- Intel Sunrise Lake Main CPU
- Faster Cache Controller/Stage Buffer FPGA
- Faster processor DRAM:  512Mb DDR2
- 3.2GBytes/sec processor to memory bandwidth & reduced latencies

## SOS 11

**Challenges of Sustained Petascale Computation**
**June 12, 2007**
**Dave Fellinger, CTO**
**dfellinger@datadirectnet.com**

**Architecting High Performance, Scalable & Highly Available Cluster Storage**