# Presentation: 380716

## Cray Roadmap(2004-2010)

**John M. Levesque**
- Senior Technologist
- (Virtual Steve Scott
- Chief Architect for X1/X1E/BW)

## Cray’s Computing Vision

**Cray’s Computing Vision**

**X1**

**‘****Strider 2’**

**X1E**

**‘****Black Widow’**

**2006**

**‘****Strider 3’**

**2006**

**‘****Black Widow 2’**

***Scalable High-Bandwidth Computing***

**Product Integration**

**RS**

**Red Storm**

**‘****Strider X’**

**2010**
**‘****Cascade’**
**Sustained**
**Petaflops**

**2004**

**2004**

**2005**

## Cray  X1

**Cray PVP**
-  Powerful vector processors
-  Very high memory bandwidth
-  Non-unit stride computation
-  Special ISA features
-  Modernized the ISA

**T3E**
-  Extreme scalability
-  Optimized communication
-  Memory hierarchy
-  Synchronization features
-  Improved via vectors

**High bandwidth, scalable shared memory supercomputer**

**Cray  X1**

**Notes:**
ISA = Instruction Set Architecture; the instructions implemented in the system

## Key Architectural Features

- New vector instruction set architecture (ISA)
- Much larger register set (32x64 vector, 64+64 scalar)
- 64- and 32-bit memory and IEEE arithmetic
- Based on 25 years of experience compiling with Cray1 ISA
- Decoupled Execution
- Scalar unit runs ahead of vector unit, doing addressing and control
- Hardware dynamically unrolls loops, and issues multiple loops concurrently
- Special sync operations keep pipeline full, even across barriers
-  Allows the processor to perform well on short nested loops
- Scalable, distributed shared memory (DSM) architecture
- Memory hierarchy: caches, local memory, remote memory
- Low latency, load/store access to entire machine (tens of TBs)
- Processors support 1000’s of outstanding refs with flexible addressing
- Very high bandwidth network
- Coherence protocol, addressing and synchronization optimized for DM

## Cray X1 Node

-  Four multistream processors (MSPs), each 12.8 Gflops
-  High bandwidth local shared memory (128 Direct Rambus channels)
-  32 network links and four I/O links per node

**51 Gflops, 200 GB/s**

**Cray X1 Node**

## NUMA Scalable up to 1024 Nodes

- Global shared memory across machine

**Interconnection**
**Network**

**NUMA Scalable up to 1024 Nodes**

**Notes:**
This slide was created by Steve Scott.

## Network Topology (16 CPUs)

- M15

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

- M0

- M0

- M1

- M1

- M15

- M0

- M0

- M1

- M1

- M15

- M15

- node 0

- node 1

- node 2

- node 3

- Section 0

- Section 1

- Section 15

**Network Topology (16 CPUs)**

**Notes:**
Interconnect diagram of an air-cooled cabinet (or tiny liquid-cooled system).  Strictly cabling between M chips on each node.   Maximum of two hops from any node to any other.

## Network Topology (128 CPUs)

**Network Topology (128 CPUs)**

**16**
**links**

**Notes:**
Example of two LC cabinets.  The “R” boxes are router modules that handle the interconnect beyond 8 nodes and between cabinets.

## Network Topology (512 CPUs)

**Network Topology (512 CPUs)**

**Notes:**
Even larger system 8 full LC cabinets.

## Cray X1 Node Module

## Cray X1 Chassis 

## 64 Processor Cray X1 System~820 Gflops

**64 Processor Cray X1 System****~820***** Gflops***

## Cray X1E Product Enhancement

## Cray X1E Mid-life Enhancement

- ~50% faster processors
- Scalar performance enhancements
- Doubling processor density
- Modest increase in memory system bandwidth
- Same interconnect and I/O
- Machine upgradeable
- Can replace Cray X1 nodes with X1E nodes
- Shipping the end of this year

**Cray X1E Mid-life Enhancement**

## Cray BlackWidow System

- Second generation Vector MPP
- Upward compatible with the Cray X1
- Shipping in 2006
- Major improvement (>> Moore’s Law rate) in:
- Single thread scalar performance
- Price performance
- BlackWidow features:
- Single chip vector microprocessor
- Globally addressable memory with 4-way SMP nodes
- Scalable to tens of thousands of processors
- Even more bandwidth per flop than the X1
- Innovative fault tolerance features
- Configurable memory capacity, memory BW and network BW

## Balanced Performance between CPU, Memory, Interconnect, and I/O
Highly scalable system hardware and software 
High speed, high bandwidth 3D mesh interconnect   
Run a set of applications 7 times faster than ASCI Red
Run an ASCI Red application on full system for 50 hours
Flexible partitioning for classified and non-classified computing
High performance I/O subsystem (File system and storage)

- Highly *scalable* system hardware and software
- High speed, high *bandwidth* 3D mesh interconnect
- Run a set of applications 7 times faster than ASCI Red
- Run an ASCI Red application on *full system for 50 hours*
- Flexible partitioning for classified and non-classified computing
- High performance I/O subsystem (File system and storage)

**System Goals**

**Notes:**
Space, Power, Cooling - High density, low power system.
Price/Performance - Excellent performance per dollar, use high volume commodity parts where feasible.
DOE ASCII Sponsored 20Tops System
Maximize use of COTS material
Build HSN 
Build on ASCII Red success
26 month Program: Start to Acceptance
Performance across entire system
Scalable to 60 Tops
Meet all Sandia Requirements and ...
Maintain Commercial product Relevance

## Red Storm System Overview

- 108 compute node cabinets, 16 service and I/O node cabinets, and 16 Red/Black switch cabinets
- 10,368 compute processors - 2.0 GHz AMD OpteronTM
- 512 service and I/O processors (256P for red, 256P for black)
- 10 TB DDR memory
- 240 TB of disk storage(120TB for red, 120TB for black)
- MPP System Software
- Linux + lightweight compute node operating system
- Managed and used as a single system
- Easy to use programming environment
- Common programming environment
- High performance file system
- Low overhead RAS and message passing
- Approximately 3,000 ft2 including disk systems

**Red Storm System Overview**

## Typical Architecture

- Memory latency ~ 160 ns and *bandwidth is shared* between mutliple processors

- Northbridge chip is 2nd most complex chip on the board.  Typical chip uses about 11 Watts

## AMD Opteron Generic System

- SDRAM memory controller and function of Northbridge is pulled onto the Opteron die.  Memory latency reduced to 60-90 ns
- No Northbridge chip results in savings in heat, power, complexity and an increase in performance
- Interface off the chip is an open standard (HyperTransport)

## Using HyperTransport to Interface With System Interconnect

**6GB/sec (3 GB/sec bi-directional)**
**3D Torus Interconnect**

## Cray BlackWidow

- The Next Generation HPC
-  System From Cray Inc.

## Cray BlackWidow System

- Second generation Vector MPP
- Upward compatible with the Cray X1
- Shipping in 2006
- Major improvement (>> Moore’s Law rate) in:
- Single thread scalar performance
- Price performance
- BlackWidow features:
- Single chip vector microprocessor
- Globally addressable memory with 4-way SMP nodes
- Scalable to tens of thousands of processors
- Even more bandwidth per flop than the X1
- Innovative fault tolerance features
- Configurable memory capacity, memory BW and network BW

## CascadeToward Sustained Petaflop Computing

## HPCS Phases

**HPCS Phases**

**Phase I: Concept Development**
**Forecast available technology**
**Propose HPCS hw/sw concepts**
**Explore productivity metrics**
**Develop research plan for Phase II**

## Cray’s Approach to HPCS

**High system efficiency at scale**
**Bandwidth is the most critical and expensive part of scalability**
**Enable *****very***** high (but configurable) global bandwidth**
**Design processor and system to use this bandwidth *****wisely***** **
**Reduce bandwidth demand architecturally**

**High human productivity and portability**
**Support legacy and emerging languages**
**Provide strong compiler, tools and runtime support**
**Support a mixed UMA/NUMA programming model**
**Develop higher-level programming language and tools**
**System robustness**
**Provide excellent fault detection and diagnosis**
**Implement automatic reconfiguration and fault containment**
**Make all resources virtualized and dynamically reconfigurable**

## Using Bandwidth Wisely

**Implement global shared memory**
**Lowest *****latency***** communication**
**Lowest *****overhead***** communication**
**Fine-grained *****overlap***** of computation and communication**
**Tolerate latency with processor *****concurrency***
**Message passing concurrency is ****constraining**** and ****hard to program**
***Vectors***** and *****streaming***** provide concurrency within a thread**
***Multithreading***** provides concurrency between threads**
**Exploit locality to reduce bandwidth demand**
**“****Heavyweight” processors (****HWPs****) to exploit temporal locality**
**“****Lightweight” processors (****LWPs****) to exploit spatial locality**
**Use other techniques to reduce network traffic**
**Atomic memory operations**
**Single word network transfers when no locality is present**

## Questions?

**Questions?**