# Presentation: 345091

## The Cray X1 Multiprocessor and Roadmap

## Cray’s Computing Vision

**Cray’s Computing Vision**

**X1**

**‘****Strider 2’**

**X1E**

**‘****Black Widow’**

**2006**

**‘****Strider 3’**

**2006**

**‘****Black Widow 2’**

***Scalable High-Bandwidth Computing***

**Product Integration**

**RS**

**Red Storm**

**‘****Strider X’**

**2010**
**‘****Cascade’**
**Sustained**
**Petaflops**

**2004**

**2004**

**2005**

## Cray  X1

**Cray PVP**
-  Powerful vector processors
-  Very high memory bandwidth
-  Non-unit stride computation
-  Special ISA features
-  Modernized the ISA

**T3E**
-  Extreme scalability
-  Optimized communication
-  Memory hierarchy
-  Synchronization features
-  Improved via vectors

**Extreme scalability with high bandwidth vector processors**

**Cray  X1**

**Notes:**
ISA = Instruction Set Architecture; the instructions implemented in the system

## Cray X1 Instruction Set Architecture

- New ISA
- Much larger register set (32x64 vector, 64+64 scalar)
- All operations performed under mask
- 64- and 32-bit memory and IEEE arithmetic
- Integrated synchronization features
- Advantages of a vector ISA
- Compiler provides useful dependence information to hardware
- Very high single processor performance with low complexity
-        ops/sec = (cycles/sec) * (instrs/cycle) * (ops/instr)
- Localized computation on processor chip
- – large register state with very regular access patterns
- – registers and functional units grouped into local clusters (pipes)
-  excellent fit with future IC technology
- Latency tolerance and pipelining to memory/network
- very well suited for scalable systems
- Easy extensibility to next generation implementation

## New instruction set
New system architecture
New processor microarchitecture

“Classic” vector machines were programmed differently
Classic vector:  Optimize for loop length with little regard for locality
Scalable micro: Optimize for locality with little regard for loop length

The Cray X1 is programmed like other parallel machines
Rewards locality: register, cache, local memory, remote memory
Decoupled microarchitecture performs well on short loop nests
(however, does require vectorizable code)

- New system architecture
- New processor microarchitecture
- “Classic” vector machines were programmed *differently*
- Classic vector:  Optimize for loop length with little regard for locality
- Scalable micro: Optimize for locality with little regard for loop length
- The Cray X1 is programmed like other parallel machines
- Rewards locality: register, cache, local memory, remote memory
- Decoupled microarchitecture performs well on short loop nests
- (however, *does* require vectorizable code)

**Not Your Father’s Vector Machine**

## Cray X1 Node

-  Four multistream processors (MSPs), each 12.8 Gflops
-  High bandwidth local shared memory (128 Direct Rambus channels)
-  32 network links and four I/O links per node

**51 Gflops, 200 GB/s**

**Cray X1 Node**

## NUMA Scalable up to 1024 Nodes

**Global shared memory across machine **

**Interconnection**
**Network**

**NUMA Scalable up to 1024 Nodes**

**Notes:**
This slide was created by Steve Scott.

## Network Topology (16 CPUs)

- M15

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

**P**

- M0

- M0

- M1

- M1

- M15

- M0

- M0

- M1

- M1

- M15

- M15

- node 0

- node 1

- node 2

- node 3

- Section 0

- Section 1

- Section 15

**Network Topology (16 CPUs)**

**Notes:**
Interconnect diagram of an air-cooled cabinet (or tiny liquid-cooled system).  Strictly cabling between M chips on each node.   Maximum of two hops from any node to any other.

## Network Topology (128 CPUs)

**16**
**links**

**Notes:**
Example of two LC cabinets.  The “R” boxes are router modules that handle the interconnect beyond 8 nodes and between cabinets.

## Network Topology (512 CPUs)

**Network Topology (512 CPUs)**

**Notes:**
Even larger system 8 full LC cabinets.

## Designed for Scalability

- Distributed shared memory (DSM) architecture
- Low latency, load/store access to entire machine (tens of TBs)
- Decoupled vector memory architecture for latency tolerance
- Thousands of outstanding references, flexible addressing
- Very high performance network
- High bandwidth, fine-grained transfers
- Same router as Origin 3000, but 32 parallel copies of the network
- Architectural features for scalability
- Remote address translation
- Global coherence protocol optimized for distributed memory
- Fast synchronization
- Parallel I/O scales with system size

**T3E Heritage**

## Decoupled Microarchitecture

- Decoupled access/execute  *and*  decoupled scalar/vector
- Scalar unit runs ahead, doing addressing and control
- Scalar and vector loads issued early
- Store addresses computed and saved for later use
- Operations queued and executed later when data arrives
- Hardware dynamically unrolls loops
- Scalar unit goes on to issue next loop before current loop has completed
- Full scalar register renaming through 8-deep shadow registers
- Vector loads renamed through load buffers
- Special sync operations keep pipeline full, even across barriers
- This is *key* to making the system *like* short-VL code

## Address Translation

- High translation bandwidth: scalar + four vector translations per cycle per P
- Source translation using 256 entry TLBs with support for multiple page sizes
- Remote (hierarchical) translation:
- allows each node to manage its own memory (eases memory mgmt.)
- TLB only needs to hold translations for one node  *scales*

## Cache Coherence

- Global coherence, but only cache memory from local node (8-32 GB)
- Supports SMP-style codes up to 4 MSP (16 SSP)
- References outside this domain converted to non-allocate
- Scalable codes use explicit communication anyway
- Keeps directory entry and protocol simple
- Explicit cache allocation control
- Per instruction hint for vector references
- Per page hint for scalar references
- Use non-allocating refs to avoid cache pollution
- Coherence directory stored on the M chips (rather than in DRAM)
- Low latency and *really* high bandwidth to support vectors
- Typical CC system: 1 directory update per proc per 100 or 200 ns
- Cray X1: 3.2 dir updates per MSP per ns (factor of several hundred!)

## System Software

## Cray X1 I/O Subsystem

**Cray X1 I/O Subsystem**

**X1 Node**

**I/O Board**

**SPC**
**Channels**
**2 x 1.2 GB/s**

**PCI-X**
**BUSES**
**800 MB/s**

**PCI-X**
**Cards**

**200 MBytes/s FC-AL**

**IP over Fiber**

**CPES**

**CNS**

**Gigabit Ethernet or Hippi Network**

**CNS**

**Gigabit Ethernet or Hippi Network**

**RS200**

**I Chip**

**I Chip**

**XFS and XVM**
**ADIC SNFS**

## Cray X1 UNICOS/mp

**UNIX kernel executes**
**on each *****Application***** node**
**(somewhat like ChorusTM **
**microkernel on UNICOS/mk)**
**Provides:  SSI, scaling & **
**resiliency**

**Commands launch**
**applications like on**
**T3E (mpprun)**

***System Service***** nodes provide**
**file services, process manage-**
**ment and other basic UNIX**
**functionality (like /mk servers). **
**User commands execute on **
***System Service***** nodes.   **

**UNICOS/mp **
**Global resource**
**manager (Psched)**
**schedules**
**applications on **
**Cray X1 nodes**

**256 CPU Cray X1**

**Single System Image**

## Programming Models

- Shared memory:
- OpenMP, pthreads
- Single node (51 Gflops, 8-32 GB)
- Fortran and C
- Distributed memory:
- MPI
- shmem(), UPC, Co-array Fortran
- Hierarchical models (DM over OpenMP)
- Both MSP and SSP execution modes supported for all programming models

## Mechanical Design

## Node board

**Node board**

**Field-Replaceable Memory**
**Daughter Cards**

**Network**
**Interconnect**

**Spray Cooling Caps**

**Air Cooling Manifold **

_**PCB**_
**17” x 22’’**

**CPU MCM**
**8 chips**

## Cray X1 Node Module

## Cray X1 Chassis 

## 64 Processor Cray X1 System~820 Gflops

**64 Processor Cray X1 System****~820***** Gflops***

## Cray BlackWidow

- The Next Generation HPC
-  System From Cray Inc.

## BlackWidow Highlights

- Designed as a follow-on to Cray X1:
- Upward compatible ISA
- FCS in 2006
- Significant improvement (>> Moore’s Law rate) in:
- Single thread scalar performance
- Price performance
- Refine X1 implementation:
- Continue to support DSM with 4-way SMP nodes
- Further improve sustained memory bandwidth
- Improve scalability up *and* down
- Improve reliability/fault tolerance
- Enhance instruction set
- A few BlackWidow features:
- Single chip vector microprocessor
- Hybrid electrical/optical interconnect
- Configurable memory capacity, memory BW and network BW

## Cascade ProjectCray Inc.StanfordCal Tech/JPLNotre Dame

## High Productivity Computing Systems

**High Productivity Computing Systems**

**Goals:**
**Provide a new generation of economically viable high productivity computing systems for the national security and industrial user community (2007 – 2010)**

**Impact:**
**Performance **(efficiency): critical national security applications by a factor of 10X to 40X
**Productivity** (time-to-solution)
**Portability** (transparency): insulate research and operational application software from system
**Robustness** (reliability): apply all known techniques to **protect against outside attacks**, hardware faults, & programming errors

**Fill the Critical Technology and Capability Gap**
**Today (late 80’s HPC technology).....*****to*****.....Future (Quantum/Bio Computing)**

**Applications:**
- Intelligence/surveillance, reconnaissance, cryptanalysis, weapons analysis, airborne contaminant modeling and biotechnology

**HPCS Program Focus Areas**

**Notes:**
Mission:

Provide a focused research and development program, creating new generations of high end programming environments, software tools, architectures, and hardware components in order to realize a new vision of high end computing, high productivity computing systems (HPCS). Address the issues of low efficiency, scalability, software tools and environments, and growing physical constraints.
Fill the high end computing between today’s late 80’s based technology High Performance Computing (HPCs) and the promise of quantum computing.
 Provide economically viable high productivity computing systems for the national security and industrial user communities with the following design attributes in the latter part of this decade:
Performance: Improve the computational efficiency and performance of critical national security applications.
Productivity: Reduce the cost of developing, operating, and maintaining HPCS application solutions.
Portability: Insulate research and operational HPCS application software from system specifics.
Robustness:  Deliver improved reliability to HPCS users and reduce risk of malicious activities.

Background:

High performance computing is at a critical juncture.  Over the past three decades, this important technology area has provided crucial superior computational capability for many important national security applications.  Government research, including substantial DoD investments, has enabled major advances in computing, contributing to the U.S. dominance of the world computer market. Unfortunately, current trends in commercial high performance computing, future complementary metal oxide semiconductor (CMOS) technology challenges, and emerging threats are creating technology gaps that threaten continued U.S. superiority in important national security applications.

As reported in recent DoD studies, there is a national security requirement for high productivity computing systems. Without government R&D and participation, high-end computing will be available only through commodity manufacturers primarily focused on mass-market consumer and business needs.  This solution would be ineffective for important national security applications.

The HPCS program will significantly contribute to DoD and industry information superiority in the following critical applications areas: operational weather and ocean forecasting; planning exercises related to analysis of the dispersion of airborne contaminants; cryptanalysis; weapons (warheads and penetrators); survivability/stealth design; intelligence/surveillance/reconnaissance systems; virtual manufacturing/failure analysis of large aircraft, ships, and

## HPCS Planned ProgramPhases 1-3

**HPCS Planned Program****Phases 1-3**

- 02

- 05

- 06

- 07

- 08

- 09

- 10

- 03

- 04

- Products

- Metrics,

- Benchmarks

- Academia

- Research

- Platforms

- Early

- Software

- Tools

- Early

- Pilot

- Platforms

**Phase 2**
**R&D**

**Phase 3 **
**Full Scale Development**
**(Planned)**

- Metrics and Benchmarks

- System
- Design
- Review

**Industry Guided**
**Research**

**Application Analysis**
**Performance**
**Assessment**

**HPCS**
**Capability or**
**Products**

- Fiscal Year

- Concept
- Reviews

- PDR &
- Early
- Prototype

- Research

- Research
- Prototypes
- & Pilot Systems

- Phase 3 Readiness Review

- Technology
- Assessments

- Requirements
- and Metrics

- Phase 2
- Readiness Reviews

**Phase 1**
**Industry**
**Concept Study**

- Reviews
- Industry BAA/RFP
- Critical Program
- Milestones

## Technology Focus of Cascade

- System Design
- Network topology and router design
- Shared memory for low latency/low overhead communication
- Processor design
- Exploring clustered vectors, multithreading and streams
- Enhancing temporal locality via deeper memory hierarchies
- Improving latency tolerance and memory/communication concurrency
- Lightweight threads in memory
- For exploitation of spatial locality with little temporal locality
- Thread  data instead of data  thread
- Exploring use of PIMs
- Programming environments and compilers
- High productivity languages
- Compiler support for heavyweight/lightweight thread decomposition
- Parallel programming performance analysis and debugging
- Operating systems
- Scalability to tens of thousands of processors
- Robustness and fault tolerance

*Lets us explore technologies we otherwise couldn’t.*
*A three year head start on typical development cycle.*

## Cray’s Computing Vision

**Cray’s Computing Vision**

**X1**

**‘****Strider 2’**

**X1E**

**‘****Black Widow’**

**2006**

**‘****Strider 3’**

**2006**

**‘****Black Widow 2’**

***Scalable High-Bandwidth Computing***

**Product Integration**

**RS**

**Red Storm**

**‘****Strider X’**

**2010**
**‘****Cascade’**
**Sustained**
**Petaflops**

**2004**

**2004**

**2005**

## Questions?

**Questions?**