# Presentation: 028555

## Exa-ArchitecturesOpportunities and Challenges forApplications Scaling 

- Bill Camp
- Chief Architect/CTO, HPC
- April 22, 2008

## Legal Disclaimers

- Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit http://www.intel.com/performance/resources/limits.htm or call (U.S.) 1-800-628-8686 or 1-916-356-3104.
- All dates and products specified are for planning purposes only and are subject to change without notice
- Relative performance is calculated by assigning a baseline value of 1.0 to one benchmark result, and then dividing the actual benchmark result for the baseline platform into each of the specific benchmark results of each of the other platforms, and assigning them a relative performance number that correlates with the performance improvements reported.
- SPEC, SPECint2000, SPECfp2000, SPECint2006, SPECfp2006, SPECjbb, SPECWeb are trademarks of the Standard Performance Evaluation Corporation.  See http://www.spec.org for more information.
- Intel® Virtualization Technology requires a computer system with an enabled Intel® processor, BIOS, virtual machine monitor (VMM) and, for some uses, certain platform software enabled for it. Functionality, performance or other benefits will vary depending on hardware and software configurations and may require a BIOS update. Software applications may not be compatible with all operating systems. Please check with your application vendor.
- Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor series, not across different processor sequences. See http://www.intel.com/products/processor_number for details.
- Intel products are not intended for use in medical, life saving, life sustaining, critical control or safety systems, or in nuclear facility applications. All dates and products specified are for planning purposes only and are subject to change without notice
- * Other names and brands may be claimed as the property of others.
- Copyright © 2007-2008 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon and Intel Core are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## I cannot say it is the beginning of the end but it looks like the end of the beginning

- This generation will have be fortunate to have seen the birth of CMOS computing and its culmination
- Computing cannot solve any really hard problem; but it can inform our understanding of it
- What is needed now more than better programming models is more imagination and more math!
- Computing’s frontiers are already moving away from HPC– we need to learn to deal with it

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Good news:

- In 2011—14 systems can be built and delivered that achieve 20—50 PF on HPL
- In 2016—18 systems can be built that achieve 1000 PF on HPL

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Bad news:

- The road to a sustained ExaOps in 2018 has some potholes in it!

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Exascale Outlook

- By 2018: With reasonably expected advances in CMOS, optical and electrical signalling technologies:
- We will be able to achieve exa-ops computing for single problems and/or exa-ops capacity to carry our many calculations simultaneously.
- The programming environment can be an evolutionary outgrowth of current parallel computing approaches with increased use of hierarchical parallelism.
- Power, area and capital costs will be significantly higher than for today’s fastest systems:
- 	$1/2B 2008 dollars
- 	~1 hectare per exa-ops system
- 	~100 MW per system

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Core  and thread count: 

- order of 25 million large (vector) cores
- Or, order of 100--400 million small (scalar) cores
- The vector model will have 1--4 threads per core
- The scalar cores might have  anything from a few to very many threads per core
- In principle, the architecture could involve billions of threads if it is entirely small-core scalar with massive multi-threading

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Programming model

- Shared memory multi-threaded languages on-core, on-tile and on-die; and distributed memory MPI or non-cache coherent load-store between dies supporting PGAS languages
- MPI among cores on a tile and aggregated MPI among tiles on-die and among dies
- Multithreaded MPI on a core, MPI among threads on a tile and aggregated or non aggregated MPI among tiles
- Flat MPI among all cores uniformly.
- Other variations
- Functional languages on-die and distributed memory between dies
- Coherent Shared memory programming models (eg PRAM-like) beyond a socket are problematic

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## A point estimate of an Exa-architecture

**25,000—40,000 sockets**
**30--40 Teraflops per socket**
**1—2 Terabytes memory per socket**
**25 to 80 Petabytes total memory**
**1 Exabytes storage**
**50—100 Megawatts**
**Think 3-D toroidal mesh**

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Exa-architecture Concept 

- Vector
- Compute

- Fabric Interconnect

- Vector
- Compute

- Vector
- Compute

- Vector
- Compute

- Vector
- Compute

- Vector
- Compute

- Vector
- Compute

- Scalar
- Compute

- Vector
- Compute

- Vector
- Compute

- Vector
- Compute

- IO

**Exa-ops**

- _Conceptual View_

- _Connected View_

-  3-dimensional mesh/torus
-  Partitioned, scalable computing architecture

- Socket A

- Memory

- Socket B

- Memory

- Memory

- FC

- N-GigE

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## A Little More Architectural Detail

- 1000 Cores per socket
- 	32—48 Gigaflops per core
- 	16 floating point or integer operations per core per cycle
- 	2-- 3 Gigahertz
- 32—48 Teraflops per socket
- 1—2 gigabytes per core: 1—2 Terabytes per socket
- 3-D toroidal mesh interconnect (+ multi-stage global interconnect?)
- 3—5 Terabytes per second bi-directional BW off and onto socket

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Even More Architectural Detail

- 1000 Cores per socket
- 	distributed L1 and L2 caches
- 	Shared L3 cache(?)
- Main memory “on package” with socket.
- DRAM will be a virtual disk.
- 	It will be large but not in proportion to peak speed
- “Disks” will  (likely SSD) be large but not in proportion to peak speed.
- I/O will support both streaming and informatics workloads

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Yet more architectural detail

- Global and local support for “Fetch and ...” atomicity
- Light weight synchronization within a core, within a socket, and globally
- Aggressive multi-threading in the spirit of MTA
- Direct addressability of all memory in the machine (but not cache coherency)

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## When we have many tera-ops processorsWe’ll need many TB/s Memory systems

- Arithmetic unit

- registers

- L1
- I$
- L1
- D$

- L2$

- DRAM

- This neglects multi-core and coherency issues

- FD
- MC

- Fast DRAM

- “Package”

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Disclaimer

**The following “theorems” are my observations based on more than 20 years experience programming, leading teams of parallel programmers, and developing architectures.**
**They in no way represent the position of my employer or of many otherwise reasonable people ******

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Programming models

- Theorem 1:
- 	MPI works just fine for scientific codes
- 	MPI works better than fine for Map-Reduce and Dynamic Master –	Slave task-level parallelism
- 	MPI works great for outer looping over codes for system ID and 	optimizing
- 	When Open-MP works, MPI works as well or better
- Theorem 2:
- 	PGAS languages play an important role:
- 		their performance can equal that of MPI codes
- 		For most humans, they are much easier to learn
- 		They can grow organically

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Programming models

- Theorem 3:
- 	True shared memory parallelism will not work at large numbers 	of threads until synchronization cost is negligible, memory contention 	is a negligible cost, and compilers are divinely inspired
- Corollary 3.1
- 	 Forget PRAM models
- Theorem 4:
- 	Functional languages and Multi-threaded languages can do things 	efficiently in graph problems and other large data problems that are 	difficult and inefficient using any messaging-passing paradigm to date.
- Theorem 5:
- 	Nonetheless, these languages remain obscure and difficult  targets for	compilation 		.

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Programming models (an easy problem)

- Look at a “typical” 3-d scientific calculation based on a 2nd order FD stencil:
- On “our” architecture we can represent 109 computational cells per socket with say 100 variables per cell and 100 operations per variable per “iteration/timestep.”
- Then a “timestep” requires 100 Tera-ops per socket
- If we sustain 10 tera-ops per socket, a TS will require 10 secs.
- We will have to send and receive 100 GB per TS per socket. At 5 TB/sec and 50% efficiency we can move this is 0.4 secs or 4% overhead. This seems reasonable for what is a less-communication intense application

- Scientific presentation of the author's opinions-- not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Programming model

- Question: Do we roll all communication up into a few socket—socket “fluxes” or do we let individual “surface” cores send messages individually?
- For this problem either way looks ok to me.
- What if we add MHD and radiation transport to the problem as we would in fusion theory?
- 	How do we divide the problem up?
- What if we look at a large system that has the above, has neutronics, has thermal-hydraulics, has materials dynamics, has aging effects, has radiation damage, ...?
- How will we parallelize the problem at this scale?
- What if we want to run an outer loop system ID or optimizer? What is the most productive approach?
- ARE THERE PROBLEMS FOR WHICH A HIERARCHICAL (FRACTAL) APPROACH IS CLEARLY SUPERIOR OR INDEED NECESSARY?

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Programming model

- These systems will have multiple failures in the timeframe of any major job:
- 	It is unrealistic to expect the HW to bear the entire resiliency 	workload.
- At 8 nm process technology, it will be harder to tell a 1 from a 0
- 	There are going to be more DUE’s and more SDC.
- We can and should make our SW and algorithms more resilient:
- 	mix task parallelism (e.g. master–slave report-back) with 	data-parallel approaches
- 	replace direct solvers with fault-tolerant iterative and	stochastically iterative methods.

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## Programming model

- Will “jitter” limit scalability?
- 	OS
- 	error recovery
- 	multi-threading vagaries
- When will communications become the limit?
- Are we using enough imagination about how to use these machines?

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007

## The end

- Scientific presentation of the author's opinions-- it is not an indication of any plans or intentions on the part of Intel Corporation

- 7/28/2007