# Presentation: 669708

## Power, Cooling, and Energy Consumption for the Petascale and Beyond

- John Shalf: *Lawrence Berkeley National Laboratory*
- Bill Tschudi: *Lawrence Berkeley National Laboratory*
- Stephen Elbert: *Pacific Northwest National Laboratory*
- Rob Pennington: *National Center for Supercomputing Applications*
- Andres Marques: *Pacific Northwest National Laboratory*
- Tim McCann: *Silicon Graphics Inc.*
- Tahir Cader:* ISR Spray Cooling*

- Atlanta Technical Center

## Looming Power Crisis

- New Constraints
- Power limits clock rates
- Cannot squeeze more performance from ILP *(complex cores)* either!
- But Moore’s Law continues!
- What to do with all of those transistors if everything else is flat-lining?
- Now, #cores per chip doubles every 18 months *instead* of clock frequency!
**The “Free Lunch” is over!**

**Figure courtesy of Kunle Olukotun, Lance Hammond, Herb Sutter, and Burton Smith**

- Atlanta Technical Center

## Microprocessors: Up Against the Wall(s)

**Microprocessors: Up Against the Wall(s)**

- Microprocessors are hitting a power wall
- Higher clock rates and greater leakage increasing power consumption
- Reaching the limits of what non-heroic heat solutions can handle
- Newer technology becoming more difficult to produce, removing the previous trend of “free” power improvement

**From Joe Gebis**

- Atlanta Technical Center

**Notes:**
What is “heroic”?  Cooligy says that they can theoretically remove heat densities as high as 1000 watts per square cm.  That’s less than three doublings of where we’re at now, and Cooligy’s system includes advanced technology such as Microchannel Heat Collectors etched directly into the silicon, custom liquid-filled Radioator, and special Electrokinetic Pump.

## New Design Constraint: POWER

- Transistors still getting smaller
- Moore’s Law is alive and well
- But Dennard scaling is dead!
- No power efficiency improvements with smaller transistors
- No clock frequency scaling with smaller transistors
- All “magical improvement of silicon goodness” has ended
- Traditional methods for extracting more performance are well-mined
- Cannot expect exotic architectures to save us from the “power wall”
- Even resources of DARPA can only accelerate existing research prototypes (not “magic” new technology)!

- Atlanta Technical Center

## ORNL Computing Power and Cooling 2006 - 2011

- Immediate need to add 8 MW to prepare for 2007 installs of new systems
- NLCF petascale system could require an additional 10 MW by 2008
- Need total of 40-50 MW for projected systems by 2011
- Numbers just for computers: add 75% for cooling
- Cooling will require 12,000 – 15,000 tons of chiller capacity

- Cost estimates based on $0.05 kW/hr

- $3M

- $17M

- $9M

- $23M

- $31M

**O****AK**** R****IDGE**** N****ATIONAL**** L****ABORATORY**
**U. S. D****EPARTMENT**** ****OF**** E****NERGY**

- Data taken from Energy Management System-4 (EMS4). EMS4 is the DOE corporate system for collecting energy information from the sites. EMS4 is a web-based system that collects energy consumption and cost information for all energy sources used at each DOE site. Information is entered into EMS4 by the site and reviewed at Headquarters for accuracy.

- Yikes!

- Atlanta Technical Center

## Power Consumption by Top500 Systems

- Atlanta Technical Center

## Other Estimates of Power Requirements

- Baltimore Sun Article (Jan 23, 2007): NSA drawing **65-75 MW** in Maryland
- Crisis: Baltimore Gas & Electric does not have sufficient power for city of Baltimore!
- expected to increase by **10-15 MW** next year!
- LBNL IJHPCA Study for ~1/5 Exaflop for Climate Science in 2008
- Extrapolation of Blue Gene and AMD design trends
- Estimate: **20 MW **for BG and **179 MW** for AMD
- DOE E3 Report
- Extrapolation of existing design trends to exascale in 2016
- Estimate: **130 MW**
- DARPA Study
- More detailed assessment of component technologies
- Estimate: **20 MW** just for memory alone, **60 MW** aggregate extrapolated from current design trends
**The current approach is not sustainable!**

- Atlanta Technical Center

## Power is an Industry Wide Problem

**“****Hiding in Plain Sight, Google Seeks More Power”, **
**by John Markoff, June 14, 2006**

**New Google Plant in The Dulles, Oregon, **
**from NYT, June 14, 2006**

- Atlanta Technical Center

## How Big is the Problem?

- Estimated Computing Power Consumption
- 200 TWh/year
- $16 billion/year
- Based on .08$/KWh, closer to $.10 now (2005)
- Nearly 150 million tonsof CO2 per year
- Roughly equivalent to 30 million cars!

- One central baseload power plant (about 7 TWh/yr)

**Numbers represent U.S. only**

- Atlanta Technical Center

## Cost of Power Will Dominate, and Ultimately Limit Practical Scale of Future Systems

- Source: Luiz André Barroso, (Google) “The Price of Performance,” *ACM Queue*, Vol. 2, No. 7, pp. 48-53, September 2005. (Modified with permission.)

- Unrestrained IT power consumption could eclipse hardware costs and put great pressure on affordability, data center infrastructure, and the environment.

- Atlanta Technical Center

## Power Efficiency BoF

- Chip Architecture Trends for Power Efficient Computing
- Review Facility Design features for improved power and cooling efficiency
- Discuss cooling technology for future HPC system designs and its impact on facility design
- System architecture features to save power
- Discuss emerging energy efficiency standards and groups
- ASHRAE
- Green Grid
- Green500

- Atlanta Technical Center

## More Information

- All of the BoF Talks Online at
- _[http://esdc.pnl.gov](http://esdc.pnl.gov/)_
- More Information on Power Efficient Datacenters:
- _[http://hightech.lbl.gov/datacenters](http://hightech.lbl.gov/datacenters)_
- Computer Architecture
- _[http://view.eecs.berkeley.edu/](http://view.eecs.berkeley.edu/)_
- _[http://www.nersc.gov/projects/SDSA/reports](http://www.nersc.gov/projects/SDSA/reports)_
- Information / Metrics / Standards Bodies
- _[http://www.ashrae.org/](http://www.ashrae.org/)_
- _[http://www.thegreengrid.org/](http://www.thegreengrid.org/)_
- _[http://www.green500.org/](http://www.green500.org/)_
- _[http://www.80plus.org/](http://www.80plus.org/)_
- _[http://www.climatesaverscomputing.org/](http://www.climatesaverscomputing.org/)_

- Atlanta Technical Center

## Some Short Remarks on Computer Architecture Trends

- Atlanta Technical Center

## What is Happening Now?

- Moore’s Law
- Silicon lithography will improve by 2x every 18 months
- Double the number of transistors per chip every 18mo.
- CMOS Power
- Total Power = V2 * f * C  + V * Ileakage
-                                  active power           passive power
- As we reduce feature size Capacitance   ( C ) decreases proportionally to transistor size
- Enables increase of clock frequency ( f ) proportionally to Moore’s law lithography improvements, with same power use
- This is called “Fixed Voltage Clock Frequency Scaling” *(Borkar `99)*
- Since ~90nm
-  V2 * f * C  ~= V * Ileakage
- Can no longer take advantage of frequency scaling because passive power (V * Ileakage ) dominates
- Result is recent clock-frequency stall reflected in Patterson Graph at right

**SPEC_Int benchmark performance since 1978 from Patterson & Hennessy Vol 4.**

- Atlanta Technical Center

## What is Happening Now?

- Moore’s Law
- Silicon lithography will improve by 2x every 18 months
- Double the number of transistors per chip every 18mo.
- CMOS Power
- Total Power = V2 * f * C  + V * Ileakage
-                                  active power           passive power
- As we reduce feature size Capacitance   ( C ) decreases proportionally to transistor size
- Enables increase of clock frequency ( f ) proportionally to Moore’s law lithography improvements, with same power use
- This is called “Fixed Voltage Clock Frequency Scaling” *(Borkar `99)*
- Since ~90nm
-  V2 * f * C  ~= V * Ileakage
- Can no longer take advantage of frequency scaling because passive power (V * Ileakage ) dominates
- Result is recent clock-frequency stall reflected in Patterson Graph at right

**SPEC_Int benchmark performance since 1978 from Patterson & Hennessy Vol 4.**

**We are here!**

- Atlanta Technical Center

## Multicore vs. Manycore

**Multicore: current trajectory**
- Stay with current fastest core design
- Replicate every 18 months (2, 4, 8 . . . Etc...)
- Advantage: Do not alienate serial workload
- Example: AMD X2 (2 core), Intel Core2 Duo (2 cores), Madison (2 cores), AMD Barcelona (4 cores)
**Manycore: converging in this direction**
- Simplify cores (shorter pipelines, lower clock frequencies, in-order processing)
- Start at 100s of cores and replicate every 18 months
- Advantage: easier verification, defect tolerance, highest compute/surface-area, best power efficiency
- Examples: Cell SPE (8 cores), Nvidia G80 (128 cores), Intel Polaris (80 cores), Cisco/Tensilica Metro (188 cores)
**Convergence: Ultimately toward Manycore**
- Manycore *if we can figure out how to program it! *
- Hedge: Heterogenous Multicore

- Atlanta Technical Center

## How Small is “Small”

- Power5 (Server)
- 389mm^2
- 120W@1900MHz
- Intel Core2 sc (laptop)
- 130mm^2
- 15W@1000MHz
- ARM Cortex A8 (automobiles)
- 5mm^2
- 0.8W@800MHz
- Tensilica DP (cell phones / printers)
- 0.8mm^2
- 0.09W@600MHz
- Tensilica Xtensa (Cisco router)
- 0.32mm^2 for 3!
- 0.05W@600MHz

- Intel Core2

- ARM

- TensilicaDP

- Xtensa x 3

- Power 5

**Each core operates at 1/3 to 1/10th efficiency of largest chip, but you **
**can pack 100x more cores onto a chip and consume 1/20 the power**

- Atlanta Technical Center

## From Doug Carmean

**From Doug Carmean**
**Intel Inc.**

- Atlanta Technical Center

## Convergence of Platforms

- Multiple parallel general-purpose processors (GPPs)
- Multiple application-specific processors (ASPs)

***“******The Processor is the new Transistor” [Rowen]***

- Atlanta Technical Center

## Power Wall Drives Concurrency Increases

**Must ride exponential wave of increasing concurrency for forseeable future!**
**	****You will hit 1M cores sooner than you think!**

- Atlanta Technical Center

## Tension between concurrency and power efficiency

- Highly concurrent systems can be more power efficient
*Dynamic power is proportional to V**2**fC*
*Build systems with even higher concurrency?*
- However, many algorithms are unable to exploit massive concurrency yet
*If higher concurrency cannot deliver faster time to solution, then power efficiency benefit wasted *
*So we should build fewer/faster processors?*

- Atlanta Technical Center

## Path to Power EfficiencyReducing Waste in Computing

- Examine methodology of low-power embedded computing market
- optimized for low power, low cost, and high computational efficiency
***    “******Years of research in low-power embedded computing have shown only one design technique to reduce power:**** reduce waste.****”**** *
********* ******Mark Horowitz, Stanford University & Rambus Inc.***
- Sources of Waste
- Wasted transistors (surface area)
- Wasted computation (useless work/speculation/stalls)
- Wasted bandwidth (data movement)
- Designing for serial performance

- Atlanta Technical Center

**Notes:**
We propose an approach to HPC system design to meet new constraints
Technology scaling has stalled, need to focus on reducing waste in our systems!
Favor parallel throughput over per-core sequential performance
Exploit Specialization!
Optimize to specific applications
Can improve energy efficiency for a class of applications

We propose an approach to HPC system design to meet new constraints
Technology scaling has stalled, need to focus on reducing waste in our systems!
Favor parallel throughput over per-core sequential performance
Exploit Specialization!
Optimize to specific applications
Can improve energy efficiency for a class of applications

## Designing for Efficiency is Application Class Specific

- Atlanta Technical Center

## Consumer Electronics Convergence

**From: Tsugio Makimoto**

- Atlanta Technical Center

## Consumer Electronics has Replaced PCs as the Dominant Market Force in CPU Design!!

- Apple Introduces IPod

- IPod+ITunes exceeds 50% of Apple’s Net Profit

- Apple Introduces Cell Phone (iPhone)

**From: Tsugio Makimoto**

- Atlanta Technical Center

## BG/L—the Rise of the Embedded Processor?

**TOP 500 Performance by Architecture**

**1**

**10**

**100**

**1000**

**10000**

**100000**

**1000000**

**10000000**

**06/1993**

**MPP**
**SMP**
**Cluster**
**Constellations**
**Single Processor**
**SIMD**
**Others**
**MPP embedded**

**06/1994**

**06/1995**

**06/1996**

**06/1997**

**06/1998**

**06/1999**

**06/2000**

**06/2001**

**06/2002**

**06/2003**

**06/2004**

**06/2005**

**Aggregate R****max ****(Tflop/s)**

- Atlanta Technical Center

## Questions

- Is Multicore really the answer? (sounds boring)
-  FPGAs?  Quantum computing?
- What else might be waiting in the wings
- What about advances in circuit fabrication?
- SOI, Hafnium doping,
- What about memory?
- Its starting to consume more memory than CPU cores!
- Packaging changes (3D Stacking? Optical Interfaces?)

- Atlanta Technical Center

## Next UpDesigning Facilities for Power Efficiency

- Atlanta Technical Center