# Presentation: 355315

## Pity the Poor Peta-Flops Programmer

**Dr. Richard Loft**
**Computational Science Section**
**Scientific Computing Division**
**National Center for Atmospheric Research**
**Boulder, CO USA**

**But not too much!**

**November 18, 2003**

## The Demand for Petaflops: High Cost of Climate Science Goals

**Climate scientists project a need for ****150x**** more computing power of the next ****5 years****.**
**Doubling ****horizontal resolution increases computational cost ****eightfold****.**
**Many additional constituents will be advected.**
**New physics: computational cost of CAM/CCM, ****holding resolution constant****, has increased ****4x ****since 1996. More coming...**
**Future: introducing super-parameterizations of moist processes would  increase physics costs dramatically ****(another x120).**

## Observed IBM Cluster Efficiencies

| System | Application efficiency
(% of peak) |
| --- | --- |
| bluesk8 | 4.1% |
| bluesk32 | 4.5% |
| blackforest | 5.7% |

**Newer systems are less efficient.**
**Larger nodes are more efficient.**
**Max sustained performance: ~450 GFlops**

## Slide 4

## Towards Scalable Climate Models

**Current dynamical core limits scalability**
**Based on global transpositions**
**New core based on spectral elements **
**Scales better (local exchanges)**
**More efficient (~16% of peak on RISC)**
**Runs on 1000’s of pe’s at 2/3o resolution**
**Cache, thread, message optimizations**
**Classic “heroic” effort with current tools.**
**Sustained PFlops: only a factor of 2000 to go!**

## Slide 6

## What are the Biggest Challenges you foresee in using Petaflops architectures?

**Petaflops Diagnosis:**
**Complexity Complex**
**Architectural angst**
**Performance Anxiety**
**Memory Latency Syndrome (MLS)**
**Bisected Bandwidth**
**Reduction disorder**
**Data Storage/Analysis paralysis**

## Complexity Complex

**Systems are getting more complex**
**Processor (CMP, SMT, etc)**
**Depth of memory heirarchies**
**Applications are getting more complex**
**Application heirarchies**
**Adaptive/dynamical algorithms**
**People are dumb/smart as ever**

## Architectural angst:What will it look like?

- RISC?

- VECTOR?

**MUTANT?**

## More Architectural Angst

**Assume the worst - you’ll be right!**
**COTS hardware**
**MPI library**
**“****print” statement**

## More Architectural Angst

**Code does not turn on a dime.**
**Rewriting models takes about 5 years**
**Changing algorithms about 10 years**
**Both are disruptive to science**
**Both cost $$$**
**Automation: Are we reaching the limits of the human programmer? **
**Source translation**
**Source generation**
**Algorithm selection**

## Performance Anxiety

**Lose the peak Flops mentality.**
**Some metrics I’d like to see:**
**Local memory bandwidth of system**
**Aggregate of parallel Streams.**
**Local memory latency**
**Random index chase.**
**System bisection bandwidth **
**All to all transpose (MB/s/proc)**
**System-wide Latency**
**global reduction of short vector.**
**global barrier synchronization.**

## Data Paralysis

**Data Storage/Analysis/Visualization**
**Storage costs $$$ : **
**Tools for analyzing data will break!**
**Sharing data difficult (LAN/Grid/Fedex)**
**Just compute on fly. Well, exploring data sets and discovering new things is part of science.**** *****Computing on the fly***** ****is only good if you *****already know***** the signal you are looking for.**

## What do Programming Models Need to provide you to exploit Pflops Architectures productively?

**The programming model should abstract away *****irrelevant***** architectural details without abstracting away performance.**
**Too much *****arcane and ephemeral knowledge***** is  required to get at performance: this applies to the system administration, user environment, language-compiler interactions, and to run-time libraries.**
**It should be easy to get *****correct and***** *****reproducible answers*****. Little in current paradigms to encourage this.**
**It should be easier to diagnose what is going on.**

## What do Programming Models Need to provide you to exploit Pflops Architectures productively? (cont)

**The programming model needs to provide a *****clear performance API***** for both local and non-local operations.**
**The parallel abstraction must not be too *****restrictive*****. In this respect, HPF was bad. **
**Allow for asynchronous communication.**
**Quasi-regular grids (icosahedra/cube topologies)**
**Transpositions/regriddings**
**Unstructured/adaptive grids**
**Parallel I/O Model**
**Should be a full functioning OO language.**

## What do Programming Models Need to provide you to exploit Pflops Architectures productively? (cont)

**Whole business of HPC languages seems trapped at the *****academic threshold*****.**
**Needs to be a substantial developer/user community, perhaps captive, from the beginning.**
**Linux open source model for domain specific languages?**
**The programming model needs to *****persist*****. Algorithmic improvements, changes to production models are decadal processes, much longer than a CS PhD thesis (hopefully).**

## What constructs or abstractions would make it easier for you to map your applications on PFlops architectures?

**OK we have local memory hierarchies, can we build smart software/hardware to deal with them? **
**Fine grained thread/relaxed coherency model is interesting. **
**Efficient and flexible grid constructs. **

##  n8 = n - MODULO(n,8)
     do j=1,n,8
        s0=y(j)
        s1=y(j+1)
        s2=y(j+2)
        s3=y(j+3)
        s4=y(j+4)
        s5=y(j+5)
        s6=y(j+6)
        s7=y(j+7)
        do i=1,n
           s0 = s0 + x(i)*a(i,j  )
           s1 = s1 + x(i)*a(i,j+1)
           s2 = s2 + x(i)*a(i,j+2)
           s3 = s3 + x(i)*a(i,j+3)
           s4 = s4 + x(i)*a(i,j+4)
           s5 = s5 + x(i)*a(i,j+5)
           s6 = s6 + x(i)*a(i,j+6)
           s7 = s7 + x(i)*a(i,j+7)
        end do
        y(j)   = s0
        y(j+1) = s1
        y(j+2) = s2
        y(j+3) = s3
        y(j+4) = s4
        y(j+5) = s5
        y(j+6) = s6
        y(j+7) = s7
     end do
     do j=n8+1,n
        y(j) = y(j) + x(i)*a(i,j)
     end do

-      do j=1,n,8
-         s0=y(j)
-         s1=y(j+1)
-         s2=y(j+2)
-         s3=y(j+3)
-         s4=y(j+4)
-         s5=y(j+5)
-         s6=y(j+6)
-         s7=y(j+7)
-         do i=1,n
-            s0 = s0 + x(i)*a(i,j  )
-            s1 = s1 + x(i)*a(i,j+1)
-            s2 = s2 + x(i)*a(i,j+2)
-            s3 = s3 + x(i)*a(i,j+3)
-            s4 = s4 + x(i)*a(i,j+4)
-            s5 = s5 + x(i)*a(i,j+5)
-            s6 = s6 + x(i)*a(i,j+6)
-            s7 = s7 + x(i)*a(i,j+7)
-         end do
-         y(j)   = s0
-         y(j+1) = s1
-         y(j+2) = s2
-         y(j+3) = s3
-         y(j+4) = s4
-         y(j+5) = s5
-         y(j+6) = s6
-         y(j+7) = s7
-      end do
-      do j=n8+1,n
-         y(j) = y(j) + x(i)*a(i,j)
-      end do

-  do j=1,n
-       do i=1,n
-            y(j) = y(j) + x(i)*a(i,j)
-       end do
-  end do

**Which code would you rather write?**

## At which point in the maturity of a new programming language would you be willing to try a new promising model?

**At the very beginning.**
**At the very end.**
**The ugly middle’s the part to avoid!**
**Seriously, in my experience these partnerships are hard to get funded.**
**NCAR has tried with ZPL and EARTH. **