# Presentation: 177788

## 64-Bit Floating-Point Accelerators for HPC Applications

**John L. Gustafson, Ph.D.**
**Chief Technology Officer, HPC**
**ClearSpeed Technology, Inc.**

## Outline

**Acceleration issues in general: host-accelerator bandwidth/latency**** **
**Accelerator performance analysis examples**
**Measured and expected application acceleration:**
- Molecular Dynamics: AMBER and NAB
- Quantum Chemistry: GAUSSIAN, Qbox, PARATEC
- Monte Carlo models for PDEs
- LS-DYNA and ANSYS
- PAM-CRASH
- MATLAB applications
**Summary**

## Thesis

**Performance analysis for accelerator cards is like analysis for message-passing parallelism, but with more levels of memory and communication.**
**Application porting success depends heavily on attention to memory bandwidths, but (surprisingly) not so much the host-accelerator bandwidth. **

## The accelerator idea is as old as supercomputing itself

- Even in 1977, HPC users faced issues of when it makes sense to use floating-point-intensive vector hardware.

**General-purpose computer**
**Runs OS, compilers, disk, printers, user interface**

- “History doesn’t repeat itself, but it does rhyme.”—*Mark Twain*

## IDC survey of planned HPC accelerator use

## Is this trip necessary? Bandwidth issues

**Acceleration software tests candidates for work on the board. If too small, it leaves them on the host.**
**Performance claims *****must *****assume host-resident data. Beware of benchmarks that leave out the time to move the data to accelerator memory.**
**Remember Bailey’s *****12 Ways...***

**Node**

**Node memory**

**Accelerator**

**Accelerator memory**

- Bandwidth = *B*

## Simple offload model is out of date

**Accelerator must be quite fast for this approach to have benefit**
**This “mental picture” may stem from early days of Intel 80x87, Motorola 6888x math coprocessors**

- latency

***Host***

- latency

***Host***

***Accelerator***

- band-width

- band-width

## Acceleration model: Host continues working

**Accelerator needs only be fast enough to make up for time lost to bandwidth + latency**
**Easiest use model: host and accelerator share the same task, load balanced to complete at same time**
**More flexible: Host, accelerator *****specialize***** what they do**

- latency

***Host***

- latency

***Host***

***Host***

***Accelerator***

- band-width

- band-width

## Host can work while data is moved

**Host can work while data is moved**

**PCI transfers might burden a *****single x86 core***** by 60%**
**Other cores on host continue productive work at full speed**

- latency

***Host***

- latency

***Host***

***Accelerator***

- band-width

- band-width

***Host***

## Card need not wait for all data before starting

**Card need not wait for *****all***** data before starting**

**In practice, latency is *****microseconds*****; the accelerator task takes *****seconds*****. Latency gaps above would be microscopic if drawn to scale.**
**The accelerator can be *****slower***** than the host, and still add performance!**

***Accelerator***

- latency

***Host***

- latency

***Host***

- band-width

- band-width

***Accelerator***

## Square DGEMM speeds as of December 2006

**Square DGEMM speeds as of December 2006**

- Doubling host-to-card bandwidth has minor effect because of I/O overlap and large task grain size.
- A zero-latency connection would not visibly affect either curve.

**2000**

**4000**

**6000**

**8000**

**10000**

**10**

**20**

**30**

**40**

**50**

**60**

**70**

**PCI-X**

- Note: curve only samples integer multiples of vector size

**Matrix size**

**GFLOPS**

## Accelerator memory hierarchy

- Total: 1.0 GB

- Total: 6.4 GB/s

- Total: 1.1 MB

- Total: 192 GB/s

- Total: 24 KB

- Total: 2 TB/s

- Total: 96 GFLOPS
- (but only 25 watts)

## Visualization of algorithm overhead: DGEMM

- Matrix multiply (DGEMM) is a perfect analog to a folded box.

## Build up the bricks through the hierarchy

**Build up the bricks through the hierarchy**

## Tier 3 overhead was originally harder to overlap

**Tier 3 overhead was originally harder to overlap**

- Simple but decent accelerator performance model:

## Unoptimized Tier 3 timing

***K*****=960, *****M*****=1920, *****N*****=1920**

## Optimized Tier 3 timing

- Can now accelerate matrices as small as N=576

## Almost doubled sustained speed

**Almost doubled sustained speed**

**2000**

**4000**

**6000**

**10**

**20**

**30**

**40**

**50**

**60**

**70**

***Previous DGEMM***
***asymptotic to ~56 GFLOPS***

- Does not include host contribution, which adds 5 to 60 GFLOPS depending on host

**Matrix size**

**GFLOPS**

- Note: curve only samples integer multiples of vector size

## Detailed profiling is essential for accelerator tuning

- Advance Accelerator Board

- Host
- CPU(s)

- Host
- CPU(s)

- Host
- CPU(s)

**Detailed profiling is essential for accelerator tuning**

- Advance Accelerator Board

- Host
- CPU(s)

- CSX600

- Pipeline

**HOST/BOARD INTERACTION**
- View host/board interactions
- Measure transfer bandwidth.
- Check overlap of host and board compute

**ACCELERATOR PIPE**
- View instruction issue.
- Visualize overlap of executing instructions. Get cycle-accurate timing.
- Remove instruction-level performance bottlenecks.

**CSX600 SYSTEM**
- Trace at system level.
- Inspect overlap of compute and I/O.
- View cache utilization.Graph performance.

- ClearSpeed
- Accelerated
- System

- CSX600

- Pipeline

**HOST CODE PROFILING**
- Visually inspect multiple host threads.
- Time specific code sections.
- Check overlap of host threads

- PCIe/X

## Pipeline view of DGEMM inner loop

**Pipeline view of DGEMM inner loop**

- Profile the code running at the instruction level

- See the pipeline performance for each instruction

- Tune the instruction scheduling for the application code

## System level: multiple DGEMM calls

**System level: multiple DGEMM calls**

- View the DGEMM calls on host and accelerator

- Much higher level of detail available from the profiler

- Each call syncs up with the host view of accelerator

## More geometric analogs: equation solving

- Ax = b

**More geometric analogs: equation solving**

*N* equations

*N* unknowns

- Dongarra’s new LAPACK will make QR, Cholesky, LU factoring, etc. much easier to accelerate.

- Volume = 1⁄3 *N*3multiply-adds

- Excellent test of hardware correctness

## Accelerated cluster model accurate to about ±5%

**Accelerated cluster model accurate to about ±5%**

*N**B* = Block size, the width of the “panels” used to update the linear system with DGEMM
*N* = Dimension of the linear system (number of equations to solve)
*P*, *Q* = Dimensions of the two-dimensional mapping of computational nodes
** = Effective point-to-point latency of MPI broadcast, in seconds
**= Effective point-to-point reciprocal bandwidth of message broadcast, in seconds per datum
**= Effective floating-point operations per second of a node independent of MPI operations

## Memory bandwidth dominates performance model

**Multicore x86**

**Node memory**

**Accelerator**

**Accelerator DRAM**

- PCIx or PCIe
- 1 to 2 GB/s

**Apps that can stage into local RAM (Tier 1) can go 10x faster than current high-end Intel, AMD hosts**
**Apps that must reside in DRAM (Tier 2) will actually run *****slower***** by about 3x (for fully optimized host code)**
**Fast Fourier Transforms can go either way!**

**Accelerator**
**Local RAM**

## Math functions reside at Tier 1, hence fast

**Typical speedup of ~8X over the fastest x86 processors, because math functions stay in the local memory on the card**

**64-bit Function Operations per Second (Billions)**

- 0.0

- 0.5

- 1.0

- 1.5

- 2.0

- 2.5

- Sqrt

- InvSqrt

- Exp

- Ln

- Cos

- Sin

- SinCos

- Inv

**Function name**

- 2.6 GHz dual-core Opteron

- 3 GHz dual-core Woodcrest

- ClearSpeed Advance card

## Monte Carlo PDE methods exploit Tier 1 bandwidth

**Monte Carlo PDE methods exploit Tier 1 bandwidth**

**No acceleration:****	****200M samples, 79 seconds**
**1 accelerator:****	****200M samples, 3.6 seconds**
**5 accelerators:****	****200M samples, 0.7 seconds**

**Real apps do work resembling “EP” of NAS Parallel Benchmarks. “Quants” solve PDEs this way for options pricing,****Black-Scholes model (a form of the Heat Equation)**

## Why do EP-type Monte Carlo apps need 64-bit?

**Accuracy increases as the square root of the number of trials, so five-decimal accuracy takes 10 billion trials.**
**But, when you *****sum***** many similar values, you start to scrape off all the significant digits.**
**64-bit summation needed to get a single-precision result!**

- Single precision:
- 1.0000x108 + 1= 1.0000x108
- Double precision:
- 1.0000x108 + 1 = 1.00000001x108

## We may need to rethink 64-bit flops...

**We may need to rethink 64-bit flops...**

**Every operation has an optimum number of bits of accuracy**
- Using too few gives unacceptable errors
- Using too many wastes memory, bandwidth, joules, dollars.
**It is unlikely that a code uses *****just the right amount***** of precision needed.**

- IEEE 754 double precision

- Optimum precision

- All floating-point operations in application

## How do HPC programmers pick FP precision?

**Assume 64-bit is plenty, and use it everywhere.**
**Use what is imposed by hardware (word size).**
**Try two precisions; if answers agree, use the less precise one, otherwise use the more precise one.**
**Compare computed answer for special cases where an analytic answer is known.**
**Compare computed answer with physical experiment (rare).**
**Perform careful analysis (very rare).**

## Can a tool estimate joules, W, $, min. precision?

**a(i,j) -= a(i,k) * a(k,j)**

- Cost and electrical power and precision are almost as important as timing... why not develop analysis tools for them? You can only optimize what you can measure.

## Model/measure power use, not just performance

- Base System: HP DL380 G5, Intel Xeon 5160 x 2 @ 3GHz, 14GB

## Accelerated Cluster Node

- Base System: HP DL380 G5, Intel Xeon 5160 x 2 @ 3GHz, 14GB
- Accelerated System:  As above + 1 ClearSpeed Advance X620

## Energy Usage: Standard vs. Accelerated

- Base System: HP DL380 G5, Intel Xeon 5160 x 2 @ 3GHz, 14GB
- Accelerated System:  As above + 1 ClearSpeed Advance X620

**53.6% Less Energy Used**

## AMBER 9 acceleration

**Model: memory motion is *****k*****1*****N*****, operations are *****k*****2*****N****** *****2****. Overheads easily overcome for typical *****N*****.**
**~4x speedup for pharmaceutical company production runs achieved recently.**
**225 hour (Opteron) run reduced to 58 hours**
**Didn’t exploit symmetry of *****i j***** forces, which will give another ~1.5–1.9x speedup (31–39 hours)**
**Solvation model GB1; 500000 time steps**
**Correctness checked incrementally throughout conversion process. **

## NAB and AMBER 10 acceleration

**Newton-Raphson refinement now possible; analytically-computed second derivatives**
**2.6x speedup obtained for this operation in three hours of effort (no source code changes)**
**Enables accurate computation of entropy and Gibbs free energy for first time.**
**Available now in NAB (Nucleic Acid Builder) code. Slated for addition to AMBER 10.**

## Plug-and-play quantum chemistry acceleration?

**DGEMM content is 18% to 65% in GAUSSIAN test suite, but typical sizes only ~10 to 100.**
**No changes to license or to any of the source code. Just invoke dynamic linking option in makefile.**
**Sample GAUSSIAN tests to date are *****too small***** to accelerate; below *****N *****= 576 threshold.**
- Need larger problem sizes
- Realistic to be that large? (Lots of occupied orbitals)
**PARATEC, Qbox much better candidates. Plane wave models are over half DGEMM, huge dimensions**

## The economics of CAE acceleration

**Each host costs $3,000.**
**Software license costs ~$30,000 *****per core*****, which discourages use of multiple cores*****.***
**MCAE engineer costs over $200,000/year.**
**In California, anyway.**
**Accelerator card would be cost-effective even with a 7% performance boost. Actual performance boost should be more like 260% for large problems.**

**Structural Analysis**
**ANSYS, LS-DYNA implicit**

## Accelerating ANSYS, LS-DYNA with Lucas’ solver

**Potentially pure plug-and-play**
**No added license fee**
**Needs ClearSpeed’s 64-bit precision and speed**
**Enabled by recent DGEMM improvements; still needs symmetric *****A*****T*****A***** variant**
**Could enable some CFD acceleration (for codes based on finite elements, low Reynolds numbers`)**

- 10 million degrees of freedom (sparse)

## PAM-CRASH and MATLAB acceleration

**Work done in China shows 1.40x PAM-CRASH speedup using one ClearSpeed accelerator**
**We await details; this is preliminary**
- Problem size?
- Explicit or implicit?
- What was done to the code?
- Compared to what host?
**Japanese industrial researcher got 5x acceleration of MATLAB waveguide model; won’t allow publication of results (?)**
********sigh********** So we continue citing TOP500 results...**

## Summary

**Accelerator tuning demands attention to memory bandwidth at all levels (bandwidth to host, less so)**
**Now seeing value for real 64-bit applications in chemistry, electromagnetics, financial modelling, crash codes, etc.**
**Fit-for-purpose analysis starts with analytical model based on memory tiers, but is verified using detailed performance tools.**