# Presentation: 803209

## Evaluation of Modern Parallel Vector Architectures

- Lenny Oliker

## Parallel Vector Systems 

- 200K/yr , funded FY03-FY05
- The goal of this study is to answer strategic questions regarding the expected performance of DOE high-end computing applications on the latest generation of vector multiprocessors: the NEC SX6, the Cray X1, and the Japanese Earth Simulator System
- Examined: SX6, X1, ES compared with Power3, Power4, Altix, Columbia
- Measuring a set of microbenchmarks and numerical kernels
- Investigating key DOE applications from
- Fusion, Material Science, Astrophysics, Cosmology, Fluid Dynamics, Climate, *Combustion*
- Vectors odds with modern sparse/dynamic codes
- Vectors specialized arch extremely effective for restricted class of apps

## Results from 2003 ES visit

- Tremendous potential of vector architectures:  4 codes running faster than ever before
- Vector systems allows resolution not possible with scalar arch (regardless of # procs)
- Opportunity to perform scientific runs at unprecedented scale
- ES shows high raw and much higher sustained performance compared with X1
- Limited X1 specific optimization - optimal programming approach still unclear (CAF, etc)
*Non-vectorizable code segments become very expensive (8:1 or even 32:1 ratio)*
- Evaluation codes contain sufficient regularity in computation for high vector performance
- GTC example code at odds with data-parallelism
*Much more difficult to evaluate codes poorly suited for vectorization*
- Vectors potentially at odds w/ emerging techniques (irregular, multi-physics, multi-scale)
- Plan to expand scope of application domains/methods, and examine latest HPC architectures

| Code | (P=64)  % peak |  |  |  |  | (P=Max avail)   Speedup ES vs.  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Pwr3 | Pwr4 | Altix | ES | X1 | Pwr3 | Pwr4 | Altix | X1 |
| LBMHD | 7% | 5% | 11% | 58% | 37% | 30.6 | 15.3 | 7.2 | 1.5 |
| CACTUS | 6% | 11% | 7% | 34% | 6% | 45.0 | 5.1 | 6.4 | 4.0 |
| GTC | 9% | 6% | 5% | 20% | 11% | 9.4 | 4.3 | 4.1 | 1.1 |
| PARATEC | 57% | 33% | 54% | 58% | 20% | 8.2 | 3.9 | 1.4 | 3.9 |
| Average |  |  |  |  |  | 23.3 | 7.2 | 4.8 | 2.6 |

**Notes:**
With the emergence of the next generation of supercomputing systems - it is clear that no single architecture is best suited for the full scope of application domanins - it is therefore critical to understand which computational areas are best suited for a given architectural paradigm, to maximize scientific productivity.

## Updated from 2004 visit

- _Paratec_: ES achieves fastest performance to date -  5.5Tflop/s on 2048 procs
- Allows simulations at resolutions previously not possible
- Compare with 2.5Tflop/s from previous visit
- Performance increased from 31% to 45% of peak for 1024 processors
- _3D-LBMHD_: ES achieves highest performance to date: 22 Tflop/s for P=4096
- Compared with superscalar systems which achieve  ~10% of peak
- Potential for Gorden Bell submission at SC05
- _GTC_: Achieved highest performance to date – 3.7 Tflop/s on 2048 procs!
- Compared with ~100Gflop/s previous best performance
- Allows new generation of high resolution simulations
- _MADCAP: _Full application functionality available - 5.1 Tflop/s on 1024 procs
- Developed MADbench for architecture evaluation and portability
- _FVCAM: _Highest resolution climate atmospheric model using FV approach
- Achieved performance 10X Seaborg, with potential to utilize many more procs than spectral approach

## Next Step:Application Characterization

- As recently performed with MADbench analysis - develop systematic method of characterizing application workload requirements across various architectures
- Use data to seed Apex-Map/Sqmat experiments, to allow more effective use of architectural probes

## Parallel Vector SystemsAccomplishments

*SC2003*: with Canning (LBNL), Carter (LBNL), Shalf(LBNL), Skinner (LBNL), Ethier (PPPL) , Biswas (NASA), Djomehri (NASA), Wijngaart (NASA)
*Concurrency and Computation Journal*
*Vecpar04*: with Borrill, Canning, Carter, Shan, Skinner, Biswas, Djomehri
*SC2004* with Canning, Carter, Shalf, and Ethier (nominated best paper)
- Submitted to *Journal of Earth Simulator*
*HiPC2004*: with Carter and Borrill
*ICPP2005* submission: with Borrill, Carter, Skinner, Biswas
- Organized 2 session minisymposium at SIAM PP04:Performance Modeling and Evaluation of Ultra-Scale Systems
- Wehner, Biswas (NASA) , Wingaart (NASA) , Nakajima (RIST),  Snavely (SDSC), Kerbyson (LANL),  Das (U of Tex)
- Submitted survey book chapter based on *SIAMPP04* meeting
- Preparing SC05 publication based on second ES visit
- Custom (ES, X1) vs  Commodity (Thunder, Jackard)

## Future Work

- Formalize performance evaluation work as integral part of BIPS
- Create publicly available application benchmark distribution
- Perform detailed workload characterization
- Relate application characteristics to Apex-Map/Sqmat architeture probes
- Investigate latest HEC systems: BG/L, Columbia, SX8, Redstorm, X1e, Purple,  Rainier
- Expand application suite to include codes poorly suited for vectorization
- Perform production runs at ES at unprecedented problem resolution/scale (based on new MOU with ESC)