# Presentation: 043328

## Programming Environments with and beyond MPI

- Steve Deitz
- 6/4/2008

## Preliminary Notes

- Slide

**Preliminary Notes**

- Background
- ZPL
- Data parallelism (beyond SIMD)
- Syntactically transparent communication
- Academic project but with some encouraging performance results
- Chapel
- Nested task and data parallelism (completely general)
- Separates processors from tasks and exposes locality
- Distributions separate implementation from algorithm
- Supports advanced “serial” language concepts
- Disclaimer
- Opinions expressed are my own, not those of Cray Inc.

## Programming Environment Challenges

- Slide

**Programming Environment Challenges**

- Goal of programming environments
- Hide the complexity of a system to make the user more productive
- Abstract for portability
- Support data-parallel computations
- Manage data decomposition details
- Manage replication, communication, and synchronization
- Hide local vs. global indexing issues
- Support task-parallel computations
- Support invocation of local and remote tasks
- Manage or ameliorate task migration and load balancing
- Separation of concerns
- How easy is it to change the data distribution? n? p?
- How easy is it to make a dense array sparse?

## MPI Strengths and Weaknesses

- Slide

**MPI Strengths and Weaknesses**

- Strengths
- It’s a proven system and an industry standard
- It’s portable
- It’s relatively easy to port and implement
- It’s been used to achieve results with near-optimal performance
- Weaknesses
- It encodes too much information about data communication
- This may not match what an architecture supports
- It’s restricted to parallelism at the “executable level”
- Applications and architectures contain parallelism at many levels
- It doesn’t provide any “separation of concerns”
- Change the data distribution
- Relax assumptions about “n” and “p”
- It doesn’t provide high-level “global-view” abstractions

## 1 When will MPI-only performance be inadequate?

- Slide

**1 When will MPI-only performance be inadequate?**

- “At least by 2000”

- ZPL out-scales F+MPI
- Uses SHMEM on T3E
- Uses MPI on clusters
- Similar observations have been made more recently

## 2 What alternate programming models are best?

- Slide

**2 What alternate programming models are best?**

- MPI+OpenMP or MPI+pThreads
- Does this address light-weight inter-processor synchronization?
- Does this address single-sided messages?
- Co-Array Fortran
- Key abstraction: co-dimensions
- Data and control still must be fragmented
- Synchronization is “less free” than in MPI
- Unified Parallel C
- Supports shared data and, weakly, distributions
- Data can be global (not fragmented), but control still is
- Communication is no longer transparent in the syntax
- Chapel
- Supports general parallelism and global-view data parallelism
- Communication is not transparent
- Adoption risk and large leap

## Why was ZPL “successful?”

- Focused
- Not a general solution, but it could excel
- This was also a major detriment
- Portable
- It was portable because of MPI!
- ZPL was designed so that it could be implemented on two-sided message passing systems.
- Chapel and more general parallel languages cannot be.
- We are using pThreads+GASNet , pThreads+ARMCI, ...
- Performance
- Matched MPI performance because it generated MPI.
- Exceeded MPI by generating SHMEM on the T3E (for example).
- Real success will be more difficult
- Performance, robustness, portability, programmability, ...
- Like applications, need a portable, low-level implementation mechanism, a next-generation MPI

- Slide

## Evolving “MPI”

- Slide

**Evolving “MPI”**

*“**I don't know what language they will be using in*
*high performance computers 10 years from now*
*but I know it will be called FORTRAN.”*
- MPI Extensions
- Flexible one-sided messages
- Memory allocation
- Thread-safe and thread-extended
- Remote thread invocation
- Light-weight synchronization
- MPI Versions
- Implementation-level version for building libraries and languages
- User-level version for writing applications

## 3 What algorithmic developments are necessary?

- Slide

**3 What algorithmic developments are necessary?**

- or, How can programming environments help?
- Separability of concerns
- General parallel programming model
- Higher level abstractions (with lower level hooks)