# Presentation: 028556

## Why not MPI-only Applications?A Case for Investigating Hybrid Parallelism

**H. Carter Edwards**
**Sandia National Laboratory**
**Sandia CSRI Workshop on**
**Next generation scalable applications:**
**When MPI-only is not enough**
**June 3-5, 2008**

## Expected HPC Environment

**Distributed memory parallelism will always be with us**
**Network of processing nodes**
**Well understood programming model, e.g. MPI**
**Processing nodes will have multiple cores**
**Cores-per-node = cores-per-socket * sockets-per-node**
**Apps must scale with #cores = #nodes * cores-per-node**
**My concerns**
**Cores contending for memory resources**
**Application’s per-socket or per-core memory overhead**
**Application’s non-deterministic parallel behavior (not discussed)**

## Expected HPC EnvironmentMy Concerns for MPI-Only

**Unmanaged contention for node’s memory resources**
**Cores contend for access to memory hierarchy**
**Cores contend for socket’s cache memory**
**MPI-only has no provision for coordinating access**
**How much will this limit on-node scalability?**
**At the mercy of the memory subsystem performance**
**Can intentional coordinated access improve scalability?**
**Could lead to increased complexity in the application**
**Per-node memory overhead**
**Will OS require each core to have its own executable image?**
**Intra-node distributed memory parallelism requires shared / ghosted data and message passing buffers**

## Conclusion: We Need to Investigate Hybrid Parallel Programming Model(s)

**Two level programming model; orthogonal parallelism**
**Outer inter-socket: distributed memory / MPI parallelism**
**Inner  intra-socket: shared memory / thread parallelism**
**Impact to scalability and performance?**
**Impact to complexity and robustness?**
**My investigation**
**Application programmer interface**
**Simple and minimalistic**
**Flexible for non-uniform parallel work**
**Performance parameters**
**thread “flight control”**
**memory access patterns**

## Simple Programming Model for Inner Level Parallelism

**Task pool / work queue strategy (old paradigm) **
**Sequential operations performed by a single thread**
**Inner level parallel operations performed by all threads**
**Inner level parallel operations have a local and temporary scope**
**Conceptually compatible with OpenMP and TBB model **

**root**
**thread**

## Scaling of double-double ‘dot(x,y)’

**Scaling of double-double ‘dot(x,y)’**

**Barcelona (AMD 2x4core) with OpenMPI**
**Hybrid parallel:  #Processes = MPI*Pthreads**
**Threads always in flight – no blocking**
**MPI_Allreduce overhead is minor; Scaling is great**

- 8

- 4

- 2

- 1

## Scaling of double-double ‘dot(x,y)’

**Clovertown (Intel 2x4core) with MPICH**
**Hybrid parallel:  #Processes = MPI*Pthreads**
**Threads always in-flight – no blocking**
**MPI_Allreduce overhead is awful.  Memory bandwidth saturates.**

- 1

- 2

- 4

- 8

## Scaling of double-double ‘dot(x,y)’

**Barcelona (AMD 2x4core) with OpenMPI**
**Hybrid parallel:  #Processes = MPI*Pthreads**
**Threads blocked between flights**

- 8

- 4

- 2

- 1

## Thread “Flight Control”

**MPI-Only**
**One thread in flight for each MPI process**
**A thread only blocks when waiting for receive (common usage) – when needed by the algorithm**
**Components / libraries share single thread resource – MPI_Comm**
**Non-MPI / Hybrid inner loop parallelism**
**Thread start / stop blocking a performance concern**
**Required for number active threads > number of cores**
**Mixed threading mechanisms, e.g. Pthreads+TBB+OpenMP+...**
**Rather not block threads – unless needed by algorithm**
**Need a shared thread resource analogous to MPI_Comm**
**Choose a single low-level mechanism ~ MPI**

## Summary

**Expect networks of manycore nodes**
**Apps required to scale with respect to core count**
**If MPI-only, critical to have multicore leveraging implementation **
**MPI-only or Hybrid Parallelism?**
**Hybrid may be necessary to address memory access contention**
**Hybrid can reduce inter-core communication overhead**
**Hybrid provides opportunities for intra-node load balancing via work-queue **
**Thread “flight control”**
**Performance (and portability) may require choosing a single low-level mechanism analogous to MPI but for intra-socket parallelism**