# Presentation: 433359

## Next-generation scalable applications: When MPI-only is not enough

June 3, 2008

Paul S. Crozier

Sandia National Labs

**June 3, 2008**
**Paul S. Crozier**
**Sandia National Labs**

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

**How we use MPI to perform scalable parallel molecular dynamics**** **

## Why MD begs to be run on parallel supercomputers:
Very popular method
Lots of useful information produced
Fundamentally limited by: force field accuracy and compute power (hardware/software)

Where’s the expensive part?
Force calculations
Neighborhood list creation
Long-range electrostatics (if needed)

**Very popular method**
**Lots of useful information produced**
**Fundamentally limited by: force field accuracy and compute power (hardware/software)**
**Where’s the expensive part?**
**Force calculations**
**Neighborhood list creation**
**Long-range electrostatics (if needed)**

**Molecular dynamics on parallel supercomputers**

## Our MD codes

**miniMD: **_**[http://software.sandia.gov/mantevo/packages.html](http://software.sandia.gov/mantevo/packages.html)**_** **
**part of Mantevo project**
**currently in licensing process (LGPL)**
**intended to be extremely simple parallel MD code that enables rapid portability and rewriting for use on novel architechtures**
**LAMMPS:  **_**[http://lammps.sandia.gov](http://lammps.sandia.gov/)**_** **
**( Large-scale Atomic/Molecular Massively Parallel Simulator )**

## Classical MD code.
Open source, highly portable C++.
Freely available for download under GPL.
Easy to download, install, and run.
Well documented.
Easy to modify or extend with new features and functionality.
Active user’s e-mail list with over 300 subscribers.
Since Sept. 2004: over 20k downloads, grown from 53 to 125 kloc.
Spatial-decomposition of simulation domain for parallelism.
Energy minimization via conjugate-gradient relaxation.
Radiation damage and two temperature model (TTM) simulations.
Atomistic, mesoscale, and coarse-grain simulations.
Variety of potentials (including many-body and coarse-grain).
Variety of boundary conditions, constraints, etc.

- Open source, highly portable C++.
- Freely available for download under GPL.
- Easy to download, install, and run.
- Well documented.
- Easy to modify or extend with new features and functionality.
- Active user’s e-mail list with over 300 subscribers.
- Since Sept. 2004: over 20k downloads, grown from 53 to 125 kloc.
- Spatial-decomposition of simulation domain for parallelism.
- Energy minimization via conjugate-gradient relaxation.
- Radiation damage and two temperature model (TTM) simulations.
- Atomistic, mesoscale, and coarse-grain simulations.
- Variety of potentials (including many-body and coarse-grain).
- Variety of boundary conditions, constraints, etc.

- LAMMPS
- (Large-scale Atomic/Molecular Massively Parallel Simulator)
- Steve Plimpton, Aidan Thompson, Paul Crozier
- _[lammps.sandia.gov](http://lammps.sandia.gov/)_

## Slide 5

## Possible parallel decompositions

**Atom:**** each processor owns a fixed subset of atoms.**
**	****requires all-to-all communication**
**Force:**** each processor owns a fixed subset of interatomic forces.**
**	****scales as O(N/sqrt(P))**
**Spatial:**** each processor owns a fixed spatial region. **
**	****scales as O(N/P)**
**Hybrid: ****typically a combination of spatial and force decompositions.**
**Other?**

- S. J. Plimpton, **Fast Parallel Algorithms for Short-Range Molecular Dynamics**, J Comp Phys, 117, 1-19 (1995).

## Parallelism via Spatial-Decomposition

**Physical domain divided into 3d boxes, one per processor**
**Each processor computes forces on atoms in its box**
**using info from nearby procs**
**Atoms "carry along" molecular topology**
**as they migrate to new procs**
**Communication via**
**nearest-neighbor 6-way stencil**
**Optimal N/P scaling for MD,**
**so long as load-balanced**
**Computation scales as N/P**
**Communication scales as N/P**
**for large problems (or better or worse)**
**Memory scales as N/P**

## Parallel Performance

**Fixed-size (32K atoms) and scaled-size (32K atoms/proc) parallel efficiencies**
**Protein (rhodopsin) in solvated lipid bilayer **
**Billions of atoms on 64K procs of Blue Gene or Red Storm**
**Opteron processor speed: 4.5E-5 sec/atom/step **
**(12x for metal, 25x for LJ)**

## Particle-mesh Methods for Coulombics

**Coulomb interactions fall off as  1/r  so require long-range for accuracy**
**Particle-mesh methods:**
** ****	****partition into short-range and long-range contributions**
**	****short-range via direct pairwise interactions**
**	****long-range:**
**	****	****interpolate atomic charge to 3d mesh**
**	****	****solve Poisson's equation on mesh (4 FFTs)**
**	****	****interpolate E-fields back to atoms**
**FFTs scale as NlogN if cutoff is held fixed**

## Parallel FFTs

**3d FFT is 3 sets of 1d FFTs**
**	****in parallel, 3d grid is distributed across procs**
**	****perform 1d FFTs on-processor**
**	****	****native library or FFTW (**_**[www.fftw.org](http://www.fftw.org/)**_**)**
**	****1d FFTs, transpose, 1d FFTs, transpose, ...**
**	****	****"transpose” = data transfer**
**	****	****transfer of entire grid is costly**
**FFTs for PPPM can scale poorly**
**	****on large # of procs and on clusters**
**Good news: Cost of PPPM is only ~2x more than 8-10 Angstrom cutoff**

## Could we do better with non-traditional hardware, or something other than MPI?

**Alternative hardware:**
**Multicore**
**GPGPUs**
**Vector machine**
**Non-MPI message passing: **
**OpenMP**
**Special-purpose message passing**
**Other suggestions?**

## Comments on morning discussion

**We really want strong scaling --- get out longer on the time axis.**
**Rather simulate a membrane protein for ms than a whole cell for a ns.**
**We want an MD algorithm that performs well on N < P.**
**LAMMPS’s spatial decomposition breaks down for N < P.   **
**We’ll need a new parallel MD algorithm for speedup along the time axis for problems where N < P.**
**We’ve looked at OpenMP vs MPI and see no advantage on LAMMPS as it stands right now.**
**Many in the MD community now looking at doing MD on “novel architechtures” --- GPUs, multicore mahcines, etc.**
**“****Canned” particle pusher middleware seems difficult. Needs many different options and flavors.**

## MPI Applications: How We Use MPI
Tuesday, June 3, 1:45-3:00 pm 

Paul S. Crozier

Sandia National Labs

**Tuesday, June 3, 1:45-3:00 pm**** **
**Paul S. Crozier**
**Sandia National Labs**

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

**How we use MPI to perform scalable parallel molecular dynamics**** **