# Presentation: 149101

## From the Heroic to the Logistical Programming Model Implicationsof New Supercomputing Applications 

**Ian Foster**
- Computation InstituteArgonne National Laboratory &The University of Chicago

- With thanks to: **Miron Livny**, **Ioan Raicu**, **Mike Wilde**, **Yong Zhao**, and many others.

- What will we do with 1+ Exaflops and 1M+ cores?

## 1) Tackle Bigger and Bigger Problems

- Computational
- Scientist
- as **Hero**

## 2) Tackle Increasingly Complex Problems

- Computational
- Scientist
- as **Logistics**
**Officer**

## “More Complex Problems”

- Use ensemble runs to quantify **climate model uncertainty**
- Identify **potential drug targets** by screening a database of ligand structures against target proteins
- Study **economic model sensitivity** to key parameters
- Analyze **turbulence dataset** from multiple perspectives
- Perform **numerical optimization** to determine optimal resource assignment in energy problems
- Mine collection of data from **advanced light sources**
- Construct databases of computed properties of **chemical compounds**
- Analyze data from the **Large Hadron Collider**
- Analyze** log data** from 100,000-node parallel computations

## Programming Model Issues

- Massive **task parallelism**
- Massive **data parallelism**
- Integrating **black box applications**
- Complex **task dependencies** (task graphs)
**Failure**, and other execution management issues
**Data management**: input, intermediate, output
**Dynamic task graphs**
**Dynamic data access** involving large amounts of data
**Long-running** computations
- Documenting **provenance** of data products

## Problem Types

- Number of tasks

- Input
- data
- size

- 1                        1K                       1M

- Hi
- Med
- Lo

- HeroicMPI
- tasks

- Dataanalysis,mining

- Many loosely coupled tasks

- Much data and complex tasks

## An Incomplete and Simplistic View ofProgramming Models and Tools

**Many Tasks**
- DAGMan+Pegasus
- Karajan+Swift

**Much Data**
- MapReduce/Hadoop
- Dryad

**Complex Tasks, Much Data**
- Dryad, Pig, Sawzall
- Swift+Falkon

**Single task, modest data**
- MPI, etc., etc., etc.

**Notes:**
We can of course use MPI in each case. But not such an obvious fit.

MapReduce is good for many tasks against much data

Complex tasks against much data  Dryad, Swift.

## Many TasksClimate Ensemble Simulations(Using FOAM,2005)

- Image courtesy Pat Behling and Yun Liu, UW Madison

- NCAR computer + grad student
- 160 ensemble members in 75 days

- TeraGrid + “Virtual Data System”250 ensemble members in 4 days

***Many Tasks******Climate ******Ensemble ******Simulations*****(Using FOAM,****2005)**

## Many Many Tasks:Identifying Potential Drug Targets

-  2M+ ligands

- Protein        x
-  target(s)

- (Mike Kubal, Benoit Roux, and others)

**Notes:**
"docking" is the identification of the low-energy binding modes of a small molecule (ligands) within the active site of a macromolecule (receptor) whose structure is known 
A compound that interacts strongly with (i.e. binds) a receptor associated with a disease may inhibit its function and thus act as a drug  
Typical Workload: 
Application Size: 7MB (static binary)
Static input data: 35MB (binary and ASCII text)
Dynamic input data:10KB (ASCII text)
Output data: 10KB (ASCII text)
Expected execution time: 5~5000 seconds
Parameter space: 1 billion tasks 

## 4 million tasks500K cpu-hrs

- start

- report

- DOCK6
- Receptor
- (1 per protein:
- defines pocket
- to bind to)

- ZINC
- 3-D
- structures

- ligands

- complexes

- NAB scriptparameters
- (defines flexible
- residues,
- #MDsteps)

- Amber Score:
- 1. AmberizeLigand
- 3. AmberizeComplex
- 5. RunNABScript

- end

- BuildNABScript

- NAB
- Script

- NAB
- Script
- Template

- Amber prep:
- 2. AmberizeReceptor
- 4. perl: gen nabscript

- FRED
- Receptor
- (1 per protein:
- defines pocket
- to bind to)

- Manually prep
- DOCK6 rec file

- Manually prep
- FRED rec file

- 1 protein
- (1MB)

- PDB
- protein
- descriptions

- 4 million tasks500K cpu-hrs

**Notes:**
More precisely, step 3 is “GCMC + hydration.”

Mike Kubal say: “This task is a Free Energy Perturbation computation using the Grand Canonical Monte Carlo algorithm for modeling the transition of the ligand (compound) between different potential states and the General Solvent Boundary Partition to explicitly model the water molecules in the volume around the ligand and pocket of the protein. The result is a binding energy just like the task at the top of the funnel; it is just a more rigorous attempt to model the actual interaction of protein and compound. To refer to the task in short hand, you can use "GCMC + hydration". This is a method that Benoit has pioneered.”

## DOCK on SiCortex

- CPU cores: 5760
- Tasks: 92160
- Elapsed time: 12821 sec
- Compute time: 1.94 CPU years
- Average task time: 660.3 sec

- (does not include ~800 sec to stage input data)

- Ioan Raicu,
- Zhao Zhang

## MARS Economic Model Parameter Study

***MARS Economic Model ******Parameter Study***

- 2,048 BG/P CPU cores
- Tasks: 49,152
- Micro-tasks: 7,077,888
- Elapsed time: 1,601 secs
- CPU Hours: 894

- Mike Wilde, Zhao Zhang

**Notes:**
(Data cached in RAM disk.)

Impact of varying Low-S Light LL and Med-S Light LL production yields for Diesel fuel on Discount Investment

## B. Berriman, J. Good (Caltech)

- B. Berriman, J. Good (Caltech)
- J. Jacob, D. Katz (JPL)

**Notes:**
The workflow shows the most basic montage workflow. This one does not show the other mAdd stage (mSubset -> mAdd ->mImgtbl ) -> mAdd 

## Montage in MPIand Swift

***Montage in MPI******and Swift***

- (Yong Zhao, Ioan Raicu, U.Chicago)

**MPI**: ~950 lines of C for one stage
**Pegasus**: ~1200 lines of C + tools to generate  DAG for specific dataset
**SwiftScript**: ~92 lines for any dataset

**Notes:**
This is a mid-sized montage workflow for M16 3x3 square degrees, with 442 images been co-added.
The GRAM submission was run with clustering enabled (a maximum of 30 jobs bundled, so use close to 16 nodes).   
Both MPI and Falkon used 16 nodes.
All resources were requested from PBS.
The workflow involves 13 stages, we show only the parallel stages in the graph, the other stages are all one-task serial execution. GRAM had more overhead for executing such stages, where MPI and Falkon were almost the same.
The last stage mAdd was parallelized in the MPI version, but not for the version for GRAM or Falkon.
We did do stage-in in the workflow, the MPI run did not involve data transfer.

## MapReduce/Hadoop

***MapReduce/Hadoop***

- Hadoop DFS Architecture

- ALCF: 80 TB memory,
- 8 PB disk,
- 78 GB/s I/O bandwidth

- Soner Balkir, Jing Tie, Quan Pham

## Extreme Scale Debugging: Stack Trace Sampling Tool (STAT)

***Extreme Scale Debugging: ******Stack Trace Sampling Tool (STAT)***

**Cost per sample on BlueGene/L**

- Bart Miller, Wisconsin

**Notes:**
Cost per stack-trace sample on the BG/L at LLNL.  The different curves represent different MRNet tree topologies.  This data is a from a VERY rough early run of the code, though still has excellent results.

NOTE that Totalview takes 2 MINUTES on just 4096 processes!

## Summary

- Peta- and exa-scale computers enable us to tackle new types of problems at far greater scales than before
- Parameter studies, ensembles, interactive data analysis, “workflows” of various kinds
- Potentially an important source of new applications
- Such apps frequently stress petascale hardware and software in interesting ways
- New programming models and tools are required
- Mixed task and data parallelism, management of many tasks, complex data management, failure, ...
- Tools for such problems (DAGman, Swift, Hadoop, ...) exist but need refinement
- Interesting connections to distributed systems community

**More info: www.ci.uchicago.edu/swift**

## Amiga Mars – Swift+Falkon

- 1024 Tasks (147456 micro-tasks)
- 256 CPU cores