# Presentation: 710681

## Robust and High Performance Tools for Scientific Computing The DOE ACTS Collection

- Robust and High Performance Tools for Scientific Computing *The DOE ACTS Collection*

**Tony Drummond and Osni Marques**
- UC Berkeley - CS-267
- October 10, 2002

## What is the DOE ACTS Collection?

**What is the DOE ACTS Collection?**

_**A**_dvanced _**C**_ompu_**T**_ational _**S**_oftware
- Tools for developing parallel applications
- Developed (primarily) at DOE Labs
- Separate projects originally
- ~ 20 tools
- ACTS is an “umbrella” project
- Leverage numerous independently funded projects
- Collect tools in a toolkit

***http://acts.nersc.gov***

## ACTS: Project Goals

**ACTS: *****Project Goals***

- Extended support for *experimental software*
- Make ACTS tools available on DOE computers
- Provide technical support (_[acts-support@](http://www.mgnet.org/mgnet-codes-gmd.html)__[nersc](http://www.mgnet.org/mgnet-codes-gmd.html)__[.](http://www.mgnet.org/mgnet-codes-gmd.html)__[gov](http://www.mgnet.org/mgnet-codes-gmd.html)_)
- Maintain ACTS information center (_[http://acts.](http://www.mgnet.org/mgnet-codes-gmd.html)__[nersc](http://www.mgnet.org/mgnet-codes-gmd.html)__[.](http://www.mgnet.org/mgnet-codes-gmd.html)__[gov](http://www.mgnet.org/mgnet-codes-gmd.html)_)
- Coordinate efforts with other supercomputing centers
- Enable large scale scientific applications
- Educate and train

## Related Activities

**Related Activities**

**Software Repositories**:
**Netlib**:*  *_[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[netlib](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org](http://www.mgnet.org/mgnet-codes-gmd.html)_
**HPC-Netlib**:*  *_[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[nhse](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org/](http://www.mgnet.org/mgnet-codes-gmd.html)__[hpc](http://www.mgnet.org/mgnet-codes-gmd.html)__[-](http://www.mgnet.org/mgnet-codes-gmd.html)__[netlib](http://www.mgnet.org/mgnet-codes-gmd.html)_
**National HPCC Software Exchange NHSE**:*  *_[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[nhse](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org](http://www.mgnet.org/mgnet-codes-gmd.html)_
**Guide to Available Mathematical Software**:*  *_[http://](http://www.mgnet.org/mgnet-codes-gmd.html)__[gams](http://www.mgnet.org/mgnet-codes-gmd.html)__[.](http://www.mgnet.org/mgnet-codes-gmd.html)__[nist](http://www.mgnet.org/mgnet-codes-gmd.html)__[.gov](http://www.mgnet.org/mgnet-codes-gmd.html)_
**MGNet**:*  *_[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[mgnet](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org](http://www.mgnet.org/mgnet-codes-gmd.html)_* *
**NEOS:***  *_[http://www-](http://www.mgnet.org/mgnet-codes-gmd.html)__[fp](http://www.mgnet.org/mgnet-codes-gmd.html)__[.](http://www.mgnet.org/mgnet-codes-gmd.html)__[mcs](http://www.mgnet.org/mgnet-codes-gmd.html)__[.](http://www.mgnet.org/mgnet-codes-gmd.html)__[anl](http://www.mgnet.org/mgnet-codes-gmd.html)__[.gov/](http://www.mgnet.org/mgnet-codes-gmd.html)__[otc](http://www.mgnet.org/mgnet-codes-gmd.html)__[/Guide](http://www.mgnet.org/mgnet-codes-gmd.html)_* *
**OO Numerics**:  _[http://](http://www.mgnet.org/mgnet-codes-gmd.html)__[oonumerics](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org/](http://www.mgnet.org/mgnet-codes-gmd.html)__[oon](http://www.mgnet.org/mgnet-codes-gmd.html)_
**Portable timing routines, tools for debugging, compiler technologies:**
**Ptools**:*  *_[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[ptools](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org](http://www.mgnet.org/mgnet-codes-gmd.html)_
**Center for Programming Models for Scalable Parallel Computing**:* *_[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[pmodels](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org](http://www.mgnet.org/mgnet-codes-gmd.html)_* *
**Education:**
**Computational Science Educational Project**:  _[http://csep1.](http://www.mgnet.org/mgnet-codes-gmd.html)__[phy](http://www.mgnet.org/mgnet-codes-gmd.html)__[.](http://www.mgnet.org/mgnet-codes-gmd.html)__[ornl](http://www.mgnet.org/mgnet-codes-gmd.html)__[.gov](http://www.mgnet.org/mgnet-codes-gmd.html)_* *
**UCB’s Applications of Parallel Computers**:*  *_[http://www.cs.berkeley.edu/~demmel/cs267_Spr99](http://www.mgnet.org/mgnet-codes-gmd.html)_
**MIT’s Applied Parallel Computing**: * *_[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[mit](http://www.mgnet.org/mgnet-codes-gmd.html)__[.](http://www.mgnet.org/mgnet-codes-gmd.html)__[edu](http://www.mgnet.org/mgnet-codes-gmd.html)__[/~](http://www.mgnet.org/mgnet-codes-gmd.html)__[cly](http://www.mgnet.org/mgnet-codes-gmd.html)__[/18.337](http://www.mgnet.org/mgnet-codes-gmd.html)_
**Dictionary of algorithms, data structures and related definitions**: _[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[nist](http://www.mgnet.org/mgnet-codes-gmd.html)__[.gov/dads](http://www.mgnet.org/mgnet-codes-gmd.html)_

## Why is ACTS unique?

- Extended support for tools
- Accumulates the expertise and user feedback on the use of the software tools and scientific applications that used them:
- independent software evaluations
- participation in the developer user groups e-mail list
- presentation of a gallery of applications
- leverage between tool developers and tool users
- workshops and tutorials
- tool classification

## ACTS: levels of support

- ACTS: *levels of support*

***High***
- Intermediate level
- Tool expertise
- Conduct tutorials

***Intermediate***
- Basic level
- Provide a higher level of support to users of the tool

***Basic***
- Basic knowledge of the tools
- Help with installation
- Compilation of user’s reports (*acts-support@nersc.gov*)

## http://acts.nersc.gov

**http://acts.nersc.gov**

## Large Scientific Codes: 

***A Common Programming Practice***

**Tuned and machine**
**Dependent  modules**

**Application**
**Data Layout**

**Control**

**I/O**

**Algorithmic Implementations**

## Using today's hardware to tackle today's Grand Challenges

- ---
- Q.  Why is it **still **difficult to obtain High Performance?

## :

- Memory latency
- Algorithms
- Programming Practices

- Some common and *interesting* answers

**:**
**:**

## Motivation - Example I

## Motivation - Example I

- Duffy et. al.,
- Lawrence Livermore National Laboratory

-  CCM3 - spectral truncations of T170 and T239
-  50 Km spatial resolution is 32 times more grid cells and takes roughly** 200 times longer** to run

## Motivation - Example II

**Non-linear demand for resources (CPU - Memory) **
**Multi-disciplinary application is more demanding**

- AGCM/ACM
- 2.5 long x 2 lat, 30 layers
- 25-chemical species

- AGCM/ACM
- 2.5 long x 2 lat, 30 layers
- 2-chemical species

## The Hardware

## Evolution of 

- Evolution of
- High Performance Computers

**FL**oating **P**oint **O**perations/**S**econd (**FLOPS**)

## Large Scientific Codes: 

***A Common Programming Practice***

**Tuned and machine**
**Dependent  modules**

**Application**
**Data Layout**

**Control**

**I/O**

**Algorithmic Implementations**

## Memory Latency

**Shared Memory**

**Distributed**
**Memory**

- Memory

- M1

- M2

- P2

- P1

- Mn

- Pn

- P2

- P1

- Pn

## Memory Hierarchy

**Memory Hierarchy**

*** ****Where is the data? Why is data locality important?*

** ****CPU**

**Speed**
**Size**

## CPU vs. DRAM Performance

**CPU vs. DRAM Performance**

-  Since 1980’s, Procs performance has increased at a rate of almost 60%/year
-  Since 1980’s, DRAM (latency) has improved at a rate of almost 9%/year

## Parallel Programming Paradigms

**Distributed**
**Memory**

- Memory

- M1

- M2

- P2

- P1

- Mn

- Pn

- P2

- P1

- Pn

-  Data parallelism
-  Message Passing

-  easier to implement
-  shared memory space
-  mutual exclusion, contention
-  shared area is use for sending and receiving data

-  virtual shared memory
-  data is implicitly available to all
-  Implicit mutual exclusion
-  Only explicit synch
-  Depends on Memory Hierarchy and Network

**Parallel Programming Paradigms**

## Shortcomings

**Tuned and machine**
**Dependent  modules**

**Application**
**Data Layout**

**Control**

**I/O**

**Algorithmic Implementations**

## Alternative Programming Approach

**Alternative Programming Approach**

**USER's APPLICATION CODE **
- (Main Control)

**Tuned and machine**
**Dependent  modules**

**Application**
**Data Layout**

**I/O**

**Algorithmic **
**Implementations**

**AVAILABLE**

**LIBRARIES & PACKAGES**

**AVAILABLE**

**LIBRARIES & PACKAGES**

**AVAILABLE**

**LIBRARIES **

## Levels of abstraction 

**Software Development **
**Levels of abstraction**** **

## What needs to be computed?

- ODEs

- PDEs

## What codes are being developed?

- Parallel programs that use large distributed arrays

- Support for Grids
- and meshes

- Language Interoperability

- Infrastructure for distributed computing

- On-line visualization and computational stearing

- Coupling distributed applications

- Performance analysis and monitoring

## The PETSc Development Team

* *Portable, Extensible Toolkit for Scientific Computation
**PETSc**

**The PETSc Development Team**

- Argonne National Laboratory
- Mathematics and Computer Science Division
- http://www-fp.mcs.anl.gov/petsc/

- Satish Balay
- Kris Buschelman
- Bill Gropp
- Dinesh Kaushik

- Mathew Knepley
- Lois Curfman-McInnes
- Barry Smith
- Hong Zhang

## What  is PETSc?

***What  is *****PETSc?**

- A toolkit that eases the difficulties of developing parallel, non-trivial PDE solvers that deliver high performance (not a PDE solver black-box!)
- Freely available (well documented + lots examples and tutorials!)
- Portable to any parallel system supporting MPI
- Begun in 1991. Over 8,500 downloads. Current version 2.1.3

## Structure of PETSc

**Structure of PETSc**

## Structure of PETSc

**Structure of PETSc**

## Structure of PETSc

**Structure of PETSc**

## Structure of PETSc

**Structure of PETSc**

## Structure of PETSc

**Structure of PETSc**

##  PETSc Numerical Components

** ****PETSc Numerical Components**

- Compressed
- Sparse Row
- (AIJ)

- Blocked Compressed
- Sparse Row
- (BAIJ)

- Block
- Diagonal
- (BDIAG)

- Dense

- Other

- Indices

- Block Indices

- Stride

- Other

- Index Sets

- Vectors

- Line Search

- Trust Region

- Newton-based Methods

- Other

- Nonlinear Solvers

- Additive
- Schwartz

- Block
- Jacobi

- Jacobi

- ILU

- ICC

- LU
- (Sequential only)

- Others

- Preconditioners

- Euler

- Backward
- Euler

- Pseudo Time
- Stepping

- Other

- Time Steppers

- GMRES

- CG

- CGS

- Bi-CG-STAB

- TFQMR

- Richardson

- Chebychev

- Other

- Krylov Subspace Methods

- Matrices

- Distributed Arrays

- Matrix-free

## Vectors (basic operations) 

**Vectors (basic operations) **

- Each process locally owns a subvector of contiguously numbered global indices
- Types: Sequential, MPI or SHARED

## Selected Vector Operations

## Matrices (basic operations) 

**Matrices (basic operations) **

- Fundamental objects for storing linear operators (e.g., Jacobians)
- Types:
- default sparse AIJ: MPIAIJ, SEQAIJ
- block sparse AIJ (for multi-component PDEs): MPIAIJ, SEQAIJ
-  symmetric block sparse AIJ: MPISBAIJ, SAEQSBAIJ
- block diagonal: MPIBDIAG, SEQBDIAG
- dense: MPIDENSE, SEQDENSE
- matrix-free

- MatCreate(MPI_Comm Comm, m,n,M,N, Mat * Mat)
- MPI_Comm - processors that share the matrix
- number of local (m x* *n)/global (M x N) rows and columns
- MatSetType(Mat,MatType)
- MatDestroy(Mat)

## Matrices (basic ops) 

**Matrices (basic ops) **

- Single user interface, e.g.,
- Matrix assembly
- MatSetValues()
- Matrix-vector multiplication
- MatMult()
- Matrix viewing
- MatView()
- Multiple underlying implementations
- AIJ, block AIJ, symmetric block AIJ, block diagonal, dense, matrix-free, etc.

## PETSc Linear Solvers

**PETSc Linear Solvers**

**Goal**:  Support the solution of linear systems,
* **Ax=b,*
- particularly for sparse, parallel problems arising within PDE-based models
- User provides:
-  Code to evaluate  *A, b*

## PETSc Linear Solvers (Preconditioning)

**PETSc Linear Solvers (Preconditioning)**

- Given the linear system of equations:
* **	**	**	**Ax=b **	**	**	**(1)*
*Krylov Projection Methods (KSP) are strongly dependent on the spectrum of A.  Use of preconditioning techniques usually accelerate the convergence rate of the iterative techniques. *

**KRYLOV SUBSPACE METHODS + PRECONDITIONERS**
- R. Freund, G. H. Golub, and N. Nachtigal. *Iterative Solution of Linear Systems,pp 57-100.*
*ACTA Numerica. Cambridge University Press, 1992. *

## PETSc Linear Solvers (SLES)

- PETSc

- Application
- Initialization

- Evaluation of* A* and *b*

- Post-
- Processing

- Solve
*Ax = b*

- PC

- KSP

- Linear Solvers (SLES)

- PETSc code

- User code

- Main Routine

**PETSc Linear Solvers (SLES)**

## PETSc Non-Linear Solvers (SNLES)

**PETSc Non-Linear Solvers (SNLES)**

- PETSc code

- User code

- Application
- Initialization

- Function
- Evaluation

- Jacobian
- Evaluation

- Post-
- Processing

- PC

- KSP

- PETSc

- Main Routine

- Linear Solvers (SLES)

- Nonlinear Solvers (SNES)

- Solve
*F(u) = 0*

## Time Dependent PDE Solution

**Time Dependent PDE Solution**

- Application
- Initialization

- Function
- Evaluation

- Jacobian
- Evaluation

- PC

- KSP

- PETSc

- Main Routine

- Linear Solvers (SLES)

- Nonlinear Solvers (SNES)

- Timestepping Solvers (TS)

- Solve
* **U** t **= F(U,U**x,**U**xx**)*

- Post-
- Processing

## 3D incompressible Euler
Tetrahedral grid
Up to 11 million unknowns 
Based on a legacy NASA code, FUN3d, developed by W. K. Anderson
Fully implicit steady-state
Primary PETSc tools: nonlinear solvers (SNES)  and vector scatters (VecScatter)

- Tetrahedral grid
- Up to 11 million unknowns
- Based on a legacy NASA code, FUN3d, developed by W. K. Anderson
- Fully implicit steady-state
- Primary PETSc tools: nonlinear solvers (SNES)  and vector scatters (VecScatter)

*Results courtesy of Dinesh Kaushik and David Keyes, Old Dominion Univ., partially funded by NSF and ASCI level 2 grant*

- CFD Application using **PETSc **

## Fixed-size Parallel Scaling Results (GFlop/s)

- Dimension=11,047,096

- CFD Application using **PETSc **

## Fixed-size Parallel Scaling Results

- (Time in seconds)

- CFD Application using **PETSc **

## TAO

**TAO**
**Toolkit for Advanced Optimization **

- S. Benson, L. Curfman-McInnes, J. More and J. Sarich
- Argonne National Laboratory

## hypre 

- Hypre TEAM
*Lawrence Livermore National Laboratory*
- Rob Falgout (project leader)
- Guillermo Castilla
- Edmond Chow
- Andy Cleary
- Van Emden Henson
- Jim Jones
- Mike Lambert
- Barry Lee
- Jeff Painter
- Charles Tong
- Tom Treadway
- Panayot Vassilevski
- Ulrike Meier Yang

## Hypre's Multiple interfaces 

## hypre  conceptual interfaces

- Structured-Grid Interface (Struct)
***applications with logically rectangular grids***
- Semi-Structured-Grid Interface (SStruct)
***applications with grids that are mostly—but not entirely—structured (e.g., block-structured, structured AMR, overset)***
- Finite Element Interface (FEI)
***unstructured-grid, finite element applications***
- Linear-Algebraic Interface (IJ)
***applications with sparse linear systems***

## CUMULVS

- Development Team: Oak Ridge National Laboratory
- Lead Pis: J. Kohl and Philip Papadopoulous

## Collaborative User Migration, User Library for Visualization and Steering
Enables parallel programming with the integration of:
Interactive visualization (local and remote)
Multiple views
Fault Tolerance
Computational Steering

- Enables parallel programming with the integration of:
- Interactive visualization (local and remote)
- Multiple views
- Fault Tolerance
- Computational Steering

- CUMULVS

## Setup input parameters to be steered 
Specify the nature and decomposition  of the data fields to be visualized. Standard data decompositions: Block, Block-Cyclic, particle decompositions  and User defined decompositions
Use existing interfacing to visualization packages or define a custom viewer on top of other visualization tools.
Setup checkpoint/restart mechanism

- Specify the nature and decomposition  of the data fields to be visualized. Standard data decompositions: Block, Block-Cyclic, particle decompositions  and User defined decompositions
- Use existing interfacing to visualization packages or define a custom viewer on top of other visualization tools.
- Setup checkpoint/restart mechanism

- CUMULVS

## CUMULVS

## The GRID

-  A large pool of resources
-  Computers
-  Networks
-  Software
-  Databases
-  Instruments
-  people
- _Requirements from GRID implementation:_
-  Ubiquitous: ability to interface to the grid at any point and leverage whatever is available
-  Resource Aware: manage heterogeneity of resources
-  Adaptive: tailored to obtain maximum performance from resources

## Globus

## High throughput computing
Schedule many tasks
Bag of resources for the Grid, offering:
Resource discovery
Data Access
Scheduling
Reservation
Security
Accounting
Code management

- Schedule many tasks
- Bag of resources for the Grid, offering:
- Resource discovery
- Data Access
- Scheduling
- Reservation
- Security
- Accounting
- Code management

- Globus

## TAU

- Profiling of Fortran 90, C, C++, HPF, and HPC++ codes
- Detailed information (much more than *prof*/*gprof*)
- C++: per-class and per-instance profiling
- Graphical display of profiling results (built-in viewers, interface to Vampir)

## TAU

- Detailed information (much more than *prof*/*gprof*)
- C++: per-class and per-instance profiling
- Graphical display of profiling results (built-in viewers, interface to Vampir)

## TAU

-  COSY: COmpile manager Status displaY
-  FANCY: File ANd Class displaY
-  CAGEY: CAll Graph Extended displaY
-  CLASSY: CLASS hierarchY browser
-  RACY: Routine and data ACcess profile displaY
-  SPEEDY: Speedup and Parallel Execution Extrapolation DisplaY

## TAU (Speedy)

## How much effort is involved in using these tools?

##  Must of the tools provide interfaces (calling functions and subroutines) from Fortran and C

- Best approach is to start with examples for beginners!
- Several efforts are targeting Tool Interoperability!

- Using the ACTS Collection

## Robust and High Performing Tools for Scientific Computing

**Robust and High Performing Tools for Scientific Computing**
*Lawrence Berkeley National Laboratory - September 4-7, 2002*

## http://acts.nersc.gov/events/Workshop2002/program.html

**http://acts.nersc.gov/events/Workshop2002/program.html**
**The material includes pdf files with tutorials on the ACTS tools, example codes and uses.  It also contains many references to applications that have benefited from the use of the ACTS Collection.**

## acts-support@nersc.gov

**acts-****support****@nersc.gov**

**http://acts.nersc.gov**