# Presentation: 148703

## DOE 2000:The Advanced Computational Testingand Simulation Toolkit

**James R. McGraw**
**Lawrence Livermore National Laboratory**

## ACTS Toolkit:  Context

**ACTS Toolkit:  Context**

- Customer Applications in Fortran or C++

- Operating System, Network, Hardware Layer
- (homogeneous and heterogeneous distributed parallel systems)

**ACTS Toolkit Contents**

***ACTS Toolkit***
***Support for***
***Customers***

## ACTS Toolkit:  Current Contents

- Application
- Support Tools

- Numerics

- Code
- Development
- Tools

- Code
- Execution
- Tools

- Adapt previously autonomous numerical libraries to permit interoperable use on selected DP applications.

- Adapt and integrate object-oriented libraries for defining and manipulating parallel distributed arrays.

- Develop a portable, parallel run-time class library and integrate it with tools for remote steering and visualization.

- Develop object-oriented tools for assisting code developers to write particle simulation-based applications.

- Round 1 Funding

- Round 2 Funding

- Expand the interoperable behavior with more numerical algorithms relevant to ER and other DOE applications.

- Expand tools for managing arrays on complex memory hierarchies and enable run-time linking to diverse class libraries.

- Incorporate use of parallel program debugging and analysis tools with the current system and improve the tools for use on computational grids.

## Metrics for Success

**More clients using the software tools**
**Improved performance of tools for clients**
**Reduced duplication of effort in tool development**
**Providing new capabilities**
**Publications **

## Overarching Theme:Interoperability

***Overarching Theme:*****Interoperability**

**Encrypt**

**Authenticate**

**Authorize**

**Control**

**Conference**

**Share**

**PETSc**

**Aztec**

**PVODE**

**Hypre**

**Opt++**

**SuperLU**

## Collaborating Projects

**Collaborating Projects**

**LBNL**
**Berkeley**
**U. Tenn.**
**USC**
**Indiana**
**U. Ore.**
**LANL**
**ANL**
**LLNL**
**PNNL**
**ORNL**
**SNL**

- Frame-
- works

- Run-
- Time

- Numerics

- ProjectsLinear Algebra
- ODEs/PDEs
- Optimization
- Comp. Arch. (CCA)
- Data Layout
- Apps. Dev. Env.
- OO Execution
- Distr. Computing

- Test and Evaluate

## Collaborating Projects

**Collaborating Projects**

**LBNL**
**Berkeley**
**U. Tenn.**
**USC**
**Indiana**
**U. Ore.**
**LANL**
**ANL**
**LLNL**
**PNNL**
**ORNL**
**SNL**

- Frame-
- works

- Run-
- Time

- Numerics

- ProjectsLinear Algebra
- ODEs/PDEs
- Optimization
- Comp. Arch. (CCA)
- Data Layout
- Apps. Dev. Env.
- OO Execution
- Distr. Computing

- Test and Evaluate

## Collaborating Projects

**Collaborating Projects**

**LBNL**
**Berkeley**
**U. Tenn.**
**USC**
**Indiana**
**U. Ore.**
**LANL**
**ANL**
**LLNL**
**PNNL**
**ORNL**
**SNL**

- Frame-
- works

- Run-
- Time

- Numerics

- ProjectsLinear Algebra
- ODEs/PDEs
- Optimization
- Comp. Arch. (CCA)
- Data Layout
- Apps. Dev. Env.
- OO Execution
- Distr. Computing

- Test and Evaluate

## Numerical Toolkit Efforts

**Large-scale (100,000,000+ DOF) simulations**
**computational fluid dynamics**
**combustion**
**structures**
**materials**
**usually PDE based**
**Large-scale optimization**** **
**often involving simulation**
**may be stochastic **

## PVODE-PETSc (LLNL-ANL)

**Complementary functionality**
**Parallel ODE integrators (PVODE)**
**sophisticated time-step control for accuracy**
**special scaled non-linear solver**
**object based**
**Scalable preconditioners and Krylov methods (PETSc)**
**run on 1000+ processors**
**highly efficient block matrix storage formats**
**object oriented**

## PVODE-PETSc Use

**Sintering - simulating the dynamics of micro-structural interactions via the method of lines, requiring the solution of a large set of coupled ODEs**
**Previously used LSODE, limited to 100s of DOF, now can handle 10,000s**
**Wen Zhang, Oakland University and Joachim Schneibel, Oak Ridge**

## Utah ASCI ASAP Center Center for the Simulation of Accidental Fires and Explosions

**“****...problem-solving environment in which fundamental chemistry and engineering physics are fully coupled with non-linear solvers, optimization, computational steering, ...”**
**PETSc + SAMRAI (LLNL) **
**Using the PETSc nonlinear PDE solvers**
**Already has fed back into PETSc nonlinear solver enhancements**

## Lab Related Toolkit Usage

**ALE3D test problems run with PETSc based parallel multigrid solver**
**Run on NERSC 512 processor T3E and LLNL ASCI Blue Pacific**
**Simulation with 16 million+ degrees of freedom**
**Mark Adams (Berkeley)**
**Version of ARES and Teton run with PETSc**
**Ground water flow simulation (gimrt-LLNL)**
**Multi-phase flow (LANL), sequential legacy code**

## System Software -Distributed Computing and Communications

**Build on Nexus communication library to provide ACTS toolkit with**
**Integrated support for multithreading and communication**
**High-performance multi-method communication**
**Dynamic process creation**
**Integration with distributed computing**
**Apply these capabilities to ACTS toolkit components & applications**

## Connections: Nexus

- Nexus

- MPICH

- HPC++ etc.

- PST

- CIF

- ALICE

- ManyWorlds

- NEOS

- CC++

- MOL

- Globus

- RIO

- SCIRun

- DAVE-2

- DOE2000 collaboratory projects

- DOE2000 ACTS toolkit projects

- Numerical
- libraries

- Other
- parallel
- tools

- AutoPilot

- Akenti

- Paradyn

- PAWS

- CAVERN

## Remote Browsing of Large Datasets

**Problem: interactively explore very large (TB+) datasets**
**Interactive client VRUI with view mgmt support**
**Data reduction at remote client (subsampling)**
**Use of Globus to authenticate, transfer data, access data**

- ANL
- USC/ISI
- UC ASAP
- CIT ASAP

## Applications in FlashASCI ASAP Center

**Prototype “global shell” that permits:**
**Sign-on once via public key technology**
**Locate available computers**
**Start computation on an appropriate system**
**Monitor progress of computation**
**Get [subsampled] output files**
**Manipulate locally **

## Cumulvs:  Collaborative Infrastructure for Interacting with Scientific Simulations

**Coordination of collection and dissemination of information to/from parallel tasks to multiple viewers.**

**task**

**task**

**task**

**task**

**task**

**task**

**task**

**task**

**task**

**task**

- viewer

- viewer

- viewer

**CUMULVS**

- Unix Host A

- NT Host B

- Unix Host C

- Remote Person
- Using AVS

- Remote Person
- Using VR Interface

- Local Viewer
- Custom GUI

## Cumulvs Capabilities

**Multiple distinct data views **
**Links to a variety of visualization tools**
**Dynamic linking to running application**
**Coordinated computational steering**
**Hetergeneous checkpointing**

## Cumulvs Integration & Interoperability

**Integration with InDEPS code development environment  (ORNL & SNL)**
**Combustion Simulations**
**Material Impact and Deformation**
**Smooth Particle Hydrodynamics**
**Remotely monitored T3E Applications (ORNL, LBL, and L_x0008_LNL)**
**Tcl/Tk language bindings for Cumulvs (NCSA, ORNL)**
**Viewers:  VTK & Tango, VR / OpenGL viewer, Immersadesk, and CAVE**
**Apps.: Chesapeake Bay Simulation, Neutron Star Collision, DOD codes**

## PADRE:  Parallel Asynchronous Data Routing Engine

- Caterpillar® Engine Geometry
- using Pro/ENGINEER®

## PADRE Purpose : Data Distribution Interoperability

**Data Distribution Library**
- e.g. Multi-block PARTI (UM),
- KeLP (UCSD), PGSLib (LANL),
- Global Arrays (PNNL)

- Communication Library
- e.g. MPI, PVM

- Application Libraries
- e.g. Overture, POOMA, AMR++

## Dynamic Distribution (redistribution, etc.)
Generation and Execution of Communication Schedules
General Array Language Support
TULIP Interface Support
Cached schedules for performance
Lazy Evaluation
Message Latency Hiding
Subarray operation support for different distributions (AMR)
Gather/Scatter support for indirect addr. (Overlapping Grid Apps)

**Generation and Execution of Communication Schedules**
**General Array Language Support**
**TULIP Interface Support**
**Cached schedules for performance**
**Lazy Evaluation**
**Message Latency Hiding**
**Subarray operation support for different distributions (AMR)**
**Gather/Scatter support for indirect addr. (Overlapping Grid Apps)**

**PADRE Services**

## PADRE in Use

**Default data distribution mechanism in Overture**
**Permits multiple distribution mechanisms within Overture for specialized applications (e.g. AMR)**
**Optimized communication using standard MPI and ROSE Optimizing preprocessor**
**Global Arrays**
**Will provide alternative to GA interface for C++ users of GA**
**Provides access to Geometry in a form identical to what they already use**
**Gives PADRE access to one-sided message passing (important for parallel grid generation mechanisms within Overture)**
**Ready for Use in POOMA II**
**KeLP distributions in PADRE close to domain layout in POOMA **
**Can provide geometry and AMR capabilities to POOMA apps.**

## Interoperability Goal:Scientific Apps. via Plug-and-Play

**Interoperability Goal:****Scientific Apps. via Plug-and-Play**

- Discretization

- Grids

- Data Reduction

- Physics Modules

- Optimization

- Derivative Computation

- Collaboration

- Adaptive Solution

**Notes:**
Two types of components:
	“one-shot” quick ones that are application specific. (e.g. steering)
	“reusable” difficult-to-write ones that are written by experts in the field (for example, solvers).

## Common Component Architecture:Active Participants

**Cal Tech**
**Indiana University**
**University of Utah**
**Sandia National Laboratory**
**Argonne National Laboratory**
**Lawrence Berkeley National Laboratory**
**Lawrence Livermore National Laboratory**
**Los Alamos National Laboratory**
**Oak Ridge National Laboratory **
**Pacific Northwest Laboratories**

**Notes:**
This is the current working group.  All of these people attend the meetings.

Good cross section of  Universities and DOE labs

## Basic Idea

**Component Concepts**
**Autonomous interchangeable parts**
**Links between components that create an application**
**Builders**
**Framework provides context or environment to components**

**Notes:**
Review of what the CCA considers a component framework to be:

	The word framework is almost meaningless.
Builders are software that identify components and prescribe links.   Framework, in our defn, is something that physically implements the links.

Framework provides the environment within which the components will run
	services
	runtime env.

## Existing Component Models:The good, bad, and ugly

**What’s good about them?**
**Everyone knows how they should work.**
**They are accepted standards.**
**What’s bad about them?**
**Not geared toward high performance computing.**
**Often based on a single framework. **
**Geared toward a single language (e.g. Beans).**
**Meant for a single environment (e.g. COM).**

**Notes:**
The major reason we exist is HP computing.

Most Component Arch’s are tied to a specific framework (e.g. Visual Basic, Beans).

## CCA Extensions to Existing Models

**gPorts (****g****eneralized ****Ports****)**
**Similar to the CORBA3.0 User/Provider Port spec.**
**Uses a type of *****IUnknown *****pattern similar to COM.**
**Stolen from data-flow component models**
**Defined by ***linked interfaces****. ***
**Draft gPort spec in pre-RFC stage:****http://z.ca.sandia.gov/~cca-forum/gport-spec**
**Scientific IDL**
**Once component model is defined, language interoperability is *****the*****  issue.**
**Include legacy programs.**
**Include legacy programmers**
**Draft IDL spec in pre-RFC stage:**** ****http://www.llnl.gov/CASC/babel**

**Notes:**
Two different legs of the CCA have been working on two different aspects:
	gPorts - the means by which interfaces are linked
	Sci IDL - the provider of language independence.

Defn of gPort is an interface that is meant to be linked to another interface.

A big imperitive is to include multiplicity of languages, including Fortran.  

## ACTS Toolkit:  Remaining Holes

- Application
- Support Tools

- Numerics

- Code
- Development
- Tools

- Code
- Execution
- Tools

- Visualization
- & Analysis Tools

- Expansion of tools to provide portable performance, multi-language programming environments, and tools for software maintenance of scientific applications.

- Integration of tools for helping application users to visualize and analyze the results of complex 3D computations.

- Expansion of tools for helping application developers share common domain-specific tools, such as adaptive mesh refinement and unstructured grids.

- Automated differentiation tools
- and  ???

- Expansion of tools to provide  parallel I/O, dynamic load-balancing, intelligent storage management.

## Slide 31