# Presentation: 391013

## Component Technology for Terascale Simulation Software

- in Parallel Numerical Components
- Lois Curfman McInnes
- Mathematics and Computer Science Division
- Argonne National Laboratory
- Computation Institute
- Brown-Bag Presentation
- February 7, 2002
- _[http://www.](http://www.cca-forum.org/ccttss)__[cca](http://www.cca-forum.org/ccttss)__[-forum.org/](http://www.cca-forum.org/ccttss)__[ccttss](http://www.cca-forum.org/ccttss)_

**Component Technology for ****Terascale Simulation Software**

- Common Component Architecture
**Participants:**
- ANL
- Indiana Univ.
- LANL
- LLNL
- ORNL
- PNNL
- SNL
- Univ. of Utah
**Partners:**
- Climate Modeling
- Quantum Chemistry
- Combustion Modeling

## Bringing Components to High Performance Computing:Challenges and Goals

**Today: **
**Single-purpose, monolithic,**
**tightly-coupled**
**parallel codes**

**Currently DOE Apps Are:**
- One-off, stove-pipe, few participants
**But Also:**
- Scalable, and low latency
- Large & rich legacy investment

**Goals**
- Enable plug-and-play parallel simulations
- Establish component "marketplace" with applications partners
- Extend commodity component technology for HPC

**Challenges**
- Practical  parallel HPC component model
- Integrate legacy software investment
- Connection to Grid components
- Multi-language

**CCA**

***HPC Component***
***    ******Framework***

***Scientific***
***    ******Components***

***“******MxN” Parallel Data***
***    ******Redistribution***

***Applications***
***    ******Integration***

## Traditional Approach to Software Interoperability

- Direct interfacing between different packages/libraries/apps
- Public interfaces are unique
*Many-to-Many* couplings require *Many **2* interfaces
- Often a heroic effort to understand the details of both codes
- Not a scalable solution

- SUMAA3d

- DAs

- GRACE

- Overture

- Trilinos

- ISIS++

- PETSc

- Data / mesh software

- Linear solvers

## Common Interface Specification

- Reduces the *Many-to-Many* problem to a *Many-to-One* problem
- Allows interchangeability and experimentation
- Difficulties
- Interface agreement
- Functionality limitations
- Maintaining performance

- SUMAA3d

- DAs

- GRACE

- Overture

- Trilinos

- ISIS++

- PETSc

## Motivation

- Existing component architecture standards such as CORBA, Java Beans, and COM  do not provide support for parallel components.

- CORBA/Java

**CCA**

- MPI

- Latency
- between
- components

- Desire to build scientific applications by hooking together  components
- DOE Common Component Architecture (CCA) provides a mechanism for interoperability of high-performance components developed by many different groups in different languages or frameworks.

## Requirements for a High-Performance Component Architecture

- Simple/Flexible
- to adopt
- to understand
- to use
- Support a composition mechanism that does not impede high-performance component interactions
- Permit the SPMD paradigm in component form
- Meant to live with and rely on **other** commodity component frameworks to provide services ...
- e.g., JavaBeans, CCM, ...

## History and Participants

- CCA forum originated in 1998 within DOE’s DOE2K program; open to everyone interested in HPC components
- Active CCA forum participants *
- ANL - Lori Freitag, Paul Hovland, Kate Keahey, Lois McInnes, Boyana Norris
- Indiana Univ. - Dennis Gannon, Randall Bramley
- LANL - Craig Rasmussen, Matt Sotille
- LLNL - Scott Kohn, Gary Kumfert, Tom Epperly
- ORNL - David Bernholdt, Jim Kohl
- PNNL - Jarek Nieplocha
- SNL - Rob Armstrong, Ben Allan, Jaideep Ray
- Univ. of Utah - Steve Parker
- Terascale - Robert Clay, Lee Taylor
- CCTTSS team is a subset of CCA forum members

- * To join CCA forum mailing list, send "subscribe cca-forum" to _[majordomo](mailto:majordomo@z.ca.sandia.gov)__[@z.ca.](mailto:majordomo@z.ca.sandia.gov)__[sandia](mailto:majordomo@z.ca.sandia.gov)__[.](mailto:majordomo@z.ca.sandia.gov)__[gov](mailto:majordomo@z.ca.sandia.gov)_

## CCA Approach

- CCA specification dictates a basic set of interfaces (and corresponding behaviors) that components should implement to be CCA compliant.
- Metaphor from visual programming (e.g., AVS); instead of data-flow, ports are linked with interfaces
- Provides/Uses design pattern
- Components are manipulatable in a framework.
- CCA specification doesn’t dictate frameworks or runtime environment
- Create components that are usable under a variety of frameworks
- Provide a means for discovering interfaces
- Specifically exclude *how* the components are linked; that is the job of a framework
- Provide language-independent means for creating components (via SIDL)

**Notes:**
The CCA bends over backwards not to legislate methodology either to the framework or component writers
while preserving the ability to do high-performance intercomponent links.

Iunknown provides for interface negotiation -- IDL will provide for reflection.

## Focus of CCTTSS

- HPC component specification and framework
- Unified reference framework implementation targeting both SPMD and distributed environments
- Tools for language interoperability via SIDL
- Suite of scientific components
- Linear and nonlinear algebra, optimization, mesh management, scientific data,visualization, steering, fault tolerance, scientific application domains, etc.
- Parallel data redistribution
- Model coupling, visualization
- Applications integration
- General outreach to the scientific community
- Close feedback loop for users/developers of CCA technology
- Collaborate with climate and chemistry applications domains as well as other ISIC centers and applications

## Ports preserve the high performance of direct connections plus the versatility of distributed object systems.

- Allow directly connected interfaces: the next component is a few function calls away (w/ SIDL).
- Adapters can create network-distributed objects out of the same components without altering them.

- Component1

- Component2

- Provides/
- Uses Port

- Component1

- Component2

- Uses Port

- ProvidesPort

- Network

## CCA Concept of SCMD (SPMD) Components

**A**

**A**

**A**

**A**

**B**

**B**

**B**

**B**

- MPI

- MPI

- Process

- MPI application using CCA for interaction between
- components A and B within the same address space

**Adaptive mesh**
** ****component**
**written by user1**

**Solver component**
**written by user2**

**Direct**
**Connection**
- supplied by
- framework at
- compile/runtime

- CCA Concept of SCMD (SPMD) Components

**Proc1**

**Proc2**

**Proc3**

**etc...**

## CCA Collective Port Modularizes Processor/Data Decomposition 

- parallel
- visualization component

**collective port**
**connecting M procs **
**with N procs**

- Combining previous parallel component with another
- parallel component in a different framework

- container composed of
- mesh and solver
- components

## Major Activities in Early Years

- Year 1
- Assess requirements of other SciDAC groups; promote and support CCA approach through tutorials, workshops, etc.
- Define interfaces and develop various base component prototypes
- Complete candidate SCMD framework specification and implementation; complete candidate distributed computing CCA runtime environment
- Year 2:
- Deploy base component prototypes; integrate base component implementations; define second level component extensions; propagate language interoperability
- Define applications interfaces; begin production componentization efforts for chemistry and climate applications
- Year 3:
- Deploy integrated SCMD/distributed framework; deploy component repository
- Streamline component and framework implementations; implement fault tolerance and recovery mechanisms; generalize and standardize system specifications

##      Given a rectangular 2-dim domain and boundary values along the edges of the domain, the objective is to find the surface with minimal area that satisfies the boundary conditions, i.e., compute
           min f(x), 
           where f: R   R

-            *min f(x),*
-            where *f: R  **** R*

- Unconstrained Minimization Problem

*n*

## Unconstrained Minimization Demo

**Optimization**
**u****i+1**** = u****i**** + ********s ...**

**Visualization**

**Linear Solver**
**H s = g**

**H**

**g**

**s**

**g**

**H**

**f**

**u****i+1**

**u****o**

- function
- solution
- Hessian
- gradient
- coordinates
- connectivity
- step direction

**f(u)**
**u **
**H**
**g **
**x **
**c **
**s**

- Compute **min f(u)**

**CCAFFEINE Framework**

- Trilinos

- PETSc

- Others ...

**Driver**

-  CCAFFEINE – Common Component Architecture Fast Framework
-            Example in Need of Everything
-  reference framework under development by B. Allan et al. (SNL)
-  _[http://www.cca-forum.org/cafe.html](http://www.cca-forum.org/cafe.html)_
-  TAO – Toolkit for Advanced Optimization
-  _[http://www.mcs.anl.gov/tao](http://www.mcs.anl.gov/tao)_
-  Demo developers: Boyana Norris, Steve Benson, and Lois McInnes

**Local Physics,**
**Discretization**

**Data **
**Redistribution**

## CCA Compliance in TAO

- Paradigm shift; both TAO and the application become components
- Each is required to provide a default constructor and to implement the CCA component interface
- contains one method: “setServices” to register ports
- All interactions between components use ports
- Application *provides* a “go” port and *uses* “taoSolver” port
- TAO *provides* a “taoSolver” port
- There is no “main”  routine

## Component Wiring Diagram

## CCA Demo Components for SC01

- 4 applications using CCAFFEINE
- Unconstrained minimization problem on a structured mesh
- Time-dependent PDE on an structured mesh
- Time-dependent PDE on an adaptive structured mesh
- Ping-pong MxN
- More than 30 components
- Many components re-used in 3 apps
- Starting points for apps interactions
- Leverage and extend parallel software developed at different institutions
- including CUMULVS, GrACE, LSODE, MPICH, PAWS, PETSc, PVM, SUMAA3d, TAO, and Trilinos.

## Component Re-Use

- Various services in CCAFFEINE
- Optimization solver
*TAOSolver*
- Integrator
*IntegratorLSODE*
- Linear solvers
*LinearSolver_Petra*
*LinearSolver_PETSc*
- Data description
*DADFactory*
- Data redistribution
*CumulvsMxN*
- Visualization
*CumulvsVizProxy*

- Component interfaces to parallel data management and visualization tools

- Component interfaces to numerical libraries, all using ESI

## SC01 Demo Code Distribution

- In progress ... Nearly ready for release “any day now”
- Tar file and web-based installation
- Will certainly evolve
- Starting-point for tutorials and apps interactions
- Some initial material for a CCA repository
- Contents
- Service components
- Mesh management and discretization
- Linear algebra, time integration, and optimization
- Data description, data redistribution, and visualization

## Service Components

***CheetahMPI**** *– Craig Rasmussen, Sue Mniszewski, Pat Fasel, and Reid Rivenburgh (LANL) – Provides a high level interface on top of MPI to allow different components to partition MPI_COMM_WORLD into separate communication groups.
***CCAFFEINE Utilities*** – Ben Allan and Rob Armstrong (SNL) –* *Provide a range of prototypes for service components, including a simple ***Logger***** **component to print diagnostics to a user-configurable file and a ***BuilderService*** prototype that renders the entire *CCAFFEINE* framework as a component to facilitate application building.

## Mesh Management and Discretization 

***TSTTMesh ***- Lori Freitag (ANL) – Provides prototype capabilities for querying unstructured meshes based on interfaces being designed within the TSTT SciDAC Center.
***FEMDiscretization*** – Lori Freitag (ANL) – Provides finite element discretization of diffusion and advection PDE operators and linear system assembly capabilities.
***GrACEComponent*** – Jaideep Ray (SNL) – Provides parallel AMR infrastructure, which follows a hierarchy-of-patches methodology for meshing and includes load-balancing; *GrACEComponent* is being used within a combustion code in the SciDAC CFRFS project.
***Various AMR Support Components*** – Jaideep Ray (SNL) – Provide support for AMR computations, including computation of the error incurred by advancing the solution and adaptors that translate a patch-based hierarchy into a *DistArrayDescriptor*, which can then used in parallel data redistribution and visualization.

## Parallel Linear Algebra, Time Integration, and Minimization 

***ESIFactory*** – Boyana Norris (ANL) – Generates linear algebra objects that employ interfaces under development by the ESI working group; these vectors, matrices, and linear solvers are used as fundamental objects within *LinearSolver,* *IntegratorLSODE*, *TAOSolver*, and *TSTTMesh*, so that applications can use ESI-compliant implementations provided by external components; currently available underlying implementations are based on the PETSc (ANL) and Trilinos (SNL) libraries.
***LinearSolver*** – Boyana Norris (ANL) – Provides a prototype linear solver port that uses ESI vector and operator interfaces, so that applications can benefit from implementations that use newly ESI-enabled parallel linear solvers.
***IntegratorLSODE*** – Ben Allan (SNL) – Provides a parallel ODE solver, based on LLNL’s ODEPACK and the ESI specification. The distribution also contains components that provide specialized mathematical operations on ESI vectors, generalized error/convergence specifications for implicit time-integrators, and logging to disk of errors/messages from different components in a parallel setting.
***TAOSolver*** – Steve Benson, Lois Curfman McInnes, and Boyana Norris (ANL) – Provides parallel unconstrained minimization solvers, which build on infrastructure within TAO (ANL) and use the ESI specification.

## Data Description, Redistribution, and Visualization

***DistArrayDescriptorFactory*** *–* David Bernholdt (ORNL) – Provides a uniform means for applications to describe dense multi-dimensional arrays and is based upon emerging interfaces from the CCA Scientific Data Components Working Group.
***CumulvsMxN**** – *Jim Kohl (ORNL) – Builds on CUMULVS (ORNL) technology to provide an initial implementation of the “MxN” parallel data redistribution interfaces that are under development by the CCA “MxN” Working Group.  *CumulvsMxN* is designed to span multiple CCA frameworks and to pass data between two distinct parallel component applications.
***PawsMxN*** *–* Sue Mniszewski, Pat Fasel, Reid Rivenburgh, and Matt Sotille (LANL) – Builds on PAWS (LANL) technology to distribute data between two data parallel components.   The *PawsMxN* component uses MPI to move data in a scalable and efficient fashion and follows the evolving CCA “MxN” interface specification.
***VizProxy*** – Jim Kohl (ORNL) – Provides a companion “MxN” endpoint for extracting parallel data from component-based applications and then passing this data to a separate front-end viewer for graphical rendering and presentation. Variants exist for structured data and unstructured triangular mesh data as well as text-based output.

## Some Ongoing Projects

- Language interoperability
- Scientific IDL (SIDL) and Babel tool
- Scott Kohn (LLNL: skohn@llnl.gov)
- Distributed components
- Line protocols, inter-framework compatibility
- Dennis Gannon (IU: gannon@cs.indiana.edu ), Gary Kumfert (LLNL: kumfert@llnl.gov)
- Scientific data components
- Structured/unstructured mesh data
- Lori Freitag (ANL: freitag@mcs.anl.gov)
- Data redistribution
- Repartitioning, inter-parallel program communication, model coupling
- Jim Kohl (ORNL: kohlja@ornl.gov)

## Collaborations with Others

- CCA does not pretend to be experts in all phases of computation, just to provide a standard way to exchange component capabilities.
- Medium of exchange: interfaces
- Equation Solver Interface Forum (ESI) has made headway toward defining interfaces for linear algebra
- _[http://z.ca.](http://z.ca.sandia.gov/esi)__[sandia](http://z.ca.sandia.gov/esi)__[.](http://z.ca.sandia.gov/esi)__[gov](http://z.ca.sandia.gov/esi)__[/](http://z.ca.sandia.gov/esi)__[esi](http://z.ca.sandia.gov/esi)_
**Need experts in various areas to define sets of domain-specific abstract interfaces**
- nonlinear solvers, optimization, meshes, visualization, scientific application domains, etc.
- e.g., Newton-Krylov methods

- This
- means
- you!

## More Information