# Presentation: 211952

## Scalable Solution of PDE-based Applications Using PETSc

- Lois Curfman McInnes
- Mathematics and Computer Science Division
- Argonne National Laboratory
- Seminar
- National Institute of Standards and Technology
- November 5, 2002

## Outline

- Motivation
- Complex, multi-physics, multi-scale applications
- Distributed, multi-level memory hierarchies
- Parallel Software for PDEs
- Approach
- Performance
- Next Generation Software
- What are components?
- Common Component Architecture (CCA) Forum
- Ongoing Challenges

## Acknowledgements

- U.S. Department of Energy – Office of Science
- Core funding in the MCS Division of Argonne through the Mathematical, Information, and Computational Science (MICS) program
- Advanced Computational Testing and Simulation (ACTS) toolkit
- Scientific Discovery through Advanced Computing (SciDAC) program
- National Science Foundation
- Multi-Model Multi-Domain Computational Methods in Aerodynamics and Acoustics

## Motivating Scientific Applications

- Discretization

- Parallel I/O

- Meshes

- Data Redistribution

- Physics

- Optimization

- Derivative Computation

- Adaptive Solution

- Astrophysics

- Molecular
- structures

- Aerodynamics

- Fusion

## Target Architectures

- Systems have an increasingly deep memory hierarchy
- Time to reference main memory 100’s of cycles

- ...

- etc.

## Challenges

- Community Perspective
- Life-cycle costs of applications are increasing
- Require the combined use of software developed by different groups
- Difficult to leverage expert knowledge and advances in subfields
- Difficult to obtain portable performance
- Individual Scientist Perspective
- Too much energy focused on too many details
- Little time to think about modeling, physics, mathematics
- Fear of bad performance without custom code
- Even when code reuse is possible, it is far too difficult
- Our Perspective
- How to manage complexity of numerical tools?
- Numerical software tools that work together
- New algorithms (e.g., interactive/dynamic techniques, algorithm composition)
- Multi-model, multi-physics simulations

## Outline

- Motivation
- Complex, multi-physics, multi-scale applications
- Distributed, multi-level memory hierarchies
- Parallel Software for PDEs
- Approach
- Performance
- Next Generation Software
- What are components?
- Common Component Architecture (CCA) Forum
- Ongoing Challenges

## What are the algorithmic needs of our target applications? 

- Large-scale, PDE-based applications
- Multi-rate, multi-scale, multi-component
- Need
- Fully or semi-implicit solvers
- Multi-level algorithms
- Support for adaptivity
- Support for user-defined customizations (e.g., physics-informed preconditioners, transfer operators, and smoothers)
- Our focus
- Algorithms
- Analysis
- Software – design, implementation, optimization
- Applications

## Interface Issues

- How to hide complexity, yet allow customization and access to a range of algorithmic options?
- How to achieve portable performance?
- How to interface among external tools?
- including multiple libraries developed by different groups that provide similar functionality (e.g., linear algebra software)
- Criteria for evaluation of success
- efficiency (both per node performance and scalability)
- usability
- extensibility

## PETSc: Portable, Extensible Toolkit for Scientific Computation

- S. Balay, K. Buschelman, B. Gropp, D. Kaushik, M. Knepley, L. C. McInnes, B. Smith, H. Zhang
- _[http://www.mcs.anl.gov/](http://www.mcs.anl.gov/petsc)__[petsc](http://www.mcs.anl.gov/petsc)_
- Targets the parallel solution of large-scale PDE-based applications
- Begun in 1991, now over 8,500 downloads since 1995
- Over 50 publications involving computations done with PETSc by outside users
- Freely available and supported research toolkits
- Hyperlinked documentation, many examples
- Usable from Fortran 77/90, C, and C++
- Portable to any parallel system supporting MPI, including
- Tightly coupled systems
- Cray  T3E, SGI Origin, IBM SP, HP 9000, Sun Enterprise
- Loosely coupled systems, e.g., networks of workstations
- Compaq, HP, IBM, SGI, Sun
- PCs running Linux or Windows

## The PETSc Team

- Dinesh
- Kaushik

- Lois
- Curfman
- McInnes

- Kris
- Buschelman

- Satish
- Balay

- Bill
- Gropp

- Barry
- Smith

- Hong
- Zhang

- Matt
- Knepley

## Programming Model

- Goals
- Portable, runs everywhere
- Performance
- Scalable parallelism
- Approach
- Distributed memory, “shared-nothing”
- Requires only a compiler (single node or processor)
- Access to data on remote machines through MPI
- Can still exploit “compiler discovered” parallelism on each node (e.g., SMP)
- Hide within parallel objects the details of the communication
- User orchestrates communication at a higher abstract level than message passing

## PDE Application Codes

- Computation and Communication Kernels
- MPI, MPI-IO, BLAS, LAPACK

- Profiling Interface

- PETSc PDE Application Codes

- Object-Oriented
- Matrices, Vectors, Indices

- Grid
- Management

- Linear Solvers
- Preconditioners + Krylov Methods

- Nonlinear Solvers,
- Unconstrained Minimization

- ODE Integrators

- Visualization

- Interface

## PETSc Numerical Libraries

- Sparse Row
- (AIJ)

- Blocked Compressed
- Sparse Row
- (BAIJ)

- Block
- Diagonal
- (BDIAG)

- Dense

- Others

- Indices

- Block Indices

- Stride

- Others

- Index Sets

- Vectors

- Line Search

- Trust Region

- Newton-based Methods

- Others

- Nonlinear Solvers

- Additive
- Schwartz

- Block
- Jacobi

- Jacobi

- ILU

- ICC

- LU
- (Sequential only)

- Others

- Preconditioners

- Euler

- Backward
- Euler

- Pseudo Time
- Stepping

- Others

- Time Steppers

- GMRES

- CG

- CGS

- Bi-CG-STAB

- TFQMR

- Richardson

- Chebychev

- Others

- Krylov Subspace Methods

- Matrices

- Distributed Arrays

- Matrix-free

## Sample: Nonlinear Solvers

**Goal**:  For problems arising from PDEs,
- support the general solution of  *f(u) = 0,*
- where *f: R     R *
- User provides:
- Code to evaluate f*(u)*
- Code to evaluate Jacobian of* f(u) *(optional)
- or use sparse finite difference approximation
- or use automatic differentiation
- AD support via collaboration with P. Hovland and B. Norris (see http://www.mcs.anl.gov/autodiff)

*n*

*n*

## Nonlinear Solvers (SNES)

- Newton-based methods, including
- Line search strategies
- Trust region approaches
- Pseudo-transient continuation
- Matrix-free variants
- Single solver interface, supporting
- uniprocessor and parallel versions
- all algorithms for a particular problem class
- real and complex arithmetic
- User can customize all phases of the solution process

## Nonlinear PDE Solution

- User code

- Application
- Initialization

- Function
- Evaluation

- Jacobian
- Evaluation

- Post-
- Processing

- PC

- KSP

- PETSc

- Linear Solvers (SLES)

- Nonlinear Solvers (SNES)

- Solve
*F(u) = 0*

- Application Driver

## Basic Nonlinear Solver Code (C/C++)

- SNES   snes;                               /*  nonlinear solver context  */
- Mat      A;                                     /*  Jacobian matrix  */
- Vec      x, F;                                 /*  solution, residual vectors  */
- int        n, its;                               /*  problem dimension, number of iterations  */
- ApplicationCtx  usercontext;       /*  user-defined application context  */
- ...
- MatCreate(MPI_COMM_WORLD,n,n,&J);
- VecCreate(MPI_COMM_WORLD,n,&x);
- VecDuplicate(x,&F);
- SNESCreate(MPI_COMM_WORLD,SNES_NONLINEAR_EQUATIONS,&snes);
- SNESSetFunction(snes,F,EvaluateFunction,usercontext);
- SNESSetJacobian(snes,J,EvaluateJacobian,usercontext);
- SNESSetFromOptions(snes);
- SNESSolve(snes,x,&its);
- MatDestroy(J);  VecDestroy(x);  VecDestroy(F);  SNESDestroy(snes);

## Basic Nonlinear Solver Code (Fortran)

- SNES   snes
- Mat      A
- Vec      x, F
- int        n, its
- ...
- call MatCreate(MPI_COMM_WORLD,n,n,J,ierr)
- call VecCreate(MPI_COMM_WORLD,n,x,ierr)
- call VecDuplicate(x,F,ierr)
- call SNESCreate(MPI_COMM_WORLD,SNES_NONLINEAR_EQUATIONS,snes,ierr)
- call SNESSetFunction(snes,F,EvaluateFunction,PETSC_NULL,ierr)
- call SNESSetJacobian(snes,J,EvaluateJacobian,PETSC_NULL,ierr)
- call SNESSetFromOptions(snes,ierr)
- call SNESSolve(snes,x,its,ierr)
- call SNESDestroy(snes,ierr)
- ...

## Uniform Access to Solvers

- Procedural interface
- SNESSetType(snes,”ls”);
- SNESSetLineSearch(snes,SNESQuadraticLineSearch);
- ...
- Run-time interface
- -ksp_type [cg,gmres,bcgs,tfqmr,...]
- -pc_type [lu,ilu,jacobi,sor,asm,...]
- -snes_type [ls,tr,...]
- -snes_line_search <line search method>
- -sles_ls <parameters>
- -snes_convergence <tolerance>
- ...

## CFD on an Unstructured Mesh

- 3D incompressible Euler
- Tetrahedral grid
- Up to 11 million unknowns
- Based on a legacy NASA code, FUN3d, developed by W. K. Anderson
- Fully implicit steady-state
- Primary PETSc tools: nonlinear solvers (SNES)  and vector scatters (VecScatter)

*Results courtesy of D. Kaushik and D. Keyes*

## Fixed-size Parallel Scaling Results

- Dimension=11,047,096

## Fixed-size Parallel Scaling Results

## Inside the Parallel Scaling Results on ASCI Red

- ONERA M6 wing test case, tetrahedral grid of 2.8 million vertices (about 11 million unknowns)
- on up to 3072 ASCI Red nodes (each with dual Pentium Pro 333 MHz processors)

## Nonlinear PDE Solution

- User code

- Application
- Initialization

- Function
- Evaluation

- Jacobian
- Evaluation

- Post-
- Processing

- PC

- KSP

- PETSc

- Linear Solvers (SLES)

- Nonlinear Solvers (SNES)

- Solve
*F(u) = 0*

- Application Driver

** ****Automatic Differentiation (AD):** a technology for automatically augmenting computer programs, including arbitrarily complex simulations, with statements for the computation of derivatives, also known as sensitivities.
** ****AD Collaborators**: P. Hovland and B. Norris (http://www.mcs.anl.gov/autodiff)

## Nonlinear PDE Solution

- Application
- Initialization

- Post-
- Processing

- PETSc

- Solve
*F(u) = 0*

- Nonlinear Solvers (SNES)

- Main Routine

- Parallel Jacobian
-  assembly

- Global-to-local
- scatter of ghost  values

- Seed matrix
- initialization

- Local Jacobian
- computation

## Using AD with PETSc

**G****lobal-to-local **
**scatter of ghost  values**

**Parallel function**
**assembly**

**Local Function **
**computation**

**Parallel Jacobian**
** ****assembly**

**G****lobal-to-local **
**scatter of ghost  values**

**Local Jacobian**
**computation**

**Local Function **
**computation**

**ADIFOR or ADIC**

**Local Jacobian**
**computation**

**Script file**

**Seed matrix **
**initialization**

-  Fully automated for structured meshes
-  Currently manual setup for unstructured    meshes; can be automated

- Current status:

## Hybrid FD/AD Strategy for Jacobian-vector Products

- FD
*F’(x) v = [ F(x+hv) - F(x)] / h*
- costs approximately 1 function evaluation
- challenges in computing the differencing parameter, *h, *since we must balance truncation and round-off errors
- AD
- costs approximately 2 function evaluations
- no difficulties in parameter estimation
- Hybrid FD/AD
- switch from FD to AD when  ||F|| / ||F   || < 

- 0

- Euler model; transonic flow over ONERA M6 wing

## One-to-one Interfacing

- Linear solvers
- AMG  _[http://www.](http://www.mgnet.org/mgnet-codes-gmd.html)__[mgnet](http://www.mgnet.org/mgnet-codes-gmd.html)__[.org/](http://www.mgnet.org/mgnet-codes-gmd.html)__[mgnet](http://www.mgnet.org/mgnet-codes-gmd.html)__[-codes-](http://www.mgnet.org/mgnet-codes-gmd.html)__[gmd](http://www.mgnet.org/mgnet-codes-gmd.html)__[.html](http://www.mgnet.org/mgnet-codes-gmd.html)_
- BlockSolve95  _[http://www.](http://www.mcs.anl.gov/BlockSolve95)__[mcs](http://www.mcs.anl.gov/BlockSolve95)__[.](http://www.mcs.anl.gov/BlockSolve95)__[anl](http://www.mcs.anl.gov/BlockSolve95)__[.](http://www.mcs.anl.gov/BlockSolve95)__[gov](http://www.mcs.anl.gov/BlockSolve95)__[/BlockSolve95](http://www.mcs.anl.gov/BlockSolve95)_
- DSCPACK  http://www.cse.psu.edu/~raghavan/dscpack
- Hypre   www.llnl.gov/casc/hypre
- ILUTP  _[http://www.](http://www.cs.umn.edu/~saad/)__[cs](http://www.cs.umn.edu/~saad/)__[.](http://www.cs.umn.edu/~saad/)__[umn](http://www.cs.umn.edu/~saad/)__[.](http://www.cs.umn.edu/~saad/)__[edu](http://www.cs.umn.edu/~saad/)__[/~](http://www.cs.umn.edu/~saad/)__[saad](http://www.cs.umn.edu/~saad/)__[/](http://www.cs.umn.edu/~saad/)_
- LUSOL  _[http://www.](http://www.sbsi-sol-optimize.com/)__[sbsi](http://www.sbsi-sol-optimize.com/)__[-sol-optimize.com](http://www.sbsi-sol-optimize.com/)_
- SPAI  _[http://www.](http://www.sam.math.ethz.ch/~grote/spai)__[sam](http://www.sam.math.ethz.ch/~grote/spai)__[.math.](http://www.sam.math.ethz.ch/~grote/spai)__[ethz](http://www.sam.math.ethz.ch/~grote/spai)__[.](http://www.sam.math.ethz.ch/~grote/spai)__[ch](http://www.sam.math.ethz.ch/~grote/spai)__[/~](http://www.sam.math.ethz.ch/~grote/spai)__[grote](http://www.sam.math.ethz.ch/~grote/spai)__[/](http://www.sam.math.ethz.ch/~grote/spai)__[spai](http://www.sam.math.ethz.ch/~grote/spai)_
- Spooles   http://www.netlib.org/linalg/spooles
- SuperLU  _[http://www.](http://www.nersc.gov/~xiaoye/SuperLU)__[nersc](http://www.nersc.gov/~xiaoye/SuperLU)__[.](http://www.nersc.gov/~xiaoye/SuperLU)__[gov](http://www.nersc.gov/~xiaoye/SuperLU)__[/~](http://www.nersc.gov/~xiaoye/SuperLU)__[xiaoye](http://www.nersc.gov/~xiaoye/SuperLU)__[/](http://www.nersc.gov/~xiaoye/SuperLU)__[SuperLU](http://www.nersc.gov/~xiaoye/SuperLU)_

- Optimization software
- TAO  http://www.mcs.anl.gov/tao
- Veltisto  http://www.cs.nyu.edu/~biros/veltisto
- Mesh and discretization tools
- Overture  _[http://www.](http://www.llnl.gov/CASC/Overture)__[llnl](http://www.llnl.gov/CASC/Overture)__[.](http://www.llnl.gov/CASC/Overture)__[gov](http://www.llnl.gov/CASC/Overture)__[/CASC/Overture](http://www.llnl.gov/CASC/Overture)_
- SAMRAI  _[http://www.](http://www.llnl.gov/CASC/SAMRAI)__[llnl](http://www.llnl.gov/CASC/SAMRAI)__[.](http://www.llnl.gov/CASC/SAMRAI)__[gov](http://www.llnl.gov/CASC/SAMRAI)__[/CASC/SAMRAI](http://www.llnl.gov/CASC/SAMRAI)_
- SUMAA3d  _[http://www.](http://www.mcs.anl.gov/sumaa3d)__[mcs](http://www.mcs.anl.gov/sumaa3d)__[.](http://www.mcs.anl.gov/sumaa3d)__[anl](http://www.mcs.anl.gov/sumaa3d)__[.](http://www.mcs.anl.gov/sumaa3d)__[gov](http://www.mcs.anl.gov/sumaa3d)__[/sumaa3d](http://www.mcs.anl.gov/sumaa3d)_
- ODE solvers
- PVODE  _[http://www.](http://www.llnl.gov/CASC/PVODE)__[llnl](http://www.llnl.gov/CASC/PVODE)__[.](http://www.llnl.gov/CASC/PVODE)__[gov](http://www.llnl.gov/CASC/PVODE)__[/CASC/PVODE](http://www.llnl.gov/CASC/PVODE)_
- Others
- Matlab  _[http://www.](http://www.mathworks.com/)__[mathworks](http://www.mathworks.com/)__[.com](http://www.mathworks.com/)_
- ParMETIS _[http://www.](http://www.cs.umn.edu/~karypis/metis/parmetis)__[cs](http://www.cs.umn.edu/~karypis/metis/parmetis)__[.](http://www.cs.umn.edu/~karypis/metis/parmetis)__[umn](http://www.cs.umn.edu/~karypis/metis/parmetis)__[.](http://www.cs.umn.edu/~karypis/metis/parmetis)__[edu](http://www.cs.umn.edu/~karypis/metis/parmetis)__[/~](http://www.cs.umn.edu/~karypis/metis/parmetis)__[karypis](http://www.cs.umn.edu/~karypis/metis/parmetis)__[/](http://www.cs.umn.edu/~karypis/metis/parmetis)__[metis](http://www.cs.umn.edu/~karypis/metis/parmetis)__[/](http://www.cs.umn.edu/~karypis/metis/parmetis)__[parmetis](http://www.cs.umn.edu/~karypis/metis/parmetis)_

- Between PETSc and ...

## Toolkit for Advanced Optimization

- S. Benson, L. C. McInnes, J. Moré, J. Sarich
- http://www.mcs.anl.gov/tao
- Targets solution of large-scale optimization problems, begun in 1997

## Optimization Using TAO

- n
- i
- t
- i
- a
- l
- i
- z
- a
- t
- i
- o
- n

- P
- o
- s
- t
- -
- P
- r
- o
- c
- e
- s
- s
- i
- n
- g

- TAO code

- User code

- External linear algebra tools

**Vector**

**Matrix**

- TAO Optimization Solvers

**Linear Solver**

- Application
- Objective, Constraint, and Derivative Evaluation

- Application Driver

## Outline

- Motivation
- Complex, multi-physics, multi-scale applications
- Distributed, multi-level memory hierarchies
- Parallel Software for PDEs
- Approach
- Performance
- Next Generation Software
- What are components?
- Common Component Architecture (CCA) Forum
- Ongoing Challenges

## What are components?

- OO techniques are useful for building individual components by relatively small teams; component technologies facilitate sharing of code developed by different groups by addressing issues in
- Language interoperability
- Via interface definition language (IDL)
- Well-defined abstract interfaces
- Enable “plug-and-play”
- Dynamic composability
- Components can discover information about their environment (e.g., interface discovery) from framework and connected components
- For more info: C. Szyperski, *Component Software: Beyond Object-Oriented Programming*, ACM Press, New York, 1998

## Goals of the Common Component Architecture (CCA) Forum

- Existing component architecture standards such as CORBA, Java Beans, and COM  do not provide support for parallel components.

- CORBA/Java

**CCA**

- MPI

- Latency
- between
- components

- Desire to build scientific applications by hooking together components
- DOE Common Component Architecture (CCA) provides a mechanism for interoperability of high-performance components developed by many different groups in different languages or frameworks.

## CCA History and Participants

- 1998: CCA Forum originated
- Participation from researchers who were exploring one-to-one software interfacing in the DOE ACTS Toolkit program
- Open to everyone interested in HPC components
- See _[http://www.](http://www.cca-forum.org/)__[cca](http://www.cca-forum.org/)__[-forum.org](http://www.cca-forum.org/)_
- Active CCA Forum participants include
- ANL - Lori Freitag, Kate Keahey, Jay Larson, Lois McInnes, Boyana Norris
- Indiana Univ. - Randall Bramley, Dennis Gannon
- LANL - Craig Rasmussen, Matt Sotille
- LLNL - Scott Kohn, Gary Kumfert, Tom Epperly
- ORNL - David Bernholdt, Jim Kohl
- PNNL - Jarek Nieplocha, Theresa Windus
- SNL - Rob Armstrong, Ben Allan, Curt Janssen, Jaideep Ray
- Univ. of Utah - Steve Parker
- And others as well ...
- 2001: Center for Component Technology for Terascale Simulation Software (CCTTSS) founded
- Support from the DOE SciDAC Initiative
- CCTTSS team is a subset of the CCA Forum
- Leader: Rob Armstrong (SNL)
- See _[http://www.](http://www.cca-forum.org/ccttss)__[cca](http://www.cca-forum.org/ccttss)__[-forum.org/](http://www.cca-forum.org/ccttss)__[ccttss](http://www.cca-forum.org/ccttss)_

## CCTTSS Multi-Pronged Approach

**HPC component specification and framework** – coordinator Scott Kohn (LLNL)
- Unified reference framework implementation targeting both SPMD and distributed environments
- Tools for language interoperability via a Scientific Interface Definition Language (SIDL)
**Suite of scientific components** – coordinator Lois Curfman McInnes (ANL)
- Linear and nonlinear algebra, optimization, mesh management, scientific data, visualization, steering, fault tolerance, scientific application domains, etc.
**Parallel data redistribution** – coordinator Jim Kohl (ORNL)
- Model coupling, visualization
**Applications integration** – coordinator David Bernholdt (ORNL)
- General outreach to the scientific community
- Close feedback loop for users/developers of CCA technology
- Collaborate with climate and chemistry applications domains as well as other groups

**CCTTSS Leader: Rob Armstrong (SNL)**

## Requirements for a High-Performance Component Architecture

- Simple/Flexible
- to adopt
- to understand
- to use
- Support a composition mechanism that does not impede high-performance component interactions
- Permit the SPMD paradigm in component form
- Meant to live with and rely on **other** commodity component frameworks to provide services ...
- e.g., JavaBeans, CORBA, ...

## CCA Approach

- CCA specification dictates a basic set of interfaces (and corresponding behaviors) that components should implement to be CCA compliant.
*Ports *define the connection model for component interactions
*Provides/Uses* design pattern
- Components are manipulatable in a framework.
- CCA specification doesn’t dictate frameworks or runtime environment.
- Create components that are usable under a variety of frameworks
- Provide a means for discovering interfaces
- Specifically exclude *how* the components are linked; that is the job of a framework
- Provide language-independent means for creating components (via SIDL)

**Component-Based**
**Scientific Application**

**Discretization**
**Engine**

**Implicit Solve**

**Visualization**

**Ports**

**Link**

**ANL**

**SNL**

**LLNL**

**PNNL**

**LANL**

**UU**

**IU**

**ORNL**

**framework**

## CCA Concept of SPMD Components

**A**

**A**

**A**

**A**

- MPI

- MPI

- Process

- MPI application using CCA for interaction between
- components A and B within the same address space

- Adaptive mesh
-  component
- written by user1

- Solver component
- written by user2

**Direct**
**Connection**
- supplied by
- framework at
- compile/runtime

- CCA Concept of SPMD Components

**Proc1**

**Proc2**

**Proc3**

**etc...**

## CCA Collective Port Modularizes Processor/Data Decomposition 

- parallel
- visualization component

- collective port
- connecting M procs
- with N procs

- Combining previous parallel component with another
- parallel component in a different framework

- container composed of
- mesh and solver
- components

## CCA References

- Web sites
- CCA Forum
- _[http://www.cca-forum.org](http://www.cca-forum.org/)_
- Center for Component Technology for Terascale Simulation Software (CCA SciDAC Center)
- _[http://www.cca-forum.org/](http://www.cca-forum.org/ccttss)__[ccttss](http://www.cca-forum.org/ccttss)_
- Sample component software and applications
- _[http://www.cca-forum.org/cca-sc01](http://www.cca-forum.org/cca-sc01)_
- Introductory paper
- R. Armstrong, D. Gannon, A. Geist, K. Keahey, S. Kohn, L. McInnes, S. Parker, and B. Smolinski, **Toward a Common Component Architecture for High-Performance Scientific Computing**, Proceedings of the High-Performance Distributed Computing Conference, pp. 115-124, 1999.

## More CCA Papers

- B. Norris, S. Balay, S. Benson, L. Freitag, P. Hovland, L. McInnes, and B. Smith, **Parallel Components for PDEs and Optimization: Some Issues and Experiences**, preprint ANL/MCS-P932-0202, February 2002, Parallel Computing (to appear).
- B. Allan, R. Armstrong, A. Wolfe, J. Ray, D. Bernholdt, and J. Kohl, **The CCA Core Specification in a Distributed Memory SPMD Framework**, August 2001, Concurrency and Computation: Practice and Experience (to appear).
- T. Epperly, S. Kohn, and G. Kumfert. **Component Technology for High-Performance Scientific Simulation Software,** Proceedings of the International Federation for Information Processing’s Working Conference on Software Architectures for Scientific Computing, 2000.
- S. Parker, **A** **Component-based Architecture for Parallel Multi-Physics PDE Simulations**, Proceedings of the 2002 International Conference on Computational Science (to appear).
- M. Sottile and C. Rasmussen, **Automated Component Creation for Legacy C++ and Fortran Codes**, Proceedings of the First International IFIP/ACM Working Conference on Component Deployment, June 2002 (submitted).
- R. Bramley, K. Chiu, S. Diwan, D. Gannon, M. Govindaraju, N. Mukhi, B. Temko, and M. Yechuri, **A Component Based Services Architecture for Building Distributed Applications*****,*** Proceedings of High Performance Distributed Computing, 2000.
- K. Keahey, P. Beckman, and J. Ahrens, **Ligature: A Component Architecture for High-Performance Applications**, International Journal of High-Performance Computing Applications, 2000.

## Related Work

- N. Furmento, A. Mayer, S. McGough, S. Newhouse, T. Field, and J. Darlington, **Optimization of Component-based Applications within a Grid Environment**, Proceedings of SC2001.
- C. René, T. Priol, and G. Alléon, **Code Coupling Using Parallel CORBA Objects**, Proceedings of the International Federation for Information Processing’s Working Conference on Software Architectures for Scientific Computing, 2000.
- E. de Sturler, J. Hoeflinger, L. Kale, and M. Bhandarkar, **A New Approach to Software Integration Frameworks for Multi-physics Simulation Codes**, Proceedings of the International Federation for Information Processing’s Working Conference on Software Architectures for Scientific Computing, 2000.
- R. Sistla, A. Dovi, P. Su, and R. Shanmugasundaram, **Aircraft Design Problem Implementation Under the Common Object Request Broker Architecture**, Proceedings of the 40th AIAA/ASME/ASCH/AHS/ASC Structures, Structural Dynamics, and Materials Conference, 1999.

## Unconstrained Minimization Example Using CCA Components

**Optimization**
**u****i+1**** = u****i**** + ********s ...**

**Visualization**

**Linear Solver**
**H s = g**

**H**

**g**

**s**

**g**

**H**

**f**

**u****i+1**

**u****o**

- function
- solution
- Hessian
- gradient
- coordinates
- connectivity
- step direction

**f(u)**
**u **
**H**
**g **
**x **
**c **
**s**

- Compute **min f(u)**

**CCAFFEINE Framework**

- Trilinos

- PETSc

- Others ...

**Driver**

-   CCAFFEINE – Common Component Architecture Fast Framework
-            Example in Need of Everything
-  reference framework under development by B. Allan et al. (SNL)
-  _[http://www.cca-forum.org/cafe.html](http://www.cca-forum.org/cafe.html)_
-  TAO – Toolkit for Advanced Optimization
-  _[http://www.mcs.anl.gov/tao](http://www.mcs.anl.gov/tao)_
-  Optimization component developers:  S. Benson, L. C. McInnes,
-             B. Norris, and J. Sarich

**Local Physics,**
**Discretization**

**Data **
**Redistribution**

## Component Wiring Diagram

- Using GUI tool within CCAFFEINE framework

## Performance on a Linux Cluster

- Newton method with line search
- Solve linear systems with the conjugate gradient method and block Jacobi preconditioning (with no-fill incomplete factorization as each block’s solver, and 1 block per process)
- Negligible component overhead; good scalability

-   Total execution time for a minimum surface minimization problem using a fixed-sized 250x250 mesh.
-   Dual 550 MHz Pentium-III nodes with 1 G RAM each, connected via Myrinet

## CCA Compliance in TAO

- Paradigm shift; both TAO and the application become components
- Each is required to provide a default constructor and to implement the CCA component interface
- contains one method: “setServices” to register ports
- All interactions between components use ports
- Application ***provides*** a “go” port and ***uses*** “taoSolver” port
- TAO ***provides*** a “taoSolver” port
- There is no “main”  routine
- Status
- TAO-1.4, released April 2002, includes prototype CCA component interfaces (C++)
- Now also have prototype SIDL interfaces for optimization components; working on some new demos and applications, expect to release in late 2002

## Molecular Geometry Optimization

- Electronic structure components based on NWChem (PNNL) and MPQC (SNL)
- Optimization components based on TAO (ANL)
- Linear algebra components based on Global Arrays (PNNL) and PETSc (ANL)

- Relativistic quantum chemistry calculation of (UO  )   (CO  )   using NWChem.  Image courtesy of Wibe deJong, PNNL.

- 2   3           3   6

- New work with T. Windus (PNNL) and C. Janssen (SNL):

## Sample CCA Components and Applications

- Developed by CCA working group for demonstration at SC01
- 4 applications using CCAFFEINE
- Unconstrained minimization problem on a structured mesh
- Time-dependent PDE on an unstructured mesh
- Time-dependent PDE on an adaptive structured mesh
- Ping-pong MxN
- More than 30 components
- Many components re-used in 3 apps
- Leverage and extend parallel software developed at different institutions
- including CUMULVS, GrACE, MPICH, ODEPACK, PAWS, PETSc, PVM, TAO, and Trilinos
- Source code and documentation available via
- http://www.cca-forum.org/cca-sc01

## Summary

- Object-oriented techniques have been effective in enabling individual libraries for high-performance numerics to explore of trade-offs in
- Algorithms, data structures, data distribution, etc.
- The CCA Forum is developing component technology specifically targeted at high-performance scientific simulations
- Addressing issues in language interoperability, dynamic composability, abstract interfaces, parallel data redistribution, etc.
- Aiming to enable the exploration of trade-offs in the broader context of multi-disciplinary simulations that require the combined use of software developed by different groups
- We have a solid start through an interdisciplinary, multi-institution team
- Open to everyone interested in high-performance scientific components (see http://www.cca-forum.org for info on joining the CCA mailing list)
**Lots** of research challenges remain!

## One Challenge:Interfaces are central

- One Challenge:Interfaces are central

- The CCA Forum participants do not pretend to be experts in all phases of computation, but rather just to be developing a standard way to exchange component capabilities.
- Medium of exchange: interfaces
**Need experts in various areas to define sets of domain-specific abstract interfaces**
- scientific application domains, meshes, discretization, nonlinear solvers, optimization, visualization, etc.
- Developing common interfaces is difficult
- Technical challenges
- Social challenges

**This **
**means **
**you!**

- Many, many additional research issues remain.

## More Information

- TAO:  http://www.mcs.anl.gov/tao
- CCA: http://www.cca-forum.org