# Presentation: 153139

## DOE 2000 Numerics Capability

- Argonne National Laboratory

- http://acts.nersc.gov

- MCS Review 1998

## Targets 

- Large Scale (100,000,000 dof) simulations
- computational fluid dynamics
- combustion
- structures
- materials
- usually PDE based
- Large scale (1000 cv) optimization
- often involving simulation
- maybe stochastic

- MCS Review 1998

##  Organization of Presentation

- Sample computations (to give an idea of scale)
- Background on computational infrastructure
- Summary of current toolkits
- Efforts on toolkit interoperability

- MCS Review 1998

## Oil Reservoir Simulation Multiphase Flow

- Three-dimensional EOS model with 3 components (8 DOF per cell)
- Structured Cartesian grid
- Over 4 million cell blocks, 32 million DOF
- Fully implicit, time-dependent
- Over 10.6 gigaflops sustained performance on 128 nodes of an IBM SP
- Entire simulation runs in less then 30 minutes

- MCS Review 1998

##  Ocean Circulation Model 

- Uses second-order finite                                                  differences with an                                                          explicit leapfrog time                                                stepping scheme on a                                                    logically rectangular grid.
- A. Wallcraft, H. Hurlburt, R. Rhodes, & J. Shriver, NRL - Stennis
- 540% speedup over the previous parallel software (64*145.6 Mflop/s) = 9.3 Gflop/s
- 2 hours is now 25 minutes

- MCS Review 1998

## CFD on an Unstructured Grid

- Three-dimensional incompressible Euler
- Tetrahedral grid
- Up to 11.0 million degrees of freedom
- Based on a legacy NASA code
- Now runs several hundred times faster than legacy code

- MCS Review 1998

## Unstructured Computational Grid

- MCS Review 1998

## Floating-Point Performance

**11,047,096 **
**Unknowns**

- MCS Review 1998

## DOE Hardware Infrastruture

- IBM SP
- Currently up to 512 nodes with 1-4 way SMP nodes
- MPI machine
- Processor peak ~ 640 MFlops
- SGI Origin
- Currently up to 128 processors in a box
- cc-NUMA - cache coherent - non-uniform memory access
- Boxes connected via message passing (MPI)
- Processor Peak ~ 500 MFlops
- Clusters (including ASCI Red, Linux, and DEC alphas)
-  7000+ processors
- MPI based

- MCS Review 1998

## DOE Numerics Software Infrastruture

- MPI basic model for parallelism
- Toolkits (libraries)
- in F77/F90, C, C++
- sometimes object oriented or at least object based
- hide much of the MPI details from application codes
- attempt to support use from multiple languages
- Applications
- F77/F90, C, C++
- some use of scripting languages (Python)

- MCS Review 1998

## Toolkit Categories

- PDE/Grid management systems
- ODE integrators
- Nonlinear solvers
- Eigenvalue computations
- Linear solvers
- Nonlinear optimization

- MCS Review 1998

## PDE/Grid management systems

- Overture
- overlapping composite grids
- complex geometry
- POOMA (Parallel Object-oriented Methods and Applications)
- Data parallel style support for
- finite difference
- particle methods

- MCS Review 1998

## Sample Overture Grid

- MCS Review 1998

## Sample Overture Grid

- MCS Review 1998

## ODE Integrators

- PVODE (Parallel variable order ODE integrators)
- can use linear solvers from other ACTS toolkits
- BDF (backward differentiation formula) multi-step methods for stiff problems
- Adams multi-step method for non-stiff
- also differential-algebraic systems

- MCS Review 1998

## Nonlinear Solvers

- PETSc (Portable Extensible Toolkit for Scientific computation)
- Kinsol (part of PVODE, may also be used independently)
- Newton-based solvers
- Globalization strategies include
- line search
- trust region
- pseudo-transient continuation (false time-stepping)
- Will support several toolkits’ linear solvers

- MCS Review 1998

## Linear Solvers

- Aztec
- General purpose iterative solver
- Hypre
- Family of application-centric (physics based) preconditioners
- PETSc
- Extensible family of general purpose iterative solvers
- QMRPack
- Full implementation of QMR
- ScaLAPACK
- Direct solvers

- MCS Review 1998

## Aztec

- General purpose parallel iterative solver
- Supports point and block storage of sparse matrices
- Includes 5 Krylov solvers
- Includes polynomial and overlapping additive Schwarz preconditioning
- Has scaled successfully to 1000s of processors (ASCI Red) on some applications

- MCS Review 1998

## Hypre

- Family of application-centric (physics or grid based) preconditioners

***Data Layout***

**structured**

**composite**

**block****-****struc**

**unstruc**

**CSR**

***Conceptual Interfaces***

***Algorithms***

**GMG****, ...**

**FAC****, ...**

**Hybrid****, ...**

**AMGe****, ...**

**ILU****, ...**

- MCS Review 1998

## PETSc

- Over 10 Krylov subspace methods including
- GMRES, TFQMR, CG, Bi-CG-stab
- A suite of preconditioners including
- additive overlapping Schwarz
- access to parallel ILU(0) and ICC(0) via BlockSolve95
- 5 matrix storage formats provided
- Many additional tools, such as vector scatter operations and mapping operations to simplify the coding of parallel PDE applications
- Modular and object oriented

- MCS Review 1998

## QMRPack

- Implements several variants of the quasi-minimal residual (QMR) algorithm for the solution of general,non-Hermitian linear systems
- Only implementation that includes the full suite of look-ahead techniques
- Includes an eigensolver
- Now interfacing with other ACTS toolkits

- MCS Review 1998

## ScaLAPACK

- SuperLU
- Direct solution of large, sparse nonsymmetric systems of linear equations.
- Runs on shared memory, MPI version expected soon
- Attained 8.3 Gflops on a 52,000 unknown linear system arising from device simulation. 512 processors on a Cray T3E. 014% to 2.4% matrix density.
- MPI and PVM based direct dense solvers
- Based on BLACS (basic linear algebra communication subroutines), BLAS and LAPACK
- Well known, trusted community standard

- MCS Review 1998

## Eigenvalue Computations

- ScaLAPACK
- Dense matrix eigensolvers
- Based on BLACS (basic linear algebra communication subroutines), BLAS and LAPACK
- Well known, trusted community standard
- Sustained 605 Gflops on ASCI Red in DOE materials simulations.

- MCS Review 1998

## Nonlinear Optimization

- Opt++
- Allows for rapid prototyping of new optimization methods
- Abstract base classes for nonlinear optimization problem:
-   NLP0:  no analytic derivatives
-  NLP1:  analytic first derivatives (gradient)
-  NLP2:  analytic first and second derivatives (gradient, Hessian)
-   Gradient methods for nonlinear least-squares problems
-    available in the NEWTON-LIKE optimization class:
- finite-difference Newton
- quasi-Newton
- Gauss-Newton
- Derived nonlinear problem classes:
- FDNLF1:  generate numerical-difference gradients
- NLF2:       first and second derivatives supplied

- MCS Review 1998

## DOE “Standards” Collaborators

- CCA - Common Component Architecture group: arising out of DOE2000 activities to develop “standard” ways of managing numerical components so that ones developed by different groups may be mixed-and-matched
- ESI - Equation Solver Interface Forum: group arising out of the DOE labs to develop common interfaces for scalable linear solvers

- MCS Review 1998

## DOE 2000 Numerics Capability

-    We provide a variety of powerful toolkits to assist in the development of scalable simulation codes
- http://acts.nersc.gov

- MCS Review 1998