# Presentation: 852164

## SciDAC TOPS PETSc Work

- http://www.mcs.anl.gov/petsc
- petsc-maint@mcs.anl.gov

- SciDAC TOPS Developers
- Satish Balay
- Chris Buschelman
- Matt Knepley
- Barry Smith
- Hong Zhang
- Other PETSc Team Members
- Lois Curfman McInnes, Bill Gropp, Dinesh Kaushik

## PETSc

- Portable, Extensible, Software for the scalable solution of nonlinear PDEs
- Focusing on algebraic solvers
- Preconditioners, multigrid, Krylov methods, Newton, FAS, time integrators

## Activities

- Derivative calculations – ADIC/ADIFOR
- Matrix-free preconditioners
- FAS (future)
- Accessing other software
- HYPRE preconditioner usage
- RAMG,SAMG
- SuperLU
- ESI
- SIDL/Babel/CCA – PETSc 3.0 plans
- SIDL/Babel MPI

## Derivatives

- Automatic tools ADIC/ADIFOR are great but require manual processing (painful when developing code)
- Would like to have a variety of derivatives “for free” (or nearly)
- Matrix element based Newton which requires explicit sparse Jacobian
- Matrix-free Newton
- SOR preconditioners and FAS require one equation at at time

## To Fully Automate

- Code comments to indicate what needs to be processed
- Predefined calling sequences
- Libraries that can use generated code
- Make system automatically generates required derivative code

## ..

- /* Process adiC: FFL */
- int FFL (DALocalInfo *info,Field **x,
-                          Field **f,void *ptr)
-  {
-   AppCtx       *user = (AppCtx*)ptr;
- DMMGSetSNESLocal(dmmg,FFL,0,ad_FFL,admf_FFL);

- typedef struct {
-   PassiveReal  lidvelocity,prandtl,grashof;
-   PetscTruth   draw_contours;
- } AppCtx;

- typedef struct {
-   double u,v,omega,temp;
- } Field;

## > make BOPT=g_c++ ex19

- Running adiC on function FFL
- Extracting structure Field
- Extracting structure AppCtx
- Extracting function FFL
- adiC -a -d gradient -i /sandbox/bsmith/petsc/bmake/adic.init ex19.c.tmp
- ADIC 1.1 beta 4 08/07/00 18:06:31
- This software was created in the course of academic and/or research
- endeavors by Argonne National Laboratory. See the disclaimer in the
- g++ -fPIC -o ex19.o -c -Dad_GRAD_MAX=36
- -I/sandbox/bsmith/petsc/include/adic -g -I/sandbox/bsmith/petsc
- ..

## Matrix-free Multiply

- Differencing (well understood)
- AD

- int mf_FFL(DALocalInfo *info,mf_Field **x,
-               mf_Field  **f,void   *ptr) {

- typedef struct {
-   double value;double  grad[1];
- } mfDERIV_TYPE;

- typedef struct  {
-   mfDERIV_TYPE  u, v, omega, temp;
- }

## Matrix-free Multigrid Preconditioners

- Krylov smoothing with
- no preconditioner
- (point block) Jacobi preconditioner
- (point block) SOR preconditioning

## Matrix-free Jacobi/SOR

- If function is provided an “(block) equation at a time” matrix-free multiply can be done an equation at a time
-      can be computed by providing a seed of
- For SOR
- Similar computations with differencing

## FAS – Full Approximation Scheme (Nonlinear Multigrid)

- Smoothing uses Newton to solve a single (block) equation at a time
- Need the same “(block) equation at a time” and its derivative as for matrix-free Newton!

## We are developing a common software infrastructure 

- and user interface for many Newton methods and FAS
- which can use AD or differencing for derivatives.

## Activities

- Derivative calculations – ADIC/ADIFOR
- Matrix-free preconditioners
- FAS (future)
- Accessing other software
- HYPRE preconditioner usage
- RAMG, SAMG
- SuperLU
- ESI
- SIDL/Babel/CCA – PETSc 3.0 plans
- SIDL/Babel MPI

## Philosophy on Solvers

- Use any solver without changing the application code
- Add to application code to take advantage of special solver features (optimizations)
- but additions cannot break other solvers

- PCILUSetDropTolerance(pc,1.e-5);
- PCSPAISetEpsilon(pc,1.e-2);

## HYPRE-PETSc Matrix Element based preconditioners

- No vector copies
- Currently a matrix copy (plan to remove)

- -pc_type hypre -pc_hypre_type
-            <pilut,parasails,boomerAMG,euclid>
- -pc_hypre_pilut_tol 1.e-4
- -pc_hypre_parasails_thresh .1
- -pc_hypre_parasails_filter .2

## RAMG, SAMG – PETScSequential Preconditioners

- AMG – written by John Ruge, Klaus Stueben, and Rolf Hempel. A classic scalar PDE AMG code.
- SAMG – written by Klaus Stueben. A complete rewrite that supports systems of PDEs (interface coming soon).
- PETSc interfaces provided by Domenico Lahaye

## SuperLU_Dist

- MPI parallel LU factorization and solve written by Sherry Li and Jim Demmel
- Henry Tufo – Paul Fischer XYT fast direct parallel solver for “small problems”, i.e. the coarse grid problem in multigrid

- -mat_aij_superlu_dist
- -mat_aij_superlu_dist_colperm
-   <MMD_AT_PLUS_A,NATURAL,MMD_ATA,COLAMD>

## ESI Equation Solver Interface

- A C++ “standard” of abstract classes for setting up and solving linear systems
- At least three implementations
- Trilinos (Sandia, NM)
- ISIS (Sandia, CA)
- PETSc

## PETSc ESI Goal

- Allow PETSc to use matrices and solvers written by others in ESI
- Allow anyone to use PETSc matrices and solvers from ESI
- Achieved by using two sets of wrappers
- PETSc objects “look like” ESI objects
- ESI objects “look like” PETSc objects

## -is_type esi –is_esi_type Petra_ESI_IndexSpace

- -vec_type esi -vec_esi_type Petra_ESI_Vector
- -mat_type esi –mat_esi_type Petra_ESI_CSR_Matrix

- PETSc using Trilinos objects

- ESI using PETSc objects

- esi::Vector<double,int> *v = new esi::petsc::Vector<double,int>(petscvector)
- esi::Operator<double,int> *o = new esi::petsc::Matrix<double,int>(petscmatrix)

## Activities

- Derivative calculations – ADIC/ADIFOR
- Matrix-free preconditioners
- FAS (future)
- Accessing other software
- HYPRE preconditioner usage
- RAMG, SAMG
- SuperLU
- ESI
- SIDL/Babel/CCA – PETSc 3.0 plans
- SIDL/Babel MPI

## SIDL/Babel (LLNL)

- Scientific Interface Definition Language
- Babel – generates language stubs for
- C, C++, Fortran 77, Python, Java, ...
- Provides
- Object model
- Exception (error) model
- Immature, but very exciting development

## PETSc 3.0

- SIDL/Babel for inter-language, object model and exception model
- Multiple precisions
- Single for preconditioners?
- Vector and matrix sizes mutable
- Adaptive mesh refinement
- Not a rewrite, just a rewrap!

## Activities

- Derivative calculations – ADIC/ADIFOR
- Matrix-free preconditioners
- FAS (future)
- Accessing other software
- HYPRE preconditioner usage
- RAMG, SAMG
- SuperLU
- ESI
- SIDL/Babel/CCA – PETSc 3.0 plans
- SIDL/Babel MPI

## Conclusions

- Many new tools and solvers
- Accessible via
- PETSc 2.0 interface
- ESI
- (coming) SIDL/Babel/CCA
- We are happy to support users, especially those interested in nontrivial problems