# Presentation: 034342

## An Overview of TrilinosMichael A. HerouxSandia National Laboratories

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

**Notes:**
Questions from audience

Can you use another matrix-vector multiply?
Are multivectors supported?
Are Petsc and Hypre available via TSF?
How does Trilinos differ from Petsc?

## The Team

- Ross Bartlett
- Paul Boggs
- David Day
- Bob Heaphy
- Mike Heroux
- Robert Hoekstra
- Russ Hooper
- Vicki Howle
- Jonathan Hu
- Kris Kampshoff
- Tammy Kolda
- Rich Lehoucq

- Kevin Long
- Mike Phenow
- Eric Phipps
- Roger Pawlowski
- Andrew Rothfuss
- Andrew Salinger
- Paul Sexton
- Ken Stanley
- Heidi Thornquist
- Ray Tuminaro
- Jim Willenbring
- Alan Williams

## Outline of Talk

- Motivation
- Overview of Current Packages.
- Overview of New Packages.
- SQA/SQE.
- Availability and support.
- Concluding remarks.

## Motivation For Trilinos

- Sandia does LOTS of solver work.
- When I started at Sandia in May 1998:
- Aztec was a mature package.  Used in many codes.
- FETI, PETSc, DSCPack, Spooles, ARPACK, DASPK, and many other codes were (and are) in use.
- New projects were underway or planned in multi-level preconditioners, eigensolvers, non-linear solvers, etc...
- The challenges:
- Little or no coordination was in place to:
- Efficiently reuse existing solver technology.
- Leverage new development across various projects.
- Support solver software processes.
- Provide consistent solver APIs for applications.
- ASCI was forming software quality assurance/engineering (SQA/SQE) requirements:
- Daunting requirements for any single solver effort to address alone.

## Evolving Trilinos Solution

- Trilinos is an evolving framework to address these challenges:
- Includes common core set of vector, graph and matrix classes (Epetra).
- Provides a common abstract solver API (TSF).
- Provides a ready-made package infrastructure:
- Source code management (cvs, bonsai).
- Build tools (autotools).
- Automated regression testing (queue directories within repository).
- Communication tools (mailman mail lists).
- Specifies requirements and suggested practices for SQA.
- In general allows us to categorize efforts:
- Efforts best done at the Trilinos level (useful to most or all packages).
- Efforts best done at a package level (peculiar or important to a package).
- Allows package developers to focus only on things that are essential to their package.

## Trilinos Packages

- Trilinos is a collection of *Packages*.
- Each package is:
- Focused on important and state-of-the-art algorithms in its problem regime.
- Developed by a small team of domain experts.
- Self-contained: No (or minimal) explicit dependencies on any other software packages (with some special exceptions).
- Configurable/buildable/documented on its own.
- Sample packages: NOX, AztecOO, IFPACK.
- Special packages: Epetra, TSF, Teuchos.

## Two-level design:
Self-contained packages
Leveraged common tools.

- Two-level design:
- Self-contained packages
- Leveraged common tools.

- Ready-made package infrastructure:
- Source code management (cvs).
- Build tools (autotools).
- Automated regression testing (queue directories within repository).
- Communication tools (mailman mail lists).
- Requirements and suggested practices for SQA.

-  Key:
- 	Leveraging investments in software infrastructure ...  Without compromising individual package autonomy

-  Notes:
- ASCI Algorithms funds much of Trilinos development (LDRD, CSRF, MICS also).
- All packages available (except TOX).
- Public release: Sept. 5th.
- All information available at Trilinos website:
- 	_software.sandia.gov/trilinos_

## Three Special Trilinos Package Collections

- Epetra: Package of concrete linear algebra classes: Operators, matrices, vectors, graphs, etc.
- Provides working, parallel code for basic linear algebra computations.
- TSF: Packages of abstract solver classes: Solvers, preconditioners, matrices, vectors, etc.
- Provides an application programmer interface (API) to any other package that implements TSF interfaces.
- Teuchos: Package of basic tools:
- Common Parameter list, smart pointer, error handler, timer.
- Interface to BLAS, LAPACK, MPI, XML, ...
- Common traits mechanism.
- Goal: Portable tools that enhance interoperability between packages.

## Dependence vs. Interoperability

- Although Trilinos packages have no explicit dependence, each package must interact with *some* other packages:
- NOX needs operator, vector and solver objects.
- AztecOO needs preconditioner, matrix, operator, vector objects.
- Trilinos is a vehicle for:
- Leveraging investments in software infrastructure.
- Establishing interoperability of Trilinos components...
- Without compromising individual package autonomy.
- Trilinos offers five basic interoperability mechanisms.

## Trilinos Interoperability Mechanisms

- M1: Package accepts user data as Epetra objects.
- M2: Package can be used via TSF abstract solver classes.
- M3: Package can use Epetra for private data.
- M4: Package accesses solver services via TSF interfaces.
- M5: Package builds under Trilinos configure scripts.

##   Interoperability Example: AztecOO

- AztecOO: Preconditioned Krylov Solver Package.
- Primary Developer: Mike Heroux.
- Minimal *explicit, essential* dependence on other Trilinos packages.
- Uses abstract interfaces to matrix/operator objects.
- Has independent configure/build process (but can be invoked at Trilinos level).
- Sole dependence is on Epetra (but easy to work around).
*Interoperable* with other Trilinos packages:
- Accepts user data as Epetra matrices/vectors.
- Can use Epetra for internal matrices/vectors.
- Can be used via TSF abstract interfaces.
- Can be built via Trilinos configure/build process.
- Can provide solver services for NOX.
- Can use IFPACK, ML or AztecOO objects as preconditioners.

## Trilinos Concrete Support Component: Petra

- 1Petra is Greek for “foundation”.

- Petra1 provides distributed matrix and vector services.
- Exists in basic form as an object model:
- Describes basic user and support classes independent of language/implementation.
- Describes objects and relationships to build and use matrices, vectors and graphs.
- Has 3 implementations under development.

## Petra Implementations

- Three version under development:
- Epetra (*Essential Petra*):
- Current production version.
- Restricted to real, double precision arithmetic.
- Uses stable core subset of C++.
- Interfaces accessible to C and Fortran users.
- Tpetra (*Templated Petra*):
- Next generation C++ version.
- Templated scalar and ordinal fields.
- Uses namespaces, and STL: Improved usability/efficiency.
- Jpetra (*Java Petra*):
- Pure Java.  Completely portable to any JVM.
- Interfaces to Java versions of MPI, LAPACK and BLAS.

## 1st Special Package: Epetra

- Package of _concrete linear algebra classes_:
- 		Operators, matrices, vectors, graphs, etc.
- Working, parallel code for basic linear algebra computations.
- Uses stable core subset of C++
- C/Fortran wrappers
- Restricted to real, double precision arithmetic
- Concrete implementation of the Petra object model

## Epetra User Class Categories

- Sparse Matrices: 	***RowMatrix***, (CrsMatrix, VbrMatrix, FECrsMatrix, FEVbrMatrix)
- Linear Operator:	***Operator*****: **(AztecOO, ML, Ifpack)
- Dense Matrices:	DenseMatrix, DenseVector, BLAS, LAPACK,			SerialDenseSolver
- Vectors: 		Vector, MultiVector
- Graphs:		CrsGraph
- Data Layout: 		Map, BlockMap, LocalMap
- Redistribution:	Import, Export, LbGraph, LbMatrix
- Aggregates: 	 	LinearProblem
- Parallel Machine: 	***Comm***, (SerialComm, MpiComm, MpiSmpComm)
- Utilities: 	   	Time, Flops

## Summary of Epetra Features

- Basic Stuff: What you would expect.
- Variable block matrix data structures.
- Multivectors.
- Arbitrary index labeling.
- Flexible, versatile parallel data redistribution.
- Language support for inheritance, polymorphism and extensions.
- View vs. Copy.

## AztecOO

- Aztec is the workhorse solver at Sandia:
- Extracted from the MPSalsa reacting flow code.
- Installed in dozens of Sandia apps.
- 1600+ external licenses.
- AztecOO leverages the investment in Aztec:
- Uses Aztec iterative methods and preconditioners.
- AztecOO improves on Aztec by:
- Using Epetra objects for defining matrix and RHS.
- Providing more preconditioners/scalings.
- Using C++ class design to enable more sophisticated use.
- AztecOO interfaces allows:
- Continued use of Aztec for functionality.
- Introduction of new solver capabilities outside of Aztec.

## Trilinos Solver Framework (TSF)

- Epetra, AztecOO, Ifpack, ML, etc.PETSc, SuperLU, Hypre, HSL,ScaLapack
- TSF is an abstract class hierarchy:
- Provides uniform API to solvers, vectors, matrices.
- Allows integration of many solvers via implementation of abstract classes.
- Supports “generic” programming.
- Provides compositional classes.
- Composed of TSFExtended, TSFCore, TSFCoreUtils.

- Lots of good solver
- components available

## Generic Programming using TSF

- Generic Programming: 	Implementation of algorithms using abstract 	interfaces.
- Example: CG using TSF interfaces.
- Allows use of CG with *any* vector/matrix classes that implement TSF interfaces.
- Very powerful for complex algorithms: Robust Block GMRES, etc. (See Belos/Anasazi talk).

## Aggregate Operator Construction

- TSF facilitates implicit (and explicit) construction of operators:
- Partitioned (block):
- Composite:
- Sum:
- Inverse:
- Others: Zero, Identity, Transpose, ...
- Recursively.

## ML: Multi-level Preconditioners

- ML package developed by Ray Tuminaro and Jonathan Hu.
- Critical technology for scalable performance of some key apps.
- ML compatible with other Trilinos packages:
- Accepts user data as Epetra_RowMatrix object (abstract interface).
- Any implementation of Epetra_RowMatrix works.
- Implements the Epetra_Operator interface.
- Allows ML preconditioners to be used with AztecOO and (soon) TSF.
- Can also be used completely independent of other Trilinos packages.

## ML Approaches

- 4 ML approaches available:
- Algebraic (Vanek) approach.
- Color matrix graph to create “balls”.
- Create projection using balls and approximation to operator null space.
- Algebraic multigrid for Maxwell’s Equations.
- Special systems *Ax=b, *where *A = S + M.*
- Prolongation constructed to respect *ker(S).*
- Adaptive Grid approach.
- Need fine grid and restriction operator.
- Coarse operator is often easy to determine, e.g., weighted injection.
- 2 Grid approach.
- Fine and (very) coarse grid required.
- Graph and coordinates required.
- No correlation required between points on each grid.

## IFPACK: Algebraic Preconditioners

- Overlapping Schwarz preconditioners.
- Accept user matrix via abstract matrix interface (Epetra versions).
- Uses Epetra for basic matrix/vector calculations.
- Supports simple perturbation stabilizations and condition estimation.
- Separates graph construction from factorization, improves performance substantially.
- Compatible with AztecOO and TSF.

## Komplex: Complex linear solver

- Most algorithms work for complex numbers (with real numbers as a special case).
- Majority of our applications produce real-valued data.
- Solver development has been focused on real-valued problems.
- Writing complex versions of all software is not appealing.
- Alternative:  Consider equivalent real formulations (ERFs).
- Komplex is an add-on module to AztecOO that:
- Intelligently builds an ERF for a complex valued problem.
- Computes the real-valued solution using AztecOO.
- Returns the complex result to user.
- This is an effective approach in important practical settings.

- (A+*i*B)(x+*i*y) = (b+*i*c)

- A  -B    x      b
-       =
- B  A     y      c

## Komplex Formulation

- Consider a complex-valued matrix *C*:
- With each entry:

**Rewrite as real-valued of twice the dimension:**

## NOX: Nonlinear Solvers

- Suite of nonlinear solution methods:
- Uses abstract vector and “group” interfaces.
- Allows flexible selection and tuning of various strategies:
- Directions.
- Line searches.
- Epetra/AztecOO, LAPACK, PETSc implementations of abstract vector/group interfaces.
- Designed to be easily integrated into existing applications.

## Amesos: Direct Solver Wrappers

- Direct sparse solver use at Sandia:
- Salinas (Structures): DSCPACK (Raghavan), SPOOLES (Ashcraft), others.
- Xyce (Circuits): Kundert, SuperLU serial.
- PCx (LP- new this year): DSCPACK, PSSPD (Sun).
- Numerous other uses.
- Amesos contains wrapper classes to important third party direct sparse solvers:
- Use Epetra objects.
- Provide data redistribution capabilities (e.g., replication).
- Provide common look-and-feel across variety of solvers.
- Provide common resource for direct solvers at Sandia.

## Epetraext: Extensions to Epetra

- Library of useful classes not needed by everyone.
- Most classes are types of “transforms”.
- Examples:
- Graph/matrix view extraction.
- Epetra/Zoltan interface.
- Explicit sparse transpose.
- Singleton removal filter.
- Static condensation filter.
- Overlapped graph constructor.
- Graph colorings.
- Permutations.
- ...
- Most classes are small, useful, but non-trivial to write.

## Some Trilinos Packages

- TSF

- Epetra

- Nonlinear solvers

- Vector, graph, matrix service classes

- Abstract solver API

- Preconditioned Krylov solvers

- Algebraic Preconditioners

- Multi-level Preconditioners

## Trilinos Package Schematic

## New Package: Meros

- Meros: Preconditioner package for incompressible NS problems.
- Addresses problems: *Ax = b.*
- where:
- Makes use of TSF to orchestrate use of:
- ML, Epetra, AztecOO, Ifpack.
- Provides rapidly-developed, scalable implementation of state-of-the-art preconditioner.

## New Packages:  Belos and Anasazi

- Next generation linear solvers (Belos) and eigensolvers (Anasazi) libraries, written in templated C++.
- Provide a generic interface to a collection of algorithms for solving large-scale linear problems and eigenproblems.
- Algorithm implementation is accomplished through the use of abstract base classes.  Interfaces are derived from these base classes to matrix-vector products, status tests, and any arbitrary linear algebra library.
- Includes _block_ linear solvers (GMRES, CG) and eigensolvers (Arnoldi, LOBPCG).

## New Package: Kokkos

- Very new project.
- Goal:
- Isolate key non-BLAS kernels for the purposes of optimization.
- Kernels:
- Dense vector/multivector updates and collective ops (not in BLAS).
- Sparse MV, MM, SV, SM.
- Serial-only for now.
- Reference implementation provided.
- Mechanism for improving performance:
- Default is aggressive compilation of reference source.
- BeBOP: Jim Demmel, Kathy Yelick, Rich Vuduc, UC Berkeley.
- Vector version: Cray.

## SuperLU Package Experiment

- Recently started making SuperLU a Trilinos Package.
- Work done by Ken Stanley with Sherry Li at LBL.
- Once experiment is complete, we will discuss next steps.

## NewPackage Package

- NewPackage provides jump start to develop/integrate a new package.
- NewPackage is a “Hello World” program and website:
- Simple but it does work with autotools.
- Compiles and builds.
- NewPackage directory contains:
- Commonly used directory structure: src, test, doc, example, config.
- Working autotools files.
- Documentation templates (doxygen).
- Working regression test setup.
- Really cuts down on:
- Time to integrate new package.
- Variation in package integration details.
- Development of website.

## Trilinos Package Dependencies

- Based on this chart:
-  AztecOO depends on Epetra, but Epetra is independent of AztecOO
-  NOX can use Epetra, but is independent of Epetra.

| 
Package |  | Depends
On |  |  |  |                                      Can
                                     Use |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Epetra | Epetra Ext | AztecOO | Komplex | Ifpack | Amesos | ML | NOX | TSF Core | TSF Ext | Belos | Meros | Anasazi |
| Epetra |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Epetra Ext |  |  |  |  |  |  |  |  |  |  |  |  |  |
| AztecOO |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Komplex |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Ifpack |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Amesos |  |  |  |  |  |  |  |  |  |  |  |  |  |
| ML |  |  |  |  |  |  |  |  |  |  |  |  |  |
| NOX |  |  |  |  |  |  |  |  |  |  |  |  |  |
| TSFCore |  |  |  |  |  |  |  |  |  |  |  |  |  |
| TSF Ext |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Belos |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Meros |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Anasazi |  |  |  |  |  |  |  |  |  |  |  |  |  |

## SQA/SQE

- Software Quality Assurance/Engineering is important.
- Trilinos facilitates SQA/SQE development/processes for packages:
- 32 of 47 ASCI SQE practices are directly handled by Trilinos (no requirements on packages).
- Trilinos provides significant support for the remaining 15.
- Trilinos Dev Guide Part II: Specific to ASCI requirements.
- Trilinos software engineering policies provide a ready-made infrastructure for new packages.
- Trilinos philosophy: Few *requirements*.  Instead mostly *suggested practices.  *Provides package with option to provide alternate process.

## Trilinos Availability/Support

- Trilinos and related packages are available via LGPL.
- Current release (3.1) is “click release”.  Unlimited availability.
- Next release scheduled for April 2004.
- New platform facilitates development and support:
- http://software.sandia.gov
- Location of cvs repository, bugzilla, bonzai and mailman servers.
- Accessible from anywhere via ssh/scp.
- Documentation (generated via doxygen) is all available online.

## Mailman Mail Lists

- Each Trilinos package, including Trilinos itself, has four mail lists:
- package-checkins@software.sandia.gov
- CVS commit emails.
- package-developers@software.sandia.gov
- Mailing list for developers.
- package-users@software.sandia.gov
- Issues for package users.
- package-announce@software.sandia.gov
- Releases and other announcements specific to the package.
- Additional list: Trilinos-Leaders@software.sandia.gov
- http://software.sandia.gov/mailman/listinfo/

## Conclusions

- Trilinos provides a variety of services to developers and users:
- Common software infrastructure for packages.
- Common SQA policies and processes.
- Simplifies installation, support for users of total collection.
- Epetra & TSF promote common APIs across all other Trilinos packages.
- Each package can be built, used independently, and exists as independent project.
- http://software.sandia.gov
- http://software.sandia.gov/trilinos
- Additional documentation at my website:http://www.cs.sandia.gov/~mheroux.