# Presentation: 458970

## Trilinos Overview and Future PlansMichael A. HerouxSandia National Laboratories

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

**Notes:**
Questions from audience

Can you use another matrix-vector multiply?
Are multivectors supported?
Are Petsc and Hypre available via Thyra?
How does Trilinos differ from Petsc?

## Trilinos Development Team

**Chris Baker**
- Developer of Anasazi
**Ross Bartlett **
- Lead Developer of Thyra
- Developer of Rythmos
**Paul Boggs **
- Developer of Thyra
**Todd Coffey**
- Lead Developer of Rythmos
**Jason Cross **
- Developer of Jpetra
**David Day**
- Developer of Komplex
**Clark Dohrmann**
- Developer of CLAPS
**Michael Gee **
- Developer of ML, NOX
**Bob Heaphy **
- Lead developer of Trilinos SQA
**Mike Heroux **
- Trilinos Project Leader
- Lead Developer of Epetra, AztecOO,
- Kokkos, Komplex, IFPACK, Thyra, Tpetra
- Developer of Amesos, Belos, EpetraExt, Jpetra
**Ulrich Hetmaniuk**
- Developer of Anasazi

**Robert Hoekstra **
- Lead Developer of EpetraExt
- Developer of Epetra, Thyra, Tpetra
**Russell Hooper**
- Developer of NOX
**Vicki Howle **
- Lead Developer of Meros
- Developer of Belos and Thyra
**Jonathan Hu **
- Developer of ML
**Sarah Knepper**
- Developer of Komplex
**Tammy Kolda **
- Lead Developer of NOX** **
**Joe Kotulski**
- Lead Developer of Pliris
**Rich Lehoucq **
- Developer of Anasazi and Belos
**Kevin Long **
- Lead Developer of Thyra,
- Developer of Belos and Teuchos
**Roger Pawlowski **
- Lead Developer of NOX
**Michael Phenow **
- Trilinos Webmaster** **
- Lead Developer of New_Package

**Eric Phipps **
- Developer of LOCA and NOX
**Marzio Sala**
- Lead Developer of Didasko and IFPACK
- Developer of ML, Amesos
**Andrew Salinger **
- Lead Developer of LOCA
**Paul Sexton **
- Developer of Epetra and Tpetra
**Bill Spotz**
- Lead Developer of PyTrilinos
- Developer of Epetra, New_Package
**Ken Stanley **
- Lead Developer of Amesos and New_Package
**Heidi Thornquist **
- Lead Developer of Anasazi, Belos and Teuchos
**Ray Tuminaro **
- Lead Developer of ML  and Meros
**Jim Willenbring **
- Developer of Epetra and New_Package.
- Trilinos library manager** **
**Alan Williams **
- Developer of Epetra, EpetraExt, AztecOO, Tpetra

## Trilinos Presentation Forums

- ACTS “Hands-on” Tutorial: Aug 22-24, 2006.
- At Lawrence Berkeley Lab.
- Next Trilinos User Group Meeting: Nov 7-9, 2006.
- Probably in new CSRI building.  Other location?

## Motivation For Trilinos

- Sandia does LOTS of solver work.
- When I started at Sandia in May 1998:
- Aztec was a mature package.  Used in many codes.
- FETI, PETSc, DSCPack, Spooles, ARPACK, DASPK, and many other codes were (and are) in use.
- New projects were underway or planned in multi-level preconditioners, eigensolvers, non-linear solvers, etc...
- The challenges:
- Little or no coordination was in place to:
- Efficiently reuse existing solver technology.
- Leverage new development across various projects.
- Support solver software processes.
- Provide consistent solver APIs for applications.
- ASCI was forming software quality assurance/engineering (SQA/SQE) requirements:
- Daunting requirements for any single solver effort to address alone.

## Evolving Trilinos Solution

**Evolving Trilinos Solution**

- Trilinos1 is an evolving framework to address these challenges:
- Fundamental atomic unit is a *package*.
- Includes core set of vector, graph and matrix classes (Epetra/Tpetra packages).
- Provides a common abstract solver API (Thyra package).
- Provides a ready-made package infrastructure (new_package package):
- Source code management (cvs, bonsai).
- Build tools (autotools).
- Automated regression testing (queue directories within repository).
- Communication tools (mailman mail lists).
- Specifies requirements and suggested practices for package SQA.
- In general allows us to categorize efforts:
- Efforts best done at the Trilinos level (useful to most or all packages).
- Efforts best done at a package level (peculiar or important to a package).
**Allows package developers to focus only on things that are unique to ****their package.**

- 1. Trilinos loose translation: “A string of pearls”

## Slide 6

## Trilinos Strategic Goals

**Scalable Solvers:** As problem size and processor counts increase, the cost of the solver will remain a nearly fixed percentage of the total solution time.
**Hardened Solvers:** Never fail unless problem essentially unsolvable, in which case we diagnose and inform the user why the problem fails and provide a reliable measure of error.
**Full Vertical Coverage:** Provide leading edge capabilities from basic linear algebra to transient and optimization solvers.
**Universal Interoperability:** All Trilinos packages will be interoperable, so that any combination of solver packages that makes sense algorithmically will be possible within Trilinos.
**Universal Solver RAS:** Trilinos will be:
- Integrated into every major application at Sandia (**Availability**).
- The leading edge hardened, efficient, scalable solutions for each of these applications (**Reliability**).
- Easy to maintain and upgrade within the application environment (**Serviceability**).

- Algorithmic
- Goals

- Software
- Goals

**Grand**

- ^

## Trilinos Package Concepts

***Package: The Atomic Unit***

## Trilinos Packages

- Trilinos is a collection of *Packages*.
- Each package is:
- Focused on important, state-of-the-art algorithms in its problem regime.
- Developed by a small team of domain experts.
- Self-contained: No explicit dependencies on any other software packages (with some special exceptions).
- Configurable/buildable/documented on its own.
- Sample packages: NOX, AztecOO, ML, IFPACK, Meros.
- Special package collections:
- Petra (Epetra, Tpetra, Jpetra): Concrete Data Objects
- Thyra: Abstract Conceptual Interfaces
- Teuchos: Common Tools.
- New_package: Jumpstart prototype.

## Trilinos Package Summary

| Objective | Package(s) |
| --- | --- |
| Distributed linear algebra objects | Epetra, Jpetra, Tpetra |
| Krylov solvers | AztecOO, Belos |
| ILU-type preconditioners | AztecOO, IFPACK |
| Multilevel preconditioners | ML, CLAPS |
| Eigenvalue problems | Anasazi |
| Block preconditioners | Meros |
| Direct sparse linear solvers | Amesos |
| Direct dense solvers | Epetra, Teuchos, Pliris |
| Abstract interfaces | Thyra |
| Nonlinear system solvers | NOX, LOCA |
| Complex linear systems | Komplex |
| C++ utilities, (some) I/O | Teuchos, EpetraExt, Kokkos, Galeri |
| Trilinos Tutorial | Didasko |
| Python Support | PyTrilinos |
| Time integrators/DAEs | Rythmos |
| Archetype package | NewPackage |

## Why Packages?

***Let’s Do Some Matrix Analysis!***

## Trilinos Development Team (Again)

**Ross Bartlett **
- Lead Developer of Thyra
- Developer of Rythmos
**Paul Boggs **
- Developer of Thyra
**Todd Coffey**
- Lead Developer of Rythmos
**Jason Cross **
- Developer of Jpetra
**David Day**
- Developer of Komplex
**Clark Dohrmann**
- Developer of CLAPS
**Michael Gee **
- Developer of ML, NOX
**Bob Heaphy **
- Lead developer of Trilinos SQA
**Mike Heroux **
- Trilinos Project Leader
- Lead Developer of Epetra, AztecOO,
- Kokkos, Komplex, IFPACK, Thyra, Tpetra
- Developer of Amesos, Belos, EpetraExt, Jpetra
**Ulrich Hetmaniuk**
- Developer of Anasazi

**Robert Hoekstra **
- Lead Developer of EpetraExt
- Developer of Epetra, Thyra, Tpetra
**Russell Hooper**
- Developer of NOX
**Vicki Howle **
- Lead Developer of Meros
- Developer of Belos and Thyra
**Jonathan Hu **
- Developer of ML
**Sarah Knepper**
- Developer of Komplex
**Tammy Kolda **
- Lead Developer of NOX** **
**Joe Kotulski**
- Lead Developer of Pliris
**Rich Lehoucq **
- Developer of Anasazi and Belos
**Kevin Long **
- Lead Developer of Thyra,
- Developer of Belos and Teuchos
**Roger Pawlowski **
- Lead Developer of NOX
**Michael Phenow **
- Trilinos Webmaster** **
- Lead Developer of New_Package

**Eric Phipps **
- Developer of LOCA and NOX
**Marzio Sala**
- Lead Developer of Didasko and IFPACK
- Developer of ML, Amesos
**Andrew Salinger **
- Lead Developer of LOCA
**Paul Sexton **
- Developer of Epetra and Tpetra
**Bill Spotz**
- Lead Developer of PyTrilinos
- Developer of Epetra, New_Package
**Ken Stanley **
- Lead Developer of Amesos and New_Package
**Heidi Thornquist **
- Lead Developer of Anasazi, Belos and Teuchos
**Ray Tuminaro **
- Lead Developer of ML  and Meros
**Jim Willenbring **
- Developer of Epetra and New_Package.
- Trilinos library manager** **
**Alan Williams **
- Developer of Epetra, EpetraExt, AztecOO, Tpetra

## Developer and Package Node IDs

_**Developer IDs**_
- RossBartlett = 1;
- PaulBoggs = 2;
- ToddCoffey = 3;
- JasonCross = 4;
- DavidDay = 5;
- ClarkDohrmann= 6;
- MichaelGee = 7;
- BobHeaphy = 8;
- MikeHeroux = 9;
- UlrichHetmaniuk = 10;
- RobHoekstra = 11;
- RussellHooper = 12;
- VickiHowle = 13;
- JonathanHu = 14;
- SarahKnepper = 15;
- TammyKolda = 16;
- JoeKotulski = 17;
- RichLehoucq = 18;
- KevinLong = 19;
- RogerPawlowski = 20;
- MichaelPhenow = 21;
- EricPhipps = 22;
- MarzioSala = 23;
- AndrewSalinger = 24;
- PaulSexton = 25;
- BillSpotz = 26;
- KenStanley = 27;
- HeidiThornquist = 28;
- RayTuminaro = 29;
- JimWillenbring = 30;
- AlanWilliams  = 31;

_**Package IDs**_
- PyTrilinos = 1;
- amesos = 2;
- anasazi = 3;
- aztecoo = 4;
- belos = 5;
- claps = 6;
- didasko = 7;
- epetra = 8;
- epetraext = 9;
- ifpack = 10;
- jpetra = 11;
- kokkos = 12;
- komplex = 13;
- meros = 14;
- loca = 15;
- ml = 16;
- new_package = 17;
- nox = 18;
- pliris = 19;
- rythmos = 20;
- teuchos = 21;
- thyra = 22;
- tpetra = 23;
- trilinosframework = 24;

## Developer-Package EdgesA(i,j) = 1 if developer i contributes to package j

- A = sparse(31,24);
- A(RossBartlett,rythmos) = 1;
- A(RossBartlett,thyra) = 1;
- A(PaulBoggs,thyra) = 1;
- A(ToddCoffey,rythmos) = 1;
- A(JasonCross,jpetra) = 1;
- A(DavidDay,komplex) = 1;
- A(ClarkDohrmann,claps) = 1;
- A(MichaelGee,ml) = 1;
- A(MichaelGee,nox) = 1;
- A(BobHeaphy,trilinosframework) = 1;
- A(MikeHeroux,trilinosframework) = 1;
- A(MikeHeroux,epetra) = 1;
- A(MikeHeroux,aztecoo) = 1;
- A(MikeHeroux,kokkos) = 1;
- A(MikeHeroux,komplex) = 1;
- A(MikeHeroux,ifpack) = 1;
- A(MikeHeroux,thyra) = 1;
- A(MikeHeroux,tpetra) = 1;
- A(MikeHeroux,amesos) = 1;
- A(MikeHeroux,belos) = 1;
- A(MikeHeroux,epetraext) = 1;
- A(MikeHeroux,jpetra) = 1;
- A(UlrichHetmaniuk,anasazi) = 1;

- A(MarzioSala,didasko) = 1;
- A(MarzioSala,ifpack) = 1;
- A(MarzioSala,ml) = 1;
- A(MarzioSala,amesos) = 1;
- A(AndrewSalinger,loca) = 1;
- A(PaulSexton,epetra) = 1;
- A(PaulSexton,tpetra) = 1;
- A(BillSpotz,PyTrilinos) = 1;
- A(BillSpotz,epetra) = 1;
- A(BillSpotz,new_package) = 1;
- A(KenStanley,amesos) = 1;
- A(KenStanley,new_package) = 1;
- A(HeidiThornquist,anasazi) = 1;
- A(HeidiThornquist,belos) = 1;
- A(HeidiThornquist,teuchos) = 1;
- A(RayTuminaro,ml) = 1;
- A(RayTuminaro,meros) = 1;
- A(JimWillenbring,epetra) = 1;
- A(JimWillenbring,new_package) = 1;
- A(JimWillenbring,trilinosframework) = 1;
- A(AlanWilliams,epetra) = 1;
- A(AlanWilliams,epetraext) = 1;
- A(AlanWilliams,aztecoo) = 1;
- A(AlanWilliams,tpetra) = 1;

- A(RobHoekstra,epetra) = 1;
- A(RobHoekstra,thyra) = 1;
- A(RobHoekstra,tpetra) = 1;
- A(RobHoekstra,epetraext) = 1;
- A(RussellHooper,nox) = 1;
- A(VickiHowle,meros) = 1;
- A(VickiHowle,belos) = 1;
- A(VickiHowle,thyra) = 1;
- A(JonathanHu,ml) = 1;
- A(SarahKnepper,komplex) = 1;
- A(TammyKolda,nox) = 1;
- A(TammyKolda,trilinosframework) = 1;
- A(JoeKotulski,pliris) = 1;
- A(RichLehoucq,anasazi) = 1;
- A(RichLehoucq,belos) = 1;
- A(KevinLong,thyra) = 1;
- A(KevinLong,belos) = 1;
- A(KevinLong,teuchos) = 1;
- A(RogerPawlowski,nox) = 1;
- A(MichaelPhenow,trilinosframework) = 1;
- A(MichaelPhenow,trilinosframework) = 1;
- A(EricPhipps,loca) = 1;
- A(EricPhipps,nox) = 1;

## Developer-Package Matrix

**Developer-Package Matrix**

- Number of developers per package:
- Maximum: max(sum(A)) = 6
- Average:	sum(sum(A))/24 = 2.875
- Number of package affiliations per developer:
- Maximum: max(sum(A’)) = 12 (me).
- Minus outlier:	             = 4.
- Average:	  sum(sum(A’))/31 = 2.26
- Observations:
- Small package teams.
- Multiple packages per developer.

- spy(A);

## Developer-Developer Matrix (A*A’)

**Developer-Developer Matrix (A*A’)**

- B = A*A’
- Apply Symmetric AMD reordering.
- Two developers connected if co-authors of a package.
- Only two developers “disconnected”:
- Clark Dohrmann: CLAPS.
- Joe Kotulski: Pliris.

- Komplex Subteam

- ML Subteam

- Anasazi Subteam

- Epetra Subteam

- NOX Subteam

- Thyra Subteam

## Package-Package Matrix (A’*A)(Not sure what to say about this)

- B = A’*A, Apply symamd.
- Two packages connected if they share a developer.
- Only two packages “disconnected”:
- Clark Dohrmann: CLAPS.
- Joe Kotulski: Pliris.

- Me

- Epetra, Amesos, Didasko, Ifpack

- Without Me

- Meros, AztecOO, Tpetra, EpetraExt

## Why Packages?

***Partial Answer: Small Inter-related teams.***

## Package Interoperability

## Interoperability vs. Dependence    (“Can Use”)                   (“Depends On”)

- Although most Trilinos packages have no explicit dependence, each package must interact with *some* other packages:
- NOX needs operator, vector and solver objects.
- AztecOO needs preconditioner, matrix, operator and vector objects.
- Interoperability is enabled at configure time.  For example, NOX:
- --enable-nox-lapack     compile NOX lapack interface libraries
- --enable-nox-epetra     compile NOX epetra interface libraries
- --enable-nox-petsc      compile NOX petsc interface libraries
- Trilinos configure script is vehicle for:
- Establishing interoperability of Trilinos components...
- Without compromising individual package autonomy.
- Trilinos offers seven basic interoperability mechanisms.

- ../configure –enable-python

## Trilinos Interoperability Mechanisms(Acquired as Package Matures)

| Package builds under Trilinos configure scripts. |  | Package can be built as part of a suite of packages; cross-package interfaces enable/disable automatically |
| --- | --- | --- |
| Package accepts user data as Epetra or Thyra objects |  | Applications using Epetra/Thyra can use package |
| Package accepts parameters from Teuchos ParameterLists |  | Applications using Teuchos ParameterLists can drive package |
| Package can be used via Thyra abstract solver classes |  | Applications or other packages using Thyra can use package |
| Package can use Epetra for private data. |  | Package  can then use other packages that understand Epetra |
| Package accesses solver services via Thyra interfaces |  | Package  can then use other packages that implement Thyra interfaces |
| Package available via PyTrilinos |  | Package  can be used with other Trilinos packages via Python. |

## “Can Use”  vs. “Depends On”

- “Can Use”
- Interoperable without dependence.
- Dense is Good.
- Encouraged.

- “Depends On”
- OK, if essential.
- Epetra: 9 clients.
- Thyra, Teuchos, NOX: 2 clients.
- Discouraged.

##   Interoperability Example: ML

- ML: Multi-level Preconditioner Package.
- Primary Developers: Ray Tuminaro, Jonathan Hu, Marzio Sala.
- No *explicit, essential* dependence on other Trilinos packages.
- Uses abstract interfaces to matrix/operator objects.
- Has independent configure/build process (but can be invoked at Trilinos level).
*Interoperable* with other Trilinos packages and other libraries:
- Accepts user data as Epetra matrices/vectors.
- Can use Epetra for internal matrices/vectors.
- Can be used via Thyra abstract interfaces.
- Can be built via Trilinos configure/build process.
- Available as preconditioner to all other Trilinos packages.
- Can use IFPACK, Amesos, AztecOO objects as smoothers, coarse solvers.
- Can be driven via Teuchos ParameterLists.
- Available to PETSc users ***without dependence on any other Trilinos packages.***

## Package Maturation Process

***Asynchronicity***

## Day 1 of Package Life

**CVS:** Each package is self-contained in Trilinos/package/ directory.
**Bugzilla:** Each package has its own Bugzilla product.
**Bonsai:** Each package is browsable via Bonsai interface.
**Mailman:** Each Trilinos package, including Trilinos itself, has four mail lists:
- package-checkins@software.sandia.gov
- CVS commit emails. “Finger on the pulse” list.
- package-developers@software.sandia.gov
- Mailing list for developers.
- package-users@software.sandia.gov
- Issues for package users.
- package-announce@software.sandia.gov
- Releases and other announcements specific to the package.
**New_package** (optional): Customizable boilerplate for
- Autoconf/Automake/Doxygen/Python/Thyra/Epetra/TestHarness/Website

## Sample Package Maturation Process

| Step | Example |
| --- | --- |
| Package added to CVS: Import existing code or start with new_package. | ML CVS repository migrated into Trilinos (July 2002). |
| Mail lists, Bugzilla Product, Bonsai database created. | ml-announce, ml-users, ml-developers, ml-checkins, ml-regression @software.sandia.gov created, linked to CVS (July 2002). |
| Package builds with configure/make, Trilinos-compatible | ML adopts Autoconf, Automake starting from new_package (June 2003). |
| Epetra objects recognized by package. | ML accepts user data as Epetra matrices and vectors (October 2002). |
| Package accessible via Thyra interfaces. | ML adaptors written for TSFCore_LinOp (Thyra) interface (May 2003). |
| Package uses Epetra for internal data. | ML able to generate Epetra matrices.  Allows use of AztecOO, Amesos, Ifpack, etc. as smoothers and coarse grid solvers (Feb-June 2004). |
| Package parameters settable via Teuchos ParameterList | ML gets manager class, driven via ParameterLists (June 2004). |
| Package usable from Python  (PyTrilinos) | ML Python wrappers written using new_package template (April 2005). |

- Startup Steps

- Maturation Steps

## Maturation Jumpstart:  NewPackage

**Maturation Jumpstart:  NewPackage**

- NewPackage provides jump start to develop/integrate a new package
- NewPackage is a “Hello World” program and website:
- Simple but it does work with autotools.
- Compiles and builds.
- NewPackage directory contains:
- Commonly used directory structure: src, test, doc, example, config.
- Working Autoconf/Automake files.
- Documentation templates (doxygen).
- Working regression test setup.
- Working Python and Thyra adaptors.
- Substantially cuts down on:
- Time to integrate new package.
- Variation in package integration details.
- Development of website.

- NOTE: NewPackage can be use independent from Trilinos

## What Trilinos is not

- Trilinos is not a single monolithic piece of software. Each package:
- Can be built independent of Trilinos.
- Has its own self-contained CVS structure.
- Has its own Bugzilla product and mail lists.
- Development team is free to make its own decisions about algorithms, coding style, release contents, testing process, etc.
- Trilinos top layer is not a large amount of source code:
- Trilinos repository contains 452,187 source lines of code (SLOC).
- Sum of the packages SLOC counts :	445,937.
- Trilinos top layer SLOC count:	    	    6, 250 (1.4%).
- Trilinos is not “indivisible”:
- You don’t need all of Trilinos to get things done.
- Any collection of packages can be combined and distributed.
- Current public release contains only 18 of the nearly 30 Trilinos packages.

## Insight from HistoryA Philosophy for Future Directions

- In the early 1800’s U.S. had many new territories.
- Question: How to incorporate into U.S.?
- Colonies? No.
- Expand boundaries of existing states? No.
- Create process for self-governing regions. Yes.
- Theme: Local control drawing on national resources.
- Trilinos package architecture has some similarities:
- Asynchronous maturation.
- Packages decide degree of interoperations, use of Trilinos facilities.
- Strength of each: Scalable growth with local control.

## Solver Collaboration: The Big Picture

## Solver Software Components and Interfaces

**Solver Software Components and Interfaces**

**2) LAL : Linear Algebra Library (e.g. vectors, sparse matrices, sparse factorizations, preconditioners)**

**ANA**

**APP**

***ANA/APP Interface***

***ANA Vector Interface***

**1) ****ANA : Abstract Numerical Algorithm (e.g. linear solvers, eigen solvers, nonlinear solvers, stability analysis, uncertainty quantification, transient solvers, optimization etc.)**

**3) APP : Application (the model: physics, discretization method etc.)**

**Example Trilinos Packages:**
-  Belos (linear solvers)
-  Anasazi (eigen solvers)
-  NOX (nonlinear equations)
-  Rhythmos (ODEs,DAEs)
-  MOOCHO (Optimization)
-  ...

**Example Trilinos Packages:**
-  Epetra/Tpetra (Mat,Vec)
-  Ifpack, AztecOO, ML (Preconditioners)
-  Meros (Preconditioners)
-  Pliris (Interface to direct solvers)
-  Amesos (Direct solvers)
-  Komplex (Complex/Real forms)
-  ...

_**Types of Software Components**_

_**Thyra**_
- ANA Interfaces to Linear Algebra

_**FEI/Thyra**_
- APP to LAL Interfaces

_**Custom/Thyra**_
- LAL to LAL Interfaces

**Thyra::Nonlin**

**Examples:**
-  SIERRA
-  NEVADA
-  Xyce
-  Sundance
-   ...

**LAL**

**Matrix**

**Preconditioner**

**Vector**

**Notes:**
There are efforts going on at Sandia to consolidate numerical software in several different areas of scientific computing.

Let’s discuss where ANAs fit in with other types of scientific computing sofware

The basic idea is to componentize the numerical scientific software along the lines shown here.

1) ANA
2) APP
3) LAL

So why is componentization desirable?  It’s all about the principle of division of labor.  It does not make sense for experts in a arrow application (such as fluid machanics) to also become experts in parallel numerical linear algebra or nonlinear solvers or especially not optimization.  Implementing advanced parallel linear algebra software and sophisticated abstract numerical algorithms is not easy.

## LAL Foundation: Petra

**LAL Foundation: Petra**

- Petra provides a “common language” for distributed linear algebra objects (operator, matrix, vector)
- Petra provides distributed matrix and vector services.
- Has 3 implementations under development.

## Petra Object Model

- Perform redistribution of distributed objects:
-  Parallel permutations.
-  “Ghosting” of values for local computations.
-  Collection of partial results from remote processors.

**Petra Object Model**

- Abstract Interface to Parallel Machine
-  Shameless mimic of MPI interface.
-  Keeps MPI dependence to a single class (through all of Trilinos!).
-  Allow trivial serial implementation.
-  Opens door to novel parallel libraries (shmem, UPC, etc...)

- Abstract Interface for Sparse All-to-All Communication
-  Supports construction of pre-recorded “plan” for data-driven communications.
-  Examples:
-  Supports gathering/scatter of off-processor x/y values when computing y = Ax.
-  Gathering overlap rows for Overlapping Schwarz.
-  Redistribution of matrices, vectors, etc...

- Describes layout of distributed objects:
-  Vectors: Number of vector entries on each processor and global ID
-  Matrices/graphs: Rows/Columns managed by a processor.
-  Called “Maps” in Epetra.

- Dense Distributed Vector and Matrices:
-  Simple local data structure.
-  BLAS-able, LAPACK-able.
-  Ghostable, restributable.
-  RTOp-able.

- Base Class for All Distributed Objects:
-  Performs all communication.
-  Requires Check, Pack, Unpack methods from derived class.

- Graph class for structure-only computations:
-  Reusable matrix structure.
-  Pattern-based preconditioners.
-  Pattern-based load balancing tools.

- Basic sparse matrix class:
-  Flexible construction process.
-  Arbitrary entry placement on parallel machine.

## Petra Implementations

- Three version under development:
**Epetra** (Essential Petra):
- Current production version.
- Restricted to real, double precision arithmetic.
- Uses stable core subset of C++ (circa 2000).
- Interfaces accessible to C and Fortran users.
**Tpetra** (Templated Petra):
- Next generation C++ version.
- Templated scalar and ordinal fields.
- Uses namespaces, and STL: Improved usability/efficiency.
**Jpetra** (Java Petra):
- Pure Java. Portable to any JVM.
- Interfaces to Java versions of MPI, LAPACK and BLAS via interfaces.

## Data Model Wrap-Up

- Our target apps require flexible data model.
- Petra Object Model (POM) supports:
- Arbitrary placement of vector, graph, matrix entries on parallel machine.
- Arbitrary redistribution, ghosting and collection of distributed data.
- This flexibility is needed by LALs:
- Algebraic and multi-level preconditioners.
- Concrete distributed matrix kernels.
- Direct methods.
- POM is complex: Non-LALs (ANAs) do not rely on it.

## Scalable Preconditioners

## Preconditioner Efforts

- ML:
- Many applications can use if matrix is SPD, or close.
- Continuing efforts in custom operators.
- Nonlinear FAS: See Michael Gee’s talk.
- CLAPS: See Clark’s talk.
- Segregated preconditioners (Meros): See Vicki’s talk.
- Mortar methods: New work.
- IFPACK: DD type algebraic preconditioners.
- Numerous other efforts focused on specific applications.
- Theme: Block preconditioners.
- Bottom line: Scalable, robust preconditioners are essential.

## Abstract Numerical Algorithms (ANAs)

## Categories of Abstract Problems and Abstract Algorithms

- Linear Problems:
- Linear equations:
- Eigen problems:

- Nonlinear Problems:
- Nonlinear equations:
- Stability analysis:

- Transient Nonlinear Problems:
- DAEs/ODEs:

- Optimization Problems:
- Unconstrained:
- Constrained:

- _Trilinos Packages_

- Belos

- Anasazi

- NOX

- LOCA

- Rythmos

- MOOCHO

**Notes:**
Many different kinds of numerical problems and algorithms for solving those problems are used in scientific computing

Just some of the categories of problems and algorithms are:

## Introducing Abstract Numerical Algorithms

**Introducing Abstract Numerical Algorithms**

-   An ANA is a numerical algorithm that can be expressed abstractly solely in terms of vectors, vector spaces, and linear operators (i.e. not with direct element access)

_**Example Linear ANA (LANA) : Linear Conjugate Gradients**_

_**Types of operations**_

_**Types of objects**_

- What is an abstract numerical algorithm (ANA)?

_**Linear Conjugate Gradient Algorithm**_

**Notes:**
So we looked at some different categories of numerical algorithms and some of the different computing environments these algorithms are run in but what is an abstract numerical algorithm?

## Examples of Nonlinear Abstract Numerical Algorithms

**Examples of Nonlinear Abstract Numerical Algorithms**

**Class of Numerical Problem**

**Example ANA**

- Nonlinear equations

- Newton’s method (e.g. NOX, MOOCHO)

- Initial Value DAE/ODE

- Backward Euler method (e.g. Rythmos?)

- Linear equations

- GMRES (e.g. Belos)

**Requires**

**Requires**

**Notes:**
Now lets look at a couple of examples of nonlinear ANAs

  ANAs build on top of one another
  One should be able to easily swap in and out linear solvers and nonlinear solvers
  Abstract interfaces between different categories of ANAs is in its infancy

## ANA Efforts

- Thyra:	Abstraction layer.
- Belos:	Generic Linear Iterative solvers.
- Anasazi:	Eigensolvers.
- Rythmos:	Transient.
- MOOCHO:	Optimization.
- Meros:	Segregated preconditioners.
- NOX/LOCA:	Nonlinear solvers.
- IFPACK:	Algebraic Preconditioners (Somewhat).
- Bottom line: ANAs are a critical enabler for next generation solver capabilities.

## Software Quality

## SQA/SQE

- Software Quality Assurance/Engineering is important.
- No longer sufficient to say “We do a good job”.
- Trilinos facilitates SQA/SQE development/processes for packages:
- 32 of 47 ASCI SQE practices are directly handled by Trilinos (no requirements on packages).
- Trilinos provides significant support for the remaining 15.
- Trilinos Dev Guide Part II: Specific to ASCI requirements.
- Trilinos software engineering policies provide a ready-made infrastructure for new packages.
- Trilinos philosophy: Few *requirements*.  Instead mostly *suggested practices.  *Provides package with option to provide alternate process.

## Slide 45

| Trilinos Service | SQE Practices Impact |
| --- | --- |
| Yearly Trilinos User Group Meeting (TUG) and Developer Forum:
Once a year gathering for tutorials, package feature updates, user/developer requirements discussion and developer  training. | — All Requirements steps: gathering,    derivation, documentation, feasibility,etc.— User  and Developer training. |
| Monthly Trilinos leaders meetings:
Trilinos leaders, including package development leaders, key managers, funding sources and other stakeholders participate in monthly phone meetings to discuss any timely issues related to the Trilinos Project. | —Developer Training.
—Design reviews.
—Policy decisions across all development    phases. |
| Trilinos and package mail lists:
Trilinos lists for leaders, announcements, developers, users, checkins and similar lists at the package level support a variety of communication. All lists are archived, providing critical artifacts for assessments and audits. | —Developer/user/client communication.
—Requirements/design/testing artifacts.
—Announcement/documenting of releases.
 |
| Trilinos and Trilinos3PL source repositories: 
All source code, development and user documentation is retained and tracked. In addition, reference versions of all external software, including BLAS, LAPACK, Umfpack, etc. are retained in Trilinos3PL. | —Source management.
—Versioning.
—Third-party software management. |
| Bugzilla Products: 
Each package has its own Bugzilla Product with standard components. | —Requirements/faults capturing and     tracking. |
| Trilinos configure script and M4 macros:
 The Trilinos configure script and related macros support portable installation of Trilinos and its packages | —Portability.
—Software release.
 |
| Trilinos test harness: 
Trilinos provides a base testing plan and automated testing across multiple platforms, plus creation of testing artifacts. Test harness results are used to derive a variety of metrics for SQE. | —Pre-checkin and regression testing.
—Software metrics.
 |

## Last Year’s Future Plans

- Trilinos Package Architecture:
- Continue refinement of *new_package.  **	***Status**: Good progress.
- Explicitly define Trilinos compatibility.  	**Status**:  Still trying to define this.
- Resolve the abstract interface issue.	**Status**:  Very good progress.
- Software Quality:
- Expand use and ease-of-use of test harness.	**Status**: Very good progress.
- Identify metrics and automate capture and display. 	**Status**: Some progress.
- Establish a life-cycle model (hybrid agile/unified process?). **Status**: Minimal progress.
- Customize the ASC SQP to our environment.	**Status**: Suspended.
- Packages:
- Foster new package development.		**Status**:  Package count still growing
- Manage the growth.  Issue: complexity of package coupling. **Status**: Very good progress.
- Harden our mature packages.		**Status**: Some progress, much more needed.
- Transition to post-delivery maintenance:	**Status**: Regress.
- Organizational issue: Tough to solve.

## Major Themes for FY06/07Framework

- Take Steps toward dynamic package addition.
- Needed to enhance external interoperability.
- Trilinos-compatibility definition or something similar.
- Define SW Lifecycle(s) and begin formalized efforts.
- Need something for external audits that is still useful to us.
- Agile vs. UP vs. hybrid.
- Make Trilinos more accessible to new users.
- We have lots of introductory material.
- Need to organize it for new users.

## Major Themes for FY06/07Algorithms and Development

- Multi-level preconditioners.
- Curl-curl.
- Mortar methods.
- DD algebraic preconditioners.
- Xyce-focused.
- Parallel KLU.
- Complex linear solvers.
- Optimization.
- Transient solvers.
- Eigensolvers.
- Maturation of Thyra and Python adapters.
- Matrix loading interface in Thyra.
- Public release of Belos, CLAPS, Galeri, Komplex, Meros, Rythmos, Tpetra, ...

## Tools on software.sandia.gov

- CVS: 		Source management.
- Bugzilla: 		Bug/feature tracking.
- Mailman: 		Mail list management.
- Bonsai: 		Interface to CVS/Bugzilla.
- Webserver: 		Webpages accessible from anywhere.
- Autotools: 		Autoconf/automake facilities.
- Wiki: 		Online logging tool.

## Trilinos Availability/Information

- Trilinos and related packages are available via LGPL.
- Current release (6.0) is “click release”.  Unlimited availability.
- Trilinos Release 7: September 2006.
- Trilinos 6.1 ?? If needed.
- Trilinos Awards:
-  2004 R&D 100 Award.
- SC2004 HPC Software Challenge Award.
- Sandia Team Employee Recognition Award.
- Lockheed-Martin Nova Award Nominee.
- More information:
- http://software.sandia.gov
- http://software.sandia.gov/trilinos
- Additional documentation at my website:http://www.cs.sandia.gov/~mheroux.

## Conclusions

- Trilinos services to developers and users:
- Infrastructure, Interfaces, Implementations.
- Simplifies installation, support for users of total collection.
- Epetra, Teuchos & Thyra promote common APIs across all other Trilinos packages.
- Each package can be built, used independently (with only explicit dependencies), and exists as independent project.
**Primary goals: **
**Rapid development and installation of robust numerical solvers.**
**High-quality production software for the critical path.**
- More information:
- http://software.sandia.gov
- http://software.sandia.gov/trilinos
- Additional documentation at my website:http://www.cs.sandia.gov/~mheroux.