# Presentation: 181192

## Replacement

**A Case Study on the Vertical Integration of Trilinos Solver Algorithms with a Production Application Code**
**Organizer:** **Roscoe A. Bartlett**
*Sandia National Laboratories*
**10:00-10:25 Overview of the Vertical Integration of Trilinos Solver Algorithms in a Production Application Code**
*Roscoe A. Bartlett*, Sandia National Laboratories
**10:30-10:55 Analytic Sensitivities in Large-scale Production Applications via Automatic Differentiation with Sacado**
*Eric Phipps*, Sandia National Laboratories
**11:00-11:25 To PDE Components and Beyond**
*Andy Salinger*, Sandia National Laboratories
**11:30-11:55 Analysis Tools for Large-scale Simulation with Application to the Stationary Magnetohydrodynamics Equations**
*Roger Pawlowski*, Eric Phipps, Heidi K. Thornquist, and Roscoe A. Bartlett, Sandia National Laboratories

## Overview of the Vertical Integration of Trilinos Solver Algorithms in a Production Application Code 

**Roscoe A. Bartlett **
**Department of Optimization & Uncertainty Estimation**
_**[http://](http://www.cs.sandia.gov/~rabartl)**__**[www.cs.sandia.gov](http://www.cs.sandia.gov/~rabartl)**__**[/~rabartl](http://www.cs.sandia.gov/~rabartl)**_
**Sandia National Laboratories**
**March 13****th****, 2008**

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

## Overview of Trilinos Vertical Integration Project (Milestone)

**Goal:** Vertically integrate Trilinos solver algorithms in Trilinos to build new predictive embedded analysis capabilities
**Impact:** Vertically integrated 10+ Trilinos algorithm packages
**Goal:** Demonstrate on relevant production applications
**Impact:** Solved steady-state parameter estimation problems and transient sensitivities on semiconductor devices in Charon
**Impact:**  Solved Eigen problems on MHD problem in Charon
**Added Goal: **Explore refined models of collaboration between production application developers and algorithm researchers.
** ****Impact: **Closer collaboration between application and algorithm developers yielding better algo and app R&D

- Bartlett, Roscoe, Scott Collis, Todd Coffey, David Day, Mike Heroux, Rob Hoekstra, Russell Hooper, Roger Pawlowski, Eric Phipps, Denis Ridzal, Andy Salinger, Heidi Thornquist, and Jim Willenbring. *ASC Vertical Integration Milestone*. SAND2007-5839, Sandia National Laboratories, 2007 [_[http://](http://www.cs.sandia.gov/~rabartl/publications.html)__[www.cs.sandia.gov/~rabartl/publications.html](http://www.cs.sandia.gov/~rabartl/publications.html)_]

## Outline

- Overview of Trilinos and Charon
- Overview of vertical solver algorithm integration
- Moving beyond the forward solve
- Challenges/barriers to embedded analysis methods
- Enabling methods
- Examples of vertically integrated algorithms with Trilinos and Charon
- Steady-state parameter estimation optimization with MOOCHO/Charon

## Outline

- Overview of Trilinos and Charon
- Overview of vertical solver algorithm integration
- Moving beyond the forward solve
- Challenges/barriers to embedded analysis methods
- Enabling methods
- Examples of vertically integrated algorithms with Trilinos and Charon
- Steady-state parameter estimation optimization with MOOCHO/Charon

## Overview of Trilinos

- Provides a suite of numerical solvers to support predictive simulation for Sandia’s customers
-   => Scope has expended to include discretizations methods, ...!
- Provides a decoupled and scalable development environment to allow for algorithmic research and production capabilities => “Packages”
- Provides support for growing SQA requirements
- Mostly C++ with some C, Fortran, Python ...
- Advanced object-oriented and generic C++ ...
- Current Status
- Current release: Trilinos 8.0.x (September 2007)
- Next release Trilinos 9.0 (September 2008)

_**Trilinos website**_
-         http://trilinos.sandia.gov

## Trilinos (8.0 & 9.0+) Package Summary

|  | Objective | Package(s) |
| --- | --- | --- |
| Discretizations | Meshing & Spatial Discretizations | phdMesh, Intrepid |
| Methods | Automatic Differentiation and UQ Prop. | Sacado, Stokos |
|  | Mortar Methods | Moertel |
| Core | Linear algebra objects | Epetra, Jpetra, Tpetra |
|  | Abstract interfaces | Thyra, Stratimikos, RTOp |
|  | Load Balancing | Zoltan, Isorropia |
|  | “Skins” | PyTrilinos, WebTrilinos, Star-P, ForTrilinos, CTrilinos |
|  | C++ utilities, (some) I/O | Teuchos, EpetraExt, Kokkos, Triutils |
| Solvers | Iterative (Krylov) linear solvers | AztecOO, Belos, Komplex |
|  | Direct sparse linear solvers | Amesos |
|  | Direct dense linear solvers | Epetra, Teuchos, Pliris |
|  | Iterative eigenvalue solvers | Anasazi |
|  | ILU-type preconditioners | AztecOO, IFPACK |
|  | Multilevel preconditioners | ML, CLAPS |
|  | Block preconditioners | Meros |
|  | Nonlinear system solvers | NOX, LOCA |
|  | Time Integration & Sensitivities | Rythmos |
| Analysis | Optimization (SAND) | MOOCHO, Aristos |

- Green:  Packages used in Vertical Integration Milestone
- Gray:  New packages that will be included in Trilinos 9.0 (September 2008) or later

## Trilinos Strategic Goals

**Scalable Computations:** As problem size and processor counts increase, the cost of the computation will remain nearly fixed.
**Hardened Computations:** Never fail unless problem essentially intractable, in which case we diagnose and inform the user why the problem fails and provide a reliable measure of error.
**Full Vertical Coverage:** Provide leading edge enabling technologies through the entire technical application software stack: from problem construction, solution, analysis and optimization.
***Grand***** Universal Interoperability**: All Trilinos packages will be interoperable, so that any combination of solver packages that makes sense algorithmically will be possible within Trilinos.
**Universal Accessibility:** All Trilinos capabilities will be available to users of major computing environments: C++, Fortran, Python and the Web, and from the desktop to the latest scalable systems.
**Universal Solver RAS**: Trilinos will be:
- _R_eliable: Leading edge hardened, scalable solutions for each of these applications
- _A_vailable: Integrated into every major application at Sandia
- _S_erviceable: Easy to maintain and upgrade within the application environment.

**Courtesy of Mike Heroux, Trilinos Project Leader**

- Algorithmic
- Goals

- Software
- Goals

## (Generalized PDE Solver)

**(Generalized PDE Solver)**

**Internal SNL Code for QASPR project**
**Large-scale parallel (MPI) **
**Unstructured grid finite elements**
**Automatic Differentiation**
**Adaptive Mesh Refinement**
**Generalized operators – fast addition of new operators/equations**
**Physics**
**Semiconductor Device**
**Multi-phase Aerosol**
**Reacting flows/gas-phase Combustion**
**MHD/Plasma**
**Algorithms testing ground**

- MHD Pump

- Semiconductor

- Reacting Flow

## Outline

- Overview of Trilinos and Charon
- Overview of vertical solver algorithm integration
- Moving beyond the forward solve
- Challenges/barriers to embedded analysis methods
- Enabling methods
- Examples of vertically integrated algorithms with Trilinos and Charon
- Steady-state parameter estimation optimization with MOOCHO/Charon

## Trilinos Strategic Goals

**Scalable Computations:** As problem size and processor counts increase, the cost of the computation will remain nearly fixed.
**Hardened Computations:** Never fail unless problem essentially intractable, in which case we diagnose and inform the user why the problem fails and provide a reliable measure of error.
**Full Vertical Coverage:** Provide leading edge enabling technologies through the entire technical application software stack: from problem construction, solution, analysis and optimization.
***Grand***** Universal Interoperability**: All Trilinos packages will be interoperable, so that any combination of solver packages that makes sense algorithmically will be possible within Trilinos.
**Universal Accessibility:** All Trilinos capabilities will be available to users of major computing environments: C++, Fortran, Python and the Web, and from the desktop to the latest scalable systems.
**Universal Solver RAS**: Trilinos will be:
- _R_eliable: Leading edge hardened, scalable solutions for each of these applications
- _A_vailable: Integrated into every major application at Sandia
- _S_erviceable: Easy to maintain and upgrade within the application environment.

**Courtesy of Mike Heroux, Trilinos Project Leader**

**Thyra** is being developed to address this issue

- Algorithmic
- Goals

- Software
- Goals

## Packages/Algorithms Most Directly to Vertical Integration Project

- Linear Problems:
- Linear equations:
- Eigen problems:

- Nonlinear Problems:
- Nonlinear equations:
- Stability analysis:

- Transient Nonlinear Problems:
- DAEs/ODEs
- ODE/DAE Sensitivities ...

- Optimization Problems:
- Unconstrained:
- Constrained:

- _Trilinos Packages_

- Belos

- Anasazi

- NOX

- LOCA

- MOOCHO

**Packages/Algorithms Most Directly to Vertical Integration Project**

- Rythmos

- Embedded Analysis Algorithms

**Notes:**
Here are some of the different types of abstract problems and the Trilinos packages implementing abstract numerical algorithms for solving these problems.

## Vertical Integration and Interoperability is Important

- Example: Numerous interactions exist between layers of  abstract numerical algorithms (ANAs) in a transient optimization problem

**Iterative Linear Solver**
**AztecOO, Amesos, Belos, ???**

**Operators and Vectors**
**Epetra, Tpetra, PETSc, ???**

**Nonlinear Solver**
**NOX, PETSc, ???**

**Nonlinear Optimizer**
**MOOCHO, ???**

**Key Points**
- Higher level algorithms, like optimization, require a lot of interoperability
- Interoperability and vertical integration must be “easy” or these configurations will not be achieved in practice

- What is needed to solve problem?
- Standard interfaces to break O(*N**2*) 1-to-1 couplings

**Application**
**Charon, Aria, ???**

**Transient Solver**
**Rythmos, SUNDIALS, ???**

**Vertical Integration and Interoperability is Important**

**Thyra** is being developed to address interoperability of ANAs by defining interfaces for:
-   Linear operators/vectors
-   Preconditioners / Linear solvers
-   Nonlinear models
-   Nonlinear solvers
-   Transient solvers

## Outline

- Overview of Trilinos and Charon
- Overview of vertical solver algorithm integration
- Moving beyond the forward solve
- Challenges/barriers to embedded analysis methods
- Enabling methods
- Examples of vertically integrated algorithms with Trilinos and Charon
- Steady-state parameter estimation optimization with MOOCHO/Charon

## Embedded Analysis Algorithms and “The Cutting Edge”

- Forward Simulator (linear solvers, preconditioners, ...)

- Complexity/Fidelity of Simulation (% of “cutting edge”)

- 0%

- 100%

- Sensitivities and Error-Estimation

- Optimization (Opt)

- UQ

- Opt and UQ

- The “Cutting Edge” for the Forward Simulation Application
- Drives capability computing (e.g., Gordan Bell, etc.)
- Drives (i.e., “Pulls”) R&D for linear solvers, preconditioners, ...
- Advanced Analysis Methods
- Lag behind the “cutting edge” of the forward simulation
- R&D reduces the lag!
- Less direct impact on the forward simulation results => Leads to “Push” instead of “Pull”
- Requires a different approach w.r.t. working with APP developers and customers!

## Challenges/Barriers to Embedded Analysis Algorithms

- Embedded Algorithms R&D with Production APPs
-  Better Algorithms R&D
-  Better Production APPs

- Version Control, Build, Test
- (incompatible dev sources, environments, tools, lack of testing, ...)
- APP + Trilinos Dev (Bartlett et. al.)

- Software Infrastructure
- (narrow forward solvers, inflexible implementation approaches, ...)
- Thyra ModelEvaluator (Bartlett et. al.)

- Derivatives
- (smoothness, accuracy, parameter derivatives, ... )
- AD/Sacado (Phipps and Gay)

- We are now addressing these barriers in a fundamental way to provide the foundation for sustained embedded algorithms R&D

## Nonlinear Algorithms and Applications : Thyra & Model Evaluator!

**Nonlinear Algorithms and Applications : Thyra & Model Evaluator!**

**Trilinos and non-Trilinos Preconditioner and Linear Solver Capability**

**NOX / LOCA**

**MOOCHO**

**Xyce**

**Charon**

**Aria**

**Tramonto**

**Aleph**

**...**

**...**

**Key Points**
- Provide single interface from nonlinear ANAs to applications
- Provides for shared, uniform access to linear solver capabilities
- Once an application implements support for one ANA, support for other ANAs can be added incrementally

- Nonlinear ANA Solvers in Trilinos

- Sandia Applications

**Rythmos**

**Stratimikos!**

**Model Evaluator**

## Some Nonlinear Problems Supported by the ModelEvaluator

- Nonlinear equations:

- Stability analysis:

- DAEs/Implicit ODEs:

- Explicit ODEs:

- DAE/Implicit ODE Forward Sensitivities:

- Unconstrained Optimization:

- Constrained Optimization:

- ODE Constrained Optimization:

- Explicit ODE Forward Sensitivities:

## APP + Trilinos Dev: Algorithms and Applications Integration

- Bartlett, Roscoe. *Daily Integration and Testing of the Development Versions of Applications and Trilinos: A stronger foundation for enhanced collaboration in application and algorithm research and development*, SAND2007-7040, Sandia National Laboratories, October 2007 [_[http://](http://www.cs.sandia.gov/~rabartl/publications.html)__[www.cs.sandia.gov/~rabartl/publications.html](http://www.cs.sandia.gov/~rabartl/publications.html)_]

- The Idea:
- Keep the development versions of APP and Trilinos code updated and tested daily
- Automated daily integrations tests
- =>Results in better production capabilities and better research
- Charon + Trilinos Dev
- Development versions of Charon and Trilinos are kept up-to-date every day!
- New embedded optimization and sensitivity capabilities are run and tested every day!
- Aria/SIERRA + Trilinos Dev
- We have automated configuration and daily integration testing of Aria/SIERRA VOTD against Trilinos Dev working!
- Now, we are addressing Aria/SIERRA software infrastructure issues and will start adding new embedded Trilinos analysis algorithms!

## Outline

- Overview of Trilinos and Charon
- Overview of vertical solver algorithm integration
- Moving beyond the forward solve
- Challenges/barriers to embedded analysis methods
- Enabling methods
- Examples of vertically integrated algorithms with Trilinos and Charon
- Steady-state parameter estimation optimization with MOOCHO/Charon

## Vertical Integrations of Trilinos Capabilities: Example 1

- Nonlinear Solver

- Time Integration

- Optimization

- Continuation

- Constrained Solves

- Sensitivity Analysis

- Stability Analysis

- Analysis Tools
-    (*embedded)*

- Data Structures

- Direct Solvers

- Linear Algebra

- Preconditioners

- Iterative Solvers

- Eigen Solver

- Matrix Partitioning

- Derivatives

- Derivative Tools

- Sensitivities

- Transient sensitivity analysis of a 2n2222 BJT
- in Charon* w/AD+Rythmos: 14x faster than FD*

- See Andy Salinger’s Talk at 11:00 AM

- Trilinos Capabilities

- See Eric Phipp’s Talk at 10:30 AM

## Vertical Integrations of Trilinos Capabilities: Example 2

- Nonlinear Solver

- Time Integration

- Optimization

- Continuation

- Constrained Solves

- Sensitivity Analysis

- Stability Analysis

- Analysis Tools
-    (*embedded)*

- Data Structures

- Direct Solvers

- Linear Algebra

- Preconditioners

- Iterative Solvers

- Eigen Solver

- Matrix Partitioning

- Derivatives

- Derivative Tools

- Sensitivities

- See Roger Pawlowski’s Talk at 11:30 AM

- Trilinos Capabilities

- Destabilizing eigen-vector for heated fluid in magnetic field

## Vertical Integrations of Trilinos Capabilities: Example 3

- Nonlinear Solver

- Time Integration

- Optimization

- Continuation

- Constrained Solves

- Sensitivity Analysis

- Stability Analysis

- Analysis Tools
-    (*embedded)*

- Data Structures

- Direct Solvers

- Linear Algebra

- Preconditioners

- Iterative Solvers

- Eigen Solver

- Matrix Partitioning

- Derivatives

- Derivative Tools

- Sensitivities

- I am talking about this next

- Trilinos Capabilities

- Steady-Sate Parameter Estimation Problem
- using 2n2222 BJT in Charon* MOOCHO + AD*

## Outline

- Overview of Trilinos and Charon
- Overview of vertical solver algorithm integration
- Moving beyond the forward solve
- Challenges/barriers to embedded analysis methods
- Enabling methods
- Examples of vertically integrated algorithms with Trilinos and Charon
- Steady-state parameter estimation optimization with MOOCHO/Charon

## QASPR Qualification of electronic devices in hostile environments

- Defect reactions

**QASPR ****Qualification of electronic devices in hostile environments**

- PDE semiconductor device simulation

- Stockpile BJT

## Steady-State Parameter Estimation with Charon/MOOCHO

- Minimize      Current model vs. target mismatch
- Subject to:   Steady-state semiconductor
-                   defect physic FE model

- Solved current matching optimization problems to calibrate model parameters against target currents
- MOOCHO (Bartlett) optimization solver converges simulation model and optimality at same time
- Faster and more robust than black-box optimization methods
- More accurate solutions
- Successes
- Very accurate inversion of currents and model parameters for contrived “inverse” problems
- Challenges
- Extremely difficult nonlinear solver convergences on model convergence                                                   => Opportunities for algorithm research
- Inability to match experimental data                                          => May indicate incomplete FE model

## Summary of Trilinos Vertical Integration Project (Milestone)

**Goal:** Vertically integrate Trilinos solver algorithms in Trilinos to build new predictive embedded analysis capabilities
**Impact:** Vertically integrated 10+ Trilinos algorithm packages
**Goal:** Demonstrate on relevant production applications
**Impact:** Solved steady-state parameter estimation problems and transient sensitivities on semiconductor devices in Charon
**Impact:**  Solved Eigen problems on MHD problem in Charon
**Added Goal: **Explore refined models of collaboration between production application developers and algorithm researchers.
** ****Impact: **Closer collaboration between application and algorithm developers yielding better algo and app R&D

- Bartlett, Roscoe, Scott Collis, Todd Coffey, David Day, Mike Heroux, Rob Hoekstra, Russell Hooper, Roger Pawlowski, Eric Phipps, Denis Ridzal, Andy Salinger, Heidi Thornquist, and Jim Willenbring. *ASC Vertical Integration Milestone*. SAND2007-5839, Sandia National Laboratories, 2007 [_[http://](http://www.cs.sandia.gov/~rabartl/publications.html)__[www.cs.sandia.gov/~rabartl/publications.html](http://www.cs.sandia.gov/~rabartl/publications.html)_]

## Replacement

**A Case Study on the Vertical Integration of Trilinos Solver Algorithms with a Production Application Code**
**Organizer:** **Roscoe A. Bartlett**
*Sandia National Laboratories*
**10:00-10:25 Overview of the Vertical Integration of Trilinos Solver Algorithms in a Production Application Code**
*Roscoe A. Bartlett*, Sandia National Laboratories
**10:30-10:55 Analytic Sensitivities in Large-scale Production Applications via Automatic Differentiation with Sacado**
*Eric Phipps*, Sandia National Laboratories
**11:00-11:25 To PDE Components and Beyond**
*Andy Salinger*, Sandia National Laboratories
**11:30-11:55 Analysis Tools for Large-scale Simulation with Application to the Stationary Magnetohydrodynamics Equations**
*Roger Pawlowski*, Eric Phipps, Heidi K. Thornquist, and Roscoe A. Bartlett, Sandia National Laboratories