# Presentation: 200104

## ModelEvaluatorScalable, Extendable Interface Between Embedded Nonlinear Analysis Algorithms and Applications

**Roscoe A. Bartlett **
**Department of Optimization & Uncertainty Estimation**
**Sandia National Laboratories**
**April 3****rd,**** 2008**

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

## Overview of Nonlinear Solver and Analysis Capabilities

## Trilinos Nonlinear Solver and Analysis Packages

- Nonlinear Problems:
- Nonlinear equations:
- Stability analysis:

- Transient Nonlinear Problems:
- DAEs/ODEs
- ODE/DAE Sensitivities ...

- Optimization Problems:
- Unconstrained:
- Constrained:

- _Trilinos Packages_

- NOX

- LOCA

- MOOCHO

**Trilinos Nonlinear Solver and Analysis Packages**

- Rythmos

**Notes:**
Here are some of the different types of abstract problems and the Trilinos packages implementing abstract numerical algorithms for solving these problems.

## ModelEvaluator Overview

## Nonlinear Algorithms and Applications : Everyone for Themselves?

**NOX / LOCA**

**Rythmos**

**MOOCHO**

**Xyce**

**Charon**

**Aria**

**Tramonto**

**Aleph**

**...**

**...**

**Trilinos and non-Trilinos Preconditioner and Linear Solver Capability**

**Key Point**
-  BAD

- Nonlinear ANA Solvers in Trilinos

- Sandia Applications

## Overview of Nonlinear Model Evaluator Interface

- Motivation: An interface for nonlinear problems is needed that will support a variety of different types of problems
- Nonlinear equations (and senstitivities)
- Stability analysis and continuation
- Explicit ODEs (and sensitivities)
- DAEs and implicit ODEs (and sensitivities)
- Unconstrained optimization
- Constrained optimization
- Uncertainty quantification
- ...
- as well as different combinations of problem types such as:
- Uncertainty in transient simulations
- Stability of an optimum under uncertainty of a transient problem

- Approach: Develop a single, scalable interface to address all of these problems

- (Some) Input arguments:
- State and differential state:
- Parameter sub-vectors:
- Time (differential):

- (Some) Output functions:
- State function:
- Auxiliary response functions:
- State/state derivative operator (LinearOpWithSolve):

**Key Point**
- All inputs and outputs are optional and the model evaluator object itself decides which ones are accepted.

**Key Point**
- The number of combinations of different problem types is large and trying to statically type all of the combinations is not realistic

## Some Nonlinear Problems Supported by the ModelEvaluator

- Nonlinear equations:

- Stability analysis:

- DAEs/Implicit ODEs:

- Explicit ODEs:

- DAE/Implicit ODE Forward Sensitivities:

- Unconstrained Optimization:

- Constrained Optimization:

- ODE Constrained Optimization:

- Explicit ODE Forward Sensitivities:

## Nonlinear Algorithms and Applications : Everyone for Themselves?

**NOX / LOCA**

**Rythmos**

**MOOCHO**

**Xyce**

**Charon**

**Aria**

**Tramonto**

**Aleph**

**...**

**...**

**Trilinos and non-Trilinos Preconditioner and Linear Solver Capability**

**Key Point**
-  BAD

- Nonlinear ANA Solvers in Trilinos

- Sandia Applications

## Nonlinear Algorithms and Applications : Thyra & Model Evaluator!

**Nonlinear Algorithms and Applications : Thyra & Model Evaluator!**

**Trilinos and non-Trilinos Preconditioner and Linear Solver Capability**

**NOX / LOCA**

**MOOCHO**

**Xyce**

**Charon**

**Aria**

**Tramonto**

**Aleph**

**...**

**...**

**Key Points**
- Provide single interface from nonlinear ANAs to applications
- Provide single interface for applications to implement to access nonlinear ANAs
- Provides shared, uniform access to linear solver capabilities
- Once an application implements support for one ANA, support for other ANAs can quickly follow

- Nonlinear ANA Solvers in Trilinos

- Sandia Applications

**Rythmos**

**Stratimikos!**

**Model Evaluator**

## Model Evaluator : Thyra and EpetraExt Versions

***Thyra::ModelEvaluator***

*create_W_op() : LinearOpBase create_W() : LinearOpWithSolveBase createInArgs() : InArgs*
*createOutArgs() : OutArgs*
*...*
*evalModel( in InArgs, out OutArgs )*

***EpetraExt::ModelEvaluator***

*createInArgs() : InArgs*
*createOutArgs() : OutArgs*
*create_W() : Epetra_Operator*
*...*
*evalModel( in InArgs, out OutArgs )*

**Thyra::EpetraModelEvaluator**

- createInArgs() : InArgs
- createOutArgs() : OutArgs
- create_W() : LinearOpWithSolveBase
- ...
- evalModel( in InArgs, out OutArgs )

***Thyra::LinearOpWithSolveFactoryBase***

**Nonlinear ANA**

**Concrete Application**

- Thyra-based interoperability interface layer

- Epetra-based application-friendly implementation support interface

- Thyra::ModelEvaluator and EpetraExt::ModelEvaluator are near mirror copies of each other.
- Thyra::EpetraModelEvaluator is fully general adapter class that can use any linear solver through a Thyra::LinearOpWithSolveFactoryBase object it is configured with
- Stateless model that allows for efficient multiple shared calculations (e.g. automatic differentiation)
- Adding input and output arguments involves
- Modifying only the Thyra::ModelEvaluator,  EpetraExt::ModelEvaluator, Thyra::EpetraModelEvaluator, and a few other auxiliary classes and tools => O(1) overhead to add new arguments!
- Only recompilation of Nonlinear ANA and Concrete Application code

**Stratimikos!**

## Simple Example EpetraExt::ModelEvaluator Subclass

## Example EpetraExt::ModelEvaluator Application Implementation

- /** \brief Simple example ModelEvaluator subclass for a 2x2 set of
-  * parameterized nonlinear equations.
-  *
-  * The equations modeled are:
-  \verbatim
-     f[0] =       x[0]      + x[1]*x[1] - p[0];
-     f[1] = d * ( x[0]*x[0] - x[1]      - p[1] );
-  \endverbatim
-  */
- class EpetraModelEval2DSim : public EpetraExt::ModelEvaluator {
- public:
-   EpetraModelEval2DSim(...);
-   /** \name Overridden from EpetraExt::ModelEvaluator . */
-   //@{
-   RCP<const Epetra_Map>      get_x_map() const;
-   RCP<const Epetra_Map>      get_f_map() const;
-   RCP<const Epetra_Vector>   get_x_init() const;
-   RCP<Epetra_Operator>       create_W() const;
-   InArgs      createInArgs() const;
-   OutArgs     createOutArgs() const;
-   void evalModel( const InArgs& inArgs, const OutArgs& outArgs ) const;
-   //@}
- private:
-   ...
- };

- Complete nonlinear equations example in:
- Trilinos/packages/epetraext/example/model_evaluator/2dsim/

## Example EpetraExt::ModelEvaluator Application Implementation

- EpetraExt::ModelEvaluator::InArgs EpetraModelEval2DSim::createInArgs() const
- {
-   InArgsSetup inArgs;
-   inArgs.setModelEvalDescription(this->description());
-   inArgs.setSupports(IN_ARG_x,true);
-   return inArgs;
- }
- EpetraExt::ModelEvaluator::OutArgs EpetraModelEval2DSim::createOutArgs() const
- {
-   OutArgsSetup outArgs;
-   outArgs.setModelEvalDescription(this->description());
-   outArgs.setSupports(OUT_ARG_f,true);
-   outArgs.setSupports(OUT_ARG_W,true);
-   outArgs.set_W_properties(
-     DerivativeProperties(DERIV_LINEARITY_NONCONST,DERIV_RANK_FULL,true)
-     );
-   return outArgs;
- }
- void EpetraModelEval2DSim::evalModel( const InArgs& inArgs, const OutArgs& outArgs ) const
- {
-   const Epetra_Vector &x = *inArgs.get_x();
-   RCP<Epetra_Vector> f_out = outArgs.get_f();
-   RCP<Epetra_Operator> W_out = outArgs.get_W();
-   if (!is_null(f_out)) {
-     ...
-   }
-   if (!is_null(W_out)) {
-     ...
-   }
- }

**Key Point**
- From looking at example code, there is not even a hint that other input and output arguments exist!

## Vertical Integration Milestone:ModelEvaluator in Action!

## Summary of Trilinos Vertical Integration Project (Milestone)

**Goal:** Vertically integrate Trilinos solver algorithms in Trilinos to build new predictive embedded analysis capabilities
**Impact:** Vertically integrated 10+ Trilinos algorithm packages
**Goal:** Demonstrate on relevant production applications
**Impact:** Solved steady-state parameter estimation problems and transient sensitivities on semiconductor devices in Charon
**Impact:**  Solved Eigen problems on MHD problem in Charon
**Added Goal: **Explore refined models of collaboration between production application developers and algorithm researchers.
** ****Impact:**** **Closer collaboration between application and algorithm developers yielding better algo and app R&D

- Bartlett, Roscoe, Scott Collis, Todd Coffey, David Day, Mike Heroux, Rob Hoekstra, Russell Hooper, Roger Pawlowski, Eric Phipps, Denis Ridzal, Andy Salinger, Heidi Thornquist, and Jim Willenbring. *ASC Vertical Integration Milestone*. SAND2007-5839, Sandia National Laboratories, 2007 [_[http://](http://www.cs.sandia.gov/~rabartl/publications.html)__[www.cs.sandia.gov/~rabartl/publications.html](http://www.cs.sandia.gov/~rabartl/publications.html)_]

## QASPR Qualification of electronic devices in hostile environments

- Defect reactions

**QASPR ****Qualification of electronic devices in hostile environments**

- PDE semiconductor device simulation

- Stockpile BJT

## Vertical Integrations of Trilinos Capabilities: Example 1

- Nonlinear Solver

- Time Integration

- Optimization

- Continuation

- Constrained Solves

- Sensitivity Analysis

- Stability Analysis

- Analysis Tools
-    (*embedded)*

- Data Structures

- Direct Solvers

- Linear Algebra

- Preconditioners

- Iterative Solvers

- Eigen Solver

- Matrix Partitioning

- Derivatives

- Derivative Tools

- Sensitivities

- Transient sensitivity analysis of a 2n2222 BJT
- in Charon* w/AD+Rythmos: 14x faster than FD*

- Trilinos Capabilities

## Vertical Integrations of Trilinos Capabilities: Example 2

- Nonlinear Solver

- Time Integration

- Optimization

- Continuation

- Constrained Solves

- Sensitivity Analysis

- Stability Analysis

- Analysis Tools
-    (*embedded)*

- Data Structures

- Direct Solvers

- Linear Algebra

- Preconditioners

- Iterative Solvers

- Eigen Solver

- Matrix Partitioning

- Derivatives

- Derivative Tools

- Sensitivities

- Trilinos Capabilities

- Destabilizing eigen-vector for heated fluid in magnetic field

## Vertical Integrations of Trilinos Capabilities: Example 3

- Nonlinear Solver

- Time Integration

- Optimization

- Continuation

- Constrained Solves

- Sensitivity Analysis

- Stability Analysis

- Analysis Tools
-    (*embedded)*

- Data Structures

- Direct Solvers

- Linear Algebra

- Preconditioners

- Iterative Solvers

- Eigen Solver

- Matrix Partitioning

- Derivatives

- Derivative Tools

- Sensitivities

- Trilinos Capabilities

- Steady-Sate Parameter Estimation Problem
- using 2n2222 BJT in Charon* MOOCHO + AD*

## Derivative Support and Sensitivity Analysis

## Model Derivatives

- State function parameter sensitivities:

- [LinearOpBase or MultiVectorBase]

- Auxiliary function state sensitivities:

- Auxiliary function parameter sensitivities:

- [LinearOpBase or MultiVectorBase2]

- [LinearOpBase or MultiVectorBase2]

- State function state sensitivities:

- [LinearOpWithSolveBase or LinearOpBase]

- First Derivatives

- Use Cases:
- Steady-state and transient sensitivity computations
- Optimization
- Multi-physics coupling
- ...

**Key Point**
- Derivative class used to store one of these three forms of a derivative

## Forward/Direct and Adjoint Sensitivities

**Forward/Direct and Adjoint Sensitivities**

- Steady-state constrained response:

- Reduced response function:

- Reduced Sensitivities:

- where:

- Forward/Direct Sensititivies (*n**g* large, *n**p* small)

- -1

- =

- +

- -

- Adjoint Sensititivies (*n**g* small, *n**p* large)

- MV

- LO

- LOWS

- MV

- MV

- MV

- -T

- =

- +

- -

*n**p*

*n**g*

*n**g*

*n**p*

- MV

- LO

- LOWS

- MV

- MV

- MV

**Key Point**
- The form of the derivatives you need depends on whether you are doing direct or adjoint sensitivities

## Transient Sensitivities: State Equations with Responses

- Fully Implicit ODE/DAE State Equations

- Composite Response Function

- Reduced Composite Response Function

- where:

- http://www.cs.sandia.gov/~rabartl/TransientSensitivitiesDerivation2007.pdf

## Transient Sensitivities: Forward Sensitivity Method

- Forward Sensitivity Equations:

- Forward Reduced Gradient

- http://www.cs.sandia.gov/~rabartl/TransientSensitivitiesDerivation2007.pdf

- Forward Sensitivity Methods:
- The forward sensitivities d(x)/d(p) are integrated right along with the forward state equation
- Can be solved using explicit or implicti time integration mehtods
- O(np) extra storage and computation per time step
- Reuse of forward solver integrator infrastructure, Jacobian/preconditioner storage

## Forward Transient Sensitivities with Charon/Rythmos

- QASPR transient current sensitivities w.r.t. reaction parameters for an irradiated semiconductor device modeled with Charon
- Embedded sensitivities with AD/Sacado (Phipps) & Rythmos
- Finite differences (steplen=1e-2) (optimal steplen=1e-1)
- Embedded sensitivities vs. finite diff.
- Much more accurate and robust!
- 10x faster for 40 parameters!

- Bartlett, Roscoe, Scott Collis, Todd Coffey, David Day, Mike Heroux, Rob Hoekstra, Russell Hooper, Roger Pawlowski, Eric Phipps, Denis Ridzal, Andy Salinger, Heidi Thornquist, and Jim Willenbring. *ASC Vertical Integration Milestone*. SAND2007-5839, Sandia National Laboratories, 2007 [_[http://](http://www.cs.sandia.gov/~rabartl/publications.html)__[www.cs.sandia.gov/~rabartl/publications.html](http://www.cs.sandia.gov/~rabartl/publications.html)__[]](http://www.cs.sandia.gov/~rabartl/publications.html)_

## ModelEvaluator Software Summary

- Large overlap in commonality between requirements for many nonlinear abstract numerical algorithms (ANAs).
- Incremental support for sensitivities, optimization, and UQ
- Mixed problem types will become more and more common and must be easy to support
- Properties of ModelEvalutor Software
- Strong Typing of Input/Object Types but Weak Typing of Problem Formulation
- Designed for Augmentation
- Incremental Development of Model Types and Capabilities
- Self-Describing Models => Smart Algorithms
- Independence/Multiplicity of Input and Output Objects
- ANAs already using or can use ModelEvaluator
- NOX (nonlinear equations)
- LOCA (stability analysis, continuation)
- Rythmos (explicit ODEs, implicit ODEs, DAEs)
- MOOCHO (constrained optimization, unconstrained optimization, nonlinear equations)
- Aristos (full space, trust-region optimization)

**ModelEvaluator Software Summary**