# Presentation: 934471

## Thyra For Developers

**Roscoe A. Bartlett **
**Department of Optimization & Uncertainty Estimation**
**Sandia National Laboratories**
**Trilinos Users Group Meeting (Developers Day), November 9****th****, 2006**

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

## Outline

- What’s New with Thyra
- Cross-Package issues
- ???

## Outline

- What’s New with Thyra
- Cross-Package issues
- ???

## What’s New with Thyra (for Developers)

- Complete implicit linear operators
- Handle classes
- Operator/Vector adapters for Tpetra
- Linear solver adapter for Belos
- Stratimikos wrapper for linear solvers and preconditioners

## Outline

- What’s New with Thyra
- Cross-Package issues
- ???

## Cross Package Issues related to Thyra

- Structured output
- Teuchos::Describable
- Teuchos::VerboseObject
- Teuchos::FancyOStream
- Reporting timings
- Teuchos::Time[Monitor]
- Accepting parameters (hieratically)
- Teuchos::ParameterList
-   Thyra interfaces and implementation depend on all of these!
-   These require inner package support for uniformity!

## Structured Outputting : Teuchos::VerboseObject

- Layered solvers will generate lots of output
- Need to indent output for each solver layer (Teuchos::FancyOStream and OSTab)
- Line prefixes with Teuchos::FancyOStream may help
- Mostly handled as the Thyra Level!
- Issues for non-Thyra code:
- Always print to an std::ostream (or a Teuchos::FancyOStream), never std::cout or std::cerr directly!
-  Using Teuchos::VerboseObjectBase::getDefaultOStream() is aways better than using std::cerr or std::cout!
- Don’t use if(procID ==0) to determine when to create output, let the std::ostream decide (i.e. use Teuchos::blackholestream or Teuchos::FancyOSteam to control process output)
- Want the option to print from specific processes for debugging etc ...
- Don’t print your main ouput to a file!

## Reporting of Timings : Teuchos::Time[Monitor]

- Teuchos::TimeMonitor
- Can be used to time almost any type of operation
- Reports timing summary across processes
- Naming of timers should be namespace! (with the package name)
- e.g. “Belos::Solve ...” instead of just “Solve ...””
- Enhancements: See Bug 2107

## The Significance of Parameter Lists and Outputting

- In the future, most users will only interact with Trilinos solvers through setting parameters and looking at output!

- <ParameterList>
-   ...
-   <Parameter name=“Time Stepper” type=“string” value=“Rythmos::ImplicitBDFStepper”/>
-   <ParameterList name=“Rythmos::SUNDIALS”> ... </ParameterList>
-   <ParameterList name=“**Rythmos::ImplicitBDFStepper**”>
-     <Parameter name=“Nonlinear Solver” type=“string” value=“NOX”/>
-     <ParameterList name=“TimeStepNewtonNonlinearSolver”> ... </ParameterList>
-     <ParameterList name=“**NOX**”>
-       <Parameter name=“Linear Solver” type=“string” value=“Stratimikos”/>
-       <ParameterList name=“**Stratimikos**”>
-         <Parameter name="Linear Solver Type" type="string" value=“Belos"/>
-         <ParameterList name="Linear Solver Types">
-           <ParameterList name=“Amesos”> ... </ParameterList>
-           <ParameterList name=“AztecOO”> ... </ParameterList>
-           <ParameterList name=“**Belos**”> ... </ParameterList>
-         </ParameterList>
-         <Parameter name="Preconditioner Type" type="string" value="ML"/>
-         <ParameterList name="Preconditioner Types">
-           <ParameterList name="Ifpack"> ... </ParameterList>
-           <ParameterList name="**ML**"> ... </ParameterList>
-           <ParameterList name=“Meros"> ... </ParameterList>
-         </ParameterList>
-       </ParameterList>
-     </ParameterList>
-   </ParameterList>
- </ParameterList>

**Time Stepper**

**Nonlinear Solver**

**Lin. Solver**

**Precond.**

**Linear Solver**

- Example:
-        Time Stepper => Nonlinear Solver => Linear Solver => Preconditioner

## User Requirements for use of Parameter Lists

- Allow specification of as many parameters as possible from an input file
- i.e. XML format or some other format
- Get back default values of parameters after a run for parameters not set by user
- e.g. NOX returns the default values of parameters
- Misspelling a parameter or sublist should result in a helpful error message and will not be silently ignored
- Make it easy change choices and sub-parameters back and forth easily
- Make it easy to find documentation and for the accepted parameter values

## Developer Requirements for use of Parameter Lists

- Make it easy to extract parameters and their values
- Make it easy to validate parameters, sublists, and parameter values
- Allow flexibility in how validation is done
- Make it easy to avoid mistakes

## Outline

- Developer Interaction with the Parameter Lists

## Standardized Approach for Object Handling of Parameter Lists

- Teuchos::ParameterListAcceptor
- Parameter list objects (not just parameter values) are “set” and remembered by the object!
- Parameters may only be read later during a solve
- Parameter list objects are non-const and can be augmented with default option values, output lists ...
- The “valid parameters” parameter list can be used for automatic documentation and for external validation is some cases ...
- Many Thyra interfaces inherit from Teuchos::ParameteListAcceptor:
- e.g. Teuchos::LinearOpWithSolveFactoryBase, Teuchos::PreconditionerFactoryBase

- Namespace Teuchos {
-   // Base class mix-in interface for objects that accept parameter lists
-   class ParameterListAcceptor {
-   public:
-     // Set parameters from a parameter list and return with default values
-     virtual void setParameterList( RCP<ParameterList> paramList ) = 0;
-     // Get the parameter list that was set using setParameterList()
-     virtual RCP<ParameterList>  getParameterList() = 0;
-     // Unset the parameter list that was set using setParameterList()
-     virtual RCP<ParameterList>  unsetParameterList ()=0
-     // Return a const parameter list of all of the valid parameters
-     virtual RCP<const ParameterList >  getValidParameters() const;
-     ...
-   };
- }

## Parameter List Validation

- A valid parameter list is compared against *this parameter list
- Parameters and sublists can be validated to any level desired
- => Gives control over how validation is done

- namespace Teuchos {
-   class ParameterList {
-   public:
-     ...
-     void validateParameters(
-       ParameterList        const& validParamList
-       ,int                 const  depth            = 1000
-       ,EValidateUsed       const  validateUsed     = VALIDATE_USED_ENABLED
-       ,EValidateDefaults   const  validateDefaults = VALIDATE_DEFAULTS_ENABLED
-       ) const;
-     ...
-   };
- }

## Recent Additions to the Teuchos::ParameterList class

- Parameter Entry documentation: Optional (multi-line) documentation string
- Parameter Entry Validators: Used in PL::validateParameters(...) the type and/or value of a parameter!

- namespace Teuchos {
-   class ParameterList {
-   public:
-     ...
-     template<typename T>
-     void set(
-       string                              const& name
-       ,T                                  const& value
-       ,std::string                        const& docString = ""
-       ,RCP<const ParameterEntryValidator> const& validator = null
-       );
-     ...
-   };
- }

- namespace Teuchos {
-   class ParameterEntryValidator {
-   public:
-     // Print documentation for this parameter.
-     virtual void printDoc(
-       std::string         const& docString
-       ,std::ostream            & out       ) const = 0;
-     // Return an array of strings of valid values if applicable.
-     virtual Teuchos::RefCountPtr<const Array<std::string> > validStringValues() const = 0;
-     // Validate a parameter entry value and throw exception if validation fails.
-     virtual void validate(
-       ParameterEntry  const& entry
-       ,std::string    const& paramName
-       ,std::string    const& sublistName ) const = 0;
-   };
- }

## Constructing a “Valid” Parameter List

- // Setup static objects that define parameters
- const std::string AztecSolver_name = "Aztec Solver";
- const Teuchos::RefCountPtr<Teuchos::StringToIntegralParameterEntryValidator<int> >
- aztecSolverValidator = Teuchos::rcp(
-   new Teuchos::StringToIntegralParameterEntryValidator<int>(
-     Teuchos::tuple<std::string>("CG","GMRES","CGS","TFQMR","BiCGStab","LU")
-     ,Teuchos::tuple<int>(AZ_cg,AZ_gmres,AZ_cgs,AZ_tfqmr,AZ_bicgstab,AZ_lu)
-     ,AztecSolver_name
-     )
-   );
- const std::string AztecSolver_default = "GMRES";
- ...
- Teuchos::RefCountPtr<Teuchos::ParameterList>  validAztecOOParams;
- ...
- // Create valid parameter list
- Teuchos::RefCountPtr<const Teuchos::ParameterList>
- getValidAztecOOParameters()
- {
-   ...
-   RefCountPtr<ParameterList> pl = validAztecOOParams;
-   if(pl.get()) return pl;
-   pl = validAztecOOParams = rcp(new ParameterList());
-   //
-   pl->set(
-     AztecSolver_name,AztecSolver_default
-     ,"Type of linear solver algorithm to use."
-     ,aztecSolverValidator
-     );
-   ..
-   //
-   return pl;
- }

- See: Trilinos/packages/aztecoo/thyra/src/AztecOOParameterList.cpp

## Reading from a User’s Parameter List

- // Setup static objects that define parameters
- const std::string AztecSolver_name = "Aztec Solver";
- const Teuchos::RefCountPtr<Teuchos::StringToIntegralParameterEntryValidator<int> >
- aztecSolverValidator = Teuchos::rcp(
-   new Teuchos::StringToIntegralParameterEntryValidator<int>(
-     Teuchos::tuple<std::string>("CG","GMRES","CGS","TFQMR","BiCGStab","LU")
-     ,Teuchos::tuple<int>(AZ_cg,AZ_gmres,AZ_cgs,AZ_tfqmr,AZ_bicgstab,AZ_lu)
-     ,AztecSolver_name
-     )
-   );
- const std::string AztecSolver_default = “GMRES";
- ...
- Teuchos::RefCountPtr<Teuchos::ParameterList>  validAztecOOParams;
- ...
- // Read the parameters (initial validation is performed by client!)
- void setAztecOOParameters(
-   Teuchos::ParameterList  *pl
-   ,AztecOO                *solver
-   )
- {
-   ...
-   // Aztec Solver
-   solver->SetAztecOption(
-     AZ_solver
-     ,aztecSolverValidator->getIntegralValue(*pl,AztecSolver_name,AztecSolver_default)
-     );
-   ...
- #ifdef TEUCHOS_DEBUG
-   // Check to make sure that I did not use the PL incorrectly!
-   pl->validateParameters(*getValidAztecOOParameters());
- #endif // TEUCHOS_DEBUG
- }

- See: Trilinos/packages/aztecoo/thyra/src/AztecOOParameterList.cpp

## Outline

- User Interaction with the Parameter Lists

## Parameter List and Sublist Structure : Stratimikos

- <ParameterList name=“Stratimikos”>
-   <Parameter name="Linear Solver Type" type="string" value=“AztecOO"/>
-   <Parameter name="Preconditioner Type" type="string" value="Ifpack"/>
-   <ParameterList name="Linear Solver Types">
-     <ParameterList name="Amesos">
-       <Parameter name="Solver Type" type="string" value="Klu"/>
-       <ParameterList name="Amesos Settings">
-         <Parameter name="MatrixProperty" type="string" value="general"/>
-         ...
-         <ParameterList name="Mumps"> ... </ParameterList>
-         <ParameterList name="Superludist"> ... </ParameterList>
-       </ParameterList>
-     </ParameterList>
-     <ParameterList name="AztecOO">
-       <ParameterList name="Forward Solve">
-         <Parameter name="Max Iterations" type="int" value="400"/>
-         <Parameter name="Tolerance" type="double" value="1e-06"/>
-         <ParameterList name="AztecOO Settings">
-           <Parameter name="Aztec Solver" type="string" value="GMRES"/>
-           ...
-         </ParameterList>
-       </ParameterList>
-       ...
-     </ParameterList>
-     <ParameterList name="Belos"> ... </ParameterList>
-   </ParameterList>
- <ParameterList name="Preconditioner Types">
-     <ParameterList name="Ifpack">
-       <Parameter name="Prec Type" type="string" value="ILU"/>
-       <Parameter name="Overlap" type="int" value="0"/>
-       <ParameterList name="Ifpack Settings">
-         <Parameter name="amesos: solver type" type="string" value="Amesos_Klu"/>
-         ...
-       </ParameterList>
-     </ParameterList>
-     <ParameterList name="ML"> ... </ParameterList>
-   </ParameterList>
- </ParameterList>

**Linear Solvers**

**Preconditioners**

- See Doxygen documentation for Thyra::DefaultRealLinearSolverBuilder!

**Sublists passed on to package code!**

**Top level parameters**

**Every parameter and sublist ****not in red**** is handled by Thyra code and is fully validated!**

## Automatically Generated Parameter List Documentation

- “Human readable” automatically generated documentation for Stratimikos

-  Linear Solver Type : string = Amesos
-     # Determines the type of linear solver that will be used.
-     # The parameters for each solver type are specified in the sublist "Linear Solver Types"
-     #   Valid values: "Belos", "Amesos", "AztecOO"
-   Preconditioner Type : string = ML
-     # Determines the type of preconditioner that will be used.
-     # This option is only meaningful for linear solvers that accept preconditioner factory objects!
-     # The parameters for each preconditioner are specified in the sublist "Preconditioner Types"
-     #   Valid values: "None", "Ifpack", "ML"
-   Linear Solver Types ->
-     AztecOO ->
-       Output Every RHS : bool = 0
-         # Determines if output is created for each individual RHS (true or 1) or if output
-         # is just created for an entire set of RHSs (false or 0).
-       Forward Solve ->
-         Max Iterations : int = 400
-           # The maximum number of iterations the AztecOO solver is allowed to perform.
-         Tolerance : double = 1e-06
-           # The tolerence used in the convergence check (see the convergence test
-           # in the sublist "AztecOO Settings")
-         AztecOO Settings ->
-           Aztec Solver : string = GMRES
-             # Type of linear solver algorithm to use.
-             #   Valid values: "CG", "GMRES", "CGS", "TFQMR", "BiCGStab", "LU"
-           Convergence Test : string = r0
-             # The convergence test to use for terminating the iterative solver.
-             #   Valid values: "r0", "rhs", "Anorm", "no scaling", "sol"
-           ...
-     ...

- Generated (hiearchially) by Thyra::DefaultRealLinearSolverBuilder::getValidParameters()->print(...)
- See simple_stratimikos_example.exe --only-print-options --print-readable-format

## Output of Default Parameter Values

- <ParameterList name="Stratimikos">
-   <Parameter name="Linear Solver Type" type="string" value="AztecOO" />
-   <Parameter name="Preconditioner Type" type="string" value="Ifpack" />
- </ParameterList>

- Example input parameter (sub)list to Stratimikos (use simple_stratimikos_example.exe)

- <ParameterList>
-   <Parameter name="Linear Solver Type" type="string" value="AztecOO"/>
-   <ParameterList name="Linear Solver Types">
-     <ParameterList name="AztecOO">
-       <ParameterList name="Forward Solve">
-         <ParameterList name="AztecOO Settings">
-           <Parameter isDefault="true" name="Aztec Solver" type="string" value="GMRES"/>
-           <Parameter isDefault="true" name="Convergence Test" type="string" value="r0"/>
-           ...
-         </ParameterList>
-         <Parameter isDefault="true" name="Max Iterations" type="int" value="400"/>
-         <Parameter isDefault="true" name="Tolerance" type="double" value="1e-06"/>
-       </ParameterList>
-       <Parameter isDefault="true" name="Output Every RHS" type="bool" value="0"/>
-     </ParameterList>
-   </ParameterList>
-   <Parameter name="Preconditioner Type" type="string" value="Ifpack"/>
-   <ParameterList name="Preconditioner Types">
-     <ParameterList name="Ifpack">
-       <Parameter isDefault="true" name="Overlap" type="int" value="0"/>
-       <Parameter isDefault="true" name="Prec Type" type="string" value="ILU"/>
-     </ParameterList>
-   </ParameterList>
- </ParameterList>

- Output (augmented) parameter (sub)list

- => Ifpack is not showing its default parameters!

## Why Parameter Validation is Important

- Silently ignoring misspelled parameter names is unacceptable
- Many users will have to wade through thousands of lines of out to find warnings
- Makes it difficult to write testing software that ensures that parameters are indeed recognized
- Almost every other kind of software will step and print error message
- e.g. compilers, scripting languages, ...

- ...
- A thousand lines of output
- ...
- Unused parameters:
- Statimikos ->
-   Linear Solver Types ->
-     AztecOO ->
-       AztecOO Settings ->
-         Aztec Solver : string = GMRES [default]
-         ...
-         tens of parameters ...
-         ...
-         ztec Solver : string = GMRES [unused]
- ...
- A hundred lines of output
- ...

- Example: User misspells “Aztec Solver” as “ztec Solver”

- Poor old user has to ask:
- Was this option spelled correctly but just not read?
- Was this option spelled incorrectly and therefore not used?

## Error Messages for Improper Parameters/Sublists

- Example: User misspells “Aztec Solver” as “ztec Solver”

- <ParameterList>
-   <Parameter name="Linear Solver Type" type="string" value="AztecOO"/>
-   <ParameterList name="Linear Solver Types">
-     <ParameterList name="AztecOO">
-       <ParameterList name="Forward Solve">
-         <ParameterList name="AztecOO Settings">
-           <Parameter name="ztec Solver" type="string" value="GMRES"/>
-         </ParameterList>
-       </ParameterList>
-     </ParameterList>
-   </ParameterList>
- </ParameterList>

- Error message generated from PL::validateParameters(...) with exception:

- p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions20InvalidParameterNameE' :
-   /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_ParameterList.cpp:352:
-   Throw test that evaluated to true: !validEntry
-   Error, the parameter {name="ztec Solver",type="string",value="GMRES"}
-   in the parameter (sub)list "DefaultRealLinearSolverBuilder->Linear Solver Types->AztecOO->Forward Solve->AztecOO Settings"
-   was not found in the list of valid parameters!
-   The valid parameters and types are:
-        {
-             "Aztec Preconditioner" : string = ilu
-             "Aztec Solver" : string = GMRES
-             ...
-        }

## Error Messages for Improper Parameters/Sublists

- Example: User specifies the wrong type for “Aztec Solver”

- <ParameterList>
-   <Parameter name="Linear Solver Type" type="string" value="AztecOO"/>
-   <Parameter name="Preconditioner Type" type="string" value="Ifpack"/>
-   <ParameterList name="Linear Solver Types">
-     <ParameterList name="AztecOO">
-       <ParameterList name="Forward Solve">
-         <ParameterList name="AztecOO Settings">
-           <Parameter name="Aztec Solver" type="int" value="GMRES"/>
-         </ParameterList>
-       </ParameterList>
-     </ParameterList>
-   </ParameterList>
- </ParameterList>

- Error message generated from PL::validateParameters(...) with exception:

- p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions20InvalidParameterTypeE' :
- /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_StandardParameterEntryValidators.hpp:279:
-   Throw test that evaluated to true: !validType
-   Teuchos::StringToIntegralParameterEntryValidator::getIntegralValue(...):
-   Error, the parameter {paramName="Aztec Solver",type="int"}
-   in the sublist "DefaultRealLinearSolverBuilder->Linear Solver Types->AztecOO->Forward Solve->AztecOO Settings"
-   has the wrong type.  The correct type is "string"!

## Error Messages for Improper Parameters/Sublists

- Example: User specifies the wrong value for “Aztec Solver”

- <ParameterList>
-   <Parameter name="Linear Solver Type" type="string" value="AztecOO"/>
-   <Parameter name="Preconditioner Type" type="string" value="Ifpack"/>
-   <ParameterList name="Linear Solver Types">
-     <ParameterList name="AztecOO">
-       <ParameterList name="Forward Solve">
-         <ParameterList name="AztecOO Settings">
-           <Parameter name="Aztec Solver" type=“string" value="GMRESS"/>
-         </ParameterList>
-       </ParameterList>
-     </ParameterList>
-   </ParameterList>
- </ParameterList>

- Error message generated from PL::validateParameters(...) with exception:

- p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions21InvalidParameterValueE' :
-   /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_StandardParameterEntryValidators.hpp:260:
-   Throw test that evaluated to true: itr == map_.end()
-   Teuchos::StringToIntegralParameterEntryValidator::getIntegralValue("GMRESS",...):
-   Error, the value "GMRESS" is not recognized for the parameter "Aztec Solver" in the sublist "".
-   Valid selections include: "CG", "GMRES", "CGS", "TFQMR", "BiCGStab", "LU".

## A few Comments on Parameter List Validation

- PL::validateParameters(...) is only a tool for validation, not a “catch-all’ solution
- How validation is performed is not important as long as it gets done!
- Validation is ultimately the responsibility of the objects that accept and read from a parameter lists
- Full validation may not occur until the underlying solver finishes
- We should not have to (dramatically) change current parameter list structures
- e.g. Ifpack, ML, Amesos, ...
- 100% validation may not be possible in all cases

## Parameter List Issues?

- Handling multiple value types for a parameter and validation?
- Accepting an string value or an enum value (e.g. Epetra_CombineMode)
- In code as: Add, Zero, Insert, InsertAdd, Average, or AbsMax
- In XML as: “Add”, “Zero”, “Insert”, “InsertAdd”, “Average”, or “AbsMax”
- Allowing for implicit type conversions (e.g. “level-of-fill”)
- Input as int, double, float ..., Output as int, double, float
- Solution => Teuchos::ParameterEntryValidator subclasases!
- Handling and validation of non-“parameter” parameters?
- Status tests in NOX or null-space vectors in ML
- Adding an RCP-wrapped derived class object and getting back base class object?
- Solution => Add specialized handling of RCP-wrapped objects for NOX PL class ...

## Parameter List Issues?

- Preserving ordering of parameter and sublists?
- Issue for Teuchos::ParameterList
- Issue for Teuchos::XMLObject
- Adding documentation and valid parameter value lists to the XML representation
- Reading in sublists from another file (in the XML parsesr)
- Broadcasting a parameter list?  Non-parameter parameters? ...
- GUI for setting XML parameters?
- Issues Ken mentioned to me:
- Case sensitivity?
- Accept “symmetric” for “Symmetric”?
- Partial matching?
- Accept “Sym” for “Symmetric”?
- Error messages?
- Suggesting correct spellings of parameters?
- Suggest “Symmetric” when user inputs “symmetric”?
- Any other issues?

## Parameter List Summary

- Consistency in the use and the look-and-feel of parameter lists is needed
- User (should) expect parameter lists to be validated
- User parameter lists and developer use of parameter lists must be validated!
- Many tools for validation
- PL::validateParameters( validParamList, ... )
- ParameterListValidator (StringToIntegralParameterListValiator, ...)