# Presentation: 145448

## The Terascale Simulation Tools and Technologies Center

*http://www.tstt-scidac.org/*

- David Brown *(Lawrence Livermore*)
- Lori Freitag *(Sandia)*
- Jim Glimm *(Brookhaven, SUNY Stony Brook)*

## The TSTT Center

*Goal:* To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales
*Adaptive methods*
*Composite or hybrid solution strategies*
*High-order discretization strategies*
*Barrier:* The lack of easy-to-use interoperable meshing, discretization, and adaptive tools requires too much software expertise by application scientists
*   **The TSTT center recognizes this gap and will address the technical and human barriers preventing use of adaptive, composite, hybrid methods*

## TSTT Participants

- ANL: *Fischer*, Leurent, Tufo
- BNL: *Glimm*, Samulyak
- LLNL: *Brown*, Chand, Fast, Henshaw, Quinlan
- ORNL: *D’ Azevedo*, de Almeida, Khamayseh
- PNNL: *Trease*
- RPI: Flaherty, Hau, Remacle, *Shephard*
- SNL: Brewer, Freitag, *Knupp*, Melander, Tautges
- SUNY SB: Glimm, *Li*
**Italics: site PI*

## TSTT Mesh Management Tools

**Structured meshes**
**Overture** - high quality, predominantly structured meshes on complex CAD geometries (LLNL)
- Variational and Elliptic Grid Generators (ORNL, SNL)
**Unstructured meshes**
**AOMD** (RPI) - primarily tetrahedral meshes, boundary layer mesh generation, curved elements, AMR
**CUBIT** (SNL) - primarily hexahedral meshes, automatic decomposition tools, common geometry module
**NWGrid** (PNNL) - hybrid meshes using combined Delaunay, AMR and block  structured algorithms
**Frontier**** **(BNL) – interface front tracking

*MEGA* Boundary Layer Mesh (RPI)

*Overture* Mesh (LLNL)

## The Challenge

- These tools all meet particular needs, but
- They do not interoperate to form hybrid, composite meshes
- They cannot be interchanged easily in an application
- In general the technology requires too much software expertise from application scientists
- Difficult to improve existing codes
- Difficult to design and implement new codes

## Near and long term approach

***Near term****:** *deployment of current TSTT mesh and discretization capabilities by partnering with SciDAC applications
***Long term****:** *development of interoperable software tools enabling
- Rapid prototyping of new applications
- Plug-and-play insertion of mesh and discretization technology through uniform software interfaces

## Near Term Strategy

- Interact with SciDAC Applications. Develop working relationships in each application area by
- Analyzing the needs of application scientists
- Inserting existing TSTT technology
- Provides a short-term impact for application scientists
- Builds trust relationship
- Developing new technologies for later insertion and new application development
- Key application areas:  Fusion, Astrophysics, Accelerator Design, Climate

## High-Order FEM for Fusion

- High-order, adaptive finite element techniques for magneto-hydrodynamics
- Fusion PI:  Jardin/Strauss (PPPL)
- TSTT PI: Shephard/Flaherty (RPI)
*Goal:* To test high-order and adaptive techniques; compare to existing linear FEM
*Progress:*
- Initial results obtained for both potential and primitive variable mixed formulations for the 2D adipole vortex flow pattern
- Two oppositely directed currents embedded in a constant magnetic field which holds them in an unstable equilibrium
- They compress and rotate to align with magnetic field to reduce energy (see below)
- Test of high-order and h-adaptive techniques available in Trellis to determine applicability to this problem
- Quadratic and cubic results

## Example : Tilt Instability

-  Initial equilibrium consists of two oppositely directed currents embedded in a constant magnetic field
-  Initial magnetic field (B) is a dipole vortex
-  Vortices are unstable to perturbations
-  Kinetic energy grows like exp(t)
-  Stream function formulation of MHD solved using stabilized finite element method

**Magnetic Flux (**********)**

- = 0.005, order = 2 and mesh of  924  elements

- t = 0

- t = 5

- t = 6

## Adaptive DG for Astrophysics

- Contact instabilities in hydrodynamics
- Application PI: Bhattacharjee/Rosner (Iowa/UofC)
- TSTT PI: Shephard (RPI)
*Goal:* to test h-p adaptive DG in hydrodynamics; compare to existing PPM
*Progress:* 3-D adaptive test to 256 processors have been done in Trellis for four contact Riemann problem
- Boltzman transport equations for neutrinos
- Application PI: Mezzacappa (ORNL)
- TSTT PI: de Almeida (ORNL)
*Goal:* to eliminate barriers imposed by discrete ordinates discretization (non-adaptive, computationally intensive) by developing a discontinuous Galerkin alternative
*Progress:* adaptive DG shows strong exponential decay, energy conservation, and outward peaking and gives better results than DOM

- Adaptive mesh and density contours after structures have evolved.  Colors on right mesh indicates processor assignment for this 4 processor case

- DOM does not reach
- asymptotic limit at large
- optical depth and does
- not conserve energy
- Adaptivity in DGM
- provides more accuracy
- the slight loss of energy
- will be corrected

- Mean Radiation Intensity (J); Net Energy Flux (H)

## GOALS

**GOALS**

**APPROACH**

**ACCOMPLISMENTS**

**SUPERNOVA  RAD  TRANSFER**

- Novel solution methods for neutrino
- Boltzmann transport as part of supernova
- simulation models
- Faster, less memory consuming, and more
- accurate than current methods

- Discontinuos Galerkin method for hyperbolic
- Operator
- Adaptive finite element of phase space
- Splitting of hyperbolic and integral operators
- Moment iteration

- Solution of the spherical Milne’s problem and
- comparison against the Discrete Ordinate
- Method
- Proposed method captures all singularities
- Faster and less memory consuming by one
- order of magnitude
- Significantly more accurate

## Mesh Quality in Accelerator Design

- Understanding the effect of mesh quality on Tau3P
- Application PI: Ko/Folwell (SLAC)
- TSTT PI: Knupp (SNL), Henshaw (LLNL)
*Goal:* Determine the mesh quality factors that most affect stability of Tau3P and to devise discretization schemes to improve the stability of Tau3P without affecting long-time accuracy
*Progress:*
- Systematic mesh quality analysis using CUBIT meshes revealed that run time varies by a factor of 3 from “best” to “worst” mesh and that smoothness and orthogonality are the most important factors
- Analytically derived sufficient conditions on mesh quality for stability of discretization in Tau3P
- Implemented basic Tau3P discretization strategy in Overture and analyzing feasibility of schemes for increasing the artificial diffusion

## Climate

- Adaptive gridding to minimize solution error
- Application PI: Drake (ORNL)
- TSTT PI: Khamayseh (ORNL)
*Goal:* Given an initial isotropic or anisotropic planar or surface mesh and a solution field with large gradient mountain heights, use solution based r-adaptation to minimize solution error
*Progress*: Proof of principle of meshing technologies demonstrated
- Geodesic mesh quality improvement
- Application PI: Randall/Ringler (Colorado)
- TSTT PI: Knupp (SNL)
*Goal:* Create smoothed geodesic grids to improve calculation accuracy
- Plan to use early version of Mesquite to create smoothed grids with respect to element area and perform calculations with smoothed grids to determine effect in Fall ’02

## Other examples where TSTT technology is helping near-term application progress

- Front tracking and adaptive techniques in Frontier and Overture for modeling of the breakup of a diesel fuel jet into spray (Argonne/BNL)
- 3D caching schemes to avoid redundant, costly evaluations of scattering kernels in phase space in astrophysics calculations (ORNL/ORNL)
- Mesh-based schemes for computational biology applications such as rat olfactory systems and human lungs (PNNL/PNNL)
- Low-order discretization schemes used as effective preconditioners in Climate applications (Colorado/ANL)

- Access pattern :red is more frequent

- Access sequence: red is accessed late in simulation

## Long Term Strategy

- Create interoperable meshing and discretization components
- Common interfaces for mesh query and modification
- Initial design will account for interoperability at all levels
- Encapsulate existing TSTT software tools into CCA-compliant components for plug and play
- Develop new technologies as needed to enable interoperability
- High-level discretization library
- Mesh quality improvement for hybrid meshes
- Terascale algorithms for adaptivity, load balancing, interpolation

## Low Level Access

- Access the mesh through the individual components
- For example
- element-by-element access to mesh components
- fortran-callable routines that return interpolation coefficients at a single point (or array of points)
- Facilitates incorporation into existing applications

## High Level Access

- Operate on the mesh components as though they were a single mesh object
- Discretization operators
- Mesh modifications
- Mesh quality improvement
- Refinement/coarsening
- Error estimation
- Multilevel data transfer
- Prototypes provided by Overture and Trellis frameworks
- Enables rapid development of new mesh-based applications

## TSTT Technology Goal

**The Data Hierarchy**
**Level A:**** Geometric description of the domain**
- provides a common frame of reference for all tools
- facilitates multilevel solvers
- facilitates transfer of information in discretizations
**Level B:**** Full geometry hybrid meshes**
- mesh components
- communication mechanisms that link them (key new research area)
- allows structured and unstructured meshes to be combined in a single computation
**Level C****: Mesh Components**

**Geometry**
**Information**
**(Level A)**

**Full **
**Geometry**
**Meshes**
**(Level B)**

**Mesh**
**Components**
**(Level C)**

**To provide *****interchangeable***** and***** ******interoperable ***
**access to different mesh management and **
**discretization strategies**
- Ease experimentation with different technologies
- Combine technologies together for hybrid solution techniques

## The Challenge

- TSTT brings together many meshing and discretization tools
- Structured Grids:  Overture, ORNL variational techniques
- Unstructured Meshes: AOMD, CUBIT, NWGrid, Frontier
- These tools all meet particular needs, but
- They do not interoperate to form hybrid, composite meshes
- They cannot be easily interchanged in an application
- In general the technology requires too much software expertise from application scientists
- Difficult to improve existing codes
- Difficult to design and implement new codes

*MEGA* Boundary Layer Mesh (RPI)

## Technology Development Strategy

- Create plug-and-play meshing and discretization components from existing technologies
- Define common interfaces for mesh query and modification
- Showcase interoperability goal through one-to-one demonstrations
- Encapsulate existing TSTT software tools into CCA-compliant components for plug and play
- Develop new technologies as needed to enable interoperability
- Mesh quality improvement for hybrid meshes
- High-level discretization library
- Terascale algorithms for adaptivity, load balancing, interpolation

## Interoperability Development Plan

- Use TSTT interfaces to use TSTT tools interoperably

## Interoperability:  TSTT Interface Specification

**Philosophy**
- Create a small sets of interfaces that existing packages can support
- Be data structure neutral
- Balance performance and flexibility
- Work with a large tool provider and application community to ensure applicability
**Status**
- Interfaces mesh geometry and topology well underway
- Mesh Query
- Entity Sets (subsetting)
- Modifiable Meshes (a basic form of adaptive meshing)
- Prototype interfaces for geometry and field data
- Prototype interface for the mesh/geometry data model manager
- Classification
- Prototype implementations
- AOMD, Overture, NWGrid, MDB/CUBIT
- C, C++, and Fortran language interoperability through SIDL/Babel (CCA)
- Used in Mesquite for interchangeablity
- Point of Contact: L. Freitag

## One-to-One Tool Interoperability

- Showcases impact of interoperable tools on applications
- Frontier (BNL/SUNY SB) - Overture (LLNL) Merge
- Combines adaptive mesh technology with Front-tracking
- Initial merge complete and used in simulations
- Next step is to parallelize adaptive schemes for a scalable solution strategy
- Will create a TSTT-compliant Frontier-Lite library for use with other TSTT mesh management tools (NWGrid, AOMD)
- Point of Contact: X. Li
- NWGrid (PNNL) - Opt-MS (ANL)
- Incorporates previously developed mesh quality improvement tools
- Uses a CCA interfaces for dynamic plug and play
- Migration to Mesquite via the TSTT interface planned for Fall 03
- Point of Contact: H. Trease

## New TSTT Tools:  Mesh Quality Improvement

- Goal:  To provide a stand-alone tool for mesh quality improvement
- hybrid, component based meshes
- development of quality metrics for high order methods
- a posteriori quality control using error estimators
- Methods
- optimization-based smoothing and untangling (based on Opt-MS and CUBIT algorithms)
- reconnection schemes
- Status
- Initial design complete and implemented
- Tri, tet, quad, hex, and hybrid meshes
- Several quality metrics and objective functions
- Conjugate Gradient, Newton, Active Set solvers
- SciDAC Impact
- SLAC mesh quality improvement
- Geodesic grids in climate
- Integrated with CUBIT, AOMD via TSTT interface
- Points of Contact: P. Knupp, L. Freitag

## Mesh Quality Improvement

*Goal*:  To provide a stand-alone tool for mesh quality improvement
- hybrid, component based meshes
- development of quality metrics for high order methods
- a posteriori quality control using error estimators
*Methods*
- optimization-based smoothing and untangling (based on Opt-MS and CUBIT algorithms)
- reconnection schemes
*Status*
- Prototype designed and most classes implemented for a simple optimization algorithm
- Opt-MS and CUBIT algorithms inserted this summer
- Built on TSTT interface
*SciDAC Application Impact*
- SLAC mesh quality improvement
- Geodesic grids in climate
- Integrated with CUBIT, NWgrid, Overture, AOMD

## New TSTT Tools:  Discretization Library

*Observation:* Complexities of using high-order methods on adaptively evolving grids has hampered their widespread use
- Tedious low level dependence on grid infrastructure
- A source of subtle bugs during development
- Bottleneck to interoperability of applications with different discretization strategies
- Difficult to implement in general way while maintaining optimal performance
- Result has been a use of sub-optimal strategies or lengthy implementation periods
*TSTT Goal:* to eliminate these barriers by developing a *Discretization Library*

## Example: Overture prototype

- CompositeGrid cg;
- floatCompositeGridFunction u,v,w;
- v = u.y();
- w = u.laplacian();
- Plotstuff ps;
- ps.plot (cg);
- ps.contour (w);

*Differentiation Operators*

- Trellis (RPI) provides similar capability for
- finite-element method

## Discretization Library Functionality and Status

- Planned functionality
- Mathematical operators will be implemented
- +, -, *, /, interpolation, prolongation, div, grad, curl, etc
- Both strong and weak (variational) forms of operators when applicable
- Many discretization strategies will be available
- Emphasize high-order and variable-order methods
- Extensive library of boundary condition operators
- The interface will be independent of the underlying mesh  by using the TSTT interface
- Status:
- Existing schemes decoupled from their frameworks
- Finite Element, Discontinuous Galerkin from Trellis; Finite Volume from Overture;  Spectral Elements from Nek5000
- Working to create a set of common interfaces for application use
- Exploring the issues associated with hybrid solution strategies (mixed element meshes, mixed discretization solution techniques)
- Point of Contact: D. Brown

## TSTT ISIC Collaborations

- TOPS: (PI: Keyes)
- provide mesh representations for multilevel techniques
- co-develop well-defined interfaces to ensure that the meshes and discretization strategies will be interoperable with solution software
- APDEC: (PI: Colella)
- provide mesh generation technologies via Overture
- co-develop common interfaces for block structured AMR strategies
- CCA: (PI: Armstrong)
- co-develop common interfaces for mesh and field data
- create CCA-compliant mesh components and provide them in the CCA component repository
- explore the role of the component model in the composition of numerous discrete operators
- Performance: (PI: Bailey)
- we will use *ROSE *preprocessor to develop highly-tuned discretization libraries
- TSTT will provide benchmarks and a testing environment for developments in the performance ISIC

## Summary

-    The TSTT Center focuses on interoperable meshing and discretization strategies on complex geometries
- Short term impact through technology insertion into existing SciDAC applications
- Long term impact through the development of
- a common mesh interface and interoperable and interchangeable mesh components
- new technologies that facilitate the use of hybrid meshes
- Working with SciDAC ISICs to ensure applicability of tools and interfaces

## Contact Information

- TSTT Web Site: _[www.tstt-scidac.org](http://www.tstt-scidac.org/)_
- David Brown: _[dlb@llnl.gov](mailto:dlb@llnl.gov)_
- Lori Freitag: _[ladiach@sandia.gov](mailto:ladiach@sandia.gov)_
- Jim Glimm: _[glimm@ams.sunysb.edu](mailto:glimm@ams.sunysb.edu)_