# Presentation: 307265

## The slides that follow were presented at the PSAAP Bidder's Meeting May 16-17, 2006 and represent the ASC Trilab authors and interests as presented in the associated White Paper for this subject area.

- Predictive Science Academic Alliance Program (PSAAP)

## Verification and Validation

**Frank R. Graziani****1**
**Scott Doebling****2, ****Richard Klein****1****, Tim Trucano****3**
**1. Lawrence Livermore National Laboratory**
**2. Los Alamos National laboratory**
**3. Sandia National Laboratory**
**May 16-17, 2006**

**This work was performed under the auspices of the U.S. Department of Energy by the University of**
**California, Lawrence Livermore National Laboratory under Contract No. W-7405-Eng-48.**

**UCRL-PRES-221297**

## Instructions for viewing slides

**These slides contain animations and need to be viewed under slide show**
**Open the slides in PowerPoint**
**Go to the tab that says “slide show”**
**Click on “view show”**
**Press “ESC” key to quite viewing**
**The animation feature works on both PC’s and MAC’s**

## Predictive science capability lies at the heart of the Advanced Simulation and Computing (ASC) mission 

***The Mission of the ASC program is to “Provide leading edge, high-end simulation capabilities needed to meet weapons assessment and certification requirements.”***
***The Vision of the ASC program is to “Predict, with confidence, the behavior of nuclear weapons, through comprehensive, science-based simulations.”***
***The Strategic Goal of the ASC program is to provide “Predictive simulations and modeling tools, supported by necessary computing resources, to sustain long-term stewardship of the stockpile.”***
***Verification and validation will provide high confidence in computational accuracy by systematically measuring, documenting, and demonstrating the predictive capability of codes***** **

- United States Department of Energy (2003), “Advanced
- Simulation and Computing Program Plan”, SAND2003-3130P

## V&V will play an important role in the Predictive Science Academic Alliance Program 

***“******The resulting simulation models lend themselves to practical verification and validation methodologies and strategies that should include the integrated use of experimental and/or observational data as a key part of model and sub-model validation, as well as demonstrations of numerical convergence and accuracy for code verification.” ***
***“...******verification, validation and prediction methodologies and results must be much more strongly emphasized as research topics and demonstrated via the proposed simulations.”***
***“******It is mandatory that proposals address the following two topics:***
***Predictability in science & engineering.***
***Verification & validation strategies for large-scale simulations, including quantification of uncertainty and numerical convergence.”***

- NNSA (2006), “Predictive Science Academic Alliance
- Program (PSAAP) - Guidelines for Applications of Interest

## Verification and validation is what distinguishes a physics code from a computer game   

**Validation : Are we solving the right equations ?**
**Comparison with experimental physics data**
**Verification : Are we solving the equations accurately ?**
**Comparison with analytic and semi-analytic solutions**
**Computer game or physics ?**
**The integrity of any computational result rests on the foundations of a solid validation and verification program****   **
**ASC Alliances**
**A V&V program must be an integral part of any Alliance code project**
**Avoid the “*****research hand grenade syndrome*****”**
**Members of the Alliances should spend time at the National Labs and vice versa**

## Code and solution verification answers the questions are the equations represented correctly and by how much ?

**Code verification**
**Are the equations represented by a code *****mathematically***** (not physically) correct?**
**Are the algorithms that provide the numerical solution of the mathematical equations themselves mathematically correct? **
** ****Is the software implementation of these algorithms correct (that is, free of faults)? **
**Analytic solutions**
**Code-to-code comparisons**
**Mesh convergence studies**
** **
**Solution verification**
**Quantification of the numerical error**
- A posteriori error estimation
- Convergence studies
- Numerical error models
- Uncertainty quantification methods

**L2 norm**

**simulation**

**“****answer”**

## The principles of structure, construction and assessment underlie verification testing and test problems

**Structure**
**Why is a given test problem chosen for a verification test suite?**
**How is the test suite organized?**
**Code coverage**
**Construction**
**Test problem formulation**
**Finding or developing new test problems**
**Assessment**
**What are the criteria for deciding whether or not a given test problem has passed or failed?**

**Cold**

**What is the effect of mesh imprinting**
**On the radiation diffusion package ?**

**Hot source**

**Is this level of **
**distortion acceptable?**

## A new generation of multi-D/coupled physics verification test problems is needed

**Analytic and semi-analytic solutions form a regular part of code development and testing at the National Labs  **
**Noh**
**Shock tube**
**Sedov blast wave**
**Su-Olson 2T Marshak like test problem**
**Graziani multi-group prompt spectrum **
**Current verification suite tends to be limited in scope**
** ****1D planar or spherical (self-similar)**
**Single physics coverage**
**Reinicke-Meyer-ter-Vehn**** **
**Critical issues****: **
**Multi-D and/or coupled physics verification test problems needed**
**Method of manufactured solutions**
**Practical methods for estimating numerical errors associated with spatial and/or temporal discretizations**
**Methods for estimating numerical errors associated with parameters that influence the algorithm (e.g. hourglassing)**
**National labs and PSAAP centers should share new and existing test problems**
**Engage the applied mathematics community to assist in the development of new problems**

**.02 cm**

**CH(Li)**

**.04 cm**

**T=.3 keV**

**simulation**

**gas **

**radiation **

**Su-Olson 2T Diffusion**

**solution**

**energy density**

**Log(X)**

**Prompt spectrum of a radiating sphere**

**simulation**

**solution**

**Timmes et. al, LA-UR-05-6865**

**J. Hayes, ZEUS-2D**

**Reinicke and Meyer-ter-Vehn, Phys. Fluids A 3, 1807 (1991)**

## Experimental and calculational validation answers the question “Are we solving the correct equations?”

**Validation quality experiments**
**     *****Experimental data that are to be compared to validation quality simulations for the purpose of inferring physical accuracy****** ***
- Was the correct experiment executed?
- Quantified experimental uncertainty
- Reproducibility
- Robustness
- Comparable to calculational set-up (diagnostics)
**Validation quality calculations**
**     *****Calculations that are compared to validation  quality data for the purpose of inferring physical accuracy***
- Verification process completed
- Faithful representation of experiment
- Quantified calculational uncertainty
- Uncertainty quantification
**Assessment**
- Are the calculations compared to validation quality experimental data?
- What has the completed validation work told us about predictive capability?
- The science of simulation-experiment comparison is immature ... we need help here!

**What are the physics priorities **
**for validation study?**

## Integrity of code produced at centers rests on the code’s ability to reproduce data coming from well diagnosed  laboratory experiments.

**Critical Issues**
**There exists experimental data that can be used for validation purposes....however, there is a need for new validation quality experimental data **
- Multi-dimensional radiation flow in homogenous and heterogeneous media
- Fluid instabilities. Material mixing, and turbulent flow
- Material fracture, failure and spall
- Charged particle transport and reaction rates in strongly coupled plasmas
- Coupled radiation-hydrodynamics
- Thermal-mechanical ignition and propagation in HE** **
** **
**Validation methodology**
- Advanced statistical methods for making quantitative measurement/prediction comparisons
- Tools to automate the process of quantitative validation
- Statistical methods for validation when there is a single well instrumented test
**Direct numerical simulation**

**Marinak et al, Phys. Rev. Lett. 80, 4426 (1998)**

**Non-linear R-T evolution of 3D multimode perturbation**

**experiment**

**simulation**

## Stopping power formulas can be validated against both experiments and direct numerical simulations  

- Belyaev et al., *Phys. Rev. E* **53**, 2701 (1996)

**Temporal behavior of the proton **
**energy loss in the plasma target**

**Experiment inferred a value for the Coulomb logarithm which was **
**consistent with theoretical predictions for weakly coupled plasmas**

**Coulomb energy loss by 1 MeV protons**

**Comparison of different approximation schemes for the stopping power with data from PIC (*) and MD (crosses) confirm the quantum kinetic approach in strongly coupled plasmas **

- Zwicknagel et al., *Phys. Rep. * **309**, 117 (1999)

## Uncertainty quantification: Answering the question “ What is the expected uncertainty in the output metrics of the calculation”

**Uncertainty quantification**
- What is the expected uncertainty in the output metrics of the calculation?
- Critical to predictive capability
**Sensitivity analysis is the first step in uncertainty quantification**
**    *****Goal: Identify and quantify the dominant sensitivities that drive the uncertainty***
- Identify all avenues of certainty for code
- Which input variabilities contribute most to the overall output metric uncertainty
- Statistical sampling and the curse of dimensionality
- Screening designs
**Rolling up input factor variabilities into an output uncertainty....*****this is a challenge and help is needed***

**“****Everything is sensitive if you make the input variation large enough”**
- Variations in parameters must be consistent with
- Experimental uncertainty and/or
- Theoretical uncertainty and/or
- Best judgment
**Identifying experimental and/or theoretical uncertainties**
**     “****How well do we know what we think we know?”**
- Forces one to understand experimental methods and evaluations
- Forces one to understand theoretical assumptions
- Exposes nature of code implementation
- Sometimes exposes the lack of any experimental and/or theoretical understandings in the phase space of interest
**“****How do we capture missing or incorrect physics”**

**Removing the “Curse”**
**Statistical samplings methods such as **
**Monte Carlo or replicated Latin Hypercube **
**reduce N**
**Screening designs such as Morris-One-At-a-Time reduce p**

- Saltelli et al., *“Sensitivity Analysis”*, **Wiley** (2000)

## Requirements include addressing verification, validation and uncertainty quantification

**Verification requirement**
- Document and execute a plan for the development of verification test problems
- Define and apply relevant analytic and semi-analytic test problems
- Include both component and coupled physics in the suite
- Describe how ASC national laboratories and the PSAAP center will exchange test problems and their results
**Validation requirement**
- Document and execute a validation plan
- Detail what experimental data is presently available for validation
- Description of approach to quantifying calculational error bars for the validation calculations
- If required experimental data do not exist, define the experiments that will be needed for validation purposes
- Description of how new experiments will be funded
-  Leveraging funds with existing in-house facilities
**Uncertainty requirement**
**     **Document and execute a sensitivity and uncertainty quantification plan
- Description of how input factor variabilities are rolled up into an overall output uncertainty
- Exploring new approaches is encouraged