# Presentation: 013082

## V&V Principles and Challenges

**Timothy Trucano**
**Optimization and Uncertainty Estimation Department**
**William Oberkampf and Martin Pilch**
**Validation and Uncertainty Processes**
**Sandia National Laboratories**
**Albuquerque, NM 87185**
**Phone: 844-8812, FAX: 844-0918**
**Email: ****tgtruca@sandia.gov**
**Trilinos Users Group Meeting 2006**
**SNL, November 9, 2006**
**SAND2006-7046C**

**Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration ****under contract DE-AC04-94AL85000.**

## Verification and Validation (V&V) Definitions

**Verification:**** Are the equations solved correctly? ****(Math)**
**Validation:**** Are the equations correct? ****(Physics)**

**ASC:**
_**Verification**_**: The process of determining that a model implementation accurately represents the developer’s conceptual description of the model and the solution to the model.**
_**Validation**_**: The process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model.**

**V&V targets **_**applications**_** of codes, not codes.**

## Verification and Validation (V&V) Definitions

**Example: linear solvers**
**“****Solve equations correctly” and “correct equations” still make sense.**
**But also keep in mind (IEEE):**
**	**_**Verification**_** = requirements implemented correctly (not quite Math)**
**	**_**Validation**_** = correct requirements (not quite Physics)**
**All the general issues project to Trilinos:**
**Hence Trilinos is part of the “problem”**
**But Trilinos may also be part of the “solution”**

## WHY do we have a V&V program?

**WHY do we have a V&V program?**

**Notes:**
Quote – comment made at LANL, December, 1997, during a meeting on a program plan for the to-be-started ASCI V&V program in October, 1998. The specific context of the quote was concern about encompassing “qualification” of codes in the program.

## 	Instead, the NNSA Advanced Simulation and Computing Program states that the purpose of computing is: 
“high-performance, full-system, high-fidelity-physics predictive codes to support weapon assessments, renewal process analyses, accident analyses, and certification.” 
	(DOE/DP-99-000010592)

**“****high-performance, full-system, high-fidelity-physics**** **_**predictive codes**_** ****to support weapon assessments, renewal process analyses, accident analyses, and certification.” **
**	****(DOE/DP-99-000010592)**

**The ASC Program’s Grasp is large!**

**“****The purpose of computing is not insight.”**

**Notes:**
Quote attributed to Hamming – but a lot of people use it and believe it.

## Outline

**History**
**Process**
**Verification**
**Validation**
**Stockpile – Toward QMU**
**Final Cautions**

**Perceived Mission Statement: Taking the fun out of computational science.**

**Notes:**
People’s faces often scrunch up with distaste when V&V is mentioned.

## A Little History

**Autumn 1997 validation workshop kicks off development of an ASCI “Validation Program” (“code developers do verification”).**
**ASCI “Validation and Verification Program” launches FY1999 (“maybe we do need to worry about verification”).**
**FY2007 begins 9****th**** year of “Verification and Validation Program” (“verification is very important and difficult”)**
**The program has been highlighted by milestones designed to optimize SSP impact.**
**The ASC Predictive Science Academic Alliance Program (PSAAP) White Paper on V&V does a reasonable job of defining the goals and values of the program.**

**1998**

**2006**

**Notes:**
The initial debate about the inclusion of verification in the “Validation Program” reflects debate over roles and responsibilities. Because V&V tends to be a cross-cutting program that combines several disciplinary concerns, including code development and experiments, roles and responsibilities debates persist and emerge in unexpected ways. (I do not accept the point of view that debates over roles and responsibilities are irrelevant to the ASC program.)

## Do you trust the calculation? 

_**Can**_** you trust the calculation?**
**	****Three reasons you may not wish to bet your life on a calculation:**
**Wrong physics (****validation****)**
**Wrong numerics (****verification****)**
**Wrong math, algorithms, software**
**Lousy numerical accuracy**
**Wrong use of the results* (****decisions****)**
**(* Especially scary!)**

## V&V is a methodology.

**Code**
**Verification**

**DP**
**Application**

**Planning**

**Experiment**
**Design, Execution**
**& Analysis**

**Metrics**

**Assessment**

**Prediction **
**& Credibility**

**Document**

**Calculation**
**Verification**

**1**

**7**

**6**

**5**

**2**

**4**

**3**

**3**

**8**

**Requirements and planning**

**Verification**

**Validation Metrics**

**Credibility**

**Permanence**

**Validation Experiments**

**Described in SAND2002-0341**

**Notes:**
This “process” was established in 2000-2001.
It really is a “methodology” (a collection of methods)
It underlies the work of the V&V program – milestones, DSW, everything
It guides collaborations (w/ code projects, Campaign 6, 7)

The origin of this methodology was one basic question – What Is A Good Validation Experiment?

## Simplified (everybody in ASC does this):

**Plan**

**Verify**

**Validate**

**Credibility**

**This methodology has been applied to all SNL ASC V&V milestones.**
**Current work includes QASPR and an FY08 HEDP milestone.**

## Planning

**It’s all about the APPLICATION of the code(s):**
**Application Requirements**
**Phenomenology Identification and Ranking Table (PIRT)**
**Priorities**
**Identify focused verification requirements (supplementing code development)**
**Identify **_**hierarchical**_** validation requirements**
**Understand resultant link to credibility for the application**
**Described in SAND2000-3101.**

**“****Everybody in the room pretty much knows what to do ...” **

**Notes:**
Quote – QASPR meeting, by someone who doesn’t want to do V&V.

## VerificationConsider the following comparison with data:

**Verification****Consider the following comparison with data:**

**What is the computational (crosses and stars) error?**
**“****Good agreement” with experimental data (circles) does not imply **_**numerical**_** accuracy!**
**(A DNS resolution study underlies the computational points.)**

**Numerical**
**Error**

**“****Calculation”**
**Verification**

**Software**
**Implementation**

**Algorithms**

**Mathematics**

**Correct?**

**Correct?**

**Correct?**

**“****Code” Verification**

**To believe any numerical error statement requires “code verification.”**

**Notes:**
The figure appears in Gregory Falkovich and K. R. Sreenivasan (2006), “Lessons from Hydrodynamic Turbulence,” Physics Today, April, 43-49.
The original source is S. Y. Chen, et al (2005), “Anomalous Scaling of Low-Order Structure Functions of Turbulent Velocity,” Journal of Fluid Mechanics, Volume 533, 183-192.
Even if there were numerical error bars, why would we believe them?
Error bars = solution verification
Evidence supporting their correctness = code verification.

## Verification

**Verification**

**Numerical**
**Error**

**Software**
**Implementation**

**Algorithms**

**Mathematics**

**Correct?**

**Correct?**

**Correct?**

**“****Code” Verification**

**Verification challenges immediately project into Trilinos.**

**“****Calculation” Verification**

## Verification is hard!

**This was clearly stated to ASCI, for example in January 1999.**
**Credible numerical error statements require a significant **_**code verification**_** foundation:**
**Proofs that math and algorithms are correct**
**Proofs that the software has no bugs**
**Anything less is an approximation and has epistemic (lack-of-knowledge) uncertainty attached to it**
**Error statements themselves (**_**solution verification**_**) come from a (presently) limited technology base:**
**Convergence studies (highly empirical – can I take these to the bank?)**
**A posteriori error estimation (not in our favorite equations)**
**Error “models” with intrinsic uncertainty (“the error probably is...”)**

## Verification is hard! Code verification:

**Two particulars:**
_**Software engineering**_**, which I will spare you since you know much more about this than I do.**
_**Specialized verification testing**_** – “functional testing” (not “unit” testing, not “regression testing,” not “structural” testing).**
**How to define tests? See Roy (2005) for current discussion of test design, including Method of Manufactured Solutions**
**There is NO AUTHORITATIVE PRESENTATION OF TESTING FOR CS&E IN ANY LITERATURE!**
**Why define formal benchmark sets (so-called Verification Test Suites)? See Oberkampf et al 2002, 2004 for some discussion.**

**Notes:**
Bob Webster’s talk at NECDC 2006 grappled with SQE – and supported its presence.

## Verification is hard! Convergence studies:

**Work at the state of the art is illustrated by the LANL 2005 Level II verification milestone.**
**Explored convergence error estimation both in asymptotic and non-asymptotic regimes, as well as other work.**
**Heavily documented.**

**Notes:**
This milestone did a lot more than worry about convergence. But it was also a very significant convergence effort. It is a Poster Child for a verification study.

## Verification is hard! A posteriori error estimates:

**Remains a “Holy Grail” effort for multi-material compressible flow hydrodynamics coupled to multiple and multiscale energy transport mechanisms and material descriptions.**
**E.g. need error **_**estimates**_**, not error **_**indicators**_**, for non-genuinely hyperbolic systems, including mixed hyperbolic-parabolic systems.**
**See, for example, Fuentes et al (2006) [Oden’s “goal-oriented error estimation program”] for current initial work on transient, nonlinear problems.**
**Ongoing debate about how long the legs on this program are for NECDC-type problems. ******
**Adaptive mesh refinement (AMR) doesn’t necessarily provide error estimates (error indicators versus error estimates).**
**Nor does it eliminate the need for verification.**

**Notes:**
I don’t believe a posteriori error estimation is the ideal long-term solution. There are too many difficulties and irregularities. Oden and I disagree on this.

## Verification is hard! Probabilistic error models:

**Glimm and colleagues: treatment of numerical error as an uncertainty (incomplete knowledge).**
**Probability used to quantify error models.**
**See Yu et al (2006) for latest published example.**
**I believe this (type of) work has great importance for the long run. For one thing, it is compatible with QMU.**

**Imploding shock wave on perturbed material interface.**

**Standard deviation of probabilistically interpreted interface position error as function of time, different meshes.**

**Time**

**Notes:**
I think this work is the future.
This is especially important because it connects to the needs of QMU.
I think JOWOG 42 provides a significant opportunity to explore these issues.

## Validation is hard!

**Uncertainty of both the calculation and the experimental data referent is dominant in validation.**

**Assume calculations are converged, say their error bar is the size of the plot symbol.**
**What does the comparison mean? **
**THERE ARE NO EXPERIMENTAL ERROR BARS**** (i.e. experimental uncertainty quantification).**
**THINK QMU.**

**“****Experience and instinct are poor substitutes for careful analysis of uncertainty.” **

**Notes:**
The quote is taken from the report of the NASA “Return to Flight” Task Group, as given by Mehta.
And if we had code error bars, what does the comparison mean? No error bars in the experimental data.

## Validation lies at the heart of “predictive codes”

**Experimental uncertainty (variability, bias, diagnostic fidelity) is remarkably hard to quantify.**
_**Quantitative**_** expt-calc differences are uncertain quantities ***
**We aren’t converging calculations yet (10 years later).**
**What ARE the calculation error bars? Why would anybody believe the reported value? (VERIFICATION IS CRITICAL)**
**How much physics do you need?**
**How much agreement is good enough?**

**EXPERIMENT**

**CALCULATION**

**In principle, a **_**simple strategy**_**: **
**“****Converge” the calculation.**
**Put in enough physics to insure “agreement” of calculation and experiment.**
**This is the 1995 charter of ASCI.**

## Validation is hard!Technically speaking, validation is -

*** Characterization of (say) the high-dimensional random field**
**Diff**** = “****Nature**** – ****Calculation****”**
**Given relatively sparse information**
**For the purpose of making a reliability statement about “****Calculation****”**
**This interpretation has historical leverage in the atmospheric sciences:**
**For example, see Jolliffe and Stephenson (2003), **_**Forecast Verification**_**; Wilks (1995), **_**Statistical Methods in the Atmospheric Sciences**_

**Notes:**
The point is: the stochastic field concept is actively addressed in forecast verification in the climate sciences
Kennedy-O’Hagan evolve a line of thought about uncertainty in calibration, in this case a particular Bayesian approach. But the line of thought goes back to the 1960’s, and is articulated for example in Leamer’s book. There is a large literature on this. We are interested in this because of the correlation with confidence in prediction from calibrated models.
The Trucano et al ref is “Read at your risk!” Not a good paper, really, but at least some concepts are thrown out.

## Validation is hard!Have we detected a trend?

**	****Where is the numerical accuracy estimation and experimental uncertainty quantification in these kinds of comparisons?**
**Re: the “Mystery Calculation” Rogue’s Gallery**

## Validation is hard!Summary of Rogue’s Gallery:

**Little or no information about V&V**
**Reported “V&V” has little or no formality**
**Experimental data have little or no quantified uncertainty**
**Little or no discussion of computational error**
**Confusion of robustness with respect to a different grid with “small numerical errors”**
**Comparison with experiment to claim small numerical errors**
**Viewgraph norms and spaghetti plots for validation**
**Information inadequate to repeat calculations**
**Information inadequate to repeat experiments (or model them with other calculations**
**Confusion of calibration and validation**
**Archaic or non-existent editorial policy for computations**
**...**
**HOW ARE THESE ARTICLES PEER REVIEWED?**
**This is a challenge for the **_**CS&E profession**_**, not just V&V weenies!**

## Viewgraph norm to quantitative differences

**	****Tieszen, et al (2005), “Validation of a Simple Turbulence Model Suitable for Closure of Temporally-Filtered Navier-Stokes Equations Using a Helium Plume,” SAND2005-3210.**

**Notes:**
Statistical analysis of the experimental data to get the confidence intervals.

## Validation is hard!Validation Metrics

**Our formal engagement with this topic goes back to 2001.**
**See Oberkampf and Barone (2006) for a recent summary of principles.**
**“****Metrics” are really metrics, but the general topic has to do with rigorous methods for quantifying ****Diff**** and drawing rigorous conclusions about predictive capability (per the ASC mission).**
**Many benefits to thinking rigorously, not least of which is strong clarification of the difference between calibration and validation (Trucano et al, 2006), which is **_**especially**_** important in prediction.**

**Estimated computational error bars disjoint from experimental uncertainty confidence intervals – target for future validation experiments.**

## Credibility for WHAT?SSP aka QMU

_**Will our calculations be used properly?**_
**More than code developers and users need to believe ASC M&S **_**and**_** understand why it is believable.**
**Best Estimate Plus Uncertainty **
**See Pilch et al (2006)**
**How much V&V is enough?**
**“****Sufficiency” ******** “Predictive Capability Maturity Model”**
**Judgment replaced? **
**Do codes replace humans?**
**Are codes certified?**

## Decisions are hard!Remember: ASC is “predictive” computational science.

**	****DO PREDICTIVE M&S!**

**Reliability Dimension**
**What can happen?**
**How likely is it?**
**What are the likely consequences if it does happen?**

**Confidence Dimension**
**What is your confidence in predicting the answers to the three questions? **

**Use the science and experience of high-consequence system design/performance assessment in rigorous decision environments.**

**“****Risk”-Informed Decision Making**

**	****Foundation = V&V**

**Large-scale computational simulations supplement or replace physical experiments and tests for stockpile stewardship.**

**What does “predictive” mean?**

**Notes:**
QMU is driven by SSP needs, even though there is not strict technical agreement about (1) what it means; (2) how to do it; (3) how success is defined.
For example, this is an important topic of consideration for the ASC Predictive Science Panel, and they don’t fully understand it. Is it a philosophy? A science? Is it a deliverable? A means to an end?
(QMU has been in the U. California contract for LANL/LLNL for years!)
LANL and LLNL have different wrinkles, but a firm, common approach has not been documented and implemented
SNL’s approach is based on past high-consequence risk-informed decision support activities for NRC (reactor safety assessment) and EPA (waste repository performance assessment, WIPP and Yucca Mountain). SNL has decades of experience in this kind of systems engineering and highly rigorous assessment.
Computation either supplements or replaces physical experimentation and testing in stockpile stewardship (exactly like it does in NRC/EPA work). This places prominence on understanding the limits of credibility of the computation. This is a problem of verification and validation.
Our applications are not “Regulation” – they are technical performance assessments.

## Decisions  are hard!What do we mean by “predictive”?

_**Predictability**_** – A technical concept, conventionally arising in the consideration of complex systems. I.e. as in “**_**predict**_** the stability of the solar system” or “**_**predict**_** the evolution of a chaotic system.”**
_**Predictive Science**_** – might just as well be a philosophical hope in the progress of the human condition. How do you measure it?**
_**Predictive Capability**_** – in particular a **_**computational**_** **_**capability**_** with some (rigorous?) basis for credible interpolation or extrapolation of current knowledge, for example existing experimental data.**

**We (ASC) believe that “predictive capability” can be measured, although such capability is **_**always relative to the intended application**_**.**

**“****Predictability” versus “Predictive Science” versus “Predictive Capability”**

**Notes:**
“Predictive Capability” is the only thing we care about (ASC Mission).

## What do we mean by “predictive”?

**M&S typically informs **_**decision making under uncertainty**_**.**
**ASC World: “Quantification of Margins and Uncertainty” (QMU)**
**Technical performance margins for engineered systems**
**Uncertainty in the underlying information and characterization of margins**
**Decisions required that reflect this uncertainty**
**Many complex factors enter into using M&S in a complex technical endeavor, like Stockpile Stewardship (or climate warming policy).**
**Our bottom line: Produce, communicate, and use M&S in the form of:**
**Best Estimate Plus Uncertainty**

## Example of BE+U: “Rivers of Blood”

**Inflation projections from the Bank of England (February 2005 Inflation Report)**
**Hendry: “Surprisingly, reporting of forecasts alone was the norm for the Bank, even until relatively recently; and it is still the norm among many forecasters.” [Hendry and Ericsson, **_**Understanding Economic Forecasts**_**, MIT, 2001]**

## “Rivers of Blood” – note missing elements

**Where is the comparison of observation with prediction?**

**Uncertainty in the observation**
**Comparison of observation with forecast (overlay past RoB)**

**Uncertainty in the uncertainty**

## Decisions are hard!V&V Sufficiency – How much is enough?

**Two options:**
**Keep going until you run out of money or until management can’t take it anymore. ****YUK! ******
**Come up with a constructive basis for assessing sufficiency.**
**The latter is inevitably tied to the application and the associated decisions, that is QMU.**
**Sufficiency raises challenges of **_**accumulation**_**, **_**communication**_** and **_**preservation**_** of information.**

**Notes:**
V&V contributes to computational margin reduction (i.e. confidence in the computational results) . No wonder V&V cares about QMU. How much margin is needed implies something about how much V&V is needed.

## Predictive Capability Maturity Model – LANL Style!

| Predictive Attribute [SNL] | Lowest
[Exploratory] | Low
[Design] | High
[Qualification] | Highest
[Certified] |
| --- | --- | --- | --- | --- |
| Simulation Geometry Fidelity | Cartoon-ish | Better | Better again | Solution time / resolution balance at will |
| Physics Model & Alg Fidelity | Ad-hoc | Calibrated / low fidelity | Physics-based / higher fidelity | Validated / algs span application |
| Model Integration in Code | Stand-alone | Limited testing | Achieved; validation | Robustness demonstrated |
| Solution Verification | Judgment only | Numerical sensitivities | Convergence, numerical errors quantified | Verification test suites created, automated |
| NTS Data Validation/QMU | Not applied | Variable calibration, less validation | Global calibration, validation | No ad-hoc calibrations, systematic val |
| Impact on codes, knob removal | Ignored | Makes little difference | Clear improvement | Designer adoption |

**Notes:**
This is an adaptation by LANL of an idea developed at SNL during 2005-2006. The idea is in progress, but gaining acceptance.

## Decisions  PCMM  Sufficiency

**Decisions**

**Risk Tolerance**

**Notes:**
This is an adaptation by LANL of an idea developed at SNL during 2005-2006. The idea is in progress, but gaining acceptance.

## Are we in the business of replacing humans or their judgment? NO!

**But what are the requirements on the codes for the SSP and the future evolution of the stockpile in a no-nuclear-test environment?**
**Designers are “certified” from a variety of perspectives, from explicit training to the tacit knowledge they embody.**
**No code will ever replace the explicit and tacit knowledge of a designer.**
**But must future codes be certified?**
**Codes **_**are**_** certified right now! – through designer use (and WILLINGNESS to use).**

**“****Over my dead body ...” **

**Hint: not a designer talking about using an ASC code.**

**Notes:**
Quote – Bruce Goodwin, April, 2006, in reference to a point made by the Predictive Science Panel to consider “qualifying” or “certifying” ASC codes.
It is reasonable that Goodwin might say this – he’s arguing for the role of humans in stockpile stewardship.
NONETHELESS ...

## Should we be constructing “Black Box” software systems?

**	****If “Users” don’t understand ASC codes they shouldn’t be using them.**

**Notes:**
Quote – Bruce Goodwin, April, 2006, in reference to a point made by the Predictive Science Panel to consider “qualifying” or “certifying” ASC codes.
It is reasonable that Goodwin might say this – he’s arguing for the role of humans in stockpile stewardship.
NONETHELESS ...

## A National “Community of Practice”?(Beyond V&V)

**Where do we stand on defining and measuring the credibility of our work for important applications?**
**“****Ground-water models cannot be validated.”**
**How do we define, use and empower benchmarks?**
**“****My code passes more benchmarks than your code.”**
**Are “standards” needed? Appropriate? Frightening?**
**“****I feel free to ignore those DMSO Guidelines.”**
**Are journals helping or hurting?**
**“****Good enough for a journal does not imply good validation.”**
**Will we ever solve enough of the technical problems to make the above questions reasonable?**
**How can education help?**

**“****Ground-water models cannot be validated.”**

**“****My code passes more benchmarks than your code.”**

**“****I feel free to ignore those DMSO Guidelines.”**

**“****Good enough for a journal does not imply good validation.”**

**Notes:**
First quote – L. F. Konikow and J. D. Bredehoeft (1992), “Ground-water Models Cannot be Validated,” Advances in Water Resources, Volume 15, 75-83.
Second quote – notional
Third quote – notional
Fourth quote – Klein, Doebling, Graziani, Pilch, Trucano (2006), “ASC Predictive Science Academic Alliance Program Verification and Validation Whitepaper,” ASC program document.

## Example:

**Mehta**

**“****NASA and DOD could learn from ASC, NNSA.”**

**Notes:**
From Mehta

## My take-home lessons from the past 11 years of worrying about this are:

**Comparing calculations with experimental data has no obvious impact on the problem of estimating numerical errors.**
**There is no obvious benefit to be gained by comparing with experimental data that have undefined uncertainty.**
**Code comparisons have no clear relationship to V&V.**
**Confusing calibration and validation is dangerous in prediction.**
**V&V is a risk-management component for high-consequence decision making under uncertainty.**
**Social elements are important.**
**Absence of evidence that something is wrong is not evidence that something is right.**

**“****I’ve had sixteen fights. I won all but twelve of them...”**

**Notes:**
The quote is from a Woody Allen movie.

## In conclusion:

**V&V is a collaboration between code developers, experimenters, designers, people with specialized V&V knowledge (to the extent they exist) **_**and**_** decision makers.**
**Success or failure in V&V directly mirrors the success or failure of this collaboration.**

**“****Too hard ... Too slow ... Too expensive ...” ?**

**Notes:**
The quote reflects a general malaise at SNL regarding the conduct, progress, and cost effectiveness of formal V&V as advocated by the ASC V&V program.

## Key technical challenges:

**Numerical error quantification.**
**Quantifying epistemic uncertainty.**
**Validation metrics for high-complexity data sets (e.g. 4-D data depending on N uncertain parameters, N>>1, with aleatory and epistemic uncertainty separately accounted for).**
**Measuring predictive capability and progress in achieving it for the important applications. **
**Sufficiency – define and implement “What’s good enough.”**
**What is the best way to collaborate with the user community, in particular beta-users.**
**Supporting QMU, that is integrating decision concerns.**
**Where is the computing going to come from?**

**Notes:**
V&V program emphasis on BE+U is present in the first document written by the program – the V&V planning guidance
Risk-informed decision making has been a critical part of the foundation from the beginning
This is why the PIRT (Phenomena Identification and Ranking Table) has been given prominence; this is a NUREG methodology widely deployed in the revised reactor safety assessment activities of the 1980’s and 1990’s
BE+U is completely  relevant for “engineering design.” (Consider the challenges of MEMS.)
The question is really what is the appropriate “Level of Formality” to attach to the computational work that supports “engineering design.”

## A few references

**ASME (2006), “Guide for Verification and Validation in Computational Solid Mechanics,” to be published.**
**D. Fuentes et al. (2006), “Extensions of Goal-Oriented Error Estimation Methods to simulations of highly-nonlinear response of shock-loaded elastomer-reinforced structures,” Computational Methods in Applied Mechanics and Engineering, Volume 195, 4659-4680.**
**Klein, et al (2006), “ASC Predictive Science Academic Alliance Program Verification and Validation Whitepaper,” UCRL-TR-220342-Rev, to be released.**
**National Science Foundation (2006), “Simulation-Based Engineering Science: Revolutionizing Engineering Science through Simulation,” Report of the National Science Foundation Blue Ribbon Panel on Simulation-Based Engineering Science.**
**Oberkampf and Trucano (2002), “Verification and validation in computational fluid dynamics,” Progress in Aerospace Sciences, Vol. 38, No. 3, 209-272. (Review)**
**Oberkampf, Trucano, and Hirsch (2004), “Verification, validation, and predictive capability in computational engineering and physics,” Applied Mechanics Reviews, Vol. 57, No. 5, 345-384. (Review)**
**W. L. Oberkampf and M. F. Barone (2006), “Measures of Agreement Between Computation and Experiment: Validation Metrics,” Journal of Computational Physics, Volume 217, 5-36.**
**Oberkampf and Trucano (2006), “Design of and Comparison with Verification and Validation Benchmarks,” SAND2006-5376C (International Workshop on “The Benchmarking of CFD Codes for Application to Nuclear Reactor Safety”**
**Pilch, et al (2000), “Guidelines for Sandia ASCI Verification and Validation Plans – Content and Format: Version 2.0,” SAND2000-3101.**
**Pilch et al (2006), “Ideas Underlying Quantification of Margins and Uncertainties (QMU): A White Paper,” SAND2006-5001.**
**C. J. Roy (2005), “Review of Code and Solution Verification Procedures for Computational Simulation,” Journal of Computational Physics, Volume 205, 131-156**
**Trucano et al (2001), “Description of the Sandia Validation Metrics Project,” SAND2001-1339.**
**Trucano et al (2002), “General Concepts for Experimental Validation of ASCI Code Applications,” SAND2002-0341.**
**Trucano et al (2003), “On the Role of Code Comparisons in Verification and Validation,” SAND2003-2752.**
**Trucano et al (2006), “Calibration, validation, and sensitivity analysis: What’s what,” Reliability Engineering and System Safety, Volume 91, 1331-1357**
**Y. Yu et al. (2006), “Uncertainty Quantification for Chaotic Computational Fluid Dynamics,” Journal of Computational Physics, Volume 217, 200-216.**