# Presentation: 226350

## Bayesian Analysis of the Thermal Challenge Problem

- F. Liu1,  M.J. Bayarri2,  J. O. Berger1,  R. Paulo3,
- J. Sacks4
- 1Duke, 2Valencia, 3Bristol, 4National Institute of Statistical Sciences
- May 22, 2006
- Sandia Laboratories
- Albuquerque, NM

## Formulation

- Reality = Code Output + Bias
- Field = Reality + Error
- [Assumption in Challenge: Error = 0]
*	**i = *1,..., *n**expt** *where *n**expt* is the number of observations in a particular experiment for fixed inputs *x,L,q*

## Strategy: Six Step (Statistical) Framework 

- 1. Specify code inputs and associated ranges
- 2. Determine evaluation criteria
- 	{*Steps 1 and 2 define the problem*}
- 3. Experimental strategy
- 	{*Includes design of data collection(s)*}
- 4. Approximation of functions
- 	{*Necessary to deal with limited data; use of GASP: Gaussian process priors on functions*}
- 5. Analysis
- 	{*Comparison of field and model by Bayesian analysis*}
- 6. Feedback
- 	{*Continuous improvement of the model*}

## Evaluation Criteria;Regulatory Requirement 

*Criteria*:  Is bias *b **≡** *0?  Is *b*(..., 1000) = 0?
*Regulatory Requirement*:

## Pure Model Prediction of Regulatory Requirement Left for medium level; pf=.08Right for high level; pf=.06

## Formulation (cont’d)

- Simplifying Assumption:
- Additional Simplification:

## Material Characterization (MC) Data

- Consider 2 levels: medium, high (contains added data)
- Use data (only for temperatures ≥ 500°) to estimate *κ**, **ρ** *:
- assume *κ**, **ρ** *are independent with *κ** ~ N*(*μ**κ** ̧**σ**κ*), *ρ** ~ N*(*μ**ρ** ̧**σ**ρ*) and non-informative priors π (*μ**κ*) = 1 and π(*σ**κ*) = 1/ *σ**κ*and same for *ρ*

## “Ensemble” Experiment (EE) Data

- Use data to estimate bias *b; *predict at accreditation configuration
**EE** data are for *x* = 0 and for the following design (configuration) for *L, q*

## Bayes Analysis

- Bayes Theorem:
- Use MCMC methods to produce a sample from *π**post* from which everything needed is obtained

## Bayes Analysis:Prior Distributions on Unknowns

*π*(*b*): Gaussian Process (GP) on functions of *z=*(*x,L,q,t*)

## “Explanation”

- Think of* b *as a random function, a realization of the GP
- When *z ~ z**′  Corr* (*b*(*z*),*b*(*z*′)) is high so *b*(*z*) ~ *b*(*z*′)
- When *z *far from* z**′  Corr* (*b*(*z*),*b*(*z*′)) is low so *b*(*z*) and *b*(*z*′) are not “connected”
- Degree of ~ (smoothness) affected by *α*’s (*α** *=2, infinitely differentiable)
- Scale determined by *β*’s

## Bayes Analysis: EE Data

- For **EE **data run MCMC for (simplified) model (2) and α’s=2
- Use the MCMC samples to obtain
- Run MCMC on model (1) with fixed
- and get samples

## πpost(b)  Lt:  L=.0127, q=1000; Rt:  L=.0127,   q=2000 Medium-level Data

## Prediction of Accreditation Data (AC) at x=0

- For predicting at new setting (the **AC **configuration) from the **EE **analysis there is a new *κ**new**, **ρ**new* with posterior distribution| **MC** Data found earlier
- To the MCMC samples above adjoin                  drawn from the posterior distribution of *κ**new**, **ρ**new*
- Adjoin draws from the distribution of  *b*(0, *L=*.019, *q=*3000,·) | ***b******EE****, ...* (***b******EE****, *-- bias functions at the four configurations of **EE)**

## Prediction of AC Data at x=0Lt: Medium-level; Rt: High-levelBlue=prediction; dashed=80% (10% below; 10% above) boundsRed = Accreditation dataGreen = Plug-in pure model prediction

## Bayesian Analysis: EE +AC Data

- Use *x* = 0 data only
- Follow same strategy as for **EE **analysis (“get *β*’s, ξ first”)

## Prediction at Regulatory Requirement

- For predicting at regulatory** **configuration) from the **EE**+**AC **analysis use new *κ**new**, **ρ**new* with posterior distribution| **MC** Data found earlier
- To the MCMC samples from **EE**+**AC **analysis above adjoin                  drawn from the posterior distribution of *κ**new**, **ρ**new*
- Adjoin draws from the distribution of  *b*(0, *L=*.019, *q=*3500,·) | ***b******EE+AC****, ...* (***b******EE+AC****, *-- bias functions at the five configurations of **EE+AC)**
- Produce sample prediction as

## Prediction at Regulatory Requirement: blue = bias-corrected prediction with 80% boundsred = pure model prediction

## Regulatory Results: Lt: medium-level  pf  ~ .02; Rt: high-level pf  ~ .01

## Bayesian Analysis 2 (1)Run MCMC assuming b = 0; (2)  run full MCMC but sample κi, ρi from their posteriors from (1)

## Prediction at Regulatory Requirement: pf  ~ .03 blue = bias-corrected prediction with 80% boundsgreen = pure model prediction

## Conclusions

- Model is somewhat imperfect (bias appears)
- Bias-correction suggests (marginally) that device may pass regulatory requirement
- Variability in *κ**, **ρ** *is substantial
- Constancy of *κ**, **ρ** *is not supportable: improve the model

## The “Specific Tasking” Questions

- 	The statistical strategy (**SAVE = Simulator Assessment and Validation Engine) **described takes into account all the issues raised and provides the essential uncertainty bounds

## References

- Bayarri, Berger, Higdon, Kennedy, Kottas, Paulo, Sacks, Cafeo, Cavendish, Lin, Tu. (2002).“A Framework for Validation of Computer Models”. In D. Pace, S. Stevenson, eds.,  *Proceedings of the Workshop on Foundations for V&V in the 21st Century. *Society for Computer Simulation. *NISS Technical Report 128, *_[www.niss.org/downloadabletechreports.html](http://www.niss.org/downloadabletechreports.html)_
- Kennedy, O’Hagan (2001). “Bayesian Calibration of Computer Models.” *JRSS B* 425-464