Typical Characteristics of a \"Regulatory Problem\" in Engineering

Prediction for the Application Requires Extrapolation.

Bottom-Up Validation Structure. A validation structure involving
(generally) several validation steps and, often, an accreditation step
encourages the modeling of more complex physics as the process
progresses.

Required Predictions, Involve the Tails of the Response Distribution.
Very often requirements are to demonstrate an event has very low
probability.

A Complete Specification is given for a Scenario to Demonstrate
Regulatory Compliance. This reduces the need for optimization, and/or
reliability analysis to try to characterize the application through a
sub-region of the test space (see the Problem Space diagram Figure #1
below). For the challenge problems:

> [Thermal Problem]{.underline}: for a specific length and flux
> determine whether or not, at time 1000 sec, the surface temperature
> will exceed a failure temperature 900º *C* with probability less than
> 10^-2^.
>
> [Mechanics Problem]{.underline}: for a specific structure determine
> what can be said about the probability that the displacement in the
> midpoint of beam 4 will not exceed 3 mm.
>
> [Structural Dynamics Problem]{.underline}: for specific structure
> determine whether or not, the max absolute acceleration of mass 3 will
> exceed 18000 in/sec with probability less than 10^-2^.

![](media/image1.jpeg){width="3.879861111111111in"
height="2.9097222222222223in"}

Figure #1 **The Problem Space**. The axes are symbolic, indicating a
range in these two elements (inputs) of the problem space. The axes are
generally multi-dimensional and not necessarily based on a real number
coordinate system. For the [mechanics problem]{.underline}, for example,
the system characteristics axis includes a component for system
structure, indicating one of several specific structures to be
considered in the analysis.

Issues and Methods in Model Calibration

Selection of Model Parameters is largely a part of the model development
effort, but calibration is critical to making the model address the
specific concerns of an application. Below is a list of the model
parameters used in the challenge problems. *X* and *Y* are inputs (see
Figure #1) and  specifies the model parameters.

[Thermal Problem]{.underline}

*X =* {*thickness, L*}

*Y =* {*flux, q: location and magnitude*}

 *=* {*heat capacity,* *C~p.~ thermal conductivity, k*}

[Mechanics Problem]{.underline}

*X =* {*system physical geometry: lengths; connections*}

*Y =* {*force, F: location; direction and magnitude*}

 *=* {*modulus of elasticity E and parameters of its spatial
distribution*}

[Structural Dynamics Problem]{.underline}

*X =* {*system physical geometry: lengths; connections*}

*Y =* {*input signal characterization*}

 *=* {*modal parameters: frequency* *~i~; damping* *~i~ and shape*
*~i~, i=1, 2, 3*}

Model parameters May Be Fixed or specified as functions of the inputs,
or (possibly) as functions of other state variables that may include the
response. For the thermal problem it appears that thermal conductivity
depends on temperature, for example, and consequently different
parameterizations may be appropriate depending on the thermal range
anticipated.

Model parameters should be conditioned, as much as possible, for the
specific application. In the structural dynamics problem, for example,
parameter estimates are provided for accelerations resulting from
excitation of low, medium and high intensity. Some of the parameters
show a significant dependence on the intensity. When the remainder of
the analysis involves only of high excitation level, than the
calibration can be conditioned on these data.

Model parameters can be re-parameterized as more data become available.
Care should be taken to make sure the data sets are parameterized
adequately using the same characterization. In the structural dynamics
problem, for example, model parameter estimates based on random
vibration data appear to be different than the estimates based on
shocked data.

Methods for Calibration of Model Parameters **(Ref 1)**

Model parameters may be characterized to include uncertainty and/or
variability. The methods used in calibration may depend on this issue.
The same application may have both types of parameters \-- uncertain
constants and variable material properties, for example.

> **Variability** (primarily)

-   "Bootstrap" using the calibration data directly **(Ref 2)**

-   Nonparametric distribution fit to the calibration data (used for
    [thermal and structural dynamics problems]{.underline})

```{=html}
<!-- -->
```
-   Parametric distribution fit to the calibration data

> **Uncertainty** (primarily)

-   Generalized likelihood **(Ref 3)** (used for the [mechanical
    problem]{.underline})

-   Bayesian parameter modeling

-   Bayesian computational modeling or forecasting **(Refs 4, 5, 6)**

Model parameters may be reduced in number by using principal components
analysis. (used for [structural dynamics problem]{.underline}).

Model parameter characterization may be determined through expert
judgment

Parameters, modeling epistemic uncertainty, may be characterized using
alternative (to those based on probability theory) representations
**(Ref 7)**.

Issues and Methods in Model Validation

The validation metric is generally a statistic based on both
computational and experimental data. Ideally, it is selected to focus
the comparison of these two sets of data on elements of the response
that are important to the application. For example, when applied to a
regulatory problem involving demonstrating a low probability event, it
is more important to focus the comparison on the tails of the
computational and experimental response distributions rather than
compare their mean values.

The model validation process is not generally a conservative process.
The validation metric (like most statistics) has an associated "power"
function or curve indicating how likely the comparison is to identify
differences between the computational and experimental responses if
there are true differences.

![](media/image2.jpeg){width="3.7194444444444446in"
height="2.797222222222222in"}

If one is willing to set a threshold for the validation metric above (or
below) which they will take action than a power curve can be constructed
(the action may be: to do further development work on the model; to
perform further experimentation; to do supplemental modeling; or to make
the prediction in another way). The processes associated with the
computational analysis are very often lacking in data, both experimental
and computational. Consequently, it may be the case that only large true
differences will be identified with reasonably high probability. This
information should be taken into account in the decision-making process.

The power curve above is somewhat of an oversimplification of the issue
in that the true difference axis is generally quite complex and must be
used to address very specific differences. Fortunately if there are
specific true differences that one wishes to investigate, the power
curve can usually be constructed using a numerical approach.

A General Approach to Comparing Computational and Experimental Responses

In an effort to focus the validation comparison on aspects of the
response that are important to the application, the following approach
was taken for all problems. It is illustrated in detail for the
[structural dynamics problem]{.underline}.

**[For a scalar response]{.underline}**

-   Step #1 -- Construct a response CDF based on the model results.

-   Step #2 -- Mimic the experimental design, (sample size and
    construction) that will be used (or was used) for the experiments
    with samples generated using the model CDF.

-   Step #3 -- Compute the empirical CDF for the sample in Step #2 and
    evaluate the 'metric' *M* comparing this empirical CDF to the model
    CDF.

> Repeat Steps #2 and #3 a large number of times and calculate the
> empirical distribution of the metric values, say E~m~

-   Step #4 -- Perform the experiments. Use the responses to compute the
    empirical experimental CDF and calculate the metric value, comparing
    it with the model CDF. If the metric value falls above a specified
    percentile on the distribution of E~m~, then the model has failed
    this test at that 'level'.

**[For a functional response]{.underline}**, the method is only slightly
more involved. One approach consists of replacing Step #1 with

-   Step \# 1' - Evaluate the principal components associated with the
    model responses and use these to generate the empirical CDFs. There
    would be one CDF for each principal component retained

> The remaining steps are similar. *M* is now a metric in
> multi-dimensional space, perhaps weighting the different dimensions to
> reflect their relative contribution to differences between responses
> as computed in the principal component analysis.

The idea is that we first characterized the model responses and sampled
using that characterization to see how a limited number of experiments
would perform if they followed that characterization exactly. Next, the
set of physical experiments is performed and their responses are
compared to the model characterization. In the case of a valid model the
responses should be close.

Issues and Methods in Model-Based Prediction

Model based prediction in regulatory problems often involves a leap of
faith because of the extrapolation associated with predictions made in
the application space. Even in cases where the model has performed well
at all other stages of the computational analysis, it is difficult to
assess the additional uncertainty that accompanies extrapolation.

When problems with the model have been identified, but the analysis is
to proceed, there are several alternatives available for supplementing
the model predictions and adjusting the modeling uncertainty. Two of the
more common approaches are 1) to adjust the model predictions through
the recalibration of model parameters and 2) to adjust the model
predictions through supplemental modeling. Each of these alternatives
has pros and cons (mainly cons) listed below.

Recalibration of model parameters, to better fit the data:

-   may distort the meaning of the parameter;

-   can lead to \"over-fitting\";

-   will generally distort the estimate of parameter uncertainty;

-   does not lead to an estimate of additional parameter uncertainty.

Supplemental empirical modeling:

-   has a difficult time modeling extrapolation when the input axes are
    not real variables. \[One approach around this is demonstrated with
    the [thermal problem]{.underline}\];

-   uncertainty estimates may be available but are rough.

Alternatives that combine these approaches are available **(Refs 4,
5).**

Thermal Problem

![](media/image3.jpeg){width="4.039583333333334in"
height="3.029861111111111in"}

[Thermal Problem]{.underline}

*X =* {*thickness, L*}

*Y =* {*flux, q: location and magnitude*}

 *=* {*heat capacity,* *C~p.~ thermal conductivity, k*}

**Solution Summary**:

Model parameter selection. Model parameters were specified in the
problem description. The parameters consisted of thermal conductivity
and heat capacity with 30 data provided through six calibration tests
performed at each of five temperatures. The first figure, that follows
illustrates the data.

![](media/image4.wmf){width="2.498611111111111in" height="2.125in"}
![](media/image5.jpeg){width="2.629861111111111in"
height="2.167361111111111in"}

Model Parameters Nonparametric Parameter Density Estimate

Model calibration. The different symbols and the first figure above,
indicate different temperatures, increasing left to right from 20° C to
1000° C. Clearly there is a dependency of the thermal conductivity
values on temperature. When these data were used to estimate the model
parameter distribution, only data at 750° C. and 1000° C. were used
because these values appeared most critical to later experiments and the
application. The second figure above shows the kernel density estimate
to the model parameter distribution. The nonparametric approach was
taken, because the data seemed inappropriate for the parametric models
considered.

To generate parameter values to be used for the validation experiments a
Latin hypercube sampling **(Ref 8)** process was applied to these
nonparametric distributions. This was followed by a process inducing a
rank correlation of zero **(Ref 9)** for the sampled parameters.

Validation tests. The means of running the model for validation
experiments was provided with the problem description. The validation
space for this problem, considered four specific sets of input
(configurations). Twenty computational runs were performed to provide
the validation data \-- 5 at each configuration. The responses over time
are plotted in the figures below for configurations 1 and 4 followed by
plots of the differences between experimental and computational
responses. The differences are computed between each pair.

Note that there are differences for each configuration, but that the
nature of the differences is different as well. Largely for this reason,
we considered it important to try to compare the entire response over
time. If extrapolation is necessary, the entire "trajectory" provides
information that may aid in supplemental modeling. The analysis was
performed using principal components of the functional data **(Ref
10).**

The formal validation analysis was performed using the functional
version of the general algorithm described above. According to this
validation testing, there were significant differences between
computational and experimental responses.

![](media/image6.wmf){width="2.623611111111111in"
height="1.8854166666666667in"}
![](media/image7.wmf){width="2.623611111111111in"
height="1.8847222222222222in"}

> Computational and Experimental Responses

![](media/image8.wmf){width="2.622916666666667in"
height="1.8944444444444444in"}
![](media/image9.wmf){width="2.623611111111111in"
height="1.917361111111111in"}

Differences Between Computational and Experimental Responses

Model Based Prediction. In order to compensate for the model
discrepancy, supplementary modeling was performed on the difference
trajectories as a function of length and flux. Each principal component
of this difference was modeled using a bilinear model, and the
uncertainty in this model was calculated and included. The first figure
below shows predictions for the accreditation problem. The green curve
represents the prediction based on the model alone; the black curve
represents the prediction based on the supplemented model. The two blue
curves represent the experimental results for the accreditation problem.
It appears (unfortunately), in this case, that the supplemented
prediction is farther from the experimental results than the model based
protection. The second figure below shows the bounds associated with
these predictions. The yellow curves come from 100 simulated results.
The black curve is the prediction of the supplemented model and the blue
curves indicate 90% bounds. The green bounds show the modeling
uncertainty, based on the un-supplemented model. Note that the upper
bound is below one of the experimental curves.

![](media/image10.jpeg){width="2.5in" height="1.875in"}
![](media/image11.jpeg){width="2.5in" height="1.875in"}

Accreditation Test Responses, Predictions and Bounds

The curves that follow provide similar information for the application
scenario. The first figure shows predictions by both the supplemented
and un-supplemented model. Bounds are provided in the second figure
below.

![](media/image12.jpeg){width="2.5in" height="1.875in"}
![](media/image13.jpeg){width="2.5in" height="1.875in"}

> Application Predictions and Bounds

The results suggest that the regulatory requirements cannot be assured
based on the present analysis. Uncertainty added the model prediction to
accommodate the extrapolation and to accommodate the model supplement
term inflate the bounds leaving substantial probability that responses
will exceed 900° C.

Mechanics Problem

![](media/image14.jpeg){width="4.129861111111111in"
height="3.0972222222222223in"}

[Mechanics Problem]{.underline}

*X =* {*system physical geometry: lengths; connections*}

*Y =* {*force, F: location; direction and magnitude*}

 *=* {*modulus of elasticity E and parameters of its spatial
distribution*}

**Solution summary**:

Model parameter selection. The parameters needed to accommodate this
problem were those necessary to characterize the material properties of
the beam. The problem description, indicated that the beam did not have
constant modulus of elasticity and consequently parameters were required
to describe a stochastic process. In the present approach, we used a
Gaussian process with correlation function:

$C(x,x') = {\lambda e}^{- \varphi|x - x'|^{1\text{.}5}}$ for any **x**
and **x**' points along the length of the beam. This leaves the two
parameters and λ to be estimated.

Model calibration. The parameters that determine the Gaussian process
are unknown constants in the present analysis, so the calibration effort
was focused on evaluating parameters that led to responses similar to
those in the calibration data set. Approximately 50 model runs were
required to find combinations of the above parameters (and a third
parameter) that appeared to match the calibration data reasonably well.
We were unable to find values of and λ that produced both the
distribution of midpoint values and the distribution of elongation
values found in the calibration data set. The third parameter added was
a "tie down" parameter that conditioned the modulus of elasticity values
found in the calibration data. The figures below give examples of the
two processes that were retained. It is likely that if more model runs
were performed alternative processes that matched the calibration data
could be obtained.

.![](media/image15.jpeg){width="2.459722222222222in"
height="1.8444444444444446in"} ![](media/image16.jpeg){width="2.5in"
height="1.875in"}

(To be Continued)

Structural Dynamics Problem

![](media/image17.jpeg){width="3.3in" height="2.475in"}

Structural Dynamics Problem

*X =* {*system physical geometry: lengths; connections*}

*Y =* {*input signal characterization*}

 *=* {*modal parameters: frequency* *~i~; damping* *~i~ and shape*
*~i~, i=1, 2, 3*}

**Solution Summary**:

Model parameter selection. Model parameters were suggested in the
problem description. The parameters consisted of three sets of modal
parameters, consisting of a frequency and damping parameter and a shape
factor; a total of 15 parameters. In order to simplify the calibration
process principal components analysis was used to reduce the number of
parameters. The first figure that follow shows the variability in the 15
parameters after standardization. The second figure illustrates the six
principal components that account for almost all (99%) of their
variability.

![](media/image18.jpeg){width="2.5in" height="1.875in"}
![](media/image19.jpeg){width="2.5in" height="1.875in"}

> Model Parameters Principal Components

Model calibration. The parameters were calibrated using only high-level
excitation calibration data provided. Only the 'high' subset of the data
was used, because as the first figure below illustrates, there can be
quite substantial differences in the parameter is depending on the
excitation level. Since the excitation levels of the accreditation and
application scenarios are high, only the high values were used for
calibration. Nonparametric distributions were constructed for
coefficients of the principal components.

To generate parameter values to be used with the forces provided for the
validation experiments a Latin hypercube sampling **(Ref 1)** process
was applied to these nonparametric distributions. This was followed by a
process inducing a rank correlation of zero **(Ref 1)** for the sampled
parameters. The resulting parameter vectors are illustrated together
with the experimental vectors, after standardization, in the second
figure below.

![](media/image20.jpeg){width="2.5in" height="1.875in"}
![](media/image21.jpeg){width="2.5in" height="1.875in"}

> Calibration Data Generated, Experimental Parameter Values

Validation tests. The means of running the model for validation
experiments was provided with the problem description. The force vectors
used in the experiments were used for the simulations as well. Twenty
computational runs were performed to provide the validation data.
Nonparametric density estimates for the maximum absolute accelerations
are shown in the first figure below (green - computational; blue -
experimental). A formal validation analysis, following the algorithm
described above is described next.

![](media/image22.jpeg){width="2.5in" height="1.875in"}
![](media/image23.jpeg){width="2.5in" height="1.875in"}

Nonparametric Density Estimates CDF Of Metric Values

-   Step #1 -- the CDF is computed from the model parameter density
    estimate above (shown as the green CDF in the second figure above).

-   Step #2 -- sets of 20 (the number of the random experiments) sample
    accelerations (500 sets) were generated at random using the model
    CDF above.

-   Step #3 -- the empirical CDF for the generated computations in step
    #2 was computed (no model runs are made at this step; values are
    just taken from the CDF at random) and evaluated using the metric:
    \[average absolute discrepancy with the model CDF across the last
    fifth of the CDF range\]. The choice of metric here is somewhat
    arbitrary, but the intent is to focus the comparison on the upper
    tail of the distribution.

-   Step #4 -- the experimental CDF is shown as the blue curve in the
    second figure above. According to the metric described in step #3,
    experimental CDF sits near the middle (p-value = .42). We conclude
    that the data do not provide evidence that the model is invalid.

Recalibration. The model appeared to perform as well as can be expected
in the validation step above. Because more data or are now available. It
makes sense to recalibrate the model parameters to accommodate this
additional information. The figure below shows the parameter estimates
for this recalibration. The dark curves are from the random vibration
testing of a light curves are from the shot tests.

![](media/image24.jpeg){width="2.5in" height="1.875in"}

Parameter Values For the Recalibration

Accreditation. The model appeared to perform as well as can be expected
in the validation step. The figures below provide similar information
for the accreditation tests and computations. The first figure, gives a
histogram a computational results and shows the experimental responses
indicated by the red stars. The results again, look quite good. Because
there are several data available is possible to perform a validation
test. The second figure below shows the CDFs for the computations and
the model. The formal validation procedure, gave a p-value of .43.

![](media/image25.jpeg){width="2.5in" height="1.875in"}
![](media/image26.jpeg){width="2.5in" height="1.875in"}

Histogram of Responses Computational and Experimental CDFs

Model Based Prediction. Aside from the disturbing segregation of
parameter values shown above resulting from the two different types of
excitation, both the validation and the accreditation step indicate that
the model is performing well. For this reason, the model alone was used
to make the application predictions; the results are shown in the
figures below. The first figure gets a histogram of the application
results. The second figure gives the CDF associated with the kernel
density estimate of the max-absolute-acceleration distribution

![](media/image27.jpeg){width="2.5in" height="1.875in"}
![](media/image28.jpeg){width="2.5in" height="1.875in"}

Histogram of Responses Application Response CDF

The results suggest that the regulatory requirements are not met for
this system. The modeling effort passed through the validation hurdles
quite well. One must keep in mind, the limitations illustrated through
the power curve (not shown here) of inferring model validity with so few
data.

**References**

1)  Campbell, K., (2002). "A Brief Survey of Statistical Model
    Calibration Ideas" Los Alamos National Laboratory, Tech Report \#
    LA-UR-02-3157.

2)  Efron, B. and Tibshirani, R., (1993). *An Introduction to the
    Bootstrap*, Chapman & Hall, New York.

3)  Beven, K. and Binley, A., (1992). "The Future of Distributed Models:
    Model Calibration and Uncertainty Prediction", *Hydrological
    Processes*, **6**, 279-298.

4)  Craig, P. S., Goldstein, M., Rougier, J. C., and Seheult, A. G.,
    (2001). "Bayesian Forecasting for Complex Systems Using Computer
    Simulators", *Journal of the American Statistical Association*,
    **96**, 717-729.

5)  Kennedy, M. C., and O'Hagan, A., (2002). "Bayesian Calibration of
    Computer Models (with discussion)", *Journal of the Royal
    Statistical Society (Series B)*, **58**, 425-464.

6)  Higdon, D., Kennedy, M., Cavendish, J., Cafeo, J., and Ryne R.,
    (2001). "Combining Field Data and Computer Simulations for
    Calibration and Prediction", *SIAM Journal on Scientific Computing,*
    **26**, 448-466.

7)  Helton, J. C. and Oberkampf, W. L., (guest editors) (2004). "Special
    Issue \-- Alternative Representations of Epistemic Uncertainty",
    Reliability Engineering & System Safety, **85**.

8)  McKay, M. D., Conover, W. J. and Beckman, R. J., (1979). "A
    Comparison of Three Methods for Selecting Values of Input Variables
    in the Analysis of Output from a Computer Code", *Technometrics*,
    Vol. **21**, No. 2, 239-245.

9)  Iman, R. L. and Conover, W. J., (1982). "A Distribution-Free
    Approach to Introducing Rank Correlation Among Input Variables",
    *Communications in Statistics, Part B - Simulation & Computation*,
    **11**, (3), 311-334.

10) Ramsay, J., O. and Silverman, B. W., (1997). *Functional Data
    Analysis*, Springer-Verlag, New York.