# Presentation: 485752

## Streamlining UncertaintyConceptual Model and Scenario Uncertainty

- FRAMES-2.0 Workshop
- U.S. Nuclear Regulatory Commission
- Bethesda, Maryland
- November 15-16, 2007
- Pacific Northwest National Laboratory
- Richland, Washington

## Model Applications

- Regulatory and design applications of hydrologic models of flow and contaminant transport often involve using the models to make predictions of future system behavior
- Performance assessment of new facilities (safety evaluation, environmental impact assessment)
- Monitoring network design for contaminant detection or performance monitoring
- License termination
- Design of a subsurface contaminant remediation system

## New Reactor Potential Model Applications

- Assessing effects of accidental releases on ground and surface waters
- groundwater flow pathways
- transport characteristics
- Assessing flood design bases
- Stream flooding
- Local flooding, site drainage
- Impacts of water use
- Watershed analysis – impacts on other users of water source upstream and downstream, particularly during drought conditions

## Framework for the Application of Hydrologic Models to Regulatory Decision Making 

**Framework for the Application of Hydrologic Models to Regulatory Decision Making **

- History Matching - reproduce observed behavior
- Demonstrate understanding of site behavior
- Provide confidence in use of models to support decisions
- Prediction – forecast future behavior
- Apply model results to decisions
- For risk-informed decision making, provide estimates of risk

- Model Development & Evaluation

- Model Application for Comparison
- with Regulatory or Design Criteria

**Notes:**
Element of risk is the uncertainty in model predictions

## Model Predictive Uncertainty Quantifies Element of Risk

**Model Predictive Uncertainty Quantifies Element of Risk**

- In general, uncertainty is assessed in the history-matching period and propagated into the predictive period
- Reduce these uncertainties by collecting additional data
- Some uncertainties only apply in the predictive period
- Irreducible characteristics of the system being modeled

**Notes:**
1st type: subjective, epistemic, variability; 2nd type: stochastic, aleatory, uncertainty

## Prediction Uncertainty

- Model conceptualization uncertainty
- A hypothesis about the behavior of the system being modeled and the relationships between the components of the system
- Each site is unique and heterogeneous/variable. Behavior typically involves complex processes. Site characterization data is limited.
- Assessed in history-matching period, applied in the predictive period
- Parameter uncertainty
- Model-specific quantities required to obtain a solution
- Measurement/sampling errors. Disparity among sampling, simulation, and actual scales of the system.
- Assessed in history-matching period, applied in the predictive period
- Scenario uncertainty
- Future state or condition that affects the hydrology
- Historical record not representative of future conditions – process variability, limited historical record, land/water use changes, climate change
- Applies to predictive period only

## Model Uncertainty

**Model Uncertainty**

- Common to rely on a single conceptual model of a system. This approach is inadequate when there are:
- different interpretations of data
- insufficient data to resolve differences between conceptualizations

**Notes:**
This is one of the key concepts of the project: failure to consider uncertainties in conceptual models can lead to underestimation of predictive uncertainty (and a resulting underestimation of risk), or to biased results when a key component of the system is unrepresented or poorly represented in the model. 

## Failure to Consider Model Uncertainty

**Failure to Consider Model Uncertainty**

- Has two potential pitfalls:
- rejection by omission of valid alternatives (underestimates uncertainty)
- reliance on an invalid model (produces biased results)

## Slide 9

| Hydrogeologic Model Application (Ref.) | Comments | Error |
| --- | --- | --- |
| Phoenix (Konikow 1986) | Assumed past groundwater pumping would continue in future | Scenario/
Conceptual |
| Cross Bar Ranch Wellfield (Stewart and Langevin 1999) | Assumed a 75-day, no-recharge scenario would represent long-term maximum drawdown | Scenario/
Conceptual |
| Arkansas Valley (Konikow and Person 1985) | Needed a longer period of calibration | Scenario/
Parameter |
| Coachella Valley (Konikow and Swain 1990) | Recharge events unanticipated | Scenario |
| INEL (Lewis and Goldstein 1982) | Dispersivities poorly estimated | Parameter |
| Milan Army Plant (Andersen and Lu 2003) | Extrapolated localized pump test results to larger area | Parameter |
| Blue River (Alley and Emery 1986) | Storativity poorly estimated | Parameter/
Conceptual |
| Houston (Jorgensen 1981) | Including subsidence in model improved predictions | Conceptual |
| HYDROCOIN (Konikow et al. 1997) | Boundary condition modeled poorly | Conceptual |
| Ontario Uranium Tailings (Flavelle et al. 1991) | Inadequate chemical reaction model | Conceptual |
| Los Alamos (Bredehoeft 2005) | Flow through unsaturated zone not understood | Conceptual |
| Los Angeles (Bredehoeft 2005) | Flow vectors 90 off in model | Conceptual |
| Summitville (Bredehoeft 2005) | Seeps on mountain unaccounted for | Conceptual |
| Santa Barbara (Bredehoeft 2005) | Fault zone flow unaccounted for | Conceptual |
| WIPP (Bredehoeft 2005) | Assumed salt had no mobile interstitial brine | Conceptual |
| Fractured Rock Waste Disposal (Bredehoeft 2005) | Preferential flow in unsaturated zone unaccounted for | Conceptual |

## How to Proceed?

- Desirable characteristics of a methodology for uncertainty assessment
- Comprehensive: as many types of uncertainty as possible should be included
- Quantitative: it should be possible to compare results with regulatory criteria or design requirements
- Systematic: able to be applied to a wide range of sites and objectives and to enable the common application of computer codes and methods

## Deterministic Approach

- Assumptions
- Model parameters are correct
- Model is correct
- Scenario is known

**Notes:**
Part 20 Subpart D (Sect. 1301) Dose Limit to Individual Members of the Public: TEDE < 100 mrem/yr
Part 20 Appendix B Table 2 gives radionuclide concentration limits
Part 20 Subpart E (License Termination) Dose Limit to Avg. Member of Critical Group is 25 mrem/yr (unrestricted use)

## Parameter Sensitivity Approach

- Assumptions
- Model parameters are unknown
- Model is correct
- Scenario is known

## Parameter Sensitivity Approach

- Results
- Probability of peak dose represents the degree of plausibility of the model result
- ? indicates that the actual values of the probabilities are unknown; statements about the relative values may be possible
- Bounding (conservative) analysis: the desired predicted value represents the worst plausible behavior of the system
- Limitations
- Can’t quantitatively estimate risk since probabilities are unknown [risk = p( > 25 mrem/yr)]
- Significance of bounding case must be assessed to avoid over-conservatism
- Significant sources of uncertainty not included

## Parameter Uncertainty Approach

- Assumptions
- Model parameters are uncertain
- Model is correct
- Scenario is known

## Parameter Uncertainty Approach

- Method
- Assign joint probability distribution to model parameters and propagate through the model (e.g., using Monte Carlo simulation)
- Results
- Peak dose probability density represents the degree of plausibility of the model result
- Quantitative estimates of probabilities can be computed
- Quantitative estimates of risk can be computed
- Limitations
- Joint probability distribution of parameters must be determined
- May be computationally expensive
- Significant sources of uncertainty not included

## Conceptual Model Sensitivity Approach

## Conceptual Model Sensitivity Approach

- Method
- Postulate alternative conceptual models for a site that are each consistent with site characterization data and observed system behavior,
- Results
- Each model is used to simulate the desired predicted quantity
- Parameters of each model (which may be different) are represented using a joint probability distribution
- Limitations
- Without a quantitative measure of the degree of plausibility of model alternatives, it is impossible to determine the risk of a decision based on the model predictions
- A conservative approach to model uncertainty relies on an implied belief that the most conservative model has a non-negligible degree of plausibility
- Requires formulation & simulation of multiple models

**Notes:**
Model 3 is most conservative. Is it justifiable to select this model if it is much less plausible than models 1 and 2?

## Quantitative Model Uncertainty

- Assign a discrete probability distribution to the conceptual model alternatives
- Analogous to the interpretation of parameter probability, the discrete model probability distribution represents the degree of plausibility of the model alternatives
- What quantity to compare with regulatory/design criteria?

## Probability-Based Model Selection

- Use the model with the highest probability for predictions
- Potentially biased result if significant probability with alternative models
- If variance due to model uncertainty is desired, must compute predicted value using each model

## Conservative Model Selection

- Use the model with the most significant consequence
- How little probability must lie with the highest consequence model before it is judged implausible?
- Consequence must be computed with each model to determine the conservative model

## Probability-Weighted Model Averaging

## Probability-Weighted Model Averaging

- Method
- Model predictions are combined using a weighted average with the weight for each model’s prediction consisting of that model’s probability
- Results
- Model-averaged probability density function represents the degree of plausibility of the predicted value that takes into consideration the joint effect of parameter and model uncertainties
- Reduces bias
- Less likely to underestimate predictive uncertainty
- Consistent treatment of parameter and model uncertainties
- Quantitative estimates of risk can be computed from the model-averaged result

## Probability-Weighted Model Averaging

- Limitations
- Model probability is a relative measure with respect to the other model alternatives considered
- Requires specifying model probability distribution
- Requires formulating & simulating multiple models
- Doesn’t consider scenario uncertainty

## Model-Averaging Informative Results

|  | Mean
Dose | Prob
(Dose > 25) | 90%ile |
| --- | --- | --- | --- |
| Model 1 (prob = 0.5) | 10.0 | 8.2 | 23.0 |
| Model 2 (prob = 0.25) | 20.0 | 23.9 | 32.7 |
| Model 3 (prob = 0.25) | 45.0 | 97.7 | 57.8 |
| Model Average | 21.2 | 34.5 | 48.5 |

- Results suggest collection of additional data to better discriminate between models (i.e., to modify model probabilities until one model dominates)
- Exceedance probability and 90th percentile suggest that a conservative regulatory action may be preferred
- based on a fully-informed consideration of model and parameter uncertainty (i.e., risk), rather than on adoption of the most conservative model

## Scenario Uncertainty: Unknown Future State or Condition of the System

- Scenario uncertainty can’t be reduced through the application of data (unlike parameter & model uncertainty)

**Notes:**
The time period of modeling can be thought of as consisting of two periods: a history-matching period in which model results can be compared to observations and a prediction period in which the model is used to forecast system behavior. The predictive period is characterized by two features: there are no observations of system behavior in this period and there is an additional source of uncertainty that we can call scenario uncertainty. A scenario is a description of the future state of the system representing such things as future climate, land use, and other natural or human-induced changes. Scenario uncertainty is represented here as a set of three alternative scenarios, one of which may be a reference scenario (e.g., continuation of current conditions). Scenarios tend to affect a model through a change in boundary conditions (e.g., climate change inducing an increase in the recharge rate over time), but may also affect a model in other ways.

## Scenario Uncertainty Sensitivity Approach

**Scenario Uncertainty Sensitivity Approach**

## Scenario Averaging Approach

## Probability-Weighted Scenario Averaging

- Method
- Model-averaged predictions for each scenario are combined using a weighted average with the weight for each scenario’s prediction consisting of that scenario’s probability
- Results
- Scenario- and model-averaged probability density function represents the degree of plausibility of the predicted value that takes into consideration the joint effect of parameter, model, and scenario uncertainties
- Quantitative estimates of risk can be computed from the scenario- and model-averaged result
- Limitations
- Requires specifying scenario probabilities
- Requires simulations of each model under each scenario

## Scenario-Averaging Informative Results

|  | Mean
Dose | Prob
(Dose > 25) | 90%ile |
| --- | --- | --- | --- |
| Scenario 1 (prob = 0.7) (model-average) | 21.2 | 34.5 | 48.5 |
| Scenario 2 (prob = 0.3)  (model-average) | 27.3 | 44.8 | 58.5 |
| Scenario Average | 23.0 | 37.6 | 52.1 |

- Mean dose results straddle regulatory threshold suggesting that a conservative regulatory action may be preferred
- based on a fully-informed consideration of model, parameter, and scenario uncertainty (i.e., risk), rather than on adoption of the most conservative modeling choices

## NRC Staff Application of Probability-Weighted Model Averaging

**MODEL 2**

**MODEL 3**

**MODEL 4**

**MODEL 5**

**Notes:**
Steve Cohen slide, application of MLBMA to model based on Rifle Uranium Mill Tailings site

## Alternative Model Development

- Models developed using Groundwater Modeling System (GMS).
- Model 2: average values for hydraulic conductivity, recharge, and evapotranspiration
- Model 3: average values for hydraulic conductivity and evapotranspiration, zonal values for recharge
- Model 4: average value for hydraulic conductivity, zonal values for recharge and evapotranspiration
- Model 5: same as model 4 with a general head boundary, recharge, and evapotranspiration

**Notes:**
Steve Cohen slide

## Model & Scenario Averaging Application Simulation Results under Two Scenarios (Well 399-1-1)

## Scenario Average (Baseline 70%)

## Project Objectives

- Improve access to the uncertainty assessment methodology by integrating methods with FRAMES
- Provide guidance on the use of model abstraction techniques to generate plausible and realistic alternative conceptual models for a site
- Parameter estimation
- Quantitative model comparison
- Simulation using multiple models and scenarios
- Demonstrate using a realistic application relevant to NRC/NRO analyses

**Notes:**
In FRAMES, assessing multiple models implies mutiple models in the Framework Development Environment

## Project Schedule

- Summer 2008
- Implementation of methods completed
- NRC workshop
- Summer 2009
- Completion of application
- NRC workshop