MODEL SENSITIVITY ANALYSIS, DATA ASSESSMENT, CALIBRATION, AND
UNCERTAINTY EVALUATION

**U.S. Geological Survey National Training Center**

**March 8-12, 2004**

# Instructors, all from U.S. Geological Survey

Mary C. Hill, Coordinator NRP, Boulder, CO <mchill@usgs.gov>

Claire R. Tiedeman NRP, Menlo Park, CA <tiedeman@usgs.gov>

Ned Banta Colorado District, Lakewood, CO <erbanta@usgs.gov>

Marshall Gannett Oregon District, Portland, OR <mgannett@usgs.gov>

Howard Reeves Michigan District, Lansing. MI <hwreeves@usgs.gov>

\*\*\*The exercise numbers differ frim those in Hill and Tiedeman
(2007), but most of the exercises are the same.

**MONDAY** 8:30 am

**I. Introduction** (Mary Hill)

> Outline of course
>
> Fourteen guidelines for effective model calibration
>
> Four field investigations are discussed:
>
> 1\. Deschutes basin, Oregon ground-water model calibration -- Tuesday
>
> 2\. Albuquerque basin ground-water model calibration -- Thursday
>
> 3\. Planning plume remediation in southwest Michigan \-- Thursday
>
> 4\. Death Valley regional ground-water flow system ground-water model
> calibration with optimal parameter estimation and three-dimensional
> GIS, and analysis of parameter observation location importance in the
> context of predictions -- Friday
>
> Some basics
>
> Model calibration using trial and error only and with nonlinear
> regression
>
> Definition of parameters and observations
>
> Objective function and some example objective function surfaces
>
> Graphical interfaces for MODFLOW-2000
>
> Primary documents used in course:
>
> Draft of a book that will replace M&G (see below), and includes the
> exercises (Hill and Tiedeman, in prep)
>
> Methods and guidelines report (Hill, 1998) (referred to as M&G)
>
> MODFOW-2000 documentation (Harbaugh +, 2000; Hill +, 2000)
>
> Advective-Transport Obs. (ADV) documentation (Anderman and Hill, 2000)
>
> Directory structure for computer files used in course

+-----------------------------------------------------------------------+
| **STEADY-STATE TEST CASE: FORWARD SOLUTION AND**                      |
|                                                                       |
| **SENSITIVITY ANALYSIS**                                              |
+-----------------------------------------------------------------------+

**II. Overview of computer programs, ground-water management problem,
and exercises**\
(Ned Banta)

> MODFLOW-2000:
>
> Processes and Packages
>
> Flow chart

LIST and GLOBAL output files

List and array data

> Parameters
>
> LPF
>
> DIS

Break 10:15 to 10:30

Description of flow system

Define spatial and temporal discretization using the Discretization file
DIS

Create the Basic Package (BAS) input file (Set IBOUND and initial head)

**III. Compare observed and simulated values using objective functions**
(Ned Banta)

Construct the model (Program Mode Forward):

List data Exercise 3a

RIV (Define a RIV parameter, which controls model input \[riverbed
conductance\] for a list of cells)

GHB (Define GHB cells without parameters)

Array data

Multiplier and Zone arrays

RCH (Define RCH parameters, which control a spatially distributed model
input \[recharge\] that applies over the top of the modeled area)
Exercise 3b

Lunch 12:00 to 1:00

LPF (Define HK parameters, which control a spatially distributed model
input \[hydraulic-conductivity\] that applies to all model layers)
Exercise 3c

LPF (Define a VKCB parameter, which controls a spatially distributed
model input \[confining-bed vertical hydraulic conductivity\] that
applies between model layers) Exercise 3d

Final preparations:

Select solver (PCG)

Output control (save heads)

Optional: LPF (Define HANI and/or VANI parameters to control horizontal
and vertical anisotropy of model layers) Exercise 3e

Optional: LPF (Additive parameters for interpolation and stochastic
methods) Exercise 3f

Break 3:00-3:15

Compare observed and simulated values using objective functions (Ned
Banta )

> Define simulated values (Program Mode Forward with Observations):

What are observations?

Explain the file applicable to all observations

Hydraulic-head observations Exercise 4a

Check simulated values Exercise 4b

Flow observations Exercise 4c

Weighting the observations Exercise 4d

Go to computers and do exercise 4 (includes break)

> Evaluate initial simulated values:

Program modes:

Forward with Parameter-Value Substitution -- Introduction to the SEN
file

Forward with Observations and Parameter-Value Substitution

Evaluate model fit resulting from the starting parameter values Exercise
5

Go to computers and do exercise 5

End at 5:30

**TUESDAY** 8:30 am

**IV. Define the information observations provide on parameters using
fit-independent statistics** (Marshall Gannett)

> Sensitivity analysis for initial model:

Program modes:

Parameter Sensitivity

Parameter Sensitivity with Observations

Sensitivity Analysis to Evaluate Potential for Parameter Estimation

Calculate sensitivities for the steady-state flow system Exercise 6a

Use dimensionless, composite, and one-percent scaled sensitivities to
evaluate observations and defined parameters Exercise 6b

Parameter correlation coefficients for evaluating parameter uniqueness
Exercise 6c

(treat correlation coefficients intuitively, they will be defined
formally later)

Evaluate contour maps of one-percent sensitivities for the steady-state
flow system Exercise 6d

Go to computers and do exercise 6 (includes break)

Break 10:00 to 10:15

  -----------------------------------------------------------------------
  **INVERSE MODELING USING NONLINEAR REGRESSION, AND ANALYSIS OF MODEL
  FIT AND PARAMETERS**

  -----------------------------------------------------------------------

**V. Estimate parameter values using nonlinear regression** (Howard
Reeves and Mary Hill)

> Objective function surfaces using hydraulic-head observations alone
> and with the
>
> flow observation, and two lumped parameters (Howard Reeves)

Relation to parameter correlation coefficients Exercise 7a

Examine the performance of the modified Gauss-Newton method Exercise 7b

> Nonlinear regression by the modified Gauss-Newton method (Mary Hill)

Discussion of theory

Lunch 12:00 to 1:00

Define range of reasonable parameter values Exercise 8a

First attempt at estimating parameters by nonlinear regression Exercise
8b

Prior information on parameters Exercise 8c

Comments on prior information and regularization

Estimation of log-transformed parameters Exercise 8d

Break 2:45 to 3:00

**Guidelines for model development (Chapters 10-11)** (Claire Tiedeman):

[Guideline 1]{.underline}: Apply the principle of parsimony

[Guideline 2]{.underline}: Use a broad range of information to constrain
the problem

[Guideline 3]{.underline}: Maintain a well-posed, comprehensive
regression problem

[Guideline 4]{.underline}: Include many kinds of data as observations in
the regression

[Guideline 5]{.underline}: Use prior information carefully

[Guideline 6]{.underline}: Assign weights which reflect measurement
errors

[Guideline 7]{.underline}: Encourage convergence by making the model
more accurate

Issues of computer execution time

**VI. Evaluate model fit using statistical and graphical analyses**
(Howard Reeves)

> Statistical measures of overall model fit
>
> Objective-function values Exercise 9a
>
> Calculated error variance, standard error, and fitted error statistics
> Exercise 9b
>
> The AIC and BIC statistics Exercise 9c

End at 5:30

**WEDNESDAY** 8:30 am

> Graphical analyses of model fit and related statistics
>
> Weighted residuals versus weighted simulated values and minimum,\
> maximum, and average weighted residuals Exercise 10a
>
> Weighted observations versus weighted simulated values and\
> correlation coefficient R Exercise 10b
>
> Graphs using independent variables and the runs statistic Exercise 10c

Break 10:00 to 10:15

> Normal probability graphs and correlation coefficient R~N~^2^ Exercise
> 10d
>
> Determining acceptable deviations from independent normal\
> weighted residuals Exercise 10e

**FIELD APPLICATION**: Ground-water modeling in the volcanic-arc
geologic setting of the Deschutes Basin, Oregon (Marshall Gannett)

Lunch 12:00 to 1:00

**VII. Evaluate estimated parameter values and parameter uncertainty
using linear-regression-based methods** (Mary Hill)

> Parameter statistics

Composite scaled sensitivities Exercise 11a

Variances and covariances Exercise 11b

Evaluate the precision of the estimates using standard deviations,\
confidence intervals, and coefficients of variation Exercise 11c

Compare estimated parameter values with reasonable ranges Exercise 11d

Evaluate the uniqueness of the parameter estimates using correlation\
coefficients Exercise 11e

Detecting non-unique parameter estimates Exercise 11f

Evaluate the precision of the estimates using nonlinear conf intervals
Exercise 11g

Evaluate the importance of individual data using influence statistics
Exercise 11h

Model linearity

Testing for linearity Exercise 12

Break 3:00-3:15

**Model testing guidelines (Chapter 12)** (Claire Tiedeman)

[Guideline]{.underline} 8: Evaluate model fit

[Guideline]{.underline} 9: Evaluate optimized parameter values

[Guideline]{.underline} 10: Test alternative models

**Demonstration of Graphical User Interfaces** (Mary Hill and Claire
Tiedeman)

Interfaces for MODFLOW

Groundwater Vistas, Argus ONE,

Groundwater Modeling System (GMS), Visual Modflow

Interface for GEOlogic Knowledge Interaction PROtocol, including
Interactive Reports

GEOPRO

End 5:30

**THURSDAY** 8:30 am

**FIELD APPLICATION**: Calibration of a flow model of the Albuquerque
Basin\
(Claire Tiedeman)

+-----------------------------------------------------------------------+
| ## PREDICTIONS USING THE STEADY-STATE MODEL                           |
+-----------------------------------------------------------------------+

**VIII. Evaluate model predictions, data needs, and prediction
uncertainty**\
(Mary Hill, Claire Tiedeman)

> Simulating predictions and their sensitivities (Mary Hill)

Predicting advective transport Exercise 13a

Break 10:00-10:15

> Determine the parameters that are important to the predictions using
> prediction\
> scaled sensitivities and parameter correlation coefficients Exercise
> 13b
>
> Assess the likely importance of potential new data to the predictions
> using\
> dimensionless scaled sensitivities and parameter correlation\
> coefficients Exercise 13c
>
> Prediction uncertainty measured using inferential statistics (Mary
> Hill)
>
> Linear confidence and prediction intervals on the components of\
> advective travel Exercise 14a
>
> Nonlinear confidence intervals on the components of\
> advective travel Exercise 14b
>
> The effect on confidence intervals of setting and regularizing\
> parameters Exercise 14c

Lunch 12:00 to 1:00

> Prediction uncertainty measured using inferential statistics --
> continued (Claire Tiedeman)
>
> Determine parameters important to the predictions using value of
> improved information statistics
>
> Assess the likely importance of potential new data to the predictions\
> using uncertainty statistics

**Guidelines for using the model to evaluate further model development
and data assessment (Chapter 13)** (Claire Tiedeman)

[Guideline]{.underline} 11: Formally consider predictions

[Guideline]{.underline} 12: Evaluate potential new data and possible
additional estimated parameters

**Guidelines for using the model to evaluate prediction uncertainty
(Chapter 14)\
**(Claire Tiedeman)

[Guideline]{.underline} 13: Use inferential statistics to quantify
prediction uncertainty

[Guideline]{.underline} 14: Use Monte Carlo methods to quantify
prediction uncertainty

Break 2:45 to 3:00

**FIELD APPLICATION**: Optimal containment strategies under parameter
uncertainty for a vinyl chloride plume in southwest Michigan (Claire
Tiedeman)

### ADDING TRANSIENT DATA TO IMPROVE THE MODEL AND PREDICTIONS

**IX. Calibrate transient and transport models and recalibrate existing
models** (Ned Banta)

> Description of flow system

MODFLOW-2000 flow process input files for transient simulation:\
Activate parameters for transient stress periods, add new parameters,\
and run simulation Exercises 15-17

> Observations for the transient problem

Hydraulic heads, flows, and temporal changes in heads Exercise 18

> Mode: Forward transient modeling with parameter substitution and
> observations

Evaluate transient model fit using starting parameter values Exercise 19

Go to computers and do exercises

End at 5:30

**FRIDAY** 8:30 am

IX\. Calibrate transient and transport models and recalibrate existing
models -- continued\
(Claire Tiedeman, Mary Hill)

> Sensitivity analysis for the initial model (Claire Tiedeman)
>
> Contour maps of one-percent scaled sensitivities for the transient\
> flow system Exercise 20a
>
> Dimensionless scaled sensitivities for evaluating observations and\
> defined parameters Exercise 20b
>
> Nonlinear Regression (Claire Tiedeman)
>
> Estimate parameters for the transient system by nonlinear regression
> Exercise 21a
>
> Compare estimated parameter values with reasonable ranges Exercise 21b

Break 10:00 to 10:15

> Model Evaluation (Claire Tiedeman)

Evaluate measures of model fit Exercise 22

Perform graphical analyses of model fit and evaluate related statistics
Exercise 23

Evaluate estimated parameters Exercise 24

Test for nonlinearity Exercise 25

> Predictions -- Are they different? Better? (Mary Hill)
>
> Predicting advective transport with the model calibrated with\
> steady-state and transient observations Exercise 26a

Sensitivities and correlations Exercise 26b

Prediction uncertainty using inferential statistics Exercise 26c

Lunch 12:00 to 1:00

**FIELD APPLICATION**: Using a ground-water model calibrated using
optimal parameter estimation and three-dimensional GIS data base and
visualization methods, and using the model to evaluate parameters and
observation locations in the context of predictions (Mary Hill and
Claire Tiedeman)

**Overview of course** (Claire Tiedeman)

**Summary comments** (Mary Hill)

> Utility of the methods, guidelines, and MODFLOW-2000
>
> Comments about UCODE
>
> Using concentration data to calibrate ground-water models
>
> Relation of methods presented to other model calibration methods

End of Class 3:30