# MODEL SENSITIVITY ANALYSIS, DATA ASSESSMENT, CALIBRATION, AND UNCERTAINTY EVALULATION

# Utrecht University, The Netherlands June 1-3, 2005

# Instructors: 

Mary C. Hill U.S. Geological Survey (USGS), National Research Program,
Boulder, Colorado, USA, <mchill@usgs.gov>

Howard Reeves USGS, Michigan District, Lansing, Michigan, USA,
<hwreeves@usgs.gov>

Frank Smits Witteveen+Bos, Deventer, the Netherlands, <f.smits@witbo.nl>

# 

# Day 1 Begin 8:30 am

I. Introduction (Mary Hill) 45 minutes

-   Instructors

-   Outline of course

-   Modeling flowchart (Frank Smits). Overview of 14 guidelines for
    effective model calibration

-   Three field investigations are discussed

    -   Flow modeling in complex glacial deposits, Berrien County,
        Michigan, USA

    -   Pumping Test Strandbad Winterswijk

    -   Death Valley (California and Nevada, U.S.A.) regional
        ground-water system

```{=html}
<!-- -->
```
-   Primary documents used in course

    -   Draft book *Effective Model Calibration with Analysis of Data,
        Sensitivity, and Uncertainty* by Hill and Tiedeman that will
        replace M&G (see below), and includes exercises

    -   USGS Methods and Guidelines Report (Hill, 1998) (referred to as
        M&G)

    -   MODFLOW-2000 Documentation:

        -   *User Guide to Modularization Concepts and the Ground-Water
            Flow Process*, Harbaugh, Banta, Hill, McDonald, 2000, USGS
            Open-File Report (OFR) 00-92, 121p.)

        -   *User Guide to the Observation, Sensitivity, and
            Parameter-Estimation Processes and Three Post-Processing
            Programs*, Hill, Banta, Harbaugh, Anderman, 2000, USGS OFR
            00-184, 209 p.

        -   *Documentation of the Advective-Transport Observation (ADV2)
            Package, Version 2*, Anderman, Hill, 2000, USGS OFR 00-342,
            89 p.

    ```{=html}
    <!-- -->
    ```
    -   MFI documentation: *A data input program (MFI2K) for
        MODFLOW-2000* (Harbaugh, 2002, USGS OFR 02-41, 55 p.)

    -   UCODE_2005 documentation: *UCODE_2005 and Three Post-Processors,
        Computer Codes for Universal Inverse Modeling* (Poeter, Hill,
        Banta, USGS OFR).

II\. Overview of MODFLOW-2000 Flow Process capabilities used in class
exercises (Howard Reeves) 30 minutes

(The three letters in parentheses are used to activate the package in
the name file)

-   Basic (BAS) Package

-   Discretization (DIS) File

-   Layer-Property Flow (LPF) Package

-   Recharge (RCH) Package

-   River (RIV) Package

-   General-Head Boundary (GHB) Package

-   Well (WEL) Package

+-----------------------------------------------------------------------+
| # STEADY-STATE TEST CASE:                                             |
|                                                                       |
| # FORWARD SOLUTION AND SENSITIVITY ANALYSIS                           |
+-----------------------------------------------------------------------+

III\. Steady-state ground-water flow system used for exercises (Mary
Hill) 30 minutes

Problem to be addressed

Description of flow system

MFI (public domain, open-source, limited graphical interface for MODFLOW
input from USGS)

Break 10:15 to 10:30

Define parameters for the steady-state problem Exercise 3.1

Define and weight hydraulic-head and flow observations for the\
steady-state problem Exercise 3.2

GW_CHART (public domain, open source, program to graph MODFLOW-2000
output)

Evaluate model fit resulting from the starting parameter values Exercise
3.3

Lunch 12:30 to 13:30

IV\. Sensitivity analysis for the initial model (Howard Reeves)

Dimensionless, composite, and one-percent scaled sensitivities for
evaluating\
observations and defined parameters Exercise 4.1b

Parameter correlation coefficients for evaluating parameter uniqueness
Exercise 4.1c

Evaluate contour maps of one-percent scaled sensitivities Exercise 4.1d

Break 3:30 to 3:45 pm

Guidelines 1-2 for model development (Mary C. Hill):

[Guideline 1]{.underline}: Apply the principle of parsimony

[Guideline 2]{.underline}: Use a broad range of information to constrain
the problem

+-----------------------------------------------------------------------+
| # INVERSE MODELING USING NONLINEAR REGRESSION                         |
+-----------------------------------------------------------------------+

V. Nonlinear regression (Frank Smits and Howard Reeves)

**Objective function surfaces for two lumped parameters using
hydraulic-head observations alone and with the flow observation (Frank
Smits)**

Use parameter correlation coefficients Exercise 5.1a

Examine performance of the modified Gauss-Newton method Exercise 5.1b

# 

End 6:00 pm

# Day 2 Begin 8:30 am

**Nonlinear regression by the modified Gauss-Newton method (Howard
Reeves)**

Discussion of theory

Define range of reasonable parameter values Exercise 5.2a

First attempt at estimating parameters by nonlinear regression Exercise
5.2b

Break 10:15-10:30

**30 minutes**

Prior information on parameters Exercise 5.2c

Comments on prior information and regularization

Guidelines 3-5 for model development (Mary C. Hill): 90 minutes

[Guideline 3]{.underline}: Maintain a well-posed, comprehensive
regression problem

[Guideline 4]{.underline}: Include many kinds of data as observations in
the regression

[Guideline 5]{.underline}: Use prior information carefully

[Guideline 6]{.underline}: Assign weights which reflect measurement
errors

[Guideline 7]{.underline}: Encourage convergence by making the model
more accurate

[Guideline 8]{.underline}: Consider alternative models

Lunch 12:30 to 1:30

VI\. Evaluate Model Fit (Frank Smits)

Statistical measures of overall model fit

Objective-function values Exercise 6.1a

Calculated error variance, standard error, and fitted error statistics
Exercise 6.1b

Graphical analyses of model fit and related statistics

Weighted residuals versus weighted simulated values and minimum,

maximum, and average weighted residuals Exercise 6.2a

Weighted observations versus weighted simulated values and correlation

coefficient R Exercise 6.2b

Graphs using independent variables Exercise 6.2c

Normal probability graphs and correlation coefficient R~N~^2^ Exercise
6.2d

Determine acceptable deviations from independent normal weighted

residuals Exercise 6.2e

Break 3:30-3:45 pm

**VII. Parameter statistics (Howard Reeves) 105 minutes**

Composite scaled sensitivities Exercise 7.1a

Variances and covariances Exercise 7.1b

Evaluate the precision of the estimates using standard deviations,

confidence intervals, and coefficients of variation Exercise 7.1c

Compare estimated parameter values with reasonable ranges Exercise 7.1d

Evaluate uniqueness of parameter estimates using correlation
coefficients Exercise 7.1e

Test for linearity Exercise 7.2

**FIELD APPLICATION:** Pumping Test Strandbad Winterswijk (Frank Smits)
30 minutes

## FIELD APPLICATION: Flow modeling in complex glacial deposits, Berrien County, Michigan, USA (Howard Reeves) 30 minutes

## 

**End at 6:30 pm**

# Day 3 Begin 8:30 am

Model testing guidelines (Mary Hill): 30 minutes

[Guideline]{.underline} 8: Evaluate model fit

[Guideline]{.underline} 9: Evaluate optimized parameter values

  -----------------------------------------------------------------------
  **PREDICTIONS USING THE STEADY-STATE MODEL**

  -----------------------------------------------------------------------

VII\. Predictions (Howard Reeves) 75 minutes

Simulate predictions and their sensitivities

Predict advective transport Exercise 8.1a

Determine the parameters that are important to the predictions using
prediction

scaled sensitivities and parameter correlation coefficients Exercise
8.1b

Break 10:15 to 10:30

**90 minutes**

Assess the likely importance of potential new data to the predictions
using

dimensionless scaled sensitivities and parameter correlation
coefficients Exercise 8.1c

Prediction uncertainty measured using inferential statistics

Linear confidence and prediction intervals on the components of

advective travel Exercise 8.2a

Guidelines for evaluating future efforts, uncertainty, and predictions
(Mary C. Hill): 30 minutes

[Guideline]{.underline} 11: Identify new data to improve model parameter
estimates

[Guideline]{.underline} 12: Identify new data to improve model
predictions

Lunch 12:30-1:30 pm

**40 minutes**

[Guideline]{.underline} 13: Evaluate prediction uncertainty and accuracy
using deterministic methods

[Guideline]{.underline} 14: Quantify prediction uncertainty using
statistical methods

**FIELD APPLICATION:** Calibrate a ground-water model of the Death
Valley regional flow system, California and Nevada, using optimal
parameter estimation and three-dimensional GIS data base and
visualization methods (Mary C. Hill) **50 minutes**

Break 3:00 to 3:15 pm

**FIELD APPLICATION (continued):** Use the model to evaluate observation
locations and parameters in the context of predictions (Mary C. Hill)
**60 minutes**

X. Summary comments (Mary C. Hill) 45 minutes

Comments about PEST and Ostrich

Use of methods with transport simulations

Relation of methods presented to other model calibration methods

Use of the guidelines with different inverse methods

Utility of the methods, guidelines, and MODFLOW-2000

End of Class 5:00