# Presentation: 529904

## VI. Evaluation of Model FitPart 2: Graphical Analysis of Model Fit and Related Statistics

- Graphs of weighted residuals versus unweighted simulated values, and minimum, maximum, and average weighted residuals
- Graphs of weighted observations versus weighted simulated values and the correlation coefficient R
- Graphics using independent variables and the runs statistic
- Normal probability plots and the correlation coefficient *R**N**2*
- Determining acceptable deviations from independent normal weighted residuals
- Use GW_Chart for plotting

## Weighted Residuals vs Simulated Values  (_ws in UCODE_2005)  (Book, p. 100 – 104)

- Two of the requirements for a valid regression are that** the weighted residuals are random and have a mean of zero**.
- Use **graph of the weighted residuals versus simulated values** to evaluate the the weighted residuals.
- The weighted residuals should be evenly distributed about zero for all weighted simulated values, and should display no trends with the simulated values.
- Trends or unequal variance indicator model bias.
- Examples of randomly and non-randomly distributed weighted residuals:
- Figures 6-1 and 6-2 of Hill and Tiedeman (page 102-103).
- Figure 6-7A (page 116) of Hill and Tiedeman shows graph for the steady-state problem.

## Change to the _ws file from the book and previous codes

- The book uses WEIGHTED simulated values on the horizontal axis of these graphs.
- There is a statistical reason for using weighted simulated values, but in practice it is more confusing and doesn’t add much.
- The _ws data-exchange file now lists weighted residuals and unweighted simulated values. This is a change from the _ws file in previous codes UCODE, MODFLOW-2000, and MODFLOWP.
- If there is a very wide range in the simulated values, you can make the horizontal axis log-transformed, or use the weighted simulated values from the _ww file.)
- The weighted residuals also can be plotted against observed values

## Wted Residuals vs. Simulated Values

- Figure 6-1 (page 102)

- Figure 6-2 of Hill and Tiedeman (page 103)

- Define scale of weighted residual axis using the standard error of the regression.

- (These figures use weighted simulated values, but the results are the same)

## 1. Minimum, Maximum, and Average Weighted Residual (Book, p. 100)

**Minimum and maximum weighted residuals** display the range of weighted residuals. Examination of these values can help reveal:
- Areas where the fit to the observed data is especially poor,
- Areas where data has been incorrectly interpreted, and
- Data input errors.
- The **average weighted residual** in nonlinear regression should be close to zero. In linear regression, the average residual is always exactly zero.
- Caution: the average weighted residual in nonlinear regression can be close to zero even if there are other problems with the model or regression.
- DO EXERCISE 6.2a: Evaluate graph of weighted residuals versus weighted simulated values.  Evaluate the minimum, maximum, and average weighted residuals.

## 1. Weighted Residuals vs. Weighted Simulated Values (can use unweighted simulated values)

- Figure 6-7A of Hill and Tiedeman (page 116)

- _ws
- data-exchange file

**Notes:**
Small sample sizes can be difficult to interpret.
Heads appear to be random about zero
Flow and prior are clustered around a value of zero on the horizontal axis, although this looks strange, it is not a problem.

## 2. Graphs of Weighted Observed vs.Weighted Simulated Values (Book, p. 105)

- Values on the graph of **weighted observed versus weighted simulated values** should plot close to a line with slope = 1.0
- Generally, for assessing model bias or desired properties of weighted residuals, these graphs are not as useful as graphs of weighted residuals vs. weighted simulated values.
- This is partly because a large range in magnitudes of weighted observations can obscure trends in the differences between the weighted observations and the weighted simulated values.
- See Figure 6-3 (page 105) of Hill and Tiedeman.

## 2. Wted Observed vs.Wted Simulated Values

- 2. Wted Observed vs.Wted Simulated Values

-        Figure 6-3 of Hill and Tiedeman (page 105)

- The data have the same problems as in the graphs with weighted residuals (repeated here). Bit these graphs do not reveal the problems as clearly!

- Figures with weighted residuals

- Figures with weighted observations

- _ww

- _ws

- In UCODE_2005, _ws has simulated
- instead of weighted simulated values

**Notes:**
In the first graph, the variance increases as the simulated values increase (not homoskedastic)
In the second graph, the residuals increase as the simulated values increase

## 2. Correlation Coefficient R

**R** is the **correlation between the weighted simulated values and the weighted observations**.
- This summary statistic reflects how well the trends in the weighted simulated values match the trends in the weighted observations.
- The equation for the correlation coefficient R can be expressed as (see Hill and Tiedeman, equation 6.11a):
-     _covariance of weighted observations and weighted simulated values    _
- (std. dev. of weighted observations)(std. dev. of weighted simulated values)
- A value of R greater than 0.90 generally indicates a good match of the trends. However, R is not too useful for assessing model fit, because of the same drawbacks as for the graph of weighted observed vs. weighted simulated values.
- DO EXERCISE 6.2b: Graph weighted observations versus weighted simulated values and examine the correlation coefficient R.
- Figure 6-7b (page 116) of Hill and Tiedeman shows the graph for the steady-state problem.

## 2. Correlation Coefficient R (Book, p. 106)

**R** is the **correlation between the weighted simulated values and the weighted observations**.
- This summary statistic reflects how well the trends in the weighted simulated values match the trends in the weighted observations.
- A value of R greater than 0.90 generally indicates a good match of the trends. However, R is not too useful for assessing model fit, because of the same drawbacks as for the graph of weighted observed vs. weighted simulated values.
- DO EXERCISE 6.2b: Graph weighted observations versus weighted simulated values and examine the correlation coefficient R.
- Figure 6-7b and c (page 116) of Hill and Tiedeman show graphs of weighted and unweighted simulated versus observed values for the steady-state problem.

## Weighted Observed vs.Weighted Simulated Values

- Figure 6-7b of Hill and Tiedeman (page 116)

- _ww

- _ws

## 3. Graphs Using Independent Variables (Book, p. 106)

**Graphs using independent variables** include plots of weighted residuals on maps of the model area or versus time.
- These plots should appear random and show no obvious patterns.
- Lack of randomness can be indicative of model error – for example if weighted residuals are all positive in a particular region of the model.
- Graph weighted residuals on maps of the model layers.  (Exercise 6.2c)
- Figure 6-9 (page 117) of Hill and Tiedeman shows these graphs for the steady-state problem.

## 3. Graphs UsingIndependent Variables

- Figure 6-9 of Hill and Tiedeman (page 117)

- Plotted using _w and .xyzt file or other source of location information.

**Notes:**
Obs 2 & 4 – the obs are in the same column.  Recall the flow system and contours – all heads are identical in a column.  Because random noise was added and the noise is of opposite sign, the regression will try to produce a head that matches both heads.  It does this by producing a head in between as this gives the lowest overall residual.  

Obs 1 – this observation has random noise.  However, because the head here is controlled by the river stage, there is little the regression can do to change this value head value with the given parameters.  Thus, the residual here is essentially equal to the noise added to the data.

## 3. Runs Statistic

- The **runs statistic** is a summary statistic that evaluates the randomness of the weighted residuals given the order that they are listed in the input file. Originally developed to check patterns in time series data.
- A ‘run’ is a **sequence of residuals with the same sign**. For example, the sequence of numbers 2, 4, -1, -3, -2, 2, 5, -9 has 4 runs: (2, 4), (-1, -3, -2), (2, 5), (-9).
- The number of runs can be defined as a random variable. If the number of positive and negative residuals are both > 10, then the number of runs should be normally distributed.
- To calculate the runs statistic, the number of runs in a list of residuals is counted, and is compared with the expected number of runs using test statistics.

## 3. Runs Statistic

- The test statistic for **too few runs **is:
- The test statistic for **too many runs **is:

- where:
*u*  = number of runs
-  = mean of *u* (expected number of runs
-  = standard deviation of *u*
*u* and  are functions of the numbers of positive and negative residuals (see Hill, 1998).

*z**f* and *z**m* can be evaluated using a **standard normal distribution**. To test whether the order of the residuals is random, we look at the probability of obtaining *z**f* or *z**m* using a standard normal distribution.

## 3. Runs Statistic

- Critical values of *z**f* or *z**m* are printed in the regression output file. These can be used to decide whether to accept or reject, at a particular significance level, the hypothesis that the residuals are random.
- This analysis is valid for n1>10 and n2 > 10.
- The runs statistic is intended to be used for a quick diagnosis of problems with patterns in the data – not necessarily a statistic you will report.
- DO EXERCISE 6.2c (page 118): Evaluate runs statistic.

**If *****z******f***** < 0, then there may be too few runs**. In this case, the probability that the values are in a random order is the area under the standard normal curve to the left of *z**f*
**If *****z******m***** > 0, then there may be too many runs**. The probability that the values are in a random order is the area under the standard normal curve to the right of *z**m*.

## 4. Normal Probability Graphs(Book, p. 108-111)

- Normality of the true errors, and therefore the weighted residuals, is not an assumption required for the regression to be valid.
- However, computation of some inferential statistics, such as parameter confidence intervals, does require that the weighted residuals be normally distributed.
- If weighted residuals are independent and normally distributed, they should plot on a straight line on a normal probability graph.

## 4. Correlation Coefficient RN2

- Summary statistic to test for independence and normality of weighted residuals is ***R******N******2***, the correlation coefficient between ordered weighted residuals and order statistics from a standard normal probability distribution function.
*R**N**2* is calculated as (see Hill and Tiedeman, equation 36):_covariance of ordered wted resids and order statistics from std normal pdf_(std dev of ordered wted resids)(std dev of order statistics from std normal pdf)
- The hypothesis tested with *R**N**2* is that the weighted residuals are independent and normally distributed. **Critical values of *****R******N******2*** are used for significance levels of 0.05 and 0.10, to test this hypothesis. The significance level is the probability that we are wrong in rejecting the hypothesis.
- For example, we choose a significance level of 0.05, and its critical value:
**If *****R******N******2*****  > critical value**, **we accept the hypothesis **that the weighted residuals are independent and normally distributed.
**If *****R******N******2***** < critical value**, **we reject the hypothesis **that the weighted residuals are independent and normally distributed, and there is a 5 percent probability that we are wrong.

## 4. Correlation Coefficient RN2 (Book, p. 110) 

- Summary statistic to test for independence and normality of weighted residuals is ***R******N******2***, the correlation coefficient between ordered weighted residuals and order statistics from a standard normal probability distribution function.
- The hypothesis tested with *R**N**2* is that the weighted residuals are independent and normally distributed. **Critical values of *****R******N******2*** are used for significance levels of 0.05 and 0.10, to test this hypothesis. The significance level is the probability that we are wrong in rejecting the hypothesis.
- For example, we choose a significance level of 0.05, and its critical value:
**If *****R******N******2*****  > critical value**, **we accept the hypothesis **that the weighted residuals are independent and normally distributed.
**If *****R******N******2***** < critical value**, **we reject the hypothesis **that the weighted residuals are independent and normally distributed, and there is a 5 percent probability that we are wrong.

## 4. Correlation Coefficient RN2

- The critical values for significance levels of 0.05 and 0.10 are printed in the UCODE and MF2K output files. Because this is a strict test for independence and normality, it is generally adequate to use **a significance level of 0.05**.
***R******N******2***** is calculated in two ways**:
- using only the weighted residuals for the dependent-variable observations.
- using the dependent-variable and the prior information weighted residuals.
- Large differences in *R**N**2* between these two data sets indicate that the data sets are not statistically similar.
- For commonly used sample sizes, this test is more powerful than other statistics used to test normality, such as Kolomogorov-Smirnov.
- DO EXERCISE 6.2d: Prepare normal probability graphs and evaluate the correlation coefficient *R**N**2*.
- Figure 6-11 (page 119) of Hill and Tiedeman shows the graph for the steady-state problem.

## 4. Normal Probability Graph

- Figure 6-11 of Hill and Tiedeman (page 119)

- _nm

## 5. Deviations from Independence and Normality (Book, p. 111-113)

- It is possible that the weighted residuals might fail the tests for independence and normality because
- There are **too few residuals**, or
- The weighted residuals are normal, but they are **correlated due to the fitting process of the regression**, instead of being independent.
- The regression methodology can result in the weighted residuals being **correlated**, **because the regression fits the errors in the data**. This correlation becomes more significant if the number of observations is small compared to the number of parameters.
- Cooley and Naff (1990) developed a method for testing whether too few residuals or correlations between residuals might be the cause of the failure of the tests for independence and normality.

## 5. Deviations from Independence and Normality

- Steps of the method developed by Cooley and Naff are:
- Generate normally distributed random numbers with and without the regression-induced correlation.
- Compare the normal probability graph of the weighted residuals with those of the generated independent normally distributed random numbers (d’s). If similar deviations from a straight line are detected in the two types of graphs, then the deviations could result from too few residuals.
- Compare the normal probability graph of the weighted residuals with those of the generated correlated normally distributed random numbers (g’s). If similar deviations from a straight line are detected in the two types of graphs, then the deviations could result from regression-induced correlation.
- UCODE and RESAN-2000 can be used to generate the independent and correlated normally distributed random numbers.
- Assess deviations from independence and normality (EXERCISE 6.2e)
- Figure 6-13 (p. 121) of Hill and Tiedeman shows graphs of generated numbers versus weighted simulated values; Figure 6-14 (p. 122) shows normal probability graphs of the generated numbers. Exer\ex6.2e-residual-analysis.bat

## 5. Deviations in normal probabilitygraphs

- Figure 6-11: **weighted residuals for steady-state model**

- Figure 6-14A

- Figure 6-14B

- Run residual_analysis to get the _rd and _rg files.
- Plot with gw_chart.
- (gw_chart reverses the axes compared to the graphs here.)

- _nm

- _rd

- _rg

**Notes:**
Weighted residuals show kinks around zero, as do the correlated normal probability plots.  The reason for the kinks around zero is because these data are the flow and two prior.  The flow is the decoupling information which prevents perfect correlation, so the regression will try to match it and thus the low residual.  For the prior, the two parameters were insensitive, and thus the prior is the dominating information, so these are matched well by the regression also.

The correlated normal probability plots are calculated using the sensitivity information, which, as explained above for the flow and prior, can lead to the same types of kinks.

Therefore, despite the apparent non random and non normality look of the residuals, the results can be explained and does not indicate any problems with the model design.