# Presentation: 640692

## DATA QUALITY and ANALYSISStrategy for Monitoring Post-fire Rehabilitation Treatments

**DATA QUALITY and ANALYSIS****Strategy for Monitoring Post-fire Rehabilitation Treatments**

**Troy Wirth and David Pyke**
**USGS – Biological Resources Division**
**Forest and Rangeland Ecosystem Science Center**
**Corvallis, Oregon**

**U.S. Department of Interior**
**U.S. Geological Survey**

**Supported by USGS - BLM Interagency **
**Agreement #HAI040045**

## Data Quality

- Assess the ability of the data to determine treatment success
- Ability to achieve high data quality will depend on variability
- Calculate data quality variables
- Confidence intervals (precision)
- Alpha (p-value) & beta levels
- Sample size estimation

**Notes:**
Monitoring data is highly variable due to the large geographic area of inference.  While it is necessary to use this monitoring data to determine treatment success – we can evaluate the data quality to aid in decision making.
Std. Deviation:  The average difference from the mean of the parameter being measured. Confidence Interval:  Interval of plausible values for the parameter being measured with a certain confidence.  The width of the interval describes how precisely the value of the parameter is known. The narrower the width, the more precise the estimate.  Sample size:  The estimated number of samples, based on current variability and defined alpha level, required to achieve a certain level of precision (CI half-width).

## Confidence Intervals

- Construct a simple confidence interval around data to determine precision of estimate
- The narrower the confidence interval, the more precise the estimate
- Must specify the alpha level

-  Alpha level, or type I error is the probability of declaring there is no difference when there is.
-  Specifies the width of the confidence interval (1 – alpha)

**Notes:**
1.  Construct confidence interval by using the mean, number of samples, t-values, and standard deviation.  

## From Elzinga et al. 1998

**From Elzinga et al. 1998**

## Confidence Intervals

**Notes:**
Examples of some treatment and control plots for plant density.  90% confidence intervals are shown.  For existing grass and shrubs. the confidence levels overlap meaning you can’t determine if there is a difference between the treatment and control. For seeded grass you can tell there is a difference even though there is only a sample size of three treatment and three control plots.

## Sample Size Estimation

- Equations which estimate the number of samples required to meet your sampling objective
- For single populations (quantitative objective)
- Confidence level (Type I error rate)
- Confidence interval width
- For detecting difference between two populations
- Confidence levels (Type I and II error rate)
- Minimum detectable change
- Iterative process

**Notes:**
1.  Calculate the estimated required sample size based on the sampling objective (alpha level and precision).  For highly variable data this estimate may be very high.  Grouping lifeforms is helpful.

## Single Population Sample Size

- Calculate sample size estimate for each parameter of interest
- Result will depend on variability of data
- For example, using equation for single population:

**Notes:**
1.  Calculate the estimated required sample size based on the sampling objective (alpha level and precision).  For highly variable data this estimate may be very high.  Grouping lifeforms is helpful.

## Single Population Sample Size Estimation Example

- Alpha = 0.1
- X = 18.5 % cover of perennial grass
- S = 4.9 % cover
- d = (18.5*0.2) = 3.7
- Initial sample = 5

- Need 3 more samples  (precision achieved 4.7)

## Improving Data Quality

- In order to increase data quality (achieve sample size) you need to:
- Reduce standard deviation (variability)
- Increase the number of samples
- In order to reduce sample size estimates without more samples you can:
- Increase alpha (less confidence)
- Increase precision or MDC (detect a larger difference)

## Graphical Analysis Using Confidence Intervals

- Compare treatment results to quantitative objectives or control areas
- Several types of analysis to fit your situation
- Types of graphical analysis:
- Comparison of treatment to quantitative standard
- Seeded plants vs. quantitative standard
- All plants at treatment plots change from time 1 to time 2
- Comparison of two populations (seeded/unseeded)
- Treatment vs. control
- Treatment vs. control (change from time 1 to time 2)

**Notes:**
Comparison of treatment to quantitative standard is used when you can definitely distinguish seeded plants from unseeded plants.  This way you know you are measuring the effect of the seeding rather than confounding it with natural regeneration.
Comparison of two populations is useful when you are not always be able to tell the difference between the seeded and unseeded plants.  This may occur in year one when seeded plants are very difficult to detect and in year three when they might be as large as some pre-existing plants.  Also, rhizomatous plants are often difficult to quantify or differentiate from seedlings.

## Flowchart for Graphical Analysis Density

**Flowchart for Graphical Analysis ****Density**

## Graphical Analysis (comparison to a standard)

- Specify quantitative objective
- Determine desired alpha level and precision
- Collect data at treatment plots
- Graph mean with confidence interval of desired width (typically 80 or 90%)
- Graphically compare to quantitative standard to determine which situation exists
- Need mean, standard deviation, and n
- Use ES&R Equation spreadsheet to help

## Graphical Analysis (comparison to a quantitative objective)

- Quantitative objective: 5 plants/m2
- Alpha level: 0.1 (90% Confidence Interval)
- X = 7 plants/m2
- S = 1.3 plants/m2
- N = 5

## Slide 15

## Comparison to a Quantitative Objective

| 
D:  The sample mean is above the quantitative objective, but the    lower limit of the confidence interval is below the objective.   |
| --- |

**Comparison to a Quantitative Objective**

## Comparison to a Quantitative Objective

| C:  The sample mean is above the objective but the lower confidence limit is below the objective.   |
| --- |

**Comparison to a Quantitative Objective**

## Comparison to a Quantitative Objective

| B: The sample mean is below the quantitative objective, but the upper limit of the confidence interval is above the objective.   |
| --- |

**Comparison to a Quantitative Objective**

## Comparison to a Quantitative Objective

| A: The sample mean and confidence interval (CI) fall below the objective.  Conclude that the objective has not been met. |
| --- |

**Comparison to a Quantitative Objective**

## Graphical Analysis (Treatment v. Control)

- Confidence interval of the difference between the treatment and control
- Uses the difference between the means of two treatments and constructs a single CI using the variance from both estimates (SE)
- A mean of 0 represents no difference between the two treatments
- Express the quantitative objective as an absolute value or as a multiple of the control.
- Use the mean and CI to make a determination of treatment effect

**Notes:**
Insert formula for confidence interval of the difference between two populations.

## Graphical Analysis  (Treatment v. Control)

- Using the ESR monitoring spreadsheet
- Specify the desired alpha level
- Enter the mean, standard deviation, and N from the data collected at the treatment and control plots
- Specify the level of quantitative objective (multiple of control or absolute difference)
- Make interpretation based on graphical analysis of the CI of the difference between the two treatments

**Notes:**
Insert formula for confidence interval of the difference between two populations.

## Graphical Analysis  (Treatment v. Control)

- Quantitative objective: twice that of control plot (2) – note that because it is a CI of difference the original amount is subtracted from quantitative objective
- Alpha level: 0.1 (90% Confidence Interval)

- Control
- X = 2.0
- S = 0.5
- N = 7

- Treatment
- X = 4.9
- S = 0.9
- N = 7

**Notes:**
Insert formula for confidence interval of the difference between two populations.

## Slide 23

**Notes:**
Insert formula for confidence interval of the difference between two populations.

## Treatment vs. Control

| A:  The mean and confidence interval for the difference between the two means is completely above the level of ecological significance (5 plants/m2).   |
| --- |

## Treatment vs. Control

|  |
| --- |

**B:  The difference of the mean between is above the level of ecological significance, but the lower confidence limit for the difference is below the level of ecological significance.  **

**Treatment vs. Control**

## Treatment vs. Control

| C:  The difference between the two means is below the level of ecological significance, but the upper confidence limit for the difference is above the level of ecological significance.   |
| --- |

**Treatment vs. Control**

## Treatment vs. Control 

**D:  The mean and confidence interval of the difference is below the level of ecological significance. Conclude that there is no ecologically significant difference between the control and treatment plots.**

**Treatment vs. Control **

## Treatment vs. Control

| E:  The mean of the difference is above zero, but the lower confidence limit is below 0 (no difference) and the upper confidence limit is above 5 plants/m2.   |
| --- |

**Treatment vs. Control**

## Treatment vs. Control

**F:  The mean of the difference is above 0, but the lower confidence limit is below 0 and the upper confidence limit is below the level of ecological significance.  **

**Treatment vs. Control**

## Flowchart for Graphical Analysis Density

**Flowchart for Graphical Analysis ****Density**

## Graphical Analysis (Treatment vs. Control T2-T1)

- Confidence interval of the difference in change between two time periods between treatment and control
- Uses the difference between the change in the means of treatment and control and constructs a single CI using the variance from both estimates (SE)
- A mean of 0 represents no difference in change between the two treatments
- Express the quantitative objective as an absolute value or as a multiple of the control.
- Use the mean and CI to make a determination of treatment effect

**Notes:**
Insert formula for confidence interval of the difference between two populations.

## Appears that there is a difference in year three when there actually was not.

**Appears that there is a difference in year three when there actually was not.**

**Accounting for initial difference in the degree of change**

## Slide 33

## Slide 34

## Change in Treatment vs. Control 

|  |
| --- |

**B:  The difference of the mean between is above the level of ecological significance, but the lower confidence limit for the difference is below the level of ecological significance.  **

**Change in Treatment vs. Control **

## Graphical Analysis (Treatment at two time periods, T2-T1)

- Confidence interval of the change between the two time periods
- Treats the two time periods as paired, reducing variability
- A mean of 0 represents no change between the two time periods
- Express the quantitative objective as the desired change between the two different time periods
- Use the mean and CI compared to the quantitative objective to make a determination of success

**Notes:**
Insert formula for confidence interval of the difference between two populations.

## Slide 37

## Reporting

**Paste graphs directly into reports and describe quantitative results e.g.**
**Perennial Grass Density**
- The density of perennial grasses is significantly greater in the treatment plots as compared to the control plots.  We are 90% confident that the difference is between 1.06 to 4.54 plants/m2 greater than the control plots with a mean of 2.8 plants/m2)

**Reporting**

## Reporting

- Link reports back to quantitative objectives
- Re-assess whether objectives were reasonable and possible reasons for success and failure.
- Make recommendations for future improvements to implementation and monitoring