# Presentation: 114558

## A New Approach For Testing the Accuracy

**of Predicted Vertebrate Occurrences**

**Sandra M. Schaefer**
**William B. Krohn**
**Raymond J. O’Connor**
- Maine Cooperative Fish and
- Wildlife Research Unit, and
- Department of Wildlife Ecology
- University of Maine, Orono

## intro

- Traditionally focused on single species being
- predicted on relatively small study areas

**Notes:**
To date, most attention on the testing of models to predict the occurrences of wildlife based on habitat characteristics (ie habitat models) has focused on assessing predictions for single species, one species at a time, generally on small study areas.

## Habitat Model Testing

**Habitat Model Testing**

**Multiple Species**

**Large geographic areas**

**Notes:**
With the advent of GAP analysis there became a need to testing habitat models for multiple species across large geographic areas.

## Traditionally, GAP models

**are tested by comparing model **
**predictions to site-specific **
**occurrence information**

- Omission Error  (OE) – failure
- to predict species known to occur
- on the site.

- Commission Error  (CE) – predicting
- the presence of species that do
- not occur on the site.

**Notes:**
Commission and omission error are calculated for each taxonomic class on each site.  An average is then taken across all sites to generate the statewide error.   

## ME-GAP Test Sites

**2. Nesowadnehunk Field, **
**    ****Baxter State Park**
**3. White Mountains National Forest**
**4. Sunkhaze Meadows **
**    ****National Wildlife Refuge**
**5. Holt Research Forest**
**6. Petit Manan **
**    ****National Wildlife Refuge**
**7. Rachel Carson **
**    ****National Wildlife Refuge**
**8. Moosehorn **
**    ****National Wildlife Refuge**
**9. Mount Desert Island/ **
**   ****Acadia National Park**

**ME-GAP Test Sites**

**Notes:**
For Example – in Maine GAP ....    Nine test sites were available for comparing to the vertebrate predictions.
Sites were wildlife refuges, a national park, a state park,  a state forest, a national forest, and 
a privately owned forestland.

Gap models were designed to capture the P/A of species breeding in inland Maine during at least 5 of the 10 year period (1984 – 1993)

To mirror this conservative definition for avian species only records which indicated confirmed breeding or frequency of occurrence were used.

Although based on shorter term field data then traditional check lists - Three smaller test sites were included (1,2,3) were to improve the geographic distribution of sites.  

## Site-Specific calculations

_**# of species predicted but not present on the test site**_
**Total number of species present on the site**

**OE =**

_**# of species present but not predicted on the test site**_
**Total number of species present on the site**

**Site-Specific calculations**

**Notes:**
Commission and Omission error were calculated for each taxonomic class on each site using these equations.
Mean error was then calculated for all taxonomic classes across all sites.  

Note how in the traditional approach the site is the focus of the test.

## Overall results of accuracy assessment on ME-GAP predicted 

**Overall results of accuracy assessment on ME-GAP predicted **
**species distributions.  Medians and ranges were calculated within **
**taxonomic group across all sites.**

**Notes:**
Low % of Omission
CE tended to be higher then OE – especially true for birds w/mean CE of 91.9%

## Challenges in Testing

-  Purpose of the model needs to be considered in testing,
-   because it may influence how the errors are interpreted.

-  Interpretation can  be complex.  Especially for commission
-   error where the cause may be either apparent or actual.

-  Errors can be caused by multiple factors (i.e test site
-    size, field inventory effort, species observability).

**Notes:**
define apparent and actual.

## Species-Specific Testing Approach

**Assessing model accuracy by calculating **
**OE and CE for each species across multiple **
**sites within the study area.**

**Notes:**
An Alternative approach to testing wildlife habitat models would be a species-specific approach.
Assessing model accuracy by calculating OE and CE for each species across multiple sites within the study area.  
Takes the focus off the sites and puts it on the species.

## Error Range

**                  ****were found during the field inventories. **

**incomplete – assumes that not all species on the site were             **
**                       ****found during the field inventories.  **

***Error Range***
***Difference in the highest and lowest possible OE and CE.***
***Calculated based on assumptions of field inventory completeness.***

***How complete are the field inventories?***
***(Nichols et al. 1998)***

## Objectives

**     ****regularly breed in Maine.**
**2) Determine if there is a relationship between the *****Error Range***** **
**     ****and the extent of a species distribution.**
**3) Determine if there is a relationship between the *****Error Range***** and **
**     ****how likely a species is to occur  during a field survey**
**    ****(for statewide species).**
**4) Compare the test results for the species-specific and site-specific **
**     ****approaches.**

**Objectives**

## Methods...Calculating the species-specific Error Range

## Potential Occurrence

**Species could potentially occur**
**on a site if the site is within the range limit.**
**Field survey data was also included if it indicates the species occurs on the site.**

**Notes:**
First step is to determine on which site the species can potentially occur

Site occurrence information was included in determining potential occurrence sites to prevent the chance of underestimating omission error.
Explain this using the incorrectly placed Sunkaze NWR which is outside the range limit yet the site indicates the species presence.  
This is an omission error yet it would go undetected unless we knew the site showed that the species breeds there.

## Species Occurrence Table

- 1 = Within ME-GAP Range     P = Presence     A = Absence

**Notes:**
For each species an occurrence table was created which was used to gather the data needed in calculating CE and OE.

The complete assumption used the data directly from the field surveys... they are complete

The incomplete assumption stated that the species absent on the survey are actually present but missed by the survey – A’ turned to P’s

## Species-Specific calculations

_**# of sites where the species was predicted but not present **_
**Total number of potential occurrence sites**

**OE  =**

_**# of sites where the species is present but not predicted**_
**Total number of potential occurrence sites**

**Species-Specific calculations**

**Notes:**
Because field inventories rarely if ever detect all species in an area we need a means to consider this likelihood in calculating test statistics for the species specific approach.  The two extreme of field surveys are totally complete and totally incomplete.

## Methods...Data Analysis

## 1) Spearman’s rho used to test for a relationship between a  

**      ****species distribution and the commission *****Error Range***** ****.**

**2) Spearman’s rho used to test for a relationship between an a **
**    *****a priori***** ranking system called Likelihood of Occurrence Ranks**
**    ****and the commission *****Error Range***** ****.**

**3) Site-specific commission error results compared to the  **
**    ****species-specific *****Error Range***** ****.**
**          **
**       **

**Notes:**
Grouped into primary breeding category (i.e., barren, early successional forest coniferous, forest deciduous, forest generalists, and wetland). 
Mean error for the site-specific method was plotted against mean error for species-specific method.
Two-way ANOVA was used to determine if a significant difference existed between the means of the 2 methods.  

## Likelihood of Occurrence Ranks (LOORs)

**(Boone and Krohn 1999)**

- LOORs are an *a priori* system of ranking species based on
- how likely they are to be seen during a standard wildlife inventory.

- Developed to help interpret the causes of commission error.
- (Schaefer and Krohn 2002).

- Atlas occurrence information was used to generate a  spatial
- incidence for each species.  The incidence came from dividing the
- number of  survey blocks in the atlas having confirmed or potential
- breeding by the number of survey blocks within the species range.

**Notes:**
Incidences where sorted and assigned a rank which became the species statewide LOOR.

## Results

## Frequency distribution of  OE and CE from predicted avian occurrences

**Assumes Complete Field Survey**

**Assumes  Incomplete Field Survey**

**Commission Error**

**Omission Error**

**Commission Error**

**Omission Error**

**Number of Species**

**Frequency distribution of  OE and CE from predicted avian occurrences**
**with the assumption of complete (A and C) and incomplete (B and D) field survey data.**

**B.**

**A.**

**C.**

**D.**

**Notes:**
179 species used in the analysis
Overall Sp-specific omission error was lower then CE

170 species never omitted under the assumption of complete inventories
W/the assumption of incomplete field surveys OE increased slightly.... 151 species never omitted. 
Despite the slight increase there were too few species omitted to use the error ranges in any analyses.

CE was much higher.  Only 25 (of 179) species were never over-predicted and 21 of the 179 were over predicted 100% of the time.
assuming the data were incomplete returned no CE.... all A’s were changed to P’s – can’t have CE.

## Commission Error Range

-           Number of potential occurrence sites

- Commission *Error Range* for all avian species across all test sites.
- rho = -0.583        P < 0.001

**Notes:**
Species with small distributions had significantly higher CE then the more broadly distributed species.  

## Relationship between commission Error Range and the Likelihood of 

- Occurrence Ranks for those species that essentially occur statewide.
- rho = -0.657      P < 0.001

- Commission Error Range

- Likelihood of Occurrence Ranks

**Notes:**
For those species with potential occurrence on all 9 sites, a negative correlation was found between CER and the 
Likelihood of Occurrence Ranks ( Rho = -0.657, P < 0.000)

## ANOVA:                 F Ratio        p-value

**Method                     20.6              0.000**
**Habitat                     4.42              0.001**
**Method*Habitat     1.54               0.180**

**Notes:**
Mean CE in each habitat group tended to higher for the site-specific then for the species specific approach.  
However this difference was only significant across all habitat categories....with F = 19.172 and P > 0.001.

## Conclusions

-    to assess the overall predictive quality of the habitat models, as well
-    as determine the variability of error for each species.
-  Commission *Error Range* was significantly correlated with species
-   distribution as well as with how likely a species was to be observed on
-    a field inventory.
-  The species-specific method generated significantly lower commission
-    error then the site-specific method.

**Conclusions**

**Notes:**
Bullet 1. read

- as in the site specific appraoch CE was higher then OE... with only 9 species reporting OC...models are not missing many species.

bullet 2. read....
- variability in CER across species comes from our inability to know understand and model every aspect of species habitat needs to determine with 100%  
accuracy where it will occur.
- correlation between distribution and CE may be related to habitat variablity, availablilty, or both.  Species that are broadly distributed may have a wider variety of habitats which suit their biological requirements.

## Conclusions (cont)

-   likelihood of occurrence then the most likely cause for the over
-   prediction is in the model.
-  However, if a species has a low likelihood of occurrence
-    and a high *Error Range*** **, then the over prediction error is likely due to
-    having incomplete field surveys for the species.

**Conclusions ****(cont)**

**Notes:**
Before bullets.... species distribution was only part of the explanation of CE....because CE still reported for broadly distributed species.
the species specific approach allows us the opportunity to infer that.... bullet 1 and bullet 2.

## Summary

- Both methods are influenced by the data available for use in the
-        testing process.  They also need data for the temporal and
-        spatial scale being assessed.
- The site-specific method provides a generalized idea of how well
- 	the models are capturing species presence and absence
-      and across the entire state.
- The species-specific approach gives a more detailed description
-  	of which species are reporting the highest levels of error.
- 	This helps to answer the question of why the predictive error
-        is being reported.

**Notes:**
Both methods indicate model performance... both models are also influenced by the data used in testing.

Species specific approach tells exactly which species are being over-predicted and by how much.

## Take Home

**Because species-specific and the site-specific methods provide **
**different information about the quality of predicted occurrences, **
**we recommend using both when assessing model accuracy .**

**Notes:**
As both methods use essentially the same data, presenting both sets of test statistics would add little work to the individual Gap projects.

## Acknowledgements

- Daniel J. Harrison
- Steve R. Sader
- Randall B. Boone
- William Halteman

**Supporting Organizations and Agencies**
- Gap Analysis Program, USGS BRD
- Maine Cooperative Fish and Wildlife Research Unit
- Department of Wildlife Ecology
- University of Maine