Evaluation of species distribution models by resampling of sites surveyed a century ago by Joseph Grinnell

Ecography 36: 1017–1031, 2013 doi: 10.1111/j.1600-0587.2013.00107.x © 2013 Th e Authors. Ecography © 2013 Nordic Society Oikos Subject Editor: David Nogues-Bravo. Accepted 25 January 2013 Evaluation of species distribution models by resampling of sites surveyed a century ago by Joseph Grinnell Adam B. Smith , Maria J. Santos , Michelle S. Koo , Karen M. C. Rowe , Kevin C. Rowe , James L. Patton , John D. Perrine , Steven R. Beissinger and Craig Moritz A. B. Smith (adam@adamlilith.net), M. S. Koo, J. L. Patton, S. R. Beissinger and C. Moritz, Museum of Vertebrate Zoology, 3101 Valley Life Sciences Building, Univ. of California, Berkeley, CA 94720-3060, USA. Present address of ABS: Center for Conservation and Sustainable Development, Missouri Botanical Garden, PO Box 299, Saint Louis, MO 63166, USA. SRB also at: Dept of Environmental Science, Policy and Management, 130 Mulford Hall, Univ. of California, Berkeley, CA 94720-3114, USA. – M. J. Santos, Spatial History Project and Bill Lane Center for the American West, History Dept, Stanford Univ., Stanford, CA 94305-2055, USA. – K. M. C. Rowe and K. C. Rowe, 4 Sciences Dept, Museum Victoria, GPO Box 666, Melbourne 3001, VIC, Australia. – J. D. Perrine, Biological Sciences Dept, California Polytech- nic State Univ., San Luis Obispo, CA 93407-0401, USA. Species distribution models (SDMs) are commonly applied to predict species ’ responses to anticipated global change, but lack of data from future time periods precludes assessment of their reliability. Instead, performance against test data in the same era is assumed to correlate with accuracy in the future. Moreover, high-conﬁ dence absence data is required for testing model accuracy but is often unavailable since a species may be present when undetected. Here we evaluate the performance of eight SDMs trained with historic (1900 – 1939) or modern (1970 – 2009) climate data and occurrence records for 18 mammalian species. Models were projected to the same or the opposing time period and evaluated with data obtained from surveys conducted by Joseph Grinnell and his colleagues in the Sierra Nevada of California from 1900 to 1939 and modern resurveys from 2003 to 2011. Occupancy modeling was used to conﬁ dently assign absences at test sites where species were undetected. SDMs were evaluated using species ’ presences combined with this high-conﬁ dence absence (HCA) set, a low-conﬁ dence set in which non-detections were assumed to indicate absence (LCA), and ran- domly located ‘ pseudoabsences ’ (PSA). Model performance increased signiﬁ cantly with the quality of absences (mean AUC SE: 0.76 0.01 for PSA, 0.79 0.01 for LCA, and 0.81 0.01 for HCA), and apparent diﬀ erences between SDMs declined as the quality of test absences increased. Models projecting across time performed as well as when pro- jecting within the same time period when assessed with threshold-independent metrics. However, accuracy of presence and absence predictions sometimes declined in cross-era projections. Although most variation in performance occurred among species, autecological traits were only weakly correlated with model accuracy. Our study indicates that a) the quality of evaluation data aﬀ ects assessments of model performance; b) within-era performance correlates positively but unreliably with cross-era performance; and c) SDMs can be reliably but cautiously projected across time. Anthropogenic climate change promises to rewrite the bio- geography of Earth ’ s species, with some expected to gain, some to lose, and some to shift their current distributions. As a result, conservation planners require reliable methods to project future distributions of species of concern and to prioritize conservation eﬀ ort (Th omas et al. 2004, Carroll et al. 2010, Ogawa-Onishi et al. 2010, Saupe et al. 2011). Species distribution models (SDMs), which correlate species occurrence data with climate variables and other factors indicative of habitat quality to produce maps of environ- mental suitability, are frequently used for such projections. Unfortunately, the reliability of projecting SDMs across time periods relevant to conservation remains largely unknown (Ara u jo et al. 2005a, b, Dormann 2007, Elith and Leathwick 2009, Kharouba et al. 2009). Scores of studies have assessed the performance of diﬀ erent SDM algorithms using within-era evaluation, testing models against records from the same region and time period used to train the models (Elith et al. 2006, Hijmans and Graham 2006, Syphard and Franklin 2009). However, within-era assessments of SDMs may give overly optimistic estimates of cross-era performance (Ara u jo et al. 2005a, Hijmans 2012). While cross-era evaluation increases the independence between training and test data, it requires data from both time periods of interest, which are rarely available for time spans relevant to conservation planning (i.e. several decades or more). SDMs should be less reliable when projecting across time than within the same era for reasons related to both biology and modeling (Ara u jo et al. 2005a, b, Dobrowski et al. 2011). From a biological perspective model perfor- mance will be diminished if species distributions are not in