Journal of Mammalogy, XX(X):1–9, 2022
https://doi.org/10.1093/jmammal/gyac021

Which mammals can be identified from camera traps andcrowdsourced photographs?

North Carolina Museum of Natural Sciences, Raleigh, NC 27601, USA
Department of Forestry and Environmental Resources, NC State University, Raleigh, NC 27695, USA
3
Department of Fish, Wildlife, and Conservation Biology, Colorado State University, Fort Collins, CO 80521, USA
4Illinois Natural History Survey, University of Illinois, Champaign, IL 61820, USA
5
Department of Biology, Angelo State University, San Angelo, TX 76909, USA
6
Division of Mammals, Department of Vertebrate Zoology, National Museum of Natural History, 10th and Constitution Avenue
NW, Washington, DC 20560, USA
7
Division of Biology, Kansas State University, Manhattan, KS 66506, USA
8
Department of Biology and Chemistry, Morehead State University, Morehead, KY 40351, USA
9
Florida Museum of Natural History, University of Florida, Gainesville, FL 32611, USA
10
Department of Biology, University of North Carolina Greensboro, Greensboro, NC 27408, USA
11
Department of Mammalogy, University of Alaska Museum, University of Alaska Fairbanks, Fairbanks, AK 99775, USA
12
Department of Ecology & Evolutionary Biology and Museum of Zoology, University of Michigan, Ann Arbor, MI 48108, USA
13
School of the Environment, Washington State University, Pullman, WA 99164, USA
14
Department of Anthropology, Yale University, New Haven, CT 06520, USA
1
2*To whom correspondence should be addressed: rwkays@ncsu.edu.
While museum voucher specimens continue to be the standard for species identifications, biodiversity data are increasingly
represented by photographic records from camera traps and amateur naturalists. Some species are easily recognized in thesepictures, others are impossible to distinguish. Here we quantify the extent to which 335 terrestrial nonvolant North American
mammals can be identified in typical photographs, with and without considering species range maps. We evaluated all pairwise
comparisons of species and judged, based on professional opinion, whether they are visually distinguishable in typical picturesfrom camera traps or the iNaturalist crowdsourced platform on a 4-point scale: (1) always, (2) usually, (3) rarely, or (4) never.
Most (96.5%) of the 55,944 pairwise comparisons were ranked as always or usually distinguishable in a photograph, leaving
exactly 2,000 pairs of species that can rarely or never be distinguished from typical pictures, primarily within clades such as
shrews and small-bodied rodents. Accounting for a species geographic range eliminates many problematic comparisons, suchthat the average number of difficult or impossible-to-distinguish species pairs from any location was 7.3 when considering
all species, or 0.37 when considering only those typically surveyed with camera traps. The greatest diversity of difficult-todistinguish species was in Arizona and New Mexico, with 57 difficult pairs of species, suggesting the problem scales withoverall species diversity. Our results show which species are most readily differentiated by photographic data and which taxa
should be identified only to higher taxonomic levels (e.g., genus). Our results are relevant to ecologists, as well as those using
artificial intelligence to identify species in photographs, but also serve as a reminder that continued study of mammals throughmuseum vouchers is critical since it is the only way to accurately identify many smaller species, provides a wealth of data
unattainable from photographs, and constrains photographic records via accurate range maps. Ongoing specimen voucher
collection, in addition to photographs, will become even more important as species ranges change, and photographic evidence
alone will not be sufficient to document these dynamics for many species.
Key words: artificial intelligence, biodiversity, camera trap, citizen science, curation, identification, iNaturalist, photograph, range
map, voucher© The Author(s) 2022. Published by Oxford University Press on behalf of the American Society of Mammalogists, www.mammalogy.org.

1

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

Roland Kays,1,2,* Monica Lasky,3 Maximilian L. Allen,4 Robert C. Dowler,5 Melissa T.R. Hawkins,6
Andrew G. Hope,7 Brooks A. Kohli,8 Verity L. Mathis,9 Bryan McLean,10 Link E. Olson,11
Cody W. Thompson,12 Daniel Thornton,13 Jane Widness,14 and Michael V. Cove1

2

JOURNAL OF MAMMALOGY

simultaneously considers the characteristics used to distinguishbetween each species pair and whether those characteristics
would be captured in a typical photograph (1) always, (2) usually, (3) rarely, or (4) never (Fig. 1). To evaluate the importanceof geography in aiding species identification, we compare geographically naive scores of identification difficulties with mapsthat plot the overlapping geographic ranges for species pairs
judged to be difficult or impossible to identify from photographs. Our results set a baseline for how well we should expect people and AI to identify mammal species and highlightwhich species and geographic areas can be most confidently
studied with photographs.

Materials and Methods
We used a list of North American mammals (north of
Mexico), and their range maps, from International Union forConservation of Nature (IUCN 2020) for all analyses. We focused on terrestrial, nonvolant species, and excluded domestic
animals and invasive species with small ranges, as these would
inflate the overall species numbers unnecessarily, but included
wide-ranging, long-established invasives, that is, Rattus rattus,
R. norvegicus, Mus musculus, Sus scrofa, and Myocastor
coypus. We expanded the list to include every pairwise species
comparison, and for each comparison we collectively scoredthe difficulty of distinguishing them in a photograph as follows:(1) easy with most pictures, (2) possible if the picture captures
a distinguishing feature, which should happen frequently, (3)possible if the picture captures a distinguishing feature, whichshould happen rarely, or (4) impossible (Fig. 1). We did not
systematically evaluate photographs, but assessed the overall
physical similarity of species pairs using a regional field guide(Kays and Wilson 2009), and based on expert opinions for eachtaxonomic group from the coauthors on this paper based on
their experience working with living animals, with museum
specimens, and with photographs. Each author evaluated thespecies groups for which they had expertise, with each grouphaving at least two experts. The penultimate draft of the matrix
was discussed by the authors and any disagreements were resolved by consensus. We considered all species pairs regardlessof whether they co-occur, but we take geographic range overlap
into account in a secondary analysis (see below).
Our scoring of 1, 2, 3, or 4 presumes a typical picture from acamera trap or citizen scientist without special efforts to capture
subtle distinguishing characteristics. This assumption makes
our results relevant to the great majority of photo-vouchereddata available. However, we note that specialized camera trap
setups (e.g., baited, close-focus, white-flash, MCCleery et al.
2014; Herrera et al. 2021) could obtain identifiable photographs
for some groups more often than we assume here. Additionally,
we did not account for identifiability of some species by experts
who know about, and can photograph, a distinguishing characteristic (e.g., while handling an animal). Although we did not
specifically examine photographs for this assessment, thesejudgments of whether a species pair could always, usually,rarely, or never be distinguished were made based on the authors’ professional experience with photos from camera traps

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

The rise of born-digital biodiversity data has led to more
and more photographs serving as visual vouchers for species occurrences (Kays et al. 2020). This is especially true on
crowdsourced platforms, such as iNaturalist (inaturalist.org),
where georeferenced photographs are posted and collectively
identified by the user community. If a photograph in iNaturalist
is identified to species (or occasionally genus or family) by thecommunity (>2 people), and is georeferenced, time-stamped,
and not captive/cultivated, the record is judged as “research
grade” and aggregated in larger data portals, including the
Global Biodiversity Information Facility (GBIF). The scale ofcurrent observations is impressive, with over 3 million photovouchered Animalia records from iNaturalist archived at GBIF
in 2019, compared to ~209,000 new museum specimens (GBIF.
org 2020). Camera traps provide another growing collection ofphoto-vouchered biodiversity records, primarily for birds andmammals. Recent studies have shown the utility of classifying
millions of photographs from thousands of locations acrossstates or countries (Cove et al. 2021; Lasky et al. 2021).
Although there has been extensive discussion—and controversy—about whether photographs should be used as typespecimens when describing a new species (Krell and Marshall2017), there has been little discussion about when these are appropriate for other applications. In particular, some species areeasy to identify through photographs, whereas others are impossible because they require examination of detailed morphological characters difficult or impossible to see in photos, color
differences missed by nocturnal camera trap pictures, or evengenetic evidence (Kays and Wilson 2009). Smaller species—
which comprise the majority of mammalian diversity—are
also more difficult to photograph, less likely to trigger camera
traps, and less likely to have obvious external features usefulfor identification. Indeed, Potter et al. (2018) found that smaller
mammal species were more difficult to distinguish than larger
mammals in camera trap pictures from Australia. Yet the importance of accurately identifying mammals from photographs
has not been systematically assessed.
The monumental task of identifying species in photographs
globally is increasingly being performed by artificial intelligence (AI). AI uses human-identified photographs to train
algorithms to make subsequent identifications automatically(Norouzzadeh et al. 2021). The accuracy of these tools has
improved with larger training data sets and increasingly sophisticated algorithms (Ahmed et al. 2020), including the useof spatial priors to account for geographic range (Mac Aodha
et al. 2019). However, these efforts usually start from the assumption that all species can be distinguished and accurately
identified in a photograph. Although we typically do not know
if algorithms are using the same physical criteria to recognize
species as humans, they are working off the same visual image,and thus would encounter the same challenges humans facewhen trying to distinguish visually similar species.
Here we quantify how often this assumption is violated
by evaluating which species of 335 North American terrestrial mammals could theoretically be identified from typicalcamera trap or citizen science photographs. We use a 4-point
scale based on the consensus expert opinion of the authors that

KAYS ET AL.—WHAT IS A PICTURE WORTH?

3

and iNaturalist photos. Our approach also inherently considers
photo quality, in that our experts took into account factors suchas nocturnal species typically being photographed in black andwhite by camera traps, and many iNaturalist contributors not
typically knowing and photographing obscure morphologicaltraits needed to identify some species. Although a more quantitative approach might be possible to evaluate the quality ofavailable photos for particular species pairs (Thornton et al.
2019), this was not practical for the large species list and enormous photo collections we considered. Thus, we consider ourapproach as a first approximation and encourage future work
considering photo quality and accuracy of particular taxon
identifications.
Based on the final matrix, we computed a Euclidean distance matrix representing all pairwise differences in species
identifiability as input to conduct a hierarchical clustering analysis, using the function hclust in R v4.0.3 (R Core Team 2020).
To aid in our goal of visualizing among-cluster distances, weused the complete linkage method for dendrogram construction(the default in hclust). The resulting dendrogram was plotted
as a phylo object using functions in the ape v 5.4-1 package in
R (Paradis and Schliep 2019). To map species that are appropriate for sampling by camera traps, we removed genera thatare typically too small to trigger a camera trap (<90 g, i.e., weremoved Neotamias, Dipodomys and smaller, but kept Tamias,
Glaucomys and larger) and subterranean species (i.e., gophersand moles). We used body mass data from Soria et al. (2021).
We overlapped range maps of different species groups using
North American mammal species ranges downloaded from the
IUCN Red List of Threatened Species (IUCN 2020). We created raster layers (50 km × 50 km pixel size) for each individual
mammal species and then extracted the areas of overlap for
species that were difficult to identify. We created all spatial figures in R version 4.0.3 (10 October 2020) using packages rgdal(version 1.5-23), tidyverse (version 1.3.0), sf, dplyr, raster, andggplot2 (Wickham et al. 2019; Hijmans et al. 2013; R CoreTeam 2020).

Results
Most (96.5%) of the pairwise comparisons (Ntotal = 55,944) of
the 335 species were considered easily or usually distinguishable from photographs, leaving 2,000 pairs of species that are
rarely or never distinguishable from typical pictures (Table
1; Supplementary Data SD1). Most of these difficult comparisons involve shrews, small rodents, or rabbits (Fig. 2A),

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

Fig. 1.—Criteria used to classify each pair of species as to whether they could be distinguished in a typical camera trap or citizen science photograph always, usually, rarely, or never. Top four pictures credit to eMammal, bottom four to iNaturalist users Dario, Christoph Moning, Ken
Kneidel, and John Sankey.

4

JOURNAL OF MAMMALOGY

Table 1.—Scores for whether a pair of species could be distinguished by a photograph for pairs of 335 terrestrial North American
mammals.
Pairwise species comparisons (n/%)
Can be distinguished
via photograph?
EasilyUsually
RarelyNever
Total

All species

Cameratrappable species

52,899/94.6%
1,046/1.9%
933/1.7%
1,066/1.9%
55,9449,480/96.0%
196/2.0%
97/1.0%97/1.0%9,870

or impossible species pairs, ranging from 0 in the southeasternand midwestern states to 9 in portions of the western UnitedStates. Of these, the most difficult groups were Sylvilagus andNeotamias (Supplementary Data SD3).

Discussion
This is a first-ever assessment of the potential to identify agroup of animals from photographs, which is timely given thegrowing importance of born-digital biodiversity data being
identified by either AI or crowdsourcing (Kays et al. 2020). Thismeasure is also relevant for the growing effort to use AI to identify animals in photographs because assessments of algorithm
accuracy should account for the fact that species-level identification is likely impossible for some groups. We found 2,000
pairs of North American mammal species that would be rarelyor never distinguishable in typical photographs. However, themajority of these difficult species are not known to overlap geographically, such that the highest number of indistinguishable
species pairs at any one site was constrained to southeastern
Arizona (57 pairs). As expected, most of the confusing species
were smaller-bodied mammals. Considering only the larger
species, which are typically studied with camera traps, mostsites have between 0 and 1 difficult or impossible-to-identify
pairs of species. These results are important for those who use
photo-based biodiversity data, because even “research-grade”
observations of small species from iNaturalist might be overconfident in species identifications, and care should be taken
to vet the photographs before using them in spatial modeling.
Our results confirm that photographic vouchers can be useful
to identify many larger mammal species, but also emphasize
the importance of traditional museum collections and continued biological surveys by experts. For instance, monitoring
the distribution and abundance of large mammals, especiallycarnivores and ungulates, with camera trap and crowdsourced
data should be possible at the species level. However, traditional collections are necessary for smaller species for accurate
species identification and to map the potentially dynamic geographic ranges of such species in the Anthropocene (in addition
to the wealth of additional “extended” data voucher specimens provide). AI should be broadly useful to classify photographs of most North American mammal species, although
the accuracy of both human and machine identifications stillneeds to be guided by extensive professional training data sets.
Additionally, our results do highlight some areas of the western
United States where extra caution is needed even for mediumto large-bodied species, such as rabbits and hares.
Perhaps the most significant challenges with camera traps
are the groups that are large enough to trigger camera traps, butdifficult to identify to the species level based on photographs
alone. Particularly problematic groups are the cottontails(Sylvilagus) and woodrats (Neotoma) in the southern UnitedStates and arid Mexico. The western chipmunks (Neotamias)
average smaller than our weight cutoff (90 g) to be considereda target of camera traps, but are still common in some camera
trap data sets (Cove et al. 2021). Most of these confusing
pairs can be distinguished based on careful consideration of

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

showing that smaller species and more speciose genera are
more likely to be difficult to identify (Fig. 2B). Considering
only species likely to be surveyed with camera traps, 98.0%
of species pairs should be easily or usually distinguished byphotographs (Table 1).
The clustering analysis provides a visualization of which
species groups are most similar in appearance (Fig. 3;
Supplementary Data SD2). In most of the shrews and smallerbodied rodents, species were ranked as being usually identifiable at the generic level, but with species-level identification
judged as rarely or never possible from photographs alone.
Conversely, higher-level groups of small mammals could always be distinguished from each other including the vole-likeanimals (small ears, eyes, and short tails), pocket mice (medium ears and eyes, and long tails), deer mice and allies (largeeyes and ears), kangaroo rats (large feet and tail, and bipedal
posture), shrews (small size, absent or reduced ears, and pointynose), rats (large size), moles (distinctive shape), and gophers(distinctive shape). The squirrels were separated into ninegroups that were judged as always distinguishable: chipmunklike, unstriped ground squirrels, antelope ground squirrels,
spotted ground squirrels, tree squirrels, red squirrels, flying
squirrels, marmots, and prairie dogs. However, within eachgroup of squirrels, many species are very difficult to distinguish, especially the chipmunks and unstriped ground squirrels,
which include many species for which skeletal or genetic characters are required. The lagomorphs were split into four groups
that could be easily distinguished from one another: cottontails,
Northern hares, jackrabbits, and pikas. Most (n = 10) of thespecies within cottontails (Sylvilagus) were judged as indistinguishable in photographs, whereas most of the jackrabbits wereranked as usually identifiable. The carnivores and ungulates included larger species that were easier to identify, including 25
unmistakable species and nine species pairs that were usually
identifiable.
The comparison of all possible species in North America revealed 2,000 pairs of species that would be difficult (n = 934)
or impossible (n = 1,066) to identify via photograph, but our
secondary mapping exercise shows that accounting for geographic range dramatically reduces this number (Fig. 4). The
average number of difficult or impossible (3 or 4) pairs ofspecies from any 50 km × 50 km grid cell was 7.3, ranging
from 0 in southern Florida to 57 in the southwestern UnitedStates. When considering only larger species likely to be detected by camera traps, the average location had 0.37 difficult

KAYS ET AL.—WHAT IS A PICTURE WORTH?

5

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

Fig. 2.—The extent to which genera of terrestrial North American mammals are identified plotted by taxonomy (A) and average body mass (B,exponential trend line r2 0.33, P < 0.0001). Difficulty is measured as the number of species pairs that would be impossible or difficult (class 3 or
4) to identify via photographs.

6

JOURNAL OF MAMMALOGY

geographic range (Supplementary Data SD3), but many ofthese boundaries are shifting as a result of climate change(e.g., Moritz et al. 2008). In most of the difficult-to-identify
species, individuals are recognizable at the generic level. Thislevel of identification could still be useful for spatial modelingif the records are used as a covariate for distribution models
of other species (e.g., a measure of prey or competitor abundance) or quantifying functional richness or ecosystem processes. Studies focusing on groups such as chipmunks might
be able to improve identification with camera traps designed
to take close-up color photographs (Gracanin et al. 2019) thatcan capture species-specific pelage markers or videos that include audio to capture their unique vocalizations (Gannon and
Lawlor 1989; Kays and Wilson 2009). Indeed, recent workby McKibben and Frey (2021) show how one pair of similar
chipmunk species, ranked 3 (rarely distinguishable in typical

pictures) in our study, can reliably be distinguished in camera
trap pictures with special camera protocols and training of staff
on distinguishing characters from museum specimens.
The use of AI to identify species in images is a rapidly
growing field and is now integrated into many platforms fordifferent uses. For example, AI is used by Wildlife Insights(https://www.wildlifeinsights.org) to speed the processing ofcamera trap images (Ahumada et al. 2019) and by iNaturalist
to suggest species-level and genus-level IDs (Van Horn et al.
2018; Ueda 2020). Creating these tools requires the use of annotated images to train algorithms to identify species, followedby an assessment of their accuracy by testing on a second, independent set of annotated images (He et al. 2016). Our results
show the importance of not restricting these tools to specieslevel identifications, which are impossible for some taxa, but
to report higher taxonomic-level results for the most difficult

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

Fig. 3.—Hierarchical clustering diagram for terrestrial North American mammals based on how difficult it is to distinguish a pair of species basedon a photograph. The color of branch tips indicates whether a taxon should be always (dark blue), usually (light blue), rarely (pink), or never (red)distinguishable in typical photographs. The large groups of red tips indicate species groups that are practically impossible to tell apart in photographs, whereas the single blue branch tips on the bottom right represent one-of-a-kind species that are unmistakable. Color of nontip branches
shows to what extent higher taxonomic groups can be distinguished. Artwork from Kays and Wilson (2009). Not all groups are pictured. All species names are presented in Supplementary Data SD2.

KAYS ET AL.—WHAT IS A PICTURE WORTH?

7

groups. We recommend that these algorithms group species
that can never be identified by most photographs (class 4 inour ranking) at the generic level to avoid compounding dubious
species-level identifications across other biodiversity data aggregators. Accounting for the underlying possibility of distinguishing pairs of species, identifying some groups only to
the generic level, and including geographic information about
species distribution (Mac Aodha et al. 2019) should result in
improved algorithmic performance and realistic metrics about
their accuracy.
Our analyses emphasize that photo-vouchered biodiversity
records are a supplement to, but not a replacement of, traditional museum specimen collections. Physical specimens havemyriad other uses unavailable from photographs, including
genetic, isotopic, disease, and detailed morphological studies(Dunnum et al. 2018; Thompson et al. 2021). As expected, ourresults show that many smaller mammals are impossible ordifficult to identify by photograph, making species-specific research on those taxa questionable from photographs alone. Our
results also suggest that geographic range can, in many cases,
help distinguish similar species, but this depends on range
maps created from densely sampled physical vouchers thatallow confident species identification and range delimitation.
Detecting range shifts in response to global change for species groups that are difficult to identify thus requires continued
active collection and museum vouchering, especially at longterm monitoring sites, which would pay extra dividends by alsoenabling updated range-assisted identification of some photovouchered data. Additionally, for species that can be identified
by photographs, the rich data available from iNaturalist could
highlight areas of species expansion that should be priorities
for additional trapping surveys. Such “ground proofing” to obtain voucher specimens will verify changing distributions, andallow for the study of both ecological and evolutionary dynamics along range peripheries. Finally, species description
and geographic range delimitation is still an active process formany taxonomic groups (including mammals) in many partsof the world (Burgin et al. 2018), and physical specimens will
be required to voucher these discoveries and highlight distinguishing phenotypic traits that may or may not be evident inphotographic data.
Our results are subjective in that they represent a consensus
of the authors on the relative difficulty of distinguishing pairsof species from photographs alone, given our knowledge

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

Fig. 4.—Maps showing the total diversity of terrestrial mammals in North America (A) and the diversity of species that are likely to be detectedin camera traps based on body size (B). Map (C) shows how many species overlap geographically in an area that would be difficult or impossible
to identify based on a photograph for all mammals, and (D) shows this for those species typically surveyed with camera traps.

8

JOURNAL OF MAMMALOGY

Acknowledgments
Thanks to all the dedicated scientists and volunteers who havecontributed specimens and photographs to public repositories
that make this kind of work possible, and the results relevant.

Supplementary Data
Supplementary data are available at Journal of
Mammalogy online.
Supplementary Data SD1.—Matrix with ranks of how difficult all pairs of 335 species of terrestrial North American
mammals are to distinguish from each other based on typicalphotographs from camera traps or citizen science projects.
Scores show if a pair of species would always (score = 1),
usually (2), rarely (3), or never (4) be distinguishable inphotographs.
Supplementary Data SD2.—Hierarchical clustering diagram
for terrestrial North American mammals based on how difficult
it is to distinguish a pair of species based on a photograph withall species names provided for Fig. 3. The color of branch tips
indicates whether a taxon should be always (dark blue), usually (light blue), rarely (pink), or never (red) distinguishable intypical photographs. The large groups of red tips indicate species groups that are practically impossible to tell apart in photographs, whereas the single blue branch tips on the bottom right
represent one-of-a-kind species that are unmistakable. Color of
nontip branches show to what extent higher taxonomic groups
can be distinguished. Species names of the same genus are colored the same.
Supplementary Data SD3.—Maps showing the range overlap
for species of mammals from North America within the more
speciose taxa that are difficult (score 3 or 4) to identify via
photographs.

Literature Cited
Ahmed A., Yousif H., Kays R., He Z. 2020. Animal species classification using deep neural networks with noise labels. Ecological
Informatics 57:101063.
Ahumada J.A., et al. 2019. Wildlife insights: a platform to maximize the potential of camera trap and other passive sensor wildlife
data for the planet. Environmental Conservation 47:1–6.
Burgin C.J., Colella J.P., Kahn P.L., Upham N.S. 2018. How many
species of mammals are there? Journal of Mammalogy 99:1–14.

Cove M.V., et al. 2021. SNAPSHOT USA 2019: a coordinated national camera trap survey of the United States. Ecology
102:2019–2020.
Dunnum J.L., et al. 2018. Mammal collections of the Western
Hemisphere: a survey and directory of collections. Journal of
Mammalogy 99:1307–1322.
Gannon W.L., Lawlor T.E. 1989. Variation of the chip vocalization of
three species of Townsend chipmunks (genus Eutamias). Journal of
Mammalogy 70:740–753.
GBIF.org. 2020. GBIF occurrence download. https://doi.
org/10.15468/dl.uvc6sr.
Gooliaff T., Hodges K.E. 2018. Measuring agreement among experts in classifying camera images of similar species. Ecology and
Evolution 8:11009–11021.
Gooliaff T.J., Hodges K.E. 2019. Error rates in wildlife image classification. Ecology and Evolution 9:9:6738–6740.
Gracanin A., Gracanin V., Mikac K.M. 2019. The selfie trap: a novel
camera trap design for accurate small mammal identification.
Ecological Management & Restoration 20:156–158.
He Z., Kays R., Zhang Z., Ning G., Huang C., Han T.X., Millspaugh J.,
Forrester T., McShea W. 2016. Visual informatics tools for supporting large-scale collaborative wildlife monitoring with citizen
scientists. IEE Circuits and Systems Magazine 16:73–86.
Herrera D.J., Moore S.M., Herrmann V., McShea W.J., Cove M.V.
2021. A shot in the dark: white and infrared LED flash camera traps
yield similar detection probabilities for common urban mammal
species. Hystrix, the Italian Journal of Mammalogy 32:72–75.
Hijmans R.J., van Etten J., Mattiuzzi M., Sumner M., Greenberg J.A.,
Lamigueiro O.P., Bevan A., Racine E.B., Shortridge A. 2013.
Raster package in R, p. 2–2.
IUCN (International Union for Conservation of Nature). 2020.
Class Mammalia from North America. (spatial data). The IUCNRed List of Threatened Species. Version 2020-2. https://www.
iucnredlist.org.
Kays R., McShea W.J., Wikelski M. 2020. Born digital biodiversity
data: millions and billions. Diversity and Distributions 26:644–648.
Kays R.W., Wilson D.E. 2009. Mammals of North America. 2nd ed.
Princeton University Press, Princeton, New Jersey, USA.
Krell F.T., Marshall S.A. 2017. New species described from photographs: yes? no? sometimes? A fierce debate and a new declarationof the ICZN. Insect Systematics and Diversity 1:3–19.
Lasky M., et al. 2021. Carolina critters: a collection of camera
trap data from wildlife surveys across North Carolina. Ecology
102:e03372.
Mac Aodha O., Elijah C., Perona P. 2019. Presence-only geographical priors for fine-grained image classification. In: Proceedingsof the IEEE/CVF International Conference on Computer Vision,p. 9596–9606.
MCCleery R.A., Zweig C.L., Desa M.A., Hunt R., Kitchens W.M.,
Percival H.F. 2014. A novel method for camera-trapping small
mammals. Wildlife Society Bulletin 38:887–891.
McKibben F.E., Frey J.K. 2021. Linking camera-trap data to taxonomy: identifying photographs of morphologically similar chipmunks. Ecology and Evolution 11:9741–9764.
Moritz C., Patton J.L., Conroy C.J., Parra J.L., White G.C.,
Beissinger S.R. 2008. Impact of a century of climate change on
small-mammal communities in Yosemite National Park, USA.
Science 2899:261–264.
Norouzzadeh M.S., Morris D., Beery S., Joshi N., Jojic N., Clune J.
2021. A deep active learning system for species identification andcounting in camera trap images. Methods in Ecology and Evolution
12:150–161.

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

of mammals, their external characters, and our experiences
working with pictures typical of camera traps or crowdsourcing.
Although it is conceivable that a different group of experts
would come to slightly different conclusions for some species pairs, we think that these results are robust in identifying
the most problematic groups, the spatial distributions of thesepairs, and the broad patterns across taxa. Despite these limitations, we hope others will build on this work to empirically explore the challenges of identifying particular groups of interest(e.g., Gooliaff and Hodges 2018, 2019; Thornton et al. 2019)with the community goal of increasing overall accuracy ofidentifications in digital biodiversity data sets at a global scale.

KAYS ET AL.—WHAT IS A PICTURE WORTH?

between closely related species from camera trap images: a reply
to Gooliaff and Hodges. Ecology and Evolution 9(11):6172–6175.
Ueda K. 2020. An overview of computer vision in iNaturalist.
Biodiversity Information Science and Standards, November 2014.
Van Horn G., Mac Aodha O., Song Y., Cui Y., Sun C., Shepard A.,
Hartwig A, Perona P., Belongie S. 2018. The inaturalist species classification and detection dataset. In: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, p. 8769–8778.
Wickham H., Averick M., Bryan J., Chang W., McGowan L.D.,
François R., Grolemund G., Hayes A., Henry L., Hester J. 2019.
Welcome to the Tidyverse. Journal of Open Source Software
4:1686.

Submitted 1 July 2021. Accepted 7 February 2022
Associate Editor was Ricardo Moratelli.

Downloaded from https://academic.oup.com/jmammal/advance-article/doi/10.1093/jmammal/gyac021/6564439 by guest on 07 April 2022

Paradis E., Schliep K. 2019. ape 5.0: an environment for modern
phylogenetics and evolutionary analyses in R. Bioinformatics
35:526–528.
Potter L.C., Brady C.J., Murphy B.P. 2018. Accuracy of identifications of mammal species from camera trap images: a northern
Australian case study. Austral Ecology 44:473–483.
R Core Team. 2020. R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
Soria C.D., Pacifici M., Di Marco M., Stephen S.M., Rondinini C.
2021. COMBINE: a coalesced mammal database of intrinsic andextrinsic traits. Ecology 102:e03344.
Thompson C.W., et al. 2021. Preserve a voucher specimen! The critical need for integrating natural history collections in infectious
disease studies. mBio 12:e02698-20.
Thornton D.H., King T.W., Scully A., Murray D. 2019. Reassessing
the success of experts and nonexperts at correctly differentiating

9