DOI: 10.5061/dryad.3ffbg79xb
title: Data from: Humans outperform Merlin SoundID in field-based point-count surveys

description: Point counts are integral to bird population monitoring, and ​multiple
 methods have been developed to improve their reliability. However, these
 improved protocols are time-consuming, expensive, and often require hiring
 surveyors with inconsistent abilities. Merlin Sound ID, an automated
 bird-sound classifier that can run in the field on mobile devices, could
 be a part of the solution to the problems associated with point counts by
 functioning as a no-cost observer with consistent performance. Here, we
 compare the accuracy of Merlin and human observers in 144 paired
 field-based point-counts. Our goals were to evaluate how the species
 identified, number of individual detections, and precision for humans and
 Merlin compared. We also examined the consistency of Merlin across
 different devices. Human observers reported 382 individual detections over
 the course of the study, 72% more than the 222 detections reported by
 Merlin. Precision was similar with 92% and 86% for humans and Merlin,
 respectively. Merlin detected four species missed by humans, but also
 detected 12 false positive species. Merlin disagreed with itself on 57% of
 point counts when running on two different devices at the same time. We
 propose Merlin not as a replacement for human point counters, but as an
 aid to increase detection probability of point-count surveys as secondary
 observers, with humans acting as reviewers and arbitrating disagreements
 between devices when needed.  This study required no ethical
 permits.  
description: This study took place at the Schoodic Forest in Winter Harbor,
 Maine, USA (Figure 1). This 687.97 hectare forest parcel is owned and
 managed by Maine Coast Heritage Trust and co-managed with Schoodic
 Institute. The Schoodic Forest property is a coastal spruce-fir forest
 with abundant <em>Acer rubrum</em> (Red Maple) that
 experienced partial harvest last in the late 1990s. The land is used by
 the local community for hunting, hiking, and other recreation. The survey
 was conducted along Curtis Road, a small gravel road that is now
 predominantly maintained as an ATV trail and access road and is the only
 road through the property. The survey route consisted
 of 10 points along the unpaved Curtis Road, starting ~500 m off from the
 paved road (Figure 2). Due to turns in the road, not all points were
 separated by 500 m, but no two points were closer than 300 m. Surveys were
 conducted beginning at point one and continuing in increasing order along
 Curtis Road to point 10 over the course of one day, except one instance,
 where we began the survey at point 10 and continued in decreasing order to
 point one. We conducted these surveys from July to October in 2023, and
 visited all 10 points during each weekly survey, unless conditions
 deteriorated, to where detection probability would be impaired as
 indicated in the National Parks Service Breeding Landbird Monitoring
 Volunteer Resource Manual (Faccio et al. 2024). This happened on one
 survey day, when we only conducted counts at points 1-4 before ending the
 survey. Whenever possible, surveys were scheduled in accordance with ideal
 weather conditions. Each point count consisted of a one-minute settlement
 period, during which no data were collected, when observers stood silently
 to allow their presence to minimize disturbance on bird behavior. At the
 end of this one minute, we began a three-minute point count, where a human
 observer auditorily and visually identified birds alongside another
 observer operating <em>Merlin</em> on a handheld device (see
 <em>Merlin Sound ID Software and Recorder Hardware</em> for
 device make and models). Observers stood within 1-2 m of each other. The
 <em>Merlin</em> operator aimed the microphone of their
 handheld device away from themselves and remained still for the three
 minutes to minimize sound interference. The human observer did not consult
 with the <em>Merlin</em> operator during the count regarding
 birds identified by the software. Occasionally there were two
 <em>Merlin</em> operators on separate devices during the same
 count. There were two human observers over the course of the study, both
 deemed intermediately skilled as they were considered level two birders
 according to the standards outlined in the Birder Certification Online
 program (Howe & Giese 2021). All point counts were conducted
 between 1100 and 1800 UTC. Six device types were used
 over the course of the study: Apple iPhone SE, Samsung Galaxy Tab A, Apple
 iPhone 12 Pro, Apple iPhone 14, Apple iPhone 13 Mini, and Apple iPhone 13.
 Three versions of the <em>Merlin</em> software (ver. 2.1.8,
 3.0, and 3.0.1) were used during the four-month study period on Apple
 iPhones, and one version (ver. 2.1.4) on the Samsung Galaxy. To our
 knowledge, no changes to the classifier used in the Sound ID feature were
 made during the duration of the study. Over the course of the study (144
 point counts), we generated 214 three-minute audio files, accounting for
 point counts where we had two devices running <em>Merlin</em>
 at the same time. Points 1-4 had 22 audio files each, while points 5-10
 had 21 each, resulting in 642 minutes of audio in total.
description: # Data from: Humans outperform Merlin SoundID in field-based point-count
 surveys Dataset DOI: [10.5061/dryad.3ffbg79xb](10.5061/dryad.3ffbg79xb) ##
 Description of the data and file structure ### Files and variables ####
 File: merlin_dataset_doublevisits_dryad.xlsx **Description:** This file
 contains the data from point counts where two observers ran Merlin at the
 same time on different devices. ##### Variables * date: The date the
 survey took place. Ten point counts happened on each survey day except one
 which was cut short due to inclement weather.  * point: Point count
 location along route on Curtis Road, starting with point one closest to
 road entrance and point ten deepest into the forest parcel. *
 what.species.1: What species did the Merlin user assign "user 1"
 for this survey day capture at this point/day combination. These species
 list include uncorrected Merlin output, so species labeled incorrect by
 the in-lab human reviewer are included. * what.species.2: What species did
 the Merlin user assigned "user 2" for this survey day capture at
 this point/day combination. These species list include uncorrected Merlin
 output, so species labeled incorrect by the in-lab human reviewer are
 included. * same = 1, diff = 0: If the species list was the same, a value
 of 1 is inout into the cell. If the species list differs in any way, a
 zero is input into the cell.  * species.misaligned: The number of species
 by which the uncorrected lists differed. * what.species.corrected.1: What
 species did the Merlin user assigned "user 1" for this survey
 day capture at this point/day combination. These species list do not
 include uncorrected Merlin output, so species labeled incorrect by the
 in-lab human reviewer are removed. * what.species.corrected.2: What
 species did the Merlin user assigned "user 2" for this survey
 day capture at this point/day combination. These species list do not
 include uncorrected Merlin output, so species labeled incorrect by the
 in-lab human reviewer are removed. * same = 1, diff = 0 corrected: If the
 species list was the same, a value of 1 is inout into the cell. If the
 species list differs in any way, a zero is input into the cell.  *
 species.misaligned.corrected: The number of species by which the corrected
 lists differed. * user.1.device: The device type used by Merlin user 1. *
 user.2.device: The device type used by Merlin user 2. ####
 File: merlin_analysis_dataset_doublesremoved.csv **Description:** This
 file contains data from each paired human-Merlin point count and the
 review of the in-lab expert. If there were two users running Merlin at the
 same time, the results from one device was randomly chosen to use
 throughout the survey for comparison to the human observer and the other
 device was removed from this data sheet. ##### Variables * date: The date
 they survey took place. Ten point counts happened on each survey day
 except one which was cut short due to inclement weather. *
 observer.initials: This is the observer ID of the in-field human
 point-counter, the Merlin operator, or the in-lab audio reviewer.  *
 obs.method: Merlin, hearing (in-field human), or playback (in-lab audio
 reviewer).  * device: If Merlin was obs.method, what type of device was
 the app run on. * point.number: Point count location along route on Curtis
 Road, starting with point one closest to road entrance and point ten
 deepest into the forest parcel. * time: Time in EST the three-minute point
 count began. * sky: Sky, or cloud cover, was recorded on a scale of 0-6 (0
 = clear skies, 1 = partly cloudy, 2 = overcast, 3 = fog that deters
 detection, 4 = light rain, 5 = rain, 6 = snow) with readings of ≥ 3
 necessitating a pause or postponement of the survey. * wind: Wind was
 recorded on the Beaufort Scale, which ranks wind speed on a scale of 0-5
 (0 = &lt; 1 mph, 1 = 1-3 mph, 2 = 4-7 mph, 3 = 8-12 mph, 4 = 13-18 mph, 5
 = 19-24 mph), with readings of ≥ 3 necessitating a pause or postponement
 of the survey. * noise: Background noise was measured by ear on a scale of
 0-4 (0 = no background noise, 1 = faint background noise, 2 = moderate
 background noise, 3 = loud background noise, 4 = intense background noise)
 with readings of ≥ 3 necessitating a pause or postponement of the survey.
 * temp: Temperature measured in Fahrenheit as recorded from the phone of
 the in-field human observer. Temperature for points 1-5 was taken at at
 point one, and temperature for points 6-10 was taken at point 6.  *
 species.code: The four-letter banding code derived from its common name.
 If species is none, UNK, or is a family name and not a species, that entry
 was discarded for analysis. None means that no species were observed for
 that point day combination, and a family name/UNK means the human observer
 had a guess on the identity of a bird they were hearing but could not ID
 to species. If species is none/UNK/a family name and not a species, that
 entry was discarded for analysis. Only entries with a species
 identification were used for the comparison of humans and Merlin. **Table
 S1.** The banding code, common name, and scientific name of every bird
 detected over the course of the study. | **Banding code** | 
 
 | 
 
 | | :--------------- | :--------------------------- |
 :------------------------ | | 
 
 | **Common name** | **Scientific name** | | AGOL | American Goldfinch |
 *Spinus tristis* | | AMCR | American Crow | *Corvus brachyrhynchos* | |
 AMRO | American Robin | *Turdus migratorius* | | BAOW | Barred Owl |
 *Strix varia* | | BAWW | Black-and-white Warbler | *Mniotilta varia* | |
 BCCH | Black-capped Chickadee | *Poecile atricapillus* | | BHVI |
 Blue-headed Vireo | *Vireo solitarius* | | BLJA | Blue Jay | *Cyanocitta
 cristata* | | BRCR | Brown Creeper | *Certhia americana* | | BTNW |
 Black-throated Green Warbler | *Setophaga virens* | | BWHA | Broad-winged
 Hawk | *Buteo platypterus* | | CANG | Canada Goose | *Branta canadensis* |
 | CEDW | Cedar Waxwing | *Bombycilla cedrorum* | | CHSP | Chipping Sparrow
 | *Spizella passerina* | | COHA | Cooper's Hawk | *Astur cooperii* |
 | CORA | Common Raven | *Corvus corax* | | COYE | Common Yellowthroat |
 *Geothlypis trichas* | | DEJU | Dark-eyed Junco | *Junco hyemalis* | |
 DOWO | Downy Woodpecker | *Dryobates pubescens* | | EAKI | Eastern
 Kingbird | *Tyrannus tyrannus* | | EAPH | Eastern Phoebe | *Sayornis
 phoebe* | | GCKI | Golden-crowned Kinglet | *Regulus satrapa* | | GRCA |
 Gray Catbird | *Dumetella carolinensis* | | HAWO | Hairy Woodpecker |
 *Dryobates villosus* | | HERG | American Herring Gull | *Larus
 smithsonianus* | | HETH | Hermit Thrush | *Catharus guttatus* | | MAWA |
 Magnolia Warbler | *Setophaga magnolia* | | MODO | Mourning Dove |
 *Zenaida macroura* | | NAWA | Nashville Warbler | *Leiothlypis
 ruficapilla* | | NOFL | Northern Flicker | *Colaptes auratus* | | NOPA |
 Northern Parula | *Setophaga americana* | | OSPR | Osprey | *Pandion
 haliaetus* | | OVEN | Ovenbird | *Seiurus aurocapilla* | | PAWA | Palm
 Warbler | *Setophaga palmarum* | | PISI | Pine Siskin | *Spinus pinus* | |
 PIWA | Pine Warbler | *Setophaga pinus* | | PIWO | Pileated Woodpecker |
 *Dryocopus pileatus* | | RBNU | Red-breasted Nuthatch | *Sitta canadensis*
 | | REVI | Red-eyed Vireo | *Vireo olivaceus* | | SOSP | Song Sparrow |
 *Melospiza melodia* | | SWTH | Swainson's Thrush | *Catharus
 ustulatus* | | WTSP | White-throated Sparrow | *Zonotrichia albicollis* |
 | YRWA | Yellow-rumped Warbler | *Setophaga coronata* |   * count.number:
 If observation type was hearing (in-field human) how many individuals of
 that species did they detect. Not recorded for Merlin or playback as a
 count of individuals cannot be determined from audio, resulting in NA
 values.  * type.a.v: If observation type was hearing (in-field human) did
 they detect the species visually or aurally. If detected both ways,
 aurally was recorded so it was noted that Merlin, in theory, could have
 also detected the bird.  * distance.50.u.o: If observation type was
 hearing (in-field human) did they infer the species was under or over 50
 meters from their position when it vocalized. If value is N/A or blank,
 distance was not recorded, likely by accident. Only the human, and not
 Merlin or playback, could estimate distance.  * notes: Any notes added by
 the in-field observers or data reviewers. * correct.id: Was this
 identification correct as determined by the in-lab human reviewer. More
 information on how observations were deemed correct or incorrect for
 humans and Merlin is available in the methods section of the manuscript.
 NA for reviewer identifications as the reviewer was determining if Merlin
 and humans were correct.