BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics
Authors: Lukas Rauch
Raphael Schwinger
Moritz Wirth
René Heinrich
Denis Huseljic
Marek Herde
Jonas Lange
Stefan Kahl
Bernhard Sick
Sven Tomforde
Christoph Scholz
Venue: ICLR 2025 Spotlight (2025)
Keywords: audio classification
multi-label
dataset collection
bioacoustics
Primary area: datasets and benchmarks
TL;DR: We introduce BirdSet, a multipurpose and large-scale dataset collection for audio classification.

Abstract: Deep learning (DL) has greatly advanced audio classification, yet the field is limited by the scarcity of large-scale benchmark datasets that have propelled progress in other domains. While AudioSet is a pivotal step to bridge this gap as a universal-domain dataset, its restricted accessibility and limited range of evaluation use cases challenge its role as the sole resource. Therefore, we introduce BirdSet, a large-scale benchmark data set for audio classification focusing on avian bioacoustics. BirdSet surpasses AudioSet with over 6,800 recording hours ($\uparrow17\%$) from nearly 10,000 classes ($\uparrow18\times$) for training and more than 400 hours ($\uparrow7\times$) across eight strongly labeled evaluation datasets. It serves as a versatile resource for use cases such as multi-label classification, covariate shift or self-supervised learning. We benchmark six well-known DL models in multi-label classification across three distinct training scenarios and outline further evaluation use cases in audio classification. We host our dataset on Hugging Face for easy accessibility and offer an extensive codebase to reproduce our results.

Published as a conference paper at ICLR 2025
BIRDSET: A LARGE-SCALE DATASET FOR AUDIO CLASSIFICATION IN AVIAN BIOACOUSTICS
Lukas Rauch Raphael Schwinger Moritz Wirth René Heinrich Denis Huseljic
Marek Herde Jonas Lange²
Stefan Kahl'
Bernhard Sick
Sven Tomforde²
Christoph Scholz
University of Kassel "Kiel University
"Fraunhofer IEF 'TU Chemnitz
TTkas rauchhuni -kans-
ABSTRACT
Deep learning (DL) has greatly advanced audio classification, yet the field is limited by the scarcity of large-scale benchmark datasets that have propelled progress in other domains. While AudioSet is a pivotal step to bridge this gap as a universal- domain dataset, its restricted accessibility and limited range evaluation use cases challenge its role as the sole resource. Therefore, we introduce BirdSet, large- scale benchmark dataset for audio classification focusing on avian bioacoustics. Bi rdSet surpasses AudioSet with over 6,800 recording hours (f 17%) from nearly 10,000 classes 181) for training and more than 400 hours (+7x) across eight strongly labeled evaluation datasets. It serves as versatile resource for use cases such as multi-label classification, covariate shift, or self-supervised learning. We benchmark six well-known DL models in multi-labe classification across three distinct training scenarios and outline further evaluation use cases in audio classification. We host dataset Hugging Face for easy accessibility and offer an extensive codebase to reproduce our results.
INTRODUCTION
Audio classification is critical in many domains such as environmental (Piczak, 2015) and wildlife monitoring (Kahl al., 2021b). Audio data presents unique challenges for deep learning (DL), includ- ing low signal-to-noise ratios, temporal dependencies of events, and recording variability (Purwins al., 2019). These challenges demand robust models capable of handling diverse evaluation cases in multi-label classification (Fonseca et al., 2021), covariate shifts (changes in environments or recording devices) (Abeßerr, 2020), class imbalance (few-shot learning) (Heggan et al., 2022), or label noise (annotation errors or weak labels) (Iqbal et al., 2022). However, large-scale datasets remain limited in audio classification compared speech recognition or computer vision. While AudioSet (Gemmeke et al., 2017) offers substantial training data with over 5,800 recording hours, its restricted accessibility that requires manual data retrieval, lack of diverse evaluation scenarios and test datasets (Wang et al., 2021), and concerns regarding transferability to real-world environmental domains (Ghani et al., 2023) challenge its role as the only training resource. Thus, advancing audio classification requires not only universal datasets but also domain-specific data that offer range of evaluation use cases for benchmarking the robustness and generalization performance of DL models.
BirdSet
SSW O
Area durationi It] L: Leain ecenol
911 houre
SNE
UHH
POW
HSN
NBP
NES
AudioSet
BirdSet 6777 houre
5,790 honms
SYC-UST-V2
Egic Sounds
ESC 50
FSDDIL i
10T
VGG
iNatSouncle
Figure I: BirdSet's 's volume compared to broader audio classification datasets. The area of the circles represents the total recording duration in [h]. More details can be found in Appendix B.
Avian bioacoustics is well-suited as such camain-specific application audio classification due to (1) cost-effective data collection through passive acoustic monitoring (PAM) (Ross et al., 2023), (2) community-driinn platforms like CanooConto (XC) (Vellinga & Planqué, 2015) with an abundance of annotated recordings, and (3) the high complexity of bird vocalizations. This complexity reflects

---

Published as a conference paper at ICLR 2025
diverse challenges relevant to broader audio classification, including diverse acoustic environments, extensive class diversity, variations in recording devices, and notable sound overlaps. The primary task in avian bioacoustics is the AudioSet. This serves as a crucial application since fluctuations in bird populations indicate broader shifts in biodiversity (Sekercioglu al., 2016). Despite growing interest in computational avian bioacoustics (Stowell, 2021), there is no large-scale, easily accessible, and curated dataset available, hindering comparability across studies and creating barriers to accessibility from the broader audio domain (Rauch et al., 2023b). To advance audio classification and address the lack of standardized benchmark in avian bioacoustics, we introduce the Bi rdSet benchmark dataset large-scale scale collection of bird vocalizations and versatile resource for broader audio classification. Bí rdset surpasses AudioSet dataset volume and offers unique test dataset collection featuring recordings from diverse regions (cf. Figure 1). We outline Bi rdSet and its contributions in the following:
classification in PAM, comparable to classification in
BirdSet: Outline and Contributions
(1) We introduce the Bi d dataset collection" on Hugging Face (HF) (Lhoest et al., 2021), featuring about 520,000 unique global bird sound recordings from nearly 10,000 species with over 6,800 hours for training and over 400 hours of PAM recordings with 170,000 annotated vocalizations across eight unique project sites for evaluation.
(2) BirdSet serves as an extensive multi-purpass dataset in audio classification with evalua- tion use cases such as self supervieed learning, event detection, multi-labed classification under covariate and domain shifts with noisy labels or few-shot, and active learning.
(3) By providing a large-scale train dataset and diverse test dataset collection, BirdSet represents a comprehensive and additional resource in audio classification (cf. Figure 1).
(4) A comprehensive literature analysis identifies and discusses challenges in computational avian bioacoustics embedded within the broader audio classification domain. We structure them to offer research guidelines and evaluation cases resulting from Bi rdSet.
(5) We benchmark multi-label classification under covariate shift with noisy labels and task shift using well-known DL models. Our extensive empirical study evaluates distinct supervised training scenarios, including large-scale training and fine-tuning BirdSet.
(6) An extensive codebase" with standardized training and evaluation protocols enables repro- ducing our results, supporting B rdSet's 's utility, and easing accessibility for newcomers.
"attpa:://hugggingffcce.. co /at researeeeeechh res.rrr--eearrrruuuuuuuu grcup/BirdSes
2 CURRENT CHALLENGES AND RELATED WORK
Avian bioacoustics exemplifies challenges in audio classification, including managing diverse and noisy acoustic environments or dealing with class imbalance. Thus, it is our primary case study for illustrating real-world evaluation this section, we outline these challenges, review how related work addresses them, and outline approach to tackling them in our benchmark, highlighting how BirdSet differs from related datasets. Detailed explanations our approaches are provided in Section and Section 4.
cases in the field represented in BirdSet's datasets. In
2.1 CHALLENGE 1: DATASETS
Challenge description. Audio data exhibits complex characteristics, including variable lengths, overlapping signals, and variability in recording sources (c.g.. recording type or device). As a domain- specific audio classification task, avian bioacoustics exemplifies and adds to these complexities, making it suitable for exploring real-world audio challenges. Avian bioacoustics differentiates between focal and soundscape recordings (Kahl et al., 2021b). Focal recordings involve recordist aiming directional microphone toward the source of bird vocalizations (i.e., sound events), capturing sequences of calls from primary and occasionally secondary species, which typically results in a multi- class problem. Their abundance and variability citizen-science platforms such as XC make them particularly suitable as training data. However, they do not represent entire acoustic environments (i.e., soundscapes) and usually weakly labeled without specifie vocalization times, making them
2

---

Published as a conference paper at ICLR 2025
unsuitable for evaluation in PAM (Van Merriënboer et al., 2024). Soundscape recordings in PAM are passively collected by omnidirectional microphones within a static area over extended periods (Kahl et al., 2021b), capturing bird vocalizations alongside environmental noise with minimal habitat disruption. Often strongly labeled from BirdCLEF competitions (Kahl et al., 2022b), soundscapes offer comprehensive audio representation of real-world domain, making them ideal for testing. Due to the simultancous occurrence of multiple sounds, soundscapes reflect multi-label problem. However, their static nature, limited geographical coverage, and high labeling cost render them unsuitable for large-scale model training ( (Van Merriënboer et al., 2024).
Table I: Datasets employed in current bird sound classification publications. Model training is architecture, and input type.
analyzed by the task (multi-label or multi-class
Sources
Focalh
Soundscapes
Mudel
Input
INA
HSN
UBB
PER
SSW
FCM
NEP
VOX
CNN
Tmssf
Spec
Wave
Bellafkir al.
Imin
cval
Bellafkir
Bain
X
X
Brave Sanchex
Fai
X
oval
x
x
X
X
8
X
Clark al.
oval
X
X
Denton et
Imin
eval
Eichinski al.
eval
al.
Bai
Gupia et al.
Inin
eval
Enin
Hochst et al.
Fam
oval
Ho al.
etal
Jeuntet & Dufcory
oval
Bain
Kahl al.
eval
Ban
Liu et al.
Bai
eval
/
Swaminathan
eval
x
Ting et al.
oval
x
Wang ct.
eval
1
Xian et
Baaid
oval
Xie & Zhu
eval
x
Zhang
Iman
B: -dSet
main
oval
Related work. Studies in audio classification commonly employ AudioSet for large-scale training and smaller datasets such as ESC-50 (Piczak, 2015) for fine-tuning and testing (Gong et al., 2021; Huang al., 2022; Chen et al., 2023). However, these datasets cannot represent all real-world complexities (e.g., class imbalance) due to limited size and quantity compared to vision datasets (e.g., ImageNet (Deng et al., 2009)). Despite the wealth of data in avian bioacoustics (Vellinga & Planqué, 2015), the absence standardized datasets similar to AudioSet results in inconsistent data selections and processing, outlined in Table 1. This lack of standardization affects comparability and complicates accessibility to this field. Studies usually create custom train and test datasets of focal recordings from XC, failing to generalize to realistic PAM scenarios with soundscapes. Additionally, using weak labels for testing compromises the practical result validity (Fonseca et al., 2021), and the dynamic nature of XC further complicates reproducibility and comparability across studies. Other research directions operate in PAM scenarios by evaluating an arbitrary set of soundscapes (Denton al., 2022: Bellafkir al., 2023) with custom training datasets from XC that are not publicly available. In contrast, Bi rdSet eliminates dataset processing and collection by providing the first curated collection with uniform metadata (e.g., label formatting) via HF. includes an extensive volume of focals for training and collection of soundscapes for testing, enabling model performance evaluation across various regions in PAM and facilitating comparability across studies.
Related benchmarks. AudioSet is universal-domain dataset for large-scale audio classification, featuring approximately 2.1 million samples across 527 classes, totaling 5,800 hours. While it is the only environmental audio classification dataset comparable in scope to BirdSet, it only includes one small test subset and requires manual retrieval of audio files. In contrast, BirdSet includes nearly 10,000 domain-specific classes, over 6,800 hours of training data, and eight distinct and
3

---

Published as a conference paper at ICLR 2025
strongly labeled test datasets. Its easy usability through HF contrasts with the complex setup required for accessing AudioSet, further highlighting BirdSet's 's benefits. ESC-50, a smaller dataset with 2,000 recordings, primarily serves for small-ssale evaluation fine-tuning. FSDK50 (Fonseca et al., 2021) offers 50,000 curated clips across 200 general-domain classes from the AudioSet ontology to improve accessibility, though it does not match Bi rdSet's 's volume. In avian bioacoustics, the BirdCLEF challenge (Kahl et al., 2023) regularly introduces novel, strongly labeled soundscape datasets for testing. However, its competition format emphasizes score optimization over broader research, and the varying datasets and metrics across editions limit its suitability for benchmarking. In contrast, Bi tt offers curated collection with standardized training and evaluation protocols, enabling comparisons across studies and complementing BirdCLEF as a foundation for future challenges. The BEANS (Hagiwara et al., 2023) and BIRB (Hamer al., 2023) benchmarks mark a shift towards structured datasets in bioacoustics but face issues in dataset diversity and accessibility. While BEANS covers various animal sounds, it lacks volume and does not provide large-scale training dataset suitable for representation learning like BirdSet, limiting its evaluation cases. Its multi-class nature also falls short in assessing generalization for avian bioacoustics and broader envirommental audio classification. In contrast, Bi rdSet offers a larger, benchmark, with circa 10,000 bird species globally for training and eight distinct evaluation locations. Its segment-based multi-label classification of soundscapes better mirrors real-world PAM and audio classification conditions. While BIRB and BirdSet focus on bird data, their task and scope differ significantly. BirdSet offers readily accessible multi-label classification task, bridging broader audio classification with various evaluation use cases. In contrast, BIRB provides a retrieval task centered on avian vocalizations, where pre-trained bioacoustic models are evaluated by ranking species-specine specific vocalizations based on a few examples. Additionally, BIRB requires manual data processing and user-defined query inputs to create datasets, hindering cross-study comparisons and limiting its applicability beyond bioacoustics. Bi rdSSe't 's simplicity and versatility make it more user- user-friendly benchmark for avian bioacoustics and general audio classification.
diverse, and versatile
2.2 CHALLENGE 2: MODEL TRAINING
Challenge description. Audio classification involves detecting and handling overlapping events in multi-label context, typically associated with weak file- labels, to reduce annotation effort (Fon- seca et al., 2021). This complexity often reduced to a more straightforward multi-class task in datasets such ESC-50 (Piczak, 2015), potentially limiting a model's ability to handle overlapping sounds and event ambiguity. In computational avian bioacoustics, tasks range from detection to identification of bird species, primarily focusing species classification through vocalizations (Van Merričnboer et al., 2024), nearly identical to general audio classification. Research typically em- ploys multi-class classification for focal recordings and multi-label classification for soundscapes. Recent studies follow a typical training process (Stowell, 2021), as outlined in Figure 2. It starts by detecting bird vocalizations in weakly labeled focals, enabling recording yield multiple samples. Comparable to audio classification, vocalizations are then converted into spectrogram images which visualize frequency intensity, reducing noise and preparing the data for classification with vision- based DL architectures (Kahl et al., 2021b). This conversion requires manual parameter tuning (c.g., frequency resolution), complicating model comparability (Gazneli et al., 2022; Rauch et al., 2023b). Augmentation techniques applied to waveforms or spectrograms enhance data variety and align the training data distribution of focal recordings more closely to the test data distribution (i.e., soundscape recordings in PAM), aiding in model generalization.
wasveform recording
event detertion
CNN training
spectrugram conversion
anguentaatio
Figure 2: Typical model training for classification in avian bioacoustics.
Related work. Most audio classification and avian bioacoustics research leverages spectrogram images inputs, shown in Table 1. While general audio classification increasingly adopts transformer architectures (Chen et al., 2023), CNNs still dominate avian bioacoustics (Kahl et al., 2021b). Bellafkir al. (2024) use a variant of the vision-based audio spectrogram transformer
4

---

Published as a conference paper at ICLR 2025
(AST) (Gong et al., 2021). Bravo Sanchez et al. (2021) and Swaminathan al. (2024) demonstrate the potential of learning directly from raw waveforms by employing SincNet (Ravanelli Bengio, 2019) and Wav2Vee 2.0 (W2V2) (Baevski et al., 2020). While audio classification has benefited from self-supervised learning (Chen et al., 2023; 2024)). avian bioacoustics has only begun initial explorations (Bellafkir al., 2024; Moummad et al., 2024). Current state-of-the-art (SOTA) models BirdNET (Kahl et al., 2021b) and Google's ' Perch (Hamer et al., 2023) still rely on conventional supervised learning with EfficientNet (Tan & Le, 2020). However, these large audio classification models, trained on approximately 10,000 bird species from XC, produce valuable embeddings for downstream tasks in avian bioacoustics. In contrast, models pre-trained on general domain audio exhibit poor transferability to the domain (Ghani et al., 2023; Nolan et al., 2023). Additionally, multi- label classification remains underrepresented in current work despite its importance in practical PAM scenarios. Our benchmark addresses this gap by introducing three training protocols for multi-label audio classification featuring large-scale representation learning and fine-tuning. We are the first to present benchmark results for the current SOTA in avian bioacousties, extending beyond typical convolutional neural network (CNN) architectures to include transformers (e.g., AST) or SOTA CNNs (ConvNext (Liu al., 2022c)). Our supervised training protocols designed to support future evaluation use cases in self-supervised representation learning, including large-scale pre-training and fine-tuning on BirdSet. All models and benchmarks open-sourced via HF.
2.3 CHALLENGE 3: MODEL ROBUSTNESS
Challenge description. A robust audio classification model must generalize across diverse acoustic environments, varying notably by their location. It requires handling recording variations, including noise, sound source distance, or recording type. Avian bioacoustics under PAM conditions exemplifies this with complex and diverse bird vocalizations (Byers & Kroodsma, 2016). A bird species emits multiple distinct vocalizations, including songs, call types, regional dialects. blending into the complex dawn chorus (Berg et al., 2006). We identified four key challenges in Bi rdSet that hinder achieving robustness in real-world audio classification tasks based avian bioacoustics, illustrated in Figure 3. We analyze how related work and our multi-label benchmark address them in the following. We also outline how Bi rdSet offers opportunities to evaluate them for audio classification.
Focal
Soundscape
covnriate shift
angreent
label uncertainty
domain shift- Pie(x")
lack shift
wlaptability
(ir(x)
Per(x'')
Figure Key obstacles for model robustness in bird sound classification.
Related work covariate and domain shift. Covariate and domain shifts occur when training and test input distributions differ (Sugiyama & Kawanabe, 2012), often due to varying recording conditions and devices. Evaluating these shifts in audio classification is challenging because most datasets are highly processed and lack variability in the test data (Fonseca et al., 2021). This gap is particularly pronounced in avian bioacousties due to the use of focals for training and soundscapes for testing. Moreover, the sound profiles of recordings vary due to differences in recording devices, environmental conditions, proximity the sound source. As shown in Table current research typically creates custom train and test datasets from XC, isolating vocalization events in advance. However, this approach bypasses the shift encountered in real-world conditions, where trcal-trained models must generalize to soundscape recordings (Kahl et al., 2021b). In a PAM setting, research employs augmentations by adding background sounds from external soundscapes (Hamer et al., 2023; Denton et al., 2022) (e.g., from VOX (Stowell et al., 2019)) or noise to align the train and test distributions more closely. Supervised pre-training on various species outside the test label distribution has enhanced model robustness under domain and covariate shift (Clark et al., 2023). Studies also incorporate soundscapes from a specific use case into training to address covariate shift (Eichinski al., 2022; Höchst et al., 2022). challenging adaption to PAM locations. BirdSet provides different training scenarios for focal- -aaed training datasets that incorporate a supervised pre-training procedure and set of augmentations to align the train and test distributions more closely.
5

---

Published as a conference paper at ICLR 2025
Related work label uncertainty. Label uncertainty or noisy labels is a common challenge in audio classification, occurring from weakly labeled recordings that provide only file-level labels without precise event timestamps or from labeling errors in large datasets (Fonseca et al., 2021). This is particularly problematic for evaluation, as weak test labels can significantly impact benchmarking accuracy (e.g., in AudioSet (Fonseca et al., 2021)). In avian bioacousties, focal recordings from XC are also usually weakly labeled, lacking exact timestamps for the bird vocalization label. However, the citizen-science aspect of XC helps reduce labeling errors. In contrast, various publicly available soundscape datasets include precise annotations by omithhlogiists (Hopping et al., 2022). Ambiguity also arises due to the lack of specific labels for different vocalization types, as different vocalizations of species are assigned to the same label. Therefore, reliable audio model must handle noisy labels in PAM. Current work typically extracts potential events in focal recordings through pre-processing. The most straightforward approach for event detection assumes bird vocalizations occur at the start of recordings (Gupta et al., 2021), ignoring temporal variations. Alternatively, a peak detection algorithm identifies vocalization events in recording (Denton et al., 2022; Hamer et al., 2023). However, these labels remain ambiguous, as an event could originate from another sound source. Other methods include random selection from recording (Bellafkir al., 2024; Eichinski et al., 2022) or dedicated detection models (Clark et al., 2023; Bellafkir et al., 2023). Additionally, curated clips of recordings are employed where label noise is removed beforehand (Tang et al., 2023; Hu et al., 2023; Jeantet & Dufourq, 2023), comparable to audio classification datasets such as AudioSet. Advanced techniques include combining clustering with peak detection (Michaud al., 2023), tracking feature changes through self-supervised learning (Bermant et al.. 2022), or isolating vocalizations via unsupervised sound separation (Denton et al., 2022). Bi rdset provides a comprehensive training dataset from XC with minimal label errors, featuring detected vocalization events and clusters to address label uncertainty but keeping event usage flexible for further research. Moreover, the test dataset collection exclusively contains strong labels, ensuring high-quality evaluation.
Related work task shift. In audio classification, pre-training on multi-label datasets like AudioSet and fine-tuning multi-class datasets like ESC-50 introduces task shift from pre-training to testing (Huang et al., 2022). A similar shift occurs in avian bioacoustics. where focal-trained models evaluated soundscapes in PAM, moving the task from multi-class multi-label classification. Most studies avoid the task shift by focusing on multi-class classification with manually filtered vocalizations in training and testing. Swaminathan et al. (2024) diverge from this trend by combining multiple XC focal recordings to mimic a multi-label scenario in training and testing. Denton et al. (2022), Bellafkir al. (2023), and Kahl al. (2021b) transition towards more realistic segment- based evaluation of soundscape recordings. They address the task shift by augmenting the focal training data with label-mixup (Zhang et al., 2018), combining multiple recordings and labels into one instance, simulating a multi-label task in training. BirdStt's 's training protocol uses focal recordings, while the evaluation protocol employs soundscapes, introducing an inherent task shift. To better align the training and test data distributions, we incorporate label-mixup.
Related work- adaptability. In real-world audio classification, models often operate in a few- shot setting with limited training data and a novel recording domain. This challenge is especially pronounced in avian bioacoustics due to many unique species and differing label distributions between focals and soundscapes (Van Merriënboer al., 2024). While models trained on diverse audios have shown to generalize and transfer across various conditions (Ghani et al., 2023; Chen et al., 2023), they exhibit subpopulation shift towards particular niches when funet for a specialized task in audio classification (Tran et al., 2022). Domain shifts between PAM environments in avian bioacoustics, such as variations in background sounds, further underscore the need for adaptable models. Currently, large-scale SOTA models (Hamer et al., 2023; Kahl et al., 202 rely on conventional supervised learning over extensive datasets that are utilized to obtain robust representations for few-shot learning and adaptation. Ghani et al. (2023) demonstrate that Perch excels in few-shot scenarios by fine-tuning only the classification layer. Methods to manage subpopulation shifts include limiting logits to relevant species in test datasets (Hamer et al., 2023) or training on tailored subsets to adapt to new domains (Denton et al., 2022). Bellafkir et al. (2024) and Rauch et al. (2024a) apply active learning to bird sound classification, addressing the limitations of static models in dynamic environments. BirdSet offers different training scenarios to evaluate model adaptability, including training on extensive datasets to create large-scale bird sound classification models and fine fine-tuning on tailored subsets for specific tasks.

---

Published as a conference paper at ICLR 2025
2.4 CHALLENGE 4: EVALUATION
Challenge description. Evaluating models in audio classification is challenging due to the temporal ambiguity of audio events. Avian bioacoustics underscores this complexity, including various practical PAM scenarios and downstream tasks (e.g., event detection, density estimation, classífication). In the detection and classification of bird vocalizations, we differentiate between event-based and segment- based evaluation (Van Merriënboer et al., 2024), as illustrated in Figure 4. Event-based evaluation isolates vocalizations (e.g.. peak event detection) in the test dataset. Isolating and applying multi-class classification is similar to ESC-50 but cannot capture temporal dynamics in complex scenarios with overlapping sounds. Segmentbased evaluation is the preferred method for assessing performance in realistic PAM settings. In this approach, long-duration soundscapes are segmented into fixed intervals, with labels assigned to each segment (Denton et al., 2022), comparable to AudioSet's 's classification task. This evaluation is challenged by ambiguous ground truth assignments in edge cases and validat- ing results during training under covariate shift. Segments simplify the task to vocalization detection multi-label classification, utilizing threshold-dependent threshold-independeer metrics.
event
multi-class
Y1
Y
Y3
segment multi-label
(x1)
(y2.)'))
(yo)
(y1)
III
IV
Figure 4: An illustration of segment- and event-based evaluations (Van Merriënboer et al., 2024).
Related work. When evaluating on focals, model performance commonly assessed through isolated vocalization events, comparable to clips of ESC-50, and multi-class metrics such as precision or recall (Liu et al., 2022a; Jeantet & Dufourq, 2023). Soundscapes PAM are typically evaluated via segment-based multi-label classification within five-second intervals (Denton et al., 2022; Hamer et al., 2023), similar to 10-second clips in AudioSet. Previous BirdCLEF competitions used the multi-label F1-Score (Kahl et al., 202 2022b), which requires class-dependent threshold tuning, complicating comparability and hindering insights into model generalization. Following general multi-label audio classification (Gemmeke al., 2017), recent work favors threshold independent metrics. The class- based mean average precision (cmAP) calculates the average precision across thresholds per class, followed by macro averaging (Kahl et al., 2023). While provides macro view of the model's ability to rank positive over negative instances, it can be noisy for species with sparse labels (Denton al., 2022). The area under the receiver operating characteristic curve (AUROC) measures a model's ability to rank a randomly selected positive over negative instance (Van Merriënboer et al.. 2024). Unlike emAP, which heavily influenced by the number of positives and negatives, AUROC is more robust to class imbalance across datasets, providing baseline of 0.5 for random rankings (Hamer et al., 2023). Top-1 accuracy (TI-Ace) measures whether the highest predicted probability matches one of the correct classes in a segment (Denton et al., 2022), making it helpful in identifying single species. Unlike ESC-50 or AudioSet, no standardized performance overview allows direct comparison across soundscapes. Thus, current research lacks standardized evaluation protocol, complicating study comparisons. Birdk rdset introduces an evaluation protocol for multi-label audio classification, allowing researchers evaluate and compare their models effectively. Our benchmark employs a collection of threshold-indeeeedden metrics.
BIRDSET: A LARGE-SCALE AUDIO DATASET COLLECTION
Collection and curation. rdSet provides a large-scale collection of curated train and test datasets with varying complexities, targeting multi-label audio event classification of bird vocalizations. All focal training data originates from XC, the largest and most popular citizen-science platform for avian bioacoustics. We collected and curated legally compliant, high-quality recordings. The test datasets include a diverse set of PAM scenarios with soundscape recordings covering different difficulty levels, class diversity, and geographical variations. Ensuring all datasets are legally compliant and equipped with strong labels allows for high-quality evaluation and realistic simulation of train and test scenarios in PAM. Inspired by the works of (Wang et al., 2018; Rauch et al., 2023a), aggregate
7

---

Published as a conference paper at ICLR 2025
various datasets into one diverse test collection to achieve representative model generalization results. Our curation process to achieve a uniform metadata format includes unifying the label format of all train and test recordings using the eBird code taxonomy (Sullivan al., 2009). Moreover, processed soundscapes into slices for segment-based evaluation, aligned the label formats with short-range focals for multi-label classification, and provided vocalization events in focals through bambird (Michaud et al., 2023). Recordings are processed in a resolution of 32 kHz, capturing the frequency range of most bird vocalizations (Kahl al., 2021b). We integrate the complete collection into one HF dataset with accompanying code to significantly simplify usability for researchers. More details are provided in Appendix B.
508,31
Carbo Langy
Big SkarE Nevada
56
NPB SIPS
NEE Gata Rira
55.000
X PER
Baen
20/40
X POW
X BNE
Nradd
SSM
Mhoci
x UHH
bland
Figure 5: BirdSet's geographical distribution of focals for training and soundscapes for testing.
Datasets. An overview of the BirdSet dataset col- lection is provided in Table with a geographical dis- tribution shown in Figure 5. We organize the datasets into three functional groups. Train comprises focal recordings suitable for large-scale training. Xeno- Canto large (XCL) covers a comprehensive snap- shot of XC, featuring approximately 530,000 curated recordings across nearly 10,000 species. Since we equip the variable-length recordings with detected events, the training dataset size can be expanded based the events extracted from each recording. XCL is Bi rdSet's 's largest dataset that is compara- ble to the training datasets of current SOTA mod- els BirdNET and Perch, but the first to be entirely publicly available. Xeno-Canto medium (XCM) is specialized subset of XCL that includes only recordings from 409 unique species across our test datasets. Bi rdSet also provides non-bird sound- scape recordings from the VOX dataset (Lostanlen et al.. 2018b), which serve as background noise or no-call segments for augmentations. POW is a relatively small, fully annotated soundscape dataset to validate performance or tune hyperparam- eters for large-scale models trained on XCL XCM. Test Dedicated Fine-Tuming consists of fully annotated and high-quality soundscapes for evaluating multi-label classification approaches. Following related work (Denton et al., 2022), divide each recording into 5-second segments and assign the labels based on ground truth timestamps. We attribute label for vocalizations spanning multiple segments if the vocalization lasts over 0.5 seconds within a segment (Kahl al., 2023). Each segment is an independent test sample containing none (0-vector), one or multiple species. Table 2 displays the number of segments and annotations, representing the overall recording duration and frequency of bird vocalizations. If the number of annotations exceeds the number of segments, it indicates overlapping vocalizations or concentrated activity a few segments, with others remaining empty. We also offer dedicated training subsets for each test dataset that include only recordings of species from XCL present in the respective set. They serve as fine- grained training or fine-tume tt-inn datasets. We present the test datasets' class imbalance through Pielou's 's evenness index J (Piclou, 1966) by calculating the relative class entropy per dataset. A value of signifies perfect balance, while 0 indicates significant imbalance across species.
Table Overview of datasets in Bi BirdSet. denotes the evenness index.
Set
[Train]
Test
#Annot
J #C
Train
528,434
528,343
0.85
9,734
XCM 89,798
89,798
0.94 409
Aux
POW
14,911
4,560
16.052
0.66
VOX 20,331
x
X
Test & Ded. Fine- Fine-Tuming Tuming
PER 16,802
15,120
14.798
0.78
132
NES 16,117
24,480
6,952
0.76
UHH 3,626
36,637
59,583
0.64
HSN 5,460
12,000
10,296
0.54 21
NEP 24.327
563
5,493
0.92 51
SSW 28,403
205,200
50,760
0.77 81
SNE 19,390
23,756
20,147
0.70 56
8

---

Published as a conference paper at ICLR 2025
4 BENCHMARK: MULTI-LABEL CLASSIFICATION
This section presents our multi-label audio classification benchmark as the primary use case for BirdSet. In this benchmark, robust models must handle BirdSet's 's inherent challenges that include covariate shift (train and test distributions), label uncertainty (weak labels), task shift (multi- class and multi-label), class imbalance, and subpopulation shift (adapting to a species subset). Detailed experimental settings and results are provided in Appendix C.
4.1 EXPERIMENTAL SETUP
Training protocol. We explore three supervised multi-label training scenarios relevant to current audio classification research (Hamer et al., 2023; Ghani et al., 2023; Purwins et al., 2019). We one-hot encode the focal recordings during training to align their labels with the multi-label soundscapes used in testing. In large training (LTT) and medium training (MT), we train models on XCL. and XCM, respectively. This results in supervised representation learning for various bird species in diverse environments, comparable to a pre-trained model on AudioSet. In dedicated fine-tuning (DT), we train the models on the specific subsets, adapting them specialized tasks, similar to fine-tuning a pre-trained AudioSet model on ESC-50. We employ five well-known audio model architectures. This includes spectrogram- based models: EfficientNet (Tan & Le, 2020) and ConvNext (Liu et al., 2022c) (CNN), and AST (Gong et al., 2021) (ViT). Additionally, we use two transformers for raw waveforms: EAT (Gazneli et al., 2022), focused on environmental sounds, and W2V2 (Baevski et al., 2020), known for speech processing. We utilize pre-trained weights based on their availability on HF. AST and EAT are pre-trained on AudioSet, corresponding to a transfer learning and fine-tuning approach in DT. Our training protocol also includes various augmentations to better align training and test data distributions: Time-shifting and background noise adjust for vocalization timings and integrate diverse noise profiles. Multi-labe mixup (Zhang al., 2018) increases sample diversity, and adding no-call training data from the auxiliary dataset VOX simulates non-vocal segments.
Evaluation protocol. We establish an evaluation protocol assess model performance across Birdset's 's test datasets in various scenarios (e.g., class imbalance and geographical diversity), benchmarking a model's generalization capability and robustness in audio classification. We use threshold-free metrics to assess generalization without threshold-tuning
minimizing application- specific biases (Hendrycks & Gimpel, 2018). Following related work, we use cmAP (Kahl et al., 2023), AUROC (Van Merriënboer et al., 2024), and T1-Acc (Denton et al., 2022). This metric collection aligns with recent work in audio classification (Gemmeke al., 2017; Chen al., 2024) and supports evaluation across various use cases. We repeat the experiments across three (MT, LT) and five (DT) random seeds and report the mean results. Refer to Appendix C for more details and the standard deviations. Following (Hamer et al., 2023), we employ POW as soundscape validation dataset for LI and MT. We save a checkpoint after each epoch and select the model with the lowest validation loss for testing. During inference, we restrict the models' logits by excluding those unrelated to the species present in the test datasets. Since validating with POW is not possible in DT the model is limited to the dedicated classes, we allocate 20% of the training data for validation using the same checkpoint method. We also incorporate inference with Perch (Hamer et al., 2023) that has already shown promising results (Ghani et al., 2023) and exclude BirdNET (Kahl al., 2021b) due to potential test data leakage.
4.2 EMPIRICAL RESULTS
Table 3 and Figure report results for the LT scenario, detailing the generalization performance of the models in audio classification. We also present overall scores averaged across all test datasets for DT and MT. The results across all metrics and models indicate that large-scale representation learning in LT and MT generally improves performance compared to DT. We can see that fine-tuning pre-trained models from general audio domains (AST and W2V2) in dedicated environments in DT without comain-specifii knowledge is less effective than employing large-scale models trained in LT or MT. Furthermore, models trained in the LT scenario demonstrate greater versatility, encompassing knowledge about approximately 10,000 bird species. Overall scores vary notably across datasets, underscoring the unique challenges in each test dataset. For instance, PER and UHH are challenging due to complex overlaps of multiple vocalizations and location- specific background noise, poorly rep- resented in the focal training data (i.e., covariate shift). This variability emphasizes the importance of

---

Published as a conference paper at ICLR 2025
diverse test data collections to effectively evaluate generalization capabilities in various environments. The T1-Acc metric indicates substantial difficulties in accurately classifying even one bird vocalizing in segment. These results BirdSet stress the necessity for ongoing research in environmental audio classification with domain-specific approaches to address the complexities of model robustness, particularly in managing label uncertainty, covariate shift, and task shift.
Table 3: Mean results across datasets and models for large train- ing on XCL. Best and second best results are highlighted. Score reflects test average for all training scenarios, also including ded- icated fine-tuning (DT) on subsets, medium training XCM.
Rif.Net = ConyNect
Perch *** EAT
Test
Score
NBP
S6W
FOW
|PER
NES
CHE
HaN
NBP
SS9
SNE
LT
MT
DT
E-T.
Net
cmAP
0.23
0.57
0.36
0.35
AUROC
0.82
0.71
0.88
0.78
0.86
0.90
0.91
0.83
0.84
0.85
0.84
T1-Acc
0.80
0.38
0.49
0.42
0.59
0.63
0.55
0.67
0.53
0.59
0.54
Coav Next
emAP
0.36
0.19
0.34
0.26
0.47
0.62
0.35
0.30
0.36
0.36
0.37
HSN
SNE
AUROC
0.82
0.72
0.88
0.79
0.89
0.92
0.93
0.83
0.85
0.84
0.83
TI-Acc
0.75
0.36
0,45
0.44
0.52
0.64
0.53
0.65
0.51
0.57
0.52
AST
CHLAP
0.33
0.18
0.32
0.21
0.44
0.61
0.33
0.28
0.34
0.31 0.29
0.6
AUROC
0.82
0.72
0.89
0.75
0.85
0.91
0.93
0.82
0.84
0.83 0.83
T1-Acc
0.79
0.40
0.48
0.39
0.48
0.61
0.50
0.57
0.49 0.47 0.49
0.8
W2v2 EAT
AUROC
emAP
0.79
0.12
0.64
0.87
0.22
0.76
0.86
0.50
0.87
0.90
0.81
0.80
0.82 0.78
0.33
UHH
0.9
PER
T1-Acc
0.69
0.32 0.46
0.40
0.47
0.61
0.46
0.58
0.48
0.47 0.47
cMAP
0.27
0.14
0.20
0.21
0.40
0.7
0.29
0.25
0.31
0.29 0.26
AUROC
0.75
0.68
0.86
0.76
0.86
0.90
0.90
0.78
0.78
0.80 0.79
T1-Acc
0.72
0.34
0.47
0.51
0.50
0.65
0.50
0.51
0.50
0.46 0.44
Figure 6: Mean AUROC in large training LT on XCL with selected models across test datasets.
Perch
cmAP
0.30
0.18
0.39
0.27
0.45
0.63
0.28
0.29
10.36
AUROC
0.84
0.70
0.90
0.76
0.86
0.91
0.91
0.83
0.84
TI-Ace
0.48 0.57
0.69
0.69
The AUROC metric shows that our ConvNext implementation consistently outperforms other models across datasets in the LT scenario, especially in its ability to discriminate between positive bird vocal- izations and negative classes or background noise. It often exceeds the SOTA model Perch (Hamer et al., 2023), which, despite its simpler EfficientNet architecture, excels in retrieval tasks reflected by the T1-Ace metric. However, ConvNext's 's consistent performance, shown by the AUROC and emAP metrics, highlights the advantages of more complex model architectures. We also that complex models such as AST or ConvNext perform better than smaller ones such as EfficientNet in LT. This could guide future research towards prioritizing large embedding models for creating representations and fine-tuning in multi-label audio classification (Ghani et al., 2023). Waveform transformers such as EAT and W2V2 offer competitive yet slightly inferior performance compared to spectrogram- based models, possibly struggling to handle input noise.
5 CONCLUSION AND LIMITATIONS
Conclusion. We introduced BirdSet, a large-scale and multi- purpose audio classification dataset in avian bioacoustics. This versatile and challenging dataset offers a complex domain-specific case study for broader audio classification tasks. We identified and discussed key challenges in audio classification that are represented as evaluation use cases in BirdSet. As the primary evaluation use case, we benchmarked supervised multi-label classification under covariate shift, task shift, and label uncertainty using five well-known deep learning models across three unique training scenarios. These scenarios include large-scale supervised learning and fine-tuning on dedicated subsets. By providing these resources, we aim to set new standard for classification tasks in passive acoustic monitoring. For future research, we aim to expand the benchmark by leveraging self-supervised representation learning techniques from the broader audio domain.
Limitations. While our benchmark indicates generalization performance in multi-label audio clas- sification, it does not analyze Bi rdSet's underlying characteristics affecting model performance, such challenges associated with model robustness. Future research should focus on these aspects to enhance model robustness and applicability across diverse environments for audio classification. Additionally, our benchmark does not include the most recent audio transformers and more thorough investigation of the influence of pre-training However, our code is flexibly designed to support additional model implementations extend our benchmark in the future.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We collected and curated data under appropriate Creative Commons (CC) licenses to ensure privacy and uphold ethical standards. By facilitating passive recordings for testing in passive acoustic moni- toring, we aim to minimize disturbance to natural habitats, promoting advancements in environmental audio classification and biodiversity monitoring. However, ethical and responsible use of both the dataset and the models derived from BirdSet is essential, The models developed from this dataset are strictly intended to advance audio research and biodiversity conservation. Each recording from XC is licensed and credited to the recordist. Researchers must respect these licenses, appropriately credit the recordist, and ensure their privacy. promote comparability, accessibility, and reproducibility, we require all researchers using this dataset and benchmark to disclose their methodologies, report results openly, and state their research objectives. We aim to foster collaboration and advance the fields through this standardized dataset, encouraging ethical research practices and sharing results.
REPRODUCIBILITY STATEMENT
Reproducibility is primary goal of this work. To ensure this, we have made the dataset collection, BirdSet, openly and easily accessible via Hugging Face, along with all model checkpoints (Ap- pendix A). The complete code for reproducing our benchmark results is available in GitHub repository, which includes detailed instructions for setting up the environment, running the code, and following the training and evaluation protocols (Appendix A). These materials ensure that all experi- ments can be easily reproduced and extended. Further details regarding data collection, preprocessing, augmentations, evaluation, and hyperparameter configurations can be found in Appendix C
ACKNOWLEDGEMENTS
This research was conducted under the DeepBirdDetect project (FKZ 67K131040E), funded by the German Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV).
REFERENCES
Jakob Abeßer. A Review of Deep Learning Based Methods for Acoustic Scene Classification. Applied Sciences, 10(6):2020, 2020. doi: 10.3390/app10062020.
Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Pieter Gijsbers. Joan Giner-Miguelez. Nitisha Jain, Michael Kuchnik, Quentin Lhoest, Pierre Marcenac, Manil Maskey, Peter Mattson, Luis Oala, Pierre Ruyssen, Rajat Shinde, Elena Simperl, Goeffry Thomas, Slava Tykhonov, Joaquin Vanschoren, Jos van der Velde, Steffen Vogler, and Carole-Jean Wu. Croissant: A metadata format for ml-ready datasets. DEEM '24 pp. 1-6. Association for Computing Machinery, 2024. doi: 10. 10.11444/66000036663366 URL https::///// /ww./ www//0..//..// 11..1..111336555020 .122203.3663333 3663326.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. Wav2vec 2.0: A Framework for Self-Supervised Learning Speech Representations. CoRR, 2020. URL https: /doi.
org/10. 400.0445500/.i..... 2006. 11477.
Hicham Bellafkir, Markus Vogelbacher, Daniel Schneider, Markus Mühling, Nikolaus Korfhage, and Bernd Freisleben. Edge-Base Based Bird Species Recognition via Active Learning, In Nenworked Systems, volume 14067, pp. 17-34. Springer Nature Switzerland, 2023. URL ttttss://do..
org/10..100 1007 101-0000777--001--0000 /tt.:/78----111777---5----
Hicham Bellafkir, Markus Vogelbucher, Daniel Schneider, Valeryia Kizik, Markus Mühling, and Bernd Freisleben. Bird Species Recognition in Soundscapes with Self-supervisce Pre-training, In Intelligent Systems and Pattern Recognition, pp. 60-74. Springer Nature Switzerland, 2024. URL 01331-46388---
https: //o///dooi// orw.org///// 1007 978
Karl S Berg, Robb T Brumfield, and Victor Apanius. Phylogenetic and ecological determinants of the neotropical dawn chorus. Proceedings of the Royal Society of London B: Biological Sciences, 273(1188):999-1100 2006. URL https: /doi .rriiorrg/11 1098/ rspb. 2005 341 0..
11

---

Published as a conference paper at ICLR 2025
Peter C. Bermant, Leandra Brickson, and Alexander J. Titus. Bioacoustic Event Detection with Self-Suuer Contrastive Learning. Preprint, Ecology, 2022. URL htt.tttts////...r///////// 1101/2023 10.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL pettps://www.aann com/. Accessed 06-05-2024.
Francisco J. Bravo Sanchez, Md Rahat Hossain, Nathan B. English, and Steven T. Moore. Bioa- coustic classification of avian calls from raw sound waveforms with an open-source deep learn- ing architecture. Scientific Reports, 11(1), 2021. URL https: /twii org/10.1038/ / s4159B- 020-00011-6555100
Bruce E. Byers and Donald E. Kroodsma. Handbook of bird biology, chapter Avian Vocal Behavior. John Wiley & Sons, 2016.
Toon "Calders and Szymon" Jaroszewicz. Efficient auc optimization for classification. In Knowledge Discovery in Databases: PKDD 2007, pp. 42-53. Springer Berlin Heidelberg, 2007. URL https:://oi. iacoi.oogg www.rrrgg...... 1007 011///9988---00200
Mark Cartwright, Jason Cramer, Ana Elisa Mendez Mendez, Yu Wang, Ho-Hsiang Wu, Vincent Lostanlen, Magdalena Fuentes, Graham Dove, Charlie Mydlarz, Justin Salamon, Oded Nov, and An urban sound tagging dataset with spatiotemporal context, Juan Pablo Bello. wtw.vv.ivvv. oww.org//ao..o... /targ/ab//////ee/eeeeeeee//eeae/ wwtp:///gg/g..//////// 2020. URL httph:tt///o
CERN. Zenodo Research. Shared. URL https:t/////mmc /zendoo org/. Accessed 06-05-2024.
Mustafa Chasmai, Alexander Shepard, Subhransu Maji, and Grant Van Horn. The inaturalist sounds dataset. In Proceedings the 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks. NeurIPS, 2024.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset, 2020. URL https::////aaaee.. /ttss////aiiiia/eaaeeeeeeeeemav/ www.iiiorgg.i.oooooooooooooo abs 1 //////2000.2.200 14368.
Sanyuan Chen, Yu WU, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio Pre- rreTTrinnng with Acoustic Tokenizers. In Proceedings of the 40th International Conference on Machine Learning, pp. 5178-5193. PMLR, 2023.
Wenxi Chen, Yuzhe Liang Ziyang Ma, Zhisheng Zheng, and Xie Chen. EAT: Self- SuperSupe Pre-Training with Efficient Audio Transformer. In Proceedings of the T Joint Conference on Artificial Intelligence, pp- 3807-3815. International Joint Conferences on Artificial Intelligence Organization, 2024. doi: 10.24963/ijeai222244421
Lauren M. Chronister, Tessa A. Rhinehart, Aidan Place, and Justin Kitzes. An annotated set of audio recordings Eastern North American birds containing frequency, time, and species information, 2021. URL https: / doi org/ 5.0.011.....11.1 /orddd/drydd.... d2547d81z.
Mary Clapp, Stefan Kahl, Erik Meyer, Megan McKenna, Holger Klinek, and Gail Patricelli. A collection of fully-annotated soundscape recordings from the southern sierra nevada mountain range, 2023. URL https: /ooi org/ / 10....... 52811 20/81/zenn /ewoodzoooooo 7525805.
Matthew L. Clark, Leonardo Salas, Shrishail Baligar, Colin A. Quinn, Rose L. Snyder, David Leland, Wendy Schackwitz, Scott Goetz, and Shawn Newsam. The effect of soundscape composition on bird vocalization classification in citizen science biodiversity monitoring project. Ecological Informatics, 75:102065, 2023. URL https: //doi. org/ 10. 0166 w.t.o/....cin..... eccinnf. 2023. 102065.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. URL htt.tt:..... / //ooi htt.:/ii..ooorooooooooooo http////...11//PP.... .0099 2009. 5206848.
12

---

Published as a conference paper at ICLR 2025
Tom Denton, Scott Wisdom, and John R. Hershey. Improving Bird Classification with Unsupervised Sound Separation. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 636-640. IEEE, 2022. URL
11 1109/ / ICASSP . 9747202.
Philip Eichinski, Callan Alexander, Paul Roe, Stuart Parsons, and Susan Fuller. A Convolutional Neural Network Bird Species Recognizer Built From Little Data by Iteratively Training, Detecting, and Labeling. Frontiers Ecology and Evolution, 10:810330, 2022. URL https: http//ddo.ooo org 1.8.399/ fevo. 2022.. 81 0330.
William Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019. URL https:// github. www.c/om///.// http:/LLigttnnnng-gg lightning....
Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. Freesound datasets: A platform for the creation of open audio datasets. 2017.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: An open dataser of human-labeled sound events. JEEF/ACM Trans. Audio, Speech and Lang. Proc., 30: 829-852, 2021. doi: 10. 109/TASL
URL https: http//ii.org/.11111/1
TASLP 2021 3133208.
Yixing Fu, Chunjiang Yu, Yan Zhang, Danjy Lv, Yue Yin, Jing Lu, and Dan Lv. Classification of birdsong spectrograms based on DR-ACGAN and dynamic convolution. Ecological Informatics, 77:102250, 2023. URL hwtpttttp/// doi. wwwp:/o..........oo 0.160....1..11 ecoinf.
http:::/dbi..or//11.11000.0.
02250.
Avi Gazneli, Gadi Zimerman, Tal Ridnik, Gilad Sharir, and Asaf Noy. End-to-End Audio Strikes Back: Boosting Augmentations Towards An Efficient Audio Classification Network. CoRR, 2022.
URL wwt.tttts///mm / org/ 100000000000 088550/ariii 2204 11479.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach,
Hal Daumé lii, and Kate Crawford. Datasheets for Datasets. Commun. ACM, 64 (1):1))86-9) 92, 2021.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, IEEE, 2017. doi: 10.1 10.1100/CAA999220.775222.
Burooj Ghani, Tom Denton, Stefan Kahl, and Holger Klinck. Feature Embeddings from Large- Scale Acoustic Bird Classifiers Enable Few-Shot Transfer Learning. CoRR, 2023. URL https: odoi..org// 10. 48550/ aat.vi/r.ivvv/.a/ 2307 06292.
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. CoRR, 2021.
URL https::s: http::///oooooooooo 100. / aat.i//rxiii....
Gaurav Gupta, Meghana Kshirsagar, Ming Zhong, Shahrzad Gholami, and Juan Lavista Fer- res. Comparing recurrent convolutional neural networks for large seale bird species clas- sification. Scientific Reports, 11(1):17085, 2021. URL https: http:///oooooo////////o org/10.1038/
Masato Hagiwara, Benjamin Hoffman, Jen- Yu Liu, Maddie Cusimano, Felix Effenberger, and Katie Zacarian. BEANS: The Benchmark of Animal Sounds. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. 09/ CCI//CCSSSPPP 49357 2023. 1.000990 6686.
URL https::///ol org 10.00000000 1109/
Jenny Hamer, Eleni Triantafilouu, Bart Van Merriënboer, Stefan Kahl, Holger Klinck. Tom Den- ton, and Vincent Dumoulin. BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics. CoRR, 2023. URL http:: ///o org/ 1 48550/ 231 07439.
Calum Heggan, Sam Budgett, Timothy Hospedales, and Mehrdad Yaghoobi. MetaAudio: A Few-Shot Audio Classification Benchmark. In Artificial Neural Networks and Machine Learning ICANN 2022: 31st International Conference Artificial Neural Networks. Bristol. UK, September 6-9, 2022, Proceedings, Part pp. 219 2309. Springer- Verlag, 2022. doi: 10.1007/978-3-031-15911 19.
13

---

Published as a conference paper at ICLR 2025
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Examples Neural Networks. CoRR, 2018. URL https: /di. wwt.oo/gg///oo/o /ttpp//.........////aaaa.a.... 48550/axxiv. 1610. 02136.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayaman. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty, CoRR, 2020. URL https: / doi org/ 48550/ (00550//rXivy 1912.027875
WA Hopping, S Kahl, and H Klinck. A collection of fully-annotated soundscape recordings from the southwestern amazon basin. 10, 2022. URL http::://doi..rgg/ee. 0ttg://1..2///eeoooo 7079124.
Addison Howard, Holger Klinck, Sohier Dane, Stefan Kahl, and Tom Denton. nell Birdeall Identification, 2020. URL tttps://kagg// cwm/ competottiinooo birdsong-recconiiiion Accessed 06-05-2024.
Cor-
Shipeng Hu, Yihang Chu. Lu Tang, Guoxiong Zhou, Aibin Chen, and Yurong Sun. A lightweight multi-sensory field-based dual-feature fusion residual network for bird song recognition. Applied Soft Computing, 146:1 10678, 2023. URL https: /ttp//ddioooo/ooooooooo wtt.dd...........11111... aaw.:.........0000.
110678.
Po- Yao Huang. Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked Autoencoders that Listen. Advances in Neural Information Processing Systems, 35:28708-28720, 2022.
Jonas Höchst, Hicham Bellafkir, Patrick Lampe, Markus Vogelbacher, Markus Mühling, Daniel Schneider, Kim Lindner, Sascha Rösner, Dana G. Schabo, Nina Farwig, and Bernd Freisleben. Bird @ Edge: Bird Species Recognition at the Edge. In Networked Systems, volume 13464, pp. 69-86. 2022. URL https: //w.///o//////////////..// org/ / 10. 1007 3-03-----1--7666---- 0__
Turab Iqbal, Yin Cao, Andrew Bailey, Mark D. Plumbley, and Wenwu Wang. ARCA23K: An audio dataset for investigating open-set set label noise, 2022.
Lorène Jeantet and Emmanuel Dufourq. Improving deep learning acoustic classifiers with contextual information for wildlife monitoring. Ecological Informatics, 77:102256, 2023. URL https:
//doi. . org/ 10. 1016/j ecoinf. 2023. 102256.
Iver Jordal, Shahul ES, Hervé BREDIN, Kento Nishi, Francis Lata, Harry Coultas Blum, Pariente Manuel, akash raj. Keunwoo Choi, FrenchKrab, Moreno La Quatra, Piotr Želasko, amiasato, Em- manuel Schmidbauer, Lasse Hansen, and Riecardo Miccini. aatroidceanntorrchaaudimmenn 10.11.1. 2024. URL https: / wow..ooo.oooooooooooooooo....... /oo//og//o0.oo.. /00. 5281 / zenodo. 10 (2628888
Stefan Kahl, Mary Clapp, W Alexander Hopping, Hervé Goëau, Hervé Glotin, Robert Planqué, Willem-Pier Vellinga, and Alexis Joly. Overview of BirdCLEF 2020: Bird Sound Recognition in Complex Acoustic Environments. In CLEF 2020 Conference and Labs of the Evaluation Forum, volume 2696 of CEUR Workshop Proceedings, 2020.
Stefan Kahl, Tom Denton, Holger Klinck, Hervé Glotin, Hervé Goëau, Willem-Pier Vellinga, Robert Planqué, and Alexis Joly. Overview of birdclef 2021: Bird call identification in soundscape recordings. In CLEF 2021 Conference and Labs the Evaluation Forum, volume 2936 of CEUR Workshop Proceedings, pp. 1437-1450, 2021a.
Stefan Kahl, Connor M. Wood, Maximilian Eibl, and Holger Klinck. BirdNET: deep learning solution for avian diversity monitoring. Ecological Informatics, 61:101236, 2021b. URL https: //doi. org/10 /00...0... 1016 6 www.ee........ccc 002222.112223. 101236.
Stefan Kahl, Russell Charif, and Holger Klinck. A collection of annly-an soundscape recordings from the northeastern united states, 2022a. URL httpt: /doi. www.orrg/.0.. /t.g/10..28. 5281 zenodo.
7079380.
Stefan Kahl, Amanda Navine, Tom Denton, Holger Klinck, Patrick Hart, Hervé Glotin, Hervé Goëau, Willem- Pier Vellinga, Robert Planqué, and Alexis Joly. Overview of BirdCLEF 2022: Endangered bird species recognition in soundscape recordings. In CLEF 2022 Conference and Labs of the Evaluation Forum, volume 3180 of CEUR Workshop Proceedings, pp. 1929-1939, 2022b.
14

---

Published as a conference paper at ICLR 2025
Stefan Kahl, Connor M. Wood, Philip Chaon, M. Zachariah Peery, and Holger Klinck. A collection of fully-annotated soundscape recordings from the western united states, 2022c. URL https: //ooi.ooo org/10.gg1000 5281/2enodo. 7050014.
Stefan Kahl, Tom Denton, Holger Klinck, Hendrik Reers, Francis Cherutich, Hervé Glotin, Hervé Goëau, Willem- Dillem-Pierr Vellinga, Robert Planqué, and Alexis Joly. Overview of BirdCLEF 2023: Automated Bird Species Identification Eastern Africa. In CLEF 2023 Conference and Labs of the Evaluation Forum, pp- 1934-1942, 2023.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Association for Computational Linguistics, 2021. URL https: : /oo.oooooooooo wrgg 0 8550/ arxiv. 21 2109. 02846.
Jiang Liu, Yan Zhang, Danjy Lv, Jing Lu, Shanshan Xie, Jiali Zi, Yue Yin, and Haifeng Xu. Birdsong classification based on ensemble multi-scale convolutional neural network. Scientific Reports, 12 (1):8636, 2022a. URL https: //wii org/10. 1038/ s 441598 0e////4118--0022222222 022-12121-8
Zhihua Liu, Wenjie Chen, Aibin Chen, Guoxiong Zhou, and Jizheng Yi. Birdsong classification based multi feature channel fusion. Multimedia Tools and Applicutions, 81(11):1549-15590 2022b. URL https: / .rroi.org/li 10. /1t0///110//--//...111 022 1257222-1.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11966-111 11966. IEEE, 2022c. URL https:
/
CVPR52688 2022. 01167.
Vincent Lostanlen, Justin Salamon, Andrew Farnsworth, Steve Kelling, and Juan Pablo Bello. Birdvox-Full-Night A Dataset and Benchmark for Avian Flight Call Detection. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 266-270. IEEE, 2018a. URL http:: http://oooooor///o//////o / 1109 TCASSP 2018. 8461400.
Vincent Lostanlen, Justin Salamon, Andrew Farnsworth, Steve Kelling, and Juan Pablo Bello. BirdVox-DCASE--0K: A dataset for bird audio detection in 10-second clips. 2018b. URL https: //doi. org/ / 1208C80.
Félix Michaud, Jérôme Sueur, Maxime Le Cesne, and Sylvain Haupert. Unsupervised classification to improve the quality of a bird song recording dataset. Ecological Informatics, 74:101952, 2023. 10.1100..2 1016/ ecoinf. 2022 1022.1011 101952.
URL https: //tp:://do//
Veronica Morfi, Yves Bas, Hanna Pamuła, Hervé Glotin, and Dan Stowell. Nips4bplus: a richly annotated birdsong audio dataset. CoRR, 2018. URL //////dodd doi. org. /00 48550///4455//xxx 1811. 02275.
llyass Moummad, Nicolas Farrugia, and Romain Serizel. Self-supervised learning for few-shot bird sound classification. In 2024 IEEE International Conference Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp. 600-604, 2024. doi: 10. 110//CASPPW244455004 10627576.
Amanda Navine, Stefan Kahl, Ann Tanimoto-Johnson, Holger Klinck, and Patrick Hart. A collection of fully-annotated soundscape recordings from the island of hawai'i, 2022. URL https: //doi. org/10. 528152281// /eoooo. 7078499.
Victoria Nolan, Chris Scott, John M. Yeiser, Nathan Wilhite, Paige E. Howell, Dallas Ingram, and James A. Martin. The development of convolutional neural network for the automatic detection of Northern Bobwhite Colinus virginianus covey calls. Remote Sensing in Ecology and Conservation, 91)):6--1, 2023. URL htt.h:tt//:/ate http://oooooo/o/o////////o org/ 1002 rse2. 294.
15

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. CoRR, 2019.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23nd ACM International Conference on Multimedia, pp- 1015-1018. ACM, 2015. URL https:
// //doo..or//// 1tt1111..1155 1ew.11114223333 2287333373828333.3 2306090.
E.C. Pielou. The measurement of diversity in different types of biological collections. Journal of Theoretical Biology. 13:131-144, 1966. doi: https:/doi..rg// 1ttp/11..10 1101/0002-51996666999000000
Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schluter, Shuo-Yin Yiin Chang, and Tara Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics Signal Processing, 13(2): 206-219, 2019. doi: 10. 10.1100 10.1100/jstspp200110 109//sspp22019.29880000 URL http:/ /xxx/////////////// doi wwt./o.gg/....... JSTSP 2019. 2.11922990000
Lukas Rauch, Matthias Aßenmacher, Denis Huseljic, Moritz Wirth, Bernd Bischl, and Bernhard Sick. Activeglae: A benchmark for deep active learning with transformers. In Machine Learning and Knowledge Discovery in Databases: Research Track, pp. 55-74. Springer Nature Switzerland, 2023a. URL http:: orr.i..rr//ooo http://.....ooo..1000//000/////0 0tt.://0-0100//-----------------
Lukas Rauch, Raphael Schwinger, Moritz Wirth, Bernhard Sick, Sven Tomforde, and Christoph Scholz. Active Bird2Vec: Towards End-to-End Bird Sound Monitoring with Transformers. CoRR, 2023b. URL httph:tt///// http://ooo.ooooo...////////oo 48550/ 11550//0Xiv...... 2308.. 07121.
Lukas Rauch, Denis Huseljic, Moritz Wirth, Jens Decke, Bernhard Sick, and Christoph Scholz. Towards Deep Active Learning in Avian Bioacoustics. CEUR Workshop Proceedings IAL'24, 2024a.
Lukas Rauch, Raphael Schwinger, Moritz Winth, and René Heinrich. Birdset, 2024b. http:: / //huggingface. co/
Mirco Ravanelli and Yoshua Bengio. Speaker Recognition from Raw Waveform with SincNet. CoRR, 2019. URL https: doi. http://///boooor....0/33333/ 48550/ wtg.l.....///.//......
1808. 00158.
Samuel R. P.-. Ross, Darren P. O' O'Cnnlll,, Jessica L. Deichmann, Camille Desjonquères, Amandine Gase, Jennifer N. Phillips, Sarab S. Sethi, Connor M. Wood, and Zuzana Burivalova. Passive acoustic monitoring provides a fresh perspective on fundamental ecological questions. Functional Ecology, 37(4):959 975, 2023. URL https: /ww. wttp:/ooooo....1166666111
14275.
Çagan H. Sekercioglu, Daniel G. Wenny, and Christopher J. Whelan. Why Birds Matter: Avian Ecological Function and Ecosystem Services. University of Chicago Press, 2016. URL https: // //doi. org/10 10. 7208/ Btt////cccc///////////// (ttp:///9822222288788 00010300777700110..11011 001.00010000000000.0 0001.
Dan Stowell. Computational bioacoustics with deep learning: A review and roadmap. CoRR, 2021.
URL https: http://o..oor///////////
2112.
Dan Stowell, Yannis Stylianou, Mike Wood, Hanna Pamula, and Hervé Glotin. Automatic acoustic detection of birds through deep learning: The first Bird Audio Detection challenge. Methods in Ecol- ogy and Evolution, 10(3):368-380 380. 2019. URL https: / //w/////////////////////o/ org/10.......... 1111122411-21xx 13103.
Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation. The MIT Press, 2012. URL https://doiiii /rr/ 10. 7551 / ///mmiittt/ mitpresss//// 9t..//7802222
10100000 0001.
Brian L. Sullivan, Christopher L. Wood, Marshall Iliff, Rick E. Bonney, Daniel Fink, and Steve Kelling. eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation, 142(10))2282-2D)) 2292, 2009. URL https: /oo.. wrg/10 10. 1016/j biocon. 3009.05.000
16

---

Published as a conference paper at ICLR 2025
Bhuvaneswari Swaminathan, M. Jagadeesh, and Subramaniyaswamy Vairavasundaram. Multi- label classification for acoustic bird species detection using transfer learning approach. Ecologi- cal Informatics, 80:102471, 2024. URL htt.pp::///....o//11.... 10t.://............ http:///wr........e.oooooo...... www.on..........22...... 102471.
Mingxing Tan and Quoe V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, 2020. URL https:/ /doi. org/10. .tleg//...855100
48550/ arxiv. 1905. 11946.
Quan Tang, Liming Xu, Bochuan Zheng, and Chunlin He. Transound: Hyper-head attention transformer for birds sound recognition. Ecological Informaties, 75:102001, 2023. URL https: //doi orr/ 10. ( ecoinf. 2023. 102001.
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. Rudner, Karan Singhal, Zachary Nado, Joost Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan. Plex: Towards Reliability using Pretrained Large Model Extensions. CoRR, 2022. URL https:///// doi. org/ / 10. 8550 / arxiv. 2207 07411.
Bart Van Merriënboer, Jenny Hamer, Vincent Dumoulin, Eleni Triantafillou, and Tom Denton. Birds, Bats and beyond: Evaluating generalization in bioacoustic models. CoRR, 2024.
Álvaro Vega-Hidalgo,, Stefan Kahl, Laurel B. Symes, Viviana Ruiz-Gutiérrez, Ingrid Molina-Mora, Fernando Cediel, Luis Sandoval, and Holger Klinek. A collection of funly-annotated soundscape recordings from neotropical coffee farms in colombia and costa rica, 2023. URL https: /doi. org/10. 5281 zenodo. 7525349.
Willem-Pier Vellinga and Robert Planqué. The xeno-canto collection and its relation to sound recognition and classification. CEUR- WS.R. 2015. URL https: wwweooccaae org/.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355. Association for Computational Linguistics, 2018. URL https: / /doi..i... org/ 10. 10..8853/va0 v/v /11 8 5446.
Hanlin Wang, Yingfan Xu, Yan Yu, Yucheng Lin, and Jianghong Ran. An Efficient Model for Vast Number Bird Species Identification Based on Acoustic Features. Animals, 12(18):2434, 2022. URL https:://///m.c / / doii..... org/10 100..//..00 3390 /aninni 1af112212444
Yu Wang, Nicholas Bryan, Justin Salamon, Mark Cartwright, and Juan Pablo Bello. Who calls the shots? Rethinking Few-Shot Learning for Audio, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 38-45, Association for Computational Linguistics, 2020.
doi: 18.18653/v 1ttp:3//1//220..m/ddddemeeee
Hanguang Xiao, Daidai Liu, Kai Chen, and Mi Zhu. AMResNet: An automatic recognition model of bird sounds in real environment. Applied Acoustics, 201:109121, 2022. URL https: //doi.
org/ /00 101 6 apacoust 2022. 109121.
Jie Xie and Mingying Zhu. Sliding-window based scale-frequency map for bird sound classification using 2D- and 3D-CNN. Expert Systems with Applications, 207:118054, 2022. URL https: //doi. 10.
2182511
Omry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019. URL https: / thub. wcom///aceebooeeseerchhh/ooooe
17

---

Published as a conference paper at ICLR 2025
Chengyun Zhang, Qingrong Li, Haisong Zhan, YiFan Li, and Xinghui Gao. One-step progressive representation transfer learning for bird sound classification. Applied Acoustics, 212:109614 2023.
URL https::t////.. //t:://doooo /ww.oooooooooooooooo www.ooo./...o.oooo 101.://....11.//........... apwcouss...ssp..oppoaopoa 2023 109614.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond Empirical 1710.
Risk Minimization. CoRR, 2018. URL ptlttp::/doi..oo
09412.
18

---

Published as a conference paper at ICLR 2025
APPENDIX
A OUTLINE
This supplementary material complements our main findings and provides additional information. Below, we outline the key resources included in external materials.
Overview Supplementary Material
1. We have uploaded our large-scale dataset collection BirdSet" (Rauch et al., 2024b) to Hugging Face Datasets and provide an extensive code repository", which includes the training and evaluation protocols, is available on GitHub. We also provide a leaderboard" on Hugging Face to promote comparability and encourage participation in our benchmark.
2. To promote transparency and reproducibility, have made all experimental results available through Weights&Biases (Biewald. 2020).
3. We provide access to our trained models" through Hugging Face Models (Wolf et al., 2020), allowing others to reproduce results without the need to train from scratch. By doing so, we offer trainable baseline models to facilitate further experimentation and development.
We provide croissant (Akhtar al., 2024) metadata file in our code repository.
group
The remainder of the supplementary material is structured follows. Scction B provides detailed insights into the Bi rdret dataset collection. Section C offers comprehensive information on the experimental setup, including the metrics used and in-depth presentation of the results. Section E presents structured datasheet that offers a clear overview of the benchmark, facilitating a thorough understanding of its scope and structure.
B BIRDSET: A DATASET COLLECTION
This section provides additional details on the Bi rdSet dataset collection. We include information on licensing, maintenance, and hosting, data collection processes, dataset descriptions, and dataset statistics.
B. LICENSING
The dataset collection BirdSet (Rauch et al., 2024b) is available under the creative commons (CC) CC-BY-NC-SA license. Researchers are permitted to use this dataset exclusively for non- commercial research and educational purposes. Each training recording in BirdSet sourced from XC is associated with its own CC license, which can be accessed via the metadata file on Hugging Face. We have excluded all recordings with non-derivative (ND) licenses. All test datasets are licensed under CC-BY-4.0, and the validation dataset is licensed under CCO-1.0.
Table 4: License overview the training dataset from XC.
License
XC recordings
CC-BY-NC
128
CC-BY-SA
7, 293
CC-BY
199
CC-HY-NC-SA
484, 350
CC-0
706
19

---

Published as a conference paper at ICLR 2025
We have carefully collected the recordings of this dataset collection. We, the authors, bear all responsibility to remove data or withdraw the paper in case of violating licensing or privacy rights upon confirmation of such violations. However, users responsible for ensuring that their dataset use complies with all licenses, applicable laws, regulations, and ethical guidelines. We make no representations warranties of any kind and accept no responsibility in the case of violations.
B.2 MAINTENANCE AND HOSTING
The dataset is hosted Hugging Face Höchst al. (2022), enabling fast and easy accessibility. We maintain a complete backup of the dataset collection on our servers to ensure long-term availability. In addition the hosted dataset, we provide data loading script and respective metadata, allowing manual data retrieval. The dataset will be supported, hosted, and maintained by the DBD-research- group, initiated by the Deep Bird Detect project. The authors of this paper are committed to maintaining the code and dataset collection. For any inquiries or support requests, please contact rauch @ @nni-kassel.de. Updates to dataset will be implemented as
the primary author at rauch necessary. While the test datasets will remain unchanged to ensure consistency in evaluations, the training datasets may be expanded or refined in response to new data that becomes available.
B.3 DATA COLLECTION
The data collection process for the test and training datasets was planned to provide high-quality, representative, and legally compliant data. We carefully curated the test datasets from range of high- quality soundscape recordings that simulate real-world PAM environment. We chose these datasets to represent different difficulty levels, encompassing class diversity, class imbalance, geographical variations, and the density of vocalizations within each segment. Such diversity is essential for evaluating the robustness and generalizability of the models across diverse scenarios, ensuring the test datasets accurately reflect practical conditions. Additionally, the collection process was guided by the availability of appropriate licenses, guaranteeing that all included recordings are legally compliant and suitable for use on platforms such as Hugging Face.
Table 5 displays the metadata we provide for BirdSet. Note that all information is available for every recording and recording type. For instance, detected events or peaks are included in the focal training dataset, not the test datasets, since soundscape recordings are precisely annotated with specific timeframes for vocalizations. Additionally, for focal recordings, sources from XC, secondary calls, or the species' sex are only sometimes available.
Table 5: Metadata of the dataset collection BirdSet (Rauch et al., 2024b) on Hugging Face Datasets. We mark the availability of the respective information in train test.
Format
Description
Train
audio
Andiolsampting ate-32
recording from hf
Kilcpath
3aluet"tring")
relutive where recording
stwrt: timee
Valuet "ont""
dant time of vocallzation
end tdtme
Valuet Hoott)
end time of vocaliation
Valuet int")
low frequency for vocalizatonn kilr
Valuet'im"")
high frequency bound vocalization kätz
cbird__ood
assigned label
Sequenceddausee Valuer's stting"))
secondary apecies recooding
eb1rd code molei labe)
Sepuencedatmeet Claed Koomesschon lon)
assigned species label mutriabel format
Sequenceddameett Valuer "smnng))
of hind vocalization
Valuet "stnin")
bord
Houre""oour)
luitude vocalleatooneeeeddugg WGS34
Malue("oour)
loagitude wocalizaloot/eeeedin WG884
langsa
Nalue(Tlu")
lenath
15cense
Valuet
license the reconinng
Naluetsttnng)
source recording
local_
Valuer'sting
local sime recording
Gotoctad_evend
Sopuencedusses Seguenceddausess Valuel (oat")))
desocted reconding bumbid,
Sapuncerdasses. Midee"lmc"))
Josected undio
clusee buntiee
Sepuncedlassss. Maueellout))
descaion
quality
aesording qualit rasanding from
Value()tring))
recordist recceding from XC:
20

---

Published as a conference paper at ICLR 2025
B.4 DATASETS OVERVIEW
Table 6: Avian data sources in related work.
Abbr.
Reference
#Labels Open-Suurce
RT2
Bird&Data
14,311
NES
Colombia Costa Rica (Vega-Hidalgo et al., 2023)
6.952
CAP
Caples Denton al. (2022)
2,944
CBI
Conell Bird Idcatification Howard et al. (2020)
21,382
ISN
High Siena Nevada Clapp ot al. (2023)
21,375
1MA
INaturalist Chasmui et al. (2024)
604,284
NAC
Macuulay Library
14,530
NBP
NIPS4BPlus Morli et al. (2018)
1,687
Amazon Basin Hopping et (2022)
14,798
POW
Powdermill Nature Chronister et al. (2021)
16,052
521.
Soundscupes] aansscupes Clark et (2023)
SNR
Siema Nevada Kahi et al. (2022c)
20,147
Sapsucker Woods Kahl al. (2022a)
50,760
U10
Hawaiian Islands Navine et al. (2022)
59,583
VOX
BirdVox Lostanlen al. (2018a)
35,402
XC
Xeno- Vene-CantVVellinga & Planqué (2015)
677,429
Amazon Basin (PER). The Amazon Basin (Hopping al., 2022) soundscape test dataset comprises 21 hours of annotated audio recordings from the Southwestern Amazon Basin, featuring 14,798 bounding box labels for 132 bird species. These recordings were collected at the Inkaterra Reserva Amazonica in Peru during early 2019. The dataset includes diverse forest conditions and was partly used for the 2020 BirdCLEF competition (Kahl al., 2020). Annotations specifically targeted the dawn hour across seven sites on multiple dates to capture the peak vocal activity of neotropical birds.
Columbia Costa Rica (NES). The Columbia Costa Rica VVga-Hidalgee et al., 2023) soundscape test dataset contains 34 hours of annotated audio recordings from landscapes of Colombian and Costa Rican coffee farms. This dataastt comprises 6,952 labels for 89 bird species, and was partially featured in the 2021 BirdCLEF competition (Kahl et al., 2021a). The recordings primarily capture dawn choruses from randomly sampled farm locations and dates.
Hawaiian Islands (UHH). The Hawaiaan Islands (Navine et al., 2022) soundscape test dataset features 635 recordings totaling nearly 51 hours, annotated by ornithologists with 59,583 labels for 27 bird species. This dataset includes recordings from endangered native species from Hawaii, gathered across four sites on Hawai'i Island from 2016 to 2022. It was utilized in the 2022 BirdCLEF competition (Kahl et al., 2022b).
High Sierra Nevada (HSN). The High Sierras (Clapp et al., 2023) soundscape test dataset comprises 100 ten-minute audio recordings from the southern Sierra Nevada in California, annotated with over 10,000 labels for 21 bird species. This dataset was featured in the 2020 BirdCLEF competition (Kahl et al., 2020). The recordings were captured in high-clevation regions of Sequoia and Kings Canyon National Parks to study the ecological impact of trout on birds. Annotations were rigorously verified by experts.
NIPS4Bplus (NBP). The Neural Information Processing Scaled for Bioacoustics Plus (Morfi et al., 2018) an enhanced version of the dataset NIPS4B 2013, developed for bird song classification with detailed temporal species annotations. Collected across diverse climates and geographical features in France and Spain, this soundscape test dataset was initially targeted at bat echolocation but also successfully captured bird songs. It is designed to maximize species diversity, distinguishing from other European datasets regarding the number of recordings and species classes. This dataset undergoes more extensive processing achieve balance among classes, simplifying the classification task compared to other datasets.
Powdermill Nature (POW). The Powdermill Nature (Chronister et al., 2021) soundscape validation dataset is a collection of strongly-labeles bird soundscapes from Powdermill Nature Reserve in the Northeastern United States. It features 385 minutes of dawn chorus recordings, captured over four
21

---

Published as a conference paper at ICLR 2025
days from April to July 2018. These recordings include vocalizations from 48 bird species, with a total of 16,052 labels. Since it includes species that overlap with other test datasets and can be processed quickly, it has been selected as a validation dataset.
Sapsucker Woods (SSW). The Sapsucker Woods soundscape test dataset (Kahl et al., 2022a) consists of 285 hour-long soundscape recordings from the Sapsucker Woods bird sanctuary in Ithaca, New York. Captured in 2017 by the Cornell Lab of Ornithology, these high-quality audio recordings include 50,760 annotations for 81 bird species. The dataset primarily aims to study vocal activity patterns, local bird species' seasonal diversity, and noise pollution's 's impact on birds. It has also been instrumental the 2019, 2020, and 2021 BirdCLEF (Kahl al., 2020; 2021a; la; 2022b) competitions. The annotation process involved carefully selecting and labeling bird calls, with each call boxed in time and frequency.
Sierra Nevada (SNE) (Kahl et al., 2022c) The Sierra Nevada soundscape test dataset features 33 hour-long audio recordings from the Sierra Nevada region in California, annotated with 20, 147 labels for 56 bird species. These recordings, collected in 2018, were specifically gathered to study the impact of forest management on avian populations. The dataset was part of the 2021 BirdCLEF (Kahl et al., 2021a) competition. It was captured across diverse ecological settings in the Lassen and Plumas National Forests, targeting various elevations and latitudes.
(VOX). Under the VOX abbreviation, compile soundscape auxiliary dataset derived from the DCASE18 bird audio detection challenge (Stowell et al., 2019). This challenge required participants design systems capable of detecting the presence of birds in binary classification setting using soundscape recordings. As these training datasets have ground truth labels regarding the presence of bird sounds, they well-suited for background augmentations for segment-based PA.M. It incorporates 20,000 audio clips from remote monitoring units in Ithaca, NY, USA, focusing on flight calls of birds (Lostanlen et al., 2018b).
Bird Vox--oXS--AEEE
Xeno-Canto (XC) (Vellinga & Planqué, 2015) XC sources all focal training datasets and acts as citizen-seience platform dedicated avian sound recording. XC was launched in 2005 and aims to popularize bird sound recording, improve accessibility to bird sounds, and expand knowledge in this field. As collaborative repository. it allows enthusiasts worldwide to upload bird sounds with essential metadata, including species, the recorder's 's name, location, and date. The platform allows anyone with internet access to upload recordings, provided they include a minimum set of metadata such species, recordist name, location, and recording date. Despite varying submission quality, all contributions are valuable, capturing rare or common bird vocalizations. XC's collection features nearly 10,00 bird species with more than 800,000 recordings, and its collection is continuously growing.
B.5 DATA COLLECTION STATISTICS
We explore the characteristies of the BirdSet dataset collection, emphasizing the soundscape test and validation datasets. We display the species composition of test and validation datasets in Figure 7. Additionally, we highlight the overlaps between test and validation segments in Figure Figure 9 shows the number of annotations per species in the XCL and evaluation datasets. This analysis not only clarifies the datasets' structures but also aids in understanding the varying difficulty levels of the test datasets, reflecting the influence of different environmental and biological factors. Table 7 complements our graphical abstract in the introduction, showcasing a range of datasets in environmental audio classification.
22

---

Published as a conference paper at ICLR 2025
000000
Absolute Specties [#]
Relative Speciae [%]
40000
60
30000
40
20000
10000
NPP (51)
NES (89)
HSS (21)
PER (132)
POW (48)
SIVE (56)
sSw (81)
umn (27)
NDP (51)
NES (89)
IISN (21)
PER (132)
pow (46)
SIE (36)
ssw (81)
UHH (27)
Figure 7: Species composition test and validation datasets, presented in absolute counts (#) and
relative percentages (%) Colored sections indicate unique species within each dataset. Identical colors do not correspond to the same species.
overlaps
total
60000
50000
Annofat heass [#]
40000
30000
20000
10000
NBP
NES
HSN
PER
POW
SNE
SSW
UHH
Figure The number annotations where overlaps are present. Annotations indicate the number of annotations present in each dataset.
Table 7: Available datasets in environmental audio classification.
Dataset
Clips Length Duration [h] Classes
Source
Domain
Label
BirdSet (train)
528k
6,877
10,296
Xeno-Cante Canto
Avian
File
BirdSet (eval)
322k
411
503
Cano-Canto
Avian
Event
INatSounds (train) Chasmai et al. (2024)
137k
1,100
5,569
iNaturalist
Avian
File
Sounds (eval) Chasmai et (2024)
95k
1,212
iNaturalist
Avian
AudioSet (train) Gemmeke
(2017)
1.8M ==10x
5,790
527
YouTube
Universal
Event
AudioSet (eval) Gemmeke et al. (2017)
300k 1010s
56
527
YouTube
Universal
File
FSD Fonseca al. (2017)
297k
632
Freesound
Universal
File
FSD50K (train) Fonseca al. (2021)
51k 0.3-30s
108
200
Freesound
Universal
File
FSD50K (eval) Fonseca et al. (2021)
10k 0.3-30s
28
200
Freesound
Universal
File
(train) Chen et (2020)
200k
310
YouTube
Universal
File
VGG (eval) Chen al. (2020)
21k
310
YouTube
Universal
File
SONY CONYC-UST-V2 (train) Cartwright et al. (2020)
14k 10s
37
30
SONYC
Urhan
File
SONYC-UST-VS (eval) Carwwiggt et (2020)
5k
30
SONYC
Urban
File
ESC50 Piczak (2015)
2k
50
Freesound
Environmental
Event
23

---

Published as a conference paper at ICLR 2025
HSN
NBP
NES
whespa
amepip
gorfin
1260
1643
5036
caswarl
eurgol
276
406
420
rucspal
whtdov
rucwar
377
762
1153
954
darwarl
241
yeboril
32
clanut
216
Iotman!
318
eBtrd Code (21)
mouchi
3 classiss unjitted...
71
<Bird Code (51)
houspa
18 cludoes oeaitted..
eBird Code (88)
gryhaw2
37 dujsses omitted...
21
moublu
34
comcha
82
culchal
21
pexspa
casfin
30
31
comchil
grywag
71
runwre!
yeceupl
20
21
yerwar
19
redero
69
bugtan
uraitted...
18 clossss cmnil
37 dassses omitted...
warvir
eurgrel
andsoll
dusfly
eurwry
20
mallar3
carcrol
yetvir
brebla
sursta
19
trsow
1
yolwar
eujjayl
acowoo
10"
102 Annotations [+]
104
10
10
10a
10a
10"
10
10"
10'
Annotations
Annotations
POW
SNE
PER
eBird Code (132)
butwoo1
wespuft
bauthrl
thlwrel
gryant2
24
28
583
764
943
1694
eBird Code (48)
amgglo
btowar
woothr
eastow
carwre
blochi
tuftit
classes omitted...
48
62
885
1058
1262
1826
eBird Code (56)
bnhcow
towwar
amerob
mouchi
gockin
herthr
classes brtuttee...
40
4.3
1366
1564
2003
2696
3232
grameul
cinmoul
23
24
buwwar
louwat
46
47
amnbum
34
39
astgual
ruckin
35
28
diasees omitted...
20 classes aesitte...
strxen1
cedwax
royfly1
brncre
wewpew
dubpar?!
reshaw
wilsap
barantl 1
pilwoo
redero 1
reatanl
comrav
amegfi
10a Annotations [W]
10"
10+
10"
Annotations [#]
Annotations [#]
SSW
UHH
XCL
eBird Code (81)
moudov
ovenbil
buwwar
wooduc
woothr
rewbla
bkechi
33 classes omitted...
106
105
2./13
2760
2981
4235
4973
Bird Code (27)
hawpetl
hawama
hawure
аpapaа
barpet
houftn
ercfra
iiwi
classes Imiited...
409
634
3612
3461
22538
eBird Code (9734)
yevwoo!
comchit
currobi
parpip1
comcha
gretit1
eurbla
14
14
temited....
3360
3477
3779
4109
4373
bahcow
98
jabwar
295
groweal
aldfly
93
hawbaw
golmon!
14
classes omitted...
cincees amitted...
classed omiited....
amered
palia
rucdovi Hl
benher
kalphe
scbant2
nuckin
comnwax
perpig2 1
rodero
bikdra
thswar9
ribgul
bawgoo
brqdovi
10"
10
10
10a
10"
Annotations (#)
Annotations [#]
Annotations [#]
Figure 9: Top, middle, and bottom five number of species appearances (vocalizations) in each evaluation and the complete XCL training dataset.
24

---

Published as a conference paper at ICLR 2025
C DETAILED EXPERIMENTAL SETTING AND RESULTS
This section offers additional experimental insights, detailing the computational resources and assets utilized. It outlines the experimental setup, including model parameters, describes the augmentations applied, and provides specifics about the evaluation metrics. We also include a comprehensive presentation the results.
C.I COMPUTATIONAL RESOURCES
We primarily utilized an internal Slurm cluster equipped with NVIDIA A100 and V100 GPU servers from the IES group at the University of Kassel, predominantly using the NVIDIA A 100 GPU servers for large-scale training. Collaboration with researchers from the University of Kiel and Fraunhofer IEE also involved using NVIDIA A100 GPUs within their internal compute clusters. Additionally, we conducted smaller-scale experiments on workstation equipped with an NVIDIA RTX 4090 GPU and an AMD Ryzen 9 7950X CPU. Due to variations in the conditions across different clusters, direct comparisons of training times are challenging. However, we provide detailed training and inference times and additional details for our computational resources on Weights& Biases.
C.2 ASSETS
In our benchmark and code, we leverage a variety of existing assets. By integrating these existing assets, can focus our research objectives utilizing well-established tools and platforms. Our primary assets include:
Hugging Face Datasets (Höchst et al., 2022) (platform and code, Apache-2.0 license): We use the Hugging Face Datasets platform for hosting and processing our data collection. This enables fast and efficient accessibility.
Hugging Face Models (Wolf et al., 2020) (platform and code, Apache-2.0 license): For model deployment, we utilize various models from Hugging Face Models, including Effi cientNet (Tan & Le, 2020), AST (Gong al., 2021), Wav2Vec2 (Baevski et al., 2020), and ConvNext (Liu et al., 2022c). Hugging Face is also utilized to host the checkpoints of our trained models.
PyTorch (Paszke al., 2019) (code) and PyTorch Lightning (Falcon & The Py Torch Light- ning team, 2019) (code, Apache-2.0 license): Our codebase is built using Py PyTorch and the PyTorch Lightning frameworks. This provides a robust and scalable environment for developing and training models.
Torch Audiomentations (Jordal et al., 2024) (code, MIT license): We employ Torch Au- diomentations for data augmentation, allowing us to enhance the variability and robustness of our training data.
BamBird (Michaud et al., 2023) (code, BSD-3-Clause license): BamBird used for detecting events in the focal recordings, facilitating training and efficient analysis.
Hydra (Yadan, 2019) (code, MIT license): We use Hydra for experiment management, enabling us to organize and streamline our experimental workflows effectively.
& (Biewald, 2020) (platform and code, MIT license): We utilize
Weights&Biasee for tracking our experiments and publishing results in detail.
Zenodo (CERN) (platform) and Xeno-Canto (Vellinga & Planqué, 2015) (platform) were the source of the train, test, and validation dataset collection.
These assets enable us to establish readily accessible dataset collection, fostering reproducibility and comparability of our results.
C.3 EXPERIMENTAL SETUP
Table provides a detailed overview of the parameters used to generate baselines for all training scenarios. It serves addition to the main article.
25

---

Published as a conference paper at ICLR 2025
Table 8: Model and training parameters for training scenarios and MT and DT Note that Perch is not further trained in our benchmark and can only be employed as a black box for inference. The species limit caps the number of samples from any one species to prevent imbalance, while the event limit restricts extractions per recording ensure dataset diversity in the training data.
Parumeter
Perch (Hamer at... 2023)
EfficientNed (Tanl
ConvNext (Liu 2023c)
AST (Gong al., 2021)
EAT (Gazneli et., 2022)
W2V2 (Baceski 2020)
2020)
Model parameters
Ingat type
Spec steps Adam None
Spec ImageNet
Spec ImaggoNet 30 AdamW So-4
Spee AudioSet
Wave
Wave Librehpeech
Pretrained
Epocas
30 Adam
Optimizeer
AdamW Sc-4
AdammW
AdamW 1e-2
We Weight decay
Te-2
1o-5
Learning rate
le.3
Se-4
Se-4
le-5
3e-4
3e-4
Scheduler
Con-Anneal.
Cos-Anneul.
Cos-Anneal.
Cos-Anneal
Cos-Annehl
Kacmup
Sc-2
Se-2
Batch Size
256
64
64
12
128
64
Validation
20%TT"
POW.
POW, 20%T
20%
POW, 20
Leas
BCE
BCE
BCE
BCE
BCE
BCE
Architecture
Base224
Farameters
19.0
19.0 M
97.5 M
93.7
5.2M
97.1 M
Spectrogram purameters
:2t
1024
2048
1024
1024
hop length
320
256
320
320
20
2.0
2.0
Melscale parameters
128
STFT
1024
513
513
GB Scale
PCEN
Processing parameters
Norm wave
peak morm
Instance norm AudioSet M//id
instance Borm
intstance barm
spec
Audiošal M/
AudioSer
Resize
1024
Seeds
3,5
Data parameters
Sampling rate
32kHz
32 KHz
32 &Hz
KHz 10, 5"
32
16 AHz
Event limit
0, 5"
5.55
1 0, 5"
10,
Species limit
500, 0, 600
500, D, 600*
500, 0, 600"
500, D. 600"
500, 600"
Train samples
750,000
1,528,068, variable, 558,455*
C.4 AUGMENTATIONS
To counteract covariate shift, we aim to pinpoint augmentations that effectively bridge the gap between the training distribution (focal recordings) and the test distribution (soundscape recordings). Moreover, seek to tackle the task shift dilemma when transitioning from multi-class to multi-label setting. We differentiate between waveform augmentations and spectrogram augmentations:
Waveform augmentations are applied before spectrogram conversion the raw waveform. They are always used independently of the input type.
Time Shifting (p=1.0) 1.0) modifies the temporal aspect of the audio to enhance model robust- against timing variations. Upon detecting vocalization event, capture 8-second window surrounding and then randomly select a 5-second segment from this window. This approach ensures that vocalizations not confined to the start of event.
Background Noise Mixing (p=0.5) incorporates diverse noise profiles to train the model in distinguishing signal from noise. This augmentation is partieularly beneficial as it allows to incorporate the VOX auxiliary dataset, which consists of background noise from soundscape recordings without bird vocalizations. plays a crucial role in enhancing the evaluation performance.
Gain Adjustments (p-0.2) vary the volume to ensure the model's resilience to amplitude fluctuations. Additionally, this method assists in aligning the training's focal recordings closer to the characteristics of soundscape recordings, which are captured with omnidirec- tional microphones featuring diverse spatial attributes.
Multi-Label Mixup (p=0.7) enhances sample diversity by mixing up two samples from a batch, extending the conventional Mixup (Hendrycks et al., 2020) technique to include the labels of the combined recordings. This adaptation aims to boost model generalization and
26

---

Published as a conference paper at ICLR 2025
address task shifts, effectively creating synthetic multi-labe challenge within a multi-class dataset. Moreover, it narrows the gap between the train and test distributions.
No-Call Mixing (p=0.075) enriches training by substituting samples with segments labeled as no-call (0-vector). This adjustment is crucial for segment- beaee evaluation in PAM scenarios, where soundscapes may contain segments devoid of bird vocalizations. This situation is not encountered in training focal recordings, where events are pre-identified by a recordist. We also leverage the VOX auxiliary dataset, which comprises soundscapes devoid of any sounds, to simulate no-call scenarios effectively.
Spectrogram augmentations are applied after the spectrogram conversion. They cannot be used when the input type only accepts raw waveforms.
Frequency Masking (p-0.5) enhances robustness to variations in spectral components by randomly obscuring a contiguous range of frequency bands in the spectrogram. This approach aims to simulate real-world PAM scenarios where certain frequency components might be masked due to environmental factors or recording anomalies. By introducing this variation during training, frequency masking helps the model to be less sensitive specific frequency absences, promoting better generalization across diverse acoustic conditions.
Time Masking (p=0.3) improves resilience temporal variations by obscuring segment of time in the spectrogram. This approach mimics interruptions or fluctuations in sound continuity that often occur in PAM settings, such as noises or overlapping sounds.
C.5 EVALUATION
We opt for threshold-free metrics to obtain a clear view of overall model performance without the necessity of fine-tuning thresholds for individual classes. This approach enhances comparability and minimizes biases toward particular applications. The metrics implemented are the following:
cmAP (class mean average precision) computes the AP (average precision) for each class e as an element independently and then averages these scores across all classes C, resulting in macro average (Kahl et al., 2023; Denton et al., 2022; Höchst et al., 2022).
cmAP = : Σ AP(c)
C
This metric reflects the model's 's ability to rank positive instances higher than negative ones across all possible threshold levels, offering comprehensive view of model performance without threshold calibration. However, it is noisy for species with sparse labels, limiting comparability across datasets (Denton et al., 2022).
Top-1 Accuracy evaluates whether the class with the highest predicted probability is (one of) the correct class for each instance:
T1-Acc 1
Σ 1 E Yi).
where N represents the total number of instances in the dataset, d. denotes the class with the highest predicted probability and Y; is the set of true labels. The indicator function [β E Yij outputs 1 the predicted class ÿs is one of the true labels in Y, and 0 otherwise. Although not a traditional multi-label metric, it offers an easily interpretable measure of generalization performance, particularly useful in practical scenarios where identifying a bird species is the primary concern.
AUROC (area under the receiver operating characteristic curve) computes the area under the receiver operating characteristic curve given model f:
AUROC(S) Ex Em (f(z)) < [(x1]]
(3)
[Do] D
where D is a set of negative and D positive examples ("Calders & Jaroszewicz, 2007). It summarizes the curve into one number, measuring the model's ability distinguish
27

---

Published as a conference paper at ICLR 2025
between classes across all thresholds. It is threshold thresholddindependderr independent and provides balanced performance view, especially in class imbalance cases with an expected value of 0.5 for each random ranking (Hamer et al., 2023).
In addition individually reporting the results for both training scenarios, compute the aggregate performance score averaged across the complete benchmark. We aim to facilitate a quick and comprehensive comparison between models.
C.6 VALIDATION
Due to the extensive empirical experiments and the volume of the training datasets required to produce our baseline results for our benchmark within the BirdSet collection, traditional hyperparameter optimization was not feasible. Additionally, the covariate shift between focals in training and soundscapes in testing complicates result validation. Utilizing a validation split from the training data is only partially functional, it does not accurately represent the soundscape data distribution. Furthermore, obtaining soundscapes from the test dataset for validation could introduce test data leakage due the static nature of soundscape recordings. Therefore, we follow (Hamer al., 2023) and employ the soundscape dataset POW for validation. However, this approach only allows us to validate the results of the and training scenarios since has dedicated classes that are not available in the validation dataset. In this case, we used validation split from the training data and employed augmentations.
Sead
Sool
Sead
0.40
Soad
Seed
0.34
Sead
0.38
0.12
BCT31 008
0.33
0.5
0.30
0:10
0.34
0.28
0.32
0.36
0.36
0.3
0.24
25
0
25
30
Chockpoints
Checkpuints
Chockpoinis
AST
2) EAT
c) ConNext
Figure 10: Validation loss in the scenario LT on POW with the models AST, EAT, and ConvNext.
We used the validation dataset to select the best- tess-pefoormng model based on the lowest loss across all training epochs POW. We show exemplary results of the validation loss curves for the training scenario LT Figure 10. illustrates that more complex models with extensive parameters (AST and ConvNext) tend overfit early the focal training data. contrast, the smaller model (EAT) benefits from extended training duration. More detailed results are available on Weights& Biases.
C.7 DETAILED BENCHIMRR RESULTS
We present detailed results for our three training scenarios in BirdSet. DT represents dedicated training on the respective subsets (see Table 9 and Figure 11). MT indicates medium training on approximately 400 species found test datasets (see Table 10 and Figure 12). LT denotes large training nearly 10,000 bird species from XC (see Table 11 and Figure 13).
28

---

Published as a conference paper at ICLR 2025
EdNe
Constiext
Eat
W2V2
NBP
SSW
NBP
SSW
HSN
SNE HSN
SNE
0.0
9.5
PER
UHH
NES
NES
a) T1 Tc-cuuuyyy
b) AUROC
Figure 11: Selected results for DT.
Table 9: Mean results and standard deviations across seeds in scenario dedicated training (DT).
POW
PER
NES
UHH
HSN
NBP
SSW
SNE
Score
emAP
0.39
0.19
0.33
0.24:
0.44
0.64:
0.35
0.28:
0.35
Edr.
AUROC
0.83# 0m1
0.72=001
0.87
0.80 6/3
0.85+
0.90=
0,91= 08
0.81 601
0.84
T1-Acc
0.70 0.02
0.40 0/0
0,47= 0.01
0,46:
0.59
0.66
0,54
0.65 6m1
0.54
cmAP
0.08
o02
Noxt
AUROC
0m
0.73 681
0,86s 0:01
0.78 0:01
0.88 9m22
0.92 0.0
0 0.
0.79 002
0.83
T1-Ace
0.67
0.45
0.43
0.69
0.57
0.52
cmAP
0.0
0.19=002
0.29: 0.01
0.15 001
0.30: o.0.
0.59
0.30 0.01
0.24 0.1
0.29
AST
AUROC
0.82s
0.72 001
0.88
0,78
0.83+
0,90 0m91
0.81
0.83
0.69
0.42=001
0.48 9:81
0:01
0.63
0.50= wee
0:1
cmAP
0.32 0.
0,14 001
0.29 0:01
0,
0.35 onz
0.
0,27= ner
0.23 001
0.33
AUROC
0.80
0.0
0.79:
0.85
0.76
T1-Acc
0m
0,37=000 001
0,45 om
0.37 001
0.43 om
0.65 om
0,48 no
0.55 o/2
0.47
emAP
0.24 0.1
0.15
0.34
0.54g
0,22
0.21
0.26
AUROC
0.75 0.02
= 0.00
0.83 0.0
0.73 0.0
0.79 o.02
0.88# ani
0.87= 0:0
0.77 0.00
0.79
T1-Acc
0.61#
0.30
0.40= 0.02
0,38s
0.36
0.64-s am
0.45
0.53 oo6
0.44
29

---

Published as a conference paper at ICLR 2025
NBP
SSW
NBP
SSW
HSN
SNE TISN
SNE
00
6.5
UHH
94
PER
UHH
0.0
PER
NES
NES
a) 11-Accuracy
AUROC
Figure 12: Selected results for MT.
Table 10: Mean results and standard deviations across 3 seeds in scenario medium training (MT).
POW
PER
NES
UHH
HSN
NBP
SSW
SNE
Score
cmAP
0.42 nor
0.18 nu
0.32: 6.1
0.26:1 oaz
0.48: ait
0.64 aM
0.36: 6.81
0.30: am
0.36
EHr.
Net
AUROC
0.85 0.00
0,70 000
0.89: 8.01
0.80 0012
0.88 o02
0.92 um0
0.92: 0.83
0.85
T1-Ace
0.79 0.
0.42 n.2
0.49: 8.01
0a5
0.69 not
0.61: 0.84
0.72: 0.33
0.59
emAP
0.40= na4
0,18 not
0.32 am
0.24 002
0.47 oot
0.64 003
0.36: 9.2
0,28
0,36
AUROC
0.0
0.2
0.86
0.84
T1-Ace
0.79 0.94
0.46 0.94
0.49: 8.01
0.41 0.99
0.46: o.8
0.66: 0.02
0.60: 0.04
0.68 0.04
0.57
0.33
0.29 0.15=um
0.38
0.61 0.30a
0.26:
0.31
AST
AUROC
0.82 6.00
0.69-nn n.
0.88
0.78# 081
0.84 am
0.91=
0.90 0.01
0.82 000
0.83
T1-Ace
0.75 a.
0.37=
0.46# 0.01
0.33
0.48 ons
0.66: ad8
0.57+
0.59 0.01
0.49
cmAP
n.r
0.09= 100
0.28
0.21a
0,40
0.54 M00
0.27 0:01
0.23 um
W2N2 EAT
AUROC
0,78 nor
0.60= m00
0.87 0:0
0.774
0.812
0.87 uon
0.86 0.00
0.79
0.80
T1-Ace
0.76
0.27=
0.01
0.37
0.63s
0.56a
0.48
cmAP
0.26a n12
0,10= uot
0.27 e.0
0.19 0m1
0.41
0.56w tos
0.26 0:11
0.2 0.0
0.29
AUROC
0.76
0.63=mm
0.84g 6.00
0.75
0.85 am
0.88
0.85a
0.77
0.80
T1-Ace
0.76 a.22
0.25 n.l
0.41 0.00
0.38 oas
0.40: 064
0.63 0.4
0.55# 0.02
0.59: 0.94
0.46
30

---

Published as a conference paper at ICLR 2025
EMNLL
Conveett
AST
...
Eat
W2V2
Pe
NBP
SNE HSN
SNE
0.0
05
UHH
UHH
9.9
NES
NES
TI --Accuracyy
b) AUROC
Figure 13: Selected results for LTT
Table 11: Mean results and standard deviations across 5 seeds scenario large training (LT).
POW
PER
NES
UHH
HSN
NBP
SSW
SNE
Score
cmAP
0.35 nm
0.1
0.30 oor
0.23 oor
0.35
0.57 0:6
0.33 nos
0.28: om
0.32
EIT.
Net
AUROC
0.82 om
0.88# 0.71=
0.78 o.
0.86
0.90=
0.91 002
0.84
T1-Acc
0.80 003
0.38 001
0.49: 001
0.424 om0
0.59 0.65
0.63 0.04
0.55 002
0.67: 003
0.53
emAP
0.36 nm
0.19 nai
0.34+ 001
0.26+ 0.01
0.47 0.82
0.62= 0.02
0.35 0.1
0.30: am
0.36
Next
AUROC
0.824 002
0.72 0.1
001
0.79. 6.2
0.01
0.92. 9.11
0.93 0.00
0.83. 6m1
0.85
T1-Acc
0.75 nos
0.36 0.7
0.45 001
0.44 o.
0.52
0.64
0.532 0o8
0.65:
0.51
cmAP
0.33 0.03
0.18 0.01
0.32# 0.0
0.21 0.1
0.44 0.02
0.61 0.02
0.33 001
0.28 001
0.34
AST
AUROC
0.82: 0.00
0.72 m01
0.89 nm
0.75+t 0.01
0.85 0.00
0.91=
0.93 0.1
0.82 001
0.84
T1-Ace
0.79#
0.02
0.48#
0.39
0.48
0.66=
0.51 0.01
0.57 0.4
0.49
cmAP
0.27 0.0
0.12 0.00
0.27 0.00
0.22+ 0.00
0.38 0.01
0.50= 0.00
0.25 0.00
0.24s 0.00
0.30
EAT
AUROC
0.79 0.01
0.64 n.1
0.87 0.03
0.76 0.01
0.86 0.01
0.87 0.00
0.90s 0.00
0.82 0.00
0.82
T1-Acc
0.69 0.02
0.32 0.01
0.46 0.02
0.40 0.02
0.47 0.02
0.61 0.01
0.46 0.2
0.58: 002
0.48
cmAP
0.27
0,14
0.30a
0.21
0.40 0.00
0.57
0.29
0.25
0.31
W2V2
AUROC
0.75 0.1
0,68 0.00
0.86. 001
0.76.1 oo1
0.86 0.00
0.90 0.01
0.90s 000
0,78 onx
0.78
T1-Acc
0.72: 8.02
0.34 0.66
0.47+ 0.01
0.51
0.50 0.05
0.65 0.06
0.50s 0.00
0,51 0.3
0.50
cmAP
0.30
0.18
0.39
0.27
0.45
0.63
0.28
0.29
0.36
Perch
AUROC
0.84
0.70
0.90
0.76
0.86
0.91
0.91
0.83
0.84
T1-Acc
0.85
0.48
0.66
0.57
0.58
0.69
0.62
0.69
0.61
31

---

Published as a conference paper at ICLR 2025
D INFERENCE RESULTS
In this section, we exemplify inference results on HSN and POW for our ConvNext model and Perch, both pretrained on XC, in a threshold-dependent setting. We adopt the same configuration as BirdNET (Kahl et al., 2021b), utilizing a global prediction threshold 0.1.
Convnext
8o call (6023)
whappa (2876)
amepip (1882)
roewre (735)
gerinn (671)
olanut (320)
borthr (266)
dagjun (175)
mouchi amerob (126)
foxqua (2))
aonubl (27)
sponan (22)
parwar (18)
caulfn growert (7)
(1
(
mallard (3)
duafty (3)
yalwar (1)
warvit
brabla
TPR
0.21
0.02
0.01 0.37 0.38
0.26
0.07
0.19
0.26
0.68
0.59
0.43
0.67 0.00 0.00 0.00:
FNR
0.00
0.79
0.95 0.98
0.99 0.63 0.62
0.74
0.93
0.81
1.00
0.74
0.32
1.00
0.41
1.00 0.43
0.57 0.33 1.00 1.00 1.00
TNR
0.19
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
FPR
0.81
.000
0.00 0.00 0.00 0.00 0.00 0.00
0.00
0.00 0.00 0.00
0.00 0.00 0.00
0.00 0.00 0.00
Perch
N6 call (6623)
whaspa (2570)
ateeppp (1082)
rocwre (733)
gertin (671)
olanut (320)
Northr (266)
daejun (175)
umerab (226)
moncch (82)
fozapa (28)
monblu (77
oposan (22)
yorwar (1m)
canlin (17)
orewer (?)
noztll (7)
warvir (7)
mallay3 (3)
duafty (3)
yalwar (1)
brahla d
TPR
0.28
0.09
0.12
0.44
0.45 0.36
0.26
0.56
0.45
0.00
0.71
1.00
0.00
0.00
0.010.01
FNR
0.01
0.72
0.85
0.91 0.88 0.75 0.56
0.55
0.64 0.74
1.00 0.44
0.55
0.83
0.59 1.00
0.43
0.29
0.00 1.00
1.00 1.00
TNR 0.29
1.00
1.00
1.00
1.00
1.00 1.00
1.00
1
1.00
1.00 1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00 1.00
FPR
0.00
0.00
0.00
0.00
0.00
Figure 14: HSN Classification performance in LT showing True Positive Rate (TPR), False Negative Rate (FNR), True Negative Rate (TNR), and False Positive Rate (FPR). The special case of no call where the segment has annotation and the model predicts O-vector. The X-axis label shows each class's occurrences (not necessarily equal to the #Annotations).
Convnext
(1ary
Ca
(T3)
(280
(355)
(3))
(31)
coloro (205
veltone (14)
(9)
(2)
(2)
(2)
7. 1.30...0.0.
0.77659.7
001.01011 000001000110
TNU 2101810 1000000101010
03
01.0010 1000011
9. 0200009000 0.0 0.0.0.0.0000000 00.00
0.0.0.0 0.0
0.00
9.0
9.9
0.0
3.0
0.0
0.0
9.0
0.0
0.0 0.0
9.0
0.00.1 9.0 0.0 0.00.00.1
0.000
Perch
madoon caves
aerdrri (1:20)
seero por (190)
ortte (878))
(aut))
2300
(oe5)
Rleper nig (212)
tap
1446
Seleere (10)
(90)
(82)
(81) (21
colerro CCT
Mangro 120g (27)
vndiove (1e)
amoirer 40
Rhawarr vecopno CD
seuignoe Ct
pullews (P)
boaale Ch
Datini G2
pttreen (2)
loarr (2)
cvrtevee (1)
(2)
20000
PN R8 48559.90000 0.0.0.0......
0.3.00 0. 0.0.000.00 0.....0...00 0.0 0.9.800
9.00.8 0.9
0.3
1.0.0.cc 3.
100.000 67777
10100.010 101000
TNR
FBA 208800R0 5 88.0 06. 00 D. 58.0000000000 8.0. 8. 20 83.893833
8............ 9.0. 0...0 8 D. 8 35 5...800.0000 5 3 8.00 000085.00000 3.0.00000000 0ww.080000000000000000000 00.0883000
Figure 15: POW Classification performance in LT showing True Positive Rate (TPR), False Negative Rate (FNR), True Negative Rate (TNR), and False Positive Rate (FPR). The special case of no call where the segment has no annotation and the model predicts 0-vector. The X-axis label shows each class's occurrences (not necessarily equal to the #Annotations).
32

---

Published as a conference paper at ICLR 2025
E DATASHEET
This section provides a comprehensive and structured datasheet BirdSet, following the guidelines outlined by Gebru et al. (2021). We aim to enhance clarity and facilitate effective communication between and researchers who wish utilize the dataset collection. The datasheet comprises the dataset's 's motivation, composition, collection process, preprocessing steps, uses, distribution, and maintenance.
E.1 MOTIVATION
For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide description.
The BirdSet dataset collection was created address the demand for large-scale audio represen- tation learning dataset, including real-world evaluation test set. In addition, Bi rdSet serves as standardized dataset and evaluation protocol in computational avian bioacoustics. The specific task is the classification of bird vocalizations in realistic passive acoustic monitoring (PAM) scenario. The goal was to consolidate fragmented research efforts, improve reproducibility, comparability, and accessibility, and ultimately enhance the effectiveness of deep learning (DL) models in analyzing bird vocalizations.
Who ereated the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? If there an associated grant, please provide the name of the grantor and the grant name and number.
The dataset collection was created by a collaborative effort of researchers from the University of Kassel at the Intelligent Embedded Systems group (IES), Kiel University at the Intelligent Systems group (INS), Fraunhofer Institute for Energy Economics and Energy System Technology (IEE), and TU Chemnitz. Throughout the creation of the dataset, the following people have been involved: Lukas Rauch (IES), Raphael Schwinger (CAU), Moritz Wirth (IEE), and René Heinrich (IEE).
Who funded the creation of the dataset? The research was funded by the German Ministry for Environment, Nature Conservation, Nuclear Safety, and Consumer Protection through the project "DeeetirdDetect- Automatic Bird Detection Endangered Species Using Deep Neural Networks" (grant numbers: 67K13 1040A, 67K131040B, 67K13 1040C).
Any other comments? A more detailed motivation and corresponding objectives can be found the main article.
E.2 COMPOSITION
What do the instances that comprise the dataset represent (e.g,, documents, photos, people, countries)? Are there multiple types of instances (e-g.. movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide description.
The instances the dataset collection are comprised of audio recordings of bird vocalizations in their natural habitats. These include focal recordings from Xeno-Canto (XC), actively recorded by recordist for training purposes, and soundscape recordings captured by omnidirectional microphones and annotated by experts for testing. For more detailed information on the unified metadata provided for each instance, refer to Table 5.
How many instances are there in total (of each type, if appropriate)?
The dataset contains a total of 712,515 instances. includes over 520,000 focal recordings from various locations worldwide and over 400 hours of soundscape recordings from passive acoustic monitoring (PAM) scenarios in South America, North America, and Europe. For more details, refer to Figure 7, Figure Figure 5, and the main article.
Does the dataset contain all possible instances, or a sample (not necessarily random) of instances from a larger set? If the dataset is sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verifted. If it is not representative of the larger set, please
33

---

Published as a conference paper at ICLR 2025
describe why not (e.g., to cover more diverse range of instances because instances were withheld or unavailable)
The dataset is a curated collection of publicly available recordings, primarily sourced from the platforms XC (Vellinga & Planqué, 2015) and Zenodo (CERN), We included all available recordings from XC as of 03/10/2022 in the XCL dataset, excluding those with an ND license. Additionally, omitted recordings of highly endangered bird species. The XCM dataset, a subset XCL, contains all bird species present across the test dataset. We also provide dedicated training datasets containing only the species represented in each test dataset. Selected test datasets from Zenodo are incorporated as available. Further details can be found in the main article.
What data does each instance consist of? "Raw" data (e.g. umprocessed text or images) or features? In either case, please provide a description
An instance consists of a raw audio recording with resolution of 32 kHz in the .OGG format and respective metadata detailed in Table 5. We also provide a list of possible vocalization events for each training recording.
Is there label or target associated with each instance? If so. please provide description.
Each instance is labeled with the eBird taxonomy (Sullivan et al., 2009) from 2021 corresponding to the bird species in the recording.
Is any information missing from individual instances? If so, please provide description, ex- plaining why this information missing (e.g.. because it was unavailable). This does not include intentionally removed information but might include, e.g,, reducied text.
No information from the source datasets has been removed. However, some recordings in the training datasets have incomplete metadata and annotations. While secondary bird vocalizations are sometimes annotated, they are not always identified. Additionally, it may not always be clear which events belong to the primary bird species or secondary background species. More information on label uncertainty is provided in the main article.
Are relationships between individual instances made explicit (e.g,, users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
Relationships between instances are not explicitly defined, each recording is treated independently for the classification tasks. However, the metadata includes details such as the recordist, location, and recording device, allowing for identifying similar instances. The soundscape test datasets recorded in static environments where segments have inherent relationships that are valuable for real-world applications. These relationships are not considered in our benchmark.
Are there recommended data splits (e.g., training, development/validation testing)? If so, please provide description of these splits, explaining the rationale behind them.
The dataset collection is split into training and testing datasets. Training datasets (XCL, XCM) are used train large-scale models, while soundscape recordings (e.g., PER, NES, UHH) are used for testing. This split reflects practical PAM scenarios, where models are trained on broad dataset and evaluated on realistic, strongly-labed labeled soundscapes. Additionally, we provide a validation dataset (POW) for tuning hyperparameters or validating model results. Additionally, we provide small fine-tuning training datasets for each test dataset with randomly generated validation split during training.
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide description.
Potential noise sources in the recordings include background sounds inherent to the natural envi- ronments where the data were collected. These are not typical errors but reflect the real-world conditions under which bird vocalizations occur. Additionally, the training recordings from XC are weakly labeled, meaning the exact timing of the vocalization events within the recordings is uncertain. Further details on label uncertainty are provided in the main paper.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the
34

---

Published as a conference paper at ICLR 2025
complete dataset (i.e,, including the external resources they existed at the time the dataset was created); c) are there any restrictions (e.g. licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
The dataset is self-contained,, with internal backup ensuring its permanence and consistency time. Given the dynamic nature of XC and the availability of additional data, the training datasets are designed to be expandable. Moreover, the metadata for each instance includes links the original sources, facilitating easy access and verification.
Does the dataset contain data that might be considered confidential (e.g., data that is pro- tected by legal privilege or by doctor pacten- confidentiality, data that includes the content of individuals' non-public communications)? so, please provide a description.
All metadata adheres to the licensing terms of the respective recordings on XC (including the recordists' names) and Zenodo.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
No, the dataset collection only contains audio recordings of bird vocalizations, which are not offensive or threatening.
Does the dataset identify any subpopulations (e.g,, by age, gender) so, please describe how these subpopulations are identified and provide description of their respective distributions within the dataset.
The dataset does not identify subpopulations based on demographic factors; it focuses solely on bird species.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how
The metadata includes the recordist's name and the recording location, which may indirectly provide information about the recordist. The respective licenses of the XC recordings also include these details.
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide description.
No, the dataset does not contain sensitive data.
Any other comments?
No further comments.
E.3 COLLECTION PROCESS
How was the data associated with each instance acquired? Was the data directly observable (e.g.. raw text, movie ratings), reported by subjects (e.g., survey responses), indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If the data was reported by subjects indirectly inferred/derived from other data, was the data velidated/verffed? If so. please describe how.
The data was downloaded from the XC and Zenodo platforms. Download and preprocessing scripts are available in the GitHub repository.
What mechanisms procedures used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? How were these mechanisms or procedures validated?
We collected the data via the XC API and Zenodo. The recordings were captured using equipment suitable for recording bird vocalizations. The recordings manually curated and annotated by
35

---

Published as a conference paper at ICLR 2025
experts enthusiasts. For further details, see the descriptions of the individual datasets listed in Section B.
If the dataset is sample from larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabiliries)?
The dataset is curated collection from larger collections, specifically from the open-source platforms XC and Zenodo. We exclude all recordings from XC that are licensed as CC-ND and sample based on training scenarios; the snapshot date is 03/10/2024. The selected datasets from Zenodo are utilized as provided. See section B and the main article for further information.
Who involved in the data collection process (e.g., students, crowd workers, contractors) and how were they compensated (e.g., how much were crowd workers paid)?
The data collection involved researchers from several institutions, including the University of Kassel, Kiel University, Fraunhofer Institute, and TU Chemnitz. The researchers were compensated according their contracts. The annotators of the focal recordings freely uploaded their recordings and annotations to XC under respective licenses.
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of new news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
The recordings were collected over several years on XC, with the date of each recording included in the metadata. A snapshot from XC was taken on 03/10/2024.
Did you collect the data from the individuals in question directly or obtain it via third parties or other sources (e.g., websites)?
The data was obtained from third-party sources, primarily the open-source platforms XC for training and Zenodo for testing datasets.
Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, otherwise reproduce, the exact language of the notification itself.
The dataset collection includes only recordings that are properly licensed for use. All recordings are under creative commons (CC) license, summarized by our collection with the license CC-BY-NC- SA.
Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
We included only recordings with proper CC license that permits usage.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future for certain uses? so. please provide description, well as a link or other access point to the mechanism (if appropriate).
See above.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? so, please provide description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.
No formal analysis has been conducted. However, all the data has already been publicly available before.
Any other comments?
No further comments.
36

---

Published as a conference paper at ICLR 2025
E.4 P
Was any prepocesiingcleenigg/bbiiinn of the data done (e.g,, diseretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If noi, you may skip the remaining questions in this section.
The audio recordings have been converted to the .OGG format and standardized to a 32kHz resolution. Metadata has been standardized, and scientific bird species names have been unified eBird (Sullivan et al., 2009) codes as labels. Refer to Table 5 for a comprehensive overview of the metadata. Additionally, we identified potential bird vocalization events from XC using the bambird (Michaud et al., 2023) event detection algorithm and scipy peak detection. The soundscape recordings have also been processed and are provided in 5-second segments for evaluation. For more details, refer to Hugging Face Dataset collection.
Was the "raw" " data saved in addition to the preproeesee/cdeaneehlaaleee data (e-g., to support unanticipated future uses)? If so, please provide link or other access point to the "raw" data.
The raw recordings remain accessible at their original sources. We also maintain an internal server backup of all data for added security and reliability.
the software that was used preprocess/dean/label the data available? If so, please provide a link other access point.
Code used for processing is available at our GitHub repository. Additionally, we detail the utilized software assets above.
Any other comments?
No further comments.
E.5 USES
Has the dataset been used for any tasks already? so, please provide description.
Yes, the dataset has been used in multi-label experiments in PAM scenario with different training protocols outlined in the main paper. Additionally, the dataset has been used for other internal experiments in the context of the research project "DeepBirdDetect".
there a repository that links to any or all papers or systems that use the dataset? If so, please provide link other access point.
To date, no papers have utilized this dataset.
What (other) tasks could the dataset be used for?
The Birdstt dataset collection can be used for various tasks beyond multi-label classification, including but not limited to:
Audio representation learning,
Audio event detection,
Noisy label learning,
Few-Shot Learning,
Domain adaption under covariate shift,
Active learning.
Conducting conservation research by monitoring bird populations and behaviors.
Is there anything about the composition the dataset or the way was collected and pre- processed/cleanec//bbeeee that might impact future uses? For example, is there anything that dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) other risks or harms (e.g.. legal risks, financial harms)? If so, please provide description. Is there anything a dataset consumer could do mitigate these risks harms?
37

---

Published as a conference paper at ICLR 2025
The dataset is composed of audio recordings of bird vocalizations and related metadata. Since it does not involve human subjects, there are no risks of unfair treatment of individuals or groups. However, users should be aware of the inherent variability in recording conditions (e.g., background noise, distance from the sound source) and the potential for label noise due to the challenges in accurately annotating bird calls in natural environments.
Are there tasks for which the dataset should not be used? If so, please provide a description. The dataset should not be used train models that can identify critically endangered bird species.
Any other comments? No further comments.
E.6 DISTRIBUTION
Will the dataset distributed to third parties outside of the entity (e.g., company, institution, organization) behalf of which the dataset was created? If so, please provide description.
Yes, see the next question.
How will the dataset be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
The dataset is available on Hugging Face with respective DOL Additionally, the metadata, including links to the sources, is available in our GitHub repository, allowing for dataset construction without using Hugging Face.
When will the dataset be distributed?
It is already public.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
The dataset collection BirdSet is available under the CC-BY-NC-SA license. Each recording in the training datasets retains the corresponding license from XC. We excluded all recordings from XC under CC-ND license. All soundscape recordings from the different datasets collected from Zenodo are licensed under CC-BY-4.0.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
No third parties have imposed IP-based or other restrictions on the data. The dataset is sourced from publicly accessible platforms that support open-access principles.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any supporting documentation.
No export controls or other regulatory restrictions apply to the dataset or its instances.
Any other comments?
No further comments.
E.7 MAINTENANCE
How can the ovneer/curator/manager of the dataset be contacted (e.g,, email address)?
The dataset can managed and inquiries addressed through the GitHub repository or by contacting the main authors involved in its creation via their institutional email addresses provided below:
Lukas Rauch: rauch @uni-kassel.de
38

---

Published as a conference paper at ICLR 2025
Raphael Schwinger: rsc@ me@@iformatik.umm uni-kiel.de
Moritz Wirth: moritz. worttz.wirtthe iee. ree.fraunhofrr.de
René Heinrich: rene.heinriche @ iee. fraunhoffer.de
there an erratum? If so, please provide link or other access point.
No erratum has been discovered so far.
Will the dataset updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to dataset consumers (e.g., mailing list, GitHub)?
Updates the dataset have not yet been planned to ensure the comparability of the benchmark. However, advancements in areas such as event detection could lead the addition vocalization events.
the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
The data does not directly relate to people.
Will older versions of the dataset continue to be supportee/nuste/naiittined? so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.
Dataset versions provided through Hugging Face.
If others want to extend/augmment/uuil on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide description. Will these contributions be validated/verifee?? so, please describe how. If not, why not? Is there a process for conmunicaainggdissriiibtt these contributions dataset consumers? If so, please provide description.
Yes, contributions from the community are welcome. Interested parties can submit their contributions via pull requests the GitHub Hugging Face repository. The dataset maintainers will review and validate all contributions before being accepted and integrated. Contributions and updates will be communicated users through the repository's 's update log.
Any other comments?
No further comments.
39


Reviews:
[{"review_id": "lx94uvhC03", "reviewer": "Reviewer_wpw3", "summary": "This is an excellent,  comprehensive and carefully curated dataset of bird sounds. The paper also provides an excellent review of the existing literature, and provides baseline code showing use of the data. The benchmark testing tasks are a bit simple, but demonstrate the breadth of the dataset.", "questions": "Please rewrite with clarity and minimizing acronyms.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Careful curation of the dataset - types of collections, varying types of birdcalls from same species; includes both soundscapes, and individual recordings; comparisons with other datasets.\nInclusion of benchmark code for future researchers to use as a baseline.\nPermissive licensing\nA large scale project", "weaknesses": "too many uncommon acronyms which require a reader to keep going back and forth - such shortening of the length was unnecessary. The acronyms are used in the figures as well. Figures and captions are supposed to stand on their own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFlwhA702T", "reviewer": "Reviewer_uitS", "summary": "This paper presents BirdSet, a new large-scale benchmark dataset specifically for multi-label audio classification within avian bioacoustics. It significantly extends the scope of existing audio datasets by including approximately 10,000 classes, covering over 6,800 hours of training data, and incorporating 400 hours across eight distinct test datasets. BirdSet addresses several real-world machine learning challenges, such as covariate shift, label noise, and task shift, providing a unique resource for evaluating model robustness in audio classification under diverse conditions. The benchmark includes evaluations of six prominent deep learning models across three different training approaches: training on the full BirdSet, training on a subset containing only classes relevant to the downstream tasks, and training on a small subset for each downstream dataset individually.\n\nAdditionally, the authors facilitate accessibility by hosting BirdSet on Hugging Face, where they provide Python code to load the data. They also offer scripts for reproducing the experiments in the paper​.", "questions": "I see significant potential in BirdSet, and I wonder if it would be feasible to define few-shot tasks within the downstream tasks to address few-shot multi-label classification?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Originality\n\nBirdSet represents a novel contribution to multi-label audio classification. With close to 10,000 classes BirdSet provides a benchmark to develop scalable methods capable of handling extreme class diversity with large imbalance. BirdSet also addresses critical machine learning challenges such as covariate shift, where the testing distribution diverges from the training distribution reflecting real-world environmental shifts in field data.\n\n\n2. Clarity\n\nThe paper is clear and well-organized, with a logical presentation of BirdSet’s structure, design choices, and challenges. The descriptions of covariate and domain shifts, as well as the evaluation protocols and training setups, are concise and well-articulated.\n\n\n3. Significance\n\nBirdSet can enable the development of new self-supervised learning, active learning, and few-shot learning approaches that are resilient to covariate and domain shifts—capabilities that are increasingly relevant for practical deployment of models in real-world applications. Moreover, its emphasis on multi-label classification reflects real-world scenarios in bioacoustics and similar domains, driving advancements that can translate to other fields beyond audio.\n\n\nIn summary, BirdSet is highly original, well-curated, and impactful dataset that serves as both a resource and a benchmark. It encourages tackling challenges related to robustness to distribution shifts, and multi-label classification, with methods that can be developed that extend across deep learning domains more broadly.", "weaknesses": "I didn’t find any weaknesses in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uOzvYCEzB1", "reviewer": "Reviewer_5AN6", "summary": "The paper proposes BirdSet, which is a dataset, or more specifically, a collection of 11 different sub-datasets. BirdSet attempts to unify \navian bioacoustic evaluation on focal and soundscape recordings under one roof, providing a large, unified, accessible test bed for testing audio classification approaches.", "questions": "In the context of how the paper is currently phrased, more direct comparisons, for instance, linear evaluation on cross-domain data between models trained on BirdSet and AudioSet on a variety of downstream tasks, spanning bioacoustics and other audio domains would be needed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Well written and easy to read.\n2. Thoroughly explains the challenges experienced not only in avian bioacoustics, such as covariate shift and mismatch in focal and soundscape recordings, but also in curating, and developing a dataset of such a size and scale. \n3. The pain points addressed in the paper are very real: poor availability and accessibility of AudioSet, lack of a unified benchmark suite for evaluating segment and event-based bioacoustics tasks, and mismatch between train and test time for avian bioacoustics, are all pressing challenges. \n4. A good variety of baseline models were evaluated.", "weaknesses": "To me, the paper, in several places, tries to pose BirdSet as a replacement for AudioSet and that it should be the exemplary benchmark for evaluating audio classification models, with statements such as \"Avian bioacoustics exemplifies challenges in audio classification...\", and how curated datasets like AudioSet and ESC-50 do not represent real-world complexities. Pain points mentioned w.r.t. AudioSet are all very real, but AudioSet is a much broader dataset than BirdSet. Several people, in industry and academia have found AudioSet useful: a blanket statement saying AudioSet is not useful in real-world scenarios is plain wrong. \nSome statements straight up downplay AudioSet: for instance, line 37-38: where AudioSet poses \"... concerns regarding transferability to real-world environmental domains\". Whereas the paper cited simply shows that AudioSet pretrained models perform \"well enough\": of course models trained on in-domain bioacoustics data will fare better for bioacoustic evaluation!\n\nAlso, as per Table 3, AST and EAT models, which are pretrained on AudioSet, seem to perform quite well in the DT setting (cross-domain transfer setting) versus their LT and MT counterparts. For EAT, DT performance matches MT and outperforms LT scenarios. \n\nBirdSet will be a tremendous contribution to the fields of audio classification and avian bioacoustics, I have no doubt, but I think the current language of the paper comes off as \"posturing\" a tad bit too much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "No3uxqTJfu", "reviewer": "Reviewer_qFwz", "summary": "The paper introduces BirdSet, a large-scale audio classification dataset for avian bioacoustics, with around 520,000 recordings from nearly 10,000 bird species. It includes over 6,800 hours of training data and 400 hours of evaluation data from diverse regions. The dataset supports tasks such as multi-label classification and self-supervised learning, with standardized training and evaluation protocols. A comprehensive literature review identifies key challenges in bioacoustics and provides research guidelines. The paper benchmarks multiple deep learning models and offers a codebase for reproducibility.", "questions": "Please refer to the weaknesses above for the questions.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The introduction of *BirdSet* fills a notable gap in audio classification by providing a large-scale, domain-specific dataset for avian bioacoustics.\n2. The paper is well-structured and clearly articulates the challenges in avian bioacoustics and audio classification more broadly. \n3. The paper provides a thorough empirical evaluation using six well-known deep learning models, covering various training scenarios, including large-scale training and fine-tuning.", "weaknesses": "1. The literature review in Section 2 is quite extensive, occupying a significant portion of the main paper. While the analysis is comprehensive, it might be more beneficial to allocate more space to the dataset description and details, rather than using five pages for the related work.\n2. The results focus mainly on the multi-label classification benchmark, with limited exploration of other use cases. Given the dataset’s availability of precise event timestamps, it would be valuable to include benchmarks for sound event detection. Additionally, providing results for other mentioned use cases would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)