$\texttt{BirdSet}$: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics

## Abstract

Abstract Deep learning (DL) models have emerged as a powerful tool in avian bioacoustics to diagnose environmental health and biodiversity. However, inconsistencies in research pose notable challenges hindering progress in this domain. Reliable DL models need to analyze bird calls flexibly across various species and environments to fully harness the potential of bioacoustics in a cost-effective passive acoustic monitoring scenario. Data fragmentation and opacity across studies complicate a comprehensive evaluation of general model performance. To overcome these challenges, we present the BirdSet benchmark, a unified framework consolidating research efforts with a holistic approach for classifying bird vocalizations in avian bioacoustics. BirdSet harmonizes open-source bird recordings into a curated dataset collection. This unified approach provides an in-depth understanding of model performance and identifies potential shortcomings across different tasks. By establishing baseline results of current models, BirdSet aims to facilitate comparability, guide subsequent data collection, and increase accessibility for newcomers to avian bioacoustics.

## 1 Introduction

Avian diversity is a crucial indicator of environmental health, as fluctuations in bird populations can reflect broader shifts in biodiversity (Sekercioglu et al., ). Additionally, as climate change accelerates the development of wind farms, the potential risks to avian species become more prevalent. Consequently, there has been a greater emphasis on avian monitoring through bioacoustics as passive acoustic monitoring (PAM) enables a cost-effective collection of extensive sound data with minimal disruption to natural habitats (Shonfield and Bayne, ). Traditionally, PAM requires human experts (i.e., ornithologists) to meticulously analyze vast quantities of audio recordings.
Recent advancements in deep learning (DL) have significantly reduced the workload of experts in avian bioacoustics, catalyzing the emergence of deep avian bioacoustics (Stowell, ; Kahl et al., ). The most prominent DL task in a PAM scenario is sound event detection (SED), focusing on identifying and classifying bird vocalizations in medium to long-range recordings. In an event or segment-based evaluation scheme, the task fundamentally translates into multi-class or multi-label classification (van Merriënboer et al., ).

Figure: Figure 1: Proposed training and evaluation pipeline with a diverse collection of training resources and a static range of test datasets.
Refer to caption: /html/2403.10380/assets/x1.png

However, deep avian bioacoustic research encounters several challenges, including the absence of a standardized evaluation protocol, leading to a fragmented research landscape. One contributing factor is the varying recording methods used interchangeably for training and testing. Recordings range from active focal recordings targeting specific bird species to passive soundscape recordings encompassing additional ambient sounds. Despite an abundance of open-source data available on platforms such as Xeno-Canto (XC) (^1^11https://xeno-canto.org/)(Vellinga and Planqué, ), there is a lack of consistent dataset and task selection (e.g., multi-class vs. multi-label) posing barriers to reproducibility, comparability and accessibility (Rauch et al., ). A complex preprocessing procedure further intensifies this issue as DL models in avian bioacoustics primarily operate on spectrograms requiring a manual conversion from raw audio to images (Stowell, ). Additionally, focal recordings from XC are weakly with varied lengths lacking precise event timings. This impedes progress towards reliable avian monitoring models that can be employed in practical PAM scenarios.

To address existing challenges and support future research, we introduce the BirdSet benchmark. Inspired by approaches in other fields (Wang et al., ; Rauch et al., ), BirdSet provides a comprehensive and standardized platform for classifying bird vocalizations in deep avian bioacoustics. BirdSet incorporates a diverse collection of datasets (c.f., [Figure 1](#S1.F1)) and serves as an overall model performance diagnostic tool, identifying specific weaknesses across different scenarios. We support BirdSet’s utility with a detailed experimental framework, establishing baselines for numerous model architectures. Our benchmark includes vision-based models that are typically employed in bioacoustics, but also extends to transformer-based models that are applied directly to raw audio (Rauch et al., ). BirdSet aims to serve as a catalyst for systematic and comprehensive research in avian bioacoustics. We outline our primary contributions below:

- 1.
Through an extensive literature analysis, we reveal current challenges and evaluation practices in deep avian bioacoustics. We embed the challenges into a comprehensive framework that establishes a foundation for future research.
- 2.
We introduce the BirdSet benchmark encompassing an extensive collection of avian bioacoustics datasets in a PAM scenario for multi-class and multi-label classification. We offer a consistent set of test datasets and a diverse array of training resources, enabling the application of diverse training approaches. Leveraging the Huggingface Datasets framework (Lhoest et al., ), we establish a standardized evaluation protocol to facilitate streamlined analysis(^2^22https://huggingface.co/datasets/DBD-research-group/BirdSet). Additionally, we provide a dataset creation tool to foster the development of community-contributed datasets.
- 3.
We conduct an extensive empirical analysis establishing baseline performances for the BirdSet benchmark. This analysis aids future research by offering best practices and methodological guidelines.
- 4.
We provide a comprehensive codebase(^3^33https://github.com/DBD-research-group/BirdSet) to support subsequent research efforts. While our implementations are based on Huggingface Datasets and Models to ensure reusability, we offer flexible utilization. Additionally, we ensure transparency by making all experimental outcomes accessible via Weights and Biases (Biewald, ).

## 2 Current Challenges and Evaluation Practice

Avian bioacoustics is characterized by the distinctive features of bird vocalizations and their ecological context. In a practical PAM scenario, rigorous model validation across various tasks is crucial to ensure reliability (Stowell, ). DL Models must demonstrate proficiency in tasks involving uncertainty, generalization, and adaptability (Tran et al., ). However, the field currently lacks a standardized evaluation practice for classification with DL models. We observe ambiguous methodologies across publications, which hinder reproducibility and comparability (e.g., multi-class vs. multi-label classification). To address these challenges, we embed them into a comprehensive framework and emphasize the importance of practical model deployment in PAM (Kahl et al., ; Denton et al., ; Hamer et al., ). We provide an overview of the current state-of-the-art challenges and briefly describe our strategies to navigate these obstacles. We pinpoint four pivotal challenges: (C1) Dataset Selection, (C2) Model Reliability, (C3) Model Training, and (C4) Evaluation. We aim to cultivate a more cohesive and collaborative research landscape in deep avian bioacoustics.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Please rewrite with clarity and minimizing acronyms.
soundness: 4
strengths: Careful curation of the dataset - types of collections, varying types of birdcalls from same species; includes both soundscapes, and individual recordings; comparisons with other datasets.
Inclusion of benchmark code for future researchers to use as a baseline.
Permissive licensing
A large scale project
summary: This is an excellent,  comprehensive and carefully curated dataset of bird sounds. The paper also provides an excellent review of the existing literature, and provides baseline code showing use of the data. The benchmark testing tasks are a bit simple, but demonstrate the breadth of the dataset.
weaknesses: too many uncommon acronyms which require a reader to keep going back and forth - such shortening of the length was unnecessary. The acronyms are used in the figures as well. Figures and captions are supposed to stand on their own.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: I see significant potential in BirdSet, and I wonder if it would be feasible to define few-shot tasks within the downstream tasks to address few-shot multi-label classification?
soundness: 4
strengths: 1. Originality

BirdSet represents a novel contribution to multi-label audio classification. With close to 10,000 classes BirdSet provides a benchmark to develop scalable methods capable of handling extreme class diversity with large imbalance. BirdSet also addresses critical machine learning challenges such as covariate shift, where the testing distribution diverges from the training distribution reflecting real-world environmental shifts in field data.


2. Clarity

The paper is clear and well-organized, with a logical presentation of BirdSet’s structure, design choices, and challenges. The descriptions of covariate and domain shifts, as well as the evaluation protocols and training setups, are concise and well-articulated.


3. Significance

BirdSet can enable the development of new self-supervised learning, active learning, and few-shot learning approaches that are resilient to covariate and domain shifts—capabilities that are increasingly relevant for practical deployment of models in real-world applications. Moreover, its emphasis on multi-label classification reflects real-world scenarios in bioacoustics and similar domains, driving advancements that can translate to other fields beyond audio.


In summary, BirdSet is highly original, well-curated, and impactful dataset that serves as both a resource and a benchmark. It encourages tackling challenges related to robustness to distribution shifts, and multi-label classification, with methods that can be developed that extend across deep learning domains more broadly.
summary: This paper presents BirdSet, a new large-scale benchmark dataset specifically for multi-label audio classification within avian bioacoustics. It significantly extends the scope of existing audio datasets by including approximately 10,000 classes, covering over 6,800 hours of training data, and incorporating 400 hours across eight distinct test datasets. BirdSet addresses several real-world machine learning challenges, such as covariate shift, label noise, and task shift, providing a unique resource for evaluating model robustness in audio classification under diverse conditions. The benchmark includes evaluations of six prominent deep learning models across three different training approaches: training on the full BirdSet, training on a subset containing only classes relevant to the downstream tasks, and training on a small subset for each downstream dataset individually.

Additionally, the authors facilitate accessibility by hosting BirdSet on Hugging Face, where they provide Python code to load the data. They also offer scripts for reproducing the experiments in the paper​.
weaknesses: I didn’t find any weaknesses in this paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: In the context of how the paper is currently phrased, more direct comparisons, for instance, linear evaluation on cross-domain data between models trained on BirdSet and AudioSet on a variety of downstream tasks, spanning bioacoustics and other audio domains would be needed.
soundness: 2
strengths: 1. Well written and easy to read.
2. Thoroughly explains the challenges experienced not only in avian bioacoustics, such as covariate shift and mismatch in focal and soundscape recordings, but also in curating, and developing a dataset of such a size and scale. 
3. The pain points addressed in the paper are very real: poor availability and accessibility of AudioSet, lack of a unified benchmark suite for evaluating segment and event-based bioacoustics tasks, and mismatch between train and test time for avian bioacoustics, are all pressing challenges. 
4. A good variety of baseline models were evaluated.
summary: The paper proposes BirdSet, which is a dataset, or more specifically, a collection of 11 different sub-datasets. BirdSet attempts to unify 
avian bioacoustic evaluation on focal and soundscape recordings under one roof, providing a large, unified, accessible test bed for testing audio classification approaches.
weaknesses: To me, the paper, in several places, tries to pose BirdSet as a replacement for AudioSet and that it should be the exemplary benchmark for evaluating audio classification models, with statements such as "Avian bioacoustics exemplifies challenges in audio classification...", and how curated datasets like AudioSet and ESC-50 do not represent real-world complexities. Pain points mentioned w.r.t. AudioSet are all very real, but AudioSet is a much broader dataset than BirdSet. Several people, in industry and academia have found AudioSet useful: a blanket statement saying AudioSet is not useful in real-world scenarios is plain wrong. 
Some statements straight up downplay AudioSet: for instance, line 37-38: where AudioSet poses "... concerns regarding transferability to real-world environmental domains". Whereas the paper cited simply shows that AudioSet pretrained models perform "well enough": of course models trained on in-domain bioacoustics data will fare better for bioacoustic evaluation!

Also, as per Table 3, AST and EAT models, which are pretrained on AudioSet, seem to perform quite well in the DT setting (cross-domain transfer setting) versus their LT and MT counterparts. For EAT, DT performance matches MT and outperforms LT scenarios. 

BirdSet will be a tremendous contribution to the fields of audio classification and avian bioacoustics, I have no doubt, but I think the current language of the paper comes off as "posturing" a tad bit too much.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: Please refer to the weaknesses above for the questions.
soundness: 4
strengths: 1. The introduction of *BirdSet* fills a notable gap in audio classification by providing a large-scale, domain-specific dataset for avian bioacoustics.
2. The paper is well-structured and clearly articulates the challenges in avian bioacoustics and audio classification more broadly. 
3. The paper provides a thorough empirical evaluation using six well-known deep learning models, covering various training scenarios, including large-scale training and fine-tuning.
summary: The paper introduces BirdSet, a large-scale audio classification dataset for avian bioacoustics, with around 520,000 recordings from nearly 10,000 bird species. It includes over 6,800 hours of training data and 400 hours of evaluation data from diverse regions. The dataset supports tasks such as multi-label classification and self-supervised learning, with standardized training and evaluation protocols. A comprehensive literature review identifies key challenges in bioacoustics and provides research guidelines. The paper benchmarks multiple deep learning models and offers a codebase for reproducibility.
weaknesses: 1. The literature review in Section 2 is quite extensive, occupying a significant portion of the main paper. While the analysis is comprehensive, it might be more beneficial to allocate more space to the dataset description and details, rather than using five pages for the related work.
2. The results focus mainly on the multi-label classification benchmark, with limited exploration of other use cases. Given the dataset’s availability of precise event timestamps, it would be valuable to include benchmarks for sound event detection. Additionally, providing results for other mentioned use cases would strengthen the paper.

Meta-review: The paper proposes a  benchmark dataset  for multi-label audio classification in bird acoustics. This dataset  improve upon existing audio datasets like audioset,  by proposing approximately thousands of classes and different test datasets.
The new dataset aims at addressing several challenges in machine learning, including domain adaptation shift, label noise, and task shift, making it an excellent resource for assessing model robustness in diverse audio classification scenarios. The benchmark evaluates deep learning models using three different training strategies.  All reviewers are happy to accept the paper.

Decision: Accept (Spotlight)