NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics
Authors: David Robinson
Marius Miron
Masato Hagiwara
Olivier Pietquin
Venue: ICLR 2025 Poster (2025)
Keywords: audio-language foundation models
multimodal large language models (llms)
bioacoustics
animal vocalizations
zero-shot learning
in-context learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large language models (LLMs) prompted with text and audio have achieved state-of-the-art performance across various auditory tasks, including speech, music, and general audio, showing emergent abilities on unseen tasks. However, their potential has yet to be fully demonstrated in bioacoustics tasks, such as detecting animal vocalizations in large recordings, classifying rare and endangered species, and labeling context and behavior—tasks that are crucial for conservation, biodiversity monitoring, and animal behavior studies. In this work, we present NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics. Our training dataset consists of carefully curated text-audio pairs spanning bioacoustics, speech, and music, designed to address the field's limited availability of annotated data. We demonstrate successful transfer of learned representations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks. We evaluate NatureLM-audio on a novel benchmark (BEANS-Zero) and it sets a new state of the art on several bioacoustics tasks, including zero-shot classification of unseen species. To advance bioacoustics research, we release our model weights, benchmark data, and open-source the code for training and benchmark data generation and model training.

Published as a conference paper at ICLR 2025
NATURELM-AUDIO: AN AUDIO-LANGUACE FOUNDATION MODEL FOR BIOACOUSTICS
David Robinson; Marius Miron, Masato Hagiwara, Olivier Pietquin Earth Species Project
ABSTRACT
Large language models (LLMs) prompted with text and audio have achieved state- of-the-art performance across various auditory tasks, including speech, music, and general audio, showing emergent abilities unseen tasks. However, their poten- tial has yet to be fully demonstrated in bioacoustics tasks, such as detecting animal vocalizations in large recordings, classifying rare and endangered species, and la- that are crucial for conservation, biodiversity
beling context and behavior- monitoring, and animal behavior studies. In this work, we present NatureLM- audio, the first audio-language foundation model specifically designed for bioa- coustics. Our training dataset consists of carefully curated text-audio pairs span- ning bioacoustics, speech, and music, designed to address the field's limited avail- ability of annotated data. We demonstrate successful transfer of learned represen- tations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks. We evaluate MaturoLM on a novel benchmark (BEANS-Zenu and it sets tasks, including zero-shot classification of unseen species. To advance bioacous- ties research, release model weights, benchmark data, and open-source the code for training and benchmark data generation and model training.
state of the art on several bioacoustics
Training Data
NatureLM-aude
Eveluation Data
Text Outpeit
Species disciPection Speciss dullectio onn teperdizoo bubknn
Blacoustics
Birds
Large Language Modiel
Marine mammelss
Bats
Amurans
Classiication Captiening
Generic
manmaats
Audio Embeddingg
Insects
Scher
Species classfication
Debection
Music
type classifican fication
Pitsh/ vetociiyyyyyyyy danection
o-fomer
slag-lage prediction
Captioning
Speaker dirization
Audio Encoder
Speaker defecciion
Figure 1: Overview of NatureLM-audio.
INTRODUCTION
Bioacoustics, the study of sound production and reception in animals, aims to understand animal behavior (Fischer et al., (013). monitor biodiversity (Stowell, 2022), and model the mechanisms underlying animal communication (Bradbury & Vehrencamp 1998). It plays vital role in conser- vation and ecological research, as animal vocalizations provide key insights into ecosystem health, species interactions, and population dynamics. By enabling the detection of endangered species and the tracking of migration patterns, bioacoustic research directly contributes to biodiversity monitor- ing and conservation efforts (Rutz al. 2023
Stevens et al. 2024
"Corresponding Buthor.avrddeerrrhheee
Project page: https:://aatttaeee github 10/ natuual aud o demo/

---

Published as a conference paper at ICLR 2025
In recent years, machine learning has taken increasingly pivotal role in bioacoustic research. Beyond its role in large-scale ecological monitoring, it has opened up new frontiers in the study of animal communication, enabling discoveries such the use of specialized vocalizations to label conspecifics in marmosets (Oren et al. 2024), dolphins (King & Janik] 2013), and elephants (Pardo et al., 2024). However, due to inherent challenges in data collection and annotation, many of these studies rely on strongly labeled small datasets [Stowell|2022 and require careful statistical analysis to ensure significance and mitigate over-fitting. Meanwhile, vast amounts of unannotated bioa- coustics data are recorded daily, particularly through passive acoustic monitoring (PAM, Dufourg al. (22TT) and citizen science platforms such as Xeno-canto (Vellinga & Planqué 2013). This growing data availability underscores the need for machine learning models capable of large-scale detection, classification, and annotation. The recent successes of large scale Al models in various domains-including natural language processing, computer vision, and game-playing Suanggssggggggstssses the possibbility of leveraging these large, raw datasets to learn robust and generalizable representations for bioacoustics (Ghani et al. 2023 Boudiat et al. 2023).
Existing bioacoustics machine learning models are typically designed for specific species or tasks (Dufourq et al.) [021] Kahl et al. 2021 Cauzinille et al., 2024), limiting their generalizabil- ity. Many traditional approaches rely on small datasets focusing on a few species and individuals, validating results through statistical measures despite the risks of over-fittigg More recent models, such as BirdNET (Kahl et al.] (022T) and Perch (Ghani et al.] (023). achieve strong performance in bird classification but require training dedicated classifiers for each target taxon. In contrast, we propose a single foundation model that generalizes across taxa. While recent self-supervised and audio-language contrastive models such AVES (Hagiwara 2023) and BioLingual (Robinson et al., 2024) have shown promising results on bioacoustics benchmarks, their discriminative and contrastive training paradigms constrain the range of tasks they can effectively address.
In recent years, foundation mondels-traines on large, diverse datasets, often via self-supervision have shown promising performance across multiple domains (Bommasani et al., 2021). While transformer-based large language models (LLMs)) are currently the most prominent examples, other architectures, such as diffusion models (Kingma et al., 202I), are also emerging as foundation mod- els in some domains. Their ability to handle unseen tasks, perform in-context learning, and respond flexibly to prompts makes them as an appealing alternative to traditional machine learning meth- ods, which typically depend on manually annotated datasets, extensive computational resources, and Soomain-specifie expertise.
While multimodal L.LMs, particularly lasion-language models (VLMss, have been explored in bio- diversity and conservation research (Miao et al. 2024), large audio-language models (LALLMs) re- main underexplored for bioacoustics. LALMS have demonstrated strong performance in human
speech (Rubenstein et al. 2023] Wang et al. 2024 Wu et al., 2023a Zhang et al. 2024). mu-
sic (Gardner al. 2024 Agostinelli et al., 2023), and general audio (Tang et al.] 2024 Chu al.,
2024 Gong et al. 2024). and they hold significant potential for advancing bioacoustics as well.
In this paper, we introduce NatureLM-audit the first audio-language foundation model specifi- cally designed for bioacoustic tasks, including classification, detection, and captioning. Inspired by cross-taxa transfer observed previous research, such as between human and gibbon or mar- mosets (Cauzinille et al., 2024, Sarkar & Magimai.-Doss, 2023) and birds and whales (Ghaani cet al. 2023), we incorporate speech and music tasks into training. We show that representations learned from these domains successfully transfer to animal vocalizations, demonstrating generaliza- tion across species. Additionally, we expand the BEANS bioacoustics benchmark (Hagiwara et al.] 2023) with new tasks, including call-type prediction, lifestage classification, captioning, and indi- vidual counting. This new benchmark, BEANS BEANS-Zero. enables us to evaluate cross- learning and zero-shot transfer to unseen taxa and tasks. Unlike existing bioacoustics benchmarks such as Perch (Ghani et (2023) for bird detection) and BirdSet (Rauch et al. (2025) for bird classification), we do not focus solely on birds and we go beyond species classification. Additionally, our dataset presents prompts and audio descriptions in natural language, fostering further research in LALMs.
Our contributions are as follows: (i) Model: We introduce MatureLM- the first audio-language foundation model for bioacoustics, trained on a carefully curated dataset spanning animal vocaliza- tions, general audio, human speech, and music. (ii) Domain transfer: We show that our model generalizes beyond the species seen during training and exhibits zero-shot capabilities on unseen taxa and species. (iii) Task transfer: We evaluate our model on BEANS-Zero, which extends be-
2

---

Published as a conference paper at ICLR 2025
yond species classification and includes unseen tasks such as individual counting. For the first time, we show positive transfer from speech and music data to bioacoustics tasks.
RELATED WORK
Most prior work on langual models has focused on human speech processing. Models such SpeechGPT (Zhang et al., 2023), Speech- (Wu et al.] 2023a), AudioLM (Borsos et al. 2023 AudioPal.M (Rubenstein et al. 2023). AudioGPT (Huang et al. (022b)) SpiRit-LM (Nguyen
al. 2025), and SpeechI. (Zhang et al. 2024) mostly focus on building language models that can perceive and produce human speech. While such models could, in principle, be fine-tuned for bioacoustic tasks, doing so would require significant computational resources and domain expertise. Instead, our model shows promising generalization to unseen species and tasks without requiring additional fine-tuning.
Recently, more general language models with audio perception capabilities have emerged. Pengi (Deshmukh et al., 2023) uses an audio encoder and a text encoder mapped onto an LLM to perform audio-to-text tasks. SALMONN (Tang et al., 2024) uses dual audio encoders and in- tegrates Q-Former (Li et al., 2023) to improve the handling of speech and general audio inputs. Qwen-audio (Chu et al. 2023) adopts a multi-task learning approach with the introduction of the Speech Recognition with Timestamp (SRWT) task. LTU (Gong et al., 2024) builds an open-ended question- answer dataset and applies currieulum learning strategies improve generalization. Sim- ilar multimodal models have been proposed for music, such as MU- MU-LLaMA (Liu et al., 2024) and LLark (Gardner et al. 2024). While recent foundation models such AVES (Hagiwara) 2023)
and Binull (Robinson et al. 2024) have demonstrated promising results in bioacoustics, their training paradigms and architectures constrain the range of tasks they can address.
Although animal sounds and vocalizations are often part of generic audio datasets such as Au- dioSet (Gemmeke al.) (2017) and audio caption datasets (Kim et al.) 2019 Mei et al. (002),, these datasets are often too broad and lack the fine-grained annotations required for bioacoustic tasks such as species classification, behavior analysis, or ecological monitoring. As result, LALMs trained on these datasets tend to produce only generic labels (c.g., 'bird' rather than a specific species name). We address this limitation by introducing diverse, multi-task training dataset and Naturel an LALM designed to produce robust representations for bioacoustics.
While specific bioacoustics benchmarks such as BIRB (Hamer al., 2023) for bird vocalization retrieval and BEANS (Hagiwara al. 2023) for classification and detection exist, the field still lacks comprehensive benchmarks comparable to those in human speech and music, such as Dynamic- SUPERB Huang et al. 2024a) or AIR-Bench (Yang al. 2024). This gap limits the evaluation of bioacoustic models, particularly in areas such as zero-shot learning and task generalization.
In this work, aim to bridge these gaps by introducing NatureLM-auce the first audio-language foundation model specifically designed for bioacoustics, and BEANS- -Zrro an expanded benchmark that evaluates cross-species and cross-task generalization.
3 TRAINING DATASET CREATION
Task
Audio Input
Text Instruction
Text Output
Classification
W
WWiS of Osse is the foool specie ae moiio Bocteearts Spenccll City Shensthssh Uheoolavci.. Aay
Gray Sholkelvuar
Detection
w
iffictof these
d-ibdl when So Sussiinn When Mavi
recording?
Captioning
the commins
Connest mudio
The sound of Sammp Sprime faliee betoown
Figure 2: Examples training instances.
3

---

Published as a conference paper at ICLR 2025
Task
Dataset
Hours
Samples
CAP
WavCaps Mei et al. 2024
7568
402k
CAP
AudioCaps [Kim
2019
145
52k
CLS
NSynth (Engel ct al. 2017
442
300k
CLS
LibriTTh Zen ct al. 2019
689
337k
VCTK Yamagiihi et al.
2019
CAP
Clotho Drossos
2020
25
4k
CLS, DET, CAP
Xeno-canto |Velinga Planqué
2015
10416
607k
CLS, DET, CAP
MNtturllist (iNaturalist)
1539
320k
CLS, DET, CAP
Watkins (Sayigh et al. (2016)
27
15k
CLS. DET
ASA (Museum für Naturkunde Berlin
78
16k
DET
Sapsucker Woods (Kahi et al. 2022
285
342k
CLS, DET
Barkley Canyon (Kanes 2021
876
309k
CLS
Urbansound (Salamon & Jacoby
2014)
10
2k
Table 1: Training tasks and datasets. CLS: classification, DET: detection, CAP: captioning.
To train an audio-text model for bioacoustics, we compile a diverse dataset of text-audio pairs (Ta- ble [T]. The data is collected through combination of prompting on existing audio datasets, gen- erating new LLM-generated text labels, and mixing new, procedurally augmented audio data. The dataset comprised of bioacoustic recordings, general audio, speech, and music datasets. Figure shows examples of training instances used for MatureLM-auuio. We plot the distribution of the training samples in Figure in the Appendix.
3.1 BIOACOUSTIC DATA
Species Classification: We standardize large-scale bioacoustic archives into common format, pro- cessing datasets such as Xeno-canto (Xeno-canto). iNaturalist (iNaturalist), Animal Sound Archive (Museum für Naturkunde Berlin), and Watkins (all-cuts, Sayigh al. (2016)). Differences in species naming conventions across datasets are reconciled using the GBIF taxonomy backbone (GBIF Sec- retariat 2023). We prompt the model to predict the scientific name, common name, or "laxonomic name" of the focal species or all species present in recording. Taxonomic names are written as "phylum class order family genus species" and are inspired by BioCLIP (Stevens et al. 2024), which found that flattening the hierarchy into text name improved generalization unseen species in computer vision. many real-world applications, an animal vocalization known to belong to a subset of species-fee example, based geographic location. To model this, generate prompts that present the model with a set of candidate species as possible answers. For 30% of prompts, we sample "random" negatives by selecting from all common names scientific names in dataset. In the remaining prompts, we introduce "hard" negatives by selecting species that share common ancestor at the family, order, or phylum level. The number of negative samples randomly selected, up to maximum of 35.
Unlike traditional bioacoustic models that predict based on audio alone, the text-audio formulation enables classification conditioned on additional context. We train the model to classify species while conditioned on recording metadata and field notes. We follow the same setup as above, but inject the time of the recording, the location, and the free-text notes of the recordist into the prompt. This data added wherever available for Xeno-canto, with time, location, or field note components randomly dropped percentage of the time.
To avoid data leakage, we exclude a set of held-out species and cbi data used in BEANS-Zero.
Species Detection: Using the same datasets as in species classification, we prompt the model to determine whether given species is present in a recording. The model selects from a provided set of candidate species chooses "None" when constructed with a mix of random and hard negatives, similar to the classification task. 50% of prompts, the correct species is omitted from the set, making "None" the correct answer.
correct option is given. Candidate sets
Because Xeno-canto comprises mostly focal recordings, we account for the covariate shifts in soundscapes by adding noise-audes that does not contain animal vocalizations, speech, sic. We source noise samples from datasets including: ShipsEar (Santos- Disitos-DDmmnuzzzz et al.] 2016), Deepship (Irfan et al. 2021) and Orcalab (Poupard et al., 2020) for boat engine sounds, as well as
4

---

Published as a conference paper at ICLR 2025
FSD50K (Fonseca et al. 2021) and Urbansound (Salamon & Jacoby 2014) for non-animal, non-
music sound classes, and all the classes from TUT2016 (Mesaros et al., 2016). IDMT (Abeßer et al,
202I), Demand (Theemann et al. 2013). and Wham noise (Wichern et al.
2019). The noise is added programmatically, using random files at a random signal-to-noise ratio (SNR) sampled from
uniform distribution between - 10dB and 20dB.
In addition, we used soundscape recording datasets for detection from Sapsucker Woods (SSW Kahl al. (2022)) for birds and from Barkley Canyon (Kanes) 2021 /oocitty 2013 201 b) for marine mammals. Following the BEANS detection dataset methodology. we segment audio into 10-second windows with a 5-second overlap, and treated as a multi- Mabli-label classification problem. Species with more than 100 occurrences were used as target labels, while those with fewer occurrences were grouped into an "other" class.
For bioacoustic captioning, we use the AnimalSpeak dataset (Robinson et al. 2024), which aggregates bioacoustic datasets into a prompts for captioning with scientific vs. common names, for "rich" captions over eight words, and for templated captions from Xeno-canto which follow strict structure.
Captioning:
dataset. We add separate
Call-type and Lifestage: We include multiple new bioacoustic tasks which can be expressed based on the Xeno-canto metadata. Specifically, predicting the life stage of birds, predicting call-types, and differentiating between calls and songs. The model is prompted using either audio alone or audio with the species name. Additionally, we include marine mammal call-type classification using Barkley Canyon recordings. These tasks go beyond species classification, providing finer-grained insights into ecological monitoring and animal behavior studies.
3.2 NON-BIOACOUSTIC DATA
General Audio We include WavCaps (Mei et al., 2024), AudioCaps (Kim et al. 2019), and Clotho (Drossos et al. (020) for general audio captioning. We observe that, during WavCaps cre- ation, some recordings originally contained metadata relevant to bioacoustics and specific species. However, this metadata was lost during general-domain captioning, resulting overly generic de- scriptions. We identify such cases by analyzing the original metadata, and re-process the metadata prompting Gemini- (imiiii-.00-pr 1.0-pro to produce bioacoustic captions. These enhanced captions are included alongside the original ones.
Music Pitch, timbre, and the number of animals in a recording are key acoustic features used by biologists to infer context and behavior. We use NSynth 2.3.3 (Engel et al.) 2017) to create a set of tasks that may help bioacoustics downstream tasks. We generate text prompts for pitch detection in Hz, instrument name, and velocity, ranging 0 1. Additionally, 'qualities' labels to create text descriptions for each audio. For instance, if the sound 'distorted,' generate descriptions such as "This sound has distinctive crunchy sound and presence of many harmonics." " or "This sound is distorted".. Moreover, we create synthetic mixtures by layering one to three different instruments. In this case we generate tasks: predicting the number instruments and identifying the instrument names.
the timbre
Speech We use LibriTTS Zen et al., 2019) and VCTK YYaaaiishi et al. 2019) to generate syn- thetic mixtures of up to four speakers, a task that may transfer to individual counting in bioacoustics. To better match the frequency variability with factors sampled from an uniform distribution between 0.25 to 4 (i.e., from 4x slower to 4x faster). Since animal vocalizations tend to be sparse, we insert random segments of silence at local minima computed the RMS of the speech signals. To enhance realism, further convolve the generated mixtures with impulse responses sampled from the DNS Challenge (Dubey et al. 2024b).
animal vocalizations, we timesccale the speech mixtures
EVALUATION DATA: THE BEANS-ZERS BENCHMARK
One of the key contributions of this work is BEANS- new benchmark for bioacoustics (Ta- 2le BEANS Zero extends beyond traditional species classification by introducing new tasks such call-type prediction, lifestage classification, captioning, and individual counting, which is not
5

---

Published as a conference paper at ICLR 2025
Task
Dataset
Description
Size
# Labels (type)
CLS esc50
generic sound
400
(sound type)
CLS watkins
marine mammals
339
31 (species)
CLS
cbi
birds
3620
264 (species)
CLS humbugdb
mosquito
1859
14 (species)
DET dcase
birds & mammals birds
13688
20 (species)
DET enabirds
4543
34 (species)
DET hiceas
cetaccans
1485
I (species)
DET rfcx
birds & frogs gibbons
10406
24 (species)
DET
g gibbons
18560
3 (call type)
CLS
unseen-species
birds etc. birds etc.
1255
200 (species)
CL.
unseen-genus
951
101 (species)
CLS
inseen-famil 1y
birds etc. birds
451
(species)
CL.S
1 festage
466
3 (stage)
CLS
call-type
birds
1000
2 (call/song)
CAP
captioning
birds etc.
29002
(open-ended)
CLS
zf-indv
zebra finches
1160
2 (# of indv.)
Table 2: Evaluation tasks and datasets of BEANS-Zero. CLSS
classification, DET: detection,
CAP: captioning. Thhe numbers of samples for classification and captioning, and the number of 5-second "chunks" for detection (see Section for more details.)
seen during training. To construct BEANS-Zen -Zro, begin with the test portion of BEANS (Hagiwara
al. 2023 evaluates models on standard bioacoustic tasks and datasets, including:
esc50 (Piczak
2015;:
Generic environmental sound classification with 50 labels.
watkins (Sayigh et al.
2016 Marine mammal species classification with 31 species.
cbi (Howard al.
2020 Bird species classification with 264 labels from the Comell
Bird Identification competition hosted on Kaggle.
humdubdb (Kiskin et al.
2021): Mosquito wingheat sound classification into 14 species.
dcase (Morfi et al. 2021 ) Mammal and bird detection from DCASE 2021 Task 5: Few-
shot Bioacoustic Event Detection (20 species).
enabirds (Chronister et al. 2021): Bird dawn chorus detection (34 species).
hiceas (Center 2022): Minke whale detection from the Hawaiian Islands Cetacean and
Ecosystem Assessment Survey (HICEAS) (1 label).
rfex (LeBien et al. 2020): Bird and frog detection from the Rainforest Connection
(RFCx) data with 24 species.
gibbons (Dufourq et al. 2021): Hainan gibbon detection with 3 call type labels.
We also include novel bioacousties datasets including:
unseen-species: 200 species held out from AnimalSpeak (Robinson et al. 2024). For controlled measure of generalization, we hold out species whose genus well- represented (at least 150 training examples)
unseen-genus: We hold out entire genus whose family is well-represented (at least 250 training examples) totaling 101 unique species.
unseen- family: hold out entire families whose class well-represented (at least training 250 examples) totaling 36 unique species and representing the hardest generaliza- tion setting.
1 festage: Predicting the lifestage of birds across multiple species. Newly curated from Xeno-canto (Xeno-canto).
call-type: Classifying song call across multiple bird species. Newly curated from Xeno-canto (Xeno-canto).
capt.ioning: Captioning bioacoustic audio AnimalSpeak (Robinson al. 2024). zf-indv (Elie & Theunissen, 2016): Determining whether recording contains multiple zebra finches, using programmatically generated mixtures (1 individuals).

---

Published as a conference paper at ICLR 2025
Some of these tasks, particularly bioacoustic captioning, have not been extensively studied before. Captioning allows for automatic generation of descriptive annotations of animal sounds, enhancing understanding of species behaviors and communication patterns. Improvements in other new tasks, such as cross-species lifestage and call-type prediction, would allow finer-grained ecological monitoring and animal communication studies scale.
For evaluation,, we use accuracy for classification, macro- macro-averaged F1 for detection, and SPIDEr (Liu al. 2017) for captioning. Unlike mean average precision (mAP), which is originally used in BEANS and assumes a smooth ranking of candidates, Fl is more appropriate for evaluating gener- ative tasks. This ensures a fairer assessment of models that generate predictions instead of ranking pre-defined classes.
5 NATURELM-AUDIO ARCHITECTURE
Our model follows a generic audio-to-text architecture similar to prior LALMs such as SALMONN (Tang et al. 2024). Qwen2-audio (Chu et al.] 2024), and LTU (Gong et al., 2024). These models are trained on paired audio-text data for tasks including speech, music, and general audio event understanding. Figure P an overview of the Nature architecture.
NatureLM-ause first encodes the input audio using BEATs (Chen et al. 2023), state-of-the-art audio encoder multiple audio tasks. To connect the BEAT' embeddings with the LLM, we use a Q-Former (Li et al., 2023) applied at the window level as proposed SALMONN (Tang et al. 2024). Similarly to the existing LALMs, we use LLM to produce text, in this case Llama 3.1- 8b (Dubey et al. 2024a), which is fine-tuned with LoRA (uu et al. 2022). During training, only the adapter layers of the L.MM are updated, while the base L.M parameters remain frozen. In contrast, the audio encoder and Q-Former remain trainable. The model takes an audio input a along with an instruction æ and produces text output y. The model is trained under the loss function:
h
Sw (Encoder(a)
P (g h)
(2)
L = - -Elog
loggl" pa I.M
(ye l0, 2, 3/4))
(3)
where Encoder is the pretrained BEATS (Chen al. converts consecutive W audio frames into a window, parameters & that converts a window into sequence text representations z using query q. and LM is the pretrained LLM with trainable parameters 6. Pg
2023) audio encoder, Jw is a function that P is the Q-Former model with trainable
6 TRAINING METHOD
Our training method follows a curriculum learning approach (Soviany et al. 2021). where the model is first trained on simpler tasks before progressively tackling more complex ones, as done in other audio foundation models (Tang et al. 2024 Gong et al. 2024). We train the two stages:
Stage 1 (Perception Pretraining): We pretrain the model exclusively on focal species classi- fication, classifying vocalizations from thousands of animal species. Species classification is a highly deterministic task, allowing opportunity to learn robust connection between language and audio. We also choose train on this task individually as it is foundational to other tasks in bioacoustics.
Stage 2 (Generalization Fine-tuning): In the second stage, we introduce a variety of bioa- coustic and other tasks. building the robust classification abilities developedin Stage 1. This includes detection, captioning, lifestage prediction, and call-type prediction. We also include speech and music data in this second stage, aimed at improving transfer to bioacoustic tasks.
We train M-aurio from scratch, initializing the Q-Former and LoRA layers randomly rather than fine-tuning existing LALM checkpoints such as SALMONN. This allows for more flexibility in terms of choosing the latest LLM with the extensive knowledge of animal species, and the most relevant architectural components (c.g., excluding memory-intensive parts of current LALMs such as the Whisper speech encoder (Radford al. 2023

---

Published as a conference paper at ICLR 2025
Model
esc50
watkins
cbi
humbugdb
dease
enabirds
hiceas
rfcx
gibbons
LLM w/o audio
0.020
0.041
0.005
0.073
0.000
0.001
0.210
0.000
0.013
SALMONN
0.320
0,041
0.004
0.090
0.005
0.004
0.097
0.002
0.005
Qwen2-audio
0.307
0,041
0.004
0.070
0,005
0.004
0.097
0,002
0.005
BioLingual 0.600
0.257
0.705
0.085
0.036
0.109
0.429
0.004
0.018
NatureLM-autt audio
0.820
0.788
0.778
0.114
0.058
0.314
0.336
0.025
0.005
Table 3: Main zero-shot results BEANS-Zero. We used accuracy for classification, and F1 for
detection tasks. The best and the second best metrics are highlighted and underlined per each dataset.
7 EXPERIMENTS
7.1 TRAINING AND EVALUATION DETAILS
We train our model on the full curated training set (Section 3). To evaluate generalization, we create hold-out splits for Xeno-canto, iNaturalist, Animal Sound Archive, and Watkins datasets, used solely for benchmarking.
We initialize the audio encoder weights using an existing BEATs checkpoint pt) and fully fine-tune it, which
(BEATs_i ter w.w.errppuuua..... www.ss.fiiettteed..o. we found to be critical in an ablation (Table 9). We initialize the LLM from 1ama--.1-8B-Insttae and apply LoRA to all attention layers (rank: 32, alpha: 32, dropout: 0.1).
We follow the proposed two-stage training strategy. In both stages, we use linear warmup followed by cosine learning rate schedule, with a peak learning rate of 9.0 x 10-5 and an end learning rate of 2.0 x 10 We use a batch size of 128 and run the first stage for 5.0 x 10 steps and the second stage for 1.6 x 10 steps. For inference, we use beam search with two beams, repetition penalty of 1.0, and a length penalty of 1.0.
We consider several inference methods depending on the task type. Spssies-casssification tasks involve single-label prediction: we prompt the model to output the species name from the recording. Since the LLM may generate text that does exactly match predefined labels, we use Levenshtein distance to map predictions to the closest species name. We choose the Levenshtein distance for its simplicity and because species names, particular Latin names. have high character-overlap with related names. However, we note that it may not be optimal for general audio classification.
For multilabel detection tasks, the number of target species varies by dataset. For tasks with 10 or fewer species, we include the species options in the prompt. Otherwise we prompt the model to list all species in the audio, any. In both cases, the model outputs all detected species, or 'None', We discard predictions with low character overlap with the valid labels.
Our baselines include CLAP-like models (Wu et al. (023b), which cannot naively perform multil- abel detection. To address this, we create negative "template" for each detection task, proposed by Miao al.] (2023). We consider each label a detection positive for CLAP if the audio more similar to the label than to the negative template in the CLAP model's embedding space.
7.2 SPECIES CLASSIFICATION AND DETECTION
Table 3 shows the main results measured on the BEANS-Zero species classification and detection datasets. Our baselines include an LLM (the original Llama-3.1-8B-Instrul model without fine-tuning, Dubey et al. (2024a)) without audio input, SALMONN (Tang et al.) 2024). BioLin- gual (Robinson et al. 2024), and Qwen2-audio (Chu et al. 2024). All baselines are evaluated in the same way as NatureL audio input, SALMONN, and Qwen2-audio are largely random on bioacoustic datasets, failing to properly interpret the input audio or follow the instructions. In contrast, NatureLM-auddi achieved state-of-the-al zero-shot performance on 7 out of 9 datasets, and delivered competitive results on the remaining tasks from the BEANS-Zero benchmark. We note that performance of baselines on the general audio dataset ESC50 (Piczak) 2015) may be reduced by the use of the Levenshtein distance, as pipeline is optimized for bioacoustic tasks.
As shown in the table, the outputs from the LLM without
8

---

Published as a conference paper at ICLR 2025
cbi
dease-bird
enabirds
BirdNET
0.609
0.035
0.490
Perch
0.744
0.035
0.164
NatureLM-audee
0.778
0.083
0.314
Table 4: Comparison with bird vocalization models.
unseen-species"
unseen-genus"
unseen-family"
Supervised SotA
0.687
0.688
0.545
random chance
0,005
0.010
0.028
baseline (CLAP)
0.014
0.026
0,082
NatureLM-auudie (cmn)
0.181
0.116
0.035
NatureLM Maudio (sci)
0.238
0.041
0.035
MatureLM-audio (tax)
0.343
0.148
0.308
Table 5: Generalization to unseen taxa terms of classification accuracy. All tasks predict species names, test sets held-out at the species genus and family level. 'arggts were not held out from "Supervised SotA" reference (BioLingual). Cmn, sci, and tax denote predictions using common, scientific, and taxonomic names respectively. Since the number of labels varies across datasets, results should not be directly compared across columns.
We also compared NatureLM-audde with bird bird-specific classification models, namely BirdNET (Kahl et al. 2021) and Perch (Ghani al.) 2023), to evaluate the zero-shot capabilities of our model. We compare on the bird-related datasets of BEANS-Zero, plus the portion DCASE with bird species. Results are presented in Table 4 Since both BirdNET and Perch
trained in supervised manner on datasets that significantly overlap with our bird evaluation datasets, this is not fully fair comparison, and their performance should be considered topline results. Nevertheless, our model demonstrated strong zero-shot bird vocalization classification capabilities. In particular, we achieve a new SotA for the cbi dataset, classifying vocalizations of hundreds of birds, and achieve competitive results with the bird-specific models on both detection tasks. We additionally compare against various models on datasets from the BirdSet benchmark (Rauch et al. (2025). where our model achieves the highest average top-1 accuracy (Appendix in Table
7.3 GENERALIZING TO UNSEEN SPECIES
We further evaluate the model's ability to generalize to completely unseen taxa using the newly added datasets families. As topline, we compare against BioLingual, which has seen these taxa in training and only indicates fully supervised performance. As baselines, we consider theoretical random base- line (1 number of classes) and CLAP-LAION (Elizalde et al., 2003), general-domain audio model which, similar to our model, is unlikely have seen these species during training. We compare the performance when predicting common, scientific, or taxonomic names.
BEANS-Zeer, held out three levels: unseen species, unseen genus, and unseen
3 the results. Across all three unseen taxa settings, Naturel.M-audin significantly out- performs the random baseline, demonstrating its ability to generalize to unseen taxa and taxonomic branches. For example, on the unseen species test set, our model achieves an accuracy of 34.3%, far surpassing the random baseline of 0.5%, indicating that the model has learned features that extend beyond the species it was trained on. The model also outperforms CLAP-LAION, further emphasiz- ing its ability to generalize. We observe that predicting with taxonomic names consistently improves performance across all settings, and is particularly critical for generalizing to unseen genus and fam- ilies where scientific (Latin) names alone fail to capture hierarchical relationships. We further note that scientific names perform relatively well when generalizing to unseen species, but perform worse than common names for generalizing to unseen genus. This suggests that common names may en- code broader hierarchical information or be more familiar to the language model.
7.4 NOVEL BIOACOUSTIC TASKS
Beyond species classification, we evaluate NatureLM-audio on novel bioacoustic tasks introduced in BEANS-Zero, which, to the best of our knowledge, have not been previously studied cross-

---

Published as a conference paper at ICLR 2025
lifestage
call-type
captioning
zf-indv
SotA NatureLM-audie
0.502
0.658
0.009
0.604
0.794
0.871
0.532
0.655
Table 6: Results BEANS-Zero novel bioacoustics tasks. We report accuracy for classification, and SPIDEr (Sharif et al.] 2018) for captioning. SotA is SALMONN for captioning and BioLingual for the remaining tasks.
species level. We additionally include 2f-indv, completely unseen task that determines whether a recording contains multiple zebra finch individuals or just one (Elie & Theunissen (016)) We compare against Biol.ingua (Robinson et al. 2024) for discriminative tasks and SALMONN (Tang)
et al. 2024) for captioning. As shown in Table 6 M-aadio sets a new state-of-the-aae across all tasks. We evaluate call-type classification more extensively (Table (), and find the model is able
to transfer this task to unseen taxa. We further find the model can improve audio classification performance by incorporating additional context as text, which we discuss in the Appendix in A.3
7.5 ABLATION ON SPEECH AND MUSIC
To investigate the impact of speech and music on downstream task performance, we run an abla- tion during stage-2 training. Specifically, we train two versions of the model for 150k tee with speech and music data and one without- and evaluate their ability to perform an unseen task: counting zebra finches. The model trained with speech achieves 67.7%, similar to our full model. The model trained without speech scored 50.0%, exactly random, and qualitatively predicted 'more than one' for all examples. These results suggest the ability to count vocalizing birds transfers from human speech and music, as our training data includes tasks such as counting human speakers in a recording. We include the ablation performance all tasks in the Appendix (Tables I0 and ITI.
8 CONCLUSION
We presented MatureL_M-audio, the first audio-language foundation model specifically designed for bioacoustics, demonstrating its potential to address critical tasks such as classifying and detecting animal vocalizations, and decoding context, call types, and individuals across species. By lever- aging carefully curated dataset spanning bioacoustics, speech, and music data, NatureLM-aude sets the new state-of-the-as on multiple tasks, including zero-shot classification of unseen species. Moreover, our model demonstrates positive transfer across both domains and tasks, performing well on a novel benchmark (BEANS-Zero) which includes new bioacoustic tasks such as captioning and individual counting. To further accelerate research and the development of more robust models in the field, we have ooen-sourced the code for generating both training and benchmarking data.
We plan to extend this work by incorporating more diverse tasks and datasets, improving the text- based LLM backbone with bioacoustie-specific texts, and enhancing the model's multilingual ca- pabilities. Another direction is the introduction of new modalities, such motion and image data, leading to multimodal models like NatureLM Momoioon and NatureLM-image. We also aim to ex- plore the model's generative abilities, particularly in producing audio tokens for applications such as animal sound synthesis and audio denoising.
While NatureL M-audoo offers significant potential for advancing biodiversity monitoring and con- servation, several ethical concerns must be addressed. First, there is a potential bias towards bird vocalizations due the overrepresentation of bird datasets, which could limit the model's effec- tiveness in other taxa. Second, the model's ability to detect and classify endangered species could be misused for illegal activities such as poaching, posing a threat wildlife. Finally, unintended consequences on animal behavior and ecology must be considered, particularly when deploying LLMs, known for their issues including hallucinations and biases (Kuan al.) 2024). These sys- tems may interfere with the behavior of the species being studied, and the long-term ecological impact of widespread passive monitoring is still unknown. Careful deployment and responsible use are essential to mitigate these risks.
Acknowledgement: We thank Benno Week, Sara Keen, Milad Alizadeh, Gagan Narula, and Matthieu Geist for their contributions this work.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jakob Abeßer, Saichand Gourishetti, András Kátai, Tobias ClauB, Prachi Sharma, and Judith Liebe- trau. IDMT-Traffic: an open benchmark dataset for acoustic traffic monitoring research. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 551-555. IEEE, 2021.
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. MusicLM: Generating music from text, 2023. URL
axg/abs/2301. 11325
Rishi Bommasani et al. On the opportunities and risk of foundation models. ArXiv, abs/2108.07258, 2021. URLL.URLheee httthttp///raatt http://awivvvorrras////aaaas wttp:oo/ggaa/oooooooo 07258
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. A A language modeling approach to audio generation. IEEF/ACM Transactions Audio, Speech, and Language Processing, 31:2523-25333, 2023. 3288409.
doi: 10.1 10.1109/TASLP22023
Malik Boudiaf, Tom Denton, Bart Van Merriënboer, Vincent Dumoulin, and Eleni Triantafillou. In search for a generalizable method for source free domain adaptation. In International Conference on Machine Learning, pp. 2914-2993 PMLR, 2023.
Jack W. Bradbury and Sandra L. Vehrencamp. Principles of animal communication, volume 132. Sinauer Associates Sunderland, MA, 1998.
Jules Cauzinille, Benoît Favre, Ricard Marxer, Dena Clink, Abdul Hamid Ahmad, and Arnaud Rey. Investigating self-supervised speech models' ability to classify animal vocalizations: The case of gibbon's ss vocal identity. In Proceedingg of Interspeech. ISCA, 2024.
NOAA Pacific Islands Fisheries Science Center. Hawaïann islands cetacean and ecosystem assess- ment survey (HICEAS) towed array data. Edited and annotated for the 9th International Work- shop on Detection, Classification, Localization, and Density Estimation of Marine Mammals Us- ing Passive Acoustics (DCLDE 2022), 2022. doi: https://oii.rr/// http://oo..rr/1..22222/eee 11tp9////1111pgp6//1/1
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 5178 5193. PMLR, 23-29 Jul 2023. URL https: htttp:://proceedinng
pres.press httppppssaa//////.. 220/// html
Lauren M. Chronister, Tessa A. Rhinehart, Aidan Place, and Justin Kitzes. An annotated set of audio recordings of Eastern North American birds containing frequency, time, and species infor- mation. Ecology. 103(6),c0332, 2021. doi:
URL https:
//esajournalgs on nel www.inneeebrrrr.llllmmmmamaaaaal wtt.leeccco//o/////////////
http:///aab......0//eeee./ ecy. 332/
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijic Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arxiv:2311.07911 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024.
URL https: http///xxvvv.o/ wrg. abs/2407 10759
Soham
Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. In Advances in Neural In-
formation Processing Systems, volume 36, pp. 18100- 2023.
URL
https::/proceeeee ceresnnnrrrpi cctpsscccppppreeeeee http://aaaroreeeee////////////// 2a225359ehooff117adddaaaaaaaaee2

---

Published as a conference paper at ICLR 2025
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Abhimanyu Dubey al. The Llama 3 herd of models, 2024a. URL https://arxiv. abb/2407.21788
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cut- ler, Alex Ju, Mehdi Zohourian, Min Tang, Mehrsa Golestaneh, et al. ICASSP 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024b.
Emmanuel Dufourq, Ian Durbach, James Hansford, Amanda Hoepfner, Heidi Ma, Jessica V. Bryant, Christina S. Stender, Wenyong Li, Zhiwei Liu, Qing Chen, Zhaoli Zhou, and Samuel T. Turvey. Automated detection of Hainan gibbon calls for passive acoustic monitoring. Remote Sensing in Ecology and Conservation, 7(3):475-455 487, 2021. doi: https://doi.org/10.00000.. 201. URL ltt.tp:://ssppubiapp http::///s..pbiaaatiossssssssss online .aclaaryywwwlyy.. wiley. com/ /oi/ / 10. 12w1102//ra r0002/rsee..m 201
Julie E Elie and Frederic E Theunissen. The vocal repertoire of the domesticated zebra finch: data- driven approach decipher the ceermattoo-bbarinn acoustic features of communication signals. Animal cognition, 19:285-315 2016.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning audio concepts from natural language supervision. In Proceedings of the International Conference Acoustics. Speech and Signal Processing (ICASSP), pp. 1-5, 2023. ISBN 978-1-7281-6327-1
doi: 10.1 10.1109//CASPP493555 90.9
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with WaveNet autoencoders. In International Conference on Machine Learning, pp. 1068-1077. PMLR, 2017.
Julia Fischer, Rahel Noser, and Kurt Hammerschmidt. Bioacoustic field research: a primer to acous- tic analyses and playback experiments with primates. American journal of primatology, 75(7); 643-663, 2013.
Fonseca, Favory, Pons, Font, and X Serra. Fsd50k: an open dataset of -auuan-labeled sound events. 1EEF/ACM Transactions Audio, Speech, and Language Processing, 30:829-852, 2021.
Josh Gardner, Simon Durand, Daniel Stoller, and Rachel Bittner. LLARK: multimodal instruction- following language model for music. In Proceedings of the 41st International Conference on Machine Learning. ICML'22. JMLR.org, 2024.
GBIF Secretariat, GBIF backbone taxonomy. 2023, URL
org/
dataset//7cddd 2-c.ecodddee22o--3333-2333 9bbbabbb9999aae333
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, 2017. doi: 10.119//CASSPP::077799555
Burooj Ghani, Tom Denton, Stefan Kahl, and Holger Klinck. Global birdsong embeddings enable superior transfer learning for bioacoustic classification. Scientific Reports, 13(1):22876, 2023.
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. Listen, think, and understand. In The Twelfth International Conference on Learning Representations, 2024.
URL https: httphhssttspseeeeeeeeee / htwp:/eeettoooommmiiooooooooo.oo
Masato Hagiwara. AVES: Animal vocalization encoder based self-supervision. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5,
2023. ISBN 978-1-7281 10081-1-281-6327-7,
doi: 10.1 10.1009/CCASS944935..0222 2022. 10095642.
12

---

Published as a conference paper at ICLR 2025
Masato Hagiwara, Benjamin Hoffman, Jen- Yu Liu, Maddie Cusimano, Felix Effenberger, and Katie Zacarian. BEANS: The benchmark of animal sounds. In Proceedings of the Interna- tional Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10.1 N0.100//AASS99995..000 200.
Jenny Hamer, Eleni Triantafillou, Bart van Merriënboer, Stefan Kahl, Holger Klinck, Tom Denton, and Vincent Dumoulin. BIRB: A generalization benchmark for information retrieval in bioacous- tics, 2023. URL bttps://aaax.
07439
Addison Howard, Klinck, Sohier Dane. Stefan Kahl, and Tom Denton. Coreell Birdcall Iden- tification. htt.cia......ggea.a/c......o//oa
recognition 2020. Accessed
URL
2023-06-01.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022. URL
hwtp//n.e///eeee. / forum?
id=nževKeeFYE
Chien- Yu Huang, Ke-Han Lu, Shih-Heng Wang. Chi-Yuan Hsiao, Chun- Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, and Hung yi Lee. Dynamic-SUPERB: Towards dy- namic, collaborative, and comprehensive instruction-tuning benchmark for speech, 2024a. URL https:://axxtt org/ abs 220. 09510
Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yun- ing Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Yuexian Zou, Zhou Zhao, and Shinji Watanabe. AudioGPT: understanding and generating speech, music, sound, and talk- ing head. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixil Conference on Innovative Applications of Artificial Intelligence and Fourteenth Sym- posium on Educational Advances Artificial Intelligence, AAAI24HAAAI24PAAN"" AAAI doi: 10.16099aaai.v33321...oo
Press, 2024b. ISBN 978-1-57735-887-55
URL https:
7/doi. org. 1111110.69//aa...
30570
iNaturalist. iNaturalist,
https://wwaz hwtp:/www.ttur..mmmm iww.unalist.orr...o...... acccessed 2023-05-01.
URL
ht https: /www.
Inatural list .org/
Muhammad Irfan, Zheng Jiangbin, Shahid Ali, Muhammad Iqbal, Zafar Masood, and Umar Hamid.
Deepship: An underwater acoustic benchmark dataset and a separable convolution based autoen- coder for classification. Expert Systems with Applications. 183:115270, 2021. ISSN 0957-4174. doi: /tt.iinne/attcce///i//////// com/
URL /httttt::///t htttps://www.... www. www.neeiieneeede.o rect.
007016
Stefan Kahl, Connor M. Wood, Maximilian Eibl, and Holger Klinck. BirdNET: deep learning solution for avian diversity monitoring. Ecological Informatics, 61:101236, 2021. ISSN 1574- 9541. doi: 1...01/J..COONN.222.....
Stefan Kahl, Russell Charif, and Holger Klinck. A collection of fully-amnotated sow.ttscpeddoonneeccccceceee record- ings from the Northeastern United States, September 2022. URL https:://6ii.og//// 5281/zenodo. 7079380
Jasper Kanes. Marine mammal phonations of Barkley Canyon: A publicly available annotated data set. J. Acoust. Soc. Am., 150(4 Supplement):A44 October 2021. doi: 10.110/10000077588. URL
https:://oi.com htt.:/oo..oogg/0// 110...12/1111 000708777777
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generat- ing captions for audios in the wild. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 119-132, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 00.18653/1/N119-001 URI //acll antho ogy. ogy.
13

---

Published as a conference paper at ICLR 2025
Stephanie L King and Vincent M Janik. Bottlenose dolphins can use learned vocal labels to address each other. Proceedings of the National Academy of Sciences, 110032)113216-1322. 2013.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707 2021.
Ivan Kiskin, Marianne Sinka, Adam D. Cobb, Waqas Rafique, Lawrence Wang, Davide Zilli, Benjamin Gutteridge, Theodoros Marinos, Yunpeng Li, Emmanuel Wilson Kaindoa, Gerard Killeen, Katherine J. Willis, and Roberts. HumBugDB: a large-scale acoustic mosquito dataset. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, 2021.
Chun-Y Yi Kuan, Wei-Ping Huang, and Hung-yi Lec. Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models. In Proceedings of Inter- speech, 2024.
Jack LeBien, Ming Zhong, Marconi Campos-Cerqueira, Julian P. Velev, Rahul Dodhia, Juan Lavista Ferres, and T. Mitchell Aide. pipeline for identification of bird and frog species in tropical soundscape recordings using a convolutional neural network. Ecological Informatics, 59:101113, 2020. ISSN 1574-9541. doi: https://doi.org/10..100 www.nf. 101113. URL https:// www. sci enced rect. com sci enceeenneeiioooo http:///c.caeo/aaoooo/////////// 277354444555220237777
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music Understanding LLaMA: Advancing text-to-music generation with question answering and captioning. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 286-290, 2024.
doi: 10.1 10.1109/CAASP44485..0000 2024444448.2222444444 10447027.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image cap- tioning via policy gradient optimization of SPIDEr. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 873-881, 2017. doi: 10. 19/11CC22001
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumb- ley, Yuexian Zou, and Wenwu Wang. WavCaps: ChatGPT weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions Audio, Speech, and Language Processing, 32:3339-3354 2024. doi: 1..1199/TASSL..2224...0
Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. TUT database for acoustic scene classi- fication and sound event detection. In 2016 24th European Signal Processing Conference (EU- SIPCO), pp. 1128-1132. IEFE, 2016.
Zhongqi Miao, Benjamin Elizalde, Soham Deshmukh, Justin Kitzes, Huaming Wang, Rahul Dodhia, and Juan M. Lavista Ferres. Zero-shot transfer for wildlife bioacoustics detection. Research Square, 2023, URL htttts:////ooo http:://ooooog/10// 21203/rs22... rs 3180218/v1
Zhongqi Miao, Yuanhan Zhang, Zalan Fabian, Andres Hernandez Celis, Sara Beery, Chunyuan Li, Ziwei Liu, Amrita Gupta, Md Nasir, Wanhua Li, Jason Holmberg. Meredith Palmer, Kait- lyn Gaynor, Rahul Dodhia, and Juan Lavista Ferres. New frontiers in Al for biodiversity re- search and conservation with multimodal language models. EcoEvoRxiv, 2024. URL https:
//ecoevoxxiv.c wwt.:v/www..ogggeeesitoooo//////
7477/
Veronica Morfi, Inês Nolasco, Vincent Lostanlen, Shubhr Singh, Ariana Strandburg- Peshkin, Lisa F. Gill, Hanna Pamula, David Benvent, and Dan Stowell. Few-shot bioacoustic event detection: A new task at the DCASE 2021 challenge. Detection and Classification of Acoustic Scenes und Events 2021, 2021.
Museum für Naturkunde Berlin. Animal sound archive. 1 Accessed via gbif.org 2023-05-09.
14

---

Published as a conference paper at ICLR 2025
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Pop- uri Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. SpiRit-LM: Interleaved spoken and written language model. Transactions of the Association doi: a URL https:
for Computational Linguistics, 13:30-52, 2025.
Guy Oren, Aner Shapira, Reuven Lifshitz, Ehud Vinepinsky, Roni Cohen, Tomer Fried, Guy P. Hadad, and David Omer. Vocal labeling of others by nonhuman primates. Science, 385(6712); 106-1003,200 2024. doi: 10.1 (001116/scieencc.dp3357777 abs abs/10.112ee wtts:ciiiecceec add3757
URLhttpss: /ttp////.....ttt///////.//aa/a httppp///www.accence.../////////
Michael A Pardo, Kurt Fristrup, David S Lolchuragi. Joyce H Poole, Petter Granli, Cynthia Moss, lain Douglas Hamilton, and George Wittemyer. African elephants address one another with indi- vidually specific name-like calls. Nature Ecology & Evolution, pp. 1-12, 2024.
Karol J. Piczak. ESC: Dataset for environmental sound classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pp. 1015-1018. ACM Press, 2015. ISBN 978-1-4503- 3459-4. doi: 10. 244144//733333288666999 URL http: //tl acm. wwtp://ggg/gctiiiiooe cfm?
doid-2733373 2806390
M Poupard, P Best, M Ferrari, P Spong. H Symonds, J-M Prévot, T Soriano, and H Glotin. From massive detections and localisations of orca at orcalab over three years to real-time survey joint to environmental conditions. In e-Forum Acusticum 2020, pp. 3235-3237, 2020.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th Interna- tional Conference on Machine Learning, ICMI.223. JMLR.org. 2023.
Lukas Rauch, Raphael Schwinger, Moritz Wirth, René Heinrich, Denis Huseljic, Marek Herde, Jonas Lange. Stefan Kahl, Bernhard Sick, Sven Tomforde, and Christoph Scholz. Birdset: A large-scale dataset for audio classification in avian bioacoustics. In The Thirteenth International Conference on Learning Representations, 2025. URI/tttps://ppeneviiee net forum?
David Robinson, Adelaide Robinson, and Lily Akrapongpisak. Transferable models for bioacoustics with human language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing, Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024, Seoul, Republic of Korea, April 14-19, 2024, pp. 1316-1320. IEEE, 2024. doi: N0.11009AASS 10447250.
URL
1109/
ICASSP48465 2024 101442550
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Han- nah Muckenhim. Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Ve- limirović, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, and Christian Frank, AudioPaL.M: large language model that can speak and listen. arXiv preprint 2rXiv:2306.1299 2023.
Christian Rutz, Michael Bronstein, Aza Raskin, Sonja C. Verne,, Katherine Zacarian, and Damián E. Blasi. Using machine learning to decode animal communication. Science, 381 (816645): 2023. doi: 10. 10.1 URL
1126/science adg731
J Salamon and JP Jacoby, Cand Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, 2014.
D Santos- Dontoo-DDmmiguez, S forres-Guijant A Cardenal- -óéez, and A Pena-Gimente ShipsEar: An underwater vessel noise database. Applied Acoustics, 1 113:64 69, 2016.
15

---

Published as a conference paper at ICLR 2025
E. Sarkar and M. Magimai-Doss Can self sulf-super neural representations pre-trained on human speech distinguish animal callers? Proceedings of Interspeech, pp. 1189-11999 1193, 2023, doi: 10.21433/Imterspeech..22 202-:/ecchhhh211990e 1968.
Laela Sayigh, Mary Ann Daher, Julie Allen, Helen Gordon, Katherine Joyce, Claire Stuhlmann, and Peter Tyack. The Watkins marine mammal sound database: An online, freely accessible resource. Proceedings of Meetings Acoustics, 27(1):440013, 2016. doi: 10.1121/2.0000358. URL
Nacha Sharif, Lyndon White, Mohammed Bennamoun, and Syed Afaq Ali Shah. Learning-base composite metrics for improved caption evaluation. In Proceedings of ACL 2018, student research workshop, pp. 14-20, 2018.
Ocean Networks Canada
Society.
Upper
slope
south
hydrophone deployed
2013-05-11.
Data
set, 2013.
URL
d644336d -b3ee---e-e----- 4b50 b2ef- 3d8ddddd
Ocean Networks
Canada Society.
Upper
slope south hydrophone deployed
2014-05-03.
Data set. 2014a.
URL
a
91 0b33e497734
Ocean Networks
Canada Society.
Upper
slope
south hydrophone deployed
2014-05-07.
Data set, 2014b.
URL https://doil http:///oo..r011.3344444
7be 12be925c-de7e0 1ilee1--0dd6 1eeo6-a78c77ffaeee
Petru Soviany, Radu Tudor Ioneseu, Paolo Rota, and N. Sebe. Curriculum learning: A survey.
International Journal of Computer Vision, 130:1526 1565, 2021. URL https://api.
Gemanticscholar. otg/CorpusID: :217099299
Samuel Stevens, Jiaman Wu, Matthew Thompson, Elizabeth Campolongo. Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Nerger-Wolly Wei Chao, and Yu Su. BioCLIP: A vision foundation model for the tree of life. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19412-19424, 2024.
Dan Stowell. Computational bioacousties with deep learningg a review and roadmap. PeerJ, 10: c13152, 2022. ISSN 2167-8359, doi: 10.77177peerj 13152. URL htttps://eurppat
org/articles (tt.artcclee//MM4442111
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview. net. fow.rorrmmmmm11mnmnomooor
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. Proceedings of Meetings on Acoustics, 19(1) 05 2013. ISSN 1939-800X. doi: 10.1121/ 1.4799597. htLh:t:s::/ //t:://////////// wrg. 1121/1. 4799597
Willem-Pier Vellinga and Robert Planqué. The xeno-canto collection and its relation to sound recog- nition and classification. In CLEF (Working Notes), 2015.
Mingqiu Wang. Izhak Shafran, Hagen Soltau, Wei Han, Yuan Cao, Dian Yu, and Laurent El Shafey. Retrieval augmented end-to-end spoken dialog models. In Proceedings of the International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12056-12060. IEEE, 2024.
Gordon Wichern, Joe Antognini. Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. In Proceedings of Interspeech, September 2019.
Jian WU, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang. Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, al. On decoder-only architecture for speech-to-text and large language model integration. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEFF, 2023a.
16

---

Published as a conference paper at ICLR 2025
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Kirk-airka and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caplie augmentation. In Proceedings of the International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023b. doi: 10. 100100/ASSS492552222000000
Xeno-canto. Xeno-canto: Bird sounds from around the world. https:://ww... xeno canto.
org/ URLbttps:: wwwp://wwwww..mmm www.- org/ Accessed 2023-05-15.
Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. CSTR VCTK Corpus: English
multi-speaker corpus for CSTR voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR), pp. 271-350, 2019.
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. AIR-Bench: Benchmarking large audio-language models via generative comprehension. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers). pp. 1979-1998, Association for Computational Linguistics. August 2024. doi: 10.18653/v1/2022 wwl.aaal.--oo..000 URL https: http://aaaaaattotaccoee org/2024. ac1 oog.on.................... 109
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from librispeech for text-to-speech. In Proceedings of Interspeech 2019, pp. 1526- 1536,153,22200 2019.
Dong Zhang. Shimin Li, Xin Zhang. Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abil- ities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15757-15773. Singapore, December 2023. As- sociation for Computational Linguisties. doi: 10.18653/v 1/2023 1100: findingss-emll.10055. URL https://accle http:s/////llaahhollggy wtg./200....g20....///// f Ending--em 1055/
Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, et al. SpeechLM: Enhanced speech pre-training with unpaired textual data. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
A APPENDIX
A.1 HELD-OUT FAMILIES
1. Elachuridae
Alytidae
Prophalangopsidae
Calyptophilidee
Castoridae
Octodontidae
3. Pelecunedidaee
Dienglosiddie
4. Phocoenidie
8. Suidue
A.2 HELD-OUT GENUS
1. Aglocutiis
18.
Hymeoops
35. Leucocurbo
Deeanortgacchs
19.
Doliorais
Gymnoptape
3. Loshia
20.
Fugerygone
Goldmania
4. Nemobius
21.
Cryplayyvioola
Oncomyats
Meconena
22.
Thaniopygia
Rhoddstthhi
Preudochorthigrus
Cathampeza
Faleipennis
7. Callectthras
24.
Eurostopodus
41. Pachyroccyx
8. Pachyaare
25.
Tylas
42. Cryptotympnn
Rhecisthrapii
26. Vini
43. Tymponostann
Astraiia
Ptychoramphes
11. Proiosscgger
28. Specitaaas
44. Cystoxipha
12. Amaronetia
29. Apheloospala
45. Afrixalus
13. Oeyalus
30. Stipituras
46. Uperoleia
14. Nandayas
31. Procanduelis
47. Urocitellus
15. Rhinorrypta
32. Respophilus
48. Chaloorana
16 Hoterocesuus
33. Nospsephotus
49. Aiolopus
17. Zacamursleyen
34. Findes
Speathos
17

---

Published as a conference paper at ICLR 2025
A.3 HULD-OUT SPECIES
1. Aethogyga shelleyi
68.
Butco japeniess
135.
Ninox natalis
2. Arachnothera dilutior
69.
Apus brodieldi
136.
Myrmelasees caurensis
3. Sitta castanca
70.
Picrocies personatuus
137.
Ruteo trizonatus
Carpodaccs rodopeplus Acthopyga ignicauda Pachyoophala
71.
Meliniporus fringillius
138.
Apalis chariessa
72.
Poccile hypermelaems
133.
Ficedola nigrorufa
73.
Ciross buffoni
140.
Pica mauritanica
Herpsilochmus romimae
74.
Pyenonouus blanfordi
141.
Antheepess reichenowwi
Amazona dufrestians
75.
Machlolophus aplonoins
142.
Sholicola mujor
9. Metallura amnoccauaa
76.
Estrlda ochrogaster
143.
Vireo osbumi
10. Thiypapiis fulvicops
77.
Truit Ivttvicas
144.
Anas capensis
11. Monarcha frater
78.
Marafra gillen
145.
Dacula lactuosa
12. Kleinofraspis reyi
79.
Piernistis ieterorhynchus
146.
Lomius newtoni
13. Aplenis magna
80.
Accipiter collaris
147.
Odonngthorss dialenens
14. Phylloscapus misariencis
81.
Knipolegus lophotes
148.
Bostryhhia olivacea
15. Apapornis pullarins
82.
Nothoproeta taczanewskil
149.
Canyrii teavoensis
16. Amazona versicolor
83. Pachycsphala modesta
150.
Placeus heaglini
17. Saliator cinztus
84. Vanellus tricolor
151.
Myoomela nigriia
18. Xiphucolapies falcinssris
85.
Caprimalgus andammaiiuus
152
Faloo chermg
19. Passer insulais
86.
Ardenna grisen
153.
Ixobeyclus sturmii
20. Chaloomitt halfiarri
87.
Mixornia kelleyi
154.
Rhipidura semirubea
21.
Arremonops tocuyensis
38.
Cinnyris johannae
155.
Haematoqus chathamensis
22.
Allapetes meridie
Recurvinosraa novachollandise
156.
Amtus benchyarus
23.
Collumiincia obscura
90.
Silta leucopsis
157.
Ocnanzbe lugens
24. Saltator maxillosus
91.
Perroica pusilla
158.
Columba rupestris
25. Philemon meyeri
92.
Amazilia lucine
159.
Rlyticrrss subcutoollis
26. Thamsophils insignis
93. Melaniporus fasciiventer
160.
Zosterops vellalavella
27. Aulacurtynchus whiteliamus
94.
Egretia picata
161.
Amths aokokensis
28. Sirystes whheanescens
95.
Columba pollenii
162.
Phacthornis idaliae
29. Sporphila nigromufa
96.
Rallus masagascariomis
163.
Picus dedemi
30. Zoother motlissima
97.
Heliadora gularis
164.
Muscicagu segregata
31. Thlypupsis inomaia
98.
Carpodacus
165.
Cyanomitra bannermani
22. Picumnus spilogaster
99.
Zesterops chioronothos
166.
Polispila facilis
33. Columis arquatrix
100.
Pachyeephala kornaci
167.
Platsteiraa albifrous
34. Petrochelidan iffcoolaris
101.
Sunceroria eyanem
168.
Dicaeum pygmarum
35.
Pyrrhura griseipectes
102.
Cinclesoma marginatum
36.
Myiothilpis cheysogaster
103.
Basco noatnmae
169.
Pullinus assimilis
37.
Thripophaga amacurensis
Conhia nipalensis
170.
Rhipicura kuhuryi
38. Hespsitocheuus motacilloodes
105.
Pachyoophulla lanioides
171.
Placess katangae
39. Progne dominicemii
106
Carpolacus trifasciatos
172.
Canis lupaster
40. Helindota bransckii
107.
Chorthipuu acroleucus
173.
Hyls andersonii
41. Asthenes arequipse
108.
Chlidonias albostristus
174.
Kanaidea uudidigita
42. Gerygone fusca
109.
Hinundo domicola
175.
Ranoidea aurea
43. Otus thilcholTmanni
110,
Falco concolor
176.
Litoria ryleri
44. subitava
111.
Drycoapus schalzzii
177.
Dendropscphus joannae
45. Charadrius montanus
Rhyticerus undulatus
178.
Okanganu accidentali
46. Petriica polymorpha
Quiscalus nicargueenii
179.
Litoria latopalmant
47.
Symposiachus vidua
114,
Cisicolla brunnesoens
180.
Magacicada tredecassini
48.
Dicnuras loyhorimas
115.
Knipolegus cynairostris
181.
Orchelimum silvaticum
40.
Pyenoneeua peniciTlauss
116.
Ardenna cameipes
182.
Occanthus celerinictus
30.
Melmnerps herminseri
117.
Lyboas rubrifacies
183.
Empidanonss ammttiattreriiitts
51.
Zostenops mysorensis
118.
Climacieris melamurus
184.
Dufotes boulengeri
52.
Oenanthe kanthoprymaa
Putlinus upisthomelas
185.
Occanthus nigricomis
53.
monachus
Manceina melanotis
186.
Myrmothera fulviventis
54.
Caprimmigus pulchellus
Celebesica abbotti
187.
Psaltoda adonis
55.
Psarocolius cassini
122.
Otus mayotiensis
188.
Rana dalmatinn
56.
Symposiactrus infelix
123.
Trachyphonus margaritatus
189.
Dendropscphus sanbomi
57.
Zostenoss cinereus
124.
Oenamthe duhia
190.
Hlperolius sticus
58.
Cireus cineress
125.
Chlompii flovigennis
191.
Hyperolins pictus
59. Georrygon chrysia
Plocess alienus
192.
Hyla eximia
60. Microspingus irifasciatus
127.
Phalacrocorax vurius
193.
Leptedactylos natalemsis
61. Piemnitiss harwooii
Ploceus polzeni
194.
Occanthus californieus
62. Coblepyris caesius
129.
Menops mentalis
195.
Hyperolnns parallehas
63. Ficedula disposit
130.
Passer gongenensis
196.
Grylka cohni
64. Treron aftinis
131.
Myzomela cineracca
197.
Pryseier macnecephalus
63. Geokichla wardii
192.
Packyeophala feminina
198.
Eleomherddccyluu unicolor
66 Campethera bennetzii
133.
Brachypeerys simencis
199.
Grylkss bemuudensis
67. Aleedo somitoqquata
Lanchura flacppeymn
200.
Anis pencloge
18

---

Published as a conference paper at ICLR 2025
A.4 EVALUATION ON BIRDSET
We evaluate Top-1 accuracy on the datasets from the BirdSet benchmark. To match other models evaluated on BirdSet, which are constrained to predict one of the allowed labels, we use coss-based classification across all datasets and make predictions using scientific names. Our model achieves the highest average Top-1 accuracy, slightly surpassing Perch, demonstrating strong generalization from primarily focal recordings to soundscape recordings, and state-of-the-art performance for re- trieval and classification on real-world bird datasets.
POW
PER
NES
UHH
HSN
NBP
SNE
AVG
EffNet
0.80
0.38
0.49
0.42
0.59
0.63
0.67
0.57
ConvNext
0.75
0.36
0.45
0.44
0.52
0.64
0.65
0.54
AST
0.79
0.40
0.48
0.39
0.48
0.61
0.57
0.53
EAT
0.69
0.32
0.46
0.40
0.47
0.61
0,58
0,50
W2V2
0.72
0.34
0.47
0.51
0.50
0.65
0.51
0.53
Perch
0.85
0.48
0.66
0.57
0.58
0.69
0.69
0.65
Naturel
0.95
0.62
0.47
0.60
0.58
0.66
0.76
0.66
Table 7: Top-1 Accuracy results for each method on the datasets BirdSet. Refer to the original paper (Rauch al. 2025) for the details of compared baseline models.
A.5 SPECIES CLASSIFICATION WITH ADDITIONAL CONTEXT
We evaluate whether VatureLM-audhe can improve species classification performance by incorpo- rating additional context as text. The CBI dataset (Howard et al.) (020)) derived from Xeno-canto, often contains metadata such as location and free-text notes written by recordists. We evaluate the model under three conditions: using audio alone, adding metadata (latitude, longitude, altitude when available, and geographic region), and further incorporating free-text notes. The model achieves an accuracy 0.776 with audio alone, 0.792 with additional metadata, and 0.798 with both meta- data and free-text notes, demonstrating that providing additional textual context can improve audio classification performance.
A.6 CALL TYPES AND TRANSFER
Configuration
call-song
multi
call-song-unseen multi-unseen
M--urio
0.871
0.667
0.769
0.678
BioL.ingual
0.658
0.303
0.665
0.419
Table 8: Accuracy of call vs. song classification (call-song), multi-call classification (multi), and the generalization of these tasks to unseen taxa (call-song-une) song-unseen, multi-unen unseen.)
We further evaluate the classification of bird call types and the transfer of this task across species. We test the model on call vs. song prediction well as call-type prediction for multiple classes (call, song, flight call, alarm call, begging call, and drumming). We then test these tasks can be transferred to unseen taxa. The call-song-unseen and multi-unseen datasets evaluate the same tasks described above, but evaluated on the held-out taxa used to test unseen species, unseen genus, and unseen family. In addition achieving state-of-the-art results these tasks, the results trans- fer strongly to unseen taxa, outperforming BioLingual- even when these taxa were held out from Naturel M M-audio but not from BioLinguat
19

---

Published as a conference paper at ICLR 2025
A.7 ABLATION ON UNFREEZING BEATs
Configuration
watkins
cbi
unseen-species
unseen-genus uuseen-family
0.723
0.680
0.320
0.124
0.306
BEATs-frozen 0.490
0.401
0.184
0.073
0.186
Table 9: Zero-shot classification results with BEATs unfrozen vs. frozen. Both models are trained on stage-1 tasks for 150k steps. We report accuracy on species classification tasks, with unseen taxa tasks predicted using taxonomic names.
A.8 SPEECI++MUSIC ABLATION: FULL RESULTS
Model
esc50
watkins
cbi
humbugdh
dease
enabirds
hiceas
rfex
gibhons
base
0.570
0.788
0.748
0.093
0.107
0.299
0.415
0.038
0.011
base w/o speech or music
0.605
0.773
0.750
0.152
0.040
0.293
0.417
0.038
0.012
Table 10: Zero-shot classification and detection results on BEANS- Zero. Base model was trained on all 2 training tasks, while "base w/o speech or music" is an ablation removing both speech and music tasks from training data. Both models were trained for 150k steps. We used accuracy classification, and FI for detection tasks.
Model
unsecn-species
unseen-genus unseen-family
lifestage
call-type
captioning
zf-indv
base
0.322
0.139
0.239
0.702
0.863
0.501
0.677
base w/o speech or music
0.354
0.137
0.330
0.690
0.852
0.503
0.500
Table 11: Zero-shot results on new tasks introduced in BEANS- Zero. Base model was trained on all stage-2 training tasks, while base w/o speech music is ablation removing both speech and music tasks from training data. Both models were trained for 150k steps. We report accuracy for classification, and SPIDEr (Sharif et al. 2018) for captioning.
20

---

Published as a conference paper at ICLR 2025
A.9
TRAINING DATA
Herarrhy Distr bution (Log-Scaled)
LPPA
LIS SANTS
PACCAAA 1991 (991
Figure Data composition across training samples including the distribution for the main data types and phylum, class, and order for non-human animals. The counts represent prompts rather than audio files i.e. various prompts may be derived from the single audio file.
21


Reviews:
[{"review_id": "81N8UcPxHO", "reviewer": "Reviewer_GfHx", "summary": "An excellent paper providing a valuable source of bioacoustic data and processing methods to the community. An attempt to comprehensively collect much of the available domain data and curate it into a usable dataset. Plans for distribution and general availability of the dataset/code to the community are missing.", "questions": "This is excellent and careful work. I particularly liked the SoTA results in classification", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "An incredible collection of datasets, and careful curation.\nA lot of ancillary code for use of the data in various learning tasks.", "weaknesses": "Unclear from the presentation if the authors intend to make the dataset widely available, and under what license.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FemM7tXA9t", "reviewer": "Reviewer_Lfuo", "summary": "The authors present a large audio-language model (Lalm) trained and evaluated on bioacoustics data. First, a dataset of text-audio pairs is compiled, partially with the use of LLMs. This dataset is then used to train an architecture similar to SALMONN. It is consisting of a LLM (Llama 3.1-8b), an audio encoder (BEATs), and a Q-Former to connect the computed audio embeddings to the LLM. During training the LLM stays frozen while the audio encoder and Q-Former are trained after a curriculum (first species classification than also detection, captioning, life stage prediction, call-type detection). The model is evaluated on the self introduced BEANS-Zero benchmark which extends BEANS with additional tasks and a zero shot evaluation protocol. The model achieves the best results compared to non bioacoustics", "questions": "1. Could you add ablation studies to access the influence of each part of your approach? What happens if the audio encoder stays frozen? Are the embeddings that BEATs generates already good enough for taking bioacoustics tasks?\n2. What SoTA model is used in the call-type comparison? I could not find that in the text.\n3. How do you evaluate the LLM without audio and why is it outperforming your model on the gibbons dataset?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Addresses a important topic from both the ML research community ( since audio and especially computational bioacoustics is a hard problem) and societal importance.\n2. Collects a comprehensive training dataset and extends an existing evaluation benchmark  with additional tasks.\n3. The performance improvements compared to a model not trained on bioacoustics data (SALMONN) supports the claim that this domain is in the need for a own foundation model.", "weaknesses": "1. Soundness of results: Your presented results only show a minor improvement compared to BioLingual (which also presents zero shot results on BEANS, there numbers differ sometimes why?), so whats the benefit of your approach and more particularly does integrating a LLM has a benefit? Or is it the different training dataset? Or the audio encoder (BEATs vs. HTS-AT)?\n2. No further details for replication of the experiments are given, e.g. pretrained models or the list of species which were hold out.\n3. Difficult covariate shift from focal training to soundscape test data is not evaluated, which is one of the major challenges in bioacoustics. E.g. for birds see Stowell,2022 or BirdSet,2024.\n4. Call-type task: birds usually have more than two call-types per species. A binary classification task ignoring the specie might have few practical applications.\n5. L. 428-219: The cbi dataset consists of XC recordings, your model should have the same advantage as BirdNet and Perch, so the comparison should be fair.\n6. Line 071-072: How do you support the claim that the generalization of BirdNet is limited? Did not Ghani et al. claim the opposite?\n\n---\n\n### Language\n\n1. l. 190-192 strong repetition\n2. Inconsistent use of abbreviations (e.g. state of the art)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lRUJ35mP84", "reviewer": "Reviewer_1ReQ", "summary": "The paper introduces NatureLM, an audio-language model specifically designed for bioacoustics, focusing on animal sounds. NatureLM leverages a diverse dataset combining bioacoustics, human speech, and music to enhance generalization across various species and tasks. NatureLM demonstrates strong in-context learning capabilities, enabling zero-shot classification for unseen species.\n\nAdditionally, the paper presents BEANS-Zero, a benchmark for bioacoustics that includes several tasks beyond species classification, such as call-type prediction, life-stage classification, and individual counting, aiming to push the boundaries of bioacoustic research.", "questions": "1. In Section 3.1.1, the method for hard negatives sampling for species detection is briefly mentioned, but the details are unclear. Could you provide a more detailed explanation of the strategy for sampling these hard negative samples?\n\n2. Could you provide more details in Section 3.2 about the selection strategy for the held-out species in the AnimalSpeak dataset for the unseen-cmn and unseen-sci subsets? Even if the selection was random, it's important to clarify the process.\n\n3. In Section 4.5, you state that including speech and music in training has a positive impact on counting zebra-finch individuals. However, while the ablation study includes speech, it is unclear whether music was also ablated. Could you clarify whether the effect of music was tested? Additionally, conducting ablation on all downstream tasks, not just individual identification, could provide more comprehensive insights into whether speech and music data enhance performance across other tasks as well. This could help clarify which types of training data are most beneficial for specific applications.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of NatureLM, the first audio-language model specifically designed for bioacoustics, represents a promising new direction for incorporating language models into biodiversity monitoring.\n\n2. The development of the BEANS-Zero benchmark extends the original BEANS benchmark by introducing new tasks, such as call-type prediction, life-stage classification, individual counting, and open-ended audio captioning. These additions have the potential to advance bioacoustics research and enable more detailed acoustic monitoring of species.\n\n3. The paper is easy to follow, and the related works are thoroughly referenced.", "weaknesses": "1. Incorrect Terminology\n\n\n1.1.  The introduction describes BioLingual as self-supervised; however, the supervision is derived from text generated based on class labels. I recommend referring to it as supervised learning with language-based supervision for greater clarity and accuracy.\n\n2.1. Both BioLingual and AVES are described in the paper as foundation models, but this classification may be misleading. BioLingual and AVES are trained on datasets with less than 2 million samples, while models trained on AudioSet with 2 million samples are not typically considered foundation models. BioLingual is evaluated on classification and retrieval tasks, while AVES is evaluated on classification and detection tasks. Typically, a foundation model is a very large model trained on extremely large datasets so that properties emerge that enable it to handle a wide variety of tasks across different domains. For improved clarity, I suggest either using alternative descriptions like self-supervised (for AVES) and audio-language contrastive (for BioLingual) to more accurately describe them, or providing a clear definition of what you consider to be a foundation model if you wish to retain this terminology. This will help avoid confusion and ensure that readers more precisely understand the capabilities of these models.\n\n2. Overstatement of Results\n\n\n2.1. The analysis of NatureLM-audio's performance on the cbi dataset (Table 4, Section 4.2) is potentially misleading due to data overlap. Since BirdNet, Perch, and NatureLM-audio are trained on Xeno-Canto, and the cbi evaluation dataset is a subset of Xeno-Canto, there is overlap between the training and evaluation data. This compromises the claim of state-of-the-art performance, especially that Perch significantly outperforms NatureLM in enabirds.\n\n2.2. In Table 5, Section 4.3, the authors claim that NatureLM-audio demonstrates generalization to completely unseen species by outperforming CLAP, a contrastive audio-language model trained on non-bioacoustic data. However, this conclusion seems too strong. The results primarily show that a model trained on bioacoustic data performs better than one trained on non-bioacoustic data, which does not necessarily indicate true generalization to unseen species. Additionally, BioLingual, which was trained similarly to CLAP but on bioacoustic data, significantly outperforms NatureLM-audio. I suggest rephrasing the conclusion to more accurately reflect these findings.\n\n2.3. In Section 4.4, the paper claims state-of-the-art performance in the captioning task. However, this comparison is based solely on the SALMONN model, which was not trained on bioacoustic data and is not specifically designed for this task. To support the claim of state-of-the-art performance, I recommend including a comparison with a captioning model trained on bioacoustic data, as this would provide a more accurate and meaningful evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)