NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics

## Abstract

Abstract Large language models (LLMs) prompted with text and audio represent the state of the art in various auditory tasks, including speech, music, and general audio, showing emergent abilities on unseen tasks.
However, these capabilities have yet to be fully demonstrated in bioacoustics tasks, such as detecting animal vocalizations in large recordings, classifying rare and endangered species, and labeling context and behavior—tasks that are crucial for conservation, biodiversity monitoring, and the study of animal behavior.
In this work, we present NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics. Our carefully curated training dataset comprises text-audio pairs spanning a diverse range of bioacoustics, speech, and music data, designed to address the challenges posed by limited annotated datasets in the field.
We demonstrate successful transfer of learned representations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks.
Importantly, we test NatureLM-audio on a novel benchmark (BEANS-Zero) and it sets the new state of the art (SotA) on several bioacoustics tasks, including zero-shot classification of unseen species.
To advance bioacoustics research, we also open-source the code for generating training and benchmark data, as well as for training the model 1 1 1 Demo page: https://earthspecies.github.io/naturelm-audio-demo/ The code will be open-sourced and available shortly. .

## 1 Introduction

Bioacoustics, the study of sound production and reception in animals, aims to understand animal behavior (Fischer et al., ), monitor biodiversity (Stowell, ), and model the mechanisms of sound production and reception used in animal communication (Bradbury & Vehrencamp, ).
It plays a vital role in conservation and ecological research, as animal vocalizations provide critical insights into ecosystem health, species interactions, and population dynamics. By enabling the detection of endangered species and tracking migration patterns, bioacoustic research directly contributes to biodiversity monitoring and conservation efforts (Rutz et al., ; Stevens et al., ).

In recent years, machine learning has taken on an increasingly pivotal role in bioacoustic research. Beyond its applications in large-scale ecological monitoring, machine learning has also opened up new frontiers in the study of animal communication, enabling discoveries like the ability of marmosets (Oren et al., ), dolphins (King & Janik, ), and elephants (Pardo et al., ) to use specialized vocalizations to label their conspecifics. Yet, because of obvious data collection and annotation difficulties, these studies often rely on strongly labeled small datasets (Stowell, ) and thus require careful statistical analysis to measure the significance of results and avoid over-fitting. At the same time, large volumes of unannotated bioacoustics data are recorded daily, particularly through passive acoustic monitoring (PAM,  Dufourq et al. ()) or citizen science platforms e.g. Xeno-canto (Vellinga & Planqué, )). There is thus a growing need for machine learning tools capable of performing tasks such as detection, classification, and annotation on these data at scale. The recent successes of large scale artificial intelligence models in various domains (e.g. natural language processing, vision, games) also point to the possibility of leveraging these huge volumes of raw data to learn accurate and generalizable representations of bioacoustics signals (Ghani et al., ; Boudiaf et al., ).

Existing bioacoustics machine learning models are typically designed for specific species or tasks (Dufourq et al., ; Kahl et al., ; Cauzinille et al., ), showing limited generalizability beyond their predefined scope. Many traditional studies rely on small datasets focused on a few species and individuals, validating results through statistical measures despite over-fitting risks. Newer models such as BirdNET (Kahl et al., ) and Perch (Ghani et al., ) perform well in specific tasks such as bird classification but still generalize poorly outside avian species. Recently, foundation models such as AVES (Hagiwara, ) and BioLingual (Robinson et al., ) have exhibited notable results using self-supervision, though they remain constrained by their training paradigms (discriminative and contrastive, respectively), which restrict the range of tasks they can address.

In recent years, foundation models, which learn patterns in large amounts of data via self-supervision, have shown promising performance across a wide range of tasks (Bommasani et al., ). While transformer-based large language models (LLMs) are currently the most prominent examples, other architectures, such as diffusion models (Kingma et al., ), are also emerging as foundation models in some domains. These models’ ability to handle unseen tasks, perform in-context learning, and respond to prompts positions them as a compelling alternative to traditional machine learning methods, which often rely on laboriously annotated data, expensive computational resources, and often-lacking machine learning expertise.

While multimodal large language models (LLMs), particularly vision-language models (VLMs), have been explored for biodiversity and conservation research (Miao et al., ), there is relatively little effort dedicated to building and investigating large audio-language models (LALMs) for bioacoustics. LALMs have shown significant promise in processing human speech (Rubenstein et al., ; Wang et al., ; Wu et al., ; Zhang et al., ), music (Gardner et al., ; Agostinelli et al., ), and general audio tasks (Tang et al., ; Chu et al., ; Gong et al., ), and they hold the potential to bring transformative advancements to bioacoustics as well.

In this paper, we present NatureLM-audio, an audio-language foundation model specifically designed for bioacoustics tasks, including classification, detection, and captioning. To the best of our knowledge, NatureLM-audio is the first model of its kind.
Inspired by the cross-taxa transfer observed in previous research, such as between human and gibbons (Cauzinille et al., ) and birds and whales (Ghani et al., ), we incorporate speech and music tasks into the training process. We show that representations learned from these domains successfully transfer to animal vocalizations, demonstrating generalization across species.
Importantly, we augment an already existing animal sounds classification and detection benchmark, BEANS (Hagiwara et al., ), with additional tasks such as call-type prediction, lifestage classification, captioning, and individual counting. With these, we test cross-domain learning capabilities of the model and zero-shot transfer to unseen taxa and tasks. We name this new benchmark BEANS-Zero.

Our contributions are thus as follows:

- •
Model: We introduce NatureLM-audio, to the best of our knowledge, the first audio-language foundation model for bioacoustics with carefully curated training datasets comprising of animal vocalization, human speech, and music.
- •
Domain transfer We show that the model transfers beyond the species originally trained on and demonstrate its zero-shot capability on unseen taxa and species.
- •
Task transfer We test our model on a novel benchmark (BEANS-Zero) that goes beyond species classification and even includes a completely unseen task (individual counting).For the first time, we show positive transfer from speech and music data to bioacoustics tasks.

## 2 Related Work

Most prior work on audio-language models has focused on human speech processing. For example, models like SpeechGPT (Zhang et al., ), Speech-LLaMA (Wu et al., ), AudioLM (Borsos et al., ), AudioPaLM (Rubenstein et al., ), AudioGPT (Huang et al., ), SpiRit-LM (Nguyen et al., ), and SpeechLM (Zhang et al., ) mostly focus on building language models that can perceive and produce human speech.
Such models may be fine-tuned for downstream bioacoustics tasks requiring expensive computational resources and expertise. Instead, our model shows promising generalization to unseen species and tasks.

Recently, more generic language models with audio perception capabilities have been released. Pengi (Deshmukh et al., ) uses an audio encoder and a text encoder mapped onto an LLM to solve audio-to-text tasks. SALMONN (Tang et al., ) uses dual audio encoders and integrates Q-Former (Li et al., ) to improve the handling of speech and general audio inputs. Qwen-audio (Chu et al., ) adopts a multi-task learning approach with the introduction of the Speech Recognition with Timestamp (SRWT) task. LTU (Gong et al., ) builds an open-ended question-answer dataset and uses curriculum learning strategies to enhance its generalization capabilities. Similar multimodal language models have been proposed for music, such as MU-LLaMA (Liu et al., ) and LLark (Gardner et al., ).
Recent foundation models such as AVES (Hagiwara, ) and BioLingual (Robinson et al., ) have exhibited notable results on bioacoustic tasks, although their training paradigms and architectures restrict the range of tasks they can address.

Although animal sounds and vocalizations are often part of generic audio datasets, such as AudioSet (Gemmeke et al., ) and audio caption datasets (Kim et al., ; Mei et al., ), these datasets are often too general and lack the fine-grained details necessary for tasks like species classification, behavior analysis, or monitoring in ecology and bioacoustics. As a consequence, LALMs trained on these datasets produce at best generic labels e.g., ‘bird’ and not the name of the species. We address this limitation by proposing an open multi-task diverse training set and a LALM, NatureLM-audio, that offers robust representations for bioacoustics.

While there are specific bioacoustics benchmarks like BIRB (Hamer et al., ) for bird vocalization retrieval and BEANS (Hagiwara et al., ) for classification/detection, the field of bioacoustics has yet to see the development of dedicated benchmarks similar to those in human speech and music, such as Dynamic-SUPERB (Huang et al., ) or AIR-Bench (Yang et al., ). This leaves a gap for advancing the evaluation of bioacoustics models, particularly in zero-shot learning and task generalization.

With this work, we aim to bridge these gaps by introducing NatureLM-audio, a model specifically designed for bioacoustics tasks, and enhancing bioacoustic benchmarks to assess cross-species and cross-task generalization, introducing BEANS-Zero.

## 3 Methods

### 3.1 Training Dataset Creation

**Table 1: Training tasks and datasets ^a CLS: classification, DET: detection, CAP: captioning**
| Task^a | Dataset | # Hours | # Samples |
| --- | --- | --- | --- |
| CAP | WavCaps (Mei et al., ) | 7568 | 402k |
| CAP | AudioCaps (Kim et al., ) | 145 | 52k |
| CLS | NSynth  (Engel et al., ) | 442 | 300k |
| CLS | LibriSpeechD  (Edwards et al., ) | 156 | 16k |
| CLS, DET, CAP | Xeno-canto  (Vellinga & Planqué, ) | 10416 | 607k |
| CLS, DET, CAP | iNaturalist () | 1539 | 320k |
| CLS, DET, CAP | Watkins  (Sayigh et al., ) | 27 | 15k |
| CLS, DET | Animal Sound Archive () | 78 | 16k |
| DET | Xeno-canto-detection  (Vellinga & Planqué, ) | 2749 | 670k |
| DET | Sapsucker Woods  (Kahl et al., ) | 285 | 342k |
| DET | Sierra Nevada  (Kahl et al., ) | 61 | 22k |
| DET | University of Hawai’i at Hilo  (Navine et al., ) | 94 | 34k |

Figure: Figure 2: Examples of training instances
Refer to caption: x2.png

To train an audio-text model for bioacoustics, we compile a diverse dataset of text-audio pairs (Table 1). The data is collected through a combination of prompting on existing audio datasets, creating new LLM-generated text labels, and mixing new, procedurally-augmented audio data. The data is comprised of bioacoustic audio, general audio, speech, and music datasets. Figure 2 shows some examples of instances used for training NatureLM-audio.

#### 3.1.1 Bioacoustic Data

Species Classification:
We curate existing large-scale bioacoustic archives into a common format. We curate Xeno-Canto (), iNaturalist (), Animal Sound Archive (), and Watkins (all-cuts, Sayigh et al. ()) into a common format. Specifically, we handle differences in common name and scientific name across datasets by joining all datasets to the GBIF taxonomy backbone (GBIF Secretariat, ). We then prompt the model to predict either the scientific or common name of the focal species, or the scientific or common names of all species in the recording. This requires the model to generate the common name or scientific name of the species directly. In many cases, we may know an animal vocalization is one of a subset of species—for example, based on location. To allow for this, we also generate prompts with a set of options injected into the question. The options may be sampled randomly from other species, or they may be “hard negative” species from the same genus, family, or order.

Species Detection:
We use the same datasets as for species classification, but prompt the model to ask whether the recording contains one of a set of options, or ‘None’. Options are sampled in the same way as for classification, with a mix of random and hard negatives. In fifty percent of prompts, the correct species is not included in the set of options, with a correct answer of ‘None’.

To help bridge the gap between focal train recordings and noisy soundscape recordings common at inference, we also generate a noise-augmented detection training set from Xeno-canto. We use per-channel energy normalization (PCEN Lostanlen et al. ()) as a form of noise-gate for bird vocalization activity detection. Then, we separate each detected segment into four stems using the 4-stem Bird-MixIT source separation model (Denton et al., ). Because the separation model may over-separate sources and does not label stems with source names, we use the YAMNet model (Howard et al., ) trained on the AudioSet dataset (Gemmeke et al., ) to select solely the stems with high probability on the AudioSet animal classes (with ids between 67 to 131). Correspondingly, for each stem we take the maximum probability across the classes, we average the values across the time frames, and we sum the stems with values higher than 0.5.

In addition, we used soundscape recording datasets from Sapsucker Woods (SSW, Kahl et al. ()), Sierra Nevada (SNE, Kahl et al. ()), and the University of Hawai’i at Hilo (UHH, Navine et al. ()) for detection tasks. Following the approach used in the detection datasets from BEANS, we split the audio into 10-second windows with a 5-second overlap, and treated it as a multi-label classification problem. Species with more than 100 occurrences were selected as target labels, while species with fewer occurrences were grouped into an “other” class.

Captioning:
We use the AnimalSpeak (Robinson et al., ) dataset for bioacoustic captioning. AnimalSpeak combines bioacoustic datasets into a language-model-captioned audio-text dataset. However, due to scale, the large segment of AnimalSpeak from Xeno-Canto was not captioned with a language-model, and used only templated captions. We further process Xeno-Canto with Gemini-1.0-pro (Gemini Team, ) following the same method used to create AnimalSpeak, and use these LLM-generated captions in addition to the original captions.

Call-type and Lifestage:
We include multiple new bioacoustic tasks which can be expressed based on the Xeno-Canto metadata. Specifically, predicting the life stage of birds, predicting call-types, and differentiating between calls and songs. Compared to species classification alone, included in existing datasets, the ability to perform these tasks at scale could significantly enhance the precision of ecological monitoring and behavior studies.

#### 3.1.2 General Audio

We include WavCaps (Mei et al., ) and AudioCaps (Kim et al., ) for general audio captioning. We observe that, in the creation of WavCaps, some recordings originally had metadata relevant to bioacoustics and specific species. However, these were lost in the general-domain captioning, producing captions which are too generic for our purpose. We detect these cases by processing the original metadata, and re-process the metadata prompting Gemini-1.0-pro to produce bioacoustic captions. We include these new bioacoustic captions in addition to the original captions.

#### 3.1.3 Music

Pitch, timbre qualities of animal vocalizations, the number of animals in a recording are often key acoustic features used by biologists to classify context and behavior.
We use NSynth 2.3.3 (Engel et al., ) to create a set of tasks that may help bioacoustics downstream tasks.
We generate text prompts for pitch detection in Hz, instrument name, and velocity, ranging 0 to 1. Additionally, we use the timbre ‘qualities’ labels to create text descriptions for each audio. For instance, if the sound is ‘distorted,’ we generate descriptions such as ‘This sound has a distinctive crunchy sound and presence of many harmonics.’ or ‘This sound is distorted’.
Moreover, we create synthetic mixtures by layering one to three different instruments. In this case we generate, two task: predicting the number of instruments and identifying the instrument names.

#### 3.1.4 Speech

We use the speech diarization dataset based on LibriSpeech (Edwards et al., ), which contains synthetic mixtures of two or three speakers. We use this to derive the number of speakers task, which we believe has interesting applications for monitoring individuals if transferred to bioacoustics.

### 3.2 Evaluation data: the BEANS-Zero benchmark

**Table 2: Evaluation tasks and datasets of BEANS-Zero. $\ {}^{a}$ CLS: classification, DET: detection, CAP: captioning. ^b The numbers of samples for classification and captioning, and the number of 5-second “chunks” for detection (see Section 3 for more details)**
| Task^a | Dataset | Description | # Size^b | # Labels (type) |
| --- | --- | --- | --- | --- |
| CLS | esc50 | generic sound | 400 | 50 (sound type) |
| CLS | watkins | marine mammals | 339 | 31 (species) |
| CLS | cbi | birds | 3620 | 264 (species) |
| CLS | humbugdb | mosquito | 1859 | 14 (species) |
| DET | dcase | birds & mammals | 13688 | 20 (species) |
| DET | enabirds | birds | 4543 | 34 (species) |
| DET | hiceas | cetaceans | 1485 | 1 (species) |
| DET | rfcx | birds & frogs | 10406 | 24 (species) |
| DET | gibbons | gibbons | 18560 | 3 (call type) |
| CLS | unseen-cmn | birds etc. | 931 | 300 (species) |
| CLS | unseen-sci | birds etc. | 931 | 300 (species) |
| CLS | lifestage | birds | 493 | 3 (stage) |
| CLS | call-type | birds | 15439 | 2 (call/song) |
| CAP | captioning | birds etc. | 29002 | (open-ended) |
| CLS | zf-indv | zebra finches | 2346 | 4 (# of indv.) |

One contribution of this work is a new benchmark for bioacoustics: BEANS-Zero (Table 2). With BEANS-Zero, we go beyond traditional species classification, introducing tasks such as call-type prediction, lifestage classification, captioning, and individual counting (which is not seen during training). To build this set of tasks, we first used the test portion of the benchmark BEANS (Hagiwara et al., ) for evaluating our models on common bioacoustics datasets and tasks, which include:

- •
esc50 (Piczak, ): Generic environmental audio classification with 50 labels.
- •
watkins (Sayigh et al., ): Marine mammal species classification with 31 species.
- •
cbi (Howard et al., ) Bird species classification with 264 labels from the Cornell Bird Identification competition hosted on Kaggle.
- •
humdubdb (Kiskin et al., ) Mosquito wingbeat sound classification into 14 species.
- •
dcase (Morfi et al., ) Mammal and bird detection from DCASE 2021 Task 5: Few-shot Bioacoustic Event Detection (20 species)
- •
enabirds (Chronister et al., ) Bird dawn chorus detection with 34 labels.
- •
hiceas (Center, ) Minke whale detection from the Hawaiian Islands Cetacean and Ecosystem Assessment Survey (HICEAS) (1 label).
- •
rfcx (LeBien et al., ): Bird and frog detection from the Rainforest Connection (RFCx) data with 24 species.
- •
gibbons (Dufourq et al., ): Hainan gibbon detection with 3 call type labels.

We also include novel bioacoustics datasets including:

- •
unseen-cmn: 300 species held out from AnimalSpeak (Robinson et al., ) with common (English) names. For a dataset of medium difficulty, we hold out species whose genus is reasonably well-represented in the training set (at least 100 recordings.)
- •
unseen-sci: same recordings as above, but predicted with scientific (Latin) names
- •
lifestage: Predicting the lifestage of birds across many species. Newly curated from Xeno-canto ().
- •
call-type: Classifying song-vs. call across multiple species of birds. Newly curated from Xeno-canto ().
- •
captioning: Captioning bioacoustic audio on AmimalSpeak (Robinson et al., )
- •
zf-indv (Elie & Theunissen, ): Counting the number of zebra finch individuals

Some of these tasks, in particular captioning of bioacoustic audio, are previously unstudied. Captioning allows for automatic generation of descriptive annotations of animal sounds, enhancing our understanding of species behaviors and communication patterns. Improvements in other new tasks, such as cross-species lifestage and call-type prediction, would allow finer-grained ecological monitoring at scale.

For evaluation, we use accuracy for classification, macro-averaged F1 for detection, and SPIDEr (Liu et al., ) for captioning. We opt for F1 instead of mean average precision (mAP), which is originally used in BEANS for detection, as F1 is better suited for generative models, whereas mAP assumes a smooth ranking of candidates, which is less appropriate for evaluating generative tasks.

### 3.3 NatureLM-audio Architecture

Our model follows a generic audio-to-text architecture used in prior works, such as SALMONN (Tang et al., ), Qwen2-audio (Chu et al., ), and LTU (Gong et al., ), which are large audio-language models trained on paired audio-text data for tasks including speech, music, and general audio events. Figure 1 provides an overview of the NatureLM-audio architecture.

Specifically, NatureLM-audio first encodes the audio input via an audio encoder, in this case BEATs (Chen et al., ), which has achieved state of the art on multiple audio tasks. To connect the BEATs embeddings with the LLM we use a Q-Former (Li et al., ) applied at the window level as proposed in SALMONN  (Tang et al., ).
Similarly to the existing LALMS we use an LLM to produce text, in this case Llamma 3.1-8b (Dubey et al., ), which is fine-tuned with LoRA (Hu et al., ). The parameters of the LLM (except for the adapter layers) remain frozen during training, while the audio encoder and Q-Former are unfrozen.
The model takes an audio ${\bm{a}}$ and an instruction ${\bm{x}}$ as its input, and produces a text sequence ${\bm{x}}_{<t}$ as the output.
The model is trained under the loss function:

$$ $\displaystyle{\bm{h}}$ $\displaystyle=$ $\displaystyle f_{W}({\rm Encoder}({\bm{a}}))$ (1) $\displaystyle{\bm{z}}$ $\displaystyle=$ $\displaystyle p_{\varphi}^{Q}({\bm{q}},{\bm{h}})$ (2) $\displaystyle L$ $\displaystyle=$ $\displaystyle-\sum\log p_{\theta}^{LM}({\bm{x}}_{<t}|{\bm{z}},{\bm{x}})$ (3) $$

where ${\rm Encoder}$ is the pretrained BEATs (Chen et al., ) audio encoder, $f_{W}$ is a function that converts consecutive $W$ audio frames into a window, $p_{\varphi}^{Q}$ is the Q-Former model with trainable parameters $\varphi$ that converts a window into a sequence of text representations ${\bm{z}}$ using query ${\bm{q}}$, and $p_{\theta}^{LM}$ is the pretrained LLM with trainable parameters $\theta$.

### 3.4 Training Method

Our training method is heavily motivated by curriculum learning (Soviany et al., ) where machine learning algorithms start with simpler, easy to learn instances and gradually shift to more difficult ones, as done in other audio foundation models (Tang et al., ; Gong et al., ). We train in the following two stages:

- •
Stage 1 (Perception Pretraining):
We pretrain the model exclusively on the task of focal species classification, classifying vocalizations of thousands of animal species. Species classification is highly deterministic, allowing opportunity to learn a robust connection between language and audio. We also choose to train on this task individually as it is foundational to other tasks in bioacoustics.
- •
Stage 2 (Generalization Fine-tuning):
In the second stage, we introduce a variety of bioacoustic and other tasks that build on the robust classification performance of the first stage. This includes detection, captioning, lifestage prediction, and call-type prediction. We also include speech and music data in this second stage, hoping to transfer to bioacoustic tasks.

We trained from scratch (i.e., random initialization of the Q-Former and LoRA) rather than fine-tuning existing models or checkpoints, such as SALMONN’s. This allows for more flexibility in terms of choosing the latest LLM, with the most knowledge of animal species, and the most relevant architectural components (e.g. excluding memory-heavy parts of current LALMs such as the speech encoder Whisper (Radford et al., )).

## 4 Experiments

### 4.1 Training and Evaluation Details

We train our model on the full curated training set (Section 3.1).
To assess the model’s generalization we created hold-out splits for Xeno-canto, iNaturalist, Animal Sound Archive, and Watkins datasets, used solely for benchmarking.

We train with our two proposed stages. In both stages we use a linear warmup, cosine scheduler, peak learning rate of $9.0\times 10^{-5}$, and a batch size of $64$. We decode using beam search with two beams, a repetition penalty of 1.0, and a length penalty of 1.0.

We consider several inference methods depending on the task type. Species-classification tasks involve single-label prediction: we prompt the model to output the species name from the recording. To handle the case where the LLM outputs text which is not an allowed label, we match to the closest label according to the Levenshtein distance. We choose the Levenshtein distance for its simplicity and because species names, in particular Latin names, have high character-overlap with related names. We note this may not be optimal for general audio classification.

For multilabel detection tasks, we range from detecting a large number of species to only a single species, depending on the dataset. When detecting only a few species (ten or less), we include the options in the prompt. Otherwise we prompt the model to predict all species in the audio window, if any. In both cases, the model outputs all detected species, or ‘None’. We discard detections with low character-overlap with the allowed labels.

Our baselines include CLAP-like models (Wu et al., ), which cannot naively perform multilabel detection. To handle this, we create a negative “template” for each detection task, as proposed in (Miao et al., ). We consider each label a detection positive for CLAP if the audio is more similar to the label than to the negative template in the CLAP model’s embedding space.

### 4.2 Species Classification and Detection

**Table 3: Main zero-shot results on BEANS-Zero. We used accuracy for classification, and F1 for detection tasks. The best and the second best metrics are highlighted and underlined per each dataset**
| Model | esc50 | watkins | cbi | humbugdb | dcase | enabirds | hiceas | rfcx | gibbons |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLM w/o audio | 0.020 | 0.041 | 0.005 | 0.073 | 0.000 | 0.001 | 0.210 | 0.000 | \ul0.013 |
| SALMONN | 0.320 | 0.041 | 0.004 | 0.090 | 0.005 | 0.004 | 0.097 | 0.002 | 0.005 |
| Qwen2-audio | 0.307 | 0.041 | 0.004 | 0.070 | 0.005 | 0.004 | 0.097 | 0.002 | 0.005 |
| BioLingual | \ul0.600 | \ul0.257 | \ul0.705 | \ul0.085 | \ul0.036 | \ul0.109 | 0.429 | \ul0.004 | 0.018 |
| NatureLM-audio | 0.635 | 0.646 | 0.755 | 0.073 | 0.052 | 0.279 | \ul0.390 | 0.039 | 0.003 |

Table 3 shows the main results measured on the BEANS-Zero species classification and detection datasets. Our baselines include an LLM (Llama-3.1-8B-Instruct, Dubey et al. ()) without audio input, SALMONN (Tang et al., ), BioLingual (Robinson et al., ), and Qwen2-audio (Chu et al., ). As shown in the table, the outputs from the LLM without audio input, SALMONN, and Qwen2-audio are largely random on the bioacoustic datasets, failing to properly interpret the input audio or follow the instructions. In contrast, NatureLM-audio achieved state-of-the-art zero-shot performance on 6 out of 9 datasets, and delivered competitive results on the remaining tasks from the BEANS-Zero benchmark. We observe that for some of those three remaining tasks, our current training data contains little signal, for example on humbugdb (Kiskin et al., ) which classifies species by mosquito wingbeat sounds not generated by a vocal tract. We also note that performance of baselines on the general audio auxiliary dataset ESC50 (Piczak, ) may be reduced by the use of the Levenshtein distance, as our pipeline is optimized for bioacoustic tasks.

**Table 4: Comparison with bird vocalization models**
|  | cbi | dcase-bird | enabirds |
| --- | --- | --- | --- |
| BirdNET | 0.609 | 0.035 | 0.164 |
| Perch | 0.744 | 0.035 | 0.490 |
| NatureLM-audio | 0.755 | 0.088 | 0.279 |

We also compared NatureLM-audio with bird vocalization classification models, namely BirdNET (Kahl et al., ) and Perch (Ghani et al., ), to evaluate the zero-shot capabilities of our model. We compare on the subset of BEANS-Zero classifying or detecting exclusively bird species, plus the portion of DCASE with bird species. The results are presented in Table 4. Since both BirdNET and Perch were trained in a supervised manner on datasets that significantly overlap with our bird evaluation datasets, this is not a fully fair comparison, and their performance should be considered as topline results. Nevertheless, our model demonstrated strong zero-shot bird vocalization classification capabilities. In particular, we achieve a new SotA for the cbi dataset, classifying vocalizations of hundreds of birds, and achieve competitive results with the bird-specific models on both detection tasks.

### 4.3 Generalizing to Unseen Species

We further evaluate the model’s ability to generalize to completely unseen taxa using the newly added datasets in BEANS-Zero. They consist of recordings of held-out species from Xeno-canto, iNaturalist, Animal Sound Archive, and Watkins. As a topline, we compare against BioLingual, which was trained on species classification and has seen these species in training. As a baseline, we consider CLAP-LAION (Elizalde et al., ), a general-domain audio model which, similar to our model, is unlikely to have seen these species during training. We compare the performance when predicting common as well as scientific names.

Our model significantly outperforms the baseline, demonstrating generalization to completely unseen species. Our model in particular excels when predicting with scientific (Latin) names (unseen-sci), which have consistent hierarchical structure it may learn to exploit.

**Table 5: Generalization to unseen species in terms of classification accuracy for: ^a common (English) names and ^b latin/scientific names**
|  | unseen-cmn^a | unseen-sci^b |
| --- | --- | --- |
| Supervised SotA | 0.547 | 0.614 |
| NatureLM-audio | 0.116 | 0.196 |
| baseline (CLAP) | 0.034 | 0.004 |

### 4.4 Novel Bioacoustic Tasks

We evaluate the model’s abilities beyond species prediction with several bioacoustic tasks newly added to BEANS-Zero, which have, to the best of our knowledge, not been studied at a cross-species level. We additionally include zf-indv, a completely unseen task counting the number zebra finches in a recording (Elie & Theunissen, ). We compare against BioLingual (Robinson et al., ) for discriminative tasks and SALMONN (Tang et al., ) for captioning. On each of these tasks, our model sets the state-of-the-art.

**Table 6: Results on BEANS-Zero novel bioacoustics tasks. We report accuracy for classification, and SPIDEr (Sharif et al., ) for captioning**
|  | lifestage | call-type | captioning | zf-indv |
| --- | --- | --- | --- | --- |
| SotA | 0.676 | 0.499 | 0.009 | 0.225 |
| NatureLM-audio | 0.763 | 0.810 | 0.494 | 0.383 |

### 4.5 Ablation on Speech and Music

To investigate the impact of speech and music on downstream task performance, we run an ablation on stage-2 training with and without speech and music data. We train both stage-2 models for 200k steps, and evaluate their ability to perform the unseen task of counting zebra-finch individuals in a recording. The model trained with speech scores .377 on this task, similar to our full model. The model trained without speech scores an accuracy of .243, approximately random, and qualitatively predicts a single speaker for every recording. This result suggests the ability to count vocalizing birds transfers from human speech and music, for which our training data includes counting human speakers in a recording.

## 5 Conclusion

We presented NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics, demonstrating its potential to address critical tasks such as classifying and detecting animal vocalizations, and decoding context, call types, and individuals across species. By leveraging a carefully curated dataset spanning bioacoustics, speech, and music data, NatureLM-audio sets the new state-of-the-art (SotA) on multiple tasks, including zero-shot classification of unseen species. Moreover, our model demonstrates positive transfer across both domain and tasks, performing well on a novel benchmark (BEANS-Zero), which includes new bioacoustic tasks such as captioning and individual counting. To further accelerate research and the development of more robust models in the field, we have open-sourced the code for generating both training and benchmarking data.

We plan to extend this work by incorporating more diverse tasks and datasets, improving the text-based LLM backbone with bioacoustic-specific texts, and enhancing the model’s multilingual capabilities. Additionally, we aim to introduce new modalities, such as motion and image data, leading to models like NatureLM-motion and NatureLM-image. Lastly, we will explore the model’s generative abilities, enabling it to produce audio tokens for tasks such as animal sound generation and audio denoising.

While NatureLM-audio offers significant potential for advancing biodiversity monitoring and conservation, several ethical concerns must be addressed. First, there is a potential bias towards bird vocalizations due to the overrepresentation of bird datasets, which could limit the model’s effectiveness in other domains. Second, the model’s ability to detect and classify endangered species could be misused for illegal activities such as poaching, posing a threat to wildlife. Finally, unintended consequences on animal behavior and ecology must be considered, particularly when deploying LLMs, known for their issues including hallucinations and biases (Kuan et al., ). These systems may interfere with the behavior of the species being studied, and the long-term ecological impact of widespread passive monitoring is still unknown. Careful deployment and responsible use are essential to mitigate these risks.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: This is excellent and careful work. I particularly liked the SoTA results in classification
soundness: 4
strengths: An incredible collection of datasets, and careful curation.
A lot of ancillary code for use of the data in various learning tasks.
summary: An excellent paper providing a valuable source of bioacoustic data and processing methods to the community. An attempt to comprehensively collect much of the available domain data and curate it into a usable dataset. Plans for distribution and general availability of the dataset/code to the community are missing.
weaknesses: Unclear from the presentation if the authors intend to make the dataset widely available, and under what license.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could you add ablation studies to access the influence of each part of your approach? What happens if the audio encoder stays frozen? Are the embeddings that BEATs generates already good enough for taking bioacoustics tasks?
2. What SoTA model is used in the call-type comparison? I could not find that in the text.
3. How do you evaluate the LLM without audio and why is it outperforming your model on the gibbons dataset?
soundness: 3
strengths: 1. Addresses a important topic from both the ML research community ( since audio and especially computational bioacoustics is a hard problem) and societal importance.
2. Collects a comprehensive training dataset and extends an existing evaluation benchmark  with additional tasks.
3. The performance improvements compared to a model not trained on bioacoustics data (SALMONN) supports the claim that this domain is in the need for a own foundation model.
summary: The authors present a large audio-language model (Lalm) trained and evaluated on bioacoustics data. First, a dataset of text-audio pairs is compiled, partially with the use of LLMs. This dataset is then used to train an architecture similar to SALMONN. It is consisting of a LLM (Llama 3.1-8b), an audio encoder (BEATs), and a Q-Former to connect the computed audio embeddings to the LLM. During training the LLM stays frozen while the audio encoder and Q-Former are trained after a curriculum (first species classification than also detection, captioning, life stage prediction, call-type detection). The model is evaluated on the self introduced BEANS-Zero benchmark which extends BEANS with additional tasks and a zero shot evaluation protocol. The model achieves the best results compared to non bioacoustics
weaknesses: 1. Soundness of results: Your presented results only show a minor improvement compared to BioLingual (which also presents zero shot results on BEANS, there numbers differ sometimes why?), so whats the benefit of your approach and more particularly does integrating a LLM has a benefit? Or is it the different training dataset? Or the audio encoder (BEATs vs. HTS-AT)?
2. No further details for replication of the experiments are given, e.g. pretrained models or the list of species which were hold out.
3. Difficult covariate shift from focal training to soundscape test data is not evaluated, which is one of the major challenges in bioacoustics. E.g. for birds see Stowell,2022 or BirdSet,2024.
4. Call-type task: birds usually have more than two call-types per species. A binary classification task ignoring the specie might have few practical applications.
5. L. 428-219: The cbi dataset consists of XC recordings, your model should have the same advantage as BirdNet and Perch, so the comparison should be fair.
6. Line 071-072: How do you support the claim that the generalization of BirdNet is limited? Did not Ghani et al. claim the opposite?

---

### Language

1. l. 190-192 strong repetition
2. Inconsistent use of abbreviations (e.g. state of the art)

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. In Section 3.1.1, the method for hard negatives sampling for species detection is briefly mentioned, but the details are unclear. Could you provide a more detailed explanation of the strategy for sampling these hard negative samples?

2. Could you provide more details in Section 3.2 about the selection strategy for the held-out species in the AnimalSpeak dataset for the unseen-cmn and unseen-sci subsets? Even if the selection was random, it's important to clarify the process.

3. In Section 4.5, you state that including speech and music in training has a positive impact on counting zebra-finch individuals. However, while the ablation study includes speech, it is unclear whether music was also ablated. Could you clarify whether the effect of music was tested? Additionally, conducting ablation on all downstream tasks, not just individual identification, could provide more comprehensive insights into whether speech and music data enhance performance across other tasks as well. This could help clarify which types of training data are most beneficial for specific applications.
soundness: 2
strengths: 1. The introduction of NatureLM, the first audio-language model specifically designed for bioacoustics, represents a promising new direction for incorporating language models into biodiversity monitoring.

2. The development of the BEANS-Zero benchmark extends the original BEANS benchmark by introducing new tasks, such as call-type prediction, life-stage classification, individual counting, and open-ended audio captioning. These additions have the potential to advance bioacoustics research and enable more detailed acoustic monitoring of species.

3. The paper is easy to follow, and the related works are thoroughly referenced.
summary: The paper introduces NatureLM, an audio-language model specifically designed for bioacoustics, focusing on animal sounds. NatureLM leverages a diverse dataset combining bioacoustics, human speech, and music to enhance generalization across various species and tasks. NatureLM demonstrates strong in-context learning capabilities, enabling zero-shot classification for unseen species.

Additionally, the paper presents BEANS-Zero, a benchmark for bioacoustics that includes several tasks beyond species classification, such as call-type prediction, life-stage classification, and individual counting, aiming to push the boundaries of bioacoustic research.
weaknesses: 1. Incorrect Terminology


1.1.  The introduction describes BioLingual as self-supervised; however, the supervision is derived from text generated based on class labels. I recommend referring to it as supervised learning with language-based supervision for greater clarity and accuracy.

2.1. Both BioLingual and AVES are described in the paper as foundation models, but this classification may be misleading. BioLingual and AVES are trained on datasets with less than 2 million samples, while models trained on AudioSet with 2 million samples are not typically considered foundation models. BioLingual is evaluated on classification and retrieval tasks, while AVES is evaluated on classification and detection tasks. Typically, a foundation model is a very large model trained on extremely large datasets so that properties emerge that enable it to handle a wide variety of tasks across different domains. For improved clarity, I suggest either using alternative descriptions like self-supervised (for AVES) and audio-language contrastive (for BioLingual) to more accurately describe them, or providing a clear definition of what you consider to be a foundation model if you wish to retain this terminology. This will help avoid confusion and ensure that readers more precisely understand the capabilities of these models.

2. Overstatement of Results


2.1. The analysis of NatureLM-audio's performance on the cbi dataset (Table 4, Section 4.2) is potentially misleading due to data overlap. Since BirdNet, Perch, and NatureLM-audio are trained on Xeno-Canto, and the cbi evaluation dataset is a subset of Xeno-Canto, there is overlap between the training and evaluation data. This compromises the claim of state-of-the-art performance, especially that Perch significantly outperforms NatureLM in enabirds.

2.2. In Table 5, Section 4.3, the authors claim that NatureLM-audio demonstrates generalization to completely unseen species by outperforming CLAP, a contrastive audio-language model trained on non-bioacoustic data. However, this conclusion seems too strong. The results primarily show that a model trained on bioacoustic data performs better than one trained on non-bioacoustic data, which does not necessarily indicate true generalization to unseen species. Additionally, BioLingual, which was trained similarly to CLAP but on bioacoustic data, significantly outperforms NatureLM-audio. I suggest rephrasing the conclusion to more accurately reflect these findings.

2.3. In Section 4.4, the paper claims state-of-the-art performance in the captioning task. However, this comparison is based solely on the SALMONN model, which was not trained on bioacoustic data and is not specifically designed for this task. To support the claim of state-of-the-art performance, I recommend including a comparison with a captioning model trained on bioacoustic data, as this would provide a more accurate and meaningful evaluation.

Meta-review: **Paper Summary:**

This paper describes an audio-language foundation model for bioacoustics: NatureLM. This model is trained on a new collection of annotated datasets consisting of bioacoustics, speech, and music. It is evaluated using the newly proposed BEANS-Zero benchmark.

**Strengths:**

Reviewers GfHx and Lfuo both appreciate the value of the training data collected to train this model. In discussion, the authors clarified that:

> We will publish the datasets on HuggingFace [...] This will be the largest open cross-taxa multi-task bioacoustics dataset.

Reviewers GfHx and 1ReQ both appreciate the development of the BEANS-Zero benchmark.

**Weaknesses:**

Reviewers Lfuo and 1ReQ raised several concerns about the experimental results, most of which were satisfactorily addressed in the discussion period.

Decision: Accept (Poster)