WhAM: Towards A Translative Model of Sperm Whale Vocalization
Authors: Orr Paradise
Liangyuan Chen
Pranav Muralikrishnan
Hugo Flores García
Bryan Pardo
Roee Diamant
David Gruber
Shane Gero
Shafi Goldwasser
Venue: NeurIPS 2025 poster ()
Keywords: Sperm Whale Communication
Bioacoustics
Masked Acoustic Token Modeling
Generative Audio Models
Representation Learning
Primary area: machine_learning_for_sciences
TL;DR: WhAM: a transformer model unifying generation, acoustic translation and classification of sperm whale vocalizations

Abstract: Sperm whales communicate in short sequences of clicks known as codas. We present WhAM (Whale Acoustics Model), the first transformer-based model capable of generating synthetic sperm whale codas from any audio prompt. WhAM is built by finetuning VampNet, a masked acoustic token model pretrained on musical audio, using 10k coda recordings collected over the past two decades. Through iterative masked token prediction, WhAM generates high-fidelity synthetic codas that preserve key acoustic features of the source recordings. We evaluate WhAM's synthetic codas using Fréchet Audio Distance and through perceptual studies with expert marine biologists. On downstream tasks including rhythm, social unit, and vowel classification, WhAM's learned representations achieve strong performance, despite being trained for generation rather than classification. Our code is available at https://github.com/Project-CETI/wham

WhAM: Towards A Translative Model of Sperm Whale Vocalization
Orr Paradise
Pranav Muralikrishnan"
Liangyuan Chen
Hugo Flores Gareía" 3
Bryan Pardo
Roee Diamant 4.2
David E. Gruber5.2
Shane Gero 6.2
Shafi Goldwasser1.2
UC Berkeley
2 Project CETI
Worthwesternn University *Haifa University
City University of New York
"Carleton University
"Equal contribution
Abstract
Sperm whales communicate in short sequences of clicks known as codas. We present WhAM (Whale Acoustics Model), the first transformer-bet -based model capa- ble of generating synthetic sperm whale codas from any audio prompt. WhAM is built by finetuning VampNet, masked acoustic token model pretrained on musical audio, using 10k coda recordings collected over the past two decades. Through iterative masked token prediction, WhAM generates high-fidelity synthetic codas that preserve key acoustic features of the source recordings. We evaluate WhAM's synthetic codas using Fréchet Audio Distance and through perceptual studies with expert marine biologists. On downstream classification tasks including rhythm, social unit, and vowel classification, WhAM's learned representations achieve strong performance, despite being trained for generation rather than classification. Our code is available at httppps: // thub. wwc.o//rroeeetCETTT1 CETICthaEET/haa
1 Introduction
Understanding the communication of sperm whales (Physeter macrocephalus) is among the most fascinating questions in animal behavioral studies.
Sperm whales communicate using codas coday-short sequences of clicks that vary in number, rhythm, and tempo WWakkins and Schevill 1977 Weilgart and Whitehead 1993 Sharma et al. 2024aa). They live in stable, Cemale-led social units that form larger vocal clans based dialect [Rendell and Whitchead 2003). That is, the dialect of a social unit determines its clan, with social units associating exclusively with other units from their clan [Gero et al. [2016a]. Furthermore, dialects believed to be learned socially rather than inherited genetically [Cantor and Whitehead 2015 Rendell et al.
2012].
The complexity of these learned vocal patterns has motivated new computational approaches to understanding codas and their functionality, Leitao et al. [2024] modeled codas as (variable-length) Markov chains, revealing new patterns of inter-clan social learning. [Beguš et al. (2025) study vowel- like spectral properties of codas, which were initially suggested by interpreting the codebook of
Generative Adversarial Network (GAN). Most recently, [Sharma et al. [2024b] trained a transformer on click timings (inter-click intervals), which is able to predict codas in exchange based long-term dependencies, as well as future diving behavior. These studies collectively highlight how
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MMH
linplet
Figure 1: Left: WhAM is trained by finetuning VampNet Gareía et al. 2023 an audio-to-aan -o-idoooo transformer pretrained large music dataset (a). Namely, we perform domain adaptation (b) on animal vocalizations followed by species-specific finetuning (c) on a novel sperm whale coda dataset. Right: WhAM synthesizes context-aware variations (d) of input codas and acoustically translates (e) natural and (f) artificial audio into coda-like audio. Illustration Alex Boersma 2025.
machine caaninng-arttcularrly transformer architectures- decode patterns imperceptible to traditional methods.
Transformers Vaswann et al. 2017] originated in natural language translation, where they revolution- ized the field by enabling high-quality, context-aware machine translation. Whereas transformers have since become ubiquitous across machine learning (e.g. [Chen et al. [021 Khan et al. 2022 Moussad et al. 2023). in this work we propose again to use transformers towards translation of animal communication.
While transformers have been used in settings where parallel data is nonexistent [Conneau and Lample 2019] and for translation from audio [Kano et al., (021).21. applying these advances to animal communication presents deep challenges. Even merely defining the problem has been the subject of
studies spanning theoretical computer science [Goldwasser et al. 2023). biology [Yovel and Rechavi
2023 Amphaeris et al., [023). linguistics [Berwick and Chomsky 2016 AAmphaeris et al. 2022 and
philosophy [Suzuki al. 2000 Hobaiter 20222.
Existing approaches to modeling sperm whale codas have made significant advances. Bermant et al. [2019] developed effective methods for coda detection and classification, while generative models based on GANs [Beguš al., 2023 Kopets et al.) [2024] have shown the potential for synthesizing coda-like audio. The atorementioned timing-based analyses of T et al.) (2024), Sharma et al. (2024b) have yielded new insight into the social and behavioral aspects of sperm whale communication.
While GAN-based models can generate 2024), they cannot easily condition on given 2024 Sharma al.) 2024b) capture important temporal patterns but may miss features only present in the raw audio, such as the recently discovered vowel-like spectral patterns [Beguš et al. 2025). Moreover, current methods train separate models for classification [Bermant et al. 2019) and generation, despite the intuition that model capable of realistic generation should also learn representations useful for classification. Lastly, none of these tackled the issue of translating across acoustic domains.
Our work will address challenges left open by these works:
coda-like audio [Beguš et al.) 2023 Kopets et al.
context. Timing-based approaches Leitao al.
To address these challenges, introduce the Whale Acoustics Model (WhAM, Figure new approach to modeling sperm whale codas that unifies three capabilities:
Acoustic franslation: WhAM can translate audio prompt (e.g. other animal vocal- izations or even noise) into the acoustic style of sperm whale codas, acting as a form of cross-domain style transfer.
Generation: WhAM can generate novel "pseudocodas" that are perceptually similar real codas, as evaluated by expert listeners.
'e emphasize that translation is in the acoustic sense; semantic translation remains distinct and more ambitious goal.

---

Classification: WhAM's learned representations are useful for range of classification tasks, including rhythm type [Sharma et al. 2024a]. social unit classification [Best 1979
Christal and Whitehead 2001 Gero et al. 2016b), and the recently discovered vowel-like
spectral patterns of BBguu et al. [2025] despite being trained primarily for generation.
Contributions. This paper presents the first unified model of sperm whale codas capable of acoustic translation, generation, and classification. Notably, WhAM demonstrates that meaningful bioacoustic features emerge from purely generative training. aligning with recent work on self-supervise (non- generative) modeling of animal vocalizations (Hagiwara 2023
WhAM serves as a proof of concept, applying advances in neural audio modeling to bioacoustics in a novel and unifying way. To facilitate further research, we will release the model and its training and evaluation code. Remarkably, WhAM achieves strong results after just five days of training on a single GPU. While the dataset is small compared to those used for large audio models [Borsos)
ct al. 2023] Agostinelli et al. 2023 our results suggest that scaling up could yield even greater improvements.
Finally, WhAM was developed in close collaboration with marine biologists and underwater acous- ticians with domain expertise in sperm whale vocalizations. The model was evaluated through perceptual studies conducted by an interdisciplinary team of specialists. To our knowledge, this is the first paper to evaluate the perception of experts on synthetically generated codas, igniting a crucial discourse for validating the utility of generative models in bioacoustics research. Code, model, and data are available at htttps://ggthiii wwwp:rojecto/o/OOO//eo CerpectttETI/ahanan
Outline. Section 2
reviews related work. Section D details our methodological framework. Section presents experimental results and expert analysis. Section 5 discusses future work.
2 Related work
Audio Generation.
The vast majority of studies on deep generative audio models focus on human
speech music (e.g. van den Oord et al. 2016 Dong al. 2018
Dhariwal et al. 2020 Lakhotia
jet al. 2021
Agostinelli et 2023). Some works are dedicated to generating the vocalization of
animals (bioacoustics) such as birds [Bhatia and Kinnunen 2022
Guei et al. [024] mice (Reilly
ctt al.
2023 cetaceans (Bergler et al. [022] Zhang et al. [222]] Honghui and Lanhao 2022 Kim
al.
2024 and in particular sperm whales BBeguš al.] 2023 Kopets al. [022]] However, our
know ledge, all techniques for bioacoustic generation are based on generative adversarial networks (GANs). Unlike our munsformer-basedd WhAM, GANs do not allow for conditioning on context in the form of an audio prompt. We emphasize that WhAM enables translation of input sounds into the acoustic style of sperm whale vocalizations, operating purely at the signal level. This is distinct from semantic translation between communication systems, which remains a far more ambitious goal requiring deep understanding of animal cognition and communication (e.g. Goldwasser et al. 2023 Amphaeris et al. 2023).
Yovel and Rechavi 2023
Animal Vocalization Modeling. Deep learning techniques have been applied towards other, non- generative, ends in bioacoustics research. Learned audio representations have been used for species recognition Chen et al. [2014], |afemann et al. [2014), Xu et al. 2019]. Kahl et al. (202)) Xic
et al. 2023 and automatic annotation (i.e., vocalization detection and classification) of bioacoustic data Bergler et al. 2019). Coffey et al. 2219), Bermant et al. (2019, Premoli et al. [202]] AVES
[Hagwaraa] 2023] utilizes HuBERT's [[su et al.) 2021] selfvsupee learning framework towards state-of-the-art performance in species classification and detection tasks. While AVES demonstrates the power of learned audio representations, its encoder-only architecture limits it to analysis tasks, con- trasting with WhAM's generative capabilities. As we show in Section 4.3) while AVES outperforms WhAM on classification tasks as expected given its specialized design, WhAM still learns meaningful representations as a byproduct of its generative training, outperforming bascline approaches despite having a different primary objective.
Sperm whale communication. challenge in marine biology for over six decades (Backus and Schevill 1966, Watkins and Schevill 1977 Whitehead and Weilgart 199| Andreas et al. 2022; see also Appendix B). Recent computa-
Understanding sperm whale communication has been a central
3

---

tional approaches have focused on analyzing click timing patterns within codas and do not directly address the acoustic properties of individual clicks within codas (Sharma al. 2024a Leitao et al.)
2024 Sharma et al. 2024b]. WhAM extends this computational trajectory by enabling systematic manipulation of click acoustics, potentially allowing a quantitative analysis of acoustic variations between clan dialects and investigation of features that make codas recognizable. While WhAM's synthetic codas may not yet match the quality needed for playback experiments. WhAM represents progress towards stimuli generation in responsible behavioral study (Tyack 1983 Deecke 2006
King and Jensen 2023, see also Appendix A
Methods
3.1 Masked Acoustic Token Modeling with VampNet
VampNet|Gaan (aaríaa et al. [023] is an audio-to-audio generative model, pretrained on 797k music tracks from thousands of artists VampNet consists of three neural models: tokenizer, a coarse-token model, and a coarse-to-fine model. For simplicity of presentation we will avoid the distinction between coarse and fine tokens, instead decomposing VampNet into an Acoustic Tokenizer and Masked Acoustic Token Model. The reader is referred to (aarífa et al. [2023) for full details of the model, and Appendix E.2 for specification of hyperparameters used in training WhAM.
Acoustic Tokenizer. The tokenizer takes as input an N audio snippet sampled at Naam Hz, and outputs a sequence of l discrete tokens from a finite vocabulary E. A jointly-trained detokenizer will convert token sequences back into audio:
T: Nax XNum Σ
T-1: T- y
Nocs X Noam
VampNet uses residual vector quantization approach known as the Descript Audio Codec (DAC, Kumar et al. 2023). At high level, audio tokenized in a temporal and hierarchical fashion, such that each interval of samples is replaced with "stack" of tokens; this means that neighboring stacks of tokens correspond to contiguous intervals samples in the audio. For example, the first five token stacks (o1....., o5) could correspond to the first 0.5 seconds of audio.
Masked Acoustic Token Model (MATM). A bidirectional transformer M is trained to perform the cloze task on acoustic token sequences. That is, each audio snippet in the pretraining dataset tokenized, and then a bidirectional transformer is trained to predict a random subset of masked tokens.
M: (E {MASK]]){ + Σ
A pretrained MATM can be finetuned in various ways. Following García et al. (2023), we finetune using Low Rank Adaptation (LoRA, Hu et al. 2022).
Generation. After training tokenizer T, detokenizer 7-1 and (possibly finetumed) MATM M. VampNet can be used to generate variations of given "prompt" audio snippets. This is done in the natural way, by randomly masking the tokenized audio; importantly, the masking scheme used in generation time does not need to be uniformly random. For example, the scheme can leave (classically-detedd than generating all masked tokens simultaneously (e.g. as in BERT, Devlin et al. 2019), VampNet uses iterative parallel decoding [Chang al. 20022 wherein tokens are gradually "unmasked" in sequence forward passes through the model.
beats unmasked, so to preserve the rhythm of the prompt. Rather
3.2 Data
WhAM is trained by finetuning VampNet (Section 3.1 on various datasets.
FSD. The Freesound Dataset (Font et al.) 2013 consists of 50k human-labeled recordings. We used recordings with the animal tag, which totaled 7h45m of audio.
AudioSet. A dataset of two million human-labeled audio clips taken from YouTube (Gemmeke ct al. 2017a. Of these, we used audio with the animal tag, totaling at about 5 hours.

---

Prompt
Generated
Tokere
(Tooee [MASK])
Toloon Boer
Delolenizer
Input andio
Generated andio
Figure 2: Overview VampNet's generation pipeline, Input audio is first converted into a grid of tokens by the Tokenizer. These tokens are then partially masked to create a prompt. The Masked Acoustic Token Model (MATM) uses parallel iterative decoding to generate new tokens, which are finally converted back into audio by the Detokenizer. The colored squares represent acoustic tokens, with grey squares indicating masked positions.
WMMS. The Watkins Marine Mammal Sound Database [Sayigh et al., 2016 totaling 4h8m. includes audio collected over seven decades in least 67 sites around the world. Sperm whales are among the 51 species recorded.
BirdSet. avian bioacoustics dataset curated for classification tasks [Rauch al. 2025]. totalling about 6,800 hours. Due to computational limits, we used 110-hour subset of audio dense with vocalizations.
DSWP. A dataset of 2507 annotated codas (Ih26m) collected over thirteen years in 2000km² area off the coast of Dominica. It consists of codas recorded using far-field boat-based hydrophones and noninvasive animal borne tags.
CETL. A more recent dataset of sperm whale vocalizations consisting of 7653 annotated codas (4h33m) collected similarly to DSWP.
The training of WhAM is split into two phases: (1) Domain adaptation, in which the base VampNet is finetuned on FSD+AudioSet+WMMI domain-adapted VampNet is finetuned on DSWP+CETI for an additional 500k iterations. Both phases follow the same (LoRA) finetuning procedure, but we find this split to be conceptually useful. Additional details are deferred Appendix E.1
for 500k iterations; (2) species-specific ppecies-specciccfffetunnngg in which
Results
We evaluate WhAM through three complementary analyses. First, we assess the quality of WhAM's synthetic codas through quantitative metrics, specifically the Fréchet Audio Distance (FAD, Kilgour et al. 2019) between generated and natural codas. Second, we conduct a perceptual study with expert marine biologists to evaluate how well our synthetic codas preserve the characteristic features of natural sperm whale vocalizations. Finally, evaluate WhAM's learned representations on downstream classification tasks to investigate whether our model captures meaningful acoustic features of sperm whale communication.
4.1 Fréchet Distance of Audio Translation
key aspect of WhAM its ability to "translate" audio inputs into the acoustic style of sperm whale codas. To evaluate this capability quantitatively, we measure the Fréchet Audio Distance (FAD) Kilgour et al. 2019) between natural and WhAM-generated synthetic codas. FAD measures the similarity between two audio datasets by comparing embeddings of the audio signals; lower FAD indicates greater acoustic similarity between the datasets.
FAD is computed using given audio embedding model. We chose BirdNET (Kahl et al. 2021 based on a principled calibration experiment that compared the sensitivity of five embedding models to the rhythmic patterns crucial to coda structure (Appendix D.I. We normalize FAD values by dividing by the maximum distance, scaling all values [0. 1). Figure portrays WhAM's translation ability using audio prompts from three domains:
Natural codas (S. Whale): A disjoint set of codas produced by sperm whales. The FAD between disjoint sets of natural codas is 0.21 11 (rather than zero) due to variance in recording
5

---

Wh-M
Fréchet Audio Distance
1.0
0.4
80
S. Whale
Beeps
C. Dolphin
Narwhal
A. Dolphin
Walrus
L Whale
Risso
Beluga
Orca
Ross Seal
B. Seal
B. Whale
Seal
Figure 3: Normalized Fréchet Audio Distance between sperm whale codas and various audio sources, before and after translation through WhAM. Lower FAD indicates greater acoustic similarity to natural codas. The horizontal line at 0.21 represents the baseline FAD between disjoint sets of natural codas. Full names of animals along with the number of samples from each can be found in Table
conditions, individual whales, and coda types. This establishes a baseline below which FAD fails to distinguish audio sources from natural variation: We therefore say that generated outputs are HAD--ndistinguishable when their FAD falls below 0.21. When passing natural codas through WhAM, we expect a slight decrease in FAD as WhAM regularizes inputs toward the mean embedding of its training distribution (a large dataset of diverse codas).
2. Animal sounds: Vocalizations from 12 species of marine mammals. Figure shows that WhAM consistently reduces the acoustic distance to natural codas, effectively translating these diverse inputs into the acoustic style sperm whale codas. WhAM -eenrrggee outputs of four species are FAD- -.--iddstinnuushabbl from natural codas.
3. Digital "beeps": Artificial audio generated by initializing arry of zeros and randomly selecting points to assign peak amplitude of 1. Remarkably, beeps and natural codas have approximately the same post- WhAM FAD. This may be because beeps' sparse structure (mostly silence with isolated peaks) gives WhAM freedom to infill patterns close to the embedding of codas, while natural codas with minimal silence constrain the model's regularization but start closer to the target distribution.
The results demonstrate WhAM's remarkable translation capability: five diverse audio sources (four non-whale species and digital beeps) become FAAD-indistinguisaalee from natural sperm whale codas after processing. This success across varied inputs suggests that WhAM has learned a robust representation of the essential acoustic properties that define sperm whale communication.
4.2 Expert Perceptual Study
To evaluate the perceptual quality of WhAM's synthetic codas, conducted a comprehensive study with domain experts to assess how well our generated outputs match natural sperm whale vocalizations with respect to human-exper distinguisher. This study measures both audio-only and spectrogram-bast baasog discrimination, while also gathering qualitative insights about specific acoustic features that distinguish synthetic from natural codas. Additional details are deferred to Appendix E.5
Expert backgrounds. Five academic experts participated in the perceptual study. Three identified as marine biologists, and two as underwater acoustics specialists. They listed between and 20 years of experience working with coda audio (field recordings), manual detection and classification, and the development of automatic detection systems. All experts had experience analyzing coda audio and spectrograms, which are the two media through which the experiment was carried out.
Experiment design.
We designed four-task study to be completed sequentially by each expert:

---

0.9
0.8
Apouracy
Accuraly
0.7
0.8
0.6
0.5
0.4
0.4
Nat. Coda
Wainua
Beepa
Smith Coda
2AFC
Mxed
2AFC Spec.
Figure 4: Expert performance audio-only 2AFC (Task 1), mixed classification (Task 2), and spectrogram-assitted 2AFC (Task 3). Er- for bars show standard deviation across ex- perts. While all tasks elicited above-chance per- formance (dashed line), spectrogram analysis showed the greatest variability between experts (a = 0.17). Task and 3 had 30 pairs each, Task 2 had a collection of 25 samples.
Figure 5: Accuracy in mixed classification (Task 2) for different input domains. Natural codas (left) were misclassified synthetic 36% of the time. The remaining columns depict perfor- mance synthetic codas generated by WhAM from walrus vocalizations, non-coda acoustic im- pulses, and codas (respectively). There were five synthetic codas from each domain, plus ten natu- ral codas for total of 25 items.
1. Audio-only twoaalternative forced choice (2AFC): Experts compared pairs of codas (one natural, one synthetic) audio-only conditions, and were asked identify the synthetic coda. Synthetic codas were generated by WhAM using the paired natural coda as input.
2. Mixed Collection Classification:
Experts classified clips as natural or synthetic. Clips were either natural codas, or synthetic codas generated from different sources: natural codas, digital beeps or walrus vocalizations (Sayigh et al. 2016). False positives (natural misclassified) and false negatives (synthetic undetected) were measured.
3. Spectrogram-assisterr 2AFC: Experts repeated the first task while visualizing audio with software of their choice. The experts were given the exact same samples as in the first task, ensuring direct comparability between audio-only and spectrogram-aided performance. This task mirrored real-world analysis workflows while quantifying the perceptual "advantage" of multimodal inspection.
4. Qualitative assessment: Experts were given five representative samples of synthetic codas. They were then asked questions about how well synthetic codas captured missed character- istics of natural codas, whether any non-natural patterns appeared in synthetic codas, and which features did they use to distinguish between codas in each of the previous tasks.
Fleiss's к quantified inter-expert agreement [Fleiss 1971]. and accuracy was calculated relative to ground-tree -ount labels. Task order was chosen towards minimizing bias (audio-first to avoid visual prim- ing), with background information collected in final section. The samples used in the experiment are attached to this submission as supplementary material. Experimental details are in Appendix E.5
Quantitative analysis. Experts achieved 81% accuracy (ı 0.41), in audio-only 2AFC (Task 1), rising marginally to 83% (w 0.41) with spectrograms visualized (Task 3). This 2% improvement suggests WhAM's synthetic codas lack glaring spectro-temporal artifacts detectable by trained analysts. As expected, accuracy with spectrograms was generally better per-expert, with one expert's performance dramatically increasing from 66% to 93% (another expert even achieved perfect score). Surprisingly, one expert's performance decreased from 83% in Task 1 to 66% in Task 3; comments in the qualitative section did not suggest an explanation.
Performance varied substantially across tasks and among experts (Figure 4 The most experienced expert ranked highest in both 2AFC tasks, but not in mixed classification. These variations reflect
Lie.. artificial sequence of clicks

---

diverging expert strategies some focused on inter-click patterns, others on spectral properties: "rhythm" to quote one expert, versus "DC offsets" and "inter-pulse structures" Mohl et al. 2003 to quote others.
some
Misclassification rates in Task 2 (Figure 3 revealed WhAM's efficacy in acoustic translation: average, experts correctly flagged walrus-to-coda audio only 75% of the time-less than digital beeps or coda-to-coda outputs of WhAM. For one expert, walrus-to-coda audio was detected only 50% of the time (random chance).
Fleiss's 's values (0.41-0.44) indicated moderate agreement across tasks, with experts showing greatest consensus on mixed classification (k = 0.44). Performance on spectrogram-ate -ided 2AFC performance was the most diverse-oon one expert achieved perfect performance while another approached chance (60%).
Qualitative feedback. Synthetic codas successfully replicated key acoustic features of natural codas. Most experts noted preservation of rhythm, referred to as inter-click intervals (ICT); that is, clicks occur "at the right time" in synthetic codas. Additionally, one expert answered that "spectral components" were overall preserved in synthetic codas.
That said, experts identified missing components which can be partitioned into three categories:
Within a single click: Some clicks "came on and disappeared too strongly," had "varying amplitude [within single coda]," and "inverted peaks." On a spectral level, an expert answered that clicks were too "broadband" compared to natural clicks which have low- frequency bias.
Rhythmichemporall One expert noted that the timing of clicks fit echolocation moreso than codas. (See Appendix B for how they differ.)
Recording-Leve level anomalies: One expert noted "DC offset" which they described as the unrealistic background noise on synthetic codas. Similarly, another noted that background noise in synthetic codas oscillated too much.
Based on this assessment, we present in Appendix C guide to the similarities and differences between natural and synthetic codas, supplemented by annotated spectrograms.
4.3 Utility of embeddings for downstream tasks
test whether WhAM's internal representations capture meaningful features of sperm whale vocalizations through three downstream classification tasks. For each task, we train small (two-layer) classifier head that takes coda embeddings as input. We compare WhAM to naive random- -edddm-e and majority-clas baselines, well AVES [Hagiwara 2023), self-supervised model achieving saate-of-thh-asd performance on bioacoustic classification tasks. Full details of the experimental setup deferred to Appendix E.4
The downstream tasks are:
1. Coda detection:
Given snippet of audio, determine whether it contains coda. The classifier is trained on positive (coda) and negative (no coda) snippets, with negative examples drawn from the same recording conditions to ensure the model learns coda features rather than recording artifacts.
2. Rhythm type:
Given a snippet of audio, classify its temporal pattern. Rhythm of inter-click intervals serves as a key axis for classification of sperm whale codas in cetacean research
Schulz et al. 2011 Bermant et al. 2019] Sharma et al. 2024a].
3. Social unit classification: The lowest level of sperm whale social structure called social units (SU) and have stable, matrilineally-rehhe rellted membership of females and their young [Christal et al. 1998). Importantly, all SUs in DSWP+CETI belong to the same vocal clan and thus share a common repertoire of coda types, making this more of a speaker identification task than dialect classification
'By analogy to human language, consider the task of classifying speakers by city of origin. It would be easier to distinguish between speakers from cities that use different dialects (simply classify the dialect). Importantly, in our data, all speakers the same dialect.

---

Table 1: Classification accuracies (%) of different audio embeddings. Each classifier head was trained using three different random seeds, with ttderr reported. The Random baseline uses a randomly initialized AVES model (training only the classifier), while Majority predicts the most common class.
TASK
WHAM
BASELINE
AVES
BIRDNET
CLAP
RAND.
MAJ.
DETECTION 91.3++0.2
60.9
60.9
92.8+00.1
93.0+1.0
96.8+1.4
RHYTHM
87.4+1.6
66.3
60.9
90.4+1.6
88.6+0.2
92.4+2.4
SOCIAL UNIT
70.
42.5
35.1
92.0=1.77
93.210.1
1.
VOWEL 85.212.5
66.3
66.3
91.8.12.9
85.914.66
84.3
4. Vowel type: Given a coda recording, classify its vowel-like pattern [Beguš et al. 2025
Table shows classification accuracies for each task. expected, non-generative embeddings specifically designed for acoustic classification tasks outperform outperform WhAM. We view those as a ceiling "sanity check" than a bascline. AVES and BirdNET perform particularly well on more specialized bio-acoustic tasks, due the fact that they were both trained on large amounts of animal vocalizations. Notably, WhAM's representations are useful despite being trained only for generation, outperforming both naive baselines. This suggests that meaningful acoustic features emerge naturally from training for coda generation, even without explicit supervision for these tasks.
We conducted an ablation study to assess how fine-tuning affects embedding quality by evaluating different WhAM variants with specific components removed (detailed Appendix The results reveal that fine-tuning did not significantly alter WhAM's downstream utility compared to base VampNet embeddings, despite WhAM's specialization on whale codas. However, as shown in Appendix D.3 species-specific fine-tuning was essential for enabling WhAM's core capability of translating audio into sperm whale vocalization acoustics.
Limitations and Future Work
The most immediate technical limitation concerns the audio codec architecture. Our current imple- mentation only finetunes the MATM while keeping the codec fixed (see Section This design choice, while computationally efficient, may limit the model's ability capture nuanced acoustic features specific to sperm whale vocalizations. For instance, the recently discovered vowel-like patterns in the 3.7-5.7kHz band [Beguš al./ (2025) may be inadequately represented by codec primarily trained on human music. Future work could explore either finetuning the entire codec or developing specialized codecs for bioacoustic signals.
Expert feedback (Section 4.2 highlighted specific limitations in click generation: unnatural onset and decay patterns, inconsistent background noise, and click properties more reminiscent of ccholocation than communication codas. These limitations might be addressed through architectural modifications, such incorporating adversarial components (Beguš al. 2023 introducing specialized modules that leverage domain knowledge about sperm whale click structure. Notably, the observation about echolocation-lie properties led to unexpected finding in our dataset preparation: the presence of echolocation sequences in datasets intended for communication codas. This discovery highlights a broader challenge in bioacoustics research-tth the difficulty of building clean, well-labeled datasets scale. Future work should focus on developing robust methods for distinguishing between different types vocalizations, perhaps by leveraging existing detection systems [Bermant et al. 2019).
These data quality challenges underscore the importance thorough evaluation protocols. Expanding the expert panel would provide more robust perceptual assessments, though acknowledge the practical challenges in recruiting specialists in sperm whale vocalizations. Additionally, developing more principled evaluation methods- and meta-evaluating these would help establish standardized benchmarks for bioacoustic generation tasks.
While our results demonstrate impressive performance with relatively small orddrss of magnitude smaller than typical in modern acoustic model training training-scaling up the training data could yield substantial improvements. This would require significant effort in aggregating and preprocessing

---

additional sperm whale datasets, as our experience with DSWP+CETI highlighted the technical challenges involved in preparing bioacoustic data for machine learning pipelines.
Looking beyond technical improvements, future work could explore unsupervised learning approaches to uncover new coda features, following the success of similar approaches in bioacoustics (Begus) et al. 2025]. This could lead to discoveries about sperm whale communication that complement traditional analytical methods while providing new directions for improving generative models of animal vocalizations.
Our methodological Tramework-froee the two-phase training approach to the expert evaluation protocol-come be adapted for studying other animal communication systems. Our experience suggests that success will require careful attention laboration with domain experts who can identify subtle but important characteristics of vocalizations.
-ppccise- acoustic features and close col-
The gap between generating vocalizations and understanding their meaning remains vast. While WhAM represents the first attempt at acoustic translation in the context of sperm whale commu- nication, future work should explore ways to bridge this semantic gap while maintaining minimal assumptions about the underlying communication system.
Acknowledgments and Disclosure of Funding
We thank Guy Gubnitsky, Yaly Mevorach, and Pernille Tonnesen for volunteering their time and expertise to the expert perceptual study. We thank anonymous reviewers for helpful comments, and in particular reviewer fPAV for suggesting adding BirdSet [Rauch et al.) 2025 to the domain adaptation stage.
This study was funded by Project CETI via grants from Dalio Philanthropies and Ocean X: Sea Grape Foundation; Virgin Unite and Rosamund Zander/Hansjorg Wyss through The Audacious Project: collaborative funding initiative housed at TED. This work was supported, in part, by National Science Foundation Award # 2222369.
Fieldwork for The Dominica Sperm Whale Project, which produced many of the coda recordings used in this work, was supported by a FNU fellowship for the Danish Council for Independent Research supplemented by Sapere Aude Research Talent Award (1325-00047A), a Carlsberg Foundation expedition grant (CF14-0789), a grant from Focused on Nature, two Explorer Grants from the National Geographic Society (WW-218R-17 17 and grants from the Arizona Center for Nature Conservation, Quarters For Conservation to SG, with supplemental funding from the Dansk Akustisks Selskab, Oticon Foundation, and the Dansk Tennis Fond to Pernille Tonnesen. Further funding was provided by Discovery and Equipment grants from the Natural Sciences and Engineering Research Council of Canada (NSERC) to Hal Whitehead of Dalhousie University and FNU large frame grant and Villum Foundation Grant (13273) to Peter Madsen of the Marine Bioacoustics Lab at Aarhus University. We thank the Chief Fisheries Officers and the Dominica Fisheries Division officers for research permits and their collaboration in data collection; all the crews of R/V Balaena and The DSWP team for data collection, curation, and annotation of audio recordings and photolD; well as Dive Dominica, Al Dive, W.E.T. Dominica, and Wacky Rollers for logistical support while in Dominica. We are grateful to Kristian Beedholm of the Marine Bioacoustics Lab at Aarhus University for CodaSorter; well as Mark Johsson and Peter Tyack for in-kind contributions of Dtags and associated code.
19), and supplementary
References
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse H. Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghi- dour, and Christian Havno Frank. Musiclm: Generating music from text. CoRR, abs/2301.11325 2023. doi: 10.4850/ARRXI..30..1133 URL https:://dii..... org/10. 081444855//aa/ 2301 11325
Jenny Amphaeris, Graeme Shannon, and Thora Tenbrink. Overlap not gap: Understanding the relationship between animal communication and language with prototype theory. Lingua, 272: 103332, 2022. ISSN 0024-3841. doi: http:///li.nnm///..0oo//nnnnnaaa
103332. URL hhtpps
//wwwascienie www.sccincedirect..... wttp:e/ddireettcccccciiiieeeieee wtw.://cccccctccc..ccooccoocoopp
10

---

Jenny Amphaeris, Daniel T Blumstein, Graeme Shannon, Thora Tenbrink, and Arik Kershenbaum. A multifaceted framework establish the presence of meaning non-human communication. Biological Reviews, 98(6))1887-111) 2023.
Jacob Andreas, Gašper Beguš,, Michael M Bronstein, Roee Diamant, Denley Delaney, Shane Gero, Shafi Goldwasser, David F Gruber, Sarah de Haas, Peter Malkin, et al. Toward understanding the communication in sperm whales. Iscience, 25(6), 2022.
Ricardo Antunes, Tyler Schulz, Shane Gero, Hal Whitehead, Jonathan Gordon, and Luke Rendell. Individually distinctive acoustic features in sperm whale codas. Animal Behaviour, 81()::73 730, 2011. ISSN 0003-3472. doi: https://oii.rrg/ 1016 1ttp:///......1...aaaaaa00000000 wnw.aa..20....2...... 12.011, URL https 7/www. wwwsscceiieentt...ccccc..eeeeeee http:///aaaaaieerrrraaa/////////
Richard H Backus and William E Schevill. Physeter clicks. Whales, dolphins and porpoises, 510: 527, 1966.
Gašper Beguš, Andrej Leban, and Shane Gero. Approaching an unknown communication system by latent space exploration and causal inference. arXiv preprint arXiv:2303.10931, 2023.
Gašper Beguš, Ronald I.. Sprouse, Andrej Leban, Marcelo Silva, and Shane Gero. Vowel- and diphthong-likk spectral patterns in sperm whale codas. Open Mind, pages 1 1-26, 2025. doi: 10.1 12/16000MMM..2222
Christian Bergler, Hendrik Schröter, Rachael Xi Cheng, Volker Barth, Michael Weher, Elmar Nöth, Heribert Hofer, and Andreas Maier. Orca-spot: An automatic killer whale sound detection toolkit using deep learning. Scientific reports, 9(1):10997, 2019.
Christian Bergler, Alexander Barnhill, Dominik Perrin, Manuel Schmitt, Andreas K. Maier, and Elmar Nöth. ORCA- DRCA-WHISPER: automatic killer whale sound type generation toolkit using deep learning. In Hanseok Ko and John H. L.. Hansen, editors, 233rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, pages 2413-2417, ISCA, 2022. doi: 10.2445//NTERSPECCH 2022-8446 URL 2022-846
https://doi org/00..rgg11... 10. 211.//1111eee
Peter C Bermant, Michael M Bronstein, Robert Wood, Shane Gero, and David Gruber. Deep machine learning techniques for the detection and classification of sperm whale bioacoustics. Scientific reports, 9(1): 2019.
Robert C Berwick and Noam Chomsky. Why only us: Language and evolution. Cambridge, MA:
MIT Press, 2016.
Peter B Best. Social organization in sperm whales, physeter macrocephalus. In Behavior of marine animals: Current perspectives in research, pages 227-289. Springer, 1979.
Rhythm Bhatia and Tomi H. Kinnunen. An initial study birdsong re-synthesis using neural vocoders. In R. Mahadeva Prasanna, Alexey Karpov, K. Samudravijaya, and Shyam S. Agrawal, editors. Speech and Computer 24th International Conference, SPECOM 2022, Gurugram, India, November 14-16, 2022, Proceedings, volume 13721 of Lecture Notes in Computer Science, pages 64-74. Springer. 2022. doi: 00.1007/978-3-03111 978-3-031 20980-992222 2__
2\_
URL httpt:t////o:: org 1007/
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. JEEE ACM Trans. Audio Speech Lang. Process. 31:2523-2553 2533, 2023, doi: 10.1109/AASLP22233288888 URL https:::::/ //doi 2023. 3288409
org 10.
Benjamin Britten. The Young Person's 's Guide to the Orchestra, Op. 34: Variations and Fugue on a Theme of Purcell. Boosey & Hawkes, London, 1945.
Claudio Campagna and Daniel Guevara. "save the whales" for their natural goodness. Marine Mammals: The Evolving Human Factor, pages 397-424. Springer, 2022.
11

---

Maurício Cantor and Hal Whitehead. How does social behavior differ among sperm whale clans? Marine Mammal Science, 31()):275-1129, 2015.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1 11305-11315. IEEE, 2022. doi: 10.1 109/
CVPRS2668.20022.01100 URL httpsttp::4 /tw.///o..oo org/ 201111/////2222 2022. 01103
Guobin Chen, Tony X. Han, Zhihai He, Roland Kays, and Tavis Forrester. Deep convolutional neural network based species recognition for wild animal monitoring. In 2014 IEEE International Conference on Image Processing, ICIP 2014, Paris, France, October 27-30, 2014, pages 858-862. IEEE, 2014. doi: 10.1 (0.1109/CCI22201470255577 URL https://ddoi: org/ 10. 1109/ ICI4.2014 70221722
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical semantcc audio transformer for sound classification and detection. In IEEE International Conference Acoustics, Speech and Signal Processing, ICASSP, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Marc Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 15084-15097, 2021.
Jenny Christal and Hal Whitehead. Social affiliations within sperm whale (physeter macrocephalus) groups. Ethology, 107(4):323-340, 2001.
Jenny Christal, Hal Whitehead, and Erland Lettevall. Sperm whale social units: variation and change. Canadian Journal of Zoology, 76(8):143 1431-1400,, 1998.
Kevin R Coffey, Ruby E. Marx, and John F Neumaier. Deepsqueak: deep learning- system for detection and analysis of ultrasonic vocalizations. Neungssychphanmacolog. 44(5):859 868. 2019.
Marie Comuzzo. Singing with whales: Exploring human and non- non-human connections. SEM Student News, 19(1), 2023. ISSN 2578-4242. URL h/tpp://cdn ymaws com/ wtw.m//thnoosscooogoooooo site-ym- wwt.coo//ooae. wwwm//eesuurceeeee http://111mmmmmmeeeeeeeeeeeeeee
a660 19c3e0f7c83e/
publications/ 19 1mlsmmsiicc www.ssccccttoocceecccoocaaaaaaaa Sww_s_semssssm__e__e 19- 1dd Music and the Anthropocene.
Special Issue:
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pages 7057 2019. URL https::/ htw.o/eedddiinnnnnnii.ses.. cc/paper/2011 / hhaa://00cccccc2244446f666626263 Abstract. html
Volker B Deecke. Studying marine mammal cognition in the wild: a review of four decades of playback experiments. Aquatic mammals, 32(4):461-482, 2006.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Trans. Mach. Learn. Res., 2023, 2023. URL http://ppenrrviie.eete///oo
Id=ivCaBz82R2
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Pupers), pages 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N 1123/11/N11... 1423. URL
https://dee org/ 10. 18653/v1/n19- 1423
12

---

Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: generative model for music. CoRR, abs/2005.00344 00041,, 2020. URL https://arxiv org/ Stg/abs//200 2005. 00341
Hao-Wen Dong, Wen- Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Sheila A. MclIraith and Kilian Q. Weinberger, editors, Proceedings of the T'hirty-Seconl AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI- 18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 34 41. AAAI Press, 2018. doi: 10. 10.1609/AAAAL3321.1133
URL wwwphhttpp:ddeee org 1609 aaai v32i1 11312
Alaina Claire Feldman. Minor listening, major influcecce: Revisitinn songs of the humpback whale. e-flux journal, (118), May 2021. URL https: flux. www///onna//////////3300 minor- wistening-nmt maajaajor- jor hatteannnn-nnirrrrrraaaaaeeeeeee ofwp::ee-rrrrititnooooooo---eeee
Accessed: October 2025.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76 (5):378, 1971.
Frederic Font, Gerard Roma, and Xavier Serra. Freesound technical demo. In Alejandro Jaimes, Nicu Sebe, Nozha Boujemaa, Daniel Gatica-Perez, David A. Shamma, Marcel Worring, and Roger Zimmermann, editors, ACM Multimedia Conference, MM '13, Barcelona, Spain, October 21-25, 2013, pages 41 1-412. ACM, 2013. doi: 10. 10.1145/2502081 255.0022220155225555. 25502445. 10. 1e.1115522020111 250002224
https: //otps://doi.. org/
Hugo Flores García, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. In Augusto Sarti, Fabio Antonacci, Mark Sandler, Paolo Bestagini, Simon Dixon, Beici Liang, Gaël Richard, and Johan Pauwels, editors, Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023, pages 359-366, 2023, doi: 10.581/ZENOD0.10265299 URL https://doi. org/r//// 10. 5281 2000///zeooo..o 10265299
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, pages 776-780, IEBE, 2017a. doi: 10. N0..109/CCASSP2207...999
URL htttttttt//o..ooo org wttp:///......./////// 1ww.11119IICSSSSS 2017 7952261
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, pages 776-780, IEEE, 2017b. doi: 10.1 10.1109ICCASPP2207...9222
URL https:tt////tttt //doi http://do..orgg///////o
www.:O/IIIASSSIO 2017 7952261
Shane Gero, Anne Bøttcher, Hal Whitehead, and Peter Teglberg Madsen. Socially segregated, sympatric sperm whale clans in the atlantic ocean. Royal Society Open Science, 3(6): 160061, 2016a.
Shane Gero, Hal Whitehead, and Luke Rendell. Individual. unit and vocal clan level identity cues in sperm whale codas. Royal Society Open Science, 311::100372, 2016b.
Shafi Goldwasser, David Gruber, Adam Tauman Kalai, and Orr Paradise. A theory of unsupervised translation motivated by understanding animal communication. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Axel-Christian Guei, Sylvain Christin, Nicolas Lecomte, and Éric Hervet. Ecogen: Bird sounds generation using deep learning. Methods in Ecology and Evolution, 15(1):69-79, 2024.
13

---

Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Emmanouilidou. Adapting frechet audio distance for generative music evaluation. IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, pages 1331-1335. IERE, 2024. doi: N0.1109/CASS4444555.. 104466444..46..... URL https 2024. 10446663
7/doi. org/ 10. 1119/IIASSP4BBEE
Luiz G. Hafemann, Luiz S. Oliveira, and Paulo Rodrigo Cavalin. Forest species recognition using deep convolutional neural networks. In 22nd International Conference on Puttern Recognition, ICPR 2014, Stockholm, Sweden, August 24-28, 2014, pages 100-11077. IEEE Computer Society, URL https: : /doi. org/ 10. 1109/ICPR,, 2014.1999999
2014. doi: N0..109/ICPR.21111
Masato Hagiwara. AVES: animal vocalization encoder based on self-supervision. In IEEE Inter- national Conference on Acoustics, Speech and Signal Processing /CASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pages 1-5. IEEE, 2023. doi: 10. (101////AAASS9337200000000000 URL attts://ddo org/10 1109 I0A.0//99SS99957.... 2023. 10095642
Taylor A. Hersh, Shane Gero. Luke Rendell, Maurício Cantor, Lindy Weilgart, Masao Amano, Stephen M. Dawson, Elisabeth Slooten, Christopher M. Johnson, Iain Kerr, Roger Payne, Andy Ro- gan, Ricardo Antunes, Olive Andrews, Elizabeth L. Ferguson, Cory Ann Hom-Weaver, Thomas F. Norris, Yvonne M. Barkley, Karlina P. Merkens, Erin M. Oleson, Thomas Pilkington, Jonathan Gordon, Manuel Fernandes, Marta Guerra, Leigh Hickmott, and Hal White- head. Evidence from sperm whale clans of symbolic marking in non-human cultures. Proceedings of the National Acudemy of Sciences, (19(6)))220069222211 19, 2022. doi: 10. 10710/0ass.2201699) 19. URL htttps//wwad pnas. org doi / abs/ 1073/ pmas_ 2201692119
James F.
Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, pages 131-135. IEEE, 2017. doi: 10.1009IAASP::21777992133
URL htttt:::/aaoo org/ 10. /IAOS/ICASP..... 2017 7952132
Catherine Hobaiter, Kirsty E Graham, and Richard W Byrne Are ape gestures like words? out- standing issues in detecting similarities and differences between human language and ape gesture. Philosophical Transactions of the Royal Society B, 377(1866):201033011 2022.
Yang Honghui and Fang Lanhao. Simulation of marine mammal calls in deep-sea environment. In International Conference on Autonomous Unmanned Systems, pages 2911 1-2920. Springer, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE ACM Trans. Audio Speech Lang. Process., 29:3451 14660000. 2021. doi: 10.1109/AALLP2220131222999 3122291
URL https://doi org (0rgg/1..00 1109 TASLPP 2021
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL //ppr///eeeeeeeeeeeeeeeeeeeeeeee net/ west//foummiddddin fot.oorrmiiddeeeeeeeeee /
Mark P Johnson and Peter I. Tyack. A digital acoustic recording tag for measuring the response of wild marine mammals to sound. IEEE journal of oceanic engineering, 28(1):3-12, 2003.
Stefan Kahl, Connor M Wood, Maximilian Eibl, and Holger Klinck. Birdnet: A deep learning solution for avian diversity monitoring. Ecological Informatics, 61:101236, 2021.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. Transformer- direct speech-to-speech translation with transcoder. In IEEE Spoken Language Technology Workshop. SL.7' 2021, Shenchen, China, Jamuary 19-22, 2021, pages 958-965, IEEE, 2021. doi: 10. 1099100///T449000000000 24919994990002229993336666 .9333496..
URL https: org/ 10. 1119//LTT89000 2021 9383496
Salman H. Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Comput. Surv., 54( 10:)::00:: 2022. doi: 10. 10.1145/350224, URL https: / //oii org/10 1145/3505244
14

---

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In Gemott Kubin and Zdravko Kacic, editors, 20th Annual Conference of the International Speech Communication Associa- tion, Interspeech 2019, Graz, Austria, September 15-19, 2019, pages 2350-2354. ISCA, 2019. doi: MM214437/NTERSSPHHHH220000 2019-2219
Utthttps: //oi: org/ 10. 21437 I
Yongcheol Kim, Seunghwan Scol, Hojun Lee, Geunho Park, and Jachak Chung. Whistlegan for biomimetic underwater acoustic covert communication. Electronics, 13(5):964, 2024.
Stephanie I. King and Frants H Jensen. Rise of the machines: Integrating technology with playhack experiments to study cetacean social cognition in the wild. Methods Ecology and Evolution, 14 (8):1873-1886, 2023.
Ekaterina Kopets, Tatiana Shpilevaya, Oleg Vasilchenko, Artur Karimov, and Denis Butusov. Gener- ating synthetic sperm whale voice data using stylegan2-ada Big Data and Cognitive Computing, (4):40, 2024.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved RVQGAN. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9: 9.1336-1355 2021. doi: 10.1162/tacl a 00430. URL https
//aclanthology... org. 2021 tacl-a.cl111
Antonio Leitao, Maxime Lucas, Simone Poetto, Taylor A. Hersh, Shane Gero, David Gruber, Michael Bronstein, and Giovanni Petri. Evidence of social learning across symbolic cultural barriers in sperm whales. eLife, May 2024. doi: 10.7554/elife. 9.362.1. URL http://dxx..... doi
org/10. 75110..554/40 http//Lifee......
Bertel Mohl, Magnus Wahlberg, Peter T Madsen, Anders Heerfordt, and Anders Lund. The monopulsed nature of sperm whale clicks. The Journal of the Acoustical Society of America, 114 (2):1143-1154 2003.
Karen E Moore, William A Watkins, and Peter L Tyack. Pattern similarity in shared codas from sperm whales (physeter catodon). Marine Manmal Science, 9(1):1-9, 1993.
Bernard Moussad, Rahmatullah Roche, and Debswapna Bhattacharya. The transformative power of transformers in protein structure prediction. Proceedings of the National Academy of Sciences, 120632);e2303499122 2023.
Roger Payne. Songs of the humpback whale, 1970. See, in particular, "Listening Instructions".
Roger S Payne and Scott Mcy. Songs of humpback whales: Humpbacks emit sounds in long, predictable patterns ranging over frequencies audible to humans. Science, 173(3997):585-577 597, 1971.
Marika Premoli, Daniele Baggi. Marco Bianchetti, Alessandro Gnutti, Marco Bondaschi, Andrea Mastinu, Pierangelo Migliorati, Alberto Signoroni, Riccardo Leonardi, Maurizio Memo, et al. Automatic classification of mice vocalizations using machine learning techniques and convolutional neural networks. PloS one, 16(1)ee244466, 2021.
Lukas Rauch, Raphael Schwinger, Moritz Wirth, René Heinrich, Denis Huseljic, Marek Herde, Jonas Lange, Stefan Kahl, Bernhard Sick, Sven Tomforde, and Christoph Scholz. Birdset: large-scale dataset for audio classification in avian bioacoustics. In The Thirfeenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
URL http:p///oppaeeiiee..eef//ooooo/
15

---

Johnny Reilly, John D Goodwin, Sihao Lu, and Andriy S Kozlov. Bidirectional generative adversarial representation learning for natural stimulus synthesis. Journal of Neurophysiology. 2023.
Luke Rendell, Sarah L Mesnick, Merel Dalebout, Jessica Burtenshaw, and Hal Whitehead. Can genetic differences explain vocal dialect variation in sperm whales, physeter macrocephalus? Behavior genetics, 42:332-344 2012.
Luke E Rendell and Hal Whitehead. Vocal clans in sperm whales (physeter macrocephalus). Pro- ceedings of the Royal Society of London, Series B: Biological Sciences, 270(1512):2252331, 2003.
César Rodriguez-Garrvito, David F. Gruber, Ashley Otilia Nemeth, and Gašper Beguš. What if we understood what animals are saying? the legal impact of Al-assisted studies of animal communication. Ecology Law Quarterly, 52(1), 2025, doi: 10. 10.1577997333X885500 URL 15779/2388888N50 Forthcoming, Fall 2025.
https://doi.org
Laela Sayigh, Mary Ann Daher, Julie Allen, Helen Gordon, Katherine Joyce, Claire Stuhlmann, and Peter Tyack. The watkins marine mammal sound database: An online, freely accessible resource. In Proceedings of Meetings on Acoustics, volume 27. AIP Publishing, 2016.
Tyler M. Schulz, Hal Whitehead, Shane Gero, and Luke Rendell. Individual vocal production in a sperm whale (physeter macrocephalus) social unit. Marine Mammal Science, 27(1):149-166, URLhtttp:///ooniiilliirre
2011. doi: https://doi..rr//1.... http:///doooro//1..0//// 11..1/001111111..1.... 1748- 7010 wiley. saaley.ccom/di//ab///000 1111/j 779217697777777777 2010. 000000039.... x
2010.00399.x
Pratyusha Sharma, Shane Gero, Roger Payne, David F Gruber, Daniela Rus, Antonio Torralba, and Jacob Andreas. Contextual and combinatorial structure in sperm whale vocalisations. Nature Communications, 15(1):3617, 2024a.
Pratyusha Sharma, Shane Gero, Daniela Rus, Antonio Torralba, and Jacob Andreas. Whalelm: Finding structure and information in sperm whale vocalizations and behavior with machine learning. bioRviv, 2024b. doi: 10.1100/022.10.11..000 10.11,61011.... URL httpstts::/ www. biorxiv org/
621077
Toshitaka N Suzuki, David Wheatcroft, and Michael Griesser. The animal vocal communication. Philosophical Transactions of the Royal Society B, 375(1889): 20180405, 2020.
interface in
Peter Tyack. Differential response of humpback whales, megaptera novacangliae, to playback of song or social sounds. Behavioral Ecology and Sociobiology, 13:49-55, -5, 1983.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Alan W. Black, editor, The 9th ISCA Speech Synthesis Workshop, SSW 2016, Sumnyvale, CA. USA, September 13-15, 2016, page 125, ISCA, 2016. URL https::/www. wwsscaaarcchin org http:/swww010///////222/2222111e http://ww/..//eaaaooorrrrdd----/ hta.//adndooddd..
ssw. html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Ro- man Garnett, editors, Advances Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 5998 6008, 2017. URL https http:/://////oe//eee///omoo www.oeeeings..nuuri....... nww.inurrnicomm cc/ ceturipps.c//pppr/21117 httpp//pppppppeeeeeeeppp /hash/
Abstract html
William A Watkins and William E Schevill. Sperm whale codas. The Journal of the Acoustical Society of America, 62(6):1455 1977.
Linda Weilgart and Hal Whitehead. Coda communication by sperm whales (physeter macrocephalus) off the galapagos islands. Canadian Journal of Zoology. 7 71(4):744-752 1993.
Linda Weilgart and Hal Whitehead, Group-specific dialects and geographical variation in coda repertoire in south pacific sperm whales. Behavioral Ecology and Sociobiology, 40:277-285, 1997.
16

---

Hal Whitehead. Sperm whales: social evolution in the ocean. University of Chicago press, 2003,
Hal Whitchead and Linda Weilgart. Patterns of visually observable behaviour and vocalizations in groups of female sperm whales. Behaviour, pages 975-296, 1991.
Yusong Wu*, Ke Chen*, Tianyu Zhang*. Yuchen Hui*, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audii pretraining with feature fusion and keyword-to-captin augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
Jiangjian Xie, Yujie Zhong. Junguo Zhang, Shuo Liu, Changqing Ding, and Andreas Triantafyllopou- los. review of automatic recognition technology for bird vocalizations in the deep learning era. Ecological Informatics, 73: 73:101927, 2023.
Lian Xu, Mohammed Bennamoun, Senjian An, Ferdous Sohel, and Farid Boussaid. Deep learning for marine species recognition. Handbook of deep learning applications, pages 129- 145, 2019.
Yossi Yovel and Oded Rechavi. Ai and the doctor dolittle challenge. Current Biology, 33(15):
R783-R787, 2023.
Lue Zhang, Hai-Ning Huang, Li Yin, Bao-Qi Li, Di Wu, Hao-Ran Liu, Xi-Feng Li, and Yong- Le Xie. Dolphin vocal sound generation via deep wavegan, Journal of Electronic Science and Technology, 20(3):100171, 2022.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Fréchet Audio Distance evaluations can be found in Section 4.1 Percep- tual studies with expert marine biologists can be found in Section 4.2 Experiments on downstream classification are reported Section 4.3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes, in Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
17

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Details on model training are in Appendix F Details on the experimental setup are in Appendix E.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same datasct, provide access to the model. In general. releasing code and data is often
18

---

one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model ge.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release the model weights, and the code used for training and evaluating the model upon publication. We will attempt to release as much of the data as possible, however we note that the data was collected by a large collaboration of marine biologists over several decades, and so we cannot commit to getting their consent to publish by publication time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ public/guides (wooeo/ummmiiiioooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https ctips.cc/publlc//eeeccc
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
19

---

Answer: [Yes]
Justification:
Yes, in Appendices E and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All figures depict error bars standard error figures, with the exception of Figures and and table For Figures and repeating the experiments with multiple seeds would have been computationally prohibitive due to the amount of categories each experiment. For Table 2 this is a minor, supplementary used to justify an experimental design choice, rather than main component of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computer resources needed train and finetune the model are reported in Appendix E.2 The resources needed to conduct the experiments (after training the model) are reported Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
20

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://ewrrr...c.pppp.tttttcccc/
Answer: [Yes]
Justification:
We adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper poses such risks.
21

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We adhere to the license of all external code we use. The data used was either collected by us (the authors), used accordance to the license of the dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpssspppppsitaeeee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide as much data appropriate and possible of the new datasets we used, without de-anonymizing ourselves. The training method and model are properly documented in Appendix
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
22

---

Answer: [Yes]
Justification:
In Appendix E.5
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [No]
The perceptual listening study presented no risk to participants. Participants were asked to evaluate audio similar to those they regularly analyze their professional work. No personal data was collected beyond coarse background questions and the assessment itself (see Appendix E..), and no compensation was provided. The task falls within the participants' scope of their normal research activities and expertise.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should should not be described.
A Broader impacts
Our work modeling sperm whale communication has potential implications for both scientific understanding and conservation efforts. Historically, advances understanding cetacean communi- cation have played crucial roles in conservation conservation-nntaable notably, the discovery of humpback whale song by
23

---

Payne and Vay 1971 contributed significantly to public awareness and the subsequent "Save
the Whales" movement Feldman 2021 Campagna and Guevara 2022 Comuzzo 2023). While we maintain that sperm whales deserve protection regardless of our ability to understand their communi- cation, we recognize that deeper scientific understanding often catalyzes public engagement with conservation efforts.
Our model's capabilittes might naturally suggest applications in behavioral experiments through playback studies. This is particularly tempting given that sperm whales often produce codas simultaneously behavior that our bidirectional model could theoretically capture by conditioning on one whale's s clicks while generating the overlapping clicks of another. However, we strongly caution against such applications at this stage. Without deeper understanding of coda semantics and functionality, playback experiments using synthetic vocalizations could have unintended and potentially harmful consequences for these social marine mammals. Instead, we propose that this work demonstrates the potential of learning from passive acoustic iinervation-studyye markable animals through careful listening rather than active intervention. With this approach, this work could potentially play role in assisting efforts reinforce existing protections create new legal protections for whales [Rodriguez-Garann Garavito al. 2025).
these re-
As noted in Section 5 our methodological framework could extend beyond sperm whales, potentially benefiting research on other marine mammals and, more broadly, any species that communicates acoustically. This scalability is particularly relevant as biodiversity monitoring becomes increasingly critical in the face of environmental changes. However, our experience underscores the importance of deep col loborrttio with domain experts-the the success of this work relied on guidance from marine biologists and acousticians with decades of experience studying sperm whales. We encourage future work in this direction to similarly prioritize partnerships with species- species-specific domain experts, as their insights are crucial for both model development and responsible deployment.
B Preliminaries on sperm whale codas
Figure The sperm whale head contains the spermaceti organ (c), a cavity filled with almost 2kL of wax-like liquid, and the junk compartment (f), comprising series of wafer-like bodies believed to act as acoustic lenses. The spermaceti organ and junk act as two connected tubes, forming a bent, conical horn of about 10m in length and 0.8m aperture in large mature males. The sound emitted by the phonic lips (i) in the front of the head is focused by traveling through the bent horn, producing a flat wavefront at the exit surface. Reproduced with permission (Andreas et al. 2022 C Alex Boersma 2021).
Sperm whales have evolved remarkable acoustic capabilities. Figure ( illlustaal the key anatomical and acoustic aspects of these capabilities, which form the basis for their complex communication system.
24

---

2048
ECHOLOCATION
TIME (seconds)
CODAS
Enter interved
ampltude
Jea
IPV pube intenso
paie stansl
Typical temporal structure of sperm whale echolocation and coda clicks. Echolo- cation signals are produced with consistent inter-click intervals (of approximately 0.4s) while coda clicks are arranged in stereotypical sequences called "codas" lasting less than 2s. Codas are character- ized by the different number of constituent clicks and the intervals between them (called inter-click intervals). Codas are typically produced in -urty exchanges that can last from about 10s to over half hour. Each click, turn, presents itself as a sequence of equally spaced pulses, with inter-pulse interval of an order of 3-4ms in an adult female, which is the result of the sound refecting within the spermaceti organ. Right: Typical structure of echolocation (dark blue, left) and coda clicks (light blue, right). When observed as a waveform zoomed into single click the type types of clicks differ observably in structure. There is far greater attenuation between the first and second pulse of echolocation click, then the coda clicks. Further, the coda clicks resonate more in the nose of the sperm whale creating additional pulses after the first one for coda clicks. Reproduced with permission (Andreas al. 2022 ② Alex Boersma 2021).
Figure 7: Left:
Sperm whales live in a multileveled social structure with female lines living together in 'units" with stable membership [Whitehead] [203)] Early acoustic research proposed that codas might serve as individual signatures [Watkins and Schevill 1977), but subsequent studies instead suggested that different coda types may have distinct functions [Antunes et al. [011]. and that variation of coda usage among units suggested a function in unit-level social identity [Moore et al.) 1993 Weilgart and Whitchead 1993] 1997). Even when living in the same waters, whales from different social units will only associate with units which share a similar repertoire of codas. This social segregation based on acoustic similarity was used to delineate the highest level of social organization which structures their populations, the vocal clan; and that codas function as symbolic markers of these cultural groups [Rendell and Whitehead 2003 Gero et al. 2016 Hersh al. 2022). Importantly, there is good evidence that these distinct dialects codas, with variation in number of clicks, as well as rhythm and tempo, the product of social learning, and not genetically inherited [Cantor and Whitehead 2015 Rendell et al. 2012].
The clicks produced by sperm whale can be generally classified as either ecolocation coda clicks. Echolocation clicks which function in navigation and hunting in the dark, wherein echoes of the clicks return and are interpreted by the whales in the darkness of the deep occan, much like bats in the night sky. Conversely, coda clicks are thought to function in communication between whales and exchanged between whales or groups of whales in social contexts at the onset of dives, during shallow dives near the surface, and during large social interactions.
Echolocation signals are produced with consistent inter-click intervals while coda clicks are arranged stereotyped, rhythmic sequences called "codas" lasting less than seconds. Codas are characterized by the different number of constituent clicks and the intervals between them (called inter-click intervals or ICIs). Rhythmic patterns and tempo of clicks define coda 'types", which are often given descriptive names. For example, a 1+1+3 coda is click-pause-click www.ll--pa--eele---iliiee click-ck--ikk-llie--- click (Figure
25

---

C A listener's guide to codas
Building on findings from our Expert Perceptual Study (Section 4.2) we present short guide detailing perceivable similarities and differences between natural and synthetic codas. We note that, unlike the Expert Study, this guide was developed by the authors under no time constraints, and with unrestricted aid of spectrograms and familiarity with model internals. This Listener's Guide to Codas structured as unifying Theme, followed by four Variations each isolating a specific cue For broad-audience listener's guide to whale (albeit humpback) vocalizations, see Payne 1970).
Theme. Synthetic codas generated by WhAM can be evaluated both visually and acoustically, using the same structural cues that characterize authentic sperm whale clicks. Each natural coda click typically consists of a sequence equally spaced pulses, with an inter-pulse interval (IPI) of approximately 3-4 ms in adult females. This is consequence of internal reflections within the spermaceti organ.
Variation A: Balance. DC offset (a shift of the waveform away from being centered zero) does sometimes occur when recording sperm whales in the wild, particularly when using handheld recording systems which run off batteries and constant DC voltage. It is often consistent, while synthetically generated clips will have quite a "wavy" offset. It is however interesting to note that WhAM picked up this feature of the authentic waveforms. In addition, sperm whales do not vary the amplitude dramatically between sequential clicks within codas, while WhAM generated codas sometimes do.
Variation B: Frequency. Using a spectrogram, one can see that the frequency content of synthetic clicks is more uniform. In Figure TO one compare the shape of spectrograms for otherwise relatively similar clicks and note that the shape is more uniform and consistent both time and frequency for synthetic clicks (bottom strong orange rectangles) compared authentic clicks which trail off both as frequency increases and across time (top, more pointed at top, with far less yellow above 10kHz, and rough along the right side). In addition, you can also observe the variation in amplitude across synthetic clicks in the waveform, but a consistent amplitude in the waveform of the authentic clicks (as described above).
Variation C: Structure. Authentic sperm whale clicks, especially coda clicks, contain the typical multiplulsed structure with a detectable inter-pulse-interval created by the head of the sperm whale and the path of the sound as it is generated (outlined above). Synthetic clicks often did not have realistic structure either by having no pulsed structure (center of Figure 99 or an exaggerated one (right of Figure 9). While some of these effects occur in authentic clicks based on the angle of recording relative to the axis of the body of the whale making the sounds, the synthetic clicks rarely had realistic structure within clicks.
Variation D: Listening alone. Taken together, these waveform- and spectrogram- bassed based cues are sometimes audible even without visual aids. A trained ear could identify synthetic codas based on subtle irregularities in amplitude, spectral consistency, and the absence of realistic multipulsed structure.
'With apologies to Britten 1945
26

---

Figure 8: Sample of a synthetic coda generated by WHAM with the variable DC offset dissimilar to natural recordings (yellow arrow) and the dramatic variation in amplitude between sequential clicks (orange arrows).
Synthetic
Synthetic
IPI
Figure 9: Pulse structure of authentic (real) and synthetic clicks.
D Additional experiments
D.1 FAD Embedding Selection
The Fr' echet Audio Distance (FAD) measures similarity between audio datasets using embeddings to map the audio into feature space. The choice of embedding is crucial, different embeddings capture different aspects that prioritizes the temporal pattems critical coda structure over background noise. This appendix describes the calibration experiment we conducted to select the most suitable embedding for our FAD analysis.
signal. For analyzing sperm whale codas, we sought an embedding
Let the coda recordings in DSWP+CETT be denoted by (.1,.., Fn), we:
1. Created denoised versions (f1,..,, 2n detailed in Appendix E.I
2. Isolated the removed noise components {x1 f1, ....... En in)
3. For each candidate embedding fi, compared:
d FAD score between codas and their denoised versions:
FAD score between codas and their noise components:
We evaluated five audio embeddings VGGISH Gemmeke et al. [017b], Hershey et al. (2017)
Encodec-embd Défossez et al. [2023), LAION CLAAP Music, LATON CLAP Audio Wu et al.
2023], Chen al. [2022), and BirdNET Kahl et al. 202 using the Fréchet Audio Distance
27

---

7B
7A
Figure 10: Waveform and spectrograms both authentic clicks recorded from wild sperm whales from clip 7B (top) and from WhAM generated synthetic clicks from clip 7A (bottom). Here "7" is because this formed the seventh pair in the expert perceptual study (Part 1).
implementation of Gui et al. (2024). The ratio dj indicates how much more weight embedding i gives background noise versus temporal structure. A larger ratio indicates stronger emphasis on temporal patterns and better suitability for the quantitative assessment of audio translation experiment. Table 2 shows these ratios for each embedding.
Table 2: Comparison of Audio Embeddings for Temporal Structure Sensitivity
EMBEDDING
dj (CODA vs. DENOISED)
(CODA vs, NOISE)
dz/a.
VGGISH ENCODEC-EMBD
2.0844
1.5027
0.7209
25,9716
3.156
0.1215
LAION CLAP MUSIC
0.1483
0.1080
0.7282
LAION CLAP AUDIO BIRDNET
0.1144
0,1098
0.9597
16.7817
22.4761
1.3393
Based on these results, we selected BirdNET for our main FAD experiments, as it maximized the ratio of distances between raw-to-noise over raw-to-signal.
D.2 Downstream Task Ablation Study
To evaluate the contributions of different components in WhAM, we conduct an ablation study by progressively removing elements and assessing performance across the same set of downstream tasks. The results presented in Figure II
28

---

No finetuning. We test the effect of skipping domain-adaptinn adaptation (step (b) in Figure I or skipping finetuning of VampNet altogether (steps b.c) Figure T]. For all tasks except Social Unit classifica- tion, removing species-specific finetuning or domain adaptation does not have significant impact on the accuracy. This indicates that the inclusion of these steps in WhAM does not significantly degrade the performance on most downstream tasks.
Tokenizer-only. We falsify the hypothesis that the neural audio codec sufficient for capturing semantic properties in the audio by testing downstream classification directly on the acoustic tokens (Figure 2) without embedding them through the MATM. This causes statistically significant performance drop, particularly in Social Unit classification (-10.9 points, from 70.5% + 0.7% to 59.6% + 2.0%)
Detection
Rhythm
90
Performance (%)
Performance (%)
90
85
88
86
80
84
No Domain Adaptation
WhAM
Base Vamptiet.
"okknizer Only
No Domain Adaptation
WhAM
Base VampNett
Takenizer Only
Vowel
Social Unit
Performance (%)
80
Performance (%)
90
T
70
80
60
75
50
No Domain Adaptation
WhAM
Base VampNet
Tokenizer Only
No Domain Adaptation
WhAM
Base VampNet
Tokenizer Only
Figure 11: Classification accuracies (%) resulting from using the output of different WhAM compo- nents on downstream tasks. Each classifier head was trained using three different random seeds, with stterr reported.
D.3 Fréchet Audio Distance Ablation Study
To complement the ablation study of Appendix D.2 the experiments detailed in section 4.1 were repeated four times with marine mammal sounds. First, using only the tokenizer. Second, training the model with only Domain Adaptation (DA, step (c) in Figure [ skipping the Species Specific Fine-Tuning step (SSFA, step (c) in Figure [p. Third, training only with Domain Adaptation. And finally, using the full version of WhAM. These results (Figure 12) show that, expected, fine-tuning WhAM on sperm whale data results in outputs that are more similar to sperm whale vocalizations.
D.4 Tokenizer Reconstruction Loss Study
WhAM uses the Descript Audio Codec (DAC) as its tokenizer (Kumar et al. 2023 DAC is tailored towards speech, music, and environmental sounds. To test possible degradation in encoding sperm whale coda audio, we conducted the following experiment.
Let each individual coda recording in the DSWP+CETI datasets be denoted by æ1, 2...
29

---

Tokenizer
0.8
0.91
0.52
0.9
0.8
DA
0.58
0.34
0.68
0.57
0.71
0.69
0.76
0.82
0.5
0.61
0.73
0.83
0.76
0.43
0.7
0.6
SSFT
0.5
WhAM
0.15
0.34
0.38
0.42
0.28
0.16
0.37
0.37
0.25
0.27
0.14
0.32
0.18
0.2
02
Sperm Whole
Beeps
Clymene Dolphin
Narwhal
Atlantic Dolphin
Wairus
Longfin Whole
Risso
Beluga
Killer Whale
Ross Seal
Bearded Seal
Bowhead Whole
Leapard Seal
Figure 12: The effect of ablating components of the model on FAD
1. For each recording reconstructed version, di was created by passing into the tokenizer to generate tokens, then passing the tokens through the decoder recover audio recording.
2. Each and d was then sliced into chunks of length C' to calculate their respective short term fourier transforms. The transform represented by the arrays (Tr,1, 2.........m } and Each T., represents the magnitude all frequencies over the jth chunk of
the recording nę
3. The mean reconstruction accuracy, denoted by E is now given by taking the average normal- ized error between all i.j and S... using the formula:
xX 178 Σ (
The mean error shown in fig. 13 for chunk size of 2.27 ms and 22.7 ms. With smaller chunk size, E indicates which pitches the tokenizer accurately reconstructs and which pitches it does not. Using a larger chunk size, E gives an indication of what type of general noise patterns the tokenizer fails to include in its reconstruction.
As can be seen with a chunk size of 22.7 ms, the error wave-like. Since sinusoidal waves in the frequency domain correspond to impulses in the time domain, this suggests tendency to misrepresent impulse-Iis like sounds the time domain. On the other hand, using chunk size of 2.27 ms provides an indication as to what pitches the tokenizer prioritizes. The spikes in the bands 1-6 kHz and 8-10 kHz suggest that, in general, the tokenizer tends to perform relatively poorly in those frequencies. However, this degradation is not severe enough to prevent our model from generating natural-sounding codas (Section 4.2). nor from its embeddings to "capture" vowels (Section
30

---

2.27ms Chunks
38
36
26
24
2000
4000
6000
8000
10000
Frequency
22.7ms Chunks
50
///
y
20
2000
3000
4000
5000
6000
7000
8000
9000
10000
Frequency (Hz)
Figure 13: Tokenizer reconstruction loss study. Normalized mean squared error (y-axis) by frequency (x-axis).
E
Methodology Details
E.1
Data
The FSD50k dataset includes 3,159 audio recordings labeled with the "animal" tag, amounting total duration of hours and 45 minutes. Noisy segments were retained to preserve real-world variability in training data.
FSD.
AudioSet. The AudioSet dataset was used to supplement training with additional animal vocaliza- tions. contains 5h8m hours of audio.
BirdSet. Consists of 6,800 total hours of recordings containing bird vocalizations [Rauch et al. 2025]. Due to space constraints and to avoid training WhAM on audio that did not contain any vocalizations, only subset of the entire dataset used, containing total of 110 hours of data.
WMMS. The Watkins Marine Mammal Sound Database consists of raw, unlabeled audio recordings. The dataset contains total of hours and 8 minutes audio. Each recording was segmented into 10- second snippets for training. No additional denoising was applied. The dataset contained vocalizations from the following mammals (names as listed on the WMMS website):
Atlantic Spotted Dolphin
Bearded Seal
Beluga (White Whale)
Bottlenose Dolphin
Boutu (Amazon Dolphin)
River
Bowhead Whale
31

---

Clymene Dolphin
Commerson's Dolphin
Common Dolphin
Dall's Porpoise
Dusky Dolphin
False Killer Whale
Fin, Finback Whale
Finless Porpoise
Fraser's Dolphin
Grampus (Risso's Dolphin)
Gray Seal
Gray Whale
Harbor Porpoise
Harbour Seal
Harp Seal
Heaviside's Dolphin
Hooded Seal
Humpback Whale
Irrawaddy Dolphin
Juan Fernandez Fur Seal
Killer Whale
Leopard Seal
Long-Beaked (Pacific)
Long-Finned Pilot Whale
Common Dolphin
Melon-Headed Whale
Minke Whale
Narwhal
New Zealand Fur Seal
Northern Right Whale
Pantropical Dolphin
Spotted
Ribbon Seal
Ringed Seal
Ross Seal
Rough-Toothed Dolphin
Sea Otter
Short-Finned (Pacific) Pilot
Whale
Southern Right Whale
Sperm Whale
Spinner Dolphin
Spotted Seal
Steller Sea Lion
Striped Dolphin
Tucuxi Dolphin
Walrus
Weddell Seal
West Indian Manatee
White-beaked Dolphin
White-sided Dolphin
in 2000km² area off the coast of Dominica. Codas were recorded using various recording systems including far-field hydrophones and animal-borne tags. Recording setups
DSWP. The dataset consists of codas collected between
follows:
2005: A Fostex VF-160 multitrack recorder (k4. sampling rate) and a custom built towed hydrophone (Benthos AQ-4 elements, frequency response: 0. 0.1-30kHz) with a filter box with high-pass filters up to 1 kHz resulting in recording chain with flat frequency response across minimum of 2-20kHz.
2006: No recordings during this short season.
2007,2009,2011: A Zoom H4 portable field recorder Mo8kHz.sampligg rate) and a Cetacean Research Technology C55 hydrophone (frequency response: 0.02-44kHz) and no filters.
2008,2010,2012,2015: custom-built towed hydrophone (Benthos AQ-4 elements, frequency re- sponse: 0. 0.1-30kHz) with filter box with high-pass filters up to I kHz resulting in recording chain with a flat frequency response across a minimum of 2-20 kHz. This was connected to computer based recording system as part of the International Fund for Ani- mal Welfare's 's (IFAW) LOGGER software package (48kHz sampling rate) or PAMGUARD (minimum 48 kHz sampling rate). In addition, recordings were also made through the deployment of animal-bornt sound and movement tags (DTag generation 3, Johnson and 2y0c3
CETL All systems were sampling above 96kHz with 16bit resolution with minimum flat (+2dB) frequency response within 1-45kHz.
The DSWP and CETI dataset contain background noise such as water sounds. To improve model performance, we denoise datasets before training on the model. A noise profile of each recording in the frequency domain generated by sampling sections which did not contain codas. Then, perform spectral subtraction to remove noise in the frequency domain, and transform back to the time domain of the audio signal.
All audio samples were downsampled 16 kHz and normalized to have zero mean and unit variance when passed into VampNet.
32

---

Table 3: Quantitative Assessment Data Summary
FULL NAME
SHORTENED NAME
NUM. SAMPLES
ATLANTIC DOLPHIN
A. DOLPHIN
58
BEARDED SEAL
B. SEAL
37
BOWHEAD WHALE
WHALE
BELUGA WHALE, WHITE WHALE
BELUGA
50
WALRUS
WALRUS
38
CLYMENE DOLPHIN
C. DOLPHIN
63
NARWHAL
NARWHAL
LEOPARD SEAL
L. SEAL
10
LONG-FINNED WHALE
L. WHALE
10
KILLER WHALE (ORCA)
ORCA
35
Ross SEAL
Ross SEAL
50
RISSO's s DOLPHIN
RISSO
Table 4: Prompt settings for each input type.
INPUT
PERIODIC PROMPT
ONSET
NUM. OF STEPS
TYPICAL MASS
SAMPLE CUTOFF
MASK WIDTH
CODAS
12
21
50
0.102
0.17
BEEPS
12
21
50
0.102
0.17
DOLPHIN
74
0.15
0.39
B. SEAL
7
I
70
0.15
0.44
B. WHALE
7
I
70
0.15
0.44
BELUGA
13
13
85
0.15
0.39
WALRUS
18
107
0.15
0.33
C. D.LLLHNNNE
12
14
72
0.15
0.25
NARWHAL
6
4
39
0.15
0.21
L. SEAL
6
4
46
0.15
0.39
L. WHALE
15
19
57
0.15
0.42
ORCA
46
0.15
0.39
ROSS SEAL
18
3
66
0.15
0.49
RISSO
13
13
85
0.15
0.39
Note that, as with any self-supervised training setup that relies on random masking, the effective number of unique training examples fär exceeds the raw audio hours. In case: First, each 2-second audio snippet becomes 14 x 120 token array. Columns correspond to time steps, and rows represent acoustic granularity During training, entire columns (i.e., time steps) are masked random; with 120 columns, there are 2120 possible masking patterns per snippet. So, for example 20 hour dataset yields 36,000 snippets, which result in 10 possible masked training inputs.
E.1.1 Generating data for Sections 4.1 and 4.2
Three different input sources used generate samples for both the Quantitative Assessment of Audio Translation and the Expert Perceptual Evaluation. The prompt settings for each input type summarized in Table
Watkins Marine Mammals. Eleven species were selected from the "Best of Watkins Marine Mammals" dataset. Due to variations in vocalization characteristics and recording conditions, prompt settings were manually optimized for each species. These species and prompt settings can be found in Table
Digital "beeps". Five digital beep sequences were generated. Each snippet was initialized as zero-filled array at a 44.1 kHz sample rate. Clicks were simulated by selecting random indices and setting them to peak amplitude of 1. To ensure realistic timing and rhythm, real coda sequences were prepended to each generated sample before synthesis. These prepended codas were then removed after generation.
33

---

E.2
Model Training
The model training procedure consisted of two phases: domain adaptation and species fine-tuning.
Acoustic Tokenizer Settings. Discrete token vocabulary size (2) = 1024. Frequency of Input Audio Naam 16kHz. Tokenizer input length Nsec 10.
Domain Adaptation. In the first phase, the model was pretrained on a mixture of general animal vocalizations, including data from FSD and AudioSet. This step aimed establish broad under- standing of bioacoustic patterns. The model was trained for 500,000 iterations using the AdamW optimizer with learning rate of 0.0001. A batch size of was used, and gradient clipping was applied to stabilize training. The model took 123 hours to train using an AWS EC2 g5 25.2xeer instance (NVIDIA A10 GPU, vCPUs, 32 GB of memory).
Species-Specific Fine-Tuning Following domain adaptation, the model was fine-tuned on whale- specific data from DSWP+CETI to adapt its representations to sperm whale vocalizations, The fine-tuning process used the same optimizer and learning rate as the pretraining phase and batch size 6. Training continued for another 500,000 iterations. This took 39 hours to run using an AWS EC2 g5. axlage.
E.3
Computational costs
All experiments were run on an AWS EC2 g5.2xlarge (NVIDIA A10 GPU, 8 vCPUs, 32 GB of memory). A full run of FAD experiments took 3 hours with a full version of Vampnet, and 1.5 hours using a Tokenizer-only model, therefore Section 4.] and appendix D.3 approximately 7.5 hours in total. Appendix [.I took approximately 1.5 hours. For downstream classification, training the
linear probe took at most 5.5 hours; thus, Section 4.3 and appendix D.2 took about 16.5 hours in total.
E.4 Utility of Embeddings for Downstream Tasks
Model Details. We run forward pass through WhAM and AVES to obtain embeddings from the audio. Both WhAM and AVES output varying embeddings over time, so we average the embeddings over time to obtain unified embedding for 1 audio snippet. After the embedding obtained, we attach two-layer feed-forward neural network connected layer that projects the embedding into a 128-dimensional hidden layer, followed by ReLJU activation. A second fully connected layer then generates class probabilities.
classifier. The network consists of fully
We evaluate embeddings from WhAM and AVES, comparing their performance against a random embedding baseline as well as a majority baseline classifier.
Training Data. For downstream task evaluation, we leveraged annotations in the DSWP+CETI datasets. Using human-annotated timestamps, we identified and extracted audio segments containing codas, each spanning 1-2 seconds. Each coda was labeled for one the following classification tasks:
Coda Detection: Determine whether given audio snippet contains a whale coda.
Rhythm Type Classification: Classify codas according to their rhythmic patterns. For this task, we choose to include samples whose rhythm types among the most common, because the remaining ones appear too infrequently for classifiers to be accurate.
Social Unit Classification: Identify the social unit associated with each coda.
Vowel Classification: Detect vowel-like patterns within whale vocalizations.
Table 5 summarizes dataset sizes for each task.
We split the dataset into 80% training and 20% testing, using stratified sampling of labels to ensure consistent label distribution. The embedding model is frozen, and only the classifier parameters are trained. Training is performed NVIDIA A10G GPU for 10 epochs, using
Training Process.
34

---

Table 5: Dataset sizes for downstream classification tasks.
TASK
NUMBER OF SAMPLES
CODA DETECTION
3,100
RHYTHM TYPE CLASSIFICATION
916
SOCIAL UNIT CLASSIFICATION
2,659
VOWEL CLASSIFICATION
486
learning rate of 10-1 and batch size of 32. Model checkpoints are saved each epoch, and the best-performing model is selected based on test set performance.
E.5
Expert Perceptual Evaluation
Five domain experts in sperm whale bioacoustics participated in the evaluation. Given the highly specialized nature of sperm whale vocalization analysis, the pool of qualified experts with years of direct experience analyzing and annotating these vocalizations is notably small. All participants were recruited from an established research collaboration studying cetacean communication, and each had at least three years of experience working with sperm whale codas. No compensation was given to participants.
The evaluation was conducted via Google Form. The form began with the following introduction:
Welcome
Thank you for participating in this study. Your expertise in analyzing sperm whale vocaliza- tions is invaluable for evaluating our model.
The study consists of four parts, to be completed in order. final section includes three short questions about your background. Technical Setup
Download and extract the distener_evaluationnn
file from provided link Complete the experiment in quiet environment
Use headphones for all listening tasks
You can take breaks between sections as needed
you encounter any technical difficulties or have questions about the procedure, please contact [omitted].
Participant Identification
Name (used for tracking responses only):
E.5.1 Audio-Only Two-Alternative Forced Choice (2AFC)
Listeners were presented with 30 pairs of codas. Each pair contained an original, denoised coda and a gener--ge counterpart. Participants were asked identify which sample was the original and which was generated.
35

---

Task Instructions
In this section, you will listen to pairs of codas. For each pair, one is natural recording and one is synthetic. Please indicate which you believe is synthetic. The audio files are located the *** s //ationn/=** / folder. Each pair consists of two files:
w + #1 +Ilb.way way
*2a. *2b.wav* etc.
Please listen to each file a* most three times** Base your decision only on the provided audio. Do not visualize the audio.
E.5.2 Mixed Two-Alternative Forced Choice (2AFC)
Listeners were presented with 25 individual samples: 10 real codas, 5 generated from real codas, generated from walrus vocalizations, and generated from digital beeps. Each listener classified each sample either real or generated.
Task Instructions
In this section, you will listen individual codas and classify each as either natural or synthetic.
The audio files are located *** ***ccctoo2**** folder:
#1.wav+
#2.wav
* etc.
Please listen to each file at most three times. **Ba your decision only on the provided audio. Do not visualize the audio. #s
E.5.3 Visualized Two-Alternative Forced Choice (2AFC)
This experiment was identical to the Audio-Only 2AFC condition, except participants were allowed to inspect the spectrograms of each recording using their preferred software before making their decision. Marine biologists preferred Adobe Auditions, while underwater acoustics experts used Matlab.
Task Instructions
Once again, you will listen to pairs of codas (a. wav and b.way). For each pair, one is a natural recording and one synthetic. Please indicate which one you believe synthetic. The audio files are located in the =**scction3/ folder. Each pair consists of two files:
l + *
# + *2b.
etc.
Please listen to each file at most three times. *ou may now visualize the audio using any software you are familiar with ** What software will you to visualize the audio?
36

---

E.5.4
Qualitative Assessment
Task Instructions
For this final section, please first listen to the reference synthetic codas provided in the section4 folder. These examples were chosen to represent typical outputs of our model. Then, based these examples and your experience with all parts of the experiment, please answer the following questions
What characteristics of natural codas well represented in the synthetic ones?
What characteristics of natural codas are missing or different in the synthetic ones?
Did you observe any patterns in the synthetic codas that do occur in natural ones? When **only listening** to the audio (sections I and 2), what helped you distinguish between natural and synthetic codas?
When visualizzng the audio pairs (section 3), what helped you distinguish between natural and synthetic codas?
E.5.5
Background Information
Task Instructions
To help contextualize the evaluations, please tell about your experience working with sperm whale codas.
How many years have you spent professionally analyzing sperm whale codas (e.g., in research, conservation, or educational contexts)?
What types of coda work have you performed?
Recording of codas in the field*
Development of recording methods for codas*
Manual detection, classification or annotation of codas
Development of automatic detection, classification or annotation systems*
Meta-analysis (e.g. methodology development, literature review)*
Otter...*
In what contexts have you worked with coda recordings?
Academic r
* Conservation work*
Indussry/coomerrial projects*
"Educational/traiiing *
*Governmentreegultoory work*
What is your primary field of expertise?
37


Reviews:
[{"review_id": "TkZo2wTin1", "reviewer": "Reviewer_fPAV", "summary": "The paper proposes a machine translation model for sperm whales. It can generate sperm whale codas either at random or conditioned on vocalizations of other animals/species. The authors do not claim semantic translation and emphasise a more style-transfer-like effect. The embeddings learnt by their model can also be used for downstream classification tasks. They start with a model pretrained with music data, train on general animal audio for domain adaptation and finally finetune on a relatively large corpus of 10K sperm whale codas. Their evaluation includes 1) perceptual similarity of generated codas and 2) downstream classification performance based on learnt representations.", "questions": "I would be happy to increase my rating if the authors can include bioacoustics embedding models and pretraining on larger bioacoustics datasets; or defend their choices.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper has good potential impact, both for sperm whale understanding and broader bioacoustics. \n2. The perceptual study using experts in the field was well conducted. I also liked the qualitative feedback included here, which gives potential directions for future work.\n3. The new dataset would also be great for the community. I understand the challenges regarding making this data public mentioned in the checklist, but urge the authors to try to release as much data as possible, after publication too. \n\n**Major Weaknesses**\n\n4. For the domain adaptation stage, the authors used animal sounds from Audioset and FSD. I was curious why the authors preferred this over larger bioacoustics datasets like BirdSet [1] and iNatSounds [2]? These have 100s to even 1000s of hours of animal sounds and can be a much better starting point. I understand if the choice was for computational constraints, but it may be worth mentioning in the paper. \n5. I am a bit skeptical about the metric FAD. In particular, I am concerned about its dependence on the choice of the embedding model. I understand that the authors analysed a few different models and then chose CLAP. However, all of the embedding models were trained for general audio, and may not appropriately capture fine grained differences. I would recommend using a bioacoustic model (eg. [3]), not necessarily trained for whales, but for birds or general species sound datasets. \n6. In continuation to the previous point, from figure 3, it seems like for before Wham (untranslated), the artificial acoustic impulses are more similar to sperm whales than quite a few animal sound sources like B Seal, B Whale and L Whale. This was quite surprising to me, and further added to my skepticism of the metric FAD. \n7. Regarding the perceptual study, I wanted to clarify if the accuracy measures how often an expert was able to correctly distinguish a synthetic code from a natural one. If this is the case, then lower the accuracy, the better Wham’s performance, right? The quantitative discussion at some points seemed to be written more as if evaluating the experts’ performance. \n8. Based on my understanding, the 2AFC experiment in Fig 4 and Natural Codas in Fig 5 should be measuring very similar things. But, the former is at 80% on average while the latter is close to 60%. I understand that mixing would increase the number of choices and the evaluation is not exactly the same anymore, but is there another reason for this difference?\n9. I would appreciate it if the authors can include a few qualitative visualizations. Spectrograms of a few original and translated audio examples could help build better intuition. I would recommend including some examples for the qualitative feedback from the expert study, if possible. \n10. The claim of learnt embeddings being “useful” for downstream classification seems to be relatively weak. I would argue even CLAP embeddings are somewhat useful, and may be better than random performance. I understand that AVES may not be a realistic baseline to beat, but can the authors include baselines with some other audio embedding models like CLAP and [3]. This would better highlight the benefit of Wham’s learnt embeddings. \n\n**Minor Weaknesses**\n\n11. I was a bit confused in Fig 5. I thought NCAIs were what was referred to as synthetic codas. Can the authors please clarify the difference. \n12. 2AFC vs 2AFC+spec. Seems like 3 experts were able to almost perfectly pick the synthetic codas with spectrograms while 2 were even more unsure. I am curious if these two groups are marine biologists and acoustics specialists. \n13. Its interesting that although the dataset consists of whales in the same vocal clan, AVES still does quite well (92%) on social unit classification. Can the authors comment on this?\n\n\n\n[1] Rauch, Lukas, et al. “BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics.” The Thirteenth International Conference on Learning Representations, 2025.\n\n[2] Chasmai, Mustafa, et al. “The iNaturalist sounds dataset.” Advances in Neural Information Processing Systems 37 (2024): 132524-132544.\n\n[3] Ghani, Burooj, et al. “Global birdsong embeddings enable superior transfer learning for bioacoustic classification.” Scientific Reports 13.1 (2023): 22876.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I am glad that the bioacoustics models were helpful. \n\nOverall, I am happy with the paper and believe it has good potential in the bioacoustics community. I am in support of it being accepted, and maintain my original rating of 5: Accept. \n\nThe worse classification performance (compared to even CLAP) is the main reason I do not increase my rating to Strong Accept.", "rebuttal": "We sincerely thank the reviewer for their thorough and constructive review, and for recognizing the potential impact of our work. We are particularly grateful for specific suggestions regarding bioacoustic embeddings and datasets, which have led to substantial improvements in our evaluation. Below we address each specific concern.\n\n**W4:** Thank you for this excellent suggestion. We will retrain our model using BirdSet/iNatSounds for the domain adaptation stage. Given the larger data volume (100s-1000s of hours vs our current dataset), this will take considerable time but will be ready for the camera-ready version. We expect this enhanced domain adaptation to improve our results (and will report the results regardless of an improvement).\n\n**W5:**\nThis suggestion significantly strengthened our evaluation. We have now evaluated FAD using BirdNET [3] embeddings as you recommended. Results strengthen our results:\n\n| Dataset | Before WhAM | After WhAM |\n|---------|-------------------|------------------|\n| Risso | 131.1760 | 99.1514 |\n| Beluga | 162.1433 | 103.3841 |\n| Orca | 204.1088 | 96.2505 |\n| L. Seal | 200.5741 | 109.8147 |\n| Narwhal | 127.1519 | 95.5090 |\n| L. Whale | 173.5660 | 93.1842 |\n| B. Seal | 213.2286 | 108.6346 |\n| B. Whale | 202.5659 | 111.2529 |\n| C. Dolphin | 148.9537 | 94.6659 |\n| Ross Seal | 205.2280 | 105.3025 |\n| A. Dolphin | 190.3531 | 80.7560 |\n| Walrus | 184.3641 | 92.8567 |\n| NCAI | 121.8530 | 59.7834 |\n| S. Whale | 91.5416 | 90.0007 |\n\nAnalysis: \n - WhAM dramatically reduces FAD scores across all non-target species (improvements of 50-110 points)\n - Sperm whale FAD remains stable (-1.5 point difference is 1-2 orders of magnitude smaller than the rest, and could be due to disjoint test vs. reference sets)\n\nWe note that the difference in scale compared to Figure 3 is because we had to reimplement FAD ourselves in order to use these embeddings; the standard library we were using with CLAP was not compatible with BirdNET. Importantly, scale differences are irrelevant to FAD.\nThese bioacoustic embeddings provide additional evidence for our model's effectiveness. Results will be added to the camera-ready version.\n\n**W6:**\nBroadly speaking, the vocalizations of bearded seals (Erignathus barbatus), bowhead whales (Balaena mysticetus) and long-finned pilot whales (Globicephala melas) are \"songlike\" (trills, moans, groans, etc.). In that regard, acoustic impulses (even digital ones) are more similar to codas than these. If anything, this shows that FAD is not just picking up on background noise—which is relatively similar in these recordings as compared to the perfect silence of digital non-coda acoustic impulses.  We will add this discussion to the paper.\n\nWe agree that FAD experiments alone would not be sufficient to support our overall claim regarding our model's acoustic translation capabilities. We chose to include them as they use an \"objective\" metric unlike the downstream classification and listener study experiments, which involve (to varying extents) human annotations.\n\n**W7:** That is correct, accuracy refers to the success rate of the listeners. We will add arrows (\\downarrow) to the y axis to emphasize that lower accuracy means better performance by WhAM.\n\n**W8:** Your understanding is correct. The key difference is that in the 2AFC task, listeners directly compare a natural recording with the model's output on that exact same audio, allowing immediate A/B comparison. When the same acoustic content is presented in both versions, listeners can detect fine-grained artifacts that become apparent through direct comparison.\n\nIn contrast, the batch classification task requires identifying synthetic vs. natural audio without matched pairs. Listeners must rely on detecting absolute qualities that distinguish synthetic from natural speech across varied content, without the benefit of direct comparative cues. This fundamental difference in task difficulty explains the performance drop from ~80% to ~60%.\n\n**W9:** Thank you for the great suggestion, we will add a sample of spectrograms to the camera-ready version. We will ask our study participants to annotate the spectrogram with their observations. We have already reached out and can confirm at least one participant is available to share their annotations.\n\n**W10:** We agree that comparing against other embedding models would better demonstrate WhAM's utility. Following your suggestion, we evaluated BirdNET [3] and CLAP embeddings on our downstream tasks:\n| Task | WhAM | AVES | BirdNET | CLAP | Random | Majority |\n|------|------|------|---------|------|---------|----------|\n| Detection | 91.3 | 92.8 | 90.0 | 96.0 | 60.9 | 60.9 |\n| Rhythm | 84.4 | 90.4 | 94.8 | 89.0 | 66.3 | 60.9 |\n| Social Unit | 70.5 | 92.0 | 93.4 | 89.0 | 42.5 | 35.1 |\n| Vowel | 79.6 | 91.8 | 85.0 | 82.0 | 66.3 | 66.3 |\n\n\nWe emphasize that AVES, BirdNET, and CLAP are all encoder-only models designed for classification, whereas WhAM can be used for generation. As expected, embeddings trained on animal datasets (AVES, BirdNET) generally outperform general audio embeddings (CLAP) on more involved bioacoustic tasks.\n\nThank you for suggesting these additional experiments—they confirm the same trend across different embedding models, and add breadth to our analysis. The complete comparison table will be included in the camera-ready version.\n\n**W11:** Apologies for the confusion.\nNCAIs = Training data (synthetic beeps/clicks that sound artificial)\nSynthetic codas = WhAM's output (intended to sound like real whale codas) We'll clarify this terminology throughout the paper.\n\n**W12:** Interesting observation! Of the (near-)perfect scores in the spectrogram-assisted 2AFC, one was a marine biologist and another was an underwater acoustician. Interestingly, one is a professor with over a decade of experience, while the other is a PhD student. However, with n=5, we cannot draw statistical conclusions about expertise-performance correlation. We can surely suggest this as future work.\n\n**W13:** In [ANONYMIZED LOCATION], Social Units are stable family groups of sperm whales - essentially extended matrilineal families that travel and socialize together. They represent the main ‘operational’ level of social structure of the whales lives, while mature males live predominantly alone, all females spend their lives in their natal units. SUs which share the same vocal dialect of codas make up a higher level of social structure called a vocal clan; however, as units move around the ocean there is a need to recognize one from another so there are some distinct features of their dialects that allow for this recognition [1]. This is likely what provides a cue for the classification task (see also [2]).\n\nAVES's 92% accuracy in distinguishing between SUs could reflect: (1) genuine group-specific vocal characteristics from these stable family units, and/or (2) confounding factors from recording conditions, as different SUs were necessarily recorded on different days and locations. While (1) represents the biological signal we're interested in, (2) is a limitation inherent to field recordings in marine environments.\n\n**In conclusion,** We believe these additional experiments and clarifications fully address your concerns. The new bioacoustic embedding results particularly strengthen our claims about both FAD metrics and downstream utility. Thank you again for your constructive feedback that has substantially improved our work.\n\n—\n### Citations\n[1] S. Gero, H. Whitehead, and L. Rendell (2016) Individual, unit, and vocal clan level identity cues in sperm whale codas. Royal Society Open Science 3: 150372.\n\n[2] P.C. Bermant, M.M. Bronstein, R.J. Wood, S. Gero, and D.F. Gruber (2019) Deep Machine Learning Techniques for Sperm Whale Bioacoustics: Detection and Classification of Echolocation Clicks and Codas. Scientific Reports 9:12588"}, {"review_id": "jg629ZZ45P", "reviewer": "Reviewer_VhbL", "summary": "The paper presents WhAM, a generative model that theoretically can convert any audio prompt into whale sound. \nTechnically, it adopts a masked language model trained on discrete audio units (DAC) and uses iterative decoding to generate the audio.\nThe model can generate whale sounds reasonably, based on the general FAD measurement as well as the evaluation by the domain experts. The representation of WhAM can be used for whale-oriented understanding tasks.", "questions": "I may request more clarification:\n\n(1) Personally, I'm a bit confused about the motivation. Why do we need to generate the whale sound **from a random audio prompt**, rather than some other more informative cues (e.g., given clan, gender, or some text-like information, if exists). \n(2) Experimentally, the authors only used around 20 hours of data, but trained the model for 1M steps. Given this small data volume and long training schedule, do the authors have a good reason for this setup? Would it make the model fully memorize every whale sound? do the authors have a plan to test the model's generalization capability?", "limitations": "The paper is solid in general. As the machine learning community may not have sufficient knowledge in this biological domain, I would recommend the authors to make some audio demos so readers could better understand this work.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n(1) The authors show great expertise in animal/whale studies, which makes it a good interdisciplinary study between machine learning and animal studies.\n(2)  The authors have conducted a good evaluation of their model, including FAD-based evaluation and the human experts' evaluation. The results are convincing. \n(3) I really appreciate that the authors provided sufficient biological background in their paper to make the paper more understandable to the community.\n\nWeakness:\n(1) Lack of baseline: the authors mentioned the GAN-based approach in their introduction and related work sections, but didn't compare with these prior works experimentally. \n(2) Lack of ablation study: The authors used non-whale sounds (e.g., AudioSet) in their first stage of training. I would suggest that the authors justify the necessity of this adaptation.", "was_revised": true, "final_justification": "The paper provides a good implementation in generating whale sound, which is an interesting topic from marine bilology.\n\nSince i'm mainly from machine learning community and have little knowledge about such field, I would only review based on my background, and is not confident enough.\n\nI score the paper 4. It should not be lower since it's complete and no tech flaws have been found. It should not be higher since the method it borrowed from prior works and is not very original, and this work can hardly be generalized beyond this whale research.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback on our work. We appreciate your recognition of our interdisciplinary approach and the thorough evaluation methodology. Your questions have helped us identify areas where we can provide additional clarity and strengthen our empirical analysis. Below, we address each of your concerns in detail, including new experimental results that validate our design choices.\n\n\n**W (1):** At time of writing the paper, we were unable to obtain samples from either of the GAN-based approaches: Kopets et al. did not have any public code or samples, while Begus et al. had no samples but did release public code; unfortunately, we were unable to run their code (we have reported the issues to the authors). If we cannot resolve the issue by the camera-ready, we will mention this in a footnote.\n\n\n**W (2):** We appreciate this suggestion. We note that we already justified the necessity of domain adaptation through downstream classification experiments (mentioned at the end of Section 4 with details in Appendix D.2), which showed that neither domain adaptation nor SSFT alone produces useful embeddings. Following your suggestion for more extensive ablations, we have now repeated this analysis for FAD:\n\n| Model Configuration | Average FAD | Sperm Whale FAD |\n|-------------------|-------------|-----------------|\n| Tokenizer only | 0.97 | 0.44 |\n| No SSFT (domain adaptation only) | 0.85 | 0.58 |\n| No domain adaptation (SSFT only) | 0.55 | 0.63 |\n| Full WhAM | 0.72 | 0.29 |\n\nImportantly, when using the \"SSFT-only\" ablation on sperm whale inputs → sperm whale outputs, FAD increases to 0.63 (worse than average). This is a red flag—the model should preserve sperm whale acoustic properties when the input is already a sperm whale coda. In contrast, full WhAM achieves 0.29 FAD on the same task (which is the lowest FAD except for the reference dataset itself).\n\nThis suggests that without domain adaptation, the model cannot properly condition on input audio. Indeed, during development, we found that without domain adaptation, the model fails to generate coherent audio when translating from marine mammal inputs: outputs are so noisy that even non-expert listeners could distinguish them with ~100% accuracy.\n\nDomain adaptation with diverse marine sounds (AudioSet, FSD) teaches the model to condition on input characteristics - a prerequisite for meaningful acoustic translation. Thank you for this suggestion - these FAD ablations revealed nuanced insights about the interaction between domain adaptation and species-specific training. We will add this interesting discussion to the ablation section of the paper alongside our existing downstream task ablations.\n\n**Regarding audio demos:** We appreciate this suggestion and included audio samples as supplementary material with our submission. As a reminder, our supplementary materials include:\n\n- Input/output pairs: Natural sperm whale codas and their WhAM translations for direct comparison\n- Cross-species examples: Synthetic codas generated from both sperm whale and walrus inputs\n\nTo better serve readers unfamiliar with marine bioacoustics, we will enhance these materials for the camera-ready version. We will (1) add sample spectrograms directly in the main paper showing input/output pairs, (2) include a \"listener's guide\" explaining what acoustic features to focus on, and (3) add more diverse examples (different coda types, various acoustic conditions).\n\nAdditionally, we intend to release the full WhAM model on HuggingFace, enabling researchers to generate their own synthetic codas and explore the model's capabilities interactively.\n\n\n**Q (1):** This is an excellent question that highlights a fundamental challenge in marine bioacoustics. We completely agree that conditioning on semantic or demographic information would be more scientifically valuable. However, the field faces severe data annotation limitations:\n\n1. Only about 10% of all recordings made of the sperm whale units in our datasets between 2005 and 2024 have been manually annotated and verified and which contain behavioral annotations. We note that recordings vary in duration from ~4mins to hours long - and this is the largest curated sperm whale dataset known to us. More have had codas detected and annotated with new algorithms, but for this work, to be conservative, we focused on manually verified annotations only.\n2. Speaker identity labels are rare as well: of the recordings which have been manually annotated only ~40% contain identified codas attributed to speakers which remains a very manual and time consuming step in the data annotation process.\n3. The vast majority of recordings in our datasets are from females and immatures of both genders (who sound very much like their mothers dialect). In particular, they mostly do not contain mature male vocalization.\n\n\nOur self-supervised acoustic translation approach was designed specifically to work with this reality—using audio context rather than requiring semantic labels. This establishes the technical foundation that future work can build upon as richer annotations become available.\n\n*Regarding clan conditioning:* All our experiments use recordings from a single clan. Multi-clan synthesis would require years of additional fieldwork across ocean basins. We note that even aggregating our current single-clan dataset from two existing sources took several months due to format inconsistencies, data loss, and lack of standardization in the marine biology community.\n\n**Q (2):**\nExcellent observation about the data-to-steps ratio. Allow us to clarify, and we will add this clarification to the model details in the paper. The key insight is that our training setup generates far more unique training examples than the raw hours suggest. At a high-level, this is how tokenization works:\n\n - Each 2-second audio snippet becomes a 14×120 token array\n - Columns represent time steps, rows represent acoustic granularity (coarse-to-fine)\n - During training, we randomly mask entire columns (time steps)\n - With 120 columns, there are 2^120 possible masking patterns per snippet\n\nIn our actual training data:\n\n - 20 hours = 36,000 snippets × 2^120 possible maskings ≈ 10^40 possible training examples\n - Training examples seen: 1M steps × 24 batch size = 24M\n - *Since 10^40 >> 24M, we effectively never see the same training example twice*\n\nDuring development, we subjectively evaluated model outputs every few thousand steps and observed a gradual shift from music-like sounds to authentic coda-like vocalizations. The model continued improving throughout the entire 1M steps without plateauing. If the reviewer believes quantifying this progression will significantly improve the paper, we can plot FAD scores versus training steps for the camera-ready version."}, {"review_id": "gsa28SnJjS", "reviewer": "Reviewer_doFd", "summary": "The paper proposed an neural Transformer architecture to generate synthetic sperm whale codas. The Sperm whales main communication is based on short sequences of clicks which called codas.\n\nThe WhAM (Whale Acoustics Model) proposed model is finetuning of VampNet which is a model that trained on musical data and on 10k coda recordings collected over the past two decades. \n\nThe paper shows that the proposed WhAM architecture generate very good synthesis codas while preserving the acoustic features.\nThe author use the Fréchet Audio Distance (FAD) metric to evaluate the proposed model. \n\nMoreover, they check the learned representations  and shows that it get very good performance for downstream tasks such as rhythm, social unit, and vowel classification.", "questions": "1. What is the limitation of the proposed model when the only trained model is the MATAM? can you train a codec?\n\n2. Can you ellborate more on the unnatural of clicks and spectral anomalies? Can you think about modification to the network in order to fix that issue?\n\n3. Can you fix the echolocation sequences problem in the dataset? Can you tell how degradation in the performance it gives?", "limitations": "--", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The idea of the proposed model WhAM is very novel for using transformer based model for generating sperm whale vocalizations.\n\n2. The paper use the well known Fréchet Audio Distance metric and and expert perceptual examination \n\n3. The proposed method develop with collaboration with domain experts \n\n\nWeaknesses:\n1. The MATM network was fine-tuned where the audio codec is fixed which can harm the results of the proposed model.\n\n2. The proposed model has noise in the generated samples for example unnatural onset/decay of click and spectral inconsistencies\n\n3. The training dataset may has echolocation clicks which affect the generation results.", "was_revised": true, "final_justification": "All of my concerns were addressed appropriately by the authors.", "rebuttal": "We thank the reviewer for their enthusiastic assessment of our work and for recognizing the novelty of applying transformer-based architectures to sperm whale vocalization synthesis. Your technical questions have prompted us to conduct additional analyses that strengthen the paper and clarify important implementation details. Below, we address each of your concerns.\n\n**Regarding training a codec:**\nThank you for this insightful question about the fixed codec limitation. Your comment prompted us to conduct a detailed analysis of the codec's impact on sperm whale vocalizations.\nWe performed systematic encoding/decoding experiments measuring frequency-dependent signal-to-noise ratio (SNR in dB):\n\n- 0.2-3kHz: +0.15 dB\n- 3-4kHz: -1.95 dB\n- 4-5kHz: +0.15 dB\n- 6-7kHz: -1.95 dB\n- 7-9kHz: -2.77 dB\n\nWhile we see some degradation in the 3-6kHz range (important for vowel-like features per Beguš et al.), the codec still preserves the primary coda energy reasonably well. The degradation is not severe enough to prevent our model from generating natural-sounding codas (Section 4.2), nor from its embeddings to \"capture\" vowels (Section 4.3).\n\nDeveloping a specialized bioacoustic codec remains valuable future work, though computationally expensive. For now, these results suggest the fixed codec, while not perfect, is sufficient for demonstrating acoustic translation capabilities. We've added this quantitative codec analysis to our limitations section. Thank you for motivating this important addition.\n\n\n**Regarding echolocation clicks:** You raise a valid concern. While our dataset curators made efforts to filter echolocation clicks from communication codas, some false-positives likely remain given the dataset's scale and technological limitations of click detectors. We will run a more recently-developed coda-detector on our dataset; if there are noticeable differences, we will retrain the model and release this as an additional checkpoint. Unfortunately, we will not be able to repeat the listener perceptual study.\n\nIt's worth noting that only one of five expert listeners reported this issue, suggesting it's not a pervasive problem in our generated outputs. The model appears to learn communication coda patterns preferentially despite possibly \"contaminated\" training data. We acknowledge this limitation in our dataset section.\n\n\n\n**Regarding fixing unnatural click properties:** Thank you for this question. The artifacts mentioned by expert listeners have different solutions:\n\n- Background noise: Standard bioacoustic preprocessing (DC offset removal, noise-gating) would remove this. Indeed, Beguš et al. (2024) apply such denoising to their sperm whale data.\n- \"Unnatural\" clicks: This could be addressed through higher p-norm losses to emphasize acoustic events of varying magnitudes. However, this optimization likely needs to happen at the codec level, which is beyond our current computational resources. More training data may also help.\n\nWe deliberately chose to use raw outputs from WhAM without classical post-processing, as this paper focuses on demonstrating our neural model's generative capabilities (and limitations). For practical applications, standard bioacoustic denoising would be applied, but including it here would obscure the model's true performance."}, {"review_id": "IiH2775DKc", "reviewer": "Reviewer_e5Hs", "summary": "This paper presents a model to generate whale codas, which is fine-tuned from VampNet. They fine-tune the VampNet on their collected dataset and confirmed that the generated codas are similar to real ones, and learned representations are useful in some downstream tasks such as audio translation and classification.", "questions": "Lack of technical novelty and important observations is the main reason for my rating. \n\nThey also do not mention that they will publish the dataset used to train their model. If their dataset is unique, they should focus on the presentation of the dataset.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The dataset used for the research is unique, which might be interesting to some people.  \n2. They seem to publish their models. \n\nWeaknesses\n1. This paper focuses on developing a model for whale vocalization. However, the model has limited technical novelty in model architecture, training objectives. Their approach seems to consist of tokenization, token modeling, and generation. These modules are borrowed from previous work, VampNet, without modification, and therefore, this work has limited technical novelty. \n\n2. It is not very clear if the success in the style transfer is quantitatively verified. In my understanding, FAD measures the distance between two distributions, yet does not measure the semantic similarity between the translated and original ones. Their generation results can be realistic enough to mimic the real codas, but they might not preserve the semantics of the input. \n\n3. The observations from their empirical results are limited. They combined several datasets to fine-tune VampNet and show that their trained model performs well on diverse whale-related downstream tasks. However, it is not clear what the key ingredient was, e.g., dataset combination or selection of VampNet. Table 1 only shows that their model outperforms the previous work. To give better observations and insight for readers, they need to provide more ablations into the combination of training datasets and base models. \n\n4. Anon1 and Anon2 seem to be their developed dataset. However, the explanation of the datasets is abbreviated from the main paper. Also, it seems it is not clear if they can publish the dataset. If they can, the novelty of this work should improve.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns about the dataset. However, it did not address my concerns about the lack of technical novelty and empirical analysis.\n\nThe author stated that their work is the first to apply the existing model to whale vocalizations, which I think does not much support the technical novelty unless there is a large barrier to applying to their task and they solved it. Also, empirical analysis is indeed limited to domain adaptation, fine-tuning, and the insight from the analysis seems to be a bit limited.\n\nConsidering that, I would like to keep leaning towards rejecting.", "rebuttal": "Thank you for your review, which has helped us clarify and strengthen our paper. Most importantly, our work is novel as it is the first application of neural acoustic translation to cetacean vocalizations (only one other paper for neural bioacoustic translations exists, and it concerns birds). We also respectfully disagree with the assessment of limited technical novelty and lack of ablations; we will highlight where those are found in the paper, below. Next, we address each concern in detail and provide additional context about our technical contributions, evaluation methodology, and the significance of this work for both machine learning and marine biology communities.\n\n### Weaknesses\n\n> This paper focuses on developing a model for whale vocalization. However, the model has limited technical novelty in model architecture, training objectives. Their approach seems to consist of tokenization, token modeling, and generation. These modules are borrowed from previous work, VampNet, without modification, and therefore, this work has limited technical novelty. ... Lack of technical novelty and important observations is the main reason for my rating.\n\nWe respectfully disagree with this assessment and are glad to more specifically state the study's contributions. Firstly, there are very few examples of acoustic translation works on animal vocalizations (birds) - and this is the first to be applied to  whale (or any cetacean) vocalizations. Our contributions include:\n- Novel evaluation framework combining FAD, expert perceptual studies, and downstream tasks\n- Largest curated sperm whale vocalization dataset\n- Training strategies bridging music-pretrained models to underwater bioacoustics\n- First demonstration that style transfer can preserve bioacoustic structure in cetaceans\n\n> It is not very clear if the success in the style transfer is quantitatively verified. In my understanding, FAD measures the distance between two distributions, yet does not measure the semantic similarity between the translated and original ones. Their generation results can be realistic enough to mimic the real codas, but they might not preserve the semantics of the input.\n\nWe appreciate this comment/concern and want to clarify a fundamental aspect of our work. As stated in the paper: \"We emphasize that translation is in the acoustic sense; semantic translation remains a distinct and more ambitious goal.\"\n\nFor this reason, we use \"acoustic translation\" rather than \"semantic translation.\" Understanding animal communication semantically would indeed be groundbreaking. For example, the 1973 Nobel Prize was awarded for decoding the semantic content of bee communication (\"waggle dances\"). No such understanding exists for whale communication.\n\nOur experiments argue that our model preserves acoustic properties and temporal structure, not semantic meaning (which remains unknown to science).\n\n> The observations from their empirical results are limited. They combined several datasets to fine-tune VampNet and show that their trained model performs well on diverse whale-related downstream tasks. However, it is not clear what the key ingredient was, e.g., dataset combination or selection of VampNet. Table 1 only shows that their model outperforms the previous work. To give better observations and insight for readers, they need to provide more ablations into the combination of training datasets and base models.\n\nThanks for this comment, We want to point out that we have indeed provided extensive ablations in our paper. Specifically:\n\n1. Section 4, final paragraph describes ablation experiments for domain adaptation and species-specific fine-tuning.\n2. Appendix D repeats the FAD and downstream classification experiments while ablating on model components and dataset composition. Additional ablations have been carried out as a result of discussions with other reviewers, and will be added to the paper.\n\n\n> Anon1 and Anon2 seem to be their developed dataset. However, the explanation of the datasets is abbreviated from the main paper. Also, it seems it is not clear if they can publish the dataset. If they can, the novelty of this work should improve. ... They also do not mention that they will publish the dataset used to train their model. If their dataset is unique, they should focus on the presentation of the dataset.\n\nWe appreciate your interest in our dataset. If this paper is accepted, we will share the Anon1 dataset. We are in conversation with the data collectors of Anon2 regarding a partial or full release.\n\nThe paper focuses on the acoustic translation methodology rather than dataset presentation because: (1) the dataset combines two existing sources with our preprocessing pipeline, and (2) the methodological contributions (translation framework, evaluation suite) have broader applicability beyond this specific dataset.\n\nWe will also release our model weights and preprocessing code, enabling researchers to apply our methods to their own bioacoustic data."}]

Author rebuttal:

Meta-review: The paper proposes WhAM, a method that re-expresses arbitrary acoustic attributes as sperm whale vocalizations. It is worth noting that modeling whale calls is an established goal in animal communication research, but this work does not attempt to provide a biologically grounded account of meaning or semantics. Rather, the system serves as an experimental probe to study what structure can be recovered and how experts engage with machine-generated calls. Its contributions are thus methodological and exploratory, and offer a new tool (acoustic translation) for analysis, expert feedback, and representation testing.

Three of four reviewers leaned toward acceptance, and the discussion addressed many concerns productively. The main outstanding issues relate to limited methodological novelty and generality at the ML level, though the paper clearly articulates and situates its novelty at the application level. In my view, the conceptual challenges inherent in this type of research, combined with the potential of this work to inspire dialogue and future development, substantially outweigh those concerns. So, I view it as a clear accept.
Decision: Accept (poster)