Acoustic Individual Identification of White-Faced Capuchin Monkeys Using Joint Multi-Species Embeddings
Authors: Álvaro Vega-Hidalgo, Artem Abzaliev, Thore Bergman, Rada Mihalcea
Accepted at: acl25

Acoustic Individual Identification of White-Faced Capuchin Monkeys Using Joint Multi-Species Embeddings

Álvaro Vega-Hidalgo1, Artem Abzaliev1, Thore Bergman2,3, Rada Mihalcea1
1Computer Science and Engineering, University of Michigan
2Department of Psychology, University of Michigan
3Department of Ecology and Evolutionary Biology, University of Michigan
{alvarovh, abzalie, thore, mihalcea}@umich.edu

Abstract
Acoustic individual identification of wild animals is an essential task for understanding animal vocalizations within their social contexts, and for facilitating conservation and wildlife monitoring efforts. However, most of the work in this space relies on human efforts, as the development of methods for automatic individual identification is hindered by the lack of data. In this paper, we explore cross-species pre-training to address the task of individual classification in white-faced capuchin monkeys. Using acoustic embeddings from birds and humans, we find that they can be effectively used to identify the calls from individual monkeys. Moreover, we find that joint multi-species representations can lead to further improvements over the use of one representation at a time. Our work demonstrates the potential of cross-species data transfer and multi-species representations, as strategies to address tasks on species with very limited data.

1 Introduction
For a long time, researchers viewed the vocalizations of non-human species as mere reactions to internal emotional states (Lorenz, 1952). Consequently, early scientific methods in animal communication research largely overlooked individual differences and did not test for the presence of linguistic features (e.g., pragmatics, semantics, syntax) in animal communication systems. This simplified view of animal communication has been overturned by the growing evidence uncovering the presence of linguistic features in non-human animals (Bergman et al., 2019), leading to the emergence of Animal Linguistics as a formal interdisciplinary research field (Bowling and Fitch, 2015; Engesser et al., 2015; Suzuki, 2024; Berthet et al., 2023; Suzuki, 2021; Scott-Phillips and Heintz, 2023). This shift in perspective highlights the need for individual-level analysis, as it allows researchers to account for the social and environmental contexts in which vocalizations occur, ultimately improving our ability to test their linguistic capacities more rigorously.

Additionally, long-term, individual-level analyses are critical for understanding and protecting wildlife. Such analyses support key approaches like social network quantification, assessing animal cognition, and performing capture–recapture techniques for tracking population dynamics (Slater, 1981; Carlson et al., 2020). Over the past decade,

![A figure showing Capuchin Twitter vocalizations, t-SNE embedding, call type clusters, and group territories](page_1016_370_388_563.png)

Figure 1: (A–D) Capuchin Twitter vocalizations show diverse structural variations. (E–F) t-SNE of Google Perch-Whisper embeddings. (E) Call type clusters. (F) Colored by individual, highlighting four diverse examples of Twitters. (G) Adult female capuchin with infant in Taboga. (H) Territories of four capuchin groups in the Taboga Reserve in northwestern Costa Rica.
acoustic monitoring has emerged as a widely adopted, cost-efficient strategy in conservation, leading to growing interest in acoustic individual identification. By enabling researchers to recognize individuals from their vocalizations, this approach paves the way for more nuanced insights into ecology, behavior, evolution, and conservation (Knight et al., 2024).

In this paper, we address the task of acoustic individual identification in white-faced capuchins. We collect a two-year dataset of individualized focal recordings, a labor-intensive yet optimized method that mitigates signal-to-noise and cocktail party problems in wild bioacoustic settings (Bermant, 2021; Miron et al., 2024). Using this dataset, we evaluate human speech- and bird bioacoustic-based pre-trained networks, comparing single-embedding models to ensembles that merge embeddings from distinct networks. We hypothesize that human speech embeddings, such as Whisper or HuBERT, complement bioacoustic embeddings like Google Perch or BirdNET–originally trained on bird sounds–and predict that heterogeneous embedding combinations will outperform single-embedding models.

Transfer learning has significantly advanced acoustic classification tasks in non-human animals (Miyaguchi et al., 2024; Kahl et al., 2023; Abzaliev et al., 2024). Recent studies on gibbons have explored the use of self-supervised speech models (e.g., HuBERT, Wav2vec 2.0), pre-trained bird classifiers (e.g., BirdNET, Perch), and non-transfer-learning deep models for primate acoustic identification, finding that speech models most effectively capture individual vocal signatures, bird classifiers perform well in automated detection but are more susceptible to background noise, and non-transfer-learning models struggle when trained on small datasets (Cauzinille et al., 2024; Clink et al., 2024). Nevertheless, it remains unclear whether using multiple joint embeddings leads to better performance by exploiting complementary features from different training data domains.

This work makes three main contributions. First, we propose white-faced capuchin monkeys as a model organism for advancing computational research on animal communication. Second, we show that combining embeddings from human speech and bird bioacoustics models significantly improves acoustic identification performance in white-faced capuchins, outperforming single-embedding baselines. Finally, our findings show that acoustic diversity and soundscape similarity play a greater role than phylogenetic proximity. Smaller models trained on diverse bird vocalizations recorded in natural environments outperform much larger speech-trained models designed for humans, despite humans being more closely related to our study species. These results highlight the value of cross-species model development in achieving better generalization for the acoustic identification task.

2 Study system: white-faced capuchin monkeys in the Taboga Reserve, Costa Rica

White-faced capuchin monkeys (*Cebus capucinus*) are ideal for studying animal communication, with 27 call types (Gros-Louis et al., 2008), complex social behavior and cognition including tool use (Goldsborough et al., 2024), complex social networks (Crofoot et al., 2011) and cultural transmission (Perry et al., 2017). Taboga hosts their highest known density (Tinsley Johnson et al., 2020).

Data collection. Our field team collected audio recordings of focal individuals by following them in the Taboga forest. We used directional microphones aimed at the subjects from January 2021 to December 2022 through the wet and dry seasons, with hours ranging from 5 am to 5 pm. Recordings were captured at 48 kHz and 16 bit resolution. These raw recordings were subsequently trimmed to isolate the precise moments when vocalizations were detected, and only the calls classified as either a “Peep” or “Twitter” were included in this dataset, according to established criteria in the literature (Gros-Louis et al., 2008).

Audio recordings. The full dataset consists of 1,257 Twitter recordings and 2,089 Peep recordings from 45 individuals, although 15% of the recordings were assigned to unknown individuals. We include data from individuals that had at least 30 recorded calls, while recordings from unidentified subjects encountered in the field are grouped into an “Unknown” class. For Peeps, this dataset includes 16 individuals, and for Twitters this dataset included 10 individuals (total sample=1609). Peep calls are typically short (mean 0.27 s, SD 0.27 s), whereas Twitter calls are more complex (Figure 1) and longer (mean 0.40 s, SD 0.18 s).
3 Cross-Species Embeddings for Individual Classification

Collecting focal audio recordings of wild animals in their natural habitat is a challenging and resource-intensive task. Even with dedicated field teams, building large enough datasets to fully exploit deep neural networks is difficult. As a result, transfer learning—which leverages the inductive bias of models pre-trained on larger, related datasets—has emerged as the most effective strategy for achieving high performance in bioacoustic classification under low-data conditions (Ghani et al., 2023a).

Audio Representation Models. We extract pre-trained embeddings from Google Perch V8 (Ghani et al., 2023a), a model primarily trained on bird vocalizations, and Whisper (Radford et al., 2022), which was predominantly pre-trained on human speech. While additional embeddings were evaluated, we focus on these two in the main text for clarity, with results from five other models detailed in Appendix A. We apply mean-pooling to obtain lower-dimensional representations from large speech models like Whisper.

Minimum Redundancy Maximum Relevance. To combine representations from multiple species, we explore a feature-select model using Minimum Redundancy and Maximum Relevance (MRMR) (Ding and Peng, 2005), alongside simple concatenation and summation. Originally developed in cancer research for gene selection, MRMR improves feature selection in high-dimensional datasets by balancing two key criteria: maximizing relevance to the target variable (measured via mutual information) while minimizing redundancy (filtered using a correlation coefficient threshold). Our implementation starts with the feature that has the highest mutual information among both embeddings, removes any features with a correlation coefficient of 0.8 or higher, and then iteratively selects the next most informative feature. This process continues until 1024 embedding features are selected from both embeddings, ensuring an optimal balance of diversity and informativeness.

Experimental Setup. To ensure a fair comparison, we carefully control parameter counts and apply hyperparameter tuning. Single-embedding models and the MRMR model compress each input into 512 units, then reduced it to 64 for final classification. Concatenation and summation ensembles apply a 256-dimensional compression to each embedding separately, then sum or concatenate the outputs before another 64-unit layer. For a robust comparison, we generate 50 random train-test splits (10 recordings per individual in the test set) and train models with all seven single embeddings as well as all pairwise combinations (concatenation, summation, and MRMR). To identify the best hyperparameters for each model trained, we conduct a search over learning rates {1e-5, 5e-5, 1e-4, 5e-4, 1e-3, 5e-3} and dropout rates {0.2, 0.3, 0.4, 0.5, 0.6}, evaluating 30 randomly sampled configurations for 100 epochs each with early stopping (patience=10, min ΔF1=0.001), and selected the highest F1-scoring setup. All models are trained using the Adam optimizer. After confirming normality and homoscedasticity, we compare each architecture’s top-performing model via ANOVA and a post-hoc Tukey test.

Whisper Layer Probing. To pinpoint which Whisper transformer layer encodes the richest individual-specific information, we trained Perch–Whisper MRMR models in which the Whisper input is systematically replaced with the hidden representation from each of the 33 encoder layers. For each layer, we retrain the model across the 50 random train-test splits using the same training schedule described above.

Spectrogram annotations and measurements. To compare explainable acoustic features with non-interpretable deep embeddings, we manually measure Peak Frequency and other acoustic parameters from spectrograms, following standard bioacoustic methods. Using Raven Pro 1.6 (K. Lisa Yang Center for Conservation Bioacoustics at the Cornell Lab of Ornithology, 2024), we select regions of interest and extract 30 interpretable features (see Appendix A), including Peak Frequency, Center Frequency, and Center Time. These measurements were taken from six individuals—one adult male, one adult female, and one infant from each of the two monkey troops—chosen for their distinct characteristics.

4 Results

Table 1 shows the results of the acoustic identification task for selected models. We present F1 scores for the models trained on bird vocalizations and
Figure 2: Peak frequency distributions for six capuchin monkey individuals, shown for Twitters (top) and Peeps (bottom) call types.

human speech data, together with their ensembles. While single-species vocalization models perform reasonably well, the models with the highest F1 scores are those that combine multiple embeddings (either using summation, concatenation or MRMR). Furthermore, the best-performing ensemble combine models developed for bioacoustic vocalizations and models developed for human speech. This highlights the potential of cross-species pre-training in a limited data regime. Pre-training on human speech does not capture enough information for the bioacoustic domain, as shown by the performance of Whisper for both vocalization types. But combined, those two models achieve an F1 score of 0.70 for Peeps and 0.66 for Twitters. This improved performance suggests that combining speech-trained and bioacoustic-trained embeddings effectively leverages complementary information. We also present the results for other models in Appendix A.

Despite its smaller size and more limited training dataset, the bioacoustic model Perch outperforms the much larger Whisper model, which was developed for human speech. Domain relevance is more important than model size, training data set size, or phylogenetic proximity for the acoustic identification task in Capuchins. Trained on data from noisy field conditions, Perch learns the acoustic variability of field conditions, contributing to its strong performance. Although our focal species is neither a bird nor a human, the top-performing models across architectures are trained using both bird- and human-derived embeddings, suggesting that joint multi-species embeddings provide better generalization for Capuchin acoustic classification tasks.

To better understand Whisper’s contribution to these multi-species embeddings, we conducted a layer-wise probing analysis across 50 training runs. We found that intermediate layers yielded slightly better classification performance for both Peeps and Twitters (Figure 3), though differences across layers were relatively modest.

Figure 3: Whisper layer-wise probing (mean F1 across 50 random train-test splits) for Twitters (top) and Peeps (bottom). Intermediate layers yield the highest performance for individual classification (maximum value highlighted).

We visualize the embeddings of the best-performing model from table 1 using t-SNE (van der Maaten, 2009) in Figure 1. Different call types formed well-defined clusters (Figure 1E), whereas individual classifications appear more diffuse (Figure 1F), illustrating the difficulty of the acoustic identification task (see Appendix A for more t-SNE visualizations). We also analyze the distribution of peak frequencies across individuals in Figure 2. Lower-pitched sounds characterize Peeps, while Twitters span a broader spectral range of peak frequencies. Notably, both call types exhibit bimodal distributions, with this pattern being more pronounced in certain individuals. This bi-
Table 1: Top-performing models for Twitters and Peeps (Mean F1 Score ± SD), with significance assessed by comparison to the best simple model (Perch). Significance levels: * for \( p < 0.05 \) and ** for \( p < 0.0001 \) (Tukey’s test).

<table>
  <tr>
    <th>Model</th>
    <th>F1 Score</th>
  </tr>
  <tr>
    <th colspan="2">Twitters</th>
  </tr>
  <tr>
    <td>Chance (uniform I/11)</td>
    <td>0.09 ± 0.00</td>
  </tr>
  <tr>
    <td>Perch (Simple)</td>
    <td>0.61 ± 0.03</td>
  </tr>
  <tr>
    <td>Whisper (Simple)</td>
    <td>0.55 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + Whisper (Concat)</td>
    <td>0.63 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + Whisper (Sum)</td>
    <td>0.63 ± 0.03*</td>
  </tr>
  <tr>
    <td><b>Perch + Whisper (MRMR)</b></td>
    <td><b>0.66 ± 0.03**</b></td>
  </tr>
  <tr>
    <th colspan="2">Peeps</th>
  </tr>
  <tr>
    <td>Chance (uniform I/17)</td>
    <td>0.06 ± 0.00</td>
  </tr>
  <tr>
    <td>Perch (Simple)</td>
    <td>0.66 ± 0.02</td>
  </tr>
  <tr>
    <td>Whisper (Simple)</td>
    <td>0.62 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + Whisper (Concat)</td>
    <td>0.67 ± 0.02*</td>
  </tr>
  <tr>
    <td>Perch + Whisper (Sum)</td>
    <td>0.68 ± 0.02**</td>
  </tr>
  <tr>
    <td><b>Perch + Whisper (MRMR)</b></td>
    <td><b>0.70 ± 0.02**</b></td>
  </tr>
</table>

modal distribution could reflect two or more call subtypes with distinct pitches and should be investigated further to test for the existence of pragmatics or semantics in their communication system through pitch modulation. Variability within the Twitter call type extends beyond overall pitch modulation. Some Twitters exhibit an n-shaped pitch contour, a continuous descending note, a final lower-pitched note, or a rising pitch throughout the call (Figure 1-A, B, C, D, respectively). Empirical studies incorporating rich social and environmental contexts will be crucial for uncovering the functional significance of this variation in Capuchin calls.

5 Conclusion

This study examined acoustic individual identification in two call types of white-faced capuchins. We established performance baselines for pre-trained embeddings and found that combining multiple embeddings (summation, concatenation, and minimum redundancy maximum relevance) improves classification performance. Our findings also indicate that domain relevance outweighs model size in noisy environments. Future work should extend these multi-species embeddings to other taxa, confirming broader applicability in bioacoustics and animal linguistics.

6 Limitations

While this study focused on acoustic identification, a deeper investigation into the behavioral and social functions of these call types remains relevant for future work. While there are other ways of improving acoustic identification, such as data augmentation (MacIsaac et al., 2024), we considered those techniques out of scope for the present study and focused on investigating the complementarity of joint multi-species embeddings. Our primary goal with this dataset is to make it accessible to the broader scientific community. We anticipate making it publicly available in a forthcoming study with further analyses.

7 Ethical Considerations

No animals were harmed during this study. All research adhered to ethical guidelines for animal welfare, recognizing the importance of studying animal communication while prioritizing their well-being, particularly in the context of climate change and habitat loss affecting this species. Additionally, all individuals involved in data collection and processing were engaged in formal employment or academic research under ethical labor practices.

Acknowledgments

We thank the entire Capuchinos de Taboga project team for their support and dedication throughout the data collection process. Their field expertise and commitment were essential to building the dataset that underpins this study. We also thank Robin Laurie for providing the capuchin monkey photograph used in Figure 1.

References

Artem Abzaliev, Humberto Pérez Espinosa, and Rada Mihalcea. 2024. Towards dog bark decoding: Leveraging human speech processing for automated bark classification. arXiv preprint arXiv:2404.18739.

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Preprint, arXiv:2006.11477.

Thore J Bergman, Jacinta C Beehner, Melissa C Painter, and Morgan L Gustison. 2019. The speech-like properties of nonhuman primate vocalizations. Animal Behaviour, 151:229–237.
Peter C Bermant. 2021. Biocppnet: automatic bioacoustic source separation with deep neural networks. Scientific Reports, 11(1):23502.

Mélissa Berthet, Camille Coye, Guillaume Dezecache, and Jeremy Kuhn. 2023. Animal linguistics: a primer. Biological reviews, 98(1):81–98.

Daniel L Bowling and W Tecumseh Fitch. 2015. Do animal communication systems have phonemes? Trends in Cognitive Sciences, 19(10):555–557.

Nora V Carlson, E McKenna Kelly, and Iain Couzin. 2020. Individual vocal recognition across taxa: a review of the literature and a look into the future. Philosophical Transactions of the Royal Society B, 375(1802):20190479.

Jules Cauzinille, Benoît Favre, Ricard Marxer, Dena Clink, Abdul Hamid Ahmad, and Arnaud Rey. 2024. Investigating self-supervised speech models’ ability to classify animal vocalizations: The case of gibbon’s vocal signatures. In Interspeech 2024, pages 132–136. ISCA; ISCA.

Dena J Clink, Hope Cross-Jaya, Jinsung Kim, Abdul Hamid Ahmad, Moeurk Hong, Roeun Sala, Hélène Birot, Cain Agger, Thinh Tien Vu, Hoa Nguyen Thi, et al. 2024. Benchmarking automated detection and classification approaches for monitoring of endangered species: a case study on gibbons from cambodia. bioRxiv, pages 2024–08.

Margaret C Crofoot, Daniel I Rubenstein, Arun S Maiya, and Tanya Y Berger-Wolf. 2011. Aggression, grooming and group-level cooperation in white-faced capuchins (cebus capucinus): Insights from social networks. American Journal of Primatology, 73(8):821–833.

Chris Ding and Hanchuan Peng. 2005. Minimum redundancy feature selection from microarray gene expression data. Journal of bioinformatics and computational biology, 3(02):185–205.

Sabrina Engesser, Jodie MS Crane, James L Savage, Andrew F Russell, and Simon W Townsend. 2015. Experimental evidence for phonemic contrasts in a nonhuman vocal system. PLoS biology, 13(6):e1002171.

Buuroj Ghani, Tom Denton, Stefan Kahl, and Holger Klinck. 2023a. Global birdsong embeddings enable superior transfer learning for bioacoustic classification. Scientific Reports, 13(1):22876.

Buuroj Ghani, Tom Denton, Stefan Kahl, and Holger Klinck. 2023b. Global birdsong embeddings enable superior transfer learning for bioacoustic classification. Scientific Reports, 13(1):22876.

Zoë Goldsborough, Margaret C Crofoot, and Brendan J Barrett. 2024. Male-biased stone tool use by wild white-faced capuchins (cebus capucinus imitator). American Journal of Primatology, 86(4):e23594.

Julie J Gros-Louis, Susan E Perry, Claudia Fichtel, Eva Wikberg, Hannah Gilkenson, Susan Wofsy, and Alex Fuentes. 2008. Vocal repertoire of cebus capucinus: acoustic structure, context, and usage. International Journal of Primatology, 29:641–670.

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. Preprint, arXiv:2106.07447.

Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. 2023. Masked autoencoders that listen. Preprint, arXiv:2207.06405.

K. Lisa Yang Center for Conservation Bioacoustics at the Cornell Lab of Ornithology. 2024. Raven pro: Interactive sound analysis software (version 1.6.5). [Computer software].

Stefan Kahl, Tom Denton, Holger Klinck, Hendrik Reers, Francis Cherutich, Hervé Glotin, Hervé Goëau, Willem-Pier Vellinga, Robert Planqué, and Alexis Joly. 2023. Overview of birdelef2023: Automated bird species identification in eastern africa. In CLEF (Working Notes), pages 1934–1942.

Stefan Kahl, Connor M. Wood, Maximilian Eibl, and Holger Klinck. 2021. Birdnet: A deep learning solution for avian diversity monitoring. Ecological Informatics, 61:101236.

Elly Knight, Tessa Rhinehart, Devin R de Zwaan, Matthew J Weldy, Mark Cartwright, Scott H Hawley, Jeffery L Larkin, Damon Lesmeister, Erin Bayne, and Justin Kitzes. 2024. Individual identification in acoustic recordings. Trends in Ecology & Evolution.

Konrad Lorenz. 1952. King Solomon’s Ring. T. Y. Crowell, New York.

Jennifer MacIsaac, Stuart Newson, Adham Ashton-Butt, Huma Pearce, and Ben Milner. 2024. Improving acoustic species identification using data augmentation within a deep learning framework. Ecological Informatics, 83:102851.

Marius Miron, Sara Keen, Jen-Yu Liu, Benjamin Hoffman, Masato Hagiwara, Olivier Pietquin, Felix Effenberger, and Maddie Cusimano. 2024. Biodenoising: animal vocalization denoising without access to clean data. arXiv preprint arXiv:2410.03427.

Anthony Miyaguchi, Adrian Cheung, Murilo Gustineli, and Ashley Kim. 2024. Transfer learning with pseudo multi-label birdcall classification for ds@ gt birdelef 2024. arXiv preprint arXiv:2407.06291.

Susan E Perry, Brendan J Barrett, and Irene Godoy. 2017. Older, sociable capuchins (cebus capucinus) invent more social behaviors, but younger monkeys innovate more in other contexts. Proceedings of the National Academy of Sciences, 114(30):7806–7813.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356.

Thom Scott-Phillips and Christophe Heintz. 2023. Animal communication in linguistic and cognitive perspective. Annual Review of Linguistics, 9(1):93–111.

PJB Slater. 1981. Individual differences in animal behavior. In Perspectives in Ethology: Volume 4 Advantages of Diversity, pages 35–49. Springer.

Toshitaka N Suzuki. 2021. Animal linguistics: exploring referentiality and compositionality in bird calls. Ecological Research, 36(2):221–231.

Toshitaka N Suzuki. 2024. Animal linguistics. Annual Review of Ecology, Evolution, and Systematics, 55.

Elizabeth Tinsley Johnson, Marcela E Benítez, Alexander Fuentes, Celia R McLean, Arik B Norford, Juan Carlos Ordoñez, Jacinta C Beehner, and Thore J Bergman. 2020. High density of white-faced capuchins (cebus capucinus) and habitat quality in the taboga forest of costa rica. American Journal of Primatology, 82(2):e23096.

Laurens van der Maaten. 2009. Learning a parametric embedding by preserving local structure. In International Conference on Artificial Intelligence and Statistics.
A Appendix

![Bar charts showing mutual information of the top features for both call type datasets, spanning seven acoustic pre-trained embeddings.](page_186_234_1077_355.png)

Figure 4: Mutual information of the top features for both call type datasets, spanning seven acoustic pre-trained embeddings. We display the five highest-performing features per pre-trained embedding, along with the top five interpretable features per model.
Figure 5: Mutual information of the top features in the Peeps call type dataset, spanning seven acoustic pre-trained embeddings. We display the five highest-performing features per pre-trained embedding, along with the top five interpretable features. Asterisks show correlation coefficients above 0.8.
Figure 6: Mutual information of the top features in the Twitters call type dataset, spanning seven acoustic pre-trained embeddings. We display the five highest-performing features per pre-trained embedding, along with the top five interpretable features. Asterisks show correlation coefficients above 0.8.

![Mutual information heatmap of top features in the Twitters call type dataset](page_324_579_1042_482.png)
Figure 7: t-SNE visualizations of five pre-trained embeddings, primarily trained on human speech data (with AudioMAE also incorporating internet-sourced audio). The first column presents the t-SNE plot of call types (Peeps in yellow and Twitters in blue), while the second and third columns show the t-SNE projections of Peeps and Twitters, respectively, with points colored by individual identity. From top to bottom, the rows correspond to HuBERT, Wav2Vec, Wav2Vec BERT, Whisper, and AudioMAE.
<table>
  <tr>
    <th>Model</th>
    <th>F1 Score (Mean ± Std)</th>
  </tr>
  <tr>
    <th colspan="2">Simple Network</th>
  </tr>
  <tr>
    <td>Perch</td>
    <td>0.66 ± 0.02</td>
  </tr>
  <tr>
    <td>Whisper</td>
    <td>0.62 ± 0.03</td>
  </tr>
  <tr>
    <td>BirdNET</td>
    <td>0.60 ± 0.02</td>
  </tr>
  <tr>
    <td>HuBERT</td>
    <td>0.55 ± 0.02</td>
  </tr>
  <tr>
    <td>Wav2Vec2</td>
    <td>0.47 ± 0.02</td>
  </tr>
  <tr>
    <th colspan="2">Concatenation (2 Embeddings)</th>
  </tr>
  <tr>
    <td>BirdNET + Perch</td>
    <td>0.67 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + Whisper</td>
    <td>0.67 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + HuBERT</td>
    <td>0.66 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + AudioMAE</td>
    <td>0.64 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + Wav2Vec2</td>
    <td>0.64 ± 0.02</td>
  </tr>
  <tr>
    <th colspan="2">Summation (2 Embeddings)</th>
  </tr>
  <tr>
    <td>Perch + Whisper</td>
    <td>0.68 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + BirdNET</td>
    <td>0.67 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + HuBERT</td>
    <td>0.66 ± 0.02</td>
  </tr>
  <tr>
    <td>BirdNET + Whisper</td>
    <td>0.64 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + Wav2Vec2</td>
    <td>0.64 ± 0.02</td>
  </tr>
  <tr>
    <th colspan="2">MRMR (2 Embeddings)</th>
  </tr>
  <tr>
    <td>Perch + Whisper</td>
    <td><b>0.70 ± 0.02</b></td>
  </tr>
  <tr>
    <td>Perch + BirdNET</td>
    <td>0.69 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + HuBERT</td>
    <td>0.68 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + Wav2Vec2</td>
    <td>0.67 ± 0.02</td>
  </tr>
  <tr>
    <td>Perch + Wav2Vec-bert</td>
    <td>0.67 ± 0.02</td>
  </tr>
</table>

Table 2: Performance of the top 5 models per method on the acoustic identification task using the Peeps dataset (Mean F1 Score ± Standard Deviation).
<table>
  <tr>
    <th>Model</th>
    <th>F1 Score (Mean ± Std)</th>
  </tr>
  <tr>
    <th colspan="2">Simple Network</th>
  </tr>
  <tr>
    <td>Perch</td>
    <td>0.61 ± 0.03</td>
  </tr>
  <tr>
    <td>BirdNET</td>
    <td>0.60 ± 0.04</td>
  </tr>
  <tr>
    <td>HuBERT</td>
    <td>0.56 ± 0.04</td>
  </tr>
  <tr>
    <td>Whisper</td>
    <td>0.55 ± 0.03</td>
  </tr>
  <tr>
    <td>Wav2Vec-bert</td>
    <td>0.43 ± 0.03</td>
  </tr>
  <tr>
    <th colspan="2">Concatenation (2 Embeddings)</th>
  </tr>
  <tr>
    <td>BirdNET + Whisper</td>
    <td>0.63 ± 0.03</td>
  </tr>
  <tr>
    <td>BirdNET + Perch</td>
    <td>0.62 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + Whisper</td>
    <td>0.62 ± 0.03</td>
  </tr>
  <tr>
    <td>BirdNET + HuBERT</td>
    <td>0.62 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + HuBERT</td>
    <td>0.61 ± 0.03</td>
  </tr>
  <tr>
    <th colspan="2">Summation (2 Embeddings)</th>
  </tr>
  <tr>
    <td>BirdNET + Whisper</td>
    <td>0.63 ± 0.04</td>
  </tr>
  <tr>
    <td>Perch + Whisper</td>
    <td>0.63 ± 0.03</td>
  </tr>
  <tr>
    <td>BirdNET + Perch</td>
    <td>0.63 ± 0.03</td>
  </tr>
  <tr>
    <td>BirdNET + HuBERT</td>
    <td>0.62 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + HuBERT</td>
    <td>0.62 ± 0.03</td>
  </tr>
  <tr>
    <th colspan="2">MRMR (2 Embeddings)</th>
  </tr>
  <tr>
    <td>Perch + Whisper</td>
    <td><b>0.66 ± 0.03</b></td>
  </tr>
  <tr>
    <td>BirdNET + Whisper</td>
    <td>0.65 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + HuBERT</td>
    <td>0.64 ± 0.03</td>
  </tr>
  <tr>
    <td>BirdNET + Perch</td>
    <td>0.64 ± 0.03</td>
  </tr>
  <tr>
    <td>Perch + Wav2Vec2</td>
    <td>0.64 ± 0.03</td>
  </tr>
</table>

Table 3: Performance of the top 5 models per method on the acoustic identification task using the Twitters dataset (Mean F1 Score ± Standard Deviation).
<table>
  <tr>
    <th>Measurement</th>
    <th>Units</th>
    <th>Definition</th>
  </tr>
  <tr>
    <td>Center Freq</td>
    <td>Hz</td>
    <td>The frequency that divides the selection into two intervals of equal energy (i.e., the 50th percentile frequency) measured on each spectrogram slice.</td>
  </tr>
  <tr>
    <td>Freq 25%</td>
    <td>Hz</td>
    <td>The 25th percentile frequency (first quartile) measured on each spectrogram slice.</td>
  </tr>
  <tr>
    <td>Freq 75%</td>
    <td>Hz</td>
    <td>The 75th percentile frequency (third quartile) measured on each spectrogram slice.</td>
  </tr>
  <tr>
    <td>Freq 5%</td>
    <td>Hz</td>
    <td>The 5th percentile frequency measured on each spectrogram slice, indicating the lower bound of the energy distribution.</td>
  </tr>
  <tr>
    <td>Freq 95%</td>
    <td>Hz</td>
    <td>The 95th percentile frequency measured on each spectrogram slice, indicating the upper bound of the energy distribution.</td>
  </tr>
  <tr>
    <td>BW 50%</td>
    <td>Hz</td>
    <td>The inter-quartile range bandwidth, computed as the difference between the 75th and 25th percentile frequencies (i.e., the bandwidth containing 50% of the energy).</td>
  </tr>
  <tr>
    <td>BW 90%</td>
    <td>Hz</td>
    <td>The bandwidth encompassing 90% of the signal’s energy, calculated as the difference between the 95th and 5th percentile frequencies.</td>
  </tr>
  <tr>
    <td>Peak Freq</td>
    <td>Hz</td>
    <td>The frequency at which the maximum power (or peak power) occurs within the selection, as observed in each spectrogram slice.</td>
  </tr>
  <tr>
    <td>Center Time</td>
    <td>s</td>
    <td>The time that divides the selection into two intervals of equal energy (i.e., the median or 50th percentile time) for the signal’s energy distribution.</td>
  </tr>
  <tr>
    <td>Time 25%</td>
    <td>s</td>
    <td>The time by which 25% of the total energy has been accumulated within the selection.</td>
  </tr>
  <tr>
    <td>Time 75%</td>
    <td>s</td>
    <td>The time by which 75% of the total energy has been accumulated within the selection.</td>
  </tr>
  <tr>
    <td>Dur 50%</td>
    <td>s</td>
    <td>The duration over which the central 50% of the signal’s energy is distributed, computed as the difference between the 75th and 25th percentile times.</td>
  </tr>
  <tr>
    <td>Time 5%</td>
    <td>s</td>
    <td>The time by which 5% of the total energy has been accumulated within the selection.</td>
  </tr>
  <tr>
    <td>Time 95%</td>
    <td>s</td>
    <td>The time by which 95% of the total energy has been accumulated within the selection.</td>
  </tr>
  <tr>
    <td>Dur 90%</td>
    <td>s</td>
    <td>The duration over which 90% of the signal’s energy is distributed, computed as the difference between the 95th and 5th percentile times.</td>
  </tr>
  <tr>
    <td>Delta Freq</td>
    <td>Hz</td>
    <td>The difference between the upper and lower frequency limits of the selection.</td>
  </tr>
  <tr>
    <td>Delta Time</td>
    <td>s</td>
    <td>The difference between the beginning and ending times of the selection.</td>
  </tr>
  <tr>
    <td>Time 5% Rel.</td>
    <td>–</td>
    <td>The relative time (as a proportion of total duration) at which 5% of the signal’s energy is accumulated.</td>
  </tr>
  <tr>
    <td>Time 25% Rel.</td>
    <td>–</td>
    <td>The relative time at which 25% of the signal’s energy is accumulated.</td>
  </tr>
  <tr>
    <td>Center Time Rel.</td>
    <td>–</td>
    <td>The relative time corresponding to the median (50%) of the signal’s energy distribution.</td>
  </tr>
  <tr>
    <td>Time 75% Rel.</td>
    <td>–</td>
    <td>The relative time at which 75% of the signal’s energy is accumulated.</td>
  </tr>
  <tr>
    <td>Time 95% Rel.</td>
    <td>–</td>
    <td>The relative time at which 95% of the signal’s energy is accumulated.</td>
  </tr>
  <tr>
    <td>Peak Time Relative</td>
    <td>–</td>
    <td>The time at which the peak amplitude occurs, expressed as a proportion of the total selection duration.</td>
  </tr>
  <tr>
    <td>PFC Avg Slope</td>
    <td>Hz/ms</td>
    <td>The average slope of the peak frequency contour over time, computed as the mean of the differences between successive peak frequencies.</td>
  </tr>
  <tr>
    <td>PFC Max Freq</td>
    <td>Hz</td>
    <td>The maximum frequency reached in the peak frequency contour.</td>
  </tr>
  <tr>
    <td>PFC Max Slope</td>
    <td>Hz/ms</td>
    <td>The maximum rate of change (slope) observed in the peak frequency contour.</td>
  </tr>
  <tr>
    <td>PFC Min Freq</td>
    <td>Hz</td>
    <td>The minimum frequency reached in the peak frequency contour.</td>
  </tr>
  <tr>
    <td>PFC Min Slope</td>
    <td>Hz/ms</td>
    <td>The minimum rate of change (slope) observed in the peak frequency contour.</td>
  </tr>
  <tr>
    <td>PFC Num Inf Pts</td>
    <td>–</td>
    <td>The number of inflection points in the peak frequency contour, indicating how frequently the slope changes sign.</td>
  </tr>
</table>

Table 4: Summary of acoustic measurements derived from Raven Pro 1.6. Definitions are adapted from the Raven Pro manual.
Figure 8: tSNE visualizations of five pre-trained embeddings, primarily trained on bioacoustics bird data (with BirdNET also incorporating other animals). The first column represents the t-SNE plot of call types (Peeps in yellow and Twitters in blue), while the second and third columns depict the t-SNE projections of Peeps and Twitters colored by individual, respectively. From top to bottom, the rows correspond to BirdNET and Perch, respectively.

<table>
  <tr>
    <th>Model name</th>
    <th>Number of parameters</th>
    <th>Training data (hours)</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>BirdNET</td>
    <td>27M</td>
    <td>8300</td>
    <td>Kahl et al. (2021)</td>
  </tr>
  <tr>
    <td>HuBERT-Large</td>
    <td>1B</td>
    <td>60960</td>
    <td>Hsu et al. (2021)</td>
  </tr>
  <tr>
    <td>Perch</td>
    <td>7.8M</td>
    <td>&lt;10k</td>
    <td>Ghani et al. (2023b)</td>
  </tr>
  <tr>
    <td>Wav2vec2</td>
    <td>317M</td>
    <td>54000</td>
    <td>Baevski et al. (2020)</td>
  </tr>
  <tr>
    <td>W2v-BERT 2.0</td>
    <td>600M</td>
    <td>60960</td>
    <td>Hsu et al. (2021)</td>
  </tr>
  <tr>
    <td>Whisper-Large-v2</td>
    <td>1.55B</td>
    <td>680000</td>
    <td>Radford et al. (2022)</td>
  </tr>
  <tr>
    <td>AudioMAE</td>
    <td>304M</td>
    <td>5500</td>
    <td>Huang et al. (2023)</td>
  </tr>
</table>

Table 5: List of considered models for acoustic embeddings, including their size, training data, and references.

Reviewer paper summaries: The paper demonstrates a way to identify non-human species from their speech using pre-trained speech identification models of a completely different species. In particular they demonstrated it on White-Faced Capuchin Monkeys. Speech from a set of 45 monkeys in Taboga forest was recorded for about 2 years, with the task of creating a model that can help identify a particular id from their "peep" and "twitter" calls. Instead of training a new ML model for identifying the monkeys, the paper uses pretrained models of Whisper and Perch which are targeted for human and bird speech respectively. The paper hypothesizes that many speech patterns are similar across species and they got decent results on using those models for monkey identification.

They expanded their research by filtering and then combining the features from both of these models for identification purposes. They tried summation, concatenation and finally Minimum Redundancy Maximum Relevance (MRMR) techniques for combining the features. They mention a technique of iteratively filtering the embedding features that provide a good balance of diversity and informativeness. Overall they got an improvement in performance when combining the two multi-species models, with MRMR providing the best performance.

The paper focuses on the acoustic individual identification of White-Faced Capuchin monkeys. Specifically, it explores a cross-species pre-training method by employing the cross-species acoustic embeddings from birds and humans to solve the data sparsity problem for some specific types of wild animals. Experimental results show that joint multi-species representations can be effectively used to identify the calls from individual monkeys.
Summary of strengths: 1. The paper body and text is well written and is able to convey the underlying approach well.
2. The explanations and analysis are quite exhaustive. The hypothesis for using cross-species models and combining the features is discussed well. Emphasis is made on domain relevance of models than their size/complexity, which is demonstrated by better performance being observed from Perch than a more complex Whisper model.
3. It seems like a lot of effort has gone into curating the dataset. This dataset would be very useful for further analysis in this field which could help in various conservation efforts.

1. The research topic of acoustic individual identification is very interesting and novel. I believe it is highly related to the personalized NLP tasks for human language and may benefit from the methods and ideas in this field. 
2. The idea of employing cross-species acoustic features is very interesting. It will be better if the findings from the machine learning method can be explained by or are consistent with the biology theory.
Summary of weaknesses: While this paper demonstrates impressive results with this novel idea of combining the encodings, I agree with the limitations section that the analysis is a bit preliminary. It is a bit early to say whether this approach will work well for call types that are more complex than peep or twitter. The encoding combination techniques might have to be revised by doing a deeper investigation into the social functions of these calls.

1. Adding a figure to show how the system works will make it easier to understand the paper.
Comments, suggestions and typos: 1. Page 7 Appendix is a blank page. Change the formatting to avoid any blank pages.
2. Figure 1 caption is confusing at a first glance. It is too concise and feels a bit cluttered. Add some more explanation to make it clearer. Or you can reverse the order in which you describe the components. I feel like it is better to start with describing (E-F), then E, then F and then show the variation in A-D. 
3. Figure-2 shows two graphs but the caption only suggests peeps call type.

It would be very beneficial for the research community if the collected dataset could be released.
Overall assessment scores: 4

4
Metareview score: 4
Metareview: The paper investigates how cross-species pre-training can enhance individual identification of White-Faced Capuchin Monkeys (WFCM). The authors combine embeddings from models trained on human speech and bird bioacoustics, demonstrating significant improvements in acoustic identification performance for WFCM. Moreover, they reveal that smaller models trained on a diverse set of bird vocalizations outperform larger models trained on human speech data, despite the closer phylogenetic relationship between humans and WFCM.
Summary of reasons to publish: The manuscript is well written and accessible, providing comprehensive explanations and detailed analyses throughout. The rationale for employing domain-specific, smaller models is well-established, underscoring their potential significance for future research in the field. Moreover, the dataset—collected over a two-year period under field conditions—enhances the study's value. Finally, the research paves the way for novel avenues in cross-species transfer learning and computational approaches to animal communication.