DOI: 10.5061/dryad.w6m905qx8
title: Vocalization data and scripts to model reindeer rut activity using on-animal acoustic recorders and machine learning

description: For decades, researchers have employed sound to study the biology of
 wildlife, with the aim to better understand their ecology and behaviour.
 By utilizing on-animal recorders to capture audio from freely moving
 animals, scientists can decipher the vocalizations and glean insights into
 their behaviour and ecosystem dynamics through advanced signal processing.
 However, the laborious task of sorting through extensive audio recordings
 has been a major bottleneck. To expedite this process, researchers have
 turned to machine learning techniques, specifically neural networks, to
 streamline the analysis of data. Nevertheless, much of the existing
 research has focused predominantly on stationary recording devices,
 overlooking the potential benefits of employing on-animal recorders in
 conjunction with machine learning. To showcase the synergy of on-animal
 recorders and machine learning, we conducted a study at the Kutuharju
 research station in Kaamanen, Finland, where the vocalizations of rutting
 reindeer were recorded during their mating season. By attaching recorders
 to seven male reindeer during the rutting periods of 2019 and 2020, we
 trained convolutional neural networks to distinguish reindeer grunts with
 a 95% accuracy rate. This high level of accuracy allowed us to examine the
 reindeers’ grunting behaviour, revealing patterns indicating that older,
 heavier males vocalized more compared to their younger, lighter
 counterparts. The success of this study underscores the potential of
 on-animal acoustic recorders coupled with machine learning techniques as
 powerful tools for wildlife research, hinting at their broader
 applications with further advancement and optimization.
description: Bioacoustics data were collected during the rutting seasons of
 2019 and 2020. In 2019, vocalizations were captured from two males, while
 in 2020, vocalizations were obtained from six males. However, the sampling
 time varied from three days to two months due to equipment issues.
 Moreover, challenges with the recorders limited our data collection to
 only seven male reindeer (two from 2019 and five from 2020).
 During the translocation of the males, they were outfitted with
 on-animal acoustic recorders. These devices housed SOROKA-15E recording
 units from TS-Market Ltd., Zelenograd, Russia, capable of capturing the
 animals’ vocalizations with an amplitude resolution of 16 bits and a
 sampling rate of 16 kHz (<strong>Figure 2</strong>). The
 recorders enabled us to obtain continuous audio data throughout the
 breeding period. For storage purposes, each recorder was equipped with a
 256-gigabyte microSD card, providing the capacity to record over 92 days
 of audio. Furthermore, a 9,000-milliamp-hour, 3.6-volt lithium-ion battery
 powered each recorder, ensuring functionality for over two
 months. Observers documented each male's status
 and harem during the ruts. Based on data gathered from agonistic
 interactions among males recorded during the rut, a clear dominance
 hierarchy was evident, established primarily on win-lose frequencies. The
 size of a male's harem was noted throughout these observations.
 Dominant males, commanding a harem, were given a corresponding label,
 whereas those without were categorized as subdominant. Fluctuations in
 dominance among numerous males occurred based on the presence or absence
 of more dominant males. Dominance during the rut was mainly determined
 through observations. Observers monitored the males' status for about
 20% of the rutting period, providing the basis for audio playbacks.
 Specific sounds and behaviors were documented during these observed hours,
 serving as a model to categorize the remaining recordings according to
 these known patterns. Each hour of each male was analyzed through audio
 playback, with status potentially changing based on displayed behaviors
 and grunts. Typically, a male's status remained constant for extended
 periods, with observed shifts from dominant to subdominant and back to
 dominant occurring in as quickly as three days. Notably, males did not
 continuously alternate statuses during these periods, and when a male lost
 a harem to another male, the displaced male often remained subdominant for
 an extended period. In cases where a male was not
 observed for periods (owing to them occasionally being unlocatable),
 assessments of their status were conducted during playback of their
 recordings. For instance, if a playback revealed a male grunting with
 audible background sounds of females and calves, it was inferred that he
 held a dominant status. Conversely, the presence of rival male grunts
 directed at the subject male, if met with silence on the latter's
 part, suggested his sub-dominance. If a recording of a male featured no
 other sounds and he wasn't grunting, it was presumed he was in search
 of a herd and thus subdominant. Nevertheless, in instances where a male’s
 status was ambiguous during playback and no observational data was
 available, his status was marked as ‘unknown.’ Confirmation of such males’
 statuses was deferred until more definitive evidence could be
 obtained. We employed Sonic Visualiser (Cannam et al.,
 2010) for the analysis and annotation of the recordings. Each recording
 underwent annotation using the “boxes layer” feature, enabling the
 delineation of intentional and unintentional noises via bounding boxes.
 This allowed for the documentation of start and end times for each sound
 within the recordings, as well as the minimum and maximum frequency of
 each vocalization. Notably, a binary label (presence or absence) was
 assigned to each bounding box. For the presence class, bounding boxes were
 strategically positioned around individual and repeating vocalizations to
 capture the grunting behaviours of the reindeer. Incidental calls from
 other reindeer were intentionally left unannotated to prevent an excessive
 impact on the CNN training due to the over-sampling of activities of the
 focal individuals. Bounding boxes of varying lengths were placed and
 utilized to annotate the absence class throughout each recording, ensuring
 comprehensive CNN training by encompassing a wide range of unintentional
 sounds (biophony, geophony, and anthropophony). The bounding boxes were
 strategically positioned between presence segments to capture a wide range
 of sounds. Throughout the process, we made a conscious effort to exclude
 segments without any noise to enhance the precision of the final
 classifier. Furthermore, within the absence class, vocalizations of any
 species overlapping with the males’ frequency range were annotated.
 Subsequently, Sonic Visualiser was utilized to validate the vocalization
 predictions made by the CNN and to eliminate any false positive or false
 negative predictions. <em>Machine Learning
 methodology and process:</em> We utilized a
 supervised learning approach to train our CNNs, employing code and
 methodology adapted from Dufourq et al. (2022). Initially, we annotated
 135 audio segments from our 2020 recordings, amounting to 25 recordings
 from five individuals and 10 from another. Unfortunately, due to technical
 issues with one of the recorders, we had to exclude the annotations from
 this individual; their inclusion led to a decrease in the network’s
 performance. Owing to limitations in computer hardware, we worked with a
 subset of the 125 annotated audio segments, specifically chosen to
 encompass a wide range of recordings across individuals, from various
 environments and weather conditions. Consequently, our 2020 training set
 comprised 8,605 presence segments (augmented to 14,000; augmentation of
 data was done through time-shifting of existing presence annotations) and
 18,000 absence segments, with 14,000 being randomly sampled to ensure a
 balanced dataset. This preliminary network was then utilized to extract
 vocalizations from our 2019 recordings. After
 annotating the recordings, we  had to train our CNN,  thus we searched for
 appropriate hyper-parameters. After experimenting with over 40 different
 hyper-parameter combinations, the values outlined in <strong>Table
 2</strong> yielded the best-performing network. In the case of CNNs,
 fixed inputs are required, and given the considerable variation in the
 duration of multiple grunts—ranging from a fraction of a second to over 30
 seconds—window lengths exceeding four seconds did not improve performance.
 Since grunt variations within a series of grunts are minimal, segments
 lasting longer than four seconds led to increased computation time without
 any performance gains. During our hyper-parameter search, we also focused
 on determining the minimum and maximum call frequency values. Given that
 the fundamental frequency of reindeer grunts falls below 100 Hz, we set
 the minimum frequency to zero Hz while the maximum frequency was set to
 4000 Hz. While formant frequencies of reindeer grunts tend to be
 indiscernible after 2500 Hz, setting the maximum frequency to 4000 Hz
 enhanced network performance, likely due to the provision of supplementary
 information that aided in distinguishing grunts from unintentional noises.
 However, setting the maximum frequency above 4000 Hz did not improve
 network performance and instead led to increased computation
 time. Four pre-processing steps were then conducted to
 prepare the inputs for the CNNs, mirroring those detailed in Dufourq et
 al. (2022). Firstly, a low-pass filter was applied to each audio file to
 capture signals below its cut-off frequency, which minimizes aliasing
 artifacts that can stem from downsampling. The filter's cut-off
 frequency was determined based on the maximum frequency of the males'
 vocalizations in our presence annotations. Notably, the grunts of the
 males became indiscernible beyond 1000 Hz in Sonic Visualizer, with
 occasional louder calls detectable beyond this point, prompting the low
 pass filter's cut-off to be set at 1000 Hz.
 Subsequently, downsampling was performed on each audio file to
 enhance computational efficiency, given that higher frequencies were
 deemed excessive for the analysis. Sampling rates beyond 4000 Hz were
 found to offer no advantage in network performance and only increased
 computational demands. Therefore, the Nyquist rate was set at 4000 Hz, and
 the downsampling rate at twice that value (Dufourq et al.,
 2022). Further, annotations for both classes were
 extracted using a sliding window approach. This involved segmenting each
 audio file into four-second windows initiated at the bounding box's
 start time, subsequently generating spectrograms that were then labeled
 accordingly (either presence or absence). This sliding window process
 continued until the window overlapped with the bounding box's end
 time, encompassing all bounding boxes across the entire dataset.
 Lastly, the audio segments were transformed into two-dimensional
 mel-frequency spectrograms. The specific values associated with this
 transformation are outlined in <strong>Table
 2</strong>. During our training process, we
 evaluated various pre-trained models detailed in  Dufourq et al. (2022),
 and among them, the ResNet152V2 model (He et al., 2016) exhibited the best
 performance. We fine-tuned both the feature extractor and the output layer
 to achieve the best possible model, although this adjustment led to
 increased computation time, it enhanced the network's performance.
 Similar to the approach outlined in Dufourq et al. (2022) and recognizing
 the necessity for pre-trained models to have a three-channel input (often
 aligning with the three channels in a colour image), we implemented the
 exponent method as described in the study. Specifically, we utilized the
 S<sup>1</sup>, S<sup>3</sup>, and
 S<sup>5</sup> channels to generate our three spectrogram
 channels in line with the methodology outlined above.
 To determine a spectrograms classification, the CNNs predicted
 two softmax outputs on each spectrogram within an entire testing file. The
 final classification (presence or absence) was determined according to the
 softmax output surpassing a value of 0.5.  Each file was predicted by
 using a four second sliding window approach. The window shifted one second
 at a time until an entire recording was processed.
 After training the preliminary CNN, we used it to collect
 vocalizations from the 2019 recordings, and verified the presence
 annotations manually to ensure accuracy and eliminate false positives. We
 also included absence annotations to provide the network with additional
 training data. We then used a subset of our 2020 annotations and all
 annotations from our 2019 recordings to train a final CNN. This final
 model was trained on a dataset of 10,778 presence and 11,546 absence
 annotations over 25 epochs. The final CNN was then used to detect reindeer
 vocalizations for the entire dataset. The number of files for each
 individual age group were: 1.5-year-old, 11 files, 2.5-year-old, 58 files,
 , 3.5-year-old 1, 22 files, 3.5-year-old 2, 53 files, 3.5-year-old 3, 21
 files, 4.5-year-old, 28 files, and 5.5-year-old, 93 files each file was
 eight hours long). It's worth noting that some files were less than 8
 hours due to recorder failure during the recording. To
 evaluate the network's performance, we conducted tests using 12 audio
 files, none of which were used during the network's training phase.
 These files represented individuals from across the rut and various
 rutting behaviours. For instance, some files contained minimal
 vocalizations, while others featured over 250 instances. Among these
 recordings, eight were from the 2020 data, while the remaining four were
 from the 2019 data. To assess the network's performance, we used
 recall rate, precision, accuracy and F1 scores  (Mesaros et al., 2016;
 Navarro et al., 2017; <strong>Equation
 A1-4</strong>). To train the networks, we
 utilized the packages listed in <strong>Table A1</strong>. The
 script was executed using Python 3, and the CNNs were implemented in
 Tensorflow 2 (Abadi et al., 2016). Each CNN underwent training across
 epochs employing the Adam optimizer with a batch size of 32 (Kingma
 & Ba, 2014). Subsequently, spectrograms were generated using the
 Librosa library (McFee et al., 2020). Model training and testing took
 place on a 2021 Apple MacBook Pro equipped with an Apple M1 Pro processor
 and 16 GB of LPDDR5 RAM, operating on MacOS Ventura 13.1.
description: # Vocalization data and scripts to model reindeer rut activity using
 on-animal acoustic recorders and machine learning \***If the user wishes
 to just analyze the data, simply download the compressed in the root
 file*** \****During the Data upload process, the data and code had to be
 uploaded separately. Consequently, to place the back into software
 folders, simply download the corresponding compressed data folder and
 place the uncompressed data folder within the root folder of the specific
 folder it corresponds to (e.g., the  2-Rut Analysis (Data) folder into the
 2-Rut Analysis and rename the  2-Rut Analysis (Data) folder to simply
 ‘data’ so that the software can find and pull data from said folder) **0 -
 File_Annotation** This file is separated into 4 subfolders, each similar
 to the other. The two SVL folders contain files that are easily imported
 and read by Sonic Visualizer. The other two are the same, but they are in
 CSV format in case a user wishes to train another machine-learning network
 using our annotations. The corresponding audio can be found in /Sample
 (with a small subset)/1-Training_Folders: Verification data/Data/Audio.
 There are 134 files and the 2020 files are 2.26 hours long, while the 210
 files are 8 hours long. 0.5-Sample-Training-and-predicting example This
 folder contains a small subset of data in case the users wish to test the
 code on their system. The code is set up in a duplicate manner to that
 found in the two following folders, just with a much less intensive audio
 subset (in case the user's computer has limited computation power).
 The ‘.ipnyb’ are the only two files that need to be interacted with and
 they can be opened with Jupyter Notebook or imported into Google Colab.
 Begin with the Training. ipynb file and follow the prompt to train a
 simple machine-learning network. The process should take about 10-15
 minutes, depending on the power of your computer. Then once that initial
 process is complete, open the Predicting_Fine_Tuning.ipynb file in the
 notebook and again follow the prompts, this process should take about 10
 minutes as well. Afterwards, SVL files will be created, with the
 network's predictions. These files, along with their respective .WAV
 files can then be imported into Sonicvisualizer to inspect its
 predictions.  Within the /Data folder, there are a number of subfolders.
 /Annotations is where the binary annotations for the .WAV files are read
 from in order to train the networks.  /Audio is where the individual .WAV
 files are stored and read from.  /DataFiles are the training and testing
 .txt files are read from. These lists are read by the script to indicate
 which annotations and .WAV files the network should either process or
 predict upon. These files, need to keep the same title as they are
 currently labelled and the names of the files within need to be exactly
 the same as the annotation and .WAV files, without their file types. 
 /Model_Ouput is where the trained networks will be saved (as a .hdf5 file,
 and later, to use the predicting script, this file must be transferred to
 the root folder, e.g., that containing the Predicting_Fine_Tuning.ipynb
 file for instance) and the .SVL files or the network-predicted sonic
 visualizer layers files, which are imported by the user to inspect the
 network's predictions. /Pickled_Data and /Saved are metadata files
 that contain script/data generated by the script, the user does not need
 to be concerned with the contents of these files
 **1-Training_Folders/Verification data** This folder is identical to the
 previous one, only that it contains all the annotation and .WAV files used
 to train our papers network. Note that while the /Audio folder contains
 134 audio files, only a subset of these were actually used to train the
 network, (34 audio files) due to hardware limitations. The full set of
 audio files corresponds to those found in the 0 - File_Annotation folder,
 should the user wish to train their network with a more powerful computer,
 with a larger annotation dataset. Consequently, the TestingFiles and
 TrainingFiles .txt files correspond to the subset of audio and annotation
 used to train our final network. The list of 12 files within the
 TestingFiles.txt corresponds to the 12 audio files used to verify the
 performance of our networks, as described in our paper. The Verification
 info subfolder includes network predictions (Both in .SVL and .SV file
 formats [the audio for these files can be found in the aforementioned
 /Data/Audio folder. To inspect them yourself, simply import them into
 Sonic Visualizer. The .xlsx file corresponds to our inspection of these
 files and our count of TP, FP and FN files and our calculations for Recall
 rate, Precision, F1 and Accuracy. The two first tabs are the performance
 of a couple of preliminary networks, and the “Final Set” tab is the
 calculations reported within our paper.  **2-Rut Analysis** Again, this
 file is set up in a duplicate manner to the previous two, but in this one,
 the subfolders within the /Data folder are empty, as such, I will leave it
 to the user to analyze what files they want, the audio and annotation
 files are organized as follows. Additionally, the training side of this
 folder, as in this folder, it is meant to simply be the predicting folder,
 whereas the above folder is meant to be one where the user trains their
 network. If the user trained a network in the above folder, and wishes to
 use their weights to predict upon files with this folder, the user should
 copy and paste their .hdf5 network file into this folder, in a manner
 duplicate to that mentioned above. -The audio files are contained within
 the /Sample (with a small subset)/2-Rut Analysis/Day_Audio_segments, here,
 the files are organized by the individual. These files are all eight hours
 long, so that each day is easily divided into three sections. Only a few
 of the audio files are less than 8 hours, as they are when the reindeer
 were put into the mating pen, or their recorders unfortunately failed. 
 -And the testing files are found within the /Sample (with a small
 subset)/2-Rut Analysis/Network Predictions/Test_Files subfolders. Again,
 organized by the individual. Should the user wish to predict some of the
 audio files, simply place the appropriate .WAV files within the
 above-mentioned/Audio folder and the appropriate TestingFiles.txt files
 within the /DataFiles folder. If the user wishes to only predict a couple
 of files, they will have to edit the TestingFiles.txt file with the
 appropriate files they wish to predict. If the user wants to view our
 network's predictions, choose the appropriate .WAV from the location
 above and find the .SVL from the file below, and place them in
 SonicVisualizer. -The annotations files are found within the Sample (with
 a small subset)/2-Rut Analysis/Network Predictions/Original unprocessed
 SVL Files/Prediction SVL subfolders, again, these are organized by the
 individual. These were the original predictions as made by our network and
 remain unprocessed by us. The user can also import CSV files into
 SonicVisualizer as well.  These are the audibly and visually verified
 prediction annotations produced by the network, as mentioned in our paper.
 -These annotations can be found in the /Sample (with a small subset)/2-Rut
 Analysis/Network Predictions/Processed CSV FIles (1/3 day segments)
 subfolder. In these ones, FP predictions were removed, and FN predictions
 added, when possible, as it is likely some vocalizations were missed.  The
 /Sample (with small subset)/2-Rut Analysis/Network Predictions/Day Segment
 Descriptions (along with original FL designations and recording notes)
 subfolder contains notes on the process as we made the 8-hour segments,
 along with their original file designations and notes as they were
 inspected.  The /Sample (with small subset)/2-Rut Analysis/Network
 Predictions/Prediction .XLS Day Segments **AND** /Sample (with small
 subset)/2-Rut Analysis/Network Predictions/Prediction .XLS Amalgamations -
 folders contain Excel sheets that were used to tabulate and calculate our
 proportional data. The ‘Day segments’ folder contains individual day files
 for each individual as we had recordings for them. Each day had 3 segments
 and we used these forms to describe their hourly proportional activity
 across the entire day. The ‘Day’ data from these files was then
 transferred to the ‘Amalgamations’ folder and the Excel sheets there, as
 they correspond to the individual, so that we could describe the reindeer
 activity as it changed over time, from hour to hour of each day, and from
 day to day. This file has tabs to describe their activity in terms of just
 days, 8-hour daily segments, and then hourly segments. This was done early
 on in the process as a form of data exploration. Finally, the
 /Users/alex_boucher/Desktop/Sample (with small subset)/2-Rut
 Analysis/Network Predictions/Rut Activity Analysis (R Analysis) include
 the R code used to analyze our data. In this file, our final master Excel
 file can be found, and the R code is streamlined for the user to analyze
 our data as well. To look at our master Excel file, refer to: Boucher et
 al. (2023) - Reindeer Rut Activity Trends/2-Rut Analysis/Network
 Predictions/Rut Activity Analysis (R Analysis)/MasterRuttingActivity.xlsx
 **Notes** -Refer to
 Boucher_A-MLReindeer/1-Training_folders/TrainingSets.xlsx and
 Boucher_A-MLReindeer/1-Training_folders/TestSets.xlsx to see a list of
 files used to train the algorithms - for both, I used the half set - for
 the absence segments, there were 18218 ‘no grunt’ spectrograms and 8506
 ‘grunt’ spectrograms - I augmented the grunt spectrograms by a factor of 2
 for a total of 17012 grunt segments and 18218 - the test vs train
 annotation sets was set 16:37, or roughly a 30:70 test to train split -
 down-sampled the no_grunt total to 17025 to balance the totals - total
 sample of 34037 segments -To train/test the algorithms, the code was run
 on an M1 Pro 16-inch MacBook Pro with 16 GBS of RAM using Apple metal
 architecture   -Training times - 10 Epoch - Training: ~600 seconds per
 Epoch, Finetuning: ~1850 - total = ~24500 seconds, 408 minutes, 6.8 hours;
 50 Epoch - Training: ~600 seconds per Epoch, Finetuning: ~1850 - total =
 ~122500 seconds, 2041 minutes, 34.3 hours -Transfer learning - used
 ResNet152V2 for transfer learning
 - [https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html](https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html) -We had an individual whose recordings were muffled, hence we did not use this individual for this study. If you reference SD46, this is a remnant from before we stopped considering this individual and should be disregarded. The Individuals, as identified in our paper, were done so as follows: GPS39 = 2019 - 5.5 GPS43 = 2019 - 2.5 SD24 = 2020 - 1.5 SD39 = 2020 - 4.5 SD42 = 2020 - 3.5 - 1 SD44 = 2020 - 3.5 - 3 SD47 = 2020 - 3.5 - 2 **Copyright Information** Note that most of this code was adapted from that supplied within the below listed study - to use code in further studies, contact Emmanuel Dufourq for permission - code repository can be found at Passive acoustic monitoring of animal populations with transfer learning - Emmanuel Dufourq, Carly Datist, Ruben Foquet, Ian Durbach - Ecological Informatics Volume 70, Sept 2022, 101688 - [https://www.sciencedirect.com/science/article/pii/S1574954122001388](https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sciencedirect.com%2Fscience%2Farticle%2Fpii%2FS1574954122001388&data=05%7C02%7Crobert.weladji%40concordia.ca%7C0ba30fb07884487c536508dc63ad0787%7C5569f185d22f4e139850ce5b1abcd2e8%7C0%7C0%7C638494842081715868%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=l7OKmhIFZFBT2OuVTAR0fQmi7NI74rNulFKafOEG6Z0%3D&reserved=0) Code used was the ‘Hainnon Gibbon - 25 sample - Fine tuning set’ Below is the copyright information copied from the original code license  MIT License: Copyright (c) 2022 Emmanuel Dufourq Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.