DOI: 10.26190/unsworks/31841
title: GAVDNet Automated Animal Call Detector - Trained Models & Validation Test Results

description: This dataset contains the validation testing results for the GAVDNet automated animal call detector framework. GAVDNet was tested on two baleen whale calls, the Antarctic blue whale Z-call, and the Chagos pygmy blue whale. For each target call, we trained and tested four different models, each with training samples synthesised at a different range of random signal-to-noise ratios. Additionally, in order to evaluate precision-recall trade-off, we also tested each of these eight (8) models with a range of post-processing activation threshold values. For each of the 2 target animal calls, this yielded 4 models, and 8 activation thresholds, for a total of 64 experimental conditions. Due to unknown reliability of the annotation logs used as ground truth, the best performing model was then subjected to human adjudication, which resolved disagreements between the detector and the ground truth. The results for all tests these tests are included in this repository. Additionally, the trained model weights are also provided. This repository does not contain the original audio data on which the detectors were tested. The original audio datasets are not the property of the authors and cannot be redistributed.
description: Two reference datasets were used to test the detector. Each one consists of a collection of audio files, and an annotated reference log. The ABWZ detector tests utilised the “Casey 2014” dataset, from the open access IWC-SORP Annotated Library (Miller et al., 2020, 2021). Audio was recorded off the coast of Casey Station, Antarctica, at latitude -63.7955, longitude 111.7871 (Figure 6), between 2013-12-25 and 2014-12-12. Audio data was sampled at 1,000 Hz, (down sampled in the present study to 250Hz) and the dataset contained 194 hours of audio. The annotations were performed by a human analyst, and the annotation log contained 1,091 ABW Z-calls. The hydrophone depth was 2,770 meters, and while mooring type was not specified, depth and bathymetry suggest direct sea-floor mooring. The CPBW detector was tested using audio data sourced from the Comprehensive Nuclear Test Ban Treaty Organisation’s International Monitoring System and was recorded on hydrophone H08S1, off the south coast of Diego Garcia Island in the Chagos Archipelago, central Indian Ocean at latitude −7.6453, longitude 72.4744 (Figure 7), throughout 2007. The hydrophone is moored by cable to the sea floor and suspended at a depth of 1,413 meters by subsurface float. The audio data was sampled at 250 Hz. The reference annotations for the CPBW dataset were generated during a previous study using an automated detector and manually reviewed by a human analyst (Leroy et al., 2021). This dataset of reference annotations has been previously used by other studies, and is considered to be of high quality (J. L. Huang et al., 2022). The complete dataset consists of over 8,000 hours of audio, so to reduce it to a more manageable size for testing, a MATLAB program was written to subsample the data creating a balanced, representative subset. The detector used in (Leroy et al., 2021) was designed to return estimates of the SNR and signal-to-interference-plus-noise ratio (SINR) for every detection, and these were used in the subsampling procedure, which operated as follows: the annotation reference log was filtered by removing any detections that referred to audio files that were missing, corrupt, did not contain valid audio data, or were shorter than 10 minutes in duration. The program then identified which audio files did and did not contain detections, based on the reference annotation log. Detections in the reference log were then stratified into bins based on the following dimensions: SNR, SINR, hour of day, and month of year. The sampling procedure selected detections proportionally from each bin to maintain the dataset’s original distribution across all these dimensions, continuing until the target test set duration of 500 hours was reached. 1% of the audio files included in the final dataset contained no detections. The final test dataset consisted of 500 hours of test audio and a reference log of 6,843 timestamps corresponding to instances of the CPBW song. After running the detector on each dataset for each experimental condition, the results were compared to the ground truth annotations. Timestamps for detections were matched to ground truth annotation timestamps using the Hungarian algorithm for optimal assignment, whereby optimal one-to-one matching of a detection to a ground truth annotation minimized temporal mismatch. Matches were only considered valid if temporal mismatch was less than 30 s. The numbers of true positives (TP), false positives (FP) and false negatives (FN) were counted, then performance metrics were calculated following standard definitions: precision=TP/(TP+FP), recall (aka. sensitivity)=TP/(TP+FN), and F1 score=2 ∙(precision∙recall)/(precision+recall). The disagreements (FP’s and FN’s) were saved for further analysis. The reference annotation logs used as ground truth were known to be unreliable. To resolve disagreements between the detector and the ground truth, an adjudication procedure was undertaken by an experienced analyst. The analyst was familiar with both target whale songs and was provided with a simple GUI application to display spectrograms and record decisions. The spectrograms displayed to the analyst began 5 s before the detection timestamp to provide context and ended either 40 s from the detection start time, or at the detection end time, whichever was longer. To avoid bias, analyst was not permitted to know whether the displayed detections were false positives, false negatives, or true detections. To evaluate the influence of chorus on the detector’s performance, the analyst was asked to respond to each disagreement with one of four options: “Discrete Call(s) Present”, “Chorus Present”, “Discrete Call(s) and Chorus Present” or “No Call(s) or Chorus Present”.