DOI: 10.6084/m9.figshare.23053382
title: Supplementary material to the journal article: An active learning framework and assessment of inter-annotator agreement facilitate automated recogniser development for vocalisations of a rare species, the southern black-throated finch (<em>Poephila cincta cincta</em>)

description: <strong>Primary author details</strong> <br> John van Osta<br> ORCID: 0000-0001-6196-1241<br> Institution: Griffith University and E2M Pty Ltd<br> Queensland, Australia<br> Email: john.vanosta@griffithuni.edu.au<br> <br> Researchers and practictioners applying and adapting the data and code provided here are encouraged to contact the primary author should they require further information.<br> <br> <strong>Sharing/Access information</strong><br> Licence CC BY 4.0. You are free to share and adapt the material provided provided appropriate attribution is given to the authors. <strong>Data and File Overview</strong> <br> This repository provides the code base, supplementary results and example audio data to reproduce the findings of the research article: 'An active learning framework and assessment of inter-annotator agreement facilitate automated recogniser development for vocalisations of a rare species, the southern black-throated finch (<em>Poephila cincta cincta</em>)', published in the Journal of Ecological Informatics. Data included within this repository are listed below.<br> <strong>Code base</strong> The code base includes:<br> - train_resnet.ipynb: Trains a resnet34 model on target and non-target audio segments (each 1.8 seconds in duration). Outputs a trained model (as a pth file).<br> - predict.ipynb: Applies the trained model to unlabelled data.<br> - BTF_detector_v1.5 is the latest version of the model, termed the 'final model' in the the research article.<br> - audio_file_extract.ipynb to extract audio frames in accordance with the active learning function. For the purpose of manual review and inclusion in the next iteration of model training.<br> - stratified_subsample.ipynb: Used to subsample predictions on unlabelled data that are stratified across the model prediction confidence scores (aka logit).<br> - macro_averaged_error.ipynb: Calculate and plot macro averaged error of the model predictions against annotator labels.<br> - inter_annotator_agreement.ipynb: Calculate and plot Krippendorff's alpha (a measure of inter-annotator agreeement) among the model's active learning iterations and human annotators.<br> - requirements.txt: Python package requirements to run the code base.<br> <br> Note: The code base has been written in Jupyter Notebooks and tested in Python version 3.6.9<br> <br> <strong>Supplementary files</strong> The file Stratified_subsample_inter_annotator_agreement.xlsx contains predictions from each model iteration and annotator labels for each of the 12,278 audio frames included in the model evaluation process, as described in the research article.<br> <br> <strong>Example audio data</strong> Example audio data provided include:<br> - Target audio files (containing black-throated finch (BTF) calls) and non-target audio files (containing other environmental noises). These are split into Training and Validation sets. To follow an active learning process, each active learning 'iteration' gets added to a new folder (i.e. IT_1, IT_2, etc..).<br> - Field recordings (10 minutes each), the majority of which contain BTF calls.<br> These audio data were collected from a field site within the Desert Uplands Bioregion of Queensland, Austrlaia, as described and mapped in the research article. Audio data were collected using two devices: Audiomoths and Bioacoustic records (Frontier Labs), which have been separated into separate folders in the 'Field_recordings'. <br> <strong>Steps to reproduce</strong> <br> <strong>General recommendations</strong> The code base has been written in Jupyter Notebooks and tested in Python version 3.6.9.<br> 1. Download the .zip file and extract to a folder on your machine.<br> 2. Open a code editor that is suitable for working with Jupyter Notebook files. We recommend Microsoft's free software: Visual Studio Code (https://code.visualstudio.com/). If using Visual Studio Code, ensure the 'Python' and 'Jupyter' extensions are installed (https://code.visualstudio.com/docs/datascience/jupyter-notebooks).<br> 3. Within the code editor open the downloaded file.<br> 4. Setup the python environment by installing the package requirements identified within the requirements.txt file contained within the repository. The steps to setup a python environment in Visual Studio Code are described here: https://code.visualstudio.com/docs/python/environments, or more generally for python described here: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/. This will download the necessary python packages to support the below code. <br> Note: We recommend running the following steps on a Windows computer with an Nvidia graphics processing unit (GPU). The code has also been tested on a Windows computer with an Intel computer processing unit (CPU), with a substantially slower runtime. Edits to the code may be required to run on a Macintosh computer or a non-Nvidia GPU; however, the core functionality will remain the same. <br> <strong>Active learning iterations to develop the final model:</strong><br> 1. Run train_resnet.ipynb to train a model from the initial target (BTF) and non-target (other environmental sounds) audio provided. The default name for the output model will be 'model.pth', however this may be adjusted manually by changing the 'MODEL_NAME' variable. The script also provides performance metrics and a confusion matrix against the validation dataset.<br> 2. Run predict.ipynb to make predictions on unlabelled data. The default code uses the final model (BTF_trained_model_v1.5.pth), as described in the research article, however this may be adjusted to link to the model created in step 4 (by changing the 'model_path' variable). Results of this step are saved in the Sample_files\Predict_results folder.<br> 3. Run audio_file_extract.ipynb to extract 1.8 second audio snips that have a 'BTF' confidence score of &gt;= 0.5. these are the sounds that range from most uncertain to the model to most likely to be BTF. The logic for this cutoff is discussed in the research article's methods section. The default extraction location is 'Sample_files\Predict_results\Audio_frames_for_review'.<br> 4. Manually review extracted audio frames and move them to the appropriate folder of the training data. E.g. for audio frames that are reviewed to contain:<br> - BTF calls, move them to the filepath 'Sample_files\Training_clips\Train\BTF\IT_2'<br> - Not BTF calls, move them to the filepath 'Sample_files\Training_clips\Train\Not BTF\IT_2'<br> IT_2 represents the second active learning iteration. Ensure 30% of the files are allocated to the validation set ('Sample_files\Training_clips\Val). Note that users will need to create subfolders for each successive iteration.<br> 5. Repeat steps 1 to 4, making sure to update the 'iterations' variable in the train_resnet.ipynb code to include all active learning iterations undertaken. For example, to include iterations 1 and 2 in the model, set the variable 'iterations'to equal ['IT_1', 'IT_2']. An example is provided in the train_resnet.ipynb code.<br> 6. Stop the active learning process when the stopping criterion is reached (e.g. when the F1 score plateaus).<br> <strong>Model evaluation steps</strong><br> 1. Run predict.ipynb using the final model on an unlabelled test dataset. By default the unlabelled audio data that will be used is saved at the filepath: example data saved in the filepath 'Sample_files\Field_recordings\Audiomoth'. However, this should be changed to data not used to train the model, such as 'Sample_files\Field_recordings\BAR', or your own audio data.<br> 2. Run stratified_subsample.ipynb to subsample the predictions that the final model made on the unlabelled data. A stratified subsample approach is used, whereby samples are stratified across confidence scores, which is described in the research article. The default output file is 'stratified_subsample_predictions.csv'<br> 3. We then manually reviewed the subsamples, including a cross review by experts on the species, as detailed in the research article.<br> We have provide the results of our model evaluation: 'Study_results\Stratified_subsample_inter_annotator_agreement.xlsx'<br> 4. Run macro_averaged_error.ipynb and inter_annotator_agreement.ipynb to reproduce the results and plots contained within the paper.<br> <br> <strong>Using the model on your own data</strong><br> The predict.ipynb code may be adapted to run the BTF call detection model on data outside of this repository.<br> <br> Notes for running on your own data:<br> - Accepts wav or flac files<br> - Accepts files from Audiomoth devices, using the file naming format: 'AM###_YYYMMDD_HHMMSS'<br> - Accepts files from Bioacoustic Recorder Devices (Frontier Labs), using the file naming format: 'BAR##_YYYMMDDTHHMMSS+TZ_REC'<br>