DOI: 10.5061/dryad.sbcc2frf7
title: Tassie BRUV: A benchmark data set for computer vision and movement quantification algorithms

description: The Tassie BRUV dataset is an image dataset that we generated as a
 benchmark dataset using Baited Remote Underwater Videos (BRUVs) from chain
 moorings and environmentally friendly moorings in Tasmania. These videos
 were collected by CSIRO and OzFish volunteers as part of a separate study
 to observe if there are differences in the community abundance of fish
 species between chain and environmentally friendly moorings in this area.
 This dataset can be used to benchmark computer vision techniques, and, in
 conjunction with the surrounding video footage, techniques that include
 movement information.
description: # Tassie BRUV: A benchmark data set for computer vision and movement
 quantification algorithms Dataset DOI:
 [10.5061/dryad.sbcc2frf7](10.5061/dryad.sbcc2frf7) The Tassie BRUV dataset
 is an image dataset that we generated as a benchmark dataset using Baited
 Remote Underwater Videos (BRUVs) from chain moorings and environmentally
 friendly moorings in Tasmania. This data set is used in the paper
 "The Motion Picture: Leveraging Movement to Enhance AI Object
 Detection in Ecology", published in Ecology and Evolution. These
 videos were collected by CSIRO and OzFish volunteers as part of a separate
 study to observe if there are differences in the community abundance of
 fish species between chain and environmentally friendly moorings in this
 area. This dataset can be used to benchmark computer vision techniques,
 and, in conjunction with the surrounding video footage, techniques that
 include movement information. For code to fit such models, refer to
 [https://github.com/BenMaslen/MCD](https://github.com/BenMaslen/MCD). With
 camouflage, occlusion, schooling, poor visibility, and bio-mimicry, this
 underwater video dataset poses unique challenges for computer vision. ##
 Description of how the data was generated 28 Baited Remote Underwater
 Videos were used to generate this benchmark data set. These BRUVs were
 placed next to chain moorings (16), control locations (6) (locations
 within a site with no moorings), and environmentally friendly moorings
 (6), being located at either North West Bay (7), Battery Point (4) or
 Sandy Bay (17) in Tasmania. The start and ends of these videos were
 cropped such that the camera being lowered and raised from the sea floor
 was removed and a time stamp in the top left hand corner of the video was
 blacked out to ensure it doesn't confound the effect of measurements
 of movement. Then, 50 frames were randomly selected from each of these
 videos and bounding boxes with fish id's to species level were
 labelled for each of them. Fish were detected using the videos as well as
 the frame image to help observe any camouflaged fish that were present.
 Then a further 100 frames were randomly selected from each of these videos
 and they were inspected if they contained any fish whose counts in the
 previous labelling process were less then 100 (referred to as 'rare
 fish'). Of the 2,800 frames, 512 had rare fish in them which were
 additionally labelled to boost the labels of rarer species in the dataset.
 This brought the total number of frames annotated to 1912, which were
 split into training (1340), test (380) and validation (192) sets (in the
 training_data_species_grouped folder). Unfortunately, there were still
 many species with low counts, so multiple species were grouped into higher
 order taxonomic levels with similar morphological attributes. In total
 5,222 fish annotations were generated, with 3,834 of these annotations
 being Platycephalus bassensis (Southern Sand Flathead), a benthic fish
 that lies on the sandy ocean floor, camouflaging itself by burying in the
 sandy sediment to ambush prey. There was also a clear baiting effect
 present in the videos with majority of fish being annotated close to the
 bait in front of the camera. ## Files, variables, and data structure This
 dataset has the following folder structure (files not included): ```
 Tassie_BRUV.zip ├── augmented_images │ ├── abs_u_15_d_1 │ ├──
 back_sub_td_400_li_500_r │ ├── diff_BS_bn_20_bs_120_u_15_d_1_r_pca │ ├──
 dir_u_15_d_1_r │ └── flow_1_r_pca ├── location_data ├── meta_data ├──
 training_data_species_grouped ├── training_data_species_high_counts └──
 surrounding_videos ``` Within each augmented images folder (`abs_u_15_d_1,
 back_sub_td_400_li_500_r, diff_BS_bn_20_bs_120_u_15_d_1_r_pca,
 dir_u_15_d_1_r, flow_1_r_pca`) and training data folder
 (`training_data_species_grouped, training_data_species_high_counts`), we
 have the following sub file structure (in accordance with training of YOLO
 models): ``` labelled_image_folder ├── images │ ├── test │ ├── train │ └──
 valid ├── labels │ ├── test │ ├── train │ └── valid ├── .cache ├──
 data.yaml ├── explained_var └── pca.pkl ``` Within each
 `images/` and`labels/`  sub folder, contains images and .txt labels (in
 YOLO format) used to test, train or validate an object detection
 algorithm. The `.cache` file is a caching file and can be ignored, and the
 `data.yaml` file is fed into a YOLO object detection algorithm and details
 the aforementioned training data structure and class label names (for the
 augmented image folders, this is using the
 `training_data_species_grouped` labels). This file will need to be edited
 appropriately to describe the parent directory structure before using in
 an object detection algorithm. Finally the `pca.pkl` file contains the
 principal components used for image augmentation, and the
 `explained_var` file details the proportion of explained variance using
 each principal component. ### augmented_images Within this folder we have
 some example augmented images that were used in the paper "The Motion
 Picture: Leveraging Movement to Enhance AI Object Detection in
 Ecology" published in Ecology and Evolution. Please refer to this
 paper for more details on the augmentation techniques. Each folder
 contains augmented images in YOLO txt format to be used for training a
 YOLO object detection algorithm. These images were augmented using code
 from [https://github.com/BenMaslen/MCD](https://github.com/BenMaslen/MCD),
 with the originals being found in the folder
 `training_data_species_grouped`. Below details the augmentations used for
 each example folder: `abs_u_15_d_1` - Frame differencing using an absolute
 value method, scaled by 15, with a displacement distance of 1 frame and
 replacing all the image data. `back_sub_td_400_li_500_r` - Background
 subtraction using the K-nearest Neighbour algorithm, replacing only the
 red image layers. `diff_BS_bn_20_bs_120_u_15_d_1_r_pca` - Background
 subtraction using frame differencing based approach, with 20 background
 images, each 120 frames apart. The values were scaled by 15, with the
 three colour layers transformed into two colour layers using PCA (the
 explained variance of the PCA can also be found in the parent directory),
 and the third layer being the average of the resulting background
 subtraction across the three layers. `dir_u_15_d_1_r` - Frame differencing
 with a direction based approach, replacing just the red layer.
 `flow_1_r_pca` - Optical flow using using PCA to transform the three
 colour layers into two colour layers, and the third layer being the
 magnitude of the measured optical flow. ### location_data Example csv file
 demonstrating the format of the `location_data.csv` file that needs to be
 generated in order to use the augmentation code in
 [https://github.com/BenMaslen/MCD](https://github.com/BenMaslen/MCD). This
 csv file has the following column headings: *image_name -* Name of the
 image in the labelled image folder *split -* Which training data split the
 image belongs to (either test, train or valid). *frame -* Frame number the
 image was sourced from. *vid_location -* Location of the video file that
 the labelled image was sourced from. ### meta_data Dataset metadata. This
 folder has two files; `species_annotation_counts.csv` and
 `TB_species_all_metadata.csv`. The `species_annotation_counts.csv` file
 has the following column headers: *Species -* The species class of each
 labelled fish species. *Family -* The family class of each labelled fish
 species. *Order -* The order class of each labelled fish species.
 *Annotation grouping -* The annotation grouping for each labelled fish
 species from the `training_data_species_grouped` labelled images.
 *Annotation counts -* The number of annotation counts per species class.
 The `TB_species_all_metadata.csv` file has the following column headers:
 *Species -* The species class of each labelled fish species. *Family -*
 The family class of each labelled fish species. *Order -* The order class
 of each labelled fish species. *Annotation counts -* The number of
 annotation counts per species class. *Mobility -* Mobility category (how
 quickly each species moves in and out of the view of the video).
 *Schooling -* How much the species likes to form schools of fish.
 *Baiting_impact -* How much the baiting effect impacts each species. ###
 training_data_species_grouped Labelled training data in YOLO txt format.
 Species have been grouped into higher order taxanomic or morphological
 groupings to avoid species groups with low counts. This is the training
 data used in the paper "The Motion Picture: Leveraging Movement to
 Enhance AI Object Detection in Ecology". You can find more details on
 how the species were grouped in this paper and by referencing the
 `meta_data/species_annotation_counts.csv` file. ###
 training_data_species_high_counts Labelled training data in YOLO txt
 format. Only species with high counts (&gt;75 annotations) are included in
 this dataset, with others not labelled. ### surrounding_videos Baited
 Remote Underwater Videos in which the labelled training data was sourced
 from. These videos are required in conjunction with the location_data
 files in order to estimate movement to augment the labelled frames using
 the code from
 [https://github.com/BenMaslen/MCD](https://github.com/BenMaslen/MCD). ##
 Accompanying code: MCD All code used to generate the example augmented
 images in this dataset as well as produce the results in the paper
 "The Motion Picture: Leveraging Movement to Enhance AI Object
 Detection in Ecology" published in Ecology and Evolution can be found
 the MCD github repository:
 [https://github.com/BenMaslen/MCD](https://github.com/BenMaslen/MCD) ##
 Access information Data was derived from the following sources: * These
 videos were collected by CSIRO and OzFish volunteers as part of a separate
 study to observe if there are differences in the community abundance of
 fish species between chain and environmentally friendly moorings in this
 area.