# Overview

This folder contains metadata related to the AquaMonitor dataset.

Folder structure:

class_maps: Contains files that are used for mapping class strings to indices

dna: contains metadata files for the DNA subset

images: image files in WebDataset format (~108GB)

thumbnail: Resized and contrast-adjusted thumbnail images in the same structure as the main dataset for visualization purposes (~10GB)

Files:

aquamonitor.parquet.gzip: Full dataset metadata, without train-test-splits. Each row corresponds to a single image.

aquamonitor-individual.parquet.gzip: Full dataset metadata, aggregated by individual.

aquamonitor-monitor.parquet.gzip: Full dataset metadata, with train-test-splits for the monitor task.

aquamonitor-classif.parquet.gzip: Full dataset metadata, with train-test-splits for the classification task.

aquamonitor-fewshot.parquet.gzip: Full dataset metadata, with train-test-splits for the few-shot task.

aquamonitor-biomass.parquet.gzip: Metadata for biomass estimation, with additional columns for biomass-related metadata.


# Metadata

## Common columns

img (str): The image filename. Unique identifier across the dataset.

individual (str): Identifier for the individual specimens.

imaging_run (str): Identifier for the imaging run. Some specimens have been imaged multiple times, and this ID refers to a single "drop" through the cuvette. Consists of individual id with running number suffix. With bulk imaged samples each individual has been imaged only once, resulting in 'individual' == 'imaging_run'

taxon (str): The scientific name of the specimen. Refers to both adult and juvenile forms.

taxon_group (str): Specifies the visual group of the specimen. Some taxa have specimens from multiple visual groups, like juvenile and adult forms. If the taxon has no such subgroups, this value is the same as 'taxon'. Used for classification benchmarks.

taxon_label (str): Original label used during morphological identification. Matches the labels used in the monitoring databases. Was used during imaging and matches image filenames.

taxon_code (str): The abbreviation of the taxon_label. Matches filenames and identifiers.

lake (str): The lake the specimen was collected from.

site (str): Sampling site.

sample (str): The taxon-site combination. Before imaging the samples were stored in containers with these groupings.

plate (str): (Only on DNA and biomass subsets) The 96 well plate ID.

position (str): (Only on DNA and biomass subsets) The position on the 96 well plate. Values range in [A-H] for rows and [1-12] for columns.

camera (int): The camera the image is from. Values in {1,2}

area (int): The area of the specimen in the image. From BIODISCOVER imaging software.

perimeter (int): The perimeter of the specimen in the image. From BIODISCOVER imaging software.

holes (int): Number of morphological holes in the image segmentation. From BIODISCOVER imaging software.

max_feret_diameter (int): Maximum Feret diameter of the segmentation object. From BIODISCOVER imaging software.

area_holes (int): The area of the morphological holes in the segmentation. From BIODISCOVER imaging software.

roi_left (int). The left position of the image crop, relational to the full cuvette. 

roi_top (int). The top position of the image crop, relational to the full cuvette.

roi_right (int). The right position of the image crop, relational to the full cuvette.

roi_bottom (int): The bottom position of the image crop, relational to the full cuvette.

width (int): width of the image.

height (int): height of the image.

imaging_time (datetime): Imaging timestamp.

dataset (str): the dataset subset image belongs to. Values in {'dna', 'biomass', 'bulk21, 'bulk22'}.

year (str): Sampling year. Values in {'2021', '2022'}.

is_bulk (bool): whether the image was taken in the bulk imaging phase.

has_dna (bool): The individual has DNA sequence data.

has_biomass (bool): The individual has biomass information.


## Unique columns

classif:

fold0-fold4 (str): The benchmark split the image belongs to, for five cross-validation folds. Values in {'train', 'val', 'test', '0'}. Values of '0' correspond to images in classes that have less than 50 specimens.


monitor:

fold0: The benchmark split the image belongs to. Values in {'train', 'test', 'val'}

taxon_group_train: taxon_group, mapped to only classes present in the 2021 (train) dataset. Classes unique to 2022 are set as 'Other'.

taxon_group_test: taxon_group, mapped to only classes present in the 2022 (test/val) dataset. Classes unique to 2021 are set as 'Other'.


classif:

fold0-fold4 (str): The benchmark split the image belongs to, for five cross-validation folds. Values in {'train', 'val', 'test'}


biomass:

well_col: column in the value 'position'. Values in range [1-12].

well_row: row in the value 'position'. Values in range [A-H].

weight: weight of the specimen in milligrams. Calculated from plate_weight and total_weight.

log_weight: natural log of the weight.

log_weight_xplus1: weight in the form ln(x+1), where x is the value in 'weight'

plate_weight: The measured plate weight.

total_weight: Weight of the plate with the specimen.

caddisfly_without_shell: If value is not nan, this is the total weight of caddisfly samples weighted without the shell.


# Mapping files

Class maps:

class_map_classif.txt: Mapping for the 42 classes with over 50 specimen, used in the Classification task

class_map_monitor_test.txt: Classes in the 2022 target dataset for the Monitor task

class_map_monitor_train.txt: Classes in the 2021 source dataset for the Monitor task



Taxa mapping files:

site_mapping:

    site_label: String of the site information present in the monitoring database.
    site_code: Abbreviation for the site. Of form abc0, with a three-letter abbreviation for the lake and number 1-3 for the site.
    lake_code: Above, but without the site number.
    lake: Lake string in ASCII format and lowercase
    lake_label: Lake string in proper spelling, including umlauts.
    
taxon_mapping:

    taxon (str): The scientific name of the specimen. Refers to both adult and juvenile forms.
    taxon_group (str): Specifies the visual group of the specimen. Some taxa have specimens from multiple visual groups, like juvenile and adult forms. If the taxon has no such subgroups, this value is the same as 'taxon'. Used for classification benchmarks.
    taxon_original: Label used during morphological identification. Matches the labels used in the monitoring databases. Was used during imaging and matches image filenames.
                    Corresponds to taxon_label in the metadata files.
    taxon_label: Above, but in computer-readable camelcase format. Example: 'Coenagrion juv.' -> 'Coenagrion_juv'
    
taxon_table.csv: Defines the hierarchical structure that can be used to map labels to higher taxonomic levels.
taxon_types.csv: Maps different subgroups and visual groups to the actual scientific name of the specimen.



# Notes

The DNA subset images are rotated 90 degrees. The roi metadata has been fixed but the images are still rotated.

The tables in folder dna/ differ in taxa names compared to the main dataset. The full dataset has taxon 'Mystacides azurea' while the same species is names 'Mystacides azureus' in the OTU data. The morpho taxon groups differ also in the way that 'Oulimnius tuberculatus' is divided to 'adult' and 'larva' groups, while in the full dataset larva is labeled just 'Oulimnius tuberculatus'.