Title: BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity
Benchmark: BIOSCAN-5M
Summary: BIOSCAN-5M is a multimodal benchmark for insect classification and contains images, taxonomic labels, raw nucleotide barcode sequences, barcode index numbers, geographic location, and size metadata. The dataset is publicly available and includes data from novel species. The benchmark supports classification, zero-shot transfer learning, and retrieval learning.
Contribution: BIOSCAN-5M is an expansion of BIOSCAN-1M. It is unique in its inclusion of 4 million additional images, and location, taxonomic rank, and size metadata. Addtitionally, BIOSCAN-5M was cleaned to resolve inconsistencies and provide more reliable labels. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: insect classification
Phenomenon defined by authors: Yes
Phenomenon definition: Insect classification is the automatic classification of inset specimens by AI tools. 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The benchmark outlines three possible tasks. The first task is insect classification, which can be performed as DNA-based and/or image-based taxonomic classification. In a closed-world setting, the task is to accurately identify species from a predefined set of existing labels. In the open-world setting the task is to group together samples of novel species. The benchmark also supports zero-shot transfer-learning, which measures how unseen datasets can be clustered using embeddings from pre-trained feature extractors, and multimodal retrieval learning by aligning image, DNA, and taxonomic label embeddings using CLIBD. 
Task item definition: A single item in the dataset contains the biological taxonomy (phylum, class, order, family, subfamily, genus, species), the genetic information (DNA barcode sequence, barcode index number), a cropped and original RBG image of the insect, size information (meas. value, scale factor, area fraction), and geographical information (coordinates, country, province/state). 
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task source detail: Dataset is an superset of previous datasets with additional metadata. 
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 47,260
Dataset size notes: There are four types of species labels that each carry their splits. Unknown samples are samples without a species label. Seen samples are samples with an established scientific name for their species. Unseen are samples with an established scientific name for the genus, and a uniquely identifying placeholder name for the species. Heldout samples are labelled with a placeholder genus and species name.   Unknown: Pretrain 4677756 Seen: Train/Validation/Test 289203/14757/38373 Unseen: Retrieval Keys/Validation/Test 36465/8819/7887 Heldout: 76590
Dataset metadata available: Yes
Dataset metadata detail: phylum, class, order, family, subfamily, genus, species, DNA barcode sequence, barcode index number, original RBG image, cropped RBG image, measured value, scale factor, area fraction, country, province/state, latitude, longitude 
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric statistics: Accuracy is reported for classification in both open and closed-world settings. Fine-tuned accuracy and linear probing accuracy are reported in a closed-world setting, while 1NN-genus probing accuracy is reported in an open-world setting. AMI is reported for zero-shot transfer learning, and in multimodal retrieval learning, micro and macro top-1 accuracy is reported. 
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors highlight that a strong dataset for insect classification requires detailed metadata, and geographic and specimen diversity. BIOSCAN-5M, compared to other benchmarks, covers 98% of discovered insects with 1.2 million labeled to the species rank, and contains geographical information, size, and DNA barcodes. The authors claim that multimodal datasets are critical for robust species classification. 
Taxonomy: Biology / 
New taxonomy: Specific Application / Physical Sciences