A Step Towards Worldwide Biodiversity Assessment:  The BIOSCAN-1M Insect Dataset
Authors: Zahra Gharaee
ZeMing Gong
Nicholas Pellegrino
Iuliia Zarubiieva
Joakim Bruslund Haurum
Scott C Lowe
Jaclyn McKeown
Chris C.Y. Ho
Joschka McLeod
Yi-Yun Catherine Wei
Jireh Agda
Sujeevan Ratnasingham
Dirk Steinke
Angel X Chang
Graham W. Taylor
Paul W. Fieguth
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Insect biodiversity
Image classification
Class-imbalance distribution
Fine-grained classification
Taxonomic classification
DNA barcode sequences
Barcode Index Number (BIN)
TL;DR: The paper presents the BIOSCAN-1M Insect Dataset, a million-image collection of hand-labelled insect images with genetic data, aimed to train computer vision models for taxonomic assessment and foster machine learning applications in biodiversity.

Abstract: In an effort to catalog insect biodiversity, we propose a new large dataset of hand-labelled insect images, the BIOSCAN-1M Insect Dataset. Each record is taxonomically classified by an expert, and also has associated genetic information including raw nucleotide barcode sequences and assigned barcode index numbers, which are genetic-based proxies for species classification. This paper presents a curated million-image dataset, primarily to train computer-vision models capable of providing image-based taxonomic assessment, however, the dataset also presents compelling characteristics, the study of which would be of interest to the broader machine learning community. Driven by the biological nature inherent to the dataset, a characteristic long-tailed class-imbalance distribution is exhibited. Furthermore, taxonomic labelling is a hierarchical classification scheme, presenting a highly fine-grained classification problem at lower levels. Beyond spurring interest in biodiversity research within the machine learning community, progress on creating an image-based taxonomic classifier will also further the ultimate goal of all BIOSCAN research: to lay the foundation for a comprehensive survey of global biodiversity. This paper introduces the dataset and explores the classification task through the implementation and analysis of a baseline classifier. The code repository of the BIOSCAN-1M-Insect dataset is available at https://github.com/zahrag/BIOSCAN-1M

A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-IM Insect Dataset
Zahra Gharaee ZeMing Gong*, Nicholas Pellegrino Iuliia Zarubiieva 2:5 Joakim Bruslund Haurum", Scott C. Lowe Jaclyn T.A. McKeown 1.2, Chris C.Y. Ho
Joschka McIeod 1.2, Yun C Wei Jireh Agda Sujeevan Ratnasingham" 1.2 Dirk Steinke Angel X. Chang Graham W. Taylor? Paul Fieguth Centre for Biodiversity Genomies, University of Guelph, University of Waterloo,
"Simon Fraser University, VVector Institute for AI, 6 AAlberta Machine Intelligence Institute (Amii), 7 AAlboor University and Pioneer Centre for AI, & Dalhousie University
Abstract
In an effort to catalog insect biodiversity, we propose a new large dataset hand-labelled insect images, the BIOSCAN-IM Insect Dataset. Each record is taxonomically classified by an expert, and also has associated genetic information including raw nucleotide barcode sequences and assigned barcode index numbers, which are genetically-basse proxies for species classification. This paper presents curated mallion-image dataset, primarily to train computer-vision models capa- ble of providing image-based taxonomic assessment, however, the dataset also presents compelling characteristies, the study of which would of interest to the broader machine learning community. Driven by the biological nature inherent the dataset, characteristic long-tailes clas--mmallance distribution is exhib- ited. Furthermore, taxonomic labelling is a hierarchical classification scheme, presenting highly fine-grained classification problem at lower levels. Beyond spurring interest in biodiversity research within the machine learning community, progress on creating an image-based taxonomic classifier will also further the ultimate goal of all BIOSCAN research: to lay the foundation for comprehensive survey of global biodiversity. This paper introduces the dataset and explores the classification task through the implementation and analysis of baseline clas- sifier. The code repository of the BIOSCAN-IM Insect dataset available at https://github. wwwpiittu..comaaaaeaeeeee www.zahraa////////ooooaaaaaaaa BIOSCAN 1M
Introduction
Global change is restructuring ecosystems on a planetary scale, creating an increasingly urgent need to track impacts on biodiversity. Such tracking is exceptionally challenging because life is highly diverse: the biosphere comprises more than 10 million multicellular species [42]. Until recently, this complexity has meant that Earth observation system for biodiversity was inconceivable. however the increased power of DNA sequencing and the recognition that living organisms can be discriminated by short stretches of DNA have revealed way forward, which has become the central focus of the International Barcode of Life (iBOL) Consortium.
can revolutionize our understanding of bio- diversity, not only by providing reliable species proxy for known and unknown species, but also by revealing their interactions and assessing their responses changes in the ecosystem. This is essential to mitigate a looming mass extinction, where an eighth of all species may become extinet by 2100 unless there is significant change in human behaviour [10
Discriminating organisms by DNA sequences [22 6
Joint first author.
Joint senior/last author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Class:
Insocta
Taxonomic Classification
Order:
Diptera
Family:
Culicidao
Subfamily: Culicinac
TTTAATTTTAATTTTGGAAACAAAAAAAAAAA
DNA Barcode
GACTTTAAATAATTTTTTTTTTAAAAAAAAAA
CTTAATTGTTAAACCCCACCCTTTAATTTTTT
Barcode Index Number (BIN)
BERD:6
Figure 1: BIOSCAN-IX 1M Insect dataset records contain high-quality microscope images of insects and labels including the taxonomic classification, raw DNA sequences, and Barcode Index Number (BIN). Pictured here is a mosquito of the subfamily Culicinae, the most populous subfamily of mosquitoes with species found around the world.
The BIOSCAN project 2], lead by iBOL, has the following three main goals:
1. Species discovery.
2. Studying the interactions between species.
3. Tracking and modelling species dynamics over geography and time.
To that end, BIOSCAN collects samples of multicellular life from around the world. Each sample individually imaged, genetically sequenced and barcoded [22], and then classified by expert taxonomists. Of particular interest to the BIOSCAN project are insects, which constitute a great proportion of the Earth's 's species and many of which remain unknown. Indeed, is estimated that 5.5 insect species exist worldwide, of which only roughly one million have been identified [54] (23). The rate of insect collection within the BIOSCAN project is increasing as the project progresses. such that 3 M insect specimens will be collected in 2023 and 10M by 2028.
Using high-resolution photographs, human taxonomists accurately classify insects from within their domain of expertise. However, human annotation cannot scale to the volume of samples needed measure and track global biodiversity. Moreover, many taxonomists with highly specialized knowledge are leaving the practice and won't It be replaced. Thus, the use of artificial intelligence and machine learning process visual and textual information collected by the BIOSCAN project is crucial to the success of a planet-scale observation system. Classification of the insect images to their taxonomic group ranking is especially useful in regions of the world where the facilities required to perform genetic barcoding are not available. Indeed, even beyond this project, there are opportunities for computer vision to transform entomology 25
This article has two main contributions.
1. The publication of the BIOSCAN- IM Insect image dataset, containing approximately 1.1 M high-quality microscope images, each of which is annotated by the insect's taxonomic ranking and accompanied by its raw DNA sequences and Barcode Index Number (BIN) 47 an example of which is shown in Figure
2. The design and implementation of deep model, classifying BIOSCAN-IM Insect images into specific taxonomic ranking groups, to serve as a baseline for future work utilizing this dataset.
2 Background and Related work
This section provides background on taxonomic classification, the use of genetic barcoding, and several challenges in the field of machine learning associated with dataset.

---

2.1 Taxonomic Classification
In biology, taxonomic classification the study of hierarchically categorizing lifeforms based on shared characteristics. In particular, Linnean taxonomy 77 20 32] forms the basis for the modern (generally accepted) system of taxonomy, of which the main hierarchical ranks are domain, kingdom, phylum, class, order, family, genus, and species, shown in Figure All insect life is part of the class Insecta.
Conventionally, expert taxonomists classify organisms based on their appearance and behaviour [7. However, this approach is susceptible to both misclassification and lacks consensus throughout the community of taxonomists, since it is difficult to prove with certainty that given classification is absolutely correct. This shortcoming of traditional taxonomy has prompted the use of classification heuristics, based on fairly concrete evidence in the form of genetic codes, that are sensitive to species identity.
Table presents a comprehensive overview of both the number of unique categories and the degree which the BIOSCAN-IM Insect dataset is labelled at each taxonomic rank. According to the table, a substantial number of samples lack labels at more specific taxonomic ranks. This limitation underscores the challenge of incomplete taxonomic labeling within the dataset. However, the ongoing efforts of expert curation are continuously improving this aspect, and future versions of the dataset are expected to include a more comprehensive taxonomic classification for larger number of samples.
Based on the statistical analysis of the BIOSCAN-1M I Insect dataset, is evident that there are 3,441 distinct categories the genus level and 8,355 the species level. However, the amount of data to train such fine-graines classifiers is limited, with 254,096 samples classified by experts at the genus level and only 84,397 samples the species level. This substantial mass-number-to-datee imbalance poses a notable challenge when training models for fine-grained classification, specifically the genus and species level.
Table 1: An overview of the number of unique categories and number proportion of expert-labelled samples within the BIOSCAN-IM Insect dataset at each taxonomic rank. Additionally, the bottom row, the corresponding information is given in relation to Barcode Index Number (BIN), presented as a genetic alternative to taxonomic labels (species proxy). Observe that all samples have an associated BIN, and there are roughly 10x more unique BINs than species labels.
Taxonomic Level
Categories
Labelled Samples
Labelled (%)
Phylum
1,128,313
100.0
Class
1,128,313
100.0
Order
16
1,128,313
100.0
Family
491
1,112,968
98.6
Subfamily
760
265,492
23.5
Tribe
535
60,477
5.4
Genus
3,441
254,096
22.5
Species
8,355
84,397
7.5
Barcode Index Number (BIN)
90,918
1,128,313
100.0
2.2
Genetic Barcoding and Barcode Index Numbers
DNA barcoding 22 employs large-scale screening of or few reference genes for assigning unknown individuals to species, as well as aiding in the discovery of new species (43. Barcoding is commonly used in several fields including taxonomy, ecology, conservation biology, diet analysis and food safety [49 [49]5]] (33. It is faster and more accurate than traditional methods, which rely on the
judgment of experts 46
Barcoding is based on the use of a short, standardized segment of mitochondrial DNA, typically portion of the mitochondrial cytochrome oxidase subunit I (COI) gene, which is nearly always unique for different species. Once the DNA sequence is obtained, can be compared to a reference library of known sequences to identify the species.
The concept of genetic barcoding can be taken a step further by mapping barcodes to clusters of organisms (characterized by their barcodes) with highly similar genetic code, known
3

---

Operational Taxonomic Unit (OTU) [2 S]. OTUs act as proxy for species based on the high degree of genetic similarity exhibited by their members. To enable indexing, each OTU is assigned a Uniform Resource Identifier (URT), commonly referred to as the Barcode Index Number (BIN) 47 which offers unique representation such that genetically identical taxa will be assigned the same BIN, and each BIN is registered in the Barcode Of Life Data system (BOLD) D. BINs additionally provide an alternative to the use of Linnean names, offering a genetics- based classification of organisms.
BOLD, the Barcode of Life Data System M. is pivotal resource in biodiversity science. It facilitates DNA barcode acquisition, storage, validation, and analysis, integrating molecular, morphological, and distributional data. BOLD hosts 17 million specimen records and 14 million barcodes, spanning over a million species worldwide. It plays a central role species identification, genetic diversity exploration, and evolutionary studies. Launched in 2005, BOLD aims to identify all eukaryotic species and offers integrated analytical tools, comprehensive data management, and secure collaboration.
2.3 Machine Learning Challenges
As will be demonstrated in Section 3 the dataset exhibits two key characteristics corresponding to open problems in the field of machine learning.
Class imbalance. The degree to which the expected quantity of instances varies between classes is known the class imbalance. In the context of a closed dataset, the class imbalance describes the disparity in number of examples among classes 27 30]. As we describe in Section B and Figure 2 the published dataset exhibits a long-tailes class distribution whereby the distribution of class sizes closely follows power-law, indicating that there is substantial class imbalance. This represents challenge due to the disproportionate amounts of available training data for majority vs. minority classes
Class Distribution: BIOSCAN-Insect Dataset
Order
1.000.000
10,000
100
896,324 896,
89,38 311
47.328
45.970
32,538
9,635
2,088
1,296
L,057
824
Diptera
Hymenoptera
Coleeptera
Hemiptera
tepidaptera
Paerodea
Thydanophara
Trichopfsra
Orthoptera
Biamadaa
Neurpptera
Ephemeroptera
Demaptera
Archasognatte
Plecoptera
Embluptera
Class Distribution: BIOSCAN-Dipters Dataset
Family Count
10,000
100
253 2.04
150.456
135,991
92.266
79:38
46.89
35,590
17,737
12,047
Hybotidde 9.505
Mycstophi lidec lidoo 6,363
Muscidae 5,974
5.307
Chrorspidae 5,164
3,764
3,496
2.756
2,557
2,296
2.267
2,144
1,340
1:138
941
833
Tophnitides tidae 805
725
Clusiddo 666
Fipur Fipurculidee 634
Sepsidae 521
Syrphidan 518
Bombyl lidee 465
402
380
369
Bibionidae 315
Richare lidoo 276
Culicidee 270
Simu lidel 270)
Psilidoo 237
Cecicom Gecicomyitcle
Chiranamidae
Phoridae
Scarridoo
Cerstopogenidas
Rsydhodidde
Dollehopodidae
Sphatroceridee
Drosophi lidae
Liman iddo
Statopsidee
Milichidaon
Ephivrrdda
Agronyzidae
Tachinidoe
Empididee
Antt myyildee
Helcomyzidae
Lauranidae iidee
Sarcophagidae
Ulielidao iidoo
Straclomyidee
Karoplatidoe
Dicomyiide
Figure 2: Class distribution and class imbalance in the BIOSCAN-IN Insect dataset. We focus
the 16 most densely populated orders (top) and the 40 most densely populated families (bottom).
Hierarchical classification. Classification problems involving data with labels that are inherently hierarchical present a unique challenge in comparison to simpler "flat" classification problems [50]. The outputs of hierarchical classification algorithms are defined over tree-like class taxonomies, where the relationship between parent and child nodes is given by the asymmetric "is-a" relationship. basic example of this is the relationship that "all dogs are canines, but not all canines are dogs". whereby "dogs" would be child node of the parent node "canines", which itself may be a child of "mammals". The dataset published here perfectly matches this paradigm and may be used to study novel approaches for handling the hierarchical classification problem. Note that the baselines we adopt in this paper do not pursue a hierarchical strategy but instead classify to fixed levels of the taxonomy: order and family. Hierarchical strategies are topic of present and future work.

---

2.4 Biological Datasets
Image-based insect classification 33 most often finds use agricultural settings, where Integrated Pest Management (IPM) systems are used to identify and count harmful insect pests [33 [1]. In combination with this, holistic systems capable of also identifying plant diseases through computer
vision are popular area of research [5 12 40).
Recently, DNA sequences have been analyzed [28 using tools from the field of Natural Language Processing 4 and in particular, through the application of bidirectional encoder representations Indeed, BERT-based models have been used to taxonomically
from transformers (BERT)
[1]
classify genetic sequences 24 41 Other recent work has used DNA barcodes as "side information"
to perform zero-shot species- recognition from images, albeit at a much smaller scale than the
BIOSCAN-IM Insect dataset
Perhaps the best known and largest biological dataset is iNaturalist [56] 26 57 the 2021 version of which contains 2.7 M images from over 10,000 different species of plants, animals, and fungi, specifically with 2,526 species of insects with 663 k annotated insect images. Many insect-specific
image datasets focus on insect as pests found in agricultural settings 62] 39 38
16
37 34
the most prominent of which, the IP102 62 dataset, contains roughly 75 insect images, covering 102 species of common crop insect pests. 19k of these are annotated with bounding boxes for object detection. In the space of plants, the PlantNer-300k [18] dataset has 306 k images labeled by species
and was constructed by sampling the larger PlantNet database 3 Table Malle[[iigh highhlghhs key biological datasets across variety of domains and indicates the degree of class imbalance [13] 3, which is defined as the ratio of the number of samples in the largest to the smallest class.
The BIOSCAN-Order and BIOSCAN-Diptera datasets, introduced in Section 4 refer subsets of the full BIOSCAN-IM Insect dataset for use in order- and family-level classification, respectively. Observe that while there significant variety in the imbalance factor among datasets. the imbalance of BIOSCAN-Order is orders of magnitude greater than that of all the other datasets. The @t@nttet- 300K 18 dataset also has remarkably high class imbalance, exceeding that of the BIOSCAN-Diptera dataset. While high imbalance ratio was expected of the BIOSCAN- IM Insect dataset based on the dataset's biological nature, the metric is highly sensitive due its dependence on only the two most extreme classes.
The iNaturalist dataset encompasses greater number of insect species than any pre-existing dataset. We measured the number of genera (plural of genus) and species that were common across both datasets. Of the 2.526 insect species in iNaturalist and 8,355 species annotated in BIOSCAN-IM Insect, only 153 genera and 62 species appeared in both datasets. This indicates the species in BIOSCAN- 1M Insect predominantly do not appear iNaturalist. Furthermore, based on the number of unique BINs present in the BIOSCAN- 1M Insect dataset, it can be assumed that the dataset in fact encompasses almost 91 distinct (possibly as yet unnamed) insect species, far greater quantity than that of Naturalist.
3 Dataset
This section describes the information made available through the publication of the BIOSCAN-IM Insect dataset, and details the procedures which generated the information.
3.1
BIOSCAN-IM Insect dataset resources
The BIOSCAN-IM Insect dataset provides four main sources of information about insect specimens. Each sample in the dataset consists of a biological taxonomic annotation, DNA barcode sequence, Barcode Index Number (BIN), and a RGB image of single specimen. In the following sections, this information is described in detail.
3.1.1 Biological taxonomy
The BIOSCAN-IM Insect dataset specifies biological taxonomic rank following the Linnean tax- onomy as described in Section 2.1 In addition to the main groups shown in Figure 3 the dataset also provides the subfamily and subspecies ranks. The subfamily rank is an auxiliary (intermediate) taxonomic rank, the next below family but more inclusive than genus. Subspecies is a taxonomic rank below species, and it is used for populations that live in different areas and vary in size, shape, or other physical characteristics, but that can successfully interbreed. Finally, we also provide "Name"
5

---

Table 2: Summary of biological fine-grained and long-tailed datasets. Note that "INaturalist- describes the subset of iNaturalist (2021 version) images that comprises insects. *For the BIOSCAN- 1M Insect dataset, report the number of unique Barcode Index Numbers (BINs) instead of the number of unique Linnean taxonomic species. The BIN is a mitochondrial DNA-based identifier which provides species-like proxy of an organism and can be used an alternative Linnean taxonomy (see Section 2.2).
Name Citation
Domain
Images
Categories
Taxonomic Rank
Imbalance, 3
INaturalist (2021)
All
2,686 k
10,000
Species
1.97
iNaturalist- Insect 57
Insects
663 k
2,526
Species
1.97
Pl@ (@ntNett300K
18
Plants
306 k
1,000
Species
3, 3,604.00
Urban Trees 660
Trees
80 k
18
Species
7.51
IP102 [62]
Insects
75 k
Species
13.63
NA Birds 55
Birds
48 k
400
Species
15.00
LeafSnap 31
Plants
31 k
184
Species
8.00
Pest24 55
Insects
25
24
Common name species
493.95
Flowers 102
45
Flowers
8
102
Genus
1.00
BIOSCAN-1M Insect
Insects
1,128
90,918
Barcode Index (BIN)*
12,491.00
BIOSCAN-Order
Insects
1,128k
16
Order
156,856.75
BIOSCAN-Diptera
Insects
891
40
Family
1,092.61
Domain
Kingdom
Phylum
Class
Order
Family
Tribe
Genus
Species
Eukaryota
Animalia
Arthropoda
Insecta
Hymenoptera Apidae Bees
Apini
Apis
melifer
Eukaryotes
Animals
Arthropods
Insects
Bees, Wasps, Ants
Honey bees
Honey boos
Western
honey bee
Figure 3:
Biological taxonomic ranking and classification. Taxonomic ranks are shown in the top row, with the classification (i.e., labels) for the Western honey bee shown below.
to indicate the lowest (most specific) known rank label. For instance, if there exists species name of an organism, "Name" will show that, but if the organism only has Family level ID, then "Name" will show that ID. Each sample has name and there are in total 10,952 unique names labelled in the dataset.
Not all data samples have labels for all taxonomic ranks recognized in the BIOSCAN-IM Insect dataset. As an example, the family group of the BIOSCAN-IM Insect dataset is indexed by 494 distinct families, however, there are 16,067 data samples that are not associated with any of these families, since they were not classified by human taxonomists. As a consequence, there are many data samples that are not classified into finer-level groups like subfamily, tribe, genus, species, or subspecies. The lack of precise annotation at all ranks is one of the major challenges of BIOSCAN-IM Insect dataset when performing classification tasks.
3.1.2 DNA Barcode and Indexing
Section 2.2 described the concept of genetic barcoding and the generation of Barcode Index Numbers (BINs). The BIOSCAN-IM Insect dataset contains genetic barcodes and BINs for all samples. This information is represented the raw nucleotide barcode sequence, under the Nuccraw field, and the Barcode Index Number (BIN), denoted by uri Independently, the field processid a unique number assigned by BOLD to each record, and sampleid is an identifier given by the collector.
3.1.3 RGB images
The BIOSCAN-IM Insect dataset offers a wealth of information through its collection insect images. The dataset contains high-resolution (2880x2160 pixel) RGB images in JPEG format; Figure Fitree/ysslyyy a selection of images representing insects from 16 most densely populated orders.
We have released multiple packages of the BIOSCAN-IM 1M Insect dataset aimed at different purposes. These packages are organized into 113 chunks, each containing 10 images. The packages include:
Original JPEG Images stored in 113 zip files (2.3 TB).

---

Cropped images organized into 113 zip files (151 GB).
Resized original images which have a size of 256 px on their smaller side (26 GB).
Resized cropped images having size of 256 px on their smaller side (7 GB).
Additionally, we also provide the dataset in HDF5 archive format for both the resized original and cropped images.
(a) Diptera
(b) Hymenoptera
(c) Coleoptera
(d) Hemiptera
896,324
89,311
47,328
46,970 (h) Trichoptera
(e) Lepidoptera
(1) Psocodea
(g) Thysanoptera
32,538 (i) Orthoptera
9,635
2,088 (k) Neuroptera
1,296 (I) Ephemeroptera
Blattodea
1,057 (m) Dermaptera
824
676 (0) Plecoptera
(n) Archaeognatha
(p) Embioptera
66
63
30
Figure 4: Examples of original insect images from 16 orders of the BIOSCAN- 1M Insect dataset. The numbers below each image identify the number of images each class, and clearly illustrate the degree of class imbalance in the BIOSCAN-IM Insect dataset. "Siphonaptera", "Strepsiptera" and "Zoraptera" are removed from classification experiments due to an insufficient number of samples.
3.2 BIOSCAN-IM Insect dataset generation
The BIOSCAN-IM IM Insect dataset consists of specimens mostly collected from three countries (Costa Rica, Canada, and South Africa) using Malaise traps. RGB images the organisms were taken with Keyence VHX-7000 microscope. Images are organized by workflow units: 96-well microplates of which 96 are used in single sequencing run (9, 120 samples at a time).
DNA barcodes of the organisms were generated using a nigh-throughput approach utilizing the Pacific Biosystems Sequel platform, which employs Single-molecule, real-time (SMRT) sequencing to generate long-read length DNA and cDNA. The taxonomic classifications were created by matching the generated barcodes to a reference library on the Barcode of Life Data System (BOLD) at the Centre for Biodiversity Genomics in Canada.
7

---

We provide a comprehensive metadata file alongside the RGB images, which includes taxonomic annotations, DNA barcode sequences, and data sample indexes and labels. The metadata file also contains image names and IDs to locate the corresponding images within the dataset packages. Additionally, it identifies the images associated with the training, validation, and test splits.
4 Experiments
By employing stratified class-based sampling, we methodically curated three subsets of varying sizes from the BIOSCAN-IM Insect dataset. Subsequently, we carried out two distinct sets of classification experiments, yielding total of six datasets. The three subsets, namely Small, Medium, and Large, each comprising approximately 50 200 k, and M data samples, respectively. The initial set of experiments primarily revolves around classifying insect images into their 16 most densely populated taxonomic orders. Subsequently, the second set experiments delves even deeper, focusing on classifying samples within the Order Diptera into their 40 most densely populated families.
4.1 Subset sampling and split mechanism
To create subsets of the BIOSCAN-IM Insect dataset, we followed a systematic two-step process. Initially, we sampled a subset exclusively from the Diptera Order, specifically selecting the 40 families with the highest number of members. This led to the creation of the BIOSCAN- Subsequently, we divided the BIOSCAN-Diptera dataset into separate train, validation, and test sets. Finally, we used these split sets of the BIOSCAN-Diptera dataset as a basis to construct the corresponding train, validation, and test sets for the BIOSCAN-Order dataset. Our methodology involved the use of stratified class-based sampling to preserve the class distribution consistently across all subsets, ensuring the integrity of our experiments.
dataset.
Table 3: The total number of samples used in the six different sized subsets of the BIOSCAN-IM Insect dataset: The entries display the number of data samples in the train, validation, and test sets, as well as the number of classes for order-level (16 orders) and Order Diptera family-level (40 families) experiments.
Dataset
Total
Train
Validation
Test
Categories
BIOSCAN- -Oderr BIOSCAN-
1,128,308
789,813
112,835
225,660
16
891,338
623,937
89,135
178,266
40
DUSSAAN-Orde:/Dipee Medium
200,000
140,000
20,000
40,000
16/40
MOSSCA--Orde/DDipeer Small
50,000
35,000
5000
10,000
16/40
The Small and Medium subsets are generated by sampling 50 and 200 data samples, respectively, from the train, validation, and test sets of the BIOSCAN-Order and BIOSCAN-Dipterz datasets. In all of our classification experiments, we used class-based stratified sampling split the dataset into train, validation and test sets. To this end, 70% of the samples of each class are randomly selected as training, 10% as validation, and 20% as test samples, as shown Table 3
The extreme class imbalances, which are an inherent characteristic of the BIOSCAN-II IM Insect dataset, are addressed to some extent by having all classes represented in the train, validation and test sets. Classes with no samples for either split set are omitted. In the insect order-level classification (Figure 4 we have sufficient data samples for 16 out of 19 orders in the train, validation, and test sets. For the Diptera famiil-level classification, we focus on the 40 most populous families within Diptera.
4.2 Data preprocessing
To improve computational efficiency, we crop and resize the images to be 256 px on the smaller dimension. Preliminary experiments with ResNet-50 comparing original images with images that are cropped show that cropping can help model learning to converge more rapidly and lead to slightly better performance. Reducing the resolution to 256 px helps to reduce the size of the large dataset from 2.3 TB down to 26 GB for the original uneropped images, and from 151 GB down to 7 GB for cropped images. We choose to run experiments on the cropped and resized images due to the small size which allows for efficient data loading from disk.
The BIOSCAN-IM image datasets have insects with varying size, pose, color and shape. Due to these variations, cropping is not a simple task. We develop cropping tool by fine-tuning DETR 99]
8

---

model with ResNet-50 [2]] backbone (pretrained on MSCOCO 35 on small set of 2,000 insect images annotated using the Toronto Annotation Tool Suite 29 In DETR, the CNN-based feature extractor extracts a set of image features that are fed into transformer-baat based encoder-detector. The detector takes a set of learned positional embeddings as object queries and uses them to attend to the encoder outputs. Each of the output decoder embeddings is then passed to a shared FFN which predicts whether there is an "insect" or "no object" and regresses the bounding box. The DETR model is further trained for 10 epochs with the AdamW optimizer with learning rate of 0.0001, weight decay of 0.0001 and a batch size 8.
To crop the image, we apply our fine-tuned DETR model and take the predicted bounding box with the highest confidence score. The finalized cropping is determined as the predicted bounding box, extended equally in width and height by 0.4 of the maximum dimension.
4.3 Classification model
To run classification experiments, fine-tuned two different pre-trained models to extract deep visual features of insects from their RGB images. Our pre-trained models are ResNet-50 [21 and a transformer based model, TT--Bas--Pathhl--2 crops from the image as input, while during validation we take the center crop. To train our model, used two loss functions, the Cross-Entropy (CH) baseline and the Focal loss, which is more
[17]. During training, we take random 224x224
suitable for datasets having class imbalances [3 8 13
5 Results
The detailed hyperparameters used for our experiments are shown in Table For conducting the experiments, we leveraged the computational resources provided by the Digital Research Alliance of Canada's 's Narval and Beluga clusters. To ensure efficient processing. Each experiment was performed using a single node equipped with I GPU, 10 CPUs per task, and a memory allocation of 128GB.
Table 4: Detailed hyperparameter settings of the experiments.
Parameters
Settings
Parameters
Settings
Model
ResNet-50; V/6-
Batch-Size
32
Loss function
Pross-Entropy:Focal
Epoch
100
Optimizer
SGD
Num- Hun-Workers
4
Weight Decay (p)
0.0001
Image-Size (Train/Val)
256
Learning rate
0.001
Crop-Size (Train)
224
Momentum
0.9
Horiddoozzztta-FFligg (Train)
Yes
K group-level
[1. 5, 10]
Centre-Crov Crop (Val)
224
Order:Family
Dataset size
L/M/S
We conducted a set of 24 trials, each executed with 3 distinct seeds. These trials were carried out to tackle classification tasks involving Insect-Order and Aaptera-Family utilizing three dataset variations: Large, Medium, and Small. Our design encompassed the creation 4 distinct models, integrating two distinct loss functions (Cross-Entro) Erross-Entropy and Focal) and two different pretrained backbones (ResNet-50 and ViT-B/16).
The combination of these diverse components led to the calculation of average performance across range of seed values. Subsequently, the model that exhibited the highest average performance on the validation set was selected for further evaluation during inference, depicted in Table 5 It's worth noting that models employing the ViT-B/16 backbone and Cross-Entropy loss function demonstrated superior performance across most of the experiments the validation set, leading to their selection for inference using test data. For the Small and Medium datasets, the models underwent 100 epochs of training, while for the Large dataset, a lesser number of epochs were applied, convergence was achieved on the validation set.
We evaluate the performance of our classification models using top-K accuracy, which extracts the K K-predicted classes with the largest probabilities for each input sample and compares them with the ground-truth class label of the sample. If the ground-truth label is among the top-K predictions then the model counts as a correct classification. The total counts are then divided by the total number of input samples to yield average. We report test results of the best model from validation

---

performance for the micro, class-averaged macro top-K, K accuracy at K E [1,3, 5] well as micro and macro Fi Scores T
Figure 5) shows the per-class top-1 test accuracy for the Order and Family classification of the Large dataset. Accuracy is quite high, above 90%, for most classes, decreasing mainly for classes with little training data.
Table 5: Top-K accuracy and class-averaged macro top-K K accuracy based on the test sets of Insect- Order and Diptera-Family classification experiments using the Small, Medium and Large datasets.
Micro Top-K
Macro Top-K
FI-Scores
Classification
Dataset
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
Micro
Macro
Insect-Order
Small
97.86
99.35
99.66
85.01
91.68
99.23
97.86
85.84
Medium
99.14
99.77
99.88
85.58
97.68
98.22
99.14
87.36
Large
99.69
99.96
99.98
90.61
98.14
99.32
99.62
92.65
Diptera-Famile
Small 94.01
97.26
98.01
92.37
96.53
97.42
94.01
93.03
Medium
96.66
98.34
98.77
91.81
96.37
97.20
96.66
92.77
Large
97.59
98.85
99.23
91.20
95.86
96.72
97.59
91.45
Micro accuracy
Micro Accuracy for Large Insect Order Dataset
3.00
0.99
0.99
0.99
0.99
0.98
0.98
0.94
0.9a
0.92
96
0.58
0.85
1.00
Diptera
Hymeroptera
Coleantera
Hemiptera
Lapidoptera
Procodea
Thysanoptera
Trichoptera
Orthaptera
Blettodee
Neuroptera
Ephemer optera
Dermaptera
Archanagnathe
Plecaptera
Emblopttrr
Micro Accuracy for the Large Diptera Family Dataset
Micro accuraty
0.98
0
0 97
Phoridae 0.99
97
0:98
0.97
0.98
0.59
6.97
0.
0.94
Muscidae 9.96
Limon lidee 0.94
0.97
0.97
0.94
0.88
0.88
9.95
9.96
Sapsidae 6.57
Syrphidae 0.91
Bombyl iida liae 0.96
Fann idae 0.2
Kerop peroppatdee 0057
Ditomy/ilda iidae 0.76
Bibienidae 0.90
Richard iddee 0.93
Cu icidae 0.67
Simul idde lide 0.85
PPii lidee 0.87
Cacidomy iide
Chironomidas
Sciaridae
Ceralopogenides
Psychedidae
Dolichopedides
Sphaerocaridas
Drosophi idoa idda
Hybotidae
Mycetophi idae
Chloropidae
Scatopsidae
Milchiide
Ephydridas
Ag Agromyzidda
Tachinidaa
Empididae
Anthomylidaa iidd
Heleomyzidae
Lauxan lidae
Sarcophagidas
Uiiriide
Tephrildde Lidde
Clus idda
Pipuruuiddo iididdd
Figure 5: Per-class top-1 test accuracy of the Insect-Order and Diptera-Family classification experi-
ments of the Large dataset. The classes are listed in descending manner with respect to the number of test samples.
Conclusion
We have described set of six novel BIOSCAN datasets, which we conducted image- based classification experiments using the taxonomic annotations of the insects. Looking ahead, iBOL's ongoing efforts will lead to further advancements in several aspects. The rate of insect sample collection is already increasing, resulting dataset that is not only larger in terms the number of records but also more comprehensive, with additional taxa at lower taxonomic levels such as genera and species. Moreover, the dataset will expand encompass diverse life forms beyond insects. Thus, while the current dataset is already the largest publicly available insect image dataset, it represents just the beginning of what lies ahead.
10

---

Acknowledgement
We acknowledge the support of the Government of Canada's New Frontiers in Research Fund (NFRF), [NFRFT-2020-00073) and a NVIDIA Academic Grant. This research was enabled in part by support provided by Calcul Québec
ca) and the Digital Research Alliance of Canada (alliancecan, ca). Data collection was enabled by funds from the Walder Foundation, a New Frontiers in Research Fund (NFRF) Transformation grant, Canada Foundation for Innovation's ' (CFT) Major Science Initiatives (MSI) Fund and CFREF funds to the Food from Thought program at the University of Guelph. The authors also wish to acknowledge the team at the Centre for Biodiversity Genomics responsible for preparing, imaging, and sequencing specimens used for this study, as well as Utku Cicek for their help with the project.
References
[1] Barcode life data system. URL httpttsss//ooldyssteettttt org.
org/
[2] BIOSCAN, Jun 2022. URL httph:tp//// //io:///bolloo org programs bioscan/
[3] Antoine Affouard, Hervé Gočau, Pierre Bonnet, Jean -han-Shhrittophe Lombardo, and Alexis Joly. Pi@ntNet
app in the era of deep learning. In ICLR: International Conference Learning Representations. 2017.
[4] Badirli, Z Akata, G Mohler. Picard, and Dundar. Fine-Grained Zero-Shot learning with DNA as side information. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021). J. December 2021. [5] Mark Blaxter, Jenna Mann, Tom Chapman, Fran Thomas, Claire Whitton, Robin Floyd, and Eyualem Abehe. Defining operational taxonomic units using DNA barcode data. Philosophical Transactions of the Royal Society B: Riological Sciences, 360( y462y: 1935-1943 2005.
[6] Thomas WA Braukmann. Natalia Ivanova, Sean Prosser, Vasco Elbrecht, Dirk Steinke, Sujeevan Rat- nasingham, Jeremy R de Waard, Jayme E Sones, Evgeny V Zakharov, and Paul DN Hebert. Metabarcoding diverse arthropod mock community. Molecular ecology resources. 19(3).11--277. 2019.
[7] Andrew VZ Brower and Randall T Schuh. Biological systematics: principles and applications. Coreel University Press, 2021.
[8] Matcusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-259, 2018.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proc. of the European Conference on Computer Vision, pages 2292 Springer, 2020.
[10] Gerardo Ceballos, Paul R Ehrlich, Anthony D Barnosky, Andrés Garefa, Robert M Pringle, and Todd M Palmer. Accelerated modern human advances,
human-induced induced species losses: Entering the sixth mass extinction. Science 2015.
[11] Gerardo Ceballos, Paul R Ehrlich, and Peter H Raven. Vertebrates the brink as indicators of biological annihilation and the sixth mass extinction. Proceedings of the National Academy of Sciences, 117() 13596-13602, 2020.
[12] Solemane Couliably. Bernard Kamsu-Foguem, Dantouma Kamissoko, and Daouda Traore. Deep learning for precision agriculture: A bibliometric analysis. Intelligent Systems with Applications, page 200102,
2022.
[13] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balancee loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-9277, 2019.
[14] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:1810.04880 1810.04805, 2018
[15] Tiago Domingues, Tomás Brandão. and João C Ferreira. Machine learning for detection and prediction of
crop diseases and pests: A comprehensive survey, Agriculture, 12(9):1350 2022.
[16] Shifeng Dong, Jianming Du, Lin Jiao, Fenmei Wang, Kang Liu, Yue Teng. and Rujing Wang. Automatic crop pest detection oriented multiscale feature fusion approach. Insects, 13(6))554, 2022.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1120 11929, 2020. [18] Camille Garcin, Alexis Joly. Pierre Bonnet, Wean-Christophe Lombardo, Antoine Affouard, Mathias Chouet, Maximilien Servajean, Titouan Lorieul, and Joseph Salmon. Pl@mlNe1-300K: a plant image dataset with high label ambiguity and a long-tailed distribution. In NeurlPS 2021-35th Conference on Newral Information Processing Systems, 2021.
[19] Jacó C Gomes and Díbio L Borges. Insect pest image recognition: A few-shot machine learning approach including maturity stages classification. Agronomy, 12(8):1733, 2022.
[20] Graham CD Griffiths. On the foundations biological systematics. Acta biotheoretica, 23(3-4):85-131,
1974.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
11

---

[22] Paul DN Hebert, Alina Cywinska, Shelley L Ball, and Jeremy R DeWaard. Biological identifications through DNA barcodes. Proceedings of the Royal Society of London. Series B: Biological Sciences, 270
(3312):11-321, 2003.
[23] Paul DN Hebert, Sujeevan Ratnasingamm, Evgeny V Zakharov, Angela C Telfer, Valerie Levesque-Bcaddin, Megan A Milton, Stephanic Pedersen, Paul Jannetta, and Jeremy R DeWaard. Counting animal species
with DNA barcodes: Canadian insects. Philosophical Transactions of the Royal Society B: Biological
Sciences, 37101702)201550333, 2016.
[24] Marwah A Helaly, Sherine Rady, and Mostafa M Aref. BERT contextual embeddings for taxonomic classification of bacterial DNA sequences. Expert Systems with Applications, 108:111972. 2022.
[25] Toke T Høye, Johanna Ärje. Kim Bjerge, Oskar LP Hansen, Alexandros Iosifidis. Florian Leese, Hjalte MR Mann, Kristian Meissner, Claus Melvad, and Jenni Raitoharju. Deep learning and computer vision will transform entomology. Proceedings of the National Academy of Sciences, 1180))e2002545117 2021. iNaturalist 2018 competition datasct. iNaturalist 2018 competition dataset. hhttps///githr com/
[26]
sasipedia/inatt comp/ trre/masterr 2018 2018.
[27] Nathalic Japkowiez and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data
analysis, 6(5):429-449, 2002.
[28]
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. DNABERT: pre-trained bidirectional encoder
representations from transformers model for DNA-language in genome. Bioinformatics, 37(15):21222122)
2021.
[29]
Amlan Kar, Seung Wook Kim, Marko Boben, Jun Gao, Tianxing Li, Huan Ling, Zian Wang. and Sanja
Fidler. Toronto annotation suite. https::/ http://aaddmsssssm cs toronto........... www.edduuaareeeeeeeeeeee /orrat 2021.
[30]
Bartosz Krawezyk. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5(4):221-232, 2016.
[31] Neeraj Kumar, Peter N Belhumeur, Arijit Biswas, David W Jacobs, W John Kress, Ida C Lopez, and João VB Soares. Leafsnap: A computer vision system for automatic plant species identification. In Computer Vision-ECCV 2012: 12th Earopean Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Purt II 12, pages 502-516. Springer, 2012.
[32] Guillaume Lecointre and Hervé Le Guyader. The Tree of Life: A Phylogenetic Classification. Socicty of Systematic Zoology, 2007.
[33] Wenyong Li, Tengfei Zheng, Zhankui Yang. Ming Li, Chuanheng Sun, and Xinting Yang. Classification and detection insects from field images using deep learning for smart pest management: A systematic review. Ecological Informatics, 66:101460, 2021.
[34] Zhiyong Li, Xueqin Jiang, Xinyu Jia, Xuliang Duan, Yuchao Wang. and Jiong Mu. Classification method of significant rice pests based on deep learning. Agronomy, 12(9):2096, 2022.
[35] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proc. of the European Conference on Computer Vision, pages 740-755. Springer, 2014.
[36] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEFF international conference on computer vision, pages 2980-2988, 2017.
[37] Liu Liu, Rujing Wang, Chengjun Xie, Rui Li, Fangyuan Wang, and Long Qi. A global activated feature pyramid network for tiny pest detection in the wild. Machine Vision and Applications. 33,5):76, 2022.
[38] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arxiv:1711.05101, 2017.
[39] Maxime Martineau, Donatello Conte, Romain Raveaux, Ingrid Arnult, Damien Munier, and Gilles
Venturini. A survey on image-based insect classification. Pattern Recognition, 65:273-288 2017.
[40] Zhonghua Miao, Guodong Huang, Nan Li, Teng Sun, and Yutao Wei. A review of plant disease and insect pest detection based on deep learning. In Proceedings of 2022 Chinese Intelligent Systems Conference: Volume II. pages 103-118, 18. Springer, 2022.
[41] Florian Mock. Fleming Kretschmer, Anton Kriese, Sebastian Böcker, and Manja Marz. Taxonomic classification of DNA sequences beyond sequence similarity using deep neural networks. Proceedings of (1V355))e11263611
the National Academy of Sciences,
2022.
[42] Camilo Mora, Derek P Tittensor, Sina Adl, Alastair GB Simpson, and Boris Worm. How many species are there on earth and in the occan? PLoS biology, 9(8xe10011227 127, 2011.
[43] Craig Moritz and Carla Cicero. DNA barcoding: promise and pitfalls. PLoS biology, 2(10)cc354, 2004,
[44] Prakash M Nadkarni, Lucila Ohno-Machadoo and Wendy W Chapman. Natural language processing: an introduction. Journal of the American Medical Informatics Association, 18(5):544-551, 201
[45] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages JEEE, 2008
[46] Jan Pawlowski, Mary Kelly-Quina Florian Altermatt, Laure Apothéloz- Apothéloz-Perret Perret-Gentil,, Pedro Beja, Angela Boggero, Angel Borja, Agnès Bouchez, Tristan Cordier, Isabelle Domaizon, et al. The future of biotic indices in the ecogenomic era: Integrating (e) DNA metaharcoding in biological assessment of aquatic ecosystems. Science of the Total Environment, 637:1295-1310,, 2018
12

---

[47] Sujeevan Ratnasingham and Paul DN Hebert. A DNA-based registry for all animal species: the barcode index number (BIN) system. PloS one, 8(7):666213, 2013.
[48] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658 6636666,222 2019.
[49] Krista M Ruppert, Richard Kline, and Md Saydur Rahman. Past, present, and future perspectives of environmental DNA (eDNA) metabarcodinn: A systematic review in methods, monitoring, and applications of global eDNA. Global Ecology and Conservation, 17:e00547, 2019.
[50] Carlos N Silla and Alex A Freitas. A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22:31-75 2011.
[51] Fábio Amaral Godoy da Silveira, Everton Castelão Tetila, Gilberto Astolfi, Anderson Bessa da Costa, and Willian Paraguassu Amorim. Performance analysis of yolov3 for real-time detection of pests in soybeans. In Intelligent Systems: 10th Brazilian Conference, BRACIS 2021, Virtual Event, November 29- December
3. 2021, Proceedings, Part II, pages 265-279. Springer, 2021.
[52] Robert R Sokal, Peter Henry Andrews Sneath, et al. Principles of numerical taxonomy. Principles of mamerical taxonomy. 1963.
[53] Thorsten Stoeck, Larissa Frühe, Dominik Forster, Tristan Cordier, Catarina IM Martins, and Jan Pawlowski. Environmental DNA metabarcoding of benthic bacterial communities indicates the benthic footprint of salmon aquaculture. Marine Pollution Bulletin, 1 149, 2018,
[54] Nigel E Stork et al. How many species of insects and other terrestrial arthropods there earth. Annual review of entomology, 63(1):31 45, 2018.
[55] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haher, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pages 595-604, 2015.
[56] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam. Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
[57] Grant Van Horn, Eljjah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12884-12893, 2021.
[58] Kaili Wang, Keyu Chen, Huiyu Du, Shuang Liu, Jingwen Xu, Junfang Zhao, Houlin Chen, Yujun Liu, and Yang Liu. New image dataset and new negative sample judgment method for crop pest recognition based deep learning models. Ecological Informatics, 69: 19260, 2022.
[59] Qi-Jin Wang. Sheng- Zhang, Shi-Feng Dong, Guang- Cai Zhang, Jin Yang, Rui Li, and Hong- Qiang Wang. Pest24: A large-scall scale very small object data set of agricultural pests for multi-target detection. Computers and Flectronics in Agriculture, 175:105585, 2020.
[60] Jan D Wegner, Steven Branson, David Hall, Konrad Schindler, and Pietro Perona, Cataloging public ohjects using acrial and street-level images-urban trees. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition. pages 6014 4- 4-224,, 2016.
[61] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006. 03677, 2020.
[62] Xiaoping Wu, Chi Zhan, Kun Lai, Ming-Ming Cheng, and Jufeng Yang. IP102: large-scale benchmark datasct for insect pest recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8787-8796, 2019.
[63] Shuli Xing and Hyo Jong Lee. Crop pests and diseases recognition using DANet with TLDP. Computers and Electronicy in Agriculure, 199:107144 144, 2022.
13

---

Appendix
A1
Data collection and organization
The BIOSCAN-IM Insect dataset consists of insect RGB images and metadata file containing taxonomic annotation, DNA barcode sequences, and an assigned Barcode Index Number (BIN). In the following sections, describe the resources available within the dataset.
A1.1
RGB images
To facilitate different levels of visual processing we created packages of color images of varying sizes. These packages are as follows:
Original full size RGB images. The original images are converted to JPEG image format. These images each have a resolution of 2880x2160, and they are typically around 5 MB in size, however some images are smaller at 600-800 kB. The package is structured as 113 zip files, each of which contains 10,000 images except the last (zip file 1 113 contains 8, 8.131 original full size images). The total size of this package is 2.5 TB. All 113 zip files are stored within the BIOSCAN project space in and the _all__aar>>>>>>> where n is the partition ID and is in the
GoogleDrive as described in Section A2 inside folder named BIOSCAN Lio_oogiiallinnnn zip files named as bioscan images_origingg range of 113.
Cropped RGB images. The images in this package are cropped by our cropping tool as described in the main body of the paper and available in the accompanying BIOSCAN- code repository. The package is structured into six zip files where each file contains 20 partitions (20x10,00) 10,000 files), except the last zip file which contains 13 partitions. The total size of this package is 151 GB, All six zip files are stored within the BIOSCAN project space in GoogleDrive as de- scribed in Section [A2 a folder named BIOSCAN BIosceN_cropped Laaeadiiseesseeee and the zip files named as bioscan aroscan_image__.. loogge_crropeeddee part<<-n>> where m-n indicate the start and end partition ID, in the range of 1-113.
Resized original RGB images. This package is available in two archive formats (zip and HDF5). The package contains downscaled versions of the original images, requiring reduced storage space. The resizing was done such as reduce the smaller dimension of image 256 pixels (and the longer side scaled to preserve the aspect ratio of the original image) and then saved in JPEG format. The total size of these packages are approximately 27 GB, and they are named ariginal_256.hdee
iiginnll_26..p0 and
Resized cropped RGB images. This package is also available in two archive formats (zip and HDF5). The package contains resized versions of the cropped images. The resizing was done such as to reduce the smaller dimension of image to 256 pixels (and the longer side scaled preserve the aspect ratio of the cropped image) and then saved in JPEG format. The total size of these packages are approximately GB, and they are named as cropped rropped_26..iip and cropped_256..n
A1.2 Metadata file
To enhance the metadata of our published dataset, we incorporated structured metadata following Web standards. The metadata file for our dataset is named BIOSCAN, Insect_ Dataset_ tatetetteetttttttteteee We created two versions of this file: one data frame in TSV format (.tsv) and the other in JSON-LD format (.jsonld). The JSON-LD file was validated using the Google Inspection Tool.
DUSSCANnnneectttt Inseet_Dataset,
The metadata file is a table with 22 columns, which contain content as described below. Note that if a sample was not labelled by taxonomist, for each taxonomy ranking group (columns 4-13) the corresponding annotation is listed as not classifiid instead. Similarly, a sample has no association with experiment shown by columns 16 21, then the sample's 's role is shown no split.
1. sampleid: An identifier given by the collector.
2. processid: A unique number assigned by BOLD each record.
3. uri:
Barcode Index Number (BIN).
4. name:
Taxonomy ranking classification label.
5. phylum:
Taxonomy ranking classification label.
14

---

6. class: Taxonomy ranking classification label.
7. order:
Taxonomy ranking classification label.
8. family:
Taxonomy ranking classification label.
9. subfamily: Taxonomy ranking classification label.
10. tribe: Taxonomy ranking classification label.
11. genus: Taxonomy ranking classification label.
12. species: Taxonomy ranking classification label.
13. subspecies: Taxonomy ranking classification label.
14. mucraw: Nucleotide barcode sequence.
15. image file: Image file name stored in structured packages.
16. surge_diptera diptera_family: Image association with the training, validation, and test split of experiment-1.
medium diptera family: Image association with the training, validation, and test split of experiment-2.
18. small diptera family: Image association with the training, validation, and test split of experiment-3.
19. large insect order: Image association with the training, validation, and test split of experiment-4.
20. medium insect order: Image association with the training, validation, and test split of experiment-5.
21. small insect order: Image association with the training, validation, and test split of experiment-6.
22. chunk number: A unique ID to locate the corresponding images within the dataset pack- ages.
A2 Informational content
The link to access the dataset and its metadata is sstt::://bioddieessttyyennn http://mmincc...eeteeeeemeeeeeee net/iM insects/
A3
Ethics and responsible use
The BIOSCAN project started by the International Barcode of Life (iBOL) Consortium, has collected a large dataset of hand-labelled images of insects. Each record is taxonomically classified by human experts, and accompanied by genetic information.
The publication of BIOSCAN-IM Insect dataset a common effort made by researchers from the University of Waterloo, Simon Fraser University, Aalborg University, Dalhousie University and the University of Guelph with support from the Vector Institute for Artificial Intelligence, Alberta Machine Intelligence Institute, Pioneer Centre for AI, and the Centre for Biodiversity Genomics.
The availability of the BIOSCAN- IM Insect dataset presents an immense opportunity for scientific advancement and understand in of insect biodiversity. However, it is important to emphasize the ethical and responsible use of this data.
First and foremost, researchers and institutions must prioritize the protection of individuals' privacy and adhere data protection regulations and guidelines. To our knowledge, there is no personal or identifiable information in the dataset. However, any such information associated with the dataset should be treated with utmost care and reported to the authors.
Furthermore, the researchers and organizations who utilize the BIOSCAN-IM Insect dataset should ensure transparency in their methodologies and practices. This includes clearly stating the purpose of their research, obtaining informed consent when applicable, and maintaining integrity in the interpretation and reporting of the results.
The responsible use the BIOSCAN- Insect dataset entails promoting open collaboration and sharing of knowledge within the scientific community. Researchers should foster environment
15

---

that encourages exchange of ideas, methodologies, and findings, while giving credit to the original dataset creators. is essential to acknowledge and respect the contributions of the human experts who hand-labelled the images by taxonomically classifying specimens. Proper attribution and recognition should be given to these individuals, as their expertise and efforts are instrumental in the creation and accuracy of the dataset.
A4 Dataset availability and maintenance
The BIOSCAN-IM Insect dataset and all its content described in the previous sections are available on GoogleDrive folder named 1M Image project. To access the BIOSCAN- 1M Insect dataset, please visit https: / wedieeeityeeeooiiiiic.
wwt.enn/...... 1M_iiMMntts///// We' ve published a code repository for dataset manipulation. including tasks like downloading dataset packages, image and metadata reading, image cropping, dataset subsampling, partitioning into train, validation, and test sets, and running the classification experiments presented in the BIOSCAN- Insect paper. To access the BIOSCAN-IM code repository, please visit http::ttps:///t thub. com/ zahrag BBISCCNOOOCANN
com//aaraggBIIOCA
A5
Licensing
Table A shows the copyright associations related to the BIOSCAN-IM Insect dataset with the corresponding names and contact information.
Table A1: Copyright associations related to the BIOSCAN-IM Insect dataset
Copyright Associations
Name & Contact
Image Photographer
CBG Robotic Imager
Copyright Holder
CBG Photography Group
Copyright Institution
Centre for Biodiversity Genomies (email:CBGImaging gmail.com)
Copyright License
Creative Commons-Attribution
Attribution Non-Commercial Share-Alike BY-NC-SA 4.0) collectionsBIO gmail.com
Copyright Contact
Copyright Year
2021
A.6
Experiment details and results
A6.1
Backbone models
We utilized distinct pretrained backbone models for our experiments with the BIOSCAN-IM-
Insect dataset. A comprehensive comparison between these models is presented in this section and in
Table A2
ResNet-50 21 is a deep convolutional neural network, which includes residual blocks that allow for the training of very deep networks without falling into the vanishing gradient problem. ViT-Base- Patch16-224 [7] signifies that the ViT model is designed to process images with a resolution of 224x224 pixels. Each image is divided into smaller patches of size 16x16 pixels, which are then fed into the transformer layers. Each transformer layer includes multi-head self-attention mechanisms and feed-forward neural networks.
Table A2: A comparison between the two pretrained backbone models used in our experiments: ResNet-50 and the ViT-Base-Path 1--24 CNN and FC denote Convolutional Neural Network, and Fully Connected layers, respectively.
Features
ResNet-50
VIT-B/16
Layers
50
12
Based Networks
CNN, Pooling and FC
Transformer
Number parameters
25.6 M
86
Overall ResNet-50 has deeper architecture with more layers than VIT-B/16. This depth can enable to learn hierarchical features in the data, while ViT's strength lies in capturing relationships between patches by applying self-attention mechanisms, which enables to capture long-range dependencies in images thus making it suitable for both local and global context understanding. Moreover, due to
16

---

its transformer architecture, ViT can parallelize training more effectively, which can result in faster convergence times despite its higher number of parameters.
A6.2
Validation results
Table A3 shows the performance of all 24 experiments conducted with 3 different seeds using the validation set. According to the validation results, ViT-B/16 with the Cross Cross-Entropy loss function consistently outperforms other models.
Comparing Focal loss to Cross-Entropy, we found that Cross-Entropy produced slightly better results. This could be due to insufficient fine-tuning of Focal loss hyperparameters (alpha and gamma). Furthermore, addressing class imbalance could involve selectively oversampling the less frequent classes during training. This strategy boosts their representation in the training process. For Focal loss, limited exposure rarer classes might hinder the effectiveness of the re- re-weighting mechanism.
The presented results table A3 depict the mean accuracy across various seeds, accompanied by the standard deviation from the average values of each model. The outcomes reveal notable consistency the performance of almost all models.
The six models highlighted bold in Table A3 are used for inference in the test experiments and to report the final results. Pretrained classification checkpoints of these six models, which achieved the best validation accuracy, are available in the GoogleDrive project folder under the directory named BIOSCAN 1M cesset_cheekpoint.
Table A3: The table displays Micro-Average-TTgn and Macro-Average-Tne validation accuracy across 24 experiments conducted using distinct seeds. These experiments encompass varying data sizes (Small, Medium, Large), loss functions (Cross-Entropp) Focal), and pretrained backbone models (ResNet-50, ViT-B/16). The experiments utilize consistent hyperparameters and extend to both insect-Order and Diptera-Famin classification levels.
Micro-Top-1
Macro-Top- Top-1
Dataset
Backbone
Loss
Insect-Order
Diptera-Family
Insect-Order
Diptera-Family
Large
ResNet-50
CE
99.65.10.10
97.30.10.02
86.26.10.30
89.98.10.27
Focal
99.62:10.06
97.15=+0.00
84.66=:0.21
89.42=0.58
ViT-B/16
CE
99.58+0.21
97.6711.011
87.36 1.20
91.47:1.31
Focal
99.52++0.27
97.58++0.02
85.80- 1.75
91.54+
Medium
ResNet-50
CE
98.98:10.04
96.2410.05
87.30
91.24410.33
Focal
98.85+00.04
95.92+00.4
86.61+0.51
90.64+-0.22
ViT-B/16
CE
99.14++0.44
96.74+0.06
88.40-+1.17
92.83+0.16
Focal 99.1110.004
96.5510.02
86.7511146
92.23
Small
ResNet-50
CE
97.79+0.08
93.23+:0.4
87.37
91.43+0.36
Focal
97.62+0000
92.57+0.07
86.55=0060
90.68=:0.0
ViT-B/16
CE
98.34-10.10
94.46
88.74411.16
92.93-1.33
Focal
98.26-10.03
94.42+0.00
88.61+0..99
92.92+0.116
A6.3
Confusion Matrix
For in-depth analysis of the performance of models trained under various configurations, provide detailed Confusion Matrices for the classification experiments conducted at the order and family levels. These experiments were carried out using the model employing ViT-B/16 and the Cooss-Entropy loss function. The evaluation was performed on the test set of the Large dataset. You can refer to Figures AT and for visual representation of the respective Confusion Matrices.
A6.4 Qualitative analysis
In this section, we provide qualitative analysis of the performance results from the order classification experiment the Small dataset. We aim shed light the misclassificatios made by model by visually examining some of the misclassified images.
Surprisingly, roughly 57% of the misclassifications in order-level classification experiments on the Small dataset, using 10,000 test samples, can be traced back to low-quality insect images. This evident when examining the examples shown in Figure A3 where image quality hampers accurate
17

---

Table A4: The table presents the Micro-Fl-Score and Macro-F1-Scone of trained models, evaluated on the validation set, and then averaged across different seeds.
Micro-Fl-Score
Macro-F Feero-F---core
Dataset
Backbone
Loss Fn
Insect-Order
Diptera-Family
Insect-Order
Diptera-Family
Large
ResNet-50
CE
99.67.10.07
97.44.10.03
87.50
90.73.10.23
Focal
99.63=0.066
97.28+0.01
84.66=t0.87
90.22=:026
ViT-B/16
CE
99.68=0.06
97.68+0.01
87.94=1..59
92.01=0.12
Focal
99.62:t0.14
97.58++0.01
86.98+42.06
91.91
Medium
ResNet-50
CE
99.0010.04
96.26110.66
87.43=1.02
92.61
Focal
08.05
95.9810.005
86.77+1.21
91.77=0.5
ViT-B/16
CE
99.14+00.04
96.75+0.04
89.33+1.21
93.58+00.8
Focal
99. 12.003
96.56+0001
87.52+1.01
93.20++0.1
Small
ResNet-50
CE
97.8410.11
93.27110335
87.8910.777
92.1010.51
Focal
97.63++0.04
92.78+0.06
87.52+0.62
91.37=0.12
ViT-B/16
CE
98.31+0.10
94.5440.16
88.92:40.74
93.56+0.23
Focal
98.28-10.03
94.42440.06
88.36:10.28
93.48
Experiment.large, leapc_innssneeeee order, Metric: micro_test hatp_eett___cc__eee accuracc
Diptera
0.00
0.00
Hymenoptera
0.01 0.99 0.00
0.00
0.00
0.00 0.00 0.00
0.00
0.00 0.00 0.00 0.00 0.00 0.00
0.00
Coleoptera
0.00 0.00 0.99
0.00
0.00
0.00
0.00
0.00 0.00 0.00
0.00
0.00
0.00
0.00 0.00 0.00
Hemiptera
0.00
0.00
Lepidoptera
0.01 0.00 0.00
0.00
0.99
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Psocodea
0.01 0.00 0.00
0.00
0.00
0.98
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Thysanoptera
0.01 0.00 0.00
0.00
0.00
0.98
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Class
Trichoptera
Orthoptera
0.02 0.00 0.00
0.04
0.00
0.00
0.00
0.00
0.93
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Blattodea
0.02 0.00 0.03 0.01 0.01 0.00 0.00
0.01
0.01
0.92
0.00
0.01
0.01
0.00
0.00
0.00
Neuroptera
Ephemeroptera
0.11 0.00 0.05
0.05
0.05 0.05 0.00
0.00
0.11
0.00
0.00
0.58
0.00
0.00
0.00
0.00
Dermaptera
0.00 0.00 0.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00
0.00
0.85
0.00
0.00
0.00
Archaeognatha
0.00 0.00
0.00 0.00
0.08
0.00
0.00
0:85
0.00
Pleroptera
0.00 0.80 0.00
0.00 0.00 0.00 0.00 0.00 0.00
0.00
0.00
0.00
0.67
Embleptera
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00
0.00
0.00 0.00
0.00
0.00
0.00
1.00
Diptera
Hymenopters
Coleoptera
Hemiptara
lapideptara
Psocodes
Thysenoptera
Tichoptera
Orthoptera
Blattodea
Neuroptera
Demmaptere
Acchaeagnathi
Plecoptera
Embloptera
Predicted Class
Figure A1: The Confusion Matrix displays the per-class predictions of the order level classification using the Large dataset of the BIOSCAN-IM Insect dataset. The test evaluation is performed on the best model achieved from validation performance results presented in Table A3
classification. A similar analysis revealed that approximately 45% of the misclassifications in order-level experiments with the Large dataset, using 225,660 test samples, were also attributed to low-quality insect images.
18

---

Experiment:large, diptera family, Metric: micro test accuracy
Cecidomyliiae
Chironomidae
Phoridae
Sciaridae
Ceratopogonidae
Psychodidae
01:000000
Delichopodidae
Sphaeroceridae
Dresophilidae
Hybotidae
Mycetophilidae
Muscidae
11 00
Limoniidae
Chloropidae
Scatopsidae
27 0000
Milichhidae
03300E00
0.6
Ephydridae
True Class
Agromyzidae
Tachinidae
Empididae
Anthomylidae
Heleomyzidae
447006000
00
Lauxanlidae
15360
Sarcophagidae
0.4
Ulidiidae
Tephritddee
Stratiomyidae
Clusiidae
Ripunculidae
Sepsidae
Syrphidae
Bombyliidae
4::00600 0E80960063004000000 000000000000000000
Fannlidae
Keroplatidae
Ditomylidae
Bibionidae
2220000000
Richardlidae
Culicidae
Simullidae
422006000
Pailidae
0.0
Cecidomyyidos
Chironomidae
phoriti
Sciaridas
Ceratopogonide
Psyc dae
dare
polichopod dae
Schaeroceridae
prosophilidde
Hy Rybotddee
prrettohilidde
Musc dae
Scatopsidas
Sahydring
Tachin acchinibbe
Empld Empididee dee
Anthomyildda dini
Helcomyzidaa
Lauxania dae
Ulididas
apphitdde
Stratiomylie aaliomyddd
Clusiidos
Pipunculi dranculldae
Semphide
Syrphice
Bambyllidde
fanniidae
Keropl popllation
Ditomyitiee
Bibiomidae
Culieidan
simullidae
Pailidae
Predicted Class
Figure A2: The Confusion Matrix displays the per-class predictions of the family level classification using the Large dataset of the BIOSCAN-IM Insect dataset. The test evaluation is performed on the best model achieved from validation performance results presented in Table A3
Another observation shows that large proportion of misclassifications are the insects belonging to different orders that are all incorrectly classified as one of the dominant classes of our Small dataset. As an example there are 16.2% of the misclassifications in order-level classification experiments on the Small dataset were insects belonging to different orders are all incorrectly classified as Diptera (flies or mosquitoes), which is the dominant class. This observation, illustrated in Figure [4] highlights specific instances where the model struggles to differentiate between various orders and tends to favour Diptera as the predicted classification.
By examining these qualitative analyses, we gain insights into the challenges faced by our model in correctly classifying insect orders, especially when dealing with low-quality images and distinguishing between similar orders when these orders have low number of training samples.
Our classification experiments have an important application in data cleaning. By identifying low- effectively detect and remove them from the
quality images that have been misclassified, dataset. This process plays a crucial role in enhancing the overall quality and reliability of the data, it ensures that only high-quality images of insects retained.
Furthermore, our classification experiments also enable us to validate the taxonomic classifications performed by human experts. By examining instances of false predictions, can investigate
19

---

natural means of expressing uncertainty, and whether a sample has been incorrectly annotated, providing valuable insights into the accuracy of the taxonomic classification process.
A6.5
Discussion
A6.5.1
Dataset: Generation, Curation and Growth
The BIOSCAN project is currently in its initial stages, with its primary goal being the facilitation of global biodiversity assessment. In this section we clarify certain aspects and procedures accomplished the generation of the BIOSCAN-IM Insect dataset.
All samples of the BIOSCAN- IM Insect dataset were processed at one facility using the same workflows and imaging equipment. This should exclude all potential biases with respect to data collection.
Regarding dataset labeling procedure, the order-level classification is conducted by taxonomists and entomologists primarily relying on morphology rather than barcode matching. For family and finer-grained classifications, combination of approaches may be employed, often supported by BIN assignment.
The annotators responsible for labeling the data were personnel affiliated with the Centre for Biodi- versity Genomics, including technicians and research scientists, all engaged in full-time roles the institute and receiving equitable compensation. A diverse team of approximately 15 to 20 individuals participated in providing labels for the dataset. These labels were derived from the images, forming the foundation for establishing higher-level taxonomies such as order and family. Additionally, the annotators conducted visual examinations of specimens for finer- amer-grained taxonomic ranks, utilizing primary literature reference where feasible. Taxonomic experts (human professionals) engaged in the barcoding process have varying level of involvement, which tends to increase when the placement of specimen is more contentious. Additionally, these experts are responsible for describing species, a task not handled by the machine. They also supply the reference ID that enables us to establish matches.
Several factors contribute to why most samples of the dataset are classified only to the family and finer- sver-graieed classes are not provided, with the primary one being the time constraints associated with completing the assignment. The complexity of the task arises from the fact that relying solely on barcode is often insufficient due to potential ambiguities, as discussed earlier. Each sample's labeling requires verification through visual inspection (images), or in some cases, examination of the original specimen, before proceeding with further classification. This process is not easily scalable, prompting the adoption of BINs as a species proxy.
However, the process utilized to expand the dataset remains consistent with the methodology em- ployed for the BIOSCAN-IM Insect dataset. The Dataset will be retrained at regular intervals and older versions archive stored on Zenodo and date stamped. Similarly we will use GitHub's 's releases mechanism to version the accompanying code.
A6.5.2 Application: Model and Tasks
In this article the baseline problems and methods we explored were chosen to be simple and accessible and result, limited. They are not the focus of the paper as our primary focus is to release the dataset and showcase its inherent potential. We expect future works will use the dataset for interesting problems such as hierarchical classification, zero-shot classification, set-valued classification and methods that improve performance in the fine-grained and long-tailed label regime.
We believe that the most promising methods will be hierarchical classifiers that yield uncertainty estimates over multiple taxonomie levels. Improving performance on minority classes and reliably delincating novel operational taxonomic units is also important. To get there, believe the most promising areas of investigation from the ML side will be remi-parametric methods that use reference libraries at test time, set-valued classification zero-shot classification.
The utilization of the IM Insect dataset in conjunction with other large biological datasets from variuos domains becomes feasible by harnessing the preprocessing module proposed this paper. By employing tools like our cropping tool and applying machine learning techniques for domain adapaationngeneraliaation,,, one can capitalize on the capabilities of pre-trained model on
20

---

(1) Original
(2) Original
(3) Original
(4) Original
(1) Cropped
(2) Cropped
(3) Cropped
(4) Cropped
(5) Original
(6) Original
(7) Original
(8) Original
(5) Cropped
(6) Cropped
(7) Cropped
(8) Cropped
9)) Original
(10) Original
(11) Original
(12) Original
(9) Cropped
(10) Cropped
(11) Cropped
(12) Cropped
Figure A3: Examples of misclassifications caused by low quality images photographed from insects.
21

---

Ephemeroptera (original)
Psocodea (original)
Psocodea (original)
Hemiptera (original)
Hemiptera (original)
Diptera (cropped)
Diptera (cropped)
Diptera (cropped)
Diptera (cropped)
Diptera (cropped)
Figure A4: Examples misclassified the dominant class Diptera (flies).
BIOSCAN-IM Insect images to effectively tackle classification challenges in out-of-distribution scenarios.
Overall, we believe that the unique annotation and metadata including the DNA barcodes will prompt interesting multimodal strategies. We intend enhance our approach by incorporating DNA barcode sequences and utilizing Barcode Index Numbers (BINs). This strategic direction aims to effectively tackle the limitations associated with the current taxonomic labels of the images. Notably, the utilization of BINs holds promise as each image is inherently associated with a distinct and unique BIN.
A7 Preprocessing: Cropping tool
Our observations showed significant improvement in processing time when we used cropped images rather than original ones. However, cropping is challenging problem since insect images have varying shapes, sizes, colors which is also shown in Figure AS The illumination and background color and surface are not the same across the original images.
Furthermore, there are cases in the original images that the insect is photographed in pieces and in such cases the cropping is quite challenging especially when the insect is small, and its less discriminative body parts like legs are distant from the main body so these pieces could be cropped instead.
To address these issues more effectively, we have developed a tool based on the DETR model for automatic identification and cropping of the main insects in images. The primary objective of this tool facilitate data storage and subsequent research, such as neural network training. The tool uses the DETR model accurately locate the main insects in images and crop accordingly. By removing irrelevant background information, the tool optimizes storage space and reduces the time spent on data management. Additionally, the cropped images can be effectively used for tasks such image classification through neural network training, leading to improved performance the following image classification task. Our crop tool checkpoint is available in the GoogleDrive project folder under the directory named BIOSCAN 1M Insect checkpoints.
A7.1
Approach
The cropping tool consists of first detecting a tight bounding box for the insect in the image using an object detector and then cropping the image by extending the bounding box. We show an overview of the cropping tool in Figure A6 To accurately locate the insect in the image, we chose the DETR [9] model which has excellent performance in the task object detection and the corresponding pre- trained ResNet-50 (21 the feature extractor. At the beginning, the CNN-based feature extractor extracts a set of image features that are fed into a transformer-basee encoder-detector. The detector takes set of learned positional embeddings as object queries and uses them to attend to the encoder
22

---

Figure A5: Examples of images used to adapt cropping tool. We include variations of insects' size, color, position and shape.
F'eetuaa rodeecia: PPimieed
Paediraned Delt for otject detection
Image bocrr
Cutput
Rounding
For traning the pood
Laben The onty thee
Figure A6: Our DETR g based cropping tool takes an input image, extracts features using a ResNet50 21 backbone, and extracts tight fitting bounding box for the insect (see red box). We then extend the bounding box (see blue box) to obtain the final cropped image. We use a DETR model pretrained MSCOCO [35]. To fine-tune the DETR model, we annotate small set of insect images with their segmentation mask.
outputs. Each of the output decoder embeddings is then passed to a shared FFN which will predict whether there "no object" or detected object with its class and bounding box. Each bounding box is parameterized as (cz, w, h) where (cz, cy) is the center of the bounding box, and (w, h) is the width and height of the box, all normalized to
The DETR network trained by optimizing bipartite set loss that matches detected boxes with the ground-truth boxes using the Hungarian algorithm minimize the overall matching loss between the matched pairs. The pairwise matching loss is a combination of the classification loss and a box regression loss (the bounding box loss is included only when the detected box matches ground truth box that corresponds to an object, and is weighted combination of GIOU 48 and L1 loss between the bounding box parameters). In our case, we have only one object class ("insectt") so the classification reduces to a binary classification between "insect" and "no object".
Note that other than the ground-truth bounding box, for training the DETR model of the cropping tool, the pixel mask of the insect in the image is also required for the training. This pixel mask is not needed during the inference phase.
Training details. We start with a DETR model pretrained on MSCOCO 35 and fine-tune on our dataset. We use the AdamW 38 optimizer with learning rate of 0.0001, weight decay of 0.0001 and batch size We train for 10 epochs. On RTX 2080 Ti with workers, for 1,000 images, training takes 1.5 minutes epoch and a total of 15 minutes for 10 epochs.
23

---

original
annotated
Figure A7: Typical instances annotated IW (left two columns) and IP (right two columns) images. To obtain accurate bounding box in reasonable annotation time, we focused drawing the external outline the main insect only excluding the small spaces between its legs. Small parts of the insect that are far away from the main body (e.g. the small leg in the first image) are also not included.
The original DETR is trained with images resized fit within 800 x1,333 tensor. We match that and resize our image (preserving the aspect ratio) so that the shortest side is less than 800 and the longest side is less than 1333. No data argumentation is applied during training.
Cropping. the cropping phase, With the predicted bounding box (the red bounding box in Figure A6). we can choose to enlarge it using certain method to include more details or meet specific image aspect ratio requirements. By default, we will choose 0.4 times the longest edge as the target and extend this size in both height and width to produce the final cropping bounding box (the blue bounding box in Figure A6).
To crop the image, we run our fine-tuned DETR model on the input image to identify the tight bounding box around the insect. We assume that each image contains one insect of interest, and during cropping, we take the predicted bounding box with the highest probability that is higher than 0.5. Before cropping, we extend the predicted bounding box by fixed ratio R 1.4 of the size of the tight bounding box. We extend the height and width by the same number of pixels by computing the extended size as: ExtendSize = (R 1) x max(width height).
If the bounding box is at the edge of the original image, we pad the image by adding pixels of maximum intensity match the white background. In this way, even the predicted bounding box does not encompass all the details of the insect, we can still include the entire insect in the cropped image. Furthermore, this maintains a more square aspect ratio, which facilitates downstream tasks such as image classification.
Runtime. The cropping tool can be run in CPU or GPU mode. On Linux machine with 16 cores and running 4 workers, using CPU only, 10k images can be cropped in 2 hours and 40 minutes (images loaded and written to local SSD). Using RTX 2080 Ti GPU, 10K images can be cropped in 30 minutes on the same machine.
A7.2 Data
We develop our tool on two sets of images of insects that are pinned (SSECCTS-PINND) and insects in wells (INL)CS-WELL). Using the Toronto Annotation Suite (TORAS) (29, we annotate each with their segmentation mask. For each set, we annotated a large (1,000 images) and small (100-150 images) training set and another small set for evaluation. The annotation was done by three volunteers and took total of hours for 1,000 images. The two sets of images are described below (see Figures[A and AS for example images and annotated masks):
INSECTS-PINNED (IP). The insect is pinned in these images (or has pin near it) with a fairly clean white background. The images are taken by a Digital SLR camera (Canon) mounted on motor-drive positioning system (OpenBuilds ACRO) equipped with stepper motors and motion control system. Pinned specimens are arrayed in sets of (8 x12 array) in a large enough distance between them to avoid including parts of neighbouring specimens in the image frame. For this set, we collected 1,000 images to form the large training set t00-tan)-,,ain,, 100 images for the small training set (P--100-rriin, and another 100 images for the validation set (IP-1004 v00-vO)
24

---

original
annotated
Figure A8: Examples of special annotation cases. Left: for an insect that is broken into multiple parts with even size, we create mask that covers all of the parts of the insect. The ideal mask should contain minimal background, and keep the edge of the mask as close to the insect's 's edge as possible (left, right). Middle: for two insects where one is in the container and the other is not, annotate the insect that is not in the container. Right: for a split insect we annotate all parts.
Table A5: The Average Precision (AP) and Average Recall (AR) were computed on the IW-150 val and 1P-100-val datasets using the DETR model, which was pre-trained with different training splits.
INSECTS-PINNED-ION Val
INSECTS-WELL-150-
Training data
AP[0.75]
AR[0.50:0.95]
AP[0.75]
AR[0.50:0.95]
IP-100
0.910
0.893
0.543
0.729
IP-1000
0.949
0.918
0.415
0.587
IW-150
0.415
0.587
0.801
0.802
IW-1000
0.665
0.695
0.872
0.835
IP-1000 + IW-1000
0.964
0.907
0.901
0.885
INSECTS-WELL (IW). In these images, the insects are placed in a well. Here the images tend to have less clean background due to the glass and uneven reflected light. The images are taken using a Keyence VHX-700 Digital Microscope system with a fully integrated head and automatic stage that permits high-resolution (4 k) microphotography of individual specimens. Because its scanning stage can hold a 96-well plate, the system automatically acquires high-resolution image of each specimen by controlling movements in the X-Y plane. As well, its capability control the z-axis position of the stage with a precision of 0.1 m allows it to photograph each specimen at multiple heights before rapidly compiling these images we collected 1,000 images to form the large training set (1W-1000-traing 150 images for the small training set NW-110-trainn and another 150 images for the validation (IW-150-val
in-focus image (depth stacking). For this set,
Note that the BIOSCAN- IM Insect Dataset consists only of insects in wells. We include the insects with pins to extend the usefulness of the cropping tool for broader spectrum backgrounds that may appear in the process that specimens are acquired in the larger BIOSCAN project.
During annotation, we focus on masking the main insect and we exclude small broken pieces of the insect that are far from its body (see Figure A7). There are also challenging cases where the insect may be broken into pieces there are multiple insects (see Figure A8). For insects that are broken into multiple pieces of similar size, we create a mask that covers all the pieces. When there are multiple insects, we mask only the central insect.
A7.3 Experiments
A7.3.1
Metrics
The metrics we used are the Average Precision (AP) and the Average Recall (AR) with the IOU of bounding box equal to [0.75] and [0.50:0.95], as they measure the precision and recall aspects of detection performance. AP reflects the accuracy of detection by considering the overlap between predicted and ground truth bounding boxes, while AR assesses how well the system captures all the ground truth objects.
25

---

original
bounding boxes
cropped
IP
Figure A9: Cropping examples of images from ESSECT-PINNED (IP) and INSECTS- WELL (IW) with the original image, image with detected bounding boxes in red, extended bounding boxes in blue, and final cropped image.
IP
IW
Figure A10: Examples of imperfect insect detection (IOU < 0.85), with ground-trute truth bounding box in green, detected bounding boxes in red, and extended in blue. In the second image of IP, note that we extend the image with the white background to fit the bounding box that escapes the original image boundaries.
A7.3.2 Cropping results
We show examples of cropped images in Figure A9 The images show the accurate identification of the insect subject by the DETR model (red bounding box) and the extended bounding box (blue bounding box) used for cropping. In Figure AAT we show cases where the predicted bounding boxes have an intersection over union (IoU) with the ground truth bounding boxes (green bounding box) less than 0.85. From these examples, we observe that the antennac certain insects and the presence of cluttered backgrounds sometimes can create disturbances to our fine-tuned DETR model. However, by expanding the predicted bounding boxes, we are still able to capture the desired information within the cropped images.
To evaluate the performance of our cropping tool with different amount and type of data, we trained the DETR model with 5 training splits (IP-100, IP-1000, IW-150, IW-1000 and IP-1000+IW-1000) and evaluate these models on two validation DaiissIP-110-wa and IW-150-val). Overall, from Table A5 we see that using the mixed training split with 1000 images from IP and 1000 images from IW results the highest accuracy. This the model that we use for cropping the images in the BIOSCAN- IM Insect Dataset.
26

---

Table A6: Comparison of classification accuracy results original images cropped images. Both
are resized to 256 on the smaller dimension. Overall, we find the cropped images yield slightly higher accuracy.
Order-level
Family-level
Micro-average
Macro-average
Mvero-average
Macro-average
Image type
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
original cropped
0.9626 0.9786
0.9970
0.8218
0.9964
0.9248
0.9802
0,9109
0.9730
0.9976
0.8757
0.9980
0.9314
0.9786
0.9154
0.9728
Twining Lis ouer Fporhs using different images
Top-1
Arr
on the val ouer Fporhs uiing different imaget
94
0.
0.4
0.10
5
10
20
25
30
5
10
20
25
30
Epochs
Epochs
Figure A11: The training loss and Top-1 accuracy the validation split during the training of family-level classification of images of insects using cropped (blue) and original (orange) images. Both are resized 256 on the shorter side.
A7.3.3 Insect classification using cropped images
We further evaluate the effectiveness of our auto-cropping tool on a downstream task: insect image classification the order/family level. In Table A6 compare the classification performance of the original vs. cropped images on the BIOSCAN small dataset following the training setup we described in the main paper. We use the ResNet-50 backbone with cross-entropy loss and train with the AdamW W optimizer with learning ceetnn of 0.001 and momentum of 0.9 for 100 epochs for order-level classification and 40 epochs for family-level classification. All images are resized such that the shorter side has size 256. During training, we apply random horizontal flip with probability of 0.5, and random crops of 224 224 are extracted and fed into the backbone to extract image features. During inference, the center 224 x 224 crop is extracted. We measure the micro and class macro average top-K K accuracy at K and K
From Table A6 we see that in most cases, using cropped images to perform training results in higher classification accuracy. In the cases where original image type outperforms cropped type, the difference is small.
To further compare the difference between using original images and cropped images for training, also compare the loss curve during training with original and cropped images.
By comparing the loss at epoch 10, 15 and 20, we see that using the cropped images can help the model converge faster. Using the cropped images also yields higher top-1 accuracy on the validation split.
27


Reviews:
[{"review_id": "FLLF84Jqlj", "reviewer": "Reviewer_qQMp", "summary": "", "questions": "", "limitations": "I do not see limitations that should have been discussed.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This is a huge dataset of 1M insects. It's a very large amount of work to have collected and labeled this, and it offers significant opportunities for algorithmic innovation both in the tasks the authors present and in other potential tasks. The data is innovative here - e.g. iNaturalist is mostly not insects, and those insects that are there are photographed in the wild by amateurs, not as they would be collected by professional entomologists, and also there is no DNA data provided.\n\nThe baselines conducted are appropriate and perform well, though show opportunities for improvement, especially in the Macro accuracy.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mhJ1eeYn0g", "reviewer": "Reviewer_Dp7Q", "summary": "", "questions": "", "limitations": "1. Most samples in the dataset are classified only to the family level and it is unclear from the paper why finer-grained classes are not provided if the data are available as this is another step that researchers will have to take (e.g. querying BOLD).\n\n2. Samples were only acquired from three countries. It would be interesting to see some discussion about use-cases for this dataset in the context of other large out-of-domain biological datasets (e.g. iNaturalist). For example, how well would a classifier trained on this dataset work on non-microscopy images? Presumably existing datasets/baseline models such as iNaturalist also contain some class overlap: would we expect a model trained on other insect imagery to also perform well at the order level?\n\n3. The authors highlight that biological datasets offer an opportunity to test hierarchical models/losses, but that it's out of the scope of the paper. This is a shame, because I think it would be a lot more interesting than an off-the-shelf classifier. Even a non-hierarchical classifier trained to predict `Name` (e.g. each class is `order_family` with 10952 classes) would be worth exploring in my opinion.\n\n4. The paper states \n\n> ... also provides the Subfamily and Subspecies rank\n\nhowever in the supplementary information, Table S2, 862,831 images do not have a subfamily and 1,128,300 images do not have a subspecies. The manuscript says there are 1.28M(?) images in the dataset, so it's incorrect to say that subspecies is provided and in 2/3 cases, subfamily is not provided either. Please update this wording to accurately reflect this.\n\n5. An ethics statement is provided in supplementary information, but no societal impacts are discussed. One potential societal impact of automated barcoding techniques is how relevant human taxonomists are in a world of low-cost, fast sequencing. As I mentioned elsewhere, one assumes that DNA sequencing provides a higher confidence than a human annotator, but presumably there are cases when humans are currently able to provide insight/confirmation of results. Note this is more of a provocation for discussion than criticism.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The motivation for the paper is compelling and it is well-written with illustrations of concepts that might not be familiar to the machine learning community (for example taxonomic classification). The dataset itself is large and well-documented, though I have noted some improvements and concerns below.\n\nThe paper is accompanied by a [public](https://github.com/zahrag/BIOSCAN-1M) code repository that contains analysis and model training code. While I noted that there were several recent commits (post-submission), they are largely cosmetic fixes and that the repository reflects the manuscript. The code I reviewed looked clean, well structured and well documented.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "58lyawv9l8", "reviewer": "Reviewer_t14s", "summary": "", "questions": "", "limitations": "- The authors did not discuss the potential limitations of the dataset nor potential negative societal impacts (though I can't imagine any potential negative societal impacts from it).\n- It seems like there is opportunity for future research using the proposed dataset to focus on specific aspects or subtasks of the dataset that are more challenging or relevant for domain scientists, which I think could have been discussed more. For example, should users of the dataset focus on improving minority class accuracy? Should they optimize for 1 shot or few shot classification accuracy? More discussion of this could help increase the impact of the dataset (even better if these were reflected in the benchmark tasks).\n- Minor suggestions:\n  - It wasn't clear in Figure 4 if the images are from the original image dataset, or cropped, or resized, etc.\n  - Line 233 uses the acronym \"FFN\" which was not previously defined (I assume this is meant to be feed forward network)", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The dataset is very large and contains substantial metadata for most samples that can enable researchers to pursue a wide variety of new methods that take advantage of metadata or the hierarchical structure. The labels should be very high quality based on the procedures for annotation described in the paper.\n- The authors did a good job of explaining the necessary domain background for understanding the dataset while also providing sufficient detail for ML practitioners to understand how to use the dataset.\n- The authors made thoughtful decisions about dataset splits and subsetting that will make the dataset (and its variants) much easier to use for ML practitioners and accessible for a wide variety of users and compute constraints. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "fjrl5rcSbH", "reviewer": "Reviewer_4KRL", "summary": "", "questions": "", "limitations": "I might have missed it, but i couldn't find a discussion on the limitations of the dataset. In particular, I was hoping there was discussion on the annotation process (whether it was single vs multiple experts), validation of the annotated labels. Hopefully the authors can clarify more on this in the response.   ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "This dataset is very substantial and represents many hours of work needed by experts to carefully annotate and collate. That in itself is very useful for anyone working in this domain and a significant addition to the body of work for taxonomy based classification. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All of the reviewers recommend accepting this paper. Authors highlighted that the dataset is very large and forms an important bridge between the machine learning and ecology communities. Other positive feedback was given to the public availability of the training code, appropriate baselines in the paper, and clear writing. Some minor issues and questions were raised (lack of species-level labels, generalization to other regions, annotation process, hierarchical losses, etc.) but most were addressed by the authors during the rebuttal. I recommend accepting this paper.
Decision: Accept (Poster)