SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zahra Gharaee $^{3*}$ , ZeMing Gong $^{4*}$ , Nicholas Pellegrino $^{3*}$ , Iuliia Zarubiieva $^{2,5}$ , Joakim Bruslund Haurum $^{7}$ , Scott C. Lowe $^{5,8}$ , Jaclyn T.A. McKeown $^{1,2}$ , Chris C.Y. Ho $^{1,2}$ , Joschka McLeod $^{1,2}$ , Yi-Yun C Wei $^{1,2}$ , Jireh Agda $^{1,2}$ , Sujeevan Ratnasingham $^{1,2}$ , Dirk Steinke $^{2\dagger}$ , Angel X. Chang $^{4,6\dagger}$ , Graham W. Taylor $^{2,5\dagger}$ , Paul Fieguth $^{3\dagger}$
<sup>1</sup>Centre for Biodiversity Genomics, <sup>2</sup>University of Guelph, <sup>3</sup>University of Waterloo, <sup>4</sup>Simon Fraser University, <sup>5</sup>Vector Institute for AI, <sup>6</sup>Alberta Machine Intelligence Institute (Amii), <sup>7</sup>Aalborg University and Pioneer Centre for AI, <sup>8</sup>Dalhousie University https://biodiversitygenomics.net/1M_insects/
In an effort to catalog insect biodiversity, we propose a new large dataset of hand-labelled insect images, the BIOSCAN-1M Insect Dataset. Each record is taxonomically classified by an expert, and also has associated genetic information including raw nucleotide barcode sequences and assigned barcode index numbers, which are genetically-based proxies for species classification. This paper presents a curated million-image dataset, primarily to train computer-vision models capable of providing image-based taxonomic assessment, however, the dataset also presents compelling characteristics, the study of which would be of interest to the broader machine learning community. Driven by the biological nature inherent to the dataset, a characteristic long-tailed class-imbalance distribution is exhibited. Furthermore, taxonomic labelling is a hierarchical classification scheme, presenting a highly fine-grained classification problem at lower levels. Beyond spurring interest in biodiversity research within the machine learning community, progress on creating an image-based taxonomic classifier will also further the ultimate goal of all BIOSCAN research: to lay the foundation for a comprehensive survey of global biodiversity. This paper introduces the dataset and explores the classification task through the implementation and analysis of a baseline classifier. The code repository of the BIOSCAN-1M-Insect dataset is available at https://github.com/zahrag/BIOSCAN-1M
Global change is restructuring ecosystems on a planetary scale, creating an increasingly urgent need to track impacts on biodiversity. Such tracking is exceptionally challenging because life is highly diverse: the biosphere comprises more than 10 million multicellular species [42]. Until recently, this complexity has meant that an Earth observation system for biodiversity was inconceivable, however the increased power of DNA sequencing and the recognition that living organisms can be discriminated by short stretches of DNA have revealed a way forward, which has become the central focus of the International Barcode of Life (iBOL) Consortium.
Discriminating organisms by DNA sequences [22, 6] can revolutionize our understanding of biodiversity, not only by providing a reliable species proxy for known and unknown species, but also by revealing their interactions and assessing their responses to changes in the ecosystem. This is essential to mitigate a looming mass extinction, where an eighth of all species may become extinct by 2100 unless there is a significant change in human behaviour [10, 11].
*Joint first author.
†Joint senior/last author.
Figure 1: BIOSCAN-1M Insect dataset records contain high-quality microscope images of insects and labels including the taxonomic classification, raw DNA sequences, and Barcode Index Number (BIN). Pictured here is a mosquito of the subfamily Culicinae, the most populous subfamily of mosquitoes with species found around the world.
The BIOSCAN project [2], lead by iBOL, has the following three main goals:
1. Species discovery.
2. Studying the interactions between species.
3. Tracking and modelling species dynamics over geography and time.
To that end, BIOSCAN collects samples of multicellular life from around the world. Each sample is individually imaged, genetically sequenced and barcoded [22], and then classified by expert taxonomists. Of particular interest to the BIOSCAN project are insects, which constitute a great proportion of the Earth's species and many of which remain unknown. Indeed, it is estimated that  $5.5\mathrm{M}$  insect species exist worldwide, of which only roughly one million have been identified [54, 23]. The rate of insect collection within the BIOSCAN project is increasing as the project progresses, such that  $3\mathrm{M}$  insect specimens will be collected in 2023 and  $10\mathrm{M}$  by 2028.
Using high-resolution photographs, human taxonomists can accurately classify insects from within their domain of expertise. However, human annotation cannot scale to the volume of samples needed to measure and track global biodiversity. Moreover, many taxonomists with highly specialized knowledge are leaving the practice and won't be replaced. Thus, the use of artificial intelligence and machine learning to process visual and textual information collected by the BIOSCAN project is crucial to the success of a planet-scale observation system. Classification of the insect images to their taxonomic group ranking is especially useful in regions of the world where the facilities required to perform genetic barcoding are not available. Indeed, even beyond this project, there are opportunities for computer vision to transform entomology [25].
This article has two main contributions.
1. The publication of the BIOSCAN-1M Insect image dataset, containing approximately 1.1 M high-quality microscope images, each of which is annotated by the insect's taxonomic ranking and accompanied by its raw DNA sequences and Barcode Index Number (BIN) [47], an example of which is shown in Figure 1.
2. The design and implementation of a deep model, classifying BIOSCAN-1M Insect images into specific taxonomic ranking groups, to serve as a baseline for future work utilizing this dataset.
2 Background and Related work
This section provides background on taxonomic classification, the use of genetic barcoding, and several challenges in the field of machine learning associated with our dataset.
2.1 Taxonomic Classification
In biology, taxonomic classification is the study of hierarchically categorizing lifeforms based on shared characteristics. In particular, Linnean taxonomy [7, 20, 32] forms the basis for the modern (generally accepted) system of taxonomy, of which the main hierarchical ranks are domain, kingdom, phylum, class, order, family, genus, and species, as shown in Figure 3. All insect life is part of the class Insecta.
Conventionally, expert taxonomists classify organisms based on their appearance and behaviour [7]. However, this approach is susceptible to both misclassification and lacks consensus throughout the community of taxonomists, since it is difficult to prove with certainty that a given classification is absolutely correct. This shortcoming of traditional taxonomy has prompted the use of classification heuristics, based on fairly concrete evidence in the form of genetic codes, that are sensitive to species identity.
Table 1 presents a comprehensive overview of both the number of unique categories and the degree to which the BIOSCAN-1M Insect dataset is labelled at each taxonomic rank. According to the table, a substantial number of samples lack labels at more specific taxonomic ranks. This limitation underscores the challenge of incomplete taxonomic labeling within the dataset. However, the ongoing efforts of expert curation are continuously improving this aspect, and future versions of the dataset are expected to include a more comprehensive taxonomic classification for a larger number of samples.
Based on the statistical analysis of the BIOSCAN-1M Insect dataset, it is evident that there are 3,441 distinct categories at the genus level and 8,355 at the species level. However, the amount of data to train such fine-grained classifiers is limited, with 254,096 samples classified by experts at the genus level and only 84,397 samples at the species level. This substantial class-number-to-data-size imbalance poses a notable challenge when training models for fine-grained classification, specifically at the genus and species level.
Table 1: An overview of the number of unique categories and number / proportion of expert-labelled samples within the BIOSCAN-1M Insect dataset at each taxonomic rank. Additionally, in the bottom row, the corresponding information is given in relation to Barcode Index Number (BIN), presented as a genetic alternative to taxonomic labels (species proxy). Observe that all samples have an associated BIN, and there are roughly  $10 \times$  more unique BINs than species labels.
Taxonomic Level	Categories	Labelled Samples	Labelled (%)
Phylum	1	1,128,313	100.0
Class	1	1,128,313	100.0
Order	16	1,128,313	100.0
Family	491	1,112,968	98.6
Subfamily	760	265,492	23.5
Tribe	535	60,477	5.4
Genus	3,441	254,096	22.5
Species	8,355	84,397	7.5
Barcode Index Number (BIN)	90,918	1,128,313	100.0
2.2 Genetic Barcoding and Barcode Index Numbers
DNA barcoding [22, 6] employs large-scale screening of one or a few reference genes for assigning unknown individuals to species, as well as aiding in the discovery of new species [43]. Barcoding is commonly used in several fields including taxonomy, ecology, conservation biology, diet analysis and food safety [49, 53]. It is faster and more accurate than traditional methods, which rely on the judgment of experts [46].
Barcoding is based on the use of a short, standardized segment of mitochondrial DNA, typically a portion of the mitochondrial cytochrome  $c$  oxidase subunit I (COI) gene, which is nearly always unique for different species. Once the DNA sequence is obtained, it can be compared to a reference library of known sequences to identify the species.
The concept of genetic barcoding can be taken a step further by mapping barcodes to clusters of organisms (characterized by their barcodes) with a highly similar genetic code, known as an
Operational Taxonomic Unit (OTU) [52, 5]. OTUs act as a proxy for species based on the high degree of genetic similarity exhibited by their members. To enable indexing, each OTU is assigned a Uniform Resource Identifier (URI), commonly referred to as the Barcode Index Number (BIN) [47], which offers a unique representation such that genetically identical taxa will be assigned the same BIN, and each BIN is registered in the Barcode Of Life Data system (BOLD) [1]. BINs additionally provide an alternative to the use of Linnean names, offering a genetics-based classification of organisms.
BOLD, the Barcode of Life Data System [1], is a pivotal resource in biodiversity science. It facilitates DNA barcode acquisition, storage, validation, and analysis, integrating molecular, morphological, and distributional data. BOLD hosts 17 million specimen records and 14 million barcodes, spanning over a million species worldwide. It plays a central role in species identification, genetic diversity exploration, and evolutionary studies. Launched in 2005, BOLD aims to identify all eukaryotic species and offers integrated analytical tools, comprehensive data management, and secure collaboration.
2.3 Machine Learning Challenges
As will be demonstrated in Section 3, the dataset exhibits two key characteristics corresponding to open problems in the field of machine learning.
Class imbalance. The degree to which the expected quantity of instances varies between classes is known as the class imbalance. In the context of a closed dataset, the class imbalance describes the disparity in number of examples among classes [27, 30]. As we describe in Section 3, and Figure 2 the published dataset exhibits a long-tailed class distribution whereby the distribution of class sizes closely follows a power-law, indicating that there is a substantial class imbalance. This represents a challenge due to the disproportionate amounts of available training data for majority vs. minority classes.
Figure 2: Class distribution and class imbalance in the BIOSCAN-1M Insect dataset. We focus on the 16 most densely populated orders (top) and the 40 most densely populated families (bottom).
Hierarchical classification. Classification problems involving data with labels that are inherently hierarchical present a unique challenge in comparison to simpler "flat" classification problems [50]. The outputs of hierarchical classification algorithms are defined over tree-like class taxonomies, where the relationship between parent and child nodes is given by the asymmetric "is-a" relationship. A basic example of this is the relationship that "all dogs are canines, but not all canines are dogs", whereby "dogs" would be a child node of the parent node "canines", which itself may be a child of "mammals". The dataset published here perfectly matches this paradigm and may be used to study novel approaches for handling the hierarchical classification problem. Note that the baselines we adopt in this paper do not pursue a hierarchical strategy but instead classify to fixed levels of the taxonomy: order and family. Hierarchical strategies are a topic of present and future work.
2.4 Biological Datasets
Image-based insect classification [39] most often finds use in agricultural settings, where Integrated Pest Management (IPM) systems are used to identify and count harmful insect pests [33, 51]. In combination with this, holistic systems capable of also identifying plant diseases through computer vision are a popular area of research [15, 12, 40].
Recently, DNA sequences have been analyzed [28] using tools from the field of Natural Language Processing [44], and in particular, through the application of bidirectional encoder representations from transformers (BERT) [14]. Indeed, BERT-based models have been used to taxonomically classify genetic sequences [24, 41]. Other recent work has used DNA barcodes as "side information" to perform zero-shot species-level recognition from images, albeit at a much smaller scale than the BIOSCAN-1M Insect dataset [4].
Perhaps the best known and largest biological dataset is iNaturalist [56, 26, 57], the 2021 version of which contains  $2.7\mathrm{M}$  images from over 10,000 different species of plants, animals, and fungi, specifically with 2,526 species of insects with  $663\mathrm{k}$  annotated insect images. Many insect-specific image datasets focus on insect as pests found in agricultural settings [62, 59, 58, 16, 63, 19, 37, 34]; the most prominent of which, the IP102 [62] dataset, contains roughly  $75\mathrm{k}$  insect images, covering 102 species of common crop insect pests.  $19\mathrm{k}$  of these are annotated with bounding boxes for object detection. In the space of plants, the PlantNet-300K [18] dataset has  $306\mathrm{k}$  images labeled by species and was constructed by sampling the larger PlantNet database [3]. Table 2 highlights key biological datasets across a variety of domains and indicates the degree of class imbalance [13],  $\beta$ , which is defined as the ratio of the number of samples in the largest to the smallest class.
The BIOSCAN-Order and BIOSCAN-Diptera datasets, introduced in Section 4, refer to subsets of the full BIOSCAN-1M Insect dataset for use in order- and family-level classification, respectively. Observe that while there is significant variety in the imbalance factor among datasets, the imbalance of BIOSCAN-Order is orders of magnitude greater than that of all the other datasets. The Pl@ntNet-300K [18] dataset also has remarkably high class imbalance, exceeding that of the BIOSCAN-Diptera dataset. While a high imbalance ratio was expected of the BIOSCAN-1M Insect dataset based on the dataset's biological nature, the metric is highly sensitive due to its dependence on only the two most extreme classes.
The iNaturalist dataset encompasses a greater number of insect species than any pre-existing dataset. We measured the number of genera (plural of genus) and species that were common across both datasets. Of the 2,526 insect species in iNaturalist and 8,355 species annotated in BIOSCAN-1M Insect, only 153 genera and 62 species appeared in both datasets. This indicates the species in BIOSCAN-1M Insect predominantly do not appear in iNaturalist. Furthermore, based on the number of unique BINs present in the BIOSCAN-1M Insect dataset, it can be assumed that the dataset in fact encompasses almost  $91\mathrm{k}$  distinct (possibly as yet unnamed) insect species, a far greater quantity than that of iNaturalist.
This section describes the information made available through the publication of the BIOSCAN-1M Insect dataset, and details the procedures which generated the information.
3.1 BIOSCAN-1M Insect dataset resources
The BIOSCAN-1M Insect dataset provides four main sources of information about insect specimens. Each sample in the dataset consists of a biological taxonomic annotation, DNA barcode sequence, Barcode Index Number (BIN), and a RGB image of a single specimen. In the following sections, this information is described in detail.
3.1.1 Biological taxonomy
The BIOSCAN-1M Insect dataset specifies biological taxonomic rank following the Linnean taxonomy as described in Section 2.1. In addition to the main groups shown in Figure 3, the dataset also provides the subfamily and subspecies ranks. The subfamily rank is an auxiliary (intermediate) taxonomic rank, the next below family but more inclusive than genus. Subspecies is a taxonomic rank below species, and it is used for populations that live in different areas and vary in size, shape, or other physical characteristics, but that can successfully interbreed. Finally, we also provide "Name"
Table 2: Summary of biological fine-grained and long-tailed datasets. Note that "iNaturalist-Insect" describes the subset of iNaturalist (2021 version) images that comprises insects. *For the BIOSCAN-1M Insect dataset, we report the number of unique Barcode Index Numbers (BINs) instead of the number of unique Linnean taxonomic species. The BIN is a mitochondrial DNA-based identifier which provides a species-like proxy of an organism and can be used as an alternative to Linnean taxonomy (see Section 2.2).
Name / Citation	Domain	Images	Categories	Taxonomic Rank	Imbalance, β
iNaturalist (2021) [57]	All	2,686 k	10,000	Species	1.97
iNaturalist-Insect [57]	Insects	663 k	2,526	Species	1.97
PI@ntNet-300K [18]	Plants	306 k	1,000	Species	3,604.00
Urban Trees [60]	Trees	80 k	18	Species	7.51
IP102 [62]	Insects	75 k	102	Species	13.63
NA Birds [55]	Birds	48 k	400	Species	15.00
LeafSnap [31]	Plants	31 k	184	Species	8.00
Pest24 [59]	Insects	25 k	24	Common name / species	493.95
Flowers 102 [45]	Flowers	8 k	102	Genus	1.00
BIOSCAN-1M Insect	Insects	1,128 k	90,918	Barcode Index (BIN)*	12,491.00
BIOSCAN-Order	Insects	1,128 k	16	Order	156,856.75
BIOSCAN-Diptera	Insects	891 k	40	Family	1,092.61
Figure 3: Biological taxonomic ranking and classification. Taxonomic ranks are shown in the top row, with the classification (i.e., labels) for the Western honey bee shown below.
to indicate the lowest (most specific) known rank label. For instance, if there exists a species name of an organism, "Name" will show that, but if the organism only has Family level ID, then "Name" will show that ID. Each sample has a name and there are in total 10,952 unique names labelled in the dataset.
Not all data samples have labels for all taxonomic ranks recognized in the BIOSCAN-1M Insect dataset. As an example, the family group of the BIOSCAN-1M Insect dataset is indexed by 494 distinct families, however, there are 16,067 data samples that are not associated with any of these families, since they were not classified by human taxonomists. As a consequence, there are many data samples that are not classified into finer-level groups like subfamily, tribe, genus, species, or subspecies. The lack of precise annotation at all ranks is one of the major challenges of the BIOSCAN-1M Insect dataset when performing classification tasks.
3.1.2 DNA Barcode and Indexing
Section 2.2 described the concept of genetic barcoding and the generation of Barcode Index Numbers (BINs). The BIOSCAN-1M Insect dataset contains genetic barcodes and BINs for all samples. This information is represented as the raw nucleotide barcode sequence, under the Nucraw field, and the Barcode Index Number (BIN), denoted by uri. Independently, the field processid is a unique number assigned by BOLD to each record, and sampleid is an identifier given by the collector.
3.1.3 RGB images
The BIOSCAN-1M Insect dataset offers a wealth of information through its collection of insect images. The dataset contains high-resolution (2880×2160 pixel) RGB images in JPEG format; Figure 4 displays a selection of images representing insects from 16 most densely populated orders.
We have released multiple packages of the BIOSCAN-1M Insect dataset aimed at different purposes. These packages are organized into 113 chunks, each containing  $10\mathrm{k}$  images. The packages include:
- Original JPEG Images stored in 113 zip files (2.3 TB).
- Cropped images organized into 113 zip files (151 GB).
- Resized original images which have a size of  ${256}\mathrm{{px}}$  on their smaller side (26 GB).
- Resized cropped images having a size of  ${256}\mathrm{{px}}$  on their smaller side (7GB).
Additionally, we also provide the dataset in HDF5 archive format for both the resized original and cropped images.
(a) Diptera
896,324
(b) Hymenoptera
89,311
(c) Coleoptera
47,328
(d) Hemiptera
46,970
(e) Lepidoptera
32,538
(f) Psocodea
9,635
(g) Thysanoptera
2,088
(h) Trichoptera
1,296
(i) Orthoptera
1,057
(j) Blattodea
824
(k) Neuroptera
676
(1) Ephemeroptera
96
(m) Dermaptera
66
Figure 4: Examples of original insect images from 16 orders of the BIOSCAN-1M Insect dataset. The numbers below each image identify the number of images in each class, and clearly illustrate the degree of class imbalance in the BIOSCAN-1M Insect dataset. "Siphonaptera", "Strepsiptera" and "Zoraptera" are removed from classification experiments due to an insufficient number of samples.
(n) Archaeognatha
63
(o) Plecoptera
30
(p) Embioptera
6
3.2 BIOSCAN-1M Insect dataset generation
The BIOSCAN-1M Insect dataset consists of specimens mostly collected from three countries (Costa Rica, Canada, and South Africa) using Malaise traps. RGB images of the organisms were taken with a Keyence VHX-7000 microscope. Images are organized by workflow units: 96-well microplates of which 96 are used in a single sequencing run (9,120 samples at a time).
DNA barcodes of the organisms were generated using a high-throughput approach utilizing the Pacific Biosystems Sequel platform, which employs Single-molecule, real-time (SMRT) sequencing to generate long-read length DNA and cDNA. The taxonomic classifications were created by matching the generated barcodes to a reference library on the Barcode of Life Data System (BOLD) at the Centre for Biodiversity Genomics in Canada.
We provide a comprehensive metadata file alongside the RGB images, which includes taxonomic annotations, DNA barcode sequences, and data sample indexes and labels. The metadata file also contains image names and IDs to locate the corresponding images within the dataset packages. Additionally, it identifies the images associated with the training, validation, and test splits.
By employing stratified class-based sampling, we methodically curated three subsets of varying sizes from the BIOSCAN-1M Insect dataset. Subsequently, we carried out two distinct sets of classification experiments, yielding a total of six datasets. The three subsets, namely Small, Medium, and Large, each comprising approximately  $50\mathrm{k}$ ,  $200\mathrm{k}$ , and  $1\mathrm{M}$  data samples, respectively. The initial set of experiments primarily revolves around classifying insect images into their 16 most densely populated taxonomic orders. Subsequently, the second set of experiments delves even deeper, focusing on classifying samples within the Order Diptera into their 40 most densely populated families.
4.1 Subset sampling and split mechanism