SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To create subsets of the BIOSCAN-1M Insect dataset, we followed a systematic two-step process. Initially, we sampled a subset exclusively from the Diptera Order, specifically selecting the 40 families with the highest number of members. This led to the creation of the BIOSCAN-Diptera dataset. Subsequently, we divided the BIOSCAN-Diptera dataset into separate train, validation, and test sets. Finally, we used these split sets of the BIOSCAN-Diptera dataset as a basis to construct the corresponding train, validation, and test sets for the BIOSCAN-Order dataset. Our methodology involved the use of stratified class-based sampling to preserve the class distribution consistently across all subsets, ensuring the integrity of our experiments.
Table 3: The total number of samples used in the six different sized subsets of the BIOSCAN-1M Insect dataset: The entries display the number of data samples in the train, validation, and test sets, as well as the number of classes for order-level (16 orders) and Order Diptera family-level (40 families) experiments.
Dataset	Total	Train	Validation	Test	Categories
BIOSCAN-Order	1,128,308	789,813	112,835	225,660	16
BIOSCAN-Diptera	891,338	623,937	89,135	178,266	40
BIOSCAN-Order/Diptera Medium	200,000	140,000	20,000	40,000	16/40
BIOSCAN-Order/Diptera Small	50,000	35,000	5000	10,000	16/40
The Small and Medium subsets are generated by sampling  $50\mathrm{k}$  and  $200\mathrm{k}$  data samples, respectively, from the train, validation, and test sets of the BIOSCAN-Order and BIOSCAN-Diptera datasets. In all of our classification experiments, we used class-based stratified sampling to split the dataset into train, validation and test sets. To this end,  $70\%$  of the samples of each class are randomly selected as training,  $10\%$  as validation, and  $20\%$  as test samples, as shown in Table 3.
The extreme class imbalances, which are an inherent characteristic of the BIOSCAN-1M Insect dataset, are addressed to some extent by having all classes represented in the train, validation and test sets. Classes with no samples for either split set are omitted. In the insect order-level classification (Figure 4), we have sufficient data samples for 16 out of 19 orders in the train, validation, and test sets. For the Diptera family-level classification, we focus on the 40 most populous families within Diptera.
4.2 Data preprocessing
To improve computational efficiency, we crop and resize the images to be  $256\mathrm{px}$  on the smaller dimension. Preliminary experiments with ResNet-50 comparing original images with images that are cropped show that cropping can help model learning to converge more rapidly and lead to slightly better performance. Reducing the resolution to  $256\mathrm{px}$  helps to reduce the size of the large dataset from 2.3 TB down to 26 GB for the original uncropped images, and from 151 GB down to 7 GB for cropped images. We choose to run experiments on the cropped and resized images due to the small size which allows for efficient data loading from disk.
The BIOSCAN-1M image datasets have insects with varying size, pose, color and shape. Due to these variations, cropping is not a simple task. We develop our cropping tool by fine-tuning a DETR [9]
model with ResNet-50 [21] backbone (pretrained on MSCOCO [35]) on a small set of 2,000 insect images annotated using the Toronto Annotation Tool Suite [29]. In DETR, the CNN-based feature extractor extracts a set of image features that are fed into a transformer-based encoder-detector. The detector takes a set of learned positional embeddings as object queries and uses them to attend to the encoder outputs. Each of the output decoder embeddings is then passed to a shared FFN which predicts whether there is an "insect" or "no object" and regresses the bounding box. The DETR model is further trained for 10 epochs with the AdamW optimizer with learning rate of 0.0001, weight decay of 0.0001 and a batch size of 8.
To crop the image, we apply our fine-tuned DETR model and take the predicted bounding box with the highest confidence score. The finalized cropping is determined as the predicted bounding box, extended equally in width and height by 0.4 of the maximum dimension.
4.3 Classification model
To run classification experiments, we fine-tuned two different pre-trained models to extract deep visual features of insects from their RGB images. Our pre-trained models are ResNet-50 [21] and a transformer based model, ViT-Base-Patch16-224 [17]. During training, we take random  $224 \times 224$  crops from the image as input, while during validation we take the center crop. To train our model, we used two loss functions, the Cross-Entropy (CE) as a baseline and the Focal loss, which is more suitable for datasets having class imbalances [36, 8, 13].
The detailed hyperparameters used for our experiments are shown in Table 4. For conducting the experiments, we leveraged the computational resources provided by the Digital Research Alliance of Canada's Narval and Beluga clusters. To ensure efficient processing. Each experiment was performed using a single node equipped with 1 GPU, 10 CPUs per task, and a memory allocation of 128GB.
Table 4: Detailed hyperparameter settings of the experiments.
Parameters	Settings
Model	ResNet-50;ViT-B/16
Loss function	Cross-Entropy;Focal
Optimizer	SGD
Weight Decay (μ)	0.0001
Learning rate	0.001
Momentum	0.9
K	[1, 3, 5, 10]
group-level	Order;Family
Parameters	Settings
Batch-Size	32
Epoch	100
Num-Workers	4
Image-Size (Train/Val)	256
Crop-Size (Train)	224
Rand-Horizontal-Flip (Train)	Yes
Centre-Crop (Val)	224
Dataset size	L/M/S
We conducted a set of 24 trials, each executed with 3 distinct seeds. These trials were carried out to tackle classification tasks involving Insect-Order and Diptera-Family utilizing three dataset variations: Large, Medium, and Small. Our design encompassed the creation of 4 distinct models, integrating two distinct loss functions (Cross-Entropy and Focal) and two different pretrained backbones (ResNet-50 and ViT-B/16).
The combination of these diverse components led to the calculation of average performance across a range of seed values. Subsequently, the model that exhibited the highest average performance on the validation set was selected for further evaluation during inference, as depicted in Table 5. It's worth noting that models employing the ViT-B/16 backbone and Cross-Entropy loss function demonstrated superior performance across most of the experiments on the validation set, leading to their selection for inference using test data. For the Small and Medium datasets, the models underwent 100 epochs of training, while for the Large dataset, a lesser number of epochs were applied, as convergence was achieved on the validation set.
We evaluate the performance of our classification models using top-K accuracy, which extracts the K-predicted classes with the largest probabilities for each input sample and compares them with the ground-truth class label of the sample. If the ground-truth label is among the top-K predictions then the model counts it as a correct classification. The total counts are then divided by the total number of input samples to yield an average. We report test results of the best model from validation
performance for the micro, class-averaged macro top-K accuracy at  $K \in [1,3,5]$  as well as micro and macro  $F_{1}$  Scores Table 5.
Figure 5 shows the per-class top-1 test accuracy for the Order and Family classification of the Large dataset. Accuracy is quite high, above  $90\%$ , for most classes, decreasing mainly for classes with little training data.
Table 5: Top-K accuracy and class-averaged macro top-K accuracy based on the test sets of Insect-Order and Diptera-Family classification experiments using the Small, Medium and Large datasets.
Classification	Dataset	Micro Top-K	Macro Top-K	F1-Scores
Top-1	Top-3	Top-5	Top-1	Top-3	Top-5	Micro	Macro
Insect-Order	Small	97.86	99.35	99.66	85.01	91.68	99.23	97.86	85.84
Medium	99.14	99.77	99.88	85.58	97.68	98.22	99.14	87.36
Large	99.69	99.96	99.98	90.61	98.14	99.32	99.62	92.65
Diptera-Family	Small	94.01	97.26	98.01	92.37	96.53	97.42	94.01	93.03
Medium	96.66	98.34	98.77	91.81	96.37	97.20	96.66	92.77
Large	97.59	98.85	99.23	91.20	95.86	96.72	97.59	91.45
Figure 5: Per-class top-1 test accuracy of the Insect-Order and Diptera-Family classification experiments of the Large dataset. The classes are listed in a descending manner with respect to the number of test samples.
We have described a set of six novel BIOSCAN datasets, on which we conducted image-based classification experiments using the taxonomic annotations of the insects. Looking ahead, iBOL's ongoing efforts will lead to further advancements in several aspects. The rate of insect sample collection is already increasing, resulting in a dataset that is not only larger in terms of the number of records but also more comprehensive, with additional taxa at lower taxonomic levels such as genera and species. Moreover, the dataset will expand to encompass diverse life forms beyond insects. Thus, while the current dataset is already the largest publicly available insect image dataset, it represents just the beginning of what lies ahead.
We acknowledge the support of the Government of Canada's New Frontiers in Research Fund (NFRF), [NFRFT-2020-00073] and a NVIDIA Academic Grant. This research was enabled in part by support provided by Calcul Québec (calculquebec.ca) and the Digital Research Alliance of Canada (alliancecan.ca). Data collection was enabled by funds from the Walder Foundation, a New Frontiers in Research Fund (NFRF) Transformation grant, a Canada Foundation for Innovation's (CFI) Major Science Initiatives (MSI) Fund and CFREF funds to the Food from Thought program at the University of Guelph. The authors also wish to acknowledge the team at the Centre for Biodiversity Genomics responsible for preparing, imaging, and sequencing specimens used for this study, as well as Utku Cicek for their help with the project.
A1 Data collection and organization
The BIOSCAN-1M Insect dataset consists of insect RGB images and a metadata file containing taxonomic annotation, DNA barcode sequences, and an assigned Barcode Index Number (BIN). In the following sections, we describe the resources available within the dataset.
A1.1 RGB images
To facilitate different levels of visual processing we created 6 packages of color images of varying sizes. These packages are as follows:
Original full size RGB images. The original images are converted to JPEG image format. These images each have a resolution of  $2880 \times 2160$ , and they are typically around 5 MB in size, however some images are smaller at  $600 - 800\mathrm{kB}$ . The package is structured as 113 zip files, each of which contains 10,000 images except the last (zip file 113 contains 8,131 original full size images). The total size of this package is 2.5 TB. All 113 zip files are stored within the BIOSCAN project space in GoogleDrive as described in Section A2 inside a folder named BIOSCAN-original_images and the zip files named as bioscan_images-original_full_part<n> where n is the partition ID and is in the range of 1 to 113.
Cropped RGB images. The images in this package are cropped by our cropping tool as described in the main body of the paper and available in the accompanying BIOSCAN-1M code repository. The package is structured into six zip files where each file contains 20 partitions (20×10,000 files), except the last zip file which contains 13 partitions. The total size of this package is 151 GB. All six zip files are stored within the BIOSCAN project space in GoogleDrive as described in Section A2 inside a folder named BIOSCAN_cropped_images and the zip files named as bioscan_images_cropped_part<m-n> where m-n indicate the start and end partition ID, in the range of 1–113.
Resized original RGB images. This package is available in two archive formats (zip and HDF5). The package contains downscaled versions of the original images, requiring reduced storage space. The resizing was done such as to reduce the smaller dimension of image to 256 pixels (and the longer side scaled to preserve the aspect ratio of the original image) and then saved in JPEG format. The total size of these packages are approximately 27 GB, and they are named as original_256.zip and original_256.hdf5.
Resized cropped RGB images. This package is also available in two archive formats (zip and HDF5). The package contains resized versions of the cropped images. The resizing was done such as to reduce the smaller dimension of image to 256 pixels (and the longer side scaled to preserve the aspect ratio of the cropped image) and then saved in JPEG format. The total size of these packages are approximately 7 GB, and they are named as cropped_256.zip and cropped_256.hdf5.
A1.2 Metadata file
To enhance the metadata of our published dataset, we incorporated structured metadata following Web standards. The metadata file for our dataset is named BIOSCAN_Insect_Dataset_metadata. We created two versions of this file: one data frame in TSV format (.tsv) and the other in JSON-LD format (.jsonld). The JSON-LD file was validated using the Google Inspection Tool.
The metadata file is a table with 22 columns, which contain content as described below. Note that if a sample was not labelled by taxonomist, for each taxonomy ranking group (columns 4-13) the corresponding annotation is listed as not_classified instead. Similarly, if a sample has no association with an experiment shown by columns 16-21, then the sample's role is shown as no_split.
1. sampleid: An identifier given by the collector.
2. processid: A unique number assigned by BOLD to each record.
3. URI: Barcode Index Number (BIN).
4. name: Taxonomy ranking classification label.
5. phylum: Taxonomy ranking classification label.
6. class: Taxonomy ranking classification label.
7. order: Taxonomy ranking classification label.
8. family: Taxonomy ranking classification label.
9. subfamily: Taxonomy ranking classification label.
10. tribe: Taxonomy ranking classification label.
11. genus: Taxonomy ranking classification label.
12. species: Taxonomy ranking classification label.
13. subspecies: Taxonomy ranking classification label.
14. nucraw: Nucleotide barcode sequence.
15. image_file: Image file name stored in structured packages.
16. large_diptera_family: Image association with the training, validation, and test split of experiment-1.
17. medium_diptera_family: Image association with the training, validation, and test split of experiment-2.
18. small_diptera_family: Image association with the training, validation, and test split of experiment-3.
19. large_insect_order: Image association with the training, validation, and test split of experiment-4.
20. medium_insect_order: Image association with the training, validation, and test split of experiment-5.
21. small_insect_order: Image association with the training, validation, and test split of experiment-6.
22. chunk_number: A unique ID to locate the corresponding images within the dataset packages.
A2 Informational content
The link to access the dataset and its metadata is https://biodiversitygenomics.net/1M_insects/.
A3 Ethics and responsible use
The BIOSCAN project started by the International Barcode of Life (iBOL) Consortium, has collected a large dataset of hand-labelled images of insects. Each record is taxonomically classified by human experts, and accompanied by genetic information.
The publication of BIOSCAN-1M Insect dataset is a common effort made by researchers from the University of Waterloo, Simon Fraser University, Aalborg University, Dalhousie University and the University of Guelph with support from the Vector Institute for Artificial Intelligence, Alberta Machine Intelligence Institute, Pioneer Centre for AI, and the Centre for Biodiversity Genomics.
The availability of the BIOSCAN-1M Insect dataset presents an immense opportunity for scientific advancement and understand in of insect biodiversity. However, it is important to emphasize the ethical and responsible use of this data.
First and foremost, researchers and institutions must prioritize the protection of individuals' privacy and adhere to data protection regulations and guidelines. To our knowledge, there is no personal or identifiable information in the dataset. However, any such information associated with the dataset should be treated with utmost care and reported to the authors.
Furthermore, the researchers and organizations who utilize the BIOSCAN-1M Insect dataset should ensure transparency in their methodologies and practices. This includes clearly stating the purpose of their research, obtaining informed consent when applicable, and maintaining integrity in the interpretation and reporting of the results.
The responsible use of the BIOSCAN-1M Insect dataset entails promoting open collaboration and sharing of knowledge within the scientific community. Researchers should foster an environment
that encourages exchange of ideas, methodologies, and findings, while giving credit to the original dataset creators. It is essential to acknowledge and respect the contributions of the human experts who hand-labelled the images by taxonomically classifying specimens. Proper attribution and recognition should be given to these individuals, as their expertise and efforts are instrumental in the creation and accuracy of the dataset.
A4 Dataset availability and maintenance
The BIOSCAN-1M Insect dataset and all its content described in the previous sections are available on a GoogleDrive folder named 1M_Photo project. To access the BIOSCAN-1M Insect dataset, please visit https://biodiversitygenomics.net/1M_insects/. We've published a code repository for dataset manipulation, including tasks like downloading dataset packages, image and metadata reading, image cropping, dataset subsampling, partitioning into train, validation, and test sets, and running the classification experiments presented in the BIOSCAN-1M Insect paper. To access the BIOSCAN-1M code repository, please visit https://github.com/zahrag/BIOSCAN-1M.
A5 Licensing
Table A1 shows the copyright associations related to the BIOSCAN-1M Insect dataset with the corresponding names and contact information.
Table A1: Copyright associations related to the BIOSCAN-1M Insect dataset
Copyright Associations	Name & Contact
Image Photographer	CBG Robotic Imager
Copyright Holder	CBG Photography Group
Copyright Institution	Centre for Biodiversity Genomics (email:CBGImaging@gmail.com)
Copyright License	Creative Commons Attribution Non-Commercial Share-Alike (CC BY-NC-SA 4.0)
Copyright Contact	collectionsBIO@gmail.com
Copyright Year	2021
A6 Experiment details and results
A6.1 Backbone models
We utilized two distinct pretrained backbone models for our experiments with the BIOSCAN-1M-Insect dataset. A comprehensive comparison between these models is presented in this section and in Table A2.
ResNet-50 [21] is a deep convolutional neural network, which includes residual blocks that allow for the training of very deep networks without falling into the vanishing gradient problem. ViT-Base-Patch16-224 [61, 17] signifies that the ViT model is designed to process images with a resolution of  $224 \times 224$  pixels. Each image is divided into smaller patches of size  $16 \times 16$  pixels, which are then fed into the transformer layers. Each transformer layer includes multi-head self-attention mechanisms and feed-forward neural networks.
Table A2: A comparison between the two pretrained backbone models used in our experiments: ResNet-50 and the ViT-Base-Path16-224. CNN and FC denote Convolutional Neural Network, and Fully Connected layers, respectively.
Features	ResNet-50	ViT-B/16
Layers	50	12
Based Networks	CNN, Pooling and FC	Transformer
Number of parameters	25.6 M	86 M
Overall ResNet-50 has a deeper architecture with more layers than ViT-B/16. This depth can enable it to learn hierarchical features in the data, while ViT's strength lies in capturing relationships between patches by applying self-attention mechanisms, which enables it to capture long-range dependencies in images thus making it suitable for both local and global context understanding. Moreover, due to
its transformer architecture, ViT can parallelize training more effectively, which can result in faster convergence times despite its higher number of parameters.
A6.2 Validation results
Table A3 shows the performance of all 24 experiments conducted with 3 different seeds using the validation set. According to the validation results, ViT-B/16 with the Cross-Entropy loss function consistently outperforms other models.
Comparing Focal loss to Cross-Entropy, we found that Cross-Entropy produced slightly better results. This could be due to insufficient fine-tuning of Focal loss hyperparameters (alpha and gamma). Furthermore, addressing class imbalance could involve selectively oversampling the less frequent classes during training. This strategy boosts their representation in the training process. For Focal loss, limited exposure to rarer classes might hinder the effectiveness of the re-weighting mechanism.
The presented results of table A3 depict the mean accuracy across various seeds, accompanied by the standard deviation from the average values of each model. The outcomes reveal a notable consistency in the performance of almost all models.
The six models highlighted in bold in Table A3 are used for inference in the test experiments and to report the final results. Pretrained classification checkpoints of these six models, which achieved the best validation accuracy, are available in the GoogleDrive project folder under the directory named BIOSCAN_1M_Insect_checkpoints.
Table A3: The table displays Micro-Average-Top-1 and Macro-Average-Top-1 validation accuracy across 24 experiments conducted using 3 distinct seeds. These experiments encompass varying data sizes (Small, Medium, Large), loss functions (Cross-Entropy, Focal), and pretrained backbone models (ResNet-50, ViT-B/16). The experiments utilize consistent hyperparameters and extend to both Insect-Order and Diptera-Family classification levels.
Dataset	Backbone	Loss Fn	Micro-Top-1	Macro-Top-1
Insect-Order	Diptera-Family	Insect-Order	Diptera-Family
Large	ResNet-50	CE	99.65±0.10	97.30±0.02	86.26±0.30	89.98±0.27
Focal	99.62±0.06	97.15±0.00	84.66±0.21	89.42±0.58
ViT-B/16	CE	99.58±0.21	97.67±0.01	87.36±1.20	91.47±0.31
Focal	99.52±0.27	97.58±0.02	85.80±1.75	91.54±0.21
Medium	ResNet-50	CE	98.98±0.04	96.24±0.05	87.30±1.29	91.24±0.33
Focal	98.85±0.04	95.92±0.04	86.61±0.51	90.64±0.22
ViT-B/16	CE	99.14±0.04	96.74±0.06	88.40±1.17	92.83±0.16
Focal	99.11±0.04	96.55±0.02	86.75±1.46	92.23±0.35
Small	ResNet-50	CE	97.79±0.08	93.23±0.24	87.37±0.56	91.43±0.36
Focal	97.62±0.09	92.57±0.07	86.55±0.60	90.68±0.20
ViT-B/16	CE	98.34±0.10	94.46±0.15	88.74±1.16	92.93±0.33
Focal	98.26±0.03	94.42±0.04	88.61±0.09	92.92±0.16
A6.3 Confusion Matrix
For an in-depth analysis of the performance of models trained under various configurations, we provide detailed Confusion Matrices for the classification experiments conducted at the order and family levels. These experiments were carried out using the model employing ViT-B/16 and the Cross-Entropy loss function. The evaluation was performed on the test set of the Large dataset. You can refer to Figures A1 and A2 for a visual representation of the respective Confusion Matrices.
A6.4 Qualitative analysis
In this section, we provide a qualitative analysis of the performance results from the order classification experiment on the Small dataset. We aim to shed light on the misclassifications made by our model by visually examining some of the misclassified images.
Surprisingly, roughly  $57\%$  of the misclassifications in order-level classification experiments on the Small dataset, using 10,000 test samples, can be traced back to low-quality insect images. This is evident when examining the examples shown in Figure A3, where image quality hampers accurate