SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A4: The table presents the Micro-F1-Score and Macro-F1-Score of our trained models, evaluated on the validation set, and then averaged across different seeds.
Dataset	Backbone	Loss Fn	Micro-F1-Score	Macro-F1-Score
Insect-Order	Diptera-Family	Insect-Order	Diptera-Family
Large	ResNet-50	CE	99.67±0.07	97.44±0.03	87.50±0.04	90.73±0.23
Focal	99.63±0.06	97.28±0.01	84.66±0.87	90.22±0.26
ViT-B/16	CE	99.68±0.06	97.68±0.01	87.94±1.59	92.01±0.12
Focal	99.62±0.14	97.58±0.01	86.98±2.06	91.91±0.22
Medium	ResNet-50	CE	99.00±0.04	96.26±0.06	87.43±1.02	92.61±0.03
Focal	98.88±0.05	95.98±0.05	86.77±1.21	91.77±0.5
ViT-B/16	CE	99.14±0.04	96.75±0.04	89.33±1.21	93.58±0.08
Focal	99.12±0.03	96.56±0.01	87.52±1.01	93.20±0.21
Small	ResNet-50	CE	97.84±0.11	93.27±0.35	87.89±0.77	92.10±0.51
Focal	97.63±0.04	92.78±0.06	87.52±0.62	91.37±0.12
ViT-B/16	CE	98.31±0.10	94.54±0.16	88.92±0.74	93.56±0.23
Focal	98.28±0.03	94.42±0.06	88.36±0.28	93.48±0.05
Figure A1: The Confusion Matrix displays the per-class predictions of the order level classification using the Large dataset of the BIOSCAN-1M Insect dataset. The test evaluation is performed on the best model achieved from validation performance results presented in Table A3.
classification. A similar analysis revealed that approximately  $45\%$  of the misclassifications in order-level experiments with the Large dataset, using 225,660 test samples, were also attributed to low-quality insect images.
Figure A2: The Confusion Matrix displays the per-class predictions of the family level classification using the Large dataset of the BIOSCAN-1M Insect dataset. The test evaluation is performed on the best model achieved from validation performance results presented in Table A3.
Another observation shows that large proportion of misclassifications are the insects belonging to different orders that are all incorrectly classified as one of the dominant classes of our Small dataset. As an example there are  $16.2\%$  of the misclassifications in order-level classification experiments on the Small dataset were insects belonging to different orders are all incorrectly classified as Diptera (flies or mosquitoes), which is the dominant class. This observation, illustrated in Figure A4, highlights specific instances where the model struggles to differentiate between various orders and tends to favour Diptera as the predicted classification.
By examining these qualitative analyses, we gain insights into the challenges faced by our model in correctly classifying insect orders, especially when dealing with low-quality images and distinguishing between similar orders when these orders have low number of training samples.
Our classification experiments have an important application in data cleaning. By identifying low-quality images that have been misclassified, we can effectively detect and remove them from the dataset. This process plays a crucial role in enhancing the overall quality and reliability of the data, as it ensures that only high-quality images of insects are retained.
Furthermore, our classification experiments also enable us to validate the taxonomic classifications performed by human experts. By examining instances of false predictions, we can investigate
whether a sample has been incorrectly annotated, providing valuable insights into the accuracy of the taxonomic classification process.
A6.5 Discussion
A6.5.1 Dataset: Generation, Curation and Growth
The BIOSCAN project is currently in its initial stages, with its primary goal being the facilitation of a global biodiversity assessment. In this section we clarify certain aspects and procedures accomplished in the generation of the BIOSCAN-1M Insect dataset.
All samples of the BIOSCAN-1M Insect dataset were processed at one facility using the same workflows and imaging equipment. This should exclude all potential biases with respect to data collection.
Regarding dataset labeling procedure, the order-level classification is conducted by taxonomists and entomologists primarily relying on morphology rather than barcode matching. For family and finer-grained classifications, a combination of approaches may be employed, often supported by BIN assignment.
The annotators responsible for labeling the data were personnel affiliated with the Centre for Biodiversity Genomics, including technicians and research scientists, all engaged in full-time roles at the institute and receiving equitable compensation. A diverse team of approximately 15 to 20 individuals participated in providing labels for the dataset. These labels were derived from the images, forming the foundation for establishing higher-level taxonomies such as order and family. Additionally, the annotators conducted visual examinations of specimens for finer-grained taxonomic ranks, utilizing primary literature as a reference where feasible. Taxonomic experts (human professionals) engaged in the barcoding process have varying level of involvement, which tends to increase when the placement of a specimen is more contentious. Additionally, these experts are responsible for describing species, a task not handled by the machine. They also supply the reference ID that enables us to establish matches.
Several factors contribute to why most samples of the dataset are classified only to the family and finer-grained classes are not provided, with the primary one being the time constraints associated with completing the assignment. The complexity of the task arises from the fact that relying solely on a barcode is often insufficient due to potential ambiguities, as discussed earlier. Each sample's labeling requires verification through visual inspection (images), or in some cases, examination of the original specimen, before proceeding with further classification. This process is not easily scalable, prompting the adoption of BINs as a species proxy.
However, the process utilized to expand the dataset remains consistent with the methodology employed for the BIOSCAN-1M Insect dataset. The Dataset will be retrained at regular intervals and older versions archive stored on Zenodo and date stamped. Similarly we will use GitHub's releases mechanism to version the accompanying code.
A6.5.2 Application: Model and Tasks
In this article the baseline problems and methods we explored were chosen to be simple and accessible and as a result, limited. They are not the focus of the paper as our primary focus is to release the dataset and showcase its inherent potential. We expect future works will use the dataset for interesting problems such as hierarchical classification, zero-shot classification, set-valued classification and methods that improve performance in the fine-grained and long-tailed label regime.
We believe that the most promising methods will be hierarchical classifiers that yield uncertainty estimates over multiple taxonomic levels. Improving performance on minority classes and reliably delineating novel operational taxonomic units is also important. To get there, we believe the most promising areas of investigation from the ML side will be semi-parametric methods that use reference libraries at test time, set-valued classification as a natural means of expressing uncertainty, and zero-shot classification.
The utilization of the BIOSCAN-1M Insect dataset in conjunction with other large biological datasets from various domains becomes feasible by harnessing the preprocessing module proposed in this paper. By employing tools like our cropping tool and applying machine learning techniques for domain adaptation/generalization, one can capitalize on the capabilities of a pre-trained model on
(1) Original
(2) Original
(2) Cropped
(3) Original
(3) Cropped
(4) Original
(4) Cropped
(1) Cropped
(5) Original
(5) Cropped
(6) Original
(7) Original
(8) Original
(7) Cropped
(8) Cropped
(9) Original
(9) Cropped
(10) Original
(10) Cropped
(11) Original
(11) Cropped
(12) Original
(12) Cropped
Figure A3: Examples of misclassifications caused by low quality images photographed from insects.
Figure A4: Examples misclassified as the dominant class Diptera (flies).
BIOSCAN-1M Insect images to effectively tackle classification challenges in out-of-distribution scenarios.
Overall, we believe that the unique annotation and metadata including the DNA barcodes will prompt interesting multimodal strategies. We intend to enhance our approach by incorporating DNA barcode sequences and utilizing Barcode Index Numbers (BINs). This strategic direction aims to effectively tackle the limitations associated with the current taxonomic labels of the images. Notably, the utilization of BINs holds promise as each image is inherently associated with a distinct and unique BIN.
A7 Preprocessing: Cropping tool
Our observations showed significant improvement in processing time when we used cropped images rather than original ones. However, cropping is a challenging problem since insect images have varying shapes, sizes, colors which is also shown in Figure A5. The illumination and background color and surface are not the same across the original images.
Furthermore, there are cases in the original images that the insect is photographed in pieces and in such cases the cropping is quite challenging especially when the insect is small, and its less discriminative body parts like legs are distant from the main body so these pieces could be cropped instead.
To address these issues more effectively, we have developed a tool based on the DETR model for automatic identification and cropping of the main insects in images. The primary objective of this tool is to facilitate data storage and subsequent research, such as neural network training. The tool uses the DETR model to accurately locate the main insects in images and crop accordingly. By removing irrelevant background information, the tool optimizes storage space and reduces the time spent on data management. Additionally, the cropped images can be effectively used for tasks such as image classification through neural network training, leading to improved performance in the following image classification task. Our crop tool checkpoint is available in the GoogleDrive project folder under the directory named BIOSCAN_1M_Insect_checkpoints.
A7.1 Approach
The cropping tool consists of first detecting a tight bounding box for the insect in the image using an object detector and then cropping the image by extending the bounding box. We show an overview of the cropping tool in Figure A6. To accurately locate the insect in the image, we chose the DETR [9] model which has excellent performance in the task of object detection and the corresponding pretrained ResNet-50 [21] as the feature extractor. At the beginning, the CNN-based feature extractor extracts a set of image features that are fed into a transformer-based encoder-detector. The detector takes a set of learned positional embeddings as object queries and uses them to attend to the encoder
Figure A5: Examples of images used to adapt our cropping tool. We include variations of insects' size, color, position and shape.
Figure A6: Our DETR [9] based cropping tool takes an input image, extracts features using a ResNet50 [21] backbone, and extracts a tight fitting bounding box for the insect (see red box). We then extend the bounding box (see blue box) to obtain the final cropped image. We use a DETR model pretrained on MSCOCO [35]. To fine-tune the DETR model, we annotate a small set of insect images with their segmentation mask.
outputs. Each of the output decoder embeddings is then passed to a shared FFN which will predict whether there is "no object" or a detected object with its class and bounding box. Each bounding box is parameterized as  $(cx, cy, w, h)$  where  $(cx, cy)$  is the center of the bounding box, and  $(w, h)$  is the width and height of the box, all normalized to 1.
The DETR network is trained by optimizing a bipartite set loss that matches detected boxes with the ground-truth boxes using the Hungarian algorithm to minimize the overall matching loss between the matched pairs. The pairwise matching loss is a combination of the classification loss and a box regression loss (the bounding box loss is included only when the detected box matches a ground truth box that corresponds to an object, and is a weighted combination of GIOU [48] and L1 loss between the bounding box parameters). In our case, we have only one object class ("insect") so the classification reduces to a binary classification between "insect" and "no object".
Note that other than the ground-truth bounding box, for training the DETR model of the cropping tool, the pixel mask of the insect in the image is also required for the training. This pixel mask is not needed during the inference phase.
Training details. We start with a DETR model pretrained on MSCOCO [35] and fine-tune it on our dataset. We use the AdamW [38] optimizer with learning rate of 0.0001, weight decay of 0.0001 and a batch size of 8. We train for 10 epochs. On a RTX 2080 Ti with 4 workers, for 1,000 images, training takes 1.5 minutes per epoch and a total of 15 minutes for 10 epochs.
Figure A7: Typical instances of annotated IW (left two columns) and IP (right two columns) images. To obtain an accurate bounding box in reasonable annotation time, we focused on drawing the external outline of the main insect only excluding the small spaces between its legs. Small parts of the insect that are far away from the main body (e.g. the small leg in the first image) are also not included.
The original DETR is trained with images resized to fit within an  $800 \times 1,333$  tensor. We match that and resize our image (preserving the aspect ratio) so that the shortest side is less than 800 and the longest side is less than 1333. No data argumentation is applied during training.
**Cropping.** In the cropping phase, With the predicted bounding box (the red bounding box in Figure A6), we can choose to enlarge it using a certain method to include more details or meet specific image aspect ratio requirements. By default, we will choose 0.4 times the longest edge as the target and extend this size in both height and width to produce the final cropping bounding box (the blue bounding box in Figure A6).
To crop the image, we run our fine-tuned DETR model on the input image to identify the tight bounding box around the insect. We assume that each image contains one insect of interest, and during cropping, we take the predicted bounding box with the highest probability that is higher than 0.5. Before cropping, we extend the predicted bounding box by a fixed ratio  $R = 1.4$  of the size of the tight bounding box. We extend the height and width by the same number of pixels by computing the extended size as: ExtendSize = (R - 1) × max(width, height).
If the bounding box is at the edge of the original image, we pad the image by adding pixels of maximum intensity to match the white background. In this way, even if the predicted bounding box does not encompass all the details of the insect, we can still include the entire insect in the cropped image. Furthermore, this maintains a more square aspect ratio, which facilitates downstream tasks such as image classification.
Runtime. The cropping tool can be run in CPU or GPU mode. On a Linux machine with 16 cores and running 4 workers, using CPU only,  $10\mathrm{k}$  images can be cropped in 2 hours and 40 minutes (images loaded and written to local SSD). Using an RTX 2080 Ti GPU, 10K images can be cropped in 30 minutes on the same machine.
A7.2 Data
We develop our tool on two sets of images of insects that are pinned (INSECTS-PINNED) and insects in wells (INSECTS-WELL). Using the Toronto Annotation Suite (TORAS) [29], we annotate each with their segmentation mask. For each set, we annotated a large (1,000 images) and a small (100-150 images) training set and another small set for evaluation. The annotation was done by three volunteers and took a total of 4 hours for 1,000 images. The two sets of images are described below (see Figures A7 and A8 for example images and annotated masks):
INSECTS-PINNED (IP). The insect is pinned in these images (or has a pin near it) with a fairly clean white background. The images are taken by a Digital SLR camera (Canon) mounted on a motor-drive positioning system (OpenBuilds ACRO) equipped with stepper motors and a motion control system. Pinned specimens are arrayed in sets of 96 ( $8 \times 12$  array) in a large enough distance between them to avoid including parts of neighbouring specimens in the image frame. For this set, we collected 1,000 images to form the large training set (IP-1000-train), 100 images for the small training set (IP-100-train), and another 100 images for the validation set (IP-100-val).
Figure A8: Examples of special annotation cases. Left: for an insect that is broken into multiple parts with even size, we create a mask that covers all of the parts of the insect. The ideal mask should contain minimal background, and keep the edge of the mask as close to the insect's edge as possible (left, right). Middle: for two insects where one is in the container and the other is not, we annotate the insect that is not in the container. Right: for a split insect we annotate all parts.
Table A5: The Average Precision (AP) and Average Recall (AR) were computed on the IW-150 val and IP-100-val datasets using the DETR model, which was pre-trained with different training splits.
Training data	INSECTS-PINNED-100-Val	INSECTS-WELL-150-Val
AP[0.75]	AR[0.50:0.95]	AP[0.75]	AR[0.50:0.95]
IP-100	0.910	0.893	0.543	0.729
IP-1000	0.949	0.918	0.415	0.587
IW-150	0.415	0.587	0.801	0.802
IW-1000	0.665	0.695	0.872	0.835
IP-1000 + IW-1000	0.964	0.907	0.901	0.885
INSECTS-WELL (IW). In these images, the insects are placed in a well. Here the images tend to have a less clean background due to the glass and uneven reflected light. The images are taken using a Keyence VHX-7000 Digital Microscope system with a fully integrated head and automatic stage that permits high-resolution  $(4\mathrm{k})$  microphotography of individual specimens. Because its scanning stage can hold a 96-well plate, the system automatically acquires a high-resolution image of each specimen by controlling movements in the X-Y plane. As well, its capability to control the z-axis position of the stage with a precision of  $0.1\mathrm{m}$  allows it to photograph each specimen at multiple heights before rapidly compiling these images into an in-focus image (depth stacking). For this set, we collected 1,000 images to form the large training set (IW-1000-train), 150 images for the small training set (IW-150-train), and another 150 images for the validation set (IW-150-val).
Note that the BIOSCAN-1M Insect Dataset consists only of insects in wells. We include the insects with pins to extend the usefulness of the cropping tool for a broader spectrum backgrounds that may appear in the process that specimens are acquired in the larger BIOSCAN project.
During annotation, we focus on masking the main insect and we exclude small broken pieces of the insect that are far from its body (see Figure A7). There are also challenging cases where the insect may be broken into pieces or there are multiple insects (see Figure A8). For insects that are broken into multiple pieces of similar size, we create a mask that covers all the pieces. When there are multiple insects, we mask only the central insect.
A7.3 Experiments
A7.3.1 Metrics
The metrics we used are the Average Precision (AP) and the Average Recall (AR) with the IOU of the bounding box equal to [0.75] and [0.50:0.95], as they measure the precision and recall aspects of detection performance. AP reflects the accuracy of detection by considering the overlap between predicted and ground truth bounding boxes, while AR assesses how well the system captures all the ground truth objects.
Figure A9: Cropping examples of images from INSECTS-PINNED (IP) and INSECTS-WELL (IW) with the original image, image with detected bounding boxes in red, extended bounding boxes in blue, and final cropped image.
IP
IW
Figure A10: Examples of imperfect insect detection (IOU  $< 0.85$ ), with ground-truth bounding box in green, detected bounding boxes in red, and extended in blue. In the second image of IP, note that we extend the image with the white background to fit the bounding box that escapes the original image boundaries.
A7.3.2 Cropping results
We show examples of cropped images in Figure A9. The images show the accurate identification of the insect subject by the DETR model (red bounding box) and the extended bounding box (blue bounding box) used for cropping. In Figure A10, we show cases where the predicted bounding boxes have an intersection over union (IoU) with the ground truth bounding boxes (green bounding box) less than 0.85. From these examples, we observe that the antennae of certain insects and the presence of cluttered backgrounds sometimes can create disturbances to our fine-tuned DETR model. However, by expanding the predicted bounding boxes, we are still able to capture all the desired information within the cropped images.
To evaluate the performance of our cropping tool with different amount and type of data, we trained the DETR model with 5 training splits (IP-100, IP-1000, IW-150, IW-1000 and IP-1000+IW-1000), and evaluate these models on two validation splits(IP-100-val and IW-150-val). Overall, from Table A5, we see that using the mixed training split with 1000 images from IP and 1000 images from IW results in the highest accuracy. This is the model that we use for cropping the images in the BIOSCAN-1M Insect Dataset.
Table A6: Comparison of classification accuracy results on original images vs. cropped images. Both are resized to 256 on the smaller dimension. Overall, we find the cropped images yield slightly higher accuracy.
Image type	Order-level	Family-level
Micro-average	Macro-average	Micro-average	Macro-average
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
original	0.9626	0.9970	0.8218	0.9964	0.9248	0.9802	0.9109	0.9730
cropped	0.9786	0.9976	0.8757	0.9980	0.9314	0.9786	0.9154	0.9728
Figure A11: The training loss and Top-1 accuracy on the validation split during the training of family-level classification of images of insects using cropped (blue) and original (orange) images. Both are resized to 256 on the shorter side.
A7.3.3 Insect classification using cropped images
We further evaluate the effectiveness of our auto-cropping tool on a downstream task: insect image classification at the order/family level. In Table A6 we compare the classification performance of the original vs. cropped images on the BIOSCAN small dataset following the training setup we described in the main paper. We use the ResNet-50 backbone with cross-entropy loss and train with the AdamW optimizer with a learning-rate of 0.001 and momentum of 0.9 for 100 epochs for order-level classification and 40 epochs for family-level classification. All images are resized such that the shorter side has size 256. During training, we apply random horizontal flip with probability of 0.5, and random crops of  $224 \times 224$  are extracted and fed into the backbone to extract image features. During inference, the center  $224 \times 224$  crop is extracted. We measure the micro and class macro average top- $K$  accuracy at  $K = 1$  and  $K = 5$ .
From Table A6, we see that in most cases, using cropped images to perform training results in higher classification accuracy. In the cases where original image type outperforms cropped type, the difference is small.
To further compare the difference between using original images and cropped images for training, we also compare the loss curve during training with original and cropped images.
By comparing the loss at epoch 10, 15 and 20, we see that using the cropped images can help the model converge faster. Using the cropped images also yields higher top-1 accuracy on the validation split.