CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
Authors: ZeMing Gong
Austin Wang
Xiaoliang Huo
Joakim Bruslund Haurum
Scott C. Lowe
Graham W. Taylor
Angel X Chang
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Learning
Contrastive Learning
DNA Barcodes
Taxonomic Classification
Fine-grained Classification
Biodiversity Monitoring
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: CLIBD uses a multimodal machine learning approach with images, DNA, and text to enhance insect classification, outperforming traditional methods in identifying known and unknown species without specific fine-tuning.

Abstract: Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multi-modal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.

Published as a conference paper at ICLR 2025
CLIBD: BRIDGING VISION AND GENOMICS FOR BIODIVERSITY MONITORING AT SCALE
ZeMing Gong'
Austin T. Wang' Xiaoliang Huo'
Joakim Bruslund Haurum?
Scott C. Lowe
Graham W. Taylor Angel X. Chang
Simon Fraser University! Aalborg University Vector Institute
University of Guelph*
Alberta Machine Intelligence Institute (Amii)
{amgong, atw7, xiaoliang huo, angelx}esfu: ca, {joha] aau. dk,
(gwtaylor) www.yyllorruuuuelphh.c https://bioscan-nll/ git.... io/e..ooolll.... co/clidd/
ABSTRACT
Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of laxonomic labels in unified em- bedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse barcode DNA and image data. Our method surpasses previous mingle-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.
INTRODUCTION
As environmental change and habitat loss accelerate, monitoring biodiversity is crucial to understand and maintain the health of ecosystems. Taxonomic classification of organisms at scale is especially important for understanding regional biodiversity and studying species interactions.
To assist in this, researchers have used computer vision to identify organisms in images [22, 69, 72, 41] for variety of applications such as ecological monitoring [13]. However, relying solely on images for identifying and classifying organisms fails to consider the rich evolutionary rela- tionship between species and may miss fine-grained differences. To better capture these distinc- tions, researchers have used DNA sequences for genome understanding and taxonomic classifica- tion [31, 79, 45, 8, 2, 54]. In particular, DNA barcodes [26]. short sections of DNA from specific genes such as the mitochondrial COI gene [40] in animals and ITS sequences in fungi, have been shown to be particularly useful for species identification [2, 54]. However, collecting DNA requires specialized equipment making it more expensive and less accessible than images. In this work, investigate whether we can leverage recent advances in multi-modal representation learning [50, 32] use information from DNA barcodes to guide the learning of image embeddings appropriate for taxonomic classification.
Recently, BioCLIP [61] used CLIP-style contrastive learning [50] to align images with common names and taxonomic descriptions to classify plants, animals, and fungi. While they showed that aligning image representations to text can help improve classification, taxonomic labels. which are not always available to the species level, are needed to obtain text descriptions. In this work, we study whether, by aligning to DNA barcodes (instead of text) during pretraining, we can learn im- proved representations of images for use in tasks relevant to biodiversity.
We propose CLIBD, which uses contrastive learning to map taxonomic Jabels, biological images and barcode DNA to the same embedding space. By leveraging DNA barcodes, we eliminate the reliance on manual taxonomic labels (as used for BioCLIP) while still incorporating rich taxonomic information into the representation. This is advantageous since DNA barcodes at scale more readily than taxonomic labels, which require manual inspection from a human ex- pert [23, 24, 60]. We also investigate leveraging partial taxonomic annotations, when available,
be obtained

---

Published as a conference paper at ICLR 2025
to build trimodal latent space that aligns all three modalities for improved representations. We demonstrate the power of using DNA as a signal for aligning image embeddings by conducting ex- periments for fine-gi eneeggraned taxonomic classification down to the species level. Our experiments show
our pretrained embeddings that align modalities can (1) improve on the representational power of image and DNA embeddings alone by obtaining higher taxonomic classification accuracy and (2) provide a bridge from image to DNA to enable image-to-DNA based retrieval.
2 RELATED WORK
We review work using images, DNA, and multi-modal models for fine tne-grained taxonomic classi- fication of species and their application in biology. Prior work has primarily explored building unimodal models for either images or DNA, and largely relied on fine-tuning classifiers known species or higher-level taxa. This limits those approaches to closed set of species, whereas we aim to also identify unseen species, for which we have no examples in the training set.
set of
Taxonomic classification of images in biology. Many studies have explored image-based taxo- nomic classification of organisms [7, 69]. However, visual identification of species remains difficult due to the abundance of fine-grained classes and data imbalance among species. To improve fine- grained taxonomic classification, methods such as coarse and weak supervision [67, 53, 63] and contrastive learning [14, 74] have been developed. Despite these advances, image-based species classification is still limited, so we leverage DNA alongside images to enhance representation learn- ing while maintaining the relative ease of acquiring visual data for new organisms.
Representation learning for DNA. Much work has focused on machine learning for DNA, such as for genome understanding [37, 34, 3, 35]. Recently, self-supervised learning has been used to develop foundation models on DNA, from masked-token prediction with transformers [31. 8, 65, 15, 79, 21. to contrastive learning [80] and next-character prediction with state-space models [47]. While much of this work focuses on human DNA, models have also been trained on large multi- species DNA datasets for taxonomic classification. BERTax [45] pretrained BERT [18] model for hierarchical taxonomic classification but focused on coarser taxa like superkingdom, phylum, and genus, which are easier than fine-grained species classification. BarC [2] showed that models pretrained on DNA barcodes rather than general DNA can be more effective for taxonomic classification. Though some of these works use contrastive learning, they do not align DNA with images. We extend these models by using cross-modal contrastive learning to align DNA and image embeddings, addressing the higher cost of obtaining DNA samples while improving image-based classification and enabling cross-modal queries.
Multimodal models for biology. While most work on taxonomic classification has been limited to single modalities, recent work started developing multimodal models for biological applications [29, 39, 77, 36). Nguyen et al. [48] introduced Insect- IM, applying contrastive learning across text and image modalities. BioCLIP [61] pretrained multimodal contrastive models images and text encodings of taxonomic labels in TrecOfLife- 10M. However, these models focus only on images and texí, limiting their with new species where taxonomic labels are unavailable. They also miss leveraging the rich taxonomic knowledge from sources like the Barcode of Life Datasystem (BOLD), which at the time of writing has nearly 19 M validated DNA barcodes. Although many records include expert-assigned taxonomic labels, only 24% are labeled to the genus level and 9% to the species level in BOLD-derived datasets like BIOSCAN-IM and BIOSCAN-5M [23, 24], By aligning images to DNA barcodes, we can use precise information in the DNA to align the image representations with the task of taxonomic classification, without requiring taxonomic labels.
One of the few works that uses both images and DNA is the Bayesian zero-shot learning (BZSL) approach by Badirli et al. [4]. This method models priors for image-based species classification by relating unseen species to nearby seen species in the DNA embedding space. Badirli et al. [5] similarly apply Bayesian techniques, with ridge regression to map image embeddings to the DNA space to predict genera for unseen species. However, this approach assumes prior knowledge of all genera and does not use taxonomic labels to learn its mapping, limiting its representational power. In this work, we show that aligning image and DNA modalities using end-to-end contrastive learning produces a more accurate model and useful representation space. By incorporating text during pretraining, we can leverage available taxonomic annotations without relying on their abundance.
2

---

Published as a conference paper at ICLR 2025
a) Contrastive Learning
b) Nearest Neighbour Clas sification
Pruds cogetter
Species A Species B
for
for
Image encoder
DNA encoder
fau
fazh
Specias C
foa
feoRf
ATGCCOCAAA ESMATATIAN DAARIGEATT.
DNA encoder
Image e encoder
Orter
Text encoder
Pish spart
Order Hormoottre Gaus sjccso
Figure I: Overview of CLIBD. (a) Our model consists of three encoders for processing images, DNA barcodes, and text. During training, we use a contrastive loss to align the image, DNA, and text embeddings. (h) At inference, we embed a query image and match it to a database of existing image and DNA embeddings (keys). We use cosine similarity to find the closest key embedding and use its taxonomic label to classify the query.
Some prior work has investigated contrastive learning of image and DNA specifically for genetics and histology [64, 75, 44]. These works extend contrastive learning for aligning images and text- introduced by COnVIRT [78] for medical images and popularized by CLIP [50]-to images and human DNA. Here, we focus on multimodal contrastive learning with DNA barcodes and images for taxonomic classification. DNA barcodes (see Appendix D for background) are much shorter (400- 800 base pairs) compared to sequences used in biomedical tasks (tens of thousands of base pairs).
3
METHOD
To align representations of images, DNA barcodes, and textual taxonomic labels, we start with pretrained encoder for each modality and fine-tune them with multimodal contrastive loss, illus- trated in Figure 1. During inference, we use our fine-tuned encoders to extract features for query image and match them against a database of image and DNA embeddings (keys) with known tax- onomic labels. To classify query image, take the taxonomic labels associated with the most similar key. Whilst we can also query against the taxonomic text embeddings, this approach does not work well for taxa that were not during training. In contrast, our model can match queries against embeddings of labelled images and barcodes acquired after training. Thus, images and DNA barcodes comprise a more robust and comprehensive set of records against which query.
3.1 TRAINING
Contrastive learning, We base our approach on a contrastive learning scheme similar to CLIP [50], which uses large-scale pretraining to learn joint embeddings of images and text. In contrastive learn- ing, embeddings for paired specimens are pulled together while non-paired specimens are pushed apart, thus aligning the semantic spaces for cross-modal retrieval. Following prior work [55], we ex- tend CLIP [50] to three modalities by considering each pair of modalities with the NT-Xent loss [58] between two modalities to align their representations. Let matrices V. D. and T represent the batch of fz-normalized embeddings of the image, DNA, and text modalities. The i-th row of each matrix corresponds to the same physical specimen instance, thus rows Vú and D; are image and DNA fea- tures from the same specimen, forming positive pair. Features in different rows V; and Dj. i # j. come from different specimens and are negative pairs. The contrastive loss for pair is
exp (V. D2/T)
exp (V/TT)
I (V llog Σリ= exp (V. Dz/T)'
L (D--V) = log Σ exp (D.V
where 7 is trainable temperature initialized to 0.07 following Radford et al. [50]. The total con- trastive loss for a pair of modalities is the sum over the loss terms for each pairs of specimens,
LVD Σ (L (
==1
wherein we apply the loss symmetrically to normalize over the possible paired embeddings for each modality [78, 55]. We repeat this for each pair of modalities and sum them to obtain the final loss, = LVD LDT + LVT-
Pretrained encoders. For each modality we use pretrained model to initialize our encoders.
Images: ViT-B pretrained on ImageNet-21k and fine-tuned on ImageNet-IK [21]. DNA barcodes:
Loaded vit base patch 16 224 in the timm library.
3

---

Published as a conference paper at ICLR 2025
BIOSCAN-1M
Training
Unlabeled species
916 species
1,088,135 records
1,043,863 records
36,279 records
Level
Classes
Order
16
Key M
Validation query Q
Test query Q
Family
491
916 spocies 1771 species
1772 species
916 species
1771 species
916 species
1772 species
Genus
3441
6350
7645
7123
4878
6952
4878
6448
Species
8355
seen
val unseen unsenn
seen
unseen
seen
unseen
Figure 2: Data partitioning. We split the BIOSCAN-IM data into training. validation, and test partitions. The training set (used for contrastive learning) has records without any species labels as well a set of seen species that are well-represented (at least records per species). The validation and test sets include seen unseen (not seen during training) species. These images are further split into subpartitions of queriex (darker color) and keys (lighter color) for evaluation. We ensure that the validation and test sets have different unseen species. Since the seen species common, have a shared set of records (Key) that we keys for seen species and combine key sets form reference database. We show the number records each box.
BarcodeBERT [2] with 5-mer tokenization, pretrained using masked language modelling on 893 DNA barcodes [19] a dataset that is similar to but not overlapping with the DNA barcodes in the BIOSCAN- IM dataset, making it ideal for our study. Tex:: we use the pretrained BERT-Small [68] for taxonomic labels. Besides these pretrained encoders, also conduct experiments using larger variants (see Appendix B.1.3).
3.2 INFERENCE
To use the model to predict taxonomic labels, we calculate the cosine similarity between the em- bedded input image (query) and reference image DNA embeddings (keys) sampled from available species. We use the taxonomic label (order, family, genus, species) associated with the closest key as prediction. This method allows us evaluate the model in zero-shot setting on species which were not seen by the model during training, provided we have appropriately labelled specimens to use as keys. The embedding space also provides the flexibility be used for other downstream tasks, such as supervised classifier or a Bayesian model [4, 5].
TASK AND DATA
To evaluate our method, we perform taxonomic classification using different combinations of input and reference modalities. The input may be biological image or DNA sequence; this matched against a reference set of labelled DNA barcodes, labelled biological images, or known taxonomic labels. We evaluate predictions at each taxonomic level by averaging accuracy over samples (micro) and taxon groups (macro). Unlike fine-tuning species classification head, our approach can iden- tify unseen species using labelled reference images or DNA, without needing to know all potential species at training time. We split the BIOSCAN- 1M dataset some species are "unseen" during training and report prediction accuracy for both seen and unseen species to study model general- ization. This simulates the case where once model is deployed, there new specimens that catalogued and labelled over time, which were not initially available for training the encoders.
Dataset. The BIOSCAN-IM dataset [23] is curated collection of over one million insect data records sourced from biodiversity monitoring workflow. Each record in the dataset includes high-quality insect image, expert-annotated taxonomic label, and a DNA barcode. However, the dataset has incomplete taxonomic labels, with fewer than 10% of records labelled at the species level. This poses a challenge for conventional supervised methods, which require comprehensive species-level annotations, but our method is able to flexibly leverage partial or missing taxonomic information during contrastive learning- The dataset also possesses long-tailed class imbalance, typical of real-world biological data. Given the vast biodiversity of insects- which estimated 80% is as-yet undescribed [62]-and the necessity to discern subtle visual differences, this dataset offers significant challenge and opportunity for model.
Data partitioning. We split BIOSCAN-IM into train/val/test sets to evaluate zero-shot classifica- tion and model generalization to unseen species. Records for well-represented species (at least records) are partitioned at 80/20 ratio into seen and unseen, with seen records allocated to each of the splits and unseen records allocated to val and test. All records without species labels are used in
4

---

Published as a conference paper at ICLR 2025
Table 1: Top-1 macr-accuurac (%) BIOSCAN-IME fest set for different combinations of modality alignment (image, DNA, text) during contrastive training. Results using DNA-to-DNA, image-to- to-gge.. and image-to- DNA query and key combinations. cross-modal alignment and unimodal training using SimCLR (V*). We report the accuracy for seen and unseen species, and their harmonic mean (H.M.) (bold: highest acc, italic: second highest acc.).
baseline, we also show results for unimodal pretrained models before
Aligned embeddings
DNA-to-DNN
Image-to Image
Image-to Moggett--DAA
Taxa
Img
DNA
Txt
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Order
78.8
91.8
84.8
54.9
51.2
7.7
9.6
8.5
69.5
59.8
64.3
99.6
97.4
98.5
v
100.0
100.0
100.0
89.5
97.6
93.4
99.7
71.8
83.5
100.0
100.0
100.0
99.7
94.4
97.0
99.4
88.5
93.6
Family
86.2
82.1
84.1
28.1
21.7
24.5
0.5
0.8
0.6
43.6
30.9
36.2
90.7
76.7
83.1
99.1
97.6
98.3
89.1
81./
84.9
90.2
44.6
59.7
100.0
98.3
99.1
90.9
81.8
86.1
90.8
50.1
64.6
Genus
82.1
69.4
75.2
14.2
10.3
11.9
0.2
0.0
0.0
24.7
16.7
19.9
72.1
49.6
58.8
97.7
93.0
95.3
74.1 I
59.7
66.1
73.4
18.7
29.8
98.2
94.7
96.4
74.6
60.4
66.8
70.6
20.8
32.1
Species
X
76.4
63.6
69.4
7.2
5.0
5,9
0.1
0.0
0.0
x
14.2
8.8
10.9
36
54.2
33.6
41.5
94.4
86.9
90.5
59.2
45,1
51.2
58.1
7.7
13.6
95.6
90.4
92.9
59.3
45.0
51.2
51.6
8.6
14.7
contrastive pretraining, and species with 2 to 8 records are divided between the unseen splits in the val and test sets, Importantly. we ensure that unseen species are mutually exclusive between the val and test sets and do not overlap with seen species for labelled records. Finally, among each of the seen and unseen sub-splits within the val and test sets, we allocate equal proportions of records as queries, to used as inputs during evaluation, and keys, to be used as our reference database. Note that keys from seen and unseen species are combined to form the reference database. This procedure ensures we have a distribution of different cases in evaluation: both seen and unseen, and common and uncommon species. See Figure 2 for split statistics and Appendix A for details.
Data preprocessing. During inference, we resize images to 256x256 and apply 224x224 center crop. For the DNA input, following Arias al. [2], set maximum length of 660 for each sequence and tokenize the input into non-overlapping 5-mers. Similar to Stevens et al. [61], we con- catenate the taxonomic levels of the insects together as text input. As did not have the common names of each record, we used the Linnean order, family, genus, and species, up to the most specific available label per specimen. With this approach, the higher-level taxonomy, even some records do not have species-level annotations.
still provide the model with knowledge of
5 EXPERIMENTS
We evaluate the model's ability to retrieve correct taxonomic labels using images and DNA bar- codes from the BIOSCAN-IM dataset [23]. This includes species that were either seen or unseen during contrastive learning. We also experiment on the INSECT dataset [4] for Bayesian zero-shot learning (BZSL. species-level image classification. We report the top-1 accuracy for the seen and unseen splits, as well as their harmonic mean (H.M.). We focus on evaluating the model using var- ious combinations of modalities on the test set. Specifically, we assess the model's performance when using images and DNA barcodes as inputs, matched against their respective image and DNA reference sets, as well as the combination of image inputs matched to DNA references. In addition, visualize the attention roll-out of the vision transformer used as image encoder to explore how the representation changes before and after contrastive learning and how aligning with different modalities affects the focus of the image encoder.
5

---

Published as a conference paper at ICLR 2025
Original
DNA to DNA
Image to Image
Image to DNA
Caenocholax
sp
10
DPM-2010
Micromus subanticus
Phylionarycter anderidae
Glyptatermes barbouri
Figure 3: Example query-key pairs. Top-3 nearest specimens from the unseen validation-key dataset retrieved aage-to-image, and image-to to---ot-DANAA retrieval. Box color
based on the cosine- similarril for DNA-to-DNA, indicates whether the retrieved samples had the same species (green), genus (light-green), family (yellow), order (orange) as the query or else not matched (red).
Implementation details. Models were trained on four 80GB A100 GPUs for 50 epochs with batch size 2000, using the Adam optimizer [33] and one-cycle learning rate schedule [57] with learning rate from 1e-6 6 to 5e-5. For efficient training, we use automatic mixed precision (AMP). We study the impact of AMP and batch size in Appendix C.
5.1 RETRIEVAL BY IMAGE AND DNA
We conducted experiments on BIOSCAN-IM, [23] to study whether the accuracy of taxonomic classification improves with contrastive learning, particularly with the inclusion of DNA barcodes as an additional modality. We compare the accuracy models trained to align different combinations of modalities: image (I), DNA (D), and text (T). We also assess performance by using different modalities as the query (input at inference time) and as the key (the embedding used for matching). As image more readily available, we primarily focus on querying by image.
Taxonomic classification. In Table 1, we report the top-1 macro- accuracy on our BIOSCAN- IM test for seen and unseen species (see Table for the top-1 accuraccc........... We report the perfor- mance of the different alignment models at different taxonomic levels (order, family, genus, species). As expected, the performance drops for more specific taxa (e-g., accuracy for order is much higher than for species), due to both the increased number of possible labels and the more fine- tne-grained differences between them. When we consider unseen species, there is a drop in accuracy compared to seen species, suggesting there is benefit to retraining encoders as more data is collected.
Are multimodal aligned embeddings useful? Our experiments show that by using contrastive learn- ing to align images and DNA barcodes, can 1) enable cross-modal querying and 2) improve the accuracy of our retrieval-baan based classifier. Unsurprisingly, we find that DNA-to-DNe retrieval is the most accurate, especially for species-level classification. Note that while direct DNA matching using methods like BLAST are quite effective, they tend to be slow and cannot be easily incorpo- rated into neural networks for other downstream tasks [2]. By using contrastive learning align different modalities, we enhance the image representation ' ability to classify (image-to-image) es- pecially the genus and species level where the macro H.M. accuracy jumps from 12.5% 69% (for genus) and 6.27% to 52% (for species) for our best model (I+D+T), Note that with alignment, the DNA-to-DNA retrieval performance also improves. To verify that the improvements in uni- modal performance come from the flow of information between modalities, we also use unimodal contrastive loss following SimCI. [11] to fine-tune the image encoder on BIOSCAN- MM. Results (Table 1, second row) show that while the performance is better than the unaligned model, it is far from the gain we see in the I+D model (see Appendix I for details).
Do DNA barcodes provide strong alignment target? Table 1 also shows that BIOSCAN-IM, using DNA provides a better alignment target than using taxonomic labels. Comparing the model that aligns image and text (I+T, row 3) the that aligns image and DNA (I+D, row 4), that the H+D model consistently gives higher accuracy than the I+T model. At the species level, the I+D outperform the I+D+T model. This is likely because the I+D+T model, despite using

---

Published as a conference paper at ICLR 2025
Table 2: Species-level top-1 macro-accuracy (%) of BioCLIP and our CLIBD model the test set, matching image embeddings (queries) against embeddings of different modalities for retrieval (image, DNA, and text keys). Note: the BioCLIP model [61] was trained on data that included BIOSCAN-IM but used different species splits, so it may have seen most of the unseen species during its training.
Aligned embeddings
Imagee
Image-to Lo-DAA
Image-to-Tees
Model
Img
DNA
Txt
Seen
Unseen
H.M.
Seen Unseen
H.M.
Seen
Unseen
H.M.
BioCLIP
20.4
14.8
17.1
4.2
3.1
3.6
CLIBD
54.2
33.6
41.5
57.6
4.6
8.5
CLIBD
59.3
45.0
51.2
51.6
8.6
14.7
56.0
4,8
8.9
all known labels for each sample, only has access to species labels for 3.36% of the pretraining data. While comprehensive species labels could yield a comparable model for taxonomic classification, the money and time cost for hiring expert scientific annotators makes it impractical to scale.
Cross-modal retrieval. Next we consider cross-modal retrieval performance from image to DNA. Without any alignment, image-to-DNA performance is effectively at chance accuracy. scoring tremely low for levels more gaue-grained than order. By using contrastive learning to align image to DNA, we improve performance at all taxonomic ranks. While the cross-modal performance is still low compared to within- modall retrieval, we see that it is feasible to perform image-to- to togge-to-DNA retrieval, which unlocks the ability to classify taxa for which no images exist in reference databases.
Retrieval examples. Figure 3 shows examples of intra-modality image and DNA retrieval as well as image-to- DNA retrieval from full model (aligning I+D+T), for which the retrieval is successful if the taxonomy of the retrieved key matches the maage's. These examples show significant similarity between query and retrieved images across taxa, suggesting effective DNA and image embedding alignment despite differences in insect orientation and placement.
5.2 COMPARISON WITH BIOCLIP
Next compare our aligned embedding space with that of BioCLIP [61] and investigate how well using taxonomic labels as keys would perform. We run experiments on BIOSCAN-IM by adapting the BioCLIP zero-shot learning demo script to perform species-level image classification. We use the BioCLIP pretrained model on the BIOSCAN-IM test set, with image query and either image or text embeddings as keys. For the text input for BioCLIP, we combined the four concatenated taxonomic levels with their provided openai.templat es as text input, while for CLIBD, we used the concatenated labels only.
In Table 2, we report the seecies-level macro-accuracy. Results on other taxonomic levels (Table 13) follow similar trend. We see CLIBD consistently outperforms BioCLIP, regardless of whether images or text is used as the key, and even for CLIBD trained only on images and text. Since BioCLIP was trained on a much broader dataset, including but not limited to BIOSCAN-IM, it may perform worse on insects as it was also trained on non-insect domains. CLIBD can also leverage DNA features during inference, while BioCLIP is limited to image and text modalities.
Does matching images to taxonomic labels work better than matching to image DNA embeddings? The performance when using text as keys is much lower than using image or DNA keys. This shows that it is more useful labeled samples with images (most preferred) DNA. Nevertheless, if no such samples are available, it is possible to directly use text labels as matching keys.
5.3 ANALYSIS
How does class size infiuence performance? Since we use retrieval for taxonomic classification, it expected that performance is linked to the number of records the key set. Figure confirms this. In general, accuracy is higher for seen species compared to unseen species in cross-modal retrieval, but this difference is less noticeable in modiaity retrieval. This suggests that contrastive training has better aligned the data it has been trained on, but is less effective for unseen species. The DNA-DNA retrieval performance remains high, regardless of number of records the key set.
Attention visualization. To investigate how the contrastive training changed the model, vi- sualize the attention roll-out of the vision transformer for the image encoder [1] in Figure 5. We reference the implementation method mentioned in Dosovitskiy al. [21] by registering forward

---

Published as a conference paper at ICLR 2025
DMV DNA
Seen Spockss Unoeer Spedis
Number of records per species in the key set
Figure 4: Average top-1 per-species accuracy, binned by count of species records in the key set, for different query and key combinations. We show in blue and unseen in orange, with size and color intensity indicating number of binned species. all cases, the accuracy for seen species increases with the number records. While trend is similar unseen species with mntra-modal retrieval (a) and d)), cross-modal retrieval (b) and c)) achieve much lower performance, underscoring the challenge of cross-modal alignment.
Before
After
Input
None
I+T
I+D
I+D+T
Input
None
I+T
I+D
I+D+T
X
Sewaalii
Eake
Radaybli
Milenkila
Rochropdd
Rachrands
Hastepeds
Ridtopaes
Lepaecer
Choaoea
Cálmeess
X
martywood
Alaghu cannd
Lepikgrac
pakkia
Macrastt
pila
Seen
Unseen
Figure 5: We visualize the attention for queries from seen and unseen species. The "Before" and "After" columns indicate if the prediction (at the species-level) was correct before (initial unaligned model) and after alignment (I+D+T model). Predicted genera + species are indicated in green correct, red incorrect. Only a few samples were predicted correctly before alignment and incorrect after (38 for seen and 69 for unseen).
hooks the VIT's attention blocks to capture attention outputs and using the Rollout method to calculate attention accumulation. We then apply the processed mask to the original image to gener- ate an attention map of the image area. Inspired by Darcet al. [16], we inspect the mask of each attention block and remove attention maps containing artifacts. Ultimately,
select the forward outputs of the second to sixth attention blocks to generate the attention map.
We show examples for both seen and unseen species. For examples where the aligned models able to predict correctly, we see that the attention more clearly focused on the insect. Furthermore, that only the I+D+T model activates highly the full insect, compared I+T and I+D. Thus, while quantitatively the models may perform comparably, the I+D+T yields the most visually interpretable predictions. This is ideal for biodiversity, interpretability is important for practical adoption. We also visualize the embedding space before and after alignment (see Appendix B.3) and show more examples of attention visualization (see Appendix B.4).
5.4 IMPROVING CROSS-MODAL CLASSIFICATION
We now more closely investigate how we can improve cross-modal ZS classification, where during inference time we have an image of an insect as a query, and we have a database seen species (with images and DNA), and unseen species (with just DNA). Badirli et al. [4] proposed hierarchical Bayesian model to classify images, using training images to learn the distribution priors and DNA embeddings to build surrogate priors for unseen classes. Here, consider similar zero-shot (7S)
8

---

Published as a conference paper at ICLR 2025
a) Nearest neighbor
Seen Species
firfa furfa fart
Myrmica spectoides (seen)
Unseen Species
fo
fi
Myrmica specioides (unseen)
Image encoder
fa
ESAATATIAA SAAKTECATT...
DNA encoder
fi2
fiaf
Query
fig
figf
fg Y1
Fabse
or
Image encader
Image
Supervised classifier
encoder
Y2
Max
b) Supervised
Figure 6: When have DNA barcodes but not images of unseen species, image and DNA as key sets. adapt CLIBD using images as keys for seen species DNA for unseen species (i.e. the IS+DU strategy) to predict the species. We first classify the input image query against seen species, using either: (a) an I-NN approach thresholding the cosine similarity score 8 = maX& fua fa: or (b) supervised classifier predicting over all seen species and thresholding the maximum softmax probability max yu by threshold If s
combination of
we subsequently query with the image feature fa using I-NN with DNA keys fix of the unseen species and predict the unseen species the closest DNA feature. "During supervised classifier training, the image encoder is finetuned only for use in the supervised pipeline.
Table 3:
BIOSCAN-IM test set using the Image+DNA+ model with image query. We compare nearest neighbour using only DNA keys (NN DNA), vs. strategies Image key for seen and DNA key for Unseen, either NN supervised linear classifier. We also compare against
Top-1 accuracy (%)
BZSL [4] with our embeddings.
Micro top-1 acc
Macro top-1 acc
Taxa
Method
Strategy
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Genus
NN
DNA
87.6
54.9
67.5
70.6
20.8
32.1
NN
IS+DU
85.7
55.0
67.0
66.8
20.8
31.7
Linear
IS+DU
83.6
55.6
66.8
61.4
21.1
31.5
BZSL
IS+DU
86.8
46.5
60.6
75.7
14.4
24.2
Species
NN
DNA
74.2
27.8
40.4
51.6
8.6
14.7
NN
IS+DU
76.1
26.2
39.0
54.8
8.5
14.8
Linear
IS+DU
72.6
25.5
37.7
41.6
9.4
15.3
BZSL
IS+DU
76.1
17.6
28.5
62.6
7.2
12.9
setting using our embeddings from our pretrained model for Bayesian zero-shot learning (BZSL), demonstrating its utility for unseen species classification.
We also consider a simpler strategy using the image embeddings for seen species, and DNA embed- dings for unseen species. A two-stage approach first determines if new image query represents seen or unseen species (see Figure 6). For seen, image-to-image matching determines the species, while for unseen, image-to-DNA matching (assuming reference set of labeled DNA samples for unseen species) determines the species. This is denoted by "IS-DU" (image seen DNA unseen).
unseen. We frame the problem as open-set recognition task [70] by using a classifier to determine whether an image query corresponds to a seen or an unseen species. This is useful for novel species detection in practice may not have reference set of labeled samples or even set of species labels for unseen species. We compare using a 1-nearest neighbor (NN) classifier, and linear supervised classifier with fine-tuned image encoder (see Figure 6 left). See Appendix B.2 for details. We show that without access to any unseen samples at inference time, our learned embeddings can be used distinguish between seen and unseen in Table 15. Using the amage-to-iman NN classifier, we obtain 83% accuracy seen, 77% unseen and a harmonic mean of 80%. For our linear classifier, we obtain lower accuracy on seen (73%). but higher on unseen (85%), with harmonic mean of 79%.
Determining
Evaluation on BIOSCAN-IM. Next we conduct experiments on BIOSCAN-IM to compare our IS+DU strategy vs. incorporating our learned embeddings in BZSL. We also compare against query- ing the seen and unseen DNA keys using I-NN directly. Table reports the top-1 micro and macro accuracy of at the genus and species level (see Table 16 for order and family level classification). We find that at the genus and species level, BZSL obtains good performance for seen species but that simple NN-based approach actually outperforms BZSL on unseen species. Using our IS-DU strategy with the supervised linear classifier, we obtain the best macro top-1 accuracy on unseen species, demonstrating that the complexity of BZSL. may not be necessary.

---

Published as a conference paper at ICLR 2025
Table 4: Macro-accuracy (%) for species classification in Bayesian zero-shot learning task the INSECT dataset. We compare our CLIBD-D with several DNA encoders: CNN encoder [4], DNABERT-2 (79), Bar- CodeBERT [2]. The baseline image encoder ResNet-101 101 used in Badirli et al. [4] is compared against our image encoder before (ViT-B) and after (CLIBD-I) pretraining BIOSCAN- IM (BS-IM). We indicate the pretrain- ing for DNA (Pre-DNA) the multi-species (M.S.) from Zhou et al. [79]. anthropods from Arias et al. [2], or BS-IM. We compare models both with and without supervised fine- tuninu (FT) for each encoder, except for CLIBD-D. D. where the comparison is with or without contrastive learning for tne-tuning on the INSECT dataset. We highlight the baseline from Badirli et al. [4] and the variant with our tneettned encoders in gray.
Data sources
Species-level acc (%)
DNA enc.
Image enc.
Pre-DNA
FT-DNA
FT-Img
Scen
Unseen
H.M.
CNN encoder
RN-101
INSECT
38.3
20.8
27.0
DNABERT-2
RN-101
M.S.
36.2
10.4
16.2
DNABERT-2
RN-101
M.S.
INSECT
30.8
8.6
13,4
BERT
RN-101
Arthro
38.4
16.5
23.1
BarcodeBERT
RN-101
Arthro
INSECT
37.3
20.8
26.7
BarcodeBERT
ViT-B
Arthro
INSECT
42.4
23.5
30.2
BarcodeBEKT
ViT-B
Arthro
INSECT
INSECT
54.1
20.1
29.3
CNN encoder
CLIBD-I
INSECT
37.7
16.0
22.5
BarcodeBEKT
CLIBD-I
Arthro
INSECT
52.0
21.6
30.6
BarT
CLIBD-I
Arthro
INSECT
INSECT
48.5
23.0
31.2
CLIBD-D
RN-101
BS-IM
34.7
21.3
26.4
CLIBD-D
RN-101
BS-IM
INSECT
32.8
25.0
28.4
CLIBD-D
CLIBD-I
BS-IM
34.2
22.1
26.9
CLIBD-D
CLIBD-I
BS-
INSECT INSECT
57.9
25.1
35.0
Evaluation on INSECT dataset with BZSL In this experiment, we study whether our image and DNA embeddings can improve performance when incorporated into BZSL. We evaluate on the INSECT dataset [4], which contains 21,212 pairs of insect images and DNA barcodes from 1 1,213 species. We record the performance of different combinations of encoders. We start with the original CNN-based DNA encoder and ResNet-100 101 image encoder pretrained on ImageNet-IK IK from Badirli et al. [4]. We also consider ViT-B [21], pretrained on ImageNet-21k k and fine-tuned ImageNet- 1k. For DNA encoders, we compare against BERT [2] (which was used to initialize our CLIBD model), and DNABERT-2 [79], BERT-based model trained on multi-species DNA.
Table 4 shows that incorporating CLJBD improves R/SL accuraacy With the baseline image encoder (RN-101), CLIBD-D performs the best of the DNA encoders (both with and without DNA fine- tuning), With BarcodeBERT, our CLIBD-I outperforms both RN-101 and VĪT-B. CLIBD-I together with CLIBD-D gives the best performance with accuracy after fine-tuning of 57.9% for seen and 25.1% for unseen. This shows the benefits of using CLIBD to learn a shared embedding space relating image and DNA data, both in performance and flexibility of applying to downstream tasks.
6 CONCLUSION
We introduced CLIBD, an approach for integrating biological images with DNA barcodes and tax- onomic labels improve taxonomic classification. It uses contrastive learning align embeddings in a shared latent space. Our experiments show that, using barcode DNA as an alignment target for image representations, CLIBD outperforms models that only align images and text. Pretrain- ing with text-encoded taxonomic labels, even when scarce at fine-grained levels, further improves performance. We further demonstrate the effectiveness of our aligned embeddings in zero-shot image-to-DNA retrieval. Our BIOSCAN-IA IM and INSECT dataset experiments showcase the value of using barcode DNA as an alignment target. However, the availability of barcode DNA is limited (see Appendix D for discussion) compared to human-curated image datasets with taxonomic labels such as TreeOfLife- 10M [61] and BioTrove [76]. would be interesting to couple the rich infor- mation found in DNA with these datasets that include natural images, and go beyond insects. It is also possible to investigate other multi-modal learning schemes beyond CLIP. Overall, we believe that using DNA barcodes for improved representation of organism images is promising avenue for future research. We hope that our work can stimulate the application of multimodal machine learning with barcode DNA to help build improved automated tools for biodiversity monitoring.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We acknowledge the support of the Government of Canada's New Frontiers in Research Fund (NFRF) INFRFT-2020-00003]. AXC and GWT are also supported by Canada CIFAR Al Chair grants. JBH is supported by the Pioneer Centre for Al (DNRF grant number PI). This research was enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca) We also thank Mrinal Goshalia and Kian Hosseinkhani for testing the code, Han- Han-Hung Lee and Yiming Zhang for helpful discussions, and Manolis Savva for proofreading and feedback.
REFERENCES
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020. 100110.1853//1/2020000 main.385.
[2] Pablo Millan Arias, Niousha Sadjadi, Monireh Safari, ZeMing Gong, Austin T. Wang, Scott C. Lowe, Joakim Bruslund Haurum, Iuliia Zarubiieva, Dirk Steinke, Lila Kari, Angel X. Chang, and Graham W. Taylor. BarcodeBERT: Transformers for biodiversity analysis. arXiv preprint arXiv:2311.02401, 2023. doi: 100.044850//axxi22311.1.. 1.240.0
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Trabska-Barwinge Kyle Taylor, Yannis Assacl, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 196-1203, Oct 2021. ISSN 1548-7105. 102.0..038/44452-00011 -012222x..
[4] Sarkhan Badirli, Zeynep Akata, George Mohler, Christine Picard, and Mehmet M Dun- dar. Fine-grained zero-shot learning with DNA side information. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neu- Information Processing Systems, volume 34, pp. 19352-19362. Curran Associates, Inc., neurips.....p.....p............. ca/ ca/paper Files/ Lales//pperr222//E1iee. Faw.-a33---aaappeeoo-
2021. URL
[5] Sarkhan Badirli, Christine Johanna Picard, George Mohler, Frannie Richert, Zeynep Akata, and Murat Dundar. Classifying the unknown: Insect identification with deep hierarchical Bayesian learning. Methods in Ecology and Evolution, 1553 2023. 101:10.1111/2041-211 14104.
[6] Dennis A Benson, Mark Cavanaugh, Karen Clark, Ilene Mizzah-Mizzachi, David J Lipman, James Ostell, and Eric W Sayers. Genbank. Nucleic acids research, 41(D1):D33 D42, 2012. doi:10. 101.00.///////gaaabl//
[7] Thomas Berg, Jiongxin Liu, Scung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Bel- humeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2019-2026, 2014, doi: 109/CVPR.0014.25
[8] Samuel Cahyawijaya, Tiezheng Yu, Zihan Liu, Xiaopu Zhou, Tze Wing Tiffany Mak, Yuk Yu Nancy Tp. and Pascale Fung. SNP2Vec: Scalable self-supervised pre-training for genome-wide association study. In Dina Demner-Fushman Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (eds.), Proceed- ings of the 21 Workshop on Biomedical Language Processing, pp. 140-154, Dublin, Ireland, May 2022. Association for Computational Linguisties. d0.: 18.5300..66/vv111 bionlp- 1.14.
[9] Xiaowei Cao, Jie Liu, Jian Chen, Guo Zheng, Matjaž Kuntner, and Ingi Agnarsson. Rapid dissemination of taxonomic discoveries based on DNA barcoding and morphology. Scientific Reports, 6(1):37066, 2016. doi: 1033//sepp30066.
[10] Shilin Chen, Xianmei Yin, Jianping Han, Wei Sun, Hui Yao, Jingyuan Song, and Xiwen Li. DNA bar- coding in herbal medicine: Retrospective and prospective. Journal of Pharmaceutical Analysis, 13(5); 431 441, 2023. doi: 10. ho1.///.pha..23........
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In International conference on machine learning, pp. 1597- mlr. 1607. PMLR, 2020. URL bectps://proceedie htt.:/eeei.inn.......rssssssss.. htteee/s/i11//cce220/eceeee pdf.
[12] Zhentao Cheng, Qiang Li, Jun Deng, Qian Liu, and Xiaolei Huang. The devil is in the details: Problems in DNA barcoding practices indicated by systematic evaluation of insect barcodes. Frontiers in Ecology and Evolution, 11:1149839, 2023. 1t..1..8fffffo..222.... 149839.
11

---

Published as a conference paper at ICLR 2025
[13] Sylvain Christin, Éric Hervet, and Nicolas Lecomte. Applications for deep learning in ecology. Methods in Ecology and Evolution, 10(10):63221166 2019, 1et.001111/2041-11 21266.1356..
[14] Elijah Cole, Xuan Yang. Kimberly Wilher, Oisin Mac Aodha, and Serge Belongie. When does contrastive visual representation learning work? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14755-14764, 2022. 200.00...090CCPP55288222222222.0
[15] Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Hen- ryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, pp. 2023 01, 2023. doi: 1111/2023.0111152222233
[16] Timothéc Dareet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need regis- ters. In International Conference Learning Representations, 2024. URL https::/ net/ / forum? 2id22:id22d::.....
[17] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hier- archical image database. In IEEE conference on computer vision and pattern recognition, pp. 248-255. IEFE, 2009. doi:10. P00./CCVR..200.20666444
[18] Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings the 2019 Conference of the North American Chopter the Association for Compu- tational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, Jun 2019. Association for Computational Linguistics. 1oi:10.18653/v 186318663/ 1/N19 1423.
[19] Jeremy R DeWaard, Sujeevan Ratnasingham, Evgeny V Zakharov, Alex Borisenko, Dirk Steinke, An- gela C Telfer, Kate HJ Perez, Jayme E Sones, Monica R Young, Valerie Levesque- Beaudin, et al. A reference library for Canadian invertebrates with 1.5 million barcodes, voucher specimens, and DNA samples. Scientific data, 6(1):308, 2019. doi: 1ete1100113/4411 1038/$41597-019-0333
[20] Angela Di Pinto, Pietro Di Pinto. Valentina Terio Giancarlo Bozzo, Elisabetta Bonerba. Edmondo Ceci. and Giuseppina Tantillo. DNA barcoding for detecting market substitution salted cod fillets and battered cod chunks. Food chemistry, 141(3):1557-1765 2013, doi: 10. 1wt.10016...o...... foodchem.200000 1....55099
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Interna- tional Conference on Learning Representations, 2021. URL https: openreve ew. net/ forum? caFYicbFdNi
[22] Camille Garein, Alexis Joly, Pierre Bonnet, Antoine Affouard, Jean-Christophe Lombardo, Mathias Chouet, Maximilien Servajean, Titouan Lorieul, and Joseph Salmon. Pl@ntNet- 300K: a plant image dataset with high label ambiguity and a long-tailed distribution. In J. Vanschoren and S. Yeung (eds.). Proceedings of the Neural Information Processing Sys- tems Track on Datasets and Benchmarks, volume 1. Curran Associates, Inc., 2021. URL proceedi ngs. neurips. wwtprips.cc/ppprr.... solfffflepppppee 1e115571112220 ael2abbb336abab3 (atece6766aaa77544 fb61 7a Papa-appr--ae round2.pdf.
https:/ / /tt.s//aaasstsssaeeeeeeaa....sea wwweeettseeeeeebarbeeeeeeee benchmarrks--prreedd
[23] Zahra Gharace ZeMing Gong, Nicholas Pellegrino, Iuliia Zarubiieva, Joakim Bruslund Hau- rum, Scott Lowe, Jaclyn McKcown, Chris Ho, Joschka McLcod, Yi-Yun Wei, Jireh Agda, Su- jeevan Ratnasingham, Dirk Steinke, Angel Chang, Graham W Taylor, and Paul Fieguth. A step towards worldwide biodiversity assessment: The BIOSCAN-IM insect dataset. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 43593-43619, Curran Associates, Inc., 2023. URL https:/ htttp:://proceedings wwwpeeedinnnn.nnnineeeeeeeeeeeee www.:eewrr..pppppeeeeeeeeeeeeeee riles wwwessppppees htt.eee/pppee/e22//. file// 8 3abbddeaa68a
Fatp:///bccc--p----attteeeeeeee- and Benchnarks pdt.
[24] Zahra Gharaee Scott C. Lowe, ZeMing Gong, Pablo Millan Arias, Nicholas Pellegrino, Austin T. Wang, Joakim Bruslund Haurum, Iuliia Zarubiieva, Lila Kari, Dirk Steinke, Graham W. Taylor, Paul Fieguth, and Angel X. Chang. BIOSCAN-5M: A multimodal dataset for insect biodiversity. In A. Globerson, Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 36285-36313. Curran Associates, Inc., 2024. URL https: /tpps://pree prw.eedingss neurips ce paperr www.ffiesspppoe stlles/paper/222/ /file// 3 3fbb44721 3041 cSece @alaceef440 F04c20c2bl esa Paper Datasets Benchmarks_nnne pdf.
12

---

Published as a conference paper at ICLR 2025
[25] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh. Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15180-15190, Jun 2023. URL https://openaccess theevf. f. con/ wtt.//otn/ttteeeeee content Coontent//veeee /CVPR2023 20123//tmml /tth//h//////////// ImageBind_Cnss
Tee_rTo___i____ee Then All 1a11__VP______ 2023 _apeerrpppp_____ html.
[26] Paul D. N. Hchert, Alina Cywinska, Shelley L. Ball, and Jeremy R. da Biological identifications through DNA barcodes. Proceedings of the Royal Society of London. Series B: Biological Sciences. 270
13312)311-321,, 2003. 1oi:10..100 htt./0/pb.b.......2222.
[27] Rosalee S Hellberg, Brenda C. Hernandez, and Eduardo I. Hernandez. Identification of meat and poultry species in food products using DNA barcoding. Food Control, 80:23-23 2017. 1011..0.......ocoooooo.
[28] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: http///ppe////reeeeeeee wttttorumt wttp://oouunidd-.eeeee wew.uniii-eeeeeeee.....
[29] Wisdom Ikezogwo, Saygin Scyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pa- van Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt- IM: One million image-text pairs for histopathology. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (cds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 37995-38017. Curran Associates, Inc., 2023. URL https: / http://oooeeenig/oocooo ngs www.iis..s...... ce/ wtt.c///ppppeeeeeee paper_ fi eiles// http//pppr///22/ /0t00/////l////////////////////e Pachoo Dataset s and Benchrarks pdd.
77 1755c577886f22612200666
[30] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Lud- wig Schmidt. Openclip. Jul 2021. URL https::pp.... : //l:://ddoo /diiiiiii.o//. org/ /10. 5281/ zenodo 5143773.
[31] Yanrong Ji. Zhihan Zhou, Han Liu, and Ramana V Davuluri. DNABERT: pre-trained bidirectional en- coder representations from transformers model for DNA DNA-language in genome. Bioinformatics 37(15): 2112-2120, Feb 2021. ISSN 1367-4803. doi: 10. (093hoontfrrmattcshuaaaa00:
[32] Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, Kun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 4904-4916. PMLR, 18-24 Jul 2021. URL https:/ /proceed/ ngs ww..mm.prrs///1333 j jia21b. 1.
[33] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Confer- ence on Learning Representations (ICLR), 2015.
[34] Nguyen Quoe Khanh Le, Quang-Thir Thai Ho, Van-Nui Nguyen, and Jung-Su Chang. BERT-Promoter An improved sequence-based predictor of DNA promoter using BERT pre-trained model and SHAP feature selection. Computational Biology and Chemistry, 99:107732, 2022. ISSN 1476-9271. doi: 10, 10.1110006........ http///...oooocceeeeeee 107732.
[35] Dohoon Lee, Jeewon Yang. and Sun Kim. Learning the histone codes with large genomic windows and thrce-dimensional chromatin interactions using transformer. Nature Communications, 13(1):6678, Nov 2022. ISSN 2041-1723. doi: 10. 44710//1144--02244552222
[36] Yuanheng Li, Christian Devenish, Marie I Tosa, Mingjic Luo. David M Bell, Damon B Lesmeister, Paul Greenfield, Maximilian Pichler, Taal Levi, and Douglas W Yu. Combining environmental DNA and remote sensing for efficient, fine-scale mapping of arthropad biodiversity. Philosophical Transactions of the Royal Society B, 379(1904):0220112) 2024. doi: 1098/stth 2023:rt1...2220.2..
[37] Zhongxiao Li, Elva Gao, Juexiao Zhou, Wenkai Han, Xiaopeng Xu, and Xin Gao. Applications of deep learning in understanding gene regulation. Cell Reports Methods, 31):100084,, 2023. ISSN 2667-2375 10, httt:/...eeeh.0222. 2022.100344.
[38] Carlos Lopez- Vaamonde. Natalia Kirichenko, Alain Cama, Camiel Doorenweerd, Charles Godfray, Antoine Guiguet. Stanislav Gomboc, Peter Huemer, Jean-François Landry, Ales Laštávka, et al. Evaluat- ing DNA barcoding for species identification and discovery in curopcan gracillariid moths. Frontiers in Ecology and Evolution, 9:626752, 2021. doi:
[39] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang. Tong Ding, Guillaume Jaume, Igor Odintsov, Andrew Zhang, Long Phi I.e, et al. Towards a visual-language foundation model for computational pathology. arXiv preprint arXiv /2914, 2023.
13

---

Published as a conference paper at ICLR 2025
[40] D. H. Lunt, D.-X. Zhang, J. M. Szymura, and O. M. Hewltt. The insect cytochrome oxidase gene: evolutionary patterns and conserved primers for phylogenetic studies. Insect Molecular Biology, 5(3); 153-165, 1996. Noi:10.11114 1365-33633323 298...99....000000
[41] Chloé Martineau, Donatello Conte, Romain Raveaux, Ingrid Arnault, Damien Munier, and Gilles Ven- turini. A survey on image-hased insect classification. Pattern Recognition, 65:273-284, 2017. ISSN 0031-3203. 10110..11//.paeeeeeee 10..1....22222.. 12.020.
[42] Leland McInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projec- tion for dimension reduction. arXiv preprint arXiv: 03426, 2018. 100.0445550//Xi111800..444
[43] Christopher P. Meyer and Gustav Paulay. DNA barcodes perform best with well-characterized taxa. PL.O Biology, 3:435, 2005. doi: 1.0:100..77/jommme 1/o///ooualllppiio. httpmaa..piio0000433333
[44] Wenwen Min, Zhiceng Shi, Jun Zhang, Jun Wan, and Changmiao Wang. Multimodal contrastive learning for spatial gene expression prediction using histology images. arXiv preprint arXiv:2407.08216, 2024. doi: 10::10,,e (00445500arxxx2240000 10821.....
[45] Florian Mock, Fleming Kretschmer, Anton Kriese, Sehastian Böcker, and Manja Marz. Taxonomic clas- sification of DNA sequences beyond sequence similarity using deep neural networks. Proceedings of the National Academy of Sciences. 119(5)):2112666111 2022. doi: 111.1//pa..21223366666
[46] Marko Mutanen, Lauri Kaila, Jukka Tabell. Wide-ranging barcoding aids discovery one-third increase of species richness in presumably well-investigated moths. Scientifie Reports, 3(1):2901, 2013. doi: Noi:10.1038//nn 1038/srep02901.
[47] Eric Nguyen, Michael Poli. Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Sykes, Stefano Massaroli, Aman Patel, Clayton Rabidcau, Yoshua Bengio, Stefano Ermon, Christopher Ré, and Stephen Baccus. HyenaDNA: Long-range genomic sequence modeling single nucleotide resolution. In A. Oh, T. Neumann, A. Gloherson, Saenko, M. Hardt, and S. Levine (cds.), Advances in Neural Information Processing Systems, volume 36, 43177-43201. Curran Asso- ciates, Inc., 2023. URL https: httptst/pprcceeediinne nwwrins. wwwperrripp.cpppppppeeeeeepeeeee fiwes/ialpppeeeeee hptpppppppee////////// file/ 6ab6666622 4ee wet.eeeeeeeeeeeeeeeeee22 orppeeeeeeeeeeeeaaaaaoooooeeeeee
[48] Hoang-Quaa Nguyen, Thanh-Dat Truong, Xuan Bac Nguyen, Ashley Dowling, Xin Li, and Khoa Luu. Fusect-Funndtion: foundation model and large-scala scale 1M dataset visual insect understanding. arXiv preprint 2rXivv2311.15200 2023.
1...
[49] Francois Pompanon, Bruce E Deagle, William OC Symondson, David Brown, Simon N Jarman, and Pierre Taberlet. Who is cating what: diet assessment using next generation sequencing. Molecular ecology, 1138))19311999)) 2012. doi:10. 111.1//11...... 1365/ 290.6652999... 1944.20111 1.040.0......
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Mcila and Tong Zhang (eds.), Proceedings of the International Conference on Machine Learning, volume 139 of Proceedings of Ma- chine Learning Research, pp. 8748-878 87633. PMLR, 18-24 Jul 2021. URL https: http///eoccce....e///// mlr. wt...press//ee wttesssvvv399 wttp33///rrrdd2222 html.
[51] Sujeevan Ratnasingham and Paul DN Hebert. BOLD: The barcode of life data system. Molecular ecology notes, 7(3):355-344, 2007. doi:10. 1111100.11111 177111.471--8886 606...x..
[52] Sujeevan Ratnasingham and Paul DN Hebert. A DNA-based registry for all animal species: the barcode index number (BIN) system. PloS one, 8(7)=666133, 2013. doi: 10i:10.137 1/jor//a/aallppoo..o.o pone. 0066213.
[53] Marko Ristin, Juergen Gall, Matthieu Guillaumin, and Luc Van Gool. From categories to sub- categories: large-scale image classification with partial class label refinement. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). pp. 231-239 Jun 2015. 200::00.1090CVPP22015557252
[54] Luuk Romeijn, Andrius Bernatavicius, and Duong Vu. MycoAl: Fast and accurate taxonomic classifi- cation for fungal ITS sequences. Molecular Ecology Resources, pp. c14006, 2024, 1oi:10.1111/1155- 0998. 1188.10000 14006.
[55] Yue Ruan. Han-Hung Lee, Yiming Zhang, Ke Zhang, and Angel X Chang. TriCoLo: Trimodal contrastive loss for text to shape retrieval. In Proceedings of the IRFF/CVF Winter Conference on Applications of Computer Vision, pp. 5815-5825, 2024. do1:10.11 1099/ACCV57001 2024.00571.
14

---

Published as a conference paper at ICLR 2025
[56] Krista M. Ruppert, Richard J. Kline, and Md Saydur Rahman. Past, present, and future perspectives of environmental DNA (cDNA) metabareoding: A systematic review in methods, monitoring, and ap- plications global cDNA. Global Ecology and Conservation, 17:e00547, 2019. ISSN 2351-9894, doi:10.1011 gew.o...ccco......coo
[57] Leslic N. Smith. A disciplined approach to neural network hyper-parameters: Part learn- ing rate, batch size, momentum, and weight decay. arXiv preprint 1003vv1803.0000 2018. doi: 10..448850///...188.... 1803.0980.
[58] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL https: httttps://proceeaangge. neurips. paper_files www.ttiiesseeeeeee pttp:/pppppppppeeeeep pdf.
[59] Eeva Soininen, Alice Valentini, Eric Coissac, Christian Miquel, Ludovic Gielly, Christian Brochmann, Anne K Brysting. Jørn H Sønstebø, Rolf A Ims, Nigel G Yoccoz, et al. Analysing diet of small herbi- vores: the efficiency of DNA barcoding coupled with high-throughput pyrasequencing for deciphering the composition of complex plant mixtures. Frontiers in zoology, 6: 1-9, 2009. doi: 10.1110.11111 11111186/1444 0101//177-999-----1
[60] Dirk Steinke, Sujeevan Ratnasingham, Jireh Agda, Hamzah Ait Boutou, Isaiah C. H. Box, Mary Boyle, Dean Chan, Corey Feng. Scott C. Lowe, Jaclyn T. A. McKeown, Joschka McLeod, Alan Sanchez, Ian Smith, Spencer Walker, Catherine Y.-Y. Wei, and Paul D. N. Hebert. Towards taxon- omy machine: A training set of 5.6 million arthropod images. Data, 9(11), 2024. ISSN 2306-5729.
doi: doi:10.33300 10..330/ddatt 10122.
[61] S. Stevens, Wu, M. Thompson, E. G. Campolongo, C. Song. D. Carlyn, L. Dong, W. M. Dahdul, C. Stewart, T. Berger-Woll W. Chao, and Y. Su. BioCLIP: A vision foundation model for the tree of life. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14412-1944, Los Alamitos, CA, USA, Jun 2024. IEEE Computer Society. 1oi: (U09/CCVR5773..2022...
[62] Nigel E Stork. How many species of insects and other terrestrial arthropods are there on Earth? Anmual review of entomology, 63( (311)31145,, 2018. 10.1 146//nnurrvveento-0000 17-043348. PMID: 28938083.
[63] Fariborz Taherkhani, Hadi Kazemi, Ali Dabouei, Jeremy Dawson, and Nasser M. Nasrabadi. A weakly supervised fine label classifier enhanced by coarse supervision. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision (ICCV), pp. 6459-6468, Oct 2019. doi: 10. 109/CCCV.00990000
[64] Aiham Taleb, Matthias Kirchler, Remo Monti, and Christoph Lippert. ConUG: Self-supervised multimodal contrastive learning for medical imaging with genetics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20908-20921, 2022. doi: 100.1..100// 2022000044..
[65] Christina Theodoris, Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R. Al Sayed, Matthew C. Hill, Helene Mantineo, Elizabeth M. Brydon, Zexian Zeng. X. Shirley Liu, and Patrick T. Ellinor. Transfer lcarning enables predictions in network biology. Nature, 618(7965):616-66 624, Jun 2023. ISSN 1476- 4687. doi: 10. (138//41586-02-4440
[66] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Ii-Jia Li. YFCC100M The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016. ISSN 0001-0782, 10 145/281 1882.
[67] Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learn- ing fine-grained image representations with coarse labels. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 874-884, Oct 2021. doi:10.1 109/CCCV4892222022 100091.
[68] Iulia Ture, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn bet- ter: On the importance of pre-training compact models. arXiv preprint (XXiv:9908.0860 2019. doi: 10.45500/rrxi.. 1908.089999....9.2...
[69] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018. 100:10.11199000 11..//////......000100
[70] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need? In 10th International Conference on Learning Representations, 2022. URI. https: httpss//openreviee net http:/ooormmmiiih...ppaa.e d-s......992......
15

---

Published as a conference paper at ICLR 2025
[71] Yunhao Wang. Yue Zhao, Audrey Bollas, Yuru Wang, and Kin Fai Au. Nanopore sequencing technology. bioinformaties and applications. Nature biotechnology. 19:11)) 1365, 2021. 1o0.10.0038/4155/1 021-01108-x.
[72] Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, and Serge Belongie, Fine-gras grained image analysis with deep learning: A survey, JEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):8927-8448 2022. doi:10. 100..0000//AAMM2221133333666
[73] Ross pytorch- Amago-models. 2019.
Wightman.
Pytorch image models.
[74] Tete Xiao, Xiaolong Wang, Alexci A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2021. URL https:
/ //openreview... net/
[75] Ronald Xie, Kuan Pang, Sai Chung. Catia Perciani. Sonya MacParland, Bo Wang, and Gary Bader. Spatially resolved gene expression prediction from histology images via bi-modal contrastive learn- ing. In Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 70626-70637. Curran Asso- ciates, Inc., 2023. URL httptpps//pproeeeegggg..eeeeeeee Cowee-oooooereeeeeeoeee Cerpraaaeeeeetpppppeeeeeeeeeeeee
ca/ wappappr.ppppppo.eeepeeeeeeeeeee files/ paale//pape//2233
file/df65 2tt.dd/f666777755eeeaaaaaaaaaae
[76] Chih-Hsuan Yang, Benjamin Feuer, Talukder "7aki" Jubery, 7i Deng, Andre Nakkab, Md Za- hid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh Singh, ARTI SINGH, Soumik Sarkar. Nirav Merchant, Chimmay Hegde, and Baskar Ganapathysubrama nian. BioTrove: A large curated image dataset enabling AI for biodiversity.
A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Neu- val Information Processing Systems, volume 37, pp. 102101-102120, Curran Associates, Inc., neurips. httpc//pppppaaeaeeeeeeeeeeeeeeee wileffieee//eeeeee saaeerrffieeepppppeeeeee wttpe//ppppp/////e/eeee and Benchnarks Track.
2024. URL
39335f80batteett73555555 pdf.
Paper
[77] Sheng Zhang, Yanho Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Ra- jesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao. Matthew P. Lungren, Tristan Naumana, Sheng Wang, and Hoifung Poon. BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.009) 2024. doi: 20..500/rxxv..233.0000
[78] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Zachary Lipton. Rajesh Ran- ganath, Mark Sendak, Michael Sjoding, and Serena Yeung (eds.). Proceedings of the 7th Machine Learn- ing for Healthcare Conference, volume 182 of Proceedings of Machine Learning Research, pp. 2-25. PMLR, 05-06 Aug 2022. URL : http///rcceeee i ngss.... mlr. prwss/eesvvee /vl 82/ zhang22a html.
[79] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. DNABERT-2: Efficient foundation model and benchmark for Learning Representations, 2024. URL httpstt:...
genomes. In International Conference on /owpn//veeeeeeee///e ew /wt//ffouume iiniiidd-oae d-OML.QB4 EZE.1
[80] Zhihan Zhou, Weimin Wu, Harrison Ho, Jiayi Wang, Lizhen Shi, Ramana V Davuluri, Zhong Wang, and Han Liu. DNABERT S: Learning species-awes aware DNA embedding with genome foundation models. arXiv preprint arXiv:2402.00 08777, 2024.
[81] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explamations by watching movies and reading books. In IEEE International Conference on Computer Vision (ICCV), pp. 19-27, Los Alamitos, CA, USA, Dec 2015, IEEE Computer Society. doi: 10. 1..11.
16

---

Published as a conference paper at ICLR 2025
APPENDICES
We provide additional details how we obtain data split (Appendix A), additional results (Appendix B) on the validation set, using different image and text encoders, additional details about the cross-modal experiments, and visualize the embedding space. We also include experiments with hyperparameter settings such as the use automatic mixed precision and batch size (Appendix C). In Appendix D, we provide additional information the utility and cost of DNA barcodes.
A ADDITIONAL DATA DETAILS
Train
70%
10%
Val Queries
Without species labet
80%
Seen species
10%
Test Queries
records
Well- represented species
10%
Keys
BIOSCAN-1M
Val unseen species
50%
Queries
With specles label
10%
Keys
Under- represented species
50%
2-8 records
50%
50%
Querles
Test unseen species
50%
Keys
Split species
Split records per species
Figure 7: Data partitioning strategy. We first partition species among the splits based on the presence of species label and the number of records per species, and then each species is designated as seen or unseen. Records from each species then partitioned among train (blue), validation (orange), and test (green). For the validation and test sets, records used as queries, and the rest used as keys for the reference database for retrieval.
Data partitioning process. We use multi-stage process establish our split of BIOSCAN- 1M [23] for our experiments (see Figure 7). Firstly, we separate records with and without species labels. Any record without species label is allocated for pretraining, during evaluation. Of the remaining records with labelled species, we partition species based on their number of samples. Species with at least 9 records allocated 80/20 seen and unseen, with unseen records split evenly between validation and test. Species with to records are used only unseen species, with partition of 50/50 between validation and test. This allows us to simulate real-world scenarios, in which most of our unseen species are represented only by a few records, ensuring realistic distribution of species sets. Species with only one record are excluded, as we need at least one record each to act the query and the key, respectively.
cannot easily use them
Finally, we allocate the records within each species into designated partitions. For the seen species, subdivide the records at a 70/10/10/10 ratio into train/val/test/kkey where the keys for the seen species are shared across evenly between queries and keys. The allocation of queries and keys ensures that we have clearly designated samples as inputs and target references for inference. We note that some samples in our data may have the exact same barcode
splits. The unseen species for each of validation and test are split
though the image may differ.
Data statistics. As a result of this process, we obtain a seen split with 916 species, and unseen val and test splits with 1771 and and 1772 species respectively (see Figure 2 in the main paper). We present the number of species and the distribution of records across species (Figure 8) and the average number of records per taxa level (Table 5). In Figure 8, show the number of records for
17

---

Published as a conference paper at ICLR 2025
Distribution of species (train, query and key)
1-10
Train
Val query
11-20
Test query
21-40
Seen key
41-80
Val unseen key
81-160
Test unseen key
161-1773
500
1000
1500
2000
2500
Number of species
seen
unseen
Figure 8: Distribution of number of records per species for our BIOSCAN-IM train, query and key splits. We show the seen and unseen species in blue and orange, with the intensity of the color indicating the number of records for that species. Note that the species shared across train, val, while the val and species are distinct from each other. From the distribution, we see that most species the unseen splits have fewer than 10 records (light orange).
Table 5: Data statistics. Average number of records per taxa level. Note that the seen and unseen species are distinct, but there is overlap at other levels of the taxa (order, family, genus) between seen and unseen species.
Species seen
Species unseen
Train
Key
Val Query
Test Query
Val Key
Val Query
Test Key
Test Query
order family
60007.72
634.80
487.60
487.60
545.71
496.29
593.58
537.33
2305.88
41.75
33.74
33.74
38.99
36.34
34.89
31.25
genus 85.63
6.10
5.17
5.17
4.98
4.73
4.56
4.36
species 21.79
2.06
1.99
1.99
1.92
1.92
1.70
1.69
each species by color intensity, with lighter hues for species with fewer records and darker hues for species with more records. Due to the long- tailed distribution of species, there are just a few species with large number of records, and most species with just a few records. This is especially true for the unseen split, where 90% of species have 4 or less records. Similarly, we see from Table 5 that each species has an average of only
samples the query set or the key set.
In Table 6, we present the number of classes at different taxonomy levels. Specifically, we show overlap of classes between the seen and unseen sets at various levels. We separate statistics for records that do not have any species labels (which go into the pretrain split), and the those that are used to establish the train/val/test splits. Note that the pretrain split, the statistics are lower bounds at the genus and species level as records label as well. Due to our data division method, there no overlap at the species level. However, for higher taxonomic levels there overlaps. As the taxonomy level increases from genus order, the proportion overlapping classes gradually increases, representing a larger percentage the total number of classes each level.
missing the species label and potential genus
Task difficulty. To understand the difficulty of classification at different levels of the taxonomy, we consider the performance of simple majority baseline classifier. In Table 7, present the micro
Table 6: Taxa overlap between seen and unseen species. Overlap at different levels of the taxa (order, family, genus) hetween seen and unseen species. We separately report the statistics for records without species labels (which form the pretrain set), and those that we used to build train/val/test set. Note that for the pretrain split, the number of classes the genus and species level are lower bounds, as labels missing. Gray highlight indicate number classes each taxonomy level either seen or unseen. Cyan highlight indicate the number classes that overlap at that level across seen / unseen.
Pretrain (no species)
Train/val/test
total
seen
unseen
overlap
seen
unseen
overlap
order
19
10
13
10
14
10
family
479
129
219
112
132
239
113
genus
2731
351
686
234
521
1400
279
species
4459
0
916
3543
18

---

Published as a conference paper at ICLR 2025
Table 7: Chance performance. Performance for most frequent class baseline for micro- auccr-ac and chance
performance (random class) macro-aee accurrccccrccyyy (%).
Micro
Macro
Seen
Unseen
H.M.
Seen
Unseen
H.M.
val/test
val
val
test
val/test
val
test
val
test
Order
76.32
58.46
57.27
66.21
65.44
10.00
7.14
8.33
9.09
7.69
Family
15.33
24.64
19.12
18.90
17.02
0.76
0.58
0.55
0.63
0.56
Genus
7.46
24.17
18.55
11.40
10.64
0.19
0.12
0.12
0.14
0.12
Species
5.21
19.52
12.14
8.22
7.29
0.11
0.06
0.06
0.07
0.06
and macro accuracy when the most frequent class at each taxonomy level is chosen for prediction. The macro accuracy is the same as chance performance of selecting a class at random, while the micro accuracy depends on the fraction records with the most frequent class. The significant difference between micro and macro accuracy indicates the large class imbalance in dataset. This imbalance reflects the phenomenon that some categories are more common than others in nature as well as the sampling bias present in data collection efforts.
B ADDITIONAL EXPERIMENTS
In this section, include additional experimental results and visualizations. We provide additional results on BIOSCAN-IM (Appendix B.1) and image to DNA retrieval results (Appendix B.2). We also visualize the aligned embedding space (Appendix B.3) to show the model's ' capability in inte- grating and representing diverse biological data, and more attention visualizations (Appendix B.4).
B. ADDITIONAL CLASSIFICATION RESULTS ON BIOSCAN-IM
Table 8: Top-1 micro accuracy (%) test for BIOSCAN-IM and different combinations of aligned embeddings (image, DNA, text) during contrastive training. show results for using image-to-iman DNA- to-DNA, and image-to-DNA query and key combinations. As baseline, we show the results prior to contrastive learning (uni-modal pretrained models without cross-modal alignment). We report the accuracy for seen and unseen species, and the harmonic
(HLM.) between these (bold: highest acc, italic: second highest acc.). also compare against unimodal training using SimCLR (2t row in each taxa, indicated with v*).
Aligned embeddings
DNA-to-DNA
Image-to-Image
Image-to-DNe -DAAA
Taxa
Img
DNA
Txt
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Order
99.1
98.5
98.8
88.8
90.8
89.8
10.5
11.0
10.7
x
94.4
94.1
94.3
x
99.7
99.6
99.6
100,0
100.0
100,0
99.6
99.7
99.6
99.7
98.9
99.3
/
100.0
100.0
100.0
99.7
99.6
99.6
99.7
99.3
99.5
Family
96.2
93.8
95.0
52.9
56.2
1.0
1.1
67.5
69.7
68.6
95.7
92.2
93.9
99.8
99.2
99.5
95.9
93.1
94.5
95.8
84.6
89.9
100.0
99.5
99.7
96.2
93.7
94,9
96.5
87.1
91.6
Genus
93.4
89.0
91.1
30.1
38.7
33,9
0.2
0.1
0.1
43.2
48.9
45.9
87.2
81.8
99.2
96.9
98.0
88.6
85.2
87.8
51.3
99.5
97.9
98.7
89.3
82.3
85.7
87.6
54.9
67.5
Species
X
90.4
84.6
87.4
18.1
26.8
21.6
0.1
0.1
0.1
28.5
36.2
31.9
76.2
61.9
68.3
97.9
94.8
96.3
79.2
70.0
74.3
75.1
25.2
37.7
98.4
96.3
97.3
79.6
69.7
74.3
74.2
27.8
40.4
19

---

Published as a conference paper at ICLR 2025
Table 9: Top-1 accuracy (9)) val for BIOSCAN-II 1M and different combinations of aligned embeddings (image, DNA, text) during contrastive training. show results for using image-to-imad to-image, DNA- to-DNA, and anage-to-DNA query and key combinations. As a baseline, we show the results prior to contrastive learning (uni-modal pretrained models without cross-modal alignment), and unimodal training using SimCLK (v").. We report the accuracy for seen and unseen species, and the harmonic mean (H.M.) between these (bold: highest acc, italic: second highest acc.).
Aligned embeddings
INA-to-DNA
Image-to to--aaee
Image-to-DNN
Taxa
Img
DNA
Txt
Seen Unseen
H.M.
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Order
98.6
81.5
89.2
54.5
45.9
6.0
59.6
55.4
57.4
89.2
85.9
87.5
100.0
100.0
100.0
99.5
94.1
96.7
99.5
72.0
83.5
100.0
100.0
100.0
98.6
96.1
97.3
99.2
76.0
86.1
Family
87.0
75.8
81.0
29.3
23,4
26.0
0.5
0.5
39.9
31.0
34.9
90.
74.7
81.7
99.9
96.4
98. /
89.6
78.6
83.7
92.2
48.5
100.0
97.9
98.9
92.9
79.7
85.8
88.6
54.5
67.5
Genus
81.2
67.4
73.7
13.8
11.4
12.5
0.1
0.0
0.0
23.3
17.3
19.9
69.7
53.1
60.3
98.
93./
95.5
75.4
61.7
67.9
73.2
23.3
35.3
99.0
95.7
97.3
76.0
69.0
68.6
25.5
37.2
Species
x
76.4
62.2
68.6
7.8
5.3
6.3
0.0
0.0
0.0
x
13.7
9.5
11.2
36
52.4
36.9
43.3
95.8
87.3
91.4
61.9
46.0
52.8
59.3
9.6
16.5
97.1
90.2
93.5
60.2
46.5
52.5
52.1
10.3
17.2
Table 10: Top-1 micro accuracy (%) on our val set for BIOSCAN-IM and different combinations of aligned embeddings (image, DNA, text) during contrastive training.
Aligned embeddings
DNA-to-DNA
Image-to-Image
Image-to-Die
Taxa
Img
DNA
Txt
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Order
99.2
98.4
98.8
89.3
90.0
32.2
29.8
31.0
94.2
94.2
99.7
99.6
99.6
100.0
100.0
100.0
99.7
99.6
99.6
99.6
98.9
99.2
100.0
100.0
100.0
99.7
99.5
99.6
99.7
99.0
99.3
Family
X
96.4
94.2
95.3
54.6
61.7
57.9
2.9
3.7
3.3
67.6
71.9
69.7
95.9
92.9
94.4
99.8
99.4
99.6
95.9
93.3
94.6
95.9
85.7
90.5
100.0
99.7
99.8
96.5
95.4
96.5
86.8
Genus
X
92.9
89.0
90.9
30.3
41.4
35.0
0.4
0.3
0.3
42.9
51.4
46.8
87.1
79.3
83.0
99.2
97.2
98.2
88.9
83.6
86.2
87.2
58.2
69.8
99.5
98.2
98.8
89.6
84.5
87.0
86.4
59.8
70.7
Species
X
89.5
84.8
87.1
18.1
31.6
23.0
0.1
0.1
0.1
X
28.4
41.1
33.6
76.1
68.0
71.8
98.1
95.2
96.6
79.7
74.0
76.7
75.4
38.8
51.2
98.8
96.5
97.6
80.0
74.3
77.0
73.3
39.6
51.4
Results for top-1 micro-accuracy and validation set. For completeness, provide the top micro-accuss accuracy on the test set (Table 8), and results on the validation set (see Table for macro accuracy, and Table 10 for miero accuracy).
20

---

Published as a conference paper at ICLR 2025
Overall, we similar trend in results as for macro accuracy the test set (see Table 1 in the main paper), with the trimodal model that aligns image (D) DÑA (D), and text (T) performing the best, and the I+D model outperforming the I+T model. We also observe that the micro averages (over individual samples) are much higher than the macro averages (over classes). This is expected as the rare classes are more challenging and pulls down the macro-average.
We further examine the reproducibility of our results across training runs. In Figure 9, we plot the mean and standard deviation of the miero and macro accuracy evaluated on our BIOSCAN-IM validation set after training with four different random seeds. The results indicate that although there is some fluctuation, the results are mostly stable across training runs. As expected, we observe higher variation for macro accuracy as there are many classes with only a few query records (see Figure 8). There also more variation for image-to-DNA, where the embeddings are less aligned.
100
Macro Accurecy
Cro Accuracy
60-
60
40
Soon Unsoon 4.M.
DM44 to OKA
Image to Imrgg
Image to DHA
Order
Familly
ganus
specias Order
Family
gein
species
Figure 9; Performance across four training runs for our I+D+T model that aligns all three embeddings during contrastive training. We plot the mean with the shaded regions for standard error for top-1 macro and micro accuracy (%) on our val set for BIOSCAN-IM. These plots clearly show that accuracy drops as we go from order to species, and that DNA-to-DNA retrieval is the most accurate with image-to-inee to-image second. Overall, the standard error within 0.5% for the miero accuracy, with higher variation in the image-to-DNa setting.
Unimodal contrastive loss. To check whether the improved unimodal performance is result of cooss-modal contrastive learning between DNA and image, or just an artifact of training on the domain-specific data, we conduct experiments comparing our results with fine-tuning the image en- coder with unimodal contrastive learning. We use SimCLR-style [11] training on the image encoder with normalized temperature scaled cross-entropy loss (NT-XEnt), and form positive pairs by using data augmentation. For data augmentation, we used the same set of augmentations as [11] by fol- lowing SimCLR's code, which focused on enhancing image variability, The augmentation process begins with randomly cropping the image to a size of 224, and includes random horizontal flips and random applications of color jitter with an 80% probability. Additionally, images are converted to grayscale 20% of the time, and Gaussian blur is applied with a kernel size equal to 10% of the image size, which is 22 in our case.
We see from the results (denoted by v* in Tables 1, 8, 9, and 10), that while SimCLR training im- proves performance for image-to-image to-image accuracy. it greatly underperforms our CLIBD multimodal model that aligns image to DNA. This shows that while training the BIOSCAN- 1M dataset helps to improve the otherwise generic image encoder, the signal from other modalities has the greatest benefit to performance.
B.1.1 DIFFERENT TRAINING STRATEGIES
CLIBD with SimCLR pretraining. We investigate whether starting with an image encoder that is already pretrained on BIOSCAN-IM images can improve the performance of our CLIBD model. In Table 11, we show that starting with a image encoder that is fine-tuned on BIOSCAN-IM using SimCLR, and then performing full CLIBD training (by aligning image, DNA, and taxonomic la- bels), we achieve improved image-to-image performance, and image-to-DNe -DAA performance at finer taxonomic levels.
ImageBind-style alignment. To determine whether it is necessary to align all the modalities to each other, or whether it would be better use Image [25] training where freeze the image and text encoders and align the DNA encoder. For these experiments, we use BioCLIP's image and text encoder (which are pre-aligned) and BarT DNA encoder.
21

---

Published as a conference paper at ICLR 2025
Table 11: Top-1 accuracy (%) the test of BIOSCAN-IM comparing starting with pre-trained image encoder that was fine-tuned using SimCLR (v jor not (x). full CLIBD training that align three modalities (the I+D+T model).
DNA-to-DNA
Image-to-Image
Image-to-DNA
Taxa
SimCLR
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Order
100.0
100.0
100.0
99.7
94.4
97.0
99.4
88.5
93.6
100.0
100.0
100.0
99.7
98.9
99.3
99.6
75.3
84.8
Family
X
100.0
98.3
99.1
90.9
81.8
86.1
90.8
50.1
64.6
V
100.0
99.5
99.7
92.2
85.8
88.9
92.6
52.1
66.7
Genus x
98.2
94.7
96.4
74.6
60.4
66.8
70.6
20.8
32.1
98.5
94.5
96.4
77.2
63.5
69.6
73.5
23.4
35.5
Species
x 95.6
90.4
92.9
59.3
45.0
51.2
51.6
14.7
V
95.7
89.8
92.7
62.2
47.9
54.1
56.5
9.7
16.5
Table 12: Top-1 macro accuracy (%) on the test set of BIOSCAN-IM, binding DNA to frozen (%) image and text encoders. We compare binding only the image (following ImageBind [25]) aligning to both image text. Here, we utilized BioCLIP's image and text encoders, with BarcodeBERT for DNA encoding.
DNA-to-DNA
Image-to-Image
Image-to-
Taxa
Bind to
Img
DNA
Txt
Soen
Unseen
H.M.
Seen
Unseen
H.M.
Seen
Unseen
H.M.
Order
Image
X
%
100.0
100.0
100.0
88.5
86.0
87.2
87.8
64.8
74.6
Image & Text
+
X
%
100.0
100.0
100.0
88.5
86.0
87.2
88.7
71.9
79.4
Family
Image Image Text
X
100.0
97.6
98.8
84.3
68.4
75.5
79.6
43.1
55.9
X
+
100.0
97.2
98.6
84.3
68.4
75.5
78.1
45,4
57.5
Genus
Image
X
98.2
92.9
95.5
62.6
47.1
53.7
56.0
17.9
27.1
Image & Text
4
X
98.9
93.2
96.0
62.6
47.1
53.7
56.5
19.7
29.2
Species
Image
X
95.11
87.9
91.3
44.1
32.4
37.4
39.5
12.5
Image
Text
X
4
95.9
88.6
92.1
44.1
32.4
37.4
38.0
13.9
The first approach follows the ImageBind paradigm, where the DNA encoder (BarcodeBERT) is aligned only to BioCLIP's image encoder. The second approach follows our CLIBD protocol and allows the unfrozen DNA encoder to align simultaneously with BioCLIP's image encoder and text encoder. The results show that allowing the DNA encoder align both the image and text encoders vs. just binding to the image encoder results in slightly higher performance. Compared to the results in Table 11, is evident that these approaches underperform compared to our proposed method CLIBD across all query and key combinations. Note that since we start with BioCL.IP's pretrained models, and since BioCLIP's training data includes BIOSCAN-IM, is not guaranteed that unseen species labels are actually unseen during training.
B. COMPARISON WITH BIOCLIP
Full taxonomic comparison with BioCLIP. In Table 13, we compare the performance of BioCLIP CLIBD classification for all four different taxonomic levels. We that CLIBD with 1+T is able to outperform BioCLIP at every taxonomic level. From Table 1, we see that the I+D model can already outperform our I+T model. By incorporating DNA barcodes (I+D+T), we able to achieve the best performance.
B.1.3 USING LARGER MODELS
Experiments with OLENCLIP. We conduct experiments using OpenCLIP as our text and image encoder, well as larger ViT and BERT models. We train our full trimodal model (with image, DNA, text alignment), and report the species-level top-1 macro accuracy on our validation set for BIOSCAN-IM in Table 14.
22

---

Published as a conference paper at ICLR 2025
Table 13: Comparison of top-1 macro-accuracy (%) of BioCLIP and our CLIBD model the test set, matching image embeddings (queries) against embeddings of different modalities for retrieval (image, DNA, and text keys). Note: the BioCLIP model [61] was trained on data that included BIOSCAN-IM but used different species splits, so it may have seen most of the unseen species during its training.
Aligned embeddings
Image- Image
Image-In Songee-t--Dee -N-
Image-to -eexxt Text
Taxon. level
Model
Img
DNA
Txt
Soen
Unseen
H.M.
Seen Unseen
H.M.
Seen
Unseen
H.M.
Order
BioCLIP
73.3
69.2
71.2
38.6
35.5
37.0
CLIBD
99.6
97,4
98.5
99.6
77.4
87.1
CLIBD
5
99.7
94.4
97.0
99.4
88.5
93.6
99.2
74.7
85.2
Family
BioCLIP
x
56.9
42.2
48.5
18.9
14.6
16.4
CLIBD
90.7
76.7
83./
94.9
49.9
65.4
CLIBD
90.9
81.8
86.1
90.8
50.1
64.6
95.8
50.9
66.5
Genus
BioCLIP
33.8
24.3
28.3
9.5
5.9
7.3
CLIBD
x
72.1
49,6
58.8
81.9
81,9
29.7
CLIBD
74.6
60.4
66.8
70.6
20.8
32.1
83.0
21.6
34.3
Species
BioCLIP
20.4
14.8
17.1
4.2
3.6
CLIBD
54.2
33.6
41.5
57.6
8.5
CLIBD
59.3
45.0
51.2
51.6
8.6
14.7
56.0
4.8
8.9
Table 14: Species-level top-1 macro accuracy (%) on vol set for BIOSCAN-IN with CLIBD using different image and text encoders. Results use image embedding to match against different embeddings for retrieval (Im- age, DNA, and Text). We compare using the OpenCLIP (OC) pretrained model with other models. For these experiments, we used OpenCLIP ViT-L/14 [30] which is pre-trained on OpenAl's dataset that combines mul- tiple pre-existing image datasets such as YFCC100M [66]. For timm ViT-B/16 (vit_base_pp wetsse..pccchhcc.eeeeeee.. wwwp_hh1122111 6_224) and timm ViT-L/16 (vit_large__aa bert-base batt-bbae-uneeedd as our text encoder, which was pretrained on BookCorpus [81]. For the DNA en- coder, we use BarTodeBERT (except for the first row, where we do not align the DNA embeddings). We
6_224) [73] both trained ImageNet [17]. We also used
highlight in gray the setting that uses the same vision and text encoder that we used in our other experiments.
Trainnn (per
Aligned embestings
Image-to- to-iaaee
Image-n--D0
Image
OC
Epech
CUDA
DNA
Text
Scmn
Linezz
H.M.
Suzn
Unsen
H.M.
Som
Lnsaee
H.M
7000
18
1.3 heur
70. G3
Opent! IPI (14)
Openi' IP
$4.4
W-11
41.
53.9
2.
12.6
200
15
1.4 hour
84.GB
OpeaCLIP(L/14)
<
OpenCTIP
55.8
41.1
47.7
26.4
9.6
24.4
51.2
7.5
13.9
200
15
1. hour
37.4GB
<
BiRTTismalt
52.7
57.3
14.0
32.1
70
11.5
46.7
522
9.4
hour
BERThwallly
40.3
47.5
44.
16.0
611
1.5 bour
72.5B
BERTibase uncased)
40.1
46.7
345
131
41.1
6.1
We select OpenCLIP ViT-L/14 [30] as a representative of pretrained vision-language model that is trained with contrastive loss. As the OpenCLIP model requires a large amount of memory, we use a batch size of 200. From Table 14, we see that using OpenCLIP (first two rows), we do achieve better performance (especially for image to text) compared to our choice of Timm VIT B/16 and BERT-small for the image and text encoder at batch size of 200 (row 3). To disentangle whether the better performance is from the prealigned image and text embeddings or from the larger model size, we compare with training with larger batch size (with similar CUDA memory usage, row 4) and larger unaligned image and text encoder, c.g. Timm ViT-L/16 and Bert-Base (row 5). Using larger batch size brings the image-to-image (row 2), and can be improved even further with larger batch size (see Table 20). However, the image-to-text performance is still lower, indicating that the pretrained aligned image-to-text model helpful despite the domain gap between the taxonomic labels and the text that makes up most of the pretraining data for OpenCLIP.
performance close to that of the model with OpenCLIP
B.2 ADDITIONAL CROSS- MODAL RETRIEVAL RESULTS
Details about the seen/unseen classifier for the IS-DU strategy. For the NN classifier, we compute the cosine similarity of the image query features with the image features of the seen species. If the most similar image key has a similarity higher than threshold f1, is considered seen. In the supervised fine-tuning approach, we add a linear classifier after the image encoder and fine-tune the encoder and classifier to predict the species out of the set of seen species. the softmax probability exceeds fg, the image is classified as seen.
We tuned t and to on the validation set using a uniform search over 1000 values between 0 and 1, maximizing the harmonic mean of the accuracy for seen and unseen species. We report the binary
23

---

Published as a conference paper at ICLR 2025
Table 13: Accuracy (9)) of our I+D+T model in predicting whether image query comessonds unseen species, binary classification problem (evaluated on our BIOSCAN-IM test set). For the "DNA" strategy with nearest neighbour (NN). we use the nearest DNA feature to classify into seen or unseen. It serves as a form of "oracle" as it has access to the samples from unseen species. For the "IS+DU" strategy and NN, we threshold the highest cosine similarity score against image keys. For the supervised lincar classifier (Linear), we threshold the confidenee score of the prediction over seen species. We report accuracy for seen and unseen species, and their harmonic mean (H.
Method
Strategy
Seen
Unseen
H.M.
NN (oracle)
DNA
82.16
76.21
79.07
NN
IS+DU
83.29
76.83
79.93
Linear
IS+DU
73.27
85.14
78.76
Table 16: Top-1 accuracy (9)) BIOSCAN 1M test using the Imagc+DNA+Tex model with image query. compare nearest neighbour (NN) using only DNA keys, vs. our two strategies use Image key for seen DNA key for Unseen, either NN or supervised linear classifier. also compare against BZSL [4] with our embeddings.
Micro top- acc
Macro top-1 acc
Taxa
Method
Strategy
Seen
Unseen
H.M.
Scen
Unseen
H.M.
Order
NN
DNA
99.7
99.3
99.5
99.4
88.5
93.6
NN
IS+DU
99.4
99.2
99.3
99.3
89.3
94.1
Linear
IS+DU
99.5
99.2
99.3
99.3
88.3
93.5
BZSL
IS+DU
99.4
98.2
98.8
98.9
59.6
74.4
Family
NN
DNA
96.5
87.1
91.6
90.8
50.1
64.6
NN
IS+DU
94.6
87.1
90.7
83.0
52.6
64.4
Linear
IS+DU
94.7
87.0
90.7
82.8
51.2
63.3
BZSL
IS+DU
95.6
80.3
87.3
88.0
32.5
47.5
classification results on our BIOSCAN-IM test in Table 15. In these experiments, we use the I+D+T model with images as the queries.
BSZL details In BZSL [4], the authors used two-layer Bayesian model with features extracted from pretrained image and DNA encoders. The model relies on hyperparameters, which shape the size and scaling of the posterior distribution used in Bayesian inference and thus significantly affect its performance. Tuning the hyperparameters on the validation split of the INSECT dataset allows for a practioner to tradeoff' between accuracy for seen vs. unseen classes. In our experiments, use the same search space as in the original paper for our hyperparameter tuning, which takes about 4 hours. We find that the resulting accuracy is volatile and that expanding the search space can lead to an impractical search time.
Order and family results for BIOSCAN-IM. In Table 16, we report the performance of the direct image-to-DNy -NN matching (NN with DNA), as well as our IS+DU strategy (with the NN and linear classifiers), as well as BZSL with embeddings from our CLIBD. NN and linear classifier, if a image is classified as seen, we will use image-to-image matching to identify the most similar key, and classify the species using that key. Otherwise, we match the image query features with the DNA key features for unseen species.
the IS-DU strategy, for both the
Results show that at the order and family-level, direct image-to- -o-AA matching and NN with IS-DU gives the highest performance, with BZSL being the worst performing.
B.3 EMBEDDING SPACE VISUALIZATION
To better understand the alignment of features in the embedding space, we visualize a mapping of the image, DNA. and text embeddings in Figure 10. We UMAP [42] with cosine similarity metric applied to the seen validation set to map the embeddings down to 2D space, and we mark points in the space based their order classification. We show the embedding space before alignment (a),
24

---

Published as a conference paper at ICLR 2025
(a) No alignment
(b) Align image and text
(c) Align image and DNA
(d) Align all three modalities
Diptera-image
Lepidoptera-Image
Hymenoptera-image
Hemiptera-imagg
Coleoptera-image
Diptera-dna
Lepideptera-dna
Hymenoptera-dna
Hemiptera-dna
Coleoptera-dna
Diptera-text
Lepidoptera-text
Hymenoptera-text
Hemiptera-text
Coleoptera-text
Psocodea-image
Thysanoptera-imagg image
"richoptera-image
Plecoptera-image
Neuroptera-iman
Psocodea-de dnn
Thysanoptera-dna
Trichoptera-dns
Plecoptera-dna
Neuroptera-dna
Psocodea-text
Thysamoptera-text
Trichoptera-text
Plecoptera-text
Neuroptera-text
Figure 10: Embedding visualization for order level. We visualize the embedding space with alignment (a), image and text aligned (b), image and DNA aligned (c), and all three modalities aligned (d) the seen validation set generated using UMAP on the image, DNA, and text embeddings, using cosine similarity distance metric. Marker hue: order taxon. Marker lightness: data modality.
with image and text (b), image and DNA (c), and all three modalities. We see that after aligning the modalities, samples for the same order (indicated by huc), from different modalities (indicated by lightness) tend overlap each other. We observe that, for some orders, there numerous outlier clusters spread out the space. However, overall the orders demonstrate some degree of clustering together, with image and DNA features close to one another within their respective clusters. Fur- thermore, we note the text embeddings tend to lie within the Image or (more often) DNA clusters, suggesting good alignment between text and other modalities.
B.4 ATTENTION MAP VISUALIZATION
We provide more attention map visualization samples in Figure 11, Figure 12 and Figure 13, includ- ing both success cases and failure cases. Beneath each attention map, we show the closest retrieval for that model. From the figure, that the retrieved images are all highly similar to the query image. This illustrates the challenges using image alone to perform taxonomic classification.
C IMPLEMENTATION DETAILS AND HYPERPARAMETER SELECTION
In this section, we provide experiments to validate the choice of hyperparameter settings and design choices made for efficient training of model.
25

---

Published as a conference paper at ICLR 2025
Input
No Align
I+T
I+D
I+D+T
Input
No Align
I+T
I+D
I+D+T
#odtelap!!
Bul
gslnecel
Chysskoms
Suegtach
exteelapi
Lmuelen
Surgarth Dale
retrieval
retrieval
Masimelle
Rewaopan
nearest retricval
nearest retrieval
pttsooott Bislephhd
sutelcpit Bullepenn
poctiracct2 Maavvitt
peckioood Mataeei4
Teplihh agalt
Prdpacers pellackss
Nepuciancios
Mopietaniion
Typiche aged
nearest retrieval
nearest retrieval
Raday221
nearest retrieval
nearest retrieval
grop PMO-20
prop is 1913-25
Jona va. 1940-2
goup 1500-208
chidenae
Theissnndios
rhieend
nearest retrieval
nearest retrieval
Seen
Unseen
Figure 1I: We visualize the attention and the nearest retrievals for queries from seen and unseen species. Predicted genera species are indicated green text for correct, red for incorrect. Here, we present the cases where only the I+D+T (Image+DNA+Tee model gives the correct results. In these cases, the attention map I+D+T model's image encoder activates well on the model's prediction results, while other models largely not attend visually to the entire insect.
C.I TRAINABLE VS FIXED TEMPERATURE
We compare using fixed temperature for the contrastive loss using trainable temperature (Ta- ble 17). We find that using the trainable temperature helps improve the performance, provided the model is trained for enough epochs.
C.2 AUTOMATIC MIXED PRECISION
For efficient training with large batch sizes, automatic mixed precision (AMP) with the bfloat16 data type. The bfloat16 data type gives a similar dynamic range as float32 at reduced precision, and provides stable training with reduced memory usage.
26

---

Published as a conference paper at ICLR 2025
Input
No Align
I+T
I+D
I+D+T
Input
No Align
I+T
I+D
I+D+T
Happortng
Xyloarda das
sege
Chompaate
retrieval
retrieval
Fankbaida
Tondbaida
Hennophke
Naalaks
Naure
nearest retricval
nearest retrieval
Miccrorl
Sytegees pistica
Macherdaa
Maccorts
ptteLoot Beileph
Cookdremn bentrr
aciinicgl Sulappo
pibetott Bicqpi
nearest retrieval
nearest retrieval
Sylema Mele
Appplaciiions
Delagi livws
Arpplaciiiin
Malaptii
Badapi?
Badapbai
nearest retrieval
nearest retrieval
the cytiion
Wallpet
nearest retrieval
nearest retrieval
Lycntela ala
Baca: Hilzah312
Lyceldia ala
Baqiia acaada
Lysnidaa alaa
Occooa
Ocaokl
nearest
nearest retrieval
retrieval
Seen
Unseen
Figure 12: We visualize the attention and the nearest retrievals for queries from seen and unseen species for which multiple cross-modally aligned models correctly predicted the species (top three examples with I+D correct, bottom three are examples correct, red for incorrect. For correct predictions, the attention map more often signals on the whole insect, and the nearest retrievals found by the correct models are more reasonable than the incorrect ones. Interestingly, that the strong activation correlate with incorrect predictions sometimes (5th,6th examples for incorrect H+T prediction, and examples for poor I+D prediction).
correct). Predicted genera species are indicated green text for
27

---

Published as a conference paper at ICLR 2025
Input
No Align
I+T
I+D
I+D+T
Input
No Align
I+T
I+D
I+D+T
Teplenctaamos
De plates
Nederainerpa
retrieval
retrieval
Tendadas
Quaapplarkp
Temosgylla
Depiraada
Deplaweta
nearest
nearest retrieval
retricval
Seen
Unseen
Figure 13: Failure cases. We visualize the attention and the nearest retrievals for queries from seen and unseen species for incorrect predictions by all four models (no align, I+T, I+D, I+D+T). Predicted genera species are indicated in green text for correct, red for incorrect. Despite strong signalling the insect, the model identifies alternative samples the nearest, though the retrieved examples still exhibit high-level visual similarities.
Table 17: Trainable temperature. Top-1 1 accuracy on the validation set for models contrastively trained with either fixed or trainable temperature. We consider the performance when training for different durations (1
and 15 epochs).
Micro Top-1 Accuracy (8)
Macro Top-1 Accuracy (%)
Temperature
Epochs
DNA-to-DNA
Image-to-Image
Image-to-DNA
DNA-to-DNA
Image-to-Imann
Image-to-DNA
Fixed
97.2
62.6
27.4
93.1
41.6
11.0
Trainable
63.5
92.0
42.8
10.4
Fixed
98.0
74.2
57.8
94.7
52.2
32.4
Trainable
15
98.4
76.9
56.5
94.1
57.8
35.7
We compare training with and without AMP in Table 18. By applying AMP, achieve comparable performance while using less memory. With AMP, the CUDA memory usage is reduced by about 15GB (~20%) and the training time by 3 hour per epoch (~75%). Although using full-precine precision (no AMP) yields slightly better accuracies, the lower memory usage and faster training time of AMP allows for efficient experiments. Additionally, the lower memory usage with AMP enables us to use larger batch sizes and is more effective for our experiments.
C.3 LORA FULL FINE-TUNING
For efficient training, we also investigate the performance of using LoRA [28] vs full fine-tuning. As shown in Table 19, we find that while LoRA does reduce the memory usage and training time, the performance is also notably worse, and thus we use full fine-tuning for the rest of our experiments.
Table 18: Automatic mixed precision. Top-1 micro and macro accuracy on the validation set with models contrastively trained either with or without automatic mixed precision (AMP). We compare accuracies across different embedding alignments (image-to-imagee ONA-to-DNA, and mage-to-DNN)) Both experiments have otherwise identical training conditions, including batch size of 300 and 15 training epochs.
Micro Top-1 Accuracy (%)
Macro Top-1 Accuracy (%)
Memory
Training Time
DNA-to-DNA
Image-to- Image
Image-to-DDA
Image-to-Inage
Image-to-DN
CUDA
per epoch,
-MP
98.07
74.97
$7.38
95.18
54.47
32.22
75.54
hour
4AMP
97.85
74.31
56.23
97.85
$3.65
30.99
60.64
1.15 hour
28

---

Published as a conference paper at ICLR 2025
Table 19: Low-rank adaptation. Top-1 micro/macro accuracy the validation for models contrastively trained with either with full fine-tuning or Low-Rank Adaptation (LoRA). We compare micro and macro Top-1 accuracies across different embedding alignments (image-to- Lmage-to-imagg -omeeee,, DNA-to-DNA, and Inage-to-DNA), Both strategics use batch size of 300 and are trained for a total of 15 epochs, allowing us to evaluate the impact of fine-tuning techniques model performance and CUDA memory usage.
Micro Top- Accuracy (%)
Macro Top Accuracy (8)
Memory
Training Time
Fine-tuning Method
DNA-DNA
Image-Inage
Image-DNA
DNA-DNA
Image-Inage
Image-DNA
CUDA (GBB
per epoch]
Pull Fine-Tuning
98.1
74.3
58.0
95.5
54.0
32.4
78.5GB
4.03 hour
LoRA
96.2
64,9
37.6
91.3
45.4
17.1
53,4GB
2.98 hour
Table 20: Batch size. Top-1 accuracy the validation set for models contrastively trained with different batch sizes. Training larger batch sizes helps improve accuracy genus and species.
fine-grained taxonomic levels such
Maco sep-1 secunay
Siaro up-i secarsy
Algaateen
DNA to DNA
Image to lrnge
Inage to DNA
DNA
Image to Image
Einge
Tuaa
ing
DNA
Tsi
Seen
Utacss
11M
Sccn
Usscss
H.M.
Seen
Usacon
H.M.
Sccn
Usaee)
H.M.
Socs.
Unieen
H.M.
Sozt.
Unieen
H.M.
Ouder
500
100.0
109.0
108.0
59.5
09.66
19..6
3915
99.2
188.0
02.9
-95.7
99.6
98.5
99.0
99.1
75.8
100.0
109.0
100.0
03.7
95.5
$9.5
73.6
100.0
109.0
100
99.7
09.7
19177
99.7
99.3
100L0
100.
100L
99.3
95.9
69.2
73.9
Funly
500
<
.0.9
03.6
05.8
55.6
03.9
06.7
34.8
56-2
50.3
100.0
97.6
58.7
85.5
19.3
$3.5
83.5
52.5
100.0
06.3
06.0
56.9
36.1
<
100.0
99.9
100.0
94.5
Genes 100
09.2
68.3
95.8
87.5
81.6
85-5
71.5
55.
56.9
00.4
05.5
07.0
71.3
61.6
66.1
54.0
31.4
10.7
1000
<
<
99.4
07.3
36.
68.7
84.5
86.6
$2.4
58.2
16.2
95.4
96.6
74.4
63.9
68.5
61.2
24.3
Spario 100
97.9
96.5
07.2
76.8
73.7
75.2
16.8
15.8
66.5
05.6
M1.7
070
15.4
65.9
10.6
15.5
87
16.0
1000
90.2
95.0
97.1
78.8
24.8
76.7
57.2
30.7
48.3
95.8
89.
92.5
59.5
47.3
52.5
44.3
15.3
7
5
921
77.8
29.8
05.1
19
61.
413
10.1
C.4 BATCH SIZE EXPERIMENTS
We conducted additional experiments investigate the impact of training batch size (from 500 to 2000) on model performance. The choice of batch size ordinarily does not have major impact on performance when using supervised learning, but can have larger impact when training using con- trastive learning since each positive pair is normalized against the pool of negative pairs appearing the same training batch.
Our results, shown in Table 20, confirm that the classification accuracy improves as the batch size increases. The effect on more pronounced for the harder, more fine- grained, taxonomic levels. Due to resource limitations, we were only able to train up to batch size of 2000. We anticipate that using larger batch sizes would further enhance the classification accuracy of CLIBD, especially on more fine grained taxonomic levels.
Taxonomic labels
DNA barcodes
Images
Label intensive
Automated (<$1 per barcode]
Easier to acquire Field-deployable
Expert labelers (biologists) Disagreement
Expensive machines
Time-conssuiingprocess
Human designed
Varistion wn species Similarities across species
Aids numan-undorssanding
Targeted and species-level precision
Predict taxonomic labels
Figure 14: Overview of modalities. (a) Taxonomic labels rely on expert biologists for annotation, providing valuable insights for human understanding. However, they may be difficult to determine at the species level, pecially given bias expert disagreement. (b) DNA barcodes
effective for species identification but require specialized equipment. (c) Images are the easiest to acquire, but visual differences between species can be hard to identify. Despite this, they are useful for analyzing similarities and differences between species. We attempt to leverage the benefits of all three modalities to learn a representation space for taxonomic classification and other downstream tasks.
29

---

Published as a conference paper at ICLR 2025
D DISCUSSION OF UTILITY AND COST OF DNA BARCODES
In this work, we focus on experiments on BIOSCAN-IM that showcase the value of using DNA barcodes as an alignment target. Here we discuss some limitations of using DNA barcodes as well as their advantages. We also provide details on the acquisition cost, well the computational and storage cost of using DNA barcodes.
D.1 WHEN ARE DNA BARCODES USEFUL?
While DNA barcodes a promising alignment target, there are also challenges with using them. For one, repositories [52, 6] and datasets [23, 24] that include DNA barcodes are relatively limited. In contrast, there are far more datasets, at larger scale, that consists only of images and taxonomic labels. For instance, Free-Of-LifelOM. [61] has more than 10M images of over 454k species and BioTrove [76] has more than 162M images of over 367k species, which is much larger than the recently introduced BIOSCAN-IM and BIOSCAN-SM. In addition, the BIOSCAN datasets are limited to images collected in a lab setting and thus models trained on such data may not readily transfer to images captured in the wild DNA barcodes are also not perfect indicators of species, especially in cases of complex inheritance (c.g. hybridization). There may also be potential errors during the data collection and barcoding process (e.g. errors during sequencing, potential contami- nation from other specimens, etc). Thus, relying solely on DNA barcodes for species identification, especially those based purely fixed thresholds of genetic distances, may lead to errors [43].
Nevertheless, we believe that DNA barcodes can be a useful signal, as validated by our experiments. Since their introduction [26). they have become widely used in large-scale biodiversity monitoring programs as they allow for relatively fast and reliable identification. The use of DNA barcodes for fine-grained species identification is highly accurate, with most errors coming from human er- rors [12]. With the use of metabarcoding [56]. it is also possible to identify multiple species from samples from the environment, allowing for the discovery of new species and monitoring of changes in biodiversity. There are ongoing efforts to scale up the collection of DNA barcoding, as well as developments in technology, such as nanopore sequencing [71], that allows for real-time barcoding of organisms in the field. With these technological advances, we foresee that there will be increasing amount of DNA barcode data the coming years. These will make DNA barcodes an important modality to leverage for building machine learning models for biodiversity monitoring and species discovery [46, 9, 383. Outside biodiversity monitoring, the use of DNA barcodes for identification can be used for dietary analyses (fecal sampling to determine gut content) [59, 49], food authentica- tion (e.g. whether fish is marketed under a different name, use of horse meat, etc.) [20, 27], herbal medicine ingredient identification [10], whether there is illegal transfer of animal remains across borders, and many others.
DNA barcodes can also be especially useful for distinguishing between cryptic species, which morphologically similar but genetically distinct, and understanding undiscovered species. For spec- imens that are not yet well established in the standard Linnaean taxonomy, clustering by DNA bar- codes can serve as an way to identify potential related group of organisms, called an Operational Taxonomic Unit. This is standardized through the use of Barcode Index Numbers (BINs) [52].
In our experiments, we attempted to simulate evaluation on wnseen species by splitting the data so that there are records with species label that were not seen during pretraining. However, our simulation does not necessarily match the real-world distribution of truly undiscovered species.
To truly investigate the usefulness of CLIBD, it is necessary to work with experts in biomonitoring in live-depl,yment, which is currently outside the scope of this work.
D.2 WORKFLOW FOR SPECIES IDENTIFICATION
The use of DNA barcodes for fine-grained identification is highly accurate. However, no automated method is perfect and thus it is coupled with human verification of morphological features and inspection of the specimen. In a typical workflow, a machine learning algorithm is used to assign family and order to new specimen, and barcodes used propose fine-grained BINs for the specimen. If there are inconsistencies between initial order identification and the proposed BIN, then human experts inspect the specimen and attempt provide the correct taxonomic label. However,
30

---

Published as a conference paper at ICLR 2025
not all specimens have well-estabbissed species that can be clearly associated with it, in which case the BIN serves as a placeholder until biologists come to a consensus on whether a new species is warranted whether the specimen (or group of specimen with the same BIN) does indeed belong to a existing species. In BOLD [. one of the largest repositories of DNA barcodes, and the source of the BIOSCAN-IM and BIOSCAN-5M datasets, there 1. 1.2M BINs but only 259k named species.
We hope that CLIBD can provide a multimodal model that can eventually be integrated into such workflow, provide potentially accurate classification, or tools for biologists retrieve simi- lar samples via either image or DNA. However, the study of how to integrate our model into the workflow beyond the scope of this this work.
D.3 ACQUISITION COST
Compared to acquiring taxonomic labels, DNA barcoding is automated process that provides an identifying DNA sequence (e.g. the DNA barcode) per specimen. The Centre for Biodiversity Ge- nomics (CBG), which contributed the BIOSCAN data, has been developing protocols that allow for barcode analysis for under $1. They are continuing to develop cheaper, less invasive DNA barcoding technology, with the goal of driving costs down below $0.01 per specimen. Metabarcoding (the barcoding of multiple specimens at once) is even more promising from a cost perspective. CBG is now targeting Oxford Nanopore Technology flow cell-based protocols that will lower the sequencing cost for barcode acquisition from $0.05 to $0.001.
On the other hand, obtaining taxonomic labels is currently a human driven process which can cost an average of at least S25 per sample. Depending on whether the species is known, the detail in the image, and potential dissection involved, taxonomic specialist may require few minutes often couple hours to identify sample. There are many specimens that are visually similar, and most specimens are not labelled to fine-graines taxa like genus or species because of ambiguities, expert disagreement, lack of established labels.
D.4 COMPUTATIONAL AND STORAGE COSTS
In addition to annotation costs, it is important to consider computational and storage costs. These costs are directly related the length of the DNA sequence. In this work, we use DNA barcodes from the mitochondrial COI gene, which have length of 660 base pairs (bp).
Token length. For the tokenization methods codes and WordPiece for taxonomic labels), we obtain approximately 133 tokens for DNA and 20 for taxonomic labels.
(5-mer non-overlapping tokens for DNA bar-
Computational cost. Validating on a batch of 500 samples. we find that the number of FLOPS for BarcodeBERT (87.4M parameters) is 5.90T, compared 0.13T FLOPS for LoRA-BERT (29.2M parameters) for taxonomic labels. Despite the difference in FLOPS, we find that the runtime per batch is nearly equivalent for both models 15 ms per batch.
Storage cost. Without compression, we find that the BIOSCAN-IM samples with species labels (84,450) require 53 MB of storage for DNA barcodes vs. 12 MB for taxonomic labels. Given that DNA consists primarily of four nucleotides (A, T, C, and G), simple compression of 2 bits per nuclcotide yields a size 13.3 MB for the DNA barcodes.
31


Reviews:
[{"review_id": "O3fcGMULxf", "reviewer": "Reviewer_Ujh5", "summary": "The authors present a paper on a CLIP-based approach combining images, DNA barcodes and textual taxonomic description for biological classification (specifically of insects) in an open-set setting. Experimental results show that the method is effective and outperforms dual-modality contrastive learning approaches, as well as other approaches from the literature on this specific task.", "questions": "1) In Tab. 1, what is the point of comparing non-aligned embeddings? It seems intuitive that there should be no correspondence in the learned representations.\n\n2) The attention visualization is not discussed in detail. Also, as mentioned above, what information comes from checking classification before alignment? Wouldn’t correct predictions be random in that case?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1) The paper is well-written and and organized. The proposed approach and the experiments are clearly described.\n\n2) The proposed approach is effective in tackling the problem at hand, while being simpler than common alternatives in the field.", "weaknesses": "1) The paper presents no methodological novelty, and mostly applies existing techniques in a standard way to a particular use case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x5DuxqDmdn", "reviewer": "Reviewer_s416", "summary": "The approach proposes multimodal contrastive learning between image, DNA and text (taxonomic labels) as opposed to Image+TaxonomicLabel only approach followed by previous work (BioCLIP). The usage of DNA is claimed to be better because, (1) classifying unseen species would be difficult with a taxonomic-label-only model because the species name would not have been seen during training, (2) DNA is easier to obtain compared to taxonomic label which requires careful examination by human experts.", "questions": "Some suggestions and questions,\n\n1.\tComparison with BioCLIP at different taxa levels: Going by the works claim, incorporation of DNA embeddings could help with classifying unseen species up to species level, but I’m guessing at higher taxonomic levels the BioCLIP performance should be comparable to CLIBD. It would be interesting to see a comparison.\n2.\tFrom Table 1, Image-to-DNA performance on seen species reduces going from (I+D) to (I+D+T), why is this happening? I would have expected the performance to improve.\n3.\tI’m also curios to know why BIOSCAN-5M was not used, considering that foundational models such as these can greatly benefit from more data.\n4.\tI personally felt Figure 7 was more informative to understand the data partitioning than Figure 2 (Just to consider in the future revisions).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tIncorporation of DNA as a modality to align the image embedding against instead of text is well motivated.\n2.\tExtensive experiments and ablations are provided.", "weaknesses": "1.\tThe accuracy when doing image to DNA on unseen species is not quite significant although it is better than BioCLIP’s approach of doing image to text. This indicates the image encoder is still not strong enough to generate a good DNA aligned embedding just from the image. Perhaps this can improve with more data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vTMo4toETV", "reviewer": "Reviewer_CfWv", "summary": "This paper proposes a CLIP-based method to jointly embed images, DNA barcodes, and taxonomic strings for different insect species. The method is evaluated on cross-modal matching and species classification, with qualitative and quantitative results provided. The paper compares against one external method (BIOCLIP) in one set of experiments and one external method (BZSL) in a second set of experiments. The key claim of the paper is that jointly learning from the modalities of images, DNA, and taxonomic information leads to stronger representations for downstream use.\n\nNote: Score increased after rebuttal.", "questions": "Please see weaknesses for primary questions. \n\nMinor comments / questions (no need to respond):\n* This work focuses on the cases where aligned data is available: images and DNA barcodes from the same individuals. It might be useful to extend the method to also take advantage of abundant unpaired data: images and DNA barcodes that are not paired. \n* Figure 2 could be clearer. Why are there different shades of blue and orange? Why don't the numbers in the boxes sum up to 36729? Generally, this figure did not aid my understanding (though many of my confusions were clarified in later text and figures). \n* It would be nice to have a chance-level baseline in Table 1. \n* Down the road, it might be interesting to try to integrate additional modalities, e.g. geospatial location (https://arxiv.org/abs/1906.05272).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The idea of jointly embedding DNA barcodes with images and taxonomic information is interesting.\n* The experiments in the paper are extensive - there is a lot of technical content, and it's clear that a lot of effort went in to this work. There are many quantitative results, in addition to interesting qualitative results (e.g. Fig. 3, Fig. 5).\n* The paper is very well-written. \n* The hyperparameters and training procedures are clearly spelled out.", "weaknesses": "My major issue with the paper is missing baselines:\n* The paper compares their multimodal representation learning approach against the unimodal pretrained models they start with. They show that their method is better, and conclude that multimodality is important. However, this claim is not justified - couldn't the benefit be from the additional training each modality received? It seems to me that the fair comparison would be to take the unimodal models and run unimodal CLIP-style training (with a similar computation / # steps budget) for each. If the multimodal model beats these CLIP-fine-tuned unimodal models, then that provides stronger evidence of the benefit of multimodal learning. \n* For the image encoder, wouldn't a model pretrained on BIOSCAN-1M be a more appropriate starting point / baseline than an ImageNet-pretrained model? \n\nThe paper should address a few items that were not discussed:\n* Are there confounders we need to worry about in this data, e.g. the facility the data was collected by? \n* The DNA to DNA matching results are very high, but shouldn't we expect this? Would simple homology methods based on string matching do a very good job at this task? What is the benefit of using deep embeddings to solve this problem? \n* Doesn't adding a modality increase the number of steps of training the model receives? Couldn't this be partly responsible for the differences between 1, 2, and 3 modalities in Table 1? \n* Image-DNA and Image-text matching don't seem very good based on Fig. 4, with the average being pulled up by a few good cases. What are those cases, and why are they different? Is there any insight to be gained there? \n\nA few claims made in the paper were not clear to me:\n* The paper claims that \"BIOCLIP... requires taxonomic labels to be available in order to obtain text descriptions. These labels can be expensive and time-consuming to obtain\" and that \"DNA barcodes can be obtained at scale more readily than taxonomic labels\". These claims are not intuitive to me. How much does DNA barcoding cost per individual, compared to the cost of having an expert inspect an image to identify the species? \n* The training data for BarcodeBERT is claimed to be \"different from, but highly similar to\" the data used in this paper  - can you expand on what this means, and the implications for the results presented?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aVDeW6Tm7W", "reviewer": "Reviewer_JVaW", "summary": "The paper proposes CLIPBD, a tri-modal embedding space consisting of image, text and DNA of insect specimens. CLIPBD is trained using a three-way contrastive learning objective between image, text and DNA on the BIOSCAN-1M dataset. Using a reference database of images and DNA, CLIPBD is able to classify images of seen and unseen species. Results reported in the paper show superior performance as compared to BioCLIP and other state-of-the-art DNA encoder models.", "questions": "Details about the reference database is missing. How many images and DNA barcodes are present in the reference database during inference?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- The motivation and problem formulation is sound and interesting.\n- The proposed model fuses images, text and DNA into a contrastive embedding space enabling zero-shot image classification of unseen species.\n- The flow of paper and writing is good in general.", "weaknesses": "- The abstract claims the paper is the first to use contrastive learning to fuse DNA and image. However, there are existing works [1, 2, 3] which have done this for other applications and they should be discussed in the related works.\n- The claim that *DNA is a better target than taxonomic labels* (**Line 316**) is highly questionable. This claim is reiterated in **Lines 359-362**. The paper clearly mentions that majority of the BIOSCAN-1M dataset does not have taxonomic labels. In fact only 3.36% pretraining data has labels upto the species level (**Line 321**). It is clearly seen from **Table-1** that aligning with taxonomic labels outperforms aligning with DNA at the order level. I believe if the authors used an **unbiased dataset** containing the same proportions of DNA labels and taxonomic labels, the results would have been similar if not worse. This is more of a problem of the dataset and not the modality itself. \n- Following the previous point, why does Image-to-DNA retrieval performance improve at the species level when aligning all three modalities (**Table 1**)?\n- For inference to work on unseen species during training, the framework assumes that their DNA and/or images are available in the lookup database. This is an unrealistic assumption. If the images and DNA are already available for unseen species, they might as well should have been used for training.\n- Limited technical novelty considering no new representation learning technique has been proposed. The paper uses an existing dataset containing 1M insect specimens and unbalanced DNA and taxonomic labels, raising questions on the effectiveness of the method on other real-world datasets. Majority of the experiments and evaluations are only shown for a single dataset.\n- The authors correctly pointed that BioCLIP was trained on diverse set of species including natural images. However, one easy way to utilize the BioCLIP embedding space would be to align the DNA modality with frozen BioCLIP vision and text encoders. Have the authors compared their method with this ImageBind-style training?\n\n\nReferences\n\n[1] Taleb, Aiham, et al. \"Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Xie, Ronald, et al. \"Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[3] Min, Wenwen, et al. \"Multimodal contrastive learning for spatial gene expression prediction using histology images.\" arXiv preprint arXiv:2407.08216 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)