CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale

## Abstract

Abstract Measuring biodiversity is crucial for understanding ecosystem health.
While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space.
This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data.
Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.

## 1 Introduction

As environmental change and habitat loss accelerate, monitoring biodiversity is crucial to understand and maintain the health of ecosystems.
Taxonomic classification of organisms at scale is especially important for understanding regional biodiversity and studying species interactions.

To assist in this, researchers have used computer vision to identify organisms in images \parencitegarcin2021pl,van2018inaturalist,wei2021finegrained, martineau2017survey for a variety of applications such as ecological monitoring \parencitechristin2019applications.
However, relying solely on images for identifying and classifying organisms fails to consider the rich evolutionary relationship between species and may miss fine-grained differences.
To better capture these distinctions, researchers have used DNA sequences for genome understanding and taxonomic classification \parenciteji2021dnabert, zhou2023dnabert, mock2021bertax, cahyawijaya2022snp2vec, arias2023barcodebert, Romeijn2024.
In particular, DNA barcodes \parencitehebert2003biological, short sections of DNA from specific genes such as the mitochondrial COI gene \parencitelunt1996insect in animals and ITS sequences in fungi, have been shown to be particularly useful for species identification \parencitearias2023barcodebert, Romeijn2024.
However, collecting DNA requires specialized equipment making it more expensive and less accessible than images.
In this work, we investigate whether we can leverage recent advances in multi-modal representation learning \parenciteradford2021learning,ALIGN to use information from DNA to guide the learning of image embeddings appropriate for taxonomic classification.

Recently, BioCLIP \parencitestevens2023bioclip used CLIP-style contrastive learning \parenciteradford2021learning to align images with common names and taxonomic descriptions to classify plants, animals, and fungi. While they showed that aligning image representation to text can help improve classification (especially for the few-shot scenario), their method requires taxonomic labels to be available in order to obtain text descriptions. These labels can be expensive and time-consuming to obtain.

We propose CLIBD, which uses contrastive learning to map taxonomic labels, biological images and barcode DNA to the same embedding space.
By leveraging DNA barcodes, we eliminate the reliance on manual taxonomic labels (as used for BioCLIP) while still incorporating rich taxonomic information into the representation.
This is advantageous since DNA barcodes can be obtained at scale more readily than taxonomic labels, which require manual inspection from a human expert \parencitegharaee2024step, bioscan5m, Steinke2024.
We also investigate leveraging partial taxonomic annotations, when available, to build a trimodal latent space that aligns all three modalities for improved representations.
We demonstrate the power of using DNA as a signal for aligning image embeddings by conducting experiments for fine-grained taxonomic classification down to the species level.
Our experiments show our pretrained embeddings that align modalities can (1) improve on the representational power of image and DNA embeddings alone by obtaining higher taxonomic classification accuracy and (2) provide a bridge from image to DNA to enable image-to-DNA based retrieval.

## 2 Related Work

We review work using images, DNA, and multi-modal models for fine-grained taxonomic classification of species and their application in biology.
Prior work has primarily explored building unimodal models for either images or DNA, and largely relied on fine-tuning classifiers on a set of known species.
This limits those approaches to a closed set of species, whereas we are concerned with being able to identify unseen species, for which we have no examples in the modality of interest.

Taxonomic classification of images in biology.
Many studies have explored image-based taxonomic classification of organisms \parenciteberg2014birdsnap, van2018inaturalist.
However, visual identification of species remains difficult due to the abundance of fine-grained classes and data imbalance among species.
To improve fine-grained taxonomic classification, methods such as coarse and weak supervision \parencitetouvron2021grafit, ristin2015categories, taherkhani2019weakly and contrastive learning \parencitecole2022does, xiao2020should have been developed.
Despite these advances, image-based species classification is still limited, so we leverage DNA alongside images to enhance representation learning while maintaining the relative ease of acquiring visual data for new organisms.

Representation learning for DNA.
Much work has focused on machine learning for DNA, such as for genome understanding \parenciteli2023applications,le2022bert, avsec2021effective,lee2022learning.
Recently, self-supervised learning has been used to develop foundation models on DNA, from masked-token prediction with transformers \parenciteji2021dnabert,cahyawijaya2022snp2vec,theodoris2023transfer,dalla2023nucleotide,zhou2023dnabert,arias2023barcodebert, to contrastive learning \parencitezhou2024dnaberts and next-character prediction with state-space models \parencitenguyen2024hyenadna.
While much of this work focuses on human DNA,
models have also been trained on large multi-species DNA datasets for taxonomic classification.
BERTax \parencitemock2021bertax pretrained a BERT \parencitedevlin2019bert model for hierarchical taxonomic classification but focused on coarser taxa like superkingdom, phylum, and genus, which are easier than fine-grained species classification. BarcodeBERT \parencitearias2023barcodebert showed that models pretrained on DNA barcodes rather than general DNA can be more effective for taxonomic classification.
Though some of these works use contrastive learning, they do not align DNA with images.
We extend these models by using cross-modal contrastive learning to align DNA and image embeddings, addressing the higher cost of obtaining DNA samples while improving image-based classification and enabling cross-modal queries.

Multimodal models for biology.
While most work on taxonomic classification has been limited to single modalities, recent work started developing multimodal models for biological applications \parenciteikezogwo2024quilt, lu2023towards, zhang2023large, li2024combining.
\textcitenguyen2023insectfoundation introduced Insect-1M, applying contrastive learning across text and image modalities.
BioCLIP \parencitestevens2023bioclip pretrained multimodal contrastive models on images and text encodings of taxonomic labels in TreeOfLife-10M.
However, these models focus only on *images and text*, limiting their use with new species where taxonomic labels are unavailable.
They also miss leveraging the rich taxonomic knowledge from sources like the Barcode of Life Datasystem (BOLD), which at the time of writing has nearly 19 M validated DNA barcodes.
Although many records include expert-assigned taxonomic labels, only 24% are labeled to the genus level and 9% to the species level in BOLD-derived datasets like BIOSCAN-1M and BIOSCAN-5M \parencitegharaee2024step, bioscan5m.
By aligning images to DNA barcodes, we can use precise information in the DNA to align the image representations with the task of taxonomic classification, without requiring taxonomic labels.

One of the few works that uses both images and DNA is the Bayesian zero-shot learning (BZSL) approach by \textcitebadirli2021fine. This method models priors for image-based species classification by relating unseen species to nearby seen species in the DNA embedding space.
\textcitebadirli2023classifying similarly apply Bayesian techniques, with ridge regression to map image embeddings to the DNA space to predict genera for unseen species. However, this approach assumes prior knowledge of all genera and does not use taxonomic labels to learn its mapping, limiting its representational power.
In this work, we show that aligning image and DNA modalities using end-to-end contrastive learning produces a more accurate model and useful representation space.
By incorporating text during pretraining, we can leverage available taxonomic annotations without relying on their abundance.

Figure: Figure 1: Overview of CLIBD. (a) Our model consists of three encoders for processing images, DNA barcodes, and text. During training, we use a contrastive loss to align the image, DNA, and text embeddings. (b) At inference time, we embed a *query* image and match it to a database of existing image and DNA embeddings (*keys*). We use cosine similarity to find the closest key embedding and use its taxonomic label to classify the query image.
Refer to caption: x1.png

## 3 Method

To align representations of images, DNA barcodes, and textual taxonomic labels, we start with a pretrained encoder for each modality
and fine-tune them with a multimodal contrastive loss, illustrated in Figure 1.
During inference, we use our fine-tuned encoders to extract features for a *query* image and match them against a database of image and DNA embeddings (*keys*) with known taxonomic information.
To classify a query image,
we
take
the taxonomic information associated with the most closely matched key.
Whilst we can also query against the taxonomic text embeddings, this approach will only work for species labels seen during training.
In contrast, the pretrained model has the potential to match queries against embeddings of labelled images and barcodes acquired after training.
Thus, images and DNA barcodes comprise a more robust and comprehensive set of records against which to query.

### 3.1 Training

Contrastive learning.
We base our approach on a contrastive learning scheme similar to CLIP \parenciteradford2021learning, which uses large-scale pretraining to learn joint embeddings of images and text.
In contrastive learning, embeddings for paired samples are pulled together while non-paired samples are pushed apart, thus aligning the semantic spaces for cross-modal retrieval.
Following prior work \parenciteruan2024tricolo, we extend CLIP \parenciteradford2021learning to three modalities by considering the modalities in pairs with the NT-Xent loss \parencitesohn2016improved between two modalities to align their representations.
Let matrices $\mathbf{X}$, $\mathbf{D}$, and $\mathbf{T}$ represent the batch of $\ell_{2}$-normalized embeddings of the image, DNA, and text modalities.
The $i$-th row of each representation matrix corresponds to the same physical specimen instance, thus rows $X_{i}$ and $D_{i}$ are image and DNA features from the same sample, forming a positive pair.
Features in different rows $X_{i}$ and $D_{j}$, $i\neq j$, come from different samples and are negative pairs. The contrastive loss for pair $i$ is

$$ $\displaystyle L_{i}^{(X\xrightarrow{}D)}=-\log\frac{\exp\left(X_{i}^{T}D_{i}/ \tau\right)}{\sum_{j=1}^{n}\exp\left(X_{i}^{T}D_{j}/\tau\right)},\qquad L_{i}^ {(D\xrightarrow{}X)}=-\log\frac{\exp\left(D_{i}^{T}X_{i}/\tau\right)}{\sum_{j= 1}^{n}\exp\left(D_{i}^{T}X_{j}/\tau\right)},$ $$

where $\tau$ is a trainable temperature initialized to 0.07 following \textciteradford2021learning.
The total contrastive loss for a pair of modalities is the sum over the loss terms for each pairs of samples,

$$ $L_{XD}=\sum_{i=1}^{n}\left(L_{i}^{(X\xrightarrow{}D)}+L_{i}^{(D\xrightarrow{}X )}\right),$ $$

wherein we apply the loss symmetrically to normalize over the possible paired embeddings for each modality \parencitezhang2021contrastive, ruan2024tricolo.
We repeat this for each pair of modalities and sum them to obtain the final loss, $L=L_{XD}+L_{DT}+L_{XT}$.

Pretrained encoders.
We use a pretrained model to initialize our encoders for each modality.
Images: ViT-B(^1^11Loaded as vit_base_patch16_224 in the timm library.) pretrained on ImageNet-21k and fine-tuned on ImageNet-1k \parencitedosovitskiy2020image.
DNA barcodes: BarcodeBERT \parencitearias2023barcodebert with $5$-mer tokenization, pretrained on about 893 k DNA barcodes using masked language modelling. The training data for BarcodeBERT was different from, but highly similar to, the DNA barcodes in the BIOSCAN-1M dataset, making it ideal for our study.
Text: we use the pretrained BERT-Small \parenciteturc2019well for taxonomic labels.

### 3.2 Inference

To use the model to predict taxonomic labels, we calculate the cosine similarity between the embedded input image (*query*) and reference image or DNA embeddings (*keys*) sampled from available species. We use the taxonomic label (order, family, genus, species) associated with the closest key as our prediction.
This method allows us to evaluate the model in a zero-shot setting on species which were not seen by the model during training, provided we have appropriately labelled samples to use as keys. The embedding space also provides the flexibility to be used for other downstream tasks, such as a supervised classifier or a Bayesian model \parencitebadirli2021fine,badirli2023classifying.

## 4 Task and data

To evaluate our method, we perform taxonomic classification using different combinations of input and reference modalities. The input may be a biological image or DNA sequence; this is matched against a reference set of labelled DNA barcodes, labelled biological images, or known taxonomic labels.
We evaluate predictions at each taxonomic level by averaging accuracy over samples (micro) and taxon groups (macro). Unlike training a classification head, our approach can identify unseen species using labelled reference images or DNA, without needing to know all potential species at training time.
We split the BIOSCAN-1M data such that some species are “unseen” during training, and report prediction accuracy for both seen and unseen species to study model generalization.

Dataset.
We use the BIOSCAN-1M dataset \parencitegharaee2024step, a curated collection of over one million insect data records. Each record in the dataset includes a high-quality insect image, expert-annotated taxonomic label, and a DNA barcode.
However, the dataset has incomplete taxonomic labels, with fewer than 10% of records labelled at the species level. This poses a challenge for conventional supervised methods, which require comprehensive species-level annotations, but our method is able to flexibly leverage partial or missing taxonomic information during contrastive learning.
The dataset also possesses a long-tailed class imbalance, typical of real-world biological data, presenting a challenge for modelling.
Given the vast biodiversity of insects—for which an estimated 80% is as-yet undescribed \parencitestork2018many—and the necessity to discern subtle visual differences, this dataset offers a significant challenge and opportunity for our model.

Data partitioning.
We split BIOSCAN-1M into train, validation, and test sets to evaluate zero-shot classification and model generalization to unseen species.
Records for well-represented species (at least 9 records) are partitioned at an 80/20 ratio into seen and unseen, with seen records allocated to each of the splits and unseen records allocated to validation and test.
All records without species labels are used in contrastive pretraining, and species with 2 to 8 records are added evenly to the *unseen* splits in the validation and test sets.
Importantly, we ensure that *unseen* species are mutually exclusive between the validation and test sets and do not overlap with *seen* species for labelled records.
Finally, among each of the seen and unseen sub-splits within the validation and test sets, we allocate equal proportions of records as *queries*, to be used as inputs during evaluation, and *keys*, to be used as our reference database.
See Figure 2 for split statistics and Appendix A for details.

Figure: Figure 2: Data partitioning. We split the BIOSCAN-1M data into training, validation, and test partitions. The training set (used for contrastive learning) has records without any species labels as well as a set of *seen* species. The validation and test sets include *seen* and *unseen* (not seen during training) species. These images are further split into subpartitions of *queries* and *keys* for evaluation. We ensure that the validation and test sets have different *unseen* species. Since the *seen* species are common, we have one common set of records that we use as *keys*.
Refer to caption: x2.png

Data preprocessing.
During inference, we resize images to 256×256 and apply a 224×224 center crop. For the DNA input, following \textcitearias2023barcodebert, we set a maximum length of 660 for each sequence and tokenized the input into non-overlapping 5-mers. Similar to \textcitestevens2023bioclip, we concatenate the taxonomic levels of the insects together as text input. As we did not have the common names of each record, we used the order, family, genus, and species, up to known labels. With this approach, we can still provide the model with knowledge of the higher-level taxonomy, even if some records do not have species-level annotations.

## 5 Experiments

**Table 1: Top-1 *macro*-accuracy (%) on BIOSCAN-1M *test* set for different combinations of modality alignment (image, DNA, text) during contrastive training. Results using DNA-to-DNA, image-to-image, and image-to-DNA query and key combinations. As a baseline, we also show results for uni-modal pretrained models before cross-modal alignment. We report the accuracy for seen and unseen species, and their harmonic mean (H.M.) (bold: highest acc, italic: second highest acc.).**
|  | Aligned embeddings | DNA-to-DNA | Image-to-Image | Image-to-DNA |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Taxa | Img | DNA | Txt | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |
| Order | ✗ | ✗ | ✗ | 78.8 | 91.8 | 84.8 | 54.9 | 48.0 | 51.2 | 7.7 | 9.6 | 8.5 |
|  | ✓ | ✗ | ✓ | — | — | — | 99.6 | 97.4 | 98.5 | — | — | — |
|  | ✓ | ✓ | ✗ | 100.0 | 100.0 | 100.0 | 89.5 | 97.6 | 93.4 | 99.7 | 71.8 | 83.5 |
|  | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 99.7 | 94.4 | 97.0 | 99.4 | 88.5 | 93.6 |
| Family | ✗ | ✗ | ✗ | 86.2 | 82.1 | 84.1 | 28.1 | 21.7 | 24.5 | 0.5 | 0.8 | 0.6 |
|  | ✓ | ✗ | ✓ | — | — | — | 90.7 | 76.7 | 83.1 | — | — | — |
|  | ✓ | ✓ | ✗ | 99.1 | 97.6 | 98.3 | 89.1 | 81.1 | 84.9 | 90.2 | 44.6 | 59.7 |
|  | ✓ | ✓ | ✓ | 100.0 | 98.3 | 99.1 | 90.9 | 81.8 | 86.1 | 90.8 | 50.1 | 64.6 |
| Genus | ✗ | ✗ | ✗ | 82.1 | 69.4 | 75.2 | 14.2 | 10.3 | 11.9 | 0.2 | 0.0 | 0.0 |
|  | ✓ | ✗ | ✓ | — | — | — | 72.1 | 49.6 | 58.8 | — | — | — |
|  | ✓ | ✓ | ✗ | 97.7 | 93.0 | 95.3 | 74.1 | 59.7 | 66.1 | 73.4 | 18.7 | 29.8 |
|  | ✓ | ✓ | ✓ | 98.2 | 94.7 | 96.4 | 74.6 | 60.4 | 66.8 | 70.6 | 20.8 | 32.1 |
| Species | ✗ | ✗ | ✗ | 76.4 | 63.6 | 69.4 | 7.2 | 5.0 | 5.9 | 0.1 | 0.0 | 0.0 |
|  | ✓ | ✗ | ✓ | — | — | — | 54.2 | 33.6 | 41.5 | — | — | — |
|  | ✓ | ✓ | ✗ | 94.4 | 86.9 | 90.5 | 59.2 | 45.1 | 51.2 | 58.1 | 7.7 | 13.6 |
|  | ✓ | ✓ | ✓ | 95.6 | 90.4 | 92.9 | 59.3 | 45.0 | 51.2 | 51.6 | 8.6 | 14.7 |

We evaluate the model’s ability to retrieve correct taxonomic labels using images and DNA barcodes from the BIOSCAN-1M dataset \parencitegharaee2024step. This includes species that were either *seen* or *unseen* during contrastive learning.
We also experiment on the INSECT dataset \parencitebadirli2021fine for Bayesian zero-shot learning (BZSL) species-level image classification. We report the top-1 accuracy for the *seen* and *unseen* splits, as well as their harmonic mean (H.M.). In the main paper, we focus on evaluating the model using various combinations of modalities on the test set. Specifically, we assess the model’s performance when using images and DNA as inputs, matched against their respective image and DNA reference sets, as well as the combination of image inputs with DNA references.
In addition, we visualize the attention roll-out of the vision transformer we used as our image encoder to explore how the representation changes before and after contrastive learning and how aligning with different modalities affects the focus of the image encoder.

Implementation details. Models were trained on four 80GB A100 GPUs for 50 epochs with batch size 2000, using the Adam optimizer \parencitekingma2014adam and one-cycle learning rate schedule \parenciteLeslie2018disciplined with learning rate from $1\mathrm{e}{-6}$ to $5\mathrm{e}{-5}$.
For efficient training, we use automatic mixed precision (AMP).
We study the impact of AMP and batch size in Appendix C.

### 5.1 Retrieval by image and DNA

We conducted experiments on BIOSCAN-1M \parencitegharaee2024step to study whether the accuracy of taxonomic classification improves with contrastive learning, particularly with the inclusion of barcode DNA as an additional modality.
We compare the inference performance in embedding spaces produced by models trained to align different combinations of modalities: image (I), DNA (D), and text (T). We also consider different modalities as query (input at inference time) and key (the embedding we match against). We focus more on querying by image since it is the more readily available modality.

Taxonomic classification.
In Table 1, we report the top-1 macro-accuracy on our BIOSCAN-1M test set for seen and unseen species (see Table 8 for the top-1 micro accuracy averaged over samples). We report the performance of the different alignment models at different taxonomic levels (order, family, genus, species). As expected, the performance drops for more specific taxa (e.g., accuracy for order is much higher than for species), due to both the increased number of possible labels and the more fine-grained differences between them. When we consider unseen species, there is a drop in accuracy compared to seen species, suggesting the model’s ability to generalize could be improved.

Figure: Figure 3: Example query-key pairs. Top-3 nearest specimens from the unseen validation-key dataset retrieved based on the cosine-similarity for DNA-to-DNA, image-to-image, and image-to-DNA retrieval. Box color indicates whether the retrieved samples had the same species (green), genus (light-green), family (yellow), or order (orange) as the query or, else not matched (red).
Refer to caption: x3.png

*Are multimodal aligned embeddings useful?* Our experiments show that by using contrastive learning to align images and DNA barcodes, we can 1) enable cross-modal querying and 2) improve the accuracy of our retrieval-based classifier. Unsurprisingly, we find that DNA-to-DNA retrieval provides the most accurate classification, especially for species-level classification. By using contrastive learning to align different modalities, we enhance the image representation’s ability to classify (image-to-image), especially at the genus and species level where the macro H.M. accuracy jumps from $12.5\%$ to $69\%$ (for genus) and $6.27\%$ to $52\%$ (for species) for our best model (I+D+T). Note that with alignment, the DNA-to-DNA retrieval performance also improves.

*DNA is a better alignment target than taxonomic labels.* We also see that using DNA provides a better alignment target than using taxonomic labels. Comparing the model that aligns image and text (I+T, row 2) vs. the one that aligns image and DNA (I+D, row 3), we see that the I+D model consistently gives higher accuracy than the I+T model. At the species level, it sometimes can even outperform the I+D+T model. This is likely because the I+T model relies on having taxonomic labels and only about 3.36% pretraining data have been labeled down to the species level.

*Cross-modal retrieval.* Next we consider cross-modal retrieval performance from image to DNA.
Without any alignment, image-to-DNA performance is effectively at chance accuracy, scoring extremely low for levels more fine-grained than order.
By using contrastive learning to align image to DNA, we improve performance at all taxonomic ranks. While the cross-modal performance is still low compared to within-modal retrieval, we see that it is feasible to perform image-to-DNA retrieval, which unlocks the ability to classify taxa for which no images exist in reference databases.

Retrieval examples.
Figure 3 shows examples of intra-modality image and DNA retrieval as well as image-to-DNA retrieval from our full model (aligning I+D+T), for which the retrieval is successful if the taxonomy of the retrieved key matches the image’s.
These examples show significant similarity between query and retrieved images across taxa, suggesting effective DNA and image embedding alignment despite differences in insect orientation and placement.

**Table 2: Species-level top-1 macro-accuracy (%) of BioCLIP and our CLIBD model on the test set, matching image embeddings (queries) against embeddings of different modalities for retrieval (image, DNA, and text keys). Note: the BioCLIP model \parencitestevens2023bioclip was trained on data that included BIOSCAN-1M but used different species splits, so it may have seen most of the unseen species during its training.**
|  | Aligned embeddings | Image-to-Image | Image-to-DNA | Image-to-Text |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Img | DNA | Txt | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |
| BioCLIP | ✓ | ✗ | ✓ | 20.4 | 14.8 | 17.1 | — | — | — | 4.2 | 3.1 | 3.6 |
| CLIBD | ✓ | ✗ | ✓ | 54.2 | 33.6 | 41.5 | — | — | — | 57.6 | 4.6 | 8.5 |
| CLIBD | ✓ | ✓ | ✓ | 59.3 | 45.0 | 51.2 | 51.6 | 8.6 | 14.7 | 59.1 | 5.6 | 10.2 |

### 5.2 Comparison with BioCLIP

Next we compare our aligned embedding space with that of BioCLIP \parencitestevens2023bioclip and investigate how well using taxonomic labels as key would perform. We run experiments on BIOSCAN-1M by adapting the BioCLIP zero-shot learning demo script to perform species-level image classification. We use the BioCLIP pretrained model on the BIOSCAN-1M test set, with image query and either image or text embeddings as keys.
For the text input for BioCLIP, we combined the four concatenated taxonomic levels with their provided openai_templates as text input, while for CLIBD, we used the concatenated labels only.

From Table 2, we see CLIBD consistently outperforms BioCLIP, regardless of whether images or text is used as the key, and even for CLIBD trained only on images and text.
Since BioCLIP was trained on a much broader dataset, including but not limited to BIOSCAN-1M, it may perform worse on insects as it was also trained on non-insect domains.
CLIBD can also leverage DNA features during inference, while BioCLIP is limited to image and text modalities.

*Does matching image to taxonomic labels work better than matching to image or DNA embeddings?*
The performance when using text as keys is much lower than using image or DNA keys. This shows that it is more useful to labeled samples with images (most prefered) or DNA. Nevertheless, if no such samples are available, it is possible to directly use text labels as matching keys.

### 5.3 Analysis

Figure: Figure 4: Average top-1 per-species accuracy, binned by count of species records in the key set, for different query and key combinations. First: using images as the query and key, the accuracy for both seen and unseen species increases as the number of records for the species in the key set rises. Second: using image as the query and DNA barcodes as the key, the accuracy of unseen species remains lower than seen species, even with the same number of records in the key set. Third: using image as the query and text as the key. Similar to using image to DNA, the accuracy of unseen species is lower than the seen species. Fourth: using DNA barcode as both the query and key. Since the DNA barcodes of the same species are always similar, the accuracy is always higher.
Refer to caption: x4.png

Figure: Figure 5: We visualize the attention for queries from seen and unseen species. The “Before” and “After” columns indicate whether the prediction (at the species-level) was correct before (e.g. the initial unaligned model) and after alignment (e.g. the I+D+T model). Predicted genera + species are indicated in green text for correct, red for incorrect. Note only a few samples were predicted correctly before alignment and incorrect after (38 for seen, and 69 for unseen).
Refer to caption: extracted/5982437/figs/images/attention_maps/origin/BIOUG76074-A02.png

### 5.4 Improving cross-modal classification

We now more closely investigate how we can improve cross-modal classification, where during inference time we have an image of an insect as a query, and we have a database of seen species (with images and DNA), and unseen species (with just DNA).
\textcitebadirli2021fine proposed a hierarchical Bayesian model to classify images, using training images to learn the distribution priors and DNA embeddings to build surrogate priors for unseen classes.
Here, we consider a similar zero-shot learning (ZSL) setting using our embeddings from our pretrained model for Bayesian zero-shot learning (BZSL), demonstrating its utility for unseen species classification.

Figure: Figure 6: When we have DNA barcodes but not images of unseen species, we can use a combination of image and DNA as key sets. We adapt CLIBD for using images as keys for seen species and DNA for unseen species (i.e. the IS+DU strategy) to predict the species. We first classify the input image query $q$ against seen species, using either: (a) an 1-NN approach thresholding the cosine similarity score $s=\max_{k}f_{ik}\cdot f_{q}$; or (b) a supervised classifier predicting over all seen species and thresholding the maximum softmax probability $s=\max y_{k}$ by threshold $t$. If $s<t$, we subsequently query with the image feature $f_{q}$ using 1-NN with the DNA keys $f_{lk}$ of the unseen species and predict the unseen species of the closest DNA feature. ^∗During supervised classifier training, the image encoder is finetuned only for use in the supervised pipeline.
Refer to caption: x5.png

We also consider a simpler strategy using the image embeddings for *seen* species, and DNA embeddings for *unseen* species.
A two-stage approach first determines if a new image query represents a seen or unseen species (see Figure 6).
For seen, image-to-image matching determines the species, while for unseen, image-to-DNA matching (assuming a reference set of labeled DNA samples for unseen species) determines the species.
This is denoted by “IS-DU” (image seen - DNA unseen).

**Table 3: Top-1 accuracy (%) on our BIOSCAN-1M test set using the Image+DNA+Text model with image query. We compare nearest neighbour using only DNA keys (NN DNA), vs. our two strategies to use Image key for seen and DNA key for Unseen, either NN or a supervised linear classifier. We also compare against BZSL \parencitebadirli2021fine with our embeddings.**
|  |  |  | Micro top-1 acc | Macro top-1 acc |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Taxa | Method | Strategy | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |
| Genus | NN | DNA | 87.6 | 54.9 | 67.5 | 70.6 | 20.8 | 32.1 |
|  | NN | IS+DU | 85.7 | 55.0 | 67.0 | 66.8 | 20.8 | 31.7 |
|  | Linear | IS+DU | 83.6 | 55.6 | 66.8 | 61.4 | 21.1 | 31.5 |
|  | BZSL | IS+DU | 86.8 | 46.5 | 60.6 | 75.7 | 14.4 | 24.2 |
| Species | NN | DNA | 74.2 | 27.8 | 40.4 | 51.6 | 8.6 | 14.7 |
|  | NN | IS+DU | 76.1 | 26.2 | 39.0 | 54.8 | 8.5 | 14.8 |
|  | Linear | IS+DU | 72.6 | 25.5 | 37.7 | 41.6 | 9.4 | 15.3 |
|  | BZSL | IS+DU | 76.1 | 17.6 | 28.5 | 62.6 | 7.2 | 12.9 |

**Table 4: Macro-accuracy (%) for species classification in a Bayesian zero-shot learning task on the INSECT dataset. We compare our CLIBD-D with several DNA encoders: CNN encoder \parencitebadirli2021fine, DNABERT-2 \parencitezhou2023dnabert, BarcodeBERT \parencitearias2023barcodebert. The baseline image encoder ResNet-101 used in \textcitebadirli2021fine is compared against our image encoder before (ViT-B) and after (CLIBD-I) pretraining on BIOSCAN-1M (BS-1M). We indicate the pretraining set for DNA (Pre-DNA) as the multi-species (M.S.) set from \textcitezhou2023dnabert, anthropods from \textcitearias2023barcodebert, or BS-1M. We compare models both with and without supervised fine-tuning (FT) for each encoder, except for CLIBD-D, where the comparison is with or without contrastive learning for fine-tuning on the INSECT dataset. We highlight the baseline from \textcitebadirli2021fine and the variant with our fine-tuned encoders in gray.**
|  |  | Data sources | Species-level acc (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DNA enc. | Image enc. | Pre-DNA | FT-DNA | FT-Img | Seen | ​​​Unseen | H.M. |
| CNN encoder | RN-101 | – | INSECT | – | 38.3 | 20.8 | 27.0 |
| DNABERT-2 | RN-101 | M.S. | – | – | 36.2 | 10.4 | 16.2 |
| DNABERT-2 | RN-101 | M.S. | INSECT | – | 30.8 | 8.6 | 13.4 |
| BarcodeBERT | RN-101 | Arthro | – | – | 38.4 | 16.5 | 23.1 |
| BarcodeBERT | RN-101 | Arthro | INSECT | – | 37.3 | 20.8 | 26.7 |
| BarcodeBERT | ViT-B | Arthro | INSECT | – | 42.4 | 23.5 | 30.2 |
| BarcodeBERT | ViT-B | Arthro | INSECT | INSECT | 54.1 | 20.1 | 29.3 |
| CNN encoder | CLIBD-I | – | INSECT | – | 37.7 | 16.0 | 22.5 |
| BarcodeBERT | CLIBD-I | Arthro | INSECT | – | 52.0 | 21.6 | 30.6 |
| BarcodeBERT | CLIBD-I | Arthro | INSECT | INSECT | 34.5 | 18.2 | 23.8 |
| CLIBD-D | RN-101 | BS-1M | – | – | 54.9 | 20.0 | 29.3 |
| CLIBD-D | RN-101 | BS-1M | INSECT | – | 32.8 | 25.0 | 28.4 |
| CLIBD-D | CLIBD-I | BS-1M | – | – | 34.2 | 22.1 | 26.9 |
| CLIBD-D | CLIBD-I | BS-1M | INSECT | INSECT | 57.9 | 25.1 | 35.0 |

Determining seen vs. unseen. We frame the problem as an open-set recognition task \parencitevaze2022openset by using a classifier to determine whether an image query corresponds to a *seen* species or an *unseen* species.
This is useful for novel species detection as in practice we may not have a reference set of labeled samples or even set of species labels for unseen species.
We compare using a 1-nearest neighbor (NN) classifier, and a linear supervised classifier with a fine-tuned image encoder (see Figure 6 left). See Appendix B.2 for details.
We evaluate the ability of our classifiers to distinguish between seen and unseen in Table 12.
Using the image-to-image NN classifier, we obtain $83\%$ accuracy on seen, $77\%$ on unseen and a harmonic mean of $80\%$. For our linear classifier, we obtain lower performance on seen ($73\%$), but higher on unseen ($85\%$), with harmonic mean of $79\%$.

Evaluation on BIOSCAN-1M.
We first conduct experiments on the BIOSCAN-1M test set to compare our IS+DU strategy vs. incorporating our learned embeddings in BZSL. We also compare against querying the seen and unseen DNA keys using 1-NN directly.
Table 3 reports the top-1 micro and macro accuracy of at the genus and species level (see Table 13 for order and family level classification).
We find that at the genus and species level, BZSL obtains the good performance for seen species but that our simple NN-based approach actually outperforms BZSL on unseen species.
Using our IS-DU strategy with the supervised linear classifier, we obtain the best macro top-1 accuracy on unseen species, demonstrating that the complexity of BZSL may not be necessary.

Evaluation on INSECT dataset with BZSL.
We evaluate on the INSECT dataset \parencitebadirli2021fine, which contains 21,212 pairs of insect images and DNA barcodes from 1,213 species.
We compare different combinations of image and DNA encoders. As baselines, we use a ResNet-101 image encoder, pretrained on ImageNet-1K (used in \parencitebadirli2021fine), and the ViT-B \parencitedosovitskiy2020image image encoder, pretrained on ImageNet-21k and fine-tuned on ImageNet-1k. For DNA encoders, we evaluate the baseline CNN from \textcitebadirli2021fine; DNABERT-2 \parencitezhou2023dnabert, a BERT-based model trained on multi-species DNA data; and BarcodeBERT \parencitearias2023barcodebert, which was pretrained on arthropodic DNA barcode data.

Table 4 shows that the baseline image encoder with CLIBD-D surpasses all baseline methods without fine-tuning on the image encoder, and matches the performance of the fine-tuned BarcodeBERT and ViT-B in harmonic mean, even without fine-tuning on the INSECT dataset, especially on unseen species. Furthermore, using CLIBD-I improves performance over the baseline image encoder, with the highest performance after fine-tuning of 57.9% seen accuracy and 25.1% unseen accuracy. This shows the benefits of learning a shared embedding space relating image and DNA data, both in performance and the flexibility of applying to downstream tasks.

## 6 Conclusion

We introduced CLIBD, an approach for integrating biological images with barcode DNA and taxonomic labels to improve taxonomic classification. It uses contrastive learning to align embeddings in a shared latent space.
Our experiments show that, using DNA as an alignment target for image representations, CLIBD outperforms models that only align images and text.
We further demonstrate the effectiveness of our aligned embedding in zero-shot image-to-DNA retrieval.
While we have demonstrated cross-modal retrieval from image to DNA, the performance still lags behind intra-modal performance, suggesting an opportunity for improvement.
Our experiments were also limited to the BIOSCAN-1M and INSECT datasets.
With the introduction of larger datasets, such as BIOSCAN-5M, a promising direction is to apply our method on a larger scale.
Additionally, we only investigated CLIP-style contrastive pretraining.
Investigating other multi-modal learning schemes with images and DNA is another promising direction.
We hope to extend our method to other species beyond insects and apply our learned representations to other downstream tasks, such as 3D model generation to better understand the characteristics of each species.

## Acknowledgement

We acknowledge the support of the Government of Canada’s New Frontiers in Research Fund (NFRF) [NFRFT-2020-00073].
AXC and GWT are also supported by Canada CIFAR AI Chair grants.
JBH is supported by the Pioneer Centre for AI (DNRF grant number P1).
This research was enabled in part by support provided by
the Digital Research Alliance of Canada ([alliancecan.ca](https://alliancecan.ca/)).
We also thank Mrinal Goshalia and Kian Hosseinkhani for testing the code,
Han-Hung Lee and Yiming Zhang for helpful discussions, and Manolis Savva for proofreading and feedback.

## Appendices

We provide additional details on how we obtain our data split (Appendix A), additional results (Appendix B) on the validation set, using different image and text encoders, additional details about the cross-modal experiments, and visualize the embedding space. We also include experiments with hyperparameter settings such as the use of automatic mixed precision and batch size (Appendix C).

## Appendix A Additional data details

Figure: Figure 7: *Data partitioning strategy*. We first partition species among the splits based on the presence of a species label and the number of records per species, and then each species is designated as seen or unseen. Records from each species are then partitioned among train (blue), validation (orange), and test (green). For the validation and test sets, some records are used as *queries*, and the rest are used as *keys* for the reference database for retrieval.
Refer to caption: x6.png

Data partitioning process. We use a multi-stage process to establish our split of BIOSCAN-1M \parencitegharaee2024step for our experiments (see Figure 7).
Firstly, we separate records with and without species labels. Any record without a species label is allocated for pretraining, as we cannot easily use them during evaluation.
Of the remaining records with labelled species, we partition species based on their number of samples. Species with at least 9 records are allocated 80/20 to *seen* and *unseen*, with unseen records split evenly between validation and test. Species with 2 to 8 records are used only as unseen species, with a partition of 50/50 between validation and test. This allows us to simulate real-world scenarios, in which most of our unseen species are represented only by a few records, ensuring a realistic distribution of species sets. Species with only one record are excluded, as we need at least one record each to act the query and the key, respectively.

## Appendix B Additional Experiments

In this section, we include additional experimental results and visualizations. We provide additional results on BIOSCAN-1M (Appendix B.1) and image to DNA retrieval results (Appendix B.2). We also visualize the aligned embedding space (Appendix B.3) to show the model’s capability in integrating and representing diverse biological data, and more attention visualizations (Appendix B.4).

### B.1 Additional classification results on BIOSCAN-1M

**Table 8: Top-1 *micro* accuracy (%) on our *test* set for BIOSCAN-1M and different combinations of aligned embeddings (image, DNA, text) during contrastive training. We show results for using image-to-image, DNA-to-DNA, and image-to-DNA query and key combinations. As a baseline, we show the results prior to contrastive learning (uni-modal pretrained models without cross-modal alignment). We report the accuracy for seen and unseen species, and the harmonic mean (H.M.) between these (bold: highest acc, italic: second highest acc.).**
|  | Aligned embeddings | DNA-to-DNA | Image-to-Image | Image-to-DNA |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Taxa | Img | DNA | Txt | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |
| Order | ✗ | ✗ | ✗ | 99.1 | 98.5 | 98.8 | 88.8 | 90.8 | 89.8 | 10.5 | 11.0 | 10.7 |
|  | ✓ | ✗ | ✓ | — | — | — | 99.7 | 99.6 | 99.6 | — | — | — |
|  | ✓ | ✓ | ✗ | 100.0 | 100.0 | 100.0 | 99.6 | 99.7 | 99.6 | 99.7 | 98.9 | 99.3 |
|  | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 99.7 | 99.6 | 99.6 | 99.7 | 99.3 | 99.5 |
| Family | ✗ | ✗ | ✗ | 96.2 | 93.8 | 95.0 | 52.9 | 60.0 | 56.2 | 1.0 | 1.1 | 1.0 |
|  | ✓ | ✗ | ✓ | — | — | — | 95.7 | 92.2 | 93.9 | — | — | — |
|  | ✓ | ✓ | ✗ | 99.8 | 99.2 | 99.5 | 95.9 | 93.1 | 94.5 | 95.8 | 84.6 | 89.9 |
|  | ✓ | ✓ | ✓ | 100.0 | 99.5 | 99.7 | 96.2 | 93.7 | 94.9 | 96.5 | 87.1 | 91.6 |
| Genus | ✗ | ✗ | ✗ | 93.4 | 89.0 | 91.1 | 30.1 | 38.7 | 33.9 | 0.2 | 0.1 | 0.1 |
|  | ✓ | ✗ | ✓ | — | — | — | 87.2 | 77.1 | 81.8 | — | — | — |
|  | ✓ | ✓ | ✗ | 99.2 | 96.9 | 98.0 | 88.6 | 82.1 | 85.2 | 87.8 | 51.3 | 64.8 |
|  | ✓ | ✓ | ✓ | 99.5 | 97.9 | 98.7 | 89.3 | 82.3 | 85.7 | 87.6 | 54.9 | 67.5 |
| Species | ✗ | ✗ | ✗ | 90.4 | 84.6 | 87.4 | 18.1 | 26.8 | 21.6 | 0.1 | 0.1 | 0.1 |
|  | ✓ | ✗ | ✓ | — | — | — | 76.2 | 61.9 | 68.3 | — | — | — |
|  | ✓ | ✓ | ✗ | 97.9 | 94.8 | 96.3 | 79.2 | 70.0 | 74.3 | 75.1 | 25.2 | 37.7 |
|  | ✓ | ✓ | ✓ | 98.4 | 96.3 | 97.3 | 79.6 | 69.7 | 74.3 | 74.2 | 27.8 | 40.4 |

Results for top-1 micro-accuracy and validation set. For completeness, we provide the top 1 micro-accuracy on the test set (Table 8), and results on the validation set (see Table 9 for macro accuracy, and Table 10 for micro accuracy).

**Table 9: Top-1 *macro* accuracy (%) on our *val* set for BIOSCAN-1M and different combinations of aligned embeddings (image, DNA, text) during contrastive training. We show results for using image-to-image, DNA-to-DNA, and image-to-DNA query and key combinations. As a baseline, we show the results prior to contrastive learning (uni-modal pretrained models without cross-modal alignment). We report the accuracy for seen and unseen species, and the harmonic mean (H.M.) between these (bold: highest acc, italic: second highest acc.).**
|  | Aligned embeddings | DNA-to-DNA | Image-to-Image | Image-to-DNA |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Taxa | Img | DNA | Txt | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |
| Order | ✗ | ✗ | ✗ | 98.6 | 81.5 | 89.2 | 54.5 | 39.7 | 45.9 | 8.4 | 6.0 | 7.0 |
|  | ✓ | ✗ | ✓ | — | — | — | 89.2 | 85.9 | 87.5 | — | — | — |
|  | ✓ | ✓ | ✗ | 100.0 | 100.0 | 100.0 | 99.5 | 94.1 | 96.7 | 99.5 | 72.0 | 83.5 |
|  | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 98.6 | 96.1 | 97.3 | 99.2 | 76.0 | 86.1 |
| Family | ✗ | ✗ | ✗ | 87.0 | 75.8 | 81.0 | 29.3 | 23.4 | 26.0 | 0.5 | 0.5 | 0.5 |
|  | ✓ | ✗ | ✓ | — | — | — | 90.1 | 74.7 | 81.7 | — | — | — |
|  | ✓ | ✓ | ✗ | 99.9 | 96.4 | 98.1 | 89.6 | 78.6 | 83.7 | 92.2 | 48.5 | 63.6 |
|  | ✓ | ✓ | ✓ | 100.0 | 97.9 | 98.9 | 92.9 | 79.7 | 85.8 | 88.6 | 54.5 | 67.5 |
| Genus | ✗ | ✗ | ✗ | 81.2 | 67.4 | 73.7 | 13.8 | 11.4 | 12.5 | 0.1 | 0.0 | 0.0 |
|  | ✓ | ✗ | ✓ | — | — | — | 69.7 | 53.1 | 60.3 | — | — | — |
|  | ✓ | ✓ | ✗ | 98.1 | 93.1 | 95.5 | 75.4 | 61.7 | 67.9 | 73.2 | 23.3 | 35.3 |
|  | ✓ | ✓ | ✓ | 99.0 | 95.7 | 97.3 | 76.0 | 63.1 | 69.0 | 68.6 | 25.5 | 37.2 |
| Species | ✗ | ✗ | ✗ | 76.4 | 62.2 | 68.6 | 7.8 | 5.3 | 6.3 | 0.0 | 0.0 | 0.0 |
|  | ✓ | ✗ | ✓ | — | — | — | 52.4 | 36.9 | 43.3 | — | — | — |
|  | ✓ | ✓ | ✗ | 95.8 | 87.3 | 91.4 | 61.9 | 46.0 | 52.8 | 59.3 | 9.6 | 16.5 |
|  | ✓ | ✓ | ✓ | 97.1 | 90.2 | 93.5 | 60.2 | 46.5 | 52.5 | 52.1 | 10.3 | 17.2 |

**Table 10: Top-1 *micro* accuracy (%) on our *val* set for BIOSCAN-1M and different combinations of aligned embeddings (image, DNA, text) during contrastive training. We show results for using image-to-image, DNA-to-DNA, and image-to-DNA query and key combinations. As a baseline, we show the results prior to contrastive learning (uni-modal pretrained models without cross-modal alignment). We report the accuracy for seen and unseen species, and the harmonic mean (H.M.) between these (bold: highest acc, italic: second highest acc.).**
|  | Aligned embeddings | DNA-to-DNA | Image-to-Image | Image-to-DNA |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Taxa | Img | DNA | Txt | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |
| Order | ✗ | ✗ | ✗ | 99.2 | 98.4 | 98.8 | 89.3 | 90.7 | 90.0 | 32.2 | 29.8 | 31.0 |
|  | ✓ | ✗ | ✓ | — | — | — | 99.7 | 99.6 | 99.6 | — | — | — |
|  | ✓ | ✓ | ✗ | 100.0 | 100.0 | 100.0 | 99.7 | 99.6 | 99.6 | 99.6 | 98.9 | 99.2 |
|  | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 99.7 | 99.5 | 99.6 | 99.7 | 99.0 | 99.3 |
| Family | ✗ | ✗ | ✗ | 96.4 | 94.2 | 95.3 | 54.6 | 61.7 | 57.9 | 2.9 | 3.7 | 3.3 |
|  | ✓ | ✗ | ✓ | — | — | — | 95.9 | 92.9 | 94.4 | — | — | — |
|  | ✓ | ✓ | ✗ | 99.8 | 99.4 | 99.6 | 95.9 | 93.3 | 94.6 | 95.9 | 85.7 | 90.5 |
|  | ✓ | ✓ | ✓ | 100.0 | 99.7 | 99.8 | 96.5 | 94.3 | 95.4 | 96.5 | 86.8 | 91.4 |
| Genus | ✗ | ✗ | ✗ | 92.9 | 89.0 | 90.9 | 30.3 | 41.4 | 35.0 | 0.4 | 0.3 | 0.3 |
|  | ✓ | ✗ | ✓ | — | — | — | 87.1 | 79.3 | 83.0 | — | — | — |
|  | ✓ | ✓ | ✗ | 99.2 | 97.2 | 98.2 | 88.9 | 83.6 | 86.2 | 87.2 | 58.2 | 69.8 |
|  | ✓ | ✓ | ✓ | 99.5 | 98.2 | 98.8 | 89.6 | 84.5 | 87.0 | 86.4 | 59.8 | 70.7 |
| Species | ✗ | ✗ | ✗ | 89.5 | 84.8 | 87.1 | 18.1 | 31.6 | 23.0 | 0.1 | 0.1 | 0.1 |
|  | ✓ | ✗ | ✓ | — | — | — | 76.1 | 68.0 | 71.8 | — | — | — |
|  | ✓ | ✓ | ✗ | 98.1 | 95.2 | 96.6 | 79.7 | 74.0 | 76.7 | 75.4 | 38.8 | 51.2 |
|  | ✓ | ✓ | ✓ | 98.8 | 96.5 | 97.6 | 80.0 | 74.3 | 77.0 | 73.3 | 39.6 | 51.4 |

Overall, we see a similar trend in results as for macro accuracy on the test set (see Table 1 in the main paper), with the trimodal model that aligns image (I), DNA (D), and text (T) performing the best, and the I+D model outperforming the I+T model.
We also observe that the micro averages (over individual samples) are much higher than the macro averages (over classes).
This is expected as the rare classes are more challenging and pulls down the macro-average.

We further examine the reproducibility of our results across training runs. In Figure 9, we plot the mean and standard deviation of the micro and macro accuracy evaluated on our BIOSCAN-1M validation set after training with four different random seeds. The results indicate that although there is some fluctuation, the results are mostly stable across training runs.
As expected, we observe higher variation for macro accuracy as there are many classes with only a few query records (see Section 5.3).
There is also more variation for image-to-DNA, where the embeddings are less aligned.

Figure: Figure 9: Performance across four training runs for our full I+D+T (image, DNA, text) model that aligns all three embeddings during contrastive training. We plot the mean with the shaded regions for standard error for top-1 macro and micro accuracy (%) on our *val* set for BIOSCAN-1M. These plots clearly show that accuracy drops as we go from order to species, and that DNA-to-DNA retrieval is the most accurate with image-to-image second. Overall, the standard error is within 0.5% for the micro accuracy, with higher variation in the image-to-DNA setting.
Refer to caption: x7.png

**Table 11: Species-level top-1 macro accuracy (%) on our *val* set for BIOSCAN-1M with CLIBD using different image and text encoder. Results use image embedding to match against different embeddings for retrieval (Image, DNA, and Text). We compare using the OpenCLIP (OC) pretrained model with other models. For these experiments, we used OpenCLIP ViT-L/14 \parenciteilharco_gabriel_2021_5143773 which is pre-trained on OpenAI’s dataset that combines multiple pre-existing image datasets such as YFCC100M \parencitethomee2016yfcc100m. For timm ViT-B/16 (vit_base_patch16_224) and timm ViT-L/16 (vit_large_patch16_224) \parenciterw2019timm both are trained on ImageNet \parencitedeng2009imagenet. We also used bert-base-uncased as our pretrained text encoder which was pretrained on BookCorpus \parenciteZhu_2015_ICCV. For the DNA encoder, we use BarcodeBERT (except for the first row, where we do not align the DNA embeddings). We highlight in gray the setting that uses the same vision and text encoder that we used in our other experiments.**
|  | Batch<br>size |  | Training time<br>(per epoch) | Memory<br>CUDA | Aligned embeddings | Image-to-Image | Image-to-DNA | Image-to-Text |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OC | Epoch | Image | DNA | Text | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |  |  |  |
| ✓ | 200 | 15 | 1.3 hour | 70.1GB | OpenCLIP(L/14) | ✗ | OpenCLIP | 54.4 | 36.7 | 43.8 | — | — | — | 53.9 | 7.1 | 12.6 |
| ✓ | 200 | 15 | 1.4 hour | 84.1GB | OpenCLIP(L/14) | ✓ | OpenCLIP | 56.8 | 41.1 | 47.7 | 36.4 | 9.0 | 14.4 | 51.2 | 7.5 | 13.0 |
| ✗ | 200 | 15 | 1.5 hour | 37.4GB | timm(B/16) | ✓ | BERT(small) | 52.7 | 37.7 | 44.0 | 32.1 | 7.0 | 11.5 | 44.7 | 5.2 | 9.4 |
| ✗ | 500 | 38 | 0.6 hour | 82.1GB | timm(B/16) | ✓ | BERT(small) | 57.8 | 40.2 | 47.5 | 44.5 | 9.8 | 16.0 | 51.4 | 6.1 | 10.9 |
| ✗ | 200 | 15 | 1.5 hour | 72.5GB | timm(L/16) | ✓ | BERT(base-uncased) | 55.9 | 40.1 | 46.7 | 34.5 | 8.1 | 13.1 | 41.1 | 6.3 | 10.9 |

Experiments with OpenCLIP.
We conduct experiments using OpenCLIP as our text and image encoder, as well as larger ViT and BERT models.
We train our full trimodal model (with image, DNA, text alignment), and report the species-level top-1 macro accuracy on our validation set for BIOSCAN-1M in Table 11.

We select OpenCLIP ViT-L/14 \parenciteilharco_gabriel_2021_5143773 as a representative of a pretrained vision-language model that is trained with contrastive loss.
As the OpenCLIP model requires a large amount of memory, we use a batch size of 200. From Table 11, we see that using OpenCLIP (first two rows), we do achieve better performance (especially for image to text) compared to our choice of Timm VIT B/16 and BERT-small for the image and text encoder at batch size of 200 (row 3).
To disentangle whether the better performance is from the prealigned image and text embeddings or from the larger model size, we compare with training with a larger batch size (with similar CUDA memory usage, row 4) and larger unaligned image and text encoder, e.g. Timm ViT-L/16 and Bert-Base (row 5).
Using a larger batch size brings the image-to-image performance close to that of the model with OpenCLIP (row 2), and can be improved even further with larger batch size (see Table 17).
However, the image-to-text performance is still lower, indicating that the pretrained aligned image-to-text model is helpful despite the domain gap between the taxonomic labels and the text that makes up most of the pretraining data for OpenCLIP.

### B.2 Additional cross-modal retrieval results

**Table 12: Accuracy (%) of our I+D+T model in predicting whether an image query corresponds to a seen or unseen species, as a binary classification problem (evaluated on our BIOSCAN-1M test set). For the “DNA” strategy with nearest neighbour (NN), we use the nearest DNA feature to classify into seen or unseen. It serves as a form of “oracle” as it has access to the samples from unseen species. For the “IS+DU” strategy and NN, we threshold the highest cosine similarity score against image keys. For the supervised linear classifier (Linear), we threshold the confidence score of the prediction over seen species. We report accuracy for seen and unseen species, and their harmonic mean (H.M).**
| Method | Strategy | Seen | ​​​Unseen | H.M. |
| --- | --- | --- | --- | --- |
| NN (oracle) | DNA | 82.16 | 76.21 | 79.07 |
| NN | IS+DU | 83.29 | 76.83 | 79.93 |
| Linear | IS+DU | 73.27 | 85.14 | 78.76 |

Details about the seen/unseen classifier for the IS-DU strategy.
For the NN classifier, we compute the cosine similarity of the image query features with the image features of the seen species.
If the most similar image key has a similarity higher than threshold $t_{1}$, it is considered *seen*.
In the supervised fine-tuning approach, we add a linear classifier after the image encoder and fine-tune the encoder and classifier to predict the species out of the set of seen species.
If the softmax probability exceeds $t_{2}$, the image is classified as *seen*.

We tuned $t_{1}$ and $t_{2}$ on the validation set using a uniform search over 1000 values between 0 and 1, maximizing the harmonic mean of the accuracy for seen and unseen species.
We report the binary classification results on our BIOSCAN-1M test set in Table 12. In these experiments, we use the I+D+T model with images as the queries.

Order and family results for BIOSCAN-1M.
In Table 13, we report the performance of the direct image-to-DNA matching (NN with DNA), as well as our IS+DU strategy (with the NN and linear classifiers), as well as BZSL with embeddings from our CLIBD.
In the IS-DU strategy, for both the NN and linear classifier, if a image is classified as *seen*, we will use image-to-image matching to identify the most similar key, and classify the species using that key. Otherwise, we match the image query features with the DNA key features for unseen species.

Results show that at the order and family-level, direct image-to-DNA matching and NN with IS-DU gives the highest performance, with BZSL being the worst performing.

**Table 13: Top-1 accuracy (%) on our BIOSCAN-1M test set using the Image+DNA+Text model with image query. We compare nearest neighbour (NN) using only DNA keys, vs. our two strategies to use Image key for seen and DNA key for Unseen, either NN or a supervised linear classifier. We also compare against BZSL \parencitebadirli2021fine with our embeddings.**
|  |  |  | Micro top-1 acc | Macro top-1 acc |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Taxa | Method | Strategy | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |
| Order | NN | DNA | 99.7 | 99.3 | 99.5 | 99.4 | 88.5 | 93.6 |
|  | NN | IS+DU | 99.4 | 99.2 | 99.3 | 99.3 | 89.3 | 94.1 |
|  | Linear | IS+DU | 99.5 | 99.2 | 99.3 | 99.3 | 88.3 | 93.5 |
|  | BZSL | IS+DU | 99.4 | 98.2 | 98.8 | 98.9 | 59.6 | 74.4 |
| Family | NN | DNA | 96.5 | 87.1 | 91.6 | 90.8 | 50.1 | 64.6 |
|  | NN | IS+DU | 94.6 | 87.1 | 90.7 | 83.0 | 52.6 | 64.4 |
|  | Linear | IS+DU | 94.7 | 87.0 | 90.7 | 82.8 | 51.2 | 63.3 |
|  | BZSL | IS+DU | 95.6 | 80.3 | 87.3 | 88.0 | 32.5 | 47.5 |

Figure: Figure 10: Embedding visualization for order level. We visualize the embedding space with no alignment (a), image and text aligned (b), image and DNA aligned (c), and all three modalities aligned (d) over the seen validation set generated using UMAP on the image, DNA, and text embeddings, using a cosine similarity distance metric. Marker hue: order taxon. Marker lightness: data modality.
Refer to caption: x8.png

Figure: Figure 11: We visualize the attention for queries from seen and unseen species. Predicted genera + species are indicated in green text for correct, red for incorrect. It showcases that in most cases, the attention map of the image encoder can well reflect the model’s prediction results, but there are also some difficult samples where the effectiveness of the attention map does not equate to the correctness of the prediction results.
Refer to caption: extracted/5982437/figs/images/attention_maps/origin/BIOUG68700-H02.png

### B.3 Embedding space visualization

To better understand the alignment of features in the embedding space, we visualize a mapping of the image, DNA, and text embeddings in Figure 10.
We use UMAP \parencitemcinnes2018umap with a cosine similarity metric applied to the seen validation set to map the embeddings down to 2D space, and we mark points in the space based on their order classification.
We show the embedding space before alignment (a), with image and text (b), image and DNA (c), and all three modalities.
We see that after aligning the modalities, samples for the same order (indicated by hue), from different modalities (indicated by lightness) tend to overlap each other.
We observe that, for some orders, there are numerous outlier clusters spread out in the space. However, overall the orders demonstrate some degree of clustering together, with image and DNA features close to one another within their respective clusters. Furthermore, we note the text embeddings tend to lie within the Image or (more often) DNA clusters, suggesting a good alignment between text and other modalities.

### B.4 Attention map visualization

We provide more attention map visualization samples in Section B.2, including both success cases and failure cases.

## Appendix C Implementation details and hyperparameter selection

In this section, we provide experiments to validate the choice of hyperparameter settings and design choices we made for efficient training of our model.

**Table 14: *Trainable temperature.* Top-1 accuracy on the validation set for models contrastively trained with either a fixed or trainable temperature. We consider the performance when training for different durations (1 and 15 epochs).**
|  |  | Micro Top-1 Accuracy (%) | Macro Top-1 Accuracy (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Temperature | Epochs | DNA-to-DNA | Image-to-Image | Image-to-DNA | DNA-to-DNA | Image-to-Image | Image-to-DNA |
| Fixed | 1 | 97.2 | 62.6 | 27.4 | 93.1 | 41.6 | 11.0 |
| Trainable | 1 | 96.8 | 63.5 | 24.2 | 92.0 | 42.8 | 10.4 |
| Fixed | 15 | 98.0 | 74.2 | 57.8 | 94.7 | 52.2 | 32.4 |
| Trainable | 15 | 98.4 | 76.9 | 56.5 | 94.1 | 57.8 | 35.7 |

### C.1 Trainable vs fixed temperature

We compare using a fixed temperature for the contrastive loss vs using trainable temperature (Table 14). We find that using the trainable temperature helps improve the performance, provided the model is trained for enough epochs.

### C.2 Automatic mixed precision

**Table 15: *Automatic mixed precision.* Top-1 micro and macro accuracy on the validation set with models contrastively trained either with or without automatic mixed precision (AMP). We compare micro and macro Top-1 accuracies across different embedding alignments (image-to-image, DNA-to-DNA, and image-to-DNA). Both experiments have otherwise identical training conditions, including a batch size of 300 and 15 training epochs.**
|  | Micro Top-1 Accuracy (%) | Macro Top-1 Accuracy (%) | Memory | Training Time |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | DNA-to-DNA | Image-to-Image | Image-to-DNA | DNA-to-DNA | Image-to-Image | Image-to-DNA | CUDA (GB) $\downarrow$ | per epoch$\downarrow$ |
| –AMP | 98.07 | 74.97 | 57.38 | 95.18 | 54.47 | 32.22 | 75.54 | 4.03 hour |
| +AMP | 97.85 | 74.31 | 56.23 | 97.85 | 53.65 | 30.99 | 60.64 | 1.15 hour |

For efficient training with large batch sizes, we use automatic mixed precision (AMP) with the bfloat16 data type. The bfloat16 data type gives a similar dynamic range as float32 at reduced precision, and provides stable training with reduced memory usage.

We compare training with and without AMP in Table 15.
By applying AMP, we achieve comparable performance while using less memory. With AMP, the CUDA memory usage is reduced by about 15GB ($\sim$20%) and the training time by 3 hour per epoch ($\sim$75%). Although using full-precision (no AMP) yields slightly better accuracies, the lower memory usage and faster training time of AMP allows for more efficient experiments.
Additionally, the lower memory usage with AMP enables us to use larger batch sizes and is more effective for our experiments.

**Table 16: *Low-rank adaptation.* Top-1 micro/macro accuracy on the validation set for models contrastively trained with either with full fine-tuning or Low-Rank Adaptation (LoRA). We compare micro and macro Top-1 accuracies across different embedding alignments (image-to-image, DNA-to-DNA, and image-to-DNA). Both strategies use a batch size of 300 and are trained for a total of 15 epochs, allowing us to evaluate the impact of fine-tuning techniques on model performance and CUDA memory usage.**
|  | Micro Top-1 Accuracy (%) | Macro Top-1 Accuracy (%) | Memory | Training Time |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Fine-tuning Method | DNA-DNA | Image-Image | Image-DNA | DNA-DNA | Image-Image | Image-DNA | CUDA (GB) $\downarrow$ | per epoch$\downarrow$ |
| Full Fine-Tuning | 98.1 | 74.3 | 58.0 | 95.5 | 54.0 | 32.4 | 78.5GB | 4.03 hour |
| LoRA | 96.2 | 64.9 | 37.6 | 91.3 | 45.4 | 17.1 | 53.4GB | 2.98 hour |

### C.3 LoRA vs full fine-tuning

For efficient training, we also investigate the performance of using LoRA [hu2021lora] vs full fine-tuning. As shown in Table 16, we find that while LoRA does reduce the memory usage and training time, the performance is also notably worse, and thus we use full fine-tuning for the rest of our experiments.

### C.4 Batch size experiments

**Table 17: *Batch size.* Top-1 accuracy on the validation set for models contrastively trained with different batch sizes. Training at larger batch sizes helps improve accuracy at more fine-grained taxonomic levels such as genus and species.**
|  |  |  |  | Micro top-1 accuracy | Macro top-1 accuracy |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Batch<br>size | Alignment | DNA to DNA | Image to Image | Image to DNA | DNA to DNA | Image to Image | Image to DNA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Taxa | Img | DNA | Txt | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. | Seen | ​​​Unseen | H.M. |  |
| Order | 500 | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 99.6 | 99.6 | 99.6 | 99.6 | 99.2 | 99.4 | 100.0 | 92.9 | 96.3 | 99.6 | 98.5 | 99.0 | 99.1 | 75.8 | 85.9 |
|  | 1000 | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 99.7 | 99.6 | 99.6 | 99.7 | 99.2 | 99.4 | 100.0 | 100.0 | 100.0 | 99.0 | 93.7 | 96.3 | 99.1 | 75.3 | 85.6 |
|  | 1500 | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 99.7 | 99.6 | 99.7 | 99.7 | 99.2 | 99.4 | 100.0 | 92.8 | 96.3 | 99.5 | 94.3 | 96.8 | 99.6 | 73.6 | 84.6 |
|  | 2000 | ✓ | ✓ | ✓ | 100.0 | 100.0 | 100.0 | 99.7 | 99.7 | 99.7 | 99.7 | 99.2 | 99.4 | 100.0 | 100.0 | 100.0 | 99.1 | 95.9 | 97.5 | 99.2 | 73.9 | 84.7 |
| Family | 500 | ✓ | ✓ | ✓ | 99.9 | 99.6 | 99.8 | 95.6 | 93.9 | 94.7 | 94.8 | 86.2 | 90.3 | 100.0 | 97.6 | 98.7 | 88.8 | 79.3 | 83.8 | 83.5 | 52.5 | 64.5 |
|  | 1000 | ✓ | ✓ | ✓ | 100.0 | 99.7 | 99.8 | 96.3 | 94.2 | 95.2 | 96.0 | 86.9 | 91.2 | 99.9 | 98.0 | 99.0 | 90.2 | 80.5 | 85.1 | 87.9 | 56.1 | 68.5 |
|  | 1500 | ✓ | ✓ | ✓ | 100.0 | 99.6 | 99.8 | 96.5 | 94.3 | 95.4 | 96.7 | 87.3 | 91.8 | 100.0 | 97.3 | 98.6 | 92.0 | 81.3 | 86.3 | 91.7 | 53.9 | 67.9 |
|  | 2000 | ✓ | ✓ | ✓ | 100.0 | 99.7 | 99.9 | 96.6 | 94.5 | 95.5 | 96.6 | 87.4 | 91.8 | 100.0 | 98.6 | 99.3 | 92.0 | 81.2 | 86.3 | 90.0 | 56.2 | 69.2 |
| Genus | 500 | ✓ | ✓ | ✓ | 99.2 | 98.3 | 98.8 | 87.5 | 83.6 | 85.5 | 77.3 | 55.9 | 64.9 | 98.4 | 95.5 | 97.0 | 71.2 | 61.6 | 66.1 | 54.0 | 21.4 | 30.7 |
|  | 1000 | ✓ | ✓ | ✓ | 99.4 | 97.9 | 98.6 | 88.7 | 84.5 | 86.6 | 82.4 | 58.2 | 68.2 | 98.4 | 94.9 | 96.6 | 74.4 | 63.9 | 68.8 | 61.2 | 24.3 | 34.8 |
|  | 1500 | ✓ | ✓ | ✓ | 99.3 | 98.2 | 98.8 | 89.6 | 84.8 | 87.1 | 83.8 | 59.7 | 69.7 | 98.0 | 95.2 | 96.6 | 75.8 | 63.7 | 69.2 | 64.9 | 23.6 | 34.6 |
|  | 2000 | ✓ | ✓ | ✓ | 99.4 | 98.4 | 98.9 | 89.6 | 84.9 | 87.2 | 84.8 | 60.1 | 70.3 | 98.9 | 96.5 | 97.7 | 76.1 | 64.2 | 69.6 | 64.9 | 24.0 | 35.0 |
| Species | 500 | ✓ | ✓ | ✓ | 97.9 | 96.5 | 97.2 | 76.8 | 73.7 | 75.2 | 58.8 | 35.8 | 44.5 | 95.5 | 90.7 | 93.0 | 56.4 | 45.9 | 50.6 | 36.5 | 8.7 | 14.0 |
|  | 1000 | ✓ | ✓ | ✓ | 98.2 | 96.0 | 97.1 | 78.8 | 74.8 | 76.7 | 67.2 | 37.7 | 48.3 | 95.8 | 89.4 | 92.5 | 59.8 | 47.3 | 52.8 | 44.3 | 9.2 | 15.3 |
|  | 1500 | ✓ | ✓ | ✓ | 98.0 | 96.2 | 97.1 | 80.5 | 74.7 | 77.5 | 69.8 | 39.9 | 50.8 | 95.1 | 89.4 | 92.2 | 61.8 | 47.9 | 54.0 | 47.7 | 10.1 | 16.6 |
|  | 2000 | ✓ | ✓ | ✓ | 98.5 | 96.5 | 97.5 | 80.0 | 74.9 | 77.3 | 70.5 | 41.3 | 52.1 | 96.9 | 90.9 | 93.8 | 61.3 | 48.0 | 53.9 | 47.7 | 9.9 | 16.4 |

We conducted additional experiments to investigate the impact of training batch size (from 500 to 2000) on model performance.
The choice of batch size ordinarily does not have a major impact on performance when using supervised learning, but can have larger impact when training using contrastive learning since each positive pair is normalized against the pool of negative pairs appearing in the same training batch.

Our results, shown in Table 17, confirm that the classification accuracy improves as the batch size increases. The effect on is more pronounced for the harder, more fine-grained, taxonomic levels.
Due to resource limitations, we were only able to train up to a batch size of 2000.
We anticipate that using larger batch sizes would further enhance the classification accuracy of CLIBD, especially on more fine-grained taxonomic levels.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: 1) In Tab. 1, what is the point of comparing non-aligned embeddings? It seems intuitive that there should be no correspondence in the learned representations.

2) The attention visualization is not discussed in detail. Also, as mentioned above, what information comes from checking classification before alignment? Wouldn’t correct predictions be random in that case?
soundness: 4
strengths: 1) The paper is well-written and and organized. The proposed approach and the experiments are clearly described.

2) The proposed approach is effective in tackling the problem at hand, while being simpler than common alternatives in the field.
summary: The authors present a paper on a CLIP-based approach combining images, DNA barcodes and textual taxonomic description for biological classification (specifically of insects) in an open-set setting. Experimental results show that the method is effective and outperforms dual-modality contrastive learning approaches, as well as other approaches from the literature on this specific task.
weaknesses: 1) The paper presents no methodological novelty, and mostly applies existing techniques in a standard way to a particular use case.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Some suggestions and questions,

1.	Comparison with BioCLIP at different taxa levels: Going by the works claim, incorporation of DNA embeddings could help with classifying unseen species up to species level, but I’m guessing at higher taxonomic levels the BioCLIP performance should be comparable to CLIBD. It would be interesting to see a comparison.
2.	From Table 1, Image-to-DNA performance on seen species reduces going from (I+D) to (I+D+T), why is this happening? I would have expected the performance to improve.
3.	I’m also curios to know why BIOSCAN-5M was not used, considering that foundational models such as these can greatly benefit from more data.
4.	I personally felt Figure 7 was more informative to understand the data partitioning than Figure 2 (Just to consider in the future revisions).
soundness: 2
strengths: 1.	Incorporation of DNA as a modality to align the image embedding against instead of text is well motivated.
2.	Extensive experiments and ablations are provided.
summary: The approach proposes multimodal contrastive learning between image, DNA and text (taxonomic labels) as opposed to Image+TaxonomicLabel only approach followed by previous work (BioCLIP). The usage of DNA is claimed to be better because, (1) classifying unseen species would be difficult with a taxonomic-label-only model because the species name would not have been seen during training, (2) DNA is easier to obtain compared to taxonomic label which requires careful examination by human experts.
weaknesses: 1.	The accuracy when doing image to DNA on unseen species is not quite significant although it is better than BioCLIP’s approach of doing image to text. This indicates the image encoder is still not strong enough to generate a good DNA aligned embedding just from the image. Perhaps this can improve with more data.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Please see weaknesses for primary questions. 

Minor comments / questions (no need to respond):
* This work focuses on the cases where aligned data is available: images and DNA barcodes from the same individuals. It might be useful to extend the method to also take advantage of abundant unpaired data: images and DNA barcodes that are not paired. 
* Figure 2 could be clearer. Why are there different shades of blue and orange? Why don't the numbers in the boxes sum up to 36729? Generally, this figure did not aid my understanding (though many of my confusions were clarified in later text and figures). 
* It would be nice to have a chance-level baseline in Table 1. 
* Down the road, it might be interesting to try to integrate additional modalities, e.g. geospatial location (https://arxiv.org/abs/1906.05272).
soundness: 3
strengths: * The idea of jointly embedding DNA barcodes with images and taxonomic information is interesting.
* The experiments in the paper are extensive - there is a lot of technical content, and it's clear that a lot of effort went in to this work. There are many quantitative results, in addition to interesting qualitative results (e.g. Fig. 3, Fig. 5).
* The paper is very well-written. 
* The hyperparameters and training procedures are clearly spelled out.
summary: This paper proposes a CLIP-based method to jointly embed images, DNA barcodes, and taxonomic strings for different insect species. The method is evaluated on cross-modal matching and species classification, with qualitative and quantitative results provided. The paper compares against one external method (BIOCLIP) in one set of experiments and one external method (BZSL) in a second set of experiments. The key claim of the paper is that jointly learning from the modalities of images, DNA, and taxonomic information leads to stronger representations for downstream use.

Note: Score increased after rebuttal.
weaknesses: My major issue with the paper is missing baselines:
* The paper compares their multimodal representation learning approach against the unimodal pretrained models they start with. They show that their method is better, and conclude that multimodality is important. However, this claim is not justified - couldn't the benefit be from the additional training each modality received? It seems to me that the fair comparison would be to take the unimodal models and run unimodal CLIP-style training (with a similar computation / # steps budget) for each. If the multimodal model beats these CLIP-fine-tuned unimodal models, then that provides stronger evidence of the benefit of multimodal learning. 
* For the image encoder, wouldn't a model pretrained on BIOSCAN-1M be a more appropriate starting point / baseline than an ImageNet-pretrained model? 

The paper should address a few items that were not discussed:
* Are there confounders we need to worry about in this data, e.g. the facility the data was collected by? 
* The DNA to DNA matching results are very high, but shouldn't we expect this? Would simple homology methods based on string matching do a very good job at this task? What is the benefit of using deep embeddings to solve this problem? 
* Doesn't adding a modality increase the number of steps of training the model receives? Couldn't this be partly responsible for the differences between 1, 2, and 3 modalities in Table 1? 
* Image-DNA and Image-text matching don't seem very good based on Fig. 4, with the average being pulled up by a few good cases. What are those cases, and why are they different? Is there any insight to be gained there? 

A few claims made in the paper were not clear to me:
* The paper claims that "BIOCLIP... requires taxonomic labels to be available in order to obtain text descriptions. These labels can be expensive and time-consuming to obtain" and that "DNA barcodes can be obtained at scale more readily than taxonomic labels". These claims are not intuitive to me. How much does DNA barcoding cost per individual, compared to the cost of having an expert inspect an image to identify the species? 
* The training data for BarcodeBERT is claimed to be "different from, but highly similar to" the data used in this paper  - can you expand on what this means, and the implications for the results presented?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Details about the reference database is missing. How many images and DNA barcodes are present in the reference database during inference?
soundness: 1
strengths: - The motivation and problem formulation is sound and interesting.
- The proposed model fuses images, text and DNA into a contrastive embedding space enabling zero-shot image classification of unseen species.
- The flow of paper and writing is good in general.
summary: The paper proposes CLIPBD, a tri-modal embedding space consisting of image, text and DNA of insect specimens. CLIPBD is trained using a three-way contrastive learning objective between image, text and DNA on the BIOSCAN-1M dataset. Using a reference database of images and DNA, CLIPBD is able to classify images of seen and unseen species. Results reported in the paper show superior performance as compared to BioCLIP and other state-of-the-art DNA encoder models.
weaknesses: - The abstract claims the paper is the first to use contrastive learning to fuse DNA and image. However, there are existing works [1, 2, 3] which have done this for other applications and they should be discussed in the related works.
- The claim that *DNA is a better target than taxonomic labels* (**Line 316**) is highly questionable. This claim is reiterated in **Lines 359-362**. The paper clearly mentions that majority of the BIOSCAN-1M dataset does not have taxonomic labels. In fact only 3.36% pretraining data has labels upto the species level (**Line 321**). It is clearly seen from **Table-1** that aligning with taxonomic labels outperforms aligning with DNA at the order level. I believe if the authors used an **unbiased dataset** containing the same proportions of DNA labels and taxonomic labels, the results would have been similar if not worse. This is more of a problem of the dataset and not the modality itself. 
- Following the previous point, why does Image-to-DNA retrieval performance improve at the species level when aligning all three modalities (**Table 1**)?
- For inference to work on unseen species during training, the framework assumes that their DNA and/or images are available in the lookup database. This is an unrealistic assumption. If the images and DNA are already available for unseen species, they might as well should have been used for training.
- Limited technical novelty considering no new representation learning technique has been proposed. The paper uses an existing dataset containing 1M insect specimens and unbalanced DNA and taxonomic labels, raising questions on the effectiveness of the method on other real-world datasets. Majority of the experiments and evaluations are only shown for a single dataset.
- The authors correctly pointed that BioCLIP was trained on diverse set of species including natural images. However, one easy way to utilize the BioCLIP embedding space would be to align the DNA modality with frozen BioCLIP vision and text encoders. Have the authors compared their method with this ImageBind-style training?


References

[1] Taleb, Aiham, et al. "Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Xie, Ronald, et al. "Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning." Advances in Neural Information Processing Systems 36 (2023).

[3] Min, Wenwen, et al. "Multimodal contrastive learning for spatial gene expression prediction using histology images." arXiv preprint arXiv:2407.08216 (2024).

Meta-review: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. Two reviewers recommend marginal acceptance, and two keep the rejection decision after discussion. The AC agreed this work is valuable to the ICLR community. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.

Decision: Accept (Poster)