CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale

Authors: ZeMing Gong, Austin Wang, Xiaoliang Huo, Joakim Bruslund Haurum, Scott C. Lowe, Graham W. Taylor, Angel X Chang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: multimodal learning, contrastive learning, dna barcodes, taxonomic classification, fine-grained classification, biodiversity monitoring

Abstract:
Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multi-modal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.

Decision: accept (poster)
Reviewer scores: 3, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1729894278917

thread_type:
official_review

<thread_object>
creator: reviewer_jvaw

<content>
summary:
The paper proposes CLIPBD, a tri-modal embedding space consisting of image, text and DNA of insect specimens. CLIPBD is trained using a three-way contrastive learning objective between image, text and DNA on the BIOSCAN-1M dataset. Using a reference database of images and DNA, CLIPBD is able to classify images of seen and unseen species. Results reported in the paper show superior performance as compared to BioCLIP and other state-of-the-art DNA encoder models.

strengths:
- The motivation and problem formulation is sound and interesting.
- The proposed model fuses images, text and DNA into a contrastive embedding space enabling zero-shot image classification of unseen species.
- The flow of paper and writing is good in general.

weaknesses:
- The abstract claims the paper is the first to use contrastive learning to fuse DNA and image. However, there are existing works [1, 2, 3] which have done this for other applications and they should be discussed in the related works.
- The claim that *DNA is a better target than taxonomic labels* (**Line 316**) is highly questionable. This claim is reiterated in **Lines 359-362**. The paper clearly mentions that majority of the BIOSCAN-1M dataset does not have taxonomic labels. In fact only 3.36% pretraining data has labels upto the species level (**Line 321**). It is clearly seen from **Table-1** that aligning with taxonomic labels outperforms aligning with DNA at the order level. I believe if the authors used an **unbiased dataset** containing the same proportions of DNA labels and taxonomic labels, the results would have been similar if not worse. This is more of a problem of the dataset and not the modality itself. 
- Following the previous point, why does Image-to-DNA retrieval performance improve at the species level when aligning all three modalities (**Table 1**)?
- For inference to work on unseen species during training, the framework assumes that their DNA and/or images are available in the lookup database. This is an unrealistic assumption. If the images and DNA are already available for unseen species, they might as well should have been used for training.
- Limited technical novelty considering no new representation learning technique has been proposed. The paper uses an existing dataset containing 1M insect specimens and unbalanced DNA and taxonomic labels, raising questions on the effectiveness of the method on other real-world datasets. Majority of the experiments and evaluations are only shown for a single dataset.
- The authors correctly pointed that BioCLIP was trained on diverse set of species including natural images. However, one easy way to utilize the BioCLIP embedding space would be to align the DNA modality with frozen BioCLIP vision and text encoders. Have the authors compared their method with this ImageBind-style training?


References

[1] Taleb, Aiham, et al. "Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Xie, Ronald, et al. "Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning." Advances in Neural Information Processing Systems 36 (2023).

[3] Min, Wenwen, et al. "Multimodal contrastive learning for spatial gene expression prediction using histology images." arXiv preprint arXiv:2407.08216 (2024).

questions:
Details about the reference database is missing. How many images and DNA barcodes are present in the reference database during inference?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729902947736

thread_type:
official_review

<thread_object>
creator: reviewer_cfwv

<content>
summary:
This paper proposes a CLIP-based method to jointly embed images, DNA barcodes, and taxonomic strings for different insect species. The method is evaluated on cross-modal matching and species classification, with qualitative and quantitative results provided. The paper compares against one external method (BIOCLIP) in one set of experiments and one external method (BZSL) in a second set of experiments. The key claim of the paper is that jointly learning from the modalities of images, DNA, and taxonomic information leads to stronger representations for downstream use.

Note: Score increased after rebuttal.

strengths:
* The idea of jointly embedding DNA barcodes with images and taxonomic information is interesting.
* The experiments in the paper are extensive - there is a lot of technical content, and it's clear that a lot of effort went in to this work. There are many quantitative results, in addition to interesting qualitative results (e.g. Fig. 3, Fig. 5).
* The paper is very well-written. 
* The hyperparameters and training procedures are clearly spelled out.

weaknesses:
My major issue with the paper is missing baselines:
* The paper compares their multimodal representation learning approach against the unimodal pretrained models they start with. They show that their method is better, and conclude that multimodality is important. However, this claim is not justified - couldn't the benefit be from the additional training each modality received? It seems to me that the fair comparison would be to take the unimodal models and run unimodal CLIP-style training (with a similar computation / # steps budget) for each. If the multimodal model beats these CLIP-fine-tuned unimodal models, then that provides stronger evidence of the benefit of multimodal learning. 
* For the image encoder, wouldn't a model pretrained on BIOSCAN-1M be a more appropriate starting point / baseline than an ImageNet-pretrained model? 

The paper should address a few items that were not discussed:
* Are there confounders we need to worry about in this data, e.g. the facility the data was collected by? 
* The DNA to DNA matching results are very high, but shouldn't we expect this? Would simple homology methods based on string matching do a very good job at this task? What is the benefit of using deep embeddings to solve this problem? 
* Doesn't adding a modality increase the number of steps of training the model receives? Couldn't this be partly responsible for the differences between 1, 2, and 3 modalities in Table 1? 
* Image-DNA and Image-text matching don't seem very good based on Fig. 4, with the average being pulled up by a few good cases. What are those cases, and why are they different? Is there any insight to be gained there? 

A few claims made in the paper were not clear to me:
* The paper claims that "BIOCLIP... requires taxonomic labels to be available in order to obtain text descriptions. These labels can be expensive and time-consuming to obtain" and that "DNA barcodes can be obtained at scale more readily than taxonomic labels". These claims are not intuitive to me. How much does DNA barcoding cost per individual, compared to the cost of having an expert inspect an image to identify the species? 
* The training data for BarcodeBERT is claimed to be "different from, but highly similar to" the data used in this paper  - can you expand on what this means, and the implications for the results presented?

questions:
Please see weaknesses for primary questions. 

Minor comments / questions (no need to respond):
* This work focuses on the cases where aligned data is available: images and DNA barcodes from the same individuals. It might be useful to extend the method to also take advantage of abundant unpaired data: images and DNA barcodes that are not paired. 
* Figure 2 could be clearer. Why are there different shades of blue and orange? Why don't the numbers in the boxes sum up to 36729? Generally, this figure did not aid my understanding (though many of my confusions were clarified in later text and figures). 
* It would be nice to have a chance-level baseline in Table 1. 
* Down the road, it might be interesting to try to integrate additional modalities, e.g. geospatial location (https://arxiv.org/abs/1906.05272).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730532099940

thread_type:
official_review

<thread_object>
creator: reviewer_s416

<content>
summary:
The approach proposes multimodal contrastive learning between image, DNA and text (taxonomic labels) as opposed to Image+TaxonomicLabel only approach followed by previous work (BioCLIP). The usage of DNA is claimed to be better because, (1) classifying unseen species would be difficult with a taxonomic-label-only model because the species name would not have been seen during training, (2) DNA is easier to obtain compared to taxonomic label which requires careful examination by human experts.

strengths:
1.	Incorporation of DNA as a modality to align the image embedding against instead of text is well motivated.
2.	Extensive experiments and ablations are provided.

weaknesses:
1.	The accuracy when doing image to DNA on unseen species is not quite significant although it is better than BioCLIP’s approach of doing image to text. This indicates the image encoder is still not strong enough to generate a good DNA aligned embedding just from the image. Perhaps this can improve with more data.

questions:
Some suggestions and questions,

1.	Comparison with BioCLIP at different taxa levels: Going by the works claim, incorporation of DNA embeddings could help with classifying unseen species up to species level, but I’m guessing at higher taxonomic levels the BioCLIP performance should be comparable to CLIBD. It would be interesting to see a comparison.
2.	From Table 1, Image-to-DNA performance on seen species reduces going from (I+D) to (I+D+T), why is this happening? I would have expected the performance to improve.
3.	I’m also curios to know why BIOSCAN-5M was not used, considering that foundational models such as these can greatly benefit from more data.
4.	I personally felt Figure 7 was more informative to understand the data partitioning than Figure 2 (Just to consider in the future revisions).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641133375

thread_type:
official_review

<thread_object>
creator: reviewer_ujh5

<content>
summary:
The authors present a paper on a CLIP-based approach combining images, DNA barcodes and textual taxonomic description for biological classification (specifically of insects) in an open-set setting. Experimental results show that the method is effective and outperforms dual-modality contrastive learning approaches, as well as other approaches from the literature on this specific task.

strengths:
1) The paper is well-written and and organized. The proposed approach and the experiments are clearly described.

2) The proposed approach is effective in tackling the problem at hand, while being simpler than common alternatives in the field.

weaknesses:
1) The paper presents no methodological novelty, and mostly applies existing techniques in a standard way to a particular use case.

questions:
1) In Tab. 1, what is the point of comparing non-aligned embeddings? It seems intuitive that there should be no correspondence in the learned representations.

2) The attention visualization is not discussed in detail. Also, as mentioned above, what information comes from checking classification before alignment? Wouldn’t correct predictions be random in that case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732137648438

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their thoughtful feedback. We appreciate that the reviewers found the problem formulation and motivation interesting (JVaW), especially with the use of DNA (s416, CfWv) and that our proposed approach simply and effectively tackled the problem (Ujh5). The reviewers found our paper and training procedure was well-organized (Ujh5, CfWv, JVaW) and that we had extensive experiments and results for our approach (s416, CfWv).

We provide responses to shared comments below and further direct responses to each respective review:

**Methodological Novelty [Ujh5, JVaW]**: We believe our work provides new value and insights both for biodiversity and for the ML community. For biodiversity, our work demonstrates the possibility of leveraging DNA barcodes for image-based taxonomic classification, significantly improving the workflow of classifying species and allowing for novel downstream tasks that would not be possible without a shared representation space. Such tasks are important for monitoring species-at-risk and evaluating/optimizing conservation efforts. For the ML community, our work is a first yet important step in showing how contrastive learning can leverage modalities of varying information density and scale to improve both unimodal and multimodal retrieval performance. Obtaining human labels is extremely costly in this domain due to the level of expertise required to classify samples to species level. Depending on the sample, it may take a few minutes to half an hour to label a sample from viewing an image, or it may require several hours to examine or dissect the specimen. We propose a solution where a dense modality (barcodes) that can be more easily collected at scale can complement a sparse, human-interpretable modality (taxonomic labels) for training a third modality (vision) ideal for deployment. We anticipate that this problem structure exists elsewhere in ML and our solution generalizes to those settings. Application areas that have such structure are diverse and include medical imaging such as with X-rays or colonoscopies, robotics with tactile sensory data, and key identification in music. We chose to focus specifically on biodiversity because we believe it is itself an important and rich problem and wanted to prioritize depth and exploration of a single problem over shallow coverage of multiple datasets.

**DNA barcode vs taxonomic labels as alignment target [CfWv,JVaW]**: Our experiments show that by aligning to DNA barcodes, instead of taxonomic labels, we learn representations that are better at taxonomic classification (especially at the more challenging genus and species levels). Reviewers CfWv and JVaW had questions about the cost (CfWv) and value (JVaW) of obtaining DNA barcodes vs taxonomic labels. We want to highlight that DNA barcodes can be obtained via a systematic and parallel process while taxonomic labels require manual annotation by scientific experts. In fact, it can be extremely challenging even for experts to classify an insect down to the species level—that is the underlying reason why a majority of the records lack species-level labels. In addition, a specimen’s DNA barcode provides intrinsic identifying markers, while taxonomic labels are designed by humans. The DNA barcodes are a property of the specimen itself, while taxonomic labels are provided by annotators and subject to ambiguity and interpretation. In this work, we are attempting to assert that by aligning image and DNA barcode embeddings, we can train models that do not rely on these costly human-provided taxonomic labels during training. Please also see individual responses for details on cost (CfWv) and challenges of obtaining taxonomic labels (JVaW). 

**Additional experiments**: We have improved our manuscript with additional experiments comparing against unimodal contrastive learning (with SimCLR). We included additional results comparing against BioCLIP at higher-level taxa and a chance-level baseline . We have also clarified a few figures and explanations, including more statistics and details about the data used for training and evaluation for each experiment as well as clarified explanations for the attention visualizations. Lastly, we have added Figure 9 to demonstrate the reproducibility of our results across multiple training runs. We mark main changes in magenta.

We thank all of the reviewers for their time in reading our paper and contributing to the discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732738473135

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Changes

comment:
We thank the reviewers for all of their feedback and comments on our paper.  We have updated the paper with the following:

**Main changes**
1. Experiment comparing initializing CLIBD with pretrained image-encoder on BIOSCAN-1M (Appendix B.1).  Thanks to **R-CfWv** for the suggestion.
   1. We used unimodal contrastive learning (e.g. SimCLR with data augmentation) to first fine-tune a pretrained image-encoder (ViT-B) on BIOSCAN-1M.  Our initial experiments show that using unimodal contrastive learning, we can improve our model performance over the initial ViT-B, but it did not perform as well as our CLIBD multimodal models.
   2. By initializing our CLIBD using this pretrained model (Appendix B.1.1), we obtain higher accuracy in image-to-image and image-to-DNA retrieval at finer taxonomic levels than when initializing with the ViT-B pretrained only on ImageNet.

| Taxa        | SimCLR   | DNA Seen | DNA Unseen | DNA H.M. | Img Seen | Img Unseen | Img H.M. | I-D Seen | I-D Unseen | I-D H.M. |
|-------------|----------|----------|------------|----------|----------|------------|----------|----------|------------|----------|
| Order       | ✗        | **100.0**| **100.0**  | **100.0**| **99.7** | 94.4       | 97.0     | 99.4     | **88.5**   | **93.6** |
|             | ✔️       | **100.0**| **100.0**  | **100.0**| **99.7** | **98.9**   | **99.3** | **99.6** | 75.3       | 84.8     |
| Family      | ✗        | **100.0**| 98.3       | 99.1     | 90.9     | 81.8       | 86.1     | 90.8     | 50.1       | 64.6     |
|             | ✔️       | **100.0**| **99.5**   | **99.7** | **92.2** | **85.8**   | **88.9** | **92.6** | **52.1**   | **66.7** |
| Genus       | ✗        | 98.2     | **94.7**   | **96.4** | 74.6     | 60.4       | 66.8     | 70.6     | 20.8       | 32.1     |
|             | ✔️       | **98.5** | 94.5       | **96.4** | **77.2** | **63.5**   | **69.6** | **73.5** | **23.4**   | **35.5** |
| Species     | ✗        | 95.6     | **90.4**   | **92.9** | 59.3     | 45.0       | 51.2     | 51.6     | 8.6        | 14.7     |
|             | ✔️       | **95.7** | 89.8       | 92.7     | **62.2** | **47.9**   | **54.1** | **56.5** | **9.7**    | **16.5** |

2. Experiment comparing ImageBind style training with frozen BioCLIP image/text encoders (Appendix B.1.1).  Thanks to **R-JVaW** for the suggestion.
   1. We experimented with starting with the BioCLIP image and text encoder and keeping them frozen while aligning the DNA encoder to the image encoder (or image and text encoder together). We found while this was very effective in aligning the DNA encoder, the image encoder performance still lagged behind that of the CLIBD model.

| Taxon. level | Model          | Img     | DNA     | Txt     | I-I Seen | I-I Unseen | I-I H.M. | I-D Seen | I-D Unseen | I-D H.M. | I-T Seen | I-T Unseen | I-T H.M. |
|--------------|----------------|---------|---------|---------|----------|------------|----------|----------|------------|----------|----------|------------|----------|
| Order        | Image          | ❄️       | ✗       | ❄️       | **100.0**| **100.0**  | **100.0**| 88.5     | 86.0       | 87.2     | 87.8     | 64.8       | 74.6     |
|              | Image & Text   | ❄️       | ✗       | ❄️       | **100.0**| **100.0**  | **100.0**| 88.5     | 86.0       | 87.2     | **88.7** | **71.9**   | **79.4** |
| Family       | Image          | ❄️       | ✗       | ❄️       | **100.0**| **97.6**   | **98.8** | 84.3     | 68.4       | 75.5     | **79.6** | 43.1       | 55.9     |
|              | Image & Text   | ❄️       | ✗       | ❄️       | **100.0**| 97.2       | 98.6     | 84.3     | 68.4       | 75.5     | 78.1     | **45.4**   | **57.5** |
| Genus        | Image          | ❄️       | ✗       | ❄️       | 98.2     | 92.9       | 95.5     | 62.6     | 47.1       | 53.7     | 56.0     | 17.9       | 27.1     |
|              | Image & Text   | ❄️       | ✗       | ❄️       | **98.9** | **93.2**   | **96.0** | 62.6     | 47.1       | 53.7     | **56.5** | **19.7**   | **29.2** |
| Species      | Image          | ❄️       | ✗       | ❄️       | 95.1     | **87.9**   | 91.3     | 44.1     | 32.4       | 37.4     | **39.5** | 7.4        | 12.5     |
|              | Image & Text   | ❄️       | ✗       | ❄️       | **95.9** | 88.6       | **92.1** | 44.1     | 32.4       | 37.4     | 38.0     | **8.5**    | **13.9** |

3. Discussion of utility and limitations of using DNA barcodes (Appendix D). This was motivated by **R-JVaW**’s questions concerning cost of acquisition, compute and storage costs of DNA barcodes compared to the costs associated with taxonomic labeling. We also updated the conclusion to acknowledge the limitations of datasets with DNA barcodes and highlight potential future directions.

(cont.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732738646167

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Contributions

comment:
**Contributions**

We want to reiterate that our main contribution is using CLIP-style contrastive training to align image and DNA barcode embeddings.  Our experiments on BIOSCAN-1M demonstrate that by aligning to DNA barcodes, we can improve image representations for taxonomic classification.  In addition, by adding sparse taxonomic labels as an alignment target, the performance of our model is further improved.  We also show that DNA barcodes can potentially be a stronger target for alignment than taxonomic labels that are not always available.  This is not necessarily an obvious finding, as shown by our discussion with R-JVaW.

**Stimulate follow up work on using multi-modal learning with DNA barcodes**

Biodiversity monitoring is an important problem.  Recently, there has been an increasing number of datasets (AMI [1], TreeOfLife-10M [2], BioTrove [3]) to engage ML experts to apply ML techniques to assist in biodiversity efforts.  Some are datasets aggregated from different sources (TreeOfLife-10M), while other datasets target a specific workflow (AMI, BIOSCAN).  While our experiments are currently limited to the BIOSCAN-1M and INSECT dataset, we want to emphasize that the BIOSCAN-1M dataset is based on data from active scientists in biodiversity monitoring ([Centre for Biodiversity Genomics](https://biodiversitygenomics.net/)).  It was contributed to help spur investigation of the application of ML techniques on DNA barcodes for various tasks, including taxonomic classification.  In our work, we took the first step in combining information from DNA barcodes, images, and taxonomic labels using CLIP-style contrastive pretraining.  While there are many different multimodal learning and other innovative techniques that can be applied, we believe our work can stimulate further research in this area.  

[1] Jain et al. "Insect Identification in the Wild: The AMI Dataset." ECCV 2024.

[2] Stevens et al. "Bioclip: A vision foundation model for the tree of life. "CVPR  2024.

[3] Yang et al. "BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity." NeurIPS D&B Track  2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. Two reviewers recommend marginal acceptance, and two keep the rejection decision after discussion. The AC agreed this work is valuable to the ICLR community. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.