BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models

## Abstract

Abstract This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a species, each capturing certain biological traits. Incorporating captions during training encourages alignment with this shared latent structure, emphasizing potentially diagnostic characters while suppressing spurious correlations. The main challenge, however, lies in obtaining faithful, instance-specific captions at scale. This requirement has limited the utilization of natural language supervision in organismal biology compared with many other scientific domains. We complement this gap by generating synthetic captions with multimodal large language models (MLLMs), guided by Wikipedia-derived visual information and taxon-tailored format examples. These domain-specific contexts help reduce hallucination and yield accurate, instance-based descriptive captions. Using these captions, we train BioCAP ( i.e ., BioCLIP with Captions), a biological foundation model that captures rich semantics and achieves strong performance in species classification and text-image retrieval. These results demonstrate the value of descriptive captions beyond labels in bridging biological images with multimodal foundation models 1 1 1 Models, data, and code available at imageomics.github.io/biocap . .

## 1 Introduction

Multimodal foundation models learn from vast datasets of paired visual and textual inputs (Radford et al., 2021; Liu et al., 2023; Zhai et al., 2023; Hurst et al., 2024). They demonstrate strong generalization across various downstream tasks, such as classification and visual question answering (VQA) (Yue et al., 2024; Mai et al., 2025). In general domains, the web provides a nearly limitless supply of such paired supervision (Gadre et al., 2023; Schuhmann et al., 2022). In scientific applications, however, such resources are unevenly distributed. Biomedical imaging benefits from clinical reports and radiology notes associated with image instances that offer detailed natural language supervision (Wang et al., 2022; Li et al., 2023; Zhang et al., 2024). By contrast, many other scientific domains—including organismal biology, astronomy, geology, and material science (Stevens et al., 2024; Parker et al., 2024; Vivanco Cepeda et al., 2023; Harnik et al., 2025)—often lack instance-level textual descriptions and must solely rely on symbolic labels (*e.g*., species names). This leaves the potential of natural-language captions untapped for scientific multimodal learning.

In this work, we focus on organismal biology. From the representation learning perspective, each species can be described by an underlying latent vector in the morphospace, which encodes its ground-truth biological characteristics, *i.e*., traits (Budd, 2021). Images and descriptive captions can be viewed as two projections of this latent vector. Each captures certain traits while also introducing noise (*e.g*., pose, lighting). Aligning both modalities encourages the model to recover the shared latent structure and focus on potentially diagnostic characters, thereby suppressing reliance on spurious correlations to noise. While the potential exists, in practice, the effectiveness relies on the availability of captions that are both instance-specific and faithful. Noisy or hallucinated captions can, instead, introduce contradictory signals that harm multimodal alignment (Huang et al., 2021).

Biological research has produced massive repositories of organismal images, but most are annotated only with species names (Stevens et al., 2023b; Yang et al., 2024). Collecting instance-based captions at scale is inherently challenging as expert-level annotation for millions of images demands exhaustive human labor (Van Horn et al., 2018). A natural solution is to leverage recent progress in multimodal large language models (MLLMs) to automatically generate image-aligned captions (Fan et al., 2023; Lai et al., 2024). However, biological species differentiation often depends on subtle morphological details. Without proper guidance, MLLMs tend to hallucinate about these details due to the aforementioned noise in the images.
[Figure 1](https://arxiv.org/html/2510.20095v2#S0.F1) shows such an example, where the model incorrectly describes the color of the Calliope Hummingbird.

Upon this observation, we suggest that domain knowledge has to be incorporated in the context. Specifically, we collect species-level visual information from Wikipedia (Wikipedia contributors, 2025) as ground-truth characters. As the model needs to attend to different traits to describe different species, we also curate format examples based on taxonomic classes to encourage explicit focus. These domain-specific contexts help MLLMs generate accurate and trait-focused descriptions grounded in the input image. As shown in [Figure 1](https://arxiv.org/html/2510.20095v2#S0.F1), contextualizing the model with Wikipedia-derived visual information and taxon-tailored format examples corrects the earlier hallucination and substitutes it with accurate biological details.

Building on these curated contexts, we generate instance-level synthetic captions for the large-scale TreeOfLife-10M dataset (Stevens et al., 2023b; Van Horn & Mac Aodha, 2021; Gharaee et al., 2023; Encyclopedia of Life, 2023). We then train the BioCAP model (*i.e*., BioCLIP with Captions) with species names and captions as complementary supervision. As shown on the right of [Figure 1](https://arxiv.org/html/2510.20095v2#S0.F1), BioCAP demonstrates a rich understanding of biological semantics.
Further comparisons with BioCLIP (trained without captions) and the initial CLIP checkpoint in [Figure 6](https://arxiv.org/html/2510.20095v2#S4.F6) show improved semantic alignment of BioCAP.
Quantitatively, BioCAP is evaluated on species classification and biological natural-language benchmarks, where it outperforms BioCLIP by $8.8\%$ and $21.3\%$, respectively. These results answer the core question of this work: descriptive captions, when grounded in biological knowledge, provide essential supervision that bridges organismal images with multimodal foundation models.

## 2 Related Work

### 2.1 Multimodal Foundation Models for Scientific Images

Multimodal large language models (MLLMs) such as GPT-4o (Hurst et al., 2024) and LLaVA (Liu et al., 2023) have demonstrated a powerful ability in tasks involving images and natural language like captioning, VQA, and open-ended reasoning.
This paradigm has been successfully extended to specialized domains: LLaVA-Med (Li et al., 2023) adapts the LLaVA framework to radiology for domain-specific VQA, ChemVLM (Li et al., 2025) reasons about molecular structures, and BiomedGPT (Zhang et al., 2024) unifies diverse biomedical tasks under a single multimodal backbone.

In parallel, CLIP-style contrastive frameworks have also been adapted to a variety of scientific imagery domains.
BioCLIP (Stevens et al., 2024; Gu et al., 2025) and BioTrove-CLIP (Yang et al., 2024) align organismal images with taxonomic names. CLIBD (Gong et al., 2025) uses DNA barcodes as supervision for biodiversity tasks.
MedCLIP (Wang et al., 2022) extracts standardized UMLS entities from radiology reports, and MoleCLIP (Harnik et al., 2025) clusters molecular fingerprints in chemistry.
CLIP allows for more flexible supervision signals when natural language resources are not available for the specific data.
Yet, it also leaves the value of natural language descriptions untapped for these scientific domains.
We provide a detailed discussion with concurrent work in §[F](https://arxiv.org/html/2510.20095v2#A6).

### 2.2 Synthetic Caption

One promising solution to overcome the lack of instance-level annotations is to generate synthetic captions that provide fine-grained, image-grounded supervision.
Recent work highlights the value of high-quality text-image pairs for CLIP-style training.
BLIP (Li et al., 2022) and LLaVA (Liu et al., 2023) are widely used to produce semantically richer descriptions from large-scale web text-image collections.
Building on this idea, FG-CLIP (Xie et al., 2025) produces long captions and constructs region-specific annotations, enabling fine-grained alignment.
The caption’s semantic relevance and granularity are critical for modality alignment. VeCLIP (Lai et al., 2024) and CapsFusion (Yu et al., 2024) refine captions with large language models to achieve more semantically aligned supervision. ALIP (Yang et al., 2023) adopts a bi-path strategy that combines web text with synthetic captions. LaCLIP (Fan et al., 2023) leverages LLMs to generate multiple paraphrases of each caption. These approaches highlight the utility of synthetic captions for improving modality alignment.

Beyond pretraining, synthetic captions have also been applied to specialized settings. Hyp-OW (Doan et al., 2024) employs dense region-level captions to expand visual vocabulary and improve open-world detection. In the MLLM space, MiniGPT-4 (Zhu et al., 2024) and LLaVA rely on GPT-4-generated image descriptions and instruction-following data to equip models with conversational capabilities.

Despite these advances, most existing efforts target general-domain imagery and emphasize caption quality or diversity. Organismal biology introduces additional challenges of fine-grained categorization and domain-specific fidelity. To address these issues, we incorporate domain knowledge into the generation process to reduce hallucination and produce faithful, instance-specific captions.

## 3 Method

There have been vast curations of organismal biology images through the efforts of biodiversity researchers and citizen scientists. Many of them have reliable species labels, geolocation metadata, and in some cases, DNA barcodes. However, descriptive captions are largely absent in these datasets.
Such captions, which describe human-interpretable visual traits and ecological context, provide complementary information that species labels alone cannot capture.
Yet at the same time, their usefulness is dependent on being faithful and instance-specific. Such requirements make them difficult to collect at scale, leaving much of their potential untapped.
In this work, we investigate descriptive captions as an additional source of training supervision for biological multimodal foundation models.
We first illustrate that descriptive captions, when grounded in biological knowledge, help align images and their species labels.
Given the absence of large-scale curated resources, we then explore the use of MLLMs to generate instance-based synthetic captions for biological images.

### 3.1 BioCAP

Let ${\bm{x}}$ be an image embedding, $y\in\mathcal{Y}$ its taxonomic label, and ${\bm{c}}$ the corresponding caption embedding.
We train BioCAP (*i.e*., BioCLIP with Captions) with two text views: the taxonomic label and the descriptive caption.
Both views are encoded by the text encoder and aligned with the image embedding.
Assume an underlying trait latent vector ${\bm{z}}^{*}$ in the morphospace encodes the phenotypic characters of taxon $y$. From a causal generation perspective, the image ${\bm{x}}$ and caption ${\bm{c}}$ are two noisy observations of the same ${\bm{z}}^{*}$:

$$ $y\rightarrow{\bm{z}}^{*}\rightarrow\{{\bm{x}},{\bm{c}}\},\quad{\bm{x}}=g({\bm{z}}^{*},{\bm{\epsilon}}),\quad{\bm{c}}=h({\bm{z}}^{*},{\bm{\epsilon}}),$ $$

where ${\bm{\epsilon}}$ represents spurious environmental factors (*e.g*., pose, lighting) that can lead to inaccurate observation.
Both the image and the caption capture certain aspects of the ground-truth traits, while being influenced by noise.
When captions are involved in the training, the contrastive objective encourages the image embeddings to emphasize trait-relevant factors of the object that are shared with captions, thereby reducing the influence of spurious environmental factors ${\bm{\epsilon}}$.
Hence, the learned representation becomes closer to the latent trait vector ${\bm{z}}^{*}$ when the caption faithfully reflects visible and potentially diagnostic characters of the species.
On the contrary, if the caption captures too much noise instead of the correct traits, the supervision may misguide optimization and degrade classification performance.
We further elaborate on this in §[A](https://arxiv.org/html/2510.20095v2#A1).

Separated visual projectors.
Given that supervision in our setting is heterogeneous, we introduce two separate visual projectors after the shared visual encoder for taxonomic labels and captions, respectively (Wang et al., 2024; Ranzinger et al., 2024).
When the paired text input is a taxonomic label, only the visual embedding after the taxonomy projector is matched, and vice versa.
The visual encoder and the text encoder remain shared across both pathways.

Figure: Figure 2: The pipeline of collecting domain-specific context for MLLMs. We query Wikipedia with the scientific name to get the corresponding webpage. After validating the full taxonomic rank, we process the descriptive excerpts with MLLMs to extract visual information. For each taxonomic class, we randomly select up to three species and curate format examples through Gemini Deep Research. These contexts help MLLMs generate accurate and grounded descriptive captions.
Refer to caption: x2.png

### 3.2 Synthetic Caption

Based on the above discussion, the captions need to provide thorough and faithful views of the images to assist alignment. Traits that represent the species should be highlighted, while the information not visible or influenced by environmental factors should not be hallucinated. As shown in [Figure 1](https://arxiv.org/html/2510.20095v2#S0.F1), when solely conditioned by the images, MLLMs tend to generate wrong descriptions due to complicated environmental factors.
Therefore, we propose to collect domain-specific contexts to regularize the caption generation.
Specifically, we use two context sources: Wikipedia-derived visual information and taxon-tailored format examples. We present the pipeline for collecting these contexts in [Figure 2](https://arxiv.org/html/2510.20095v2#S3.F2). The prompts used to collect captions and detailed processes are listed in §[B](https://arxiv.org/html/2510.20095v2#A2).

Wikipedia visual information extraction.
Wikipedia provides a comprehensive and accessible knowledge base across the Tree of Life, with species-level descriptions that often contain appearance information.
As each instance within a species may have significant appearance variations, these descriptions cannot be directly associated with images.
Even so, they offer morphological vocabulary that can be systematically extracted and leveraged in captions.

As shown on the left of [Figure 2](https://arxiv.org/html/2510.20095v2#S3.F2), we scrape Wikipedia pages based on scientific names and validate the page with the full taxonomy rank. Each page may contain various information regarding the species, such as appearance, habitat, and distribution. We perform a quick filtering to keep the sections that potentially have visual descriptions according to their title (*e.g*., “Description”). Qwen3 32B (Yang et al., 2025) is then used to verify whether the kept paragraph contains visual information and to extract such information.
The model is forced to only focus on attributes such as color, pattern, shape, texture, and other morphological characteristics.
When no usable visual information is found for a species, we apply the same process for the corresponding genus and map the description back to the species if available.
This pipeline yields $132$K descriptions, covering $29.5\%$ of the $447$K taxa in TreeOfLife-10M (Stevens et al., 2024).
The extracted Wikipedia visual information forms the foundation of the subsequent caption generation process.

Format example design.
MLLMs might struggle to decide which traits are salient for a given organism.
Direct prompting without guidance often leads to hallucination or oversight of important details.
Therefore, we design taxon-tailored format examples that illustrate the desired style and content of descriptive captions.
For each of the $347$ taxonomic classes in TreeOfLife-10M, we query Gemini Deep Research (Comanici et al., 2025) to retrieve candidate textual descriptions of representative species.
Each query returns descriptions of up to six species, from which we manually validate trait accuracy and format consistency, and keep at most three per class.
When reliable sources are scarce, fewer than three species are included.
This process yields $896$ curated examples, which explicitly encourage the model to attend to important traits given species labels.

Caption generation.
With Wikipedia-derived visual information and taxon-tailored format examples as domain-specific contexts, we leverage MLLMs to generate synthetic captions grounded in the target images.
Specifically, we use InternVL3 38B  (Zhu et al., 2025) as the backbone and accelerate inference with the vLLM framework  (Kwon et al., 2023).
When no Wikipedia information is available for the species, we only incorporate format examples in the context.
Such a process helps the model generate trait-focused descriptions that emphasize visible appearance while avoiding irrelevant or hallucinated details.
Furthermore, this pipeline enables large-scale, efficient generation of instance-level captions that complement taxonomy-based supervision.

## 4 Experiments

BioCAP is initialized from the OpenAI ViT-B/16 CLIP checkpoint (Radford et al., 2021) and trained on TreeOfLife-10M (Stevens et al., 2023b; Van Horn & Mac Aodha, 2021; Gharaee et al., 2023; Encyclopedia of Life, 2023) for 50 epochs with species labels and captions.
We evaluate BioCAP on species classification tasks following BioCLIP 2 (Gu et al., 2025), including NABirds (Van Horn et al., 2015), Meta-Album (Ullah et al., 2022),
IDLE-OO Camera Traps (Campolongo et al., 2025; Island-Conservation,; Desert-Lion-Conservation, 2024; Vélez et al., 2022; Balasubramaniam, 2024; Yousif et al., 2019), and Rare Species (Stevens et al., 2023a; Encyclopedia of Life, 2023; IUCN, 2022).
For evaluating the understanding of natural language, we use INQUIRE-Rerank (Vendrow et al., 2024).
In addition, we collect paired text-image data from PlantID (Bruce Homer-Smith and contributors to PlantID.net, 2025) and Cornell Bird (Macaulay Library, Cornell Lab of Ornithology, 2025) to test retrieval performance in organismal domains.
Hyperparameter settings and more data information are presented in §[C](https://arxiv.org/html/2510.20095v2#A3) and §[D](https://arxiv.org/html/2510.20095v2#A4), respectively.

**Table 1: Zero-shot species classification top-1 accuracy across $\mathbf{10}$ tasks for different models. Bold and underlined entries indicate the best and second best accuracies, respectively. BioCAP achieves the best performance across all benchmarks, with an average improvement of $8.8\%$ over BioCLIP. All the compared models are based on the ViT-B/16 visual encoder.**
|  | Animals | Plants & Fungi |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | NABirds | Plankton | Insects | Insects 2 | Camera Trap | PlantNet | Fungi | PlantVillage | Med. Leaf | Rare Species | Mean |
| Random Guessing | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}0.2$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}1.2$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}1.0$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}1.0$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}3.5$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}4.0$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}4.0$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}2.6$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}4.0$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}0.3$ | $\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}2.2$ |
| CLIP (ViT-B/16) | $39.0$ | $3.3$ | $7.4$ | $9.3$ | $28.1$ | $52.5$ | $8.6$ | $5.1$ | $15.0$ | $25.7$ | $19.4$ |
| SigLIP | $50.2$ | $3.7$ | $17.6$ | $17.6$ | $26.7$ | $76.3$ | $28.3$ | $\underline{26.1}$ | $\underline{45.4}$ | $30.7$ | $32.3$ |
| FG-CLIP | $48.3$ | $1.9$ | $6.9$ | $9.3$ | $26.4$ | $55.6$ | $7.3$ | $5.9$ | $15.7$ | $29.4$ | $20.7$ |
| BioTrove-CLIP | $39.4$ | $1.0$ | $20.5$ | $15.7$ | $10.7$ | $64.4$ | $38.2$ | $15.7$ | $31.6$ | $24.6$ | $26.2$ |
| BioCLIP | $\underline{58.8}$ | $\underline{6.1}$ | $\underline{34.9}$ | $\underline{20.5}$ | $\underline{31.7}$ | $\underline{88.2}$ | $\underline{40.9}$ | $19.0$ | $38.5$ | $\underline{37.1}$ | $\underline{37.6}$ |
| BioCAP | $\mathbf{67.6}$ | $\mathbf{7.2}$ | $\mathbf{41.9}$ | $\mathbf{23.7}$ | $\mathbf{37.4}$ | $\mathbf{93.6}$ | $\mathbf{64.4}$ | $\mathbf{33.0}$ | $\mathbf{51.4}$ | $\mathbf{44.2}$ | $\mathbf{46.4}$ |

### 4.1 Main Results

Classification.
We evaluate models in the zero-shot setting on ten species classification benchmarks in [Table 1](https://arxiv.org/html/2510.20095v2#S4.T1).
Compared with BioCLIP, where captions are not involved in training, the accuracy increases by $23.5\%$ on Fungi and $7.1\%$ on Rare Species, demonstrating stronger generalization to challenging real-world settings.
Overall, BioCAP achieves an average top-1 accuracy margin of $8.8\%$ over BioCLIP and $27.0\%$ over the original CLIP model (Radford et al., 2021).

Retrieval.

**Table 2: Performances on natural language tasks, including INQUIRE-Rerank (AP@$\mathbf{50}$) and two text-image retrieval (Recall@$\mathbf{10}$) benchmarks. Bold and underlined entries indicate the best and second best accuracies, respectively. I2T means image-to-text retrieval, and vice versa. With the additional supervision from descriptive captions, BioCAP achieves an average performance advantage of $21.3\%$ over BioCLIP.**
|  | INQUIRE Rerank | Cornell Bird | PlantID |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Appear. | Behav. | Context | Species | I2T | T2I | I2T | T2I | Mean |
| CLIP (ViT-B/16) | $30.8$ | $32.9$ | $37.2$ | $37.1$ | $33.8$ | $29.1$ | $25.0$ | $22.1$ | $31.0$ |
| SigLIP | $\underline{34.6}$ | $\mathbf{37.2}$ | $\mathbf{41.4}$ | $\underline{36.2}$ | $47.7$ | $\underline{50.2}$ | $42.1$ | $38.1$ | $\underline{40.9}$ |
| FG-CLIP | $28.8$ | $31.1$ | $32.5$ | $41.0$ | $\underline{49.4}$ | $48.1$ | $28.7$ | $27.4$ | $35.9$ |
| BioTrove-CLIP | $28.5$ | $22.2$ | $30.5$ | $39.5$ | $16.5$ | $13.8$ | $47.4$ | $\underline{50.1}$ | $31.1$ |
| BioCLIP | $27.4$ | $27.2$ | $30.8$ | $41.1$ | $15.1$ | $16.2$ | $\underline{47.8}$ | $45.0$ | $31.3$ |
| BioCAP | $\mathbf{37.1}$ | $\underline{33.6}$ | $\underline{37.0}$ | $\mathbf{43.0}$ | $\mathbf{54.0}$ | $\mathbf{52.0}$ | $\mathbf{81.4}$ | $\mathbf{83.0}$ | $\mathbf{52.6}$ |

We evaluate the natural language understanding on INQUIRE-Rerank (AP@$50$), Cornell Bird and PlantID(Recall@$10$) in [Table 2](https://arxiv.org/html/2510.20095v2#S4.T2).
Except for the Behavior and Context tasks in INQUIRE-Rerank, BioCAP achieves the best performance across all benchmarks.
FG-CLIP is trained for fine-grained retrieval tasks (Xie et al., 2025), yet BioCAP shows clear advantages with an average performance margin of $16.7\%$.
These results indicate that trait-grounded synthetic captions substantially enhance fine-grained, expert-style retrieval of biological knowledge.

### 4.2 Ablation Study

For ablation studies, we train models with TreeOfLife-1M (Stevens et al., 2024) for 100 epochs.

**Table 3: Influence of different captions. *None*: no caption is used in training (BioCLIP); *Wiki Page*: a sentence from the Wikipedia visual information; *Synthetic*: captions generated by MLLMs. Synthetic captions involve the option of simple prompts (*Base*) and trait-focused prompts (*Trait*). Domain-specific contexts in MLLM inputs: *Example*: format examples; *Wiki*: Wikipedia-derived visual information. The average performances on each benchmark category (*CLS*: classification; *INQ*: INQUIRE-Rerank) are reported.**
| Strategy | CLS | Retrieval | INQ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Caption | Prompt | Context |  | I2T | T2I |  |
| None | – | – | $30.2$ | $29.8$ | $30.9$ | $30.0$ |
| Wiki Page | – | – | $30.0$ | $45.8$ | $46.2$ | $30.3$ |
| Synthetic | Base | – | $27.0$ | $25.4$ | $25.2$ | $29.9$ |
| Synthetic | Trait | – | $30.8$ | $43.4$ | $43.9$ | $33.2$ |
| Synthetic | Trait | Example | $\underline{31.8}$ | $\underline{46.7}$ | $\underline{46.4}$ | $\underline{33.9}$ |
| Synthetic | Trait | Example+Wiki | $\mathbf{33.8}$ | $\mathbf{53.8}$ | $\mathbf{53.6}$ | $\mathbf{34.8}$ |

Influence of different captions.
As illustrated before, the alignment between captions and potential diagnostic traits is critical to the model training.
We conduct ablations to examine the impact of different caption generation strategies in [Table 4](https://arxiv.org/html/2510.20095v2#S4.T4).
Using only taxonomic labels without captions (*None*) corresponds to BioCLIP, and is set as the baseline.
Adding raw Wikipedia text as captions is a straightforward way to involve domain knowledge.
We demonstrate by *Wiki Page* that using non-instance-specific captions provides improvements in retrieval but does not help classification.

Then we incorporate MLLMs to generate instance-based synthetic captions.
The *Base* prompt asks the model to “describe this image in short.” Without emphasis on traits, the captions contain inaccurate descriptions and degrade the performance. It corresponds to our previous analysis that noise in captions harms multimodal alignment.
The *Trait* prompt explicitly requires the model to focus on traits, which substantially reduces the influence of environmental nuisance.
Building upon the *Trait* prompt, we further introduce taxon-tailored format examples and Wikipedia-derived visual information as domain-specific contexts. *Trait+Example+Wiki* demonstrates the best performance, and is used to generate captions for the other experiments.
All the adopted prompts are listed in §[B.3](https://arxiv.org/html/2510.20095v2#A2.SS3).

Training recipe.
In BioCAP, we adopt two separate visual projectors to align species names and captions, respectively.
To ablate this design, we train a model with a single visual projector that aligns species and captions at the same time, named as *single proj* in [Table 4](https://arxiv.org/html/2510.20095v2#S4.T4).
Our two-projector design is named *dual proj*.
The results show that *dual proj* consistently outperforms *single proj*, which validates the necessity of decoupling supervision signals in heterogeneous multimodal training.
Additionally, caption generation introduces additional computation time.
To assess the fairness, we naively double the training epochs and report the results as *200 epochs* in [Table 4](https://arxiv.org/html/2510.20095v2#S4.T4).
The results show an obvious gap between *200 epochs* and other variants, supporting the worth of synthetic captions.

**Table 5: The influence of the training set components. We incrementally add Wikipedia-covered and non-covered species into training with different supervisions to validate the generalization of synthetic captions. The results on the covered/non-covered test species are reported. *Name*: species name; *Caption*: synthetic captions.**
| Training Component | Classification |  |  |
| --- | --- | --- | --- |
| Covered | Non-covered | Covered | Non-covered |
| Name | – | $35.7$ | $40.1$ |
| Name+Caption | – | $37.5$ | $42.7$ |
| Name | Name | $35.8$ | $43.5$ |
| Name+Caption | Name | $38.6$ | $46.6$ |
| Name+Caption | Name+Caption | $\mathbf{45.3}$ | $\mathbf{48.8}$ |

Training set components.
Wikipedia provides faithful domain knowledge as contexts for MLLMs. However, it does not cover all the species for usable visual descriptions (See §[B.2](https://arxiv.org/html/2510.20095v2#A2.SS2)).
We incrementally add different training components to evaluate the generalization of synthetic captions in [Table 6](https://arxiv.org/html/2510.20095v2#S4.T6).
Specifically, we partition the training set based on whether the species are covered by Wikipedia ($37.1\%$ of the total taxa in TreeOfLife-1M).
We also separate classification benchmarks into the two groups ($76.2\%$ covered by Wikipedia).
In the table, *Name* refers to taxonomy-only supervision, while *Name+Caption* denotes the joint supervision with synthetic captions.
The results show that adding captions to Wiki-covered species also enhances the understanding of non-covered species.
Incorporating both taxonomy and captions for all species achieves the best results, even if many species are not covered by Wikipedia, indicating generalization of caption-encoded knowledge.

Caption Evaluation.
We conduct a human evaluation to assess the quality of the generated captions.
As shown in [Table 6](https://arxiv.org/html/2510.20095v2#S4.T6), we evaluate the captions along four metrics (*Groundedness*: if the description is visible in the image; *Specificity*: if the caption describes distinctive traits; *Completeness*: if the caption covers 2-3 most salient aspects; *Clarity*: preciseness and objectiveness).
We randomly sample $200$ images from TreeOfLife, covering $200$ distinct taxonomic classes.
For each image, we provide three captions: one generated by our full method (*Trait+Example+Wiki*) and two obtained from alternative strategies (*Base* and *Trait*).
The order of captions is randomized.
Human evaluators are asked to select the best caption under each metric. Each caption is independently assessed by two evaluators.
The results show that captions generated by our full pipeline are recognized by human evaluators to be accurate and image-specific.
Detailed statistics are included in §[G](https://arxiv.org/html/2510.20095v2#A7).

### 4.3 Analysis and Discussion

Figure: Figure 3: Captions generated by different strategies (refer to [Table 4](https://arxiv.org/html/2510.20095v2#S4.T4) and §[B.3](https://arxiv.org/html/2510.20095v2#A2.SS3)). The domain-specific contexts of taxon-tailored format examples and Wikipedia-derived visual information significantly reduce hallucination and provide more accurate descriptions of the target object.
Refer to caption: x4.png

Qualitative comparison of captions. [Figure 3](https://arxiv.org/html/2510.20095v2#S4.F3) illustrates qualitative comparisons of generated captions. The second column corresponds to our full strategy (*Trait+Example+Wiki*). Using the *Base* or *Trait* prompts without domain-specific contexts leads to hallucination (the common elbow orchid flower misrecognized as an insect) and inaccurate descriptions (vague color descriptions for the white-eared honeyeater). In contrast, introducing format examples and Wikipedia-derived visual information significantly improves the instance-specificity and faithfulness of the synthetic captions.

Figure: Figure 4: Grad-CAM visualization of CLIP, BioCLIP, and BioCAP, given species names and biological concepts frequently mentioned in their captions. Comparatively, BioCAP offers a comprehensive understanding of these concepts and connects them to the corresponding species.
Refer to caption: x5.png

Figure: Figure 5: Grad-CAM visualization of CLIP, BioCLIP, and BioCAP, given behaviors. BioCAP correctly highlights the body parts related to the behaviors.
Refer to caption: x6.png

Why are captions helpful for classification?
To better understand the effects of captions, we use Grad-CAM (Zhou et al., 2016) to visualize model attention given species names and high-frequency biological concepts mentioned in the corresponding captions in [Figure 4](https://arxiv.org/html/2510.20095v2#S4.F4).
The visualization shows that BioCAP learns to localize biologically meaningful traits and associate them with the species names.
It corresponds to our analysis in §[3.1](https://arxiv.org/html/2510.20095v2#S3.SS1) that captions help model focus on diagnostic characters during training, and thereby improve classification performance.
The information on Grad-CAM implementation and high-frequency concept is presented in §[C.3](https://arxiv.org/html/2510.20095v2#A3.SS3).

Figure: Figure 6: Embedding distribution of three bird species with sex and behavior annotations. On the right, we provide example images corresponding to each label. DINOv3 and CLIP fail to align male and female red-winged blackbirds while mixing male and female hummingbirds. BioCLIP does not capture the semantical difference between behaviors. With the guidance of captions, BioCAP tells the subtle difference between *perch* and *stand* and accurately separates the behavior variants.
Refer to caption: x7.png

Semantic Understanding.
In addition to specific traits, we further analyze whether captions help align embeddings with broader biologically meaningful semantics ([Figure 5](https://arxiv.org/html/2510.20095v2#S4.F5)). Using Grad-CAM, we visualize model attention given behavior-related words such as fly, feed, and crawl. Compared with CLIP and BioCLIP, BioCAP accurately highlights the body parts related to these behaviors, for instance, activation of wings for “fly” and legs for “crawl.”

Beyond instance-level understanding, we also visualize the relationship between instances with t-SNE across three bird species in [Figure 6](https://arxiv.org/html/2510.20095v2#S4.F6), annotated with both behaviors (perch, fly, stand) and sex (male, female/immature). General-purpose models like CLIP and DINOv3 form loose species clusters and conflate sex distinctions. They also mistakenly align female/immature red-winged blackbirds with brown-headed cowbirds.
BioCLIP learns the species distinction, but fails to differentiate the behavior variations. Comparatively, BioCAP produces compact species clusters and separation of various biological semantics.
These results further demonstrate the effectiveness of descriptive captions in enhancing the understanding of various biological concepts.
The collection process of behavior labels is presented in §[C.4](https://arxiv.org/html/2510.20095v2#A3.SS4). We provide more qualitative results in §[E](https://arxiv.org/html/2510.20095v2#A5).

## 5 Conclusion

This paper investigates using instance-level descriptive captions as a complementary supervision for biological multimodal foundation models. Due to the lack of such resources at scale, we incorporate multimodal large language models (MLLMs) to generate synthetic captions. We curate Wikipedia-derived visual information and format examples to reduce hallucination and produce accurate, instance-specific captions. Aligning images with captions encourages the model to emphasize potential diagnostic traits while reducing the influence of environmental factors. The acquired BioCAP model demonstrates rich understanding of a broad range of biological semantics. The superior performance in species classification and biological text-image retrieval highlights the value of descriptive captions in bridging biological images with multimodal foundation models.

## Ethical Statement

Our study involves human evaluation of automatically generated captions for organismal biology images. Participants were asked only to express preferences between different caption candidates based on the associated image. No personal or identifiable information was collected, and the task posed minimal risk. Participation was voluntary, and participants were engaged without coercion. As the task did not involve sensitive data, medical decisions, or personal attributes, institutional review board (IRB) approval was not required under our institution’s policies. We emphasize that the generated captions are intended for scientific research purposes. Nonetheless, we acknowledge the potential risk of inaccurate or misleading captions, and we therefore recommend their use only as research tools and not as authoritative sources.

## Reproducibility statement

We have released all source code, including modules for model training, evaluation, caption generation, and Wikipedia scraping, at [https://github.com/Imageomics/biocap](https://github.com/Imageomics/biocap). The corresponding dataset, extended from TreeOfLife-10M with generated captions, is available at [TreeOfLife-10M-Captions](https://huggingface.co/datasets/imageomics/TreeOfLife-10M-Captions).
The details regarding caption generation are presented in §[B](https://arxiv.org/html/2510.20095v2#A2). The training hyperparameters and details to reproduce our evaluation results are included in §[C](https://arxiv.org/html/2510.20095v2#A3). The information on adopted benchmarks is listed in §[D](https://arxiv.org/html/2510.20095v2#A4).

## Acknowledgments

We would like to thank Wasila Dahdul, Zhiyuan Tao, Yifan Liu, Fangxun Liu, Shuheng Wang, Ziqi Li, David Carlyn, Quang-Huy Nguyen, Yintie Lei, and Junke Yang for their help with the human evaluation, and the [Imageomics Team](https://imageomics.osu.edu/about/team) for their constructive feedback.

We sincerely thank PlantID.net (Bruce Homer-Smith and contributors to PlantID.net, 2025), as well as the Cornell Lab of Ornithology (Macaulay Library, Cornell Lab of Ornithology, 2025) for providing access to their biological media collections. The data made our retrieval evaluation possible.

Our research is supported by NSF OAC 2118240 and resources from the Ohio Supercomputer Center (Center, 1987). The authors are grateful for the generous support of the computational resources from the Ohio Supercomputer Center.

## References

- Balasubramaniam (2024)
S. Balasubramaniam.
Optimized classification in camera trap images: An approach with smart camera traps, machine learning, and human inference.
Master’s thesis, The Ohio State University, 2024.
URL [http://rave.ohiolink.edu/etdc/view?acc_num=osu1721417695430687](http://rave.ohiolink.edu/etdc/view?acc_num=osu1721417695430687).
- Bruce Homer-Smith and contributors to PlantID.net (2025)
Bruce Homer-Smith and contributors to PlantID.net.
Plantid – online plant identification resource, 2025.
URL [https://plantid.net/](https://plantid.net/).
Content licensed under CC BY-NC 3.0. Developed and produced by Bruce Homer-Smith with contributions from Dave Long, Doreen Smith, Kristin Jakob, John Malpas, and others.
- Budd (2021)
Graham E Budd.
Morphospace.
*Current Biology*, 31(19):R1181–R1185, 2021.
- Campolongo et al. (2025)
Elizabeth G Campolongo, Jianyang Gu, and Net Zhang.
IDLE-OO Camera Traps, 2025.
URL [https://huggingface.co/datasets/imageomics/IDLE-OO-Camera-Traps](https://huggingface.co/datasets/imageomics/IDLE-OO-Camera-Traps).
- Center (1987)
Ohio Supercomputer Center.
Ohio supercomputer center, 1987.
URL [http://osc.edu/ark:/19495/f5s1ph73](http://osc.edu/ark:/19495/f5s1ph73).
- Cherti et al. (2023)
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.
Reproducible scaling laws for contrastive language-image learning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 2818–2829, 2023.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025.
- Desert-Lion-Conservation (2024)
Desert-Lion-Conservation.
Desert lion conservation camera traps, July 2024.
URL [https://lila.science/datasets/desert-lion-conservation-camera-traps/](https://lila.science/datasets/desert-lion-conservation-camera-traps/).
- Doan et al. (2024)
Thang Doan, Xin Li, Sima Behpour, Wenbin He, Liang Gou, and Liu Ren.
Hyp-ow: Exploiting hierarchical structure learning with hyperbolic distance enhances open world object detection.
In *AAAI*, pp. 1555–1563, 2024.
- Encyclopedia of Life (2023)
Encyclopedia of Life.
Encyclopedia of life (eol).
[http://eol.org](http://eol.org), 2023.
Accessed 29 July 2023.
- Fan et al. (2023)
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian.
Improving clip training with language rewrites.
In *NeurIPS*, volume 36, pp. 35544–35575, 2023.
- G. & J. (2019)
Geetharamani G. and Arun Pandian J.
Identification of plant leaf diseases using a nine-layer deep convolutional neural network.
*Computers & Electrical Engineering*, 76:323–338, 2019.
ISSN 0045-7906.
doi: https://doi.org/10.1016/j.compeleceng.2019.04.011.
URL [https://www.sciencedirect.com/science/article/pii/S0045790619300023](https://www.sciencedirect.com/science/article/pii/S0045790619300023).
- Gadre et al. (2023)
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.
Datacomp: In search of the next generation of multimodal datasets.
In *Advances in Neural Information Processing Systems*, volume 36, pp. 27092–27112, 2023.
- Garcin et al. (2021)
Camille Garcin, Alexis Joly, Pierre Bonnet, Antoine Affouard, Jean-Christophe Lombardo, Mathias Chouet, Maximilien Servajean, Titouan Lorieul, and Joseph Salmon.
Pl@ntnet-300k: a plant image dataset with high label ambiguity and a long-tailed distribution.
In *Advances in Neural Information Processing Systems (Datasets and Benchmarks Track)*, volume 36, 2021.
URL [https://openreview.net/forum?id=eLYinD0TtIt](https://openreview.net/forum?id=eLYinD0TtIt).
- Gharaee et al. (2023)
Z. Gharaee, Z. Gong, N. Pellegrino, I. Zarubiieva, J. B. Haurum, S. C. Lowe, J. T. A. McKeown, C. Y. Ho, J. McLeod, Y. C. Wei, J. Agda, S. Ratnasingham, D. Steinke, A. X. Chang, G. W. Taylor, and P. Fieguth.
A step towards worldwide biodiversity assessment: The BIOSCAN-1M insect dataset.
In *NeurIPS*, 2023.
- Gharaee et al. (2024)
Zahra Gharaee, Scott C Lowe, ZeMing Gong, Pablo Millan Arias, Nicholas Pellegrino, Austin T Wang, Joakim Bruslund Haurum, Iuliia Eyriay, Lila Kari, Dirk Steinke, et al.
Bioscan-5m: a multimodal dataset for insect biodiversity.
In *NeurIPS*, volume 37, pp. 36285–36313, 2024.
- Gong et al. (2025)
ZeMing Gong, Austin Wang, Xiaoliang Huo, Joakim Bruslund Haurum, Scott C. Lowe, Graham W. Taylor, and Angel X Chang.
CLIBD: Bridging vision and genomics for biodiversity monitoring at scale.
In *ICLR*, 2025.
- Gu et al. (2025)
Jianyang Gu, Samuel Stevens, Elizabeth G Campolongo, Matthew J Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E White, James Balhoff, et al.
Bioclip 2: Emergent properties from scaling hierarchical contrastive learning.
In *NeurIPS*, 2025.
- Gwet (2008)
K. L. Gwet.
Computing inter‐rater reliability and its variance in the presence of high agreement.
*British Journal of Mathematical and Statistical Psychology*, 61(Pt 1):29–48, 2008.
- Harnik et al. (2025)
Yonatan Harnik, Hadas Shalit Peleg, Amit H. Bermano, and Anat Milo.
Data efficient molecular image representation learning using foundation models.
*Chemical Science*, 16(24):10833–10841, 2025.
- Heidi M. Sosik (2015)
Emily F. Brownlee Heidi M. Sosik, Emily E. Peacock.
Annotated plankton images - data set for developing and evaluating classification methods, 2015.
URL [https://hdl.handle.net/10.1575/1912/7341](https://hdl.handle.net/10.1575/1912/7341).
- Huang et al. (2021)
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng.
Learning with noisy correspondence for cross-modal matching.
In *NeurIPS*, volume 34, pp. 29406–29419, 2021.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- (24)
Island-Conservation.
Island conservation camera traps.
URL [https://lila.science/datasets/island-conservation-camera-traps/](https://lila.science/datasets/island-conservation-camera-traps/).
- IUCN (2022)
IUCN.
The iucn red list of threatened species. version 2022-2.
[https://www.iucnredlist.org](https://www.iucnredlist.org), 2022.
Accessed on 5 July 2023.
- Kingma & Ba (2015)
Diederik Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In *ICLR*, 2015.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- Lai et al. (2024)
Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao.
Veclip: Improving clip training via visual-enriched captions.
In *ECCV*, pp. 111–127, 2024.
- Landis & Koch (1977)
J. R. Landis and G. G. Koch.
The measurement of observer agreement for categorical data.
*Biometrics*, 33(1):159–174, 1977.
- Li et al. (2023)
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.
LLaVA-med: Training a large language-and-vision assistant for biomedicine in one day.
In *NeurIPS*, 2023.
- Li et al. (2022)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
In *ICML*, volume 162, pp. 12888–12900, 2022.
- Li et al. (2025)
Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Wei Li, Mao Su, Shufei Zhang, Wanli Ouyang, Yuqiang Li, and Dongzhan Zhou.
Chemvlm: Exploring the power of multimodal large language models in chemistry area.
In *AAAI*, pp. 415–423, 2025.
- Liu et al. (2023)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *NeurIPS*, volume 36, pp. 34892–34916, 2023.
- Macaulay Library, Cornell Lab of Ornithology (2025)
Macaulay Library, Cornell Lab of Ornithology.
Macaulay library: Multimedia resources for birds and other animals, 2025.
URL [https://www.macaulaylibrary.org](https://www.macaulaylibrary.org).
- Mai et al. (2025)
Zheda Mai, Arpita Chowdhury, Zihe Wang, Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, and Wei-Lun Chao.
Ava-bench: Atomic visual ability benchmark for vision foundation models.
*arXiv preprint arXiv:2506.09082*, 2025.
- Parker et al. (2024)
Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Rudy Morel, Ruben Ohana, Mariel Pettee, Bruno Régaldo-Saint Blancard, Kyunghyun Cho, and Shirley Ho.
Astroclip: a cross-modal foundation model for galaxies.
*Monthly Notices of the Royal Astronomical Society*, 531(4):4990–5011, June 2024.
ISSN 1365-2966.
- Picek et al. (2022)
Lukáš Picek, Milan Šulc, Jiří Matas, Thomas S Jeppesen, Jacob Heilmann-Clausen, Thomas Læssøe, and Tobias Frøslev.
Danish fungi 2020-not just another image recognition dataset.
In *Proceedings of the IEEE Winter Conference on Applications of Computer Vision*, pp. 1525–1535, 2022.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamila Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *ICML*, pp. 8748–8763, 2021.
- Ranzinger et al. (2024)
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov.
Am-radio: Agglomerative vision foundation model reduce all domains into one.
In *CVPR*, pp. 12490–12500, 2024.
- S & J (2020)
Roopashree S and Anitha J.
Medicinal leaf dataset, Oct 2020.
URL [https://data.mendeley.com/datasets/nnytj2v3n5/1](https://data.mendeley.com/datasets/nnytj2v3n5/1).
- Sastry et al. (2025)
Srikumar Sastry, Subash Khanal, Aayush Dhakal, Adeel Ahmad, and Nathan Jacobs.
Taxabind: A unified embedding space for ecological applications.
In *WACV*, pp. 1765–1774. IEEE, 2025.
- Schuhmann et al. (2022)
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation image-text models.
In *Advances in neural information processing systems*, volume 35, pp. 25278–25294, 2022.
- Serret et al. (2019)
Hortense Serret, Nicolas Deguines, Yikweon Jang, Gregoire Lois, and Romain Julliard.
Data quality and participant engagement in citizen science: comparing two approaches for monitoring pollinators in france and south korea.
*Citizen Science: Theory and Practice*, 4(1):22, 2019.
- Singh et al. (2020)
Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra.
Plantdoc: A dataset for visual plant disease detection.
In *Proceedings of the 7th ACM IKDD CoDS and 25th COMAD*, CoDS COMAD 2020, pp. 249–253, New York, NY, USA, 2020. Association for Computing Machinery.
ISBN 9781450377386.
doi: 10.1145/3371158.3371196.
URL [https://doi.org/10.1145/3371158.3371196](https://doi.org/10.1145/3371158.3371196).
- Stevens et al. (2023a)
Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su.
Rare species, 2023a.
URL [https://huggingface.co/datasets/imageomics/rare-species](https://huggingface.co/datasets/imageomics/rare-species).
- Stevens et al. (2023b)
Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su.
TreeOfLife-10M, 2023b.
URL [https://huggingface.co/datasets/imageomics/TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M).
- Stevens et al. (2024)
Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su.
BioCLIP: A vision foundation model for the tree of life.
In *CVPR*, pp. 19412–19424, June 2024.
- Ullah et al. (2022)
Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle M Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu.
Meta-album: Multi-domain meta-dataset for few-shot image classification.
In *NeurIPS*, 2022.
URL [https://meta-album.github.io/](https://meta-album.github.io/).
- Van Horn & Mac Aodha (2021)
Grant Van Horn and Oisin Mac Aodha.
inat challenge 2021 - fgvc8, 2021.
URL [https://kaggle.com/competitions/inaturalist-2021](https://kaggle.com/competitions/inaturalist-2021).
- Van Horn et al. (2015)
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.
Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.
In *CVPR*, pp. 595–604, 2015.
- Van Horn et al. (2018)
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.
The inaturalist species classification and detection dataset.
In *CVPR*, pp. 8769–8778, 2018.
- Vélez et al. (2022)
Juliana Vélez, Paula J Castiblanco-Camacho, Michael A Tabak, Carl Chalmers, Paul Fergus, and John Fieberg.
Choosing an appropriate platform and workflow for processing camera trap data using artificial intelligence.
*arXiv preprint arXiv:2202.02283*, 2022.
- Vendrow et al. (2024)
Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate E Jones, Oisin Mac Aodha, Sara Beery, and Grant Van Horn.
Inquire: A natural world text-to-image retrieval benchmark.
*NeurIPS*, 2024.
- Vivanco Cepeda et al. (2023)
Vicente Vivanco Cepeda, Gaurav Kumar Nayak, and Mubarak Shah.
Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization.
In *NeurIPS*, volume 36, pp. 8690–8701. Curran Associates, Inc., 2023.
- Wang et al. (2024)
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari.
Sam-clip: Merging vision foundation models towards semantic and spatial understanding.
In *CVPR*, pp. 3635–3647, 2024.
- Wang & Isola (2020)
Tongzhou Wang and Phillip Isola.
Understanding contrastive representation learning through alignment and uniformity on the hypersphere.
In *ICML*, pp. 9929–9939. PMLR, 2020.
- Wang et al. (2022)
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun.
MedCLIP: Contrastive learning from unpaired medical images and text.
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), *EMNLP*, pp. 3876–3887, 2022.
- Wikipedia contributors (2025)
Wikipedia contributors.
Wikipedia, the free encyclopedia.
[https://en.wikipedia.org/wiki/](https://en.wikipedia.org/wiki/), 2025.
Accessed: August 2025.
- Wu et al. (2019)
Xiaoping Wu, Chi Zhan, Yukun Lai, Ming-Ming Cheng, and Jufeng Yang.
IP102: A large-scale benchmark dataset for insect pest recognition.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pp. 8787–8796, 2019.
- Xie et al. (2025)
Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin.
FG-CLIP: Fine-grained visual and textual alignment.
In *ICML*, 2025.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yang et al. (2024)
Chih-Hsuan Yang, Ben Feuer, Zaki Jubery, Zi K. Deng, Andre Nakkab, Zahid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh K Singh, Arti Singh, Soumik Sarkar, Nirav Merchant, Chinmay Hegde, and Baskar Ganapathysubramanian.
Biotrove: A large curated image dataset enabling ai for biodiversity.
In *NeurIPS*, volume 37, pp. 102101–102120, 2024.
- Yang et al. (2023)
Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu.
Alip: Adaptive language-image pre-training with synthetic caption.
In *ICCV*, pp. 2910–2919, 2023.
- Yousif et al. (2019)
Hayder Yousif, Roland Kays, and Zhihai He.
Dynamic programming selection of object proposals for sequence-level animal species classification in the wild.
*IEEE Transactions on Circuits and Systems for Video Technology*, 2019.
- Yu et al. (2024)
Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu.
Capsfusion: Rethinking image-text data at scale.
In *CVPR*, pp. 14022–14032, June 2024.
- Yue et al. (2024)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In *CVPR*, pp. 9556–9567, 2024.
- Zhai et al. (2023)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
Sigmoid loss for language image pre-training.
In *ICCV*, pp. 11941–11952, 2023.
- Zhang et al. (2024)
Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al.
A generalist vision–language foundation model for diverse biomedical tasks.
*Nature Medicine*, pp. 1–13, 2024.
- Zhang et al. (2025)
Ziheng Zhang, Jianyang Gu, Arpita Chowdhury, Zheda Mai, David Carlyn, Tanya Berger-Wolf, Yu Su, and Wei-Lun Chao.
Finer-cam: Spotting the difference reveals finer details for visual explanation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)*, pp. 9611–9620, June 2025.
- Zhou et al. (2016)
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.
Learning deep features for discriminative localization.
In *CVPR*, pp. 2921–2929, 2016.
- Zhu et al. (2024)
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
MiniGPT-4: Enhancing vision-language understanding with advanced large language models.
In *ICLR*, 2024.
- Zhu et al. (2025)
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
*arXiv preprint arXiv:2504.10479*, 2025.

## Appendix A Analysis on the Impact of Captions in Training

Let ${\bm{x}}$ be an image embedding extracted by the visual encoder, $y\in\mathcal{Y}$ the taxonomic label, ${\bm{c}}$ the textual embedding of a corresponding descriptive caption, ${\bm{z}}^{*}$ the underlying latent vector of ground-truth traits associated with the taxon $y$, and ${\bm{\epsilon}}\perp{\bm{z}}^{*}$ the environmental noise influencing the trait observation. Assume the image ${\bm{x}}$ and the caption ${\bm{c}}$ are derived from a linear transformation on the latent vector ${\bm{z}}^{*}$ and noise ${\bm{\epsilon}}$:

$$ ${\bm{x}}={\bm{A}}{\bm{z}}^{*}+{\bm{G}}{\bm{\epsilon}}+\eta_{x},\quad{\bm{c}}={\bm{B}}{\bm{z}}^{*}+{\bm{D}}{\bm{\epsilon}}+\eta_{c},$ $$

where $\eta_{x}$ and $\eta_{c}$ are independent zero-mean Gaussian noises. The label $y$ can be directly determined by ${\bm{z}}^{*}$ and is irrelevant to the noise $\epsilon$.
The target is to optimize the encoders so that ${\bm{x}}$ is aligned with ${\bm{z}}^{*}$, and thereby the label $y$ can be derived from ${\bm{x}}$.
We train the model with InfoNCE loss:

$$ $\mathcal{L}_{\text{NCE}}=\mathbb{E}_{p({\bm{x}},{\bm{c}})}\left[-s({\bm{x}},{\bm{c}})+\log\mathbb{E}_{p({\bm{x}}^{{}^{\prime}})}\sum\exp s({\bm{x}}^{{}^{\prime}},{\bm{c}})\right],$ $$

where

$$ $s({\bm{x}},{\bm{c}})=\frac{1}{\tau}\phi({\bm{x}})^{\top}\psi({\bm{c}}).$ $$

Wang & Isola (2020) show that the contrastive loss optimizes alignment and uniformity properties.
The alignment part increases the image-caption inner product $\mathbb{E}[{\bm{x}}^{\top}{\bm{c}}]$, while the uniformity part preserves maximum information.
For $l_{2}$-normalized features, $\mathbb{E}[{\bm{x}}^{\top}{\bm{c}}]=\text{tr}(\Sigma_{{\bm{x}}{\bm{c}}})$, *i.e*., the cross-covariance between the two views. We have:

$$ $\Sigma_{{\bm{x}}{\bm{c}}}={\bm{A}}{\bm{B}}^{\top}+{\bm{G}}{\bm{D}}^{\top},$ $$

which decomposes into the trait-shared component ${\bm{A}}{\bm{B}}^{\top}$ and the nuisance-shared component ${\bm{G}}{\bm{D}}^{\top}$.
If captions capture the diagnostic characters without noise disturbance (${\bm{D}}=0$), the trait term ${\bm{A}}{\bm{B}}^{\top}$ dominates the cross-covariance. The learned image projection, therefore, aligns with trait directions and drops nuisance. Thus, the faithful caption ${\bm{c}}$ helps species classification.
On the contrary, if the caption covaries with image nuisance ${\bm{\epsilon}}$, ${\bm{G}}{\bm{D}}^{\top}$ adds to $\Sigma_{{\bm{x}}{\bm{c}}}$. The InfoNCE loss pulls the image embedding toward the range of ${\bm{G}}$, which leads to spurious correlation and potential performance drop.
Based on these analyses, we propose to generate synthetic captions that ground biological knowledge to enhance the alignment between images and labels.

## Appendix B Details on Caption Generation

In this section, we list the adopted prompts and pipeline details for collecting descriptive captions. We also present some ablation of the visual information extraction pipeline.

### B.1 Prompts

Format example generation. For each of the $347$ taxonomic classes in TreeOfLife-10M, we query Gemini Deep Research (Comanici et al., 2025) to retrieve candidate textual descriptions of representative species.
Each query returns up to six candidate descriptions together with images, each representing one species within the class.
We manually validate trait accuracy and format consistency based on the images of the species, and keep at most three per class.
The images are only used to validate the format example, but are not used in our model training.
When reliable sources are scarce, fewer than three species are included. This process yields $896$ curated examples, which are then incorporated into the context to guide the model in generating trait-focused captions in a consistent style. The prompt used for Gemini Deep Research is listed below.

Wikipedia visual information extraction.
Format examples encourage MLLMs to focus on important traits for different species. We also leverage Wikipedia as a large-scale resource to provide detailed visual information across the Tree of Life.
The descriptions of visual information on Wikipedia pages are often mixed with habitat, behavior, and distributional information.
We perform a quick filtering based on the section titles of the Wikipedia pages and keep those with potential visual information, including “description”, “morphology”, “appearance”, “identification”, “feature”, “characteristics”, “physical”, “structure”, and “explanation of names.”

After the quick filtering, we design a verifying and extraction pipeline to further extract visual information such as color, pattern, shape, and texture.
This process yields over $132$K trait-focused descriptions, covering $29.5\%$ of the $447$K taxa in TreeOfLife-10M (Stevens et al., 2024).
The prompt used to filter and extract morphological traits from Wikipedia is provided below.

The design of separating the verification and extraction steps is based on the consideration of efficiency.
In our experiment, Qwen3 8B is used for verification and Qwen3 32B is used for extraction (Yang et al., 2025). Compared with a single-step design integrating both verification and extraction with Qwen3 32B, the separate design saves $13\%$ computational time. After manual examination on a $200$-sample validation set, the accuracy of Qwen3 8B in verifying if the paragraph contains visual information is consistent with Qwen3 32B. Therefore, we adopt the separate design in our experiment.

Caption generation.
After acquiring the domain-specific contexts, we query MLLMs to generate instance-based captions.
When Wikipedia-derived visual information is not available, only format examples are used in the context.
This ensures the model consistently generates accurate and instance-specific descriptions that emphasize visible morphology while avoiding hallucination.
The exact prompt template is shown below.

### B.2 Statistics of Caption Collection

**Table 7: Taxa coverage and sample coverage across taxonomic ranks.**
| Rank | Taxa coverage | Sample coverage |  |  |
| --- | --- | --- | --- | --- |
| Covered taxa / Total | Ratio | Covered images / Total | Ratio |  |
| Order | $1{,}137/1{,}486$ | $76.5\%$ | $9{,}256{,}964/9{,}533{,}174$ | $97.1\%$ |
| Family | $5{,}127/7{,}920$ | $64.7\%$ | $9{,}164{,}272/9{,}533{,}174$ | $96.1\%$ |
| Genus | $32{,}725/73{,}290$ | $44.7\%$ | $7{,}436{,}080/9{,}533{,}174$ | $78.0\%$ |
| Species | $122{,}243/406{,}293$ | $30.0\%$ | $4{,}989{,}956/9{,}533{,}174$ | $52.3\%$ |

Wikipedia coverage.
As illustrated above, we apply an LLM-based extractor to only keep Wikipedia-derived descriptions related to visual information.
After excluding non-visual descriptions, We obtain a total of $131{,}869$ Wikipedia-derived visual descriptions, collected through three complementary cases.
First, using available species-level labels from TreeOfLife-10M, we collect $109{,}482$ species-level descriptions directly from Wikipedia.
Second, for species whose Wikipedia pages lack visual information, we fall back to their corresponding genus pages and map the retrieved genus descriptions back to these species, producing an additional $12{,}761$ species-level descriptions.
Together, these two cases provide $122{,}243$ species-level descriptions in total.
Third, for samples that lack a species-level label, we use the genus pages directly, adding another $9{,}626$ entries.
In total, we obtain $131{,}869$ visual descriptions, consisting of $122{,}243$ species-level and $9{,}626$ genus-level descriptions. These $131{,}869$ descriptions correspond to $29.5\%$ of the $447{,}593$ unique taxa in TreeOfLife-10M.

[Table 7](https://arxiv.org/html/2510.20095v2#A2.T7) shows the Wikipedia description coverage at different taxonomic levels. One taxon is counted as covered when at least one species within the taxa has associated visual description through the above scraping process. The species-level Wikipedia-derived visual information contributes to $30.0\%$ of the total species-level taxa.
$44.7\%$ taxonomic genera have Wikipedia visual information associated with at least one species within them, and the number becomes $76.5\%$ for orders.

Sample-wise, $52.3\%$ samples are covered with the Wikipedia-derived visual information of exactly the corresponding species. When it comes to the family level, $96.1\%$ samples have at least one species covered by Wikipedia within the same family. In such a way, even if the species is not exactly covered, there is at least another similar species with diagnostic characters described in the context. Thereby, the knowledge is able to be generalized across different species during training. We provide quantitative analysis toward the generalization in [Table 6](https://arxiv.org/html/2510.20095v2#S4.T6).

Computational time. For caption generation, we employ InternVL3 38B with the vLLM  (Kwon et al., 2023),
running on $12$ NVIDIA H100 GPUs for about $30$ hours to process $10$ million samples.
Note that the caption generation process is designed to be highly scalable. The adopted vLLM framework allows for efficient distributed inference with optimized memory management and parallel sampling. The system can handle larger datasets by increasing the number of GPUs or extending the running time. This scalability ensures that generating captions for hundreds of millions of images is feasible, which is valuable for existing biological image repositories.

### B.3 Baseline Prompts

In addition to introducing domain-specific contexts, we also explore different formats for the instruction.
We design two different prompts: Base, which generates generic short captions without any domain hints,
and Trait, which encourages more detailed, image-grounded trait descriptions.
The two prompts are listed below.

We provide the quantitative comparison between the captions generated with the two prompts in [Table 4](https://arxiv.org/html/2510.20095v2#S4.T4).
The qualitative comparisons are presented in [Figure 3](https://arxiv.org/html/2510.20095v2#S4.F3) and [Figure 7](https://arxiv.org/html/2510.20095v2#A5.F7). MLLMs tend to produce more detailed descriptions when the prompt explicitly asks the model to focus on traits.
Based on the comparison, we use the *Trait* prompt in other experiments and further incorporate domain-specific contexts to ground biological knowledge.

## Appendix C Experimental Details

We generate synthetic captions using InternVL3 38B.
The generation is conducted on $12$ NVIDIA H100 GPUs for $30$ hours with the vLLM framework (Kwon et al., 2023).
We apply nucleus sampling (top-$p=0.8$) with a temperature of $0.6$.
With captions and species labels obtained, we train our model on $8$ H100 GPUs for $50$ epochs using the openclip framework (Cherti et al., 2023).
Each GPU processes $4{,}096$ text-image pairs per batch, resulting in a global batch size of $32{,}768$.
We use the AdamW optimizer (Kingma & Ba, 2015) with a learning rate of $1\times 10^{-4}$, weight decay of $0.2$, and a linear warm-up during the first $500$ iterations.
Images are resized to $224\times 224$ for training and evaluation.
As described in §[4.2](https://arxiv.org/html/2510.20095v2#S4.SS2), we adopt a dual-projector design with two separate visual projectors: one for taxonomy supervision and the other for caption supervision. All embeddings used for evaluation are extracted from the taxonomy projector.

### C.1 Hyperparameters

**Table 8: The adopted hyper-parameter setting in training BioCAP.**
| Hyper-parameter | Value |
| --- | --- |
| Architecture | ViT-B/16 |
| Optimizer | Adam |
| Batch size/GPU (organism) | $4{,}096$ |
| GPUs | $8$ H100s |
| Epochs | $50$ |
| Max learning rate | $1\times 10^{-4}$ |
| Warm-up steps | $500$ |
| Weight decay | $0.2$ |
| Input resolution | $224$ |

We summarize the hyper-parameter configurations in [Table 9](https://arxiv.org/html/2510.20095v2#A3.T9) and [Table 9](https://arxiv.org/html/2510.20095v2#A3.T9), corresponding to the main training of BioCAP and the ablation study.
The batch size reported in both tables refers to the per-GPU value.
Compared with the full training, the ablation uses fewer GPUs and a shorter warm-up schedule, while keeping the overall architecture unchanged.

### C.2 Text-Image Retrieval

We evaluate zero-shot retrieval on Cornell Bird  (Macaulay Library, Cornell Lab of Ornithology, 2025), PlantID (Bruce Homer-Smith and contributors to PlantID.net, 2025). We follow the standard text-image retrieval protocol using paired text-image data.
Both text-to-image and image-to-text retrieval are considered by embedding the two modalities into a joint representation space and ranking candidates according to cosine similarity.
Performance is measured by Recall@$10$, which reflects how often the correct match is retrieved within the top results.

### C.3 Grad-CAM Visualization

Implementation. We adopt Grad-CAM (Zhou et al., 2016) for visualizing CLIP attention. Grad-CAM highlights image regions most relevant to a target output by weighting feature maps with their corresponding gradients.
For a CLIP model, given an image $\mathbf{I}$ and a text prompt, their embeddings are obtained via the image encoder $f_{\text{img}}$ and text encoder $f_{\text{text}}$:

$$ $\mathbf{V}=f_{\text{img}}(\mathbf{I}),\qquad\mathbf{T}=f_{\text{text}}(\text{prompt}).$ $$

The cosine similarity logit is:

$$ $s=\frac{\mathbf{V}\cdot\mathbf{T}}{\|\mathbf{V}\|\,\|\mathbf{T}\|}.$ $$

We backpropagate this logit as the target signal, and apply Grad-CAM to the final transformer block of the image encoder to obtain heatmaps. Since the CLIP image encoder is ViT-based, we reshape the patch-token activations from the target layer into a 2D spatial grid before upsampling.

High-frequency concept collection.
For each target species, we first aggregate all captions and compute the frequency of words after removing common stopwords (*e.g*., *and*). Then we manually select biologically meaningful concepts starting from the most frequent words, such as body
structures (*e.g*., antennae, petals, tails). The high-frequency concepts are then used as the text prompt for Grad-CAM. This procedure ensures that the concepts used in visualization correspond to the supervision signal of the synthetic captions. Based on the visualization examples, we explicitly show that aligning captions and images guides the model to focus on the diagnostic characters. Thereby, BioCAP demonstrates better species classification performance.

### C.4 Behavior Semantic Annotation

In addition to static biological concepts related to organs or body parts of the object, we also intend to investigate the model’s understanding of behavioral semantics.
Given that NABirds (Van Horn et al., 2015) does not provide behavior annotations in the original dataset, we use GPT 4o (Hurst et al., 2024) to automatically assign one of three mutually exclusive
behavior categories: fly, perch, or stand. Here, we do not rely on manual labeling, which can be subjective and inconsistent across annotators.
These three categories cover the majority of common bird poses. The model is prompted with the following instruction:

## Appendix D Benchmark Details

### D.1 Collection of Retrieval Benchmarks

**Table 10: Benchmarks collected for image-text retrieval evaluation.**
| Benchmark | Description | Species | Image-Text Pairs |
| --- | --- | --- | --- |
| Cornell Bird | Sourced from the Macaulay Library of the Cornell Lab of Ornithology, containing image-text pairs of North American bird species. | $700$ | $7{,}007$ |
| PlantID | Collected from PlantID, providing paired images and textual descriptions for a wide range of plants. | $794$ | $2{,}349$ |

We collect two retrieval benchmarks to evaluate our model under diverse biological domains:
Cornell Bird (Macaulay Library, Cornell Lab of Ornithology, 2025) and PlantID (Bruce Homer-Smith and contributors to PlantID.net, 2025).
These datasets cover bird and plant species, each providing paired images and textual descriptions for fine-grained retrieval tasks.
[Table 10](https://arxiv.org/html/2510.20095v2#A4.T10) summarizes the key characteristics of these benchmarks.

### D.2 Information of Other Benchmarks

**Table 11: Datasets used for zero-shot classification evaluation. Top-1 accuracy is reported for all the listed benchmarks.**
|  | Name |  | Examples | Classes | Labels |
| --- | --- | --- | --- | --- | --- |
| Animals | NABird | (Van Horn et al., 2015) | $48,000$ | $400$ | Common |
| Plankton | (Heidi M. Sosik, 2015) | $4,080$ | $102$ | Mixed |  |
| Insects | (Serret et al., 2019) | $4,680$ | $117$ | Scientific |  |
| Insects 2 | (Wu et al., 2019) | $4,080$ | $102$ | Mixed |  |
| Plants & Fungi | PlantNet | (Garcin et al., 2021) | $1,000$ | $25$ | Scientific |
| Fungi | (Picek et al., 2022) | $1,000$ | $25$ | Scientific |  |
| PlantVillage | (G. & J., 2019) | $1,520$ | $38$ | Common |  |
| Medicinal Leaf | (S & J, 2020) | $1,040$ | $26$ | Scientific |  |
| PlantDoc | (Singh et al., 2020) | $1,080$ | $27$ | Common |  |
| CameraTrap | Desert-lion | (Desert-Lion-Conservation, 2024) | $352$ | $32$ | Taxonomic |
| ENA24 | (Yousif et al., 2019) | $1120$ | $20$ | Taxonomic |  |
| Island | (Island-Conservation,) | $310$ | $17$ | Taxonomic |  |
| Ohio-small-animals | (Balasubramaniam, 2024) | $468$ | $39$ | Taxonomic |  |
| Orinoquia | (Vélez et al., 2022) | $336$ | $28$ | Taxonomic |  |
|  | Rare Species | (Stevens et al., 2023a) | $12,000$ | $400$ | Taxonomic |

We summarize the datasets used for zero-shot classification evaluation in [Table 11](https://arxiv.org/html/2510.20095v2#A4.T11). These benchmarks cover diverse biological
domains, including animals, plants, fungi, and camera-trap datasets, and all tasks are evaluated with Top-$1$ accuracy. We additionally evaluate on INQUIRE-Rerank (Vendrow et al., 2024), a benchmark where
the goal is to re-rank $100$ candidate images for each of $200$ text queries ($20$K
images in total), ensuring relevant images appear higher in the order.

### D.3 License Information

For retrieval evaluation, we additionally used paired text-image data from
PlantID  (Bruce Homer-Smith and contributors to PlantID.net, 2025) and the Cornell Bird Macaulay Library (Macaulay Library, Cornell Lab of Ornithology, 2025).

PlantID is developed and maintained by Bruce Homer-Smith with contributions
from numerous experts and organizations. The website content, including most
textual and photographic materials, is released under the Creative Commons
CC BY-NC 3.0 license, which allows reuse for non-commercial purposes with proper
attribution. We confirm that our usage strictly followed these terms.

The Cornell Bird Macaulay Library is maintained by the Cornell Lab of Ornithology.
Use of its media assets in scientific publications requires attribution following
their official guidelines. We obtained explicit approval for our use and
acknowledge receipt of media from the Cornell Lab of Ornithology | Macaulay Library.

We sincerely thank PlantID, its contributors, and the Cornell Lab of Ornithology
for making these invaluable resources available to the community, which enables
our retrieval evaluation.

## Appendix E More Qualitative Evaluations

### E.1 Synthetic Caption Comparisons.

Figure: Figure 7: More qualitative comparison on the captions generated by different prompts. Our full pipeline, which emphasizes focusing on traits and introduces domain-specific contexts, demonstrates accurate and instance-specific captions.
Refer to caption: x8.png

[Figure 7](https://arxiv.org/html/2510.20095v2#A5.F7) shows more captions generated by different prompts, including *Base*, *Trait*, and our full pipeline (*Trait+Example+Wiki*). The *Base* captions are often not detailed and not based on biological knowledge. Even if the model is explicitly prompted to focus on traits given the species name, MLLMs can also hallucinate and lose attention to the target object. As shown in the “Paranapiacaba significata” example, the *Trait* caption falsely describes the flower instead of the insect. Our full pipeline, in contrast, accurately describes the insect. When the caption contains too much noise while losing focus on the target object, the provided supervision can misguide the multimodal alignment and harm the model performance.

### E.2 Grad-CAM Results.

Figure: Figure 8: Grad-CAM visualization of CLIP, BioCLIP, and BioCAP, given species names and biological concepts frequently mentioned in their captions. BioCAP accurately highlights these concepts and associate them with classification.
Refer to caption: x9.png

Figure: Figure 9: Finer-CAM visualization of CLIP, BioCLIP, and BioCAP, given biological concepts frequently mentioned in their captions. BioCAP accurately localizes these fine-grained concepts.
Refer to caption: x10.png

Alignment with biological traits.
Various biological concepts are mentioned in the synthetic captions to describe potential diagnostic characters of the species. We present more visualizations regarding these concepts using Grad-CAM and Finer-CAM (Zhang et al., 2025) in [Figure 8](https://arxiv.org/html/2510.20095v2#A5.F8) and [Figure 9](https://arxiv.org/html/2510.20095v2#A5.F9) as a supplement to [Figure 4](https://arxiv.org/html/2510.20095v2#S4.F4).
Compared with CLIP and BioCLIP, BioCAP demonstrates significantly better localization of these concepts. Moreover, when Grad-CAM is applied to the species name, the parts highlighted by BioCAP align with these concepts. The results indicate that the synthetic captions guide the model to focus on potential diagnostic traits, while suppressing the spurious correlation. Thereby, BioCAP demonstrates better species classification performance, which also aligns with our analyses in §[3](https://arxiv.org/html/2510.20095v2#S3) and §[A](https://arxiv.org/html/2510.20095v2#A1).

Figure: Figure 10: Grad-CAM visualization of CLIP, BioCLIP, and BioCAP, given behaviors. BioCAP correctly highlights the body parts related to the behaviors.
Refer to caption: x11.png

Alignment with behavior semantics.
In addition to the organism parts that are potentially discriminative traits of the species, the generated captions also contain descriptions regarding behaviors. We provide more Grad-CAM visualizations toward behavior in [Figure 10](https://arxiv.org/html/2510.20095v2#A5.F10) as a supplement to the previous demonstration in [Figure 5](https://arxiv.org/html/2510.20095v2#S4.F5).
For “fly,” “perch,” and “feed,” BioCAP accurately highlights wings, legs/feet, and mouth/beak/food in the images, respectively. Based on the comparison with CLIP and BioCLIP, the understanding of these behavior concepts is derived from the newly curated synthetic captions. It also validates that our synthetic captions have successfully captured a variety of biological semantics. The rich semantic understanding of BioCAP potentially enables broader applications in various biology-related tasks.

## Appendix F Discussion with Recent Work

This work proposes a new biological multimodal foundation model based on the combined supervision of taxonomic names and descriptive captions. This direction is complementary to recent advances that enrich biological models via scale or additional modalities.
Gu et al. (2025) curate a large-scale TreeOfLife-200M dataset, with which they demonstrate that scaling hierarchical contrastive learning enables emergent properties.
Gharaee et al. (2024) introduce DNA barcoding information as additional supervision.
This work does not aim to replace scaling or other supervision sources, but focuses on an orthogonal direction of how synthetic captions help bridge biological images with multimodal foundation models.
We demonstrate in this paper that descriptive captions, when grounded in biological knowledge, can significantly enhance the model’s understanding of rich semantics and its species classification performance.
The value of descriptive captions is, however, largely underexplored so far Vendrow et al. (2024).
It is also notable that the designed caption generation pipeline can be feasibly scaled up for larger datasets and integrated with more supervision dimensions through techniques like TaxaBind (Sastry et al., 2025).

A parallel line of work strengthens CLIP-style training by improving or expanding captions. FG-CLIP (Xie et al., 2025) constructs region-specific annotations, targeting fine-grained alignment capabilities.
LaCLIP (Fan et al., 2023) uses LLMs to generate multiple versions of each caption according to format examples.
VeCLIP (Lai et al., 2024) rewrites noisy web text into visual-enriched captions and mixes them with alt-text for training. CapsFusion (Yu et al., 2024) obtains descriptive captions from captioning models and refines them with large language models to achieve more semantically aligned supervision.
These advances have pushed the frontier of general-domain multimodal foundation models and also enabled the pipeline of our approach. However, previous efforts mainly focus on general images, with limited exploration of scientific domains like organismal biology. When the pipeline is naively applied to biology, we empirically observe more hallucinations about biological details and oversight on discriminative traits. As discussed in §[3](https://arxiv.org/html/2510.20095v2#S3), noisy captions can potentially harm the multimodal alignment. To this end, we explicitly inject the domain knowledge into the MLLM context. We demonstrate that the domain-specific contexts reduce hallucination and produce captions with more accurate biological details. Our work complements the gap for biological foundation models to explore the value of descriptive captions.

## Appendix G Human Evaluation

### G.1 Evaluation Instruction

### G.2 Evaluation Statistics

**Table 12: Agreement between different human evaluators. Overall values are computed across all data (micro average).**
|  | Metric |  |
| --- | --- | --- |
| Attributes | Raw Agreement | Gwet’s AC1 |
| Groundedness | $66.5$ | $0.509$ |
| Specificity | $64.7$ | $0.482$ |
| Completeness | $78.8$ | $0.724$ |
| Clarity | $80.0$ | $0.750$ |
| Overall | $72.5$ | $0.615$ |

We conduct a human evaluation to assess the quality of captions generated by three different strategies: *Base*, *Trait*, and *Trait+Example+Wiki* (*Ours*). A total of $16$ participants are involved in the study, including one ecologist and $15$ computer science students.
The evaluation covers $20$ sets of images, each set containing $10$ images, resulting in $200$ images in total.
For each image, we provide three candidate captions (one from each strategy) and ask evaluators to select the best one according to four evaluation attributes: *Groundedness*, *Specificity*, *Completeness*, and *Clarity*. Each image is independently evaluated by two different participants, ensuring two evaluation results.

We first report the win rate of each caption generation method, defined as the percentage of times a method’s caption is selected as the best among the three candidates. As shown in Table [6](https://arxiv.org/html/2510.20095v2#S4.T6), our full method (*Trait+Example+Wiki*) consistently outperforms both baselines across all four evaluation attributes, supporting the effectiveness of domain-specific contexts in improving the caption quality.

In addition, we assess the agreement between the two human evaluators. We report both the *raw agreement* (the proportion of samples where two evaluators select the same caption) and *Gwet’s AC1* (Gwet, 2008), a chance-corrected agreement coefficient designed to provide stable estimates under imbalanced category distributions. Agreement is reported separately for each attribute, and we also report a micro-average across all data. As summarized in Table [12](https://arxiv.org/html/2510.20095v2#A7.T12), both raw agreement ($72.5\%$ overall) and Gwet’s AC1 ($0.615$ overall) indicate substantial inter-evaluator agreement, corresponding to the “substantial” level in standard interpretation scales (Landis & Koch, 1977).

## Appendix H Disclosure of LLM Usage

Portions of this manuscript were polished for clarity and readability using an LLM. The LLM was not used to generate research ideas, design experiments, analyze data, or draw conclusions. All scientific content, methods, and results are the authors’ original work.

## Appendix I Limitations

We use InternVL3 38B to generate the synthetic captions. The caption generation process is biased toward the adopted MLLM. Although we employ Wikipedia-derived visual information and taxon-tailored format examples to provide biological contexts, the emphasized traits may still vary across different MLLMs. In this work, we have not investigated the influence of different MLLMs on the generated captions. We treat this as an important question and will explore it in the future.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: can caption-aligned features also help zero-shot retrieval/classification?
How do results change with different captioners?
What is the dedup protocol used? How do you ensure held-out test set?
Are there any cases of observed hallucination in the final captions?
soundness: 2
strengths: The paper proposes interesting tweaks like using images and captions as complementary views of a species’ latent trait vector and training with contrastive learning to emphasize diagnostic features. The dual projector cleanly separates heterogeneous supervision as compared to a single projector. 

The paper will be of reasonable interest for the broader community interested in training VLMs for life sciences.
summary: The paper proposes a BIOCLIP-style model that, during training, aligns images with taxonomic labels and instance-specific synthetic captions generated by an MLLM. The captions are constrained by Wikipedia-derived visual descriptions and taxon-tailored format examples to reduce hallucinations. The evaluation show strong performance for species classification and text-image retrieval by training on TreeOfLife-10: +8.8% average top-1 over BIOCLIP across 10 zero-shot species classification sets, and +21.3% average on natural-language tasks/retrieval. Ablations show that human eval favors the proposed model's captions over others. 
The qualitative analysis with Grad-CAM images and t-SNE embeddings further highlight the improved embeddings with BioCAP.
weaknesses: Captions are biased toward the chosen InternVL3-38B; and there is no cross-MLLM comparison. 
Behavior labels for analysis are auto-assigned by GPT-4o that may cause label drift. 
Large deltas are shown in experimental results, but statistical significance/confidence intervals aren’t reported for benchmarks.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: - Is there any caption noise? how did you filter those?
- What effort is needed to extend the framework to video data?
- The method is specifically designed for biology domain. How is the generalization capability to other scientific domains?
- Is there any quantitative result for Grad-CAM?
- Who are the human annotator? are they biology expert?
soundness: 3
strengths: - Multimodal alignment in biology is an under-explored and important task.
- The paper propose an interesting idea that images and captions are treated as complementary projections of a species’ latent morphospace, so aligning them helps capture diagnostic traits while suppressing noise.
- The paper introduced a dual-projector architecture elegantly separates taxonomy vs. caption supervision, and use context-guided caption generation (Wikipedia + format examples) effectively mitigates LLM hallucination.
- The evaluation is comprehensive, including multiple benchmarks, both quantitative and qualitative analysis.
summary: This work present a biological multimodal foundation model that integrates synthetic, descriptive captions as additional supervision beyond taxonomic labels to improve species understanding. The authors propose to generate synthetic instance-level captions using multimodal LLMs guided by Wikipedia-derived visual information and taxon-specific format exam, then train a model with dual text views(species name + caption, )and dual visual projectors to decouple supervision. The results shown more than 20% improvements across 10 classification tasks.
weaknesses: - The caption reliance on Wikipedia-derived descriptors could reinforce taxonomic bias and exclude rare or poorly documented species. How does the framework handle species without any Wikipedia entry or minimal trait descriptions?
- The repeated LLM re-generation could produce inconsistent style or attribute focus across species, causing the semantic drift.
- Potential scalable issue since the caption generation and derail-view training are computational costly.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - What is the taxonomic breakdown of the resulting descriptions? At line 224 you indicate the descriptions cover ~32% of the species in TreeOfLife-10M. Are there any biases in how the generated captions or is it uniformly distributed across organism groups?
- What are the limitations of the caption generation strategy? Is the 32% coverage a fundamental limit of what is available on Wikipedia? 
- Are there any potential issues when mapping genus level descriptions onto species? Does that result in multiple species having the same descriptions?
- Can you elaborate on the human evaluation task? Per the supplement, there was only one domain expert in the group of annotators and 15 computer science students. How well prepared are your human validators to assess the quality of the captions? What is the ecologist's area of expertise? 
- What is the overlap in the species coverage in the zero-shot datasets? How much of the performance improvement is related to prior knowledge of the organism groups being tested?
- At line 230, it is noted that there are 347 taxonomic classes in TreeOfLife-10M. Is that a typo? Or are classes in this case the level of the taxonomic tree?
soundness: 3
strengths: - Originality: the authors use an interesting approach to caption generation, prompting MLLMs with format specification on a per-class basis. The idea was to constrain output to focus on salient morphological description that can be challenging to pick out of raw text without guidance. The approach to leveraging the captions in BioCLIP is appealingly simple. 
- Quality: The overall qualtiy is quite good, lots of experiments executed with sufficient data. The ablation studies do a nice job of illustrating the benefits of adding these captions to the model. 
- Clarity: The paper is well-written and relatively easy to follow. There are a few sections (noted below) that could benefit from some rewriting. 
- Significance: This work provides more evidence that species classification is easier when you can direct the model to pay more attention to informative parts of the target. The use of captions, and the way in which they were generated, is a useful addition to the toolkit.
summary: The authors create synthetic instance-level captions of images from the TreeOfLife-10M dataset and use them to train a version of BioCLIP with a caption encoder. The captions are morphological trait-based descriptions of the animals that correspond to subregions in the images. The authors articulate their synthetic caption generation strategy, report benchmarks on classification and retrieval, and perform a detailed ablation study. The results indicate that the trait captions improve classification and help constrain model attention to animal parts that help with classification.
weaknesses: The format example design discussion needs expansion. This seems to be a critical element of the work and it is treated very narrowly. It isn't clear what the 'classes' are that were used to query Gemini Deep Research, who did the winnowing of the results, or how consistent the examples were across the classes. This element may itself benefit from an exploration of how variable those exemplars are between model runs and how consistent the human overseers where in selecting appropriate descriptions.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: **Questions for the Authors**

1. Why was BIOCLIP 2, the current state-of-the-art biological multimodal model, not used as the base model or included as a baseline?
2. Have the authors evaluated performance specifically on underrepresented or rare species to assess generalization across the long tail of biodiversity?
3. How sensitive is the model to the number and diversity of curated examples per taxonomic class? Would increasing this number improve generalization or trait coverage?
4. Are there analyses on how synthetic caption errors propagate through the training process, especially for species with limited textual coverage?

**Actionable Feedback**

1. Compare BioCAP against BIOCLIP 2, the current state-of-the-art, and provide justification for choosing BIOCLIP as the base model. Discussion of alternative multimodal foundation models (e.g., LLaVA) would also strengthen the paper.
2. Include few-shot classification, additional biological visual benchmarks (FishNet, Newt-AWA2, Herb-19, PlantDoc), and more diverse retrieval tasks. Explicit analysis of underrepresented or rare species would help assess generalization.  
3. Assess whether limiting to three examples per taxonomic class biases the model, and consider ablations with more examples.
soundness: 2
strengths: - **Novel approach**: The use of synthetic descriptive captions for biological multimodal models is innovative and addresses a clear bottleneck in organismal biology.
- **Strong empirical performance**: BioCAP outperforms multiple baselines (CLIP, SigLIP, BioTrove-CLIP, BioCLIP, FG-CLIP) on ten species classification benchmarks, retrieval tasks (PlantID, Cornell Bird), and natural language understanding (INQUIRE-Rerank).
- **Comprehensive evaluation**: The authors evaluate classification, retrieval, and language understanding, covering multiple organismal domains.
- **Human evaluation and careful ablation studies**: Caption quality is assessed using four metrics (groundedness, specificity, completeness, and clarity) showing the effectiveness of synthetic captions and addressing hallucination concerns. Ablations confirm that dual projectors outperform a single projector, and that adding synthetic captions improves generalization, even for species without Wikipedia coverage.
summary: This paper proposes BioCAP, a biological multimodal model that leverages descriptive captions as an additional supervision signal alongside species labels. The authors argue that images and captions provide complementary views of a species’ latent morphospace, capturing biologically meaningful traits while suppressing spurious correlations. To overcome the scarcity of instance-level captions, they generate synthetic captions using a large multimodal language model (InternVL3 38B) guided by Wikipedia-derived visual information and taxon-tailored format examples. BioCAP is trained with a shared visual and text encoder, but uses dual visual projectors for taxonomic labels and captions. Experiments show BioCAP improves species classification and text–image retrieval with several baselines, while careful ablation studies demonstrate the benefits of their approach.
weaknesses: - **Choice of base model and baselines**: BioCAP is initialized from BIOCLIP, but the paper does not justify why **BIOCLIP 2**, the current state-of-the-art, was not used as the base model or included as a baseline, despite being cited in the experimentation section. Furthermore, there is no discussion of alternative multimodal models (e.g., LLaVA) or rationale for selecting BIOCLIP.
- **Evaluation scope and claims**: Evaluation is limited to zero-shot species classification and a subset of retrieval benchmarks; prior work includes few-shot tasks and additional biological visual datasets (FishNet, NeWT, AwA2, Herb. 19, PlantDoc). 
- **Coverage limitations**: While ablation studies indicate improvements even for non-covered species, the paper does not explicitly analyze performance specifically on underrepresented species, leaving it unclear whether the approach benefits the full long-tail of species. 
- **Trait diversity in curated examples**: Only up to three examples per taxonomic class may bias the model toward common traits, potentially underrepresenting rare or atypical characteristics. An ablation study on different number of curated examples would be a nice addition.
- **Propagation of synthetic caption errors**: While human evaluation supports caption quality, errors or omissions in synthetic captions could still affect model performance for species with limited or no reliable descriptions.


Decision: Accept (Poster)