BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models
Authors: Ziheng Zhang
Xinyue Ma
Arpita Chowdhury
Elizabeth G Campolongo
Matthew J Thompson
Net Zhang
Samuel Stevens
Hilmar Lapp
Tanya Berger-Wolf
Yu Su
Wei-Lun Chao
Jianyang Gu
Venue: ICLR 2026 Poster (2026)
Keywords: AI for biology
foundation models
synthetic captions
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a species, each capturing certain biological traits. Incorporating captions during training encourages alignment with this shared latent structure, emphasizing potentially diagnostic characters while suppressing spurious correlations. The main challenge, however, lies in obtaining faithful, instance-specific captions at scale. This requirement has limited the utilization of natural language supervision in organismal biology compared with many other scientific domains. We complement this gap by generating synthetic captions with multimodal large language models (MLLMs), guided by Wikipedia-derived visual information and taxon-tailored format examples. These domain-specific contexts help reduce hallucination and yield accurate, instance-based descriptive captions. Using these captions, we train BioCAP (i.e., BioCLIP with Captions), a biological foundation model that captures rich semantics and achieves strong performance in species classification and text-image retrieval. These results demonstrate the value of descriptive captions beyond labels in bridging biological images with multimodal foundation models

Published as a conference paper at ICLR 2026
BIOCAP: EXPLOITING SYNTHETIC CAPTIONS BEYOND LABELS IN BIOLOGICAL FOUNDATION MODELS
Ziheng Zhang Xinyue Ma
Arpita Chowdhury Elizabeth G Campolongo
Matthew J Thompson
Net Zhang Samuel Stevens
Hilmar Lapp* *
Tanya Berger-Woll Wolf Yu Su Wei-Lun Chao * Jianyang Gu
The Ohio State University Duke University
+ Boston University
[zhang. 13617, gu. gu.1220)0oszl
Embedding Space
Wikipedia Derived Visual Information
Callope hummingbirds have glossy green on the back and crown with white underparts. The adult male has wine-red streoks the gorget, green and a dark tail ... Femoles and immotures hove a pinkish wash on the flanks, dark streaks throot, dark
white
Format Example
House sparrow perches on wooden fence urban setting, streoked brown back, and pole underparts biending with the surroundings..
conical bill,
MLLM
MLLM
Fly
Calliope hummingbird with green feathers, white throot, and bluish tint along the toil, with wings motion.
Female Cailiape hummingbird hovers mid air, showing glassy green upperparts, white underparts, and faint streaks on the thront, with short dark edged white.
Male
Figure I: Left: Different strategies create captions for biological images. Wikipedia offers rich domain knowledge, but descriptions are often generic and not directly grounded in the given image. Multimodal large language models (MLLMs) may hallucinate details when conditioned solely by images (wrong color description in this example). Incorporating Wikipedia-derived visual information and taxon-tailored format examples as contexts helps generate accurate, image-specific captions. Right: Using these descriptive captions additional supervision, BIOCAP captures fine- me-ggrined biological semantics. Please refer to Figure for detailed comparisons with other models.
ABSTRACT
This work investigates descriptive captions an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a species, each cap- turing certain biological traits. Incorporating captions during training encourages alignment with this shared latent structure, emphasizing potentially diagnostic characters while suppressing spurious correlations. The main challenge, however, lies in obtaining faithful, instance-specific captions at scale. This requirement has limited the utilization of natural language supervision in organismal biol- ogy compared with many other scientific domains. We complement this gap by generating synthetic captions with multimodal large language models (MLLMs), guided by Wikipedia-derived visual information and taxon-tailored format exam- ples. These domain-specific contexts help reduce hallucination and yield accurate, bastance-based descriptive captions. Using these captions, we train BIoCAP (i.e., BIOCLIP with Captions), a biological foundation model that captures rich se- mantics and achieves strong performance species classification and text-image retrieval. These results demonstrate the value of descriptive captions beyond labels in bridging biological images with multimodal foundation models!
Models, data, and code available imageomics hta.:commcc..iuuu..ohiceeeag

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Multimodal foundation models learn from vast datasets of paired visual and textual inputs (Radford et al., 2021; Liu et al., 2023; Zhai et al., 2023; Hurst et al., 2024). They demonstrate strong generalization across various downstream tasks, such as classification and visual question answering (VQA) (Yue et al., 2024; Mai et al., 2025). In general domains, the web provides a nearly limitless supply of such paired supervision (Gadre et al., 2023; Schuhmann et al., 2022). In scientific applications, however, such resources are unevenly distributed. Biomedical imaging benefits from clinical reports and radiology notes associated with image instances that offer detailed natural language supervision (Wang et al., 2022; Li et al., 2023; Zhang et al., 2024a). By contrast, many other scientific domains- including organismal biology, astronomy, geology, and material science (Stevens et al., 2024; Parker et al., 2024; Vivanco Cepeda et al., 2023; Harnik et al., 2025) lack instance-level textual descriptions and must solely rely on symbolic labels (e.g.. species names). This leaves the potential of natural- languagg captions untapped for scientific multimodal learning,
In this work, we focus on organismal biology. From the representation learning perspective, each species can be described by an underlying latent vector in the morphospace. which encodes its ground-truth biological characteristics, Le., traits (Budd, 2021). Images and descriptive captions can be viewed as two projections of this latent vector. Each captures certain traits while also introducing noise (e.g., pose, lighting). Aligning both modalities encourages the model to recover the shared latent structure and focus on potentially diagnostic characters, thereby suppressing reliance on spurious correlations to noise. While the potential exists, in practice, the effectiveness relies on the availability of captions that are both instance-specifie and faithful. Noisy or hallucinated captions can, instead, introduce contradictory signals that harm multimodal alignment (Huang et al., 2021).
Biological research has produced massive repositories of organismal images, but most are annotated only with species names (Stevens et al., 2023b; Yang et al., 2024). Collecting instance- based captions at scale is inherently challenging as expert-level annotation for millions of images demands exhaustive human labor (Van Horn et al., 2018). A natural solution is to leverage recent progress in multimodal large language models (MLLMs) automatically generate image-aligned captions (Fan et al., 2023; Lai et al., 2024). However, biological species differentiation often depends on subtle morphological details. Without proper guidance, MLLMs tend to hallucinate about these details due to the aforementioned noise in the images. Figure 1 shows such an example, where the model incorrectly describes the color of the Calliope humminghird.
Upon this observation, we suggest that domain knowledge has to be incorporated in the context. Specifically, we collect species-level visual information from Wikipedia (Wikipedia contributors, 2025) as ground-truth characters. As different species feature distinct traits, we also curate format examples based on taxonomic classes to encourage explicit focus. These domain-specific contexts help MLLMs generate accurate and trait-focused descriptions grounded in the input image. As shown in Figure 1, contextualizing the model with iikipedia-derivee visual information and taxon-tailoon format examples corrects the hallucination and grounds the caption in accurate biological details.
Building on these curated contexts, we generate instance-level synthetic captions for the large-scale TreeOfLife- 10M dataset (Stevens et al., 2023b; Van Horn & Mac Aodha, 2021; Gharaee et al., 2023; Encyclopedia of Life, 2023). We then train the BIOCAP model (i.e., BIOCLIP with Captions) with species names and captions complementary supervision. As shown the right of Figure 1, BIOCAP demonstrates a rich understanding of biological semantics. Further comparisons with BIOCLIP (trained without captions) and CLIP in Figure 7 show improved semantic alignment of BIOCAP. Quantitatively, B1oCAP is evaluated on species classification and biological natural- language benchmarks, where outperforms BIOCLIP by 8.8% and 21. 21.3%. respectively. These results answer the core question of this work: descriptive captions, when grounded in biological knowledge, provide essential supervision that bridges organismal images with multimodal understanding.
2 RELATED WORK
2.1 MULTIMODAL FOUNDATION MODELS FOR SCIENTIFIC IMAGES
Multimodal large language models (MLLMs) such as GPT-4o (Hurst et al., 2024) and LLaVA (Liu al., 2023) have demonstrated powerful ability in tasks involving images and natural language
2

---

Published as a conference paper at ICLR 2026
like captioning, VQA, and open-ended reasoning. This paradigm has been successfully extended to specialized domains: LLeVM (Li et al., 2023) adapts LLaVA to radiology for domain -peccfc VQA, ChemVLM (Li et al., 2025) reasons about molecular structures, and BiomedGPT (Zhang et al., 2024a) unifies diverse biomedical tasks under a single multimodal hackbone.
In parallel, CLIP-style contrastive frameworks have also been adapted variety of scientific imagery domains. BIOCLIP (Stevens et al., 2024; Gu et al., 2025) and BioTrove-CLII (Yang et al., 2024) align organismal images with laxonomic names. CLIBD (Gong et al., 2025) uses DNA barcodes as supervision for biodiversity tasks. MedCLIP (Wang et al., 2022) extracts standardized UMLS entities from radiology reports, and MoleCLIP (Harnik et al., 2025) clusters molecular fingerprints in chemistry, CLIP allows for more flexible supervision signals when natural language resources are not available for the specific data. Yet, also leaves the value of natural language descriptions untapped for these scientific domains. We provide a detailed discussion with concurrent work in $F.
2.2 SYNTHETIC CAPTION
One promising solution overcome the lack of instance-level annotations is to generate synthetic captions that provide fine-grained, image-grounded supervision. Recent work highlights the value of high-quality text-image pairs for CLIP-style training. BLIP (Li et al., 2022) and LLaVA (Liu et al., 2023) are widely used to produce semantically richer descriptions from large-scale web text-image collections. Building on this idea, FG-CLIP (Xie et al., 2025) produces long captions and constructs region-specific annotations, enabling fine tneegrained alignment. The caption's 's semantic relevance and granularity are critical for modality alignment. VeCLIIP (Lai et al., 2024) and CapsFusion (Yu et al., 2024) refine captions with large language models to achieve more semantically aligned supervision. ALIP (Yang et al., 2023) adopts a bi-path strategy that combines web text with synthetic captions. LaCLIP (Fan al., 2023) leverages LLMs to generate multiple paraphrases of each caption. These approaches highlight the utility of synthetic captions for improving modality alignment.
Beyond pretraining, synthetic captions have also been applied to specialized settings. Hyp-OW (Doan et al., 2024) employs dense region-level captions to expand visual vocabulary and improve worldworld detection. In the MLLM space, MiniGPT-4 (Zhu al., 2024) and LLaVA rely on GPT-4-generatee image descriptions and instruction-follovinn data to equip models with conversational capabilities.
Despite these advances, most existing efforts target general-domain imagery and emphasize caption quality or diversity. Organismal biology introduces additional challenges of fine-grat mee-grained categoriza- tion and domain-specific fidelity. To address these issues, we incorporate domain knowledge into the generation process to reduce hallucination and produce faithful, instance-specific captions.
METHOD
There have been vast curations of organismal biology images through the efforts of biodiversity researchers and citizen scientists. Many of them have reliable species labels, geolocation metadata, and in some cases, DNA barcodes. However, descriptive captions are largely absent in these datasets. Such captions, which describe human-interpretable visual traits and ecological context, provide complementary information that species labels alone cannot capture. Yet at the same time, their usefulness is dependent on being faithful and instance-specifie Such requirements make them difficult to collect at scale, leaving much of their potential untapped. In this work, we investigate descriptive captions as additional source of training supervision for biological multimodal foundation models. We first illustrate that descriptive captions, when grounded in biological knowledge, help align images and their species labels. Given the absence of large-scale curated resources, we then explore the use of MLLMS to generate instance-based synthetic captions for biological images.
3.1 BIOCAP
Let be an image embedding, y E y its taxonomic label, and the corresponding caption embedding. We train BIOCAP (i.e,, BIOCLIP with Captions) with two text views: the taxonomic label and the descriptive caption. Both views encoded by the text encoder and aligned with the image embedding. Assume an underlying trait latent vector x" in the morphospace encodes the phenotypic characters of taxon y. From causal generation perspective, the image and caption e two noisy
3

---

Published as a conference paper at ICLR 2026
"Selasphorus calllope"
Wikipedia Visual Info
Format Example
Query
Wikipedia Page
Wikipedia Descriptive Excerpts
Taxonomic Classes
Calliope humminghid
The callape is the amollest creeding ald jound in Conada and the Unwted
"Dencription': :
Amphibian
Osteichthyes
Stoter Collape hummingtiids have plany grees on the back and crown with white undiepparts.
Reptilia
Ave
Insecta
col
>tgpeorance
"Marphofogy" "
"Identificaton".. "Froture"s
sparrow; Ardea herodlas;
Redi colled hawk.
Does it contaio viusl Information?
Gemini
"Decrriptoon"
Yest it hos visuol info,
MILM
wooden
Deactptiin
levsing clets Aucia
Taxonomy
you oxtract them?
the and mepyst poan 30
above apon
and eteret the yoe in
Chocsee
Colliape humeningbirss glassy green on the back and crowe with white underparts. asluit male has wine ind streoks on the garget
MLLM
flede, brood wange and ware....
Domain- specifte Context
cidietion vabryaer oil and hergee se
Conu
Halve Bottrrtion 2o))
Caption
MILLM
Figure 2: The pipeline of collecting domain-specific context for MLLMs. We query Wikipedia with the scientific name to get the corresponding webpage. After validating the full taxonomic rank, we process the descriptive excerpts with MLLMs to extract visual information. For each taxonomic class, we randomly select up to three species and curate format examples through Gemini Deep Research. These contexts help MLLMs generate accurate and grounded descriptive captions.
observations of the same z": - x z= == E), c= c=h(2",6)
where e represents spurious environmental factors (e.g., pose, lighting) that can lead to inaccurate observation. Both the image and the caption capture certain aspects of the ground-truth traits, while being influenced by noise. When captions are involved in the training, the contrastive objective encourages the image embeddings to emphasize trait-relevant factors the object that are shared with captions, thereby reducing the influence of spurious environmental factors 6. Hence, the learned representation becomes closer to the latent trait vector 2" when the caption faithfully reflects visible and potentially diagnostic characters of the species. On the contrary, if the caption captures too much noise instead of the correct traits, the supervision may misguide optimization and degrade classification performance. We further elaborate on this in $A.
Separated visual projectors. Given that supervision in our setting is heterogeneous, we introduce two separate visual projectors after the shared visual encoder for taxonomic labels and captions, respectively (Wang et al., 2024; Ranzinger et al., 2024). When the paired text input is a taxonomic label, only the visual embedding after the taxonomy projector is matched, and vice versa. The visual encoder and the text encoder remain shared across both pathways.
3.2 SYNTHETIC CAPTION
Based on the above discussion, the captions need to provide thorough and faithful views of the images assist alignment. Traits that represent the species should be highlighted, while the information not visible or influenced by environmental factors should not he hallucinated. As shown in Figure 1, when solely conditioned by the images, MLLMs tend to generate wrong descriptions due complicated environmental factors. Therefore, we propose to collect domain-specific contexts to regularize the caption generation. Specifically, we use two context sources: wikipedia-deriver visual information and taxon-taiie axxon-tailoedd format examples. We present the pipeline for collecting these contexts in Figure 2. The prompts used to collect captions and detailed processes are listed in SB.
Wikipedia visual information extraction. Wikipedia provides a comprehensive and accessible knowledge base across the Tree of Life, with species-level descriptions that often contain appearance information. As each instance within species may have significant appearance variations, these descriptions cannot be directly associated with images. Even so, they offer morphological vocabulary that can be systematically extracted and leveraged in captions.
4

---

Published as a conference paper at ICLR 2026
As shown on the left of Figure 2, scrape Wikipedia pages based on scientific names and validate the page with the full taxonomy rank. Each page may contain various information regarding the species, such appearance, habitat, and distribution. We perform quick filtering keep the sections that potentially have visual descriptions according to their title (e.g. "Description"). Qwen3 32B (Yang et al., 2025) is then used to verify whether the kept paragraph contains visual information and to extract such information. The model is forced to only focus on attributes such as color, pattern, shape, texture, and other morphological characteristics. When no usable visual information is found for species, we apply the same process for the corresponding genus and map the description back to the species if available. This pipeline yields 132K descriptions, covering 29.5% of the 447K taxa in TreeOfLife- 10M (Stevens et al., 2024). The extracted Wikipedia visual information forms the foundation of the subsequent caption generation process.
Format example design. MLLMs might struggle to decide which traits are salient for given organism. Direct prompting without guidance often leads to hallucination or oversight of important details. Therefore, we design taxon-tailored format examples that illustrate the desired style and content of descriptive captions. For each of the 347 taxonomic classes in TreeOfl.ife TreeOff.ife-10M. 10M, we query Gemini Deep Research (Comanici et al., 2025) to retrieve candidate textual descriptions of representative species. Each query returns descriptions of up to six species, from which we manually validate trait accuracy and format consistency, and keep at most three per class. When reliable sources are scarce, fewer than three species are included. This process yields 896 curated examples, which explicitly encourage the model to attend to important traits given species labels.
Caption generation. With Wikipedia-derived visual information and taxon-tailored format examples as domain-specific contexts, we leverage MILMs to generate synthetic captions grounded in the target images. Specifically, we use Intern 38B (Zhu et al., 2025) as the backbone and accelerate inference with the vLLM framework (Kwon et al., 2023). When no Wikipedia information is available for the species, we only incorporate format examples in the context. Such process helps the model generate trait-focused deseriptions that emphasize visible appearance while avoiding irrelevant or hallucinated details. Furthermore, this pipeline enables large-scale, efficient generation of instance-level captions that complement taxonomy-based supervision.
EXPERIMENTS
BioCAP is initialized from the OpenAl ViT-B/16 CLIP checkpoint (Radford et al., 2021) and trained on TreeOfLiif- 10M (Stevens et al., 2023b; Van Horn & Mac Aodha, 2021; Gharaee et al., 2023;
Encyclopedia of Life, 2023) for 50 epochs with species labels and captions. We evaluate BIOCAP on species classification tasks following BIOCLIP 2 (Gu et al., 2025), including NABirds (Van Horn et al., 2015), Meta-Album (Ullah et al., 2022), IDLE-OO Camera Traps (Campolongo et al., 2025;
2024; Vélez et al., 2022; Balasubramaniam, 2024; Yousif et al., 2019), and Rare Species (Stevens et al., 2023a; Encyclopedia of Life, 2023; IUCN, 2022). For evaluating the understanding of natural language, we use INQUIRE-Rerank (Vendrow et al., 2024). In addition, we collect paired text-image data from PlantID (Bruce Homer-Smith and contributors to PlantID.net nett, 2025) and Comeel Bird (Macaulay Library, Cornell Lab of Ornithology. 2025) to test retrieval performance in organismal domains. Hyperparameter settings and more data information are presented $C and $D, respectively.
Island-Conservation
Seseet-Loon-Conervaaee
4.1 MAIN RESULTS
Classification, We evaluate models in the zero-shot setting on ten species classification benchmarks in Table Compared with BIOCLIP, where captions are not involved training, the accuracy increases by 23.5% on Fungi and 7.1% on Rare Species, demonstrating stronger generalization to challenging real-world settings. Overall, BIOCAP achieves an average top-1 accuracy margin of 8.8% over BIOCLIP and 27.0% over the original CLIP model (Radford et al., 2021).
Retrieval. We evaluate the natural language understanding on Bird and PlantDDRRecall@ @ 10) in Table 2. Except for the Behavior and Context tasks in INQUIRE- Rerank, BIOCAP achieves the best performance across all benchmarks. FG-CLIP is trained for fine-grained retrieval tasks (Xie al., 2025), yet BIOCAP shows clear advantages with aver-
Rerank (AP@50), Corell
5

---

Published as a conference paper at ICLR 2026
Table 1: Zero-shot species classification top-1 accuracy across 10 tasks for different models. Bold and underlined entries indicate the best and second best accuracies, respectively. BIOCAP achieves the best performance across all benchmarks, with an average improvement of 8.8% over BIOCLIP. All the compared models are based on the ViT-B/16 visual encoder.
Animals
Plants & Fungi
Camera Trap
Plant Village
Rare Species
Med. Leaf
NABirds
Plankton
PlantNet
Insccts
Insects
Fungi
Model
Random Guessing
0.2
1.2
1.0
1.0
3.5
4.0
4.0
2.6
4.0
0.3
2.2
CLIP (VIT-B/16)
39.0
3.3
7.4
9.3
28.1
52.5
8.6
5.1
15.0
25.7
19.4
SigLIP
50.2
3.7
17.6
17.6
26.7
76.3
28.3
26.1
45.4
30.7
32.3
FG-CLIP
48.3
1.9
6.9
9.3
26.4
55.6
7.3
5.9
15.7
29.4
20.7
BroThove-CLIP
39.4
1.0
20.5
15.7
64.4
38.2
15.7
31.6
24.6
26.2
BIOCLIP
58.8
6.1
34.9
20.5
31.7
88,2
40.9
19.0
38.5
37.1
37.6
BIOCAP
67.6
7.2
41.9
23.7
37.4
93.6
64.4
33.0
51.4
44.2
46,9 4
Table Performances on natural language tasks, including INQUIRE-Rerank (AP@50) and two text-image retrieval (Recall @ 10) benchmarks. Bold and underlined entries indicate the best and second best accuracies, respectively. 12T means image-to-text retrieval, and vice versa. With the additional supervision from descriptive captions, BIOCAP achieves an average performance advantage 21.9% over BIOCLIP.
INQUIRE Rerank
Cornell Bird
PlantID
Model
Appear.
Behav
Context
Species
12T
T21
I2T
T2I
Mean
CLIP (VIT-B/16)
30.8
32.9
37.2
37.1
33.8
33.0
26.0
23.7
31.8
SigLIP
34.6
37.2
41.4
36.2
48.4
48.0
43.3
39.3
41.1
FG-CLIP
28.8
31.1
32.5
41.0
50.3
48.4
28.9
28.4
36.2
SeTTrove-CLPP
28.5
22.2
30.5
39.5
16.6
15.3
48.0
51.5
31.5
BIOCLIP
27.4
27.2
30.8
41.1
16.7
48.4
45.5
31.6
BIOCAP
37.1
33.6
37.0
43.0
56.5
55.0
82.6
83.3
53.5
age performance margin of 17.3%.
These results indicate that trait-grounded synthetic captions substantially enhance fine-grained, expert-style retrieval of biological knowledge.
4.2
ABLATION STUDY
For ablation studies, we train models with TreeOfLife- 1M (Stevens et al., 2024) for 100 epochs.
Influence of different captions. As illustrated before, the alignment between captions and potential diagnostic traits is critical to the model training. We conduct ablations to examine the impact of different caption generation strategies in Table Using only taxonomic labels without captions (None) corresponds to BIOCLIP, and is set as the baseline. Adding raw Wikipedia text as captions straightforward way to involve domain knowledge. We demonstrate by Wiki Page that using non-instance-speccfic captions provides improvements in retrieval but does not help classification.
Then we incorporate MILMSs to generate instance- based synthetic captions. The Base prompt asks the model to "describe this image in short." Without emphasis on traits, the captions contain inaccurate descriptions and degrade the performance. corresponds to our previous analysis that noise in captions harms multimodal alignment. The Trait prompt explicitly requires the model to focus on traits, which substantially reduces the influence environmental nuisance. Building upon the Trait prompt, further introduce taxon-tailored format examples and Wikipedia-deriver visual information as comain--ppccffi contexts. Trait+ Example+ Wiki demonstrates the best performance, and is used generate captions for the other experiments. All the adopted prompts listed in $B.3.

---

Published as a conference paper at ICLR 2026
Table Influence of different captions. None: no caption is used in training (B1oCLIP); Wiki Page: sentence from the Wikipedia visual information; Syuthetic: captions generated by MLLMs. Syn- thetic captions involve the option of simple prompts (Base) and trait-focused prompts (Trait). Domain-specific contexts MLLM inputs: Example: format examples; Wiki: Wikipedia-derived vi- sual information. The average performances on each benchmark category (CLS: classification; INQ: ANQUIRE-Rerann are reported.
200 opochs single proj
dual proj
Performance
30
20
10-
Strategy
Retrieval
Classification
INQUIRE
Retrieval I2'
Retrieval T21
CLS
INQ
Caption
Prompt
Context
12T
T21
Figure 3: Ablation study on
None
30.2
30.5
31.2
30.0
Wiki Page
30.0
47.2
47.1
30.3
training recipes. 200 epochs: model trained for 200 epochs
Synthetic
Base
27.0
26.7
28.2
29.9
Synthetic
Trait
30.8
44.6
45.2
33.2
with species labels; single pro-
Synthetic
Trait
Example
31.8
47.5
48.1
33.9
j/dual proj: projector settings
Synthetic
Trait
Example+ Wiki
33.8
54.7
54.3
34.8
for caption alignment.
Table 4: The influence of the training set components. We incrementally add Wikipedia-coveree and non-covered species into training with different supervisions to validate the generalization of synthetic captions. The results on the covered/non-covers test species are reported. Name: species name; Caption: synthetic captions.
Table 5: Win rate of different syn- thetic captions in the human evalua- tion. Base: naive prompt: Trait: trait- focused prompt; Ours: Trait-domain- specific contexts.
Method
Training Component
Classification
Covered
Non-covered
Covered
Non-covered
Attributes
Base
Trait
Ours
Name
35.7
40.1
Groundedness
5.7
11.9
82.4
CapeoCa
37.5
42.7
Specificity
10.3
8.9
80.8
Name
Name
35.8
43.5
Completeness
5.1
7.6
87.3
Name+Caption
Name
38.6
46.6
Clarity
5.1
5.7
89.2
Name-Caption
NamtoCaption 45.3
48.8
Average
6.6
8.5
85.0
Training recipe. In BIOCAP, we adopt two separate visual projectors to align species names and captions, respectively. To ablate this design, train model with single visual projector that aligns species and captions at the same time, named as single proj in Figure 3. Our two-projector design is named dual proj. The results show that dual proj consistently outperforms single proj. which validates the necessity of decoupling supervision signals in heterogeneous multimodal training. Additionally, caption generation introduces additional computation time. To assess the fairness, we naively double the training epochs and report the results as 200 epochs in Figure 3. The results show an obvious gap between 200 epochs and other variants, supporting the worth of synthetic captions.
Training set components. Wikipedia provides faithful domain knowledge as contexts for MLLMs. However, it does not cover all the species for usable visual descriptions (See BB.2). We incrementally add different training components to evaluate the generalization of synthetic captions in Table 4. Specifically, we partition the training set based on whether the species are covered by Wikipedia (37.1% of the total taxa in TreeOfL.ife-IM) We also separate classification benchmarks into the two groups (76.2% covered by Wikipedia). In the table, Name refers to taxonomy-only supervision, while Name+ Caption denotes the joint supervision with synthetic captions. The results show that adding captions to Wiki-covered species also enhances the understanding of non-covered species. Incorporating both taxonomy and captions for all species achieves the best results, even if many species are not covered by Wikipedia, indicating generalization of caption-encoded knowledge.
Caption Evaluation. We conduct a human evaluation to assess the quality of the generated captions. As shown in Table 5, we evaluate the captions along four metries (Groundedness: if the description the
is visible in the image; Specificitty if the caption describes distinctive traits; Completeness: caption covers 2-3 most salient aspects; Clarity: preciseness and objectiveness). We randomly sample 200 images from TreeOfLife, covering 200 distinct taxonomic classes. For each image, we provide three captions: generated by our full method (Trait + Example + Wiki) and two obtained from
7

---

Published as a conference paper at ICLR 2026
Table 6: Influence of example number. We vary the number of taxon-tailored format examples per class and report the resulting performances.
Table Comparison across caption genera- tors and model sizes. We report the resulting performances under each setting.
Num
CLS
Retrieval
INQ
Generator
Size
CLS
Retrieval
INQ
Examples
12T
T21
12T
T21
InterVL33
8B
31.7
49.4
50.2
33.7
33.2
52.7
53.2
33.9
InterVVL3
78B
33.9
54.8
55.6
34.5
3
33.8
54.7
54.3
34.8
IntervVL
38B
33.8
54.7
54.3
34.8
5
34.0
54.3
54.7
34.5
Qwen-2.5-VV 2.ccn-2.-VLL VI.
32B
34.1
53.3
55.5
35.2
7
33.7
54.5
54.6
34.6
I.aaVANeXT
34B
32.9
54.1
34.7
34.4
Image & Species
Trait+Exampler Wild (Ours)
Trait
Base
The red hot poker shows tall conical infforescence orange- to-yellow tubulor flowers rising obove pond with water lilles.
striking red hot poker with fiery tubular flowers fading yellow, rising above pond lilies, ottrocting hummingbird's nearby.
A tail arange and yellow flower standing by water, shoped like torch, with green plants around it.
Red Hot Poker
The White- Mhite-Eared Honeyeater displays black head, white ear- coverts, diive-green body while perched branch
The White- Eared Honeyeater shows an olive back, pale ear spot, dark standing
White-Eared
small bird with green feathers and white potch sits quietly on branch.
White Eored Honeyeater
The common elbow archid displays small, insect-like flowers on slender, arching stem with few bracts.
The common elbow archid shows what looks like small insect clinging to thin stem delicate ports.
A tiny bug shope sits stick, seen against blurry background.
Common Elbow Orchid
Figure 4: Captions generated by different strategies (refer to Table 3 and $B.3). The domain- specific contexts of taxon-tailored format examples and Wikipedia-derived visual information signifi- cantly reduce hallucination and provide more accurate descriptions of the target object.
alternative strategies (Base and Trait). The order of captions is randomized. Human evaluators are asked to select the best caption under each metric. Each caption is independently assessed by two evaluators. The results show that captions generated by our full pipeline recognized by human evaluators to be accurate and image-specific. Detailed statistics are included SH.
Ablation format examples. Format examples guide MLLMs to emphasize different traits across species. We examine the influence of varying the number of examples per taxonomic class in Table 6. Using only example for the entire class limits diversity, leading suboptimal performances. Three or more examples yield similar results. Considering performances and computational cost, we adopt three examples per class in the other experiments. We also include generating format examples at the order level instead of the class level and assess the stability across multiple runs in $E.3.
MLLMM family and model size. We evaluate using different MILMSs to generate descriptive captions, including LLaVA-NEXT (Zhang et al., 2024b), Qwen2.5-VL (Bai et al., 2025), and Intern VL3. We also vary the parameter scales of VL.3 (8B, 38B, 78B) and report the results in Table We observe that captions produced by Qwen2.5-VL. and VL3 lead almost the same downstream performance, while SaaaV--ggneatee captions result in slightly lower performance. Overall, the proposed caption generation pipeline is robust across different MLLMs. For model size, the 8B variant produces weaker results, whereas the 38B and 78B models show almost the same performance. Therefore, select Intern VI.3 38B for the other experiments. Overall, the caption-generation pipeline does not rely on extremely large or highly specialized MLLMs, and the proposed framework remains stable across both model families and parameter scales.
4.3 ANALYSIS AND DISCUSSION
Qualitative comparison of captions. Figure illustrates qualitative comparisons of generated captions. The second column corresponds to our full strategy (Trait+ Example+ Wiki). Using the Base or Trail prompts without domain-specific contexts leads hallucination (the common elbow
8

---

Published as a conference paper at ICLR 2026
Input
CLIP
BioCLIP
BioCAP
CLIP
BioCLIP
BioCAP
CLIP
BioCLIP
BioCAP
"Tail"
"Crest"
"Northern Cardinal"
"Antennae"
"Legs"
*Lygus lineolaris'
"Petal"
"Pistil"
"Peruvian Lily"
Figure 5: Grad-CAM visualization of CLIP, BIOCLIP, and B1oCAP, given species names and biological concepts frequently mentioned in their captions. Comparatively, BIOCAP offers comprehensive understanding of these concepts and connects them to the corresponding species.
Input
CLIP
BioCLIP
BioCAP
Input
CLIP
BioCLIP
BioCAP
"Fly"
"Crawl"
"Feed"
"Fly"
Figure Grad-CAM visualization CLIP, BIOCLIP, and BIOCAP, given behaviors. BIOCAP correctly highlights the body parts related to the behaviors.
orchid flower misrecognized as an insect) and inaccurate descriptions (vague color descriptions for the white-eared honeyeater). In contrast, introducing format examples and Wikipedia-derived visual information significantly improves the instance-specd instance-speciffiity and faithfulness of the synthetic captions.
Why are captions helpful for classification? To better understand the effects of captions, we use Grad-CAM (Zhou et al., 2016) to visualize model attention given species names and hign-frequency biological concepts mentioned the corresponding captions in Figure The visualization shows that B1oCAP learns to localize biologically meaningful traits and associate them with the species names. corresponds to our analysis in $3.1 that captions help model focus on diagnostic characters during training, and thereby improve classification performance. The information Grad-CAM implementation and figh-frequency concept is presented SC.3.
Semantic Understanding, In addition specific traits, further analyze whether captions help align embeddings with broader biologically meaningful semantics (Figure 6). Using Grad-CAM, we visualize model attention given behavior-related words such as fly, feed, and crawl. Compared with CLIP and BIOCLIP, BIoCAP accurately highlights the body parts related to these behaviors, for instance, activation of wings for "fly" and legs for "crawl."
Beyond instance-level understanding, we also visualize the relationship between instances with t-SNE across three bird species Figure 7, annotated with both behaviors (perch, fly, stand) and sex (male, female/immature). General-purpose models like CLIP and DINOv3 form loose species clusters and conflate sex distinctions. They also mistakenly align female/immature red-winged blackbirds with brown-headed cowbirds. BIOCLIP learns the species distinction, but fails to differentiate the behavior variations. Comparatively, BIOCAP produces compact species clusters and separation of various biological semantics. These results further demonstrate the effectiveness of descriptive captions in enhancing the understanding various biological concepts. The collection process of behavior labels is presented in $C.4. We provide more qualitative results in $F.

---

Published as a conference paper at ICLR 2026
DINOv3
CLIP
BioCLIP
BioCAP
Condhied
Blackbird
Blackbied
Red-wing winged blackbird
Calliope hummingbird
Brown-headed cowbird
Immature
Male
Fly
Fly
Stenn
Perch
Perch
Fly
Stand
Figure 7: Embedding distribution of three bird species with sex and behavior annotations. On the right, we provide example images corresponding to each label. DINOv3 and CLIP fail to align male and female red-winged blackbirds while mixing male and female hummingbirds. BIOCLIP does not capture the semantical difference between behaviors. With the guidance captions, BIOCAP tells the subtle difference between perch and stand and accurately separates the behavior variants.
CONCLUSION
This paper investigates using instance-level descriptive captions as complementary supervision for biological multimodal foundation models. Due lack of such resources at scale, we incorporate multimodal large language models (MLLMs) generate synthetic captions. We curate Wikipedia- derived visual information and format examples to reduce hallucination and produce accurate, instance-specific captions. Aligning images with captions encourages the model to emphasize potential diagnostic traits while reducing the influence environmental factors. The acquired broad range of biological semantics. The
BIoCAP model demonstrates rich understanding superior performance in species classification and biological text-image retrieval highlights the value of descriptive captions bridging biological images with multimodal foundation models.
ETHICAL STATEMENT
Our study involves human evaluation of automatically generated captions for organismal biology images. Participants were asked only express preferences between different caption candidates based on the associated image. No personal or identifiable information was collected, and the task posed minimal risk. Participation was voluntary. and participants were engaged without coercion. As the task did not involve sensitive data, medical decisions, or personal attributes, institutional review board (IRB) approval was not required under our institution's policies. We emphasize that the generated captions are intended for scientific research purposes. Nonetheless, we acknowledge the potential risk of inaccurate or misleading captions, and we therefore recommend their use only as research tools and not authoritative sources.
REPRODUCIBILITY STATEMENT
We have released all source code, including modules for model training, evaluation, caption genera- tion, and Wikipedia scraping, https://github. httptthhb..ooommmggommmm///iie
The corresponding dataset, extended from TreeOfLife. 10M with generated captions, is available at TreeOfLife 10M- NOM-Captions. The details regarding caption generation presented in ŞB. The training hyperparameters and details to reproduce our evaluation results are included in SC. The information on adopted benchmarks is listed in ŞD.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like to thank Wasila Dahdul, Zhiyuan Tao, Yifan Liu, Fangxun Liu, Shuheng Wang, Ziqi Li, David Carlyn, Quang-Huy Nguyen, Yintie Lei, and Junke Yang for their help with the human evaluation. We also thank Hong-You Chen and the Imageomics Team for their constructive feedback.
We sincerely thank PlantID.net (Bruce Homer-Smith and contributors PlantID. 2025), well as the Cornell Lab of Ornithology (Macaulay Library, Cornell Lab of Omithology, 2025) for providing access to their biological media collections. The data made our retrieval evaluation possible.
Our research is supported by NSF OAC 2118240 and resources from the Ohio Supercomputer Center (Center, 1987). The authors are grateful for the generous support of the computational resources from the Ohio Supercomputer Center.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong. Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-v vl technical report, 2025. URL https: //arxivaaaivv.//aa//aaa org/abggss...oo abs///b/////////// 25t./2502..222 13923.
S. Balasubramaniam. Optimized classification in camera trap images: An approach with smart camera traps, machine learning, and human inference. Master's thesis, The Ohio State University, 2024. URL http://rave ohiolink edt./ettccvieee/cccccccccccccc num-
osu172141769543066
Bruce Homer-Smith and contributors to PlantID.net. Plantid - online plant identification resource, 2025. URL https:://// aanntid. net/. Content licensed under CC BY-NC 3.0. Developed and produced by Bruce Homer-Smith with contributions from Dave Long, Doreen Smith, Kristin Jakob, John Malpas, and others.
Graham E Budd. Morphospace. Current Biology, 31(19):R1181-R1185 2021.
Elizabeth G Campolongo, Jianyang Gu, and Net Zhang. IDLE-00 Camera Traps, 2025, URL 1tt.-//------aa---araaaaa- https: httpps:/hugginngacce co/ dataaatss/////a// www./maaaaammoooomomm..........a es/ (atmaaaeeomme/ii------------
Ohio Supercomputer Center. Ohio supercomputer center, 1987. URL htt.htttp//osc..d//aaat
/19495/f5s1phT3.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025.
Sitert-Lion-Conervaall Desert lion conservation camera traps, July 2024. URL https:// camera camera-trapst
lila. scwence//eececc.cc..coo.. setttcassts//eeeeeee
Thang Doan, Xin Li, Sima Behpour, Wenbin He, Liang Gou, and Liu Ren. Hyp-ow: Exploiting hierarchical structure learning with hyperbolicc distance enhances open world object detection. In AAAI, pp. 1555-1563, 2024.
Encyclopedia of Life. Encyclopedia of life (col). http:://ol..../ 2023. Accessed 29 July 2023.
Facebook, Inc. Threatexchance. 2019.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. In NewrIPS, volume 36, pp. 35544-355555 2023.

---

Published as a conference paper at ICLR 2026
Geetharamani G. and Arun Pandian J. Identification of plant leaf diseases using nine-layer deep convolutional neural network. Computers & Electrical Engineering, 76:323-338, 2019. ISSN 0045-7906. doi: rect. com/ sci ence/ /ttci//ece/aattcccee////////////
URL https: /www. 9300023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymiis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems, volume 36, pp. 27092-27112, 2023.
Camille Garcin, Alexis Joly, Pierre Bonnet, Antoine Affouard, Jean-Christophe Lombardo, Mathias Chouet, Maximilien Servajean, Titouan Lorieul, and Joseph Salmon. PI@ ntnet-300k: a plant image dataset with high label ambiguity and long-tailed distribution. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 36, 2021. URL /ttep:://opereeee net 1twpe//iiee.eee/orouidd------ www.ummiddeLYinnia
Z. Gharaee, Z. Gong. N. Pellegrino, I. Zarubiieva, J. B. Haurum, S. C. Lowe, J. T. A. McKeown, C. Y. Ho, J. McLeod, Y. C. Wei, J. Agda, S. Ratnasingham, D. Steinke, A. X. Chang, G. W. Taylor, and P. Fieguth. step towards worldwide biodiversity assessment: The BIOSCAN-IM insect dataset. In NeurlPS, 2023.
Zahra Gharaee, Scott C Lowe, ZeMing Gong, Pablo Millan Arias, Nicholas Pellegrino, Austin T Wang, Joakim Bruslund Haurum, Iuliia Eyriay, Lila Kari, Dirk Steinke, al. Bioscan-5m: multimodal dataset for insect biodiversity. In NewrIPS, volume 37, pp. 36285-36311 2024.
ZeMing Gong, Austin Wang, Xiaoliang Huo, Joakim Bruslund Haurum, Scott C. Lowe, Graham W. Taylor, and Angel X Chang. CLIBD: Bridging vision and genomics for biodiversity monitoring at scale. In ICLR, 2025.
Jianyang Gu, Samuel Stevens. Elizabeth G Campolongo, Matthew Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E White, James Balhoff, et al. Bioclip 2: Emergent properties from scaling hierarchical contrastive learning. In NeurIPS, 2025.
K. L. Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, (Pt 18:29-48, 2008.
Yonatan Harik, Hadas Shalit Peleg, Amit H. Bermano, and Anat Milo. Data efficient molecular
image representation learning using foundation models. Chemical Science, 16(24); 10833310411 2025.
Emily F. Brownlee Heidi M. Sosik, Emily E. Peacock. Annotated plankton images data set for developing and evaluating classification methods, 2015. URL http:: //hll han.le..ntt........ 10. 10.1555/199 /1112/111731.
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. In NeurIPS, volume 34, pp. 29406-294199 2021.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276, 2024.
-sland-Conservation
Island conservation camera traps. datasets/is conservation camera- camera-traps/
URL https:://lla.cciiee
IUCN. The iuen red list of threatened species. version 2022-2. https://www.uucet org, 2022. Accessed on 5 July 2023.
Faizan Farooq Khan, Xiang Li, Andrew J Temple, and Mohamed Elhoseiny. Fishnet: A large-scale dataset and benchmark for fish recognition, detection, and functional trait prediction. In ICCV, pp. 80496-20506, 2023.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Alcksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Improving clip training via visual-enriched captions. In ECCV, pp. 111-127, 2024.
R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-177) 1977.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama. Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. LLa Training large lunguage-and-vist assistant for biomedicine in one day. In NeurlPS, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-iman image pre-training for unified vision-language understanding and generation. In /CML., volume 162, pp. 12888-12900 2022.
Junxian Li, Di Zhang, Xunzhi Wang. Zeying Hao, Jingdi Lei. Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Wei Li, Mao Su, Shufei Zhang, Wanli Ouyang, Yuqiang Li, and Dongzhan Zhou. Chemvim: Exploring the power of multimodal large language models chemistry area. In AAAI, pp. 415-423, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), NeurlPS, volume 36, pp. 34892-34916, 2023.
Macaulay Library, Comell Lab of Omithology. Macaulay library: Multimedia resources for birds and other animals, 2025. URL / http://wwwwaaaaaaaa////a/aaa hat.caullllbbbrryyyy org.
Zheda Mai, Arpita Chowdhury, Zihe Wang, Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, and Wei-Lun Chao. Ava-bench: Atomic visual ability benchmark for vision foundation models. arXiv preprint ar arXiv:2506.0000 2025.
Ernst Mayr and Peter D. Ashlock. Principles of Systematic Zoology. McGraw-Hill, 2nd edition, 1991.
Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Rudy Morel, Ruben Ohana, Mariel Pettee, Bruno Régaldo-Sain Blancard, Kyunghyun Cho, and Shirley Ho. Astroclip: a cross- modal foundation model for galaxies. Monthly Notices of the Royal Astronomical Society, 531(4): 4990-501 1, June 2024. ISSN 1365-2966.
Lukáš Picek, Milan Šule, Jifí Matas, Thomas S Jeppesen, Jacob Heilmann Clauusen,, Thomas Læssøe, and Tobias Frøslev. Danish fungi 2020-not just another image recognition dataset. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pp. 1525-1535. 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamila Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI. pp. 8748-8763, 2021.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In CVPR, pp. 12490-12500, 2024.
Roopashree S and Anitha J. Medicinal leaf dataset, Oct 2020. URL hwtph:tpp//eattt /ttpa///aaaaaaaaaaaaaaaaaaa wwndeleneeeeeeee ey.
com/dataset //n.settsstttttttteeeeee nnyt: j2v3n5/1.
Srikumar Sastry, Subash Khanal, Aayush Dhakal, Adeel Ahmad, and Nathan Jacobs. Taxabind: A
unified embedding space for ecological applications. In WACV, pp. 1765-1774. IEFE, 2025.
13

---

Published as a conference paper at ICLR 2026
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Advances in neural information processing systems, volume 35, pp. 25278-25299 2022.
Hortense Serret, Nicolas Deguines, Yikweon Jang, Gregoire Lois, and Romain Julliard. Data quality and participant engagement in citizen science: comparing two approaches for monitoring pollinators in france and south korea. Citizen Science: Theory and Practice. 4(1):22, 2019.
Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra. Plantdoc: A dataset for visual plant disease detection. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, CoDS COMAD 2020, pp. 249-253, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450377386. doi: 10.1145/3371158.3371196 3371196.
URL wwtptttp/////m /tt.///oiiiio/// www.ooog//......o... 1tt.//1.1111155 11w..111533333
Samuel Stevens, Jiaman Wu, Matthew Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su. Rare species, 2023a. URL https:/ htapps///hhgiingggaa wtt.aaccc..oooaaaeeeeeeeeeoeeeee co/datasess/ imageemicc es/ rare-
Samuel Stevens, Jiaman Wu, Matthew Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger- Wei-Lun Chao, and Yu Su. TreeOfLife- 10M, 2023b. URL httph::t////ttt /hwgg inggiinnaacee wtt.aeeeeeceeeeeeeoooooooeeeoeee co/datasetss imageomice Saaageomlcs/TreeefLlff....
Samuel Stevens, Jiaman Wu, Matthew Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger- Wolf, Wei- Lun Chao, and Yu Su. BioCLIP: A vision foundation model for the tree of life. In CVPR, pp. 19412-19424, June 2024.
Perger-Wolf,
Kiat Chuan Tan and Yulong Liu. Herbarium challenge 2019 fgvc6. http:::s//agggle..o com/
um 2 fgvc,, 2019. Kaggle.
Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. Meta-album: Multi-domain meta- dataset for few-shot image classification. In NeurIPS, 2022. URL https: /ttp////aaaaaa///eeeeeeee aumb.m github. ic/.
Grant Van Horn and Oisin Mac Aodha. inat inat challenge 2021 fgve8, 2021. URL httttt:tt//kaggt com/ cwwpetotttiionooooooe /in/tnnnatuaasttttmaa 2021.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen The fine print in ene-graieed dataset collection. In CVPR, pp. 595-604, 2015.
scientists:
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, pp. 8769-8778. 2018
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In CVPR, pp. 12884- 12893. 2021.
Juliana Vélez, Paula J
Michael A Tabak, Carl Chalmers, Paul Fergus, and John Fieberg, Choosing an appropriate platform and workflow for processing camera trap data using artificial intelligence. arXiv preprint arXiv:2202.02283. 2022.
Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate E Jones, Oisin Mac Aodha, Sara Beery, and Grant Van Horn. Inquire: natural world text-to-image retrieval benchmark. NewrIPS, 2024.
Vicente Vivanco Cepeda, Gaurav Kumar Nayak, and Mubarak Shah. Geoclip: Clip-inspired alignment between locations and images for effective worldwide gco-localization. In NeurlPS, volume 36, pp. 8690-8701. Curran Associates, Inc., 2023.
14

---

Published as a conference paper at ICLR 2026
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-200 dataset. 2011.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: weightee visual explanations for convolutional neural networks. In CVPRW, pp. 24-25, 2020.
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. CVPR, pp. 3635-3647, 2024.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICMI., pp. 9929-9939. PMLR, 2020.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. MedCLIP: Contrastive learning from unpaired medical images and text. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), EMNLP, pp. 3876-3887, 2022.
Wikipedia contributors. Wikipedia, the free encyclopedia. https://en.wwkiipeett wiki/, 2025. Accessed: August 2025.
Xiaoping Wu, Chi Zhan, Yukun Lai, Ming-Ming Cheng, and Jufeng Yang. IP102: A large-scale benchmark dataset for insect pest recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8787-8796, 2019.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE TPAMI, (199:225512225. 2018.
Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang. Dawei Leng, and Yuhui Yin. FG-CLIP: Fine- -rained visual and textual alignment. In ICML, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505. 09388, 2025.
Chih-Hsuan Yang, Ben Feuer, Zaki Jubery, Zi K. Deng, Andre Nakkab, Zahid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh K Singh, Arti Singh, Soumik Sarkar, Nirav Merchant, Chinmay Hegde, and Baskar Ganapathysubramanian Biotrove: A large curated image dataset enabling ai for biodiversity. In NeurIPS, volume 37, pp. 102101-102120, 2024.
Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic caption. In ICCV, pp. 2910-2919, 2023.
Hayder Yousif, Roland Kays, and Zhihai He. Dynamic programming selection of object proposals for sequence-level animal species classification in the wild, IEEE Transactions on Circuits and Systems for Video Technology. 2019.
Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data scale. In CVPR, pp. 14022-14032, June 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, pp. 9556-95677 2024.
15

---

Published as a conference paper at ICLR 2026
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pp. 11941-11952, 2023.
Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision-language foundation model for diverse biomedical tasks. Nature Medicine, pp. 1-13, 2024a.
Yuanhan Zhang, Bo Li, Haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model. Llava-next: A strong zero-shot video understanding model, 8, 2024b.
Ziheng Zhang, Jianyang Gu, Arpita Chowdhury, Zheda Mai, David Carlyn, Tanya Berger- Wolf, Yu Su, and Wei-Lun Chao. Finer-cam: Spotting the difference reveals finer details for visual explanation. In Proceedings the Computer Vision and Pattern Recognition Conference (CVPR), pp. 961 1-9620, June 2025.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. CVPR, pp. 2921-2929) 2016.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
16

---

Published as a conference paper at ICLR 2026
Table of Contents in Appendix
A Analysis on the Impact of Captions in Training
18
Details on Caption Generation
19
B.1 Prompts
19
B.2 Statistics of Caption Collection
22
B.3 Baseline Prompts
22
B.4 Genus-to-Species Description Mapping
23
C Experimental Details
24
C.1 Hyperparameters
24
C.2 Text-Image Retrieval
24
C33 Grad-CAM Visualization
24
C,4 Behavior Semantic Annotation
25
Benchmark Details
26
D.I Collection of Retrieval Benchmarks
26
D.2 Information of Other Benchmarks
................................
26
D.3 Duplicate and Leakage Control
26
D.4 License Information
27
E More Qualitative and Quantitative Evaluations
28
E.1
Synthetic Caption Comparisons.
28
E.2
Grad-CAM Results.
29
E,3
Format Example Design
E,4
Performance Underrepresented Species
31
E,5 Performance on Few-shot Classification and Biological Visual tasks
32
F
Discussion with Recent Work
34
G Broader Applicability to Scientific Domains
35
Human Evaluation
36
H.l Evaluation Instruction
36
H.2 Evaluation Statistics
37
I Disclosure of LLM Usage
37
17

---

Published as a conference paper at ICLR 2026
A ANALYSIS ON THE IMPACT OF CAPTIONS IN TRAINING
Let æ be an image embedding extracted by the visual encoder, y E y the taxonomic label, c the textual embedding of a corresponding descriptive caption, z" the underlying latent vector of ground- truth traits associated with the taxon y, and e I x* the environmental noise influencing the trait observation. Assume the image a and the caption are derived from a linear transformation on the latent vector x" and noise E:
x Az Ge la, c= + DE TJc,
where V/ and l/e are independent zero-mean Gaussian noises. The label y can be directly determined by z" and is irrelevant to the noise The target is to optimize the encoders so that æ: is aligned with x", and thereby the label y can derived from æ. We train the model with InfoNCE loss:
LZCE = (p(mn.) [s(x,, c) + log p( Σ exp exp
where
1 (ar, c) - (cc).
Wang & Isola (2020) show that the contrastive loss optimizes alignment and uniformity properties. The alignment part increases the image-caption inner product E j cl. while the uniformity part preserves maximum information. For 12 fg-normalized features, E x tt(Ewe), iee, the cross- covariance between the two views. We have:
Ege AB + -GD..
which decomposes into the trait-shared component AB and the nuisance-shared component GDT captions capture the diagnostic characters without noise disturbance (D = 0), the trait term AB dominates the cross-covariance. The learned image projection, therefore, aligns with trait directions and drops nuisance. Thus, the faithful caption e helps species classification. On the contrary, if the caption covaries with image nuisance c, GD adds to Eze. The InfoNCE loss pulls the image embedding toward the range of G, which leads to spurious correlation and potential performance drop. Based on these analyses, we propose to generate synthetic captions that ground biological knowledge enhance the alignment between images and labels.
18

---

Published as a conference paper at ICLR 2026
B DETAILS ON CAPTION GENERATION
In this section, we list the adopted prompts and pipeline details for collecting descriptive captions. also present some ablation the visual information extraction pipeline.
B. PROMPTS
Format example generation. For each of the 347 taxonomic classes in TreeOfLife-IOM. query Gemini Deep Research (Comanici et al., 2025) to retrieve candidate textual descriptions of representative species. Each query returns up six candidate descriptions together with images, each representing one species within the class. We manually validate trait accuracy and format consistency based on the images of the species through winnowing process conducted by two evaluators, both with interdisciplinary training in biology and computer science. Each candidate description is independently evaluated and retained only if it satisfies the following criteria: (I) visual groundedness, where all mentioned morphological traits, colors, and structural attributes must be directly observable in the corresponding exemplar image, with additional cross-checking against reliable public sources; (2) format consistency, requiring adherence to the predefined caption template with concise descriptions focusing on 2-3 salient visual traits while excluding non-visual attributes such as habitat; and (3) diversity coverage, ensuring that the retained examples within each taxonomic class represent distinct species or visually diverse variants rather than redundant descriptions. Candidate descriptions containing hallucinated unverifiable traits discarded. The two evaluators first assess candidates independently, and any disagreements are resolved through joint re-examination. We keep at most three curated examples per class, and when reliable sources are scarce, fewer than three species are included. The images are used solely for validation of format examples and are not used in model training. This process yields 896 curated examples, which are then incorporated into the context to guide the model in generating trait-focused captions in consistent and grounded style. The prompt used for Gemini Deep Research is listed below.
Prompt for Format Example Generation
You are a biologist describing organisms strictly on the basis of visible characteristics in an image. Your task is to generate short, fluent, and biologically meaningful captions for a given class, using examples drawn from different species within that class. Captions must be based on real samples and grounded in visual evidence.
Requirements:
Provide diverse format examples captions from different species within the class.
Captions must emphasize salient visual traits (e.g., color, shape, pattern, texture, body structure).
If clearly visible, background or environmental features may be included, but only when they are explicitly apparent in the image.
Each caption must contain either the scientific name or the common name (not both).
Do not begin directly with the name; instead, weave it naturally into the caption text.
Each caption must not exceed 35 words.
Each caption must be linked to a corresponding image URL, which should point to the actual visual sample used for description.
Maintain concise, scientific style, with variation across examples.
If the class contains too few distinctive species, provide fewer than three examples; if no usable information is available, output "N/A".
Output Format:
Two columns for all provided classes:
1. Class name
19

---

Published as a conference paper at ICLR 2026
2. Examples (listed 1, 3, ...), each followed by its corresponding image URL
Now, generate examples for given classes: {classes)
Wikipedia visual information extraction. Format examples encourage MLLMs to focus on im- portant traits for different species. We also leverage Wikipedia as large-scale resource provide detailed visual information across the Tree of Life. The descriptions of visual information on Wikipedia pages often mixed with habitat, behavior, and distributional information. We perform a quick filtering based on the section titles of the Wikipedia pages and keep those with potential visual information, including "description", "morphology", "appearance", "identification", "feature", "characteristies., "physical", "structure", and "explanation of names."
After the quick filtering, design verifying and extraction pipeline to further extract visual information such as color, pattern, shape, and texture. This process yields over 132K trait-focused descriptions, covering 29.5% the 447K taxa in TreeOfLife- (Stevens et al., 2024). The prompt used to filter and extract morphological traits from Wikipedia is provided below.
Prompt for Wikipedia Visual Information Verification
You are given textual description of a species. Your task is to determine whether the description contains any information about the species' visible appearance (including features, colors, shapes, patterns, textures, or other morpho- logical characteristics).
Respond strictly with: "Yes" or "No".
Examples:
"Bagada is genus of moths of the family Noctuidae." No
"Aetheolacna rosana is species of flowering plant in the family Asteraceae. It is found only in Ecuador. Its natural habitat subtropical or tropical moist montane forests. It is threatened by habitat loss." No
"The fur of the African wild dog differs significantly from that of other canids, consisting entirely of stiff bristle-harrs with no underfur. Colour variation is extreme, and may serve in visual identification." + Yes
"The most characteristic physical feature of the raccoon is the area of black fur around the eyes, which contrasts sharply with the surrounding white face coloring." + Yes
Now classify the following description: "(content)"
Prompt for Wikipedia Visual Information Extraction
You expert taxonomy editor. Extract only the sentences (or partial sentences) that describe visual appearance:
Colours, patterns, shapes, sizes, textures, diagnostic marks - anything visible photo.
Visual differences in sex, form, or life stage should be preserved.
Do not include behaviour, distribution, threats, taxonomy, dates, or references.
Remove all non-visual parts from the original paragraph while maintaining sentence structure.
Keep exactly the same descriptions from the original input; do not rewrite or rephrase.
Return exactly in the format: <species>
<caption>
20

---

Painted Bunting Published as a conference paper at ICLR 2026
User Examples:
"The fur of the African wild dog differs significantly from that of other canids, consisting entirely of stiff bristle-hairs with no under-fur. Colour pattern is patchy black, yellow ochre and white." " + Lycaon pictus The fur of the African wild dog consists entirely of stiff bristle-hairs with no under-fuse fur. Colour pattern is patchy black, yellow ochre and white.
"The most characteristic physical feature of the raccoon is the area of black fur around the eyes, which contrasts sharply with the surrounding white face colouring." Procyon lotor the area of black fur around the eyes, which contrasts sharply with the surrounding white face colouring.
"The male painted bunting is often described as the most beautiful bird in North America...
colors, dark blue head, green back, red rump, and un- derparts, make it extremely easy to identify... The plumage of female and juvenile painted buntings is green and yellow-green... The adult female brighter, truer green than other similar songbirds."
The male painted bunting has a dark blue head, green back, red rump, and red underparts, making it extremely easy to identify, though it often hides in foliage. The female and juvenile painted buntings have green and yel sellowggreen plumage, which serves as camouflage, The adult female is brighter, truer green than other similar songbirds.
Now extract:
<species>: (species) <description>: "(description)"
The design of separating the verification and extraction steps based the consideration of efficiency. In our experiment, Qwen3 8B is used for verification and Qwen3 32B is used for extraction (Yang al., 2025). Compared with a single-step design integrating both verification and extraction with Qwen3 32B, the separate design saves 13% computational time. After manual examination on 200-sample validation set, the accuracy of Qwen3 8B verifying if the paragraph contains visual information is consistent with Qwen3 32B. Therefore, we adopt the separate design in our experiment.
Caption generation. After acquiring the domain-specific contexts, we query MLLMs to generate instance-based captions. When Wikipedia-derive visual information is not available, only format examples are used in the context. This ensures the model consistently generates accurate and instance- specific deseriptions that emphasize visible morphology while avoiding hallucination. The exact prompt template shown below.
Prompt for Caption Generation
You biologist describing organisms based strietly on what is visible in the image. Your goal is to produce a concise caption that highlights diagnostic, image-based traits. Focus primarily anatomical structures (e.g., color, shape, pattern, texture, position). If clearly visible, you may mention substrate, scale cues, or explicit interactions. Use precise biological terminology. Avoid vague generic words.
Examples of good captions: (format examples)
If Wikipedia excerpt is available: Reference excerpt about
sepeciessnameee use only to standardize correct terms that match visible traits; do not copy text; do not add traits not visible in the image: (wiki.excerei
The caption must not exceed moord.limin words.
Include the species name "(species __mee] naturally in the sentence.
21

---

Published as a conference paper at ICLR 2026
Table 8: Taxa coverage and sample coverage across taxonomic ranks.
Taxa coverage
Sample coverage
Rank
Covered taxa Total
Ratio
Covered images / Total
Ratio
Order
1,137/1 184
76.5%
9,256,964/99. /9,34933,31117
97.1%
Family
5.127/7.00 /7,90
64.7%
9,161,272 /9,53,,517
96.1%
Genus 52,725/73,299
44.7%
7,436,080 00,6,0000/,,,,,11111
78.0%
Species 122,243/ 012,243/406,290 406,293
30.0%
(989,956/9,53333
52.3%
Priority order: (1) the most diagnostic visible trait, (2) secondary distinctive trait, (3) contextual detail only if it strengthens identification.
Final instruction: For the following image of (species name), write single, concise sentence describing its visible traits.
B.2 STATISTICS OF CAPTION COLLECTION
Wikipedia coverage. As illustrated above, we apply an LLM-based extractor to only keep Wikipedia- derived descriptions related to visual information. After excluding non-visual descriptions, obtain a total of 131,869 Wikipedia-derived visual descriptions, collected through three complementary cases. First, using available species-level labels from TreeOfLife- 1OM, we collect 109,482 species- level descriptions directly from Wikipedia. Second, for species whose Wikipedia pages lack visual information, we fall back to their corresponding genus pages and map the retrieved genus descriptions back to these species. producing an additional 761 species-level descriptions. Together, these two cases provide 122,243 species-level descriptions in total. Third, for samples that lack species-level label, we use the genus pages directly, adding another 9,626 entries. In total, we obtain 131 8669 visual descriptions, consisting of 122,243 species-level and 9,626 genus-level descriptions. These 131,869 descriptions correspond to 29.5% of the 447,593 unique taxa in TreeOfLife-1OM.
Table 8 shows the Wikipedia description coverage at different taxonomic levels. One taxon counted as covered when at least one species within the taxa has associated visual description through the above scraping process. The species-level Wikipedia-derivee of the total species-level taxa. 44.7% taxonomic genera have Wikipedia visual information associated with at least one species within them, and the number becomes 76.5% for orders.
visual information contributes to 30.0%
Sample-wise 52.3% samples covered with the Wikipedia-derie derived visual information of exactly the corresponding species. When it comes to the family level, 96.1% % samples have at least one species covered Wikipedia within the same family. In such way, even if the species is not exactly covered, there at least another similar species with diagnostic characters described in the context. Thereby, the knowledge is able to be generalized across different species during training. We provide quantitative analysis toward the generalization in Table 4.
Computational time. For caption generation, employ Intern 38B with the vLLM (Kwon et al., 2023), running on 12 NVIDIA H100 GPUs for about 30 hours to process 10 million samples. Note that the caption generation process is designed to be highly scalable. The adopted vLLM framework allows for efficient distributed inference with optimized memory management and parallel sampling. The system can handle larger datasets by increasing the number of GPUs or extending running time. This scalability ensures that generating captions for hundreds of millions of images is feasible, which is valuable for existing biological image repositories.
B.3 BASELINE PROMPTS
In addition to introducing camain-speciffe contexts, we also explore different formats for the instruc- tion. We design two different prompts: Base, which generates generic short captions without any domain hints, and Trait, which encourages more detailed, inage-grounded trait descriptions. The two prompts are listed below.
22

---

Published as a conference paper at ICLR 2026
Baseline Prompt: Base
Describe this image in short.
Baseline Prompt: Trait
You are a biologist describing organisms strictly on the basis of visible characteristics in an image.
Your goal is to produce a concise caption that highlights diagnostic, image-based traits. Focus primarily on anatomical structures (e.g., color, shape, patter,, texture, position). If clearly visible, you may mention substrate, scale cues, or explicit interactions.
Use precise biological terminology. Avoid vague or generic words.
The caption must not exceed {word_1 imi t words.
Include the species name "( "(species.ue name )" naturally in the sentence.
We provide the quantitative comparison between the captions generated with the two prompts in Table 3. The qualitative comparisons are presented in Figure 4 and Figure 9. MLLMs tend produce more detailed descriptions when the prompt explicitly asks the model to focus on traits. Based on the comparison, we use the Trait prompt in other experiments and further incorporate domain -ppciif contexts to ground biological knowledge.
B.4 GENUS-TO-SPECIRS DESCRIPTION MAPPING
Image & Species
Captions
Moraeo flovescens features a slender, elongated yellow flower atop thin green stem, set against backdrop of dry, woody ground.
Moraea flavescens
Moraeo neglecta features bright yellow, iris-like flowers with twisted petals, growing amidst dry, grassy substrate.
Moraea neglecta
Figure 8: Examples of genus-level Wikipedia descriptions used as fallback and the resulting captions.
When species-level Wikipedia visual descriptions are unavailable, we map the corresponding genus- level descriptions to species within that genus. This strategy is biologically grounded: in taxonomy, species belonging to the same genus typically share most of their diagnostic morphological characters and differ only in a small subset of traits (Mayr & Ashlock, 1991). Genus-level descriptions therefore provide a coherent approximation of the shared morphological context for closely related species. These descriptions are not used directly as captions, but serve as contextual information to guide the caption-gas gentrrttnnnrrtionnnn MLLM, which still conditions the input image. As result, even if multiple species receive the same genus-level Wikipedia visual description, the generation process leads to instance-specific captions that reflect visual traits present in the image, shown in Figure 8.
23

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
We generate synthetic captions using InternVL3 38B. The generation is conducted on 12 NVIDIA H100 GPUs for 30 hours with the vLLM framework (Kwon et al., 2023). We apply nucleus sampling (top-p = 0.8) with a temperature model 8 H100 GPUs for 50 epochs using the openclip framework (Cherti et al., 2023). Each GPU processes 4,096 text-image pairs per batch, resulting in global batch size of 768. We use the AdamW optimizer (Kingma & Ba, 2015) with learning rate of 10 weight decay of 0.2, and linear warm-up during the first 500 iterations. Images are resized to 224 x 224 for training and evaluation. As described in $4.2, we adopt dual projecror design with two separate visual projectors: one for taxonomy supervision and the other for caption supervision. All embeddings used for evaluation are extracted from the taxonomy projector.
0.6. With captions and species labels obtained, we train our
C.I HYPERPARAMETERS
Table 9: The adopted hyper-parameter setting in training BIOCAP.
Table 10: The adopted hyper-paramenn setting in ablation study.
Hypereparameter
Value
Hyper-parameter
Value
Architecture
ViT-B/16
Architecture
ViT-B/16
Optimizer
Adam
Optimizer
Adam
Batch size/GPU (organism)
4,096
Batch size/GPU (organism)
4,096
GPUs
8 H100s
GPUs
4 H100s
Epochs
50
Epochs
100
Max learning rate
x 10-1
Max learning rate
1 x 10-1
Warm-up steps
500
Warm-up steps
200
Weight decay
0.2
Weight decay
0.2
Input resolution
224
Input resolution
224
summarize the hyper-parameter configurations in Table and Table 10, corresponding to the main training of BioCAP and the ablation study. The batch size reported in both tables refers to the per-GPU value. Compared with the full training, the ablation uses fewer GPUs and a shorter warm-up schedule, while keeping the overall architecture unchanged.
C.2 TEXT-IMAGE RETRIEVAL
We evaluate zero-shot retrieval on Cornell Bird (Macaulay Library, Cornell Lab of Ornithology, 2025), PlantID (Bruce Homer-Smith and contributors PlantID.net, 2025). We follow the standard text-image retrieval protocol using paired text-image data. Both text-to-image and image-to-tes retrieval considered by embedding the two modalities into joint representation space and ranking candidates according to cosine similarity. Performance is measured by Recall@ 10, which reflects how often the correct match is retrieved within the top results.
C.3 GRAD-CAM VISUALIZATION
Implementation. We adopt Grad-CAM (Zhou et al., 2016) for visualizing CLIP attention. Grad- CAM highlights image regions most relevant to a target output by weighting feature maps with their corresponding gradients. For CLIP model, given image I and text prompt, their embeddings are obtained via the image encoder fing and text encoder fexti
Sung (T),
(prompt)
The cosine similarity logit
V
T
We backpropagate this logit as the target signal, and apply Grad-CAM the final transformer block of the image encoder to obtain heatmaps. Since the CLIP image encoder is ViT-based, we reshape the patch-token activations from the target layer into 2D spatial grid before upsampling.
24

---

Published as a conference paper at ICLR 2026
Table 11: Agreement between GPT-4o, Gemini 2.5 Pro, and human annotators for behavior labels.
Annotator
Annotator B
Agreement
GPT-4o Gemini 2.5 Pro
Human
95.1
Human
92.9
GPT-4o
Gemini 2.5 Pro
94.0
High-frequency concept collection. For each target species, we first aggregate all captions and compute the frequency of words after removing common stopwords (e.g., and). Then manually select biologically meaningful concepts starting from the most frequent words, such as body structures (e.g., antennae, petals, tails). The high-frequency concepts CAM. This procedure ensures that the concepts used in visualization correspond to the supervision signal of the synthetic captions. Based the visualization examples, explicitly show that aligning captions and images guides the model to focus on the diagnostic characters. Thereby, BIOCAP demonstrates better species classification performance.
then used the text prompt for Grad-
C.4 BEHAVIOR SEMANTIC ANNOTATION
In addition to static biological concepts related to organs body parts of the object, we also intend to investigate the model's understanding of behavioral semantics. Given that NABirds (Van Horn et al., 2015) does not provide behavior annotations in the original dataset, et al., 2024) to automatically assign one of three mutually exclusive behavior categories: fly, perch, stand. Here, do not rely manual labeling, which be subjective and inconsistent across annotators. These three categories cover the majority of common bird poses. The model is prompted with the following instruction:
use GPT-4o (Hurst
Prompt for Behavior Annotation
You are an ornithologist tasked with identifying bird behaviors from images. Looking at this bird image, classify the bird's behavior into exactly ONE of these three categories:
fly > Wings are spread/extended, bird appears to be in flight or airborne perch - Bird's feet are gripping thin branches, reeds, wires, similar thin supports stand + Bird's feet are on ground, soil, rocks, thick surfaces, or flat platforms Output only the behavior label: fly, perch, or stand.
To further verify the reliability these labels, we collect annotations from Gemini 2.5 pro and from human annotators. The agreement rates summarized in Table 11. Across all comparisons, the agreement exceeds 92%, indicating that behavior classification a cow-ambiguity task. The two MLLMs produce labels highly consistent with human judgment, supporting the robustness of the automatic annotation pipeline.
25

---

Published as a conference paper at ICLR 2026
D BENCHMARK DETAILS
D.1 COLLECTION OF RETRIEVAL BENCHMARKS
Table 12: Benchmarks collected for image-text retrieval evaluation.
Benchmark Description
Species
Image- Text Pairs
Cornell Bird Sourced from the Macaulay Library of the Cornell Lab of Ornithol-
700
7,000
ogy, containing image-text pairs of North American bird species.
PlantID
Collected from PlantID, providing paired images and textual de- scriptions wide range plants.
794
2,251
We collect two retrieval benchmarks to evaluate our model under diverse biological domains: Cornell Bird (Macaulay Library, Cornell Lab of Ornithology, 2025) and PlantID (Bruce Homer-Smith and contributors to PlantID.net, 2025). These datasets cover bird and plant species, each providing paired images and textual descriptions for fine- ene-grained retrieval tasks. Table summarizes the key characteristics of these benchmarks.
D.2 INFORMATION OF OTHER BENCHMARKS
Table 13: Datasets used for zero-shot classífication evaluation. Top-1 accuracy is reported for all the listed benchmarks.
Name
Examples
Classes
Labels
NABird
(Van Horn et al., 2015)
48,000
400
Common
Animals
Plankton
(Heidi Sosik, 2015)
4,080
Mixed
Insects
(Serret al., 2019)
4,680
Scientific
Insects 2
(Wu et al., 2019)
4,080
102
Mixed
Plants & Fungi
PlantNet
(Garcin et al., 2021)
1,000
25
Scientific
Fungi
(Picek et al., 2022)
1,000
25
Scientific
PlantVillage
(G. & J., 2019)
1,520
38
Common
Medicinal Leaf
(S & J, 2020)
1,040
26
Scientific
PlantDoc
(Singh al., 2020)
1,080
27
Common
Desert-lion
(Desert- Lion- Conseoonrrvvtinn, 2024)
352
32
Taxonomic
CameraTrap
ENA24
(Yousif et al., 2019)
1120
20
Taxonomic
Island
Island-Conservation
310
17
Taxonomic
Ohio-small-animan
(Balasubramaniam, 2024)
168
39
Taxonomic
Orinoquia
(Vélez al., 2022)
336
28
Taxonomic
Rare Species
(Stevens al., 2023a)
12,000
400
Taxonomic
We summarize the datasets used for zero-shot classification evaluation in Table 13. These benchmarks cover diverse biological domains, including animals, plants, fungi, and camera-trap datasets, and all tasks are evaluated with Top-1 accuracy. We additionally evaluate on INQUIRE Rerankk (Vendrow et al., 2024), benchmark where the goal is to re-rank 100 candidate images for each of 200 text queries (20K images in total), ensuring relevant images appear higher in the order.
D.3 DUPLICATE AND LEAKAGE CONTROL.
To ensure rigorous evaluation and eliminate any potential information leakage, we conduct duplicate control for two retrieval benchmarks used in this study. Since the two collected retrieval datasets (PlantID (Bruce Homer-Smith and contributors to PlantDD.ne net, 2025) and Cornell Bird (Macaulay Library, Cornell Lab of Ornithology, 2025)) are uploaded individually by users rather than systemati- cally aggregated, their provenance metadata is noisy and inconsistent. We therefore apply perceptual hashing (Facebook, Inc, 2019) with distance threshold of 10 to detect visually similar images that cannot be identified through metadata or MD5. Images flagged as near-duplicates are treated conservatively avoid leakage from any retrieval dataset into the training corpus. Using this pipeline,
26

---

Published as a conference paper at ICLR 2026
find that 4.1% of PlantID images and less than 0.1% of Cornell Bird images appear in the training set before filtering. All overlapping images are removed. This deduplication ensures that retrieval benchmarks are cleanly disjoint from the training data.
D.4 LICENSE INFORMATION
For retrieval evaluation, we additionally used paired text-image data from PlantID (Bruce Homer- Smith and contributors to PlantID.net, 2025) and the Comeell Bird Macaulay Library (Macaulay Library, Cornell Lab of Ornithology, 2025).
PlantID is developed and maintained by Bruce Homer-Smith with contributions from numerous experts and organizations. The website content, including most textual and photographic materials, is released under the Creative Commons CC BY-NC 3.0 license, which allows reuse for non-commercial purposes with proper attribution. We confirm that our usage strictly followed these terms.
The Cornell Bird Macaulay Library is maintained by the Cornell Lab of Ornithology. Use of its media assets in scientific publications requires attribution following their official guidelines. We obtained explicit approval for our use and acknowledge reccipt of media from the Cornell Lab of Ornithology Macaulay Library.
We sincerely thank PlantID, its contributors, and the Comell Lab Ornithology for making these invaluable resources available to the community, which enables our retrieval evaluation.
27

---

Published as a conference paper at ICLR 2026
E MORE QUALITATIVE AND QUANTITATIVE EVALUATIONS
E.
SYNTHETIC CAPTION COMPARISONS.
Image Species
Trait+ Example+ Wiki (Ours)
Trait
Base
Lesser Horseshoe wetch features small, bright yellow flowers with compound, green leaves and reddish stems, growing low to the ground.
image shows a plant small yellow flowers and dellcate, feathery green leaves, typicai of Lesser Horseshoe- vetch, growing among rocks.
Small plant delicate yellow flowers and green leaves growing in rocky soil.
Lesser
Horseshoe- vetch
Cloesio digna displays iridescent blue wings with metallic sheen ond distinct brownish fringes, set against a plain white bockground.
dark, textured wings with bluish tint and distinctive body, matching its scientific name Cloesio digna.
dork-colared
intricate potterns wings is disployed agoinst white background.
Cloesia digna
Habronattus fallax exhibits mottled brown with distinct white stripes on the cepholothorar, positioned on textured substrate.
image a spider with speckled pattern, likely Hobronottus failax, featuring distinct morkings and slender legs on textured surface.
small spider with striped and body sits textured surface neor strow.
Hobronattus fallax
Gorden Orbweovers display distinctive zigzng stabilimentum in their web, with central white cross pottern and rodial symmetry, set against backdrop of green folioge.
Lorvoe of Larvae weaver spider. The intricate web set in green, leafy environment, likely garden or forest.
A spider meticulously creates intricate, decorotive web against backdrop of green leaves.
Garden Orbweaver
small, reddish-orange possibly Paranapiacaba significato, is perched on a white, five-petoled flower with dork spots against green faliage.
image shows white flower with green center, jeaturing smail black spots ond an ont on its petais, set agoinst green foliage.
A white flower smoll insect on it, set against green faliage.
Paranapiacaba significata
The Pink Leof Moth disploys vibrant pink and yellow wings with subtle gradation, delicote furry texture, prominent ontennae.
The moth has vibrant pink ond yellow wings, delicate oppeoronce,
A colorful moth with pink and yellow wings displayed ogainst white background.
Pink Leaf Moth
soft, pastel color grodient, typical Pink Leaf Moth species.
Rattlesnake mannagrass displays on open panicle with drooping spikelets, nestled omong brood, green grossy habitat. sel. photo Glycerio canadensis.
The image shows dense, gross with small, light-colored seed heads, typicol of rattlesnoke mannagrass, growing natural setting.
Close-up of grass with small seed heads in a natural, outdoor setting.
Rattlesnake Mannagrass
The Atlantic threetooth features brown, spiraled shell with fine growth
The image shows o small, colled, three- three-tootheds toothed snail with brown, textured shell resting on soft. fibrous surfoce.
A small, coiled snoil lorva is resting on a soft, fibrous surface.
Atlantic Threetooth
pale, open umbilicus, resting soft, fiórous surface.
Figure 9: More qualitative comparison on the captions generated by different prompts. Our full pipeline, which emphasizes focusing traits and introduces domain-specific contexts, demonstrates accurate and restance-speciffe captions.
Figure shows more captions generated by different prompts, including Base, Trait, and our full pipeline (Trait+ Wikí). The Base captions are often not detailed and not based on biological knowledge. Even the model is explicitly prompted to focus on traits given the species name, MLLMs can also hallucinate and lose attention to the target object. As shown in the "Paranapiacaba significata" example, the Trait caption falsely describes the flower instead of the insect. Our full pipeline, in contrast, accurately describes the insect. When the caption contains too much noise while losing focus on the target object, the provided supervision can misguide the multimodal alignment and harm the model performance.
28

---

Published as a conference paper at ICLR 2026
E.2
GRAD-CAM RESULTS.
Input
CLIP
BioCLIP
BioCAP
BioCLIP
BioCAP
CLIP
BioCLIP
BicCAP
'Legs"
"Abdomen"
"Weak shelled Shore Crab"
"Legs"
"Abdomen" *
""Wasp Spider"
"Wing"
"Crown"
"Regulus satrapa"
"Gill"
"Stem"
"Panaeolus foenisecii"
Figure 10: Grad-CAM visualization CLIP, BIOCLIP, and BioCAP, given species names and biological concepts frequently mentioned in their captions. BIoCAP accurately highlights these concepts and associate them with classification.
Input
CLIP
BioCLIP
BioCAP
CLIP
BioCLIP
BioCAP
CLIP
BioCLIP
BicCAP
"Banner fin"
"Snout"
"Pelvic fin"
"Bud"
"Petal"
"Anther"
"Nostrill"
"Bye"
"Toe"
"Antenna"
"Eyespots"
"Head"
Finer-CAM visualization of CLIP, BIOCLIP, and BIoCAP, given biological concepts frequently mentioned in their captions. BIOCAP accurately localizes these fine-grained concepts.
Figure 11:
Alignment with biological traits. Various biological concepts are mentioned the synthetic captions to describe potential diagnostic characters of the species. We present more visualizations regarding these concepts using Grad-CAM and Finer-CAM (Zhang al., 2025) in Figure 10 and Figure 11 as supplement to Figure 5. Compared with CLIP and BIOCLIP, BIOCAP demonstrates significantly better localization these concepts. Moreover, when Grad-CAM applied to the species name, the parts highlighted by BIOCAP align with these concepts. The results indicate that the synthetic captions guide the model to focus on potential diagnostic traits, while suppressing the spurious correlation. Thereby, BIOCAP demonstrates better species classification performance, which also aligns with our analyses $3 and SA.
29

---

Published as a conference paper at ICLR 2026
Input
CLIP
BioCLIP
BioCAP
Input
CLIP
BioCLIP
BioCAP
"Fly"
"Fly"
"Fly"
"Fly"
"Perch"
"Perch"
"Perch"
"Perch"
"Feed"
"Feed"
"Feed"
"Feed"
Figure 12: Grad-CAM visualization of CLIP, BIoCLIP, and BIoCAP, given behaviors. BIOCAP correctly highlights the body parts related to the behaviors.
Alignment with behavior semantics. In addition to the organism parts that are potentially discrim- inative traits of the species, the generated captions also contain descriptions regarding behaviors. We provide more Grad-CAM visualizations toward behavior in Figure 12 as a supplement to the previous demonstration in Figure 6. For "fly." "perch," and "feed," BIOCAP accurately highlights wings, legs/feet, and mouth/beak/food in the images, respectively. Based on the comparison with CLIP and B1OCLIP, the understanding of these behavior concepts is derived from the newly curated synthetic captions. It also validates that our synthetic captions have successfully captured a variety of biological semantics. The rich semantic understanding of BIOCAP potentially enables broader applications in various biology-related tasks.
Table 14: Localization performance of different models on CUB, evaluated by the energy-based pointing game.
Model
Localization score
CLIP
0.36
BIOCLIP
0.43
BIOCAP
0.47
Grad-CAM quantitative results. We provide a quantitative evaluation of localization quality on CUB (Wah et al.. 2011), using ground-trui truth bounding boxes offered in the dataset. For each model (CLIP, BIOCLIP, and BIoCAP), we compute Grad-CAM maps with species names as text prompts and measure localization accuracy using the energy-based pointing game (Wang al., 2020). This
30

---

Published as a conference paper at ICLR 2026
Table 15: Class-level vs. order-level format examples. We compare using class and order level examples guide caption generation.
Table 16: Stability across examples genera- tion rounds. We evaluate BIOCAP using two independently generated exemplar sets.
Retrieval
Retrieval
Taxonomic Level
CLSS
INQ
Generation
CLS
INQ
12T
T21
Round
12T
T21
Class Order
33.8
54.7
54.3
34.8
1
33.8
54.7
54.3
34.8
33.2
55.1
53.9
33.6
2
33.6
54.9
54.5
34.7
Table 17: Performance on underrepresented species groups. We report classification accuracy across conditions defined by Wikipedia coverage. training image availability, and the Rare Species set. Bold and underlined entries indicate the best and second best results, respectively.
Classification
Model
Few-images
Many-imager
Rare
non-covered
covered
non-covered
covered
CLIP
16.2
32.3
21.1
23.4
25.7
BIOCLIP
54.0
30.4
35.4
42.1
37.6
BIOCAP
61.0
45.0
43.7
48.9
46.4
metric quantifies the fraction of activation energy that falls within the annotated bounding box. The results are reported in Table 14. BIoCAP achieves higher localization scores than both CLIP and BioCLIP, indicating that the trait-focused captions help guide the model toward biologically meaningful regions.
E.3 FORMAT EXAMPLE DESIGN
In $4.2, we analyze the effect of format examples count and show that using three curated examples per taxonomic class provides stable and sufficient guidance for caption generation. Here, we provide detailed results for two more design choices: the taxonomic level used to generate format examples, and the stability of the pipeline across independent generation rounds.
Taxonomic level of format examples. We compare class-level and order-level format examples for guiding caption generation, with results shown Table 15. The two configurations yield comparable performance across all metrics, while order-level format examples introduce slightly higher variability in few cases. These findings indicate that class-level format examples strike an effective balance between trait specificity and taxonomic coverage without over-constraining the captioner.
Stability across generation rounds. To assess robustness against sampling variability, we regenerate full second set of format examples using the same Gemini-based pipeline and retrain BIOCAP. As reported in Table 16, while the regenerated format examples are different, the resulting models achieve nearly identical performance. This shows that the format examples generation process remains stable across model runs.
E.4 PERFORMANCE UNDERREPRESENTED SPECIES
It has been shown in Table that BIOCAP improves the species not covered by Wikipedia-derived visual descriptions. We further partition species the sets using two criteria: whether the species is sxipedia-coverre or non-covered, and the number of available training images. We rank species by sample count and define the bottom as "few-image" species, with the remainder treated as "many-image." Combining these two factors yields four groups: few-image non-covered, few-iidae + covered, many-image + non-covered, and many-image + covered. We report the results these groups in Table 17, together with Rare Species, of which the species are not seen during training. Across all groups, BIOCAP consistently outperforms CLIP and BIOCLIP. The improvements are most pronounced for the two most challenging conditions (few-image non-covered and few-image covered), indicating that caption-guide -uiided training is beneficial even when both visual data and external descriptions are limited. B1OCAP also shows clear gains the Rare Species set, suggesting that
31

---

Published as a conference paper at ICLR 2026
Table 18: Few-shot species classification top-1 aceuracy across 10 tasks. Bold and underlined
values indicate the best and second best results. All models use the ViT-B/16 visual encoder.
Animals
Plants & Fungi
Camem Trip
PlantVillage
Rare Speciee
Med. Leaf
NABirds
Plankton
Insects
Inseets
PlantNet
Fungl
Model
Moan
One-Shor Classification
CLIP (VII-B/16)
24.5+10
21.8+11
19.8+0.5
11.3+0.
34.0+2.1
38.9+s.7
15.5+22
46.0=22
67.3-22
26.5ee..
30.6
SigLIP
30.3g0.
28.2.007
27.5409
17.141s
35.1
57.1
21.9:1a
58.9.24
79.6-1.
32.7.0.3
38.8
Sipervised-IN21K
45.4
25.6.00
23.9
20.8
34.31as
58.2144
28.613.
50.5
81.3
36.1
41.4
DINOv3
47.820.7
36.4211
10.1404
19.3206
48.0ga5
60.72a.
23.821.
66.4-.11
94.3
41.720.
44.4
BLLII
61.91os
16.
67.1114
20.
31.21za
69.7134
47.3
47
55.8144
83.5
31.91.
48.9
BIOCLIP
67-4+12
2
17.1
20.4
35.0gas
67.723.
44.0+2.
50.5
83.7
44.9
50.0
BIOCAP
53.9210
31.2,07
$3.
23.5
33.143.0
68.9
41.2+14
66.9
86.9 1.8
45 45.4=0.
50.5
Five Shot Classification
CLIP (VIT-B/16)
48.22o.3
36.2207
36.710.
22.0ga.
61.7
59.6
24. 24.1
60.9
16.1
43.3
47.8
SigLIP
51.2a0.4
17.9
48.0+0.
30.2+0.7
52.2+2.
76.621.
36.2+2.
78.520.7
92.1-17
50.840.4
56.7
SupervisedINNIK
66.7e0.
51.0+0.4
47.740.6
35.9+1.2
57.6
80.7+1.
51.5+1.
84.5+1a
981.-112
57.7=0.2
62.9
DINOv3
75.5
61.0210
28.7
37.1
69.3
86.3216
50.322.
85.6.1.7
09.2.0.5
67.5.0.
BioI
78.5+002
41.640.6
77.0+0.s
31.2+0.
47.9+2.
86.0
65.2+0.8
75.1=0.8
16..2-0.7
51.3+0.2
65.6
B1OCLIP
78.210.
49.2.11
78.0
33.9
54.3+22
85.7
61.6:1.
81.7a1.1
96.720.
65.7g0.4
68.5
BIOCAP
77.0
51.1
77.0
38. +0.4
48.442.4
85.421.
63.2
86.2
96.9 .o.
67.3 6T.3
69.1
Table 19: Biological visual tasks beyond species classification. Bold and underlined entries indicate the best and second best accuracies.
Animals
Plants
FishNet
Herh. 19
Plant Doc
NoW?
AwA2
Model
Mean
CLIP (VIT-B/16)
25.3
79.7
66.0
15.6
17.5g3.
40.8
SigLIP
31.94o.1
83.2
67.310.6
18.6
28.2
45.8
Supervised-IN2IK
29.440.1
75.8
52.7+1.6
14.9
25.121.1
39.6
DINOv3
37.9+0.1
85.7+0.0
48.0+2.
21.2
40.3+1.2
48.6
BioTrove-CLIP
22.110.0
82.5.o.1
45.710.7
20.410.
37.721.2
41.7
BIOCLIP
30.1+0.2
82.7+0.1
65.9+0.3
26.8+0.4
39.5
49.0
BIOCAP
29.540.3
84.5
65.641.1
28.1
37.7
49.1
the morphological priors encoded in synthetic captions support strong generalization even to species unseen during training.
E.5 PERFORMANCE ON FEW-SHOT CLASSIFICATION AND BIOLOGICAL VISUAL TASKS
Beyond zero-shot classification and retrieval, we further evaluate BIOCAP on few-shot species classification and additional biological visual tasks (Khan al., 2023; Van Horn et al., 2021; Xian et al., 2018; Tan & Liu, 2019; Singh et al., 2020) in Table 18 and Table 19, respectively. We observe an overall improvement of BIOCAP over BIOCLIP in few-shot classification and similar result in other biological visual tasks. We believe the smaller gap compared with zero-shot and multimodal retrieval is expected, given the nature of the added supervision. Compared with species names alone, descriptive captions introduce more "disturbances" that enrich the semantics carried by the embedding but also distract embeddings from the species prototypes. BIOCAP yields more interpretable intra- class structure tied to biological semantics. However, this supervision does not explicitly enforce more separation between visual embeddings of different species. Therefore, when captions provide better semantic organization and multimodal alignment, they do not contribute to better few-shot classification performances. This is supported by results Table 20. We compare models trained with varying caption qualities. We observe that while higher-quality captions significantly boost zero-shot and retrieval metrics shown Table 2, few-shot performance shows minimal variance. This weakly correlated with caption quality. On the other hand, the
indicates that few-shot capability is
32

---

Published as a conference paper at ICLR 2026
Table 20: Effects of different captions few-shot species classification top-1 aceuracy across 10 tasks. None uses caption; Wiki Page uses Wikipedia visual text; Synthetic uses LLLL--gnnerated captions via simple (Base) or trait-focused (Trait) prompts. Domain-specific contexts include format examples (Example) and Wikipedia-derived info (Wiki). The results demonstrate that varying caption generation strategies yields marginal performance differences the few-shot setting. Bold and underlined values indicate the best and second best results.
Fungi
Camera Trup
Plant Village
Rane Species
Med. Leaf
NABidds
Planktton
PlantNor
Insects
Insects
Pungl
Strategy
Meun
Caption
Prompt
Context
Ohr-Shot Claveifiatiin
45.1
29.9
45.6g1
18.7
33.3
64.1
35.8
80.8
78.0g
368
Wiki Page
40.811.
31.9
42.4
19.5.1
35.3
60.7a39
35.0go.s
58.9
82 82.1
38.1407
44.5
Synthetic
38.619
40.3
20.0
84.4
62.2
81.9
85.91as
44.0
Synthetic
Trait
11.5eaa
29.9a1.
22.1a12
19.5
343332
63.1
35.3
60.9
81.1ea1
37.2 37.2
41.5
Synihetie
Trait
Example
41.92a-
31.0a1a
43.5a14
10.3
34.5259
64.
61.5g8
79.6241
36.8194
44.8
Synihetic
Examplc+Wiki
13. 31.8t1s 18.5
19.3e1
33.8.4
63.5
35.0
61.
78.7aar
37.8
Caption
Prompt
Context
Sho-Shot Clacsification
07.0 10.6
65.0go4
34.8
52.4
18.3g1.
50.6
57.1.a 95.8gaa
Wiki Page
69.0 02.0 49.210.7
69.9
334424
63.0
82.0
54.6
70.6
93.3ga
57 37.9
64.3
Synthetic
Base
66 66.2 15.0
67.3e07
315
55.3
78.9
54.8
82.5-1
93.6a18
58.2ea.
64.6
Synthetic
Trait
07.0gaa
49.7an5
67.7
34.5.a.
62.7a3
80.230.
53.9
83.2-a
94.6eae
68.2294
64.2
Synthetic
Trait
Esample
67.0ea.
51.640.4
68 68.1
33.9203
53.3e11
81.0sLo
51.
81.8-1
93.2caa
58.4
61.3
Synihetic
Examplc-Wiki
6744aa 51.211.0 68.3.n: 31.3.0
53.0g.
8122,
58.
38.9
82.721
93.9g
58.8
generalization across other biological visual tasks arises from larger training scale, stated in the BIOCLIP 2 paper (Gu et al., 2025). Here, we use the same amount of data as B1OCLIP. Therefore, it is also understandable that such "emergent properties" occur.
33

---

Published as a conference paper at ICLR 2026
F DISCUSSION WITH RECENT WORK
This work proposes a new biological multimodal foundation model based on the combined supervision of taxonomic names and descriptive captions. This direction is complementary recent advances that enrich biological models via scale or additional modalities. Gu et al. (2025) curate large- scale TreeOfLife-2000 dataset, with which they demonstrate that scaling hierarchical contrastive learning enables emergent properties. Gharaee et al. (2024) introduce DNA barcoding information as additional supervision. This work does not aim to replace scaling or other supervision sources, but focuses on orthogonal direction of how synthetic captions help bridge biological images with multimodal foundation models. We demonstrate in this paper that descriptive captions, when grounded in biological knowledge, significantly enhance the model''s understanding of rich semantics and its species classification performance. The value descriptive captions is, however, largely underexplored so far (Vendrow et al., 2024). It is also notable that the designed caption generation pipeline can be feasibly scaled up for larger datasets and integrated with more supervision dimensions through techniques like TaxaBind (Sastry al., 2025).
parallel line of work strengthens CLIP-style training by improving or expanding captions. FG- CLIP (Xie et al., 2025) constructs region-specific annotations, targeting fine-grained alignment capabilities. LaCLIP (Fan et al., 2023) uses LLMs to generate multiple versions of each caption according to format examples. VeCLIP (Lai et al., 2024) rewrites noisy web text into visual-enriched captions and mixes them with alt-text for training. CapsFusion (Yu et al., 2024) obtains descriptive captions from captioning models and refines them with large language models to achieve more semantically aligned supervision. These advances have pushed the frontier of general- multimodal foundation models and also enabled the pipeline our approach. However, previous efforts mainly focus on general images, with limited exploration of scientific domains like organismal biology. When the pipeline is naively applied to biology, we empirically observe more hallucinations about biological details and oversight of discriminative traits. As discussed in $3, noisy captions can potentially harm the multimodal alignment. To this end, we explicitly inject the domain knowledge into the MILM context. We demonstrate that the domain-specifie contexts reduce hallucination and produce captions with more accurate biological details. Our work complements the gap for biological foundation models explore the value of descriptive captions.
34

---

Published as a conference paper at ICLR 2026
G BROADER APPLICABILITY TO SCIENTIFIC DOMAINS
growing line of work aligns scientific images with domain-specific labels or short text surrogates (e.g., object IDs, catalog entries, class names) to train multimodal representations (Parker al., 2024; Vivanco Cepeda et al., 2023; Harnik et al., 2025). While effective for recognition and retrieval, this supervision often semantically thin: many scientific labels are rare in everyday language. encode limited visual semantics by themselves, and are weak proxies for the causal or functional attributes that scientists care about. As a result, the learned alignment can overfit to dataset-specific correlates (instrument signatures, acquisition context, background cues) and provide limited support for higher-level tasks that require grounded interpretation.
BioCAP targets such a structural bottleneck in biodiversity. While images with taxonomic labels are abundant, these labels usually contain limited semantics and do not indicate traits of the species. Instance-speciftc, altribute-rich text is scarce and expensive to obtain. Similar gaps arise in many fields where visual observations are routinely collected, but natural- -anguule reports are incomplete, non-standardized, or locked behind domain conventions. A naive approach is to directly generate captions at scale using general-purpose MILMs; however, this can suffer from hallucination, which is shown in Figure 1, making synthetic supervision unreliable for scientific training.
The key idea of BioCAP is to convert external scientific structure into grounded language supervision. Coneretely, we inject domain knowledge (Wikipedia-deriveed visual information) and curated format exemplars to constrain generation. This transforms open-ended captioning into a constrained, schema- driven description problem. The resulting captions not intended be free-form narratives; rather, they serve as reliable, attribute- attribute-oriented supervision that better matches how scientists record observations and how downstream models should reason.
This recipe generalizes beyond biodiversity to scientific domains that share three properties: (I) large-scale imagery (or image-like like measurements), (2) weak native text alignment (IDs or labels that under-specify visual content), and (3) available external structure that can be encoded as constraints. Examples include: medical imaging (findings structured by radiology ontologies and reporting guidelines), materials science and microscopy (morphology, phases, defects, acquisition parameters), chemistry and molecular imaging (functional groups, conformations, assay condi- tions), remote sensing (land-cover attributes, phenology, sensor metadata, geospatial context), and astronomy (object properties, observation bands, instrument settings, survey metadata). In each case, schema-driven captioning can synthesize attribute-level text that is both grounded in the image and standardized across sources, enabling multimodal foundation models to learn representations that transfer tasks requiring explanation, attribute retrieval, and hypothesis generation.
More broadly, BioCAP suggests a general pathway for scientific multimodal alignment: when paired image-text data searce unreliable, use domain constraints manufacture trustworthy language supervision can learn representations that are not only predictive, but also interpretable.
35

---

Published as a conference paper at ICLR 2026
H HUMAN EVALUATION
H.1 EVALUATION INSTRUCTION
Evaluation Task and Criteria
Objective
The goal this study to validate the quality and reliability automatically generated captions. Your task is to independently evaluate them along predefined criteria. Specifically, we have sampled a set of examples, each paired with multiple candidate captions. For each example, you will review the captions and select the one that performs best under each evaluation criterion.
Criterion-Level Assessment
For each example, you will be provided with an image and three candidate captions. You are asked to evaluate the captions on four criteria:
Groundedness: Does the caption align with features actually visible the image?
Trait Specificity: Does the caption highlight the most distinctive traits (e.g., color, patterns, morphology)?
Completeness: Does the caption cover the 2-3 most salient visible aspects?
Clarity / Scientific Tone: Is the caption written clearly, using precise and objective language?
For each criterion, select the caption that best satisfies the requirement.
Notes
Each criterion is independent: you may choose different captions across criteria.
Please read the criteria carefully and base your judgments only on the provided image and captions.
Please only select the best caption per criterion.
Participant Information and Consent
Thank you for agreeing to participate in this human evaluation study. Before you begin, please carefully read the following information:
Voluntary Participation
Your participation in this study is entirely voluntary. You may stop at any time without penalty.
Purpose
The goal of this study is to evaluate the quality of automatically generated captions for biological images, using expert judgments across specific evaluation criteria.
Data Collected
Only your evaluation responses (e.g., which caption you select for each criterion) will be recorded.
No Personal Information
We will not collect any personally identifiable information (PII), such your name, email address, or IP address. Your responses will remain anonymous.
Confidentiality
All data will be stored securely and used solely for research purposes. Results may be reported in aggregate form but will never be linked to individual evaluators.
Risks and Benefits
There anticipated risks associated with this study. While you may not receive di- rect personal benefit, your participation will contribute to the advancement of methods for evaluating scientific image descriptions.
36

---

Published as a conference paper at ICLR 2026
Consent Statement
By clicking "Yes," you confirm that you have read the information above, agree to participate in this evaluation, and acknowledge that no personal data will be collected.
H.2 EVALUATION STATISTICS
Table 21: Agreement between different human evaluators. Overall values are computed across all data (micro average).
Metric
Attributes
Raw Agreement
Gwet's 's ACI
Groundedness
66.5
0.509
Specificity
64.7
0.482
Completeness
78.8
0.724
Clarity
80.0
0.750
Overall
72.5
0.615
We conduct human evaluation to assess the quality of captions generated by three different strategies: Base, Trait, and Trait+ Example+ Wiki (Ours). A total of 16 participants are involved in the study, including one ecologist and 15 computer science students. The evaluation covers 20 sets of images, each set containing 10 images, resulting in 200 images in total. For each image, we provide three candidate captions (one from each strategy) and ask evaluators to select the best one according to four evaluation attributes: Groundedness, Specificity, Completeness, and Clariry. Each image is independently evaluated by two different participants, ensuring two evaluation results.
We first report the win rate of each caption generation method, defined as the percentage of times method's caption is selected the best among the three candidates. As shown in Table our full method (Trait+ Example+ Wiki) consistently outperforms both baselines across all four evaluation attributes, supporting the effectiveness of domain-specifie contexts in improving the caption quality.
In addition, we assess the agreement between the two human evaluators. We report both the raw agreement (the proportion of samples where two evaluators select the same caption) and Gwet's ACI (Gwet, 2008), chance-corrected agreement coefficient designed to provide stable estimates under imbalanced category distributions. Agreement is reported separately for each attribute, and we also report micro-average across all data. As summarized in Table 21, both raw agreement (72.5% overall) and Gwet's AC1 (0.615 overall) indicate substantial inter-evaluator agreement, corresponding to the "substantial" level in standard interpretation scales (Landis & Koch, 1977).
I DISCLOSURE OF LLM USAGE
Portions of this manuscript polished for clarity and readability using LLM. The LLM was not used to generate research ideas, design experiments, analyze data, or draw conclusions. All scientific content, methods, and results the authors' original work.
37


Reviews:
[{"review_id": "YZRg6yYVRe", "reviewer": "Reviewer_XfBu", "summary": "The paper proposes a BIOCLIP-style model that, during training, aligns images with taxonomic labels and instance-specific synthetic captions generated by an MLLM. The captions are constrained by Wikipedia-derived visual descriptions and taxon-tailored format examples to reduce hallucinations. The evaluation show strong performance for species classification and text-image retrieval by training on TreeOfLife-10: +8.8% average top-1 over BIOCLIP across 10 zero-shot species classification sets, and +21.3% average on natural-language tasks/retrieval. Ablations show that human eval favors the proposed model's captions over others. \nThe qualitative analysis with Grad-CAM images and t-SNE embeddings further highlight the improved embeddings with BioCAP.", "questions": "can caption-aligned features also help zero-shot retrieval/classification?\nHow do results change with different captioners?\nWhat is the dedup protocol used? How do you ensure held-out test set?\nAre there any cases of observed hallucination in the final captions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposes interesting tweaks like using images and captions as complementary views of a species’ latent trait vector and training with contrastive learning to emphasize diagnostic features. The dual projector cleanly separates heterogeneous supervision as compared to a single projector. \n\nThe paper will be of reasonable interest for the broader community interested in training VLMs for life sciences.", "weaknesses": "Captions are biased toward the chosen InternVL3-38B; and there is no cross-MLLM comparison. \nBehavior labels for analysis are auto-assigned by GPT-4o that may cause label drift. \nLarge deltas are shown in experimental results, but statistical significance/confidence intervals aren’t reported for benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wwh29aVMsF", "reviewer": "Reviewer_KyEW", "summary": "This work present a biological multimodal foundation model that integrates synthetic, descriptive captions as additional supervision beyond taxonomic labels to improve species understanding. The authors propose to generate synthetic instance-level captions using multimodal LLMs guided by Wikipedia-derived visual information and taxon-specific format exam, then train a model with dual text views(species name + caption, )and dual visual projectors to decouple supervision. The results shown more than 20% improvements across 10 classification tasks.", "questions": "- Is there any caption noise? how did you filter those?\n- What effort is needed to extend the framework to video data?\n- The method is specifically designed for biology domain. How is the generalization capability to other scientific domains?\n- Is there any quantitative result for Grad-CAM?\n- Who are the human annotator? are they biology expert?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Multimodal alignment in biology is an under-explored and important task.\n- The paper propose an interesting idea that images and captions are treated as complementary projections of a species’ latent morphospace, so aligning them helps capture diagnostic traits while suppressing noise.\n- The paper introduced a dual-projector architecture elegantly separates taxonomy vs. caption supervision, and use context-guided caption generation (Wikipedia + format examples) effectively mitigates LLM hallucination.\n- The evaluation is comprehensive, including multiple benchmarks, both quantitative and qualitative analysis.", "weaknesses": "- The caption reliance on Wikipedia-derived descriptors could reinforce taxonomic bias and exclude rare or poorly documented species. How does the framework handle species without any Wikipedia entry or minimal trait descriptions?\n- The repeated LLM re-generation could produce inconsistent style or attribute focus across species, causing the semantic drift.\n- Potential scalable issue since the caption generation and derail-view training are computational costly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mn9g8FL7gV", "reviewer": "Reviewer_uX5T", "summary": "The authors create synthetic instance-level captions of images from the TreeOfLife-10M dataset and use them to train a version of BioCLIP with a caption encoder. The captions are morphological trait-based descriptions of the animals that correspond to subregions in the images. The authors articulate their synthetic caption generation strategy, report benchmarks on classification and retrieval, and perform a detailed ablation study. The results indicate that the trait captions improve classification and help constrain model attention to animal parts that help with classification.", "questions": "- What is the taxonomic breakdown of the resulting descriptions? At line 224 you indicate the descriptions cover ~32% of the species in TreeOfLife-10M. Are there any biases in how the generated captions or is it uniformly distributed across organism groups?\n- What are the limitations of the caption generation strategy? Is the 32% coverage a fundamental limit of what is available on Wikipedia? \n- Are there any potential issues when mapping genus level descriptions onto species? Does that result in multiple species having the same descriptions?\n- Can you elaborate on the human evaluation task? Per the supplement, there was only one domain expert in the group of annotators and 15 computer science students. How well prepared are your human validators to assess the quality of the captions? What is the ecologist's area of expertise? \n- What is the overlap in the species coverage in the zero-shot datasets? How much of the performance improvement is related to prior knowledge of the organism groups being tested?\n- At line 230, it is noted that there are 347 taxonomic classes in TreeOfLife-10M. Is that a typo? Or are classes in this case the level of the taxonomic tree?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: the authors use an interesting approach to caption generation, prompting MLLMs with format specification on a per-class basis. The idea was to constrain output to focus on salient morphological description that can be challenging to pick out of raw text without guidance. The approach to leveraging the captions in BioCLIP is appealingly simple. \n- Quality: The overall qualtiy is quite good, lots of experiments executed with sufficient data. The ablation studies do a nice job of illustrating the benefits of adding these captions to the model. \n- Clarity: The paper is well-written and relatively easy to follow. There are a few sections (noted below) that could benefit from some rewriting. \n- Significance: This work provides more evidence that species classification is easier when you can direct the model to pay more attention to informative parts of the target. The use of captions, and the way in which they were generated, is a useful addition to the toolkit.", "weaknesses": "The format example design discussion needs expansion. This seems to be a critical element of the work and it is treated very narrowly. It isn't clear what the 'classes' are that were used to query Gemini Deep Research, who did the winnowing of the results, or how consistent the examples were across the classes. This element may itself benefit from an exploration of how variable those exemplars are between model runs and how consistent the human overseers where in selecting appropriate descriptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbGGmKTBRH", "reviewer": "Reviewer_qs4b", "summary": "This paper proposes BioCAP, a biological multimodal model that leverages descriptive captions as an additional supervision signal alongside species labels. The authors argue that images and captions provide complementary views of a species’ latent morphospace, capturing biologically meaningful traits while suppressing spurious correlations. To overcome the scarcity of instance-level captions, they generate synthetic captions using a large multimodal language model (InternVL3 38B) guided by Wikipedia-derived visual information and taxon-tailored format examples. BioCAP is trained with a shared visual and text encoder, but uses dual visual projectors for taxonomic labels and captions. Experiments show BioCAP improves species classification and text–image retrieval with several baselines, while careful ablation studies demonstrate the benefits of their approach.", "questions": "**Questions for the Authors**\n\n1. Why was BIOCLIP 2, the current state-of-the-art biological multimodal model, not used as the base model or included as a baseline?\n2. Have the authors evaluated performance specifically on underrepresented or rare species to assess generalization across the long tail of biodiversity?\n3. How sensitive is the model to the number and diversity of curated examples per taxonomic class? Would increasing this number improve generalization or trait coverage?\n4. Are there analyses on how synthetic caption errors propagate through the training process, especially for species with limited textual coverage?\n\n**Actionable Feedback**\n\n1. Compare BioCAP against BIOCLIP 2, the current state-of-the-art, and provide justification for choosing BIOCLIP as the base model. Discussion of alternative multimodal foundation models (e.g., LLaVA) would also strengthen the paper.\n2. Include few-shot classification, additional biological visual benchmarks (FishNet, Newt-AWA2, Herb-19, PlantDoc), and more diverse retrieval tasks. Explicit analysis of underrepresented or rare species would help assess generalization.  \n3. Assess whether limiting to three examples per taxonomic class biases the model, and consider ablations with more examples.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Novel approach**: The use of synthetic descriptive captions for biological multimodal models is innovative and addresses a clear bottleneck in organismal biology.\n- **Strong empirical performance**: BioCAP outperforms multiple baselines (CLIP, SigLIP, BioTrove-CLIP, BioCLIP, FG-CLIP) on ten species classification benchmarks, retrieval tasks (PlantID, Cornell Bird), and natural language understanding (INQUIRE-Rerank).\n- **Comprehensive evaluation**: The authors evaluate classification, retrieval, and language understanding, covering multiple organismal domains.\n- **Human evaluation and careful ablation studies**: Caption quality is assessed using four metrics (groundedness, specificity, completeness, and clarity) showing the effectiveness of synthetic captions and addressing hallucination concerns. Ablations confirm that dual projectors outperform a single projector, and that adding synthetic captions improves generalization, even for species without Wikipedia coverage.", "weaknesses": "- **Choice of base model and baselines**: BioCAP is initialized from BIOCLIP, but the paper does not justify why **BIOCLIP 2**, the current state-of-the-art, was not used as the base model or included as a baseline, despite being cited in the experimentation section. Furthermore, there is no discussion of alternative multimodal models (e.g., LLaVA) or rationale for selecting BIOCLIP.\n- **Evaluation scope and claims**: Evaluation is limited to zero-shot species classification and a subset of retrieval benchmarks; prior work includes few-shot tasks and additional biological visual datasets (FishNet, NeWT, AwA2, Herb. 19, PlantDoc). \n- **Coverage limitations**: While ablation studies indicate improvements even for non-covered species, the paper does not explicitly analyze performance specifically on underrepresented species, leaving it unclear whether the approach benefits the full long-tail of species. \n- **Trait diversity in curated examples**: Only up to three examples per taxonomic class may bias the model toward common traits, potentially underrepresenting rare or atypical characteristics. An ablation study on different number of curated examples would be a nice addition.\n- **Propagation of synthetic caption errors**: While human evaluation supports caption quality, errors or omissions in synthetic captions could still affect model performance for species with limited or no reliable descriptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)