Title: BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Biology Foundation Models

Multi-modal Representation Learning
Abstract: Foundation models trained at scale exhibit remarkable emergent behaviors, learning new capabilities beyond their initial training objectives. We find such emergent behaviors in biological vision models via large-scale contrastive vision-language training. To achieve this, we first curate TreeOfLife-200M, comprising 214 million images of living organisms, the largest and most diverse biological organism image dataset to date. We then train BioCLIP 2 on TreeOfLife-200M to distinguish different species. Despite the narrow training objective, BioCLIP 2 yields extraordinary accuracy when applied to various biological visual tasks such as habitat classification and trait prediction. We identify emergent properties in the learned embedding space of BioCLIP 2. At the inter-species level, the embedding distribution of different species aligns closely with functional and ecological meanings (e.g., beak sizes and habitats). At the intra-species level, instead of being diminished, the intra-species variations (e.g., life stages and sexes) are preserved and better separated in subspaces orthogonal to inter-species distinctions. We provide formal proof and analyses to explain why hierarchical supervision and contrastive objectives encourage these emergent properties. Crucially, our results reveal that these properties become increasingly significant with larger-scale training data, leading to a biologically meaningful embedding space.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents BioCLIP‑XL, a CLIP-based model that maps biological images and taxonomic labels into a shared embedding space. Trained with hierarchical contrastive learning on the new TreeOfLife‑200M dataset, it boosts zero-shot species classification accuracy by about 18% over the original BioCLIP. The authors also identify “emergent properties” in the embeddings: species naturally sort by ecological or functional traits, while subtle within‑species differences stay isolated in their own subspaces. Ablation tests and theoretical analysis confirm these effects and explain how they arise.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1) The paper improves the hierarchical contrastive learning model based on BioCLIP, effectively improving the model's efficiency in utilizing hierarchical information.\n\n2) The TreeOfLife-200M dataset is proposed, providing high-quality data resources for subsequent biological vision research\nIt is very valuable for in-depth analysis of emergent properties.\n\nWeaknesses:\n\nAside from the dataset imbalance and incomplete theoretical analysis noted in the paper, another potential drawback is the simplicity of the text input: it only uses taxonomic labels (scientific or common names) and does not include ecological or appearance descriptions for each rank. Despite this limitation, due to data availability and missing fine‑grained descriptions, the ablation studies and downstream evaluations show that the model still generalizes well across tasks.\n\nQUESTIONS:\n1) Could you include visualizations of the sample distribution at each taxonomic rank to more clearly illustrate the dataset’s imbalance?\n\n2) Given the data imbalance, might you consider reporting class‑averaged or balanced accuracy alongside top‑1 accuracy for a more comprehensive evaluation?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper applies large-scale contrastive vision-language models to classify a dataset comprising 214 million images of living organisms. The objective is species classification, along with additional biological visual tasks such as habitat classification and trait prediction.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The experiments leverage a large-scale dataset containing 214 million images, which is an impressive data scale that strengthens the validity of the reported results and supports robust model training.\n\n- The study performs classification over 10 distinct tasks, demonstrating the flexibility and broad applicability of the proposed approach.\n\n- The authors systematically investigate the impact of data scaling on model performance across these tasks, providing valuable insights into how increased data size translates into improvements in generalization and task transfer.\n\n- The presentation of ideas and results is well-organized and clear, supported by thoughtfully designed figures and tables that effectively communicate key findings to the reader.\n\n- The paper goes beyond empirical observations by introducing and proving a theorem that explains the ecological alignment across species while separating intra-species variation, offering a sound theoretical foundation for the proposed methods.\n\n- The experimental evaluation includes thorough comparisons against several baseline models on multiple benchmark datasets, helping to validate the effectiveness and superiority of the proposed approach through comprehensive empirical evidence.\n\n**Weaknesses**\n- The classification tasks evaluated in the paper may be considered relatively trivial downstream tasks, as they primarily rely on morphological data (images) to distinguish between organisms and their associated feature attributes. This could limit the broader applicability of the conclusions to more complex or less morphologically distinct domains.\n\n- In the BIOSCAN-5M project, which the authors themselves cite and leverage in their experiments, multimodal data were used in conjunction with a contrastive loss function to perform species classification. However, the current paper does not adequately compare its proposed approach to BIOSCAN-5M, nor does it sufficiently discuss how its architecture builds upon or improves over the BIOSCAN-5M framework.\n\n- The primary novelty of the paper appears to be centered on the scale of the dataset used in experiments. While this scale is impressive, there is a risk that the contribution could be perceived as incremental if the architectural innovations or methodological advances are not clearly distinguished beyond simply scaling up data.\n\nQUESTIONS:\nI have no further questions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Bio-CLIP-XL, a powerful CLIP model trained on the mega-scale TREEOFLIFE-200M dataset, comprising 214 million image-text pairs of living organisms from sources including GBIF, BIOSCAN, and EOL. By achieving state-of-the-art performance on multiple benchmarks, this work represents a significant contribution to the machine learning community. The paper also provides insightful discussions on the model's training strategy to avoid neural collapse and its resulting emergent properties.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Scale and Diversity of the Training Data: The assembly of TREEOFLIFE-200M is a major achievement. \n\n2. Robust Model Training and State-of-the-Art Performance: The paper demonstrates a robust and technically sophisticated training strategy, effectively leveraging large-scale computational resources. The resulting model, BIOCLIP-XL, sets a new state-of-the-art on a wide range of biological benchmarks, showcasing its capabilities as a powerful foundation model for the life sciences.\n\n3. Avoided the Neural Collpsing often seen in constrast learning on fine grained data, and provided insightful discussion.\n\n4. Insightful Theoretical Framework: The theoretical analysis, particularly Theorem 5.1, provides a compelling mathematical justification for the observed preservation of fine-grained features. It elegantly explains how the model learns to represent different factors of variation (species identity vs. intra-species traits) in orthogonal subspaces, providing a solid theoretical foundation for the empirical results.\n\nWeakness\n1. Questionable Foundation of Taxonomic Standardization: The decision to standardize all taxonomic labels against the GBIF Backbone Taxonomy as a single source of truth is a significant concern from a biological standpoint. While pragmatic for automation, GBIF is a secondary aggregator, not a primary taxonomic authority. Biological classifications are dynamic and constantly revised in peer-reviewed literature. By anchoring the entire dataset to a single, potentially lagging source, the model may have learned outdated species concepts, which could lead to critical errors in downstream applications (e.g., misidentifying recently split species with different conservation statuses).\n\n2. Potential Mismatch with Core Scientific Needs for ecologists and conservationists: The model's exclusive focus on image identification (\"what\") overlooks the spatiotemporal metadata (location and date) that is often more critical for its target users. For ecologists and conservationists, core tasks like tracking species ranges, modeling habitats, or monitoring invasive species depend on knowing \"where\" and \"when\" an observation occurred. By discarding this essential context, the model's practical utility for a large portion of the biological community is significantly limited. The impact of the work on biology would be significantly increased if the dataset could provide high quality data on where and when the animals are spotted.\n\n3. Fragility of the Text-Embedding-Based Prototypes: The theory relies on class prototypes being defined by text embeddings of taxonomic names. This creates a brittle system that is highly vulnerable to taxonomic revisions. For example, if a species is reclassified (a common occurrence), its text-based prototype could become instantly obsolete. A name change could cause a prototype to point to an entirely different concept, a species merger could cause two distinct prototypes to collapse into one, and a species split would leave the model with no prototype for the newly described entity. This tight coupling of the model's core structure to a non-permanent string of text could be a vulnerability.\n\nQUESTIONS:\n1. This work represents a significant contribution to morphology-based species identification using deep learning. In modern biology, however, there is a strong trend towards using genomic sequencing (e.g., DNA barcoding) for this task, often seen as providing a more direct \"ground truth\". Could the authors elaborate on the specific scientific and practical advantages of their image-based approach? In what scenarios is a model like Bio-CLIP-XL superior to, or a necessary complement to, genomic methods for understanding biodiversity?\n\n2. Could the authors clarify the composition of the 5 million images from the BIOSCAN dataset? The supplementary material suggests all 5M images were used, but the original BIOSCAN[1] dataset documentation indicates only ~473,000 images have expert-verified morphological labels, with others labeled via DNA barcoding.\n\n\n   2.1. What was the labeling methodology for the ~4.5 million images that lacked expert morphological IDs? \n\n\n   2.2. Could you also explain the reduction in class count from the 22,622 species in the source data to the 18,711 classes used in your final dataset?\n\n3. Back to weakness 3. I am thinking a possibile case it could become the strength. What if, the text embeddings of different names are too similar in Bio-CLIP-XL may suggesting a taxnonmy merger? And this merge is happened in academia. If this is confirmed, the model can be a guide for taxnonmy revisement.\n\n[1]. Gharaee, Z., Lowe, S. C., Gong, Z., Millan Arias, P., Pellegrino, N., Wang, A. T., ... & Chang, A. (2024). Bioscan-5m: A multimodal dataset for insect biodiversity. Advances in Neural Information Processing Systems, 37, 36285-36313.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}