Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs

## Abstract

Abstract Scientific Large Language Models (Sci-LLMs) have emerged as a promising frontier for accelerating biological discovery. However, these models face a fundamental challenge when processing raw biomolecular sequences: the tokenization dilemma .
Whether treating sequences as a specialized language, risking the loss of functional motif information, or as a separate modality, introducing formidable alignment challenges, current strategies fundamentally limit their reasoning capacity.
We challenge this sequence-centric paradigm by positing that a more effective strategy is to provide Sci-LLMs with high-level structured context derived from established bioinformatics tools, thereby bypassing the need to interpret low-level noisy sequence data directly. Through a systematic comparison of leading Sci-LLMs on biological reasoning tasks, we tested three input modes: sequence-only, context-only, and a combination of both. Our findings are striking: the context-only approach consistently and substantially outperforms all other modes. Even more revealing, the inclusion of the raw sequence alongside its high-level context consistently degrades performance, indicating that raw sequences act as informational noise, even for models with specialized tokenization schemes. These results suggest that the primary strength of existing Sci-LLMs lies not in their nascent ability to interpret biomolecular syntax from scratch, but in their profound capacity for reasoning over structured, human-readable knowledge. Therefore, we argue for reframing Sci-LLMs not as sequence decoders, but as powerful reasoning engines over expert knowledge. This work lays the foundation for a new class of hybrid scientific AI agents, repositioning the developmental focus from direct sequence interpretation towards high-level knowledge synthesis.

## 1 Introduction

The convergence of artificial intelligence and the life sciences has given rise to a new class of powerful tools: Scientific Large Language Models (Sci-LLMs). Built on Transformer architectures (e.g. BERT, GPT) that have revolutionized natural language processing [14], these models hold immense promise for accelerating biological discovery [22]. From predicting protein function [8] to designing novel therapeutics [16], Sci-LLMs such as Intern-S1 [5], Evolla [38], and NatureLM [34] are being developed to interpret the complex “language of life” encoded in DNA, RNA, and protein sequences [32]. Early efforts have demonstrated their potential, sparking visions of an AI-driven future for scientific research. This burgeoning field has largely coalesced around two primary strategies for integrating biomolecular data [19]. The first “sequence-as-language” approach treats sequences as a specialized form of language, extending the model’s vocabulary to include individual amino acids or nucleotides and pre-training it on vast corpora of sequence and text data. The second “sequence-as-modality” approach, inspired by multimodal learning, treats sequences as a distinct modality, employing a specialized encoder (e.g., a pre-trained biological foundation model like ESM [23] and Evo [12]) to generate rich embeddings that are then aligned with and injected into the language model’s input space, allowing LLMs to reason over high-level features of the sequence provided by the encoder, rather than the raw sequence itself [1, 25, 10].

While both paradigms have shown progress, they share a fundamental, yet often overlooked, vulnerability that we term the tokenization dilemma. In the “sequence-as-language” paradigm, the tokenization process is often too granular
[29, 8]. By breaking down sequences into their atomic components—single amino acids or nucleotides—it destroys the very structures that carry biological meaning: functional motifs, domains, and regulatory elements [14]. The model is consequently forced into the complicated task of re-learning these fundamental “words” of biology from a stream of disconnected “letters,” a process that is both inefficient and struggles with generalization. Conversely, the “sequence-as-modality” paradigm, while preserving structural information within its high-fidelity embeddings, introduces a formidable alignment challenge [17]. The hidden space learned by a bioinformatics encoder is governed by the principles of evolution and biophysics, a world of alpha-helices and selective pressure. The hidden space of an LLM, however, is shaped by human language. Bridging this profound semantic gap between the two modalities is a non-trivial task, and imperfect alignment can introduce ambiguity or even misinterpretation, limiting the model’s ability to ground its reasoning accurately in the underlying biological reality. We are, in essence, asking these models to perform a task for which they are ill-equipped: they are becoming lost in tokenization.

Figure: Figure 1: Paradigms for integrating biomolecular sequences into Sci-LLMs. (a) The sequence-as-language approach, tokenization fragments sequences into atomic symbols, potentially destroying functional motifs. (b) The sequence-as-modality approach preserves structure via specialized encoders but suffers from semantic misalignment with natural language. (c) The context-driven approach leverages bioinformatics tools to provide high-density, natively aligned textual context.
Refer to caption: x3.png

In this work, we challenge the prevailing sequence-centric view and propose an alternative, more effective paradigm to overcome the tokenization dilemma. We hypothesize that rather than forcing LLMs to directly decipher the noisy, low-level syntax of raw biomolecular sequences, we should leverage their core strength: reasoning over high-level, structured knowledge. Decades of accumulated biological wisdom are embedded in expert tools and databases – from BLAST for sequence homology to Pfam for conserved domains and Gene Ontology for functional terms. As shown in Figure [1](https://arxiv.org/html/2510.23127v2#S1.F1), we posit that these resources can be transformed into an information-rich textual context for the LLM. This “context”, presented as human-readable text, is not only information-dense, having already distilled functional insights from the raw sequence, but is also natively aligned with the LLM’s linguistic domain, entirely circumventing the tokenization dilemma.

We conduct a systematic empirical study across a representative set of state-of-the-art Sci-LLMs. Surprisingly, we observe that adding the raw sequence to an already informative context often degrades performance: the sequence acts as a form of “informational noise” that confuses an otherwise well-informed model. When both sequence and context are given, the sequence introduces misleading signals that reduce accuracy, suggesting that the true power of current Sci-LLMs lies not in their ability to serve as de novo sequence interpreters, but as sophisticated reasoning engines over integrated domain knowledge. Models that are fed high-level biological context can make insightful connections and generalizations whereas those fed only raw sequences struggle to draw any inference until they essentially “learn biology” from scratch.

## 2 Related Work

### 2.1 Foundation Models in Biological Representation

Foundation models for biological sequences have made rapid strides in representation learning. In the protein domain, large language models like ProtBERT [11] and the ESM series [23, 15] are trained on massive sequence corpora, capturing signals of evolutionary conservation, structural motifs, and residue co-variation that enable downstream generalization. On the nucleotide side, models such as DNABERT [20] and the more recent Nucleotide Transformer [9] apply $k$-mer tokenization or other subword strategies to genome-scale data, achieving high accuracy in identifying promoters, splice sites, and transcription factor binding locations. Multi-species genome models like DNABERT-2 [39] further improve efficiency by replacing $k$-mers with Byte-Pair Encoding to accommodate longer input sequences. Meanwhile, specialized transformer architectures have extended context lengths to capture distal regulatory interactions and boost gene expression prediction [4, 27, 28]. Despite their powerful representational capacity, these bio-sequence foundation models largely act as “black boxes”. Their internal embeddings are high-dimensional and not straightforwardly mapped to human-interpretable biological units like motifs, domains, or pathways, making it difficult to extract mechanistic insight.

### 2.2 Scientific Large Language Models

Large language models tailored to scientific domains (Sci–LLMs) have rapidly advanced, extending the success of general LLMs into tasks like protein or molecule design, genomic analysis, and scientific reasoning. Galactica [32], a 120-billion-parameter model trained on a corpus of papers and knowledge bases, was introduced to store and reason over scientific knowledge. Domain-focused sequence models have also emerged: NatureLM [34], for example, is a unified sequence-based model pre-trained across proteins, nucleic acids and small molecules. Likewise, Intern–S1 [5] is a recent large multimodal MoE model (28B activated parameters) with specialized tokenization and encoders for different scientific modalities. In this work, we focus on biomolecular understanding as a representative scientific challenge: information is inherently encoded in sequences (genes or proteins), which can be expressed in textual form or as a distinct modality, making it an ideal testbed for probing how well Sci-LLMs integrate domain knowledge and whether they truly understand biological sequences.

### 2.3 Existing Strategies in Bridging Sequences and Language

Sci-LLMs have adopted several strategies to bridge low-level biomolecular sequences with higher-level reasoning and knowledge. One common approach is treating sequences as a specialized language. Models like NatureLM [34] and Intern–S1 [5] ingest raw or tokenized sequences directly as input, training on vast datasets of sequences annotated with text so that the model learns joint representations. Another emerging strategy is treating sequences as a separate modality. For example, EvoLLaMA [25] incorporates a protein structure encoder and a sequence encoder alongside an LLM to enable multimodal protein question-answering, and Evolla [38] employs SaProt [30] as the structure encoder. BioReason [12] similarly couples a frozen DNA foundation model Evo [28] with a language model Qwen3 [35], so that genomic sequences are converted into contextual embeddings which the LLM can reason over in natural language. A third line of work explores agent-based or tool-augmented approaches. Rather than having a single model directly analyze sequences, the LLM is equipped with the ability to call external tools or databases as needed. Notable examples include GeneAgent [33], which self-verifies for gene-set analysis using domain databases, and ChemCrow [7], which uses an agent to plan multi-step chemistry tasks by invoking a suite of expert tools. While all these strategies have pushed the frontier of scientific AI [18], it remains unclear how much of the success in Sci-LLMs comes from genuine reasoning over raw sequences. In this work, we adopt a deliberately context-driven baseline—providing the model with only high-level, structured annotations of the sequence. By comparing this setup to one where the model sees the raw sequence, we can assess how and when sequence information truly adds value.

## 3 Preliminaries

### 3.1 The Biomolecular Understanding Task

Let $\mathcal{S}$ be the space of all possible biomolecular sequences (e.g., protein, RNA, DNA, and small molecules), $\mathcal{Q}$ be the space of natural language questions about a sequence, and $\mathcal{A}$ be the space of plausible natural language answers. The general task is to learn a function $f:\mathcal{S}\times\mathcal{Q}\to\mathcal{A}$ that maps a sequence $s\in\mathcal{S}$ and a question $q\in\mathcal{Q}$ to a factually correct and relevant answer $a\in\mathcal{A}$.

A Scientific LLM, denoted as $\mathcal{M}$, aims to approximate this function by learning a set of optimal parameters $\theta$. The generation of an answer can be expressed as:

$$ $a=\mathcal{M}(s,q;\theta)$ (1) $$

The fundamental distinction between the paradigms we investigate lies in how the sequence $s$ and question $q$ are represented and processed by the model $\mathcal{M}$.

### 3.2 Sequence-as-Language

This approach, utilized by models such as NatureLM [34] and Intern-S1 [5], treats a biomolecular sequence as a specialized string of text. Let $T_{seq}$ be a tokenizer that maps a sequence $s$ into a series of tokens from a biological vocabulary, $V_{bio}$, and let $T_{text}$ be a standard tokenizer for a natural language question $q$ with vocabulary $V_{text}$. The model operates on an extended vocabulary $V_{ext}=V_{text}\cup V_{bio}$. The input to the LLM, $X_{\text{input}}$, is formed by the concatenation of the tokenized question and sequence:

$$ $X_{\text{input}}=[T_{\text{text}}(q);T_{\text{seq}}(s)]$ (2) $$

The model $\mathcal{M}$ then processes this unified token sequence autoregressively to generate the answer $a$:

$$ $P(a|s,q)=\prod_{k=1}^{|a|}P(a_{k}|a_{<k},X_{\text{input}};\theta)$ (3) $$

It introduces the first horn of the tokenization dilemma: the weak representation comes from the low-level tokenization atomizes the sequence, destroying the hierarchical structures of functional motifs. The model receives a high-dimensional but low-information-density signal, from which it must re-learn the fundamental grammar of biology, a notoriously difficult and data-intensive task.

### 3.3 Sequence-as-Modality

Inspired by successes in vision-language modeling, this paradigm—employed by models like Evolla [38] and BioReason [12]—treats the biomolecular sequence as a distinct, non-textual modality. A specialized, pre-trained biomolecular encoder, $\mathcal{E}_{bio}:\mathcal{S}\to\mathbb{R}^{L\times d}$, first transforms the sequence $s$ into a sequence of rich, contextualized embeddings. An alignment module, $\mathcal{A}_{\text{align}}$, then projects these biological embeddings into the LLM’s semantic space, creating an aligned sequence representation $E_{\text{aligned_seq}}\in\mathbb{R}^{K\times d}$. The final input to the LLM is a structured combination of the embedded text and the aligned sequence embeddings:

$$ $X_{\text{input}}=[T_{\text{text}}(q);E_{\text{aligned_seq}}]$ (4) $$

While this approach preserves the sequence’s structural integrity, it introduces the second horn of the tokenization dilemma: the challenge of semantic misalignment. The semantic space of $\mathcal{E}_{bio}$ is governed by the principles of biophysics and evolution, whereas the LLM’s space is structured by human linguistics and logic. The alignment module $\mathcal{A}_{\text{align}}$ must learn to bridge this profound semantic gap. Any imperfection in this translation can inject ambiguity or noise.

## 4 The Context-Driven Approach

In this work, we propose and investigate a third paradigm that circumvents the tokenization dilemma entirely. This approach posits that the most effective way to leverage an LLM is to provide it with what it processes best: high-quality, human-readable text.

We define a set of established bioinformatics tools as a function $\mathcal{C}:\mathcal{S}\to\mathcal{T}_{\text{context}}$, where $\mathcal{T}_{\text{context}}$ is the space of structured, human-readable textual descriptions. This function transforms a raw sequence $s$ into a high-level context $c=\mathcal{C}(s)$. The model’s input deliberately omits the raw sequence $s$:

$$ $X_{\text{input}}=[T_{\text{text}}(q);T_{\text{text}}(c)]$ (5) $$

The model approximates the answer’s probability by conditioning only on high-level knowledge:

$$ $P(a|s,q)\approx P(a|c,q)=\prod_{k=1}^{|a|}P(a_{k}|a_{<k},X_{\text{input}};\theta)$ (6) $$

This paradigm reframes the task from one of low-level sequence interpretation to one of high-level knowledge synthesis. The context $c$ is information-dense and natively aligned with the LLM’s natural language space, shifting the model’s role from low-level sequence interpretation to high-level knowledge synthesis and reasoning.

Specifically, we design a pipeline to generate and structure the context for any given protein sequence. First, we generate a comprehensive functional profile by executing a multi-source toolchain. InterProScan [21] is used to identify conserved domains and motifs based on the sequence’s intrinsic features, while BLASTp [2] retrieves annotations from close homologs in the Swiss-Prot database [6]. For novel orphan sequences lacking hits from these tools, we use the tri-modal retrieval model ProTrek [31] as a fallback to generate a basic semantic description. The outputs from these tools are then integrated into a final context using an empirically-driven hierarchical strategy. The details are in the Appendix [7](https://arxiv.org/html/2510.23127v2#S7).

A central concern in fair evaluation is the prevention of information leakage. Our context-driven approach is explicitly designed to avoid label leakage along two complementary axes:

##### Intrinsic analysis rather than identity lookup.

We employ InterProScan to detect conserved domains and motifs intrinsic to the query sequence. This constitutes an *ab initio*, feature-based analysis grounded in domain knowledge bases, not in annotation records of the query protein itself. Consequently, even for genuinely novel proteins, recognizable elements such as a kinase domain can be identified without ground-truth labels.

##### Homology-based inference rather than direct annotation matching.

When using BLASTp, we restrict our context-driven approach to reading GO annotations from the homologous sequences, rather than from the query protein’s own record. This reflects standard bioinformatics practice: predicting the function of unknown sequences by analogy to characterized homologs rather than simply retrieving pre-annotated answers.

## 5 The Tokenization Dilemma in Practice

### 5.1 The Primacy of Context over Sequence

Following a standardized protocol inspired by Evolla [38], our benchmark focuses on three fundamental aspects of protein biology: molecular function, metabolic pathway involvement, and subcellular localization. For each protein in our test set, we generated queries corresponding to these categories (e.g., “What is the function of this protein?”). To ensure a set of factually grounded and verifiable ground truths, a question was only included if its corresponding annotation field was explicitly present in the source database entry, from which the answer was directly excerpted. Performance was quantified using an automated pipeline, leveraging a general-purpose LLM as an expert judge, a metric we term the LLM-Score. A detailed description of the dataset construction, evaluation protocol, and prompt design is provided in Appendices [8](https://arxiv.org/html/2510.23127v2#S8) and [9](https://arxiv.org/html/2510.23127v2#S9).

We evaluate the performance of both specialized Sci-LLMs and leading general-purpose LLMs across three distinct input configurations: (i) Sequence-Only, where the model receives only the raw protein sequence; (ii) Sequence + Context, a combined input; (iii) Context-Only, where the model receives only the high-level context. The results are presented in Table [5.1](https://arxiv.org/html/2510.23127v2#S5.SS1).

**Table 1: Comparison of performance across specialized Sci-LLMs and general-purpose LLMs on our protein QA benchmark. $\checkmark$ indicates that the corresponding input modality was provided to the model. Results are reported on three task-specific subsets—Function (Func.), Pathway (Path.), and Subcellular Location (Sub. Loc.)—as well as the overall average (All). The best score for each model is underlined, and the overall best performance across all models is highlighted in bold.**
| Model | Sequence | Context | Func. | Path. | Sub. Loc. | All |
| --- | --- | --- | --- | --- | --- | --- |
| \rowcolor[HTML]EFEFEF     Specialized Sci-LLMs |  |  |  |  |  |  |
| Intern-S1 | $\checkmark$ |  | 20.57 | 26.56 | 69.75 | 43.33 |
| Intern-S1 | $\checkmark$ | $\checkmark$ | 74.18 | 98.85 | 93.00 | 84.03 |
| Intern-S1 |  | $\checkmark$ | 76.22 | 97.60 | 95.60 | 86.15 |
| Evolla | $\checkmark$ |  | 40.23 | 72.71 | 79.76 | 59.93 |
| Evolla | $\checkmark$ | $\checkmark$ | 57.46 | 84.69 | 83.05 | 70.53 |
| Evolla |  | $\checkmark$ | 65.77 | 83.33 | 81.88 | 74.02 |
| NatureLM | $\checkmark$ |  | 3.58 | 5.52 | 10.45 | 6.82 |
| NatureLM | $\checkmark$ | $\checkmark$ | 42.33 | 64.25 | 32.30 | 38.86 |
| NatureLM |  | $\checkmark$ | 44.77 | 51.35 | 32.51 | 39.50 |
| \rowcolor[HTML]EFEFEF     General LLMs |  |  |  |  |  |  |
| Deepseek-v3 | $\checkmark$ |  | 10.98 | 24.54 | 74.72 | 40.77 |
| Deepseek-v3 | $\checkmark$ | $\checkmark$ | 77.40 | 91.35 | 94.75 | 86.03 |
| Deepseek-v3 |  | $\checkmark$ | 75.79 | 93.96 | 93.65 | 84.99 |
| Gemini2.5 Pro | $\checkmark$ |  | 10.40 | 13.85 | 77.58 | 41.25 |
| Gemini2.5 Pro | $\checkmark$ | $\checkmark$ | 79.12 | 94.17 | 94.65 | 86.98 |
| Gemini2.5 Pro |  | $\checkmark$ | 79.17 | 98.65 | 94.56 | 87.19 |
| GPT-5 | $\checkmark$ |  | 19.64 | 17.08 | 64.15 | 39.83 |
| GPT-5 | $\checkmark$ | $\checkmark$ | 79.89 | 89.48 | 71.30 | 76.45 |
| GPT-5 |  | $\checkmark$ | 77.25 | 85.73 | 73.05 | 75.76 |

### 5.2 Deconstructing the Dilemma I: The Weak Representation

We visualize the embeddings of the outputs, where ground-truth classes were established by clustering homologous proteins using MMseqs2 at a 50% sequence identity threshold. For each model, we extracted the final-layer embeddings for their outputs. We employed t-SNE [26] to project them into a 2D space. The quality of the resulting functional separation was then quantified by performing clustering on the high-dimensional embeddings and calculating the Adjusted Rand Index (ARI) against the MMseqs2 ground-truth clusters. For our context-driven approach, we generated embeddings from the structured context itself using the text embedding model Qwen-embedding [37]. The results are visualized in Figure [2](https://arxiv.org/html/2510.23127v2#S5.F2).

Figure: Figure 2: The visualization of representation spaces.
Refer to caption: x4.png

The visualizations confirm the weak representation horn of the tokenization dilemma. The sequence-as-language models, NatureLM (c) and Intern-S1 (b), exhibit highly disorganized latent spaces, quantitatively confirmed by their low ARI scores of 0.492 and 0.690, respectively. Evolla (a), which employs the sequence-as-modality paradigm, demonstrates a significantly more structured representation, highlighting the benefit of using a specialized sequence encoder. However, both paradigms are dramatically outperformed by our context-driven approach (d). The representation derived purely from the textual context achieves near-perfect functional separation.

### 5.3 Deconstructing the Dilemma II: The Semantic Misalignment

While the sequence-as-modality paradigm, exemplified by Evolla, overcomes the weak representation problem, it introduces a more subtle yet equally critical challenge: semantic misalignment. The specialized encoder and the generalist LLM operate in fundamentally different semantic worlds—one governed by biophysics, the other by linguistics. We performed a layer-wise representational analysis of the Evolla-10B model, tracing the informational journey of a protein sequence from its biological embedding to its final interpretation by the language model. As shown in Figure [3](https://arxiv.org/html/2510.23127v2#S5.F3), the initial SaProt encoder generates a well-structured latent space. As the Q-Former works to translate these biological embeddings for the LLM, the functional clarity begins to blur.

Figure: Figure 3: Visualization of representation spaces at different stages within the Evolla-10B model.
Refer to caption: x5.png

### 5.4 Collapse on Novel Protein Families

A critical limitation of many large-scale models is their tendency to overfit to training data, leading to poor generalization on novel examples. We adopted the evaluation protocol from Evolla [38], which partitions the test set into three subsets based on sequence identity to the training set: Easy, Medium, and Hard. The division of these subsets is described in Appendix [8](https://arxiv.org/html/2510.23127v2#S8).

Figure: Figure 4: Comparison of Evolla-10B and our approach across the easy, medium, and hard subsets.
Refer to caption: x6.png

The results, illustrated in Figure 4, reveal a dramatic divergence in generalization capability. Evolla’s performance exhibits a steep, monotonic decline as the data hardness increases. It performs well on the Easy subset with an LLM score of 81.9, where it can likely rely on memorized patterns from similar training sequences. The performance collapse of about 30% from Easy to Hard is a classic symptom of poor generalization. In stark contrast, our context-driven method demonstrates remarkable robustness. Its performance remains consistently high across all levels of difficulty. The performance is virtually unaffected by the novelty of the protein sequence. This stability stems from the fact that our approach does not rely on interpreting the raw sequence itself. Instead, it leverages high-level knowledge that are inherently designed to generalize well.

### 5.5 Degrading Phenomenon Across Time

We curated a dataset by randomly sampling about 100 proteins for each year from 1995 to 2024 based on the first publication year. The relationship between a protein’s first publication year and the models’ LLM-Scores is visualized in Figure [5](https://arxiv.org/html/2510.23127v2#S5.F5).

Figure: Figure 5: Analysis of model performance versus protein’s first publication year.
Refer to caption: x7.png

For this analysis, our context-driven approach employed DeepSeek-V3 [24] as its base LLM to ensure a fair comparison against models with similar training data cut-off dates. Our context-driven approach (a), while maintaining the highest overall performance, exhibits a slight negative trend over time due to the diminishing availability of rich, homologous information in the knowledge bases. For very recent proteins, homology-based tools like BLAST find fewer well-characterized relatives, leading to a sparser context and thus slightly less precise answers. The sequence-as-modality model, Evolla (b), displays a much more pronounced degradation. Its performance on well-studied proteins from the 1990s and early 2000s is strong, but it deteriorates significantly for proteins discovered in the last decade. It is crucial to note that Evolla’s training data, sourced from Swiss-Prot Release (202303), has a temporal bias. Therefore, part of this decline can be attributed to its lack of exposure to the most recent protein data. However, this training bias alone does not fully account for the steepness of the collapse. The trend suggests a deeper issue: Evolla’s encoder appears to rely heavily on the dense web of evolutionary information available for older, larger protein families. When faced with recent, potentially more unique proteins that lack this deep evolutionary context—a problem exacerbated by its training data cutoff—the encoder’s ability to generate meaningful representations weakens considerably. The sequence-as-language model, Intern-S1 (c), shows a performance profile that is almost entirely flat and consistently low across the entire 30-year period. This lack of temporal trend, combined with its overall poor performance, indicates a fundamental failure to extract meaningful biological signals from the raw sequence.

The above dual robustness confirms that reasoning over stable, high-level knowledge is a more robust foundation for AI in biology than relying on the difficult task of raw sequence interpretation.

## 6 Conclusion and Limitation

In this work, we confronted a fundamental challenge at the heart of modern Sci-LLMs: the tokenization dilemma. We demonstrated that current paradigms, whether treating biomolecular sequences as a specialized language or as a distinct modality, are fundamentally handicapped by issues of weak representation and semantic misalignment. Our central contribution is the validation of a third paradigm that resolves this dilemma. By shifting the focus from low-level sequence interpretation to high-level knowledge synthesis, our context-driven approach entirely circumvents the tokenization problem, as illustrated in the conceptual landscape of Figure [6](https://arxiv.org/html/2510.23127v2#S6.F6). Our approach is computationally efficient, as it leverages generalist LLMs without the retraining required by domain-specific Sci-LLMs.

While our findings are compelling, we acknowledge several limitations. For truly novel orphan proteins from unexplored regions of the protein universe, our method’s performance may be constrained. Furthermore, our current analysis has primarily focused on proteins; although we provide some preliminary exploration in Appendix [13](https://arxiv.org/html/2510.23127v2#S13), a more comprehensive treatment remains for future work.

Figure: Figure 6: The trade-off landscape of representation vs. semantic alignment. The x-axis quantifies the quality of the biological representation (measured by ARI), while the y-axis conceptually represents the degree of semantic alignment with natural language. The area of each circle is proportional to the computational cost, with larger circles indicating higher computational expenses.
Refer to caption: x8.png

## References

- Abdine et al. [2024]
Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, and Michalis Vazirgiannis.
Prot2text: Multimodal protein’s function generation with gnns and transformers.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 10757–10765, 2024.
- Altschul et al. [1990]
Stephen F Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David J Lipman.
Basic local alignment search tool.
*Journal of molecular biology*, 215(3):403–410, 1990.
- Ashburner et al. [2000]
Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al.
Gene ontology: tool for the unification of biology.
*Nature genetics*, 25(1):25–29, 2000.
- Avsec et al. [2021]
Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley.
Effective gene expression prediction from sequence by integrating long-range interactions.
*Nature methods*, 18(10):1196–1203, 2021.
- Bai et al. [2025]
Lei Bai, Zhongrui Cai, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, et al.
Intern-s1: A scientific multimodal foundation model.
*arXiv preprint arXiv:2508.15763*, 2025.
- Boeckmann et al. [2003]
Brigitte Boeckmann, Amos Bairoch, Rolf Apweiler, Marie-Claude Blatter, Anne Estreicher, Elisabeth Gasteiger, Maria J Martin, Karine Michoud, Claire O’Donovan, Isabelle Phan, et al.
The swiss-prot protein knowledgebase and its supplement trembl in 2003.
*Nucleic acids research*, 31(1):365–370, 2003.
- Bran et al. [2023]
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller.
Chemcrow: Augmenting large-language models with chemistry tools.
*arXiv preprint arXiv:2304.05376*, 2023.
- Brandes et al. [2023]
Nadav Brandes, Grant Goldman, Charlotte H Wang, Chun Jimmie Ye, and Vasilis Ntranos.
Genome-wide prediction of disease variant effects with a deep protein language model.
*Nature Genetics*, 55(9):1512–1522, 2023.
- Dalla-Torre et al. [2025]
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al.
Nucleotide transformer: building and evaluating robust foundation models for human genomics.
*Nature Methods*, 22(2):287–297, 2025.
- de Almeida et al. [2025]
Bernardo P de Almeida, Guillaume Richard, Hugo Dalla-Torre, Christopher Blum, Lorenz Hexemer, Priyanka Pandey, Stefan Laurent, Chandana Rajesh, Marie Lopez, Alexandre Laterre, et al.
A multimodal conversational agent for dna, rna and protein tasks.
*Nature Machine Intelligence*, pages 1–14, 2025.
- Elnaggar et al. [2021]
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al.
Prottrans: Toward understanding the language of life through self-supervised learning.
*IEEE transactions on pattern analysis and machine intelligence*, 44(10):7112–7127, 2021.
- Fallahpour et al. [2025]
Adibvafa Fallahpour, Andrew Magnuson, Purav Gupta, Shihao Ma, Jack Naimer, Arnav Shah, Haonan Duan, Omar Ibrahim, Hani Goodarzi, Chris J. Maddison, and Bo Wang.
Bioreason: Incentivizing multimodal biological reasoning within a dna-llm model, 2025.
URL [https://arxiv.org/abs/2505.23579](https://arxiv.org/abs/2505.23579).
- Fang et al. [2024]
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen.
Mol-instructions: A large-scale biomolecular instruction dataset for large language models.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Ferruz et al. [2022]
Noelia Ferruz, Steffen Schmidt, and Birte Höcker.
Protgpt2 is a deep unsupervised language model for protein design.
*Nature communications*, 13(1):4348, 2022.
- Hayes et al. [2025]
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al.
Simulating 500 million years of evolution with a language model.
*Science*, 387(6736):850–858, 2025.
- Hie et al. [2024]
Brian L Hie, Varun R Shanker, Duo Xu, Theodora UJ Bruun, Payton A Weidenbacher, Shaogeng Tang, Wesley Wu, John E Pak, and Peter S Kim.
Efficient evolution of human antibodies from general protein language models.
*Nature biotechnology*, 42(2):275–283, 2024.
- Hu et al. [2025]
Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, Tianbin Li, Guohang Zhuang, Jiaqi Liu, Yingzhou Lu, et al.
A survey of scientific large language models: From data foundations to agent frontiers.
*arXiv preprint arXiv:2508.21148*, 2025.
- Huang et al. [2025]
Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Yusuf Roohani, Ryan Li, Lin Qiu, Gavin Li, Junze Zhang, et al.
Biomni: A general-purpose biomedical ai agent.
*biorxiv*, 2025.
- Hwang et al. [2024]
Yunha Hwang, Andre L Cornman, Elizabeth H Kellogg, Sergey Ovchinnikov, and Peter R Girguis.
Genomic language model predicts protein co-regulation and function.
*Nature communications*, 15(1):2880, 2024.
- Ji et al. [2021]
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri.
Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome.
*Bioinformatics*, 37(15):2112–2120, 2021.
- Jones et al. [2014]
Philip Jones, David Binns, Hsin-Yu Chang, Matthew Fraser, Weizhong Li, Craig McAnulla, Hamish McWilliam, John Maslen, Alex Mitchell, Gift Nuka, et al.
Interproscan 5: genome-scale protein function classification.
*Bioinformatics*, 30(9):1236–1240, 2014.
- Karpatne et al. [2025]
Anuj Karpatne, Aryan Deshwal, Xiaowei Jia, Wei Ding, Michael Steinbach, Aidong Zhang, and Vipin Kumar.
Ai-enabled scientific revolution in the age of generative ai: second nsf workshop report.
*npj Artificial Intelligence*, 1(1):18, 2025.
- Lin et al. [2023]
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al.
Evolutionary-scale prediction of atomic-level protein structure with a language model.
*Science*, 379(6637):1123–1130, 2023.
- Liu et al. [2024a]
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024a.
- Liu et al. [2024b]
Nuowei Liu, Changzhi Sun, Tao Ji, Junfeng Tian, Jianxin Tang, Yuanbin Wu, and Man Lan.
Evollama: Enhancing llms’ understanding of proteins via multimodal structure and sequence representations.
*arXiv preprint arXiv:2412.11618*, 2024b.
- Maaten and Hinton [2008]
Laurens van der Maaten and Geoffrey Hinton.
Visualizing data using t-sne.
*Journal of machine learning research*, 9(Nov):2579–2605, 2008.
- Nguyen et al. [2023]
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al.
Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution.
*Advances in neural information processing systems*, 36:43177–43201, 2023.
- Nguyen et al. [2024]
Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam J Bartie, Armin W Thomas, Samuel H King, Garyk Brixi, et al.
Sequence modeling and design from molecular to genome scale with evo.
*Science*, 386(6723):eado9336, 2024.
- Rao et al. [2020]
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives.
Transformer protein language models are unsupervised structure learners.
*Biorxiv*, pages 2020–12, 2020.
- Su et al. [2024a]
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan.
Saprot: Protein language modeling with structure-aware vocabulary.
In *The Twelfth International Conference on Learning Representations*, 2024a.
- Su et al. [2024b]
Jin Su, Xibin Zhou, Xuting Zhang, and Fajie Yuan.
Protrek: Navigating the protein universe through tri-modal contrastive learning.
*bioRxiv*, pages 2024–05, 2024b.
- Taylor et al. [2022]
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic.
Galactica: A large language model for science.
*arXiv preprint arXiv:2211.09085*, 2022.
- Wang et al. [2025]
Zhizheng Wang, Qiao Jin, Chih-Hsuan Wei, Shubo Tian, Po-Ting Lai, Qingqing Zhu, Chi-Ping Day, Christina Ross, Robert Leaman, and Zhiyong Lu.
Geneagent: self-verification language agent for gene-set analysis using domain databases.
*Nature Methods*, pages 1–9, 2025.
- Xia et al. [2025]
Yingce Xia, Peiran Jin, Shufang Xie, Liang He, Chuan Cao, Renqian Luo, Guoqing Liu, Yue Wang, Zequn Liu, Yuan-Jyue Chen, et al.
Naturelm: Deciphering the language of nature for scientific discovery.
*arXiv e-prints*, pages arXiv–2502, 2025.
- Yang et al. [2025]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yu et al. [2023]
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao.
Enzyme function prediction using contrastive learning.
*Science*, 379(6639):1358–1363, 2023.
- Zhang et al. [2025]
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al.
Qwen3 embedding: Advancing text embedding and reranking through foundation models.
*arXiv preprint arXiv:2506.05176*, 2025.
- Zhou et al. [2025]
Xibin Zhou, Chenchen Han, Yingqi Zhang, Jin Su, Kai Zhuang, Shiyu Jiang, Zichen Yuan, Wei Zheng, Fengyuan Dai, Yuyang Zhou, et al.
Decoding the molecular language of proteins with evolla.
*bioRxiv*, pages 2025–01, 2025.
- Zhou et al. [2024]
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu.
Dnabert-2: Efficient foundation model and benchmark for multi-species genomes.
In *The Twelfth International Conference on Learning Representations*, 2024.

## 7 Context details

Our context-driven consists of three stages: (1) generation of a multi-source evidence profile, (2) hierarchical construction of a textual context, and (3) context-based inference by LLMs.

### 7.1 Context Generation

For any given biomolecular sequence (we use proteins as the running example), we first generate a comprehensive functional profile using a deliberately crafted, multi-source toolchain.

Input: A single protein FASTA sequence.

Toolbox Execution:

- 1.
Feature-intrinsic domain analysis: We first scan the input sequence with InterProScan [21]
to identify conserved domains and key motifs by integrating multiple signature libraries (e.g., Pfam, PROSITE, SMART). This step is an analysis grounded in *intrinsic* sequence features: even for a completely novel, unannotated protein, InterProScan can recognize known modular features. We extract textual descriptions of detected domains together with any directly linked Gene Ontology (GO) annotations [3]
.
- 2.
Homology-based functional inference. In parallel, we run BLASTp [2]
against a curated reference database (e.g., Swiss-Prot) to retrieve close homologs. We *transfer* GO annotations from the most similar sequences to the query. *Critically, we never use the query’s own (possibly unknown) labels.*
- 3.
Information integration. We merge GO evidence obtained from InterProScan (feature-intrinsic) and from BLASTp (homology-based) to form a comprehensive functional profile.
- 4.
Fallback mechanism. In rare cases where neither BLASTp nor InterProScan yields informative signals (“orphan” sequences), we invoke ProTrek [31]
to synthesize a concise, model-based textual description that serves as minimal context.

### 7.2 Context Construction

The raw outputs from these tools can be redundant, conflicting, or noisy, especially for novel proteins. A naive combination of all outputs is suboptimal. Therefore, based on rigorous empirical evaluation (see Ablation Study), we developed a hierarchical strategy designed to gracefully handle the spectrum of protein novelty:

- 1.
Prioritizing High-Confidence Homology: Our analysis revealed that for generating a list of candidate GO terms, the single most reliable source is the annotation of the top homolog found by BLAST. This strategy maximizes precision while maintaining high recall (see Table [11](https://arxiv.org/html/2510.23127v2#S11)).
- 2.
Integrating Domain Information: Pfam motifs identified by InterProScan are added as a separate, complementary source of evidence, providing structural and functional context.
- 3.
Semantic Evidence: Our experiments showed that ProTrek’s semantic hits, while powerful, could introduce noise when combined with high-quality GO/Pfam data. Therefore, ProTrek’s output is used as a fallback—it is only added to the context when primary sources like GO and Pfam are sparse or absent.

This empirically-driven, hierarchical process culminates in a final textual context engineered to be as factually dense and noise-free as possible, ready for the final inference stage.

### 7.3 Context-based Inference

The final stage transforms the structured biological evidence into a query that the LLM can process. The constructed context and the original user question are formatted into a unified prompt using a predefined template as shown in Figure [4](https://arxiv.org/html/2510.23127v2#S4). The LLM’s role is to act as a knowledge synthesizer. It processes the prompt, which contains a series of factual statements derived from the context. This final step leverages the LLM’s core strength in natural language understanding and reasoning, entirely bypassing the need for it to interpret the complex, low-level syntax of the raw biomolecular sequence itself.

## 8 Dataset details

### 8.1 Protein Dataset

To ensure a comprehensive and fair evaluation of our model against Evolla, we employed a multi-faceted dataset strategy. This approach incorporates not only the benchmark datasets used in the original Evolla study but also a dataset we have meticulously reconstructed to address specific limitations in their evaluation methodology. Our assessment is primarily based on the following datasets:

- 1.

We initiated our evaluation on the original Evolla Question–Answering (QA) dataset to establish a performance baseline (Figure [4](https://arxiv.org/html/2510.23127v2#S5.F4)). Following the protocol from the original study (Zhou et al., 2025), this dataset’s test set is partitioned into three subsets based on sequence identity to the training set, designed to assess model generalization at varying levels of difficulty:

- •
Easy: Proteins with $>$30% sequence identity to the major representative clusters in the training set (covering 50% of sequences).
- •
Medium: Proteins with $>$30% sequence identity to the remaining training clusters.
- •
Hard: Proteins with $<$30% sequence identity to any training cluster.

However, despite this structured difficulty, we identified inherent limitations within the dataset. A significant portion of the questions are open-ended (e.g., “Can you share furthermore important details regarding this protein?”), for which there is no single, definitive correct answer. This ambiguity complicates evaluation, as a model’s response may be valid even if it does not align perfectly with the provided ground truth (GT). Furthermore, the GT answers were themselves generated by feeding protein database entries into a Large Language Model (LLM), rather than using the original source text, a process that may introduce stylistic biases or factual inaccuracies.

Our Benchmark Dataset
To overcome these evaluation challenges, we reconstructed the “Hard” subset of the Evolla dataset to create a more rigorous and objective benchmark. This subset, containing proteins with less than 30% sequence identity to the training set, is critical for assessing a model’s generalization capabilities on novel proteins. Our reconstruction was guided by the following principles:

- •
Standardized Questions: We replaced open-ended queries with a fixed set of three targeted question templates: “What is the function of this protein?”, “What is the pathway of this protein?”, and “What is the subcellular location of this protein?”
- •
Conditional Question Generation: To ensure every question has a verifiable answer, we only generated a specific question if the corresponding field (“Function,” “Pathway,” or “Subcellular location”) was explicitly present in the protein’s source database entry.
- •
Authentic Ground Truth: Crucially, our ground truth answers are direct excerpts from the protein’s database entry. Unlike the original dataset, we did not use an LLM to generate answers, thereby ensuring the objectivity and factual accuracy of the GT and creating a more reliable scoring standard.

The results for our benchmark dataset are presented in Table [5.1](https://arxiv.org/html/2510.23127v2#S5.SS1), where we compare the performance of different methods using sequence-only, context-only, and sequence+context inputs.

Enzyme Commission (EC) Number Dataset
To evaluate the model’s performance on a specific, structured bioinformatics task, we also utilized the EC Number dataset from the Evolla study. This task requires the model to accurately predict the functional class of enzymes, which is a standardized and important functional annotation task. Testing on this dataset allows us to gauge the model’s capabilities in handling classification-oriented protein function prediction, and is presented in Figure [7](https://arxiv.org/html/2510.23127v2#S10.F7).

Time-Split Dataset
To investigate the model’s performance over time and the impact of sequence novelty, we curated a dataset by randomly sampling about 100 proteins for each year from 1995 to 2024 based on the first publication year. This allows us to analyze the relationship between a protein’s first publication year and the model’s LLM-Score, as shown in the “Degrading Phenomenon Across Time” section (Figure [5](https://arxiv.org/html/2510.23127v2#S5.F5)). The time-split dataset is valuable for understanding how well the model generalizes to older versus more recent proteins, and whether its performance degrades as the data becomes more novel, less represented in training, or more temporally distant from the model’s training data cutoff.

Mol-Instruction Dataset
To further evaluate the model’s ability to handle diverse protein tasks, we tested it on the Mol-Instruction benchmark [13]. This benchmark provides datasets for assessing molecular understanding across a range of tasks. We specifically evaluated the model on three functionally distinct protein subsets: Catalytic Activity, General Function, and Protein Function. The performance on these datasets, as illustrated in the “Performance on Protein Function Prediction” section (Figure [9](https://arxiv.org/html/2510.23127v2#S13.F9)), highlights the model’s ability to accurately predict protein function even across different protein families, demonstrating the robustness and flexibility of our context-driven approach.

### 8.2 DNA Dataset

To assess whether our context-driven paradigm extends beyond proteomics, we evaluated its performance on a DNA-based mechanistic reasoning task. We utilized the KEGG Disease Pathway dataset curated by BioReason [12], which provides a unique benchmark for connecting genomic variants to disease phenotypes through multi-step biological pathways. Each entry in the dataset consists of a reference and a variant DNA sequence, the associated KEGG pathway definition, and the ground-truth disease outcome. The task requires the model to reason from the mutation and its functional context to predict the correct disease.

We designed three experimental setups to investigate the impact of different data configurations on the model’s performance. The first setup included only pathway-related contextual information (context-only). The second setup incorporated both the pathway context and the raw DNA sequence data (context and sequence). The third setup focused solely on the DNA sequence itself (sequence-only). These configurations allowed us to evaluate the effect of using context, sequence, or both on the model’s ability to predict mutations.

The KEGG dataset’s comprehensive pathway data, paired with precise mutation annotations, provided a solid foundation for designing these experiments. By varying the inclusion of sequence and context information, we aimed to assess the model’s ability to predict the effects of DNA mutations based on both pathway context and raw sequence data.

## 9 Evaluation Metric

To conduct a comprehensive and multi-dimensional assessment of our model’s performance, we designed specific evaluation metrics tailored to each of our distinct tasks.

### 9.1 LLM-Score for General Protein QA Tasks

For the open-ended protein question–answering task, traditional metrics based on lexical overlap (e.g., BLEU, ROUGE) are inadequate for assessing the semantic accuracy and factual consistency of generated answers. To address this, we adopted an automated evaluation methodology leveraging a LLM as an adjudicator, which we term the LLM-Score. The core principle of this metric is to use a powerful, independent third-party LLM (in this case, a DeepSeek-V3 [24] model) to score the quality of our model’s generated answer against the ground truth. The evaluation process is as follows:

- 1.
Prompt Construction: We embed the generated answer and the ground truth answer into a carefully designed prompt template. This prompt instructs the adjudicator LLM to act as an expert biologist and perform a holistic evaluation based on factual accuracy. The exact prompt is shown below.
LLM-Score Adjudicator Prompt
As an expert biologist, you are assigned to check one paragraph is aligned with facts or not. You will receive some facts, and one paragraph. Score the paragraph between 0 to 100.
The score should be the format of {"score": score}
---------
Here’s the facts:
[Ground Truth Text from Database]
---------
Here’s the paragraph:
[Generated Answer from Model to be Scored]
- 2.
Score Generation: The adjudicator LLM processes the prompt and returns a numerical score on a scale from 0 to 100, where a higher score indicates that the generated answer is of higher quality and more closely aligned with the ground truth.
- 3.
Score Extraction and Aggregation: A robust parsing function extracts the numerical score from the LLM’s textual response. The model’s final performance on the dataset is reported as the *average LLM-Score* across all test samples.

This approach moves beyond surface-level text matching to provide a deeper, more semantically aware assessment of the model’s ability to comprehend and articulate biological knowledge.

### 9.2 Hierarchical Metrics for EC Number Prediction

The Enzyme Commission (EC) number is a four-level hierarchical classification system (e.g., A.B.C.D). A proficient model should be rewarded not only for predicting the exact four-digit code but also for correctly identifying the broader functional classes at higher levels of the hierarchy. Therefore, an exact-match accuracy metric at a single level is insufficient.

To capture this, we implemented a more nuanced, hierarchical evaluation scheme. We calculate F1-Score at each of the four functional levels.

The methodology is as follows:

- •
Hierarchical Matching: To evaluate performance at level-N, all predicted and ground truth EC numbers are truncated to their first N digits for comparison. For example, at level-3, a prediction of 1.2.3.5 is considered a match for a ground truth of 1.2.3.4.
- •
Multi-Label Formulation: As a single protein can be associated with multiple EC numbers, the task is treated as a multi-label classification problem.
- •
Micro-Averaging: We compute the total number of True Positives (TP), False Positives (FP), and False Negatives (FN) by aggregating their counts over the entire test set. Global Precision, Recall, and F1-Score are then calculated from these aggregate sums.

$$ $\mathrm{Precision}_{\mathrm{micro}}=\frac{\sum\mathrm{TP}_{i}}{\sum\mathrm{TP}_{i}+\sum\mathrm{FP}_{i}}$ (7) $$

$$ $\mathrm{Recall}_{\mathrm{micro}}=\frac{\sum\mathrm{TP}_{i}}{\sum\mathrm{TP}_{i}+\sum\mathrm{FN}_{i}}$ (8) $$

$$ $\mathrm{F1}_{\mathrm{micro}}=2\cdot\frac{\mathrm{Precision}_{\mathrm{micro}}\cdot\mathrm{Recall}_{\mathrm{micro}}}{\mathrm{Precision}_{\mathrm{micro}}+\mathrm{Recall}_{\mathrm{micro}}}$ (9) $$

This suite of metrics provides a comprehensive view of the model’s predictive accuracy at varying degrees of granularity and effectively handles the multi-label nature of the data, offering a more equitable measure of true performance.

## 10 Quantitative Benchmark: EC Number Prediction

To further validate our central thesis on the primacy of context over sequence, we introduce a quantitative benchmark: EC number prediction. This task provides an objective, verifiable measure of a model’s ability to understand a protein’s precise biochemical function. The hierarchical nature of EC numbers allows us to evaluate performance at four increasing levels of specificity (from 1-digit to 4-digit matches), with the F1-Score serving as our primary metric.

We compare two categories of models: “Sequence-Only” models, which include both general-purpose LLMs and CLEAN [36] (a model specifically trained for this task), and “Context-Driven” models, which leverage the contextual information as described in the main text. The comparative performance is visualized in Figure [7](https://arxiv.org/html/2510.23127v2#S10.F7).

Figure: Figure 7: Performance on EC Number Prediction (F1-Score). The plot is divided into Sequence-Only models (left) and Context-Driven models (right). A clear and dramatic performance gap is visible between the two groups. Context-driven approaches significantly outperform even specialized sequence-based models like CLEAN, especially at higher levels (3 and 4 digits).
Refer to caption: x9.png

The results presented in Figure [7](https://arxiv.org/html/2510.23127v2#S10.F7) are unequivocal and offer several key insights:

- •
Failure of Sequence-Only LLMs: In the Sequence-Only setting, both general-purpose LLM DeepSeek-V3 and Sci-LLM Intern-S1 perform poorly. Their F1-scores plummet as the required specificity increases, demonstrating their inability to decipher complex enzymatic function from raw sequence data alone. This reinforces our “lost in tokenization” hypothesis.
- •
Context Outperforms Specialization: CLEAN, a model specifically trained on sequences for EC prediction, establishes a respectable baseline. However, every model in the Context-Driven category outperforms CLEAN across the first three levels of precision (1-digit, 2-digit, and 3-digit). This demonstrates that providing high-level context to a general model is more effective than training a specialized model on sequence data for these levels.
- •
Robustness of the Context-Driven Approach: While all models show a natural decline in performance as the task becomes harder (from 1-digit to 4-digit prediction), the context-driven models exhibit a much more graceful degradation. Gemini-2.5 Pro, using only context, achieves the highest F1-score of 0.406 on the most challenging 4-digit prediction task, a score comparable to the specialized CLEAN model’s performance (0.408).

In summary, this quantitative benchmark provides strong, direct evidence that high-level biological context is a far more effective and reliable representation of protein function for LLMs than the raw amino acid sequence. It enables general-purpose models to excel at highly specific bioinformatics tasks without needing task-specific architectures or fine-tuning.

## 11 Ablation Study

To dissect the contribution of each component within our framework, we conducted a comprehensive ablation study. The experiments were performed on our benchmark dataset, which is ideal for evaluating generalization, as it contains proteins with less than 40% sequence identity to the Evolla training set. We systematically evaluated the performance by providing different combinations of contextual information—Pfam, GO, and ProTrek—to the DeepSeek model. The results, summarized in Table [11](https://arxiv.org/html/2510.23127v2#S11), reveal the individual and synergistic effects of these components.

**Table 2: Ablation study of context components on our benchmark dataset. Scores reflect the model’s performance when provided with different combinations of contextual information. Our final, conditional approach yields the best result.**
| Context Components Provided | LLM Score |
| --- | --- |
| \rowcolor[HTML]EFEFEF               Single Components |  |
| Pfam only | 74.90 |
| GO only | 84.02 |
| ProTrek only | 66.44 |
| \rowcolor[HTML]EFEFEF               Pairwise Combinations |  |
| Pfam + GO | 84.60 |
| Pfam + ProTrek | 77.00 |
| GO + ProTrek | 77.78 |
| \rowcolor[HTML]EFEFEF               Full Combinations |  |
| Pfam + GO + ProTrek (Unconditional) | 81.56 |
| Pfam + GO + ProTrek (Conditional) | 84.99 |

##### Analysis of Individual Components

The results from single-component experiments establish a clear hierarchy of information value. Gene Ontology (GO) annotations emerge as the most powerful single source of context, achieving a high score of 84.02 on its own. Pfam provides a moderately strong signal, scoring 74.90. In contrast, ProTrek alone is the least informative component, with a score of 66.44, suggesting its raw output may be noisy or less directly useful for functional queries.

##### Synergistic and Antagonistic Effects in Combinations

Combining Pfam and GO yields a score of 84.60, a slight improvement over GO alone, indicating a positive, synergistic relationship where Pfam provides complementary information. However, a critical observation arises when combining components with ProTrek. Both ‘Pfam + ProTrek‘ (77.00) and ‘GO + ProTrek‘ (77.78) perform worse than their stronger counterparts (Pfam and GO, respectively) alone. This trend is amplified when all three are combined unconditionally (‘Pfam + GO + ProTrek‘), resulting in a score of 81.56, which is substantially lower than ‘Pfam + GO‘. This strongly suggests that naively adding ProTrek’s information introduces noise that dilutes the high-quality signals from Pfam and GO, ultimately degrading the model’s performance.

##### Justification for the Conditional Strategy

Based on this insight, we implemented our final, conditional strategy: ProTrek information is only included as a fallback when both Pfam and GO annotations are unavailable for a given protein. This intelligent inclusion prevents ProTrek from interfering with higher-quality data while still providing a baseline of information for sparsely annotated proteins. As shown in the final row of Table [11](https://arxiv.org/html/2510.23127v2#S11), this conditional approach achieves the highest score of 84.99. It effectively captures the synergy of Pfam and GO while mitigating the negative, noisy impact of ProTrek, thus justifying its selection as our final methodology.

## 12 Impact of Semantic Alignment on Mutation Sensitivity

A critical capability for any protein model is the ability to detect and represent the effects of small sequence variations, such as point mutations. To investigate how the internal mechanisms of sequence-as-modality models like Evolla handle such changes, we conducted an analysis on the feature representations before and after its Q-Former alignment module.

We introduced a series of mutations (from 1 to 4 differing sites) into a sample protein sequence. We then extracted the resulting feature embeddings at two key stages: (1) directly from the SaProt protein encoder, and (2) after they had been processed by the Q-Former. The difference between the pre-mutation (wild-type) and post-mutation embeddings at each stage was then visualized and quantified. The results are presented in Figure [8](https://arxiv.org/html/2510.23127v2#S12.F8).

Figure: Figure 8: Effect of mutations on internal representations of Evolla. The top row shows feature differences from the SaProt encoder, and the bottom row from the Q-Former. Heatmaps visualize the difference vector (‘mutated - original‘). SaProt’s representation is clearly perturbed by mutations, showing localized and significant changes. In contrast, the Q-Former’s output shows almost no change, indicating that the alignment process erases the fine-grained signal of the mutation.
Refer to caption: x10.png

##### High Sensitivity at the Protein Encoder Stage

As expected, the SaProt protein encoder is highly sensitive to sequence mutations. The visualizations (Figure [8](https://arxiv.org/html/2510.23127v2#S12.F8), top row) show clear, localized changes in the feature map corresponding to the mutation sites. Quantitatively, while the cosine similarities between pre- and post-mutation embeddings remain high (0.980–0.995), the Euclidean distances are substantial (ranging from 13.3 to 26.2). This confirms that the encoder accurately captures the perturbation caused by the mutation, altering the feature vector in a meaningful way. This sensitivity is the foundation required for any downstream mutation effect analysis.

##### Loss of Sensitivity After Q-Former Alignment

A starkly different picture emerges after the features pass through the Q-Former. The difference heatmaps (Figure [8](https://arxiv.org/html/2510.23127v2#S12.F8), bottom row) are almost entirely uniform, indicating a negligible change between the wild-type and mutated representations. This visual observation is confirmed by the quantitative metrics: the cosine similarities are nearly perfect (approaching 1.0, e.g., $>0.9999$), and the Euclidean distances (5.8–9.9) are significantly smaller than those observed from the encoder.

##### Implications for Downstream Tasks

This analysis reveals a critical limitation of the sequence-as-modality paradigm employed by models like Evolla. The Q-Former, in its role of compressing and aligning the detailed protein features into a fixed set of tokens for the language model, effectively “smooths out” or discards the fine-grained information related to single point mutations. While this may be sufficient for generating high-level functional descriptions, it renders the final representation insensitive to the subtle yet critical differences that underpin tasks like mutation effect prediction, disease variant analysis, and protein engineering. This inherent loss of information at the alignment stage explains why such architectures are fundamentally ill-suited for these precision tasks.

## 13 Generalizability Across Biomolecular Types

To demonstrate the broad applicability and robustness of our context-driven methodology, we evaluated its performance on standard benchmarks beyond our primary QA dataset. This tests the approach on different tasks and different biomolecular types.

### 13.1 Performance on Protein Function Predication

We first evaluate our approach on the protein classification tasks from the Mol-Instruction benchmark [13]. This benchmark contains curated datasets for assessing molecular understanding. We specifically tested on three functionally distinct protein subsets: Catalytic Activity, General Function, and Protein Function. The performance is shown in Figure [9](https://arxiv.org/html/2510.23127v2#S13.F9).

Figure: Figure 9: Performance on Mol-Instruction Protein Classification. Score distributions of our context-driven method on three sub-datasets. The results show consistently high performance across all categories, with mean scores above 85 and the vast majority of answers ($>84\%$) falling into the highest score bracket ($81-100$), demonstrating robust generalization.
Refer to caption: x11.png

As illustrated in Figure [9](https://arxiv.org/html/2510.23127v2#S13.F9), our method achieves excellent performance across all three predication tasks. The mean score were exceptionally high: 91.2 for Protein Function, 90.2 for Catalytic Activity, and 85.5 for General Function. The score distributions are heavily skewed towards the maximum, with over 84% of answers in all three tasks receiving a score in the 81-100 range. This demonstrates that our method is not only effective on our QA benchmark but also generalizes robustly to standard, multi-category protein function predication tasks, validating its broad utility.

### 13.2 Performance on DNA Mutation Prediction

Figure: Figure 10: Model performance on the KEGG DNA dataset under different input configurations. Both Qwen3-4B and GPT-4o were evaluated across context-only, sequence-only, and combined inputs, with both models showing consistent improvements when using context-based inputs compared to sequence-based approaches.
Refer to caption: x12.png

We evaluated two powerful generalist LLMs, GPT-4o and Qwen3-4B, across our three standard input configurations: Context-Only, Sequence + Context, and Sequence-Only. To provide a comprehensive view of performance, we measured not only classification accuracy but also F1 Score, Precision, and Recall, accounting for potential class imbalances in the dataset. The results, presented in Figure [10](https://arxiv.org/html/2510.23127v2#S13.F10), strongly corroborate our findings from the protein-based tasks and confirm the paradigm’s generalizability. For both models, the Context-Only configuration consistently achieved the highest scores across all four evaluation metrics. Crucially, the “informational noise” effect of raw sequences persists in the DNA domain. The Sequence + Context configuration consistently underperformed the Context-Only setup, indicating that the models were again “lost in tokenization,” struggling to integrate the low-level signal from the raw DNA sequence with the clear, high-level context. The Sequence-Only approach yielded the poorest results, confirming that atomic tokenization of nucleotide sequences is insufficient for complex biological reasoning.

## 14 Independence from Clustering Metrics

To verify that our conclusions are robust and not contingent on a single definition of protein families, we evaluated all embeddings against two distinct ground-truth labeling schemes: UniClust50 and UniClust30. These standards group proteins at 50% and a stricter 30% sequence identity threshold, respectively, providing different granularities for functional classification. While both are generated using the MMseqs2 algorithm, they represent conceptually different criteria for defining protein homology. We performed the same hierarchical clustering analysis for each representation against both sets of labels. The resulting ARI scores are presented in Table [3](https://arxiv.org/html/2510.23127v2#S14.T3).

**Table 3: Performance (ARI) against ground-truth labels from UniClust50 and UniClust30.**
| Model / Representation Stage | ARI (vs. UniClust50) | ARI (vs. UniClust30) |
| --- | --- | --- |
| Ours | 0.958 | 0.958 |
| Evolla Encoder (SaProt) | 0.945 | 0.945 |
| Evolla Alignment (Q-Former) | 0.916 | 0.916 |
| Evolla Decoder (Final) | 0.809 | 0.809 |
| Intern-S1 8B | 0.690 | 0.690 |
| NatureLM | 0.492 | 0.492 |

As shown in Table [3](https://arxiv.org/html/2510.23127v2#S14.T3), the performance hierarchy and the specific ARI scores remain identical across both labeling schemes. While the identical results suggest that our test set’s structure is stable across these two identity thresholds, the key insight is the unwavering consistency of our central observations. Specifically, three conclusions hold firm regardless of the ground-truth definition:

- •
Our context-driven approach consistently achieves the highest functional separation (ARI 0.958).
- •
The ’semantic misalignment’ within Evolla is consistently demonstrated by the progressive drop in ARI from 0.945 to 0.809.
- •
The ’weak representation’ of sequence-to-language models (Intern-S1 and NatureLM) is consistently evident from their significantly lower scores.

This stability across different conceptual standards for protein families validates our findings, confirming that the tokenization dilemma and the superiority of our context-driven solution are fundamental phenomena, not artifacts of a particular evaluation metric.

## 15 QA examples

To provide concrete illustrations of the performance differences reported in the main paper, this section presents qualitative examples from our benchmark. We analyze the outputs for a single protein (UniProt ID: A6LHQ9) across the three input modalities: Context-Only, Sequence-Only, and Sequence + Context. These case studies offer tangible insights into why our context-driven approach excels and how raw sequence inputs can be detrimental.

### 15.1 Comparative Analysis on Protein A6LHQ9

The following examples use the same protein and question (“What is the function of this protein?”) to demonstrate how the input modality directly impacts the model’s reasoning and final output. The ground truth is: “Putative component of the fimbrium tip. Fimbriae are filamentous appendages on the cell surface that mediate cell adhesion and biofilm formation”.

#### 15.1.1 Case 1: Context-Only (Success)

In the Context-Only setting, the model receives curated information from Pfam. The model demonstrates a strong ability to synthesize this structured knowledge into a comprehensive and accurate answer, achieving a perfect score.

#### 15.1.2 Case 2: Sequence-Only (Total Failure)

When provided with only the raw amino acid sequence, the model fails completely. It hallucinates a function (Chitin-Binding Module) that is entirely incorrect, likely by misinterpreting spurious patterns in the tokenized sequence. This is a stark example of the “lost in tokenization” phenomenon.

#### 15.1.3 Case 3: Sequence + Context (Noise Effect)

In the combined setting, the model produces a correct but slightly less focused answer than the Context-Only version. The inclusion of the sequence acts as informational noise, causing a minor degradation in answer quality and a corresponding drop in the score. This supports our quantitative finding that the sequence is not merely unhelpful but can be actively detrimental.

The comparative analysis of protein A6LHQ9 clearly illustrates the core tenets of our findings: context is the primary driver of accurate biological reasoning, while raw sequence data can lead to catastrophic failures or introduce performance-degrading noise.

### 15.2 Showcase of Our Method’s Success Cases

Having established the superiority of the context-driven paradigm, we now showcase its robustness and versatility across the three primary query types in our benchmark: molecular function, metabolic pathway, and subcellular localization. The following examples demonstrate the model’s ability to consistently generate high-quality, accurate, and detailed answers for each category.

#### 15.2.1 Example 1: Function Prediction

#### 15.2.2 Example 2: Pathway Prediction

#### 15.2.3 Example 3: Subcellular Location Prediction

## 16 Limitations

While our context-driven approach demonstrates impressive performance in various bioinformatics tasks, there are still significant challenges, particularly in predicting the effects of protein mutations. One of the primary difficulties is that InterProScan and BLAST—two key bioinformatics tools that we use for generating functional context—are not sensitive to subtle mutations at a single or few amino acid positions. This is because both tools rely on sequence-level comparisons, and mutations that alter a few amino acids may not result in any meaningful changes in the overall sequence context, making the corresponding Pfam domains or GO terms identical before and after mutation.

Thus, when we apply our method to predicting the effects of mutations, we encounter a situation where the context generated for the wild-type and mutated proteins is essentially the same. This leads to the limitation that our approach, at present, cannot effectively predict changes in the protein’s function or characteristics due to small mutations.

Below are two examples demonstrating this limitation: one shows the context for a wild-type protein, and the other for a mutated version of the same protein. The only difference between the two sequences is the mutation at two amino acid positions, which we have highlighted in red. For clarity and ease of comparison, we have provided only the Pfam domain and GO annotations (numbers) rather than the complete context, which would otherwise be too long to display for these examples.

### 16.1 Wild-Type Protein Example

### 16.2 Mutated Protein Example

In both cases, the context generated for the wild-type and mutated proteins is identical, as InterProScan and BLAST fail to identify significant differences caused by mutations at amino acid positions. This limitation underscores the challenge of predicting the effects of mutations based on sequence context alone, as even small changes in the sequence can have profound effects on protein function that are not captured by these tools.

### 16.3 Summary of the Limitation

The core limitation of our current method is its inability to distinguish the functional consequences of amino acid mutations, due to the lack of sensitivity of InterProScan and BLAST to small sequence changes. This highlights the complexity of mutation prediction tasks, where even subtle changes to the sequence can lead to vastly different biological outcomes.

## 17 Comparison of Model Performance on the Same Protein and Question

In this section, we compare the performance of different models on the task of predicting the function of the same protein, specifically protein A7YWC4, using the question: “What is the function of this protein” The ground truth for this protein is as follows:

Below, we compare the answers generated by our method (using context), Interns1, Evolla, and NatureLM.

### 17.1 Analysis of Model Differences

Our Method provided the most accurate and complete functional analysis, correctly identifying the protein as ATAD3 and detailing its involvement in mitochondrial network organization, metabolism, and cell growth, which closely aligns with the ground truth. The score of 100 reflects the comprehensive nature of the answer and the high alignment with the expected protein function.

Interns1, relying solely on sequence input, inferred the protein to be an ATPase, which is a reasonable prediction given the presence of specific ATP-binding motifs. However, it failed to identify the specific protein (ATAD3) and did not connect its functions to the mitochondrial network organization, leading to a much lower score of 30.

Evolla also struggled with a correct protein identification, suggesting a mitochondrial import receptor for the endoplasmic reticulum (MIA40), which does not match the true function of ATAD3. This error resulted in a score of 20.

NatureLM provided a very generic answer, linking the protein to RNA binding and mRNA transport, which is not at all related to the actual function of ATAD3. This misinterpretation also earned a score of 20.

In conclusion, while all models gave some plausible biological functions, none of them fully captured the detailed and specific roles of ATAD3 within the mitochondrial network, as outlined in the ground truth. Our method, leveraging context, was able to provide the most accurate and thorough analysis of the protein’s function, demonstrating the advantage of context-driven approaches over sequence-based models in protein functional prediction tasks.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Given the issue with mutations, do you have ideas for how this context-driven method could be adapted for those tasks? Maybe by using different tools that are sensitive to mutations to generate the context?

You mention your method is efficient because it avoids retraining, but running tools like InterProScan and BLAST for every query isn't free. How does the real-world inference time/cost of your pipeline compare to running a big, end-to-end model?
soundness: 3
strengths: The paper's "tokenization dilemma" concept is a really clear and smart way to frame a major hurdle for Sci-LLMs. The main idea—that feeding LLMs text context from tools like BLAST is better than giving them the raw sequence—is surprising but backed up well by the experiments. The finding that raw sequences just add "noise" and make things worse is a big deal. The visualizations (like in Figure 3) showing how alignment fails are also very convincing . This work is important because it questions the push for end-to-end models and offers a practical, hybrid alternative.
summary: This paper tackles how Sci-LLMs handle biomolecular sequences. It argues current methods are stuck in a "tokenization dilemma": either they treat sequences as language, breaking up important motifs, or as a separate modality, which creates an alignment gap. The authors propose a "context-driven" approach, skipping raw sequences entirely. Instead, they use bioinformatics tools (BLAST, Pfam) to create a text summary for the LLM . Their experiments show this context-only method works best, and that adding the raw sequence back in actually hurts performance, acting like noise.
weaknesses: The main drawback, which the authors rightly point out, is that this method can't handle mutation effect prediction. The bio-tools (BLAST, etc.) used to create the context just aren't sensitive to tiny, single-point changes, so the context for a normal protein and its mutant look the same . This is a major limitation, as it rules out a big area of computational biology. Also, the claims about it working on DNA are mostly tucked away in the appendix, not fully explored in the main paper.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	How do you address the circular reasoning between the strengths of the approach (incorporating high-quality expert annotations) and using this approach to predict those annotations where they do not yet exist?  Could tool-calling agents solve this rather than building directly into the LLM?  What are the tradeoffs?
2.	Along this line of questioning, does the core contribution put a focus on the LLMs, or are you simply demonstrating that tradition bioinformatics pipelines already solve most of the problems around understanding protein function?
3.	Have these results been validated against human expert annotators?
soundness: 2
strengths: 1.	Overall: The paper and aims to address a novel challenge in the Sci-LLM space, making a case that Sci-LLMs are better served as “reasoning engines over expert knowledge”, rather than pure sequence decoders. While this is noted and there is some evidence that this is the case, it does raise some circular logic around the quality of the annotations derived from the bioinformatics knowledgebases (addressed below in the weaknesses).
2.	Generalizability: The solution in generalizable, with applications ranging from known proteins to “novel” proteins, as well as different biomolecular types.
3.	Practicality: The solution as it is described is practical, as it allows to more easily keep models up to date with new biological knowledge with lower development costs.  (Although it could be argued that most of the effort is derived from maintaining the bioinformatics knowledgebases).
summary: The paper proposes a “paradigm shift” for how Scientific Large Language Models (Sci-LLMs) are trained, leveraging context-centric approaches driven by high-level structured knowledge from bioinformatics tools (e.g., GeneOntology, ProTrek, BLASTp, etc.).  The solution addresses two key tokenization “dilemmas” that have posed challenges on the Sci-LLM space: sequence-as-language and sequence-as-modality.  This approach accounts for multiple levels of language used to describe biomolecular phenomena – from human-encoded knowledge to genetics/evolutionary-encoded knowledge. Strikingly, the context-only approach largely outperforms joint context + raw sequences, suggesting that raw sequences contribute more to information noise.  The contribution suggests that Sci-LLMs don’t necessarily require solving complex biological “language” from scratch but can leverage decades of accumulated biological knowledge contained within structured databases.
weaknesses: 1.	Circular Logic: The approach works well when high-quality annotations exist, yet the solution also exists to propose annotations to fill in knowledge gaps. This counter-intuitively raises a bit of a “Catch 22” scenario.
2.	Core Argument: The basis of the manuscript suggests that there is in fact valuable information encoded within the evolutionary language through sequence tokens, yet the results suggest the opposite – and that human context exclusively drives the value.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: Pros:
- The authors proposed a new “context-only” method, which achieved significantly 
- The context-driven approach achieve good performance.
summary: This paper argues that Scientific Large Language Models face a "tokenization dilemma," struggling to interpret raw biomolecular sequences, which are either broken down into meaningless components or difficult to align with natural language. Through systematic experiments, the authors demonstrate that a "context-only" approach, where models are given high-level, human-readable knowledge from bioinformatics tools (like BLAST or Pfam) , consistently and substantially outperforms models given the raw sequence.
weaknesses: Cons:
- Context-only approach sounds interesting. However, compared with raw biomolecular sequences input, an inevitable con of this approach would be significant information loss (by discarding too many detailed information).
- The capability of this approach is capped by the bioinformatics tools being used, e.g., InterProScan and BLAST.
- As the context-only model relies majority on prior, it may not be a good tool for exploring “novel” findings (which may be out of distribution a bit).
- Why in Table 1, QWEN series of models are not considered, while in Figure 2, for “ours” model, the author choose to use Qwen-embedding. What about the embedding visualization for specialized language models [1] like ESM series


[1] Zheng, Y., Koh, H. Y., Ju, J., Yang, M., May, L. T., Webb, G. I., ... & Church, G. (2025). Large language models for drug discovery and development. Patterns.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Given the high performance of general-purpose LLMs like Deepseek-v3, Gemini2.5 Pro and GPT-5, what steps did you take to ensure that the ground-truth annotations for your test proteins were not present in these models' pre-training data? Could the results be partly explained by memorization rather than reasoning?
- Your approach depends on external tools. Can you provide a qualitative analysis or failure case study for proteins where InterProScan and BLASTp return no or incorrect hits? How does the performance of your method degrade in such "orphan" scenarios, and what are the potential remedies?
- The paper convincingly shows that context is superior for *retrieving* known functional annotations. However, do you believe your paradigm can be extended to tasks that require *discovering* novel functions or reasoning about structure-sequence relationships that are not yet captured in existing databases?
- You note your method is computationally efficient as it avoids Sci-LLM retraining. However, running InterProScan and BLASTp for every query in a real-time application could be costly and slow. Could you comment on the latency and scalability of the full context-generation pipeline compared to a single forward pass of a sequence-as-language and a sequence-as-modality model?
soundness: 3
strengths: - The paper clearly articulates the "tokenization dilemma" as a critical, yet overlooked, bottleneck in Sci-LLMs. The conceptual framing of the two existing paradigms and their respective weaknesses is compelling and well-supported by prior work.
- The central claim—that raw sequences can be detrimental when combined with high-level context—is counter-intuitive and strongly supported by systematic experiments across multiple models (Intern-S1, Evolla, NatureLM, GPT-4o, etc.) and tasks (protein function, pathway, localization, EC prediction). The consistent performance drop in "Sequence + Context" settings is a powerful result.
- The authors evaluate their method on a wide range of benchmarks, including their own reconstructed dataset, temporal splits, and sequence identity-based splits (Easy/Medium/Hard). The inclusion of DNA-based tasks also demonstrates generalizability beyond proteomics.
- The paper goes beyond mere performance comparisons. The layer-wise analysis of Evolla (Section 5.3, Appendix F) convincingly shows how semantic alignment (via Q-Former) erases fine-grained mutation signals, providing a mechanistic explanation for the limitations of the sequence-as-modality approach.
summary: This paper identifies and investigates a fundamental challenge in Scientific Large Language Models (Sci-LLMs) for biomolecular understanding, which the authors term the "tokenization dilemma." They argue that existing paradigms—"sequence-as-language" (tokenizing sequences into atomic units) and "sequence-as-modality" (encoding sequences via specialized encoders)—suffer from weak representation and semantic misalignment, respectively. As a solution, the authors propose a "context-driven" paradigm, which bypasses raw sequence input. Instead, it leverages established bioinformatics tools (e.g., InterProScan, BLASTp) to generate high-level, human-readable textual context (e.g., functional domains, GO terms) that is natively aligned with the LLM's linguistic space.  The authors evaluated three input modes: sequence-only, context-only, and a combination of both. Through extensive empirical evaluation on protein QA, EC number prediction, and DNA mutation tasks, the authors demonstrate that the context-only approach consistently and substantially outperforms all other modes. They find that adding raw sequence information to context often degrades performance, acting as "informational noise."
weaknesses: - The context-driven approach relies heavily on the quality and coverage of external tools (InterProScan, BLAST). While an ablation study is provided (Appendix E), it does not fully explore the performance ceiling—what happens when these tools fail completely on highly novel proteins? The method's performance is inherently tied to the underlying databases' completeness and timeliness.
- The paper equates "biomolecular understanding" primarily with high-level functional annotation (GO terms, pathways). It does not assess whether the model gains *mechanistic* or *structural* insights that might require raw sequence analysis (e.g., predicting the effect of a point mutation). The limitation section (Appendix J) correctly notes this but underscores a fundamental constraint of the proposed paradigm.
- The strong performance of general LLMs (Gemini, GPT) in the context-only setting raises questions about potential memorization of public protein annotations from their vast pre-training corpora. While the authors take care to prevent label leakage in their *context generation*, they do not explicitly audit whether the test proteins' annotations were already in the LLMs' training data.
- The primary metric (LLM-Score) relies on another LLM (DeepSeek-V3) to judge answer quality. While this is a reasonable approach for open-ended QA, it introduces potential biases and lacks the objectivity of exact-match metrics used in tasks like EC prediction.
- Code is not provided in the current submission, providing it would be helpful to make work reproducible.


Decision: Accept (Poster)