Evolutionary Reasoning Does Not Arise in Standard Usage of Protein Language Models
Authors: Yasha Ektefaie
Andrew Shen
Lavik Jain
Maha Farhat
Marinka Zitnik
Venue: NeurIPS 2025 poster ()
Keywords: Protein Language Models
Reasoning
Sequence Modeling
Evolution
Primary area: machine_learning_for_sciences
TL;DR: State-of-the-art protein language models, under standard usage, cannot infer evolutionary relationships so we introduce Phyla, a model explicitly trained for evolutionary reasoning that achieves state-of-the-art phylogenetic performance.

Abstract: Protein language models (PLMs) are often assumed to capture evolutionary information by training on large protein sequence datasets. Yet it remains unclear whether PLMs can reason about evolution—that is, infer evolutionary relationships between sequences. We test this capability by evaluating whether standard PLM usage, frozen or fine-tuned embeddings with distance-based comparison, supports evolutionary reasoning. Existing PLMs consistently fail to recover phylogenetic structure, despite strong performance on sequence-level tasks such as masked-token and contact prediction. We present Phyla, a hybrid state-space and transformer model that jointly processes multiple sequences and is trained using a tree-based objective across 3,000 phylogenies spanning diverse protein families. Phyla outperforms the next-best PLM by 9\% on tree reconstruction and 23\% on taxonomic clustering while remaining alignment- and guide-tree-free. Although classical alignment pipelines achieve higher absolute accuracy, Phyla narrows the gap and achieves markedly lower end-to-end runtime. Applied to real data, Phyla reconstructs biologically accurate clades in the tree of life and resolves genome-scale relationships among Mycobacterium tuberculosis isolates. These findings suggest that, under standard usage, evolutionary reasoning does not reliably emerge from large-scale sequence modeling. Instead, Phyla shows that models trained with phylogenetic supervision can reason about evolution more effectively, offering a biologically grounded path toward evolutionary foundation models.

Evolutionary Reasoning Does Not Arise in Standard Usage of Protein Language Models
Yasha Ektefaie &
Andrew Shen Department of Biomedical Data Science Stanford University Stanford, CA 94305
Eric and Wendy Schmidt Center Broad Institute of Harvard and MIT
Cambridge, MA 02142
yektefaai@booddisst.. tute.
ashen7@stanford
Lavik Jain
Maha Farhat
Department of Biomedical Informatics
Department of Biomedical Informatics
Harvard Medical School
Harvard Medical School
Boston, MA 02115
Boston, MA 02115
lavikjain@college harvard. edu
fahhaffrhattee harvard. edu
Marinka Zitnik
Department of Biomedical Informatics
Harvard Medical School
Boston, MA 02115
marinka@hms harvard edu
Abstract
Protein language models (PLMs) are often assumed to capture evolutionary in- formation by training large protein sequence datasets. Yet it remains unclear whether PLMs can reason about evolution- is, infer evolutionary relation- ships between sequences. We test this capability by evaluating whether standard PLM usage, frozen or fine-tuned embeddings with distance-based comparison, supports evolutionary reasoning. Existing PLMs consistently fail recover phy- logenetic structure, despite strong performance on sequence-level tasks such as masked-token and contact prediction. We present PHYLA, a hybrid stace-space and transformer model that jointly processes multiple sequences and is trained using a brsed objective across 3,000 phylogenies spanning diverse protein families. PHYLA outperforms the next-best PLM by 9% on tree reconstruction and 23% on taxonomic clustering while remaining alignment- and guide-tree-fie Although classical alignment pipelines achieve higher absolute accuracy, PHYLA narrows the gap and achieves markedly lower end-to-end runtime. Applied real data, PIIYLA reconstructs biologically accurate clades in the tree of life and resolves genome-scale relationships among Mycobacterium tuberculovis isolates. These findings suggest that, under standard usage, evolutionary reasoning does not reliably emerge from large-scale sequence modeling. Instead, PHYLA shows that models trained with phylogenetic supervision reason about evolution effectively, offering a biologically grounded path toward evolutionary foundation models.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Protein language models (PLMs) use transformers with masked language or autoregressive self-
supervision to model molecular sequences (Rives al. [021 Lin ct al. 2022; Alley EC
2019
Madani 2023 Notin 2022). PLMs have shown state-of- performance across predictive
(Meier al. 2021 Rives et al.] 2021 Rao et al. 2021] Elnaggar et al. 2021 Alley EC
2019
Rao et al.] 2020) and generative (Lin et al. 2022: Hayes et [ Madani [023]
Ferruz
2022
tasks. At their core, PLMs learn statistical distributions over amino acid sequences, modeling which
residues are likely to appear given their surrounding context. This enables powerful zero-shot predictions of mutational effects by observing how the likelihood of substitution changes under the model. Over time, this paradigm has been framed as a form of "evolutionary modeling," with the assumption that capturing residue-level distributions implicitly reflects evolutionary dynamics. In practice, researchers often quantify these relationships using distance measures- typically Euclidean or cosine distances- between frozen or fine-tuned embeddings Alley EC (2019): Hie et al. (2022):
et al. 22024); Pantolini et al. (2024): Muir et al. (2025).
However, in this work, argue that this paradigm conflates evolutionary modeling with evolutionary reasoning. Evolutionary modeling, as practiced by current PLMs, involves capturing local amino acid distributions to estimate residue likelihoods. In contrast, evolutionary reasoning requires a deeper level of abstraction: given a set of sequences, can model infer the evolutionary relationships that generated them? This includes identifying shared ancestry, relative divergence, and tree-like structure tasks that cannot be solved by distributional modeling alone.
This distinction is critical. Evolutionary reasoning is the foundation of phylogenetic analysis problem that predates modern biology and underlies centuries of effort to reconstruct how life has changed through time. Classical phylogenetics attempts to recover the tree of life Hug (016) from observed sequences, but current PLMs are not evaluated on their ability to do this. They operate largely in isolation, modeling individual sequences without explicit multi-sequence comparison, structure discovery, or relational inference. Beyond phylogeny, important biological insights can be discerned from reasoning across sequences, whether it is determining the impact of protein variant
(Meier et al. 2021: Brandes 2023 et al.) 2023) or annotating functions of poorly characterized
proteins (Nguyen et al. 2024 Avsec 2021 Zvyagin et al. 2022 Queen et al. 2024).
This work poses the following question: Are current PLMs capable of evolutionary reasoning? And if not, how must we reimagine their architecture and training to enable them to do so?
Present Work. To address the limitations of current PLMs in performing evolutionary reasoning. we make several contributions. First, we introduce benchmark for evolutionary reasoning over protein sequences, built from curated datasets containing ground-truth phylogenetic trees derived from established phylogenetic studies and databases. This benchmark enables rigorous evaluation of whether models can infer evolutionary structure from sets of sequences, rather than simply modeling local residue distributions. Second, we evaluate existing PLMs on this benchmark. Using their learned embeddings, we attempt to reconstruct phylogenetic trees and find that these models Lin et al. (202)) Hayes et al. (2024); ESM Team (2024); Nijkamp et al. (2022) Brixi et al. (2025 fail to outperform a simple baseline. Hamming distance, when evaluated under standard usage (frozen or fine-tuned embeddings with distance- based comparison). This analysis highlights fundamental limitation: current PLMS are architecturally and functionally optimized for intra-sequence modeling, not inter-sequence reasoning.
To address this gap, we propose PHYLA, a new model architecture designed for evolutionary reasoning. Phyla is a hybrid state-space and sparsified attention model that processes multiple sequences jointly. alternates between inter-sequence modules, which detect conserved motifs across sequences, and intra-sequence modules, which contextualize these motifs within individual sequences- -allwwes--llloiing the model to reason over sequence sets. We also introduce a novel pretraining objective tailored to phylogenetic structure. Instead of using masked language modeling. PHYLA is trained using a tree loss that teaches the model to embed sequences such that their pairwise distances reflect the correct evolutionary topology.
Despite its compact size (24M parameters), PHYLA outperforms existing PLMs in phylogenetic tree reconstruction and competitive accuracy in mutation effect prediction. On TreeBase- dataset of protein sequences paired with phylogenetic trees from published studies-PHYi.. outperforms much larger models, including ESM-3 (1.4B) and ProGen-XLaree (6.4B), by 8% and 13% respectively,

---

Unaligned protein sequences
Phylogenetic tree construction
Evolutionary embeddings
2 2
3 WWW.LAGGHKKKROAKKKAFFAAAAEEEEE
Phyla
4 FYAGTCLERNKVDM DMIKTYGGRA/HNGPPYYORR
Bacteda Archaes Eukarya
Figure I: PHYLA takes unaligned protein sequences and generates embeddings that reflects relative evolutionary distance between input sequences which we can to construct phylogenetic trees.
based normalized Robinson-Foulds (normRF) distance. However, state-of-the-art accuracy re- mains with traditional phylogenetic algorithms Katoh et al. (2002); Price al. (2010). PIYLLA is substantially faster and narrows this gap. but further advances in model architecture and training objectives will be required to surpass classical methods. On the ProteinGym mutation effect predic- tion benchmark, PHYLA achieves Spearman correlation of 0.64, matching exceeding models with several orders of magnitude more parameters. We further demonstrate PHYLA'S 's capacity for evolutionary reasoning by applying it to reconstruct the tree of life from ribosomal protein sequences spanning Bacteria, Archaea, and Eukarya. Additionally, we use PHYLA to infer whole- moole-genome evolutionary relationships among Mycobacterium tuberculosis isolates. In both cases, the resulting trees deviate from canonical topologies and instead reflect known functional distinctions, suggesting that PHYYLA learns representations aligned with true biological structure. These results position PHYLA as a foundation for a new generation of protein models grounded in evolutionary reasoning rather than sequence-lever pattern recognition alone.
2 Related Work
Protein Language Models (PLMs). State-of-the-ate protein language models include transformer- based models such as ESM2 (Lin et al. 2022), ESMC Team (2024), ProGen (Madani (023).. and Progen2 Nijkamp Nijkanp et al. (202)) that are trained using masked or autoregressive language modeling. These models learn to model the language proteins by learning
co-occurrence of amino acid residues within diverse training set. Other PLMs, such ESM3 (Hayes et al.) (024), model additional data modalities. ESM3 considers structural and functional information addition to the background amino acid sequences. These models have demonstrated good performance intra-sequence reasoning from sequence modeling pre-training tasks, but have not explicitly been trained to perform inter-sequence reasoning between different sequences in the training set.
Alternatives self-attention. Self-attention is the backbone of the transformer but suffers from quadratic scaling with sequence length, making modeling longer protein sequences difficult (Vaswani) al. (2017). The Mamba state-space architecture has been proposed as an alternative backbone architecture for sequence-based foundation models. The architecture builds upon the S4 class of structured state-space models (Gu et al. (2022) by adding a selection mechanism and a hardware- aware parallel algorithm. These advances allow Mamba to model long sequences efficiently. Beyond Mamba, other approaches use similar ideas to extend context length, including Hyena (Poli et al. 2023ab and xLSTM (Beck et al. 2024). More recent work has shown hybrid state-space and translormer architectures allow for long-context modeling while maintaining the advantages of transformers Poli et al. (2023b); Ren et al. (2025).
Bioinformatics approaches to phylogenetic analysis. Traditional tree reconstruction methods for set of input protein sequences consist of generating multiple sequence alignment (MSA) using one of many alignment algorithms. The MAFFT and Clustal Omega alignment algorithms are popular choices for efficient and accurate MSA generation (Katoh et al. (2002) Sievers et al. (201T)). These alignment algorithms align the input sequences by matching the location the most conserved amino acids within the sequences. After generating the MSA, phylogenetic tree is reconstructed using

---

tree reconstruction algorithm, like FastTree and Ireeee (Price et al. (2010); Nguyen et al. (2014)) These algorithms infer the structure of the phylogenetic tree with and without parametric models and usually with various heuristics to generate the most likely phylogenetic tree topology. The primary limitation of tree reconstruction is runtime inefficiency as tree sizes grow. More recent works can be found in the Appendix A.6
3 Problem Definition
Evolutionary Reasoning: Problem Formulation Let {......, .n3 denote set of protein sequences drawn from an unknown evolutionary process that produced true, strictly binary phyloge- netic tree 7 whose leaves correspond bijectively to S. The patristic (evolutionary) distance between sequences and 8j T is denoted (i)o
Our goal is to learn an encoder fo Σ- - Rd that maps each sequence to an embedding space, together with a distance function dø I2d Rd + Rzo, such that the embedding- distances
Dij de (fo(s))) fo(s;))
approximate the true evolutionary distances (iva)(i.
model fe is said to perform evolutionary reasoning on S if there exists a computable mapping (e.g., distance-based tree builder) such that the reconstructed tree
f h D h(prd))
where Dpred = Dij. minimizes a lopology-level loss
Luve ( T dist(T (T))
with dist denoting a structural metric such as normalized Robinson-Foulds (normRF) distance.
In our experiments, we use neighbor joining for /, Euclidean distance for de and the normalized Robinson-Fould distance for Luee Under this formulation, evolutionary reasoning corresponds to learning embeddings and distance functions whose geometry maximally preserves the information contained in the true phylogenetic structure.
Alignment-Free and Freee---re--ree Unlike most prior methods, our approach does not rely on multiple sequence alignment (MSAs are used solely to derive supervision distances, as model inputs) or a predefined guide tree (i.e. a fixed topology to scaffold inference). Removing both introduces new challenges: alignment and tree inference are jointly NP-hard WANG & (1994) Roch (2005), and guide trees offer strong structural priors. By dispensing with both, PHYLA must discover meaningful patterns directly from unaligned sequences sequences-ccppurrnn capturing evolutionary structure without relying handcrafted heuristics.
4 Methods
4.1
PHYLA Model Architecture
To do well on evolutionary reasoning, models must be able to perform good inter- and intra-sequence reasoning. Inter-sequence reasoning requires the model to detect motifs that recur across different sequences, revealing shared evolutionary signatures. Intra-sequence reasoning requires the model to contextualize identified motifs within the context an individual sequence to determine evolutionary divergence from the identified motif.
To meet these requirements, we introduce a hybrid state-space transformer model, PHYLA. The PHYLA architecture alternates between inter-sequence and intra-sequence reasoning blocks. Inter- sequence reasoning is performed using BiMamba layers, a -ffficiieeee state-space module with gated linear recurrence and deep receptive fields, enabling the model to detect long-range evolutionary motifs across sequences [chhff et al. (2024). Intra-sequence reasoning is handled by sparsified attention layers to contextualize these motifs within each sequence. The current 24M- parameter PHYL.A model comprises three inter-sequence blocks, each containing 16 BiMamba layers followed by single sparsified attention layer. Each layer operates with hidden dimension of 256.

---

Phyla
Discantidd
Antiro
Tree
Phyla
C
Intra sequuancn block
Inter- sequence reasoning ing
Intra-
Sparsified Atlention
reasoning
block
block
Inter-
Bidirectional
Context
block
Window
Figure 2: Overview of the PHYLA. (a) PHYLA processes batches of protein sequences sampled from tree, learning embeddings used to reconstruct phylogenies. (b) It alternates inter- and intra-sequence reasoning blocks. (c) Inter-sequence reasoning uses long bidirectional convolutions to extract shared motifs; intra-sequence reasoning applies sparsified attention to maintain per-sequence context.
Unlike most PLMs which operate single sequence at time Lii Linet al. (2022) Hayes et al. (2024) ESM Team (2024); Notin et al. (2023b); Nijkamp et al.) (2022) PHYLA is the first PLM designed process multiple sequences concurrently. This enables to directly compare sequences, identify conserved motifs, and exploit these inter-sequence relationships to infer evolutionary structure at inference time. While prior models have employed hybrid state-space transformer architectures (Poli et al. (2023b)), PHYLA introduces a sparsification mechanism: an attention mask M that constrains each token to attend only tokens within its own sequence, preserving intra-sequence context while enabling efficient multi-sequence processing. Specifically M:
1, if the j-th token within the i-th sequence,
Mi = 0, otherwise.
4.2 PHYLA Model Training
Tree Loss Pretraining. To teach PIIYLA how model input sequences recapitulate evolutionary relationships, we designed a novel tree loss pretraining function. Constructing a phylogenetic tree contains series of decisions in how split sequences based on their relative distance to each other. With this in mind, we use quarter loss to supervise training. The quartet loss encourages the model embed sequences such that inferred distances preserve correct quartet topologies. For a of sequences with predicted pairwise distances D prod E Rn*x and true distances D E R=** we sample set of quartets Q {(i,j, k. E)). For each quartet, we compute three possible pairwise distance sums:
Tislke Dij Dke:
Tsklfe Dia + Dje:
Fiijkk Die Dik:
and convert them into logits using a softmax-like transformation:
logits, q T ([zajike
]mjj]] mean Faklee Fee)))))
where T temperature hyperparameter controlling softness. We compute the cross-entropy loss between these logits and the ground-truie minimum sum (from Dune):
Cquartet
Σ
CrossEntropy (logits arg ag (3]k(AA)))))))))
min
5

---

Quartet loss is the sole pretraining objective used to train PHYLA. In contrast to most protein language models, which rely on masked language modeling (MLM) over amino acids, PHyLa does not use MLM.
Training Details. PHYLA was trained on distances derived from 3,321 high-quality multiple se- quence alignments (MSAs) curated from the OpenProteinSet (Adritzz et al. (2023). Low-quality MSAs were removed using the filtering procedure described in EVE Frazer al. (202I). For each MSA, we compute pairwise distances between sequences by counting the number of matching positions and dividing by the total alignment length. These normalized pairwise distances are used as supervision during training. We used MSA-derived pairwise distances rather than tree- base distances, as experiments showed that training on tree distances did not yield sigmificant performance gains (Appendix A.I.2). Using the precomputed MSAs from OpenProteinSet therefore provided comparable results at substantially lower computational cost. The current 24M parameter model was trained on single 80GB H100 GPU for 3 days with the Adam W optimizer using 10,000 step linear warmup up to a learning rate of le-5 (Loshchilov & Hutter (2019))
Adaptive Batch Size Sampling. We employ an adaptive batch sizing approach to efficiently utilize GPU memory and avoid overfitting t E T at every training step that can fit within the available GPU memory. Next, we randomly sample subtree size n such that 10 < n S [t], where |t] is the number of sequences in t. Finally, we identify how many subtrees of the sampled size |t] can be accommodated within the GPU memory. If the model encounters an out-of-memovy
specific tree topology. We determine the largest subtree
(OOM) error during this process, the subtrees are resampled with both the subtree size and the number of subtrees halved. We empirically determined that PHYL.A can process input lengths up to 213,350 tokens on a 32 GB GPU and up to 302,350 tokens on 48 GB GPU. For other GPU memory sizes, we used linear model to estimate the maximum allowable input length. Given the length of the longest protein in the input, we computed the maximum number of sequences that could fit within the memory limit.
Training Procedure. During training, phylogenetic tree T is sampled, where T consists of N sequences S. Each sequence is tokenized with an alphabet of 23 tokens, corresponding to 20 standard amino acids, a CLS token, mask token, and pad token. The input to PHYLA S with a [CLS] token concatenated in front of cach tokenized sequence, E S: [[CLS] | |2 $3....[CLLSS [C'LSss,, J. The size and number of trees considered in each training step are determined at each training step by the adaptive batch size sampler.
5 Experiments
Datasets. We evaluate PHYLA on range of evolutionary and functional reasoning tasks. For phylogenetic tree reconstruction, we use two held-out datasets: TreeBase, which includes 1,533 curated phylogenetic trees across diverse species (Piel & Tannen (2009)) and TreeFam, which contains 9,586 gene-family trees spanning a wide evolutionary range (Li et al. (2006)). Dataset statisties are provided in Table 3 and Table 4 with further details in Appendix A.I.I To assess taxonomic classification, we use bacterial isolate sequences from the Genome Taxonomy Database (GTDB) (Parks et al. (202T)). We define five classification tasks at different levels of taxonomic granularity: Class, Order, Family, Genus, and Species. These tasks measure the model's abillity to cluster sequences according to hierarchical evolutionary relationships. Together, these seven tasks-tree reconstruction in TreeBase and TreeFam, and taxonomic classification at five levels using GTDB-form our evolutionary reasoning benchmark. To evaluate performance beyond evolutionary structure, we also assess performance on functional prediction using the ProteinGyym benchmark, which consists of 83 protein mutation effect datasets (Notin et al. 2023a) (Appendix A.1.3)
Baselines. We consider four protein language models, one genomic foundation model, six models from the ProteinGym benchmark, and naive Hamming distance baseline. The protein language
models include ESM2, ESM3, ESM C, and ProGen2 (Lin et al.] (2022); Hayes et al. (2024); ESM Team (2024): Nijkamp et al. (2022). ). The genomic foundation model Evo 2 (Brixi et al. (2025)). The six models from the ProteinGym benchmark are ProteinNPT, MSA Transtormer, ESM-Iv. Tranception, TranceptEVE, and DeepSequence (Notin et al. 22023 b): Rao et al. (202T); Meier
et al. (2021): Notin et al. (2022); Riesselman 20; Notin (2022)). ). We also evaluate traditional
phylogenetic pipeline consisting of a multiple sequence alignment constructed via MAFFT
Katoh
ct al. (2002) and then tree constructed via FastTree Price al. (2010).

---

Evaluation setup. We consider three evaluation settings. Tree reconstruction: This setting evaluates the model's 's ability to reconstruct phylogenetic trees given solely the original sequences. For each method, we generated protein embeddings, computed pairwise distances between embeddings, and reconstructed trees using the neighbor-joining algorithm. We evaluate tree reconstruction by comparing the predicted tree to the reference tree using the normalized Robinson- -ouulds metric (Robinson & Foulds (1981). Taxonomic clustering: This setting evaluates the model's s ability to perform fine-grained and coarse-grained classification of sequences. We cluster 500 sequences at each taxonomic level of Class, Order, Family, Genus, and Species and assess the homogeneity of the predicted clusters. Functional prediction: This setting evaluates the model's ability to predict functional labels given solely the original sequences. We assess functional prediction by training a linear probe classifier on the generated embeddings, and also by utilizing the predicted tree structure to assign labels. More details can be found in Appendix A.1.I
5.1 PHYLA can Reason over Protein Sequences
Experimental setup. To assess the ability of PHYLA to reason over sequences, we assess PHYLA's s ability to reconstruct phylogenetic trees the TreeBase and TreeFam datasets. We use the metric of Robinson-Foon -obinson-Foulds distance, or "RF", whereby a larger RF value is equivalent to a larger distance between predicted and reference tree, and can be interpreted as a lower quality predicted tree. The RF metric not invariant to tree size, so we compute the normalized RF, or "normRF", to directly compare the tree reconstruction performance between trees of different sizes. We utilize the ETE3 Toolkit implementation of RF and normRF distance (Jaime Huerta-Cepas & Bork] In addition, we assess PHYYLA's 's ability to cluster sequences across various taxonomic levels. We report the metric of homogeneity across predicted clusters as a measure of the ability to classify sequences. We compare the performance of PHYLA against PLMs (ESM2, ESM3, ESM C, ProGen2), genomic foundation model (Evo 2), and traditional phylogenetics workflow (MAFFT and FastTree) (Lin al. (2022): Hayes et al. (2024) ESM Team (2024); Nijkaapp et al. (2022); Brixi et al. (2025) KKathh
et al. 2002); Price et al. (2010). j. Table [ shows the normRF performance of PHYLA and benchmark
models on the TreeBase and TreeFam datasets.
Results. PHYLA achieves the best performance out of all tested PLMs on both the TreeBase and
TreeFam tree reconstruction benchmarks, outperforming all tested baselines-including models with 12 to 266 times more parameters, such as ESM2 (650M) and ProGen2-XLarge (6.4B) (Table []. On TreeFam, PHYLA reduces normRF by 13.4% compared the next-best model (ESM2 650M). This improvement is not attributable to data overlap with the pretraining corpus (Appendix I.I.I). Fine-tuning ESM2 embeddings with the same tree-based loss did not yield meaningful gains (Ap- pendix A.1.2). and training PHYLA with tree-derived versus MSA-derived supervision produced comparable results (Appendix(A) A.1.2 indicating that performance arises from the model architecture rather than specific training heuristics.
Traditional alignment-based methods remain the accuracy upper bound: the MAFFT + FastTree pipeline achieves normRF of 0.65 on TreeBase and 0.32 on TreeFam, while PIIYLA attains 0.73 and 0.58, rsspectivelyccosiiee
roughly half of this gap. Notably, the classical workflow required roughly 2 hours on TreeBase and 66 hours on TreeFam across multiple CPU nodes, whereas PHYLA completed both in under an hour on a single H100 GPU. On the taxonomy clustering benchmark. PHYLA also achieves the best performance across all taxonomic levels, improving species-level homogeneity by 23.2% over the strongest baseline. Notably, all PLMs underperform compared to the naive Hamming distance baseline on TreeBase, highlighting the challenge of this dataset. These results underscore the effectiveness of incorporating evolutionary structure into PHYLA architecture and training.
5.2 PHYLA Trees Encode Protein Functional Information
Experimental setup. To evaluate the expressivity of the learned embeddings, we use two comple- mentary strategies for PHYLA and the baseline PLMs. For PHYLA, we convert its embeddings into pairwise distances and construct trees using the neighbor-joining algorithm, as in the previous bench- mark. Functional labels from the training sequences are then overlaid onto the resulting tree. Local sequence clusters are identified using TreeCluster (Balaban M 2019), which assigns each evaluation sequence to cluster. The functional label of a test sequence is inferred by averaging the labels of training samples within its cluster, and performance is measured using Spearman rank correlation
7

---

Table 1: Evolutionary reasoning benchmark comparing PHyLa to existing protein language models (PLMs). Tree reconstruction performance on TreeBase and TreeFam is evaluated using normalized Robinson-Foulds distance (normRF), where lower values indicate better agreement with ground truth phylogenies. Clustering performance at the Class, Order, Family, Genus, and Species levels is measured using homogeneity scores, with higher values indicating more accurate unsupervised grouping according to known taxonomic labels. For other clustering metrics see Appendix A.5
Model
TreeBase +
TreeFam +
Class
Order 1
Family
Genus
Species
Hamming Distance
0.75
0.75
MAFFT+Fas(Tree
0.65
0.32
ESM2 (650M)
0.78
0.67
0.64
0.66
0.68
0.71
0.75
ESM2 (3B)
0.79
0.67
0.55
0.56
0.57
0.59
0.67
ESM3 (1.4B)
0.81
0.72
0.61
0.63
0.66
0.67
ESM C (300M)
0.77
0.71
0.57
0.60
0.62
ESM C (600M)
0.80
0.73
0.61
0.66
0.66
0.71
0.75
Evo (7B)
0.84
0.84
0.50
0.54
0.55
0.55
ProGen2-Large (2.7B)
0.77
0.68
0.60
0.65
0.66
0.71
0.75
ProGen2-XX MooGen2-XLare (6.4B)
0.86
0.82
0.52
0.55
0.57
0.61
PHYLA (24M)
0.73
0.58
0.71
0.76
0.87
0.93
0.98
between predicted and true labels. For baseline PLMs, we follow the standard evaluation protocol and train linear probe to predict functional labels directly from their embeddings. Applying this linear-probe approach to PHYLA performed poorly. We utilize the 83 datasets from the ProteinGym evaluation set (Appendix A.I.3). Figure shows the average
(Notin al. (2023a) benchmark Spearman correlation metric with both the dataset size (in number of sequences trained on) and the model size (in parameters) on the 83 ProteinGym evaluation datasets.
Results. PHYLA achieves competitive performance on the ProteinGym benchmark, ranking among the top-performing models across the 83 functional prediction tasks (Figure 3 Despite being trained on 34 times less data and with 27 times fewer parameters than models like ESM2 (650M), PHYT.A achieves comparable Spearman correlation of 0.64. Notably, PHYLA outperforms several larger PLMs. including ESM3 (I.4B) and Tranception, demonstrating that its tree-based representations encode meaningful biological function. These results suggest that the structured embeddings pro- duced by PHYLA not only capture phylogenetic and taxonomic information but also generalize to downstream functional prediction-dee despite minimal model size and data requirements.
5.3 Ablation Analyses
Experimental setup. To understand the effect of the tree loss versus traditional masked language modeling loss, we trained PIIYLA with only masked language modeling (PHYLA-MLM), To probe the effect of PHYLA architecture on performance, we ablated the sparsified attention layers, which the intra-sequence reasoning blocks MHYLL--NAtttention) We then evaluated both ablated PHYLA models on the tree reasoning benchmark and on functional prediction.
pretraining on a masked language modeling (MLM) objective resulted in substantial performance degradation across all tasks: 19.3% increase in normalized Robinson- Foulds (normRF) distance for tree reconstruction (TreeBase and TreeFam), a 16.4% drop in Spearman correlation for functional prediction (ProteinGym), and 5.3% reduction in average taxonomic clustering homogeneity, Similarly, ablating sparsified attention led to even greater degradation: 20.4% increase in normRF, 45.5% drop in functional prediction, and 7.4% reduction in taxonomic clustering. These results highlight that both the tree-based training objective and the sparsified attention mechanism are critical PHYLA's overall performance.
Results. As shown Table 2
5.4 PHyLA application to real world datasets
PHYLA demonstrates promising performance in sequence reasoning. To showcase its capabilities, applied PHYLA to the task of phylogenetic tree construction. The tree of life is fundamental

---

Figure 3: Functional prediction performance ProteinGym with model sizes and dataset sizes. Spearman correlation of existing PLMs and PHYLA on ProteinGym benchmark as a function of pretraining dataset size. Model points are sized based the number of parameters in the model.
Functional Prediction Performance Protein Gym
Model Size (Parameters)
ESM2 (650M)
10M Parameters
100M
16 Parameters
ESM2 (3B)
0.7
MSA Transformer Embeddings*
Phyla
ESM-Iv Embeddings*
Spearman Correlation
Tranception Embeddings*
0.6
TranceptEVE + One-Hot Encodings*
0.5
8
Tranception One-Hot Encodings*
MSA Transformer One-Hot Encodings'
ESM-1v + One-Hot Encodings*
0.4
ESM3 (1.4B)
1030
Number of Sequences Trained On
Table 2: Ablation study on PHYLA's architecture and training objectives. Tree reconstruction is evaluated on TreeBase and TreeFam using normalized Robinson-Foulds distance (normRF), where lower values indicate better alignment with ground truth trees. Functional prediction is assessed on ProteinGym using Spearman correlation (higher is better). Taxonomic clustering is evaluated five hierarchical levels-class, order, family, genus, and species-usine homogeneity scores, with higher values indicating more taxonomically consistent clusters. Additional clustering metrics are reported in Appendix A.4
Model
TreeBase
TreeFam ProteinGym
Class
Order
Family f
Genus 1
Species t
PHYLA-MLM
0.83
0.79
0.55
0.66
0.76
0.80
0.87
0.95
PHYLA-NoAttention
0.86
0.78
0.44
0.69
0.74
0.78
0.82
0.92
PHYLA
0.73
0.58
0.64
0.71
0.76
0.87
0.93
0.98
framework in biology, delineating evolutionary relationships between organisms and serving as an indicator of relative phenotypic traits. Current approaches to constructing the tree of life typically rely on multiple sequence alignments of ribosomal proteins (Hug et al., 2016). We used PHYLA to analyze set of 3,084 phylogenetic sequences, successfully reconstructing the tree of life in just 16 hours, compared the 3,840 hours required by traditional methods (Hug et al. 2016) (Appendix A.2).
In order to evaluate the validity of the reconstructed tree of life, we compare the tree to the multiple sequence alignment and perform manual feature inspection. As shown in Figure [] PHYLA accurately places sequences within their respective domains in the tree of life. PHYLA identifies overlap between certain Archaeal isolates and Bacteria, result consistent with current phylogenetic reasoning. Lokiarchaeota, an Archaeal lineage clustered with Bacteria, known to have mosaic genome with over 30% of its genome derived from Bacteria (Levasseur et al. (017). Within this genus. PHYLA (L.-5) is paraphyletic to Archaea (Figure 4p.. Examination of the multiple sequence alignment of L-45 and L-A with their immediate phylogenetic neighbors revealed that L.-45 harbors a deletion of the S3 ribosomal protein while L-A retains this protein (Figure 4p). The S3 deletion has been noted in
placed Lokiarchaeaota archaeon loki (L-A) paraphyletic to Bacteria while Lokiarchaeota

---

Baddea Eutorpo
Lekarchaeaci archanoe
Sutsllution scored accepadking Sksuret matri
Agyeneet
lonyboairieiir introoor
Lakardheocte ardwasnn lou
Loterchedds db
Artras GBS
Archaes 33
Figure 4: PHYLA created a new placement Lokiarchaca. (a) Lokiarchaeaota archeaon loki (L-A) placed among Bacterial neighbors while Lokiarchaeota 45 8 (L-45) was placed among Archaeal neighbors. (b) Analysis of the multiple sequence aligmment revealed that L-A placed with Bacteria retained conserved S3 ribosomal protein, aligning with its Bacterial neighbors. In contrast, the L-45 placed with Archaea exhibited a deletion of the S3 ribosomal protein, aligning with its Archaeal neighbors.
previous studies of Lokiarchaea genomes Da Cunha al. (2077). Biologically, these differences may relate to adaptation to extreme environments. L-45 was isolated from the bottom of the Arctic Ocean, while L-A was isolated from the Horonobe Underground Research Laboratory (URL) in Japan. In fact, L-A's neighbor, Methylacidiphilum infernorum, is an acidophilic methanotroph originally isolated from geothermal area in New Zealand Hou al. (2008.. This environment shares similarities with the conditions in the URL. where extensive methane metabolism has been Amseroed/Amano
observed Amano al. (0024). This highlights PHYLA's ability discover potentially biologically meaningful evolutionary relationships through probing the output reconstructed tree. We also apply PIYLLA whole-genome sequences of Mycobacterium tuberculosis, aligning over 4 megabases of DNA to infer novel phylogenetic relationships among clinical isolates (Appendix A.3
6 Conclusion
We introduced an evolutionary reasoning benchmark showing that existing PLMs fail to recover phylogenetic structure, revealing a gap between sequence modeling and evolutionary reasoning. To address this, we proposed PHYLA, hybrid state-space transformer trained with a tree-based objective. PHYLA achieves sfateoo-tthe-ar performance on evolutionary reasoning and competitive accuracy on functional prediction, despite using far fewer parameters and data than large PL.Ss.
Traditional tligmment-based methods such MAFFT FastTree remain the accuracy benchmark, but PHYLA narrows this gap while offering reliance on distance-based tree reconstruction, operation in Euclidean space that may underrepresent hierarchical structure, and an evolutionary benchmark that, while practical, cannot fully capture the diversity open-ended nature of protein evolution. As more curated phylogenetic datasets become available, expanding this benchmark is an important next step. Overall, PHYLA shows that evolutionary reasoning does not emerge from standard sqqeecce-modelng paradigms and points toward a new class of models that learn evolution directly.
faster inference. Limitations include
Acknowledgments and Disclosure of Funding
Y.E. is supported by grant T32 HG002295 from the National Human Genome Research Institute, the NSDEG fellowship, and the Eric and Wendy Schmidt Fellowship. A.S. is supported by NSF GRFP grant DGEE2114777
Y.E. and M.Z. gratefully acknowledge the support of NIH R01-HD108794, NSF CAREER 2339524, US DoD PA8702-15-D-00
awards from Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Al- liance with Distinguished Scientists, Sanofi iDEA-iTECH Award, Pfizer Research, Chan Zuckerberg Initiative, John and Virginia Kaneb Fellowship award at Harvard Medical School, Biswas Computa- tional Biology Initiative in partnership with the Milken Institute, and Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. Any opinions, findings, conclusions
10

---

recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders.
Competing Interests
The authors declare no competing interests.
Code Availability Statement
Code to run PHYLA can be found in the project Github
References
Gustaf Ahdritz, Nazim Bouatta, Sachin Kadyan, Lukas Jarosch, Daniel Berenberg, Ian Fisk, An- drew M. Watkins, Stephen Ra, Richard Bonneau, and Mohammed AlQuraishi. Openproteinset: Training data for structural biology scale, 2023. URL URL https::/arxii: org/ abs 2308 05326
Biswas S AlQuraishi M Church GM Alley EC, Khimulya G. Unified rational protein engineer- ing with sequence-based deep representation learning. Nat Methods, 2019. doi: 10.1038/ 941592-019-0598
Stephen Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David Lipman. Basic local alignment search tool. Journal molecular biology, 215(3):403-410, 1990.
Yusuke Amano, Rishabh Sachdeva, Daniel Gittins, Karthik Anantharaman, Shuwei Lei, Laura E. Valentin-Alvarado, Samuel Diamond, Hiroshi Beppu, Teruki Iwatsuki, Ayako Mochizuki, Keiko Miyakawa, Eri Ishii, Hiroshi Murakami, Adam L. Jaffe, Cindy Castelle, Adi Lavy, Yohei Suzuki, and Jillian F. Banfield. Diverse microbiome functions, limited temporal variation and substantial genomic conservation within sedimentary and granite rock deep underground research laboratories. Environmental Microbiome, 19( 190, Dec 2024. doi: 10. 00.1186//47799-024000699999
Agarwal V. Visentin D. et al. Avsec, 2. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methooss 2021. doi: 10. 10010088411392-00--1152222 URL https:tpp:. www. nature. com/ http//rrrcclee//455-221111112110
Mai U Jia X Mirarab Balaban M, Moshiri N. Treecluster: Clustering biological sequences using phylogenetic trees. PLoS ONE. 14(8):e0221068, 2019. doi: 10.1371/journal.po wwt.:ppp........... 0221066.
Maximilian Beck, Korbinian Pöppel. Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, 2024. URL /tLL:tttp://aaxxv..or/// /400 2405. 224400055...77.
Goldman G. Wang C.H. et al. Brandes, N. Genome-wide prediction of disease variant effects with deep protein language model. Nature Genetics, 2023, doi: 10. 10300003111150 588-123--8888011888 023-01465-0. URL
020-5588-----------0000000-- hattps:/wwwt nature. aom/articles arr.:cceee//1111-0002-----00
Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A. Gonzalez, Samuel H. King, David B. Li, Aditi T. Merchant, Mohsen Naghipour- far, Eric Nguyen, Chiara Ricci-Tam, David W. Romero, Gwanggyu Sun, Ali Taghibakshi, Anton Vorontsov, Brandon Yang, Myra Deng, Liv Gorton, Nam Nguyen, Nicholas K. Wang, Etowah Adams, Stephen A. Baccus, Steven Dillmann, Stefano Ermon, Daniel Guo, Rajesh Ilango, Ken Janik, Amy X. Lu, Reshma Mehta, Mohammad R.K. Mofrad, Madelena Y. Ng, Jaspreet Pannu, Christopher Ré, Jonathan C. Schmok, John St. John, Jeremy Sullivan, Kevin Zhu, Greg Zynda, Daniel Balsam, Patrick Collison, Anthony B. Costa, Tina Bernaandz--Buussard Eric Ho, Ming- Yu Liu, Thomas McGrath, Kimberly Powell, Dave P. Burke, Hani Goodarzi, Patrick D. Hsu, and Brian L. Hie. Genome modeling and design across all domains of life with evo 2. bioRxiv. 2025. doi: 10. N0.1100/2025.02 00............1. URL https: h/wp:::://www... www.riii.iivv........ www.bbiwwoogo.o......o.o....weee wwt.:r//o/ooeee//////////5///55 02/21/2023 02. 638918
Violette Da Cunha, Morgan Gaia, Daniele Gadelle, Arshan Nasir, and Patrick Forterre. Lokiarchaea are close relatives of euryarchaeota, not bridging the gap between prokaryotes and eukaryotes. PLoS Genet., 13(6):e 1006810 10, June 2017.
11

---

Xian Ding, Zijun Zou, and Charles L. Brooks III. Deciphering protein evolution and fit- ness landscapes with latent space models. Nature Communicaations, 10:5644, 2019. doi: 10. 1210086441467--11-1333333
https://:: http:////ooo.oo111100133333
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language life's 's code through self-supervised deep learning and high performance computing, 2021. https://::// attp//iiii....gabaaaa//
06225
ESM Team. Esm cambrian: Revealing the mysteries of proteins with unsupervised learning, 2024. ai htwp//llggommaaaaaaaeeeeee cambrrin
URL htt.:tttp:://evoluttooaaaaeeeeee http:/a/aalllloo//cmmmmbr...a./.
J. Cheng al. Accurate proteome-wide missense variant effect prediction with alphamissense. Science, 2023. doi: 10. 126/science..ddg4922 URL htttps://wwtt science.
1126/science 4dg7492
Schmidt S. Höcker B. Ferruz, N. Protgpt2 is a deep unsupervised language model for protein design. URL htttps//www nature
Nature Communicattonns, 2022. doi:
com com/ /tt///rccce///1//--222/222 -000 32007-7
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K Min, Kelly P. Brock, Yarin Gal, and Debora Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 2021.
Lorenzo Freschi, Ronald Vargas, Amna Husain, Population structure, biogeography and transmissibility of Mycobacterium tuberculosis. Nature Communicatiions, 12:6099, 2021. doi: 10.1038//414477001--222
URL httpss:s::///o http::///.ooo.r.....10.1100 10..0..013//4144// 021-56777221-12-8----- 26248-1
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022. URL
00396
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry, Raul S. Molina, Neil Thomas, Yousuf Khan, Chetan Mishra, Carolyn Kim, Liam J. Bartie, Matthew Nemeth, Patrick D. Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with language model. bioRxiv, 2024. doi: 10.1101/2024.07 000./22022..7.0116055553 01.600583. URL httph: //wp:::///www.. biorxiv. org/ wctpe/nt/earll///22////////// 02/2024 07. 01. 600583
Brian L. Hie, Kevin K. Yang, and Peter S. Kim. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins. Cell Systems, 13(4):774-285c66 e6,, Apr 2022. ISSN 2405-4712, doi: 1.10066/.cel.222220000000 .003. URL (RL
2022.01.00 01.003
Shaobin Hou, Kira S Makarova, Jimmy H W Saw, Pavel Senin, Benjamin V Ly. Zhemin Zhou, Yan Ren, Jianmei Wang, Michael Galperin, Marina V Omelchenko, Yuri I Wolf, Natalya Yutin, Bugene V Koonin, Matthew B Stott, Bruce W Mountain, Michelle Crowe, Angela V Smirnova, Peter F Dunfield, Lu Feng, Lei Wang, and Maqsudul Alam. Complete genome sequence of the extremely acidophilic methanotroph isolate v4, methylacidiphilum infernorum, representative of the hacterial phylum verrucomicrobia. Biol. Direct, 3:26, July 2008
Baker B. Anantharaman K. et al. Hug, L. A new view of the tree of life. Nature Microbiol- ogy, 2016. doi: 10. 00.00///microbbol2066...88 URL :ostps://vit nature. com/ ham/articles. nmi mmicrobio1201648
Laura A Hug. Brett Baker, Karthik Anantharaman, Christopher T Brown, Alexander Probst, Cindy J Castelle. Cristina N Butterfield, Alex W Hernsdorf, Yuki Amano, Kotaro Ise, Yohey Suzuki, Natasha Dudek, David A Relman, Kari M Finstad, Ronald Amundson, Brian C Thomas, and Jillian F Banfield. A new view of the tree of life. Nat. Microbiol. 1:16048, April 2016.
Francois Serra Jaime Huerta-Cepas and Peer Bork. Ete 3: Reconstruction, analysis and visualization of phylogenomic data. Mol Biol Evol, 2016. doi: 10. 10.1099/molbe///s00046
12

---

Yueyu Jiang, Metin Balaban, Qiyun Zhu, and Siavash Mirarab. Depp: Deep learning enables extending species trees using single genes. Systematic Biology, 72(1):17-34, 04 2022. ISSN 1063-5157. doi: 10. (0.10993/ysshoo/yycc0111 tRI.https://daa. org/ htt.:/.110.///yso///sa.a....m
Kazutaka Katoh, Kazuharu Misawa, Kei-ichi Kuma, and Takashi Miyata. Mafit: a novel method for rapid multiple sequence alignment based on fast fourier transform. Nucleic Acids Research, 30 (14):3059-3066, 07 2002. ISSN 0305-1048. doi: 10.1093/nar/gkf446.. URL httts:/p:://d org/
10. 1wr11999nnar// g3f436
Anthony Levasseur, Vicky Merhej, Emeline Baptiste, Vikas Sharma, Pierre Pontarotti, and Didier Raoult. The rhizome of lokiarchaeota illustrates the mosaicity of archacal genomes. Genome Biol. Evol., 9(0):2263-,263, October 2017.
Heng Li, Avril Coghlan, Jue Ruan, Lachlan James Coin, Jean-Karing Hériché, Lara Osmotherly, Ruiqiang Li, Tao Liu, Zhang Zhang, Lars Bolund, Gane Ka-Shu Wong, Weimou Zheng, Paramvir Dehal, Jun Wang, and Richard Durbin. Treefam: a curated database of phylogenetic trees of animal gene families. Nucleic Acids Research, 34(suppl,) D572 D580, 012006. ISSN0305 1048. : URL httthts:://doi org 10. 1093/ nar /gk////
Zeming Lin. Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel-Zarante Tom Sercu. Sal Candido. al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. //arxiv. owgooog////./.o 1711.051011
URL https
Krause Greene E.R. et al. Madani, A. Large language models generate functional protein sequences across diverse families. Nat Biotechnol 41, 1099-1106, 2023. 10.10388441587702220066688
Aritra Mahapatra and Jayanta Mukherjee. Deephy: A deep learning model to reconstruct phylogenetic tree from unaligned nucleotide sequences. bioRtxiv, 2025. 10. 10.11000/022.03.09..o 6422399.
URL
https://www wttp://xwwwbbooxiv.v.vo. biorxiv. /ttp://nntteeeeaaal//22///////// .ww.2020500000000. 033.09.642222
Maximillian G Marin, Natalia Quinones-Olvera. Christoph Wippel, Mahboobeh Behruznia, Bren- dan M Jeffrey. Michael Harris, Brendon C Mann, Alex Rosenthal, Karen R Jacobson, Robin M Warren, Heng Li, Conor J Meehan, and Maha R Farhat. Pitfalls of bacterial pan-genome anal- ysis approaches: a case study of m. tuberculosis and two less clonal bacterial species. Bioin- formatics, pp. btaf219, 05 2025. ISSN 1367-4811. https://doi.nnn oog 10. http://....rrr1111.100/00ooooon. http///iooooffommaac/////aaaaaaa
URL
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, 2021. 10.1101/2021.00 1...0.0000.... 10w.1.000.40666600 4506488. 2021.07.08 4566444
URL https:tts::// //wp:::://www... www.wwv....ii....mmoo http://wwwwboooooorooooooooo////
Takahiro Mimori and Michiaki Hamada. Geophy: www.eeeereettaaaleeeyyoegeeeeete inference via geometric gradients of tree topologies, 2023. URL Ut..hattss://rravv.orgg// wwwp:ogggbbbbs/aegee 230p://000/3300707755533
Duncan F. Muir, Garrison P. R. Asper, Pascal Notin, Jacob A. Posner, Debora S. Marks, Michael J. Keiser, and Margaux M. Pinney. Eyolutionary-scale enzymology enables exploration of rugged catalytic landscape. Science, (88 1058, 2025. 10.1 126/sciencc..dul1558. URL https 7/www www.:sccence.coo htt.e/ence..rgg/////aa/////e///e 110000...288 1126/ science..cc.cccccc adu1058
Luca Nesterenko, Luc Blassel, Philippe Veber. Bastien Boussau, and Laurent Jacob. Phyloformer: Fast, accurate, and versatile phylogenetic reconstruction with deep neural networks. Molecular Biology and Evolutionn m2s4)mmsa005 03 2025. ISSN 1537-1719. 10.1093/mollev/mssa0011. URL wwwgg/10...0 11t111...1199//9 wttp///mmlee////af005.55
https://doi. org/
Eric Nguyen, Michael Poli, Matthew G. Durrant, Armin W. Thomas, Brian Kang, Jeremy Sullivan, Madelena Ng, Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Bermandez-Boussaae Christopher Ré, Patrick D. Hsu, and Brian L. Hie. Sequence modeling and design from molecular to genome scale with evo. bioRxiv, 2024. 10. 10.1101/204.00..2..5522 02. 27 582234
URL
https://wwar biorxiv. http://oteeetteel///////////////
13

---

Lam-Tung Nguyen, Heiko A. Schmidt, Arndt von Haeseler, and Bui Quang Minh. Iq-tree: A fast and effective stochastic algorithm for estimating maximum-likeline phylogenies. Molecular
Biology and Evolution, (2(1))268-2744 11 2014. ISSN 0737-4038. 10. 10.1033/mllbev//s33300 URL
https://doi.org/ 10. http://11100//mm////e////s/ii httppp////aaaoooo//////oo//////0
Erik Nijkamp, Jeffrey Ruffolo, Eli N. Weinstein, Nikhil Naik, and Ali Madani, Progen2: Exploring the boundaries of protein language models, 2022. URL https://arras org at//abbs/22000 2206. 13517
Dias M. Frazer J. Marchena- Hurttao J. Gomez A. Marks D.S. Gal Y. Notin, P. Tranception: Protein fitness prediction with autoregressive transformers and inference-time retrieval. ICML, 2022.
Pascal Notin, Lood Van Nickerk, Aaron W Kollasch, Daniel Ritter, Yarin Gal, and Debora S. Marks. Trancepteve: Combining family-specific and family-agnostic models of protein sequences for improved fitness prediction. bioRxiv, 2022. 10.1 10.1101/2022.12.777 12.0222............. 12.00//22222.2200.11995555 519495. //www.biorxxt org/ content/early2222/// /ttp0//2222///0/222222/2002
URL https
Pascal Notin, Aaron W. Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Hansen Spinner, Nathan Rollins, Ada Shaw, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Rose Orenbuch, Yarin Gal, and Debora S. Marks. Proteingym: Large-scale benchmarks for protein design and fitness prediction. bioRxiv, 2023a. 10.1101/2023 12.07..12...... 12.00.7777.. 7/www. biorxiv. org/ /tt.eeet//erl///222//2222/////// 20t///3///////223333/3 12. 5770777
URL https
Pascal Notin, Ruben Weitzman, Debora S. Marks, and Yarin Gal.
Proteinnpt: Improving
protein property prediction and design with non-parametric transformers. bioRxiv, 2023b. 10. 10.1101/2023 12.00.5773. URL RtLpsttts://www http///wwwbbbbxx..../////////t.t org/ cott//t/ttettteeeeeeeeee
http://wwww...ooooootttt//aaa/// (oontett/earlly/20/////000
2023. 177.122066.50777
Lorenzo Pantolini, Gabriel Studer, Joana Pereira, Janani Durairaj. Gerardo Tauriello, and Torsten Schwede. Embedding- based alignment: combining protein language models with dynamic pro- gramming aligmment to detect structural similarities in the twilight-zone. Bioinformatics, 40 (1):btad786, 01 2024. ISSN 1367-4811. 10. 10.1093/biinformmate 10.100/biinoomati///aaal....... //doi. 10.
URL https
Donovan Parks, Maria Chuvochina, Christian Rinke, Aaron J Mussig, Pierre-Alain Chaumeil, and Philip Hugenholtz. Gtdb: ongoing census of bacterial and archaeal diversity through phylogenetically consistent, rank normalized and complete genome-based taxonomy. Nucleic
Acids Research, 50(D1):D78--D744 09 2021. ISSN 1048. 10. 10.1093/nar/gkab777. URL https://doi. org/ 10. 1003//00////aa////// gkab776
William R. Pearson. An introduction to sequence similarity ("homology") searching. Current
Protocols in Bioinformatics, Chapter 3:3.1.1-3.1.8. 2013. (10000047725095566003003313
Chan L. Dominus M. J. Ruan Vos A. Piel, W. H. and Tannen. Treebase v. 2: A database of phylogenetic knowledge. e-BioSphere, 2009.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Re. Hyena hierarchy: Towards larger convolutional language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28043-2800 PMLR, 23-29 Jul 2023a. URL https: htttp::/pprocedding mlr prress/v202/pol-1 htmml
Michael Poli, Jue Wang, Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models, 12 2023b. URL htt.:tttt::/giiiii
hata///toethheercopptteeet sstr htwpr/eyerrrpeeenaaaeeaa
Morgan N. Price, Paramvir S. Dehal, and Adam P. Arkin. Fasttree 2 - approximately maximum- likelihood trees for large alignments. PLOS ONE. 103):1-10, 03 2010. 10.137 /jourmll.pone..00000 0009490.
URL http::///ooiii. org / 10. 1371/journal pone 0009490
Owen Queen, Yepeng Huang. Robert Calef, Valentina Giunchiglia, Tianlong Chen, George Dasoulas, LeAnn Tai, Yasha Ektefaie, Ayush Noori, Joseph Brown, et al. Procyon: multimodal foundation model for protein phenotypes. bioRxiv, pp. 2024-12, 2024.
14

---

Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov. and Alexander Rives. Transformer pro- tein language models are unsupervised structure leamerr. bioRxiv, 2020. 10.1101/2020.1..155 htl.https:/ / www. biorxiv. org/ wttp:/ooeeet//arr///2022222 http:///22222/11////220000 12. 15. 422761
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine URL http:::/
Learning Research, pp. 8844-8856, PMLR, 18-24 Jul 2021.
mlr
press/v139/ http://r22iaaaaaa.. html
Liliang Ren, Yang Liu, Yadong Lu, yelong shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. In The Thirleenth International Conference on Learning Reprexentations, 2025. URL https:tpps... htt.:httt////peeeee http:///pp/eeieieeeeeee net/ forum?id-bIlnpWWWe
Ingraham J.B. Marks D.S. Riesselman, A.J. Deep generative models of genetic variation capture the effects of mutations. Nature Method, 2018. 00.1088/41192--08-00133 URL https 018-038---
7/www. www.:e.ntur...m
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 18(15))c2011223111 2021. 10. 20.0073/pmas22112331118 URL
https://www. www.wwwwww.........mm........... pnws wtt./////////////////ea/e http://bbbs/111.1/./p...... 2016239111211191
D.F. Robinson and L.R. Foulds Comparison of phylogenetic trees. Mathematical Biosciences,
53(1):131-147, 1981 ISSN 0025-5564. http:///ooooo////100000000000050 90001////00000000554444999900922 990043-2. URL
S. Roch. A short proof that phylogemettic tree reconstruction by maximum likelihood is hard, 2005. URL htttt://p//xiii wrgg/bbs/att///////0 0504377
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov.
Caduceus: Bi-directional equivariant long-range dna sequence modeling, 2024. URL https
//arxiv. org/abs// 220//2233..0 03234
Fabian Sievers, Andreas Wilm, David Dineen, Toby J Gibson, Kevin Karplus, Weizhong Li, Rodrigo Lopez, Hamish McWilliam, Michael Remmert, Johannes Söding, Julie D Thompson, and Desmond G Higgins. Fast, scalable generation of high-quality protein multiple sequence alignments using clustal omega. Molecular Systems Biology, 7(1):539, 201 https://doi.org o00./.....//O.0oo 10..00//m...001.... URL https://wwe
1038/msb. 2011.
Jason Thorpe, Wipaporn Sawaengdee, Danielle Ward, et al. Multi-platform whole genome se- quencing for tuberculosis clinical and surveillance applications. Scientific Reports. 14:5201, 2024. 10.1008/s41988/00 12--5586--11.. URL httpp::sss//:...oo org/ 10. 10111003///4 024-
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
LUSHENG WANG and TAO JIANG. On the complexity of multiple sequence alignment. Journal of Computational Biology, 1(4):337-348 1994. 10. 10.1089/cmb. 1994.1.333 https:: org/ 10. 108999mbmbmm.. 1994. 337
PMID: 8790475.
Eli Weinstein, Alan Amin, Jonathan Frazer, and Debora Marks. Non-identifiabiiliy and the blessings of misspecification in models of molecular fitness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Bel- grave, K. Cho. and A. Oh (eds.), Advances in Neural Informatioon Processing Systems, volume 35, pp. 5484-5497. Curran Associates, Inc., 2022. URL http:/tt/tcccceeeieiiiiieeo iewpeeeeeedinnnenenneeeeeeeeeeee neurips. cc/paper 2ttper//2222/11e/222121 /ttp://22222222e4222222222111111
Papere Conference, pdf
Jacob West-Roberts, Joshua Kravitz, Nishant Jha, Andre Cornman, and Yunha Hwang. Diverse genomic embedding benchmark for functional evaluation across the tree of life. bioRxiv, 2024.
15

---

10.110//202400.10000233.. URL 2024.07.10.60223.
http://wwwoooxxx.rrrrro http:s//www.oorr.....cooootttttr
World Health Organization. Global tuberculosis report 2024, 2024. URL httpss://www.whh.... int/ hta.iaaaiooon//tttte/////2200000
Licence: CC BY-NC-SA 3.0 IGO.
Tianyu Xie and Cheng Zhang, Artree: A deep autoregressive model for phylogenetic inference. In Advances in Neural Information Processing Systems, volume 36, 2023. Spotlight.
Tianyu Xie, Harry Richman, Jiansi Gao, Frederick A. Matsen, and Cheng Zhang. Phylovae: Unsupervised learning of phylogenetic trees via variational autoencoders. In Proceedings of the International Conference on Learning Representations (ICLR), Jan 2025. Accepted Poster.
Maxim Zvyagin, Alexander Brace, Kyle Hippe, Yuntian Deng, Bin Zhang, Cindy Orozco Bohorquez, Austin Clyde, Bharat Kale, Danilo Perez-Rivera, Heng Ma, Carla M. Mann, Michael Irvin, J. Gregory Pauloski, Logan Ward, Valerie Hayot, Murali Emani. Sam Foreman, Zhen Xie, Diangen Lin, Maulik Shukla, Weili Nie. Josh Romero, Christian Dallago, Arash Vahdat, Chaowei Xiao, Thomas Gibbs, lan Foster, James J. Davis, Michael E. Papka. Thomas Brettin, Rick Stevens, Anima Anandkumar, Venkatram Vishwanath, and Arvind Ramanathan. Genslms: Genome-scale language models reveal sars-cov-2 evolutionary dynamics. bioRxiv, 2022. 10.1100//0222.01105511155 URL https 7/www. biorxiv ottp://ocnaeeeeer//////////1///0
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce a new evolutionary reasoning benchmark that exposes a clear limitation of current PLMs. We also introduce new model PHYlA that is explicitly trained to model multiple sequences and performs well on evolutionary reasoning tasks while maintaining strong functional prediction performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss all potential limitations of our work in the Supplemental.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer:
[NA].
Justification:
We do not include any theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include code to run model and all IDs of datasets to train and evaluate our model.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification: We include a link to an anonymized version of our Github repository, which includes code to our model. We also include the IDs of the datasets
train
model from the OpenProteinSet (Ahdritz et al. (2023)) and the datasets we use for evaluation
from TreeBase, TreeFam, GTDB, and ProteinGym (Piel & Tannen 2009) Li al. 2006);
Parks et al. 2021); Notin et al. (203a)).
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips. ec/
gublic/guidess httii//guiiies/oeeuubbiisss
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/nips. //nips.ccppubll. wwtpppliccguuueeeeeeeeeeeeeeeeee wttpiiccggiieeeeeeeeeeeeeeesseee Some/uumeeieeeeeeeeeeeeoe iorrrdeSubiissioonnne ssisssooicciyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all details about hyperparameters used for model training in the Supplemental. We also include details about all evaluation datasets in the Experiments section, with further details in the Supplemental.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Since we perform one run for each of our evaluation results, we do not report error bars to represent standard deviation of the results over multiple runs. We include statistical significance tests of our results in the Supplemental.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include all details on compute resources used to train PHYLA in the Supplemental.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We do involve any human subjects in research, utilize only public datasets that do not require external consent forms, we include open access to code and data, and outline any potential safety concerns of work in the Supplemental.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
20

---

We include details about both positive and negative societal impacts of our work the Supplemental.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The current version of our PHYLA model very small and light- 24M parameters so it does not pose significant risk compared to larger genomic and proteomic foundation models trained in the billions parameters.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include citations for all code, data, and models that were referenced in our work used as inspiration to develop our methods.
Guidelines:
The answer NA means that the paper does not existing assets.
21

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaaaetts,ppersstt.t. paperswi thcode. com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: introduce a new benchmark and PHYLA model our paper, and we document these novel contributions in our anonymized Github.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not utilize any human subjects in our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
22

---

We do not utilize any human subjects in our work, and thus do not require IRB approval.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We do not use LLMs important, original, or non-standard components of the core methods in our work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:e///orrrpscc/oooooeeeeeee// for what should or should not be described.
23

---

A Technical Appendices and Supplementary Material
Table 3: Information about the 1,533 TreeBase evaluation datasets.
Dataset Metric
Min
Mean
Max
Standard Deviation
Average Sequence Length Number of Sequences
5
322
3658
288
53
1284
65
Table 4: Information about the 9,586 TreeFam evaluation datasets
Dataset Metric
Min
Mean
Max
Standard Deviation
Average Sequence Length
40
599
31926
687
Number of Sequences
33
375
34
A.1
Tree reasoning and Protein Gym benchmark
A.1.1
Tree reasoning benchmark
Tree Reconstruction. We evaluate tree reconstruction performance using two external datasets: TreeBase Piel & Tannen (2009) and TreeFam [i al. (2006). TreeBase is a curated repository of 12.817 phylogenetic trees from pecr-reviewed studies, along with associated sequence data contributed by users. After filtering for trees with complete and parseable sequence information, retained 1,533 trees for analysis. TreeFam is database of gene family trees focused on animal genomes. We extracted 9,586 gene trees along with their corresponding protein sequences. To evaluate reconstruction quality, computed the normalized Kobinson-Foulde distance (lower is better) between the predicted and reference trees. We found that PHYLA had the lowest normalized Robinson-Foulds distance among all models. This difference was significant (p 0.05, one-sided 1-test, Table 5) for all models.
We also evaluate the overlap between the training corpus (OpenProteinSett) and the evaluation sets (TreeBase, TreeFam). We ran BLAST Altschul et al. (1990) between each tree in TreeBase and TreeFam and all the sequences in the training corpus using e-value cutoff of le-5 and bitscore cutoff of 50, We found in TreeBase only 0.56% evaluation trees had any match to the training set. The overall mean percent identity of sequences in TreeBase is 0.38%. In TreeFam only 3.38% of evaluation trees had any match to the training set. The overall mean percent identity of sequences in TreeFam is 0.23%. These results indicate minimal leakage and highlights the ability of PHYLA to generalize.
Table 5: P-values from paired t-tests comparing PHYLA with baseline PLMs on normalized Robinson- Foulds scores for TreeBase and TreeFam. Lower values indicate statistically significant improvements.
Baseline Model
TreeBase P-value
TreeFam P-value
ESM2 (650M)
6.78 10-16
1.54 x 10-102
ESMC (300M)
1.69 x 10-9
7.95 x 10-222
ESMC (600M)
4.61 x 10 10-27
7.16 x 10-288
ESM2 (3B)
7.72 x 10-21
3.95 x 10-118
ProGen2-Large (2.7B)
1.37 x 10-12
4.63 x 10-112
ProGen2-XLarge (6.4B)
7.99 x 10 10-99
1.37 X 10 10-308
ESM3 (1.4B)
3.20 x 10 10-40
5.41 X 10 -4-
Evo2 (7B)
4.71 x 10
1.37 x 10-308
Taxonomic Clustering Evaluation. We evaluate taxonomic structure using data from the Genome Taxonomy Database (GTDB), high-quality and frequently updated resource for microbial taxonomy. GTDB taxonomic labels are derived by concatenating multiple sequence alignments of 120 marker
24

---

proteins per organism. We use Release 10-RS226 (released April 16, 2025), which contains 715,230
organisms.
To construct our evaluation dataset, we randomly sample 50 groups of 10 organisms each, ensuring that all organisms within group share the same taxonomic label at either the class, order, family, genus, or species level depending on the benchmark. For each model, we compute organism- level embeddings follows: (1) For baseline models, we embed each of the 120 marker proteins independently, then average the resulting embeddings to produce a single vector per organism. (2) For PHYLA, we batch all 500 organisms for each marker protein (e.g. protein 1) and compute embeddings jointly. We repeat this for all 120 proteins, then average across them to obtain organism- level embeddings. We then perform k-means clustering (as implemented in scikit-learn) with k=50, corresponding to the number of groups.
In the main text, report results for such sample of 500 organisms. However, find that performance is consistent across clustering metrics (completeness, homogeneity, NMI) and across random seeds. We generate four additional random samples and report mean and standard deviation across these replicates. PIIYLA achieves state-of-the-art performance in taxonomic clustering across all taxonomic levels. It is only outperformed by ProGen2-XLarge in completeness the class level (Table 6).
A.1.2 Extended Benchmarking
Fine-tuned ESM2-650M. We fine-tuned the strongest baseline model. ESM2-650M. using two lightweight heads feed-forward network and transformer head-trained with the same quartet loss as PHYLA, while freezing the backbone (as backpropagation through 650M parameters for trees with more than 100 leaves is computationally infeasible). Despite this targeted fine-tuning, the best variant achieved normalized Robinson-Foulde (normRF) scores of 0.77 on TreeBase and 0.79 on TreeFam, whereas PIYYLA trained end-to-end reason jointly over sets of loquennes-achieee 0.73 and 0.58, respectively. These results indicate that the performance gap cannot be attributed solely to evaluating PLMs a zero-shot setting.
Input-Order Robustness. For each evaluation tree in TreeBase and TreeFam, PHYLA processes the full set of sequences in single forward pass. Because the model is trained on randomly sampled and permuted sequence subsets, its predictions are invariant to input order. To verify this property. we randomly shuffled the sequence order for 250 trees from TreeBase five times each and measured the standard deviation in normalized RF scores. The mean deviation was 0.01, confirming that PHYLA's embeddings and resulting distance matrices are effectively deterministic with respect sequence order.
Training signal comparison. To assess whether the choice of supervision signal influences per- formance, we compared two variants of PHYLA trained using (i) normalized Hamming distances computed from multiple sequence alignments (MSAs) and (ii) pairwise patristic distances extracted from trees reconstructed with FastTree on the same MSAs. Both variants achieved nearly identical performance TreeBase and TreeFam benchmarks, with differences below 0.01 in normalized Robinson-Foulds distance (7).le
These results indicate that PHYLA does not simply replicate a specific distance metric but learns generalizable representations that capture consistent topological structure across distance definitions. In practice, Hamming distances serve an efficient proxy for topological supervision without materially affecting model behavior or accuracy.
A.1.3 ProteinGym benchmark
The 83 ProteinGym datasets were chosen based on the memory constraint of single 80GB H100 GPU. We stratified performance on the ProteinGym functional prediction benchmark across various levels of overlap between pre-traine me-trrinnng and evaluation datasets. We quantified overlap by running the Basic Local Aligmment Search Tool (BLAST) algorithm between each representative sequence of the 83 ProteinGym datasets and each model's pre-training dataset (Altschul et al. (1990)). Overlap was calculated as the average percent similarity of output hits. We did not include the total number of hits when calculating the overlap metric in order to not further penalize models with larger pre-training datasets.
25

---

Table 6: Clustering evaluation metrics (mean std) across taxonomic levels. Standard deviation and mean across 5 random taxonomic group samples from GTDB.
Taxonomic Level
Model
Homogeneity
Completeness
NMI
ESM2 (650M)
0,54 0.05
0.64 0.04
0.38 + 0.05
ESM3 (1.4B)
0.67 = 0.01
0.01
ESMC (300M)
0.53 0.10
0.10
0.10
ESMC (600M)
0.02
0.09
0.02
Class
ProGen2-Large (2.7B)
0.67
0.02
0.00
ProGen2-XLarge (6.4B)
0.66
0.75
0.00
Evo2 (7B)
0.02
0.67 0.01
0.62 0.01
PHYLA (24M)
0.69 - 0.02
0.73 # 0.02
0.71 + 0.02
ESM2 (650M)
0.57 0.05
0.68 # 0.04
0.62 0.04
ESM3 (1.4B)
0.75 0.02
0.78
0.76
ESMC (300M)
0.59 +
0.17
0.13
ESMC (600M)
0.48 0.04
0.04
0.04
Family
ProGen2- (2.7B)
0.04
0.01
0.07
ProGen2-XLare (6.4B)
0.07
0.05
0.00
Evo2 (7B)
0.02
0.05
0.67 0.05
PHYLA (24M)
0.85 + 0.02
0.88 # 0.02
0.86 0.02
ESM2 (650M)
0.64
0.74 = 0.04
0.68 - 0.05
ESM3 (1.4B)
0,83
0.85 # 0.02
0.02
ESMC (300M)
0,65
0.11
0.19
ESMC (600M)
0.09
0.06
0.08
Genus
ProGen2- Large (2.7B)
0.06
0.04
0.05
ProGen2-XL.n XLarge (6.4B)
0,68 0.08
0.79
0.07
Evo2 (7B)
0.74 0.06
0.00
0.77
PHYLA (24M)
0.95 + 0.02
0.97 # 0.01
0.96 0.01
ESM2 (650M)
0.57 + 0.03
0.67 + 0.03
0.62 + 0.03
ESM3 (1.4B)
0,72 + 0.03
0.02
0.04
ESMC (300M)
0.58 + 0.10
0.10
0.11
ESMC (600M)
0.00
0.02
0.03
Order
ProGen2-Lan Largee (2.7B)
0.04
0.76
0.04
ProGen2-XLarge (6.4B)
0.03
0.05
0.09
Evo2 (7B)
0.60
0.09
0.04
PHYLA (24M)
0.78 0.03
0.81 = 0.02
4.79
ESM2 (650M)
0.69 0.05
0.79 + 0.04
0.73 0.05
ESM3 (1.4B)
0.85 + 0.03
0.88 # 0.03
0.87 + 0.03
ESMC (300M)
0.73 0.06
0.82
0.77 + 0.04
ESMC (600M)
0.52 - 0.04
0.04
0.60 0.04
Species
ProGen2-Large (2.7B)
0,85 + 0.04
0.03
0.87 0.03
MoGen22-Laaree (6.4B)
0.73 2 0.04
0.77 = 0.03
Evo2 (7B)
0.82 + 0.03
0.87 + 0.03
0.84 0.03
PIIYLA (24M)
0.98 + 0.01
0.99 + 0.00
0.99 + 0.01
When comparing functional prediction performance between PHYLA and ESM2 on all 83 ProteinGym datasets, ESM2 significantly outperforms PHYLA by 0.14. However, on low-overlap regimes with less than 40% similarity to the model's pre-training dataset, PHYLA outperforms ESM2 by 0.03 as shown in Table g (Pearson (2013). Performance degrades on low-overlap regimes for all models, but PHYlA'S ''s performance degrades less. addition, PHYT.A occupies a lower overlap regime than ESM2, which is likely due to the smaller pre-training set used for PHYLA.
26

---

Table 7: Performance comparison when training with Hamming versus tree-based supervision signals.
Training Signal
Tree (normRF)
TreeFam (normRF)
Hamming Distance (original)
0.73
0,58
FastTree Distance
0.73
0,58
Table 8: Functional Prediction Across Various Overlap Settings. Model performance on predicting functional effects in ProteinGyy stratified into low-overlap and high-overlap settings based on alignment overlap.
Model
Low-Overlap High-Overlap All Datasets
ESM2
0.59
0.80
0.78
PHYLA-MLM
0.53
0.61
0.55
PHYLA-NoAAteetion
0.40
0.53
0.44
PHYLA
0.62
0.68
0.64
A.2 Tree of Life analysis
To reconstruct the tree of life, use dataset of 3,083 organisms, each represented by the catenated multiple sequence alignment (MSA) of conserved ribosomal proteins Hug (2016). For input PHYLA. concatenate the amino acid sequences of all 16 ribosomal proteins per organism, yielding sequence lengths of approximately 6 million tokens. We process these sequences using PHyLA on high-memory CPU machine (800 GB RAM), requiring approximately 16 hours of compute time for a full forward pass. We then compute pairwise distances between organism-level embeddings and apply the neighbor-joining algorithm to construct the resulting phylogenetic tree.
A.3 Tuberculosis analysis
Tuberculosis (TB) is one of the leading causes of death globally due infectious disease World Health Organization 2024). A major challenge in controlling TB its capacity to rapidly evolve drug resistance. Genomic surveillance has become a critical tool in addressing this issue: by tracking the evolutionary dynamics of TB, we can detect emerging resistance before it becomes widespread and adapt treatment strategies accordingly Thorpe whole-genome phylogenetic tree of Mycobacterium tuberculosis isolates. Specifically, we analyze 151 complete TB genomes (around 4 Mb each) Marin et al. (2025). To generate organism-level representations, we divide each genome into non non-overlapping 500 bp segments (shared across all 151 genomes), embed each chunk using PHYLA, and then average the resulting embeddings across the genome. This approach is well-suited to TB, which is a highly clonal organism with limited variation across peilaatesmakinn direct genome chunking and alignment across samples both feasible and meaningful Freschi et al, (202T). We compute pairwise distances between the organism-level embeddings and reconstruct the phylogenetic tree using the jeighbor-joining algorithm (Figure
(2024). We use PHYLA to construct the first
PHYLA accurately reconstructs the major global lineages of Mycobacteriumn tuberculosis, recovering coherent clades corresponding Lineages 1-6 and the recently described Lineage Freschi al. (2021). The topology is broadly consistent with established TB phylogenies, with carly-branching Lineage and distinct clusters for the derived East Asian (Lineage and Euro-American (Lineage 4) groups. Deviations are observed in the relative positioning of Lineages 3 and 4, which appear slightly closer than expected. Given that PHYLA infers these relationships directly from sequence embeddings rather than explicit substitution models, the recovery of lineage-level structure without supervision underscores that the learned representation captures genuine evolutionary signal beyond surface sequence similarity.
27

---

10000
1441
09190
0014
637022
1969 11g
-8054
10150
75024
10009004
Dode
R03371
Lineage
Lineage
Lineage
Lineage
Lineage
Lineage
Lineage
Figure 5: PHYI-A-generated tree of whole genome Tuberculosis (TB) isolates. By comparing chunks of the genome of TB across 151 TB isolates, PHYLA constructed the first whole genome tree of TB. Labeled are the main lineages of TB (lineage2, lineage 3, and lineage 4).
A.4 Ablations
We retested our ablations, PHYLA-MLM trained with only the MLM loss and PHYLA-NoAttention trained without sparsified attention, on the extended taxonomic clustering dataset described in Section A.1.I We found across all taxonomic levels, except class, PHYLA had the best performance across all metrics assessed. PHYlA also has significantly better tree reconstmuction TreeBase and TreeFam than all ablations (Table 10
28

---

Table 9; Clustering evaluation metrics (mean + std) for PHYLA variants across taxonomic levels. Standard deviation and mean across random taxonomic group samples from GTDB.
Taxonomic Level
Model
Homogeneity
Completeness
NMI
PHYLA-MLM
0.65 + 0.02
0.69 = 0.02
0.67 = 0.02 0.71 = 0.02
Class
PIYLA-NoAttention
0.69 + 0.02
0.74 = 0.01
PHYLA
0.69 + 0.02
0.73 = 0.02
0.71 = 0.02
PHYLA-MLM
0.78 + 0.02
0.80 0.02
0.79 = 0.02 0.81 = 0.02
Family
PHYLA-NoAttention
0.79 + 0.02
0.83 # 0.02
PHYLA
0,85 = 0.02
0.88 0.02
0.87 = 0.02
PHYLA-MLM
0.89 0.02
0.92 0.02
0.90 = 0.02
Genus
PHYLA-NoAttention
0.85 = 0.03
0.88 # 0.02
0.87 = 0.03
PHYLA
0.95 + 0.02
0.97 = 0.01
0.96 + 0.01
PHYLA-MLM
0.74 + 0.01
0.78 + 0.01
0.76 + 0.01 0.76 + 0.02
Order
PHYLA-NoAttention
0.74 + 0.03
0.79 + 0.02
PIIYLA
0.78 + 0.03
0.81 = 0.02
0.79 # 0.02
PHYLA-MLM
0.94 + 0.01
0.96 = 0.01
0.95 # 0.01
Species
PHYLA-NoAttention
0.92 + 0.01
0.94 + 0.01
0.93 = 0.01
PHYLA
0.98 + 0.01
0.99 + 0.00
0.99 + 0.01
Table 10: P-values from paired t-fests comparing PHYLA with PHYLA ablations normalized Robinson-Foulds scores for TreeBase and TreeFam. Lower p-values indicate stronger statistical significance.
Baseline Model
TreeBase P-value
TreeFam P-value
PUYLA-MMM PHYLA-NoAttention
1.14 x 10-37
1.94 x 10-91
0.0
A.5
Evolution Reasoning Result
Please see Section A.I.1
A.6 Recent Work
Evolutionary reasoning versus evolutionary modeling. Previous studies distinguish between fitting generative model to observed sequence data and reasoning about the underlying fitness landscape that produced those sequences Weinstein al. (2022): Ding al. (2019). From these works, we know generative sequence models aim to match the marginal distribution of extant proteins (evolutionary modeling), whereas evolutionary reasoning seeks to recover the selective pressures and functional constraints driving those distributions.
Al for Phylogenetic Tree Construction. Recent years have seen surge of Al-driven approaches for cvolutionary-bioli problems. For instance, PhyloVAE and ARTrec Xie et al. (205);; Xie & Zhang (0023) generate plausible tree topologies from collections of existing phylogenies. GeoPhy and
Phyloformer Mimori & Hamada (2023): : Nesterenko et al. (2025) take multiple sequence alignment (MSA) as input and infer both tree topology and branch lengths by maximizing the likelihood under fixed substitution model. Other methods such as DEPP and DeePhy DeenhyDiang et al. (2022); |Mahapatra Mukherjee (2025) are designed place unaligned query sequences into a reference phylogenetic tree or into existing triplets of sequences. In contrast, PHYLA focuses on task that combines these capabilities: starting from unaligned sequences alone, simultaneously infers complete tree topology without requiring any prior alignment.
29

---

A.7 Evaluation on Masked Token Prediction
In addition evaluating the ability of PHYLA and benchmark models to perform evolutionary reasoning, we also assess performance on standard PLM task of masked-token prediction. PHYLA trained only with masked-lange languagg modeling attains 30% top-1 accuracy with reduced evolutionary reasoning performance; training with tree loss decreases accuracy to 11% but improves evolutionary reasoning performance (See PHYLA-MMLM performance in 2 for evolutionary reasoning performance). Conventional PLMs (ESM-2 650M, ProGin22 achieve 45-55%. These results confirm trade-off: per- token objectives favor single-sequence reconstruction, whereas the tree loss preserves cross-sequence signals essential for evolutionary reasoning.
A.8 Societal Impacts and Safety Concerns
Our method's ability to generalize beyond its training distribution can lead to more accurate identifi- cation of disease-causing variants in previously unseen protein families, potentially accelerating both basic research and the development of personalized therapeutics. However, because this the first model that infers a phylogenetic tree directly from raw sequence data, there is risk that users may overestimate its reliability and substitute for established, alignment-basee pipelines. Such overre- liance could produce misleading evolutionary hypotheses or clinical interpretations if the model's assumptions and limitations are not carefully considered. To mitigate these dangers, we stress that our approach is intended complement to to-rather than a replacement for traditioaao phylogenetic methods, and that any high-stakes decisions should always include orthogonal validation.
30


Reviews:
[{"review_id": "ad2DwtxcWk", "reviewer": "Reviewer_xecw", "summary": "The paper challenges the assumption that protein language models (PLMs), which are trained on large protein sequence datasets, can inherently perform evolutionary reasoning. While PLMs excel at tasks like masked token prediction, they fail to infer phylogenetic relationships between sequences. To address this, the authors introduce a new model called PHYLA, which combines state-space and transformer architectures and is trained using a tree-based objective across thousands of phylogenies. PHYLA significantly outperforms existing models in reconstructing evolutionary trees and clustering sequences by taxonomy. Unlike traditional PLMs, PHYLA is explicitly designed to reason across multiple sequences, capturing evolutionary structure without relying on sequence alignment. The study concludes that true evolutionary reasoning requires specialized training and architecture, not just large-scale sequence modeling.", "questions": "refer to Strengths And Weaknesses", "limitations": "refer to Strengths And Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper makes a compelling case that current protein language models (PLMs) conflate evolutionary modeling with evolutionary reasoning, and it clearly defines the difference.\n\n2. Introduces the first benchmark specifically designed to evaluate evolutionary reasoning over protein sequences, enabling rigorous assessment.\n\n3. Innovative Architecture: Proposes PHYLA, a hybrid state-space and transformer model that processes multiple sequences jointly—unlike traditional PLMs.\n\n4. PHYLA outperforms much larger models (e.g., ESM3, ProGen2) in phylogenetic tree reconstruction and taxonomic clustering, despite having far fewer parameters (24M).\n\n5. Demonstrates that PHYLA’s embeddings align with known biological and functional structures, including real-world applications like reconstructing the tree of life.\n\n6. Achieves high accuracy without relying on multiple sequence alignment or guide trees, and is computationally efficient compared to traditional methods.\n\nWeaknesses\n1. While PHYLA excels at evolutionary reasoning, its performance on broader protein modeling tasks (e.g., generative tasks) is not deeply explored.\n\n2. The evolutionary reasoning benchmark, though novel, may not capture the full complexity of evolutionary dynamics across all protein families.\n\n3. The paper does not deeply address how interpretable PHYLA’s embeddings are in terms of biological insight beyond tree topology.\n\n4. PHYLA’s success relies on curated phylogenetic trees for training, which may limit its applicability in domains lacking such data.\n\n5.  While outperforming PLMs in evolutionary reasoning, PHYLA’s performance on standard PLM tasks like masked token prediction is not benchmarked.", "was_revised": false, "final_justification": "", "rebuttal": "# Weakness 1\n\nWe thank the reviewer for their suggestion. PHYLA is deliberately positioned as the start of a new class of protein language models, one that prioritises evolutionary reasoning (recovering accurate phylogenetic relationships) over traditional single-sequence objectives such as masked-token recovery. Existing benchmark suites (e.g., EC/GO function prediction, remote-homology classification, generative design) were built for models whose primary signal comes from residue-level co-occurrence patterns within individual sequences. They do not assess whether a model can reconstruct or reason over the evolutionary structure among sequences, which is the central question we address.\n\nTo reflect this shift in emphasis, we introduced the Evolutionary-Reasoning Benchmark and related tasks tailored to multi-sequence inference. We believe the field benefits from recognising this emerging class of evolution-focused PLMs and from developing corresponding evaluation protocols, just as earlier work developed function-oriented tests for classical PLMs. Pre-trained weights and code are public, so future studies can certainly explore hybrid objectives or broader task coverage; however, such experiments would not alter the paper’s key claims.\n\n# Weakness 2\n\nWe agree. At present no dataset, or set of reference trees, exhaustively represents evolutionary dynamics for every protein family. Because evolution is both open-ended and unevenly sampled, any benchmark necessarily covers only a subset of sequence space. Our benchmark is therefore a practical, but incomplete, proxy, an inherent limitation shared with all current phylogenetic evaluations. We will make this explicit in the manuscript’s limitations section, noting that expanding the benchmark as new data become available is an important direction for future work.\n\n# Weakness 3\n\nWe thank the reviewer for highlighting this point. In the Tree-of-Life case study we show that PHYLA’s output, when paired with a standard multiple-sequence alignment (MSA), highlights specific sequence features that drive the inferred topology. In that example, the alignment revealed a protein deletion that separates two clades. This workflow—tree → alignment → feature inspection—mirrors how phylogeneticists routinely move from a topology to biological hypotheses (e.g., gene loss, domain rearrangement) for downstream validation. We will clarify this interpretability pipeline in the results and discussion to make explicit that PHYLA, like traditional trees, serves as a starting point for generating testable biological insights beyond the topology itself.\n\n# Weakness 4\n\nWe thank the reviewer for raising this important point. PHYLA does not require expert-curated trees. To train PHYLA we need high-quality MSAs which can be generated by a slew of existing algorithms. We agree that these algorithms are computationally costly, however given a new domain with new sequencing data, it is possible to construct MSAs. Moreover, at inference time PHYLA operates on unaligned sequences; MSAs are needed only for training. We have also shown PHYLA can generalize to unseen sequences, so given a domain previously lacking sequencing data, PHYLA can be deployed effectively. \n\n# Weakness 5\n\nWe thank the reviewer for pointing out the missing benchmark. We have now evaluated PHYLA on masked-token prediction (MTP) on our training set. PHYLA trained only with MTP attains 30% top-1 accuracy with reduced evolutionary reasoning performance; training with tree loss decreases accuracy to 11% but improves evolutionary-reasoning performance (See Table 2 for evolutionary reasoning performance). Conventional PLMs (ESM-2 650 M, ProGen2) achieve 45–55%. These results confirm a trade-off: per-token objectives favour single-sequence reconstruction, whereas the tree loss preserves cross-sequence signals essential for evolutionary reasoning. We will incorporate this benchmark, training MTP curves, and a discussion in the manuscript."}, {"review_id": "F2WYJYtSIc", "reviewer": "Reviewer_8BeB", "summary": "This work argues that current Protein Language Models (PLMs) are incapable of performing proper phylogenetic analysis. The authors address this limitation by proposing an approach that incorporates inter-sequence elements (in addition to the usual intra-sequence elements) in both the model architecture and the training objective. The authors also propose a benchmark to evaluate the level of phylogenetic analysis that different PLMs are capable of. The proposed approach is compared against different PLMs and the Hamming Distance in this benchmark and functional prediction using the ProteinGym benchmark.", "questions": "- The authors state that PHYLA is the first PLM designed to process multiple sequences concurrently. In the training schema, we see that the loss depends on sequence quartets. Does that mean that PHYLA process always involves 4 proteins? Or does it process more than that? \n\n- The final embedding of a protein depends on the context of other protein sequences. If this is the case, protein embeddings and distances between those embeddings are not deterministic. How are the test sequences fed to the model to obtain the representations and the distances used in the experiments to assess PHYLA? \n\n- Are all the predictions of the competing PLMs obtained similarly to PHYLA? If that is the case, PHYLA was explicitly trained to reflect phylogenetic trees by means of the representation distances. Given that function prediction depends on tree reconstruction, this is a clear disadvantage for the competitors in this task. \n\n- How does PHYLA perform on other standard PLM tasks? (Enzyme Commission Number Prediction, Gene Ontology Term Prediction, Remote homology detection)", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths *\n\n- The idea of informing protein embedding learning by including context from other proteins, using inter-protein modules in this work, is interesting and is shown to enhance PHYLA performance in the presented ablation studies \n\n- Introduction of an interesting benchmark to assess the performance of PLMs at phylogenetic analysis. \n\n*Weaknesses*\n\n- It is interesting to evaluate different widely used PLMs for performing phylogenetic analysis. However, it is unclear how these analyses were made for the different models from reading the manuscript. I suspect that the experiments were based on the distances between protein representations, and if that is the case, there is no reason to assume that those distances should reflect phylogenetic structures.  \n\n- The authors attempt to make a distinction between evolutionary modeling and evolutionary reasoning. I believe that a more formal definition of the latter should be included, given that one of the main objectives of this manuscript is to show that PHYLA, in contrast to other PLMs, is capable of evolutionary reasoning. \n\n- Fairness of valuation is lacking: no comparison with SOTA Phylogenetic models was provided, the experiment's results on clustering and tree reconstruction compare a trained Phyla model with PLMs on a zero-shot task.  \n\n- It's unclear to me if Protein Language Models should be evaluated on their ability to perform phylogenetic analysis without further fine-tuning. \n\n- Lack of reference/explanation on some statements. For example, lines 27 to 30 state that PLMs are thought of as being capable of evolutionary reasoning without offering any reference.\n\n- A clearer explanation of the model architecture is needed. Currently, Figure 2 seems to be the best explanation of the model and should be accompanied by a textual description. As it stands now, this work cannot be reproduced. \n\n- No information on Training data in the main Manuscript. Supplementary material is the first place where it can be found that the training is informed by distances that are derived from an MSA dataset. \n\n- Line 129 claims that PHYLA does not depend on MSA; however, the supplementary material says that it was trained on high-quality multiple sequence alignments (MSAs).", "was_revised": true, "final_justification": "The authors have run experiments including MAFFT + FastTree workflow and a fine-tuned ESM650M, the addition of the discussion of the results provided by the authors of these experiments in addition to a more accurate description of the contribution of they work further convinced me to raise my score. The authors proposal to improve the architecture and experiment descriptions allows for the work to be reproducible. Nonetheless, I agree with reviewer 9GvF that the comparison with the PLMs as is currently frame is not convincing, and that the results obtained in comparison to MAFFT + FastTree workflow are not specially good.", "rebuttal": "# Weakness 1\n\nWe thank the reviewer for pointing out the need to clarify the evaluation protocol.\n\n### How PLM baselines were evaluated\n1. Input: each raw (unaligned) protein sequence is fed through the frozen PLM\n2. Embedding extraction: we extracted protein embeddings as detailed by each PLM\n3. Distance computation: Pairwise Euclidean distance is computed on these vectors\n\nTree scoring: We ran a neighbor-joining algorithm on these distances to construct a tree from which we calculated the norm-RF.\nThis description is in the appendix, but as other reviewers have also pointed out, belongs in the main text and we have moved it there.\n### Why evaluate PLMs this way\nWe agree there is no a‑priori guarantee that these embedding distances encode phylogeny. Yet several influential papers and benchmarks (ESM2 [1], ESM3 [2], DGEB Benchmark [3], ESMC [4], EVO2[5]) argue that PLMs capture evolutionary relationships. Our study rigorously tests that claim and, as the reviewer notes, finds little evidence in support.\n### Contribution clarified\nThe key contribution is therefore two‑fold:\n\n**Empirical:** we show systematically that state‑of‑the‑art PLMs do not recover phylogenetic structure under standard usage.\n\n**Methodological:** we introduce PHYLA, whose quartet loss and architecture enables it to reason over evolutionary splits, achieving state of the art performance on our benchmark\n\nThese results highlight a gap in current protein representation learning and propose a concrete path forward.\n__________________________\n# Weakness 2\n\nWe thank the author for this important comment. We define evolutionary reasoning as the following:\n\nLet $S = \\{s_1, \\ldots, s_n\\}$ be a set of protein sequences drawn from an (unknown) evolutionary process that generated a true, strictly-binary phylogenetic tree $T^\\star$ with leaves in bijection with $S$.\n\nDenote by $d_{\\text{evol}}(i, j)$ the patristic distance between $s_i$ and $s_j$ on $T^\\star$.\n\nA model $f_\\theta : \\Sigma^\\star \\rightarrow \\mathbb{R}^d$ is said to perform evolutionary reasoning on $S$ if there exists a computable mapping $h$ (e.g., a distance-based tree builder) such that the tree\n$$\n\\hat{T} = h(D_{\\text{pred}}), \\quad \\text{where } D_{\\text{pred}}[i, j] = \\|f_\\theta(s_i) - f_\\theta(s_j)\\|,\n$$\nminimizes a tree-level loss $\\mathcal{L}_{\\text{tree}}(\\hat{T}, T^\\star)$.\n\nWhere in our case $h$ is the neighbor joining algorithm and $\\mathcal{L}_{\\text{tree}}$ is the normalized Robinson–Foulds distance. We have added this definition to the problem definition portion of the manuscript.\n__________________________\n# Weakness 3\n\nWe thank the reviewer for raising this point. Our original focus was to test whether general‑purpose PLMs capture phylogenetic structure, but we agree that including a classical pipeline provides an essential upper bound. We have therefore run the standard MAFFT + FastTree workflow on TreeBase and TreeFam. It achieves the best tree‑reconstruction scores (normRF ≈ 0.65 on TreeBase and ≈ 0.32 on TreeFam), while PHYLA attains 0.73 and 0.58 respectively—closing roughly half of the gap that separates PLMs from classical methods. Importantly, the classical run required ~2 hours on TreeBase and ~66 hours on TreeFam across multiple CPU nodes, whereas PHYLA completed both benchmarks in under an hour on a single H100 GPU. These results, now reported in the appendix, clarify the landscape: classical align‑then‑tree pipelines remain the accuracy benchmark, but PHYLA offers a substantial speed advantage and demonstrates that models can approach SOTA accuracy without an explicit alignment step. We believe this addition resolves the fairness concern and sets a clear, measurable target for future improvements.\n__________________________\n# Weakness 4\n\nWe thank the reviewer for raising this important concern. We evaluated PLMs in a zero‑shot setting because a substantial body of work [1, 2, 4, 5] claims that their pretrained embeddings already encode evolutionary relationships. To ensure this choice did not disadvantage the PLMs, we fine‑tuned the strongest model, ESM‑650 M, with two lightweight heads (a FNN head and transformer head) trained using the same quartet loss as PHYLA while freezing the backbone (back‑propagating through 650 M parameters for >100‑leaf trees is infeasible). Despite this targeted correction, the best model reached a normRF 0.77 on TreeBase and 0.79 on TreeFam, whereas PHYLA—trained end‑to‑end to reason over sequence sets—achieves 0.73 and 0.58 respectively. These results confirm that the performance gap is not simply a consequence of evaluating PLMs in a zero‑shot setting. We have added these results to the appendix.\n__________________________\n# Weakness 5\n\nWe thank the reviewer for catching this. We have inserted explicit citations where we state that PLMs are often described as capturing evolutionary signals (lines 27–30,  ESM2 [1], ESM3 [2], DGEB Benchmark [3], ESMC [4], EVO2[5]). We also re‑scanned the manuscript and added citations wherever similar assertions were previously unsupported.\n__________________________\n# Weakness 6\n\nWe thank the reviewer for raising this important point. To meet the page requirement we moved model architecture details to the appendix, but after concerns raised by other reviewers as well we will move it back to the main text. From the appendix:  \n\nThe PHYLA architecture alternates between inter-sequence and intra-sequence reasoning blocks.Inter-sequence reasoning is performed using BiMamba layers, while intra-sequence reasoning is handled by sparsified attention layers. The current 24M-parameter PHYLA model comprises three inter-sequence blocks, each containing 16 BiMamba layers followed by a single sparsified attention layer. Each layer operates with a hidden dimension of 256.\n\nWe believe this satisfies the reviewer’s request and makes the work reproducible without exceeding the page limit.\n__________________________\n# Weakness 7\n\nWe thank the reviewer for pointing this out. We have moved the essential training data description into the main text. The paragraph now states that PHYLA is trained on the 3k‑MSA subset of OpenProteinSet, with supervision coming from normalized Hamming distances computed on those MSAs, while the model itself sees only the raw, unaligned sequences. We believe this resolves the concern without inflating the manuscript.\n__________________________\n# Weakness 8\n\nWe thank the reviewer for flagging the ambiguity. The intended point is that PHYLA’s inputs are always raw, unaligned sequences; it never ingests gap‑padded MSAs at inference time. MSAs are used only to compute target distances (Hamming) during training. We have re‑phrased line 129 to:\n\n“Unlike most prior methods, our approach does not rely on a multiple sequence alignment (MSAs are used solely to derive supervision distances, not as model inputs), …”\n\nThis clarification is also echoed in appendix information pulled into the main text in response to earlier comments.\n__________________________\n# Question 1\n\nWe thank the reviewer for pointing this out. PHYLA processes hundreds of unaligned sequences in a single forward pass. During training we then sample many quartets from that batch and apply the quartet loss, which rewards the model for making the correct topological decisions. This sequence context is enabled by the BiMamba inter‑sequence layers, a capability not present in prior PLMs. The new “Model architecture” and “Training details” paragraphs in the main text (moved from the appendix) now explain this step‑by‑step, so the distinction should be clear.\n__________________________\n# Question 2\n\nWe thank the reviewer for highlighting this. For every evaluation tree (TreeBase or TreeFam) we feed PHYLA the entire set of its sequences in one batch; the sequence order is arbitrary. Re‑ordering the sequences in the batch simply permutes the output rows, and pairwise distances changes minimally. We believe this happens because in training we sample arbitrary sub-trees of any order, so PHYLA in training sees that arbitrary ordering of the same sequences leads to the same quartet loss. We confirmed this empirically by shuffling the sequences of 250 random trees from Treebase fives times and taking the average standard deviation in resulting normRF for each tree and got 0.01. These details and numbers are now noted in the main text and appendix. Thus the embeddings and the resulting distance matrix are effectively deterministic for the purposes of evaluation.\n__________________________\n# Question 3\n\nWe thank the reviewer for highlighting this point. See response to earlier comment with ESM2 (650M) finetuning.\n__________________________\n# Question 4\n\nWe thank the reviewer for their suggestion. Existing benchmark suites (e.g., EC/GO function prediction, remote-homology classification) were built for models whose primary signal comes from residue-level co-occurrence patterns within individual sequences. They do not assess whether a model can reconstruct or reason over the evolutionary structure across sequences, which is the central question we address. To reflect this shift in emphasis, we introduced the Evolutionary-Reasoning Benchmark and related tasks tailored to multi-sequence inference. We believe the field benefits from recognizing this emerging class of evolution-focused PLMs and from developing corresponding evaluation protocols, just as earlier work developed function-oriented tests for classical PLMs. Pre-trained weights and code are public, so future studies can certainly explore hybrid objectives or broader task coverage; however, such experiments would not alter the paper’s key claims and contributions.\n__________________________\n**References**\n\n[1] Lin et al. Science 2023\n\n[2] Hayes et al. Science 2025\n\n[3] West-Roberts et al. BioArxiv 2024\n\n[4] ESM Team, 2024\n\n[5] Brixi et al. BioArxiv 2025"}, {"review_id": "J8FPnYVN1G", "reviewer": "Reviewer_9GvF", "summary": "This work's main contribution is a novel tree-based loss function for (pre-)training PLMs, which is crucially distinct from MLM. The work trains a new multi-sequence model called PHYLA using this loss. Thanks to this loss, PHYLA's latent space better reflects true tree evolutionary distances. Using PHYLA's latent space improves performance on phylogenetic tree reconstruction, compared to using the embeddings of traditional PLMs.", "questions": "1) In the appendix, line 586, why do you state that tree reconstruction could introduce biases during training? You are using the Hamming distance, is that not \"more biased\"? To my understanding, this essentially contradicts the whole body of work on statistical phylogenetics.\n2) Why do you use the Euclidean distance on the embedding space of traditional PLMs? Might fitting a network to \"correct\" potential \"nonlinearities\" in these embeddings make them perform well on the phylogenetic tree reconstruction task? This would put into question the claim that traditional PLMs are not well positioned to perform evolutionary reasoning.\n3) Is it possible that there is test set leakage in the phylogenetic tree reconstruction task? I.e., how do OpenProteinSet and TreeBase/TreeFam overlap?\n4) If PHYLA was trained on aligned sequences, why is there any expectation that it would generalize to unaligned sequences?", "limitations": "As mentioned in my review, many important facts -- specifically, limitations -- are deferred to the appendix and not clearly communicated and discussed upfront in the main text.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Stregths:\n1) The tree-based loss function is interesting and clearly distinct from MLM. It reminds me of contrastive losses which try to  to place closely related objects closer in latent space. These kinds of losses have seen success in other areas such as computer vision.\n\nWeaknesses:\n1) Firstly, the way in which PLMs are used in the tree reconstruction benchmark is by just taking the Euclidean distance in embedding space. This fact deferred to the appendix (line 607) and is quite important: none of these PLMs (e.g. ESM) have been trained to recapitulate true tree distance, so in a sense it is expected that they will not do well on phylogenetic tree reconstruction -- at least not as well as PHYLA, since PHYLA has been trained to recover quartets. There could be non-linear effects on the latent space of traditional PLMs, but if you trained a network to \"correct\" these embeddings such that they recapitulate tree distance (using, for example, your quartet-based tree loss, or by other means such as matching to true tree distances), the corrected embeddings might do just as well as PHYLA on tree reconstruction. I thus find the presented results are unfairly harsh with current PLMs.\n2) Secondly -- and also deferred to the appendix (line 585) -- is the fact that the model is trained using (normalized) Hamming distances instead of true tree distances, or more reallistically, distances obtained from reconstructed tree using software such as FastTree. The manuscript says that \"To create trees we would have to run an algorithm on top of the MSA which could introduce biases while training\". As someone with experience in statistical phylogenetics, I find this statement alarming and reversed: I would argue that using Hamming distances is biased whereas using tree reconstruction software such as FastTree or IQTree will give more accurate and less biased results. For example, Hamming distances disregard the charge of the residues, and in general are a non-linear transformation of true tree distance. Instead, CTMC-based software such as FastTree uses amino acid substitution rate matrices which account for residue charge and attempt to reconstruct true tree distance. Therefore, it appears to me that PHYLA is essentially learning to \"imitate\" the Hamming distance quartet resolution criteria, which seems an uninteresting task (compared to MLM, for example) via a continuous relaxation.\n3) While PHYLA does not require aligned sequences, the fact that it was trained exclusively on MSAs (Section A.1.2: training details) makes me wonder why it would even generalize to unaligned sequences.\n4) I didn't find any discussion on how the training set (OpenProteinSet) overlaps with the test set (TreeBase and TreeFam). If there is overlap, then there is test set leakage, and since PHYLA is tarined to explicitly recover quartets it would be unsurprising that it did well on the tree reconstruction task.", "was_revised": true, "final_justification": "I find the rebuttal of the authors is still not convincing. Moreover, the fact that the authors confirmed that important information was originally hidden in the appendix (specifically, the fact that Euclidean distance in embedding space is used to evaluate PLMs) further decrease my confidence in the submission, so I continue to recommend rejection. I understand that reviewer Ya7G enthusiastically recommends the paper for acceptance but I disagree with them. The reviewer has mentioned in their review as a strength the \"astonishing fact that the existing PLMs are not applicable on evolutionary studies\". Both myself and reviewer 8BeB have found this unsurprising (in fact, trivial) because of how PLMs are not trained to explicitly match tree distances, unlike PHYLA. I disagree with reviewer Ya7G, and am surprised by their high confidence in their score. I think the quartet loss is interesting, but as marketed the paper is making strong claims about the limitations of PLMs which I am still not convinced by, and, to reiterate, the fact that important information was indeed deferred to the appendix reduces my confidence in the quality of the submission. I have maintained my score of 2 and increased my confidence from 3 to 4.\n\nUPDATE: The authors have promised to overhaul their paper to remove their strong claims about the limitations of PLMs. Please see my \"Second Response to Authors\" and the author's final response regarding this. In particular, this involves changing the title of the paper. I agree that with the proposed changes the paper would be of interest, but since the changes with respect to the original submission are so substantial, it is hard for me to confidently recommend acceptance without reading the promised version, so I will maintain my score of 2 recommending rejection, but I am open to discussion.", "rebuttal": "# Weakness 1\n\nWe thank the reviewer for this thoughtful and constructive comment. We agree that it is an important point that the way PLMs are used in the tree reconstruction benchmark is by simply computing Euclidean distances in embedding space (previously only mentioned in the appendix, line 607). We will move this detail to the main text to make the evaluation setting clearer.\nTo address the concern that this setup may be unfairly harsh on current PLMs, we performed an additional experiment with the best-performing PLM on the benchmark, ESM-650M, by training a network on top of its embeddings using the same tree-loss employed by PHYLA. We froze ESM-650M during training, as it was not computationally feasible to store gradients for larger trees containing over 100 proteins; this limitation highlights a key advantage of PHYLA’s architecture, which is designed to jointly process all sequences efficiently.\n\nWe evaluated two variants: (i) a simple feed-forward network (FNN) head and (ii) a transformer-based head over the protein embeddings. In both cases, ESM-650M was unable to match PHYLA’s performance. On the TreeBase benchmark, the corrected ESM-650M models achieved normalized RF scores starting at 0.78 and plateauing at 0.77/0.76, while on the TreeFam benchmark performance degraded to 0.79/0.80. These experiments show that PHYLA’s superior accuracy stems from its architecture’s ability to reason jointly over entire sequence sets, not from the PLMs being evaluated in zero‑shot settings. We have added these results to the appendix.\n\n# Weakness 2\n\nWe thank the reviewer for highlighting this crucial point. Our intent was not to claim that normalized Hamming distance (HD) is less biased than tree‑based algorithms and distances, but rather to avoid injecting the hyper‑parameter choices and heuristic assumptions of a particular tree‑building algorithm into the supervision signal. We have clarified this wording in the paper and explicitly acknowledge that FastTree/IQ‑Tree (along with other algorithms, RaxML, etc) are state‑of‑the‑art in this space.\n\n### Why HD was used originally.\n\n**Definition & clarity:** To be clear, the (normalized) HDs are computed column‑wise on each multiple‑sequence alignment (MSA) not on raw, unaligned sequences.\n\n**Practicality:** MSAs were already computed on the OpenProteinSet and HDs can be obtained from MSAs in O(L × N²) time, whereas building trees even with FastTree with a large number of trees is computationally expensive.\n\n**Focus on topology:** PHYLA’s quartet loss optimizes which branches merge, independent of branch length. FastTree likewise derives its initial topology from the pairwise similarity in the underlying MSA, then refines topology and branch lengths. We therefore treated HD as a proxy for topological signal, not a perfect metric of evolutionary time.\n\n### New experiment requested by the reviewer.\n\nWe reconstructed FastTree trees for all training MSAs, extracted pairwise tree distances, and re‑trained PHYLA with these distances in place of HD. Test results:\n\n| **Training Signal**     | **TreeBase** | **TreeFam** |\n|-------------------------|-------------:|------------:|\n| *HD (original)*         | **0.7269**   | **0.5777**  |\n| *FastTree Distance*     | **0.7312**   | **0.5777**  |\n\n\nThe near‑identical scores indicate that PHYLA is not merely copying HD; it is learning to compare sequences in a way that generalizes across distance definitions.\n\n### Take‑away.\n\nUsing HD does not handicap PHYLA on the evaluated topological tasks, while avoiding dependence on specific FastTree/IQ‑Tree settings. We have added this explanation, the table above, and a note that incorporating richer substitution models for branch‑length prediction is valuable future work.\n\n# Weakness 3 \n\nWe thank the reviewer for raising this point. The current wording in Section A 1.2 is confusing, and we have revised it. To clarify, PHYLA is always fed raw, unaligned amino‑acid sequences. The only role of the multiple‑sequence alignments (MSAs) in training is to provide a target distance matrix—we compute normalized Hamming distances on the MSA, then discard the alignment. Thus the model never “sees” gap symbols and is explicitly trained on unaligned inputs.\n\nBecause the supervision signal is derived from MSAs while the inputs are raw sequences, PHYLA must implicitly learn to align (or otherwise compare) sequences in a differentiable manner to minimise the quartet loss. This built‑in alignment‑free processing is precisely why the model generalises to novel, unaligned sequences at test time.\n\n# Weakness 4\n\nWe thank the reviewer for pointing out the need to quantify any overlap between the training corpus (OpenProteinSet) and the evaluation sets (TreeBase, TreeFam). We ran BLAST between each tree in TreeBase and TreeFam and all the sequences in the training corpus using an e-value cutoff of 1e-5 and bitscore cutoff of 50. We found in TreeBase only 0.56% of evaluation trees had any match to the pretraining set. The overall mean percent identity of sequences in TreeBase is 0.38%. In TreeFam only 3.38% of evaluation trees had any match to the pretraining set. The overall mean percent identity of sequences in TreeFam is 0.23%. These results indicate minimal leakage and highlights Phyla’s ability to generalize. We will add this to the appendix and cite it in the main text.\n\n# Questions\n\nWe thank the reviewer for raising these important questions, we believe our detailed responses above (the new experiments and clarifications) fully resolve these points.\n\n# Limitations\n\nWe thank the reviewer for emphasizing that several limitations were in the appendix. Space constraints forced difficult trade‑offs, but we agree these details belong in the main narrative. In the revised draft we have moved all limitations from the appendix to the discussion and added more training details from the appendix to the main text. These changes make the scope and limitations explicit while respecting the page limit."}, {"review_id": "VFkUNMSFVa", "reviewer": "Reviewer_Ya7G", "summary": "This model proposed an evaluation benchmark to exam the existing PLMs on phylogenetic reconstruction and a new model PHYLA for evolutionary reasoning. The new model rely on a hybrid state-space and attention-based architecture to capture both long-range dependency and local context. It can process multiple sequences in parallel to discover inter and intra sequence relationship. The model is trained on a tree loss function to reflect evolutionary distance and phylogenetic structure in the embedding. \nCompared to existing PLMs, who often failed on phylogenetic reconstruction, PHYLA achieved good accuracy and efficiency on phylogenetic inference on benchmark datasets.\nThis method is effective on evolutionary inference without sequence alignment and guide tree.", "questions": "1. It would be interesting to discuss about why PLMs can’t even beat hamming distance on phylogenetic inference. That means PLMs does not capture basic evolutionary information from the sequences. \n2. How does the model performance compare to traditional phylogenetic inference methods? On both accuracy, running time and memory usage.", "limitations": "Overall, this paper is a solid study on a biologically significant problem, phylogenetic inference. This study not only identified an astonishing fact that the existing PLMs are not applicable on evolutionary studies, but also proposed a solid solution to resolve this problem by introducing both intra and inter sequence inference blocks, tree loss to capture tree information directly from the model. \nThe limitation of this work mainly include leak of traditional phylogenetic inference baselines, model robustness analysis, and theoretical analysis on the model effectiveness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This paper reveals an important question that the evolutionary reasoning is different from sequence modeling. Evaluation on existing PLMs shows that the limitation of large language models on specific biological tasks. The impact is huge for comparative studies.\nThe design of model combines state-space module with sparsified attention mechanism. Effectively capture both intra and inter sequence relationships. The tree loss enables the model to learn evolutionary relationships directly from sequences, which is quite novel and biologically significant.   \nThe model can process multiple sequence semutaniousy, infer phylogenetic tree without multiple sequence alignment and guide tree, greatly improved inference efficiency. \nThe experiment is comprehensive, covers multiple tasks, include multiple benchmark datasets.\nSome weaknesses include that this study does not include traditional phylogenetic tree inference, such as RAxML, Fasttree, etc. It is unclear about the model robustness and effectiveness on datasets with high sequence diversity. It would be helpful to include simulated datasets with artificially designed sequence diversity to simulate sequences with different evolutionary distance.", "was_revised": true, "final_justification": "Thanks for the authors' detailed rebuttal and the new experiments. They have addressed my concerns.\nBesides, I disagree with the comments of reviewer 9GvF. The concerns this reviewer raised was based on a fundamental misunderstanding of the core contribution and misinterpretation of the experiment results. In my point of view, this work is a solid and impactful work that deserved to be accepted. \nRegarding to the problem with Euclidean distance, the authors' entire point is that even with a standard and widely used metric like Euclidean distance, PLMs still fail to produce meaningful evolutionary representations. The failure isn't in the evaluation method but in the model itself. \nReviewer 9GvF also raised another concern that this new method is not good enough because it does not beat the FastTree. This totally missed the point. The author has already included FastTree as a critical baseline to provide a performance upper bound. I would suggest that the author make it clear that their methodology are completely different. Traditional methods like FastTree rely heavily on MSAs, which is extremely computational expensive as tree size grow. The key contribution of this work is developed an efficient and scalable alternative and it is alignment-free. This isn't about simply beating an existing method, it is about introduce a pioneer approach for evolutionary reasoning. \nI felt that the reviewer 9GvF is not familiar with phylogenetic inference background and I totally understand their concern on the performance side. \nI believe the authors have already made a strong and convincing argument. This work is a significant contribution to the field. I recommend for acceptance.", "rebuttal": "# Question 1 \n\nWe thank the reviewer for highlighting this important point. Conventional PLMs are trained with a masked-language-model (MLM) loss on single sequences. That objective optimises per-token reconstruction, not the relationships among sequences, so the embeddings do not preserve evolutionary distance. A simple Hamming count is crude but is monotonically related to divergence, so outperforms these models on tree reconstruction.\n\nOur experiments make this explicit. Swapping PHYLA’s quartet-based tree loss for MLM decreases the ability for PHYLA to do evolutionary reasoning; fine-tuning a head on the 650M-parameter ESM-2 backbone likewise fails to beat Hamming, and removing PHYLA’s multi-sequence attention blocks drops performance dramatically. Scale cannot rescue a model whose objective and architecture is misaligned with the task.\n\nThis result matters for the community. It shows that objective and architecture alignment, not parameter count, is the bottleneck, and it motivates our Evolutionary-Reasoning Benchmark and PHYLA’s design. Publishing these findings will prevent researchers from assuming that large MLM-trained PLMs “implicitly learn evolution” and will guide future work toward loss functions and architectures that capture the phylogenetic signal. PHYLA represents the first step in a new class of evolutionary-focused PLMs with its own unique model architectures, loss functions, and benchmarks.\n\n# Question 2\n\nWe thank the reviewer for raising this point. Our original focus was to test whether general‑purpose PLMs capture phylogenetic structure, but we agree that including a classical pipeline provides an essential upper bound. We have therefore run the standard MAFFT + FastTree workflow on the same TreeBase and TreeFam splits. It achieves the best tree‑reconstruction scores (normRF ≈ 0.65 on TreeBase and ≈ 0.32 on TreeFam), while PHYLA attains 0.73 and 0.58 respectively—closing roughly half of the gap that separates PLMs from classical methods. Importantly, the classical run required ~2 hours on TreeBase and ~66 hours on TreeFam across multiple CPU nodes (30 nodes with 75 GB memory each with  52% usage each), whereas PHYLA produced both trees in under an hour on a single H100 GPU (83% memory usage). These results, now reported in the appendix, clarify the landscape: classical align‑then‑tree pipelines remain the accuracy benchmark, but PHYLA offers a substantial speed advantage and demonstrates that models can approach SOTA accuracy without an explicit alignment step."}]

Author rebuttal:

Meta-review: This paper presents PHYLA, a protein language model–based approach for phylogenetic inference. Rather than the standard masked language modeling loss, the authors introduce a tree-based training objective and evaluate on TreeBase and TreeFam, showing improved performance over PLM baselines.

This was a borderline paper with diverging reviews, largely due to framing. Assertions of state-of-the-art performance should be tempered: during rebuttal, the authors added baselines against classical phylogenetic methods (MAFFT + FastTree), which remain the clear state of the art by a wide margin. The claim of introducing a new benchmark could also be misleading, as TreeBase and TreeFam have been established since the late 2000s. One reviewer noted that the central argument -- that PLMs do not learn embeddings that produce accurate phylogenetic trees -- is unsurprising, and the authors have indicated they will revise their title and claims accordingly. Overall, while concerns about framing and benchmarking remain, the authors have committed to addressing these issues in the revision, and the proposed model represents a novel and promising research direction of phylogenetic inference from protein language models.
Decision: Accept (poster)