Title: Understanding protein function with a multimodal retrieval-augmented foundation model
Venue: NeurIPS 2025
Accepted: yes
Keywords: protein fitness prediction

transformer

retrieval

language model

MSA

generative model

protein engineering
Abstract: Protein language models (PLMs) learn probability distributions over natural protein sequences. By learning from hundreds of millions of natural protein sequences, protein understanding and design capabilities emerge. Recent works have shown that scaling these models improves structure prediction, but does not seem to improve mutation understanding and representation quality for protein function prediction. We introduce PoET-2, a multimodal, retrieval-augmented protein foundation model that incorporates in-context learning of family-specific evolutionary constraints with optional structure conditioning to learn generative distributions over protein sequences. PoET-2 uses a hierarchical transformer encoder that is equivariant to sequence context ordering and a dual decoder architecture with both causal and masked language modeling objectives, allowing PoET-2 to operate in both fully generative and bidirectional representation learning modes. PoET-2 achieves state-of-the-art performance on zero-shot variant effect prediction, excelling at scoring variants with multiple mutations and challenging indel mutations. In supervised settings, PoET-2 embeddings outperform previous methods for learning sequence-function relationships, especially with small datasets. This work highlights the benefits of combining retrieval augmentation with multimodal, family-centric modeling for advancing protein foundation models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents PoET-2, a multimodal, retrieval-augmented protein language model designed to improve protein variant effect prediction. Unlike prior PLMs, PoET-2 integrates both sequence and structure information through a hierarchical encoder and dual decoders optimized for both causal and masked language modeling. It can ingest evolutionary homologs and optionally structural templates to inform its predictions. The model shows strong performance on tasks involving indels and higher-order mutations, surpassing or matching state-of-the-art methods such as VenusREM and PoET-1, especially when ensembling is applied. PoET-2 also demonstrates strong generalization in supervised learning settings, particularly with limited training data, highlighting its utility in practical protein engineering applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- PoET-2 introduces a flexible architecture that incorporates homologous sequences and structures as context for language modeling, enabling more informed and family-specific variant predictions.\n\n- The model is particularly effective for predicting the effects of indels and higher-order mutations, addressing a known limitation in many existing PLMs.\n\nWeakness\n- In the key evaluation benchmark (zero-shot DMS substitution), PoET-2 underperforms VenusREM on the main metric (Spearman correlation). While PoET-2 outperforms other baselines, its results are reported as ensemble averages over multiple prompts and context lengths, whereas baselines are not ensembled. It is unclear how much of the performance gain is attributable to architectural advantages versus ensembling.\n\n- The model relies heavily on retrieval of homologous sequences or structures, which may not be available for orphan proteins or rapidly evolving families such as antibodies. The model’s performance on proteins with rare homologs should be reported.\n\n- The paper does not provide a clear conclusion on which application scenarios PoET-2 is best suited for. Its performance is mixed--PoET-2 is better at predicting the mutation effect of indels but not substitution, excelling in indel and higher-order mutation prediction but lagging in single-substitution prediction. The practical implications of these trade-offs should be better discussed.\n\nQUESTIONS:\n- Why did you choose embeddings from the MLM decoder and LLRs from the CLM decoder? Have you tested alternative configurations such as using embeddings and LLRs from both decoders or swapping their roles?\n\n- Figure 2 shows performance trends with decreasing training data size, but no error bars or standard deviations. Given the performance for small sample sizes (e.g., 10 examples) could be unstable on different draws of training data, can you report the variance across multiple runs?\n\n- Please report PoET-2’s performance without the length reweighting adjustment for indel variants to better understand its importance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes PoET-2 – a multimodal foundation model achieving state-of-the-art results on standard protein fitness prediction benchmarks. Additionally, PoET-2 enables addressing previously unsolvable tasks with PLMs, such as indel mutations. The method introduces several new components, which are well-justified by ablation studies and thus provide a valuable contribution for future research in the field.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Strong empirical results across several standard protein fitness prediction benchmarks (e.g., ProteinGym mutational scanning and clinical datasets).\n- PoET-2 enables zero-shot indel and higher-order variant effect prediction, which were either not addressable or achieved only limited performance in prior methods.\n- The proposed method is technically solid and well justified through extensive ablation experiments presented in the supplemental information.\n- The method is shown to be both parameter- and data-efficient.\n\nWeaknesses\n\n- Without ensambling, PoET-2 underperforms VenusREM on the most well-established DMS substitutions benchmark. While, this is a minor issue considering that the authors demonstrate PoET-2 can be effectively ensembled with VenusREM, it would be great if the authors could provide more detailed comparison of the methods or more detailed discussions on the differences in their performance.\n- Section 3 of the main text mainly discusses well-established machine learning components (e.g., “autoregressive decoder”, PLM and CLM losses) or technical details (e.g., binning of distances between alpha carbons), but omits seemingly important model components described in B.5 (e.g., 3Di token tracks). Could the authors revise Section 3 to make the method description more concise and complete?\n- Figure 1 lacks a caption and is not straightforward to understand without one. For example, what exactly are the outputs of PoET-2?\n\nMinor comments\n\n- The phrase \"protein function\" in the title may be confusing, as the paper primarily focuses on protein fitness prediction. Protein function typically refers to tasks like enzyme EC number prediction, which are not directly addressed in this paper.\n- The title, abstract, and introduction mention “retrieval-augmented,” but afterwards this concept is only mentioned in the discussion and not in the methods section. Please clarify what is meant by “retrieval-augmented.” Does it refer to the model’s ability to retrieve MSA sequences given a query sequences?\n- Please discuss VenusREM in related work and describe the ensembling procedure.\n- Lines 130–132. This statement does not appear to be supported by the experiments.\n- Line 154. It is unclear how D is used in the model.\n- The paper lacks the discussion of future work.\n\nQUESTIONS:\n- Both PoET-2 and VenusREM leverage sequence, structure, and MSA-derived homolog information. Could the authors comment on why PoET-2 may underperform VenusREM on the DMS substitutions benchmark?\n- Could the authors present additional experiments demonstrating, for example, the generative capabilities of the model, in addition to zero-shot or supervised fitness prediction benchmarks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PoET-2, a multimodal and retrieval-augmented protein foundation model that combines sequence and structure information. By using homologous proteins as prompts, it enables context modeling without requiring multiple sequence alignment (MSA). PoET-2 uses a hierarchical Transformer encoder and two decoders, supporting both autoregressive generation and masked language modeling. Experiments show that PoET-2 achieves state-of-the-art zero-shot performance on insertion and deletion (indel) prediction in the ProteinGym benchmark, and performs significantly better than existing methods on higher-order mutations and clinical variant prediction tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- Lightweight and efficient: PoET-2 uses only 180 million parameters but achieves performance comparable to models with hundreds of millions or even trillions of parameters. It is fast and easy to deploy.\n- Supports complex mutation prediction: PoET-2 is one of the few protein language models that can accurately predict the effects of insertions and deletions (indels), making it suitable for a wider range of mutation types.\n- Highly data-efficient: In supervised learning, it only needs about 1/30 of the training data to outperform strong baselines like Kermut, which is especially useful in data-scarce scenarios.\n\n**Weaknesses:**\n\n- VenusREM performs much better than PoET-2 on substitution tasks. Although combining the two models helps, this does not mean PoET-2 alone is comparable to VenusREM. Also, VenusREM is only half the size of PoET-2 and can also handle structural inputs. Its only weakness is handling indels. Overall, VenusREM still appears to be a stronger tool for protein engineering.\n\nQUESTIONS:\n1. Since the pairwise distances between backbone atoms are already given, the position of each $C_{\\alpha}$ atom is mostly determined. Is the additional local distance feature $x_{atomb}$ really necessary? Is there any ablation study to show its impact?\n\n2. For the retrieval step, the model only uses AlphaFoldDB structures corresponding to the query sequences. These predicted structures may be low-quality in some cases. Could the authors consider using a template search strategy, similar to AlphaFold's, to improve retrieval quality?\n\n3. Although the paper shows that combining PoET-2 with VenusREM improves overall performance, Table 1 indicates that VenusREM consistently outperforms PoET-2 on substitution tasks. Ignoring indels, doesn’t this suggest that VenusREM is already a stronger model for typical protein engineering tasks?\n   In addition, VenusREM is only half the size of PoET-2 (110M vs. 182M) and also supports structural inputs. Other than its lack of indel prediction, its capabilities are mostly similar. Since most real-world tasks focus on substitutions, how do the authors view the overall advantages of PoET-2 compared to VenusREM? Are there specific settings where PoET-2 clearly outperforms VenusREM on its own, without relying on model ensembling?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PoET-2, a multimodal, retrieval-augmented protein foundation model designed to improve protein function prediction. Unlike prior PLMs, PoET-2 incorporates sequence and structural information from homologous proteins and leverages both causal and masked language modeling objectives. This allows it to operate in generative and bidirectional representation learning modes. PoET-2 shows state-of-the-art performance in zero-shot prediction of complex mutations (e.g., indels, higher-order mutations), clinical variant effect prediction, and few-shot supervised learning tasks. It achieves this with a modest 182M parameter footprint, demonstrating superior data efficiency and performance without requiring excessively large models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Multimodality: Simultaneous use of sequence and structural data enhances the model’s understanding of protein function.\n\n- Retrieval-Augmentation: Allows in-context learning from unaligned homologs, reducing reliance on massive parameter counts.\n\n- Dual Decoders: Supports both generative and representation tasks, providing flexibility.\n\n- State-of-the-Art Performance: Outperforms existing models on several benchmarks, especially in indel mutation and clinical variant effect prediction.\n\n- Efficiency: Only 182M parameters yet comparable or superior to billion-scale models.\n\nWeaknesses:\n\n- Structure Conditioning Limitations: Shows limited improvement for clinical and supervised prediction tasks.\n\n\n- Lack of Public Code: The paper states code and model weights will be released later, hindering immediate reproducibility.\n\n- Slight Underperformance on DMS Stability Data: Slightly lags behind newer masked structure-based models in this specific setting.\n\nQUESTIONS:\n- How does PoET-2 balance between structure-aware modeling and data availability when predicted structures are missing or low confidence?\n\n\n- How sensitive is the model to the number and diversity of homologs provided during prompting?\n\n- Are there specific protein families where PoET-2 underperforms, and why?\n\n- How does PoET-2 perform in real-world protein engineering campaigns compared to baseline PLMs?\n\nLIMITATIONS:\n- Structure information contributes selectively to task performance, being more useful for stability prediction than functional classification.\n\n\n- Although designed for few-shot learning, performance might still vary with extremely sparse data.\n\n- The model relies on precomputed homologs and structures, potentially limiting use cases with novel or uncharacterized proteins.\n\n- Current benchmarks may not fully represent in vivo protein function complexity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}