Title: Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins
Venue: NeurIPS 2025
Accepted: yes
Keywords: protein

language model

generative

infilling

scaling laws

mixture of experts

sparse

alignment

DPO
Abstract: Generative protein language models (PLMs) are powerful tools for designing proteins purpose-built to solve problems in medicine, agriculture, and industrial processes.
Recent work has trained ever larger language models, but there has been little systematic study of the optimal training distributions and the influence of model scale on the sequences generated by PLMs.
We introduce the ProGen3 family of sparse generative PLMs, and we develop compute-optimal scaling laws to scale up to a 46B-parameter model pre-trained on 1.5T amino acid tokens.
ProGen3's pre-training data is sampled from an optimized data distribution over the PPA v1, a carefully curated dataset of 3.4B full-length proteins. 
We evaluate for the first time in the wet lab the influence of model scale on the sequences generated by PLMs, and we find that larger models generate viable proteins for a much wider diversity of protein families.
Finally, we find both computationally and experimentally that larger models are more responsive to alignment with laboratory data, resulting in improved protein fitness prediction and sequence generation capabilities.
These results indicate that larger PLMs like ProGen3-46B trained on larger, well-curated datasets are powerful foundation models that push the frontier of protein design.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a systematic study of scaling laws in protein language models (PLMs), using a newly curated dataset called PPA that contains over 150 million protein sequences grouped into clusters at different sequence identity thresholds. The authors train a suite of ProGen3 models ranging from 44M to 4.7B parameters, analyzing how model scale and training data distribution affect generative quality, functional diversity, and downstream performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper introduces a well-curated and biologically meaningful dataset (PPA), organized with sequence clustering and rigorous data splits, which provides a strong foundation for benchmarking PLMs.\n\n- It presents a thorough empirical study of how training distribution and model scale influence protein generation, offering insights that can benefit the field without requiring similarly expensive experiments.nd similar computational cost, especially in academic settings, where resources are constrained.\n\n- Beyond language modeling metrics like perplexity, the study evaluates sequence diversity and functional viability using in vitro experiments on generated sequences from the luciferase family. The paper offers practical insights into how data and model scaling impact protein generation and design, particularly in resource-constrained academic settings.\n\n\nWeakness\n- While the paper makes strong contributions in dataset curation, model scaling, and evaluation, it would benefit from a clear commitment to open-sourcing resources. In particular, the authors should clarify whether they intend to release the PPA dataset, trained model checkpoints, and training/inference code. Making these resources public would significantly enhance the reproducibility and long-term impact of the work.\n\n- While ProGen3 demonstrates promising performance, virtually all benchmarks presented in the paper are internal—comparing different variants of ProGen3 across model sizes or training distributions. Since this work introduces a new model family, it would be valuable to include direct comparisons with existing models such as ESM-3 to better contextualize ProGen3’s capabilities relative to the current state of the art.\n\nQUESTIONS:\n- Is there a particular reason why Figures 1e and 4a only show a single FLOP-performance point for the 4B model, whereas other model sizes are represented with full scaling curves?\n\n- The paper mentions that segments were randomly masked for the infilling objective. Could this result in overly easy infilling tasks compared to biologically meaningful challenges? Please consider discussing alternative infilling objectives, such as fixed-region scaffolding or motif-conditioned design, which may provide more biologically relevant evaluation.\n\n- While the paper focuses on scaling within the ProGen3 framework, incorporating additional baseline models such as ESM-3 into perplexity, diversity, or cluster coverage plots (even if just as reference lines) would better contextualize ProGen3’s performance relative to existing methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ProGen3, a sparse mixture-of-experts generative protein language model. The authors explore Scaling-Laws in this architecture and introduce an optimized pre-training dataset, PPA v1. Experimental results demonstrate that larger PLMs can generate much wider diversity across protein families. Furthermore, after using iterative reasoning preference optimization (IRPO) with wet lab data, ProGen3 demonstrates strong performance on protein fitness prediction tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The authors curate and present the PPA-1 dataset, showing the advantages of training on de-biased data.\n2. They optimize the LLM with wet lab data and highlight the effectiveness of such experimental reinforcement.\n3. They train a large (46B parameter) generative protein language model, providing a new base model for the field.\n\nWeaknesses:\n\n1. The paper lacks comparisons of protein generation capabilities with other PLM models, such as ESM3 [1], AIDO.Protein [2], and ProGen2 [3], all of which also support protein generation. It would strengthen the work to include direct comparisons to demonstrate ProGen3’s advantages over these models.\n2. In Section 3.1, the authors claim that “larger PLMs generate more diverse proteins.” However, larger models were trained with more data (Lines 135-137), so it is unclear if the gains come from the increased model capacity or simply from more diverse training data. The necessity of scaling up model size should be better demonstrated.\n3. Methodologically, the innovations seem somewhat incremental. Prior works have already validated MoE architectures for PLMs [2], explored scaling laws with different PLM architectures [3], and shown that preference optimization improves model fitness prediction ability [5]. While the value of using wet lab data is acknowledged, the technical novelty warrants further clarification.\n\nQUESTIONS:\n1. The authors show that IRPO with wet lab data improves ProGen3’s performance on protein fitness prediction. Does this strategy generalize to other downstream tasks (e.g., protein property prediction), or would each task require different experiments? Additional results on various tasks would strengthen the value of wet lab results.\n2. The authors demonstrate the effectiveness of IRPO with wet lab data. Is this approach model-agnostic? If the same data and IRPO method were applied to ProteinDPO [5] or other PLMs, could even better results be obtained? Such discussion or experiments would strengthen the paper’s contribution to data and experimental methodology.\n\n\n\n[1] Simulating 500 million years of evolution with a language model\n\n[2] Mixture of Experts Enable Efficient and Effective Protein Understanding and Design\n\n[3] ProGen2: Exploring the boundaries of protein language models\n\n[4] Training Compute-Optimal Protein Language Models\n\n[5] Aligning protein generative models with experimental fitness via Direct Preference Optimization\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a suite of MoE PLMs named ProGen3 that demonstrate strong scaling performance on not only the pretraining task, but also after being finetuned for downstream tasks such as fitness prediction and stable protein sequence generation. Wet lab experiments also verify that the generated sequences are diverse and have high expression rate.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors conduct an extensive scaling study on the output proteins (and not just the embeddings) of ProGen3 at multiple model sizes and with a validation metric that also accounts for OOD proteins. Such scaling studies are very underexplored in the PLM literature. The authors also empirically identified a data mixture that yielded lower OOD loss than a uniform or unmodified mixture.\n- A large proportion of the generated proteins have are viable/expressible, even for the generations with <30%ID to any of the natural proteins in the data.\n- The ProGen3 models also demonstrate strong performance when finetuned for downstream tasks such as fitness prediction and generation of stable proteins. \n- The ProGen3 suite of models is also provided at a wide range of model scales, making it suitable for a variety of downstream uses as well as further scaling studies.\n- The paper is extremely detailed, providing extensive explanations of both the dry and wet lab experiments.\n\nWeaknesses:\n- The use of IRPO/preference learning for supervised fitness prediction seems poorly motivated -- since IRPO (and most offline preference learning algorithms) trains on ranked pairs, using this algorithm necessarily involves some loss of information (i.e., the magnitudes of the experimentally-determined fitness values). Additionally, such preference learning algorithms have been shown to struggle with learning likelihoods that correctly align with the rankings of the data (see [Chen et al.](https://arxiv.org/abs/2405.19534)). Although it is trendy now to use IRPO and other rankings-based preference learning methods for fine-tuning pLMs, using it for a regression task is not very principled. (Additionally, it is an unfortunate fact that many popular offline preference learning algorithms still show a non-monotonic relationship between ranking performance and generality quality, so it may be better to not try to optimize for both at once with these algorithms. See [Tang et al.](https://arxiv.org/abs/2405.08448) and Sec. 5 of [Chen et al.](https://arxiv.org/abs/2405.19534) for evidence of this.)\n- Very minor nits:\n  - I find figure 1(a) to be somewhat incomprehensible. Even as someone who already understands what infilling generation is, I still did not understand this diagram.\n  - Typo in Fig. 3 caption: \"generated generated\" -> \"generated\"\n\nQUESTIONS:\n1. Would the fitness prediction performance improve if you used an alignment algorithm that incorporated the actual experimental fitness values (e.g. offline/off-policy RLHF or policy gradient), rather than only ranked pairs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ProGen3, a protein LLM scaled up to 46 billion parameters.  The paper goes into significant depth on the development of a new training dataset and its impacts on performance as the number of parameters/data is scaled.  The paper then proceeds with significant evaluation of the ProGen3 family of models, finding that as parameters scale the model is able to generate more diverse protein sequences.  Samples from ProGen3 are evaluated in the wet lab where it is demonstrated that the generated proteins express.  Finally, the authors perform preference alignment on the ProGen3 model and demonstrate that scaling impacts model performance after fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper proposes ProGen3 and details its training procedures (including a new training dataset) and goes into great depth on analysis of the impact of scaling on downstream performance.  The writing is clear and concise and the findings are quite comprehensive.  I would like to in particular commend the detail in this paper on the specifics of scaling up ProGen3 and the various design factors which impacted downstream performance.  Moreover, the evaluation of the model was very comprehensive, covering many different axes of model performance, including wet lab validation.\n\nI do have one concern regarding the reproducibility of the experiments detailed in the paper.  While ProGen3 model weights have been released on HuggingFace for model sizes up to 3B, the 46B parameter model has not been open sourced.  Further, as training set mixes and setups are one of the key drivers of performance in LLM training, it would be very beneficial if the PPA-1 training set described in the manuscript were open sourced, especially as small discrepancies in training sets can cause large impacts on downstream model performance.  I would strongly encourage the authors to consider open sourcing both of these to better the reproducibility of their work, as training a 46B model on 256 H100s for 17 days is impractical for nearly any academic lab.\n\nQUESTIONS:\n1. What are the authors' hunches for why the largest models don't lead to better zero-shot fitness prediction?\n\n2. Similarly, what are the authors' intuitions for why scaling model size doesn't seem to impact expression rates all that much besides on the infilling task?  Also, why do the authors think scaling improves infilling performance in particular?\n\n3. Will model weights for the 46B parameter model as well as the training dataset be open-sourced?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}