Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins
Authors: Aadyot Bhatnagar
Sarthak Jain
Joel Beazer
Samuel C. Curran
Alexander M. Hoffnagle
Kyle Shan Ching
Michael Martyn
Stephen Nayfach
Jeffrey A. Ruffolo
Ali Madani
Venue: NeurIPS 2025 spotlight ()
Keywords: protein
language model
generative
infilling
scaling laws
mixture of experts
sparse
alignment
DPO
Primary area: machine_learning_for_sciences

Abstract: Generative protein language models (PLMs) are powerful tools for designing proteins purpose-built to solve problems in medicine, agriculture, and industrial processes.
Recent work has trained ever larger language models, but there has been little systematic study of the optimal training distributions and the influence of model scale on the sequences generated by PLMs.
We introduce the ProGen3 family of sparse generative PLMs, and we develop compute-optimal scaling laws to scale up to a 46B-parameter model pre-trained on 1.5T amino acid tokens.
ProGen3's pre-training data is sampled from an optimized data distribution over the PPA v1, a carefully curated dataset of 3.4B full-length proteins. 
We evaluate for the first time in the wet lab the influence of model scale on the sequences generated by PLMs, and we find that larger models generate viable proteins for a much wider diversity of protein families.
Finally, we find both computationally and experimentally that larger models are more responsive to alignment with laboratory data, resulting in improved protein fitness prediction and sequence generation capabilities.
These results indicate that larger PLMs like ProGen3-46B trained on larger, well-curated datasets are powerful foundation models that push the frontier of protein design.

Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins
Aadyot Bhatnagar"
Sarthak Jain"
Joel Beazer'
Samuel C. Curran"
Alexander M. Hoffnagle"
Kyle S. Ching"
Michael Martyn'
Stephen Nayfach"
Jeffrey A. Ruffolo*-1
Ali Madani--i
Abstract
Generative protein language models (PLMs) are powerful tools for designing pro- teins purpose-built to solve problems in medicine, agriculture, and industrial pro- cesses. Recent work has trained little systematic study of the optimal training distributions and the influence model scale the sequences generated by PLMs. We introduce the ProGen3 family of sparse generative PLMs, and we develop compute-optmma scaling laws scale up 46B-parameter model pre-trained Gen3's pre-training data is sampled from an optimized data distribution over the Profluent Protein Atlas vl, carefully curated dataset of 3.4B full-length proteins. We evaluate for the first time in the wet lab the influence of model scale on the sequences generated by PLMs, and we find that larger models generate viable pro- teins for much wider diversity of protein families. Finally, we find both compu- tationally and experimentally that larger models are more responsive to alignment with laboratory data, resulting in improved protein fitness prediction and sequence generation capabilities. These results indicate that larger PL.Ss like Pr6Gen3-46B trained larger, well-curated datasets are powerful foundation models that push the frontier of protein design."
larger language models, but there has been
1.5T amino acid tokens. Pro-
Introduction
Proteins ubiquitous molecules that play central role in most biological processes. They catalyze reactions, contribute to immune function, regulate cellular pathways, and facilitate the transport of other molecules. They enable solutions for multiple industries including therapeutics, diagnosties, agriculture, energy, and manufacturing. Until recently, we have been limited to finding relevant proteins in nature through serendipitous discovery. Techniques to engineer proteins for desired purposes largely rely on laborious laboratory techniques that randomly mutate these initial hits in hopes of discovering variant with enhanced function [G] [a]]. Relying these inherently random processes presents major challenge to the bespoke design of proteins for specific use cases.
The dramatic reduction of DNA sequencing costs has enabled exponential growth in ability to sample naturally occurring protein sequences that are the product of selective pressures over billions of years. Generative protein language models (PLMs) have emerged as powerful tools to learn the complex distribution of proteins found in nature, and their ability to design highly functional novel proteins has been experimentally verified for wide range of applications [HI, 45. 63, 68 79, 80 96]
"Profluent Bio, Inc.
To whom correspondence should be addressed: (abhatnagar jeff jelff,,ai.@pproflueett bio Code and model weights are available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, a number of important questions have remained relatively under-explored in the PLM lit- erature. Despite the rapidly accelerating growth of protein sequence datasets, little work has been done to determine the optimal data distributions for training ever larger PLMs [G3]] While evalua- The trajectory of these models mirrors those found in natural language processing (NLP) research. Earlier works in NLP demonstrated that larger models learn representations that can be fine-tuned to perform downstream tasks more effectively [30, I25, αταI, [TOT], and elucidated the compute-optimal ways to scale up the size of both models and datasets to reap these benefits [9%] 1. Similarly, has been shown that PLM embeddings implicitly capture notions of protein fitness that can be elicited in both supervised and unsupervised settings [[65] 55.
90X MSS) and recent work has derived
compute-optimal scaling laws for PLMs [T9].
tions in NLP have shifted towards analyzing the sequences generated by LMs rather than properties of their embeddings [20X 23, HIl unclear how scale influences the proteins generated by PL. both computationally and experimentally. Finally, post-training strategies to align models with user- defined preferences have gained traction in both domains [74, 88. [00,, [TY but the influence of an aligned PLM's scale on its performance has received only limited attention [EM]]
In this work, we introduce ProGen3, family of autoregressive generative PLMs that leverages sparse mixture of experts architecture [52. D6 85] to improve efficiency while maintaining perfor- mance. To train ProGen3, we assemble the Profluent Protein Atlas v1 (PPA-1), a highly curated dataset of 3.4B full-length proteins and 1.1T amino acid tokens, and we optimize the data distribu- tion for pre-training We determine compute-optimal scaling laws for sparse PLMs and use them to scale ProGen3 up to a 46B parameter model trained on 1.5T tokens from PPA-1. Next, we evaluate for the first time in the wet lab the influence of model scale on the sequences generated by PLMs. We find that larger models generate viable proteins for much wider swaths of sequence space. Finally, we demonstrate both computationally and experimentally that models at all scales can be aligned with laboratory data for improved protein fitness prediction and sequence generation capabilities, and that the larger models receive the greatest lift from alignment. Taken together, these results present clear case that larger protein language models trained on larger, well-curated datasets are more useful tools for a broad range of protein design challenges.
1.1 Related Work
Mixture of Experts Scaling up model size is one of the most important ways to improve the
performance of deep learning models [43, 51]] However, larger models are also more compute- intensive. Mixture of Experts (MoE) layers leverage sparsity to increase model efficiency [85]. These layers consist of multiple expert sub-networks, and they route each individual token to a
different subset of experts. Transformer models trained in the NLP [A 62, 29, 56]] computer
vision [IZZ]] and protein [20] domains have replaced the feedforward network with a MoE to unlock significant speedups while maintaining performance comparable to dense model.
Protein Language Models Protein language models (PLMs) encompass wide range of ap- proaches including causal decoders that generate novel proteins from scratch [63,, 68, 80] or infill spans in the middle of a protein [ISS, 86], bidirectional encoders useful for protein understanding [HD. 59 78. I0], and inverse folding models that generate proteins conditioned on a user-specified structure [27. 9S. 7y), among others. These models are typically pre-trained on large databases of naturally occurring proteins. Compute-optimal scaling laws have been derived for dense PLMs [[9]] but it is unclear whether they also apply to sparse PLMs.
Limited work has been done to determine the best data distributions for training PLMs, and prior works have largely relied on range of ad-hoc distributions. Lin et al. [5 ]] and Hayes et al. [MM] de- replicated their data at 90% ID and respectively sampled each 50% and 70% ID cluster with equal probability. Nijkamp et al. [68] and Cheng al.
[1T]] de-replicated their genomic data at 90% ID and their metagenomic data at 30% ID. Chen et al. [1S]] and Sun et al. [XX] de-replicated their genomic data at 90% ID and included at most 10 sequences per 30% ID cluster for their metagenomic data. Meanwhile, Fournier et al. 3 found the best results without any resampling.
Model Alignment As language models grow, they acquire increasingly useful emergent proper- ties over the course of unsupervised training [8, 21, go, 199 68, B5). However, out of the box, they often ill-suited to tasks that their users care about, from generating performant code to design-

---

a
N-to-C Generation
C-to-N Generation
DOKK
Residue Prediction
Expert
Experts
Expert
Infiling Generation
Router
Causal Self Atlention
Expert n-1
Input Embedoing
Expert
M
Y
1.0
Distributoo
Aggrogate
Square Root
1988
2.4
213M
112M
109
3,08
0.8
Inverse Los
353ms
468
Unmodittd
76280
Op..
Distinct: @ 30%
0.6
2.176
123
Distinct 50%
990
1.
Dieniuttio Unform
1.6
0.2
Square Root Lop Unmodiied
Distinct @ 90%
L,675
10
10
10a
10*
22
202h
10
10TH
20
10
1077
Sequences Per Cluster
Figure 1: Determining the optimal data distribution and scaling laws to train ProGen3, sparse generative PLM. (a) ProGen3 can generate proteins from the N- to C-terminal from the C- to N-terminal. It can also generate spans in the middle of a protein. (b) ProGen3 is an autoregressive transformer with sparse mixture of experts architecture. (c) Diversity of PPA-1 data distributions as measured by the CDF of 50% ID cluster sizes.
(d) Validation losses of 1.4B parameter models trained on 80B tokens from different data distributions. (e) Validation losses of models with 112M
to 46B parameters trained on 10B to 1.5T tokens from the Inverse Log distribution. "Opt." indicates the predicted losses for the compute-optina configurations given by Equation MI
ing more stable proteins. Alignment algorithms leverage supervised data to direct language model generations to match user's preferences. These include fine-tuning
curated set of positive examples [68. 82 [II),, reinforcement learning to maximize the user-specified rewards achieved by language model's conditional generations [22, ITI) and direct alignment algorithms that reformu- late the reinforcement learning task as an easier supervised learning task [7]] ZZ]] For PLMs, direct alignment algorithms have improved models for inverse folding [20, [06], fitness prediction [I00%]
function- Lunction-guided generation [40] 88], and binder design [48, 67]
ProGen3
2.1 Architecture
ProGen3 a family of autoregressive (AR) transformer based PLMs with a sparse mixture of experts (MoE) architecture that only activates 27% of its parameters per forward pass [32, 56 85, ø9] (Fig- ure (b). Models span sizes ranging from 112M to 46B parameters, and all models have a context length of 8192 tokens. See Appendix B.l for more details. Like prior work in NLP AA BUJ, we find that for a fixed compute budget, sparse models meaningfully outperform dense ones (Appendix B-))
A standard AR PLM enables causal language modeling (CLM), where users can generate proteins one amino acid at time either from the N- C-terminal, or from the C- N-terminal [63] 68). In addition to CLM, ProGen3 also performs generalized language modeling (GLM) to infill spans the middle of a protein [G) IS. 65,
NE 92]. does so by replacing target spans with special sentinel tokens and placing those spans (preceded by their corresponding sentinel tokens) at the end of the sequence (Figure Пl). This capability allows users to redesign domains in the middle of protein while attending to all surrounding residues. See Appendix B for more details.
15.

---

2.2 Training Data
To train ProGen3, we curate the Profluent Protein Atlas v1 (PPA-1), a dataset of 3.4B proteins and I.IT amino acid tokens. PPA-1 draws from wide range of genomic and metagenomic sources, and we apply multiple layers of quality filters to ensure the dataset is appropriate for pre-training a generative PLM. In particular, we exclude all protein fragments sequences. PPA-1 is of similar scale and diversity to the OMG dataset [[ It has a similar number of sequences as ESM3's dataset, but ESM3's dataset does include protein fragments [90]. It is also considerably larger than UniRef [РI] and BFD [SUL 87]]. See Appendix A for more details.
can train only full protein
While protein datasets are rapidly expanding. they have biases that influence the quality of PLMs trained on them [ 31 [OK]] Therefore, we aim to more rigorously study the impact of the training distribution, an aspect which has been relatively underexplored in prior PLM literature (Section I.. We consider 4 different schemes of balancing data diversity over 50% clusters, which we refer to (in descending order of diversity) as Uniform, Square Root, Inverse Log, and Unmodified (Fig- ure mc). In each of these distributions, we respectively sample a 50% ID) cluster of size n with probability proportional to 1 (equal probability), vn, n// n1 + log ni), and (the natural sequence dis- tribution). These distributions are listed in descending order of diversity: 80% of sequences sampled from these distributions reside in clusters of size at most 8, 94, 591, and 1471, respectively. After sampling a 50% ID cluster, we sample a 90% ID sub-cluster from the same distribution, before sampling sequence from that 90% ID sub-cluster uniformly at random.
To measure model generalization, we construct validation sets distinct from our training data at 30%, 50%, and 90% ID. Each x% ID validation set consists of 2.5M sequences distributed uniformly between X% ID clusters of sizes 1-10, 11-100, and 101-1000. The average loss thus avoids over- weighting highly represented parts of protein space and more accurately measures generalization. We also average the losses on these sets to compute an aggregate validation loss.
Figure IId shows the validation losses achieved by 1.4B parameter models trained for 80B tokens sampled from each distribution. For proteins close to our training data distinct at 90% ID - training on the Unmodified distribution yields the best performance. The Inverse Log distribution is a close second, while the other two are considerably worse. However, training on the Inverse Log distribu- tion yields the lowest loss for out-of of-distribution sequences that are distinct from the training data at 50% and even 30% ID. We therefore train all subsequent models on the Inverse Log distribution.
Interestingly, training on the Uniform distribution, which is similar in spirit to de-replicating the training data 50% ID, consistently obtains the worst performance on all validation sets. This suggests that PLMs can learn important signals from the frequencies with which related proteins occur in the data. However, some rebalancing can improve out-of-distribution generalization.
2.3 Scaling Up to a 46B Parameter Model
Given a fixed computational budget, scaling laws describe the optimal way to allocate pre-training resources between model size and dataset size [[9 83, 55I]. They do so by predicting a model's loss L as function of the number of parameters N and the number of tokens in the training dataset D. We fit a variant of the equation proposed by Kaplan et al. [ST]], L(N, D) = (AN-o/3 + BD-1) + C, to the validation losses achieved by models spanning wide range of parameter counts and dataset sizes. This equation can then be manipulated to yield power law Nops (D) x D3/a which tells us the compute-optine model size to pre-train on a dataset with D tokens.
We train models with 1 112M to 10B parameters for 10k to 500k steps (10B to IT tokens, depending on batch size; see Appendix B. Because we use a cosine learning rate schedule, we train separate We plot their validation losses in Figure Me and find
model for each dataset size
Nopo (D) (2.462 x 10 DDDDDl-
This result for sparse AR PLMs agrees closely with Cheng et al. [ who report that Nops (D) DI-370 for dense AR PLMs. Equation 0 predicts that the optimal allocation for 1.1 x 10ah FLOPs to train a 90B parameter model for 753B tokens. This model predicted to achieve a validation loss of 1.376. We make the practical decision to instead train a 46B parameter model for 1.5T tokens because it is predicted to achieve a slightly higher validation loss of 1.397 but can be used for inference on 4x 40GB GPUs

---

Generoing model
330M
g
Gemera Pasning fiters
Nonura Currers Cavered
2.50
Priteei 23394
Proteenh
expressao
40-505000
2.50
40-40%ID
33
40%
1M
333M
302k
ProGer3-463
50%ID
Overisp Covernd Clasters
-05% Rupeats
Termination wSow Coverage
25ck 500k. 750k
1.00
Volid Gener ations
Expressing (95)
333M08/468
TTL Covered Clater
Gan, Model
Can 329M
30
36
1.5
10/663
10-
1Ck-
Valid
3k-
333M
38
50-30500
465
338M
3K 100K
Scoring Mide
Covered Cluter
339M
18 463
Expressing
Figure 2: Larger models generate viable proteins for a much more diverse set of families than smaller
models. (a) Larger models generate more valid sequences that pass series of quality filters. (b)
Valid generations from larger models cover a wider array of naturally occurring 30% ID clusters. (c)
The clusters covered by larger models are largely superset of those covered by smaller models. (d)
Larger models generate sequences from clusters whose natural members are assigned high perplexity by smaller models, but not vice versa. (e) Larger models are better able generate sequences from smaller clusters. (f) Visualization of the clusters selected for experimental characterization.
accompanied by ESMFold predicted structures for selected generations. (g) In vitro expression rates of model generations for different groups of clusters. Black circles indicate natural protein expression rates for the same clusters.
Interestingly, this 46B parameter model achieves validation loss of 1.345, which lies below the predicted compute-optimal frontier (Figure Ile, bottom right). We had to increase the warmup period stabilize the training for this model, which could explain the discrepancy, since longer warmup improved the final training and validation losses of a 1.4B parameter model trained for 80B tokens (Appendix B.4))
Computational and Experimental Analysis of Model Generations
We have shown that scaling up the computational resources to pre-train PLMs predictably improves their validation losses for proteins both near and distant to their training data. In other words, larger models better represent the naturally occurring distribution of proteins. However, the implications of scaling on the generative capabilities of PLMs have largely gone unexplored by prior works. We focus on three models separated by three orders of magnitude
pre-training compute: ProGen3- 339M (trained on 200B tokens with 1.2 x 10a0 FLOPs), ProGen3-3B (trained on 500B tokens with 2.6 1021 FLOPs), and ProGen3-46B (trained on 1.5T tokens with 1.1 1028 FLOPs). We also evaluate ProGen2-XX XL [68], and ProGen3-3B-200
dense autoregressive model with 6.4B parameters, variant of ProGen3-3B trained on only 200B tokens.
3.1 Larger PLMs generate more diverse proteins that express in vitro
From each ProGen3 model, respectively generate 4M, 3M, and 2M sequences unconditionally using top-p sampling [E]]] with p = 0.95 and temperatures E [0.5, 1]. We also sample 4M generations from ProGen2-XL using the same hyperparameters.
We then apply series of quality filters to extract "protein-like" sequences. Because language mod- els often generate repetitive sequences [24] 1055, we first remove all sequences consisting of >25% low-complexity regions [36]]; 95.5% PPA-1 also passes this filter. We also require that genera- tions include the appropriate termination token, and that they have >80% alignment coverage to any natural sequence in PPA-1 [2] 8D). Larger ProGen3 models generate more walid sequences that pass each of these quality filters, and scaling up from 339M to 3B especially reduces the number of repet-

---

itive generations (Figure 2a). Interestingly, only 4% of the sequences generated by ProGen2-XL are valid, compared with 30%, 40%, and 49% from ProGen3-339M, 3B, and 46B, respectively.
Next, we align each valid generation to the 30% ID cluster representatives. in PPA-1 and say that it covers cluster if it is >30% IID to the cluster representative with >90% alignment coverage. Note that single generation can cover multiple clusters because a sequence can be >30% ID to multiple cluster representatives that are <30% ID each other. Larger models generate sequences from a more diverse set of clusters than smaller models (Figure [b). Moreover, the clusters covered by smaller models are almost entirely covered by larger models, while larger models generate se- quences from many clusters that are not covered by smaller models (Figure 2c). Amongst its valid generations, ProGen2-XX XL covers 18% fewer clusters than ProGen3-3B despite being larger model.
We also find that for ProGen3,, model size has more influence than dataset size, further corroborating our scaling laws (Equation II). ProGen3-3B-2000 generates 38% valid sequences, compared to 30% for ProGien3-339M (200B tokens) and 40% for ProGen3-3B (500B tokens). Meanwhile, ProGen3- 3B covers 81% % more clusters than ProGen3-3399 but only 14% more than ProGen3-3B-200B.
To determine whether sampling more sequences would allow smaller ProGen3 models to generate from the clusters covered by larger models, we compute the average perplexities of the naturally occurring proteins in those clusters. If given model assigns high perplexity to a cluster's 's naturals, it is unlikely to generate sequence from that cluster. Figure Dd shows that larger models routinely generate from clusters that smaller models assign high perplexity, but not vice versa. This implies that larger models can cover all the clusters that smaller models generate from, but that some clusters covered by larger models are out-of -f-ttiiibitttnonnn for smaller models. This trend is partially explained by the fact that as models scale up, they become better covering smaller clusters (Figure De).
We now seek to characterize the viability of these generated proteins in the lab. It is infeasible to directly measure protein function for a wide range of families with potentially unknown functions. Therefore, we instead perform the split-GFP E. coli protein expression assay [ILI, [2] (Appendix (D) to measure soluble protein abundance, which depends stability, translational yield, thermodynamic folding equilibrium, kinetic stability, susceptibility to proteases, aggregation propensity, and toxicity [R, 60). This assay correlates well with protein ex- pression assayed by direct methods such as PASGE and Western blotting, is amenable to high throughput assay [6]]], and has been used to benchmark generative protein models [+7]]. All of these qualities are important components of a protein's 's fitness.
factors including mRNA abundance and
Given that increased diversity is one of the clearest impacts of scaling up PLMs, we select for exper- imental characterization 42 clusters that all 3 models generated from, 62 clusters that ProGen3-3B and ProGen3-46B generated from, and 45 clusters that only ProGen3-46B generated from (Fig- ure 2). From each cluster, we select one sequence per model that is 40-60% ID to a natural sequence in PPA-1, and one sequence that is 60-80% ID. For each (cluster, %ID bucket) pair, the selected se- quences have the lowest perplexity according to the model that generated them. We also include as controls two random natural proteins for each cluster, and ensure that the sequences in the selected clusters span a wide range of lengths (75-300aa) and structural diversity.
Figure 2g reports the cluster-averaged percentage of sequences that showed soluble expression in vitro. All models attain similar rates of expression, but larger models do so for a wider range of clusters. As one might expect, generated proteins that are 40-60% ID to a natural tend to have lower rates of expression than those that are 60-80% ID. Interestingly, for all models, the generated proteins expressed at rates similar to random naturals from the clusters they reside in.
3.2 PL generate viable proteins outside natural sequence space
Having demonstrated that proteins generated by PLMs residing in sequence clusters alongside nat- urals are broadly viable, we next turn to sequences without a natural reference point, i.c. they are <30% identity to any protein in PPA-1 (or could not be aligned). From the same set of uncondi- tional generations, we select 10 each that have secondary structure compositions (as predicted by ESMFold [59]] from ProGen3-339M, ProGen3- 3B, and ProGen3-46B, for a total of 90 sequences. We consider the rate of expression using the split-GFP assay (Figure Ba), as well as the expression levels relative to our SPIit-GFP positive con- trol (Figure (b)). We find that all models are comparable in their ability to generate sequences that express in vitro, but generations from larger models generally have higher expression.
mostly-betar and mixed alpha/beta

---

Generations with <30%/D to Naturals
Expression vs, WT
C
Short infil s: Expression vs WT
Long Infili:: Expression vs WT
0.5
Hel. Expressioo (408)
0.6
Expressing (N)
Riil Expression
60
201
0.4
Bel.
0.-
Parameters
3.00
468 (4)
N.5. 11.31
46B (5)
N.5. [16]
9.0
38001 35
668
3:9M 35 468 Generating Model
Lamg Generation
Shert
Rel Expression (28)
10"
Rel Expression (38)
10"
Generating Model
Figure 3: Protein language model generate novel proteins and infill effectively. (a) Expression rates (left) and levels relative to positive control (right) for generated proteins with less than 30% identity to any natural sequence. (b) Expression levels of infilled sequences relative to respective scaffold proteins for long- and short-span infilling tasks. (c) Comparison of relative expression levels for infills generated for the same spans by models with 3B and 46B parameters.
3.3 Larger PLMs are better infillers
ProGen3's infilling capabilities enable redesigning key segments in the middle of proteins quence. To test these capabilities, we select nine therapeutically or industrially relevant proteins 100-400aa in length and with some precedent for compatibility with recombinant E. coli expression. For each protein, we use ProGen3-3B and ProGen3-46B to infill sequences for two randomly se- lected short spans (20% of the protein's 's length) and two randomly selected long spans (50% of the protein's 's length). We also randomly select nine additional cytosolic proteins between 100aa and 300aa from the E. coli proteome and the models infill sequences for short span and one long span.
For each span, select from each model the two infills with the lowest perplexity according to the model that generated them. We experimentally determine the expression levels of two selected sequences from each model, as well as the expression of the corresponding natural protein, using our split-GFP assay.
11 of the 18 natural proteins come from the E. coli proteome and may therefore be expected to express well in our split-GFP assay. While infilled sequences have reduced expression on average compared to their corresponding natural protein (Figure (b), there are nonetheless examples of in- filled sequences with improved expression over their natural counterpart. Overall, ProGen3-46B slightly outperforms ProGen3-3B, both by the average expression across all infills (Figure Bb) and by the relative expression for a given infilled span (Figure Be). This suggests that larger models may be more adept understanding the context and constraints present in given sequence for infilling.
Aligning Protein Language Models to Laboratory Data
Zero-shot fitness prediction is often used evaluate PLMs pre-trained evolutionary data. Per- formance is typically measured by the Spearman correlation between model likelihoods and ex- perimentally determined fitness values for the proteins in deep mutational scan (DMS) dataset [#2, 65, 69). However, due to correlations present in evolutionary dynamics, a PLM's task of faith- fully capturing the natural protein distribution can be odds with zero-shot fitness prediction [Π[К]]
Indeed, despite the fact that scaling up model size decreases the validation loss of proteins highly distant from their training data and allows them to generate much greater diversity of expressing proteins, we find that models larger than 3B parameters often perform worse on the ProteinGym benchmark [69]] (Figure Bk). This finding strengthens with more extensive experiments the hypothe- ses Weinstein et al. [103]] and Gordon et al. [G8], that beyond a certain threshold, better estimators of the natural protein distribution can be worse fitness predictors.
Instead, we find that a key advantage of larger models lies their latent abilities which can be elicited through supervised training. To this end, we use iterative reasoning preference optimization (IRPO, (ZII) to align ProGen3 likelihoods with experimentally measured protein properties includ- ing activity, binding, organismal fitness, and stability. See Appendix edl for more details.

---

ProteinGym Subs (Zero-Shot)
b
ProteinGym Multi-Sule Subt
Megascale All Variants
0.40
0.35
Spoarman
Spcarra
Pro Tra Aligned
Spearman
0.6
Pre-Trained Aligned
0.30
Number of Paremeters
112M
0:25
219M
1.00
0.5
0.4
10
10aa
10*
1016
10
1010
Pre-Training FLOPs
Number of Parameters
Number of Parameters
Megascale Single-Subs
e
Megascale e Mu ti-Subs
In Siiico Stabilization
In Vitro Expression
Rel, Expression (Algnee))
REU from Natural
Pre-Trained A iineed
Spearman P
0.6
Spearman
7
10a
06
-9
0.5
Pre-Trained Allgned
Pre Trained igned
10a
Pre-Trained (3)
0.4
0.4
-15
10-1
Alligned (8)
N.S. (21)
10a
1030
10a
1010
10
10:0
10
Number Parameters
Number Parameters
Number of Paramaters
Rel. Expression (Pre: (Pre-Trained)
Figure 4: Aligning model with laboratory data improves its utility for practical design tasks, and larger models are more amenable to alignment. Solid lines indicate ProGen3, while dashed lines indicate ProGen2. (a) Increasing model scale beyond certain point degrades zero-shot fitness prediction. (b) Aligning models on a local mutational landscape improves their fitness prediction capabilities for much more distant variants. ProGen3 outperforms ProGen2. (c) Aligning models on protein stability datasets allows them to perform out-of-distribution stability prediction. ProGen3 outperforms ProGen2. (d) Stability prediction performance on single-substiution variants. (e) Sta- bility prediction performance on multi-substitution variants. (f) Average change in Rosetta energy (REU) between natural proteins and sequences generated using those proteins as prompts. Align- ment and model scale both improve the in silico stability of generated sequences. (g) Aligning ProGen3-46B on stability data improves the in vitro expression of its generations in many cases.
4.1 Supervised Fitness Prediction
First, we demonstrate how alignment can accelerate directed evolution campaign by training Pro- Gen3 models on a local mutational landscape and performing fitness prediction on more distant variants. We identify all assays in ProteinGym [[9]] with at least 3 mutations, and we train all variants at most i mutations from the wild type, where l is the smallest number required for the train split exceed 500 sequences. To ensure that the train and test splits contain proteins of similar fitness, we require that the total variation distance between the train and test distributions of fitness scores be less than 1. These filters yield 8 assays that measure diverse functional attributes for wide range of proteins (Supp. Table R
With as few as 500 experimental measurements of single-substitun protein variants, aligned mod- els learn generalizable concepts of fitness that allow them to accurately rank proteins that are tens of mutations away from the wild type. The overall fitness prediction performance of aligned models increases with their size (Figure Hb, Supp. Figure R) An aligned ProGen3-46B (p 0.673) outper- forms KERMUT [B9] (p 0.628) and obtains similar performance to ConFit [ICE]] (p = 0.679). Both baselines are state-of-the-art methods that leverage inductive biases specifically designed to predict the effects of substitutions. In contrast, ProGen3 can also generate novel proteins, and our IRPO formulation can accommodate insertions and deletions as well as substitutions.
Next, we align ProGen3 models with folding free energy measurements (AG) from the Megascale protein stability dataset [17]], which consists of almost 800,000 single- and couble-mutation variants of 479 protein domains. We evaluate how well the models learn universal concept of stability that generalizes to proteins highly distinct from the training data. Like Widatalla et al. [II6),, we use FoldSeek easy-cluster [98] with 50% alignment coverage to structurally cluster the domains. We hold out 5% of clusters for validation and 5% for testing. Additionally, we only train on single- substitution variants.
8

---

As before, increasing model size uniformly improves the stability prediction performance of an aligned model (Figure Hk-e). Using a similar train/val/test split, ProteinDPO [[UК], similar method that additionally leverages structural information, obtains comparable overall performance (p 0.72) to the aligned ProGen3-46B (p = 0.737, Figure Be). However, despite only being trained on single-substitttion variants, our sequence-only method is a much better stability predictor for multi-substitution variants, achieving p = 0.820 compared to ProteinDPO's = 0.468 (Figure De).
Finally, across all fitness prediction tasks, we find that for a fixed parameter count, applying IRPO to ProGen3 outperforms applying IRPO to ProGen2 (Figure Hb- but ProGen3 is much more efficient due to its sparse MoE implementation. This further highlights the importance of the improved architecture and pre-training data distribution that differentiate ProGen3 from ProGen2.
4.2 Sequence Generation
Encouraged by these results, we now evaluate the generative capabilities of Mability-aligned models. From the set tested in Section B.I. we select as prompts 32 structurally diverse natural proteins that are 98-282 amino acids long (considerably longer than the domains in the Megascale dataset, which are 40-72 amino acids long). We generate 50,000 sequences per model per prompt subject to the quality filters described in Figure Ca, either the C-terminal 75% conditioned on the N-terminal 25%, or the N-terminal 75% conditioned on the C-terminal 25%. For each (model, prompt) combination, select the 100 sequences passing these quality filters that have the lowest perplexity according to the model that generated them.
As an initial evaluation, we perform an in silico characterization of protein stability. For each batch of generated sequences and the corresponding natural sequence prompt, we predict the structure with ESMFold [59] and perform four cycles of minimization (MinMover) and relaxation (FastRelax) across five independent trajectories in PyRosetta [0=]]. To evaluate the effects of model scale and alignment the stability of generated proteins, energy achieved for each sequence relative to the corresponding natural (lower is more stable). We find that larger models generally produce more stable proteins and that alignment further stabilizes proteins across model scales (Figure B!)
calculate the difference in the minimum Rosetta
Finally, prior work has found that the split-GFP assay correlates positively with protein stability [5), 60J. To determine whether alignment on protein stability data actually improves the stability of generated proteins, we select for each prompt the two (owest-perplexig sequences from both pre- trained and stability-aligned ProGen3-46B models, and we characterize them with the split-GFP assay. In Figure Blg, we plot for each prompt the average expression of each model's generations, normalized by the expression of the wild type protein used to construct each prompt. Using a paired- sample Welch's f-test [[оое]] with p 0.01. we find that alignment improves in vitro expression for 8/32 prompts, degrades it for 3/32 prompts, and has no significant effect for 21/32 prompts.
We thus verify both computationally and experimentally that alignment not only improves ProGen3 model's ' stability prediction capabilities, but also improves the stability of the sequences it generates. Moreover, alignment imbues these models with concept of protein stability that generalizes to proteins that are structurally dissimilar to and considerably longer than any of the sequences found in the alignment dataset. Since alignment also improves out-of-distribution fitness prediction more broadly, we expect that it can be applied to improve models' abilitics to generate sequences that optimize diverse array of functional attributes, and that larger models will see greater gains.
5 Discussion
In this work, we introduce ProGen3, family of sparse, optimally-scaled generative protein language models (PLMs) trained on one of the largest high-quality protein datasets constructed to date. We systematically demonstrate the importance of curating appropriate data distributions for pre-training PLMs, and we use our findings to train a 46B parameter instantiation of ProGen3 on 1.5T tokens. To our knowledge, ProGen3-46B is the largest sparse PLM yet, and it required 5-10x less compute to pre-trat trair than dense frontier models of a comparable scale [[35 40].
We perform wet lab studies on the relationship between model scale and the fitness of proteins gen- erated by PLMs, and we find that larger models generate viable proteins for a much wider diversity of protein families. Thus, an immediate consequence of model scaling is the ability to better rep-

---

resent rare protein families. Larger models' more general representations also improve their ability redesign protein of interest for improved expression, even if that protein well-represented in the training data. Finally, we show both computationally and experimentally that larger models reap the greatest benefits to their protein fitness prediction and sequence generation capabilities when we align them with laboratory data. In particular, aligned ProGen3-466 consistently matches or beats state-of-the-art supervised fitness prediction methods while being a far more flexible model.
We have thus demonstrated that larger generative PLMs are more useful tools for wide range of real-world protein design tasks. Continued model scaling can leverage the exponentially growing amounts of protein sequence data while employing more sophisticated implementations of sparsity remain highly efficient [29]]. However, our results show that ProGen3-468 already well poised to advance the vision of designing bespoke proteins for use cases that include drug discovery, en- zyme engineering, and industrial processes.
Safety and Ethics
Computational protein design carries the dual potential accelerate the development of novel ther- apeutics and other society-improving molecules, while providing parallel capabilities for nefarious uses, such engineering of bioweapons. When bolstered by current and future iterations of gener- ative AI, these capabilities are heightened and expected to grow further. The global protein design community has begun to establish appropriate regulations and guidelines towards the continued beneficial development and application of these technologies. We support having a set of commu- nity values, guiding principles, and commitments for the responsible development of Al for protein design httpss//resoonsblebbddessss
Gene synthesis represents a critical step in the actualiza- tion of designed protein sequences. The International Gene Synthesis Consortium (IGSC) unites major gene synthesis providers under a commitment to screen all incoming orders against known pathogens and potentially dangerous sequences. As concrete step towards safe application of pro- tein design technology, all gene synthesis work in support of the present study was performed with IGSC members. For all protein design projects, we urge researchers maintain ethical oversight throughout project initiation, experimental characterization, and subsequent deployment phases to ensure safety and avoid unintended harmful outcomes. For the current models described in this paper for release, we find the benefit of model accessibility to greatly outweigh any theoretical risks.
Competing Interests
The authors are current or former employees, contractors, or executives of Profluent Bio, Inc. and may hold shares in Profluent Bio, Inc. Funding for this work was provided by Profluent Bio, Inc.
Acknowledgments
We would like to thank Adeline Chen and Rachel Lee for developing our high-throughput lab in- frastructure, Eleanor Dunietz for developing lab informaties systems, and Joseph Gallagher and Peter Cameron for overarching support.
Author Contributions
Data: S.N.
Pre-Training: A.B., S.J.
Alignment: A.B.
Analysis and Experimental Design: A.B., A. A.M.H., SJ.., A.M.
Split-GFP Experiment: S.C.C., K.C., M.M.
Open Source Development: S.J.
Overall Scientific Direction: A.M., A.B., J.A.R.
10

---

References
[1] J. Ainslic, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai. GQA: Training generalized mueri-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023. URL httpp::/openrest www.oeeerrveee..eeefooooee
[2] F. Altschul, Gish, W. Miller, W. Myers, D. Lipman. Basic local alignment search tool. Journal of Molecular Biology, 215(3)4433-40, 1990. ISSN 0022-2836. doi: https://doi.org// (016650002-2233605558888888 50022283605805661
URL
com www.s/////e/////wccccoecooceceee awt.a//c///atacic/pii..
[3] E. H. Arool. Design by directed evolution. Accounts of Chemical Research, 315))1 Mar 1998. ISSN 0001 -4822. doi: 10.1021/ar960117.. URI. httpp:://6b..or//1.. 1tt.:////.....1/...1./1/21 1
[4] P. Avasthi and R. York. The known protein universe phylogenetically biased. Arcadia Science, Aug 2024.
protein-universe www.euue--pylloeeeeeeeeeee phylophnetic-
[5] D. Balchin, M. Hayer-Hartl, and U. Hartl, vivo aspects protein folding and quality control. Science, a634444),, 2016.
[6] M. Bavarian, H. Jun, N. Tezak, Schulman, C. McLeavey, Tworek, M. Chen. Efficient training of language models to fill in the middle, 2022. URL https://arxiv org/ abs/ 2207 14255,
[7] Belliardo, G. D. Koutsovoulos, C. Rancurel, M. Clément, J. Lipuma, M. Bailly- Bechet, and G. J. Danchin. Improvement eukaryotic protein predictions from soil metagenomes. Scientifie Data, 9(1): 311, Jun 2022. ISSN 2052-4463. 1038/ 1193/44459770022014000
doi: 10. 10100///1157-022-0--22220
URL https://don org/10.
[8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Vosse G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Bemner, McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, Hadsell, Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Asso- ciates, Inc., 2020. URL httpe://proceeeeee (45T00060010055574100000088
neuriperrpp... heuripe.cc/pape.. files/paper/2000 hatpr//222/ffeee Paper pdf.
[9] B. Buchfink, K. Reuter, and H.-G. Drost. Sensitive protein alignments at trec-of- life scale using diamond. Nature Methods, 18(4):366 368, Apr 2021. ISSN 1548-7105. doi: 0001118/441122022--0000000 URL
https: http://dooo.rgg/////// /00 0030///4111200210000000100
[10] B. Bushnell. BBDuk, Mar. 2025. URL sourceforge metttpprojecss/bbmay original-date: 2016- 03-08713:29:362.
[11] S. Cabantous and G. S. Waldo. In vivo and in vitro protein solubility assays using split glp. Nature methods, 3 3(10):845-854 854, 2006.
[12] S. Cabantous, T. C. Terwilliger, and G. S. Waldo, Protein tagging and detection with engineered self- assembling fragments of green fluorescent protein. Nature Biotechnology, 102)-1022-077 Jan 2005. ISSN 1646-1696. doi: 10. 10.1008//b11044. URL https://doi. org. 10. 1038/nbt1044
[13] A. P. Camargo, S. Roux, F. Schulz, M. Babinski, Y. Xu, B. Hu, P. S. G. Chain, S. Nayfach, and N. C. Kyrpides. Identification of mobile genetic elements with genomad. Nature Biotechnology, 1303- 1312, Aug 2024. ISSN 1546 1696. doi: 10.10038/641587-03-0195533 URL (ttps://do1.org/// 1038/ 0035784158770223-11553
[14] Chaudhury, S. Lyskov, and J. Gray. Pyrosetta: a script-based interface for implementing molecular modeling algorithms using rosetta. Bioinformatics, 26(5):689-69) 2010.
[15] B. Chen, X. Cheng. P. Li, Y.a. Geng. J. Gong, S. Li. Z. Bei, X. Tan, B. Wang, X. Zeng, et al. xt- rimopgim: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint arXiv:2401.066 199999999999...... 2024.
[16] I.M. A. Chen, K. Chu, K. Palaniappan, A. Ratner, Huang, M. Huntemann, P. Hajek, S. Ritter, N. Varghese, Seshadri, Roux, Woyke, A. Aado-Fadrosh, N. N. Ivanova, and N. Kyrpides. The img/m data management and analysis system v.6.0: new tools and advanced capabilities. Nucleic Acids Research, 49(5)): D761-3335 D763, Oct 2020. ISSN 0305- 1048. doi: 10. 10.193//ar//kaa999, URL https:/ btbps:///oi... http///ii.ogg//////

---

[17] S. Chen. C. He. Y. Li, Z. Li, and C. E. Melançon III. A computational toolset for rapid identification of sars-cov-2, other viruses and microorganisms from sequencing data. Briefings in bioinformatics, 22(2): 924-935, 2021.
[18] T. Chen, B. Xu, Zhang, and C. Guestrin. Training deep nets with sublinear memory cost, 2016. URL attps://arxis.
[19] X. Cheng, Chen, Li. Gong, J. Tang, and L. Song. Training compule-optimal protein language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, edi- tors, Advances in Neural Information Processing Systems, volume 37, pages 69386-69418. Curran As- sociates, Inc., 2024, URL https://pproceedding.. neuraiiiiiiinnnnns....... nwuripe. cc/pape wwwes/paper//eeeeeeeee /02//////////11e///// 0rb.ag/4442220000111166666606662
[20]
W.-L. Chiang, L. Zheng. Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 4/st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 8359 8388. PMLR, 21-27 Jul 2024. URL https://proceediee
mir.presse wwwss//2355 chi ang24b.
[21] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, Sutton, Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko. J. Maynez. A. Rao, P. Barnes, Y. Tay. N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury. J. Austin, M. Isard, G. Gur-Ari, P. Yin. T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski. X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, Dohan, S. Agrawal, M. Omernick, A. M. Dai, S. Pillai, M. Pellat, A. Lowkowyez,
Morcira,
R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Sacta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113 2023. URL http://jmlr org/ papers/v24/22-1144 1244 html.
[22] P. F. Christiano, Leike, T. Brown, M. Martic, S. Legg. and D. Amodei. Deep reinforcement learning from human preferences. In Guyon, V. Luxburg, S. Bengio, H. Wallach, R. Forgus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https://proceeding neur seuripss.o cccppapcrccpcrcpppppppppp abszcuadaabUUcc91111i 20w.::///e par.
[23] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun. L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers solve math word problems, 2021. URL
https://arxiv: wwtpi//.rr//bb//110... 14111
[24] A. Cormman, J. West-Roberts, A. P. Camargo, S. Roux, M. Beracochea, M. Mirdita, S. Ovchinnikov, and Hwang. The OMG dataset: An open metagenomic corpus for mixed-modality genomic language modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openrevie wwt/ ddd
[25] P. Danecek, J. K. Bonfield, J. Liddle, J. Marshall, V. Ohan, M. O. Pollard, A. Whitwham, T. Keane, S. A. McCarthy, R. M. Davies, al. Twelve years of samtools and beftools. Gigascience, 10(2)giaa0088, 2021.
[26] T. Dao. FlashAttention 2: Faster attention with better parallelism and work partitioning, In International Conference on Learning Representations (ICLR), 2024.
[27] J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, Y. Leung. T. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker, Robust deep learningbased protein sequence design using proteinmpnn. Science, 378(661 (7786665))499556 5),4955,, 2022. doi: 10/ssee/seee add2187. URL https://www scwence. wwtpooog//oo/o// abto/////ssssll 10. 1 1126/ scissciiecso www.en..e...........aa add2187.
[28] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 233-941. PMLR, 06-11 htmll
Aug 2017. URL
mlr, caosss/v/0/dauphii
[29] DeepSeek-Ar AI, A. Liu, B. Feng. B. Xue, B. Wang. B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin. H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo.
12

---

J. N.. J. Li, J. Wang, J. Chen, J. Chen, J. Yuan, J. Qiu. J. Li, J. Song, K. Dong, K. Hu. K. Gao, K. Guan, K. Huang, K. Yu. L. Wang. L. Zhang, L. Xu, L. Xia, L. Zhao. L. Wang. L. Zhang, M. Li, M. Wang, M. Zhang, M. Zhang. M. Tang, M. Li, N. Tian, P. Huang. P. Wang, P. Zhang, Q. Wang. Q. Zhu, Q. Chen,
Q. Du, R. J. Chen, R. L. Jin, R. Gc, R. Zhang, R. Pan, R. Wang, R. Xu, R. Zhang, R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Yc, S. Ma, S. Wang. S. Zhou, S. Yu, S. Zhou, S. Pan, T. Wang. T. Yun,
T. Pei. T. Sun, W. L. Xiao, W. Zeng. W. Zhao, W. An, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, X. Q. Li, X. Jin, X. Wang. X. Bi, X. Liu, X. Wang. X. Shen, X. Chen, X. Zhang, X. Chen, X. Nie, X. Sun, Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yu, X. Song. X. Shan, X. Zhou, X. Yang. X. Li, X. Su, X. Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Y. Zhang, Y. Xu, Y. Xu, Y. Huang, Y. Li, Y. Zhao,
Y. Sun, Y. Li, Y. Wang, Y. Yu, Y. Zheng, Y. Zhang, Y. Shi, Y. Xiong, Y. He. Y. Tang. Y. Piao. Y. Wang.
Y. Tan. Y. Ma. Y. Liu. Y. Guo. Y. Wu, Y. Ou, Y. Zhu, Y. Wang. Y. Gong. Y. Zou. Y. He, Y. Zha, Y. Xiong. Ma. Yan, Y. Luo. Y. You, Y. Liu, Y. Zhou.
F. Wu, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu. Z. Huang, Z. Zhang, Z. Xie, Z. Zhang, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Xu, Z. Wu, Z. Zhang,
Z. Li, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Gao, and Z. Pan. Deepscck-v3 technical report,
2024. URL /ttps://arxiv www.wiv.rrrg/e/o wwwggb////2....... 19437.
[30] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre- Pre-training of deep bidirectional trans- formers for language understanding. In J. Burstein, C. Doran, and I. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, Jun 2019. Association for Computational Linguistics. doi: (0018653/11/N19-144 URL. https://acllantholee wwwp://y...//N///11..100
[31] F. Ding and J. Steinhardt. Protein language models are biased by unequal sequence sampling across the tree of life. bioRxiv, 2024. doi: 10111/202240300.5840000000 URL httpss:/www. http///wwwwwxxx...co// wworxiv.org. content/ear wttpa////alllll//////22//2
03.07 03.00.584001
[32] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Mach. Learn. Res., 23(1), Jan. 2022. ISSN 1532-4435.
[33] Q. Fournier, R. M. Vernon, A. van der Sloot, B. Schulz, S. Chandar, and C. J. Langmead. Protein language models: Is scaling necessary? bioRxiv, 2024. doi: 10.1101/2024.09 0tt..999...660000 URL https: //wwu. www.xiiorxiv.omm org/ /ttp///ente/eaall/////////////// 20224 09. 23, 614603.
[34] D. M. Fowler and S. Fields. Deep mutational scanning: a new style of protein science. Nature Methods,
l 18)8:800--877 Aug 2014. ISSN 1548-7105. doi: 00.1038/mmeth.307.. URL https://dol org/ 10. 1038/ 20t8/mmeth.3077
[35] D. Fried, A. Aghajanyam, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URL httpa://openreviees http://ppeeeeeeeeeee///ooomeeeee net/forum?dd 1U86-16M6ED.
Yih, L. Zettlemoyer, and
[36] M. Frith. A new repeat-maskinn method enables specific detection homologous sequences. Nucleic Acids Research, 3944)x233-223, Nov 2010. ISSN 0305-1048. doi: 10. (0.109///ar/ URL https!
7ddoi. owg/rog//// 10/g/10.100 1093/narz /tt///ar////// /2gq1222
[37] T. Gale, D. Narayanan, C. Young, and M. Zaharia. Megablocks: Efficient sparse training with mixture- ef-extsss. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems, volume 5. pages 288 304. Curan, 2023, URL https::/proceediinng wtw.:/ssswssss...o//aaaa.e.... org/paper files/ http://eee....eeeeeaeeeeeeeeeeea
pdd.
[38] C. Gordon, A. X. Lu, and P. Abbeel. Protein language model fitness is matter of preference. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevier net/forua?id-UuPPppAAnn
[39] M. r. Groth, M. Kerr, Olsen, Salomon, and W. Boomsma. Kermut: Composite kernel regression for protein variant effects. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang. editors. Advances in Neural Information P Processing Systems. volume 37, pages 29514-29565. Curran Associates, Inc.. 2024. URL https::/procceedinii neurips. . cc/paper files/paper/ htap://2232a770000eeaa11aaaaaaa2
Coaaaperr-ooofeeeeee Conferencee.....mooomooooooooc.e pdf.
[40]
T. Hayes, R. Rao, H. Akin, J. Sofroniew, D. Oktay, Z. Lin, R. Verkuil, V. Q. Tran, Deaton, M. Wig- gert, R. Badkundri, 1. Shafkat, Gong, A. Derry, R. Molina, N. Thomas, Y. A. Khan, C. Mishra, C. Kim, L. J. Bartie, M. Nemeth, P. D. Hsu, T. Sercu, S. Candido, and A. Rives. Simulating 500 million years of evolution with language model. Science, e00)e0dd) Jan 2025. doi: 10. 126/sceccee ads0018. URL https: //www//wwwwww// www.wssceencc... org/ 1ttp://bba///.1111//ceeeeee ads0018
13

---

[41] D. Hendrycks, C. Burns, S. Basart. A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring mas- sive multitask language understanding. In International Conference on Learning Reprexentations, 2021. URL https://openrevize http://wwwwweeeeeoooeeeeeeieeeee jmt
[42] B. L., Hie, K. K. Yang, and P. S. Kim. Evolutionary velocity with protein language models predicts evolutionary dynamies of diverse proteins. Cell Systems, (3(4))274-285.cc 2022. ISSN 2405-4712. doi: httpss//doi.org//
URL attps://wwk. http://www..cinnctrrrrcc.t wtt.e//aaccareee....oooaeeeieeea com/science/
[43] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E, Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals. J. W. Rae, and L. Sifre. Training compute- optimal large language models. In Proceedings of the 36th International Conference on Neural Infor- mation Processing Systems, NeurIPS 2022, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713877
[44] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degencration. In International Conference on Learning Representations, 2020. URL https://openreviokk neet/
Torum TYGGEEEEE
[45] C. Hsu. R. Verkuil. J. Liu, Z. Lin, B. Hie. T. Sercu, A. Lerer, and A. Rives. Learning inverse folding from millions of predicted structures. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8946-897777 8970, PMLR, 17-23 Jul 2022. URL https://proceeding: http://nwwww....pp///eeeemm..e/e wwwpp//esse.. hatsss/v12hhhsu22e html,
mlr
[46] D. Hyatt, G.-L. Chen, P. F. LoCascio, M. L. Land, F. W. Larimer, and L. J. Hauser. Prodigal: prokary- otic gene recognition and translation initiation site identification. BMC Bioinformatics, 11(1):1 19, Mar 2010. ISSN 1471-2105. doi: 10. 21111///471-210--1111
URL https://doi. org/10. 11867
1471-200511-11
[47] J. B. Ingraham, M. Baranov, Z. Costello, K. W. Barber, W. Wang. A. Ismail, V. Frappier, D. M. Lord, Ng- Mig-Thow-Hing Hing, E. Van Vlack, S. Tie, Xue, C. Cowles, A. Leung, V. Rodrigues, C. Morales-Perez, A. M. Ayouh, R. Green, K. Puentes, Oplinger, N. V. Panwar, F. Obermeyer, A. Root, A. Beam, J. Poelwijk, and G. Grigoryan. Illuminating protein space with a programmable generative model. Nature, 6 1078-078, Nov 2023. ISSN 1476-4687, doi: 10. 1038/04 1586-023-067788& URL https: org/ 10. 1038/ 8250/441188002-0072-7
[48] K. Janocha, A. Ling, A. Godson, Y. Lampi, S. Bornschein, and N. Y. Hammerla. Harnessing preference optimisation in protein Ims for hit maturation cell therapy, 2024. URL https://arxiv.. org./ii..rr///eo /bs//
2112.01388
[49] A. A. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang. T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024. URL https: http:///axvvvvvvvv org/ abs/ ab///401....2..../// 2401. 04008
[50] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunaknol, R. Bates, Židek, A. Potapenko, A. Bridgland, C. Meyer, A. A. Kohl, A. J. Ballard, A. Cowie, Romera- Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):83-588) Aug 2021. ISSN 1476-4687. doi: 00.0088641586--01-88812222 URL
Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, Petersen, D. Reiman,
https://doi.. http:///doooorr1100/ org/10. 1038
[51] J. Kaplan, S. McCandlish, T. Henighan, T. Brown, Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei. Scaling laws for neural language models, 2020. URL https://aaxiv: http//arrrv..or///a/aa//// org/abs/
2001 088851
[52] P. J. Keeling, F. Burki, H. M. Wilcox, B. Allam, E, E. Allen, L. A. Amaral-Zettler, E. V. Armbrust, J. M. Archibald, A. K. Bharti, C. Bell, B. Beszteri, K. D. Bidle, C. T. Cameron, L. Campbell, D. A. Caron, R. A. Cattolico, J. L. Collier, K. Coyne. S. K. Davy, P. Deschamps. S. T. Dyhrman, B. Edvardsen, R. D. Gates, J. Gobler, J. Greenwood, S. M. Guida, L. Jacobi, K. S. Jakobsen, E. R. James, B. Jenkins, U. John, M. D. Johnson, A. R. Juhl, A. Kamp. A. Katz, R. Kiene, A. Kudryavtsev, B. Leander, S. Lin, C. Lovejoy. D. Lynn, A. Marchetti, G. McManus, A. M. Nedelcu, S. Menden- Deuer, C. Miceli, T. Mock, M. Montresor, M. A. Moran, S. Murray, G. Nadathur, S. Nagai, P. B. Ngam, B. Palenik,
J. Pawlowski, G. Petroni, G. Piganeau, M. C. Posewitz, K. Rengefors, G. Romano, M. E. Rumpho,
14

---

T. Rynearson, K. B. Schilling, D. C. Schroeder. A. G. B. Simpson, C. H. Slamovits, D. R. Smith, G. J. Smith, S. R. Smith, H. M. Sosik, P. Stief, E. Theriot, S. N. Twary, P. E. Umale. D. Vaulot, B. Wawrik, G. L. Wheeler. W. H. Wilson, Xu, A. Zingone, and A. Z. Worden. The marine microbial eukaryote transcriptome sequencing project (mmetsp);; Illuminating the functional diversity of cukaryotic life in the occans through transcriptome sequencing. PLOS Biology, 12(6):1 6, Jun 2014. doi: 10.1371/jourm pbio. 100189, URL https: wttp///o...../.... 1371/ pbio. 1001889.
[53] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https: wtt./bs.g/bbbaaeeeee.. 1412. 6980,
[54] S. Kurtz, A. Phillippy, A. L. Delcher, M. Smoot, M. Shumway, C. Antonescu, and S. L. Salzberg, Versa- tile and open software for comparing large genomes. Genome biology. 5:1-9, 2004.
[55] R. Leinonen, R. Akhtar, E. Birney, L. Bower, A. Cerdeno-Tárraga, Y. Cheng, I. Cleland, N. Faruque, N. Goodgame, R. Gibson, G. Hoad, M. Jang, N. Pakseresht, S. Plaister, R. Radhakrishnan, K. Reddy, Sobhany, Ten Hoopen, R. Vaughan, V. Zalumin, and Cochrane. The european nucleotide archive. Nucleic Acids Research, 39(1))D22 D31, Oct 2010, ISSN 1048, doi: 10. 1693//////// URL attps://doi... org/ 10. w0t.0/000mmmm/a/////a gkq967
[56] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. (GS) Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL attps://openrest ew het/foormmidd-qreeeeeeeeeeeeeeee
[57] F. Levy Karin, M. Mirdita, and J. Südding. Senstuk-sensitive, high-throughput gene discovery, and annotation for large-scale eukaryotic metagenomics. Microbiome, 8(1 8(1):48, Apr 2020. ISSN 2049-2618. doi: 10. 186//40168-0000 68000000000xxxx URL https://doi. orw//o///// 1111/110016-0000000000000
[58] H. Li. Minimap2: pairwise alignment for nucleotide sequences. Bioinformatics, 34(18):3094-3100 2018.
[59] Z. Lin, H. Akin, R. Rao, B. Hic, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli, A. dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, and A. Rives. Exolutionary-scalc prediction of atomic-level protein structure with language model. Science, 379 2023. doi: 10. 1 1266science..dee25... URL https: /www www.wwwwwccc...wwwwwwwww.ww ott/ 1121a/........./////../ scwence.ecc.cccco..eo ade2574. 2221
[60] Lindman, A. Hernandez-Garcia, O. Svczepankiewicz, B. Frohm, and S. Linse. vivo protein stabilization hased on fragment complementation and a split GFP system. Proceedings of the Na- tional Academy of Sciences, 107(46):1986--12))
2010. doi: 10.1073/pnas. 10005//1..0005681000 1005689107. URL
https:/ /wwp:///ww... pnas. wwtp////////////////////oeareeea 107300011011....... 1005689107.
[61] P. Listwan, T. C. Terwilliger, and G. S. Waldo. Automated, high-throughput platform for protein sol- ubility screening using split-gfp system. Journal of structural and functional genomics, 10:47-55, 2009.
[62] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httts::/ wttp::/wwww...eteoeo/ooe/oorrooa
[63] A. Madani, B. Krause, F. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. I.. Olmos, C. Xiong, 7. Sun, R. Sncher, J. S. Fraser, and N. Naik. Large language models generate functional protein sequences across diverse families, Nature Biotechnology, 41(8): 1099-1190000111 1106, Aug 2023, ISSN 1546-1696 doi: 1LL110384415577-02--166633
URL https: //doi. org/ 10. 1038/ 022-6622111111812
[64] G. Marçais and C. Kingsford. A fast, lock-free approach for efficient parallel counting of occurrences of k-mers. Bioinformatiis, 764 770, 01 2011. ISSN 1367-4803, doi: 10. 1093/bioinfoomaticc//t0ll1.. URL https://dol. org/ 10. 1093/
[65] J. Meier, R. Rao. R. Verkuil, J. Liu, T. Sercu, and A. Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29287- 29303. Curran Associates, Inc., 2021. URL /ttps://proceeddnng nwwripe..c/ Papee7777..---........ pdf.
files/
[66] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL http::///opppeeeeeeeee////r////e
[67] P. Mistani and V. Mysore. Preference optimization of protein language models as a multi-objective binder design paradigm, 2024. URL https::/aawwv..rr///eee//// 04188 04187.
15

---

[68] E. Nijkamp. J. A. Ruffolo, E. N. Weinstein, N. Naik, and A. Madani. Progen2: Exploring the boundaries of protein language models. Cell Systems, (111)):66-978.4422 Nov 2023. ISSN 2405-4712. doi: 10. 1016/j.cels.2023.100 w0....ss..22....000 URL
1023.10.002
[69] P. Notin,
Kollasch, D. Ritter, L. van Nickerk, S. Paul, H. Spinner, N. Rollins, A. Shaw, R. Orenhuch, R. Weitzman, J. Frazer, M. Dias, D. Franceschi, Y. Gal, and D. Marks. Proteingym: Large-scale benchmarks for protein fitness prediction and design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine. editors, Ad- vances in Neural Information Processing Systems, volume 36, pages 64331-64379. Curran Asso- ciates, Inc., 2023. URL https://proceeeding: mew.riinnnninnnnpp.......... ww/ppppecccppppppppppppppp filess/paper/002//E1 65e33 Papeeeee--eePPPPr---aaa-a--- Datasets, and Banaaaeet__ad__BBccaaaaass. pdr,
[70] Notin, R. Weitzman, D. Marks, and Y. Gal. Proteinnpt: Improving protein property prediction and design with non-parametric transformers. In A. Oh, T. Naumann, A, Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 33529 33563. Curran Associates, Inc., 2023. URL tttps://prrccediiig www.eiiinnnnnnnunreeeeeeeeee cc/paper files/
[71] Y. Pang. W. Yuan, He, K. Cho, S. Sukhbaatar, and Weston. Iterative reasoning preference op- timization. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 1 116617-116637. Cur- ran Associates, Inc., 2024, URL https://proceedinngg neurips cc/paper hiiles/pappr/202// .pff.
f file/
[72] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killcen, Z. Lin, N. Gimelshein, Antiga, Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL sctps://proceedinng neuripa wwtpppppcccpppppppccppppppppeeep /iles//appp//0/01///11 00002228866771111100010177777720
[73] L. Pronk and M. H. Medema. Whokaryote: distinguishing cukaryotic and prokaryotic contigs in metagenomes based on gene structure. Microbial Gemomics, 805):000833, 2022. ISSN 2057-5858. doi: 10. 10.10999mgen.0.omm 0.0.000....... URL ottp:://bbl.org//00.
org..o..o111000. 10999//0...9999/m..... 0.000823.
[74] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, Ermon, and C. Finn, Direct preference opti- mization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https::/openreett
[75] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Li,,, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal Machine Learning Research, 21(140):1-61 2020. URL http: http://ww...ggpppppe///////////0 html]
[76] Rajbhandari,, J. Rasley, Ruwase, parameter models, 2020. URL https://arxiv http:://awwvvvoooe.eeee/ www.orrg///bggrooeee att.a110.00000
He. Zero: Memory optimizations toward training trillion
[77] C. Riquelme, Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8583 8595. Curran Associates, Inc., 2021. URL https: https:://proceedings neur ips.ripp.... CC/ ccppaper pdf.
[78] A. Rives, Meier, Sercu, Goyal, Z. Lin, J. Liu, Guo, M. Ott, L. Zitnick, J. Ma, and Fergus, Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2019. doi: 10.1101/622803. URL https:/ ://ps////www biorxiv. org/ content/100 100
1101/6228037
[79] J. A. Ruffolo, A. Bhatnagar, J. Beazer, S. Nayfach. J. Russ, E. Hill, R. Hussain. J. Gallagher, and A. Madani. Adapting protein language models for structure-conditiooed design. biokxiv, 2024. doi: 10. http://22.......606.0 2024. 08. 03. 506485.
[80] J. Ruffolo, Nayfach, Gallagher, A. Bhatnagar, Beazer, Hussain, J. Russ. J. Yip. E. Hill, M. Pacesa, A. J. Meeske, P. Cameron, and A. Madani. Design of highly functional genome editors by modeling the universe of crispr-cas cas sequences. bioRxiv. 2024, doi: 10. 2024 04. 22, 590591.
24.52999. URL
https://www biorxiv www.:org/ccooo /ttp///////ooooo////////////////
16

---

[81] E. W. Sayers, E. E. Bolton, J. Brister. K. Canese, J. Chan. D. C. Comeau, C. M. Farrell, M. Feldgarden.
A. M. Fine, K. Funk, E. Hatcher, S. Kannan, C. Kelly, S. Kim, W. Klimke, M. J. Landrum, S. Lathrop, Lu, T. L. Madden, A. Malheiro, A. Marchler-Bauer, T. D. Murphy, L. Phan. Pujar. S. H. Rangwala, V. A. Schneider, T. Tse, J. Wang, J. Ye, B. W. Trawick, K. D. Pruitt, and S. T. Sherry. Database resources of the national center for biotcchnology information in 2023. Nucleic Acids Research, S1(D)) D38, Nov 2022. ISSN 0305- 1048, doi: 10. 10.1093/nar/gkac1053 URL https://dol orw.:/...rrr.... 10937
nar/ nargkkac1332
[82] R. Schmirler, M. Heinzinger. and B. Rost. Fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications, 15(1):7407, Aug 2024. ISSN 2041-1723, doi: 10.1038/
0467-024-518444 URL https:///oo..or//1 10. 1038/ 541467-024-51844--
[83] D. Sgarbossa, C. Malbranke, and A.-F Bitbol. Protmamba: homology-aware but alignment- pro-
tein state space model. bioRxiv, 2024. doi: 10.1 10.1100/0224005244..222 24.555... URL https://www...... htt.pp/o//ww..oooxi...
org/ ht5p://otnaat//aar12///////////0
05. 24. 595730
[84] N. Shazeer. GLU variants improve transformer, 2020. URL. https://arxi:: wwwp://wwwwrrbbb///o.ooo
[85] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference Learning Representations, 2017. URL https::/openrevvees net/ffrummmmmeec Bii.orn
[86] R. W. Shuai, J. A. Ruffolo, and J. J. Gray. Iglm: Infilling language modeling for antibody sequence design. Cell Systems, 14(1 (411))979-999 2023.
[87] M. Steinegger, M. Mirdita, and J. Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Nature Methods, 16(7))603-60)) 606. Jul 2019. ISSN 1548-7105, doi: 10.
URL https://do1 otg/10. 01108///15922219----99
[88] F. Stocco, M. Artigues-Lleixa, A. Hunklinger, T. Widatalla, M. Guell, and N. Ferruz Guiding generative protein language models with reinforcement learning, 2025. URL https://arxiv. www.wwwwww.....oaeeoooooooooooet arg/abs/412. 2412. 129749
[89] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568(C). Feb 2024. ISSN 0925-2312. doi: n..c066).eeucomm2223.3333 222....00....2..........
URL https://doi. orggoog///// 10 1016/j neucom. 2023. 127063.
[90] N. Sun, S. Zou. T. Tao, S. Mahbub. D. Li, Y. Zhuang. H. Wang, X. Cheng, L. Song. and E. P. Xing. Mixture of experts enable efficient and effective protein understanding and design. bioRxiv, 2024. doi: 10.1101/2024.11.29900 62.5225.
2024 11 29. 625425.
[91] B. E. Suzek, H. Huang. P. McGarvey, R. Mazumder, and C. H. Wu. Uniref: comprehensive and non- redundant uniprot reference clusters. Bioinformatics, 23(10): Mar 2007. ISSN 1367-4803. doi: 10. 1000/bioinfoomaaii///mm0s URL https: org/ 10. wttp///ooooorratt.ioiooooooooooe
[92] Y. Tay, M. Dehghani, V. Q. Tran, X. Gareia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, Zheng, D. Zhou, N. Houlsby, and D. Metzler. UL22: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviin net/forum (tt-orror//--6B6LLLZZZT
[93] The Mosaic MI. Team. composer. httg:://ggthbb..cc///aaaaaaa//pp
2021.
[94] The Mosaic ML Team, streaming. https///ithabb.ccmmmmmaaaaaa///
2022.
[95] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, F, Ham- bro, F. Azhar, A. Rodriguez, A. Joulin, F. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023. URL https://arxiy, http://///wwoooo//////////////// wtgg/ba//3//0000 2391/ 13971.
[96] T. Truong Jr and T. Bepler. Poet: A generative model of protein families as sequences-of- sequences. In A. Oh. T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine. editors, Advances in Neural Information Processing Systems, volume 36, pages 77379-77415. Curran Asso- ciates, Inc., 2023. URI. https://proceediin neur neuri cc/pappcrccppppppppppppppp fwwee/ pttp://////ppp///////1eee. 05566626662222299972888883338332 Pawpeouaaabrraaa---aaaa wow.aparrrrorereeeeeeeeaa erence. pdf.
[97] K. Tsuboyama, Dauparas. J. Chen, E. Laine, Mohseni Behbahani, J. J. Weinstein, N. M. Man- gan, S. Ovchinnikov, and G. J. Rocklin. Mega-scale experimental analysis of protein folding sta- bility in biology and design. Nature, 620(7973):444 444, Aug 2023. ISSN 1476-4687. doi: 10.
011111888660000060 URL https:://oo..o org/ 10. 1038/ $41886 023 06328-6
17

---

[98] M. van Kempen, S. S. Kim, C. Tumescheit, M. Mirdita, J. Lee, C. L. M. Gilchrist, J. Söding, and M. Steinegger. Fast and accurate protein structure search with foldseek. Nature Biotechnology. 42(2): 243-246, Feb 2024. ISSN 1546-1696. doi: 10.1008841137--02--017333 URL https://doi.co
10.
023
[99] A. Vaswani, N. Shazeer. N. Parmar, J. Uszkoreit, Jones, A. N. Gomez, L. u. Kaiser, and Polosukhin. Attention is all you need. In Guyon, U. V. V.xuburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL (rr Paper .pdi
weweannggeenuuuure cc/papp httec/pppr_ffeeeeeee fwwes/paper///eeeeeoeeeaaaaaaeea http//ppaae/21///11eee. /2077/f11e/
Paper.
[100] A. Wang. Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bow- man. Superglue: A stickier benchmark for gereral-purpose language understanding sys- tems, In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and Gar- nett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL bttps://proceediin nwur cct.ippp.ceppprr...e files//pper/2209//E1 pdr.
[101] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019. URL https://openreet
[102] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https:// net/ waw.rmmmnddgggeoozeeeeae
[103] N. Weinstein, A. N. Amin, Frazer, and D. Marks. Non-identifiabiiiyy and the blessings of misspecification in models of molecular fitness. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttt: net/ dorum///eeea--
[104] B. L. Welch. The generalization of 'student's' problem when several different population variances are involved. Biometrika, 34(1-2):28 35, Jan 1947. doi: 10.2307/2332510.
[105] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with un- likelihood training. In International Conference on Learning Representations, 2020. URL https: //opparevee..nt//fouumntcccc hot.:forrum/eeeeleeeo0tttt
[106] T. Widatalla, R. Rafailov, and B. Hie. Aligning protein generative models with experimental fitness via direct preference optimization. bioRxiv, 2024. doi: 010110//224.0.220.9560.00 URL hitps: 20. 595026.
//wu.. biorxiv....v. org/ wtwpg//ooeeeteeaaeaeeeeeeeeeeeee ear wttp://eeelll//2aaae///// 1///22////////22222222
[107] Xiong, Y. Yang, D. He, K. Zheng. Zheng, C. Xing, H. Zhang. Lan, L. Wang, and Liu. On layer normalization the transformer architecture. In D. III and Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524-10533. PMIR, 13-18 Jul 2020. URL https: https://proceeddings www.mmrrrsss....... v119 xio.:/ooonnbbmmmmmm
[108] B. Zhang and R. Sennrich. Root mean square layer normalization. In H. Wallach, H. Larochelle, Beygelzimer, F. d' d'Alché-Buc, Fox, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 32, Curran Associates, Inc., 2019. URL Ectps: wew.eeeedddnnne..eni........ neurips. ..ff.
cc/ paper,
[109] J. Zhao, C. Zhang, and Luo. Contrastive fitness learning: Reprogramming protein language models for low-n learning of protein fitness landscape. In J. Ma, editor, Research in Computational Molecular Biology, pages 470-4 474, Cham, 2024, Springer Nature Switzerland. ISBN 978-1-0716 10--1-0716-3938 -4.
[110] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences, 2020. URL https://aaxiii org/ abs/1909 08593
[111] V. Zulkower and S. Rosser. DNA Chisel, a versatile sequence optimizer. Bioinformatics, 36( 36(16):4508 4509, 2020.
18

---

A Profluent Protein Atlas
To train ProGen3, we curated the Profluent Protein Atlas v1 (PPA-1). Table II summarizes the com- position of the dataset. We collected genomic and metagenomic datasets from IMG/M [T ENA [55]] and NCBI GenBank [II], and we performed protein coding prediction using prodigal-gy [I3, 9d]. We discarded contigs shorter than 1,000 bp well contigs with coding density <65% to remove nispredicted proteins from eukaryotic sequences [Z3]]. These were combined with pro- teins from UniRef [91]], NCBI NR [8]]] and five additional metagenome and eukaryotic-focused databases, resulting in total of 12.2B proteins. All proteins were subject to additional filters to remove proteins with >8000 amino acids, protein fragments, and proteins with invalid amino acids, k-mer repeats [80], or >50% low complexity regions [[66]. This resulted in a final filtered set of 3.4B proteins.
On this filtered set, we performed hierarchical clustering at 90%, 50%, and 30% amino acid identity using DIAMOND [9]. At each clustering step, used the representative sequences from the previ- ous round (e.g.. 50% identity clustering was performed on the 90% identity cluster representatives). Clustering at each level
conducted iteratively across four sensitivity settings, where the repre- sentatives from one setting were clustered further at the next. The number sensitivity levels varied by identity threshold: at 90% identity, applied only --linclust, while at 50% and 30% identity, we used --linclust, -- fast, --default, and --more- meore-sensiivv. This pipeline yielded 1.2B 90% ID clusters, 183M 50% ID clusters, and 1 113M 30% ID clusters.
We compare PPA-1 to other protein datasets in Table After filtering and clustering, PPA-1 of similar scale and diversity to the OMG dataset [24]]. It is of similar scale to the dataset used to train ESM3 [40]. but does not include any protein fragments while ESM3's s dataset does. It is also considerably larger than resources like UniRef [Р]] and BFD [50, 87].
Data Source
Type Metagenomic
Proteins (Total)
Proteins (Filtered)
Percentage
IMG/M
[6]]
5,313,407,975
1,402,061,25 1,3
41.28%
ENA
55
Metagenomic
3,324.676.018
946,087,610
27.85%
NCBI-nr
H
Genomic
542.683,057
471,482,099
13.88%
GenBank
Genomic
455,377,65
362,034,076
10.66%
UniRef90
Genomic
170,669,87
144,914,015
4.27%
SRC 87
Metagenomic
2,020,821,851
60,061,204 2004
1.77%
MMETSP [52]
Metagenomic
28,029,947
3,837,925
0.11%
SoilEuk
Metagenomic 5,366,252
3,450,399
0.10%
MERC
Metagenomic
291,261,719
1,994.696
0.06%
MetaEuk [7
Metagenomic
6,310,729
258,456
0.01%
Total
12,158,605,083
3.396,181,727
100%
Table 1: Composition of PPA-1.
Dataset
Fragments Removed
30% ID clusters
50% ID Clusters
Proteins
UniRef [91]]
69M
454M
BFD [50 87
66M
2204M
ESM3
X
3143M
OMG
[24
207M
3300M
PPA-1 (ours)
113M
183M
3396M
Table 2: Comparison of PPA-1 other large protein datasets.
B Pre-Traine Pre-Training
B.1 Implementation and Hyperparameters
All ProGen3 models are transformer based decoder-only models[99] with context length of 8192 tokens and Transformer++ optimizations [РS]: feedforward networks with SwiGLU activations
19

---

128. X4]. rotary positional embeddings (RoPE [89]), grouped query attention [T]] and pre-layer nor- malization [[оА]] using RMSNorm [[08]. For RoPE, we use 0 10 Unless otherwise specified. we replace all feed-forward layers with Mixture of Experts (MoE) that activates 2 experts (out of total) for each token [32, 85]::
MoE(x) = Σ (oftmax(Top2(W, FFN,
(2)
where I E IRd is a single token embedding, W E R&xd is linear routing layer, and FFN, : R R is an expert sub-network. We regularize our language modeling loss with the load balancing loss used by SwitchTransformer [62] multiplied by weight À 0.05.
All models are trained using the AdamW optimizer [53L 62] with 31 = 0.9, 32 = 0.95, and BF16
mixed precision [66í]. After initial warmup period, we decay the learning rate to 10% of its peak value following a cosine schedule. We leverage fully sharded data parallel training [76]] and gradient
checkpointing
[IS]] for memory-efficient distributed training.
We implement our models using PyTorch [
To improve efficiency, we use FlashAttention2 [20] and Megablocks [37]] for our attention and MoE layers, respectively. Finally, we orchestrate training and data loading with MosaicML's composer [XX] and streaming [[4] libraries, respec- tively. We trained all models on H100s hosted by MosaicMl./Datarricks Pre-training ProGen3-46B
took approximately 17 days on cluster of 256xH100.
Table B describes our model configurations and pre-training hyperparameters in more detail.
Params (Active)
Layers
dmodel
Attn Heads
dFFN
LR
WD
BSZ
WU
112M
(32M)
384
1152
5e-4
5e-6
1M
2000
219M (63M)
11
512
8
1536
5e-4
5e-6
1M
2000
339M (98M)
512
1536
5e-4
5e-6
1M
2000
762M (221M)
17
768
12
2304
5e-4
5e-6
1M
2000
1.4B (393M)
1024
16
3072
5e-4
5e-6
1M
2000
3.0B (866M)
24
1280
16
3840
5e-4
5e-6
1M
2000
10B (2.7B)
24
2048
16 (8 KV)
8192
5c-4
5e-6
2M
2000
46B
(12.6B)
32
4096
32 (8 KV)
14336
3,2c-4
4e-6
3M
10000
Table 3: Model configurations and pre- treiiirin hyperparameters. BSZ is the number of tokens per batch. WU is the number of warmup steps and is kept constant regardless of total training duration.
B.2 Mixture of Experts Performance
To validate the performance of sparse architectures, we train three models for 80B tokens: dense 1.4B parameter model, a sparse 1.4B parameter model, and a sparse 3B parameter model. Table we find that the dense 1 1.4B parameter model slightly outperforms the sparse 1.4B parameter model. However, the sparse 3B parameter model significantly outperforms both while requiring about half the FLOPs of the smaller dense model.
Architecture Dense
Parameters 1,386,084.355
Active Parameters
FLOPs
Validation Loss
1,386,084,352
1.10 x 1000
1.920
Sparse
1,355,871,22 2,,2
393,174.016
3.14 x 10
1,948
Sparse
2,989,920,000
866,369,280
6.93 x 1019
1.849
Table 4: Validation loss of dense and sparse architectures trained for 80B tokens. For a fixed number of parameters, dense models slightly outperform sparse ones. For a fixed FLOPs budget, sparse models outperform dense ones.
B.3 Training Tasks
Training Protein Language Models with standard autoregressive modeling objective supports gen- erating protein sequences from the N-to-C terminal C-to-N terminal, conditioned on the previous
20

---

Validation PPL (Distinct @ 50%)
Hyperparameter
CLM
GLM (L 10)
GLM 50)
GLM (L = 200)
Infilling to CLM Ratio
8.40
1:10
8.82
20.86
>10
>100
1:4
8,98
14.84
>10
>100
1:2
9.13
12.05
>10
>10
1:1
9,41
12.47
>10
>10
Mask Fractions and Span
Lengths
L 10, 0.15
9.13
12.05
>10
>100
L f 0.25
9.08
9.64
49.16
>100
T 70, 0.50
9.09
9.71
9.68
9.44
T, S 200, < 0.50
9.02
9.54
9.34
8.91
Positional Encoding Strategy
Naive
8,98
9.47
9.30
8.87
Dosition-Preserving Fuzzy Prestion-Preserving
8.99
9.08
8.86
8.56
8.90
8.30
8.87
8.56
Table 5: Validation Loss CLM and Single Span Infilling Tasks for various infilling setups.
residues in the sequence [63, [8]. This formulation precludes settings where only want to gen- erate a segment in the middle the protein while keeping both prior and subsequent amino acids fixed. To support this, augment the ProGen3 model training with infilling or generalized language modeling (GLM) learning objective, taking inspiration from multiple prior works in NLP and Protein Language Modeling [I5, 83
[6. 33 75, 92]
Infilling Training Details During ProGen3 training, we used the infilling objective for 1/3 of by the model (i.e., the GLM CLM ratio is 1:2). For each such case, sample N spans from the sequence, up until a maximum length fraction f. The span length of each span is sampled from truncated mixture Gaussians, where we assign equal probability to each of ((10, 5), N (30, 10), (700, (7), N 50), and (400, 100), denoted as (u,, σT. The maximum length fraction f is itself sampled from the (0.15, 0.25, 0.5, 0.8} with probabilities (0.28, 0.3, 0.28, 0.14) respectively.
the sequences
For each span, sample the start position uniformly from the protein sequence and replace the corresponding span of residues (span start, span start + span length) with single sentinel token <span i>. At the same time, the same sentinel token is appended followed by the residues in the span being replaced and finally an end of span token. The constructed sequence is then used train the model with a causal attention mask and autoregressive modeling loss over the whole sequence. Similar to ProGen2 [68], we use 1 token to denote the N-terminal and token denote the C-terminal, enabling us to perform both CLM and GLM tasks in the N-to-C or C-to-N direction.
the end of the sequence,
Position Preserving Fuzzy Encoding When the infilled residues are transposed the end of the sequence, they retain their original position ids, and the sentinel span tokens get the same position id as the first token of the infilled span. A drawback of this scheme is that the model will have strong bias towards exactly the same number of residues that have been removed from the prefix. To mitigate this and allow the model to stop early, for each infilled span, we add fuzzy length factor to the position ids of all tokens that come after that span. The fuzzy factor is sampled from a geometric distribution with a mean of 0.2 times the length of the span. We provide an example below with infilled spans highlighted in color.
21

---

Original Sequence: ASVVEFKAAWVVDYKKITTTTTEEEEEEI
Infilling Sequence,
<bos _gmm_ggmm>>> S VASVGE <span
K
T
Y
Position lds
0
1 2 2 5
12 13 14
15
16
17
18
19
20
<span 6> L
D
<eos>
<span 38>
KA
V
<eos 8pan>
<span
67
P
21
29 30
31 32 33
34
891011
12
21
22
23
Y E T <eos span>
24 25 26
27
Ablations We validated various design decisions described in the previous paragraphs using val- idation perplexity, measured on the sequences from the held-out clusters distinct at 50% from the training set for both CLM task (to measure any degradation in CLM performance) and single span infilling. For the latter, remove random span of length L from each validation example and measure the perplexity the suffix after the infilling transformation as described previously (i.e. per- plexity of tokens after the <eos> token). The results from these ablations Initially, we only infilled spans up to length 10 for infilling and varied the proportion of infilling examples seen during training. We observed that CLM performance shows small degradation (with Itt perplexity difference) as we vary the proportion from to 1:1, while infilling perplexity varies significantly. We selected ratio of 1:2 compromise. In addition, of models had very high perplexity for infilling spans of length 50 and 200. Therefore, in the next of experiments, we vary the length of spans and the mask fraction seen by the model. We that increasing lengths and mask fraction generally improve performance across all validation sets, validating our choice of using a mixture of Gaussians with mean lengths of up to 400. Finally, we observed that using position-ppeserving position encoding improved perplexities, while fuzzy length encoding did not lead any significant performance degradation. Hence, we adopted both of these enhancements.
be found in Table B.
also observed that this set
B.4 Importance of Warmup Duration
found that the 46B parameter model achieved validation loss lower than the compute-optimal frontier our scaling laws predicted. A key difference between this model and the smaller ones is that inereased its warmup from 2,000 steps to 10,000 steps stabilize training (Table S) This discrepancy could explain the gap in performance, as we find that using 8,000 steps of warmup instead of 2,000 steps improves the final training and validation loss of 1.4B parameter model trained for 80B tokens (80,000 steps) (Figure 5))
In Section P
2.8
warmup 2k (val loss 1.948) warmup 8k loss 1.939)
2.6
Training Loss
2.4
2.2
2
3
4
5
6
7
6
Training Tokens
1e10
Figure 5: Increasing warmup from 2000 steps to 8000 steps improves the final training and validation loss for 1.4B parameter model trained on 80B tokens.
22

---

Parameters
Pre-Train Tokens
3
# Epochhs (ProteinGym)
# Epochs (Megascale)
339M
200B
0.10
0.05
25
1.4B
500B
0.10
0.05
25
6
3B
500B
0.10
0.02
25
46B
1.5T
0.10
0.01
25
10
Table 6: Training hyperparameters for IRPO.
C Model Alignment
C.1
Implementation and Hyperparameters
The most relevant alignment algorithms for this paper direct preference optimization (DPO, [Z4]] and its generalization iterative reasoning preference optimization (IRPO, [[]]]) Let Pa be the model we are aligning, Pref be the pre-trained model, be our alignment dataset, be conditioning information, y x be a completion, and R be the reward assigned to x. IRPO can be formulated
min (.,, (e. RaD
E
Σ R.D
llo
Blog
Po (x) Pref a:)
33 log
Po( | Pref (1/ xr)
xr)
(3)
log po (yl x))
Σ
1/4
R ma= R........
The first term encourages the aligned model po to assign higher likelihood to preferred completions than dispreferred ones. 3 controls the amount we allow the aligned model po to deviate from the pre-trained model Pref; larger values allow less deviation. The richness of this loss also increases with the block size n, i.e. the number of completions that are directly comparable with each other. The second term is the negative log likelihood loss of the completions with the highest reward. we set the regularization strength
0, IRPO reduces to DPO.
In this paper, we let the conditioning information æ denote single DMS assay, y4 denote pro- tein characterized in that assay, and R; its experimentally measured fitness (AG for the Megascale dataset). We use a cosine learning rate schedule that decays to 50% of the peak learning rate over training and uses the first 10% of steps for linear warmup. We train all models with peak learning rate 3 10-6 and peak weight decay 3 x 10-8 For models smaller than ProGen3-46B, we run alignment jobs on 8xH100; each job takes at most 3hr. For ProGen3-460 we require 16xH100, and jobs take 2-6hr, depending on the size of the dataset.
For the ProteinGym experiments, we train a separate set of models for each assay in T. We use common set of hyperparameters across all configurations: the only exception is that decrease the batch size from 64 to 32 if the training dataset contains fewer than 1000 samples. Since we are only training on single assay, the block size is the same the batch size.
For all Megascale experiments, we a single model on the entire training set with a batch size of 1024 and block size of 64. We select the training duration via early stopping on the validation Spearman p. Table R reports additional training hyperparameters that vary between models.
C.2 IRPO Ablation Studies
We select the a hyperparameter and block size n for each model by training on the Megascale protein stability dataset and evaluating fitness prediction performance the Megascale validation set, as well as the model's perplexity on the validation set used for pre-training.
Table Z shows representative trends for a 1.4B parameter model. In general, increasing the block size n improves fitness prediction performance with minimal impact to validation perplexity. This makes sense because the number of comparisons in Equation B scales quadratically with n, so increasing n imbues the loss with much richer information about the overall fitness landscape. For this reason, proceed with fairly large block size of 64 wherever possible.
23

---

Model
Block Size
Spearman p
Validation Perplexity
1.4B (Pre-Trainn Trained)
0.471
8.150
1.4B
0.05
8
0.562
9.384
1.4B
0.05
16
0.639
9.607
1.4B
0.05
64
0.643
9.712
1.4B (DPÓ)
64
0.670
13.874
1,4B
0,02
64
0.650
10.551
1.4B
0.05
64
0.643
9.712
1.4B
0.1
64
0.629
9.241
Table Larger block sizes improve fitness prediction performance. Increasing the IRPO regulariza- tion coefficient improves validation perplexity at the cost of fitness prediction performance.
We also find that the NLL loss term of IRPO is an important regularizer that prevents the validation perplexity from degrading too much, i.. it allows aligned models to retain the broad knowledge about proteins that they gained during pre-training For each model size, select the value of which achieves the best fitness prediction performance while obtaining a validation perplexity no more than 2 points higher than the pre-trained model's. This process yields the values in Table R
C.3 ProteinGym Alignment Experiments
In Figure H. we report the fitness prediction performance of both pre-trained and aligned models on each of the 8 assays described in Table S. Figure !b in the main text reports the average Spearman p across these assays. Across the board, aligned models outperform their zero-shot counterparts.
For the five assays with least 500 single- single-mutation mutation variants (i.e. the ones where trained only on single-mutation variants), the performance of the aligned models improves monotonically with their scale. KERMUT and ConFit obtain similar performance to ProGen3-46B on the 4 activity assays; ConFit also performs similarly on CAPSD CAPSD_AAV2S, AAV2s_Sinai 2021, but KERMUT has meaningfully worse performance this assay (Table B
For the other three assays, there is no such correlation between model scale and fitness prediction performance, though using larger model almost never causes meaningful performance drop. This suggests that IRPO learns most effectively from rich single-mutation data, and that on balance, larger models benefit more from alignment than smaller models. Each of the baseline methods also has noticeably worse performance than ProGen3-46B on two of these three assays (Table El
We also observe that the two assays where all methods obtain the worst performance (GCN4_ YEAST_ YEAST_SStalll Sww.Slalllrrr..e 2018 and SPG1_STRS Si__STRSG___S Bacis__w____0 #u_20 both measure binding affinity. Given that the pre-training data contains no signals about a protein's 's binding partner, predicting pro- tein's binding affinity is likely a challenging task for models of this sort. However, access richer single-mutation mutation data may still improve performance for this task.
D
Split-GFP Methods and Supporting Information
D.1
Competent cell preparation
Two days in advance, E. coli BLJ-21 DE3 (either with without P15a-GFP1.nn 1-10) was struck onto LB agar with or without kanamycin 50 /g/L., and incubated overnight at 37 One day advance, a single colony was inoculated into 50 mL LB with or without kanamycin 50 pg/L in 250 mL shake flask, and incubated overnight at C 225 RPM. On the day of, the overnight pre-culture was inoc- ulated 1:100 into 500 mL to 2L LB with or without kanamycin 50 pg/mL 2L shake flasks, and incubated at 37 225 RPM, until OD600 was between 0.4 and 0.6. The flasks were then chilled for 30 minutes on ice and then pelleted by centrifugation (10min, 3500 RCF, 4ºC) chilled 500mL plastic centrifuge bottles. The supernatant was decanted and pellets were suspended in TFB I (100 mM rubidium chloride, 50 mM manganese chloride (MnC12 H20), 30 mM potassium acetate, 10 mM calcium chloride (CaC12H20), 15% v/v glycerol) at ratio of 1:10 TFBI to starting culture volume. Resuspended cells were incubated on ice for minutes and then pelleted by centrifuga- tion (10min, 3500 RCF, 4'C). The supernatant was decanted and pellets were suspended in TFB II

---

Dataset ID
Coarse
Max Mutations (Train)
# Train Variants
Max # Mutations (Test)
Test Variants
Selection
Type
D7PM05_ D7PMO5_CLYGR, Swwermeyereeeeeee 2022
Activity
1169
23
23,346
GFP ____EQI_____ Sarri_Sarissaaaa 2016
Activity
1084
15
50,630
NGWV12_9MAX Somermeyer 2022
Activity
1141
13
30,260
QSWTC7 [SWIC7_9CNID Somermayer 2022
Activity
1201
43
32,309
GCN4 YEAST Staller 2018
Binding
4
617
44
2021
SPG1_ SPG1_STRSC _u__016
Binding
2
2167
4
147,193
CAPSD A
Sinai 2021
Organismal
532
28
41,796
Fitness
HIS7
HIS7_YEAST YEAST Pokusaeva_2011
Organismal
1643
28
494,494
Fitness
Table 8: Summary of ProteinGym datasets used for alignment.
Avg
D7PM05
GFP
Q6WV12
Q8WTC7
GCN4
SPG1
CAPSD
HIS7
KERMUT
0.628
0.820
0.764
0.843
0.773
0,461
0,227
0,494
0.645
ConFit
0.679
0.871
0.829
0.845
0,800
0.250
0.460
0.714
0.585
ProGen3-46B
0.673
0.825
0.837
0.798
0.706
0.339
0.451
0.753
0.672
Table 9: Supervised fitness prediction performance different methods.
(10mM propppppaliiiooooppaessllllooooo acid (MOPS), 10mM rubidium chloride, 75 mM calcium chloride, 15% glycerol) at ratio of 1:25 TFBII to starting culture volume. The resuspensions were incubated on ice for minutes, before being gently homogenized by inversion, and aliquoted to 96w plates, before storage at C
D.2 Cloning of expression plasmid backbones
The expression plasmids for the PoI-GFP, and GFP p'eet-Pol-GFP, and p15a-GFPL110 re- spectively, were modeled after Cabantous 2005, synthesized by Twist Bioscience with overlapping fragments, assembled via NEB HiFi following manufacturer recommendations. transformed into 10-beta competent E. coli (New England Bioscience), and plated on LB agar containing the appro- priate antibiotic. Colonies were picked, miniprepped (Zymo), and sequence verified via long-read nanopore sequencing (Plasmidsaurus) (Table IT]).
D.3 Cloning of the GFP11 fusion libraries
The Pol insert libraries were codon optimized for E. coli with DNA Chisel [[Л]]] flanked with Bsal Type II restriction sites and synthesized as gene fragments via Twist Bioscience. Plate maps specified such that generated and wild type proteins from the same cluster were located on the same plate. Positive and negative controls (human dihydrofolate reductase DHFR, NP JP_000782.1, and human GPCR beta-3 adrenergic receptor, NP UP_000016.1, respectively) Additionally, several wells were left empty on each plate to permit background subtraction at the flu- orescent measurement stage. DNA fragments
added each plate.
suspended in 100 pl H20. Bsal-HFv2 Golden Gate reactions were performed using manufacturer recommendations (New England Bioscience) scaled to 10 μL per assembly. Golden Gate reactions consisted of 30 cycles (5 min 37 min 16ºC) followed by 5 min 60%C. μL completed Golden Gate reaction was transformed follow- ing manufacturer recommendations into BL-21 DE3 competent cells either with without p15a- GFP1-10 (New England Bioscience). Transformations were plated onto LB agar plates containing 75 μg/L spectinomycin either with or without wg/L kanamycin using Hamilton liquid handler. Up to four colonies per assembly were picked using Qpix robotic colony picker into liquid LB con- taining 75 pg//L spectinomycin with or without 50 pg//L kanamycin. Colony PCR was performed using primers splitgpf-Pol-fie Pol-fwd and splitgpf-Pol- (Table I[I) and RedTaq PCR mastermix (Thermo Fischer) following manufacturer recommendations, scaled to 10 pl.. To spot check, PCR reactions were diluted 1 1:40 in water and size-verified by capillary electrophoresis (Revvity LabChip).
25

---

D7PNOS_CLVED CLYGR, SUVGR_Somaaie Commrreyerrroo 2022
ABQW] Sarkisyan_201e
QeWV12 SMAXI Somemayyer_ 2022
Q#WTC7 SCNID Somermaya 2022
0.8
0.8
0.6
0.0
0.6
Zero-Shok
0.4
Shot Aligned
0.4
0.4
0.2
0.2
Alignd
0.0
GCN4 YEAST YEAST_Staaeerrreee Staleer
SPG1_STRSG 2016
CAPSD APS00AAV25,555
AM225,5in Sinal 2022
H157 YEAST, Pokusseva_ 2019
0.4
0.4
0.7
0.65
0.60
Aligned
0.2
Algred
0.5
Aligned
Algnet
0.2
0.4
0.1
0.0
0.3
10"
101H
20th
10"
10tH
10th
Number Parameters
Number Parameters
Numberr Parameters
Number Parameters
Figure 6: Fitness prediction by zero-shot and aligned models on the assays described in Table N.
D.4 Sequence verification
For each plate of assemblies, generated in silico pool of references sequences, correspond- ing to the PCR amplicons using primers splitgpf-Pol-fwd and splitgpf-Pol-ry (Table I[]. Prior to sequencing, all pools were pre-validated ensure sufficient sequence diversity using Jellyfish with kmer size set to 31 [64]]. A custom Python script was run across the resulting kmer file to ensure each member of the pool had kmer with a sufficient edit distance from all other members of the pool.
PCR reactions were pooled, size-selected (1x Beckman Coulter Ampure beads, 2x 80% ethanol wash), and sequenced via Oxford Nanopore PromethION sequencing using R 110.4.1 I flow cells (Plas- midsaurus). For each sequenced pool, Plasmidsaurus returned a single FASTQ containing all reads. Reference amplicons of the expected library members were compiled into a multi-FASTA. The FASTQ and reference FASTA went through the following analysis steps: All unique 31-mers per reference sequence were identified by Uniquekmer [[Z)]. Reads were filtered based on length. Reads with fewer than 50 bp were discarded using BBDuk [IC]. Reads were demultiplexed based on exact match to unique 31-mers using BBSeal [M Demultiplexed reads were mapped to refer- ence sequences using minimap2 [58]] Consensus sequence generated based on aligned reads using samtools [25]] Variants were called between the consensus and the reference using DNADiff [54]] Colonies were filtered for sequence fidelity, coverage, and depth (>10 reads covering the ORF), and single correct representative of each genotype was hitpicked using Hamilton liquid handler. Hit- pick plates were grown overnight liquid LB containing 5g/l. spectinomyein with or without 50 pg/l. kanamycin, mixed 1:1 with 50% glycerol, and stored at -80 until assay.
D.5 Split-GFP Assay
In general, the preparation and assay of the Pols and GFP1-10 and assay followed the protocols delineated in Cabantous, 2006 [IT]]. The split-GFP assay can be either in cells using two orthog- onal expression plasmids, or in lysate. In a pilot study, we observed a poor correlation between the two methods 2r -0.27, n 91 proteins), and that with GFP fluorescence under sequential induction (Figures D We further validated the sequential induction approach with SDS-PAGE of 88 proteins (Figure III) Although the relatively high limit of detection for SDS- IAG- prevents quantitative correlation across the range of measured GFP val- ues, we observed excellent identification of highly expressing and lowly expressing proteins. 19 of 23 proteins in the top quartile had distinct bands at the correct size, while 22 of 22 proteins in the bottom quartile had no detectable band (Figure H). the top quartile, two bands were absent. The other two proteins without proper migration were two highly-expressing proteins that failed to migrate on the gel, implying high stability or resistance to SDS-mediate denaturation (Figure IIO). While boiling (up to 100ºC, 10 min) or proteinase K treatment afforded partial purification, neither
abundance was better correlated
26

---

Plasmid Ple--Pol-GFPi
DNA Sequence 2
stppopiliicd agan tiing
cort goaa spreter oo coeng poogpiias
sa masiperse poppoocaaeo caagi cmpprraaaaa
for meghiiggann paa by oogan
AB Akstaooest
sagii ar ano
GCTAAC AAAAAATTINNN TTAA AGEAA AGE 7046
P15a-GFP1110 1-10
CACCAC WWA.GGGRCCCCCCOTCTTTTT TTC.CCCCCCCCCCCCCCCCCCCC
AGOCUMAAII
GTAATOGUSCICAT TAA.TOOCOOOOCCCOOOOOOOOOO CATTTAATTS SATTSTTTTOOOAAAAA TOO.OOOOOAAAAAAAA CEREAATTGAAAAA AAA.AAATTOOOOOAOOO CAA.GAAGGGGAAAATAOOOOOOAA WTW.AAAAATTTTCCCCCCCCCC ATTTCCATTTTTTTTTTTTTTT TGAAAC
CGGACATION Tare ACTEC
AGA
COOTIGA SAA.IGAAGGGTTTTOooo..o www.oettTTTTTT.T........ TOO.AAAGGGGAATT.... AAAAAAAAAAAA AACAAAAAAAAOOO MATAAAEEE DEGAAC AAAAAAAAAATOOAAAA COREEEAGGEEA ADC CACCAAAGAAATTOAE AEECAAECTOOON WAA.TAATTAAAACCC
CADOCAG TETTAAATOOOOAAAATTTTTTT SAAATATTTTTTOA DETEEEEEEEGE GAGALTTI POO.GAAAOOOOOOOAA AOTTTEEAAGGAAAAAAA SECCCTOOAGTTE COSTTRSAAA AETOU AATAAAATAAAAAAAA CAAACAAA
COOOOCAGTTT TTO POCCAGGE TOUGAA
LESC ACCCCOO COC.CCCCCCCCCCCCCC ACTITT JOCLUNT
FOGC TOsCe
COOOKTICTA ATCA ATCCC ATACCOOOO AOOCOAAOTTTT
TATTAAAIONNN TO CITTCI PECTIC TOMOO CIIC TTAAA TOOTOAA DAA TOCAT CUCCATTMAA AOGAaOCAGC
CAOIT
CGGTEM
TO
CEDCACSASAAACCC AAAAAAAAEEE TCA TEATTAAGROOGA
SETEOTTTTAAATE TTTTAATTTOOOEAAA AADORAAOOAG AAE ESCACCTTTOOE
GE OLOGGIGAT FOCEOOC LACOA TO CO CORCED ACA CALAAA
GELLEGGGAAA AATAAAAALA EEACTE TCAC TATMOOOOAA CAATTTTT TGACK
ATTEMAATAGAATAA TAAAATTTTTATTT AAAATRRCC ACAAT
TACCTITTOO
WOR MOTTAAAAAAAA YOU
STITICARGACTE TOECATGEAANNN BOGOOOG AAACCTTT
MAGAAAGGGOOOO POCTETT AAAAAATGAAAA
GIGAA MECTTTAOOTTTIII EETTITTTA AATTAATTOTAA PATAAAATTTTT TAATTAAAOOO
TAAAA MAAAAAAATTAAAAAAA TETCCGGACAAAAA.. ACAAAA TOULG TACAAC TTTAAC AAAC/CCCCCA
CAACAAATACTUC
AATTTACOOOO THOCCGA TOGK ECTTUT
TAACE CAAEEEAAAAAAAAA TOTOL TTT COCAAGGGGIIO AAAAAAAAAAAAAAAA
AGUTTEL TOP EAAE
TO
A TAAAAAA
TEG FOC
AATAATTAGCATA ATAAAAATTAOOOOOOO
AGCOSCUCATE AAACCAC VOCOGG IOGOO EOCTOOTTE
CA MOACT THO CAGGEAAAGGOOO CECTEEECTTTAEEE
ATTACOGTOATOOO TOOTITTTTTOOOTAAA TOCTIO ACCTACTOOOO
TOGATTOAATTE CACOT ECTTT
EZOTTOCAA
ACTORTAATAA
TIC AATATTTTTAAAA AETAATTTEE TECTI TIAA TTATT EITT AC SAAAAAA TAAGE ACAAA EGALAA CTEEE AAT ATTTAA ATATT
CAATACAACTAATTI TOCCCCTTTTTTTAT AAAATAAAAAAAAAAAA SATTTAAAAAAAAA
CATTICTTO TTX ACACTT TOTT CaCace AGCC
AA TC TCAGGATA PICTICT AA TACC TOGIA TOE TOTT TTC GOOOD POT TOOA FAACE TGC PCAT ACT
OOSA MOSAAAAATOE ATAATTICUU CAATIICTTTE ACCE TTAST CIEDAC ICATE IGEAACATE PAGE
ARPOOTARCTTTE MOCEEE AACAO
ROAL
OCTOLAUTATCGE COLLTACGIIATTTT MATTCAR AT TOCTAACOA MALAEAAAGEE AAAAAAAAAAAA CCCCCCCCCCCCCCCCCCCCOCCOOCCCOOOO CCOC AACCCAANOOOO MOCOOO TOUTO ACAACAACMO MOCAGGAAAAAA GA CATACATOOO CATOCOCACOOA AOOUROOO
CGECA COGUGATA AA TGRT TOU TIC
MADE GOGGGG AOGAAE TOACTA AGOE OAGG HOCRO FOR AACAT CAGAA
KAUCGMOMORCE TOOTC
DOTU
GA CAAMEACATTICAAA ATAATTTIO ACGAT AETT TROTT DET METEE MARAAAAAAAEE TEA TOGEST TIA ARECICTIO AARGGE AADEAATIO TUGGI TOTTGG TOCOOR
TOCC CACTTATGAGGGAAT CAAAECAATTAAAAAAAAAAA WAA.TTAAAATTTTTOOTTTTTACOO TOTTTTTTTTTTTOOCCCCOOOOOOOOOCCCO TOACTECCSGE
Table 10: Plasmids used in this study.
Plasmid splitgpf-Poll Powwfwd splitgpf-Pol-rv
DNA Sequence 2
guaggacgcaclgaaoggggte ugggggggggattgtccttt
Table 11: Primers used for the size and sequence validation of Pol inserts in GTPt-PPo-GFPPp-
led to proper migration of the bands. Interestingly, both of these proteins shared the same cluster with the most closely related natural protein
dehydratase FabZ.
Due to the higher correlation with SDS-PAGE, we chose the in-lysate approach for experimental validation of the broader set. We describe both approaches in more detail below.
D.5.1 Whole-cell assay
Glycerol stocks containing pTee--Po-GGFPj and p15a-GFP 1-10 plasmids were inoculated into 1 mL liquid LB containing 75 μg/L spectinomycin and 50 jg/liter kanamycin sulfate and grown overnight in shaking plate incubator (37'c, 1000RPM). In the morning, overnight growth was inoculated in quadrupticate plates 1:100 LB containing 75 μg/L spectinomycin and 50 jg/liter kanamycin sul- fate, and grown in a shaking plate incubator (37'C, 1000RPM) until OD600 reached 0.5 to 1.0. Pol-GFP11 expression was induced by the addition of anhydroterracycline
ethanol to final concentration of 250 ng/L. Simultancous GFP1-10 was induced by the addition of isopropyl 3-D-
27

---

120k
606
gfg/0D600 co-induction
Figure 7: Scatter plot of mean split- under sequential induction (y-axis) vs mean co-induction (x-axis), == 16 hours, for 91 unique proteins. Fach color is a unique protein sequence. Error bars standard deviation of replicates.
thiogalactopyranoside (IPTG) to a final concentration of 0.1 mM. Protein production continued for two hours in shaking incubator (37"C, 1000RPM). Cultures were diluted 1:1 in LB final vol- ume of 100 pl in a 384-well black clear bottom plate (Greiner 781 781097). Fluorescence and OD600 were read on Tecan Spark plate reader (Ex: 475nm, Em: 530 nm, Gain; 69, Z-position: 30000).
D.5.2 Lysate assay
Preparation of protein of interest: Glycerol stocks containing PTét-PPo-GFPP1 plasmids were inoculated into 1mL liquid LB containing 75 /g/L spectinomycin and grown overnight in shaking plate incubator (37 1000RPM). In the morning, overnight growth was inoculated 1:50 in quadru- plicate in 1mL LB containing 75 μg/L spectinomycin, and grown shaking plate incubator (37%C, (000RPM) until OD600 reached 0.5 to 1.0. Pol was induced by the addition of anhydrotetracycling in ethanol to a final concentration of 250 ng/L. Protein production continued for hours in shak- ing incubator C379C, 1000RPM). Culture-containin plates were spun down (3500 RCF, 5 minutes), decanted, and 100 μl BugBuster MasterMix (EMD Millipore) containing 25 sg//LL chlorampheni- col was added to lyse cells and stop protein expression. Cells were lysed by incubation in a shaking incubator (25eC, 300RPM, I hour). Lysates were stored overnight at 4" before assay.
Preparation of GFP1-10 P15a-GFPi-10 transformed into coli BL-21 DE3 competent cells (New England Bioscience) following manufacturer recommendations. A single colony was grown overnight in LB 50 pg/liter kanamycin sulfate, diluted 1:1 with 50% glycerol, and stored at -80 Biomass from the glycerol stock was inoculated into 50 ml. LB 50 pg/liter kanamycin sulfate in 250 mL shake flask, and incubated overnight C37 225 RPM). The following morning, the overnight culture was inoculated 1:100 into LB 50 pg/liter kanamycin sulfate in 2 L shake flask. After hours (37'C, 225 RPM), GFP 1-10 expression was induced with a final concentration of mM isopropyl ai-thhiggalccoopyrnnodde (IPTG). After five hours of incubation (379C, 225 RPM), the culture was placed 4 and left overnight. The following morning, the culture was spun down (3500 RCF x 15 minutes) and re-suspended in 1/5 of the culture volume (i.e. 200 mL for L culture) in lysis buffer containing 100 mm Tris HCI 7.5, 10% glycerol, and 150 mm sodium chloride (TNG buffer). The resuspended culture was sonicated on ice (VWR 76193-590 50% duty, 10 minutes, second pulse). The lysate was centrifuged (3500 RCF x 15 minutes) and the GFP,. containing supernatant was decanted.
28

---

D.5.3 In trans folding and fluorescence measurement
10 pl. of Pol-are1 lysate were added to 90uL. of the G-P1-10 lysate in a 384-well black clear bottom plate (Greiner 781097). Plates were sealed and incubated at 25 overnight. Fluorescence was read on Tecan Spark plate reader (Ex: 475nm, Em: 530 nm, Gain: 69, Z-position 30000), Samples were background subtracted using the mean fluorescence of the empty wells on a per plate basis.
D.6 SDS-PAGE validation
10 yL of cell lysate were diluted with 30 μL BugBuster Protein Extraction Reagent with 25 pg/mL chloramphenicol and then added 12 pl. Laemmli + DTT and pipet mixed. Samples were heated at C for min. 10 yL were loaded on Mini-PROTEAAN TGX Precast Gel (BioRad) and run at 120V for 50 minutes. Gels were rinsed DI water then opened and transferred into troughs filled with DI water. Troughs were gently shaken for 15 minutes on an Ohaus light duty orbital shaker at 130 rpm at room temperature. Then DI water was replaced and this wash process was repeated two more times. Gels were then stained with GelCode Blue Stain Reagent for 45 minutes while shaking 130 rpm. Gel was de-stained by decanting dye and adding water and shaking at 130 rpm for another hour. Photographs were taken via smartphone camera.
Kag 0FP
64
87
25
20
18
att 1523-5)
0517 1nr 2022.0)
Expocted size (-ADa)
120 24
18
18
18
52
Figure 9: SDS-PAGE of 9 proteins. Top rows show mean GFP fluorescence in either co- se- quential induction experiments, t= 16 hours. The bottom row shows expected size in kDa.
Figure 8: Count of SDS-PAGE bands ob- served correct size for 92 proteins, orga- nized by GFP quartile.
29

---

2211
414
Red
Figure 10: SDS-PAGE gels of induced proteins. Each lane labeled by plate position, background- subtracted GFP, and boolean call on band presence at the anticipated size. Negative control is un- induced plasmid.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our first claim is that we introduce compute-optimal scaling laws for sparse protein language models (PLMs), and use them to train 46B parameter sparse PLM on 1.5T tokens of data from curated data distribution. Section deseribes our model ar- chitecture, data distribution, and optimal scaling laws. Appendix A describes the dataset in more detail. Section E describes the validation of our models in the wet lab, showing that larger models generate viable proteins for a much wider diversity of protein families. Finally, Section E describes how alignment is effective technique for supervised fitness prediction, and that larger models benefit more for alignment.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We note in Section [ that the split-GFP assay does not directly measure protein function, which is an inherent limitation of any assay that measures properties of diverse protein families with divergent (and potentially unknown) function.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
31

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
There are no theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all relevant model architecture and pre-training details in Ap- pendix B all relevant details pertaining to model alignment in Appendix Q and all experi- mental methods for the split-GFP method in Appendix D We describe how we constructed our pre-training dataset in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
specific model and empirical evaluation,
While NeurlPS does not require releasing code, the conference does require all sub- reasonable avenue for reproducibility, which may depend
missions to provide on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
32

---

(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release the model code on GitHub and weights (up to 3B parameters) on Huggingface. However, we do release the dataset, training code, and ProGen3-46B weights.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, understand that this might be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.com wwwblic/ggides//Cee httes/CooeSSummissseeee asionnloicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All these details are enumerated throughout the main text and appendices. We provide all relevant model architecture and pre-training details in Appendix [B, details on the train/test splits for pre-training Appendix A, and all relevant details pertaining to model alignment (and the relevant datasets) in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
33

---

Question: Does the paper report eтror bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: For wet lab data, we include error bars corresponding to 2 standard deviations from the experiment. We not include error bars for most computational experiments be- cause they would be too expensive to construct. First, it would be too costly to replicate pre-training runs to obtaín error bars. Second, we perform model alignment using differ- ent models 9 different datasets; cross-validation would have been too expensive, use a single train/val/test split throughout.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of eπrors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We note in Appendix B. that pre-trained ProGen3-46B for approximately days on 256xH100 hosted by Databricks; all other pre-training jobs were considerably smaller. We also note in Appendix eil the amount of compute required for each align- ment training run (<3hr on 8xH100 for models smaller than ProGen3-4633 and 2-6hr on 16xH100 for ProGen3-46B).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neurips.cco.. httppp///.oorrrppccp/p/////hchaa
Answer: [Yes]
34

---

Justification: We do no research and use no data involving human subjects, and all our data complies with the terms of the datasets use. We consider potential negative ramifications of our work and determine that the potential to improve society greatly outweighs any theoretical risks (see "Safety and Ethics" section). We also include notes on responsible development in our GitHub repo and Huggingface model cards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discuss the broader impacts of protein language models in both the intro- duction and discussion sections, where we overview wide range of protein engineering use cases that models like ProGen3 enable. We also discuss potential negative impacts in the Safety and Ethics section after the discussion.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We describe appropriate safeguards in the "Safety and Ethics" section at the end of the main text.
Guidelines:
The answer NA means that the paper poses such risks.
35

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all raw data repositories used to create our dataset in Appendix At these are the only assets we use that we did produce ourselves. Licenses for the datasets are respected, but we do not list them in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, paperswithoon thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We release well-doce model code on GitHub, along with weights for all models up to ProGen3-3B on Huggingface.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
36

---

Answer:
[NA]
Justification:
There was no crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: None the work in this paper involved human study participants.
Guidelines:
The answer NA
that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This paper does not use traditional LLMs anywhere.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non nta-staddar components.
Please refer to LLM policy for what should or should not be described.
37


Reviews:
[{"review_id": "wh2k8miSWW", "reviewer": "Reviewer_GjZd", "summary": "This paper presents ProGen3, a protein LLM scaled up to 46 billion parameters.  The paper goes into significant depth on the development of a new training dataset and its impacts on performance as the number of parameters/data is scaled.  The paper then proceeds with significant evaluation of the ProGen3 family of models, finding that as parameters scale the model is able to generate more diverse protein sequences.  Samples from ProGen3 are evaluated in the wet lab where it is demonstrated that the generated proteins express.  Finally, the authors perform preference alignment on the ProGen3 model and demonstrate that scaling impacts model performance after fine-tuning.", "questions": "1. What are the authors' hunches for why the largest models don't lead to better zero-shot fitness prediction?\n\n2. Similarly, what are the authors' intuitions for why scaling model size doesn't seem to impact expression rates all that much besides on the infilling task?  Also, why do the authors think scaling improves infilling performance in particular?\n\n3. Will model weights for the 46B parameter model as well as the training dataset be open-sourced?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "This paper proposes ProGen3 and details its training procedures (including a new training dataset) and goes into great depth on analysis of the impact of scaling on downstream performance.  The writing is clear and concise and the findings are quite comprehensive.  I would like to in particular commend the detail in this paper on the specifics of scaling up ProGen3 and the various design factors which impacted downstream performance.  Moreover, the evaluation of the model was very comprehensive, covering many different axes of model performance, including wet lab validation.\n\nI do have one concern regarding the reproducibility of the experiments detailed in the paper.  While ProGen3 model weights have been released on HuggingFace for model sizes up to 3B, the 46B parameter model has not been open sourced.  Further, as training set mixes and setups are one of the key drivers of performance in LLM training, it would be very beneficial if the PPA-1 training set described in the manuscript were open sourced, especially as small discrepancies in training sets can cause large impacts on downstream model performance.  I would strongly encourage the authors to consider open sourcing both of these to better the reproducibility of their work, as training a 46B model on 256 H100s for 17 days is impractical for nearly any academic lab.", "was_revised": true, "final_justification": "The paper is well motivated, delves into the engineering of training a scaled PLM, and shows impressive wet lab results.  As such, I recommend acceptance for this paper.", "rebuttal": "Thank you for your thoughtful feedback. Addressing your points one at a time,\n1. As you mentioned, we have open sourced all model sizes up to 3B parameters, but we do not intend to open source PPA-1 or ProGen3-46B.\n2. As models grow larger, they improve at estimating the distribution of naturally occurring proteins. However, since the forces driving evolution don’t always correlate with the experimental read-outs of laboratory assays, the average protein found in nature is rarely the most optimal. Thus, effectively modeling the evolutionary distribution of proteins can be at odds with zero-shot fitness prediction, since the most optimal variants may be quite unlikely in nature. Others have also theorized about and documented this trend [1-2], and we allude to their work in Lines 219-226. Rather, we find that larger models learn better representations that are more useful for downstream tasks.\n3. Based on our findings, we expect that it is actually quite easy for ProGen3 models to generate expressing (but potentially non-functional) proteins, *provided that those proteins come from families that are in-distribution for the model*. Consequently, unconditional generations from all models have similar expression rates when compared head-to-head, but larger models trained on more data can generate expressing proteins for families that are out-of-distribution for the smaller models. For the infilling task, we pre-selected protein scaffolds for the model to infill, and a number of these scaffolds happened to be in-distribution for ProGen3-46B but out-of-distribution for ProGen3-3B (but not vice versa).\n\n[1] Weinstein et al. \"Non-identifiability and the blessings of misspecification in models of molecular fitness.” NeurIPS, 2022.\n\n[2] Gordon et al. “Protein language model fitness is a matter of preference.” ICLR, 2025."}, {"review_id": "A4meJUfJAP", "reviewer": "Reviewer_RnRR", "summary": "This paper presents a suite of MoE PLMs named ProGen3 that demonstrate strong scaling performance on not only the pretraining task, but also after being finetuned for downstream tasks such as fitness prediction and stable protein sequence generation. Wet lab experiments also verify that the generated sequences are diverse and have high expression rate.", "questions": "1. Would the fitness prediction performance improve if you used an alignment algorithm that incorporated the actual experimental fitness values (e.g. offline/off-policy RLHF or policy gradient), rather than only ranked pairs?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors conduct an extensive scaling study on the output proteins (and not just the embeddings) of ProGen3 at multiple model sizes and with a validation metric that also accounts for OOD proteins. Such scaling studies are very underexplored in the PLM literature. The authors also empirically identified a data mixture that yielded lower OOD loss than a uniform or unmodified mixture.\n- A large proportion of the generated proteins have are viable/expressible, even for the generations with <30%ID to any of the natural proteins in the data.\n- The ProGen3 models also demonstrate strong performance when finetuned for downstream tasks such as fitness prediction and generation of stable proteins. \n- The ProGen3 suite of models is also provided at a wide range of model scales, making it suitable for a variety of downstream uses as well as further scaling studies.\n- The paper is extremely detailed, providing extensive explanations of both the dry and wet lab experiments.\n\nWeaknesses:\n- The use of IRPO/preference learning for supervised fitness prediction seems poorly motivated -- since IRPO (and most offline preference learning algorithms) trains on ranked pairs, using this algorithm necessarily involves some loss of information (i.e., the magnitudes of the experimentally-determined fitness values). Additionally, such preference learning algorithms have been shown to struggle with learning likelihoods that correctly align with the rankings of the data (see [Chen et al.](https://arxiv.org/abs/2405.19534)). Although it is trendy now to use IRPO and other rankings-based preference learning methods for fine-tuning pLMs, using it for a regression task is not very principled. (Additionally, it is an unfortunate fact that many popular offline preference learning algorithms still show a non-monotonic relationship between ranking performance and generality quality, so it may be better to not try to optimize for both at once with these algorithms. See [Tang et al.](https://arxiv.org/abs/2405.08448) and Sec. 5 of [Chen et al.](https://arxiv.org/abs/2405.19534) for evidence of this.)\n- Very minor nits:\n  - I find figure 1(a) to be somewhat incomprehensible. Even as someone who already understands what infilling generation is, I still did not understand this diagram.\n  - Typo in Fig. 3 caption: \"generated generated\" -> \"generated\"", "was_revised": true, "final_justification": "I remain satisfied with the paper and did not identify any reasons to either increase or decrease my score after the discussion.", "rebuttal": "Thank you for your thoughtful feedback and the highly relevant references. We did some early experiments with offline preference learning algorithms like LiPO [1] that make full use of the continuous fitness values, but we found that they generally underperformed DPO/IRPO. Prior work has also shown that contrastive ranking losses can be more effective than regression losses for protein variant effect prediction, especially in few-shot settings like those we consider in Figure 4b [2-3]. However, we agree that moving beyond ranked pairs is an important direction for future research in aligning protein language models with continuous-valued experimental data.\n\nYour points about the gap between predictive & generative capabilities and between online & offline algorithms are interesting ones. We do find that aligning on the Megascale stability dataset allows us to generate proteins that are more stable both computationally (Figure 4f) and experimentally (Figure 4g); additionally, larger models generate more stable proteins than smaller models (Figure 4f), matching the trends of the stability prediction results (Figure 4c-e). A key difference between our setting and the NLP one is that we select <100 low-perplexity sequences out of 50,000+ generated candidates, while in NLP, LLMs rarely generate >100 candidates. Our setting may therefore increase the importance of the model’s ranking capabilities and mitigate some of its generative shortcomings.\n\nWe chose not to use an online algorithm due to the additional complexities it introduces: we would need to train a different reward model for each dataset, ensure that reward model generalized from the local mutational landscapes it was trained on to model generations, and fine-tune and/or prompt ProGen3 for each dataset to ensure that we sample generations from the right protein cluster. Given that we trained 4 models on 9 datasets each, it was impractical to fully tune all these hyperparameters for each setting, but comparing online and offline methods (especially in settings where the generations from each can be tested in the lab) is an exciting future direction.\n\nWe are more than happy to address your minor points, especially if you have any specific feedback on what would make Figure 1a more interpretable.\n\n[1] Liu et al. “LiPO: Listwise Preference Optimization through Learning-to-Rank.” NAACL, 2025.\n\n[2] Brookes et al. “Contrastive losses as generalized models of global epistasis.” NeurIPS, 2024.\n\n[3] Zhao et al. “Contrastive Fitness Learning: Reprogramming Protein Language Models for Low-N Learning of Protein Fitness Landscape.” RECOMB, 2024."}, {"review_id": "BNWXZHm73x", "reviewer": "Reviewer_RM9H", "summary": "This paper presents ProGen3, a sparse mixture-of-experts generative protein language model. The authors explore Scaling-Laws in this architecture and introduce an optimized pre-training dataset, PPA v1. Experimental results demonstrate that larger PLMs can generate much wider diversity across protein families. Furthermore, after using iterative reasoning preference optimization (IRPO) with wet lab data, ProGen3 demonstrates strong performance on protein fitness prediction tasks.", "questions": "1. The authors show that IRPO with wet lab data improves ProGen3’s performance on protein fitness prediction. Does this strategy generalize to other downstream tasks (e.g., protein property prediction), or would each task require different experiments? Additional results on various tasks would strengthen the value of wet lab results.\n2. The authors demonstrate the effectiveness of IRPO with wet lab data. Is this approach model-agnostic? If the same data and IRPO method were applied to ProteinDPO [5] or other PLMs, could even better results be obtained? Such discussion or experiments would strengthen the paper’s contribution to data and experimental methodology.\n\n\n\n[1] Simulating 500 million years of evolution with a language model\n\n[2] Mixture of Experts Enable Efficient and Effective Protein Understanding and Design\n\n[3] ProGen2: Exploring the boundaries of protein language models\n\n[4] Training Compute-Optimal Protein Language Models\n\n[5] Aligning protein generative models with experimental fitness via Direct Preference Optimization", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The authors curate and present the PPA-1 dataset, showing the advantages of training on de-biased data.\n2. They optimize the LLM with wet lab data and highlight the effectiveness of such experimental reinforcement.\n3. They train a large (46B parameter) generative protein language model, providing a new base model for the field.\n\nWeaknesses:\n\n1. The paper lacks comparisons of protein generation capabilities with other PLM models, such as ESM3 [1], AIDO.Protein [2], and ProGen2 [3], all of which also support protein generation. It would strengthen the work to include direct comparisons to demonstrate ProGen3’s advantages over these models.\n2. In Section 3.1, the authors claim that “larger PLMs generate more diverse proteins.” However, larger models were trained with more data (Lines 135-137), so it is unclear if the gains come from the increased model capacity or simply from more diverse training data. The necessity of scaling up model size should be better demonstrated.\n3. Methodologically, the innovations seem somewhat incremental. Prior works have already validated MoE architectures for PLMs [2], explored scaling laws with different PLM architectures [3], and shown that preference optimization improves model fitness prediction ability [5]. While the value of using wet lab data is acknowledged, the technical novelty warrants further clarification.", "was_revised": true, "final_justification": "The authors' reply solve my questions. I decide to raise my rating.", "rebuttal": "Thank you for your thoughtful feedback. Addressing your points one at a time,\n1. In Section 4, we do compare applying IRPO to ProGen3 against state-of-the-art supervised fitness prediction methods ProteinDPO [1] for stability prediction, and ConFit [2] and KERMUT [3] for general fitness prediction. However, we will add additional comparisons to ProGen2-XL [4], a 6.4B parameter dense autoregressive model, for both unconditional generation and alignment. We find that only only 4.1% of generations from ProGen2-XL (out of 4M total) are valid proteins (compared to 39.8% / 49.3% from ProGen3-3B/46B), and they cover 0.82x / 0.66x as many clusters as ProGen3-3B/46B. For a fixed parameter count, ProGen3 also outperforms ProGen2 on all our alignment benchmarks, but ProGen3 is much more efficient due to its sparse MoE implementation. We do not compare against ESM3 [5] because its license precludes us from using it, and we do not compare against AIDO.Protein [6] or ESM-IF1 [7] because they perform structure-conditioned generation, not unconditional generation.\n2. While it is true that the larger models in Section 3 were trained on more data, the point still stands that generation diversity improves when using more pre-training FLOPs. The mechanism is simple: a lower validation loss implies a better understanding of the natural protein distribution, especially for rare or out-of-distribution proteins (Lines 93-97). However, to address your concern more directly, we will create supplementary figures that compare the quality and diversity of generations from ProGen3-339M trained on 200B tokens (PG3-339M_200B), ProGen3-3B trained on 200B tokens (PG3-3B_200B), and ProGen3-3B trained on 500B tokens (PG3-3B_500B). We find that increasing ProGen3-3B’s dataset size from 200B to 500B tokens has a much smaller impact than scaling up from 339M to 3B parameters. 30.0%, 37.8%, and 39.8% of the generations are valid from PG3-339M_200B, PG3B-3B_200B, and PG3-3B_500B, respectively. Additionally, PG3-3B_500B covers 81% more clusters than PG3-339M_200B, but only 14% more clusters than PG3-3B_200B.\n3. We are the first to systematically study the effect of the pre-training data distribution for PLMs, and we find that it has a significant effect on model performance (Figure 1d). Given that prior work has shown that the data distribution [8] and pre-training task [9] can affect scaling behavior, we feel that deriving compute-optimal scaling laws for this optimized distribution and for the infilling task are significant contributions. The fact that our results agree with Cheng et al. [9], who operate in a totally different setting, demonstrates the robustness of these scaling laws for autoregressive PLMs more broadly. We also performed the first wet lab experiment to characterize the impact of model scaling on the quality and diversity of generated sequences, and the findings in Section 3 are highly novel. Finally, while prior work has shown that preference optimization with wet lab data improves PLM performance, we are the first to show that larger pre-trained models yield better aligned models, especially in few-shot settings (Figure 4b).\n4. IRPO can be used to increase the correlation of a model’s likelihood with any metric that we wish to optimize, but it is not suited for multi-class classification tasks. Section 4 shows that IRPO can greatly improve the correlation of ProGen3’s likelihoods with a wide range of protein properties. For general properties like thermostability, we show that aligned models can generalize to protein families that are highly distinct from those they were trained on, both for fitness prediction and sequence generation (Lines 251-254, Figures 4c-4g). We also show that for properties that are more specific to a protein family, including activity, binding affinity, and organismal fitness, as few as 500 experimental samples (a modest number that many labs can reasonably generate) can greatly improve the model’s performance for that specific task (Figure 4b, Supplementary Tables 8-9).\n5. IRPO is a model-agnostic method. It is a variant of DPO that adds a negative log likelihood term to DPO’s contrastive loss (Appendix D.1). To strengthen our paper and address your concerns about having an apples-to-apples comparison with a different model family, we will include the results of applying IRPO to ProGen2-XL [4] in our revisions (see point 1 for more discussion of baselines). As mentioned above, we find that ProGen3 outperforms ProGen2 across the board. That said, similar to how IRPO is a variant of DPO that we found was well-optimized for ProGen3, ProteinDPO [1] is a variant of DPO optimized for stability prediction with ESM-IF1 [7], an inverse folding model that generates a protein sequence conditioned on its structure, and ConFit [2] is a variant of DPO that is optimized for ESM2 [10], a masked language model. Note that since masked language models don’t have well-defined likelihoods, vanilla DPO/IRPO cannot be applied to them. Moreover, we used identical train/val/test splits to evaluate each method. Thus, the baselines already in our paper are essentially the comparisons you are asking for, and we show that the aligned ProGen3-46B outperforms ProteinDPO and matches ConFit while being a much more flexible model that is capable of unconditional sequence generation.\n\n[1] Widatalla et al. “Aligning protein generative models with experimental fitness via Direct Preference Optimization.” bioRxiv, 2024.\n\n[2] Zhao et al. “Contrastive Fitness Learning: Reprogramming Protein Language Models for Low-N Learning of Protein Fitness Landscape.” RECOMB, 2024.\n\n[3] Groth et al. “Kermut: Composite kernel regression for protein variant effects.” NeurIPS, 2024.\n\n[4] Nijkamp et al. “ProGen2: Exploring the boundaries of protein language models.” Cell Systems, 2023.\n\n[5] Hayes et al. “Simulating 500 million years of evolution with a language model.” Science, 2025.\n\n[6] Sun et al. “Mixture of Experts Enable Efficient and Effective Protein Understanding and Design.” bioRxiv, 2024.\n\n[7] Hsu et al. \"Learning inverse folding from millions of predicted structures.\" PMLR, 2022.\n\n[8] Mayilvahanan et al. “LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws.” ICML, 2025.\n\n[9] Cheng et al. “Training Compute-Optimal Protein Language Models.” NeurIPS, 2024.\n\n[10] Lin et al. “Evolutionary-scale prediction of atomic-level protein structure with a language model.” Science, 2023."}, {"review_id": "s4VvZ9dJ9v", "reviewer": "Reviewer_Hphy", "summary": "This paper presents a systematic study of scaling laws in protein language models (PLMs), using a newly curated dataset called PPA that contains over 150 million protein sequences grouped into clusters at different sequence identity thresholds. The authors train a suite of ProGen3 models ranging from 44M to 4.7B parameters, analyzing how model scale and training data distribution affect generative quality, functional diversity, and downstream performance.", "questions": "- Is there a particular reason why Figures 1e and 4a only show a single FLOP-performance point for the 4B model, whereas other model sizes are represented with full scaling curves?\n\n- The paper mentions that segments were randomly masked for the infilling objective. Could this result in overly easy infilling tasks compared to biologically meaningful challenges? Please consider discussing alternative infilling objectives, such as fixed-region scaffolding or motif-conditioned design, which may provide more biologically relevant evaluation.\n\n- While the paper focuses on scaling within the ProGen3 framework, incorporating additional baseline models such as ESM-3 into perplexity, diversity, or cluster coverage plots (even if just as reference lines) would better contextualize ProGen3’s performance relative to existing methods.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper introduces a well-curated and biologically meaningful dataset (PPA), organized with sequence clustering and rigorous data splits, which provides a strong foundation for benchmarking PLMs.\n\n- It presents a thorough empirical study of how training distribution and model scale influence protein generation, offering insights that can benefit the field without requiring similarly expensive experiments.nd similar computational cost, especially in academic settings, where resources are constrained.\n\n- Beyond language modeling metrics like perplexity, the study evaluates sequence diversity and functional viability using in vitro experiments on generated sequences from the luciferase family. The paper offers practical insights into how data and model scaling impact protein generation and design, particularly in resource-constrained academic settings.\n\n\nWeakness\n- While the paper makes strong contributions in dataset curation, model scaling, and evaluation, it would benefit from a clear commitment to open-sourcing resources. In particular, the authors should clarify whether they intend to release the PPA dataset, trained model checkpoints, and training/inference code. Making these resources public would significantly enhance the reproducibility and long-term impact of the work.\n\n- While ProGen3 demonstrates promising performance, virtually all benchmarks presented in the paper are internal—comparing different variants of ProGen3 across model sizes or training distributions. Since this work introduces a new model family, it would be valuable to include direct comparisons with existing models such as ESM-3 to better contextualize ProGen3’s capabilities relative to the current state of the art.", "was_revised": true, "final_justification": "The authors have addressed my questions/concerns. I would keep my recommendation for the acceptance of this submission.", "rebuttal": "Thank you for your thoughtful feedback. We want to briefly correct the claim that “The authors train a suite of ProGen3 models ranging from 44M to 4.7B parameters” -- ProGen3 models range in size from 112M to 46B parameters. Addressing your points one at a time,\n1. We have open sourced all model sizes up to 3B parameters, but we do not intend to open source PPA-1 or ProGen3-46B.\n2. In Section 4, we do compare applying IRPO to ProGen3 against state-of-the-art supervised fitness prediction methods ProteinDPO [1] for stability prediction, and ConFit [2] and KERMUT [3] for general fitness prediction. However, we will add additional comparisons to ProGen2-XL [4], a 6.4B parameter dense autoregressive model, for both unconditional generation and alignment. We find that only only 4.1% of generations from ProGen2-XL (out of 4M total) are valid proteins (compared to 39.8% / 49.3% from ProGen3-3B/46B), and they cover 0.82x / 0.66x as many clusters as ProGen3-3B/46B. For a fixed parameter count, ProGen3 also outperforms ProGen2 on all our alignment benchmarks, but ProGen3 is much more efficient due to its sparse MoE implementation. We do not compare against ESM3 [5] because its license precludes us from using it, and we do not compare against AIDO.Protein [6] or ESM-IF1 [7] because they perform structure-conditioned generation, not unconditional generation.\n3. Figures 1e and 4a only show a single point for 46B parameters because we only trained this model size on 1.5T tokens. Because we need to train a separate model for each dataset size (Lines 117-118), it was prohibitively expensive to train multiple versions of the 46B parameter model. Instead, we used our scaling laws to determine the right parameter/token split for 1.1e23 FLOPs.\n4. We do not expect that masking random spans would yield an overly easy infilling objective. First, we still apply autoregressive attention to the full sequence, so infilling a span of length L is just as hard as generating the L residues at the C-terminal when performing N-to-C generation. Second, we train to infill long spans that can cover up to 80% of the protein and can be >400 residues (Lines 856-862). That said, your suggestions to apply fixed-region scaffolding or motif-conditioned design would be interesting extensions that could improve performance in fine-tuning regimes where more structural data is available; unfortunately, they are infeasible to apply on large-scale unlabeled datasets like PPA-1.\n\n[1] Widatalla et al. “Aligning protein generative models with experimental fitness via Direct Preference Optimization.” bioRxiv, 2024.\n\n[2] Zhao et al. “Contrastive Fitness Learning: Reprogramming Protein Language Models for Low-N Learning of Protein Fitness Landscape.” RECOMB, 2024.\n\n[3] Groth et al. “Kermut: Composite kernel regression for protein variant effects.” NeurIPS, 2024.\n\n[4] Nijkamp et al. “ProGen2: Exploring the boundaries of protein language models.” Cell Systems, 2023.\n\n[5] Hayes et al. “Simulating 500 million years of evolution with a language model.” Science, 2025.\n\n[6] Sun et al. “Mixture of Experts Enable Efficient and Effective Protein Understanding and Design.” bioRxiv, 2024.\n\n[7] Hsu et al. \"Learning inverse folding from millions of predicted structures.\" PMLR, 2022."}]

Author rebuttal:

Meta-review: This work presents ProGen3, a family of protein language models up to 46B parameters. Reviewers appreciated the extensive scaling analysis, creation of a new dataset, and compelling wet-lab validation. All reviewers recommended acceptance, agreeing that the paper's contributions are significant and its strengths outweigh its weaknesses.

The paper is a clear accept. The scale of the models combined with a comprehensive evaluation that includes wet-lab validation makes this a useful study in protein generation. The insights on scaling and data curation are a contribution to the field. While the reproducibility concerns are valid, they do not detract from the paper's overall impact and the value of the released models.
Decision: Accept (spotlight)