Training Compute-Optimal Protein Language Models
Authors: Xingyi Cheng
Bo Chen
Pan Li
Jing Gong
Jie Tang
Le Song
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Protein Language Model
Scaling Law
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We explore optimally training protein language models and propose the scaling law, an area of significant interest in biological research with limited guidance on best practices.

Abstract: We explore optimally training protein language models, an area of significant interest in biological research where guidance on best practices is limited.
Most models are trained with extensive compute resources until performance gains plateau, focusing primarily on increasing model sizes rather than optimizing the efficient compute frontier that balances performance and compute budgets.
Our investigation is grounded in a massive dataset consisting of 939 million protein sequences. 
We trained over 300 models ranging from 3.5 million to 10.7 billion parameters on 5 to 200 billion unique tokens, to investigate the relations between model sizes, training token numbers, and objectives.
First, we observed the effect of diminishing returns for the Causal Language Model (CLM) and that of overfitting for Masked Language Model (MLM) when repeating the commonly used Uniref database. To address this, we included metagenomic protein sequences in the training set to increase the diversity and avoid the plateau or overfitting effects. 
Second, we obtained the scaling laws of CLM and MLM on Transformer, tailored to the specific characteristics of protein sequence data. 
Third, we observe a transfer scaling phenomenon from CLM to MLM, further demonstrating the effectiveness of transfer through scaling behaviors based on estimated Effectively Transferred Tokens.
Finally, to validate our scaling laws, we compare the large-scale versions of ESM-2 and PROGEN2 on downstream tasks, encompassing evaluations of protein generation as well as structure- and function-related tasks, all within less or equivalent pre-training compute budgets.

Training Compute-Optimal Protein Language Models
Xingyi Cheng',, Bo Chen?*t, Pan Li', Jing Gong', .Ji Tang², 2 Le Song! 1.3
BioMap Research TTinghua University "MBZUAI
derrickzy@gmail. com, cb21 @mails. twwngnua.nn.... edu. cn
Code: http:///........o./////////////s
Abstract
We explore optimally training protein language models, an area of significant interest in biological research where guidance on best practices is limited. Most models are trained with extensive compute resources until performance gains plateau, focusing primarily on increasing model sizes rather than optimizing the efficient compute frontier that balances performance and compute budgets. Our investigation is grounded in massive dataset consisting of 939 million protein sequences. We trained over 300 models ranging from 3,5 million to 10.7 billion parameters on 5 200 billion unique tokens, investigate the relations between model sizes, training token numbers, and objectives. First, we observed the effect of diminishing returns for the Causal Language Model (CLM) and that of overfitting for the Masked Language Model (MLM) when repeating the commonly used Uniref database. To address this, we included metagenomic protein sequences the training set to increase the diversity and avoid the plateau or overfitting effects. Second, we obtained the scaling laws of CLM and MLM on Transformer, tailored to the specific characteristics of protein sequence data. Third, we observe transfer scaling phenomenon from CLM to MLM, further demonstrating the effectiveness of transfer through scaling behaviors based estimated Effectively Transferred Tokens. Finally, to validate our scaling laws, we compare the large-scale versions of ESM-2 and PROGEN2
downstream tasks, encompassing evaluations of protein generation as well as structure- and function-related tasks, all within less or equivalent pre-training compute budgets.
Introduction
Scaling up transformer-basee models has become a guiding principle for enhancing model perfor-
mance across broad domains, particularly in Natural Language Processing (NLP) [4, 12, 24, 64, 79] and Computer Vision (CV) [20, 67, 89]. In recent years, large transformer-base Protein Language Models (PLMs) such PROGEN familiy [51, 61], ESM familiy [68, 47] and XTrimoPGLM [14] have also been developed, which leads to significant improvements over model performance on complex downstream tasks [27, 45]. Current language models utilize two main training objectives to encode sequence information: the BERT-like [23] Masked Language Model (MLM) and the GPT-like Causal Language Model (CLM) [11], each applied either separately or unified fashion. common understanding is that bi-directionally MIM excels in sample efficiency and shows enhanced performance in downstream task fine-tuning. This is particularly true tasks that emphasize under- standing complex patterns, making it prevalent learning objective in modeling protein sequences*
XC and BC contributed equally. Work done while interned BioMap. Appendix also compared CLM and MLM on the protein contact prediction task through fine-tuning and freeze probing. with MLM demonstrating superior performance relative to CLM.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

[46, 14]. On the other hand, uni- mniddirettioal CLM, due to its sequential generation ability, is better suited for generating more coherent and realistic sequences compared to MLM [19, 61, 63].
However, training large protein language models (PL.M) are computational-intnniiii for optimally allocating compute budgets for training PLMS are relatively underexplored, with most efforts focusing on scaling model parameters based on a fixed set of training tokens to achieve performance improvements. A key insight [36, 42, 76] is that large models should not be trained to their lowest possible loss to optimize computing; instead, models and data should be scaled proportionally based on available compute budgets. These scaling laws are broadly found in natural language models [42, 36, 33, 2. 58, 78, 17, 90]. But their applicability has not been validated within biological datasets, such as the primary structures of proteins, which are composed of amino acid sequences forming protein chains. Unlike natural languages, protein sequences are scientific data that are precisely represented using a vocabulary of 20 amino acids, with very little redundancy and are not as semantically smooth. Thus, we consider such data as a distinct modality and ask the question: What are the scaling behaviors for MLM and CLM in protein language modeling?
and strategies
We focus on the best practices, which include revisiting datasets, optimization objectives, and model parameters key factors. Our goal is investigate an optimal training scheme for protein language models given predetermined compute budgets. Our core findings are as follows:
We revisited the protein sequence data used for training PLMs and collected a dataset of 194 billion unique tokens on 939M unique sequences from publicly available sources to address the issue of overfitting and perform plateau in protein language modeling.
We find that, in both MLM and CLM, training data scales sublinearly in the model sizes but follow distinct power-laws. In other words, 10x increase in compute leads to 6x increase in MI. model size and a 70% increase in data, versus a 4x increase in CLM model size and 3x increase in training tokens.
We also find that models trained with CLM can be transferred to MLM. When given a predetermined amount of computation, and one wants to obtain both a CLM and a MLM model, there is a trade-off in allocating the training token each model to jointly optimize the performance of the two. Interestingly, the allocation for CLM pre-training was determined by the scaling law of CLM and MLM, and the Effectively Transferred Tokens D, from CLM to MLM. Furthermore, we verify this method experimentally using a 470M model and fine-tuning on downstream tasks.
Building on our scaling strategies, we re-allocate of model size and training tokens under the compute budgets of established PROGEN2-xlarg and FSSM22 (3B) setups. Consequently, with the same compute budgets, we trained two corresponding models, one with 7.2B parameters and the other with 10.7B parameters, which exhibited enhanced performance in a diverse range of downstream tasks.
2 Scaling up data
First, we explore the effects of training PLMs across multiple epochs under token scarcity conditions. We then introduce dataset, UniMeta200B, used throughout this work. This dataset enhancement alleviates the challenge of insufficient training for protein language models.
2.1 A Data-hungry Observation
Using the UniParc database with 250 million protein sequences, research on ESM [68] shows that the datasets UR50/S and UR50/D, with 45M and 65M unique sequences respectively, outperform Uniref100 in perplexity (PPL) on a -670M parameter MLM model. These datasets contain -15B and -20B unique amino acid tokens. The ESM-2 family models, ranging from 150M to 15B parameters, are trained extensively with nearly 1 trillion tokens over 45 epochs on the UR50/D dataset. In observing the scaling of ESM-2 models, it becomes apparent that increasing model size to 15B parameters from 3B shows marginal improvement. On the other hand, contemporary LLMs are often trained for only one or a few epochs [43, 36. 79, 80, 1 11]. The repetition of data with limited unique tokens has diminishing returns and hinders scaling model size [65, 34, 58, 70J. This underscores the importance of using rich datasets for training large-scale language models to ensure robust performance across applications. We evaluated models with 150M and 3B parameters on the UR50/S dataset, trained on 200B tokens, as shown in Figure 1. We focus on the Independent and Identically
2

---

Training Loss
Validation Perplexity
OOD Dataset Perplexity
URS0/S-150M
U
16
URS0/S-38
17-
14
12-
16-
2.4
10
2.2
2.6
16
17-
Masked Languag
2.4
14
2.2
12
16-
10
158
778
1388
2008
2008
1388
2008
Tokens
(1)
(5) (9)
(13)
Tokens
Tokens
(UR50/S Epochs)
(UR50/S Epochs)
Figure I: Learning curves for UR50/S and UniMeta200B. Training loss and validation PPL, OOD test PPL, were tracked over 200 billion training tokens for both the 150M and 3B models. As we scaled the model from 150M to 3B, we observed diminishing returns on CLM (First line) and a tendency to overfit MLM (Second line) when repeating the Uniref50 (UR50/S) dataset. We totally evaluate repeating methods on MLM 3B models, all of which present overfitting (see Appendix B).
Distributed (IID) validation and Out- Out-Of-Distribution (OOD) test PPL, which measures the model's randomness in amino acid selection. For our OOD dataset, we utilized the MMseqs2 tool [72] conduct searches within the UniRef90 database for sequences post-training dataset timestamp. retaining those with no defectable identity. From these, a random sample of 3,000 sequences was selected to constitute the OOD dataset. Notably, we do not adopt dropout regularization, practice that often reduces model capacity and is infrequently used in contemporary LLMs [43]. This choice consistent with recent LLM configuration findings [38], including ESM-2 [47].
The results show the 150M model lacks good generalization while increasing to 3B model resulted in diminishing returns for CLM and severe overfitting for MLM. Principally, the bidirectional self- attention mechanisms used in MLM have a higher capacity to overfit compared to the unidirectional self-attention used CLM. This because MLM can utilize the entire context surrounding masked token, leading to faster memorization of the training data.
2.2 Expanding Diversified Metagenomic Data
To tackle the challenge of data scarcity, we leveraged the Colab- FoldDB database [56], which focuses on metagenomic data sources such BFD [1]. MGnify [57], and spe- cific eukaryotic and viral datasets in- cluding SMAG [22], MetaEuk [44],
Table 1:
The Pre-training data, aggregates various public sources and specifies sampling proportions for single epoch of training on 194 billion unique amino acids.
Datasets
Prot. Seq.
Tokens (AAs)
Samp. Prop.
TOPAZ [3], MGV [59], and GPD [13].
Uniret50/S
54M
15.2B
8.5%
Uniref90/50
102M
37.8B
19.5%
We applied stringent deduplication process with a maximum similarity threshold of 0.3 to preserve the diver- sity of the protein universe. Given that the Uniref90 dataset has proven effective for pre-training across various Uniref clustering levels per ESM-1v [54], we incorporated Uniref90/50 (Before 2022-12) 12). which includes incremental data relative Uniref50/S representatives. ColabFoldDB, and ColabFoldDB play dominant roles within the dataset, corresponding to cluster representatives and members. respectively. To ensure uniformity during training, we allocate weights within each batch to allow each amino acid token to be evenly processed through the model. This dataset, termed UniMeta200B, contains 939 million unique protein sequences and 194 billion amino acids, which is order of magnitude larger than UR50/D. We observed significant improvements
ColabFoldDB.
208M
37.7B
19.5%
ColabFoldDB
575M
103B
52.5%
Total
939M
194B

---

in the OOD test set and consistent learning curve on the IID validation subset extracted from the training set (Figure 1). These enhancements not only ensure a controlled diversity to maintain sample efficiency but also significantly increase the quantity and uniformity of data, facilitating model scaling.
Findings 1. Scaling the model from 150M to 3B, we noted diminishing returns for CLM and an overfitting tendency for MLM when repeating the UR50/S dataset. The proposed Expanding Diversified Metagenomic Data (UmiMeta200B) addresses these problems.
3 Parameters and Datasize Optimal Allocation
2.4
C.H Valdaaion 1o35
MLM Valeaatonnn Less
2.5
2.3-
244
18 FLOPs
3e118 FLOPs
1e+19 FLOPs
1c+19 FLOPs
3c+19 FLOPs
2.1-
2c+19 FLOPs
1e+20 FLOPs
1c+20
20 FLOPs
3c+20 FLOPs
2.2
1c+21 FLOPs
2.0
1c+21 FLOPs
101
Model size
Model size
10
1014
N95 - 1..1--..1----000 CAN
D/M 2.0 2.02-10".
1041
N33 1.26-10-f...a
DSM m1 1.24 10F gooss
1072
1050
Nodel dre
Data 627
10
2011
10a
1020
10'
105
1.2
1044
1044
FLOPs
FLOPs
Figure 2: IsoFLOPs curves and parametric fit for CLM and MIM. We selected training tokens to ensure uniform final FLOP count for different model sizes. The lowest loss of each curve revealed an optimal model size for FLOP budget (above). We use these rainbow points the valley to plot the efficient frontier for estimating the optimal model size and training tokens for scaling models (below). The interval range was estimated by model points with similar loss.
In this section, we propose a scaling law for protein sequences with MLM and CLM objectives, aiming optimally balancing model size and data size under fixed compute budget to improve efficiency on expanded resources.
3.1 Scaling laws for CLM and MLL
Table 2: Coefficient of Equation 1.
We first fit our models in the form of a fundamental power-law based on the
Parameter
a
B
existing work [42, 36, 33, 69, 2, 17, 90] in the field of LLMs. Specifically, given a fixed FLOPPs formula of C.
CLM MLM
0.578
0.422
1.26 x 10 3
1.23 x 10
0.776
0.230
6.19 x 10 2.02 x 10
6 x N x D, where N represents the
number of how should one navigate the trade-off between model size and the number of training tokens? The
non-embedding parameters, and D is the number of training tokens,
Appendix E compare the training performed separately on two datasets, and we find that the ColabFoldDB does not affect downstream results.

---

model parameters N and data size D can be directly fit with simple power-law:
N(C)= = AxC", D(C)= = B
We employed the IsoFLOPs profiling approach [36, 9], setting 7 distinct training FLOP counts ranging from x 1018 to 102t. For each FLOOP count, we selected models from a pool of candidates (see Appendix N). Models were excluded if the estimated data size (C/(6 * N)) resulted in more than 200B tokens or if the training steps were fewer than 20K. U'timately, approximately 260 models were used for fitting. We considered the final validation loss for each model to ensure that every model completed full cosine cycle with 10x learning rate decay. For each fixed FLOP count, we employ smoothed loss to determine the optimal model size with the smallest loss (Figure 2 (above)). Subsequently, we use Equation and apply the least_squares method to the model. Given
Table 3: Coefficient of Equation 2
Objective
ON
ap
ac
NN
BD
3c
CLM
-0.037
-0.051
-0.027
4.835
7.904
8.251
MLM
-0.040
-0.120
-0.034
4.530
42.614
10.125
the minimal variations in the final loss among a set of (N, D) configurations, we classify these configurations operating under "IsoLoss" conditions (see Appendix K. Figure A15), considered optimal for training. In Figure 2 (below), we illustrate an efficient frontier interval that demonstrates permissible fluctuations in model size and dataset size at specific FLOP count, while still achieving nearly identical losses. The variation in loss is quantified at 0.25 on a logarithmic scale with a base of 10. This indicates that within this FLOP counts, the model size can be adjusted within a range, increasing up to 80% or decreasing up to 40% without repeating data, to maintain loss variation within 0.01.
We observe distinct growth rates in the proportional relation- ship between model size and training tokens for the ML. model compared to the CLM, as detailed in Table 2. Both models demonstrate increase in the growth of model size that surpasses the growth of training tokens. Up to the inter- section point around 1 10 (see Figure 2, left below), the model size of MLM tends to be smaller than the CLM, there- after, the MLM rapidly exceeds that of the CLM. Notably, the growth of the MLM's training tokens is greatly lower than that for the CLM, possibly due MLM's 's higher sample efficiency. For instance, if the compute budget is increased by 10x, the size of the CL.M model should increase by and the training data by 3x, aligning more closely with equally proportional scaling. For the MLM, the model size should increase by 6x and the training data size by 1.7x.
Model
10
10
1077
FLOPs (Cam!
Figure 3: Compute allocation for two objectives with the same model size.
In exploring the scaling relations of loss, we analyzed various model sizes N, compute budgets C, and training dataset tokens D. These can be described by similar power-law relation defined as:
L(x) 3r x x
(2)
where O₂ is the scaling exponent for different variables. For each FLOP count, we aimed identify the minimal loss as the fitting target along with the corresponding independent variable x. Table presents these fitting coefficients.
Based on the coefficients obtained from the fitting described above, we can establish the relationship between D and N by eliminating L. The relationship is expressed by the following equation:
BN 3D
oD
ON
D(N) =
X N RD
D By substituting the learned coefficients into this formula, we derive MLM and CLM when given N. The estimation may be affected when the data exceeds 200 billion or when the quality or quantity of the training dataset changes.
5

---

3.2
Scaling law for training two models
When our goal is optimize both CLM and MLM simultaneously, the strategic allocation of compute resources between these two objectives becomes essential. To facilitate this, we equalize model parameters across objectives to assess specific compute budgets for dual-objective training. Specifically, we seek the compute budgets, CMLMM and CCLM. for configurations where the optimal model size the same, i.e., (CMM)) N N(CCLM). These individual computations are then aggregated to formulate the overall compute budget:
0.776
1.25 x 10 0.578
6.2 X 10-8
Csum (N) CMLM(N) + CCLM(N) =
(4)
N
N
These two objectives share the same parameter size, their compute budget C. and the number of training tokens D differ. Thus we further introduce a model-to-ratio r(N) as DMLM(N) (N). We then achieve the relationship between N and Csum by fitted power-law (Figure 3) form:
(C 1.497 x 10-6 6 CO.703 r(N) R 8.449 x 10a x N N0..392
(5)
The ratio r(N) informs us about the allocation proportion of training tokens. Specifically, under equal parameters, the data for MLM should exceed that for CLM until 10B threshold (achieving 1:1) is reached, after which more training tokens are allocated to CLM.
We further find that the scaling behavior of sparse parameter counts in Mixture of Experts (MoE) protein model [74], configured with eight experts (see Appendix I), as well as a combined power-law formula used to fit our data (see Appendix J), both exhibit similarity to the scaling behavior have proposed.
Findings 2. In both CLM and MLM, training data scales sublinearly with model size, following distinct power laws. With an "infinite" dataset, where samples are not repeated and training for less one epoch, MLM's 's model size grows faster than CLM's s.
4 Transfer Scaling
We have outlined two independent scaling laws and how to allocate FLOPs under fixed budget for training two optimal models, one with MLM and the other with CLM. However, we have not explored the interaction between these objectives. This raises important questions: Can models trained with one objective transferred to one with another objective? Is there a synergistic effect from training two models? Does training order impact the results?
4.1 Transferability
We conduct transfer learning experiments on MLM and CLM objectives, selecting eight optimal model sizes based on Equation 1. These models correspond to four increasing FLOP counts from 3 x 19 to 1 x 10 and undergo training from scratch followed by transfer training. Transfer training involves initially training on MLM CLM, then training on the alternate model for each size.
We find that optimal pre-training on one objective benefits the target objective in transfer learning. though effects vary between methods. Starting with CLM and then training MLM, benefits increase with model scale. In contrast, starting with MLM then training CLM sees diminishing benefits. As shown in Figure 4 (left), for model size of 230M with 3 x 1019 FLOPs, CLM from MLM pre- tre-training reduces the loss by 0.02 compared to CLM from scratch. however, benefit that nears zero for the 1.7B model. Conversely, for models from 85M 1.2B, transfer benefits grow with model size, the compared validation loss gap increasing from 0.025 to 0.045. This likely stems from the higher loss utilization rate in CLM; CLM calculates losses for all tokens in a protein sequence, whereas MLM only calculates losses for 15% of the tokens.
Appendix C analyzes the mask ratios.

---

1.28
CLM
2.9
880M
2.8
CLM from MLM pre-training
2.0
CUM loss curve from scratch
680M
2.7
13++19 FLOPs)
2.7
550M
adation Loss
470M (1e+20 FLOPs)
2.6
470M
2.6
1.7b (1e+21 FLOPs)
13e+20 FLOPs)
25
393M
300M
2.5
24
230M
2.4
23
170M
MLM from C.M mommer
154M
2.3-
2-
127M
CLM from scratch fronlier
2.2
58
258
458
65B
93B
+[C]
1015
10
1022
106M
85M
Training tokens
FLOPs
1.28
MLM from scratch
27
880M 55
MLM from CLM pre-training
2.6
curve from
736M
85M (3e+19 FLOPs]
2.5
630M
2.5
200M (1e+20 FLOP))
2.4
550M
609M
24
(3e-20 FUOPs)
233
470M
Validatoo
2.3
1.28 (1e+21 FLOPs)
22
393M
300M
21
280M g
2.
2.0
230M
21
MLM from CLM fronder +
170M
19-
20-
MLM from scratch frontirr 21
fronter
127M
1.8
106M
35B
65B
95B
1258
1558
10ts
1000
1022
85M
Training tokens
FLOPs
Figure 4: Left: The upper graph compares validation loss of CLM trained from scratch with those transferred from MLM, showing diminishing transfer benefits as model size increases. The lower graph depicts increased benefits for MLM from pre-trained CLM with larger sizes, indicating scale- dependent efficiency gains. Right: Shows loss curves for CLM and MLM across different FLOPs, emphasizing the efficient frontiers (or Pareto Frontier) from various transfer strategies. It highlights that the benefits of transferring from CLM to MLM grow with model size, reflecting scale-dependent synergy between training objectives.
We use a power-law model the transfer scaling law, initially excluding the pre-training FLOPs, The scaling behavior of transfer learning is modeled by:
L(C,) = A, x Ca L(C)) = B, x CT
(6)
where I(C)) and I(C.)) represent the loss for transfer learning and training from seratch.
Figure (right) shows that the efficient frontier for L(C,) has shifted relative L(C.) (it can be directly obtained from Table 3, repeated here for convenience.), indicating an improvement. The coefficients from both are shown in Table 4,
Table 4: Coefficients for I(C,) and I(C))
Parameter
A
B.
MLM
10.125
-0.034
11.133
-0,038
CLM
8,251
-0.027
7.191
-0.024
dt
where we can infer that Ct x C = C
suggesting that training MLM from scratch with 10x the compute requires approximately 7.7x the compute compared to MLM from CLM pre-training. Another observation is that mixing training objectives in single batch tends to be detrimental. Detailed results and settings are in Appendix H. The recommended transfer learning schedule involves pre-training CLM before MLM, as mixed training and order swapping show no benefits. We speculate that this primarily occurs because our MLM, which focuses solely on recovering corruption tokens, is not causal. If it predicted a middle segment in left-to-right manner, it could mutually adapt with the context accelerate training [86].
Findings Transferring from MLM to CLM results in diminishing returns. Conversely, transferring from CLM models to MLM models remains effective as compute scales.
4.2 Effectively Transferred Tokens
Although we observe that MLM benefits from transfer learning from CLM, the pre-training compute budget remains unaccounted for. We focus on two aspects: (1) the actual benefit CL.M provides to MLM and its predictability, and (2) performance differences between MLM trained from pre-trained
7

---

From scratch
Optima Interval
156
Valdeaton PPL (Optmma) MIM Stte (TUSPH)
8SM NLM from pre trained CLN
125
85M NLM from scratch
(2612)
(2112)
4T0M
Trun.
12.0
Pre-trained CLM Tokens
-by
(L:20)
11.5
9.0
11.
10.5
10.0
50B
Compute arai-ii
Figure 5: Left: Valid perplexity of % compute allocated for the CLM pre- pre-training. For instance, % compute indicates first training on CLM and then the rest compute fine-tuning on MLM. The optimal CLM pre-training % compute range with [10, 20]. And the fitted Dz/(D, + Dy) drops in the optimal loss range. Right: Comparison of validation perplexity for models trained from scratch (red) and those fine-tuned from a pre-trained CLM (green), demonstrating that fine-tuning from CLM reduces perplexity with similar or even fewer tokens.
CLM (MLM-CLM) and MLM from scratch (MLM-S) under identical FLOP constraints. We define Effectively Transferred Tokens Dr as the additional data model of the same size would need to train from scratch on MLM to achieve the same loss as model pre-trained on CLM. the token number in the pre-trained CLM model exceeds D,, then the computations for CLM pre-training was excessive. Knowing D, in advance would guide the allocation of tokens for CLM pre- pre-training.
We compare MLM-S and MLM-CLM models ranging from 33M to 1.2B with FLOP counts from 3 x 1049 to 1021, By calculating the token distance the same loss level between these models, we establish our fitting target Dr, collecting approximately 2800 sample points. Following similar methods in scaling transfer works [35, 90], D, is defined by a simple multiplicative scaling formula:
D. =
N 3 x 10 Rz R 001137
-0.369
D
where Dy represents the tokens used for MLM-CLM. and N is the number of parameters, with k, and y as fitting coefficients. For instance, a 10x increase in D, would roughly triple the model size and double D,- We validate these findings by evaluating the compute ratio of CLM pre-training under four specified parameters and FLOPs, as shown in Figure 5 (left), finding that MLM-CLM generally outperforms MLM-S. Specifically, (D, + Dy) ranges from 10% to 20% of the compute budget for CLM pre-training. Figure 5 (right) schematically illustrates the learning curves of two 85M (3ell 19 FLOPs) models, with MLM-CLM achieving similar or better loss levels with equal or fewer tokens.
Findings 4. Training MLM from scratch with 10x the compute requires approximately 7.7x the compute compared to MLM from CLM pre- pre-training. implying that around 20% of the compute budget should be allocated for CLM pre-training to get better MLM models transferred from CLM pre-training.
5 Experimental Validation
Based on the scaling laws we observe, we estimate the model size and training tokens for current leading models by analyzing their FLOPs. our configuration, the PROGEN2-Xlaree model, with 6.4B parameters, is estimated to require training with 7.2B parameters and 265B tokens. Similarly, ESM-2 model, with 3B parameters, should be trained with model size of 10.7B parameters and 260B tokens. Additionally, we employed two 470M models to test the transfer scaling strategy, one trained from scratch (470M scratch) and the other from CLM pre-training (470M trans.). The model's 's details reported in Table 5.
8

---

Table 5: Model architecture details. We compare popular models PROGEN2 and ESM-2 using similar FLOPs with our models estimated by proposed scaling law.
Params
Objective
Nheu
Dim.
Mayer
Train. Tokens
FLOPs
PROGEN2-Xlarge (6.4B)
CLM
16
4096
32
350B
1.34 10
Our 7.2B
CLM
32
4096
36
265B
1.14 x 10a2
ESM-2 (3B)
MLM
40
2560
36
1.68 1022
Our 10.7B
MLM
32
4352
47
260B
1.68 x 1022
470M scratch
MLM
16
1280
24
106B
3.0 1000
470M Trans.
CLM MLM
16
1280
24
21B + 85B
3.0 1020
A
B
D
Sequence clusering 50% cusof
FSMFold pl DDT
20
PROGENZ-xtarge
Our 7.20 CLM
100
1.00
10
PROGEN2 xlarge
4,818 clusters
8,466 seqs
FoMSeek FoMSeek.MM moosooee
Perplesity
40-
TT
0.75
Chuster site
2000
4000
6000
20-
20
8,263 sops
0.25
10
7,28 CLM
7,097 cluster
(N-514)
(N-7789)
0.3
(N-1756)
0.5
20
7:2 CLM
0.00
MaxID
(N-8456)
(N-4263)
0.0
0.5
1.0
2000
4000
6000
SoqID
Closters
Figure 6: Comparative Analysis of CLM Models. A. Perplexity analysis for PROGEN2- ealaree
and our 7.2B CLM shows lower values for our model across various MaxID levels, suggesting better sequence handling. B. Box plots of pLDDT scores for protein structures by PROGEN2-xlarg and our 7.2B CLM. C. Contour and line plots show our 7.2B CLM sequences mimic natural sequences more closely than PROGEN2-xlarge, assessed using Foldseek with the PDB database. D. Clustering at 50% sequence identity reveals our 7.2B CLM generates more clusters, indicating higher diversity.
5.1 Protein Generation Comparison: 7.2B CLM vs. 6.4B PROGEN-xlarge
We first evaluate the perplexity OOD data and then compare the protein generation capabilities of the 7.2B CLM and PROGÉN2-xXAN -llagg models. Each model generated 2,000 sequences for each parameter combination of top-p (0.5, 0.7, 0.9, 1.0) and temperature {0.2, 0.4, 0.6, 0.8, 1.0), totaling 40,000 sequences per model. Sequences with a perplexity greater than 10 and duplicates were removed, leaving 8,263 and 8,466 sequences for the 7.2B CLM and PROGEN-xlarge, respectively. We used four metrics to assess the quality of the models and the generated sequences (See Appendix for details).
OOD Dataset PPL Analysis We randomly sampled 5,000 sequences from UniProt released after 2023-01-01 and aligned them to our and PROGEN2's s training data (Uniref90 and BFD) using HHblits [66] or Jackhmmer [29]. Sequences below maximum identity cutoff' were used assess the models' PPL., as shown in Figure 6A. Our 7.2B CLM exhibited lower PPL on three subsets.
PLDDT scores from ESMFold Atomic structures of 8,263 and 8,466 generated sequences were predicted using ESMFold, and compared based on pLDDT scores, displayed in Figure 6B. The 7.2B model's average PLDDT score was 78.69, higher than PROGEN2-xlarge' 74.33.
Natural Sequences Comparisons with Foldseek Using Foldseek [81]. we searched the PDB database for sequences similar to those generated by our 7.2B CLM model, which showed better mimicry natural sequence properties with higher average TM-scores (0.655 vs 0.522) and SeqID 19.194 vs 0.165), as shown in Figure 6C.
Diversity Analysis Generated sequences were clustered using MMsegs2 [72] with a 50% similarity cutoff. The 7.2B CLM model resulted in higher diversity with 7,097 clusters compared to 4,818 clusters for PROGEN2-xlagge, detailed in Figure 6D.
5.2 Protein understanding tasks: 10.7B MLM vs. 3B ESM2
We evaluate different task types from the protein benchmark [14]: Contact prediction as binary classification at the amino acid pair level: fold classification into 1 1195 classes at the sequence level: and fluorescence as regression tasks. Following [14], we add Multi-Layer Perceptron (MLP) head to each pre-trained model and apply Low-Rank Adaptation (LoRA) [37] (r=8, a=16) for fine-tuning (see Appendix G for convergence details).

---

Table 6: Tasks performance of MLM Model on the test dataset with LoRA fine-tuning
Models
Contact Pred. (P@L/S)
Fold Class. (1195 class.)
Fluor. (reg.)
ESM-2 (3B)
0.91
0.69
0.65
Our 10.7B
0.91
0.72
0.69
470M scratch
0.78
0.65
0.67
470M trans.
0.80
0.66
0.67
The results, shown in Table 6 and A7, demonstrate that our 10.7B model outperforms ESM-3B on 7 of tasks. This confirms the rationale behind the observed scaling law and addresses concerns about the scope and rigor of our evaluation tasks. Additionally, the 470M model transferred from CLM pre-training continues perform effectively in this task, showing the efficacy of the observed transfer scaling law.
6 Discussion and Limitations
Data Repeat Scaling Law. Our scaling law derived within a single-epoch training setting. is well known that MLM exhibits higher sample efficiency than CLM due to its dynamic masking strategies across multiple epochs. However, this advantage diminishes when training is restricted to just one epoch. We present empirical study comparing 2.8B model trained IT tokens (approximately five epochs) with a 10.7B model trained on 265B tokens (roughly 1.4 epochs). The two models achieve similar performance in terms of OOD PPL (10.33 10.21) while utilizing the same amount of FLOPs. This finding suggests that repeating multiple rounds of MLM training has minimal impact on reducing loss. Notably, the smaller models more user-friendly during inference and fine-tuning, Therefore, we suggest an alternative approach that adjusts the optimal training token count and model size within the data-repeat scaling law.
Multi-modality Scaling. The multi-moda auto-regressive work [33] suggests the existence of a nearly universal scaling law across various modalities, including images, videos, math, code, and languages. Our results appear in this trend as well, such as, the scaling laws for CLM exhibit similarities those in natural languages. The same situation may apply other modalities of biological data, such as RNA and DNA [60].
Various Pre-train Datasets and Strategies. Our datasets cover substantial portion of the protein universe, yet they might not be entirely representative. Combining BFD [8]. Uniref [75], MetaClust [44], and IMG/IGI [52] with 90% clustering results in least 600 billion unique tokens. However, variations in datasets could affect the power-law behavior. Future work could explore applying our findings different model architectures. There is ongoing research on scaling LLMs for long sequences [7, 15, 16, 18, 39, 48, 87], and MSA augmentation could significantly improve protein representation regarding contacts and structure. Investigating scaling laws in this context could be a promising direction for future research.
7 Conclusion
In this work, we are the first to establish a practical pathway for researchers to develop faithful and powerful protein language models optimized by both CLM and MLM objective in an end-to-end manner. This includes everything from pre-training dataset construction, expanded metagenomic databases such as ColabFoldDB, emphasizing the critical importance of data quality and quantity for scaling language models, to optimal parameter and dataset allocation along with the potential loss prediction, as well as knowledge transfer from other pre-training objectives. Our work holds significant potential for the application of large language models across various scientific domains.
This work has been supported by the National Key R&D Program of China 2021ZD01 13304, NSFC for Distinguished Young Scholar 62425601, New Cornerstone Science Foundation through the XPLORER PRIZE.
Acknowledgments.
10

---

References
[1] Bfd big fantastic database. httpps://bft htttp::/bbddmmmeqs.cco com.
[2] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. International Conference on Machine Learning. pages 265-279. PMLR, 2023.
[3] Harriet Alexander, Sarah K Hu, Arianna I Krinos, Maria Pachiadaki, Benjamin J Tully, Christo- pher J Neely, and Taylor Reiter. Eukaryotic genomes from global metagenomic dataset illuminate trophic modes and biogeography of ocean plankton. bioRxiv, pages 2021-07, 2021.
[4] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv: 000.666770666..... 2016.
[6] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv: 14255, 2022.
[7] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-doeument transformer.
arXiv preprint arXvv:2004.051500 2020.
[8] BFD Team. Big fantastic database. BFD Official Website, n.d.
[9] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint S..iv:2401.02955 2024.
[10] Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102- 2110, 2022.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in newral information processing systems, 33:1 1877-197911 2020.
[12] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[13] Luis F Camarillo-Guerrero,, Alexandre Almeida, Guillermo Rangel-Pineros, Robert D Finn, and Trevor D Lawley. Massive expansion of human gut bacteriophage diversity. Cell, 184(4):1098- 1 1109, 2021.
[14] Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, et al. xtrimopgim: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint arXiv:2401 Seti::2001.06999,, 2024.
[15] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 10509, 2019.
[16] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 2020.
[17] Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pages 4057-4086. PMLR, 2022.
[18] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35: 2022.
11

---

[19]
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, al. Robust deep learning-based protein sequence design using proteinmpnn. Science, ():06661))49-5555 2022.
[20] Mostafa Dehghani. Josip Djolonga. Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[21]
Grégoire Delétang, Anian Ruoss, Paul-Ambrois Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint 2rXiv:2309..000 10668, 2023.
[22] Tom Delmont, Morgan Gaia, Damien D Hinsinger, Paul Frémont, Chiara Vanni, Antonio Fernandez--ueria A Murat Eren, Artem Kourlaiev, Leo d' Agatt, Quentin Clayssen, al. Functional repertoire convergence of distantly related cukaryotic plankton lineages abundant in the sunlit ocean. Cell Genomics, 2(5):10012) 2022.
[23] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
[24] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pages 5547-5569. PMLR, 2022.
[25] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360 2021.
[26] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 320-335, 2022.
[27] Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general-purpose modelling. arXiv preprint Stti::3001.065666 2023.
[28] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):112-711) 2021.
[29] European Bioinformatics Institute. Jackhmmer tool. EBI Tools Documentation, n.d.
[30] fast.ai. How could the memorization hypothesis be true. fast.ai Blog, 2023. Retrieved
May 21. 2024, from https://www.ffstt.i//aaaa saaooriiaatioohhottheeisss berttu...
[31] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 a deep unsupervised language model for protein design. Nature communications, 13(1 (3(1),4348, 2022.
[32] Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Martin Steinegger, and Burkhard Rost. Prost1:: Bilingual language model for protein sequence and structure. bioRxiv, pages 2023-07, 2023.
[33] Tom Henighan, Jared Kaplan. Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 100/////4470)) /4701, 2020.
[34] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk. Nelson Elhage, Zac Hatfield-Dodds Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10483 10487, 2022.
[35] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.0122
2021.
[36] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
12

---

[37] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.09000
2021.
[38] Hugging Face. Llana 2 model documentation, n.d.
[39] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song. Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint 2xXiv:2309.14500 2023.
[40] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401 240.002200.0000888, 2024.
[41] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger. Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek. Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583- 2021.
[42]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alce Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. 2020.
[43] Aran Komatsuzaki. One epoch is all you need. arXiv preprint
2019.
[44] Eli Levy Karin, Milot Mirdita, and Johannes Söding. Metaeuk-sensittee high-throughput gene discovery, and annotation for large-scale eukaryotic metagenomics. Microbiome, 8:1-15, 2020.
[45] Trancesca-Zhoufal Li, Ava P Amini, Yisong Yue, Kevin K Yang, and Alex X Lu. Feature reuse and scaling: Understanding transfer learning with protein language models. bioRxiv, pages 2024-02 02,0002222 2024.
[46] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic level protein structure with a language model. bioRxiv, 2022.
[47] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6633) 2023.
[48] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023.
[49] Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXivv2310.05209, 2023.
[50] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 2017.
[51] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497, 2020.
[52] Victor M Markowitz, Frank Korzeniewski, Krishna Palaniappan, Ernest Szeto, Greg Werner, Anu Padki, Xueling Zhao, Inna Dubchak, Philip Hugenholtz, lain Anderson, et al. The integrated microbial genomes (img) system. Nucleic acids research,
2006.
[53] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv: 1812,066 162, 2018.
[54] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sereu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in neural information processing systems, 34:29287-29303, 2021.
[55] William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. arXiv preprint arXiv:2010.09699 2020.
[56] Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature methods, 19(6):679-688 2022.
13

---

[57] Alex L Mitchell, Alexandre Almeida, Martin Beracochea, Miguel Boland, Josephine Burgin, Guy Cochrane, Michael R Crusoe, Varsha Kale, Simon C Potter, Lorna J Richardson, al. Mgnify: the microbiome analysis resource in 2020. Nucleic acids research, 48(DD))D570-D558 2020.
[58] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Seao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models, Advances in Neural Information Processing Systems, 36, 2024.
[59] Stephen Nayfach, David Páez-Espino, Lee Call, Soo Jen Low, Hila Sberro, Natalia N Ivanova, Amy D Proal, Michael A Fischbach, Ami S Bhatt, Philip Hugenholtz, et al. Metagenomic compendium of 189,680 dna viruses from the human gut microbiome. Nature microbiology. 6(7):960-970, 2021.
[60] Eric Nguyen, Michael Poli, Matthew G Durrant, Armin W Thomas, Brian Kang, Jeremy Sullivan, Madelena Y Ng. Ashley Lewis, Aman Patel, Aaron Lou, et al. Sequence modeling and design from molecular to genome scale with evo. bioRxiv, pages 2024-02, 2024.
[61] Erik Nijkamp. Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, (4(11))968-97 2023.
[62] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: large- scale benchmarks for protein fitness prediction and design. Advances Neural Information Processing Systems, 36, 2024.
[63] Jiezhong Qiu, Junde Xu, Jie Hu, Hanqun Cao, Liya Hou, Zijun Gao, Xinyi Zhou, Anni Li, Xiujuan Li, Bin Cui, et al. Instructplm: Aligning protein language models follow protein structure instructions. bioRxiv, pages 2024-04, 2024.
[64] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11442 2021.
[65] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[66] Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding. Hhblits: lightning- fast iterative protein sequence searching by hmm-hmm alignment. Nature methods, 973)
175, 2012.
[67] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-855 8395,559,222 2021.
[68] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e201223))) 68. 2021.
[69] Philippe A Robert, Rahmad Akbar, Robert Frank, Milena Pavlović, Michael Widrich, Igor Snapkov, Andrei Slabodkin, Maria Chernigovskaya, Lonneke Scheffer, Eva Smorodina, et al. Unconstrained generation of synthetic antibody-antige structures to guide machine learning methodology for antibody specificity prediction. Nature Computational Science, 2(12):845 865, 2022.
[70] Elke Schaper, Andrey V Kajava, Alain Hauser, and Maria Anisimova. Repeat not re- statistical peatt-statistical validation of tandem repeat prediction in genomic sequences. Nucleic acids research, 10005-00000 2012.
[71] Noam Shazcer. Glu variants improve transformer. arXiv preprint
2020.
[72] Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(1 (5(11):022-1105 2017.
[73] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Nenrocomputing. 127063,, 2024.
14

---

[74] Ning Sun, Shuxian Zou, Tianhua Tao, Sazan Mahbub, Dian Li, Yonghao Zhuang, Hongyi Wang. Xingyi Cheng, Le Song, and Eric P Xing. Mixture of experts enable efficient and effective protein understanding and design. bioRxiv, pages 2024-11, 2024.
[75] Baris E Suzek, Yuqi Wang. Hongzham Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, (1(6))926-92) 9322, 2015.
[76] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani. and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10688 2021.
[77] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, et al. U12: Unifying language learning paradigms. arXiv preprint wwx.::2225.051333 2022.
[78] Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210. 11399, 2022.
[79] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[80] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv: 2307.09288. 2023.
[81] Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. Biorxiv, pages 2022-02, 2022.
[82] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[83] Robert Verkuil, Ori Kabeli, Yilun Du, Basile IM Wicky, Lukas F Milles, Justas Dauparas, David Baker, Sergey Ovchinnikov, Tom Sercu, and Alexander Rives. Language models generalize beyond natural proteins. bioRxiv, pages 2022-12, 12, 2022.
[84] Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv: 19X.x:990200499 2019.
[85] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[86] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung. Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pages 22964-22988 PMLR, 2022.
[87] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13:17283-17297, 2020.
[88] Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai. Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:22 10.02410 14, 2022.
[89] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 12104-12113, 2022.
[90] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets I1m finetuning: The effect of data, model and finetuning method. arXiv preprint 172.1. 2024.
[91] Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure- informed language models are protein designers. In International Conference on Machine Learning, pages 42317-42338. PMLR, 2023.
15

---

Table A7: Tasks performance of MLM Model the 5 test dataset, Fitness Prediction (Fit P.) as a regression task measured by Spearman coefficient at sequence level, Localization (Loc.) as 10 sub-cellular classification task at sequence level, Metal Ion Binding (MIB) binary classification task at sequence level, Solubility (Sol.) as a binary classification task at sequence level, Stability (Sta.) regression task measured by Spearman coefficient sequence level, with LoRA fine-tuning.
Model
Fit P. (SP)
Loc. (ACC)
MIB (ACC)
Sol. (ACC)
Sta. (SP)
ESM2 (3b)
0.94
0.81
0.82
0.74
0.82
Our 10.7B
0.96
0.79
0.83
0.79
0.83
Appendix
Table of Contents
A Related Work
16
B UR50/S Repeat Experiments
17
C Choice of Masking Ratio
18
D MLM/CLM for Protein Contact Prediction
19
E Pre-training Dataset Quality
F The Evaluation of Protein Generation.
G Convergence Analysis of Downstream Fine-tuning Tasks
21
H Mixed Objectives Training
21
MoE Scaling
23
Combined Power-law
23
K IsoLoss
24
L Training Procedure
24
M Broader Impact
25
N Model Parameters
25
A Related Work
Protein Language Model Since the advent of AlphaFold2 [41], the masked language model (MLM) has been integrated as subtask within the Evoformer architecture. In this context, an assumption lossless compression method [21]. This was
that large language models can be considered followed by a series of language modeling efforts [31, 10, 32, 28, 27], which aimed to conduct pre-training on single-sequence proteins using larger datasets and model scales. These efforts sought to hamness the scale of the models to learn complex co-evolutionary information, although detailed investigations on how to optimally scale these models remain scarce. Our work primarily focuses on these finer aspects, aiming fill this gap in the research.
16

---

Training objectives In natural language processing (NLP), masked language models (MLM) are rarely adopted due the self-explanatory nature of natural language, which inherently prompts the meta-knowledge of tasks and generates task targets through CLM (Conditional Language Modeling) training models. However, a unified language modeling objective for Protein Language Models has yet to be fully consented. Those based on causal language modeling (CLM) have been primarily explored for protein design. Benchmarks in protein design using MLM [84] have also shown promising results for generation [62], exhibiting variable performance when compared to CLM [91, 83]. Additionally, the potential of the in-filling task objective remains largely unexplored [6. 77, 25J. Our research aims to thoroughly discern the scaling behavior of the two most common optimization objectives in this domain.
Scaling Laws To our knowledge, the concept of scaling laws of language model is first introduced by OpenAI [42]. Subsequently, numerous variants and modifications [36] have been developed around this theme. Recently, an array of new scaling laws has emerged. These include scaling laws related to learning rates and batch sizes [9], data-constrained scaling laws [58], scaling laws for downstream tasks and Transfer [90, 35], well scaling laws within the Mixture of Experts (MoE) framework [17], and those concerning long sequences and positional encoding [49]. While these laws primarily derived using auto-regressive models in resource-rich domains, their application in the biological data sector is less common. Our work seeks to address this gap. Furthermore, scaling laws for Masked Language Models (MLM) are notably scarce. Given that MLMs are currently one of the most effective training methods for biological data, our research on MLMs could also be extended to other non-text domains.
B UR50/S Repeat Experiments
Tranning loss curve
Validation loss curve
Grad norm curve
10
UnWMen2008
UniMera2008
Un Mn:
505 teoistap
2.5
JRSW'S bookstrp
08
USSAS brotstrpp
2.2
JR 50.5 ever, mpoch
U85005 ylota_stulte
rport_ sbuffe
2.3
2.4
JRSA'S every stuth
JRSWSS globo_auufe
0.7
0.8
L185A every, -pprrt cruftee
USS0S glosal 2hulft
2.2
0.6
0.5
2.3
044
2.0
0.3
1.3
02-
3.5
0K
SGK
100K Stop 200K
15
1.8
0K
50K
100K
15
200x
0.1
0K
50c
100K
153K
200K
Stop
Stop
Figure A7: Learning curve for UR50/S dataset repetition methods. Our 194B tokens dataset (UniMeta200B) shown in blue, serves as the reference with an approximate single epoch run. The bootstrapping method, depicted in orange, processes 200 billion tokens with replacement, indicating a tendency towards zero unsampled tokens by the fifth epoch. The every-epoch shuffle method, in green, ensures all tokens are used per epoch, forming a stair-step pattern in training loss. Lastly, the global shuffle method, in red, loosely uses all tokens each epoch but ensures the strict number of epoch passes for every token. The rightmost plot of gradient norms shows an uptick for curves corresponding to overfitting, signifying a lack of further optimization, with steep or erratic gradients indicated by the ascending gradient norms.
We employed three different methods repeat training on the UR50/S dataset, all of which ultimately led to overfitting. The reference for these experiments is shown by the blue curve in Figure A7, which represents UniMeta's loss for approximately one epoch.
Firstly, using bootstrapping, we processed 200 billion tokens from UR50/S with replacement. In each epoch, 65% of the dataset was randomly selected, leading to a diminished proportion of unsampled tokens by the fifth epoch, as depicted by the orange curve.
Secondly, we shuffled the unique data for each epoch to ensure that all UR50/S tokens were used per epoch, resulting in a stair-step pattern [30] in the training loss, illustrated by the green curve, has simply memorized the dataset but isn't 't improving at generalizing. Over-confident predictions of the first batch of the next epoch lead to a big step update, and then the model is not adapted to the next batches, resulting in no longer a decrease in loss.
17

---

Lastly. we shuffled the entire training dataset less stringently, which did not strictly ensure that all UR50/S tokens were used every epoch, but guaranteed that each token was used an equal number of times over the entire training period. We term it global shuffle, this approach is shown by the red curve.
From the gradient norm curve shown in Figure A7 (right), we observe an uptick in gradient norm for the overfitting curves, indicating that the model no longer optimizing effectively. In machine learning, such an increase in gradient norm typically suggests that the model is encountering areas of the parameter space where gradients are steeper or more erratic, often occurring when the model starts to memorize the training data rather than generalize from it, approaching a saturated network [55]. This behavior can result from overly complex models, too many training epochs without sufficient regularization, or training on non-representative data.
C Choice of Masking Ratio
model: 154M
mocel: 85M
Velidation loss
2.6
mask10
2.6
mask10
2.5-
mask20
Velidet on
2.5
mesk20
2.4-
mask25
mask25
2.3-
mask40
mask40
22-
mesk50
mesk50
50
Training tokens (billlon)
Taining tokenss (bil llon
Figure A8: Validation loss of different masking ratios. Two models (154M and 85M) are trained from 5% to 60% masking intervals.
Contact Prediction (154M)
Fold Prediction (154M)
0.6
Performance
0.6
Performance
0.4
0.0
0.0
0.05
0.1
0.15
0.2
0.25
0.3
0.4
0.6
0.05
0.1
0.15
0.2
0.25
0.3
0.4
0.6
Mask prob.
Mask prob.
Contact Prediction (85M)
Fold Prediction (85M)
Performance
Performance
0.3
0.4
0.2
0.2
0.0
0.05
0.1
0.15
0.3 Mask prob.
0.2
0.25
0.4
0.6
0.05
0.1
0.15
0.2
0.25
0.3
0.4
0.6
Mask prob.
Figure A9: Abalation different masking ratios. Two models (154M and 85M) are trained from 5% to 60% masking intervals, and evaluated on contact map and fold classification downstream tasks.
In the original BERT work [23], the absence of masked tokens in downstream tasks presented mismatch with the pre-training data distribution. The authors investigated various masking ratios and concluded that 15% masking rate most beneficial for downstream tasks. This was implemented alongside an 90-10-10 strategy: 80% of the tokens were replaced with a mask, 10% were randomly substituted, and the remaining 10% were left unchanged.
However, given the significant differences between protein sequences and natural language processing data, employed two models, sized at 85M and 154M, to explore range of masking ratios from
18

---

5% to 60% (see Figure A8). The best masking ratios for validation loss drop ranged from 10% to 20%; ratios too small (5%) or too large (greater than 25%) degraded the performance.
We further used pre-trained eight different models to perform full fine-tuning on downstream tasks such as Contact Prediction and Fold Classification in Figure A9. Results from the test datasets revealed that, similar to NLP, the optimal performance was achieved within 10%-20% masking range. Specifically, 20% masking ratio slightly outperformed 15% in Contact Prediction, while the 15% ratio yielded the best results in Fold Prediction. Consequently, for our Masked Language Model (MLM), we decided to adhere to the 15% masking ratio with the 80-10-18 strategy for training all our models.
D MLM/CLM for Protein Contact Prediction
Similar Pre-training Compute
Similar Pre-training Loss
0.9
0.8-
0.7
0.7-
10 0.5
0.5
0.4
0.4
38 MUM LoKA
BOOM MLM LoRA
38 CLM LoRA
7.28 CLM LORA
0.2
Probing
Probing
38 CLM Probing
0.2
7.28 CLM Probing
10000
20000
3ndoo
40000
50don
10000
20000
30000
40d00
50000
Figure A10: Contact Prediction on MLM and CLM models. Two 3B models (CLM and MLM) trained using identical computational resources, represented by the probing and LoRA fine- tuning methods. On the right, performance of a 7.2B CLM model is compared with an 880M MLM model under similar pre-training loss conditions. These models exhibit differing rates of convergence, highlighting the impact of uni-directional and bi-directional model architectures on learning dynamics.
We compared the effectiveness of CLM the downstream task of contact prediction, using two different setups (Figure A10). In the first setup, two 3B models were trained under identical computational resources on 200 billion tokens, 3.4 x 2 FLOPs. Their performance was evaluated through two training approaches: Probing (freezing the pre-trained model) and LoRA fine-tuning, with an added MLP head for comparison.
In the second setup, we compared the effects of MLM and CLM under similar loss conditions. Here, 7.2B CLM model and an 880M MLM model were selected, both achieving loss of 1.98 on our validation set. Despite the MLM model having simpler loss calculation, involving a 15% mask rather than one-by-one mask-which would result in a higher loss-the MLM significantly outperformed the CLM. Importantly, the CLM model's 's computational power was an order of magnitude greater than the MLM model (1.68 1022 vs 1.0 x 1021 FLOPs). This suggests that despite the lower loss achievable by the CLM model compared to MLM with one-by-on mask, the unidirectional limitations CLM do not translate into better downstream task performance.
E Pre-training Dataset Quality
Compared Uniref90, ColabFoldDB offers a higher diversity and larger numbers of protein se- quences, though with generally shorter sequence lengths, likely suggesting potentially lower data quality. To evaluate the efficacy of our expanded dataset, ColabFoldDB, we initially trained two 85M models separately on Uniref90 and ColabFoldDB. Uniref90 in our dataset comprises two subsets: Uniref50/S and the incremental dataset over Uniref50/S, termed Uniref90/50. Similarly, ColabFoldDB consists of representative and member data, We controlled the sampling proportion to ensure uniform sampling across both datasets, with results reported in Table A8, Both models were then trained using identical configurations on 50B scale.
19

---

From the perspective of validation loss in pre-training, the higher loss on ColabFoldDB might be attributed to its lower diversity and shorter sequence lengths compared to Uniref90. However, the performance on downstream tasks, such contact prediction and fold classification, shows negligible differences between models trained solely on ColabFoldDB and those trained on Uniref90, as illustrated in Figure F. This confirms that ColabFoldDB is an effective expansion of Uniref90 that maintains sample efficiency.
Table A8: Compared two dataset characteristics. Protein sequence count, token number, and sampling proportions for Uniref50/S, Uniref90/50, and ColabFoldDB representative and member data.
Datasets
Prot. Seq.
Tokens (AAs)
Sampling Prop.
Uniret50/S
54M
15.2B
28.67%
Unire190/50
102M
37.8B
71.33%
ColabFoldDB.
208M
37.7B
26.75%
ColabFoldDB...
575M
103B
73.52%
Learning curve
Contact Prediction
Fold Classification
2.7
85M, ColabFoldDD
0.7
2.6
85M_Uniref90
0.4
0.6
Validaton loss
2.5
0.5
0.3
2.4
Accuracy
0.4
2.3
0.2
0.3
2.2
0.1
85M, ColabFoldDB
85M, ColabFoldDB
85M Unvilfeo
#SM_Unirefoo
20000 60000 60000 800001 DO000
1000 2000
3000
4000 Stop
5000
6000
1000
2000
3000
4000
Stop
Step
Figure All: Data quality check. Comparison of learning dynamics and downstream task perfor- mance for two 85M models trained on ColabFoldDB and Uniref90. Left: Validation loss curves demonstrating initial training differences. Middle: Contact prediction performance showing the response testing on similar tasks. Right: Fold classification accuracy, comparing model responses to structural prediction tasks. Despite initial differences in loss, both datasets yield comparable performance in downstream applications.
F The Evaluation of Protein Generation.
We explain more details about Protein Generation Comparison as follows:
OOD Dataset PPL Analysis. PPL represents the probability of the test sequences in the model distribution. The lower the PPL, the closer the distribution of the model and the test set is. In order to test the generalization ability of the model on new data, we use different sequence identity (0.0, 0.3, 0.5) as thresholds to select the test set.
PLDDT Scores from ESMFold. Predicted Local Distance Difference Test is the confidence level of ESMFold protein structure prediction. This metric is widely used in methods such as AlphaFold2. ESMFold, and OpenFold. PLDDT filters are often used in protein design (such as RFDiffusion), which can significantly improve the success rate of protein design;
Natural Sequences Comparisons with Foldseek. Foldseck takes protein structure as input and searches for proteins with similar structure in the database. We use the experinentally--esllvee protein structure as the database (PDB database) explore how the structure of the generated sequences close to PDB (a higher TM-score indicates higher structural similarity). This method has been used to evaluate other methods for protein sequence generation (ProGen2, ProtGPT2);
Diversity Analysis. We cluster the two sets of sequences (ProGen2-xlarge and CLM) according to sequence similarity, Sequences with identity higher than 50% will stay in one cluster. Since the
20

---

number of input sequences is similar (8,466 vs 8,263), we can measure the diversity of the generated sequences by comparing the number of clusters.
G Convergence Analysis of Downstream Fine-tuning Tasks
Observing the learning curves in Figure A22a, we can assess the effectiveness of different fine- tuning scenarios. For the contact prediction task, the convergence speed under the LoRA setting is very similar for both models. Our testing reveals closely matching results for ESM-2 models with capacities of 650M, 3B, 15B, consistent with the findings reported by Ankh et al. [27]. This similarity suggests possible saturation of the dataset under single-sequence pre-trained models. Additionally, the convergence rates for tasks such as fold classification and fluorescence are generally faster than those for ESM-2, indicating robust generalization following data augmentation strategies.
Based on the two 470M models defined in our Table 5, despite using the same computational power, we observe distinct outcomes (Figure A A1)b) in contact prediction and fold classification tasks. The MLM model from CLM pre-training converges slightly faster than MLM from scratch. However, the distinction is less pronounced in the two downstream regression tasks. This suggests that perplexity more sensitive to protein structure related tasks, i.e., contact prediction and fold classification, but shows less sensitivity to regression tasks, particularly when assessed using the Spearman metric, which is prone to variability.
Contact Prediction
Fold Classification
Flucresence Prediction
0.8
0.6
Spearman an coefiiceent
0.6
0.5
PaLLSS
0.6
Accuracy
0.4
0.4
0.4
0.3
0.3
0.2
our 10.76
(38)
0.1
ESM 2 (381
our 10.7b
our 10.7b
ESM-2 (38)
0.0
2000
4000
6000
8000
20000
40000
60000
80000
20000
40000
60000
Stop
Step
Stop
(a) Learning Curve Convergence Speed. LORA fine-tuning our 10.7B model and ESM-2 (3B) model on three downstream tasks.
Contact Prediction
Classification
Fluoresence Prediction
0.7
0.8
0.7
0.6
Spearman corefficient.
0.6
0.5
Acourary
0.5
0.4
0.3
0.3
02
0.2-
0.2
470M from scratch
0.1
470M scretch
0.1-
470M from scrwith
pre-traming
pre-inaring
0.0
5000 1000015022000020000000
4000 Step
5030
7500 20000150001500
Step
Step
(b) Learning Curve Convergence Rate Detection. LORA fine-tuning two 470M models on three downstream tasks. transfer means first pre-training 21B tokens on CLM then fine-tuning on MLM with 85B tokens, from scratch means training on 106B tokens from scratch.
H Mixed Objectives Training
We also employed an untied model to simultaneously optimize two objectives:
LCLM CE(Vo(Wi encoder(z))), Mnext),
21

---

CLM Learning Curve
2.9
scratch
1.78
130M mix training with MLM
2.
BBCM mix training with MLM
2.8
1.78 mix training with MLM
27
2.7
Valiation Loss
26
valdaton
validation
2.2-
08
28
46
68
8B
108
08
108
208
31B
41B
518
08
198
388
578
768
958
Elapsed Tokens
CLM Elapsed Tokens
CLM Elaosed Toleens
MLM Learning Curve
2.6
2.6
25M from
470M scratch
25M mix training with CLM
2.5
470M mix training with CLM
2.5
1.28 mix training with CLM
2.4-
2.4
2.4
Valddtion Lo35
2.3
2.3
2.
1.9
08
98
19B
296
398
499
08
218
429
63B
84B
1058
08
278
558
B2B
1108
1378
MLM Elapsed Tokens
MLM Elapsed Tokenns
MLM Elapsed Takens
Figure A13: Mixed objective validation loss. Comparative validation loss curves for models trained from scratch versus mixed training approaches. Each panel corresponds to different model sizes, as indicated by the parameters. For each model, two training strategies were compared over an identical number of elapsed tokens: training from scratch (blue) and mixed training with the other objective (orange). Across all model sizes, training from scratch consistently achieves lower validation loss compared to mixed training, suggesting that mixed training may not be as effective as dedicated training for each individual objective.
and
MMLM CEVVo((a((ncood((a))) ymask).
where represents the protein vocabulary embedding, and W1 and W2 are the parameters corre- sponding to the CLM and MLM tasks, respectively. CE is the cross-entropy operator. The is the Tanh activation function.
We compared CLM and MLM under our scaling law of optimal model and data size distributions. One approach involved training from scratch, while the other used mixed training. In the mixed training approach, the actual number of training tokens was higher due to the additional FLOPs consumed by another optimally trained objective, other words. In other words, mixed training consumes the FLOPs of two optimal allocations; we only extracted the loss curve of one target for comparison. We extracted the loss curve of just one target for comparison with the from-scratch training. Our findings indicate that mixed training of the two targets can lead to detrimental interference, an effect not observable smaller models, as depicted Figure A13. As the model size increases hundred million or billion parameters, the differences become more pronounced. The possible reason for this situation that mixed training has reduced the batch size for one of the objectives, making optimization difficult. We did not further investigate the impact of increasing the batch size and only observed based on the training tokens. However, we cannot rule out the possibility that they are mutually detrimental. Therefore, if both objectives are to be optimized concurrently, sequential training strategy should be employed: first optimizing CLM, followed by MLM training. We consider that CLM is more challenging to predict than MLM, which may allow the model to capture more complex and implicit sequential features initially, thereby enhancing its ability to understand and predict masked words in subsequent MLM training.
22

---

1012
10"
2,60
1014
NGsT 1:48 100
m1557 10 305-CA4A
2.55
4023
CUR Va
Model
100
1014
2.45
10
2.40
36 FLOPs
10"
1e+20 PLOP4
10
1077
1619
10at
109
1017
100
1073
Model sice
FLOPS
FLOPs
1014
10t
2,5
101L
Ndt 6.36 6.35-10" 10-0 g2000
0 2.04. 100 .000
MLM Validatioon Lass
30
Model size
1.0
1.04
36
1e-19
10
2.2
3e-19 FLOPs
107
1e+20 FLOPs
167
10
1037
1019 10
FLOPs 10
1017
10
1003
Model size
FLOPs
Figure A14: Scaling laws of MoE.The scaling behaviors of sparse parameter counts (8 experts) in MoE models, highlighting IsoFLOPS curves for different model sizes and FL.OPs configurations. Each graph represents the relationship between model size, FLOPs, and validation loss for both CLM and MLM using MoE configurations. The power-law fits indicate optimal model size and data requirements for efficient scaling, showing that MoE models closely align with dense models in terms of scaling efficiency, with power-law coefficients for MoE-CLM and MoE-MLM approximating those of their dense counterparts. This suggests that MoE models achieve similar scaling behaviors with potentially lower computational costs.
I
MoE Scaling
We find that the scaling behaviors of sparse parameter counts Mixture of Expert (MoE) models are remarkably similar to those of dense model sizes, potentially allowing for reduced compute budget for modeling scaling behaviors due to less activated parameters per token.
In our experiments, we evaluate MoE models ranging from 10M to 500M sparse parameter counts, using model size of 17 with experts, following the settings outlined in Mixtral of experts [40], including its boad-bllancing scheme. The figure below shows different IsoFLOPs curves. Notably, the FLOPs here are calculated based on sparse parameters rather than actually activated ones. We use the method described in the main text to select optimal loss points and fit these around the sample points, enabling us project the optimal model size and number of tokens for larger models (center and right). We observe that the power-law coefficients for CLM and MLM are similar to those of dense models, with MoE CLM vs. Dense CLM at approximately 0.57 vs. 0.58, and MoE MLM vs. Dense MLM at 0.74 vs. 0.77.
Our study strictly focuses models with eight experts, which may not be entirely rigorous. Clark et al. [17] proposed a unified scaling law defining effective training parameters for MoE, aiming to harmonize the scaling laws for Dense and MoE models. Investigation of biological data will be considered as future work.
J Combined Power-law
We applied the fitting function proposed by Chinchilla [36], detailed in Equation 8, to model the effects of various factors on model performance. can provide a loss prediction where neither the parameters or model size not optimal allocation. This loss function simultaneously depends parameters N and
A
B D3
L(N, D)
+E
(8)
No
23

---

where E denotes the irreducible loss. Parameters A, B, and 3 are learned through the fitting process. As N + oe or D oc. the function degenerates form similar to Equation 2, which indicates that it models the scenarios under perfect conditions of other variables.
Given that most of training tokens are used for less than equal one epoch, and that the model size is prone to underfitting at fixed FLOPs, the asymptotic behaviors ((N) at D + oo and L(D) at D).
N + 0o are enough for determining the parameters
To enrich data points, we randomly added several FLOP counts into 25% of the model size and trained these models for 0.25, 0.5, 0.75, and 1 epoch. And adopt the Huber loss fit these coefficients:
min Σ Hubera (LSE (a a log N₁, b 3 log D log Li))
a,b,
where LSE represents the log-sum-exp operator, and 5 10-3 The terms N;, Di and L, de- note the model size, dataset size, and loss of the i-th run, respectively. We fitted the MLM val- idation loss from 110 samples and the CLM validation loss from 149 samples using grid search with E {0, 0.5, ................................ 2), 3 E {0, 0.5, ... 2), E {-1, -0.5- ... ..... .............................. 1}. E {0,5,... 25). and b E {0, 5, ................................ 25}. The final initialized parameters of CLM and MLM both are [e,a, b, a, 3] = [1,5, 10, 0.5, 0.5]. We set the maximum number of iterations to 1000, and the two objectives were essentially achieved after 360 iterations. The exponential powers of learned a and b yielded the coefficients A, B, which were reported in Table A9.
Table A9: Coefficient of Equation
Objective
A
B
CLM
143.9
22036.5
0.367
0.496
MLM
3.365
7.569
0.042
0.099
Substituting all learned coefficients into the following Equation from the original Chinchilla paper:
C
Nope oCp G
(C) = G
6
6
(10)
aA
at
B
a
where G=
b
BB
+
3
+
B
The results closely approximate the trends given in Equations
and 2, confirming our overall findings.
K
IsoLoss
In addition to using the seven different FLOPs counts reported in the main text to determine the optimal model sizes and our scaling law, we also incorporated additional model points into our analysis. We trained using the final loss points of all the CLM and MI.M that run. Figure A15 depicts the contour the fitted function L and the efficient frontier as a red dashed line, presented in log-log space. The frontier interval of Figure 2 is computed from this observation. From this approach, revealed the scaling exponents for model size to be 0.77 in MLM and 0.57 in CLM, very similar the IsoFLOPs profiling method in Section 3.1.
L Training Procedure
We conducted all experiments using Ampere A100 GPUs (80G) equipped with NVLink, utilizing the GLM framework [88, 26] developed based on DeepSpeed and Megatron. We have used total of around I million GPU hours. Our approach predominantly utilized data parallelism, avoiding model parallelism and pipeline parallelism to simplify deployment. Modifications were made to the standard Transformer architecture [82], adopting a DeepNorm [85] strategy and layer normalization [5]. The activation function was set to GLU [71]. RoPE [73] was used to encode position, similar to the settings found in the Transformer++ architecture [79]. We further adopt FlashAttention [18] to
24

---

IsoLoss for CLM
IsoLoss for MLM
Efficient trontier
Efficient frontier N x C
N C93s
109
10a
Model Size
Model Size
108
10a
1019
10
101
1020
10
FLOPS
FLOPS
Figure A15: Parametric fit for CLM and MLM. Unlike the IsoFLOPs method used in the main text to select the optimal model size, these plots use all available data points to fit the models. The left panel shows the contour of the function L and the efficient frontier (indicated by the red dashed line) for the CLM, and the right panel for the MLM. The rainbow dots represent identical loss. The results closely align with using the IsoFLOPs profiling method.
accelerate our training process. The used max LR empirically found to range between 6 x 10-1 and 1.2 10-4 from small to large model size, was used along with a cosine decay strategy to reduce to 0.1 X max LR. Both CLM and MLM were trained under similar settings for model size, with consistent LR and minimum warm-up period of 2.5% steps, extending to at least 100K training steps. All sequences were set to length of 1024, with sequences concatenated using <EOS> delimiter. Based on findings related to loss magnitude and batch size [53]. The Adam W optimizer [50] was used with 31 0.9, $2 0.95,
10 and weight decay of 0.01. All experiments omitted the dropout (it reduced the capacity to hinder model scaling) and trained with bfloat Most pre-training experiments
models extending up 30% beyond one epoch. For the transfer learning setting, we load the finished checkpoint of the pre-training model and disregard the pre-trained optimized state, and learn rest tokens with warmup 5% steps the max LR.
confined to the < epoch, with
M Broader Impact
If the scaling law of the protein language model improves predictions or understanding of protein structure and function, it could potentially have positive impacts on scientific research in fields such as biology, medicine, and drug development. This may facilitate the development of new drugs, accelerate progress in disease diagnosis, or drive advancements frontier research in the life sciences.
N
Model Parameters
Table A10 details the sizes and configurations of all models utilized in this research, training only with data parallel expcept 10B with tensor parallel size 2:
25

---

Table A10: All model eyperparameters. Several of the models presented have been trained using
various learning rate schedules and differing amounts training tokens.
params
d
model
ffw
kv size
head num
layers
4M
192
512
24
8
8
5M
256
683
32
8
7
6M
256
683
32
8
8
10M
320
853
40
8
8
13M
320
1280
40
8
8
19M
448
1194
64
8
25M
512
1365
64
8
8
34M
512
2048
64
8
8
40M
576
1536
64
8
10
47M
576
1536
12
66M
640
2560
64
10
10
77M
480
1280
20
28
85M
768
2048
64
12
12
106M
768
2048
48
16
15
127M
768
2048
48
16
18
154M
896
2389
64
14
16
157M
640
1707
32
20
32
170M
768
2048
48
16
24
200M
896
2389
64
14
21
230M
896
2389
64
14
24
300M
1024
2731
64
16
24
393M
1280
3413
80
16
20
470M
1280
3413
80
16
24
550M
1280
3413
80
16
28
670M
1536
4096
96
16
24
880M
1792
4778
64
28
23
1.2B
2048
5461
64
32
24
1.5B
2304
6144
64
36
24
1.7B
2304
6144
36
28
2.0B
2560
6832
64
40
26
2.4B
2560
6832
64
40
30
2.8B
2560
6832
64
40
36
3.IB
2688
7168
64
42
36
3.4B
2816
15040
128
22
22
4.0B
3072
8192
128
24
36
5.7B
3328
8874
128
26
40
6.2B
3584
9556
128
28
40
7.2B
4096
10923
128
36
36
10.7B
4352
11605
136
32
47
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly made statements to claim the contributions in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed several limitations of our work in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

This work is primarily focused on empirical experiments and extensive val- idation rather than providing theoretical results. Therefore, the paper does not include set of assumptions or proofs for theoretical results since it is not the main emphasis of the research.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details in the main text and Appendix to ensure the reproduction all the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
28

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the implementation details in the Appendix to ensure the production of all the experimental results. Moreover, we release code and script at https://gil ttw.b//gtttuu.... httpc///ctttttteeeeaaalngnoooooo
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
aublic/guides/Code
ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (htt / //niss. cttpsscccp/llllp/ h/ // wewgggiddsss/// /Coeee/biiiS wwt.suuuiiisioiooooiio..oooossoo wwwonPoliossssoo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details in the Appendix to ensure the reproduction all the experimental results. Moreover, we release code and script at https://github.. http://www..oeectttt/t////////ea
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We have presented our results with the error bar in Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the implementation details in the Appendix to ensure the reproduction of all the experimental results.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///rrpp..cc//o/t/ttctceiaue
Answer: [Yes]
Justification:
We have reviewed and obeyed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impact in Appendix M.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have mentioned all the source of used data, code, and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets
31

---

has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release code and script at
ScalingProteinLM.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't include human subjects or crowdsourcing this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't 't include human subjects crowdsourcing in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "yruBzIVolf", "reviewer": "Reviewer_b9DN", "summary": "This paper studies scaling law for two types of protein language models: masked language modelling and causal modelling. The paper finds that both types scale sublinearly but the coefficients of the scaling law are very different. Based on the derived scaling law, the paper then reevaluates the allocation of resources for ESM and PROGEN and shows improved results.", "questions": "See above", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this paper presents a well-motivated study with solid and extensive experimental work, \n* The paper addresses scaling laws in the context of AI4Science, an area that has been rarely studied but is highly necessary.\n* The experimental studies are comprehensive, robust and sound. \n* The paper is very well-written and well-structured.", "weaknesses": "* Evaluations in Section 5 need more rigor:\n    a. Narrow Task Selection: The evaluation tasks, particularly for protein understanding, are limited in scope. The absence of important tasks such as fitness prediction and EC number predictions means that the chosen tasks may not provide a complete picture of the model's performance across various aspects of protein understanding.\n   b. Missing Experimental Details: The paper omits crucial experimental details, including: hyperparameter selection process for the evaluation, train/validation/test set division. Also, it's not clear whether the evaluation data is omitted from training. These make it challenging to evaluate the fairness of comparisons.\n\n\n* Lack of Diverse Sampling Strategies: The paper does not explore different sampling methods, which is a common practices in the field. For instance, it doesn't consider the approach used in training models like ESM, where sampling by UniRef is employed to upsample rare protein sequences. This practice might drastically change the scaling behaviour.\n\n\n* Lack of insights: The paper can feel a bit like a lot of experiment findings stacked without insights. Tt would be good to provide some intuitions about the findings, such as why is masked language modelling overfitting while causal modelling does not, and the difference in the scaling behaviour between protein sequences and natural language sequences. \n\nHowever,  these weaknesses do not negate the paper's contributions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**About Weakness-1: the protein understanding tasks and missing experimental details.** We have provided a comprehensive task performance comparison across 8 protein understanding benchmarks, as adopted from the protein understanding benchmarks available at biomap-research on Hugging Face. These benchmarks span four distinct categories of protein-related challenges:\n\n- **Protein Structure:** Contact Prediction, Fold Prediction\n- **Protein Function:** Fluorescence, Fitness-GB1, Localization\n- **Protein Interaction:** Metal Ion Binding\n- **Protein Developability:** Solubility, Stability\n\nThe results, shown in Table 1 in the attached PDF, demonstrate that our 10.7B model outperforms ESM-3B on 7 out of 8 tasks. This confirms the rationale behind the observed scaling law and addresses concerns about the scope and rigor of our evaluation tasks.\n\n*About the missing experimental details.* The downstream data are collected from corresponding publication and have been well split, we employed the same division for evaluation. We will add the description in the future version. \nAdditionally, we did not implement any specific strategy to filter out downstream data during evaluation. Given the nature of pre-training, the pre-training process is self-supervised, and no label-related information is seen. Although some samples have a high sequence identity with pre-trained sequences, the lack of exposure to label information during pre-training should prevent any substantial data leakage effects.\n\n**About Weakness-2: the lack of diverse sampling strategies.** Thank you for your question and great thinking. Indeed, different sampling strategies may change the loss curve and potentially alter the scaling laws.\nRegarding the diversity of data selection, we chose to use  i.e., UniRef90, and ColabFoldDB. These is one result of sampling strategies from the original datasets. Our choice is based on balancing diversity and data volume. For example, previous ESM models performed very well on UniRef90, as shown in Figure 4 of the referenced paper [3]. Using the entire Uniref100 would degrade model performance [2], while using UniRef50 would result in insufficient data, necessitating data repetition, which introduces another variable. This would require considering scaling laws under different sampling strategies as well as under repetition [1].\nThus, our initial assumption was to pass the data through only once or just slightly more, following a uniform distribution. This assumption allows us to preliminarily avoid the issue of data repetition. Introducing high diversity sampling strategies would require further research on data-constrained scaling laws [1]. Therefore, we initially considered simpler settings, such as those used in early OpenAI and Chinchilla studies.\nRegarding ColabFoldDB, its sequence identity is relatively low. In Appendix G, we compared two small models on UniRef90 and ColabFold datasets, both under uniform sampling conditions, and their performances were quite similar. This outlines our rationale for data and sampling strategy selection.\nThe proposed diverse sampling strategy is an excellent point, In the field of LLMs, there has been some work done[4], but there is still no optimal conclusion. Exploring new scaling laws by considering sampling strategies and data repetition variables would be very interesting for future work.\n\n[1] Muennighoff, Niklas, et al. \"Scaling data-constrained language models.\" NIPS'24.\n\n[2] Rives, Alexander, et al. \"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.\" PNAS'21\n\n[3] Meier, Joshua, et al. \"Language models enable zero-shot prediction of the effects of mutations on protein function.\" NIPS'21\n\n[4] Xie, Sang Michael, et al. \"Doremi: Optimizing data mixtures speeds up language model pertaining.\" NIPS'24\n\n**About Weakness-3: the lack of insights.**\n\n*Regarding the overfitting issue in masked language modeling*: In MLM, a certain percentage of input tokens are masked randomly, and the model is trained to predict these masked tokens using bidirectional context. The bidirectional self-attention mechanisms used in MLM have a higher capacity to overfit compared to the unidirectional self-attentions used in CLM. This is because MLM can utilize the entire context surrounding a masked token, leading to faster memorization of the training data.\nA more systematic study investigating the training dynamics of MLM and CLM is presented by Tirumala et al. [1]. This study observes that MLM tends to memorize faster than CLM, which leads to overfitting, especially when the pre-training data is repeated over multiple epochs. The bidirectional nature of MLM allows for a more comprehensive understanding of the context, but it also means that the model can become overly reliant on specific patterns in the training data, leading to overfitting.\n\n[1] Tirumala, Kushal, et al. \"Memorization without overfitting: Analyzing the training dynamics of large language models.\" NIPS'22.\n\n*Differences in Scaling Behavior: Protein Sequences vs. Natural Language Sequences*\nThe scaling behavior between protein sequences and natural language sequences also differs significantly. Protein sequences are governed by the rules of biological systems and have a different structural complexity compared to natural language sequences. Proteins have specific folding patterns and functional constraints that are not present in natural language.\n\nWhen scaling models for protein sequences, the complexity of the biological rules and constraints means that the models need to capture a wide range of interactions and dependencies. This can lead to different scaling laws compared to natural language models, which primarily focus on syntactic and semantic coherence. The specific coefficient parameters between protein language model (PLM) scaling laws and natural language processing (NLP) scaling laws confirm this difference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oduigb9Hef", "reviewer": "Reviewer_qqtc", "summary": "In this work authors propose to explore training compute-optimal protein language models and aim to derive scaling laws. Specifically, they (i) explore new pre-training data for protein language models (ii) derive scaling laws for protein language models using these datasets for both Masked Language Modeling and Causal Language Modeling objectives (iii) explore scaling laws and transferability between those objectives (iv) use the findings to pre-train two pLMs which reach SOTA performance on downstream tasks.", "questions": "- In Section 5, did you try training the 470M version of your model for the same computing budget as ESM2 ? This would emulate works in NLP [2] where authors find that training smaller models for longer can yield good results with limited inference cost.\n- It would be interesting to see if the derived scaling laws are also reflected in downstream task performance, *e.g.* by fine-tuning models pre-trained for the same computing cost and training data (as I understand, PROGEN2 and ESM2 use different pre-training data).\n\n[2] Touvron, Hugo, et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).", "limitations": "I believe limitations have been properly addressed in Appendix.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Clarity**\n- The paper is clearly motivated and finding domain-specific scaling laws is valuable for practitioners.\n- The paper is very easy to follow and experiments are well described.\n\n**Experiments**\n- Authors conduct extensive experiments and the results on scaling laws are solid.\n- Notably they also derive scaling laws for transfer between CLM and MLM and mixture-of-experts.", "weaknesses": "**Lack of discussion of results** \n- Conclusion, Related Works and Discussions and Limitations have been moved to the Appendix. While I understand the 9-page constraints, I recommend to include at least a discussion/conclusion part in the main body.  \n\n**Novelty**\n- This work closely follows previous work on English LMs such as Chinchilla and presents very limited conceptual novelty.\n\n**Experimental Validation**\n- While the scaling laws analysis is solid, the experimental validation on Protein understanding tasks is limited to 3 tasks whereas previous works have proposed a more extensive benchmark [1]\n\n[1] Elnaggar, Ahmed, et al. \"Ankh: Optimized protein language model unlocks general-purpose modelling.\" arXiv preprint arXiv:2301.06568 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**About the Weakness-1: the lack of discussions.** Thank you for your insightful suggestions. We will make the following modifications to significantly enhance the clarity and impact of our manuscript.\n\n- Adding conclusions or takeaway findings at the end of each main section.\n- Moving the detailed discussion and limitations of our work from Appendix Sections A, B, and C to the main body of the paper to highlight the great potential and future research directions of our work.\n\n**About the Weakness-2: the clarification of novelty.** While Chinchilla focuses on optimizing the model size and number of tokens for training a causal language model within a given compute budget and provides open-source models, our work takes a novel approach. We aim to establish a practical pathway for researchers to develop faithful and powerful protein language models optimized by both CLM and MLM objective in an end-to-end manner. This includes everything from pre-training dataset construction to optimal parameter and dataset allocation, as well as knowledge transfer from other pre-training objectives. Our work holds significant potential for the application of large language models across various scientific domains.\n\n**About the Weakness-3: About the experimental validation**. We have provided a comprehensive task performance comparison across 8 protein understanding benchmarks, as adopted from the protein understanding benchmarks available at biomap-research on Hugging Face. These benchmarks span four distinct categories of protein-related challenges:\n\n- **Protein Structure:** Contact Prediction, Fold Prediction\n- **Protein Function:** Fluorescence, Fitness-GB1, Localization\n- **Protein Interaction:** Metal Ion Binding\n- **Protein Developability:** Solubility, Stability\n\nThe results, shown in Table 1 in the attached PDF, demonstrate that our 10.7B model outperforms ESM-3B on x out of x tasks. This confirms the rationale behind the observed scaling law and addresses concerns about the scope and rigor of our evaluation tasks.\n\n**About the Question-1: the scaling law.** It is indeed common practice to use smaller models in many practical applications, even though these models may not follow Compute-Optimal Training principles. Smaller models are more deployment-friendly, especially for models with high traffic and for downstream task fine-tuning.\nWe did not perform this experiment because it falls beyond the scope of this paper, which primarily discusses how to train compute-optimal models. If we were to discuss the scenario of small models with high computational power, it would require further examination of data repetition issues, introducing a new variable into the analysis.\nWe addressed this in the limitations section (Appendix C), noting that for MLM models, smaller models trained with relatively high computational power can have comparable performance to optimally trained models while being more inference and deployment-friendly. For example, a 3B model trained with 5 epochs (1T tokens) uses the same computational budget as an optimally trained 10.7B model with 265B tokens, and the performance gap between the 3B and 10.7B models is minimal.\nHowever, we know that continuously repeating data can lead to diminishing returns, and some models may even suffer from overfitting. Different model sizes correspond to an optimal data repetition variable. Without accurately determining this variable, it's challenging to predict how much loss smaller models might incur due to data repetition, unless we have infinite data.\nThis area requires further exploration and generalization of current scaling laws, such as data-constrained scaling laws [1]. We anticipate more work in this direction soon.\n\n[1] Scaling Data-Constrained Language Models\n\n**About the Question-2: the downstream task scaling law.** Thank you for your suggestion. \nResearch on the downstream performance of scaling has already emerged in the field of LLMs [1]. This research is indeed fascinating, but it falls beyond the scope of this paper. Due to limitations of the paper space and computational resources, a deeper discussion would introduce other scaling factors such as downstream fine-tuning data size.\nWe briefly compared CLM and MLM under the same data and two different model sizes in Appendix F, using the same computational cost (same FLOPs) for the Contact Prediction downstream task. The conclusion is that MLM consistently outperforms CLM. For a more detailed exploration of scaling laws in downstream tasks, including CLM generation and MLM, you can refer to the xTrimoPGLM paper [2].\n\n[1] When Scaling Meets LLM Finetuning: The Effect of Data, Model, and Finetuning Method\n\n[2] xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1NPSUdcFKL", "reviewer": "Reviewer_wQuW", "summary": "The authors fit scaling laws to determine how to train protein language models compute-optimally, for both causal LM and masked LM tasks. They also consider a setup to consider possible effective data transfer when training on one task and transfer training on the other. Finally, they use their derived scaling laws to train new protein language models compute-optimally, achieving better downstream performance.", "questions": "- Would it be possible to add a lot more detail on the implications of the work, as well as context on how it improves upon prior attempts? In particular, to what extent have previous models been suboptimally trained, and what are the core recommendations the authors have for future related work?\n\n[copied from the weaknesses section] \n- What are the confidence intervals for the scaling exponents for model size and data?\n- Why was this particular functional form for the effectively transferred tokens chosen? (equation 7) Why does it make sense for the data and parameter components to be multiplicative rather than additive?", "limitations": "I thought the discussion of limitations in appendix C was well done.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I think the biggest strength of this paper is its importance – I think that the impact of these kinds of systems will likely be very substantial over the coming years and empirical scaling laws for protein LMs could be very influential. It’s also fairly original because there hasn’t been much work done on this in the past. So I really appreciated that the authors took the time to obtain these results on compute-optimal training.", "weaknesses": "Personally, I feel the presentation of the writeup could be substantially improved. For example: The main body of the paper doesn’t have a discussion/conclusion, which unfortunately made the paper hard to read. In general, I found the writeup to be largely listing results without much discussion of significance. \n- Personally I think it’d be helpful if the authors made some of their discussion more concise (e.g. in section 2.1), and instead adding more detail about how significant their results are. \n- E.g. this could mean calculations saying something like “using our approach, we were able to reach X performance using a factor of Y less training compute, which suggests…”. \n- In addition, I’d suggest including a discussion section in the main body of the paper that outlines the key results and implications, especially in the context of previous work (in a quantitative fashion). What does their work mean that future AI researchers should do? How far off of compute-optimality was previous work? How important are overtraining considerations? Etc.\n\nI think if these kinds of concerns are addressed I'd be happy to increase my score. \n\nMore minor questions/concerns: \n- Lines 164-166: I feel it’d be best to include more decimal places in the reported numbers, so that the product of the data and model size increase equals 10x.\n- What are the confidence intervals for the scaling exponents for model size and data?\n- Why was this particular functional form for the effectively transferred tokens chosen? (equation 7) Why does it make sense for the data and parameter components to be multiplicative rather than additive?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**About the Question-1: the re-organization of the paper.** We appreciate your insights and have taken them into consideration for our revisions. Although we cannot directly modify the paper PDF during the rebuttal period, we demonstrate how we will reorganize and improve the writing style of our manuscript in the next version of our paper.\n\n*Add Discussion/Conclusion Sections*:\nWe will add conclusions or takeaway findings at the end of each main section. Specifically:\n\nFor Section 2: Studying the Data-hungry Observation & Scaling Up Data, we will add discussions to support the two key findings:\n- Scaling the model from 150M to 3B, we noted diminishing returns for CLM and an overfitting tendency for MLM when repeating the UR50/S dataset.\n- Our proposed Expanding Diversified Metagenomic Data (UniMeta200B) addresses these problems.\n\nFor Section 3: Scaling Law for CLM/MLM Protein Language Models, besides the detailed scaling law equations, we will add discussions for the two key findings:\n- In CLM, training data scales sublinearly with model size but follows a distinct power-law with a compute exponent of 0.57. A 10× increase in compute leads to a 4× increase in model size and a 3× increase in training tokens.\n- In MLM, training data scales sublinearly with model size, approximately following a power-law with a compute exponent of 0.77. A 10× increase in compute results in a 6× increase in model size and a 70% increase in training data.\n\nThese simplified takeaway scaling laws will benefit AI researchers who want to train their own optimal PLM under a computational budget.\n\nFor Section 4: Transfer Scaling Scenarios, besides the specific transfer scaling law equations obtained by empirical experiments, we will add discussions about:\n- The significance of the transfer scaling law: Researchers can obtain better MLM models transferred from CLM pre-training under the guidance of the transfer scaling law than merely MLM from scratch following the General MLM scaling law obtained in Section 3.\n- The simplified takeaway transfer scaling law: Training MLM from scratch with 10× the compute requires approximately 7.7× the compute compared to MLM from CLM pre-training, implying that around 20% of the compute budget should be allocated for CLM pre-training to get better MLM models transferred from CLM pre-training.\n\nFor Section 5: Confirming the Scaling Law Effect in Downstream Tasks:\n- Using our concluded scaling law, we obtain a training-optimal 7.2B CLM under the total compute of ProGen2-xlarge, which shows better generation capacities as shown in Figure 6.\n- We also obtain a training-optimal 10.7B MLM model under the same FLOPs as ESM2-3B, which performs better on 8 protein understanding tasks, as shown in Table 1 in the attached PDF.\n- The 470M MLM transferred from CLM outperforms MLM from scratch under the same training budget.\n\nTo summarize: we aim to establish a practical pathway for researchers to develop faithful and powerful protein language models optimized by both CLM and MLM objective in an end-to-end manner. This includes everything from pre-training dataset construction to optimal parameter and dataset allocation, as well as knowledge transfer from other pre-training objectives. Using our approach, we are able to obtain better MLM or CLM models compared to other well-known suboptimal PLM models such as ProGEN2 and ESM. Our work holds significant potential for the application of large language models across various scientific domains.\n\nFinally, we will move the discussion and limitations of our work from the Appendix Sections A, B, and C to the main body of the paper to highlight the great potential and future research directions of our work.\n\nWe believe that these revisions address the concerns raised and significantly enhance the clarity and impact of our manuscript.\n\n**About the Question-2: the confidence interval**. We used `scipy`'s `curve_fit` to fit the parameters, such as in the following example:\n\n```python\nparams, covariance = curve_fit(exponential_fit, x, y, maxfev=10000)\n```\n\nWe did not detail the confidence intervals in our paper, but we will add them using the pseudo code below:\n\n```python\ndef get_lower_and_upper_bound(params, cov):     \n    perr = np.sqrt(np.diag(cov))     \n    alpha = 0.05\n    z = stats.norm.ppf(1 - alpha/2)  # 1.96 for 95% CI\n    lower_bounds = params - z * perr       \n    upper_bounds = params + z * perr \n    return lower_bounds, upper_bounds\n```\n\nThe calculated z×perr for the scaling exponents of the CLM is 0.028, and for the MLM, z×perr is 0.014. Thus, we include the confidence intervals at a confidence level of 1−α=0.95 with the lower and upper bounds shown in parentheses:\n\n- **CLM:**\n  - Model size scaling exponent: 0.578 (0.554, 0.602)\n  - Data size scaling exponent: 0.422 (0.394, 0.449)\n\n- **MLM:**\n  - Model size scaling exponent: 0.776 (0.768, 0.784)\n  - Data size scaling exponent: 0.230 (0.216, 0.244)\n\n**About the Question-3: the particular functional form of transfer scaling law.** Thank you very much for your suggestion. You raise a very good point about the lack of a single standard definition for the scaling law formula. For instance, OpenAI's scaling law differs from Chinchilla's approach. In fact, an additive scaling law could also be used to address your question.\nThe reason we chose the multiplicative form is mainly empirical and was inspired by the following two papers [1,2] ~(in section 4.2).\n[2] provides a detailed comparison between additive and multiplicative scaling laws, concluding that there is not much difference between the two, although the multiplicative form achieves slightly lower extrapolation error on average. Therefore, we adopt this for follow-up analysis. Due to space limitations, we couldn't enumerate all possible experiments, so we naturally chose this one. \nWe will add this explanation in the paper.\n\n[1] Scaling Law for Transfer\n\n[2] WHEN SCALING MEETS LLM FINETUNING: THE EFFECT OF DATA, MODEL AND FINETUNING METHOD", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldEmq0WhYw", "reviewer": "Reviewer_8Frn", "summary": "The main objective of the paper is to explore the scalability of Protein Language Models (PLMs) and to provide models to determine the optimal number of parameters, pre-training dataset size, and transferability between pre-training tasks as functions of the available computational budget.\n\nThe study contains a comprehensive number of experiments and its main findings are:\n\n* The diversity of sequences of the pre-training database can lead to lower perplexity in out-of-distribution validation dataset. This is reflected by the overfitting observed when training models with masked language modelling (MLM) pre-training objective on the Uniref 50S database and a custom made pre-training database enriched with more diverse metagenomic sequences from a variety of sources.\n\n* It establishes scaling laws for that allow for determining the optimal model size and optimal pre-training data size  given computational budgets in the interval 10e17-10e21 both for the MLM pre-training objective and for the causal language modelling (CLM) pre-training objective.\n\n* It also established what the optimal proportion of computational budget should be allocated to pre-training two models one on CLM and another on MLM finding that in general the computational resources allocated to MLM should exceed those for CLM the training tokens reach the 10B threshold where a 1:1 ratio is reached, after which more training tokens should be allocated to CLM.\n\n* It also explores the benefits of transferring models pre-trained with one objective to another objective (MLM to CLM) or (CLM to MLM). Its findings are that the benefits of transferring from MLM to CLM diminish with model size in the 2.7M to 1.7B parameters range; and that the benefits from CLM to MLM grow with model size in the 85M to 1.2B parameters range.\n\n* They validate the scaling laws by pre-training and evaluating models for CLM and MLM tasks with the optimal sizes and comparing them with state-of-the-art methods: ProGen-2 and ESM-2 3B. In the case of CLM, their proposed method achieves greater sequence diversity between the sequences generated, but less structural diversity. In the case of MLM, their proposed method (10.7B) outperforms ESM2 3B by a small margin in two out of the three tasks (fold classification, improvement of 0.03; fluorescence prediction, improvement of 0.04).\n\n* Regarding transfer from CLM to MLM, they also show that a 470M parameter model pre-trained with the CLM to MLM strategy outperforms a model pre-trained in MLM from scratch in two of the benchmarks by tiny margins (contact, improvement of 0.02; fold classification, improvement of 0.01).", "questions": "- Do you have any intuition about why the CLM transferability to MLM scales with model size and the inverse is not true?", "limitations": "I think the authors have overall addressed properly the limitations of their setup and I don't think there is any significant societal impact on their work that has not being acknowledged. The only limitation that I don't think is properly address is that the results are limited to the range of model sizes and compute power they have experimented on and that the experimental setup does not allow for any claim about whether the laws induced can extrapolate outside that range. I don't think the authors have properly addressed this limitation, as I stated in the weaknesses the layout of Figure 2, seems to argue the opposite.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The paper is clearly written and despite the diversity of analysis conducted the main findings and their importance are well conveyed in most cases.\n\n* The paper provides the most comprehensive analysis to date on protein language model scaling and how to optimise the computational resources available.\n\n* The paper demonstrates that the scaling laws induced from their experiments can be used to derive the most optimal pre-training configuration for PLMs both for the generative and predictive purposes.", "weaknesses": "* Figure 2 has quite an extended x axis from 10e17 to 10e24. The interpretation a reader may make of this (and it is the one I'm making) is that the authors are implicitly claiming (or at least suggesting) that the scaling laws are able to extrapolate to that range. However, their data only supports the claim that they are valid in the 10e17 to 10e21. In order, to make the claim (or suggestion) that they extrapolate to the range they show in the figure, there should be at least one data point close to the upper bound of the range (10e24).\n\n* Sections 4.1 and 4.2 were the most hard to follow, in contrast with the rest of the paper that was quite clear despite the complexity and diversity of the analysis. I'm not entirely sure how it could be improved, but I had to read it attentively 3 or 4 times to get the point of the experiment and what the results were. However, it may be a personal issue.\n\n* I think that section 5.1 could benefit of a more detailed discussion on what the results mean regarding protein sequence generation and how the optimal model proposed is better/worse than the state-of-the-art. I am aware of the the constraints on length, so I would suggest including it as an additional section on the appendix. I don't think this is a significant issue, just a minor improvement to make the contributions to the overall field clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**About the Weakness-1**. Thank you for your valuable feedback. Based on your suggestion, we will adjust the x-axis range and include the trained maximum models for 10**22 FLOPs, namely MLM 10.7B and CLM 7.2B in section 5. \n\n**About the Weakness-3: the evaluation of protein generation.** We explain more details about Protein Generation Comparison as follows:\n+ *OOD Dataset PPL Analysis*: PPL represents the probability of the test sequences in the model distribution. The lower the PPL, the closer the distribution of the model and the test set is. In order to test the generalization ability of the model on new data, we use different sequence identity (0.0, 0.3, 0.5) as thresholds to select the test set;\n+ *pLDDT scores from ESMFold* predicted Local Distance Difference Test is the confidence level of ESMFold protein structure prediction. This metric is widely used in methods such as AlphaFold2, ESMFold, and OpenFold. pLDDT filters are often used in protein design (such as RFDiffusion), which can significantly improve the success rate of protein design;\n+ *Natural Sequences Comparisons with Foldseek* Foldseek takes protein structure as input and searches for proteins with similar structure in the database. We use the experimentally-resolved protein structure as the database (PDB database) to explore how the structure of the generated sequences close to PDB (a higher TM-score indicates higher structural similarity). This method has been used to evaluate other methods for protein sequence generation (ProGen2, ProtGPT2);\n+ *Diversity Analysis*: We cluster the two sets of sequences (ProGen2-xlarge and CLM) according to sequence similarity. Sequences with a identity higher than 50% will stay in one cluster. Since the number of input sequences is similar (8,466 vs 8,263), we can measure the diversity of the generated sequences by comparing the number of clusters.\n\n**About Question-1: transfer between mlm and clm.** The effectiveness of pre-training models under a given training budget fundamentally determines their capacity. In Causal Language Modeling (CLM), the loss gradient is calculated for all tokens in the sequence, meaning that each token contributes to the model's learning process. This comprehensive contribution makes CLM training more efficient.\nConversely, in Masked Language Modeling (MLM), the loss is calculated only for a subset of tokens—typically around 20% that are masked. Consequently, only these masked tokens contribute to the model's learning, resulting in lower training efficiency compared to CLM.\nTherefore, when MLM is transferred from a pre-trained CLM, it benefits from the higher overall training efficiency of CLM, leading to improved performance of the finalized MLM model. However, when CLM is transferred from a pre-trained MLM, the lower training efficiency of MLM results in a sub-optimal CLM model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for your valuable feedback on our manuscript. We have taken your comments seriously and have made several important revisions to enhance the quality and readability of our work. Below is a summary of the key changes:

### Additional Experiments

- **Expanded Understanding Experiments:** We have expanded our evaluation to include 8 comprehensive protein understanding benchmarks in the attached PDF. These benchmarks span diverse protein-related challenges, including Protein Structure, Protein Function, Protein Interaction, and Protein Developability. This broader evaluation confirms the validity of our scaling laws and demonstrates our model's superior performance across various tasks.

### Manuscript Reorganization

- **Improved Structure and Clarity:** We have reorganized the manuscript to improve its flow and readability. Sections have been restructured to provide a clearer and more logical progression of ideas, making it easier for readers to follow our research narrative. We will add conclusions or takeaway findings at the end of each main section.

### Detailed Responses to Reviewer Comments

- **Targeted Revisions:** We have addressed each of the reviewers' comments and identified weaknesses point by point. Specific changes have been made in response to your suggestions, and detailed explanations are included in our revised manuscript. We encourage you to refer to these targeted responses for more information on how we have addressed individual concerns.

We appreciate your guidance in helping us refine our work. Thank you once again for your constructive feedback.

Meta-review: The paper explores scaling laws for masked and causal protein language models, finding distinct sublinear scaling coefficients for each. It reevaluates resource allocation when training such models, achieving improved results by training compute-optimal models and exploring transferability between objectives. The study validates these scaling laws through comprehensive experiments, demonstrating better downstream performance and optimal computational budget allocation.

The reviews were generally very positive, highlighting the importance of the topic, and the comprehensive experimental studies. The identified weaknesses were primarily related to presentation, with requests for more in-depth discussion/insights. This was addressed by the authors in the rebuttal, which also expanded on the set of experiments. Post-rebuttal, all reviewers recommended acceptance of the paper.
Decision: Accept (spotlight)