ProtFIM: Fill-in-Middle Protein Sequence Design via Protein Language Models
Authors: Youhan Lee
Hasun Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Protein language modeling
Protein engineering
Text infilling

Abstract: Following the investigation that protein sequence determines its structure and function, engineering protein sequences allows us to optimize the functions of proteins for specific purposes such as enhancement of catalytic activity or binding affinity maturation. In protein engineering, there are many cases where the amino acids in the middle of a protein sequence are changed while maintaining the remaining residues to avoid unwanted functional changes from remaining residues. However, existing research on protein sequence design via protein language models (PLMs) has focused on modifying suffix residues by prompting prefix residues to the model or mutating the overall sequence residues. This is unsuitable for scenarios where the residues located in the middle of the sequence are to be optimized. In this work, we suggest a PLM-based framework to solve the fill-in-middle (FIM) protein engineering tasks. To evaluate the performance of PLMs on the FIM tasks, we design a novel evaluation scheme where PLMs are tasked to generate new sequences while maintaining the secondary structures. Also, we propose a new PROTein language model specialized for the Fill-In-Middle task, ProtFIM. Experiments confirm that ProtFIM performs FIM engineering efficiently, especially for alpha-helix structures, and provides decent protein representations of sequence-function relationships. Finally, we demonstrate an artificial protein sequence design framework composed of ProtFIM and a high-quality structure predictor as a novel tool to optimize protein sequences.

Under review as a conference paper at ICLR 2023
PROTFIM: FILL-IN-MIDDLE PROTEIN SEQUENCE DESIGN VIA PROTEIN LANGUAGE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Following the investigation that protein sequence determines its structure and function, engineering protein sequences allows us to optimize the functions of proteins for specific purposes such as enhancement of catalytic activity or binding affinity maturation. In protein engineering, there are many cases where the amino acids in the middle of protein sequence are changed while maintaining the re- maining residues to avoid unwanted functional changes from remaining residues. However, existing research on protein sequence design via protein language mod- els (pLMs) has focused on modifying suffix residues by prompting prefix residues to the model mutating the overall sequence residues. This is unsuitable for scenarios where the residues located in the middle of the sequence are to be op- timized. In this work, suggest a pLM-based framework to solve the fill-in- middle (FIM) protein engineering tasks. To evaluate the performance of pLMs on the FIM tasks, we design a novel evaluation scheme where pLMs are tasked generate new sequences while maintaining the secondary structures. Also, we pro- pose new PROTein language model specialized for the Fill-In-Midae task, Prot- FIM. Experiments confirm that ProtFIM performs FIM engineering efficiently, especially for alpha-helix structures, and provides decent protein representations of sequence-function relationships. Finally, we demonstrate artificial protein sequence design framework composed ProtFIM and a high-quality structure predictor novel tool to optimize protein sequences.
INTRODUCTION
Proteins play crucial role in various parts of biological processes, and the ensemble of diverse functioning proteins is the basis of life's ' activities, such as immune response and metabolism. Such essential and versatile functions of proteins are encoded in protein sequences which are the arrange- ment amino acid residues. The sequences determine their structures via complex biophysical interactions between residues and these structures directly linked to the functions of proteins. Thus, optimizing the protein's 's function by changing amino acid residues of protein of interest, called protein engineering, has been of great interest in diverse industries such as biofuel (Wen al. 2009).
pharmaceuticals (H Tobin et al. 2014). and agriculture [Rao] 2008).
One of the representatives of protein sequence design methods is mutagenesis technique, which gives evolutionarily plausible candidate protein sequence libraries with the help of genetic engineer- ing (Amold 1998). However, this approach requires substantial efforts in high-throughput screen- ing experiments. Recently, machine learning-guider protein sequence design strategies have been proposed achieve a more efficient sequence space search using experimentally acquired labeled data (Yang et al. 2019a).
With both advances in high-throughput sequencing technologies and language modeling in the field of natural language processing (NLP), protein language models (pLMs), which are trained in an unsupervised manner using tremendous sets of unlabeled protein sequences (Consortium) 2019) have been developed for generating de novo protein sequence (Madani et al. 2020 Hesslow et al.
2022 Moffat et al.) 2022, Ferruz et al.] 2022 Nijkamp et al., (022)). Existing generative PLMs trained using an auto-regressive (AR) strategy (Radford et al. 2019: Brown et al. 2020), and generate sequences conditioning on the prefix protein sequences. Unfortunately, the target region where want change amino acid residues is located the front, existing pL.Ms only

---

Under review as a conference paper at ICLR 2023
Original protein
FIM engineered proteins
WSWWLLADDDDCWIT
TDAARAYDE
TOGROGUUP.DSWAKKT....
Sequence generation
EEGYDKFAIH
KPHIPLIOLR
Well-structured sequence
Poorly-structured sequence
Original structure
Original sequence
An illustrative example of FIM protein engineering. The changed sequences for the target region are generated by generative ML.Ms or the like, and the structures are altered accordingly.
Figure I:
few preceding amino acid residues ("prompts") for sequence generation. The interaction sites, positions that interact with other proteins or molecules to perform their functions and are mainly modified to improve functionality, are evenly located on the protein sequence. To prove this, we collect 3D protein structures from Protein Data Bank (PDB) database (Sussman et al., 1998) and calculate the relative positions of protein-protein interaction sites on the protein sequences (see details in Appendix(A.]]. As illustrated in Figure55 interacting sites are evenly present on the protein sequence. This result suggests that in protein engineering, modifying the amino acid sequence will be done for the middle part of the sequence in many cases. In this case, existing pLMs may not effectively utilize the information behind them, which can result in poor quality of generation.
In this work, regard the middle protein engineering ation problem as in Figure and investigate the possibility of pLMs in FIM protein engineering framework. With the emergence of highly accurate protein structure predictors (Jumper al. 2021 Baek et al. 2021), protein structures are predicted very quickly and accurately with a low cost. Us- ing these vances, we propose new evaluation scheme, Secondary structurE InFilling rEcoveRy, SEIFER, for FIM protein sequence generation. The secondary structures are usually desirable to be preserved (Rubio et al. 2019) since the binding pockets of other interacting proteins or molecules are fixed to some extent. In SEIFER, models are tasked to recommend protein sequences and achieve two conditions: the new sequences must be different from the original sequences and their secondary structures must be fully maintained. So, SEIFER can assess both the diversity and structure of new sequences simultaneously and we believe that SEIFER is suitable for assessing generated sequences in the field of protein engineering which modifies the amino acid residues of original sequences to improve functions. Also, inspired by the latest research in the field of language models (Bavarian Protein language model specialized for the Filldle-Middle task, ProtFIM. Compared to existing PLMs, our proposed ProtFIM use both front ("prefixx)) and back ("suffix") sequence information during training and inference.
fill-inmiddle (FIM) sequence gener-
ct al. 2022b), we propose
Through SEIFER evaluation, we show that ProtFIM can generate diverse sequences while main- taining secondary structure, especially for a-helix. Furthermore, ProtFIM outperforms when engi- neering on residues positioned in the front part of a protein sequence compared to existing pLMs, proving that the FIM training is more suitable for FIM engineering compared to AR pLMs. Finally, through analysis and visualization, we prove that ProtFIM has decent representations of protein se- quences and can serve as a sequence optimization tool accompanied by AlphaFold2. summary, our contributions are:
We define FIM protein engineering as protein sequence infilling tasks and provide the ap- plicability of protein language models on the task.
We propose new evaluation scheme, SEIFER, that can used to evaluate the perfor- mance of ML protein infilling sequence design tasks by considering structural con- servation. Through this evaluation, we find that existing AR pLMs are capable sequence design having a-helix structure.
We propose new type of pLM, ProtFIM, that has both AR and FIM capability. Compre- hensive results show that ProtFIM has efficient and comparable performances in protein infilling and protein representations for protein engineering compared to other ML.
2

---

Under review as a conference paper at ICLR 2023
We show that the ProtFIM acts sequence optimizer, which generates novel sequences with high pLDDT of AlphaFold2 while maintaining the structures essential for the function of the protein.
RELATED WORK
Protein language models Pretraining-based language modeling such as Transformer (Vaswani et al. 2017). BERT (Devlin et al., 2018), and GPT (Ferruz et al.) 2022) have revolutionized natural language processing and shown remarkable performance on various tasks such as language under- standing, sentence generation, and infilling over the last few years. With huge increase in the amount of unlabeled protein sequences (Consortium] 2019) produced by high throughput sequenc- ing technologies, pLMs have been introduced and resolved ved the challenges in protein science and en- gineering by learning protein languages. BERT-style models primarily provide protein embeddings to solve prediction problems, including protein structure prediction (Rao et al.) 2020 Jumper al.) 2021 Lin et al., 2022). function prediction (Brandes et al., [022). and property prediction (Rives) al. 2021). GPT-style architectures are utilized in resolving generation challenges such as protein
sequence design (Madani et al. 2020 Hesslow et al.
2022 Moffat et al., 2022 Ferruz et al. 2022
Nijkamp et al. 2022).
Protein sequence design Attempts to efficiently design protein sequences can be divided into two categories: a method for conducting large number of high-throughput experiments with mutagen- esis and machine learning-based sequence generation method. Recent advances in experimental- based methods (Fowler& & Fields (014)) allow us to assess the functional changes of mutated pro- tein sequences at a large scale and produce a lot of labeled data. Many machine learning- based sequence design methods generate the optimized sequences iteratively based on the feedback of la- beled data (Yang et al., 2019a] Xu al. 2020: Wu al. 2021: [Shin et al.) 2021). Unfortunately, both approaches require lot of cost and effort in experiments. Recently, several works generate protein sequences conditioned on given 3D structures using a single energy function (Alford et al.) 2017). convolutional neural networks (Zhang et al. 2020; Qi & Zhang 2020). graph neural net- works (Ingraham et al.) 2019 Jing et al. 2020; Strokach et al.) 2020) Dauparas et al. 2022), or Transformers(Hsu et al. 2022). Since these works require 3D coordinate information to generate
sequences, generation may be limited only to areas where high-quality structures exist. Also, in these works, CATH (Orengo et al.) 1997) is used to evaluate how similar the generated sequences are to the original sequence. This evaluation method may not be suitable for protein engineering, which aims to change the sequence to have a better function. In parallel, generative pLMs such as RITA (Hesslow et al. 2022), DARK (Moffat et al. 2022). ProtGPT2 (Ferruz et al.) 2022). and Progen2 (Nijkamp et al. 2022 have been developed. These generative pL.Ms generate protein se- quences having well-folded and viable structures even though these methods do not employ any structural information. However, due to the nature of the AR model itself, these methods utilize only the preceding sequence information during sequence generation. Our proposed ProtFIM has both AR and FIM property, resulting in efficient FIM protein engineering.
3 METHOD
Problem Setup In NLP. infilling is defined as generating complete text æ given incomplete text i, including one or more missing spans. Similarly, we can regard protein engineering on middle residues as an infilling task where models are tasked to return new protein sequences s given in- complete protein sequence containing missing residues on the target region. Additionally, in the protein infilling task, there is special structure conservation constraint where the secondary struc- ture of the target site is maintained to approximate the protein engineering scenario properly. Taken together, our goal is to develop pLM, f(; 0), which outputs complete protein sequence based on a distribution (({{)) and sequence must have different residues while having the same secondary structure as that of original residues.
3.1 MODEL REQUIREMENTS
We suggest four key characteristics of MS suitable for protein infilling tasks follows:
3

---

Under review as a conference paper at ICLR 2023
Dynamic property: The model can handle various lengths of protein sequences because the lengths the middle sites are diverse depending on various applications.
Previous studies reveal that AR pLMs have data-driven co-evolutionary rules across natural protein sequences and generate plausible sequences that tend to be well-folded. So, ML. which have both AR and infilling capability would be optimal.
Causal modeling:
Efficiency: Various strategies, such as pre-processing training data, modifying the model architecture, and using special tokens for controlling,
be used. However, these ap- proaches must be fulfiiled as efficiently as possible because protein sequence length is relatively long (we use the maximum length of residues 1024 in this work).
Diversity: Because there are many combinations giving the same secondary structures, PLMs which generate diverse sequences different from existing sequences are preferred.
To achieve the above characteristics, adopt the idea of FIM transformation, which is very recently proposed FIM causal language modeling strategy by [avarian et al. (2022a). The following section explains how to develop FIM pLMs and generate protein sequences using the model.
3.2 MODEL DEVELOPMENT
FIM training In FIM transformation, span of text from the middle of a whole sentence is moved to its end, and additional special tokens are introduced for marking where spans are from. The trans- formation is stochastically fulfilled during causal language modeling training. Intriguingly, this sim- ple and straightforward transformation successfully gives fillinethe-mmidle (FIM) capability to the model without modifying model architecture and sacrificing left-to-right causal generation capacity. The transformation is easily applied to protein sequence modeling as follows. First, we tokenize each residue R of a protein sequence with length N to the sequence consisting of corresponding tokens T (see eqn. and
(Ri, R2 Rx)
(1)
S = (71, T2 T,....,TTT)
(2)
Second, conduct uniform sampling to get the start position K of the middle span of length L and add special tokens [PRE], [MID,, and [SUF] at the beginning of each prefix, middle, and suffix part, respectively. Finally, HIM-transformer sentences created by concatenating prefix, suffix middle in order as eqn. 3
= =(PRF, Ri...., RK-1, (SUF), RK+L-1, Ry. (MTD), RK...., RK+L)
(3)
Because several residues are needed to form secondary structure, the middle residue sampling conducted so that both prefix and suffix parts have at least four residues. The traditional GPT2 architecture from Hugging Face (Wolf al. 2019) is used for training, and FIM transformation is applied to the input with a 50% frequency. We denote pLMs trained using FIM transformation as ProtFIM in this work. More details written in Appendix A.4.
FIM inference for middle residue engineering For generating complete sequences in protein infilling tasks, we consider the target region the middle part, and the front and back regions the target region are prefixes and suffixes. Then, we make prompt for FIM generation by concatenating prefix part, suffix part, and [MID] token as eqn.
p = ((PR), RK-1, [UFF), RK+L+I. RN (MID))
(4)
EXPERIMENTS
Section illustrates our proposed evaluation scheme, SEIFER, specially designed for protein in- filling tasks. Section 4.2 describes metrics and various baseline models covering representative language modeling approaches such as causal language modeling (CLM) and permutation language modeling (PLM). Section 4.3 includes evaluation results of SEIFER tasks. Then, section and
4

---

Under review as a conference paper at ICLR 2023
FIM generator
Alpha Helix UAAAA
[C]
Sequence
generation
Generated sequences
Generated structures
Generated secondary structures
_GGEEGYDKFAIH..
_EEEHHHHCCCC...
AlphaFold
.GHHOHEFPLLOUR...
EEEECCCCCCC...
Recovery
evaluation
Original sequence
Original structure
Original secondary structure
KETDAARAIIDE...
Esperiment AlphaFold
VESAR
..EEHHHHCCC....
Ground
truth
Figure 2: Illustration of SEIFER evaluation scheme, which estimates the recovery rates of the secondary structures of the generated structure for the original secondary structure.
provide ablation studies of the SEIFER task concerning relative position and length of target re- gion. Additionally, other metrics for evaluating pLMs, such as perplexity and sequence recovery, are provided Appendix(.66
4.1 EVALUATION
Protein secondary structure Protein secondary structures play key role as intermediate be- tween the primary sequences and functional tertiary structures that determines the function of pro- teins in variety of biological process. Therefore, designing properly optimized combinations of residues having the same secondary structures can lead to enhanced function of protein (Rubio et al., 2019). Protein secondary structures are categorized into regular and irregular categories. First, the regular structure includes o-helix (H). 8-sheet (E) (Pauling et al. 1951). and the irregular structure type coil (C). this work, we adopt a 3-class secondary structure definition and those structures are calculated via DSSP (Kabsch & Sander 1983).
Protein secondary structure recovery via infilling this work, we propose a new evaluation scheme, Secondary structurE InFilling rEcoveRy, called SEIFER, evaluating the sequence genera- tion and structure conservation simultaneously. In the task, first, models are tasked to generate var- ious sequences to fill the provided target sites. Since secondary structures are calculated based three-dimensional structural information, the characterization of tertiary protein structures for each generated sequence must be preceded. Unfortunately, conducting experimental characterization on all the new sequences is practically impossible. Instead of this, we utilize Alphafold2 (Jumper et al., 202T]. which has shown near-experiments prediction performance, to predict tertiary structures of all generated sequences. Then, secondary structures of each new sequence are calculated via DSSP algorithm using DSSP module Biopython (Cock et al.) 2009). Finally, the secondary structures of new sequences are compared to the original secondary structures. We assign positive value, 1, on the case where all new residues have the same secondary structure as the original sequences. And all other cases negative, O. We illustrate the process of SEIFER in Figure We use pro- teins presented in CASP14 to obtain candidate middle sites for SEIFER tasks. And, we argue that experimental setting is reliable because AlphaFold2 was stringently assessed and proved by markable prediction performance on the proteins in the CASP14. Additionally, we use the middle sites, which have minimum lengths of 10, 6, and for helix, beta, and coil structures, respectively, considering the average number of residues for the structures.
Difference of SEIFER over protein residue recovery task Sequence recovery has been widely used to evaluate the generation performance of protein generative language models (Ingraham et al.) 2019 However, considering that the objective sequence optimization is to design new sequences with better target properties, recovery of original residues would not be proper. So, a metric is needed to evaluate whether the model can generate a variety of sequences while maintaining the function of the protein. Because the function of protein is directly linked to local structure, evalu- ating the model ability that generates different residues with the same local structure is a promising way. So, argue that our proposed SEIFER tasks appropriate for simulating sequence engineer-
5

---

Under review as a conference paper at ICLR 2023
ing scenarios, because in SEIFER tasks models are tasked to recover the protein's ' local structure, especially, secondary structures, not residues.
4.2
EXPERIMENTAL SETUP SETUP
Baseline
We compare our ProtFIM with other pLMs covering diverse generation strategies.
. ProGPT2-C:
To prove the effect of FIM over AR in protein engineering, we train AR-based pLMs using the same data, hyperparameters, and the number of parameters compared to ProtFIM. It is similar to the previous AR model, Pro1GPT2 (Ferruz et al. 2022). but our model is trained in the amino acid level. Thus, we name the amino acid residue-level (character-level) version of ProtGPT as ProtGPT2-C.
ProGen2: ProGen2 (Nijkamp et al. 2022) is concurrently released suite of AR pLMs with various parameters.
ProtXLNet:
XLNet Yang et al. 2019b) can protein sequences using prefix and suffix in- formation. Like FIM, sequences of target sites are generated rugresegryyyvvely with con- ditioning on bidirectional context from both the prefix and suffix. We borrow the publicly released ProtXI.Net model, a variant of XL.Net for protein (EInaggar al. (022).
This generator is used to approximate the random mutagenesis tech- nique, error-prone PCR (McCullum al. 2010), which is still commonly used in protein engineering (Dror et al. 2014).
Random generator:
Evaluation metrics SEIFER measures how many sequences with the same secondary structure exist among new sequences created by generative model. It is like retrieval recommendation engine for protein sequences. In the SEIFER task, all models generate K sequences for N middle sites, and all sequences are recovered. If the whole secondary structures are recovered, it is a true positive (TP). Then, Precision@e @ K is the mean of TP/K for N sites. Also, we use Retrieval which assumes posi- tive case where any true positive sequence exists in generated K sequences, zero otherwise. Thus, Retrieval @ K is (the number of sites having TP among KY/N.
evaluated by whether the whole secondary structures at each target site
4.3 EXPERIMENTAL RESULTS ON SEIFER TASKS
As shown in Table T and all ML have better performance than the random generator in he- lix structure recovery in views of both retrieval and precision. These results describe that pLMs are promising tools to fill in middle residues target protein during protein engineering on he- lix structure. In contrast, all pLMs perform similarly worse than the random generator in the 3-sheet and coil recovery. To investigate the result, we check the distribution of secondary struc- tures for the proteins with known structures by calculating the distribution of secondary structures in proteins from PDB (details are described in AppendixA.2). Figure 6a and (6b illustrate 3-classes and 4-classes secondary structure distribution, showing that the a-helix structures are dominant in natural protein structures. This empirical result is consistent with the widely known observation in the protein community. We conjecture that this imbalance gives unwanted a-helix bias in existing protein sequence datasets. Additionally, the coil usually has unordered noisy structures. Taking the above facts together, is possible say that the similar or worse performances of pL.ss 3-sheets and coil cases are reasonable because helix bias makes it models hard to learn the rules of generating residues consisting of the coil and 3-sheet.
Meanwhile, we compare ProtFIM over ProtGPT2-C to see the effectiveness of FIM compared to CLM. As shown in each table's fifth and sixth rows, ProtFIM performs better than Pro1GPT2-C in helix recovery. Because both models are trained using the same data, hyperparameters, and the number of parameters except for the utilization of fill-in-midlle transformation, these results support that conditioning both prefixes and suffixes during generation is essential for better sequence design for protein engineering.
We also compare ProtFIM with other pLMs, such ProGen2 and ProtXLNet. XLNet is another possible model which is able to fil--n-middle protein engineering using both the prefix It is found that Net shows strong performance in a-helix compared to the similar scale of
and suffix.

---

Under review as a conference paper at ICLR 2023
Table 1: Model performances on SEIFER tasks in terms of retrieval.
II (a-heliix R@3
E (3-sheet)
C (Coil)
Model
#Params
Objective
R@s
R@3
R@5
R@3
R@s
Random Generator
0.46
0.60
0.76
0.82
0.76
0.80
ProGen2-smad
151M
CLM
0.59
0.71
0.67
0.64
0.69
0.80
mroGen2-mediun
764M
CLM
0.54
0.66
0.69
0.79
0.70
0.79
ProGen2-large
2.7B
CLM
0.59
0.64
0.70
0.79
0.69
0.82
ProtXI.Net
409M
PLM
0.61
0.69
0.66
0.75
0.66
0.76
ProtGPT2-C
85M
CLM
0.58
0.66
0.71
0.79
0.69
0.74
ProtFIM (ours)
85M
FIM
0.57
0.71
0.74
0.82
0.74
0.81
Table 2: Model performances SEIFER tasks in terms of precision.
H (a-helix)
E (S-shect)
C (Coil)
Model
#Params
Objective
P@3
P@5
P@3
P@5
P@3
P@5
Random Generator
0.25
0.25
0.49
0.47
0.47
0.45
ProGen2-small
151M
CLM
0.32
0.32
0.45
0.43
0.47
0.48
764M
CLM
0.31
0.32
0.45
0.47
0.49
0.47
ProGen2-large
2.7B
CLM
0.36
0.34
0.44
0.45
0.50
0.51
ProtXINet
409M
PLM
0.37
0.36
0.42
0.42
0.49
0.49
ProtGPT2-C
85M
CLM
0.31
0.31
0.48
0.47
0.45
0.45
ProtFIM (ours)
80M
FIM
0.31
0.32
0.45
0.48
0.48
models, such as ProGen2-small and ProGen2-large. These results prove that sequence design needs
to be conducted using the surrounding context of target sites. On the other hand, proposed ProtFIM shows comparable performance in term of retrieval and competitive performances in term of precision compared to other larger models by 2-20 times. These results show that the FIM scheme is parametrically efficient for protein middle engineering tasks.
4.4 PERFORMANCE WITH REGARD THE POSITION OF TARGET REGION
We start with an assumption that previous AR pLMs would be weak in FIM protein engineering because the sequence generation AR PLMs fulfilled by conditioning on only prefixes residues. To verify whether this phenomenon occurs, we ablate the SEIFER performance concerning the relative position the target middle sites. After dividing each protein sequence into four parts, the a-helix recovery performances of each model corresponding each part averaged and illustrated in Figure Figure3aalnd Figure 7al Interestingly, in the first part (front part), only two models, ProtFIM and ProtXLNet, which consider both prefix and suffix part outperforms the random generator, while AR models such as ProtGPT2-C and ProGen-series do not. These results prove our assumption empirically.
Additionally, the fact that ProtFIM outperforms ProtXI in the front part shows the effectiveness of the FIM training scheme because ProtFIM has five times fewer parameters. Meanwhile, it is found that PLMs are generally better than the random generator in other parts, supporting the effectiveness of pLMs on protein middle engincering. In addition, it can be seen that the model's performance is not uniform over positions. We think that it is due to the lack of an evaluation dataset because the number of used CASP proteins 28. However, since the models are compared under the same conditions, the insight obtained from the performance comparison in the experiments is reliable.
4.5 PERFORMANCE WITH REGARD LENGTII OF TARGET REGION
We can see that the random generator shows comparable performances to PLMs in several tests in the above results. To investigate this phenomenon, we ablate the SEIFER performances according to the length of the middle sites. We partition the range of lengths into four parts, and plot correspond- ing averaged Recall@ K and Precision @ K as Figure 3b and Figure 7b Interestingly, the random generator performs similarly to ML in the first quarter (short length size). However, the perfor-

---

Under review as a conference paper at ICLR 2023
Preerit ProtaPT2-C Rardom
hotinn--maal mooden--madum
Motann-lagge PobXLne
ProffN ProteeTz-C
Proteen-mmall
ProGen2emeciam
PrGenn/aage
Pettile
Random
0.6
0.4
0.6
0.3
0.5
0.2
0.4
0.1
0.3
0.0
(0.0, 0.25]
(0.25, 0.5] 10.5, 0.75] Relative Position Bins
(0.75, 1.0]
(10, 20]
(20, 30]
[30, 40)
(40, 50]
Length Bins
(a)
(b)
Figure 3: Performance changes in the term of retrieval with regard to (a) relative positions and (b) length middle sites.
mance of the random generator drastically drops as the length of the target sites becomes longer, and fails all predictions when the middle sites are longer than 30.
Meanwhile, the performance of pL.Ms degrades gradually and fails at the last part, where the middle sites are longer than 40. All the results imply that the length of the middle sites is the main factor for model performance. We explain this using the degree of freedom on possible protein structures of target middle sites. Since the nigh-dimensional interactions between amino acids make the structure of the protein, the structure determined some extent by the structural context from other residues except residues of the middle sites. In other words, the degree of freedom in the structure of the middle sites relatively small due to the non-target residues. Considering that any amino acid is building block of a a-helix, B-sheet, and coil structure, even if the amino acid is randomly sampled, there will be a high probability of obtaining the desired original structure in FIM scenarios. Meanwhile, the observation that pLMs still work at the longer middle sites shows that pLMs would be promising solution for long FIM protein sequence design, giving efficient sequence search compared to random generator.
ANALYSIS AND VISUALIZATION
Table 3: Zero-shot fitness prediction FLIP tasks. All scores Spearman correlation.
Model
#Params
Objective
AAV
GB1
Meltome
Meta Avg.
ESM-1b (mean)
750M
MLM
0.36
0.34
0.71
0.47
ESM-1v (mean))
750M
MLM
0.72
ProGen2-small
151M
CLM
0.39
-0.21
0.56
0.25
ProGen2-mmdium
764M
CLM
0.18
-0.11
0.59
0.22
ProGen2-large
2.7B
CLM
0.41
0.24
0.68
0.44
ProtXLNet
409M
PLM
0.33
0.29
0.47
0,36
ProIGPT2-C
80M
CTM
0.40
0.18
0.53
0.37
ProtFIM
80M
FIM
0.39
0.25
0.60
0.41
5.1 REPRESENTATION QUALITY
Collecting experimental functional properties of protein sequence gives insights into sequence-to- function relationship called fitness landscape. In protein engineering, the fitness landscape used to rank designed sequences. To this end, pLMs can provide sequence representation for fitness pre- diction. Recently, FLIP benchmarks have been introduced to assess the quality of representations of Ms (Dallago al. 2021). Using FL.IP, compare the embeddings of ProtFIM with baselines.
8

---

Under review as a conference paper at ICLR 2023
Generated structures
1.0
ProtFIM
Original structure _MPDILOALREIARVURH..
ProGen-large
Cumulative Dersily
0.8-
Random
RLQALREIAR
0.6
PLDOT: 97.14
KADALREMV
pLDOT: 97.05
-40
-30
-20
-10
0
4pLDDT
(a)
(b)
Figure 4: (a) Cumulative density plot on pLDDT change and (b) an example of a case where pLDDT increases or decreases after protein sequence design via ProtFIM.
Additionally, ESM-1b and 1v (Rao al. 2020 added compare FIM with masked language modeling (MLM). In FLIP, embeddings of sequences are directly used to predict fitness without fine-tuning pLMs. For a fair comparison, embeddings are obtained via averaging of all residue rep- resentations. Table 3 shows that ProtFIM has comparable zero-shot fitness prediction performance even the ProtFIM capacity is multiple times smaller than other models. This result implies that the embeddings of ProtFIM are effective for both FIM protein engineering and zero-shot fitness prediction. Detailed scores are included Appendix
5.2 PLDDT CHANGE AND VISUALIZATION
AlphaFold2 gives per-residue confidence metric called the predicted local distance difference test (pLDDT)) ranging from 0 to 100. Recently, several works have used the metric as a scoring criterion to assess designed protein sequence by assuming that the higher PLDDT, the better and more plau- sible structure (Moffat et al. 2022; Wang et al. 2022). To assess the FIM engineering performance of models in terms of PLDDT, I, visualize the difference between pLDDT of the structure of both new sequences and the corresponding original sequence using a cumulative density plot. Figure 4a reveals that positive cases where pLDDT increases after FIM engineering but pLMs have more chance to get sequences with higher PLDDT. We cherry-pick a protein and visualize the original structure and modified structures through ProtFIM shown in Figure 4b) The new two sequences of middle sites are different from the original sequences, but all have a-helix. In- terestingly, in-depth visualization considering the side-chain unveils the subtle difference, resulting in well or poorly-optimized sequences. All the above results demonstrate that our model, with the help of AlphaFold2, can serve as sequence design framework, which optimizes the target sequence while maintaining the structures essential for the protein's function.
for all models,
CONCLUSION
this work, we show the FIM protein sequence design framework via pLMs and propose a new protein language model, ProtFIM, which is specialized for the framework. By evaluating various models via our proposed new evaluation scheme, SEIFER, ProtFIM performs FIM protein sequence design efficiently compared to existing pLMs. Additional analysis and visualization also prove that ProtFIM is a promising tool for practical protein engincering such as fitness prediction and sequence optimization.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O'Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6):3031-304 2017.
Frances Arnol. Design by directed evolution. Accounts of chemical research, 31(3):125-131, 1998.
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong. Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373 876, 2021.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint 11Xx:22277.125555 2022a.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint Xrxiv:2007 220.:2222.4.55 2022b.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: A uni- versal deep-learning model of protein sequence and function. Bioinformatics, 210))2100- 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Drew H Bryant, Ali Bashir, Sam Sinai, Nina K Jain, Pierce J Ogden, Patrick Riley, George M Church, Lucy Colwell, and Eric D) Kelsic. Deep diversification of an aav capsid protein by machine learning. Nature Biotechnology. (9(6))99--696, 2021.
Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilezynski, al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 2009.
UniProt Consortium. Uniprot: a worldwide hub of protein knowledge. Nucleic acids research, 47 (D1):D506-D515, 2019.
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, 2021.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpan. Science, pp. eadd2187, 2022.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810..00
2018.
Adi Dror, Einav Shemesh, Natali Dayan, and Ayelet Fishman. Protein engineering by random muta- genesis and structure-guided consensus of geobacillus stearothermophilus lipase t6 for enhanced stability in methanol. Applied and environmental microbiology, 80(4):1515- 2014.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10)7112--1127, 2022. doi: 10. 10111199/PPIMMMM 1099TPPAM1.22213309553
10

---

Under review as a conference paper at ICLR 2023
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is deep unsupervised language model for protein design. Nature communications, 13(1):1-10, 2022.
Douglas M Fowler and Stanley Fields. Deep mutational scanning: new style of protein science.
Nature methods, 11(8)801-887, 2014.
Peter H Tobin, David H Richards, Randolph A Callender, and Corey Wilson. Protein engineering: new frontier for biological therapeutics. Current drug metabolism, 15(7):743-756, 2014.
Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli, and Debora Marks. Rita: a study on scaling up generative protein sequence models. arXiv preprint Stti:::22000078888 2022.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in neural information processing systems, 32, 2019.
Anna Jarzab, Nils Kurzawa, Thomas Hopf, Matthias Moerch, Jana Zecha, Niels Leijten, Yangyang Bian, Eva Musiol, Melanie Maschberger, Gabriele Stoehr, et al. Meltome atlas-thermal pro- teome stability across the tree of life. Nature methods, 1755)=95-5503, 2020.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning from protein structure with geometrie vector perceptrons. arXiv preprint arXiv: 20te::009.0141,,2000 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583- -899, 2021.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recog- nition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12):5577-263) 1983.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint /412.6 69800 2014.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel- Zazzl-Zarandi, Tom Sercu, Sal Candido, al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled 171 2017.
weight decay regularization. arXiv preprint
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03440 0..490 2020.
Elizabeth O McCullum, Berea AR Williams, Jinglei Zhang, and John C Chaput. Random mutagen- esis by error-prone per. In In vitro mutagenesis protocols, pp. 103-109. Springer, 2010.
Lewis Moffat, Shaun M Kandathil, and David T Jones. Design in the dark: Learning deep generative models for de novo protein design. bioRxiv, 2022.
Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. arXiv preprint arXiv:2206.13551 2022.
Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thomton. Cath-a hierarchic classification of protein domain structures. Structure, 508) 1109, 1997.
Linus Pauling. Robert B Corey, and Herman R Branson. The structure of proteins: two hydrogen- bonded helical configurations of the polypeptide chain. Proceedings of the National Academy of Sciences, 37(4):205-21 1951.
11

---

Under review as a conference paper at ICLR 2023
Yifei Qi and John ZH Zhang. Denseepd: improving the accuracy of neurallnetworr-baseee compu- tational protein sequence design with densenet. Journal of chemical information and modeling, 60(3):1245-1252, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
A Gururaj Rao. The outlook for protein engineering in crop improvement. Plant physiology. 147 (1):6-12, 2008.
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure leamnrr. Biorxiv, 2020.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e001233111,, 2021.
Marcelo Ventura Rubio, César Rafael Fanchini Terrasan, Fabiano Jares Contesini, Mari- ane Paludetti Zubieta, Jaqueline Aline Gerhardt, Leandro Cristante Oliveira, Any Elisa de Souza Schmidt Gonçalves, Fausto Almeida, Bradley Joseph Smith, Gustavo Henrique Martins Ferreira De Souza, et al. Redesigning n-glycosylation sites in gh3 3-xylosidase improves the enzymatic efficiency. Biotechnology for biofuels, 12(1):1-14, 2019.
Jung-Eun Shin, Adam Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C. Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
Alexey Strokach, David Becerra, Carles Corbi- Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell systems, 1 1444:4022-11, 2020.
Joel L Sussman, Dawei Lin, Jiansheng Jiang, Nancy Manning. Jaime Prilusky, Otto Ritter, and Enrique E Abola. Protein data bank (pdb): database of three-dimensional structural information of biological macromolecules. Acta Crystallographica Section D: Biological Crystallography, 54 (6):107--1084, 1998.
Baris E. Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consor- tium. Uniref clusters: a comprchensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31(6):926-99) 932, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- tion processing systems, 30, 2017.
Jue Wang, Sidney Lisanza, David Juergens, Doug Tischer, Joseph L Watson, Karla M Castro, Robert Ragotte, Amijai Saragovi, Lukas F Milles, Minkyung Back, et al. Scaffolding protein functional sites using deep learning. Science, 377(6604):387-394 2022.
Fei Wen, Nikhil U Nair, and Huimin Zhao. Protein engineering in designing tailored enzymes and microorganisms for biofuels production. Current opinion in biotechnology. 20(4):412 2009.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of the-art natural language processing. arXiv preprint arX1v:1910.03771, 2019.
Nicholas C Wu, Lei Dai, C Anders Olson, James O Lloyd-Smith, and Ren Sun. Adaptation in protein fitness landscapes facilitated by indirect paths. Elife, 5:e16965, 2016.
Zachary Wu, Kadina E Johnston, Frances H Amold, and Kevin K Yang. Protein sequence design with deep generative models. Current opinion in chemical biology, 65:18-27, 2021.
Yuting Xu, Deeptak Verma, Robert P Sheridan, Andy Liaw, Junshui Ma, Nicholas M Marshall, John McIntosh, Edward C Sherer, Vladimir Svetnik, and Jennifer M Johnston. Deep dive into machine learning models for protein engineering. Journal of chemical information and modeling. 60(6): 2773-2790, 2020.
12

---

Under review as a conference paper at ICLR 2023
Kevin K. Yang, Zachary Wu, and Frances H Arold. Seaihiin--eanning-gge directed evolution for protein engineering, Nature methods, 16(8):687-694, 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019b.
Yuan Zhang, Yang Chen, Chenran Wang, Chun-Chao Lo, Xiuwen Liu, Wei Wu, and Jinfeng Zhang. Prodconn: Protein design using convolutional neural network. Proteins: Structure, Function, and Bioinformatics, 88(7):819-88) 2020.
A APPENDIX
A.I INTERACTION SITES EXTRACTION
Density
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Relative position
Figure 5: Relative positions of the interacting sites on the protein sequences.
We download 3D protein structures from PDB database (Sussman al., 1998) and extract protein structures satisfying several conditions: having more than two protein chains; having UniProt ID and a length of the entire sequence in mmCIF dictionaries from MMCIF2Dict module of Biopy- thon (Cock et al. 2209).. We hypothesize that the two residue pairs of two different chains would be involved in the interaction if any atom excluding hydrogen of the residues were at a Euclidean dis- tance of 8Â less. Then, identify all residues which likely to be involved in the interactions and find where these residues are located on the entire protein sequence.
A.2 SECONDARY STRUCTURE STATISTICS
We analyze the secondary structures of 166,512 structures that can be processed through a DSSP module of Biopython. Biopython classifies the secondary structures as eight classes by default:
alpha helix (4-12) (code: 'H'). isolated beta-bridge residue (code: 'B'), strand (code: 'E'), 3-10
helix (code: 'G'), pi helix (code: 'T'), turn (code: "T"), bend (code: 'S'), and none (code: '-'). In
our study, The eight classes are mapped the three classes as follows: 'H', 'G", and 'I' are mapped to the a-helix class 'H'; 'B' and 'E' are mapped to the 8-sheet class 'E'; 'T', 'S', 'C', and '.' are mapped to the coil class 'C', addition, in Figure 6b is displayed separately.
Figure 6a and 6b show that a-helix substructures are dominant in natural proteins, meaning im- balance. Furthermore, coil structures have rules that are difficult to capture. Therefore, the model trained using the existing natural protein database would be familiar with the a-helix generation. Therefore, preprocessing or encoding technique that can alleviate the a-helix bias can be a good research topic in the future.
13

---

Under review as a conference paper at ICLR 2023
Counts
Counts
H
Secondary structure
Secondary
(a) 3-classes
(b) 3-classes and
Figure 6: Secondary structure distribution of proteins in PDB database. H, E, C, and correspond to a-helix, 8-sheet, coil, and none-type. (a) describes 3-classes secondary structure distribution. And, because coil can be divided into two categories, the coil and none-type structure in DSSP algorithm, can calculate 4-classes distribution as shown in (b).
A.3 TRAINING DATASETS
For training, protein sequences from UniRef50(Suzek al. 2013) dated March 28, 2018 version are used to avoid leakage of CASPI3, 14 and conduct a fair comparison with other models. 5% of protein sequences in the UniRef50 are randomly selected as a held-out validation set. The total number of sequences in training data is 25M
A.4 TRAINING DETAILS
ProtFIM is trained with a batch size 128. The maximum length of each protein sequence we used for training is 1024. For ProtFIM optimization, we use AdamW optimizer Kingma & Ba (2014); Loshchilov & Hutter (2017) with weight decay ratio of 1e-5. The learning rate is scheduled using cosine-warmus strategy. The total optimization step is 500k with 1k warmup steps. We train the model on 8 NVIDIA A100s in days. FIM transformation is applied with 50% of probability. The model consists of 12 layers with feature dimension of 768. The architecture is based on the released GPT2-base model by HuggingFace (Wolf al., 2019).
A.5 GENERATION HYPER- PARAMETERS
We conduct sequence generation using HuggingFace generation API. Th topK and topP values are set 100 and 0.95. We set the temperature as 1.0. After sequence generation, we select top-K sequences. If shorter sequences are generated compared to the length of middle sites, we increase topK by 10 and conduct generation until sentences are collected. We use default option of HuggingFace API for other hyper-parametent These hyper-parameters and generation processes are applied on ProtFIM, ProtGPT2, and ProGen2 models for a fair comparison. Also, we use ProtXINet to generate sequences of target sites regressegrysivvlyy with conditioning bidirectional context using topK sampling as other AR models.
A.6 PERPLEXITY AND SEQUENCE RECOVERY
We also add other evaluation metrics, such as perplexity and sequence recovery rates, which are widely used for evaluating language models in inverse folding. Table]]shows the result of perplexity and sequence recovery rates. ProtFIM performs poorly in terms of perplexity and sequence recovery rates. In the FIM paper (Bavarian et al., 2022b), some experiments find that perplexity alone is insufficient for evaluating the infilling task because infilling is conducted in somewhat different nature compared to conventional left-to-light generation as expressed like Pr (M P,S) (AR(M P) where P. M, indicate prefix, middle, and suffix part, respectively.
14

---

Under review as a conference paper at ICLR 2023
PotFil W
Praden2 small ProGen2-medum
PoGen2iarge ProtXiNer
Proshin
Proden2 small
ProGer2 aage Protithe
Bandom
0.50
0.45
0.30
0.40
Precison@e
0.35
0.20
0.30
0.25
0.10
0.20
0.05
0.15
0.00
(0.0, 0.25]
(0:25, 0.5] (0.5, 0.75]
(0.75, 1.0]
(10, 20)
[20, 30]
(30. 40]
(40. 50)
Relative Position Bins
Length Bins
(a)
(b)
Figure 7: Performance changes in the metric of Precision @3 with regard to relative positions and length of middle sites.
Additionally, targeted infilling task aims design various sequences in the presence of local structure constraints considering the surrounding context, which is quite different from restoring residues much as possible. So, there needs be an appropriate evaluation scheme simulating protein middle engineering tasks that change amino acid residues of local parts of the protein to optimize the target protein, such as enzymes and antibodies. So, sequence residue recovery rate, widely used metric to evaluate models' sequence design performance, is insufficient for the protein infilling task. Based on the above results and descriptions, we argue that our proposed SEIFER tasks are more appropriate for evaluating protein infilling tasks than existing metrics such as perplexity and sequence recovery rates.
Table 4: Perplexity and sequence recovery rates
Model
IParams
Objective
Perplexity (1)
Recovery rale (%) (1)
ProGen2-small
151M
CLM
16.88
ProGen2-medium
764M
CLM
16.17
10
ProGen2-large
2.7B
CLM
16,24
ProtXI Net
409M
PLM
16.58
ProtGPT2-C
80M
17.08
ProtFIM
80M
FIM
17.04
A.7 PRECISION @K WITH REGARD TO POSITION AND LENGTH
Table 7a and 7b include ablation studies of SEIFER performance in term of precision according relative positions and length of target sites in protein.
A.8 FLIP
Table and 7 contain the zero-shot fitness prediction performances of various MLMs on three fitness landscapes.
15

---

Under review as a conference paper at ICLR 2023
Table 5: Zero-shot fitness prediction on adeno-associated virus (AAV) capsid proteins (Bryntt et al.
2021
All scores are Spearman correlation.
Model
#Params
Objoctive
Mut-Des
Des-Mut
1-V5-reat
2-v5-test
7-va-rest
low-vs-high
Avg.
ESM-Ib
750M
MLM
0.63
0.59
0.04
0.26
0.46
0.36
ESM-Iv (mean)
750M
MLM
0.55
0.44
0.18
0.16
0.45
0.20
0.33
ProGen2-small
151M
CLM
0.38
0.53
0.43
0.39
PtoGien--medium
764M
CLM
0.19
0:25
0.14
0.30
0.22
0,00
0.18
ProGen2-large
2.7B
CLM
0.68
0.67
0.33
0.42
0.41
ProiXLNet
400M
PLM
0.55
0.58
0.21
0.02
0.42
0,20
0.33
Pro1GPT2-C
80M
CLM
0.59
0.66
0.24
0.41
0.40
ProiFIM
8UM
FIM
0.53
0.56
0.32
0.24
0.44
0,28
0.39
Table 6: Zero-shot fitness prediction on adeno-associated virus GB1 landscape All scores are Spearman correlation.
(Wu et al.
2016
Model
#Params
Objective
1-vs-rest
2-vs-rest
3-ve-rest
low-vs-high
Avg.
ESM-1h (mean)
750M
ML.M
0.32
0.36
0.54
0.13
0.34
ESM-Iv (mean)
750M
MLM
0.32
0.32
0.77
0.10
0.38
2nmeel2-small
151M
CLM
-0.27
-0.30
-0.26
-0.03
-0.21
ProGen2-medium
764M
CLM
-0.06
-0.16
-0.12
-0.10
-0.11
ProGen2-large
2.7B
CLM
0.19
0.44
0.06
0.24
ProtXLNet
409M
PLM
0.18
0.33
0.44
0.21
0.29
ProrGPT2-C
80M
CLM
0.02
0.05
0.44
0,20
0.18
ProtFIM
80M
FIM
0.01
0.18
0.63
0.18
0.25
Table 7: Zero-shot fitness prediction landscape from the Meltome Atlas (Jarzab et al. scores are Spearman correlation.
2020 All
Model
#Params
Objective
Mixed
Human
Human-Cell
Avg.
ESM-1b (mean)
750M
MLM
0.68
0.70
0.75
0.71
ESM-ly (mean)
750M
MLM
0.67
0.75
0.74
0.72
ProGen2-small
151M
CLM
0.46
0.59
0.56
ProGen2-medium
CLM
0.59
ProGen2-large
2.7B
CLM
0.67
0.70
0.66
0.68
ProtXLNet
409M
PLM
0.44
0.52
0.47
0.47
ProtGPT2-C
80M
CLM
0.49
0.55
0.54
0.53
ProtFIM
80M
FIM
0.51
0.63
16


Reviews:
[{"review_id": "wtpjbmHy4Q", "reviewer": "Reviewer_tRtc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a new protein language model ProtFIM, and a new protein sequence design framework via ProtFIM. \nBy comparing the performance with previous models via their new evaluation scheme ( SEIFER), ProtFIM achieved similar performance but with less parameters.", "summary_of_the_review": "The idea of this paper is easy to understand, but not novel enough.  The performance of ProtFIM is similar to Progen-large, but ProtFIM has less parameters. More competing methods and some other metrics should be included. ", "strength_and_weaknesses": "Strengths: \n1.\tThe article is well written and easy to understand.\n\nWeaknesses: \n\n 1. There are too few metrics for experimental evaluation. Only P@k and R@k are included in evaluating infilling.  Some metrics provided in Progen2 can be used (e.g. complexity).\n\n 2. The method is not throughly evaluated.  The number of comparison methods is too small. There are still a lot of related work on generative PLMs.   The performance of the model is similar to Progen-large, and the main advantage is that the model size is smaller. Is it possible to add other models with different scales provided in Progen, especially small-scale models?  Then we can see whether ProtFIM still has an advantage over the models sharing similar scales.\n\n3. The evaluation requires the consistence of secondary structure between the generated protein and the original protein. However, a new protein with better performance may have a different secondary structure. Whether asking the generated protein is completely similar to the original one in secondary structure will affect the innovation of the model?\n\n4. It’s better to explain what (a) and (b) are in the caption of Figures 3 and 4. In addition, Figure 4 (a), (b) are both “Precision”. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pH6kRl8-Meq", "reviewer": "Reviewer_c4rP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a protein language model (pLM) that is trained on a fill-in-middle task that has seen application in the NLP domain. This is motivated by the fact that existing pLMs that are used for protein generation are autoregressive, and as such cannot condition on bidirectional context, which would actually be the most prominent case when re-engineering a region of a protein, as many residues relevant to function are found in the middle of the sequence rather than at its ends. The performance of the infilling model is evaluated by predicting the secondary structure of the filled in region and comparing it to the ground truth.", "summary_of_the_review": "The paper introduces a FIM-trained pLM and argues in favor of it's performance based on the fact that it has a far lower number of parameters than the ProGen model used for comparison. However, this leaves it unclear whether the model is truly more parameter efficient because of the FIM task - an AR pLM of equal size would be required to show that. Also, additional baseline methods should be considered to convincingly show that FIM boosts performance for the investigated tasks.", "strength_and_weaknesses": "Strengths\n- The FIM paradigm was not applied to pLMs so far.\n\nWeaknesses\n- No clear performance improvement over the ProGen baseline. \n- The comparison to ProGen-large does not prove that FIM is more efficient per parameter. You should compare to an autogressive pLM of equal size without the FIM task to show that.\n- If I understand correcty, also XLNet-style models should be capable of generating sequences with bidirectional context. Why is the ProtTrans XLNet model not considered?\n- ESM-1b is not SOTA for zero-shot fitness prediction. Also, this evaluation again fails to prove that the parameter efficiency can be attributed to the FIM task.\n\nQuestions\n- The definition of the @k metrics is hard to follow. Could it be expressed in a more straightforward way that directly explains what TP, FP, TN, FN are in the context?\n- What is the reason for choosing a constructed secondary structure metric over perplexity? Perplexity can directly measure how well the model captures the masked span.\n- Does the alpha helix bias mean that the FIM residue SS does not really depend much on the surrounding context?\n\nAdditional comments\nIn the introductory session, describing directed evolution as \"random guessing or brute-force search\" doesn't capture its essence, as it is actually meant to avoid doing just that.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jbB2WgxJh7", "reviewer": "Reviewer_ps6t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a new type of protein language model capable of filling in the middle of a sequence (which wasn't easily doable with autoregressive or with masked language models). ", "summary_of_the_review": "Important direction for supporting protein infilling with language models, but limited technical or empirical contributions.", "strength_and_weaknesses": "Strengths:\n+ The \"fill in the middle\" use case is very common in protein engineering, and such a model could indeed be very useful.\n\nWeaknesses:\n+ The training setup choices seem somewhat limited: For example, it only allows one masked region in the middle. Often, in protein engineering, there might be multiple segments for joint design. There is a much more general version of this (e.g. see CM3 https://arxiv.org/pdf/2201.07520.pdf).\n+ The empirical results on fitness prediction seem worse than existing methods.\n+ I'm unconvinced that secondary structure conservation is the best evaluation scheme. What about sequence recovery or perplexity on the \"middle\" sequence?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "jJrjJzMv0J", "reviewer": "Reviewer_2Lrn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper's aim is sequence-based protein engineering based on protein language models (PLMs). For this purpose, it uses a recently developed self-supervised in-filling language model. The model rearranges the middle (to be infilled) part of a sequence and move it to the end of the sequence which enables the use of standard forward prediction. The paper then tests these PLM models for identifying other possible residue sequences in the middle of a protein sequence that preserves the general 3D structure of the protein (secondary structural type).  ", "summary_of_the_review": "The paper uses a novel method for protein language models that is quite efficient in number of parameters and shows the results are comparable for protein sequence engineering compared to one other recent PLM baseline. However, considering that the results are clearly positive compared to the possible baselines, the lack of technical novelty (even in form of an increment on existing methods), the lack of a thorough set of baselines, and presentation of the work, I believe the paper is not ready for publication, at least at a ML venue. More developments on the method and/or a more thorough empirical evidence will increase the quality of the paper in another revision.", "strength_and_weaknesses": "- the paper provides a novel method for PLM especially suitable for sequence-based protein optimization.\n- despite being a language model, the proposed architecture is relatively small in number of parameters which enables efficient use at test time and enables research for protein-engineering with limited computational resources.\n- the additional experiments regarding the quality and the general analysis of the learnt representations are interesting, encouraging and informative.\n\n*Weaknesses*:\n- For an ML conference there seems to be no technical novelty, neither fundamental, nor incremental. The model is almost an exact copy of the standard LM with the addition of the structure constraint.\n- The number of available protein (and non protein) language models are vast. The paper only compares with one method while it could (and should have) compared with other PLMs but also possibly other language models with the same trick (moving the missing part to the end).\n- the main results that is the goal of the work for protein engineering seem quite comparable with the only baseline that is used (ProGen).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject