Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design
Authors: Yue Cao
Payel Das
Pin-Yu Chen
Vijil Chenthamarakshan
Igor Melnyk
Yang Shen
Venue:  (2021)
Keywords: Joint Embedding Learning
Generative Model
Transformer Autoencoder
Inverse Protein Folding
Sequence Design

Abstract: Designing novel protein sequences consistent with a desired 3D structure  or fold, often referred to as the inverse protein folding problem, is a central, but non-trivial, task in protein engineering.  It has a wide range of applications  in energy, biomedicine, and materials science. However, challenges exist due to the complex sequence-fold relationship and difficulties associated with  modeling 3D folds. To overcome these challenges, we propose Fold2Seq, a novel transformer-based generative framework for designing protein sequences conditioned on a specific fold. Our model learns a fold  embedding from the density of the secondary structural elements in 3D voxels, and then models  the complex sequence-structure relationship by learning a joint sequence-fold embedding. Experiments on  high-resolution, complete, and single-structure test set demonstrate improved  performance of Fold2Seq in terms of speed and reliability for sequence design, compared to existing baselines including the state-of-the-art RosettaDesign and other neural net-based approaches. The unique advantages of fold-based Fold2Seq becomes more evident on diverse real-world test sets comprised of low-resolution, incomplete, or ensemble structures,  in comparison to a structure-based model. 

Under review as a conference paper at ICLR 2021
FOLD2SEQ: A JOINT (EQUENCE(ID)-FOLD(3] EMBEDDING-BASED GENERATIVE MODEL FOR PRO- TEIN DESIGN
Anonymous authors Paper under double-blind review
ABSTRACT
Designing novel protein sequences consistent with desired 3D structure or fold, often referred to as the inverse protein folding problem, a central, but non- trivial task in protein engineering. It has wide range of applications in en- ergy, biomedicine, and materials science. However, challenges exist due to the complex sequence-fole relationship and difficulties associated with modeling 3D folds. To overcome these challenges, we propose Fold2Seq. a novel transformer- based generative framework for designing protein sequences conditioned cific fold. Our model learns a fold embedding from the density of the secondary structural elements in 3D voxels, and then models the complex sequence-structure relationship by learning joint sequence-fold embedding. Experiments on high- resolution, complete, and lingle-strccture test set demonstrate improved perfor- mance of Fold2Seq in terms of speed and reliability for sequence design, com- pared to existing baselines including the state-of-the-an RosettaDesign and other neural net-based approaches. The unique advantages of fold-based Fold2Seq be- comes more evident on diverse world test sets comprised of resouuuoo,,nn incomplete, ensemble structures, in comparison to structure- based model.
spe-
INTRODUCTION
Computational protein design is the conceptual inverse of the protein structure prediction problem, and aims to infer an amino acid sequence that will fold into a given 3D structure. Designing protein sequences that will fold into desired structure has broad range of applications, from therapeu- ties to materials (Kremmer-Pecore et al. 2001). Despite significant advancements in methodologies as well as in computing power, inverse protein design still remains challenging, primarily due to the vast size of the sequence space and the difficulty of learning function that maps from the 3D structure space to the sequence space. Earlier works rely mostly on energy minimization-bant approaches (Koga et al., 2012, Rocklin al., 2017] Huang et al.) [201]). which follow scoring function (force fields, statistical potentials, or machine learning (ML) models,) and sample both sequence and conformational space. Such methods often suffer from drawbacks such low aceu- racy of energy functions or force-fields (Khan & Vihinen 2010) and low efficiency in sequence and conformational search (Koga et al. 2012).
Recently, as the data on both protein sequences (hundreds of millions) and structures (a few hundreds of thousands) is quickly accumulating, data-driven approaches for inverse protein design rapidly emerging (Greener et al. 2018: Karimi et al., 2020, Ingraham et al.) 2019). Generally, data-driven protein design. attempts model the probability distribution over sequences conditioned on the structures: (())g), where x and y are protein sequences and structures, respectively. Two key challenges remain: (1) defining good representation (3) of the protein structure and (2) modelling the sequence generation process conditioned
y. Current protein design methods use protein backbone information from a single protein structure (fixed backbone) or from a set of backbone structures consistent with a single fold (flexible backbone). In earlier studies, the protein structures are represented as a 1D string (Greener et al.] 2018), a 1D vector (Karimi et al. 2020). a 2D image graph (Ingraham et al. 2019). The sequence generation methods used in the protein design studies can be classified as con-autoregressive (Karimi et al. 2020 Greener
Strokach et al. 2020),
et al. 2018 Strokach et al. 2020) and autoregeressive (Ingraham al. 2019 Madani
2020)

---

Under review as a conference paper at ICLR 2021
In non-autoregerssive methods, is usually concatenated with Gaussian random noise x (which is the latent vector of the sequence) please check here to be the input to a sequence generator P(x x y) = fy, while in autoregressive methods, P( yy is decomposed through the chain rule: P( y) m (1, 1; y). where a = (211 T2,,..,nn.) T.))
In this paper, focus designing sequences conditioned a protein fold. A protein fold is defined as the arrangement (or topology) of the secondary structure elements the protein relative to each other (Hou al. 2003). secondary structural element can be defined as the three dimensional form of local segments of a protein sequence. Protein folds are therefore necessarily based on sets of structural elements that distinguish domains. As protein structure inherently hierarchical, the complete native structure can have multiple folds and a fold can be present in many protein structures. A single structure (fixed backbone) or an ensemble of structures (flexible backbone) can be used as representatives of fold. The ensemble representation is often a better choice, as it captures the protein dynamies.
Despite the recent progress in using ML models for protein design, significant gaps still remain in addressing both aforementioned challenges (fold representation and conditional sequence genera- tion). First, the current fold representation methods are either hand-designed, or constrained and do not capture the complete original fold space (See Sec. 2.2 for details), resulting in low generalization capacity efficiency. Second, the sequence encoding and the fold encoding are learned separately in previous methods, which makes two latent domains heterogeneous. Such heterogeneity across two domains actually increases the difficulty of learning the complex sequence- fold relationship.
To fill the aforementioned gaps, the main contributions of this work are as follows:
We propose a novel fold representation. through first representing the 3D structure by the voxels of the density of secondary structures elements (SSEs), and then learning the fold representation through a transformer-base representation has several advantages: first, it preserves the entire spatial information of SSEs. Second, does not need any pre-defined rules, so that the paramterized fold space is not nei- ther limited or biased toward any particular fold. Third, the representation extracted from a given protein structure. Lastly, the density model also loosens the rigidity of structures so that the structural variation and lack of structural information of the protein is better handled.
structure encoder. Compared to previous fold representations, this
be automatically
We employ a novel joint sequence-fold embedding learning framework into the transformer-be auto-encoder model. By learning a joint latent space between sequences and folds, our model, Fold2Seq. mitigates the heterogeneity between two different domains and is able to better capture the sequence-fold relationship, as reflected in the results.
Experiments on standard test sets demonstrate that Fold2Seq has superior performance on per- plexity, native sequence recovery rate, and native structure recovery accuracy, when compared to competing methods including the state-of-the-aa RosettaDesign and other neural net models. Ablation study shows that this superior performance is directly attributed to our algorithmic inno- vations. Experiments on real-world test sets further demonstrates the unique practical utility and versatility of Fold2Seq compared to the structure-based baselines.
RELATED WORK
Data-driven Protein Design A significant surge of protein design studies that deeply exploit the data through modern artificial intelligence algorithms has been witnessed in the last two years. Greener et al. (2018) used the conditional variational autoencoder for generating protein sequences conditioned on given fold. Karimi et al. (2020) developed guided conditional Wasserstein Gen- erative Adversarial Networks (gcWGAN) also for inverse fold design. |Mdanni et al. (2020) trained an extreme large (1.2B parameters) language model conditioned on taxonomic and keyword tags such as molecular functions for generating protein sequences. Ingraham et al. (2019) developed graph-based transformer for generating protein sequences conditioned on a fixed backbone. Lastly, Strokach et al. (2020) formulated the inverse protein design constraint satisfaction problem (CSP) and applied the graph neural networks for generating protein sequences conditioned on the residue-residue distance map that is static representation of the structure.
2

---

Under review as a conference paper at ICLR 2021
Protein Fold Representation For extensive overview of molecular representations, including those of proteins, please see David et al. (2020). Murzin et al. (1995) and [Orenoo et al. (1997)
manually classified protein structures in hierarchical manner based on their structural similarity. These classifications can be regarded as one-hot encoding of the fold representations. Taylor (2002) represents protein fold using "periodic table". This representation was later used for inverse fold design (Greener et al., 2018). However, it only considers three pre-defined folds (( layer, layer and barrel) for a fold. Hou et al. (2003) chose hundreds of representative proteins and calculated the similarity scores among them. This similarity matrix was then used for kernel Principle Component Analysis (kPCA). A similar idea was used in Karimi et al. (2020) for inverse protein design. This representation needs a pre-defined set ((alpha only, beta only, and alpha+beta) of structures along with a similarity metric. Such representation could lead to biased or constrained representations of the fold space and also may not preserve the detailed spatial information of the fold. Finally, Koga et al. (2012) summarized three rules that describe the junctions between adjacent secondary structure elements for a specific fold. Again, these rules are hand designed for a subset of structures, which makes the representation restricted to small part of the fold space and offers limited generalizability during conditional sequence generation.
structures, which significantly limits the spatial information content of the
Joint Embedding Learning Joint embedding learning across multiple different data modalities
was first proposed by Nggia et al. (2011) on audio and video signals. Since then, such approaches
have been then widely used in cross modal retrieval or captioning tasks (Arandjelovic & Zisserman, 2018 Gu et al. 2018 Peng & Qi 2019; Chen et al. 2018 Wang et al. Dognin et al. [2019). In few/zero-shot leanning joint feature- embedding was used for predicting the label of instances belonging unseen classes (Zhang & Saligrama, 2016 Socher et al. 2013). Several papers have demonstrated that learning joint embedding is useful for the single modal classification tasks (Ngiam) jet al., 2011; Wang et al. 2018 'Toutanova al., 2015). Moreover, Chen al. (2018) used joint embedding learning for text to shape generation. Lastly. Joint sequence- emhedding is also
applied for molecular prediction/generation (Cao Shen 2020
Das et al. 2018).
METHODS
3.1 BACKGROUND AND NOTATION
A protein is formed by a linear chain of amino acids (residues) that defines its 1D sequence. Chemi- cal nature, as well as physical and chemical interactions with neighboring residues drive the folding of sequence into different secondary structure elements or SSEs (helix, beta-sheet, loop, etc., see Fig[Ta)). that eventually forms a complete native 3D structure. A protein fold captures the topology and the composition of secondary structure elements, thus serving an intermediate between the 1D sequence and the full 3D structure.
3.2 STRUCTURE REPRESENTATION THROUGH 3D VOXELS OF THE DENSITY OF SSES
(a)
(c)
Rescale
esturization
Figure 1: (a) The structure of PDB 107L. The secondary structures are colored as: helices in red, beta sheets in yellow and loops in green. (b) The structure is rescaled to fit the 40Å x 40 x 40A cubic box. (c) The box discretized into voxels. (d) Features of each voxel are obtained from the structure content of the voxel (bend/turn in blue).
In this subsection, we describe how we represent the 3D structure to explicitly capture the fold information. We denote the Pos coordinate) of each residue by its alpha carbon. For given protein with length N. we first translate the structure to match its center of mass with the origin of the coordinate system. We then rotate the protein around the origin let the first residue be on the
3

---

Under review as a conference paper at ICLR 2021
negative side of z-axis. We denote the resulting residue coordinates C1. C2...., CN. We assign the secondary structure label to each residue based on their SSE assignment (Kabsch & Sander 1983 in Protein Data Bank (Berman al. 2000). We consider 4 types of secondary structure labels: helix, beta strand, loop and bend/turn. In order to consider the distribution of different secondary structure discretize the 3D space into voxels, shown in Fig A technical
labels in the 3D space, challenge here is that the sizes of different proteins vary drastically. As we are only considering the arrangement of SSEs, not their exact coordinates, we here rescale the original structure, so that it fits into a fixed-size cubic box. Based on the distribution of sizes of single-chain proteins in the CATH database (Sillitoe et al. 2019), we choose 40A x 40A x 40A box with each voxel of size 2A x 2Á x 2A. We denote the scaling ratio as r E R For voxel i, we denote the coordinates of its center as U.. We assume that the contribution of residues j to voxel follows the Gaussian form:
| o - v.
Yij = exp(
where ł, E {0, 134 is the one-hot encoding of the secondary structure label of amino acid j. The standard deviation is chosen to be 2Å. We sum up all residues together to obtain the final features of the voxel i: y: Σÿ= Vij- The structure representation is the vector of y:'s over all voxels. In the next subsection, we will describe how we encode into the latent space and how we learn a joint sequence-for fold embedding in order for generating sequences consistent with a desired 3D structure.
3.3 FOLD2SBQ WITH JOINT SEQUENCE-FOLD EMBEDDING
Training Scheme
Inference Scheme
Cyclic Loss
Sequerce
Soguzna: Emmtediing
Trensfiemmr Encoder
(x)))
Fold Classsiication
Sequence Encoder
Cosnze Similarly Loss
Bleck
Com
Pux Encoder
Traraferaa Encoder
x
Residual Lonr Block
3D Pre. Einooder
Tranffence Innoder
brly)
Fold Ulossiication
Structure Encoder
x
Structure Encoder
Sequence Embeitlin
Traisfomer Decodlrr
Ganerated
Sequence
Sequance
Sequence Enbedding
Decoder
ple k())
Outpur
Autoregreesive
Sequence Decoder
Sequence Decoder
The architecture of the model during the training and inference stages. (Training Scheme): During training. the model includes three major components: (top) Sequence Encoder, (middle) Structure Encoder and (bottom) Sequence Decoder. The dashed arrows represent the pro- cess for getting cyclic loss. (Inference Scheme): During the inference, the model only needs the structure encoder and the sequence decoder for decoding sequences. check here and the figure
Figure 2:
Model Architecture In the training stage, our model consists of three major components: a se- quence encoder: h-(-), a structure encoder: h-)-) and sequence decoder: p(æ (k(-),, as shown in Fig. 2 (Left). Both sequence encoder and decoder are implemented using the vanilla transformer model Vaswani al. 2017). The structure encoder contains 6 residual blocks followed by 3D positional encoding. Each residual block has two 3D-convolutional layers (3 x 3 x 3) and batch normalization layers. The 3D positional encoding
simple 3D extension of the sinusoidal en- coding described in the vanilla transformer model, shown in Appendix A After the positional encoding, the 3D vector is flattened to be ID as the input of a transformer encoder. The length of the transformer input is fixed to be ly 53 125. The output of the transformer encoder: h (3) is the latent fold representation of y.
We propose a simple fold-to-sen to-senuence reconstruction loss based the auto-encoder model: RE, (x))))())) However, as mentioned earlier, training based on RE, alone suffers due to the het- erogeneity of x and y. To overcome this challenge, we first encode the sequence through the
4

---

Under review as a conference paper at ICLR 2021
sequence encoder into the latent space ha(2)), which could be done through simple sequence- to io-sequence reconstruction loss: RE, = p(æ) ha (z)). We then learn joint latent space between (y()) and (xx) through a novel sequence-fold fold embedding learning framework.
Joint Embedding Learning Typically, learning a joint embedding across two domains needs two intra-domain losses and one cross-domain loss (Chen et al.) 2018). An intra-domain loss forces two semantically similar samples from the same domain to be close to each other in the latent space, while a cross-domain loss forces two semantically similar samples in different domains to be closer. In our case, the meaning of 'semantically similar' is that the proteins should have the same fold(s). Therefore, we consider a supervised learning task for learning intra-domain similarity: fold classi- fication. Specifically, the outputs of both encoders: hy
R and h,(x) R! will be averaged along (, and l, dimensions, followed by a MLP+softmax layer to perform fold classifica- tion (shown as two blue blocks in Fig. 2). where 4. L and d are the length of the sequence, the fold and the latent state, respectively. The two MLP layers' parameters are shared. The category labels follow the fold (topology) level of hierarchical protein structure classification in CATH4.2 dataset (Sillitoe et al. 2019) (see Section 3.4). As a result, we propose the following two intra-domain losses: FC, and FC,, the cross entropy losses of fold classification from hy( and (ar) respec- tively. The benefits of these two classification tasks are obvious: First, it will force the structure encoder to learn the fold representation. Second, as we perform the same supervised learning task on the latent vectors from two domains, it will not only learn the intra- similarity, but also cross-domain similarity. However, without explicit cross-domain learning, the two latent vectors (V), hg (x) could still have minimal alignment between them. We will describe the cross-domain loss the rest of this subsection.
In the transformer decoder, each element in the non-self attention matrix is calculated by the cosine similarity between the latent vectors from the encoder and the decoder, respectively. Inspired by this observation, we consider to maximize the cosine similarity (shown as the 'Cosine Similarity" pink block in Fig 2) between hyy E R and h2 (x) E Rl. xd as the cross-domain loss. We first calculate the matrix-product between hy( and (x(2)) as Q = hy(s) - (x))² Q E R4y xi, The ith row in Q represents the similarity between ith position in the fold and every position of the sequence. We would like to find the best- matching sequence piece with each position in the fold. To achieve this, the similarity matrix Q first goes through row-wise average pooling with kernel size k, followed by the row-wise max operation:
max (AvgPool (Q)). E R
(2)
yow
where row means the operation is row-wised. We choose k = 3, which means the scores of every 3 continuous positions in the sequence will be averaged. We finally average over all positions in the fold to get the final similarity score: CS = mean(q). please read this subsection until here. I did some equation-in-lin here.
Besides the cosine similarity loss, inspired from the earlier CycleGAN work (Zhu et al., 2017) we add a cyclic loss (shown as the *Cyelic Loss' red block in Fig 2.) to be another term of our cross-domain loss. Specifically, we take the argmax of the output of fold-to-sequence model: x' = arg max p(æ) (y)), and send back to the sequence encoder for generating the cyclic-seq latent state: h, (xx' (shown as the dashed line in Fig 2). This cyclic-seq latent state will compare with the native seq latent state (x) through the square the L2 distance:
CY = =11h,(x') ) - (z))2
(3)
To summarize, the complete loss objective is the following:
L L RE, + 22 + FC + AgFC. + -C)
CS)
(4)
where \ through A5 are the hyperparameters for controlling the importance of these losses.
Training and Decoding Strategy During experiments we found that, if the sequence encoder and the structure encoder were trained together, the structure encoder had little parameter improvement while the sequence encoder dominated the training. To overcome this issue, we consider two- stage training strategy. In the first stage, we train the the sequence intra-domain loss: L RR, + FaFC.. After the first stage is finished, we start the
model regularized by
5

---

Under review as a conference paper at ICLR 2021
second training stage. We train the -eeuuuee model regularized by the fold intra-domain loss and the cross-domain loss while keeping the sequence encoder frozen: L2 = X1 + >3FC, + As [CY- CS). The comparison between the one-stage training and two-stage training strategies are described in details in Appendix E
We implement model in Pytorch (Paszke al. 2019 Each transformer block has 4 layers and 256 latent dimensions. In order to increase the robustness of our model for rotated structures, augment our training data by right-hand rotating the each structure by 90" 180 and 270 along each axis (x.y.z). As a result, we augment our training data by 3 x 3 9 times. The learning rate schedule follows the original transformer paper (Vaswani et al. 2017 We use the exponential decay (Blundell et al., 2015) for À5 = 1/2 pe
in the loss function. while A through λ and e are tuned based on the validation set, resulting À = 1.0, X2 1.0, A 0.02, 1.0, e 3 We train our model on 2 Tesla K80 GPUs, with batch size 128. In every training stage we train up to 200 epochs with an early stopping strategy based on the loss on the validation set.
During inference, we only need the Fold2seq model for decoding sequences (Fig 2 (Right)). Top-k sampling strategy (Fan et al. 2018) is used for sequence generation, where set to be (tuned based on the validation set).
3.4 BENCHMARK DATASETS
We used the protein structure data from CATH 4.2 (Sillitoe et al. 2019) filtered by 100% sequence identity. We remove proteins that (1) are multi-chain or non-continuous in sequence; (2) contain other than 20 natural amino acids; (3) have length longer than 200. We randomly split the dataset based on the fold-level classification of protein structures into 95%, 2.5%, 2.5% as dataset (a), (b) and (c), which means that the three datasets have non-overlapping folds. We further randomly split the dataset (a) into 95%, 2.5% and 2.5% as dataset (al), (a2) and (a3). Datasets (al) (a2) (a3) have overlapping folds. We use dataset (al) as the training set, (b)+(a2) as the validation set, (a3) as the In-distribution (ID) test set and (c) as the Dut-of-disttibution (OD) test set. The folds of the test set overlaps with the training set, whereas the folds of OD) test set do not overlap with the training set. The statistics of these datasets are presented in Appendix B
In order to quantitatively measure their difficulty levels, we calculate the averaged maximum quence similarity (amsi) between a given test set T and the training set, defined as: amsir = DDT ExED, ma (SIM(æ,,. æ2,),, where Duain and Dr are the training set and test set T, re- spectively; SIM(2;, œ, B is the sequence similarity between sequence D; and æj. As a result, we found we have amsin 36.3% and amsion 16.3%. This shows that the sequence similarity between the ID test set and the training set is more than twice that between the OD test set and the training set. These values clearly demonstrate that the OD test set represents a much more difficult generalization task compared to the ID) test set.
4 EXPERIMENTS ON BENCHMARK TEST SETS.
Ideally, the most appropriate criteria for evaluating inverse design methods is the extent that the structure of the generated sequence matches the desired fold/structure. However, as protein structure prediction is very time-consuming, and similar sequences usually indicate similar folds/structures, many carlier methods report performance in the sequence domain. We hereby perform comprehen- sive evaluations of Fold2Seq against two data-driven fold design methods: cVAE (Greener et al. 2018) and ge gGGA (Karimi et al. 2020) as well as the state-of-the-arr principle-drieen method, RosettaDesign [3 (Huang et al. 2011 For this comparison, on inverse folding problem with a flexible backbone constraint (e.g. (Madani et al., 22020), (Strokach) al. 2020)). Three evaluation metries are used: 1) Perplexity: Ability assign high likelihood to
leave out methods that do not focus
1 We plan to relcase the code upon acceptance. "Sequence similarity is measured through the Needleman Wunsch algorithm Needleman & Wunsch 1970 with Blossum66 scoring matrix.
"RosettaDesign is principle-based method. It uses MCMC sampling and energy calculation to search for best sequences. The input to RosettaDesign consists of the hackbone of the native structure and the SSE of each residue.

---

Under review as a conference paper at ICLR 2021
Table
Performance of different methods assessed in the sequence domain.
(b) Avg. SeRR = std. dev. (%)
(a) Per-residue Perplexity.
Model
ID Test
OD Test
Model
ID Test
OD Test
Random across two folds
12.8 + 7.94
12.8 = 7.94
Uniform
20.0
20.0
cVAE
17.7 + 7.34
15.3 = 5.34
Natural
18.0
18.0
geWGAN
17.5 + 6.35
14.1 3.45
cVAE
14.8
16.3
RosettaDesign
20.3 5.13
20.2 = 2.98
geWGAN
13.5
15.2
Fold2Seq
27.1 + 6.31
24.1 2.64
Fold2Seq
8.1
11.9
Random within same fold
39.1 + 9.35
39.1 9.35
native (ground-truth) sequences; 2) Native sequence recovery rate (SeRR): Ability to recover native sequences; 3) Native structural recovery accuracy (StRA): Ability to recover native structures.
We note that the Graph trans (Ingraham et al. 2019 model can be applied to flexible backbone scenario. However, Graph _eaph_taassii focuses on structure- based sequence design using the detailed back- bone structure as input, whereas the fold-based Fold2Seq designs sequences using the high-level fold information. Therefore, direct comparison between Fold2Seq and Graph trans on the same train/test data containing full structure information is not fair. To highlight that Fold2Seq better cap- tures fold information, we first re- re-train Graph_trams on our training set. We use tSNE to visualize the fold embeddings h after the fold encoder for the proteins in the OD test sets. The results are shown in Appendix F is evident that the embeddings of same-fold proteins from Graph trans are less clustered than those from Fold2Seq. Later, we will compare Fold2Seq with Graph_trans three real-world design tasks, which will be described in details in Sec. 5.
Perplexity and Sequence Recovery Comparison. We first compare Fold2Seq against geWGAN and eVAE in terms of per-residue Perplexity. Perplexity is metric widely used in the language mod- cling and is not applicable to RosettaDesign. For reference, we also show the per-residue perplexity under the uniform distribution and the frequencies through all natural sequences in UniRef50 (Suzek itt al. 2015). Performance on two test sets is summarized in Table 3a showing that Fold2Seq has the smallest per-residue perplexity for both ID and OD test sets. We also note that the performance of all methods on the OD test set is worse than that on the ID test set, which matches our expectation of their difficulty levels.
Next, we compare the ability of different methods for recovering the native sequences given a desired fold. We calculate the averaged sequence similarity between native sequence and the generated sequence on two test sets. Also, for comparison, two random sequences in our whole dataset belonging to two different folds and belonging to the same folds. The results are summarized in Table 1b In general, Fold2Seq performs significantly better, when compared to existing methods. Specifically, compared to RosettaDesign (second-best) Fold2Seq improves the recovery rate by 33.5% and by 19.3% the ID and OD test set, respectively.
calculate the expected similarity between
Structural Recovery Comparison with RosettaDesign. Although the comparison in sequence domain reflects the reliability of generated sequences to some extent, our final goal is to ensure that the structure of generated sequence is similar to the native structure. Specifically, iTasser Suite (Yang et al. 2015), one of the state-of-the-art protein structure prediction software, predict the structure of the designed sequences. As iTTssser usually takes least one day for predicting the structure of a single protein, we only compare Fold2Seq to RosettaDesign in terms of the structure recovery. The similarity between two 3D protein structures is measured by the TM-score E [0, 1] (Zhang & Skolnick 2004), where a higher score implies greater similarity.
used the
For the ith protein in the test set, compare TM(s, Fold2Seq s;
Naativ)) against TM(s Tos . .NNative),
Rosetta
where s; Fold2Seq s Rosetta and Native are the 3D structures of the generated sequence from Fold2Seq.
the generated sequence from RosettaDesign, and the native sequence, respectively.
We first compare the distributions of TM(s, MMsssall2See s Native) and (M( Nativvv on two test sets (ID and OD). The results are shown in Fig 3(). For the ID test set, Fold2Seq shows signif- icant improvement against RosettaDesign, while for the OD test set, Fold2Seq shows slightly better
Fold2Seq
performance. We then define ATM; TM(s Fold2Seq
s s.Naaive) TM( (s Rosetta s.aatve) and perform
7

---

Under review as a conference paper at ICLR 2021
(8)
Icl5
RosettaDesign
10a
ResittaDeiign
Seq Recovery Rator
Fois25es Graph
Fold2Seq
10a
30
Scare
Running Time
10
109
Folo25eq GPU
0.4
20"4
15
10
0.2
200
15 Residue Fraction!%)
Protein Longth (AA)
Figure 3: (a). TM score distributions of RosettaDesign and Fold2Seq. (b). Run time of Fold2Seq and RosettaDesign for generating one protein sequence: CPU: Intel Xeon E5-2680 v4 2.40GHz, GPU: Nvidia Tesla K80. (c). Avg. SeRR for the OD test set with string missing residues.
one-sided one-sampled t-test over ATM, with null hypothesis as "ATM & 0.0" on two test sets, re- vealing P-valuem 2. 24E 60, P-valueon 0.019, which demonstrate that Fold2Seq can overall generate more reliable structures compared to RosettaDesign. The two distributions over TMM are shown in Fig 4 in Appendix [ We also randomly pick some designed structures with ATM > 0.0 and ATM 0.0, and visualize them on Fig 5 and Fig 6 in Appendix respectively.
The sequence generation efficiency of Fold2Seq against RosettaDesign is shown Fig 3()). Fold2Seq is almost 100 times faster than RosettaDesign when running CPU, and is 5000 times faster when running on GPU, while RosettaDesign can only run on CPU.
Ablation Study. In order to rigorously delineate the contributions of each algorithmic innovations, we perform an ablation study as following:
VAE: We use eVAE (Greener et al. 2018) as baseline with 1D string fold representation and MLP-based VAE,
Trans Traiigstring__ REy: We replace the MLP-based VAE in cVAE with transformer autoencoder model.
The loss is L RE
Trans voxel RE We replace the 1D string fold representation in "Trans string RE," with 3D voxel representation. We also add the convolutional residual block and 3D positional encoding.
The loss is L = RE,,
+RE,+CS: We add the sequence encoder, together with the reconstruction loss and the cosine similarity loss the previous loss: L RE + RgRE, AsCS.
+2FC: We add the two FC losses. I, A RE, + RE, + AgFC, + X,FC, - AsCS.
+CY (Fold2Seq): We add the cyclic loss into the former model with the final loss = X RE, + >₂RE, A3FCC F. As(CY CS).
We use average of native sequence recovery rate (SeRR) the assessment metric. The performance on two test sets is summarized in Table 2a Our key observations are: (i) By changing "string" to 'voxel' and adding 2 FC losses, the performance has the largest improvement by 3-4%. (ii) By adding the cyclic loss (CY), the performance also improves around 2%. (iii) contrast, by adding RE, and CS, the improvement is trivial. It also shows that including the FC losses the intra- domain loss is crucial for the joint embedding learning. voxel representation and the joint embedding learning framework, which includes intra-domain and cyclic losses, results significant improvement of the performance.
conclusion, our novel design of the 3D
5 EXPERIMENTS ON REAL-WORDE DESIGN TASKS.
In order further explore the practical utility of our model, we perform three real-world challenging design tasks: (1) Low-resolution structures; (2) Structure with region of missing residues; and (3) NMR ensembles. For (1) and (2), we compare with Graph trans with a flexible backbone constraint which is re-trained on training set.
now-Resolution Structures. We first create the low-resolution structure dataset from Protein Data Bank, which contains 164 low resolution proteins (single-chain), resolution varying from 6Å to
8

---

Under review as a conference paper at ICLR 2021
Table 2: Average sequence recovery rate =std. dev. (%) for (a) different models in ablation study,
(b) low resolution structures and NMR ensemble.
(a)
(b)
Model
ID Test
OD Test
Model
Low res Set
cVAR
17.7 + 7.34
15.3 + 5.34
Graph trans
19.9 4.8
Trans_string_RRl
,
18.5 + 8.26
16.1 + 3.23
Desenn
17.2 + 6.3
Traes_ ,
+.23
19.2 + 3.46
Fold2Seq
21.2 3.1
+RE,+CS
7.54
19.1 2.24
Input
ID
OD
+2FC
25.3
22.2 +
Single
22.7 3.4
20.9 + 4.2
+CY (Fold2Seq)
27.1 6.31
24.1 + 2.64
Ensemble
24.1 + 4.6
22.3 3.1
12A. This set has maximum sequence similarity (MSI) below 30% to the training set. We compare Fold2Seq's 's performance on this set with that of Sraph_trans and RosettaDesign, As shown in Table 2b Fold2seq outperforms other baselines, because Fold2Seq only learns from the fold information by re-scaling the structure, discretizing the space, and smoothing the spatial secondary structure element information by neighborhood averaging. As Fold2Seq uses the high-level fold information, the method's ' performance is expected to be less sensitive compared to RosettaDesign or Graph trans, when test structures are of lower resolution.
Structures with missing string of residues. We then perform the design task where the in- put structures have missing residues. In order to mimic the real-world scenario, for every pro- tein in our OD test set, we select stretch of residues at random starting positions with length for which residue information was removed. We compared Fold2Seq with Graph trans at p = (5%, 10%, 15%, 20%, 25%, 30%]. As shown in Fig. 3e, initially, when p is small, the perfor- mance of Fold2Seq is on par with Graph_trans. As p increases, Fold2Seq outperforms Graph_ _trans with a consistent margin. This pattern matches our expectation, as Fold2seq is less sensitive to the availability of complete and detailed backbone structure information.
NMR Structural Ensemble. We finally apply Fold2Seq to structural ensemble NMR struc- tures. We filter the NMR structures from our two test sets and obtaín 57 proteins from the ID set and 10 proteins from the OD set. On average each protein has around 20 structures. Handling NMR ensembles using Fold2Seq is straightforward, when compared to Graph.tran trans and RosettaDesign: after we obtain the voxel-based features through Eq [ for each model (structure) within one NMR ensemble, we simply average them across all models. Fold2Seq results for NMR ensembles are shown in Table 2bb along with a single structure baseline. Results show that Fold2Seg performs better on both ID and OD proteins, when ensemble structure information is available. This is con- sistent with the fold-based learning framework of Fold2Seq as fold representation better captures structural variations present within single fold.
CONCLUSION
In this paper, we design novel neural network model to learn a fold representation from the 3D vox- els of density of secondary structure elements. In order mitigate the heterogeneity between the sequence domain and the fold domain, we learn the joint sequence-fold representation through novel intra-domain and cross-domain losses. Our model, Fold2Seq outperforms existing data-driven methods and the state-of-the-art principle-driven method RosettaDesign, in terms of perplexity, se- quence recovery rate, and structural recovery accuracy. We also show that our method is signifi- cantly faster compared to RosettaDesign. Ablation study shows that this superior performance can be directly attributed to our novel algorithmic innovations, including the fold representation, joint sequence-fold embedding, and various losses. Moreover, we demonstrate the unique practical utility of Fold2Seq compared to structure- based models in a set of real-world design tasks, including low resolution structures, structures with region of missing residues, and NMR structural ensembles.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 435-451, 2018.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Πya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1):
235-242, 2000.
Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv: 055444 2015.
Yue Cao and Yang Shen. Tale: Transformer-based protein function annotation with joint se- quence-label embedding. bioRxiv, 2020. doi: 10.1101/200000922.11533333
URL https:
//www. http:///ww......eeaaaaa/////////
09 09.0999.2.... 315937
Kevin Chen, Christopher Choy, Manolis Savva, Angel Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Asian Conference on Computer Vision, pp. 100-116. Springer, 2018.
Payel Das, Kahini Wadhawan, Oscar Chang, Tom Sercu, Cicero Dos Santos, Matthew Riemer, Vijil Chenthamaraksham, Inkit Padhi, and Aleksandra Mojsilovic. Pepovae: Semi-supervised targeted design of antimicrobial peptide sequences. arXiv preprint arXiv: arXiv:/810.07743, 2018.
Laurianne David, Amol Thakkar, Rocío Mercado, and Ola Engkvist. Molecular representations in ai-driven drug discovery: a review and practical guide. Journal of Cheminformatiis, 12(1):1-22, 2020.
Pierre Dognin, Igor Melnyk, Youssef Mrouch, Jerret Ross, and Tom Sercu. Adversarial semantic alignment for improved image captions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 10463-10471, 2019.
Angela Fan. Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint orXiv:1805.04833 2018
Joe G Greener, Lewis Moffat, and David T Jones. Design of metalloproteins and novel protein folds using variational autoencoders. Scientific reports, 8-1):1-12, 2018.
Jiuxiang Gu, Jianfei Cai, Shafiq R Joty, Li Niu, and Gang Wang. Look, imagine and match: Im- proving textual-visual cross-modal retrieval with generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7 7181-7189, 2018.
Jingtong Hou, Gregory E Sims, Chao Zhang, and Sung-Hou Kim. A global representation of the protein fold space. Proceedings of the National Academy of Sciences, 10065))3386-2390, 2003.
Po-Ssu Huang, Yih-En Andrew Ban, Florian Richter, Ingemar Andre, Robert Vernon, William R Schief, and David Baker. Rosettaremodel: generalized framework for flexible backbone protein design. PloS one, 6(8):e2409,, 2011.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. In Advances in Neural Information Processing Systems, pp. 15820-15831, 2019.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recog- nition of vydrogen-bonder and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12):5577-2637 1983.
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of Chemical Infor- mation and Modeling, 2020.
Sofia Khan and Mauno Vihinen. Performance of protein stability predictors. Human mutation, 31 (6):675-684, 2010.
10

---

Under review as a conference paper at ICLR 2021
Nobuyasu Koga, Rie Missumi-Koge Gaohua Liu, Rong Xiao, Thomas B Acton, Gaetano T Mon- telione, and David Baker. Principles for designing ideal protein structures. Nature, 491(7423): 222-227, 2012.
Christina M Graemer-Pecore Andrew M Wollacott, and John R Desjarlais. Computational protein design. Current Opinion Chemical Biology. 5(6):690-695, 695, 2001.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint wwXiv:2004.03499 2020.
Alexey G Murzin, Steven E Brenner, Tim Hubbard, Cyrus Chothia, et al. Scop: a structural classifi- cation of proteins database for the investigation of sequences and structures. Journal of molecular biology, 247(4):536-540, 1995.
Saul B Needleman and Christian Wunsch. A general method applicable to the search for similar- ities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443-453, 1970.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In ICML, 2011.
Christine Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thortoo.. Cath-a hierarchic classification of protein domain structures. Structure, 508): 1109, 1997.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, pp. 8026-8037,2011
Yuxin Peng and Jinwei Qi. Cm-gans: Cross-modal generative adversarial networks for common representation learning. ACM Transactions Multimedia Computing, Communications, and Applications (TOMM), 15(1):1-24, 2019.
Gabriel Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexander Lemak, Lauren Carter, Rashmi Ravichandran, Vikram K Mulligan, Aaron Chevalier, et al. Global analysis of protein folding using massively parallel design, synthesis, and testing. Science, 357 (6347):168-1 -151668-1111111 2017.
Sillitoe, Natalie Dawson, Tony Lewis, Sayoni Das, Jonathan G Lees, Paul Ashford, Adeyelu Tolulope, Harry M Scholes, Ilya Senatorov, Andra Bujan, et al. Cath: expanding the horizons of structure-bart based functional annotations for genome sequences. Nucleic acids research, 47(D1);
D280-D284, 2019.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In Advances in neural information processing systems, pp. 935- 943, 2013.
Alexey Strokach, David Becerra, Carles Vergi-Verge, Albert Perez-Riba, and Philip Kim. Fast and flexible design of novel proteins using graph neural networks, bioRxiv, pp. 868935, 2020.
Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consor- tium. Uniref clusters: comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31(6))926-932. 2015.
William R Taylor. A 'periodic table' for protein structures. Nature, 4 416(6681))6677-4 2002.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 conference on empirical methody in natural language processing, pp. 1499-1509, 2015.

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. Joint embedding of words and labels for text classification. arXiv preprint arXiv: 04174, 2018.
Kaiye Wang, Ran He, Wei Wang. Liang Wang, and Tieniu Tan. Learning coupled feature spaces for cross-modal matching. In Proceedings of the IEEE International Conference Computer Vision, pp. 2088-2095, 2013.
Jianyi Yang, Renxiang Yan, Ambrish Roy, Dong Xu, Jonathan Poisson, and Yang Zhang. The i-tasser suite: protein structure and function prediction. Nature methods, 12(1):7-8, 2015.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4))702-710, 2004.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via joint latent similarity embedding. In proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 6034- 6042, 2016.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2232, 2017.
A 3D EXTENSION OF THE SINUSOIDAL ENCODING
We use a simple extension of the sinusoidal encoding described in the original transformer model (Vaswani et al. 2017) encode each position our Structure Encoder.
PE(x, 2, 20) sin(z/100000"/h) + sin(g//100002//h) + sin(2/100002//)
(5)
2E(2, cos(z/100000 + cos(s/10000 cos(z/100002//1)
B DATASET STATISTICS
The statistics our various datasets are given below.
Training includes 45995 proteins belonging total 1093 folds.
Validation set includes 1590 proteins belonging total 256 folds.
In-distribution (ID) test set includes 1230 proteins belonging to total 230 folds.
Out-of-distrrbation (OD) test set includes 204 proteins belonging to 28 folds.
C DISTRIBUTIONS OVER ATM
For the jth protein, we define ATM; = TM(s Fold2Seq Native) (M (M tions of ATM for the ID and OD are shown in Figure
Rosetta, NaNaive). The distribu-
12

---

Under review as a conference paper at ICLR 2021
[a)6
(b)
P-value=2.14E-60 60
Pvalue-0.019
5-
Density
Density
2-
1-
-0.4 00.2 0.0 0.1 0.2
0.3
0.4
-0.4
-.3--03--0
-..
0.1
0.0 0.1
0.2
0.3 0.4
ATM Test Set
ATM OD Test Set
Figure 4: The distributions of ATM for two test sets.
D COMPARISON OF FOLDED STRUCTURES
In this section, we show some representative folded structures whose sequences are designed by RosettaDesign and Fold2Seq. The folded structures were predicted using iTTasse, a state of the art program for protein structure prediction. [ some structures where Fold2Seq performs better than RosettaDesign and Figure showss some structures where RosettaDesign is better.
Rosetta Designed Strutures
TM-0.241
TM=0
TM50.355
204
Fold2Seq Designed Structures
TM-0.972
TM-0.961
TM+0 443
TM-0 324
Ground-truth Structures
1oufA00-1-130 (In-domain Set)
1nh0A00-1-99 (In-domain Sct)
2c12A00-1-88
(Out-of-domain Test Sety(Out-ol domain Test Sel)
Figure 5: The native and designed structures with ATM, > 0. The IDs at the bottom are the CATH domain names of each structure.
13

---

Under review as a conference paper at ICLR 2021
Rosetta Designed Strutures
TM-0 985
TM-0 838
TM-0
TM-0.582
Fold2Seq Designed Structures
m
Ground-truth Structures
2elfA01-4-73 (In-domain Test Set)
1dbyA00-1-44 1-107
1w2/AD0-2-91
2rmA01-447-550
(In-domain Test Set) (Out-of-domain Test Set) (Out-of-dee -out-of-domain Test Set)
Figure 6: The native and designed structures with ATM; 0. The IDs at the bottom the CATH domain names of each structure.
E COMPARISON BETWEEN TWO TRAINING STRATEGIES
In this section, we compare the performance between one-stage training and two-stage training strategies. In the one-stage strategy, train our model through the loss terms in Eq (4) together. While in the two-stage strategy, we train our model through L1 first and then train it through Lz-
We first compare in the learning curves. As the RE, loss represents the quality of the model, plot the RE, loss vs epochs on both training and validation sets.
3.2
Two-stage Training
Two-stage Validation
3.0
One-stage Training
One-stage Validation
2.8
RE
2.6
2.4
2.
2.0
1.8
25
50
100
125
150
175
200
Epoch
Figure The fold2seg loss(RE,) curves of two training strategies on training and validation
14

---

Under review as a conference paper at ICLR 2021
Table
Performance of two training strategies assessed in the sequence domain.
(a) Per-residue Perplexity.
(b) Avg. SeRR + std. dev. (%)
Model
ID Test
OD Test
Model
ID Test
OD Test
Uniform
20.0
20.0
Random across two folds
12.8 +: 7.94
12.8 7.94
Natural
18.0
18.0
One-stage strategy
19.2 + 7.2
16.7 + 4.3
One-stage strategy
14.0
16.2
Two-stage strategy
27.1 + 6.31
24.1
2.64
Two-stage strategy
8.1
11.9
Random within same fold
39.1 + 9.35
39.1
9.35
As shown in Fig 7 the two-stage strategy significantly outperformed one-stage strategy. To further demonstrate that, we calculate the per-residue perplexity and average sequence recovery rate on the two test sets. As shown in T the same conclusion can be drawn. Those results validate our design choice for two-stage strategy.
F TSNE FOR FOLD/STRUCTURE EMBEDDINGS
Graph Trans
Fold25eq
Component
Component
-3-
Component
Component
The (SNE visualization of the averaged structure(fold) latent embeddings h after the structure encoder of two methods on OD test set. Each protein colored by its fold category. Same color indicates the same fold, except that gray points represent outliers, which is defined by its fold having < 5 proteins in the test set.
Figure 8:
G AVG. SEQ RECOVERY RATE PER FOLD.
155
Density
125
Avg. seq recovery rate per fold.
Figure 9: Avg. SeRR per fold for the test set.
15


Reviews:
[{"review_id": "x5IFXZfLkZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper on predicting the underlying amino acid sequence from a given protein tertiary (3D) structure, an important problem with multiple applications. The authors have proposed an elaborate system and performed an impressive set of experiments to demonstrate the efficacy of their model. The proposed method (Fold2Seq) has several novel contributions, which I list below alongside my comments on them.\n\n1. A novel fold representation based on voxels of the density of secondary structure elements (SSEs). This is promising, however, I am not sure how sensitive to small changes in the 3D structure this representation is. The authors shift the structure such that its center of mass is at the origin and its first CA is on the negative side of the Z-axis. What happens only if the first residue is moved around? This is considering that the residues in the N-terminal usually do not have a solid form. Moreover, since the authors focus on short proteins (N < 200), I am curious to know how they handle structures determined by NMR, in which an ensemble of structures is determined, instead of a single structure.\n2. A novel joint sequence-fold embedding is learned that is employed in the transformer-based auto-encoder model. This is a great idea and seems to play an important role in Fold2Seq’s performance. Imagine two proteins that have two domains each and share one domain. How would their fold and sequence embeddings look? I am concerned that Fold2Seq might memorize some domains. For example, the drastic difference between Fold2Seq and RosettaDesign TM scores in ID and OD test sets is concerning. While RosettaDesign performs similarly in ID and OD sets, Fold2Seq has a significant drop in performance. I think the distribution of sequence similarities between each test sequence and the closest sequence in ID and OD sets should be provided. \n3. I am not sure if splitting the datasets solely based on “fold” is enough to prevent information leakage. I think it should be both based on fold and sequence similarity. Also, since proteins are so small, it might be helpful if their domains are also considered, e.g., using Pfam or computational methods such as doi.org/10.1101/626507.\n4. It is not clear to me why the authors did not compare to methods without the flexible backbone constraint (e.g., Ingraham 2019), the rationale is not fully convincing.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "wJEX1kC4RB3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackle the challenge of designing protein sequences that are consistent with a given 3D fold. To address this challenge, the authors propose a transformer-based generative framework that designs protein sequences conditioned on a given fold. There are two central contributions - the first is a novel fold representation, in which the 3D structure is represented by the voxels of secondary structure elements, and then a fold representation is learned via a transformer-based structure encoder. The second is a joint sequence-fold embedding learning framework. The authors use ablation studies to show that learning a joint latent space between sequences and folds enables the model to better capture the the sequence-fold relationship, improving experimental results. The authors provide a good summary of related work. \n\nTo proceed with the novel representation, the authors first scale each protein structure to fit into a fixed size cubic box, discretized into fixed size voxels, and extract the alpha carbon coordinates of each amino acid. They assign secondary structure labels drawn from a 4-letter alphabet to each residue, and compute the resulting features of each voxel. These features are used to train a structure encoder, which is used together with a trained sequence encoder to learn a joint sequence-fold embedding. Here fold classification is used for the intra-domain task, while for the cross domain loss they maximize the cosine similarity between the outputs of the sequence and structure encoders in addition to a cyclic loss. \n\nThe authors use the data from CATH 4.2 to define two tasks based on a random (ID) split, and a fold-level (OD) split of the data. In each case 2.5% of the data is held out as validation and test sets respectively. They perform various experiments that compare the performance of their approach against cVAE, gcWGAN and RosettaDesign. The performance of this approach is impressive, however I am surprised by the decision to leave out methods that do not focus on the inverse folding problem with a flexible backbone constraint - I would like to see the performance comparison with these methods. Please could the authors adequately justify this decision, or provide these comparisons. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "i6ICc4POer", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript presents a method for generating protein sequences conditioned on protein structures. The core idea is to represented protein structures by their secondary structures in 3D space. This voxel grid is then encoded into a vector representation and decoded to a distribution over sequences. The authors propose to learn this model jointly with a sequence encoder, combining the sequence and structure representations to decode the sequence during training. For inference, the sequence encoder component is not used. Learning to generate protein sequence conditioned on structure is an interesting and important problem and has been attracting increasing attention from the ML community. Representing structures as voxel grids is an approach worth exploring and flexible structure representations could be promising. However, it isn’t clear to me that this work achieves those goals and comparisons against key baselines (namely Ingraham 2019) are missing. Furthermore, the authors make many unsupported and unsubstantiated claims about their method. Specific comments and questions follow below.\n\n1.\tThe claim that this method allows flexible fold representations seems to be undermined by the need to specify a specific structure for which sequences are decoded. How does this method allow for more flexible structure representations than Ingraham 2019 or other methods that require 3D coordinates of each residue? It isn’t clear that this representation is more flexible than that required by RosettaDesign either. The authors need to justify this claim or remove it.\n2.\tThe authors also claim that their method preserves spatial secondary structure information in contrast to others. However, all backbone-based protein structure representations preserve secondary structure information as this is defined by the backbone angles. The authors also claim that their method does not need predefined rules or structures. However, their structure representation is derived from a predefined structure, so this claim makes no sense.\n3.\tHow does the density model loosen the rigidity of structures? The density model only represents a single structure and the authors do not use structure ensembles or other means to represent structural flexibility. I fail to see how this claim is supported by the presented method or experiments. In principle, any structure->sequence model can account for flexibility by ensembling over possible 3D structure.\n4.\tHave the authors compared their method with Ingraham et al. 2019? Ingraham et al solve the same problem with, arguably, an even more flexible representation of protein structures, because the graph based representation is invariant to rotation and translation of the structure in 3D space. A comparison against Ingraham et al is critical to understand if the proposed voxel-based structure representation is better than other methods. At face value, Ingraham 2019 reports better perplexities with a difficult train/test split and fewer training examples than this work. In fact, Ingraham et al’s out of distribution perplexity surpasses this method’s in distribution perplexity calling into question the utility of this method. The authors should compare their method on the same train/val/test split used by Ingraham et al.\n5.\tFor the in distribution test, how do sequence models fit per fold perform? For example, how well do profile HMMs fit to each fold model the heldout sequences from those folds?\n6.\tWhy are the val and test splits so small (only 2.5%)? Given these splits, how many sequences occur in each?\n7.\tBecause this method encodes structures into a voxel grid, it is not invariant to rotation and translation of proteins in 3D space. The authors align their structures based on center of mass and orient the protein such that the first residue points in the negative direction of the z-axis. How sensitive is the method to misaligned proteins? Is this alignment scheme robust to alternative conformation of the same protein? What happens when structures are rotated? The described alignment scheme also has ambiguity in the rotation around the z-axis. How is this addressed?\n\nThings that would improve my score:\n1.\tProvide support for unsupported claims in the introduction.\n2.\tCompare against Ingraham et al 2019.\n3.\tExamine the sensitivity of this method to rotated structures.\n\nEdit: I have read the authors' response and update  manuscript. I appreciate the new experiments and some of the clarifications. However, some of my fundamental issues with this work are not addressed.\n\nThe distinction with Ingraham et al feels forced. It is important to compare head-to-head on Ingraham's dataset in order to truly understand the tradeoffs between these methods and to understand the differences in performance between flexible backbone approaches and a well tuned rigid backbone approach. Why are perplexities for Ingraham et al not reported for all cases? Improving over Ingraham when only low resolution structures or incomplete structures are available is interesting, but I also question how useful this case is. When attempting to do _structure-based design_ of new proteins, how often are fully specified structures not available for those tasks? Ingraham et al presented a variant of their method based only on a flexible structure specification and it isn't clear how that contrasts with this flexible specification. It is certainly possible that this approach is better, but we need a head-to-head comparison to know.\n\nWith that in mind, I'll raise my score to 5. I think this work has promise, but is not yet ready for publication in its current state.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6EdRK_ac6nt", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this paper propose a Transformer-based multimodal autoencoder architecture (including a sequence encoder, a structure encoder, and a sequence decoder) for protein design. In the method, a claimed novelty is that a 3D proteins structure is represented by 3D voxels of density of secondary structures. Transformer is used for sequence modelling and structural modelling. A another major claim of this paper is the joint sequence-fold representation learning. The objective function reflects the needs for both intra-domain representation learning and cross-domain representation learning. Comparisons with a principled-based method - RosettaDesign and two machine learning methods - cVAE and gcWGAN at both sequence level and structural level show that the proposed method can achieve better performance on sets of proteins with limited sizes. Overall, this is an interesting work for 3D structure learning and design. However, I also have the following concerns. \n\nMajors:\n1. A possible major drawback of the proposed structural representation method is that it relies on rescaling of very large structures to the 3D frame of fixed size. In the experiments, proteins with more than 200 amino acids were filtered out. Moreover, in the inference phase, this representation can cause a complication to the length of generated sequences.\n\n2. In the training phase, it is unclear whether L_1 and L_2 are optimized alternatingly along multiple epochs, or L_1 is optimized for multiple epochs then L_2 for multiple epochs. How much gain was obtained in comparison with just direct optimization of the objective function with five terms in terms of learning curves and performance at sequence and structure levels? I guess the unbalanced training progress is due to the fact that training sequences is easier than training 3D structures. \n\nMinors:\nposition(3D -> position (3D\ntransformer -> Transformer\nIn the text, the authors state that \"we consider to maximize the cosine similarity (shown as the ‘Cosine Similarity’\npink block in Fig 2) ...\", however, \"Cross Similarity\" is given in Figure 2. \na MLP -? an MLP\nPytorch -> PyTorch\nai-driven -> AI-driven [in the references, check similar issues]", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a promising idea, without enough empirics to substantiate its potential utility, and also with a lack of clarity on the importance of the outlined task itself (fold-based rather than structure-based conditional sequence generation). There remain concerns about the lack of a more comprehensive comparison to methods for structure-to-sequence (e.g. Ingraham was added during the revision but only in a limited capacity), or easy generalizations of them, and about the quality of some of the presented results. Additionally, the concern about sensitivity to rotational invariances, and related issues wrt the fixed-size cubic grid were not satisfactorily addressed. As a side-note, the quality of the manuscript in terms of scholarliness of presentation was overall lacking.
Decision: Reject