De novo Protein Design Using Geometric Vector Field Networks
Authors: Weian Mao
Muzhi Zhu
Zheng Sun
Shuaike Shen
Lin Yuanbo Wu
Hao Chen
Chunhua Shen
Venue: ICLR 2024 spotlight (2024)
Keywords: Protein design
Protein structure encoder
Inverse folding
Protein diffusion
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Advances like protein diffusion have marked revolutionary progress in $\textit{de novo}$ protein design, a central topic in life science. These methods typically depend on protein structure encoders to model residue backbone frames, where atoms do not exist. Most prior encoders rely on atom-wise features, such as angles and distances between atoms, which are not available in this context. Only a few basic encoders, like IPA, have been proposed for this scenario, exposing the frame modeling as a bottleneck. In this work, we introduce the Vector Field Network (VFN), that enables network layers to perform learnable vector computations between coordinates of frame-anchored virtual atoms, thus achieving a higher capability for modeling frames. The vector computation operates in a manner similar to a linear layer, with each input channel receiving 3D virtual atom coordinates instead of scalar values. The multiple feature vectors output by the vector computation are then used to update the residue representations and virtual atom coordinates via attention aggregation. Remarkably, VFN also excels in modeling both frames and atoms, as the real atoms can be treated as the virtual atoms for modeling, positioning VFN as a potential $\textit{universal encoder}$. In protein diffusion (frame modeling), VFN exhibits a impressive performance advantage over IPA, excelling in terms of both designability ($\textbf{67.04}$\% vs. 53.58\%) and diversity ($\textbf{66.54}$\% vs. 51.98\%). In inverse folding(frame and atom modeling), VFN outperforms the previous SoTA model, PiFold ($\textbf{54.7}$\% vs. 51.66\%), on sequence recovery rate; we also propose a method of equipping VFN with the ESM model, which significantly surpasses the previous ESM-based SoTA ($\textbf{62.67}$\% vs. 55.65\%), LM-Design, by a substantial margin. Code is available at https://github.com/aim-uofa/VFN

Published as a conference paper at ICLR 2024
DE NOVO PROTEIN DESIGN USING GEOMETRIC VECTOR FIELD NETWORKS
Weian Mao!.2 Muzhi Zhu';
Zheng Sun Shuaike Shen',
Lin Yuanbo Wu', Hao Chen',
Chunhua Shen 1.4
Zhejiang University, China
The University of Adelaide, Australia Ant Group
Swansea University, UK
ABSTRACT
Innovations like protein diffusion have enabled significant progress in de novo protein design, which is vital topic in life science. These methods typically depend on protein structure encoders to model residue backbone frames, where atoms do exist, Most prior encoders rely on atom-wise features, such angles and distances between atoms, which are not available in this context. Thus far, only several simple encoders, such as IPA (Jumper et al., 2021), have been proposed for this scenario, exposing the frame modeling as bottleneck. In this work, we proffer the Vector Field Network (VFN), which enables network layers to perform learnable vector computations between coordinates of trame-anchored virtual atoms, thus achieving higher capability for modeling frames. The vector computation operates in manner similar to linear layer, with each input channel receiving 3D virtual atom coordinates instead of scalar values. The multiple feature vectors output then used to update the residue representations
by the vector computation and virtual atom coordinates via attention aggregation. Remarkably, VPN also excels modeling both frames and atoms, as the real atoms can treated the virtual atoms for modeling, positioning VFN as a potential universal encoder. In protein diffusion (frame modeling), VFN exhibits an impressive performance advantage over IPA, excelling in terms of both designability (67.04% vs. 53.58% and diversity (66.54% vs. 51.98%). inverse folding (frame and atom modeling), VFN outperforms the previous SoTA model, PiFold (54.7% 7% vs. 51.66%), on sequence recovery rate. We also propose a method of equipping VFN with the ESM model (Lin et al., 2023), which significantly surpasses the previous ESM-based best result (62.67% vs. 55.65%) LM-Design (Zheng et al., 2023), by substantial margin. Code is available at https::///// / thub. com/ molm--ofaa fa/VFN
A
B
1.0
$4.70
54
54.11
0.9
988
0.7
52.68
0.6
51.70
51.74
0.5
51
51.20
50.85
+ PiFoic
RameDiff Diff
VPN-FF
10
14
19
25
70
100
200
300
infer time (ms)
Longth Ranges
Figure 1: Experimental results on protein diffusion and inverse folding. A) VFN-IF compared to PiFold with varying numbers of layers, showcasing the trade-off between speed and sequence recovery rate. B) VFN-Diff compared to FrameDiff (IPA) for designability across proteins of varying lengths.
*WM, 7S and MZ contributed equally. Work was done when WM was visiting Zhejiang University. HC and CS the corresponding authors.

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
The field of de novo protein design (Huang et al., 2016) represents a pivotal frontier in the realms of bioengineering and drug development, holding the potential to bring about a revolutionary shift in the creation of innovative therapeutic agents. Recent advancements in this domain have been driven by a groundbreaking paradigm that combines protein structure diffusion models (Watson et al., 2023; Yim et al., 2023) with inverse folding networks (Dauparas et al., 2022). Specifically, this paradigm initiates by employing a protein diffusion model to stochastically generate the backbone structure of protein, represented by residue frames. Since the types of amino acids in the generated protein are initially unknown, an inverse folding network is then utilized to design the protein sequences for each residue based the backbone residue frames. While this paradigm has shown great success for protein design, it also brings challenges for deep learning-basse protein structure encoders.
Methods like protein diffusion rely protein structure encoders to model and sample protein structures, However, prior structure encoders are unsuitable or severely limited in this case, primarily because the representation of residues is highly specialized in methods like diffusion. Specifically, in those methods, backbone frames are commonly employed to represent the spatial information of residues, with atom-level representations typically absent. The absence of atoms has rendered most previous encoders ineffective for protein design, as they typically rely on atom-level input features such as interatomic angles and distances. Although few basic encoders, such as IPA (Jumper et al., 2021), were designed for frame modeling, they still faced significant limitations. For instance, IPA simply performs distance pooling between frame- anchored virtual atoms as geometric features. Obviously, such pooling operation and the single pooled distance value clearly lacks the expressiveness required for representation. We refer to these limitations as the atom representation bottleneck, which is explained in detail in $2.4. In response to this challenge, we propose novel structure encoder called the Vector Field Network (VFN).
The core idea of VFN revolves around the utilization of vector-specific linear layer to extract multiple geometric feature vectors by mapping the coordinates of frame-anchored virtual atoms. Specifically, VFN introduces virtual atoms in Euclidean space for each amino acid, and these virtual atoms move in conjunction with the frames, serving as dynamic representations of the frames. When modeling the relationship between two amino acids, module called the 'vector field operator' takes the virtual atom coordinates of the two amino acids as vector inputs and performs operations similar to linear layer. Within the vector field operator, each vector initially multiplied by a learnable scalar weight, and these weighted vectors are accumulated to compute the output feature vector. Like a linear layer, the vector field operator has multiple output channels, with each output channel generating Euclidean vector. Subsequently, these output feature vectors are directed into MLP layer for processing, facilitating the fusion of residue features and enabling the modeling of residue frames.
VFN exhibits significant advantages over previous encoders in terms of its expressive power and versatility, VFN circumvents the atom representation bottleneck present in IPA. As previously mentioned, IPA relies on a single scalar variable, which represents the sum of distances between virtual atoms, serving as both geometric features and attention bias. This single scalar variable and corresponding pooling operation limit the expressive capacity of IPA. In contrast, VFN can flexibly extract multiple feature vectors through a vector field operator, thereby circumventing this bottleneck. Please refer to $2.4 for more details.
In scenarios where frame and backbone atoms coexist, such as inverse folding, VFN maintains excellent expressiveness while simultaneously offering enhanced generality and fiexibility. This because real atoms can be treated as virtual atoms in VFN, and the coordinates of both real and virtual atoms can be used to model the relationships between amino acids using the vector field operator, naturally forming hierarchical representations. Compared to other atom-based inverse folding methods, VFN achieves superior performance by potentially modeling frames more effectivvly. Thus, VFN also surpasses the current SoTA in inverse folding. is worth emphasizing that, IPA is not suitable for treating the real atoms as the virtual atoms, since the IPA network cannot provide learnable weights for the coordinates of each atom, facing the atom representation bottleneck.
To assess the performance of VFN for de novo protein design, have implemented two models based on VFN, namely VFN-IF and VFN-Diff, tailored for protein inverse folding and diffusion based generative modeling, respectively. Experimental results consistently demonstrate the remarkable
2

---

Published as a conference paper at ICLR 2024
performance of VFN. For protein diffusion, VFN-Diff significantly outperforms the prior solid baseline, FrameDiff (Yim et al., 2023), in terms of designability (67.04% vs. 53.58%) and diversity (66.54% vs. 51.98%). It is important emphasize that the key distinction between VFN-Diff and FrameDiff lies in the replacement of the encoder-IPA, with VEN layers. This substantiates the superior geometric feature extraction capability of VFN over the widely adopted IPA.
In the domain of inverse folding. VFN-IF exhibits a substantial performance boost over the current state-of-the-art method, PiFold (54.74% vs. 51.66%, sequence recovery rate). Furthermore, we have trained a larger-scale VFN-IF model on the entire PDB database, achieving an impressive sequence recovery rate of 57. 14%, underscoring the scalability of VFN-IF. Additionally, we propose novel variant of VFN-IF, called VFN-IFE, which is equipped with an external knowledge base, achieves remarkable precision at 62.64%, %, surpassing SoTA approaches this regard, L.M-Design (Zheng et al., 2023) (55.65%), by a substantial margin.
The main contributions of this work can be summarized as follows:
We propose the Vector Field Network (VFN), which employs a vector field operator to extract geometric feature vectors between frame anchored virtual atoms, resembling a linear layer. This approach overcomes the atom representation bottleneck, thereby enhancing representational capabilities. Notably, VFN can also incorporate real atoms as the virtual atoms for hierarchical modeling, positioning it as a potential universal encoder.
For protein diffusion, VFN significantly enhances the designability((77 (67.0%% vs. 53.58%) and diversity(66.54% vs. 51.98%) of protein generation compared to IPA.
For inverse folding, VFN significantly surpasses the previous SoTA model, PiFold (54.74% vs. 51.66%, sequence recovery rate). Additionally, we propose a method for equipping VFN with an external knowledge base, achieving substantial breakthrough over the SoTA approach, LM-Design, in this regard, thus elevating model accuracy to next level (62.64% vs. 55.65%).
2 RELATED WORK
2.1 DE NOVO PROTEIN DESIGN
De novo protein design, which involves the creation of proteins from scratch, holds paramount significance in the fields of enzyme engineering and protein engineering. Traditional approaches such as RosettaDesign (Liu & Kuhlman, 2006) were prevalent before the advent of machine learning- based methods. In recent years, with the maturation of machine learning techniques. advanced deep learning- based methods have emerged, exemplified by Huang al. (2022) (Side Chain Unknown Backbone Arrangement) and protein diffusion (Watson et al., 2023; Fu et al., 2023).
The paradigm of protein diffusion is regarded
of the most promising methods in the realm of protein design, which encompasses protein diffusion (Yim et al., 2023) and inverse folding (Gao al., 2022; Jendrusch et al., 2021: Wu al., 2021; Ovchinnikov & Huang, 2021; Dauparas et al., 2022; Ingraham et al., 2019; Hsu et al., 2022; Gao et al., 2024; Derevyanko et al., 2018). Specifically, a protein diffusion model first generates the backbone structure of a protein, followed by an inverse folding network that designs the corresponding sequence for this backbone. The feasibility of both these steps has been experimentally validated through cryo-electron microscopy (Watson et al., 2023; Dauparas et al., 2022), marking significant breakthrough in the field of protein design. However, while protein diffusion methods based on frame representation achieve significant success, these methods, atom representation is absent, rendering previous general purpose encoders unusable.
2.2 GENERAL PURPOSE ENCODER
In the past, numerous encoders (Hermosilla et al., 2020; Zhang et al., 2022; Hermosilla & Ropinski, 2022; Veličković et al., 2018; Baldassarre et al., 2021; Li et al., 2022; Shroff et al., 2019; Dumortier et al., 2022; McPartlon et al., 2022; Cao et al., 2021; Anishchenko et al., 2021; Karimi al., 2020; Zhang et al., 2020; Wang et al., 2022b; Derevyanko et al., 2018) have been proposed for tasks such model quality assessment (Townshend et al., 2021) and fold classification (Hou et al., 2018), where atomic information is available. However, these methods are not suitable for protein design tasks where atomic representations of proteins unavailable. For instance, GVP (Jing al., 2020)
3

---

Published as a conference paper at ICLR 2024
transforms input atomic coordinates into vector and scalars variables as the network input, facilitating the model's ' SE(3) invariance. Meanwhile, Wang et al. (2022a); Jin et al. (2022) proposed efficient approaches for modeling protein structures using hierarchical protein representations, enabling GNNs to possess both residue-level and atom-level hierarchical representations. Nevertheless, such approaches are evidently less applicable in the methods like protein diffusion, atomic information is unattainable.
2.3 FRAME-BASED ENCODER
The use of crame-based representations in protein structure encoders has been relatively unexplored. with few existing encoders being rather rudimentary in nature. Historically, these encoders have primarily served as auxiliary modules in protein structure prediction models, such as RoseTTA (Back et al., 2021) and AlphaFold2 (Jumper et al., 2021). l). In RoseTTAFold, the explicit concept of frames is absent; instead, employs the origin of frames to represent frames and employs SE(3)- transformers (Fuchs et al., 2020) model these frames. However, this approach conspicuously neglects rotational information of frames, considering only positional offsets. AlphaFold2, on the other hand, introduces the IPA structural encoding module address this limitation. IPA represents frames using framed-anchored virtual atoms and extracts geometric information between two frames by computing the sum of distances between virtual atom pairs. This approach takes into account both the coordinate offsets between frames and their rotational information.
2.4 ATOM REPRESENTATION BOTTLENECK
However, IPA still faces the atom representation bottleneck. Firstly, IPA does not incorporate learnable weights when extracting features for virtual atoms; it directly applies a simple summation pooling operation to the atom-pair distances, thereby lacking flexibility. Secondly, the sum of distances between virtual atom pairs yields only scalar variable, making it challenging to represent complex geometric information, thus lacking expressiveness. In VFN, these issues have been successfully resolved. VFN introduces operator similar linear layer, characterized by the inclusion of learnalle weights associated with individual atom coordinates. These weights facilitate flexible vector computations, effectively mitigating the flexibility challenges inherent in IPA's pooling operation. Furthermore, the operator in VFN can return multiple feature vectors, eliminating the expressiveness bottleneck caused by the single scalar variable (the pooled distance) in the IPA. For further details regarding the vector field operator, please refer to $3.2.
3 VECTOR FIELD NETWORK LAYERS
The Vector Field Network (VFN) is designed to extract geometric features between amino acids using a module, named the vector field operator (refer to $3.2). In each layer of VFN, the protein's representation undergoes a sequential process including the vector field operator, node interactions ($3.3), edge interactions ($3.4) and virtual atom updating ($3.4). The vector field operator is crucial for extracting geometric features between pairs of amino acids through vector computations and virtual atoms. Subsequently, these extracted geometric features are aggregated and employed to update the representation of each amino acid through node and edge interactions. At the end of each layer, the coordinates of the mentioned virtual atom are updated by aggregation residue representations. The mentioned modules and the overall pipeline are elucidated subsequently.
3.1 PROTEIN REPRESENTATION
In VFN, protein consisting of n amino acids represented as a graph denoted as 9 (S,ET Q). Here, S {ss E d ) li= represents the set of node features. To encode the positional information of each amino acid in space, we use a set of local frames, T = (T.).= to represent the position of each amino acid. We introduce set of Trame-anchored virtual atoms, whose coordinates are maintained by Eq. equation 8 or Eq. equation 9. Specifically, we denote all virtual atom coordinates @ (Qi)i,,,.....,, with Q {qı E RR J=-,,,, representing set of virtual atom coordinates associated with the i-th residue w.s.f T.. dy denotes the number of virtual atoms treated as vectors in the vector field operator,
in each residue. These virtual atom coordinates allowing for vector computations to extract geometric features. Additionally, we construct edges in the graph based on specific rules, such as a complete graph or k-nearest neighbors. The complete graph is taken by default, so the set of edge features can be denoted as E = {ci. E Rd fi.==......
4

---

Published as a conference paper at ICLR 2024
A
B
T4
Q
Q
Kj
Kg
q:
h2
k, Kj = Titj
W 1 wb
0
-1
Figure 2: Pipeline for the Vector Field Operator. A) Transforming the virtual atomic coordinates Q; from frame T, to frame T, to obtain Kj. B) An example vector computation involving vectors Q and K, using learnable weights w and w as defined in Equation equation When w and w are specific weights (as shown in figure), the vector field can yield the Euclidean vector, h1 and hig between two particular atoms.
3.2 VECTOR FIELD OPERATOR
As shown in Figure 2, the vector field operator extracts geometric features between two amino acids by performing learnable vector calculations on their virtual atom coordinates. Specifically, when modeling the geometric relationship between two amino acids, such as the i-th and j-th amino acids, the input to the vector field operator consists of virtual atom groups, Q; and Qj- To ensure that all vectors are represented in the same local coordinate system, we first transform Q, from the local
frame T, to T,, and denote the transformed coordinates as K [kg E R3 } which can be written (as shown in Figure 2.A):
K, + Q₁ where T T T
Here, Ti-j represents the transformation matrix transforming the coordinates from T, Tr. Next, to select specific virtual atoms for calculating feature vectors, similar to a linear layer, we introduce two of learnable weights, w {wf, RR..:=.....d and w {w E R.,,=...... Those weights are then utilized perform vector calculations between Q; and Kj. resulting in extracted feature vectors H.j (h& E R )=I..,.... (as shown in Figure 2.B ). This can be expressed as follows:
hi Σ Wh 1 + Σ kk
where q Qi Ky E H.y. Here, H.j serves vector representation for captured geometric features. However, due to its large numerical range (ranging from -00AA to 200A), it can lead to instability during network training and requires further processing.
To avoid this issue, H. decomposed into two variables: unit direction vectors and vector lengths. The vector lengths will be mapped using a radial basis function, RBF, This can be written as:
h
gij concat&
gi, E Rd
gi.j is a vector that represents the geometric relationship between two residues and is used in the following module for aggregating and updating the features s; and eij. The concat, represents the concatenation of all the feature' resulting from all the vectors H = [he (he d represents the number of channels in gi.j-
3.3 NODE INTERACTIONS
Here, a MLP-based multi-head attention mechanism is designed to aggregate geometric features gij+ node features s; and 8j. edge features oı and update the node representations s;. Specifically, the pair-wise features mentioned above are first fed into MLP, followed by softmax operation to obtain attention weights, denoted as:
a. softmax, so
e1,)));
representing the flattened unit direction vectors and the values output by RBF
5

---

Published as a conference paper at ICLR 2024
where o,,j represents the attention weight for the interaction between nodes and j. Next, another MLP is employed to generate the values Vi,j for the multi-head attention, which are subsequently aggregated using the attention mechanism, expressed as follows:
Σ ViyV where Vij MLP(s), gijj, gi,, esg)
(5)
Here, o₂ represents the aggregated features, which utilized to update the features s; after undergo- ing an MLP layer, written as:
S; S, + MLP(o.)
Up this point, node features have been updated and utilized the input for the subsequent layer and following operations.
3.4 EDGE INTERACTIONS
Next, we introduce the edge interactions, which is designed aggregate geometric information gij node features s; and $y, and edge features Ci,j, to update the representation the edge e,,j. This can be written follows:
eijj t Big + MLP((s; 8j, Bi.j ev.j)
3.5 VIRTUAL ATOM COORDINATES UPDATING
In the final stage of each VFN layer, the coordinates of virtual atoms Q are updated. We have devised two different methods for updating these virtual atom coordinates, which are respectively referred to 'node feature-based updating' and 'coordinate aggregating updating." These approaches be selected based on the specific task, and their detailed methodologies are elucidated following.
processed through linear layer generate set of virtual atom coordinates for updating Q This process can be represented as follows:
Node Feature-Based Updating. Node features
Q2 Linear(s:)
Coordinate Aggregating Updating. For updating Q virtual atom coordinates K; are firstly aggre- gated through an attention mechanism to obtain the aggregated atom coordinates Q Subsequently, Qf fed into an MLP layer to update the coordinates Qü. denoted follows:
Q < V-MLP(Q, Q?), where Q
Σ as. K
Here, V-MLP represents a dedicated MLP layer designed specifically for vectors, as described in Appendix 1 di, is computed in Eq. equation 4.
4 IMPLEMENTATION FOR DE NOVO PROTEIN DESIGN
To establish the recent paradigm in de novo protein design, we have developed two dis-
A
B
tinct models, namely VFN-Diff and VFN- IF, each dedicated to protein structure dif- fusion and inverse folding, respectively. In the protein diffusion part, the protein struc- ture is designed and represented using back- bone frames T. Subsequently, these back- bone frames fed into the inverse folding network to obtain the corresponding pro- tein sequence for the designed structure. the following subsections, we present an overview of the implementations for VFN- Diff' and VFN-IF. For more implementation details, please refer to Appendix A.2.2 and Appendix A.2.3.
VEN-Diff
ESMFold
FrameDiff
Figure 3: Visual Comparison of VEN-IF and Frame-Difi
'BSMFold" represents protein structures recovered using Pro- teinMPNN and ESMFold, with closer structural resemblance being preferable.

---

Published as a conference paper at ICLR 2024
4.1 PROTEIN DIFFUSION
We adopted the FifmeDiff (Yim et al., 2023) paradigm, diffusion model used to sample protein backbone structures by updating residue frames T. In FrameDiff, a network called FramePred is employed to model the protein backbone frames T during each diffusion step. This network relies invariant point attention (IPA), which consists of three components: node attention, edge attention, and point attention. Importantly, the point attention module is the operation that causes the atom representation bottleneck, as mentioned in $2.4. Therefore, to fairly evaluate whether VFN can bypass this bottleneck, replaced the point attention with our VFN attention mechanism. The remaining parts of VFN-Diff remain consistent with FrameDiff.
4.2 INVERSE FOLDING
The purpose of the inverse folding task to map the frames T (generated by the diffusion model) to amino acid categories (1.....0))' denoted fa T These predicted amino acid categories aim enable the protein fold into the de- signed structure. In this task, we adopt the previous paradigm. Specifically, the VFN- IF network is composed of 15 sequential VFN layers, and constructs edges in the graph using a k-NN approach. The net- work takes backbone frames and backbone atoms input. It's important to note that the coordinates backbone atoms are used to initialize some of the virtual atom coor- dinates, enabling VFN to achieve higher modeling capacity through hierarchical rep- resentation. The prediction head VFN-
Sequence Recovery Length Rarge
0.6
Sequence
0.4
UMDesigr
VFN-IF
0-100
100-200
200-300 300-420
403-500
Length Rarge
Figure 4: Visualization comparison, conducted on in- verse folding and CATH 4.2, between sequence recov- ery rate and protein length.
IF follows the PiFold approach (Gao al., 2022), directly predicting amino acid categories for each node through a linear layer and supervising them using cross-entropy loss function. Additionally, we propose an approach fine-tune the ESM model (Lin et al., 2023) with LoRA (Hu al., 2021) to correct the predictions of VFN-IF. We name this approach VFN-IFE, and the process can be represented as fESM (ir(T) : -> c, where fir represents VFN-IF, and fESM : e - c denotes ESM model fine-tuned with LoRA to correct the VFN-IF predictions e", For more VFN-IFE details, please refer to Appendix A.2.4.
EXPERIMENTS
As mentioned, we comprehensively validated the superiority of VFN-IF and VFN-Diff through experiments involving protein diffusion and inverse folding. The following sections will present the main experimental results for these two tasks separately. Additionally, we have included experiment details, extensive visual analyses and ablation studies based on inverse folding in Appendix A.4. We encourage readers to refer to the appendix for more in-depth information.
5.1 INVERSE FOLDING
In inverse folding, we followed the settings of (Gao al., 2022) and tested the sequence recovery performance of VFN on the CATH 4.2 (Orengo et al., 1997), TS50, and TS500 datasets (Li et al., 2014). Furthermore, state-of-the-an models. Additionally, we demonstrated the superiority of VFN in trades-off between speed and accuracy.
tested the structure recovery performance of VFN and compared it with
In this task, we have devised three variations of the VFN model, namely VFN-IR, VFN-IF+, and VFN-IFE. VFN-IF represents the vanilla version of VFN, trained solely on the CATH 4.2 dataset. VFN-IF+ is an extended version of VFN-IF, scaled up incorporate the entire PDB dataset during training. VFN-IFE denotes version of VFN-IF equipped with an external knowledge base (ESM).
7

---

Published as a conference paper at ICLR 2024
Perplexity
Recovery's (%)
Model
Short
Single
All
Short
Single
All
StructGNN
8.29
8.74
6.40
29.44
28.26
35.91
GraphTrans
8.39
8.83
6.63
28.14
28.46
35.82
GCA
7.09
7.49
6,05
32.62
31.10
37.64
GVP
7.23
7.84
5.36
30.60
28.95
39.47
w/o ESM
GVP-large'
7.68
6.12
6.17
32.60
39.40
39.20
Design
7.32
7.63
6.30
34.16
32.66
41.31
ESM-IF!
8.18
6.33
6,44
31.30
38.50
38.30
ProteinMPNN
6.21
6.68
36.35
34.43
45,96
PiFold
6.04
6.31
4,55
39.84
38.53
51.66
VEN-IF
5.70
5,86
4.17
41.34
40.98
54.74
ESM
ESM-IF!
6.05
4,00
4.01
38.10
51,50
51.60
LM-Design
6.77
6.46
4.52
37.88
42.47
55.65
VFN-IFE
4.92
4.22
3.36
50.00
52.13
62.67
Table 1: Experimental results comparison on the CATH dataset (inverse folding). Some results are
reproduced by Gao et (2022). "f" denotes that the version of CATH used 4.3, while for the remaining methods, the CATH version is 4.2.
Sequence recovery. The performance of VFN on the CATH dataset and TS50, TS500 is presented in Table 1, Table 3 and Figure 4, respectively.
w/o ESM
ESM
Metric
PiFold
VEN-IF
LM-Design
VEN-IFE
these ex- periments, VFN is compared to other advanced models, such as StructGNN (Ingraham al., 2019), TraphTrans (In- graham et al., 2019), GCA (Tan et al., 2022), GVP (Jing et al., 2020), ESM-IF (Hsu et al., 2022), ProteinMPNN (Dau- paras al., 2022), PiFold (Gao et al., 2022), LM-design (Zheng et al., 2023), in terms of perplexity and sequence recovery performance. Table 1 provides detailed results for various subsets. Among the subsets considered in our analysis, the "Short" subset refers to proteins with length of to 100 amino acids, while the "Single" subset exclusively includes single chain proteins. 'w/o ESM' and 'ESM' refer to methods without and with the use of an external knowledge base (ESM), respectively.
ScTM > 0.5 scRMSD < 2
90.98%
92.37 %
89 42%
93.29 %
60.35%
62.89% %
58.41%
64.16%
Table 2: Experimental results on structure recovery (in- verse folding). 'scTM > 0.5' represents the percentage of designed proteins that exhibit a structural similarity ex- ceeding 0.5 with the desired protein. The same applies to 'scRMSD < 2'.
Structure recovery. We compared VFN's performance in terms of pro- tein structure recovery with several ad- vanced methods on CATH 4.2, as shown
TS50
TS500
Model
Perp.t
Rec.(%)t
Perp..
Rec,(%)
StructGNN
5.40
43.89
4.98
45.69
in Table 2. We followed standard eval- uation procedures( Yim et al., 2023).
GraphTrans
5.60
42.20
5.16
44,66
w/o ESM
GVP
4.71
44.14
4.20
49.14
Specifically, we used ESMFold to pre-
GCA
5.09
47.02
4.72
47.74
AlphaDesign
5.25
48.36
4.93
49.23
diet whether sequences designed by in-
ProteinMPNN
3.93
54.43
3.53
58.08
verse folding networks could fold pro-
PiFold
3.86
58.72
3.44
60.42
teins into the desired structures, i.c., the
VEN-IF
3.58
59.54
3.19
63.65
input structures of the inverse folding network. We employed two metrics, scTMt and scRMSDJ, to assess the sim- ilarity between the desired protein struc- tures and the structures of proteins de-
ESM
LM-Design
3.50
57.89
3.19
67.78
VFN-IFE
2.52
73.30
2.54
72.49
Table 3: Experimental results TS50 and TS500 (inverse
folding).
signed through inverse folding. Our experimental results demonstrate that VEN has a significant advantage compared to state-of-the-an methods.
Speed and accuracy trade-off. terms of the trade-off between speed and accuracy, we compared VFN-IF with the SoTA model in this regard, PiFold. We conducted comparisons between different layers of VFN and PiFold, as shown in Figure I(A). Experimental results demonstrate that VFN-IF achieves SoTA efficiency. Even with just 5 layers, VEN-IF achieves higher accuracy (52.68%) than 10-layer PiFold while maintaining faster inference speeds. Furthermore, PiFold's accuracy saturates 10 layers, whereas VFN-IF does not encounter this issue. For more details, please refer to Table 6a in Appendix.
8

---

Published as a conference paper at ICLR 2024
5.2 PROTEIN DIFFUSION
We followed the settings and benchmarks of FrameDiff (Yim et al., 2023), conducting a detailed comparison between VFN-Diff and FrfmeDiff in terms of designability and diversity. It is worth emphasizing that FrameDiff and VFN-Diff differ only in the protein structure encoder (VFN vs. IPA). All other settings are identical, and the parameter counts are similar (18.3M vs. 17.4M), making an ablation study. On the other hand, RFDiffusion (Watson et al., 2023) is a recent advance in the field. However, as FrfmeDifff pointed out, RFDiffusion performs noticeably worse than FrameDiff in the same setting (without traitrained weights). Additionally, RFDiffusion has a larger number of parame- ters and is trained on larger datasets (i.e., complex data). Therefore, comparing with RFDiffusion is beyond the scope of this work. We leave these engineering implementations for future research.
Designability. As shown in Table and Fig- ure 5, we compare the protein designability of VFN-Diff with that of FrameDiff. For evalua- tion, Proteins generated by the diffusion model reconstructed using the inverse folding net- work (ProteinMPNN) and structural prediction network (ESMFold). The designability of the generated proteins is then assessed by compar- ing their structural similarity (scTM, scRMSD) to the reconstructed proteins. Experimental re- sults demonstrate that VFN-Diff outperforms FrameDiff noticeably in terms of designability.
Protcin ScRMSD
16
Frame Diff
VFN-Difr
MYRNO Veluaes
10
Diversity. Diversity is an important metric follow
for generative models. Similarly, FrameDiff's relevant evaluation metrics, namely 'diversity' and 'pdbTM', to compare the diver- sity of VFN-IF and FrfmeDiff. Specifically, 'di- versity' represents the clustering center density of generated samples. To be more specific, we first excluded undesignable proteins (scTTM < 0.5). Then, we used MaxCluster (Herbert & Sternberg, 2008) cluster the remaining samples and ob- tain clustering centers. Finally, clustering center density can be calculated as follows: (number of clustering center) (number of generated samples). 'pdbTM' represents structural similarity of generated samples to the most similar structures in the PDB database.
Length Ranges
Figure 5: Visualization of designability. New repre- sents the number of attempts when reconstructing protein structures using Protein MPNN and ESM- Fold.
Visualization Comparison. Through visualization, we observed that FrfmeDiff
Setting
Noise Scale
1.0
0.5
0.1
0.1
0.1
Num, Step
500
500
500
500
100
Metric
Num. Soq.
100
suffers from very low des- ignability when designing longer protein, while VFN- Diff does not have this issue. To illustrate, typical generated proteins and compared the results generated by VFN-Diff and FrameDiff through visual- ization, respectively. The visualization results demon- strate the superiority of VEN- in generating pro- tein, shown in Figure 3.
Diverity Designabilty
FrameDiff
53.58%
76.42%
77.41%
87.04%
76.67%
scTMa.5
VFN Diff
67.04%
81.23%
83.95%
92.84%
83,83%
FrameDiff VEN-Diff
10.62%
23. 23.4
28. 18.02%
37. 37.78%
26.42%
scRMSD,
25.93%
40.00%
44.20%
56.30
40.25%
selected
Diversity
FrameDiff Diff
51.98%
74.57%
75.56%
85.43%
74.94%
66.54%
801.49 %
83.33%
90.61
82.59%
pdbTM,,7
FrameDiff
5
30
86
35
VEN-DiI
102
48
Table 4: Experimental results protein structure diffusion assessing the designability and diversity VFN-Diff. 'scTMo.5 and 'scRMSD, represent the percentages of generated proteins with scTM 0.5 and scRMSD < 2, respectively. 'pdbTMo.r signifies the count of generated proteins with pdbTM < 0.7, measuring the novelty of the generated protein. For more details on metrics, please refer to appendix A.4.3.
CONCLUSION
By introducing VFN, we address the atom representation bottleneck though the vector field oper- ator. enhancing its capacity for modeling frames. We demonstrate VEN's expressiveness through comprehensive experiments de novo protein design. Significant improvements or sfate-of-the--e performance achieved in protein diffusion and inverse folding tasks.

---

Published as a conference paper at ICLR 2024
Ethics Statement We do not foresee any obvious undesirable ethical social impacts
ACKNOWLEDGEMENT
This work was supported by National Key R&D Program China (No. 20222D0118700, The authors would like to thanks Hangzhou City University for accessing its GPU cluster.
REFERENCES
Ivan Anishchenko, Samuel J. Pellock, Tamuka M. Chidyausiku, Theresa A. Ramelot, Sergey Ovchin- nikov, Jingzhou Hao, Khushboo Bafna, Christoffer Norn, Alex Kang, and et al. Bera, Asim K. De novo protein design by deep network hallucination. Nature, 600(7889):547-555 2021.
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang. Qian Cong. Lisa N. Kinch, and et al. Schaeffer, R. Dustin. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):771-876 2021.
Federico Baldassarre, David Menéndez Hurtado, Arne Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinformatics, 37(3): 360-366, 2021.
Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, T. N. Bhat, Helge Weissig, llya N. Shindyalov, and Philip E. Bourne. The Protein Data Bank. Nucleic Acids Research, 2811)233--24, 2000. 18
Stephen K. Burley, Helen M. Berman, Gerard J. Kleywegt, John L. Markley, Haruki Nakamura, and Sameer Velankar. Protein data bank (pdb): the single global macromolecular structure archive. Protein Crystallography: Methods and Protocols, pp. 627-641, 2017. 19
Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin-Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence (Id)-fold (3d) embedding-basee generative model for protein design. In Proc. Int. Conf. Mach. Learn., pp. 1261-1271, 2021.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J. Ragotte, Lukas F. Milles, Basile IM Wicky, Alexis Courbet, Rob J. de Haas, and et al. Bethel, Neville. Robust deep leamnig-bbseed protein sequence design using proteinmpnn. Science, 378(6655):49566, 2022.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolu- tional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046 4051444044444,,,,, 2018.
Baldwin Dumortier, Antoine Liutkus, Clément Carré, and Gabriel Krouk. PeTriBERT: Augmenting bert with tridimensional encoding for inverse protein folding and design. bioRxiv, pp. 2022-08, 2022. 3
Cong Fu, Keqiang Yan, Limei Wang, Wing Yee Au, Michael McThrow, Tao Komikado, Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and Shuiwang Ji. A latent diffusion model for protein structure generation, 2023. 3
Fabian Fuchs, Daniel Worrall. Volker Fischer, and Max Welling. transforrnsoommrs::: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020. 4
Zhangyang Gao, Cheng Tan, and Stan Li. PiFold: Toward effective and efficient protein inverse folding. Int. Conf. Learning Representations, 2022. 3, 7, 8, 18, 19
Zhangyang Gao, Cheng Tan, and Stan Li. Ke-Design: Pushing the limit of protein deign via knowledge refinement. Proc. Int. Conf. Learning Representations, 2024. 3
10

---

Published as a conference paper at ICLR 2024
Alex Herbert and M. Stembberg, Maxcluster: tool for protein structure comparison and clustering, 2008. URL http: /www.wwww....o........... sbg. bio. ic. ac. maxclusteel 9, 19
Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. arXiv preprint arXiv:2205./5675. 2022. 3
Pedro Hermosilla, Marco Schäfer, Matej Lang, Gloria Fackelmann, Pere-Pau Vázquez, Barbora Kozlikova, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrisicc convolution and pooling for learning on 3d protein structures. In Proc. Int. Conf. on Learning Representations, 2020. 3
Brian Hie, Salvatore Candido, Zeming Lin, Ori Kabeli, Roshan Rao, Nikita Smetanin, Tom Sercu, and Alexander Rives. A high-level programming language for generative protein design. bioRxiv, 2022. 18
Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: deep convolutional neural network for mapping protein sequences folds. Bioinformatics, 1295):1295-3333,, 2018. 3
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In Int. Conf. Machine Learning, pp. 8946-89 8970, 2022. 8,8 8
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:2/06.09685, 2021.
Bin Huang, Yang Xu, Xiuhong Hu, Yongrui Liu, Shanhui Liao, Jiahai Zhang, Chengdong Huang, Jingjun Hong, Quan Chen, and Haiyan Liu. A backbone-centree energy function of neural networks for protein design. Nature, 602(7897):523-3)
2022. 3
Po-Ssu Huang, Scott E. Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537(7620):320-322 2016. 2
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Proc. Advances in Neural Information Processing Systems, 32, 2019. 3,
Michael Jendrusch, Jan Korbel, and Kashif Sadiq. AlphaDesign: de novo protein design framework based AlphaFold. bioRxiv, pp. 2021-10, 2021. 3
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Antibody-antigen docking and design via hierarchical equivariant refinement. arXiv preprint arXiv:2207.06661, 2022. 4
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. Int. Conf. on Learning Representations, 2020. 3, 8. 18, 19
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, and et al. Potapenko, Anna. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588
2021. 1, 2, 4,
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of Chemical Information and Modeling, 60X11):5667-55681 2020. 3
Jiahan Li, Shitong Luo, Congyue Deng, Chaoran Cheng, Jiaqi Guan, Leonidas Guibas, Jian Peng, and Jianzhu Ma. Directed weight neural networks for protein structure representation learning. arXiv preprint arXi::2201.13299 2022. 3
Zhixiu Li, Yuedong Yang, Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction of profiles of sequences compatible with protein strueture by neural networks with fragment-base based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82(10): 2565-2572 2014.
11

---

Published as a conference paper at ICLR 2024
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel-ZZrandi,, Tom Sercu, Sal Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with language model. Science, 2023. 7,7
Yi Liu and Brian Kuhlman. RosettaDesign server for protein design. Nucleic Acids Research, 34: W235-W238, 2006.
Matt McPartlon, Ben Lai, and Jinbo Xu. A deep SE(3)-equivarian model for learning inverse protein folding. bioRxiv doi: 11100/2022.04.154484442 2022.
Christine Orengo, Alex Michie, Susan Jones, David Jones, Mark Swindells, and Janet Thornton. Cath-a hierarchic classification of protein domain structures. Structure, 5(8):1093-1109, 1997. 7, 18
Sergey Ovchinnikov and Po-Ssu Huang. Structure-based protein design with deep learning. Current Opinion in Structural Biology, 65:136-144, 2021.
Raghav Shroff, Austin Cole, Barrett Morrow, Daniel Diaz, Isaac Donnell, Jimmy Gollihar, Andrew Ellington, and Ross Thyer. A structure-based deep learning framework for protein engineering. bioRxiv, 2019. 3
Cheng Tan, Zhangyang Gao, Jun Xia, and Stan Li. Generative de novo protein design with global context. Proc. Int. Conf. Acoustics, Speech, and Signal Processing, 2022. 8
Raphael John Lamarre Townshend, Martin Vögele, Patricia Adriana Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon M Anderson, Stephan Eismann, et al. Atom3d: Tasks on molecules in three dimensions. In Thirty-fifth Conf. on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Michel van Kempen, Stephanic Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron L.M. Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature Biotechnology, 2024. 20
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. Proc. Int. Conf. Learning Representations, 2018.
Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. In Proc. Int. Conf. Learning Representations, 2022a. 4
Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. Comenet: Towards complete and efficient message passing for 3d molccular graphs. Advances Neural Information Processing Systems, 35:650-664, 2022b. 3
Joseph Watson, David Juergens, Nathaniel Bennett, Brian Trippe. Jason Yim, Helen Eisenach, Woody Ahern, Andrew Borst, Robert Ragotte, Lukas Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, pp. 1-3, 2023. 2, 3,
Zachary Wu, Kadina F. Johnston, Frances H. Arnold, and Kevin K. Yang. Protein sequence design with deep generative models. Current Opinion in Structural Biology, 65:18-27, 2021. 3
Jason Yim, Brian L. Trippe Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. SE(() diffusion model with application to protein backbone generation. Proc. Int. Conf. Machine Learning. 2023. 2, 3, 7, 7, 9, 18, 19,20 20
Yuan Zhang, Yang Chen, Chenran Wang, Chun-Chao Lo, Xiuwen Liu, Wei Wu, and Jinfeng Zhang Prodconn: Protein design using a convolutional neural network. Proteins: Structure, Function, and Bioinformatics, 88(7):819 19-889, 2020. 3
Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In Proc. Int. Conf. Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2024
Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure-informed language models are protein designers. In Proc. Int. Conf. Machine Learning, 2023. 1, 3, 8
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. Proc. Int. Conf. Learning Representations, 2023. 19
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A. THE COMPARISON BETWEEN VEN AND IPA
In this section, we first present the pscudocode for IPA and VFN to provide intuitive comparison, as shown in Subsection A.1.1. Subsequently, we elaborate on the differences between IPA and VFN, outlined in Subsection 1.1.2. In order to further elucidate the atom representation bottleneck, expound on how VFN addresses the bottleneck in Subsection A.1.3. Finally, in Subsection A.1.4, introduce the pipeline of IPA facilitate our readers' understanding of the specific processes involved in IPA.
A.1.1 THE PSEUDO-CODE FOR VEN AND IPA
In this section, we initially compare IPA with VFN in the provided Algorithm and below and explain the meaning the notation. Please note that the following code is provided for the convenience of our readers to distinguish between IPA and VFN's core differences. The specific implementation of VEN may vary across different tasks and may differ from Algorithm 2. Please refer the main text for the exact implementation.
Algorithm The pseudo-code for the IPA mod-
Algorithm The pseudo-code for the VEN mod- ule.
ule.
1:
der der IPA((ss: (T.))
der (FN((s:), (eg), (Tr-)):
2:
qa, k., v; Linear(s:)
3:
qu, ku, Vu Linear(s.)
2:
qu Linear(s,)
4:
qa, ku, Vu + T, {Ta, Ka, VuF
3:
K₁ Tins o qst
5:
point = E, | kjrll
4: h Σ. whaq + D kark
gij = concaty
hi [
RBF ( (i)ll)
0i
q. k,
7:
edge
Linear(e:;)
8:
aij softmax, (a. 0ij + Poinn))
odge
6:
0. softmax, (ML.P(s 83. Bij, ei.))
7: ViJ MIP(s),.B.,,
9:
node ij
E, ai.V
8: O, Σ, EjaiVis
alge
E, dij
1j
Gpain T.² E, av. vzi
12:
+- Linear(o
edge oodde @
9:
MLP(o.)
10: ei < MLP Bj.
13:
return {s:}
11: return {si), (es,5}
In the aforementioned pseudocode, largely adhere the notations used in the main text, with some distinctions certain notations. Specifically, qu denotes the I-th feature vector of the i-th node, and K3 follows the same convention. T, represents the transformation matrix from the i-th local frame (residue frame) to the global frame, while T signifies the transformation matrix from the global frame to the i-th local frame. T.-j denotes the transformation matrix from the j-th local frame to the i-th local frame. s; represents the representation of the i-th node, and eij represents the representation of the edge between i-th and j-th nodes. wi, and wh are learnable weights. The specific process of VFN is detailed the main text; the procedure for IPA elaborated Subsection A.1.4. For the sake of conciseness in comparison, the pseudocode overlooks certain non-essential factors. For instance, in the pseudocode, we illustrate a single-head attention mechanism, but in fact, both IPA and VFN employ multi-head attention mechanism.
14

---

Published as a conference paper at ICLR 2024
A.1.2 THE DIFFERENCES BETWEEN IPA AND VEN
As indicated in the pseudocode, the attention mechanism for virtual atoms in IPA and VFN is fundamentally different. The most crucial distinction lies the fact that, due to the constraints of SE(3) invariance, IPA cannot directly employ an activation function when extracting features for virtual atoms, as explained in Subsection A. 1.3. In contrast, VFN circumvents this limitation and utilizes a vector field operator to extract features, denoted as hg. and complementing it with an MLP (ReLU inside) for feature extraction. To accommodate this design choice, the overall architecture of VFN diverges significantly from that of IPA.
A.1.3 BYPASSING IPA BOTTLENECKS
The design of IPA has led to the atom representation bottleneck, while the design of VFN avoids this issue. Specifically, in IPA, and kj are with respect to the global frame, and the operator corresponding q. k, needs maintain SE(3) invariance. This constraint results in the inability to directly apply activation functions on qú kj. as doing so would compromise the SE(3) invariance. Consequently, IPA can only employ operations similar to distance pooling. We refer to this limitation as the atom representation bottleneck.
In VFN, we place these virtual atoms in the same local frame T, utilizing the local frame to ensure SE(3) invariance, as proved in Subsection A.2.6. This eliminates the need to impose constraints on operators to achieve SE(3) invariance. Operators in VFN can freely utilize activation functions without disrupting SE(3) invariance. This characteristic allows VFN to circumvent the atom representation bottleneck present in IPA.
A.1.4 THE PIPELINE OF IPA
As shown in Algorithm IPA employs attention mechanism. Unlike VFN, its attention mechanism consists of three parts: node attention, edge attention, and point attention. Specifically, node attention and edge attention utilize common methods. In particular, node attention employs mechanism similar to that of the transformer, obtaining attention weights modle through dot product of q: and odge through a linear ky. In edge attention, the representation of edges e,y introduces attention bias o1
layer. Point attention the core of IPA, where attention weights ogoina are obtained through distance pooling operation on the virtual atomic distances (Ilqa- 1 [). Subsequently, these attention weights are summed and normalized through softmax to obtain the final attention weights as,j- These attention weights di.j are then used to calculate weighted averages for the corresponding representations Vi, eij, Vst producing the output for each type of attention. ofi 0 edge apriin Finally, these outputs are collectively updated for each node's 's representation s; through linear layer, achieving the frame modeling.
A.2 METHODOLOGY
A.2.1 MULTI-LAYER VECTOR PERCEPTRON MODULE
In this section, we elaborate on the process of the multi-layer vector perceptron module, V-MLP, as illustrated in Algorithm 3:
15

---

Published as a conference paper at ICLR 2024
Algorithm 3 The pseudo-code for the multi-layer vector perceptron module.
1: def V-MLP(Q); Qú:
2: w I 1.I.,.11..... w {wg, E R...,.l....
# initialize learnable weight w w
3:
V = Ei 1 + Ei wh, qi"
VA R q E Q q Q
4:
tk [we Illvel
W Vk
We R (we vx) E
We are learnable weights.
5:
w = (wen, E {I] [m.
# initialize learnable weight w
6:
cm = Ex Wm,,k 2ik
cm E R3
7:
return {Em 1m=1.....dd
A.2.2
IMPLEMENTATION OF VEN-DIFF
edge stth
edge aton
cdge attn
edge attr
node attn
node attn
node attn
node attr
point attn
point attn
point attn
VFN
Figure In the context FrameDiff (A) and VFN-Diff (B), the traditional point attention module in FrameDiff has been replaced with a VFN layer, introducing enhanced geometric feature extraction capabilities
We have implemented VFN-Diff based on FrameDiff, as shown in Figure 6. Apart from modifications to the structural encoder, IPA, other parts remain unchanged. Within IPA, attention divided into three components: node attention, edge attention, and point attention. In the case of VFN-Difi we replace the point attention component in IPA with the VFN layer, while keeping the remaining portions unaltered. Despite the removal of node attention and edge attention in VFN-Diff, the model still functions effectively, achieving performance comparable to FrameDiff. However, is important that, this stage, VFN-Diff's parameter count only one-ninth that of FrameDiff. Clearly, such a comparison would be unfair. Therefore, we retained the additional parameter-rich "vanilla" components, node attention and edge attention. This adjustment brings the parameter count of VFN-Diff close to that of FrameDifi (18.3 million vs. 17.4 million), facilitating a fair comparison. Moreover, node attention and edge attention are well-established practices, widely adopted previous works such as PiFold. Consequently, these modules do not constitute the core of IPA; rather, point attention does. Hence, the replacement of point attention with the VFN-Diff approach represents rigorously comparative implementation.
A.2.3 IMPLEMENTATION OF VFN-IF
We have adopted the PiFold framework implement VFN, consisting of three components: the input layer, network layer, and decoder. Concerning the decoder and corresponding loss functions, VFN remains consistent with PiFold. For the network layer, we have substituted PiFold's layer, PiGNN,
16

---

Published as a conference paper at ICLR 2024
RAAK
*
WODILWSIOORRRSS
ESM
PETORYWARIK RVVLIEMARGYD
VFN-IF
VEEAYNKYL
LoRA
ELDALAB ABALAAELRALVAAR RALLRALLLLLAAE ARALLAAAPAALEA
AAPALARLLG
protein file
Sequence
Figure 8: Flowchart depicting the fine-tuning process of frozen VFN-IF and ESM with LoRA.
with the VFN layer. However, to ensure a fair comparison, the Global Context Attention module from PiFold has also been retained in VFN-IF.
Regarding the network in- put layer, PiFold employs manual featurizer to ex- tract interatomic features such as angles and dis- tances. In VFN, we have completely removed this featurizer since the VFN layer itself extract ge- ometric features without re- lying on featurizer. We initialize a portion of vir- tual atomic coordinates us- ing known backbone atomic coordinates to provide the network with this information.
VPN laper
VFN
VEN
kyer
Sequenee
Rackbone Arom
Figure We harnessed the capabilities of VFN layers effectively extract both frame and backbone atom information, facilitating the generation sequences for inverse folding.
A.2.4 IMPLEMENTATION OF VEN-IFE
In VEN-IFE, the ESM model is employed to refine the predictions made by VEN-IF. During training. the pretrained VFN-IF frozen, and its predictions are used input the ESM model (as described later). The ESM model is initialized with pretrained weights and then frozen. We leverage LoRA provide learnable weights and fine-tune the ESM. ESM is supervised with a cross-entropy loss using ground truth amino acid labels. The most critical aspect of our approach is the introduction of relaxation method, allowing the ESM to accept probability-bass
inputs from VFN-IF.
For i-th amino acid, VFN-IF predicts a set of probabilities corresponding to 20 amino acid categories, denoted as p {0 < Pj < 1) Similarly. ESM includes word embeddings corresponding these categories, denoted as W WESM [w, R. /=o.....00- Therefore, we use the predicted probabilities p from VFN to perform a weighted sum of the corresponding word embeddings W Wis for producing the input token a ESM to ESM, denoted as:
ESM Σ EpW,
(10)
A.2.5 LOCAL FRAMES OF RESIDUES
In VFN, a local frame is set up for each residue via (Jumper et al., 2021), refer to rigidFrom3PPints algorithm in their paper.
process proposed by AlphaFold2
17

---

Published as a conference paper at ICLR 2024
A.2.6 PROOF OF SE(3) INVARIANCE
The output and the vector representation of VFN are SE(3) invariant, which are crucial for networks to attain higher performance. The proof is simple. In short, the local frame of residues is SE(3) equivariant, which ensures the invariance for the inputs of VFN and the outputs of vector field operator. Here, we provide the proof following:
The SE(3) invariance of initial representation. Due to all the initial representations are with respect to the local frame, the input of VFN, 8; o, 9,, is SE(3) invariant.
The SE(3) invariance of the vector field operator. In our main paper, the transform matrix Tą-5 is employed in our operators (refer to Eq. equation in our main paper):
K
T T o Q3 where T T o T
(11)
The transform matrix T is SE(3) invariant w.r.t the global reference frame, because the global frame cancels out in the computation of the transform matrix Tz-;:
(T global T,) (T global TTy T globall T Tglobal T
(12)
T Ty
where T global denotes any global reference frame. Therefore, the outputs h Bi, of the vector field operator are SE(3) invariant.
The conclusion of SE(3) invariance. Finally, because the input of VFN and the vector field operator (Only these operators associated with the global reference frame) are SE(3) invariant, the intermediate variables and outputs of VFN satisfy SE(3) invariance.
A.3 DE NOVO PROTEIN DESIGN USING VFN
In this section, we explored de novo protein design through the utilization of two distinct pipelines. The first pipeline involves the established flow of FrameDiff coupled with Protein MPNN, while the alternative approach adpots VFN-Dit in tandem with VFN-IFE. Both pipleines employed the ESMFold (Hie et al., 2022) to fold amino acid sequences into ptrotein Seuuturrs..emarkeee one-shot experiments with Noise Scale parameter set to 1.0, our VFN-based pipeline outperformed Prattein MPNN approach in terms of designability, as shown in $A.4.3.
in
Frame Diff
Protein MPNN
SELDELL JAII ARPREFI FAFT ME BLWGRICRVABSALII
ESMFold
FSEL REEDRLALLVANEE
VFN- DIFF
VFN- IFE
Figure 9: Whole pipelines
A.4 EXPERIMENT
A.4.1 DATASET DETAILS
Protein diffsuion. FrameDiff was re-trained using the same standards as in (Yim et al., 2023), while VFN-Diff was trained on four NVIDIA 4090 GPUs for a total duration
days and hours. The training dataset consisted of proteins from the PDB database (Berman et al., 2000) in August 2023, encompassing 21,399 proteins with lengths ranging from 60 to 512 and a resolution of < SÄ.
Inverse folding. Unless specified, our experiments conducted the CATH 4.2 (Orengo et al., 1997) dataset using the same data splitting as previous works such as GVP (Jing et al., 2020) and PiFold (Gao et al., 2022). The dataset consists of 18,024 proteins for training, 608 for validation, and
18

---

Published as a conference paper at ICLR 2024
Setting
Noise Scale
1.0
0.5
0.1
0.1
Number Steps
500
500
500
100
Metric
Number Sequences
8
8
8
FrfmeDiff + ProteinMPNN
53.58%
76.42%
77.41%
76.67%
Designabiity
sc f
VFN-Diff + ProteinMPNN
67.04%
81.23%
83.95%
83.83%
VFN-DifI' + VEN-IF
72.84%
91.60% %
93.46 %
90.49%
FrameDiff + ProteinMPNN
10.62%
23.46%
28.02%
26.42%
scRMSD2 +
VEN-DifI + ProteinMPNN
25.93%
40.00%
44.20%
40.25%
VFN-Diff + VFN-IF
26.79%
53.33% %
58.27 %
51.36%
Diversity
Frfme + ProteinMPNN
51.98%
74.57%
75.56%
74.94%
Diversity
VFN-Diff + ProteinMPNN
66.54%
80.49%
83.33%
82.59%
VFN-Diff + VEN-IF
69.75%
86.91 %
87.03 %
85.43 %
Table Comparison the complete pipeline. All settings are aligned with Table in the main text. Here, VFN-IF adopts the settings of ProteinMPNNN.
1 1120 for testing. During the evaluation, we also test our model on two smaller datasets, TS50 and TS500 (Jing al., 2020; Gao al., 2022), validate the generalizability. Furthermore, we also create another larger training set by incorporating data from the PDB (Burley et al., 2017). We apply the same strategy as (Zhou al., 2023) to collect and filter structures. Additionally, the proteins with sequences highly similar to test set proteins are also removed. By using the expanded dataset, we are able to scale up the VFN-IF.
A.4.2 IMPLEMENT DETAILS
VFN-Diff. Our training regimen employs the Adam optimizer with the following hyperparameters: learning rate of 0.0001, 81 set to 0.9, and 32 set to 0.999. We follow the setting of FrfmeDiff. For more implement details, please refer thier paper.
VFN-IF. VFN-IF are trained with batch size and are optimized by Adam with weight decay of 0.1. We apply a OneCycle scheduler with a learning rate of 0.001 and train our model for total of 100,000 iterations.
VEN-IFE. We constructed VFN-IFE by employing 15B ESM model in conjunction with the standard VFN-iF. The rank for LoRA applied to ESM was set to 8. All other training settings remained consistent with those VFN-iF.
A.4.3 METRIC
Protein diffusion inference. Inferences were conducted with protein lengths ranging from 100 to 500, using step size of 5. This resulted in the generation of 10 diffusion samples of protein at each step, with a total of 810 diffusion samples, denoted as Nasff. 'Noise Scale' represents the initial noise scale in diffusion, while 'Num. Step' represent the diffusion steps during inference. Protein MPNN was employed to generate 'Num. Seq.' (Neeg) sequences for each sample, followed by ESMFold to create protein structure files, referred to as Nesm = Nais X Nseq-
Protein diffusion Metries. Metrics of protein diffusion inference experiments can be categorized into the following sections:
Structural Similarity Metrics: This metric evaluates the proportion of samples with structural consensus TM score (scTM) and RMSD meeting the criteria scTM > 0.5 and scRMSD < 2 A, indicated in Table 4 by scTMM.. and scRMSD, In this context, scTM measures the structural similarity between ESMFold generated proteins and diffusion generated structures, while scRMSD quantifies the not-mean-square deviation in atomic positions between these structures.
Diversity: To gauge the diversity of the generated protein sequences, we adhered to the methodology laid out in (Yim et al., 2023) and leveraged MaxCluster (Herbert & Sternberg, 2008) for hierarchical clustering of protein backbones. However, we opted for a higher threshold of 0.6 TM-score, in contrast to the 0.5 T'M-score referenced in the literature,
19

---

Published as a conference paper at ICLR 2024
impose rigorous clustering criterion. Furthermore, enforeed selection criterion for cluster inclusion, requiring scTM 0.5, with the intention of mitigating the impact of proteins with low designability a singular protein generated through ESMFold was chosen based on the highest scTM score. The diversity metric was computed the ratio of the number of clusters to Naiff- These experimental modifications were introduced to yield results that are not only more scientifically sound but also align more closely with anticipated trends within the inference data.
diversity assessments. For each diffusion sample,
PdbTM: To assess protein novelty, we compared the ESMFold ESMFold-generrtee PDB files, each containing the protein with the highest scTM score from diffusion sample, to the PDB database using the Foldseek tool (van Kempen et al., 2024). The highest TM-scores of the generated samples were compared with any chain in the PDB database, and the resulting value was denoted as pdbTM. To exclude proteins with limited designabilityy we applied cutoff criterion of RMSD pdbTM served as a robust metric for quantifying protein novelty, reflecting the structural similarity between the generated proteins and those documented in the PDB database. In contrast to the threshold of pdbTM < 0.6 employed in (Yim et al., 2023), we considered proteins with pdbTM values less than 0.7, which refers pdbTM,.7 in Table as novel designs due to their substantial dissimilarity from existing proteins, resulting in the inclusion of greater number of novel proteins in both selection.
consistent with the approach used in (Yim al., 2023).
Inverse folding. We conducted a comparative analysis involving VEN-IF, VIN-IFE, Pifold, and LM-Design. We utilized the CATH 4.2 dataset for validation and compared the results by performing ESMFold the generated sequences (with N Nseq 1) and comparing them with the original PDB files to calculate scTM and scRMSD. This analysis was performed as a one-shot comparison.
Whole Pipeline. This set of experiments compared VFN-Diff + VFN-IFE with FrameDiff + Protein- MPNN. To ensure alignment of comparison standards, we used N Nseg = 1 and Noise Scale = 1.0 for one-shot comparisons.
1.0
FrameDllf WINNFFE
x
20,5000
0.5
8.5
0.4
5.4
100
150
200
230
300
350
400
430
500
Pramectff MPAN
VEN-IIT VPNIFF
Ieagth
Rpeins
Figure 10: Whole pipeline in one-shot: FrameDDif+PPrttiiMMMNN and VEN-DIII-VNN-IFE
A.4.4
ABLATION STUDY
A.5 ABLATIONS
In this section, we carefully investigate the design choices of vector field modules proposed here based CATH 4.2 (inverse folding).
The number of layers. We investigate the impact modifying the number of layers on recovery and perplexity in Table 6a. Increasing the number of layers from 5 to 15 results in marginal improvement
20

---

Published as a conference paper at ICLR 2024
VEN-IF
2.0
0.8
0.6
04
core
surface
loop
Figure 11: VFN PiFold different structural contexts
Table 6: Ablation studies on the CATH 4.2 dataset. We use the default model settings unless otherwise specified. When calculating the number of parameters, we only count the number of parameters occupied by this module in one layer.
(a) Varying the number of layers.
#layors
5
10
12
15
Recovery(%) Perplexity
52.53
54.12
54
54.3
54.08
edge feature
4.3114
4.1986
4.1883
4.1536
4.2185
Recovery (%)
52.68
53.78
54.28
54.11
54.7
w/o edge feature
Perplexity
4.3192
4.1983
4.1766
4.1829
4.1687
(b) Varying the number of vectors.
#Vec.
16
32
64
Recovery (%)
53.97
54.28
53.63
Perplexity
4.1598
4.1766
4.2636
(c) Varying the V-MLP.
(d) Varying the vector field.
Recovery (%)
Params
Recovery (%)
Perplexity
V-MLP
54.28
4.2K
Baseline
54.28
4.1766
MLP
53.42
36.0K
w/o Vec. field
35.43
7.5455
Wo V-MLP
53.67
0.0K
w/o gh.5 in Eq. equation 7
53.05
4.2907
(e) Varying the transformation T.
Ghf
h
Transformation
Recovery($$)
Perplexity
54.28
4.1766
53.55
4.2495
V
/
53.36
4.2238
X 52.89
4.3268
in recovery scores, with the highest recovery achieved at 54.72% for 12 layers. Ablation experiments without the edge featurizer show that with or without edge features, the performance is comparable. Especially when the number layers reaches 15, even achieves better results, indirectly proving the effectiveness of VEN-IF and its potential for reducing reliance hand-crafted features.
21

---

Published as a conference paper at ICLR 2024
The number of vectors. In Table 6b also perform ablation study determine the optimal number for vectors. It demonstrates that increasing the vector number from to 32 brings a slight improvement in recovery 4.1 However, further increasing the number to 64 results in a decrease in recovery score to 53.83% and higher perplexity
from 53.72% 54.26%, while maintaining low perplexity score of
of 4.20. Overall, these findings suggest that 32 is more suitable choice.
V-MLP. In Table 6c, we observe that using V-MLP outperforms using a regular MLP or not using all. Compared to using regular MLP, V-MLP significantly reduces the parameter count.
Vector field design. Results Table 6d show that removing the whole vector field operator leads signifieant drop in recovery, indicating its importance in capturing protein folding patterns. The incorporation of the gi,j in the edge aggregation module also has a substantial effect on the performance of the model.
Vector field operator design. We validate the effectiveness RBF and / as shown in Table 6e. Furthermore, it illustrates that completely excluding the Eq. equation 3 and directly flattening the vectors H into gi,j leads to an obvious performance decrease.
Eq. equation
A.5.1 VISUALIZATION RESULTS
In this section, we visualize some of the protein structure restoration results of VFN-IF, shown in Figure 13, Figure 14 and Figure 12.
VFN
Experiment
PiFold
r.m.dd. 1.820 A
Recovery 51.2 %
r.m.s... 3.227 A
Recovery 39.7
Figure 12: Visualization results of a challenging sample (PDB 2KRT). We use AlphaFold2 to recover the structure based on the predicted sequence and compare against the experimentally determined ground-truth structure.
22

---

Published as a conference paper at ICLR 2024
ID2I
1GT7 recevery 54.37
r.m.sd. 0.317
recovery 54.70 %
r.m.sd. =
IWTD
2BMO
r.m.dd. - 1.846 A
recovery -4.90 %
r.m.s.d 0.403
recovery 52.06 %
2KIN
2LT5 recovery 49.17%
r.m.d. 0.453
recovery 50.00
r.m.d.
VFN
Experiment
Figure 13: The visualization result of VFN.
23

---

Published as a conference paper at ICLR 2024
2LWY
2R41
r.m.sd. 0.827
recovery 55.225 %
r.m.d. 0.730 A
recovery 48.78 %
3F7S
3GZB
r...d. 0.629 A
recovery 50.00
r.m.s.d. 0.575
recovery 50.00 %
411R6
4U13
r.m.s.d. 0.544 A
recovery 52.47
r.m.s.d. 0.457 A
recovery 49,54 %
VFN
Experiment
Figure 14: The visualization result of VFN.
24


Reviews:
[{"review_id": "xTJL6w5eCo", "reviewer": "Reviewer_unsL", "summary": "The authors introduce a new architecture (VFN) for processing protein structures, which allows to represent residue positions with many virtual atoms. This allows for more fine-grained modeling of residue interactions. The proposed VFN architecture is shown to outperform standard architectures in both protein generation using diffusion models and inverse folding tasks.", "questions": "I would mainly like the authors to provide a more detailed theoretical and/or experimental analysis of the introduced atom representation bottleneck, as I mentioned in the weaknesses.\n\n\n-------\n### After Rebuttal\nThank you for the extensive rebuttal. I read through all the reviews and all the answers and I think the work has noticeably improved.\nIt's a bit strange the authors have not updated the paper itself with all the new results, but I trust that they will for the final version.\nI now recommend acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I agree with the authors that there has been an over-reliance on IPA in the literature for protein tasks. It makes a lot of sense to investigate improvements to it, so the paper does target a very important problem in my eyes.\nIntroducing effectively more data channels into the model to increase its capacity is also very sensible. Importantly the model is shown to improve the results on the most common and important protein modeling tasks.", "weaknesses": "The general reasoning of why the proposed architecture works better and should be constructed the way it is lies on the concept of atom representation bottleneck. But this bottleneck is not really introduced or investigated in a rigorous manner. Maybe the authors can at least give concrete theoretical counter examples of what problem can be modeled with VFN but not IPA. At least an experimental ablation on varying the virtual node count would be interesting to see how the performance changes.\n\nIt would also be nice if authors could show the theoretical expressivity of their proposed construction. E.g. is there something it cannot model or is it provably universal.\n\nGeneral GNNs have seen some improvements on modeling equivariant structures, e.g. [1, 2, 3] which in some ways have some similarities to the current work (e.g. higher dimensional embedding in frame averaging is a bit like virtual nodes here). It would be nice to see how the proposed VFN architecture stacks up to those general GNN constructions. Although admittedly, those papers usually test their models on molecular and other physics-inspired 3D tasks, but not proteins tackled in this paper. However the extension, especially in case of frame averaging or the multi-channel EGNN, would be trivial. Note that there are also many more improved 3D GNNs, especially in molecule domain that could be applied to this problem. I would like to see the authors test against at least a few of these options, especially frame-averaging as it has been used for proteins a couple of times now [4, 5] and would tackle a similar problems as the proposed model in a very general way.\n\nSpeaking of proteins, in the abstract authors say that only basic encoders such as IPA have been proposed for proteins so far. E.g. [4] applies frame averaging [2] to antibodies, with a quite intersting non-relational architecture for antibody design. While its restricted to a certain protein family it's still worth mentioning that 'less simple' encoders do exist, at least in specific cases.\n \n[1] Du, Weitao, et al. \"A new perspective on building efficient and expressive 3D equivariant graph neural networks.\"\n\n[2] Puny, Omri, et al. \"Frame averaging for invariant and equivariant network design.\" \n\n[3] Levy, Daniel, et al. \"Using Multiple Vector Channels Improves E (n)-Equivariant Graph Neural Networks.\"\n\n[4] Martinkus, Karolis, et al. \"Abdiffuser: Full-atom generation of in-vitro functioning antibodies.\"\n\n[5] Jin, Wengong, et al. \"Unsupervised Protein-Ligand Binding Energy Prediction via Neural Euler's Rotation Equation.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NLbx2AB3RB", "reviewer": "Reviewer_MXeu", "summary": "This paper proposes vector field network to model to better model local frames. Building on this VFN, this paper constructs two sequential models, respectively targeting at designing protein structures and generating protein sequence based on given protein backbone structure.  The paper achieves a new SOTA score on CATH 4.2 on inverse folding task and performs better than FrameDiff on protein structure design.", "questions": "1. In figure 2, $T_{i\\rightarrow j}$ means T_j to T_i, while in line below Equation 1, it means T_i to T_j. What does this term really mean?\n\n2. Why he range of H can be negative, say -200 A?\n\n3. $d_q$ is the number of channels in $g_{i,j}$. Does that mean the number of virtual atoms between node i and node j? What is the specific value used in this paper?\n\n4. Are the MLP in equation 7 and 4 are the same one?\n\n5. Using the ESM as initialization and then testing the model on CATH benchmark may have data leakage issues. How did the author deal with this problem?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed model achieves a new SOTA score on the CATH 4.2 benchmark.", "weaknesses": "1. **The architecture design lacks some novelty:** It seems for the protein structure design part (VFN-Diff) borrows some ideas from FrameDiff, while for the inverse folding part (VFN-IF), the virtual atom is similar to that of PiFold and the node interaction (Equation 4, 5, 6) is similar to the node gating mechanism in PiFold.\n\n2. **The problem setting is unfair:** In the first paragraph of section 4, the author mentioned \"In the protein diffusion part, the protein structure is designed and represented using backbone frames T. Subsequently, these backbone frames are fed into the inverse folding network to obtain the corresponding protein sequence for the designed structure.\" If I didn't understand wrongly, this means the author first design the protein structure  and then generate protein sequence based on the designed structures.  Therefore, it's kind of like a pipeline. However, in Table 1, for the sequence design task, the author only compared with the inverse folding models, say sequence design based on real structure instead of designed structures. I think a more fair comparison should compare to baseline like structure design model plus inverse folding model, such as FrameDiff + ProteinMPNN, RFDiffusion [1]+ProteinMPNN and also some structure-sequence co-design model like ProtSeed [2]. \n\n[1] De novo design of protein structure and function with RFdiffusion. Nature 2023.\n\n[2] Protein Sequence and Structure Co-Design with Equivariant Translation. ICLR 2023.\n\n3.**Lack of baselines:** The paper lacks some important baselines. For example, [2] for sequence part. For structure design part, the author only compares with FrameDiff, while the current SOTA protein structure design model is RFdiffusion [1]. Also, there is some other protein structure design model like SCMDiff [3], \n\n[3] DIFFUSION PROBABILISTIC MODELING OF PROTEIN BACKBONES IN 3D FOR THE MOTIF-SCAFFOLDING PROBLEM. ICLR 2023.\n\n[4] Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Clouds. ICML 2023.\n\n4. **The writing is unclear:** The author may need to use a unified annotation system, like sometimes {i, j} sometimes k is confusing while they mean the same thing. Additionally, the author may need to provide an overall graph of the architecture to help reader understand this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A7LT3OGgPV", "reviewer": "Reviewer_AKqf", "summary": "This paper presents a frame-based architecture for operating on protein structure. The key idea is to parameterize a set of virtual atoms within each residue frame. For each pair of interacting frames, the coordinates of both sets of virtual atoms in the destination frame are used to compute attention weights and message values. The aggregated message is then used to update the node features and virtual atom coordinates. In experiments, the authors swap out the architectures of PiFold and FrameDiff and achieve significantly improved performance under identical experimental settings.", "questions": "* Please describe more details on how the VFN-IF+ training split is constructed.\n* Table 5 shows results for FrameDiff+ProteinMPNN vs VFN-Diff+VFN-IFE. What was the exact setting for reporting these numbers reported here, relative to Table 4? Are there equivalent results for FrameDiff+VFN-IF or VFN-Diff+VFN-IF? Please show these for all experimental settings in Table 4.\n* In Figure 1, why does FrameDiff suffer in scTM more on medium-length proteins than the longest proteins?\n* Why was it necessary to retrain FrameDiff, as claimed in the appendix?\n* Please clarify if the PiFold and FrameDiff numbers are taken directly from the respective papers. Please affirm that the results and claims made here are indeed under *identical experimental conditions* relative to PiFold and FrameDiff, or if they have been modified, please be very direct about these modifications.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper establishes a new entry in the design space of residue frame-based architectures, an exciting direction for protein representation learning.\n* The experimental results are quite strong and establish that VFN could be used as a drop-in replacement for alternative SOTA architectures.\n* The non-exchangeable treatment of virtual atoms (unlike IPA) leaves open the possibility of using the framework for real sidechain atoms.", "weaknesses": "* The thesis of the paper would be improved by better contextualization relative to IPA. The authors should not shy away from acknowledging significant similarities, but highlight the key changes and the insights behind them. I would suggest a side-by-side algorithmic comparison.\n* The paper could be further strengthened by additional comparisons with IPA. Particularly, if we replace IPA in AlphaFold/ESMFold with VFN, does the performance persist? It should not be too hard to run this experiment since the structural module is not very large.\n* The claims about a \"universal encoder\" are not well-supported. It would be nice to see actual experiments where sidechains are also involved.\n* From the novelty standpoint, an argument can be made that the architecture is similar enough to IPA and / or PiFold to count against its technical significance.\n\nJustification for score: I think this is a good paper and am happy to recommend acceptance if the Questions are fully addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)