ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention

Authors: mingchen  li
yang  tan
xinzhu  ma
bozitao  zhong
huiqun  yu
ziyi  zhou
wanli  ouyang
bingxin  zhou
pan  tan
hong  liang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 18:34:22

Abstract: Protein language models (PLMs) have shown remarkable capabilities in various protein function prediction tasks. However, while protein function is intricately tied to structure, most existing PLMs do not incorporate protein structure information. To address this issue, we introduce ProSST, a Transformer-based protein language model that seamlessly integrates both protein sequences and structures. ProSST incorporates a structure quantization module and a Transformer architecture with disentangled attention. The structure quantization module translates a 3D protein structure into a sequence of discrete tokens by first serializing the protein structure into residue-level local structures and then embeds them into dense vector space. These vectors are then quantized into discrete structure tokens by a pre-trained clustering model. These tokens serve as an effective protein structure representation. Furthermore, ProSST explicitly learns the relationship between protein residue token sequences and structure token sequences through the sequence-structure disentangled attention. We pre-train ProSST on millions of protein structures using a masked language model objective, enabling it to learn comprehensive contextual representations of proteins. To evaluate the proposed ProSST, we conduct extensive experiments on the zero-shot mutation effect prediction and several supervised downstream tasks, where ProSST achieves the state-of-the-art performance among all baselines. Our code and pre-trained models are publicly available.

Main paper: 
 introduction: 
 Predicting the functions of proteins is one of the most critical areas in life sciences [1] . In recent decades, protein sequence databases have experienced exponential growth [2] , making it possible to learn the fundamental representations of protein sequences with large-scale models in a datadriven manner. Inspired by pre-trained language models in natural language processing [3, 4] , many pre-trained Protein Language Models (PLMs) have emerged [5, 6, 7, 8, 9] . Benefiting from remarkable protein representation capabilities, they have become fundamental tools for bioinformatics in protein-related tasks.The function of a protein is determined by its structure [10] . However, most PLMs mainly focus on modeling protein sequences, neglecting the importance of structural information, and one significant reason for this phenomenon is the lack of structural data. Fortunately, some excellent works, such as AlphaFold [11] and RoseTTAFold [12] , are proposed, which can accurately predict protein structures. These works significantly expand the protein structure dataset [13] to millions and enable the pre-training of large-scale structure-aware PLMs. After that, the major challenge is how to effectively integrate protein structure information into PLMs. Specifically, existing structure-aware PLMs [14, 15] first use Foldseek [16] to convert protein structures into discrete structure tokens and then integrate these structural data into the Transformer architecture. However, despite achieving promising performance on several tasks, this approach still faces two main issues. First, Foldseek encodes the structure of a residue within a protein by considering only the features of its previous and next residues. This representation is insufficient and may overlook subtle differences in the local structure of proteins, such as catalytic sites or binding pockets, which are crucial for protein function [17] . Second, the naive Transformer architecture lacks the ability to explicitly model the relationship between protein sequences and structure token sequences, making it challenging to effectively leverage structural cues.In this paper, we develop ProSST (Protein Sequence-Structure Transformer), a structure-aware pre-trained protein language model. Specifically, ProSST mainly consists of two modules: a structure quantization module and a Transformer with sequence-structure disentangled attention. The structure quantization module is based on a GVP (Geometric Vector Perceptron) [18] encoder, which can encode a residue structure along with its neighborhoods in its local structure and quantize the encoding vectors into discrete tokens. Compared to Foldseek, which only considers individual residues, this encoder can take into account more information from the micro-environment of residue. The sequence-structure disentangled attention module replaces the self-attention module in the Transformer model. This can make Transformer model explicitly model the relationship between protein sequence tokens and structure tokens, enabling it to capture more complex features of protein sequences and structures. To enable ProSST to learn the contextual representation comprehensively, we pre-train our model with the Masked Language Modeling (MLM) objective on a large dataset containing 18.8 million protein structures. To summarize, our main contributions are as follows:• We propose a protein structure quantizer, which can convert a protein structure into a sequence of discrete tokens. These token sequences effectively represent the local structure information of residues within a protein.• We propose a disentangled attention mechanism to explicitly learn the relationship between protein structure and residue, facilitating more efficient integration of structural token sequences and amino acid sequences.To evaluate the proposed ProSST, we conduct extensive experiments on zero-shot mutation effect prediction and multiple supervised downstream tasks, where the proposed model achieves state-ofthe-art results among all baselines. Besides, we also provide detailed ablations to demonstrate the effectiveness of each design in ProSST.2 Related Work 
protein representation models: 
 Based on the input modality, protein representation models can be divided into three categories: sequence-based models, structure-based models, and structure-sequence hybrid models.Sequence-based models. Sequence-based models treat proteins as a sequence of residue tokens, using the Transformer model [19] for unsupervised pre-training on extensive datasets of sequence.According to the pre-training objective, current models can be further divided into BERT-based models [4] , GPT-based models [3] , and span-mask based models. Specifically, BERT-style models, including ESM-series models [5, 6, 7] , ProteinBert [9] , and TAPE [20] , aim to recover the masked tokens in the training phase. The GPT-style models, such as Tranception [21] , ProGen2 [22] , and ProtGPT2 [23] , progressively generate the token sequences in an auto-regressive manner. Lastly, models that use span-mask as the training objective include Ankh [24] , ProtT5 [8] , and xTrimo [25] .Structure-based models. Protein structures play a dominant role in protein functionality. Therefore, models leveraging structure information generally get more accurate predictions. Recently, various techniques have been applied in learning protein structure representation, including CNN-based models [26] and GNN-based models [18, 27, 28, 29, 30] , where the GNN-based ones have demonstrated significant versatility in integrating protein-specific features through node or edge attributes. Moreover, the recent advancements in protein folding models [7, 11, 31] enable the structure-based models access to extensive datasets of protein structures. This led to a growing interest in developing PLMs that leverage protein structure cues [14, 15, 32] .Structure-sequence hybrid models. Hybrid models, which incorporate both sequence and structure information of proteins, offer more effective representations of proteins. For example, the LM-GVP [33] model employs ProtBERT-BFD [9] embeddings as input features for the GVP [18] model, while ESM-GearNet [34] investigates various methods of integrating ESM-1b [5] representations with GearNet [32] . Similarly, the recent ProtSSN [35] model leverages ESM-2 [7] embeddings as input for the EGNN [36] model, resulting in notable advancements. Both ESM-IF1 [37] and MIF-ST [38] target inverse folding, utilizing the structure to predict corresponding protein residues, whereas ProstT5 [15] focuses on the transformation between residue sequences and their structure token sequences [16] as a pre-training objective. SaProt [14] constructs a structure-aware vocabulary using structure tokens generated by foldseek [16] . Both SaProt and ProstT5 extensively utilize large structure databases [13] for their pre-training datasets. ProSST is also a hybrid structure-sequence model. Compared to previous work, ProSST develops an advanced structure quantization method and a better attention formulation to leverage the structure cues.The most intuitive way to represent a protein structure is using continuous features, such as coordinates, dihedral angles and distance map. However, directly using these continuous features in the pre-training may lead to overfitting [14] . This issue arises from the mismatched representations of the structure between the training set (derived from model predictions) and the test set (measured by wet-lab experiments). As the bridge to eliminate this gap, structure quantization has been investigated by a few works. These methods can be divided into two groups based on the way to generate the discrete secondary structure, including the methods based on physical computing, such as DSSP [39] , and the methods based on deep learning, such as Foldseek [16] , which have been successfully applied to structure-aware PLMs [14, 15] . The structure quantization module of ProSST also relies on learning-based approaches but provides a more detailed residue structure representation than Foldseek.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] See Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we introduce the architecture of ProSST. ProSST mainly contains two modules: structure quantization (Section 3.1) module and a-transformer-based model with sequence-structure disentangled attention. (Section 3.2).The structure quantization module aims to transform a residue's local structure into a discrete token. Initially, the local structure is encoded into a dense vector using a pre-trained structure encoder. Subsequently, a pre-trained k-means clustering model assigns a category label to the local structure based on the encoded vector. Finally, the category label is assigned to the residue as the structure token. The pipeline of structure quantization is shown in Figure 1 .Structure representation. We categorize protein structures into two distinct levels: protein structure and local structure. Protein structure denotes the complete architecture of a protein, including all its residues. The local structure focuses on specific individual residues. It describes the local environment of a residue by centering on a specific residue and including it along with the nearest 40 residues surrounding it in three-dimensional space [18] . Compared to protein structure, local structures are in finer granularity, which allows for a more accurate description of the structure of residue. Therefore, a protein containing L residues has one protein structure and L local structures. Despite the different levels of structure, we can use graphs to represent it. Formally, we represent a structure using graph G = (V , E), where V and E denote the residue-level nodes and edges, respectively. For any given node v ∈ V , it contains only the structure information of the residue, without any residue type information of the residue itself. This ensures that the structure encoder is solely focused on the structure cues. The edge set E = {e ij } includes all i, j for which v j is one of the top-40 nearest neighbors of v i , determined by the distance between their Cα atoms.Structure encoder. Based on the above-mentioned definition of structure, we use geometric vector perceptrons (GVP) [18] as the (local) structure encoder. In particular, the GVP can be represented as a structure feature extraction function π θ (G) ∈ R l×d , where l is the number of nodes, d is the embedding dimension, and θ is trainable parameters. We integrate GVP with a decoder that includes a position-wise multi-layer perceptron (MLP) to form an auto-encoder model. The entire model is trained using a de-noising pre-training objective. In this process, we perturb Cα coordinates with 3D Gaussian noise (Figure 1A ) and use Brownian motion on the manifold of rotation matrices, according to RF-Diffusion [40] . The model is then tasked with recovering the structure to its original, noise-free state. After being trained on the C.A.T.H dataset [41] (see Appendix A.2), we exclude the decoder and utilize solely the mean pooled output of the encoder as the final representation of structures. Although the structure encoder is trained on protein structures, it can effectively encode local structures. Therefore, for a graph G of a protein structure, the encoding is: r = 1 l l i=1 π θ (g i ), where g i represents the graph of the local structure associated with the i-th residue in the graph G, and π θ (g i ) ∈ R d is the output of the encoder for the i-th node. Here, r ∈ R d is the mean pooled output of the encoder and the vectorized representation of the local structure.Local structure codebook. The structure code book quantizes dense vectors representing protein structure into discrete tokens (Figure 1B ). To build this, we employ a structure encoder to embed the local structures of all residues from the C.A.T.H dataset (See in Appendix A.2) into a continuous latent space. Then we apply the k-means algorithm to identify K centroids within this latent space, denoted as {e i } K i=1 . These centroids constitute the structure codebook, as shown in Figure 1B . For any local-structure embedding, it is quantized by the nearest vector e j within the codebook and j serving as the structure token. In this paper, the clustering number K is also referred to as the structure vocabulary size.Protein serialization and quantization. In general, for a residue at position i in a protein sequence, we first build a graph g i only based on its local structure, and then use the structure encoder to embed it into a continuous vector r i . Then we use the codebook to assign a structure token s i ∈ {1, 2, ..., K} to this vector as the structure token of the residue. Overall, the entire protein structure can be serialized and quantized into a sequence of structure tokens (Figure 1C ). Inspired by DeBerta [42] , we use an expanded form of disentangled attention to combine the attention of residual sequences and structure sequences as well as relative positions. Specifically, for a residue at position i in a protein sequence, it can be represented by three items: R i denotes its residue token hidden state, S i represents the embedding of residue-level local structure, and P i|j is the embedding of relative position with the token at position j. The calculation of the cross attention A i,j between residue i and residue j can be decomposed into nine components by:EQUATIONAs formulated in Equation 1, the attention weight of a residue pair can be calculated by separate matrices, including residue tokens, structure tokens, and relative positions. These matrices are utilized for various interactions such as residue-to-residue, residue-to-structure, residue-to-position, structure-to-residue, structure-to-structure, structure-to-position, position-to-residue, position-tostructure, and position-to-position. Since our model concentrates on learning contextual embeddings for residues, the terms structure-to-structure (S i S ⊤ j ), structure-to-position (S i P ⊤ j|i ), position-tostructure (P j|i S ⊤ j ), and position-to-position (P j|i P ⊤ j|i ) do not provide relevant information about residues and thus do not contribute significantly. Consequently, these terms are removed from our implementation of the attention weight calculation. As shown in Figure 2 , our sequence-structure disentangled attention mechanism includes 5 types of attention.In the following part, we use single-head attention as an example to demonstrate the operation of sequence-structure disentangled attention. To begin, we define the relative position of the i-th to the j-th residue, denoted as δ(i, j):δ(i, j) =    0 if i -j ≤ -L max 2L max -1 if i -j ≥ L max , i -j + L max otherwise (2)where, L max represents the maximum relative distance we consider, which is set to 1024 in the implementation. Similar to standard self-attention operation [19] , the computation of query, key for structure, residue and relative position, and the value for residue is as follows:Q r = RW q r K r = RW k r V r = RW v r Q s = SW q s K s = SW k s Q p = P W q p K p = SW k p (3)and the the attention score Âi,j from residue i to residue j can be calculated as follows:EQUATIONwhere Q r i represents the i-th row of the matrix Q r , and K r j denotes the j-th row of K r . Q s i and K s j are the i-th and j-th rows of Q s and K s , respectively. The term K p δ(i,j) refers to the row in K p indexed by the relative distance δ(i, j), and Q p δ(j,i) refers to the row in Q p indexed by the relative distance δ(j, i). To normalize the attention scores, a scaling factor of 1 √ 5d is applied to Â. This scaling is crucial for ensuring the stability of model training [42] , particularly when dealing with large-scale language models. All the Âij form the attention matrix, and the final output residue hidden state is R o :EQUATIONwhich is used as the input for the hidden state of the next layer.ProSST is pre-trained with the structure-conditioned masked language modeling. In this approach, each input sequence x is noised by substituting a fraction of the residues with a special mask token or other residues. The objective of ProSST is to predict the original tokens that have been noise in the input sequence, utilizing both the corrupted sequence and its structure token sequence s as context:EQUATIONWe randomly select 15% indices from the set M for nosing and computing loss for back-propagation.At each selected index i, there is an 80% chance of substituting the residue with a mask token, a 10% chance of replacing it with a random residue token, and the remaining residues are unchanged. The training objective is to minimize the negative log-likelihood for each noised residue x i , based on the partially noised sequence x/M and the un-noised structure tokens, serving as contextual cues. Therefore, to accurately predict the noised tokens, this objective enables the model not only to learn the dependencies between residues but also the relationship between residues and structures. The details of pre-training dataset and hyper-parameter configuration can be found in Appendix A.2.Model Model Type ρ s ↑ NDCG ↑ Top-recall ↑ EVE [49] 0.439 0.781 0.230 EVmutation [53] 0.395 0.777 0.222 DeepSequence [51] 0.407 0.774 0.225 WaveNet [50] 0.373 0.761 0.203 GEMME [47] 0.457 0.777 0.211 MSA-Transformer [48] Evolution-based 0.434 0.779 0.217 Tranception [21] 0.434 0.779 0.220 RITA [44] 0.372 0.751 0.193 UniRep [45] 0.190 0.647 0.139 ESM-1v [6] 0.374 0.732 0.211 ESM-2 [7] 0.414 0.747 0.217 ProGen2 [22] 0.391 0.767 0.199 VESPA [46] Sequence-based 0.394 0.759 0.201 ESM-IF [37] 0.422 0.748 0.223 MIF-ST [38] Inverse-folding 0.401 0.765 0.226 Trancepiton-EVE [52] 0 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] This paper does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we comprehensively evaluate the representation ability of ProSST in several benchmarks, covering zero-shot mutant effective prediction tasks (Section 4.1) and various supervised function prediction tasks (Section 4.2). Additionally, we also provide ablation studies and discussions to further show the effectiveness of the detailed designs in our model (Section 4.3).Datasets. To evaluate the effectiveness of ProSST in zero-shot mutant effect prediction, we conduct experiments on ProteinGym [43] and utilize AlphaFold2 [11] to generate the structures of wild-type sequences. See Appendix A.2 for the details of the dataset and Appendix A.1 for scoring method.Baselines. We compare ProSST with the current state-of-the-art models, including sequence-based models [6, 7, 21, 44, 45, 22, 46] , sequence-structure model [14] , inverse folding models [37, 38] , evolutionary models [47, 48, 49, 50, 51] , and ensemble models [6, 52, 51] .Results. Table 1 shows the performance of zero-shot mutant effect prediction on ProteinGYM. Based on the results, we draw several noteworthy conclusions:• ProSST outperforms all baselines on zero-shot mutant effect predictions of ProteinGYM. We used the non-parametric bootstrap method to calculate the standard error of the difference in Spearman performance between each model and ProSST. The results showed that all standard errors were less than 0.01. This calculation was based on 10,000 bootstrap samples extracted from proteins in the ProteinGym benchmark. Furthermore, ProSST was compared against other models on subsets of ProteinGYM categorized by function, such as stability, activity, binding, and expression. ProSST achieves state-of-the-art (SOTA) performance in the stability, binding, and expression subsets, as detailed in Appendix A.4. Notably, ProSST achieves the best performance in predicting stability, aligning with the previous findings that models incorporating structure information typically perform better in stability predictions [43] .• The degraded version of ProSST (without structure) gets results similar to other sequencebased models. This demonstrates that the performance improvement of our model stems from the efficient modeling of structure information, rather than other factors such as more powerful backbones. Baselines. We compared ProSST with other PLMs including ESM-2 [7] , ESM-1b [5] , and the sequence-structure model SaProt [14] (two parameter versions, 35M and 650M), MIF-ST [38] , as well as the protein structure representation model GearNet [32] and ESM-GearNet [34] .Results. The results of the supervised fine-tuning tasks are shown in Table 4 .2, and we can get the following conclusions:• ProSST gets the best results among all models with 4 firsts in all 6 settings. For the tasks (settings) of DeepLoc, Metal Ion Binding, ProSST largely surpasses other methods, and ESM-GearNet gets comparable (or slightly better) results for thermostability and GO-BP and GO-CC with ProSST, at the price of more than 6× model size. • The sequence-structure models, ESM-GearNet, SaProt and ProSST, show better results than other counterparts, which suggests the importance of the structure cues in protein modeling. Furthermore, ProSST is more capable of integrating sequence and structure information of proteins than SaProt, which confirms the effectiveness of our designs.Combined with the results in Section 4.1, ProSST exhibits powerful ability in multiple settings.In this section, we provide additional ablation studies and discussions to show the necessity and effectiveness of the detailed designs in ProSST. Specifically, we use zero-shot mutant effect prediction on ProteinGYM, supervised downstream task DeepLoc, and the perplexity in the pre-training validation set to conduct corresponding experiments.Ablations on quantized structure. The ablation results of quantized structure are shown in Table 3 and Figure 3 (a), and we can get the following findings:• We can find, as the increases of K (the size of local structure vocabulary), the performance of ProSST shows an upward trend on all metrics, and most metrics achieve the best results with K = 2048. Based on that, we set K = 2048 as our default setting. • As the increase of K, the convergence of ProSST improves progressively (Figure 3(a) ), which suggests incorporating structure cues can improve the representation capabilities of models. • Based on the same network architecture, the proposed structure quantization method (with an appropriate hyper-parameter K) performs better than Foldseek [16] and DSSP [39] , which shows the effectiveness of our design. • ProSST (Foldseek), ProSST (DSSP), and all ProSST (K>0) models significantly surpass ProSST (K=0) in all metrics, which confirms the importance of the structure cues again.• There is almost no difference in performance between ProSST (K=1) and ProSST (K=0), indicating that the improvement does not come from the rise in parameters of disentangled attention.Ablations on disentangled attention. Here we show detailed ablations and comparisons of disentangled attention in Table 4 and Figure 3 (b), and we can get the following observations:• All items in Equation 4 are necessary to our attention formulation. Also note that 'P2R' attention has the least impact on model capacity, with the Perplexity slightly increasing from 9.033 to 9.173, suggesting that positional attention to amino acids is relatively less critical than other items. Conversely, removing 'R2S' item results in a significant increase in Perplexity from 9.033 to 12.142, underscoring the important role of structure information in enhancing the model's representation capability.• Compared with standard self-attention, our attention formulation gets better results for all metrics, indicating that explicitly modeling structure cues is crucial for integrating such information. Besides, positional encoding is also necessary in our design.As we have mentioned in the Section 2, our disentangled should learn the connections between structure and residue sequence. To valid these, we conduct further experiments to analyze disentangled attention in the Appendix A.5.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] This paper has disclosed all the necessary information to reproduce the main experimental results, including the dataset (See Appendix A.2), network architecture (See Section 3.2), and network hyper-parameters (See Appendix A.3). 
conclusion and limitations: 
 This paper introduces ProSST, a protein sequence-structure transformer for PLM. ProSST includes two key techniques, protein structure quantization and sequence-structure disentangled attention. The structure quantization module contains an encoder and a k-means clustering model. The encoder is trained with a denoising objective and is utilized for encoding protein structures. Leveraging this encoder, we embed the local structures of each residue within every protein in the C.A.T.H dataset into a continuous latent space. Then we utilize k-means clustering algorithm to obtain K (default setting is 2048) centroids. These centroids are then utilized to discretize the local structures of residues based on the index of the nearest centroid of its structure embedding vectors. A protein structure can be transformed into a sequence of discrete numbers (or referred to tokens) and each token representing the corresponding local structure of residue. The sequence-structure attention enhances standard self-attention by not only considering self-attention residues but also incorporating attention between residues and structures, and vice versa. This enables the model to learn the relationships between residues and structures, thereby acquiring improved adequate contextual representations of residues. Furthermore, we pre-train ProSST with 18.8 million protein structures using a MLM objective. Experimental results show that ProSST can outperform existing models in ProteinGYM benchmark and other supervised learning tasks. Despite of this, there are some limitations of ProSST. For example, the local structure construction and encoding requires heavy computations. In the future work, we aim to speed up the protein structure quantization process. Another threat is that the structural and sequential data are required for ProSST to derive the final protein representations, since the amount of available structural data is lower than that of sequence data. We provide solutions in the Appendix Section A.6. Additionally, we plan to enhance ProSST by training it with larger structure datasets and expanding its parameter, which may further improve its performance.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Our code and pre-trained models is available at https://github.com/ ai4protein/ProSST.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] We provide implementation details of our model in Section A.3. Baseline methods are pre-trained by their authors, which requires no further supervision or learning procedures.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Measurements used to in this work return fixed results on a determined model, and a pre-trained model barely trains a set of models under the same dataset or hyper-parameter settings.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] We cite all the papers for baseline methods. The baseline results of Prote-inGYM are copied from the latest release on their GitHub repositories, which we attached URLs in Appendix A.2. The baselines results of supervised fine-tuning are copied from the latest reviewed version of SaProt.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Zero-Shot ScoringPrevious studies have demonstrated that PLMs, when trained on extensive and varied protein sequence databases, are capable of predicting experimental measurements of protein mutants function without further supervision [6, 14] . For those PLMs that are trained with masked language modeling objective, the calculation of mutation scores can be formalized as follows:EQUATIONHere F is a single or multi-point mutant, andF = {(p i , f i , w i )|i = 1, 2, ..., |F |} is a set of triplets,where p i ∈ N represents the mutation position, f i is the residue after the point mutation, and w i is the original residue of the point mutation. x is the sequence of residues of the wild type. We slightly modify the formula above to adapt to ProSST, where the structure sequence is an additional condition to score mutants:EQUATIONHere, s is the structure token sequence of the wild type.Dataset for pre-training. The pre-training data is collected from AlphaFoldDB [13] , which contains more than 214 million structures predicted by AlphaFold [11] . We downloaded the 90% reduced version, containing 18.8 million structures. 3 . From this collection, we randomly select 100,000 structures for validation (sequences with a similarity of over 30 to the training set will be removed for data deduplication.), enabling us to monitor the perplexity in the training phase. During pre-training, proteins with more than 2048 resiudes (594 samples) are removed for training efficacy.Dataset for training structure encoder. The dataset used for training the structure encoder originates from CATH43-S40 4 . This dataset is manually annotated and comprises protein crystal structural domains that have been deduplicated for sequence similarity by 40%. The original dataset contains 31,885 structures. After removing structural domains missing atoms such as Cα and N, the dataset is reduced to 31,270 entries. From this, 200 structures were randomly selected to serve as a validation set. The auto-encoder model was then trained using the configuration that yielded the lowest loss on this validation set.Dataset for training structure codebook. The dataset for training the structure codebook consists of local structures extracted from CATH43-S40. Given a protein structure, slide along the residue sequence to select a segment with a chosen residue as the anchor. Connect up to 40 residues within 10 Å [54] to form a star-shaped graph. As shown in Figure A4 , local structures with more than 40 neighbors account for only 0.00052%, indicating that our choice covers most cases. For pairwise amino acid pairs in this graph, if the Euclidean distance is less than 10 Å, a link will be assigned to them. This process yields a number of protein local structures equal to the length of the protein multiplied by the total number of proteins, resulting in 4,735,677 local structures from the protein structures in CATH43-S40. These sub-structures are fed into a structural encoder to obtain embeddings. By setting various quantities for K, different structure codebooks are obtained using the k-means clustering algorithm.Dataset and metrics for zero-shot mutant effect prediction.We utilize the ProteinGYM benchmark [43] to assess the zero-shot mutant effect prediction capabilities of ProSST. ProteinGYM offers comprehensive benchmarks specifically collected for predicting protein fitness. It contains a wide range of deep mutational scanning assays with millions of mutated sequences. ProSST is evaluated using the most extensively utilized datasets for substitution mutations, which include 217 experimental assays. Each assay incorporates both the sequence and structure of the protein, with a particular emphasis on 66 datasets that focus on thermo-stability. The evaluation metrics employed are the Spearman coefficient, Top-recall, and NDCG, where higher values signify superior model performance. These metrics are computed using scripts5 provided by ProteinGYM.Datasets and metrics for downstream tasks. The downstream datasets have the same train, valid, and test splits as SaProt's and are downloaded from SaProt. Data statistics are provided in Table A5 .• Thermostability. The task is to predict the thermostability values of proteins using the "Human-cell" divisions from the Thermostability task in FLIP [55] . For this regression task, the Spearman correlation coefficient is utilized as the evaluation metric to evaluate the prediction results.• DeepLoc (Protein Sub-cellular Localization). The task is to output a probability distribution across two sub-cellular localization categories for a protein. This is a binary classification task, and we utilize accuracy as the metric to evaluate the predictions. This dataset was introduced by DeepLoc [56] and we use the original data split.• Metal Ion Binding. The task is to predict whether metal ion-binding sites exist within a protein. This is also a binary classification task, and we utilize accuracy as the metric to evaluate the predictions. This dataset was introduced by TAPE [20] , and we use the original data split.• GO annotations prediction. This task is to predict Gene Ontology terms to evaluate the model's ability to predict protein functions. This task was introduced by DeepFRI [26], and we use three types of GO labels: Molecular Function (MF), Biological Process (BP), and Cellular Component (CC). This is a multi-label classification task, and we evaluate the model using the Max F1-Score.Structure encoder. We describe a structure with the graph G = (V , E), adopting the characterizations of V and E as outlined in the GVP framework [18] . The GVP encoder includes a six-layer message-passing graph neural network in which a geometric perceptron replaces the MLP to ensure translational and rotational invariance of the input structure. Our GVP encoder is consistent with the original GVP-GNN [18] , except that we removed the residue type information. The GVP encoder is trained from scratch. We conducted additional experiments to analyze the relationship between disentangled attention.Experiment 1. We replaced all structure tokens in the ProteinGYM and the validation set with zeros or random numbers from a uniform distribution and re-evaluated ProSST. The results are shown in Table A7 The results show that the incorrect structure tokens harmed the performance of ProSST significantly, suggesting that disentangled attention learned the sequence-structure relationship.Otherwise, the performance would only have a minimal impact.Experiment 2. To verify if disentangled attention can indeed enhance performance, we train ProSST (K=1) by substituting the structural tokens with a constant value of 1. This configuration maintains the disentangled attention mechanism even though the structure input is useless. If ProSST (K=1) still improves performance, it indicates that the improvement is solely due to the disentangled attention. We evaluate this model on DeepLoc, ProteinGYM and the valid set. The results are shown in Table A8 There is little difference between K=1 and K=0 since their perplexity curves (refer to Figure 3 (a)) nearly overlap. This suggests that disentangled attention alone cannot enhance performance without correct structure tokens.Experiment 3. We visualize the learned different types of attentions on Green Fluorescent Protein (GFP, Unipro ID:P422127 ), including 238 residues, in Figure A6 . We can see that disentangled attention learns different attention patterns, with notable differences between "R2S" and "S2R". We can conclude from Experiment 1 and Experiment 2 that the disentangled attention and structure tokens sequence are inseparable. The disentangled attention mechanism cannot function without correct structure tokens. This also indicates that the performance improvement of our model stems from the design of the model rather than increasing its parameters of the attention layer. Furthermore, the Experiment 3 shows that our disentangled attention actually learned different patterns of attentions.In conclusion, we offer two solutions for obtain representations of the sequence-only protein datasets:• Utilize AlphaFold 2 [11] or ESMFold [7] 8 for structure prediction as they are highly reliable methods.• Use ProSST (MST), which is trained with structure masking and supporting sequence-only inputs. The MST denotes "Masked Structure Training (MST)", which means that during pre-training, each sample's structure sequence has a 50% probability of being replaced by a fully masked sequence [, simulating missing protein structure. Therefore, when applying ProSST to sequence-only datasets, we need to use the masked sequence [1,1,1,1,1,...,1] as a substitute for the structure token sequence.We have evaluated the two approaches on the ProteinGym benchmark, binary localization prediction (BLP) from a sequence-only benchmark, PEER [58] , and perplexity on the validation set. The results are show in Table A9 : In the Table A9 , the first two rows show the performance differences between AlphaFold and ESMFold. Rows 3-4 show the performance of the new model ProSST(MST). And row 5 shows the performance of the sequence-only model.A.7 AlphaFold pLDDT versus Zero-shot mutant effect performance Protein structures containing disorder region may not be accurately predicted by AlphaFold 2, potentially leading to reduced performance of structure-aware models. Here, we test the relationship between AlphaFold pLDDT scores and the performance of structure-aware models including ProSST, SaProt, and ESM-IF on ProteinGYM, as illustrated in Figures A7. Our findings reveal a positive correlation between pLDDT values and model performance: a correlation coefficient of 0.30 for ProSST, 0.31 for SaProt, and 0.42 for ESM-IF1, where the correlation coefficient (ρ p ) represents the strength of the relationship. These results suggest that structure-aware models may exhibit limitations in accurately predicting the structures of disordered proteins.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] We state clearly the scope and contribution of both communities of protein language modeling and computer science in the abstract and introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://cluster.foldseek.com/http://download.cathdb.info/cath/releases/all-releases/v4_3_0/non-redundant-data-sets/https://github.com/OATML-Markslab/ProteinGym/blob/main/scripts/https://github.com/westlake-repl/SaProthttps://www.uniprot.org/uniprotkb/P42212/entryhttps://esmatlas.com/resources?action=fold 


Decision: The work introduces ProSST, a novel protein language model that simultaneously leverages sequence and structure information to produce SOTA performance on various widely-used benchmarks. The novelty is in the use of (1) vector-quantized graph autoencoder to tokenize protein structure into local structure tokens and (2) disentagled attention to model the relationship between sequence tokens and structure tokens. The consensus is that the novelty, clarity, and performance introduced in this work will be of interest to NeurIPS audience.

Reviews: review: paper_summary: paper_summary: This paper presents ProSST, a new language model for protein data that captures both structural and sequential modalities of proteins. The protein's structure is tokenized using a graph-based auto-encoder architecture, where each residue's local structure is embedded into a vector in a high-dimensional embedding space and then classified using a $k$-means clustering algorithm within that embedding space. The two sequence-based and structure-based data streams are then passed to multiple disentangled attention layers, allowing the model to better capture the connections between structure and sequence data. Numerical results showcase the outperformance of ProSST over several baselines across a range of downstream tasks.
strength_weakness: strength_weakness: The main weakness of this work, in my opinion, is that both structural and sequential data are required for the model to derive the final protein representations, since the amount of available structural data is lower than that of sequence data. I wonder if there is any way in which the model can operate on sequence input alone, with the structure knowledge embedded into the model parameters. Especially, how does the model work on protein sequences for which no structural data is available? Is there a "sequence-only" mode that the model can revert to (e.g., by feeding a "default/noise" structure input alongside the actual sequence)? I see the ProSST (-structure) model in Table 1, but that seems to be a model that was only trained on sequence data, so the training pipeline and its parameterization are completely different than the complete ProSST framework.
questions: questions: - What is the reasoning behind choosing **40** residues as the neighborhood of each node in Section 3.1?

- Is $L$ on line 128 the same as $l$ on line 137? 

- Could you please elaborate on how the GVP encoder is parameterized?

- Have you studied what would happen if, instead of discrete structure tokens on the bottom-right of Figure 1, the centroid embedding of each cluster is used as the input structural vector? Also, what if the continuous local structure embeddings are directly used and the clustering is removed altogether?

- Could you explain the average pooling that happens on line 146? In particular, what does $l$ precisely represent here? Is it the 40-node neighborhood of each residue's local graph?

- What is the difference between the disentangled attention in Eq. (1) and a regular attention mechanism, where the tokenized representations of each residue are simply concatenated together?

- What is $k$ in Eq. (2)? Is it the same as the number of clusters in $k$-means clustering of structural embeddings?

- Why is the structure $s$ in Eq. (6) left unmasked? Shouldn't you also mask $s$ when you make the residue tokens in the sequence data? On a related note, shouldn't the structure of the variant sequence be changed in the first term of Eq. (8) when calculating the scores?

- It would be helpful if you could also compare your approach with ESM-GearNet, and also provide the results of supervised downstream prediction with frozen embeddings in Section 4.2 (as opposed to fine-tuning the model).

- Minor points: $e$ should be replaced by $e_i$ on line 154, and $m$ should be replaced by $M$ in Eq. (6).
limitations: limitations: As the authors allude to, the main limitation of the model is the assumption of the availability of structural data and the computational complexity of the structural quantization process.
paper_summary: paper_summary: The paper proposes ProSST (Protein Structure-Sequence Transformer), a novel PLM incorporating sequence and structure information. ProSST can be split into two parts: a modified version of the Transformer architecture and a quantization module. The quantization module consists of a structure encoder using a Geometric Vector Perceptron (GVP). The authors train the GVP on a large dataset of protein structures (18.8 million) in which they encode residues by contextualizing local structures with surrounding ones in a manner more robust than current methods. They then discretize local residues into tokens using a pre-trained k-means clustering model. ProSST adds relative positional information to the structure before feeding it into the Transformer architecture. Instead of standard self-attention, the authors propose using disentangled attention, a method that allows the model to process contextual information from sequence and structure tokens. They test their model on various downstream protein function tasks, for which they achieve state-of-the-art performance across almost all metrics.
The main contributions of the paper are as follows:
● Introduced a novel method for protein structure quantization by incorporating contextual structural information into a PLM using a structure autoencoder
● Proposed a nonstandard attention calculation that allows for sequence and structure information to be incorporated into protein function prediction
strength_weakness: strength_weakness: - Despite having clearly formulated reasons for choosing their quantization methods, the authors need a similar justification for disentangled attention. There is little discussion on this design choice until after the related works section. Similarly, there is little experimental analysis of disentangled attention outside the ablation study. This is surprising as the departure from standard self-attention allows for a nearly four percent increase in performance, which is just as impactful as their quantization module. The paper talks about learning the connections between structural tokens and sequence tokens as the main contribution of ProSST but never demonstrates the connection.

- The paper conveys the generalization power of their model by testing it on multiple downstream protein function prediction tasks. ProSST outperforms nearly every other model on these benchmarks, yet some experimental setups raise concerns. Most glaringly, the paper provides little information on their testing regimen across all training tasks. In the experiments that do outline testing, the authors describe extremely small validation sets. For example, in the training phase, the authors train ProSST on 18.8 million samples, with 100,000 being used for validation (~ 99:1 split). Similarly, the split for training the structure codebook is 31,070 train points with only 200 points for validation (~ 99:1 split). The paper needs to provide more information on the testing regimen for the fine-tuning tasks. Along with omitting these training details, the authors seemingly make many mistakes in reporting their results. First and foremost, the paper provides no error or significance analysis despite including empirical evaluations of ProSST (which is required in the NeurIPS paper checklist). 

- In Table 3 and Table 4 on page 8, the paper evaluates the effectiveness of ProSST, marking the best results in bold. When comparing the best results from both tables, it is clear that the ablation study was done with K=2048 being their clustering hyperparameter due to the matching accuracies (although this is not reported explicitly). However, almost all of the other metrics outside of accuracy for this experiment do not match, which is either a copying mistake by the authors or is indicative of a different experiment. If the latter explanation is true, then the variance in the results indicates that the paper should have been more thorough in its analysis of the model’s performance across initializations and data splits. Along with the extremely small validation set sizes and little testing information, this makes me skeptical of the empirical results section. In addition, in Table 1, the authors incorrectly bold their model’s performance on the NDCG metric when it is clear that five other models outperform theirs on this specific metric.

Another area for improvement of the paper is its analysis of the structure quantization module. The model has a four percent increase in performance when jumping from K=0 to K=20 and has diminishing returns from there onward. At K=0, there are no structure tokens passed to the model, meaning that disentangled attention is unable to work effectively in this experimental setting. This becomes an issue when you compare the results to the ablation study on disentangled attention; it is difficult to compare the effects of the quantization module to disentangled attention. Because of this, their results might suggest that the increase in performance is due to disentangled attention, not necessarily their quantization module.

The conclusion section is also a weakness of thenot give much insight into the methods proposed and instead reviews the methods section. It lacks a thorough ana paper. It does lysis of the paper's results.
questions: questions: Is there a reason why several of the baselines (e.g., EVmutation, DeepSequence, WaveNet, UniRep, RITA, ProGen2, and VESPA) and datasets (DMS data other than thermostability) from ProteinGym are not included in this study?
limitations: limitations: - Contribution is limited given that now ESM3 and many other PLMs consider structure in the language model
- There has been no discussion on several protein functions that do not benefit from rigid 3D structure, as predicted by AF2/3, e.g., disordered protein functions with floppy regions: how would ProSST handle those cases?

Conclusion:

This paper provides interesting methods for improving PLM performance by intelligently incorporating structural information into protein function prediction problems. The two novelties of this paper come from their quantization method and disentangled attention application. Despite providing compelling methods and promising results, there are many details that are glossed over, which over the course of the paper accrue many concerns from the reader on the validity of the results. My suggestion to the authors is to provide more insight into their choice of disentangled attention (related works section) and analysis of the results (conclusion). Similarly, the experiment section and supplemental materials need to be updated with enough information for the reader to faithfully reproduce the results according to best practices. In general, there is enough concern from the experimental results that warrant justification and response from the authors.
paper_summary: paper_summary: This paper focuses on the protein representation task. ProSST introduces a quantized method to combine the information from protein structure. Additionally, the authors also propose a disentangled attention mechanism on top of the quantized structures to learn the relationship between residue and structure token sequences. They claim that the proposed method outperforms state-of-the-art methods in several tasks under both zero-shot and supervised learning settings.
strength_weakness: strength_weakness: 1. The novelty is somewhat limited. I do believe such work is meaningful for computational biology, but this is not the first hybrid approach to protein language models (such as ESM-GearNet [1]).
2. Regarding the experiment, I strongly suggest comparing ProSST to ESM-GearNet, which I think is a fairer comparison considering the utilization of both sequence and structure. 
3. Besides comparing the number of parameters, I suggest the authors provide experiments on inference speed. In my experience, clustering is often time-consuming, especially for long sequences and multiple cluster centers (K > 2000 in this paper).

[1] A Systematic Study of Joint Representation Learning on Protein Sequences and Structures
questions: questions: 1. How does ProSST deal with extremely long protein sequences?
2. How to deploy this method to sequence-only datasets like the PEER benchmark [2]? It seems ProSST requires both sequence and structure as inputs.

[2] Peer: a comprehensive and multi-task benchmark for protein sequence understanding
limitations: limitations: My main concern is the experiment setting mentioned in the Weaknesses part. Authors are welcome to answer my questions and I am leaning to raise my rate.
paper_summary: paper_summary: In this paper the authors introduce ProSST, a method for training protein language models on both sequence and structure. They use a geometric vector perceptron trained with a structural denoising objective to obtain a structure encoder, and then perform k-means clustering on the embeddings of local residue neighborhoods in CATH to obtain a codebook of structural tokens. A masked language model is trained with disentangled attention, conditioned on the structural token inputs and the corrupted sequence.
strength_weakness: strength_weakness: It is not clear why the structural quantization is a two-step process, involving training a separate structure encoder with a denoising objective, and then clustering to form the local codebook. An advantage of SaProt is the simplicity of the codebook and almost negligible cost of structural tokenization. How does this compare to the pre-processing needed for ProSST?
questions: questions: Can the authors give some indication about the increased time and computational cost of tokenizing structures using their approach?

Why is the two-step structural quantization approach used, rather than a typical VQ-VAE as used in FoldSeek? Are there advantages to training stability, simplicity, or inference time tokenization? 

Have the authors investigated using smaller neighborhood sizes than 40 nearest neighbors? Is this a route to reducing the computational burden of structural quantization?
limitations: limitations: Yes

Comments: title: Thanks for the response
comment: We deeply appreciate the dedication you have shown in evaluating our submission and offering your comments. Your insightful recommendations have played a crucial role in improving the clarity and rigor of our work. Regardless of the ultimate outcome, your input has enhanced the clarity and quality of our work. We extend our heartfelt thanks to you!
We thank you for taking the time and effort to review our research and engage with our responses. Your comments have helped us refine our paper. Regardless of the final outcome, your feedback has helped us better define and present our research objectives.
We are truly grateful for the time and effort you dedicated to reviewing our work and considering our responses. Your constructive feedback and expert advice have significantly enhanced the quality of our research. Regardless of the final decision, your thoughtful engagement has greatly clarified the objectives and structure of our paper. Thank you for your invaluable contribution!
We sincerely appreciate the time and effort you dedicated to reviewing our paper and reading our responses. Your valuable suggestions and insightful comments have significantly contributed to refining the quality of our work. No matter what the final result will be, the thoughtful communication with you has better clarified our paper's goals and logic. We would like to express our great gratitude to you!
I thank the authors for their response and will maintain my score.
My concerns are mostly addressed. I increase my score.
Thank you for the responses, I believe my concerns have mostly been addressed, although I still think there exists many similarities with previous work. I do think that this is now above the acceptance threshold and have changed my score.
Thank you. I have decided to increase my score in favor of acceptance after reading the rebuttal and the rest of the reviews. I believe the paper presents a worthwhile contribution that is of interest to the NeurIPS audience.