Evaluating Representation Learning on the Protein Structure Universe
Authors: Arian Rokkum Jamasb
Alex Morehead
Chaitanya K. Joshi
Zuobai Zhang
Kieran Didi
Simon V Mathis
Charles Harris
Jian Tang
Jianlin Cheng
Pietro Lio
Tom Leon Blundell
Venue: ICLR 2024 poster (2024)
Keywords: Protein
Representation
Learning
Protein Structure
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Benchmark for evaluating pre-training and representation learning on protein structures.

Abstract: We introduce ProteinWorkshop, a comprehensive benchmark suite for representation learning on protein structures with Geometric Graph Neural Networks. We consider large-scale pre-training and downstream tasks on both experimental and predicted structures to enable the systematic evaluation of the quality of the learned structural representation and their usefulness in capturing functional relationships for downstream tasks. We find that: (1) large-scale pretraining on AlphaFold structures and auxiliary tasks consistently improve the performance of both rotation-invariant and equivariant GNNs, and (2) more expressive equivariant GNNs benefit from pretraining to a greater extent compared to invariant models.
We aim to establish a common ground for the machine learning and computational biology communities to rigorously compare and advance protein structure representation learning. Our open-source codebase reduces the barrier to entry for working with large protein structure datasets by providing: (1) storage-efficient dataloaders for large-scale structural databases including AlphaFoldDB and ESM Atlas, as well as (2) utilities for constructing new tasks from the entire PDB. ProteinWorkshop is available at: github.com/a-r-j/ProteinWorkshop.

Published as a conference paper at ICLR 2024
EVALUATING REPRESENTATION LEARNING ON THE PROTEIN STRUCTURE UNIVERSE
Arian R. Jamasb 1.t Alex Morehead
Chaitanya K. Joshi"!
Zuobai Zhang Jianlin Cheng',
Kieran Didi', Simon Mathis',
Charles Harris', Jian Tang",
Pietro Lio', Tom L., Blundell
University Cambridge,
University of Missouri,
3 Mila - Québec Al Institute
ABSTRACT
We introduce Protein SooteinWorkshop. comprehensive benchmark suite for represen- tation learning on protein structures with Geometric Graph Neural Networks. We consider large-scale pre-training and downstream tasks on both experimen- tal and predicted structures to enable the systematic evaluation of the quality of the learned structural representation and their usefulness in capturing functional relationships for downstream tasks. We find that: (1) large-scale pretraining on AlphaFold structures and auxiliary tasks consistently improve the performance of both rotation- invariant and equivariant GNNs, and (2) more expressive equivariant GNNs benefit from pretraining to greater extent compared to invariant models.
We aim to establish a common ground for the machine learning and computational biology communities to rigorously compare and advance protein structure represen- tation learning. Our open-source codebase reduces the barrier to entry for working with large protein structure datasets by providing: (1) storage-efficient dataloaders for large-scale structural databases including AlphaFoldDB and ESM Atlas, as well (2) utilities for constructing new tasks from the entire PDB. ProteinWorkshop is available at: github. com/a a r- hateaaaarrrpPooeeeeeeeee j/ProteinWorksho......
INTRODUCTION
Modern protein structure prediction methods have led to an explosion in the availability of structural data (Jumper et al., 2021; Back et al., 2021). While many sequence-based functional annotations can be directly mapped to structures, this has resulted in a bggnffcantly-incrrasing gap between structures and meaningful structural annotations (Varadi et al., 2021). Recent work has focused on developing methods to draw biological insights from large volumes of structural data by either determining representative structures that can be used to provide such annotations (Holm, 2022) or representing structures in a simplified and compact manner such as sequence alphabets (van Kempen al., 2023) graph embeddings (Greener & Jamali, 2022). These works have significantly reduced the computational resources required to process and analyse such structural representatives at scale. Nonetheless, remains be shown how such results can help us better understand the relationship between protein sequence, structure, and function through the use of deep learning algorithms.
Several deep learning methods have been developed for protein structures. In particular, Geometric Graph Neural Networks (GNNs) (Duval et al., 2023) have emerged as the architecture of choice for learning structural representations of biomolecules (Schütt et al., 2018: Gasteiger et al., 2020; Jing et al., 2020; Schütt et al., 2021; Morehead et al., 2022; Zhang et al., 2023b). Methods can be categorised according to (1) the featurisation schemes and level of granularity of input structures (Co, backbones, all-atom); as well as (2) the enforcement of physical symmetries and inductive biases (invariant or equivariant representations) (Joshi et al., 2023a). However, there remains a need for robust, standardised benchmark to track the progress of new and established methods with greater granularity and relevance to downstream applications.
"quual contribution
Current affiliation: Prescient Design, Genentech

---

Published as a conference paper at ICLR 2024
Node-lent tocks
idartiiia
PDi
Vere sincing
Astrea
Invers 4
ESM
Virtud TOrMITTR angles
PTM
Inverse foiding
Metagenomic Allas
Hull epp Dibaa
deecarin
AlpliaaFold
Tanslonal Seecking
Foldd
Sibechsh
Youi Servece equ mutrrr Shike
pLDOT predicion
daas
Protein Stracture Datacets
Featurisation Schemee
Geometric GNN Encoders
Pre-training & Auxiliary Tasks
Downstream Teeks
Figure Overview of Protein Worksho. a comprehensive benchmark suite for evaluating pre- training and representation learning of Geometric GNNs on large-scale protein structure data.
In this work, develop unified and rigorous framework for evaluating protein structural encoders, providing pretraining corpora that span known foldspace and tasks that assess the ability of models learn informative representations at different levels of structural granularity. Previous works in protein structure representation learning have focused on learning effective global (i.e. graph-level) representations of protein structure, typically evaluating the methods on function or fold classification tasks (Gligorijević et al., 2021; Zhang al., 2023b). However, there has been comparatively little in- vestigation into the ability different methods to learn informative local (node-level) representations. Good node-level representations
important for variety of annotation tasks, such as binding interaction site prediction (Gainza et al., 2020), as well as providing conditioning signals in structure- conditioned molecule design methods (Schneuing al., 2022; Corso al., 2023). Understanding the dtructure-functin relationship at this granular level can drive progress in protein design by revealing structural motifs that underlie desirable properties, enabling them to be incorporated into designs.
Our contributions are as follows:
We curate numerous structure-buse -aased pretraining and fine-tuning datasets from the literature with a focus on tasks that can enable structural annotation of predicted structures, We compile highly-modular benchmark, enabling the community to rapidly evaluate protein representation learning methods across tasks, models, and pretraining setups.
We benchmark Geometric GNNs for representation learning of proteins at different levels of structural granularity (Co, backbones, sidechain) and across several classes of models, ranging from general-purpose (Schütt et al., 201 8; Satorras et al., 2021) protein-specific architectures (Morehead Cheng, 2024; Zhang al., 2023b). We are the first to evaluate higher order equivariant GNNs (Thomas al., 2018; Batatia et al., 2022) for proteins.
We pretrain and evaluate models on, to our knowledge, the largest non non-redundant protein structure corpus containing 2.27 million structures from AlphaFoldDBB.
Our benchmarks show that sequence and structure demoising- auxiliary tasks and structure denoising-basee pretraining consistently improve Geometric GNNs. Moreover, we surprisingly observe that sequence-based pretrained ESM-2-650M (Lin et al., 2022) augmented with our structural featurisation matches of-e-offthhe-at GNNs on (super)family fold and gene ontology prediction.
2
PROTEIN WORKSHOP
The overarching goal of ProteinWorkshon is effectively cover the design space of protein structure representation learning methods. To achieve this, the benchmark is highly modular by design, enabling evaluation of different combinations of structural encoders, protein featurisation schemes, and auxiliary tasks over a wide range of both supervised and unsupervised tasks. user manual is available in Appendix D, containing detailed listings and descriptions of all components.
2

---

Published as a conference paper at ICLR 2024
2.1 FEATURISATION SCHEMES
Protein structures are typically represented as geometric graphs, with researchers opting to use a coarse-grained Co atoms graph as full atom representations can quickly become computationally intractable due to a large number of nodes. However, this is a lossy representation, with much of the structural detail, such as orientation of the backbone and sidechain structure, being only implicitly encoded. Due to the computational burden incurred by operating on full-atom node representations, we focus primarily on Co-based graph representations, investigating featurisation strategies to incorporate higher-level structural information. Note that we do provide utilities to enable users to work with backbone and full-atom graphs in the benchmark. Details about different featurisation schemes are provided in Appendix D.4 and Table 5.
2.2 PRE-TRAINING TASKS
The benchmark contains a comprehensive suite of pretraining tasks. Broadly, these can be categorised into: masked-atrribute prediction, denoising-basee and contrastive learning-based tasks. These can be used as both a pretraining objective or as auxiliary tasks in a downstream supervised task.
Sequence Denoising. The benchmark contains variations based
sequence corruption processes C) that receive an amino acid sequence S [0, 1] and return sequence SE [0, 1]lv) with fraction of its positions corrupted. The first scheme is based on mutating fraction of the residues to a random amino acid and tasking the model with recovering the uncorrupted sequence. The second is masked residue prediction task, where a fraction of the residues are altered to mask value and the model is tasked to recover the uncorrupted sequence.
Structure Denoising, We provide two structure-based denoising tasks: coordinate denoising and torsional denoising. In the coordinate denoising task, noise is sampled from normal or uniform distribution and scaled by noise factor, v R, and applied each of the atom coordinates in the structure to ensure structural features, such as backbone or sidechain torsion angles, are also corrupted. The model is then tasked with predicting either the per-node noise or the original uncorrupted coordinates. For torsional denoising, the noise is applied to the backbone torsion angles and Cartesian coordinates are recomputed using PNcRF (AlQuraishi, 2019) and the uncorrupted bond lengths and angles prior to feature computation. Similarly the coordinate denoising task, the model is then tasked with predicting either the per-residue angular noise or the original dihedral angles.
Sequence-Strucctee Co-Denoising. This is multitask formulation of the previously described structure and sequence denoising tasks, with separate output heads for denoising each modality.
Masked Attribute Prediction. We use inverse folding (Section 2.3.1) as pretraining task. The benchmark additionally incorporates the distance, angle and dihedral angle masked-attribut. prediction (Zhang al., 2023b) well backbone dihedral angle prediction task.
PLDDT Prediction. Structure prediction models typically provide per-residue PLDDT (predicted Local Distance Difference Test) scores local confidence measures which have been shown to correlate well with disordered regions (Wilson et al., 2022). We formulate a node-level regression task on predicted structures, somewhat analogous structure quality assessment, where the model is PLDDT DLDssidduupDDDDDD>p
tasked with predicting the scaled per-residue
E [0, 1] values.
2.3 DOWNSTREAM TASKS
We curate several structure- and sequence-based datasets from the literature and existing benchmarks summarised in Table 1. The tasks are selected to evaluate not only the global structure representational power of each method, but also to evaluate the ability of each method to learn informative local representations for residue-level prediction and annotation tasks.
The raw structures are, where possible and accounting for obsolescence, retrieved directly from the PDB (or another structural source) as several processed datasets used by the community discard full atomic coordinates in favour of retaining only C. positions, making them unsuitable for in-depth
'To retain focus on protein representation learning, we deliberately exclude commonly used tasks based on protein-small molecule interactions as it is hard to disentangle the effect of the small molecule representation and the potential for bias (Boyles et al., 2019).
3

---

Published as a conference paper at ICLR 2024
Table 1: Overview of supervised tasks and datasets.
Task
Dataset Origin
Structures
# Train
# Validation
# Test
Metric
Graph- lovel Node level
Inverse Folding
Ingraham et al. (2019)
Experimental
3,9 M
105 K
180 K
Perplexity
PPI Site Prediction
Gainza et al. (2020)
Experimental
478 K
K
117 K
AUPRC
Metal Bind Site Prediction
Experimental
1.1 M
13.7
29.8 K
Accuracy
FTM Site Prediction
Yan et al. (2023)
Predicted
44
2,4
2.5
ROC-AUC
Fold Prediction
Hou et al. (2017)
Experimental
12.3
0.7 K
1.3/0.7/1. K
Accuracy
Gene Ontology Prediction
Gligorijevič al. (2021)
Experimental
27.5 K
3.1 K
3.0 K
Finx
Reaction Class Prodiction
Hermosilla et al. (2020)
Experimental Experimental 1.7
29.2 K
2.6
K
Accuracy
Antibody Dev. Prediction
Huang et (2021)
0.24
0.48 K
AUPRC
experimentation. This provides an entry point for users to apply a custom sequence of pre-processing steps, such deprotonation or fixing missing regions which are common in experimental data.
2.3.1 NODE-LEVEL TASKS
Inverse Folding. Many generative methods for protein design produce backbone structures that require the design of an associated sequence. As a result, inverse folding is an important part of de novo design pipelines for proteins (Dauparas et al., 2022). Formally, this is node-level classification task where the model learns a mapping for each residue r; to an amino acid type y E (1,...,n), where n is the vocabulary size (n = 20 for the canonical set of amino acids). Inverse folding is a generic task that can be applied to any dataset in the benchmark. In the literature, it is commonly evaluated on the CATH dataset (Section 2.4) compiled by Ingraham et al. (2019).
PPI Site Prediction. Identifying protein ppreein interaction sites has important applications in developing refined protein-protein interaction networks and docking tools, providing biological context to guide protein engineering and target identification in drug discovery campaigns (Jamasb et al., 3021). This task is a node-level binary classification task where the goal is to predict whether or not a residue is involved in a protein-protein interaction interface. We the dataset of experimental structures curated from the PDB by Gainza et al. (2020) and retain the original splits, though we modify the labelling scheme to be based on inter-atomic proximity (3.5 A), which can be user-defined, rather than solvent exclusion. The dataset is curated from the PDB by preprocessing such as the presence of one of the seven specified ligands (e.g., ADP or FAD), clustering based on 30% sequence contains 1,459 structures, which are randomly assigned to
identity and random subsampling. training (72%), validation (8%) and test (20%). 12 (A) radius patches were extracted from the generated structures and patch labelled as part of binding pocket if its centre point was < 3 (Å) away from an atom of the corresponding ligand.
Metal Binding Site Prediction. Many proteins coordinate transition metal ions to carry their functions. Predicting the binding sites metal ions can elucidate the role of metal binding on protein function. This is a binary node classification task where each residue is mapped to label y {0, 1) indicating whether the residue (or its constituent atoms) is within 3.5 (Å) of user-defined metal ion or ligand heteroatom, respectively. We provide tooling to curate a dataset of experimental structures from the PDB for this task, where binding site assignments for each residue are computed on-the-fly. While the benchmark supports this task on arbitrary subsets of the PDB and ligands, we provide the Zinc-binding dataset from Dürr et al. (2023) specifically for this task. The dataset constructed by sequence- siquedce-based clustering of the PDB at 30% sequence identity to remove sequence and structural redundancy, Clusters with a member shorter than 3000 residues, containing at least one zinc atom with resolution better than 2.5 (Å) determined by x-ray crystallography and not containing nucleic acids are used to compose the dataset. If multiple structures fulfil these criteria, the highest resolution structure used. The train (2,085) validation (26) test (59) splits are constructed such that proteins in the validation and test sets have no partial overlap with any protein in the training data.
Tess-Traallaiilll Modification Site Prediction. Identifying the precise sites where post- translational modifications (PTMs) occur is essential for understanding protein behaviour and designing targeted therapeutic interventions. We frame prediction of PTM sites as a multilabel classification task where each residue is mapped to a label y {1..... 13) distinguishing between modifications on different amino acids (c.g. phosphorylation on S/T/Y and N-linked glycosylation on N). We use dataset of 48,811 (alaaaooll--pedicett structures curated by Yan et al. (2023),
4

---

Published as a conference paper at ICLR 2024
where each structure contains the PTM metadata necessary construct residue- site prediction labels. The dataset is split into training (43,907, validation (2,393) and test (2,511) sets based on 50% sequence identity and 80% coverage.
2.3.2 GRAPH-LEVEL TASKS
Fold Prediction. The utility of this task is that it serves as a litmus test for the ability of a model to distinguish different structural folds. It stands to reason that models that perform poorly
distinguishing fold classes likely learn limited or low-quality structural representations. This is a multiclass graph classification task where each protein, G, is mapped to label y E (1 (1..... 1195)
denoting the fold class. We adopt the fold classification dataset originally curated from SCOP 1.75 by (Hou et al., 2017). This provides three different test sets stratified based on topological similarity: Fold, in which proteins originating from the same superfamily are absent during training; Superfamily, in which proteins originating from the same family are absent during training; and Family, which proteins from the same family are present during training.
Gene Ontology Prediction. Predicting protein function in the form of functional annotations such as GO terms has important applications in protein analysis and engineering, providing researchers with the ability to cluster functionally-related structures or to guide protein generation methods to design new proteins with desired functional properties. This is multilabel classification task, assigning functional Gene Ontology (GO) annotation to structures. GO annotations are assigned within three ontologies: biological process (BP), cellular component (CC) and molecular function (MF). We use the dataset of experimental structures curated from the PDB by Gligorijević et al. (2021) and retain the original multi-cutof based splits, with cutoff at 30% sequence similarity.
Reaction Class Prediction. As proteins' reaction classifications are based on their enzyme- cattlyyed reaction according to all four levels of the standard Enzyme Commission (EC) number, methods that predict such classifications may help elucidate the function of newly-designed proteins as they are developed. This is multiclass graph classification task where each protein, G. mapped to label E {I,.., .... 384} denoting which class of reactions a given protein catalyzes; all four levels of the EC assignment are employed to define the reaction class label. We adopt the reaction class prediction dataset originally curated from the PDB by Hermosilla et al. (2020), split on the basis of sequence similarity using a 50% threshold.
Antibody Developability Prediction. Therapeutic antibodies must be optimised for favourable physicochemical properties in addition to target binding affinity and specificity to be viable de- velopment candidates. Consequently, we frame prediction of antibody developability as binary graph classification task indicating whether a given antibody is developable, We adopt the antibody developability dataset originally curated from SabDab (Dunbar et al., 2014) by Chen et al. (2020). This dataset contains 2,426 antibodies that have both sequences and PDB structures available, where each example contains both a heavy chain and a light chain with resolution < (Å). The label based on thresholding the developability index (DI) (Lauer et al., 2012) as computed by BIOVIA's platform (Systèmes, 2016), which relics on an antibody's hydrophobic and electrostatic interactions. This task is interesting from a benchmarking perspective as it enables targeted performance assessment of models on a specific (immunoglobulin) fold, providing insight into whether general-purpose muueture-based encoders be applicable to fold-specific tasks.
2.4 PRE-TRAINING DATASETS
The benchmark contains several large corpora of both experimental and predicted structures that can be used for pretraining inference. We provide utilities for configuring supervised tasks and splits directly from the PDB. Additionally, we build storage-efficins pretraining corpora of predicted structures (AlphaFoldDB. ESM Atlas). We believe codebase will considerably reduce the barrier to entry for working with large structure- baased datasets.
dataloaders for large
2.4.1 EXPERIMENTAL STRUCTURES
PDB. We provide utilities for curating datasets directly from the Protein Data Bank (Berman, 2000). In addition to using the collection its entirety, users can define filters to subset and split the data using combination of structural similarity, sequence similarity or temporal strategies. Structures can
5

---

Published as a conference paper at ICLR 2024
be filtered by length, number of chains, resolution, deposition date, presence/absence of particular ligands and structure determination method.
CATH. We provide the dataset derived from CATH 4.2 40% (Knudsen & Wiuf, 2010) non-redundant chains developed by Ingraham et al. (2019) as an additional, smaller, pretraining dataset.
ASTRAL. ASTRAL (Brenner, 2000) provides protein domain structures which are regions of proteins that can maintain their structure and function independently of the rest of the protein. Domains typically exhibit highly-specifie functions and can be considered structural building blocks.
2.4.2 PREDICTED STRUCTURES
AlphaFoldDB Representative Structures. This dataset contains 2.27 million representative struc- tures, identified through large-scale sigguuall-smmilrrt--bbssss clustering of the 214 million structures contained in the AlphaFold Database (Varadi al., 2021) using FoldSeek (van Kempen et al., 2023). We additionally provide a subset of this collection - the so-called dark proteome - corresponding the 31% % of the representative structures that lack annotations.
ESM Atlas, ESM High Quality. These datasets are compressed collections of predicted structures produced by ESMFold. ESM Atlas is the full collection of all 772m predicted structures for the MGnify 2023 release (Richardson et al., 2022). ESM High Quality is a curated subset of high confidence (mean pLDDT) structures from the collection.
METHODS AND EXPERIMENTAL SETUP
Overview. To demonstrate the utility of our benchmark, we investigate how combinations of protein structure representation, architecture choice and pretraining/auxilarry tasks affect predictive performance across range of tasks. The tasks are selected to focus on important real-world structural annotation tasks and such that we can evaluate these combinations terms of both the local and global representational power. To this end, we select state-of-the- protein structure encoders and generic geometric GNN architectures that span the design space of geometric GNN models with regard to both message passing body order and tensor order (Joshi et al., 2023a). We evaluate several structural representations that, to varying degrees, capture the full detail of the protein structure.
Architectures. We provide unified implementation of several rotation invariant and equivariant architectures. We benchmark general purpose models: SchNet (Schütt et al., 2018), EGNN (Satorras et al. 2021), TFN (Thomas et al., 2018), MACE (Batatia et al., 2022); and 2 protein-specific architectures: GCPNet (Morchead & Cheng, 2024), GearNet (Zhang et al., 2023b). We also compare geometric GNNs to the pretrained sequence-based language model ESM (Lin et al., 2022) augmented with structural featurisation. We chose the 650M pretrained ESM-2 because this is the scale at which significant structure-related abilities were observed for ESM
Featurisation Schemes. We consider five featurisation schemes, progressively increasing the amount of structural information provided the model by incorporating sequence positional information, virtual dihedral and bond angles over the Ca trace, backbone torsion angles, and sidechain torsion angles. Featurisation schemes are detailed in Table in the Appendix.
Pretraining Dataset. For all pretraining experiments we use AlphaFoldDB Herrin- al., 2023). This dataset provides rich diversity of 2.27 million non-redundant protein structures and, to our knowledge, is substantially more diverse than any other previously used structure- based pretraining corpus, whilst remaining of size that is amenable to experimentation. Models pretrained on AlphaFoldDB should, in principle, exhibit strong generalisation to the currently known (and predicted) natural protein structure universe as it would have 'seen' the same protein fold during pretraining. To facilitate working with large-scale AlphaFoldDB and ESM Atlas, we developed storage-efficient dataloaders based on FoldComp (Kim et al., 2023), described in Appendix D.6.
Pretraining and Auxiliary Tasks. In our evaluation, we focus predominantly on denoising based pretraining and auxiliary tasks as these are comparatively less explored than contrastive or masked- attribute prediction tasks (Zhang et al., 2023b). We consider five pretraining tasks: (1) structure-bait based denoising, (2) sequence denoising, (3) torsional denoising,
(4) inverse folding and (5) PLDDT prediction. Structure and sequence denoising also used as auxiliary tasks in our experiments. We

---

Published as a conference paper at ICLR 2024
also investigate an inverse folding pre-trale mee-triiing task which we subsequently finetune the CATH dataset for benchmarking inverse folding as a downstream task (see below).
Noising Schemes. For structure- denoising draw i..d. noise samples from Gaussian distri- bution and scale by a = 0.1 to corrupt the input coordinates or dihedral angles. Geometric scalar and vector-valued features are computed from the noised structure, í.e. ف = (V. E, X, S, V), where o; = 2; + oe, and e ~ (0, Iz). For eqquence-based denoising, use the mutation strategy and corrupt 25% of the residues in each protein. When sequence denoising is used as an auxiliary task, we weight the loss with coefficient À= 0.1, similar to NoisyNodes (Godwin et al., 2021).
Training. As we are interested benchmarking large-scale datasets and models, we try to consistently use six layers for all models, each with 512 hidden channels. For equivariant GNNs, we reduced the number of layers and hidden channels to fit 80GB of GPU memory on one NVIDIA A100 GPU. For downstream tasks, we set the maximum number of epochs to 150 and use the Adam optimizer with a batch size of 32 and ReduceLLROPlltteer validation metric with patience of epochs and reduction of 0.6. See Appendix D.10 for details on hyperparameter tuning for optimal learning rates and dropouts for each architecture. We train models to convergence, monitoring the validation metric and performing early stopping with a patience of 10 epochs. Pretraining is performed for 10 epochs using a linear warm-up with cosine schedule. We report standard deviations over three runs across three random seeds.
learning rate scheduler, monitoring the
4 RESULTS & DISCUSSIONS
4.1 AUXILIARY DENOISING CONSISTENTLY IMPROVES BASELINE PERFORMANCE
In Table 2, first set out determine the following questions about architectural choices in conjunction with denoising auxiliary tasks and without pretraining:
1. Whether invariant or equivariant models perform better? Across 10 tasks, equivariant models such as EGNN and GCPNet attain the best performance on 5. Notably, sequence-based ESM-2-650M augmented with our structural featurisation matches tfate-of-the-ar protein- specific GNNs (Fan et al., 2023) on (super)family and gene ontology prediction.
2. Which input representation is the best for each respective task? Featurising models with Co atoms, virtual angles, and backbone torsions provides the best performance overall on 22 out of 60 combinations of models and tasks. This suggests that letting models implicitly learn about side chain orientation and flexibility by using backbone-only featurisation may prevent overfitting crystallisation artifacts (Dauparas et al., 2022).
3. Whether auxiliary denoising tasks improve model performance? Both sequence and struc- ture denoising are particularly useful auxiliary tasks for training protein structure encoders, until sufficient structural detail makes the tasks trivial, improving results over not using auxiliary tasks for 50 out of 60 combinations of models and primary tasks. Notably, structure denoising helped stabilise the training of MACE models the GO and Reaction tasks, where other runs did not converge.
4.2 INCORPORATING MORE STRUCTURAL DETAIL IMPROVES
PERFORMANCE
We then investigated protein structure pre-training in Table 3 to determine:
1. Which input representation is best for pre-training? Incorporating greater structural detail with dihedral angles generally improves validation metrics on pre-trate wve-training tasks, more so than architecture.
2. Which GNNs benefit from which pre-training task? Inverse folding, structure denoising, sequence denoising, and torsional denoising benefit equivariant models the most in the context of pre-training, whereas PLDDT prediction benefits invariant models the most, suggesting that certain pre- vre-trainig tasks benefit certain classes of models more than other tasks. Unfortunately, were currently unable to pre-train spherical equivariant GNNs (TEN, MACE) due to the high computational requirements of these models.
7

---

Published as a conference paper at ICLR 2024
Table 2: Baseline benchmark results without pretraining. Results for each model and featurisation pair are given as: no auxiliary task +structure denoising. Coloured boxes mark the best auxiliary tasks per model and featurisation, underlined results denote the best featurisation choice per model, and bold results are the best models for each task. Greyed cells denote invalid task-setup combinations (e.g. inverse folding and sequence denoising as auxiliary task), and - denote runs that did not converge. [*]] denotes results for ESM- lb without structural features, taken from Zhang et al. (2023b), whereas [f] denotes results for new tasks with ESM-2-650M. Key takeaways: (1) Auxiliary tasks consistently improve performance across models and primary tasks compared to performance without auxiliary tasks. (2) Equivariant GNNs outperform invariant GNNs in general though protein-specific architectures are consistently performant. (3) Ca, virtual angles, and backbone torsions provide the best fcaturisation. (4) Augmenting ESM-2 2-650M with structural features provides compelling performance for (super)family fold classification and gene ontology prediction compared to GNNs. For enhanced readability, in Section A of the Appendix, we provide series of analysis figures characterizing each task's 's results.
+sequence denoising
KJS AET B.S COMI-O0
BUTANOE MSSUEOO (184)008
OSSHER HAADNAAR FESO BXO
BUNIC SANAAA
DONI
X354 2000 1.55 DORCOO
MUSHAR /0036 90001
Gcatin
CBSIEX: 1a001600
(53866-00
AMILAAI (MHG-AL
(4390
25
31MIC.00 (2) (83000000 (1.0100
(11.310-0
semi 0279 (7.3600
EIMESI
2000
C%
201119
2112 300
$111
10325600
473644 (0:000000000001

---

Published as a conference paper at ICLR 2024
Table 3:
Validation performance for pretraining tasks on AlphaFoldDB. Met-
rics: Inverse Folding: perplexity; pLDDT, Structure Denoising, Torsional Denoising: RMSE;
Seq. Denoising: Accuracy. Best (second-best) results are bolded (underlined). Key takeaway:
Incorporating backbone structural features (i.e., adding torsion angles Φ 1 wi ), in general, im-
proves pretraining performance compared using only virtual angles along the sequence.
Task
Method
Inverse Folding ())
PLDDT Pred.
Structure Denoising ())
Sey. Denoising
Torsional Denoising ())
SchNet
7.791
0.2397
0.0704
36.81
0.0586
GeurNet- Eddge
6.596
0.2326
0.0672
43,76
0,0615
Co
EGNN
6.016
0.2406
0.0700
40.51
0.0586
GCPNet
6,243
0,2395
0.0679
44,81
0.0562
SchNet
5,562
0.2388
45.61
0.0489
GearNet. Edge
5.124
0.2402
0.0562
50.15
0.0533
EGNN
5.962
0.2403
0.0595
53,80
0.0487
GCPNet
3,839
0.2399
0.0561
$9.54
0.0443
4.3 PRE-TRAINING AND GREATER STRUCTURAL DETAIL. BENEFIT DOWNSTREAM TASKS
Following the observation that more fine-grained input representations improve pretraining perfor- mance, Table 4 explores finetuning on downstream tasks:
Whether these lessons from pretraining translate downstream tasks? Equivariant GNNs outperform invariant GNNs in the majority of cases and generally benefit the most from pre- training on structure-basee based tasks, particularly when provided with greater structural detail in input features.
Which combination of parameters performs best downstream tasks? Overall, providing a greater amount of structural detail compared to strict Ca atom representation benefits downstream performance for both invariant and equivariant models. Notably, structure denoising generally improves downstream performance for both types of models.
Table 4: Pretrained model benchmark results. Results for each model and featurisation pair given as: no pretraining / sequence denoising / structure denoising except for inverse folding on CATH, which is pretrained with inverse folding on AFDB. Key takeaway: The equivariant GCPNet model benefits most from pretraining and maximum structural detail.
Fvataro
G0
GB-CC
Reackee
lavere Evidine
Fill 20.99
Peeis
41.90
12:50/
1:41
20.90
OCTRA
5 CONCLUSIONS
This work focuses on building a comprehensive and multi-task benchmark for protein structure representation learning. ProteinWorkshop provides unified implementation of large pretraining corpora, featurisation schemes, Geometric GNN models and benchmarking tasks to evaluate the effectiveness of protein structure encoding methods. Key findings include that structural pretraining. as well auxiliary sequence and structure denoising tasks, improve performance on wide range of downstream tasks and that incorporating more structural detail in featurisation improves performance. Our benchmark is flexible for including new tasks and datasets and is open to the wider research community.
Availability, The Protein Workshop codebase is available under permissive MIT License at github. wow/aaooi//rooteeooeoooooooooeee inMorkshop and accompanying documentation, preprocessed datasets and pretrained model weights hosted publicly proteins. sh. Preprocessed datasets and pretrained model weights are deposited on Zenodo at the following URLs, respec- tively: zenodo. oroo hetpg//eerd////eeeeee/ooooooo 8282470 and zenado. org rtt./d/coordeeeeee 822.77..///877.....

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We acknowledge that this work was supported by a variety of institutions. ARJ was funded by Biotechnology and Biological Sciences Research Council (BBSRC) DTP studentship (BB/M011194/1) AM and JC were supported by U.S. NSF grant (DB12308699) and two U.S. NIH grants 13093133 and R01GM146340), CKJ was supported by the A Singapore National Science Scholarship (PhD). This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (www csd3 cam ac. uk), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/T022159/D), and DiRAC funding from the Science and Technology Facilities Council (www dirac ac uk). Additionally, this work was performed using high performance computing infrastructure provided by Research Support Services at the University of Missouri-Columble (DOI: We thank Martin Steinegger and Do- Yoon Kim for allowing us to use the illustrations Figure 1.
BROADER IMPACTS
Our benchmark unifies protein representation learning tasks, large-scale pre-training datasets, fea- turisation schemes, and models. The wide range of tasks studied in our benchmark can enable us to develop insight into effective pre-training strategies, and whether pre-trained protein structural representations can have material impact in real-world computational biology and drug discovery. It is not lost on us that these models can play role in developing harmful chemical matter in the hands of bad actor. Additionally, training very large models can contribute to climate change. However, we hope that developing highly effective structural representations will have broad, positive implications across biology and medicine that significantly outweigh the potential for misuse.
REFERENCES
Mohammed AlQuraishi. Parallelized natural extension reference frame: Parallelized conversion from internal to cartesian coordinates. Journal of Computational Chemistry, 2019. (Cited page 24)
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 2021. (Cited on page 1, 23)
Inigo Hernandzr Jingi Yeo, Jürgen Jänes, Tanita Wein, Mihaly Varadi, Sameer Velankar, Pedro Beltrao, and Martin Steinegger. Clustering predicted structures at the scale of the known protein universe. Bioinformatics, 2023. doi: 10. 0001/003..0300.5311111
(Cited on page 6)
Ilyes Batatia, Dávid Péter Kovács, Gregor NC Simm, Christoph Ortner, and Gábor Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. In NeurlPS, 2022. (Cited on page 2, 6, 23)
H. M. Berman. The protein data bank. Nucleic Acids Research, 2000. doi: 10. 10.10093/ar/28...255 (Cited on page 5, 23)
Fergus Boyles, Charlotte M Deane, and Garrett M Morris. Learning from the ligand: using ligand- based features to improve binding affinity prediction. Bioinformatics, 2019. (Cited on page 25)
Anthony R. Bradley, Alexander S. Rose, Antomín Pavelka, Yana Valasatava, Jose M. Duarte, Andreas Prlić, and Peter W. Rose. MMTF-an efficient file format for the transmission, visualization, and analysis of macromolecular structures. PLOS Computational Biology, 2017. (Cited page 23)
S. E. Brenner. The ASTRAL compendium for protein structure and sequence analysis. Nucleic Acids Research, (6(1):254-255 January 2000. (Cited on page 6, 24)
Martin Buttenschoen, Garrett M Morris, and Charlotte M Deane. Posebusters: Ai-based docking methods fail to generate physically valid poses or generalise to novel sequences. arXiv preprint arXiv:2308. 05777, 2023. (Cited page 20)
10

---

Published as a conference paper at ICLR 2024
Henriette Capel, Robin Weiler, Maurits Dijkstra, Reinier Vleugels, Peter Bloem, and K. Anton Feenstra. ProteinGLUE multi-task benchmark suite for self-supervised protein modeling. Scientific Reports, 12(1), September 2022. (Cited on page 19)
Chen Chen, Xiao Chen, Alex Morehead, Tianqi Wu, and Jianlin Cheng. 3d-equivariant graph neural networks for protein model quality assessment. Bioinformatiis, 39(1):btad030, 2023. (Cited on page 19)
Xingyao Chen, Thomas Dougherty, Chan Hong, Rachel Schibler, Yi Cong Zhao, Reza Sadeghi, Naim Matasci, Yi-Chich Wu, and lan Kerman. Predicting antibody developability from sequence using machine learning. bioRxiv, June 2020. (Cited on page 30)
Gabriele Corso. Modeling molecular structures with intrinsic diffusion models. arXiv preprint 12X2: 2023. (Cited on page 19)
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking. In ICLR, 2023. (Cited on page 2, 23)
Bowen Dai and Chris Bailey-Kellogg. Protein interaction interface region prediction by geometric deep learning. Bioinformatics, (1177))2580-5883 2021. (Cited page 19)
Christian Dallago, Jody Mou, Jody Mou, Kadina Johnston, Bruce Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021. (Cited page 19)
Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. King, and D. Baker. Robust deep learning-bar based protein sequence design using ProteinMPNN, Science, 2022. (Cited on page 4, 7, 19, 20)
Performance and structural coverage of the latest, in-development alphafold model. DeepMind, 2023. (Cited on page 20)
James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, 42 (D1): 140-D11 146, 2014. (Cited on page 30)
Simon L. Dürr, Andrea Levy, and Ursula Rothlisberger. Metal3d: a general deep learning framework for accurate metal ion location prediction in proteins. Nature Communications, 14(1), May 2023.
(Cited on page 4, 28)
Alexandre Duval, Simon V Mathis, Chaitanya K Joshi, Victor Schmidt, Santiago Miret, Fragkiskos D Malliaros, Taco Cohen, Pietro Liò, Yoshua Bengio, and Michael Bronstein. A hitchhiker's guide to geometric gnns for 3d atomic systems. arXiv preprint, 2023. (Cited page 1, 21)
Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 2023. (Cited on page 19)
Stephan Eismann, Patricia Suriana, Bowen Jing. Raphael JL Townshend, and Ron O Dror. Protein model quality assessment using rotation- equtiioneuuvvaiiant, hierarchical neural networks. arXiv preprint 1.Xi::2011.13551: 2020. (Cited page 19)
Sara El-Gebali, Jaina Mistry, Alex Bateman, Sean R Eddy, Aurélien Luciani, Simon C Potter, Matloob Qureshi, Lorna J Richardson, Gustavo A Salazar, Alfredo Smart, Erik L L Sonnhammer, Layla Hirsh, Lisanna Paladin, Damiano Piovesan, Silvio C E Tosatto, and Robert D Finn. The pfam protein families database in 2019. Nucleic Acids Research, 47(D1):D42- October 2018. (Cited on page 19)
William A Falcon. Pytorch lightning. GitHub, 2019. (Cited on page 20)

---

Published as a conference paper at ICLR 2024
Hehe Fan, Zhangyang Wang. Yi Yang. and Mohan Kankanhalli. Continuous-diserete convolution for geometry-sequeene modeling in proteins. In The Eleventh International Conference on Learning Representations, 2023. (Cited on page 7)
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint, 2019. (Cited on page 20)
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):144-19,, 2020. (Cited on page 2, 4, 19, 26, 27)
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. ICLR, 2020. (Cited on page 1, 21)
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 2021. (Cited on page 25)
Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXi::2207.0443 2022. (Cited page 20)
Vladimir Gligorijević, P. Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C. Taylor, Ian M. Fisk, Hera Vlamakis, Ramnik J. Xavier, Rob Knight, Kyunghyun Cho, and Richard Bonneau. Structure-based protein function prediction using graph convolutional networks. Nature Communications, 2021, (Cited on page 2, 5, 19, 26, 29)
Jonathan Godwin, Michael Schaarschmidt, Alexander Gaunt, Alvaro Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple gnn regularisation for 3d molecular property prediction & beyond. arXiv preprint, 2021. (Cited on page 7, 19)
Yulia
Joe Greener and Kiarash Jamali. Fast protein structure searching using structure graph embeddings. bioRxiv, 2022. (Cited on page 1)
Charles Harris, Kieran Didi, Arian Jamasb, Chaitanya Joshi, Simon Mathis, Pietro Lio, and Tom Blundell. Posecheck: Generative models for 3d structure-based drug design produce unrealistic poses. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023. (Cited on page 20)
Pedro Hermosilla, Marco Schäfer, Matěj Lang. Gloria Fackelmann, Pere Pau Vázquez. Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. arXiv preprint 20XXi:00077000552. 2020. (Cited page 4, 5, 19, 26, 30)
Liisa Holm. Dali server: structural unification of protein families. Nucleic acids research, 2022. (Cited on page 1)
Jie Hou, Badri Adhikari, and Jianlin Cheng. DeepSF: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 2017. (Cited on page 4, 5, 26, 29)
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021. (Cited on page 4, 19, 26)
Yufei Huang, Lirong Wu, Haitao Lin, Jiangbin Zheng, Ge Wang, and Stan Z Li. Data-efficient protein 3d geometric pretraining via refinement of diffused protein structure decoy. arXiv preprint arXiv:2302. 10888, 2023. (Cited on page 19)
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. In NeuriPS, 2019. (Cited on page 6, 23, 26, 27)
12

---

Published as a conference paper at ICLR 2024
Arian R. Jamasb, Ben Day, Cätälina Cangea, Pietro Liò, and Tom L. Blundell. Deep learning for protein protein-protein protein interaction site prediction. In Methods in Molecular Biology. pp. 263-288 Springer US, 2021. (Cited on page 4, 27)
Arian Rokkum Jamasb, Ramon Viñas Torné, Eric J Ma, Yuanqi Du, Charles Harris, Kexin Huang, Dominic Hall, Pietro Lio, and Tom Leon Blundell. Graphein a python library for geometric deep learning and network analysis on biomolecular structures and interaction networks. In NeurIPS, 2022. (Cited on page 20)
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. arXiv preprint wrXiv:2110.04624, 2021.
(Cited on page 19)
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometrie vector perceptrons. In ICLR, 2020. (Cited on page 1, 19)
Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Liò. On the expressive power of geometric graph neural networks. In International Conference on Machine Learning. 2023a. (Cited on page 1, 6, 21)
Chaitanya K Joshi, Arian R Jamasb, Ramon Viñas, Charles Harris, Simon V Mathis, Alex Morehead, and Pietro Liò. grnade: Geometric deep learning for 3d ma inverse design. arXiv preprint, 2023b. (Cited on page 20)
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, et al. Highly accurate protein structure prediction with alphafold. Nature, 2021. (Cited on page 1, 24)
Hyunbin Kim, Milot Mirdita, and Martin Steinegger. Foldcomp: a library and format for compressing and indexing large protein structure sets. Bioinformatics, 39(4), March 2023. (Cited on page 6, 24)
Michael Knudsen and Carsten Wiuf. The CATH database. Human Genomics, 4(3):207, 2010. (Cited on page 6, 23)
Rohith Krishna, Jue Wang, Woody Ahern, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix Morey-Burroww Ivan Anishchenko, Ian Humphreys, et al. Generalized biomolecular modeling and design with rosettafold all-atom. bioRxiv, 2023. (Cited on page 20)
Timothy M. Lauer, Neeraj J. Agrawal, Naresh Chennamsetty, Kamal Egodage, Bernhard Helk, and Bernarrtt L. Trout. Developability index: A rapid silico tool for the screening of antibody aggregation propensity. Journal of Pharmaceutical Sciences, 101()):002-115, January 2012.
(Cited on page 30)
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
(Cited on page 2, 24)
Shengchao Liu, Weitao Du, Yanjing Li, Zhuoxinran Li, Zhiling Zheng, Chenru Duan, Zhiming Ma, Omar Yaghi, Anima Anandkumar, Christian Borgs, et al. Symmetry-informed geometric representation for molecules, proteins, and crystalline materials. arXiv preprint 09X7552306.09355 2023a. (Cited on page 20)
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with SE(3)-invariant de- noising distance matching. In The Eleventh International Conference on Learning Representations, 2023b. (Cited on page 20)
Sajid Mahmud, Alex Morehead, and Jianlin Cheng. Accurate prediction protein tertiary structural changes induced by single-site mutations with equivariant graph neural networks. bioRxiv, pp. 2023-10, 2023. (Cited on page 19)
Alex Morehead and Jianlin Cheng. eeometryycompleer diffusion for 3d molecule generation. In ICLR Machine Learning for Drug Discovery 2023, 2023. (Cited on page 22)
13

---

Published as a conference paper at ICLR 2024
Alex Morehead and Jianlin Cheng. Deometry-complee perceptron networks for 3d molecular graphs. Bioinformatics, 2024. (Cited on page 2, 6, 19, 22)
Alex Morehead, Chen Chen, and Jianlin Cheng. Geometric transformers for protein interface contact prediction. In International Conference on Learning Representations, 2022. (Cited on page 1)
Alex Morehead, Chen Chen, Ada Sedova, and Jianlin Cheng. Dips-plus: The enhanced database of interacting protein structures for interface prediction, Scientific Data (Nature), 2023a. (Cited on page 19)
Alex Morehead, Jeffrey A Ruffolo, Aadyot Bhatnagar, and Ali Madani. Towards joint sequence- structure generation of nucleic acid and protein complexes with se(3)-discrete diffusion. NeurIPS 2023 Workshop on Machine Learning in Structural Biology, 2023b. (Cited on page 20)
Alex Morehead. Jian Liu, and Jianlin Cheng. Protein structure accuracy estimation using geometry- complete perceptron networks. Protein Science, 2024. (Cited on page 19)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, gegh-performncce deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. (Cited on page 20)
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. (Cited on page 19)
Lorna Richardson, Ben Allen, Germana Baldi, Martin Beracochea, Maxwell I. Bileschi, Tony Burdett, Josephine Burgin, Juan Caballero-Pérez, Guy Cochrane, Lucy J Colwell, Tom Curtis, Alejandra Escobar-Zepeda, Tatiana A Gurbich, Varsha Kale, Anton Korobeynikov, Shriya Raj. Alexander B Rogers, Ekaterina Sakharova, Santiago Sanchez, Darren J Wilkinson, and Robert D Finn. MGnify: the microbiome sequence data analysis resource in 2023. Nucleic Acids Research, 2022. (Cited on page 6, 24)
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. In ICML, 2021. (Cited page 2, 6. 22)
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, et al. Structure-basee drug design with equivariant diffusion models. arXiv preprint, 2022. (Cited on page 2)
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In ICML, 2021. (Cited on page 1)
Kristof T Schütt, Huziel Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 2018. (Cited on page 1, 2, 21)
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. Advances in Neural Information Processing Systems, 34, 2021. (Cited on page 19)
Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In CVPR, 2021. (Cited on page 19)
Dassault Systèmes. Biovia discovery studio, 2016. (Cited on page 30)
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-an and cranslationeeqiiariiin neural networks for 3d point clouds. arXiv preprint, 2018. (Cited page 2, 6, 23)
14

---

Published as a conference paper at ICLR 2024
Raphael Townshend, Martin Vögele, Patricia Suriana, Alex Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon Anderson, Stephan Eismann, Risi Kondor, Russ Altman, and Ron Dror. Atom3d: Tasks on molecules in three dimensions. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021. (Cited page 19)
Michel van Kempen, Stephanie S. Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron L. M. Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature Biotechnology, 2023. (Cited on page 1, 6, 24)
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, Augustin Zídek, Tim Green, Kathryn Tunyasuvunakool. Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, Michael Figurnov, Andrew Cowie, Nicole Hobbs, Pushmeet Kohli, Gerard Kleywegt, Ewan Birney, Demis Hassabis, and Sameer Velankar. AlphaFold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 2021. (Cited page 1, 6, 24)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. (Cited on page 21)
Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. In ICLRR 2023. (Cited on page 19)
Carter Wilson, Wing-Yin Yiu Choy, and Mikko Karttunen. Alphafold2: a role for disordered pro- tein/region prediction? International Journal of Molecular Sciences, 23(9):4591, 2022. (Cited on page 3, 25)
Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Ma Chang, Runcheng Liu, and Jian Tang. Peer: A comprehensive and multi-task benchmark for protein sequence understanding. In NeurIPS, 2022. (Cited on page 19)
Omry Yadan. Hydra- a framework for elegantly configuring complex applications. Github, 2019. (Cited on page 20)
Yu Yan, Jyun- Yu Jiang, Mingzhou Fu, Ding Wang, Alexander R. Pelletier, Dibakar Sigdel, Do- minic C.M. Ng, Wei Wang, and Peipei Ping. MIND-s is a deep-lcarning prediction model for elucidating protein post-translational modifications in human diseases. Cell Reports Methods, 2023. (Cited on page 4, 26, 28)
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33, 2020. (Cited on page 17)
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. The Eleventh International Conference on Learning Representations, 2023. (Cited on page 19)
Zuobai Zhang, Minghao Xu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. Enhancing protein language models with structure- based encoder and pre-training, arXiv preprint arXiv:2303.06275 2023a. (Cited on page 20)
Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In The Eleventh International Conference on Learning Representations, 2023b. (Cited on page 1, 2, 3, 6, 8, 19, 29,21,225 25)
Zuobai Zhang, Minghao Xu, Aurélie Lozano, Vijil Chenthamarakshan, Payel Das, and Jian Tang. Physics-inspired protein encoder pre-training via siamese sequence-structure diffusion trajectory prediction, 2023c. (Cited on page 25)
Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengehao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, et al. Torchdrug: A powerful and flexible machine learning platform for drug discovery. arXiv preprint, 2022. (Cited on page 19, 20)
15

---

Published as a conference paper at ICLR 2024
APPENDICES
Illustrated Results
17
Related Work
19
C Discussion and Future Work
19
D Protein Workshop User Manual
20
D.I Dependencies
20
D.2 Usage
20
D.3 Computational Resources
20
D.4 Featurisation Schemes
20
D.5 Protein Structure Encoder Architectures
21
D.6 Pretraining Datasets
23
D.7 Pretraining Tasks
24
D.8 Downstream Tasks
25
D.9 SE(3) Equivariant Noise Predictor
25
D.10
Hyperparameter Selection
26
Documentation for Datasets
26
E.1
Inverse Folding.....
26
E,2
MaSIF-Site PPI Site Prediction
................................
27
E.3
ccPDB Metal Binding Site Prediction
28
E.4
PTM Pes.-oorasssaaee Trasstraanlationall Modification Site Prediction
................................
28
E.5
FOLD Fold Prediction...
29
E.6
GO Gene Ontology Prediction
29
E.7
EC Reaction Reaction Class Prediction
30
E.8
TDC Antibody Developability Prediction
30
16

---

Published as a conference paper at ICLR 2024
A ILLUSTRATED RESULTS
Following You et al. (2020), this section, we provide an alternative means of interpreting the main results in Table of the main text. Across Figures 2-10, illustrate the test metric stability of each encoder, featurization scheme, and auxiliary task across each dataset included in Table 2 via ranking analysis. For each configuration, we rank design choices by their performance, deeming performance to be tied when the difference < 0.02 (c 0.001 for PPI site prediction). We collect rankings over all configurations and report the mean ranking and their smoothed distribution for each task with bar and violin plots, respectively (lower is better). An average ranking score of indicates the design choice invariably results in the highest performance over other possibilities its category. The smoothed distributions provide further insight into the broader behaviour of a design choice.
Ontology
Parhesnten Banking Arerage
IPS
FIZK
TE.
16.00
Desacry Viscala Beedry
Encader
Fea ture
Auxilarr lask
Figure 2: Ranking analysis of Gene -iioooggg- Process (GO-BP) test performance across different encoders, feature sets, and auxiliary tasks.
Gene Ontology
Oerroraio Banking Average
PIXE
Sccurnc leoory Jncar Beresig
Encader
TCRE
Secenic Lessarg Siscare bersang Auxilory
Figure 3: Ranking analysis of Gene Ontology-Molest Molecular Function (GO-MF) test performance across different encoders, feature sets, and auxiliary tasks.
Gene Ontology CC
Pertemae Ranking Nverage
Encoder
Feature
Auxilirry bask
Figure 4: Ranking analysis of Gene Ontology Ontlllay-Cellular Component (GO-CC) test performance across different encoders, feature sets, and auxiliary tasks.
Antibody Developability
Periiutton Panking
fiver age
2o7i Mertintee Mil Wol IIS
FIXE
Iere
ScREPT Croory Sicare Boreing
Encoder
Feature
Tewo
Auxiliary Task
Figure 5: Ranking analysis of Antibody Developability test performance different encoders, feature sets, and auxiliary tasks.
17

---

Published as a conference paper at ICLR 2024
Fold Classification Fold
Discrnuttin Ranking
hirereg
SOree Ceatialiige BAN
Auxiliary
Socuatc banuorg Siscara Dereary
Figure 6: Ranking analysis of Fold Classification-olle test performance across different encoders, feature sets, and auxiliary tasks.
Family
destoittoon Ranking
Averege
166
1800
72k
Ieee
Sceaent Leorr Siecarg bereirg
Encader
Festure
Task
Siecre bersang
Figure 7: Ranking analysis of Fold Classification- test performance across different encoders, feature sets, and auxiliary tasks.
Fold Classification Superfamily
Bestemman Ranking Pvee age
Encaddr
Fea ture
Auxilory bask
Figure 8: Ranking analysis of Fold Clasification-Supefaailll test performance across different encoders, feature sets, and auxiliary tasks.
Reaction Classification
Riutenn Barking
org
17%
PICK
Seteing Lanaarg Siscara berearg Auxiliary
Figure 9: Ranking analysis of Reaction Classification test performance across different encoders, feature sets, and auxiliary tasks.
Prediction
Performance Banking
Dittybunte Average
PICE
T/.
sican'co becaera Anndarr
Sear Teraace liary
Encoder
Figure 10: Ranking analysis of Protein- Proeein Interaction (PPI) Site Prediction test performance across different encoders, feature sets, and auxiliary tasks.
18

---

Published as a conference paper at ICLR 2024
B RELATED WORK
Protein Structure Representation Learning. Several structure- encoders for proteins have been designed extract information from different levels of granulariy, such as residue-level, atom- level and surfaces. Previous works have aimed to encode protein structural priors directly within architectures model proteins hierarchically (Somnath al., 2021; Hermosilla et al., 2020), as computationally efficient point clouds (Gainza et al., 2020; Sverrisson et al., 2021), or as geometric graphs (Jing et al., 2020; Jin et al., 2021; Morehead Cheng, 2024; Wang et al., 2023; Zhang et al., 2023b; Mahmud et al., 2023) for tasks such protein function prediction (Gligorijevié et al., 2021), protein model quality assessment (Eismann et al., 2020; Chen et al., 2023; Morehead et al., 2024), and protein interaction region prediction (Dai & Bailey-Kellogg, 2021; Morehead et al., 2023a).
Protein Benchmarks. Several benchmarks have been proposed for evaluating the efficacy of learnt protein sequence representations. However, gructure-based benchmarks are comparatively unaddressed. Rao et al. (2019) developed TAPE (Tasks Assessing Protein Embeddings), providing a large pretraining corpus of protein sequences curated from Pfam (EI-Gebali et al., 2018), well as a collection of five supervised benchmark tasks assessing the ability of protein language models to predict structural qualities (contact prediction and secondary structure prediction), and functional properties (fuuresccence and stability prediction), Xu et al. (2022) proposed PEER (Protein Sequence Understanding). focussing on multitask evaluation of protein sequence models. Therapeutic Data Commons (Huang et al., 2021) provide several datasets relevant to therapeutic development, however the few protein structure-derived datasets it contains are cast as sequence- tasks. Dallago et al. (2021) developed FLIP, a requence-based benchmark of protein fitness landscapes. ProteinGLUE (Capel et al., 2022) is another sequenced- based benchmark focussing on per-residue tasks.
To our best knowledge, the only protein sencturr-benchmart to date is ATOM3D (Townshend et al., 2021), which proposes a collection of tasks largely assessing geometric methods at predicting graph-level properties of protein structures. TorchProtein (Zhu et al., 2022) also provides small collection of global-structura datasets. Most existing benchmarks do not exhaustively evaluate both the local and global representation learning power of proposed methods. As the field develops, identify a need for a consistent benchmarking framework of diverse tasks to ensure improving results reported in the literature map on to progress in the downstream problems we hope to address. Similar benchmarking efforts for general purpose GNNs have provided experimental rigour to architectural research (Dwivedi et al., 2023).
Denoising-Basee Pre-training and Regularisation. Several methods have been developed for pre- training GNNs, predominantly focussing on cases where 3D coordinate information is only implicitly encoded in the graph structures. In this work, we build on work by Godwin et al. (2021) and Zaidi al. (2023) to investigate whether denoising- baased auxillary and pre-training tasks are effective methods for pre-training geometric GNNs operating on protein structures, similar to concurrent works bridging the gap between denoising objectives for geometric neural networks and diffusion generative modeling for biomolecules (Huang al., 2023; Corso, 2023).
C DISCUSSION AND FUTURE WORK
Completeness of input featurisation. The extent to which providing complete information about all atoms and side chain orientations of each residue in a protein is debatable, as the exact coordinates from PDB files are known to contain artifacts from structure determination via crystallography, This was most recently noted by Dauparas et al. (2022) in the context of developing and experimentally validating an inverse folding model. Our benchmarking results provides similar insights at present, letting models implicitly learn about side chain orientation by using backbone-only featurisation performs better or equally well as explicitly providing complete side chain atomic information across both global and node level tasks.
On the choice of pre-trainng pre-training tasks. We currenly focussed on pre-training tasks that roughly fall under the category of denoising, i.e. corrupting information in the input (sequence identity, coordi- nates) and tasking the model with producing the uncorrupted input. We were particularly interested in self- sulf-super objectives that were (1) extremely scalable, so to pre-train on the sarge-scale AlphaFoldDB of 2.4M structures; and (2) train protein representations at the fine-grained node level, so to be general-purpose across the downstream tasks considered. We did not benchmark other
19

---

Published as a conference paper at ICLR 2024
tasks from the literature, such contrastive learning and generative moddlllng-inssiiree objectives (Liu et al., 2023b;a; Zhang et al., 2023a). Such tasks are generally (1) computationally heavier and more cumbersome to set up than corruption-typ objectives, making them harder to scale up, and (2) only train protein representations for the global/graph level and do not operate at the node level. Naturally, would like to continue exploring more pre-training strategies Protein Workshop.
continue expand
Beyond protein-only representation learning. Recently, Krishna et al. (2023) and DeepMind- Isomorphic (2023) generalised protein structure prediction models to full biological assemblies including proteins, small molecules, nucleic acids, and other ligands. Geometric graphs are natural choice for representation learning across biomolecular systems: Advances in geometric GNN methodology should, in principle, be adaptable to modelling other biomolecules (Joshi al., 2023b), their complexes (Morehead et al., 2023b), and quaternary structures.
We currently focus on protein representation learning because (1) large scale datasets for self- supervised learning, as well as well-defined downstream tasks, are readily available and accepted by the community; and (2) we see protein representation learning as a fundamental task, improving upon which should also advance the modelling of proteins in complex with other biomolecules. Comparatively, the scale of data available for biomolecular complexes is smaller and there less consensus among the community evaluation (Harris et al., 2023; Buttenschoen al., 2023).
D ProteinWorkshop USER MANUAL
D.1 DEPENDENCIES
The benchmark is developed using PyTorch (Paszke et al., 2019), PyTorch Geometric (Fey Lenssen, 2019), PyTorch Lightning (Falcon, 2019), and Graphein (Jamash et al., 2022). Experiment configuration is performed using Hydra (Yadan, 2019), Certain architectures introduce additional dependencies, such as TorchDrug rug (Zhu et al., 2022) and e3nn (Geiger Smidt, 2022).
D.2 USAGE
The modular design of our benchmark means it can be readily adapted into different workflows easily. Firstly, the benchmark is pi pip-installable from PyPI and contains several importable modules, such dataloaders, featurisers and models, that can be imported into projects. This will aid in standardising the datasets and workflows used in protein representation learning. Secondly, the benchmark serves as easily extendable template, which users fork and work directly in, thereby reducing implementation burden. Lastly, we provide a CLI that can be used to quickly run single experiments and hyperparameter sweeps with minimal development time overhead.
D.3
COMPUTATIONAL. RESOURCES
All models are trained on 80GB NVIDIA A100 GPUs. All baseline and finetuning results are performed using a single GPU while pre-training performed using four GPUs.
D.4 FEATURISATION SCHEMES
Protein structures are typically represented as graphs, with researchers typically opting to use a coarse-grained Co atoms graph as full atom representations can quickly become computationally intractable due to a large number of nodes. The extent to which coarse soasse-rrainig schemes are 'complete' representation of the geometry and structure of the protein residue is variable. For instance, backbone-only features ignore the orientations of the side chain atoms in the residue, so models must account for this information implicitly. However, providing complete information about all atoms and side chain orientations is debatable as exact coordinates from PDB files are known to contain crystallography artefacts (Dauparas et al., 2022). Due to the computational burden incurred by operating on full-atom node representations, we focus primarily Ca-based graph representations, investigating featurisation strategies to incorporate higher-Ievel structural information. Note that we do provide utilities to enable to work with backbone and full-atom graphs in the benchmark.
20

---

Published as a conference paper at ICLR 2024
We represent protein structures as geometric graphs, 9 EV, V), where V is set of nodes, is set of edges, X V|x3 is matrix of Cartesian node coordinates, [ matrix of d-dimension scalar node features, and V E R VVxxxx3 is a tensor vector-valued features.
The five scalar featurisation schemes considered in the baselines are provided in Table 5. Pretraining experiments are performed on the featurisation schemes rows three and four.
Table Structural featurisation schemes. Residue type is one-hot encoding of the amino acid type for each node; positional encoding is 16-dimensional transformer-like positional encoding (Vaswani et al., 2017); and o, w, w E g and X1-4 E IR are backbone dihedral angles and sidechain torsion angles, respectively, embedded on the unit circle. Similarly, K, E R' are virtual torsion and bond angles defined over Ca atoms. In our experimental evaluation, we consistently use k-NN edge construction with 16.
Granularity
Co Features
Backbone
Sidechain
Co
Residue Type
Ca
Residue Type, Positional Encoding
Ca
Residue Type, Positional Encoding, K, at
Ca
Residue Type, Positional Encoding,
d, th Ø, th
Ca
Residue Type, Positional Encoding, K, a
X1 X2. X3. Xa
Additionally, vector message passing methods such as GCPNet receive node orientation vectors (i.e., 0y--1 and 0gr+1) and edge directional vectors (i.e., Vgs)) ) as vector input features for nodes and edges, respectively. These vector features, prior to heing normalised unit vectors, are defined as:
= ē;-1 Z.. Joit1 E.+1 C. and Jas 5. zz..
D.5 PROTEIN STRUCTURE ENCODER ARCHITECTURES
We provide a unified implementation several rotation invariant and equivariant geometric GNNs, spanning the range of message passing body order and tensor order (Joshi al., 2023a), including general-purpose as well as protein-specific models. See Duval et al. (2023) for a self-contained introduction to geometric GNNs.
D.5.1 INVARIANT GNNS
SchNet (Schütt et al., 2018) SchNet is of the most popular and simplest instantiation of E(3) invariant message passing GNNs. SchNet constructs messages through element-wise multiplication of scalar features modulated by radial filter conditioned on the pairwise distance |ق,ي|| between two neighbours. Scalar features are update from iteration i to + via:
8 ((+1)
8 + Σ fi
|
(2)
JEA.
Hyperparameters: number of layers = 6, hidden channels 512.
DimeNet++ (Gasteiger et al., 2020) DimeNet is an E(3) invariant GNN which uses both distances | and angles Tey Za perform message passing among triplets, follows:
(++1)
Σ
(
fz
8
8
(x)
| Tej au
(3)
JEN:
REANDA
Hyperparameters: number of layers = 6, hidden channels 512.
GearNet-Edge (Zhang et al., 2023b) GearNet-Edge is an SE(3) invariant architecture leveraging relational graph convolutional layers and edge message passing. The original MaaNNet-Edge formula- tion presented in Zhang et al. (2023b) operates on multirelational protein structure graphs making use of several edge construction schemes (I:NN, euclidean distance and sequence distance based). Our
21

---

Published as a conference paper at ICLR 2024
benchmark contains full capabilitiies for working with multirelational graphs but single edge type (i.c. [R] 1) in our experiments to enable more direct architectural comparisons.
The relational graph convolutional layer is defined for relation type
(++1)
8, (1) +o
BN
Σ PER
W
Σ
JEAS(1)
The edge message passing layer defined for relation type r as:
m (8.j.r)
(++1)
=
BN
Σ
W
E
m (m,k.r2)
(5)
rEIRY
(++1)
BN
Σ FEJR
W
Σ JEN-(()
FC
(6)
where FC(-) denotes a linear transformation upon the message function. Hyperparameters: number layers 6, hidden channels 512.
D.5.2 EQUIVARIANT GNNS IN CARTESIAN COORDINATES
EGNN (Satorras et al., 2021) We consider E(3) equivariant GNN layers proposed by Satorras al. (2021) which updates both scalar features s; as well as node coordinates 2; as follows:
= J2
8
Σ JEN
J
8
:
.8j
|
4;
ij
@ S3
|
(8)
JEN,
Hyperparameters: number of layers = 6, hidden channels 512.
GCPNet (Morehead & Cheng, 2024) GCPNet is an SE(3) equivariant architecture that jointly learns scalar and vector-valued features from geometric protein structure inputs and, through the use of geometry-complete frame embeddings, sensitises its predictions to account for potential changes induced by the effects of molecular chirality on protein structure. In contrast to the original GCPNet formulation presented in Morehead & Cheng (2024), the implementation we provide in the benchmark incorporates the architectural enhancements proposed in Morehead Cheng (2023) which include the addition of a scalar message attention gate (i.e.. fa(-)) and simplified structure for the model's ' geometric graph convolution layers (i.e., fx(-)) With geometry-complee graph convolution in mind, for node i and layer f, scalar edge features our and vector edge features ea are used along with scalar node features " and vector node features (1) to update each node feature type as:
(8
(++1) mis
V ((+1)) = s((+1) (1s (00), (88) (
(f((+1) (t) (8 ( F.),
(9)
(1+1) (o m0 v
(1+1)) = p(t+1)
(8 20
Σ
(a (
((=1))
(10)
JEN())
where the geometry-complete and chirality-sensitivee local frames for node (i.e., its edges) are defined as F, (aij, bij, C.), with a; Hyperparameters: number of layers 6, hidden scalar channels 128. hidden vector channels 16.
=: - bij
xxx,; T= and Cij aij x bij, respectively.
22

---

Published as a conference paper at ICLR 2024
D.5.3 EQUIVARIANT GNNS IN SPHERICAL COORDINATES
Tensor Field Network (Thomas al., 2018) Tensor Field Networks are B(3) or SE(3) equivariant GNNs that have been successfully used in protein structure prediction (Baek et al., 2021) and protein- ligand docking (Corso et al., 2023). These models use higher order spherical tensors h.r E 2 as node features, starting from order
L. The first two orders correspond to scalar features 8, and vector features V, respectively. The higher order tensors h, are updated via tensor products @ of neighbourhood features h, for j NG with the higher order spherical harmonic representations Y of the relative displacement
up to arbitrary
Ey Beyt E
h
(++1)
h
+
Σ JENN
Y (ā.r) &w n
(1I)
where the weights w of the tensor product are computed via learnt radial basis function of the relative distance, i.e. w = f (1]
Hyperparameters: choice of symmetry SO(3), number of layers = spherical harmonic tensor order
= 2, hidden irreps per tensor type = 6 64x0e 64x0o + 8xle 8x1o + 4x2e 4x2o.
We were particularly interested in benchmarking the impact of higher order tensors and SO(3) equivariance.
MACE MACE (Batatia et al., 2022) a higher order E(3) or SE(3) equivariant GNN originally developed for molecular dynamics simulations. MACE provides an efficient approach computing high body order equivariant features in the Tensor Field Network framework via Atomic Cluster Expansion: They first aggregate neighbourhood features analogous to equation II (the functions in Batatia et al. (2022) (eq. 9)) and then take h 1 repeated self-tensor products of these neighbourhood features, In our formalism, this corresponds to:
h (++1) h (++1) 2r
2w h
(++1)
(12)
Hyperparameters: choice of symmetry = 0(3), number of layers = 2, spherical harmonic tensor order 2, hidden irreps per tensor type 32x0e 32xlo 32x2e, body order =. Note that the number channels for all tensor types must be the same for MACE, which is restrictive for scaling the depth and number of parameters.
D.6 PRETRAINING DATASETS
The benchmark contains several large corpora of both experimental and predicted structural data that can be used for pretraining or inference. We provide utilities for configuring supervised tasks and splits directly from the PDB (Berman, 2000). Additionally, large pretraining corpora of predicted structures including the AlphaFoldDB and ESM Atlas. We believe our codebase will considerably reduce the barrier entry for pretraining and working with large structure- based datasets.
build storage-efficient dataloaders for
D.6.1 EXPERIMENTAL STRUCTURES
PDB. We provide utilities for curating datasets directly from the Protein Data Bank (Berman, 2000). In addition to using the collection in its entirety, users can define filters to subset and split the data using combination of structural similarity, sequence similarity or temporal strategies. Structures can be filtered by length, number chains, resolution, deposition date, presence/absence of particular ligands and structure determination method. The benchmark supports working with PDB structures both pdb and mmt format (Bradley et al., 2017), which significantly reduces the requirements for data storage.
CATH. We provide the dataset derived from CATH 4.2 40% (Knudsen & Wiuf, 2010) non-redundant chains developed by Ingraham al. (2019) as additional, smaller, pretraining dataset. These data are split based on random assignment of the CATH topology classifications based on an 80/10/10 split.
23

---

Published as a conference paper at ICLR 2024
ASTRAL. ASTRAL (Brenner, 2000) provides compendia of protein domain structures, regions of proteins that can maintain their structure and function independently of the rest of the protein. Domains typically exhibit highly-specific functions and can be considered structural building blocks of proteins.
D.6.2 PREDICTED STRUCTURES
We provide ready-to-go dataloaders for several large-scale collections of predicted structures derived from both AlphaFold2 (Jumper et al., 2021) and ESMFold (Lin et al., 2022). This is facilitated by FoldComp (Kim et al., 2023), (minimally) lossy compression scheme for predicted protein structures. FoldComp stores protein structures as a collection of discretised dihedral and bond angles which can be used to reconstruct the whole structure using fixed bond lengths and canonical amino acid geometry. FoldComp achieves a disk-space reduction of almost an order of magnitude, describing residue with only 13 bytes down from 97 bytes per-residue in a traditional uncompressed format. Whilst lossy, this procedure results in 0.08 Á and 0.14 Å RMSD for backbone and all-atom reconstruction, making it highly suitable for pretraining tasks which use input representations complete up to the backbone. Furthermore, this lightweight format enables the dataloaders in the benchmark read structures directly from disk with no pre-processing caching required.
AlphaFoldDB Representative Structures. This dataset contains 2.27 million representative struc- tures, identified through large-scale srtucturalsimmilartt--boo based clustering of the 214 million structures contained in the AlphaFold Database (Varadi et al., 2021) using FoldSeek (van Kempen et al., 2023). We additionally provide subset of this collection the so-called dark proteome corresponding to the 31% % of the representative structures that lack annotations.
ESM Atlas, ESM High Quality. These datasets are compressed collections of predicted structures produced by ESMFold. ESM Atlas is the full collection of all 772m predicted structures for the MGnify 2023 release (Richardson al., 2022). ESM High Quality is curated subset of high confidence (mean PLDDT) structures from the collection.
D.7 PRETRAINING TASKS
The benchmark contains comprehensive suite of pretraining tasks. Broadly, these tasks can be categorised into: masked-attribute prediction, denoising- based and contrastive learning-basee tasks. In most cases, these tasks can be used both pretraining objective as auxiliary tasks in downstream supervised task.
Sequence Denoising. The benchmark contains two variations based
sequence corruption processes C($)S,v) that receive amino acid sequence S [0, 1] and return sequence SE (0, 1 || with fraction of its positions corrupted. The first scheme is based on mutating fraction of the residues to a random amino acid and tasking the model with recovering the uncorrupted sequence. The second is masked residue prediction task, where fraction of the residues are altered to mask value and the model is tasked to recover the uncorrupted sequence.
Structure Denoising. We provide two structure-based denoising tasks: coordinate denoising and torsional denoising. In the coordinate denoising task, noise is sampled from normal or uniform distribution and scaled by noise factor, v R, and applied each of the atom coordinates in the structure to ensure structural features, such as backbone or sidechain torsion angles, are also corrupted. The model is then tasked with predicting either the per-node noise or the original uncorrupted coordinates. For the torsional denoising variant, the noise is applied to the backbone torsion angles and Cartesian coordinates are recomputed using pNeRF (AIQuraishi, 2019) and the uncorrupted bond lengths and angles prior to feature computation. Similarly to the coordinate denoising task, the model is then tasked with predicting either the per-residue angular noise or the original dihedral angles.
This is a multitask formulation of the previously described structure and sequence denoising tasks, with separate output heads for denoising each modality.
Sequence-Structur Co-Denoising.
Masked Attribute Prediction Tasks We use inverse folding (Section 2.3.1) 1) as a pretraining task. The benchmark additionally incorporates the distance, angle and dihedral angle masked-atribute
24

---

Published as a conference paper at ICLR 2024
prediction tasks proposed in Zhang et al. (2023b) well backbone dihedral angle prediction task.
PLDDT Prediction. Protein structure prediction models typically provide per-residue pLDDT (predicted Local Distance Difference Test) scores as local confidence measures in the quality of the prediction shown correlate well with disordered regions (Wilson et al., 2022). We formulate self-supervised node-level regression task on predicted structures, somewhat analogous to structure quality assessment (QA), where the model tasked with predicting the scaled per-residue pLDDT E [0, 1J values.
D.8 DOWNSTREAM TASKS
We curate several structure-based and sequence-based datasets from the literature and existing benchmarks The tasks are selected to evaluate not only the global structure representational power of each method, but also to evaluate the ability of each method to learn informative local representations for resid prediction and annotation tasks.
The raw structures are, where possible and accounting for obsolescence, retrieved directly from the PDB (or another structural source) as several processed datasets used by the community discard full atomic coordinates in favour of retaining only C. positions making them unsuitable for in-depth experimentation. This provides entry point for users to apply custom sequence of pee-processing steps, such as deprotonation or fixing missing regions which are common in experimental data.
The following downstream tasks are available in benchmark. Detailed documentation including composition, splitting details, metrics, and data sheets (Gebru et al., 2021) are available in Appendix
Node-level Tasks
CATH Inverse folding.
PPI Site Prediction.
Metal Binding Site Prediction.
Post-Translationa, Modification Site Prediction.
Graph-level Tasks
Fold Prediction.
Gene Ontology Prediction.
Reaction Class Prediction.
Antibody Developability Prediction.
D.9 SE(3) EQUIVARIANT NOISE PREDICTOR
Similar to Zhang et al. (2023c), for structure- paaueturr-bbedddddeeee denoising tasks we use an SE(3) equivariant noise predictor network to predict per-residue perturbations from SE(3) invariant scalar embeddings and
corrupted atomic coordinates X. Each edge Eij is featurised by concatenating the two adjoining
scalar node representations š; sj and the euclidean distance between them [ E1112- We use two-layer MLP to produce score m.y.
mij = ML, s,, MLP( - -2,(1)))
(13)
Subsequently, Equation 13 is used to aggregate normalised directional edge vectors over the neigh- bourhood Nr of each node:
T. - 2;
Eo
Σ myj
(14)
| - -0,
JEN;
'To retain focus on protein representation learning, we deliberately exclude commonly used tasks based on protein-small molecule interactions as it is hard to disentangle the effect of the small molecule representation and the potential for bias (Boyles al., 2019)
25

---

Published as a conference paper at ICLR 2024
Table Overview of supervised tasks and datasets.
Task
Dataset Origin
Structures
Train
# Validation
Test
Folding
Ingraham et al. (Ingraham 2019)
Experimental
PPT Site Prodiction
Gainza et al. (2020)
Experimental
478 K
53 K
117
Metal Binding Site Prediction
Experimental
1.1 M
13.7 K
29.8
Post-Trans, Modification
(2023)
Predicted
Crappl- Aevel
Fold Prediction
Hou et al. (Hou al., 2017)
Experimental
12.3
0.7
3. K
Gene Ontology Prediction
Gligorijevó al. (2021)
Experimental
K
3.
Reaction Class Prediction
Hermosilla (2020)
Experimental
29.2
2.6K
5.6
Antibody Develoability Prediction
Huang al. (Huang al, 2021)
Esperimental
1.7
0.24 K
0.48
D.10 HYPERPARAMETER SELECTION
Given the large number of models and featurisation schemes, we try our best to do consistent and fair hyperparameter search. We fix consistently high number layers and large hidden dimension across models, as we wanted to focus scaling model size and dataset size via pre-training. We use the Fold Classification task to select the best learning rate and dropout per model and featurisation scheme for downstream tasks. While our best performing models sometimes do not outperform the best reported results in the literature, we have obtained these results consistent experimental setup. Our goal was to demonstrate the utility of our benchmarking framework and uncover the impact of architectural considerations such as featurisation schemes, geometric GNN models, and mre-traninngaaxxlliary tasks under fair and rigorous experimental settings.
Protein Structure Encoders
1. We use consistent batch size of 32.
2. For all models, we try to consistently use six layers, each with 512 hidden channels. For tensor-based equivariant GNNs (TFN, MACE), we reduced the number of layers and hidden channels to fit 80GB of GPU memory on one NVIDIA A100 GPU.
3. For each encoder and featurisation baseline, we search over learning rates: 0.00001, 0.0001, 0.0003, 0.001 and select the best based on the validation performance on the fold classification task.
4. For each finetuning configuration, 0.0001, 0.0003, 0.00001 and perform model selection based on the validation performance.
conduct small sweep
learning rates:
5. We train all models to convergence or maximum of 24 hours on single A100 GPU.
Output Heads
1. All primary output heads use three-layer MLP with 512 as the hidden dimension.
2. For all auxiliary tasks we use a two-layer MLP with 128 as the hidden dimension.
3. For all structure denoising tasks we use two-layer SE(3) equivariant noise predictor network (Section D.9). The message and distance MLPs each consist of two layers of 128 hidden units.
4. For each encoder and featurisation, we search over decoder dropout: 0.0, 0.1, 0.3, 0.5, and select the best based the validation performance on the Fold Classification task.
E DOCUMENTATION FOR DATASETS
Below, we provide detailed documentation for each dataset included in our benchmark, summarised Table Each dataset will be made available for download processed and raw forms from Zenodo upon publication. Note that, for all datasets, we authors bear all responsibility in case of any violation of rights regarding the usage of such datasets, whether they were compiled from existing sources curated from scratch.
E. CATH INVERSE FOLDING
This is a common protein engineering task where the goal is to recover an amino acid sequence given structure up to backbone completeness. Formally, this is node-level classification task where the
26

---

Published as a conference paper at ICLR 2024
model learns a mapping for each residue r; to amino acid type y {I,..., n), where is the vocabulary size (n = 20 for the canonical set of amino acids).
Note that inverse folding is generic task that can be applied to any dataset. In the literature, it is commonly evaluated on the CATH dataset compiled by Ingraham et al. (2019). We additionally use inverse folding on AlphaFoldDB pretraining task.
Motivation Several generative methods for protein design produce backbone structures that require the design of an associated sequence. As a result, inverse folding is an important part of de novo design pipelines for proteins.
Collection For this dataset, adopt the commonly-used CATH dataset originally compiled by Ingraham et al. (2019).
Composition The dataset consists of protein structures randomly split into training, valida- tion and test sets such that proteins in different sets do not share the same CATH topology classification (i.e., CAT code).
Hosting A preprocessed version of the dataset can be downloaded from the benchmark's Zenodo data record.
Licensing We have released a preprocessed version of the dataset under Creative Commons Attribution 4.0 International license. The original dataset is available under a Creative Commons Attribution 4.0 International license at http:: cathdb. inf.
Maintenance We will announce any errata discovered in or changes made to the dataset using the benchmark's 's GitHub repository.
Uses This dataset can be used for multilabel node classification tasks where model learns mapping for each residue r; to an amino acid type E {I,...,,y) nJ, where n is the vocabulary size (e.g., n 20 for the canonical of amino acids).
Metric Perplexity.
E.2
MASIF-SITE PPI SITE PREDICTION
This task is a node-level binary classification task where the goal is to predict whether or not residue involved protein-protein interaction interface.
Motivation Identifying protein-protett interaction (PPI) sites has important applications in developing improved protein-protein interaction networks and docking tools, providing biologi- cal context to guide protein engineering and target identification in drug discovery campaigns (Jamash et al., 2021).
Collection We adopt the dataset of experimental structures curated from the PDB by Gainza et al. (2020) and retain the original splits, though we modify the labelling scheme to be based on inter-atomic proximity (3.5 Å), which can be user-defined, rather than solvent exclusion.
Composition The dataset is curated from the PDB by preprocessing such as the presence of one of the seven specified ligands (e.g., ADP or FAD), clustering based on 30% sequence identity and random subsampling. contains 1,459 structures, which are randomly assigned to training (72%), validation (8%) and test set (20%). 12 (Å) radius patches were extracted from the generated structures and a patch labelled as part of binding pocket if its centre point was < 3 (A) away from an atom of the corresponding ligand.
Hosting The original dataset is made available by the authors on Zenodo.
Licensing We have released a preprocessed version of the dataset under a Creative Commons Attribution 4.0 International license. The original dataset is available under an Apache 2.0 license at https:tpp............ github Bttp:/PDD-EPFL//AE /alssii/lloo bbllb//b////
Maintenance We will announce any errata discovered in or changes made to the dataset using the benchmark's GitHub repository.
Uses This dataset can be used for binary node classification tasks where the goal is to predict
whether or a residue is involved in protein-protein interaction interface.
Metric AUPRC.
27

---

Published as a conference paper at ICLR 2024
E.3 CCPDB METAL BINDING SITE PREDICTION
This is a binary node classification task where each residue is mapped to a label E {0, 1 indicating whether the residue (or its constituent atoms) is within 3.5 (A) of a user-defined metal ion or ligand heteroatom, respectively.
Motivation Several proteins coordinate transition metal ions to carry out their functions. As such, predicting the binding sites of metal ions elucidate the role of metal binding on protein function.
Collection The dataset is constructed from experimental structures curated from the PDB, where binding site assignments for each residue are computed on-the-fly. While the benchmark supports this task on arbitrary subsets of the PDB and ligands, we provide the Zinc-binding dataset from Dürr et al. (2023) specifically for this task.
Composition The dataset constructed by sequence-basee clustering of the PDB at 30% sequence identity to remove sequence and structural redundancy. Clusters with a member shorter than 3000 residues, containing at least one zinc atom with resolution better than 2.5 (A) determined by x-ray crystallography and not containing nucleic acids are used to compose the dataset. If multiple structures fulfill these criteria, the highest resolution structure is used. The train (2,085) / validation (26) / test (59) splits are constructed such that proteins in the validation and test sets have no partial overlap with any protein in the training data.
Hosting A preprocessed version of the dataset can be downloaded from the benchmark's 's Zenodo data record.
Licensing We have released a preprocessed version of the dataset under a Creative Commons Attribution 4.0 International license. The original dataset is freely available without a license at https: / htt////aaammccccccc oup. com/ / da database/ /ttpaa/a//eatticceeee/////////// http//attcceeeeeeeeee hwt.c//e////ii/eee 1093/ dtt//attabaasss /bay142/ 0t1111////2299331111110
Maintenance We will announce any errata discovered in changes made to the dataset using the benchmark's 's GitHub repository.
Uses This dataset can be used for binary node classification tasks where each residue is mapped to a label E (0.1 } indicating whether the residue (or its constituent atoms) is within 3.5 (Å) of a user-defined metal ion ligand heteroatom, respectively.
Metric Accuracy.
E.4
PTM POST- TRANSLATIONAL MODIFICATION SITE PREDICTION
We frame prediction of post-translational modification (PTM) sites as multilabel classification task where each residue is mapped to label E (1...., 13) distinguishing between modifications on different amino acids (e.g. phosphorylation on S/T/Y and N-linked glycosylation on N).
Motivation Identifying the exact sites where post-translational modifications (PTMs) occur is essential for understanding protein behaviour and designing targeted therapeutic interventions.
Collection We adopt dataset of 48,811 AlphaFold22-rediiced structures curated by Yan et al. (2023), where each structure contains the PTM metadata necessary to construct residue-wise site prediction labels.
Composition The dataset is split into training (43,907), validation (2,393) and test (2,511) sets based on 50% sequence identity and 80% coverage. In total, there are 240,090 PTMs present in the dataset compared to 3,391, 1,208 residues where PTMs could possible but are not present. The most common PTMs are phosphorylations serine (93,734) and N-linked glycosylation at asparagine (59,143) which together account for around 70% the PTMs.
Hosting A preprocessed version of the dataset can be downloaded from the benchmark's Zenodo data record.
Licensing We have released preprocessed version of the dataset under Creative Commons Attribution 4.0 International license. The original dataset is available under a Creative Commons Attribution 4.0 International license at ht zenodo. org/ http///rroooo//00/////
28

---

Published as a conference paper at ICLR 2024
Maintenance We will announce any errata discovered in changes made to the dataset using the benchmark's GitHub repository.
Uses This dataset can be used for multilabel node classification tasks where each residue is distinguishing between modifications different amino acids (e.g., phosphorylation on S/T/Y and N-linked glycosylation on N).
mapped label y (1,..., 13)
Metric ROC-AUC.
E.5 FOLD FOLD PREDICTION
This is multiclass graph classification task where each protein, 9, is mapped to label (1...., 1195] denoting the fold class.
Motivation The utility of fold prediction is that it serves as a litmus test for the ability of a model to distinguish different structural folds. It stands reason that models that perform poorly on distinguishing fold classes likely learn limited or low-quality structural representations.
Collection We adopt the fold classification dataset originally curated from SCOP 1.75 by (Hou et al., 2017).
Composition This dataset provides three different test sets stratified based on topological similarity: Fold, in which proteins originating from the same superfamily are absent during training: Superfamily, in which proteins originating from the same family are absent during training; and Family, in which proteins from the same family are present during training.
Hosting A preprocessed version of the dataset can be downloaded from the benchmark's s Zenodo data record.
Licensing We have released a preprocessed version of the dataset under a Creative Commons Attribution 4.0 International license. The original dataset is available under Creative Commons Attribution 4.0 International license at https: /ttp//acaammcccc/// oup com/ bioinformat ics ics/article// /4/ 8 1295/ 4708302
Maintenance We will announce any errata discovered in or changes made to the dataset using the benchmark's GitHub repository.
Uses This dataset can be used for multilabel graph classification tasks where each protein, G. is
mapped to a label y E {I...., 1195) denoting the fold class.
Metric Micro-averaged accuracy.
E.6
GO GENE ONTOLOGY PREDICTION
This a multilabel classification task, assigning functional Gene Ontology (GO) annotation to structures, GO annotations are assigned within three ontologies: biological process (BP), cellular component (CC) and molecular function (MF).
Motivation Predicting protein function in the form of functional annotations such gene ontology (GO) terms has important applications in protein analysis and engineering, providing researchers with the ability cluster funetienally-related structures to guide protein generation methods to design new proteins with desired functional properties.
Collection We adopt the dataset of experimental structures originally curated from the PDB by Gligorijević al. (2021).
Composition We retain the original multi-cuton based dataset splits proposed by (Gligorijević et al., 2021), with cutoff at 30% sequence similarity.
Hosting A preprocessed version of the dataset can be downloaded from the benchmark's 's Zenodo data record.
Licensing We have released a preprocessed version of the dataset under a Creative Commons Attribution 4.0 International license. The original dataset is available under Creative Commons Attribution 4.0 International license at https: www. ure. com/ axt iecee/cle/// / s414
17-021-23303--
29

---

Published as a conference paper at ICLR 2024
Maintenance We will announce any errata discovered in changes made to the dataset using the benchmark's 's GitHub repository.
Uses This dataset can be used for multilabel graph classification tasks, assigning functional
Gene Ontology (GO) annotation to protein structures.
Metric F max score.
E.7
EC REACTION REACTION CLASS PREDICTION
This is multiclass graph classification task where each protein, G, is mapped to label y 384) denoting which class of reactions given protein catalyzes; all four levels of the EC assignment employed to define the reaction class label.
Motivation As proteins' reaction classifications are based on their enzyme-catalyzed reaction according to all four levels of the standard Enzyme Commission (EC) number, methods that predict such classifications can help elucidate the function of newly-designed proteins as they are developed.
Collection We adopt the reaction class prediction dataset originally curated from the PDB by Hermosilla et (2020).
Composition The dataset is split the basis of sequence similarity using 50% threshold.
Hosting A preprocessed version of the dataset can be downloaded from the benchmark's 's Zenodo data record.
Licensing We have released preprocessed version of the dataset under Creative Commons Attribution 4.0 International license. The original dataset is available under an MIT license at attps://giiee ENSE.
com/ honm/phermossiee BBt///ECCovvvcee http:cooww.proeeins/iioooo.. hww.lo///matee/IIII1
Maintenance We will announce any errata discovered in changes made to the dataset using the benchmark's s GitHub repository.
Uses This dataset can be used for multilabel graph classification tasks where each protein, G, is mapped to label E {1,... 384) denoting which class of reactions given protein catalyzes. Metric Accuracy.
E.8
TDC ANTIBODY DEVELOPABILITY PREDICTION
Therapeutic antibodies must be optimised for favourable physicochemical properties in addition to target binding affinity and specificity to be viable development candidates. Consequently, we frame prediction antibody developability as binary graph classification task indicating whether a given antibody is developable.
Motivation From a benchmarking perspective, predicting the developability a given antibody is important as it enables targeted performance assessment of models specific (immunoglob- ulin) fold, providing insight into whether general-purpose structure-bast encoders can be applicable to fold-specific tasks.
Collection We adopt the antibody developability dataset originally curated from SabDab (Dunbar et al., 2014) by Chen et al. (2020).
Composition This dataset contains 2,426 antibodies that have both sequences and PDB structures available, where each example contains both a heavy chain and a light chain with resolution <
(Å). Labels are based on thresholding the developability index (DI) ((Lauer et al., 2012)) as computed by BIOVIA's platform ((Systèmes, 2016)), which relies antibody's hydrophobic and electrostatic interactions.
Hosting preprocessed version of the dataset can be downloaded from the benchmark's 's Zenodo data record.
Licensing We have released a preprocessed version of the dataset under a Creative Commons Attribution 4.0 International license. The original dataset is available under Creative Commons Attribution 3.0 Unported license at https: /dtoomoocs ai/ ingge_predd___eeeeeeeegeegee tasks /devel op/ 4 asaaabbbeooo chen-et-al.
30

---

Published as a conference paper at ICLR 2024
Maintenance We will announce any errata discovered in changes made to the dataset using the benchmark's 's GitHub repository.
Uses This dataset can be used for binary graph classification tasks indicating whether given antibody is developable.
Metric AUPRC.
31


Reviews:
[{"review_id": "u4pm8v6N7C", "reviewer": "Reviewer_W2RU", "summary": "This work presents an open benchmark for evaluating protein structure representation-learning methods. The benchmark includes a diverse set of pre-training methods, downstream tasks, and corpora and includes experimental and predicted protein structures. The structure-based pre-training and fine-tuning datasets and tasks emphasize tasks that enable structural annotation.", "questions": "Can the work be adapted for other biological macromolecules beyond protein structures?\nSee, for example, the recent reference:\nPerformance and structural coverage of the latest, in-development AlphaFold model, 2023.\nPredicting structure of proteins, nucleic acids, small molecules, ions, and modified residues. Providing a quantitative benchmark, and improving accuracy of protein-ligand structure prediction, protein-DNA and protein-RNA interface structure prediction, and protein-protein interfaces.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Modular benchmark enabling rapid evaluation of protein representation learning methods across various tasks, models, representations, and pre-training setups.\n\n2. Analysis of model performance across these different representations and architectures.\n\n3. Using auxiliary tasks to improve the performance of both invariant and equivariant models.\n\n4. Providing tools and procedures for training and evaluating models.", "weaknesses": "1. The work is missing an explanation of the limitations of the featurization schemes and pre-training tasks.\n\n2. Would be beneficial to include a discussion about the generalizability of the benchmark results \nto the overall protein structure space, and how this translates to proteins not included in the current dataset.\n\n3. Missing a discussion about how geometric models may be improved to surpass sequence-based models.\n\n4. Missing information about the ease of use of the tools, and details about the computational resources required for using the benchmark.\n\n5. Missing (i) aggregation of methods for improving model performance; and (ii) computation of uncertainties in evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zWVL77A7Qy", "reviewer": "Reviewer_5uRv", "summary": "This paper focuses on the task of protein structure representation learning and aims to provide a robust and standardized benchmark for this task. \n\nIn this paper, the authors provide different pretraining datasets, downstream datasets, pretraining tasks, auxiliary tasks, featurisation schemes, and model architectures. They cover most of the widely used training strategies, datasets, and model architectures. \n\nIn addition, the authors run experiments using the provided code base and provide some observations and insights.", "questions": "See weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well-written and easy to follow.\n\nThe provided datasets, GNN models, and training strategies are comprehensive.", "weaknesses": "1. In addition to datasets etc, I think a good benchmark should also provide experimental results with well-searched hyperparameters. In such case, future researchers can directly take results for a fair comparison. \n - However, in the current version, the authors didn’t provide results on all downstream tasks. \n - In addition, I am not sure whether the hyperparameters are well-searched, since the best results reported here are still worse than some existing methods. For example, the best results on Fold (considering both with and without auxiliary tasks) are still worse than ProNet [1] and CDConv [2] which don’t use any auxiliary tasks.\n\n2. Do ESM results in the table use pre-trained ESM weights?\n\n3. Some other pretraining strategies are used in Geom3D [3] and ESM-GearNet [4].  \n\n[1] Learning Hierarchical Protein Representations via Complete 3D Graph Networks.  \n[2] Continuous-Discrete Convolution for Geometry-Sequence Modeling in Proteins.  \n[3] Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials.  \n[4] A Systematic Study of Joint Representation Learning on Protein Sequences and Structures", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AiZm1pdoRA", "reviewer": "Reviewer_foh9", "summary": "The authors have introduced a novel framework aimed at curating datasets sourced from public repositories like PDB. Their objective is to construct benchmark datasets that facilitate the evaluation of protein structure representation. This framework takes into account various settings, including different backbone models such as various GNN-based models and diverse feature representations, to enhance our understanding of protein structure presentation. Additionally, the authors explore a wide range of pretraining tasks, including sequence/structure denoising and inverse folding, before subjecting these models to a battery of diverse downstream tasks, operating at the Alpha carbon, residue, or overall protein levels.\n\nTo support this work, the authors have generously shared an anonymous GitHub link containing scripts for data preprocessing, which enables the creation of datasets for both pretraining and benchmarking. This contribution is particularly noteworthy as it addresses a critical gap in the field of protein structure representation learning. Historically, the lack of systematically curated benchmarking datasets covering aspects such as featurization, pretraining, and downstream task evaluation has hindered progress. The absence of such a framework has made it exceedingly challenging to compare or reproduce research findings in this specialized domain.\n\nHowever, there are some concerns regarding the paper's completeness. Notably, the authors have outlined a set of downstream tasks in Figure 1, yet a significant portion of these tasks remains unreported in the results section.", "questions": "1. Could you please address the potential leakage as discussed in the weakness comments?\n\n2. Could you please consider additional featurization as pointed out in the weakness comments?\n\n3. Could you please complete the experiments and provide the results with the downstream tasks that are missing in the paper?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The unveiling of this framework, designed for assembling public datasets in order to generate pretraining and downstream benchmark datasets for the study of protein structure representation, is a noteworthy development.\n\nThe examination of how pretraining and featurization affect various downstream architectures and tasks proves to be engaging and insightful.\n\nThe timeliness and significance of this research topic cannot be understated, as it addresses the pressing need for a standardized framework that enables the comparison of various state-of-the-art methods using the same benchmark datasets.", "weaknesses": "Addressing the Issue of Potential Leakage:\n\nEfforts to mitigate potential data leakage are crucial to ensuring the integrity of benchmarking results, as such leakage could introduce misleading elements into the research findings. Have you considered the removal of overlapping sequences between the pretraining datasets and the downstream testing datasets to further safeguard against such issues?\n\nExpanding Featurization Methods:\n\nIn terms of featurization, the paper seems to primarily focus on simple feature extraction methods. It might be valuable if the authors explored the integration of the following additional featurization techniques:\n\n    Incorporating 3D presentations, such as Uni-Mol (available at https://github.com/dptech-corp/Uni-Mol).\n    Utilizing pretrained models trained on 2D data, like ESM-1b and ESM-2. Considering that Table 1 showcases results using ESM as a backbone model, which produced superior outcomes on the fold dataset, it could be beneficial to include the results for other ESM models in the table.\n\nDiverse Downstream Tasks:\n\nTable 1 predominantly presents results for two specific tasks. However, it appears that several other downstream tasks have not been included in this experiment. It is worth noting that many of these unreported downstream tasks are of substantial importance and are conspicuously absent from the paper. Is there a particular reason for not considering these tasks, and could they potentially be included to provide a more comprehensive view of the research findings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qa9TsGua2r", "reviewer": "Reviewer_keJ2", "summary": "The authors introduce a large suite of benchmarks for evaluating learned embeddings of proteins. Included are node-level evaluations (at the level of individual residues, e.g. inverse folding, metal binding site prediction) and graph-level evaluations (at the level of the entire protein, e.g. fold classification). The authors also provide a number of software tools, including dataloaders for various pretraining tasks. Finally, they evaluate selected architectures on various pretraining task/benchmark combinations.", "questions": "In Table 3, inverse folding (a downstream node-level task) is listed alongside four other tasks explicitly identified as pretraining tasks above. Is this intentional? \n\nAre the experiments in Table 2 at all realistic? I don't think these tasks ever be attempted using models with no pretraining at all in practice. Also, the ESM model without any pretraining can hardly be called an ESM model at all.\n\nDid you consider adding any restrictions on the interfaces between model embeddings and the downstream tasks? If the goal is to evaluate the embeddings themselves, it seems to me like there should be an attempt to standardize e.g. the size of the task-specific MLPs used.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "New benchmarks are always welcome, and there is also great value in consolidating existing benchmarks. This paper does that well, as the selection here is quite broad, with good coverage. I appreciate the inclusion of both node-level and graph-level evaluations. Within each category, each choice of benchmark is accompanied by a specific rationale, which is also a strength. The documentation of the codebase also seems clear and easy to follow.", "weaknesses": "It would have been nice to see baselines for all of the downstream tasks in the benchmark using the tools in this software suite (or, lacking that, even scores copied from their respective papers if needs be). Certain tasks like \"reaction class prediction\" are currently missing. Part of the value-add here is the ease of running experiments on all of these tasks, and that isn't currently demonstrated in the current version of the manuscript. Also, consolidated baseline scores are useful sanity checks for reproduction experiments down the line.\n\nMiscellaneous stuff:\n\n- Please provide details about the confidence intervals in Table 2.\n> whereas pLDDT prediction and structure denoising benefit invariant models the most\n- I don't really understand what the basis of this claim (^) is. All four models do approximately as well in the pLDDT column, e.g.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)