Split and Merge Proxy: pre-training protein-protein contact prediction by mining rich information from monomer data
Authors: Hao Du
Yuchen Ren
Yan Lu
He Huang
Yating Liu
Zhendong Mao
Xinqi Gong
Wanli Ouyang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Protein Bioinformatics
Protein-Protein Contact Prediction
Pre-training

Abstract: Protein-protein contact prediction is a key intelligent biology computation technology for complex multimer protein function analysis but still sufferers from low accuracy. An important problem is that the number of training data cannot meet the requirements of deep-learning-based methods due to the expensive cost of capturing structure information of multimer data. In this paper, we solve this data volume bottleneck in a cheap way, borrowing rich information from monomer data. To utilize monomer (single chain) data in this multimer (multiple chains) problem, we propose a simple but effective pre-training method called Split and Merger Proxy (SMP), which utilizes monomer data to construct a proxy task for model pre-training. This proxy task cuts monomer data into two sub-parts, called pseudo multimer, and pre-trains the model to merge them back together by predicting their pseudo contacts. The pre-trained model is then used to initialize for our target – protein-protein contact prediction. Because of the consistency between this proxy task and the final target, the whole method brings a stronger pre-trained model for subsequent fine-tuning, leading to significant performance gains. Extensive experiments validate the effectiveness of our method and show the model performs better than the state of the art by 11.40% and 2.97% on the P@ L/10 metric for bounded benchmarks DIPS-Plus and CASP-CAPRI, respectively. Further, the model also achieves almost 1.5 times performance superiority to the state of the art on the harder unbounded benchmark DB5. The code, model, and pre-training data will be released after this paper is accepted. 

Under review as a conference paper at ICLR 2023
SPLIT AND MERGE PROXY: PRE-TRAINING PROTEIN- PROTEIN CONTACT PREDICTION BY MINING RICH IN- FORMATION FROM MONOMER DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Protein-protein contact prediction is a key intelligent biology computation tech- nology for complex multimer protein function analysis but still sufferers from low accuracy. An important problem is that the number of training data cannot meet the requirements of deep-learning-base capturing structure information of multimer data. In this paper, we solve this data volume bottleneck in a cheap way. borrowing rich information from monomer data. To utilize monomer (single chain) data in this multimer (multiple chains) problem, we propose a simple but effective pre-training method called Split and Merger Proxy (SMP), which utilizes monomer data to construct proxy task for model pre-training. This proxy task cuts monomer data into two sub-parts, called pseudo multimer, and pre-trains the model to merge them back together by pre- dicting their pseudo contacts. The pre-trained model is then used to initialize for our target protein-protein protein contact prediction. Because of the consistency be- tween this proxy task and the final target, the whole method brings stronger pre-trained model for subsequent fine-tuning, leading to significant performance gains. Extensive experiments validate the effectiveness of our method and show the model performs better than the state of the art by 11.40% and 2.97% on the P@ L/10 metric for bounded benchmarks DIPS-Plus and CASP-CAPRL respec- tively. Further, the model also achieves almost 1.5 times performance superiority to the state of the the harder unbounded benchmark DB5. The code, model, and pre- pre-training data will be released after this paper is accepted.
methods due to the expensive cost
INTRODUCTION
Proteins are large molecules consisting of amino acids (also called residues) sequences. Protein- protein contact prediction aims to compute the constraints between given protein sequences (specif- ically whether residue (can be understood as individual amino acid) one protein are in contact with residue on the other protein), which is important for the structural or functional analysis of protein complexes. 'he predicted constraints reveal the relationships between each residue pair of the two protein sequences, which can not only benefit complex protein structure prediction but also be useful for many kinds of protein function analysis scenarios, e. g. developing new drugs and designing new proteins. The success of RaptorX Wang et al. (2017); Xu et al. (2021) and Duuppr AlphaFold2 Jumper et al. (202T) demonstrates the application potential of deep learning in the com- putational biology field and inspired series of new biological computation methods. However, when extending the deep model protein-protein (inter-chain) contact prediction, recent works have not achieved satisfying performance as the aforementioned successful works do. An important bottleneck is data quantity limitation.
Many well-known successful deep learning systems are almost trained under large-scale datasets.
For example, computer vision (CV), ConvNet Krizhevsky et al. (2012); Simonyya & Zisserman (2014); He et al. (2016)) and ViT Dosovitskiy et al. (2020); Liu al. (2021); Yuan et al. (2021) are trained on ImageNet Deng et al. (2009) which has 14 million labeled data who provides enough vision category information of real word. For natural language processing (NLP), the most popular language model BERT Devlin et al. (2018) is trained on document-level data BooksCorpus Zhu
et al. (2015) and English Wikipedia in unsupervised manner. And in computational biology, the

---

Under review as a conference paper at ICLR 2023
Split
Merge
monomer
pseudo multimer
Figure I: The main idea of Split and Merger Proxy (best viewed in color). In the pre-training stage, monomer (single chain) is firstly split into two sub-parts that are treated as pseudo multimers (a pair of chains). And then the deep model is pre-trained by learning to merge the pseudo multimers back by predicting their protein-protein contacts.
recent most popular protein structure prediction model AlphaFold2 [uuperrrerrrr al. (2021) is trained
on about 400k monomer data, 60k with 3D structure labels of Protein Data Bank (PDB) wwp (019) and 350k protein sequence, and achieves electron mieroscope accuracy. Obviously, existing human- level accurate and successful artificial intelligence models also need big data to train. However, the number of the current largest open-sourced multimer training data Morehead et al. (2022) is much lower than the aforementioned topics, which is only 15k and limits the performance of the deep model. The main reason is the expensive cost of capturing the multimer protein structural information by high- high-accurate devices. So tackle the problem of the scarcity of training data, focus on finding a cheap way to obtain additional data and avoid the extra cost.
Our main idea is to expand the training data by introducing the monomer data into the training step for protein-protein contact prediction. The existing monomer data is free and also can provide useful biological prior. ComplexConie
Zeng et al. (2018) the first work introducing the monomer data into the multimer contact prediction task. It proves the potential value of the monomer data to the multimer task. But obviously, there an unneglectable task gap between the monomer and the multimer. Specifically, the monomer can only provide information about one chain while the multimer task requires more. So ComplexContact Zeng al. (2018) suffers from that task gap and existing contact prediction methods often neglect these data. In this paper, we design a novel and effective pre- pre-training method called Split and Merger Proxy (SMP) to introduce monomer data into the protein- protein contact prediction task more effectively, which reduces the aforementioned task gap and leads to better results.
The proposed SMP is proxy task for contact prediction pre-training. As shown in Figure SMP generates pseudo multimer data from monomers and utilizes that data to pre-train the contact pre- diction model. In particular, single protein is split into two sub-pau habb-arts that treated pseudo multimer. That pseudo data are used to train the contact prediction model, equal to guide the model merge these split data back. Although the pseudo multimer data contain biological noise, they can provide additional richer information that complements the existing multimer data. The training targets of SMP and the final task are both contact prediction, so there no task gap the fine-tuning stage. The pre-trained model can be fine-tuned on the real multimer data without any modification, leading to a better final model and more accurate contact results.
Our main contributions are as follows:
We design novel proxy task, Split and Merger Proxy (SMP). to pre-train contact predic- tion models on the monomer data more effectively. From the best of our knowledge, this is the first work leverage the monomer protein data pre-train the multimer protein contact prediction task.
Experiments show that achieve new state-of-the-art and improve the P@ L/10 met- ric by a large margin 11.40% and 2.97% respectively on DIPS-Plus and CASP-CAPRI benchmarks when compared with the latest state-of-the--rt DeepInteract Morehead et al.
2

---

Under review as a conference paper at ICLR 2023
Moreover, almost achieve 1.5 times more performance than GeoTrans the 2022
harder unbounded benchmark DB5.
2 RELATED WORKS
2.1
PROTEIN-PROTEIN CONTACT PREDICTION
Intra-protein contact prediction has been well treated Jumper et al. (2221; Back et al. (2021). but protein-protein contact prediction has not been extensively studied. Some early works Weigt et al. (2009); Morcos Morcoss et al. 1201T): Ekeberg al. (2014) used direct-coupling analysis (DCA) to disentan- gle direct and indirect correlations to infer potential relationships between amino acids at different positions. With the great success of Convolutional Neural Network (CNN) LeCun et al. (1998) in
CV area, Zeng et al. (2018); Yan & Huang (2021); Roy et al. (2022) applied CNN to multimer con- tact prediction. Zeng al.] (2018) used two CNNs, one with ID convolution processed sequence information and the other with 2D convolution encoded MSA information. Yan & Huang (2021) utilized more biological features (e.g. inter-protein docking pattern, physico-chemical information and sequence conservation) as inputs to the neural network to enrich the information carried by multimer data. Because He et al. (2016) demonstrated that deeper networks could learn more dis- criminative features from the dataset, |ooy et al. (2022) used a deeper dilated residual network Yu al. (2017) to capture relationships between residues. Due to each protein has 3D structure, Fout et al. (2017); Liu al. (2020); Morchead et al. (2022); Xie & Xu (2022) designed graph neural Scarselli et al. (2008) to predict contacts between proteins. They first built graph for each protein, the residue each protein is regarded
network [GNN)|Scarsclll]
node, and whether the residues in the protein are connected is regarded as an edge. FFout et al. (2017) used graph convolution Kipf & Welling (2016) to get the graph representation of the underlying protein structure and fully convo- lutional network (FCN) was utilized to determine contacts between two proteins. Liu et al. (2020) employed weights sharing GNNs to obtain the residue features of each protein, then they devised multilayer CNNs as the interaction module to perform contact prediction. Based on this, Morehead al. (2022) designed graph transformers to encode the geometric information in multimers, e.g., the distance and direction between residues and the amide angle. Xie & Xu (2022) believed that simply building the residue graph was not enough, so they built two more graphs, e.g., atom graph and surface graph, then they did message passing in each graph. Since AlphaFold2 Jumper et al. (202I) has achieved surprising results in monomer structure prediction, [vvans et al. (2021) ); Bryant cet al. (2022); Gao et al. (2022) extended it to multimer contact prediction. [vvans et al. (2021) took into account permutation symmetry. position encoding of different chains in multimer, and multi- mer multiple sequence aligmment (MSA) construction for contact prediction. |Bryant et al. (2022);
Gao et al. (2022) directly spliced multimer as monomer and fed it into AlphaFold2 to get contact prediction. But due to the small level of existing multimer data, current models are less accurate in moteinn-prteeio contact prediction.
2.2 PRE-TRAINING IN PROTEIN MODELING
Pre-training from a lot of data can provide good prior knowledge for the model, so it achieves great success data science community, such as computer vision and natural language processing
areas. Some recent works introduced pre- pre-training paradigm to protein modeling area. Rao et al.
20211; Rives et al. 2021 Elnaggar et al. (202 Chowdhury et al. (202I); [ Fang et al. 2022)): ; Lin al. (2022) used Masked Language Model (MLM) proxy task Devlin et al. (2018)
learn
residue embedding from massive protein sequences. Rives et al. (2021) Elnaggar et al. 202
Chowdhury al. (202I) directly utilized transformer Vaswani et al. (2017 as pre-training network to capture potential biological patterns of amino acids. Since MSAs can provide a certain biological
prior for the model, Fang et al. (2022); Lin et al. (2022) devised the same Evoformer network as AlphaFold2 JphaPelldJJupper et al. (2021) and Rao et al. (2021) designed MSA transformer to fully integrate the MSA information into the transformer architecture pre-training stage, which can make the
network directly learn evolutionary information. Because each atom of the protein in PDB Database
wwp (2019) has 3D coordinates, Gligorijević et al. (202I): Chen et al. (2022) designed distance prediction and the dihedral angle prediction proxy tasks, then they got the underlying structural representations for monomers and achieved excellent performance in protein classification tasks. Due to the lack of multimer data and the cost of collecting multimer data is expensive, it is difficult
3

---

Under review as a conference paper at ICLR 2023
Split stage
Merge stage
pseudo ground truth
64
Graph Feature Extraction
Interleave
oas
poeuda
Fine-tuning
Graph Feature Extraction
Interleave
0
0
12
ground truth
multimer
The Framework of the proposed Split and Merge Proxy (SMP) pre-training method (best viewed in color). The split stage cuts monomer (the sequence "APFLDERL" in the figure) into a pseudo multimer consisting of two sub-parts (the two sub-parts "APFLD" and "LRL" in the figure) and computes the contact ground truth. The merge step is the pre-training process, which trains the model to predict the contact relationships on the split data, essentially equal to merging the split sequences back. Note that the sequence here is just an example, not the real chain. And the fine- tuning stage, the whole contact predictor, including the graph feature extractor and the interleave module, is directly fine-tuned without any modification on the real multimer data.
Figure 2:
to build effective pre- vre-training on existing multimer data. So in this paper, we design novel proxy task to adapt the monomer into multimer contact prediction, which can pre-train the model getting stronger performance.
METHOD
3.1
TASK DEFINITION
The protein-protein contact prediction, also called residue-residue interface interaction prediction, aims to compute the contact relationship map E (0, 1 between the two given protein chains. The element A,, is 0 or 1. indicating whether the i-th residue in one sequence interacts with the j-th residue in another sequence or not. The contact prediction models take multiple kinds of biological features as inputs, such amino acid sequences E TpL (P is the set of the amino acid, including 20 kinds of amino acids) and residue 3D coordinates e e L which is essentially the location of the non-hydrogen atoms. The computation pipeline can be defined as:
Apred = f(I1, where = (s; cJ, i (1,2)
(1)
To achieve this, whatever details of the function f. proteins often regarded as graph so the residues are treated as graph nodes and processed by graphic models to extract features. After that, an interleave module fuses these node features and measures the similarities between each residue pair to calculate the contact scores. The whole process is the same as the Fine-tuning block in Figure
3.2 SPLIT AND MERGER PROXY
The Split and Merger Proxy (SMP) is effective proxy task proposed to pre-train the contact prediction model. The main pipeline includes a split step and a merge step, shown in Figure 2 Each monomer sequence is cut into two sub-parts to generate the pseudo multimer data. In the merge step, the model learns the contact prediction task directly on the aforementioned split data without any modification. After that, the model would be fine-tuned real multimer data.
4

---

Under review as a conference paper at ICLR 2023
Split stage: We use the monomers from the Protein Data Bank (PDB) wwp (2019) dataset because its monomer data including both amino acid sequences s E Ph and corresponding 3D coordinates E R The target of the split stage is to generate the pseudo multimer that has the same data structure as the real one, including two sub-sequences (81 and 82) with structural information (C1 and (2) and their corresponding contact ground truth A. We first cut the amino acid sequence into two sub-sequences at random location:
= s|= $2 = s|]
(2)
where / means the random split index uniformly sampled from the range R. keeping each cut se- quence informative and avoiding too short split results that contain only small amounts of residues. In other words, this split location is around the center of the given sequence.
And for the 3D structure, we do a similar split operation:
61 = cl: |/,c2 C2 c[1 :].
(3)
We do not operate any kind of normalization on these 3D coordinates, keeping their values still in the monomer coordinate system so ground-truth A could be computed by the following formula directly:
A., I D Dij >> Da, -
where A the threshold to determine whether the i-th and j-th residue pair contact or not. ||-|| means the Euclidean distance. This process could be interpreted that the ground-trus truth contact of pscudo multimer is equal to the intra-contact of the original monomer. Based on the steps mentioned above, monomer data is converted to the pseudo multimer in the form of (81, 82, C1, C2 A).
Merge stage:
The merge stage is essentially the mimicking learning of the standard contact pre- diction training. The model learns to predict the A based on the given pseudo multimer inputs {81, 82, C1, ).
We first extract and calculate co-evolution, conservation and geometric information for each cut sub- sequence by Multiple Sequence Alignments (MSA) and Protein Structure and Interaction Analyzer (PSAIA), respectively. And then, these pieces of information combined with the protein sequence and 3D structural information are sent to a weights-sharing graph feature extractor to extract residue features F1 E R 4, xC and F- E RLaxC like Figure shows. Note that the coordinate values in c1, C2 all belong in the same monomer coordinate system. So they are all treated to the relative distances of residue pairs each protein sequence to avoid information leakage. After that, an interleave module computes the interaction features F, E R24xL2XC* which stores the high-level relationship patterns for each residue pair. Finally, contact prediction head, often a fully convolutional neural network (FCN), predicts a contact map based on that features. For the prediction, we train it as a binary classification task by utilizing the cross-entropy loss function.
Fine-tuning stage: The SMP task is the same as the final contact prediction task, both predicting the protein-protein protein contact maps. So there is not any task gap between this proxy task and fine-tuning Every module and parameter of the pre-trained model could be re-used in the final model. So, We feed the real multimer data into the pre-trained model and fine-tune the whole model directly.
EXPERIMENTS
4.1 DATASET AND EVALUATION PROTOCOL
In this section, we conduct several experiments on three popular benchmarks DIPS-Plus Morehead al. (202T), CAPPCCAPRLLeesiii al. (2019 202T) and DB5 Dreven al. (2015) datasets.
DIPS-Plus is latest open-sourced dataset for protein-protein contact prediction. It provides amino acid sequences and residue coordinates for each multimer data. Except for these pieces of basic information, DIPS-Plus also offers additional different types of biological features such as protrusion index and amide plane normal vector, composing much richer information. After filtering extreme data, such too long, too short sequences and high relative data with other datasets, the DIPS-Plus dataset still has 15,618 and 3,548 protein complexes for training and validation, respectively, which is the recent known largest open-sourced benchmark. For testing, it provides 32 protein complexes consisting of 16 homodimers and 16 heterodimers to evaluate the model's 's ability to handle samples of different difficulties.
5

---

Under review as a conference paper at ICLR 2023
Table 1: The average top-k precision (P@k) and recall (R@k) HIPS-Plus test dataset (%)
(Homo)
(Heter))
Method
Pe L/10
P@ L/5
Pe L/2
P@ L/10
p L/5
pe L/2
BIFSPI Sanchez Gey-Garcia Gareia et
2018)
2,00
DeepHomo Yan Huane
2021
12.00
9.00
ComplexConiee Conlexe Zeng el 2018
16.00
15.00
GCN Morcheao
2022
20.00
18.00
8.00
7.00
Geo'Itans Morchead gral
2022
25.00
23.00
14.00
11.00
Trans SMP
39.81
33,33
26.02
20.99
20.07
15.00
32 (All Proteins)
Method
L/10
Ple L75
Pe L/2
R
RØ L/2
R L/5
RIPSPI /IPSP| Sancher Garcia
2018
1.00
1.00
1.00
0.40
0.30
GCN Morcheed stali 2
16.00
12.00
10.00
6.00
3.00
GeoTrans
9.00
Ged SoTTram Frans + SMP
30.40
26.70
20.51
24.00
16.02
8.56
Table 2: The average top-k precision and recall CASP-CAPRI 13 & 14 dataset.
14 (Toom)
(lltero))
Method
L/10
P@ L/5
Pe L/2
P@ L/10
Pe L/5
Pe L/2
BIPSPI Sanchez-Garee
2018)
5.00
DecpHome Yon & Humne 2021
2.00
2,00
ComplexContur
8:00
5.00
GCN Morchaha e al. 2022
11.00
13.00
11.00
9,00
Geolirms Morcheed
Goo Trans SMP
18.63
14.37
11.57
32,00
23.49
18.35
19 (All Proteins)
Method
L/10
Pe L/5
Pe L/2
R46
R L/2
R L/5
RIPSPI RIPSPI Sanchez Garcia et al
2018)
0
1,00
2,00
1.00
0.1
GCN Morchend ef al.
10.00
6.00
GeoTrans (a (Morenemn arai 2022
19.00
14.00
13,00
8.00
4,00
Geu Trans SKP
21.97
16.77
13.36
14.33
8.34
3.91
CASP-CAPRI has been well known biologically joint challenge since 2014, aiming assess
the computational methods of modeling protein structures. Morehead et al.) (2022) re-organized the data of the 13th and 14th CASP-CAPRY challenge sessions Lensink al. (2019 202T), filtering the overlap between the original CASP-CAPRI data and the DIPS-Plus. These filtered data include 14 homodimers and 5 heterodimers and are used to evaluate the ability of real-world applications and cross-set generalization of models trained on the DIPS-Plus training set.
DB5 (Docking Benchmarks version 5 Vreven al. (2015)) traditional benchmark dataset for protein-protee contact prediction, including 140 training, 35 evaluation and 55 testing samples. DB5 consists of unbounded protein complexes that have varying contact types. In contrast, com- plexes in DIPS-Plus and CASP-CAPR are bounded and their multiple chains are already conformed with each other. So can indicate the performance and effectiveness of our model on different types of complexes.
Evaluation All the experiments follow the standard evaluation protocol in existing multimer contact prediction benchmarks. To assess the accuracy of the prediction, the top-k precision and recall are adopted as the evaluation metrics, where E {L/30, L/20, length of the shortest chain.
L/10, L/2, L} with L being the
4.2 IMPLEMENTATION DETAILS
We generate the pscudd multimer data from all monomers before 2018-4-30 from Protein Data Bank (PDB) wwp 2019). There are 60, 206 pdb files in total. Each file contains sequence and structural information for the protein. Monomers that cannot be parsed by Biopython Cock et al. (2009) (containing unknown atoms; missing atoms, chain numbers are not in order and so on) are filtered out. Except that each protein file contains several conformations, we only keep the first one and abandon the other. We set the split range R [1/3 2/3} so that the cut position is close to the middle of the given sequence to get pseudo multimers. Too short split proteins whose length of any chain is less than 20 are dropped. The threshold À used to calculate the contact ground truth is set as Á following the same procedure that real multimer utilizes Morchead et al. (2021). Finally, there 22, 589 pseudo multimers, about 1.5 times of the existing real multimer dataset. Whatever for the pseudo or real multimer data, we also use HHBlits Remmert et al. (2012) with Uniclust30 Mirdita et al. (017) database for MSA, and PSAIA Mihel et al. (2008) to calculate geometric features.
Our SMP is a pre- pre-trainin method that is not tightly bound to a specific model. So we combine SMP with the Geometric Transformer (GeoTrans) Morehead al. (2022) ) to evaluate the effectiveness of

---

Under review as a conference paper at ICLR 2023
Table 3: The average top-k precision and recall DB5 test dataset.
(Hetero)
Methed
Pe L/10
P@ L/5
P@ L/2
R
R L/2
R@ L/5
BIPSP] SSanchez- Garcia et (211
0.10
0.04
Complex Contac Zong et al 2018
0.30
0.30
0.70
0.30
0.10
GCN Moreheau el al 2022)
0.60
0.70
1.30
0.80
0.30
Geo Mastiaiisspoooeeeeee Moreica at 2022
0.90
1.10
1.80
1.00
0.34
Goo rans + SMP
1.78
1.88
1.55
2.53
1,45
0.69
Table 4: SMP vs self-supervised pre-training (SSL) on DIPS-Plin test dataset.
Row
Model
PreTrann
P@ 1,/10
P@ I,/5
P@ T./2
P@
R
R T./2
R@ 1,/5
R 1,/10
GCN
16.00
12.00
10.00
6.00
3,00
GCN
SMP
13.58
10.04
Geo'Trans
19.00
17.00
15.00
9.00
4,00
Geo'Irans
SSL
17.46
Geolirans
SMP
30.40
26.70
20.51
15.87
24.00
16.02
8,56
4.79
SMP in the following experiments. The batch size of pre-training and fine-tuning are all set as 48 (except the fine-tuning one of CASP-CAPRI set as 32 because of the cross-domain evaluation setting of CASP-CAPRI, Other experimental settings, including loss function, optimizer, learning rate and so on, are all kept the same the latest open-soureed state-of-the-art GeoTrans.
4.3 COMPARISON WITH STATE-OF-THE- METHODS
We compare several state-of-the-art multimer contact prediction methods including BIP- SPI Saanchez-Garria et al. (2018), ComplexContact /Zeng et al.] (2018), DeepHomo Yan Huang (2021). GCN Morehead et al. (2022) and Gea Morehead et (2022). Except that the input of ComplexContace is the amino acid sequence, the other methods take both amino acid sequence and 3D structural information inputs, which are the same as our model.
Table 1 shows the comparison results between SMP and other methods on the DIPS- Plus dataset, demonstrating that SMP outperforms existing state-of-the-ar method GeoTrans Morehead et al.) 2022) by large margin. For homologous complexes, SMP outperforms Geo Trans by 7.43% on the harder metric P@ 1.22 and even 18.36% on P@ L,20,, demonstrating that SMP can learn more use- ful residue representation and contact prediction knowledge from additional pseudo multimer data. For more difficult heterologous complexes, SMP also surpasses Geo 4.49% on harder P@ L/2. These heterologous performances benefit from the potential consistency with the pseudo multimer and heterologous proteins. Specifically, the cut chains usually have low sequence identities, sharing certain similar properties and distributions of the real heterologous data, making SMP obvious improvement on heterologous multimers. From an overall perspective, the proposed SMP brings significant gains compared with GeoTrans by 11.4% at P@ L/10 and 8.00% at R @ L. proving that our SMP brings more discriminative expression for multimer contact prediction whatever homolo- gous or heterologous complexes.
Tables 2 presents the average top-k metrics of SMP on the CASP-CAPRI dataset, specifically, 19 challenging protein complexes (14 homodimers and 5 heterodimers). SMP also surpasses the state- of-the-an art method BrenTrans on P@ L/ 10 by 5.63% on 14 homologous when keeping comparable performances for 5 heterologous. SMP achieves improvements for several different settings, demon- strating that the pre-training of SMP learns many valuable patterns of contact prediction from pseudo multimers to help learn real multimer prediction effectively.
On the DB5 dataset in Table 3 SMP also exceeds the precision of GeoTTaan for all metrics. All methods perform poorly due to testing hard and unseen unbound complexes with varying contact types that are not necessarily conformal. However, SMP still shows more than 1.5 times better performance than GeoTrans almost all metrics. indicates that SMP has good cross cross-domain capabilities and has the potential to be used in real-worke applications of complex contact prediction.
Overall, this pre-training paradigm plays a considerable role in various types of downstream protein- protein contact prediction tasks (cross set and unbound set), showing good robustness with SMP.
4.4
ABLATION STUDIES
4.4.1
COMPARISON WITH DIFFERENT PRE-TRAINING PARADIGM AND CONTACT PREDICTOR
Previous comparisons show the effectiveness the combination of our SMP with the latest state- of-the-art model GeoTrans Morehead al. (2022). In this ablation study, further investigate the

---

Under review as a conference paper at ICLR 2023
Table 5: Partial pre-training results DIPS-Plus test dataset.
Row
Ratio
Pie L/10
Pel L75
PΦ L/2
Pg
R@ L
RØ L/2
R
L,6
R L7T0
19.00
17.00
15.00
9.00
4,00
1/5
18.22
15.58
13.35
11.06
16.80
10.48
4.84
2.76
3
1/4
18.61
19.02
15.10
11.76
17.08
11.14
5.68
2.90
1/3
24.64
21.36
16.59
11.92
16.93
12.08
6.14
3.35
5
1/2
26.20
21.29
15.85
12.77
18.09
11.50
6.40
3.84
Table Partial tinettuiig results on DIPS-Plus test dataset.
Row
Kahi
Pie 1,/10
Pos 1,75
Pl 1./2
P6l
Roi 1,/2
K L/55
Roi L7To
0
5.98
3.70
2,08
1.98
1.78
0.82
0.62
0.56
1/5
12.25
3
1/4
19.83
16.12
12.53
10.80
15.87
9.71
4.91
3.10
1/3
12.96
1/2
23.99
19.55
15.49
11.92
16.97
11.11
5.53
3.22
superiority of our SMP. We combine SMP with different contact predictor to prove its generalization and also compare the SMP with other pre-training method show the advantage the SMP design. All related results are shown in Table 4
To investigate the influence of the combined contact predictor with SMP, we change the graph feature transfer module from Transformer into the Graph Convolutional Network (GCN) Kipf & Welling (2016). This GCN only has a total of 33k parameters, which is quite much lower than the I.4m parameters of the Transformer one. So this setting can show the generalization of the SMP small-scale model. From the 1st and 2nd lines of Table it can he seen that our SMP still brings 3.64% performance increase under the p@ L/5. It indicates that the SMP paradigm keeps strong generalization on the small-scale model, showing the potential for extensions of future different types and levels of contact predictors.
To show the superiority of the SMP design, we construct another pre-training method by adapting popular self-supervised learning (SSL) paradigm Proteinbert Brandes al. (2022) train the monomer data with 3D structural cues. This method pre-trains the model by self-supervised proxy task that guides the model to reconstruct the inputs by partial observation, providing different pre- training mechanisms compared with our SMP. As shown in the 3rd ~ 4th lines of Table 4 this SSL. method provides average 1 % gains on all metries, proving the fact that monomers bring much useful information for this multimer task from different view. But when compared with the SMP (5th line), SMP still shows stronger performance and outperforms the SSL by 5.89% on the harder metric P@ L/2, indicating the superiority of the SMP design that utilize information in 3D structures more effectively and further eliminate the task gap between the pre- pre-training and fine-tuning.
4.4.2 PARTIAL TRAINING RESULTS
We study the effectiveness of pre-training data volume for SMP and conduct partial pre-training experiments with different degrees of monomer data. We set five partial pre-training ratios (1/5, 1/3, 1/2, 1). When comparing the 2nd line of Table with the 1st line (without pre- training), we find that the performance has some fluctuation when the number of introduced pseudo multimers is small. This is caused by the biological noise introduced by the small-scale pseudo data, which is eliminated when the scale increases and clearly indicated in Table 5 3rd~6thh lines. Obviously, when the amount of pre-trained data reaches 1/4 (in the 3rd line), SMP has introduced certain precision and recall gains except on P@ L/ 10 metric than GeoTran (in the 1st line), with an average improvement of 2%. Moreover, as the amount of pre-trained data increases, the perfor- mance gradually improves, proving that SMP guides the model to learn rich contact prediction to provide beneficial initialization parameters for contact prediction models.
4.4.3 PARTIAL FINE-TUNING RESULTS
The pre-trained model has the potential to achieve satisfying performances only trained with small- scale training data. So we aim to explore the effect of SMP for fine-tuning with different scale data. We use six partial fine-tuning ratios, which belong to the set {0, 1/5, 1/4, 1/3, 1/3, 1st line of Table 6 shows that SMP surpasses the traditional method BIPSPI (Table B without any fine-tuning, which indicates that pseudo multimer can provide prior knowledge that relevant to the real multimers contact prediction. Moreover, the 3rd line of Table δ shows that model achieves
The
8

---

Under review as a conference paper at ICLR 2023
Geo
SMP
Ground Truth
Figure 3: Contact visualization results of the 2 multimer 4LIW (first row) and 4DR5 (second row). The GeoTrans's predictions, SMP's predictions and ground truths are corresponding to the left, middle and right columns, respectively.
Table Different split ranges results on DIPS-Plus test dataset.
Range
L,/1
L,5
T./2
R
1./2
L,75
L/10
2/5
3/5
21.94
19.47
14.37
11,66
17.63
11.43
6.39
3.65
1/3
2/3
30.40
26.70
20.51
15.87
24.00
16.02
8,56
4.79
1/4 3/4
23.11
20.00
17.22
14.35
20.80
12.80
5.85
3.35
1/5 4/5
23.90
20.75
14.17
11.36
16.84
10.71
6.16
3,40
comparable results to our combined predictor Geo Morchead et al. (2022) only with 1/4 train-
ing data demonstrating that the SMP pre-training can provide knowledge that can be effectively re-used and transferred to the real multimer scenario. With further increasing the data volume the 4th ~ 6th line of Table 6 it can finally achieve 30.4% metric P@ L/10, surpassing the previ- ous state-of-the-art method GeoTrans. These experiments prove that our pre-training paradigm can effectively reduce the dependence real data and make the model adapt to different volume-level training data situations, having the potential to save the extra cost of collecting multimer data.
4.4.4 DIFFERENT SPLIT RANGE RESULTS
We study the influence of different split ranges on the split stage for SMP to find the optimal one. We set four split intervals settings (2/5 ~ 3/5, 1/3 ~ 2/3, 1/4 3/4, 1/5 4/5). As shown in the 2nd line of Table 7 we find the performance is best when the split interval is 1/3 ~ 2/3. This appropriate range makes the random interval relatively close to the middle of the protein chain and avoids yielding one of the monomers with a too short length simultaneously.
4.5 VISUALIZATION
We also visualize some prediction results of GeoTrans in Figure exhibit a homologous mul- timer (i.e., PDB ID: 4LIW) and heterologous multimer (i.e., 4DR5) from the DIPS-Plus test set. The blue box in FigureBjidiccaee indicates that SMP successfully can predict several positive contacts that GeoTrans neglects. And the green box in Figure 3 shows that SMP can eliminate some false positives provided by GeoTTran. All these bounded areas demonstrate that SMP is more accurate method GeoTrans, demonstrating that the
multimer contact prediction than the state-of-the model pre-trained by SMP can carry several types of new advantages over the original one.
5 CONCLUSION
This paper introduces the Split and Merger Proxy (SMP), simple yet effective pre-training frame- work for protein-protein contact prediction to solve the limited number of multimers by using rich monomer information. SMP splits monomer data into pscudo multimers and trains the model to merge them back together by predicting its pseudo contact interaction, which reduces the task gap between this proxy task and the final target, leading significant performance gain. It demonstrates that splitting monomers benefit multimer contact prediction tasks and also implies that monomers data may have the potential for other downstream computational multimer protein tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Protein data bank: the single global archive for 3d macromolecular structure data. Nucleic acids research, 47(DI):D520-DSC D528, 2019.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong. Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876, 2021.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: universal deep-learning model of protein sequence and function. Bioinform., 38(8):2102 2110, 2022. doi: 10. 20.100//bonfoommaiss/uz2200 URL bioinformate www.ormaattttttttoootoeoo.o
Patrick Bryant, Gabriele Pozzati, and Ame Elofsson. Improved prediction of protein-protein inter- actions using alphafold2. Nature communications, 13(1):1-11, 2022.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. supervised learning. arXiv preprint arXiv:2204.042113 2022.
Structure-aware protein self-
Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Charlotte Rochereau, George Church, Peter K Sorger, and Mohammed AlQuraishi. Single-sequence protein structure prediction using language models from deep learning. bioRxiv, 2021.
Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon J Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilczynski, et al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11):1422-1423 2009.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /olo.04800 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11929, 2020.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprini arXiv:2012.09699 2020.
Magnus Ekeberg, Tuomo Hartonen, and Erik Aurell, Fast pseudolikelihood maximization for direct- coupling analysis protein structure from many homologous amino-acid sequences. Journal of Computational Physics, 276:341-355 2014.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Towards crack- ing the language of lifes code through self-supervised deep learning and high performance com- puting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Richard Evans, Michael O'Neill, Alexander Pritzel, Natasha Antropova, Andrew W Senior, Tim- othy Green, Augustin Židek, Russell Bates, Sam Blackwell, Jason Yim, et al. Protein complex prediction with alphafold-muline multimer. BioRxiv, 2021.
Xiaomin Fang, Fan Wang, Lihang Liu, Jingzhou He. Dayong Lin, Yingfei Xiang, Xiaonan Zhang, Hua Wu, Hui Li, and Le Song. Helixfold-single Msa-free protein structure prediction by using protein language model as alternative, arXiv preprint 13221222077.9911,, 2022.
10

---

Under review as a conference paper at ICLR 2023
Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. Protein interface prediction using graph convolutional networks. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 6530-6539. Curran Associates, Inc.. 2017. URL http::tt/pttststeetttt ctc//por72777/11/f66777777722222
neurips
Mu Gao, Davi Nakajima An, Jerry M Parks, and Jeffrey Skolnick. Af2complex predicts direct physical interactions in multimeric proteins with deep learning. Nature communications, 13(1): 1-13, 2022.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12(1):1-14, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-excittton networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1857-1867, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-53) 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: arXiv: 1609,02907 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Pereira, C.J. Burges, L. Bottou, and K.Q Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Asso- ciates, Inc., 2012. URL https: http:::::://pccceeeeccoo neuripeiiss c399862d3b9ds5 wtt.aa/cbbbcccee3266666353------ ppf
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2324 1998.
Mare F Lensink, Guillaume Brysbaert, Nurul Nadzirin, Sameer Velankar, Raphaël AG Chaleil, Tereza Gerguri, Paul A Bates, Elodie Laine, Alessandra Carbone, Sergei Grudinin, et al. Blind prediction of homo-and hetero-protein complexes: The casp1 3-capri experiment. Proteins: Struc- ture, Function, and Bioinformatics, 1200-12100-1221, 2019.
Mare F Lensink, Guillaume Brysbaert, Théo Mauri, Nurul Nadzirin, Sameer Velankar, Raphael AG Chaleil, Tereza Clarence, Paul Bates, Ren Kong, Bin Liu, et al. Prediction of protein as- semblies, the next frontier: The casp 1ssp14-cppri experiment. Proteins: Structure, Fanction, and Bioinformatics, 2021.
Pengyong Li, Jun Wang, Ziliang Li, Yixuan Qiao, Xianggen Liu, Fei Ma, Peng Gao, Seng Song, and Guotong Xie. Pairwise half-graph discrimination: A simple graph-level self--upe strategy for pre-training graph neural networks. arXiv preprint arXiv:2 135677, 2021.
11

---

Under review as a conference paper at ICLR 2023
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel-Zarande Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Yi Liu, Hao Yuan, Lei Cai, and Shuiwang Ji. Deep learning of high-order interactions for pro- tein interface prediction. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery data mining, pp. 679 687, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Josip Mihel, Mile Šikić, Sanja Tomić, Branko Jeren, and Kristian Vlahoviček. Psaia Psaia-proteid protein struc- ture and interaction analyzer. BMC structural biology, 8(1):1-11, 2008.
Milot Mirdita, Lars Von Den Driesch, Clovis Galiez, Maria J Martin, Johannes Söding, and Martin Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and align- ments. Nucleic acids research, 45(D1):D100-D170 2017.
Faruck Morcos, Andrea Pagnani, Bryan Lunt, Arianna Bertolino, Debora S Marks, Chris Sander, Riccardo Zecchina, José Onuchic, Terence Hwa, and Martin Weigt. Direct-coupling analysis of residue coevolution captures native contacts across many protein families. Proceedings of the National Academy of Sciences, 108(49):E1295) 221-E1131,,, 2011.
Alex Morehead, Chen Chen, Ada Sedova, and Jianlin Cheng. Dips-plus: The enhanced database of interacting protein structures for interface prediction. CoRR, abs/2106.0436,, 2021. URL
httts::/aaxiv.com wwtp://////////a
04362
Alex Morehead, Chen Chen, and Jianlin Cheng. Geometric transformers for protein interface contact
prediction. The Tenth International Conference on Learning Representations, ICLR 2022,
Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URUnnttps://ppenrriiee
Forum? (arrum?i--C44410 4 463xx6H1
Franco P Preparata and Michael Shamos. Computational geometry: an introduction. Springer Science & Business Media, 2012.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pp- 8844-8856. PMLR, 2021.
Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding. Hhblits: lightning-fast iterative protein sequence searching by hmm-hmm alignment. Nature methods, 9(2):173-775 2012.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):c001623911 2021.
Raj S Roy, Farhan Quadir, Elham Soltanikazemi, and Jianlin Cheng. A deep dilated convolutional residual network for predicting interchain contacts of protein homodimers. Bioinformatics, 38(7): 1904-1910, 2022.
Ruben Sanchez- -ariia, CO Sorzano, M Carazo, and Joan Segura. BIPSPI: method for the prediction of partner-specifie protein-protein interfaces. Bioinformatics, 35(3):40-4477007 2018. ISSN 1367-4803 doi: 10. 100.00/oooooommmtt///y544.... bioinformatics//tty..
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEEE transactions on neural networks, 20(1): 161-800 2008.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
12

---

Under review as a conference paper at ICLR 2023
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955) 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Thom Vreven, lain H Moal, Anna Vangone, Brian G Pierce, Panagiotis L Kastritis, Mieczyslaw Torchala, Raphael Chaleil, Brian Jiménez-García, Paul A Bates, Juan Femandez-Recio, et al. Updates to the integrated protein-protein interaction benchmarks: docking benchmark version and affinity benchmark version 2. Journal of molecular biology, 427(11):3031-304 2015.
Sheng Wang, Siqi Sun, Zhen Li, Renyu Zhang, and Jinbo Xu. Accurate de prediction of protein contact map by ultra-deep learning model. PLoS computational biology, (3(1)=e100532) 2017.
Martin Weigt, Robert A White, Hendrik Szurmant, James A Hoch, and Terence Hwa. Identification of direct residue contacts in protein protein interaction by message passing. Proceedings of the National Academy of Sciences, 106(1):67-72, 2009.
Ziwei Xie and Jinbo Xu. Deep graph learning of inter-protein contacts. Bioinformatics, 38(4): 947-953, 2022.
Jinbo Xu, Matthew Mcpartion, and Jin Li. Improved protein structure prediction by deep learning irrespective of co-evolution information. Nature Machine Intelligence, 3(7):601-609, 2021.
Yumeng Yan and Sheng- You Huang. Accurate prediction of inter-proteii residue- residue contacts for hoooooooooieoo moigomoiigomrric protein complexes. Briefings in bioinformatics, 22(5):bbab038 2021.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 472-480, 2017.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-toker vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 558-567, 2021.
Hong Zeng, Sheng Wang. Tianming Zhou, Feifeng Zhao, Xiufeng Li, Qing Wu, and Jinbo Xu. Complexcontact: web server for inter-protein contact prediction using deep learning. Nucleic acids research, (6W11):443-W4377 2018.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
13

---

Under review as a conference paper at ICLR 2023
Table 8: Different batch size results three benchmarks. DS: dataset, BZ: batch size.
DS
BZ
Per
L/IO
Pgr L75
P@ L/2
Pe L
R L
R L/2
R L76
R L/10
bc=32
14.19
DIPS-Plus
bsm48
30.40
26,70
20.51
15.87
24,00
16.02
8.56
4.79
21.02
bx=32
23.52
18.94
13.78
11.17
12.58
7,96
4.35
2.52
CASP-CAPRI
bsm48
23.36
18.84
13.92
11.14
13.88
8.64
4.33
2.57
bs=72
16.28
14,91
12.90
10.02
14.34
9.15
4.83
2.68
bs=32
1.39
1.69
2.24
2,05
3.96
2.04
0.56
0.22
DB5
2.53
bx=72
1.42
1.84
1,60
1.65
2.96
1.46
0.69
0.24
A APPENDIX
A. DETAILED NETWORK DESIGN
We use the same network architecture as GenoTrans Morehead al. (2022), the details fol- lows, a nearess neighbor graph Preparata & Shamos (2012) is applied to construct graph of each chain, the residue each chain regarded distance from other residues are considered for connecting with the current residue, the is 20. For network architecture, we use the 2 layers of graph transformer| Dwivedi & Bresson (2020) with batch normalization loffe & Szegedy (2015), each transformer has 4 attention head and 128 hidden size get rich node and edge representations. We also use a layers of dilated residual network Yu et al. (217)) for interleave module. It contains 4 residual block, and each block is composed of 2D convolution with kernel size 3 and instance normalization. A squeeze-ande saneeeze-and-xeitattiion (SE) attention Hu et al. (218)) is added after each block to capture channel-wise information. In addition, we use Adam optimizer Kingma Ba (2014) with the learning rate of - 3, the weight decay rate of 1e- and the batch size of 48 to train the network. The dropout Srivastava et al. (2014) of 0.2 and an early-stopping patience period of 5 epochs are applied to avoid network over-fitting. Since there more non-contact sites than contacts in the protein-proteil contact prediction task, there a huge class imbalance, and we use weighted cross entropy with a positive class weight of 5 to overcome this imbalance. To experiment unbounded dataset DB5, we fine-tune models on 140 training and 35 validation complexes of DB5 with the learning rate of 1e-5
node, and the first k residues with the smallest
A.2 BATCH SIZE TUNING
The benchmarks that we utilize for fine-tunning and evaluating our model cover several contact prediction scenarios, including bounded Morehead et al. (2021). cross-set Lensink et al. (2019 2021) and unbounded situations Vreven et al. (2015). So the hyper-parameter find the optimal batch size on these downstream
fine-tuning may
affect the final results. In this sub-section, we datasets respectively. We set different batch size settings of 32, 48 and 72 on all utilized benchmarks as shown in Table We find that when pre-training with batch size of achieving the best result of 30.40 % on P@ L/10 metric on the bounded benchmark DIPS-Plus On the cross-set benckmark CASP-CAPER fine-tumin tne-tuning with batch size of 32 optimal and can obtain 23.52 % on P@ L/10. For the unbounded DB5 dataset, it achieves the best result of 1.78 % on P@ T,/10 metric with batch size 48.
A.3 COMPARISON WITH DIFFERENT GRAPH PRE-TRAINING METHODS
Since GeoTrans graph-based model, we use different graph pre-training methods to compare with SMP to show the superiority of the SMP design. We compare several simple graph pre-training methods including Mask node Hu et al. (020), Mask edge Hu et al. (2020). and PHD Li et al. 2021). Table shows the contrast results between SMP and other graph pre-training methods. We find that all these graph pre-tree training methods have improved to the baseline method
and SMP outperforms other methods by large margin in all metrics. This experiment indicates that SMP has no task gap between the pre- pre-training stage and the fine-tuning stage in the protein- protein contact prediction task, but other graph pre-training methods have.
14

---

Under review as a conference paper at ICLR 2023
Table 9: Comparison with different graph pre-training methods DIPS-Plus test dataset.
Row
Model
Pe L7T0
Pier L76
P6 L/2
Ríe L
R@ L/2
RØ L75
GeaTrans Morehead et
2022
19.00
17.00
15.00
9.00
4,00
GeoTrans Mask Node Ho
2020
20.87
18.19
14.62
17.46
9.88
4.87
3
Geo'Irans Mask Edge HD vt
2020
20.26
17.67
14.31
16.37
10.47
5.16
GooTrans PHD Li
20.33
17.32
14.50
16.51
10.75
5.34
5
Geo'rans + SMP
30.40
26.70
20.51
24.00
16.02
8.56
Table 10: Comparison with different protein language methods on DIPS-Plus test dataset.
Row
Model
Pg L/10
P@ L/5
Pe L/2
R4 L
R@ L/2
R@ L/5
Geo Trans Morehead et (2022)
19.00
17.00
15.00
9.00
4,00
ESM-10 Kivess
18.96
16.97
14.26
12.85
8.44
3.98
ESM-MSA- Kao er ar
23.49
20.68
16.86
14.73
9.71
4.89
GeoTrans
30.40
8.56
A.4 COMPARISON WITH DIFFERENT PROTEIN LANGUAGE MODELS
The large protein language model has the potential to learn prior biological knowledge from mas- sive monomer sequences. We investigate large-scall protein language models ESM-1b Rives et al. (2211) and SEM-MMA-11 Rao et al. (2021) on DIPS-Plus test dataset as shown in Table I0 It demonstrates that SMP has higher accuracy than other pretrained protein language models for all metrics. Both ESM- (650M parameters) and ESM-MSA-Ib (100M parameters) have much more parameters than our SMP (4.5M parameters), reflecting the effectiveness and light weight of our SMP. Since ESM-Ib does not any MSA and geometric information, its accuracy is lower than the baseline method GeoTrans. ESM-MSA-IE acquires strong biological prior knowledge to get higher accuracy. The above experiments demonstrate that SMP splits the monomer into the pseudo multimer and pre-training the pseudo multimer could be more beneficial than using monomer pre- training directly.
15


Reviews:
[{"review_id": "V4fX_GW1pkn", "reviewer": "Reviewer_HQ2g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new approach for protein-protein contact prediction that is based on monomer data (this has been used in related work) and particularly cuts monomers into two sub-parts  and pre-trains the model to merge them back. The results show huge performance gains compared to related works.\n", "summary_of_the_review": "Altogether, to me this seems to be a solid contribution with some issues that could be corrected. I am no expert for reasoning over proteins, therefore other reviewers would have to confirm if the claims w.r.t. novelty and SOTA are true. \n\nMy main concern is that the work is very application-focused and the paper does not really represent new, more general ML technology. But this seems to be a more general question, how such papers are to be handled. I didn't find guidance for this from ICLR. I would find a more general consideration of GNN pretraining more insightful for the community.\n", "strength_and_weaknesses": "(+) The paper is mostly straightforward and concise, the approach is reasonable.\n\n(+) The evaluation is extensive, contains various datasets and models, ablation studies, and more detailed analysis. \n\n(-) For someone coming from ML some parts are a bit unclear. For example:\np.1 \"the constraints between given protein sequences\" - what exactly is the form of these constraints?\n\n(-) Related work:\n- Overall the paper focuses heavily on the application. Pretraining has become a huge field for graph neural networks in very short time, and the paper is missing related work from there completely, which could be applied to the monomers. At a venue such as ICLR, that work should be mentioned. For example:\nPairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks, IJCAI 21\n- \"ComplexContact Zeng et al. (2018) is the first work introducing the monomer data into the multimer contact prediction task.\" - This seems to be an important related work, how exactly do they use the data?\n\n(-) The results seem nearly a bit too good and I wonder what is the issue. It seems that the other models do not use pre-training, therefore I wonder why the authors do not compare to ComplexContact in more metrics, and don't apply some existing GNN pre-training approaches.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NgO-5zENq8", "reviewer": "Reviewer_2Wyz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes simple pre-training strategy for multi-mer contact prediction.  The strategy is shown to provide empirical benefit across several benchmark tasks.  However the paper does not compare to some of the most widely used methods for predicting contacts (AlphaFold and RosettaFold).", "summary_of_the_review": "A clear explanation and empirical demonstration of simple idea, but with insufficient demonstration of the novelty and comparison + discussion of existing work (namely AlphaFold and RosettaFold).", "strength_and_weaknesses": "Strengths:\n* The chief benefit of the suggested approach is it's simplicity.\n* The comparison of the performance of GeoTrans with and without SMP is good ablation for checking the extent to which the method improves performance.\n\nWeaknesses:\n* Discussion of prior work.  RoseTTAFold2 is explicitly geared towards interaction prediction, as is AF2 Multimer.  While these methods are primarily built for structure prediction (rather than phrased as for contact prediction) they may easily used for contact prediction.  For example using the predicted alignment error (PAE) output by AF2.\n*  Predicted alignment error as been shown to be effective in predicting existence and strength of contacts in the context of binder design (Bennet et al. 2022).   For RosettaFold a simple option could be pLDDT at the interacting residues.\n* How does pre-training compare to relying on generalization from structure prediction with large (e.g. 384 residue crops), as in AlphaFold?\n* An empirical comparison to AF2 or RosettaFold is crucial.  I will not consider changing my score if a comparison is not to be added.\n\nIn some cases, it seems that the pre-training strategy does not help (in particular, in table 2 in some comparisons to GeoTrans).  Could the authors comment on the situations in which they expect the pre-training will have the largest / smallest improvements?\n\n\nNit:\n* Check for typos: e.g. suppresses —> surpasses?\n* What do you mean by “rich” information in the title?  Would be helpful to clarify this vague language in the main text or choose alternative phrasing.\n\n\nReferences:\nBennett, Nathaniel, et al. \"Improving de novo protein binder design with deep learning.\" bioRxiv (2022).\nBaek, Minkyung, et al. \"Accurate prediction of protein structures and interactions using a three-track neural network.\" Science 373.6557 (2021): 871-876.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4XtPUmSKeT", "reviewer": "Reviewer_oqAy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary of the paper\nThe paper proposes a new method for pretraining models for protein-protein contact prediction. The authors created pretraining data by collecting sequences and 3D data from the PDB database. \n\nThey break a monomer (a single chain) into a multimer with two chains at a random location in the sequence and train a model to predict the contacts of these two chains. The pseudo-contact ground truth between pseudo multimers is defined using the Euclidean distance between residue coordinates provided within the 3D data (residues within 5 Angstrom are considered as having a contact). \n\nThe pre-trained models are then fine-tuned further on protein-protein contact prediction datasets using supervised training approaches. \n\nThe authors demonstrated that although this is a simple pretraining approach, it provides a significant improvement over SOTA-supervised methods and the other methods that use pre-trained models for protein representation using language models such as the ProteinBert.\n\n\n\n", "summary_of_the_review": "Summary of the comments\nThis is an interesting work that has a good impact on the application domain.\nMy main concerns are:\n\n+  potential leakage, please fix this and report new results, if the new results are still significant, I am happy to change my score\n\n+ additional comparisons when replacing ProteinBert with ESM-1b, ESM-MSA-1b are needed because these models demonstrated SOTA results on supervised/unsupervised contact prediction.\n\n+ additional comparisons with pretraining monomer contact prediction using PDB data and ESM-1b as back-bond features for proteins are needed to demonstrate that the Split and Merge Proxy is needed.", "strength_and_weaknesses": "Strengths\n\nInteresting idea of creating pretraining data.\n\nThe improvement over the SOTA-supervised learning approach is impressive.\n\n\nWeaknesses\n\nPotential leakage happens\nSince the SMP model was pretrained on the PDB dataset. There might be potential leakage happens if the test set of DIPS-Plus, DB5, and CASP-CAPRI  overlap with the data used for pretraining. Even though the objective of SMP which focuses on reconstructing the contacts of broken monomer and the objective of a multimer are different, seeing test instances during pretraining steps provides an unfair advantage over other methods that do not see the test data distribution during their learning. I suggest the authors check for overlapping instances between PDB and  DIPS-Plus, DB5, and CASP-CAPRI and remove them before pretraining to avoid leakage.\n\nComparison with pretrained protein models\nEven the authors have compared their methods to the pretrained language models ProteinBert with the results reported in Table 4, I would suggest the authors compare their approaches to the following pretrained models available at https://github.com/facebookresearch/esm:\n\n+ ESM-1b: this is a standard language model trained on the large database of  protein sequences, demonstrated very good results on contact prediction\n\n+ ESM-MSA-1b: leveraging MSA information for predicting better contact \n\nThis ESM pretrained model family was trained with very large data and demonstrated SOTA results on contact prediction.\n\nComparison with pretrained monomer contact prediction \nTo demonstrate that it is necessary to break the monomer into two chains and predict the merged contact with the Split and Merger proxy, the following experiments need to be done on the PDB sequences with 3D structure:\n\n+ Take the representation from ESM-1b as the representation of the protein sequence.\n\n+ Predict the contact matrix constructed by looking at the Euclidean distance between residues coordinates provided in the protein 3D data\n\n+ Use the new representation of ESM-1b pretrained on the monomer contact prediction for GeoTrans to further fine-tune for multimer contact prediction.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-d9Nf4Naokn", "reviewer": "Reviewer_bLda", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel self-supervised approach to pretrain models on protein-protein contact predictions. The approach, called Split and Merger Proxy (SMP), uses protein monomer samples by splitting them into two sub-parts as a proxy to protein-protein complex and pre-trains the model to predict its original ‘merged’ form. Through this approach, the models can break away from the limitations of the small dataset of labeled protein-protein contact. Experiments have shown that SMP improves the performance of current baselines to achieve state-of-the-art results in downstream protein-protein contact prediction.", "summary_of_the_review": "The proposed approach is novel and a clever way to exploit the relatively larger dataset of protein monomers. The empirical performance gains are impressive. However, the impact might be limited to only protein-protein contact prediction.", "strength_and_weaknesses": "Strength:\nThe proposed SMP approach is a smart and novel trick to pretrain models without the use of labeled protein-protein contact dataset.\nStrong empirical results are shown with proposed approach.\n\nWeaknesses:\nImpact seems limited to protein-protein contact prediction\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject