# LPBERT: A Protein-Protein Interaction Prediction Method Based on a Pre-Trained Language Model
## 1 Introduction
Similarly, SpatialIPPI [22] first generated corresponding complex structures for protein sequences via AlphaFold Multimer [23], and then processed the complex structures via one-hot encoding, volume encoding, or distance encoding. In current research, there are many preliminary processing methods for protein sequences, most of which convert character sequences into corresponding vector representations through methods such as one-hot encoding and word embeddings for subsequent studies. Although these methods have made some progress in sequence processing, they still have limitations in capturing complex semantic relationships and long-distance dependencies. For example, one-hot encoding cannot express the semantic relationships between words. While word embedding methods improve on this, their fixed window size limits their ability to handle long-distance dependencies. The success of language models in the NLP field has proven their potential in completing natural language tasks. Leveraging the similarity between protein sequences and natural language has also promoted the application of language models in proteomics. TAPE [24], ProtTrans [25], ESM [26], and ProteinBERT [27] have become popular protein language models in recent years. By pre-training on large-scale protein sequence datasets, they can effectively capture the characteristic information of proteins from sequences and provide support for downstream protein tasks. xCAPT5 [28] innovatively selected the ProtT5-XL-UniRef50 [29] protein language model to generate encoded representations for protein sequences. It used multi-core CNNs to capture complex interaction features at both the micro and macro levels and integrated XGBoost [30] to enhance the predictive performance. [PERSON] et al. [31] performed a similar study by training the MindSpore ProteinBERT (MP-BERT) model from transformers. They used transfer learning techniques to construct the MPB-PPI architecture for PPI prediction. Protein sequences are extremely similar to a natural language in form or can be viewed as a unique natural language, which makes it a reasonable choice to apply language models to process protein sequences. The successful application of xCAPT5 and MPB-PPI for PPI prediction demonstrated that this research direction is feasible. This paper introduces the LPBERT deep learning framework. It utilizes the protein language model ProteinBERT [27] as the encoder to obtain embedding representations from protein sequences. By integrating CNNs, Transformer encoders [32], and Bidirectional Long Short-Term Memory (BiLSTM), LPBERT can further process the embedding representations and enhance the predictive ability of the method. In this work, our main contributions are as follows:  
* First, we introduced ProteinBERT in the field of PPI prediction to obtain rich embedding representations of protein sequences. * Second, with ProteinBERT as the core, we designed the Local Convolutional Recurrent Neural Network (LCR) module and the Global Convolutional Transformer encoder (GCT) module to process the extracted embedding representations and construct the LPBERT framework, which achieved satisfactory results in the PPI prediction tasks. * Third, we conducted extensive experiments and performance evaluations to demonstrate the superior performance of the LPBERT framework on various PPI prediction benchmark datasets. In addition, we conducted ablation experiments to verify the contribution of the LCR and GCT modules to the overall model performance. Through these experiments, we were able to fully understand the superiority and potential application value of LPBERT in the PPI prediction task.