# LPBERT: A Protein-Protein Interaction Prediction Method Based on a Pre-Trained Language Model
## 1 Introduction  
Proteins play a core role within organisms. They participate in numerous important biological processes in complex biological environments and are a crucial cornerstone of biomolecules. Proteins perform a variety of functions, including facilitating reactions [1, 2], assisting in signal transduction [3], regulating biological activities [4], and transporting key substances [5, 6]. These functions manifest externally as interactions between two or more proteins, known as protein-protein interactions (PPIs). Research on PPIs is instrumental in revealing the operational mechanisms within organisms, understanding disease mechanisms, and developing new drugs. Initially, researchers verified the interactions between proteins by traditional experimental methods, such as yeast two-hybrid [7, 8] and co-immunoprecipitation [9]. These methods can accurately detect interactions between proteins. However, long experimental cycles, a low efficiency, and experimental failure risks are major drawbacks of traditional methods. These factors prompted some researchers to seek more efficient and reliable alternative methods. Protein sequences, which represent the primary structure of proteins, contain key information about their composition. They are more readily available and understandable than 3D structures of proteins, amino acid physicochemical properties, and other information. The advancement of high-throughput sequencing technologies [10] has provided a wealth of protein sequence data for PPI research, which has driven the development of numerous protein-sequence-based PPI prediction methods. These methods use protein sequences as the entry point and achieve good results in PPI prediction by combining deep learning techniques. PIPR [11] employed residual Recurrent Convolutional Neural Networks (RCNNs) to capture the latent semantic features of protein sequences, then used different loss functions in a Multilayer Perceptron (MLP) to adapt to different prediction tasks. OR-RCNN [12] used the Skip-Gram method for pre-training protein sequences, employed the RCNN module to encode protein embedding representations, and combined it with an ordinal regression-based classifier to calculate interaction scores. DeepTrio [13] utilized multiple parallel Convolutional Neural Networks (CNNs) to capture the contextual information of protein sequences from multiple scales. Notably, DeepTrio also introduced a single-protein class to distinguish the relative and intrinsic properties. RAPPPID [14] utilized the Sentencepiece [15] algorithm to classify protein sequences into common groupings and encoded the grouped sequences with an AWD-LSTM [16] regularization encoder. This method provided a solution to the generalization problem in PPI prediction tasks. The DCSE [17] applied the concepts of natural language processing (NLP). Initially, each amino acid is treated as a word, and the amino acids are mapped to an N-dimensional vector. Subsequently, DCSE employed two different networks to extract global and local features from the protein sequences. ADH-PPI [18] employed two types of dropout strategies to generate corresponding embedding matrices for sequences. It then utilized CNN and Long Short-Term Memory (LSTM) to extract matrix information and employed an attention mechanism to sift important features. The ECA-PHY [19] integrated five sequence-encoding methods: AAC, DDE, MMI, CT, and GTPC. It combined effective channel attention, a Bidirectional Gated Recurrent Unit (BiGRU), and 1D-CNNs to more effectively predict human-virus PPIs. TAGPPI [20] used a CNN to process sequences and applied AlphaFold [21] to predict the corresponding contact maps of sequences. It further processed the contact maps through a graph attention neural network to extract three-dimensional protein structural features. The model then combined sequence and structural features through weighted addition to generate the final fused feature vector.