SlowGuess/ACL_2020_OCR huggingface dataset:
Pratik Dutta, Sriparna Saha
Indian Institute of Technology Patna
(pratik.pcs16, sriparna)@iitp.ac.in
An in-depth exploration of protein-protein interactions (PPI) is essential to understand the metabolism in addition to the regulations of biological entities like proteins, carbohydrates, and many more. Most of the recent PPI tasks in BioNLP domain have been carried out solely using textual data. In this paper, we argue that incorporation of multimodal cues can improve the automatic identification of PPI. As a first step towards enabling the development of multimodal approaches for PPI identification, we have developed two multimodal datasets which are extensions and multimodal versions of two popular benchmark PPI corpora (BioInfer and HRPD50). Besides, existing textual modalities, two new modalities, 3D protein structure and underlying genomic sequence, are also added to each instance. Further, a novel deep multi-modal architecture is also implemented to efficiently predict the protein interactions from the developed datasets. A detailed experimental analysis reveals the superiority of the multi-modal approach in comparison to the strong baselines including uni-modal approaches and state-of-the-art methods over both the generated multimodal datasets. The developed multi-modal datasets are available for use at https://github.com/sduttap16/MM_PPI_NLP.
Understanding protein-protein interactions (PPI) is indispensable to comprehend different biological processes such as translation, protein functions (Kulmanov et al., 2017), gene functions (Dutta and Saha, 2017; Dutta et al., 2019b), metabolic pathways, etc. The PPI information helps researchers to discover disease mechanisms and plays seminal role in designing the therapeutic drugs (Goncearenco et al., 2017). Over the years, a significant amount of protein-protein interaction information has been published in scientific articles
in unstructured text formats. However, in recent years, there has been an exponential rise in the number of biomedical publications (Khare et al., 2014). Therefore, it becomes imperative, urgent and of extreme interest to develop an intelligent information extraction system to assist biologists in curating and maintaining PPI databases.
This pressing need has motivated Biomedical Natural Language Processing (BioNLP) researchers to automatically extract PPI information by exploring various AI techniques. Recent advancements in deep learning (LeCun et al., 2015)(Bengio et al., 2007) have opened up new avenues in solving different well-known problems ranging from computational biology (Alipanahi et al., 2015; Dutta et al., 2019a), machine translations (Cho et al., 2014), image captioning (Chen et al., 2017). Subsequently, there is a notable trend in using deep learning for solving different natural language processing (NLP) tasks in the biomedical and clinical domains (Asada et al., 2018; Alimova and Tutubalina, 2019) including the identification of protein-protein interactions from biomedical corpora (Yadav et al., 2019; Peng and Lu, 2017). Multi-modal deep learning models, combining information from multiple sources/modalities, show promising results compared to the conventional single modal-based models while solving various NLP tasks like sentiment and emotion recognition (Qureshi et al., 2019, 2020), natural language generation, machine translation (Poria et al., 2018; Zhang et al., 2019; Qiao et al., 2019; Fan et al., 2019) etc. There exist few popular multi-modal datasets which are extensively used in solving various problems in NLP like emotion recognition from conversations (Poria et al., 2018; Chen et al., 2018), image captioning (Lin et al., 2014), sentiment analysis (Zadeh et al., 2016), etc. Compared to single modal-based approaches, multi-modal techniques provide a more comprehensive perspective of the
6396
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6396-6407 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
dataset under consideration.
Despite the popularity of multi-modal approaches in solving traditional NLP tasks, there is a dearth of multi-modal datasets in BioNLP domain especially for the PPI identification task. The available PPI benchmark datasets contain solely the textual knowledge of different protein pairs, which do not help in anticipating the molecular properties of the proteins. Hence, along with the textual information, incorporation of molecular structure or underlying genomic sequence can aid in understanding the regulations of the protein interactions. The integration of multi-modal features can help in obtaining deeper insights but the concept of multimodal architecture, for textual and biological aspects, has not been cultivated much in the BioNLP domain (Peissig et al., 2012; Jin et al., 2018).
1.1 Motivation and Contribution
The main motivation for this research work is to generate multi-modal datasets for PPI identification task, where along with the textual information present in the biomedical literature, we did explore the genetic and structure information of the proteins. The biomedical and clinical text database is an important resource for learning about physical interactions amongst protein molecules; however, it may not be adequate for exploring biological aspects of these interactions. In the field of Bioinformatics, there are various web-based enriched archives<sup>12</sup> that contain multi-omics biological information regarding protein interactions. The integration of multi-omics information from these aforementioned databases helps in understanding the various physiological characteristics (Sun et al., 2019; Ray et al., 2014; Amemiya et al., 2019; Hsieh et al., 2017; Dutta et al., 2020). Hence, in our current work, along with the textual information from biomedical corpora, we have also incorporated structural properties of protein molecules as biological information for solving PPI task. For structural information of proteins, we have considered the atomic structure (3D PDB structure) and underlying nucleotide sequence (FASTA sequence) of protein molecules. In the BioNLP domain, collection of biological data (multi-omics information) from the text corpus is little difficult. To obtain the aforementioned information about other modalities, we need to exploit different web-based archives that
are meant for biological structures.
Drawing inspirations from these findings, we have generated a protein-protein interaction-based multi-modal dataset which includes not only textual information, but also the structural counterparts of the proteins. Finally, a novel deep multimodal architecture is developed to efficiently predict the protein-protein interactions by considering all modalities. The main contributions of this study are summarized as follows:
1. For this study, we extend and further improve two biomedical corpora containing PPI information for multi-modal scenario by manually annotating and web-crawling two different bio-enriched archives.
2. Our proposed multi-modal architecture uses self-attention mechanism to integrate the extracted features of different modalities.
3. This work is a step towards integrating multi-omics information with text-mining from biomedical articles for enhancing PPI identification. To the best of our knowledge, this is the first attempt in this direction.
4. The results and the comparative study prove the effectiveness of our developed multimodal datasets along with proposed multimodal architecture.
There are few works (Ono et al., 2001; Blaschke et al., 1999; Huang et al., 2004) which focus on rule-based PPI information extraction method such as co-occurrence rules (Stapley and Benoit, 1999) from the biomedical texts. In (Giuliano et al., 2006), relation is extracted from entire sentence by considering the shallow syntactic information. (Erkan et al., 2007) utilize semi-supervised learning and cosine similarity to find the shortest dependency path (SDP) between protein entities. Some important kernel-based methods for PPI extraction task are graph kernel (Airola et al., 2008a), bag-of-word (BoW) kernel (Sætre et al., 2007), edit-distance kernel (Erkan et al., 2007) and all-path kernel (Airola et al., 2008b). (Yadav et al., 2019) presented an attention-based bidirectional long short-term memory networks (BiLSTM) model that uses SDP between protein pairs, latent PoS and position
<sup>1</sup>https://www.cancer.gov
$^{2}$ https://www.ncbi.nlm.nih.gov/
6397
Generated Instances of our multi-modal dataset	Protein pairs	Gene pairs	PDB ID pairs	Ensembl ID pairs	Interaction type
Protein1	Protein2	Gene1	Gene2	PDB1	PDB2	Ensembl1	Ensembl2
Megalin and cubilin: multifunctional endocytic receptors PROTEIN1 and PROTEIN2 are two structurally different endocytic receptors that interact to serve such functions	Megalin	cubilin	LRP2	CUBN	2MOP	3KQ4	ENSG00000081479	ENSG00000107611	TRUE
Megalin and PROTEIN1: multifunctional endocytic receptors Megalin and PROTEIN2 are two structurally different endocytic receptors that interact to serve such functions	cubilin	cubilin	CUBN	CUBN	3KQ4	3KQ4	ENSG00000107611	ENSG00000107611	FALSE
PROTEIN1 and cubilin: multifunctional endocytic receptors Megalin and PROTEIN2 are two structurally different endocytic receptors that interact to serve such functions	cubilin	Megalin	CUBN	LRP2	3KQ4	2MOP	ENSG00000107611	ENSG00000081479	FALSE
Megalin and PROTEIN1: multifunctional endocytic receptors PROTEIN2 and cubilin are two structurally different endocytic receptors that interact to serve such functions	cubilin	Megalin	CUBN	LRP2	3KQ4	2MOP	ENSG00000107611	ENSG00000081479	FALSE
PROTEIN1 and PROTEIN2: multifunctional endocytic receptors Megalin and cubilin are two structurally different endocytic receptors that interact to serve such functions	cubilin	Megalin	CUBN	LRP2	3KQ4	2MOP	ENSG00000107611	ENSG00000081479	FALSE
PROTEIN1 and cubilin: multifunctional endocytic receptors PROTEIN2 and cubilin are two structurally different endocytic receptors that interact to serve such functions	Megalin	Megalin	LRP2	LRP2	2MOP	2MOP	ENSG00000081479	ENSG00000081479	FALSE
embeddings for PPI extraction. Some of the popular deep learning based PPI extraction techniques are reported by (Shweta et al., 2016; Zhao et al., 2016; Hua and Quan, 2016; Hsieh et al., 2017).
3 Dataset Formation and Preprocessing
In this study, we have extended, improved, and further developed two popular benchmark PPI corpora, namely BioInfer $^{3}$  and HRPD50 $^{4}$  dataset for the multi-modal scenario. Along with the textual information, these enhanced multi-modal datasets contain the biological counterparts of the interacting or non-interacting protein pairs. Biological information comes from the underlying FASTA sequence and the atomic structures of interacting protein pairs.
<sup>3</sup>http://corpora.informatik.hu-berlin.de/
4https://goo.gl/M5tEJj
Figure 1: An example of generating instances along with the structural and sequence counterparts of our multimodal dataset from HRPD50 dataset. PDB ID and Ensembl ID are utilized for obtaining protein 3D atomic structure and underlying FASTA sequence, respectively.
Figure 2: Statistics of positive and negative instances across our developed multi-modal datasets.
3.1 Dataset Preparation
Firstly, we have extracted data, primarily consisting of two and more protein entities, from the XML representations of two PPI corpora mentioned earlier. To simplify this complex relations among multiple protein entities, we have considered only a single protein pair at a time and found out if they are interacting or not. Among these relations, we have considered positive instances that are directly mentioned in the dataset. The other interactions are considered as non-interacting proteins, i.e., negative instances.
Consider an instance of HRPD50 dataset, "Megalin and cubilin: multifunctional endocytic receptors Megalin and cubilin are two structurally different endocytic receptors that interact to serve such functions"(Figure 1). In this particular example, we have four protein entities but we have considered the interactions between two proteins at a time and arrived at six possible relations (shown in table of Figure 1). Among these relations, only one pair (Megalin, cubilin) is denoted as interacting proteins in the HRPD50 dataset. Hence, the number of instances in our dataset is much higher than those in BioInfer and HRPD50 datasets.
After generating both positive and negative instances, next we have downloaded other two modalities. To download the genomic sequence and the 3d structure of proteins, the ensemble ID and PDB ID of the proteins are required to be known. But all the biological archives contain the relationships between gene and PDB ID or Ensemble ID instead
An Instance from HRPD50
- Megalin and cubilin: multifunctional endocytic receptors Megalin and cubilin are two structurally different endocytic receptors that interact to serve such functions
6398
Figure 3: An overview of the proposed deep multi-modal architecture for predicting protein-protein interactions. For each modality, we have designed different deep learning based models which are finally integrated using self-attention mechanism.
of any relationship between the proteins and aforementioned IDs. Hence, we have used manual annotation to find out the respective gene names of each protein name and then python based methodologies to find out Ensembl ID and PDB ID of each of these genes. These IDs help us in downloading the underlying genomic sequence (FASTA sequence) from  $^{5}$  and structures of these proteins (3D PDB structure) from the RCSB Protein Data Bank  $^{6}$  archive. The pre-processing and generation of the multimodal datasets from the biomedical corpora
are pictorially depicted in Figure 1. The complete exemplified multi-modal datasets are available at the provided GitHub link.
3.2 Dataset Annotation and Statistics
A major challenge in creating the dataset is to manually encode the relationships between genes and proteins, a many to many mapping for biological reasons. Hence, to find out the genes which are more related to a particular protein, we asked three annotators who have strong biological knowledge. The disagreement between the annotators was less than  $1\%$  and the disagreement is solved by the ma
<sup>5</sup>https://useast.ensembl.org/index.html
<sup>6</sup>http://www.rcsb.org/
jority voting. The total number of instances of the developed multi-modal datasets are shown in Figure 2.
4 Problem Formalization
Our goal is to develop a deep multi-modal architecture that can efficiently predict whether two proteins are interact with each other or not from the developed multi-modal datasets. Formally, consider the multi-modal dataset  $\mathbb{D} = \{S^i\}_{i=1}^N = \{(I_{Text}^i, I_{Struc}^i, I_{Seq}^i)\}_{i=1}^N$  consisting of  $N$  instances.  $\forall i \in \{1, 2, \dots, N\}, I_{Text}^i, I_{Struc}^i$  and  $I_{Seq}^i$  represent the textual, structural and sequence modality of  $S^i$  sentence/instance, respectively. The proposed PPI task for an instance  $S^i$  is mathematically formulated as
$$
f _ {a c t} \Big (f _ {s a} \big (\mathbb {M} _ {1} (I _ {T e x t} ^ {i}), \mathbb {M} _ {2} (I _ {S t r u c} ^ {i}), \mathbb {M} _ {3} (I _ {S e q} ^ {i}) \big) \Big)
$$
Here  $\mathbb{M}_1, \mathbb{M}_2, \mathbb{M}_3$  are three different deep learning based models for text, structure and sequence modality, respectively. The extracted features are fused by self attention mechanism ( $f_{sa}$ ) which is finally fed to an activation function ( $f_{act}$ ) for predicting protein interactions.
The major steps of our proposed multi-modal architecture are shown in Figure 3.
5.1 Feature Extraction from Textual Modality
The proposed deep learning model  $(\mathbb{M}_1)$  for extracting features from textual modality is described in Figure 4. Firstly, we use BioBERT v1.1(Lee et al., 2019) model to provide a vector representation  $(u^i \in \mathbb{R}^d)$  of the textual instance  $(I_{Text}^i)$ . With almost same architecture of BERT (Bidirectional Encoder Representation from Transformers) model (Devlin et al., 2018), BioBERT v1.1 is pre-trained on 1M PubMed abstracts. Here, each sentence is embedded as a unique vector of size 768 (i.e.,  $d = 768$ ) by averaging the last four transformer layers of the first token ([CLS]) of BioBERT model. Inspired by the efficient usage of stacked Bidirectional long short term memory (BiLSTM)(Yadav et al., 2019), we use this to encode the embedded representation  $(u^i)$ . In stacked BiLSTM, the  $l^{th}$  level BiLSTM computes the forward  $\left(\overleftarrow{h_{u^i}^l}\right)$  and backward hidden states  $\left(h_{u^i}^l\right)$  which are then concatenated and fed to the next  $(l + 1)^{th}$  level of
Figure 4: Proposed hybrid model combining BioBERT and stacked BiLSTM for the Textual modality.
BiLSTM layer. Therefore, the final representation  $(F_{Text}^{i})$  of  $I_{Text}^{i}$  is obtained from the last layer  $(L)$  of the stacked BiLSTM model as
$$
F _ {T e x t} ^ {i} = \mathbb {M} _ {1} \left(I _ {T e x t} ^ {i}\right) = \left[ \overrightarrow {h _ {u ^ {i}} ^ {L}} \bigoplus \overleftarrow {h _ {u ^ {i}} ^ {L}} \right] \tag {1}
$$
5.2 Sequence Feature Extraction
Firstly, we have downloaded the FASTA sequence of protein pairs of an instance  $(S^i)$  from Ensembl genome browser. In this modality, each protein  $(I_{Seq}^{i})$  is represented as string of four nucleotides, i.e.,  $I_{Seq}^{i} = \{A,T,G,C\}^{+}$ . The underlying genomic sequence is considered as a separate channel of the text modality. Since molecular properties of protein molecules are heavily dependent on the sequence of nucleotides, we apply capsule network (Sabour et al., 2017) to capture the spatial information between the nucleotides. In this regard, firstly, we have converted all four nucleotides into one-hot vector representation, i.e., the protein is represented as a 2D matrix,  $\mathbb{O} = \{0,1\}^{4\times m}$  where  $m$  is the number of nucleotides in the sequence. Now, three convolutional layers  $(f_{conv})$  are applied on  $\mathbb{O}$  where the output of the third layer is fed to the primary capsule. Finally, the output of the primary capsule is fed to secondary capsule which
Figure 5: Capsule network-based deep model for extracting features from underlying genomic sequence of proteins.
Figure 6: Graph convolutional neural network-based deep model for extracting features from molecular structure of proteins.
provides the final representation  $(F_{Seq}^{i})$  of the sequence modality. The final feature vector obtained from the developed deep architecture  $(\mathbb{M}_2)$  is
$$
F _ {S e q} ^ {i} = \mathbb {M} _ {2} \big (I _ {S e q} ^ {i} \big) = f _ {c a p s u l e} \Bigg (f _ {C O N V} \big (\mathbb {O} \big) \Bigg)
$$
5.3 Structural Feature Extraction
For the structure modality, firstly we have downloaded protein 3D structure from RCSB protein data bank website and obtained the atomic coordinates from the PDB file. Among all the modalities, structural modality is the most relevant modality for inferring biological information. In this modality, we have considered the atomic structure of the proteins. Inspired by the inherent capabilities of graph convolutional neural network (Kipf and Welling,
2016; Zamora-Resendiz and Crivelli, 2019) for understanding the effective latent representation of the graph, we have used it to learn a local neighborhood representation around each atom of the proteins. For this structural modality, the developed model (Figure 6) learns the chemical bonding information from the atomic structure of the proteins rather from its corresponding image. Each protein, which consists of a set of atoms  $\{a_1,a_2,\dots ,a_n\}$ , has an adjacency matrix,  $A\in \{0,1\}^{n\times n}$ , and a node feature matrix,  $X\in \mathbb{R}^{n\times d_v}$ . In this study, we have considered two proteins  $(P_{1},P_{2})$  in an instance and extracted the insightful features  $(y_{1},y_{2})$  using GCNN and then concatenated them for the final representation  $(F_{Struc}^{i})$ . The GCNN takes  $A$
6401
and  $X$  as inputs of the proteins and the structural feature represented as
$$
F _ {S t r u c} ^ {i} = \mathbb {M} \left(I _ {S t r u c} ^ {i}\right) = \left[ y _ {1} \bigoplus y _ {2} \right] \tag {2}
$$
$$
y _ {j \mid j \in \{1, 2 \}} = f \left(H _ {j} ^ {i}, A _ {j}\right) = \sigma \left(A _ {j}, H _ {j} ^ {i}, W _ {j} ^ {i}\right) \tag {3}
$$
Here,  $\oplus ,f,\sigma$  are the concatenation operator, a non-linear activation function and the propagation rule, respectively.  $W_{j}^{i}$  is the weight matrix of layer  $i$  of protein  $P_{j}$  and  $H_{j}^{i}$  is defined as  $f(H_j^{i - 1},A)$  where  $H_{j}^{0} = X_{j}$
5.4 Attention-based Multi-modal Integration
After extracting the features of three modalities (textual, protein sequence and protein structure), we have fused the features using attention mechanism. Attention mechanism has the ability to focus on the features which are the most relevant to a context specific task. In this study, we have used self-attention mechanism of the transformer model which concatenates the final integrated feature representations  $(\mathbb{F})$  of  $i^{th}$  instance  $(S_{i})$  using the following formula.
$$
\mathbb {F} = \left[ W _ {T e x t} ^ {i} F _ {T e x t} ^ {i} \bigoplus W _ {S e q} ^ {i} F _ {S e q} ^ {i} \bigoplus W _ {S t r c} ^ {i} F _ {S t r c} ^ {i} \right] \tag {4}
$$
Here,  $W_{i}$  represents the attention weight of  $i$ th modality. Finally, this final representation  $(\mathbb{F})$  is fed to softmax layer for final classification.
6 Experimental Results and Analysis
In this section, we have briefly described the details of the hyper-parameters and the comparative analysis of the proposed deep multi-modal architecture. To explore the role of developed multi-modal datasets along with the proposed multi-modal architecture for predicting the protein interactions, several experiments are conducted for evaluating each modality and also different combinations of the modalities. Additionally, we have compared the performance of our multi-modal approach with various state-of-the-art methods.
6.1 Details of Hyper-parameters
In our proposed multi-modal architecture, for the final classification we have used softmax. Adam optimizer is used through out the multi-modal architecture. In stacked BiLSTM model for textual modality, 6 (i.e.,  $L = 6$ ) layers of BiLSTM are used.
In case of structural features, graph convolutional neural network with two hidden layers is used. For sequence modality, capsule network followed by three ReLU convolutional layers are used. In the developed capsule network, the number of primary capsules are eight along with two secondary capsules. Finally, self-attention of transformer model is utilized for integrating the features of different modalities. For self-attention, we have used three encoders which are followed by a fully connected network with two hidden layers. The output of the fully connected network is then fed to softmax for final classification.
6.2 Comparative analysis with baselines
For baselines, we have compared our multi-modal approach with three uni-modal, three bi-modal and two other multi-modal architectures.