SlowGuess/ACL_2020_OCR huggingface dataset:
- Textual modality BioBERT and stacked BiLSTM are utilized for this model.
- Protein sequence modality Capsule network is utilized to understand the underlying features extracted from the protein sequences.
- Protein structural modality Inspired by the effective performance of GCNN in understanding the graph representation, GCNN is applied on atomic structure of proteins.
- 3D structural + sequence modality In this bimodal architecture, GCNN and capsule network are used for structural and sequence modality, respectively. Finally, self-attention is utilized to understand the integrated features of these two modalities.
- Textual + sequence modality In this model, self-attention is applied on the extracted features of textual and sequence modality.
- Textual + 3D structure modality: To learn the different attributes discussed in the text and protein structural modality, self-attention mechanism is applied to fuse them.
- Multi-modal approach 1 This architecture of this baseline is the same as the proposed multimodal approach, except the learned features of each modality are simply concatenated instead of using any attention mechanism.
- Multi-modal approach 2 In this model, attention mechanism is applied for integrating
6402
Textual
modality	Protein sequence
modality	Protein structural
modality	Textual + sequence
modality	Textual + 3D
structure modality	3D structural +
sequence modality	Multi-modal
approach 1	Multi-modal
approach 2	Proposed
approach
BioInfer	Precision	54.42	50.63	59.34	64.51	69.04	68.15	79.16	83.77	86.81
Recall	87.45	83.68	91.63	87.45	88.49	89.53	87.44	86.40	89.53
F-measure	67.09	63.09	72.04	74.25	77.54	77.39	83.11	85.07	88.15
HRPD50	Precision	90.44	86.95	91.75	91.01	94.79	93.57	96.51	96.61	96.93
Recall	58.67	41.32	69.01	62.81	75.21	75.21	74.38	76.44	78.51
F-measure	71.17	56.02	78.77	74.32	83.87	83.39	84.01	85.35	86.75
the features of textual, protein sequence and structural modalities. For extracting the features from textual, protein sequence and protein structure, we use BioBERT, BiLSTM and CNN, respectively.
The results reported in Table 1 illustrate the supremacy of the proposed multi-modal approach over other baselines.
6.3 Comparison with State-of-the-art
Additionally, along with the baselines, we have compared the performance of our multi-modal approach with several existing works reported in the literature. For BioInfer dataset, we have compared our proposed method with nine state-of-the-art models. These existing methods are based on different techniques like kernel-based (Choi and Myaeng, 2010; Tikk et al., 2010; Qian and Zhou, 2012; Li et al., 2015), deep neural network-based (Zhao et al., 2016), multi-channel dependency-based convolutional neural network model (Peng and Lu, 2017), semantic feature embedding (Choi, 2018) and shortest dependency path (Hua and Quan, 2016). Along with the aforementioned methods, we have also compared our approach with a recent deep learning-based approach proposed by (Yadav et al., 2019). The comparative performance analysis for BioInfer dataset is tabulated in Table
Table 1: Comparative study of our proposed deep multi-modal approach with several baselines in terms of precision, recall,  $F$ -measure
Precision	Recall	F-score
Proposed Model	86.81	89.53	88.15
(Yadav et al., 2019)	80.81	82.57	81.68
(Hua and Quan, 2016)	73.40	77.00	75.20
(Choi, 2018)	72.05	77.51	74.68
(Qian and Zhou, 2012)	63.61	61.24	62.40
(Peng and Lu, 2017)	62.70	68.2	65.30
(Zhao et al., 2016)	53.90	72.9	61.60
(Tikk et al., 2010)	53.30	70.10	60.00
(Li et al., 2015)	72.33	74.94	73.61
(Choi and Myaeng, 2010)	74.50	70.90	72.60
2. We have also compared our approach with nine existing approaches for HRPD50 dataset. The comparative results for HRPD50 dataset are presented in Table 3.
By analyzing the above comparative study, we can infer that the overall performance of our proposed multi-modal approach surpasses other baselines and existing methods. Among the baseline models, proposed multi-modal approach outperforms its unimodal and bimodal counterparts. Among the uni-modal architecture, structural modality outperforms other two modalities which suggests the importance of structural modality over textual and sequence modalities. The sequence modality performs poorly because of its huge length (length of most of the sequences is approx 10,000 nucleotides).
Among the bimodal architectures, (textual + structural) model surpasses other bimodal and unimodal counterparts. This fusion shows improvements of  $5.1\%$  and  $5.5\%$  F-score values over the best unimodal architecture for HRPD50 and BioInfer data sets, respectively. Similarly, our proposed multi-modal architecture shows an improvement over bi-modal counterparts. Also, the proposed multi-modal architecture shows an aver
Table 2: Comparative analysis of the proposed multimodal approach with state-of-the-art techniques for BioInfer dataset.
Precision	Recall	F-score
Proposed Model	96.93	78.51	86.75
(Yadav et al., 2019)	79.92	77.58	78.73
(Tikk et al., 2010)	68.20	69.80	67.80
(Tikk et al., 2010)(with SVM)	68.20	69.80	67.80
(Palaga, 2009)	66.70	80.20	70.90
(Airola et al., 2008a)(APG)	64.30	65.80	63.40
(Van Landeghem et al., 2008)	60.00	51.00	55.00
(Miwa et al., 2009)	68.50	76.10	70.90
(Airola et al., 2008a)(Co-occ)	38.90	100	55.40
(Pyysalo et al., 2008)	76.00	64.00	69.00
Table 3: Comparative analysis of the proposed multimodal approach with other state-of-the-art approaches for HRPD50 dataset.
6403
age improvement of  $3.87\%$  and  $2.24\%$  F-scores over multi-modal approach1 and multi-modal approach2, respectively. This improvement indicates that in addition to multiple modalities, underlying deep learning models and fusion technique contribute significantly in improving the performance of the overall architecture.
In addition, Table 2 and Table 3 indicate that the proposed multi-modal architecture outperforms the best and recent existing methods for both BioInfer and HRPD50 dataset, respectively. We have performed Welch's t-test to show that obtained improvements by the proposed approach are statistically significant. From the above comparative study, it is evident that our proposed multi-modal approach identifies the protein interactions in an efficient way and can be further improved in different ways.
6.5 Error Analysis
After thoroughly analyzing false positive and false negative instances, it can be inferred that following are the possible reasons of errors:
1. The instances which contain huge number of protein entities lead to misclassification. The maximum number of proteins in an instance of HRPD50 and BioInfer are 26 and 24, respectively; this has a huge chance of misclassification. For example: "Mutations in Saccharomyces cerevisiae RFC5, DPB11, MEC1, DDC2, MEC3, PDS1, CHK1, PDS1, and DUN1 have increased the rate of genome rearrangements up to 200-fold whereas mutations in RAD9, RAD17, RAD24, BUB3, and MAD3 have little effect."
2. Repetitive mentions of the same protein entity adds noise that leads to loose contextual information. For example “Here we demonstrate ... CLIP-170 and LIS1 Overexpression of CLIP-170 results ... phospho-LIS1 ... that CLIP-170 and LIS1 regulate ... that LIS1 is a regulated adapter between CLIP-170 ... MT dynamics”.
3. For sequence modality, we consider underlying FASTA sequence of proteins. The length of the sequence varies from 100 to 10000 nucleotides. This increased protein length leads to misclassification as the deep learning-based model is unable to possess this long chain of nucleotides.
In this work, we have generated some multi-modal protein-protein interaction databases by amalgamating protein structures and sequences with existing text information available in the biomedical literature. The process of generating multi-modal datasets from PPI corpora is illustrated with some examples. Besides, we have proposed a novel deep multi-modal architecture for managing the multimodal scenario for PPIs. For each modality (textual, protein sequence and protein atomic structure), we have developed different deep learning models for efficient feature extractions. A detailed comparative analysis proves that the proposed multi-modal architecture outperforms other strong baselines and existing models. Future work aims at enhancing sequence feature extraction methods to improve the classification performance as those suffer from low accuracy. Further there are plenty of options for improving the fusion technique to enhance the overall performance of the model.
Pratik Dutta acknowledges Visvesvaraya PhD Scheme for Electronics and IT, an initiative of Ministry of Electronics and Information Technology (MeitY), Government of India for fellowship support. Dr. Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya PhD scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia) for carrying out this research.
Antti Airola, Sampo Pyysalo, Jari Björne, Tapio Pahikkala, Filip Ginter, and Tapio Salakoski. 2008a. All-paths graph kernel for protein-protein interaction extraction with evaluation of cross-corpus learning. BMC bioinformatics, 9(11):S2.
Antti Airola, Sampo Pyysalo, Jari Björne, Tapio Pahikkala, Filip Ginter, and Tapio Salakoski. 2008b. A graph kernel for protein-protein interaction extraction. In Proceedings of the workshop on current trends in biomedical natural language processing, pages 1-9. Association for Computational Linguistics.
Ilseyar Alimova and Elena Tutubalina. 2019. Detecting adverse drug reactions from biomedical texts with
6404
neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 415-421.
Babak Alipanahi, Andrew Delong, Matthew T Weirauch, and Brendan J Frey. 2015. Predicting the sequence specificities of dna-and rna-binding proteins by deep learning. Nature biotechnology, 33(8):831.
Takayuki Amemiya, M Michael Gromiha, Katsuhisa Horimoto, and Kazuhiko Fukui. 2019. Drug repositioning for dengue haemorrhagic fever by integrating multiple omics analyses. Scientific reports, 9(1):523.
Masaki Asada, Makoto Miwa, and Yutaka Sasaki. 2018. Enhancing drug-drug interaction extraction from texts by molecular structure information. arXiv preprint arXiv:1805.05593.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. 2007. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pages 153-160.
Christian Blaschke, Miguel A Andrade, Christos A Ouzounis, and Alfonso Valencia. 1999. Automatic extraction of biological information from scientific text: protein-protein interactions. In Ismb, volume 7, pages 60-67.
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. 2017. Scacnn: Spatial and channel-wise attention in convolutional networks for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5659-5667.
Sheng-Yeh Chen, Chao-Chun Hsu, Chuan-Chun Kuo, Lun-Wei Ku, et al. 2018. Emotionlines: An emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Sung-Pil Choi. 2018. Extraction of protein-protein interactions (ppis) from the literature by deep convolutional neural networks with various feature embeddings. Journal of Information Science, 44(1):60-73.
Sung-Pil Choi and Sung-Hyon Myaeng. 2010. Simplicity is better: revisiting single kernel ppi extraction. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 206-214. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Pratik Dutta and Sriparna Saha. 2017. Fusion of expression values and protein interaction information using multi-objective optimization for improving gene clustering. Computers in biology and medicine, 89:31-43.
Pratik Dutta, Sriparna Saha, Saraansh Chopra, and Varnika Miglani. 2019a. Ensembling of gene clusters utilizing deep learning and protein-protein interaction information. IEEE/ACM transactions on computational biology and bioinformatics.
Pratik Dutta, Sriparna Saha, and Saurabh Gulati. 2019b. Graph-based hub gene selection technique using protein interaction information: Application to sample classification. *IEEE journal of biomedical and health informatics*, 23(6):2670-2676.
Pratik Dutta, Sriparna Saha, Sanket Pai, and Aviral Kumar. 2020. A protein interaction information-based generative model for enhancing gene clustering. Scientific Reports (Nature Publisher Group), 10(1).
Gunes Erkan, Arzucan Ozgur, and Dragomir R Radev. 2007. Semi-supervised classification for extracting protein interaction sentences using dependency parsing. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. 2019. Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1999-2007.
Claudio Giuliano, Alberto Lavelli, and Lorenzo Romano. 2006. Exploiting shallow linguistic information for relation extraction from biomedical literature. In 11th Conference of the European Chapter of the Association for Computational Linguistics.
Alexander Goncearenco, Minghui Li, Franco L Simonetti, Benjamin A Shoemaker, and Anna R Panchenko. 2017. Exploring protein-protein interactions as drug targets for anti-cancer therapy with in silico workflows. In *Proteomics for Drug Discovery*, pages 221-236. Springer.
Yu-Lun Hsieh, Yung-Chun Chang, Nai-Wen Chang, and Wen-Lian Hsu. 2017. Identifying protein-protein interactions in biomedical literature using recurrent neural networks with long short-term memory. In Proceedings of the eighth international joint conference on natural language processing (volume 2: short papers), pages 240-245.
Lei Hua and Chanqin Quan. 2016. A shortest dependency path based convolutional neural network for protein-protein relation extraction. *BioMed research international*, 2016.
6405
Minlie Huang, Xiaoyan Zhu, Yu Hao, Donald G Payan, Kunbin Qu, and Ming Li. 2004. Discovering patterns to extract protein-protein interactions from full texts. Bioinformatics, 20(18):3604-3612.
Mengqi Jin, Mohammad Taha Bahadori, Aaron Colak, Parminder Bhatia, Busra Celikkaya, Ram Bhakta, Selvan Senthivel, Mohammed Khalilia, Daniel Navarro, Borui Zhang, et al. 2018. Improving hospital mortality prediction with medical named entities and multimodal learning. arXiv preprint arXiv:1811.12276.
Ritu Khare, Robert Leaman, and Zhiyong Lu. 2014. Accessing biomedical literature in the current information landscape. In Biomedical Literature Mining, pages 11-31. Springer.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Maxat Kulmanov, Mohammed Asif Khan, and Robert Hoehndorf. 2017. Deepgo: predicting protein functions from sequence and interactions using a deep ontology-aware classifier. Bioinformatics, 34(4):660-668.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, 521(7553):436.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics.
Lishuang Li, Rui Guo, Zhenchao Jiang, and Degen Huang. 2015. An approach to improve kernel-based protein-protein interaction extraction by learning from large-scale network data. Methods, 83:44-50.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Makoto Miwa, Rune Sætre, Yusuke Miyao, and Jun'ichi Tsujii. 2009. Protein-protein interaction extraction by leveraging multiple kernels and parsers. International journal of medical informatics, 78(12):e39-e46.
Toshihide Ono, Haretsugu Hishigaki, Akira Tanigami, and Toshihisa Takagi. 2001. Automated extraction of information on protein-protein interactions from the biological literature. Bioinformatics, 17(2):155-161.
Peter Palaga. 2009. Extracting relations from biomedical texts using syntactic information. Mémoire de DEA, Technische Universität Berlin, 138.
Peggy L Peissig, Luke V Rasmussen, Richard L Berg, James G Linneman, Catherine A McCarty, Carol Waudby, Lin Chen, Joshua C Denny, Russell A Wilke, Jyotishman Pathak, et al. 2012. Importance of multi-modal approaches to effectively identify cataract cases from electronic health records. Journal of the American Medical Informatics Association, 19(2):225-234.
Yifan Peng and Zhiyong Lu. 2017. Deep learning for extracting protein-protein interactions from biomedical literature. arXiv preprint arXiv:1706.01556.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508.
Sampo Pyysalo, Antti Airola, Juho Heimonen, Jari Björne, Filip Ginter, and Tapio Salakoski. 2008. Comparative analysis of five protein-protein interaction corpora. In BMC bioinformatics, volume 9, page S6. BioMed Central.
Longhua Qian and Guodong Zhou. 2012. Tree kernel-based protein-protein interaction extraction from biomedical literature. Journal of biomedical informatics, 45(3):535-543.
Zhi Qiao, Xian Wu, Shen Ge, and Wei Fan. 2019. Mnn: multimodal attentional neural networks for diagnosis prediction. Extraction, 1:A1.
Syed Arbaaz Qureshi, Gael Dias, Mohammed Hasanuzzaman, and Sriparna Saha. 2020. Improving depression level estimation by concurrently learning emotion intensity. IEEE Computational Intelligence Magazine.
Syed Arbaaz Qureshi, Sriparna Saha, Mohammed Hasanuzzaman, and Gael Dias. 2019. Multitask representation learning for multimodal estimation of depression level. IEEE Intelligent Systems, 34(5):45-52.
Bisakha Ray, Mikael Henaff, Sisi Ma, Efstratios Efstathiadis, Eric R Peskin, Marco Picone, Tito Poli, Constantin F Aliferis, and Alexander Statnikov. 2014. Information content and analysis methods for multi-modal high-throughput biomedical data. Scientific reports, 4:4411.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In Advances in neural information processing systems, pages 3856-3866.
Rune Sætre, Kenji Sagae, and Jun'ichi Tsujii. 2007. Syntactic features for protein-protein interaction extraction. LBM (Short Papers), 319.
Shweta, A. Ekbal, S. Saha, and P. Bhattacharyya. 2016. A deep learning architecture for protein-protein interaction article identification. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 3128-3133.
6406
Benjamin J Stapley and Gerry Benoit. 1999. Biobibliometrics: information retrieval and visualization from co-occurrences of gene names in medline abstracts. In Biocomputing 2000, pages 529-540. World Scientific.
Dongdong Sun, Minghui Wang, and Ao Li. 2019. A multimodal deep neural network for human breast cancer prognosis prediction by integrating multidimensional data. IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB), 16(3):841-850.
Domonkos Tikk, Philippe Thomas, Peter Palaga, Jorg Hakenberg, and Ulf Leser. 2010. A comprehensive benchmark of kernel methods to extract protein-protein interactions from literature. PLoS computational biology, 6(7):e1000837.
Sofie Van Landeghem, Yvan Saeys, Bernard De Baets, and Yves Van de Peer. 2008. Extracting protein-protein interactions from text using rich feature vectors and feature selection. In 3rd International symposium on Semantic Mining in Biomedicine (SMBM 2008), pages 77-84. Turku Centre for Computer Sciences (TUCS).
Shweta Yadav, Asif Ekbal, Sriparna Saha, Ankit Kumar, and Pushpak Bhattacharyya. 2019. Feature assisted stacked attentive shortest dependency path based bi-lstm model for protein-protein interaction. Knowledge-Based Systems, 166:18-29.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259.
Rafael Zamora-Resendiz and Silvia Crivelli. 2019. Structural learning of proteins using graph convolutional neural networks. *bioRxiv*, page 610444.
Shifeng Zhang, Xiaobo Wang, Ajian Liu, Chenxu Zhao, Jun Wan, Sergio Escalera, Hailin Shi, Zezheng Wang, and Stan Z Li. 2019. A dataset and benchmark for large-scale multi-modal face antispoofing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 919-928.
Zhehuan Zhao, Zhihao Yang, Hongfei Lin, Jian Wang, and Song Gao. 2016. A protein-protein interaction extraction approach based on deep neural network. International Journal of Data Mining and Bioinformatics, 15(2):145-164.
6407