The Impact of Self-Interaction Attention on the Extraction of Drug-Drug Interactions
Authors: Luca Putelli, Alfonso Emilio Gerevini, Alberto Lavelli, Ivan Serina (Università di Brescia, Fondazione Bruno Kessler, 2019)
Topic: In-domain IR and IE

Abstract
Since a large amount of medical treatments requires the assumption of multiple drugs, the discovery of how these interact with each other, potentially causing health problems to the patients, is the subject of a huge quantity of documents. In order to obtain this information from free text, several methods involving deep learning have been proposed over the years. In this paper we introduce a Recurrent Neural Network-based method combined with the Self-Interaction Attention Mechanism. Such a method is applied to the DDI2013-Extraction task, a popular challenge concerning the extraction and classification of drug-drug interactions. Our focus is to show its effect over the tendency to predict the majority class and how it differs from the other types of attention mechanisms.

Introduction
Given the increase of publications regarding side effects, adverse drug reactions and, more in general, how the assumption of drugs can cause risks of health issues that may affect patients, a large quantity of free-text containing crucial information has become available. For doctors and researchers, accessing this information is a very demanding task, given the number and the complexity of such documents. Hence, the automatic extraction of Drug-Drug Interactions (DDI), i.e. situations where the simultaneous assumption of drugs can cause adverse drug reactions, is the goal of the DDIExtraction-2013 task (Segura-Bedmar et al., 2014). DDIs have to be extracted from a corpus of free-text sentences, combining machine learning with natural language processing (NLP). Starting from the introduction of word embedding techniques like Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) for word representation, Recurrent Neural Networks (RNN) and in particular Long Short Term Memory networks (LSTM) have become the state-of-the-art technology for most of natural language processing tasks like text classification or relation extraction. The main idea behind the attention mechanism (Bahdanau et al., 2014) is that the model “pays attention" only to the parts of the input where the most relevant information is present. In our case, this mechanism assigns a higher weight to the most influential words, i.e. the ones which describe an interaction between drugs. Several attention mechanisms have been proposed in the last few years (Hu, 2018), in particular self-interaction mechanism (Zheng et al., 2018) applies attention with a different weight vector for each word in the sequence, producing a matrix that represents the influence between all word pairs. We consider this information very meaningful, especially in a task like this one where we need to discover connections between pairs of words. In this paper we show how self-interaction attention improves the results in the DDI-2013 task, comparing it to other types of attention mechanisms. Given that this dataset is strongly unbalanced, the main focus of the analysis is how each attention mechanism deals with the tendency to predict the majority class.

Conclusion
We have compared the self-interaction attention model to alternative configurations using the standard attention mechanism introduced by Bahdanau et al. (2014) and the context-attention mechanism of Yang et al. (2016). Our experiments show that the self-interaction mechanism improves the performance with respect to other versions, in particular reducing the tendency of predicting the majority class, hence decreasing the number of false negatives. The standard attention mechanism produces better results than the context attention. As future work, our objective is to exploit or adapt the Transformer architecture (Vaswani et al., 2017), which has become quite popular for machine translation tasks and relies almost only on attention mechanisms, and apply it to relation extraction tasks like DDI-2013. Another direction includes the exploitation of a different pre-trained language modeling. For example, BioBERT (Lee et al., 2019) obtains good results for several NLP tasks like Named Entity Recognition or Question Answering and we plan to apply it to our task.