Negation Scope Resolution: Quantifying Neural Uncertainty In An Imbalanced Setting

Negation scope detection is an interesting task for neural machine learning models, because of the sequential dependencies in the input data. Having a neural classifier being able to untangle negated parts of a sentence from the non-negated part is useful for downstream tasks. Additionally, generally in classification tasks one has to work with quite imbalanced data sets. Within natural language only a subset of sentences contain negations – thus negation annotated data might be prone to imbalance in such a way that there are many annotated sentences without any negations (positive sentences) versus sentences with negations (negative sentences). This paper looks at how this kind of imbalance affects neural model performance by comparing models trained on the full data set, with models trained on a subset in which the positive sentences have been filtered out. The results evaluated on the *SEM 2012 shared task on negation scope detection show that there does seem to be a difference in how the classifiers are affected by imbalance, depending on architecture; and how including part-of-speech (PoS) features helps to reduce this difference. 1 The Task: Introduction & Data Within Negation Analysis the aim is to be able to automatically detect cues that lead to negation(s) within a sentence; scope resolution to relate the cue to its arguments; and event identification which is the key event of the scope. This can be useful in itself for example in triggering different services in web applications; but it can also be used as a pre-processing step for Sentiment Analysis (Lapponi et al., 2012). Negation Analysis can thus be separated into three sub-problems which can be solved in a sequential manner. In (1) the cue is shown in bold. It acts on the scope which has been underlined, and the key event of the scope is italicised (and by definition will also always be underlined). (1) Some people without possessing genius have a remarkable power of stimulating it. Language can however be complicated, and there are many cases where cues can act on cues, and scopes be within scopes. The sentence in (2) shows one such example. Which is quite difficult to even grasp as a human reader, but showcases the complexity of language. The subscripts correspond to cues which act on the scopes in braces with the corresponding number as superscript. The events are also annotated with the subscript in front of the word. Here we see how a word can be both a cue, be within multiple scopes, and be an event as well. (2) Mr. Sherlock Holmes, {who 2was} usually {very 2late in the mornings,}2 save2 {upon {those}0}1 not1 {0,1in0{frequent occasions when he was up all night}0}1}2, was seated at the breakfast table. The word infrequent in (2) displays all these properties, and we also see how the cue can be an affix of the word; e.g. the in in infrequent. Furthermore cue words can also be multi-token words, e.g. no more or neither . nor. This complexity is difficult to encode in a reasonable way for a machine learning system to take advantage of, and thus we will make some welcome simplifications to adapt the data for neural methods. As stated, Negation Analysis consists of three sub-tasks: (a) cue detection, (b) scope resolution and (c) event identification. The first simplification will be to only consider (b) given (a). The second simplification is to transform the data into a (sequential) binary classification task – which is to distribute the sentences with multiple cues (that have their own scopes) into separate sentences. We therefore get a copy of each sentence corresponding to how many cues it originally contains. To illustrate this, we can take (2) and transform it into (2a)-(2c). (2a) Mr. Sherlock Holmes, who was usually very late in the mornings, save upon those not infrequent occasions when he was up all night, was seated at the breakfast table. (2b) Mr. Sherlock Holmes, who was usually very late in the mornings, save upon those not infrequent occasions when he was up all night, was seated at the breakfast table. (2c) Mr. Sherlock Holmes, who was usually very late in the mornings, save upon those not infrequent occasions when he was up all night, was seated at the breakfast table. We have also removed the events in the latter three examples, because they are not part of our simplified variant of the task anyway. Lastly, since we are given (a), we will take the cue information and use it to solve the task. That means we will use the gold cue in the data set as features, together with the sentences (and later PoS tags). To sum up, the task for a neural classifier will be to output a vector of binary values of the same length as the input sentence. For each position in the vector the classifier must predict whether the word is within a scope or not. Input to the system will be the sentences themselves, and additionally some binary value saying whether the word is (part of) a cue or not. The details of this are explained in section 3. All this said, a model can not be built without data. The example sentences above have given a hint about what kind of data we are working with: the *SEM 2012 negation annotations over short stories by Sir Arthur Conan Doyle (Morante and Daelemans, 2012). Conveniently the data has been compiled into a training, development and evaluation (held-out) set which permits for reliable comparisons between systems. Table 1 gives an overview of the distribution. The second column denotes the number of unique sentences in each data set. The third one is the number of sentences when we have distributed the sentences as in examples (2a)-(2c). While the last column is the number of sentences Data set Unique Distributed Negative Train 3643 3779 983 Development 787 816 173 Evaluation 1089 1118 264 Table 1: Distribution of the Conan Doyle data set. Unique is the number of unique sentences. Distributed is the number of sentences when they have been multiplied out as described in examples (2a)-(2c). The Negative column describes the number of sentences when we filter out positive sentences from the distributed set. left when we filter out the positive sentences (i.e. sentences with no negations) after distributing sentences. Therefore the number of negative sentences comprise about 20-25% of the data, meaning we have quite an imbalanced data set. We will set out to explore how this affects the uncertainty of model performance in different settings. By uncertainty here we mean how the use of Distributed versus Negative data influences the final predictions of a neural classifier.