SlowGuess/NAACL_2021_OCR huggingface dataset:
Andrew Moore*
and Communications,
a.moore@lancaster.ac.uk
Jeremy Barnes
jeremycb@ifi.uio.no
The majority of work in targeted sentiment analysis has concentrated on finding better methods to improve the overall results. Within this paper we show that these models are not robust to linguistic phenomena, specifically negation and speculation. In this paper, we propose a multi-task learning method to incorporate information from syntactic and semantic auxiliary tasks, including negation and speculation scope detection, to create English-language models that are more robust to these phenomena. Further we create two challenge datasets to evaluate model performance on negated and speculative samples. We find that multi-task models and transfer learning via language modelling can improve performance on these challenge datasets, but the overall performances indicate that there is still much room for improvement. We release both the datasets and the source code at https://github.com/ jerbarnes/multitask_negation_ for_targeted_sentiment.
Targeted sentiment analysis (TSA) involves jointly predicting entities which are the targets of an opinion, as well as the polarity expressed towards them (Mitchell et al., 2013). The TSA task, which is part of the larger set of fine-grained sentiment analysis tasks, can enable companies to provide better recommendations (Bauman et al., 2017), as well as give digital humanities scholars a quantitative approach to identifying how sentiment and emotions develop in literature (Alm et al., 2005; Kim and Klinger, 2019).
Modelling TSA has moved from sequence labeling using conditional random fields (CRFs) (Mitchell et al., 2013) or Recurrent Neural Networks (RNN) (Zhang et al., 2015a; Katiyar and Cardie, 2016; Ma et al., 2018), to Transformer
models (Hu et al., 2019). However, all these improvements have concentrated on making the best of the relatively small task-specific datasets. As annotation for fine-grained sentiment is difficult and often has low inter-annotator agreement (Wiebe et al., 2005; Øvrelid et al., 2020), this data tends to be small and of varying quality. This lack of high-quality training data prevents TSA models from learning complex, compositional linguistic phenomena. For sentence-level sentiment classification, incorporating compositional information from relatively small amounts of negation or speculation data improves both robustness and general performance (Council et al., 2010; Cruz et al., 2016; Barnes et al., 2020). Furthermore, transfer learning via language-modelling also improves fine-grained sentiment analysis (Hu et al., 2019; Li et al., 2019b). In this paper, we wish to explore two research questions:
1. Does multi-task learning of negation and speculation lead to more robust targeted sentiment models?
2. Does transfer learning based on language-modelling already incorporate this information in a way that is useful for targeted sentiment models?
We explore a multi-task learning (MTL) approach to incorporate auxiliary task information in targeted sentiment classifiers in English in order to investigate the effects of negation and speculation in detail, we also annotate two new challenge datasets which contain negated and speculative examples. We find that the performance is negatively affected by negation and speculation, but MTL and transfer learning (TL) models are more robust than single task learning (STL). TL reduces the improvements of MTL, suggesting that TL is similarly effective at learning negation and speculation. The overall performance on the challenge datasets,
*The authors contributed equally.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2838-2869 June 6-11, 2021. ©2021 Association for Computational Linguistics
however, confirms that there is still room for improvement.
The contributions of the paper are the following: i) we introduce two English challenge datasets annotated for negation and speculation, ii) we propose a multi-task model to incorporate negation and speculation information and evaluate it across four English datasets, iii) Finally, using the challenge datasets, we show the quantitative effect of negation and speculation on TSA.
Fine-grained sentiment analysis is a complex task which can be broken into four subtasks (Liu, 2015): i) opinion holder extraction, ii) opinion target extraction, iii) opinion expression extraction, iv) and resolving the polarity relationship between the holder, target, and expression. From these four subtasks, targeted sentiment analysis (TSA) (Jin and Ho, 2009; Chen et al., 2012; Mitchell et al., 2013) reduces the fine-grained task to only the second and final subtasks, namely extracting the opinion target and the polarity towards it.
English TSA datasets include MPQA (Wiebe et al., 2005), the SemEval Laptop and Restaurant reviews (Pontiki et al., 2014, 2016), and Twitter datasets (Mitchell et al., 2013; Wang et al., 2017). Further annotation projects have led to review datasets for Arabic, Dutch, French, Russian, and Spanish (Pontiki et al., 2016) and Twitter datasets for Spanish (Mitchell et al., 2013) and Turkish (Pontiki et al., 2016). Prior work has also explored the effects of different phenomena on TSA through error analysis and challenge datasets. Wang et al. (2017), Xue and Li (2018), and Jiang et al. (2019) showed the difficulties of polarity classification of targets on texts with multiple different polarities through the distinct sentiment error splits, the hard split, and the MAMS challenge dataset respectively. Both Kaushik et al. (2020) and Gardner et al. (2020) augment document sentiment datasets by asking annotators to create counterfactual examples for the IMDB dataset. More recently, Ribeiro et al. (2020) showed how sentence-level sentiment models are affected by various linguistic phenomena including negation, semantic role labelling, temporal changes, and name entity recognition. Previous approaches to modelling TSA have often relied on general sequence labelling models, e. g. CRFs (Mitchell et al., 2013), probabilistic graphical models (Klinger and Cimiano, 2013), RNNs (Zhang
et al., 2015b; Ma et al., 2018), and more recently pretrained Transformer models (Li et al., 2019b).
Multi-task and transfer learning The main idea of MTL (Caruana, 1993) is that a model which receives signal from two or more correlated tasks will more quickly develop a useful inductive bias, allowing it to generalize better. This approach has gained traction in NLP, where several benchmark datasets have been created (Wang et al., 2019b,a). Under some circumstances, MTL can also be seen as a kind of data augmentation, where a model takes advantage of extra training data available in an auxiliary task to improve the main task (Kshirsagar et al., 2015; Plank, 2016). Much of MTL uses hard parameter sharing (Caruana, 1993), which shares all parameters across some layers of a neural network. When the main task and auxiliary task are closely related, this approach can be an effective way to improve model performance (Collobert et al., 2011; Peng and Dredze, 2017; Martínez Alonso and Plank, 2017; Augenstein et al., 2018), although it is often preferable to make predictions for low-level auxiliary tasks at lower layers of a multi-layer MTL setup (Søgaard and Goldberg, 2016), which we refer to as hierarchical MTL.
Transfer learning methods (Mikolov et al., 2013; Peters et al., 2018a; Devlin et al., 2019) can leverage unlabeled data, but require training large models on large amounts of data. However, it seems even these models can be sensitive to negation (Ettinger, 2020; Ribeiro et al., 2020; Kassner and Schütze, 2020)
Specific to TSA, previous research has used MTL to incorporate document-level sentiment (He et al., 2019), or to jointly learn to extract opinion expressions (Li et al., 2019b; Chen and Qian, 2020).
Negation and Speculation Detection As negation is such a common linguistic phenomenon and one that has a direct impact on sentiment, previous work has shown that incorporating negation information is crucial for accurate sentiment prediction. Feature-based approaches did this by including features from negation detection modules (Das and Chen, 2007; Councill et al., 2010; Lapponi et al., 2012), while it has now become more common to assume that neural models learn negation features in an end-to-end fashion (Socher et al., 2013). However, recent research suggests that end-to-end
models are not able to robustly interpret the effect of negation on sentiment (Barnes et al., 2019), and that explicitly learning negation can improve sentiment results (Barnes, 2019; Barnes et al., 2020).
On the other hand, speculation refers to whether a statement is described as a fact, a possibility, or a counterfact (Saurí and Pustejovsky, 2009). Although there are fewer speculation annotated corpora available (Vincze et al., 2008; Kim et al., 2013; Konstantinova et al., 2012), including speculation information has shown promise for improving sentiment analysis at document-level (Cruz et al., 2016).
There has, however, been little research on how these phenomena specifically affect fine-grained approaches to sentiment analysis. This is important because, compared to document- or sentence-level tasks where there is often a certain redundancy in sentiment signal, for fine-grained tasks negation and speculation often completely change the sentiment (see Table 2), making their identification and integration within a fine-grained sentiment models essential to resolve.
We perform the main experiments on four English language datasets: The Laptop dataset from SemEval 2014 (Pontiki et al., 2014), the Restaurant dataset which combines the SemEval 2014 (Pontiki et al., 2014), 2015 (Pontiki et al., 2015), and 2016 (Pontiki et al., 2016), the Multi-aspect Multisentiment (MAMS) dataset (Jiang et al., 2019), and finally the Multi-perspective Question Answering (MPQA) dataset (Wiebe et al.,  $2005)^{1}$  shows the distribution of the sentiment classes. We take the pre-processed Laptop and Restaurant datasets from Li et al. (2019a), and use the train, dev, and test splits that they provide. We use the NLTK word tokenizer to tokenise the Laptop, Restaurant, and MPQA datasets and Spacy for the MAMS dataset.
We choose datasets that differ largely in their domain, size, and annotation style in order to determine if any trends we see are robust to these data characteristics or whether they are instead correlated. We convert all datasets to a targeted setup by extracting only the aspect targets and their polarity. We use the unified tagging scheme $^2$  following recent work (Li et al., 2019a,b) and convert all data
to BIOUL format<sup>3</sup> with unified sentiment tags, e.g.  $B-POS$  for a beginning tag with a positive sentiment, so that we can cast the TSA problem as a sequence labeling task.
The statistics for these datasets are shown in Table 1. MAMS has the largest number of training targets (11,162), followed by Restaurant (3,896), Laptop (2,044) and finally MPQA has the fewest (1,264). MPQA, however, has the longest average targets (6.3 tokens) compared to 1.3-1.5 for the other datasets. This derives from the fact that entire phrases are often targets in MPQA. Finally, due to the annotation criteria, the MAMS data also has the highest number of sentences with multiple aspects with multiple polarities – nearly  $100\%$  in train, compared to less than  $10\%$  for Restaurant.
3.1 Annotation for negation and speculation
Although negation and speculation are prevalent in the original data – negation and speculation occur in  $13 - 25\%$  and  $9 - 20\%$  of the sentences, respectively – it is difficult to pry apart improvement on the original data with improvement on these two phenomena. Therefore, we further annotate the dev and test set for the Laptop and Restaurant datasets<sup>4</sup>, and when possible<sup>5</sup>, insert negation and speculation cues into sentences lacking them, which we call LaptopNeg, LaptopSpec, RestaurantNeg, and RestaurantSpec. Inserting negation and speculation cues often leads to a change in polarity from the original annotation, as shown in the example in Table 2. We finally keep all sentences that contain a negation or speculation cue, including those that occur naturally in the data. As this process could introduce errors regarding the polarity expressed towards the targets, we doubly annotate the polarity for 50 sentences from the original dev data, the negated dev data, and the speculation dev data and calculate Cohen's Kappa scores. The statistics and inter-annotator agreement scores (IAA) are shown in Table 1<sup>6</sup>. The new annotations have similarly high IAA scores (0.66-0.70) to the original data
1All datasets contain the following three sentiment classes positive, neutral, and negative. The MPQA dataset also includes a fourth rare class, both. Table 7 of Appendix A. 2This is also known as collapsed tagging scheme (Hu et al., 2019)
$^3$ BIOUL format tags each token as either B: beginning token, I: inside token, O: outside token, U: unit (single token), or L: last token.
4For clarification this is the SemEval 2014 Laptop dataset and the 2014, 2015, and 2016 combined Restaurant dataset.
While inserting negation into new sentences is quite trivial, as one can always negate full clauses, e.g. It's good  $\rightarrow$  It's not true that it's good, adding speculation often requires rewording of the sentence. We did not include sentences that speculation made unnatural.
Table 7 of Appendix A shows the distribution of the sentiment classes.
Train	Dev	Test
sents.	targs.	len.	mult.	sents.	targs.	len.	mult.	sents.	targs.	len.	mult.	IAA
Laptop	2,741	2,044	1.5	136	304	256	1.5	18	800	634	1.6	38	0.67
LaptopNeg	-	-	-	-	147	181	1.5	41	403	470	1.6	79	0.70
LaptopSpec	-	-	-	-	110	142	1.4	10	208	220	1.5	19	0.64
Restaurant	3,490	3,896	1.4	312	387	414	1.4	34	2,158	2,288	1.4	136	0.71
RestaurantNeg	-	-	-	-	198	274	1.4	61	818	1,013	1.4	161	0.66
RestaurantSpec	-	-	-	-	138	200	1.3	35	400	451	1.4	49	0.66
MAMS	4,297	11,162	1.3	4,287	500	1,329	1.3	498	500	1,332	1.3	500	-
MPQA	4,195	1,264	6.3	94	1,389	400	5.4	29	1,620	365	6.7	22	-
Table 1: Statistics for the sentiment datasets used in the experiments. The table indicates the number of sentences in each split (sents.), the number of targets (targs.), the average length of the targets (len.), as well as how many sentences in each have multiple targets with differing polarity (mult.). IAA scores are reported on a subset of the data.
(0.67-0.71), confirming the quality of the annotations.
3.2 Auxiliary task data
For the multi-task learning experiments, we use six auxiliary tasks: negation scope detection using the Conan Doyle  $(\mathbf{NEG}_{CD})$  (Morante and Daelemans, 2012), both negation detection  $(\mathbf{NEG}_{SFU})$  and speculation detection (SPEC) on the  $\mathbf{SFU}_{NegSpec}$  dataset (Konstantinova et al., 2012), and Universal Part-of-Speech tagging (UPOS), Dependency Relation prediction (DR) and prediction of full lexical analysis (LEX) on the Streusle dataset (Schneider and Smith, 2015). We show the train, dev, test splits, as well as the number of labels, label entropy and label kurtosis (Martínez Alonso and Plank, 2017) in Table 3. An example sentence with auxiliary labels is shown in Appendix B. Although it may appear that the SFU dataset is an order of magnitude larger than the Conan Doyle dataset, in reality, most of the training sentences do not contain annotations, leaving similar sized data if these are filtered. Similar to the sentiment data, we convert the auxiliary tasks to BIO format and treat them as sequence labelling tasks.
We experiment with a single task baseline (STL) and a hierarchical multi-task model with a skipconnection (MTL), both of which are shown in Figure 1. For the STL model, we first embed a sentence and then pass the embeddings to a Bidirectional LSTM (Bi-LSTM). These features are then concatenated to the input embeddings and fed to the second Bi-LSTM layer, ending with the token-wise sentiment predictions from the CRF tagger. For
the MTL model, we additionally use the output of the first Bi-LSTM layer as features for the separate auxiliary task CRF tagger. As seen from Figure 1, the STL model and the MTL main task model use the same the green layers. The MTL additionally uses the pink layer for the auxiliary task, adding less than  $3.4\%$  trainable parameters<sup>7</sup> for all auxiliary tasks except LEX, which adds  $221.4\%$  due to the large label set (see Table 3). Furthermore, at inference time the MTL model is as efficient as STL, given that it only uses the green layers when predicting the targeted sentiment, of which this is empirically shown in Table 20 of Appendix F.
Figure 1: The overall architecture where the STL model contains all of the green layers and the MTL uses the additional pink auxiliary CRF tagger. The second Bi-LSTM has a skip connection from the embedding layer which concatenates the word embeddings with the output from the first Bi-LSTM.
Embeddings: For the embedding layer, we perform experiments using 300 dimensional GloVe
7The STL model had 1,785,967 parameters of which 364,042 were trainable as the embedding layer was frozen.
original	this is good, inexpensive sushi.
negated	this is not good, inexpensive sushi.
speculative	I'm not sure if this is good, inexpensive sushi.
Table 2: Example of how adding negation and speculation can change the polarity of a target (added tokens are shown in **bold**). While in the original, the target "sushi" has a positive polarity, in the negated example it is negative, and in the speculative example it is neutral.
train	dev	test	# labels	label entropy	label kurtosis
NEGCD	842	144	235	5	1.0	-0.8
NEGSEFU	13,712	1,713	1,703	5	0.2	0.2
SPEC	13,712	1,713	1,703	5	0.1	0.2
UPOS	2,723	554	535	17	2.5	-0.6
DR	2,723	554	535	49	3.1	1.3
LEX	2,723	554	535	570	3.9	75.7
Table 3: Statistics for the auxiliary datasets.
embeddings (Pennington et al., 2014), as well as TL from Transformer ELMo embeddings (Peters et al., 2018b) $^{8}$ . The GloVe embeddings are publicly available and trained on English Wikipedia and Gigaword data. For the MPQA dataset we use the Transformer ELMo from Peters et al. (2018b) $^{9}$  which was trained on the 1 billion word benchmark (Chelba et al., 2014). For the MAMS and Restaurant datasets we tuned a Transformer ELMo on 27 million (M) sentences from the 2019 Yelp review dataset $^{10}$ , and for the Laptop dataset on 28M sentences $^{11}$  from the Amazon electronics reviews dataset (McAuley et al., 2015) $^{12}$ . Training these models on large amounts of in-domain data gives superior performance to models trained on more generic data, e.g. BERT (Devlin et al., 2019). For all experiments we freeze the embedding layer in order to make the results between GloVe and TL more comparable with respect to the number of trainable parameters. For TL, we learn a summed weighting of all layers $^{13}$ , as this is more effective
This is a 6 layer transformer model with a bi-directional language model objective that contains 56 million parameters excluding the softmax. In comparison BERT uses a masked language modelling objective and contains 110 and 340 million parameters for the base and large versions (Devlin et al., 2019).
<sup>9</sup>Found at https://allennlp.org/elmo under Transformer ELMo.
10 https://www.yelp.com/dataset
11 More specifically there was 9M unique sentences and the model was trained for 3 epochs.
12For full details of on how the fine tuned Transformer ELMo models were trained see https://github.com/ apmoorel/language-model.
13For this Transformer ELMo it uses the output from the 6
than using the last layer (Peters et al., 2018a). For more details on the number of parameters used for each model see Table 19 in Appendix F.
Training: For the STL and the MTL models, we tune hyperparameters using AllenTune (Dodge et al., 2019) on the Laptop development dataset. We then use the best hyperparameters on the Laptop dataset for all the STL and MTL experiments, in order to reduce hyperparameter search. We follow the result checklist for hyperparameter searches from (Dodge et al., 2019) (details found in Tables 17 and 18 of Appendix E along with Figure 2 showing the expected validation scores from the hyperparameter tuning). For the MTL model, a single epoch involves training for one epoch on the auxiliary task and then an epoch on the main task, as previous work has shown training the lower-level task first improves overall results (Hashimoto et al., 2017). In this work, we assume all of the auxiliary training tasks are conceptually lower than TSA.
Evaluation: For all experiments, we run each model five times (Reimers and Gurevych, 2017) and report the mean and standard derivation. We also take the distribution of the five runs to perform significance testing (Reimers and Gurevych, 2018), eliminating the need for Bonferroni correction. Following Dror et al. (2018), we use the nonparametric Wilcoxon signed-rank test (Wilcoxon, 1945) for the  $F_{1}$  metrics and a more powerful parametric Welch's t-test (Welch, 1947) for the accu
transformer layers and the output from the non-contextualised character encoder, thus in total 7 layers are weighted and summed.
Laptop	MAMS	Restaurant	MPQA
Aux.	GloVe	TL	GloVe	TL	GloVe	TL	GloVe	TL
MTL	NEGCD	54.65(1.37)	62.89(1.18)	62.50(0.42)	65.17(0.35)	65.06(2.66)	71.04(1.13)	18.88(1.17)	22.25*(2.00)
DR	53.67(0.94)	62.29(1.32)	62.05*(0.32)	65.10(0.63)	66.06(2.63)	71.45(1.47)	17.03(1.12)	22.09*(0.70)
LEX	54.85(0.99)	62.55(1.66)	62.14*(0.83)	64.65*(0.88)	65.89(1.32)	71.77(1.88)	18.66(1.22)	22.74(1.68)
NEG\( _{SFU} \)	53.73(1.93)	62.61(1.79)	62.34*(0.54)	65.00*(0.48)	65.82(1.31)	71.63(1.64)	17.60(0.57)	22.30*(1.19)
SPEC	51.65(2.32)	62.03(1.14)	62.16(0.71)	64.50*(0.79)	65.16(1.50)	71.51(1.16)	16.70(2.26)	22.86*(0.98)
UPOS	54.17(2.26)	62.35(0.77)	62.79(0.37)	64.88(0.46)	65.73(1.46)	70.38(1.63)	18.70(0.25)	23.05*(0.88)
STL	54.37(2.56)	63.70(1.14)	63.20(0.65)	65.70(0.55)	65.60(1.06)	70.68(1.53)	18.11(2.83)	24.66(1.07)
Table 4: The  $F_{1}$ -i results for the test split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best performing model for that dataset and embedding. The * represent the models that perform statistically significantly worse than the STL model for that dataset and embedding at a 95% confidence level.
racy metric.
We report the  $F_{1}$  score for the target extraction  $(F_{1} - a)$ , macro  $F_{1}$ $(F_{1} - s)$  and accuracy score  $(acc - s)$  for the sentiment classification for all targets that have been correctly identified by the model, and finally the  $F_{1}$  score for the full targeted task  $(F_{1} - i)$ , following He et al. (2019). Unlike He et al. (2019), we do not use any of the samples that contain the conflict label on Laptop or Restaurant. The test results for the main  $F_{1} - i$  metric are reported in Table 4, and the other metrics for the test split are reported in Tables 9 and 10 of Appendix C.
The MTL models outperform STL on four of the eight experiments (see Table 4), although the STL TL model is significantly better than the majority of MTL models on MPQA. Of the MTL models,  $\mathrm{NEG}_{CD} + \mathrm{GloVe}$  performs best on MPQA (18.88),  $\mathrm{DR} + \mathrm{GloVe}$  is best on Restaurant (66.06), and LEX is the best model on Laptop (54.85) with GloVe and Restaurant (71.77) with TL. The TL models consistently outperform the GloVe models - by an average of 5.4 percentage points (pp) across all experiments - and give the best performance on all datasets.
The results suggest that transfer learning reduces the beneficial effects of MTL. At the same time, the results suggest that MTL does not hurt the STL models, as no STL model is significantly better than all of the MTL models across the datasets and
embeddings for the  $F_{1}$  -i metric. 14
5 Challenge Dataset Results