Venue: NAACL 2021

# Multi-task Learning of Negation and Speculation

for Targeted Sentiment Classification

Andrew Moore1

School of Computing

and Communications,

Lancaster University

a.moore@lancaster.ac.uk &Jeremy Barnes

University of Oslo

Department of Informatics

jeremycb@ifi.uio.no

Footnote 1: The authors contributed equally.

###### Abstract

The majority of work in targeted sentiment analysis has concentrated on finding better methods to improve the overall results. Within this paper we show that these models are not robust to linguistic phenomena, specifically negation and speculation. In this paper, we propose a multi-task learning method to incorporate information from syntactic and semantic auxiliary tasks, including negation and speculation scope detection, to create English-language models that are more robust to these phenomena. Further we create two challenge datasets to evaluate model performance on negated and speculative samples. We find that multi-task models and transfer learning via language modelling can improve performance on these challenge datasets, but the overall performances indicate that there is still much room for improvement. We release both the datasets and the source code at [https://github.com/jerbarnes/multitask_negation_for_targeted_sentiment](https://github.com/jerbarnes/multitask_negation_for_targeted_sentiment).

## 1 Introduction

Targeted sentiment analysis (TSA) involves jointly predicting entities which are the targets of an opinion, as well as the polarity expressed towards them Mitchell et al. (2013). The TSA task, which is part of the larger set of fine-grained sentiment analysis tasks, can enable companies to provide better recommendations Bauman et al. (2017), as well as give digital humanities scholars a quantitative approach to identifying how sentiment and emotions develop in literature Alm et al. (2005); Kim and Klinger (2019).

Modelling TSA has moved from sequence labeling using conditional random fields (CRFs) Mitchell et al. (2013) or Recurrent Neural Networks (RNN) Zhang et al. (2015); Katiyar and Cardie (2016); Ma et al. (2018), to Transformer models Hu et al. (2019). However, all these improvements have concentrated on making the best of the relatively small task-specific datasets. As annotation for fine-grained sentiment is difficult and often has low inter-annotator agreement Wiebe et al. (2005); Ovrelid et al. (2020), this data tends to be small and of varying quality. This lack of high-quality training data prevents TSA models from learning complex, compositional linguistic phenomena. For sentence-level sentiment classification, incorporating compositional information from relatively small amounts of negation or speculation data improves both robustness and general performance Councill et al. (2010); Cruz et al. (2016); Barnes et al. (2020). Furthermore, transfer learning via language-modelling also improves fine-grained sentiment analysis Hu et al. (2019); Li et al. (2019). In this paper, we wish to explore **two research questions**:

1. Does multi-task learning of negation and speculation lead to more robust targeted sentiment models?
2. Does transfer learning based on language-modelling already incorporate this information in a way that is useful for targeted sentiment models?

We explore a **multi-task learning (MTL)** approach to incorporate auxiliary task information in targeted sentiment classifiers in English in order to investigate the effects of negation and speculation in detail, we also annotate two new challenge datasets which contain negated and speculative examples. We find that the performance is negatively affected by negation and speculation, but MTL and **transfer learning (TL)** models are more robust than **single task learning (STL)**. TL reduces the improvements of MTL, suggesting that TL is similarly effective at learning negation and speculation. The overall performance on the challenge datasets,

however, confirms that there is still room for improvement.

The contributions of the paper are the following: i) we introduce two English challenge datasets annotated for negation and speculation, ii) we propose a multi-task model to incorporate negation and speculation information and evaluate it across four English datasets, iii) Finally, using the challenge datasets, we show the quantitative effect of negation and speculation on TSA.

## 2 Background and related work

Fine-grained sentiment analysisis a complex task which can be broken into four subtasks Liu (2015): i) opinion holder extraction, ii) opinion target extraction, iii) opinion expression extraction, iv) and resolving the polarity relationship between the holder, target, and expression. From these four subtasks, targeted sentiment analysis (TSA) Jin and Ho (2009); Chen et al. (2012); Mitchell et al. (2013) reduces the fine-grained task to only the second and final subtasks, namely extracting the opinion target and the polarity towards it.

English TSA datasets include MPQA Wiebe et al. (2005), the SemEval Laptop and Restaurant reviews Pontiki et al. (2014, 2016), and Twitter datasets Mitchell et al. (2013); Wang et al. (2017). Further annotation projects have led to review datasets for Arabic, Dutch, French, Russian, and Spanish Pontiki et al. (2016) and Twitter datasets for Spanish Mitchell et al. (2013) and Turkish Pontiki et al. (2016). Prior work has also explored the effects of different phenomena on TSA through error analysis and challenge datasets. Wang et al. (2017), Xue and Li (2018), and Jiang et al. (2019) showed the difficulties of polarity classification of targets on texts with multiple different polarities through the distinct sentiment error splits, the hard split, and the MAMS challenge dataset respectively. Both Kaushik et al. (2020) and Gardner et al. (2020) augment document sentiment datasets by asking annotators to create counterfactual examples for the IMDB dataset. More recently, Ribeiro et al. (2020) showed how sentence-level sentiment models are affected by various linguistic phenomena including negation, semantic role labelling, temporal changes, and name entity recognition. Previous approaches to modelling TSA have often relied on general sequence labelling models, _e. g._ CRFs Mitchell et al. (2013), probabilistic graphical models Klinger and Cimiano (2013), RNNs Zhang et al. (2015); Ma et al. (2018), and more recently pretrained Transformer models Li et al. (2019b).

Multi-task and transfer learningThe main idea of MTL Caruana (1993) is that a model which receives signal from two or more correlated tasks will more quickly develop a useful inductive bias, allowing it to generalize better. This approach has gained traction in NLP, where several benchmark datasets have been created Wang et al. (2019b,a). Under some circumstances, MTL can also be seen as a kind of data augmentation, where a model takes advantage of extra training data available in an auxiliary task to improve the main task Kshirsagar et al. (2015); Plank (2016). Much of MTL uses _hard parameter sharing_Caruana (1993), which shares all parameters across some layers of a neural network. When the main task and auxiliary task are closely related, this approach can be an effective way to improve model performance Collobert et al. (2011); Peng and Dredze (2017); Martinez Alonso and Plank (2017); Augenstein et al. (2018), although it is often preferable to make predictions for low-level auxiliary tasks at lower layers of a multi-layer MTL setup Sogaard and Goldberg (2016), which we refer to as _hierarchical MTL_.

Transfer learning methods Mikolov et al. (2013); Peters et al. (2018); Devlin et al. (2019) can leverage unlabeled data, but require training large models on large amounts of data. However, it seems even these models can be sensitive to negation Ettinger (2020); Ribeiro et al. (2020); Kassner and Schutze (2020)

Specific to TSA, previous research has used MTL to incorporate document-level sentiment He et al. (2019), or to jointly learn to extract opinion expressions Li et al. (2019b); Chen and Qian (2020).

Negation and Speculation DetectionAs negation is such a common linguistic phenomenon and one that has a direct impact on sentiment, previous work has shown that incorporating negation information is crucial for accurate sentiment prediction. Feature-based approaches did this by including features from negation detection modules Das and Chen (2007); Councill et al. (2010); Lapponi et al. (2012), while it has now become more common to assume that neural models learn negation features in an end-to-end fashion Socher et al. (2013). However, recent research suggests that end-to-end 

models are not able to robustly interpret the effect of negation on sentiment (Barnes et al., 2019), and that explicitly learning negation can improve sentiment results (Barnes, 2019; Barnes et al., 2020).

On the other hand, speculation refers to whether a statement is described as a fact, a possibility, or a counterfactual (Saurf and Pustejovsky, 2009). Although there are fewer speculation annotated corpora available (Vincze et al., 2008; Kim et al., 2013; Konstantinova et al., 2012), including speculation information has shown promise for improving sentiment analysis at document-level (Cruz et al., 2016).

There has, however, been little research on how these phenomena specifically affect fine-grained approaches to sentiment analysis. This is important because, compared to document- or sentence-level tasks where there is often a certain redundancy in sentiment signal, for fine-grained tasks negation and speculation often completely change the sentiment (see Table 2), making their identification and integration within a fine-grained sentiment models essential to resolve.

## 3 Data

We perform the main experiments on four English language datasets: The **Laptop** dataset from SemEval 2014 (Pontiki et al., 2014), the **Restaurant** dataset which combines the SemEval 2014 (Pontiki et al., 2014), 2015 (Pontiki et al., 2015), and 2016 (Pontiki et al., 2016), the Multi-aspect Multi-sentiment (**MAMS**) dataset (Jiang et al., 2019), and finally the Multi-perspective Question Answering (**MPQA**) dataset (Wiebe et al., 2005)1 shows the distribution of the sentiment classes . We take the pre-processed Laptop and Restaurant datasets from Li et al. (2019), and use the train, dev, and test splits that they provide. We use the NLTK word tokenizer to tokenise the Laptop, Restaurant, and MPQA datasets and Spacy for the MAMS dataset.

Footnote 1: All datasets contain the following three sentiment classes positive, neutral, and negative. The MPQA dataset also includes a fourth rare class, both. Table 7 of Appendix A.

We choose datasets that differ largely in their domain, size, and annotation style in order to determine if any trends we see are robust to these data characteristics or whether they are instead correlated. We convert all datasets to a targeted setup by extracting only the aspect targets and their polarity. We use the unified tagging scheme2 following recent work (Li et al., 2019, 2019) and convert all data to BIOUL format3 with unified sentiment tags, _e. g. B-POS_ for a beginning tag with a positive sentiment, so that we can cast the TSA problem as a sequence labeling task.

Footnote 2: This is also known as collapsed tagging scheme (Hu et al., 2019)

The statistics for these datasets are shown in Table 1. MAMS has the largest number of training targets (11,162), followed by Restaurant (3,896), Laptop (2,044) and finally MPQA has the fewest (1,264). MPQA, however, has the longest average targets (6.3 tokens) compared to 1.3-1.5 for the other datasets. This derives from the fact that entire phrases are often targets in MPQA. Finally, due to the annotation criteria, the MAMS data also has the highest number of sentences with multiple aspects with multiple polarities - nearly 100% in train, compared to less than 10% for Restaurant.

### Annotation for negation and speculation

Although negation and speculation are prevalent in the original data - negation and speculation occur in 13-25% and 9-20% of the sentences, respectively - it is difficult to pry apart improvement on the original data with improvement on these two phenomena. Therefore, we further annotate the dev and test set for the Laptop and Restaurant datasets4, and when possible5, insert negation and speculation cues into sentences lacking them, which we call Laptop\({}_{Neg}\), Laptop\({}_{Spec}\), Restaurant\({}_{Seg}\), and Restaurant\({}_{Spec}\). Inserting negation and speculation cues often leads to a change in polarity from the original annotation, as shown in the example in Table 2. We finally keep all sentences that contain a negation or speculation cue, including those that occur naturally in the data. As this process could introduce errors regarding the polarity expressed towards the targets, we doubly annotate the polarity for 50 sentences from the original dev data, the negated dev data, and the speculation dev data and calculate Cohen's Kappa scores. The statistics and inter-annotator agreement scores (IAA) are shown in Table 16. The new annotations have similarly high IAA scores (0.66-0.70) to the original data 

(0.67-0.71), confirming the quality of the annotations.

### Auxiliary task data

For the multi-task learning experiments, we use six auxiliary tasks: negation scope detection using the Conan Doyle (**NEG\({}_{CD}\)**) Morante and Daelemans (2012), both negation detection (**NEG\({}_{SFU}\)**) and speculation detection (**SPEC**) on the SFU\({}_{NegSpec}\) dataset Konstantinova et al. (2012), and Universal Part-of-Speech tagging (**UPOS**), Dependency Relation prediction (**DR**) and prediction of full lexical analysis (**LEX**) on the Streusle dataset Schneider and Smith (2015). We show the train, dev, test splits, as well as the number of labels, label entropy and label kurtosis Martinez Alonso and Plank (2017) in Table 3. An example sentence with auxiliary labels is shown in Appendix B. Although it may appear that the SFU dataset is an order of magnitude larger than the Conan Doyle dataset, in reality, most of the training sentences do not contain annotations, leaving similar sized data if these are filtered. Similar to the sentiment data, we convert the auxiliary tasks to BIO format and treat them as sequence labelling tasks.

## 4 Experiments

We experiment with a single task baseline (STL) and a hierarchical multi-task model with a skip-connection (MTL), both of which are shown in Figure 1. For the STL model, we first embed a sentence and then pass the embeddings to a Bidirectional LSTM (Bi-LSTM). These features are then concatenated to the input embeddings and fed to the second Bi-LSTM layer, ending with the token-wise sentiment predictions from the CRF tagger. For the MTL model, we additionally use the output of the first Bi-LSTM layer as features for the separate auxiliary task CRF tagger. As seen from Figure 1, the STL model and the MTL main task model use the same the green layers. The MTL additionally uses the pink layer for the auxiliary task, adding less than 3.4% trainable parameters7 for all auxiliary tasks except LEX, which adds 221.4% due to the large label set (see Table 3). Furthermore, at inference time the MTL model is as efficient as STL, given that it only uses the green layers when predicting the targeted sentiment, of which this is empirically shown in Table 20 of Appendix F.

Footnote 7: The STL model had 1,785,967 parameters of which 364,042 were trainable as the embedding layer was frozen.

**Embeddings:** For the embedding layer, we perform experiments using 300 dimensional **GloVe**

Figure 1: The overall architecture where the STL model contains all of the green layers and the MTL uses the additional pink auxiliary CRF tagger. The second Bi-LSTM has a skip connection from the embedding layer which concatenates the word embeddings with the output from the first Bi-LSTM.

 

embeddings (Pennington et al., 2014), as well as **TL** from Transformer ELMo embeddings (Peters et al., 2018b)8. The GloVe embeddings are publicly available and trained on English Wikipedia and Gigaword data. For the MPQA dataset we use the Transformer ELMo from Peters et al. (2018b)9 which was trained on the 1 billion word benchmark (Chelba et al., 2014). For the MAMS and Restaurant datasets we tuned a Transformer ELMo on 27 million (M) sentences from the 2019 Yelp review dataset10, and for the Laptop dataset on 28M sentences11 from the Amazon electronics reviews dataset (McAuley et al., 2015)12. Training these models on large amounts of in-domain data gives superior performance to models trained on more generic data, _e. g._ BERT (Devlin et al., 2019). For all experiments we freeze the embedding layer in order to make the results between GloVe and TL more comparable with respect to the number of trainable parameters. For TL, we learn a summed weighting of all layers13, as this is more effective than using the last layer (Peters et al., 2018a). For more details on the number of parameters used for each model see Table 19 in Appendix F.

Footnote 8: This is a 6 layer transformer model with a bi-directional language model objective that contains 56 million parameters excluding the softmax. In comparison BERT uses a masked language modelling objective and contains 110 and 340 million parameters for the base and large versions (Devlin et al., 2019).

Footnote 9: Found at [https://allennlp.org/elmo](https://allennlp.org/elmo) under Transformer ELMo.

Footnote 10: [https://www.yelp.com/dataset](https://www.yelp.com/dataset)

Footnote 11: More specifically there was 9M unique sentences and the model was trained for 3 epochs.

Footnote 12: For full details of on how the fine tuned Transformer ELMo models were trained sok[https://github.com/apnoerel/language-model](https://github.com/apnoerel/language-model).

**Training:** For the STL and the MTL models, we tune hyperparameters using _AllenTune_(Dodge et al., 2019) on the Laptop development dataset. We then use the best hyperparameters on the Laptop dataset for all the STL and MTL experiments, in order to reduce hyperparameter search. We follow the result checklist for hyperparameter searches from (Dodge et al., 2019) (details found in Tables 17 and 18 of Appendix E along with Figure 2 showing the expected validation scores from the hyperparameter tuning). For the MTL model, a single epoch involves training for one epoch on the auxiliary task and then an epoch on the main task, as previous work has shown training the lower-level task first improves overall results (Hashimoto et al., 2017). In this work, we assume all of the auxiliary training tasks are conceptually lower than TSA.

**Evaluation:** For all experiments, we run each model five times (Reimers and Gurevych, 2017) and report the mean and standard derivation. We also take the distribution of the five runs to perform significance testing (Reimers and Gurevych, 2018), eliminating the need for Bonferroni correction. Following Dror et al. (2018), we use the non-parametric Wilcoxon signed-rank test (Wilcoxon, 1945) for the \(F_{1}\) metrics and a more powerful parametric Welch's t-test (Welch, 1947) for the accu

\begin{table}
\begin{tabular}{l l r c c c c c} \hline \hline  & train & dev & test & \# labels & label entropy & label kurtosis \\ \cline{2-7} NEG\({}_{CD}\) & 842 & 144 & 235 & 5 & 1.0 & -0.8 \\ NEG\({}_{SFU}\) & 13,712 & 1,713 & 1,703 & 5 & 0.2 & 0.2 \\ SPEC & 13,712 & 1,713 & 1,703 & 5 & 0.1 & 0.2 \\ UPOS & 2,723 & 554 & 535 & 17 & 2.5 & -0.6 \\ DR & 2,723 & 554 & 535 & 49 & 3.1 & 1.3 \\ LEX & 2,723 & 554 & 535 & 570 & 3.9 & 75.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Example of how adding negation and speculation can change the polarity of a target (added tokens are shown in **bold**). While in the original, the target “sushi” has a positive polarity, in the negated example it is negative, and in the speculative example it is neutral.

\begin{table}
\begin{tabular}{c p{284.5pt}} \hline \hline original & this is good, inexpensive sushi. \\ negated & this is **not** good, inexpensive sushi. \\ speculative & **I’m not sure if** this is good, inexpensive sushi. \\ \hline \hline \end{tabular}
\end{table}
Table 3: Statistics for the auxiliary datasets.

 

racy metric.

### Results

We report the \(F_{1}\) score for the target extraction (\(F_{1}\)-\(a\)), macro \(F_{1}\) (\(F_{1}\)-\(s\)) and accuracy score (\(acc\)-\(s\)) for the sentiment classification for all targets that have been correctly identified by the model, and finally the \(F_{1}\) score for the full targeted task (\(F_{1}\)-\(i\)), following He et al. (2019). Unlike He et al. (2019), we do not use any of the samples that contain the _conflict_ label on Laptop or Restaurant. The test results for the main \(F_{1}\)-\(i\) metric are reported in Table 4, and the other metrics for the test split are reported in Tables 9 and 10 of Appendix C.

The MTL models outperform STL on four of the eight experiments (see Table 4), although the STL TL model is significantly better than the majority of MTL models on MPQA. Of the MTL models, NEG\({}_{CD}\) + GloVe performs best on MPQA (18.88), DR + GloVe is best on Restaurant (66.06), and LEX is the best model on Laptop (54.85) with GloVe and Restaurant (71.77) with TL. The TL models consistently outperform the GloVe models - by an average of 5.4 percentage points (pp) across all experiments - and give the best performance on all datasets.

The results suggest that transfer learning reduces the beneficial effects of MTL. At the same time, the results suggest that MTL does not hurt the STL models, as no STL model is significantly better than all of the MTL models across the datasets and embeddings for the \(F_{1}\)-\(i\) metric. 14

Footnote 14: These findings also generalise to the results on the development splits, shown in Tables 11 and 12 within Appendix C.

## 5 Challenge Dataset Results

In order to isolate the effects of negation and speculation on the results, we test all models trained on the original Laptop and Restaurant datasets on the Laptop\({}_{Neg}\), Restaurant\({}_{Neg}\), LaptoP\({}_{Spec}\), and Restaurant\({}_{Spec}\) test splits. Tables 5 and 6 show the results for negation and speculation, respectively. The results for the dev split and the \(F_{1}\)-\(s\) of the test split are shown in Appendix D.

Firstly, all models perform comparatively worse on the challenge datasets, dropping an average of 24 and 25 pp on \(F_{1}\)-\(i\) on the negation and speculation data, respectively. Nearly all of this drop comes from poorer classification (\(acc\)-\(s\), \(F_{1}\)-\(s\)), while target extraction (\(F_{1}\)-\(a\)) is relatively stable. This demonstrates the importance of resolving negation and speculation for TSA and the usefulness of the annotated data to determine these effects.

On Laptop\({}_{Neg}\) and Restaurant\({}_{Neg}\) incorporating negation auxiliary tasks gives an average improvement of 3.8 pp on the \(F_{1}\)-\(i\) metric when using GloVe embeddings. More specifically, MTL with negation improves the sentiment classification scores, but does not help extraction. This makes sense conceptually, as negation has little effect on whether or not a word is part of a sentiment target. Instead,

\begin{table}
\begin{tabular}{l l c r r r r r r r} \hline \hline  & \multicolumn{3}{c}{Laptop} & \multicolumn{3}{c}{MAMS} & \multicolumn{3}{c}{Restaurant} & \multicolumn{3}{c}{MPQA} \\ \cline{2-10}  & Aux. & GloVe & TL & GloVe & TL & GloVe & TL & GloVe & TL \\ \cline{2-10}  & NEG\({}_{CD}\) & 54.65 & 62.89 & 62.50 & 65.17 & 65.06 & 71.04 & **18.88** & 22.25* \\  & DR & 53.67 & 62.29 & 62.05* & 65.10 & (2.66) & (1.13) & (1.17) & (2.00) \\  & DEX & (0.94) & (1.32) & (0.32) & (0.63) & (2.63) & (1.47) & (1.12) & (0.70) \\ MTL & LEX & **54.85** & 62.55 & 62.14* & 64.65* & 65.89 & **71.77** & 18.66 & 22.74 \\  & (0.99) & (1.66) & (0.83) & (0.88) & (1.32) & (1.88) & (1.22) & (1.68) \\  & NEG\({}_{SFU}\) & 53.73 & 62.61 & 62.34* & 65.00* & 65.82 & 71.63 & 17.60 & 22.30* \\  & NEG\({}_{(1.93)}\) & (1.79) & (0.54) & (0.48) & (1.31) & (1.84) & (0.57) & (1.19) \\  & SPEC & 51.65 & 62.03 & 62.16 & 64.50* & 65.16 & 71.51 & 16.70 & 22.86* \\  & (2.32) & (1.14) & (0.71) & (0.79) & (1.50) & (1.16) & (2.26) & (0.98) \\  & UPOS & 54.17 & 62.35 & 62.79 & 64.88 & 65.73 & 70.38 & 18.70 & 23.05* \\  & (2.26) & (0.77) & (0.37) & (0.46) & (1.46) & (1.83) & (0.25) & (0.88) \\ STL & 54.37 & **63.70** & **63.20** & **65.70** & 65.60 & 70.68 & 18.11 & **24.66** \\  & (2.56) & (1.14) & (0.65) & (0.55) & (1.06) & (1.53) & (2.83) & (1.07) \\ \hline \hline \end{tabular}
\end{table}
Table 4: The \(F_{1}\)-\(i\) results for the test split, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best performing model for that dataset and embedding. The * represent the models that perform statistically significantly worse than the STL model for that dataset and embedding at a 95% confidence level.

 

jointly learning dependency relations (DR) and full lexical analysis (LEX) improve extraction results. Furthermore, when using TL instead of GloVe embeddings, the best MTL model (NEG\({}_{SFU}\)) does marginally beat the STL TL equivalent on average, indicating that multi-task learning is still able to contribute something to transfer learning.

On Laptop\({}_{Spec}\) and Restaurant\({}_{Spec}\) MTL models improve results when using GloVe embeddings, with the additional speculation (SPEC) and dependency relation (DR) data improving the _F\({}_{1}\)-i_ metric by 0.5 pp and 0.49 pp respectively on average. However, with TL, MTL only leads to benefits on the Restaurant dataset. Unlike the negation data results, the speculation results appear to be helped more by syntactic auxiliary tasks like DR than semantic tasks like NEG\({}_{CD}\) and to some extent NEG\({}_{SFU}\).

The best MTL GloVe models on the original datasets (LEX16 and DR, respectively) also outperform the STL GloVe models on the challenge data, indicating that MTL leads to greater robustness. When comparing the STL model using GloVe and TL on average the model improves by 9.55 pp on the negation dataset compared to 3.65 pp for the speculation suggesting that transfer learning is less effective for speculation.

Footnote 16: The development _F\({}_{1}\)-i_ result for LEqTop\({}_{Neg}\) and Restaurant\({}_{Neg}\) test split, where the values represent the mean (standard deviation) of five runs with a different random seeds. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The * represents the models that are significantly worse (p \(<0.05\)) than the best performing model on the respective dataset, metric, and embedding.

## 6 Conclusion

In this paper, we have compared the effects of MTL using various auxiliary tasks for TSA and have created a negation and speculation annotated challenge dataset17 for TSA in order to isolate the effects of MTL. We show that TSA methods are drastically affected by negation and speculation effects in the data. These effects can be similarly reduced by either incorporating auxiliary task information into the model through MTL or through transfer learning. Additionally, MTL of negation

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & & & NEG\({}_{CD}\) & DR & Lex & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\ \multirow{8}{*}{\begin{tabular}{c} Sentiment \\ \end{tabular} } & \multirow{2}{*}{GloVe} & 42.80 & 38.54\({}^{\star}\) & 38.72\({}^{\star}\) & 45.26 & 41.23\({}^{\star}\) & 38.92\({}^{\star}\) & 38.32\({}^{\star}\) \\  & & & (2.48) & (0.98) & (3.00) & (1.45) & (2.90) & (1.74) & (1.73) \\  & & \multicolumn{1}{c}{TL} & **38.49** & 45.90 & 45.93 & 47.04 & 45.71 & 46.29 & 46.50 \\  & & \multicolumn{1}{c}{(2.32)} & \multicolumn{1}{c}{(3.54)} & \multicolumn{1}{c}{(2.93)} & \multicolumn{1}{c}{(2.19)} & \multicolumn{1}{c}{(2.03)} & \multicolumn{1}{c}{(3.30)} \\ \cline{4-11}  & & & (2.32) & \multicolumn{1}{c}{(3.54)} & \multicolumn{1}{c}{(2.03)} & \multicolumn{1}{c}{(3.72)} & \multicolumn{1}{c}{(3.72)} & \multicolumn{1}{c}{(3.72)} & \multicolumn{1}{c}{(3.72)} & \multicolumn{1}{c}{(3.72)} \\ \cline{4-11}  & & \multirow{2}{*}{extraction} & \multirow{2}{*}{GloVe} & 75.36\({}^{\star}\) & 76.05\({}^{\star}\) & 78.68 & 75.04\({}^{\star}\) & 76.14 & 77.98 & 76.52\({}^{\star}\) \\  & & & (0.91) & (1.20) & (0.97) & (1.92) & (2.06) & (1.41) & (1.24) \\  & & \multicolumn{1}{c}{TL} & 82.39 & 82.95 & 83.47 & 83.25 & 82.24\({}^{\star}\) & 82.58 & 82.10 \\  & & \multicolumn{1}{c}{(1.34)} & \multicolumn{1}{c}{(1.36)} & \multicolumn{1}{c}{(1.26)} & \multicolumn{1}{c}{(1.80)} & \multicolumn{1}{c}{(1.39)} & \multicolumn{1}{c}{(1.58)} & \multicolumn{1}{c}{(1.11)} \\ \cline{4-11}  & & \multirow{2}{*}{targeted} & \multirow{2}{*}{GloVe} & 32.28 & 29.30\({}^{\star}\) & 30.47\({}^{\star}\) & 33.96 & 31.36\({}^{\star}\) & 30.36\({}^{\star}\) & 29.33\({}^{\star}\) \\  & & & (2.23) & (0.54) & (2.45) & (1.30) & (1.78) & (1.56) & (1.47) \\  & & \multicolumn{1}{c}{TL} & **39.95** & 38.08 & 83.35 & 39.18 & 37.59 & 38.23 & 38.14 \\  & & \multicolumn{1}{c}{(2.02)} & \multicolumn{1}{c}{(3.13)} & \multicolumn{1}{c}{(2.01)} & \multicolumn{1}{c}{(2.88)} & \multicolumn{1}{c}{(1.99)} & \multicolumn{1}{c}{(1.89)} & \multicolumn{1}{c}{(2.23)} & \multicolumn{1}{c}{(2.23)} \\ \hline \multirow{8}{*}{
\begin{tabular}{c} Sentiment \\ \end{tabular} } & \multirow{2}{*}{Sentiment} & GloVe & 53.41 & 49.78\({}^{\star}\) & 47.69\({}^{\star}\) & 56.01 & 48.86\({}^{\star}\) & 50.58\({}^{\star}\) & 49.86\({}^{\star}\) \\  & & \multicolumn{1}{c}{(4.28)} & \multicolumn{1}{c}{(2.10)} & \multicolumn{1}{c}{(1.19)} & \multicolumn{1}{c}{(1.07)} & \multicolumn{1}{c}{(3.94)} & \multicolumn{1}{c}{(2.18)} & \multicolumn{1}{c}{(1.77)} \\  & & \multicolumn{1}{c}{TL} & **60.69** & **62.61** & **60.80** & **60.45** & **61.70** & **60.06** & **60.66** \\  & & \multicolumn{1}{c}{(1.91)} & \multicolumn{1}{c}{(2.11)} & \multicolumn{1}{c}{(3.20)} & \multicolumn{1}{c}{(2.04)} & \multicolumn{1}{c}{(1.42)} & \multicolumn{1}{c}{(2.13)} & \multicolumn{1}{c}{(2.24)} \\ \cline{4-11}  & & \multicolumn{1}{c}{extraction} & GloVe & 80.97 & **82.22** & **82.15** & **80.74** & **81.53** & **81.92** & **80.97** \\  & & \multicolumn{1}{c}{TL} & 83.04 & 82.94\({}^{\star}\) & **84.10** & **83.94** & **83.48** & **82.33\({}^{\star}\)** & **83.50** \\  & & \multicolumn{1}{c}{(1.26)} & \multicolumn{1}{c}{(0.97)} & \multicolumn{1}{c}{(0.86)} & \multicolumn{1}{c}{(1.67)} & \multicolumn{1}{c}{(1.59)} & \multicolumn{1}{c}{(1.37)} & \multicolumn{1}{c}{(1.16)} \\ \cline{4-11}  & & \multirow{2}{*}{targeted} & GloVe & 43.28 & 40.95\({}^{\star}\) & 39.19\({}^{\star}\) & **45.22** & **39.85\({}^{\star}\)** & **41.43\({}^{\star}\)** & **40.38\({}^{\star}\)** \\  & & \multicolumn{1}{c}{TL} & 50.40 & **51.92** & **51.15** & **50.75** & **51.49** & **49.45** & **50.68** \\  & & \multicolumn{1}{c}{(2.03)} & \multicolumn{1}{c}{(1.64)} & \multicolumn{1}{c}{(3.04)} & \multicolumn{1}{c}{(2.10)} & 

can lead to small improvements when combined with transfer learning. Returning to the two original research questions, we can conclude that in general 1) MTL using negation (speculation) as an auxiliary task does make TSA models more robust to negated (speculative) samples and 2) transfer learning seems to incorporate much of the same knowledge. Additionally, incorporating syntactic information as an auxiliary task within MTL creates models that are more robust to both negation and speculation.

Neither MTL nor TL are currently guarantees for improved performance17. Additionally, the results from the challenge datasets indicate that different auxiliary tasks improve the performance of different subtasks of TSA. This may suggest that the target extraction and sentiment classification tasks should not be treated as a collapsed labelling task, as the sentiment and extraction tasks are too dissimilar [Hu et al.2019]. Future work should consider using pipeline or joint approaches, where each sub-task can be paired with the most beneficial auxiliary tasks. This decoupling could also allow MTL and transfer learning to compliment each other more.

Footnote 17: Compare the performance of LEX using GloVe (28.59) to when it uses TL (28.56) in Table 6 for the Laptop dataset.

Finally, in order to improve reproducibility and to encourage further work, we release the code18, dataset, and trained models associated with this paper, hyperparameter search details with compute infrastructure (Appendix E), number of parameters and runtime details (Appendix F), and further detailed dev and test results (appendices C and D), in line with the result checklist from dodge2019.

Footnote 18: [https://github.com/jerbarnes/multitask_negation_for_targeted_sentiment](https://github.com/jerbarnes/multitask_negation_for_targeted_sentiment)

## Acknowledgements

This work has been carried out as part of the SANT project (Sentiment Analysis for Norwegian Text), funded by the Research Council of Norway (grant number 270908). Andrew has been funded by Lan

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\ \cline{4-13}  & \multirow{2}{*}{sentiment} & \multirow{2}{*}{GloVe} & 34.32 & 35.67 & **36.75** & 35.98 & 36.74 & **35.57** & 34.67 \\  & & & (1.86) & (1.00) & (1.91) & (2.05) & (1.64) & (1.31) & (1.40) \\  & & & 35.42 & 34.76 & 35.06 & 34.08\({}^{\star}\) & 35.03 & 35.01 & **35.97** \\  & & & (3.54) & (1.63) & (1.97) & (0.40) & (2.36) & (1.04) & (1.45) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{targeted}}}} & \multirow{2}{*}{GloVe} & 74.77\({}^{\star}\) & 74.01\({}^{\star}\) & **77.80** & 75.99 & 73.39\({}^{\star}\) & **76.80** & 75.01 \\  & & & (1.54) & (1.93) & (1.34) & (2.48) & (1.74) & (0.99) & (1.93) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{targeted}}}} & \multirow{2}{*}{TLA} & 80.11\({}^{\star}\) & 80.27 & 81.47\({}^{\star}\) & **83.14** & 81.49 & 81.07 & 79.84\({}^{\star}\) \\  & & & (1.23) & (0.50) & (2.22) & (1.24) & (1.38) & (0.58) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\begin{} \end{tabular} } }}} & \multirow{2}{*}{GloVe} & 25.67\({}^{\star}\) & 26.39\({}^{\star}\) & **28.59** & 27.33 & 26.95 & 27.31 & 26.01\({}^{\star}\) \\  & & & (1.62) & (0.60) & (1.42) & (1.70) & (1.07) & (0.82) & (1.26) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\begin{} \end{tabular} } }}} & \multirow{2}{*}{TL} & 28.36 & 28.09 & 28.56 & 28.33 & 28.54 & 28.37 & **28.72** \\  & & & (2.81) & (1.68) & (1.60) & (0.52) & (1.83) & (0.77) & (1.20) \\ \hline \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\begin{} \end{tabular} } }} & \multirow{4}{*}{sentiment} & \multirow{2}{*}{GloVe} & 62.38 & 64.01 & 63.44 & 63.33 & **64.30** & 63.15 & 63.94 \\  & & & (3.75) & (2.72) & (2.21) & (1.87) & (3.14) & (3.38) & (1.84) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\begin{} \end{tabular} } }}} & \multirow{2}{*}{TL} & 67.23 & 68.98 & **69.70** & 67.62 & 66.93 & 68.13 & 68.17 \\  & & & (1.08) & (1.17) & (2.51) & (1.58) & (1.79) & (1.25) & (2.44) \\ \cline{1-1} \cline{3-11}  & \multirow{2}{*}{extraction} & \multirow{2}{*}{GloVe} & 75.53 & **76.40** & 75.75 & 75.66 & 75.29 & 75.87 & 75.58 \\  & & & (1.03) & (1.90) & (1.18) & (1.65) & (0.77) & (0.97) & (1.48) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{targeted}}}} & \multirow{2}{*}{TL} & 77.92 & 77.84 & **79.10** & 78.76 & 78.20 & 77.15 & 77.61 \\  & & & (1.36) & (0.84) & (1.48) & (1.27) & (1.80) & (1.92) & (1.87) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\begin{} \end{tabular} } }}} & \multirow{2}{*}{GloVe} & 47.14 & **48.94** & 48.07 & 47.90 & 48.41 & 47.94 & 48.35 \\  & & & (3.24) & (3.06) & (2.22) & (1.25) & (2.48) & (3.14) & (2.32) \\ \multirow{4}{*}{\multirow{4}{*}{\multirow{4}{*}{\begin{} \end{tabular} } }} & \multirow{2}{*}{TL} & 52.39 & **53.69** & **55.15** & 53.25 & 52.34 & 52.55 & 52.94 \\  & & & (1.18) & (0.69) & (2.70) & (1.10) & (1.85) & (1.22) & (2.99) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Sentiment (_acc-s_), extraction (\(F_{1}\)-_a_) and full targeted (\(F_{1}\)-_f_) results for Laptop\({}_{Spec}\) and Restaurant\({}_{Spec}\) test split, where the values represent the mean (standard deviation) of five runs with a different random seeds. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The \({}^{\star}\) represents the models that are significantly worse (p \(<0.05\)) than the best performing model on the respective dataset, metric, and embedding.

 

caster University by an EPSRC Doctoral Training Grant. The authors thank the UCREL research centre for hosting the models created from this research.

## References

* Ovesdotter Alm et al. (2005) Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_, pages 579-586, Vancouver, BC, Canada.
* Augenstein et al. (2018) Isabelle Augenstein, Sebastian Ruder, and Anders Sogaard. 2018. Multi-task learning of pairwise sequence classification tasks over disparate label spaces. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1896-1906, New Orleans, USA.
* Barnes (2019) Jeremy Barnes. 2019. Ltg-oslo hierarchical multi-task network: The importance of negation for document-level sentiment in spanish. In _Proceedings of the Iberian Languages Evaluation Forum co-located with 35th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2019, Bilbao, Spain, September 24th, 2019._, pages 378-389.
* Barnes et al. (2019) Jeremy Barnes, Lilja Ovrelid, and Erik Velldal. 2019. Sentiment analysis is not solved! Assessing and probing sentiment classification. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 12-23, Florence, Italy.
* Barnes et al. (2020) Jeremy Barnes, Erik Velldal, and Lilja Ovrelid. 2020. Improving sentiment analysis with multitask learning of negation. _Natural Language Engineering_, 27.
* Bauman et al. (2017) Konstantin Bauman, Bing Liu, and Alexander Tuzhilin. 2017. Aspect based recommendations: Recommending items with the most valuable aspects based on user reviews. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 717-725. ACM.
* Caruana (1993) Richard Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In _Proceedings of the Tenth International Conference on Machine Learning_, pages 41-48. Morgan Kaufmann.
* Chelba et al. (2014) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In _Fifteenth Annual Conference of the International Speech Communication Association_.
* Chen et al. (2012) Lu Chen, Wenbo Wang, Meenakshi Nagarajan, Shaojun Wang, and Amit Sheth. 2012. Extracting diverse sentiment expressions with target-dependent polarity from twitter. In _Proceedings of the International AAAI Conference on Web and Social Media_.
* Chen and Qian (2020) Zhuang Chen and Tieyun Qian. 2020. Relation-aware collaborative learning for unified aspect-based sentiment analysis. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3685-3694, Online. Association for Computational Linguistics.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Council et al. (2010) Isaac Councill, Ryan McDonald, and Leonid Velikovich. 2010. What's great and what's not: learning to classify the scope of negation for improved sentiment analysis. In _Proceedings of the Workshop on Negation and Speculation in Natural Language Processing_, pages 51-59, Uppsala, Sweden.
* Cruz et al. (2016) Noa P. Cruz, Maite Taboada, and Ruslan Mitkov. 2016. A machine-learning approach to negation and speculation detection for sentiment analysis. _Journal of the Association for Information Science and Technology_, 67(9):2118-2136.
* Das and Chen (2007) Sanjiv R. Das and Mike Y. Chen. 2007. Yahoo! for Amazon: Sentiment extraction from small talk on the web. _Management Science_, 53(9):1375-1388.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2185-2194, Hong Kong, China. Association for Computational Linguistics.
* Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics 

* Ettinger (2020) Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. _Transactions of the Association for Computational Linguistics_, 8:34-48.
* Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Aliy Zhang, and Ben Zhou. 2020. Evaluating nlp models via contrast sets.
* Hashimoto et al. (2017) Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsurrouka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1923-1933, Copenhagen, Denmark. Association for Computational Linguistics.
* He et al. (2019) Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 504-515, Florence, Italy. Association for Computational Linguistics.
* Hu et al. (2019) Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted sentiment analysis via span-based extraction and classification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 537-546, Florence, Italy. Association for Computational Linguistics.
* Jiang et al. (2019) Qingnan Jiang, Lei Chen, Ruitfeng Xu, Xiang Ao, and Min Yang. 2019. A challenge dataset and effective models for aspect-based sentiment analysis. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6279-6284, Hong Kong, China. Association for Computational Linguistics.
* Jin and Ho (2009) Wei Jin and Hung Hay Ho. 2009. A novel lexicalized hmm-based learning framework for web opinion mining. In _Proceedings of the 26th Annual International Conference on Machine Learning_, ICML '09, page 465-472, New York, NY, USA. Association for Computing Machinery.
* Kassner and Schutze (2020) Nora Kassner and Hinrich Schutze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7811-7818, Online. Association for Computational Linguistics.
* Katiyan and Cardie (2016) Arzoo Katiyan and Claire Cardie. 2016. Investigating LSTMs for joint extraction of opinion entities and relations. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 919-929, Berlin, Germany. Association for Computational Linguistics.
* Kaushik et al. (2020) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In _International Conference on Learning Representations_.
* Kim and Klinger (2019) Evgeny Kim and Roman Klinger. 2019. Fromwing Frodo, wincing Leia, and a seriously great friendship: Learning to classify emotional relationships of fictional characters. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 647-653, Minneapolis, Minnesota. Association for Computational Linguistics.
* overview. In _Proceedings of the BioNLP Shared Task 2013 Workshop_, pages 8-15, Sofia, Bulgaria. Association for Computational Linguistics.
* Klinger and Cimiano (2013) Roman Klinger and Philipp Cimiano. 2013. Bi-directional inter-dependencies of subjective expressions and targets and their value for a joint model. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 848-854.
* Konstantinova et al. (2012) Natalia Konstantinova, Sheila C.M. de Sousa, Noa P. Cruz, Manuel J. Mana, Maite Taboada, and Ruslan Mitkov. 2012. A review corpus annotated for negation, speculation and their scope. In _Proceedings of the 8th International Conference on Language Resources and Evaluation_, pages 3190-3195, Istanbul, Turkey.
* Kshirsagar et al. (2015) Meghana Kshirsagar, Sam Thomson, Nathan Schneider, Jaime Carbonell, Noah A. Smith, and Chris Dyer. 2015. Frame-semantic role labeling with heterogeneous annotations. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_, pages 218-224, Beijing, China.
* Lapponi et al. (2012) Emanele Lapponi, Jonathon Read, and Lilja Ovrelid. 2012. Representing and resolving negation for sentiment analysis. In _Proceedings of the 2012 IEEE 12th International Conference on Data Mining Workshops_, pages 687-692, Washington, DC, USA.
* Li et al. (2019a) Xin Li, Lidong Bing, Piji Li, and Wai Lam. 2019a. A unified model for opinion target extraction and target sentiment prediction. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 6714-6721.
* Li et al. (2019b) Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. 2019b. Exploiting BERT for end-to-end aspect-based sentiment analysis. In _Proceedings of the 5th 

Workshop on Noisy User-generated Text (W-NUT 2019)_, pages 34-41, Hong Kong, China. Association for Computational Linguistics.
* Liu [2015] Bing Liu. 2015. _Sentiment analysis: Mining Opinions, Sentiments, and Emotions_. Cambridge University Press, Cambridge, United Kingdom.
* Ma et al. [2018] Dehong Ma, Sujian Li, and Houfeng Wang. 2018. Joint learning for targeted sentiment analysis. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4737-4742, Brussels, Belgium. Association for Computational Linguistics.
* Alonso and Plank [2017] Hector Martinez Alonso and Barbara Plank. 2017. When is multitask learning effective? Semantic sequence prediction under varying data conditions. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_, pages 44-53, Valencia, Spain.
* McAuley et al. [2015] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '15, page 43-52, New York, NY, USA. Association for Computing Machinery.
* Mikolov et al. [2013] Tomas Mikolov, Greg Corrado, Kai Chen, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _Proceedings of the International Conference on Learning Representations_, pages 1-12. Mitchell et al. [2013] Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1643-1654, Seattle, Washington, USA. Association for Computational Linguistics.
* Morante and Daelemans [2012] Roser Morante and Walter Daelemans. 2012. ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories. In _Proceedings of the 8th International Conference on Language Resources and Evaluation_, Istanbul, Turkey.
* Ovrelid et al. [2020] Lilja Ovrelid, Petter Maehlum, Jeremy Barnes, and Erik Velldal. 2020. A fine-grained sentiment dataset for norwegian. In _Proceedings of The 12th Language Resources and Evaluation Conference_, pages 5025-5033, Marseille, France. European Language Resources Association.
* Peng and Dredze [2017] Nanyun Peng and Mark Dredze. 2017. Multi-task domain adaptation for sequence tagging. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_, pages 91-100, Vancouver, Canada.
* Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1532-1543, Doha, Qatar.
* Peters et al. [2018a] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Peters et al. [2018b] Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1499-1509, Brussels, Belgium. Association for Computational Linguistics.
* Plank [2016] Barbara Plank. 2016. Keystroke dynamics as signal for shallow syntactic parsing. In _Proceedings of the 26th International Conference on Computational Linguistics_, pages 609-619, Osaka, Japan.
* Pontiki et al. [2016] Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphee De Clercq, Veronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud Maria Jimenez-Zafra, and Gulsen Eryigit. 2016. SemEval-2016 task 5: Aspect based sentiment analysis. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, pages 19-30, San Diego, California. Association for Computational Linguistics.
* Pontiki et al. [2015] Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 task 12: Aspect based sentiment analysis. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_, pages 486-495, Denver, Colorado. Association for Computational Linguistics.
* Pontiki et al. [2014] Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_, pages 27-35, Dublin, Ireland. Association for Computational Linguistics.
* Reimers and Gurevych [2017] Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 338-348, Copenhagen, Denmark. Association for Computational Linguistics.
* Reimers and Gurevych [2018] Nils Reimers and Iryna Gurevych. 2018. Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. _arXiv preprint arXiv:1803.09578_ 

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4902-4912, Online. Association for Computational Linguistics.
* Sauri and Pustejovsky (2009) Roser Sauri and James Pustejovsky. 2009. Factbank: a corpus annotated with event factuality. _Language Resources and Evaluation_, 43:227-268.
* Schneider and Smith (2015) Nathan Schneider and Noah A. Smith. 2015. A corpus and model integrating multiword expressions and supersenses. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1537-1547, Denver, Colorado. Association for Computational Linguistics.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jy Wu, Jason Chuang, Chris Manning, Andrew Ng, and Chris Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, USA.
* Sogaard and Goldberg (2016) Anders Sogaard and Yoav Goldberg. 2016. Deep multitask learning with low level tasks supervised at lower layers. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 231-235, Berlin, Germany.
* Vincze et al. (2008) Veronika Vincze, Gyorgy Szarvas, Richard Farkas, Gyorgy Mora, and Janos Csirik. 2008. The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes. _BMC bioinformatics_, Suppl 11.
* Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. _arXiv preprint 1905.00537_.
* Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _International Conference on Learning Representations_.
* Wang et al. (2017) Bo Wang, Maria Liakata, Arkaitz Zubiaga, and Rob Procter. 2017. TDParse: Multi-target-specific sentiment recognition on twitter. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 483-493, Valencia, Spain. Association for Computational Linguistics.
* Welch (1947) B. L. Welch. 1947. The generalization of 'student's problem when several different population variances are involved. _Biometrika_, 34(1/2):28-35.
* Wiebe et al. (2005) Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. _Language Resources and Evaluation_, 39(2-3):165-210.
* Wilcoxon (1945) Frank Wilcoxon. 1945. Individual comparisons by ranking methods. _Biometrics Bulletin_, 1(6):80-83.
* Xue and Li (2018) Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2514-2523, Melbourne, Australia. Association for Computational Linguistics.
* Zhang et al. (2015a) Meishan Zhang, Yue Zhang, and Duy Tin Vo. 2015a. Neural networks for open domain targeted sentiment. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)_, pages 612-621.
* Zhang et al. (2015b) Meishan Zhang, Yue Zhang, and Duy-Tin Vo. 2015b. Neural networks for open domain targeted sentiment. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 612-621, Lisbon, Portugal. Association for Computational Linguistics.

 

Class Distribution of the Sentiment Datasets 



\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c}  & \multicolumn{4}{c}{**Train**} & \multicolumn{4}{c}{**Dev**} & \multicolumn{4}{c}{**Test**} \\ \cline{2-13}  & pos & neu & neg & both & pos & neu & neg & both & pos & neu & neg & both \\ Laptop & 19.9 & 43.2 & 36.9 & - & 18.0 & 40.6 & 41.4 & - & 26.0 & 53.5 & 20.5 & - \\ Laptop\(N_{Neg}\) & - & - & - & - & 17.1 & 35.4 & 47.5 & - & 26.7 & 23.3 & 50.0 & - \\ Laptop\(S_{Spec}\) & - & - & - & - & 50.7 & 16.2 & 33.1 & - & 38.2 & 20.5 & 41.4 & - \\ Restaurant & 15.8 & 60.0 & 24.2 & - & 12.3 & 65.2 & 22.5 & - & 11.5 & 66.6 & 21.9 & - \\ Restaurant\(N_{Neg}\) & - & - & - & - & 16.4 & 32.5 & 51.1 & - & 15.0 & 32.2 & 52.8 & - \\ Restaurant\(S_{Spec}\) & - & - & - & - & 30.0 & 29.0 & 41.0 & - & 16.9 & 39.7 & 43.5 & - \\ MAMS & 45.1 & 30.2 & 24.7 & - & 45.5 & 30.3 & 24.3 & - & 45.5 & 29.9 & 24.6 & - \\ MPQA & 13.3 & 43.9 & 39.1 & 3.7 & 17.0 & 42.5 & 37.0 & 3.5 & 19.2 & 33.2 & 41.4 & 6.3 \\ \hline \end{tabular}
\end{table}
Table 7: Sentiment class distribution statistics as a percentage of the number of targets (samples), for the sentiment datasets used in the experiments. pos, neu, neg, and both represent the sentiment classes positive, neutral, negative, and both respectively.



Examples of Auxiliary Tasks 



\begin{table}
\begin{tabular}{l l l l l l l}  & you & might & not & like & the & service \\ NEG\({}_{CD}\) & B\({}_{scope}\) & I\({}_{scope}\) & B\({}_{cue}\) & B\({}_{scope}\) & I\({}_{scope}\) & I\({}_{scope}\) \\ NEG\({}_{SFU}\) & B\({}_{scope}\) & I\({}_{scope}\) & B\({}_{cue}\) & B\({}_{scope}\) & I\({}_{scope}\) & I\({}_{scope}\) \\ SPEC & B\({}_{scope}\) & B\({}_{cue}\) & B\({}_{scope}\) & I\({}_{scope}\) & I\({}_{scope}\) & I\({}_{scope}\) \\ UPOS & PRON & AUX & PART & VERB & DET & NOUN \\ DR & nsubj & aux & advmod & root & det & obj \\ LEX & O\({}_{PRON}\) & O\({}_{AUX}\) & O\({}_{ADV}\) & B\({}_{V-v.emotion}\) & O\({}_{DET}\) & B\({}_{N-n.ACT}\) \\ \hline \end{tabular}
\end{table}
Table 8: A toy example sentence with the labels from each auxiliary task

Additional Main Result Tables 



\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPDS & STL \\ \multirow{4}{*}{2-5} & acc-s & GloVe & **71.90** & 70.66 & 70.36 & 70.30 & 68.11\({}^{*}\) & 69.60\({}^{*}\) & 70.80 \\  & acc-s & & (1.32) & (1.55) & (2.24) & (1.86) & (2.19) & (1.98) & (2.02) \\  & & TL & 75.30 & 74.75 & 74.56 & 74.36 & 74.47\({}^{*}\) & 74.70\({}^{*}\) & **76.85** \\  & & & (0.54) & (1.14) & (1.49) & (1.74) & (0.82) & (1.02) & (1.96) \\ \cline{2-10}  & F1-s & GloVe & **65.00** & **63.19** & **63.07** & **62.60\({}^{\star}\)** & 59.83\({}^{\star}\) & 61.51 & 61.90 \\  & F1-s & & (1.36) & (2.32) & (3.51) & (2.74) & (2.46) & (2.66) & (3.32) \\  & & TL & 66.92 & 67.76 & 67.26 & 66.00 & 66.92\({}^{\star}\) & 66.63\({}^{\star}\) & **69.91** \\  & & (2.41) & (1.75) & (2.27) & (3.03) & (1.21) & (1.61) & (2.72) \\ \cline{2-10}  & extraction & GloVe & 76.00\({}^{*}\) & 75.98\({}^{\star}\) & (1.17) & (1.14) & (1.57) & (1.57) & (1.10) & 76.76 \\  & & TL & **83.51** & 83.32 & 83.88 & **84.21** & 83.29 & 83.48 & 82.90 \\  & & (1.09) & (0.94) & (0.88) & (1.81) & (1.15) & (1.30) & (0.72) \\ \cline{2-10}  & targeted & GloVe & 54.65 & 53.67 & **54.85** & 53.73 & 51.65\({}^{*}\) & 54.17 & 54.37 \\  & & (1.37) & (0.94) & (0.99) & (1.93) & (2.32) & (2.26) & (2.56) \\  & & TL & 62.89 & 62.29 & 62.55 & 62.61 & 62.03 & 62.35 & **63.70** \\  & & (1.18) & (1.32) & (1.66) & (1.79) & (1.14) & (0.77) & (1.14) \\ \hline  & acc-s & GloVe & **78.18** & 74.37 & 75.94 & 77.38 & 72.82\({}^{\star}\) & 73.83\({}^{\star}\) & 73.01\({}^{*}\) \\  & & (2.72) & (3.47) & (3.02) & (4.91) & (3.88) & (2.30) & (3.41) \\  & & TL & 71.84 & 72.01 & **73.08** & 70.96 & 72.79 & 72.61 & 70.47 \\  & & (3.46) & (3.01) & (4.01) & (2.03) & (3.13) & (3.84) & (1.51) \\ \cline{2-10}  & F1-s & GloVe & **42.03** & 39.96 & 40.58 & 41.16 & 39.19 & 39.90\({}^{*}\) & 39.00 \\  & & (1.50) & (1.66) & (1.28) & (2.32) & (1.94) & (1.06) & (1.86) \\  & & TL & 39.92 & 40.27 & **41.13** & 39.17 & 39.84 & 39.90 \\  & & (1.15) & (0.80) & (3.05) & (0.79) & (1.56) & (1.66) & (0.68) \\ \cline{2-10}  & extraction & GloVe & 24.17 & 22.93\({}^{\star}\) & 24.58 & 22.84\({}^{\star}\) & 22.90\({}^{\star}\) & **25.34** & 24.77 \\  & & (1.44) & (1.57) & (1.36) & (1.58) & (2.49) & (0.46) & (3.55) \\  & & TL & 30.98\({}^{\star}\) & 30.71\({}^{\star}\) & 31.19\({}^{\star}\) & 31.41\({}^{\star}\) & 31.41\({}^{\star}\) & 31.88\({}^{\star}\) & **34.99** \\  & & (2.40) & (1.10) & (2.69) & (1.16) & (0.51) & (2.62) & (1.10) \\ \cline{2-10}  & & (1.88) & 17.03\({}^{\star}\) & 18.66 & 17.60\({}^{\star}\) & 16.70\({}^{\star}\) & 18.70 & 18.11 \\  & & (1.17) & (1.12) & (1.22) & (0.57) & (2.26) & (0.25) & (2.83) \\ \cline{2-10}  & TL & 22.25\({}^{\star}\) & 22.09\({}^{\star}\) & 22.74 & 22.30\({}^{\star}\) & 22.86\({}^{\star}\) & 23.05\({}^{\star}\) & **24.66** \\  & & (2.00) & (0.70) & (1.68) & (1.19) & (0.98) & (0.88) & (1.07) \\ \hline \hline \end{tabular}
\end{table}
Table 9: _acc-s, \(F_{1}\)-s, extraction (\(F_{1}\)-\(a\)) and full targeted (\(F_{1}\)-\(i\)) results for Laptop and MPQA **test split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The \({}^{*}\) represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and TL at a 95% confidence level.





\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\  & acc-s & GloVe & **81.72** & 81.42 (0.32) & 81.24 (0.44) & 81.10 (0.46) & 80.90 (0.70) & 81.58 (0.96) & 81.70 (0.79) \\  & & TL & 84.59 (0.50) & **84.81** & 83.99* (0.84) & 83.73 (0.68) & 84.28 (0.39) & 83.90* (0.51) & 84.67 (0.56) \\
**F1-s** & GloVe & **81.05** & 80.71 (0.43) & 80.53 (0.60) & 80.39 (0.69) & 80.22 (0.80) & 80.81 (0.01) & 80.94 (0.88) \\  & & TL & 84.24 (0.51) & **84.39** & 83.59* (0.87) & 83.30 (0.67) & 83.89 (0.32) & 83.44* (0.60) & 84.24 (0.58) \\  & & GloVe & 76.49 (0.99) & 76.21* (0.39) & 76.49 (0.106) & 76.87 (0.64) & 76.83 (0.48) & 76.97 (0.54) & **77.36** (0.19) \\  & & TL & 77.04 (0.35) & 76.76* (0.13) & 76.98 (0.84) & 77.64 (0.36) & 77.33 (0.79) & 77.33 (0.61) & 77.59 (0.35) \\  & targeted & GloVe & 62.50 (0.42) & 62.05* (0.32) & 62.14* (0.83) & 62.34* (0.54) & 62.16 (0.71) & 62.79 (0.37) & **63.20** (0.65) \\  & & TL & 65.17 (0.35) & 65.10 (0.63) & 64.65* (0.88) & 65.00* (0.48) & 64.50* (0.79) & 64.88 (0.46) & **65.70** (0.55) \\ \hline  & & Acc-s & GloVe & 83.02 (1.82) & 83.23 (1.69) & 83.26 (0.89) & 83.80 (0.78) & 83.01 (1.16) & 83.36 (1.09) & 83.65 (0.48) \\  & & TL & 87.40 (0.67) & **87.63** (0.76) & 87.37 (0.90) & 87.26 (0.96) & 87.36 (0.48) & 87.00* (0.56) & 87.32 (0.66) \\  & & GloVe & 66.75 (0.75) & 67.79 (0.30) & 67.59 (1.39) & 67.75 (1.92) & 67.35 (0.02) & 67.13 (0.31) & **68.00** (1.61) \\  & & TL & 72.27 (1.14) & 72.96 (1.79) & 73.73 (2.60) & 72.12 (2.30) & 73.90 (2.82) & 71.61 (1.13) & 73.47 (1.10) \\  & & GLoVe & 78.33 (1.55) & 79.34 (1.60) & 79.13 (0.93) & 78.53 (0.96) & 78.48 (0.81) & 78.84 (0.78) & 78.42 (0.85) \\  & & TL & 81.27 (0.90) & 81.53 (1.01) & 82.13 (1.09) & 82.08 (1.22) & 81.85 (1.37) & 80.94 (1.18) \\ targeted & GloVe & 65.06 (2.66) & **66.06** (2.63) & 65.89 (1.32) & 65.82 (1.31) & 65.16 (1.50) & 65.73 (1.46) & 65.60 (1.06) \\  & & TL & 71.04 (1.13) & 71.45 (1.47) & 71.77 (1.88) & 71.63 (1.64) & 71.51 (1.16) & 70.38 (1.53) \\ \hline \hline \end{tabular}
\end{table}
Table 10: _acc-s, \(F_{1}\)-s, extraction (\(F_{1}\)-_a_) and full targeted (\(F_{1}\)-_i_) results for MAMS and Restaurant **test split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The \({}^{*}\) represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and TL at a 95% confidence level.





\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\ \multirow{4}{*}{2-5} & acc-s & GloVe & 71.47 & 77.53 & 76.52 & 78.34 & 77.32 & 76.67 & 77.22 \\  & & (1.43) & (0.69) & (1.38) & (2.23) & (1.06) & (1.77) & (1.05) \\  & & TL & 81.22 & 79.63\({}^{\star}\) & 80.54 & 80.05\({}^{\star}\) & 80.87 & 79.31\({}^{\star}\) & **81.89** \\  & & (1.42) & (1.56) & (2.27) & (1.28) & (1.64) & (1.12) & (1.07) \\ \cline{2-10}  & F1-s & GloVe & 71.55 & 70.75 & 70.61 & 72.04 & 70.70 & 67.82 & 67.27 \\  & & (0.81) & (2.75) & (2.46) & (3.85) & (1.83) & (1.20) & (2.80) \\  & & TL & 74.53 & 74.84 & 74.89 & 73.70 & 75.22 & 71.90\({}^{\star}\) & **75.33** \\  & & (3.09) & (1.87) & (3.32) & (1.98) & (2.03) & (1.77) & (1.61) \\ \multirow{4}{*}{extraction} & GloVe & 74.87 & 73.55 & 75.70 & 74.81 & 74.75 & 74.15\({}^{\star}\) & 75.06 \\  & & (1.15) & (2.09) & (0.96) & (1.18) & (0.57) & (1.84) & (1.06) \\ \multirow{4}{*}{targeted} & TL & 80.82 & 80.89 & 80.39 & 81.86 & 80.05\({}^{\star}\) & 81.37 & 81.23 \\  & & (1.35) & (0.83) & (1.02) & (1.24) & (0.59) & (0.99) & (0.82) \\ \cline{2-10}  & & 57.99 & 57.02 & 57.92 & 58.62 & 57.80 & 56.82 & 57.97 \\  & & (0.69) & (1.53) & (1.08) & (2.19) & (1.10) & (0.71) & (1.24) \\ \multirow{4}{*}{targeted} & \multirow{2}{*}{TL} & \multirow{2}{*}{TL} & \multirow{2}{*}{65.62\({}^{\star}\)} & \multirow{2}{*}{64.42} & \multirow{2}{*}{64.73} & \multirow{2}{*}{65.52\({}^{\star}\)} & \multirow{2}{*}{64.72\({}^{\star}\)} & \multirow{2}{*}{64.55\({}^{\star}\)} & \multirow{2}{*}{66.51} \\  & & (0.76) & (1.64) & (1.30) & (0.52) & (0.97) & (1.53) & (0.43) \\ \hline \multirow{4}{*}{2-5} & acc-s & GloVe & 87.75 & 88.65 & 87.64 & 89.11 & 86.85 & 88.16 & 85.29\({}^{\star}\) \\  & & (3.15) & (4.20) & (3.71) & (3.29) & (1.46) & (3.20) & (0.89) \\ \multirow{4}{*}{2-5} & \multirow{2}{*}{TL} & \multirow{2}{*}{RL} & \multirow{2}{*}{88.63} & \multirow{2}{*}{90.08} & \multirow{2}{*}{87.23} & \multirow{2}{*}{85.62} & \multirow{2}{*}{88.71} & \multirow{2}{*}{88.75} & \multirow{2}{*}{88.01} \\  & & (1.83) & (2.94) & (1.80) & (4.14) & (2.89) & (1.56) & (1.36) \\ \cline{1-1} \cline{5-12}  & & 54.18 & 59.87 & 51.51 & 56.39 & 48.09 & 52.32 & 45.92 \\ \multirow{4}{*}{2-5} & \multirow{2}{*}{TL} & \multirow{2}{*}{(8.15)} & \multirow{2}{*}{(14.46)} & \multirow{2}{*}{(8.83)} & \multirow{2}{*}{(10.28)} & \multirow{2}{*}{(4.83)} & \multirow{2}{*}{(9.96)} & \multirow{2}{*}{(3.06)} \\  & & (8.15) & (14.46) & (8.83) & (10.28) & (4.83) & (9.96) & (3.06) \\ \multirow{4}{*}{2-5} & \multirow{2}{*}{TL} & \multirow{2}{*}{52.83\({}^{\star}\)} & \multirow{2}{*}{(3.40)} & \multirow{2}{*}{(5.28)} & \multirow{2}{*}{(5.99)} & \multirow{2}{*}{(9.33)} & \multirow{2}{*}{(3.28)} & \multirow{2}{*}{(7.01)} & \multirow{2}{*}{(6.52)} \\  & & (3.40) & (5.28) & (5.99) & (9.33) & (3.28) & (7.01) & (6.52) \\ \multirow{4}{*}{extraction} & GloVe & 20.68 & 21.00 & 20.78 & 20.48\({}^{\star}\) & 19.54\({}^{\star}\) & 21.73 & 20.11 \\  & & (0.65) & (1.73) & (1.52) & (0.85) & (2.18) & (0.74) & (2.38) \\ \multirow{4}{*}{2-5} & \multirow{2}{*}{TL} & \multirow{2}{*}{TL} & 32.33 & 30.39\({}^{\star}\) & 31.75 & 32.18 & 30.65\({}^{\star}\) & 31.00\({}^{\star}\) & **33.38** \\  & & (3.71) & (1.09) & (1.78) & (1.29) & (1.52) & (1.92) & (1.92) \\ \multirow{4}{*}{2-5} & \multirow{2}{*}{GloVe} & \multirow{2}{*}{LR} & 18.14 & 18.57 & 18.18 & 18.23 & 16.94\({}^{\star}\) & **19.16** & 17.16 \\  & & (0.58) & (1.13) & (1.10) & (0.55) & (1.66) & (0.89) & (2.07) \\ \multirow{4}{*}{2-5} & \multirow{2}{*}{TL} & \multirow{2}{*}{LR} & 28.60 & 27.35\({}^{\star}\) & 27.67\({}^{\star}\) & 27.53\({}^{\star}\) & 27.15\({}^{\star}\) & 27.49\({}^{\star}\) & **29.39** \\  & & (2.82) & (0.72) & (1.29) & (1.30) & (0.92) & (1.38) & (0.95) \\ \hline \hline \end{tabular}
\end{table}
Table 11: _acc-s, \(F_{1}\)-s, extraction (\(F_{1}\)-_a_) and full targeted (\(F_{1}\)-_i_) results for Laptop and MPQA **development split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The \({}^{\star}\) represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and TL at a 95% confidence level.





\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\ \multirow{4}{*}{**acc-s**} & \multirow{2}{*}{GloVe} & 80.73\({}^{\star}\) & 80.98 & 80.89\({}^{\star}\) & 80.68\({}^{\star}\) & 80.87\({}^{\star}\) & 81.36 & **82.14** \\  & & (0.22) & (1.17) & (0.45) & (0.48) & (0.53) & (0.82) & (0.77) \\  & & TL & 84.37 & 83.99\({}^{\star}\) & **84.73** & 84.18 & 84.17 & 83.79\({}^{\star}\) & 84.43 \\  & & (0.22) & (0.34) & (0.34) & (0.49) & (0.53) & (0.40) & (0.41) \\ \cline{2-10}  & \multirow{2}{*}{F1-s} & GloVe & 80.20\({}^{\star}\) & 80.48\({}^{\star}\) & 80.38\({}^{\star}\) & 80.14\({}^{\star}\) & 80.38\({}^{\star}\) & 80.76 & **81.67** \\  & & (0.24) & (1.15) & (0.50) & (0.57) & (0.56) & (0.95) & (0.77) \\ \cline{2-10}  & \multirow{2}{*}{TL} & 84.14\({}^{\star}\) & 83.72\({}^{\star}\) & **84.46** & 83.96\({}^{\star}\) & 83.93 & 83.54\({}^{\star}\) & 84.12 \\  & & (0.24) & (0.21) & (0.35) & (0.47) & (0.47) & (0.42) & (0.45) \\ \cline{2-10}  & \multirow{2}{*}{extraction} & GloVe & 78.93 & 79.11 & **79.24** & 79.00 & 78.86 & 78.68 & 79.15 \\  & & (0.66) & (0.52) & (0.62) & (0.47) & (0.67) & (0.35) & (0.39) \\ \cline{2-10}  & \multirow{2}{*}{TL} & 77.81 & 77.86 & 77.62\({}^{\star}\) & 78.32 & 77.59\({}^{\star}\) & **78.54** & 78.35 \\  & & (0.48) & (0.59) & (0.34) & (0.31) & (0.21) & (0.38) & (0.40) \\ \cline{2-10}  & \multirow{2}{*}{targeted} & GloVe & 63.72\({}^{\star}\) & 64.06\({}^{\star}\) & 64.10\({}^{\star}\) & 63.74\({}^{\star}\) & 63.76\({}^{\star}\) & 64.01\({}^{\star}\) & **65.01** \\  & & (0.63) & (0.66) & (0.48) & (0.28) & (0.21) & (0.62) & (0.44) \\ \cline{2-10}  & \multirow{2}{*}{TL} & 65.65 & 65.39\({}^{\star}\) & 65.77\({}^{\star}\) & 65.93 & 65.31\({}^{\star}\) & 65.81 & **66.15** \\  & & (0.39) & (0.47) & (0.44) & (0.20) & (0.56) & (0.41) & (0.54) \\ \hline \multirow{4}{*}{**acc-s**} & \multirow{2}{*}{GloVe} & 78.42\({}^{\star}\) & 78.78 & 79.58 & 78.75 & 78.31\({}^{\star}\) & 79.14 & 78.76 \\  & & (0.78) & (0.67) & (0.89) & (0.52) & (0.78) & (0.41) & (0.37) \\ \cline{2-10}  & \multirow{2}{*}{FL} & 81.90 & **81.90** & 81.53 & 81.89 & 81.02 & 80.47\({}^{\star}\) & 81.77 \\  & & (0.69) & (0.69) & (0.86) & (0.84) & (0.85) & (1.10) & (0.46) \\ \cline{2-10}  & \multirow{2}{*}{F1-s} & GloVe & 62.89 & 64.01 & **65.37** & 62.49\({}^{\star}\) & 62.54\({}^{\star}\) & 63.00\({}^{\star}\) & 63.15\({}^{\star}\) \\  & & (2.84) & (2.56) & (1.47) & (0.86) & (2.28) & (1.64) & (1.94) \\ \cline{2-10}  & \multirow{2}{*}{TL} & 67.98 & 69.26 & 69.09 & 68.18 & 67.54 & 67.14\({}^{\star}\) & **69.37** \\  & & (3.46) & (1.07) & (1.72) & (1.90) & (3.88) & (1.05) & (0.97) \\ \cline{2-10}  & \multirow{2}{*}{extraction} & GloVe & 78.22\({}^{\star}\) & 79.20 & 78.85 & 78.38\({}^{\star}\) & 78.21 & **79.62** & 79.18 \\  & & (0.78) & (1.07) & (1.08) & (0.73) & (1.37) & (0.48) & (0.76) \\ \cline{2-10}  & \multirow{2}{*}{TL} & 81.69 & 81.84 & **82.56** & 82.25 & 82.07 & 82.48 & 82.33 \\  & & (0.71) & (0.88) & (0.79) & (0.22) & (0.68) & (0.61) & (0.52) \\ \cline{2-10}  & \multirow{2}{*}{GloVe} & 61.34\({}^{\star}\) & 62.39 & 62.75 & 61.73\({}^{\star}\) & 61.25\({}^{\star}\) & **63.02** & 62.36 \\ \cline{2-10}  & & (0.73) & (0.95) & (1.26) & (0.77) & (1.26) & (0.42) & (0.60) \\ \cline{2-10}  & \multirow{2}{*}{TL} & 66.90 & 67.03 & 67.31 & **67.36** & 66.49 & 66.38\({}^{\star}\) & 67.32 \\ \cline{2-10}  & & (0.40) & (1.06) & (0.32) & (0.66) & (0.52) & (1.31) & (0.55) \\ \hline \hline \end{tabular}
\end{table}
Table 12: _acc-s, \(F_{1}\)-s, extraction (\(F_{1}\)-\(a\)) and full targeted (\(F_{1}\)-\(i\)) results for MAMS and Restaurant **development split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The \({}^{\star}\) represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.



D Additional Negation and Speculation Result Tables
* [1] 



\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\ \cline{3-10} F1-s & GloVe & 46.58 & 44.16* & 42.74* & 47.65 & 44.00 & 44.78 & 44.14* \\  & & (3.24) & (2.18) & (1.09) & (1.35) & (3.99) & (1.85) & (1.89) \\  & TL & 52.85* & 54.41 & 54.08 & 52.54* & **55.63** & 52.16* & 53.59* \\  & & (1.69) & (1.51) & (3.87) & (1.99) & (1.65) & (2.01) & (1.95) \\ \hline \hline \end{tabular}
\end{table}
Table 13: \(F_{1}\)-\(s\) results for the **negation test split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The * represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\ \cline{3-10} F1-s & GloVe & 32.74 & 33.14 & **35.24** & 33.62 & 33.83 & 33.21 & 31.99* \\  & TL & (2.35) & (0.98) & (2.16) & (2.99) & (1.62) & (1.00) & (1.92) \\  & TL & 33.02 & 33.33 & 33.14 & 31.72* & 33.25 & 32.71 & **34.08** \\  & & (4.07) & (1.56) & (2.10) & (0.88) & (2.14) & (1.38) & (1.40) \\ \hline \multirow{4}{*}{F1-s} & GloVe & 55.27 & **57.77** & 56.27* & 55.59* & 57.35 & 56.55 & 57.32 \\  & & (3.82) & (2.91) & (2.36) & (1.09) & (3.75) & (3.14) & (2.30) \\ \cline{1-1}  & TL & 58.84* & 60.95 & **62.36** & 58.44* & 60.52 & 59.23 & 60.74 \\ \cline{1-1}  & & (1.58) & (0.98) & (2.86) & (2.24) & (3.25) & (1.81) & (2.32) \\ \hline \hline \end{tabular}
\end{table}
Table 14: \(F_{1}\)-\(s\) results for the **speculation test split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The * represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.





\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\ \multirow{4}{*}{acc-s} & \multirow{2}{*}{GloVe} & **37.07** & **32.99\({}^{\star}\)** & **33.98** & **36.64** & **34.83** & **32.09\({}^{\star}\)** & **29.45\({}^{\star}\)** \\  & & & (3.78) & (0.96) & (2.60) & (3.17) & (1.09) & (3.41) & (1.46) \\  & & TL & **41.84\({}^{\star}\)** & **39.38\({}^{\star}\)** & **40.85\({}^{\star}\)** & **45.40** & **43.00\({}^{\star}\)** & **38.77\({}^{\star}\)** & **41.15\({}^{\star}\)** \\  & & & (0.54) & (2.69) & (2.50) & (1.70) & (1.57) & (2.64) & (3.28) \\ \cline{3-8}  & \multirow{2}{*}{F1-s} & \multirow{2}{*}{GloVe} & **34.02** & **31.48\({}^{\star}\)** & **33.06** & **34.61** & **33.66** & **28.92\({}^{\star}\)** & **24.92\({}^{\star}\)** \\  & & & (2.05) & (2.66) & (2.21) & (3.34) & (1.13) & (4.11) & (1.33) \\  & & TL & **38.26\({}^{\star}\)** & **38.05** & **39.42** & **42.32** & **41.39** & **36.73** & **38.92** \\  & & & (1.67) & (3.06) & (4.02) & (3.45) & (1.97) & (3.67) & (3.45) \\ \cline{3-8}  & \multirow{2}{*}{extraction} & \multirow{2}{*}{GloVe} & **72.49** & **74.40** & **73.91** & **71.25\({}^{\star}\)** & **73.83** & **73.33** & **74.60** \\  & & & (1.45) & (1.64) & (0.99) & (1.20) & (0.93) & (1.90) & (1.51) \\  & & TL & **80.94** & **81.31** & **81.21** & **81.94** & **79.00\({}^{\star}\)** & **81.92** & **82.75** \\  & & & (1.64) & (1.29) & (1.36) & (1.46) & (0.87) & (1.36) & (1.80) \\ \cline{3-8}  & \multirow{2}{*}{targeted} & \multirow{2}{*}{GloVe} & **26.87** & **24.56** & **25.12** & **26.14** & **25.71** & **23.54** & **21.98\({}^{\star}\)** \\  & & & (2.75) & (1.21) & (1.99) & (2.55) & (0.68) & (2.66) & (1.26) \\  & & TL & **33.86\({}^{\star}\)** & **32.04\({}^{\star}\)** & **33.14\({}^{\star}\)** & **37.20** & **33.98\({}^{\star}\)** & **31.78\({}^{\star}\)** & **34.01\({}^{\star}\)** \\  & & & (0.71) & (2.54) & (1.56) & (1.59) & (1.47) & (2.46) & (2.29) \\ \hline \multirow{4}{*}{GloVe} & \multirow{2}{*}{acc-s} & \multirow{2}{*}{GloVe} & **46.02** & **43.13\({}^{\star}\)** & **41.06\({}^{\star}\)** & **49.02** & **41.65\({}^{\star}\)** & **44.02\({}^{\star}\)** & **42.69\({}^{\star}\)** \\  & & & (4.88) & (3.24) & (3.36) & (1.31) & (4.06) & (3.09) & (2.01) \\ \cline{3-8}  & \multirow{2}{*}{TL} & \multirow{2}{*}{TLA} & **53.79** & **54.40** & **52.25** & **54.42** & **53.16** & **54.31** & **52.25** \\  & & & (2.56) & (3.63) & (3.18) & (1.92) & (2.49) & (2.51) \\ \cline{3-8}  & \multirow{2}{*}{F1-s} & \multirow{2}{*}{GloVe} & **40.03** & **38.56\({}^{\star}\)** & **37.54** & **41.05** & **37.01** & **38.03** & **38.45** \\  & & & (5.30) & (3.10) & (3.69) & (2.45) & (4.16) & (3.40) & (2.28) \\ \cline{3-8}  & \multirow{2}{*}{TL} & \multirow{2}{*}{TL} & **48.03** & **49.13** & **48.31** & **49.18** & **48.80** & **49.42** & **48.06** \\  & & & (2.62) & (3.71) & (3.58) & (2.03) & (2.76) & (2.12) \\ \cline{3-8}  & \multirow{2}{*}{extraction} & \multirow{2}{*}{GloVe} & **81.74** & **82.37** & **82.36** & **80.61\({}^{\star}\)** & **81.34** & **82.38** & **81.32** \\  & & & (0.77) & (0.64) & (0.80) & (0.72) & (1.48) & (1.00) & (0.37) \\ \cline{3-8}  & \multirow{2}{*}{TL} & **84.08** & **82.87\({}^{\star}\)** & **84.32** & **83.71** & **83.45** & **84.02** & **84.84** \\  & & & (0.72) & (0.66) & (0.68) & (0.55) & (1.09) & (1.12) & **84.84** \\ \cline{3-8}  & \multirow{2}{*}{Glove} & **37.61** & **35.54\({}^{\star}\)** & **33.82\({}^{\star}\)** & **39.51** & **33.88\({}^{\star}\)** & **36.27\({}^{\star}\)** & **34.71\({}^{\star}\)** \\  & & & (3.86) & (2.81) & (2.78) & (1.03) & (3.38) & (2.64) & (1.58) \\ \cline{3-8}  & \multirow{2}{*}{TL} & **45.23** & **45.07** & **44.04** & **45.57** & **44.38** & **45.62** & **44.31** \\ \cline{3-8}  & & & (2.19) & (2.90) & (2.77) & (2.91) & (2.05) & (1.88) & (1.74) \\ \hline \hline \end{tabular}
\end{table}
Table 15: _acc-s, F\({}_{1}\)-s, extraction (\(F_{1}\)-_a_) and full targeted (\(F_{1}\)-_i_) results for the **negation development split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The \({}^{\star}\) represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.





\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & NEG\({}_{CD}\) & DR & LEX & NEG\({}_{SFU}\) & SPEC & UPOS & STL \\  & acc-s & GloVe & **33.56\({}^{\star}\)** & **35.00** & 31.70\({}^{\star}\) & **34.67** & **37.24** & 31.59\({}^{\star}\) & 32.57\({}^{\star}\) \\  & acc-s & ClloVe & **33.56\({}^{\star}\)** & **35.00** & **(2.94)** & **(1.64)** & **(2.21)** & **(2.48)** & **(1.57)** & **(1.94)** \\  & & TL & **34.73** & **35.02** & **35.16** & **32.79** & **34.24** & **34.27** & **34.28** \\  & & & (1.94) & **(3.50)** & **(2.48)** & **(1.43)** & **(1.95)** & **(1.56)** & **(1.19)** \\ \cline{3-8}  & F1-s & GloVe & **32.66** & **33.99** & **30.72\({}^{\star}\)** & **33.20** & **34.99** & **29.88\({}^{\star}\)** & **30.75** \\  & & & (2.33) & **(3.14)** & **(1.33)** & **(2.36)** & **(2.91)** & **(1.50)** & **(2.02)** \\  & & TL & **33.60** & **34.14** & **33.83** & **30.92** & **33.36** & **32.45** & **32.47** \\  & & & (2.30) & **(3.47)** & **(2.83)** & **(1.48)** & **(1.98)** & **(1.32)** & **(1.67)** \\ \cline{3-8}  & extraction & GloVe & **71.10** & **69.42** & **72.61** & **70.46** & **69.33\({}^{\star}\)** & **70.58\({}^{\star}\)** & **73.04** \\  & & TL & **(1.97)** & **(2.25)** & **(0.95)** & **(1.41)** & **(1.58)** & **(2.46)** & **(2.48)** \\  & & TL & **80.50** & **80.25** & **79.91\({}^{\star}\)** & **80.83** & **79.35\({}^{\star}\)** & **80.95** & **82.00** \\  & & & (0.95) & **(0.90)** & **(0.61)** & **(0.61)** & **(1.34)** & **(0.43)** & **(1.32)** \\ \cline{3-8}  & targeted & GloVe & **23.88** & **24.26** & **23.02\({}^{\star}\)** & **24.44\({}^{\star}\)** & **25.82** & **22.32\({}^{\star}\)** & **23.80** \\  & & & (2.05) & **(1.81)** & **(1.35)** & **(1.81)** & **(1.82)** & **(1.73)** & **(1.77)** \\  & & TL & **27.96** & **28.11** & **28.08** & **26.51** & **27.16** & **27.74** & **28.12** \\  & & & (1.60) & **(2.92)** & **(1.81)** & **(1.34)** & **(1.39)** & **(1.31)** & **(1.32)** \\ \hline  & & & & & & & & & \\ \multirow{4}{*}{F1-s} & acc-s & GloVe & **35.54\({}^{\star}\)** & **40.09** & **37.98** & **37.18** & **37.97** & **38.32** & **37.23** \\  & & & (0.90) & **(2.95)** & **(2.38)** & **(2.14)** & **(2.28)** & **(1.31)** & **(1.65)** \\  & & TL & **38.80** & **38.50** & **40.72** & **40.84** & **39.69** & **39.49** & **40.55** \\  & & & (2.17) & **(1.19)** & **(1.18)** & **(2.30)** & **(1.69)** & **(0.89)** & **(1.16)** \\ \cline{3-8}  & & FloVe & **31.46\({}^{\star}\)** & **35.99** & **34.37** & **32.53** & **34.02** & **32.92** & **33.18** \\  & & & (1.47) & **(3.98)** & **(2.89)** & **(1.82)** & **(2.03)** & **(1.74)** & **(1.15)** \\  & & TL & **33.00\({}^{\star}\)** & **33.28** & **35.92** & **35.26** & **34.47** & **35.08** & **35.40** \\  & & & (3.34) & **(1.65)** & **(1.84)** & **(3.37)** & **(3.78)** & **(1.78)** & **(1.59)** \\  & & & & & & & & & \\ \multirow{4}{*}{extraction} & GloVe & **78.26\({}^{\star}\)** & **80.38** & **80.33** & **79.75\({}^{\star}\)** & **79.99\({}^{\star}\)** & **81.98** & **80.59** \\  & & & (0.92) & **(1.86)** & **(1.13)** & **(1.05)** & **(1.20)** & **(1.04)** & **(1.29)** \\ \cline{3-8}  & & TL & **83.91** & **83.93** & **84.60** & **84.77** & **83.40** & **85.03** & **84.85** \\  & & & (0.67) & **(1.03)** & **(0.63)** & **(0.29)** & **(1.58)** & **(1.47)** & **(0.83)** \\ \cline{3-8}  & & Glove & **27.82\({}^{\star}\)** & **32.25** & **30.51\({}^{\star}\)** & **29.64\({}^{\star}\)** & **30.37** & **31.43** & **30.02** \\  & & & (0.79) & **(2.79)** & **(2.06)** & **(1.60)** & **(1.91)** & **(1.42)** & **(1.74)** \\ \cline{3-8}  & TL & **32.56** & **32.31\({}^{\star}\)** & **34.45** & **34.62** & **33.08** & **33.59** & **34.41** \\ \cline{3-8}  & & & (1.83) & **(1.13)** & **(1.00)** & **(1.96)** & **(1.01)** & **(1.20)** & **(1.23)** \\ \hline \hline \end{tabular}
\end{table}
Table 16: _acc-s, \(F_{1}\)-s, extraction (\(F_{1}\)-_a_) and full targeted (\(F_{1}\)-_i_) results for the **speculation development split**, where the values represent the mean (standard deviation) of five runs with a different random seed. The **bold** values represent the best model, while highlighted models are those that perform better than the single task baseline. The \({}^{\star}\) represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.





## Appendix E Hyperparameter Search Space



\begin{table}
\begin{tabular}{c c c} \hline
**GPU Infrastructure** & 1 GeForce GTX 1060 6GB GPU \\ \hline
**CPU Infrastructure** & AMD Ryzen 5 1600 CPU \\ \hline
**Number of search trials** & 30 \\ \hline
**Search strategy** & uniform sampling \\ \hline
**Best validation span F1/F1-i** & 0.6156 \\ \hline
**Training duration** & 14232 sec \\ \hline
**Model implementation** & [https://bit.ly/3lAz6yf](https://bit.ly/3lAz6yf) \\ \hline \end{tabular} 
\begin{tabular}{c c} \hline
**Hyperparameter** & **Search space** & **Best assignment** \\ \hline embedding & GloVe 300D & GloVe 300D \\ \hline embedding trainable & False & False \\ \hline number of epochs & 150 & 150 \\ \hline patience & 10 & 10 \\ \hline metric early stopping & Span F1/F1-i & Span F1/F1-i \\ monitored & & \\ \hline batch size & 32 & 32 \\ \hline dropout & _uniform-float[0, 0.5]_ & 0.5 \\ \hline
1st layer LSTM hidden dimension & _uniform-integer[30, 110]_ & 60 \\ \hline main task LSTM hidden & 50 & 50 \\ dimension & & \\ \hline skip connection between & True & True \\ embedding and main task layer & & \\ \hline learning rate optimiser & Adam & Adam \\ \hline learning rate & _loguniform-float[1e-4, 1e-2]_ & 1.5e-3 \\ \hline gradient norm & 5.0 & 5.0 \\ \hline regularisation type & L2 & L2 \\ \hline regularisation value & 1e-4 & 1e-4 \\ \hline \end{tabular}
\end{table}
Table 17: STL search space and best assignment using the Laptop dataset.





\begin{table}
\begin{tabular}{c c c} \hline
**GPU Infrastructure** & 1 GeForce GTX 1060 6GB GPU \\ \hline
**CPU Infrastructure** & AMD Ryzen 5 1600 CPU \\ \hline
**Number of search trials** & 30 \\ \hline
**Search strategy** & uniform sampling \\ \hline
**Best validation span F1/F1-i** & 0.6017 \\ \hline
**Training duration** & 18473 sec \\ \hline
**Model implementation** & [https://bit.ly/3lAz6yf](https://bit.ly/3lAz6yf) \\ \hline \end{tabular} 
\begin{tabular}{c c} \hline
**Hyperparameter** & **Search space** & **Best assignment** \\ \hline embedding & GloVe 300D & GloVe 300D \\ \hline embedding trainable & False & False \\ \hline number of epochs & 150 & 150 \\ \hline patience & 10 & 10 \\ \hline metric early stopping & Span F1/F1-i & Span F1/F1-i \\ monitored & \\ \hline batch size & 32 & 32 \\ \hline dropout & _uniform-float[0, 0.5]_ & 0.27 \\ \hline Shared/1st layer LSTM hidden dimension & _uniform-integer[30, 110]_ & 65 \\ \hline main task LSTM hidden dimension & 50 & 50 \\ \hline skip connection between & True & True \\ embedding and main task layer & & \\ \hline learning rate optimiser & Adam & Adam \\ \hline learning rate & _loguniform-float[1e-4, 1e-2]_ & 1.9e-3 \\ \hline gradient norm & 5.0 & 5.0 \\ \hline regularisation type & L2 & L2 \\ \hline regularisation value & 1e-4 & 1e-4 \\ \hline \end{tabular}
\end{table}
Table 18: MTL search space and best assignment using the Laptop dataset. The auxiliary task was negation detection using the Conan Doyle (NEG\({}_{CD}\)) dataset.





## 6 Conclusion

Figure 2: Hyperparameter budget against expected span F1/_F\({}_{1}\)-i_ performance for the STL and MTL models. The hyperparameter search space is stated within Tables 17 and 18 for the STL and MTL models respectively. The shaded areas represent the expected performance \(\pm 1\) standard deviation. Note the shaded area does not go beyond the maximum observed validation score as recommended Dodge et al. (2019).



Additional Reproducibility Statistics 



\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline  & \multirow{2}{*}{Model} & \multicolumn{4}{c|}{Number of Parameters} \\ \cline{3-5}  & & \multicolumn{3}{c|}{Including Auxiliary Task} \\ \cline{3-6}  & & \multicolumn{2}{c|}{Yes} & \multicolumn{2}{c|}{No} \\ \cline{3-7}  & & Trainable & All & Trainable & All \\ \hline \multirow{10}{*}{**Cooling**} & STL & 364,042 & 1,785,967 & 364,042 & 1,785,967 \\ \cline{2-6}  & NEG\({}_{CD}\) & 385,851 & 2,403,876 & 385,122 & 2,403,147 \\ \cline{2-6}  & NEG\({}_{SFU}\) & 385,851 & 7,066,176 & 385,122 & 7,065,447 \\ \cline{2-6}  & SPEC & 385,851 & 7,066,176 & 385,122 & 7,065,447 \\ \cline{2-6}  & UPOS & 388,413 & 2,952,738 & 385,122 & 2,949,447 \\ \cline{2-6}  & DR & 397,213 & 2,961,538 & 385,122 & 2,949,447 \\ \cline{2-6}  & LEX & 1,191,204 & 3,755,529 & 385,122 & 2,949,447 \\ \hline \multirow{10}{*}{**Finite**} & STL & 1,001,170 & 56,870,931 & 1,001,170 & 56,870,931 \\ \cline{2-6}  & NEG\({}_{CD}\) & 1,051,939 & 56,921,700 & 1,051,210 & 56,920,971 \\ \cline{2-6}  & NEG\({}_{SFU}\) & 1,051,939 & 56,921,700 & 1,051,210 & 56,920,971 \\ \cline{2-6}  & SPEC & 1,051,939 & 56,921,700 & 1,051,210 & 56,920,971 \\ \cline{2-6}  & UPOS & 1,054,501 & 56,924,262 & 1,051,210 & 56,920,971 \\ \cline{2-6}  & DR & 1,063,301 & 56,933,062 & 1,051,210 & 56,920,971 \\ \cline{2-6}  & LEX & 1,857,292 & 57,727,053 & 1,051,210 & 56,920,971 \\ \hline \end{tabular}
\end{table}
Table 19: Number of parameters for each model using different embeddings ordered by number of trainable parameters. The number of parameters is different for the MTL models depending on whether the parameters from the auxiliary task are included or not. The auxiliary task specific layer is shown as the pink layer in Figure 1. The number of parameters including and not including the auxiliary task is stated as the MTL models at inference time would not use the auxiliary task parameters. There are many more trainable parameters for the MTL models ignoring the auxiliary task parameters. This is because the hyperparameter search finds a larger shared LSTM hidden dimension to be preferable for the MTL models (see Tables 17 and 18). For the GloVe MTL models the total number of parameters changes depending on the auxiliary task. This is because the GloVe embeddings contain different numbers of vocabulary words, as we filter words based on those in the auxiliary and main task datasets/corpora. The large difference in the number of trainable parameters between GloVe and TL models is due to the fact that the TL is 724 parameters larger than the 300 parameter GloVe embeddings. Lastly, the number of trainable parameters is dataset agnostic, the number of all parameters is not dataset agnostic for the GloVe models due to the vocabulary size, for clarification the model parameters reported here are for those trained on the Laptop dataset.





\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline Embedding & Model & Device & Batch Size & Min Time (s) & Max Time (s) \\ \hline  & & \multirow{3}{*}{CPU} & 1 & 10.24 & 10.45 \\ \cline{6-7}  & & & 8 & 7.00 & 7.21 \\ \cline{3-7}  & & & 16 & 6.67 & 6.91 \\ \cline{3-7}  & & & 32 & 6.35 & 6.51 \\ \cline{3-7}  & & & 1 & 9.24 & 9.26 \\ \cline{3-7}  & & GPU & 8 & 6.58 & 6.67 \\ \cline{3-7} GloVe & & & 16 & 6.34 & 6.36 \\ \cline{3-7}  & & & 32 & 6.12 & 6.26 \\ \cline{3-7}  & & CPU & 1 & 10.06 & 10.26 \\ \cline{3-7}  & & & 8 & 7.05 & 7.19 \\ \cline{3-7}  & & & 16 & 6.90 & 6.99 \\ \cline{3-7}  & & & 32 & 6.41 & 6.46 \\ \cline{3-7}  & & & 1 & 9.43 & 9.49 \\ \cline{3-7}  & & GPU & 8 & 6.60 & 6.70 \\ \cline{3-7}  & & & 16 & 6.26 & 6.55 \\ \cline{3-7}  & & & 32 & 6.10 & 6.20 \\ \hline  & & \multirow{3}{*}{CPU} & 1 & 64.79 & 71.26 \\ \cline{3-7}  & & & 8 & 43.62 & 49.70 \\ \cline{3-7}  & & & 16 & 47.06 & 48.41 \\ \cline{3-7}  & & & 32 & 56.76 & 62.77 \\ \cline{3-7}  & & & 1 & 23.26 & 23.79 \\ \cline{3-7}  & & GPU & 8 & 8.82 & 9.09 \\ \cline{3-7} TL & & & 16 & 8.57 & 8.86 \\ \cline{3-7}  & & & 32 & 8.45 & 9.78 \\ \cline{3-7}  & & & 1 & 64.01 & 67.90 \\ \cline{3-7}  & & CPU & 8 & 49.05 & 50.00 \\ \cline{3-7}  & & & 16 & 53.47 & 56.42 \\ \cline{3-7}  & & & 32 & 55.33 & 55.79 \\ \cline{3-7}  & & & 1 & 23.81 & 23.97 \\ \cline{3-7}  & & GPU & 8 & 9.19 & 9.49 \\ \cline{3-7}  & & & 16 & 8.54 & 8.92 \\ \cline{3-7}  & & & 32 & 8.43 & 8.70 \\ \hline \end{tabular}
\end{table}
Table 20: Run/inference times for STL and MTL models that have been trained on the Laptop dataset using either GloVe or TL embeddings. Each model was timed in seconds (s) to generate predictions for 800 sentences, that were taken from the Laptop test split, of which this process was repeated five times and here we report the minimum (min) and maximum (max) time to generate predictions for those 800 sentences. We report these timings across different model configurations based on different batch sizes at prediction time and different devices. The trained MTL model used in this experiment was the MTL (NEG\({}_{SFU}\)) version, this was chosen as it contains the largest number of total parameters as shown in Table 19. Further all of these times were based on the model already loaded into memory and using the Python timeit library for timings. Additionally the GPU used was a GeForce GTX 1060 6GB GPU, CPU was an AMD Ryzen 5 1600 CPU, and the computer had 16GB of RAM.