Venue: NAACL 2018

# Multi-task Learning of Pairwise Sequence Classification Tasks

Over Disparate Label Spaces

Isabelle Augenstein1, Sebastian Ruder23, Anders Sogaard1

1Department of Computer Science, University of Copenhagen, Denmark

2Insight Research Centre, National University of Ireland, Galway

3Aylien Ltd., Dublin, Ireland {augenstein|soegaard}@di.ku.dk, sebastian@ruder.io

The first two authors contributed equally.

###### Abstract

We combine multi-task learning and semi-supervised learning by inducing a joint embedding space between disparate label spaces and learning transfer functions between label embeddings, enabling us to jointly leverage unlabelled data and auxiliary, annotated datasets. We evaluate our approach on a variety of sequence classification tasks with disparate labeled spaces. We outperform strong single and multi-task baselines and achieve a new state-of-the-art for topic-based sentiment analysis.

## 1 Introduction

Multi-task learning (MTL) and semi-supervised learning are both successful paradigms for learning in scenarios with limited labelled data and have in recent years been applied to almost all areas of NLP. Applications of MTL in NLP, for example, include partial parsing [4], text normalisation [5], neural machine translation [7], and keyphrase boundary classification [8].

Contemporary work in MTL for NLP typically focuses on learning representations that are useful across tasks, often through hard parameter sharing of hidden layers of neural networks [9, 6]. If tasks share optimal hypothesis classes at the level of these representations, MTL leads to improvements [1]. However, while sharing hidden layers of neural networks is an effective regulariser [6], we potentially _loose synergies between the classification functions_ trained to associate these representations with class labels. This paper sets out to build an architecture in which such synergies are exploited, with an application to pairwise sequence classification tasks. Doing so, we achieve a new state of the art on topic-based sentiment analysis.

For many NLP tasks, disparate label sets are weakly correlated, e.g. part-of-speech tags correlate with dependencies [10], sentiment correlates with emotion [12, 13], etc. We thus propose to induce a joint label embedding space (visualised in Figure 2) using a Label Embedding Layer that allows us to model these relationships, which we show helps with learning.

In addition, for tasks where labels are closely related, we should be able to not only model their relationship, but also to directly estimate the corresponding label of the target task based on auxiliary predictions. To this end, we propose to train a Label Transfer Network (LTN) jointly with the model to produce pseudo-labels across tasks.

The LTN can be used to label unlabelled and auxiliary task data by utilising the 'dark knowledge' [14] contained in auxiliary model predictions. This pseudo-labelled data is then incorporated into the model via semi-supervised learning, leading to a natural combination of multi-task learning and semi-supervised learning. We additionally augment the LTN with data-specific diversity features [15] that aid in learning.

ContributionsOur contributions are: a) We model the relationships between labels by inducing a joint label space for multi-task learning. b) We propose a Label Transfer Network that learns to transfer labels between tasks and propose to use semi-supervised learning to leverage them for training. c) We evaluate MTL approaches on a variety of classification tasks and shed new light on settings where multi-task learning works. d) We perform an extensive ablation study of our model.



e) We report state-of-the-art performance on topic-based sentiment analysis.

## 2 Related work

Learning task similaritiesExisting approaches for learning similarities between tasks enforce a clustering of tasks Evgeniou et al. (2005); Jacob et al. (2009), induce a shared prior Yu et al. (2005); Xue et al. (2007); Daume III (2009), or learn a grouping Kang et al. (2011); Kumar and Daume III (2012). These approaches focus on homogeneous tasks and employ linear or Bayesian models. They can thus not be directly applied to our setting with tasks using disparate label sets.

Multi-task learning with neural networksRecent work in multi-task learning goes beyond hard parameter sharing Caruana (1993) and considers different sharing structures, e.g. only sharing at lower layers Sogaard and Goldberg (2016) and induces private and shared subspaces Liu et al. (2017); Ruder et al. (2017). These approaches, however, are not able to take into account relationships between labels that may aid in learning. Another related direction is to train on disparate annotations of the same task Chen et al. (2016); Peng et al. (2017). In contrast, the different nature of our tasks requires a modelling of their label spaces.

Semi-supervised learningThere exists a wide range of semi-supervised learning algorithms, e.g., self-training, co-training, tri-training, EM, and combinations thereof, several of which have also been used in NLP. Our approach is probably most closely related to an algorithm called _codorest_Li and Zhou (2007). In co-forest, like here, each learner is improved with unlabeled instances labeled by the ensemble consisting of all the other learners. Note also that several researchers have proposed using auxiliary tasks that are unsupervised Plank et al. (2016); Rei (2017), which also leads to a form of semi-supervised models.

Label transformationsThe idea of manually mapping between label sets or learning such a mapping to facilitate transfer is not new. Zhang et al. (2012) use distributional information to map from a language-specific tagset to a tagset used for other languages, in order to facilitate cross-lingual transfer. More related to this work, Kim et al. (2015) use canonical correlation analysis to transfer between tasks with disparate label spaces. There has also been work on label transformations in the context of multi-label classification problems Yeh et al. (2017).

## 3 Multi-task learning with disparate label spaces

### Problem definition

In our multi-task learning scenario, we have access to labelled datasets for tasks \(T\) tasks \({\mathcal{T}}_{1},\ldots,{\mathcal{T}}_{T}\) at training time with a target task \({\mathcal{T}}_{T}\) that we particularly care about. The training dataset for task \({\mathcal{T}}_{i}\) consists of \(N_{k}\) examples \(X_{{\mathcal{T}}_{i}}=\{x^{{\mathcal{T}}_{i}}_{1},\ldots,x^{{\mathcal{T}}_{i}}_ {N_{k}}\}\) and their labels \(Y_{{\mathcal{T}}_{i}}=\{y^{{\mathcal{T}}_{i}}_{1},\ldots,y^{{\mathcal{T}}_{i}}_ {N_{k}}\}\). Our base model is a deep neural network that performs classic hard parameter sharing Caruana (1993): It shares its parameters across tasks and has task-specific softmax output layers, which output a probability distribution \({\mathbf{p}}^{{\mathcal{T}}_{i}}\) for task \({\mathcal{T}}_{i}\) according to the following equation:

\[{\mathbf{p}}^{{\mathcal{T}}_{i}}={\rm softmax}({\mathbf{W}}^{{\mathcal{T}}_{ i}}{\bf h}+{\mathbf{b}}^{{\mathcal{T}}_{i}}) \tag{1}\]

where \({\rm softmax}({\mathbf{x}})=e^{{\mathbf{x}}}/\sum_{i=1}^{|{\mathbf{x}}|}e^{{ \mathbf{x}}_{i}}\), \({\mathbf{W}}^{{\mathcal{T}}_{i}}\in{\mathbb{R}}^{L_{i}\times h}\), \({\mathbf{b}}^{{\mathcal{T}}_{i}}\in{\mathbb{R}}^{L_{i}}\) is the weight matrix and bias term of the output layer of task \({\mathcal{T}}_{i}\) respectively, \({\bf h}\in{\mathbb{R}}^{h}\) is the jointly learned hidden representation, \(L_{i}\) is the number of labels for task \({\mathcal{T}}_{i}\), and \(h\) is the dimensionality of \({\bf h}\).

The MTL model is then trained to minimise the sum of the individual task losses:

\[{\cal L}=\lambda_{1}{\cal L}_{1}+\ldots+\lambda_{T}{\cal L}_{T} \tag{2}\]

where \({\cal L}_{i}\) is the negative log-likelihood objective \({\cal L}_{i}={\cal H}({\mathbf{p}}^{{\mathcal{T}}_{i}},{\mathbf{y}}^{{ \mathcal{T}}_{i}})=-\frac{1}{N}\sum_{n}\sum_{j}\log{\mathbf{p}}^{{\mathcal{T}}_ {i}}_{j}{\mathbf{y}}^{{\mathcal{T}}_{i}}_{j}\) and \(\lambda_{i}\) is a parameter that determines the weight of task \({\mathcal{T}}_{i}\). In practice, we apply the same weight to all tasks. We show the full set-up in Figure 0(a).

### Label Embedding Layer

In order to learn the relationships between labels, we propose a Label Embedding Layer (LEL) that embeds the labels of all tasks in a joint space. Instead of training separate softmax output layers as above, we introduce a label compatibility function \(c(\cdot,\cdot)\) that measures how similar a label with embedding \({\bf l}\) is to the hidden representation \({\bf h}\):

\[c({\bf l},{\bf h})={\bf l}\cdot{\bf h} \tag{3}\]

where \(\cdot\) is the dot product. This is similar to the Universal Schema Latent Feature Model introduced by Riedel et al. (2013). In contrast to 

other models that use the dot product in the objective function, we do not have to rely on negative sampling and a hinge loss (Collobert and Weston, 2008) as negative instances (labels) are known. For efficiency purposes, we use matrix multiplication instead of a single dot product and softmax instead of sigmoid activations:

\[{\bf p}={\rm softmax}({\bf Lh}) \tag{4}\]

where \({\bf L}\in\mathbb{R}^{(\sum_{i}L_{i})\times l}\) is the label embedding matrix for all tasks and \(l\) is the dimensionality of the label embeddings. In practice, we set \(l\) to the hidden dimensionality \(h\). We use padding if \(l<h\). We apply a task-specific mask to \({\bf L}\) in order to obtain a task-specific probability distribution \({\bf p}^{\mathcal{T}_{l}}\). The LEL is shared across all tasks, which allows us to learn the relationships between the labels in the joint embedding space. We show MTL with the LEL in Figure 1b.

### Label Transfer Network

The LEL allows us to learn the relationships between labels. In order to make use of these relationships, we would like to leverage the predictions of our auxiliary tasks to estimate a label for the target task. To this end, we introduce the Label Transfer Network (LTN). This network takes the auxiliary task outputs as input. In particular, we define the output label embedding \({\mathbf{o}}_{i}\) of task \(\mathcal{T}_{i}\) as the sum of the task's label embeddings \({\mathbf{l}}_{j}\) weighted with their probability \({\bf p}^{\mathcal{T}_{i}}_{j}\):

\[{\mathbf{o}}_{i}=\sum_{j=1}^{L_{i}}{\bf p}^{\mathcal{T}_{i}}_{j}{\mathbf{l}}_ {j} \tag{5}\]

The label embeddings I encode general relationship between labels, while the model's probability distribution \({\bf p}^{\mathcal{T}_{i}}\) over its predictions encodes fine-grained information useful for learning (Hinton et al., 2015). The LTN is trained on labelled target task data. For each example, the corresponding label output embeddings of the auxiliary tasks are fed into a multi-layer perceptron (MLP), which is trained with a negative log-likelihood objective \(\mathcal{L}_{\mathrm{LTN}}\) to produce a pseudo-label \({\bf z}^{\mathcal{T}_{T}}\) for the target task \(\mathcal{T}_{T}\):

\[{\rm LTN}_{T}={\rm MLP}([{\mathbf{o}}_{1},\dots,{\mathbf{o}}_{T-1}]) \tag{6}\]

where \([\cdot,\cdot]\) designates concatenation. The mapping of the tasks in the LTN yields another signal that can be useful for optimisation and act as a regulariser. The LTN can also be seen as a mixture-of-experts layer (Jacobs et al., 1991) where the experts are the auxiliary task models. As the label embeddings are learned jointly with the main model, the LTN is more sensitive to the relationships between labels than a separately learned mixture-of-experts model that only relies on the experts' output distributions. As such, the LTN

Figure 1: a) Multi-task learning (MTL) with hard parameter sharing and 3 tasks \(\mathcal{T}_{1-3}\) and \(L_{1-3}\) labels per task. A shared representation \({\bf h}\) is used as input to task-specific softmax layers, which optimise cross-entropy losses \(\mathcal{L}_{1-3}\). b) MTL with the Label Embedding Layer (LEL) embeds task labels \({\mathbf{l}}^{7_{1-3}}_{1-L_{A}}\) in a joint embedding space and uses these for prediction with a label compatibility function. c) Semi-supervised MTL with the Label Transfer Network (LTN) in addition optimises an unsupervised loss \(\mathcal{L}_{pseudo}\) over pseudo-labels \({\bf z}^{\mathcal{T}_{T}}\) on auxiliary/unlabelled data. The pseudo-labels \({\bf z}^{\mathcal{T}_{T}}\) are produced by the LTN for the main task \(\mathcal{T}_{T}\) using the concatenation of auxiliary task label output embeddings \([{\mathbf{o}}_{i-1},{\mathbf{o}}_{i},{\mathbf{o}}_{i+1}]\) as input.

 

can be directly used to produce predictions on unseen data.

### Semi-supervised MTL

The downside of the LTN is that it requires additional parameters and relies on the predictions of the auxiliary models, which impacts the runtime during testing. Instead, of using the LTN for prediction directly, we can use it to provide pseudo-labels for unlabelled or auxiliary task data by utilising auxiliary predictions for semi-supervised learning.

We train the target task model on the pseudo-labelled data to minimise the squared error between the model predictions \({\bf p}^{\mathcal{T}_{i}}\) and the pseudo labels \({\bf z}^{\mathcal{T}_{i}}\) produced by the LTN:

\[\mathcal{L}_{pseudo}=MSE({\bf p}^{\mathcal{T}_{T}},{\bf z}^{\mathcal{T}_{T}}) =||{\bf p}^{\mathcal{T}_{T}}-{\bf z}^{\mathcal{T}_{T}}||^{2} \tag{7}\]

We add this loss term to the MTL loss in Equation 2. As the LTN is learned together with the MTL model, pseudo-labels produced early during training will likely not be helpful as they are based on unreliable auxiliary predictions. For this reason, we first train the base MTL model until convergence and then augment it with the LTN. We show the full semi-supervised learning procedure in Figure 0(c).

### Data-specific features

When there is a domain shift between the datasets of different tasks as is common for instance when learning NER models with different label sets, the output label embeddings might not contain sufficient information to bridge the domain gap.

To mitigate this discrepancy, we augment the LTN's input with features that have been found useful for transfer learning Ruder and Plank (2017). In particular, we use the number of word types, type-token ratio, entropy, Simpson's index, and Renyi entropy as diversity features. We calculate each feature for each example.1 The features are then concatenated with the input of the LTN.

Footnote 1: For more information regarding the feature calculation, refer to Ruder and Plank (2017).

### Other multi-task improvements

Hard parameter sharing can be overly restrictive and provide a regularisation that is too heavy when jointly learning many tasks. For this reason, we propose several additional improvements that seek to alleviate this burden: We use skip-connections, which have been shown to be useful for multi-task learning in recent work Ruder et al. (2017). Furthermore, we add a task-specific layer before the output layer, which is useful for learning task-specific transformations of the shared representations Sogaard and Goldberg (2016); Ruder et al. (2017).

## 4 Experiments

For our experiments, we evaluate on a wide range of text classification tasks. In particular, we choose pairwise classification tasks--i.e. those that condition the reading of one sequence on another sequence--as we are interested in understanding if knowledge can be transferred even for these more complex interactions. To the best of our knowledge, this is the first work on transfer learning between such pairwise sequence classification tasks. We implement all our models in Tensorflow Abadi et al. (2016) and release the code at [https://github.com/coastalcph/mtl-disparate](https://github.com/coastalcph/mtl-disparate).

### Tasks and datasets

We use the following tasks and datasets for our experiments, show task statistics in Table 1, and summarise examples in Table 2:

**Topic-based sentiment analysis** Topic-based sentiment analysis aims to estimate the sentiment of a tweet known to be about a given topic. We use the data from SemEval-2016 Task 4 Subtask B and C Nakov et al. (2016) for predicting on a two-point scale of positive and negative Topic-2) and five-point scale ranging from highly negative to highly positive Topic-5 respectively. An example from this dataset would be to classify the

\begin{table}
\begin{tabular}{p{39.8pt} c c c c} \hline \hline Task & Domain & \(N\) & \(L\) & Metric \\ \hline Topic-2 & Twitter & 4,346 & 2 & \(\rho^{PN}\) \\ Topic-5 & Twitter & 6,000 & 5 & \(MAE^{M}\) \\ Target & Twitter & 6,248 & 3 & \(F_{1}^{M}\) \\ Stance & Twitter & 2,914 & 3 & \(F_{1}^{BA}\) \\ ABSA-L & Reviews & 2,909 & 3 & \(Acc\) \\ ABSA-R & Reviews & 2,507 & 3 & \(Acc\) \\ FNC-I & News & 39,741 & 4 & \(Acc\) \\ MultiNLI & Diverse & 392,702 & 3 & \(Acc\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Training set statistics and evaluation metrics of every task. \(N\): # of examples. \(L\): # of labels.

 

tweet "No power at home, sat in the dark listening to AC/DC in the hope it'll make the electricity come back again" known to be about the topic "AC/DC", which is labelled as a positive sentiment. The evaluation metrics for Topic-2 and Topic-5 are macro-averaged recall (\(\rho^{PN}\)) and macro-averaged mean absolute error (\(MAE^{M}\)) respectively, which are both averaged across topics.

Target-dependent sentiment analysisTarget-dependent sentiment analysis (Target) seeks to classify the sentiment of a text's author towards an entity that occurs in the text as positive, negative, or neutral. We use the data from Dong et al. (2014). An example instance is the expression "how do you like settlers of catan for the wii?" which is labelled as neutral towards the target "wii'.' The evaluation metric is macro-averaged \(F_{1}\) (\(F_{1}^{M}\)).

Aspect-based sentiment analysisAspect-based sentiment analysis is the task of identifying whether an aspect, i.e. a particular property of an item is associated with a positive, negative, or neutral sentiment Ruder et al. (2016). We use the data of SemEval-2016 Task 5 Subtask 1 Slot 3 Pontiki et al. (2016) for the laptopps (ABSA-L) and restaurants (ABSA-R) domains. An example is the sentence "For the price, you cannot eat this well in Manhattan", labelled as positive towards both the aspects "restaurant prices" and "food quality". The evaluation metric for both domains is accuracy (\(Acc\)).

Stance detectionStance detection(Stance) requires a model, given a text and a target entity, which might not appear in the text, to predict whether the author of the text is in favour or against the target or whether neither inference is likely Augenstein et al. (2016). We use the data of SemEval-2016 Task 6 Subtask B Mohammad et al. (2016). An example from this dataset would be to predict the stance of the tweet "Be prepared - if we continue the policies of the liberal left, we will be #Greece" towards the topic "Donald Trump", labelled as "favor". The evaluation metric is the macro-averaged \(F_{1}\) score of the "favour" and "against" classes (\(F_{1}^{FA}\)).

Fake news detectionThe goal of fake news detection in the context of the Fake News Challenge2 is to estimate whether the body of a news article agrees, disagrees, discusses, or is unrelated towards a headline. We use the data from the first stage of the Fake News Challenge (FNC-1). An example for this dataset is the document "Dino Ferrari hooked the whopper wells catfish, (...), which could be the biggest in the world." with the headline "Fisherman lands 19 STONE catfish which could be the biggest in the world to be hooked" labelled as "agree". The evaluation metric is accuracy (\(Acc\))3.

Footnote 2: [http://www.fakenewschallenge.org/](http://www.fakenewschallenge.org/)

Natural language inferenceNatural language inference is the task of predicting whether one sentences entails, contradicts, or is neutral towards another one. We use the Multi-Genre NLI corpus (MultiNLI) from the RepEval 2017 shared task Nangia et al. (2017). An example for an instance would be the sentence pair "Fun for only children", "Fun for adults and children", which are in a "contradiction" relationship. The evaluation metric is accuracy (\(Acc\)).

\begin{table}
\begin{tabular}{|c|} \hline
**Topic-based sentiment analysis**: \\ _Tweet_: No power at home, sat in the dark listening \\ to AC/DC in the hope it’ll make the electricity come back again \\ _Topic_: AC/DC \\ _Label_: positive \\ \hline
**Target-dependent sentiment analysis**: \\ _Text_: how do you like settlers of catan for the wii? \\ _Target_: win \\ _Label_: neutral \\ \hline
**Aspect-based sentiment analysis**: \\ _Text_: For the price, you cannot eat this well in \\ Manhattan \\ _Aspects_: restaurant prices, food quality \\ _Label_: positive \\ \hline
* if we continue the policies of \\ the liberal left, we will be \#Greece \\ _Target_: Donald Trump \\ _Label_: favor \\ \hline
**Fake news detection**: \\ _Document_: Dino Ferrari hooked the whopper wells \\ catfish, (...), which could be the biggest in the world. \\ _Headline_: Fisherman lands 19 STONE catfish which \\ could be the biggest in the world to be hooked \\ _Label_: agree \\ \hline
**Natural language inference**: \\ _Premise_: Fun for only children \\ _Hypothesis_: Fun for adults and children \\ _Label_: contradiction \\ \hline \end{tabular}
\end{table}
Table 2: Example instances from the datasets described in Section 4.1.

 



### Base model

Our base model is the Bidirectional Encoding model (Augenstein et al., 2016), a state-of-the-art model for stance detection that conditions a bidirectional LSTM (BiLSTM) encoding of a text on the BiLSTM encoding of the target. Unlike Augenstein et al. (2016), we do not pre-train word embeddings on a larger set of unlabelled in-domain text for each task as we are mainly interested in exploring the benefit of multi-task learning for generalisation.

### Training settings

We use BiLSTMs with one hidden layer of \(100\) dimensions, \(100\)-dimensional randomly initialised word embeddings, a label embedding size of \(100\). We train our models with RMSProp, a learning rate of \(0.001\), a batch size of \(128\), and early stopping on the validation set of the main task with a patience of \(3\).

## 5 Results

Our main results are shown in Table 3, with a comparison against the state of the art. We present the results of our multi-task learning network with label embeddings (MTL + LEL), multi-task learning with label transfer (MTL + LEL + LTN), and the semi-supervised extension of this model. On 7/8 tasks, at least one of our architectures is better than single-task learning; and in 4/8, all our architectures are much better than single-task learning.

The state-of-the-art systems we compare against are often highly specialised, task-dependent architectures. Our architectures, in contrast, have not been optimised to compare favourably against the state of the art, as our main objective is to develop a novel approach to multi-task learning leveraging synergies between label sets and knowledge of marginal distributions from unlabeled data. For example, we do not use pre-trained word embeddings (Augenstein et al., 2016; Palogiannidi et al., 2016; Vo and Zhang, 2015), class weighting to deal with label imbalance (Balikas and Amini, 2016), or domain-specific sentiment lexicons (Brun et al., 2016; Kumar et al., 2016). Nevertheless, our approach outperforms the state-of-the-art on two-way topic-based sentiment analysis (Topic-2).

The poor performance compared to the state-of-the-art on FNC and MultiNLI is expected; as we alternate among the tasks during training, our model only sees a comparatively small number of examples of both corpora, which are one and two orders of magnitude larger than the other datasets. For this reason, we do not achieve good performance on these tasks as main tasks, but they are still useful as auxiliary tasks as seen in Table 4.

## 6 Analysis

### Label Embeddings

Our results above show that, indeed, modelling the similarity between tasks using label embeddings sometimes leads to much better performance. Figure 2 shows why. In Figure 2, we visualise the label embeddings of an MTL+LEL model trained on all tasks, using PCA. As we can see, similar labels are clustered together across tasks, e.g. there are two positive clusters (middle-right and top-right), two negative clusters (middle-left and bottom-left), and two neutral clusters (middle-top

\begin{table}
\begin{tabular}{l c c c c c c c} \hline  & Stance & FNC & MultiNLI & Topic-2 & Topic-S* & ABSA–L & ABSA-R & Target \\ \hline Augenstein et al. (2016) & **49.01** & - & - & - & - & - & - \\ Riedel et al. (2017) & - & **88.46** & - & - & - & - & - \\ Chen et al. (2017) & - & - & **74.90** & - & - & - & - \\ Palogiannidi et al. (2016) & - & - & - & 79.90 & - & - & - \\ Balikas and Amini (2016) & - & - & - & - & **0.719** & - & - \\ Brun et al. (2016) & - & - & - & - & - & **88.13** & - \\ Kumar et al. (2016) & - & - & - & - & - & **82.77** & 86.73 & - \\ Vo and Zhang (2015) & - & - & - & - & - & - & - & **69.90** \\ \hline STL & 41.1 & 72.72 & 49.25 & 63.92 & 0.919 & _76.74_ & 67.47 & 64.01 \\ \hline MTL + LEL & 46.26 & 72.71 & 49.94 & **80.52** & 0.814 & 74.94 & 79.90 & 66.42 \\ MTL + LEL + LTN, main model & 43.16 & 72.73 & 48.75 & 73.90 & 0.810 & 75.06 & 83.71 & 66.10 \\ MTL + LEL + LTN + semi, main model & 43.56 & 72.72 & 48.00 & 72.35 & 0.821 & 75.42 & 83.26 & 63.00 \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of our best performing models on the test set against a single task baseline and the state of the art, with task specific metrics. *: lower is better. Bold: best. Underlined: second-best.



and middle-bottom).

Our visualisation also provides us with a picture of what auxilary tasks are beneficial, and to what extent we can expect synergies from multi-task learning. For instance, the notion of positive sentiment appears to be very similar across the topic-based and aspect-based tasks, while the conceptions of negative and neutral sentiment differ. In addition, we can see that the model has failed to learn a relationship between MultiNLI labels and those of other tasks, possibly accounting for its poor performance on the inference task. We did not evaluate the correlation between label embeddings and task performance, but Bjerva (2017) recently suggested that mutual information of target and auxiliary task label sets is a good predictor of gains from multi-task learning.

### Auxiliary Tasks

For each task, we show the auxiliary tasks that achieved the best performance on the development data in Table 4. In contrast to most existing work, we did not restrict ourselves to performing multi-task learning with only one auxiliary task (Sogaard and Goldberg, 2016; Bingel and Sogaard, 2017). Indeed we find that most often a combination of auxiliary tasks achieves the best performance. In-domain tasks are less used than we assumed; only Target is consistently used by all Twitter main tasks. In addition, tasks with a higher number of labels, e.g. Topic-5 are used more often. Such tasks provide a more fine-grained reward signal, which may help in learning representations that generalise better. Finally, tasks with large amounts of training data such as FNC-1 and MultiNLI are also used more often. Even if not directly related, the larger amount of training data that can be indirectly leveraged via multi-task learning may help the model focus on relevant parts of the representation space (Caruana, 1993). These observations shed additional light on when multi-task learning may be useful that go beyond existing studies (Bingel and Sogaard, 2017).

### Ablation analysis

We now perform a detailed ablation analysis of our model, the results of which are shown in Table 5. We ablate whether to use the LEL (+ _LEL_), whether to use the LTN (+ _LTN_), whether to use the LEL output or the main model output for prediction (main model output is indicated by _, main model_), and whether to use the LTN as a regulariser or for semi-supervised learning (semi-supervised learning is indicated by _+ semi_). We further test whether to use diversity features (_-diversity feats_) and whether to use main model predictions for the LTN (_+ main model feats_).

Overall, the addition of the Label Embedding Layer improves the performance over regular MTL in almost all cases.

### Label transfer network

To understand the performance of the LTN, we analyse learning curves of the relabelling function vs. the main model. Examples for all tasks without semi-supervised learning are shown in Figure 3. One can observe that the relabelling model does not take long to converge as it has fewer parameters than the main model. Once the relabelling model is learned alongside the main

\begin{table}
\begin{tabular}{l l} \hline \hline Main task & Auxiliary tasks \\ \hline Topic-2 & FNC-1, MultiNLI, Target \\  & FNC-1, MultiNLI, ABSA-L, \\ Topic-5 & Target \\ Target & FNC-1, MultiNLI, Topic-5 \\ Stance & FNC-1, MultiNLI, Target \\ ABSA-L & Topic-5 \\ ABSA-R & Topic-5, ABSA-L, Target \\  & Stance, MultiNLI, Topic-5, \\ FNC-1 & ABSA-R, Target \\ MultiNLI & Topic-5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Best-performing auxiliary tasks for different main tasks.

Figure 2: Label embeddings of all tasks. Positive, negative, and neutral labels are clustered together.

 

model, the main model performance first stagnates, then starts to increase again. For some of the tasks, the main model ends up with a higher task score than the relabelling model. We hypothesise that the softmax predictions of other, even highly related tasks are less helpful for predicting main labels than the output layer of the main task model. At best, learning the relabelling model alongside the main model might act as a regulariser to the main model and thus improve the main model's performance over a baseline MTL model, as it is the case for TOPIC-5 (see Table 5).

To further analyse the performance of the LTN, we look into to what degree predictions of the main model and the relabelling model for individual instances are complementary to one another. Or, said differently, we measure the percentage of correct predictions made only by the relabelling model or made only by the main model, relative to the number of correct predictions overall. Results of this for each task are shown in Table 6 for the LTN with and without semi-supervised learning. One can observe that, even though the relabelling function overall contributes to the score to a lesser degree than the main model, a substantial number of correct predictions are made by the relabelling function that are missed by the main model. This is most prominently pronounced for ABSA-R, where the proportion is 14.6.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Task & Main & LTN & Main (Semi) & LTN (Semi) \\ \hline Stance & 2.12 & 2.62 & 1.94 & 1.28 \\ FNC & 4.28 & 2.49 & 6.92 & 4.84 \\ MultiNLI & 1.5 & 1.95 & 1.94 & 1.28 \\ Topic-2 & 6.45 & 4.44 & 5.87 & 5.59 \\ Topic-5* & 9.22 & 9.71 & 11.3 & 5.90 \\ ABSA-L & 3.79 & 2.52 & 9.06 & 6.63 \\ ABSA-R & 10.6 & 6.70 & 9.06 & 6.63 \\ Target & 26.3 & 14.6 & 20.1 & 15.7 \\ \hline \end{tabular}
\end{table}
Table 6: Error analysis of LTN with and without semi-supervised learning for all tasks. Metric shown: percentage of correct predictions only made by either the relabelling function or the main model, respectively, relative to the the number of all correct predictions.

Figure 3: Learning curves with LTN for selected tasks, dev performances shown. The main model is pre-trained for 10 epochs, after which the relabelling function is trained.

\begin{table}
\begin{tabular}{l r r r r r r r} \hline \hline  & \multicolumn{2}{c}{Stance} & \multicolumn{2}{c}{FNC} & \multicolumn{2}{c}{MultiNLI} & \multicolumn{2}{c}{Topic-2 Topic-5* ABSA-L ABSA-R Target} \\ \hline MTL & 44.12 & 72.75 & 49.39 & **80.74** & 0.859 & 74.94 & 82.25 & 65.73 \\ \hline MTL + LEL & **46.26** & 72.71 & **49.94** & 80.52 & 0.814 & 74.94 & 79.90 & **66.42** \\ MTL + LTI + LTN & 40.95 & 72.72 & 44.14 & 78.31 & 0.851 & 73.98 & 82.37 & 63.71 \\ MTL + LTLN, main model & 41.60 & 72.72 & 47.62 & 79.98 & 0.814 & 75.54 & 81.70 & 65.61 \\ MTL + LEL + LTN & 44.8 & **72.76** & 43.72 & 74.07 & 0.821 & **75.66** & 81.92 & 65.00 \\ \hline MTL + LEL + LTN + main model & 43.16 & 72.73 & 48.75 & 73.90 & 0.810 & 75.06 & **83.71** & 66.10 \\ \hline MTL + LEL + LTN + main prods feats & 42.78 & 72.72 & 45.41 & 66.30 & 0.835 & 73.86 & 81.81 & 65.08 \\ \hline MTL + LEL + LTN + main prods feats - drivenk & 42.65 & 72.73 & 48.81 & 67.53 & **0.803** & 75.18 & 82.59 & 63.95 \\ \hline MTL + LEL + LTN + main prods feats – diversity feats & 42.78 & 72.72 & 43.13 & 66.3 & 0.835 & 73.5 & 81.7 & 63.95 \\ MTL + LEL + LTN + main prods feats – diversity feats, main model & 42.47 & 72.74 & 47.84 & 67.53 & 0.807 & 74.82 & 82.14 & 65.11 \\ \hline MTL + LEL + LTN + semi & 42.65 & 72.75 & 44.28 & 77.81 & 0.841 & 74.10 & 81.36 & 64.45 \\ MTI + LEL + LTN + semi, main model & 43.56 & 72.72 & 48.00 & 72.35 & 0.821 & 75.42 & 83.26 & 63.00 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation results with task-specific evaluation metrics on test set with early stopping on dev set. _LTN_ means the output of the relabelling function is shown, which does not use the task predictions, only predictions from other tasks. _LTN + main preds feats_ means main model predictions are used as features for the relabelling function. _LTN, main model_ means that the main model predictions of the model that trains a relabelling function are used. Note that for MultiNLI, we down-sample the training data. *: lower is better. Bold: best. Underlined: second-best.

 

Conclusion

We have presented a multi-task learning architecture that (i) leverages potential synergies between classifier functions relating shared representations with disparate label spaces and (ii) enables learning from mixtures of labeled and unlabeled data. We have presented experiments with combinations of eight pairwise sequence classification tasks. Our results show that leveraging synergies between label spaces sometimes leads to big improvements, and we have presented a new state of the art for topic-based sentiment analysis. Our analysis further showed that (a) the learned label embeddings were indicative of gains from multi-task learning, (b) auxiliary tasks were often beneficial across domains, and (c) label embeddings almost always led to better performance. We also investigated the dynamics of the label transfer network we use for exploiting the synergies between disparate label spaces.

## Acknowledgments

Sebastian Ruder is supported by the Irish Research Council Grant Number EBPPG/2014/30 and Science Foundation Ireland Grant Number SFI/12/RC/2289. Anders Sogaard is supported by the ERC Starting Grant Number 313695. Isabelle Augenstein is supported by Eurostars grant Number E10138. We further gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

## References

* Abadi et al. (2016) Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. _arXiv preprint arXiv:1603.04467_ .
* Augenstein et al. (2016) Isabelle Augenstein, Tim Rocktaschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Twitter Stance Detection with Bidirectional Conditional Encoding. In _Proceedings of EMNLP_.
* Augenstein and Sogaard (2017) Isabelle Augenstein and Anders Sogaard. 2017. Multi-task learning of keyphrase boundary detection. In _Proceedings of ACL_.
* Balikas and Amini (2016) Georgios Balikas and Massih-Reza Amini. 2016. TwiSE at SemEval-2016 Task 4: Twitter Sentiment Classification. In _Proceedings of SemEval_.
* Baxter (2000) Jonathan Baxter. 2000. A Model of Inductive Bias Learning. _JAIR_ 12:149-198.
* Bingel and Sogaard (2017) Joachim Bingel and Anders Sogaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In _Proceedings of EACL_.
* Bjerva (2017) Johannes Bjerva. 2017. Will my auxiliary tagging task help? Estimating Auxiliary Tasks Effectivity in Multi-Task Learning. In _Proceedings of NODALIDA_.
* Bollman et al. (2017) Marcel Bollman, Joachim Bingel, and Anders Sogaard. 2017. Learning attention for historical text normalization by learning to pronounce. In _Proceedings of ACL_.
* Brun et al. (2016) Caroline Brun, Julien Perez, and Claude Roux. 2016. XRCE at SemEval-2016 Task 5: Feedback Ensemble Modelling on Syntactico-Semantic Knowledge for Aspect Based Sentiment Analysis. _Proceedings of SemEval_ .
* Caruana (1993) Rich Caruana. 1993. Multitask Learning: A Knowledge-Based Source of Inductive Bias. In _Proceedings of ICML_.
* Chen et al. (2016) Hongshen Chen, Yue Zhang, and Qun Liu. 2016. Neural Network for Heterogeneous Annotations. In _Proceedings of EMNLP_.
* Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Recurrent neural network-based sentence encoder with gated attention for natural language inference. _arXiv preprint arXiv:1708.01353_ .
* Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In _Proceedings of ICML_.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _The Journal of Machine Learning Research_ 12:2493-2537.
* Daume III (2009) Hal Daume III. 2009. Bayesian multitask learning with latent hierarchies. In _Proceedings of UAI_.
* Dong et al. (2014) Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification. In _Proceedings of ACL_. pages 49-54.
* Eisner et al. (2016) Ben Eisner, Tim Rocktaschel, Isabelle Augenstein, Matko Bosnjak, and Sebastian Riedel. 2016. emoji2vec: Learning Emoji Representations from their Description. In _Proceedings of SocialNLP_.
* Evgeniou et al. (2005) Theodoros Evgeniou, Charles A. Micchelli, and Massimiliano Pontil. 2005. Learning multiple tasks with kernel methods. _Journal of Machine Learning Research_ 6:615-637.
* Felbo et al. (2017) Bjarke Felbo, Alan Mislove, Anders Sogaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoj occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In _Proceedings of EMNLP_ 

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks. In _Proceedings of EMNLP_.
* Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. _arXiv preprint arXiv:1503.02531_ .
* Jacob et al. (2009) Laurent Jacob, Jean-Philippe Vert, Francis R Bach, and Jean-philippe Vert. 2009. Clustered Multi-Task Learning: A Convex Formulation. In _Proceedings of NIPS_. pages 745-752.
* Jacobs et al. (1991) Robert a. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive Mixtures of Local Experts. _Neural Computation_ 3(1):79-87.
* Kang et al. (2011) Zhuoliang Kang, Kristen Grauman, and Fei Sha. 2011. Learning with Whom to Share in Multi-task Feature Learning. In _Proceedings of ICML_.
* Kim et al. (2015) Young-Bum Kim, Karl Stratos, Ruhi Sarikaya, and Minwoo Jeong. 2015. New Transfer Learning Techniques for Disparate Label Sets. In _Proceedings of ACL_.
* Kumar and Daume III (2012) Abhishek Kumar and Hal Daume III. 2012. Learning Task Grouping and Overlap in Multi-task Learning. _Proceedings of the 29th International Conference on Machine Learning_ pages 1383-1390.
* Kumar et al. (2016) Ayush Kumar, Sarah Kohail, Amit Kumar, Asif Ekbal, and Chris Biemann. 2016. IIT-TUDA at SemEval-2016 Task 5: Beyond Sentiment Lexicon: Combining Domain Dependency and Distributional Semantics Features for Aspect Based Sentiment Analysis. _Proceedings of SemEval_ .
* Li and Zhou (2007) Ming Li and Zhi-Hua Zhou. 2007. Improve Computer-Aided Diagnosis With Machine Learning Techniques Using Undiagnosed Samples. _IEEE Transactions on Systems, Man and Cybernetics_ 37(6):1088-1098.
* Liu et al. (2017) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial Multi-task Learning for Text Classification. In _Proceedings of ACL_.
* Luong et al. (2016) Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task Sequence to Sequence Learning. In _Proceedings of ICLR_.
* Mohammad et al. (2016) Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In _Proceedings of SemEval_.
* Nakov et al. (2016) Preslav Nakov, Alan Ritter, Sara Rosenthal, Veselin Stoyanov, and Fabrizio Sebastiani. 2016. SemEval-2016 Task 4: Sentiment Analysis in Twitter. In _Proceedings of SemEval_. San Diego, California.
* Nangia et al. (2017) Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel R. Bowman. 2017. The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations. In _Proceedings of RepEval_.
* Palogiannidi et al. (2016) Elisavet Palogiannidi, Athanasa Kolovou, Fenia Christopoulou, Filippos Kokkinos, Elias Iosif, Nikolaos Malandrakis, Haris Papageorgiou, Shrikanth Narayanan, and Alexandros Potamianos. 2016. Tweester at SemEval-2016 Task 4: Sentiment Analysis in Twitter Using Semantic-Affective Model Adaptation. In _Proceedings of SemEval_. pages 155-163.
* Peng et al. (2017) Hao Peng, Sam Thomson, Noah A Smith, and Paul G Allen. 2017. Deep Multitask Learning for Semantic Dependency Parsing. In _Proceedings of ACL 2017_.
* Plank et al. (2016) Barbara Plank, Anders Sogaard, and Yoav Goldberg. 2016. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. In _Proceedings of ACL_.
* Pontik et al. (2016) Maria Pontik, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammed AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphee De Clercq, Veronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud Maria Jimenez-Zafra, and Gulsen Erygit. 2016. SemEval-2016 Task 5: Aspect Based Sentiment Analysis. In _Proceedings of SemEval_.
* Rei (2017) Marek Rei. 2017. Semi-supervised Multitask Learning for Sequence Labeling. In _Proceedings of ACL 2017_.
* Riedel et al. (2017) Benjamin Riedel, Isabelle Augenstein, Georgios P Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. In _arXiv preprint arXiv:1707.03264_.
* Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation Extraction with Matrix Factorization and Universal Schemas. _Proceedings of NAACL-HLT_ pages 74-84.
* Ruder et al. (2017) Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Sogaard. 2017. Sluice networks: Learning what to share between loosely related tasks. In _CoRR, abs/1705.08142_.
* Ruder et al. (2016) Sebastian Ruder, Parsa Ghaffari, and John G. Breslin. 2016. A Hierarchical Model of Reviews for Aspect-based Sentiment Analysis. _Proceedings of EMNLP_ pages 999-1005.
* Ruder and Plank (2017) Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with Bayesian Optimization. In _Proceedings of EMNLP_.
* Sogaard and Goldberg (2016) Anders Sogaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In _Proceedings of ACL_ 

* Vo and Zhang (2015) Duy-Tin Vo and Yue Zhang. 2015. Target-Dependent Twitter Sentiment Classification with Rich Automatic Features. In _Proceedings of IJCAI_. pages 1347-1353.
* Xue et al. (2007) Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. 2007. Multi-Task Learning for Classification with Dirichlet Process Priors. _Journal of Machine Learning Research_ 8:35-63.
* Yeh et al. (2017) Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. 2017. Learning Deep Latent Space for Multi-Label Classification. In _Proceedings of AAAI_.
* Yu et al. (2005) Kai Yu, Volker Tresp, and Anton Schwaighofer. 2005. Learning Gaussian processes from multiple tasks. _Proceedings of ICML_ 22:1012-1019.
* Zhang et al. (2012) Yuan Zhang, Roi Reichart, Regina Barzilay, and Amir Globerson. 2012. Learning to Map into a Universal POS Tagset. In _Proceedings of EMNLP_.