Frustratingly easy quasi-multitask learning
Authors: Gábor Berend
Norbert Kis-Szabó
Venue:  (2020)
Keywords: multitask learning
ensembling

Abstract: We propose the technique of quasi-multitask learning (Q-MTL), a simple and easy to implement modification of standard multitask learning, in which the  tasks to be modeled are identical. We illustrate it through a series of sequence labeling experiments over a diverse set of languages, that applying Q-MTL consistently increases the generalization ability of the applied models. The proposed architecture can be regarded as a new regularization technique encouraging the model to develop an internal representation of the problem at hand that is beneficial to multiple output units of the classifier at the same time. This property hampers the convergence to such internal representations which are highly specific and tailored for a classifier with a particular set of parameters. Our experiments corroborate that by relying on the proposed algorithm, we can approximate the quality of an ensemble of classifiers at a fraction of computational resources required. Additionally, our results suggest that Q-MTL handles the presence of noisy training labels better than ensembles.


Under review as a conference paper at ICLR 2020
FRUSTRATINGLY EASY QUASI-MULTITASK LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
We propose the technique of quasi-multitask learning (Q-MTL), simple and easy to implement modification of standard multitask learning, in which the tasks to be modeled are identical. We illustrate it through series of sequence labeling experi- ments over diverse set of languages, that applying Q-MTL consistently increases the generalization ability of the applied models. The proposed architecture can be regarded as new regularization technique encouraging the model to develop an internal representation of the problem hand that beneficial to multiple output units of the classifier at the same time. This property hampers the convergence to such internal representations which are highly specific and tailored for a classifier with particular set of parameters. Our experiments corroborate that by relying on the proposed algorithm, we can approximate the quality of an ensemble of classifiers at fraction of computational resources required. Additionally, our re- sults suggest that Q-MTL handles the presence of noisy training labels better than ensembles.
INTRODUCTION
Ensemble methods are frequently used in machine learning applications due to their tendency of increasing model performance. While the increase the prediction performance undoubtedly an important aspect when we train a model, it should not be forgotten that the increased performance of ensembling comes at the price of training multiple models for solving the same task.
The question that we tackle in this paper is the following: Can enjoy the benefits of ensemble learning, while avoiding its overhead for training models from from scratch multiple times? This question is highly relevant these days, since state-of-the- -he--trr-rt neural models tend to be extremely resource- intensive on their own (Strubell et al. 2019), prohibiting their inclusion in a traditional ensemble setting.
Our proposed architecture simultaneously offers the benefit of ensemble learning, while avoiding its drawback of training multiple models. The method introduced here employs special form of multitask learning (MTL). Caruana 1997) argues in his seminal work that MTL can be a useful source of introducing inductive bias machine learning models. Standard MTL have been shown to be fruitfully applicable in solving series of NLP tasks: Collobert & Weston (2008) Plank et al. (016); Rei (2017); Kiperwasser & Ballesteros 2018); [anh et al.) (2018), inter alia. We introduce quasi-multitask learning (Q-MTL), where the goal is to simultaneously learn multiple neural models that solve identical tasks, while relying on shared representation layer.
Besides the considerable speedup that comes with the proposed technique, we additionally argue that by applying multiple output units on top shared parameter set is beneficial, as we can avoid converging such degenerate internal representations that are highly tailored for particular classification model. In that sense, Q-MTL can also be viewed as an implicit regularizer, which prevents neural networks to develop such an internal representation which is not generic enough to provide useful input to multiple classification units simultancously.
Our experiments with Q-MTL illustrate that the presence of multiple classifier layers for the same task affect each other positively- - similar to ensemble learning without the additional overhead of actually training multiple models.
A similar technique have already been derived from MTL called Pseudo-Tase Augmentation MMyy- erson & Miikkulainen 2018), which builds on the idea of common representation, but the manage-

---

Under review as a conference paper at ICLR 2020
d
Σ
Noun
Verbh
Noun
Noun
Verbb
Noun
Noun
Verb
Noun
Retl
Rell
ReLLi
Aelli
ReLUi
A
BILSTM
BILSTM
BILSTM
BiLSTM
BILSTH
BLSTH
BILSTM
A
coffee
coffee
coffee
coffee
coffee
coffee
coffee
(a) Sequence of STLS
Q-MTL
Ensemble
Figure 1: A schematic illustration of the different architectures employed in experiments. Quasi- Multitask Learning (Q-MTL) averages the predictions multiple classification units similar to en- sembling without the computational bottleneck of adjusting the parameters of multiple LSTM cells.
ment of these tasks differs. We conducted experiments comparing the two methods for a greater comprehension of the differences.
APPLIED MODELS
We release all our source code used for our experiments at anonymized Our models are based on the sequence classification framework from Plank et al. (2016) implemented in DyNet (Neubig al. 2017). Figure provides a visual summary of the different architectures we implemented. Figure 1b highhlhiss that Q-MTL has the benefit of training multiple classification models over the same internal representation, as opposed to traditional ensemble model, which requires training of multiple LSTM parameters as well (cf. Figure Tc..
2.1 BASELINE ARCHITECTURE
Our baseline classifier is a bidirectional LSTM (Hochreiter & Schmidhuber 1997) incorporating character and word level embeddings. We first compute the input embedding for the network at position as
CT
where ① is the concatenation operator, w; denotes the word embedding, с and & refers to the left-to-right and right-to-left Staracter-based embeddings, respectively. We subsequently feed e; into a bi-LSTM, which determines a hidden representation h; E Rm for every token position as h; h h, i.c., the concatenation of the hidden states of the two LSTMs processing the input from its beginning to the end, and in reverse direction.
The final output of the network for token position gets computed
Yi softmar by) -bw)
with V. hoom and by E 2h denoting the weight matrix and the bias of perceptron unit, whereas W Raxe and bw E I are the parameters of the neuron performing classification over the target classes.
2

---

Under review as a conference paper at ICLR 2020
2.2 Q-MTL ARCHITECTURE
The Q-MTL network behaves similarly to the model introduced in Section 2.1 with the notable exception that it trains k distinct classification models, all of which operate over the same hidden representation as input obtained from a single bi-LSTM unit.
More concretely, we replace the single prediction of the STL model from Eq. I by a series of predictions for Q-MTL according to
Yij softmar (RetmaaRRRLLVVaaaa by WOO +bW),
(2)
with j E {1...., .......,)..... As argued before, this approach behaves favorably from a computational point of view, it relies on shared representation h; for all the & classification units.
The loss of the network for token position and gold standard class label y" can be conveniently generalized
(T-MTL(1) Σ viJy
where CE denotes categorical cross entropy loss and the number of (identical) tasks in the Q-MTL model, with the special case of l = 1 resulting in standard single task learning (STL).
Losses from the different outputs can be aggregated efficiently during backpropagation, hence the shared LSTM cell benefit from multiple error signals without the actual need of going through multiple individual forward and backward passes.
Q-MTL outputs & predictions by all of its prediction units, however, we can as well derive a com- bined prediction from the distinet outputs of Q-MTI. according to
(B b W
(
k
which essentially a weighted average according the predicted probabilities of the distinct mod- els. As introducing averaging at the model-level would eliminate diversity of the individual classi- fiers (Lee et al. 2015), this kind of averaging took place in a post-hoc manner, only when making predictions.
2.3 TRADITIONAL ENSEMBLE MODEL
As an additional model, we also employ a traditional ensemble of & independently trained STL mod- els. We define the prediction of the ensemble model by averaging the predictions of k independent models as
(1) (Re ()) bby W b
(4)
The distinctive difference between Eq. 4 and the Q-MTL model formulation in Eq. 3 that ensem- bling relies on the hidden representations originating from independently trained LSTM models as denoted by the superscripts of the hidden states in h Such an ensemble necessarily requires approximately k-times as much computational resources compared to Q-MTL, due to the LSTM models being trained in total isolation. For the above reason, ensembling is strictly more expen- sive form of training a model, for which reason we regard its performance as a glass ceiling for Q-MTL.
EXPERIMENTS
Our model uses character embeddings of 100 dimensions and the word representations get initialized by the 64-dimensional pre-trained polyglot word emheddings (AI-Rfou et al.) 2013) as suggested by Plank & Agié (2018). We one-layered bi-L.TMM which outputs hidden vectors h; E 22000 as a concatenation of h, h E 100 Instead of directly applying a fully-conmected layer to perform classification based h;, first transform h; by intermediate perceptron unit with ReLU
3

---

Under review as a conference paper at ICLR 2020
Table 1: Statistics training data size.
el
en
eu
hr
hu
id
n1
ta
# sentences
1662
2738
5369
14980
6983
910
4477
12269
400
3685
word forms
9035
7436
19222
39717
33382
7767
19223
266665
2637
13781
# total words
42326
50096
72974
127602
154055
20166
97531
186046
6329
38082
Table 2: Results of Q-MTL on the dev sets for varying number of tasks employed (k).
k:
el
en
eu
fi
hr
hu
id
nl
ta
tr
Avg.
1
95.61
94.99
94.49
93.19
96.84
93.95
93.05
96.05
82.74
93.61
93,45
10
95.84
95.23
94.81
93.30
96.99
94.25
92.98
96.53
84.48
93.78
93.82
30
95.86
95.21
94.59
93.09
96.93
93.79
93.25
96.27
83.85
93.46
93.63
activation. The perceptron transforms h; into 20 dimensions, that is, we have V E 20x Our motivation with the extra non-linearity introduced by ReLU encourage increased diversity in the behavior of the different output units.
Upon training the LSTMs, used the default architectural settings employed by Plank al. (2016), i.c. we relied on a word dropout rate of 0.25 (Kiperwasser & Goldberg 2016) and an additive Gaussian noise (with o 0.2) over the input embeddings. We trained all our models for 20 epochs using stochastic gradient descent with batch size of 1. First, we assess the quality of Q-MTL towards POS tagging, then we evaluate on named entity recognition well.
When comparing the performance of different approaches, Q-MTL models are compared against the average performance of l: STL models, where denotes the number of task the case of Q-MTI. The & STL models are also used to derive single prediction by the ensemble model.
3.1 POS TAGGING EXPERIMENTS
We set our POS tagging related experiments on 10 treebanks from the Universal Dependencies dataset v2.2 (Nivre et al. 2018), namely the Greek-GDT (el), English-LinFSS (en), Basque-BITT (eu), Finnish-FTB (fi), Croatian-SFT (hr), Hungarian-Szeged (hu), Indonesian-GSD (id), Dutch- Alpino (nl), Tamil-TTB (ta) and Turkish-IMST (tr) treebanks. These treebanks not only cover typologically diverse set of languages, but they also vary substantially in the number of available training sequences, illustrated in Table [ Table [ also illustrates the typological diversity of the investigated languages, as the average number of occurrences per distinct word forms vary substan- tially, i.e. between 2.4 for Tamil and 6.9 for Dutch.
3.1.1 EXPERIMENTS WITH THE NUMBER OF TASKS
We first investigate how does changing the value for k, i.c., the number of simultancously learned tasks, affects the performance of Q-MTL. We experimented with & E {1, 10, 30). Based on the re- sults in T2 we set the number of tasks to be employed as for all upcoming experiments. In order to choose k without overfitting to the training data, this experiment was conducted on the development set.
3.1.2 COMPARING Q-MTL WITH STL
Following the recommendation in (Dodge al., 2019) report learning curves over the develop- ment set as function of the number epochs in Figure As general observation, we can see that Q-MTL tends to perform consistently better than STL models right from the beginning of training.
Directly comparing the classifiers One benefit of Q-MTL is that it learns A different classifica- tion models during training with only a marginal computational overhead compared to training a STL baseline, since all the tasks share common internal representation. As discussed earlier, can combine the predictions from the & classifiers from Q-MTL according to Eq. [ is also pos- sible, however, to the l distinct predictions of Q-MTL. In what follows next, compare the
4

---

Under review as a conference paper at ICLR 2020
97
Accuracy (%6
95.0
94
92
90
96
90.0
90
85
95
hu
id
ta
tr
93
96
80
90
Accuracy
85
70
85
60
0
10
20
0
10
20
10
20
10
20
0
10
20
Number epochs
Number epochs
Number of epochs
Number epochs
Number epochs
Figure The accuracy of the different model types over the training epochs on the dev set.
performance of the STL models train to the & classifiers that are incorporated within Q-MTL model.
Upon comparing the performance of Q-MTL classifier with STL model, made it sure that the overlapping parameters (matrices V and W) were initialized with the same values and that they receive the training instances in the exact same order. This way the performance achieved by the ith output of Q-MTL is directly comparable with the jt STL baseline. Comparison of the results of the individual outputs of Q-MTL and their corresponding STL counterpart are included in Figure
dataset
Cast
dev
en
ou
hr
97
90.88
94.6
<80
94.4
08 AVP
85
94.25 94.50 94.75
93.0
97.0
Id
ta
tr
QJ3TT accurary (96)
<B,
93.50
96.5
96.0
86
93.25
99.5
93.00
84
93.5
05.5
2
92.5
82
93.0
93
94
93.5 STL accuracy (%)
93.0
95.5 96.0 96.5
82.5
85.0
87.5
93
94
STL accuracy (%6
STL accuracy (%6
STL
accuracy (%6
STL accuracy (%)
Figure 3: Scatter plot comparing the accuracy of the individual classifiers from Q-MTL (k = 10) and their corresponding STL counterpart. Each model that is above the diagonal line performs better after training in the Q-MTL setting.
Training Q-MTL models with & tasks simultaneously is not only faster than training distinct STL models separately, but the individual Q-MTL models typically outperform their baseline counter- parts evaluated against both the development and the test data.
The regularizing effect of Q-MTL We have argued earlier that Q-MTL has an implicit regular- izing effect. Among most recent techniques, such dropout (Srivastava al.) 2014), weight decay (Krogh & Hertz 1992) is one of the most typical form of regularization for fostering the gener- alization capability of the learned models. When employing weight decay, add extra term penalizing the magnitude of the values learned by our model, which results in an overall shrinkage in the values of the model parameters.
5

---

Under review as a conference paper at ICLR 2020
W
Frobenius norm
20
10
hu
nl
ta
tr
Frobenius norm
10
STL Q-MTL
STL
Q-MTL
STL
Q-MTL
STL
Q-MTL
STL
Q-MTL
Figure 4: The average Frobenius norms of the learned parameter matrices and W for the different approaches and treebanks.
Figure iiluustttss that the effects of employing Q-MTL is similar to applying weight decay, as the Frobenius norm the parameter matrices from the classifiers of Q-MTL are substantially smaller than those of the STL classifiers. This observation holds for both the of parameter sets V and W. Recall that the initial values for these matrices were identical for both Q-MTL and STL.
3.1.3 COMPARISON TO AN ENSEMBLE OF CLASSIFIERS
have provided a detailed comparison of the STL and Q-MTL models so far. We next extend their comparative evaluation with the ensemble model. Upon providing comparison for the different approaches, we also assess their sensitivity towards the presence of noisily labeled tokens during training. To do replaced the true class label of token by some predefined probability E {0, 0.1, 0.2, 0.3). During the random replacement of the class labels, ensured that the same tokens got randomly relabeled by the same label for the different approaches.
conducted multiple experiments for each language, for which randomly
Figure 5 contains the performance of the three different models in conjunction with the different amounts of noisy labels introduced to the training set. We can observe from Figure that Q-MTI. outperforms STL for all the languages irrespective to the amount of noisy tokens being present countered during training. Figure further reveals that the performances of the ensemble models which are based on the predictions of the STL classifiers are dominantly better than the average performance of the individual STL models. When mislabeled tokens are not present in the training data all, ensemble also has a slight advantage over Q-MLT, however, this advantage of the ensem- bling model gradually fades out as the proportion of noisy training labels increases. Indeed, for the performance of Q-MTL reaches
case when 30% the training labels are randomly replaced, that of the ensemble model (cf. the rightmost subplot in Figure 5). The Q-MTL approach has the additional benefit over the ensemble model that requires a fraction of computational resources as will demonstrate it in Section 3.2
As final interesting note, the Q-MTL has an improved performance for Indonesian as the amount of noisy training labels increases. A possible explanation for this is that corrupting the class labels of the training data can be viewed as an alternative form label smoothing (Szegedy et al. 2016), which is known increase the generalization ability of neural models.
3.2 COMPARISON OF TRAINING TIMES
One of the main benefits of Q-MTL resides in its training efficiency compared traditional semble models as also demonstrated by Figure which includes the training times for the different approaches. We plot the training times the logarithmic scale for better readability for both & 10

---

Under review as a conference paper at ICLR 2020
and k = 30. We can that the training times for STL and Q-MTL practically concur, whereas the overall costs of ensembling exceeds the training time of STL and Q-MTL models by a factor of k.
The training times reported in Figure were obtained without GPU acceleration E7-4820 CPU - order to simulate a setting with limited computational resources. We also repeated training TITAN Xp GPU. The GPU-based training was 3 to 10 times quicker depending on the languages, but the relative performance between the different approaches remained the same, i.e., STL and Q-MTL training times did not differ substantially, whereas the ensemble model took k- times as much time to be created.
Intel Xeon
3.3 EVALUATION ON NAMED ENTITY RECOGNITION
We also conducted experiments on CoNLL 2002/2003 shared task data on named entity recogni- tion (NER) in English, Spanish and Dutch (Tong Kim Sang. 2002; Tjong Kim Sang & De Meulder 2003). For these experiments, we report performance in terms of overall F1 scores calculated by the official scorer of the shared task. We trained models with k 10 and compared the average performance of the individual STL models to the performance of the Q-MTL and ensemble models.
Table 4a shows the results for NER over the different languages, corroborating vation that Q-MTL is capable closing the gap between the performance of STL models and the much more resource-intensive ensemble model derived from k independent models.
previous obser-
In our POS tagging experiment, we trained models on treebanks of radically differing sizes (cf. Ta- had access training data sets of comparable
ble [. whereas during NER experiments, sizes (ranging between 218K and 273K tokens). In order to simulate the effects of having access limited training data on NER as well, we artificially relied on only 10% of the available training sets.
These results for the limited training data setting are included in Table 4b from which we can see that Q-MTL manages to preserve larger fraction of its original performance, i.e., 87.5% on average as opposed to the ensemble and STL models, which preserved only 86. 86.7% and 86.4% of their original F-scores, respectively.
COMPARISON TO PSEUDO-TASH AUGMENTATION
An achitecture introduced by Meyerson & Miikkulainen 2018) called Pseudo- Augmentation (PTA) uses similar architecture to leverage a better representation of the task by fitting multiple outputs to the same task. This architecture makes series of predictions according
Yi.j softmar ( h; +bW)
(5)
STL
Q-MTL
ensemble
Accuracy (95)
f1
97.75
hr
95.5
Average
95
97.0
96.0
97.50
95.0
96.5
95.5
97.25
94.5
bu
id
13
tr
94.0
Accuracy (8S)
95.0
95
93.5
94.5
93.75
94.0
93.50
86
94
93.0
10
20
30
0
10 20
30
10 20 30 Label noise (16)
30 Label noise (%)
0 10 20
0.
10
20
30
0
10
20
30
Label noise (%)
Label naise (%)
Label noise (16)
Label naise (56
Figure Model performances obtained by the different approaches when a varying amount of noisy training samples are introduced during training. The rightmost plot (titled Average) contains averaged accuracies over the 10 treebanks.

---

Under review as a conference paper at ICLR 2020
10
= 30
Elapsed time (hours)
10a
method
10a
STL
Q-MTL
Ensemble
10
hr id ta eu hr id
Figure 6: Training times of the different approaches for the different languages.
Table 3: F1 performance scores for the NER experiments.
Avg. STL
Q-MTL
Ensemble
Avg. STL
Q-MTL
Ensemble
86.68
86.88
87.86
en
77.54
80.24
78.52
es
82.28
82.35
83.76
es
70.71
71.57
72.56
nl
81.84
83.15
83.61
68.47
69.16
70.33
Avg.
83.60
84.13
85.07
Avg. 72.42
73.66
73.80
(a) 100% training data used
(b) 10% training data used
The main difference between PTA and Q-MTL is that Q-MTL uses an extra transformation and a ReLU non-linearity the hidden representation of the LSTM (ef. Eq. 2 and Eq. for Q-MTL and PTA, respectively).
Another key difference is that PTA model selection (MS), whereas Q-MTL relies model averaging (MA). This means that PTA makes prediction for test instances during inference based on the model which achieves best performing dev accuracy. Q-MTI., however, aggregates all the models according to 3
As Figure 7 shows the different settings for model selection and Multi-Layered Perceptron (MLP) that explore the methods used in PTA (MS @ O MLP) to Q-MTL (MA @ 20 MLP). We also we evaluated the different combinations of model selection/averagingg and the usage of additional non- linearity for completeness. In these experiments 20 MLP means the MLP has 20 dimensional
mlp
MA
Average
Accuracy (86)
96.25
94.75
97.00
96.75
96.00
93
94.5
95.75
94.25
hu
94.0
Accuracy (96)
94.8
93.8
88
93.5
94.50
95.0
87 0 10
10 20 30 Label noise (56)
0 10 20 30
10 20
30
20 30 Label naise (%)
0.
10 20 30
0
10
20
30
Label noise (%)
Label noise (1)
Label noise (1)
Label noise (16)
Figure 7: PTA and Q-MTL compared in an analogue manner. Model selection (MS) compared to model averaging (MA) and MLP (20 MLP) compared to linear classifier (0 MLP). From PTA (MS @ O MLP) to Q-MTL (MA @ 20 MLP) we can see all combinations of these parameters.
8

---

Under review as a conference paper at ICLR 2020
output, while 0 MLP means there is no MLP layer added to the tasks. Note, that by removing the MLP layer we also remove the ReLU activation.
Figure demonstrates that the Q-MTL model with its MLP layer facilitate the of model averaging shown in Eq. B as it outperforms the Q-MTL using model selection (MS @ 20 MLP). On the other hand in the ensemble the models, we can see that MS often outperforms MA in case 0 MLP. Additionally, can see that the MLP layer improves the tolerance of the models to the increasing label noise, as it outperforms 7 out of 10 treebanks the model not employing extra non-linearity. Interestingly when the train set contains high label noise, the ensemble of linear classifiers outperforms the model selected ones.
MLP is used (0 MIP) with which it is indeed discouraged to
RELATED WORK
Caruana 1997) showed that neural networks can be trained for multiple tasks, leveraging cross domain in information. More recently, Søgaard & Goldberg (2016); [Sanh et al. (2018) argues that solving low-level NLP tasks can improve the performance
high level tasks. Additionally, Plaank ett al. (2016); Bingel & Søgaard (201)) show that better performing models can be trained by intro- ducing multiple auxiliary tasks. Rei (2017) proposes an auxiliary task for NLP sequence labeling tasks, where the auxiliary tasks is predict the previous and next word in the sequence. Our results complement these findings by showing that hetter generalization can also be achieved if we learn multiple models for the same task concurrently.
Meyerson & Miikkulainen (0188) introduced Pseudo-Task Augmentation a similar architecture that aims to build a robust internal representation from multiple classifier units that optimize for the same task in the same neural network. Their work contains policies that simulate linear indepen- dence (refered as: perturb, hyperturb, independent dropout) in the set of linearly dependent classifier units, while making use of the dependence in other cases to improve accuracy (refered as: greedy). Section describes the similarities and differences to our method, PTA architecture evaluated on multitask as well, while our work only considers single tasks at the moment.
Ruder Plank (2018) has shown that self-learning and tri-training can be adapted to deep neural nets in the semi-supervised regime. Their tri-training architecture resembles our approach in that they were utilizing multiple classifier units that were built on top of a common representation layer for providing labels to previously unlabeled data.
Cross-view training (CVT) (Clark et al. 2018) resembles Q-MTL in that it also employs a shared bi- LSTM layer used by multiple output layers. The main difference between CVT and Q-MTL is that we are utilizing an bi-LSTM to solve the same task multiple times a supervised setting, whereas Clark al. 2018) used it solve different tasks in semi-supervised scenario.
A series of studies have made use of ensemble learning in the context of deep learning (Hansen & &Salamon 1990, Krogh & Vedeisby 1995 Lee et al., 2015 Huang al.) 2017). Our proposed model is also related to the line of research on mixture of experts proposed by Jacobs et al. (991), which has already been applied successfully in NLP before (Le al., 2016). The main difference in our proposed architecture is that the internal LSTM representation that we train are shared across the classifiers, hence more efficient training could be achieved as opposed to training multiple independent expert models as it was done in (Shazeer et al. 2017)
Model distillation (Hinton et al., 2015) an alternative approach for making computationally de- manding models more effective during inference, however, the approach still requires training of a "cumbersome" model first.
6 CONCLUSIONS
We proposed quasi-multitask learning (Q-MTL), which can be viewed as efficiently trainable alternative of traditional ensembles. We additionally demonstrated that it acts as an implicit form of regularization well. In our experiments, Q-MTL consistently outperformed the single task leam- ing (STL) baseline for both POS tagging and NER. We have also illustrated that Q-MTL generalizes better smaller and noisy datasets compared to both STL and ensemble models.

---

Under review as a conference paper at ICLR 2020
The computational overhead for the additional classification units in Q-MTL is infinitesimal due to the effective aggregation of the losses and the shared recurrent unit between the identical tasks. Although evaluated approach sequence classification tasks, the general idea can be applied for other network architectures and beyond NLPP applications as well.
REFERENCES
Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. Polyglot: Distributed word representations for multilingual nlp. In Proceedings of the Seventeenth Conference on Computational Natu- ral Language Learning, pp. 183-192. Association for Computational Linguistics, 2013. URL
org/ wttpaatttoooooooooooooo
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 164-169, Association for Computational Linguistics, 2017. http///cc/////aaa...oo//aaaaaaa/
Rich Caruana. Multitask learning. Machine Learning 28():)))2222)))1)1 1997. 10.1003/A41100777966744444
1997. 1573-055 doi:
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. Semi-supervised quence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1914 1925. Association for Computational Lin- guistics, 2018. httt:t/t://acl http://www......taoooooooo//--/-
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning. ICML '08, pp. 160-167, New York, NY, USA, 2008. ACM. ISBN 978- 1-60558-2055 doi: 10. 10.1145/1390156.1 19911/
1399177. URL http: //tp:://oi: acm. org/ 10. wtop/o./.011.... 1145/
1390156. 1390177739177
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting 03004
/ repotinng of experimental results. CoRR, abs/1909.03000 2019. (RLhtt::
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis & Machine Intelligence, (10):993-1001, 1990.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network.
In NIPS Deep Learning and Representation Learning Workshop, 2015. Uttp:t/://aaxi
org/ bog/abs//e 1503/ 02531
Sepp Hochreiter and Jrgen Schmidhuber. Long short-term memory. Neural computation, 9:1735- 80, 12 1997. doi: 10.116220
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get M for free. CORR, abs/1704.00109, 2017. URL http://
00109
Robert Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, March 1991. ISSN 0899-7667. doi:
1.97.3.1.79. URL /tLhtttp://ee dei oro./o.gg.1.00 11 w2w1122nneeeee neco. .e..o.999111 19...
Eliyahu Kiperwasser and Miguel Ballesteros. Scheduled multi-task learning: From syntax to trans-
lation. Transactions of the Association for Computational Linguistics, 6:225-240 2018. URL
http://aclweb.rr org/ http://ahhhooooo/ooooaaa
Eliyahu Kiperwasser and Yoav Goldberg. Simple and accurate dependency parsing using bidirec- tional Istm feature representations. Transactions of the Association for Computatiooal ConpuratiioallLnngiittsssss Linguistics,
4:313-327, 2016. U //ac web. org/ httpa/oorraaaaaologgg1111--11111
10

---

Under review as a conference paper at ICLR 2020
Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In J. E. Moody, S. J. Hanson, and R. P. Lippmann (eds.), Advances in Neural Information Processing Sys- Bargan-Kauffmanng 563 a- s mp e we i ght - dew.y- can mprove genera izat 1 iontioon....
tems 4, pp. 950- 957.
1992. URL http: /ttptt:://ppppaa// papers nips. cC/ ccppaper/ paper/
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in neural information processing systems, pp. 231-238, 1995.
Phong Le, Mare Dymetman, and Jean-Michel Renders. Lstm-based mixture-of-exper for knowledge-aware dialogues. In Proceedings of the 1st Workshop on Representation Learning for NLP, pp. 94-99. Association for Computational Linguisties, 2016. doi: 10.18633/11/1/ HiW16110 1611. URL
org/ aatholoogy/W16-111
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. Why M heads are better than one: Training a diverse ensemble of deep networks. CoRR, abs/1511.00314, 2015. /
org wttp://////.....e/eee.o/ C6311
Elliot Meyerson and Risto Miikkulainen. Pseudo-task augmentation: From deep multitask learning to intratask sharing- and back. 03 2018.
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anasta- sopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. Dynet: The dynamic neural network toolkit. arXiv preprint arxiv: 1701003800 2017.
Joakim Nivre, Mitchell Abrams, and et al. Universal dependencies 2.2, 2018. htt::///:/hhd handle e. net. 11234/1-22823 LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Barbara Plank and Željko Agié. Distant supervision from disparate sources for low-resource part- of-speech tagging. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 614-620. Association for Computational Linguisties, 2018. URL
1061
Barbara Plank, Anders Søgaard, and Yoav Goldberg. Multilingual part-of-speech tagging with bidi- rectional long short-term memory models and auxiliary loss. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 412 418. Association for Computational Linguistics, 2016. doi: 10.18653/v1/16--200 URL
2067
Marek Rei. Semi-supervised multitask learning for sequence labeling. In Proceedings of the 55th
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
2121-2130. Association for Computational Linguistics, 2017. doi: 10. 10.18653/11/11--1144 URL
http://aclweb. http:///cc.eeb.orr/atttoooooo/e wttp///nnncclloyy//11111113
Sebastian Ruder and Barbara Plank. Strong baselines for neural semi-supervised learning under domain shift. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1044-1054. Association for Computational Linguistics, 2018. httl:ttppp/cclaae
org wwwg/aathooogg///11111 1096
Victor Sanh, Thomas Wolf, and Sebastian Ruder. A hierarchical multi-task approach for learning embeddings from semantic tasks, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of-experts layer. 2017. URL
Anders Søgaard and Yoav Goldberg. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 231-235, Association for Computational Linguistics, 2016. doi: 10. 10.18653/11/11-20000

---

Under review as a conference paper at ICLR 2020
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Ma- chine Learning Research, 15:1929-1958 2014. URL http://jmlz: wwt.://pppppp...ueeeeee
sr vastava 4a.ht.l
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning NLP. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 3645-3643335 3650, Florence, Italy, July 2019. Association for Computational Linguisties. URL https:: / wwww ac web. wwb.orrg/atcoooogg/// wgw. w 1355
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR. pp. 2818-28222 IEEE Computer Society, 2016.
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-independen named entity recognition. In Proceedings CoNLL-2002, pp. 155-158. Taipei, Taiwan, 2002.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning al IILT-NAAL 2003 Volume 4, CONLL '03, pp. 142-147, Stroudsburg, PA, USA, 2003, Association for Computational Linguistics. doi: 10.3115/1119111 1119195. URL http: / dx. doi org. 3111111111111111111111111 1119199
Language-indepente
12


Reviews:
[{"review_id": "rkgOxr70FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers a regularization technique, derived from multi-task learning, where multiple models with some shared parameters are jointly trained to solve copies of the same task. The technique is well-motivated as an efficient alternative to ensemble learning. The method is validated for a BiLSTM NLP model, which is applied to several POS tagging and named entity recognition tasks. The power of the technique as a regularizer is also demonstrated in the case of highly noisy labels, surprisingly, even outperforming ensemble learning in this setting.\n\nDespite this, my inclination is to reject the paper because of the substantial overlap with previous work and the limited scope of experiments.\n\nMy primary concern is that the proposed technique was previously introduced in [1]. This prior work is not acknowledged in the current paper. Perhaps it was overlooked because it is situated tightly in Multi-task Learning, whereas the present work is motivated mainly with respect to Ensembling.\n\nAlthough the general method was introduced previously, the paper does have some key experimental differences that would be interesting to see explored further.\n\n(1)\tThe paper uses a hidden layer in the separate classification heads, whereas previous work only used a linear classifier. The intuition that more complex heads will yield more diverse models is clear, but it would be great to see experimental evidence that this complexity helps. The conclusion states that the computational overhead is “infinitesimal”; does increasing the complexity of the classifier trade cost for performance?\n(2)\tThis paper uses Eq. 3 to make predictions, whereas previous work found that this did not improve over simply using the best single prediction model, which makes prediction somewhat more efficient. Is there some experimental evidence that Eq. 3 leads to improvements?\n(3)\tThis paper considers the comparison to ensembling, whereas previous work only considered comparisons to single task and standard multitask learning. Additional experiments showing the advantages over ensembling could make this extension a significant contribution.\n(4)\tThis paper presents novel investigation of the regularization effects of the method, i.e., the resilience to noisy labels and the analysis of learned weight matrices. Is there a real problem where this resilience to noise will improve over ensembles, i.e., without randomly replacing labels? Such an experiment would make this point more compelling. Also, is there some underlying reason why the method outperforms ensembles in this case? Is it simply because the method is less expressive so cannot overfit?\n\nIn effect, if the paper could clearly show that (1) or other practical extensions lead to improvements over ensembling in settings where ensembling is commonly used, or enable ensembling in settings where vanilla ensembling fails (i.e., the case of noisy labels), then it could be a substantial contribution. The current scope of the experiments is too limited to conclusively show these points. For example, the technique can be applied to any architecture, but the experiments in the paper are limited to a single architecture; and additional experiments with architectures and tasks that commonly use ensembling would make the experiments more compelling, ideally with comparisons to external results.\n\nOther minor comments:\n-\tIt would be good to see the number of model parameters for easy comparison, especially in table 2 with different value of k.\n-\tIt looks like the x and y axis labels are swapped in Figure 3; from the Figure it looks like STL gets higher accuracies.\n-\tFigure 2 should say epochs instead of iterations.\n\n[1] Meyerson, E. & Miikkulainen R. “Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back”, ICML 2018.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bygykw94qH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a quasi-multitask learning (Q-MTL) for supervised learning. The network architecture in Q-MTL borrows the idea of multi-task neural networks by sharing the latent representation among different classifiers which are designed for a single task.\n\nWhat is the difference among multiple classifiers for the single task? Can they become identical? The rationale behind Q-MTL is unclear to me. Authors need to conduct more analyses to show why Q-MTL is superior to supervised learning.\n\nAuthors claim that Q-MTL is equivalent to performing some regularization. However, I did not see any analysis on this aspect.\n\nIn experiments, the performance of Q-MTL is not so good when compared with ensemble learning.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: One of the reviewers pointed out similarity to existing very recent work which would require significant reframing of the current paper. Hence, this work is below the bar at the moment.
Decision: Reject