Frustratingly easy quasi-multitask learning

1 INTRODUCTION . Ensemble methods are frequently used in machine learning applications due to their tendency of increasing model performance . While the increase in the prediction performance is undoubtedly an important aspect when we train a model , it should not be forgotten that the increased performance of ensembling comes at the price of training multiple models for solving the same task . The question that we tackle in this paper is the following : Can we enjoy the benefits of ensemble learning , while avoiding its overhead for training models from scratch multiple times ? This question is highly relevant these days , since state-of-the-art neural models tend to be extremely resourceintensive on their own ( Strubell et al. , 2019 ) , prohibiting their inclusion in a traditional ensemble setting . Our proposed architecture simultaneously offers the benefit of ensemble learning , while avoiding its drawback of training multiple models . The method introduced here employs a special form of multitask learning ( MTL ) . Caruana ( 1997 ) argues in his seminal work that MTL can be a useful source of introducing inductive bias into machine learning models . Standard MTL have been shown to be fruitfully applicable in solving a series of NLP tasks : Collobert & Weston ( 2008 ) ; Plank et al . ( 2016 ) ; Rei ( 2017 ) ; Kiperwasser & Ballesteros ( 2018 ) ; Sanh et al . ( 2018 ) , inter alia . We introduce quasi-multitask learning ( Q-MTL ) , where the goal is to simultaneously learn multiple neural models that solve identical tasks , while relying on a shared representation layer . Besides the considerable speedup that comes with the proposed technique , we additionally argue that by applying multiple output units on top of a shared parameter set is beneficial , as we can avoid converging to such degenerate internal representations that are highly tailored for a particular classification model . In that sense , Q-MTL can also be viewed as an implicit regularizer , which prevents neural networks to develop such an internal representation which is not generic enough to provide useful input to multiple classification units simultaneously . Our experiments with Q-MTL illustrate that the presence of multiple classifier layers for the same task affect each other positively – similar to ensemble learning – without the additional overhead of actually training multiple models . A similar technique have already been derived from MTL called Pseudo-Task Augmentation ( Meyerson & Miikkulainen , 2018 ) , which builds on the idea of common representation , but the manage- ment of these tasks differs . We conducted experiments comparing the two methods for a greater comprehension of the differences . 2 APPLIED MODELS . We release all our source code used for our experiments at anonymized . Our models are based on the sequence classification framework from Plank et al . ( 2016 ) implemented in DyNet ( Neubig et al. , 2017 ) . Figure 1 provides a visual summary of the different architectures we implemented . Figure 1b highlights that Q-MTL has the benefit of training multiple classification models over the same internal representation , as opposed to traditional ensemble model , which requires the training of multiple LSTM parameters as well ( cf . Figure 1c ) . 2.1 BASELINE ARCHITECTURE . Our baseline classifier is a bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) incorporating character and word level embeddings . We first compute the input embedding for the network at position i as ei = wi ⊕−→ci ⊕←−ci , where ⊕ is the concatenation operator , wi denotes the word embedding , −→ci and ←−ci refers to the left-to-right and right-to-left character-based embeddings , respectively . We subsequently feed ei into a bi-LSTM , which determines a hidden representation hi ∈ Rm for every token position as hi = −→ hi ⊕ ←− hi , i.e. , the concatenation of the hidden states of the two LSTMs processing the input from its beginning to the end , and in reverse direction . The final output of the network for token position i gets computed as yi = softmax ( ReLU ( V hi + bV ) W + bW ) ( 1 ) with V ∈ Rh×m and bV ∈ Rh denoting the weight matrix and the bias of a perceptron unit , whereas W ∈ Rh×c and bW ∈ Rc are the parameters of the neuron performing classification over the c target classes . 2.2 Q-MTL ARCHITECTURE . The Q-MTL network behaves similarly to the model introduced in Section 2.1 , with the notable exception that it trains k distinct classification models , all of which operate over the same hidden representation as input obtained from a single bi-LSTM unit . More concretely , we replace the single prediction of the STL model from Eq . 1 by a series of predictions for Q-MTL according to yi , j = softmax ( ReLU ( V ( j ) hi + b ( j ) V ) W ( j ) + b ( j ) W ) , ( 2 ) with j ∈ { 1 , . . . , k } . As argued before , this approach behaves favorably from a computational point of view , as it relies on a shared representation hi for all the k classification units . The loss of the network for token position i and gold standard class label y∗i can be conveniently generalized as lQ−MTL ( i ) = k∑ j=1 CE ( y∗i , yi , j ) , where CE denotes categorical cross entropy loss and k is the number of ( identical ) tasks in the Q-MTL model , with the special case of k = 1 resulting in standard single task learning ( STL ) . Losses from the different outputs can be aggregated efficiently during backpropagation , hence the shared LSTM cell benefit from multiple error signals without the actual need of going through multiple individual forward and backward passes . Q-MTL outputs k predictions by all of its prediction units , however , we can as well derive a combined prediction from the distinct outputs of Q-MTL according to 1 k k∑ j=1 softmax ( ReLU ( V ( j ) hi + b ( j ) V ) W ( j ) + b ( j ) W ) , ( 3 ) which essentially is a weighted average according to the predicted probabilities of the distinct models . As introducing averaging at the model-level would eliminate diversity of the individual classifiers ( Lee et al. , 2015 ) , this kind of averaging took place in a post-hoc manner , only when making predictions . 2.3 TRADITIONAL ENSEMBLE MODEL . As an additional model , we also employ a traditional ensemble of k independently trained STL models . We define the prediction of the ensemble model by averaging the predictions of k independent models as 1 k k∑ j=1 softmax ( ReLU ( V ( j ) h ( j ) i + b ( j ) V ) W ( j ) + b ( j ) W ) . ( 4 ) The distinctive difference between Eq . 4 and the Q-MTL model formulation in Eq . 3 is that ensembling relies on the hidden representations originating from k independently trained LSTM models as denoted by the superscripts of the hidden states in h ( j ) i . Such an ensemble necessarily requires approximately k-times as much computational resources compared to Q-MTL , due to the LSTM models being trained in total isolation . For the above reason , ensembling is a strictly more expensive form of training a model , for which reason we regard its performance as a glass ceiling for Q-MTL . 3 EXPERIMENTS . Our model uses character embeddings of 100 dimensions and the word representations get initialized by the 64-dimensional pre-trained polyglot word embeddings ( Al-Rfou et al. , 2013 ) as suggested by Plank & Agić ( 2018 ) . We use a one-layered bi-LSTM which outputs hidden vectors hi ∈ R200 as a concatenation of −→ hi , ←− hi ∈ R100 . Instead of directly applying a fully-connected layer to perform classification based on hi , we first transform hi by an intermediate perceptron unit with ReLU 1 95.61 94.99 94.49 93.19 96.84 93.95 93.05 96.05 82.74 93.61 93.45. activation . The perceptron transforms hi into 20 dimensions , that is , we have V ∈ R20×200 . Our motivation with the extra non-linearity introduced by ReLU is to encourage an increased diversity in the behavior of the different output units . Upon training the LSTMs , we used the default architectural settings employed by Plank et al . ( 2016 ) , i.e. , we relied on a word dropout rate of 0.25 ( Kiperwasser & Goldberg , 2016 ) and an additive Gaussian noise ( with σ = 0.2 ) over the input embeddings . We trained all our models for 20 epochs using stochastic gradient descent with a batch size of 1 . First , we assess the quality of Q-MTL towards POS tagging , then we evaluate it on named entity recognition as well . When comparing the performance of different approaches , Q-MTL models are compared against the average performance of k STL models , where k denotes the number of task in the case of Q-MTL . The k STL models are also used to derive a single prediction by the ensemble model . 3.1 POS TAGGING EXPERIMENTS . We set our POS tagging related experiments on 10 treebanks from the Universal Dependencies dataset v2.2 ( Nivre et al. , 2018 ) , namely the Greek-GDT ( el ) , English-LinES ( en ) , Basque-BDT ( eu ) , Finnish-FTB ( fi ) , Croatian-SET ( hr ) , Hungarian-Szeged ( hu ) , Indonesian-GSD ( id ) , DutchAlpino ( nl ) , Tamil-TTB ( ta ) and Turkish-IMST ( tr ) treebanks . These treebanks not only cover a typologically diverse set of languages , but they also vary substantially in the number of available training sequences , as illustrated in Table 1 . Table 1 also illustrates the typological diversity of the investigated languages , as the average number of occurrences per distinct word forms vary substantially , i.e. , between 2.4 for Tamil and 6.9 for Dutch . 3.1.1 EXPERIMENTS WITH THE NUMBER OF TASKS . We first investigate how does changing the value for k , i.e. , the number of simultaneously learned tasks , affects the performance of Q-MTL . We experimented with k ∈ { 1 , 10 , 30 } . Based on the results in Table 2 , we set the number of tasks to be employed as k = 10 for all upcoming experiments . In order to choose k without overfitting to the training data , this experiment was conducted on the development set . 3.1.2 COMPARING Q-MTL WITH STL . Following the recommendation in ( Dodge et al. , 2019 ) , we report learning curves over the development set as a function of the number epochs in Figure 2 As a general observation , we can see that Q-MTL tends to perform consistently better than STL models right from the beginning of training . Directly comparing the classifiers One benefit of Q-MTL is that it learns k different classification models during training with only a marginal computational overhead compared to training a STL baseline , since all the tasks share a common internal representation . As discussed earlier , we can combine the predictions from the k classifiers from Q-MTL according to Eq . 2 . It is also possible , however , to use the k distinct predictions of Q-MTL . In what follows next , we compare the performance of the k STL models we train to the k classifiers that are incorporated within a Q-MTL model . Upon comparing the performance of a Q-MTL classifier with a STL model , we made it sure that the overlapping parameters ( matrices V and W ) were initialized with the same values and that they receive the training instances in the exact same order . This way the performance achieved by the ith output of Q-MTL is directly comparable with the ith STL baseline . Comparison of the results of the individual outputs of Q-MTL and their corresponding STL counterpart are included in Figure 3 . 95 96 97 95 96 97 QM TL a cc ur ac y ( % ) el 95 96 95.0 95.5 96.0 en 94.25 94.50 94.75 94.2 94.4 94.6 94.8 eu 93.0 93.5 94.0 93.0 93.5 94.0 fi 97.0 97.5 96.8 97.0 97.2 97.4 hr 93 94 STL accuracy ( % ) 92.5 93.0 93.5 94.0 94.5 QM TL a cc ur ac y ( % ) hu 93.0 93.5 STL accuracy ( % ) 92.75 93.00 93.25 93.50 id 95.5 96.0 96.5 STL accuracy ( % ) 95.5 96.0 96.5 nl 82.5 85.0 87.5 STL accuracy ( % ) 82 84 86 88 ta 93 94 STL accuracy ( % ) 93.0 93.5 94.0 94.5 tr dataset test dev Figure 3 : Scatter plot comparing the accuracy of the individual classifiers from Q-MTL ( k = 10 ) and their corresponding STL counterpart . Each model that is above the diagonal line performs better after training in the Q-MTL setting . Training Q-MTL models with k tasks simultaneously is not only faster than training k distinct STL models separately , but the individual Q-MTL models typically outperform their baseline counterparts evaluated against both the development and the test data . The regularizing effect of Q-MTL We have argued earlier that Q-MTL has an implicit regularizing effect . Among most recent techniques , such as dropout ( Srivastava et al. , 2014 ) , weight decay ( Krogh & Hertz , 1992 ) is one of the most typical form of regularization for fostering the generalization capability of the learned models . When employing weight decay , we add an extra term penalizing the magnitude of the values learned by our model , which results in an overall shrinkage in the values of the model parameters . Figure 4 illustrates that the effects of employing Q-MTL is similar to applying weight decay , as the Frobenius norm of the parameter matrices from the classifiers of Q-MTL are substantially smaller than those of the STL classifiers . This observation holds for both the of parameter sets V and W . Recall that the initial values for these matrices were identical for both Q-MTL and STL .

Summary: This paper considers a regularization technique, derived from multi-task learning, where multiple models with some shared parameters are jointly trained to solve copies of the same task. The technique is well-motivated as an efficient alternative to ensemble learning. The method is validated for a BiLSTM NLP model, which is applied to several POS tagging and named entity recognition tasks. The power of the technique as a regularizer is also demonstrated in the case of highly noisy labels, surprisingly, even outperforming ensemble learning in this setting.