Title: A JOINT MANY-TASK MODEL: GROWING A NEURAL NETWORK FOR MULTIPLE NLP TASKS

Abstract: Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce such a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. All layers include shortcut connections to both word representations and lower-level task predictions. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end trainable model obtains state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment. It also performs competitively on POS tagging. Our dependency parsing layer relies only on a single feed-forward pass and does not require a beam search.

1. INTRODUCTION:

The potential for leveraging multiple levels of representation has been demonstrated in a variety of ways in the field of Natural Language Processing (NLP). For example, Part-Of-Speech (POS) tags are used to train syntactic parsers. The parsers are used to improve higher-level tasks, such as natural language inference (Chen et al., 2016) , relation classification (Socher et al., 2012) , sentiment analysis (Socher et al., 2013; Tai et al., 2015) , or machine translation (Eriguchi et al., 2016) . However, higher level tasks are not usually able to improve lower level tasks, often because systems are pipelines and not trained end-to-end.

In deep learning, unsupervised word vectors are useful representations and often used to initialize recurrent neural networks for subsequent tasks (Pennington et al., 2014) . However, not being jointly trained, deep NLP models have yet shown benefits from predicting many (> 4) increasingly complex linguistic tasks each at a successively deeper layer. Instead, existing models are often designed to predict different tasks either entirely separately or at the same depth (Collobert et al., 2011) , ignoring linguistic hierarchies.

We introduce a Joint Many-Task (JMT) model, outlined in Fig. 1 , which predicts increasingly complex NLP tasks at successively deeper layers. Unlike traditional NLP pipeline systems, our single JMT model can be trained end-to-end for POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment. We propose an adaptive training and regularization strategy to grow this model in its depth. With the help of this strategy we avoid catastrophic interference between tasks, and instead show that both lower and higher level tasks benefit from the joint training. Our model is influenced by the observation of Søgaard & Goldberg (2016) who showed that predicting two different tasks is more accurate when performed in different layers than in the same layer (Collobert et al., 2011) . 

2. THE JOINT MANY-TASK MODEL:

In this section, we assume that the model is trained and describe its inference procedure. We begin at the lowest level and work our way to higher layers and more complex tasks.

2.1. WORD REPRESENTATIONS:

For each word w t in the input sentence s of length L, we construct a representation by concatenating a word and a character embedding.

Word embeddings: We use Skip-gram (Mikolov et al., 2013) to train a word embedding matrix, which will be shared across all of the tasks. The words which are not included in the vocabulary are mapped to a special UNK token.

Character n-gram embeddings: Character n-gram embeddings are learned using the same skipgram objective function as the word vectors. We construct the vocabulary of the character n-grams in the training data and assign an embedding for each character n-gram. The final character embedding is the average of the unique character n-gram embeddings of a word w t . 1 For example, the character n-grams (n = 1, 2, 3) of the word "Cat" are {C, a, t, #BEGIN#C, Ca, at, t#END#, #BEGIN#Ca, Cat, at#END#}, where "#BEGIN#" and "#END#" represent the beginning and the end of each word, respectively. The use of the character n-gram embeddings efficiently provides morphological features and information about unknown words. The training procedure for the character n-gram embeddings is described in Section 3.1, and for further details, please see Appendix A. Each word is subsequently represented as x t , the concatenation of its corresponding word and character vectors.

2.2. WORD-LEVEL TASK: POS TAGGING:

The first layer of the model is a bi-directional LSTM (Graves & Schmidhuber, 2005; Hochreiter & Schmidhuber, 1997) whose hidden states are used to predict POS tags. We use the following Long Short-Term Memory (LSTM) units for the forward direction: where we define the input g t as

EQUATION

EQUATION

g t = [ - → h t-1 ; x t ],

i.e. the concatenation of the previous hidden state and the word representation of w t . The backward pass is expanded in the same way, but a different set of weights are used.

For predicting the POS tag of w t , we use the concatenation of the forward and backward states in a one-layer bi-LSTM layer corresponding to the t-th word:

h t = [ - → h t ; ← - h t ]. Then each h t (1 ≤ t ≤ L

) is fed into a standard softmax classifier with a single ReLU layer which outputs the probability vector y (1) for each of the POS tags.

2.3. WORD-LEVEL TASK: CHUNKING:

Chunking is also a word-level classification task which assigns a chunking tag (B-NP, I-VP, etc.) for each word. The tag specifies the region of major phrases (or chunks) in the sentence.

Chunking is performed in the second bi-LSTM layer on top of the POS layer. When stacking the bi-LSTM layers, we use Eq. ( 1) with input g

(2) t = [h (2) t-1 ; h (1) t ; x t ; y (pos) t

], where h

(1) t is the hidden state of the first (POS) layer. We define the weighted label embedding y (pos) t as follows:

EQUATION

)

where C is the number of the POS tags, p(y

(1) t = j|h (1)

t ) is the probability value that the j-th POS tag is assigned to w t , and (j) is the corresponding label embedding. The probability values are automatically predicted by the POS layer working like a built-in POS tagger, and thus no gold POS tags are needed. This output embedding can be regarded as a similar feature to the K-best POS tag feature which has been shown to be effective in syntactic tasks (Andor et al., 2016; Alberti et al., 2015) . For predicting the chunking tags, we employ the same strategy as POS tagging by using the concatenated bi-directional hidden states h

(2) t = [ - → h (2) t ; ← - h (2)

t ] in the chunking layer. We also use a single ReLU hidden layer before the classifier.

2.4. SYNTACTIC TASK: DEPENDENCY PARSING:

Dependency parsing identifies syntactic relationships (such as an adjective modifying a noun) between pairs of words in a sentence. We use the third bi-LSTM layer on top of the POS and chunking layers to classify relationships between all pairs of words. The input vector for the LSTM includes hidden states, word representations, and the label embeddings for the two previous tasks: g

(3) t = [h (3) t-1 ; h (2) t ; x t ; (y (pos) t + y (chk) t

)], where we computed the chunking vector in a similar fashion as the POS vector in Eq. (2). The POS and chunking tags are commonly used to improve dependency parsing (Attardi & DellOrletta, 2008) .

Like a sequential labeling task, we simply predict the parent node (head) for each word in the sentence. Then a dependency label is predicted for each of the child-parent node pairs. To predict the parent node of the t-th word w t , we define a matching function between w t and the candidates of the parent node as m (t, j) = h (3) L+1 = r as a parameterized vector. To compute the probability that w j (or the root node) is the parent of w t , the scores are normalized:

(3) t T W d h (3) j ,

h (4) 2 h (4) 3 label embedding h (3) 1 y (chk) 1 y (pos) 1 y (rel)

EQUATION

)

where L is the sentence length.

Next, the dependency labels are predicted using [h

t ; h

(3) j ] as input to a standard softmax classifier with a single ReLU layer. At test time, we greedily select the parent node and the dependency label for each word in the sentence. 2 At training time, we use the gold child-parent pairs to train the label predictor.

2.5. SEMANTIC TASK: SEMANTIC RELATEDNESS:

The next two tasks model the semantic relationships between two input sentences. The first task measures the semantic relatedness between two sentences. The output is a real-valued relatedness score for the input sentence pair. The second task is a textual entailment task, which requires one to determine whether a premise sentence entails a hypothesis sentence. There are typically three classes: entailment, contradiction, and neutral.

The two semantic tasks are closely related to each other. If the semantic relatedness between two sentences is very low, they are unlikely to entail each other. Based on this intuition and to make use of the information from lower layers, we use the fourth and fifth bi-LSTM layer for the relatedness and entailment task, respectively. Now it is required to obtain the sentence-level representation rather than the word-level representation h (4) t used in the first three tasks. We compute the sentence-level representation h (4) s as the element-wise maximum values across all of the word-level representations in the fourth layer:

EQUATION

.

(4)

To model the semantic relatedness between s and s , we follow Tai et al. (2015) . The feature vector for representing the semantic relatedness is computed as follows:

EQUATION

where h

(4) s -h (4) s

is the absolute values of the element-wise subtraction, and h

(4) s h (4)

s is the element-wise multiplication. Both of them can be regarded as two different similarity metrics of the two vectors. Then d 1 (s, s ) is fed into a softmax classifier with a single Maxout hidden layer (Goodfellow et al., 2013) to output a relatedness score (from 1 to 5 in our case) for the sentence pair.

2.6. SEMANTIC TASK: TEXTUAL ENTAILMENT:

For entailment classification between two sentences, we also use the max-pooling technique as in the semantic relatedness task. To classify the premise-hypothesis pair (s, s ) into one of the three classes, we compute the feature vector d 2 (s, s ) as in Eq. ( 5) except that we do not use the absolute values of the element-wise subtraction, because we need to identify which is the premise (or hypothesis). Then d 2 (s, s ) is fed into a standard softmax classifier.

To make use of the output from the relatedness layer directly, we use the label embeddings for the relatedness task. More concretely, we compute the class label embeddings for the semantic relatedness task similar to Eq. ( 2). The final feature vectors that are concatenated and fed into the entailment classifier are the weighted relatedness label embedding and the feature vector d 2 (s, s ). 3We use three Maxout hidden layers before the classifier.

3. TRAINING THE JMT MODEL:

The model is trained jointly over all datasets. During each epoch, the optimization iterates over each full training dataset in the same order as the corresponding tasks described in the modeling section.

3.1. PRE-TRAINING WORD REPRESENTATIONS:

We pre-train word embeddings using the Skip-gram model with negative sampling (Mikolov et al., 2013) . We also pre-train the character n-gram embeddings using Skip-gram. The only difference is that each input word embedding in the Skip-gram model is replaced with its corresponding average embedding of the character n-gram embeddings described in Section 2.1. These embeddings are fine-tuned during the training of our JMT model. We denote the embedding parameters as θ e .

3.2. TRAINING THE POS LAYER:

Let θ POS = (W POS , b POS , θ e ) denote the set of model parameters associated with the POS layer, where W POS is the set of the weight matrices in the first bi-LSTM and the classifier, and b POS is the set of the bias vectors. The objective function to optimize θ POS is defined as follows:

EQUATION

where p(y

(1) t = α wt |h (1)

t ) is the probability value that the correct label α is assigned to w t in the sentence s, λ W POS 2 is the L2-norm regularization term, and λ is a hyperparameter.

We call the second regularization term δ θ e -θ e 2 a successive regularization term. The successive regularization is based on the idea that we do not want the model to forget the information learned for the other tasks. In the case of POS tagging, the regularization is applied to θ e , and θ e is the embedding parameter after training the final task in the top-most layer at the previous training epoch. δ is a hyperparameter.

3.3. TRAINING THE CHUNKING LAYER:

The objective function is defined as follows:

EQUATION

which is similar to that of POS tagging, and

θ chk is (W chk , b chk , E POS , θ e )

, where W chk and b chk are the weight and bias parameters including those in θ POS , and E POS is the set of the POS label embeddings. θ POS is the one after training the POS layer at the current training epoch.

3.4. TRAINING THE DEPENDENCY LAYER:

The objective function is defined as follows:

EQUATION

)

where p(α|h

(3) t )

is the probability value assigned to the correct parent node α for w t , and p(β|h

EQUATION

α ) is the probability value assigned to the correct dependency label β for the childparent pair (w t , α). θ dep is defined as

(W dep , b dep , W d , r, E POS , E chk , θ e )

, where W dep and b dep are the weight and bias parameters including those in θ chk , and E chk is the set of the chunking label embeddings.

3.5. TRAINING THE RELATEDNESS LAYER:

Following Tai et al. (2015) , the objective function is defined as follows:

EQUATION

EQUATION

where p(s, s ) is the gold distribution over the defined relatedness scores, p(h

EQUATION

s ) is the predicted distribution given the the sentence representations, and KL p(s, s ) p(h

(4) s , h (4)

s ) is the KL-divergence between the two distributions. θ rel is defined as (W rel , b rel , E POS , E chk , θ e ).

3.6. TRAINING THE ENTAILMENT LAYER:

The objective function is defined as follows:

EQUATION

EQUATION

)

where p(y

EQUATION

s ) is the probability value that the correct label α is assigned to the premise-hypothesis pair (s, s ). θ ent is defined as

(W ent , b ent , E POS , E chk , E rel , θ e )

, where E rel is the set of the relatedness label embeddings.

4. RELATED WORK:

Many deep learning approaches have proven to be effective in a variety of NLP tasks and are becoming more and more complex. They are typically designed to handle single tasks, or some of them are designed as general-purpose models (Kumar et al., 2016; Sutskever et al., 2014) but applied to different tasks independently.

For handling multiple NLP tasks, multi-task learning models with deep neural networks have been proposed (Collobert et al., 2011; Luong et al., 2016) , and more recently Søgaard & Goldberg (2016) have suggested that using different layers for different tasks is more effective than using the same layer in jointly learning closely-related tasks, such as POS tagging and chunking. However, the number of tasks was limited or they have very similar task settings like word-level tagging, and it was not clear how lower-level tasks could be also improved by combining higher-level tasks.

In the field of computer vision, some transfer and multi-task learning approaches have also been proposed (Li & Hoiem, 2016; Misra et al., 2016) . For example, Misra et al. (2016) proposed a multi-task learning model to handle different tasks. However, they assume that each data sample has annotations for the different tasks, and do not explicitly consider task hierarchies.

Recently, Rusu et al. (2016) have proposed a progressive neural network model to handle multiple reinforcement learning tasks, such as Atari games. Like our JMT model, their model is also successively trained according to different tasks using different layers called columns in their paper. In their model, once the first task is completed, the model parameters for the first task are fixed, and then the second task is handled by adding new model parameters. Therefore, accuracy of the previously trained tasks is never improved. In NLP tasks, multi-task learning has the potential to improve not only higher-level tasks, but also lower-level tasks. Rather than fixing the pre-trained model parameters, our successive regularization allows our model to continuously train the lower-level tasks without significant accuracy drops.

5. EXPERIMENTAL SETTINGS:



5.1. DATASETS:

POS tagging: To train the POS tagging layer, we used the Wall Street Journal (WSJ) portion of Penn Treebank, and followed the standard split for the training (Section 0-18), development (Section 19-21), and test (Section 22-24) sets. The evaluation metric is the word-level accuracy.

Chunking: For chunking, we also used the WSJ corpus, and followed the standard split for the training (Section 15-18) and test (Section 20) sets as in the CoNLL 2000 shared task. We used Section 19 as the development set, following Søgaard & Goldberg (2016) , and employed the IOBES tagging scheme. The evaluation metric is the F1 score defined in the shared task.

Dependency parsing: We also used the WSJ corpus for dependency parsing, and followed the standard split for the training (Section 2-21), development (Section 22), and test (Section 23) sets. We converted the treebank data to Stanford style dependencies using the version 3.3.0 of the Stanford converter. The evaluation metrics are the Unlabeled Attachment Score (UAS) and the Labeled Attachment Score (LAS), and punctuations are excluded for the evaluation.

Semantic relatedness: For the semantic relatedness task, we used the SICK dataset (Marelli et al., 2014) , and followed the standard split for the training (SICK train.txt), development (SICK trial.txt), and test (SICK test annotated.txt) sets. The evaluation metric is the Mean Squared Error (MSE) between the gold and predicted scores.

Textual entailment: For textual entailment, we also used the SICK dataset and exactly the same data split as the semantic relatedness dataset. The evaluation metric is the accuracy.

5.2. TRAINING DETAILS:

Pre-training embeddings: We used the word2vec toolkit to pre-train the word embeddings. We created our training corpus by selecting lowercased English Wikipedia text and obtained 100dimensional Skip-gram word embeddings trained with the context window size 1, the negative sampling method (15 negative samples), and the sub-sampling method (10 -5 of the sub-sampling coefficient). 4 We also pre-trained the character n-gram embeddings using the same parameter settings with the case-sensitive Wikipedia text. We trained the character n-gram embeddings for n = 1, 2, 3, 4 in the pre-training step.

Embedding initialization: We used the pre-trained word embeddings to initialize the word embeddings, and the word vocabulary was built based on the training data of the five tasks. All words in the training data were included in the word vocabulary, and we employed the word-dropout method (Kiperwasser & Goldberg, 2016) to train the word embedding for the unknown words. We also built the character n-gram vocabulary for n = 2, 3, 4, following Wieting et al. (2016) , and the character n-gram embeddings were initialized with the pre-trained embeddings. All of the label embeddings were initialized with uniform random values in [-6/(dim + C), 6/(dim + C)], where dim = 100 is the dimensionality of the label embeddings and C is the number of labels.

Weight initialization: The dimensionality of the hidden layers in the bi-LSTMs was set to 100. We initialized all of the softmax parameters and bias vectors, except for the forget biases in the LSTMs, with zeros, and the weight matrix W d and the root node vector r for dependency parsing were also initialized with zeros. All of the forget biases were initialized with ones. The other weight matrices were initialized with uniform random values in [-6/(row + col), 6/(row + col)], where row and col are the number of rows and columns of the matrices, respectively.

Optimization: At each epoch, we trained our model in the order of POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment. We used mini-batch stochastic gradient decent to train our model. The mini-batch size was set to 25 for POS tagging, chunking, and the SICK tasks, and 15 for dependency parsing. We used a gradient clipping strategy with growing clipping values for the different tasks; concretely, we employed the simple function: min(3.0, depth), where depth is the number of bi-LSTM layers involved in each task, and 3.0 is the maximum value. The learning rate at the k-th epoch was set to ε 1.0+ρ(k-1) , where ε is the initial learning rate, and ρ is the hyperparameter to decrease the learning rate. We set ε to 1.0 and ρ to 0.3. At each epoch, the same learning rate was shared across all of the tasks.

Regularization::

We set the regularization coefficient to 10 -6 for the LSTM weight matrices, 10 -5 for the weight matrices in the classifiers, and 10 -3 for the successive regularization term excluding the classifier parameters of the lower-level tasks, respectively. The successive regularization coefficient for the classifier parameters was set to 10 -2 . We also used dropout (Hinton et al., 2012) . The dropout rate was set to 0.2 for the vertical connections in the multi-layer bi-LSTMs (Pham et al., 2014) , the word representations and the label embeddings of the entailment layer, and the classifier of the POS tagging, chunking, dependency parsing, and entailment. A different dropout rate of 0.4 was used for the word representations and the label embeddings of the POS, chunking, and dependency layers, and the classifier of the relatedness layer.

6. RESULTS AND DISCUSSION:



6.1. SUMMARY OF MULTI-TASK RESULTS:

Table 1 shows our results of the test sets on the five different tasks. 5 The column "Single" shows the results of handling each task separately using single-layer bi-LSTMs, and the column "JMT all " shows the results of our JMT model. The single task settings only use the annotations of their own tasks. For example, when treating dependency parsing as a single task, the POS and chunking tags are not used. We can see that all results of the five different tasks are improved in our JMT model, which shows that our JMT model can handle the five different tasks in a single model. Our JMT model allows us to access arbitrary information learned from the different tasks. If we want to use the model just as a POS tagger, we can use the output from the first bi-LSTM layer. The output can be the weighted POS label embeddings as well as the discrete POS tags.

Table 1 also shows the results of three subsets of the different tasks. For example, in the case of "JMT ABC ", only the first three layers of the bi-LSTMs are used to handle the three tasks. In the case of "JMT DE ", only the top two layers are used just as a two-layer bi-LSTM by omitting all information from the first three layers. The results of the closely-related tasks show that our JMT model improves not only the high-level tasks, but also the low-level tasks. Table 6 : Textual entailment results.

6.2. COMPARISON WITH PUBLISHED RESULTS:



POS tagging::

Table 2 shows the results of POS tagging, and our JMT model achieves the score close to the state-of-the-art results. The best result to date has been achieved by Ling et al. (2015) , which uses character-based LSTMs. Incorporating the character-based encoders into our JMT model would be an interesting direction, but we have shown that the simple pre-trained character n-gram embeddings lead to the promising result.

Chunking: Table 3 shows the results of chunking, and our JMT model achieves the state-of-the-art result. Søgaard & Goldberg (2016) proposed to jointly learn POS tagging and chunking in different layers, but they only showed improvement for chunking. By contrast, our results show that the low-level tasks are also improved by the joint learning.

Dependency parsing: Table 4 shows the results of dependency parsing by using only the WSJ corpus in terms of the dependency annotations, and our JMT model achieves the state-of-the-art result. 6 It is notable that our simple greedy dependency parser outperforms the previous state-ofthe-art result which is based on beam search with global information. The result suggests that the bi-LSTMs efficiently capture global information necessary for dependency parsing. Moreover, our single task result already achieves high accuracy without the POS and chunking information. Further analysis on our dependency parser can be found in Appendix B.

Semantic relatedness: Table 5 shows the results of the semantic relatedness task, and our JMT model achieves the state-of-the-art result. The result of "JMT DE " is already better than the previous state-of-the-art results. Both of Zhou et al. (2016) and Tai et al. (2015) explicitly used syntactic tree structures, and Zhou et al. (2016) relied on attention mechanisms. However, our method uses the simple max-pooling strategy, which suggests that it is worth investigating such simple methods before developing complex methods for simple tasks. Currently, our JMT model does not explicitly use the learned dependency structures, and thus the explicit use of the output from the dependency layer should be an interesting direction of future work.

Textual entailment: Table 6 shows the results of textual entailment, and our JMT model achieves the state-of-the-art result. 7 The previous state-of-the-art result in Yin et al. (2016) relied on attention mechanisms and dataset-specific data pre-processing and features. Again, our simple max-pooling strategy achieves the state-of-the-art result boosted by the joint training. These results show the importance of jointly handling related tasks. Error analysis can be found in Appendix C.

6.3. ANALYSIS ON MULTI-TASK LEARNING ARCHITECTURES:

Here, we first investigate the effects of using deeper layers for the five different single tasks. We then show the effectiveness of our training strategy: the successive regularization, the shortcut connections of the word representations, the embeddings of the output labels, the character n-gram embeddings, the use of the different layers for the different tasks, and the vertical connections of multi-layer bi-LSTMs. All of the results shown in this section are the development set results. -Depth: The single task settings shown in Table 1 are obtained by using single layer bi-LSTMs, but in our JMT model, the higher-level tasks use successively deeper layers. To investigate the gap between the different number of the layers for each task, we also show the results of using multi-layer bi-LSTMs for the single task settings, in the column of "Single+" in Table 7 . More concretely, we use the same number of the layers with our JMT model; for example, three layers are used for dependency parsing, and five layers are used for textual entailment. As shown in these results, deeper layers do not always lead to better results, and the joint learning is more important than making the models complex only for single tasks. -Successive regularization: In Table 8 , the column of "w/o SR" shows the results of omitting the successive regularization terms described in Section 3. We can see that the accuracy of chunking is improved by the successive regularization, while other results are not affected so much. The chunking dataset used here is relatively small compared with other low-level tasks, POS tagging and dependency parsing. Thus, these results suggest that the successive regularization is effective when dataset sizes are imbalanced.

JMT -Shortcut connections: Our JMT model feeds the word representations into all of the bi-LSTM layers, which is called the shortcut connection. Table 9 shows the results of "JMT all " with and without the shortcut connections. The results without the shortcut connections are shown in the column of "w/o SC". These results clearly show that the importance of the shortcut connections in our JMT model, and in particular, the semantic tasks in the higher layers strongly rely on the shortcut connections. That is, simply stacking the LSTM layers is not sufficient to handle a variety of NLP tasks in a single model. In Appendix D, we show how the shared word representations change according to each task (or layer). -Output label embeddings: Table 10 shows the results without using the output labels of the POS, chunking, and relatedness layers, in the column of "w/o LE". These results show that the explicit use of the output information from the classifiers of the lower layers is important in our JMT model. The results in the column of "w/o SC&LE" are the ones without both of the shortcut connections and the label embeddings. -Character n-gram embeddings: Table 11 shows the results for the three single tasks, POS tagging, chunking, and dependency parsing, with and without the pre-trained character n-gram embeddings. The column of "W&C" corresponds to using both of the word and character n-gram embeddings, and that of "Only W" corresponds to using only the word embeddings. These results clearly show that jointly using the pre-trained word and character n-gram embeddings is helpful in improving the results.

The pre-training of the character n-gram embeddings is also effective; for example, without the pre-training, the POS accuracy drops from 97.52% to 97.38% and the chunking accuracy drops from 95.65% to 95.14%, but they are still better than those of using word2vec embeddings alone. Further analysis can be found in Appendix A. -Different layers for different tasks: Table 12 shows the results for the three tasks of our "JMT ABC " setting and that of not using the shortcut connections and the label embeddings as in Table 10. In addition, in the column of "All-3", we show the results of using the highest (i.e., the third) layer for all of the three tasks without any shortcut connections and label embeddings, and thus the two settings "w/o SC&LE" and "All-3" require exactly the same number of the model parameters. The results show that using the same layers for the three different tasks hampers the effectiveness of our JMT model, and the design of the model is much more important than the number of the model parameters. -Vertical connections: Finally, we investigated our JMT results without using the vertical connections in the five-layer bi-LSTMs. More concretely, when constructing the input vectors g t , we do not use the bi-LSTM hidden states of the previous layers. Table 13 shows the JMT all results with and without the vertical connections. As shown in the column of "w/o VC", we observed the competitive results. Therefore, in the target tasks used in our model, sharing the word representations and the output label embeddings is more effective than just stacking the bi-LSTM layers.

7. CONCLUSION:

We presented a joint many-task model to handle a variety of NLP tasks with growing depth of layers in a single end-to-end deep model. Our model is successively trained by considering linguistic hierarchies, directly connecting word representations to all layers, explicitly using predictions in lower tasks, and applying successive regularization. where σ(•) is the logistic sigmoid function, ṽ(w) is the weight vector for the context word w, and w i is a negative sample. It should be noted that the weight vectors for the context words are parameterized for the words without any character information.

A.2 EFFECTIVENESS ON UNKNOWN WORDS:

One expectation from the use of the character n-gram embeddings is to better handle unknown words. We verified this assumption in the single task setting for POS tagging, based on the results reported in Table 11 . Table 14 shows that the joint use of the word and character n-gram embeddings improves the score by about 19% in terms of the accuracy for unknown words.

We also show the results of the single task setting for dependency parsing in Table 15 . Again, we can see that using the character-level information is effective, and in particular, the improvement of the LAS score is large. These results suggest that it is better to use not only the word embeddings, but also the character n-gram embeddings by default. Recently, the joint use of word and character information has proven to be effective in language modeling (Miyamoto & Cho, 2016) , but just using the simple character n-gram embeddings is fast and also effective.

B ANALYSIS ON DEPENDENCY PARSING:

Our dependency parser is based on the idea of predicting a head (or parent) for each word, and thus the parsing results do not always lead to correct trees. To inspect this aspect, we checked the parsing results on the development set (1,700 sentences), using the "JMT ABC " setting.

In the dependency annotations used in this work, each sentence has only one root node, and we have found 11 sentences with multiple root nodes and 11 sentences with no root nodes in our parsing results. We show two examples below:

(a) Underneath the headline " Diversification , " it counsels , " Based on the events of the past week , all investors need to know their portfolios are balanced to help protect them against the market 's volatility . " (b) Mr. Eskandarian , who resigned his Della Femina post in September , becomes chairman and chief executive of Arnold .

In the example (a), the two boldfaced words "counsels" and "need" are predicted as child nodes of the root node, and the underlined word "counsels" is the correct one based on the gold annotations. This example sentence (a) consists of multiple internal sentences, and our parser misunderstood that both of the two verbs are the heads of the sentence.

In the example (b), none of the words is connected to the root node, and the correct child node of the root is the underlined word "chairman". Without the internal phrase "who resigned... in September", the example sentence (b) is very simple, but we have found that such a simplified sentence is still not parsed correctly. In many cases, verbs are linked to the root nodes, but sometimes other types of words like nouns can be the candidates. In our model, the single parameterized vector r is used to represent the root node for each sentence. Therefore, the results of the examples (a) and (b) suggest that it would be needed to capture various types of root nodes, and using sentence-dependent root representations would lead to better results in future work.

C ANALYSIS ON SEMANTIC TASKS:

We inspected the development set results on the semantic tasks using the "JMT all " setting. In our model, the highest-level task is the textual entailment task. We show an example premise-hypothesis pair which is misclassified in our results:

Premise: "A surfer is riding a big wave across dark green water", and Hypothesis: "The surfer is riding a small wave".

The predicted label is entailment, but the gold label is contradiction. This example is very easy by focusing on the difference between the two words "big" and "small". However, our model fails to correctly classify this example because there are few opportunities to learn the difference. Our model relies on the pre-trained word embeddings based on word co-occurrence statistics (Mikolov et al., 2013) , and it is widely known that such co-occurrence-based embeddings can rarely discriminate between antonyms and synonyms (Ono et al., 2015) . Moreover, the other four tasks in our JMT model do not explicitly provide the opportunities to learn such semantic aspects. Even in the training data of the textual entailment task, we can find only one example to learn the difference between the two words, which is not enough to obtain generalization capacities. Therefore, it is worth investigating the explicit use of external dictionaries or the use of pre-trained word embeddings learned with such dictionaries (Ono et al., 2015) , to see whether our JMT model is further improved not only for the semantic tasks, but also for the low-level tasks.

D HOW DO SHARED EMBEDDINGS CHANGE:

In our JMT model, the word and character n-gram embedding matrices are shared across all of the five different tasks. To better qualitatively explain the importance of the shortcut connections shown in Table 9 , we inspected how the shared embeddings change when fed into the different bi-LSTM layers. More concretely, we checked closest neighbors in terms of the cosine similarity for the word representations before and after fed into the forward LSTM layers. In particular, we used the corresponding part of W u in Eq. (1) to perform linear transformation of the input embeddings, because u t directly affects the hidden states of the LSTMs. Thus, this is a context-independent analysis.

Table 16 shows the examples of the word "standing". The row of "Embedding" shows the cases of using the shared embeddings, and the others show the results of using the linear-transformed embeddings. In the column of "Only word", the results of using only the word embeddings are shown. The closest neighbors in the case of "Embedding" capture the semantic similarity, but after fed into the POS layer, the semantic similarity is almost washed out. This is not surprising because it is sufficient to cluster the words of the same POS tags: here, NN, VBG, etc. In the chunking layer, the similarity in terms of verbs is captured, and this is because it is sufficient to identify the coarse chunking tags: here, VP. In the dependency layer, the closest neighbors are adverbs, gerunds of verbs, and nouns, and all of them can be child nodes of verbs in dependency trees. However, this information is not sufficient in further classifying the dependency labels. Then we can see that in the column of "Word and char", jointly using the character n-gram embeddings adds the morphological information, and as shown in Table 11 , the LAS score is substantially improved.

In the case of semantic tasks, the projected embeddings capture not only syntactic, but also semantic similarities. These results show that different tasks need different aspects of the word similarities, and our JMT model efficiently transforms the shared embeddings for the different tasks by the simple linear transformation. Therefore, without the shortcut connections, the information about the word representations are fed into the semantic tasks after transformed in the lower layers where the semantic similarities are not always important. Indeed, the results of the semantic tasks are very poor without the shortcut connections.

:

Wieting et al. (2016) used a nonlinearity, but we have observed that the simple averaging also works well.

This method currently assumes that each word has only one parent node, but it can be expanded to handle multiple parent nodes, which leads to cyclic graphs.

This modification does not affect the LSTM transitions, and thus it is still possible to add other singlesentence-level tasks on top of our model.

It is empirically known that such a small window size in leads to better results on syntactic tasks than large window sizes. Moreover, we have found that such word embeddings work well even on the semantic tasks.

The development and test sentences of the chunking dataset are included in the dependency parsing dataset, although our model does not explicitly use the chunking annotations of the development and test data. In such cases, we show the results in parentheses.

Choe & Charniak (2016) employed the tri-training technique to expand the training data with automatically-generated 400,000 trees in addition to the WSJ data, and they reported 95.9 UAS and 94.1 LAS.

The result of "JMT all " is slightly worse than that of "JMTDE", but the difference is not significant because the training data is small.

Figures and Tables:

Figure 1: Overview of the joint many-task model predicting different linguistic outputs at successively deeper layers.

Figure 2: Overview of the POS tagging and chunking tasks in the first and second layers of the JMT model.

Figure 4: Overview of the semantic tasks in the top layers of the JMT model.

where W d is a parameter matrix. For the root, we

Table content: <table><tr><td colspan="6">Dependency Parsing:</td><td/><td colspan="2">softmax</td></tr><tr><td/><td/><td/><td/><td/><td colspan="2">softmax</td><td/><td>softmax</td></tr><tr><td/><td/><td/><td colspan="2">h</td><td>(3) 1</td><td>h</td><td>(3) 2</td><td>h</td><td>(3) 3</td><td>h</td><td>(3) 4</td></tr><tr><td/><td/><td colspan="4">LSTM</td><td colspan="2">LSTM</td><td>LSTM</td><td>LSTM</td></tr><tr><td/><td>x 1</td><td>h</td><td>(2) 1 y</td><td colspan="2">1 (chk) (pos) y 1</td><td/><td/></tr><tr><td colspan="9">Figure 3: Overview of dependency parsing in the third layer of the JMT model.</td></tr><tr><td colspan="2">Semantic</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">relatedness:</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>softmax</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">Feature extracton</td></tr><tr><td/><td/><td/><td/><td/><td>temporal max-pooling</td><td/><td/><td>temporal max-pooling</td></tr><tr><td>h</td><td>(4) 1</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">LSTM</td><td/><td colspan="3">LSTM</td><td colspan="2">LSTM</td><td>LSTM</td><td>LSTM</td><td>LSTM</td></tr><tr><td>x 1</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td colspan="4">Sentence1</td><td/><td/><td>Sentence2</td></tr></table>

Single JMT all JMT AB JMT ABC JMT DE Test set results for the five tasks. In the relatedness task, the lower scores are better.

Table content: <table><tr><td colspan="2">A POS</td><td/><td>97.45</td><td>97.55</td><td>97.52</td><td>97.54</td><td>n/a</td></tr><tr><td colspan="3">B Chunking</td><td colspan="2">95.02 (97.12)</td><td>95.77</td><td>(97.28)</td><td>n/a</td></tr><tr><td>C</td><td colspan="3">Dependency UAS 93.35 Dependency LAS 91.42</td><td>94.67 92.90</td><td>n/a n/a</td><td>94.71 92.92</td><td>n/a n/a</td></tr><tr><td colspan="3">D Relatedness</td><td>0.247</td><td>0.233</td><td>n/a</td><td>n/a</td><td>0.238</td></tr><tr><td colspan="3">E Entailment</td><td>81.8</td><td>86.2</td><td>n/a</td><td>n/a</td><td>86.8</td></tr><tr><td>Method</td><td/><td>Acc.</td><td/><td/><td/></tr><tr><td>JMT all</td><td/><td>97.55</td><td/><td/><td/></tr><tr><td colspan="2">Ling et al. (2015)</td><td>97.78</td><td/><td/><td/></tr><tr><td colspan="2">Kumar et al. (2016)</td><td>97.56</td><td/><td/><td/></tr><tr><td colspan="2">Ma &amp; Hovy (2016)</td><td>97.55</td><td/><td/><td/></tr><tr><td>Søgaard (2011)</td><td/><td>97.50</td><td/><td/><td/></tr><tr><td colspan="2">Collobert et al. (2011)</td><td>97.29</td><td/><td/><td/></tr><tr><td colspan="2">Tsuruoka et al. (2011)</td><td>97.28</td><td/><td/><td/></tr><tr><td colspan="2">Toutanova et al. (2003)</td><td>97.27</td><td/><td/><td/></tr></table>

POS tagging results.

Table content: <table><tr><td>Method</td><td>F1</td></tr><tr><td>JMTAB</td><td>95.77</td></tr><tr><td>Søgaard &amp; Goldberg (2016)</td><td>95.56</td></tr><tr><td>Suzuki &amp; Isozaki (2008)</td><td>95.15</td></tr><tr><td>Collobert et al. (2011)</td><td>94.32</td></tr><tr><td>Kudo &amp; Matsumoto (2001)</td><td>93.91</td></tr><tr><td>Tsuruoka et al. (2011)</td><td>93.81</td></tr></table>

Chunking results.

Table content: <table><tr><td>Method</td><td>UAS</td><td>LAS</td></tr><tr><td>JMT all</td><td>94.67</td><td>92.90</td></tr><tr><td>Single</td><td>93.35</td><td>91.42</td></tr><tr><td>Andor et al. (2016)</td><td>94.61</td><td>92.79</td></tr><tr><td>Alberti et al. (2015)</td><td>94.23</td><td>92.36</td></tr><tr><td>Weiss et al. (2015)</td><td>93.99</td><td>92.05</td></tr><tr><td>Dyer et al. (2015)</td><td>93.10</td><td>90.90</td></tr><tr><td>Bohnet (2010)</td><td>92.88</td><td>90.71</td></tr></table>

Dependency results.

Table content: <table><tr><td>Method</td><td>MSE</td></tr><tr><td>JMT all</td><td>0.233</td></tr><tr><td>JMTDE</td><td>0.238</td></tr><tr><td>Zhou et al. (2016)</td><td>0.243</td></tr><tr><td>Tai et al. (2015)</td><td>0.253</td></tr></table>

Semantic relatedness results.

Table content: <table><tr><td>Method</td><td>Acc.</td></tr><tr><td>JMT all</td><td>86.2</td></tr><tr><td>JMTDE</td><td>86.8</td></tr><tr><td>Yin et al. (2016)</td><td>86.2</td></tr><tr><td>Lai &amp; Hockenmaier (2014)</td><td>84.6</td></tr></table>

Effects of depth for the single task settings.

Table content: <table><tr><td/><td>Single</td><td>Single+</td></tr><tr><td>POS</td><td/><td>97.52</td></tr><tr><td>Chunking</td><td>95.65</td><td>96.08</td></tr><tr><td>Dependency UAS</td><td>93.38</td><td>93.88</td></tr><tr><td>Dependency LAS</td><td>91.37</td><td>91.83</td></tr><tr><td>Relatedness</td><td>0.239</td><td>0.665</td></tr><tr><td>Entailment</td><td>83.8</td><td>66.4</td></tr></table>

Effectiveness of the Successive Regularization (SR).

Table content: <table><tr><td/><td>JMT all</td><td>w/o SR</td></tr><tr><td>POS</td><td>97.88</td><td>97.85</td></tr><tr><td>Chunking</td><td>97.59</td><td>97.13</td></tr><tr><td>Dependency UAS</td><td>94.51</td><td>94.46</td></tr><tr><td>Dependency LAS</td><td>92.60</td><td>92.57</td></tr><tr><td>Relatedness</td><td>0.236</td><td>0.239</td></tr><tr><td>Entailment</td><td>84.6</td><td>84.2</td></tr></table>

Effectiveness of the Shortcut Connections (SC).

Table content: <table><tr><td/><td>all</td><td>w/o SC</td></tr><tr><td>POS</td><td>97.88</td><td>97.79</td></tr><tr><td>Chunking</td><td>97.59</td><td>97.08</td></tr><tr><td>Dependency UAS</td><td>94.51</td><td>94.52</td></tr><tr><td>Dependency LAS</td><td>92.60</td><td>92.62</td></tr><tr><td>Relatedness</td><td>0.236</td><td>0.698</td></tr><tr><td>Entailment</td><td>84.6</td><td>75.0</td></tr></table>

Effectiveness of the Label Embeddings (LE).

Table content: <table><tr><td/><td>JMT all</td><td>w/o LE</td><td>w/o SC&amp;LE</td></tr><tr><td>POS</td><td>97.88</td><td>97.85</td><td>97.87</td></tr><tr><td>Chunking</td><td>97.59</td><td>97.40</td><td>97.33</td></tr><tr><td>Dependency UAS</td><td>94.51</td><td>94.09</td><td>94.04</td></tr><tr><td>Dependency LAS</td><td>92.60</td><td>92.14</td><td>92.03</td></tr><tr><td>Relatedness</td><td>0.236</td><td>0.261</td><td>0.765</td></tr><tr><td>Entailment</td><td>84.6</td><td>81.6</td><td>71.2</td></tr></table>

Effectiveness of the character n-gram embeddings.

Table content: <table><tr><td>Single</td><td>W&amp;C</td><td>Only W</td></tr><tr><td>POS</td><td>97.52</td><td>96.26</td></tr><tr><td>Chunking</td><td>95.65</td><td>94.92</td></tr><tr><td>Dependency UAS</td><td>93.38</td><td>92.90</td></tr><tr><td>Dependency LAS</td><td>91.37</td><td>90.44</td></tr></table>

Effectiveness of using different layers for different tasks.

Table content: <table><tr><td/><td>JMTABC</td><td>w/o SC&amp;LE</td><td>All-3</td></tr><tr><td>POS</td><td>97.90</td><td>97.87</td><td>97.62</td></tr><tr><td>Chunking</td><td>97.80</td><td>97.41</td><td>96.52</td></tr><tr><td>Dependency UAS</td><td>94.52</td><td>94.13</td><td>93.59</td></tr><tr><td>Dependency LAS</td><td>92.61</td><td>92.16</td><td>91.47</td></tr></table>

In our experiments on five different types of NLP tasks, our single model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness, and textual entailment. POS tagging scores on the development set with and without the character n-gram embeddings, focusing on accuracy for unknown words. The overall accuracy scores are taken from Table11. There are 3,862 unknown words in the sentences of the development set.

Table content: <table><tr><td colspan="5">Single (POS) Overall Acc. Acc. for unknown words</td></tr><tr><td>W&amp;C</td><td/><td>97.52</td><td>90.68 (3,502/3,862)</td></tr><tr><td>Only W</td><td/><td>96.26</td><td>71.44 (2,759/3,862)</td></tr><tr><td/><td colspan="2">Overall scores</td><td colspan="2">Scores for unknown words</td></tr><tr><td colspan="2">Single (Dependency) UAS</td><td>LAS</td><td>UAS</td><td>LAS</td></tr><tr><td>W&amp;C</td><td colspan="4">93.38 91.37 92.21 (900/976) 87.81 (857/976)</td></tr><tr><td>Only W</td><td colspan="4">92.90 90.44 91.39 (892/976) 81.05 (791/976)</td></tr></table>

Dependency parsing scores on the development set with and without the character n-gram embeddings, focusing on UAS and LAS for unknown words. The overall scores are taken from Table11. There are 976 unknown words in the sentences of the development set.

Table content: <table/>

Closest neighbors of the word "standing" in the embedding space and the projected space in each forward LSTM.

Table content: <table><tr><td/><td>Word and char</td><td>Only word</td></tr><tr><td/><td>leaning</td><td>stood</td></tr><tr><td/><td>kneeling</td><td>stands</td></tr><tr><td>Embedding</td><td>saluting</td><td>sit</td></tr><tr><td/><td>clinging</td><td>pillar</td></tr><tr><td/><td>railing</td><td>cross-legged</td></tr><tr><td/><td>warning</td><td>ladder</td></tr><tr><td/><td>waxing</td><td>rc6280</td></tr><tr><td>POS</td><td>dunking</td><td>bethle</td></tr><tr><td/><td>proving</td><td>warning</td></tr><tr><td/><td>tipping</td><td>f-a-18</td></tr><tr><td/><td>applauding</td><td>fight</td></tr><tr><td/><td>disdaining</td><td>favor</td></tr><tr><td>Chunking</td><td>pickin</td><td>pick</td></tr><tr><td/><td>readjusting</td><td>rejoin</td></tr><tr><td/><td>reclaiming</td><td>answer</td></tr><tr><td/><td>guaranteeing</td><td>patiently</td></tr><tr><td/><td>resting</td><td>hugging</td></tr><tr><td colspan="2">Dependency grounding</td><td>anxiously</td></tr><tr><td/><td>hanging</td><td>resting</td></tr><tr><td/><td>hugging</td><td>disappointment</td></tr><tr><td/><td>stood</td><td>stood</td></tr><tr><td/><td>stands</td><td>unchallenged</td></tr><tr><td colspan="2">Relatedness unchallenged</td><td>stands</td></tr><tr><td/><td colspan="2">notwithstanding beside</td></tr><tr><td/><td>judging</td><td>exists</td></tr><tr><td/><td>nudging</td><td>beside</td></tr><tr><td/><td>skirting</td><td>stands</td></tr><tr><td>Entailment</td><td>straddling</td><td>pillar</td></tr><tr><td/><td>contesting</td><td>swung</td></tr><tr><td/><td>footing</td><td>ovation</td></tr></table>

Peer review (iclr2017):
The authors propose a transfer learning approach applied to a number of NLP tasks; the set of tasks appear to have an order in terms of complexity (from easy syntactic tasks to somewhat harder semantic tasks). Novelty: the way the authors propose to do transfer learning is by plugging models corresponding to each task, in a way that respects the known hierarchy (in terms of NLP *complexity*) of those tasks. In that respect, the overall architecture looks more like a cascaded architecture than a transfer learning one. There are some existing literature in the area (first two Google results found: https://arxiv.org/pdf/1512.04412v1.pdf, (computer vision) and https://www.aclweb.org/anthology/P/P16/P16-1147.pdf (NLP)). In addition to the architecture, the authors propose a regularization technique they call *successive regularization*. Experiments: - The authors performed a number of experimental analysis to clarify what parts of their architecture are important, which is very valuable; - The information *transferred* from one task to the next one is represented both using a smooth label embedding and the hidden representation of the previous task. At this point there is no analysis of which one is actually important, or if they are redundant (update: the authors mentioned they would add something there). Also, it is likely one would have tried first to feed label scores from one task to the next one, instead of using the trick of the label embedding -- it is unclear what the latter is actually bringing. - The successive regularization does not appear to be important in Table 8; a variance analysis would help to conclude.