Introduction
Deep learning models have been widely used in many natural language processing (NLP) tasks. A major challenge is how to design and learn the semantic composition function while modeling a text sequence. The typical composition models involve sequential BIBREF0 , BIBREF1 , convolutional BIBREF2 , BIBREF3 , BIBREF4 and syntactic BIBREF5 , BIBREF6 , BIBREF7 compositional models.
In spite of their success, these models have two major limitations. First, they usually use a shared composition function for all kinds of semantic compositions, even though the compositions have different characteristics in nature. For example, the composition of the adjective and the noun differs significantly from the composition of the verb and the noun. Second, different composition functions are learned from scratch in different tasks. However, given a certain natural language, its composition functions should be the same (on meta-knowledge level at least), even if the tasks are different.
To address these problems, we need to design a dynamic composition function which can vary with different positions and contexts in a sequence, and share it across the different tasks. To share some meta-knowledge of composition function, we can adopt the multi-task learning BIBREF8 . However, the sharing scheme of most neural multi-task learning methods is feature-level sharing, where a subspace of the feature space is shared across all the tasks. Although these sharing schemes are successfully used in various NLP tasks BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , they are not suitable to share the composition function.
In this paper, inspired by recent work on dynamic parameter generation BIBREF15 , BIBREF16 , BIBREF17 , we propose a function-level sharing scheme for multi-task learning, in which a shared meta-network is used to learn the meta-knowledge of semantic composition among the different tasks. The task-specific semantic composition function is generated by the meta-network. Then the task-specific composition function is used to obtain the task-specific representation of a text sequence. The difference between two sharing schemes is shown in Figure 1 . Specifically, we use two LSTMs as meta and basic (task-specific) network respectively. The meta LSTM is shared for all the tasks. The parameters of the basic LSTM are generated based on the current context by the meta LSTM, therefore the composition function is not only task-specific but also position-specific. The whole network is differentiable with respect to the model parameters and can be trained end-to-end.
We demonstrate the effectiveness of our architectures on two kinds of NLP tasks: text classification and sequence tagging. Experimental results show that jointly learning of multiple related tasks can improve the performance of each task relative to learning them independently.
Our contributions are of three-folds:
Generic Neural Architecture of Multi-Task Learning for Sequence Modeling
In this section, we briefly describe generic neural architecture of multi-task learning .
The generic neural architecture of multi-task learning is to share some lower layers to determine common features. After the shared layers, the remaining higher layers are parallel and independent respective to each specific task. Figure 2 illustrates the generic architecture of multi-task learning. BIBREF9 , BIBREF11 , BIBREF12
There are many neural sentence models, which can be used for sequence modeling, including recurrent neural networks BIBREF0 , BIBREF1 , convolutional neural networks BIBREF2 , BIBREF3 , and recursive neural networks BIBREF5 . Here we adopt recurrent neural network with long short-term memory (LSTM) due to their superior performance in various NLP tasks.
LSTM BIBREF19 is a type of recurrent neural network (RNN), and specifically addresses the issue of learning long-term dependencies. While there are numerous LSTM variants, here we use the LSTM architecture used by BIBREF20 , which is similar to the architecture of BIBREF21 but without peep-hole connections.
We define the LSTM units at each time step $t$ to be a collection of vectors in $\mathbb {R}^h$ : an input gate $\mathbf {i}_t$ , a forget gate $\textbf {f}_t$ , an output gate $\mathbf {o}_t$ , a memory cell $\mathbf {c}_t$ and a hidden state $\textbf {h}_t$ . $d$ is the number of the LSTM units. The elements of the gating vectors $\mathbf {i}_t$ , $\textbf {f}_t$ and $\mathbb {R}^h$0 are in $\mathbb {R}^h$1 .
The LSTM is compactly specified as follows.
$$\begin{bmatrix} \mathbf {g}_{t} \\ \mathbf {o}_{t} \\ \mathbf {i}_{t} \\ \mathbf {f}_{t} \end{bmatrix} &= \begin{bmatrix} \tanh \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \begin{pmatrix} \mathbf {W}\begin{bmatrix} \mathbf {x}_{t} \\ \mathbf {h}_{t-1} \end{bmatrix}+\mathbf {b}\end{pmatrix}, \\ \mathbf {c}_{t} &= \mathbf {g}_{t} \odot \mathbf {i}_{t} + \mathbf {c}_{t-1} \odot \mathbf {f}_{t}, \\ \mathbf {h}_{t} &= \mathbf {o}_{t} \odot \tanh \left( \mathbf {c}_{t} \right),$$   (Eq. 9)
where $\mathbf {x}_t \in \mathbb {R}^{d}$ is the input at the current time step; $\mathbf {W}\in \mathbb {R}^{4h\times (h+d)}$ and $\mathbf {b}\in \mathbb {R}^{4h}$ are parameters of affine transformation; $\sigma $ denotes the logistic sigmoid function and $\odot $ denotes elementwise multiplication.
The update of each LSTM unit can be written precisely as follows:
$$\textbf {h}_t &= \mathbf {LSTM}(\textbf {h}_{t-1},\mathbf {x}_t, \theta ). $$   (Eq. 10)
Here, the function $\mathbf {LSTM}(\cdot , \cdot , \cdot )$ is a shorthand for Eq. ( 9 -), and $\theta $ represents all the parameters of LSTM.
Given a text sequence $X = \lbrace x_1, x_2, \cdots , x_T\rbrace $ , we first use a lookup layer to get the vector representation (embeddings) $\mathbf {x}_t$ of each word $x_t$ . The output at the last moment $\textbf {h}_T$ can be regarded as the representation of the whole sequence.
To exploit the shared information between these different tasks, the general deep multi-task architecture consists of a private (task-specific) layer and a shared (task-invariant) layer. The shared layer captures the shared information for all the tasks.
The shared layer and private layer is arranged in stacked manner. The private layer takes the output of the shared layer as input. For task $k$ , the hidden states of shared layer and private layer are:
$$\textbf {h}^{(s)}_t& = \text{LSTM}(\mathbf {x}_{t}, \textbf {h}^{(s)}_{t-1},\theta _s),\\ \textbf {h}^{(k)}_t &= \text{LSTM}(\begin{bmatrix} \mathbf {x}_{t}\\ \textbf {h}^{(s)}_t \end{bmatrix}, \textbf {h}^{(k)}_{t-1},\theta _k)$$   (Eq. 13)
where $\textbf {h}^{(s)}_t$ and $\textbf {h}^{(k)}_t$ are hidden states of the shared layer and the $k$ -th task-specific layer respectively; $\theta _s$ and $\theta _k$ denote their parameters.
The task-specific representations $\textbf {h}^{(k)}$ , which is emitted by the multi-task architecture, are ultimately fed into different task-specific output layers.
Here, we use two kinds of tasks: text classification and sequence tagging.
For task $k$ in , the label predictor is defined as
$${\hat{\mathbf {y}}}^{(k)} = \textbf {softmax}(\mathbf {W}^{(k)}\textbf {h}^{(k)} + \mathbf {b}^{(k)}),$$   (Eq. 16)
where ${\hat{\mathbf {y}}}^{(k)}$ is prediction probabilities for task $k$ , $\mathbf {W}^{(k)}$ is the weight matrix which needs to be learned, and $\mathbf {b}^{(k)}$ is a bias term.
Following the idea of BIBREF22 , BIBREF23 , we use a conditional random field (CRF) BIBREF24 as output layer.
Task Definition
The task of Sequence Modeling is to assign a label sequence $Y=\lbrace y_1,y_2,\cdots ,y_T\rbrace $ . to a text sequence $X=\lbrace x_1,x_2,\cdots ,x_T\rbrace $ . In classification task, $Y$ is a single label. Assuming that there are $K$ related tasks, we refer $\mathcal {D}_k$ as the corpus of the $k$ -th task with $N_k$ samples:
$$\mathcal {D}_k = \lbrace (X_i^{(k)},Y_i^{(k)})\rbrace _{i=1}^{N_k},$$   (Eq. 6)
where $X_i^k$ and $Y_i^k$ denote the $i$ -th sample and its label respectively in the $k$ -th task.
Multi-task learning BIBREF8 is an approach to learn multiple related tasks simultaneously to significantly improve performance relative to learning each task independently. The main challenge of multi-task learning is how to design the sharing scheme. For the shallow classifier with discrete features, it is relatively difficult to design the shared feature spaces, usually resulting in a complex model. Fortunately, deep neural models provide a convenient way to share information among multiple tasks.
Training
The parameters of the network are trained to minimise the cross-entropy of the predicted and true distributions for all tasks.
$$\mathcal {L}(\Theta ) = -\sum _{k=1}^{K} {\lambda }_k \sum _{i=1}^{N_k} \mathbf {y}_i^{(k)} \log (\hat{\mathbf {y}}_i^{(k)}),$$   (Eq. 19)
where $\lambda _k$ is the weights for each task $k$ respectively; $\mathbf {y}_i^{(k)}$ is the one-hot vector of the ground-truth label of the sample $X_i^{(k)}$ ; $\hat{y}_i^{(k)}$ is its prediction probabilities.
It is worth noticing that labeled data for training each task can come from completely different datasets. Following BIBREF9 , the training is achieved in a stochastic manner by looping over the tasks:
Select a random task.
Select a mini-batch of examples from this task.
Update the parameters for this task by taking a gradient step with respect to this mini-batch.
Go to 1.
After the joint learning phase, we can use a fine tuning strategy to further optimize the performance for each task.
Meta Multi-Task Learning
In this paper, we take a very different multi-task architecture from meta-learning perspective BIBREF25 . One goal of meta-learning is to find efficient mechanisms to transfer knowledge across domains or tasks BIBREF26 .
Different from the generic architecture with the representational sharing (feature sharing) scheme, our proposed architecture uses a functional sharing scheme, which consists of two kinds of networks. As shown in Figure 3 , for each task, a basic network is used for task-specific prediction, whose parameters are controlled by a shared meta network across all the tasks.
We firstly introduce our architecture on single task, then apply it for multi-task learning.
Meta-LSTMs for Single Task
Inspired by recent work on dynamic parameter prediction BIBREF15 , BIBREF16 , BIBREF17 , we also use a meta network to generate the parameters of the task network (basic network). Specific to text classification, we use LSTM for both the networks in this paper, but other options are possible.
There are two networks for each single task: a basic LSTM and a meta LSTM.
For each specific task, we use a basic LSTM to encode the text sequence. Different from the standard LSTM, the parameters of the basic LSTM is controlled by a meta vector $\mathbf {z}_t$ , generated by the meta LSTM. The new equations of the basic LSTM are
$$\begin{bmatrix} \mathbf {g}_{t} \\ \mathbf {o}_{t} \\ \mathbf {i}_{t} \\ \mathbf {f}_{t} \end{bmatrix} &= \begin{bmatrix} \tanh \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \begin{pmatrix} \mathbf {W}(\mathbf {z}_t) \begin{bmatrix} \mathbf {x}_{t} \\ \mathbf {h}_{t-1} \end{bmatrix} + \mathbf {b}(\mathbf {z}_t) \end{pmatrix}, \\ \mathbf {c}_{t} &= \mathbf {g}_{t} \odot \mathbf {i}_{t} + \mathbf {c}_{t-1} \odot \mathbf {f}_{t}, \\ \mathbf {h}_{t} &= \mathbf {o}_{t} \odot \tanh \left( \mathbf {c}_{t} \right),$$   (Eq. 27)
where $\mathbf {W}(\mathbf {z}_t): \mathbb {R}^z\rightarrow \mathbb {R}^{4h\times (h+d)}$ and $\mathbf {b}(\mathbf {z}_t): \mathbb {R}^z\rightarrow \mathbb {R}^{4h}$ are dynamic parameters controlled by the meta network.
Since the output space of the dynamic parameters $\mathbf {W}(\mathbf {z}_t)$ is very large, its computation is slow without considering matrix optimization algorithms. Moreover, the large parameters makes the model suffer from the risk of overfitting. To remedy this, we define $\mathbf {W}(\mathbf {z}_t)$ with a low-rank factorized representation of the weights, analogous to the Singular Value Decomposition.
The parameters $\mathbf {W}(\mathbf {z}_t)$ and $\mathbf {b}(\mathbf {z}_t)$ of the basic LSTM are computed by
$$\mathbf {W}(\mathbf {z}_t) &= \begin{bmatrix} P_c \mathbf {D}(\mathbf {z}_t) Q_c\\ P_o \mathbf {D}(\mathbf {z}_t) Q_o\\ P_i \mathbf {D}(\mathbf {z}_t) Q_i\\ P_f \mathbf {D}(\mathbf {z}_t) Q_f  \end{bmatrix}\\ \mathbf {b}(\mathbf {z}_t)&=\begin{bmatrix} B_c \mathbf {z}_t\\ B_o \mathbf {z}_t\\ B_i \mathbf {z}_t\\ B_f \mathbf {z}_t \end{bmatrix}$$   (Eq. 28)
where $P_*\in \mathbb {R}^{h\times z}$ , $Q_*\in \mathbb {R}^{z\times d}$ and $B_*\in \mathbb {R}^{h\times z}$ are parameters for $*\in \lbrace c,o,i,f\rbrace $ .
Thus, our basic LSTM needs $(8hz + 4dz)$ parameters, while the standard LSTM has $(4h^2+4hd+4h)$ parameters. With a small $z$ , the basic LSTM needs less parameters than the standard LSTM. For example, if we set $d = h = 100$ and $z=20$ , our basic LSTM just needs $24,000$ parameter while the standard LSTM needs $80,400$ parameters.
The Meta-LSTM is usually a smaller network, which depends on the input $\mathbf {x}_t$ and the previous hidden state $\textbf {h}_{t-1}$ of the basic LSTM.
The Meta-LSTM cell is given by:
$$\begin{bmatrix} \hat{\mathbf {g}}_{t} \\ \hat{\mathbf {o}}_{t} \\ \hat{\mathbf {i}}_{t} \\ \hat{\mathbf {f}}_{t} \end{bmatrix} &= \begin{bmatrix} \tanh \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \begin{pmatrix} \mathbf {W}_m \begin{bmatrix} \mathbf {x}_{t} \\ \mathbf {\hat{h}}_{t-1}\\ \mathbf {h}_{t-1} \end{bmatrix}+\mathbf {b}_m \end{pmatrix}, \\ \hat{\mathbf {c}}_{t} &= \hat{\mathbf {g}}_{t} \odot \hat{\mathbf {i}}_{t} + \hat{\mathbf {c}}_{t-1} \odot \hat{\mathbf {f}}_{t}, \\ \hat{\mathbf {h}}_{t} &= \hat{\mathbf {o}}_{t} \odot \tanh \left( \hat{\mathbf {c}}_{t} \right),\\ \mathbf {z}_t &= \mathbf {W}_z \hat{\mathbf {h}}_{t},$$   (Eq. 30)
where $\mathbf {W}_{m} \in \mathbb {R}^{4m \times (d+h+m)}$ and $\mathbf {b}_{m} \in \mathbb {R}^{4m}$ are parameters of Meta-LSTM; $\mathbf {W}_{z} \in \mathbb {R}^{z \times m}$ is a transformation matrix.
Thus, the Meta-LSTM needs $(4m(d+h+m+1)+mz)$ parameters. When $d = h = 100$ and $z=m=20$ , its parameter number is $18,080$ . The total parameter number of the whole networks is $42,080$ , nearly half of the standard LSTM.
We precisely describe the update of the units of the Meta-LSTMs as follows:
$$[\hat{\textbf {h}}_t , \mathbf {z}_t] & = \text{Meta-LSTM}(\mathbf {x}_{t}, \hat{\textbf {h}}_{t-1},\textbf {h}_{t-1};\theta _m),\\ \textbf {h}_t &= \text{Basic-LSTM}(\mathbf {x}_{t}, \textbf {h}_{t-1};\mathbf {z}_t, \theta _b)$$   (Eq. 31)
where $\theta _m$ and $\theta _b$ denote the parameters of the Meta-LSTM and Basic-LSTM respectively.
Compared to the standard LSTM, the Meta-LSTMs have two advantages. One is the parameters of the Basic-LSTM is dynamically generated conditioned on the input at the position, while the parameters of the standard LSTM are the same for all the positions, even though different positions have very different characteristics. Another is that the Meta-LSTMs usually have less parameters than the standard LSTM.
Meta-LSTMs for Multi-Task Learning
For multi-task learning, we can assign a basic network to each task, while sharing a meta network among tasks. The meta network captures the meta (shared) knowledge of different tasks. The meta network can learn at the “meta-level” of predicting parameters for the basic task-specific network.
For task $k$ , the hidden states of the shared layer and the private layer are:
$$[\hat{\textbf {h}}^{(s)}_t, \mathbf {z}^{(s)}_t]& = \text{Meta-LSTM}(\mathbf {x}_{t}, \hat{\textbf {h}}^{(s)}_{t-1},\textbf {h}^{(k)}_{t-1};\theta ^{(s)}_m),\\ \textbf {h}^{(k)}_t &= \text{Basic-LSTM}(\mathbf {x}_{t}, \textbf {h}^{(k)}_{t-1};\mathbf {z}^{(s)}_t, \theta ^{(k)}_b)$$   (Eq. 33)
where $\hat{\textbf {h}}^{(s)}_t$ and $\textbf {h}^{(k)}_t$ are the hidden states of the shared meta LSTM and the $k$ -th task-specific basic LSTM respectively; $\theta ^{(s)}_m$ and $\theta ^{(k)}_b$ denote their parameters. The superscript $(s)$ indicates the parameters or variables are shared across the different tasks.
Experiment
In this section, we investigate the empirical performances of our proposed model on two multi-task datasets. Each dataset contains several related tasks.
Exp-I: Multi-task Learning of text classification
We first conduct our experiment on classification tasks.
For classification task, we test our model on 16 classification datasets, the first 14 datasets are product reviews that collected based on the dataset, constructed by BIBREF27 , contains Amazon product reviews from different domains: Books, DVDs, Electronics and Kitchen and so on. The goal in each domain is to classify a product review as either positive or negative. The datasets in each domain are partitioned randomly into training data, development data and testing data with the proportion of 70%, 10% and 20% respectively. The detailed statistics are listed in Table 1 .
The remaining two datasets are two sub-datasets about movie reviews.
IMDB The movie reviews with labels of subjective or objective BIBREF28 .
MR The movie reviews with two classes BIBREF29 .
For single-task learning, we compare our Meta-LSTMs with three models.
LSTM: the standard LSTM with one hidden layer;
HyperLSTMs: a similar model which also uses a small network to generate the weights for a larger network BIBREF17 .
For multi-task learning, we compare our Meta-LSTMs with the generic shared-private sharing scheme.
ASP-MTL: Proposed by BIBREF30 , using adversarial training method on PSP-MTL.
PSP-MTL: Parallel shared-private sharing scheme, using a fully-shared LSTM to extract features for all tasks and concatenate with the outputs from task-specific LSTM.
SSP-MTL: Stacked shared-private sharing scheme, introduced in Section 2.
The networks are trained with backpropagation and the gradient-based optimization is performed using the Adagrad update rule BIBREF31 .
The word embeddings for all of the models are initialized with the 200d GloVe vectors (6B token version, BIBREF32 ) and fine-tuned during training to improve the performance. The mini-batch size is set to 16. The final hyper-parameters are set as Table 2 .
Table 3 shows the classification accuracies on the tasks of product reviews.
The row of “Single Task” shows the results for single-task learning. With the help of Meta-LSTMs, the performances of the 16 subtasks are improved by an average of $3.2\%$ , compared to the standard LSTM. However, the number of parameters is a little more than standard LSTM and much less than the HyperLSTMs.
For multi-task Learning, our model also achieves a better performance than our competitor models, with an average improvement of $5.1\%$ to average accuracy of single task and $2.2\%$ to best competitor Multi-task model. The main reason is that our models can capture more abstractive shared information. With a meta LSTM to generate the matrices, the layer will become more flexible.
With the meta network, our model can use quite a few parameters to achieve the state-of-the-art performances.
We have experimented various $z$ size in our multi-task model, where $z \in [20, 30,...,60]$ , and the difference of the average accuracies of sixteen datasets is less than $0.8\%$ , which indicates that the meta network with less parameters can also generate a basic network with a considerable good performance.
To illustrates the insight of our model, we randomly sample a sequence from the development set of Toys task. In Figure 4 we predict the sentiment scores each time step. Moreover, to describe how our model works, we visualize the changes of matrices generated by Meta-LSTM, the changes $\textbf {diff}$ are calculate by Eq. 54 .
As we see it, the matrices change obviously facing the emotional vocabulary like ”friendly”, ”refund”, and slowly change to a normal state. They can also capture words that affect sentiments like ”not”. For this case, SSP-MTL give a wrong answer, it captures the emotion word”refund”, but it makes an error on pattern ”not user friendly”, we consider that it's because fixed matrices don't have satisfactorily ability to capture long patterns' emotions and information. Dynamic matrices generated by Meta-LSTM will make the layer more flexible.
$$\mathbf {diff}^{(k)} = \textbf {mean}(\frac{\textbf {abs}(\mathbf {W}^{(k)}-\mathbf {W}^{(k-1)})}{\textbf {abs}(\mathbf {W}^{(k-1)})}),$$   (Eq. 54)
Figure 5 shows the learning curves of various multi-task model on the 16 classification datasets.
Because it's inappropriate to evaluate different tasks every training step during shared parameters training since mini-batch of which tasks are selected randomly, so we use the average loss after every epoch. We can find that our proposed model is more efficient to fit the train datasets than our competitor models, and get better performance on the dev datasets. Therefore, we can consider that our model could learn shareable knowledge more effectively.
Since our Meta-LSTM captures some meta knowledge of semantic composition, which should have an ability of being transfered to a new task. Under this view, a new task can no longer be simply seen as an isolated task that starts accumulating knowledge afresh. As more tasks are observed, the learning mechanism is expected to benefit from previous experience.
The meta network can be considered as off-the-shelf knowledge and then be used for unseen new tasks.
To test the transferability of our learned Meta-LSTM, we also design an experiment, in which we take turns choosing 15 tasks to train our model with multi-task learning, then the learned Meta-LSTM are transferred to the remaining one task. The parameters of transferred Meta-LSTM, $\theta ^{(s)}_m$ in Eq.( 33 ), are fixed and cannot be updated on the new task.
The results are also shown in the last column of Table 3 . With the help of meta knowledge, we observe an average improvement of $3.1\%$ over the average accuracy of single models, and even better than other competitor multi-task models. This observation indicates that we can save the meta knowledge into a meta network, which is quite useful for a new task.
Exp-II: Multi-task Learning of Sequence Tagging
In this section, we conduct experiment for sequence tagging. Similar to BIBREF22 , BIBREF23 , we use the bi-directional Meta-LSTM layers to encode the sequence and a conditional random field (CRF) BIBREF24 as output layer. The hyperparameters settings are same to Exp-I, but with 100d embedding size and 30d Meta-LSTM size.
For sequence tagging task, we use the Wall Street Journal(WSJ) portion of Penn Treebank (PTB) BIBREF33 , CoNLL 2000 chunking, and CoNLL 2003 English NER datasets. The statistics of these datasets are described in Table 4 .
Table 5 shows the accuracies or F1 scores on the sequence tagging datasets of our models, compared to some state-of-the-art results. As shown, our proposed Meta-LSTM performs better than our competitor models whether it is single or multi-task learning.
Result Analysis
From the above two experiments, we have empirically observed that our model is consistently better than the competitor models, which shows our model is very robust. Explicit to multi-task learning, our model outperforms SSP-MTL and PSP-MTL by a large margin with fewer parameters, which indicates the effectiveness of our proposed functional sharing mechanism.
Related Work
One thread of related work is neural networks based multi-task learning, which has been proven effective in many NLP problems BIBREF9 , BIBREF34 , BIBREF11 , BIBREF12 . In most of these models, the lower layers are shared across all tasks, while top layers are task-specific. This kind of sharing scheme divide the feature space into two parts: the shared part and the private part. The shared information is representation-level, whose capacity grows linearly as the size of shared layers increases.
Different from these models, our model captures the function-level sharing information, in which a meta-network captures the meta-knowledge across tasks and controls the parameters of task-specific networks.
Another thread of related work is the idea of using one network to predict the parameters of another network. BIBREF15 used a filter-generating network to generate the parameters of another dynamic filter network, which implicitly learn a variety of filtering operations. BIBREF16 introduced a learnet for one-shot learning, which can predicts the parameters of a second network given a single exemplar. BIBREF17 proposed the model hypernetwork, which uses a small network to generate the weights for a larger network. In particular, their proposed hyperLSTMs is same with our Meta-LSTMs except for the computational formulation of the dynamic parameters. Besides, we also use a low-rank approximation to generate the parameter matrix, which can reduce greatly the model complexity, while keeping the model ability.
Conclusion and Future Work
In this paper, we introduce a novel knowledge sharing scheme for multi-task learning. The difference from the previous models is the mechanisms of sharing information among several tasks. We design a meta network to store the knowledge shared by several related tasks. With the help of the meta network, we can obtain better task-specific sentence representation by utilizing the knowledge obtained by other related tasks. Experimental results show that our model can improve the performances of several related tasks by exploring common features and outperforms the representational sharing scheme. The knowledge captured by the meta network can be transferred across other new tasks.
In future work, we would like to investigate other functional sharing mechanisms of neural network based multi-task learning.
Acknowledgement
We would like to thank the anonymous reviewers for their valuable comments. The research work is supported by the National Key Research and Development Program of China (No. 2017YFB1002104), Shanghai Municipal Science and Technology Commission (No. 17JC1404100), and National Natural Science Foundation of China (No. 61672162).

Question: What datasets do they use in the experiment?
Answer: Wall Street Journal(WSJ) portion of Penn Treebank (PTB) , CoNLL 2000 chunking, CoNLL 2003 English NER , Amazon product reviews from different domains: Books, DVDs, Electronics and Kitchen, IMDB The movie reviews with labels of subjective or objective, MR The movie reviews with two classes

CoNLL 2000 chunking, CoNLL 2003 English NER, Wall Street Journal(WSJ) portion of Penn Treebank (PTB), 14 datasets are product reviews, two sub-datasets about movie reviews

Question: What new tasks do they use to show the transferring ability of the shared meta-knowledge?
Answer: choosing 15 tasks to train our model with multi-task learning, then the learned Meta-LSTM are transferred to the remaining one task

we take turns choosing 15 tasks to train our model with multi-task learning, then the learned Meta-LSTM are transferred to the remaining one task.

Question: What kind of meta learning algorithm do they use?
Answer: a function-level sharing scheme for multi-task learning

a function-level sharing scheme for multi-task learning, in which a shared meta-network is used to learn the meta-knowledge of semantic composition among the different tasks