Question: What is the shared composition function used to do?
Evidence: Deep learning models have been widely used in many natural language processing (NLP) tasks. A major challenge is how to design and learn the semantic composition function while modeling a text sequence. The typical composition models involve sequential BIBREF0 , BIBREF1 , convolutional BIBREF2 , BIBREF3 , BIBREF4 and syntactic BIBREF5 , BIBREF6 , BIBREF7 compositional models.
In spite of their success, these models have two major limitations. First, they usually use a shared composition function for all kinds of semantic compositions, even though the compositions have different characteristics in nature. For example, the composition of the adjective and the noun differs significantly from the composition of the verb and the noun. Second, different composition functions are learned from scratch in different tasks. However, given a certain natural language, its composition functions should be the same (on meta-knowledge level at least), even if the tasks are different.
To address these problems, we need to design a dynamic composition function which can vary with different positions and contexts in a sequence, and share it across the different tasks. To share some meta-knowledge of composition function, we can adopt the multi-task learning BIBREF8 . However, the sharing scheme of most neural multi-task learning methods is feature-level sharing, where a subspace of the feature space is shared across all the tasks. Although these sharing schemes are successfully used in various NLP tasks BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , they are not suitable to share the composition function.
In this paper, inspired by recent work on dynamic parameter generation BIBREF15 , BIBREF16 , BIBREF17 , we propose a function-level sharing scheme for multi-task learning, in which a shared meta-network is used to learn the meta-knowledge of semantic composition among the different tasks. The task-specific semantic composition function is generated by the meta-network. Then the task-specific composition function is used to obtain the task-specific representation of a text sequence. The difference between two sharing schemes is shown in Figure 1 . Specifically, we use two LSTMs as meta and basic (task-specific) network respectively. The meta LSTM is shared for all the tasks. The parameters of the basic LSTM are generated based on the current context by the meta LSTM, therefore the composition function is not only task-specific but also position-specific. The whole network is differentiable with respect to the model parameters and can be trained end-to-end.
We demonstrate the effectiveness of our architectures on two kinds of NLP tasks: text classification and sequence tagging. Experimental results show that jointly learning of multiple related tasks can improve the performance of each task relative to learning them independently.
Our contributions are of three-folds:
In this paper, inspired by recent work on dynamic parameter generation, we propose a function-level sharing scheme for multi-task learning, in which a shared meta-network is used to learn the meta-knowledge of semantic composition among the different tasks.
Answer: It is used for multi-task learning, in which a shared meta-network is used to learn the meta-knowledge of semantic composition among the different tasks.