Question: What are the weaknesses of most existing models?
Evidence: Learning the distributed representation for long spans of text from its constituents has been a key step for various natural language processing (NLP) tasks, such as text classification BIBREF0 , BIBREF1 , semantic matching BIBREF2 , BIBREF3 , and machine translation BIBREF4 . Existing deep learning approaches take a compositional function with different forms to compose word vectors recursively until obtaining a sentential representation. Typically, these compositional functions involve recurrent neural networks BIBREF5 , BIBREF6 , convolutional neural networks BIBREF7 , BIBREF8 , and tree-structured neural networks BIBREF9 , BIBREF10 .
Among these methods, tree-structured neural networks (Tree-NNs) show theirs superior performance in many NLP tasks BIBREF11 , BIBREF12 . Following the syntactic tree structure, Tree-NNs assign a fixed-length vector to each word at the leaves of the tree, and combine word and phrase pairs recursively to create intermediate node vectors, eventually obtaining one final vector to represent the whole sentence.
However, these models have a major limitation in their inability to fully capture the richness of compositionality BIBREF13 . The same parameters are used for all kinds of semantic compositions, even though the compositions have different characteristics in nature. For example, the composition of the adjective and the noun differs significantly from the composition of the verb and the noun. Moreover, many semantic phenomena, such as semantic idiomaticity or transparency, call for more powerful compositional mechanisms BIBREF14 . Therefore, Tree-NNs suffer from the underfitting problem.
To alleviate this problem, some researchers propose to use multiple compositional functions, which are arranged beforehand according to some partition criterion BIBREF11 , BIBREF13 , BIBREF15 . Intuitively, using different parameters for different types of compositions has the potential to greatly reduce underfitting. BIBREF13 [ BIBREF13 ] defined different compositional functions in terms of syntactic categories, and a suitable compositional function is selected based on the syntactic categories. BIBREF15 [ BIBREF15 ] introduced multiple compositional functions and during compositional phase, a proper one is selected based on the input information. Although these models accomplished their mission to a certain extent, they still suffer from the following three challenges. First, the predefined compositional functions cannot cover all the compositional rules; Second, they require more learnable parameters, suffering from the problem of overfitting; Third, it is difficult to determine a universal criterion for semantic composition based solely on syntactic categories.
In this paper, we propose dynamic compositional neural networks over tree structure, in which a meta network is used to generate the context-specific parameters of a dynamic compositional network. Specifically, we construct our models based on two kinds of tree-structured neural networks: recursive neural network (Tree-RecNN) BIBREF11 and tree-structure long short-term memory neural network (Tree-LSTM) BIBREF9 . Our work is inspired by recent work on dynamic parameter prediction BIBREF16 , BIBREF17 , BIBREF18 . The meta network is used to extract the shared meta-knowledge across different compositional rules and to dynamically generate the context-specific compositional function. Thus, the compositional function of our models varies with positions, contexts and samples. The dynamic compositional network then applies those context-specific parameters to the current input information. Both meta and dynamic networks are differentiable such that the overall networks can be trained in an end-to-end fashion. Additional, to reduce the complexity of the whole networks, we define the dynamic weight matrix in a manner simulating low-rank matrix decomposition.
We evaluate our models on two typical tasks: text classification and text semantic matching. The results show that our models are more expressive due to their learning to learn nature, yet without increasing the number of model's parameters. Moreover, we find certain composition operations can be learned implicitly by meta TreeNN, such as the composition of noun phrases and verb phrases.
The contributions of the paper can be summed up as follows.
However, these models have a major limitation in their inability to fully capture the richness of compositionality.
Answer: However, these models have a major limitation in their inability to fully capture the richness of compositionality.