Neural Language Modeling by Jointly Learning Syntax and Lexicon

Abstract: We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.

Linguistic theories generally regard natural language as consisting of two part: a lexicon, the complete set of all possible words in a language; and a syntax, the set of rules, principles, and processes that govern the structure of sentences BIBREF0 . To generate a proper sentence, tokens are put together with a specific syntactic structure. Understanding a sentence also requires lexical information to provide meanings, and syntactical knowledge to correctly combine meanings. Current neural language models can provide meaningful word represent BIBREF1 , BIBREF2 , BIBREF3 . However, standard recurrent neural networks only implicitly model syntax, thus fail to efficiently use structure information BIBREF4 .
Developing a deep neural network that can leverage syntactic knowledge to form a better semantic representation has received a great deal of attention in recent years BIBREF5 , BIBREF4 , BIBREF6 . Integrating syntactic structure into a language model is important for different reasons: 1) to obtain a hierarchical representation with increasing levels of abstraction, which is a key feature of deep neural networks and of the human brain BIBREF7 , BIBREF8 , BIBREF9 ; 2) to capture complex linguistic phenomena, like long-term dependency problem BIBREF4 and the compositional effects BIBREF5 ; 3) to provide shortcut for gradient back-propagation BIBREF6 .
A syntactic parser is the most common source for structure information. Supervised parsers can achieve very high performance on well constructed sentences. Hence, parsers can provide accurate information about how to compose word semantics into sentence semantics BIBREF5 , or how to generate the next word given previous words BIBREF10 . However, only major languages have treebank data for training parsers, and it request expensive human expert annotation. People also tend to break language rules in many circumstances (such as writing a tweet). These defects limit the generalization capability of supervised parsers.
Unsupervised syntactic structure induction has been among the longstanding challenges of computational linguistic BIBREF11 , BIBREF12 , BIBREF13 . Researchers are interested in this problem for a variety of reasons: to be able to parse languages for which no annotated treebanks exist BIBREF14 ; to create a dependency structure to better suit a particular NLP application BIBREF10 ; to empirically argue for or against the poverty of the stimulus BIBREF15 , BIBREF16 ; and to examine cognitive issues in language learning BIBREF17 .
In this paper, we propose a novel neural language model: Parsing-Reading-Predict Networks (PRPN), which can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to form a better language model. With our model, we assume that language can be naturally represented as a tree-structured graph. The model is composed of three parts:
We evaluate our model on three tasks: word-level language modeling, character-level language modeling, and unsupervised constituency parsing. The proposed model achieves (or is close to) the state-of-the-art on both word-level and character-level language modeling. The model's unsupervised parsing outperforms some strong baseline models, demonstrating that the structure found by our model is similar to the intrinsic structure provided by human experts.
The idea of introducing some structures, especially trees, into language understanding to help a downstream task has been explored in various ways. For example, BIBREF5 , BIBREF4 learn a bottom-up encoder, taking as an input a parse tree supplied from an external parser. There are models that are able to infer a tree during test time, while still need supervised signal on tree structure during training. For example, BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 , etc. Moreover, BIBREF22 did an in-depth analysis of recursive models that are able to learn tree structure without being exposed to any grammar trees. Our model is also able to infer tree structure in an unsupervised setting, but different from theirs, it is a recurrent network that implicitly models tree structure through attention.
Apart from the approach of using recursive networks to capture structures, there is another line of research which try to learn recurrent features at multiple scales, which can be dated back to 1990s (e.g. BIBREF23 , BIBREF24 , BIBREF25 ). The NARX RNN BIBREF25 is another example which used a feed forward net taking different inputs with predefined time delays to model long-term dependencies. More recently, BIBREF26 also used multiple layers of recurrent networks with different pre-defined updating frequencies. Instead, our model tries to learn the structure from data, rather than predefining it. In that respect, BIBREF6 relates to our model since it proposes a hierarchical multi-scale structure with binary gates controlling intra-layer connections, and the gating mechanism is learned from data too. The difference is that their gating mechanism controls the updates of higher layers directly, while ours control it softly through an attention mechanism.
In terms of language modeling, syntactic language modeling can be dated back to BIBREF27 . BIBREF28 , BIBREF29 have also proposed language models with a top-down parsing mechanism. Recently BIBREF30 , BIBREF31 have introduced neural networks into this space. It learns both a discriminative and a generative model with top-down parsing, trained with a supervision signal from parsed sentences in the corpus. There are also dependency-based approaches using neural networks, including BIBREF32 , BIBREF33 , BIBREF34 .
Parsers are also related to our work since they are all inferring grammatical tree structure given a sentence. For example, SPINN BIBREF35 is a shift-reduce parser that uses an LSTM as its composition function. The transition classifier in SPINN is supervisedly trained on the Stanford PCFG Parser BIBREF36 output. Unsupervised parsers are more aligned with what our model is doing. BIBREF12 presented a generative model for the unsupervised learning of dependency structures. BIBREF11 is a generative distributional model for the unsupervised induction of natural language syntax which explicitly models constituent yields and contexts. We compare our parsing quality with the aforementioned two papers in Section SECREF43 .
Suppose we have a sequence of tokens INLINEFORM0 governed by the tree structure showed in Figure FIGREF4 . The leafs INLINEFORM1 are observed tokens. Node INLINEFORM2 represents the meaning of the constituent formed by its leaves INLINEFORM3 , where INLINEFORM4 and INLINEFORM5 stands for the leftmost child and right most child. Root INLINEFORM6 represents the meaning of the whole sequence. Arrows represent the dependency relations between nodes. The underlying assumption is that each node depends only on its parent and its left siblings.
Directly modeling the tree structure is a challenging task, usually requiring supervision to learn BIBREF4 . In addition, relying on tree structures can result in a model that is not sufficiently robust to face ungrammatical sentences BIBREF37 . In contrast, recurrent models provide a convenient way to model sequential data, with the current hidden state only depends on the last hidden state. This makes models more robust when facing nonconforming sequential data, but it suffers from neglecting the real dependency relation that dominates the structure of natural language sentences.
In this paper, we use skip-connection to integrate structured dependency relations with recurrent neural network. In other words, the current hidden state does not only depend on the last hidden state, but also on previous hidden states that have a direct syntactic relation to the current one.
Figure FIGREF5 shows the structure of our model. The non-leaf node INLINEFORM0 is represented by a set of hidden states INLINEFORM1 , where INLINEFORM2 is the left most descendant leaf and INLINEFORM3 is the right most one. Arrows shows skip connections built by our model according to the latent structure. Skip connections are controlled by gates INLINEFORM4 . In order to define INLINEFORM5 , we introduce a latent variable INLINEFORM6 to represent local structural context of INLINEFORM7 :
and gates are defined as: DISPLAYFORM0 
Given this architecture, the siblings dependency relation is modeled by at least one skip-connect. The skip connection will directly feed information forward, and pass gradient backward. The parent-to-child relation will be implicitly modeled by skip-connect relation between nodes.
The model recurrently updates the hidden states according to: DISPLAYFORM0 
and the probability distribution for next word is approximated by: DISPLAYFORM0 
 where INLINEFORM0 are gates that control skip-connections. Both INLINEFORM1 and INLINEFORM2 have a structured attention mechanism that takes INLINEFORM3 as input and forces the model to focus on the most related information. Since INLINEFORM4 is an unobserved latent variable, We explain an approximation for INLINEFORM5 in the next section. The structured attention mechanism is explained in section SECREF21 .
In this section we give a probabilistic view on how to model the local structure of language. A detailed elaboration for this section is given in Appendix . At time step INLINEFORM0 , INLINEFORM1 represents the probability of choosing one out of INLINEFORM2 possible local structures. We propose to model the distribution by the Stick-Breaking Process: DISPLAYFORM0 
The formula can be understood by noting that after the time step INLINEFORM0 have their probabilities assigned, INLINEFORM1 is remaining probability, INLINEFORM2 is the portion of remaining probability that we assign to time step INLINEFORM3 . Variable INLINEFORM4 is parametrized in the next section.
As shown in Appendix , the expectation of gate value INLINEFORM0 is the Cumulative Distribution Function (CDF) of INLINEFORM1 . Thus, we can replace the discrete gate value by its expectation: DISPLAYFORM0 
With these relaxations, Eq. EQREF9 and EQREF10 can be approximated by using a soft gating vector to update the hidden state and predict the next token.
In Eq. EQREF12 , INLINEFORM0 is the portion of the remaining probability that we assign to position INLINEFORM1 . Because the stick-breaking process should assign high probability to INLINEFORM2 , which is the closest constituent-beginning word. The model should assign large INLINEFORM3 to words beginning new constituents. While INLINEFORM4 itself is a constituent-beginning word, the model should assign large INLINEFORM5 to words beginning larger constituents. In other words, the model will consider longer dependency relations for the first word in constituent. Given the sentence in Figure FIGREF4 , at time step INLINEFORM6 , both INLINEFORM7 and INLINEFORM8 should be close to 1, and all other INLINEFORM9 should be close to 0.
In order to parametrize INLINEFORM0 , our basic hypothesis is that words in the same constituent should have a closer syntactic relation within themselves, and that this syntactical proximity can be represented by a scalar value. From the tree structure point of view, the shortest path between leafs in same subtree is shorter than the one between leafs in different subtree.
To model syntactical proximity, we introduce a new feature Syntactic Distance. For a sentence with length INLINEFORM0 , we define a set of INLINEFORM1 real valued scalar variables INLINEFORM2 , with INLINEFORM3 representing a measure of the syntactic relation between the pair of adjacent words INLINEFORM4 . INLINEFORM5 could be the last word in previous sentence or a padding token. For time step INLINEFORM6 , we want to find the closest words INLINEFORM7 , that have larger syntactic distance than INLINEFORM8 . Thus INLINEFORM9 can be defined as: DISPLAYFORM0 
where INLINEFORM0 . INLINEFORM1 is the temperature parameter that controls the sensitivity of INLINEFORM2 to the differences between distances.
The Syntactic Distance has some nice properties that both allow us to infer a tree structure from it and be robust to intermediate non-valid tree structures that the model may encounter during learning. In Appendix and we list these properties and further explain the meanings of their values.
 BIBREF38 shows that it's possible to identify the beginning and ending words of a constituent using local information. In our model, the syntactic distance between a given token (which is usually represented as a vector word embedding INLINEFORM0 ) and its previous token INLINEFORM1 , is provided by a convolutional kernel over a set of consecutive previous tokens INLINEFORM2 . This convolution is depicted as the gray triangles shown in Figure FIGREF20 . Each triangle here represent 2 layers of convolution. Formally, the syntactic distance INLINEFORM3 between token INLINEFORM4 and INLINEFORM5 is computed by DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 , INLINEFORM1 are the kernel parameters. INLINEFORM2 and INLINEFORM3 can be seen as another convolutional kernel with window size 1, convolved over INLINEFORM4 's. Here the kernel window size INLINEFORM5 determines how far back into the history node INLINEFORM6 can reach while computing its syntactic distance INLINEFORM7 . Thus we call it the look-back range.
Convolving INLINEFORM0 and INLINEFORM1 on the whole sequence with length INLINEFORM2 yields a set of distances. For the tokens in the beginning of the sequence, we simply pad INLINEFORM3 zero vectors to the front of the sequence in order to get INLINEFORM4 outputs.
The Reading Network generate new states INLINEFORM0 considering on input INLINEFORM1 , previous memory states INLINEFORM2 , and gates INLINEFORM3 , as shown in Eq. EQREF9 .
Similar to Long Short-Term Memory-Network (LSTMN) BIBREF39 , the Reading Network maintains the memory states by maintaining two sets of vectors: a hidden tape INLINEFORM0 , and a memory tape INLINEFORM1 , where INLINEFORM2 is the upper bound for the memory span. Hidden states INLINEFORM3 is now represented by a tuple of two vectors INLINEFORM4 . The Reading Network captures the dependency relation by a modified attention mechanism: structured attention. At each step of recurrence, the model summarizes the previous recurrent states via the structured attention mechanism, then performs a normal LSTM update, with hidden and cell states output by the attention mechanism.
At each time step INLINEFORM0 , the read operation attentively links the current token to previous memories with a structured attention layer: DISPLAYFORM0 
 where, INLINEFORM0 is the dimension of the hidden state. Modulated by the gates in Eq. EQREF13 , the structured intra-attention weight is defined as: DISPLAYFORM0 
 This yields a probability distribution over the hidden state vectors of previous tokens. We can then compute an adaptive summary vector for the previous hidden tape and memory denoting by INLINEFORM0 and INLINEFORM1 : DISPLAYFORM0 
Structured attention provides a way to model the dependency relations shown in Figure FIGREF4 .
The Reading Network takes INLINEFORM0 , INLINEFORM1 and INLINEFORM2 as input, computes the values of INLINEFORM3 and INLINEFORM4 by the LSTM recurrent update BIBREF40 . Then the write operation concatenates INLINEFORM5 and INLINEFORM6 to the end of hidden and memory tape.
Predict Network models the probability distribution of next word INLINEFORM0 , considering on hidden states INLINEFORM1 , and gates INLINEFORM2 . Note that, at time step INLINEFORM3 , the model cannot observe INLINEFORM4 , a temporary estimation of INLINEFORM5 is computed considering on INLINEFORM6 : DISPLAYFORM0 
From there we compute its corresponding INLINEFORM0 and INLINEFORM1 for Eq. EQREF10 . We parametrize INLINEFORM2 function as: DISPLAYFORM0 
 where INLINEFORM0 is an adaptive summary of INLINEFORM1 , output by structured attention controlled by INLINEFORM2 . INLINEFORM3 could be a simple feed-forward MLP, or more complex architecture, like ResNet, to add more depth to the model.
We evaluate the proposed model on three tasks, character-level language modeling, word-level language modeling, and unsupervised constituency parsing.
From a character-level view, natural language is a discrete sequence of data, where discrete symbols form a distinct and shallow tree structure: the sentence is the root, words are children of the root, and characters are leafs. However, compared to word-level language modeling, character-level language modeling requires the model to handle longer-term dependencies. We evaluate a character-level variant of our proposed language model over a preprocessed version of the Penn Treebank (PTB) and Text8 datasets.
When training, we use truncated back-propagation, and feed the final memory position from the previous batch as the initial memory of next one. At the beginning of training and test time, the model initial hidden states are filled with zero. Optimization is performed with Adam using learning rate INLINEFORM0 , weight decay INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 . We carry out gradient clipping with maximum norm 1.0. The learning rate is multiplied by 0.1 whenever validation performance does not improve during 2 checkpoints. These checkpoints are performed at the end of each epoch. We also apply layer normalization BIBREF41 to the Reading Network and batch normalization to the Predict Network and parsing network. For all of the character-level language modeling experiments, we apply the same procedure, varying only the number of hidden units, mini-batch size and dropout rate.
we process the Penn Treebank dataset BIBREF42 by following the procedure introduced in BIBREF43 . For character-level PTB, Reading Network has two recurrent layers, Predict Network has one residual block. Hidden state size is 1024 units. The input and output embedding size are 128, and not shared. Look-back range INLINEFORM0 , temperature parameter INLINEFORM1 , upper band of memory span INLINEFORM2 . We use a batch size of 64, truncated back-propagation with 100 timesteps. The values used of dropout on input/output embeddings, between recurrent layers, and on recurrent states were (0, 0.25, 0.1) respectively.
In Figure FIGREF32 , we visualize the syntactic distance estimated by the Parsing Network, while reading three different sequences from the PTB test set. We observe that the syntactic distance tends to be higher between the last character of a word and a space, which is a reasonable breakpoint to separate between words. In other words, if the model sees a space, it will attend on all previous step. If the model sees a letter, it will attend no further then the last space step. The model autonomously discovered to avoid inter-word attention connection, and use the hidden states of space (separator) tokens to summarize previous information. This is strong proof that the model can understand the latent structure of data. As a result our model achieve state-of-the-art performance and significantly outperform baseline models. It is worth noting that HM-LSTM BIBREF6 also unsupervisedly induce similar structure from data. But discrete operations in HM-LSTM make their training procedure more complicated then ours.
Comparing to character-level language modeling, word-level language modeling needs to deal with complex syntactic structure and various linguistic phenomena. But it has less long-term dependencies. We evaluate the word-level variant of our language model on a preprocessed version of the Penn Treebank (PTB) BIBREF42 and Text8 BIBREF49 dataset.
We apply the same procedure and hyper-parameters as in character-level language model. Except optimization is performed with Adam with INLINEFORM0 . This turns off the exponential moving average for estimates of the means of the gradients BIBREF50 . We also adapt the number of hidden units, mini-batch size and the dropout rate according to the different tasks.
we process the Penn Treebank dataset BIBREF43 by following the procedure introduced in BIBREF51 . For word-level PTB, the Reading Network has two recurrent layers and the Predict Network do not have residual block. The hidden state size is 1200 units and the input and output embedding sizes are 800, and shared BIBREF52 , BIBREF53 . Look-back range INLINEFORM0 , temperature parameter INLINEFORM1 and the upper band of memory span INLINEFORM2 . We use a batch size of 64, truncated back-propagation with 35 time-steps. The values used of dropout on input/output embeddings, between recurrent layers, and on recurrent states were (0.7, 0.5, 0.5) respectively.
dataset contains 17M training tokens and has a vocabulary size of 44k words. The dataset is partitioned into a training set (first 99M characters) and a development set (last 1M characters) that is used to report performance. As this dataset contains various articles from Wikipedia, the longer term information (such as current topic) plays a bigger role than in the PTB experiments BIBREF61 . We apply the same procedure and hyper-parameters as in character-level PTB, except we use a batch size of 128. The values used of dropout on input/output embeddings, between Recurrent Layers and on recurrent states were (0.4, 0.2, 0.2) respectively.
In Table TABREF39 , our results are comparable to the state-of-the-art methods. Since we do not have the same computational resource used in BIBREF50 to tune hyper-parameters at large scale, we expect that our model could achieve better performance after an aggressive hyperparameter tuning process. As shown in Table TABREF42 , our method outperform baseline methods. It is worth noticing that the continuous cache pointer can also be applied to output of our Predict Network without modification. Visualizations of tree structure generated from learned PTB language model are included in Appendix . In Table TABREF40 , we show the value of test perplexity for different variants of PRPN, each variant remove part of the model. By removing Parsing Network, we observe a significant drop of performance. This stands as empirical evidence regarding the benefit of having structure information to control attention.
The unsupervised constituency parsing task compares hte tree structure inferred by the model with those annotated by human experts. The experiment is performed on WSJ10 dataset. WSJ10 is the 7422 sentences in the Penn Treebank Wall Street Journal section which contained 10 words or less after the removal of punctuation and null elements. Evaluation was done by seeing whether proposed constituent spans are also in the Treebank parse, measuring unlabeled F1 ( INLINEFORM0 ) of unlabeled constituent precision and recall. Constituents which could not be gotten wrong (those of span one and those spanning entire sentences) were discarded. Given the mechanism discussed in Section SECREF14 , our model generates a binary tree. Although standard constituency parsing tree is not limited to binary tree. Previous unsupervised constituency parsing model also generate binary trees BIBREF11 , BIBREF13 . Our model is compared with the several baseline methods, that are explained in Appendix .
Different from the previous experiment setting, the model treat each sentence independently during train and test time. When training, we feed one batch of sentences at each iteration. In a batch, shorter sentences are padded with 0. At the beginning of the iteration, the model's initial hidden states are filled with zero. When testing, we feed on sentence one by one to the model, then use the gate value output by the model to recursively combine tokens into constituents, as described in Appendix .
Table TABREF44 summarizes the results. Our model significantly outperform the RANDOM baseline indicate a high consistency with human annotation. Our model also shows a comparable performance with CCM model. In fact our parsing network and CCM both focus on the relation between successive tokens. As described in Section SECREF14 , our model computes syntactic distance between all successive pair of tokens, then our parsing algorithm recursively assemble tokens into constituents according to the learned distance. CCM also recursively model the probability whether a contiguous subsequences of a sentence is a constituent. Thus, one can understand how our model is outperformed by DMV+CCM and UML-DOP models. The DMV+CCM model has extra information from a dependency parser. The UML-DOP approach captures both contiguous and non-contiguous lexical dependencies BIBREF13 .
In this paper, we propose a novel neural language model that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. We introduce a new neural parsing network: Parsing-Reading-Predict Network, that can make differentiable parsing decisions. We use a new structured attention mechanism to control skip connections in a recurrent neural network. Hence induced syntactic structure information can be used to improve the model's performance. Via this mechanism, the gradient can be directly back-propagated from the language model loss function into the neural Parsing Network. The proposed model achieve (or is close to) the state-of-the-art on both word/character-level language modeling tasks. Experiment also shows that the inferred syntactic structure highly correlated to human expert annotation.
The authors would like to thank Timothy J. O'Donnell and Chris Dyer for the helpful discussions.

Figure and table captions:
Figure 1: Hard arrow represents syntactic tree structure and parent-to-child dependency relation, dash arrow represents dependency relation between siblings
Figure 2: Proposed model architecture, hard line indicate valid connection in Reading Network, dash line indicate valid connection in Predict Network.
Figure 3: Convolutional network for computing syntactic distance. Gray triangles represent 2 layers of convolution, d0 to d7 are the syntactic distance output by each of the kernel position. The blue bars indicate the amplitude of di’s, and yi’s are the inferred constituents.
Figure 4: Syntactic distance estimated by Parsing Network. The model is trained on PTB dataset at the character level. Each blue bar is positioned between two characters, and represents the syntactic distance between them. From these distances we can infer a tree structure according to Section 4.2.
Table 1: BPC on the Penn Treebank test set
Table 2: PPL on the Penn Treebank test set
Table 3: Ablation test on the Penn Treebank. “- Parsing Net” means that we remove Parsing Network and replace Structured Attention with normal attention mechanism; “- Reading Net Attention” means that we remove Structured Attention from Reading Network, that is equivalent to replace Reading Network with a normal 2-layer LSTM; “- Predict Net Attention” means that we remove Structured Attention from Predict Network, that is equivalent to have a standard projection layer; “Our 2-layer LSTM” is equivalent to remove Parsing Network and remove Structured Attention from both Reading and Predict Network.
Table 4: PPL on the Text8 valid set
Table 5: Parsing Performance on the WSJ10 dataset
Figure 5: Syntactic structures of two different sentences inferred from {di} given by Parsing Network.

Questions about this paper:
How do they show their model discovers underlying syntactic structure?
How do they measure performance of language model tasks?

Answers:
By visualizing syntactic distance estimated by the parsing network
BPC, Perplexity