Title: MAJOR-MINOR LSTMS FOR WORD-LEVEL LAN-GUAGE MODEL

Abstract: As a widely-accepted evaluation criterion, complexity has attracted more and more attention in the design of language models. The parameter count is a proxy for complexity, which is often reported and compared in research papers. In general, more parameters means better model performance, but higher complexity. Therefore, reconciling the contradiction between the complexity and the model performance is necessary. In this paper, we propose a simple method to make use of model parameters more effectively, so that the LSTM-based language models can reach better results without the cost of increasing parameters. The method constructs another small-scale LSTM with a part of parameters originally belonging to the vanilla LSTM in each layer, whose output can assist the next layer in processing the output of the vanilla LSTM. We name these two LSTMs Major-Minor LSTMs. In experiments, we demonstrate the language model with Major-Minor LSTMs surpasses the existing state-of-the-art model on Penn Treebank and WikiText-2 with fewer parameters.

1. INTRODUCTION:

Language model (LM) is a foundational component of natural language processing (NLP) tasks, which estimates the probability distribution of a sequence of words (w 0 , ..., w n ) by modeling the probability of the next word (w i ) given preceding words (w 0 , ..., w i-1 ), i.e. P (w 0 , ..., w n ) = P (w 0 ) N i=1 P (w i |w 0 , ..., w i-1 )

Language model plays an important role of systems for machine translation Koehn (2009) , speech recognition (Yu & Deng, 2014) , learning token embeddings (Botha & Blunsom, 2014; Press & Wolf, 2016) , natural language generation (Radford et al., 2017; Merity et al., 2017 ) and text classification (Howard & Ruder, 2018) .

For word-level language model, the vanilla multi-layer Long Short Term Memory (LSTM) networks have been demonstrated to achieve state-of-the-art performance (Yang et al., 2017; Merity et al., 2017) . As one of the successful variants of Recurrent Neural Network (RNN), LSTM (Hochreiter & Schmidhuber, 1997) can not only process word sequences of any length, but also alleviate the issue of gradient vanishing effectively.

Since the probability distribution to be predicted is complex, current LSTM-based language models often utilize large-scale LSTMs with multiple layers to enhance generalization capability of the language model, which lead to a huge amount of parameters. In fact, we have found through a lot of experiments that when the scale of a certain LSTM layer is large enough, the improvement of overall model performance will reduce as the LSTM output dimension increases, even the change within a certain range will not cause obvious difference of the model result. But for these largescale LSTM layers, a small increase in their output size requires a large number of parameters to generate. This means that in order to achieve model effect as good as possible, there will inevitably be a part of parameters in the LSTMs with little contribution. Therefore, how to effectively improve the utilization of model parameters is a challenge for language model. For the sake of clarity, we call the LSTM in each layer of the vanilla LSTM-based language model Major LSTM. In this paper, we propose to reduce the dimension of Major LSTM appropriately, and set up a Minor LSTM with the saved parameters, under the premise of not impairing the performance of the Major LSTM significantly. The Minor LSTM generates a set of auxiliary features, which has capability to assist the next layer in processing the output of the current Major LSTM. Since the scale of the Minor LSTM is much smaller than the Major LSTM, the parameters building the Minor LSTM do not exceed the saved parameters. The architecture of the layer we proposed is called Major-Minor LSTMs (MMLSTMs), and for convenience the language model consisting of MMLSTMs is called Major-Minor Language Model (MMLM). We will detailed introduce the architecture of MMLSTMs and explain the rationality of the Minor LSTM later.

We integrate some existing methods (e.g. regularization, optimization, Mixture of Softmax) into our MMLM, and evaluate it on standard language modeling benchmarks. The experimental results show that the MMLM surpasses existing state-of-the-art language model on both Penn Treebank and WikiText-2 datasets with fewer parameters.

2. RELATED WORK:

Language modeling has gone through significant development from conventional N-gram language models (Kneser & Ney, 1995; Chen & Goodman, 1999) to neural language models (Bengio et al., 2003; Mikolov et al., 2010; Jozefowicz et al., 2016) in these decades. These classical N-gram models suffer from data sparsity, which makes it difficult to represent large contexts and thus, long-range dependencies. The LSTM-based language models effectively alleviate the problem of long-range dependency, so their performance greatly exceed other neural network language models. In addition, these LSTM-based language models also use a large amount of parameters for better generalization. Zaremba et al. (2014) applied dropout to the non-recurrent connections in LSTM language models with medium and large sizes, and achieved the best results with the large one at that time. Although, the large model was only slightly better than the medium model in the final performance, its parameter count is three times that of the latter. Gal & Ghahramani (2016) used the large LSTM language model with a dropout variation to surpass the result of Zaremba et al. (2014) . But there was still no effective way to reduce the model complexity. Inan et al. (2016) introduced a novel theoretical framework leadingto tying together the input embedding and the output projection matrices, greatly reducing the number of trainable parameters. Merity et al. (2017) proposed a weight-dropped LSTM, which used DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, they introduced a variant of averaged stochastic gradient method named NT-AvSGD, which was a successful improvement in the language model optimization method. Zolna et al. (2017) proposed to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their predictions. Yang et al. (2017) identified the Softmax bottleneck by formulating language modeling as a matrix factorization problem, and proposed a method called Mixture of Softmax to address it. However, for these language models using multi-layer LSTMs, in order to improve the model performance, they mainly leveraged the regularization method to reduce the over-fitting of the models under the same parameter counts. In contrast, our method is more straightforward. We assist the Major LSTM directly y constructing the Minor LSTM without increasing the cost of parameters. In addition, our model can be combined with existing regularization methods to produce better results.

3. MAJOR-MINOR LSTMS (MMLSTMS) FOR LANGUAGE MODEL:



3.1. THE VANILLA LSTM-BASED LANGUAGE MODEL:

A vanilla LSTM-based language model can be simply divided into three components: the input of word embeddings, the model of LSTMs, and the Softmax layer.

The mathematical formulation of the LSTM is as follow:

i j = σ (W i [h j-1 ; v j ] + b j ) (1) f j = σ (W f [h j-1 ; v j ] + b f ) (2) o j = σ (W o [h j-1 ; v j ] + b o ) (3) g j = tanh (W r [h j-1 ; v j ] + b r ) (4) c j = i j g j + f j c j-1

(5)

EQUATION

Where

[W i , W f , W o , W r ] , [b i , b f , b o , b r ]

are weight matrices and bias vectors. v j is the j th input vector, h j is the current exposed hidden state, c j is the memory cell state, and is element-wise multiplication. The embedding of word sequence is fed into LSTM layers, and the output of the last LSTM layer is used in predicting the probabilities distribution.

The Softmax layer calculates the probabilities of each word in the word sequence:

y j = e Ws•xj 1 T • e Ws•xj (7)

Where [W s ] is weight matrix, x j is the output of the last LSTM layer corresponding to j th word, 1 T = (1, ..., 1).

3.2. MAJOR-MINOR LSTMS FOR LANGUAGE MODEL:

Major LSTM

Minor LSTM As we discussed before, the size of LSTMs should be large enough so that the language model has sufficient learning capability. However, the promotion of the model performance will become unnoticeable when the parameter count reachs a certain level. For this reason, instead of keeping increasing the parameters of each layer of Major LSTM for a slight performance improvement, it is better to build a Minor LSTM with a low cost to generate an auxiliary feature vector, which assists the next layer in handling the output of the Major LSTM.

The architecture of a Major-Minor LSTMs layer is shown as Figure 1 . The inputs of the Major and Minor LSTMs are the output of the previous layer (v 1 , ..., v n ) and the original embedding of word sequence (e 1 , ..., e n ), respectively. And the size of the Major LSTM output vector (a 1 , ..., a n ) is much larger than Minor LSTM (c 1 , ..., c n ). For facilitating adjustment, we only set the size of entire Major-Minor LSTMs layer, and adjust the proportions of the Major and Minor LSTMs.

The reason why we do not use the same input of the Major LSTM as the input of the Minor LSTM is three-fold. First, according to our hypothesis, the Major LSTM has enough capability to process the output of the previous layer, so that there is no need to process it with the Minor LSTM. Second, if we consider the layers before the current layer as a whole, the output of the current layer can be written as the function of the original embedding of the word sequence. So the output of the previous layer is only an intermediate result for generating the current layer output. The original embeddings, by contrast, can generate the auxiliary features needed by the current layer more directly. Third, as in Merity et al. (2017) , the size of embedding is smaller than that size of hidden state, so that ease the issue of overfitting of the word embeddings. Therefore, compared with the output of the previous LSTM layer, using the original embedding as the input of the Minor LSTM can save more parameters.

3.3. THE RATIONALITY OF THE MINOR LSTM:

In MMLM, the output of Major-Minor LSTMs layers will flow into the next MMLSTM layer or the last Softmax layer. But from the Formula (1-7) we can infer that whether the next layer is LSTM or Softmax layer, it can be simplified to the combination of different components in a form of nonlinear affine transformation:

y t = f W 1 • x M ajor t + U 1 • s t-1 + b 1 + W 2 • x M inor t (8)

where x M ajor t

x M inor t denotes the t th output of the Major (Minor) LSTM in the previous layer, s t-1 denotes the (t -1) th hidden state in LSTM or 0 in the Softmax layer, [W 1 , W 2 , U 1 , b 1 ] are parameters of the component. f denotes the nonlinear function of the component, which is sigmoid or tanh function in LSTM and softmax function in Softmax layer. The Formula (8) indicates that we can understand W 2 • x M inor t as an adaptive bias vector, which adjusts the values of the component y t for different input word sequence. As a result, the auxiliary feature vector x M inor t has the capability to assist the next layer in processing the output of the current Major LSTM.

But for general LSTM-based language models (only have Major LSTMs), we can expand the vector x M ajor t to x M ajor t , x M ajor t by increasing the dimension of the Major LSTM. In this way, the 8) is replaced by W 2 • x M ajor t , and the form of the Formula (8) does not have obvious changes. But since the x M ajor t and the x M ajor t are generated by the same LSTM, x M ajor t cannot be freely and independently transformed into an auxiliary feature vector for x M ajor t like x M inor t generated by another LSTM. Concretely, if we simplify the Major LSTM to a vanilla RNN, the x M ajor t and the x M ajor t can be calculated as:

W 2 • x M inor t in Formula (

EQUATION

)

Where f is the nonlinear function, z t is the input corresponding to the t th word. Any change in each parameter will affect every output dimension of the LSTM, so it is difficult to split an independent portion of its output which can be freely transformed as the auxiliary feature vector of the other part.

For MMLSTMs, the auxiliary feature vector from the Minor LSTM can generate an adaptive bias vector to adjust the calculation of current Major LSTM output. By contrast, splitting a part of output in a single LSTM as the auxiliary feature vector will bring a heavier parameter burden to the LSTM. 

Hyper:



4. EXPERIMENT:



4.1. EXPERIMENT DETAILS:

We evaluate our MMLM on a preprocessed version of the Penn Treebank (PTB) (Marcus et al., 1993) and the WikiText-2 (WT2) dataset (Merity et al., 2016) , and use perplexity as the primary metric. In order to directly verify that our method can further heighten the effect of LSTM-based language model on the basis of some existing techniques, we apply MMLSTMs in AWD-LSTM-MoS architecture proposed by Yang et al. (2017) , named AWD-MMLSTMs-MoS. We finetune the hyperparameters in Yang et al. (2017) based on the validation performance while controlling the model size, and list the hyper-parameters we modified in Table 1 . For facilitating the adjustment of the regularization methods on Major and Minor LSTMs, we tie up their embedding and hidden state Variational dropout values. Regarding the dimensional proportion of the Major and Minor LSTMs, we divide each layer into 10 parts, and search various combinations heuristically and manually. It was found that the dimensions of the Major LSTMs on three layers accounted for [90%, 90%, 60%] respectively, and this model can achieve the best results on both datasets. We also apply dynamic evaluation Krause et al. (2017) to further improve our model performance, whose hyper-parameters are listed in Table 2 .

4.2. MAIN RESULTS:

We present the perplexity results from both our models (AWD-MMLSTMs-MoS) and other competitive models in Table 3 and 4 for PTB and WT2 respectively. We improve the state-of-the-art results on both datasets, 46.81 on PTB and 40.15 on WT2. In conclusion, obtaining our model requires three stages: training, fine-tuning, and dynamic estimation. Our model surpasses AWD-LSTM-MoS in all stages, which fully proves the effectiveness of our method.

4.3. STUDY OF DIMENSIONAL CHANGES ON DIFFERENT LSTM LAYERS:

The idea of Major-Minor LSTMs layer is based on our observation that reducing the dimensions of Major LSTMs in each layer properly does not significantly affect the performance of the Major LSTMs. So we build a small-scale Minor LSTM to extract a set of auxiliary features to assist the next layer in processing the output of the current Major LSTM. In this subsection, we study the influence of dimensional changes in different LSTM layers on the vanilla LSTM-based language In order to obtain the overall trend of perplexities with dimensional changes in each LSTM layer, we firstly use three line graphs2 to observe in a large dimension range from 200 to 1200 at intervals of 200. From these line graphs we can clearly see that with the increase of the dimension, the perplexity keeps decreasing, but the rate of decline is getting slower and slower. This means that the contribution of newly added parameters to the improvement of model performance decreases, so it is not worthwhile to strengthen the final effect by setting larger dimensions of LSTM layers. Moreover, we can also find that the dimensional change of the second LSTM layer has more influence than the first LSTM layer on the total model performance.

On the other hand, for intuitively verifying that the dimensional changes in a certain range for each LSTM layer will not cause obvious fluctuation in perplexity, we take the dimensions of each LSTM layer in Merity et al. (2017) and Yang et al. (2017) as standard, then draw the perplexity bands of models with dimensions from 0.6c to 1.2c in each layers at intervals of 0.1c(≈ 100). In each perplexity band, we mark the part whose perplexity differs from the standard perplexity within the range of [-0.25, 0.25] with a red box. We find the dimensions of 0.9c in the most layers are included in red boxes, and the red box of the third LSTM layer in AWD-LSTM-MoS includes the dimension of 0.6c. This demonstrates that we can greatly reduce the useless parameters and maintain the model performance by setting the dimension ratio of the Major LSTMs to 90% and 60% in the first two and the last MMLSTMs layers.

4.4. STUDY OF MINOR LSTM INPUT:

In Section 3.2 we explained the reason why we use the original embeddings as the input of the Minor LSTMs instead of the output of the previous layer. To compare the influence of the two types inputs on the final results, we conduct experiments on PTB and WT2, while limiting the amount of parameters. We construct two AWD-MMLSTMs-MoS with the two types input of Minor LSTMs, and denote them as MMLM-1 (embedding input) and MMLM-2. The dimension of the first two LSTM layers are set to 1150 for MMLM-2, so that the total parameters of the two models 

4.5. ABLATION STUDY:

Through the above experiments, we can reach the conclusion that using MMLSTMs layers can control the amount of parameters effectively and improve the model performance. To further study the contribution of the Minor LSTM in each MMLSTMs layer, we conduct an ablation study on both PTB and WT2. We remove the Minor LSTM from each MMLSTMs layer by layer, and denote the model with removed layer as MMLM-w/o-Minor-i (i = 1, 2, 3). The dimension of the layer removed the Minor LSTM will be adjusted to keep total parameters consistent, and the other hyper-parameters are the same to ensure a single variable contrast experiment. We also exclude the fine-tuning and dynamic evaluation steps, and the results are reported in Table 6 .

Comparing with our original three-layer MMLM, we can conclude that the Minor LSTM of the third layer has the greatest effect on the final performance of the model, followed by the first Minor LSTM and finally the second. In addition, this experiment demonstrates once again that MMLSTMs layer can not only reduce the amount of parameters in each layer, but also benefit the model performance.

5. CONCLUSIONS:

We observed that when the scale of LSTM is large enough, the benefits to the vanilla LSTM-based language model of increasing the dimension of LSTM are very limited, while leading to a lot of parameter waste. In this paper, we studied the phenomenon systematically, then proposed to reduce the dimension of the Major LSTM in the original LSTM layer appropriately, and construct a smallscale Minor LSTM to extract a set of auxiliary features, so that the next layer can process the output of the Major LSTM better. In experiments, we verified the MMLM can control the total parameter counts effectively, and further improve the performance of the language model based on the existing methods such as regularization, optimization and Mixture of Softmax. Moreover, our method improved the current state-of-the-art results on standard benchmarks with less amount of parameters.

:

The third LSTM layer of AWD-LSTM is connected to the Softmax layer, whose input size is fixed after tying the word vectors and word classifier. So we only experiment over the first two LSTM layers.

Since the dimension of the third LSTM of AWD-LSTM-MoS is much smaller than the first two layers, it is not suitable to draw their dimensional changes in the same coordinate system, so we only plot the dimensional changes of the first two LSTM layers in the line graph.

Figures and Tables:

Figure 1: The architecture of MMLSTMs layer. (v 1 , ..., v n ) and (e 1 , ..., e n ) denote the output of the previous layer and the word embedding of the original word sequence respectively. (a 1 , ..., a n ) and (c 1 , ..., c n ) represent the output of the Major LSTM and the Minor LSTM.

[W, b] and [W , b ] are parameters corresponding to x M ajor t and x M ajor t respectively, and they do not interfere with each other. But because of the appearance of x procedure, which is controlled by both [W, U, b] and [W , U , b ].

Figure 2: The line graphs of models with changed LSTM layers dimensions perplexities on PTB and WT2. The solid line and the dotted line denote the perplexity variation trend of the first LSTM and the second LSTM layer respectively. And the dimensional interval of these three line graphs is 200.

Hyper-parameters used for AWD-MMLSTMs-MoS. V-dropout abbreviates variational dropout(Gal & Ghahramani, 2016). SeeMerity et al. (2017)for more detailed descriptions.

Table content: <table><tr><td>-parameter</td><td>PTB</td><td>WT2</td></tr><tr><td>Learning rate</td><td>20</td><td>20</td></tr><tr><td colspan="3">MMLSTMs layer size [1080, 1080, 620] [1200, 1200, 650]</td></tr><tr><td colspan="3">Ratio of Major LSTM [90%, 90%, 60%] [90%, 90%, 60%]</td></tr><tr><td>Embedding V-dropout</td><td>0.5</td><td>0.55</td></tr><tr><td>Hidden state V-dropout</td><td>0.25</td><td>0.2</td></tr><tr><td>Non-monotone interval</td><td>10</td><td>10</td></tr><tr><td>Hyper-parameter</td><td>PTB</td><td>WT2</td></tr><tr><td>Batch size</td><td>150</td><td>130</td></tr><tr><td>Learning rate(η)</td><td colspan="2">0.0024 0.00198</td></tr><tr><td>ε</td><td colspan="2">0.0025 0.0023</td></tr><tr><td>λ</td><td>0.07</td><td>0.0245</td></tr><tr><td>bptt</td><td>7</td><td>7</td></tr></table>

Hyper-parameters used for dynamic evaluation of AWD-MMLSTMs-MoS. SeeKrause et al. (2017) for more detailed descriptions.

Table content: <table/>

The perplexities of MMLM-1 and MMLM-2 over PTB and WT2 datasets. are roughly equal. The other hyper-parameters of MMLM-1 and MMLM-2 are the same, and we still remove the fine-tuning and dynamic evaluation steps. The results are shown in Table5. The MMLM-1 surpasses MMLM-2 on both datasets steadily, which demonstrates using the embedding of input word sequence can better generate auxiliary features for Major-LSTM.

Table content: <table><tr><td>Model</td><td/><td/><td>PTB</td><td/><td/><td>WT2</td></tr><tr><td/><td colspan="2">#Param</td><td>Val</td><td colspan="3">Test #Param</td><td>Val</td><td>Test</td></tr><tr><td colspan="7">MMLM-1 21.3M 56.52 54.51 32.3M 64.30 61.77</td></tr><tr><td colspan="7">MMLM-2 21.7M 57.95 55.93 32.5M 65.43 62.76</td></tr><tr><td>Model</td><td/><td colspan="2">PTB</td><td/><td/><td>WT2</td></tr><tr><td/><td colspan="3">AD #Param</td><td>Val</td><td>Test</td><td>AD #Param</td><td>Val</td><td>Test</td></tr><tr><td colspan="4">MMLM-w/o-Minor-1 1030 21.5M</td><td colspan="3">57.3 55.10 1150 32.5M 65.34 62.73</td></tr><tr><td colspan="7">MMLM-w/o-Minor-2 1030 21.5M 57.14 54.93 1130 32.6M 65.07 62.43</td></tr><tr><td colspan="2">MMLM-w/o-Minor-3 520</td><td colspan="5">21.5M 57.84 55.74 550</td><td>32.6M 65.66 63.02</td></tr><tr><td>MMLM</td><td>-</td><td colspan="4">21.3M 56.52 54.51</td><td>-</td><td>32.3M</td><td>64.3 61.77</td></tr></table>

Ablation study on PTB and WT2 datasets without fine-tuning or dynamical evaluation. AD ( Adjusted Dimension) means adjusted dimension of MMLSTM layer whose Minor LSTM is removed, and the columns with the title Val and Test denote the perplexities on the validation sets and test sets respectively.

Table content: <table/>

Peer review (iclr2019):
 The paper suggests more efficient spending of the parameters in LSTM language models: instead of increasing the hidden layer size of the LSTM, the authors suggest building a small assistant LSTM and its output is concatenated with the output of the main LSTM. The approach reminds me the SCRN Model of Mikolov et al. (2015). I don*t claim it is the same, as you have an independent LSTM, but there are some similarities: in both cases an assisting small layer is added, and in both cases one needs to choose an appropriate size of this assisting layer. It would be interesting to see if your Minor LSTM plays the same role as the context state in SCRN, e.g. does the hidden state in Minor LSTM change slower than that of the Major LSTM? If yes, then we could say that the Minor LSTM focuses more on what is being talked about (context). Experimental evaluation of the idea seems adequate. However, my feeling is that the recent optimization and regularization techniques (such as those used in AWD-LSTM) may hide the effect of your idea. If possible, could you please evaluate your approach under less recent regularization, e.g. variational dropout with tied weights as in Inan et al. (2016). Minor comment on "more parameters means better model performance" in the abstract: If by model performance we understand performance on held-out data, then I would say that "more parameters means more flexible model". Note after reading other reviews: I agree with Reviewer1 and Reviewer2 that the use of skip-connections (Melis et al., 2017; Zilly et al., 2016) is one of the main reasons why the model achieves its performance (Table 5). Without skip-connections the proposed Major-Minor LSTM is practically on par with the AWD-LSTM-MoS of Yang et al. (2017) on PTB and only 0.57 perplexity units better on WT-2. References - Inan, H., Khosravi, K. and Socher, R., 2016. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462. - Melis, G., Dyer, C. and Blunsom, P., 2017. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589. - Mikolov, T., Joulin, A., Chopra, S., Mathieu, M. and Ranzato, M.A., 2014. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753. - Yang, Z., Dai, Z., Salakhutdinov, R. and Cohen, W.W., 2017. Breaking the softmax bottleneck: A high-rank RNN language model. arXiv preprint arXiv:1711.03953. - Zilly, J.G., Srivastava, R.K., Koutník, J. and Schmidhuber, J., 2016. Recurrent highway networks. arXiv preprint arXiv:1607.03474.