Title: Multiplicative LSTM for sequence modelling
Authors: Ben Krause
Iain Murray
Steve Renals
Liang Lu
Venue: ICLR 2017
Keywords: Deep learning
Natural language processing
Unsupervised Learning

Abstract: We introduce multiplicative LSTM (mLSTM), a novel recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level modelling tasks, and that this improvement increases with the complexity of the task. This model achieves a test error of 1.19 bits/character on the last 4 million characters of the Hutter prize dataset when combined with dynamic evaluation.

TL;DR: Combines LSTM and multiplicative RNN architectures; achieves 1.19 bits/character on Hutter prize dataset with dynamic evaluation.

Review discussion:
What is the reference to tensor RNN in section 1.2?
I think you can compare mRNN and mLSTM also to Multi-Function Recurrent Unit (MuFuRU) (https://arxiv.org/abs/1606.03002)?
Why do you chose the variant in equation (16) to use tanh after the output gating?
How does it differ in performance to the usual variant?
In Table 1, 2 & 3, what are the numbers of parameters and the hidden dimensions for each of the presented models?
In Table 3, is it fair to compare with models which use the test data indirectly? How is it justified to use the error signal from test data? In a real-world evaluation, when you don't know the real target, you cannot do that.
Did you compare with multiplicative integration LSTMs? And other LSTM variants such as Associative LSTMs?
Your stacked LSTM models were all only 2 layers. Have you tried deeper LSTM models?
You say that mLSTM's advantage over stacked LSTM was greater after a suprising input than it is in general. By how much?
1. What is the reference to tensor RNN in section 1.2?
The tensor RNN is an idea described in the original multiplicative RNN paper. The tensor RNN is not a practical model (and has never been used in practice as far as we know) because it has too many parameters, but it illustrates what the multiplicative RNN is attempting to express with fewer parameters.
2. I think you can compare mRNN and mLSTM also to Multi-Function Recurrent Unit (MuFuRU) (https://arxiv.org/abs/1606.03002)?
Thanks for pointing us towards this paper, it is an interesting approach that we had not spotted. It seems like their main motivation is to learn task specific transition functions, whereas our main motivation is to learn transition functions specific to each possible input unit.
Also, it doesn’t seem like they have any results we could easily include in our paper.
3. Why do you chose the variant in equation (16) to use tanh after the output gating?
How does it differ in performance to the usual variant?
We found it slightly better in our own preliminary experiments on the text 8 dataset, but our LSTM baseline on text8 (using equation (16)) performs about equally with a baseline in the literature (which used the usual variant) of the same size on text8 in table 2. Having the LSTM output gate on the outside of the tanh, as in the usual variant, gives it the ability to shut off the LSTM unit completely no matter how large the input to the tanh is. We think this is useful for the synthetic tasks LSTM was originally designed for, but not necessarily for real world tasks. Having the output gate inside the tanh still has the ability to greatly reduce the output in the linear regions of the tanh. It could also help prevent saturation in the tanh.
Overall we don’t think the two approaches are very different.
4. In Table 1, 2 & 3, what are the numbers of parameters and the hidden dimensions for each of the presented models?
We report the sizes of the models we trained, but we do think it would be useful to have the sizes of other models as well, especially on the Hutter prize dataset where over fitting is somewhat less of a concern. Some papers report hidden dimensionality while others report number of parameters, so we will go back through all the papers for the other results we report and see if we can report size information in a coherent way, either as a table column or a footnote.
5. In Table 3, is it fair to compare with models which use the test data indirectly? How is it justified to use the error signal from test data? In a real-world evaluation, when you don't know the real target, you cannot do that.
It is true that this would not be possible for supervised tasks, however the main focus on our paper was application to generative sequence modelling, where the outputs are conditioned on after they are observed.
So all generative RNNs inherently have access to this information anyway, it is just that they have trouble using it. Using the error signal after the output is conditioned on still defines a proper probability distribution over sequences (independently of what the test data actually is), and can be used to improve compression and conditional sequence generation. It is also possible to sample directly from the probability distribution defined by a dynamically adapted RNN by adapting to recent sequences generated by the network.
6. Did you compare with multiplicative integration LSTMs? And other LSTM variants such as Associative LSTMs?
We have a paragraph in our related approaches section comparing mLSTM with multiplicative integration LSTM. We also include results from multiplicative integration LSTM in tables 2 and 3. Associative LSTM is difficult to compare our results against; they did have an experiment on the Hutter prize dataset, but it used a very small network. The main motivation of associative LSTM was improving memory capacity, which is fairly different from our motivation to improve input-dependent transition flexibility.
7. Your stacked LSTM models were all only 2 layers. Have you tried deeper LSTM models?
No we haven’t tried this. It could be interesting, but we also suspect there would be a quick diminishing return for adding more stacked layers for the tasks tested in this paper.
Also, we do compare with one 7-layer stacked LSTM from the literature in table 3, although that result is somewhat outdated.
8. You say that mLSTM's advantage over stacked LSTM was greater after a suprising input than it is in general. By how much?
When we said that mLSTM's advantage over stacked LSTM was greater after a surprising input than it is in general (on page 7), that was the conclusion we were drawing from the results presented in the preceding lines. In these results we present how large mLSTM's advantage was 1-4 timesteps after a surprising input, and also note how large mLSTM's advantage was in general.
From the paper:
One to four time-steps after a surprising input occurred, mLSTM and stacked LSTM took average losses of (2.26, 2.04, 1.61, 1.51) and (2.48, 2.25, 1.79, 1.67) bits per character respectively. mLSTM's overall advantage over stacked LSTM was 1.42 bits/char to 1.53 bits/char
so for instance, one step after a surprising input mLSTM's advantage over stacked LSTM was 0.22 bits/char, whereas in general it was 0.11 bits/char
* What is the memory consumption and efficiency of this technique compared to regular RNNs or LSTMs? Can you add a few sentence for the clarifications into the paper?
* Do you think if it is possible, apply this model to the word-level language modeling? What kind of challenges need to be solved for that?
* On char-LM tasks to be able to obtain more comparable results to the SOTAs, have you tried recurrent/variation dropout and layer/batch norm?
* What other tasks do you think this model can be useful for?
* What kind of initialization have you used for the transition matrices?
1.What is the memory consumption and efficiency of this technique compared to regular RNNs or LSTMs? Can you add a few sentence for the clarifications into the paper?
mLSTM uses one additional input-to-hidden and hidden-to-hidden matrix in addition to the normal input-to-hidden and hidden-to-hidden matrix in LSTM.
However, the additional weight matrices are only one quarter of the size of the original LSTM weight matrices.
So excluding the output layer (which is the same in both cases), mLSTM has 5/4 times as many weights as regular LSTM, therefore requiring 5/4 times the computation and 5/4 the memory for weight storage compared with regular LSTM. We do mention that mLSTM has 5/4 the number of recurrent weights as LSTM for the same number of hidden units, but we will try to make this more clear and explicitly mention memory and efficiency in the paper.
2.Do you think if it is possible, apply this model to the word-level language modeling? What kind of challenges need to be solved for that?
It is possible to apply the model directly to word-level language modelling in its present form, and this is something we are currently researching.
3.On char-LM tasks to be able to obtain more comparable results to the SOTAs, have you tried recurrent/variation dropout and layer/batch norm?
We have not tried this yet, although we do suspect that using these techniques would make mLSTM more competitive with other models that use them.
4.What other tasks do you think this model can be useful for?
This model works best when the input is a sequence of one-hot vectors. The model could in theory be used for sequences of non-one-hot vectors but there are some optimisation difficulties that we have ran into in practice, that are likely due to instability that results from not having a fixed number of possible transition matrices (since the input-dependant diagonal matrix in equation (8) would be a linear combination of vectors rather than a single vector).
5.What kind of initialization have you used for the transition matrices?
We used independent Gaussian noise with a very small standard deviation for the transition matrices, and set the LSTM forget gate biases to 3 for all experiments. Large forget gate biases were helpful for performance, but this required using transition matrices that are smaller to keep the network from blowing up. More specifically, for all experiments for mLSTM, we set the standard deviation of initialization of the hidden-intermediate and intermediate-hidden weight matrices to 0.02, and the input-dependent diagonal intermediate matrix to 0.1 (I'm referring to the 3 weights matrices in equation (8)). Since the input dependent matrices are diagonal, they have a smaller spectral radius and greatly reduce the spectral radius of the overall transition matrix when the three transition matrices are multiplied together (which is then made up for by having positively biased forget gates).
What is the motivation behind Eq. (16)?
What if the multiplicative LSTM is trained with additional regularization or training methods (zoneout, dropout, batch normalization, layer normalization)?
What are the learning rates used in your experiments (for each dataset) and hyper parameters of RMSProp?
1.What is the motivation behind Eq. (16)?
Having the output gate inside the tanh could potentially reduce saturation in the tanh. We found it performed slightly better than the usual variant in our own preliminary experiments on the text 8 dataset, but our LSTM baseline (using E.q. (16)) on text8 performs about equally with a baseline in the literature (which used the usual variant) of the same size on text8 in table 2. Having the LSTM output gate on the outside of the tanh, as in the usual variant, gives it the ability to shut off the LSTM unit completely no matter how large the input to the tanh is. We think this is useful for the synthetic tasks LSTM was originally designed for, but not necessarily for real world tasks. Having the output gate inside the tanh still has the ability to greatly reduce the output in the linear regions of the tanh.
Overall, we don’t think the two approaches are very different.
2.What if the multiplicative LSTM is trained with additional regularization or training methods (zoneout, dropout, batch normalization, layer normalization)?
This would definitely possible. It would require design decisions such as whether to apply regularisation just at the LSTM units, or at the intermediate transition states in Eq. (9) as well.
3.What are the learning rates used in your experiments (for each dataset) and hyper parameters of RMSProp?
for the text8, Hutter prize and multilingual experiments:
for mLSTM: we used a decay rate of 0.9 for the running average of the squared gradient, we used a learning rate of 10 (although it was really more of a learning norm because we used normalized updates in Eq.(22)), and we used a learning rate decay of 0.9999 per update.
for stacked LSTM: Same as above but with learning rate/norm of 2. mLSTM was able to use a larger learning rate than LSTM .
for mLSTM on Penn Treebank: learning rate/norm of 7, learning rate decay of 0.9995 per update.
Pros:
* Clearly written.
* New model mLSTM which seems to be useful according to the results.
* Some interesting experiments on big data.
Cons:
* Number of parameters in comparisons of different models is missing.
* mLSTM is behind some other models in most tasks.
Thank you for your review.  Similar to our response to another reviewer, we would point out that mLSTM is a much simpler architectural modification that uses less depth than models that outperform it.
This paper proposes an extension of the multiplicative RNN [1] where the authors apply the same reparametrization trick to the weight matrices of the LSTM.
The paper proposes some interesting tricks, but none of them seems to be very crucial.
For instance, in Eq. (16), the authors propose to multiply the output gate inside the activation function in order to alleviate the saturation problem in logistic sigmoid or hyperbolic tangent.
Also, the authors share m_t across the inference of different gating units and cell-state candidates, at the end this brings only 1.25 times increase on the number of model parameters.
Lastly, the authors use a variant of RMSProp where they add an additional hyper-parameter $\ell$ and schedule it across the training time. It would be nicer to apply the same tricks to other baseline models and show the improvement with regard to each trick.
With the new architectural modification to the LSTM and all the tricks combined, the performance is not as great as we would expect. Why didn’t the authors apply batch normalization, layer normalization or zoneout to their models? Was there any issue with applying one of those regularization or optimization techniques?
At the fourth paragraph of Section 4.4, where the authors connect dynamic evaluation with fast weights is misleading. I find it a bit hard to connect dynamic evaluation as a variant of fast weights. Fast weights do not use test error signal. In the paper, the authors claim that “dynamic evaluation uses the error signal and gradients to update the weights, which potentially increases its effectiveness, but also limits its scope to conditional generative modelling, when the outputs can be observed after they are predicted”, and I am afraid to tell that this assumption is very misleading. We should never assume that test label information is given at the inference time. The test label information is there to evaluate the generalization performance of the model. In some applications, we may get the label information at test time, e.g., stock prediction, weather forecasting, however, in many other applications, we don’t.
For instance, in machine translation, we don't know what's the best translation at the end, unlike weather forecasting.
Also, it would be fair to apply dynamic evaluation to all the other baseline models as well to compare with the BPC score 1.19 achieved by the proposed mLSTM.
The quality of the work is not that bad, but the novelty of the paper is not that good either. The performance of the proposed model is oftentime worse than other methods, and it is only better when dynamic evaluation is coupled together.
However, dynamic evaluation can improve the other methods as well.
[1] Ilya et al., “Generating Text with Recurrent Neural Networks”, ICML’11
Thank you for your review. The review contains misunderstandings about the scope of dynamic evaluation. Dynamic evaluation can be applied without the ground truth labels by using labels generated by the model. This approach has been extensively studied in speech recognition and machine translation, through cache-based n-gram language models [1,2] and dynamic adaptation of recurrent neural networks [3]. This is similar to how a generative RNN can either use ground truth or generated sequence elements to update its hidden state. If applying dynamic evaluation to the ground truth sequence achieves a lower perplexity, it means that the network has a higher probability of generating the correct sequence from scratch by applying dynamic evaluation to sequence elements generated by the network, and would likely achieve a better word error rate, because it assigns a higher density to the correct answer.
We included dynamic evaluation in the paper because we felt it added to the paper's concept of being able to adjust to unexpected inputs.
[1] Kuhn et al. A cache-based natural language model for speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 6, pp. 570-583, Jun 1990.
[2] Nepveu et al. Adaptive Language and Translation Models for Interactive Machine Translation. EMNLP-2004.
[3] Kombrink et al. Recurrent Neural Network Based Language Modeling in Meeting Recognition. Interspeech-2011.
Could you please clarify if you used test ground truth labels in this experiment as like as done in [1]?
I wonder that if this assumption was made: output is observed after the prediction, and it can be used to compute the (test) error gradients.
[1] Graves, Generating Sequences With Recurrent Neural Networks, 2013
We applied dynamic evaluation in the same way as the referenced paper. For evaluating perplexity or bits/char on sequences, all standard generative RNNs observe test labels after they predict them.
So this is an assumption required to compute our cost function, but not inherently required by our model. We did not look at word error rate or related cost functions that do not use this assumption in this paper, but usually (but not always) a model that has a higher probability of generating the test data (as measured by perplexity or bits/char) can be decoded to achieve a lower word error rate on that test data.
* Brief Summary:
This paper explores an extension of multiplicative RNNs to the LSTM type of models. The resulting proposal is very similar to [1]. Authors show experimental results on character-level language modeling tasks. In general, I think the paper is well-written and the explanations are quite clear.
* Criticisms:
- In terms of contributions, the paper is weak. The motivation makes sense, however, very similar work has been done in [1] and already an extension over [2]. Because of that this paper mainly stands as an application paper.
- The results are encouraging.
On the other hand, they are still behind the state of art without using dynamic evaluation.
- There are some non-standard choices on modifications on the standard algorithms, such as "l" parameter of RMSProp and multiplying output gate before the nonlinearity.
- The experimental results are only limited to character-level language modeling only.
* An Overview of the Review:
Pros:
- A simple modification that seems to reasonably well in practice.
- Well-written.
Cons:
- Lack of good enough experimental results.
- Not enough contributions (almost trivial extension over existing algorithms).
- Non-standard modifications over the existing algorithms.
[1] Wu Y, Zhang S, Zhang Y, Bengio Y, Salakhutdinov RR. On multiplicative integration with recurrent neural networks. InAdvances in Neural Information Processing Systems 2016 (pp. 2856-2864).
[2] Sutskever I, Martens J, Hinton GE. Generating text with recurrent neural networks. InProceedings of the 28th International Conference on Machine Learning (ICML-11) 2011 (pp. 1017-1024).
Thank you for your review.
While the modifications could be considered trivial, we justify them clearly in the paper, in terms of architectural expressiveness. The presented approach achieves better results than those models that use similar but not the same modifications.  Our results are behind recurrent highway networks [1] and bytenet [2] on Wikipedia (without dynamic evaluation) - we use a much simpler model that is able to achieve competitive results with much less depth, and without any regularization method. Recurrent highway networks actually achieved a similar result to mLSTM (both 1.42 bits/char on Wikipedia) before recurrent highway networks were updated to include recurrent batch normalization.
Also, both of these results were submitted to arXiv around the same time as our paper was submitted to ICLR.
[1] Zilly et al. Recurrent Highway Networks. arXiv:1607.03474
[2] Kalchbrenner et al. Neural Machine Translation in Linear Time. arXiv:1610.10099

Meta-review:
role: PC
data: comment: The paper presents a new way of doing multiplicative / tensored recurrent weights in RNNs. The multiplicative weights are input dependent. Results are presented on language modeling (PTB and Hutter). We found the paper to be clearly written, and the idea well motivated. However, as pointed out by the reviewers, the results were not state of the art. We feel that is that this is because the authors did not make a strong attempt at regularizing the training. Better results on a larger set of tasks would have probably made this paper easier to accept. 
 
 Pros:
 - interesting idea, and reasonable results
 Cons:
 - only shown on language modeling tasks
 - results were not very strong, when compared to other methods (which typically used strong regularization and training like batch normalization etc).
 - reviewers did not find the experiments convincing enough, and felt that a fair comparison would be to compare with dynamic weights on the competing RNNs.

Decision: Invite to Workshop Track