Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: * Brief Summary: 

This paper explores an extension of multiplicative RNNs to the LSTM type of models. The resulting proposal is very similar to [1]. Authors show experimental results on character-level language modeling tasks. In general, I think the paper is well-written and the explanations are quite clear.

* Criticisms:

- In terms of contributions, the paper is weak. The motivation makes sense, however, very similar work has been done in [1] and already an extension over [2]. Because of that this paper mainly stands as an application paper.
- The results are encouraging. On the other hand, they are still behind the state of art without using dynamic evaluation. 
- There are some non-standard choices on modifications on the standard algorithms, such as "l" parameter of RMSProp and multiplying output gate before the nonlinearity.
- The experimental results are only limited to character-level language modeling only. 

* An Overview of the Review:

Pros:
- A simple modification that seems to reasonably well in practice.
- Well-written.

Cons:
- Lack of good enough experimental results.
- Not enough contributions (almost trivial extension over existing algorithms).
- Non-standard modifications over the existing algorithms.

[1] Wu Y, Zhang S, Zhang Y, Bengio Y, Salakhutdinov RR. On multiplicative integration with recurrent neural networks. InAdvances in Neural Information Processing Systems 2016 (pp. 2856-2864).
[2] Sutskever I, Martens J, Hinton GE. Generating text with recurrent neural networks. InProceedings of the 28th International Conference on Machine Learning (ICML-11) 2011 (pp. 1017-1024).
review_title: An Interesting paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an extension of the multiplicative RNN [1] where the authors apply the same reparametrization trick to the weight matrices of the LSTM. 

The paper proposes some interesting tricks, but none of them seems to be very crucial. For instance, in Eq. (16), the authors propose to multiply the output gate inside the activation function in order to alleviate the saturation problem in logistic sigmoid or hyperbolic tangent. Also, the authors share m_t across the inference of different gating units and cell-state candidates, at the end this brings only 1.25 times increase on the number of model parameters. Lastly, the authors use a variant of RMSProp where they add an additional hyper-parameter $\ell$ and schedule it across the training time. It would be nicer to apply the same tricks to other baseline models and show the improvement with regard to each trick.

With the new architectural modification to the LSTM and all the tricks combined, the performance is not as great as we would expect. Why didn’t the authors apply batch normalization, layer normalization or zoneout to their models? Was there any issue with applying one of those regularization or optimization techniques? 

At the fourth paragraph of Section 4.4, where the authors connect dynamic evaluation with fast weights is misleading. I find it a bit hard to connect dynamic evaluation as a variant of fast weights. Fast weights do not use test error signal. In the paper, the authors claim that “dynamic evaluation uses the error signal and gradients to update the weights, which potentially increases its effectiveness, but also limits its scope to conditional generative modelling, when the outputs can be observed after they are predicted”, and I am afraid to tell that this assumption is very misleading. We should never assume that test label information is given at the inference time. The test label information is there to evaluate the generalization performance of the model. In some applications, we may get the label information at test time, e.g., stock prediction, weather forecasting, however, in many other applications, we don’t. For instance, in machine translation, we don't know what's the best translation at the end, unlike weather forecasting. Also, it would be fair to apply dynamic evaluation to all the other baseline models as well to compare with the BPC score 1.19 achieved by the proposed mLSTM.

The quality of the work is not that bad, but the novelty of the paper is not that good either. The performance of the proposed model is oftentime worse than other methods, and it is only better when dynamic evaluation is coupled together. However, dynamic evaluation can improve the other methods as well.

[1] Ilya et al., “Generating Text with Recurrent Neural Networks”, ICML’11
review_title: Not bad work

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
* Clearly written.
* New model mLSTM which seems to be useful according to the results.
* Some interesting experiments on big data.

Cons:
* Number of parameters in comparisons of different models is missing.
* mLSTM is behind some other models in most tasks.

review_title: useful new LSTM variant

final_score: rating: null
review_content: What is the reference to tensor RNN in section 1.2?

I think you can compare mRNN and mLSTM also to Multi-Function Recurrent Unit (MuFuRU) (https://arxiv.org/abs/1606.03002)?

Why do you chose the variant in equation (16) to use tanh after the output gating?
How does it differ in performance to the usual variant?

In Table 1, 2 & 3, what are the numbers of parameters and the hidden dimensions for each of the presented models?

In Table 3, is it fair to compare with models which use the test data indirectly? How is it justified to use the error signal from test data? In a real-world evaluation, when you don't know the real target, you cannot do that.

Did you compare with multiplicative integration LSTMs? And other LSTM variants such as Associative LSTMs?

Your stacked LSTM models were all only 2 layers. Have you tried deeper LSTM models?

You say that mLSTM's advantage over stacked LSTM was greater after a suprising input than it is in general. By how much?


review_title: other models, more details

final_score: rating: null
review_content: What is the motivation behind Eq. (16)?

What if the multiplicative LSTM is trained with additional regularization or training methods (zoneout, dropout, batch normalization, layer normalization)?

What are the learning rates used in your experiments (for each dataset) and hyper parameters of RMSProp?
review_title: Some more details

final_score: rating: null
review_content: 
* What is the memory consumption and efficiency of this technique compared to regular RNNs or LSTMs? Can you add a few sentence for the clarifications into the paper?
* Do you think if it is possible, apply this model to the word-level language modeling? What kind of challenges need to be solved for that?
* On char-LM tasks to be able to obtain more comparable results to the SOTAs, have you tried recurrent/variation dropout and layer/batch norm?
* What other tasks do you think this model can be useful for?
* What kind of initialization have you used for the transition matrices?
 
review_title: Some small details  

Meta-review: The paper presents a new way of doing multiplicative / tensored recurrent weights in RNNs. The multiplicative weights are input dependent. Results are presented on language modeling (PTB and Hutter). We found the paper to be clearly written, and the idea well motivated. However, as pointed out by the reviewers, the results were not state of the art. We feel that is that this is because the authors did not make a strong attempt at regularizing the training. Better results on a larger set of tasks would have probably made this paper easier to accept. 
 
 Pros:
 - interesting idea, and reasonable results
 Cons:
 - only shown on language modeling tasks
 - results were not very strong, when compared to other methods (which typically used strong regularization and training like batch normalization etc).
 - reviewers did not find the experiments convincing enough, and felt that a fair comparison would be to compare with dynamic weights on the competing RNNs.
Decision: Invite to Workshop Track