Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Comments:
- LSTM with projection is quite standard. Some variant (LSTMP) was introduced in Long short-term memory recurrent neural network architectures for large scale acoustic modeling, by Google, in 2014. You don't compare your work to that. I think it is very related and should be compared.
- I think Convolutional LSTM Network (https://arxiv.org/abs/1506.04214, from 2015) are also related because you can also see that as kind of grouping.
- The plot in Figure 2 of the training loss is very nice. I think, in addition, it would be nice to see the same plot but with the training computation time on the X-axis, so you can better see, e.g. after 1 week of training, where you are with each model. In TensorBoard, I think there is even an option to do that.

Cons:
- Lacking related work and comparisons.
- Lacking experiments on other tasks.

Pros:
- State-of-the-art result on OBW.
- Open Source code.

review_title: nice work

final_score: rating: 6: Marginally above acceptance threshold
review_content: Factorization Tricks for LSTMs

Summary:

This paper empirically investigates different ways of learning parameters of LSTMs to optimize the computational and the memory efficiency of the models. They use LSTMP model from Sak et al 2014 as their baseline and implement two different tricks to speed up their model. The first trick is to use factorize the weight matrices of the neural network to introduce a lower-dimensional bottleneck and the second approach is to create block structure in  They report convincing results on 1-billion word language modeling benchmark. The factorization methods investigated in this paper seems to act like a regularizer and the improve the generalization error as well. 

General Comment:
It seems like this paper is missing some important references. I would cite [1] for the factorization trick, and progressive networks [2] and block-diagonal recurrent neural networks[3](there have been several other similar papers can be found in the literature, this is the oldest one I could find) for the group structure that you are introducing. The results are interesting.

Overall Review:

Pros,
The empirical investigation of two different ways to reparametrize LSTM models to speed up and lower the memory consumptions.
Experiments are convincing and the results are good

Cons,
I think you are missing an important baseline, a regular LSTM language model without projection(not the LSTMP by Sak et al).
The writing needs more work, the notation used is a bit difficult to parse.

Detailed and some Minor comments:

On Page 1, “transformation 1” —> "Equation 1”
Figure 1, needs more description and clarification it is not very clear what d1, d, d2 means. 
Please use a more formal notation, use subscript for the weights. There are variables used in the equations without being properly defined.
I would like to see a discussion about the computational complexity of those approaches as well.
I would like to see the Figure 2 with respect to wall-clock time in the x-axis.
After some more work, this paper can be made much easier to read. This version of the paper is not very easy to understand, unfortunately.
Can you plot Figure 2 in log-scale, the differences between the learning curves are not very clear to me.


[1] Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, Predicting parameters in deep learning, NIPS 2013.
[2] Rusu, Andrei A., Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. "Progressive neural networks." arXiv preprint arXiv:1606.04671 (2016).
[3] Sivakumar, Shyamala C., William Robertson, and William J. Phillips. "Online stabilization of block-diagonal recurrent neural networks." IEEE Transactions on Neural Networks 10.1 (1999): 167-175.
review_title: Interesting paper but needs some more work

Decision: Accept