Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I have a few questions regarding the experimental section.


1) Why don't you compare low-rank GRU with a GRU baseline for the memory variant and addition tasks?

2) Memory and adding tasks are usually used to show that model can capture long-term dependencies. Do you think that the low-rank and low-rank+diagonal parametrizations help with that problem?

4) What is the total number of parameters for the different models PTB and the permuted sequential mnist tasks?

5) How exactly is computed test per-char perplexity on the char-level PTB experiment and how your model compare to state-of-art models in term of bits per character ? 

review_title: Prereview question

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors study the use of low-rank approximation to the matrix-multiply in RNNs. This reduces the number of parameters by a large factor, and with a diagonal addition (called low-rank plus diagonal) it is shown to work as well as a fully-parametrized network on a number of tasks.

The paper is solid, the only weakness being some claims about conceptual unification (e.g., the first line of the conclusion -- "We presented a framework that unifies the description various types of recurrent and feed-forward
neural networks as passthrough neural networks." -- claiming this framework as a contribution of this paper is untrue, the general framework is well known in the community and RNNs have been presented in this way before.)

Aside from the above small point, the true contribution is in making low-rank RNNs work, the results are generally as good as fully-parametrized networks. They are hardly better though, which makes it unclear why low-rank networks should be used. The contribution is thus not very strong in terms of results, but even achieving the same results with fewer parameters is not easy and the studies were well-executed and explained.
review_title: Exploring a solid idea, but results are not convincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: The author proposes the use of low-rank matrix in feedfoward and RNNs. In particular, they try their approach in a GRU and a feedforward highway network.

Author also presents as a contribution the passthrough framework, which can describe feedforward and recurrent networks. However, this framework seems hardly novel, relatively to the formalism introduced by LSTM or highway networks.

An empirical evaluation is performed on different datasets (MNIST, memory/addition tasks, sequential permuted MNIST and character level penntreebank). 

However, there are few problems with the evaluation:

- In the highway network experiment, the author does not compare with a baseline.
We can not assess what it the impact of the low-rank parameterization. Also, it would be interesting to compare the result with a highway network that have this capacity bottleneck across layer  (first layer of size $n$, second layer of size $d$, third layer of size $n$) and not in the gate functions. Also, how did you select the hyperparameter values?.

- It is unfortunate that the character level penntreebank does not use the same experimental setting than previous works as it prevents from direct comparison.
Also the overall bpc perplexity seems relatively high for this dataset. It is therefore not clear how low-rank decomposition would perform on this task applied on a stronger baseline.

-Author claims state-of-art in the memory task. However, their approach uses  more parameters than the uRNN (41K against 6.5K for the memory) which makes the comparison a little bit unfair toward uRNN. It would be informative to see how low-rank RNN performs using overall 6.5K parameters. Generally, it would be good to see what is the impact of the matrix rank given a fix state size.

- It would be informative as well to have the baseline and the uRNN curve in Figure 2 for the memory/addition task.

- it is not clear when to use low-rank or low-rank + diagonal from the experiments.

Overall, the evaluation in its current form in not really convincing, except for the sequential MNIST dataset.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a low-rank version of pass-through networks to better control capacity, which can be useful in some cases, as shown in the experiments.
That said, I found the results not very convincing overall. Results are overall not as good as state-of-the-art on sequential MNIST or the memory task, but add one more hyper-parameter to tune. As I said, it would help to show in Tables and/or Figures competing approaches like uRNNs.
review_title: my review

final_score: rating: null
review_content: The paper is clear, just to make the task go away.
review_title: No question

final_score: rating: null
review_content: - Two proposals are made (passthrough and lowrank) and experiments are shown with lowrank only, lowrank and passthrough, but not passthrough only. Why?
- Figure 2 only shows curves from the proposed models, but not for baselines; why?
- In Table 1 and Table 2, what happens when the state size is higher for the baseline?

review_title: Questions

Meta-review: The reviewers seem to agree that the framework presented is not very novel, something I agree with.
 The experiments show that the low rank + diagonal parameterization can be useful, however. The paper could be improved by making a more tightened message, and clearer arguments. As it currently stands, however it does not seem ready for publication in ICLR.
Decision: Reject