Title: Low-rank passthrough neural networks
Authors: Antonio Valerio Miceli Barone
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Deep learning consists in training neural networks to perform computations that sequentially unfold in many steps over a time dimension or an intrinsic depth dimension. For large depths, this is usually accomplished by specialized network architectures that are designed to mitigate the vanishing gradient problem, e.g. LSTMs, GRUs, Highway Networks and Deep Residual Networks, which are based on a single structural principle: the state passthrough. We observe that these "Passthrough Networks" architectures enable the decoupling of the network state size from the number of parameters of the network, a possibility that is exploited in some recent works but not thoroughly explored. In this work we propose simple, yet effective, low-rank and low-rank plus diagonal matrix parametrizations for Passthrough Networks which exploit this decoupling property, reducing the data complexity and memory requirements of the network while preserving its memory capacity. We present competitive experimental results on several tasks, including a near state of the art result on sequential randomly-permuted MNIST classification, a hard task on natural data.

TL;DR: Describe low-rank and low-rank plus diagonal parametrizations for Highway Neural Networks, GRUs and other kinds of passthrough neural networks. Present competitive experimental results.

Review discussion:
- Two proposals are made (passthrough and lowrank) and experiments are shown with lowrank only, lowrank and passthrough, but not passthrough only. Why?
- Figure 2 only shows curves from the proposed models, but not for baselines; why?
- In Table 1 and Table 2, what happens when the state size is higher for the baseline?
- The two proposal are "low-rank" and "low-rank plus diagonal". Diagonal-only, which would be essentially a bag-of-words model with some temporal weighting, was not considered because I conjecture that it would not be expressive enough for practical tasks.
- I did not re-implement the uRNN and Associative LSTM models. Training curves are available in the referenced papers, and they are comparable since the experimental setups are the same.
- For Sequential permuted MNIST, I experimented with a full-rank GRU with state size equal to 256 and it failed to converge due to training instability. Different choices of learning rate, etc. could possibly make it work, but this is larger than what is commonly used in the literature for this task so I didn't investigate it any further. For character-level language model I did not try to use baselines larger than 1000 (which is the common size found in the literature) but I expect overfitting, especially for the largest setting. GPU memory utilization could be also an issue. If I can get some GPU time in the next few days I may try to run a few more experiments on language modeling. Anyway, the main claim for this task is that our proposal obtains a better perplexity for the same number of parameters.
I ran a LM experiment with a larger GRU baseline and it did not overfit more than the other models (thanks to recurrent dropout, I presume).
In fact it performed similarly to my best low-rank+diagonal model, although with a much higher number of parameters. With that many parameters, it should be possible in principle to train low-rank and low-rank+diagonal models with a much larger state size, but I found it difficult to it on the GPUs that I have access to because of memory constraints (a low-level implementation optimized for memory consumptions could possibly help, although I didn't investigate this).
I also fixed a result in table 2 (the 5459-64 GRU) and some details in appendix A.2 (learning rate and lack of character embeddings).
I have a few questions regarding the experimental section.
1) Why don't you compare low-rank GRU with a GRU baseline for the memory variant and addition tasks?
2) Memory and adding tasks are usually used to show that model can capture long-term dependencies. Do you think that the low-rank and low-rank+diagonal parametrizations help with that problem?
4) What is the total number of parameters for the different models PTB and the permuted sequential mnist tasks?
5) How exactly is computed test per-char perplexity on the char-level PTB experiment and how your model compare to state-of-art models in term of bits per character ?
> 1) Why don't you compare low-rank GRU with a GRU baseline for the memory variant and addition tasks?
The original papers report results for LSTM models, and papers that compare GRUs and LSTMs generally found that they perform comparably for the same state size, which is consistent with the results I obtained here when I did the comparisons, therefore I assumed that I could avoid doing the comparison with GRU baselines for these tasks in order to save computational time.
> Memory and adding tasks are usually used to show that model can capture long-term dependencies. Do you think that the low-rank and low-rank+diagonal parametrizations help with that problem?
Low-rank and low-rank+diagonal parametrizations allow to use larger state size for the same number of parameters (which is practically limited by available dataset size and computational resources), increasing the memory capacity of the model, therefore helping it to capture long-term dependencies.
In tasks on synthetic data, which is in principle unlimited, overfitting is impossible, therefore even a full-matrix model should eventually reach the performance of a low-rank or low-rank+diagonal model if trained for long enough (assuming that the optimizer doesn't get stuck at some bad critical point). In tasks on natural data, which is limited in size, a full-matrix model can overfit sooner than a low-dimensional one, resulting in lower generalization accuracy.
Specifically, adding a single node to a hidden layer parametrized by full nXn matrices adds O(n) parameters to the model, therefore when n is large, even adding a single node can change a model that underfits because it has too little parameters into one that overfits because it has too many parameters, without ever hitting close to the ideal number of parameters for the task. Low-dimensional parametrizations can help controlling the parameter number without sacrificing the state size.
> 4) What is the total number of parameters for the different models PTB and the permuted sequential mnist tasks?
Added in the paper.
> 5) How exactly is computed test per-char perplexity on the char-level PTB experiment and how your model compare to state-of-art models in term of bits per character ?
The test per-char perplexity is the base-2 exponential of the bits-per-character entropy. I added a citation to existing work. My model performs worse than published results, although the experimental setup is slightly different (published results usually maintain the RNN state between sentences and use a slightly different character set).
In any case, my goal here is not to set a new SOTA but to show a relative improvement of the proposed parametrizations w.r.t. the full-matrix baseline.
The paper is clear, just to make the task go away.
The authors study the use of low-rank approximation to the matrix-multiply in RNNs. This reduces the number of parameters by a large factor, and with a diagonal addition (called low-rank plus diagonal) it is shown to work as well as a fully-parametrized network on a number of tasks.
The paper is solid, the only weakness being some claims about conceptual unification (e.g., the first line of the conclusion -- "We presented a framework that unifies the description various types of recurrent and feed-forward
neural networks as passthrough neural networks." -- claiming this framework as a contribution of this paper is untrue, the general framework is well known in the community and RNNs have been presented in this way before.)
Aside from the above small point, the true contribution is in making low-rank RNNs work, the results are generally as good as fully-parametrized networks. They are hardly better though, which makes it unclear why low-rank networks should be used. The contribution is thus not very strong in terms of results, but even achieving the same results with fewer parameters is not easy and the studies were well-executed and explained.
Thanks for your comments.
> the only weakness being some claims about conceptual unification (e.g., the first line of the conclusion -- "We presented a framework that unifies the description various types of recurrent and feed-forward
neural networks as passthrough neural networks." -- claiming this framework as a contribution of this paper is untrue, the general framework is well known in the community and RNNs have been presented in this way before.)
Fair point, I removed that sentence.
> They are hardly better though, which makes it unclear why low-rank networks should be used.
For memory, addition and permuted sequential MNIST they work better than full-rank networks, presumably due to a regularization effect. The contribution is thus not very strong in terms of results, but even achieving the same results with fewer parameters is not easy and the studies were well-executed and explained.
On language modelling, even with the additional regularization of dropout, they work pretty much as good as full-rank networks for the same state size, and better for the same parameter size, thus at the very least they could be used as a from of model compression.
The paper proposes a low-rank version of pass-through networks to better control capacity, which can be useful in some cases, as shown in the experiments.
That said, I found the results not very convincing overall. Results are overall not as good as state-of-the-art on sequential MNIST or the memory task, but add one more hyper-parameter to tune. As I said, it would help to show in Tables and/or Figures competing approaches like uRNNs.
Results are SOTA on the memory tasks as far as I know.
sounds good, why not show the competitive approaches in the tables/figures then?
Results are presented as plots in the original papers (uRNN, Associative LSTM and Orthogonal RNN) and they are a bit difficult to regenerate given the limited hardware resources I have currently access to (uRNN and Associative LSTM in particular are slow to train). Code for Associative LSTM and Orthogonal RNN is also not available, as far as I know.
I've added uRNN results in the figure and results for an additional variant of LRD-GRU.
I've also added LSTM and LRD-LSTM results for the language modeling task.
The author proposes the use of low-rank matrix in feedfoward and RNNs.
In particular, they try their approach in a GRU and a feedforward highway network.
Author also presents as a contribution the passthrough framework, which can describe feedforward and recurrent networks.
However, this framework seems hardly novel, relatively to the formalism introduced by LSTM or highway networks.
An empirical evaluation is performed on different datasets (MNIST, memory/addition tasks, sequential permuted MNIST and character level penntreebank).
However, there are few problems with the evaluation:
- In the highway network experiment, the author does not compare with a baseline.
We can not assess what it the impact of the low-rank parameterization.
Also, it would be interesting to compare the result with a highway network that have this capacity bottleneck across layer  (first layer of size $n$, second layer of size $d$, third layer of size $n$) and not in the gate functions.
Also, how did you select the hyperparameter values?.
- It is unfortunate that the character level penntreebank does not use the same experimental setting than previous works as it prevents from direct comparison.
Also the overall bpc perplexity seems relatively high for this dataset. It is therefore not clear how low-rank decomposition would perform on this task applied on a stronger baseline.
-Author claims state-of-art in the memory task.
However, their approach uses  more parameters than the uRNN (41K against 6.5K for the memory) which makes the comparison a little bit unfair toward uRNN. It would be informative to see how low-rank RNN performs using overall 6.5K parameters. Generally, it would be good to see what is the impact of the matrix rank given a fix state size.
- It would be informative as well to have the baseline and the uRNN curve in Figure 2 for the memory/addition task.
- it is not clear when to use low-rank or low-rank + diagonal from the experiments.
Overall, the evaluation in its current form in not really convincing, except for the sequential MNIST dataset.
Thanks for your comments.
> - In the highway network experiment, the author does not compare with a baseline
The highway network experiment was exploratory, and I didn't have time to carry out all the proper evaluation.
Therefore I moved it to the appendix. May remove it from the paper if the reviewers agree.
> - It is unfortunate that the character level penntreebank does not use the same experimental setting than previous works as it prevents from direct comparison.
I now tried to replicate (Graves 2013) setup, although I still get worse results.
Nevertheless, I still get an improvement using the Low-rank plus diagonal parametrization on an LSTM model, indicating that the method is robust w.r.t. different architectures.
> -Author claims state-of-art in the memory task.
However, their approach uses  more parameters than the uRNN (41K against 6.5K for the memory) which makes the comparison a little bit unfair toward uRNN.
Fair point, I have now included a direct comparison to a uRNN with a large state size and also some experiments with a variant of the LRD-GRU.
> -  It would be informative to see how low-rank RNN performs using overall 6.5K parameters.
> Overall, the evaluation in its current form in not really convincing, except for the sequential MNIST dataset.
Hope that now it's better.
I've tried in different configurations but it fails.
> - It would be informative as well to have the baseline and the uRNN curve in Figure 2 for the memory/addition task.
Added the curve for the memory task, although for a different state size than in the uRNN paper.
> -  it is not clear when to use low-rank or low-rank + diagonal from the experiments.
The low-rank is sufficient in most cases but not on the memory task. Since the low-rank plus diagonal adds a very small amount of complexity and parameters, there seems no reasons not to always use it.
I added additional results on the memory tasks and language modeling task.
For the memory task, as suggested by the reviewers, I compare against uRNN models with approx. the same number of parameters as the LRD-GRU. This uRNN, much larger than the one used in the original uRNN paper, converges quickly on the simple memory task but overfits (training was terminated by early stopping in the reported graph).
On the variable sequence length task (Danihelka 2016) it trains faster (with somewhat larger oscillations) than the LRD-GRU, but on the variable lag task (Henaff 2016) it fails.
I've also considered a variant of the LRD-GRU which I tried only on these two subtasks. In this variant I add weight normalization (Salimans 2016) and a weight max-row-norm constraint in order to reduce the amounts of time that NaN recovery triggers. With this modification, NaN recovery is unnecessary, and the models train much faster.
In fact, it performs on par with the uRNN even on the (Danihelka 2016) task.
For the language modelling task, I did experiments with a LSTM baseline and a low-rank plus diagonal LSTM, while trying to use the same setup of (Graves 2013). I still can't replicate those results (the code is online at https://github.com/Avmb/lowrank-lstm feel free to have a look at it), but I get an improvement from the low-rank plus diagonal parametrization compared to the baseline.
I also moved the experiments on the Highway Networks in the appendix, describing them as exploratory and preliminary. It may make sense to remove them completely from the paper if the reviewers agree.
I removed the phrasing that implied that the framework was a novel contribution.

Meta-review:
role: PC
data: comment: The reviewers seem to agree that the framework presented is not very novel, something I agree with.
 The experiments show that the low rank + diagonal parameterization can be useful, however. The paper could be improved by making a more tightened message, and clearer arguments. As it currently stands, however it does not seem ready for publication in ICLR.

Decision: Reject