Reduced-Gate Convolutional LSTM Design Using Predictive Coding for Next-Frame Video Prediction

Abstract: Spatiotemporal sequence prediction is an important problem in deep learning. We
      study next-frame video prediction using a deep-learning-based predictive coding
      framework that uses convolutional, long short-term memory (convLSTM) modules.
      We introduce a novel reduced-gate convolutional LSTM architecture. Our
      reduced-gate model achieves better next-frame prediction accuracy than the original
      convolutional LSTM while using a smaller parameter budget, thereby reducing
      training time. We tested our reduced gate modules within a predictive coding architecture
      on the moving MNIST and KITTI datasets. We found that our reduced-gate
      model has a significant reduction of approximately 40 percent of the total
      number of training parameters and training time in comparison with the standard
      LSTM model which makes it attractive for hardware implementation especially
      on small devices.

Reviews and author responses:

Thanks for pointing out the link to the revision. 

After rereading the paper, I'm still not entirely convinced by the proposed model, neither by the intuition or by the experiments. Since this paper conducts experimental research, it may make sense to compare the standard LSTM and the proposed rgcLSTM using a simpler task before moving on to the complex model, predNet. A simple case can more straightforwardly show what we can gain and will lose by using rgcLSTM. As shown in the experiments, both quantitative and qualitative results are not significantly better than the original predNet. It is reasonable to question that the rgcLSTM might lose some capacity, e.g., memorizing a long sequence, but this is not noticeable maybe because it reduces the model complexity of the predNet, which simplifies the optimization and reaches to a comparable local minimum. We could use rgcLSTM in predNet to gain some efficiency, but how about other models built on convLSTM? If the paper only focuses on this particular application, the contribution is incremental. 

Dear AnonReviewer2,
We apologize for this mistake during uploading the last version of our paper. We contacted to the ICLR 2019 Program Chairs about this issue and they said to us that we can mention to you the previous version as we can not upload the new one. 
We have the same version that was uploaded correctly on (17th November )
in the Revision History link:  https://openreview.net/revisions?id=rJEyrjRqYX
Or its direct link: 
https://openreview.net/references/pdf?id=SJCkcu0pm

The version that we intended to upload but we missed the correct file uploading is the same as the draft version that we mentioned above except that we had corrected Figure 1 as last tanh on the right hand side is not on the C^t arrow.  It was a mistake during a trial to reduce the figure design size which caused the mis-matching between the equations and the figure. 
We appreciate your time and considerations. We would be pleased if you review the version that mentioned above which contains all the corrections and recommendation you and the other reviewers mentioned. And please consider the correction about  Figure 1 what we did but missed to upload correctly.
We apologize for this inconvenience due to our mistake that happened due our fear of missing the deadlines. 


After reading the authors' response, I'd like to check the updated manuscript. However, the attached pdf only includes the formatting instructions. Could you fix this first?



1. Apologies, I misread your sentence in the paper. I will amend my review accordingly.

2. The term "vanilla" means indeed "plain" or "basic", which is why would be better referred to the 1997 implementation rather than the more advanced one with the forget gates. Nonetheless, being a matter of semantics I don't want to further object to your choice of referring to the 2000 implementation as vanilla. What is more important is that in the manuscript you seem to imply that both the implementation by Gers et al (2000) and that of Lotter et al. (2017) shall be referred to vanilla LSTMs: "[referring to Lotter et al (2017)] Greff et al (2017) has termed this design the vanilla LSTM". This is confusing.

3. Apologies for not being more clear. I referred to this sentence: "There were other attempts to design smaller recurrent gated models [...] based on either removing one of the gates or the activation functions [...] these models had no significant reduction in trainable parameters".

4. Thank you for adding subsection 4.1. I believe this subsection would be more clear if it was part of the model description in section 3 and referred to the formulas in that section. 
Most importantly, while I understand the practical motivations behind this modelling choice, stacking a tanh and a sigmoid nonlinearities in a row will make the optimization problem harder, constrain the network and limit its manoeuvre space. In particular, the sigmoid activation of the gate will likely be in the [0.3, 0.7] range when given an input in [-1, 1] if I am not mistaken. This halves the range of values one would expect the gate to be able to take. 
I believe this is the main problem of the suggested approach and that a much more convincing justification than the findings about the LSTMs gates reported by Greff et Al is needed.

It is also unclear to me why it would be sensible to tie the values of the input, forget and output gates, given that they play a very different and in some cases complementary role. Again, this approach should be fully justified with an intuition as to why this should be a sensible idea.

Furthermore, I still find the description of the network very difficult to follow. I suggest to move the discussion on the number of parameters and the description of the dimensionality of the layers to the end of the paragraphs/section, in the interest of readability. I also recommend to mimic the notation used in the original convLSTM paper that the authors compare against (I refer to eq 3 of the original paper), to make it easier for the reader to spot differences and similarities.

More to this, it is unclear to me the difference between the "network gate value" f_gate and the "gate value" f. In particular the former doesn't seem to be used in any of the following equations, and neither of them is depicted in the figures.

To summarise, I believe that the manuscript still fails to provide a convincing justification for the proposed approach of tying the LSTM gates together and the clarity of the description of the model is still unsatisfactory. I am also very sceptical about the modelling choice of stacking two nonlinearities in a row to compute the activations of the gate. Finally, I agree with Reviewer 2 that the contribution of tying the weights together and replacing Hadamard product with convolution seems to be marginal.

* there is a typo in the updated version of the manuscript, just after eq 1: hight --> height 

The authors propose a variant to convLSTMs with convolutional peephole units (as opposed to the original Hadamard product) and tied gates. The description of the model is confusing, the authors don't offer a strong justification for the proposed approach, some of the technical choices seem flawed.

It is also obviously false that removing an LSTM gate does not incur in a reduction of trainable parameters: "There were other attempts to design smaller recurrent gated models [...] based on either removing one of the gates or the activation functions [...] these models had no significant reduction in trainable parameters".

The model description is difficult to follow, but from what I can gather the model depiction is flawed. In particular, the proposed model bases on the idea to use tied gates to reduce the number of parameters. f^{(t)}, i.e., the value that governs the input, forger and output gates, is derived via a sigmoid. When the sigmoid is zero, the network will hence set all the gates to zero, ignoring the input, forgetting the memory and suppressing the output, which is hardly a behaviour one could ever want. Unfortunately, almost half of the sigmoid spectrum will lead to such behavior. This seems to be a poor modeling decision.

The nodes in Figure 1 seem to suggest that two consecutive non-linearities are applied in a row, if W_fc is clamped to zero, as reported by the authors, there is no reason to specify it, the corresponding part of the equation can be removed and the narrative simplified. In sec3, the meaning of “net input image” and “network gate image value” is unclear. Also the description of the equation is hard to follow, e.g., the square bracket notation is eventually explained only after 8 lines of text.

At the beginning of Sec4 err_1 and err_2 are defined as the difference between the predicted frame and the target frame, and vice versa. This error is then fed to the “rgcLSTM input arranger unit and to the next higher layer”. By my understanding the error of one layer is fed in the next as an input. I wonder if such error is provided also at inference time, giving a clear guidance to the network to produce the correct output exploiting privileged information. This could also explain why the training process was completed in only one epoch.

- Typos:
*Intro: More important → more importantly
* page5: ReL -> ReLu

1. More datasets could be used to demonstrate the enhancement in performance.
Response: This is a helpful suggestion which, because of practical limitations, we will reserve for future work. We have added this point to the conclusions.

1. Criticism: Update gate in GRU does not combine input gate, forget gate, and memory gate and memory unit of LSTMs.
Response: We quote below from the following paper as our source, which we cited in our paper, and of which Schmidhuber is a co-author.

. . . Gated Recurrent Unit (GRU). They used neither peephole connections
nor output activation functions, and coupled the input and the forget gate
into an update gate. Finally, their output gate (called reset gate) . . .

The above is from the last paragraph in Section III of the paper by Greff, Srivastava, Koutnik, Steunebrink, and Schmidhuber \LSTM: A Search Space Odyssey," Transactions on Neural Networks and Learning Systems, 2017, axXiv:1503:04069v2.
Please reread the last paragraph of page 2 of our paper and then look at the above quotation.


2. The term vanilla is \normally" used as a synonym to \original." Vanilla refers to the
original 1997 implementation not to Greff et al.'s work . . .
Response: We stand by our use of the term \vanilla." This term comes from a Cambridge ice-cream shop which was frequented in the 1970's by the developers of the first Lisp-based object-oriented system known as Flavors. It referred to the base class in an object-oriented system. To see if the meaning changed or evolved over the
decades, I searched for a definition on the web. It still means "plain" or "basic," not "original."

More to the point, the original (1997) LSTM of Hochreiter & Schmidhuber does not have a forget gate and all modern "basic" LSTMs do have forget gates. This gate wasn't added until the (2000) publication that we cite in our submitted paper. All modern LSTMs use a forget gate and it has since become known that the forget gate is the most important gate as stated in Greff et al (2017). Schmidhuber, an inventor of the LSTM, is the senior author on that (2017) paper cited above in point 1 which designated that the LSTM with all three gates and no peephole connections is the
"vanilla" LSTM. Section II of the Greff et al paper is titled Vanilla LSTM. We followed their lead.

3. "It is obviously false that removing an LSTM gate does not incur a reduction in parameters."
Response: Either the above is a misstatment, or we agree with the referee and never stated otherwise.
Each gate has a set of associated incoming weights (parameters). Removing a gate removes the associated weights. We've added an appendix showing how the parameters were counted to remove any doubt about the parameter reduction.

4. When the sigmoid is zero, the network will hence set all the gates to zero, ignoring the input, forgetting the memory and suppressing the output, which is hardly the behavior once could ever want.
Response: Yes, the gate value must stay in a functional range. We have added a subsection 4.1 titled "Keeping the gate output within a functional range" to explain how this is addressed in our application.
The referee asked why there where two consecutive non-linearities (tanh) in a row. The positioning of the second tanh forces both c and h stay within the range (0; 1), and since they are both input to the network gate, this improves the likelihood of the network gate value staying in a functional range.

5. In sec3, the meaning of "net input image" and "network gate image value" is unclear.
Response: The confusion seems to come from use of the word "image." We've added a sentence at the beginning of Sec3 after the introduction of Figure 1 which states that "Since this is a convolutional LSTM, the information on each wire is a multi-channel image." We have also removed the word "image" from the above mentioned phrases.

6. The square bracket notation is eventually explained only after 8 lines of text.
Response: We moved the square bracket sentence forward to just after the equation where the notation is first used and revised the sentence to be more explicit: "The square brackets indicate that multi-channel images or filters with compatible dimensions are stacked on top of each other."




1. The description in the paper is very confusing.
Response: We have improved the clarity of figure 2. We have also added an appendix giving many more details of the model architecture. The main purpose of the appendix is to spell out how the parameter counts were performed but a side-effect is to offer a quite detailed specification of the architecture. We have also put the source code on github and listed the link in Section 4 of the paper.

2. Something is wrong with Eqns. 14 and 15.
Response: We have been continuing work on the project since our initial submission to ICLR and have found these errors. We have corrected these parameter count equations in the revision.

3. Eqns 5 and 6 are not consistent.
Response: The inconsistency was due to an error in Eqn 6. We have corrected this.

4. The intuition of having one gate instead of three is not clear. Mixing all the functions, i.e., input, forget, and output filters, does provide freedom for learning but also introduces learning complexity. Decoupling a complex function into a couple of simpler functions is a common way to narrow down the search space. This is what the standard LSTM does. The authors should provide reasonable arguments to explain intuitively why using one gate is better.
Response: The decoupling advice of the referee is very persuasive. We weren't fully cognizant of this at the start of the project. Our main inspiration comes from the paper of Greff et al (2017) which concludes on the basis of a vast number of empirical studies that the LSTM variant with three gates and no peephole connections gives the best overall performance. They also note that the forget gate is the most important gate in the LSTM. Inspired by the GRU and the MGU, we just tried to see what would happen if we used one gate for all three functions. Since this was a convolutional LSTM, we also decided to replace the original peephole connections that were implemented by an elementwise multiply in the Shi et al model with a convolutional operation. This seemed far more appropriate for image processing. We tried various alternatives and had the most success with the model proposed in our paper.

5. The experimental performance comparisons would be fairer if we let the models converge.
Response: For the KITTI dataset, we used the initialization and number of epochs (150) used by the original PredNet authors. In this sense, we were fair to the model. For the moving MNIST dataset, the original authors didn't do an experiment using it. However, the models converged in one epoch. This can be seen in the MSE and SSIM values. When using three epochs, these values were approximately the same.

6. The rgcLSTM may be performing better, despite the use of fewer parameters, because the LSTM may be suffering from the vanishing gradient problem but this doesn't indicate that the rgcLSTM is better than the standard LSTM.
Response: This point is well taken and we don't have an answer at the present time. We've amended the conclusions to admit this possibility. We definitely have to examine this possibility in future work, include the ResNet idea suggested by the referee.

7. Summary: experimental results are good to verify the goodness of a model, but the theoretical support or intuitions are more important to understand the benefits that we can gain by making this change to the standard LSTM. This part is not clear from the paper.
Response: We've added subsection 4.1 to the paper which enhances the theory
and provides intuitions. Unfortunately, we acknowledge that we probably have not
completely address this concern.

This manuscript proposes replacing the three gates in the standard LSTM with one gate to reduce the number of parameters and the computation time. The proposed reduce-gate convolutional LSTM is applied in PredNet to predict next frames of a video. 

The main contribution of this paper is proposing an efficient convolutional LSTM. Although the number of the parameters and the training time in the experiments support this statement, the description in the paper is very confusing. 

1) In the standard LSTM, the cell state c^{t-1} is not an input for the computation of the three gates and the cell state's candidate. That is, in Eq(15) 2\kappa and 2n should be \kappa and n. Compared to Eq(14), it may not show that the standard LSTM has more parameters than rgcLSTM.

2) Eq(5) and Eq(6) are not consistent. If I_g = I_f, the coefficient before \kappa should be 2; otherwise, the input update shouldn't include c^{t-1}. 

3) The intuition of having one gate instead of three is not very clear in the paper. Mixing all the functions, i.e., input, forget, and output filters, does provide freedom for learning but also introduces the learning complexity. Decoupling a complex function into a couple of simpler functions is a common way to narrow down the searching space. This is exactly what the standard LSTM does. The authors may want to provide reasonable arguments to explain intuitively why using one gate is better. 

The model performance comparison in the experiments would be fairer if let the models converge. Perhaps, the standard LSTM is just suffering the gradient-vanishing issue and using ResNet design, for example, might improve the performance. Similarly, the rgcLSTM has fewer parameters as shown in the experiments. A possible explanation for its demonstrated better performance could be that it's less suffering the vanishing gradient. But, this doesn't indicate it's better than the standard LSTM. In summary, experimental results are good to verify the goodness of a model, but the theoretical support or intuitions are more important to understand the benefits that we can gain by making this change to the standard LSTM. This part is not clear from the paper. 

Authors present a new LSTM architecture, i.e. reduced gate convolutional LSTM. Authors use only one trainable gate, i.e. the forget gate which leads to less trainable parameters. They demonstrate the superiority of ti in two datasets, the moving MNIST and KITTI. Their results show that their architecture performs better than others in more accurately predicting next-frame. The paper is clearly written and the evaluation is based on other proposed similar architectures. For the moving MNIST they compared their model against the vanilla convLSTM with three gates and no peephole connections, based on previous work which has shown that it exceeds the accuracy performance of other LSTM based approaches. 
The results show that the training time is reduced and the standard error alike. The only limitations is that more databases could have been used to demonstrate the enhancement in performance.

Meta-review: The submission suggests reducing the parameters in a conv-lSTM by replacing the 3 gates in the standard LSTM with one gate. The idea is to get a more efficient convolutional LSTM and use it for video prediction. Two of the reviewers found the manuscript and description of the work difficult to follow and the justification for the proposed method lacking. Additionally, the contribution of this submission feels rather thin, and the experimental results are not very convincing: the absolute training time is too coarse of a measurement (and convergence may depend on many factors), and the improvements over PredNet seem somewhat marginal.

Finally, I agree with the reviewer that mentioned that a proper comparison with baselines should be done in such a way that the number of parameters is comparable (if #params is a main claim of the paper!). It is entirely plausible that if you reduce the number of parameters in PredNet by 40% (in some other way), its performance would also benefit.

With all this in mind, I do not recommend this paper be accepted at this time.

Decision: rejected-papers