Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper build a higher-order spatio-temporal model by means of combining Convolutional Tensor-Train Decomposition(CTTD) and ConvLSTM, and utilize the combination method to solve long-term video prediction problems. The CTTD factorizes a large convolutional kernel into a chain of smaller tensors, so as to relieve the difficult convergence and overfitting problems caused by too much model params. 
Experiments on Moving-MNIST and KTH datasets show that the proposed method achieved better results than standard ConvLSTM, and in some way comparable with SOTA model. Ablation Studies are also provided.

Although it seems novel combing CTTD with ConvLSTM, the idea of CTTD and the combination mainly comes from [Yu et al.,2017] and [Yang et al.,2017]，this paper use the method in a new problem of video prediction,  I think the theoretical innovation is not enough for ICLR.
Although the experimental results were better than ConvLSTM(2015), but not as good as PredRNN++(2018), especially in terms of the MSE metrics. Since the prediction accuracy has not yet achieved, I don't think the reduction of model params is a matter of primary importance.  What’s more, Moving-MNIST and KTH are relatively simple datasets, video prediction on a more complicated datasets such as UCF101 will be more convincing.
Conclusion:
This paper is in some way novel, but not enough for ICLR, and the experiment results seems not enough convincing, so I will give a weak reject.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposed a convolutional tensor-train (CTT) format based high-order and convolutional LSTM approach for long-term video prediction. This paper is well-motivated. Video data usually have high dimensional input, and the proposed method aims to explicitly take into account more than one hidden representation of previous frames - both lead to a huge number of parameters. Therefore, some sort of parameter reduction is needed. This paper considers two different types of operations - convolution and tensor-train (TT) decomposition - in an interleaved way. The basic model considered in this paper is a high-order variant of convolutional LSTM (convLSTM).
 
There exist several works using tensor decomposition methods including TT to compress a fully connected layer or a convolutional layer in neural nets, to break the memory bottleneck and accelerate computation. This paper takes a different direction - it further embeds convolution into the TT decomposition and thus defines a new type of tensor decomposition, termed convolutional tensor train (CTT) decomposition. CTT is used to represent the huge weight matrices arisen in the high-order convLSTM. To my best knowledge, this combination of convolution and TT decomposition is new.
 
The paper is well-written as the literature review is well done. Experimental results demonstrate improved performance over the convolutional LSTM baseline, a fewer number of parameters, and the qualitative results show sharp and clean digits. This improvement could be attributed to multiple causes: the high-order, the tensor decomposition-based compression, or the CTT. The authors also provide an ablation study, but it mainly concerns comparisons with ConvLSTM.   
 
Despite the promising results, this paper is not ready for ICLR yet. Below is a list of suggested points needed to address:
(1) Yang et al 2017 claim that TT-RNN without convolution can also capture spatial and temporal dependence patterns in video modeling. This is an important baseline but missing in the current version of the paper. 
(2) The justification of high-order modeling in long-term prediction. The first-order model also implicitly aggregates multiple past steps. It would be good to add more experimental evidence to support the necessity of the high-order.
(3) There exists some unjustified complexity for the CTT approach. How does it compare to TT for high-order ConvLSTM?
 
Perhaps, a more complete ablation study should include:
(1) LSTM with TT but without high-order and convolution
(2) LSTM with high-order and TT but without convolution
(3) ConvLSTM with TT
(4) ConvLSTM with CTT
(5) ConvLSTM with high-order and TT
(6) ConvLSTM with high-order and CTT
 
Question:
• How is the backpropagation done for the CTT core tensors? 
• What is the error propagation issue of first-order methods and how does the high-order one not prone to it? 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes a method that saves memory and computation in the task of video prediction by low-rank tensor representations via tensor decomposition. The method is able to outperform standard convolutional lstm and other methods by using less parameters when testing it in the Moving MNIST and KTH datasets. The authors also present a proof to validate their method.


Pros:
+ Interesting method for decomposing tensors operations in convolutional architectures
+ Outperforms immediate baseline (Convolutional LSTM)

Weaknesses / comments:
- Weak experimental section
The authors mainly compare against Convolutional LSTM. The performance increase is there but the difference in parameters is not that significant in comparison to the performance. Needing fewer parameters is one of the claims in this paper and I am not fully convinced of the trade-off between the complexity of the model and the gain in parameter reduction / performance. In addition, the show videos do not look that much improved. The paper is also missing baselines from Villegas et al., 2017 and Denton et al., 2017 which both have available models for the KTH dataset.

- No videos provided
The paper does not provide any videos which is a must for video prediction papers. Judging the video quality from images in the paper is not easy, and also the used metrics have been shown to not be very objective in terms of video prediction quality or image generation in general.


Conclusion:
The proposed decomposition method is interesting, but the experimental section fails to convince me as to whether the methods performance validates the complicated formulations. My current score is between weak reject and reject so I will give a weak reject.
review_title: Official Blind Review #1

Meta-review: This paper proposes Conv-TT-LSTM for long-term video prediction. The proposed method saves memory and computation by low-rank tensor representations via tensor decomposition and is evaluated in Moving MNIST and KTH datasets.

All reviews argue that the novelty of the paper does not meet the standard of ICLR. In the rebuttal, the authors polish the experiment design, which fails to change any reviewer’s decision.

Overall, the paper is not good enough for ICLR.
Decision: Reject