Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed CrevNet, a conditionally reversible network, that performs video prediction efficiently and effectively.
The proposed CrevNet used the bijective two-way autoencoder to make sure the frames are inherently invertible from the latent space.

While the idea of the paper is good, I am not convinced of some points raised in the paper. And I hope to get response from the authors.
(1) The latent space of the proposed method is not very small, it should be at least the same dimension as the input given invertibility, so it is w*h*c. What are the dimensions of resolution-preserving methods?
(2) According to previous studies, although i-RevNet preserves all information in the latent features, it is not make robust mappings: a small perturbation on the features will ruin the inverted image. Is it still the case in CrevNet? Can we some how visualize what is happening/changing in the latent space? 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a new method of self-supervised feature learning from videos based on learning future frame prediction. The idea is similar as BERT like NLP tasks, but for videos, the computational cost and memory cost could be very large. To solve this problem efficiently, the authors adopt several existing techniques such as pixel shuffle layer, 3D-CNN,  ConvRNN and Attention module to efficiently and effectively capture video information. Experiments on several datasets are conducted to show the effectiveness of the proposed method.

The idea of self-supervised feature learning from videos are not novel. The key is how to learn good features that can generalize very well. The authors show that the learned features in this paper can be used on other tasks, such as object detection. And state-of-the-art results on KITTI dataset could be achieved based on the learned features with fixed backbone parameters. Although the provided results may not be state-of-the-art (for car, it seems that KITTI best results are 97%+ for easy, 95%+ for medium, 90%+ for hard, instead of 92%, 92%, and 85% provided in the paper), the generalization ability on object detection looks very interesting.

Is the generalization ability of the proposed method better that existing methods such as CycleGAN, PredNet and ContextVP? More experiments about the quality of the learned features of the proposed method are recommended to improve the importance of this paper. The advantage of the proposed method looks weak without these comparisons.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper introduces Conditionally Reversible Network (CrevNet) that consists of the invertible autoencoder and a reversible predictive module (RPM). The two-way autoencoder is an invertible network that preserves the volume with no information loss while reducing memory consumption by using bijective downsampling. The RPM is a recurrent extension of two-way autoencoder that provides the reversiblity in temporal domain. The experiments on Moving MNIST, Traffic4cast, KITTI, and 2D object detection on KITTI show the improvement compare to other state-of-the-art models. 

The paper is well-written and the contributions are clear. The experiments on diverse tasks and datasets are provided. Especially, the use of pretrained generative model for the object detection task shows the benefit of the model.   

However, it is not clear the necessity of the proposed model and how the memory demand and computation cost are reduced. 

To have a better support for the contributions, I have following suggestions. 

- The benefit or necessity of each module is not clear. The comparisons with 1) only the invertible two-way autoencoder (without Section 2.2 and 2.3)", 2) only RPM (without Section 2.1 and/or 2.3), and 3)two-way autoencoder + reversible predictive model without 3D convolutions (without Section 2.3) are required. 

- Model size/memory comparison is provided only for the MNIST dataset. Since the main benefit of the proposed model is low memory demand and computation cost, I suggest to provide the the comparisons of the memory demand with other models as well.  
1) Other models on on Caltech Pedestrian dataset, Especially, the performance gap on Caltech Pedestrian dataset between the proposed method and CycleGAN is very small. 
2) Some of the compared models on 2D KITTI detection are also lightweighted e.g., SqeeqeDet. The comparison of the model size/memery/test time with these models are necessary. 

---- 
<After rebuttal>
Authors have mostly addressed my concerns, and the contributions are clearer. I adjusted my rating accordingly. 
review_title: Official Blind Review #2

Meta-review: This paper introduces a new approach that consists of the invertible autoencoder and a reversible predictive module (RPM) for video future-frame prediction.

Reviewers agree that the paper is well-written and the contributions are clear. It achieves new state-of-the-art results on a diverse set of video prediction datasets and with techniques that enable more efficient computation and memory footprint. Also, the video representation learned in a self-supervised way by the approach can have good generalization ability on downstream tasks such as object detection. The concerns of the paper were relatively minor, and were successfully addressed in the rebuttal.

AC feels that this work makes a solid contribution with well-designed model and strong empirical performance, which will attain wide interests in the area of video future-frame prediction and self-supervised video representation learning.

Hence, I recommend accepting this paper.
Decision: Accept (Poster)