Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper addresses the problem of reconstructing a video sequence that contains occlusions. The paper focusses on remove four particular types of simulated occlusion - Raindrops, Remove  pixel, Cloud and Vertically moving Bar. The authors show good results in reconstructing these video sequences in an unsupervised manner. The paper uses a GAN based network to accomplish inpainting in the occluded regions.  The authors claim that the method is very flexible in terms of the data that needs to come in, and test this by deploying the method to solve quite different missing data problems (Using different type of occlusion and in different contexts). 

One interesting note about the experiments is that the presented method is outperformed by Newson et. al[2014] in all experiments where Newson's method does not complete. I think we need more data for this to be a reasonable constraint. How long was the experiment allowed to run without completing? Do you think there is a reasonable prospect that it could finish, given more time? Other than that the method performs extremely well across all tasks.

The paper is well written, and the methods and experiments are convincing. Another very good aspect of this paper is that the accompanying website is very good and contains code for the methods and experiments used.  The work is also sufficiently novel, and has some clever tricks (such as having the discriminator consider the inter frame difference).

Minor issues
On  pg 7 - I am unfamiliar with the uses of sota, and my intuition is that is should perhaps be SOTA. If this is common usage, please ignore me :)
On pg 9 - framwork (typo)

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary
The paper presents an approach to perform video inpainting from corrupted input only . Their approach proposed a method that uses GANs for denoising images to now handle full sequences of images by building on top work that handles inpainting in single images.

Strengths
1) The authors present extensive experiments on many datasets. 
2) The presented approach is simple and general enough to be applied to many video problems.
3) The provided code is well structured and easy to read and atrached webpage showcases their results well.
4) The paper is well-written.

Weakness
1) From the code and the description in paper, it seems that a different corruption (https://github.com/anon-ustdi/ustdi/blob/7a81db4972ef9d4eabbd8fe354a8984a7771ae5d/src/datasets/corrupted.py) is applied for each step. Can the authors confirm if that is the case? 

If this is the case, I am afraid the method cannot be called unsupervised as across many steps the model would have seen different corruptions of the same video and across many such corruptions the model can learn what an uncorrupted video looks like. It is okay if that is the case but the claim of unsupervised would not hold. If that is the case, the authors need to make comparison with other supervised inpainting methods as well [1,2]

2) Is there a dataset for which these corruptions exist naturally? May be [3, 4].  It would be nice to have experiments on a dataset where the corruptions are present naturally.

3) While the experiments are incomplete for Table 1,  [4] outperforms the proposed approach for the one dataset the numbers have been reported. Performance comparison with [4] are not fair as [4]'s implementation is in MATLAB.

Decision
While the presented approach is good, further experiments are required to further validate the effectiveness of their approach in an unsupervised setting.  

References
[1] Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details.
[2] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. 
[3] https://github.com/stayhungry1/Video-Rain-Removal
[4] https://github.com/nnUyi/DerainZoo
[5] Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting of complex scenes. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a GAN based approach for unsupervised video inpainting. The idea is to learn an auto encoder to reconstruct the unoccluded regions of the frames, and reuse the autoencoder as the generator for a video GAN, which discriminates between real noisy videos and generated videos masked by a known mask distribution. 

To me the general idea is a straightforward application of GANs to the general inpainting/denoising problem, with the novelty being 1) not relying on paired or unpaired references and 2) extension to the video domain. The main critique I have is that the work seems incomplete and does not show great empirical results, which makes this work a much weaker contribution. Obviously, the authors should complete the experimental results on baseline method 2. It'd also be informative to compare with SOTA unsupervised image inpainting algorithms, if more such baselines are available than video based ones. 

Another limitation of this work is that a known mask distribution is assumed, while this might be a reasonable assumption in some cases, it can also be easily violated in real world problems. It'd be great improvement if such an assumption can be relaxed, for example, assuming a family of mask distributions but not the exact instantiation. 

Overall I think this work attempts to solve an interesting problem with an incremental but reasonable approach, and more empirical evaluation is needed to make the paper meet the bar for acceptance.
review_title: Official Blind Review #2

Meta-review: This paper studies the problem of unsupervised inpainting occluded areas in spatiotemporal sequences and propose a GAN-based framework which is able to complete the occluded areas given the stochastic model of the occlusion process. The reviewers agree that the problem is interesting, the paper is well written, and that the proposed approach is reasonable. However, after the discussion phase the critical point raised by AnonReviewer1 remains: in principle, when applying different corruptions in each step, the model is able to see the entire video over the duration of the training. This coupled with the strong assumptions on the mask distribution makes it questionable whether the approach should be considered unsupervised. Given that the results of the supervised methods significantly outperform the unsupervised ones, this issue needs to be carefully addressed to provide a clear and convincing selling point. Hence, I will recommend rejection and encourage the authors to address the remaining issues (the answers in the rebuttal are a good starting point).
Decision: Reject