Unsupervised Spatiotemporal Data Inpainting
Authors: Yuan Yin
Arthur Pajot
Emmanuel de Bézenac
Patrick Gallinari
Venue:  (2020)
Keywords: Deep Learning
Adversarial
MAP
GAN
neural networks
video

Abstract: We tackle the problem of inpainting occluded area in spatiotemporal sequences, such as cloud occluded satellite observations, in an unsupervised manner. We place ourselves in the setting where there is neither access to paired nor unpaired training data. We consider several cases in which the underlying information of the observed sequence in certain areas is lost through an observation operator. In this case, the only available information is provided by the observation of the sequence, the nature of the measurement process and its associated statistics. We propose an unsupervised-learning framework to retrieve the most probable sequence using a generative adversarial network. We demonstrate the capacity of our model to exhibit strong reconstruction capacity on several video datasets such as satellite sequences or natural videos.


Under review as a conference paper at ICLR 2020
UNSUPERVISED SPATIOTEMPORAL DATA INPAINTING
Anonymous authors
Paper under double-blind review
ABSTRACT
We tackle the problem of inpainting occluded area in spatiotemporal sequences, such as cloud occluded satellite observations, in an unsupervised manner. We place ourselves in the setting where there is neither access to paired nor unpaired training data. We consider several cases in which the underlying information of the observed sequence in certain areas lost through an observation operator. In this case, the only available information is provided by the observation of the sequence, the nature of the measurement process and its associated statistics. We propose mnsupervised-eerniin framework to retrieve the most probable sequence using a generative adversarial network. We demonstrate the capacity our model to exhibit strong reconstruction capacity on several video datasets such satellite sequences or natural videos.
INTRODUCTION
We consider the problem of reconstructing missing information from image sequences. The problem occurs in many different settings and for different types of sequences. For example, in remote sensing applications, satellite imagery are frequently occluded by meteorological perturbations such as clouds and rains (Singh & Komodakis (018). Recovering missing satellite data is an active research topic. Approaches range from simple interpolation to sophisticated data assimilation methods. The latter is often a model-based approach that relies on analytical models of the underlying observed phenomenon (Ubelmann al. 2013 Sirjacobs et al. 201T Lguensat et al.) (017)) Model-free data based methods have also been developed such as DINEOF physical observation modeling problems of this type, there is never any direct supervision available. Another example concerns natural videos. Here, information can be occluded by moving objects such as fences (Yamashita et al. 20100) raindrops (Qian et al., 2018), persons (Kim et al., 2019), stains on photographic films (Tang et al.) 201T). Video and image imputation have given rise to large body of literature, Recent Deep Learning (DL) advances have motivated the development of general imputation methods relying on generative models such GANs (Wang et al. 201 Sa Xu et al. 2019; Kim et al., 2019). They all make use of supervision and require the availability of a ground truth, which is absent in many real- world problems. Data driven supervised methods have thus attained impressive results and are able to accurately complete a large missing region. However, reconstructing the missing information in videos when supervision is unavailable is still open problem and there have been only a few works exploring this direction. For example, NNwson et al. (2014) propose simple but effective method for occlusions in natural videos that replaces occluded parts with information from their neighborhood.
Alvera-Azcárau [201I). Note that for
We consider here unsupervised video reconstruction. We propose a model which can be used on different types of image sequences, physical or natural videos, and for a large variety of occlusion processes. Our method does not make any assumption on the nature of the image sequence, it does not require any prior knowledge like most methods used for physical images do. It is especially well suited when the occlusion complex thus forbidding the use of ad hoc techniques, e.g., the patch method of Newson et al. (2014). The method extends to sequences from ideas recently developed for still images based on generative networks (Bora et al. [0018] Pajot et al. 2019 Li et al. (2019). This up our knowledge the first attempt to solve the problem of unsupervised video completion using general ML methods. This method is fully data driven and does not use any prior on the signal. Priors the unobserved signal are directly learned from the data for solving an underlying inverse problem. The method is then applicable to a large variety of video signals.
analytical
Our main contributions are the following:

---

Under review as a conference paper at ICLR 2020
We propose framework and model for large-scale image sequence inpainting learning, in a fully unsupervised context.
This model can be used for a variety of image sequences and for different occlusion processes.
Extensive evaluations are performed on realistic simulated satellite data and on natural videos with different occlusion processes.
METHOD
2.1 PROBLEM SETTING
We suppose that there exists an unknown spatiotemporal sequence Px. E ROXTXIXXW where æ is a tensor denoting C-channel sequence composed of frames of x W pixels. We denote the 1-th frame of the sequence and i the subsequence from the 41-th to the 12-th frame inclusive. With this notation, to corrupted observation sequences of this signal y ~ Py- E 2CCxTx>XXX Our objective is to reconstruct from the corresponding observation y- For example, (SST) at successive times while image sequence y is SST measurements via IR satellites occluded by moving clouds. We will suppose that y obtained from via a measurement process modeled through a stochastic operator F' as follows:
zT. We do not have access to the original signal but only
be surface temperature
y = F(x, m) = O m c.
where PM is occlusion mask, generated from known distribution with the same size and with components in {0, 1}. where 0 holds for masked pixel. m denotes the complement of m, is the element-wise multiplication, all the masked pixels are supposed be reset constant which could be 0 or depending on the observation process (see Section B Random variables X and M are assumed be independent and F is assumed differentiable w.r.t. æ. the following, we will suppose that one can retrieve the mask m directly from the observation y. This is not very restrictive since most situations this easy to do. denote T the mask extractor T(y) = m.
Our objective is then to recover the sequence æ from the observations y and the corresponding binary masks m. Adopting a probabilistic viewpoint, we want select a reconstruction which the most plausible under the posterior distribution Px|y (-1y).
2.2 MODEL.
We formulate the problem finding the most probable sequence conditioned observations:
xx = arg max log Px y (olly) = arg max log Px (æ) + log P
The prior term log Px (x) is unknown since we are in an unsupervised setting, while the likelihood log P (y/=)) does not lead analytical simple computational solution.
To tackle these issues, let us introduce a mapping G: Y + X, parameterized by neural network & and associating mcasurement to its estimate æ. will allow us to approximate the underlying distribution of training sequences. By plugging G(yy into Equation the objective becomes:
G* arg max Eyyoy [log Px (G(y)) + Eyrry [log Py x (3) G(y))
G
likelbbood
2.3 PRIOR HANDLING
Let us first handle the prior term in Equation 3 We want the distribution induced from G(y) to be close to Px In order to do so, we will use adversarial approach. We will build on the ideas introduced in Bora et al. 2018); Pajot et al. (0099 for still images. The process is illustrated in Figure I For given observation y, want to generate approximation of the unknown true sequence t = G The prior Px being unknown, the only available information source is the observation y and the noise prior PM- For given generated signal á we compute corrupted version of á through
2

---

Under review as a conference paper at ICLR 2020
real frame
fake frame
PM
G
y ~ Py real sequence
0 D
fake sequence
Figure 1: Schema of our model. Generator G takes a sequence y and outputs an inpainted sequence i; measurement process F takes the inpainted sequence then outputs fake observations ý.
the known mask m. ý F@t, m) with mn ~ PM- We will train G to make the distributions of y and ý indistinguishable. In order to succeed, the generator G' will have remove the corruption from y and recover sample from distribution Px- Generator G' will then act as an inpainter conditioned on y. This will enforce the distribution of the reconstructed sequences & to be close the distribution of true ones a and maximize the prior term.
A direct application of the adversarial training idea suggests using discriminator operating directly on the sequences. We found out that using an additional discriminator on frames worked better than using unique operating sequences. We then associated with whole sequences and with individual frames to optimize G. D. separates sequences y and ÿ. Dy distinguishes real frames y: from fake ones 9t. The loss function used for training G, D., and D, is:
discriminators D, and Dy respectively
1 Σ log (ogg
min L(G) max E Ey 8mpp [log Dr.D,
(4)
with & (y) E ma
(w m)), corresponding the distribution of the corrupted sequences ÿ generated via the measurement operator F. px (x) is the distribution of & induced by G from y. i.e. & Gly).
2.4 LIKELIHOOD HANDLING
Let us now handle the likelihood term in Equation
Byrpy llo P
(5)
This likelihood is maximised when we are able to perfectly reconstruct y from G(y). One way to ensure this property is to constrain G to directly use y for the non occluded area of the reconstructed image G(y). This can be easily achieved through the following mapping:
G( (
m
m
where is a NN responsible for reconstructing the missing part of y. m = T(y) is the mask retrieved from y. G maps Y X with the help of mask m to ensure that the network will only generate values for occluded pixel, while keeping all the information from y. To summarize, optimizing the prior term amounts training y for inputting the missing pixels while optimizing the likelihood term simply achieved by copying the non occluded portion of y.
2.5 TRAINING
G' is optimized by descending the prior loss and D,, Dr by ascending it. Sequence discriminator D, focuses on temporal dependence and coherence of pixel changes. Frame discriminator D, keeps an eye spatial feature coherence of observation frames.

---

Under review as a conference paper at ICLR 2020
3 EXPERIMENTS
We evaluate model on four datasets, characteristic of different types of image sequences. The first one, SST, a realistic simulation of satellite observations. The other three are natural video datasets: FaceForensiest+, KTH, and BAIR, initially respectively used benchmarks for forgery detection, motion detection, and video prediction.
3.1 DATASETS
The Sea Surface Temperature dataset used for the experiments includes 2 subsets of GLOBAL Sea Physical Analysis and Forecasting Product] from E.U. Copernicus Marine Service Information. This is monitor system providing simulated but realistic global ocean SST data, which integrates satellite-derived and in situ data by assimilation. Our dataset is a part of the hourly mean SST, the finest timescale we have access to. The data we use is a part of the archive of analysis integrating real-world data. We retrieved our training-and and-validation set and test set respectively from two different marine regions. Detailed data description and information for accessing the dataset are provided in Appendix
SST
FaceForensies++ (Rössler et al., 2019) This dataset contains 1000 videos of non-occluded face movements on a static background. was initially created for forgery detection. In our case, we or thus keeping
extracted the faces from the original unforged videos with only the changing component of the videos. The faces have been cropped and resized to 64x64.
KTH (Schuldt et al. 2004) A human action dataset containing 2391 video clips of 6 human actions. The videos have been recorded with 25 subjects in different environments. All frames have been resized to 64 64.
BAIR Robot Pushing Dataset (Ebert al. 2017) This dataset contains 44374 videos recorded by an one-armed robot. It pushes objects and changes movement direction in a stochastic manner. All videos share similar tabletop with static background. All frames have been resized 64 x6.
3.2 MEASUREMENT PROCESSES
The above datasets provide ground truth videos without corruption. In order to generate corrupted observation sequences, we simulate different types of occlusion depending on the nature of the videos. Each corruption process is defined as a stochastic operator F as in Equation [ with mask distribution PM- For a given video one then generates a sequence of random masks, one mask being then associated to each frame of the sequence. Note that except for the Remove- PPxel corruption process where two successive corruptions are independent, for all processes, the generated corruption sequences are deeendepeddnn: the corruption pattern at time t will depend on the one at time t 1.
Cloud This process is specific for the SST dataset. It simulates realistically video cloud masks satellite images. Cloud masks are simulated using Liquid Water Path (LWP) data (measured in g/m²), which characterizes the total amount of liquid water present in the atmosphere between two points. The LWP data are generated by PyCLES Pressel et al. 2015 3 a large eddy simulation system. It simulates the evolution of clouds in time based on a variant of anclastic equations of atmospheric motion. Collected LWP data record mask videos of clouds. The binary masks are then obtained by setting the image pixels to 0 when their LWP value is above a threshold. This produces realistic cloud coverage of the captured regions, see Figure 2a Pixels occluded by the mask are set to 1. Thresholds are selected in the interval 55 to g/m" to simulate clouds at different occlusion rates. Statistics about the occluded area at different thresholds are presented in Table 2al For simulating occlusion, for each SST image sequence, we sample randomly a sequence of masks from the LWP dataset to be applied to the SST sequence.
htrps:/gitttbb.com///eeee//////p
4

---

Under review as a conference paper at ICLR 2020
Raindrops This process is a simplified model of random raindrops between subject and camera. taking into account a blurring effect when raindrops leave traces during exposure. It generates a set of white bars, each with random length 0 and constant width Bars move down at random speed O., starting from a random initial position 0z. All these values are normalized w.r.f the frame edge length in j0. [. The number of raindrops is pre-defined. Bars return to the top once completely out of frame, see Figure 2 Pixels occluded by the mask are reset to = 1. Note that as for Cloud, this is a dime-dependent measurement process, meaning that two successive masks are correlated.
Remove-Pixel This measurement roughly mimics severe damages on vintage films. It masks randomly a fixed proportion p E ]o, 1[ of pixels at each time step and reset them to c 0. see Figure 2 Mask for each frame is generated independently regardless the evolution of time. This is the only time-independent measurement considered here.
Moving- Vertical--aa This simple measurement operator generates a vertical white bar crossing the sequence, very roughly mimicking a fence or any similar obstacle. The bar is generated with the following distribution parameters: width 8w. initial position 8z. horizontal constant velocity 0,. These values are in J0, as for Raindrops. The moving direction is chosen randomly. The bar reappears on the opposite side once it reaches the border. Masked pixels in observation are reset to This is a time-dependent measurement.
3.3 BASELINES
Unsupervised Approaches We use two unsupervised baselines, one adapted for SST and the other one specific of natural videos. The former is DINEOF (Azáart-Azeáratc 2011). This is a state-of- the-art data-driven completion method in geophysics, and it has been used for SST observations, chlorophyll, salinity etc. It is a parameter-fire interpolation technique based on empirical orthogonal function (EOF). It adopts an iterative algorithm that calculates each iteration truncated decompo- sition of EOF from known pixels, then replaces the values marked as missing by a reconstruction from calculated EOF, DINEOF does not make any assumption on the form of missing area and as such could be used for other domains as well and for different types of complex occlusion processes. However, DINEOF has been developed for remote sensing and does not ensure the coherence between different input channels (c.g. for RGB images).
The other one is Newson et al. (014), one of the very few methods for unsupervised natural video inpainting. It is representative of patch-based approaches and it is still today state-of-the-art for many natural video occlusion processes. searches for the nearest neighbors of occluded area using an Approximate Nearest neighbor (ANN) search with PatchMatch (Barnes et al.) [2009) finding for each occluded pixel the best corresponding non-occluded patch. The occluded area is then reconstructed by assembling information from these neighbors at multiple scales. The form of the researched patches is supposed to be rectangular cuboids, c.g. a 5 x x 5 spatiotemporal tensor, which limits its capability to adapt to more complex cases like Cloud, Raindrops, Remove-Pixel.
Supervised Approaches As already mentioned, there exists several supervised approaches to sequence inpainting (Huang et al. 2016; Xu et al. 2019 Kim al. 2019). In order to evaluate the performance of our unsupervised method w.r.t supervised ones, we compared with two supervised baselines. As our goal is not to beat state-of-the-an supervised techniques, used two supervised adaptation of our model, respectively trained using unpaired and paired supervision. They are described below.
UNPAIRED VARIANT This is a supervised variant of our model in which we have access to unpaired samples from Px and py- The model is illustrated in Appendix C Because we have access to clean data, it is then possible to supervise the approximation & G by discriminating directly between samples x from the signal distribution and the output of the reconstruction network á.
PAIRED VARIANT Here we have access to corrupted -nnpeeed-uucrruptedd pairs (y, œ) from the joint distribution PY,x Given the masked image y. the reconstruction obtained by regressing y to the associated complete image x using a L' loss. In order to avoid blurry samples, we add an adversarial term in the objective, which helps G to produce realistic samples. This model is similar to the Vid2Vid (Wang et al. (0188)) model, except that they rely on optical flow which not available in case because of the masked regions. The model is illustrated in appendix C
5

---

Under review as a conference paper at ICLR 2020
(a) SST
(b) FaceForensics**
(c) KTH
(d) BAIR
Figure 2: Samples from test sets. SST data (a) are masked with Cloud, natural video datasets (b,c,d) are masked with Secmvve-Pixel and Raindrops. Sequences are accelerated times to make movements more visible. Each sample from top row to bottom: observed y, and recovered t.
3.4 NETWORK ARCHITECTURE AND TRAINING DETAILS
We use the same networks for all the experiments. For generator G, we use a ResNet-type self- attention network (Zhang et al. 2019), which is composed of 3D-ResNet blocks and spatial self- attention layers. Frame discriminator Dy is 2D convolutional NN trained for binary classification. Sequence discriminator D, uses the same structure as D but with 3D convolutions. These networks can process sequences of any time length. See Appendix B for more details about the networks.
Let us now detail the training procedure for each dataset: (a) For SST data, the model trained on 300 sequences, validated on 66 sequences, and tested on 60 sequences. Each sequence composed of frames. We use SST data degraded by cloud masks at LWP threshold 70 g/m² for training since they include sufficient information for both SST and cloud dynamics. and BAIR, we pick randomly 5% data for validation, another 5% data for test, and keep the remaining for training. Sequences truncated or padded 35 frames be able to fit into GPU memory.
(b) For FaceForensicst*. KTH,
For time-independent process Remove-Pixel, we use plain pixel value as feature and directly let the sequence discriminator capture the dynamics. For time-dependent Raindrops, Moving-Vertical- Vertical-Bar and Cloud, we further reinforce the sequence discriminator D, to focus on temporal component by extracting inter-frame difference feature since the underlying dynamics reflected by this feature is more expressive than plain pixels. D, will therefore distinguish between & yy2 y1..... UN
yN-1] and Ø y 91. JN -N-1).
We use hinge loss for Equation as in Zhang et al. 2019). Following standard practice, all three networks are trained using Adam optimizer with learning rate of 10-4 and (01, 82) = (0, 0.999). All networks are initialized with normal distribution with a gain of 0.02. We apply spectral normalization for all parametric layers. The experiments were made on NVIDIA GeForce GTX TITAN X G
'Code and video samples unsup-ve veo-- inaaiiiiittn
available
at:
httpa://sites. com/i///

---

Under review as a conference paper at ICLR 2020
Raindrops
Remove-Pixel
Moving-Vortieal-Be
Dataset
Method
FTD
FVD
MAE
FID
FVD
MAR
FID
FVD
MAF
43.72
1574,89
0834
0187
93.28
1460.02
0894 0137
493.57
.1304
FF++
(1)
75.93
3424,11
12084.0272
110.15
3091.67
.0552 0161
56.58
5775.25
.3286-20888
9.04
316.58
.494::.0501
Ours
56.56
2522.81
008801.0002
56.16
2639.24
.049
39.05
588.94
00711 0505
KTH
(1)
71.69
6400,44
0072
82.45
6660,02
0400=,0000
34.90
3408.19
.0559 0402
354.01
.0268.
Ours
27.33
1194.19
0153
$3,80
2073.90
0997=00087
11.55
496.38
.1619: 00590
BAIR
89.87
4456.08
.2345...0244
140.20
4014.17
1424 0103
67.06
7361.77
.5579 1.066
10.31
340.97
.18821.0877
Table I: Results for FaceForensics, KTH, and BAIR. Compared with (1) Alvera-Azcárate 201 and (2) Newson et al. (2014). *Could not terminate.
3.5 EVALUATION METRICS
Our objective is to find the most plausible sequence. We use as main performance measures of the generated frames, Fréchet Inception Distance (FID, HiDDell et al. (017) and Fréchet Video Distance (FVD. D.VD.uuntrrhiner al.] (0188). Both compare the activation distribution of the generated samples from PX to the real one sampled from Px- These distributions are extracted from activation layers of NNs, which are pre-trained respectively on natural image classification tasks for FID and video classification tasks for FVD. The two distances are calculated for the whole sequence including occluded and non-occluded region. Besides FID and FVD, we also evaluate the reconstruction error as complimentary metric. We use for that Mean Average Error (MAE), which indicates the absolute deviation from the real data. MAE is calculated solely within the occluded area.
4 RESULTS
4.1 COMPARISON WITH BASELINES
Results for SST Data Table 2a shows the results for SST data with simulated clouds different oc- clusion rates. For most occlusion rates, the generated sequences have an MAE under 0.1 C which is well below the reference baseline (see Table[a).. They also have good FID and FVD values, which means that they are spatially and temporally realistic (See Fig- ure 2a for examples). For heavily occluded area, our model can realistically reconstruct the data around the border, the reconstruction near the center of the cloud is of lower quality. We compare our results in Table 2b at 70% occlusion, with DINEOF, the state-of-the-art agnostic method for image sequence reconstruction in IR images. The error reduction w.r.t. DINEOF is about 40% for MAE. We have not been able to obtain results for Newson et al.] (2014) be- cause the highly complex Cloud masks make search- ing for valid cuboid patches in both occluded and non-occluded area extremely hard. Consequently, the
LWP
Occluded Area (9)
(p/m²)
FID
FVD
MAE ("C)
79.9 9.6
.127311.043
60
69.
22.95
79.13
1047-.0396
65
55.9+15.1
17.75
75.07
09884.0378
70
14.5
8.01
40.76
.03999.0324
75
24.5+11.5
5,58
30.07
0098
7.8
1.77
.4977.0237
All
47.1
14.76
61.55
.0874400347
Results with clouds generated at different LWP thresholds.
Method
FID
FVD
MAF
Ours
8.01
40.76
.739.1.0324
Alvera-Azcärate 2011
27.99
323.61
.121411.0288
Newson (0014)
(b) Comparison of results with clouds LWP threshold g/m". *Could not terminate.
Table Results for SST dataset. algorithm remains blocked in its initialization step and is prevented from the completion. Note that Newson et al. (2014) is specifically designed for inpainting in natural videos so that it is not adapted for this type of data.
Results for Videos Table D gatherss the results obtained for the three natural video datasets with artificial measurements (Raindrops, Remove- Pixel, and Moving- the FID and FVD performance obtained by model are 20%-50% better than DINEOF. This means that our model better controls the both spatial and the temporal generation quality than DINEOF. Globally, we achieve better MAE scores notably for color videos with few exceptions (performance
Moving-Vertical-Bart Vertical-Bar). For all measurements,
7

---

Under review as a conference paper at ICLR 2020
is close for (emove-Pixel) As for Newson et al. (2014), the completion could not be finished for Raindrops and Remove-Pixe masks for the same reason mentioned above caused by highly complex masks breaking cuboid patches. [Nwson et al. (2014) performs better when the form of masks is simple such as Moving- Vertical-Bar, for which matching patches could be easily found in neighbour frames. However, the computation time Newson al. (2014) is much longer than our model in terms of best performance could be achieved. Note that reduced computation time was an argument put forward in their publication. For a 30-frame x4 video, [Nesson al. (2014) costs on average I minute and GPU speedup is not available, versus around 0.5 seconds by our model, with an end-to-end GPU speedup.
Comparison with Supervised Baselines Table compares our model with the two supervised (un-
Method
FID
FVD
MAE
paired and paired) variants described in Section 3.3
Ours, Unsupervised
43.72
1574.89
0834-1.015
Unpaired, Supervised
20.86
$75.08
.0547=.0105
Unsurprisingly, the performance of supervised mod- els is far better than the ones of our unsupervised model. We can find out that the access to the ground truth reduce dramatically all three metrics. By using supervision, FID is halved and FVD is between two and three times smaller. The error reduction is smaller with MAE. We also notice that the unpaired version performs better than the paired one in terms of sequence completion quality (FVD) as the I loss introduces a strong constraint for the reconstruction. This shows to what extent the absence of ground truth will affect generation quality and the extra difficulty while dealing with partial observations.
Paired, Supervised
22.17
720.75
.555:+.0108
Table 3:
Comparison with supervised base- lines for FaceForensicst* with Raindrops.
4.2 ABLATION STUDY
We also conduct additional experiments in order to quantify the importance the temporal component.
Method
FID
FVD
MAE ("C)
Ours
8.01
40.76
.0739:1.0324
In a first series of experiments, quence component from our model, i.c. removing the sequence discriminator D, and replacing 3D genera- tor by 2D one generating individually frames. Table 4 shows that model clearly improves temporal qual- ity by reducing FVD by a ratio of 7 compared to the model without the temporal component (denoted Static variant in the table). Note that FID is also clearly improved by factor of 4. this gives more evidence that the model is able to exploit temporal dependency for its image completion task. We provide samples for this part in Appendix [ in Figure 12 We further compared our model in Appendix E with Criminisi et al. (2004) and Ulynnov jtt al. 2017). two other state-of-the the-art unsupervised image inpainting methods. The results show the same improvement on temporal quality as mentioned above.
remove the se-
Recurrent variant
13.29
67.37
.096011.4311
Static variant
35,91
279.78
.103611.0047
Table 4: Comparison of results for SST data for ablation study.
Our model generates frame at time t, ã, from a whole sequence of observations y. In second series of experiments, we conditioned the generation of frames é; only on past observations. We feed past observations into a convolutional RNN (we used GRU our experiments) and generate the reconstructed frame, still denoted G(y) by abuse of notation, from the last hidden state of the RNN, which encodes all past observations. The spatial discriminator operates as before, while the sequence discriminator operates
past observations only, instead of the full sequence of observations in model. See Appendix [rrendix[[forr an illustration and for further description. Results in Table Recurrent variant, show that using only past observations makes the completion less realistic and less accurate, but it still clearly outperforms the model without time dependency.
RELATED WORK
There is currently, up to our knowledge, no other learning- approach trying to solve the problem of spatiotemporal data completion in a purely unsupervised manner. We will review below related contributions for image and video reconstruction, data assimilation, and domain translation.
Image Reconstruction Video or more generally spatiotemporal sequence completion can be con- sidered as extension of image completion problems. the first attempts for image completion and
8

---

Under review as a conference paper at ICLR 2020
inpainting were all supervised. Xie et al. (2012) uses convolutional NNs for regressing observations to ground truth images. This typically produces blurry outputs. To overcome this issue, some authors introduce textures (Yang et al., 2016), while many others make use of GANs (Pathak et al. [2016] Yu al. 2018). More recently, unsupervised approaches have been developed by considering only corrupted images. [Iyanov et al. (017); [ehtinen et al. (2018) show that it is possible to learn the underlying data distribution and to reconstruct images from observations when model of observation process is given or when the noise is zero-mean. Such restrictive hypothesis have been removed in the seminal work of Bora et al. (2018) 8). They introduce AmbientGAN to unconditionally generate images without supervision from corrupted observations under the assumption that the stochastic measurement process is known. MisGAN (Li et al., 2019) extend this idea and to learn jointly the mask and the original data distributions. Both contributions objective is data generation and not com- pletion like we do here. Pajot et al. (2019) propose to conditionally recover images from corrupted observations only by solving a maximum posteriori (MAP) estimation problem, implemented with an adversarial framework. This is limited to still images.
Video Inpainting Video inpainting has been mainly considered a supervised framework. Object- based (Cheung et al.] (006) and patch-based (Newson et al. (014) approaches introduced before the deep learning era generally rely on prior segmentation of moving objects and background or strong assumptions on video content. Flow-based methods have been used to model spatial appearance and local pixel movement between consecutive frames. [uang et al. (2016) propose to guide non- parametric patch- based optimization with forward and backward optical flow. Xu et al. 2019) Kim cet al. (2019) try to resolve the problem through neural optical flow estimation, which requires extra pre-trained network. More recently end-to-end learning approaches have been proposed. For example, Wang et al. (2018a) propose frame-level generation decomposition by combining a video inpainter with frame-wise refinement inpainter. Extensions of image inpainting methods are also proposed in Chang et al. (2019). All these learning-based methods are trained with supervision and have been developed for natural videos.
Data Assimilation for Remote Sensing For remote sensing applications, Optimal Interpolation (OI) is widely used in operational products (Donlon et al. [2012). It produces linear estimate for the occluded area. Model-based assimilation methods (Ubelmann et al. 2015) rely on explicit physical dynamic priors and demand significant computational power. Purely data-driven methods based on empirical orthogonal functions (EOF, Beckers & Rixen 2003) use basically matrix factorization to achieve temporal interpolation. Recent advances in Analog Data Assimilation (AnDA, Lguensat et al. 2017 Fablet et al. 2018) combine analog forecasting methods with data-driven assimilation using implicit knowledge of dynamical prior. These methods rely either on interpolation or exploit some priors on the nature of the underlying process. Recently, learning methods have started to be exploited in this field. Shibata et al. (017)) propose to apply learning-based frame-level inpainting enhanced with optical flow using simple assumptions et al. (2018), they recover the missing data using an adversarial approach to supervise extra occluded area w.r.t the original partial observations. This approach still reconstruct data frame by frame.
pixel movement. In a later paper, (Shibata
Domain Translation Reconstruction can also be considered as a translation problem between two domains, incomplete observations and full unobserved data. For images, Pix2Pix (Isola et al.) 2016) utilizes GANs to project data from domain A to domain with paired data. CycleGAN (Zhu al. 2017) propose to two cenerator-discriiian two domains. For videos, Wang et al. (2018b) propose Vid2Vid by adding a multi-scale temporal discriminator in Pix2Pix to supervise the optical flow. RecycleGAN (Bansal al. 2018) is based on the idea of CycleGAN by adding a temporal transformation in both domains. However, these methods require full data from the two domains, and sometimes the supervision on motion, when no supervision is available in our setting.
pairs to model the transformation between
6 CONCLUSION
We have proposed a GAN-based framework to complete partially observed spatiotemporal data, Our model utilizes a generator to complete missing pixels in observation sequences with the help of two discriminators classifying real and generated observation sequences. We show that our model is able to complete spatiotemporal data without ground truth supervision when have stochastic

---

Under review as a conference paper at ICLR 2020
model of the occlusion process. Our results for SST data and natural videos show that the recovered sequences are realistic, especially when the occluded area is highly complex.
REFERENCES
Barth A. Sirjacobs D. Lenartz Beckers J. Alvera-Azcárate, A. Data interpolating empirical orthogonal functions (dineof): tool for geophysical data analyses. Mediterranean Marine Science, 12(3), 2011. doi: 10. 10.12681/mm 1/mm5666.
Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle- Unsupervised video retargeting. CoRR, 01211888.55777 2018. htlthhtt://arttii org.iiii.o.roooooo wrtp:ss//gaaaaaaaooeooooosss abs /108 /
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (Proc. SIGGRAPHI, 28(3), August 2009.
J. M. Beckers and M. Rixen. EOF calculations and data filling from incomplete oceanographic datasets. Journal of Atmospheric and Oceanic Technology, 20(12):1839-165 2003. doi: 10. 1520-0426 (2003)
C.C,0;
Ashish Bora, Eric Price, and Alexandros G. Dimakis. AmbientGAN: Generative models from
lossy measurements. In International Conference on Learning Representations, 2018. URL
https:://pennev/ ew net (erpuummid=yyyyoooo
Ya- Va-Liang Chang, Zhe Yu Liu, Kuan- Ying Lee, and Winston Hsu. Free-form video inpainting with
3D gated convolution and temporal PatchGAN, CoRR, abs/1904.10247, 2019. URL http:
wwrxii.org/aaba.... /104. 10247
S. Cheung, J. Zhao, and M. V. Venkatesh. Efficient object-based video inpainting. In 2006 International Conference on Image Processing, pp. 705-708, Oct 2006. doi: 10.1109//C1P.206. 312432.
A. Criminisi, Pérez, and K. Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing, 1200:1200- Sep. 2004. doi: 10. N0.1109/TIP2204.33333
Craig Donlon, Matthew Martin, John Stark, Jonah Roberts-Jones, Emma Fiedler, and Werenfrid Wimmer. The operational sea surface temperature and sea ice analysis (OSTIA) system. Remote Sensing of Environment, 116:140 158, 2012. ISSN 0034-4257. doi: https://doi.org/10.1001 rse. 1sc.2010.10.017 URL http:////// www. scis sciencedirect com/ http://eeeneeaiieeeeeeeee/// 5003442571100119 Advanced Along Track Scanning Radiometer (AATSR) Special Issue.
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-Supervised Visual Planning with Temporal Skip Connections. arXiv e-prints, art. arXiv:1710.05260 Oct 2017.
Ronan Fablet, Phi Huynh Viet, Redouane Lguensat, Pierre-Henri Horrein, and Bertrand Chapron. Spatio-temporal interpolation of cloudy SST fields using conditional analog data assimilation.
Remote Sensing, 10(2), 2018. ISSN 2072-4292, doi: 10.3390/ss002033 URL https
7/ , mdpi. com/ wttp://022222222222222 0000-4222/20///335
Madec Gurvan, Romain Bourdallé- Badie, Pierre-Antoine Bouttier, Clément Bricaud, Diego Brucia- ferri, Daley Calvert, Jérôme Chanut, Emanuela Clementi, Andrew Coward, Damiano Delrosso, Christian Ethé Simona Flavoni, Tim Graham, James Harle, Doroteaciro Iovino, Dan Lea, Claire Lévy, Tomas Lovato, Nicolas Martin, Sébastien Masson, Silvia Mocavero, Julien Paul, Clément Rousset, Dave Storkey, Andrea Storto, and Martin Vancoppenolle. NEMO ocean engine, October 2017. UtL doi htw./oiioooorrr/ org/10. 0tt.:/10..281/eeee0 htt/////eoo..omm 3248739 still revision 8625 from SVN repository.
Fix broken referneee,,,ssss
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - EECCV 2016, pp. 630-645, Cham, 2016. Springer International Publishing. ISBN 978-3-319- 46493-0.
10

---

Under review as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. GANs trained by a two sime- update rule converge to nash equilibrium. CoRR, abs/1706.08500, 2017. /tLLattp:/aaxx...rrrt http:::/aaaxvvvv.o
1706 08500
Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf. Temporally coherent comple- tion of dynamic video. ACM Trans. Graph., 111:::119661111911111oo 2016. ISSN 0730-
0301. doi: (1L1144/29801792298288888 URL http: 2982398
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. CoRR, tess/1611.07000 2016. URL http:://axxi..oort
/bss1611... 07004
Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting, CoRR, abs/1905.01639, 2019. URL http:h/apaa/iiii org.:.iiiroororoooooo... abs 1900/ 01639
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. CoRR, abs/1 05///803.04111 189, 2018. Uttp://::/aaxx
/bss 1803. 0018800.1111111 C4189
Redouane Lguensat, Pierre Tandeo, Pierre Ailliot, Manuel PULIDO, and Ronan Fablet. The Analog Data Assimilation. Monthly Weather Review, 145(10)4403 4107, October 2017. doi: 10.1175/
MWR-D-16-0441.. URL htt.hhttt/s//cm /taas://hal..e acch ves ouvertes. ot fr/hal//aaa 016011141
Steven Cheng-Xiare Xian Li, Bo Jiang, and Benjamin M. Marlin. MisGAN: Learning from incomplete data with generative adversarial networks. CoRR, abs/1902.09599, 2019. URL http:
org/abs 9122. 09599
Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting of complex scenes. SIAM J. Imaging Sciences, 7:1993-2019, 2014.
Arthur Pajot, Emmanuel de Bezenac, and Patrick Gallinari. Unsupervised adversarial image re- construction. In International Conference on Learning Representations, 2019. URL https
//openreview. nww.eeeeeeeeeeeeeeeeeeeeeee wew.:e/oooommmddd
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. CoRR, abs/1604.07379, 2016. URL http:://axiy
org/ / 07379
Kyle G. Pressel, Colleen M. Kaul, Tapio Schneider, Zhihong Tan, and Siddhartha Mishra. Large-eddy simulation in anelastic framework with closed water and entropy balances. Journal of Advances in Modeling Earth Systems, 7(3): 1235:1425-14 2015. doi: 10. (002000 URL https //gguubss..
in l
ey www. doi abs / TTT.TT000000020000000000000
Rui Qian, Robby T. Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from single image. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thics, and Matthias Nießner. Faceforensics+#: Learning to detect manipulated facial images. CoRR, abs/190 1901 008971, 2019.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: A local svm approach. In Proceedings of the Pattern Recognition, 17th International Conference on (ICPR '04) Volume 3- Volume 03, ICPR '04, pp. 32-36, Washington, DC, USA, 2004. IEEE Computer Society. ISBN 0-7695-2128-2. doi: TCPR. 2004. 747
2077.747. URLhttp.
dx doi wtw..oooorr...00////00000001
S. Shibata, M. liyama, A. Hashimoto, and M. Minoh. Restoration of sea surface temperature images by learning- based and optical-flow-baue inpainting. In 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 198-198, July 2017. doi: 10.1 10.1109//CM...007.0000 9019401.
11

---

Under review as a conference paper at ICLR 2020
S. Shibata, M. liyama, A. Hashimoto, and M. Minoh. Restoration of sea surface temperature satellite images using a partially occluded training set. 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2771-2776 Aug 2018.
Mennatullah Siam, Sepehr Valipour, Martin Jagersand, and Nilanjan Ray. Convolutional Gated Recurrent Networks for Video Segmentation. arXiv e-prints, art. arXiv: wtt.:6111.054335 Nov 2016.
P. Singh and N. Komodakis. Cloud-GAN: Cloud removal for Sentinel-2 imagery using a cyclic consistent generative adversarial networks. In IGARSS 2018 2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 1772-1775, July 2018. doi: 10.1109/1GARSS.211....
Damien Sirjacobs, Aida Alvera-Azcárate, Alexander Barth, Geneviève Lacroix, YoungJe Park, Bouchra Nechad, Kevin Ruddick, and Jean-Marie Beckers. Cloud filling of ocean colour and sea surface temperature remote sensing products over the southern north sea by the data inter- polating empirical orthogonal functions methodology. Journal of Sea Research, 65(1): 14 130, 2011. ISSN 1385-1101.
doi: https://tb.orr///00100ooo..sssee 201000000.....00...........00 002. URL http:
/ 7/www. www.sisieeeeddreecc..... wtt.:/eecccccccieeecceeeeeeeeeee ott.a//cc........o.oo.....o.aaaa
N. C. Tang, C. Hsu, C. Su, K. Shih, and H. M. Liao. Video inpainting on digitized vintage films via maintaining spatiotemporal continuity. IEEE Transactions on Multimedia, 13(4):602-614, Aug
2011. ISSN 1520-9210. doi: 10. 109 1.211264..
Clement Ubelmann, Patrice Klein, and Lee-Lueng Fu. Dynamic interpolation of sea surface height and potential applications for future high-resolution altimetry mapping. Journal of Atmospheric and Oceanic Technology, 32(1):177-184, 2015. doi: 10. (0.177//TTECH---14000011111 URL
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Deep image prior. CoRR, abs/1711.1099 2017. UBL/ittpsr//arriv.orr//////////1
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. CoRR, abs/1812.01717, 2018. URL Uttp::t: org abs /atp://111..01111
Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details. CoRR, fbs//8006084822. 2018a. URL httt:://arxe org/abs/ 1806.00000 08482
Ting-Chun Wang. Ming- Yu Liu, Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. CoRR, has/1808.066011 2018b. URL http:://arxiv org/abs//eee /ww.b////.... 06601
X. Wang, R. Girshick, A. Gupta, and K. H. Non-local neural networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7794-7803, June 2018c. doi: 10.1109/CVPR.2016 1.000....
Junyuan Xie, Linli Xu, and Enhong Chen. Image denoising and inpainting with deep neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems 25, pp. 341- 349. Curran Associates, Inc., 2012. URL
4686 image www.ddinnnnnn-eeeeeeeee wand-inpainting with
pdf
Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
A. Yamashita, A. Matsui, and T. Kaneko. Fence removal from multi-focur images. In 2010 20th International Conference on Pattern Recognition, pp. 4532-4535, Aug 2010. doi: 10. 10.1109/ICPR. 2010.1101.
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi scale neural patch synthesis. CoRR, abs/161 1,09669, 2016. URL http: 7/ rxiv. htt.v/oorgaab///eeeeeoo 6611.99969
12

---

Under review as a conference paper at ICLR 2020
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Generative image inpainting with contextual attention. CoRR, abs//180.07892, 2018. URL /RL/ttp://axxiv.org abs/ 07892
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363 Long Beach, Califormia, USA, 09-15 15 Jun 2019. PMLR. URL
http://proceecingg mww. wwwss/ v97 wtteees/99//hhagg9..... www.glaa.nnaa...
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. CoRR, abs/1703.1059) 2017. URL http://
arxiv.org/abs/17000 10593
A SUPPLEMENTARY INFORMATION ON SST DATASET
GLOBAL Sea Physical Analysis and Forecasting Product is a monitor system providing simulated but realistic global ocean SST data. In analysis engine. integrates satellite-derived and in situ data by assimilation from OSTLA SST system (OSTIA, The Operational Sea Surface Temperature and Sea Ice Analysis, Donlon et al. 2012). The analysis is based on a sophisticated ocean model, NEMO ocean engine (Nucleus for European Modeling of the Ocean, Gurvan et al., 2017), state-of-the-an art modelling framework of ocean related physics. Oceanic output variables of this product are hourly, daily and monthly means of temperature, salinity, currents, sea surface height, etc.
Our dataset is a subset of global ollaalaaaalllll ssh, which includes hourly mean Sea Temperature sea level height, i.c. SST. The data we use is part of the archive of analysis integrating real-world data. Training-and-valicee
set is hourly SST in 2018 (2018-01-4 00:30 to 2018-12-31 23:30) on marine region of 64 64 pixels in North Atlantic Ocean). Test set corresponds to the data of the first 60 days of 2019 (2019-01-01 00:30 to 2019-03-01 23:30) in another region of the same size (14.759-20"5, 14.75% W0, in South Atlantic Ocean). To obtain the dataset, check out the product via http://marine www.enicce.nccooooeo.. www.:eeieeeeeeeeeeeeeeeeesesseee www.ccessppooooooooooooooooooooo hacefooooocccceeesssososs to-ttttoprrre prooduuuuss// ?option=com
(20--25.20 N, 34.75%.40%V
id=GLOBAL ANALYSIS FORECAST PHY 001 024 then enter the critcria above for each subset.
B ARCHITECTURE DETAILS
We detail the architecture networks utilized in our experiments in this section.
Generator Table 5a lists modules for generator G. It is a ResNet-type self-attention network as in Zhang et al.] (2019). It composed of 3D ResNet blocks as in He et al. (2016) and spatial self-altention layer from Wang al. (2018c), which means the attention calculation is limited within each frame.
Discriminators Table the architecture of 2D or 3D PatchGAN discriminator as in Isola et al. (2016). Frame discriminator D, is 2D convolutional NN. Sequence discriminator D, uses the same structure as Dy but with 3D convolutions except that the stride still for temporal dimension, such that the number of frames is not limited.
C DETAILS OF VARIANTS OF OUR MODEL
We describe two supervised variants as supervised baselines in Section 4.1 and recurrent variant of our model Section
Unpaired Variant, Supervised (Figure This variant gives model access the distribution Px Instead of distinguishing between true observations y and fake ones ÿ. Dy and D, will discriminate between original sequences and the output of the generator á
13

---

Under review as a conference paper at ICLR 2020
Nb, Inpot
Nb, Output Channel
Module
Activation
Channel
Encoder
Nb. Input
Nb. Output
Spatial
ResNet block
Cine
ReLU
Module
Channel
Channel
Activation
2 3D ResNet block
Cass
16Chese 16Chce
ReLU
2DV3D Conv.
Can
Cuas
2
LeakyReLU
3
3D ResNet block
16Chm
ReLU
2D/3D Conv.
Chow
2Cho
LenkyRet.1
Decoder
2D/3D Conv:
2Chac
4Chsse
LeakyReLLU
2D/3D Conv:
4Chee
LeakyReLU
ResNet block
16Cas
8Chua
ReLU
2D/3D Cony.
8Chee
8Chee
LeakyReLU
block
8Chese
4Chase
ReLU
2D/3D Conv,
8Chms
8Chow
2
LenkyReL.U
ResNet block
1Chse
2Chxe
ReLU
2D/3D Conv,
8Chac
8Chss
LeakyReLU"
7 Spotial Sell-Attention
2Cose
2Chsa.
ReLU
2D/3D
8Cher
8 3D ResNet block
20
Chose
ReLU"
2Cese
2Chse
(b) PatchGAN Diseriminator. Kernel size Stride of temporal dimension is always convolution for frame discriminator D,, 3D for sequence one Da- NNegative slope 0.2.
10 3D Conv.
Chas
Cin
lanh
(a) Generator structure. Kernel size 3, stride 1.
* Activation inside the module
Table 5: Architecture of networks.
This variant gives our model access not only to the dis- tribution Px but also the joint distribution Px, y by adding sequence-to-seqqeeeee L' reconstruction loss. We use the duo-discriminator setting as our model to prevent generating blurry frames, which is essentially the same techniques in Isola et al. (0166) and Wang al. (2018))
Paired Variant, Supervised (Figure 3b)
Recurrent Variant, Unsupervised (Figure In this variant, we simply insert a convolutional recurrent network cell of any type, ConvGRU in our case (Convolutional Gated Recurrent Unit. [Siam et al. 2016), into PPjoot et al. (2019). See Appendix for the illustration and further description. This cell models temporal dependency feature maps after encoding the sequence into the space with maximum number of channels. Instead of discriminating image by image in Pajot et al. (2019), we use this variant the same duo-diseriminator setting as our model. D, distinguishes frame by frame while D, discriminates true sequence clip y and fake clip ύî 1+11 where L is the maximum number consecutive frames.
D ADDITIONAL SAMPLES
We provide more longer samples from with samples from basclines in this section, see figures 5
7 and 8 We also show extra samples for unsupervised baselines (Figure I0.. supervised baselines
(Figure 11. ablation study (Figure
E ADDITIONAL COMPARISON WITH STATE-OF-THE-ART UNSUPERVISED IMAGE INPAINTING APPROACHES
We provide here the additional comparison of results with state-of-the-art unsupervised inpainting methods, Criminisi et al. (2044) and Ulyanov et al. (017)). for FaceForensics** dataset with Raindrops noise. Unsurprisingly. Table 6 shows that model outperforms these methods, especially in terms of temporal quality indicated by FVD. We show also some samples in Figure [3
Method
FID
FVD
MAE
Criminisi al. (2004)
147.86
3617.92
.533+.1246
Ulyanov et al. (2017)
44.84
2410.62
2271+.1560
Ours
43.72
1574.89
.08341.0187
Table 6: Additional comparison with state-of-the-art unsupervised image inpainting methods for FaceForensics*+ with Raindrops.
14

---

Under review as a conference paper at ICLR 2020
Dj
Dj
resl frame.
fake frame
real frame
fake frame
1 Py
~ by
real sequerie
fake sequence
fake sequence
D,
loss
sequence D2
(a) Unpaired variant of our model, supervised
(b) Paired variant of model, supervised
Figure 3: Supervised variants, used as baseline in Section 3.3
RNN Gell
h
(decoding)
m
(encoding) C
F
1//s
Vi
De L+1
real frame
fake frame
real
fake
sequence clip
sequence clip
D.
Figure 4: Recurrent variant.
15

---

Under review as a conference paper at ICLR 2020
Figure 5: Samples from our model for SST. From top to bottom: Cloud at LWP threshold 55, 60, 65,
70, 75, and 80 g/m².
Figure 6: Samples from our model for FaceForensicstt. From top to bottom: Raindrops, Remove- Pixel, and Moving- Vertical-Bar.
16

---

Under review as a conference paper at ICLR 2020
J
Figure 7: Samples from our model for KTH. From top to bottom: Raindrops, Remove-Pixel, and Moving-Vertical- Vertical-Bar
Figure 8: Samples from our model for BAIR. From top to bottom: Raindrops, Remove- and Moving- Bering-Vertical-Bar.
17

---

Under review as a conference paper at ICLR 2020
Figure 9; Samples from DINEOF Alvera-Azcárate 2011 for SST, FaceForensicstt, KTH and BAIR.
18

---

Under review as a conference paper at ICLR 2020
Figure 10: Samples from Newson et al. 2014) for FaceForensicst+, KTH, and BAIR. Only with
Moving- Vering-ertticl-Bar Samples for other measurements cannot be calculated in reasonable time.
(a) Unpaired variant
(b) Paired variant
Figure 11: Samples from supervised variants.
(a) Pajot et al.
2019 (left) versus ours (right)
(b) Pajot et al. (2019). Abrupt inter-frame changes degrade tem- poral quality.
(c) Recurrent variant. Frames on the right are better than those on the left.
(d) Ours.
Figure 12: Samples for ablation study.
19

---

Under review as a conference paper at ICLR 2020
Ours
Criminisi et al. (0004)
Ulyanov et al. (017))
Ours
Criminisi et al. (2004)
Ulyanov al. (0117)
Ours
Criminisi et al. 2004
Ulyanov et al. (017)
Figure 13: Comparison of samples from model and two state-of-the-art unsupervised image inpainting methods. Note that the temporal coherence broken for Ulyanov et al, (017) due to the instant appearance and disappearance of facial parts.
20


Reviews:
[{"review_id": "rkl9Dco75S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of reconstructing a video sequence that contains occlusions. The paper focusses on remove four particular types of simulated occlusion - Raindrops, Remove  pixel, Cloud and Vertically moving Bar. The authors show good results in reconstructing these video sequences in an unsupervised manner. The paper uses a GAN based network to accomplish inpainting in the occluded regions.  The authors claim that the method is very flexible in terms of the data that needs to come in, and test this by deploying the method to solve quite different missing data problems (Using different type of occlusion and in different contexts). \n\nOne interesting note about the experiments is that the presented method is outperformed by Newson et. al[2014] in all experiments where Newson's method does not complete. I think we need more data for this to be a reasonable constraint. How long was the experiment allowed to run without completing? Do you think there is a reasonable prospect that it could finish, given more time? Other than that the method performs extremely well across all tasks.\n\nThe paper is well written, and the methods and experiments are convincing. Another very good aspect of this paper is that the accompanying website is very good and contains code for the methods and experiments used.  The work is also sufficiently novel, and has some clever tricks (such as having the discriminator consider the inter frame difference).\n\nMinor issues\nOn  pg 7 - I am unfamiliar with the uses of sota, and my intuition is that is should perhaps be SOTA. If this is common usage, please ignore me :)\nOn pg 9 - framwork (typo)\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJeZL4PtYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper presents an approach to perform video inpainting from corrupted input only . Their approach proposed a method that uses GANs for denoising images to now handle full sequences of images by building on top work that handles inpainting in single images.\n\nStrengths\n1) The authors present extensive experiments on many datasets. \n2) The presented approach is simple and general enough to be applied to many video problems.\n3) The provided code is well structured and easy to read and atrached webpage showcases their results well.\n4) The paper is well-written.\n\nWeakness\n1) From the code and the description in paper, it seems that a different corruption (https://github.com/anon-ustdi/ustdi/blob/7a81db4972ef9d4eabbd8fe354a8984a7771ae5d/src/datasets/corrupted.py) is applied for each step. Can the authors confirm if that is the case? \n\nIf this is the case, I am afraid the method cannot be called unsupervised as across many steps the model would have seen different corruptions of the same video and across many such corruptions the model can learn what an uncorrupted video looks like. It is okay if that is the case but the claim of unsupervised would not hold. If that is the case, the authors need to make comparison with other supervised inpainting methods as well [1,2]\n\n2) Is there a dataset for which these corruptions exist naturally? May be [3, 4].  It would be nice to have experiments on a dataset where the corruptions are present naturally.\n\n3) While the experiments are incomplete for Table 1,  [4] outperforms the proposed approach for the one dataset the numbers have been reported. Performance comparison with [4] are not fair as [4]'s implementation is in MATLAB.\n\nDecision\nWhile the presented approach is good, further experiments are required to further validate the effectiveness of their approach in an unsupervised setting.  \n\nReferences\n[1] Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details.\n[2] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. \n[3] https://github.com/stayhungry1/Video-Rain-Removal\n[4] https://github.com/nnUyi/DerainZoo\n[5] Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting of complex scenes. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJlXPY_SFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a GAN based approach for unsupervised video inpainting. The idea is to learn an auto encoder to reconstruct the unoccluded regions of the frames, and reuse the autoencoder as the generator for a video GAN, which discriminates between real noisy videos and generated videos masked by a known mask distribution. \n\nTo me the general idea is a straightforward application of GANs to the general inpainting/denoising problem, with the novelty being 1) not relying on paired or unpaired references and 2) extension to the video domain. The main critique I have is that the work seems incomplete and does not show great empirical results, which makes this work a much weaker contribution. Obviously, the authors should complete the experimental results on baseline method 2. It'd also be informative to compare with SOTA unsupervised image inpainting algorithms, if more such baselines are available than video based ones. \n\nAnother limitation of this work is that a known mask distribution is assumed, while this might be a reasonable assumption in some cases, it can also be easily violated in real world problems. It'd be great improvement if such an assumption can be relaxed, for example, assuming a family of mask distributions but not the exact instantiation. \n\nOverall I think this work attempts to solve an interesting problem with an incremental but reasonable approach, and more empirical evaluation is needed to make the paper meet the bar for acceptance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of unsupervised inpainting occluded areas in spatiotemporal sequences and propose a GAN-based framework which is able to complete the occluded areas given the stochastic model of the occlusion process. The reviewers agree that the problem is interesting, the paper is well written, and that the proposed approach is reasonable. However, after the discussion phase the critical point raised by AnonReviewer1 remains: in principle, when applying different corruptions in each step, the model is able to see the entire video over the duration of the training. This coupled with the strong assumptions on the mask distribution makes it questionable whether the approach should be considered unsupervised. Given that the results of the supervised methods significantly outperform the unsupervised ones, this issue needs to be carefully addressed to provide a clear and convincing selling point. Hence, I will recommend rejection and encourage the authors to address the remaining issues (the answers in the rebuttal are a good starting point).
Decision: Reject