Title: UNSUPERVISED SPATIOTEMPORAL DATA INPAINTING

Abstract: We tackle the problem of inpainting occluded area in spatiotemporal sequences, such as cloud occluded satellite observations, in an unsupervised manner. We place ourselves in the setting where there is neither access to paired nor unpaired training data. We consider several cases in which the underlying information of the observed sequence in certain areas is lost through an observation operator. In this case, the only available information is provided by the observation of the sequence, the nature of the measurement process and its associated statistics. We propose an unsupervised-learning framework to retrieve the most probable sequence using a generative adversarial network. We demonstrate the capacity of our model to exhibit strong reconstruction capacity on several video datasets such as satellite sequences or natural videos.

1. INTRODUCTION:

We consider the problem of reconstructing missing information from image sequences. The problem occurs in many different settings and for different types of sequences. For example, in remote sensing applications, satellite imagery are frequently occluded by meteorological perturbations such as clouds and rains (Singh & Komodakis, 2018) . Recovering missing satellite data is an active research topic. Approaches range from simple interpolation to sophisticated data assimilation methods. The latter is often a model-based approach that relies on analytical models of the underlying observed phenomenon (Ubelmann et al., 2015; Sirjacobs et al., 2011; Lguensat et al., 2017) . Model-free data based methods have also been developed such as DINEOF (Alvera-Azcárate, 2011) . Note that for physical observation modeling problems of this type, there is never any direct supervision available. Another example concerns natural videos. Here, information can be occluded by moving objects such as fences (Yamashita et al., 2010) , raindrops (Qian et al., 2018) , persons (Kim et al., 2019) , stains on photographic films (Tang et al., 2011) . Video and image imputation have given rise to a large body of literature. Recent Deep Learning (DL) advances have motivated the development of general imputation methods relying on generative models such as GANs (Wang et al., 2018a; Xu et al., 2019; Kim et al., 2019) . They all make use of supervision and require the availability of a ground truth, which is absent in many real-world problems. Data driven supervised methods have thus attained impressive results and are able to accurately complete a large missing region. However, reconstructing the missing information in videos when supervision is unavailable is still an open problem and there have been only a few works exploring this direction. For example, Newson et al. (2014) propose a simple but effective method for occlusions in natural videos that replaces occluded parts with information from their neighborhood.

We consider here unsupervised video reconstruction. We propose a model which can be used on different types of image sequences, physical or natural videos, and for a large variety of occlusion processes. Our method does not make any assumption on the nature of the image sequence, it does not require any prior knowledge like most methods used for physical images do. It is especially well suited when the occlusion is complex thus forbidding the use of ad hoc techniques, e.g., the patch method of Newson et al. (2014) . The method extends to sequences from ideas recently developed for still images based on generative networks (Bora et al., 2018; Pajot et al., 2019; Li et al., 2019) . This is up to our knowledge the first attempt to solve the problem of unsupervised video completion using general ML methods. This method is fully data driven and does not use any hand-defined analytical prior on the signal. Priors on the unobserved signal are directly learned from the data for solving an underlying inverse problem. The method is then applicable to a large variety of video signals.

Our main contributions are the following:

• We propose a new framework and model for large-scale image sequence inpainting learning, in a fully unsupervised context. • This model can be used for a variety of image sequences and for different occlusion processes. • Extensive evaluations are performed on realistic simulated satellite data and on natural videos with different occlusion processes.

2. METHOD:



2.1. PROBLEM SETTING:

We suppose that there exists an unknown spatiotemporal sequence x ∼ p X , x ∈ R C×T ×H×W , where x is a tensor denoting a C-channel sequence composed of T frames of H × W pixels. We denote x t the t-th frame of the sequence and x t2 t1 the subsequence from the t 1 -th to the t 2 -th frame inclusive. With this notation, x = x T 1 . We do not have access to the original signal x but only to corrupted observation sequences of this signal y ∼ p Y , y ∈ R C×T ×H×W . Our objective is to reconstruct x from the corresponding observation y. For example, x can be sea surface temperature (SST) at successive times while image sequence y is SST measurements via IR satellites occluded by moving clouds. We will suppose that y is obtained from x via a measurement process modeled through a stochastic operator F as follows:

y = F (x, m) = x m + c • m (1)

where m ∼ p M is an occlusion mask, generated from a known distribution with the same size as x and with components in {0, 1}, where 0 holds for a masked pixel. m denotes the complement of m, is the element-wise multiplication, all the masked pixels are supposed to be reset to a constant c which could be 0 or 1 depending on the observation process (see Section 3). Random variables X and M are assumed to be independent and F is assumed differentiable w.r.t. x. In the following, we will suppose that one can retrieve the mask m directly from the observation y. This is not very restrictive since in most situations this is easy to do. We denote T the mask extractor T (y) = m.

Our objective is then to recover the sequence x from the observations y and the corresponding binary masks m. Adopting a probabilistic viewpoint, we want to select a reconstruction x * which is the most plausible under the posterior distribution p X|Y (•|y).

2.2. MODEL:

We formulate the problem as finding the most probable sequence conditioned on observations:

EQUATION

The prior term log p X (x) is unknown since we are in an unsupervised setting, while the likelihood log p Y|X (y|x) does not lead to analytical or simple computational solution.

To tackle these issues, let us introduce a mapping G : Y → X, parameterized by a neural network ϕ and associating measurement y to its estimate x. G will allow us to approximate the underlying distribution of training sequences. By plugging G(y) into Equation 2, the objective becomes:

G * = arg max G E y∼p Y [log p X (G(y)) prior ] + E y∼p Y [log p Y|X (y|G(y))] likelihood (3)

2.3. PRIOR HANDLING:

Let us first handle the prior term in Equation 3. We want the distribution induced from G(y) to be close to p X . In order to do so, we will use an adversarial approach. We will build on the ideas introduced in Bora et al. ( 2018); Pajot et al. (2019) for still images. The process is illustrated in Figure 1 . For a given observation y, we want to generate an approximation of the unknown true sequence x ≡ G(y). The prior p X being unknown, the only available information source is the observation y and the noise prior p M . For a given generated signal x, we compute a corrupted version of x through

G G F F y y …. …. ŷ ŷ D f D f D s D s real frame fake frame real sequence fake sequence m m ⇠ p M ⇠ p M ⇠ p Y ⇠ p Y ….

x x

Figure 1 : Schema of our model. Generator G takes a sequence y and outputs an inpainted sequence x; measurement process F takes the inpainted sequence then outputs fake observations ŷ.

the known mask m, ŷ ≡ F ( x, m) with m ∼ p M . We will train G to make the distributions of y and ŷ indistinguishable. In order to succeed, the generator G will have to remove the corruption from y and recover a sample x from distribution p X . Generator G will then act as an inpainter conditioned on y. This will enforce the distribution of the reconstructed sequences x to be close to the distribution of true ones x and maximize the prior term.

A direct application of the adversarial training idea suggests using a discriminator operating directly on the sequences. We found out that using an additional discriminator on frames worked better than using a unique one operating on sequences. We then use two discriminators D s and D f respectively associated with whole sequences and with individual frames to optimize G. D s separates sequences y and ŷ. D f distinguishes real frames y t from fake ones ŷt . The loss function used for training G, D s , and D f is:

min G L(G) = max Ds,D f E y∼p Y , ŷ∼p G Y [log D s (y)+log(1-D s ( ŷ))+ 1 T T t=1 log D f (y t )+log(1-D f ( ŷt ))] (4) with p G Y (y) ≡ E m∼p M ,x∼p G X [p Y|X,M (y|x, m)],

corresponding to the distribution of the corrupted sequences ŷ generated via the measurement operator F . p G X (x) is the distribution of x induced by G from y, i.e. x = G(y).

2.4. LIKELIHOOD HANDLING:

Let us now handle the likelihood term in Equation 3:

E y∼p Y [log p Y|X (y|G(y))].

(5)

This likelihood is maximised when we are able to perfectly reconstruct y from G(y). One way to ensure this property is to constrain G to directly use y for the non occluded area of the reconstructed image G(y). This can be easily achieved through the following mapping:

G(y) ≡ ϕ(y) m + y m (6)

where ϕ is a NN responsible for reconstructing the missing part of y, m = T (y) is the mask retrieved from y. G maps Y to X with the help of mask m to ensure that the network will only generate values for occluded pixel, while keeping all the information from y. To summarize, optimizing the prior term amounts at training ϕ for inputting the missing pixels while optimizing the likelihood term is simply achieved by copying the non occluded portion of y.

2.5. TRAINING:

G is optimized by descending the prior loss and D s , D f by ascending it. Sequence discriminator D s focuses on temporal dependence and coherence of pixel changes. Frame discriminator D f keeps an eye on spatial feature coherence of observation frames.

3. EXPERIMENTS:

We evaluate our model on four datasets, characteristic of different types of image sequences. The first one, SST, is a realistic simulation of satellite observations. The other three are natural video datasets: FaceForensics++, KTH, and BAIR, initially respectively used as benchmarks for forgery detection, motion detection, and video prediction.

3.1. DATASETS:

SST The Sea Surface Temperature dataset used for the experiments includes 2 subsets of GLOBAL Sea Physical Analysis and Forecasting Product1 from E.U. Copernicus Marine Service Information. This is a monitor system providing simulated but realistic global ocean SST data, which integrates satellite-derived and in situ data by assimilation. Our dataset is a part of the hourly mean SST, the finest timescale we have access to. The data we use is a part of the archive of analysis integrating real-world data. We retrieved our training-and-validation set and test set respectively from two different marine regions. Detailed data description and information for accessing the dataset are provided in Appendix A.

FaceForensics++ (Rössler et al., 2019) This dataset contains 1000 videos of non-occluded face movements on a static background. It was initially created for forgery detection. In our case, we extracted the faces from the original unforged videos with face_recognition2 , thus keeping only the changing component of the videos. The faces have been cropped and resized to 64×64.

KTH (Schuldt et al., 2004 ) A human action dataset containing 2391 video clips of 6 human actions. The videos have been recorded with 25 subjects in different environments. All frames have been resized to 64×64.

BAIR Robot Pushing Dataset (Ebert et al., 2017) This dataset contains 44374 videos recorded by an one-armed robot. It pushes objects and changes movement direction in a stochastic manner.

All videos share similar tabletop with static background. All frames have been resized to 64×64.

3.2. MEASUREMENT PROCESSES:

The above datasets provide ground truth videos without corruption. In order to generate corrupted observation sequences, we simulate different types of occlusion depending on the nature of the videos. Each corruption process is defined as a stochastic operator F as in Equation 1with mask distribution p M . For a given video one then generates a sequence of random masks, one mask being then associated to each frame of the sequence. Note that except for the Remove-Pixel corruption process where two successive corruptions are independent, for all processes, the generated corruption sequences are time-dependent: the corruption pattern at time t will depend on the one at time t -1.

Cloud This process is specific for the SST dataset. It simulates realistically video cloud masks on satellite images. Cloud masks are simulated using Liquid Water Path (LWP) data (measured in g/m 2 ), which characterizes the total amount of liquid water present in the atmosphere between two points.

The LWP data are generated by PyCLES (Pressel et al., 2015) 3 , a large eddy simulation system. It simulates the evolution of clouds in time based on a variant of anelastic equations of atmospheric motion. Collected LWP data record mask videos of clouds. The binary masks are then obtained by setting the image pixels to 0 when their LWP value is above a threshold. This produces realistic cloud coverage of the captured regions, see Figure 2a . Pixels occluded by the mask are set to c = 1.

Thresholds are selected in the interval 55 to 80 g/m 2 to simulate clouds at different occlusion rates. Statistics about the occluded area at different thresholds are presented in Table 2a . For simulating occlusion, for each SST image sequence, we sample randomly a sequence of masks from the LWP dataset to be applied to the SST sequence.

Raindrops This process is a simplified model of random raindrops between subject and camera, taking into account a blurring effect when raindrops leave traces during exposure. It generates a set of white bars, each with a random length θ l and a constant width w. Bars move down at a random speed θ v , starting from a random initial position θ p . All these values are normalized w.r.t the frame edge length in ]0, 1[. The number of raindrops is pre-defined. Bars return to the top once completely out of frame, see Figure 2 . Pixels occluded by the mask are reset to c = 1. Note that as for Cloud, this is a time-dependent measurement process, meaning that two successive masks are correlated.

Remove-Pixel This measurement roughly mimics severe damages on vintage films. It masks randomly a fixed proportion p ∈ ]0, 1[ of pixels at each time step and reset them to c = 0, see Figure 2 . Mask for each frame is generated independently regardless the evolution of time. This is the only time-independent measurement considered here.

Moving-Vertical-Bar This simple measurement operator generates a vertical white bar crossing the sequence, very roughly mimicking a fence or any similar obstacle. The bar is generated with the following distribution parameters: width θ w , initial position θ p , horizontal constant velocity θ v . These values are in ]0, 1[ as for Raindrops. The moving direction is chosen randomly. The bar reappears on the opposite side once it reaches the border. Masked pixels in observation are reset to c = 1. This is a time-dependent measurement.

3.3. BASELINES:

Unsupervised Approaches We use two unsupervised baselines, one adapted for SST and the other one specific of natural videos. The former is DINEOF (Alvera-Azcárate, 2011). This is a state-ofthe-art data-driven completion method in geophysics, and it has been used for SST observations, chlorophyll, salinity etc. It is a parameter-free interpolation technique based on empirical orthogonal function (EOF). It adopts an iterative algorithm that calculates at each iteration a truncated decomposition of EOF from known pixels, then replaces the values marked as missing by a reconstruction from calculated EOF. DINEOF does not make any assumption on the form of missing area and as such could be used for other domains as well and for different types of complex occlusion processes. However, DINEOF has been developed for remote sensing and does not ensure the coherence between different input channels (e.g. for RGB images).

The other one is Newson et al. (2014) , one of the very few methods for unsupervised natural video inpainting. It is representative of patch-based approaches and it is still today state-of-the-art for many natural video occlusion processes. It searches for the nearest neighbors of occluded area using an Approximate Nearest neighbor (ANN) search with PatchMatch (Barnes et al., 2009) finding for each occluded pixel the best corresponding non-occluded patch. The occluded area is then reconstructed by assembling information from these neighbors at multiple scales. The form of the researched patches is supposed to be rectangular cuboids, e.g. a 5 × 5 × 5 spatiotemporal tensor, which limits its capability to adapt to more complex cases like Cloud, Raindrops, or Remove-Pixel.

Supervised Approaches As already mentioned, there exists several supervised approaches to sequence inpainting (Huang et al., 2016; Xu et al., 2019; Kim et al., 2019) . In order to evaluate the performance of our unsupervised method w.r.t supervised ones, we compared with two supervised baselines. As our goal is not to beat state-of-the-art supervised techniques, we used two supervised adaptation of our model, respectively trained using unpaired and paired supervision. They are described below.

UNPAIRED VARIANT This is a supervised variant of our model in which we have access to unpaired samples from p X and p Y . The model is illustrated in Appendix C. Because we have access to clean x data, it is then possible to supervise the approximation x = G(y) by discriminating directly between samples x from the signal distribution and the output of the reconstruction network x.

PAIRED VARIANT Here we have access to corrupted-uncorrupted pairs (y, x) from the joint distribution p Y,X . Given the masked image y, the reconstruction is obtained by regressing y to the associated complete image x using a L 1 loss. In order to avoid blurry samples, we add an adversarial term in the objective, which helps G to produce realistic samples. This model is similar to the Vid2Vid (Wang et al., 2018b) model, except that they rely on optical flow which is not available in our case because of the masked regions. The model is illustrated in appendix C. For time-independent process Remove-Pixel, we use plain pixel value as feature and directly let the sequence discriminator capture the dynamics. For time-dependent Raindrops, Moving-Vertical-Bar and Cloud, we further reinforce the sequence discriminator D s to focus on temporal component by extracting inter-frame difference feature since the underlying dynamics reflected by this feature is more expressive than plain pixels. D s will therefore distinguish between φ ≡ [y 2y 1 , . . . , y Ny N -1 ] and φ ≡ [ ŷ2 -ŷ1 , . . . , ŷN -ŷN-1 ].

We use hinge loss for Equation 4 as in Zhang et al. (2019) . Following standard practice, all three networks are trained using Adam optimizer with a learning rate of 1 × 10 -4 and (β 1 , β 2 ) = (0, 0.999). All networks are initialized with normal distribution with a gain of 0.02. We apply spectral normalization for all parametric layers. The experiments were made on one NVIDIA GeForce GTX TITAN X GPU. 

3.5. EVALUATION METRICS:

Our objective is to find the most plausible sequence. We use as main performance measures of the generated frames, Fréchet Inception Distance (FID, Heusel et al., 2017) and Fréchet Video Distance (FVD, Unterthiner et al., 2018) . Both compare the activation distribution of the generated samples from p G X to the real one sampled from p X . These distributions are extracted from activation layers of NNs, which are pre-trained respectively on natural image classification tasks for FID and video classification tasks for FVD. The two distances are calculated for the whole sequence including occluded and non-occluded region. Besides FID and FVD, we also evaluate the reconstruction error as a complimentary metric. We use for that Mean Average Error (MAE), which indicates the absolute deviation from the real data. MAE is calculated solely within the occluded area. Results for SST Data Table 2a shows the results for SST data with simulated clouds at different occlusion rates. For most occlusion rates, the generated sequences have an MAE under 0.1°C which is well below the reference baseline (see 2014) because the highly complex Cloud masks make searching for valid cuboid patches in both occluded and non-occluded area extremely hard. Consequently, the algorithm remains blocked in its initialization step and is prevented from the completion. Note that Newson et al. (2014) is specifically designed for inpainting in natural videos so that it is not adapted for this type of data.

4. RESULTS:



4.1. COMPARISON WITH BASELINES:

Results for Videos Table 1 gathers the results obtained for the three natural video datasets with artificial measurements (Raindrops, Remove-Pixel, and Moving-Vertical-Bar). For all measurements, the FID and FVD performance obtained by our model are 20%-50% better than DINEOF. This means that our model better controls the both spatial and the temporal generation quality than DINEOF. Globally, we achieve better MAE scores notably for color videos with few exceptions (performance is close for Remove-Pixel). As for Newson et al. (2014) , the completion could not be finished for Raindrops and Remove-Pixel masks for the same reason mentioned above caused by highly complex masks breaking cuboid patches. Newson et al. (2014) performs better when the form of masks is simple such as Moving-Vertical-Bar, for which matching patches could be easily found in neighbour frames. However, the computation time of Newson et al. (2014) is much longer than our model in terms of best performance could be achieved. Note that reduced computation time was an argument put forward in their publication. For a 30-frame 64×64 video, Newson et al. (2014) Comparison with Supervised Baselines Table 3 compares our model with the two supervised (unpaired and paired) variants described in Section 3.3. Unsurprisingly, the performance of supervised models is far better than the ones of our unsupervised model. We can find out that the access to the ground truth reduce dramatically all three metrics. By using supervision, FID is halved and FVD is between two and three times smaller. The error reduction is smaller with MAE. We also notice that the unpaired version performs better than the paired one in terms of sequence completion quality (FVD) as the L 1 loss introduces a strong constraint for the reconstruction. This shows to what extent the absence of ground truth will affect generation quality and the extra difficulty while dealing with partial observations. We also conduct additional experiments in order to quantify the importance of the temporal component.

4.2. ABLATION STUDY:

In a first series of experiments, we remove the sequence component from our model, i.e. removing the sequence discriminator D s and replacing 3D generator by 2D one generating individually frames. Table 4 shows that our model clearly improves temporal quality by reducing FVD by a ratio of 7 compared to the model without the temporal component (denoted Static variant in the table). Note that FID is also clearly improved by a factor of 4. this gives more evidence that the model is able to exploit temporal dependency for its image completion task. We provide samples for this part in Appendix D in Figure 12 . We further compared our model in Appendix E with Criminisi et al. (2004) and Ulyanov et al. (2017) , two other state-of-the-art unsupervised image inpainting methods. The results show the same improvement on temporal quality as mentioned above.

Our model generates a frame at time t, xt from a whole sequence of observations y. In a second series of experiments, we conditioned the generation of frames xt only on past observations. We feed past observations into a convolutional RNN (we used GRU in our experiments) and generate the reconstructed frame, still denoted G(y) by abuse of notation, from the last hidden state of the RNN, which encodes all past observations. The spatial discriminator operates as before, while the sequence discriminator operates on past observations only, instead of the full sequence of observations in our model. See Appendix C for an illustration and for further description. Results in Table 4 -Recurrent variant, show that using only past observations makes the completion less realistic and less accurate, but it still clearly outperforms the model without time dependency.

5. RELATED WORK:

There is currently, up to our knowledge, no other learning-based approach trying to solve the problem of spatiotemporal data completion in a purely unsupervised manner. We will review below related contributions for image and video reconstruction, data assimilation, and domain translation.

Image Reconstruction Video or more generally spatiotemporal sequence completion can be considered as an extension of image completion problems. the first attempts for image completion and inpainting were all supervised. Xie et al. (2012) uses convolutional NNs for regressing observations to ground truth images. This typically produces blurry outputs. To overcome this issue, some authors introduce textures (Yang et al., 2016) , while many others make use of GANs (Pathak et al., 2016; Yu et al., 2018) . More recently, unsupervised approaches have been developed by considering only corrupted images. Ulyanov et al. (2017) ; Lehtinen et al. (2018) show that it is possible to learn the underlying data distribution and to reconstruct images from observations when a model of observation process is given or when the noise is zero-mean. Such restrictive hypothesis have been removed in the seminal work of Bora et al. (2018) . They introduce AmbientGAN to unconditionally generate images without supervision from corrupted observations under the assumption that the stochastic measurement process is known. MisGAN (Li et al., 2019) extend this idea and to learn jointly the mask and the original data distributions. Both contributions objective is data generation and not completion like we do here. Pajot et al. (2019) propose to conditionally recover images from corrupted observations only by solving a maximum a posteriori (MAP) estimation problem, implemented with an adversarial framework. This is limited to still images.

Video Inpainting Video inpainting has been mainly considered in a supervised framework. Objectbased (Cheung et al., 2006) and patch-based (Newson et al., 2014) approaches introduced before the deep learning era generally rely on prior segmentation of moving objects and background or strong assumptions on video content. Flow-based methods have been used to model spatial appearance and local pixel movement between consecutive frames. Huang et al. (2016) propose to guide nonparametric patch-based optimization with forward and backward optical flow. Xu et al. (2019) ; Kim et al. (2019) try to resolve the problem through neural optical flow estimation, which requires extra pre-trained network. More recently end-to-end learning approaches have been proposed. For example, Wang et al. (2018a) propose frame-level generation decomposition by combining a video inpainter with a frame-wise refinement inpainter. Extensions of image inpainting methods are also proposed in Chang et al. (2019) . All these learning-based methods are trained with supervision and have been developed for natural videos.

Data Assimilation for Remote Sensing For remote sensing applications, Optimal Interpolation (OI) is widely used in operational products (Donlon et al., 2012) . It produces a linear estimate for the occluded area. Model-based assimilation methods (Ubelmann et al., 2015) rely on explicit physical dynamic priors and demand significant computational power. Purely data-driven methods based on empirical orthogonal functions (EOF, Beckers & Rixen, 2003) use basically matrix factorization to achieve temporal interpolation. Recent advances in Analog Data Assimilation (AnDA, Lguensat et al., 2017; Fablet et al., 2018) combine analog forecasting methods with data-driven assimilation using implicit knowledge of dynamical prior. These methods rely either on interpolation or exploit some priors on the nature of the underlying process. Recently, learning methods have started to be exploited in this field. Shibata et al. (2017) propose to apply learning-based frame-level inpainting enhanced with optical flow using simple assumptions on pixel movement. In a later paper, Shibata et al. (2018) , they recover the missing data using an adversarial approach to supervise extra occluded area w.r.t the original partial observations. This approach still reconstruct data frame by frame.

Domain Translation:

Reconstruction can also be considered as a translation problem between two domains, incomplete observations and full unobserved data. For images, Pix2Pix (Isola et al., 2016) utilizes GANs to project data from domain A to domain B with paired data. CycleGAN (Zhu et al., 2017) propose to use two generator-discriminator pairs to model the transformation between two domains. For videos, Wang et al. (2018b) propose Vid2Vid by adding a multi-scale temporal discriminator in Pix2Pix to supervise the optical flow. RecycleGAN (Bansal et al., 2018) is based on the idea of CycleGAN by adding a temporal transformation in both domains. However, these methods require full data from the two domains, and sometimes the supervision on motion, when no supervision is available in our setting.

6. CONCLUSION:

We have proposed a GAN-based framework to complete partially observed spatiotemporal data. Our model utilizes a generator to complete missing pixels in observation sequences with the help of two discriminators classifying real and generated observation sequences. We show that our model is able to complete spatiotemporal data without ground truth supervision when we have a stochastic model of the occlusion process. Our results for SST data and natural videos show that the recovered sequences are realistic, especially when the occluded area is highly complex. Paired Variant, Supervised (Figure 3b ) This variant gives our model access not only to the distribution p X but also the joint distribution p X,Y by adding a sequence-to-sequence L 1 reconstruction loss. We use the duo-discriminator setting as our model to prevent generating blurry frames, which is essentially the same techniques in Isola et al. (2016) and Wang et al. (2018b) .

Recurrent Variant, Unsupervised (Figure 4 ) In this variant, we simply insert a convolutional recurrent network cell of any type, ConvGRU in our case (Convolutional Gated Recurrent Unit, Siam et al., 2016) , into Pajot et al. (2019) . See Appendix C for the illustration and further description. This cell models temporal dependency in feature maps after encoding the sequence into the space with maximum number of channels. Instead of discriminating image by image as in Pajot et al. (2019) , we use in this variant the same duo-discriminator setting as our model. D f distinguishes frame by frame while D s discriminates true sequence clip y t t-L+1 and fake clip ŷt t-L+1 , where L is the maximum number of consecutive frames.

D ADDITIONAL SAMPLES:

We provide more longer samples from with samples from baselines in this section, see figures 5, 6, 7, and 8. We also show extra samples for unsupervised baselines (Figure 10 ), supervised baselines (Figure 11 ), ablation study (Figure 12 ).

E ADDITIONAL COMPARISON WITH STATE-OF-THE-ART UNSUPERVISED IMAGE INPAINTING APPROACHES:

We provide here the additional comparison of results with two state-of-the-art unsupervised inpainting methods, Criminisi et al. (2004) and Ulyanov et al. (2017) , for FaceForensics++ dataset with Raindrops noise. Unsurprisingly, Table 6 shows that our model outperforms these methods, especially in terms of temporal quality indicated by FVD. We show also some samples in Figure 13 .

Method FID FVD MAE Criminisi et al. (2004) 

:

http://marine.copernicus.eu/services-portfolio/access-to-products/

https://github.com/ageitgey/face_recognition

https://github.com/pressel/pycles

Figures and Tables:

Figure 2: Samples from test sets. SST data (a) are masked with Cloud, natural video datasets (b,c,d) are masked with Remove-Pixel and Raindrops. Sequences are accelerated 3 times to make movements more visible. Each sample from top row to bottom: observed y, and recovered x.3.4 NETWORK ARCHITECTURE AND TRAINING DETAILSWe use the same networks for all the experiments. For generator G, we use a ResNet-type selfattention network(Zhang et al., 2019), which is composed of 3D-ResNet blocks and spatial selfattention layers. Frame discriminator D f is a 2D convolutional NN trained for binary classification. Sequence discriminator D s uses the same structure as D f but with 3D convolutions. These networks can process sequences of any time length. See Appendix B for more details about the networks.Let us now detail the training procedure for each dataset: (a) For SST data, the model is trained on 300 sequences, validated on 66 sequences, and tested on 60 sequences. Each sequence is composed of 24 frames. We use SST data degraded by cloud masks at LWP threshold 70 g/m 2 for training since they include sufficient information for both SST and cloud dynamics. (b) For FaceForensics++, KTH, and BAIR, we pick randomly 5% data for validation, another 5% data for test, and keep the remaining for training. Sequences are truncated or padded to 35 frames to be able to fit into GPU memory.

Figure 3: Supervised variants, used as baseline in Section 3.3

Figure 5: Samples from our model for SST. From top to bottom: Cloud at LWP threshold 55, 60, 65, 70, 75, and 80 g/m 2 .

Figure 6: Samples from our model for FaceForensics++. From top to bottom: Raindrops, Remove-Pixel, and Moving-Vertical-Bar.

Figure 7: Samples from our model for KTH. From top to bottom: Raindrops, Remove-Pixel, and Moving-Vertical-Bar.

Figure 8: Samples from our model for BAIR. From top to bottom: Raindrops, Remove-Pixel, and Moving-Vertical-Bar.

Figure 9: Samples from DINEOF (Alvera-Azcárate, 2011) for SST, FaceForensics++, KTH and BAIR.

Figure 10: Samples from Newson et al. (2014) for FaceForensics++, KTH, and BAIR. Only with Moving-Vertical-Bar. Samples for other measurements cannot be calculated in reasonable time.

Figure 11: Samples from supervised variants.

(a) Pajot et al. (2019) (left) versus ours (right) (b) Pajot et al. (2019). Abrupt inter-frame changes degrade temporal quality. (c) Recurrent variant. Frames on the right are better than those on the left. (d) Ours.

Figure 12: Samples for ablation study.

Results for FaceForensics, KTH, and BAIR. Compared with (1) Alvera-Azcárate (2011) and (2)Newson et al. (2014). *Could not terminate.

Table content: <table><tr><td/><td>4</td><td/><td/><td/><td/><td/></tr><tr><td>4 Code</td><td>and</td><td>video</td><td>samples</td><td>available</td><td>at:</td><td>https://sites.google.com/view/</td></tr><tr><td colspan="4">unsup-video-inpaiting</td><td/><td/><td/></tr></table>

Results for SST dataset.

Table content: <table/>

Table 2a). They also have good FID and FVD values, which means that they are spatially and temporally realistic (See Figure2afor examples). For heavily occluded area, our model can realistically reconstruct the data around the border, the reconstruction near the center of the cloud is of lower quality. We compare our results in Table2bat 70% occlusion, with DINEOF, the state-of-the-art agnostic method for image sequence reconstruction in IR images. The error reduction w.r.t. DINEOF is about 40% for MAE. We have not been able to obtain results forNewson et al. (

Table content: <table/>

Comparison with supervised baselines for FaceForensics++ with Raindrops.

Table content: <table><tr><td>costs on average</td></tr></table>

Comparison of results for SST data for ablation study.

Table content: <table><tr><td>Method</td><td>FID</td><td>FVD</td><td>MAE (°C)</td></tr><tr><td>Ours</td><td>8.01</td><td colspan="2">40.76 .0739±.0324</td></tr><tr><td colspan="2">Recurrent variant 13.29</td><td colspan="2">67.37 .0960±.0431</td></tr><tr><td>Static variant</td><td colspan="3">35.91 279.78 .1036±.0047</td></tr></table>

Architecture of networks.

Table content: <table/>

Additional comparison with state-of-the-art unsupervised image inpainting methods for FaceForensics++ with Raindrops.

Table content: <table><tr><td/><td>147.86 3617.92 .5533±.1246</td></tr><tr><td>Ulyanov et al. (2017)</td><td>44.84 2410.62 .2271±.1560</td></tr><tr><td>Ours</td><td>43.72 1574.89 .0834±.0187</td></tr></table>

Peer review (iclr2020):
The paper addresses the problem of reconstructing a video sequence that contains occlusions. The paper focusses on remove four particular types of simulated occlusion - Raindrops, Remove pixel, Cloud and Vertically moving Bar. The authors show good results in reconstructing these video sequences in an unsupervised manner. The paper uses a GAN based network to accomplish inpainting in the occluded regions. The authors claim that the method is very flexible in terms of the data that needs to come in, and test this by deploying the method to solve quite different missing data problems (Using different type of occlusion and in different contexts). One interesting note about the experiments is that the presented method is outperformed by Newson et. al[2014] in all experiments where Newson*s method does not complete. I think we need more data for this to be a reasonable constraint. How long was the experiment allowed to run without completing? Do you think there is a reasonable prospect that it could finish, given more time? Other than that the method performs extremely well across all tasks. The paper is well written, and the methods and experiments are convincing. Another very good aspect of this paper is that the accompanying website is very good and contains code for the methods and experiments used. The work is also sufficiently novel, and has some clever tricks (such as having the discriminator consider the inter frame difference). Minor issues On pg 7 - I am unfamiliar with the uses of sota, and my intuition is that is should perhaps be SOTA. If this is common usage, please ignore me :) On pg 9 - framwork (typo)