Unsupervised Spatiotemporal Data Inpainting

1 INTRODUCTION . We consider the problem of reconstructing missing information from image sequences . The problem occurs in many different settings and for different types of sequences . For example , in remote sensing applications , satellite imagery are frequently occluded by meteorological perturbations such as clouds and rains ( Singh & Komodakis , 2018 ) . Recovering missing satellite data is an active research topic . Approaches range from simple interpolation to sophisticated data assimilation methods . The latter is often a model-based approach that relies on analytical models of the underlying observed phenomenon ( Ubelmann et al. , 2015 ; Sirjacobs et al. , 2011 ; Lguensat et al. , 2017 ) . Model-free data based methods have also been developed such as DINEOF ( Alvera-Azcárate , 2011 ) . Note that for physical observation modeling problems of this type , there is never any direct supervision available . Another example concerns natural videos . Here , information can be occluded by moving objects such as fences ( Yamashita et al. , 2010 ) , raindrops ( Qian et al. , 2018 ) , persons ( Kim et al. , 2019 ) , stains on photographic films ( Tang et al. , 2011 ) . Video and image imputation have given rise to a large body of literature . Recent Deep Learning ( DL ) advances have motivated the development of general imputation methods relying on generative models such as GANs ( Wang et al. , 2018a ; Xu et al. , 2019 ; Kim et al. , 2019 ) . They all make use of supervision and require the availability of a ground truth , which is absent in many real-world problems . Data driven supervised methods have thus attained impressive results and are able to accurately complete a large missing region . However , reconstructing the missing information in videos when supervision is unavailable is still an open problem and there have been only a few works exploring this direction . For example , Newson et al . ( 2014 ) propose a simple but effective method for occlusions in natural videos that replaces occluded parts with information from their neighborhood . We consider here unsupervised video reconstruction . We propose a model which can be used on different types of image sequences , physical or natural videos , and for a large variety of occlusion processes . Our method does not make any assumption on the nature of the image sequence , it does not require any prior knowledge like most methods used for physical images do . It is especially well suited when the occlusion is complex thus forbidding the use of ad hoc techniques , e.g. , the patch method of Newson et al . ( 2014 ) . The method extends to sequences from ideas recently developed for still images based on generative networks ( Bora et al. , 2018 ; Pajot et al. , 2019 ; Li et al. , 2019 ) . This is up to our knowledge the first attempt to solve the problem of unsupervised video completion using general ML methods . This method is fully data driven and does not use any hand-defined analytical prior on the signal . Priors on the unobserved signal are directly learned from the data for solving an underlying inverse problem . The method is then applicable to a large variety of video signals . Our main contributions are the following : • We propose a new framework and model for large-scale image sequence inpainting learning , in a fully unsupervised context . • This model can be used for a variety of image sequences and for different occlusion processes . • Extensive evaluations are performed on realistic simulated satellite data and on natural videos with different occlusion processes . 2 METHOD . 2.1 PROBLEM SETTING . We suppose that there exists an unknown spatiotemporal sequence x ∼ pX , x ∈ RC×T×H×W , where x is a tensor denoting a C-channel sequence composed of T frames of H ×W pixels . We denote xt the t-th frame of the sequence and xt2t1 the subsequence from the t1-th to the t2-th frame inclusive . With this notation , x = xT1 . We do not have access to the original signal x but only to corrupted observation sequences of this signal y ∼ pY , y ∈ RC×T×H×W . Our objective is to reconstruct x from the corresponding observation y . For example , x can be sea surface temperature ( SST ) at successive times while image sequence y is SST measurements via IR satellites occluded by moving clouds . We will suppose that y is obtained from x via a measurement process modeled through a stochastic operator F as follows : y = F ( x , m ) = x m+ c · m̄ ( 1 ) wherem ∼ pM is an occlusion mask , generated from a known distribution with the same size as x and with components in { 0 , 1 } , where 0 holds for a masked pixel . m̄ denotes the complement of m , is the element-wise multiplication , all the masked pixels are supposed to be reset to a constant c which could be 0 or 1 depending on the observation process ( see Section 3 ) . Random variables X and M are assumed to be independent and F is assumed differentiable w.r.t . x . In the following , we will suppose that one can retrieve the maskm directly from the observation y . This is not very restrictive since in most situations this is easy to do . We denote T the mask extractor T ( y ) = m. Our objective is then to recover the sequence x from the observations y and the corresponding binary masks m. Adopting a probabilistic viewpoint , we want to select a reconstruction x∗ which is the most plausible under the posterior distribution pX|Y ( ·|y ) . 2.2 MODEL . We formulate the problem as finding the most probable sequence conditioned on observations : x∗ = arg max x log pX|Y ( x|y ) = arg max x log pX ( x ) + log pY|X ( y|x ) ( 2 ) The prior term log pX ( x ) is unknown since we are in an unsupervised setting , while the likelihood log pY|X ( y|x ) does not lead to analytical or simple computational solution . To tackle these issues , let us introduce a mapping G : Y 7→ X , parameterized by a neural network ϕ and associating measurement y to its estimate x. G will allow us to approximate the underlying distribution of training sequences . By plugging G ( y ) into Equation 2 , the objective becomes : G∗ = arg max G Ey∼pY [ log pX ( G ( y ) ) ︸ ︷︷ ︸ prior ] + Ey∼pY [ log pY|X ( y|G ( y ) ) ] ︸ ︷︷ ︸ likelihood ( 3 ) 2.3 PRIOR HANDLING . Let us first handle the prior term in Equation 3 . We want the distribution induced from G ( y ) to be close to pX . In order to do so , we will use an adversarial approach . We will build on the ideas introduced in Bora et al . ( 2018 ) ; Pajot et al . ( 2019 ) for still images . The process is illustrated in Figure 1 . For a given observation y , we want to generate an approximation of the unknown true sequence x̂ ≡ G ( y ) . The prior pX being unknown , the only available information source is the observation y and the noise prior pM . For a given generated signal x̂ , we compute a corrupted version of x̂ through the known mask m̂ , ŷ ≡ F ( x̂ , m̂ ) with m̂ ∼ pM . We will train G to make the distributions of y and ŷ indistinguishable . In order to succeed , the generator G will have to remove the corruption from y and recover a sample x̂ from distribution pX . Generator G will then act as an inpainter conditioned on y . This will enforce the distribution of the reconstructed sequences x̂ to be close to the distribution of true ones x and maximize the prior term . A direct application of the adversarial training idea suggests using a discriminator operating directly on the sequences . We found out that using an additional discriminator on frames worked better than using a unique one operating on sequences . We then use two discriminators Ds and Df respectively associated with whole sequences and with individual frames to optimize G. Ds separates sequences y and ŷ . Df distinguishes real frames yt from fake ones ŷt . The loss function used for training G , Ds , and Df is : min G L ( G ) = max Ds , Df Ey∼pY , ŷ∼pGY [ logDs ( y ) +log ( 1−Ds ( ŷ ) ) + 1 T T∑ t=1 logDf ( yt ) +log ( 1−Df ( ŷt ) ) ] ( 4 ) with pGY ( y ) ≡ Em∼pM , x∼pGX [ pY|X , M ( y|x , m ) ] , corresponding to the distribution of the corrupted sequences ŷ generated via the measurement operator F . pGX ( x ) is the distribution of x̂ induced by G from y , i.e . x̂ = G ( y ) . 2.4 LIKELIHOOD HANDLING . Let us now handle the likelihood term in Equation 3 : Ey∼pY [ log pY|X ( y|G ( y ) ) ] . ( 5 ) This likelihood is maximised when we are able to perfectly reconstruct y from G ( y ) . One way to ensure this property is to constrain G to directly use y for the non occluded area of the reconstructed image G ( y ) . This can be easily achieved through the following mapping : G ( y ) ≡ ϕ ( y ) m̄+ y m ( 6 ) where ϕ is a NN responsible for reconstructing the missing part of y , m = T ( y ) is the mask retrieved from y. G maps Y to X with the help of mask m to ensure that the network will only generate values for occluded pixel , while keeping all the information from y . To summarize , optimizing the prior term amounts at training ϕ for inputting the missing pixels while optimizing the likelihood term is simply achieved by copying the non occluded portion of y . 2.5 TRAINING . G is optimized by descending the prior loss and Ds , Df by ascending it . Sequence discriminator Ds focuses on temporal dependence and coherence of pixel changes . Frame discriminator Df keeps an eye on spatial feature coherence of observation frames . 3 EXPERIMENTS . We evaluate our model on four datasets , characteristic of different types of image sequences . The first one , SST , is a realistic simulation of satellite observations . The other three are natural video datasets : FaceForensics++ , KTH , and BAIR , initially respectively used as benchmarks for forgery detection , motion detection , and video prediction . 3.1 DATASETS . SST The Sea Surface Temperature dataset used for the experiments includes 2 subsets of GLOBAL Sea Physical Analysis and Forecasting Product1 from E.U . Copernicus Marine Service Information . This is a monitor system providing simulated but realistic global ocean SST data , which integrates satellite-derived and in situ data by assimilation . Our dataset is a part of the hourly mean SST , the finest timescale we have access to . The data we use is a part of the archive of analysis integrating real-world data . We retrieved our training-and-validation set and test set respectively from two different marine regions . Detailed data description and information for accessing the dataset are provided in Appendix A. FaceForensics++ ( Rössler et al. , 2019 ) This dataset contains 1000 videos of non-occluded face movements on a static background . It was initially created for forgery detection . In our case , we extracted the faces from the original unforged videos with face_recognition2 , thus keeping only the changing component of the videos . The faces have been cropped and resized to 64×64 . KTH ( Schuldt et al. , 2004 ) A human action dataset containing 2391 video clips of 6 human actions . The videos have been recorded with 25 subjects in different environments . All frames have been resized to 64×64 . BAIR Robot Pushing Dataset ( Ebert et al. , 2017 ) This dataset contains 44374 videos recorded by an one-armed robot . It pushes objects and changes movement direction in a stochastic manner . All videos share similar tabletop with static background . All frames have been resized to 64×64 .

Summary: The paper addresses the problem of reconstructing a video sequence that contains occlusions. The paper focusses on remove four particular types of simulated occlusion - Raindrops, Remove  pixel, Cloud and Vertically moving Bar. The authors show good results in reconstructing these video sequences in an unsupervised manner. The paper uses a GAN based network to accomplish inpainting in the occluded regions.  The authors claim that the method is very flexible in terms of the data that needs to come in, and test this by deploying the method to solve quite different missing data problems (Using different type of occlusion and in different contexts).