Unsupervised Adversarial Image Reconstruction

1 INTRODUCTION . Many real world applications require acquiring information about the state of some physical system from incomplete and inaccurate measurements . For example , in infrared satellite imagery , one has to deal with the presence of clouds and a variety of other external factors perturbing the acquisition of temperature maps . This raises questions on how to recover the correct information and eliminate the contribution of external factors hindering the overall signal acquisition . In this context , signal recovery does not usually yield a unique solution , meaning that multiple signal reconstructions could trivially explain the measurements . For the above example , different missing temperature values could accurately explain the observations . To cope with this indeterminacy , one usually relies on prior information on the structure of the true signal in order to constrain the reconstruction to plausible solutions ( Stuart ( 2010 ) ) . A common approach is to use handcrafted , analytically tractable priors ( Candès et al . ( 2005 ) , Mota et al . ( 2017 ) ) . This approach is limited to situations for which the underlying signal structure can be easily described , which are rarely observed in the wild . Recent developments in generative models parameterized by neural networks ( Goodfellow et al . ( 2014 ) , Kingma & Welling ( 2013 ) , Dinh et al . ( 2016 ) ) offer a promising statistical approach to signal recovery , for which priors on the signal are not handcrafted anymore , but learned from large amounts of data . Despite exhibiting interesting results ( Bora et al . ( 2017 ) , Mardani et al . ( 2017 ) , Ledig et al . ( 2016 ) ) , these methods all require some form of supervision , either observation measurement-signal pairs , or at least unpaired samples from observations and underlying signals . For many practical problems , obtaining these samples is too expensive and/or impractical , which makes these approaches not suitable for such situations . We address the problem of image reconstruction in an unsupervised setting , when only corrupted observations are available , together with some prior information on the nature of the measurement process . ∗equal contribution The learning problem is formulated as finding the maximum a posteriori estimate of signals given their measurements on the training set . We derive a natural objective for our reconstruction network , composed of a linear combination of an adversarial loss for recovering realistic signals , and a reconstruction loss to tie the reconstruction to its associated observation ( Section 2.2 ) . This model is evaluated and compared to baselines on 3 image datasets , CelebA ( Liu et al . ( 2015 ) ) , LSUN Bedrooms ( Yu et al . ( 2015 ) ) , Recipe-1M ( Marin et al . ( 2018 ) ) , where we experiment with different types of measurement processes corrupting the images . Our contributions are : • A novel , computationally efficient framework for dealing with large scale signal recovery in an unsupervised context , applicable to a wide range of situations , • A model and a new way of training a deep learning architecture for implementing this framework , • Extensive evaluations on a number of image datasets with different measurement processes . 2 PRELIMINARIES . Notations . We use capital letters ( e.g . X ) for random variables , and lower-case letters ( e.g . x ) for their values . pX ( x ) denotes the distribution ( or its density in the appropriate context ) ofX evaluated at x . 2.1 PROBLEM SETTING .. Suppose there exists a signal X ∼ pX we wish to acquire , but we only have access to this signal through lossy , inaccurate measurements Y ∼ pY . The measurement process is modeled through a stochastic operator F mapping signals X to their associated observations Y . We will refer to F as the measurement process , which corrupts the input signal . F is parameterized by a random variable Θ ∼ pΘ following an underlying distribution pΘ we can sample from , which represents the factors of corruption . Thus , given a specific signal x , we can simulate its measurement by first sampling θ from pΘ , and then computing F ( x ; θ ) . Additional sources of uncertainty , e.g . due to unknown factors , can be modeled using additive i.i.d . Gaussian noise E ∼ N ( 0 , σ2I ) , so that the overall observation model becomes : Y = F ( X ; Θ ) + E ( 1 ) F is assumed to be differentiable w.r.t . its first argumentX , and Θ andX to be independent ( denoted X ⊥ Θ ) . Different instances of F will be considered ( refer to Section 4.2 ) , like random occlusions , information acquisition from a sparse subset of the signal , overly smoothing out and corrupting the original distribution with additive noise , etc . In such cases , the factors of corruption Θ might respectively represent the position of the occlusion , the coordinates of the acquired information , or simply the values of the additive noise . 2.2 APPROACH . Given an observation y , our objective is to find a signal x̂ as close as possible to the associated true signal x . From a probabilistic viewpoint , it is natural to formulate the problem as finding the maximum a posteriori ( MAP ) estimate , which consists in selecting the most probable signal x∗ under the posterior distribution pX|Y ( ·|y ) : x∗ = arg max x log pX|Y ( x|y ) ( 2 ) or equivalently : x∗ = arg max x log pY |X ( y|x ) + log pX ( x ) ( 3 ) where pY |X ( y|x ) is the likelihood of the signal x given observation y , and pX ( x ) is the prior probability evaluated at x . Therefore , a good reconstruction must be likely to have generated the data , i.e . yield high likelihood , and look realistic , i.e . yield high probability under the prior . In the general case , calculating the likelihood term pY |X ( y|x ) requires marginalizing over noise parameters Θ and this does not yield an analytic form . As for the prior pX ( x ) , it is unknown , and we have no access to samples fromX since we are in an unsupervised setting : there is then no direct way to estimate pX either . In the general case considered here , with no assumption on the form of the distributions , solving Equation ( 3 ) is up to our knowledge an open problem . In the following sections , we will introduce an approach to deal with the likelihood term ( Section 3.1 ) , and the unknown prior term ( Section 3.2 ) in order to provide an approximate solution to equation ( 3 ) ( Section 3.3 ) . For that , we will formulate the problem as learning a mapping G : Y → X that links each measurement y to its associated MAP estimate x∗ on the training set . The associated objective is then : G∗ = arg max G EpY { log pY |X ( y|G ( y ) ) + log pX ( G ( y ) ) } ( 4 ) Which is obtained by plugging G ( y ) = x into equation 3 and taking the expectation w.r.t . the distribution of observations pY . 3 METHOD . From Equation ( 4 ) , we see that a valid reconstruction mapping G must yield high probability for the likelihood and the prior . This will guide the design of an appropriate objective during the following section , where the reconstruction mapping G will be implemented using a neural network . 3.1 HANDLING THE LIKELIHOOD TERM In the general case , evaluating the likelihood pY |X ( y|x ) in equation ( 3 ) requires marginalizing on the unobserved noise variable Θ : pY |X ( y|x ) = EpΘpY |X , Θ ( y|x , θ ) , which involves computing an intractable integral . Most probabilistic model for image denoising make assumptions on the structure of the measurement operator F ( . , Θ ) and on the distribution of Θ in order to obtain an analytic form for the expectation ( Boyat & Joshi ( 2015 ) , Alkinani & El-Sakka ( 2017 ) ) . Here , we consider more general measurement operators which do not necessarily lead to such a simplification and therefore proceed in a different way . We outline below , the main steps of the method for handling the likelihood term pY |X ( y|x ) in equation ( 4 ) . The complete derivation is provided in Appendix A . 1 . Making use of the independence between X and Θ , the expectation term EpY log pY |X ( y|G ( y ) ) in equation ( 4 ) can be rewritten as : EpΘpXpY |X , Θ log pY |X , Θ ( y|G ( y ) , θ ) + c1 ( 5 ) , with c1 constant w.r.t . G. 2 . The general measurement process described in equation ( 1 ) induces log pY |X , Θ ( y|G ( y ) , θ ) to yield a simple analytic expression : log p ( y|G ( y ) , θ ) = − 1 2σ2 ‖y − F ( G ( y ) ; θ ) ‖22 + c2 ( 6 ) with c2 a constant . 3 . The likelihood term EpY log pY |X ( y|G ( y ) ) can then be replaced in objective ( 4 ) by − EpΘpXpY |X , Θ 1 2σ2 ‖y − F ( G ( y ) ; θ ) ‖22 ( 7 ) Equation ( 7 ) shows that the likelihood term can be evaluated by first sampling a measurement y conditioned on a corruption parameter θ and signal x , and then constrainG such that ‖y − F ( G ( y ) ; θ ) ‖22 is close to zero . Note that in this expression , the same parameter θ is used for simulating ỹ from x̃ and y from x ( see Figure 1 and section 3.3 for more details ) . Unfortunately , this requires first sampling x from the signal distribution pX which is unknown . In the following sections , we will see how we work around this problem . 3.2 HANDLING THE PRIOR TERM Maximizing w.r.t . the prior term pX ( G ( y ) ) in equation ( 4 ) is similar to learning a mapping G , such that the distribution induced by G ( y ) , EpY pX ( G ( y ) ) is close to the distribution pX . The prior pX being unknown , the only sources of information are the lossy measurements y and the known prior pΘ on the measurement process . In order to learn an approximation of the true prior pX , we will use a form of generative adversarial learning , and build on an idea introduced in the AmbientGAN model by Bora et al . ( 2018 ) . AmbientGAN aims at learning an unconditional generative model G of the true signal distribution pX , when only lossy measurements y of the signal are available together with a known stochastic measurement operator F . In AmbientGAN , a generator is trained to produce uncorrupted signal samples from a latent code so that the generated signals when corrupted are indistinguishable from the observation measurements . In Bora et al . ( 2018 ) , the authors show that for some families of noise distributions pΘ , the generator ’ s induced distribution matches the signal ’ s true distribution . Note that even if the generation process of the observations y in AmbientGAN is similar to the one considered in this paper ( see Section 2.1 ) , the objective is however different : when the aim of AmbientGAN is to learn a distribution of the underlying signal by sampling a latent space , ours is to reconstruct corrupted signals . In order for G to produce uncorrupted signals we will use an approach inspired from AmbientGAN , as illustrated in Figure 2 . Given an observation y , one wants to reconstruct a latent signal approximation x̂ = G ( y ) so that a corrupted version of this signal ŷ = F ( x̂ ) will have a distribution indistinguishable from the one of the observations y . The generator G and a discriminator D are trained on observations y and generated samples ŷ . The corresponding loss is the following1 : Lprior ( G ) : = max D EY∼pY , Ŷ∼pGY { logD ( y ) + log ( 1−D ( ŷ ) ) } ( 8 ) where pGY corresponds to the distribution induced by G ’ s corrupted outputs ( ŷ in Figure 2 ) , i.e . pGY ( y ) : = EpΘpGX { p ( y|x , θ ) } and pGX denotes the marginal distribution induced by G ’ s outputs ( x̂ in Figure 2 ) : pGX ( x ) : = EpY p G X|Y ( x|y ) = EpY δ ( x−G ( y ) ) 2 . This penalty enforces the marginal pGX to be close to the true prior distribution pX , and thus forces G to map its input measurements onto pX . 3.3 PUTTING EVERYTHING TOGETHER In Section 3.1 , we have shown that it is possible to maximize the average log-likelihood term in equation ( 4 ) , given that we can sample from the unknown prior distribution pX . In Section 3.2 , we have shown how it is possible to enforce the generator to produce samples from pX , without ever having access to uncorrupted samples . The idea is then to use the distribution induced by the generator ’ s output pGX as a proxy for pX to compute an approximate value of the expectation in equation ( 5 ) : This gives us the following penalty term ( see appendix A ) : Llikeli ( G ) : = EpΘpGX , Ŷ∼pY |X , Θ ‖ŷ − F ( G ( ŷ ) ; θ ) ‖ 2 2 ( 9 ) 1the min term of the adversarial loss will be introduced later , see Equation ( 10 ) 2δ ( x ) is the Dirac delta function , which is equal to zero everywhere except in x . The full objective is a linear combination of penalties ( 8 ) and ( 9 ) : arg min G Lprior ( G ) + λ · Llikeli ( G ) ( 10 ) As illustrated by the dependencies highlighted in Figure 2 , in the process of minimizing Lprior , we sample from the marginal likelihood pGY ( y ) : = EpΘpGX { p ( y|x , θ ) } . The expectancy in the likelihood term Llikeli is precisely computed w.r.t . this distribution . We can then use the same samples in order to minimize the full objective ( 10 ) . This gives us the Algorithm 1 described below , along with the dependency structure illustrated in Figure 3 . Algorithm 1 Training Procedure . Require : Initialize parameters of the generator G and the discriminator D. while ( G , D ) not converged do Sample { yi } 1≤i≤n from data distribution pY Sample { θi } 1≤i≤n from PΘ Sample { εi } 1≤i≤n from PE Set ŷi to F ( G ( yi ) , θi ) + εi for 1 ≤ i ≤ n Update D by ascending : 1 n n∑ i=1 logD ( yi ) + log ( 1−D ( ŷi ) ) Update G by descending : 1 n n∑ i=1 λ · ‖ŷi − F ( G ( ŷi ) ; θi ) ‖22 + log ( 1−D ( ŷi ) ) 3 end while

Summary: This paper presents a method to reconstruct images using only noisy measurements. This problem is practically interesting, since the noiseless signal may be unavailable in many applications. The approach combines ideas from recent development in compressed sensing and GANs. However, the model’s presentation is confusing, and many important details of the experiments are missing.