Title: UNSUPERVISED ADVERSARIAL IMAGE RECONSTRUCTION

Abstract: We address the problem of recovering an underlying signal from lossy, inaccurate observations in an unsupervised setting. Typically, we consider situations where there is little to no background knowledge on the structure of the underlying signal, no access to signal-measurement pairs, nor even unpaired signal-measurement data. The only available information is provided by the observations and the measurement process statistics. We cast the problem as finding the maximum a posteriori estimate of the signal given each measurement, and propose a general framework for the reconstruction problem. We use a formulation of generative adversarial networks, where the generator takes as input a corrupted observation in order to produce realistic reconstructions, and add a penalty term tying the reconstruction to the associated observation. We evaluate our reconstructions on several image datasets with different types of corruptions. The proposed approach yields better results than alternative baselines, and comparable performance with model variants trained with additional supervision.

1. INTRODUCTION:

Many real world applications require acquiring information about the state of some physical system from incomplete and inaccurate measurements. For example, in infrared satellite imagery, one has to deal with the presence of clouds and a variety of other external factors perturbing the acquisition of temperature maps. This raises questions on how to recover the correct information and eliminate the contribution of external factors hindering the overall signal acquisition.

In this context, signal recovery does not usually yield a unique solution, meaning that multiple signal reconstructions could trivially explain the measurements. For the above example, different missing temperature values could accurately explain the observations. To cope with this indeterminacy, one usually relies on prior information on the structure of the true signal in order to constrain the reconstruction to plausible solutions (Stuart (2010) ). A common approach is to use handcrafted, analytically tractable priors (Candès et al. (2005) , Mota et al. (2017) ). This approach is limited to situations for which the underlying signal structure can be easily described, which are rarely observed in the wild.

Recent developments in generative models parameterized by neural networks (Goodfellow et al. (2014) , Kingma & Welling (2013) , Dinh et al. (2016) ) offer a promising statistical approach to signal recovery, for which priors on the signal are not handcrafted anymore, but learned from large amounts of data. Despite exhibiting interesting results (Bora et al. (2017) , Mardani et al. (2017) , Ledig et al. (2016) ), these methods all require some form of supervision, either observation measurement-signal pairs, or at least unpaired samples from observations and underlying signals.

For many practical problems, obtaining these samples is too expensive and/or impractical, which makes these approaches not suitable for such situations.

We address the problem of image reconstruction in an unsupervised setting, when only corrupted observations are available, together with some prior information on the nature of the measurement process.

The learning problem is formulated as finding the maximum a posteriori estimate of signals given their measurements on the training set. We derive a natural objective for our reconstruction network, composed of a linear combination of an adversarial loss for recovering realistic signals, and a reconstruction loss to tie the reconstruction to its associated observation (Section 2.2). This model is evaluated and compared to baselines on 3 image datasets, CelebA (Liu et al. (2015) ), LSUN Bedrooms (Yu et al. (2015) ), Recipe-1M (Marin et al. (2018) ), where we experiment with different types of measurement processes corrupting the images.

Our contributions are:

• A novel, computationally efficient framework for dealing with large scale signal recovery in an unsupervised context, applicable to a wide range of situations,

• A model and a new way of training a deep learning architecture for implementing this framework,

• Extensive evaluations on a number of image datasets with different measurement processes.

2. PRELIMINARIES:

Notations. We use capital letters (e.g. X) for random variables, and lower-case letters (e.g. x) for their values. p X (x) denotes the distribution (or its density in the appropriate context) of X evaluated at x.

2.1. PROBLEM SETTING.:

Suppose there exists a signal X ∼ p X we wish to acquire, but we only have access to this signal through lossy, inaccurate measurements Y ∼ p Y . The measurement process is modeled through a stochastic operator F mapping signals X to their associated observations Y . We will refer to F as the measurement process, which corrupts the input signal. F is parameterized by a random variable Θ ∼ p Θ following an underlying distribution p Θ we can sample from, which represents the factors of corruption. Thus, given a specific signal x, we can simulate its measurement by first sampling θ from p Θ , and then computing F (x; θ). Additional sources of uncertainty, e.g. due to unknown factors, can be modeled using additive i.i.d. Gaussian noise E ∼ N (0, σ 2 I), so that the overall observation model becomes:

EQUATION

F is assumed to be differentiable w.r.t. its first argument X, and Θ and X to be independent (denoted X ⊥ ⊥ Θ). Different instances of F will be considered (refer to Section 4.2), like random occlusions, information acquisition from a sparse subset of the signal, overly smoothing out and corrupting the original distribution with additive noise, etc. In such cases, the factors of corruption Θ might respectively represent the position of the occlusion, the coordinates of the acquired information, or simply the values of the additive noise.

2.2. APPROACH:

Given an observation y, our objective is to find a signal x as close as possible to the associated true signal x. From a probabilistic viewpoint, it is natural to formulate the problem as finding the maximum a posteriori (MAP) estimate, which consists in selecting the most probable signal x * under the posterior distribution p X|Y (•|y):

x * = arg max

EQUATION

or equivalently:

EQUATION

where p Y |X (y|x) is the likelihood of the signal x given observation y, and p X (x) is the prior probability evaluated at x. Therefore, a good reconstruction must be likely to have generated the data, i.e. yield high likelihood, and look realistic, i.e. yield high probability under the prior.

In the general case, calculating the likelihood term p Y |X (y|x) requires marginalizing over noise parameters Θ and this does not yield an analytic form. As for the prior p X (x), it is unknown, and we have no access to samples from X since we are in an unsupervised setting: there is then no direct way to estimate p X either. In the general case considered here, with no assumption on the form of the distributions, solving Equation ( 3) is up to our knowledge an open problem.

In the following sections, we will introduce an approach to deal with the likelihood term (Section 3.1), and the unknown prior term (Section 3.2) in order to provide an approximate solution to equation (3) (Section 3.3). For that, we will formulate the problem as learning a mapping G : Y → X that links each measurement y to its associated MAP estimate x * on the training set. The associated objective is then:

EQUATION

Which is obtained by plugging G(y) = x into equation 3 and taking the expectation w.r.t. the distribution of observations p Y .

3. METHOD:

From Equation (4), we see that a valid reconstruction mapping G must yield high probability for the likelihood and the prior. This will guide the design of an appropriate objective during the following section, where the reconstruction mapping G will be implemented using a neural network.

3.1. HANDLING THE LIKELIHOOD TERM:

x F < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 7). To compute this expectation, one first simulates an observation y from a signal x using F (x; θ) , then generates x = G(y) and ỹ = F (x; θ), as in the Figure . This allows us to compute the MSE term in the above expression.

A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n

In the general case, evaluating the likelihood p Y |X (y|x) in equation (3) requires marginalizing on the unobserved noise variable Θ: (y|x, θ) , which involves computing an intractable integral. Most probabilistic model for image denoising make assumptions on the structure of the measurement operator F (., Θ) and on the distribution of Θ in order to obtain an analytic form for the expectation (Boyat & Joshi (2015) , Alkinani & El-Sakka (2017) ). Here, we consider more general measurement operators which do not necessarily lead to such a simplification and therefore proceed in a different way.

p Y |X (y|x) = E p Θ p Y |X,Θ

We outline below, the main steps of the method for handling the likelihood term p Y |X (y|x) in equation ( 4). The complete derivation is provided in Appendix A.

1. Making use of the independence between X and Θ, the expectation term E p Y log p Y |X (y|G(y)) in equation ( 4) can be rewritten as :

EQUATION

, with c 1 constant w.r.t. G.

2. The general measurement process described in equation ( 1) induces log p Y |X,Θ (y|G(y), θ) to yield a simple analytic expression:

log p(y|G(y), θ) = - 1 2σ 2 y -F (G(y); θ) + c 2 (6)

with c 2 a constant.

3. The likelihood term E p Y log p Y |X (y|G(y)) can then be replaced in objective (4) by

-E p Θ p X p Y |X,Θ 1 2σ 2 y -F (G(y); θ) 2 2 (7)

Equation ( 7) shows that the likelihood term can be evaluated by first sampling a measurement y conditioned on a corruption parameter θ and signal x, and then constrain G such that y -F (G(y); θ)

2 2

is close to zero. Note that in this expression, the same parameter θ is used for simulating ỹ from x and y from x (see Figure 1 and section 3.3 for more details). Unfortunately, this requires first sampling x from the signal distribution p X which is unknown. In the following sections, we will see how we work around this problem.

3.2. HANDLING:

THE PRIOR TERM y < l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k z D Q K b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M 7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k z D Q K b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M 7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k z D Q K b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M 7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k z D Q K b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M 7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > ŷ < l a t e x i t s h a 1 _ b a s e 6 4 = " V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s 1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G n G F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c

q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 

V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s 1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G n G F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s 1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G n G F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s 1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G n G F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " + 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d 4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d 4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d 4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d 4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > G < l a t e x i t s h a 1 _ b a s e 6 4 = " T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M 2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M

T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M

T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M

/ L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = <

/ l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5

N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = <

/ l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5

N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = <

/ l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 4). An observation y is sampled, and then transformed by the generative network into a reconstructed signal x = G(y). One then simulates a measurement ŷ := F (x; θ) from this reconstruction. We then enforce the distributions of observations p Y and simulated measurements p G Y to be similar using an adversarial loss. In order to produce indistinguishable distributions, the generator G has to remove the corruption and recover a sample x from p X . Maximizing w.r.t. the prior term p X (G(y)) in equation ( 4) is similar to learning a mapping G, such that the distribution induced by G(y), E p Y p X (G(y)) is close to the distribution p X . The prior p X being unknown, the only sources of information are the lossy measurements y and the known prior p Θ on the measurement process. In order to learn an approximation of the true prior p X , we will use a form of generative adversarial learning, and build on an idea introduced in the AmbientGAN model by Bora et al. (2018) . AmbientGAN aims at learning an unconditional generative model G of the true signal distribution p X , when only lossy measurements y of the signal are available together with a known stochastic measurement operator F . In AmbientGAN, a generator is trained to produce uncorrupted signal samples from a latent code so that the generated signals when corrupted are indistinguishable from the observation measurements. In Bora et al. (2018) , the authors show that for some families of noise distributions p Θ , the generator's induced distribution matches the signal's true distribution. Note that even if the generation process of the observations y in AmbientGAN is similar to the one considered in this paper (see Section 2.1), the objective is however different: when the aim of AmbientGAN is to learn a distribution of the underlying signal by sampling a latent space, ours is to reconstruct corrupted signals.

N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n

In order for G to produce uncorrupted signals we will use an approach inspired from AmbientGAN, as illustrated in Figure 2 . Given an observation y, one wants to reconstruct a latent signal approximation x = G(y) so that a corrupted version of this signal ŷ = F (x) will have a distribution indistinguishable from the one of the observations y. The generator G and a discriminator D are trained on observations y and generated samples ŷ. The corresponding loss is the following 1 :

EQUATION

where p G Y corresponds to the distribution induced by G's corrupted outputs (ŷ in Figure 2 ) , i.e. p G Y (y) := E p Θ p G X p(y|x, θ) and p G X denotes the marginal distribution induced by G's outputs (x in Figure 2 

): p G X (x) := E p Y p G X|Y (x|y) = E p Y δ x -G(y) 2 .

This penalty enforces the marginal p G X to be close to the true prior distribution p X , and thus forces G to map its input measurements onto p X . 

3.3. PUTTING EVERYTHING TOGETHER:

n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f

n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f

n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f

n I G T X W a m a D S t W t u X O R V f A K q E K h x q D y 1 R / G L I 1 Q G i a o 1 j 3 P T Y y f U 2 U 4 E z g t 9 1 O N C W U T O s K e R U k j 1 H 4 + X 3 R K z q 0 z J G G s 7 J O G z N 3 f E z m N t M 6 i w H Z G 1 I z 1 c m 1 m / l f

z D Q K b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M

z D Q K b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M

b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M

b G d E c W y W a 7 n 5 X 6 2 X Y n j t Z 0 I l K X L F F h + F q S Q Y k / x 2 M h S a M 5 R T C 5 R p Y X c l b E w 1 Z W g T q t g Q v O W T V 6 F 9 U f c s 3 1 / W G j d F H G U 4 g V M 4 B w + u o A F 3 0 I Q W M J j A M

E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G n

G F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " 

+ 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d 4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d 4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d 4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + 2 a + Y d A k K 5 g X A K P W K N I Z / 6 d j I O k = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F 8 F R 2 R d B j 0 Y v H C v Y D 2 q V k 0 7 S N z S Z L M i u U p f / B i w d F v P p / v P l v T N s 9 a O s L g Y d 3 Z s j M G y V S W P T 9 b 6 + w t r 6 x u V X c L u 3 s 7 u 0 f l A + P m l a n h v E G 0 1 K b d k Q t l 0 L x B g q U v J 0 Y T u N I 8 l Y 0 v p 3 V W 0 / c W K H V A 0 4 S H s Z 0 q M R A M I r O a n Z x x J H 2 y h W / 6 s 9 F V i H I o Q K 5 6 r 3 y V 7 e v W R p z h U x S a z u B n 2 C Y U Y O C S T 4 t d V P L E 8 r G d M g 7 D h W N u Q 2 z + b Z T c u a c P h l o 4 5 5 C M n d / T 2 Q 0 t n Y S R 6 4 z p j i y y 7 W Z + V + t k + L g O s y E S l L k i i 0 + G q S S o C a z 0 0 l f G M 5 Q T h x Q Z o T b l b A R N Z S h C 6 j k Q g i W T 1 6 F 5 k U 1 c H x / W a n d 5 H E U 4 Q R O 4 R w C u I I a 3 E E d G s D g E Z 7 h F d

i i 0 + i j J J M C G z 6 0 k o N G c o J x Y o 0 8 L u S t i I a s r Q Z l S x I X j L J 6 9 C + 6 L u W b 6 / r D V u i j j K c A K n c A 4 e X E E D 7 q A J L W A Q w z O 8 w p u j n R f n 3 f l Y t J a c Y u Y Y / s j 5 / A F E 5 J C z < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T d k k W 5 f p M Z r c a d 8 M X R D z r R q Y X l c = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 Q N p T N Z t M u 3 W z C 7 k Q o o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L A z b 5 B K Y d B 1 v 5 3 S 2 v

i i 0 + i j J J M C G z 6 0 k o N G c o J x Y o 0 8 L u S t i I a s r Q Z l S x I X j L J 6 9 C + 6 L u W b 6 / r D V u i j j K c A K n c A 4 e X E E D 7 q A J L W A Q w z O 8 w p u j n R f n 3 f l Y t J a c Y u Y Y / s j 5 / A F E 5 J C z < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T d k k W 5 f p M Z r c a d 8 M X R D z r R q Y X l c = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 Q N p T N Z t M u 3 W z C 7 k Q o o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L A z b 5 B K Y d B 1 v 5 3 S 2 v

i i 0 + i j J J M C G z 6 0 k o N G c o J x Y o 0 8 L u S t i I a s r Q Z l S x I X j L J 6 9 C + 6 L u W b 6 / r D V u i j j K c A K n c A 4 e X E E D 7 q A J L W A Q w z O 8 w p u j n R f n 3 f l Y t J a c Y u Y Y / s j 5 / A F E 5 J C z < / l a t e x i t > G < l a t e x i t s h a 1 _ b a s e 6 4 = " T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M 2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M 2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M 2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T f B f c O T Z N Z T e l V x 2 S R w U a 3 Y x d L E = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 K D H F u w H t K F s t p N 2 7 W Y T d j d C C f 0 F X j w o 4 t W f 5 M 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E c G 1 c 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 5 a O U 8 W w y W I R q 0 5 A N Q o u s W m 4 E d h J F N I o E N g O x r e z e v s J l e a x f D C T B P 2 I D i U P O a P G W o 2 7 f r n i V t 2 5 y C p 4 O V Q g V 7 1 f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 6 K k E W o / m y 8 6 J W f W G Z A w V v Z J Q + b u 7 4 m M R l p P o s B 2 R t S M 9 H J t Z v 5 X 6 6 Y m v P Y z L p P U o G S L j 8 J U E B O T 2 d V k w B U y I y Y W K F P c 7 k r Y i C r K j M 2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > F < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = <

/ l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5

N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = <

/ l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5

N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U U v H q W L Y Z L G I V S e g G g W X 2 D T c C O w k C m k U C G w H 4 9 t Z v f 2 E S v N Y P p h J g n 5 E h 5 K H n F F j r c Z d v 1 x x q + 5 c Z B W 8 H C q Q q 9 4 v f / U G M U s j l I Y J q n X X c x P j Z 1 Q Z z g R O S 7 1 U Y 0 L Z m A 6 x a 1 H S C L W f z R e d k j P r D E g Y K / u k I X P 3 9 0 R G I 6 0 n U W A 7 I 2 p G e r k 2 M / + r d V M T X v s Z l 0 l q U L L F R 2 E q i I n J 7 G o y 4 A q Z E R M L l C l u d y V s R B V l x m Z T s i F 4 y y e v Q u u i 6 l l u X F Z q N 3 k c R T i B U z g H D 6 6 g B v d Q h y Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Q W n H z m G P 7 I + f w B m n O M y g = = < / l a t e x i t >

Discriminator MSE:

Figure 3 -General Approach. We wish to train G to recover a plausible signal from lossy measurements. As is shown in Section 2.2, this requires the reconstructions x := G(y) to have high probability under the likelihood and the prior. For simplicity, variable E has been omitted. Prior : we sample a measurement y from the data, produce a reconstruction x, and sample a perturbation parameter θ. We enforce the simulated measurement ŷ := F (x; θ) to be similar to measurements in the data using an adversarial penalty. Intuitively, this requires the network to remove the corruption.

Likelihood : to enforce G to produce reconstructions with high likelihood, it is not possible to add a penalty to constrain the mean square error (MSE) between y and ŷ to be small. This is because the underlying perturbation that caused y is unknown, and may be different from θ. Starting from ŷ we generate a ỹ (see figure 3 ) using the same θ as the one used for generating ŷ. We then constrain ŷỹ 2 2 to be small (ỹ = F (G(ŷ)).

In Section 3.1, we have shown that it is possible to maximize the average log-likelihood term in equation ( 4), given that we can sample from the unknown prior distribution p X . In Section 3.2, we have shown how it is possible to enforce the generator to produce samples from p X , without ever having access to uncorrupted samples. The idea is then to use the distribution induced by the generator's output p G X as a proxy for p X to compute an approximate value of the expectation in equation ( 5): This gives us the following penalty term (see appendix A):

L likeli (G) := E p Θ p G X , Ŷ ∼p Y |X,Θ ŷ -F (G(ŷ); θ) 2 2 (9)

1 the min term of the adversarial loss will be introduced later, see Equation (10) 2 δ(x) is the Dirac delta function, which is equal to zero everywhere except in x.

4.2. CORRUPTIONS:

Let us present the different measurement processes F used in the experiments, also named corruptions:

Remove-Pixel. This measurement process randomly samples a fraction p of pixels uniformly and sets the associated channel values to 0. All the corresponding channel values are set to 0. Remove-Pixel-Channel. Instead of setting to 0 a pixel for all channels as in Remove-Pixel, one samples a pixel coordinate and a channel, and sets the corresponding value to 0. Convolve-Noise. Here F (x; θ) := k * x + θ, where * is the convolution operator and k is a mean filter of size l. For each pixel, noise θ sampled from a zero-mean Gaussian of variance σ 2 C is added to the previous result. Patch-Band. A horizontal band of height h whose vertical position in the image is uniformly sampled from the set of possible positions. For each pixel falling inside the band, its associated value is set to 0. The resulting measurement for pixel at column i and row j can be summarized as:

EQUATION

where θ is uniformly sampled from {1, . . . , H -h}, and H is the image height. In the experiments, h is set to 20.

4.3. BASELINES:



4.3.1. CONDITIONAL AMBIENTGAN.:

This is our only unsupervised baseline. The context is the same as for our model: the measurement process F is assumed known, there is no access to samples from the uncorrupted signal distribution p X , but only to their corrupted counterpart p Y . This baseline is a combination of two recent techniques in the field of signal recovery: the aforementioned AmbientGan Bora et al. (2018) and CS- GAN Bora et al. (2017) .

An unconditional generator G is trained using the AmbientGan framework (Bora et al. (2018) ) for each type of measurement process F , in order to produce samples from p X (see Section 4.2).

The distribution induced by the generator p G X is an approximation of p X (at the optimum, both distributions match, i.e. p G X = p X ). Given a specific measurement y, the reconstruction x is the signal from G that is closest to y, as in Bora et al. (2017) . To find x = G(ẑ), we search for the latent code ẑ of G, such that ẑ = arg min z y -G(z) 2 2 + R(z). R(z) is a regularizing term that enforces the latent code to stay in G's input domain. This objective is optimized using stochastic gradient descent. To train G, we use the same architectures and hyper-parameters as those provided by the authors. Because this approach may be sensitive to the initial latent code, we reiterate this approach three times and select the best resulting image.

4.3.2. UNPAIRED VARIANT.:

This is a variant of our model where we have access to samples of the signal distribution p X . This means that although we have no paired samples from p X,Y , we have access to unpaired samples from p X and p Y . This baseline is similar to our model but instead of discriminating between a measurement from the data y and a simulated measurement ŷ, we directly discriminate between samples x from the signal distribution and the output of the reconstruction network x. For a diagram describing the model, refer to appendix C.1.

4.3.3. PAIRED VARIANT.:

This is a variant of our model where we have access to signal measurement pairs (y, x) from the joint distribution p Y,X . Given input measurement y, the reconstruction is obtained by regressing y to the associated signal x using a MSE loss. In order to avoid blurry samples, we add an adversarial term in the objective in order to constrain G to produce realistic samples, as in Isola et al. (2016) .

The model is trained using the same architectures as our model, and the hyperparameters have been found using cross-validation. For a diagram describing the model, refer to appendix C.2.

4.3.4. MEASUREMENT SPECIFIC BASELINES.:

We also compare our model to baselines that where designed to remove specific corruptions.

Deep Image Prior (Ulyanov et al. (2017) ). Given a generator G φ parametrized by randomly initialized weights φ and a measurement y, this method seeks to find a reconstruction from G φ that is close to the measurement. For corruptions processes Patch-Band, Remove-Pixel and Remove-Pixel-Channel, we assume the access to the θ associated to the observations in the data (i.e. in this case, the mask). For more details, please refer to appendix C. Biharmonic Inpainting (Damelin & Hoang ( 2018)). By considering inpainting as a smooth surface extension domain, this baseline resolves a biharmonic equation to obtain a high order approximation of the image. This approximation is then extended to the missing part of the image. This method assumes access to the θ associated to the observations in the data (i.e. in this case, the mask).

Total Variation Denoising (Chambolle (2004) ). This denoising baseline aims to minimize the total variation of an image i.e the integral of the absolute gradient of the image. Reducing the total variation of the image removes unwanted detail, such as white noise artifacts while preserving important details such as edges and corners.

5. RESULTS:

We will now present our results. First, we compare quantitatively our model with non-measurement specific baselines on CelebA. We then present qualitative results with samples from our model and these baselines. Comparisons with measurement specific baselines are presented in appendix D for the three datasets.

5.1. QUANTITATIVE RESULTS:

We compare our model with baselines introduced in the previous section. We report mean square error (MSE) scores between the reconstructed x and the true signal x used to generate the input y. Table 1 shows the MSE computed on the test set, a randomly selected subset of CelebA comprised of 40000 images. Because the Conditional AmbientGan model is too computationally expensive, we only report the MSE on 40 randomly chosen samples of the test set. Quantitatively, our model performs well. Except for the Conditional Ambiant GAN, all the methods are quite similar in terms of MSE. Our unsupervised model reaches performance similar to its variants trained using additional supervision. We also note that when the aligned signal-observation pairs are not used (as in our Unpaired Variant), results are comparable -sometimes better -than when these pairs are used directly (as in our Paired Variant). This suggests that our likelihood term is sufficient to condition the reconstruction on the input signal.

5.2. QUALITATIVE RESULTS:

However, quantitative results gives us only partial information. We now evaluate the quality of our reconstruction on three different datasets (Section 4.1). Figure 4 shows reconstructions obtained from different models on the CelebA dataset.We observe that Conditional AmbientGAN yields visually poor results, especially for the Remove-Pixel and Remove-Pixel-Channel measurement processes. We hypothesize that this is due to the large Euclidean distance between the measurements and the associated signals, and the suboptimality of the generator. Visually, the quality of our model's reconstructions are coherent with the quantitative results: they are comparable to its paired and unpaired counterparts (Section 4.3). Figures ( 5), ( 6), ( 7), and (8) each show reconstructions from a given measurement processe on different datasets. Our model is able to produce images with good visual quality while remaining coherent with the underlying uncorrupted images. In Figures ( 11), ( 12), ( 13) and ( 14) in appendix D, we compare our model with commonly used inpainting or denoising methods. We can see that contrary to these methods, we are able to capture semantic information from the dataset. Typically, in Figure 14 , the model infers missing eyes or noses, without ever having seen them. Additional samples are available in the appendix D, refer to Figures ( 15), ( 16), ( 17), and (18). 

6. RELATED WORK:

To our knowledge, there is no other Deep Learning approach attempting to solve the unsupervised signal reconstruction problem. However, some of the ideas developed here are close to or even inspired from recent work. 

)) attempt to retrieve maximum a posteriori estimates of the super resolution image conditioned on an input image. They too use a generative model of the signal trained in an adversarial fashion using samples from signal distribution to constrain their reconstructions. Their approach is fully supervised.

Other works attempt to solve ill-posed inverse problems using generative models (Bora et al. (2017) , Asim et al. (2018 ), Tripathi et al. (2018) , Van Veen et al. ( 2018)). The general approach in all these papers consists to first train a generative model on the uncorrupted signal distribution. Then, given a measurement from which we wish to reconstruct the signal, it is inverted by finding the latent input code that generated the uncorrupted image, by minimizing the mean square error between the corrupted reconstruction and the measurement. This requires solving an optimization problem for each image, which takes several minutes (Ulyanov et al. (2017) ) on GPU, and requires random restarts to avoid falling a bad local minima. Again, the setting is fully supervised.

Finally Lehtinen et al. (2018) propose a method for denoising images without direct supervision. They train a network to regress a corrupted image to the same image with a different corruption value. Assuming the corruption has zero-mean, their network learns to remove the corruption by the conditional expectation. This setting implicitly assumes access to the distribution of uncorrupted images in order to generate different noisy versions of the same image, which is not our case.

7. CONCLUSION:

We have proposed a general formulation to recover a signal from lossy measurements using a neural network, without having access to uncorrupted signal data. We have formulated the problem as finding a maximum a posteriori estimate of the signal given its observation, for all observations in the training set. This gives us a natural objective for our neural network, composed of a linear combination of an adversarial loss for recovering realistic signals, and a reconstruction loss to tie the reconstruction to its associated observation. Our approach yields results superior to the baselines, while staying competitive with other model variants that have access to higher forms of supervision.

For future work, we plan to apply our framework to different corruption processes, and evaluate our model's performance in real world settings, specifically for retrieving uncorrupted scientific data. Another interesting research direction would be to make our reconstruction network stochastic, in order to approximate the true posterior of the signal given the measurement, and to obtain uncertainty estimates.

A ADDITIONAL STEPS FOR HANDLING THE LIKELIHOOD:

We develop below the different steps for handling the likelihood summarized in Section 3.1:

1. Making use of the independence between X and Θ, we rewrite the expectation term E p Y log p Y |X (y|G(y)) in equation ( 4 with c 2 constant. This result is directly obtained using the fact that E ∼ N (0, σ 2 I). 

B ARCHITECTURE DETAILS:

Network architecture. Our network architectures are inspired by the Self-Attention GAN architecture in Zhang et al. (2018) . They use residual networks (He et al. (2016)) , where each residual block of the generator and discriminator is comprised of 2 repeated sequences of batch normalization (Ioffe & Szegedy (2015) ), ReLU activation, spectral normalization (Miyato et al. (2018) ) and 3 × 3 convolutional layers. For the discriminator, we use the same as Zhang et al. (2018) , and for reconstruction network G, we propose an image-to-image variant of their generator. We have not added downsampling layers: we have found that they degraded the overall model's performance. For corruption processes that yield observations that are very correlated with the input, such as Patch Band and Convolve-Noise, we have found that using G(y) := y + Net(y) for the reconstruction network allows us to initialize G close to identity, accelerates training and augments the overall quality of the samples.

Hyperparameters. Hyperparameters have been selected on the validation set, based on the mean square error between the reconstructions x and the image x. As in Zhang et al. (2018) , we use imbalanced learning rates for the generator and the discriminator (0.0001 and 0.0004, respectively), using the Adam optimizer (Kingma & Ba (2014)), using β 1 = 0 and β 2 = 0.9. The weights are initialized using orthogonal initialization. We set λ = 2, and exponentially decay the learning rate every 400 iterations, setting the decay factor to 0.995.

C.3 DEEP IMAGE PRIOR (DIP):

Given an input measurement y, a generator G φ parameterized by random parameters φ, and a random latent code z, the reconstruction G φ * (z) is obtained by resolving the following optimization problem:

φ * = arg min φ y -G φ (z) 2 2 (16)

For measurement processes Patch-Band, Remove-Pixel and Remove-Pixel-Channel (refer to Section 4.2), the resulting reconstruction G φ * (z) was not satisfactory: G was consistently regressing to the corrupted values in the measurement y, which led to unsatisfactory results. Instead of presenting these results, we have chosen to remove the contribution of the error terms where the measurement process induced null values from objective ( 16), and present the latter instead. However, this assumes access to the true value θ that corrupted datum y: y = F (x; θ). Formally, we resolve the following objective: Where F acts as a mask, and eliminates the terms associated to the pixels from y that have been put to 0. Note that this method corresponds to the inpainting formulation in Ulyanov et al. (2017) . We used the implementation provided by the authors4 . 

D ADDITIONAL SAMPLES:



:

In practice we optimize -log D(ŷi) instead of log(1 -D(ŷi))

https://dmitryulyanov.github.io/deep_image_prior

Figures and Tables:

Figure 1 -The Figure illustrates the dependencies between the variables considered for handling the likelihood term when solving (4). The likelihood term in (4) can be replaced by the expectation of 1 2σ 2 y -F (G(y); θ) 2 2 (see Equation7). To compute this expectation, one first simulates an observation y from a signal x using F (x; θ) , then generates x = G(y) and ỹ = F (x; θ), as in the Figure.This allows us to compute the MSE term in the above expression.

2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

2 m Z E P w l k 9 e h d Z F 1 b P c u K z U b v I 4 i n A C p 3 A O H l x B D e 6 h D k 1 g g P A M r / D m P D o v z r v z s W g t O P n M M f y R 8 / k D m / e M y w = = < / l a t e x i t > F < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 y v i 8 g S 7 b S a q Q j U N S m l 2 7 H 5 N / t 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y F M R j C / Y D 2 l A 2 2 0 m 7 d r M J u x u h h P 4 C L x 4 U 8 e p P 8 u a / c d v m o K 0 v L D y 8 M 8 P O v E E i u D a u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g

Figure 2 -The figure illustrates the dependencies of the variables used for dealing with the prior term in (4). An observation y is sampled, and then transformed by the generative network into a reconstructed signal x = G(y). One then simulates a measurement ŷ := F (x; θ) from this reconstruction. We then enforce the distributions of observations p Y and simulated measurements p G Y to be similar using an adversarial loss. In order to produce indistinguishable distributions, the generator G has to remove the corruption and recover a sample x from p X .

< l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H

r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H

r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H

r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " k B W p E t b V W 3 x l z 0 t 9 E J b d P D f c Z n 0 = " > A A A B 6 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I n k o i g h 6 L X j y 2 Y D + g D W W z n b R r N 5 u w u x F C 6 C / w 4 k E R r / 4 k b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y S C a + O 6 3 8 7 a + s b m 1 n Z p p 7 y 7 t 3 9 w W D k 6 b u s 4 V Q x b L B a x 6 g Z U o + A S W 4 Y b g d 1 E I Y 0 C g Z 1 g c j e r d 5 5 Q a R 7 L B 5 M l 6 E d 0 J H

r p S a 8 8 X M u k 9 S g Z I u P w l Q Q E 5 P Z 1 W T I F T I j M g u U K W 5 3 J W x M F W X G Z l O 2 I X j L J 6 9 C + 7 L m W W 5 e V e u 3 R R w l O I U z u A A P r q E O 9 9 C A F j B A e I Z X e H M e n R f n 3 f l Y t K 4 5 x c w J / J H z + Q P n v 4 z 9 < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k

7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k

7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k z D Q K

7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M + L v H / A I / N d 5 o o + o j M 2 S L m 0 r c Y = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 A N p T N d t M u 3 W z C 7 k Q s o T / C i w d F v P p 7 v P l v 3 L Q 5 a O s L C w / v z L A z b 5 B I Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s 4 1 Y y 3 W C x j 3 Q 2 o 4 V I o 3 k K B k n c T z W k U S N 4 J J r d 5 v f P I t R G x e s B p w v 2 I j p Q I B a N o r U 5 / T D F 7 m g 2 q N b f u z k V W w S u g B o W a g + p X f x i z N O I K m a T G 9 D w 3 Q T + j G g W T f F b p p 4 Y n l E 3 o i P c s K h p x 4 2 f z d W f k z D p D E s b a P o V k 7 v 6 e y G h k z D Q K

7 z C m 5 M 4 L 8 6 7 8 7 F o L T n F z D H 8 k f P 5 A 7 A + j 8 k = < / l a t e x i t > ŷ < l a t e x i t s h a 1 _ b a s e 6 4 = " V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G nG F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s 1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G nG F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s 1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b E 6 h G z h P s R H S s R C k b R W t 3 B h G K e z Y b V m l t 3 F y L r 4 B V Q g 0 K t Y f V r M I p Z G nG F T F J j + p 6 b o J 9 T j Y J J P q s M U s M T y q Z 0 z P s W F Y 2 4 8 f P F u j N y Y Z 0 R C W N t n 0 K y c H 9 P 5 D Q y J o s C 2 x l R n J j V 2 t z 8 r 9 Z P M b z x c 6 G S F L l i y 4 / C V B K M y f x 2 M h K a M 5 S Z B c q 0 s L s S N q G a M r Q J V W w I 3 u r J 6 9 C 5 q n u W H 6 5 r z d s i j j K c w T l c g g c N a M I 9 t K A N D K b w D K / w 5 i T O i / P u f C x b S 0 4 x c w p / 5 H z + A L H D j 8 o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V i 9 5 Y w k n F r F z c B 5 L y q g i Y S o M f 0 U = " > A A A B 7 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j B f s B b S i b 7 a Z d u t m E 3 Y k Q Q n + E F w + K e P X 3 e P P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u l j c 2 t 7 Z 3 y b m V v / + D w q H p 8 0 j F x q h l v s 1 j G u h d Q w 6 V Q v I 0 C J e 8 l m t M o k L w b T O / m 9 e 4 T 1 0 b

4 8 7 b 1 4 7 9 7 H o r X g 5 T P H 8 E f e 5 w + j + Y 8 o < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 f S / b p r + s s m e y t 6 S N 2 8 Z C U b o k h c = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g m 2 V N p T N Z t I u 3 W z C 7 k Y s o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L A z b 5 A K r o 3 r f j u l l d W 1 9 Y 3 y Z m V r e 2 d 3 r 7 p / 0 N Z J p h i 2 W C I S d R 9 Q j Y J L b B l u B N 6 n C m k c C O w E o + t p v f O I S v N E 3 p l x i n 5 M B 5 J H n F F j r Y e e 4 S L E / G n S r 9 b c u j s T W Q a v g B o U a v a r X 7 0 w Y V m M 0 j B B t e 5 6 b m r 8 n C r D m c B J p Z d p T C k b 0Q F 2 L U o a o / b z 2 c I T c m K d k E S J s k 8 a M n N / T + Q 0 1 n o c B 7 Y z p m a o F 2 t T 8 7 9 a N z P R p Z 9 z m W Y G J Z t / F G W C m I R M r y c h V 8 i M G F u g T H G 7 K 2 F D q i g z N q O K D c F b P H k Z 2 m d 1 z / L t e a 1 x V c R R h i M4 h l P w 4 A I a c A N N a A G D G J 7 h F d 4 c 5 b w 4 7 8 7 H v L X k F D O H 8 E f O 5 w 9 D X 5 C y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 f S / b p r + s s m e y t 6 S N 2 8 Z C U b o k hc = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g m 2 V N p T N Z t I u 3 W z C 7 k Y s o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L A z b 5 A K r o 3 r f j u l l d W 1 9 Y 3 y Z m V r e 2 d 3 r 7 p / 0 N Z J p h i 2 W C I S d R 9 Q j Y J L b B l u B N 6 n C m k c C O w E o + t p v f O I S v N E 3 p l x i n 5 M B 5 J H n F F j r Y e e 4 S L E / G n S r 9 b c u j s T W Q a v g B o U a v a r X 7 0 w Y V m M 0 j B B t e 5 6 b m r 8 n C r D m c B J p Z d p T C k b 0 Q F 2 L U o a o / b z 2 c I T c m K d k E S J s k 8 a M n N / T + Q 0 1 n o c B 7 Y z p m a o F 2 t T 8 7 9 a N z P R p Z 9 z m W Y G J Z t / F G W C m I R M r y c h V 8 i M G F u g T H G 7 K 2 F D q i g z N q O K D c F b P H k Z 2 m d 1 z / L t e a 1 x V c R R h i M4 h l P w 4 A I a c A N N a A G D G J 7 h F d 4 c 5 b w 4 7 8 7 H v L X k F D O H 8 E f O 5 w 9 D X 5 C y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 f S / b p r + s s m e y t 6 S N 2 8 Z C U b o k h c = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g m 2 V N p T N Z t I u 3 W z C 7 k Y s o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L A z b 5 A K r o 3 r f j u l l d W 1 9 Y 3 y Z m V r e 2 d 3 r 7 p / 0 N Z J p h i 2 W C I S d R 9 Q j Y J L b B l u B N 6 n C m k c C O w E o + t p v f O I S v N E 3 p l x i n 5 M B 5 J H n F F j r Y e e 4 S L E / G n S r 9 b c u j s T W Q a v g B o U a v a r X 7 0 w Y V m M 0 j B B t e 5 6 b m r 8 n C r D m c B J p Z d p T C k b 0 Q F 2 L U o a o / b z 2 c I T c m K d k E S J s k 8 a M n N / T + Q 0 1 n o c B 7 Y z p m a o F 2 t T 8 7 9 a N z P R p Z 9 z m W Y G J Z t / F G W C m I R M r y c h V 8 i M G F u g T H G 7 K 2 F D q i g z N q O K D c F b P H k Z 2 m d 1 z / L t e a 1 x V c R R h i M 4 h l P w 4 A I a c A N N a A G D G J 7 h F d 4 c 5 b w 4 7 8 7 H v L X k F D O H 8 E f O 5 w 9 D X 5 C y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 f S / b p r + s s m e y t 6 S N 2 8 Z C U b o k h c = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g m 2 V N p T N Z t I u 3 W z C 7 k Y s o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L A z b 5 A K r o 3 r f j u l l d W 1 9 Y 3 y Z m V r e 2 d 3 r 7 p / 0 N Z J p h i 2 W C I S d R 9 Q j Y J L b B l u B N 6 n C m k c C O w E o + t p v f O I S v N E 3 p l x i n 5 M B 5 J H n F F j r Y e e 4 S L E / G n S r 9 b c u j s T W Q a v g B o U a v a r X 7 0 w Y V m M 0 j B B t e 5 6 b m r 8 n C r D m c B J p Z d p T C k b 0 Q F 2 L U o a o / b z 2 c I T c m K d k E S J s k 8 a M n N / T + Q 0 1 n o c B 7 Y z p m a o F 2 t T 8 7 9 a N z P R p Z 9 z mW Y G J Z t / F G W C m I R M r y c h V 8 i M G F u g T H G 7 K 2 F D q i g z N q O K D c F b P H k Z 2 m d 1 z / L t e a 1 x V c R R h i M 4 h l P w 4 A I a c A N N a A G D G J 7 h F d 4 c 5 b w 4 7 8 7 H v L X k F D O H 8 E f O 5 w 9 D X 5 C y < / l a t e x i t > ỹ < l a t e x i t s h a 1 _ b a s e 6 4 = " T d k k W 5 f p M Z r c a d 8 M X R D z r R q Y X l c = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 Q N p T N Z t M u 3 W z C 7 k Q o o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L Az b 5 B K Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s k 0 4 y 3 W C I T 3 Q 2 o 4 V I o 3 k K B k n d T z W k c S N 4 J x r e z e u e J a y M S 9 Y C T l P s x H S o R C U b R W o 9 9 F D L k + W Q 6 q N b c u j s X W Q W v g B o U a g 6 q X / 0 w Y V n M F T J J j e l 5 b o p + T j U K J v m 0 0 s 8 M T y k b 0 y H v W V Q 0 5 s b P 5 w t P y Z l 1 Q h I l 2 j 6 F Z O 7 + n s h p b M w k D m x n T H F k l m s z 8 7 9 a L 8 P o 2 s + F S j P k i i 0 + i j J J M C G z 6 0 k o N G c o J x Y o 0 8 L u S t i I a s r Q Z l S x I X j L J 6 9 C + 6 L u W b 6 / r D V u i j j K c A K n c A 4 e X E E D 7 q A J L W A Q w z O 8 w p u j n R f n 3 f l Y t J a c Y u Y Y / s j 5 / A F E 5 J C z < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T d k k W 5 f p M Z r c a d 8 M X R D z r R q Y X l c = " > A A A B 8 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g v 2 Q N p T N Z t M u 3 W z C 7 k Q o o b / C i w d F v P p z v P l v 3 L Y 5 a O s L C w / v z L A z b 5 B K Y d B 1 v 5 3 S 2 v r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s k 0 4 y 3 W C I T 3 Q 2 o 4 V I o 3 k K B k n d T z W k c S N 4 J x r e z e u e J a y M S 9 Y C T l P s x H S o R C U b R W o 9 9 F D L k + W Q 6 q N b c u j s X W Q W v g B o U a g 6 q X / 0 w Y V n M F T J J j e l 5 b o p + T j U K J v m 0 0 s 8 M T y k b 0 y H v W V Q 0 5 s b P 5 w t P y Z l 1 Q h I l 2 j 6 F Z O 7 + n s h p b M w k D m x n T H F k l m s z 8 7 9 a L 8 P o 2 s + F S j P k

r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s k 0 4 y 3 W C I T 3 Q 2 o 4 V I o 3 k K B k n d T z W k c S N 4 J x r e z e u e J a y M S 9 Y C T l P s x H S o R C U b R W o 9 9 F D L k + W Q 6 q N b c u j s X W Q W v g B o U a g 6 q X / 0 w Y V n M F T J J j e l 5 b o p + T j U K J v m 0 0 s 8 M T y k b 0 y H v W V Q 0 5 s b P 5 w t P y Z l 1 Q h I l 2 j 6 F Z O 7 + n s h p b M w k D m x n T H F k l m s z 8 7 9 a L 8 P o 2 s + F S j P k

r G 5 l Z 5 u 7 K z u 7 d / U D 0 8 a p s k 0 4 y 3 W C I T 3 Q 2 o 4 V I o 3 k K B k n d T z W k c S N 4 J x r e z e u e J a y M S 9 Y C T l P s x H S o R C U b R W o 9 9 F D L k + W Q 6 q N b c u j s X W Q W v g B o U a g 6 q X / 0 w Y V n M F T J J j e l 5 b o p + T j U K J v m 0 0 s 8 M T y k b 0 y H v W V Q 0 5 s b P 5 w t P y Z l 1 Q h I l 2 j 6 F Z O 7 + n s h p b M w k D m x n T H F k l m s z 8 7 9 a L 8 P o 2 s + F S j P k

Figure 4 -Model reconstructions for different corruption processes, on CelebA. Each row corresponds to a specific corruption process, and each column to a particular model.

Figure 5 -On the top row, randomly sampled test set measurements from CelebA corrupted using Patch-Band(h = 20), and below, our associated reconstructions.

Figure 6 -On the top row, randomly sampled test set measurements from CelebA corrupted using Remove-Pixel(p = 0.95), and below, our associated reconstructions.

Figure 7 -On the top row, randomly sampled test set measurements from LSUN corrupted using Patch-Band(h = 20), and below, our associated reconstructions.

Figure8-On the top row, randomly sampled test set measurements from Recipe-1M corrupted using Remove-Pixel (p = 0.9), and below, our associated reconstructions.

The general measurement process in equation (1), Y = F (X; Θ) + E induces log p Y |X,Θ (y|G(y), θ) to yield a simple analytic expression: log p(y|G(y), θ) = -1 2σ 2 y -F (G(y); θ)

The likelihood term E p Y log p Y |X (y|G(y)) can then be replaced by-E p Θ p X p Y |X,Θ 1 2σ 2 y -F (G(y); θ) 2 2 in objective (4). This is because the constant c 2 does not change the objective.

(y -G φ (z); θ)

Figure 11 -Baseline comparison for the CelebA dataset. Corruption is Remove-Pixel (p = 0.95).

Figure 15 -On the top row, randomly sampled test set images from LSUN. Below, associated couples of corrupted observations and subsequent reconstructions from our model. From top to bottom, corruptions are Remove-Pixel-Channel(p = 0.95), Remove-Pixel(p = 0.90), Patch-Band(h = 20), Convnoise(σ C = 0.15, l = 3).

Figure 16 -On the top row, randomly sampled test set images from Recipe. Below, associated couples of corrupted observations and subsequent reconstructions from our model. From top to bottom, corruptions are Remove-Pixel-Channel(p = 0.95), Remove-Pixel(p = 0.90), Patch-Band(h = 20), Convnoise(σ C = 0.15, l = 3).

Figure 17 -Additional samples from our model of the LSUN Bedrooms dataset. From top to bottom, corruptions are Convnoise(σ C = 0.15, l = 3), Patch-Band(h = 20), Remove-Pixel-Channel(p = 0.90) and Remove-Pixel(p = 0.95). 20

Figure 18 -Additional sample from our model, on the Recipe dataset. From top to bottom, corruptions are Convnoise(σ C = 0.3, l = 5), Patch-Band(h = 20), Remove-Pixel-Channel(p = 0.90) and Remove-Pixel(p = 0.90).

: Average mean square error of neural network based models on the test set of CelebA, for different measurement processes. The first two rows are model trained with no supervision, the last two row with additional supervision.

Table content: <table><tr><td/><td>Remove-</td><td>Remove-</td><td>Patch-Band</td><td>Convolve-</td></tr><tr><td/><td>Pixel</td><td>Pixel-</td><td/><td>Noise</td></tr><tr><td/><td/><td>Channel</td><td/><td/></tr><tr><td>Conditional AmbientGan</td><td>0.292</td><td>0.2829</td><td>0.1421</td><td>0.0814</td></tr><tr><td>Our Model</td><td>0.0414</td><td>0.0409</td><td>0.0165</td><td>0.0088</td></tr><tr><td>Unpaired Variant</td><td>0.037</td><td>0.0336</td><td>0.034</td><td>0.0103</td></tr><tr><td>Paired Variant</td><td>0.0383</td><td>0.0401</td><td>0.0147</td><td>0.0084</td></tr></table>

) to E p Θ p X p Y |X,Θ log p Y |X,Θ (y|G(y), θ) + c 1 , with c 1 constant w.r.t. G: For all x, (in particular for G(y)), if X and Θ are independent, the log-likelihood can be decomposed as:log p Y |X (y|x) = log p Y,Θ|X (y, θ|x) -log p Θ|X,Y (θ|x, y) X⊥ ⊥Θ = log p Y |X,Θ (y|x, θ) + log p Θ (θ) -log p Θ|Y (θ|y)(12)Applying the expectation w.r.t to the joint p Y,Θ on both sides, we obtainE p Y log p Y |X (y|x) = E p Y,Θ log p Y |X,Θ (y|x, θ) + log p Θ (θ) -log p Y,Θ (θ|y) = E p Y,Θ log p Y |X,Θ (y|x, θ) + c 1 (13)The terms log p Θ (θ) and log p Θ|Y (θ|y) do not depend on x, hence c 1 is a constant w.r.t. x. Plugging back G(y) in place of x and applying the law of total probabilities w.r.t. X on the right hand side, we obtain: E p Y log p Y |X (y|G(y)) = E p Θ p X p Y |X,Θ log p Y |X,Θ (y|G(y), θ) + c 1 (14)

Table content: <table/>

Peer review (iclr2019):
 This is a very interesting paper that achieves something that seems initially impossible: to learn to reconstruct clear images from only seeing noisy or blurry images. The paper builds on the closely related prior work AmbientGAN which shows that it is possible to learn the *distribution* of uncorrupted samples using only corrupted samples, again a very surprising finding. However, AmbientGAN does not try to reconstruct a single image, only to to learn the clear image distribution. The key idea that makes this is possible is knowledge of the statistics of the corruption process: the generator tries to create images that *after they have been corrupted* they look indistinguishable from real corrupted images. This surprisingly works and provably recovers the true distribution under a very wide set of corruption distributions, but tells us nothing about reconstructing an actual image from measurements. Given access to a generative model for clear images, an image can be reconstructed from measurements by maximizing the likelihood term. This method (CS-GAN) was introduced by Bora et al. in 2017. Therefore one approach to solve the problem that this paper tackles is to first use AmbientGAN to get a generative model for clear images and then use CS-GAN using the learned GAN. If I understand correctly, this is the *Conditional AmbientGAN* approach that is used as a baseline. This is a sensible approach given prior work. However, the authors show that their method (*Unpaired Supervision*) performs significantly better compared to the Conditional AmbientGAN baseline. This is very surprising and interesting to me. Please discuss this a bit more ? As far as I understand the proposed method is a merging of AmbientGAN and CS-GAN, but much better than the naive separation. Could you give a bit more intuition on why ? I would like to add also that the authors can use their approach to learn a better AmbientGAN. After getting their denoised images, these can be used to train a new AmbientGAN, with cleaner images as input , which should be even better no ? In the appendix where is the proposed method in fig 5- 8 ? Does the proposed method outperform Deep Image Prior ?