UNSUPERVISED ADVERSARIAL IMAGE RECONSTRUCTION
Year: 2019

Abstract:
We address the problem of recovering an underlying signal from lossy, inaccurate observations in an unsupervised setting. Typically, we consider situations where there is little to no background knowledge on the structure of the underlying signal, no access to signal-measurement pairs, nor even unpaired signal-measurement data. The only available information is provided by the observations and the measurement process statistics. We cast the problem as finding the maximum a posteriori estimate of the signal given each measurement, and propose a general framework for the reconstruction problem. We use a formulation of generative adversarial networks, where the generator takes as input a corrupted observation in order to produce realistic reconstructions, and add a penalty term tying the reconstruction to the associated observation. We evaluate our reconstructions on several image datasets with different types of corruptions. The proposed approach yields better results than alternative baselines, and comparable performance with model variants trained with additional supervision.

heading: 1 INTRODUCTION
text: Many real world applications require acquiring information about the state of some physical system from incomplete and inaccurate measurements. For example, in infrared satellite imagery, one has to deal with the presence of clouds and a variety of other external factors perturbing the acquisition of temperature maps. This raises questions on how to recover the correct information and eliminate the contribution of external factors hindering the overall signal acquisition.
In this context, signal recovery does not usually yield a unique solution, meaning that multiple signal reconstructions could trivially explain the measurements. For the above example, different missing temperature values could accurately explain the observations. To cope with this indeterminacy, one usually relies on prior information on the structure of the true signal in order to constrain the reconstruction to plausible solutions (Stuart (2010)). A common approach is to use handcrafted, analytically tractable priors (Candès et al. (2005), Mota et al. (2017)). This approach is limited to situations for which the underlying signal structure can be easily described, which are rarely observed in the wild.
Recent developments in generative models parameterized by neural networks (Goodfellow et al. (2014), Kingma & Welling (2013), Dinh et al. (2016)) offer a promising statistical approach to signal recovery, for which priors on the signal are not handcrafted anymore, but learned from large amounts of data. Despite exhibiting interesting results (Bora et al. (2017), Mardani et al. (2017), Ledig et al. (2016)), these methods all require some form of supervision, either observation measurement-signal pairs, or at least unpaired samples from observations and underlying signals. For many practical problems, obtaining these samples is too expensive and/or impractical, which makes these approaches not suitable for such situations.
We address the problem of image reconstruction in an unsupervised setting, when only corrupted observations are available, together with some prior information on the nature of the measurement process.
∗equal contribution
The learning problem is formulated as finding the maximum a posteriori estimate of signals given their measurements on the training set. We derive a natural objective for our reconstruction network, composed of a linear combination of an adversarial loss for recovering realistic signals, and a reconstruction loss to tie the reconstruction to its associated observation (Section 2.2). This model is evaluated and compared to baselines on 3 image datasets, CelebA (Liu et al. (2015)), LSUN Bedrooms (Yu et al. (2015)), Recipe-1M (Marin et al. (2018)), where we experiment with different types of measurement processes corrupting the images.
Our contributions are:
• A novel, computationally efficient framework for dealing with large scale signal recovery in an unsupervised context, applicable to a wide range of situations,
• A model and a new way of training a deep learning architecture for implementing this framework,
• Extensive evaluations on a number of image datasets with different measurement processes.

heading: 2 PRELIMINARIES
text: Notations. We use capital letters (e.g. X) for random variables, and lower-case letters (e.g. x) for their values. pX(x) denotes the distribution (or its density in the appropriate context) ofX evaluated at x.

heading: 2.1 PROBLEM SETTING.
text: Suppose there exists a signal X ∼ pX we wish to acquire, but we only have access to this signal through lossy, inaccurate measurements Y ∼ pY . The measurement process is modeled through a stochastic operator F mapping signals X to their associated observations Y . We will refer to F as the measurement process, which corrupts the input signal. F is parameterized by a random variable Θ ∼ pΘ following an underlying distribution pΘ we can sample from, which represents the factors of corruption. Thus, given a specific signal x, we can simulate its measurement by first sampling θ from pΘ, and then computing F (x; θ). Additional sources of uncertainty, e.g. due to unknown factors, can be modeled using additive i.i.d. Gaussian noise E ∼ N (0, σ2I), so that the overall observation model becomes:
Y = F (X; Θ) + E (1)
F is assumed to be differentiable w.r.t. its first argumentX , and Θ andX to be independent (denoted X ⊥ Θ). Different instances of F will be considered (refer to Section 4.2), like random occlusions, information acquisition from a sparse subset of the signal, overly smoothing out and corrupting the original distribution with additive noise, etc. In such cases, the factors of corruption Θ might respectively represent the position of the occlusion, the coordinates of the acquired information, or simply the values of the additive noise.

heading: 2.2 APPROACH
text: Given an observation y, our objective is to find a signal x̂ as close as possible to the associated true signal x. From a probabilistic viewpoint, it is natural to formulate the problem as finding the maximum a posteriori (MAP) estimate, which consists in selecting the most probable signal x∗ under the posterior distribution pX|Y (·|y):
x∗ = arg max x log pX|Y (x|y) (2)
or equivalently:
x∗ = arg max x log pY |X(y|x) + log pX(x) (3)
where pY |X(y|x) is the likelihood of the signal x given observation y, and pX(x) is the prior probability evaluated at x. Therefore, a good reconstruction must be likely to have generated the data, i.e. yield high likelihood, and look realistic, i.e. yield high probability under the prior.
In the general case, calculating the likelihood term pY |X(y|x) requires marginalizing over noise parameters Θ and this does not yield an analytic form. As for the prior pX(x), it is unknown, and we have no access to samples fromX since we are in an unsupervised setting: there is then no direct way to estimate pX either. In the general case considered here, with no assumption on the form of the distributions, solving Equation (3) is up to our knowledge an open problem.
In the following sections, we will introduce an approach to deal with the likelihood term (Section 3.1), and the unknown prior term (Section 3.2) in order to provide an approximate solution to equation (3) (Section 3.3). For that, we will formulate the problem as learning a mapping G : Y → X that links each measurement y to its associated MAP estimate x∗ on the training set. The associated objective is then:
G∗ = arg max G
EpY { log pY |X(y|G(y)) + log pX(G(y)) }
(4)
Which is obtained by plugging G(y) = x into equation 3 and taking the expectation w.r.t. the distribution of observations pY .

heading: 3 METHOD
text: From Equation (4), we see that a valid reconstruction mapping G must yield high probability for the likelihood and the prior. This will guide the design of an appropriate objective during the following section, where the reconstruction mapping G will be implemented using a neural network.
3.1 HANDLING THE LIKELIHOOD TERM
In the general case, evaluating the likelihood pY |X(y|x) in equation (3) requires marginalizing on the unobserved noise variable Θ: pY |X(y|x) = EpΘpY |X,Θ(y|x, θ), which involves computing an intractable integral. Most probabilistic model for image denoising make assumptions on the structure of the measurement operator F (.,Θ) and on the distribution of Θ in order to obtain an analytic form for the expectation (Boyat & Joshi (2015), Alkinani & El-Sakka (2017)). Here, we consider more general measurement operators which do not necessarily lead to such a simplification and therefore proceed in a different way.
We outline below, the main steps of the method for handling the likelihood term pY |X(y|x) in equation (4). The complete derivation is provided in Appendix A.
1. Making use of the independence between X and Θ, the expectation term EpY log pY |X(y|G(y)) in equation (4) can be rewritten as :
EpΘpXpY |X,Θ log pY |X,Θ(y|G(y), θ) + c1 (5)
, with c1 constant w.r.t. G.
2. The general measurement process described in equation (1) induces log pY |X,Θ(y|G(y), θ) to yield a simple analytic expression:
log p(y|G(y), θ) = − 1 2σ2 ‖y − F (G(y); θ)‖22 + c2 (6)
with c2 a constant.
3. The likelihood term EpY log pY |X(y|G(y)) can then be replaced in objective (4) by
− EpΘpXpY |X,Θ 1
2σ2 ‖y − F (G(y); θ)‖22 (7)
Equation (7) shows that the likelihood term can be evaluated by first sampling a measurement y conditioned on a corruption parameter θ and signal x, and then constrainG such that ‖y − F (G(y); θ)‖22 is close to zero. Note that in this expression, the same parameter θ is used for simulating ỹ from x̃ and y from x (see Figure 1 and section 3.3 for more details). Unfortunately, this requires first sampling x from the signal distribution pX which is unknown. In the following sections, we will see how we work around this problem.
3.2 HANDLING THE PRIOR TERM
Maximizing w.r.t. the prior term pX(G(y)) in equation (4) is similar to learning a mapping G, such that the distribution induced by G(y), EpY pX(G(y)) is close to the distribution pX . The prior pX being unknown, the only sources of information are the lossy measurements y and the known prior pΘ on the measurement process. In order to learn an approximation of the true prior pX , we will use a form of generative adversarial learning, and build on an idea introduced in the AmbientGAN model by Bora et al. (2018).
AmbientGAN aims at learning an unconditional generative model G of the true signal distribution pX , when only lossy measurements y of the signal are available together with a known stochastic measurement operator F . In AmbientGAN, a generator is trained to produce uncorrupted signal samples from a latent code so that the generated signals when corrupted are indistinguishable from
the observation measurements. In Bora et al. (2018), the authors show that for some families of noise distributions pΘ, the generator’s induced distribution matches the signal’s true distribution. Note that even if the generation process of the observations y in AmbientGAN is similar to the one considered in this paper (see Section 2.1), the objective is however different: when the aim of AmbientGAN is to learn a distribution of the underlying signal by sampling a latent space, ours is to reconstruct corrupted signals.
In order for G to produce uncorrupted signals we will use an approach inspired from AmbientGAN, as illustrated in Figure 2. Given an observation y, one wants to reconstruct a latent signal approximation x̂ = G(y) so that a corrupted version of this signal ŷ = F (x̂) will have a distribution indistinguishable from the one of the observations y. The generator G and a discriminator D are trained on observations y and generated samples ŷ. The corresponding loss is the following1:
Lprior(G) := max D EY∼pY ,Ŷ∼pGY { logD(y) + log ( 1−D(ŷ) )} (8)
where pGY corresponds to the distribution induced by G’s corrupted outputs (ŷ in Figure 2) , i.e. pGY (y) := EpΘpGX { p(y|x, θ) } and pGX denotes the marginal distribution induced by G’s outputs (x̂ in Figure 2): pGX(x) := EpY p G X|Y (x|y) = EpY δ ( x−G(y) ) 2. This penalty enforces the marginal pGX to be close to the true prior distribution pX , and thus forces G to map its input measurements onto pX .
3.3 PUTTING EVERYTHING TOGETHER
In Section 3.1, we have shown that it is possible to maximize the average log-likelihood term in equation (4), given that we can sample from the unknown prior distribution pX . In Section 3.2, we have shown how it is possible to enforce the generator to produce samples from pX , without ever having access to uncorrupted samples. The idea is then to use the distribution induced by the generator’s output pGX as a proxy for pX to compute an approximate value of the expectation in equation (5): This gives us the following penalty term (see appendix A):
Llikeli(G) := EpΘpGX ,Ŷ∼pY |X,Θ ‖ŷ − F (G(ŷ); θ)‖ 2 2 (9)
1the min term of the adversarial loss will be introduced later, see Equation (10) 2δ(x) is the Dirac delta function, which is equal to zero everywhere except in x.
The full objective is a linear combination of penalties (8) and (9):
arg min G
Lprior(G) + λ · Llikeli(G) (10)
As illustrated by the dependencies highlighted in Figure 2, in the process of minimizing Lprior, we sample from the marginal likelihood pGY (y) := EpΘpGX
{ p(y|x, θ) } . The expectancy in the likelihood
term Llikeli is precisely computed w.r.t. this distribution. We can then use the same samples in order to minimize the full objective (10). This gives us the Algorithm 1 described below, along with the dependency structure illustrated in Figure 3.
Algorithm 1 Training Procedure. Require: Initialize parameters of the generator G and the discriminator D.
while (G,D) not converged do Sample {yi}1≤i≤n from data distribution pY Sample {θi}1≤i≤n from PΘ Sample {εi}1≤i≤n from PE Set ŷi to F (G(yi), θi) + εi for 1 ≤ i ≤ n Update D by ascending:
1
n n∑ i=1 logD(yi) + log(1−D(ŷi))
Update G by descending:
1
n n∑ i=1 λ · ‖ŷi − F (G(ŷi); θi)‖22 + log(1−D(ŷi))3
end while

heading: 4 EXPERIMENTS

heading: 4.1 MODEL ARCHITECTURES AND DATASETS
text: Architectures. We will briefly describe the architectures, additional details on architectures and hyperparameters can be found in appendix B. Our network architectures are inspired by the GAN architecture in Zhang et al. (2018). We use the same discriminator, and we propose an image-toimage variant of their latent-to-image generator for the reconstruction network G.
Datasets. We evaluate our approach using three different image datasets :
• CelebA. Dataset of celebrities, containing approximately 200 000 samples. As Bora et al. (2018), the images are center-cropped.
• LSUN Bedrooms. Dataset of bedrooms, containing 3 million samples. • Recipe-1M. Dataset of cooked meals, containing approximately 600 000 samples.
All the images have been resized to 64× 64. In order to place ourselves in the most realistic setting possible, every image has been corrupted once, i.e. there is never multiple occurrences of an image corrupted with different corruption parameters.
We withhold 15% of the training set for validation, selected uniformly at random for each dataset.
3In practice we optimize − logD(ŷi) instead of log(1−D(ŷi))

heading: 4.2 CORRUPTIONS
text: Let us present the different measurement processes F used in the experiments, also named corruptions:
Remove-Pixel. This measurement process randomly samples a fraction p of pixels uniformly and sets the associated channel values to 0. All the corresponding channel values are set to 0.
Remove-Pixel-Channel. Instead of setting to 0 a pixel for all channels as in Remove-Pixel, one samples a pixel coordinate and a channel, and sets the corresponding value to 0.
Convolve-Noise. Here F (x; θ) := k ∗ x+ θ, where ∗ is the convolution operator and k is a mean filter of size l. For each pixel, noise θ sampled from a zero-mean Gaussian of variance σ2C is added to the previous result.
Patch-Band. A horizontal band of height h whose vertical position in the image is uniformly sampled from the set of possible positions. For each pixel falling inside the band, its associated value is set to 0. The resulting measurement for pixel at column i and row j can be summarized as:
F (x; θ)i,j := { 0, if j ∈ {θ, . . . , θ + h} xi,j , otherwise
(11)
where θ is uniformly sampled from {1, . . . ,H − h}, and H is the image height. In the experiments, h is set to 20.

heading: 4.3 BASELINES

heading: 4.3.1 CONDITIONAL AMBIENTGAN.
text: This is our only unsupervised baseline. The context is the same as for our model: the measurement process F is assumed known, there is no access to samples from the uncorrupted signal distribution pX , but only to their corrupted counterpart pY . This baseline is a combination of two recent techniques in the field of signal recovery: the aforementioned AmbientGan Bora et al. (2018) and CS-GAN Bora et al. (2017).
An unconditional generator G is trained using the AmbientGan framework (Bora et al. (2018)) for each type of measurement process F , in order to produce samples from pX (see Section 4.2). The distribution induced by the generator pGX is an approximation of pX (at the optimum, both distributions match, i.e. pGX = pX ). Given a specific measurement y, the reconstruction x̂ is the signal from G that is closest to y, as in Bora et al. (2017). To find x̂ = G(ẑ), we search for the latent code ẑ of G, such that ẑ = arg minz ‖y −G(z)‖ 2 2 +R(z). R(z) is a regularizing term that enforces the latent code to stay in G’s input domain. This objective is optimized using stochastic gradient descent. To train G, we use the same architectures and hyper-parameters as those provided by the authors. Because this approach may be sensitive to the initial latent code, we reiterate this approach three times and select the best resulting image.

heading: 4.3.2 UNPAIRED VARIANT.
text: This is a variant of our model where we have access to samples of the signal distribution pX . This means that although we have no paired samples from pX,Y , we have access to unpaired samples from pX and pY . This baseline is similar to our model but instead of discriminating between a measurement from the data y and a simulated measurement ŷ, we directly discriminate between samples x from the signal distribution and the output of the reconstruction network x̂. For a diagram describing the model, refer to appendix C.1.

heading: 4.3.3 PAIRED VARIANT.
text: This is a variant of our model where we have access to signal measurement pairs (y, x) from the joint distribution pY,X . Given input measurement y, the reconstruction is obtained by regressing y to the associated signal x using a MSE loss. In order to avoid blurry samples, we add an adversarial term in the objective in order to constrain G to produce realistic samples, as in Isola et al. (2016).
The model is trained using the same architectures as our model, and the hyperparameters have been found using cross-validation. For a diagram describing the model, refer to appendix C.2.

heading: 4.3.4 MEASUREMENT SPECIFIC BASELINES.
text: We also compare our model to baselines that where designed to remove specific corruptions.
Deep Image Prior (Ulyanov et al. (2017)). Given a generator Gφ parametrized by randomly initialized weights φ and a measurement y, this method seeks to find a reconstruction from Gφ that is close to the measurement. For corruptions processes Patch-Band, Remove-Pixel and RemovePixel-Channel, we assume the access to the θ associated to the observations in the data (i.e. in this case, the mask). For more details, please refer to appendix C.
Biharmonic Inpainting (Damelin & Hoang (2018)). By considering inpainting as a smooth surface extension domain, this baseline resolves a biharmonic equation to obtain a high order approximation of the image. This approximation is then extended to the missing part of the image. This method assumes access to the θ associated to the observations in the data (i.e. in this case, the mask).
Total Variation Denoising (Chambolle (2004)). This denoising baseline aims to minimize the total variation of an image i.e the integral of the absolute gradient of the image. Reducing the total variation of the image removes unwanted detail, such as white noise artifacts while preserving important details such as edges and corners.

heading: 5 RESULTS
text: We will now present our results. First, we compare quantitatively our model with non-measurement specific baselines on CelebA. We then present qualitative results with samples from our model and these baselines. Comparisons with measurement specific baselines are presented in appendix D for the three datasets.

heading: 5.1 QUANTITATIVE RESULTS
text: We compare our model with baselines introduced in the previous section. We report mean square error (MSE) scores between the reconstructed x̂ and the true signal x used to generate the input y. Table 1 shows the MSE computed on the test set, a randomly selected subset of CelebA comprised of 40000 images. Because the Conditional AmbientGan model is too computationally expensive, we only report the MSE on 40 randomly chosen samples of the test set.
Quantitatively, our model performs well. Except for the Conditional Ambiant GAN, all the methods are quite similar in terms of MSE. Our unsupervised model reaches performance similar to its variants trained using additional supervision. We also note that when the aligned signal-observation pairs are not used (as in our Unpaired Variant), results are comparable – sometimes better – than when these pairs are used directly (as in our Paired Variant). This suggests that our likelihood term is sufficient to condition the reconstruction on the input signal.

heading: 5.2 QUALITATIVE RESULTS
text: However, quantitative results gives us only partial information. We now evaluate the quality of our reconstruction on three different datasets (Section 4.1). Figure 4 shows reconstructions obtained from different models on the CelebA dataset.We observe that Conditional AmbientGAN yields visually poor results, especially for the Remove-Pixel and Remove-Pixel-Channel measurement processes. We hypothesize that this is due to the large Euclidean distance between the measurements and the associated signals, and the suboptimality of the generator. Visually, the quality of our model’s reconstructions are coherent with the quantitative results: they are comparable to its paired and unpaired counterparts (Section 4.3). Figures (5), (6), (7), and (8) each show reconstructions from a given measurement processe on different datasets. Our model is able to produce images with good visual quality while remaining coherent with the underlying uncorrupted images. In Figures (11), (12), (13) and (14) in appendix D, we compare our model with commonly used inpainting or denoising methods. We can see that contrary to these methods, we are able to capture semantic information from the dataset. Typically, in Figure 14, the model infers missing eyes or noses, without ever having seen them. Additional samples are available in the appendix D, refer to Figures (15), (16), (17), and (18).

heading: 6 RELATED WORK
text: To our knowledge, there is no other Deep Learning approach attempting to solve the unsupervised signal reconstruction problem. However, some of the ideas developed here are close to or even inspired from recent work.
In order to enforce high likelihood, we incorporate a penalty in our objective that is similar to the Cycle Consistency loss, used in several contexts (Zhu et al. (2017), Lample et al. (2017), Almahairi et al. (2018)). This constraint is used to learn from unpaired data sets. Moreover, they too use adversarial training to constrain the marginal distribution induced by the generator.
In the context of image super resolution, (Ledig et al. (2016), Sønderby et al. (2016), Mardani et al. (2017)) attempt to retrieve maximum a posteriori estimates of the super resolution image conditioned on an input image. They too use a generative model of the signal trained in an adversarial fashion using samples from signal distribution to constrain their reconstructions. Their approach is fully supervised.
Other works attempt to solve ill-posed inverse problems using generative models (Bora et al. (2017), Asim et al. (2018), Tripathi et al. (2018), Van Veen et al. (2018)). The general approach in all these papers consists to first train a generative model on the uncorrupted signal distribution. Then, given a measurement from which we wish to reconstruct the signal, it is inverted by finding the latent input code that generated the uncorrupted image, by minimizing the mean square error between the corrupted reconstruction and the measurement. This requires solving an optimization problem for each image, which takes several minutes (Ulyanov et al. (2017)) on GPU, and requires random restarts to avoid falling a bad local minima. Again, the setting is fully supervised.
Finally Lehtinen et al. (2018) propose a method for denoising images without direct supervision. They train a network to regress a corrupted image to the same image with a different corruption value. Assuming the corruption has zero-mean, their network learns to remove the corruption by the conditional expectation. This setting implicitly assumes access to the distribution of uncorrupted images in order to generate different noisy versions of the same image, which is not our case.

heading: 7 CONCLUSION
text: We have proposed a general formulation to recover a signal from lossy measurements using a neural network, without having access to uncorrupted signal data. We have formulated the problem as
finding a maximum a posteriori estimate of the signal given its observation, for all observations in the training set. This gives us a natural objective for our neural network, composed of a linear combination of an adversarial loss for recovering realistic signals, and a reconstruction loss to tie the reconstruction to its associated observation. Our approach yields results superior to the baselines, while staying competitive with other model variants that have access to higher forms of supervision.
For future work, we plan to apply our framework to different corruption processes, and evaluate our model’s performance in real world settings, specifically for retrieving uncorrupted scientific data. Another interesting research direction would be to make our reconstruction network stochastic, in order to approximate the true posterior of the signal given the measurement, and to obtain uncertainty estimates.

heading: ACKNOWLEDGMENTS
text: This work was partially funded by ANR project LOCUST - ANR-15-CE23-0027 and by CLEAR - Center for LEArning & data Retrieval - joint lab. With Thales (www.thalesgroup.com).

heading: A ADDITIONAL STEPS FOR HANDLING THE LIKELIHOOD
text: We develop below the different steps for handling the likelihood summarized in Section 3.1:
1. Making use of the independence between X and Θ, we rewrite the expectation term EpY log pY |X(y|G(y)) in equation (4) to EpΘpXpY |X,Θ log pY |X,Θ(y|G(y), θ) + c1, with c1 constant w.r.t. G:
For all x, (in particular for G(y)), if X and Θ are independent, the log-likelihood can be decomposed as:
log pY |X(y|x) = log pY,Θ|X(y, θ|x)− log pΘ|X,Y (θ|x, y) X⊥Θ
= log pY |X,Θ(y|x, θ) + log pΘ(θ)− log pΘ|Y (θ|y) (12)
Applying the expectation w.r.t to the joint pY,Θ on both sides, we obtain
EpY log pY |X(y|x) = EpY,Θ { log pY |X,Θ(y|x, θ) + log pΘ(θ)− log pY,Θ(θ|y) }
= EpY,Θ { log pY |X,Θ(y|x, θ) } + c1 (13)
The terms log pΘ(θ) and log pΘ|Y (θ|y) do not depend on x, hence c1 is a constant w.r.t. x. Plugging back G(y) in place of x and applying the law of total probabilities w.r.t. X on the right hand side, we obtain:
EpY log pY |X(y|G(y)) = EpΘpXpY |X,Θ log pY |X,Θ(y|G(y), θ) + c1 (14)
2. The general measurement process in equation (1), Y = F (X; Θ) + E induces log pY |X,Θ(y|G(y), θ) to yield a simple analytic expression:
log p(y|G(y), θ) = − 1 2σ2 ‖y − F (G(y); θ)‖22 + c2 (15)
with c2 constant. This result is directly obtained using the fact that E ∼ N (0, σ2I).
3. The likelihood term EpY log pY |X(y|G(y)) can then be replaced by −EpΘpXpY |X,Θ 1 2σ2 ‖y − F (G(y); θ)‖ 2 2 in objective (4). This is because the constant
c2 does not change the objective.

heading: B ARCHITECTURE DETAILS
text: Network architecture. Our network architectures are inspired by the Self-Attention GAN architecture in Zhang et al. (2018). They use residual networks (He et al. (2016)), where each residual block of the generator and discriminator is comprised of 2 repeated sequences of batch normalization (Ioffe & Szegedy (2015)), ReLU activation, spectral normalization (Miyato et al. (2018)) and 3 × 3 convolutional layers. For the discriminator, we use the same as Zhang et al. (2018), and for reconstruction network G, we propose an image-to-image variant of their generator. We have not added downsampling layers: we have found that they degraded the overall model’s performance. For corruption processes that yield observations that are very correlated with the input, such as Patch Band and Convolve-Noise, we have found that usingG(y) := y + Net(y) for the reconstruction network allows us to initialize G close to identity, accelerates training and augments the overall quality of the samples.
Hyperparameters. Hyperparameters have been selected on the validation set, based on the mean square error between the reconstructions x̂ and the image x. As in Zhang et al. (2018), we use imbalanced learning rates for the generator and the discriminator (0.0001 and 0.0004, respectively), using the Adam optimizer (Kingma & Ba (2014)), using β1 = 0 and β2 = 0.9. The weights are initialized using orthogonal initialization. We set λ = 2, and exponentially decay the learning rate every 400 iterations, setting the decay factor to 0.995.

heading: C ADDITIONAL INFORMATION ON THE BASELINES
text: C.1 UNPAIRED VARIANT
C.2 PAIRED VARIANT
C.3 DEEP IMAGE PRIOR (DIP)
Given an input measurement y, a generator Gφ parameterized by random parameters φ, and a random latent code z, the reconstruction Gφ∗(z) is obtained by resolving the following optimization problem:
φ∗ = arg min φ ‖y −Gφ(z)‖22 (16)
For measurement processes Patch-Band, Remove-Pixel and Remove-Pixel-Channel (refer to Section 4.2), the resulting reconstruction Gφ∗(z) was not satisfactory: G was consistently regressing to the corrupted values in the measurement y, which led to unsatisfactory results. Instead of presenting these results, we have chosen to remove the contribution of the error terms where the measurement process induced null values from objective (16), and present the latter instead. However, this assumes access to the true value θ that corrupted datum y: y = F (x; θ). Formally, we resolve the following objective:
arg min φ
‖F (y −Gφ(z); θ)‖22 (17)
Where F acts as a mask, and eliminates the terms associated to the pixels from y that have been put to 0. Note that this method corresponds to the inpainting formulation in Ulyanov et al. (2017). We used the implementation provided by the authors4.

heading: D ADDITIONAL SAMPLES
text: 4https://dmitryulyanov.github.io/deep_image_prior

Review summaries:
This paper presents a method to reconstruct images using only noisy measurements. This problem is practically interesting, since the noiseless signal may be unavailable in many applications. The approach combines ideas from recent development in compressed sensing and GANs. However, the model’s presentation is confusing, and many important details of the experiments are missing.