Title: Amortised MAP Inference for Image Super-resolution
Authors: Casper Kaae Sønderby
Jose Caballero
Lucas Theis
Wenzhe Shi
Ferenc Huszár
Venue: ICLR 2017
Keywords: Theory
Computer vision
Deep learning

Abstract: Image super-resolution (SR) is an underdetermined inverse problem, where a large number of plausible high resolution images can explain the same downsampled image. Most current single image SR methods use empirical risk minimisation, often with a pixel-wise mean squared error (MSE) loss.
However, the outputs from such methods tend to be blurry, over-smoothed and generally appear implausible. A more desirable approach would employ Maximum a Posteriori (MAP) inference, preferring solutions that always have a high probability under the image prior, and thus appear more plausible. Direct MAP estimation for SR is non-trivial, as it requires us to build a model for the image prior from samples. Here we introduce new methods for \emph{amortised MAP inference} whereby we calculate the MAP estimate directly using a convolutional neural network. We first introduce a novel neural network architecture that performs a projection to the affine subspace of valid SR solutions ensuring that the high resolution output of the network is always consistent with the low resolution input. We show that, using this architecture, the amortised MAP inference problem reduces to minimising the cross-entropy between two distributions, similar to training generative models. We propose three methods to solve this optimisation problem: (1) Generative Adversarial Networks (GAN) (2) denoiser-guided SR which backpropagates gradient-estimates from denoising to train the network, and (3) a baseline method using a maximum-likelihood-trained image prior. Our experiments show that the GAN based approach performs best on real image data. Lastly, we establish a connection between GANs and amortised variational inference as in e.g. variational autoencoders.

TL;DR: Probabilisticly motivated image superresolution using a projection to the subspace of valid solutions

Review discussion:
The paper presents an amortised MAP estimation method for SR problems. By learning a neural network which learns to project to an affine subspace of SR solutions which are consistent with the LR method the method enables finding propoer solutions with by using a variety of methods: GANs, noise assisted and density assisted optimisation.
Results are nicely demonstrated on several datasets.
I like the paper all in all, though I feel the writing can be polished by quite a bit and presentation should be made clearer. It was hard to follow at times and considering the subject matter is quite complicated making it clearer would help.
Also, I would love to see some more analysis of the resulting the networks - what kind of features to they learn?
Thanks for the review!
> It was hard to follow at times and considering the subject matter is quite complicated making it clearer would help.
We attempted to strike a balance between accessibility of the presentation and technical correctness.
However we might have missed this balance in some places. If you could point us to any specific equations/paragraphs/concepts that are unclear we would be very happy to try to clarify these?
> I would love to see some more analysis of the resulting the networks - what kind of features to they learn
We consider the main contributions of the paper is the proposed methods for MAP inference in image models (the GAN, Likelihood-based and Denoising Autoencoders guided.
Hence we spend the main part of the text deriving these models and qualitatively comparing the results to highlight the relative merits of each approach. We agree that investigating the behaviour of the filters would be useful especially if we used the networks as a self-supervised representation learning, but this was not our goal and we believe that this analysis is outside the scope of the current work.
Our actual superresolution networks are relatively simple, and we do not expect them to learn anything beyond recognising textured regions and generating finer resolution texture.
Thus we don't expect these features to be very conceptually meaningful for humans to look at and they would be hard to visualise and explain. The more interesting form of representation learning - if any - happens in the methods which we use to represent the image prior (the GAN, pixel CNN, denoising function), our main contribution here is exploiting these learnt representations of natural image statistics in a novel and theoretically grounded way for image superresolution and low-level vision tasks in general.
Sincere apologies for the late review.
This paper argues to approach Super-Resolution as amortised MAP estimation. A projection step to keep consistent HR-LR dependencies is proposed and experimentally verified to obtain better results throughout.
Further three different methods to solve the resulting cross-entropy problem in Eq.9 are proposed and tested.
Summary: Very good paper, very well written and presented. Experimental results are sufficient, the paper presents well chosen toy examples and real world applications. From my understanding the contributions for the field of super-resolutions are novel (3.2,3.3,3.4), parts that are specific for the training of GANs may have appeared in different variants elsewhere (see also discussion). I believe that this paper will be relevant to future work on super-resolution, the finding that GAN based model training yields most visually appealing results suggests further work in this domain.
Manuscript should be proof-read once more, there were some very few typos that may be worth fixing.
We thank the reviewer for the comments, please find our response below.
We agree that the general algorithm for training GANs is not our contribution.
However, the specific variant used here that minimizes the KL divergence between data and model distribution, eq 10 in the paper, is new (to our knowledge this was first discussed in a blog post by Ferenc Huszar - one of the coauthors of this submission, and has since been built upon in independent work by Arjovsky & Buttou and Mohamed & Lakshminarayanan - both parallel ICLR submissions).
Furthermore, we present  ‘instance noise’ as a new and theoretically motivated technique for improving the stability of GAN training. This has also been independently discovered by Arjovsky & Buttou, and we believe it is a useful step in understanding the instability GANs.
Apart from sections (3.2,3.3,3.4) that the reviewer points out we also believe that the analysis of the loss function in section 3.0 and the affine projection for maximising the likelihood term described in section 3.1 are valuable contributions.
Furthermore, Appendix F establishes a connection between GANs and amortised variational inference which we believe is a very important link - this opens the door for approximate Bayesian image superresolution which can have several applications in medical diagnosis where calibrated uncertainty estimates are important.
The paper presents a new framework to solve the SR problem - amortized MAP inference and adopts a pre-learned affine projection layer to ensure the output is consistent with LR.
Also, it proposes three different methods to solve the problem of minimizing cross-entropy. Generally, it is a great paper.
However, I still have several comments:
1) The proposed amortized MAP inference is novel and different from the previous SR methods. Combined with GAN, this framework can obtain plausible and good results. Compared with another GAN-based SR methods - Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, question may arise as to what this new formulation adds to the latest state-of-the-art.
2) Using an affine projection architecture as a constraint, the model do not need any corresponding {HR, LR} image pairs for training.
However, when training the affine projection layer, we still need the {HR, LR} image pairs. Does it mean that we merely transfer this training procedure to the training of affine projection?
3) The paper presents many results of the framework, including the results of natural images from ImageNet. Can the author also provide the results of Set5, Set14 or BSD100, which are conventional test dataset for SR, so that we can perform a fair comparison with previous work.
4) I see that the size of the results of nature images presented in this paper are limited to 128*128. Can this framework perform well on images with larger size? Because SR will encounter input with arbitrary size.
5) A normal GAN will have a noise term as a latent space, so that it can be better illustrated as learning a distribution. Do the author try the noise vector?
Overall, this paper provides a new framework for SR with solid theoretical analysis. The idea is novel and the author explore many methods.
Though there still exist questions like the necessity and more experiments are needed. I think this work will will provide good inspiration to the community.
We thank the reviewer for the comments, please find our response below.
> 1) The proposed amortized MAP inference is novel and different from the previous SR methods. Combined with GAN, this framework can obtain plausible and good results. Compared with another GAN-based SR methods - Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, question may arise as to what this new formulation adds to the latest state-of-the-art.
Our work primarily focuses on theoretically grounded methods for MAP inference in images whereas other recent papers focusses primarily on empirical evaluations and improvements of the image quality. We believe that our contributions are orthogonal and complementary to these works and further that by combining our work i.e. the affine projection layer presented here with the improved model architectures in i.e. Ledig et. al. 2016 the performance can be further improved. We agree that our results do not surpass the visual quality of reconstructions in i.e. Ledig et. al. 2016, however we believe our work have several contributions that make it a valid stand-alone contribution:
a) Recent work combined the GAN objective with either VGG feature matching and total variation loss (Ledig et. al. 2016, Dosovitskiy et. al. 2016) or L1 penalty in the low-resolution space (Garcia 2016, unpublished), as the GAN alone does not lead to good results in general. Our work shows that it is possible to use a GAN-only training objective with an appropriately modified architecture.
Moreover we show that training of the modified architecture is faster and reaches a lower eventual minimum when trained with MSE (Figure 2), which makes us believe it is a better architecture choice overall.
b) Ledig et al, (2016) provide only intuitive reasoning for using GANs and focus on empirical evaluation of the proposed method, this submission provides a solid theoretical motivation and interpretation for using GANs, and establishes connections to both MAP and variational inference (Appendix F).
c) Compared to other recent works we first establish a theoretically motivated loss function (MAP inference) that we want to minimize. We then present three potential solutions and compare their relative merits (GAN, Likelihood based using pixelCNNs and Denoiser Guided).
While the conclusion is that GANs performed best in practice, we believe it is a valuable contribution to study alternative approaches.
Further, to our knowledge the use of denoising in this context is also novel, and a non-trivial contribution.
d) Although not the main contribution, this paper introduces the 'Instance noise' heuristic, and provides a novel analysis of the GAN algorithm in Appendix C. Note that this was discovered in parallel by (Arjovsky and Buttou, in review for ICLR 2017).
> 2) Using an affine projection architecture as a constraint, the model do not need any corresponding {HR, LR} image pairs for training.
However, when training the affine projection layer, we still need the {HR, LR} image pairs. Does it mean that we merely transfer this training procedure to the training of affine projection?
For training the affine projection we do not need to use natural images, in fact we trained the projection using only white Gaussian noise. This approach approximates a numerical solution to matrix inversion. In figure 2 we provide results using combinations of learnable, pretrained and fixed affine projections. We find that pretraining the affine projection using white gaussian noise (as described above) and keeping the parameters fixed during training of the main model works the best. We have provided details in Appendix B, but we’ll try to clarify this in the main text.
> 3) The paper presents many results of the framework, including the results of natural images from ImageNet. Can the author also provide the results of Set5, Set14 or BSD100, which are conventional test dataset for SR, so that we can perform a fair comparison with previous work.
Our focus have not been on achieving state of the art performance on benchmark datasets but on introducing and analyzing theoretically grounded methods for amortized MAP inference. We proposed three different methods for this and the included experiments are chosen to facilitate comparison of the relative merits of these methods.
However we agree that future work combining the work present here with the methods proposed in i.e. Ledig et. al. 2016 should include such experiments.
> 4) I see that the size of the results of nature images presented in this paper are limited to 128*128. Can this framework perform well on images with larger size? Because SR will encounter input with arbitrary size.
Our models are fully convolutional it can be applied to arbitrarily sized input. There is nothing preventing the models from working on larger images, including the affine projection layer.
> 5) A normal GAN will have a noise term as a latent space, so that it can be better illustrated as learning a distribution. Do the author try the noise vector?
We did experiment with introducing noise into the model as described below. In Section 5.6 we argue it might be desirable from a perceptual quality point of view and further in Appendix F we provide a theoretical motivation for doing so: it would be equivalent to a form of variational inference.
There are multiple ways one can introduce noise to the superresolution network i.e. via an input noise vector, input dropout or feature dropout.  We observed that the model simply chooses to ignore the introduced stochasticity if possible, such as when it is provided as an an input vector.
Hence we believe that dropout noise or similar is preferred for training of AffGAN models since the model is forced to depend on the introduced stochasticity. Experimentally we found that the stochastic models do work as we hoped: conditioned on a LR face a noisy network can sample different plausible reconstructions, sometimes smiling, sometimes not for example.
However we don't feel that these results are fully conclusive at this stage why we have chosen not to include them.

Meta-review:
role: PC
data: comment: All the reviewers agreed that the paper is original, of high quality, and worth publishing.

Decision: Accept (Oral)