Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes a network architecture for inverse problems in computer vision. Example inverse problems considered are image inpainting, computing intrinsic image decomposition and foreground/background separation.
The architecture is composed of (i) a generator that produces target (latent) output (such as foreground / background regions), 
(ii) renderer that composes that latent output back to the image that can be compared with the input to measure reconstruction error, 
and (iii) adversarial prior that ensures the target output (latent) image respects a certain image statistics.

Strong  points.
- The proposed architecture with memory database is interesting and appears to be novel. 

Weak points:
- Experimental results are only proof-of-concept in toy set-ups and do not clearly demonstrate benefits of the proposed architecture.
- It is unclear whether the memory retrieval engine that retrieves images based on L2 distance on pixel values is going generalize to other more realistic scenarios. 
- Clarity. The clarity of explanation can be also improved (see below).


Detailed evaluation.

Originality:
- The novelty of this work lies in the (iii) adversarial prior that places an adversarial loss between the generated latent output and a single image retrieved from a large unlabelled database of target output examples (called memory). The adversarial prior has a convolutional form matching local image statistics, rather than the entire image.  The particular form of network architecture with the memory-based fully convolutional adversarial loss appears to be novel and potentially interesting.

- Motivation for the Architecture. The weakest point of the proposed architecture is the "Memory retrieval engine" R (section 2.4),
where images are retrieved from the memory by measuring L2 distance on pixel intensities. While this maybe ok for simple problems considered in this work, it is unclear how this can generalize to other more complicated datasets and problems.  
This should be better discussed, better justified and ideally results in some more realistic set-up shown (see below).


Quality:
- Experiments. Results are shown for inpainting of MNIST digits, intrinsic image decomposition on the MIT intrinsic image database, and figure/ground layer extraction on the synthesized dataset of 3D chairs rendered onto background from real photographs.  
 The experimental validation of the model is not very strong and proof-of-concept only. All the experiments are performed in simplified toy set-ups. The MNIST digit inpainting is far from current state-of-the-art on image inpainting in real photographs (see e.g. Pathak et al., 2016). The foreground background separation is done on  only synthetically generated test data. Even for intrinsic image demposition problem there is now relatively large-scale dataset of (Bell et al., 2014), see the citation below.  

While this is probably ok for the ICLR paper, it diminishes the significance of the work. Is this model going to be useful in a real settings? One possibility to address this would be to focus on one of the problems and show results on a challenging state-of-the-art data. It would be great to see the benefits of the memory database. 

S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild.
ACM Transactions on Graphics, 33(4):159, 2014.

Clarity:
- The clarity of the writing can be improved. I found some of the terminology of the paper, specially the “imagination” and “memory” confusing. From figure 2, it is not clear how the “memories” for the given input image are obtained, which also took me some time to understand.

- To help understand the proposed architecture, it would be useful to draw an illustration of what is happening in the "feature space”, similar in spirit e.g. to figure 2 in https://arxiv.org/pdf/1612.02136.pdf. Specially, it would be interesting to understand the role of the memory database in this way.


Significance:
- The paper describes potentially interesting architecture. Given the only proof-of-concept results in toy set-ups, the significance, in the current version, appears only limited. Rather than addressing many different problems, it would be interesting to see benefits of the proposed architecture on realistic challenging data for one of the problems.

Overall:
- The proposed architecture seems novel and potentially interesting, but experiments are only proof-of-concept and clarity can be improved. It is unclear whether the memory matching engine will generalize to other more complicated datasets and problems.  Overall, I am on the edge with this paper, giving the authors the benefit of doubt with a score slightly above the threshold.


review_title: The proposed architecture seems novel and potentially interesting, but experiments are only proof-of-concept and clarity can be improved

final_score: rating: 3: Clear rejection
review_content: In this work, the authors propose to use a (perhaps deterministic) retrieval function to replace uniform sampling over the train data in training the discriminator of a GAN.
Although I like the basic idea, the experiments are very weak.  There are essentially no quantitative results, no real baselines, and only a small amount of not especially convincing qualititative results.   It is honestly hard to review the paper- there isn't any semblance of normal experimental validation.

Note:  what is happening with the curves in fig. 6?
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a model that generates a latent representation of input image(s) and optimizes a reconstruction loss with an adversarial loss (Eq (1)) over nearest neighbors from a bank of images (“memory”).   The framework is adapted to three tasks: (i) image in-painting, (ii) intrinsic image decomposition, (iii) figure-ground layer extraction.  Qualitative results are shown for all three tasks.

I think the proposed model has potential merits.  I particularly like the fact that it seems to be reasoning over image composites via matching against a bank of images (somewhat similar to “Segmenting Scenes by Matching Image Composites” work in NIPS 2009).  However, I won’t champion the paper as the overall clarity and evaluation could be improved.

More detailed comments:

I believe the fatal flaw of the paper is there is no quantitative evaluation of the approach.  At the very least, there should be a comparison against prior work on intrinsic image decomposition (e.g., SIRFS, maybe benchmark on "intrinsic images in the wild” dataset).

I found the writing vague and confusing throughout.  For instance, “memory database” could mean a number of things, and in the end it seems that it’s simply a set of images.  “Imagination” is also vague.  On page 4, R(M,x) has the database and input image as arguments, but Fig 2 doesn’t show the input image as an input to R.  The contributions listed on page 3 should be tightened (e.g., it’s not clear what “Relevant memory retrieval for informative adversarial priors” means).  Fig 3 seems inconsistent with Fig 2 as the module for “memory database” is not present.  The fully-convolutional discriminator could use more details; one possibility is to provide a cost function.
review_title: The proposed model has potential merits, but lack of quantitative evaluation and paper clarity issues put the paper below the bar.

Meta-review: All three reviewers point to significant deficiencies (particularly the lack of quantitative evaluation and clarity problems). No response or engagement from the authors. I see no basis for supporting this paper.
Decision: Reject