Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper considers the problem of generating conditional samples from unconditional models, such that one can query the learned model with a particular set of attributes to receive conditional samples.  Key to achieving this is the introduction of a realism constraint that encourages samples to be more realistic without degrading their reconstruction and a critic which identifies regions of the latent space with targeted attributes.   Generating conditional samples then involves finding points in the latent space which satisfy both the realism constraint and the critic.  This is carried out either used gradient-based optimization or using an actor function which tries to amortize this process.

This paper is clearly on a timely topic and addresses an important problem.  The low-level writing is good and the paper uses figures effectively to explain its points.  The qualitative results presented are compelling and the approaches taken seem reasonable.  On the downside, the quantitative evaluation of method does not seem very thorough and the approach seems quite heuristical at times. Overall though, the paper seems like a solid step in a good direction with some clearly novel ideas.

My two main criticisms are as follows
1. The evaluation of the method is generally subjective without clear use of baselines or demonstration of what would do in the absence of this work - it seems like it works, but I feel like I have a very poor grasp of relative gains.  There is little in the way of quantitative results and no indication of timing is given at any point.  Given that the much of the aim of the work is to avoid retraining, I think it is clear to show that the approach can be run sufficiently quickly to justify its approach over naive alternatives.

2. I found the paper rather hard to follow at times, even though the low-level writing is good.  I think a large part of this is my own unfamiliarity with the literature, but I also think that space has been prioritized to showing off the qualitative results at the expense of more careful description of the approach and the evaluation methods.  This is a hard trade-off to juggle but I feel that the balance is not quite right at the moment.  I think this is a paper where it would be reasonable to go over the soft page limit by a page or so to provide more precise descriptions.  Relatedly, I think the authors could do a better job of linking the different components of the paper together as they come across a little disjointed at the moment.
review_title: Seems like a solid paper on a timely topic

final_score: rating: 7: Good paper, accept
review_content: UPDATE: I think the authors' rebuttal and updated draft address my points sufficiently well for me to update my score and align myself with the other reviewers.

-----

ORIGINAL REVIEW: The paper proposes a method for learning post-hoc to condition a decoder-based generative model which was trained unconditionally. Starting from a VAE trained with an emphasis on good reconstructions (and at the expense of sample quality, via a small hard-coded standard deviation on the conditional p(x | z)), the authors propose to train two "critic" networks on the latent representation:

1. The "realism" critic receives either a sample z ~ q(z) (which is implicitly defined as the marginal of q(z | x) over all empirical samples) or a sample z ~ p(z) and must tell them apart.
2. The "attribute" critic receives either a (latent code, attribute) pair from the dataset or a synthetic (latent code, attribute) pair (obtained by passing both the attribute and a prior sample z ~ p(z) through a generator) and must tell them apart.

The goal is to find a latent code which satisfies both the realism and the attribute-exhibiting criteria, subject to a regularization penalty that encourages it to stay close to its starting point.

It seems to me that the proposed realism constraint hinges exclusively on the ability to implictly capture the marginal distribution q(z) via a trained discriminator. Because of that, any autoencoder could be used in conjunction with the realism constraint to obtain good-looking samples, including the identity encoder-decoder pair (in which case the problem reduces to generative adversarial training). I fail to see why this observation is VAE-specific. The authors do mention that the VAE semantics allow to provide some weak form of regularization on q(z) during training, but the way in which the choice of decoder standard deviation alters the shape of q(z) is not explained, and there is no justification for choosing one standard deviation value in particular.

With that in mind, the fact that the generator mapping prior samples to "realistic" latent codes works is expected: if the VAE is trained in a way that encourages it to focus almost exclusively on reconstruction, then its prior p(z) and its marginal q(z) have almost nothing to do with each other, and it is more convenient to view the proposed method as a two-step procedure in which an autoencoder is first trained, and an appropriate prior on latent codes is then learned. In other words, the generator represents the true prior by definition.

The paper is also rather sparse in terms of comparison with existing work. Table 1 does compare with Perarnau et al., but as the caption mentions, the two methods are not directly comparable due to differences in attribute labels.

Some additional comments:

- BiGAN [1] should be cited as concurrent work when citing (Dumoulin et al., 2016).
- [2] and [3] should be cited as concurrent work when citing (Ulyanov et al., 2016).

Overall, the relative lack of novelty and comparison with previous work make me hesitant to recommend the acceptance of this paper.

References:

[1] Donahue, J., Krähenbühl, P., and Darrell, T. (2017). Adversarial feature learning. In Proceedings of the International Conference on Learning Representations.
[2] Li, C., and Wand, M. (2016). Precomputed real-time texture synthesis with markovian generative adversarial networks. In European Conference on Computer Vision.
[3] Johnson, J., Alahi, A., and Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: # Paper overview:
This paper presents an analysis of a basket of approaches which together enable one to sample conditionally from a class of 
generative models which have been trained to match a joint distribution. Latent space constraints (framed as critics) are learned which confine the generating distribution to lie in a conditional subspace, which when combined with what is termed a 'realism' constraint enables the generation of realistic conditional images from a more-or-less standard VAE trained to match the joint data-distribution.

'Identity preserving' transformations are then introduced within the latent space, which allow the retrospective minimal modification of sample points such that they lie in the conditional set of interest (or not).  Finally, a brief foray into unsupervised techniques for learning these conditional constraints is made, a straightforward extension which I think clouds rather than enlightens the overall exposition.

# Paper discussion:
I think this is a nicely written paper, which gives a good explanation of the problem and their proposed innovations, however I am curious to see that the more recent "Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space" by Nguyen et al. was not cited.  This is an empirically very successful approach for conditional generation at 'test-time'. 

Other minor criticisms include:
* I find the 'realism' constraint a bit weak, but perhaps it is simply a naming issue.  Did you experiment with alternative approaches for encouraging marginal probability mass?

* The regularisation term L_dist, why this and not log(1 + exp(z' - z)) (or many arbitrary others)? 

* The claim of identity preservation is (to me) a strong one: it would truly be hard to minimise the trajectory distance wrt. the actual 'identity' of the subject.

* For Figure 6 I would prefer a different colourscheme: the red does not show up well on screen.

* "Furthermore, CGANs and CVAEs suffer from the same problems of mode-collapse and blurriness as their unconditional cousins" -> this is debateable, there are many papers which employ various methods to (attempt to) alleviate this issue.


# Conclusion:
I think this is a nice piece of work, if the authors can confirm why "Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space" is not placed relative to this work in the paper, I would be happy to see it published.  If stuck for space, I would personally recommend moving the one-shot generation section to the appendix as I do not think it adds a huge amount to the overall exposition.
review_title: Nicely written paper: one key missing reference.

Meta-review: This paper clearly surveys a set of methods related to using generative models to produce samples with desired characteristics.  It explores several approaches and extensions to the standard recipe to try to address some weaknesses.  It also demonstrates a wide variety of tasks.  The exposition and figures are well-done.
Decision: Accept (Poster)