Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a method to control pre-trained generative models (e.g. to condition samples on a text prompt, or to control the value of an attribute, or to debias samples). The control can be specified using a different model (e.g. CLIP or inverse graphics or a classifier) via an energy-based formulation that outputs a controlled-version of the pre-trained generative model. 
strengths_and_weaknesses: **Strengths**
- Compatibility with a wide range of control models (classifiers, inverse graphics, embedding)
- Compatibility with a wide range of generative models (results include StyleNerf, NVAE, StyleGAN2, and BigGan)
- Compositionality of controls
- No optimization at inference
- The writing is easy to follow. (But here's a specific point. The following motif is repeated at several points in the paper: PromptGen leverages “the knowledge of various off-the-shelf models” for distributional control of pre-trained generative models. It is quite unclear especially in the abstract what “off-the-shelf models” you are referring to, and whether you are still talking about generative models. Perhaps it would be clearer to state that you are trying to control pre-trained generative models using knowledge from “other models” rather than “off-the-shelf models”?)

**Weaknesses/concerns**:
- Most results are on faces. The authors could have evaluated on different data or chosen a wider variety of prompts. So far it is only clear that the model can control/yield pictures of babies, cats, and persons (especially controlling for age, make-up, or race).
- When evaluated on non-facial data, the results lack diversity (PromptGen on ImageNet 512 in Figure 5(d)). “A photo of a glow and light dog” seems to yield a single specimen. The melancholy robot always has the same helmet and the background is a consistent color. There’s no further results from ImageNet in the supplementary. Also the size of the generated samples is too small given they were 512x512.
- CLIP control doesn’t always seem to work well e.g. in Fig 12(a), at least a third of the images have cats with open eyes. It is unclear whether it is CLIP to blame, or a bias in the image dataset.
questions: - Functional composition is an important property that needs further evidence. Consider providing results that show how well CLIP captions can be composed (e.g. C_1=“photo of an Asian man” and C_2=“photo of a person with eyeglasses” and C_3=“photo of a bald person”). If you could expound on the performance difference between using a multi-component control C (as suggested in Figure 2’s caption) or using two separate controls C_1 and C_2, that would be super helpful.
- Table 2: could you please add at least one more column here so we can ensure the attributes aren’t cherry-picked? How about hairColor (say “black” if you need a binary attribute)?
- Fig 9: the latents learnt by the GAN looks fairly well separated between the two clusters. How well is PromptGen able to cope with less structured latent spaces?
- Could you clarify whether equation 8 holds for all invertible NNs, or only the architecture you've chosen?
limitations: Section F in the appendix covers some interesting failure modes. But it places the burden of failure mostly on the pre-trained generative model or control model. It would be nice to see where the overall idea/method might fail.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces an approach to use supervision from off-the-shelf pre-trained models, including classifiers, scoring functions, and multi-modal encoders, as controls for the generation of images with custom properties. In particular, the proposed approach transforms random noise, using an invertible neural network, that is fed to a pre-trained generative model.
strengths_and_weaknesses: +This paper includes a through set of experiments that showcase the effectiveness of the proposed approach across several pre-trained generative models (e.g. StyleNeRF, NVAE, BigGAN, StyleGAN2) and different types of controls (binary properties, continuous values, text descriptions). 

+A unified approach to custom controls for successfully steering pre-trained generative models can facilitate several down-stream applications and has high significance. In particular, success in compositional controls makes this work even more interesting.

-Both the positioning of the work and the explanation of the method can be further clarified.  See below for more details.

questions: * The authors attempt at differentiating this paper from prior work suggests that [prior methods] “are either model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiring all training samples to be labeled for explicit conditions), limiting their generality and practical use.” However, there are several prior works that enable sampling from a specific region of the latent space in a frozen generative model without doing any optimization at inference time (e.g. [1]) (not needing labels at training time, using a frozen generative model, using interpolation of previously learnt vectors at inference time, applicable to both GAN and VAE models). Additionally, it is unclear what being “model-dependent” means here. Does it mean learning to navigate the latent space that is specific to a single pre-trained generative model and may not generalize to other pretrained generative models? If so, the parameters $\theta$ in PromptGen also rely on the pretrained generative model at hand (In Algorithm 2, the gradients pass through the frozen $G_{\phi}$.) Conceptually, both PromptGen and the aforementioned approaches apply to different pre-trained generative models, though the parameters are going to be tuned for each frozen model. The authors might have missed such prior research in their literature review or the writeup does not properly communicate that. Please clarify the distinctions and positioning of this work.

*  Method clarification: It is unclear where the different formulations of energy for the classifier/CLIP type models/and scoring functions (i.e. the inverse graphics case) come from. Is this well-defined in an EBM? Is this inspired by prior work? Is this a design choice? 

*  Prompt tuning uses a (small) dataset for tuning additional parameters that are (usually) prepended to a meaningful (textual) input to steer the model’s outputs in a direction of interest. Some variants do it in the discrete natural language format and some in continuous embedding space. Some form of concatenation to an existing meaningful input is the fingerprint of what is usually dubbed as “prompt tuning”. So in my opinion, drawing a parallel between “learning transformations over random noise” and “prompt tuning” seems a bit tangential and potentially distracting at first. Perhaps authors meant other connections? If yes, would appreciate clarifications.

[1] Shen, Yujun, et al. "Interpreting the latent space of gans for semantic face editing." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

limitations: The authors have briefly described the societal impact of their work. Additionally, one can argue that more granular controls for generative models might make it easier for generating targeted Deepfakes and its resulting potential negative impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes PromptGen, a method for sampling pre-trained generator networks (e.g. StyleGAN) conditioned on a prompt (label, text, etc) using off-the-shelf discriminative models (classifiers, CLIP etc). The proposed method uses an invertible neural network (INN) to express a distribution p(z ; C) over generator latents z given constraint C. A prompted sample is obtained by sampling p(z ; C) and then passing through the generator to obtain a data-space sample (all experiments are conducted with images but in principle the method could be applied to other modalities). INNs are trained either for a specific instant of the prompt C, or in a prompt conditional setting.

The method is demonstrated on a range of tasks, including de-biasing StyleGAN, and generating conditioned on a text prompt (with CLIP).




strengths_and_weaknesses: Strengths
* Unlike iterative inference time methods like PPGM, PromptGen can be applied (after training) to sample latents in a single forward pass, resulting in significantly improved inference speeds. This is particularly useful in cases where the INN can effectively represent a constraint that can be applied to all samples, e.g. StyleGAN race de-biasing. 
* The compositionality of the proposed method, where prompting mechanisms can be applied in sequence, is a desirable property. 
* The experiments: race / gender de-biasing, text-conditioning are well chosen for their real-world importance.

Weaknesses
* The broad usefulness of the method depends on the quality and coverage of the pre-trained generator network. GAN generators in general are prone to mode-dropping, especially those trained on complex natural image distributions (BigGAN). This is addressed in the limitations section.
* For complex conditional tasks, like text-conditioned generation, the INN will have to learn a very complex mapping (e.g. from text -> latents). This will require lots of data / compute / optimization etc and as such might be prohibitive in practice. 
* In general there is a balance between the benefits of amortization: fast inference speeds, and the costs: training time and compute. And which is preferable depends on a user's constraints. I think the paper would benefit from a discussion of these issues. 
questions: * The prospect of using such models to reveal the limitations of pre-trained models (e.g. CLIP and the makeup example) is potentially very useful. But how do we know if the limitation belongs to the prompting model, or if it is a limitation of the INR-based p(z ; C)? For some prompts, p(z ; C) might be very complex, and therefore challenging to learn. So how do we know that we haven't just failed to (fully) learn the mapping when training the INR? If there isn't a good way of knowing, then this is a limitation, as in general it won't be possible to confidently use the proposed method to diagnose other models.
* For the text conditional experiments, is the INN trained just for a single prompt? Why not train a text-conditional model? It is impractical to train separate models for each text input, and is a limitation if it is not possible with the proposed approach.
* Can PromptGen be applied to diffusion models?   
* Could you expand on this comment in the limitations:  "Except for the case of generative models with a class-embedding space, PromptGen focuses on mode-seeking and mode-reweighting instead of domain adaptation" - I didn't really understand what was meant by domain adaptation vs mode seeking in this context. 
limitations: * The authors adequately addressed the fact that the method is dependent on the data coverage of the generator network used. E.g. if you're using StyleGAN you're going to get faces and not spaceships. 
* The need to train a potentially very powerful conditional INN for complex tasks such as text-conditional generation wasn't really addressed (as mentioned in the weaknesses section). For this use case the burden of training such a network could easily be so high that a practicioner may prefer an iterative inference time method (like PPGN), even with its higher inference costs.
* Potential negative societal impacts were not really discussed, however the experimental work on de-biasing pre-trained models (StyleGAN, CLIP) communicates to the reader that bias is an issue with such models that should be addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a unified method called PromptGen to control the image synthesis of pre-trained generative models, such as StyleGAN, NVAE and StyleNeRF. The basic idea is to 1) first formulate the latent variable z distribution conditioned on the control \mathcal{C}, i.e., $p(x|\mathcal{C})$ as a latent-space EBM, where the latent-space EBM can use pre-trained image classifiers, CLIP model and inverse graphics model to specify the control; and 2) train an invertible neural network (INN) to approximate the latent-space EBM using an KL divergence as the training objective. In experiments across different image datasets, this work shows the efficacy and efficiency of PromptGen in the tasks of image synthesis based on text description, de-biasing generative models, pose-guided face synthesis, and iterative control via functional composition.

strengths_and_weaknesses: Strengths:

(1) Controlling generative models using EBM with MCMC Langevin dynamics is effective but slow (it needs multiple inference/optimization iterations). Thus, this work proposes to train another latent generator $z=f_\theta(\epsilon)$ that approximates the latent-space EBM, so the sampling can be performed with one forward pass, which I think is the main originality and significance of this work.

(2) For the second contribution, with the latent generator $z=f_\theta(\epsilon)$, this work also demonstrates 1) the generality for different controllable generation tasks, and 2) the iterative controls by composing the latent generators and the pre-trained generative model. 

(3) The paper is well-written and easy to read. Extensive experiments were performed to show the wide applicability, effectiveness and efficiency of the proposed method.

Weaknesses:

My main concern is that in experiments, this work didn’t provide sufficient quantitative results of evaluating the controllability. I like the quantitative results of evaluating the de-biasing performance, but regarding other controllable generation tasks, such as text-guided generation and pose-guided generation, I didn’t see the quantitative results of how the proposed method controls the generation to satisfy the specified attribute or text. 

For other concerns, please see the questions below.

questions: (1) Can more quantitative results on the controllability be added to show the performance of PromptGen? 

(2) In Figure 5(d), I think we see a severe mode collapse issue, as the background colors and object patterns are very similar across different generated images. How do you explain this phenomenon?

(3) For the text-guided generation, can PromptGen perform well with more complex sentences, where we have a composition of multiple attributes, such as “a photo of a smiling baby with glasses”? These kinds of results will better demonstrate the compositionality of the proposed method. 

(4) For the generative models with a class-embedding space, I wonder why we need to train another latent-space generator $y=h_\theta(\xi)$? I feel like we can also directly train a single latent-space generation $z=f_\theta(\epsilon, y)$ to approximate the EBM. Any justification?

limitations: This work has well addressed their limitations. But I didn’t see many discussions about their negative societal impact. I think this work shares with other image synthesis tools similar potential benefits and risks, which have been discussed extensively in [1]. I suggest adding more discussions about the risks of controllable image synthesis. 

[1] Vaccari, C. and Chadwick, A. (2020). Deepfakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news. Social Media+ Society.
review_title: null

Meta-review: metareview: This work concerns a unifying method for repurposing "off the shelf" conditional models in order to define an energy-based model of vectors in the latent space of a pre-trained generative model, for the purpose of controlling synthesis, and a feed-forward approximation using invertible neural networks. The authors present several use cases and experiments on each across a range of different model types.

Reviewers were positive on the presentation, originality and usefulness, and generally felt the experiments were well chosen. There were some concerns regarding discussion of societal impact (gbfq), the fact that most results involved faces and those that didn't were less compelling (5eQN), and clarity around the derived energy function and positioning relative to prior work (byc9). Most concerns were addressed in rebuttal, however QXTM felt quantitative results evaluating controllability, specifically, left much to be desired, and lowered their score following a rebuttal that they felt failed to address this issue.

Based upon the discussion and my own reading of the paper, the AC views this work in an overall positive light, the valid concerns of QXTM notwithstanding. With some reservations, I recommend acceptance.
confidence: Less certain
Decision: Accept