Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose to combine disentangled representations of generative models (GANs) and quantile regression for stochastic encoding of corrupted data. Thanks to quantile prediction, the encoder predicts uncertainty intervals instead of single points, which allows to get a more sound representation of corrupted data.  Also, the authors use calibration algorithms to ensure, with a certain confidence threshold, that the predicted quantiles contain the target value. 
strengths_and_weaknesses: Strengths:
* 1) Original idea of using generative models to visualize uncertainties on corrupted data, and of combining disentangled representations of GANs and encoding with a quantile regression loss. 
* 2) An advantage of the approach is the interpretability of the results and the possibility to reuse large models. 

Weakness
* 1) There are no experiments on real attributes. In this setting (real attributes), the errors can come from: a) lack of disentanglement in the pre-trained generative model b) the proposed method of encoding with quantile regression and calibration. It would be more convincing to have studies on real attributes, and see the relative impact of the two types of errors. 
* 2) The hypothesis that a) each dimension of the generative model corresponds to a separated factor of variation and b) that these factor of variations are independant, is a very strong hypothesis.
* 3) There is no ablation study on the impact of the different loss terms. For example, what happens without the identity loss? 
* 4) The idea is close to previous works that leverage generative models for inversion tasks. For example, [1] uses a similar procedure of training an encoder on masked images with frozen and pre-trained generator. Although the proposed method is different because it proposes learned quantiles, [1] and similar works should be discussed in related work section.
* 5) Small typos in the definition affect the readability, especially the mathematical details. L82: "$(\alpha,\delta)$ risk-controlling prediction set if $(\alpha,\delta)$-Risk-Controlling Prediction Set if". In Equation (5), $Y$ seems to be used instead of $Z$.  L202, "the the".

[1] Chai, Lucy, Jonas Wulff, and Phillip Isola. "Using latent space regression to analyze and leverage compositionality in GANs." International Conference on Learning Representations. 2020.
questions: Beta-VAEs [2] are known to be disentangled. In the VAE framework, each point is encoded as a Gaussian distribution. This model naturally induces uncertainty estimation. Moreover, there exists some variants of VAEs, like Beta-VAEs, known for their disentangled representations. How does your framework could be compared to such models?

[2] Higgins et al.  "beta-vae: Learning basic visual concepts with a constrained variational framework". International Conference on Learning Representations. 2017.

limitations: The authors addressed the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a method to obtain the uncertainty intervals of the semantic latent variables in the generative model. The method takes a corrupted image input and then predicts each semantic factor along with an uncertainty interval. These uncertainty intervals can propagate through the generative adversarial network (GAN) to visualize the uncertainty in image space. To fulfill the goal, the author proposes two steps: (1) use the quantile regression to output a heuristic uncertainty interval for each element in the latent space; (2) calibrates these uncertainties such that they contain the true value of the latent for new, unseen input. Experiments show that the proposed method can produce visually meaningful uncertainties and the model exhibits more uncertainty with increased information loss at the input image. 
strengths_and_weaknesses: Originality: 
To the best of my knowledge, this is the first work to integrate the distribution-free uncertainty quantification with the latent space analysis in GAN. The proposed method is novel and original given the considered application and the new training pipeline to achieve the uncertainty interval by means of quantile regression, although the core of the proposed method is originated and motivated by [7] and [2]. 

Quality and Clarity: 
Overall, the author provides a clear description of the proposed method with detailed and straightforward illustrations to help the reviewer understand the paper. 
Weakness:
However, the introduction of the distribution-free uncertainty quantification and the Risk-Controlling Prediction Set (RCPS) is not sufficient. As being the core underlying theory to backup the feasibility of the proposed method, the author should provide more explanation to convey the connection clearly and intuitively between the theory and the considered application. This is crucial since the potential reader may not have enough background on the distribution-free uncertainty quantification and the RCPS in [7]. The current writing of lines 79-90 on page 3 and section 2.3 seems to assume that the potential reader has already known the related techniques, which may hurt the readability of the paper. For line 132 on Page 5, it would be better to include more details about the exact section or paragraph for the potential reader to locate. 

Minor Format Problem:
The Figure 5 caption is too close to the text in line 244, which makes it hard to interpret.

Significance:
The qualitative gives several examples of the predicted semantic uncertainties based on the controlled corrupted input image. 
Weakness:
However, it is hard to say whether these results are general enough to support the claims in the paper, and the author may want to include more examples to give a more comprehensive demonstration.  

Moreover, the author may want to show the visual difference between the lower and upper quantiles with and without calibration to demonstrate that the proposed method in Section 2.3 works as expected.

Although the proposed method provides a first step to quantify the uncertainty of the latent variable in GAN, the present work relies strongly on the accessibility of the disentangled latent space and thus the present work can only show the feasibility of GAN. Moreover, the concrete application of such an uncertainty calibration method in generative models is not clear. The author may want to provide some illustrations that what kinds of existing applications related to the generative models they expect the proposed method may provide some benefits.
questions: The questions and suggestions are already included in “Strengths and Weaknesses”. The reviewer may expect the author to respond to the weaknesses listed in the “Strengths and Weaknesses” to consider changing the rate.
limitations: The author has already stated three limitations of the present work in lines 289-294 on Page 9, where these limitations are all related to the core assumptions of the proposed method and indispensable for the present work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes an architecture to predict uncertainty intervals for (pre-learned) latent coordinates. It starts off with an existing disentanglement architecture, samples artificial images from latents, and noisifies them to get training samples for their method. The resulting training data is then fed to a quantile regression network to learn both bounds of the uncertainty interval per coordinate. I was expecting a disentaglement paper with some disentanglement pureness guarantees, so my review is an educated guess.

The training procedure optimizes a four-component loss with hyperparameters ($c$ and the potentially non-unit weights on the quantile losses) and a fine-tuning (calibration) stage with its own loss. It also depends on third-party embeddings and identified causal dimensions in the latent space.



strengths_and_weaknesses: Strengths:
- Predicts a trainable bound on the probability that the latent deviates far from its point-estimate over the assumed noise

Main remarks/weaknesses:
- After reading the paper it's not very clear, what would be a killer application for this method and what would be a hypothetical actionable workflow after getting such predictions.
- On the theoretical side, it seems to refer to the properties of quantile regression loss, and appears to echo the result that enough data and finite hypotheses classes (discretized NN weights) are PAC-learnable (here, with a particular loss function) -- hence the ability to satisfy $\alpha$ and $\delta$ bounds by generating more data.

questions: Some more explanations needed:
- how hard is it to tune the $c$ for a task at hand?
- 120: what would be the calibration loss function here?
- 129: why are you doing scanning and not binary search?
- 171: please add a few words on how to pick disentangled dimensions
- Ablation would be nice to see if only for the setting w/o calibration, to get a feeling of how visually important it is.


Minor:
- 64: lacking space after i.e. 
- 83: something wrong with the sentence ("risk-controlling prediction" appears twice)
- 127: what does "drive delta to zero" mean, if it's user-specified? Probably the whole last term is meant.
- Figure 5: explain the x-axis "set size" (not clear why it's int and float in different images)
limitations: Overall, the method appears to produce what it is trained for but it's not clear from the draft when to apply it, and how costly the procedure is given the multi-phase learning (and HPO) and data generation.
review_title: null

Meta-review: metareview: While the reviewers raised concerns about assumptions of disentangling uncertainty from spurious disentanglement (and the strong  disentanglement hypothesis used), they believe the work is interesting and novel, and I concur.  I think even if it is not perfect (re assumptions, etc.) this is a nice place to start a conversation.
confidence: Less certain
Decision: Accept