Semantic uncertainty intervals for disentangled latent spaces
Authors: Swami Sankaranarayanan
Anastasios Nikolas Angelopoulos
Stephen Bates
Yaniv Romano
Phillip Isola
Venue: NeurIPS 2022 Accept (2022)
Keywords: generative models
uncertainty estimation
distribution free statistics

Abstract: Meaningful uncertainty quantification in computer vision requires reasoning about semantic information---say, the hair color of the person in a photo or the location of a car on the street. To this end, recent breakthroughs in generative modeling allow us to represent semantic information in disentangled latent spaces, but providing uncertainties on the semantic latent variables has remained challenging. In this work, we provide principled uncertainty intervals that are guaranteed to contain the true semantic factors for any underlying generative model. The method does the following: (1) it uses quantile regression to output a heuristic uncertainty interval for each element in the latent space (2) calibrates these uncertainties such that they contain the true value of the latent for a new, unseen input. The endpoints of these calibrated intervals can then be propagated through the generator to produce interpretable uncertainty visualizations for each semantic factor. This technique reliably communicates semantically meaningful, principled, and instance-adaptive uncertainty in inverse problems like image super-resolution and image completion. Project page: https://swamiviv.github.io/semantic_uncertainty_intervals/

Semantic uncertainty intervals for disentangled latent spaces
Swami Sankaranarayanan' Anastasios N. Angelopoulos? Stephen Bates?, Yaniv Romano and Phillip Isola'
'MTT
University of California, Berkeley "Technion-Isra" Institute of Technology
Abstract
Meaningful uncertainty quantification in computer vision requires reasoning about semantic information- say, the hair color of the person in a photo the location of a car on the street. To this end, recent breakthroughs in generative modeling allow to represent semantic information in disentangled latent spaces, but providing uncertainties on the semantic latent variables has remained challenging. In this work, provide principled uncertainty intervals that guaranteed to contain the true semantic factors for any underlying generative model. The method does the following: (1) it uses quantile regression to output heuristic uncertainty interval for each element in the latent space (2) calibrates these uncertainties such that they contain the true value of the latent for new, unseen input. The endpoints of these calibrated intervals can then be propagated through the generator to produce interpretable uncertainty visualizations for each semantic factor. This technique reliably communicates semantically meaningful, principled, and instance-adaptive uncertainty in inverse problems like image super-resolution and image completion. Code and demos can be found on our project page.
Introduction
When making decisions with visual data, such as automated vehicle navigation with blurry images, uncertainty quantification is critical. The relevant uncertainty pertains to a low-dimensional set of semantic properties, such as the locations of objects. However, there is a wide class image- valued estimation problems- forrimss--oommmooo super-resolution to inpainting for which there does not currently exist a method producing semantically meaningful uncertainties. As an example, imagine doing uncertainty quantification for medical image reconstruction from, say, fast but undersampled MRI are not very useful. We need uncertainty on the underlying semantics e.g. whether there is tumor, and if so, of what size and shape.
sean. In such a setting, pixelwise intervals 16 40 5
We make progress on this problem by bringing techniques from quantile regression and distribution- free uncertainty quantification together with a disentangled latent space learned by generative adversarial network (GAN). We call coordinates of this latent space semantic semanticfactors, as each controls one meaningful aspect of the image, like age hair color. We do not require these semantic factors to be statistically independent. Our method takes a corrupted image input and predicts each semantic factor along with uncertainty interval that is guaranteed to contain the true semantic factor. When the model is unsure, the intervals are large, and vice-versa. By propagating these intervals through the GAN coordinate-wise, we can visualize uncertainty directly in image-space without resorting to per-pixel intervals- see Figure The result of our procedure a rich form of uncertainty quantification directly the estimates of semantic properties of the image.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Predicted images
Predicted semantic interval Intent space)
uggerr
corrupted
ground truth
Input
not smiling
smiling
andilng?
glasso??
gasses
glasees
Figure 1: Uncertainty intervals semantic factors produced by method. We express the intervals in a disentangled latent space that allows us to factorize the uncertainty into meaningful components.
More concretely, we receive input images X and then predict an uncertainty interval for each of D semantic factors Za, d = I,,,... ..., D, which are the elements of disentangled latent space. The method involves training an encoder (a neural network that takes images as input and produces outputs the latent space of the GAN) on (X; Z) pairs to give three different outputs:
1. The point prediction, f(X). This is the encoder's best guess at the semantic factors Z.
2. The estimated lower conditional quantile, 9g (X). The encoder believes that 2g (X) is lower bound on the value of Z given X.
3. The estimated upper conditional quantile, Q1- (X). (.).TT The encoder believes that q1 (X) is upper bound the value of Z given X.
Once the above encoder is trained, as described in Section 2.2 we use it to form an uncertainty interval for each semantic factor. However, for the dth element the latent code, the naive interval (gy (X)a, )a Q1- (X)a) is not guaranteed to contain the ground truth value in finite samples. We propose to perform calibration procedure to this problem, yielding the sets
T(X)a _
cal (X), qi- (X)a
(1)
where qn is a calibrated version of constructed using the tools in Section 2.3 Once we have done so, the intervals will contain fraction of the true latent codes with high probability. In other words, for any user-chosen levels a and 6, we can output intervals that with probability satisfy
R D (d: Za T(()aa]
2 11-a.
(2)
for a new test point (X, Z). regardless of the distribution of the data, the encoder used, and the number of data points used in the calibration procedure. This guarantee, described more carefully in Definition 2.1 says that the intervals cover calibration data is not representative of our test data (which only happens with a probability 8 which goes the number of calibration data points grows). We visualize each of the {1...., D} intervals in latent space by propagating the dth lower and upper endpoints through the generator with all other entries the latent fixed to the point prediction (see Section 2.3 for a formal explanation).
a fraction of the semantic factors unless our
1.1 Central Contribution
To our knowledge, this the first algorithm for uncertainty intervals on a learned semantic represen- tation with formal statistical guarantees. By propagating these intervals through the generator, we are able to visualize uncertainty an intuitive new way that directly encodes semantic meaning. This an important step towards interpretable uncertainties in general image-valued estimation problems. However, the reader should note that our technique requires access disentangled latent space, such that of StyleGAN, and provides no guarantees about the degree of disentanglement.

---

2 Method
2.1 Notation and goal
Our data consist of pairs (X,, the corrupted image X in A' = [0, 1) and the latent code As mentioned in the introduction, we think of Z a disentangled representation with d semantic factors-ia factors of variation corresponding to interpretable features in an image, such as hair color and expression. For simplicity, assume each of the d dimensions controls single semantic factor; in practice, we ignore those that do not.
Z Z, where Z IR
functions. The first function is fixed generator G 2 + y where y = [0. 1| which takes the latent vector Z and produces the ground truth image Y Y (for ease of notation, we assume X and Y have the same shape). The second function is a corruption model, F y - X. which degrades the ground truth image to produce the corrupted image X, for example by randomly masking out part of the image. To summarize our data-generating process, we have
In our sampling model, X is generated from Z by composing
= G(Z) and X F(Y):
(3)
Goal #1. Our first goal is to train encoder E recover Z from X-in other words, invert the mapping F with a heuristic notion of uncertainty. The encoder's point prediction will be function X + Z. The uncertainty will be parameterized by two functions, Q1- X' Z and 9g X + Z, denoting our estimates of the 1 & and & conditional quantiles, respectively. These conditional quantiles potentially bad estimates; they do not natively possess the statistical guarantee we desire.
Goal #2. Having trained the encoder and the conditional quantile estimates, will output uncertainty intervals in the disentangled latent space. Each dimension will get its own interval, which has the form in 2). Ultimately, our uncertainty intervals will come with statistical risk control guarantee, like in (2), that is distribution-free
valid irrespective of the model or data distribution. The risk function controlled is the false negative rate in the latent space. In other words, bound the fraction of semantic factors not covered by their calibrated intervals above by a (say, a = 10%).
Prediction Set (RCPS)). A set-valued function T :x - 2 an (a,, 8) risk-controlling prediction set if
Definition 2.1
P [[[[L(((X), Z)] >a) Sd
(4)
where
V( {d: La T(X)a}
(5)
HW
The reader should note here that the function T depends the calibration data. The outer probability in 4 is over the randomness in this calibration procedure; the inner expectation is over the new test point. (X, Y). The reader should note that in our setting, because we can generate infinite data from the GAN, it is always possible to take ō arbitrarily close to O, effectively making T nonrandom. In the two following subsections, address each of our goals separately.
2.2 Goal #1: Training the encoder for quantile regression
Our job in this subsection is to learn the three functions f. qg. and 91 91-2. We do so by training a neural network with three different loss functions, for each of three linear heads top of the same feature extractor-see Figure 2 for the training protocol.
Loss function for point prediction. We supervise the point prediction with three loss functions. The first is an L, loss directly in the latent space, and encourages f(X) to be close to Z.
L1 (f(x)) = 115(x) - = =|1,
(6)
The second loss term imposes domain specific prior on the generated image G(J()). For our experiments on faces, we use an identity loss which encourages G(J(X) to contain attributes that have the same "identity" or semantic content
Y. We caleulate the identity loss using
3

---

image space
latent space predict
X
{99. n-3
encoder
gradients
C losses
corrupt
Y
Z
sample
generator
Figure 2: Our training pipeline, visualized above, shows our training process for the prediction f. lower quantile qş and the upper quantile qi q1-
pretrained embedding function, ID : Rd for some d', which projects Y an embedding space where images with different identities land far away from one another. Finally, we calculate our loss using cosine similarity,
(I(x))ID(V)) (I) (=)] [IDD)II-IID((]II
The third is perceptual loss the generated image GX)(X) that imposes similarity between G()(X) and in the embedding space of a large pretrained image classification model. We impose perceptual similarity by computing the LPIPS loss [50] which has been shown to preserve image quality [7]. The perceptual loss calculated using pretrained feature extractor in a similar manner Bq Following standard practice, the feature extractor used in this case is VGG network, pretrained on Imagenet.
Finally, combine the loss functions to form the loss function for the prediction f, (f 2) = L, (f(x), 2) + Ci (=(f())), G(z)) + C.LIPss (G(f G(=)). where 6₁ and 02 are hyper-parameters whose values are chosen based on a held out set.
(8)
Loss function for quantile regression. Quantile regression 30 13 32 27 28 29 31 is a statistical
method for estimating the conditional quantiles of distribution. The key idea of quantile regression to supervise the regressor using quantile loss,
(9) The minimizer of the quantile risk is the true 3 conditional quantile of 2 We supervise our conditional quantile estimates o3 and 91-9 with two separate instances of the quantile loss, L and C 1 respectively.
C (gg)(()) =) = (= - (=)) 31 {z > (=())} + (ga(x) - =) - 3)1 {= S (a()))
This concludes our explanation of the model training procedure, summarized in Algorithm Experi- mental details, such as the particular model architecture use, available in Section 3
Algorithm 1 Quantile GAN encoder training
Input: training dataset D, risk level a, number of epochs E, fixed generator G Output: trained functions f. qq. and q1-9. f.gg and 91-9 random mödel initialization
for é = 1 to do
loss <- 0
for (X train Zurain) in D do
L1 +- Ppeed (f (Xtrri))) Ztrain)
L2 - L (19 (X (ttraiu) Strain)
L3 -- L (1) (01=2(Xrain) (trrai)) Ztraiu)
loss += L1 12 + L3
end for
loss. backward
end for

---

2.3 Goal #2: Calibration
Having trained the model, we now calibrate achieve the statistical guarantee in Definition 2.1 using set of calibration data {(x; [x)) ;=1 generated from the model and the upper-confidence bound procedure from [8]. The output of the procedure will be the function T from 2): specifically, we will learn the calibrated conditional quantiles o# and caal
Our procedure will calibrate the conditional quantiles by rescaling their size multiplicatively. We will ultimately choose multiplicative factor À that gives us the desired guarantee. Towards that end, we index a family of uncertainty intervals scaled by a free parameter À for each semantic factor,
TXX)a ((X)a-X((C))
4 I(X)a + X(n- ((())) (10)
When À grows, the interval (X()a also grows, and thus, the loss function (X)) Y) shrinks. Therefore, by taking À large enough, can always ensure the loss is zero. The challenge ahead is to pick λ to be the smallest value such that 7x is an RCPS as in
The algorithm for selecting involves forming an upper confidence bound (UCB) for the risk, then picking the smallest value of À such that the upper confidence bound falls below a. We give Hoeffding's s UCB below, although we use the stronger Hoeffding-Benie Bentkus bound from g in practice:
(²())= = 1 Σ (Tx(X)), Y3) + V
2n log 5
(11)
Note that our setting, we can always generate enough samples to drive the last term to for any 5; however, if we only had finite sample from some population, this would not be the case. We
can then select Ā by scanning from large to small values, Ả = min [x: Ret (X) a, Va' 2 a). or running binary search.
Proposition 2.2 (Tx is an RCPS With À selected as above, Ts satisfies Definition 2.1
For the proof of this fact, along with a discussion the tighter confidence bounds used in our experi- ments and extensions to the underlying theory, see and 3
Having proven that T5 is an RCPS, we can simply set T(X) = (X) in 2 : in other words, we set ( (X), f(X)a ((0)axx((())))) 9g (X)a), and of (X), f(X)a a X (g1 (X)a
Visualizing uncertainty intervals in image space
We briefly describe our method for visualizing latent-space uncertainty intervals. In order the effect of a single semantic factor, we set it to either the lower or upper quantile and hold the other factors fixed to the point prediction. More specifically, for particular dimension d {1, D}. define the following vector:
2 =
9k cal (X)a: [(X)a4)...,
(12)
We visualize the lower and upper quantiles in image space as G(22) and G(2 4 respectively. Since each semantic factor corresponds to attribute, visualizing the lower and upper quantiles per-factor gives interpretable meaning to the latent-space intervals. For example, Figure | the images of the child smiling give range of possible expressions the model believes are consistent with the underlying image.
Experiments
3.1 Dataset descriptions
FFHQ We use the StyleGAN framework pretrained using the Flickr-Faces-HQ (FFHQ) dataset [25 FFHQ is a publicly available dataset consisting of 70,000 high-quality images at 1024 x 1024 resolution. The data used to train the quantile encoder and for the experiments in this section sampled from the generator pretrained on FFHQ.
5

---

HH We use the CelebA-HQ dataset [23] to demonstrate the effectiveness of our approach on real world data. The dataset contains 30,000 high-quality images of celebrity faces. Since the real dataset is used only for evaluation, we use images from standard test split.
CLEVR. In order have controlled setup where can easily identify disentangled factors of variation, we generate synthetic images of objects based on the CLEVR dataset [22]. This dataset provides a programmatic way of generating synthetic data by explicitly varying specific semantic factors. We create a synthetic dataset by varying {color, shape) and fixing the other factors such as lighting. material and camera jitter. 3.2 Experimental setup
Model architectures In all our experiments. we use the StyleGAN2 [24] framework for the generator architecture G. For the experiments involving faces, we use the pretrained model available from the official repository. For the CLEVR-2D experiments, we train a simpler variant of StyleGAN2 generator from scratch. For the quantile regression, we use standard architecture: the encoder network consists of ResNet-50 backbone [99] with the final layer branching into the point prediction and conditional quantiles. The branching module is a standard combination of convolution and activation blocks followed by a fully connected layer of the expected output dimension; we call these the heads of the model. Specific details of the model architecture are provided in the supplementary material.
Model training We start by pretraining the generative model or acquiring an off-the-sheil pretrained generative model for the task at hand. In generative models such as StyleGAN, the style space that offers fine grained control over image attributes, is very high dimensional. From this high dimensional space, we extract only the disentangled dimensions following previous work on style space analysis [42] In order to better focus the encoder's 's capacity only on the disentangled dimensions, we mask out the irrelevant dimensions for applying the quantile loss. However, the pointwise loss in (6) is applied to the full style vector to ensure that the pointwise prediction is able to match the true latents accurately, while the quantile heads focus on learning variablity only in the disentangled dimensions.
During the encoder training 2.2. the generative model G is held frozen and only the parameters of the encoder E' are updated. The point prediction and conditional quantile heads are trained jointly with the Ranger optimizer [42] and a flat learning rate of 0.001 for all our experiments. The hyperparameter weights (Eq(8) are set C1 c2 10.0.
For the image super-resolution training. we augment the input dataset by using different levels of downsampled inputs, i.e., take the raw input and apply random downsampling factor from {1, 8, 16, 32} and resize it to the original dimensions. For the image inpainting task, we vary the difficulty by choosing a random threshold to create the mask lower thresholds implies fewer pixels are masked and vice-versa. The mask is concatenated to the image resulting in a C' + 1-channel input to the encoder, where C is the number of image channels. The detailed description of the mask generation procedure can be found in the supplementary material.
Calibration and Evaluation For both the synthetic object experiments with CLEVR and face experiments with FFHQ, we train the quantile encoder on data points sampled from the latent space of the pretrained generative model. This ensures that we have access to the true latents that resulted in each image. We generate 100k samples per model and generate a random 80- 10-10-10 10 split for training, calibration and validation.
3.3 Findings
In the following experiments, we explore different properties of our intervals. The problem types include image super-resolution and image inpainting. The risk level a and the user-specified error threshold 8 are fixed to 0.1, unless specified otherwise.
3.3.1 Producing semantic uncertainties
Goal, We qualitatively verify that the proposed approach outlined in Section 2 produces visually meaningful uncertainties.
Description. We train quantile encoder with Algorithm 1 Then we generate n = 5000 images by sampling latents and propagating them through the encoder-generator combination. We use these images as calibration data for the procedure in 2.3 Finally, we randomly sample new test point, pass it through the calibrated encoder, and form the uncertainty intervals in image space in Section 2.3

---

encoder
generator
encoder
generator
input
prediction
input
prediction
semantic factor
intervals in latent space truth prediction uncalibrated
endpoints visualized in image space
semantic factor
intervals in latent space truth
endpuints visualized in image space
upper
upper
prediction
uncalibrated
glasses
color
Hel
bair color
shape
HH
Figure 3: Semantically meaningful uncertainty intervals produced by our method on example images sampled from the generator trained on the FFHQ dataset (left) and CLEVR dataset (right). The corrupt image is provided as input to the encoder which outputs a pointwise prediction and quantile predictions for each style dimension. We plot the calibrated and uncalibrated intervals as well as their visualizations in image-space. [Best viewed in color, zoom in for detail.]
Results. The results are illustrated in Figure 3 on images sampled from the generator trained on FFHQ (left) and CLEVR (right). In case of the FFHQ image, the person wearing glasses in the lower quantile image and not in the upper; hence, the model is not certain that the person is wearing glasses. The model also expresses some uncertainty about the amount gray versus brown hair. This outcome was predictable from the input image, where the fact that the person is wearing glasses is obvious, and there is some hair color ambiguity. The results on the CLEVR dataset are analogous. The lower and upper quantile images yield similar colors, which is predictable from the blurry input. The model predicts that both cylinder and sphere would consistent with this blurry input. The calibrated quantiles cover the ground truth color value, while the uncalibrated ones do not.
3.3.2 Experiments with real data
factor
calibrated endpoints
factor
effect of calibration
lower
upper
before
after
smile
smile
input
hair
hair
prediction
glasses
glasses
ground truth
Figure 4: Uncertainty intervals for real data We demonstrate the outputs our approach for a real image sampled from the CelebA-HQ faces dataset. [Left] The input is corrupted by downsampling the true image by a factor of 32. Though the prediction does not match the ground truth, the uncertainty captured by the quantiles remains meaningful. [Right] For the same input, we show the effect of our calibration procedure by visualizing the edges of the quantile intervals before and after calibration. Observe that in some cases, such smile and glasses, there clear visible changes due to the calibration approach. In other cases such as hair, the changes are not visually explicit. More such examples can be found in the supplementary material. [Best viewed in color. Zoom in for detail]

---

Goal. We calibrate our encoder with a combination of fake data sampled from the pretrained StyleGAN model and real data sampled from the CelebA-HQ face dataset for the image resolution task. The objective is to test to what extent we can extract meaningful uncertainty intervals for real data without explicitly training with them.
Description The difficulty with calibrating using real data is that the targets i.c the true style vectors that generated the real images are available to us. To circumvent that difficulty, back-project the real images into the latent space of the GAN using existing approaches to GAN inversion 24 Even though the mapping from the GAN latent space to the output optimization procedure is able to visually match the appearance of the real images convincingly. This procedure provides us with the estimated "true" style vectors for each image in the real calibration set. We combine this with the generated data and perform the RCPS calibration procedure as described in Note that, our encoder was only trained on generated data i.e. the same model as used in 3.3.1 we use the real data only for the calibration procedure.
one-one, the back paojeprroe
Section 2.3
Section
Qualitative results The predicted quantiles and the effect of the calibration procedure for a real test image drawn from the Ce dataset is visualized in Figure 4 The pointwise prediction of the encoder does match the identity of the true image due the high level of input corruption. However, using our approach, we are able to visualize the predicted uncertainty intervals in meaningful manner. The visual effect our calibration procedure demonstrated on the right. Notice that some attributes such as smile and glasses have a visible change but others do not. The change due the fact that the calibration procedure adjusts the intervals in such a way to satisfy the required coverage guarantee.
3.3.3 Exploratory results with purposeful corruptions
corrupted input
pointwise prediction
upper quantile
corrupted input
pointwise prediction
lower quantile
upper quantile
quantile
Figure 5: Visualizing adaptivity [Left] A random mask applied to the same input image in each row. When there is no mask (1st row), the lower and upper quantiles are extremely close to the pointwise prediction. As we increase the regions that are being masked, the intervals predicted by the quantile encoder expand, as indicated by the variability on the lower and upper quantile predictions. [Right] We show the results the encoder on two sets of images. The corruption intensity varied across each set, the input image in the top row is not corrupted while the input in the bottom row is under-sampled by 16x. In both case, we can observe that the most diverse prediction in the bottom row where the input is corrupted the most. [Best viewed in color. Zoom in for detail]
Goal. We probe our procedure to see if will have the expected qualitative behavior.
Description of experiment. We sampled images from the held out set of the FFHQ pretrained GAN and applied purposeful corruptions to check if the resulting quantile estimates had semantic meaning. Both image resolution and image masking were used as corruption models. We qualitatively analyze

---

the results by visualizing the predictions. We include quantitative measurement of variability estimated using image based metries the supplementary material.
Qualitative results. Figure 5 shows the results of this experiment for Image inpainting (left) and super-resolution (right). In the inpainting case, when nothing is masked, the quantiles roughly identical. When the eyes are masked, the quantiles indicate the model does not know if the person was wearing glasses. When the mouth masked, the model expresses uncertainty as whether the expresseese
woman is showing her teeth in the smile. Finally, when almost everything is masked, the quantile images are very different, representing individuals with entirely different identities. Similar behavior can be observed in the super-resolution case. The results are shown for two separate inputs; in both cases, the input in the top row is uncorrupted while in the bottom row, it is undersampled by 16x. The model is able to predict almost perfectly in the absence of corruption. The quantile predictions are extremely close as well. the presence of corruption, both the pointwise prediction is off (as expected) and the quantile edges display much higher variability including in attributes such as hair shape, glasses, facial hair and perceived gender. The results from both these experiments point to an expected qualitative behavior of our proposed approach: the model exhibits more uncertainty with increased information loss at the input.
4 Interval sizes function of problem difficulty
Goal. We seek to construct intervals that adapt to the uncertainty of the input, i.e. result in lower values for easier inputs and higher values for harder inputs. this experiment we verify quantitatively how informative our intervals are as function of increasing input corruption.
FFH Super-resolution
CLEVR Super-resolution
medium
Ilstogram count
Helogram count
-0.90
0.00
0.50
0.75
Set size
Set size
Figure 6: Adapting to varying corruption levels: Distribution of set-sizes for different input corruption levels for super-resolution on FFHQ and CLEVR.
Description of experiment. To simulate varying levels of corruption for image super-resolution, create the difficulty levels feasy, medium, hard] that correspond to {1x, 8x, 32x} downsampled versions. All downsampled versions are resized to the same dimensions before presenting to the encoder. The results computed CelebA-HQ dataset. In order to obtain the set size for each input, we scale the quantile width using the threshold obtained after the RCPS calibration procedure.
set of 2000 images sampled from the validation split of the
Results. Figure 6 shows the set sizes for the super-resolution corruption model as a function of problem difficulty two datasets, FFHQ and CLEVR. As expected, the set sizes increase with increasing problem difficulty indicating increasing uncertainty as corruption level increases.
5 Related Work
GANs for Inverse problems. The remarkable image generation properties of recent approaches such as BigGAN [T]] and StyleGAN 24 has led to the increasing use of these models to solve inverse problems relating image restoration such as image super-resolution and completion. All prior methods that use GANs for inverse problems such as the ones that use the pretrained generative model as image prior [6 9 38 36) the ones that train encoder model to project the input

---

into the generator's 's latent space [5]] 42 45 41 on the uncertainty level of the input.
focus on the accuracy of the point estimate but not
GA for Interpretability. Despite providing no guarantees of image likelihood, unlike others such Normalized Flow (26] Score-based models [52), GANs have been used to develop interpretable approaches to image generation. The widespread use of GANs as opposed to other generative models in the interpretability is done due the availability of an disentangled latent space [8] 49 a property we utilize in our work.
which is
Quantile Regression. Quantile regression was first proposed in 33). Since then, many papers have 39), medical research [7]. and more.
used the technique, applying it to machine learning [21 37
Most relevant to is conformalized quantile regression 4 which gives quantile regression
marginal coverage guarantee using conformal prediction. Our work instead uses risk-controlling prediction sets, different distribution-free uncertainty quantification technique.
Conformal prediction and distribution-Free uncertainty quantification. At the core of our proposed method is the distribution-free marginal risk-control technique studied in [8] and [3]. These ideas have their roots in the distribution-fret marginal guarantees of conformal prediction, proposed in [48.. Conformal prediction is flexible method for forming prediction sets that satisfy a marginal coverage guarantee, under no assumptions besides the exchangeability of the test point with the calibration data [48] 47 34 35| 33 12. Conformal prediction has been studied in computer vision 22 12
natural language 14, drug discovery [5], criminal justice [00] and more.
We are not aware of work applying the notions of conformal prediction and quantile regression to the latent space generative models.
Conclusion
Experiments indicate that with an appropriately disentangled model, latent space uncertainty intervals express a useful semantic notion of uncertainty previously unavailable in computer vision. Limitations of our method include that the calibration data must be reflective of the data distribution, and that we assume access to disentangled latent space. We see our work as part of a larger tapestry of results in generative models, and our technique will remain applicable disentanglement and backprojection. Furthermore, by introducing a notion of statistical rigor to GAN based approaches, our work creates path for application of such models to safety critical settings.
progress gets made
7 Ethics
The ethics of generative modeling itself has been called into question given recent events, e.g., the development of deep fakes. Nonetheless, likely be positive. The techniques herein do not change the predictions of a generative model; they simply provide a calibrated notion of its uncertainty in relevant semantic space. Thus, the standard criticism of generative modeling- ttat will enable widespread deep fakes-is not applicable. Furthermore, we expect having statistically valid and semantically rich notion of uncertainty will provide a sobering reliability assessment of these models, perhaps mitigating the chance of harmful failures. Finally, although use face datasets due their ubiquity in this literature, have attempted to ethically treat topics like gender and race where they arise.
believe the downstream consequences of this work will
Acknowledgements
A. N. A. was supported by NSF GRFP. S. B. was supported by the Foundations of Data Science Institute and the Simons Institute. Y. R. was supported by the Israel Science Foundation (grant No. 729/21). Y. R. thanks the Career Advancement Fellowship, Technion, for providing research support. S. S. and P. I. 's research for this project was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750- 19-2- 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. S. S. also acknowledges the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing compute resources that have contributed to the results reported in this work.
10

---

References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432 4441, 2019.
[2] Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution- frree uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
[3] Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint 1r.iv:21 2021.
[4] Anastasios N Angelopoulos, Stephen Bates, Tijana Zrnic, and Michael Jordan. Private prediction sets. arXiv preprint arXiv:2102.06202, 2021.
[5] Anastasios N Angelopoulos, Amit P Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution- uncertainty quantification and applications in imaging. arXiv preprint arXiv:2202.05265, 2022.
[6] Anastasios Nikolas Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Un- certainty sets for image classifiers using conformal prediction. In International Conference Learning Representations (ICLR), 2021.
[7] Peter Armitage, Geoffrey Berry, and John Nigel Scott Matthews. Statistical methods in medical research. John Wiley & Sons, 2008.
[8] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I Jordan. Distribution-free, risk-controlling prediction sets. arXiv preprint arXiv:2101.0277 2021.
[9] David Bau, Hendrik Strobelt, William S. Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. CoRR, abs/2005.07727, 2020.
[10] Richard A Berk and Arun Kumar Kuchibhotla. Improving fairness in criminal justice algorithmic risk assessments using conformal prediction sets. arXiv preprint arXiv:2008.1 /664, 2020.
[11] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[12] Maxime Cauchois, Suyash Gupta, and John C. Duchi. Knowing what you know: Valid and validated confidence sets in multiclass and multilahel prediction. Journal of Machine Learning Research, 22(81):1-42) 2021.
[13] Probal Chaudhuri. Global nonparametric estimation of conditional quantile functions and their derivatives. Journal of multivariate analysis, 39(2):246-269, 1991.
[14] Adam Fisch, Tal Schuster. Tommi Jaakkola, and Regina Barzilay. Efficient conformal prediction via cascaded inference with expanded admission. arXiv preprint arXiv:2007.03114, 2020.
[15] Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Few-shot conformal prediction with auxiliary tasks. arXiv preprint 200.:2200,00888,, 2021.
[16] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation:
Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[17] Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang. Collabo- rative learning for faster stylegan embedding. arXiv preprint arXiv:2007.01758. 2020.

---

[18] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable GAN controls. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Yotam Hechtlinger. Barnabás Póczos, and Larry Wasserman. Cautious deep learning. arXiv preprint arXiv: 1805.09660 2018.
[21] Changha Hwang and Jooyong Shim. A simple quantile regression via support vector machine.
In International Conference on Natural Computation, pages 512-520. Springer, 2005.
[22] Justin Johnson, Bharath Hariharan, Laurens der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1988-1997. IEEE Computer Society, 2017.
[23] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[24] Tero Karras, Mika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[25] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4401-4410. Computer Vision Foundation IEEE, 2019.
[26] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. In Samy Bengio. Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianche and Roman Garnett, editors, Advances Neural Information Processing Sys- tems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 10236-10245, 2018
[27] Roger Koenker. Quantile Regression. Cambridge University Press, 2005.
[28) Roger Koenker. Additive models for quantile regression: Model selection and confidence bandaids. Brazilian Journal of Probability and Statistics, 25(3):239-262, 2011.
[29] Roger Koenker. Quantile regression: 40 years on. Annual Review of Economics, 9:155-176. 2017.
[30] Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: Journal of the Econometric Sociery, 46(1):33-50, 1978.
[31] Roger Koenker, Victor Chernozhukov, Xuming He, and Limin Peng. Handbook of Quantile Regression. CRC press, 2018.
[32] Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 133-)1143155,, 2001.
[33] Jing Lei. Classification with confidence. Biometrika, 101(4):555-76, 10 2014.
[34] Jing Lei, Alessandro Rinaldo, and Larry Wasserman. conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence, 74:29 2015.
12

---

[35] Jing Lei, James Robins, and Larry Wasserman. Distribution free prediction sets, Journal of the American Statistical Association, 108(501):278-255) 2013.
[36] Razvan V. Marinescu, Daniel Moyer, and Polina Golland. Bayesian image reconstruction using deep generative models. CoRR, abs/2012.04567, 2020.
[37] Nicolai Meinshausen and Greg Ridgeway. Quantile regression forests. Journal of Machine Learning Research, 7(6), 2006.
[38] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: self-supervised photo upsampling via latent space exploration of generative models. In 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2434-2442. Computer Vision Foundation IEEE, 2020.
[39] Alexey Natekin and Alois Knoll. Gradient boosting machines, a tutorial. Frontiers in Neuro- robotics, 7:21, 2013.
[40] Luis Oala, Cosmas Heiß, Jan Macdonald, Maximilian März, Wojeiech Samek, and Gitta Kutyniok. Interval neural networks: Uncertainty scores. arXiv preprint arXiv:2003. 2020.
[41] Guy Ohayon, Theo Adrai, Gregory Vaksman, Michael Elad, and Peyman Milanfar. High perceptual quality image denoising with a posterior sampling cgan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1805-1813, 2021.
[42] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: A stylegan encoder for anage-to-imaag translation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 2287-2296. Computer Vision Foundation IEEE, 2021.
[43] Yaniv Romano, Evan Patterson, and Emmanuel Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32, pages 3543-3553, 2019.
[44] Yaniv Romano, Matteo Sesia, and Emmanuel J Candès. Classification with valid and adaptive coverage. arXiv preprint arXiv:2006.00 2005:22006.02544, 2020.
[45] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or Designing an encoder for stylegan image manipulation. ACM Trans. Graph., 40<4):133:11133:14 2021.
[46] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Deep image prior. CoRR, abs/1711.100255, 2017.
[47] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, New York, NY, USA, 2005.
[48] Vladimir Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. In International Conference on Machine Learning, pages 444-453, 1999.
[49] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 12863- 12872. Computer Vision Foundation / IEEE, 2021.
[50] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[51] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain GAN inversion real image editing. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII, volume 12362 of Lecture Notes in Computer Science, pages 592-608. Springer, 2020.
[52] Roland S. Zimmermann, Lukas Schott, Yang Song, Benjamin A. Dunn, and David Klindt. Score-based generative classifiers. CoRR, 10.001 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We explicitly state the assumptions in Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Explicitly address this in Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Explicitly address this in Section 7
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Ys]] The proposition
stated in Section 2.3 We also state the assumptions Section 6 For more details, we redirect the reader to previous work where we derive the framework from. ( g
and 3
(b) Did you include complete proofs of all theoretical results? [No] The proposition is stated in Section 2.3 For complete proofs, we redirect the reader to previous work where we derive the framework from. [ and (3]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Training details are provided in Section 3 More complete details about the model architecture are provided in the supplementary material. The datasets we use are publicly available, we include references to that in Section 3 The code will be released in the near future.
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] While expect there to be some randomness, the behavior of the encoder is consistent across different runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We cite the dataset creators who have included their respective licenses along with the dataset. We ensure that we follow the terms of the license i.e. use the data for research use only.
(c) Did you include any new assets either the supplemental material or as URL?? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No] Since we use dataset that comes with a reseach license, we defer such concerns to the dataset creators.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] Since we use dataset that comes with reseach license, we defer such concerns to the dataset creators.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "eHYhFftsaSC", "reviewer": "Reviewer_kKQD", "summary": "The paper proposes an architecture to predict uncertainty intervals for (pre-learned) latent coordinates. It starts off with an existing disentanglement architecture, samples artificial images from latents, and noisifies them to get training samples for their method. The resulting training data is then fed to a quantile regression network to learn both bounds of the uncertainty interval per coordinate. I was expecting a disentaglement paper with some disentanglement pureness guarantees, so my review is an educated guess.\n\nThe training procedure optimizes a four-component loss with hyperparameters ($c$ and the potentially non-unit weights on the quantile losses) and a fine-tuning (calibration) stage with its own loss. It also depends on third-party embeddings and identified causal dimensions in the latent space.\n\n\n", "questions": "Some more explanations needed:\n- how hard is it to tune the $c$ for a task at hand?\n- 120: what would be the calibration loss function here?\n- 129: why are you doing scanning and not binary search?\n- 171: please add a few words on how to pick disentangled dimensions\n- Ablation would be nice to see if only for the setting w/o calibration, to get a feeling of how visually important it is.\n\n\nMinor:\n- 64: lacking space after i.e. \n- 83: something wrong with the sentence (\"risk-controlling prediction\" appears twice)\n- 127: what does \"drive delta to zero\" mean, if it's user-specified? Probably the whole last term is meant.\n- Figure 5: explain the x-axis \"set size\" (not clear why it's int and float in different images)", "limitations": "Overall, the method appears to produce what it is trained for but it's not clear from the draft when to apply it, and how costly the procedure is given the multi-phase learning (and HPO) and data generation.", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Predicts a trainable bound on the probability that the latent deviates far from its point-estimate over the assumed noise\n\nMain remarks/weaknesses:\n- After reading the paper it's not very clear, what would be a killer application for this method and what would be a hypothetical actionable workflow after getting such predictions.\n- On the theoretical side, it seems to refer to the properties of quantile regression loss, and appears to echo the result that enough data and finite hypotheses classes (discretized NN weights) are PAC-learnable (here, with a particular loss function) -- hence the ability to satisfy $\\alpha$ and $\\delta$ bounds by generating more data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aztTk30--M", "reviewer": "Reviewer_GBAe", "summary": "The authors propose to combine disentangled representations of generative models (GANs) and quantile regression for stochastic encoding of corrupted data. Thanks to quantile prediction, the encoder predicts uncertainty intervals instead of single points, which allows to get a more sound representation of corrupted data.  Also, the authors use calibration algorithms to ensure, with a certain confidence threshold, that the predicted quantiles contain the target value. ", "questions": "Beta-VAEs [2] are known to be disentangled. In the VAE framework, each point is encoded as a Gaussian distribution. This model naturally induces uncertainty estimation. Moreover, there exists some variants of VAEs, like Beta-VAEs, known for their disentangled representations. How does your framework could be compared to such models?\n\n[2] Higgins et al.  \"beta-vae: Learning basic visual concepts with a constrained variational framework\". International Conference on Learning Representations. 2017.\n", "limitations": "The authors addressed the limitations. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* 1) Original idea of using generative models to visualize uncertainties on corrupted data, and of combining disentangled representations of GANs and encoding with a quantile regression loss. \n* 2) An advantage of the approach is the interpretability of the results and the possibility to reuse large models. \n\nWeakness\n* 1) There are no experiments on real attributes. In this setting (real attributes), the errors can come from: a) lack of disentanglement in the pre-trained generative model b) the proposed method of encoding with quantile regression and calibration. It would be more convincing to have studies on real attributes, and see the relative impact of the two types of errors. \n* 2) The hypothesis that a) each dimension of the generative model corresponds to a separated factor of variation and b) that these factor of variations are independant, is a very strong hypothesis.\n* 3) There is no ablation study on the impact of the different loss terms. For example, what happens without the identity loss? \n* 4) The idea is close to previous works that leverage generative models for inversion tasks. For example, [1] uses a similar procedure of training an encoder on masked images with frozen and pre-trained generator. Although the proposed method is different because it proposes learned quantiles, [1] and similar works should be discussed in related work section.\n* 5) Small typos in the definition affect the readability, especially the mathematical details. L82: \"$(\\alpha,\\delta)$ risk-controlling prediction set if $(\\alpha,\\delta)$-Risk-Controlling Prediction Set if\". In Equation (5), $Y$ seems to be used instead of $Z$.  L202, \"the the\".\n\n[1] Chai, Lucy, Jonas Wulff, and Phillip Isola. \"Using latent space regression to analyze and leverage compositionality in GANs.\" International Conference on Learning Representations. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K4puAIKTTEI", "reviewer": "Reviewer_3nDa", "summary": "This paper proposes a method to obtain the uncertainty intervals of the semantic latent variables in the generative model. The method takes a corrupted image input and then predicts each semantic factor along with an uncertainty interval. These uncertainty intervals can propagate through the generative adversarial network (GAN) to visualize the uncertainty in image space. To fulfill the goal, the author proposes two steps: (1) use the quantile regression to output a heuristic uncertainty interval for each element in the latent space; (2) calibrates these uncertainties such that they contain the true value of the latent for new, unseen input. Experiments show that the proposed method can produce visually meaningful uncertainties and the model exhibits more uncertainty with increased information loss at the input image. ", "questions": "The questions and suggestions are already included in “Strengths and Weaknesses”. The reviewer may expect the author to respond to the weaknesses listed in the “Strengths and Weaknesses” to consider changing the rate.", "limitations": "The author has already stated three limitations of the present work in lines 289-294 on Page 9, where these limitations are all related to the core assumptions of the proposed method and indispensable for the present work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \nTo the best of my knowledge, this is the first work to integrate the distribution-free uncertainty quantification with the latent space analysis in GAN. The proposed method is novel and original given the considered application and the new training pipeline to achieve the uncertainty interval by means of quantile regression, although the core of the proposed method is originated and motivated by [7] and [2]. \n\nQuality and Clarity: \nOverall, the author provides a clear description of the proposed method with detailed and straightforward illustrations to help the reviewer understand the paper. \nWeakness:\nHowever, the introduction of the distribution-free uncertainty quantification and the Risk-Controlling Prediction Set (RCPS) is not sufficient. As being the core underlying theory to backup the feasibility of the proposed method, the author should provide more explanation to convey the connection clearly and intuitively between the theory and the considered application. This is crucial since the potential reader may not have enough background on the distribution-free uncertainty quantification and the RCPS in [7]. The current writing of lines 79-90 on page 3 and section 2.3 seems to assume that the potential reader has already known the related techniques, which may hurt the readability of the paper. For line 132 on Page 5, it would be better to include more details about the exact section or paragraph for the potential reader to locate. \n\nMinor Format Problem:\nThe Figure 5 caption is too close to the text in line 244, which makes it hard to interpret.\n\nSignificance:\nThe qualitative gives several examples of the predicted semantic uncertainties based on the controlled corrupted input image. \nWeakness:\nHowever, it is hard to say whether these results are general enough to support the claims in the paper, and the author may want to include more examples to give a more comprehensive demonstration.  \n\nMoreover, the author may want to show the visual difference between the lower and upper quantiles with and without calibration to demonstrate that the proposed method in Section 2.3 works as expected.\n\nAlthough the proposed method provides a first step to quantify the uncertainty of the latent variable in GAN, the present work relies strongly on the accessibility of the disentangled latent space and thus the present work can only show the feasibility of GAN. Moreover, the concrete application of such an uncertainty calibration method in generative models is not clear. The author may want to provide some illustrations that what kinds of existing applications related to the generative models they expect the proposed method may provide some benefits.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept