Structuring Uncertainty for Fine-Grained Sampling in Stochastic Segmentation Networks
Authors: Frank Nussbaum
Jakob Gawlikowski
Julia Niebling
Venue: NeurIPS 2022 Accept (2022)
Keywords: stochastic segmentation
sample control
uncertainty representation
factor model
factor rotations
aleatoric uncertainty

Abstract: In image segmentation, the classic approach of learning a deterministic segmentation neither accounts for noise and ambiguity in the data nor for expert disagreements about the correct segmentation. This has been addressed by architectures that predict heteroscedastic (input-dependent) segmentation uncertainty, which indicates regions of segmentations that should be treated with care. What is missing are structural insights into the uncertainty, which would be desirable for interpretability and systematic adjustments. In the context of state-of-the-art stochastic segmentation networks (SSNs), we solve this issue by dismantling the overall predicted uncertainty into smaller uncertainty components. We obtain them directly from the low-rank Gaussian distribution for the logits in the network head of SSNs, based on a previously unconsidered view of this distribution as a factor model. The rank subsequently encodes a number of latent variables, each of which controls an individual uncertainty component. Hence, we can use the latent variables (called factors) for fine-grained sample control, thereby solving an open problem from previous work. There is one caveat though--factors are only unique up to orthogonal rotations. Factor rotations allow us to structure the uncertainty in a way that endorses simplicity, non-redundancy, and separation among the individual uncertainty components. To make the overall and factor-specific uncertainties at play comprehensible, we introduce flow probabilities that quantify deviations from the mean prediction and can also be used for uncertainty visualization. We show on medical-imaging, earth-observation, and traffic-scene data that rotation criteria based on factor-specific flow probabilities consistently yield the best factors for fine-grained sampling.

Structuring Uncertainty for Fine-Grained Sampling in Stochastic Segmentation Networks
Frank Nussbaum" Friedrich-SShhiee University Jena Fürstengraben 1, 07743 Jena, Germany and German Aerospace Center (DLR) Institute of Data Science Mälzerstraße 3-5, 07745 Jena, Germany frank. nussbaum@uni jeaaiddeeeeeeeeee
Jakob Gawlikowski" Technical University of Munich Arcisstraße 21, 80333 Munich, Germany
and German Aerospace Center (DLR) Institute of Data Science Mälzerstraße 3-5, 07745 Jena, Germany jakob gwwl www.wiikwwwii@di.. .de
Julia Niebling
German Aerospace Center (DLR)
Institute of Data Science
Mälzerstraße 3-5, 07745 Jena, Germany wuw.a..ieelinngei.......
Abstract
In image segmentation, the classic approach of learning a deterministic segmenta- tion neither accounts for noise and ambiguity in the data nor for expert disagree- ments about the correct segmentation. This has been addressed by architectures that predict heteroscedastic (input-dependent) segmentation uncertainty, which indi- cates regions of segmentations that should be treated with care. What is missing are structural insights into the uncertainty, which would desirable for interpretability and systematic adjustments. In the context of state-of-the-an stochastic segmen- tation networks (SSNs), we solve this issue by dismantling the overall predicted uncertainty into smaller uncertainty components. We obtain them directly from the low-rank Gaussian distribution for the logits in the network head of SSNs, based on previously unconsidered view of this distribution The rank subsequently encodes a number latent variables, each which con- trols individual uncertainty component. Hence, (called factors) for fine-grained sample control, thereby solving an open problem from previous work. There is one caveat though-factors factors only unique up orthogonal rotations. Factor rotations allow us to structure the uncertainty in a way that endorses simplicity, non-redundancy, and separation among the individual uncertainty components. To make the overall and factor-specific uncertainties at play comprehensible, introduce flow probabilities that quantify deviations from the mean prediction and can also be used for uncertainty visualization. We show medical-imaginn carth-observation, and traffic-scene data that rotation criteria based on f'actor-specific flow probabilities consistently yield the best factors for fine- ane-grained sampling.
factor model.
the latent variables
"both authors contributed equally
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
Semantic Segmentation the computer vision task of assigning class each pixel of image.
Examples for popular applications are the segmentation of medical images []] 2 39, land-cover
classification earth observation 28 37 42], and segmentation of imagery taken by autonomous vehicles 14, 25 43 Semantic segmentation tasks are affected by heteroscedastic (input-dependent)
aleatoric uncertainty 26 27 that is also called data uncertainty. Aleatoric uncertainty can emerge in form of label noise in the training data, differing expert opinions about the true segmentation, or ambiguity already contained in the data, for example, caused by technical restrictions like the image resolution [16 20 27].
To account for the prevailing aleatoric uncertainty, various probabilistic architectures have been proposed 3 21 24 28 31. The work [ demonstrates that in general, uncertainties predicted
by probabilistic segmentation architectures correlate positively with estimation errors, including Nevertheless, deterministic
those obtained from ensemble methods (23 29 and MC-dropout [15]
This may be because for practitioners it is often not clear how to take advantage of the predicted uncertainties. For instance, limited number of sampled segmentations usually does not represent the predicted uncertainty well. As shown in Figure (top), overview plots for the predicted uncertainty like entropy [11 27 can be generated. They indicate areas of high uncertainty, where practitioners should take care. However, they do not convey pixel-wise correlations of uncertainty, that is, how changes in the segmentation of one region an image affect changes in another. Moreover, overview plots cannot explain the overall uncertainty in terms of smaller uncertainty components, which would be desirable towards an interpretation of the uncertainty following the principles of problem decomposition and wwvide-and-conuuee [30]
segmentation architectures 7 351 are still pre-dominantly used 2 44
Image
Ground truth
Mean prediction
FP full
Entropy
Gressland
Croslands
Snowroe
Barren
Factor
Factor
Factor
Factor
Factor
Factor 6
Factor
Factor
Factor 9
Factor 10
IP-Quartiaa Unrotated
Figure 1: Segmentation uncertainty for an image from the SENI 2MS data set (37]. Top row: image, ground truth with coarse labels, mean prediction, overview plot for the predicted uncertainty using full flow probabilities (FP), entropy (bright colors mean high entropy). Bottom rows: Factor-specific flow probabilities for the unrotated factors from the network output (top) and rP-Quartimax factors (bottom). Bright colors mean that the prediction changes with high probability to the class indicated by the color. Flow probability rotations like FP-Quartimax greatly reduce redundancy. Here, they yield only three distinguished factors that encode larger uncertainty components and can used as the main controls for ime-grained sampling, which was requested in [31
In the context of the recently introduced stochastic segmentation networks (SSNs) 3 that we explain below, the lack of understanding the overall uncertainty in terms of independent components connects to problem that was also observed by the authors of [31]. They pointed out in their accompanying demo that 'fine-grained sample control' is still missing. Indeed, the identification of independent or at least reasonably distinguished components of uncertainty represents natural solution to this problem. We seek such components for SSNs with the goal of manipulating them individually for generating and fine-adjusting segmentations.

---

SSNs model uncertainty via low-rank multivariate Gaussian distribution on the logits, that is, on the network output before the softmax is applied, see Section [ Surprisingly, has not been made use of the fact that this uncertainty model itself offers a straightforward way for distinguishing between uncertainty components. This may be because the low-rank model was originally proposed for reducing the number of parameters. Here, we center our approach around the semantics of the low-rank model as factor model. Factor models structure the overall uncertainty into individual components of uncertainty, each of which is governed by a single latent variable (called factor). Therefore, we solve the open problem from [T by using the latent factor variables as control variables for a systematic exploration of the predieted heteroscedastic segmentation uncertainty. For the best result, however, some additional work is necessary: As is known from exploratory factor analysis [3 40), the latent variables in factor models only unique up orthogonal rotations. Hence, they should be rotated for increased interpretability 6 40), which in our case amounts to generating more useful controls.
Good controls encode uncertainty components that are simple, non-redundant, and separable in the sense that they affect distinguished image regions classes. To evaluate these aspects, in Section we introduce flow probabilities that quantify deviations from the mean prediction, which also enables uncertainty visualization, see Figure I Specifically, we compute factor-specifie flow probabilities that quantify the impact of the uncertainty components encoded by individual factors. In Section 4 fuse factor-specific flow probabilities with classic rotation criteria [6] TC from exploratory factor analysis. In Section we show that these fused criteria generally result in the best possible controls. Note that by-product, computing full flow probabilities for the overall uncertainty also yields new type of overview plot that does not aggregate class-specific information about the uncertainty, see Figure (top).
Before we summarize our main contributions, we would like to emphasize that we do not benchmark SSNs as they have already been proven to produce state-of- satee-of-the-ee art results w.r.t. various metrics, for instance, generalized energy distance to the ground truth distribution 31). Hence, it is safe for us assume that after successful training. SSNs are capable of predicting the aleatoric uncertainty for given input image reasonably well. Based on that our main contributions are:
control variables for the contributions of individual, factor-specific uncertainty components for fine-grained sampling (given by the latent factor variables).
flow probabilities for quantifying and visualizing overall and factor-specific uncertainties,
rotations based on factor-specific flow probabilities, which structure the uncertainty compo- nents and thereby provide simpler, less redundant, and well-separated control variables.
Please find an overview figure of our contributions in Section A of our supplement. Ad- ditionally, we made the code for the proposed methods and experiments available under
2 Factor modeling in stochastic segmentation networks
Stochastic segmentation networks (SSNs) 31 are characterized by modeling the pixel-wise logits in the network head as a low-rank multivariate Gaussian distribution, that is,
p(n x>) A
+ +.()))
Here, 17 R" are the problem with e classes. The parameters of the Gaussian distribution are the mean (xx) E TR" and the covariance matrix, which decomposes into matrix of rank bounded by & with square root T(x) E Raxr and a diagonal matrix (x) E R Rxxn with positive diagonal elements. The parameters of the Gaussian distribution the output of backbone segmentation network with input 77 35
hwe logits for input image of size h x and classification
Originally, the low-rank parameterization was solely introduced as a means for reducing the number of parameters (3]. However, the low-rank covariance model has deeper structural meaning as
a factor model 13 38 40). Factor models are characterized by a typically small number of latent
variables, called factors, that explain all correlations among a larger number of observed variables. In
our case, the joint distribution of the observed logits and the latent factor variables E RR is given by
(1, x) ~ N ((5'
T (
Pr
+ V
I, ))
r
3

---

where I, is the (r x r) identity matrix, and for brevity, we omit the dependence on the input æ in the notation from now on. Here, the interactions of the latent variables with the observed logits are described in the matrix F of factor loadings: Each column contains the loadings of one latent factor variable on the observed logits, yielding structured uncertainty. The loading characteristic becomes clear in the following sampling procedure for the logits from the factor model:
7 la + Tz + 1///2,, where
(O, L,, & NOO,, In)-
This procedure results from sampling from the joint distribution p(n, =) p(z)p(1) z) as follows: First, the latent variables are sampled according to 2 ~ (0, I,). Second, the logits are sampled from the conditional distribution = ~N Tz, V. Subsequently, only the logits are observed.
Sampling the logits as in Equation ! provides control over the contributions of the different latent factor variables. This invites for individual manipulation of the factors, enabling fine-gra tinegrrined sampling. However, as pointed out in the introduction, factors should be rotated beforehand because they are only unique up to orthogonal rotations, see Lemma in the supplement. In particular, orthogonal rotations of the latent factor variables do not change the marginal distribution p(1) of the logits. Indeed, replacing rz by rO for an orthogonal matrix O R in Equation (I) yields an equivalent sampling procedure. Therefore, one way to understand orthogonal rotations that they change the basis of the r-dimensional affine space { + T2 E RR' of the (noiseless) logits, where the basis elements are the columns of the factor loading matrix.
3 Flow probabilities
In this section, we develop the notion of flow probabilities as our main tool for the analysis of factor models in SSNs. Flow probabilities can roughly be understood as probabilities of deviations from the mean prediction. We use flow probabilities for uncertainty quantification and visualization.
3.1 Factor-specific flow probabilities
Because we need understand and assess individual factors, is important to analyze and quantify the uncertainty that is encoded in them. A useful tool for that is to compute factor-wise distributions of class predictions, for which we vary an individual latent factor variable with associated factor loadings (a column of T, later we use the notation T. for specific column of T). We keep the influence of all other latent factor and noise variables fixed to zero. Consequently, for 10, we compute the following expected value:
P= - [ r2)(()d e |0, [[
(2)
Here, E(u++22 + 72) E {0, (w)k)xx is the matrix whose rows correspond to the pixel-wise one-hot encoded class predictions, which are obtained by reshaping the logits μ + yz into shape (wh, c) and then applying an arg max along the class dimension. Note that the class probabilities P can be understood as function of the factor loadings 7 since the mean logits μ are fixed. We solve Equation (2) analytically. For that, we consider pixels separately since for (flat) spatial index i [wh]] = [1...., wh), the probabilities Pik from the i-th row of only depend on its associated mean logits and factor loadings. Specifically, with the definition 9&x (2) -4kk + Vix z for & E [c], for fixed the predicted class is arg max, gss (z). Hence, from Equation
? we get that
/ [[k arg max 9ik (z))p(z)dz,
Pik pTx(7)
k E [c] (1....,c),
where 1 is the indicator function. We solve Equation 3 for binary classification first (k: E (1, 2)). Assuming that only the logits for the class are learned, we set 9n (2) for consistency in Equation (3). Then, with f /4:2 and assuming that -22 * 0, the probability P:2 evaluates as
P12 f 1
f 72 0lp(2ddz
{ (v((-1/)), [1 11- (-H//)),
7<0
7>0"
Here, di the cumulative distribution function of a standard normal random variable. For the last equality, observe that -p/y is the intersection point of the straight line g12(z) μ + 72 with the

---

2-axis 9:1- If O, the arg max is not unique and probabilities can be split. For clarity of the technical exposition, we assume that the arg max is unique the following.
For general multi-class problems, the probabilities Pik: in Equation (3) can be derived from the class-predicte preddrrediiion function 2 + arg max gix (2). In this function, the class prediction can only change at intersection points 2 of two non-parallel straight lines Gik and fik', that is, = (Pit jik') ) (T "ak)). Generally, if class & is predicted for some 2, then all 2 values for which the k-th class is predicted form non-empty interval (Z:k: C R. The end points of this interval can either be -O, an intersection point of gik. or cc. practice, the intervals (Z.R) Ea) can be computed by sorting all intersection points and checking the values of the class-prediction function on the resulting partition of the z-axis. If class is never predicted, we set 2sk = ž = -00. Finally, the class probability is given by Pit (z (2xx). where we use the conventions that (0000) = and (x())) = I. Observe that the formula for binary problems given above is a special case of the one given for Pak here. Overall, we obtain the following result:
Proposition 1. Define Z (Zsk) and Z (3xx) with entries [wh]] and k E [c]. Then, the distribution of predicted classes under variation of the factor with associated loadings "y is given by
P(x) (Z) (Z))
where va applies the cumulative distribution function of a standard normal variable element-wise
Now, to highlight the difference to the prediction from the mean μe, we compure flow probabilities
F(>)= = P(7) - F(x) v/ Z - 1/ Z E(j)) [-1,1]
Positive entries in the k-th column F(r)... indicate that the prediction for the corresponding pixels changes with positive probability from the mean prediction to class k. Based on this fact, factor- specific flow probabilities enable visualizations of the impact of individual factors, see Figure []] (bottom rows). The visualizations are obtained by calculating a mixture of class-specific colors with weights given by the (factor-specific) flow probabilities, see the supplement for details.
As factor-specific flow probabilities represent the real impact of a factor on output segmentations, they will also be key quality assessment of the factors, see Section 4 For future reference, denote by F(T) E [-1, 1](whe)xr the matrix of all factor-specific flow probabilities that is obtained
by concatenating the factor-specific flow probabilities F(T:5) as columns after flattening, where the latent factor variables as control variables
T. is the j-th column of T. Finally, since for fine- grained sampling, it is helpful to also compute one-sided flow probabilities that encode the uncertainty for respectively positive and negative values of the latent factor variable.
the one-sided fuideeffctorr.. factor- cicitto--ppeffccciiii probabilities Corollary 1. Using the notation from Proposition 7 for a factor with loadings y compute as
/ E(u + (z)p(2)) E(R) = (mmax(0. Z)) - (max(0 Z)) - E(H)),
F+ (r)
(0,00)
F (7)))
E(u) (min Z)) + (min(0. Z)) E(H).
(-00,0
3.2 Uncertainty quantification for the full factor model
The idea of computing factor-specifie flow probabilities for uncertainty quantification and visualiza- tion extends to the full factor model. For that, analogous to Equation (2). we compute the distribution of class predictions. However, this time we take the expected value over the full distribution of the logits given in Equation IT)
/ E(u++ "222e)p(2)p(edddde e [0, [[(x))x
pfull = E(n)p(n)n
(4)
The change from the mean prediction E(H) is then given by the full flow probabilities, which we compute as Ffull pfull E(H)) E [11,1] weighted mixtures of class-specific colors yields a new type of overview plot for the uncertainty, see Figure I (top row) for example. Though for our work only by-product, it has the advantage that
Visualizing full flow probabilities as above by
5

---

it does not aggregate information about class-specific uncertainties, in contrast to overview plots like entropy (see also Figure IT top row).
In practice, the integral from Equation 4 is difficult to evaluate. This is because the arg max in E(n) technically amounts to determining maximum of multivariate linear functions. Hence, we approximate the integral using Monte- Carlo integration with i..d. samples z(1), ...... (mm) R" drawn from (0, Ir) and i.i.d. samples e(1),...,, .....y e((m) E IR whe drawn from (0, mhe)) The matrix pfull of class probabilities is thus approximated by
plul
E(u+ 12 + 1//2 E [0, (wk)xx
The matrix f-aal of flow probabilities can be approximated similarly. In the supplement, we show empirically that the diagonal noise term has little impact on the flow probabilities. Hence, we can focus on the structural uncertainty that is induced by the latent factor variables.
4 Factor rotations
As pointed out in Section 2 the latent variables/factors factor models are only unique up to orthogonal rotations. Therefore, it is common practice in exploratory factor analysis to rotate them in order to maximize their interpretability [6 22 38). The factor model in a SSN represents the predicted uncertainty for given imput image, where the factors themselves encode components of the overall uncertainty. We intend to use them as control variables for fine-grained sampling. From that we derive the following quality criteria:
(1) The number of 'relevant' factors should be small, where relevant factors are characterized by having 'significant' effect output segmentations.
(2) Relevant factors should be separable from each other in the sense that they encode distin- guished uncertainty components,
(3) Each area in the input image should be affected by only few factors.
Here, the first criterion ensures that the number of impactful control variables is reduced to a necessary minimum, and the second criterion requires that the corresponding uncertainty components are distinct. Together, the first two criteria discourage factor redundancy. The last criterion reflects the general requirement of sparsity and simplicity that is also found among Thurstone's 's rules [41] for simple structure a factor loading matrix, which the primary goal in exploratory factor analysis [3]. However, in our we rather require simple structure the matrix F(T) of factor-specific flow probabilities (see Section 3.1) since they measure the actual impact of the factors on output segmentations. In Section we evaluate different rotation criteria that we present in the following.
First, we consider classic rotation criteria. Here, for a factor loading matrix = (165) E R"xr, Crawson and Ferguson [10] defined the CF family of rotation criteria:
4xe (T) = (l-
E [0,
lijght
1=1
Fiight
The CF family is a generalization of the widely used orthomax family 17 where the parameter k controls trade-off between row complexity (first sum) and column complexity (second sum). We focus on popular choices: к = 1/n yields an equivalent version of the Varimax criterion [22). which the most used method. Intuitively, it tries to maximize the variance the squared factor loadings. Next, K = 0 yields the Quartimax criterion that minimizes the number of factors needed to explain variable (in our case segmentation uncertainty of pixel). Finally, x = r2n) yields the Equamax criterion that represents combination of Varimax and Quartimax.
Classic rotation criteria do not consider the actual impact of factors on predicted segmentations because they only take the factor loadings I but not the mean into account. Therefore, we incorpo- rate factor-specific flow probabilities into rotation criteria by applying a base rotation criterion on the flow probabilities instead of the factor loadings. Hence, the objective function to be minimized becomes O + q( (TOO) instead of + (TOO). We call the new family of rotation criteria the FP family. For instance, FP- Varimax applies the Varimax criterion on the flow probabilities.

---

Experiments
The purpose of our experiments is to (1) evaluate rotation criteria based on the quality of rotated factors, (2) demonstrate the merits of fine- tnee-griied sample control based on reasonably-rotatee factors.
Data sets and training. First, we use the LIDC data set [1] in its pre-processed version from [28 that contains 2D slices of 3D thorax scans size 128 x 128 pixels. Each slice respectively has four ground truth segmentations from different experts. Second, we use the multi-spectral Sentinel-2 data from the SENI2MS data set (36] with images of size 244 x 244 pixels and coarse labels for semantic segmentation of 10 types of land cover. Third, we use the Cam Vid data set [5], which contains images of road scenes resolution 480 x 360 and is pixel-wise labeled into 11 different classes. Additional details and statistics about the data sets (including splits) can be found in the supplement, where we also detail all training procedures. We respectively use r = 10 our experiments, which accounts for the varying uncertainty in different images and has also been used in [3.. We would like to emphasize again that we do not benchmark SSNs since they have already shown be state of the art 3 I1.
24
For examples of uncertainty predictions, see Figure I Figure 2
and the supplement.
Image
Ground truth Mean pred.
FP full
Image
Ground truth
Mean prediction
FP full
Figure 2: Left: LIDC example including ground truth (split to show annotations of four experts), mean prediction, full flow-probability (FP) overview plot for the uncertainty. Right: Cam example with uncertainty mostly at class borders. More examples can be found in the supplement.
Computational aspects. We used Python 3.7, particularly with the libraries PyTorch 1.// [32 seikit-learn (33, NumPy [8]. and einops 34. On single of Intel Xeon Platinum 8260, factor-specific and full flow probabilities can be computed in the sub-second range without significant differences w.r.t. the used rotation, see the supplement for details.
To obtain the optimal rotation matrices for the different rotation criteria, we adapted gradient projection algorithms from [4] based on flow probabilities can take up to few minutes, see the supplement for details. In practice, recommend pre-computing rotations whenever possible.
needs. In current implementation, optimization for criteria
5.1 Evaluation of rotation criteria
We evaluate rotations according the quality criteria from Section 4 that is, (1) the relevance of individual factors, (2) the separability of the relevant factors, and (3) the sparsity of the factors.
5.1.1 Factor relevance
Here, we measure the impact of individual factors on the segmentation. In this section, we use the notation I to denote a matrix of factor loadings that can be either rotated or unrotated. simple measure for the impact of the j-th factor with loadings F E IR" is given by the -nom ( II) of the factor-specific flow probabilities. In what follows, we consider relevance curves that show how many factors exceed the overall uncertainty for varying thresholds 7 2 0. Specifically, we compute
T4 = R l,
where R, = { [[ (1) 2 T| P (T)
and we measure the overall uncertainty by the l -orm of the full flow probabilities, approximated by 100 Monte-Carlo samples.
Results and discussion. The results of averaging n, over the respective test images are shown in Figure 3 (top row). First, classic rotations barely reduce the number of relevant factors compared to the unrotated representation. This is surprise since they do not take the mean logits into account and only try simplify the structure of the factor loadings F' Nevertheless, even classic rotations
7

---

already seem to decrease redundancy. However, intended by design, FP rotations reduce the number of relevant factors to a much greater extent. Especially for LIDC and SEN12MS, already small thresholds are sufficient to cut off most factors below the threshold: Figure 3 (top) shows that all FP rotations behave similarly with curves declining sharply for small т. Consequently, FP rotations tend to produce huge gap between small number of relevant factors and the remaining ones, see D for a visual example. This is desirable since allows to focus only on a few relevant and meaningful factors during the exploration of the predicted uncertainty. It may be harder to find such factors if the predicted uncertainty has less inherent structure. Cam Vid an example this regard as uncertainty predictions are often restricted to class borders, which means that they are less spatially correlated. However, even for CamVid Vid, there is structured uncertainty, see Figure 13 (Section D.2.3) in the supplement. Figure 3 (top) shows that also for Cam FP rotations significantly reduce the number relevant factors.
Figure
5.1.2 Separability of relevant factors
The second quality criterion from Section concerns factor separation. Here, for a separation threshold p E [0, 1], we compute the largest possible fraction of pairwise separated relevant factors:
5- (p) = n max[1 : J C R,, cos(((() F(C.,)) S for all j + j' E J} E [0, 1].
If n2 = 0, we set (-(p) = for all The separation of two factors is measured by the cosine similarity of their factor-specific flow probabilities, which is always non-negative since corresponding entries cannot have opposing signs. For s, (p). a value of one is best since it means that all relevant factors are also separated. For fixed relevance thresholds т, we also compute the area under the curve AUC(s,) for the comparison of different rotation criteria.
Results and discussion. In Figure (bottom row), we show the separation scores AUC(s,) for different relevance thresholds т, respectively averaged over all test images. FP rotations consistently beat classic rotations by a factor of around two terms of AUC. Classic rotation criteria are still better than the unrotated representations (which form the real baseline). The AUC separation scores drop for thresholds t that fail to determine the number of relevant factors sensefully because they too small too large. Notably, for classic rotation criteria, the separation scores AUC(s,) respectively peak at a threshold T for which the number of relevant factors nearly coincides with the one from FP rotations, compare the intersection of the curves in Figure (top row). The peak of the separation scores is less pronounced for FP rotations, particularly for LTDC and SENI2MS where the set of relevant factors is more stable across different thresholds For SENI2MS, the results also distinguish among the FP- rotation criteria, where Pe-Quattimxx seems to be slightly favored over the
LIDC
SEN12MS
CamVid
Unrotated
8-
8-
8-
Varimax
Equamax
6
6
Quartimax
FP-Varimax
PP Eqqumaxx
2-
2-
2-
qua
0-
0
0
155
2.0
2.5
0.0
2.0
25
1.0
1.0
1.0
0.8
0.8
0.8
AUCISO
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.5
1.0
1.5
2.0
7.5
0.0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.0
0.5
1.0
3.5
2.0
2.5
Figure 3: Number of relevant factors n, (top row) and separation via area under the curve AUC(s>) (bottom row), respectively for different relevance thresholds 5 and averaged over all images from the corresponding test data sets. In the bottom row, high values signify a high degree of separation among the relevant factors. Error bars are deferred to the supplement for visual clarity.
8

---

other FP rotations. This may be because Quartimax emphasizes row sparsity the most, which reduces cosine similarities. We investigate row sparsity further the next section.
5.1.3 Factor sparsity
To evaluate to which degree different factors affect the same regions of the input image, we measure row sparsity of the factor-specific flow probabilities F(T) [-1, 1, (whhe)x For that, for (row) vector v E IR" let
h(v) VF ||||| /110 [0,
be the Hoyer measure [19] where values close to one indicate high degree of sparsity. For us, sparsity only matters in rows with actual uncertainty for the pixel/class, therefore we additionally weigh each row proportional to its -norm. Hence, as final measure, we compute weighted Hoyer measure
whe
H(T) = =IF Σ |F(T) ((F(T)).)
Results and discussion. FP rotations generally concentrate the uncertainty for single re- gions/classes in only few components, see [ This means that FP rotation yield the most disen- tangled uncertainty components, which also indicates strong separation. For LIDC and SEN12MS, the amount of predicted uncertainty varies greatly across test images, causing high standard deviations. However, in general, large correlating uncertainty components can be found, allowing high row sparsity. This in contrast to Cam Vid, where uncertainty typically predicted for class borders.
Table 1:
Mean and standard deviation of row sparsity (100 times weighted Hoyer measure) for different rotations. FP rotations yield the most sparsity, For more detailed figure, see the supplement.
Unrotated
Varimax
Equamax
Quartimax
FP-Vari
FP-Equa
FP-Quarti
LIDC
19.3#:14
48.2::19
37.9+18
47.7
82.8
81.8+14
82.7
SEN12MS
33.0+11
52.7+:1
50.4117
45.6::19
74.1=11
74.3+11
77.7
Cim
26.1
30.4
30.3 3
30.7 3
50.1 3
50.0 3
49.9 3
5.2 Fine-grained sampling
Monteiro al. 31 already manipulated samples post-hoc by simple linear inter- extrapolation w.r.t. the mean. However, they noted that additional more fine-grained sample control is necessary for systematic exploration of the sample space: The interpolation approach lacks solid foundation in the uncertainty model, and it relies on having useful sample to start with. The meaningful control variables that obtain by rotating factors provide all that has been missing. They enable to systematically explore the sample space by fine-gr ainingrieeed sampling: Starting from the mean prediction, they can inspect alternatives, correct possible mistakes, and ane-adjust borders. Particularly, they can
Factor 4 Fector 3
Factor Factor
Figure Sxeudo-sample obtained by an individual manipulation of the four most relevant factors of the FP- UP-Quartimax rotation for the example image from Figure [ For each factor, five pseudo-samples
are respectively obtained by setting the associated factor to 1,, -0.5, 0, 0.5. 1, while keeping all other factors fixed to zero. The pseude-sample continuum is intuitively described by one-sided flow The fourth factor (bottom right) has little impact, which can also be deduced from its nearly zero flow probabilities.
probabilities (shown the left and right of the boxes) derived in Corollary [

---

manipulate the contribution of individual uncertainty components by manually setting the values of the corresponding factors. samples obtained in this way are shown in Figure 4
Alongside this paper, we provide an interface for fine- fine-grained sampling It allows the selection of rotation criterion for a given input image (we recommend FP-Quartimax for start), and control variables can be set conveniently using sliders. In the supplement, we provide some visuals.
Discussion and conclusion
In this work, we interpreted the uncertainty model of stochastic segmentation networks (SSNs) as factor model, which provides control variables for fine- mne-grained sampling as requested by the authors of [31] By (re- (re-)structuring the uncertainty using rotations, we improved the controls and obtained as few as possible, but as many as necessary relevant uncertainty components. Here, it turned out that rotation criteria based on flow probabilities yield the most meaningful controls, where flow probabilities are a new quantification and visualization technique for the uncertainty in SSNs.
Our controls allow to systematically explore the predicted uncertainty and to fine-adjust samples. However, the exploration of the sample space is only useful if the overall predicted uncertainty makes sense. This can be ensured by proper training. Structuring and examining the uncertainty is especially useful if there is a significant amount of aleatoric uncertainty. We note that one limitation caused by current implementation is that the computation of pow-obbbyylyity based rotations may take too long for performing it in an interactive scenario. However, there is significant potential for improving the optimization (scheme and parallelization). In any case, rotations should be pre- computed whenever possible.
Overall, we see broader impact of our approach, which extends beyond the scope of SSNs. For instance, we believe that it could used for large-scale image classification, where factor models have recently been employed for modeling class correlations [8]. Another promising application to learn and inspect more structured latent spaces in (variational) autoencoders [2]. Structuring uncertainty as do may be useful whenever multivariate Gaussian forms part of model. Next, flow probabilities can also be used for other probabilistic segmentation architectures that have a mean prediction reference point. Notably, flow-probability overview plots for the predicted uncertainty keep class-specific information, in contrast to other overview plots like entropy.
To sum up, is often easier understand the whole in terms of smaller parts. In this light, we structured the predicted uncertainty of SSNs into meaningful smaller uncertainty components. Jointly, they enable fine-grained sample control, so for us, the sum of the parts is also greater than the whole.
Acknowledgements
We thank Prof. Dr.-Ing. Joachim Denzler for helpful comments and Ferdinand Rewicki for checking our work multiple times. We also thank all anonymous reviewers for their insightful feedback.
References
[1] Armato III, S.G., McLennan, G., Bidaut, I.., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D.R., Henschke, C.I., Hoffman, E.A., et al.: The lung image database consortium (LIDC) and image database resource initiative (IDRI): completed reference database of lung nodules on CT scans. Medical Physics 38(2), 915-931 (2011)
[2] Asgari Taghanaki, S., Abhishek, K., Cohen, J.P., Cohen-Adad, J., Hamarneh, G.: Deep semantic segmentation of natural and medical images: a review. Artificial Intelligence Review 54(1),
137-178 (2021)
[3] Baumgartner, C.F. Tezcan, K.C., Chaitanya, K., Hötker, A.M., Muehlematter, U.J., Schawkat, K., Becker, A.S., Donati, O.,, Konukoglu, E.: Phiseg: Capturing uncertainty in medical image segmentation. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. pp. 119-127. Springer (2019)
10

---

[4] Bernaards, C.A., Jennrich, R.I.: Gradient projection algorithms and software for arbitrary
rotation criteria in factor analysis. Educational and Psychological Measurement 65(5), 676-696 (2005)
[5] Brostow, G.J., Shotton, J., Fauqueur, J., Cipolla, R.: Segmentation and recognition using
structure from motion point clouds. In: ECCV (1). pp. 44-57 (2008)
[6] Browne, M.W.: An overview of analytic rotation exploratory factor analysis. Multivariate behavioral research 36(1), 111-150 (2001)
[7] Chen, L.C.,, Zhu, Y., Papandreou, G., Schroff, F,, Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp. 801-818 (2018)
[8] Collier, M., Mustafa, B., Kokiopoulou, E., Jenatton, R., Berent, J.: Correlated input-dependent
label noise in large-scale image classification. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 1551-1560 (2021)
[9] Costello, A.B., Osborne, J.: Best practices in exploratory factor analysis: Four recommendations
for getting the most from your analysis. Practical assessment, research, and evaluation 10(1), (2005)
[10] Crawford, C.B., Ferguson, G.A.: A general rotation criterion and its use orthogonal rotation. Psychometrika 35(3), 321-333 (1970)
[11] Czolbe, S., Arnavaz, K., Krause, O., Feragen, A.: Is segmentation uncertainty useful? In: International Conference on Information Processing in Medical Imaging. pp. 115-726. Springer (2021)
[12] Dorta, G., Vicente, S., Agapito, L., Campbell, N.D.F.. Simpson, I: Structured uncertainty prediction networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5477-5485 (2018)
[13] Fabrigar, L.R., Wegener, D.T.: Exploratory factor analysis. Oxford University Press (2011)
[14] Feng, D.. Haase-Schütz. C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., Wiesbeck, W., Dietmayer, K.: Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions Intelligent Transportation Systems 22(3), 1341-1360 (2020)
[15] Gal. Y., Ghahramani, Z.: Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In: international conference on machine learning. pp. 1059. PMLR (2016)
[16] Gawlikowski, J., Tassi, C.R.N,, Ali, M., Lee, J., Humt, M., Feng. Kruspe, A.. Triebel, R., Jung, P., Roscher, R., et al.: A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.033. 0..3440...322 (2021)
[17] Harman, H.H.: Modern factor analysis. University Chicago press (1976)
[18) Harris, C.R., Millman, K.J., der Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D.,
Wieser, E., Taylor, J., Berg. S., Smith, N.J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M.H., Brett, M., Haldane, A., del Río, J.F,, Wiebe, M., Peterson, P., Gérard-Marchanm P.. Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., Oliphant, T.E.: Array programming with NumPy. Nature 585(7825), 357-362 (Sep 2020). https://ddo.org//// (038//41586-020-
[19] Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. Journal of machine learning research 5(9) (2004)
[20] Hüllermeier, E., Waegeman, W.: Aleatoric and epistemic uncertainty in machine learningg An
introduction concepts and methods. Machine Learning 110(3), 457-506 (2021)
[21] Islam, M., Glocker, B.: Spatially varying label smoothing: Capturing uncertainty from expert annotations. In: International Conference on Information Processing in Medical Imaging, pp. 677-688. Springer (2021)
[22] Kaiser, H.F.: The varimax criterion for analytic rotation in factor analysis. Psychometrika 23(3),
187-200 (1958)
[23]
Kamnitsas, K., Bai, W., Ferrante, E., McDonagh, S., Sinclair, M., Pawlowski, N., Rajchl, M., Lee, M., Kainz, B., Rueckert, D.. et al.: Ensembles of multiple models and architectures
11

---

for robust brain tumour segmentation. In: International MICCAI brainlesion workshop. pp. 450-462. Springer (2017)
[24] Kassapis, E., Dikov, G., Gupta, D.K., Nugteren, C.: Calibrated adversarial refinement for
stochastic semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7057-7067 (2021)
[25] Kaymak, Ç.. Uçar, A.: A brief survey and application of semantic image segmentation for autonomous driving. In: Handbook of Deep Learning Applications, pp. 161-200. Springer (2019)
[26] Kendall, A., Badrinarayanan, V., Cipolla, R.: Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. In: British Machine Vision Conference 2017, BMVC 2017 (2017)
[27] Kendall, A., Gal. Y.: What uncertainties do need in Bayesian deep learning for computer vision? Advances in neural information processing systems 30 (2017)
[28] Kohl, S., Pareees, B., Meyer, C., De Fauw, J., Ledsam, J.R., Maier-Hein, K., Eslami, S., Jimenez Rezende, D., Ronneberger, O.: A probabilistic U-net for segmentation of ambiguous images. Advances in neural information processing systems 31 (2018)
[29]
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30 (2017)
[30] Michie, D.: Problem decomposition and the learning of skills. In: European Conference on Machine Learning. pp. 17-31. Springer (1995)
[31] Monteiro, M., Le Folgoc, L.., Coelho de Castro, D., Pawlowski, N., Marques, B., Kamnitsas, K., van der Wilk, M., Glocker, B.: Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty. Advances in Neural Information Processing Systems 33, 12756-12761 (2020)
[32] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., Veto Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An imperative style, high segh-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024-8033 8035. Curran Associates, Inc. (2019)
[33] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V. Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D.,, Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825-2830 (2011)
[34] Rogozhnikov, A.: Einops: Clear and reliable tensor manipulations with Einstein-like notation. In: International Conference Learning Representations (2022)
[35] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image
segmentation. In: International Conference on Medical image computing and computer- assputer-assisted
intervention. pp. 234-241 Springer (2015)
[36] Schmitt, M., Hughes, L.H., Qiu, C., Zhu, X.X.: SENI2MS a curated dataset of georeferenced multi-spectral Sentinel- 1/2 imagery for deep learning and data fusion. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences. vol. IV-2/W7, pp. 153-160 (2019)
[37] Schmitr, M., Prexl, J., Ebel, Liebel, L., Zhu, X.X.: Weakly supervised semantic segmentation of satellite images for land cover mapping-challenge and opportunities. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 3, 795-802 (2020)
[38] Spearman, C.: "General intelligence," objectively determined and measured. American Journal of Psychology 15, 201-293 (1904)
[39] Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J.N., Wu, Z., Ding, X.: Embracing imperfect
datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis 63, 101693 (2020)
[40] Thompson, B.: Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC 10694 (2004)
12

---

[41] Thurstone, L.L.; Multiple-faceee factoo analysis; development and expansion of The Vectors of Mind. University of Chicago Press (1947)
[42] Wang, S., Chen, W., Xie, S.M., Azzari, G., Lobell, D.B.: Weakly supervised deep learning for segmentation of remote sensing imagery. Remote Sensing 12(2), 207 (2020)
[43] Yang. M., Yu, K., Zhang, C., Li, Z., Yang, K.: Denseaspp for semantic segmentation in street scenes. In: Proceedings of the IEEE conference computer vision and pattern recognition, pp.
3684-3692 (2018)
[44] Yuan, X., Shi, J., Gu, L.: A review of deep learning methods for semantic segmentation of remote sensing imagery. Expert Systems with Applications 169, 114417 (2021)
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes) See the discussion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We do not see direct negative consequences, however, segmentation as a general technique can in principle also be used in negative contexts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We submit a zipped version of our code repository supplemental material. Further, the code is available on GitHub: https://gtthub.com/ wttp:///oooooooooo/ee/teuiiieeee is fully anonymous and will be made public with the submission of our supplemental material.
The repository
The online version of the repository also contains demo with few test images (including pre-computed rotations). They could not be included into the zipped version, due to the limitation of 100MB.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See the supplement or the implementation.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] For the number of relevant factors and separability, we deferred eтror bars the supplement for visual clarity in the plots in the main paper. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3 and the supplement for details.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] The licenses stated within the more detailed description of the data sets in the supplement.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See Section 5.2 and the GitHub repository (see Question 3(a)).
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The consent is included in the publication and licensing of the sources.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] We describe the data sets in the supplement in more detail and discuss the uncritical nature of the data shortly.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "TBHTVVw_U4J", "reviewer": "Reviewer_N33x", "summary": "This paper tackles the issue of segmentation uncertainty. Using state-of-the-art SSNs, authors view these as factor models. They derive flow probabilities on these factors to visualize and quantify uncertainty associated with them, while looking for a \"minimal rotation\" of factors through orthogonal rotations.\nThey show that this technique is suited to derive fine-grained maps for assessing uncertainty in segmentation results, as well as to tweak computed segmentations - which could prove useful for experts using such tools.", "questions": "1. Why do you think rotated factors (from FP-Quartimax for instance) seem to be so different across tested datasets? In particular, as you mention at lines 266-268, uncertainty seems to be significant only on class borders for CamVid (cf Fig 11 in the supplemental material), which is the dataset with the highest number of classes. Is your method capable of showing uncertainty for attributed classes not only on borders, and if not, isn't this a major limitation of your model?\n\n2. Are you using loadings or rotated loadings (columns of $\\Gamma$ or $\\Gamma \\cdot O$) in the separation criterion exposed introduced 5.1.2? It is confusing to me that you should use loadings here, when you use rotated loadings line 222.", "limitations": "Authors mention computation time of flow-probability based rotation, but I think question 1 could be an important limitation of this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find this piece of work sound and interesting. Flow probabilities (FP), after factors have been rotated in a meaningful way, make for a nice object to intuitively visualize uncertainty and authors show a convincing piece of code allowing users to update segmentations thanks to these FP.\n\n**Significance** The described method could prove very useful if it can indeed help domain experts perform fast segmentation tasks while providing them with intuitive uncertainty measures. However, it is yet unclear to me wether it can help discard entire segmented zones when the number of classes is higher than 2 or 3 (see Question 1).\n\n**Originality** This paper could be considered mildly original as it mostly combines existing results from SSNs, factor analysis and flow probabilities. However, I think the idea to view low-rank models underlying SSNs as factor models sheds a nice perspective on these models, and that this aspect is more important than pure originality.\n\n**Quality** I find this piece of work to be well written and illustrated. I appreciate that authors bundled a repo that I could use out of the box (I tested the notebook and read a great deal of the code). As is, the codebase is hard to grasp and use though, and I think it would greatly benefit from being documented more extensively (docstrings for all methods would be nice) and tested (tests make for a nice way to understand the overall structure of the codebase and usage of each method).\nPlease cite used packages in the main document (numpy, scipy, sklearn, torch, einops to name a few).\n\n**Clarity** Although this work calls for very visual and easily understood experiments, I found the article a bit difficult to dive in. I think it would benefit from having a figure describing the overall procedure, training steps, and connection between concepts (loadings, factors, latent variables, FPs, rotations and rotation criteria). It could be included in the supp. mat. I think intuitions leading to Proposition 1 would also benefit from being illustrated in a Figure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S5wZrUv44Kr", "reviewer": "Reviewer_CT7G", "summary": "The paper proposes a novel method for structuring uncertainty in the context of stochastic segmentation networks(SSNs). They use low-rank multivariate gaussian distribution to solve SSNs model uncertainty. They also develop a tool for the analysis of factor models in SSNs and apply rotation criteria to provide simple and well-separated control variables. In the experimental part, the proposed method has been outperformed the current state of the art on several datasets.", "questions": "How to determine how many factors we should use from Figure 1?\n\nHow to determine the ‘significant’ effect where the relevant factors are characterized.\n", "limitations": "one limitation of this work is the heavy computation of the flow-probability rotation.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of this proposed method appears to be a novel combination of stochastic segmentation networks. The authors try to solve the segmentation uncertainty by using smaller latent factor variables based on the recent work called SSNs. The model proposed by the paper is clearly explained and it includes sufficient and detailed experiments in the supplement. \n\n\nOne possible weakness maybe somewhat missing the explanations of the so called ‘significant’ effect on output segmentations; another weakness maybe more detailed description about the rotation criteria. It will be great if the authors could explain more about the interface of fine-grained sampling. In the supplement, the proposed methods seems to have heavy computation cost, I expect them to better address this issue if possible.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "L43xotEAsa", "reviewer": "Reviewer_LLek", "summary": "The manuscripts reinterprets stochastic segmentation networks (SSN 2020) as a factor model and thus adds latent factors governing the noise components within the single covariance of SSNs. Additionally rotation of the factors with imposed sparsity leads to a parsimonious, and supposedly more interpretable, representation of the factors.  The manuscript provides derivations of the reasoning behind the proposed representation and performs a rigorous empirical comparison of already available rotation approaches. The results in the main manuscript and the supplement, including the video demonstrate that the approach works in providing uncertainty factors that could be individually manipulated.\n", "questions": "- Is there any way to show an intuitive example where the proposed way of displaying uncertainty can be useful, since everything shown is as not helpful as the rial methods of entropy and others?\n- Can you report timing information for vanilla SSN vs the proposed modification to give a rough idea how severe is the slowdown?", "limitations": "- The value of the proposed approach is not clear from the paper, this limits further impact of the paper since practitioners won't be able to appreciate the need for this method.\n- Minor: computational complexity, as noted by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- A simple and pragmatic approach to extending SSN with an ability to control uncertainty components independently\n- A needed take on sparsity and uncertainty to address interpretability of uncertainty representation.\n\n## Weaknesses\n- The major weakness, in my view, is that besides meeting the goals of factorization and sparsification of uncertainty, the manuscript is not convincing in that the created tool is interpretable and thus useful. The value of the method for interpretation of the model is not coming through in any of the examples of the manuscript, supplement, and the video.  The most intuitively interpretable examples from the CamVid dataset does not add any information and looks like uncertainty of all classes but, possibly, the cars is simply mixed around all the objects. Other demonstrations are also not helpful and it is unclear how would a user of the model benefit from the new approach in either of the remaining examples. I do not look at the satellite images every day and that may be the reason the flagship example in the main manuscript and the supplied video does not convey much information since the segmentation seems to be very poor (the DICE score would be really low).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept