Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the issue of segmentation uncertainty. Using state-of-the-art SSNs, authors view these as factor models. They derive flow probabilities on these factors to visualize and quantify uncertainty associated with them, while looking for a "minimal rotation" of factors through orthogonal rotations.
They show that this technique is suited to derive fine-grained maps for assessing uncertainty in segmentation results, as well as to tweak computed segmentations - which could prove useful for experts using such tools.
strengths_and_weaknesses: I find this piece of work sound and interesting. Flow probabilities (FP), after factors have been rotated in a meaningful way, make for a nice object to intuitively visualize uncertainty and authors show a convincing piece of code allowing users to update segmentations thanks to these FP.

**Significance** The described method could prove very useful if it can indeed help domain experts perform fast segmentation tasks while providing them with intuitive uncertainty measures. However, it is yet unclear to me wether it can help discard entire segmented zones when the number of classes is higher than 2 or 3 (see Question 1).

**Originality** This paper could be considered mildly original as it mostly combines existing results from SSNs, factor analysis and flow probabilities. However, I think the idea to view low-rank models underlying SSNs as factor models sheds a nice perspective on these models, and that this aspect is more important than pure originality.

**Quality** I find this piece of work to be well written and illustrated. I appreciate that authors bundled a repo that I could use out of the box (I tested the notebook and read a great deal of the code). As is, the codebase is hard to grasp and use though, and I think it would greatly benefit from being documented more extensively (docstrings for all methods would be nice) and tested (tests make for a nice way to understand the overall structure of the codebase and usage of each method).
Please cite used packages in the main document (numpy, scipy, sklearn, torch, einops to name a few).

**Clarity** Although this work calls for very visual and easily understood experiments, I found the article a bit difficult to dive in. I think it would benefit from having a figure describing the overall procedure, training steps, and connection between concepts (loadings, factors, latent variables, FPs, rotations and rotation criteria). It could be included in the supp. mat. I think intuitions leading to Proposition 1 would also benefit from being illustrated in a Figure.
questions: 1. Why do you think rotated factors (from FP-Quartimax for instance) seem to be so different across tested datasets? In particular, as you mention at lines 266-268, uncertainty seems to be significant only on class borders for CamVid (cf Fig 11 in the supplemental material), which is the dataset with the highest number of classes. Is your method capable of showing uncertainty for attributed classes not only on borders, and if not, isn't this a major limitation of your model?

2. Are you using loadings or rotated loadings (columns of $\Gamma$ or $\Gamma \cdot O$) in the separation criterion exposed introduced 5.1.2? It is confusing to me that you should use loadings here, when you use rotated loadings line 222.
limitations: Authors mention computation time of flow-probability based rotation, but I think question 1 could be an important limitation of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The manuscripts reinterprets stochastic segmentation networks (SSN 2020) as a factor model and thus adds latent factors governing the noise components within the single covariance of SSNs. Additionally rotation of the factors with imposed sparsity leads to a parsimonious, and supposedly more interpretable, representation of the factors.  The manuscript provides derivations of the reasoning behind the proposed representation and performs a rigorous empirical comparison of already available rotation approaches. The results in the main manuscript and the supplement, including the video demonstrate that the approach works in providing uncertainty factors that could be individually manipulated.

strengths_and_weaknesses: ## Strengths
- A simple and pragmatic approach to extending SSN with an ability to control uncertainty components independently
- A needed take on sparsity and uncertainty to address interpretability of uncertainty representation.

## Weaknesses
- The major weakness, in my view, is that besides meeting the goals of factorization and sparsification of uncertainty, the manuscript is not convincing in that the created tool is interpretable and thus useful. The value of the method for interpretation of the model is not coming through in any of the examples of the manuscript, supplement, and the video.  The most intuitively interpretable examples from the CamVid dataset does not add any information and looks like uncertainty of all classes but, possibly, the cars is simply mixed around all the objects. Other demonstrations are also not helpful and it is unclear how would a user of the model benefit from the new approach in either of the remaining examples. I do not look at the satellite images every day and that may be the reason the flagship example in the main manuscript and the supplied video does not convey much information since the segmentation seems to be very poor (the DICE score would be really low).
questions: - Is there any way to show an intuitive example where the proposed way of displaying uncertainty can be useful, since everything shown is as not helpful as the rial methods of entropy and others?
- Can you report timing information for vanilla SSN vs the proposed modification to give a rough idea how severe is the slowdown?
limitations: - The value of the proposed approach is not clear from the paper, this limits further impact of the paper since practitioners won't be able to appreciate the need for this method.
- Minor: computational complexity, as noted by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel method for structuring uncertainty in the context of stochastic segmentation networks(SSNs). They use low-rank multivariate gaussian distribution to solve SSNs model uncertainty. They also develop a tool for the analysis of factor models in SSNs and apply rotation criteria to provide simple and well-separated control variables. In the experimental part, the proposed method has been outperformed the current state of the art on several datasets.
strengths_and_weaknesses: The idea of this proposed method appears to be a novel combination of stochastic segmentation networks. The authors try to solve the segmentation uncertainty by using smaller latent factor variables based on the recent work called SSNs. The model proposed by the paper is clearly explained and it includes sufficient and detailed experiments in the supplement. 


One possible weakness maybe somewhat missing the explanations of the so called ‘significant’ effect on output segmentations; another weakness maybe more detailed description about the rotation criteria. It will be great if the authors could explain more about the interface of fine-grained sampling. In the supplement, the proposed methods seems to have heavy computation cost, I expect them to better address this issue if possible.

questions: How to determine how many factors we should use from Figure 1?

How to determine the ‘significant’ effect where the relevant factors are characterized.

limitations: one limitation of this work is the heavy computation of the flow-probability rotation.
review_title: null

Meta-review: metareview: The manuscript interprets stochastic segmentation networks (SSN 2020) as a factor model and thus adds latent factors governing the noise components within the single covariance of SSNs. Additionally, well-chosen rotations of the factors with imposed sparsity lead to a parsimonious, and supposedly more interpretable, representation of the factors. The manuscript provides derivations of the reasoning behind the proposed representation and performs a rigorous empirical comparison of already available rotation approaches. The results in the main manuscript and the supplement, including the video, demonstrate that the approach works in providing uncertainty factors that could be individually manipulated.

According to all reviewers, the paper is well written and the technical work looks solid. The proposed solution is pragamatic. Experiments are sufficient.

The main issue of the paper is that the value of the method for interpretation of the model does not come through the examples of the manuscript, supplement, and the video. It is unclear how would a user of the model benefit from the new approach in either of the examples. The issue is that according to the authors, the new feature for fine-grained sampling and fine-adjustments of segmentations should be integrated alongside with classic manual techniques for editing segmentations in a full user application. However, the present work only focused on the new technique and not on building a full user application. Given this, there remains a doubt wehther the proposed technique is actually useful or not.

Other issues related to presentation clarity and quality of the visual material have been well addressed. 

The consensus is that the paper should be accepted at NeurIPS 2022.
confidence: Less certain
Decision: Accept