Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the statistical reliability of segmentation results obtained by Neural Network-based Image Segmentation. The authors developed a conditional selective inference (SI) framework to compute exact valid p-values for the segmentation results. The experimental results on the brain image dataset demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: [Strengths]
+ The studied problem is interesting and important.
+ Compared to the Permutation Test, the proposed methods could bring significant improvements.

[Weaknesses]
- The authors just conduct the experiments on a small dataset, which makes it difficult to evaluate the robustness of the proposed method. 
- The evaluated NNs model is somehow simple.  There are many SOTA image segmentation / Semantic segmentation methods, e.g., Deeplabv3+, and datasets, e.g., Pascal VOC. Could you evaluate these segmentation methods on the dataset with the proposed method?
- What’s the relationship between Uncertainty Estimation for Semantic Segmentation and statistical reliability?

questions: Please refer to the Question in the Weaknesses.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method that extends previous work on conditional selective inference (SI) for statistical significance quantification in segmentation models to deep neural networks with affine operations and piecewise linear activation functions (the previous method was applicable to threshold- and graph-cut-based algorithms only). The core idea of this approach is to correct for the segmentation bias and perform SI on a conditional subspace of images that yield the same segmentation result as that actually observed. To identify this subspace, the paper proposes to perform search on a line parametrized by a scalar $z$ and reduces the core problem to finding the set of $z$'s compatible with the actual image ("truncation region"). To do this, it proposes to overcondition on "selected pieces" of the activations of the network nodes, shows that this can be expressed in the form of linear inequalities, and that multiple overconditioning cases can be combined with a homotopy method by finding "breakpoints" where exactly one network node changes its value. The method is experimentally evaluated on synthetic random data and on real images of brain tumors, using a simple conv-pooling network.

strengths_and_weaknesses: The paper is very well written, cites the relevant literature, and includes a brief introduction to SI for simple segmentation methods, summarizing the results from ref [24]. The reduction of the data subspace to 1D is based on prior results from ref [16], which the present paper acknowledges. The major contribution of the present work is showing how overconditioning and the homotopy method allow the application of SI to neural network-based segmentation models, and the demonstration of its effectiveness on sample synthetic and real data. This is a novel result and should be of significant interest.

The paper is generally upfront about the limitations (affine operations only, hard to approximate softmax), but could be a bit more clear about the computational complexity, which is important for real-life applications. 

It is also commendable that the authors included the code necessary to reproduce all the experiments and the figures.
questions: - The paper generally presents the method as "efficient" in a few places, and on page 8 mentions that the homotopy method can be exponential in the worst case, but tends to be linear in practice. I think it would be helpful for practitioners to also mention explicitly in the text that the linear complexity has a high "constant factor" in the form of neural network inference (obvious in Alg. 2, but a bit hard to see from just skimming the paper).
- nit: "Therefore, all of the operations that only operate in the training process such as Dropout or Batch Normalization"; this is not strictly true about dropout and BN, as these operations also affect inference (albeit in a way that is compatible with the claims in the paper and does break the assumption of the operations being affine)
- in ref [24], it is assumed that $\Sigma$ can be estimated from a null image known not to contain any "foreground" pixels; It would be useful to mention how it was estimated to generate the results in Fig. 7.
- nit: in Alg. 2, the initialization of T should be {z_t} instead of $z_t$

limitations: No concerns.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper concentrates on evaluating the statistical reliability of the segmentation, necessary to develop a plausible statistical inference methodologies. This is difficult in a DNN context. Addressing such a challenge allows to derive exact (non-asymptotic) sampling distribution of the test statistic for obtaining valid statistical inference.
The main goal in this paper t is to conduct the statistical inference in the test phase, specifically, the goal is to quantify the reliability of the segmentation result when a new test image is given to a DNN.

strengths_and_weaknesses: Strengths
Selective Inference (SI) is known in the community and has been applied in a quite diverse range of applications. However, delving into details in DNN context is a less explored topic. Furthermore, proposed methodologies such as [24] are not applicable in the case of DNN-based segmentation, since the selection event is difficult to be accomplished in this context, as such this  constitutes strength of the proposed approach.
I also think that the background and the theoretical preliminary ingredients of the SI are well written in Section 2.
 
Weaknesses 
Although I think this is a valuable contribution to this research field, I have some questions that perhaps constitute the main weaknesses of the present paper (see my questions below).

questions: I consider that this paper is well written, with a clear exposition of the ideas, as well as its analytical development. However, I could no follow all the details provided, as I describe next:
- Perhaps I did not follow the eq.(10) - left side -> x(z) = a+ b*z, I performed some calculations and I obtained  x(z) = a+ b*eta^T * z. 
So I wonder if it is possible to obtain this derivation ? 

-Seems that in the process of segmentation  we obtain a large number of breakpoints, since the breakpoints are dependent on the number of the neurons in the architecture. Is this a disadvantage of the proposal ? 
-By the way, is it possible to have an idea of the  order of magnitude concerning the number of changes in a given segmentation task ?
-The equation in Lemma 3, is computed for each neuron ? If so, I think this is a limitation of the framework
-I understand the authors are concerned to “close” all the operations in a DNN, concerning network operations, activation and pooling functions. However, and following the material provided in the paper, I find a bit short the justifications in Remarks given (page 6). For instance, the universal polling does not fit in Remark 3, that must be, somehow, extended.
 
(see Junhyuk Hyun, Hongje Seong, Euntai Kim, Universal Pooling -- A New Pooling Method for Convolutional Neural)

- Finally, Although I think that this can constitute an advance in terms of refining the p-value, and consequently, the segmentation confidence, these contributions seem a bit short, when evaluating more complex deep network architectures.  So I may guess that, perhaps this theoretical framework may not be extended to a more drastic situation, where the comçlxity is greatly enlarged. 


limitations: See my questions  above. But fundamentally, extending this framework to more complex architectures is not straightforward.
review_title: null

Meta-review: metareview: This paper proposes a conditional selective inference based technique for quantifying the statistical significance of an image segmentation generated by a DNN. Estimating the statistical significance of a DNN generated image segmentation is an important problem in medical image analysis, and the technique proposed extends the applicability of selective inference beyond the previously applicable domains of thresholding or graph partitioning based segmentation methods to segmentations generated by DNNs with piecewise linear activation functions.

While all reviewers found the proposed method to be of value to the community at large, they commented on the limited evaluation of the method. The method has only been applied to simple segmentation networks, and not state of the art architectures, further it has only been applied to a limited set set of datasets, suggesting the practical challenges with scaling this technique to realistic networks and datasets. Regardless, the reviewers unanimously recommend acceptance, and I hope this work will stimulate further practical progress in this important area.


confidence: Certain
Decision: Accept