Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

In this paper, the authors propose a novel approach for quantifying the statistical significance of binary masks predicted by a subclass of deep neural network (DNN) models for image segmentation problems.

In brief, the manuscript considers the particular setting where a model has been pretrained to produce a binary output of the same dimension as the input, which can be interpreted as a binary attention mask / image segmentation output. In this scenario, the main contribution is an approach to test the null hypothesis that a linear contrast of the image, depending on the segmentation outputs, equals zero. In particular, the specific formulation put forward in the paper tests the null hypothesis that the average value of pixels classified as 1 (e.g. in the mask) is equal to the average value of pixels classified as 0 (e.g. not in the mask).

Since the linear contrast is a function of the input, in this case, a nonlinear function implemented by the pre-trained DNN model, classical statistical inference (such as e.g. a two-sided t-test) does not apply. Instead, the authors propose to leverage recent advances in the field of selective inference to derive the null distribution of the test statistic conditioned on the DNN’s output (plus additional constraints to get rid of nuisance parameters).

In a nutshell, to accomplish this, they:

1) Assume that images follow a multivariate normal distribution with known covariance matrix,
and
2) Restrict the class of DNN models to those whose internal representation in every layer can be expressed as a piecewise affine function of the input. Namely, this excludes activation functions other than piecewise linear functions (e.g. ReLU), which the authors propose to approximate using piecewise linear functions.

The modelling assumption (1) together with the model constraints (2) permit the problem to be cast in a form for which the celebrated Polyhedral lemma (Lee et al. 2016) applies.

Finally, the authors also tackle the algorithmic problem of efficiently integrating over the resulting truncation regions. To this end, they propose two approaches: one based on over-conditioning, inspired by Lee et al. 2016, and another based on the homotopy method, possibly inspired by Liu et al. 2018.

Experimental results are provided for a synthetic toy problem with “images” that follow a multivariate normal distribution with no correlation between pixels. The ground-truth attention mask is a large square region in the center of the image, and pixels within this region have a larger mean than those outside. DNN models consisting of 2 convolutional layers are pretrained on data generated under the same distribution, with the entire ground-truth binary segmentation masks used for training for each “image”. Results suggest that, under these conditions, the proposed approach succeeds in controlling the type I error while achieving non-trivial statistical power. The supplementary material shows that type I error remains controlled for other distributions such as Laplace or Student-t with 20 DoF. 

Finally, results are also shown for a real-world brain image dataset, where type I error appears to be only sensibly larger than the tower and power non-trivial as well.

### High-level assessment

From a methodological perspective, the authors have identified a novel, interesting application of existing work, namely, the now seminal work of Lee et al. 2016 and recent improvements over those ideas presented by Liu et al. 2018, to the problem of image segmentation using certain DNN models. Moreover, they also proposed a sound algorithmic implementation of these ideas targeting the application at hand and provided some nascent, preliminary results that suggest the proposed approach might be of use in certain applications such as medical imaging.

All in all, this represents an original contribution of relevance to the field that I wish to see published eventually. However, I believe the current version of the manuscript falls short in certain areas, and should be improved prior to publication. Mainly:

1) I have strong reservations regarding the adequacy of the modelling assumptions required by the selective inference framework for this application.

2) The experimental results are substantially lacking, both in terms of breadth and depth.

3) The paper is also lacking in clarity. Many key, low-level aspects such as model architecture, (pre)-training procedure or assumptions between the relation of pre-training and testing data are either not clearly stated or mentioned without proper justification or discussion of its implications and impact on the overall results. Likewise, I feel that some of the limitations of the proposed approach are understated and that its generality is overstated, especially in the abstract and the introduction.

Because of this, I am as of now leaning towards recommending rejection, as I believe the paper would greatly benefit from a strong, non-incremental revision.

### Major points / suggestions for improvement

1) In order to frame the problem within the assumptions of the Polyhedral lemma, the authors postulate that images in the dataset follow a multivariate normal distribution, with unknown mean but known covariance matrix.

Unfortunately, while selective inference is known to abound on strong parametric assumptions, I believe this assumption is just too strong and hard to justify in the context of image data relative to, say, interpreting the coefficients of a linear model as done in more “typical” applications of selective inference.

While it might be unrealistic to expect the authors to overcome inherent limitations of the current state of the art in selective inference, I believe they should at least carry out a much more thorough investigation of the extent to which these assumptions are applicable to real-world data, and what are the consequences of violations for the inference process.

The robustness results provided in the supplementary material are a good step in this direction, but keep many unrealistic assumptions, such as unimodality or simplistic / inexistent correlation structures that remain far from representative of natural or medical images.

2) The proposed approach relies on pre-training the DNN model on a separate dataset, since the authors have made no attempt to account for the impact of training itself on the null distribution. 

This can be an important limitation, especially in applications where data does not abound, which unfortunately might have a strong overlap with applications for which statistical significance might be of special relevance. Moreover, the need to split the data into “development” and “inference” sets will introduce randomness of the inference results, apart from incurring a loss in statistical power.

To this end, I believe the authors should: (1) discuss in much greater clarity how the pre-training of the DNN models interacts with their proposed approach; (2) study the extent to which the resulting inferences are robust to data splitting & retraining of the DNN model and (3) the impact that data splitting has on the resulting statistical power.

3) The paper is substantially lacking in clarity, leaving the reader to second-guess many key aspects of the proposed approach.

One such example is precisely the issue of pre-training. It is never clearly stated in the main manuscript how the DNN models are (pre)-trained, and what the relation between the training and inferences datasets is exactly.

Other, related aspects are (i) which type of supervision the models require for pre-training and (ii) which type of output layers are required in order for the null hypothesis to be relevant. For example, I had to look at the code in order to know that, in their simulation experiments, models are trained with the segmentation masks as supervision targets. To the best of my knowledge, this is nowhere stated in the paper. Similarly, I am uncertain of how the models for the real-world brain image dataset were trained. Given the change in architecture reported in the supplementary material, I could guess that in this case only tumor/no tumor labels were provided, but this is neither stated nor, as far as I could see, shown in the code provided. Also, in the latter case, I imagine that in order to keep the null hypothesis relevant, the model must rely on a global average pooling layer prior to classification since, otherwise, a null hypothesis defined in terms of the difference of pixel intensity values would not capture well the behavior of the output layer(s). More generally, the choice of architecture for the models is not justified in any way.

As a final example, it is also never precisely stated how exactly the output representation is thresholded in order to define the binary attention mask, how the threshold parameter is supposed to be chosen by a user and what its impact is in overall performance.

All in all, in my opinion, at the moment the manuscript does not provide sufficient information for a reader to accurately reproduce its results from the text alone.

4) In my opinion, the abstract and introduction currently overstate the generality of the proposed approach. I believe that claims such as “testing the reliability of DNN representations” implicitly imply the approach can be applied to any kind of DNN model/problem. However, the target application for the framework here developed is narrower and applies only to image segmentation problems and perhaps to other strongly related settings (e.g. problems where the target representation has the same size as the input and can be interpreted as an attention mask of sorts).

To clarify, I do not intend to imply the author’s contribution is insufficient. I believe proposing an approach to quantify the statistical significance of image segmentation masks is on its own a worthwhile contribution. However, I do believe it would be preferable that the abstract and introduction were written in a way that was more specific to what the manuscript truly proposes, implements and tests.

5) The experimental results, while providing some encouraging preliminary results supporting the proposed approach, in my opinion fall short in multiple aspects.

Firstly, as mentioned in point 1. above, I believe the experiments on synthetic data as they stand now are barely a “sanity-check” for the model. Instead, I suggest that the authors explore ways to create more challenging synthetic datasets capturing characteristics of real-world data, perhaps (but not necessarily) by exploiting deep generative models trained on natural / imaging data. In brief, the goal should be to exhaustively characterize (1) the extent to which the modelling assumptions required by selective inference are applicable to real-world data and (2) which failure modes / limitations the proposed approach might have.

Secondly, the models used by the authors are too small and not representative of those deployed in the target applications. I would encourage the authors to show results pertaining larger models (e.g. ResNet-based architectures with > 10 layers).

Finally, related to the previous point, it would be of interest to explore the effect of model size & architecture (depth, number of units per layer, presence of residual connections / normalization) on the number of intervals encountered, false positive rate and statistical power.

### Minor points (not related to the manuscript’s rating)

1) In my opinion, Example 1 would be much more informative if it included inactive units rather than the special, much easier case where all hidden units are active.

2) Personally, I do not find the definition of power in the supplementary material to be sufficiently clear. I would encourage the authors to define mathematically what # detected and # rejected are in their context.

3) To the best of my knowledge, the code does not currently set the seed for the PRNG, which might be detrimental for reproducibility.

4) I would recommend having the submission proof-read for English style and grammar issues.
review_title: Paper1095 review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a framework of selective inference to rigorously study the problem of finding significant attention areas of a neural network model (with certain constraints). 

Pros
- The proposed method yields good performance in power in practice.

- The FDR of the proposed method is theoretically controlled.

- The authors proposed an efficient algorithm to find certain kinds of attention areas. 

Cons & Questions: 

- Is the assumption (1) necessary for the proposed non-asymptotic method? Please address. If it is necessary, please also address why this assumption is practical, or existing work where the same was assumed.

- The proposed method has several assumptions on the neural network, such as the layers need to be affine / max pooling / piecewise linear. Given the design of popular neural networks, the method is widely applicable enough. However, it will be helpful if the authors can address the commonly used network structure where the proposed method cannot be used. 

For example, can one uses this method to test the alternative hypothesis that a region found by BERT's internal attention mechanism is significant? Such a case study will help readers gain more clarity of the method.

Also, if the method is not applicable to softmax layer, does the score produced from the logit? Please clarify.
review_title: A nice application of selective inference to finding statistically significant attention area of an image

final_score: rating: 7: Good paper, accept
review_content: The authors present a new method to quantify the representational strength of the neural networks. In particular, they address a known issue about how to judge a network's result relative to a hypothesis if the source of both the hypothesis and the network training was the same dataset.
I believe the goal and motivation of this paper are clear and very well useful. Having a robust and fair theoretical framework would be a good step in analyzing a lot of vision approaches and approaches in NLP, control, and others. They also give some promising experimental results.
The techniques used in this work (e.g., selection of extra conditioning for traceability, Merging results of condition sets) have a satisfactory amount of novelty and can be of independent interest to the ML community. Since their approach (at least partially) addresses the issue explained above, it may find more interest among researchers of downstream tasks for explainability and reliability evaluation.
I suggest the authors address famous problems with frequentist statistics and commonly misinterpreted p-value. I want to see the author's opinion about the position of their approach relative to such issues. Perhaps having a discussion section or limitations of the study would help. Also, does this approach have a Bayesian counterpart so that it doesn't rely on p-values?
review_title: Important problem and nice approach

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposed a novel method which to quantify the reliability of DNN-driven hypotheses in a statistical hypothesis testing framework. Naive statistical testings are not appropriate for the DNN-driven hypotheses, where the hypotheses are selected by looking at the data(i.e. The selection bias exists). To address this problem, the authors developed a novel homotopy method under the Selective-Inference(SI) framework, which can derive the exact sampling distribution of the DNN-driven hypotheses. In this paper,  the authors mainly focus on DNNs which consist of affine operations, max-operations, and piecewise-linear activation. As described by Lee et al. (2016), the main idea of SI is to make the inference conditional on the selection event. Specifically to the DNN-driven hypotheses, the authors proposed a novel method that consists of two steps, 1) Adding extra conditioning to make the problem traceable. 2) Combining multiple over-conditioning cases by homotopy method to solve the over-conditioning problem. The experimental results on both synthetic and real-world datasets illustrate the proposed method can successfully control the FP error rate.

Considering that there are more and more interests in the research on neural representation learning, the problem that this work is trying to solve is pretty important. The SI framework has been well studied on the lasso and other problems. To the best of my knowledge, this is the first work that deploys SI to test DNN representation driven hypotheses. Although the authors only demonstrate it on simple DNNs, I can see the potentials of this method to apply on more practical and complex DNNs. The community may benefit from it on understanding neural representations.

Pros:
- The problem that this paper is trying to solve, is clearly defined and is essential in understanding the representations DNN learned.
- This proposed SI algorithm based on the homotopy method can derive the exact conditional sampling distribution of DNN-driven hypotheses in an efficient way, and it is proved to be effective in practical by experimental results. It also shows its novelty in solving the problem.
- The authors provided comprehensive supplementary materials to help readers to understand the proposed method as well as to reproduce the experiments.
- The structure of the paper is well designed, and the writing is clear.

Cons:
- While this paper claims that the proposed method can quantify the reliability of neural network representation-driven hypotheses, there could be more examples (more realistic models on more tasks) to demonstrate the method's effectiveness in more scenarios. Currently, the examples in this paper are almost all using basic NN components on image inputs. The performances are not obvious if the networks contain parts like residual connections or recurrent structures, either the inputs are sentences rather than images.
- Related to the previous point, the boundary between where this method can be applied and can not be applied is not so clear. It would be better if the authors could give such guidance for people to use this method.
review_title: Good work on reliability analysis of neural network representation learning

Meta-review: Four reviewers evaluated your work and provided a detailed review with many suggestions. I also think that there is an interesting idea and encouraging results but there is a lack of numerical results and still some parts are still unclear and need to be polished. Consequently in its  current form, the paper can not be accepted for publication. I would advise you to carefully follow the remarks of reviewer 1 to improve  the paper. 


Decision: Reject