Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to analyze a binary neural classifier's input-output relationship.
A straightforward, brute-force sampling is forbidden due to the high dimension and continuity of input distribution.
To achieve efficiency, the authors reformulate the problem as density of states (DOS) sampling in physics, and propose to apply a gradient-based proposal which facilitates the exploration of under-explored output values.
The main contributions are:

- making the output histogram problem tractable.
- analyzing the input-output relationship in the entire input space and revealing some interesting properties by sampled statistics.
- introducing a new perspective to understand the behavior of neural network classifiers.

strength_and_weaknesses: **[Strength]**

- The idea of trying to understand neural networks through input-output mapping relationships is groundbreaking; it also makes sense to draw on solutions from a related field (condensed matter physics).
- Various experiments well illustrate that the statistics sampled by the proposed method (GWL) are close to the ground truth (enumeration) and significantly better than directly sampling (from in-dist test samples).

**[Weaknesses]**

- While it is possible to describe the output histogram problem using a domain specific approach (physical language), I still wonder if there is another way (perhaps a more general sampling algorithm) to solve this problem? The authors do not seem to discuss this sufficiently in the Related Work section.
- Are the properties of neural network input-output mapping observed by the authors in their experiments (e.g., mapping unrecognizable human samples to very negative logit values) consistent or contradictory to some previous literature? Making some comparisons would make the overall analysis a bit more comprehensive.
- Fig. 6 is interesting, but I don't particularly understand why logit values for larger samples look "darker" implying that the *background* is more important than the *foreground* for the classification task? Perhaps the authors could give a more detailed explanation on this?


clarity,_quality,_novelty_and_reproducibility: The articles are clearly written and of good quality.

The use of a physical perspective for interpretable analysis of the mapping of neural networks is novel.
summary_of_the_review: In general I think it is novel and interesting to use a physical perspective to understand the mapping of neural networks. However, there is very little discussion or comparison of related work throughout the paper (e.g., other non-physically inspired sampling algorithms, or other work on the analysis of neural network mappings; see the Weakness section above). This results in this work looking less than complete, and weaken their conclusions. I am inclined to marginally reject it.


========== post rebuttal ==========

> We update section 5.3 (and in appendix D) to present some of our interpretation of the results, such as why the black background appears when the logit gets larger.
> In the related work section, we compare the methods with MCMC samplers and position our method in terms of performance characterization literature.

Thank you for clarifying this and adding Section 5.3 as well as the supplementary material for more explanation. I feel that the two most significant flaws (inadequate discussion of the related work and too few explanations of the Fig. 6) have been addressed, thus improving my rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The essence of the paper is that one should examine the behavior of a neural network by studying how the output logits change as a function of the entire input space rather than only examine the behavior based on the sampled data distribution. The paper then illustrates how one could utilize intuition regarding a correspondence between state density representations in physics to histograms of logits to design a sampling scheme that combines wang-landau algorithm with gradient with gibbs sampler to explore the vast space of output possibilities.  The algorithm is applied to a toy data set with reduced dimensionality (of 25 dimensions) constructed out of MNIST to illustrate the utility and inner working of the concept. This is then followed by tests on MNIST and illustration of certain insights (e.g. how negative logits correspond to human unrecognizable images and about the nature of what information may be used by the neural net).  Open issues in the sampler are discussed and it is articulated that this form of extensive exploration of the energy density is important for understanding neural net behavior in the entire input space of possibilities.
strength_and_weaknesses: Strengths:
Offers the important point that one should explore the entire input space and the input's correspondence to output logits. Sampling methods used in physics literature are offered as a possible way to do the exploration efficiently instead of a brute force approach.

Weaknesses:
The emphasis that one has to explore the relationship between the entire input space and output space is well known (see for example: performance characterization literature in the 90's,  http://haralick.org papers on performance characterization. There were explorations involving mapping of boolean random series to outputs for nonlinear operators (e.g. mathematical morphology).  Another example of such characterization in pattern classification settings is illustrated in Gao et al, Statistical Characterization of Morphological Operator Sequences in ECCV 2002).  However, the results in the present paper are not yet convincing to me - I find the visualizations in figures 2, 3, 5, 6 hard to judge. While there is illustration in the toy example that the algorithm seems to be performing correctly via quantitative overlay of the enumeration (groundtruth) against the paper's method, the practical utility of the methodology to gain insights is still not clear. Moreover, there are open challenges with respect to getting the sampling scheme to be efficient as rightly articulated by the authors.
Apart from the fact that the exploration of the input space is shown in the paper, the interpretation of the results is vague. 


clarity,_quality,_novelty_and_reproducibility: Clarity:  The paper is reasonably clear.
Quality:  The paper quality can be strengthened as the results are still preliminary.
Originality:  The paper combines well known elements in the sampling and physics literature. The problem addressed is a difficult one and the importance of the problem has been articulated in the past. Thus, the authors need to be commended for their work.  
Reproducibility:  While adequate information is provided to implement the algorithm, it would be recommended to have an open source implementation with the ability to reproduce the experiments.
summary_of_the_review: I like the overall idea of the paper and believe that complete characterization of the input to output behavior, over the entire input space, for a trained net will be invaluable for providing guarantees of performance.  However, in my view, the paper in its present form is still not ready for publication.

After the revisions presented by the authors I have increased my original rating and have presented my viewpoints to the reviewers and area-chair.  The paper has been strengthened and is addressing an important aspect of performance analysis.  After my discussion with the other reviewer and area-chair my revised rating does not change.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to explore the daunting task of "modeling the entire space of input" to the output of a binary neural network. This is obviously impossible for any reasonably sized feature space. So, the paper proposes to do this by modeling the output of a neural network to the "density of states" concept in physics, where an efficient way to estimate it via the "Wang-Landau Algorithm", which seems to be a metropolis-hasting MCMC method. Wang-Landau algorithm can still yield many unrelated samples, so the paper proposes another recent work called Gibbs-With-Gradients (GWG) to be used as the sampler. GWG can leverage the model's learned weights to propose distributions that are closer to seen examples. Doing this exercise yields some interesting observations, the biggest of which is the fact that an enormous amount of possible inputs map to a very high-confidence prediction regime for a neural network.  
strength_and_weaknesses: **Strengths**

*[S1]:* Very interesting results: As mentioned in the summary above, the main result of this paper is quite interesting. The result can be potentially used to better understand adversarial attacks, out-of-distribution generalization, and learning mechanisms of neural networks in general. While the paper does not fully address the above possibilities, I remain hopeful :) 

*[S2]:* Clearly written: The paper is very easy to read, despite this reviewer's unfamiliarity with either the "density of states" or the "Wang-Landau Algorithm"; After brief Wikipedia-based learning of these concepts, the paper made the parallels clear and I felt like I could map these concepts to things already known in ML literature and statistical properties. (Though I might have misunderstood something, hence noted confidence of 3 below).  Furthermore, the experiments are clearly written and well-explained.

**Weaknesses**

*[W1]:* Some obvious questions were left unanswered: For me, the paper was a little baffling due to how little analysis beyond the "main result" this paper goes into. Some questions that I would like to have seen discussed or at least mentioned (even if it is in "future works"):

W1.1. Why do most of the "human unrecognizable images" map to very negative logit values, instead of vice-versa; The positive-negative logit is normally considered arbitrary in binary classification (Two classes can be "assigned" to either side of them). Is there a reason for this asymmetry? What are its implications? Is one of the classes harder to create an adversarial attack and/or higher generalization than the other? 

W.1.2 What are the implications of this finding in the context of decision-making, explainability, and "model evaluation? The abstract mentions that it will be explored in "future works" but in what manner? I am certainly not expecting this paper to do any of these "works" but I am quite unclear about how the findings from the paper can go beyond an interesting result and more towards how, in the words of the paper, "opens a new gate for neural model evaluation"
clarity,_quality,_novelty_and_reproducibility: **Clarity:** To me, it is quite clear; I do think there are a few more things that I wish were on the paper. See W1 above, but the clarity is excellent for what is on the paper. 

**Quality:** See strengths and weaknesses above.

**Novelty:** It is quite new and exciting! 

**Reproducibility:** The analysis is built on existing tools, and clearly explained. While no code is promised, I think reproducibility is excellent. 
summary_of_the_review: I am currently moderately positive about the paper and the very interesting "main" finding of the paper. However, I refrain from a higher score because the discussion leaves a lot to be desired about whether this is a special case or can be used as a general-purpose analysis of any binary (or multi-class) classifiers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of efficiently understanding the relationship between the input-output mapping of a neural network over a large and representative set of input spaces. To do so, the authors draw an analogy between a neural network model and physics's density of states problem. In doing so, they can design a new sampling strategy inspired by the Wang-Landau algorithm. They augment the algorithm with a gradient-based approach that can sample inputs with maeaningful structure. They conduct experiments on ResNet and CNN to show the approach's effectiveness. 

strength_and_weaknesses: -= S1 =- The analogy that the paper draws between neural networks' output histogram and the density of states problem in physics is well-motivated. This open opportunities to apply established methods from physics to the domain of  neural networks. 

-= S2 =- The authors make the observation that the randomized sampling method of WL is not as applicable in the context of neural networks and modify that by adding a gradient-based approach that includes more structured inputs. 

-= W1 =- The paper considers a very limited set of neural network models that are not state-of-the-art. The paper can benefit from using a recent architecture to showcase the utility of the approach. The same applies to datasets. It is unclear why binary classification on MNIST is an interesting case to apply this technique to. 

-= W2 =- It is not sufficiently clear how the new insights uncovered by applying this technique augment our understanding of neural networks. The paper can benefit from spelling that out more explicitly. 

-= W3 =- It is also unclear how this approach can be extended to the case of multiple labels. The paper can benefit from a discussion on that aspect as >2 label classification is more prevalent than a binary classifier. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. There is novelty in the proposed algorithm. No artefacts were made available for reproducibility. 

summary_of_the_review: While the paper is well-motivated and the algorithm proposed is relatively novel, the experiments are limited in data set and models studied. Additionally, the scope of the paper -- binary classification -- is not very broad.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thank you for your submission to ICLR.  Overall, this submission received a very large amount of discussion, both within OpenReview and within the AC-reviewer meeting held on this paper.  As such, I will summarize the basic understanding of the paper as it currently stands, highlighting the positives and negatives.

On the positive side, this paper considers an interesting and potentially valuable contribution to the field: trying to better understand how networks react to the "entire" distribution over input space, to see what kind of distributional properties exist in the outputs. While (as reviewers pointed out, and as the authors acknowledged in their responses), this is by no means a truly new perspective, the paper nonetheless tackles some important problems in this space and brings to bear interesting techniques (namely, the Wang-Landau algorithm) to this problem.  Naturally, understanding the _full_ distribution of possible outputs over a high-dimensional input space represents an intractable problem, the approaches here cast a new perspective on this challenge, one which may be valuable to researchers going forward.

The main downside to this work, however, is simply that it was unclear to the reviewers and myself "just how well" the proposed approach really works at casting insight into this problem.  Specifically, the authors ultimately propose a method based upon the Wang-Landau, essentially a particular form of MCMC estimator with gradient information, to compute the necessarily integrals over input space.  However, there is very little comparison of the approach to alternative, and similar, MCMC methods that are more widely used within the ML space, such as Hamiltonian MC methods (see e.g. https://arxiv.org/pdf/1701.02434.pdf).  The experimental results don't really compare the proposed method to alternative MCMC approaches, or fully illustrate the consequences of this approach is a manner that substantially challenges the common knowledge of adversarial inputs.

For these reasons, the ultimate feeling is that the paper is not quite ready yet for publication, and would benefit substantially from additional experimentation, evaluation, and situation in contrast to alternative approaches within the field.

 
justification_for_why_not_higher_score:  It would be reasonable to accept this paper, if value was to be placed mainly on the conceptual contributions.  However, it was also felt that additional experimentation, comparison, and situation within existing methods in ML would be valuable here.
justification_for_why_not_lower_score: NA
Decision: Reject