Output Distribution over the Entire Input Space: A Novel Perspective to Understand Neural Networks
Authors: Weitang Liu
Yi-Zhuang You
Jingbo Shang
Venue: Submitted to ICLR 2023 (2023)
Keywords: model evaluation
comprehensive input-output mapping relation

Abstract: Understanding the input-output mapping relationship in the \emph{entire input space} contributes a novel perspective to a comprehensive understanding of deep neural networks. In this paper, we focus on binary neural classifiers and propose to first uncover the histogram about the number of inputs that are mapped to certain output values and then scrutinize the representative inputs from a certain output range of interest, such as the positive-logit region that corresponds to one of the classes. A straightforward solution is uniform sampling (or exhaustive enumeration) in the entire input space but when the inputs are high dimensional, it can take almost forever to converge. We connect the output histogram to the \emph{density of states} in physics by making an analogy between the energy of a system and the neural network output. Inspired by the Wang-Landau algorithm designed for sampling the density of states, we propose an efficient sampler that is driven to explore the under-explored output values through a gradient-based proposal. Compared with the random proposal in Wang-Landau algorithm, our gradient-based proposal converges faster as it can propose the inputs corresponding to the under-explored output values. Extensive experiments have verified the accuracy of the histogram generated by our sampler and also demonstrated interesting findings. For example, the models map many human unrecognizable images to very negative logit values. These properties of a neural model are revealed for the first time through our sampled statistics. We believe that our approach opens a new gate for neural model evaluation and shall be further explored in future works.


Under review as a conference paper at ICLR 2023
OUTPUT DISTRIBUTION OVER THE ENTIRE INPUT SPACE: A NOVEL PERSPECTIVE TO UNDERSTAND NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Understanding the input-output mapping relationship in the entire input space contributes novel perspective to comprehensive understanding of deep neu- ral networks. In this paper, we focus on binary neural classifiers and propose to first uncover the histogram about the number of inputs that are mapped to certain output values and then scrutinize the representative inputs from a certain output range of interest, such the positive-logit region that corresponds one of the classes. A straightforward solution is uniform sampling (or exhaustive enumer- ation) in the entire input space but when the inputs are high dimensional, can take almost forever to converge. We connect the output histogram to the den- sity of states in physies by making an analogy between the energy of a system and the neural network output. Inspired by the Wang-Landau Landau algorithm designed for sampling the density of states, we propose an efficient sampler that driven to explore the under-explored output values through a gradient-based proposal. Compared with the random proposal in Wang-Landat algorithm, our gradient- based proposal converges faster as it propose the inputs corresponding to the under-explored output values. Extensive experiments have verified the accuracy of the histogram generated by our sampler and also demonstrated interesting find- ings. For example, the models map many human unrecognizable images to very negative logit values. These properties of neural model are revealed for the first time through our sampled statistics. We believe that our approach opens a new gate for neural model evaluation and shall be further explored in future works.
INTRODUCTION
Understanding the input-output mapping relationship in the entire input space contributes novel perspective to a comprehensive understanding of deep neural networks. Existing methods approx- imate such mapping relations through the evaluation on a certain subset of the entire input space, such measuring the accuracy on in-distributtio test sets Dosovitskiy et al. (202 Tolstikhin et al. (002); Steiner et al. (202I): Chen et al.) (202I); Zhuang et al. (002);; ); He et al. (2015), out-of distribution (OOD) test sets (Liu et al. 2020 Hendrycks & Gimpel 2016; Hendrycks et al. 2019 Hsu et al. 2020 Lee et al. 2017;2018) and adversarial test sets Szegedy et (2013); Rozsa et al. (2016): Miyato al. 2018); Kurakin al. (016)) However, none of the existing evaluations can offer a comprehensive understanding that covers the entire input space, including all kinds of inputs mentioned above and even those human unrecognizable inputs as shown in Fig
As a pilot study, we focus on binary classification - given a trained binary classifier, we aim to uncover histogram that counts how many samples the entire input space mapped certain logit values, i.c., the distribution of the output values, as shown in Fig lb A straightforward solution uniform sampling (or exhaustive enumeration) in the entire input space but when the inputs are high dimensional, it can take almost forever to converge. Therefore, it calls for a novel efficient sampling method over a neural model's output space. Note that, as side product of the sampling procedure, one can expect that this histogram also offers fine-grained information such as some representative input samples corresponding to a certain range output values.

---

Under review as a conference paper at ICLR 2023
Human recognizable
Unrecognizable
The number'ratio of samples
(e.g. great generalization)
Humans
agree w/ model
1
y-9
y-10
y=
Humnns
(e.g. adversarial)
(e.g OOD)
y
0)
1
disagree
w/ modcl
(a) Different types of input samples
(b) The histogram of an example binary classifier
Figure I: Input types and the example output histogram when the task is binary classification be- tween digits 0 and 1. The entire input space covers all possible gray-scale images of the same shape. y is the output (logit) of input x.
We connect the output histogram problem to the density of states (DOS) prob- lem in physics by making analogy between the system energy and neu- ral network output, as shown in the right figure. If one follows the physics language to describe be viewed as the configuration x of the system; the neural network output (e.g., logit values in binary classifier) y(x) corresponds to the energy func- tion E(x): the desired output histogram can be obtained through the DOS (a.k.a,, the entropy, S ((E()), the log scale of DOS), which is the count of the configurations given the energy value. Note that the density of states by defi- nition is over the entire input space, which aligns perfectly with our objective.
Physiks
ML imsacc)
configuraion
problem, the imput to the neural network
Isit modk
Nearal Network
+ Clapat
Emerge taxt
$
Donsty of case
Ontger hosogram Skyta0
(D0B) StEinO
Inspired by the Wang-Landau algorithm (Wang & Landau 2001) designed for density of states, we propose an efficient sampler that driven to explore the under-explored output values through gradient-based proposal. If the binary classifier is well-trained, it is reasonable to believe that in-distribution inputs (images) which usually have certain semantic structures are concentrated in certain output ranges. If one follows the random proposal in the Wang-Landau algorithm, it is dif- ficult to propose the inputs with meaningful structure (or even in-distribution inputs), thus possibly preventing the sampler from exploring the corresponding output values. Thus, we propose to apply gradient-basse baased proposal called Gibbs- Seehos-wit--rrriiiie Gradients (GWG) (Grathwohl et al. 202I) which proves to be efficient to propose in- di-triiibutionn inputs for trained model.
With the help of this new sampler, we can reveal some new understanding of the models for the real-world dataset, the dominant output values
entire input space. First, in our experiments very negative and correspond to the human-unrecon uumna--ureecoiizabble inputs. This indicates the models may map an overwhelmingly large number of unrecognizable images to the overconfident prediction probabilities. Second, we can derive the relative difference between the dominant peak of output values and the other output values, especially those where the in-distribution inputs correspond to. The output values where the in in-distribution inputs correspond to are also dominated by the human- unrecognizable inputs. This result presents significant challenges to the OOD detection problems. Third, we observe a clear trend of the representative samples in a CNN model and speculate it simply utilizes the background to predict the labels of the digits.
Our contributions are summarized as follows.
We work on the challenging problem to uncover the output distribution over the entire input space. Such output distributions offer a novel perspective to understand deep neural networks.
We connect this output distribution problem to the density of states problem in physic and suc- cessfully tailor Wang-Landau algorithm using gradient-based proposal, which is must-have component to sample the entire output space as much as possible, improving the efficiency.
We conduct extensive experiments toy and real-world datasets based on CNN and ResNet-18 to confirm the correctness of our proposed sampler and discover novel and interesting findings.
We believe that approach opens explored in future works. For example, one can can utilize our sampler to estimate the intrinsic ratio of in-distribution samples given range of interest with human evaluation shown in Sec. 5.3
gate for neural model evaluation and shall be further
2

---

Under review as a conference paper at ICLR 2023
2 PROBLEM DEFINITION
In the traditional setting, binary neural classifiers model the class distribution through logit 2. A neural classifier parameterized by 9 learns po po(z)x) 5(2 yo(x)) through function yo - R. where x E Ω Ω S {0,... N] for images, and is the Dirac delta function. Ω2 is aligned with setting be discrete.
What the above model does not define is the distribution of the data x. This paper aims to obtain the output value distribution of binary classifiers in the entire input space: 52 {0, ..... N) Here assume that the data distribution p(x) follows the uniform distribution over the domain ! of x and denote its measure by μ.. We define the joint distribution
pa(z, x) (=(2)x)u(x)
Our goal is only the logit (output) distribution. We marginalize the above joint distribution define the density given the logit 2:
(o(2)= = Σ po(2|x))(()) = Σ -
xgn
To sample from the distribution (=(2), we can first sample x; ~ Uniform((2), then condition on the sampled x,, sample 2; (z(xx)) While uniform sampler in principle can resolve our problem, takes almost forever to converge.
3 METHOD
In this section, we first discuss the connection between our problem to density of states (DOS), background, and present
introduce both Wang-Landau algorithm and new sampler Gradient- Gradient-Wang-Landde algorithm.
3.1 CONNECTION TO DENSITY OF STATES (DOS) IN PHYSICS
In statistical physics, given the energy function E x 3 E R. the DOS p(E) defined as
((E) = Σ 8( - FI()))
XER
where 5 is the Dirac delta function and Q is the domain of where x is valid. The DOS treated as probability distribution in the energy space, whose fog-probability is defined as the entropy S:
p(E) cxp(S(E))
Boltzmann constant is assumed to be 1 in our setting. DOS is meaningful because many physical quantities depend on energy or its integration but the specific input x.
We connect the output histogram to DOS in physics by making an analogy between the system energy & = Exx) and neural network output y(x). This connection is based the observation that the energy function in physics maps an input configuration to ecalar-valued energy; similarly, a binary neural classifier maps an image to logit. Both the logit and energy are treated as the direct output of the mapping. Other quantities, such as the loss, are derived from the output. The desired output histogram can be obtained similarly through sampling the DOS (a.k.a,, the entropy (xE())) or (xx(x) ) in the log scale) which is the count of the configurations given the energy value. The output histogram and DOS are defined in the entire input space.
3.2 WANG-LANDAU ALGORITHM AND GIBBS- WITH-GRADIENT
Wang-Landau algorithm is Markov chain Monte Carlo sampler that samples DOS. Since the true distribution ((E) is what we are interested in sampling but its formula/model is unknown, we need to approximate it. The Wang-Landau algorithm uses a histogram to store the current estimation Š. improves the sampling efficiency by sampling the inverted distribution:
p(x) exp(-$((F)))) (F(()))
3

---

Under review as a conference paper at ICLR 2023
By sampling p(x). we can get ensemble of & via E() whose probability distribution is:
x(E)
Σ
- E(x))
x=p(xx
When SE)) approaches S(E), the energy distribution =((' approaches to E-independent constant for all the accessible energy E.
Gibbs-Wiie WitdeGra (GWG) is used for energy-based models (EBM) by sampling
log) = f(x) log Z,
where f(x) the unnormalized log-probability, Z the partition function, and x is discrete. Typical Gibbs sampler iterates every dimension 2; of x, computes the conditional probability p(a: #441, ..., TD), and samples according to this conditional probability.
When the training data x are natural images and the EBM learns decently well, the traditional Gibbs sampler wastes much of the computation. For example, most pixel-by-pixel iterations over F, in MNIST dataset will be on the background which should stay black. GWG proposes smart proposal that picks the pixel zy that is more likely to change, such as the pixels around the edge between the bright and dark region of the digits.
3.3 WANG-LANDAU WITH GRADIENT PROPOSAL
Directly applying Wang-Landal algorithm is not enough as it uses random proposal, because a trained neural model learns preferred mapping through the loss function. For example, binary classifier should map the training inputs to either the sufficiently positive or negative logit values which ideally should correspond the sampler explores and generates the peak centered at 0 where most random samples correspond to as shown in Fig. [b] is almost impossible for the sampler with a random proposal to propose the inputs with meaningful structure (or even in-distribution inputs) so that the other possible output values are explored. Of course, whether those output values correspond to in-distribution inputs is only confirmable after sampling. In summary, it is extremely difficult for the random proposal in Wang-Lander algorithm to explore (almost all) the possible output values.
the extremely rare but semantically meaningful inputs. After
We propose to use the framework of Wang-Landau algorithm but replace the proposal distribution with the Gibbs- With-Gradients (GWG) sampler which has a gradient proposal, since the gradient proposal takes the advantage of model's learned weights to propose inputs. In order to sample the distribution of the output prediction through GWG, we define log- log-probability [(x as:
f(x) = S(x(x)
where is the count for the bin corresponding to y(x). The fixed f(-) in the original GWG is changing in our sampling process given the input x, since the formula for S is unknown and we can only estimate the output distribution as we did in Wang-Landau algorithm. Moreover, the GWG requires the gradient of f. but the S is not differentiable since it is approximated through discrete bins. We adopt a first-order differentiable interpolation for the discrete histogram of entropy.
In summary, similar the original Wang-Landau algorithm, first initialize two histograms with all of their bins to O. One of these histograms is for entropy S, and the other histogram, H, is a counter of how many times the sampler workers visited a specific bin and H is also for the flatness check. We first preset the number of iterations. When H passes the flatness check, it enters the next iteration loop with the step counter reset to Every step in the while loop until the flatness check passes, we interpolate the entropy in the histogram to get differentiable interpolation and take the derivative of the negation of the entropy with respect to the output Z and the inputs x through the chain rule. GWG uses this gradient to propose the next input that is likely to have lower entropy and be accepted by the sampler. his procedure drives the sampler to visit rare samples whose logit values correspond to the lower entropy until S converges. This proposal also goes through accept-reject procedure in the GWG. Once the flatness is met, the bins of H are reset to 0 and the step size is halved before new iteration. Our proposed algorithm is in Alg. 1 in Appendix.
Monte-Carle
4

---

Under review as a conference paper at ICLR 2023
4 RELATED WORKS AND DISCUSSIONS
Performance Characterization has long been explored even before the era of deep learning (Haral-
ick 1992 Klette et al. 2000; Thacker et al. 2008). The input-output relationship has been explored
for simple functions (Hammitt & Bartlett 1995) and mathematical morphological operators Gao
et al. 2002] Kanungo& & Haralick] 1990). Compared to existing performance characterization ap-
proaches (Ramesh et 1997; Bowyer Phillips 1998 Aghdasi 1994, Ramesh Haralick 1992
1994). our work focuses on the output distribution (Greiffenhagen et al. 2001) of neural network
over the entire input space (i.e., not task specific) following the blackbox approach (Courtney et al.)
1997 Cho et al. (197) where the system transfer function from input to output is unknown. Our set- ting shall be viewed as the most general forward uncertainty quantification case (Lee & Chen 2009) where the model performance is characterized when the inputs are perturbed (Roberts et al. 2021).
To our best knowledge, we demonstrate for the first time that the challenging task of sampling the entire input space for modern neural networks is feasible and efficient by drawing the connection between neural network and physics models. Our proposed method can offer samples to be further integrated with the performance characterization methods mentioned above.
Density Estimation and Energy Landscape Mapping Previous works in density estimation fo- cus on data density Tabak & Turner (2013); Liu et al. (202I), where class samples are given and the goal is to estimate the density of samples. Here we are not interested in the density of the given dataset, but the density of all the valid samples in the pixel space for a trained model. Hiill et al. (011)); Barbu & Zhu (2020) have done the pioneering work in sampling the energy landscape for energy- models. Their methods specifically focus on the local minimum and barriers of the energy landscape. We can relax the requirement and generalize the mapping on the "output" space where either sufficiently positive or sufficiently negative output (logit) values are meaningful in binary classifiers and other models.
Open-world Model Evaluation Though many neural models have achieved the SOTA perfor- mance, most of them are only on in-distribution test sets (Dosovitskiy et al.] [202T] Tolstikhin et al.]
2021 Steiner et al.] 2021; Chen al. [021]] Zhuang et al. 2022] He et al. [0015] Simonyan &
Zisserman 2014 Szegedy et al. 2015 Huang et al. 2017]] Zagoruyko & Komodakis 2016). Open-
world settings where the test set distribution differs from the in distriiibttion training set create special challenges for the model. While the models have to detect the OOD samples from in-distribution
samples (Liu et al. 20200 Hendrycks & Gimpel| 2016 Hendryeks et al.] 2019 Hsu et al. [020]
Lee et al. 2017, 2018: Liang et al. 2018)] Mohseni et al. 2020; Ren et | 2019)) we also expect sometimes the model could generalize what it learns to OOD datasets (Cao et al. 2022 Sun Lij 2022). It has been discovered that models have over-confident predictions for some OOD samples that obviously do not align with human judgments (Nguyen et al. 2015). The OOD generalization becomes more challenging because of this discovery, because the models may not be as reliable as we thought they were. Adversarial test sets Szegedy et al. 2):13;; Rozsa et al. 2016); MMyato et al.) (2018): Kurakin et al. (2016); Xie al. (2019); Madry et al. (2017) 7) also present special challenges as models decisions are different from those of humans. Having full view of input-output relation with all the above different kinds of test sets under consideration important.
Samplers MCMC samplers (Chen et al. 2014 Welling & Tch 2011 Li et al. 2016 Xu et al. 2018) are developed to scale to big datasets and sample efficient with gradients. Recently, Gibbs- With-Gradients (GWG) (Grathwohl et al.] (02TT) is proposed to pick the promising pixel(s) as the proposal. To further improve sampling efficiency, CSGLD (Deng et al., 2020) drives the sampler to explore the under-explored energy using similar idea as Wang-Landau algorithm (Wang & Lan- dau] 2001). The important difference between our problem setting and the previous ones solved by other MCMC samplers is the function or model as distribution to be sampled from is unknown. Wang-Landau algorithm utilizes previous approximation of the distribution to drive the sampler to explore the under-explored energy regions. This algorithm can be more efficient through paralleliza- tion (Vogel et al. (2013) CCnha- Netto et al.) 2008), fire (Junghans et al.) 2014 Li & Eisenbach) 2017) and extended to multi-dimensional outputs (Zhou et al. (0066). While the previous samplers can be applied to high dimensional inputs, the energy functions written by physicists are relative simple and symmetric. However, modern neural networks are complex and hard to characterize per- formance (Roberts al., 2)21). We assume agnostic of the output properties of the model and thus apply the Wang-Landan algorithm to sample the entropy as a function of energy but with the gradi- proposal in GWG to make the sampler more efficient. Similar to GWG, our sampler can propose
5

---

Under review as a conference paper at ICLR 2023
the imputs corresponding to the under-explored regions of outputs. Improvements of efficiency can benefit from a patch of pixel changes.
EXPERIMENTS
In this section, we apply our proposed Gradient Wang-Landee sampler to inspect a few neural net- work models and present the discovered output histogram together with representative samples. The dataset and model training details are introduced in Sec. 5.1 We first empirically confirm our sam-
pler performance through a toy example in Sec. 5.2 We then discuss results for modern binary classifiers in Sec. 5.3 and Sec. 5.4. Hyperparameters of the samplers tested in are Appendix
5.1 DATASETS, MODELS, AND OTHER EXPERIMENT SETTINGS Datasets As aforementioned, we focus on binary classification. Therefore, derive two datasets from the MNIST datasets by only including samples with labels (0, 1}. The training and test splits are the same as those in the original MNIST dataset.
Toy is simple dataset with 5 x binary input images we construct. is designed to make feasible the bruteforce enumeration over the entire input space (only 25x5 different samples). We center crop the MNIST samples from {0, 1} classes and resize them to 5 x 5 images. We compute the average of the pixel values and use the average as the threshold to binarize the images the pixel value lower than this threshold becomes O; otherwise, it becomes 1. The duplicates are not removed for accuracy after resizing since PyTorch does not find duplicate row indices.
MNIST-0/1 is an MNIST dataset whose samples only have the 0,1 labels. To align with the GWG setting. the inputs are discrete and not Z-normalized. Therefore, in this dataset, the input x is 28 28 dimensional with discrete pixel values from (0,..., 255).
Neural Network Models for Evaluation Since the focus of this paper is not to compare different neural architectures, given the relatively small datasets have, simple CNN and ResNet-18 (He et (..15) Each pixel of the inputs is first transformed to the one-hot encoding and passed to 3-by-3 convolution layer with 3 channel output. The CNN model contains 2 convolution layers with 3-by-3 filter size. The output channels are 32 and 128. The final features peraage-pooled and passed to fully-connected layer for the binary classification.
train two types of models,
Please keep in mind that our goal in this experiment section to showcase that our proposed sampler can uncover some novel interesting empirical insights for neural network models. Models with different architectures, weights due to different initialization, optimization, and/or datasets will lead to different results. Therefore, our results and discussions are all model-specific. Specifically, we train a simple CNN model to classify the 5 x 5 binary images in the Toy dataset (CNN-Toy). The test accuracy of this CNN-Toy model reaches 99.7%, which is almost perfect. We train simple CNN model to classify the 28 x 28 grey-scale images in the MNIST-0/1 dataset (CNN-MNIST-O/T) The test accuracy of CNN-MNIST-OI/ model is 97.8%. We train a ResNet-18 model to classify the 28 28 grey-scale images in the MNIST-0/1 dataset (ResNet-18--W) 18-MNIST-0/1). The test accuracy of ResNet- 18-MNIST-O/1 model is 100%.
Sampling Methods for Comparison We compare several different sampling methods (including our proposed method) to obtain the output histogram over the entire input space.
Enumeration generates the histogram by enumerating all the possible pixel values as inputs. This is a rather slow but the most accurate method.
In-dist Test Samples generates the histogram of the inputs based on the fixed test set. This is commonly used in machine learning evaluation. It is based on very small and potentially biased subset of the entire input space.
Wang-Landau algorithm (WL) generates the histogram the Wang-Landau algorithm with the ran- dom proposal. Specifically, we randomly pick one pixel at time and change it to any valid (discrete) value as in this implementation
Gradient Wang-Landar (GWL) generates the histogram by our proposed sampler of Wang- Landau algorithm with gradient proposal.
http:::://wwww.haatt/ es. wanglaand .py

---

Under review as a conference paper at ICLR 2023
Relative entropy of output values Toy dataset
5
Enumeration (ground truth)
GWL (ours)
Slybo))
0
In-dist samples
-5
10
30
20
10
Figure 2: Output histograms of CNN-Toy obtained by different sampling methods. The in- distribution samples are only a very small portion in the output histogram. We also present the representative samples obtained by GWL given different logit values.
Relative Entropy of output values
40
GWL
-1000
in-dist test samples
30
Sty(s))
-2000
20
-3000
4000
-10
X(O)
Figure 3: Output histograms of CNN-MNIST-0/1 obtained by different sampling methods. The blue scale for GWL and the red scale is for In-distribution Test Samples. We also present the representative samples obtained by GWL given different logit values (more in Fig. in Appendix)
5.2 RESULTS OF CNN-TOY
Given the CNN-Toy model, we apply Enumeration. GWL, and In-dist Test Samples to obtain the output histograms, shown in Fig. 2 Note that our GWL method samples the relative entropy of different energy values as duplicate x may be proposed. After normalization with the maximum entropy, the GWL histogram almost exactly matches the Enumeration histogram which is the ground truth histogram. This confirms the accuracy of our GWL sampler and we can apply it further to more complicated models with confidence.
Remarkably, this histogram is quite different from the expectation we presented in Fig. Tb this histogram is even not centered at has the expected subdominant peaks on both the positive and negative sides. Instead, the dominant peak is so wide that it covers almost the entire spectrum of the possible output values. From coarse-grained overview, most of the samples are mapped to the center of logit -5 with a decay from -5 to both sides in the CNN-Toy model. This shows the CNN-Toy model is biased to predict more samples to the negative logit values.
In Fig. 2] we also present the representative samples obtained by GWL given different logit values the CNN-Toy model. The visualization results suggest that the CNN-Toy model probably learns the digit "1" for positive logit values as the center pixels of the representative samples are white (see the three representative samples with logit values from to 20) and "0" for the very negative logit values as the center pixels of the representative samples are black (see two representative samples with logit values from -20 to -30.. From this example, one can see that the output histogram over the entire input space offer comprehensive understanding of the neural network models, helping the outputs, where the
researchers better understand critical questions such as the distribution model maps the samples to, and what the representative samples with high likelihood are.
5.3 RESULTS OF CNN-MNIST-O/1
Histogram Results by GWL The trial of applying GWL on the CNN-Toy model is encouraging and we now apply GWL to the CNN-MNIST-0/1 that is trained on a real-world dataset. The results shown in Fig. As our GWL reveals, the output histogram of CNN-MNIST-0/L similar to CNN-Toy's histogram, does not have the subdominant peaks. is also different from the presumed case in Fig. Tbl Compared with the output histogram of the CNN-Toy model (i.e., Fig. 2). this

---

Under review as a conference paper at ICLR 2023
Histogram plot for different stops
Histogram plot per iteration (itI)
1e6
Histogrrm plot per iteration fist)
4000
Sty(e)
5kpp=5000
1tops-1090
4000
xt-
Skytel)
in-0
2000
sieps- 00000
2000
21-1
stops 20300
-50
0
50
100
150
-00
-10
-20
0
-50
-10
-20
y(X)
(a) S of GWL((uur) for steps.
(b) S of GWL(ours) per iteration.
(c) S of WL per iteration.
Figure 4: Intermediate output histogram S per iteration. (a) GWL gradually explores the logit values in the first iteration. (b) GWL discovers the output histogram well within 2 iterations. (c) The original WL explores the output distribution much slower.
time, the peak is on the negative boundary and the histogram is skewed towards the negative logit values. S almost linearly decays to the positive logit values. While the in-distribution samples have logit values between -0 and 15 we expect, these samples are exponentially (i.e., e3000 at logit value -20 to e3100 at logit value 13, thousands in log scale) less often found than the majority samples whose logit values are around -55. From a fine-graines view, the CNN-MNIST-0/1 model tends to map the work (Nguyen et al., 2015) showed the existence of the overconfident prediction samples, our result shows a rough but quantitative performance of this CNN which serve baseline for further improvements. One may notice that in Fig. 3 there is still some output values (c.g., the rightmost positive logit region) that not yet covered by our GWL sampler. We believe that this calls for more future work to follow on more advanced efficient samplers.
samples to the very negative logit values. While previous
GWL is much more efficient than WL Since WL takes a much longer time to converge, we not able to obtain the converged results from WL. For the comparison purpose, we inspect the intermediate S results of the GWL and WL samplers, as shown in Fig. 4 As one can see from Fig. 4al in the first iteration, GWL has already been able to gradually explore the logit values efficiently from the most dominant output value around -55 to the positive logit values. Within only two iterations, as shown in Fig. Ab GWL can discover the output histogram covering the value range from -55 to 13. On the other hand, as presented in Fig. 4c in the first two iterations. the original WI. can only explore the output ranges from around 55 to -45; in the 3rd iteration, WI. converges significantly slower and never ends in a reasonable time. This result indicates that the GWL converges much faster than the original WL and is able to explore a much more diverse range of output values.
Manual inspection more representative samples As show in Fig. for the CNN-MNIST-
0/1 model, GWL can effectively sample input images from logit values ranging from -55 to 13. We further group these logit values per 100 bins (100 bins correspond to difference of 10 in logit value) in S, resulting in about 7 groups. For every group, we sample 200 representative input images. To make sure they not correlated, randomly pick 10-out-of-200 samples from every group in Fig. 6 in Appendix. We manually inspect the sufficiently positive group (e.g.. the last column in Fig. 6 and the sufficiently negative groups (e.g., the first five columns in Fig. (6) and there are no human recognizable samples of digits. We also observe an interesting pattern that the logit value increases, more and more representative samples have black background. This result suggests that the CNN-MNIST-0// model may heavily rely on the background classify the images (Xiao et al. 2020). We conjecture that because the samples in the most dominant peak are closer to class O samples than class samples (Appendix.) (D)- In summary, although CNN-MNIST-0/1 holds a very high in-distribution test accuracy, is far from a robust model because it does not truly understand the semantic structure of the digits.
sample every 1000 steps. For demonstration purposes,
Discussion Fig. presents challenges to the OOD detection methods that may be more model- dependent than we thought before. If the model cannot map most of the human unrecognizable sam- ples with high uncertainty, the likelihood-based OOD detection methods (Liu et al. 2020 Hendrycks & Gimpel| 2016) cannot perform well for samples in the entire input space. Fig. 6 shows the in- puts with the in-distribution output values (output logits of the red plot) of the CNN model may not uniquely correspond to in-distribution samples. More rigorous experiments to a definite conclusion are yet required as future work.
8

---

Under review as a conference paper at ICLR 2023
Relative Entropy of valure valure
Relative Entrogy of ourpot values
GWL
GNL
bidia
lndiet
-0000
-506
30
-1006
-1000
Figure 5: Output histograms of ResNeet18-MNISSTI (b) Results with test set re-initialization.
(a) Results with random re-initializaaion..
obtained by different sampling methods. There may be sharp local minima in the output landscape causing a cliff around the logit value of -33 and making GWL "trapped". We have tried two variants address the "trapped" issue via (a) random re-in re-initialization and (b) test set re- re-initialization. The blue scale is for GWL and the red scale for In-distribution Test Samples. We also present the representative samples obtained by GWL given different logit values.
5.4 RESULTS OF RESNET-18 DESNET-1--MNIST-O//
When applying our GWL samplers to the can easily get trapped in some output regions. This is a fairly common phenomenon for Wang- Landau-based samplers,
model, we observe that the sampler
reported in (Vogel et al., 2018). We follow the common practice to re-initialize the sampler to the random samples every time it gets trapped. We let those workers run 1000 steps (10,000 pixels selected with replacement) before counting to S again. As shown in Fig. 5a. the smallest logit values in ResNet- 101-188-NISTTOO are around -220, much lower than those wide range of negative logit values corresponds to human unrecognizable inputs and there is obvious pattern observed in contrast to CNN-MNIST-0/I 's results.
of CNN-MNIST-0/1.
Interestingly, we observe a cliff around the logit value of -33. We try another variant for re- initialization: we re-initialize using the test set samples every time it gets trapped in certain output value. This time, as shown in Figure 5b) it can explore the output values larger than -33, We believe there may be sharp local minima in the output landscape, similar to the case discussed before Vogel et al. 2018).
Because of the "trapping" issue and the complexity of ResNet- 18 over CNN, have relax the flatness check a bit to let GWL converge for the first iteration. Because of the less rigorous flatness check, we do not draw conclusions about BesNNet-18-NNISS entropy differences. Compared with the CNN-MNIST-OV model, ResNet-18-MNN interesting phenomena and further exploration is needed to understand these phenomena.
evaluation of the relative has more
6 CONCLUSION
We aim to get full picture of the input-output relationship of a model through the inputs valid in the pixel space. We propose to use histogram to better understand the input-output distribution. When the inputs are high- high-dimensiooal,, enumeration uniform sampling is either impossible takes too long to converge. We connect the density of states physics to this histogram sampling problem. We propose use efficient sampler achieve this goal. We confirm empirically this can be achieved and uncover some new aspects of neural networks.
For future work, it is interesting to develop guarantees to acquire this input-output relationship in order to sample with more pixels, such as the ImageNet (Deng al. 2009). Most importantly, with this new sampler, can develop new insights into network architectures developed in the last decade for open-world applications.
and more efficient sampler that has theoretical
REPRODUCIBILITY
We provide fairly amount information to re-implement our sampler. The data processing is in the Sec. [. and algorithm is in Appendix B The hyperparameters and sampling details also listed in the Sec. [ We also provide different time stamp of steps for our samplers to indicate what to expect during the sampling procedure in Fig. 4 Of course, the Wang-Landau algorithm adopted
9

---

words: Transformers for image recognition at Under review as a conference paper at ICLR 2023
the prototypical and it subjects to some issues reported in its follow-up works, such as the discontinuity the boundaries between bins and trapping in one of the bins. These problems lead to some issues in our experiments and discussed them in Sec. 5.4 More advanced algorithms have been developed to resolve these issues.
8 ETHICS STATEMENT
Our method aims to provide a comprehensitve understanding of the neural models. This work will be applicable many applications, such as those in the safety and trusty-worthy machine learning. As pilor study, we do not anticipate the negative aspects of our work.
REFERENCES
Farzin Aghdasi. Digitization and analysis of manmographic images for early detection of breast cancer. PhD thesis, University British Columbia, 1994.
Adrian Barbu and Song-Chum Zhu. Mapping the energy landscape. In Monte Carlo Methods, pp- 367-420. Springer, 2020.
Kevin Bowyer and Jonathon Phillips. Empirical evaluation techniques in computer vision. IEEE Computer Society Press, 1998.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning, In International Conference Learning Representations, 2022. URLattpss///ppeevviiee http:eewwweeeeortreeeeee forum?
1d==0-rLLOR-CCA
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691. PMLR, 2014.
Xiangning Chen, Cho-Jui Hsich, and Boqing Gong. When vision transformers outperform resnets without pretraining strong data augmentations. arXiv preprint 2101::2006.01448. 2021.
Kyujin Cho, Peter Meer, and Javier Cabrera. Performance assessment through bootstrap. IEEE Transactions Pattern Analysis and Machine Intelligence, 19(11):1155-1198 1997.
Patrick Courtney, Neil Thacker, and Adrian F Clark, Algorithmic modelling for performance evalu- ation. Machine Vision and Applications, 95):219-222, 1997.
Antônio Gonçalves da Cunha-Netto, AA Caparica, Shan-Ho Tsai, Ronald Dickman, and David Paul Landau. Improving wang-landau sampling with adaptive windows. Physical Review E, 78(5): 055701, 2008
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 109/CV/R.2009.5500 55266448.
Wei Deng, Guang Lin, and Faming Liang. A contour stochastic gradient langevin dynamics algo- rithm for simulations of multi-mouu modal distributions. In Advances in Neural Information Processing Systems, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x scale. ICIR, 2021.
Xiang Gao, Visvanathan Ramesh, and Terry Boult. Statistical characterization of morphological operator sequences. In European Conference on Computer Vision, pp. 590-605. Springer, 2002.
Will Grathwohl, Kevin Swersky, Milad Hashemi, David Duvenaud, and Chris Maddison. Oops i took a gradient: Scalable sampling for discrete distributions. In International Conference on Machine Learning, pp. 3831-3841. PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Michael Greiffenhagen, Dorin Comaniciu, Heinrich Niemann, and Visvanathan Ramesh. Design, analysis, and engineering of video monitoring systems: An approach and a case study. Proceed- ings of the IEEE, 89(10): 1498-117,,, 2001.
AM Hammitt and EB Bartlett. Determining functional relationships from trained neural networks. Mathematical and computer modelling, 22(3):83-103, 1995.
Robert M Haralick. Performance characterization in computer vision. In BMVC92, pp. 1-8.
Springer, 1992.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint 1rXiv:1512.03385, 2015.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv: 01161.0..16,,... 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier
exposure. In International Conference on Learning Representations, 2019. URL https:// openreview net forum? [d==Hxcxhhchxc
Mitch Hill, Erik Nijkamp, and Song-Chun image spaces. Quarterly of Applied Mathematics, 77(2):269 321, 2019.
Chun Zhu. Building telescope to look into high-dimensional
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN: Detecting out- of-distribution image without learning from out-of-distribttion data. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Christoph Junghans, Danny Perez, and Thomas Vogel. Molecular dynamics in the multicanonical ensemble: Equivalence of wang-landau sampling, statistical temperature molecular dynamics, and metadynamics. Journal of chemical theory and computation, 10(5):1843-188) 2014.
Tapas Kanungo and Robert M Haralick. Character recognition using mathematical morphology. In Proc. of the Fourth USPS Conference Advanced Technology, pp. 973-986, 1990.
Reinhard Klette, H Siegfried Stiehl, Max A Viergever, and Koen L Vincken. Performance charac- terization in computer vision. Springer, 2000.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples the physical world, 2016.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidencce-calibratee classifiers for detecting out-of-distribution samples. arXiv preprint arXiv:1711.09325 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, pp. 71.........7 7167-7177, 2018.
Sang Hoon Lee and Wei Chen. A comparative study uncertainty propagation methods for black- box-type problems. Structural and multidisciplinary optimization, 37(3):239-255 2009.
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Ying Wai Li and Markus Eisenbach. A histogram- free multicanonical monte carlo algorithm for the basis expansion of density of states. In Proceedings of the Platform for Advanced Scientific Computing Conference, pp. 1-7, 2017.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of of distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
11

---

Under review as a conference paper at ICLR 2023
Qiao Liu, Jiaze Xu, Rui Jiang, and Wing Hung Wong. Density estimation using deep generative neural networks. Proceedings of the NationalAcadems Academy of Sciences, ( 118(15)c2101344411 2021.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy Energy-based out- -ittr--dittribution detec- tion. Advances in Neural Information Processing Systems, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, arXiv preprint 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learnin. IEEE transactions on pattern analysis and machine intelligence, 1979)99979 2018.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):5216-522) April 2020. ISSN 2159-5399. doi: 10.1609/aaai, v34i04.5966
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 427-436, 2015.
Ramesh and RM Haralick. A methodology for automatic selection of iu algorithm tuning param- eters. In ARPA Image Understanding Workshop, 1994.
Visvanathan Ramesh and Robert M Haralick. Random perturbation models and performance char- acterization in computer vision. In Proceedings 1992 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 521-522. IEEE Computer Society, 1992.
Visvanathan Ramesh, RM Haralick, AS Bedekar, X Liu, DC Nadadur, KB Thornoon, and X Zhang. Computer vision performance characterization. RADIUS: Image Understanding for Imagery In- relligence, pp. 1-22 1997.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14680-14691, 2019.
Daniel Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. arXiv 2021.
preprint 100.
Andras Rozsa, Ethan M Rudd, and Terrance E Boult. Adversarial diversity and hard positive gen- eration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 25 32, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXhv:2106.10270, 2021.
Yiyou Sun and Yixuan Li. Open-world contrastive learning. arXiv preprint arXiv:2208.027644 2022.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communications Pure and Applied Mathematics, 66(2):145-164 2013.
12

---

Under review as a conference paper at ICLR 2023
Neil A Thacker, Adrian F Clark, John L Barron, Ross Beveridge, Patrick Courtney, William R Crum, Visvanathan Ramesh, and Christine Clark. Performance characterization computer vi- sion: A guide to best practices. Computer vision and image understanding, 109(3):305-334, 2008.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
Thomas Vogel, Ying Wai Li, Thomas Wüst, and David P Landau. Generic, hierarchical framework for massively parallel wang-landau sampling. Physical review letters. 110(21):21060)) 2013.
Thomas Vogel, Ying Wai Li, and David P. Landau. A practical guide replica-exchange wang-landau simulations. 1012, 4 2018. ISSN 1742-6588. doi: 10.10888/144-65566000// 1/012003. BtRt::tps://wwe osti hgt.//bb1111/1117799308
Fugao Wang and David Landau. Efficient, multiple-range random walk algorithm to calculate the density of states. Physical review letters, 86(10):2050 2001.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pp. 681-688,
2011.
Kai Xiao, Logan Engstrom, Andrew Iyyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006. 099944 2020.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 2730-2739, 2019.
Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynam- ics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Chenggang Zhou, T. C. Schulthess, Stefan Torbrügge, and D. P. Landau. Wang-landau algorithm for continuous models and joint density of states. Phys. Rew Lett., 06:120 Mar 2006. doi: 10. 9.t PhysRevLett. 120201
URL https:
aps wttp://///oo/oooooo.
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvomek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. ICLR, 2022.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX A: REPRESENTATIVE INPUTS
Here we list more representative samples of the CNN-MNIST-0/1 scenario. bounded by a black box of boundaries.
The samples are
-50.6
-43.2
-31.5
-18.3
-13.8
-0.5
10.3
-54.2
-43.9
-28.9
-18.5
-13.9
0.6
8.8
-46.7 7
-44.2
-32.8
-17.4
-13.9
-3.5
11.2
-49.3
-39.0
-26.0
-0022
-9.8
0.7
7.9
-49.1
-39.6
-33.4
-23.4
-11.0
0.8
6.3
-47.9
-35.7
-32.2
-18.6
-7.6
-0.7
9.3
-47.2
-43.1
-33.7
-16.2
-14.9
3.4
10,4
-45.1
-38.1
-28.8
-23.4
-12.2
3.9
14.8
-53.1
-36.7
-33.5
-22.5
-5.7
-1.9
8.8
-48.7 7
-43.8
-3..7
-22.2
-.0
0.2
5.8
Figure 6: More representative samples of the CNN-MNIST-0/ model obtained by GWL different logit values, grouped by logit values. We further group these logit values per 100 bins (100 bins correspond difference of 10 in logit value) in resulting in about 10 groups. The output values in the first column are within the range [-55,-45) and the second is from [-45,-35], etc.
14

---

Under review as a conference paper at ICLR 2023
B APPENDIX B: GRADIENT WANG-LANDAU ALGORITHM
Here we provide the algorithms the GWL algorithm. The input and output are listed. The hyper- parameters are determined mostly by the toy-example.
Algorithm Our proposed Gradient Wang-I LangaLLnndau (GWI.)
Require:
pretrained model
+ flat histogram H 0, entropy histogram $ O,
increment/step-size Inf, number of iterations T, test set Die, GWG sampler GWG(zz S), interpo- lation function ((, for t = 1,2,.... T' do
Die while H is not flat do y(xx Sin g(,
Get the continuous interpolation entropy Sin at output Take the negation of Sü,. for Wang-Landau exploration D. Round z to the nearest z' that corresponds to one of the bins
GWG( Sin)
= round(2)
S[ž] S[
H[ž] + H[ē]+1
end while
Inf < Inf/2
H 4
Reset the bins in counter to
end for return S(y)
C HYPER-PARAMETERS AND IMPLEMENTATION DETAILS FOR GWL AND WL
The hyper-parameters for GWL and WL are extremely similar, if not identical, the only major difference between GWL and WL is the gradient proposal versus the random proposal. We first preset large enough range of output values for the sampler explore the trained neural network models. In our experiments, we found that the output (logit) values of the binary classifiers typically fall in the range of -300 to 100 (based on ResNet). Therefore, we use this range for experiments. For flatness histogram H. the bin window size is set to be 1, resulting in 400 bins. The histogram H considered flat the difference between maximum bin value and minimum bin value is smaller than the average bin value. For output histogram S, set the bin window size to be 0. resulting in 4000 bins. Instead of updating one bin at a time for S, we update the neighbor bins with exponential decay. We use the linear interpolation to approximate the bins for continuous queries. We iterate times with test set initialization. Every step the GWG tries to at most update 10 pixels.
D SAMPLES SIMILARITY
The samples in the most dominant peak may be closer to class O than to class We compute the 1.2 pixel-wise distance from the uniform noise image the samples of class and respectively. The mean L2 distance from uniform noise to 0 is around 0.3121 and that from uniform noise to is around 0.3236. The distance between and 0 samples 0.1652. This result shows the samples the most dominant peak are closer to class 0 samples than class samples. More rigorous experiments definite conclusion is yet required as future work.
15


Reviews:
[{"review_id": "vvEhyvDDCJ", "reviewer": "Reviewer_JRJF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The essence of the paper is that one should examine the behavior of a neural network by studying how the output logits change as a function of the entire input space rather than only examine the behavior based on the sampled data distribution. The paper then illustrates how one could utilize intuition regarding a correspondence between state density representations in physics to histograms of logits to design a sampling scheme that combines wang-landau algorithm with gradient with gibbs sampler to explore the vast space of output possibilities.  The algorithm is applied to a toy data set with reduced dimensionality (of 25 dimensions) constructed out of MNIST to illustrate the utility and inner working of the concept. This is then followed by tests on MNIST and illustration of certain insights (e.g. how negative logits correspond to human unrecognizable images and about the nature of what information may be used by the neural net).  Open issues in the sampler are discussed and it is articulated that this form of extensive exploration of the energy density is important for understanding neural net behavior in the entire input space of possibilities.", "summary_of_the_review": "I like the overall idea of the paper and believe that complete characterization of the input to output behavior, over the entire input space, for a trained net will be invaluable for providing guarantees of performance.  However, in my view, the paper in its present form is still not ready for publication.\n\nAfter the revisions presented by the authors I have increased my original rating and have presented my viewpoints to the reviewers and area-chair.  The paper has been strengthened and is addressing an important aspect of performance analysis.  After my discussion with the other reviewer and area-chair my revised rating does not change.", "strength_and_weaknesses": "Strengths:\nOffers the important point that one should explore the entire input space and the input's correspondence to output logits. Sampling methods used in physics literature are offered as a possible way to do the exploration efficiently instead of a brute force approach.\n\nWeaknesses:\nThe emphasis that one has to explore the relationship between the entire input space and output space is well known (see for example: performance characterization literature in the 90's,  http://haralick.org papers on performance characterization. There were explorations involving mapping of boolean random series to outputs for nonlinear operators (e.g. mathematical morphology).  Another example of such characterization in pattern classification settings is illustrated in Gao et al, Statistical Characterization of Morphological Operator Sequences in ECCV 2002).  However, the results in the present paper are not yet convincing to me - I find the visualizations in figures 2, 3, 5, 6 hard to judge. While there is illustration in the toy example that the algorithm seems to be performing correctly via quantitative overlay of the enumeration (groundtruth) against the paper's method, the practical utility of the methodology to gain insights is still not clear. Moreover, there are open challenges with respect to getting the sampling scheme to be efficient as rightly articulated by the authors.\nApart from the fact that the exploration of the input space is shown in the paper, the interpretation of the results is vague. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mcPng8S-kv6", "reviewer": "Reviewer_ucC6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of efficiently understanding the relationship between the input-output mapping of a neural network over a large and representative set of input spaces. To do so, the authors draw an analogy between a neural network model and physics's density of states problem. In doing so, they can design a new sampling strategy inspired by the Wang-Landau algorithm. They augment the algorithm with a gradient-based approach that can sample inputs with maeaningful structure. They conduct experiments on ResNet and CNN to show the approach's effectiveness. \n", "summary_of_the_review": "While the paper is well-motivated and the algorithm proposed is relatively novel, the experiments are limited in data set and models studied. Additionally, the scope of the paper -- binary classification -- is not very broad. ", "strength_and_weaknesses": "-= S1 =- The analogy that the paper draws between neural networks' output histogram and the density of states problem in physics is well-motivated. This open opportunities to apply established methods from physics to the domain of  neural networks. \n\n-= S2 =- The authors make the observation that the randomized sampling method of WL is not as applicable in the context of neural networks and modify that by adding a gradient-based approach that includes more structured inputs. \n\n-= W1 =- The paper considers a very limited set of neural network models that are not state-of-the-art. The paper can benefit from using a recent architecture to showcase the utility of the approach. The same applies to datasets. It is unclear why binary classification on MNIST is an interesting case to apply this technique to. \n\n-= W2 =- It is not sufficiently clear how the new insights uncovered by applying this technique augment our understanding of neural networks. The paper can benefit from spelling that out more explicitly. \n\n-= W3 =- It is also unclear how this approach can be extended to the case of multiple labels. The paper can benefit from a discussion on that aspect as >2 label classification is more prevalent than a binary classifier. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ggCOwlcRGH", "reviewer": "Reviewer_KRCc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to explore the daunting task of \"modeling the entire space of input\" to the output of a binary neural network. This is obviously impossible for any reasonably sized feature space. So, the paper proposes to do this by modeling the output of a neural network to the \"density of states\" concept in physics, where an efficient way to estimate it via the \"Wang-Landau Algorithm\", which seems to be a metropolis-hasting MCMC method. Wang-Landau algorithm can still yield many unrelated samples, so the paper proposes another recent work called Gibbs-With-Gradients (GWG) to be used as the sampler. GWG can leverage the model's learned weights to propose distributions that are closer to seen examples. Doing this exercise yields some interesting observations, the biggest of which is the fact that an enormous amount of possible inputs map to a very high-confidence prediction regime for a neural network.  ", "summary_of_the_review": "I am currently moderately positive about the paper and the very interesting \"main\" finding of the paper. However, I refrain from a higher score because the discussion leaves a lot to be desired about whether this is a special case or can be used as a general-purpose analysis of any binary (or multi-class) classifiers. ", "strength_and_weaknesses": "**Strengths**\n\n*[S1]:* Very interesting results: As mentioned in the summary above, the main result of this paper is quite interesting. The result can be potentially used to better understand adversarial attacks, out-of-distribution generalization, and learning mechanisms of neural networks in general. While the paper does not fully address the above possibilities, I remain hopeful :) \n\n*[S2]:* Clearly written: The paper is very easy to read, despite this reviewer's unfamiliarity with either the \"density of states\" or the \"Wang-Landau Algorithm\"; After brief Wikipedia-based learning of these concepts, the paper made the parallels clear and I felt like I could map these concepts to things already known in ML literature and statistical properties. (Though I might have misunderstood something, hence noted confidence of 3 below).  Furthermore, the experiments are clearly written and well-explained.\n\n**Weaknesses**\n\n*[W1]:* Some obvious questions were left unanswered: For me, the paper was a little baffling due to how little analysis beyond the \"main result\" this paper goes into. Some questions that I would like to have seen discussed or at least mentioned (even if it is in \"future works\"):\n\nW1.1. Why do most of the \"human unrecognizable images\" map to very negative logit values, instead of vice-versa; The positive-negative logit is normally considered arbitrary in binary classification (Two classes can be \"assigned\" to either side of them). Is there a reason for this asymmetry? What are its implications? Is one of the classes harder to create an adversarial attack and/or higher generalization than the other? \n\nW.1.2 What are the implications of this finding in the context of decision-making, explainability, and \"model evaluation? The abstract mentions that it will be explored in \"future works\" but in what manner? I am certainly not expecting this paper to do any of these \"works\" but I am quite unclear about how the findings from the paper can go beyond an interesting result and more towards how, in the words of the paper, \"opens a new gate for neural model evaluation\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gXnMR7LhUc", "reviewer": "Reviewer_gv1q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to analyze a binary neural classifier's input-output relationship.\nA straightforward, brute-force sampling is forbidden due to the high dimension and continuity of input distribution.\nTo achieve efficiency, the authors reformulate the problem as density of states (DOS) sampling in physics, and propose to apply a gradient-based proposal which facilitates the exploration of under-explored output values.\nThe main contributions are:\n\n- making the output histogram problem tractable.\n- analyzing the input-output relationship in the entire input space and revealing some interesting properties by sampled statistics.\n- introducing a new perspective to understand the behavior of neural network classifiers.\n", "summary_of_the_review": "In general I think it is novel and interesting to use a physical perspective to understand the mapping of neural networks. However, there is very little discussion or comparison of related work throughout the paper (e.g., other non-physically inspired sampling algorithms, or other work on the analysis of neural network mappings; see the Weakness section above). This results in this work looking less than complete, and weaken their conclusions. I am inclined to marginally reject it.\n\n\n========== post rebuttal ==========\n\n> We update section 5.3 (and in appendix D) to present some of our interpretation of the results, such as why the black background appears when the logit gets larger.\n> In the related work section, we compare the methods with MCMC samplers and position our method in terms of performance characterization literature.\n\nThank you for clarifying this and adding Section 5.3 as well as the supplementary material for more explanation. I feel that the two most significant flaws (inadequate discussion of the related work and too few explanations of the Fig. 6) have been addressed, thus improving my rating.\n", "strength_and_weaknesses": "**[Strength]**\n\n- The idea of trying to understand neural networks through input-output mapping relationships is groundbreaking; it also makes sense to draw on solutions from a related field (condensed matter physics).\n- Various experiments well illustrate that the statistics sampled by the proposed method (GWL) are close to the ground truth (enumeration) and significantly better than directly sampling (from in-dist test samples).\n\n**[Weaknesses]**\n\n- While it is possible to describe the output histogram problem using a domain specific approach (physical language), I still wonder if there is another way (perhaps a more general sampling algorithm) to solve this problem? The authors do not seem to discuss this sufficiently in the Related Work section.\n- Are the properties of neural network input-output mapping observed by the authors in their experiments (e.g., mapping unrecognizable human samples to very negative logit values) consistent or contradictory to some previous literature? Making some comparisons would make the overall analysis a bit more comprehensive.\n- Fig. 6 is interesting, but I don't particularly understand why logit values for larger samples look \"darker\" implying that the *background* is more important than the *foreground* for the classification task? Perhaps the authors could give a more detailed explanation on this?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject