The Analysis of the Expected Change in the Classification Probability of the Predicted Label
Authors: Ruo Yang
Ping Liu
Mustafa Bilgic
Venue: Accepted by TMLR ()

Abstract: We present a formalism for estimating the expected change in the probability distribution of the predicted label of an object, with respect to all small perturbations to the object. We first derive analytically an estimate of the expected probability change as a function of the input noise. We then conduct three empirical studies: in the first study, experimental results on image classification show that the proposed measure can be used to distinguish the not-robust label predictions from those that are robust, even when they are all predicted with high confidence. The second study shows that the proposed robustness measure is almost always higher for the predictions on the corrupted images, compared to the predictions on the original versions of them. The final study shows that the proposed measure is lower for models when they are trained using adversarial training approaches.

Published in Transactions on Machine Learning Research (11/2023)
The Analysis of the Expected Change in the Classification Probability of the Predicted Label
Ruo Yang
ryang23@hawk. itt. edu
Deportment of Computer Science, Illinois Institute of Technology Chicago, IL, USA
Ping Liu
pliuu@linkedän. com
LinkedIn Corporation Mountain View, CA, USA
Mustafa Bilgic Department of Computer Science, Illinois Institute of Technology Chicago, II, USA
mbilgic@hil edu
Reviewed on OpenReview: https: /ttps: opwp://veeeeeee.....mmmm net/ forum? id- guqzuUVPLQ
Abstract
We present a formalism for estimating the expected change in the probability distribution of the predicted label of an object, with respect to all small perturbations to the object. We first derive analytically an estimate of the expected probability change as a function of the input noise. We then conduct three empirical studies: the first study, experimental results on image classification show that the proposed measure can be used to distinguish the not-robust label predictions from those that robust, even when they all predicted with high confidence. The second study shows that the proposed robustness measure is almost always higher for the predictions on the corrupted images, compared to the predictions on the original versions of them. The final study shows that the proposed measure is lower for models when they are trained using adversarial training approaches.
1 Introduction
Deep learning models are used for numerous industrial, governmental, and personal applications that include medical image analysis Liu & Bilgie (2021), machine translation [Sutskever et al. (2014), face recognition Schroff et al. (2015), automated driving Chen al. (2017). loan application evaluations [Petropoulos al, (2019), and recommender systems Cheng et al. 2016 As deep learning models have been deployed for critical applications such as automated driving and medical diagnosis and as it has been found that these models are vulnerable to simple noise and adversarial attacks Szegedy et al. (2014), the analysis of the robustness of these models has been of paramount importance Athalye et al. (2018).
The robustness of the model's prediction probability is crucial for decision making, including how much confidence associate with that prediction, which action take, and what information to gather next. For example, in medical diagnosis, the predicted diagnosis is rarely useful by itself; the classifier's confidence in that prediction well the robustness of that prediction crucial for making treatment plans and ordering additional diagnostic tests.
It is known by the research communily that even high probability predictions of a highly accurate model might be unreliable Goodfellow et al. (2015). For example, model a high-variance model or its probabilities are not calibrated, its predictions can be unstable and its probability outputs cannot be used indicate uncertainty Platt al. (1999). Even for models whose probability distributions are well calibrated, if the object under consideration lies region where the probability distribution changes drastically,

---

Published in Transactions on Machine Learning Research (11/2023)
the probability prediction might be unstable. Object 22 in Figure demonstrates this situation where the probability distribution has high curvature around the input, and the predicted probability distributions of 22's close neighbors are different from the probability value of z2.
In this paper, we study, analytically and empirically, quantifying the predicted probability difference between an object and all hypothetical objects at Ar distance to æ; i.e., x + Az for all small Vär E RR. We derive an estimate of this measure through Taylor expansion and the Divergence theorem. We then conduct empirical analyses four datascts. Note that one standard strategy to estimate this measure is to sample Az. While has only two neighbors in ID, the number of Ar is infinite in 2D; hence, provide analytical derivation, rather than reverting to sampling. Our main contributions include:
We prove that the expected change in the probability distribution of the object, with respect to all small perturbations around the object, is proportional to the Laplace operator (TO), which is defined as the sum of the second partial derivative of the classifier's output with respect to each input dimension
We conduct three empirical studies four datasets, analyzing the derived estimate (LO) and its relationship to the robustness of the predicted label to random noise, the robustness of the prediction to several lypes of image corruption (blur, brightness, etc.), and comparing the robustness of models when they are trained using traditional training methods versus adversarial training approaches.
fxx)
(122)
f(
Ar
f0r + Ar)
A
Ar
15.
Figure 1: Illustration of the expected probability difference between an object x and its close neighbors x + Ar, i.e, EAz (fxx+ Ar) - f(x)) = Es=(A (A)())). In this simple 1 feature classifier, the expected value computed with its left and right neighbours. In this example, (Az())())) < 0, (Af(2)()2) & O, Es (Af(2))) ss 0, and Esd (Af(r)) > 0. We provide analytical and empirical analyses of EA. (Af(z))) for all small Yår E R...
The of this paper organized as follows. We discuss related work in Section 2 We formulate the problem and derive its analytical estimate in Section 3 We present empirical analyses in Section We then
discuss the limitations and future directions in Section 5 and conclude in Section R
The provided later in Section
2

---

Published in Transactions on Machine Learning Research (11/2023)
2
Related Work
Studies show that the stale-offthe-art deep neural network models are vulnerable to adversarial examples that. are created with small perturbations to the original examples Szegedy et al. (2014). For example, (oooffeloo et al. (0155) showed that two images that are indistinguishable to humans could mislead high-accuracy neural networks. Several papers focused on constructing adversarial examples Kurakin et al. (2017); Xiao et al. (2018), measuring the overall robustness of the models to attacks Peck et al. (2017), and training the models to be more robust to attacks Madry et al. (2018): Jakubovitz & Giryes (2018). While the robustness of the models to adversarial attacks is a closely related area, our sctting is not the adversarial sctting. The objective of this paper is to evaluate the robustness of the predicted probability against random noise instead of adversarial attacks. We focus on quantifying the expected change to the probability distributions under small and random perturbations to the data.
Antheer related area of work is the calibration and quantification of the uncertainty of the predictions of neural networks. Even though the outputs of the neurons passed through sigmoid (or softmax) function are often treated as a probability distribution, these distributions can often be at the extreme ends of [o, I]. displaying an "over-confiience" that is not necessarily warranted by the data. Hence, several papers studied calibrating the probability distributions of these models Guo et al. (2017); Corbière et al. (2019); [Jiang et al. (2018); Lakshminarayanan et al. (2017). Altornatively, the Baycsian modeling and inference approaches for neural networks Denker & LeCun (1990); MacKay
(1992) compute a ribution over the predicted probability distribution and hence provide a mechanism to measure the uncertainty of the prediction probability itself. Finally, several approaches used alternative mechanisms for quantifying the uncertainty of the neural networks. For example, while the drop-out Srivastava et al. (2014) is often used to prevent overfitting, it can also be used for measuring the uncertainty of the model Gal & Ghahramani (2016).
A closely related work to ours is the work by Jiang et al. (2018). They estimate the uncortainty of the predicted label of an object as the ratio between the distance to the closest neighbor that has the same predicted label and the distance to the closest neighbor that has different predicted label. We estimate the expected change in the probability distribution of the object's label with respect to all possible small perturbations to the object, whereas [Jiang et al. (2018) estimate the uncertainty of an object's 's label via the nearest two objects in its neighborhood.
Our focus in this paper is orthogonal to the approaches that quantify and calibrate the uncertainty of neural networks. Regardless of whether the probability distribution calibrated or not, and regardless whether the method is Bayesian or based on drop-out, we quantify how much the predicted probability distribution is expected to change with respect to small perturbations to an object. Even though this is similar to the Bayesian approach where a confidence around the probability prediction can be computed, the Bayesian approach is with respect to the posterior distribution, which is combination of the prior distribution and the observed data. Our approach, on the other hand, mcasures the robustness of the probability distribution with respect to small perturbations to an object, measuring robustness to noise, and identifying probability regions where the distribution is expected to stay the same or change drastically.
3 Our Approach
Let x E RR" be point of interest and F(x): 18= + |0, 1] be a differentiable probabilistic classifier where the number of classes is m. Let Fo(x) be the probability that æ belongs to class c, 1 c m. We then denote f(x) = maxx Fe(x)) as the probability of the predicted class. Let S be a n-dimensional sphere centered at 0 isesm
with radius 2 O. Consider random perturbation vector Ar that is uniformly distributed the sphere S, i.c., Ar ~ Unif(S). Then x + Ar represents displacement around x with a random noise of length r. Further assume that V is an r--imensional ball centered at 0 with radius r, such that S is the surface of the ball V, Le. =
Our goal is to quantitatively mcasure the vulncrability of the classifier's prodicted probability on the point x for a class e against random input perturbations. In other words, we are interested in computing the expected change in the probability of the predicted class e with respect to all possible random perturbations + Ar:
3

---

Published in Transactions on Machine Learning Research (11/2023)
(ar(AS(z)) = (a + Ax) - f(x)).
One way to approach this problem is to use Monte Carlo sampling for perturbation Az, then take an expectation the probability change with respect to the sampled perturbations. While sampling is a viable approach, comes with own challenges, such as the need for generating large number of samples to create reliable estimates, and more importantly, requires the noise level as hyper-parameter. We tackle the problem of determining this quantity via analytical methods, without resorting to sampling, and solve it as function of the noise level. We first present the main theorem of our paper and then describe the general sketch of the proof. We provide the detailed steps of the proofs in the supplementary material.
Theorem 1. Denote 2 as the Laplace operator, and LO as the Laplacian of given function g 9: R n ('((x) at point æ. That is, LO V2g(z) = Σ
1 0u," Then, the expected change the probobility distribution of the predicted class of v, with respect to all random perturbations Ar is:
22 v2 f(x) + +0(Ax))) ~ (Af( = 2n
x2 2n
LO.
(2)
As defined earlier, all possible perturbations Ar with L2 norm r uniformly live on the surface of S. This indicates that the probability density function of the perturbation, p(Ar), is a constant with respect to perturbations. Furthermore, the expected change the probability of the predicted class c the integration of the probability variation caused by the perturbation Ar in expectation its corresponding probability density function. Formally:
Lemma 2. Denote r'()) as the Gamma Function, then the expected change in the value of f around input x is:
I( I-n / As(s)ds
Ea (Af(z))
(3)
2T
We prove Lemma 2 in the supplementary material. We next use the sccond-order Taylor expansion for estimating f(r + Ar):
0f(z)
a ff(()) Ar, 2
ffz)) AN,A,, + O((Ar(?). (4)
[1x (x+ f(x) +
r;
2 Σ dx,²
+
Or
0r,0x,
Then, the integral of Af(x) over all possible random perturbation Ar can be estimated as:
J AS(F)dS
0f(r) Or: 2
Σ am1
f2
82 f(x) Ar, Ar, dz,On,
Ar.
+
Axe
(5)
dr
The Divergence thcorem states that the volume integral of a vector field over close region inside of surface equals the surface integral of the same vector field over the close surface. For the ball V, the unit normal vector directed outward from V is ने (A Ara
Aza
Lemma 3. We have Sg Ar, = 0, Ss Az, Ax, 2ds
11 and Ig Ar, 0, for & i,j < n with r +
* j.
In Equation
5 the 0f(x) fxx)
0z: 02,02,² and f())
ōz,² are independent constants from the variable Az. By Lemma
have:
4

---

Published in Transactions on Machine Learning Research (11/2023)
0f(x) dri
0, and / 2
12
8 ((x) dr,Od, A Ar, AaydS 0.
Ar,dS
i
Using the lemmas above, we derive that the expected change in the probability of the predicted class with respect to all possible random perturbations with length r has the following approximate value:
1
8-1
T
(A)())
m+1)
2T 2
[ + 1) dr,"
22
T
0
,2
Σ
v f(f())
(6)
4 [ + +1)
dz, 2 2n
concluding the proof of Theorem
Given an n dimensional input x and a fixed radius r, Theorem | shows that the expected change in the probability of the predicted class e at x with respect to all possible permutations is linearly proportional to LO. If the LO is a large negative (positive), we expect that the predicted probability at decreases (incrcases) rapidly due to random perturbations. On the other hand, if [LO] is small and close to zero, Theorem suggests that the predicted probability is robust to noise at the input.
3.1 L0 in Practice
There are three lasks where the quantification of the expected change in the predicted probability distribution can be used in practice. We first discuss the robustness of the predicted label a classifier, then the robustness of the predicted probability distribution, and finally the robustness of the model.
3.1.1 The Robustness of the Predicted Label
In the first task, we desire to know if the predicted label by the classifier is robust under noise. One way measure the robustness of the predicted label to noise is to sample white noise, apply it to the test object, and analyze number of times the predicted label changes.
Another approach for measuring the robustness of the predicted label to noise is to calculate the uncertainty of the prediction. If the object is close to the decision boundary, then its label is not expected to be robust to noise. The distance to the decision boundary can be calculated in several ways. For example, for support vector machines, the distance lo the margin can be used. For probabilistic classifiers, the probability output. of the classifier can be used. In the binary classification case, if the prediction probability close 0.5, then the object is treated as uncertain. In the multi-class case, one can use 1 Pe where Pa is the probability of the predicted class, or the margin Pe - Ps. where Pe is the probability of the predicted class and p.. is the probability of the next likely class, or the entropy of the distribution - Σ log
We hypothesize that the value of LO can be used to identify objects whose label prediction is not uncertain and yet the prediction is still not robust to noise. Let us analyze the binary classification and multiclass classification cases separately. Let LO(i) be the LO value for class In the binary case, let the classes be A, B. In this case, we have (LA) (B). Without loss of generality, let the predicted class be A and let its probability be PA- If (A) < O, then the probability of class A is expected to drop in expectation with respect to perturbations and hence the predicted label is not expected to be robust. On the other hand, if LO(A)) ~ 0 or better yet if LO) > 0, the prediction of label A is expected to be robust to noise. In the multiclass case, assume there are m classes, let the predicted class be &; and its probability be pot- If (O(c)) s 0 or better yet LO(c) > O, the prediction of label e is expected to be robust to noise. If LD(c)) < 0, however, then the prodiction is not expected be robust. Further, the summation of LO cross all classes 0, c.g. Σ,, LO(c)) o. When LO(c)) & O, observing LO(c,) >> for some * j adds further evidence to the unstability of the predicted label G noise.
5

---

Published in Transactions on Machine Learning Research (11/2023)
3.1.2 The Robustness of the Probability Distribution
In this task, rather than the predicted label, we are interested in the robustness of the probability distribution itself to small perturbations. Theorem
| states that the expected probability change approximately equal to r' LO.. We emphasize that r LO approximates the expectation of the change, which is the mean change, and it 2n
2n
is not an estimate the variance. Hence, LO means the mean change of the probability distribution with respect to the perturbations to an object is close to 0 and hence the probability in the object's neighborhood is approximately equal to the object's probability claim that the predicted probability is not robust to perturbation, whereas when LO 0, we make claims regarding the robustness of the probability distribution. All we can claim when LO R 0 is that the object's 's probability distribution is similar, on average, to the distribution in its immediate neighborhood within r.
average. Therefore, when LO > 0 or LO < O,
Theorem measures Eaz (Ax))), which is useful for identifying objects that have high confidence but unstable predictions. Another measure that would be useful determine the robustness of the probability distribution is (l Using triangle inequality and the theorems from the prior work Sykora 1974 prove in the supplementary material the following bound:
T 1-n
/ [af())
Es. (AAf(()))
2T
r r(3)
0f(x)
r2
8 f(x) dr,0z,
r² 2n
+
v2f(x).
(7)
π +
dx,
nu² i=1 iff
3.1.3 The Robustness of the Model
In this task, we compare different models in terms of their robustness to noise. A robust model's predicted probability distribution is expected to change less compared to a less robust one, under minor changes to the input. In essence, this is also related to the variance of the models. Given two models that are comparably accurate on validation set, the model that is more robust noise is preferred as it expected to generalize better to unscen data.
We illustrate an extreme case of this phenomenon in Figure 2 The more robust. model has smooth curvature in its probability distribution, and hence the noise in inputs causes smaller changes in the predicted probability values when compared to the non-robust model. Because LO is proportional to the expected change in the probability distribution with respect to all perturbations per Theorem [ we hypothesize that the absolute value of LO can be used to compare the robustness of two models. More specifically, we propose computing the expected [LO| for a given model A:
f [())))O,(())
In the experiments section, we estimate this measure using held-out set, T:
Σ [00x(0)
T
PET
Given two models A and B, the model that has the higher value of this measure is expected to be less robust to noise. We present experimental results in Section 4.4 using this mcasure to compare models that are trained "normally" versus "adversarial training methods."
4 Experimental Methodology and Results
We conduct three empirical studies. In the first study, we study the relationship between L0 and the stability of the label prediction on four image classification datascts. Second, we compare the magnitude of [LO] on the original versus corrupted versions of images on two benchmark datasets. Finally, we compare LO on models that are trained using traditional approaches versus adversarial training methods.

---

Published in Transactions on Machine Learning Research (11/2023)
f(x:)
Robust Model Not Robust Model
I
The illustration of the robustness of two models in 1D. 24 and zy are two example test instances. Both robust model (bluc solid line) and not-robust model (red dashed lino) have the same prediction for inst instances&o 2; and zy- However, the curvatures of the robust model are smoother at those two points. In this case, [LO] computed using the robust model lower than the one computed using the not-robust model.
Figure 2:
4.1 Datasets and the Models
We conduct experiments on the MNIST digit classification LeCun et al. (1998), CIFAR-10 10 object classification Krizhevsky et al. (2009), histology images of colorectal cancer (CRC) dataset Sirinukunwattana et al. (2016), and the street view house numbers (SVHN) dataset Netzer et al. (2011). Theorem applies to any differentiable classifier. We focus on neural networks in this study. We train LeNet LeCun et al. (1998) for MNIST and ResNet He et al. (2016) for the other three datasets. The details of training, validation, and test splits, the model structure, activation functions, and the model hyper-parameters are as follow.
We split the MNIST data as 60K for training and 10K for testing. We adopt the LeNet architecture, use the hyperbolic tangent function for all convolution layers, and linear activations for the dense layers. We use all samples in the training set to train with 20 epochs, batch size of 500, and Adam Kingma. & Ba (2015) optimizer with a learning rate of 0.01. The accuracy of this model on the test set is 98%.
The CIFAR-10 dataset consists of 45K training, 5K validation, and 10K testing images A Abadi et al. 2015 We create a ResNet model with 3 Res-blocks. We initialize all weights following He et al. (2015a) and utilize the PRelu He et al. (2015b) activation function. We augment the training by flipping the images horizontally and shifting both height and width with maximum 12.5% range. We use a batch size of 128, regularization constant of 0.0001, learning rate of 0.001, and SGD with the momentum of 0.9. We optimize the epoch number using the validation accuracy. The final model achieves top-1 test accuracy of 85%. We adapt the same model structure of CIFAR-10 to the SVHN dataset. We keep the original test set (26,032 objects) of SVHN for testing, and reserve random 5K objects from training set as the validation set. The final model has a test accuracy of 92%.
We also use the histology images colorectal cancer dataset (CRC), which contains 100 H&E stained colorectal adenocarcinomas images where each image contains several cells. The cells in the stained images are labeled Fipttelial, Inflammatory, Fibroblast, Miscellaneous, and the location of the center of each labeled cell
7

---

Published in Transactions on Machine Learning Research (11/2023)
is provided in the data. We extract 27x27x3 image for each cell at the locations provided in the data. The total samples for each class are 7,057, 6,278, 5, 5,130, and 1,842 respectively. We split the dataset. into 70% train, 15% validation, and 15% test. We used the same ResNet architecture of SVHN, but changed the input and output size as needed. The model achieves a top-1 test accuracy of 74%.
We implemented both LeNet and ResNet models using Tensorflow Abadi al. (015). compute L.O for each image, we first calculate the Hessian matrix, H, of the maximum predicted probability of the image by using the automatic differentiation function provided by Tensorflow. LO is then simply the trace of the H. We used single modern GPU (Quadro RTX 5000) for the experiments.
4.2 1. and Label Flip
(a) Predicted label:
(b) Predicted label: Bird
Prediction prob: 0.803 True label: Cat
Prediction prob: 0.814
True label: Deer
Pro))) -0.230
((AProb) -0.248 LO 1523
LO 1413
Figure Two example images from the CIFAR-10 dataset. Even though the predictions by the ResNet model are confident, our estimate shows that the prediction probabilities are expected to decrease small perturbations are applied these images. Sampling and applying noise vector of length 1, which is equivalent to average 0.037 change per pixel, shows that the cat prediction flips to different label in 57% of the samples (43% 'cat', 29% 'dog', and 28% 'frog') and the bird prediction flips to different label in 64% of the samples (36% 'bird', 63% 'deer', and 1% 'frog).
In this section, we study if LO can be used to identify objects whose label predictions are not robust to noise. As discussed in Section 3.1 even if an object's 's label is predicted with high confidence, if LO < for the predicted class, the prediction is likely to not robust to noise.
To test our hypothesis, we take a random sample of 1K objects from the test set for cach datasct, and calculate the probability and LO for their predieted labels. Then, we generate 10K noisy versions of each of these images by sampling noise vector of length (we tested r=0.5, and 2) and appending it to the original image. We then compare the predicted labels for the noisy versions of the original images with the original labels.
the first experiment, we focus on objects where the model confident in its prediction; we specifically focus on objects whose predicted label has 0.8 or higher probability value. This filtering results 903 test objects for MNIST, 891 test objects for CIFAR-10 10, 603 test objects for CRC, and 805 test objects for SVHN. We group these objects into two: top 50 objects where LO is lowest (referred "Bottom") and
8

---

Published in Transactions on Machine Learning Research (11/2023)
top 50 objects where LO is highest (referred "Top "). We compute, for each test object, the ratio of its noisy versions that have a different, predicted label from the original image (also referred as "label flip"). We report the average label flip rates for both the "Bottom" and the "Top group, under varying values of r, in Table The results show that the label flip percentage is almost always small for the "Top LO" group and the label flip percentage is drastically higher for the "Bottom IL group. These results indicate that large negative values of L.O ("Bottom") can identify objects whose label predictions are not stable, oven when they are predicted with high confidence. We present two example images from the CIFAR-10 dataset in Figure Both images were predicted with high confidence but had large negative LO values. Sampling and adding noise vectors to these images and re-predicting their labels confirm that the original predictions are not stable, as the prediction switches to a different label majority of the time.
T
= 0.5
r = 1.0
= 2.0
LO
Top
Bottom
Top
Bottom
Тop
Bottom
MNIST 0.02%
5.1%
2.0%
9.9%
3.4%
19.4%
CIFAR-10
1.8%
2.2%
2.0%
22.1%
2.0%
59.9%
CRC 0.07%
12.3%
0.18%
18.9%
0.05%
27.6%
SVHN 0%
1.6%
0%
6.1%
0%
21.6%
Table 1: Label flip percentages for all datasets. 1K random test objects were selected from each dataset. The lest objects whose label were predicted with at least 0.8 probability were subjected to random noise whose length was For these confident objects, we computed what percentage of those samples changed their predicted labels under noise. The group that had the smallest (i.e, largest negative) LO value had higher label flip percentages.
In our next analysis, we group the test objects into two: those who have at least one noisy version (among 10K samples) whose label is different from the original test object versus those whose all noisy versions' labels agree with the original test object's 's label. We present the distribution of LO values as box plots and the p-values for the unpaired t-tests comparing "flipped at least once" to "never flip" for each in Figure 4 As the results show, LO is significantly smaller (more negative) for the "flipped at least once" group.
4.3
|LO|| and Corrupted Data
In this section, we analyze two benchmark robustness datasels: MNIST-C by Mu Gilmer 2019 and CIFAR-10-C by Hendryyks & Dietterich (2019). The MNIST-C dataset contains MNIST images corrupted by 15 different corruption methods such as guassian blur, scotter, and etc. Each original test image of the MNIST dataset is corrupted separately using one of these 15 corruption methods, which leads to 15 groups of corrupted images. The CTFAR-10-C dataset has 19 different corruption methods and each corruption has 5 levels of corruption that can be added to each image.
We expect the original images to have more stable predictions compared to their corrupted versions. Figure 5a compares the [T.O] on all corrupted sets and the original images for the MNIST-C dataset, and figure 5b on all corrupted image with corruption level 3 and the original images for the CIFAR-10 datasct. (Appendix includes the comparison with other corruption levels). The result for the original images is shown as dotted line because it does not depend the corruption method. The results show that the corrupted images always have a higher LO] than original images from MNIST, and for majority of corrupted images for CIFAR-10.
4.4
[L|| and Model robustness
In this section, we study the connection between the robustness of the model and values of [[LO| as discussed in Section 3.1.3] We compare a "standard" model that trained using traditional learning methods with models that are trained using adversarial training. To be clear, our primary objective is not to conduct a comprehensive evaluation of adversarial training methods to enhance the robustness of the model. However, utilize the adversarial training process along with [T.0] to evaluate model robustness from a random
9

---

Published in Transactions on Machine Learning Research (11/2023)
Flipped least
Never
MNIST
CIFAR10
50
P.s 1e-5
P.s 1e-5
P.s 1e-5
100
p.5 Te:5
p.s Te:5
p.s 1e-5
25
0
-100
-25
-200
9
-50
-300
-75
-400
-125
-150
-600
1.0
20
100
p.s 1e-5
P.S 1e-5
p.S 1e-5
0.000
5.00
S.s 1e.4
-100
-10
-200
9
-20
-300
-400
-30
-40
-600
Figure 4: Average LO within two groups. Flipped at least once contains the test objects who has at least one noisy version whose label has changed. The Never flip contains the objects whose noisy versions all agree with the test object's label. Noisy versions are created by adding random noise with length r to original object.
noise perspective. For adversarial training of the model, we adopt the strategy from Tramèr et al. (2018) Goodfellow et al. (2015). Table summarizes the accuracies and the L2 norm of the weights of these models. As the table shows, these models have relatively comparable accuracies, except for CIFAR-10 dataset where the accuracy drops 10% for adversarial training. The adversarial model for MNIST has a significantly lower Tu2 norm (suggesting simpler model), whereas the L2 norms are comparable for the other datascts (suggesting similar model complexity).
We next compute |LO| for each model for each test data point. We present the boxplots, in Figure 6 of these |O] values and the p-values of the t-test comparing the adversarially-traien model. We observed that the predictions trained by adversarial training have statistically significantly lower LOl values for all datasets. Morever, as expected, [LOO] is smaller if the input dimension smaller (the equation 2), hence the [IO] values for MNIST are smaller, absolute terms, than the other three datasets. This leaves smaller room for objects to differ in [LO] values. However, the reported p-value indicates a statistically significant difference in the [T.O| values for adversarial-trained versus standard-trained models.
model and standard-trained
10

---

Published in Transactions on Machine Learning Research (11/2023)
Corrupted Original
700
Original
600
25
Mean. [LO)
Mean. (LD
500
20
15
300
10
200
gaussien, noise
shot nolse
impulse hoise defocis _uluur
frasted gloss, blur motion Blur
zoom, butu
snow
frost
fog
brightnnes
contrast
elastic
pixelate
gaussian, jolur sabirate
spatter
speckle noise
jeeg compression
identity
shot noise
impulse, nobo
glass Blur
sheal
Scale
rolate
brightness
translate
stripe
Spatter
detted Jine
zigrag
motion,
Corruption
Computtion
(a) MNIST
(b) CIFAR-10
Figure 5: Comparing average [LO] for the original images (dotted line) versus the corrupted groups of images for the MNIST-C and the CIFAR-10-C datasets. [T.O] is always higher for all the corrupted images from MNIST-C, and higher for most of the corrupted images from CIFAR-10-C. Results for other levels of corruption for CIFAR-10-C included in the appendix.
Dataset
MNIST
CIFAR-10
CRC
SVHN
Dimension
28x28x1
32x32x3
27x27x3
32x32x3
Accuracy (Standard Train)
.98
.85
.74
.92
Accuracy (Adversarial Train)
.99
.75
.71
.89
L2 of weights (Standard Train) 1933
1190
1130
1041
L2 of weights (Adversarial Train)
1492
1006
1335
1228
Table 2: Summary of model performance from standard training and adversarial training. Adversarial training
results lower or similar training accuracy as expected compared to the standard training method. L2 norm is smaller for the adversarial model for MNIST (suggesting a simpler model), and comparable for other models (suggesting comparable complexity). Note that the models that correspond to "standard" training and "adversarial" training have the same model architecture.
5 Limitations and Future work
The exact computation of LO requires the second derivatives at These derivatives can be derived analytically for a given neural network structure and can be computed efficiently based on this analytical derivation. However, most packages including Tensorflow, do not support the computation of the second derivate of with respect to input æ. Some instcad provide the computation of the Hessian matrix, which is compulationally expensive, and is overkill for computing LO, as LO needs only the diagonals of the Hessian matrix, and not the full matrix. Thus, using LO for robustness is difficult for many of the state-of-the-art models.
In this paper, we studied the mean change in the predicted probability with respect to perturbations, E[A((r) We showed that for high confidence predictions, a large negative LO can indicate a label flip under small perturbations. Though this quantity can also be used study the stability of the probability distribution itself, it measures only the mean change. promising future direction probability change, with respect to perturbations: [[A E(A/(z)))
derive the variance of the
11

---

Published in Transactions on Machine Learning Research (11/2023)
p.
p.5
MNIST
CIFAR10
0.175
0.150
250
0.125
200
0.100
0.075
100
0.050
0.025
0.000
Standard
Standard
Adversarial
p 1e-5
P.o 1e-5
40
120
30
100
80
LO1
20
10
40
10
20
Standard
Standard
Adversarial
Figure 6: The model robustness and [LO] for each dataset. The p-values show that the models that are trained using adversarial training have ssatistically significantly lower [TO]] values compared to the same architectures that are trained using standard procedures.
6 Conclusions
We investigated the robustness of the probability of the predicted class for a differentiable model. We derived using the Taylor expansion and Divergence theorem that the expected change in the probability of the predicted class, with respect to random perturbations around the input, is multiple of the Laplace operator at the input. We conducted empirical analyses four image classification datasets. The first empirical study showed that we were able to identify objects whose labels were predicted with high confidence but yet were still unstable under random noise. The second empirical study on two robustness benchmark datasets showed that the absolute value of the Laplace operator was higher for corrupted images than the original ones. Lastly, the experiments demonstrated that [LO] can distinguish with statistical significance a standard model from model that is trained using adversarial training, and hence can be used as a measure of overall model robustness.
12

---

Published in Transactions on Machine Learning Research (11/2023)
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Trving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Πlya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pele Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Largo-scale machine learning on heterogencous systems, 2015.
Anish Athalye, Logan Engstrom, Andrew Nyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International Conference on Machine Learning, pp. 284-293, 2018.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
Heng-Twa Tze Cheng, Levent Koc, Jeremiahh Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Charles Corbière, Nicolas THOME, Avner Bar-Hen, Malthieu Cord, and Patrick Pérez. Address- ing failure prediction by learning model confidence, In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL stttoe///proceeeiigg htw.eedddiinnneeriss...ce.ooeeee wwwriips.ccpppper///... [557554aasScd0000000000002222210
pdf
John S Denker and Yann LeCun. Transforming neural-net output levels probability distributions. In Proceedings of the 3rd. International Conference on Neural Information Processing Systems, pp. 853 859, 1990.
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Jake Gipple. The volume of n-balls. Rose-Hulman Undergraduate Mathematics Journal, 15(1):14, 2014.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.
PMLR, 06 11 Aug 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026 10641034,11 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In Proceedings of the European Conference on Compuler Vision (ECCV), pp. 514-529, 2018.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In S. Bengio, Н. Wallach, Н. Larochelle, K. Grauman, N. Resa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (ods.), 3rd International Conference on Learning Representations, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Tan J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Truck Proceedings, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable prodictive uncer- lainty estimation using deep ensembles. In Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bass learning applied to document rocognition. Proccedings of the IEEE, (6(11):2278-233) 2324, 1998.
Ping Liu and Mustafa Bilgic. Relational classification of biological cells in microscopy images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 344-352, 2021.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neurol computation, 4(3): 448-472, 1992.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. ar preprint arXiv:1906. 10X.:37906.02377, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Jonathan Peck, Joris Roels, Bart Goossens, and Yvan Saeys. Lower bounds on the robustness to adversarial perturbations. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 804-813, 2017.
Anastasios Petropoulos, Vasilis Siakoulis, Evaggelos Stavroulakis, Aristotelis Klamargias, et al. A robust machine learning approach for credit risk analysis of large loan level datasets using deep learning and extreme gradient boosting. Are Post-crisis Statistical Initiatives Completed, 49:49-49, 2019.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815 823, 2015.
Korsuk Sirinukunwattane, Shan E Ahmed Raza, Yee- Tsang, David R. J. Snead, Ian A. Cree, and Nasir M. Rajpoot. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Transactions on Medical Imaging, 35(5):1196-1200 2016. doi: (0000//M..000......22..
14

---

Published in Transactions on Machine Learning Research (11/2023)
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Lya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929- 1958, 2014.
Nya Sutskever, Oriol Vinyals, and Quoc V Te. Sequence to sequence learning with neural networks. Tn Advances in neural information processing systems, pp. 3104 3112, 2014.
Stanislav Sykora. Quantum theory and the bayesian inference problems. Journal of Statistical Physics, 11(1): 17 27, 1974.
Christian Szegedy, Wojeiech Zaremba, Πlya Sutskover, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, 2014.
Florian Tramer, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. January 2018, 6th International Conference on Learning Reprosentations, ICLR 2018; Conference date: 30-04-2018 Through 03-05-2018.
Chaowei Xiao, Bo Li, Jun Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. In 27th International Joint Conference on Artificial Intelligence, IJCAI 2018, pp. 3905 3911, 2018.
A Appendix
A.1 Lemmas and Proofs
fa(z)), where (Z,,AT) E R", I() is the Gamma Function, T R 5' =| Tbllz =r. Then the expected Change around point Tb:
Lemma 4. Define Afe,s (56) f(z + +At)
I( 2 p- 2 2
/ ASc.s
E(Change)
(8)
Proof. To prove the Lemma 1, we first define a n dimensional ball with radius r in the space and centered at zg. The S' is the dimensional sphere of the ball such that any point 7, n6 + A7, with the Euclidean distance r the point x is on the sphere S E In other words, with attack vector which has length r, | r, every attack sample point E around un-attacked sample z is the surface S'. Then define a infinitely small region A the surface S' such that Z' E A. The probability density function p(Z) on the surface S is the relative likelihood for point on sphere S' and also in the region of A. Formally, Prob(7 E A) = pA )())((() 2)). Roughly speaking, p( x)) measures the likelihood of a point / to be selected on the surface S and of course for any particular point Z, the measure is O, that is p(ž)) for a single point is 0 since it is probability density function. There is Change respect to each possible attack sample ø. So the expectation of Change is the integral over each Change on its point Z times the likelihood of the point z. Formally, this integral in surface integral over S' So:
E(Afe e (x6)) = E(Se(zo + AT fc(
J g/ A
(
(9)
p()) is the probability density function of a point on a (n - 1) Dimension sphere S', so the integral of p( 7
over all z 1. That is:
/ p p())dss ds =1
(10)
15

---

Published in Transactions on Machine Learning Research (11/2023)
Since sphere S is surface of ball, every point sphere S' has equal likelihood to be selected, then probability density function p( ) is uniform probability distribution, which means p( xt) is a constant for a fixed dimension n then:
pl p(Z)
/ s
ids
(11)
For any given n dimensional ball with radius r, the area of the surface is SS 1dS is the Gamma function, so:
2x """"""= where I(.)
T(2)
p( 2)
p 1-n
(12)
2
and:
[ 1-n fS. Afc.s' (20)ds
Eq. 9
(13)
2
It completes the proof of Lemma 1.
Lemma 5. is surface of the n dimensional ball V centered at O with radius F ,.1,...,,,, kn) S then: JS x;dS = 0, for 1 <is Snn.
Proof. To prove Lemma 2, we first construct a vector filed F() for each E [1, with value is 1 for 7th dimension and 0 for others.
F() (O,.
0)
(14)
Then, the unit normal vector directed outward from V is:
πt T2
(15)
and:
p(1) ने. 3i
(16)
r
Then:
x,
p() .t
(17)
The divergence of the vector filed is 0:
ap(1) 0
V FO =
(18)
1=1 Oz,
Clearly, F(O is a vector field whose component functions have a continuous partial derivatives in V, so by the statement of The Divergence Theorem, we have:
[ F 7ds /1. (V F(O) S/ odV 0
(19)
This completes the proof of Lemma 2
J
Lemma 6. S is surface of the dimensional ball V centered at D' with radius 3 ( S
then: SS 2dS 7 r for 1 & Eisn.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Proof. To prove Lemma 3, the similar idea from above, but construct vector filed F() for each E [1 with value is z; for 8th dimension and 0 for others.
F (O..... 0)
(20)
The divergence of the vector filed is
OF(1)
V. F()
(21)
/=1 dz,
so by the Divergence Theorem:
| x 2S r
F(0) 78S = rrrr 1dV
(22)
Jfy 1dV is the volume of the ball V, and for any n dimensional ball, the volume Gipple 2014 is:
m 2
rm:
r( +1)
π 2
1dV
(23)
T(
So:
14 RdS
R+1
(24)
I + 1)
Lemma 7. S is surface of the n dimensional ball V centered at o' with radius 3 = (21,... ..,,,,,,,,,,,,, E S then: Ig xr = 0, for 1 <is n, <j < and i 4 j.
Proof. By the similar idea, we construct a vector filed F() for each pairs of * j [1,m] with value Bry for 8th dimension, (1 B)x; for jth dimension, and 0 for others, where 3 [0,
Fli (O,.... Brg
(1 3)z, 0)
(25)
3th
Then, the divergence of this vector filed is 0:
OF(1)
p(i)
(26)
dz,
This completes the proof by using the Divergence Theorem again.
A.2
Additional figures
A.3
Model details
The MNIST dataset is split into train and test as follows: 60K images set has 10K images. We adopt the LoNet architecture from LeCun et al. (1998) and train on MNIST with normalized pixel values. Specifically, following the seltings in LeCun et al. (1998), we use the hyperbolic tangent function as activation function for all convolution layers, and then linear activations are used for the dense layers. We utilize all samples in the MNIST training set train with 20 epochs, batch size 500, and Adam Kingma Ba (2055 optimizer with learning rate 0.01. The accuracy of this model on the test set is 98%.
the training set and the test
17

---

Published in Transactions on Machine Learning Research (11/2023)
3
15-
20-
10 15
1.0) 011090001155280310000
N
3
stoce
and
25
26
25
01601
14171224611000
509555
4751158520114
5-
3
3
3
B
3
15
12
5 15
15
15
[LO] 1. /29:3810109/0004
[00 051/11/99195/775
1.0 1Q--9546544188330
1.0
2.00122405557444 246055
3
3
3
10
20
20
35
[10]
[[]]
Figure 7: Clean and Corrupted Images with [LO]
The CIFAR-10 dataset is split into train, validation, and test as follows: 45K images are on the training set, 5K images are on the validation set, and the test set has 10K images Abadi et al. (2015). We create a ResNet He et al. (2016) model for this datasct. We create 3 Res-blocks construct the ResNet model. We initialize all weights following He et al. (201 5a) and utilize the PRelu He et al. (2015b) activation function. We use data augmentation before passing the training data into the model, where randomly horizontally flip, and shift both height and width with maximum 12.5% range. Furthermore, we use batch size of 128, regularization constant of 0.0001, learning rate of 0.001, and SGD with momentum of 0.9, and we optimize the epoch number using the validation accuracy. The final model achieves top-1 test accuracy of 85%.
For SVHN dataset, we adapt the same model structure and training parameters of CIFAR-10 to SVHN dataset Netzer al. (2011). We hold original test dataset (26032 objects) for testing, and randomly reserve 5K objects from training sets validation set. The accuracy test dataset is 92%.
Sirinukunwatlana et al. (2016) introduced the histology images colorectal cancer dataset (CRC), which contains 100 H&&E stained colorectal adenocareinomas images where each image contains many cells. The cells in the stained images are labeled as: Epithelial, Inflammatory, Fibroblast, or Miscellancous, and the location of the center each labeled cell is provided the data. We extract a 27x27x3 image for each cell at the locations provided in the data. The total number samples of each class are 7,057, 6,278, 5,130, and 1,842. We split the dataset into 70% train, 15% validation, and 15% test. We used the same ResNet architecture that used for CIFAR-10 and SVHN, but changed the input and output size as needed. The model achieves top-1 test accuracy of 74%.
18

---

Published in Transactions on Machine Learning Research (11/2023)
brightness
contrast
defocus blur
elastic
fog
Mean. [LO]
240
Moan. [[a]
1000
[LO]
Mean. (LO)
500
Mean. [Lo]
1000
400
220
500
300
300
500
Corruption
Cornuption frosted glass blur
Corruption
Corruption
Corupption
frost
gaussian blur
gaussian_ noise
impulse noise
Mean. ILO
350
500
400-
Mean, I.
300
300
300
Mean.
400
Mean:
Mean,
250
250
300
300
200
4
Corrustion
Corrustion
Corruption Level pixelate
Corruption Level saturate
Corruption Level
jeg compression
motion blur
shot noise
600
Mean, [LO
350
Mean, LO
400
Mean LO
Mean. IO
300
300
300
400
300
250
200
Corrustion
Corrustion spatter
Corruption Level speckle noise
Corruption Level
Corruption
snow
zoom blur
Mean. LO
Mean. (LO)
300
300
Mesn, (LO
100
I
Original
250
250
250
Corrupted
225
2 3
Corruption
Corruption Level
Corruption Level
Corruption
Level
Figure 8: Comparing average |LO|| for the original images (dotted line) versus the corrupted groups of images for the CIFAR-10-C dataset. [LO] almost (83 out of 95) always higher for the corrupted images. [LO] is positively correlated with the corruption level for most corruption methods.
19


Reviews:
[{"review_id": "jmMA5IzGOn", "reviewer": "Reviewer_jsEK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper formalizes a new idea to perform adversarial machine learning.\n\nThis paper is easy to follow, and the main contribution is clearly stated.\n\nWeaknesses:\n\nThis paper presents a novel approach to evaluating how minor perturbations can impact the predicted label probabilities of an object. Initially, the authors present an analytical estimate of anticipated probability variations tied to input noise. Subsequently, they conduct three empirical studies to support their theoretical findings. The first study illustrates the ability of their proposed metric to distinguish robust from non-robust label predictions, even when predictions exhibit high confidence. In the second study, predictions on corrupted images consistently show a higher measure of robustness in comparison to those on original images. Finally, models subjected to adversarial training approaches exhibit lower values of the proposed measure, as demonstrated in the last study. However, the paper's experimental justification is somewhat limited due to the lack of incorporation of advanced adversarial training methods as baselines. This omission could potentially undermine the full validation of the proposed method and its efficiency, thus necessitating more comprehensive experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel framework for assessing the change in predicted label probabilities for an object in response to small perturbations. The authors first develop an analytical estimate for expected probability alterations in relation to input noise. Following this, they substantiate their findings with three empirical studies. The first experiment demonstrates that their proposed metric can differentiate between robust and non-robust label predictions, even when these predictions are made with high confidence. The second study reveals that their measure typically indicates greater robustness for predictions made on corrupted images, compared to those made on their original counterparts. Lastly, the study finds that their proposed robustness measure tends to be lower for models that have undergone adversarial training. In summary, this paper provides an innovative analytical approach, backed by empirical studies, to better understand and measure the impact of small perturbations on label prediction probabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper's experimental justification is somewhat limited due to the lack of incorporation of advanced adversarial training methods as baselines. This omission could potentially undermine the full validation of the proposed method and its efficiency, thus necessitating more comprehensive experiments.", "broader_impact_concerns": "There is no broader impact concern regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oMad8SWuMG", "reviewer": "Reviewer_b491", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n\n- **Well-motivated problem.** Measuring the robustness of model prediction is a practically important direction for its reliable deployment to the real world.\n- **Intuitive approach.** The proposed estimation via the Laplacian operator is intuitive and well-grounded by the provided derivation.\n\n**Weaknesses.**\n\n- **Novelty.** The second derivation (or Hessian) has been widely studied in the literature on model robustness [1,2,3,4]. Although the authors propose new applications with this, it’s hard to find a technical novelty compare to the existing works.\n- **Lack of baseline**. The authors do not compare with the naive baselines. For example, instead of Laplacian Operator, one can use model confidence or entropy for selecting Top and Bottom samples in Table 1. Also, Figure 4 can be drawn by adopting other measurements as y-axis. Hence, the authors should provide such baseline results and validate that the proposed estimation is better than other choices.\n- **Limited experimental demonstration**. The authors only adopt the small models (LeNet or 3-layer ResNet) and benchmarks (MNIST, SVHN, CNC, and CIFAR-10) for the experiments. However, the proposed scheme can be easily applied to the larger scale of benchmarks (e.g., ImageNet, CIFAR100) with state-of-the-art models (e.g., pre-trained ResNet or ViT). If the authors can show that the proposed method also well performs in those scenarios, it will be really interesting and valuable.\n\n[1] Ros and Doshi., Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients., AAAI 2018 \\\\\\\n[2] Yao et al., Hessian-based Analysis of Large Batch Training and Robustness to Adversaries., NeurIPS 2018 \\\\\\\n[3] Singla and Feizi., Second-Order Provable Defenses against Adversarial Attacks., ICML 2020 \\\\\\\n[4] Tsiligkaridis and Roberts., Second Order Optimization for Adversarial Robustness and Interpretability., arXiv \\\\", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new estimation of the expected change in the probability distribution with respect to the small perturbations, offering a potential tool for assessing the robustness of the model prediction. Specifically, the authors analytically derive that the Laplacian operator becomes such an estimation. With this estimation, they demonstrate its effectiveness across three different applications on four different computer vision benchmarks: 1) distinguishing not-robust prediction with high confidence, 2) detecting corrupted images, and 3) measuring the adversarial robustness of neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns above. Specifically, \n\n1. Clarification of the contribution and comparison with previous works\n2. Comparison with baseline methods\n3. Experiments with larger benchmark", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VBIKnrqwjB", "reviewer": "Reviewer_sGRe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- paper presentation is clear\n- problem studied is interesting and well-motivated\n- theoretical contributions are interesting and findings are backed by experimental results\n- good experimental scope across datasets\n\nWeaknesses:\n- In Section 3, the authors list 3 practical applications (robustness of predicted label, robustness of probability distribution, robustness of model) of the Laplace operator.  I think the experimental results provided generally demonstrate the first and third applications, but it doesn't seem like there are experimental results that demonstrate the second applications.  Since the authors mention model calibration and uncertainty quantification in the related works section, I wonder if experiments could be used to demonstrate that more calibrated models have Laplace operator closer to 0 than uncalibrated models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors prove that the expected change in a function's classification probability for random perturbations that lie within a fixed radius around a data point can be estimated using the Laplace operator of the function.  They then conduct experiments on a variety of datasets (MNIST, CIFAR-10, SVHN, CRC) and demonstrate that the Laplace operator is correlated with the rate at which the classifier flips the predicted label for noisy images and model robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the point under Weaknesses, addressing this point would strengthen the work in my view.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an analytical estimate of the expected change in the probability distribution for random perturbations on an $l_2$ ball surface. The idea of using the second derivation (or Hessian's diagonal) is not surprising, but the potential to evaluate the robustness of the classifier is interesting. The reviewers had a number of concerns about novelty, contributions, and experiments, which were mostly resolved through the discussion process. Overall, this is an interesting and useful paper and I recommend its acceptance.

---

- As in the author's comments to Reviewer jsEK, related clarification should be clearly demonstrated in the paper to avoid an overclaim.
- As Reviewer b491 also requested, I highly recommend conducting experiments with a larger benchmark. I understand that computing the Hessian diagonal will be a bottleneck, but recent advances in deep learning frameworks can help accelerate this computation, e.g., a library for PyTorch - https://docs.backpack.pt/en/master/extensions.html#backpack.extensions.BatchDiagHessian.
Decision: Accept (poster)