How Sampling Impacts the Robustness of Stochastic Neural Networks
Authors: Sina Däubener
Asja Fischer
Venue: NeurIPS 2022 Accept (2022)
Keywords: Stochastic neural network
robustness
adversarial attacks

Abstract: Stochastic neural networks (SNNs) are random functions whose predictions are gained by averaging over multiple realizations. 
Consequently, a gradient-based adversarial example is calculated based on one set of samples and its classification on another set. 
In this paper, we derive a sufficient condition for such a stochastic prediction to be robust against a given sample-based attack. 
This allows us to identify the factors that lead to an increased robustness of SNNs and gives theoretical explanations for: 
(i) the well known observation, that increasing the amount of samples drawn for the estimation of adversarial examples increases the attack's strength,
(ii) why increasing the number of samples during an attack can not fully reduce the effect of stochasticity, 
(iii) why the sample size during inference does not influence the robustness, and
(iv) why a higher gradient variance and a shorter expected value of the gradient relates to a higher robustness. 
Our theoretical findings give a unified view on the mechanisms underlying previously proposed approaches for increasing attack strengths or model robustness and are verified by an extensive empirical analysis.

How Sampling Impacts the Robustness of Stochastic Neural Networks
Sina Däubener and Asja Fischer
Department of Computer Science
Ruhr University Bochum, Germany
{sina. daeubener, asja. fischer)@ruu.c... de
Abstract
Stochastic neural networks (SNNs) are random functions whose predictions are gained by averaging over multiple realizations. Consequently, gradient-based adversarial example is calculated based on one set of samples and its classification another set. In this paper, derive sufficient condition for such a stochas- tic prediction to be robust against given sample-based attack. This allows us identify the factors that lead to an increased robustness of SNNs and gives theoretical explanations for: (i) the well known observation, that increasing the amount of samples drawn for the estimation of adversarial examples increases the attack's s strength, (ii) why increasing the number of samples during an attack can not fully reduce the effect of stochasticity, (iii) why the sample size during inference does not influence the robustness, and (iv) why a higher gradient variance and shorter expected value of the gradient relates theoretical findings give a unified view on the mechanisms underlying previously proposed approaches for increasing attack strengths or model robustness and are verified by an extensive empirical analysis.
higher robustness. Our
Introduction
Since the discovery of adversarial examples [Biggio et al. 2013 Szegedy et al. 2014), significant
amount of research was dedicated to hinder attacks [e.g. Madry et al. 2018 Papernot al. 2016
Zhang et al.] [2019]. to enhance attack strategies [e.g. [Athalye et al.] 2018]Akktt and Mian 2018
Carlini and Wagner 2017 (esstoo al. 2018 to derive ways to certify model robustness (e.g.
/
Cohen et al. 2019 Lécuyer et al. 2019]. Robustness guarantees often specify an e-ball around input
points in which perturbations do not lead to a label change (Hein and Andriushchenko 2017]] Croce
and Hein 2020). 'he maximal possible radius such an e-ball corresponds to the distance of the input point to the nearest decision boundary, which on the other hand equal to the length of the smallest perturbation vector that leads to misclassification (c.f. figure Ia)). Such a robustness analysis assumes that the decision boundaries are fixed and that the attacker able estimate (at least approximately) this minimal perturbation vector, which is a reasonable assumption for deterministic networks but usually does not hold for stochastic neural networks (SNNs).
Stochastic neural networks, and stochastic classifiers more generally, are random functions and predictions are given by the expected value of the random function for the given input. In practice this expectation is usually not tractable and hence it is approximated by averaging over multiple realizations of the random function. This approximation leads the challenging setting where predictions, decision boundaries, and gradients become random variables themselves. Hence, under an adversarial attack, the decision boundaries used for calculating the adversarial example and those used when predicting the label of the resulting adversarial example differ. This means that the attacker can not estimate the optimal perturbation direction i.e., the direction to the closest decision boundary of the network that will be sampled during inference c.f. figure b), c).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Jeady
Fddy
1
(a)
(b)
Figure 1: (Un-)successffl attacks on binary stochastic classifier with linear decision boundary. a) an adversarial example Tady = x + J created by shifting the direction of the closest decision boundary, indicated by the blue arrow. is the minimal needed attack length and the attack can only be an adversarial example if | 2 e holds. b) and c) show the stochastic decision boundaries during attack (dashed) and inference (solid). The red arrows indicate the shortest direction to the latter, respectively. In b) 8 moves x even further away from the decision boundary used during inference, while in e) the magnitude of 5 is too short to result in successful attack.
In this paper, we study how robustness of SNNs arises from this misalignment of the attack direction and the optimal perturbation direction during inference that results from the stochasticity inherent to stochastic classifiers.
We make the following contributions: First, we derive a sufficient condition for a SNN prediction relying on one set of samples to be robust against an attack that was calculated on a second set of samples. Second, we discuss how model properties and sample sizes impact this condition which does not only allows us to answer the questions stated in the abstract but also to explain the success of recently proposed defense mechanism from a simple unifying geometric perspective. Lastly, conduct an empirical analysis that demonstrates that the novel theoretical insights perfectly match what we observe in practice.
2 Related work
Several works proposed stochastic defense mechanism to increase adversarial robustness [e-g. Raff eet al.] 2219 Xie et al.] 2018). /Athalye et al. [2018) linked their success to gradient obfuscating during the attack and showed that increasing the number samples for approximating the gradient during attack leads to sever decrease in adversarial accuracy.
However, SNNs were still found have an increased robustness even w.r.t. stronger attacks Yu et
2021 He He et al. 2019 Eustratiadis et al. 2021 Jeddi et al.] 2020) Their success was attributed to
different effects of stochasticity, e.g. model smoothing [Liu al. 2018 Addepalli al., [02 or
diversification of the gradients [Lee et al., 2022 Bender et al. 2020
While a lot of work analyzed
the robustness of deterministic neural networks le.g. Madry et al. 2018 Croce et al. 2019 Croce and
Hein 2020 Dabouei et al.) 2020 Yang et al.] [2022] the robustness of SNNs is less well understood. One line of research focused on the robustness Bayesian neural networks (BNNs). Wicker et al.)
(2200 certified robustness of BNNs using interval bound propagation techniques which they later also employed to derive guarantees for the robustness of BNNs with modified adversarial training [Wicker et al. 2021). Moreover, CCrboone et al. (2020) investigate robustness in the infinite-widre infinite- sample limit. Lastly. Pinot et al. 2019 derived thcoretical robustness guarantees for randomized networks, where the randomization is based on additive noise from an exponential family distribution. This leads to a generalization of the robustness guarantees derived previously by Lécuyer et al. (2019) and Cohen et al. [ Those certified robustness guarantees specify the radius of ball in which the prediction does not change. In contrast, our results specify the robustness that results from the difficulty to identify the ideal attack direction and imply that even for perturbations outside this confidence ball, a gradient- gradient-based attack has a chance of not being successful.
To the best of our knowledge, no existing theoretical analysis of SNNs explicitly discusses either the effect of stochasticity during inference nor the impact of the sample size during attack and prediction on the robustness.

---

3 Preliminaries
We first clarify the terminology before we state the main theoretical results of our paper.
Stochastic classifiers We use the term stochastic classifiers for all classifiers which have an inherent stochasticity through the use of random variables in the model. Formally, we define them as follows: Definition 3.1 (Stochastic classifiers). A stochastic classifier with l: classes corresponds
function
f: IRd x 22 + IR* that maps a pair (I, ①) to the output f(x,, θ) = (Si (z, O),... (2, O))² where x E R2 is an input vector, θ E Ω* ~ p p()) is a random vector, and fa(r, θ) with c E {l...., . k} the discriminant functions for each class. The prediction of stochastic classifier for an input is given by [f } and the predicted class by arg max, Ef 6)
This generic definition of stochastic classifiers covers linear models with random weights, but also more complicated methods like BNNs [Neal, [996]. infinite mixtures [Däubener and Fischer 2020). Monte Carlo dropout networks [Gal and Ghahramani 2016), randomized smoothing as proposed by Lécuyer et al. [2019 T1 and any other class of neural networks which use stochasticity at the
input level le.g Raff et al. 2019 or within the network [e.g. He al. [2199] Jeddi et al.] 2020 Liu
ct al. 2018 Yu et al. 2021 Eustratiadis et al. 2021). In cases where Eelf +) is not tractable which in practice usually the case the prediction of the stochastic classifier is approximated
by its Monte Carlo (MC) estimate f (x) == s E. f(x, 0.), where the samples in the sample set
S [01, 0g) are drawn from p()).
Adversarial attacks on stochastic classifiers Informally speaking, adversarial examples are inputs that are modified such that the network predicts wrong classes even though the changes to the inputs are not perceptible for human. More precisely, let æ be an input with corresponding true label E {1. .......K. }} that is classified correctly by the multi-class classifier f()), that is arg max. fe(z)) y- We consider the most common attack form which aims at misclassifying æ by allowing for some predefined maximum magnitude of perturbation. That is, the attacker targets the optimization problem
marimize L(f(x + 6), s.t. |101pp S.1
where -(.,)) is the loss function, || ||. llpp with 2 1 is the én-norm, and is the perturbation strength, i.e the maximal allowed magnitude of the attack (see [ for an illustration). Common choices of loss functions include the cross-entropy loss and the negative margin loss Lmargin (f + 8). y) (I + 6) maxezy fa(x + 8)). In practice, targeting the optimization problem in eq. (T) usually involves estimating 8 by performing some kind of gradient-based optimization on the loss function (e.g. Goodfellow et al. 2015, Madry al. 2018 If f (-) is a stochastic classifier (as introduced in the previous paragraph) approximated by its MC estimate, the loss gradient with respect to the input is stochastic as well and given by
8
VzL(SS()),)) = V.L.
1 S
Σ
[(x, ((s,,.)..))
(2)
Note, that for linear C, as for example the margin loss for a fixed class c, the loss gradient of the mean prediction is equivalent the mean of the loss gradients for single sample predictions. However, in general (e.g. for the cross entropy loss) this is not the case.
4 Geometrical robustness analysis
After clarifying our understanding of stochastic classifiers and on how gradient- based adversarial attacks are conducted on them, we are now able to present simple but general geometrical, adversarial robustness analysis for stochastic classifiers. It motivated by the following observation: Each time a stochastic classifier is used for a prediction, another set of realizations from the random vectors are
'Note, that, in contrast Lécuyer al. 2019) the variant of randomized smoothing proposed by Co- hen et al. 2019) does not defne the prediction of the SNN to be given by the expectation over but by arg maX.gy (z + <) = c). where ~ (1 o² I). This makes our results not directly applicable to their networks. However, they can probably be transferred to the decision boundaries and attacks corresponding to their form of generating predictions.
3

---

drawn, resulting in another set of discriminant functions and corresponding decision boundaries. To put it into other words, the classifier gets a random variable itself. As a consequence, calculating a gradient-based adversarial attack for an input is done with respect to drawn set of realizations A (0f. .g..................... , (x ). and thus 8 from eq. ① is specifie for realization f^(x)) of the classifier, which we specify by writing 54. During inference, the resulting adversarial example Tadv = 2 + 8- is then fed to another random classifier f". which is based different set of realizations from the random vector I = {0) f8 8z }. From this perspective, the prediction model ft is robust against the attack if the distance from x to the decision boundary (given by (x) f (x) = 0)
in the direction of 8-A is larger than the length of 6A, as illustrated in figures [D) and c).
Robustness conditions for stochastic attacks For classifier with linear discriminant functions, we are able to turn the previously described observation into a theorem in which we derive a sufficient and necessary condition for the prediction model to be robust against a given attack.
Theorem 4.1 (Sufficient and necessary robustness condition for linear classifiers). Let f IR x R be a stochastic classifier with linear discriminant functions and f and f be two MC estimates of the classifier. Let x E Id be data point with label y E {1....,k) and arg max, fz)(z) arg max. (z) = y, and let Tody = x + 6 be an adversarial example computed for solving the minimization problem ① for f It holds that arg max. f2 + 54) y if and only if
min xI > 110A with cy
(3)
00
if cos(a
S
P/
((() (1((0) (2 costa? ooo
otherwise
where A.A is the angle between - V. (x)) and 8-4
The proof which is based on Taylor expansion is given in supplement A. The conditions for cos have a nice geometrical interpretation: An angle of more than 90º (which corresponds cos(az cosine value) indicates that the gradient - V. (z(((())
(a
negative (x)) and the perturbation 6-4 point into "opposite" directions and thus even for infinitely long moves into the direction of 8-4, the predicted label for x will not change to class c # y. For positive cosine values, --. specifies the distance to the decision boundary in the attack direction. It looks similar the minimal distance the decision boundary in a deterministic setting which is given by [V= fc(x)) and which recovered
f,(x)-fo(x)
if 2-) 1. A cosine value of one however can only occur if the gradient and perturbation direction are identical i.e., if the margin loss is used for calculating the attack direction and if attack and inference model are identical. In practice, the latter is almost surely not the case due to the finite sample approximation, and thus cos(a? I..A < 1. This illustrates the robustness advantage induces by stochasticity.
The derived conditions may locally hold for classifiers which can be reasonably well approximated by first-order Taylor approximation. To derive further guarantees, can relax the linearity assumption by assuming discriminant functions which are L-smooth as defined in the following.
Definition 4.1 (L-smoothness (Yang et al., 2022). ). A differentiable function f : RRd - R is
L-smooth, if for any 31, I2 Ra and any output c (I,...,k))
fc(r)) V.,fa fc,)(ll)) < L
b x2
For such smooth discriminant function we can derive sufficient (but not necessary robustness condition specified by the following theorem, which proven in supplement
Theorem 4.2 (Sufficient condition for the robustness of a L-smooth stochastic classifier). In the setting of Theorem 4.1 let f and f be L-smooth (instead of linear) discriminant functions. Then it
'The necessity of this condition is not given since for non linear decision boundaries it is possible that behind region of different class there exists another region of class y that attack ends in if 8 is long enough.

---

holds that arg max, (z (8A) = if if sy re > 118~112 with
00 ,0, if VV- - fe [2) cos(a + 2 1184112 S 0,
re
(
otherwise
[+ (0)) cos ) 1111||12
with cos (.A) in theorem 4.1
Factors influencing the robustness of stochastic classifiers Since in practice neither the parameter set A nor I is fixed, the quantity better describing the practical robustness of stochastic classifier with linear discriminant functions is
P(min p2 Caty
[10~112)
where F7 and 8-4 are random variables. For L-smooth models, replacing F.. by r leads to lower bound the robustness. Deriving analytic expression for this probability is hard problem. However, based on theorems 4.1 and 4.2 it becomes clear that a larger min Te relates to increasing the probability in eq. ( and thus to an increased robustness. We note that Te with c + grows with i) larger prediction margins 1 (1)) - fz (æt), ii) smaller gradient norms |/ [ (x) - Le (z)) and iii) larger angles Z.A, Larger prediction margins and smaller gradient norms were also found to positively impact the robustness of deterministic networks [c.g. Ross and Doshi- Velez 2018). In contrast, the dependency on the angle is unique to the stochastic setting. We therefore focus on the
analysis of this factor in the following.
Analyzing the expected angle The angle A depends on both terms (=(f,(x) -())))((()))))) (for which we use the shorthand - in the following) and 8 For further analysis, we first rewrite the gradient
1
-V -v
Σ
Sn Sg-o(s,
V V (efg-o(x,0)
5T
5I
Let 1z Bel V 2=fp-c(1; 03)] and E be the covariance of - -V (x))).). then it follows from the N (w, ( ). For simplicity
central limit theorem that for sufficiently many samples -V f= let us assume that the attack is based on the margin loss and only one iteration of gradient ascent. In
this case, 5-A Tólla
7 6, with 8 ~ N (H, ) Note, that we can neglect the scaling of the attack
vector, since the angle only depends on 8. Therefore, estimating the distribution of T.A corresponds to estimating the distribution of the angle between two independent multivariate Gaussian random vectors with the same mean and (potentially) differently scaled versions of the same covariance It is known for vectors from normalized standard multivariate Gaussian distributions that the mode of the distribution of the angle between two random vectors is equal to 90 and that with increased dimension the concentration around this mode gets tighter [Cai et al., 2013). Unfortunately, deriving a closed form expression for the distribution or expectation in the general case is a challenging task and beyond the scope of this paper. However, we conjecture that the expectation of the angle increases proportionally with the variance and anti-propottionalll with the norm of the mean. We illustrate the intuition behind this hypothesis in figure 2 To empirically verify the correctness of this hypothesis we conducted an experiment where we estimated the expected angle between two identically distributed 1.000-dimensiom Gaussian random vectors for different choices of means and diagonal covariances. More precisely, we sampled the mean and variances uniformly from [0, 4). where i. increased from zero to ten with step size 0.2, and estimated the expected angle based on 10,000 vector pairs drawn from the resulting distributions. Results are shown in figure 2 on the right side. As hy hypothesized, the smaller the length of μ and the higher the average variances, the higher the expected angle.
II the case of more complicated attacks the mean will differ as well but we expect the findings of these section generalize also this scenario.
5

---

80
60
Implications for the model From the perspective of the defender, the analysis of the angle shows that models with an increased gradient variance (which is often associated to high prediction variance) and small norm of the mean gradient are connected to larger values of a and thus to higher probability of unsuccessfully attacks. This explains why including the norm of the mean Average standard
40
-0
50
75
100
125
150
175
Longth of mcan voctor
Figure 2: Dependence of the angle w.r.t mean and variance of the gradient. Left, top: Sketch of decreased variance which leads to smaller angles. The mean of the gradient is shown the blue arrow. The circles indicate the areas of high probability for different scales of the covariance matrix. When the covariance is decreased (dark compared light gray region) the maximum angle between vectors in these region to vectors from the blue region is decreased as well (a1 compared to 0r2). Left, bottom: Sketch of increased mean which leads smaller maximal angles. When the covariance is fixed and the mean is increased (dark compared to light blue arrow) the maximum angle between vectors from high probability regions decreases (a3 compared a44.. Right: Expected angle between two identically distributed 1, 4.000-dimensional Gaussian random vectors, for Gaussian with different means and variances. Expected angles calculated based 10,000 pairwise draws from the respective Gaussian distributions.
Implications for the attack Based on the previous discussion, the only way the attacker can influence the probability of a successful attack, and in this sense the robustness of the model against the attack, is by increasing the amount of samples and thereby reducing the variance of the attack vector. A reduction of the variance leads to a decrease of the expected angle as described in the previous section. This gives more elaborated explanation of what was often loosely described as finding the "correct" gradient direction in previous work [Athalye et al.) 2018). However, even if the attacker would be able to take infinitely many samples, and thus the variance of the attack direction would be reduced to zero, the expected value of the angle will be larger than zero because of the still existing stochasticity in the inference process. This stochasticity can even lead to an expected angle close to 90 if short, and/or the covariance Σ is high. That is, the advantage of obfuscating the optimal attack direction by incorporating stochasticity into the classifier can be decreased but not fully counterbalanced by taking more samples during the attack. This might explain the finding in He et al. (019), that the accuracy under attack stagnates at a higher level than the deterministic counterpart when increasing the number of iterations of iterative gradient- based attack methods.
gradient, the gradient variance, or the angle between gradients as regularization terms in the training of SNNs, as for example proposed by Bender al. 2020 and Lee al. (2022 lead to an increased empirical robustness.
would be naturally to suspect that increasing the number of samples used during inference also decreases the robustness, since it decreases the expected angle. However, this is not the case, as it is counterbalanced by decrease of the norm of the gradient estimate (i.e. the second term in the denominator of F.) with growing sample size []] This can be seen by rewriting the expected denominator with respect to the two sample sets I and A as
'In these works the regularization terms motivated by maintaining input sensitivity and bounding the expected loss increase, respectively.
*ee present preposition showing that the interval incorporating the gradient norm decreases to the norm of with increasing amount of samples in supplement A.

---

(- -Vaoy-c(() 841
Ez.1 [1 V Valy-a(z)]l cos(a [- = BI,A
V Wafy-c(())1
11V (z))l2 ||
= =Ez, (-
|
Ex [- (x)] .EA Σ 1=1
=
1811122
where the last equation holds due to the independence of 8-h and VV -V The expectation of
[6 does not depend the samples taken during inference and the expectation of the estimate of the derivative w.r.t. the i-th input Ez [~V.,, F. [z)] is the same for different amounts of samples. Therefore, the expectation of the denominator does not change when changing the sample size during inference. This observation explains why the robustness of stochastic classifier does not depend on the amount of samples taken during inference and gives an justification for picking an arbitrary
6
sample size that allows for good trade-off between efficiency and reduction of the variance of the MC estimate used for prediction.
Experimental robustness analysis
In this section empirically demonstrate that the findings of our theoretical analysis transferable to SNNs and help to explain the mechanisms leveraging the experimentally observed robustness of previously proposed SNNs.
Experimental setup Our experiments are conducted on two different image datasets: FashionM-
NIST (Xiao et al. 2017] and CIFARIO (Krizhevsky et al.) ( For experiments MNIST we used feedforward neural networks (FNN) with two stochastic hidden layers, each with 128 neurons.
Variational Matrix Gaussian (VMG, the BNN proposed by Louizos and Welling 20166) via variational inference or as an infinite mixture (IM) with the maximum likelihood objective proposed by Däubener and Fischer [2020), with matrix variate normal distribution placed over the weights. We also trained FNNs of the same architecture, where we added Gaussian noise with 2 {0.05, 0.1} the input, by minimizing the cross-entropy. We refer these models as stochastic input networks (SINs) 0.05 and SIN 0.1, respectively. Note, that these networks correspond the basic networks proposed for randomized smoothing by Lécuyer et al.) (2019). For experiments on CIFARIO 10 we trained two wide residual networks (ResNet) with MC dropout layers (Gal and Ghahramani 2016) applied after the convolution blocks and dropout probabilities p 0.3 and p 0.6. If not specified otherwise we used 100 samples of p(O) for inference on all datasets and calculated adversarial attacks with the fast gradient (sign) method (FGM) [Goodfellow et al. 2015). the cross-entropy loss for IMs, BNNs, and ResNets, the margin loss Cmargin specified in section 3 for SINs, and the 42-nomm constraint based on the CleverHans repository [Papernot sin- gle NVIDIA GeForce RTX 2080 Ti. We refer
We trained the FNN
1.0
0.8
al. 2018 All experiments were
Accuracy
0.6
the reader supplement for more details
0.4
the datasets, models, and the training procedure.
0.2
Actual accuracy Linear r-accuracy
Accuracy of robustness conditions We first investigated the practical transferability of the derived theorems. For enforcing the smoothness condition used in theorem (.2 we built on the result from Yang et al. 2022], who showed that the L-smoothness parameter of classifier x [f e)] smoothed with random noise ~ NO0 )²), is bounded by L < 2/0². We therefore applied randomized smoothing during training of the models. For the BNN, on which
0.0
Smooth raccurracy
0.0
0.1
0.2
0.3
0.4
Perturbation strength
Figure 3: Adversarial accuracy of the smoothed BNN model for attacks based on 10 samples vs percentage of images with min, (smooth) and min, 1 > ||8A1]2 (linear) for the first 100 images from the NAST dataset.
||8- [|
Additional experiments for CIFAR100 are presented in the supplement C.S.
7

---

we focus in this section due to space restrictions (results for the other networks look qualitatively similar and can be found in supplement C), we replaced each image in the batch by two noisy copies with Gaussian noise e ~ (0, 0.1) which ensures I. I 20. During prediction we estimated the expectation under Gaussian noise with 10 samples and also used 10 samples for calculating the FGM attack. We estimated the percentage of resulting attacks for which min r2 > ||0~1| and min, -2 > ||8-112 and compared this to the adversarial accuracy (i.e. the percentage of perturbed samples classified correctly) in figure The percentage of samples fulfilling the condition mine r > ||ō~||2 approaches zero with growing perturbation strength, indicating that the lower bound provided by re is rather loose due to the high (upper bound of the) L-smoothness constant. On the other hand the percentage of samples for which (3) is fulfilled is a good approximation of the real accuracy for small attack length, which suggests that the discriminant functions are approximately linear in a small neighborhood of the input.
0.6
sttack sample
9.0
0.8
Accurncy
0.8
1.00 wttert samples 0.6
wttacck 0.6
attacc sample
attwrkc samples 0.6
5 sttakk samplees 97
sttacck strppee SR D.1
120 atoact samples 0.3
2.4
100 ettert samples IN
1002 atteck semplees
atare comples IN
0.4
100 wieck wamples 0.1
attteck samplee SNV
atherc samp
0.4
0.2
1000 allack sempike BMN
0/2
100 wllekt wamplea
0.0
0.5
3.5 Porturtation strength
1.0
2.0
0.0
0.5
1.0
3.5
2.0
0.0
0.2
0.4
0.5
0.8
Porturtation strength
Perturbation strengsh
(a) IM
(b) SIN
(c) ResNet with dropout prob. 0.6
Figure 4: Accuracy under attack for different perturbation strength and amount of samples used for calculating the attack. The dashed line shows the adversarial accuracy for the models with the same architecture but less prediction variance.
Stronger attacks by increased sample size In this section we investigate the effect of varying the
amount of samples used for calculating the attack, e.g. for S E (1,5, 10, 100, (1000)]- Figure 4 shows the resulting adversarial robustness of the IM, SIN 0.1, and the ResNet with dropout probability 0.6. The accuracy under attack decreases for all models with increasing amount of samples used for calculating the attack as conjectured. We found this to hold also for stronger attacks, i.e. attacks based on logits for IM and BNN (where we observe highly confident softmax predictions), attacks with Lo constraint, or projected gradient descent (PGD) [aarry et al. 2018] attacks with 100 iterations as shown in supplement C. Note, that iterative attacks already increase the sample size due to their iterative nature, and therefore only few samples per iteration may sufficiently increase attack strength.
Simultaneously the accuracy we estimated the corresponding values of cos(a TTA), for j arg min, F2, for the first 1,000 test images and depicted them in figure 5 It can be seen that the cosine values are increasing (which corresponds to decreasing angles) with growing sample size and that the larger the observed values, the lower the accuracy under attack as shown in figure 4 This observation in accordance to our theoretical analysis which predicts that an increased amount of samples leads to higher cosine values and in turn to less adversarial robustness. Note however, that even when taking many samples (os I.AA < 1, which underlines the fact that the optimal attack direction can not be recovered due the still existing stochasticity in the inference procedure. Further
results on the angle under different amounts of samples during attack can be found in supplement C.2.
Prediction variance as robustness indicator In this section we compare the properties of SNNs that have the same network architecture and a similar training procedure but different prediction variances. That is, we compare the BNN against the IM, SIN 0.05 against SIN 0.1, and the two ResNets with different dropout probabilities against each other. First, we estimated the standard deviation of the prediction and the average standard deviation of the gradient entries of the models by calculating the average of the empirical estimates over the first 1,000 examples from the respective test sets (see [T. As expected, the IM, SIN 0.1, and ResNet with dropout probability 0.6 have a larger standard deviation of the prediction and hence prediction variance than their respective counterpart. The higher standard deviation of the prediction translates to an increased standard deviation of the gradient. This explains why we observe smaller cosine values for these models compared to their counterparts (see right most boxplots in figure 5 a) and c)) which in turn translate
8

---

1.0
costangle
0:00
coslangle)
costangle)
0.8
0.2
0.2
0.5
6.4
1.M
PM
10:00
200 1000
3082 DIM
5.0006
10d9s samples
samples
(a) IM
(b) SIN 0.1
(c) ResNet with dropout prob.
Figure 5: Cosine values for adversarial examples with perturbation strength 1.5 for a) and b) and 0.3 for c) and different amounts of samples. White crosses indicate mean values.
Table 1: Empirical standard deviation of the prediction (for the correct class) based on 1,000 single predictions (each with inference sample) and average length and variance the corresponding gradient. We report averages over the first 1,000 images from the respective test set.
FASHIONMNIST
CIFARI0 10
BNN
IM
SIN 0.05
SIN 0.1
DR 0.3
DR 0.6
AVG. STD OF PREDICTIONS
0.0186
0.0473
85,4657
186.5599
0.0146
0.0196
AVG. GRADIENT LENGTH
0.5218
0.5044
49.0120
48,8312
0.0713
0.0599
AvG. STD OF GRADIENT
0.0316
0.0959
1.5741
1.8323
0.0000
0.0000
to an increased robustness (as indicated by the dotted accuracy curves in figure We also found that the length of the mean of the gradient that estimated based on 1,000 samples is smaller for the models with higher variance This might be another reason for the observed smaller angles as discussed in the previous section.
Robustness in dependence of the amount of samples used during inference practice the amount of samples 5t drawn during inference is fixed to an arbitrary number. In this section we investigate the impact of varying S {1.5, 10, 100} which are values frequently used. First, observed that only few samples are necessary to get reliable predictions as shown by the test accuracies in table Second, we found that increasing the amount of samples did not affect the adversarial accuracy. This can be explained by the observation that the decrease of I.AA caused by increasing the sample set is counterbalanced by an simultaneous decrease of the average norm of the gradient estimate, as can be seen by inspecting the results in figure 6 That is, the product V g= cos(a?.4) stays approximately the same regardless of the inference sample size as predicted by our theoretical analysis. Interestingly, the effect of increasing the number of samples during inference has almost no effect on the prediction margin (more results are shown in supplement C.4).
clean data for all models
Prediction margin
0.4
8.4
02
-02
10
100
100
10
320
102
10
120 samples
100
NUTrON samples
inference samples
(a) cos(az I.AA
(b)
V=fg-c(z)]l
(c) fg-e(#)
Figure 6: Values of single factors of F in dependence of varying number of samples used during inference for the BNN trained FashionMNIST.
Forr SINs, is known that a higher variance used during training relates to a lower Lipschitz-constee constann [e.g. Yang et all 2022 Salman al. 2019) which leads to stronger smoothing effect.

---

Table 2: Test set and adversarial accuracy with 100 samples during the attack and allowed perturbation
strength of 1.5 on FashionMNIST and 0.3 on CIFAR10 for increasing number of samples used during prediction. We estimated the average accuracy 10 times and report the average and standard deviation.
FASHIONMNIST
CIFARIO
I
BNN
IM
SIN 0.05
SIN 0.1
DR 0.3
DR 0.6
TEST SET ACCURACY
83.03
0.12
79.15 0.27
86,98 0.17
85.76 1 0.17
92.08 1 0.13
92.68 1 0.15
5
84.81 + 0.14
84.29 0.21
88.21 + 0.09
87.49 + 0.15
0.05
93.51 + 0.09
10
85.03 0.13
85.02 0.18
88.47 + 0.06
87.92 + 0.10
92.66 0.06
93.60 = 0.06
100
85.21 0.06
85.72 0.08
88.63 + 0.07
88.27 + 0.06
92.74
93.69 = 0.04
ADVERSARIAL ACCURACY
1 35.76
+ 0.92
44.67 1.19
36.15 + 0.79 0.79
10.63 + 0.79
42.64 + 0.39
17.18 + 0.70
5
36.19
0.47
47.08 + 0.73
35.92 + 0.51
0.44
12.73 + 0.31
17.00 0.38
10
36.30
0.50
17.23 + 0.62
36.06 0.41
39.93 + 0.66
12.69 0.37
47.01 0.33
36.50
+ 0.39
47.96 0.24
35.91 0.23
39.67 0.29
42.82 0.17
47.02 0.20
Conclusion
this work we have stressed the fact that stochastic neural networks (SNNs) (and stochastic classifiers in general) often depend on samples and thus their predictions are random variables themselves. For gradient-basee adversarial attacks this means that the attack calculated based on one realization of the stochastic network (which depends on multiple samples of the random variables used in the network) and applied to another which is used for inference. We derived a sufficient condition for this inference network to be robust against the calculated attack. This allowed to identify the factors that lead to an increased robustness of stochastic classifiers: i) larger prediction margins ii) a smaller norm of the gradient estimates and iii) higher angles between the attack direction and the direction to the closest decision boundary during inference. The observed angles depend inverse proportionally on the norm of the expected gradient and proportionally variance can be reduced by increasing the sample size. These insights enable us to explain previously reported empirical findings for SNNs from geometrical perspective, e.g. that the robustness of SNNs is higher than the robustness of their deterministic counterparts even for strong attacks that are based on several samples (He et al.) 2019 Eustratiadis al.) 202T], why regularization of the gradient variance [Bender et al.) [2020]. norm of the mean gradient, and angle (Lee et al. [022) improves the adversarial robustness, and last but not least why increasing the sample size during attack is important to exploit its potential [Athalye et al., 2018). Therefore, our work poses a general applicable and simple framework, that helps understanding the mode of operation of existing stochastic defense mechanisms, even if they were motivated from a different point of view. Moreover, we derived justification for the common practice of choosing the sample size during inference in way that balances its prediction certainty against the computational cost. Finally, we believe our findings will be useful to evaluate and compare the robustness of different models, since they point out that they might require different amounts of samples during attack to sufficiently reduce variance and hope that they will help to improve the robustness of stochastic classifiers in future.
the variance of the gradient estimates. This
Potential negative societal impact and limitations. Since we do not propose new attack strategy, but contribute a better understanding of the robustness of SNNs and advise to cautiousness when determining sample sizes, we do not see negative impact.
Acknowledgments
We would like to thank Denis Lukovnikov for his useful comments work and beautifying our graphics. This work funded by the Deutsche Forschungsgemeinschaff (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2092 CASA 390781972.
10

---

References
Sravanti Addepalli, Samyak Jain, Gaurang Sriramanan, and R. Venkatesh Babu. Boosting adversarial robustness using feature level stochastic smoothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 93-102, 2021.
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6:14410-14430, 2018.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Cireumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 274 273, 2018.
Christopher Bender, Yang Li, Yifeng Shi, Michael K. Reiter, and Junier Oliva. Defense through diverse directions. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 756-766, 2020.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Sıniić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Železný, editors, Machine Learning and Knowledge Discovery in Databases, pages 387-402. Springer Berlin Heidelberg, 2013.
Tony Cai, Jianqing Fan, and Tiefeng Jiang. Distributions of angles in random packing on spheres. Journal of Machine Learning Research, 14321):1877-184, 2013.
Ginevra Carbone, Matthew Wicker, Luca Laurenti, Andrea Patane', Luca Bortolussi, and Guido Sanguinetti. Robustness of bayesian neural networks to gradient- attacks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 15602-15613, 2020.
Nicholas Carlini and David Wagner. Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods, page 3-14. Association for Computing Machinery, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference Machine Learning (ICML), volume 97, pages 1310-1320, 2019.
Francesco Croce and Matthias Hein. Provable robustness against all adversarial L, perturrbaaion for 1. In International Conference Learning Representations (ICLR), 2020.
Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of relu networks via maximization of linear regions. In Proceedings of the Twenty-Secone International Conference on Artificial Inielligence and Statistics (AISTATS), volume 89, pages 2057-2066, 2019.
Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, Jeremy Dawson, and Nasser M. Nasrabadi. Exploiting joint robustness to adversarial perturbations. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1119-1128, 2020.
Sina Däubener and Asja Fischer. Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification. In arXiv preprint https: // arziv. org/ abs/ 2008, 03209 2020.
Panagiotis Eustratiadis, Henry Gouk, Da Li, and Timothy Hospedales. Weight-covariance alignment for adversarially robust neural networks. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pages 3047-3056, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference Machine Learning (ICML), volume 48, pages 1050-1059, 2016.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, (ICLR), 2015.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
11

---

Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Ahmadreza Jeddi, Mohammad Javad Shafiee, Michelle Karg, Christian Scharfenberger, and Alexan- der Wong. 2earn2perturb:: An end-to-end feature perturbation learning to improve adversarial robustness. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1238-1247, 2020.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). In http: // wwwr. toronto. edu/ "kriz/ cifar. html
Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy (SP), pages 656-672, 2019.
Sungyoon Lee, Hoki Kim, and Jaewook Lec. Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2022.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Computer Vision - ECCV, pages 381-397, 2018.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In Proceedings of The 33rd International Conference on Machine Learning (ICML), volume 48, pages 1708-1716, 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018.
Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag, Berlin, Heidelberg, 1996.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy (SP), pages 582-597, 2016.
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the eleverhans 12. adversarial examples library. In arXiv preprint https: // arziv. org/ abs/ 1610. 00768 2018.
Rafael Pinot, Laurent Meunier, Alexandre Araujo, Hisashi Kashima. Florian Yger, Cedric Gouy- Pailler, and Jamal Atif. Theoretical evidence for adversarial robustness through randomization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. Barrage of random transforms for adversarially robust defense. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6521-6530, 2019.
Andrew Ross and Finale Doshi-Veles Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sehastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations (ICLR), 2014.
12

---

Jonathan Uesato, Brendan O' D'Donoghue, Pushmeet Kohli, and Aäron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pages 5032-5041, 2018.
Matthew Wicker, Luca Laurenti, Andrea Patane, and Marta Kwiatkowska. Probabilistic safety for bayesian neural networks. In Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence (UAI), volume 124, pages 1198 1207, 2020.
Matthew Wicker, Luca Laurenti, Andrea Patane, Zhuotong Chen, Zheng Zhang. and Marta Kwiatkowska. Bayesian inference with certifiable adversarial robustness. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130, pages 2431-2433 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Mushion-mnise novel image dataset for benchmarking machine learning algorithms. In arXiv preprint https: // arziv. org/ abs/ 1708. 07747 2017.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. International Conference on Learning Representations (ICLR), 2018.
Zhuolin Yang, Linyi Li, Xiaojun Xu, Bhavya Kailkhura, Tao Xie, and Bo Li. On the certified robust- ness for ensemble models and beyond. In International Conference Learning Representations (ICLR), 2022.
Tianyuan Yu, Yongxin Yang, Da Li, Timothy Hospedales, and Tao Xiang. Simple and effective stochastic neural networks. Proceedings of the AAAI Conference on Artificial Intellignnce, 35(4): 3252-3260, 2021.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference Machine Learning (ICML), volume 97, pages 7472-7482, 2019.
13

---

7 Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The theoretical results are limited
by their assumptions in section 4
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Below the conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes] In section
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs are given in supplement A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material description of datasets, models and training procedure from section [5 in supplement B and provided the code used in the main paper in the supplemental material.
URL)? [Yes] We extended the
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Details for the experiments which are not mention in section 5 are given supplement B.
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Partially, where it applicable and interesting (c.f. table 2)
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] All experiments are run one NVIDIA GeForce RTX 2080 Ti and package version requirements for the virtual environment are given in the code base.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Main python packages and models are cited in the paper as well as highlighted in the code.
(b) Did you mention the license of the assets? [Yes] The used code bases from others were published under MIT license.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] Not applicable.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Not applicable.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Not applicable.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Not applicable.
14


Reviews:
[{"review_id": "Zyb6dURcXT9", "reviewer": "Reviewer_kUU3", "summary": "The authors perform a systematic study on the robustness of stochastic neural networks (SNNs). They make several theorectical corollaries: 1) In order to increase the probability of a successful attack against a SNN, the attacker need to increase the amount of samples to estimate the gradient. 2) The defender need to increase gradient variance and decrease the norm of the mean gradient, in order to improve the robustness of a SNN. 3) The robustness of a stochastic classifier does not depend on the amount of samples taken during inference.\nThe authors also conuduct extensive experiments on Fashion-MNIST and CAIFAR-10 datasets to emperically verify the correctness of these corollaries.", "questions": "The paper \"Certified Adversarial Robustness via Randomized Smoothing\" uses randomized smoothing to provide a robustness certificate for a neural network. A neural network with randomized smoothing can be seen as a type a stochastic neural network. How can Theorem 4.1 and Theorem 4.2 be used to certify robustness for a neural network with randomized smoothing? And how is it related to the robustness certificate in the paper \"Certified Adversarial Robustness via Randomized Smoothing\"?", "limitations": "They discuss the limitations and potential negative societal impact of their work in section 4 & 6.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is written with good clarity and easy to follow. It also provide valuable theoretical insights on the robustness analysis of stochastic neural networks. The theoretical corollaries are well supported in the experiments.\n\nWeaknesses: The datasets used in the experiments are relatively small. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M6qOxtSZjHD", "reviewer": "Reviewer_izQZ", "summary": "This paper drives a sufficient condition for the robustness of stochastic neural networks (SNNs). The theorems show that SNNs can classify adversarial examples correctly if the angle between adversarial perturbations and gradient of the inference estimate of SNNs satisfies specific properties. These theoretical results imply several strategies for attack and defense for SNNs.", "questions": "1. How is random smoothing related to the implications on SNN?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ This paper gives a theoretical condition to show when SNNs can classify adversarial examples correctly. \n+ Most parts of theorems and proofs are well-written and clear for understanding.\n+ Empirical results are aligned with the theoretical results.\n\nWeakness:\n\n- Missing information in theorem 4.1: $c \\neq y$ is used in proof but not in Theorem 4.1.  Supplementary material defines $\\alpha$ in line 29, but Theorem 4.1 does not describe it.\n\n- I think the theorems discussed in the paper are a bit too general and do not go deeper inside the design of SNNs from a probabilistic perspective:  The estimate used for attack and the estimate used for inference could be arbitrary two models with different decision boundaries.\nIt would be interesting to discuss more on how some properties (like the number of realizations) of SNNs influence the adversarial classification.\n\n- A type of related work is not well discussed: Random smoothing is a popular certified defense method and could be also treated as a type of SNN. (Random variables only change the input and do not influence the weight). It would be good to mention and discuss how random smoothing fits into the proposed theory. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KJIUsghjUXH", "reviewer": "Reviewer_nbL1", "summary": "The authors study the adversarial attacks for stochastic classifiers. The attacker has the disadvantage of not knowing the exact direction of the attack since the hypothesis is a random variable and it would change randomly every time that it is queried. It has been observed that increasing the sample size would decrease the attack success rate, which goes against intuition on its face.\n\nA sufficient condition for the robustness of the stochastic classifier against the calculated attack is proposed. Based on this condition, we can identify the factors that lead to an increased robustness of stochastic classifiers: \n1- Larger expected prediction margins \n2- A smaller expected norm of the gradient estimates \n3- Higher angles between the attack direction and the direction to the closest decision boundary during inference.\n\nThe observed angles depend inverse proportionally on the norm of the expected gradient and proportionally on the variance of the gradient estimates. This variance can be reduced by increasing the sample size. The authors proposal explains the previously reported empirical findings for SNNs from a geometrical perspective, that the robustness of SNNs is higher than the robustness of their deterministic counterparts even for strong attacks that are based on several samples, why regularization of the gradient variance, norm of the mean gradient, and angle improves the adversarial robustness, and why increasing the sample size during attack is important to exploit its potential.", "questions": "What is the difference between a robust stochastic model and an stochastic model that have not converged in its derivatives?", "limitations": "The authors make a good effort to describe the limitations. However, they have not included the learning theory aspects of the phenomenon. The proposal is also positioned as an explanation for observations, where as it is more likely that the observations are not causally related to the phenomenon.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality:\nThe paper tries to bring insight on the phenomenon from the perspective of stochastic classifiers. The proposal is positioned as an explanation for the already observed aspects of the phenomenon is SNNs. From the three proposed conditions of robustness, large margin and the mean gradient norm are already studied in the literature. The angle is unique to the stochastic setting, and is the main result of the paper.\n\n- Quality:\nThe authors have made the claims rigorous. The theorems and the results are intuitive enough. I have not read the supplementary material and the proofs. Nevertheless, the proposal lacks an analysis of the convergence of the classifiers in derivatives. The observations could be trivially explained if we assume that the derivatives of the trained SNNs have not converged in probability, or that the convergence is pointwise.\n\n- Clarity:\nThe intuition behind the proposal is clear.\n\n- Significance:\nThis is the my main concern with the paper. The main object of study in the paper is only definable for stochastic models. Consequently, it cannot be a good foundation for more general settings. It would be of help if the authors describe the proposal from the perspective of convergence in learning theory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D-p1NA-Lntr", "reviewer": "Reviewer_qGjN", "summary": "This paper investigates the robustness of stochastic neural networks against adversarial attacks. To that end, it derives a sufficient condition on the robustness in terms of the decision margin, the gradient norms, and the angle between the decision boundary and the perturbation vector. The sufficient condition holds for linear and L-smooth classifiers and the paper provides some intuition as to why the conditions should generalize to arbitrary networks. The condition confirms various empirical approaches trying to improve robustness: more samples increase the attack performance, reducing the gradient variance and expected value increase the network’s robustness, and the number of samples used for inference is largely irrelevant.", "questions": "What are the implications for randomized smoothing methods aimed at increasing the certified robustness of neural networks?", "limitations": "The paper addresses the potential negative societal impact.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides a novel theoretical perspective on the robustness of stochastic neural networks. It derives a sufficient condition on the robustness in terms of the decision margin, the gradient norms, and the angles between the perturbation vector and the decision boundary. As a consequence, the paper can analyze the influence of the different terms, which have been empirically used to increase robustness in prior work, on the robustness of neural networks. Thus, the paper presents a unified view of the robustness of stochastic neural networks, which can be useful for future research in this area.\n\nSome typos:\n\n- Line 97: the input *is* stochastic\n- Line 269: theaccuracy -> the accuracy\n- Line 271: many -> Many\n- Caption of Table 1: correspondinggradient -> corresponding gradient", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept