RoMA: a Method for Neural Network Robustness Measurement and Assessment 
Authors: Natan Levy
Guy Katz
Venue: ICLR 2022 Submitted (2022)
Keywords: Neuran Network
Robustness
Safety Critical Software
Categorial Robustness

Abstract: Neural network models have become the leading solution for various tasks, such
as classification, language processing, protein folding, and others. However, their
reliability is heavily plagued by adversarial inputs: small input perturbations that
cause the model to produce erroneous output, thus impairing the model’s robustness.
Adversarial inputs can occur naturally when the system’s environment behaves
randomly, even in the absence of a malicious adversary, and are thus a severe
cause for concern when attempting to deploy neural networks within critical
systems. In this paper, we present a new statistical method, called Robustness
Measurement and Assessment (RoMA), which can accurately measure the robustness
of a neural network model. Specifically, RoMA determines the probability
that a random input perturbation might cause misclassification. The method allows
us to provide formal guarantees regarding the expected number of errors a
trained model will have after deployment. Our approach can be implemented on
large-scale, black-box neural networks, which is a significant advantage compared
to recently proposed verification methods. We apply our approach in two ways:
comparing the robustness of different models, and measuring how a model’s robustness
is affected by the scale of adversarial perturbation. One interesting insight
obtained through this work is that, in a classification network, different output
labels can exhibit very different robustness levels. We term this phenomenon
Categorial Robustness. Our ability to perform risk and robustness assessments
on a categorial basis opens the door to risk mitigation, which may prove to be a
significant step towards neural network certification in safety-critical applications.

ROMA: A METHOD FOR NEURAL NETWORK ROBUST- NESS MEASUREMENT AND ASSESSMENT CONFERENCE SUBMISSION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network models have become the leading solution for a large variety of tasks, such as classification, language processing, protein folding, and others. However, their reliability is heavily plagued by adversarial inputs: small input perturbations that cause the model to produce erroneous outputs. Adversarial in- puts occur naturally when the system's 's environment behaves randomly, even in the absence a malicious adversary, and are a severe cause for concern when attempting to deploy neural networks within critical systems. In this paper, we present a new statistical method, called Robustness Measurement and Assessment (RoMA), which can measure the expected robustness of neural network model, Specifically, RoMA determines the probability that a random input perturbation might cause misclassification. The method allows us to provide formal guarantees regarding the expected frequency of errors that trained model will encounter after deployment. Our approach can applied to large-scale, black-box neural networks, which is a significant advantage compared to recently proposed verifi- cation methods. We apply our approach two ways: comparing the robustness of different models, and measuring how a model's 'ss robustness is affected by the mag- nitude of input perturbation. One interesting insight obtained through this work is that, in classification network, different output labels can exhibit very different robustness levels. We term this phenomenon categorial robustness. Our ability to perform risk and robustness assessments risk mitigation, which may prove to be a significant step towards neural network certification in safety-critical applications.
categorial basis opens the door
INTRODUCTION
In the passing decade, deep neural networks (DNNs) have emerged one of the most exciting and innovative developments in computer science, allowing computers to outperform humans in various classification tasks. However, a major disadvantage of the DNN approach is the existence of adversarial inputs Goodffllow et al.) (2014): inputs which are very close (according to some metrics) correctly-classifee input, but which are misclassified themselves. It has been observed that most state-of-the-at DNNs are highly vulnerable to adversarial inputs Carlini & Wagner (2017); and it has been suggested that adversarial inputs an inescapable part of the neural network architecture, and are thus not an issue that can be solved entirely Ilyas et al. (011).. In this perspective, it is crucial find method for containing this phenomenon and mitigating the risk that causes; especially in order to allow DNNs to be deployed in -articaciitical settings (e.g., automotive, aerospace, trains, and medical devices), where regulatory requirements and public opinion set high bar for reliability.
As the impact of the Al revolution is becoming evident, regulatory agencies are starting to address the challenge safely integrating DNNs into safety-critical systems - by forming workgroups to create the needed guidelines. Notable examples in the European Union include SAE G-34 and EUROCAE WG-114 Pereira Thomas (000);; Vidot et al. (202]); ; and the efforts made by the European Union Safety Agency (EASA). which is responsible for civil aviation safety, and which has published a road map for certifying Al-based systems) European Union Aviation Safety Agency 2020). These efforts highlight the dire need for certification methods for DNN-based systems.

---

The existence of adversarial inputs does not seem to be deal-breaker as far as regulatory agencies are concerned; indeed, these agencies are used to certifying systems with components that might fail due to an unexpected hazard. A common example is the certification of jet engines for civilian aircraft, with a known mean time between failures (MTBF). Based on the MTBF value, there ex- ist certification processes that perform functional hazard analysis (FHA) and risk mitigation [FA]] (202I). To perform similar processes for DNN-based systems, one needs to assess the likelihood of failure (i.e., an adversarial input), but this essential ability still missing.
In this paper, we attempt to address this crucial gap by introducing a novel, straightforward, and scalable statistical method for measuring the probability that a DNN classifier will misclassify in- puts. The method, which we term Robustness Measurement and Assessment (RoMA), assumes that the misclassification of the neural network is not due to a malicious attack, but rather due to random perturbations of the input, which occur naturally as part of the system's 's operation. Under this as- sumption, RoMA can be used to measure the model's robustness to aandoll-produced adversarial inputs. The proposed method is useful for several applications, such as comparing the robustness of multiple models and picking the best one, or for checking the impact of various configurable parameters (e.g., the number of training epochs, or the magnitude of the input perturbation) on the resulting model's robustness.
RoMA is method for estimating rare events in a large population - in case, adversarial inputs within a space of inputs that are generally correctly classified. The method relies on the properties of normal distributions. If it is known that the rare events (adversarial inputs) are distributed normally, it is usually sufficient to sample few thousands random points, use them to draw Gaussian curve, and then use the normal distribution function evaluate the probability of a rare event in the popu- lation. Unfortunately, adversarial inputs are often not distributed normally in the population; and to overcome this difficulty, RoMA first applies a statistical transformation, after which the distribution often becomes normal and can be analyzed.
At a high level, RoMA consists of the following steps:
1. for an arbitrary input point Fo. we randomly sample n perturbations of æn (usually, a few thousands), obtaining a set of perturbed inputs (
2. we evaluate the DNN on each znj obtaining the corresponding outputs (vb..... y6 }:
3. for each yá that is classified incorrectly (that is, which indicates a labeling different than that of æo), we collect the maximal entry cº of yú- This value is the confidence score assigned by the DNN to the incorrect label;
4. if needed, we apply a statistical transformation called Box-Cox Box & Cox 1982) to nor- malize the distribution of e² values; and
5. if the distribution is now normal, we use the properties of the normal distribution function to calculate the probability for an adversarial input around zp.
A key component in our method is the Box-Cox statistical power transformation, which is a well- established method, and which does not pose any restrictions on the DNN in question (e.g.. Lipschitz continuity, certain kinds of activation functions, or a specific network topology). Further, the method does not require access to the network's topology or weights, and is thus applicable to black-box DNNs.
We implemented our method as proof-of-concept tool, and evaluated it on standard DNN architec- tures: 16 Simonyan & Zisserman (2015). Resnet He et al. (2016). and Densenet Huang et al. (2017a). all trained on the CIFARIO data set set Krizhessyy et al. (2009). We used RoMA to compare the robustness of these DNN models, and found, as expected, that a higher number of epochs leads to higher robustness score. Additionally, we used RoMA to measure how the allowed magnitude of perturbation affects the robustness of DNN model, and computed the rate at which robustness deteriorates as the perturbation level increases. Finally, using RoMA, we found that the categorial robustness score of DNN (i.e., the robustness score of inputs labeled a particular category) varies significantly among the different categories. This finding could allow users and regulators to specify different acceptable robustness thresholds for each target category, instead of single global threshold, which may be more difficult to obtain.
To summarize, main contributions are:
2

---

Introducing RoMA: a new method for measuring the robustness of DNN model. The new method scalable and can run on black- box DNNs.
Comparing the robustness of multiple sate-of-the-ar DNN models.
Using RoMA measure the effect of perturbation level has on the robustness of the DNN model.
Introducing the notion of categorial robustness, which is measure of robustness computed for each target label.
Related work. The topic of adversarial robustness has been studied extensively. Some notable approaches for estimating a model's robustness include:
Statistical approaches that evaluate the probability of encountering an adversarial input in the population. In a recent paper Huang et al.] (202T). Huang et al. use random sampling, which is similar in spirit to RoMA, but which assumes that the DNN's adversarial inputs are distributed normally. As demonstrate later, this is often not the case. In another paper, Webb at al. Webb et al. (2018) use a sampling method called multi-level splitring, which provides no formal guarantee of the DNN's robustness. Mangal et al. Mangal et al. (2019) use importance sampling, which might be biased due to lack of sampling in areas of the population that are deemed unimportant. Moreover, this approach assumes that the network's 's output is Lipschitz continuous, which limits its applicability. In contrast, RoMA requires no hipschitz-continuii assumptions, does not assume a-priori that the adversarial inputs are distributed normally, and provides rigorous robustness guarantees.
Lipschitz-comtinuuus
Formal-verificttion based approaches Katz et al. (2017): y Wang et al. (2018); Jacoby et al. (2020), which allow for computing a DNN's exact adversarial robustness score. These approaches typically convert the problem into a constraint satisfiability problem, and then apply search and deduction procedures solve efficiently. However, verification- baased approaches afford only limited scalability, and operate strictly on white-box DNNs. In contrast, RoMA scalable technique, and can operate on black-box DNNs.
Approaches for computing an estimate bound on the probability that a classifier's margin function exceeds a given value Weng et al. Anderson & Sojoudi 2020); Dvvjotham et al. (2018). These analyses focus on the worst-case behavior, thus producing bounds that might be inadequate for regulatory certification. In contrast, RoMA focuses on the average case, which is more realistic many application domains.
Outline. We begin with some needed background on adversarial robustness in Section 2 We then describe our proposed method for measuring adversarial robustness in Section 3 followed by description of our evaluation setup in Section In Section we summarize and diseuss our results.
BACKGROUND
Neural Network. neural network N is function N : R + RT which maps a real-valued input vector æ E RD to real-valued output vector y E Rm. For classification networks, which is our subject matter here, the entries of y represent confidence scores for m possible classes; and classified label if y's l'th entry has the highest score; i.c., if arg max(N(x)) = L. We use c(x) to denote this highest score, i.c. c(x) = max(N(x)) (x)))
Local Adversarial Robustness. The local adversarial robustness of DNN is measure of how silient that network is against adversarial perturbations to specific inputs. Intuitively. a network with high robustness behaves "smoothly", i.e., small perturbations to its input do not cause significant spikes in its output. More formally [Bastani et al. (2016): Huang et al. (017b):
Definition 1. A DNN is e-locally-robus at input point &o iff
Va. x zulloo I e = arg maN())) arg max (axo))
Intuitively, Definition I states that for input vector æ. which is at distance at most from fixed input Zn. the network assigns to æ the same label that it assigns to wn (for simplicity, we use here the To norm, but other metrics could also be used). When network is not e-local-robust point
3

---

To, there exists point that is at distance of at most from 2o, which is misclassified; this is called an adversarial input. In this context, local refers to the fact that Dy is fixed. Larger values of imply larger distance from zo. and hence stronger robustness guarantee if the property holds. Intuitively, in a DNN for image classification that is e-local-robust, small perturbations to 2n. i.., perturbations small that human would fail detect them, should not result in a change of predicted class.
Distinct Adversarial Robustness. Classification networks assign value to each output label, which expresses the level of confidence in that label; and the label with the highest confidence score wins. We are interested in an adversarial imput æ only if it clearly misclassified, i.e., if x's assigned label received a significantly higher confidence score than that of the label assigned to æo- For example, arg max N (mr)) F arg max max(N (x)) but c(2r))
0.41 and c(x)) = 0.42, x is not distinctly an adversarial input; whereas case where c(x) = 0.8 is clearly much more relevant. We refer to adversarial inputs whose assigned confidence value is sufficiently high (greater than some 5) as distinct adversarial inputs, and refine Definition Io only consider them, follows:
Definition 2. A DNN N is (c, 8)-locally- at input point To. iff
Vx. ||=0
moll> 6 (arg max (Nx)) = arg max (Nxx((((x))) V c(x) 5)
THE PROPOSED METHOD
3.1 PROBARILISTIC ROBUSTNESS
Definitions and which are fairly common, are geared for malicious adversary: they are con- cerned with the existence of an adversarial input, implicitly assuming the adversary will be suc- cessful in finding it if such an input exists. Here, we focus instead on a random setting, where perturbations can occur naturally, and are not necessarily malicious. We argue that this setting is more realistic for widely-deployed systems, such medical devices, aerospace, and trains, which are expected to operate at a large scale for prolonged period, and are more likely to randomly en- counter adversarial inputs than those crafted by malicious adversary. In this case, natural method for assessing a model's 's robustness is to randomly perform input perturbations, and check whether these result in adversarial inputs, i.c., cause misclassification.
Towards this end, we propose to use a probabilistic measure of robustness, which signifies the probability of randomly perturbing æg into an input which is not a distinct adversarial input:
Definition 3. The (8, waabaabblistic-lla---obbeeee viated plrs,e (N, 2o), is defined as:
score of DNN N at input point xo abbre-
pirs, (N, xx) & Pa:ll= Po:l:==0llmms [arg max(N(x:)) arg max((((x))) V Ve(z) < 6)]
The key point is that probabilistic robustness, as defined in Definition is scalar value; the closer this value is to 1, the less likely it is a random perturbation to 2o would produce an adversarial input. This is in contrast to Definitions [ and which are Boolean in nature. We also note that the probability value in Definition 3 can be computed with respect to values of æ drawn according to any input distribution of interest. For simplicity, unless otherwise stated, we will assume that x is drawn uniformly at random.
To assess a DNN's probabilistic robustness using Definition 3 we need to measure how many inputs are in the c-ball around 2g are adversarial. Estimating this measure directly, e.g., with the Monte Carlo or Bernoulli Hammersley (013)) methods, is not feasible due to the typical extreme sparsity of adversarial inputs, and the large number of samples required to achieve reasonable accuracy Webb et al. (2018). Thus, we require a different statistical approach to obtain this measure, using only a small number of samples. We next propose such an approach.
3.2 THE NORMAL DISTRIBUTION
Our goal is to measure the probability of randomly encountering an adversarial input, by looking a finite set of perturbed samples around xy Each sample is created by perturbing the input features of ao. so that the overall perturbation size does not exceed e; and its adversariality is determined
4

---

according to Definition 2 The main question is how to extrapolate conclusion regarding the whole population from these samples.
The normal distribution is a useful notion in this context: if we know that the perturbed imputs distributed normally, straightforward to obtain such a measure, even adversarial inputs are scarce.
To illustrate this point, we trained a VGG16 DNN model, and examined an arbitrary point zo, classi- fied as some label lo. from its training set. We randomly generated 10,000 perturbed images from æn, and ran them through the DNN. For each output vector obtained this way, we collected the highest confidence score assigned to any label other than lo: these plotted the blue histogram in Fig- ure I The green curve represents the normal distribution using the average and standard deviation of the raw data. As the figure shows, the data is normally distributed: and this claim is supported by running a "goodness-of-fii ffttit" test (explained later). Our goal is to compute the probability of fresh, aunomll-perturben
input to be misclassified, with confidence score that exceeds given 8, say 0.6. For data distributed normally, as in this case, we begin with calculating the statistical standard score (Z-Score), which is the number standard deviations by which the value of raw score (in our case, d) exceeds the mean value. Once the Z-score is obtained, we can use the normal distribution function, which computes the correct probability of the event using the Gaussian function. In our case, get Xo ~ [p 0.473, Σ 0.053²). where μ is the average score and Σ is the variance. 2.396, where is the standard deviation. Finally,
The resulting Z-score is 5-1 0.6-0.473 get:
0.053
plro.c,o.o((N,20)
1 NormalDistriutiionn--soor)
= 1 NormalDistributionn
1
2.369
2
1
c
dt
v 2m
=11 0.008288) 0.991712
Thus, in this example, a perturbed image drawn uniformly at random has a chance of 0.82% of constituting an adversarial input.
0.22 0.5
0.6
Figure 1: dirsally adversarial inputs.
Of course, given data obtained empirically, as in our case, we need a way to determine whether the data is distributed normally before applying the aforementioned approach. A goodness-of test is a procedure for determining whether set of n samples can be considered as drawn from specified distribution. A common goodness-of-fit test for the normal distribution is the Anderson- Darlin Anderson (201T) test, which is used by widespread statistical, commercial applications such SPSS IBM (2001) and JMP SAS (200T). Usually, few hundred samples to prove normal distribution with the Anderson-Darlin test.
than enough
3.3 THE BOx-CoX TRANSFORMATION
Unfortunately, most often the adversarial inputs around æp are not normally distributed, and so the aforementioned approach does not immediately apply. For example, in our VGG16 model, out of the 10,000 points in the training set, only 1131 had normally-distributed distribute adversarial inputs (as determined by the Anderson-Darlin test). Figure 2a illustrates the abnormal distribution for of these input points, where we consequently cannot use the normal distribution function to estimate the probability of adversarial inputs in the population.
5

---

006
008
0.1
0:12
0.14
0.18
018
02
004 -338
0.36
03c
032
03
028
0.28
(a) Adversarial inputs that are not (b) The now normal distribution, normally distributed.
after applying Box-Cox.
Figure 2: Adversarial imputs that are initially not normally distributed.
The strategy that we propose for handling abnormal distributions of data like the one depicted in Figure 2 to apply statistical transformations, capable of converting the given distribution into normal one. This practice is widely used by statisticians, and is common in standard statistical ap- plications such as SPSS and JMP. There are two main transformations used normalize probability distributions: Box-Cox Box & Cox (1982) and Yeo-Johnson Yeo & Johnson (2000). Here, we focus on the Box-Cox power transformation, which preferred for distributions of positive values (as in our case). Box-Cox is continuous, piecewise-lineat power transform function, parameterized by real-valued X, defined as follows:
Definition 4. The Box-Coxx power transformation of imput x is:
0 In(x) =fA=
Box-Coxx(2)
Selecting the parameter À in often performed using the maximum--lkkllihod estimation (MLE) method. In this method, À is chosen by heuristically maximizing the goodness-of-fit score of the re- sulting distribution, so that it most closely resembles normal distribution Rossi 2018 Asar al.
2017
Figure 2b depicts the distribution of the data from Figure 2a after applying the Box-Cox transfor- mation, with an automatically calculated X 0.257 value. As the figure shows, the data is now N( -0.33, Σ = 0.020 this confirmed by the Anderson-Darlin test with confidence score of over 99%. Following the Cox-Box Box transformation, we can now calcu- late the Z-Score, which gives 7.45; and the corresponding pir score, which turns out to be extremely high: 1 4.66 10-14
normally distributed:
3.4 THE ROMA CERTIFICATION ALGORITHM
Based on the previous sections, our algorithm for computing pir scores is given Algorithm
Algorithm Compute Probabilistic Local Robustness(O, n, N,, Do. D)
1: for = to n do
2:
zi -CCRAATTPPTUURRRPPNNTTT e, DD)
3: confidence(i] + PREDICT xx))
4: if
, NORMAL then
5:
confidence + Box-Coxoi Cox(coofiddenc))
6: if ANDERSON-DARLLICccfiiien
NORMAL then
7:
return FAIL.
8: avg +- AVERAGE((ooffddecee)
9: std < STDDEV(confidenee)
z-score Z- -)
11: return - NORM.LDDISTRIBUTTONESSSOS
The inputs to the algorithm are: 8, the confidence level distinct adversarial input; e, the maxi- mum perturbation that be added Sot n, the number of perturbed samples generate around

---

To; N, the neural network, and Zg. the input point whose pir score is being computed: and D, the distribution from which adversarial inputs are drawn. The algorithm starts by generating n per- turbed imputs of the provided Zn. each drawn according to the provided distribution D and with perturbation that does not exceed e (lines ( Next, on lines 7 it confirms that the samples' confidence values distribute normally, possibly applying the Box- Cox transformation if needed. Fi- nally, on lines [HIT the algorithm calculates the probability for a random adversarial input using the properties of the normal distribution, and returns the computed plrs,e (IN, æo) score.
Soundness and Completeness. Algorithm depends on the distribution of adversarial inputs being normal. If the distribution is initially not normal, the algorithm attempts normalize using the -ox transformation. The Anderson Maderson-Darlin goodness of fit tests ensure that the algorithm will not treat an abnormal distribution as normal one, and thus guarantee the soundness of the computed plr scores.
The algorithm's completeness depends on its ability to always obtain normal distribution. our evaluation (described later) demonstrates, the Box-Cox transformation can indeed accomplish this very often. However, the transformation might fail in producing normal distribution; this will be identified by the Inderson-Darlin test, and our algorithm will stops with failure notice in such cases. In that sense, Algorithm [ is incomplete. In practice, failure notices by the algorithm can sometimes be circumvented by increasing the sample size, or by evaluating the robustness of other input points.
In evaluation, observed that the success of Box-Cox often depends the value of An analysis of the results indicated that very small or very large c values more often led to failures, whereas mid-range values more often led success. We speculate that this is because very small values lead to almost no adversarial inputs - i.e., the resulting distribution of adversarial inputs is close to uniform, and is consequently impossible normalize. A similar situation occurs for very large values, which introduce large number of adversarial inputs distributed uniformly. We argue that the remaining, mid-range values of e are the more relevant ones. Adding better support for cases where Box-Cor fails, for example by using additional statistical transformations, remains a work in progress.
EVALUATION
For evaluation purposes, we implemented A proof-of-concept tool. The tool is written in Python 30,.10 and uses the TensorFlow 2.5 and Keras 2.4 frameworks, For our models, we used Resnet- 10. Resnet-100, VGG16- 10, VGG16 and Densenet, as described in Table [ all trained using the CIFAR10 data set. All experiments mentioned in this section were run using the Google Colab Pro environment, with an NVIDIA-SM 470. 74 GPU and a single-core Intel(R) Xeon(R) CPU @ 2.20GHz. The code for the tool and experiments is (anonymously) available online Annonymized 20211, and will be publicly released with the final version of this paper.
Table
Neural network models' properties
Model Name
Base Model
Reference
# Epochs
Accuracy
Loss
Resnet- 10
Resnet
He ct al.
2016
10
0.72
1.07
Resenet- 100
Resnet
He ct
2016
100
0.91
0.4456
VGG16-10
VGG16
Simonyan Zisserman
2015
10
0.73
0.8329
VGG16-200
VGG16
Simonyan & Zisserman
2015
200
0.76
2,7082
Densenet
Densnct
Huang et al. 2017a
200
0.93
0.5335
4.1 EXPERIMENT 1: MEASURING ROBUSTNESS' SENSITIVITY TO PERTURBATION SIZE
By our notion of robustness given in Definition 3 it is likely that the plrs, (N, .o) score decreases as é increases. For our first experiment, we set out to measure the rate of this decrease. Using our Densenet model, repeatedly invoked Algorithm to compute pir scores for increasing values of e. For our on. we arbitrarily selected the first 100 images from the CIFARIO 10 test set, and measured the average robustness of the images for each The averaged results (depicted in Figure indicate

---

almost linear correlation between and the robustness score. This result is supported by earlier findings Webb et al. 2018).
D-1316631
13-1-20-111
618
211
6:16
Value
Figure 3: Average probabilistic robustness as a function of
The experiment conducted by running Algorithm I with 5 = 0.6 each of the 100 input im- ages, and generating 10,000 perturbed samples for each image. Running the algorithm took less than 7 seconds per sample, and under one hour for the entire experiment. We note here that Algorithm naturally lends itself to additional parallelization, as each perturbed input can be evaluated indepen- dently of the others; we leave adding these capabilities our proof-of -f-nccceppt implementation for future work.
4.2 EXPERIMENT 2: COMPARING ROBUSTNESS ACROSS MODELS
The ability to efficiently compute pir scores allows us to compare multiple models based on their robustness. Using Algorithm computed the pir scores for each of our 5 models, averaged over the first 100 images from the CIFARI0 10 test set. We arbitrarily set = 0.04 and 5 = 0.6. The average pir scores appear in Figure 4 and indicate that higher number of epochs leads higher robustness score.
0-548500
B 0-G08600
Model
Figure 4: Robustness comparation between models
Running Algorithm D as part of this experiment with 10,000 perturbation samples for each image took between five seconds (with the VGG16- model) to seven seconds (with the Densenet model), for each sample. The whole experiment lasted less than one hour.
4.3 EXPERIMENT 3: CATEGORIAL ROBUSTNESS
For our third experiment, we focused categorial robusiness, where we first measure the robustness of inputs labeled a specific category, and then compare the robustness scores across categories. We Algorithm on VGG16- 10 model, for all 10,000 CIFARIO test set images. We calculated the average robustness score for each the ten categories separately. Table 2 depicts the results.
The results expose an interesting insight, namely the high variability in robustness between the different categories. For example, the average robustness score for inputs classified as Bird is more than ten times that of inputs classified Dog. We applied T-rest and binomial test, which a well-established statistical tools for measuring the difference between two sets values, to the Bird and Frog categories. The tests produced similarity score of less than 0.3%, indicating that the
8

---

Table 2: Pivot analysis of all samples in the
Category
# Samples
Robustness Avg
Robustness variance
Airplane
1000
2.8894 10-1
1.1362
10-3
Automotive
1000
8.3363 10-
1.0430
10-6
Bird
1000
1 1.1877
10
1.5064
10
Cat
1000
1 7.0537 10
7.8409
10
Deer
1000
1.3545
100-
1.9256
10-6
Dog
1000
1 7.1440
10-6
3.8810 10-9
Frog
1000
1 6.6058
10
6.0929 10
Horse
1000
2.5776
7.9067 10
Ship
1000
1 1.6059
10-4
3.0538
10-6
Truck
1000
1 2.7746 10-1
6.8932
10
two categories are indeed distinctly different. From this, we draw the important conclusion that the per-category robustness of models can be far from uniform.
To the best of our knowledge, this is the first time such extreme differences in categorial robustness have been reported; and we believe this insight could affect DNN certification efforts, by allowing engineers to require separate robustness thresholds for different categories. For example, in DNN for traffic signs recognition, user might require high robustness score for stop sign and a low robustness score for parking sign.
Running Algorithm ! on the entire CIFATIO test set with one thousand samples per input, using 0.6 and 0.04, took under 71 minutes.
Completeness Level. We observed that the completeness rate (the number of complete samples divided by the number of samples) varies between the models: in VGG16-10 and VGG-200, the completeness rate was 77% and 26% respectively; in Resnet-10 and Resnet-100 the rate was 70% and 57% respectively; and in Densenet, the rate was 30%, In order to improve these rates, the steps described in Section 3.4 can be applied.
SUMMARY AND DISCUSSION
5.1 SUMMARY
In this paper, we introduced RoMA novel statistical and scalable method for measuring the probabilistic local robustness black-box DNN model. We demonstrated RoMA's applicability in several aspects and multiple common DNN models. The key advantages of RoMA over isting methods are: (i) it uses straightforward and intuitive statistical method for measuring DNN robustness; (ii) it is scalable; (iii) it works on black-box DNN models, and makes no assumptions such as Lipschitz continuity or piecewise-lineee constraints; and (iv) the method is quick in compar- ison formal verification methods and other methods that require hours or more for analyzing local robustness Wang et al.) (2018). The key limitation of our approach is that it depends on the normal distribution of the adversarial inputs, and will fail whenever the Box-Cor transformation does not normalize this distribution.
The pir scores computed by RoMA indicate the risk of using DNN model, and can allow regulatory agencies conduct risk mitigation procedures: common practice for integrating sub-systems with a known MTBF into safety-critical systems. The ability to perform risk and robustness assessment an important step towards using DNN models in the world of safety-criticll applications, such as medical devices, UAVs, automotive, and others. We believe that the newly defined notion of categorial robustness could also play key role in this endeavor.
Moving forward, we intend to: (i) evaluate our tool on additional norms, beyond L and (ii) better characterize the cases where the Box-Cox transformation fails, and search for other statistical tools can succeed those cases; and (iii) improve the scalability of our tool by adding parallelization capabilities.

---

Ethics Statement This paper does not raise any ethical concerns.
Reproducibility Statement All the details required to reproduce the results for this paper are de- tailed in the evaluation part of this paper in section 4. The code for the tool and experiments is (anonymously) available online Annonymized (202I). and will be publicly released with the final version of this paper.
REFERENCES
Brendon G. Anderson and Somayeh Sojoudi. Data-Driven Assessment Of Deep Neural Networks With Random Input Uncertainty, arXiv preprint arXiv:2010.01177, 2020. ISSN 23318422. URL
http://arx.v.orrr/bbs//// 01111
Theodore W. Anderson. Anderson-Darling Tests Of Goodness-of-Fit. Int. Encycl. Stat. Sci., 1(2):
52-54, 2011. doi: 10.1007/978-36642-4499---2
Annonymized. RoMA: Code and Experiments, 2021.
https://drive.googl. com/
EKpH4V28sbx
Özgür Asar, Ozlem Ilk, and Osman Dag. Estimating Box-Cox Power Transformation Parameter Via Goodness-of-Fit Tests. Communications coutiticx-Simulatiin and Computation, 46(1): 91-105, 2017.
Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi. Measuring Neural Net Robustness With Constraints. Adv. Neural Inf. Process. Syst., pp. 2621-2629, 2016. ISSN 10495258.
G. E. Box and D. R. Cox. An Analysis Of Transformations Revisited, Rebutted. J. Am. Stat. Assoc., 77(377):209-211 1982. ISSN 1537274X. doi: 10. 10.1080/01621455 118.16111...991117777711717
Nicholas Carlini and David Wagner. Towards Evaluating The Robustness Of Neural Networks. In Proc. 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017.
Krishnamurthy Dvijotham, Marta Garnelo, Alhussein Fawzi, and Pushmeet Kohli. Verification of Deep Probabilistic Models. arXiv preprint arXiv:1812.02795, 2018. URL http://arxiv
org/abs aba 1812. 02795
European Union Aviation Safety Agency. A Human-Centra Approach To AI
Artificial Aviation,
Intelligence
Roadmap: URL
In
2020.
https: http:://wwww... easa www..a.....a..aac............... w and events/ newws/
easa -tt.aaaaiiiiiill-teelleeeeeooe roadmap 10 1a-tmmapp----upihhsheeeee
FAA. System Safety Handbook, 2021.
seliciesshandboooo httpp//wwww.ooooaaaaaaaaeeeeeeee hat.aaaa////eeeaaeeeoo/aaaaaaaan httpe/enennttssaaaoooooooo/ handbook/ accessed 2020-09-14.
J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Haressinng Adversarial Examples. 3rd Int. Conf. Learn. Represent. ICLR 2015 Conf. Track Proc., pp. 1-11, dec 2014.
URL httptttp///xii..att/ www.oo/gg/.//.....o//
John Hammersley. Monte Carlo Methods. Springer Science & Business Media, 2013.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning For limage Recognition. In Proc. Proceedings of the IEEE conference on computer vision and pattern recog- nition(IEEEE,, pp. 770-778, 2016.
Chengqiang Huang, , Zheng Hu, Xiaowei Huang, and Ke Pei. Statistical Certification of Acceptable Robustness for Neural Networks. Artificial Neural Networks and Machine Learning ICANN 2021, (377))777) 2021. ISSN 161 1611-3349.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely Connected Convolutional Networks. Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, 2017-January:2261- 2017a. doi: 10. 109//CPR..2117.444
10

---

Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety Verification Of Deep Neural Networks. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioin- formatics), 10426 LNCS:3-28 2017b. ISSN 16113349, doi: 10.10779978-33319-6333
1.
IBM. IBM SPSS Website, 2001. httphhtps://// //ww ibm. http://ooopoocccc//pppeoaaa sttp:aaacccssspppsssaaaaaasat.ss
accessed 2021-09-14.
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial Examples Are Not Bugs, They Are Features. arXiv, 2019. ISSN 23318422.
Y. Jacoby, C. Barrett, and G. Katz. Verifying Recurrent Neural Networks using Invariant Inference. In Proc. 18th Int. Symposium Automated Technology for Verification and Analysis (ATVA), pp- 57-74, 2020.
Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Reluplex: An Efficient Smt Solver For Verifying Deep Neural Networks. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), 10426 LNCS:97-117, 2017. ISSN 16113349. doi:
Alex Krizhevsky, Geoffrey Hinton, et Learning Multiple Layers of Features from Tiny Images. 2009.
Ravi Mangal, Aditya V Nori, and Alessandro Orso. Robustness of Neural Networks: A Probabilistic and Practical Approach. In Proc. 2019 IEEF/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), pp. 93 96. IREE, 2019.
Ana Pereira and Carsten Thomas. Challenges of Machine Learning Applied to Lafety-Critical Cyber-Physicaa Systems. Machine Learning and Knowledge Extraction, 2(4))579-608 2020.
Richard Rossi. Mathematical Statistics: Introduction to Likelihood Based Inference. John Wiley & Sons, 2018
SAS. Sas jmp website, 2001. wttt:://wwww...nn.comoooooooooss accessed 2021-09- 14.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks For Large-Scale Im- age Recognition. In Proc. 3rd Int. Conf. Learn. Represent. ICLR 2015 Conf. Track (ICL,( pp. 1-14, 2015.
Guillaume Vidot, Christophe Gabreau, Ileana Ober, and lulian Ober. Certification of Embedded Systems Based on Machine Learning: A Survey, arXiv preprint arXiv:2106.07221, 1(1):1-23, 2021. URL httttttp///axitta
httgg//bb/21066. C7221
Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Formal Security Anal- ysis of Neural Networks Using Symbolic Intervals. Proc. 27th USENIX Secur. Symp., pp. 1599-
1614, 2018.
Stefan Webb, Tom Rainforth, Yee Whye Teh, and M. Pawan Kumar. A Statistical Approach to Assessing Neural Network Robustness. arXiv preprint 1811 1.07009, pp. 1-15, 2018. ISSN 23318422.
Lily Weng, Pin-Yu Chen, Lam Nguyen, Mark Squillante, Akhilan Boopathy, Ivan Oseledets, and Luca Daniel. PROVEN: Verifying Robustness Neural Networks With a Probabilistic Approach. In Proc. International Conference on Machine Caturning(PMLRR pp. 6727-6736, 2019.
Kw-K Yeo and Richard A Johnson. A New Family of Power Transformations Improve Nor- mality or Symmetry. Biometrika, 87(4):954-995 959, 2000.
11


Reviews:
[{"review_id": "rxrKnq9vvBg", "reviewer": "Reviewer_JNVz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper appears quite novel and interesting, I did not feel the text reflected the abstract as a whole. The paper was quite opaque, and I remained unconvinced that the method if applied in practice would provide useful insights. \n\n- Needs proofreading, the repeated citations in the related work section is distracting. \n- Additional motivation for Definition 3 is warranted. Can the authors present a practical motivating example, which does, or could believably, occur in reality?\n- Grammatical errors: \"we need to measure how many inputs are in the $\\epsilon$-ball around $x_0$ are adversarial\"\n- \"To illustrate this point, we trained a VGG16 DNN model,\" on what dataset? with what training parameters? These details should at least be put in the appendix. How exactly are the perturbations generated, what attack, what attack parameters?\n- While I appreciate the acknowledgement that most often the adversarial inputs around a datapoint are not normally distributed, the example in Figure (2a) is very close to normal. I would hypothesise that a wide variety of distributions are possible given different parameterisations of  dataset, downstream task, label distribution, architecture, training parameters altogether. On ImageNet, if the perturbations are generated by non-targeted attacks, it is significantly easier to induce misclassification via related categories (different types of dogs) than classes like \"frogs\". Altogether, I am unconvinced that the figures as presented correspond to expected scenarios in reality, and even if so, I expect the variance across datasets and training parameters to be high. \n- Does the Box-Cox transformation preserve the statistics of the distribution, what statistics are and aren't preserved? From a functional perspective, can the transformation be considered bijective, can one reverse the Box-Cox transformation and recover the original distribution. One can transform one distribution to an entirely different distribution, but that isn't interesting if the transformed distribution does not reflect the original distribution in any unique way? This should be clarified in the text. ", "summary_of_the_paper": "Present a method to measure the expected robustness of a neural network model, by determining  the probability that a random input perturbation might cause misclassification, providing formal guarantees regarding the expected frequency of errors that a trained model will encounter after deployment. The method can be applied black-box. Applied the approach to compare the robustness of different models, and measure how a model’s robustness is affected by the magnitude of input perturbation. ", "summary_of_the_review": "Recommend rejection as the paper did not demonstrate its utility in practice, nor corroborated their abstract in a clear manner. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nsNyaZYTyP", "reviewer": "Reviewer_Y2p2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\n \n1. The paper tries to address an important problem in the field of adversarial machine learning: how to evaluate \nrobustness of deep neural networks efficiently, which is of great importance and has wide application.\n\n \n2. Experiments show that the proposed statistic decreases as epsilon (the distortion bound) increases, which shows \nthat the proposed statistic behaves expectedly. Since larger epsilon usually leads to weaker robustness (as the \nprobability of adversarial examples existing improves), a reasonable statistic that measures robustness should \ndecrease as epsilon increases.\n\n \n3. It's nice to see a comparison of robustness of different architectures. \nCode is provided for reproducing the experimental results. \n\n \n##########################################################################\n\nCons: \n\n1. Definition 3 is problematic. Based on the sentence before the definition \"..., which signifies the probability of \nrandomly perturbing $x_0$ into an input which is not a distinct adversarial input\", the robustness score should \nbe $1-$probability of perturbed input being an adversarial input. Thus, the definition of score should be:\n\\begin{align*}\nplr_{\\delta, \\epsilon}(N,x_0):=1-P_{x:\\|x-x_0\\|\\le\\epsilon}[argmax(N(x))\\ne argmax(N(x_0)) \\bigcap c(x)>\\delta].\n\\end{align*}\nThe definition in the paper does not satisfy the claim \"the closer this value is to 1, the less likely it is a random \nperturbation to $x_0$ would produce an adversarial input.\" Based on the original definition, the closer the value \nis to 1, the closer the value of $P_{x:\\|x-x_0\\|\\le\\epsilon}[argmax(N(x))=argmax(N(x_0)) \\bigcup c(x)<\\delta]$ is to 0, \nwhich makes the probability of robustness close to 0.\n\n \n2. The proposed method section is not clear. Based on the description of z-score calculation, my understanding \nis that the method is trying to measure $1-P(c(x)>\\delta)$, which is $1-$the probability of the maximum confidence \nscore of perturbed input greater than a preset threshold. Here're some problems of it:\n\n- In this section, it says that $x_0\\sim N(\\mu=0.473, \\Sigma=0.053^2)$. Isn't $x_0$ the original input? Why suddenly \nit follows a normal distribution? Based on the context, I guess it represents the distribution of maximum confidence \nscores of perturbed inputs. Another possibility is that it represents the distribution of maximum confidence scores of \nadversarial examples out of the perturbed inputs. If my understanding is wrong, please correct me. \n\n- If it represents the distribution of maximum confidence scores of perturbed inputs, why $1-P(c(x)>\\delta)$ can be used to measure robustness? $P(c(x)>\\delta)$ does not represent the probability of $x$ being adversarial. The probability of adversarial should be: $P[argmax(N(x))\\ne argmax(N(x_0))\\bigcap c(x)>\\delta]$. Where is the requirement of $argmax(N(x))\\ne argmax(N(x_0))$ reflected in the method?\n\n- Even if the authors compute the empirical distributions of maximum confidence scores based on only adversarial \nexamples out of the perturbed samples, it's only a conditional probability and the component of $P[argmax(N(x))\\ne argmax(N(x_0))]$ \nis still missing. Besides, estimating $P[argmax(N(x))\\ne argmax(N(x_0))]$ is a difficult problem. The probability cannot simply be estimated by number of adversarial examples in the perturbed inputs divided by total number of perturbed inputs, because we don't know how adversarial samples are distributed in the input space.\n\n3. For experiment 4.1, why not report plr scores of $\\epsilon<0.1$? On CIFAR10, a white-box attack like PGD with perturbation budget $0.03$ can reduce the accuracy to $0$. So, it would be better to report the plr scores for $\\epsilon<0.1$ as well.\n\n4. The paper uses \"adversarial inputs\" many times. In the abstract, \"adversarial inputs\" is defined as: \"small input perturbations that cause the model to produce erroneous outputs.\" However, later on, it is used to describe maximum confidence scores of adversarial examples. For example, in Figure 1, the caption is \"Normally-distributed adversarial inputs\". Based on the description, the histogram is generated with highest confidence scores assigned to any label other than original label. So, a better caption can be \"Histogram of highest confidence scores of adversarial examples of 10,000 perturbed images\". This happens several times, making the definition of adversarial inputs very confusing.\n\n##########################################################################\n\nMinor:\n\n1. \"CONFERENCE SUBMISSION\" should be removed from the title.\n\n2. Page 5, when calculating $plr_{0.6, 0.04}(N,x_0)$, the value $0.008288$ implies that upper tail value is calculated, but the integration represents lower tail.", "summary_of_the_paper": "Summary:\n\nThe paper introduces a statistical method to measure the robustness of deep neural networks. The novel \npart of the method is that it's designed to measure the probability of random points near an input being \nadversarial, instead of probability of adversarial examples existing in the vicinity of an input. To measure it, \nthe authors proposes to use Box-cox transformation to transform distribution of confidence scores to normal, \nthen calculate the probability based on it.\n\n", "summary_of_the_review": "Reasons for score: \n\n \nOverall, I vote for rejection. I like the idea of measuring the probability of random inputs being adversarial \ninstead of finding the extreme case. However, I think whether the proposed method is measuring such \nprobability is questionable. Some notations and terms are unclear, so it is possible that I missed some \ncomponents. If the authors can address my major concerns in the rebuttal period, I'm ok to increase \nmy score and accept the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "hw6uhS2Xwa2", "reviewer": "Reviewer_FezE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+ The paper addresses the very important challenge of characterizing robustness of deep learning models.\n\n+ The described approach is simple - transform the neighborhood confidence c_i values to a normal distribution  and then estimate the z-score.\n\nWeaknesses:\n\n- Some observations are rather obvious and well-known. Different classes exhibit different robustness. Confusion matrices have been used widely to understand these class-conditional error distributions. The reviewer is not convinced that \"categorical robustness\" is a new notion. \n\n- The decision of normalizing c^i on sampled input appears arbitrary. How do we decide what inputs to sample? The model is likely going to be robust in some part of the input space and not robust in another. Sampling inputs on which the perturbation is applied appears a crucial decision. Evaluating the model in just the neighborhood of the training data is not very useful. \n\n- Could authors provide more details on how lambda was selected? The paper states that \"Selecting the parameter λ in often performed using the maximum-likelihood estimation (MLE) method. In this method, λ is chosen by heuristically maximizing the goodness-of-fit score of the resulting distribution, so that it most closely resembles a normal distribution.\" This appears to describe the usual practice. In this paper how was λ selected? Was a specific tool used to implement this?\n\n- \"An analysis of the results indicated that very small or very large \u000f values more often led to failures, whereas mid-range values more often led to success. We speculate that this is because very small values lead to almost no adversarial inputs — i.e., the resulting distribution of adversarial inputs is close to uniform, and is consequently impossible to normalize. A similar situation occurs for very large \u000f values, which introduce a large number of adversarial inputs distributed uniformly\". Could authors clarify what is meant by uniformly distributed? It is not intuitively obvious why adversarial examples would be uniformly distributed in the input space. The decision of the model would be more sensitive with respect to some features (or pixels) than others. Also, given the observations that the technique only works on some mid-range values of epsilon. Could authors comment on how this limitation would impact its practical utility? \n\n- A minor suggestion - forward references such as \"explained later\" should be avoided. Putting a citation for standard textbook approaches such as goodness-of-fit tests would suffice. ", "summary_of_the_paper": "The paper presents a statistical method - Robustness Measurement and Assessment (RoMA) to measure the expected robustness of a neural network model. The robustness is defined as the probability that a random input perturbation causes an incorrect prediction. The presented approach is a blackbox approach. Different output labels are observed to exhibit different robustness values. \n\nThe basic premise of the paper is that the adversarial perturbations are not naturally normal, but a transformation (Box-Cox) can be applied to make them a normal distribution before applying statistical estimation techniques (Anderson-Darling test + z score). ", "summary_of_the_review": "The paper presents a simple approach to characterize robustness. Some limitations of the techniques need to be clarified for the reviewer to understand the contribution and limitations of the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bYCEwIL_Y0y", "reviewer": "Reviewer_uGd4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has several drawbacks, which are detailed as follows.\n\n1. There are several existing methods that can provide similar probabilistic robustness measures. Although the authors emphasize that the proposed method uses random sampling to evaluate robustness and obtain a statistical robustness score. Similar ideas are already widely used in the literature. For example, randomized smoothing [R1], which is one of the most widely used certification techniques, already provides probabilistic bounds and confidence intervals for local robustness assessment. Moreover, [Weng 2019] considered the problem setting of a random perturbation to a local data sample. The resulting bound is also similar to the proposed local sampling technique. Since the idea is similar and the authors did not compare to those existing methods, I don't see many insights from the proposed method compared to existing works.\n\n2. The experimental results are not convincing. First, the authors did not provide any empirical robustness results to compare with the robustness evaluations. How can one know the reported results are correct and meaningful? Without rigorous justification and empirical evidence, the results could be questionable. Second, the authors did not compare to existing methods, which makes it difficult to assess the quality of the proposed method.\n\n\n[R1] https://arxiv.org/abs/1902.02918\n   \n", "summary_of_the_paper": "This paper proposes RoMA, a robustness evaluation framework based on local sampling and probability computation.\n\nThe main contributions are:\n1. Proposal of the ($\\epsilon$,$\\delta$) local robustness score for assessing the probability of random local samples that have different predictions than a given data input with a $\\delta$-confined top-1 confidence.\n\n2. Use of Box-Cox transformation for input data to improve statistical estimation.\n\n3. The method can be implemented in a model-agnostic fashion.\n", "summary_of_the_review": "The current version is quite incomplete. It lacks comparisons and distinctions to prior arts for technical novelty, and it also lacks comparisons to empirical robustness and other baseline methods.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a technique to measure the *expected* robustness of a
neural network by measuring the probability random input perturbations will
cause the model to make a mistake.

The reviewers are not convinced by the results in this paper. The methods
are not carefully evaluated against prior work, and it is not exactly
clear what lesson one can draw from the resulting statistical evaluation.
The experimental setup is not clearly explained in several places, making the
paper difficult to fully follow.

Since the authors do not respond to the reviewer concerns, there was no
opportunity to address these concerns.
Decision: Reject