Network robustness as a mathematical property: training, evaluation and attack
Authors: Marco Casadio
Matthew L Daggitt
Ekaterina Komendantskaya
Wen Kokke
Robert Stewart
Venue: ICLR 2022 Submitted (2022)

Abstract: Neural networks are widely used in AI for their ability to detect general patterns in noisy data. Paradoxically, by default they are also known to not be particularly robust, i.e. moving a small distance in the input space can result in the network's output changing significantly.
Many methods for improving neural network robustness have been proposed recently. This growing body of research gave rise to numerous explicit or implicit notions of robustness.  Connections between these notions are often subtle, and a systematic comparison of these different definitions was lacking in the literature.
In this paper we attempt to address this gap by performing an in-depth comparison of the different definitions of robustness, by analysing their relationships, assumptions, interpretability and verifiability.
By abstracting robustness as a stand-alone mathematical property, we are able to show that, having a choice of several  definitions of robustness, one can combine them in a modular way when defining training modes, evaluation metrics, and attacks on neural networks.
We also perform experiments to compare the applicability and efficacy of different training methods for ensuring the network obeys these different definitions.

NETWORK ROBUSTNESS AS A MATHEMATICAL PROP- ERTY: TRAINING, EVALUATION AND ATTACK
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks are widely used to detect general patterns in noisy data, however they are also known to not be particularly robust, i.c. moving small distance in the imput space can change the network's output significantly. Recently, many methods for improving network robustness have been proposed and this growing body of research has given rise to numerous explicit or implicit notions ro- bustness. Connections between these notions are often subtle, and systematic comparison of these different definitions is missing in the literature.
In this paper attempt to address this gap by performing an in-depth comparison the different definitions of robustness by analysing their relationships, assump- tions, interpretability and verifiability. By viewing robustness mathematical property, we are able to show that, having a choice of several def- initions of robustness, one combine them in modular way when defining training modes, evaluation metrics, and attacks on neural networks. We also per- form experiments compare the applicability and efficacy of different training methods for ensuring the network obeys these different definitions.
stand-alone
INTRODUCTION
Safety and security are critical for some complex AI systems involving neural networks, yet they difficult ensure. The most famous instance of this problem guaranteeing robustness against adversarial altacks (Szegedy et al., 2014; Goodfellow et al., 2015). Intuitively, an image is e-ball robust if, when you move no more than e away from in the input space, the output does not change much, or alternatively, the classification decision that the network gives does not change. Adversarial robustness is property that even very accurate neural networks fail to satisfy.
The proposed solution is to (re)train the network with robustness specifically in mind. Such training can seen as way to impose a formal specification, and so may contribute to explainability as well as verification. This work considers four of the most prominent families of techniques:
1. Data augmentation (Shorten & Khoshgoftaar, 2019)
2. Adversarial training (Goodfellow et al., 2015; Madry et al., 2018)
3. Lipschitz robustness training (Anil et al., 2019; Pauli et al., 2021)
4. Training with logical constraints (Xu et al., 2018; Fischer et al., 2019)
The last technique, training with logical constraints, is a more general approach that can train for not just robustness, but wide range of constraints expressed in some logical language.
Although the first three families of methods all seek to represent the same high-level knowledge in the neural network, each technique seeks to optimise for subtly different definitions of robustness. We formally identify these as standard (SR), classification (CR), Lipschitz (LR) and strong classifi- cation (SCR) robustness. Given these differences, some natural questions ask are: What the relationships between them? What assumptions do they make about the training dataset? Are some more effective than others? Are some more interpretable by users than others?
Contributions, In this work, we attempt to answer these questions. We take classification problems example domain, interpreting neural network f RRn I a procedure that separates

---

the n-dimennsional data into classes. This enables the following findings: We observe that from the security perspective, different definitions robustness ultimately determine the nature of attack, thus giving rise to SR, CR, LR, SCR attacks. Thus, but attack with SR constraint. This raises questions about relative strengths of these different training methods and attacks. Some constraint-driven training methods special cases of others. For example, adversarial training known in the literature can be seen as a form of training with SR constraint, with certain amount of parameter tuning.
train for example with SCR constraint
We can order some robustness constraints based on their strength, for example, we show that LR implies SR, and SCR implies CR. In this case, training with a stronger constraint (e.g. LR) will protect better against both kinds of attacks (in this case, both SR and LR attacks). Some pairs of constraints cannot ordered by strength (e-g. SR and SCR, LR and SCR), and in this case, optimising training for given constraint defends better only against attacks with this same target constraint. Moreover, we show that training with logical constraints defends against adversarial attacks better than data augmentation for any choice of robustness definition as training constraint. Finally, we show that there are additional common criteria that can be used to qualitatively compare different modes of constraint-drive training, e.g. interpretability. global or local nature. E.g. CR is the most interpretable, but not globally desirable, LR least interpretable, but globally desirable.
To our knowledge, this is the first systematic study of robustness training from the point of view of the impact of precise formalisation of robustness on training, evaluation and attack. Note that some of the previous work reported on unstable performance of constraint-driven training when defending against attacks (Ayers et al., 2020), which we do not observe in our experiments. Some papers (Fischer et al., 2019) listed and even implemented some kinds of robustness constraints that we study here, but gave no indication of their relative performance. We are not aware of any prior analysis of SR, LR, CR, SCR abstractly as logical constraints.
The paper is organised as follows. Section 2 explains how different robustness constraints arise from different machine learning approaches to constraint-driven training. Section 3 abstractly analysises these robustness definitions, establishing their relative strength, interpretability and applicability. Section shows how these robustness constraints determine different evaluation metrics and attacks, and provides a comprehensive empirical evaluation of the robustness constraints deployed as training constraints and as attacks. Section 5 concludes the paper and outlines future work.
EXISTING TRAINING TECHNIQUES
Data augmentation is one of the simplest methods of improving the robustness of neural network via training (Shorten & Khoshgoftaar, 2019). is applicable to any transformation of the input (e.g. addition of noise, translation, rotation, scaling) that leaves the output label unchanged. To make the network robust against such a transformation, one augments the dataset with instances sampled via the transformation. Although it may seem that this simple solution has nothing to do with formal logic, it imposes significant choices from the point of view of the constraint specification:
cl. the choice of will reflect assumptions about admissible range of perturbations;
c2. the choice of the distance function --- that measures the e-proximity will reflect our assump- tions on desirable geometric properties of the perturbations;
c3. the choice of the sampling method (random sampling, adversarial attacks, generative algorithm, prior knowledge of images etc.) will determine the constraint we optimise for.
But, perhaps even more significantly for us, this method determines the exact definition of robustness that optimise for when train neural network R -> m We call it classification robustness and formally define as follows: given a training dataset input-output pair (x, and a distance function
for all inputs within the e-ball distance of ensure that class y has the largest score in output fxx). In other words:
Definition 1 (Classification robustness)
C)((e e Vx xx x S arg max f(x) =
Used as a spec for training, this constraint does not account for possibility of having "uncertain" images in the dataset, for which small perturbation ideally should change the class. For datasets
2

---

that contain significant number of such images, it will lead to significant reduction in accuracy of the trained neural networks; and, as we show later it may even have a detrimental effect on a network's ' robustness.
Adversarial training is the current state-of the-art method to robustify a neural network. Whereas standard training tries minimise loss between the predicted value, f(x), and the true value, y, for each entry (x,y) y) in the training dataset, adversarial training minimises the loss with respect to the worst-case perturbation of each sample in the training dataset. It therefore replaces the standard training objective L(x, with:
max y( Vxc S
Algorithmic solutions to the maximisation problem that find the worst-case perturbation has been the subject of several papers. The earliest suggestion was the Fast Gradient Sign Method (FGSM) algorithm introduced by Goodfellow et al. (2015):
FGSM(*) -roj( + sign( V (xx,x)))))
However, modern adversarial training methods usual rely some variant of the Projected Gradient Descent (PGD) algorithm (Gu & Rigazio, 2014) which iterates FGSM some number of times:
PG = X: ( = PGD, (FGSM(())
has been empirically observed that neural networks trained using this family of methods exhibit greater robustness at the expense of increased generalization error (Tsipras al., 2018; Madry et al., 2018; Zhang et al., 2019), which frequently referred to as the accuracy-vobustess tradeoff' for neural networks (although this effect has been observed to disappear as the size of the training dataset grows (Raghunathan et al., 2019).
In logical terms what is this procedure trying train for? Obviously it's unreasonable expect that adversarial training will ever succeed in driving the loss of all perturbations down to zero Therefore let assume that there's some maximum distance, that is acceptable for the output to be perturbed given the size of perturbations in the input. This leads us to the following definition, where || - || suitable distance function over the output space:
Definition (Standard robustness)
SR(e, 8, x) Vx: x- S llf(x) yll 5
In the of adversarial training the distance between the outputs ||x y|| is equal to L(x, y)-
We note that, just as with data augmentation, choices el - c3 are still there to be made, although the sampling methods are usually given by special-purpose FGSM/PGD heuristics based on computing the loss function gradients.
Training for Lipschitz robustness. More recently, third competing definition of robustness has been proposed: Lipschitz robustness (Balan et al., 20. Inspired by the well-established concept of Lipschitz continuity, Lipschitz robustness asserts that the distance between the original output and the perturbed output is at most a constant L times the change in the distance between the inputs.
Definition 3 (Lipschitz robustness)
LR(e, L, 2 Vx - - -
Se (1f(x) - yll S L/x
As will be discussed in Section 3, this stronger requirement than standard robustness. Tech- niques for training for Lipschitz robustness include formulating it semi-definite programming optimisation problem (Pauli et al., 2021) or including a projection step that restricts the weight ma- trices those with suitable Lipschitz constants (Gouk al., 2021).
Training with logical constraints. Logically, this discussion leads one to ask whether a more general approach to constraint formulation may exist, and several attempts in the literature addressed this research question (Xu et al., 2018; Fischer et al., 2019), by proposing methods that can translate first-order logical formula C. into constraint loss function Lc- The loss function penalises the
3

---

Table comparison of the different types of robustness studied this paper.
Definition
Standard robustness
Lipschitz robustness
Classification robustness
Strong classification
robustness
Symbol
SR(c,6)
LR((,L)
CR(c)
SCR(e,
Problem domain Interpretability
General Medium
General
Classification
Classification
Low
High
Medium
Globally desirable
X
X
Has loss functions
Adversarial training
X
Data augmentation
X
/
Logical-constrantt training
X
network when outputs do not satisfy given Boolean constraint, and universal quantification is handled by choice of sampling method. Our standard loss function L. is substituted with:
(* y) = y) + (Lc(8,
where weights a and B control the balance between the standard loss and the constraint loss.
This method looks deceivingly as generalisation on the previous approaches. However, even given suitable choices for el c3, classification robustness cannot be modelled via constraint loss in the DL2 framework, as orgmar not differentiable. Instead Fischer et (2019) define an alternative constraint will call strong classification robustness:
Definition 4 (Strong classification robustness)
SCR(e Vx x
S6
f(x)
which looks only at the prediction of the true class and checks whether it is greater than some value (chosen to be 0.52 in their work).
In summary, we have hopefully demonstrated how non-trivial knowledge representation choices and problems arise on the boundary between logical form of the desired constraints and their machine- learning realisations loss functions. In the next section, we analyse the advantages and disadvan- tages of each definition in order to help people better make these choices in future.
3 COMPARISON OF DEFINITIONS
Table shows summary of the points discussed in this section. The first aspect discuss is the logical relationship between the various definitions, i.e. when the definitions agree and disagree.
Standard and Lipschitz robustness. The easiest relationship quantify is the between stan- dard robustness and Lipschitz. robustness. In particular, the latter is a strictly stronger constraint than the former, in the sense that when network satisfies LR(e, L) then also satisfies SR(e, eL). However, the converse does not hold, as standard robustness does not relate the distances between the inputs and the outputs. Consequently, there are SR(e, 5) robust models that are not LR(e, L) robust for any L, as for any fixed L one can always make the distance xx x arbitrarily small in order violate the Lipschitz inequality.
(Strong) classification robustness. The next relationship we discuss is that of classification ro- bustness and strong classification robustness. As discussed earlier, the latter designed to over- approximate the former whilst providing a logical loss function with meaningful gradient. We work under the assumption that the last layer the classification network is a softmax layer and therefore the output forms probability distribution. When > 0.5 then, as would be hoped, any network that satisfies SCR(e, 7) also satisfies CR((c.. For n < 0.5 this relationship breaks down the true class may be assigned probability greater than 7 but may still not be the class with the highest probability. We therefore recommended that one only uses value of n) > 0.5 when using strong classification robustness (for example 0.52 in Fischer et al. (2019)).
4

---

Given that LR is stronger than SR and SCR is stronger than CR, the obvious question is whether there is a relationship between these two groups? In short, the answer to this question no. In particular, although the two sets of definitions agree whether a network is robust around images with high-confidence, they disagree over whether network is robust around images with low confidence. We illustrate this with example, comparing SR against CR. We note that similar analysis holds for any pairing from the two groups.
Standard vs classification robustness. The key insight is that standard robustness bounds the drop in confidence that neural network can exhibit after a perturbation, whereas classifica- tion robustness does not. Figure shows two hypothetical images from the MNIST dataset. Our network predicts that Figure has an 85% chance of being 7. Now consider adding small perturbation to the image and consider two different scenarios. In the first scenario the output of the network for class 7 decreases from 85% to 83% and therefore the classifica- Figure I: Hypotherical images from the MNIST dataset tion stays in the same. In the second scenario the the output of the network for class 7 decreases from 85% 45%, and results in the classification changing from 7 to 9. When considering the two definitions, small change in the output leads to no change in the classification and large change in the output leads a change in classification and robustness and classification robustness both agree with each other.
7
(a) P(7) 85%
(b) P(7) 51%
However now consider Figure 1b with relatively high uncertainty. this case the network is (cor- rectly) less sure about the image, only narrowly deciding that it's a 7. Again consider adding a small perturbation. In the first scenario the prediction of the network changes dramatically with the proba- bility of it being 7 increasing from 51 % to 91% but leaves the classification unchanged as 7. In the second scenario the output of the network only changes very slightly, decreasing from 51% to 49% flipping the classification from to 9. Now, the definitions of SR and CR disagree. In the first case, adding a small amount of noise has erroneously massively increased the network's confidence and therefore the SR definition correctly identifies that this is a problem. In contrast CR has no problem with this massive increase in confidence as the chosen output class remains unchanged. Given this, it is clear that although standard and classification robustness agree low-uncertainty examples, classification robustness breaks down and gives what we argue are both false positives and false negatives when considering examples with high-unncrtaintt.
Dataset assumptions. A related question that we have found to be rarely discussed in the literature what assumptions the different definitions are making about the distribution of the training data with respect to the data manifold of the true distribution of inputs.
For SR and LR it is, at minimum, desirable for the network be robust over the entire data man- ifold. In the most domains the shape of the manifold is unknown and therefore it is necessary to approximate by taking the union of the balls around the inputs in the training dataset. We are not particularly interested about whether the network is robust in regions of the input space that lie off the data manifold, but there is no problem the network is robust in these regions. Therefore these definitions make no assumptions about the distribution of the training dataset.
This is in contrast to (S)CR. As discussed the previous section, rather than requiring that there is only a small change in the output, they require that there is no change to the classification. This is only a desirable constraint when the region being considered does not contain a decision boundary. Consequently when one is training for some form of classification robustness, one is implicitly making the assumption that the training data points lie away from any decision boundaries within the manifold. In practice, most datasets for classification problems assign single label instead of an entire probability distribution to each input point, and so this assumption is usually valid. However, we feel it is important note that classification robustness is not an appropriate definition train for if the dataset contains input points that may lie close to the decision boundaries and in such case may result in a logically inconsistent specification.
5

---

Interpretability. One of the key selling points of training with logical constraints is that, by ensur- ing that the network obeys understandable constraints, it improves the explainability of the neural network. Each of the robustness constraints encode that "small changes to the input only result in small changes to the output", but the interpretability of each definition is also important.
All of the definitions share the relatively interpretable parameter, which measures how large perturbation from the input is acceptable. Despite the other drawbacks discussed so far, CR is inherently the most interpretable it has no second parameter. In contrast, SR and SCR require extra parameters, and respectively, which measure the allowable deviation in the output. Their addition make these models less interpretable.
Finally we argue that, although LR is the most desirable constraint, it is also the least interpretable. Its second parameter L measures the allowable change in the output as a proportion of the allowable change in the input. It therefore requires one to not only have an interpretation of distance for both the input and output spaces, but to be able to relate them. In most domains, this relationship simply doesn't exist. Consider the MNIST dataset, both the commonly used notion of pixel-wise distance used in the input set, although crude, and the distance between the output distributions are both interpretable. However, the relationship between them is not. For example, what does allowing the distance between the output probability distributions being no more than twice the distance between the images actually mean? This therefore highlights common trade-off between complexity of the constraint and its interpretability.
Another interpretability consideration that appears when training with logical constraints is the mantics of the parameters a and / which allow the users to decide how to weight the accuracy of the network vs the importance that adheres to the constraint. Section I describes how the loss function used in logical-constrantt training is split into two parts: C* = a.((x, y) + (8, y). where the first part designed maximise the accuracy of the predictions and the second encodes the con- straint. However, in practice the constraints (and therefore the derived loss functions) refer to the true label y rather than the current output of the network f(x). e.g. Vx: xx x S c = [f(x) y S 6. This leads to scenarios where network that is robust around & but gives the wrong prediction, being penalised by Le which on paper is designed maximise robustness. Essentially Lo is trying to maximise both accuracy and constraint adherence concurrently. Instead, we argue that to preserve the intended semantics of a and 3 is important to instead compare against the current output the network e.g. Vx
x = fxx f(x) < 8. Of course, this doesn's 't work for SCR because in order to derive the most popular class from the output f() you need arg max, the very function SCR seeks avoid using. This is another argument why (S)CR should be avoided if possible.
4 EXPERIMENTS
Evaluation metrics. Given a particular definition of robustness, a natural question is how to quantify how close a given network is satisfying it? We argue that there three different measures that one should be interested in:
1. Does the constraint hold? This is a binary measure and the answer is either true or false.
2. If the constraint doesn't 't hold, how easy for an attacker to find a violation?
3. If the constraint doesn't 't hold, how often does the average encounter a violation?
Based off of these measures, we define three concrete metrics: constraint satisfaction, constraint security, constraint accuracy'. Let X' be the training dataset, B(x,)) e) - (x R" be the e-ball around x and P be the right hand side of the implication in each of the definitions of robustness. Let Is be the standard indicator function which is if constraint o(x) holds and otherwise. The constraint satisfaction metric measures the proportion of the training dataset for which the constraint holds. Unfortunately, depending on the network architecture and the constraint in question, the indicator function may not always be feasible to evaluate.
x| <
'Our naming scheme differs from Fischer et al. (2019) who use the term constraint accuracy (without ex- plicitly defining it in the paper) to refer to what we term constraint security. In our opinion, the term constraint accuracy is less appropriate here than the name constraint security given the of an adversarial attack.

---

Definition 5 (Constraint satisfaction)
CS(2)
IIxxE3(..):P(O))
X
REX
In contrast, constraint security measures the proportion of inputs in the training dataset such that an attack A unable to find an adversarial example for constraint P. In our experiments we use the PGD attack for A, although in general any strong attack can be used.
Definition 6 (Constraint security)
CR(2)
=
1 X
Σ (p(A()) SEX
Finally constraint accuracy estimates the probability of random user coming across a counter- example to the constraint, usually referred as 1 - success rate in the robustness literature. Let S(&, n) be a set of n elements randomly uniformly sampled from B(& e). Then constraint accuracy follows:
Definition (Constraint accuracy)
= CL(X)
1 X
Σ [p(x) (E)(),))
n
R
Note that there is no relationship between constraint accuracy and constraint security: an attacker may succeed in finding an adversarial example where random sampling fails and vice-versa. Also note the role of sampling in this discussion and compare it to the discussion of the choice c3 in Section 2. Firstly, sampling procedures affect both training and evaluation of networks. But the same time, their choice is orthogonal to choosing the verification constraint for which we optimise or evaluate. For example, we measure constraint security with respect to the PGD attack, and this determines the way we sample; but having made that choice still leaves us to decide which constraint, SCR, SR, LR, or other we will be measuring as we sample.
Choosing an evaluation metric. It is important to note that for all three evaluation metrics, one still has to make a choice for constraint P, namely SR, SCR or LR, as defined in Section 2. As constraint security always uses PGD to find input perturbations, the choice of SR, SCR and LR effectively amounts to us making a judgement of what an adversarial perturbation consists of: class change defined by SCR, or is it a violation of the more nuanced metrics defined by SR and LR? Therefore we will evaluate constraint security on the SR/SC//LR constraints using a PGD attack. This will be further analysed in the experiment E1 below.
For large search spaces in n dimensions, random sampling deployed in constraint accuracy fails to find the trickier adversarial examples, and usually has deceivingly high performance: we found 100% and > 98% constraint accuracy for SR and SCR, respectively. We will therefore not discuss these experiments in detail, but refer the reader to Appendix A.
Constraint satisfaction is different from constraint security and accuracy, in that must evaluate constraints over infinite domains rather than merely sampling from them. Verifiers such Marabou or ERAN are built with this purpose in mind. It is out of scope for this paper to conduct experiments by this measure, and we leave for future work. Our preliminary experiments with Marabou confirm the generally accepted assumption that it is very hard for training techniques to achieve a high-score for constraint satisfaction.
Networks. We use the FASHION MNIST (or just FASHION) (Xiao et al., 2017) and the GT- SRB (Stallkamp al., 2011) datasets. For our baseline architecture, we use two fully connected layers: the first layer uses the ReLU activation function, and the second uses the clamp function to restrict each output to the range [-100, 100). We use a clamp function to evaluate the constraints instead of the traditional softmax function because the former is compatible with the constraint ver- ification tools such as Marabou whereas the latter is not. The predicted classification is then taken the output with the maximum score. However, only during training, clamp function through a softmax layer before feeding it to the loss. For instance, fANIST = Fo o F,, .0 LeLU and on (x:) clamp(s, - 100, 100).
pass the output from the
where F, IR 784 - R 100 and Fo R100
IR
7

---

Neura
net trained with Constraint Loss (L FASHION]
Neural net trained with Constraint Loss (3R) (FASHION)
3R Attock
SCR Atacs
LR Abacs
3R Adacc
BCR Atack
125
-2
50
52
26
28
0,000
0.35
3.000
0,275
0200
0.000
0,35
0.050
1,005
7,100
Epalos
Easton
Figure 2: Experiments that show how the two networks trained with LR and SR constraints perform when evaluated against different definitions of robustness underlying the attack.
Rocustness againsl SR Allack (FASH ON). "he different lines show performance reural networks
Robusimeus ugainsl SR Alack (GTSRB). The differen. lines show pe formance diffe neural networks trained
Combrart Laua Lantram (SCR)
Corab Lana Combrr Love (SCR)
Cordinar LUR
0.355
3,000
0,275
0015
0.550
1,005
Lpsten
London
Figure 3: Experiments show how different choices of constraint affect standard robustness of neural networks, for PGD anacks with various values.
Loss functions. Since our experiments study classification problems, we will use the cross-entropy loss function as our baseline loss function:
m
Lee (x,, y) = Σ y, log(f(x))
For the Lo component of the loss function L" we use the constraint-to-lot function translation of Fischer et al. (2019). all experiments, we use the Adam optimiser (Kingma & Ba, 2015) with the following learning parameters:
0.0001, 100 epochs, batch size of 128.
Settings. We keep the architecture the same throughout the experiments, and only vary the training. Thus our Baseline network is trained just with coos-enttropy. Data Augmentation adds 2 additional images in the 0.lle-ball around each image, and samples either randomly from a uniform distribu- tion (RU) or using FGSM attack. Adversarial training refers the training procedure described in Section 2, with FGSM sampling. In all other cases, we use a constraint loss function Le defined as in Fischer et al. (2019), and use the constraints SR, SCR, LR defined in Section with 1, 3 3= 0.2, = 0.1, = 10, 1 = 0.52, L = 10 and _ distance metrics. All networks trained with L use sampling by the PGD attack, for efficiency (as well as comparability).
Results. We start with noting the standard test set accuracy of the resulting neural networks in Table 2, making sure that our different train- ing regimes do not deteriorate networks' gen- eral performance too drastically. The most no- table accuracy drop occurs for adversarial train- ing.
Table 2: Standard test accuracy (as % of the dataset instances) for chosen trained networks.
Training Regime:
FASHION
GTSRB
Baseline
88.2
92.4
Data Augmentation (RU)
88.6
92.8
We now highlight groups of experiments that confirm extend main theoretical conclu- sions; the complete experiment description is available in Appendix A.
Data Augmentation (FGSM)
88.8
94.5
Adversarial Training
85.1
83.5
Constraint Loss (SR)
88.2
93.3
Constraint Loss (SCR)
91.9
Experiment set E1. Comparable constraints. In Section 3, we established that LR con- straint stronger than SR, and both are not strictly comparable to SCR. This would suggest that, train two neural networks, one with the SR, and the other with the LR constraint, then the latter
Constraint Loss (LR)
86.6
93.1
8

---

Rocustness against SR Allack (FASHION), "he diferent lree show formmnaaece differen: reural networks trained with.
Robusiness againsl SR Attakk (GTSRB). The dffferen. lines show formance diffe end neural nelworks trained with.
Rursen
Unto fagiwersttoo (-555)
Aeversant moinng
Aughnettinnnn 3B1 Loratra 11 (5-1)
lraning
100
76
78
88
35
0.35 0
0,275
0.000 0,055
0,050
1,005
3,100
Epalos
Exsiton
Figure 4: Experiments that show how adversaria training, training with data augmentation, and training with constraint loss affect standard robustness of neural networks, for PGD attacks with various values.
should always have higher constraint security against both SR and LR attacks than the former. This indeed confirmed by the experiment shown in Figure 2. We also discussed that depending on the properties of the dataset, SR may not guarantee SCR, and Figure 2 shows exactly that case. It also confirms that generally, stronger constraints are harder to obtain: whether a network is trained with SR or LR constraints, it is less robust against LR attack than any other attack.
Experiment set E2. Incomparable constraints. The results Figure tell us that using the SCR constraint for training does not help to increase defences against SR attacks. A similar picture, but in reverse, can be seen when we optimise for SR but attack with SCR, see Appendix A.
Experiment set E3. Constraint training versus data augmentation and adversarial training. Next, we confirm our assumptions about the relative inefficiency of using data augmentation com- pared to adversarial training training with constraints, attack, but graphs for SL and SCR attacks show the same trends, see Appendix A. Surprisingly, neural networks trained with data augmentation give is encouraging to see that Constraint Loss (SR) is almost as good as adversarial training while the latter also gives 11% drop in terms of the networks accuracy (Table 2).
Figure 4. We show this only for SR
results than even the baseline network.
Experiment set E4. Role of other parameters. As previously discussed, random uniform sampling struggles to find adversarial inputs in large searching spaces. It is logical expect that using random uniform sampling when training will be less successful than training with sampling that uses FGSM or PGD as heuristics. Indeed, Figure shows this effect for data augmentation, but similar trends are expected for any form of training.
Finally, one may ask whether the trends just described would be replicated for more complex archi- tectures of neural networks. In particular, data augmentation is known to require larger networks. By replicating the results of Figure 4 with a large 18 layer convolutional network of Fischer et al. (2019) indeed confirms that larger networks handle data augmentation better, and data augmentation improves robustness compared to the baseline. Nevertheless, data augmentation still lags behind all other modes of constraint driven training, and thus this major trend remains stable across network architectures, see Appendix A.
5 CONCLUSIONS AND RELATED WORK
We have presented a comprehensive study of constraint-driven training from the formal point of view. Taking robustness as representative constraint, we abstractly studied different forms of bustness; and showed how the existing literature constraint- through this prism. Moreover, we proposed a method that separates out the logical study of con- straints from their implementation as loss functions for training on the one hand, and their use as evaluation methods and attacks on the other. We showed that this method allows us to make general conclusions about relations of different modes of constraint-driven training that were not possible before.
training can be understood
For translation of constraints into loss functions, we used the implementation from Fischer et al. (2019); thus our results are compatible and comparable with that prior study. In particular we have identified that the method of Fischer al. (2019) is not strict generalisation of the other techniques and that significant trade-offs and decisions have to be made in order to represent some of the robustness definitions.

---

REFERENCES
Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. In International Conference on Machine Learning, pp. 291-301. PMLR, 2019.
Edward W. Ayers, Francisco Eiras, Majd Hawasly, and lain Whiteside. PaRoT: practical frame- work for robust deep neural network training. In NASA Formal Methods 12th International Symposium, NFM 2020, Moffett Field, CA, USA, May 11-15, 2020, Proceedings, volume 12229 of LNCS, pp. 63-84. Springer, 2020.
Radu Balan, Maneesh Singh, and Dongmian Zou. Lipschitz properties for deep convolutional net- works. Contemporary Mathematics, 706:129-18 19-1151,111 2018.
Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen. Timon Gehr, Ce Zhang, and Martin T. Vechev. DL2: training and querying neural networks with logic. In Proc. of the 36th Int. Conf. Machine Learning. ICML 2019, volume 97, pp. 1931-1941. PMLR, 2019.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural net- works by enforcing Lipschitz continuity. Machine Learning, 110(2):393-416, 2021.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5063 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Infernational Conference on Learning Representations, 2018.
Patricia Pauli, Anne Koch, Julian Berberich, Paul Kohler, and Frank Allgower. Training robust neural networks using Lipschitz bounds. IEEE Control Systems Letters, 2021.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint
2019.
Connor Shorten and Taghi M. Khoshgoftaar. A survey image data augmentation for deep learn- ing. J. Big Data, 6:60, 2019.
Johannes Stallkamp, Marc Schlipsing. Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. IEEE International Joint Conference on Neural Networks, pp. 1453-1460) 2011.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- tions, 2018
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNISTI novel image dataset for bench- marking machine learning algorithms, 2017.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmssmässan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. -5997-507. PMLR, 2018
10

---

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent EI Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.

---

A APPENDIX
This appendix will present all the results that, due to space, were not possible to include in the main text. We will show in detail how systematically evaluated all the trends that we reported. Figures 5, 8 are the same as Figures 4, 3, and Figures 13A, 14A correspond to Figure 2, we report them again here for completeness.
A.1 DATA AUGMENTATION VS ADVERSARIAL. TRAINING VS TRAINING WITH CONSTRAINT LOSS
Figures 5, 6 and 7 show how the networks trained with the different methods are robust against attacks. For SR and SCR attacks, Adversarial Training improves significantly the robustness of the model while training with Constraint Loss (SR) also improves it, although not much. Training with both the Data Augmentation methods actually reduce robustness of the network. None of the training techniques succeed in ensuring robustness against LR attacks, indicating that as discussed in Section 1. LR is strong property to hold. Nonetheless we report the graphs for completeness.
Rocustness againsl SR Allack (FASHION). The diferent lines show performance differem: neural networks trained with:
Robusiness againsl SR Alack (GTSRB). The differen. lines shaw formance diffe rent neural networks trained with:
Data fagnersttr (50))) Apverardl Troking
Doto Auggenttettttioonn 2BV Advorsara Tianing
Corserent Lace (SR
Ccraita Lo (914)
100)
02
0.25
0:55
0,275
0.000
0.035
0.050
1,075
2,100
Epsies
Figure 5: Experiments that show how adversarial training. training with data augmentation. and training with constraint loss affect standard robustness of neural networks. for varying sizes of the PGD attack (measured by values).
Robustness against LR Attack (FASHION). The different lines show performance different reural natworks trained with:
Robustness against LR Altack (GTERB). The different ines show fooranne diffe neural notworks trained
Aug Acverses of Treking
Datas Argmaritter CSM)
DatuAggmatiia BBM Abhama Turnng Conera Lose (59)
Conment Lose IBR
100
50
0125
Uhall
0,2/5
R,02
36020
1,025
11550
3
2,001
Fystes
Figure 6: Experiments that show how adversarial training, training with data augmentation, and training with constraint loss affect lipschitz robustness of neural networks, for varying sizes of the PGD attack (measured by values).
A.2 TRAINING WITH DIFFERENT CONSTRAINT LOSSES
Figures 8, and 10 show how the networks trained with the different constraint losses are robust against attacks. For SR and SCR attacks, we can see trend in the networks trained with the same constraint that are generally more robust against the respective attacks. On the other hand, models trained with LR are generally more robust and they have the best average improvement against all attacks. None of the training techniques succeed in ensuring robustness against LR attacks, except the network trained with LR on the FASHION dataset.
A.3 TRAINING A BIGGER NETWORK
Figure 11 show some of the training methods scale with the architecture. On the left we have the small architecture used for all the experiments, while the right we have a bigger architecture. The
12

---

Robusiness against SCR Allack (FASHION) The different ines show perfoorance differen reural networks trained with.
Rebusiress against SCR Altack (GTSRB): The cifferent Ines show formanue diffe neural nelworks trained with.
Joheno
Upto fagiwersatoon (-555))
moning
Auginettiionn 3B1 Loratr 11 (9-1)
lraning
100
76
35
0,000
0.35
3020
0,375
0200
0.000
0,15
0,050
3,005
7,100
Epalos
Easton
Figure 7: Experiments that show how adversarial training, training with data augmentation, and training with constraint loss affect strong classification robustness of neural networks, for varying sizes of the PGD attack (measured by values).
Rocustness against SR Allack (FASHION). "he different lires show performance differem: reural networks trained with:
Robusiness against SR Attack (GTSRB). The different lines show formance diffe neural networks trained with.
Comeraat
Constr Loos
Conctro Looo
20
0,26
0,275
0,102
3.000
1,025
0.050
2,015
2,003
Fystos
Eishn
Figure 8: Experiments that show how different choices of constraint loss affect standard robustness of neural networks, for varying sizes of the PGD attack (measured by e values).
relative behaviour between the training methods remain the same, except from the baseline network that present decrease in robustness and it exhibits similar behaviour to the models trained with Data Augmentation.
A.4 NETWORKS' BEHAVIOURS AGAINST DIFFERENT ATTACKS
Previous experiments show that the most robust models the ones trained with Adversarial Train- ing, LR and SR. Figures 12, 13 and 14 select the relevant data from previous experiments to provide comparison of how these models perform against the different attacks. All the networks strug- gle the most against LR attacks, while they present significant robustness against SR attacks and a slightly less but still important robustness against SCR attacks. Overall we can see that Adversarial Training provides the best defence against SR and SCR attacks, immediately followed by training with Constraint Loss LR. However, as already reported above, the only network that show some robustness against LR attacks is the one trained with Constraint Loss LR.
13

---

Robustness against LR Attack (FASHION), The different lines show performance differem, neural natworks trained with:
Rebustness against LR Altack (GTSRB). The different ines show pe formnnce diffe fnnt noural networks trained with:
Tlreeine
Croddwarl 1500
Covernen tox (1inr)
Cardd er les (18] Convdro ef nex (501))
** Comehei
25
27
1,025
Eystos
Eishin
Figure 9: Experiments that show how different choices of a constraint loss affect lipschitz robustness of neural networks, for varying sizes of the PGD attack (measured by values).
Robustness against SCR (FASHION) The different of ow per formance difleren, reural networkss trained will.
Robustness against SCR Altacx (3TSRB): The oifferent
show per formanue diffe ent neural nelworks Irained with.
Biudite
Cunduntt Lite 18R) .. Coeehrett (105
Caniaain Lasa (SOR)
Buadi:
Caral dn. Lusa (9R)
Cuando Lon (SCR)
*. Cordham (85
125
25
0,77
0,035
1
Epalie
Exskan
Figure 10: Experiments that show how different choices of a constraint loss affect strong classification robust- ness of neural networks, for varying sizes of the PGD attack (measured by values).
Robusiness against SCR (FASHION). The thow performance of differem: reural networks trained with:
Robusiness aga nsi SCR (FASHIOON) difeernn show pe formance different neural networks trained with:
miseeim ex LData Audeem (Vanoom Untamm
Duta Angieergtttooo (FC6M)
Constrane staan Loos (9OR)
Blssiine Lara Augrerrat Rartem Univam
125
2obb Aimeetann (FGBN)
Condraint
100
78
0.35
2030
0,275 0/002
0035
0.550
2,005
2,100
Figure 11: Experiments that show comparison of how training with data augmentation, and training with constraint loss affect strong classification robustness of neural networks, for varying sizes of the PGD attack (measured by values) of our standard network with a bigger architecture.
Neural mel. trained with Adversanial Training (FASHION)
Neural mel trained with Acher satial Training (GTSRB)
Alaac
1775
100
Securar
100
26
25
0.35
3,20
0,27
0/02
0.355
0.550
1,055
Lpsten
Landon
Figure 12: Experiments that show how the networky trained with Adversarial Training perform when evalu- ated against different definitions of rohusmess underlying the attack.
14

---

Neural net trained with Constraint Loss (SR) (FASH
Neural net trained with Constraint Loss (SR) (GTSRB)
100
100
Fyddes
Figure 13: Experiments that show how the networks trained with SR constraints perform when evaluated against different definitions of robustness underlying the attuck.
Veura nel trained with Consirant sink Loes (L R) (FASHION)
Neural me. L L'ained with Constrain Loss (LR) (GTSRB)
100
100
0,275
CO35
Upstes
Figure 14: Experiments that show how the networks trained with I.R constraints perform when evaluated against different definitions of robustness underlying the attack.
15


Reviews:
[{"review_id": "RUwu1kFmEC", "reviewer": "Reviewer_6o25", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors aim at creating a general and formal framework for uniting the different notions of robustness of neural networks. In particular, they consider four notions. \n\nFirst, the restriction to only four approaches in such a vast field of research is not preferable for a general framework (just as an example, there are probabilistic robustness [1], stability training [2], targeted robustness [3]). \n\nSecond, the authors introduce their own meaning for \"interpretability\" without actually discussing it which makes it hard to follow the discussion. While usually in the literature interpretability relates to the understanding of the decisions of the black-box model and reasoning behind them, in this paper interpretability relates to the intelligibility of the robustness constraints - which of course does contribute to expert-level understandability of the model, but not much connected to the classical interpretability.\n\nFurther, the authors introduce different kind of attacks for different classes of robustness considered, but in the end everything converges to only adversarial attacks (that is only one class of robustness) without proper justification for this. Moreover, exclusively PGD attacks are used, motivated by state-of-the-art adversarial robustness checks.\n\nThe formal definitions of the robustness types are very sloppy and do not introduce all the variables used (mainly, for what x and what y the definition is considered is not included in the definition itself).\n\nWhile being centered around adversarial research, the paper makes very rough mistakes with regard to it: (i) FGSM attack is using sign of the gradient, not the projected gradient (ii) the citation reference for PGD attack is absolutely wrong (it is not Gu&Rigazio paper from 2014). Moreover, adversarial training currently is much more advanced compared to FGSM training (e.g., [4]). \nLipschitz continuity of the function is defined with respect to the distance between outputs of the function (while in Definition3 in the paper the correct label is used right away). Moreover, one cannot just \"assume\" the value of the Lipschitz constant as it is done in the experiments.\n\nThe discussion comparing Lipschitz robustness and Standard robustness is wrong, since the difference is more that the difference of predictions for SR is fixed in \\delta, while LR should become less with moving the inputs close to each other.\n\nThe introduction of Constraint Properties is given in the evaluation section which makes it unconnected to the framework definition. Moreover it is unclear how the constraints are evaluated in the experiments (possibly only due to the not proper discussion).\n\nFinally, the authors omit comparison to the multitudes of other robustness frameworks, saying that it is out of scope of the current work, which is wrong (e.g., [5]).\n\nminor details:\n- The list of the approaches in the introduction does not correspond to the names introduced, which makes it very confusing between standard robustness and classification robustness.\n\n[1] Mangal, Ravi, Aditya V. Nori, and Alessandro Orso. \"Robustness of neural networks: A probabilistic and practical approach.\" 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, 2019.\n\n[2] Zheng, Stephan, et al. \"Improving the robustness of deep neural networks via stability training.\" Proceedings of the ieee conference on computer vision and pattern recognition. 2016.\n\n[3] Gopinath, Divya, et al. \"Deepsafe: A data-driven approach for assessing robustness of neural networks.\" International symposium on automated technology for verification and analysis. Springer, Cham, 2018.\n\n[4] Wong, Eric, Leslie Rice, and J. Zico Kolter. \"Fast is better than free: Revisiting adversarial training.\" arXiv preprint arXiv:2001.03994 (2020).\n\n[5] Gehr, Timon, et al. \"Ai2: Safety and robustness certification of neural networks with abstract interpretation.\" 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 2018.", "summary_of_the_paper": "The authors are proposing an approach to systematization of the types of robustness of neural network. In particular, they are discussing four types of robustness: through augmentation, through adversarial training, through Lipschitz constraint training and through logical constraints training. The authors discuss the interconnections between the four classes of robustness, hypothesize which one is more general than others, propose possible measures for how easily the robustness can be violated - randomly or intentionally. In the experimental evaluation the authors are checking the attacks vulnerability for the networks with different types of robustness, confirming discussions about them.", "summary_of_the_review": "Based on the mistakes in the text of the paper, sloppiness of the formal definitions, and unclear contribution, I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RMAU9cU8x3", "reviewer": "Reviewer_DHPy", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* This paper attempts to systematically compare different robustness definitions.\n \nWeaknesses:\n* Many arguments in the paper are too strong yet without sufficient justification and are not convincing (more in detailed comments).\n* The major contribution and insights of this paper remains unclear.  Robustness can be viewed as a mathematical property, which is known. If the authors aim to make a systematic discussion, I think they need to extensively cite related works to support the insights. \n* The paper is not clearly structured and is difficult for readers to get the core insights of this paper.  I suggest there to be subsections and some reorganization.\n \nDetailed comments:\n* \"Definition 2 (Standard robustness)\": This doesn’t really appear to be “standard” from my knowledge. I see “classification robustness” more in the literature, but this paper claims Def 2 as the “standard” without sufficient justification. \n* It is unclear what “globally desirable” means. The authors put “standard robustness” as “globally desirable” but “classification robustness” as not “globally desirable”, which does not look correct to me.\n* I think the “classification robustness” definition is closer to the goal of adversarial training but this paper says adversarial training is for “standard robustness”, and I think this is wrong. References are needed if the authors want to argue on that.\"\n* “Finally we argue that, although LR is the most desirable constraint\": It is not convincing why LR is the most desirable. In the rebuttal, the authors argue that “LR is strictly stronger”, but this is not convincing that being strictly stronger means more desirable. Other factors (such as the feasibility) are also important. There are similar issues with the strong argument \"(S)CR should be avoided if possible”.\n* Def 5~7: Literally \"satisfaction\", \"security\", \"accuracy\" have similar meanings, but the paper assigns different definitions to \"Constraint satisfaction\", \"Constraint security\", \"Constraint accuracy\", which is quite confusing .\n* \"This work considers four of the most prominent families of techniques\": This sentence asserts the listed categories as the \"most prominent\" but misses lots of important works on certified adversarial robustness (including randomized smoothing and deterministic methods). To list a few examples (there are many more others):\n  * Wong, Eric, and Zico Kolter. \"Provable defenses against adversarial examples via the convex outer adversarial polytope.\" International Conference on Machine Learning. PMLR, 2018.\n  * Cohen, Jeremy, Elan Rosenfeld, and Zico Kolter. \"Certified adversarial robustness via randomized smoothing.\" International Conference on Machine Learning. PMLR, 2019.\n  * Li, Bai, et al. \"Certified adversarial robustness with additive noise.\" Advances in Neural Information Processing Systems 32 (2019): 9464-9474.\n  * Salman, Hadi, et al. \"Provably robust deep learning via adversarially trained smoothed classifiers.\" Advances in Neural Information Processing Systems 32 (2019): 11292-11303.\n  * Gowal, Sven, et al. \"On the effectiveness of interval bound propagation for training verifiably robust models.\" arXiv preprint arXiv:1810.12715 (2018).\n  * Zhai, Runtian, et al. \"MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius.\" International Conference on Learning Representations. 2019.\n  * Zhang, Huan, et al. \"Towards stable and efficient training of verifiably robust neural networks.\" ICLR 2021.\n  * Zhang, Bohang, et al. \"Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons.\" International Conference on Machine Learning. PMLR, 2021.\n  * Shi, Zhouxing, et al. \"Fast Certified Robust Training with Short Warmup.\" Advances in Neural Information Processing Systems 34 (2021).\n", "summary_of_the_paper": "This paper attempts to compare different robustness definitions. The paper discusses the relations between various definitions and conducts experiments to show the relations.", "summary_of_the_review": "There are too many strong arguments or definitions that do not have sufficient justification and are not convincing or reasonable. And this paper is not quite clear and not well-structured. Author response does not address my concerns and the authors did not show up to answer my follow-up questions. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I8gffd74uwR", "reviewer": "Reviewer_2Rrw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The summarization and proposed formal framework is very helpful in terms of unifying notions and categorizing existing work in the literature.\n- Some empirical findings are interesting like LR constraint training maintains competitive robustness as adversarial training.\n- Code is provided for reproducibility.\n\nWeaknesses:\n- The theoretical analyses seem a bit preliminary. The implication relationships: LR -> SR, SCR -> CR immediately follow from their definitions. The comparison table in Table 1 is mainly based on empirical reasoning.\n- The empirical evaluation does not provide sufficient novel findings. It is intuitive that adversarial training provides the highest CR or SCR. Also, it is a bit straightforward that incorporating corresponding constraints provides the highest robustness against corresponding robustness notions. The implication relationships directly guarantee the findings of Experiment set E1.\n- Also, the empirical evaluation lacks generalizability in terms of datasets and covered works. In terms of datasets: large-scale datasets like CIFAR-10, CIFAR-100, ImageNet, etc need to be included. In terms of covered works: the paper seems to focus heavily on the DL2 work by Fischer et al. Some work, especially for LR (see below), is not yet included. Therefore, the evaluation is not comprehensive enough.\n\nSuggestions:\n- In terms of Lipschitz robustness, recent literature imposes this constraint via proposing specialized network architectures [1,2] or Lipschitz-aware training [3,4]. These methods may need to be evaluated for comprehensiveness.\n- Details on the PGD attack may need to be provided. For example, how does PGD find violations of LR? What are the used step size and iterations for PGD?\n\nMinor:\n- last paragraph of Section 1: analysises -> analyzes\n\n[1] Trockman, Asher, and J. Zico Kolter. \"Orthogonalizing Convolutional Layers with the Cayley Transform.\" International Conference on Learning Representations 2021.\n\n[2] Zhang, Bohang, et al. \"Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Tsuzuku, Yusuke, Issei Sato, and Masashi Sugiyama. \"Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks.\" NIPS 2018.\n\n[4] Leino, Klas, Zifan Wang, and Matt Fredrikson. \"Globally-Robust Neural Networks.\" International Conference on Machine Learning. PMLR, 2021.", "summary_of_the_paper": "This paper formally summarizes common robustness notions in the literature as: standard robustness (SR), classification robustness (CR), Lipschitz robustness (LR), and strong classification robustness (SCR). The paper proves LR implies SR, SCR implies CR. Then some empirical studies are conducted.", "summary_of_the_review": "The summarization and formalization of robustness notions is a good contribution of this paper, which helps the community to sort out different approaches and their applicability. However, from the empirical front and theoretical front, the paper may not provide enough novel findings nor useful approaches. Thus, the paper may be below the acceptance threshold at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies and compares different notions of robustness. However, reviewers found there are many unjustified claims in the analysis, and the paper does not provide novel findings nor useful approaches.
Decision: Reject