Year: 2021

Abstract:
Neural networks are widely used to detect general patterns in noisy data, however they are also known to not be particularly robust, i.e. moving a small distance in the input space can change the network’s output significantly. Recently, many methods for improving network robustness have been proposed and this growing body of research has given rise to numerous explicit or implicit notions of robustness. Connections between these notions are often subtle, and a systematic comparison of these different definitions is missing in the literature. In this paper we attempt to address this gap by performing an in-depth comparison of the different definitions of robustness by analysing their relationships, assumptions, interpretability and verifiability. By viewing robustness as a stand-alone mathematical property, we are able to show that, having a choice of several definitions of robustness, one can combine them in a modular way when defining training modes, evaluation metrics, and attacks on neural networks. We also perform experiments to compare the applicability and efficacy of different training methods for ensuring the network obeys these different definitions.

heading: 1 INTRODUCTION
text: Safety and security are critical for some complex AI systems involving neural networks, yet they are difficult to ensure. The most famous instance of this problem is guaranteeing robustness against adversarial attacks (Szegedy et al., 2014; Goodfellow et al., 2015). Intuitively, an image is -ball robust if, when you move no more than away from it in the input space, the output does not change much, or alternatively, the classification decision that the network gives does not change. Adversarial robustness is a property that even very accurate neural networks fail to satisfy.
The proposed solution is to (re)train the network with robustness specifically in mind. Such training can be seen as a way to impose a formal specification, and so may contribute to explainability as well as verification. This work considers four of the most prominent families of techniques:
1. Data augmentation (Shorten & Khoshgoftaar, 2019)
2. Adversarial training (Goodfellow et al., 2015; Madry et al., 2018)
3. Lipschitz robustness training (Anil et al., 2019; Pauli et al., 2021)
4. Training with logical constraints (Xu et al., 2018; Fischer et al., 2019)
The last technique, training with logical constraints, is a more general approach that can train for not just robustness, but a wide range of constraints expressed in some logical language.
Although the first three families of methods all seek to represent the same high-level knowledge in the neural network, each technique seeks to optimise for subtly different definitions of robustness. We formally identify these as standard (SR), classification (CR), Lipschitz (LR) and strong classification (SCR) robustness. Given these differences, some natural questions to ask are: What are the relationships between them? What assumptions do they make about the training dataset? Are some more effective than others? Are some more interpretable by users than others?
Contributions. In this work, we attempt to answer these questions. We take classification problems as an example domain, interpreting a neural network f : Rn → Rm as a procedure that separates
the n-dimensional data into m classes. This enables the following findings: We observe that from the security perspective, different definitions of robustness ultimately determine the nature of attack, thus giving rise to SR, CR, LR, SCR attacks. Thus, one can train for example with SCR constraint but attack with SR constraint. This raises questions about relative strengths of these different training methods and attacks. Some constraint-driven training methods are special cases of others. For example, adversarial training known in the literature can be seen as a form of training with SR constraint, with certain amount of parameter tuning.
We can order some robustness constraints based on their strength, for example, we show that LR implies SR, and SCR implies CR. In this case, training with a stronger constraint (e.g. LR) will protect better against both kinds of attacks (in this case, both SR and LR attacks). Some pairs of constraints cannot be ordered by strength (e.g. SR and SCR, LR and SCR), and in this case, optimising training for a given constraint defends better only against attacks with this same target constraint. Moreover, we show that training with logical constraints defends against adversarial attacks better than data augmentation for any choice of robustness definition as a training constraint. Finally, we show that there are additional common criteria that can be used to qualitatively compare different modes of constraint-driven training, e.g. interpretability, global or local nature. E.g. CR is the most interpretable, but not globally desirable, LR is least interpretable, but globally desirable.
To our knowledge, this is the first systematic study of robustness training from the point of view of the impact of precise formalisation of robustness on training, evaluation and attack. Note that some of the previous work reported on unstable performance of constraint-driven training when defending against attacks (Ayers et al., 2020), which we do not observe in our experiments. Some papers (Fischer et al., 2019) listed and even implemented some kinds of robustness constraints that we study here, but gave no indication of their relative performance. We are not aware of any prior analysis of SR, LR, CR, SCR abstractly as logical constraints.
The paper is organised as follows. Section 2 explains how different robustness constraints arise from different machine learning approaches to constraint-driven training. Section 3 abstractly analysises these robustness definitions, establishing their relative strength, interpretability and applicability. Section 4 shows how these robustness constraints determine different evaluation metrics and attacks, and provides a comprehensive empirical evaluation of the robustness constraints deployed as training constraints and as attacks. Section 5 concludes the paper and outlines future work.

heading: 2 EXISTING TRAINING TECHNIQUES
text: Data augmentation is one of the simplest methods of improving the robustness of a neural network via training (Shorten & Khoshgoftaar, 2019). It is applicable to any transformation of the input (e.g. addition of noise, translation, rotation, scaling) that leaves the output label unchanged. To make the network robust against such a transformation, one augments the dataset with instances sampled via the transformation. Although it may seem that this simple solution has nothing to do with formal logic, it imposes significant choices from the point of view of the constraint specification:
c1. the choice of will reflect our assumptions about admissible range of perturbations;
c2. the choice of the distance function | · − · | that measures the -proximity will reflect our assumptions on desirable geometric properties of the perturbations;
c3. the choice of the sampling method (random sampling, adversarial attacks, generative algorithm, prior knowledge of images etc.) will determine the constraint we optimise for.
But, perhaps even more significantly for us, this method determines the exact definition of robustness that we optimise for when we train our neural network f : Rn → Rm. We call it classification robustness and formally define as follows: given a training dataset input-output pair (x̂,y) and a distance function | · − · |, for all inputs x within the -ball distance of x̂, ensure that class y has the largest score in output f(x). In other words:
Definition 1 (Classification robustness) CR( , x̂) , ∀x : |x− x̂| ≤ ⇒ argmax f(x) = y
Used as a spec for training, this constraint does not account for possibility of having “uncertain” images in the dataset, for which a small perturbation ideally should change the class. For datasets
that contain a significant number of such images, it will lead to significant reduction in accuracy of the trained neural networks; and, as we show later it may even have a detrimental effect on a network’s robustness.
Adversarial training is the current state-of the-art method to robustify a neural network. Whereas standard training tries to minimise loss between the predicted value, f(x̂), and the true value, y, for each entry (x̂,y) in the training dataset, adversarial training minimises the loss with respect to the worst-case perturbation of each sample in the training dataset. It therefore replaces the standard training objective L(x̂,y) with:
max ∀x:|x−x̂|≤
L(x,y) (1)
Algorithmic solutions to the maximisation problem that find the worst-case perturbation has been the subject of several papers. The earliest suggestion was the Fast Gradient Sign Method (FGSM) algorithm introduced by Goodfellow et al. (2015):
FGSM(x̂) = proj(x̂+ · sign(∇xL(x,y)))
However, modern adversarial training methods usual rely on some variant of the Projected Gradient Descent (PGD) algorithm (Gu & Rigazio, 2014) which iterates FGSM some number of times:
PGD0(x̂) = x̂; PGDt+1(x̂) = PGDt(FGSM(x̂))
It has been empirically observed that neural networks trained using this family of methods exhibit greater robustness at the expense of an increased generalization error (Tsipras et al., 2018; Madry et al., 2018; Zhang et al., 2019), which is frequently referred to as the accuracy-robustness tradeoff for neural networks (although this effect has been observed to disappear as the size of the training dataset grows (Raghunathan et al., 2019).
In logical terms what is this procedure trying to train for? Obviously it’s unreasonable to expect that adversarial training will ever succeed in driving the loss of all perturbations down to zero. Therefore let us assume that there’s some maximum distance, δ, that it is acceptable for the output to be perturbed given the size of perturbations in the input. This leads us to the following definition, where || · − · || is a suitable distance function over the output space:
Definition 2 (Standard robustness)
SR( , δ, x̂) , ∀x : |x− x̂| ≤ ⇒ ||f(x)− y|| ≤ δ
In the case of adversarial training the distance between the outputs ||x− y|| is equal to L(x,y). We note that, just as with data augmentation, choices c1 – c3 are still there to be made, although the sampling methods are usually given by special-purpose FGSM/PGD heuristics based on computing the loss function gradients.
Training for Lipschitz robustness. More recently, a third competing definition of robustness has been proposed: Lipschitz robustness (Balan et al., 2018). Inspired by the well-established concept of Lipschitz continuity, Lipschitz robustness asserts that the distance between the original output and the perturbed output is at most a constant L times the change in the distance between the inputs.
Definition 3 (Lipschitz robustness)
LR( , L, x̂) , ∀x : |x− x̂| ≤ ⇒ ||f(x)− y|| ≤ L|x− x̂|
As will be discussed in Section 3, this is a stronger requirement than standard robustness. Techniques for training for Lipschitz robustness include formulating it as a semi-definite programming optimisation problem (Pauli et al., 2021) or including a projection step that restricts the weight matrices to those with suitable Lipschitz constants (Gouk et al., 2021).
Training with logical constraints. Logically, this discussion leads one to ask whether a more general approach to constraint formulation may exist, and several attempts in the literature addressed this research question (Xu et al., 2018; Fischer et al., 2019), by proposing methods that can translate a first-order logical formula C into a constraint loss function LC . The loss function penalises the
network when outputs do not satisfy a given Boolean constraint, and universal quantification is handled by a choice of sampling method. Our standard loss function L is substituted with:
L∗(x̂,y) = αL(x̂,y) + βLC(x̂,y) (2)
where weights α and β control the balance between the standard loss and the constraint loss.
This method looks deceivingly as a generalisation on the previous approaches. However, even given suitable choices for c1 – c3, classification robustness cannot be modelled via a constraint loss in the DL2 framework, as argmax is not differentiable. Instead Fischer et al. (2019) define an alternative constraint we will call strong classification robustness:
Definition 4 (Strong classification robustness)
SCR( , η, x̂) , ∀x : |x− x̂| ≤ ⇒ f(x)c ≥ η
which looks only at the prediction of the true class and checks whether it is greater than some value η (chosen to be 0.52 in their work).
In summary, we have hopefully demonstrated how non-trivial knowledge representation choices and problems arise on the boundary between logical form of the desired constraints and their machinelearning realisations as loss functions. In the next section, we analyse the advantages and disadvantages of each definition in order to help people better make these choices in future.

heading: 3 COMPARISON OF DEFINITIONS
text: Table 1 shows a summary of the points discussed in this section. The first aspect we discuss is the logical relationship between the various definitions, i.e. when the definitions agree and disagree.
Standard and Lipschitz robustness. The easiest relationship to quantify is the one between standard robustness and Lipschitz robustness. In particular, the latter is a strictly stronger constraint than the former, in the sense that when a network satisfies LR( , L) then it also satisfies SR( , L). However, the converse does not hold, as standard robustness does not relate the distances between the inputs and the outputs. Consequently, there are SR( , δ) robust models that are not LR( , L) robust for any L, as for any fixed L one can always make the distance |x − x̂| arbitrarily small in order to violate the Lipschitz inequality.
(Strong) classification robustness. The next relationship we discuss is that of classification robustness and strong classification robustness. As discussed earlier, the latter is designed to overapproximate the former whilst providing a logical loss function with a meaningful gradient. We work under the assumption that the last layer of the classification network is a softmax layer and therefore the output forms a probability distribution. When η > 0.5 then, as would be hoped, any network that satisfies SCR( , η) also satisfies CR( ). For η ≤ 0.5 this relationship breaks down as the true class may be assigned a probability greater than η but may still not be the class with the highest probability. We therefore recommended that one only uses value of η > 0.5 when using strong classification robustness (for example η = 0.52 in Fischer et al. (2019)).
Given that LR is stronger than SR and SCR is stronger than CR, the obvious question is whether there is a relationship between these two groups? In short, the answer to this question is no. In particular, although the two sets of definitions agree whether a network is robust around images with high-confidence, they disagree over whether a network is robust around images with low confidence. We illustrate this with an example, comparing SR against CR. We note that similar analysis holds for any pairing from the two groups.
Standard vs classification robustness. The key insight is that standard robustness bounds the drop in confidence that a neural network can exhibit after a perturbation, whereas classification robustness does not. Figure 1 shows two hypothetical images from the MNIST dataset. Our network predicts that Figure 1a has an 85% chance of being a 7. Now consider adding a small perturbation to the image and consider two different scenarios. In the first scenario the output of the network for class 7 decreases from 85% to 83% and therefore the classifica-
tion stays in the same. In the second scenario the the output of the network for class 7 decreases from 85% to 45%, and results in the classification changing from 7 to 9. When considering the two definitions, a small change in the output leads to no change in the classification and a large change in the output leads to a change in classification and so robustness and classification robustness both agree with each other.
However now consider Figure 1b with relatively high uncertainty. In this case the network is (correctly) less sure about the image, only narrowly deciding that it’s a 7. Again consider adding a small perturbation. In the first scenario the prediction of the network changes dramatically with the probability of it being a 7 increasing from 51% to 91% but leaves the classification unchanged as 7. In the second scenario the output of the network only changes very slightly, decreasing from 51% to 49% flipping the classification from 7 to 9. Now, the definitions of SR and CR disagree. In the first case, adding a small amount of noise has erroneously massively increased the network’s confidence and therefore the SR definition correctly identifies that this is a problem. In contrast CR has no problem with this massive increase in confidence as the chosen output class remains unchanged. Given this, it is clear that although standard and classification robustness agree on low-uncertainty examples, classification robustness breaks down and gives what we argue are both false positives and false negatives when considering examples with high-uncertainty.
Dataset assumptions. A related question that we have found to be rarely discussed in the literature is what assumptions the different definitions are making about the distribution of the training data with respect to the data manifold of the true distribution of inputs.
For SR and LR it is, at minimum, desirable for the network to be robust over the entire data manifold. In the most domains the shape of the manifold is unknown and therefore it is necessary to approximate it by taking the union of the balls around the inputs in the training dataset. We are not particularly interested about whether the network is robust in regions of the input space that lie off the data manifold, but there is no problem if the network is robust in these regions. Therefore these definitions make no assumptions about the distribution of the training dataset.
This is in contrast to (S)CR. As discussed in the previous section, rather than requiring that there is only a small change in the output, they require that there is no change to the classification. This is only a desirable constraint when the region being considered does not contain a decision boundary. Consequently when one is training for some form of classification robustness, one is implicitly making the assumption that the training data points lie away from any decision boundaries within the manifold. In practice, most datasets for classification problems assign a single label instead of an entire probability distribution to each input point, and so this assumption is usually valid. However, we feel it is important to note that classification robustness is not an appropriate definition to train for if the dataset contains input points that may lie close to the decision boundaries and in such a case may result in a logically inconsistent specification.
Interpretability. One of the key selling points of training with logical constraints is that, by ensuring that the network obeys understandable constraints, it improves the explainability of the neural network. Each of the robustness constraints encode that “small changes to the input only result in small changes to the output”, but the interpretability of each definition is also important.
All of the definitions share the relatively interpretable parameter, which measures how large a perturbation from the input is acceptable. Despite the other drawbacks discussed so far, CR is inherently the most interpretable as it has no second parameter. In contrast, SR and SCR require extra parameters, δ and η respectively, which measure the allowable deviation in the output. Their addition make these models less interpretable.
Finally we argue that, although LR is the most desirable constraint, it is also the least interpretable. Its second parameter L measures the allowable change in the output as a proportion of the allowable change in the input. It therefore requires one to not only have an interpretation of distance for both the input and output spaces, but to be able to relate them. In most domains, this relationship simply doesn’t exist. Consider the MNIST dataset, both the commonly used notion of pixel-wise distance used in the input set, although crude, and the distance between the output distributions are both interpretable. However, the relationship between them is not. For example, what does allowing the distance between the output probability distributions being no more than twice the distance between the images actually mean? This therefore highlights a common trade-off between complexity of the constraint and its interpretability.
Another interpretability consideration that appears when training with logical constraints is the semantics of the parameters α and β which allow the users to decide how to weight the accuracy of the network vs the importance that it adheres to the constraint. Section 1 describes how the loss function used in logical-constraint training is split into two parts: L∗(x̂,y) = αL(x̂,y)+βLC(x̂,y), where the first part is designed to maximise the accuracy of the predictions and the second encodes the constraint. However, in practice the constraints (and therefore the derived loss functions) refer to the true label y rather than the current output of the network f(x̂), e.g. ∀x : |x− x̂| ≤ ⇒ |f(x)− y| ≤ δ. This leads to scenarios where a network that is robust around x̂ but gives the wrong prediction, being penalised by LC which on paper is designed to maximise robustness. Essentially LC is trying to maximise both accuracy and constraint adherence concurrently. Instead, we argue that to preserve the intended semantics of α and β it is important to instead compare against the current output of the network e.g. ∀x : |x− x̂| ≤ ⇒ |f(x)− f(x̂)| ≤ δ. Of course, this doesn’t work for SCR because in order to derive the most popular class from the output f(x̂) you need argmax, the very function SCR seeks to avoid using. This is another argument why (S)CR should be avoided if possible.

heading: 4 EXPERIMENTS
text: Evaluation metrics. Given a particular definition of robustness, a natural question is how to quantify how close a given network is to satisfying it? We argue that there are three different measures that one should be interested in:
1. Does the constraint hold? This is a binary measure and the answer is either true or false.
2. If the constraint doesn’t hold, how easy is it for an attacker to find a violation?
3. If the constraint doesn’t hold, how often does the average user encounter a violation?
Based off of these measures, we define three concrete metrics: constraint satisfaction, constraint security, constraint accuracy1. Let X be the training dataset, B(x̂, ) , {x ∈ Rn | |x − x̂| ≤ } be the -ball around x̂ and P be the right hand side of the implication in each of the definitions of robustness. Let Iφ be the standard indicator function which is 1 if constraint φ(x) holds and 0 otherwise. The constraint satisfaction metric measures the proportion of the training dataset for which the constraint holds. Unfortunately, depending on the network architecture and the constraint in question, the indicator function may not always be feasible to evaluate.
1Our naming scheme differs from Fischer et al. (2019) who use the term constraint accuracy (without explicitly defining it in the paper) to refer to what we term constraint security. In our opinion, the term constraint accuracy is less appropriate here than the name constraint security given the use of an adversarial attack.
Definition 5 (Constraint satisfaction)
CS(X ) = 1 |X | ∑ x̂∈X I∀x∈B(x̂, ):P (x)
In contrast, constraint security measures the proportion of inputs in the training dataset such that an attack A is unable to find an adversarial example for constraint P . In our experiments we use the PGD attack for A, although in general any strong attack can be used.
Definition 6 (Constraint security)
CR(X ) = 1 |X | ∑ x̂∈X IP (A(x̂))
Finally constraint accuracy estimates the probability of a random user coming across a counterexample to the constraint, usually referred as 1 - success rate in the robustness literature. Let S(x̂, n) be a set of n elements randomly uniformly sampled from B(x̂, ). Then constraint accuracy follows:
Definition 7 (Constraint accuracy)
CL(X ) = 1 |X | ∑ x̂∈X  1 n ∑ x∈S(x̂,n) IP (x)  Note that there is no relationship between constraint accuracy and constraint security: an attacker may succeed in finding an adversarial example where random sampling fails and vice-versa. Also note the role of sampling in this discussion and compare it to the discussion of the choice c3 in Section 2. Firstly, sampling procedures affect both training and evaluation of networks. But at the same time, their choice is orthogonal to choosing the verification constraint for which we optimise or evaluate. For example, we measure constraint security with respect to the PGD attack, and this determines the way we sample; but having made that choice still leaves us to decide which constraint, SCR, SR, LR, or other we will be measuring as we sample.
Choosing an evaluation metric. It is important to note that for all three evaluation metrics, one still has to make a choice for constraint P , namely SR, SCR or LR, as defined in Section 2. As constraint security always uses PGD to find input perturbations, the choice of SR, SCR and LR effectively amounts to us making a judgement of what an adversarial perturbation consists of: is it a class change as defined by SCR, or is it a violation of the more nuanced metrics defined by SR and LR? Therefore we will evaluate constraint security on the SR/SCR/LR constraints using a PGD attack. This will be further analysed in the experiment E1 below.
For large search spaces in n dimensions, random sampling deployed in constraint accuracy fails to find the trickier adversarial examples, and usually has deceivingly high performance: we found 100% and > 98% constraint accuracy for SR and SCR, respectively. We will therefore not discuss these experiments in detail, but refer the reader to Appendix A.
Constraint satisfaction is different from constraint security and accuracy, in that it must evaluate constraints over infinite domains rather than merely sampling from them. Verifiers such as Marabou or ERAN are built with this purpose in mind. It is out of scope for this paper to conduct experiments by this measure, and so we leave it for future work. Our preliminary experiments with Marabou confirm the generally accepted assumption that it is very hard for training techniques to achieve a high-score for constraint satisfaction.
Networks. We use the FASHION MNIST (or just FASHION) (Xiao et al., 2017) and the GTSRB (Stallkamp et al., 2011) datasets. For our baseline architecture, we use two fully connected layers: the first layer uses the ReLU activation function, and the second uses the clamp function to restrict each output to the range [−100, 100]. We use a clamp function to evaluate the constraints instead of the traditional softmax function because the former is compatible with the constraint verification tools such as Marabou whereas the latter is not. The predicted classification is then taken as the output with the maximum score. However, only during training, we pass the output from the clamp function through a softmax layer before feeding it to the loss. For instance, fMNIST = F0 ◦F1, where F1 : R784 → R100 and F0 : R100 → R10, α0 = ReLU and α1(x) = clamp(x,−100, 100).
Loss functions. Since our experiments study classification problems, we will use the cross-entropy loss function as our baseline loss function:
Lce(x,y) = − m∑ i=1 yi log(f(x)i)
For the LC component of the loss function L∗, we use the constraint-to-loss function translation of Fischer et al. (2019). In all experiments, we use the Adam optimiser (Kingma & Ba, 2015) with the following learning parameters: η = 0.0001, 100 epochs, a batch size of 128.
Settings. We keep the architecture the same throughout the experiments, and only vary the training. Thus our Baseline network is trained just with cross-entropy. Data Augmentation adds 2 additional images in the 0.1- -ball around each image, and it samples either randomly from a uniform distribution (RU) or using an FGSM attack. Adversarial training refers the training procedure described in Section 2, with FGSM sampling. In all other cases, we use a constraint loss function LC defined as in Fischer et al. (2019), and we use the constraints SR, SCR, LR as defined in Section 2 with α = 1, β = 0.2, = 0.1, δ = 10, η = 0.52, L = 10 and L∞ distance metrics. All networks trained with L∗ use sampling by the PGD attack, for efficiency (as well as comparability).
Results. We start with noting the standard test set accuracy of the resulting neural networks in Table 2, making sure that our different training regimes do not deteriorate networks’ general performance too drastically. The most notable accuracy drop occurs for adversarial training.
We now highlight groups of experiments that confirm or extend our main theoretical conclusions; the complete experiment description is available in Appendix A.

heading: Experiment set E1. Comparable constraints.
text: In Section 3, we established that LR as a constraint is stronger than SR, and both are not strictly comparable to SCR. This would suggest that, if we train two neural networks, one with the SR, and the other with the LR constraint, then the latter
should always have higher constraint security against both SR and LR attacks than the former. This is indeed confirmed by the experiment shown in Figure 2. We also discussed that depending on the properties of the dataset, SR may not guarantee SCR, and Figure 2 shows exactly that case. It also confirms that generally, stronger constraints are harder to obtain: whether a network is trained with SR or LR constraints, it is less robust against LR attack than any other attack.
Experiment set E2. Incomparable constraints. The results in Figure 3 tell us that using the SCR constraint for training does not help to increase defences against SR attacks. A similar picture, but in reverse, can be seen when we optimise for SR but attack with SCR, see Appendix A.
Experiment set E3. Constraint training versus data augmentation and adversarial training. Next, we confirm our assumptions about the relative inefficiency of using data augmentation compared to adversarial training or training with constraints, see Figure 4. We show this only for SR attack, but graphs for SL and SCR attacks show the same trends, see Appendix A. Surprisingly, neural networks trained with data augmentation give worse results than even the baseline network. It is encouraging to see that Constraint Loss (SR) is almost as good as adversarial training while the latter also gives 11% drop in terms of the networks accuracy (Table 2).
Experiment set E4. Role of other parameters. As previously discussed, random uniform sampling struggles to find adversarial inputs in large searching spaces. It is logical to expect that using random uniform sampling when training will be less successful than training with sampling that uses FGSM or PGD as heuristics. Indeed, Figure 4 shows this effect for data augmentation, but similar trends are expected for any form of training.
Finally, one may ask whether the trends just described would be replicated for more complex architectures of neural networks. In particular, data augmentation is known to require larger networks. By replicating the results of Figure 4 with a large 18 layer convolutional network of Fischer et al. (2019) indeed confirms that larger networks handle data augmentation better, and data augmentation improves robustness compared to the baseline. Nevertheless, data augmentation still lags behind all other modes of constraint driven training, and thus this major trend remains stable across network architectures, see Appendix A.

heading: 5 CONCLUSIONS AND RELATED WORK
text: We have presented a comprehensive study of constraint-driven training from the formal point of view. Taking robustness as a representative constraint, we abstractly studied different forms of robustness; and showed how the existing literature on constraint-driven training can be understood through this prism. Moreover, we proposed a method that separates out the logical study of constraints from their implementation as loss functions for training on the one hand, and their use as evaluation methods and attacks on the other. We showed that this method allows us to make general conclusions about relations of different modes of constraint-driven training that were not possible before.
For translation of constraints into loss functions, we used the implementation from Fischer et al. (2019); thus our results are compatible and comparable with that prior study. In particular we have identified that the method of Fischer et al. (2019) is not a strict generalisation of the other techniques and that significant trade-offs and decisions have to be made in order to represent some of the robustness definitions.

heading: A APPENDIX
text: This appendix will present all the results that, due to space, were not possible to include in the main text. We will show in detail how we systematically evaluated all the trends that we reported. Figures 5, 8 are the same as Figures 4, 3, and Figures 13A, 14A correspond to Figure 2, we report them again here for completeness.

heading: A.1 DATA AUGMENTATION VS ADVERSARIAL TRAINING VS TRAINING WITH CONSTRAINT LOSS
text: Figures 5, 6 and 7 show how the networks trained with the different methods are robust against attacks. For SR and SCR attacks, Adversarial Training improves significantly the robustness of the model while training with Constraint Loss (SR) also improves it, although not as much. Training with both the Data Augmentation methods actually reduce robustness of the network. None of the training techniques succeed in ensuring robustness against LR attacks, indicating that as discussed in Section 3.1, LR is a strong property to hold. Nonetheless we report the graphs for completeness.

heading: A.2 TRAINING WITH DIFFERENT CONSTRAINT LOSSES
text: Figures 8, 9 and 10 show how the networks trained with the different constraint losses are robust against attacks. For SR and SCR attacks, we can see a trend in the networks trained with the same constraint that are generally more robust against the respective attacks. On the other hand, models trained with LR are generally more robust and they have the best average improvement against all attacks. None of the training techniques succeed in ensuring robustness against LR attacks, except the network trained with LR on the FASHION dataset.

heading: A.3 TRAINING A BIGGER NETWORK
text: Figure 11 show some of the training methods scale with the architecture. On the left we have the small architecture used for all the experiments, while on the right we have a bigger architecture. The
relative behaviour between the training methods remain the same, except from the baseline network that present a decrease in robustness and it exhibits similar behaviour to the models trained with Data Augmentation.

heading: A.4 NETWORKS’ BEHAVIOURS AGAINST DIFFERENT ATTACKS
text: Previous experiments show that the most robust models are the ones trained with Adversarial Training, LR and SR. Figures 12, 13 and 14 select the relevant data from previous experiments to provide a comparison of how these models perform against the different attacks. All the networks struggle the most against LR attacks, while they present significant robustness against SR attacks and a slightly less but still important robustness against SCR attacks. Overall we can see that Adversarial Training provides the best defence against SR and SCR attacks, immediately followed by training with Constraint Loss LR. However, as already reported above, the only network that show some robustness against LR attacks is the one trained with Constraint Loss LR.

Review summaries:
The authors are proposing an approach to systematization of the types of robustness of neural network. In particular, they are discussing four types of robustness: through augmentation, through adversarial training, through Lipschitz constraint training and through logical constraints training. The authors discuss the interconnections between the four classes of robustness, hypothesize which one is more general than others, propose possible measures for how easily the robustness can be violated - randomly or intentionally. In the experimental evaluation the authors are checking the attacks vulnerability for the networks with different types of robustness, confirming discussions about them.

This paper formally summarizes common robustness notions in the literature as: standard robustness (SR), classification robustness (CR), Lipschitz robustness (LR), and strong classification robustness (SCR). The paper proves LR implies SR, SCR implies CR. Then some empirical studies are conducted.