Network robustness as a mathematical property: training, evaluation and attack

1 INTRODUCTION . Safety and security are critical for some complex AI systems involving neural networks , yet they are difficult to ensure . The most famous instance of this problem is guaranteeing robustness against adversarial attacks ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) . Intuitively , an image is -ball robust if , when you move no more than away from it in the input space , the output does not change much , or alternatively , the classification decision that the network gives does not change . Adversarial robustness is a property that even very accurate neural networks fail to satisfy . The proposed solution is to ( re ) train the network with robustness specifically in mind . Such training can be seen as a way to impose a formal specification , and so may contribute to explainability as well as verification . This work considers four of the most prominent families of techniques : 1 . Data augmentation ( Shorten & Khoshgoftaar , 2019 ) 2 . Adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) 3 . Lipschitz robustness training ( Anil et al. , 2019 ; Pauli et al. , 2021 ) 4 . Training with logical constraints ( Xu et al. , 2018 ; Fischer et al. , 2019 ) The last technique , training with logical constraints , is a more general approach that can train for not just robustness , but a wide range of constraints expressed in some logical language . Although the first three families of methods all seek to represent the same high-level knowledge in the neural network , each technique seeks to optimise for subtly different definitions of robustness . We formally identify these as standard ( SR ) , classification ( CR ) , Lipschitz ( LR ) and strong classification ( SCR ) robustness . Given these differences , some natural questions to ask are : What are the relationships between them ? What assumptions do they make about the training dataset ? Are some more effective than others ? Are some more interpretable by users than others ? Contributions . In this work , we attempt to answer these questions . We take classification problems as an example domain , interpreting a neural network f : Rn → Rm as a procedure that separates the n-dimensional data into m classes . This enables the following findings : We observe that from the security perspective , different definitions of robustness ultimately determine the nature of attack , thus giving rise to SR , CR , LR , SCR attacks . Thus , one can train for example with SCR constraint but attack with SR constraint . This raises questions about relative strengths of these different training methods and attacks . Some constraint-driven training methods are special cases of others . For example , adversarial training known in the literature can be seen as a form of training with SR constraint , with certain amount of parameter tuning . We can order some robustness constraints based on their strength , for example , we show that LR implies SR , and SCR implies CR . In this case , training with a stronger constraint ( e.g . LR ) will protect better against both kinds of attacks ( in this case , both SR and LR attacks ) . Some pairs of constraints can not be ordered by strength ( e.g . SR and SCR , LR and SCR ) , and in this case , optimising training for a given constraint defends better only against attacks with this same target constraint . Moreover , we show that training with logical constraints defends against adversarial attacks better than data augmentation for any choice of robustness definition as a training constraint . Finally , we show that there are additional common criteria that can be used to qualitatively compare different modes of constraint-driven training , e.g . interpretability , global or local nature . E.g . CR is the most interpretable , but not globally desirable , LR is least interpretable , but globally desirable . To our knowledge , this is the first systematic study of robustness training from the point of view of the impact of precise formalisation of robustness on training , evaluation and attack . Note that some of the previous work reported on unstable performance of constraint-driven training when defending against attacks ( Ayers et al. , 2020 ) , which we do not observe in our experiments . Some papers ( Fischer et al. , 2019 ) listed and even implemented some kinds of robustness constraints that we study here , but gave no indication of their relative performance . We are not aware of any prior analysis of SR , LR , CR , SCR abstractly as logical constraints . The paper is organised as follows . Section 2 explains how different robustness constraints arise from different machine learning approaches to constraint-driven training . Section 3 abstractly analysises these robustness definitions , establishing their relative strength , interpretability and applicability . Section 4 shows how these robustness constraints determine different evaluation metrics and attacks , and provides a comprehensive empirical evaluation of the robustness constraints deployed as training constraints and as attacks . Section 5 concludes the paper and outlines future work . 2 EXISTING TRAINING TECHNIQUES . Data augmentation is one of the simplest methods of improving the robustness of a neural network via training ( Shorten & Khoshgoftaar , 2019 ) . It is applicable to any transformation of the input ( e.g . addition of noise , translation , rotation , scaling ) that leaves the output label unchanged . To make the network robust against such a transformation , one augments the dataset with instances sampled via the transformation . Although it may seem that this simple solution has nothing to do with formal logic , it imposes significant choices from the point of view of the constraint specification : c1 . the choice of will reflect our assumptions about admissible range of perturbations ; c2 . the choice of the distance function | · − · | that measures the -proximity will reflect our assumptions on desirable geometric properties of the perturbations ; c3 . the choice of the sampling method ( random sampling , adversarial attacks , generative algorithm , prior knowledge of images etc . ) will determine the constraint we optimise for . But , perhaps even more significantly for us , this method determines the exact definition of robustness that we optimise for when we train our neural network f : Rn → Rm . We call it classification robustness and formally define as follows : given a training dataset input-output pair ( x̂ , y ) and a distance function | · − · | , for all inputs x within the -ball distance of x̂ , ensure that class y has the largest score in output f ( x ) . In other words : Definition 1 ( Classification robustness ) CR ( , x̂ ) , ∀x : |x− x̂| ≤ ⇒ argmax f ( x ) = y Used as a spec for training , this constraint does not account for possibility of having “ uncertain ” images in the dataset , for which a small perturbation ideally should change the class . For datasets that contain a significant number of such images , it will lead to significant reduction in accuracy of the trained neural networks ; and , as we show later it may even have a detrimental effect on a network ’ s robustness . Adversarial training is the current state-of the-art method to robustify a neural network . Whereas standard training tries to minimise loss between the predicted value , f ( x̂ ) , and the true value , y , for each entry ( x̂ , y ) in the training dataset , adversarial training minimises the loss with respect to the worst-case perturbation of each sample in the training dataset . It therefore replaces the standard training objective L ( x̂ , y ) with : max ∀x : |x−x̂|≤ L ( x , y ) ( 1 ) Algorithmic solutions to the maximisation problem that find the worst-case perturbation has been the subject of several papers . The earliest suggestion was the Fast Gradient Sign Method ( FGSM ) algorithm introduced by Goodfellow et al . ( 2015 ) : FGSM ( x̂ ) = proj ( x̂+ · sign ( ∇xL ( x , y ) ) ) However , modern adversarial training methods usual rely on some variant of the Projected Gradient Descent ( PGD ) algorithm ( Gu & Rigazio , 2014 ) which iterates FGSM some number of times : PGD0 ( x̂ ) = x̂ ; PGDt+1 ( x̂ ) = PGDt ( FGSM ( x̂ ) ) It has been empirically observed that neural networks trained using this family of methods exhibit greater robustness at the expense of an increased generalization error ( Tsipras et al. , 2018 ; Madry et al. , 2018 ; Zhang et al. , 2019 ) , which is frequently referred to as the accuracy-robustness tradeoff for neural networks ( although this effect has been observed to disappear as the size of the training dataset grows ( Raghunathan et al. , 2019 ) . In logical terms what is this procedure trying to train for ? Obviously it ’ s unreasonable to expect that adversarial training will ever succeed in driving the loss of all perturbations down to zero . Therefore let us assume that there ’ s some maximum distance , δ , that it is acceptable for the output to be perturbed given the size of perturbations in the input . This leads us to the following definition , where || · − · || is a suitable distance function over the output space : Definition 2 ( Standard robustness ) SR ( , δ , x̂ ) , ∀x : |x− x̂| ≤ ⇒ ||f ( x ) − y|| ≤ δ In the case of adversarial training the distance between the outputs ||x− y|| is equal to L ( x , y ) . We note that , just as with data augmentation , choices c1 – c3 are still there to be made , although the sampling methods are usually given by special-purpose FGSM/PGD heuristics based on computing the loss function gradients . Training for Lipschitz robustness . More recently , a third competing definition of robustness has been proposed : Lipschitz robustness ( Balan et al. , 2018 ) . Inspired by the well-established concept of Lipschitz continuity , Lipschitz robustness asserts that the distance between the original output and the perturbed output is at most a constant L times the change in the distance between the inputs . Definition 3 ( Lipschitz robustness ) LR ( , L , x̂ ) , ∀x : |x− x̂| ≤ ⇒ ||f ( x ) − y|| ≤ L|x− x̂| As will be discussed in Section 3 , this is a stronger requirement than standard robustness . Techniques for training for Lipschitz robustness include formulating it as a semi-definite programming optimisation problem ( Pauli et al. , 2021 ) or including a projection step that restricts the weight matrices to those with suitable Lipschitz constants ( Gouk et al. , 2021 ) . Training with logical constraints . Logically , this discussion leads one to ask whether a more general approach to constraint formulation may exist , and several attempts in the literature addressed this research question ( Xu et al. , 2018 ; Fischer et al. , 2019 ) , by proposing methods that can translate a first-order logical formula C into a constraint loss function LC . The loss function penalises the network when outputs do not satisfy a given Boolean constraint , and universal quantification is handled by a choice of sampling method . Our standard loss function L is substituted with : L∗ ( x̂ , y ) = αL ( x̂ , y ) + βLC ( x̂ , y ) ( 2 ) where weights α and β control the balance between the standard loss and the constraint loss . This method looks deceivingly as a generalisation on the previous approaches . However , even given suitable choices for c1 – c3 , classification robustness can not be modelled via a constraint loss in the DL2 framework , as argmax is not differentiable . Instead Fischer et al . ( 2019 ) define an alternative constraint we will call strong classification robustness : Definition 4 ( Strong classification robustness ) SCR ( , η , x̂ ) , ∀x : |x− x̂| ≤ ⇒ f ( x ) c ≥ η which looks only at the prediction of the true class and checks whether it is greater than some value η ( chosen to be 0.52 in their work ) . In summary , we have hopefully demonstrated how non-trivial knowledge representation choices and problems arise on the boundary between logical form of the desired constraints and their machinelearning realisations as loss functions . In the next section , we analyse the advantages and disadvantages of each definition in order to help people better make these choices in future .

Summary: The authors are proposing an approach to systematization of the types of robustness of neural network. In particular, they are discussing four types of robustness: through augmentation, through adversarial training, through Lipschitz constraint training and through logical constraints training. The authors discuss the interconnections between the four classes of robustness, hypothesize which one is more general than others, propose possible measures for how easily the robustness can be violated - randomly or intentionally. In the experimental evaluation the authors are checking the attacks vulnerability for the networks with different types of robustness, confirming discussions about them.