Network robustness as a mathematical property: training, evaluation and attack

1 INTRODUCTION . Safety and security are critical for some complex AI systems involving neural networks , yet they are difficult to ensure . The most famous instance of this problem is guaranteeing robustness against adversarial attacks ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) . Intuitively , an image is -ball robust if , when you move no more than away from it in the input space , the output does not change much , or alternatively , the classification decision that the network gives does not change . Adversarial robustness is a property that even very accurate neural networks fail to satisfy . The proposed solution is to ( re ) train the network with robustness specifically in mind . Such training can be seen as a way to impose a formal specification , and so may contribute to explainability as well as verification . This work considers four of the most prominent families of techniques : 1 . Data augmentation ( Shorten & Khoshgoftaar , 2019 ) 2 . Adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) 3 . Lipschitz robustness training ( Anil et al. , 2019 ; Pauli et al. , 2021 ) 4 . Training with logical constraints ( Xu et al. , 2018 ; Fischer et al. , 2019 ) The last technique , training with logical constraints , is a more general approach that can train for not just robustness , but a wide range of constraints expressed in some logical language . Although the first three families of methods all seek to represent the same high-level knowledge in the neural network , each technique seeks to optimise for subtly different definitions of robustness . We formally identify these as standard ( SR ) , classification ( CR ) , Lipschitz ( LR ) and strong classification ( SCR ) robustness . Given these differences , some natural questions to ask are : What are the relationships between them ? What assumptions do they make about the training dataset ? Are some more effective than others ? Are some more interpretable by users than others ? Contributions . In this work , we attempt to answer these questions . We take classification problems as an example domain , interpreting a neural network f : Rn → Rm as a procedure that separates the n-dimensional data into m classes . This enables the following findings : We observe that from the security perspective , different definitions of robustness ultimately determine the nature of attack , thus giving rise to SR , CR , LR , SCR attacks . Thus , one can train for example with SCR constraint but attack with SR constraint . This raises questions about relative strengths of these different training methods and attacks . Some constraint-driven training methods are special cases of others . For example , adversarial training known in the literature can be seen as a form of training with SR constraint , with certain amount of parameter tuning . We can order some robustness constraints based on their strength , for example , we show that LR implies SR , and SCR implies CR . In this case , training with a stronger constraint ( e.g . LR ) will protect better against both kinds of attacks ( in this case , both SR and LR attacks ) . Some pairs of constraints can not be ordered by strength ( e.g . SR and SCR , LR and SCR ) , and in this case , optimising training for a given constraint defends better only against attacks with this same target constraint . Moreover , we show that training with logical constraints defends against adversarial attacks better than data augmentation for any choice of robustness definition as a training constraint . Finally , we show that there are additional common criteria that can be used to qualitatively compare different modes of constraint-driven training , e.g . interpretability , global or local nature . E.g . CR is the most interpretable , but not globally desirable , LR is least interpretable , but globally desirable . To our knowledge , this is the first systematic study of robustness training from the point of view of the impact of precise formalisation of robustness on training , evaluation and attack . Note that some of the previous work reported on unstable performance of constraint-driven training when defending against attacks ( Ayers et al. , 2020 ) , which we do not observe in our experiments . Some papers ( Fischer et al. , 2019 ) listed and even implemented some kinds of robustness constraints that we study here , but gave no indication of their relative performance . We are not aware of any prior analysis of SR , LR , CR , SCR abstractly as logical constraints . The paper is organised as follows . Section 2 explains how different robustness constraints arise from different machine learning approaches to constraint-driven training . Section 3 abstractly analysises these robustness definitions , establishing their relative strength , interpretability and applicability . Section 4 shows how these robustness constraints determine different evaluation metrics and attacks , and provides a comprehensive empirical evaluation of the robustness constraints deployed as training constraints and as attacks . Section 5 concludes the paper and outlines future work . 2 EXISTING TRAINING TECHNIQUES . Data augmentation is one of the simplest methods of improving the robustness of a neural network via training ( Shorten & Khoshgoftaar , 2019 ) . It is applicable to any transformation of the input ( e.g . addition of noise , translation , rotation , scaling ) that leaves the output label unchanged . To make the network robust against such a transformation , one augments the dataset with instances sampled via the transformation . Although it may seem that this simple solution has nothing to do with formal logic , it imposes significant choices from the point of view of the constraint specification : c1 . the choice of will reflect our assumptions about admissible range of perturbations ; c2 . the choice of the distance function | · − · | that measures the -proximity will reflect our assumptions on desirable geometric properties of the perturbations ; c3 . the choice of the sampling method ( random sampling , adversarial attacks , generative algorithm , prior knowledge of images etc . ) will determine the constraint we optimise for . But , perhaps even more significantly for us , this method determines the exact definition of robustness that we optimise for when we train our neural network f : Rn → Rm . We call it classification robustness and formally define as follows : given a training dataset input-output pair ( x̂ , y ) and a distance function | · − · | , for all inputs x within the -ball distance of x̂ , ensure that class y has the largest score in output f ( x ) . In other words : Definition 1 ( Classification robustness ) CR ( , x̂ ) , ∀x : |x− x̂| ≤ ⇒ argmax f ( x ) = y Used as a spec for training , this constraint does not account for possibility of having “ uncertain ” images in the dataset , for which a small perturbation ideally should change the class . For datasets that contain a significant number of such images , it will lead to significant reduction in accuracy of the trained neural networks ; and , as we show later it may even have a detrimental effect on a network ’ s robustness . Adversarial training is the current state-of the-art method to robustify a neural network . Whereas standard training tries to minimise loss between the predicted value , f ( x̂ ) , and the true value , y , for each entry ( x̂ , y ) in the training dataset , adversarial training minimises the loss with respect to the worst-case perturbation of each sample in the training dataset . It therefore replaces the standard training objective L ( x̂ , y ) with : max ∀x : |x−x̂|≤ L ( x , y ) ( 1 ) Algorithmic solutions to the maximisation problem that find the worst-case perturbation has been the subject of several papers . The earliest suggestion was the Fast Gradient Sign Method ( FGSM ) algorithm introduced by Goodfellow et al . ( 2015 ) : FGSM ( x̂ ) = proj ( x̂+ · sign ( ∇xL ( x , y ) ) ) However , modern adversarial training methods usual rely on some variant of the Projected Gradient Descent ( PGD ) algorithm ( Gu & Rigazio , 2014 ) which iterates FGSM some number of times : PGD0 ( x̂ ) = x̂ ; PGDt+1 ( x̂ ) = PGDt ( FGSM ( x̂ ) ) It has been empirically observed that neural networks trained using this family of methods exhibit greater robustness at the expense of an increased generalization error ( Tsipras et al. , 2018 ; Madry et al. , 2018 ; Zhang et al. , 2019 ) , which is frequently referred to as the accuracy-robustness tradeoff for neural networks ( although this effect has been observed to disappear as the size of the training dataset grows ( Raghunathan et al. , 2019 ) . In logical terms what is this procedure trying to train for ? Obviously it ’ s unreasonable to expect that adversarial training will ever succeed in driving the loss of all perturbations down to zero . Therefore let us assume that there ’ s some maximum distance , δ , that it is acceptable for the output to be perturbed given the size of perturbations in the input . This leads us to the following definition , where || · − · || is a suitable distance function over the output space : Definition 2 ( Standard robustness ) SR ( , δ , x̂ ) , ∀x : |x− x̂| ≤ ⇒ ||f ( x ) − y|| ≤ δ In the case of adversarial training the distance between the outputs ||x− y|| is equal to L ( x , y ) . We note that , just as with data augmentation , choices c1 – c3 are still there to be made , although the sampling methods are usually given by special-purpose FGSM/PGD heuristics based on computing the loss function gradients . Training for Lipschitz robustness . More recently , a third competing definition of robustness has been proposed : Lipschitz robustness ( Balan et al. , 2018 ) . Inspired by the well-established concept of Lipschitz continuity , Lipschitz robustness asserts that the distance between the original output and the perturbed output is at most a constant L times the change in the distance between the inputs . Definition 3 ( Lipschitz robustness ) LR ( , L , x̂ ) , ∀x : |x− x̂| ≤ ⇒ ||f ( x ) − y|| ≤ L|x− x̂| As will be discussed in Section 3 , this is a stronger requirement than standard robustness . Techniques for training for Lipschitz robustness include formulating it as a semi-definite programming optimisation problem ( Pauli et al. , 2021 ) or including a projection step that restricts the weight matrices to those with suitable Lipschitz constants ( Gouk et al. , 2021 ) . Training with logical constraints . Logically , this discussion leads one to ask whether a more general approach to constraint formulation may exist , and several attempts in the literature addressed this research question ( Xu et al. , 2018 ; Fischer et al. , 2019 ) , by proposing methods that can translate a first-order logical formula C into a constraint loss function LC . The loss function penalises the network when outputs do not satisfy a given Boolean constraint , and universal quantification is handled by a choice of sampling method . Our standard loss function L is substituted with : L∗ ( x̂ , y ) = αL ( x̂ , y ) + βLC ( x̂ , y ) ( 2 ) where weights α and β control the balance between the standard loss and the constraint loss . This method looks deceivingly as a generalisation on the previous approaches . However , even given suitable choices for c1 – c3 , classification robustness can not be modelled via a constraint loss in the DL2 framework , as argmax is not differentiable . Instead Fischer et al . ( 2019 ) define an alternative constraint we will call strong classification robustness : Definition 4 ( Strong classification robustness ) SCR ( , η , x̂ ) , ∀x : |x− x̂| ≤ ⇒ f ( x ) c ≥ η which looks only at the prediction of the true class and checks whether it is greater than some value η ( chosen to be 0.52 in their work ) . In summary , we have hopefully demonstrated how non-trivial knowledge representation choices and problems arise on the boundary between logical form of the desired constraints and their machinelearning realisations as loss functions . In the next section , we analyse the advantages and disadvantages of each definition in order to help people better make these choices in future .

Summary: This paper formally summarizes common robustness notions in the literature as: standard robustness (SR), classification robustness (CR), Lipschitz robustness (LR), and strong classification robustness (SCR). The paper proves LR implies SR, SCR implies CR. Then some empirical studies are conducted.