Robustness Verification for Transformers

1 INTRODUCTION . Deep neural networks have been successfully applied to many domains . However , these blackbox models are generally difficult to analyze and their behavior is not guaranteed . Moreover , it has been shown that the predictions of deep networks become unreliable and unstable when tested in unseen situations , e.g. , in the presence of small adversarial perturbations to the input ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Lin et al. , 2019 ) . Therefore , neural network verification has become an important tool for analyzing and understanding the behavior of neural networks , with applications in safety-critical applications ( Katz et al. , 2017 ; Julian et al. , 2019 ; Lin et al. , 2019 ) , model explanation ( Shih et al. , 2018 ) and robustness analysis ( Tjeng et al. , 2019 ; Wang et al. , 2018c ; Gehr et al. , 2018 ; Wong & Kolter , 2018 ; Singh et al. , 2018 ; Weng et al. , 2018 ; Zhang et al. , 2018 ) . Formally , a neural network verification algorithm aims to provably characterize the prediction of a network within some input space . For example , given a K-way classification model f : Rd → RK , where fi ( x ) stands for the predicted score of class i , we can verify some linear specification ( defined by a vector c ) as below : min x ∑ i cifi ( x ) s.t . x ∈ S , ( 1 ) where S is a predefined input space . In the robustness verification problem , S = { x | ‖x−x0‖p ≤ } is defined as some small ` p-ball around the original example x0 , and setting up c = 1y0 − 1y enables us to verify whether the logit output of class y0 is always greater than another class y for any input within S. This is a nonconvex optimization problem which makes computing the exact solution challenging , and thus several algorithms are recently proposed to find the lower bounds of Eq . ( 1 ) in order to efficiently obtain a safety guarantee ( Gehr et al. , 2018 ; Weng et al. , 2018 ; Zhang et al. , 2018 ; Singh et al. , 2019 ) . Moreover , extensions of these algorithms can be used for verifying properties beyond robustness , such as rotation or shift invariant ( Singh et al. , 2019 ) , conservation of energy ( Qin et al. , 2019 ) and model correctness ( Yang & Rinard , 2019 ) . However , most of existing verification methods focus on relatively simple neural network architectures , such as feed-forward and recurrent neural networks , while they can not handle complex structures . In this paper , we develop the first robustness verification algorithm for Transformers ( Vaswani et al. , 2017 ) with self-attention layers . Transformers have been widely used in natural language processing ( Devlin et al. , 2019 ; Yang et al. , 2019 ; Liu et al. , 2019 ) and many other domains ( Parmar et al. , 2018 ; Kang & McAuley , 2018 ; Li et al. , 2019b ; Su et al. , 2019 ; Li et al. , 2019a ) . For frames under perturbation in the input sequence , we aim to compute a lower bound such that when these frames are perturbed within ` p-balls centered at the original frames respectively and with a radius of , the model prediction is certified to be unchanged . To compute such bounds efficiently , we adopt the linear-relaxation framework ( Weng et al. , 2018 ; Zhang et al. , 2018 ) – we recursively propagate and compute linear lower and upper bounds for each neuron w.r.t the input within the perturbation space S. We resolve several particular challenges in verifying Transformers . First , Transformers with selfattention layers have a complicated architecture . Unlike simpler networks , they can not be written as multiple layers of affine transformations or element-wise activation functions . Therefore , we need to propagate linear bounds differently for self-attention layers . Second , dot products , softmax , and weighted summation in self-attention layers involve multiplication or division of two variables both under perturbation , namely cross-nonlinearity , which is not present in feed-forward networks . Ko et al . ( 2019 ) proposed a gradient descent based approach to find linear bounds , however it is inefficient and poses a computational challenge for Transformer verification since self-attention is the core of Transformers . In contrast , we derive closed-form linear bounds that can be computed in O ( 1 ) complexity . Third , in the computation of self-attention , output neurons in each position depend on all input neurons from different positions ( namely cross-position dependency ) , unlike the case in recurrent neural networks where outputs depend on only the hidden features from the previous position and the current input . Previous works ( Zhang et al. , 2018 ; Weng et al. , 2018 ; Ko et al. , 2019 ) have to track all such dependency and thus is costly in time and memory . To tackle this , we introduce an efficient bound propagating process in a forward manner specially for self-attention layers , enabling the tighter backward bounding process for other layers to utilize bounds computed by the forward process . In this way , we avoid cross-position dependency in the backward process which is relatively slower but produces tighter bounds . Combined with the forward process , the complexity of the backward process is reduced by O ( n ) for input length n , while the computed bounds remain comparably tight . Our contributions are summarized below : • We propose an effective and efficient algorithm for verifying the robustness of Transformers with self-attention layers . To our best knowledge , this is the first method for verifying Transformers . • We resolve key challenges in verifying Transformers , including cross-nonlinearity and cross-position dependency . Our bounds are significantly tighter than those by adapting Interval Bound Propagation ( IBP ) ( Mirman et al. , 2018 ; Gowal et al. , 2018 ) . • We quantitatively and qualitatively show that the certified bounds computed by our algorithm consistently reflect the importance of input words in sentiment analysis , which justifies that these bounds are meaningful in practice and they shed light on interpreting Transformers . 2 RELATED WORK . Robustness Verification for Neural Networks . Given an input x0 and a small region Bp ( x0 , ) : = { x | ‖x − x0‖p ≤ } , the goal of robustness verification is to verify whether the prediction of the neural network is unchanged within this region . This problem can be mathematically formulated as Eq . ( 1 ) . If Eq . ( 1 ) can be solved optimally , then we can derive the minimum adversarial perturbation of x by conducting binary search on . Equivalently , we obtain the maximum such that any perturbation within Bp ( x0 , ) can not change the predicted label . Several works focus on solving Eq . ( 1 ) exactly and optimally , using mixed integer linear programming ( MILP ) ( Tjeng et al. , 2019 ; Dutta et al. , 2018 ) , branch and bound ( BaB ) ( Bunel et al. , 2018 ) , and satisfiability modulo theory ( SMT ) ( Ehlers , 2017 ; Katz et al. , 2017 ) . Unfortunately , due to the nonconvexity of model f , solving Eq . ( 1 ) is NP-hard even for a simple ReLU network ( Katz et al. , 2017 ) . Therefore , we can only expect to compute a lower bound of Eq . ( 1 ) efficiently by using relaxations . Many algorithms can be seen as using convex relaxations for non-linear activation functions ( Salman et al. , 2019 ) , including using duality ( Wong & Kolter , 2018 ; Dvijotham et al. , 2018 ) , abstract domains ( Gehr et al. , 2018 ; Singh et al. , 2018 ; Mirman et al. , 2018 ; Singh et al. , 2019 ) , layer-by-layer reachability analysis ( Wang et al. , 2018b ; Weng et al. , 2018 ; Zhang et al. , 2018 ; Gowal et al. , 2018 ) and semi-definite relaxations ( Raghunathan et al. , 2018 ; Dvijotham et al. , 2019 ) . Additionally , robustness verification can rely on analysis on local Lipschitz constants ( Hein & Andriushchenko , 2017 ; Zhang et al. , 2019 ) . However , existing methods are mostly limited to verifying networks with relatively simple architectures , such as feed-forward networks and RNNs ( Wang et al. , 2018a ; Akintunde et al. , 2019 ; Ko et al. , 2019 ) , while none of them are able to handle Transformers . Transformers and Self-Attentive Models . Transformers ( Vaswani et al. , 2017 ) based on selfattention mechanism , further with pre-training on large-scale corpora , such as BERT ( Devlin et al. , 2019 ) , XLNet ( Yang et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , achieved state-of-the-art performance on many NLP tasks . Self-attentive models are also useful beyond NLP , including VisualBERT on vision and language applications ( Li et al. , 2019b ; Su et al. , 2019 ) , image transformer for image generation ( Parmar et al. , 2018 ) , acoustic models for speech recognition ( Zhou et al. , 2018 ) , sequential recommendation ( Kang & McAuley , 2018 ) and graph embedding ( Li et al. , 2019a ) . The robustness of NLP models has been studied , especially many methods have been proposed to generate adversarial examples ( Papernot et al. , 2016 ; Jia & Liang , 2017 ; Zhao et al. , 2017 ; Alzantot et al. , 2018 ; Cheng et al. , 2018 ; Ebrahimi et al. , 2018 ; Shi et al. , 2019 ) . In particular , Hsieh et al . ( 2019 ) showed that Transformers are more robust than LSTMs . However , there is not much work on robustness verification for NLP models . Ko et al . ( 2019 ) verified RNN/LSTM . Jia et al . ( 2019 ) ; Huang et al . ( 2019 ) used Interval Bound Propagation ( IBP ) for certified robustness training of CNN and LSTM . In this paper , we propose the first verification method for Transformers . 3 METHODOLOGY . We aim to verify the robustness of a Transformer whose input is a sequence of frames X = [ x ( 1 ) , x ( 2 ) , · · · , x ( n ) ] . We take binary text classification as a running example , where x ( i ) is a word embedding and the model outputs a score yc ( X ) for each class c ( c ∈ { 0 , 1 } ) . Nevertheless , our method for verifying Transformers is general and can also be applied in other applications . For a clean input sequence X0 = [ x ( 1 ) 0 , x ( 2 ) 0 , · · · , x ( n ) 0 ] correctly classified by the model , let P = { r1 , r2 , · · · , rt } ( 1 ≤ rk ≤ n ) be the set of perturbed positions , where t is the number of perturbed positions . Thus the perturbed input belongs to S : = { X = [ x ( 1 ) , x ( 2 ) , · · · , x ( n ) ] : ‖x ( rk ) − x ( rk ) 0 ‖p ≤ , 1 ≤ k ≤ t , x ( i ) = x ( i ) 0 , ∀i /∈ P } . Assuming that c is the gold class , the goal of robustness verification is to compute { min X∈S yc ( X ) − y1−c ( X ) } : = δ . If δ > 0 , the output score of the correct class is always larger than the incorrect one for any input within S . As mentioned previously , computing the exact values of δ is NP-hard , and thus our goal is to efficiently compute a lower bound δL ≤ δ .

Summary: This paper builds upon the CROWN framework (Zhang et al 2018) to provide robustness verification for transformers. The  CROWN framework is based upon the idea of propagating linear bounds and has been applied to architectures like MLP, CNNs and RNNs. However, in Transformers, the presence of cross-nonlinearities and cross-position dependencies makes the backward propagation of bounds in CROWN computationally intensive. A major contribution of this paper is to use forward propagation of bounds in self attention layers along with the usual back-propagation of bounds in all other layers. The proposed method provides overall reduction in computational complexity by a factor of O(n). Although the fully forward propagation leads to loose bounds, the mixed approach (forward-backward) presented in this work provides bounds which are as tight as fully backward method.