Certified Robustness for Deep Equilibrium Models via Interval Bound Propagation

1 INTRODUCTION . A recent development in neural network design has been the introduction of implicit layers ( Amos & Kolter , 2017 ; Chen et al. , 2018 ; Agrawal et al. , 2019 ; Bai et al. , 2019 ; 2020 ; El Ghaoui et al. , 2021 ) , where the output is defined implicity as the solution to certain sets of conditions , rather than explicitly , via closed-form functions . These layers are promising alternatives to standard explicit deep learning layers and have demonstrated improved expressivity and inductive biases in a variety of settings , for example , processing time series ( Rubanova et al. , 2019 ) , generative modeling ( Grathwohl et al. , 2018 ) , solving logical reasoning tasks ( Wang et al. , 2019 ) , solving two player games ( Ling et al. , 2018 ) , and many others . One particularly promising class of implicit layers is deep equilibrium layers ( DEQs ) ( Bai et al. , 2019 ) , which define the output as the solution to an input-dependent fixed point equation . DEQ-based models have matched or outperformed traditional explicit models even in commonly benchmarked settings ( Bai et al. , 2019 ; 2020 ) . Though recent empirical successes of DEQs have been promising , their implicit nature and inherent mathematical complexity also give rise to basic concerns . In order for DEQs to realize their promise , these concerns should ideally be mitigated or resolved . For example , one major issue with DEQs is well-posedness – a solution to the fixed point equation defining the layer might not exist . On the other hand , explicit layers always have well-defined outputs . A recent line of work has focused on addressing this important concern ( Winston & Kolter , 2020 ; Revay et al. , 2020 ; Xie et al. , 2021 ) . This paper tackles a less-studied , but also important , question for DEQs : certified adversarial robustness . Because robustness is a basic concern for safe deployment of deep models ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) , for explicit models there is a large literature dedicated to certifying robustness , or guaranteeing correctness of the predictions even when the input is subject to imperceptible adversarial perturbations ( see e.g . ( Raghunathan et al. , 2018a ; Wong & Kolter , 2018 ; Gowal et al. , 2018 ; Dvijotham et al . ; Xiao et al. , 2018 ; Cohen et al. , 2019 ) ) . Many certified robustness methods require opening up the black box of the model and therefore only work for explicit models . It is unclear how to certify robustness of DEQs , which are only defined implicitly . The certified robustness method motivating this work is interval bound propagation ( IBP ) ( Mirman et al. , 2018 ; Gowal et al. , 2018 ) , a simple and cheap way to certify robustness to ` 8 perturbations . IBP computes layerwise upper and lower bounds for each coordinate of the adversarially perturbed hidden layers . These bounds follow basic rules of interval arithmetic and are simple to obtain in closed form as explicit functions of a layer ’ s weights . However , it is unclear how to apply this idea when layers are only defined implicitly . In this paper , we propose the IBP-MonDEQ , a deep equilibrium layer which is certifiably robust to ` 8 perturbations . Motivated by principles from IBP , we define the IBP-MonDEQ output as the solution of an augmented fixed-point equation involving 3 quantities : the unperturbed output of the layer and upper and lower interval bounds on this output . As with IBP for explicit models , interval bounds on the IBP-MonDEQ output are computed during the forward pass of the network and can be composed with other layers to certify robustness of the entire model . More concretely , we build upon monotone operator deep equilibrium ( MonDEQ ) layers proposed by Winston & Kolter ( 2020 ) , which take the preceding layer vpxq as input and outputs the solution z‹ , which is guaranteed existence and uniqueness , to the following fixed-point equation : z‹ “ σpWz‹ ` vpxqq ( 1.1 ) We propose an IBP-inspired fixed-point equation „ sz‹ z‹ “ σ ˆ „ pW q ` pW q´ pW q´ pW q ` „ sz‹ z‹ ` „ svpxq vpxq ˙ ( 1.2 ) which maps upper and lower interval bounds sv , v on v to sz‹ and z‹ , which provide upper and lower interval bounds on z‹ . Figure 1 depicts this process . The augmented fixed-point equation is derived by unrolling the computation of z‹ into an infinitely deep , explicit network and applying IBP to the forward pass of this network . One immediate challenge is that it is not clear that a fixed point solution to ( 1.2 ) should always exist , especially given its interpretation as the result of applying IBP to an infinitely deep network . Indeed , a major drawback of IBP is that its performance degrades with deeper networks , as observed by Shi et al . ( 2021 ) and also shown in Figure 2 ( left ) . One potential explanation for this failure is that IBP bounds tend to be unstable with depth and can diverge for deeper models ( Figure 2 , right ) . On the other hand , we show that a unique fixed-point solution to ( 1.2 ) is guaranteed to exist whenW admits a simple unconstrained parameterization which is easy to enforce throughout training . Thus , our results pinpoint a class of infinitely deep networks for which IBP bounds are provably stable , which may be of independent interest . We experimentally compare the proposed IBP-MonDEQ layer against IBP for standard explicit models . We consider common certified robustness benchmark settings and evaluate architectures of various sizes . Our results show that models with IBP-MonDEQ layers can achieve comparable or better ` 8 certified robustness relative to fully explicit models with similar parameter counts . In summary , our contributions are as follows : 1 ) We study the certified robustness of DEQs , proposing the IBP-MonDEQ , a class of DEQs with a guaranteed unique fixed point with provable interval bounds on the fixed point value . 2 ) The proposed IBP-MonDEQs form an expressive class of infinitely-deep models for which IBP is provably stable , which may be of independent interest . 3 ) Our experiments demonstrate that IBP-MonDEQ layers are competitive with standard explicit layers for ` 8-certified robustness . 2 BACKGROUND . ` 8 certified robustness . Consider a K-way classification task with neural network classifier F : Rd Ñ RK . For a given input x with true label y , the classifier F is adversarially robust to ` 8 perturbations with radius if min δPRd : } δ } 8ď F px ` δqy ´ F px ` δqy1 ą 0 @ y1 ‰ y ( 2.1 ) Safety-critical applications require certifying the robustness of F , i.e. , verifying whether 2.1 holds . Directly optimizing over δ to verify ( 2.1 ) is challenging because the objective is non-convex ( Madry et al. , 2017 ) . Thus , recent work on certified robustness has focused on verifying ( 2.1 ) via computationally tractable relaxations ( Raghunathan et al. , 2018a ; Wong & Kolter , 2018 ; Dvijotham et al . ; Raghunathan et al. , 2018b ; Weng et al. , 2018 ; Gowal et al. , 2018 ; Salman et al. , 2019b ) . Interval bound propagation . IBP is a computationally efficient method for certifying ` 8 robustness of neural networks ( Mirman et al. , 2018 ; Gowal et al. , 2018 ) . It proposes to verify ( 2.1 ) via a ( potentially loose ) upper bound on F px ` δqy ´ F px ` δqy1 which is obtained by computing propagating upper and lower bounds on each layer through the forward pass of the network . More precisely , let zpxq compute some hidden layer of the network on input x . We say that szpx , q , zpx , q are interval bounds on z at x for perturbation if the following holds for all coordinates i : pzpx , qqi ď min δPRd : } δ } 8ď pzpx ` δqqi ď max δPRd : } δ } 8ď pzpx ` δqqi ď pszpx , qqi ( 2.2 ) We omit dependencies on x and when clear from context . Letting k denote the layer index , the bounds szk and zk are obtained inductively via simple interval arithmetic . For an affine layer zk “ Wzk´1 ` b and pre-computed bounds zk´1 , szk´1 , IBP computes szk , zk as follows : „ szk zk “ „ pW q ` pW q´ pW q´ pW q ` „ szk´1 zk´1 ` „ b b ( 2.3 ) Here pW q ` fi maxtW , 0u and pW q´ fi mintW , 0u denote the matrix W with negative or positive values truncated to 0 . For simplicity we focus on ReLU networks , with σ denoting the ReLU activation . For layers zk “ σpzk´1q which apply σ coordinate-wise , IBP computes szk “ σpszk´1q and zk “ σpzk´1q . Initial bounds are obtained via psz0px , q , z0px , qq “ px ` 1 , x´ 1q , where 1 denotes the all 1 ’ s vector . The interval bounds are propagated through all layers of the network by following the simple rules above . To certify ( 2.1 ) for the whole network , one straightforward method is to confirm that the margins of the interval bounds on the logits are positive : pF px , qqy ´ p sF px , qqy1 ą 0 @ y1 ‰ y . One important note about IBP is that the bounds should be optimized during training in order for the method to provide nontrivial robustness guarantees . Monotone operator equilibrium networks . Proposed by Winston & Kolter ( 2020 ) , MonDEQs are a class of DEQs inspired by monotone operator theory ( Ryu & Boyd , 2016 ) which have guaranteed unique fixed point solutions to the following equilibrium equation : z‹ “ σpWz‹ ` vpxqq ( 2.4 ) Let Ihˆh denote the identity matrix on h dimensions , with the subscript omitted when clear . A unique fixed-point solution is guaranteed for ( 2.4 ) for the following class of W : Proposition 2.1 ( ( Winston & Kolter , 2020 ) ) . Suppose W P Rhˆh satisfies that I ´W is positive definite ( PD ) , i.e . I ´W ą 0 . Then @ vpxq P Rh , a solution z‹ to ( 2.4 ) exists and is unique . Here 0 denotes the all 0 ’ s matrix and A ą 0 indicates that uJAu ą 0 for all nonzero u ( note that A does not need to be symmetric ) . Winston & Kolter ( 2020 ) guarantee that I ´ W is PD using the following unconstrained parameterization , which is enforced throughout training : W “ p1´mqI ´AAJ ` B ´BJ , for positive hyperparameter m. Section 3.1 builds on these results to derive certified upper and lower bounds on z‹ . 3 CERTIFYING ROBUSTNESS OF MONDEQS USING IBP . In this section , we describe our core methodology for developing certifiably robust MonDEQ layers . We will first demonstrate how to obtain interval bounds for MonDEQ layers by computing the solution to a certain IBP-inspired fixed point equation ( 3.3 ) . In Section 3.1 , we characterize a new parameterization for W for which a unique fixed point exists . In Section 3.3 , we provide theoretical justification that the resulting IBP-MonDEQ layers remain expressive . Our aim is to derive upper and lower interval bounds sz‹ and z‹ for the fixed point solution to ( 2.4 ) . A common interpretation of DEQs is that they compute an infinitely deep , unrolled explicit network ( Bai et al. , 2019 ; Winston & Kolter , 2020 ) : zk “ σpWzk´1 ` vq ( 3.1 ) with limkÑ8 zk “ z‹ . This equivalence is informal and mainly serves to motivate our derivation of the IBP-MonDEQ . Given interval bounds sv and v on v satisfying v ď v ď sv , where the inequalities hold elementwise , we can also follow IBP and ( 2.3 ) to iteratively obtain interval bounds on zk : „ szk zk “ σ ˆ „ pW q ` pW q´ pW q´ pW q ` „ szk´1 zk´1 ` „ sv v ˙ ( 3.2 ) Just as we took limkÑ8 zk , we consider limkÑ8 szk , zk , motivating another fixed-point problem : „ sz‹ z‹ “ σ ˆ „ pW q ` pW q´ pW q´ pW q ` „ sz‹ z‹ ` „ sv v ˙ ( 3.3 ) As shown in Figure 1 , this IBP-inspired fixed point equation essentially maps the region tv1 : v ď v1 ď svu to the region tz‹1 : z‹ ď z‹1 ď sz‹u , where inequalities are coordinate-wise . If they exist , the fixed points sz‹ and z‹ will indeed lead to valid interval bounds ( as defined in ( 2.2 ) ) on z‹ . Proposition 3.2 states this observation formally .

Summary: The paper presents a new class of neural networks called IBP-MonDEQs that are an extension of the recently introduced implicit networks MonDEQs. The authors identify a class of weight matrices of the network that ensures that fixed point of the implicit layers exist with respect to the interval analysis and is unique. The construction of IBP_MonDEQ is motivated by the goal of obtaining networks that can be certified to be robust. The authors then train such networks comparing against explicit networks of the same architecture constructed using the certified IBP training. The results show that IBP-MonDEQs can obtain better certified robustness than explicit networks on the MNIST and CIFAR10 dataset.