SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mahyar Fazlyab*, Taha Entesari; Aniket Roy, Rama Chellappa
{mahyarfazlyab, tentesa1, aroy28, rchella4}@jhu.edu
To improve the robustness of deep classifiers against adversarial perturbations, many approaches have been proposed, such as designing new architectures with better robustness properties (e.g., Lipschitz-capped networks), or modifying the training process itself (e.g., min-max optimization, constrained learning, or regularization). These approaches, however, might not be effective at increasing the margin in the input (feature) space. In this paper, we propose a differentiable regularizer that is a lower bound on the distance of the data points to the classification boundary. The proposed regularizer requires knowledge of the model's Lipschitz constant along certain directions. To this end, we develop a scalable method for calculating guaranteed differentiable upper bounds on the Lipschitz constant of neural networks accurately and efficiently. The relative accuracy of the bounds prevents excessive regularization and allows for more direct manipulation of the decision boundary. Furthermore, our Lipschitz bounding algorithm exploits the monotonicity and Lipschitz continuity of the activation layers, and the resulting bounds can be used to design new layers with controllable bounds on their Lipschitz constant. Experiments on the MNIST, CIFAR-10, and Tiny-ImageNet data sets verify that our proposed algorithm obtains competitively improved results compared to the state-of-the-art.
Motivated by the vulnerability of deep neural networks to adversarial attacks [1], i.e., imperceptible perturbations that can drastically change a model's prediction, researchers and practitioners have proposed various approaches to enhance the robustness of deep neural networks, including adversarial training [2-4], regularization [5, 6], constrained learning [7], randomized smoothing [8-10], relaxation-based defenses [11, 12], and model ensembles [13, 14]. These approaches modify the model architecture, the optimization procedure (loss function and algorithm), the inference mechanism, or the dataset itself [15] to enhance accuracy on both natural and adversarial examples.
However, most of these methods typically do not directly operate on input margins and rather target the output margin or its surrogates. For example, it is an established property that deep models trained with the cross-entropy loss, which is a surrogate for maximizing the output margin (see Appendix A.2), are prone to adversarial attacks [1]. From this perspective, it is critical to design regularized loss functions that can explicitly and effectively target the margin in the input space [16-18].
Our Contribution (1) Using first principles, we design a novel regularized loss function for training adversarially robust deep classifiers. We design a differentiable regularizer, using the Lipschitz constants of the logit differences, that is a lower bound on the input margin. We empirically show that this regularizer promotes larger margins in the input space. (2) We develop a scalable method for calculating guaranteed analytic and differentiable upper bounds on the Lipschitz constant of deep networks accurately and efficiently. Our method, called LipLT, hinges on the idea of Loop Transformation on the nonlinearities, which allows us to exploit the monotonicity and Lipschitz continuity of activations functions effectively. We prove that the resulting upper bound is better than the product of the Lipschitz constants of all layers (the so-called naive bound), and in practice, it is significantly better. Furthermore, our Lipschitz bounding algorithm can be used to design new layers with controllable bound on their Lipschitz constant. (3) We integrate our Lipschitz estimation algorithm within the proposed training loss to develop a robust training algorithm that eliminates the need for any inner-loop optimization subroutine. We utilize the recurring structure of the calculations that enable parallelized implementation on GPUs. When integrated into training, the relative accuracy of the bounds prevents excessive regularization and allows for more direct manipulation of the decision boundary.
Experiments on the MNIST, CIFAR-10, and Tiny-ImageNet data sets verify that our proposed algorithm obtains competitively improved results compared to the state-of-the-art. Code available on https://github.com/o4lc/CRM-LipLT.
In the interest of space, we only review the most relevant literature and defer the comprehensive version to the supplementary materials.
Enhancing Robustness via Optimization The most well-known approach in adversarial defenses is perhaps adversarial training (AT) [4] and its certified variants [11, 19-23], which involves minimizing a worst-case loss (or its approximation) using uniformly-bounded perturbations to the training data. Since these approaches might hurt accuracy on non-adversarial examples, several regularization-based methods have been proposed to trade off adversarial robustness against natural accuracy [5, 6, 24, 25]. Intuitively, these approaches aim to control the variations of the model close to the decision boundary or around the training data points. In general, the main challenge is to formulate computationally efficient differentiable regularizers that can directly increase the margin of the classifier. Of notable recent work in this direction is [18], in which the authors design a regularizer that directly increases the margin in the input space and prioritizes more vulnerable points by leveraging the dynamics of the decision boundary during training. Our loss function also takes advantage of this approach, but rather than using an iterative algorithm to compute the margin, we use Lipschitz continuity arguments to find closed-form differentiable lower bounds on the margin.
Robustness and Lipschitz Regularity To obtain efficient and scalable certificates of robustness one can bound or control the global Lipschitz constant of the model. To train robust networks, the global Lipschitz bound is typically computed as the product of the spectral norm of linear layers (the naive bound), which provides computationally efficient but overly conservative certificates [5, 19, 26]. If these bounds are localized (e.g., in a neighborhood of each training data point), conservatism can be mitigated at the expense of losing computational efficiency [20, 27]. However, it is not clear if this approach provides any advantages over other local bounding schemes [28, 29] that can be more accurate with comparable complexity. Hence, for training purposes, it is highly desirable to have global but less conservative differentiable Lipschitz bounds. Of notable work in this direction is the semidefinite programming (SDP) approach of [30] (as well as its local version [31]) to provide accurate numerical bounds, which has also been leveraged in training low-Lipschitz networks [32]. However, these SDP-based approaches are restricted to small-scale models. In comparison, LipLT is less accurate than LipSDP but can scale to significantly larger models and larger input dimensions. Compared to the naive method, LipLT is significantly more accurate but has a comparable practical complexity thanks to a specialized GPU implementation that exploits the recursive nature of the algorithm.
Lipschitz Layers Another approach to improving robustness is to design new layers with controllable Lipschitz constant. Various methods have been proposed to design the so-called 1-Lipschitz networks [7, 33-37]. Inspired by the LipSDP framework of [30], the recent work [38] designs 1-Lipschitz layers that generalize many of the aforementioned 1-Lipschitz designs. However, their proposed approach is currently not applicable to multi-layer networks. In [39], the authors propose a reparameterization that directly satisfies the SDP constraint of LipSDP, but the proposed method cannot handle skip connections. [21] introduces  $\ell_{\infty}$  distance neurons that are 1-Lipschitz and provide inherent robustness against  $\ell_{\infty}$  perturbations.
1.2 Preliminaries and Notation
The log-sum-exp function is defined as  $LSE(x) = \log(\sum_{i=1}^{n} \exp(x_i))$ . For any  $t > 0$ , the perspective [40] of the log-sum-exp function satisfies the inequalities  $\max_i(x_i) < t^{-1}LSE(tx) \leq \max_i(x_i) + t^{-1}\log(n)$ . We denote the cross-entropy loss by  $CE(q,p) = -\sum_{k=1}^{K} p_k \log(q_k)$ , where  $p, q \in \mathbb{R}^K$  are on the probability simplex. For an integer  $y \in \{1, \dots, K\}$ , we define  $u_y \in \mathbb{R}^K$  as the  $y$ -th standard unit vector. For a vector  $a \in \mathbb{R}^K$ , we have  $\|a\|_p = (\sum_{i=1}^{K} |a_i|^p)^{1/p}$ . Additionally, we have  $\|a^\top\|_p = \|a\|_q$ , where  $1/p + 1/q = 1$ . When  $p = 1$ , it is understood that  $q = \infty$ . For a matrix  $A \in \mathbb{R}^{m \times n}$ ,  $\|A\|$  denotes a matrix norm of  $A$ . In this paper we focus on  $\|A\|_2 = \sigma_{\max}(A)$ , i.e., the spectral norm of the matrix  $A$ . We, however, note that except for the implementation notes regarding the power iteration, the rest of the Lipschitz estimation algorithm is valid in any other matrix norm as well. We denote the indicator function of the event  $E$  as  $1_E$ , where  $1_E$  outputs 1 if the event  $E$  is realized and 0 otherwise.
2 Certified Radius Maximization (CRM)
Consider a  $K$ -class deep classifier  $C(x;\theta) = \arg \max_{1\leq i\leq K}f_i(x;\theta)$ , where  $f(x;\theta) = \operatorname{softmax}(z(x;\theta))$  is the vector of class probabilities, and  $z(x;\theta)$  is a deep network. If a data point  $x$  is classified correctly as  $y\in \{1,\dots K\}$ , then the logit margin  $\gamma (z(x;\theta),y)$ , the difference between the largest and second-largest logit, is  $\gamma (z(x;\theta),y) = z_y(x;\theta) - \max_{j\neq y}z_j(x;\theta)$ .
Several existing loss functions for training classifiers are differentiable surrogates for maximizing the logit margin, including the cross entropy, and its combination with label smoothing (see Appendix A.2 for more details). However, maximizing the logit margin, or its surrogates, does not necessarily increase the margin in the input space. This can happen, for example, when the model has rapid variations close to the decision boundary. Thus, we need a regularizer targeted for maximizing the input margin.
Regularization Based on Certified Radius For a correctly-classified data  $x$  with label  $y$ , the distance of  $x$  to the decision boundary can be calculated by solving the optimization problem [18],
$$
R (x, y; \theta) = \inf  _ {\hat {x}} \| x - \hat {x} \| \text {s u b j e c t} \gamma (z (\hat {x}; \theta), y) = 0, \tag {1}
$$
where the constraint enforces  $\hat{x}$  to be on the decision boundary. To ensure a large margin in the input space, we must, in principle, maximize the radius  $R(x,y;\theta)$  of all correctly classified data points. To achieve this goal, we can add a regularizer that penalizes small input margins,
$$
\min  _ {\theta} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ - \gamma (z (x; \theta), y) ] + \lambda \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ 1 _ {\{\gamma (z (x; \theta), y) > 0 \}} g (R (x, y; \theta)) ], \tag {2}
$$
where  $\lambda > 0$  is the regularization constant and  $g\colon \mathbb{R} \to \mathbb{R}$  is a decreasing function to promote larger certified radii. See appendix A.2 for more discussion on the properties of  $g$ .
To optimize (2), we must compute and differentiate through  $R(x, y; \theta)$ . For  $\ell_{\infty}$  norm and ReLU activations, we can reformulate (1) as a mixed-integer linear program by using a binary representation
2We note that we can replace the equality constraint with the inequality constraint  $\gamma (z(\hat{x};\theta),y)\leq 0$  and get the same result.
of the activation functions. However, it is not affordable to solve this optimization problem during training. Hence, we must resort to computing lower bounds on  $R(x,y;\theta)$  instead.
By replacing the logit margin with its soft lower bound in the constraint of (1), we obtain a lower bound on  $R(x, y; \theta)$ ,
$$
R (x, y; \theta) \geq \min  _ {\hat {x}} \| x - \hat {x} \|
$$
$$
\text {s . t .} \quad z _ {y} (x; \theta) - \frac {1}{t} \log \sum_ {j \neq y} e ^ {t z _ {j} (x; \theta)} \leq 0. \tag {3}
$$
To solve this non-convex problem, the authors of [18] adapt the successive linearization algorithm of [41], which in turn does not provide a provable lower bound. To avoid this iterative algorithm and to provide sound lower bounds, we propose to use Lipschitz continuity arguments.
Lipschitz-Based Surrogates for Certified Radius Suppose we add a norm-bounded perturbation  $\delta$  to a data point  $x$  that is classified correctly as  $y$ . For the label of  $x + \delta$  to not change, we must enforce
$$
\Delta z _ {y i} (x + \delta) := z _ {y} (x + \delta ; \theta) - z _ {i} (x + \delta ; \theta) > 0 \quad \forall i \neq y.
$$
Suppose the logit difference  $x \mapsto \Delta z_{yi}(x)$  is Lipschitz continuous with constant  $L_{yi}$  (in  $p$  norm), implying  $|\Delta z_{yi}(x + \delta; \theta) - \Delta z_{yi}(x)| \leq L_{yi} \| \delta \|_p$ ,  $\forall x, \delta$ . Then we can write
$$
\Delta z _ {y i} (x + \delta ; \theta) \geq \Delta z _ {y i} (x; \theta) - L _ {y i} \| \delta \| _ {p} \quad \forall i \neq y.
$$
The right-hand side remaining positive for all  $i \neq y$  is a sufficient condition for the correct classification of  $x + \delta$  as  $y$ . This sufficient condition yields a lower bound on  $R(x, y; \theta)$  as follows,
$$
\underline {{R}} (x, y; \theta) := \min  _ {i \neq y} \frac {z _ {y} (x ; \theta) - z _ {i} (x ; \theta)}{L _ {y i}} \leq R (x, y; \theta). \tag {4}
$$
This lower bound is the pointwise minimum of logit differences normalized by their Lipschitz constants. This lower bound is not differentiable everywhere, but similar to the logit margin, we propose a smooth underapproximation of the min operator using scaled LSE:
$$
\underline {{R}} _ {t} ^ {\text {s o f t}} (x, y; \theta) := - \frac {1}{t} \log \left(\sum_ {i \neq y} \exp \left(- t \frac {z _ {y} (x ; \theta) - z _ {i} (x ; \theta)}{L _ {y i}}\right)\right). \tag {5}
$$
In addition to differentiability, another advantage of using this soft lower bound is that as opposed to  $\underline{R}(x,y;\theta)$ , which involves only two logits, the soft lower bound  $\underline{R}_t^{\text{soft}}(x,y;\theta)$  includes all the logits, and hence, makes more effective use of information.
Proposition 1 We have the following relationship between  $\underline{R}_t^{\text{soft}}(x,y;\theta)$  defined in (5) and  $\underline{R}(x,y;\theta)$  defined in (4).
$$
\underline {{R}} _ {t} ^ {s o f t} (x, y; \theta) \leq \underline {{R}} (x, y; \theta) \leq \underline {{R}} _ {t} ^ {s o f t} (x, y; \theta) + \frac {\log (K - 1)}{t}.
$$
In summary, we use the following loss function to train our classifier,
$$
\min  _ {\theta} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ - \underline {{\gamma}} (z (x; \theta), y) ] + \lambda \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ 1 _ {\{\gamma (z (x; \theta), y) > 0 \}} g (\underline {{R}} _ {t} ^ {s o f t} (x, y; \theta)) ]. \tag {6}
$$
The first and the second terms are differentiable surrogates to the negative logit margin, and the input margin, respectively. For example, as we show in Appendix A.2 the negative cross-entropy loss is a differentiable lower bound on the logit margin, i.e., we can choose  $\underline{\gamma}(z(x; \theta), y) = -CE(z(x; \theta), u_y)$ .
It now remains to estimate the  $L_{yi}$ 's (the Lipschitz constants of  $x \mapsto \Delta z_{yi}(x)$ ) that are needed to compute the second term in the loss function. While any guaranteed upper bound on  $L_{yi}$  would suffice to preserve the lower bound property in (4), a more accurate upper bound prevents excessive regularization and allows for more direct manipulation of the decision boundary. To achieve this goal, we propose a new method which we will discuss next.
3 Scalable Estimation of Lipschitz Constants via Loop Transformation (LipLT)
In this section, we propose a general-purpose algorithm for computing a differentiable upper bound on the Lipschitz constant of deep neural networks, which is also of independent interest. For simplicity in the exposition, we first consider single hidden layer neural networks of the form  $h(x) = W^{1}\phi (W^{0}x)$ , where  $W^1, W^0$  have compatible dimensions, and the bias terms are ignored without loss of generality. The activation layers  $\phi$  are of the form  $\phi (z) = (\varphi (z_{1}),\dots ,\varphi (z_{n_{1}}))\quad z\in \mathbb{R}^{n_{1}}$ , where  $\varphi \colon \mathbb{R}\to \mathbb{R}$  is the activation function, which we assume to be monotone and Lipschitz continuous, implying  $\alpha \leq (\varphi (x) - \varphi (x')) / (x - x')\leq \beta \quad \forall x\neq x'$  for some  $0\leq \alpha \leq \beta < \infty$  [30, 42]. In [30], the authors propose an SDP for computing an upper bound on the global Lipschitz constant of multi-layer neural networks when  $\ell_2$  norm is considered in both input and output domains. This result for the single hidden layer case is formally stated in the following theorem.
Theorem 1 ([30]) Consider a single-layer neural network described by  $h(x) = W^{1}\phi (W^{0}x)$ . Suppose  $\phi (x)\colon \mathbb{R}^{n_1}\to \mathbb{R}^{n_1} = [\varphi (x_1)\dots \varphi (x_{n_1})]$ , where  $\varphi$  is slope-restricted over  $\mathbb{R}$  with parameters  $0\leq \alpha \leq \beta < \infty$ . Suppose there exist  $\rho >0$  and diagonal  $T\in \mathbb{S}_{+}^{n_{1}}$  such that the matrix inequality
$$
M (\rho , T) := \left[ \begin{array}{c c} - 2 \alpha \beta W ^ {0} ^ {\top} T W ^ {0} - \rho I _ {n _ {0}} & (\alpha + \beta) W ^ {0} ^ {\top} T \\ (\alpha + \beta) T W ^ {0} & - 2 T + W ^ {1} ^ {\top} W ^ {1} \end{array} \right] \preceq 0, \tag {7}
$$
holds. Then  $\| h(x) - h(y)\| _2\leq \sqrt{\rho}\| x - y\| _2$  for all  $x,y\in \mathbb{R}^{n_0}$ .
The key advantage of this SDP formulation is that we can exploit several properties of the structure, namely monotonicity  $(\alpha \geq 0)$  and Lipschitz continuity  $(\beta < \infty)$  of the activation functions, as well as using the same activation function in the activation layer. However, solving this SDP and enforcing them during training can be challenging even for small-scale neural networks. The recent work [43] has exploited the chordal structure of the resulting SDP imposed by the sequential structure of the network to solve the SDP for larger instances. However, these approaches are still unable to scale to larger problems and are not suitable for training purposes.
To guide the search for analytic solutions to the linear matrix inequality (LMI) of Theorem 1, the authors in [38] consider the following residual structure,
$$
h (x) = H x + G \phi (W x). \tag {8}
$$
Then it can be shown that the LMI condition (7) generalizes to condition (9) (see Appendix A.3 for details).
$$
M (\rho , T) := \left[ \begin{array}{c c} - 2 \alpha \beta W ^ {\top} T W + H ^ {\top} H - \rho I _ {n _ {0}} & (\alpha + \beta) W ^ {\top} T + H ^ {\top} G \\ (\alpha + \beta) T W + G ^ {\top} H & - 2 T + G ^ {\top} G \end{array} \right] \preceq 0, \tag {9}
$$
By choosing  $\rho = 1$ ,  $H = I$  and  $G = -(\alpha + \beta)W^\top T$ , then the LMI (9) simplifies to  $(\alpha + \beta)^2 TWW^\top T \preceq 2T$  (all blocks in the LMI except for the lower diagonal block become zero). When we restrict  $T$  to be positive definite, then the latter condition is equivalent to  $(\alpha + \beta)^2 WW^\top \preceq 2T^{-1}$ , which can be satisfied analytically using various choices of  $T$  [38]. In summary, the function  $h(x) = x - (\alpha + \beta)W^\top T\phi(Wx)$  is guaranteed to be 1-Lipschitz as long as  $(\alpha + \beta)^2 WW^\top \preceq 2T^{-1}$ .
A potential issue with the above parameterization (and 1-Lipschitz networks in general) is that, since the true Lipschitz constant of the layer can be less than one, the multi-layer concatenation can become overly contractive. One way to resolve this limitation is to modify the parameterization as follows.
Proposition 2 Suppose  $WW^{\top} \preceq \frac{2\rho}{(\alpha + \beta)^{2}} T^{-1}$  for some  $\rho > 0$  and some diagonal positive definite  $T$ . Then the following function is  $\sqrt{\rho}$ -Lipschitz.
$$
h (x) = \sqrt {\rho} x - \frac {\alpha + \beta}{\sqrt {\rho}} W ^ {\top} T \phi (W x). \tag {10}
$$
Now if we make a cascade connection of  $L \geq 2$  layers of the form (10) (each having its own  $\rho$ ), a naive bound on the Lipschitz constant of the corresponding deep network would be  $\prod_{i=1}^{L} \rho_i^{1/2}$ . However, this upper bound can still be very crude. We now propose an alternative approach to compute an analytic upper bound on the Lipschitz constant of (8). For the multi-layer case, we then show that our method can capture the coupling between different layers to improve the naive bound. For the sake of space, we defer all the proofs to the supplementary materials.
Figure 1: Loop transformation on a residual layer of the form  $h(x) = Hx + G\phi (Wx)$ . Here we use  $\varphi (x) = \tanh x$  for the illustration of the loop transformation.
The power of loop transformation Starting from (8), using the fact that  $\phi$  is  $\beta$ -Lipschitz, a global Lipschitz constant of  $h$  can be computed as
$$
\begin{array}{l} \left\| h (x) - h \left(x ^ {\prime}\right) \right\| \leq \left\| H \left(x - x ^ {\prime}\right) \right\| + \left\| G \left(\phi \left(W x\right) - \phi \left(W x ^ {\prime}\right)\right) \right\| \\ \leq \underbrace {\left(\| H \| + \beta \| G \| \| W \|\right)} _ {L _ {\text {n a i v e}}} \| x - x ^ {\prime} \|. \tag {11} \\ \end{array}
$$
This upper bound is pessimistic as it does not exploit the monotonicity  $(\alpha \geq 0)$  of activation functions. In other words, this bound would not change as long as  $\alpha \geq -\beta$ . To inform the bound with monotonicity, we perform a loop transformation [44, 45] to bring the non-linearities to the symmetric sector  $[-(\beta - \alpha) / 2, (\beta - \alpha) / 2]$ , resulting in the representation
$$
h (x) = \left(H + \frac {\alpha + \beta}{2} G W\right) x + G \psi (W x), \tag {12}
$$
where  $\psi(x) \coloneqq \phi(x) - \frac{\alpha + \beta}{2}x$  is the loop transformed nonlinearity, which is no longer monotone, and is  $\frac{\beta - \alpha}{2}$ -Lipschitz- see Figure 1. We can now compute a Lipschitz constant as
$$
\begin{array}{l} \| h (x) - h \left(x ^ {\prime}\right) \| = \| \left(H + \frac {\alpha + \beta}{2} G W\right) \left(x - x ^ {\prime}\right) + G \left(\Psi \left(W x\right) - \Psi \left(W x ^ {\prime}\right)\right) \| \\ \leq \left\| \left(H + \frac {\alpha + \beta}{2} G W\right) \left(x - x ^ {\prime}\right) \right\| + \left\| G \left(\Psi \left(W x\right) - \Psi \left(W x ^ {\prime}\right)\right) \right\| \tag {13} \\ \leq \underbrace {(\| H + \frac {\alpha + \beta}{2} G W \| + \frac {\beta - \alpha}{2} \| G \| \| W \|)} _ {L _ {\mathrm {L T}}} \| x - x ^ {\prime} \|. \\ \end{array}
$$
As we show below, this loop transformation does improve the naive bound. In Appendix A.3 we also prove the optimality of the transformation.
Proposition 3 Consider a single-layer residual structure given by  $h(x) = Hx + G\phi (Wx)$ . Suppose  $\phi \colon \mathbb{R}^{n_1}\to \mathbb{R}^{n_1}$ ,  $\phi (x) = [\varphi (x_{1})\dots \varphi (x_{n_{1}})]$ , where  $\varphi$  is slope restricted in  $[\alpha ,\beta ]$ . Then  $h$  is Lipschitz continuous with constant  $\| H + \frac{\alpha + \beta}{2} GW\| +\frac{\beta - \alpha}{2}\| G\| \| W\| \leq L_{\mathrm{naive}}$ .
Bound refinement. For the case of the  $\ell_2$  norm, we can further improve the derived bound in (13). Specifically, we can bound the Lipschitz constant of the second term of  $h$  in (12) analytically using LipSDP, which we summarize in the following theorem.
Theorem 2 Suppose there exists a diagonal  $T$  that satisfies  $G^{\top}G\preceq T$ . Then a Lipschitz constant of (8) in  $\ell_2$  norm is given by
$$
L _ {\mathrm {L T - S D P}} (T) = \| H + \frac {\alpha + \beta}{2} G W \| _ {2} + \frac {\beta - \alpha}{2} \| W ^ {\top} T W \| _ {2} ^ {\frac {1}{2}}.
$$