SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dayana Savostianova Gran Sasso Science Institute 67100 L'Aquila (Italy) dayana.savostianova@gssi.it
Gianluca Ceruti  
University of Innsbruck  
6020 Innsbruck (Austria)  
gianluca.ceruti@uibk.ac.at
Emanuele Zangrando Gran Sasso Science Institute 67100 L'Aquila (Italy) emanuele.zangrando@gssi.it
Francesco Tudisco  
Gran Sasso Science Institute  
67100 L'Aquila (Italy)  
francesco.tudisco@gssi.it
With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultaneously enforcing approximate orthonormal constraints. The resulting model reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.
Deep learning and neural networks have achieved great success in a variety of applications in computer vision, signal processing, and scientific computing, to name a few. However, their robustness with respect to perturbations of the input data may considerably impact security and trustworthiness and poses a major drawback to their real-world application. Moreover, the memory and computational requirements for both training and inferring phases render them impractical in application settings with limited resources. While a broad literature on pruning methods and adversarial robustness has been developed to address these two issues in isolation, much less has been done to design neural networks that are both energy-saving and robust. Actually, in many approaches the two problems seem to compete against each other as most adversarial robustness improving-techniques require even larger networks [36, 41, 46, 47, 79] or computationally more demanding loss functions, and thus more expensive training phases [13, 23, 34, 44, 66].
The limited work available so far on robust pruned networks is mostly focused on reducing memory and computational costs of the inference phase, while retaining adversarial robustness [20, 28, 42, 56, 72, 76]. However, the inference phase amounts to only a very limited fraction of the cost of the
whole deep learning pipeline, which is instead largely dominated by the training phase. Reducing both inference and training costs is a challenging but desirable goal, especially in view of a more accessible AI and its effective use on limited-resource and limited-connectivity devices such as drones or satellites.
Some of the most effective techniques for the reduction of training costs so far have been based on low-rank weights parametrizations [29, 55, 71]. These methods exploit the intrinsic low-rank structure of parameter matrices and large data matrices in general [17, 49, 59, 69]. Thus, assuming a low-rank structure for the neural network's weights  $W = USV^{\top}$ , the resulting training procedures only use the small individual factors  $U, S, V$ . This results in a training cost that scales linearly with the number of neurons, as opposed to a quadratic scaling required by training full-rank weights. Despite significantly reducing training parameters, these methods achieve accuracy comparable with the original full networks. However, their robustness with respect to adversarial perturbations has been largely unexplored so far.
In this paper, we observe that the adversarial robustness of low-rank networks may actually deteriorate with respect to the full baseline. By modeling the robustness of the network in terms of the neural network's condition number, we argue that this loss of robustness is due to the exploding condition number of the low-rank weight matrices, whose singular values grow very large in order to match the baseline accuracy and to compensate for the lack of parameters. Thus, to mitigate this growing instability, we design an algorithm that trains the network using only the low-rank factors  $U, S, V$  while simultaneously ensuring the condition number of the network remains small. To this end, we interpret the loss optimization problem as a continuous-time gradient flow and use techniques from geometric integration theory on manifolds [12, 31, 55, 70] to derive three separate projected gradient flows for  $U, S, V$ , individually, which ensure the condition number of the network remains bounded to a desired tolerance  $1 + \tau$ , throughout the epochs. For a fixed small constant  $\varepsilon > 0$ , this is done by bounding the singular values of the small rank matrices within a narrow band  $[s - \varepsilon, s + \varepsilon]$  around a value  $s$ , chosen to best approximate the original singular values.
We provide several experimental evaluations on different architectures and datasets, where the robust low-rank networks are compared against a variety of baselines. The results show that the proposed technique allows us to compute from scratch low-rank weights with bounded singular values, significantly reducing the memory demand and computational cost of training while at the same time retaining or improving both the accuracy and the robust accuracy of the original model. On top of the experimental evidence, we provide a key approximation theorem that shows that if a high-performing low-rank network with bounded singular values exists, then our algorithm computes it up to a first-order approximation error.
This paper focuses on feed-forward neural networks. However, our techniques and analysis apply straightforwardly to convolutional filters reshaped in matrix form, as done in e.g. [29, 55, 71]. Other ways exist to promote orthogonality of convolutional filters, e.g. [60, 65, 77], which we do not consider in this work.
Neural networks' robustness against adversarial perturbations has been extensively studied in the machine learning community. It is well-known that the adversarial robustness of a neural network is closely related to its Lipschitz continuity [13, 23, 62, 68], see also Section 3. Accordingly, training neural networks with bounded Lipschitz constant is a widely employed strategy to address the problem. A variety of works studied Lipschitz architectures [37, 60, 65, 68], and a number of certified robustness guarantees have been proposed [23, 51, 61]. While scaling each layer to impose 1-Lipschitz constraints is a possibility, this approach may lead to vanishing gradients and it is known that a more effective way to reduce the Lipschitz constant and increase robustness is obtained by promoting orthogonality on each layer [5, 13]. On top of robustness, small Lipschitz constants and orthogonal layers are known to lead to improved generalization bounds [11, 45] and more interpretable gradients [67]. Orthogonality was also shown to improve signal propagation in (very) deep networks [52, 74].
A variety of methods to integrate orthogonal constraints in deep neural networks have been developed over the years. Notable example approaches include methods based on regularization and landing [1, 13], cheap parametrizations of the orthogonal group [6, 38, 39, 48, 50], Riemannian and projected gradient descent schemes [2, 3, 10].
In parallel to the development of methods to promote orthogonality, an active line of research has grown to develop effective training strategies to enforce low-rank weights. Unlike sparsity-promoting pruning strategies that primarily aim at reducing the parameters required for inference [8, 21, 22, 30, 43], low-rank neural network models are designed to train directly on the low-parametric manifold of low-rank matrices and are particularly effective to reduce the number of parameters required by both inference and training phases. Similar to orthogonal training, methods for low-rank training include methods based on regularization [26, 29], as well as methods based on efficient parametrizations of the low-rank manifold using the SVD, randomized tensor dropout or the polar decomposition [32, 71, 75], and Riemannian optimization-based training models [55, 57].
By combining low-rank training with approximate orthogonal constraints, in this work we propose a strategy that simultaneously enforces robustness while only requiring a reduced percentage of the network's parameters during training. The method is based on a gradient flow differential formulation of the training problem, and the use of geometric integration theory to derive the governing equations of the low-rank factors. With this formulation, we are able to reduce the sensitivity of the network during training at almost no cost, yielding well-conditioned low-rank neural networks. Our experimental findings are supported by an approximation theorem that shows that, if the ideal full network can be approximated by a low-rank one, then our method computes a good approximation. This is well-aligned with recent work that shows the existence of high-performing low-rank nets in e.g. deep linear models [7, 17, 25, 49]. Moreover, as orthogonality helps in training really deep networks, low-rank orthogonal models may be used to mitigate the effect of increased effective depth when training low-rank networks [54].
3 The condition number of a neural network
The adversarial robustness of a neural network model  $f$  can be measured by the worst-case sensitivity of  $f$  with respect to small perturbations of the input data  $x$ . In an absolute sense, this boils down to measuring the best global and local Lipschitz constant of  $f$  with respect to suitable distances, as discussed in a variety of papers [13, 15, 23, 62]. However, as the model and the data may assume arbitrary large and arbitrary small values in general, a relative measure of the sensitivity of  $f$  may be more informative. In other words, if we assume a perturbation  $\delta$  of small size as compared to  $x$ , we would like to quantify the largest relative change in  $f(x + \delta)$ , as compared to  $f(x)$ . This is a well-known problem of conditioning, as we review next, and naturally leads to the concept of condition number of a neural network.
In the linear setting, the condition number of a matrix is a widely adopted relative measure of the worst-case sensitivity of linear problems with respect to noise in the data. For a matrix  $A$  and the matrix operator norm  $\| A \| = \sup_{x \neq 0} \| Ax \| / \| x \|$  the condition number of  $A$  is defined as  $\operatorname{cond}(A) = \| A \| \| A^+ \|$ , where  $A^+$  denotes the pseudo-inverse of  $A$ . Note that it is immediate to verify that  $\operatorname{cond}(A) \geq 1$ . Now, if for example  $u$  and  $u_{\varepsilon}$  are the solutions to the linear system  $Au = b$ , when  $A$  and  $b$  are exact data or when they are perturbed with noise  $\delta_A$ ,  $\delta_b$  of relative norm  $\| \delta_A \| / \| A \| \leq \varepsilon$  and  $\| \delta_b \| / \| b \| \leq \varepsilon$ , respectively, then the following relative error bound holds
$$
\frac {\| u - u _ {\varepsilon} \|}{\| u \|} \lesssim \operatorname {c o n d} (A) \varepsilon .
$$
Thus, small perturbations in the data  $A$ ,  $b$  imply small alterations in the solution if and only if  $A$  is well conditioned, i.e.  $\operatorname{cond}(A)$  is close to one.
As in the linear case, it is possible to define the concept of condition number for general functions  $f$ , [24, 53]. Let us start by defining the relative error ratio of a function  $f: \mathbb{R}^d \to \mathbb{R}^m$  in the point  $x$ :
$$
R (f, x; \delta) = \frac {\| f (x + \delta) - f (x) \|}{\| f (x) \|} / \frac {\| \delta \|}{\| x \|}. \tag {1}
$$
In order to take into account the worst-case scenario, the local condition number of  $f$  at  $x$  is defined by taking the sup of (1) over all perturbations of relative size  $\varepsilon$ , i.e. such that  $\| \delta \| \leq \varepsilon \| x \|$ , in the
limit of small  $\varepsilon$ . Namely,  $\operatorname{cond}(f; x) = \lim_{\varepsilon \downarrow 0} \sup_{\delta \neq 0: \| \delta \| \leq \varepsilon \| x \|} R(f, x; \delta)$ . This quantity is a local measure of the "infinitesimal" conditioning of  $f$  around the point  $x$ . In fact, a direct computation reveals that
$$
\frac {\| f (x + \delta) - f (x) \|}{\| f (x) \|} \lesssim \operatorname {c o n d} (f; x) \varepsilon , \tag {2}
$$
as long as  $\| \delta \| \leq \varepsilon \| x \|$ . Thus,  $\operatorname{cond}(f; x)$  provides a form of relative local Lipschitz constant for  $f$  which in particular shows that, if  $\| \delta \| / \| x \|$  is smaller than  $\operatorname{cond}(f; x)^{-1}$ , we expect limited change in  $f$  when  $x$  is perturbed with  $\delta$ . A similar conclusion is obtained using an absolute local Lipschitz constant in e.g. [23]. Similarly to the absolute case, a global relative Lipschitz constant can be obtained by looking at the worst-case over  $x$ , setting  $\operatorname{cond}(f) = \sup_{x \in \mathcal{X}} \operatorname{cond}(f; x)$ . Clearly, the same bound (2) holds for  $\operatorname{cond}(f)$ . Note that this effectively generalizes the linear case, as when  $f(x) = Ax$  we have  $\operatorname{cond}(f) = \operatorname{cond}(f, x) = \operatorname{cond}(A)$ .
When  $f$  is a neural network,  $\operatorname{cond}(f)$  is a function of the network's weights and robustness may be enforced by reducing  $\operatorname{cond}(f)$  while training. In fact,  $\operatorname{cond}(f)$  is the relative equivalent of the network's Lipschitz constant and thus standard Lipschitz-based robustness certificates [23, 40, 68] can be recast in terms of  $\operatorname{cond}(f)$ . However, for general functions  $f$  and general norms  $\| \cdot \|$ ,  $\operatorname{cond}(f)$  may be (very) expensive to compute, it may be non-differentiable, and  $\operatorname{cond}(f) > 1$  can hold [24]. Fortunately, for feed-forward neural networks, it holds (proof and additional details moved to Appendix B in the supplementary material)
Proposition 1. Let  $\mathcal{X}$  be the input space and let  $f(x) = z_{L + 1}$  be a network with  $L$  linear layers  $z_{i + 1} = \sigma_i(W_iz_i)$ ,  $i = 1,\ldots ,L$ . Then,
$$
\operatorname {c o n d} (f) = \sup  _ {x \in \mathcal {X} \backslash \{0 \}} \operatorname {c o n d} (f; x) \leq \left(\prod_ {i = 1} ^ {L} \sup  _ {x \in \mathcal {X} _ {i} \backslash \{0 \}} \operatorname {c o n d} (\sigma_ {i}; x)\right) \left(\prod_ {i = 1} ^ {L} \operatorname {c o n d} (W _ {i})\right).
$$
In particular, for typical  $\mathcal{X}_i$  and typical choices of  $\sigma_{i}$ , including  $\sigma_{i} \in \{\text{leakyReLU}, \text{sigmoid}, \text{tanh}, \text{hardtanh}, \text{softplus}, \text{siLU}\}$ , we have
$$
\sup  _ {x \in \mathcal {X} _ {i} \backslash \{0 \}} \operatorname {c o n d} (\sigma_ {i}; x) \leq C <   + \infty
$$
for a positive constant  $C > 0$  that depends only on the activation function  $\sigma_{i}$ .
Note that for entrywise nonlinearities  $\sigma$ , the condition number  $\operatorname{cond}(\sigma; x)$  can be computed straightforwardly. In fact, when  $\sigma$  is Lipschitz, the problem can be reduced to a one-dimensional function, and it follows directly from its definition that (see also [64])
$$
\operatorname {c o n d} (f; x) = \sup  _ {\nu_ {x} \in \partial \sigma (x)} | \nu_ {x} | | x | | \sigma (x) | ^ {- 1}, \qquad x \in \mathbb {R}
$$
where  $\partial \sigma (x)$  denotes Clarke's generalized gradient [14] of  $\sigma$  at the point  $x$ . Thus, for example, if  $\sigma$  is LeakyRelu with slope  $\alpha$ , we have  $\mathrm{cond}(\sigma) = 1$ ; if  $\sigma$  is the logistic sigmoid  $(1 + e^{-x})^{-1}$  and the feature space  $\mathcal{X}_i$  is such that  $\mathcal{X}_i = W_i\mathcal{X}_{i - 1}$ , then if  $|z_{i - 1}|\leq c_{i - 1}$  entry-wise, we have  $|x_{i}|\leq c_{i - 1}\max_{uv}|W_{i}|_{uv}:= c_{i}$  and  $\mathrm{cond}(\sigma)\leq \sup_{x\geq -c_i}|x|e^{-x}(1 + e^{-x})^{-1}\leq \max \{c_i,1 / e\}$ .
From Proposition 1 we see that when  $f$  is a feed-forward network, to reduce the condition number of  $f$  it is enough to reduce the conditioning of all its weights. When  $\| \cdot \| = \| \cdot \|_2$  is the Euclidean  $L^2$  norm, we have  $\mathrm{cond}_2(W) = s_{\max}(W) / s_{\min}(W)$ , the ratio between the largest and the smallest singular value of  $W$ . This implies that orthogonal weight matrices, for example, are optimally conditioned with respect to the  $L^2$  metric. Thus, a notable and well-known consequence of Proposition 1 is that imposing orthogonality constraints on  $W$  improves the robustness of the network [13, 27, 38, 50].
While orthogonal constraints are widely studied in the literature, orthogonal matrices are not the only optimally conditioned ones. In fact,  $\mathrm{cond}_2(W) = 1$  for any  $W$  with constant singular values. In the next section, we will use this observation to design a low-rank and low-cost algorithm that trains well-conditioned networks by ensuring  $\mathrm{cond}_2(W) \leq 1 + \tau$ , for all layers  $W$  and a desired tolerance  $\tau > 0$ .
4 Robust low-rank training
4.1 Instability of low-rank networks
Low-rank methods are popular strategies to reduce the memory storage and the computational cost of both training and inference phases of deep learning models [29, 55, 71]. Leveraging the intrinsic
Figure 1: Evolution of layers' condition numbers during training for LeNet5 on MNIST. From left to right: standard full-rank baseline model; [71] vanilla low-rank training; [55] dynamical low-rank training based on gradient flow; [75] low-rank training through regularization. All low-rank training strategies are set to  $80\%$  compression ratio (percentage of removed parameters with respect to the full baseline model).
low-rank structure of parameter matrices [7, 17, 49, 59], these methods train a subnetwork with weight matrices parametrized as  $W = USV^{\top}$ , for "tall and skinny" matrices  $U, V$  with  $r$  columns, and a small  $r \times r$  matrix  $S$ . Training low-rank weight matrices has proven to effectively reduce training parameters while retaining performance comparable to those of the full model. However, while a variety of contributions have analyzed and refined low-rank methods to match the full model's accuracy, the robust accuracy of low-rank models has been partially overlooked in the literature.
Here we observe that reducing the rank of the layer may actually deteriorate the network's robustness. We argue that this phenomenon is imputable to the exploding condition number of the network. In Figure 1 we plot the evolution of the condition number  $\mathrm{cond}_2$  for the four internal layers of LeNet5 during training using different low-rank training strategies and compare them with the full model. While the condition number of the full model grows moderately with the iteration count, the condition number of low-rank layers blows up drastically. This singular value instability leads to poor robustness performance of the methods, as observed in the experimental evaluation of Section 5.
In the following, we design a low-rank training model that allows imposing simple yet effective training constraints, bounding the condition number of the trained network to a desired tolerance  $1 + \tau$ , and improving the network robustness without affecting training nor inference costs.
4.2 Low-rank gradient flow with bounded singular values
Let  $W \in \mathbb{R}^{n \times m}$  be the weight matrix of a linear layer within  $f$ . For an integer  $r \leq \min \{m, n\}$  let  $\mathcal{M}_r = \{W : \mathrm{rank}(W) = r\}$  be the manifold of rank-  $r$  matrices which we parametrize as
$$
\mathcal {M} _ {r} = \left\{U S V ^ {\top}: U \in \mathbb {R} ^ {n \times r}, V \in \mathbb {R} ^ {m \times r} \text {w i t h o r t h o n o r m a l c o l u m n s ,} S \in \mathbb {R} ^ {r \times r} \text {i n v e r t i b l e} \right\}.
$$
Obviously, the singular values of  $W = USV^{\top} \in \mathcal{M}_r$  coincide with the singular values of  $S$ . For  $s, \varepsilon$  such that  $0 < \varepsilon < s$ , define  $\Sigma_s(\varepsilon)$  as the set of matrices with singular values in the interval  $[s - \varepsilon, s + \varepsilon]$ . Note that  $\Sigma_s(0)$  is a Riemannian manifold obtained essentially by an  $s$  scaling of the standard Stiefel manifold (the manifold of matrices with orthonormal columns) and any  $A \in \Sigma_s(0)$  is optimally conditioned, i.e.  $\mathrm{cond}_2(A) = 1$ . Thus,  $\varepsilon$  can be interpreted as an approximation parameter that controls how close  $\Sigma_s(\varepsilon)$  is to the "optimal" manifold  $\Sigma_s(0)$ . To enhance the network robustness, in the following we will constrain the parameter weight matrix  $S$  to  $\Sigma_s(\varepsilon)$ . With this constraint, we get  $\mathrm{cond}_2(W) \leq (s - \varepsilon)^{-1}(s + \varepsilon) = 1 + \tau$ , with  $\tau = 2(s - \varepsilon)^{-1}\varepsilon$ , so that the tolerance  $\tau$  on the network's conditioning can be tuned by suitably choosing the approximation parameter  $\varepsilon$ .
Given the loss function  $\mathcal{L}$ , we are interested in the constrained optimization problem
$$
\min  \mathcal {L} \quad \text {s . t .} \quad W = U S V ^ {\top} \in \mathcal {M} _ {r} \text {a n d} S \in \Sigma_ {s} (\varepsilon), \text {f o r a l l l a y e r s} W. \tag {3}
$$
To approach (3), we use standard arguments from geometric integration theory [31, 70] to design a training scheme that updates only the factors  $U, S, V$  and the gradient of  $\mathcal{L}$  with respect to  $U, S, V$ , without ever forming the full weights nor the full gradients. To this end, following [55], we first recast the optimization of  $\mathcal{L}$  with respect to each layer  $W$  as a continuous-time gradient flow
$$
\dot {W} (t) = - \nabla_ {W} \mathcal {L} (W (t)), \tag {4}
$$
where "dot" denotes the time derivative and where we write  $\mathcal{L}$  as a function of  $W$  only, for brevity. Along the solution of the differential equation above, the loss decreases and a stationary point is
Algorithm 1: Pseudocode of robust well-Conditioned Low-Rank (CondLR) training scheme
Input: Chosen compression rate, i.e. for each layer  $W$  choose a rank  $r$ ; Initial layers' weights parametrized as  $W = USV^{\top}$ , with  $S \sim r \times r$ ; Second singular value moment of  $S$ ,  $s = \sqrt{\sum_{k} s_k(S)^2 / r}$  Conditioning tolerance  $\tau > 0$   
1 for each iteration and each layer do (each block in parallel)  
2  $\begin{array}{l} U \leftarrow \text{one optimization step with gradient } G_1 \text{ and initial point } U \\ U \leftarrow \text{project } U \text{ on Stiefel manifold with } r \text{ orthonormal columns} \end{array}$   
3  
4  
5  
6  
7  
8  
9
approached as  $t \to \infty$ . Now, if we assume  $W \in \mathcal{M}_r$ , then  $\dot{W} \in T_W \mathcal{M}_r$ , the tangent space of  $\mathcal{M}_r$  at the point  $W$ . Thus, to ensure the whole trajectory  $W(t) \in \mathcal{M}_r$ , we can consider the projected gradient flow  $\dot{W}(t) = -P_{W(t)} \nabla_W \mathcal{L}(W(t))$ , where  $P_W$  denotes the orthogonal projection (in the ambient space of matrices) onto  $T_W \mathcal{M}_r$ . Next, we notice that the projection  $P_W \nabla_W \mathcal{L}$  can be defined by imposing orthogonality with respect to any point  $Y \in T_W \mathcal{M}_r$ , namely
$$
\left\langle P _ {W} \nabla_ {W} \mathcal {L} - \nabla_ {W} \mathcal {L}, Y \right\rangle = 0 \quad \text {f o r a l l} Y \in T _ {W} \mathcal {M} _ {r}
$$