Title: Robust low-rank training via approximate orthonormal constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: low-rank neural networks

Stiefel manifold

orthogonal neural networks

pruning

adversarial robustness

neural network condition number

neural network singular values
Abstract: With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy,  we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while  simultaneously enforcing approximate orthonormal constraints. The resulting model 
reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.

Robust low-rank training via approximate orthonormal constraints

Abstract

With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultaneously enforcing approximate orthonormal constraints. The resulting model reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.

1 Introduction

Deep learning and neural networks have achieved great success in a variety of applications in computer vision, signal processing, and scientific computing, to name a few. However, their robustness with respect to perturbations of the input data may considerably impact security and trustworthiness and poses a major drawback to their real-world application. Moreover, the memory and computational requirements for both training and inferring phases render them impractical in application settings with limited resources. While a broad literature on pruning methods and adversarial robustness has been developed to address these two issues in isolation, much less has been done to design neural networks that are both energy-saving and robust. Actually, in many approaches the two problems seem to compete against each other as most adversarial robustness improving-techniques require even larger networks [36, 41, 46, 47, 79] or computationally more demanding loss functions, and thus more expensive training phases [13, 23, 34, 44, 66].

The limited work available so far on robust pruned networks is mostly focused on reducing memory and computational costs of the inference phase, while retaining adversarial robustness [20, 28, 42, 56, 72, 76]. However, the inference phase amounts to only a very limited fraction of the cost of the

Emanuele Zangrando

Francesco Tudisco

whole deep learning pipeline, which is instead largely dominated by the training phase. Reducing both inference and training costs is a challenging but desirable goal, especially in view of a more accessible AI and its effective use on limited-resource and limited-connectivity devices such as drones or satellites.

Some of the most effective techniques for the reduction of training costs so far have been based on low-rank weights parametrizations [29, 55, 71]. These methods exploit the intrinsic low-rank structure of parameter matrices and large data matrices in general [17, 49, 59, 69]. Thus, assuming a low-rank structure for the neural network's weights W = USV ⊤ , the resulting training procedures only use the small individual factors U, S, V . This results in a training cost that scales linearly with the number of neurons, as opposed to a quadratic scaling required by training full-rank weights. Despite significantly reducing training parameters, these methods achieve accuracy comparable with the original full networks. However, their robustness with respect to adversarial perturbations has been largely unexplored so far.

Contributions

In this paper, we observe that the adversarial robustness of low-rank networks may actually deteriorate with respect to the full baseline. By modeling the robustness of the network in terms of the neural network's condition number, we argue that this loss of robustness is due to the exploding condition number of the low-rank weight matrices, whose singular values grow very large in order to match the baseline accuracy and to compensate for the lack of parameters. Thus, to mitigate this growing instability, we design an algorithm that trains the network using only the low-rank factors U, S, V while simultaneously ensuring the condition number of the network remains small. To this end, we interpret the loss optimization problem as a continuous-time gradient flow and use techniques from geometric integration theory on manifolds [12, 31, 55, 70] to derive three separate projected gradient flows for U, S, V , individually, which ensure the condition number of the network remains bounded to a desired tolerance 1 + τ , throughout the epochs. For a fixed small constant ε > 0 , this is done by bounding the singular values of the small rank matrices within a narrow band [ s -ε, s + ε ] around a value s , chosen to best approximate the original singular values.

We provide several experimental evaluations on different architectures and datasets, where the robust low-rank networks are compared against a variety of baselines. The results show that the proposed technique allows us to compute from scratch low-rank weights with bounded singular values, significantly reducing the memory demand and computational cost of training while at the same time retaining or improving both the accuracy and the robust accuracy of the original model. On top of the experimental evidence, we provide a key approximation theorem that shows that if a high-performing low-rank network with bounded singular values exists, then our algorithm computes it up to a first-order approximation error.

This paper focuses on feed-forward neural networks. However, our techniques and analysis apply straightforwardly to convolutional filters reshaped in matrix form, as done in e.g. [29, 55, 71]. Other ways exist to promote orthogonality of convolutional filters, e.g. [60, 65, 77], which we do not consider in this work.

2 Related work

Neural networks' robustness against adversarial perturbations has been extensively studied in the machine learning community. It is well-known that the adversarial robustness of a neural network is closely related to its Lipschitz continuity [13, 23, 62, 68], see also Section 3. Accordingly, training neural networks with bounded Lipschitz constant is a widely employed strategy to address the problem. A variety of works studied Lipschitz architectures [37, 60, 65, 68], and a number of certified robustness guarantees have been proposed [23, 51, 61]. While scaling each layer to impose 1-Lipschitz constraints is a possibility, this approach may lead to vanishing gradients and it is known that a more effective way to reduce the Lipschitz constant and increase robustness is obtained by promoting orthogonality on each layer [5, 13]. On top of robustness, small Lipschitz constants and orthogonal layers are known to lead to improved generalization bounds [11, 45] and more interpretable gradients [67]. Orthogonality was also shown to improve signal propagation in (very) deep networks [52, 74].

A variety of methods to integrate orthogonal constraints in deep neural networks have been developed over the years. Notable example approaches include methods based on regularization and landing [1, 13], cheap parametrizations of the orthogonal group [6, 38, 39, 48, 50], Riemannian and projected gradient descent schemes [2, 3, 10].

In parallel to the development of methods to promote orthogonality, an active line of research has grown to develop effective training strategies to enforce low-rank weights. Unlike sparsitypromoting pruning strategies that primarily aim at reducing the parameters required for inference [8, 21, 22, 30, 43], low-rank neural network models are designed to train directly on the lowparametric manifold of low-rank matrices and are particularly effective to reduce the number of parameters required by both inference and training phases. Similar to orthogonal training, methods for low-rank training include methods based on regularization [26, 29], as well as methods based on efficient parametrizations of the low-rank manifold using the SVD, randomized tensor dropout or the polar decomposition [32, 71, 75], and Riemannian optimization-based training models [55, 57].

By combining low-rank training with approximate orthogonal constraints, in this work we propose a strategy that simultaneously enforces robustness while only requiring a reduced percentage of the network's parameters during training. The method is based on a gradient flow differential formulation of the training problem, and the use of geometric integration theory to derive the governing equations of the low-rank factors. With this formulation, we are able to reduce the sensitivity of the network during training at almost no cost, yielding well-conditioned low-rank neural networks. Our experimental findings are supported by an approximation theorem that shows that, if the ideal full network can be approximated by a low-rank one, then our method computes a good approximation. This is well-aligned with recent work that shows the existence of high-performing low-rank nets in e.g. deep linear models [7, 17, 25, 49]. Moreover, as orthogonality helps in training really deep networks, low-rank orthogonal models may be used to mitigate the effect of increased effective depth when training low-rank networks [54].

3 The condition number of a neural network

The adversarial robustness of a neural network model f can be measured by the worst-case sensitivity of f with respect to small perturbations of the input data x . In an absolute sense, this boils down to measuring the best global and local Lipschitz constant of f with respect to suitable distances, as discussed in a variety of papers [13, 15, 23, 62]. However, as the model and the data may assume arbitrary large and arbitrary small values in general, a relative measure of the sensitivity of f may be more informative. In other words, if we assume a perturbation δ of small size as compared to x , we would like to quantify the largest relative change in f ( x + δ ) , as compared to f ( x ) . This is a well-known problem of conditioning, as we review next, and naturally leads to the concept of condition number of a neural network.

̸

In the linear setting, the condition number of a matrix is a widely adopted relative measure of the worst-case sensitivity of linear problems with respect to noise in the data. For a matrix A and the matrix operator norm ∥ A ∥ = sup x =0 ∥ Ax ∥ / ∥ x ∥ the condition number of A is defined as cond( A ) = ∥ A ∥∥ A + ∥ , where A + denotes the pseudo-inverse of A . Note that it is immediate to verify that cond( A ) ≥ 1 . Now, if for example u and u ε are the solutions to the linear system Au = b , when A and b are exact data or when they are perturbed with noise δ A , δ b of relative norm ∥ δ A ∥ / ∥ A ∥ ≤ ε and ∥ δ b ∥ / ∥ b ∥ ≤ ε , respectively, then the following relative error bound holds

Thus, small perturbations in the data A,b imply small alterations in the solution if and only if A is well conditioned, i.e. cond( A ) is close to one.

As in the linear case, it is possible to define the concept of condition number for general functions f , [24, 53]. Let us start by defining the relative error ratio of a function f : R d → R m in the point x :

In order to take into account the worst-case scenario, the local condition number of f at x is defined by taking the sup of (1) over all perturbations of relative size ε , i.e. such that ∥ δ ∥ ≤ ε ∥ x ∥ , in the

̸

limit of small ε . Namely, cond( f ; x ) = lim ε ↓ 0 sup δ =0: ∥ δ ∥≤ ε ∥ x ∥ R ( f, x ; δ ) . This quantity is a local measure of the 'infinitesimal' conditioning of f around the point x . In fact, a direct computation reveals that as long as ∥ δ ∥ ≤ ε ∥ x ∥ . Thus, cond( f ; x ) provides a form of relative local Lipschitz constant for f which in particular shows that, if ∥ δ ∥ / ∥ x ∥ is smaller than cond( f ; x ) -1 , we expect limited change in f when x is perturbed with δ . A similar conclusion is obtained using an absolute local Lipschitz constant in e.g. [23]. Similarly to the absolute case, a global relative Lipschitz constant can be obtained by looking at the worst-case over x , setting cond( f ) = sup x ∈X cond( f ; x ) . Clearly, the same bound (2) holds for cond( f ) . Note that this effectively generalizes the linear case, as when f ( x ) = Ax we have cond( f ) = cond( f, x ) = cond( A ) .

When f is a neural network, cond( f ) is a function of the network's weights and robustness may be enforced by reducing cond( f ) while training. In fact, cond( f ) is the relative equivalent of the network's Lipschitz constant and thus standard Lipschitz-based robustness certificates [23, 40, 68] can be recast in terms of cond( f ) . However, for general functions f and general norms ∥· ∥ , cond( f ) may be (very) expensive to compute, it may be non-differentiable, and cond( f ) > 1 can hold [24]. Fortunately, for feed-forward neural networks, it holds (proof and additional details moved to Appendix B in the supplementary material)

Proposition 1. Let X be the input space and let f ( x ) = z L +1 be a network with L linear layers z i +1 = σ i ( W i z i ) , i = 1 , . . . , L . Then,

In particular, for typical X i and typical choices of σ i , including σ i ∈ { leakyReLU , sigmoid , tanh , hardtanh , softplus , siLU } , we have

for a positive constant C > 0 that depends only on the activation function σ i .

Note that for entrywise nonlinearities σ , the condition number cond( σ ; x ) can be computed straightforwardly. In fact, when σ is Lipschitz, the problem can be reduced to a one-dimensional function, and it follows directly from its definition that (see also [64])

where ∂σ ( x ) denotes Clarke's generalized gradient [14] of σ at the point x . Thus, for example, if σ is LeakyRelu with slope α , we have cond( σ ) = 1 ; if σ is the logistic sigmoid (1 + e -x ) -1 and the feature space X i is such that X i = W i X i -1 , then if | z i -1 | ≤ c i -1 entry-wise, we have | x i | ≤ c i -1 max uv | W i | uv := c i and cond( σ ) ≤ sup x ≥-c i | x | e -x (1 + e -x ) -1 ≤ max { c i , 1 /e } .

From Proposition 1 we see that when f is a feed-forward network, to reduce the condition number of f it is enough to reduce the conditioning of all its weights. When ∥·∥ = ∥·∥ 2 is the Euclidean L 2 norm, we have cond 2 ( W ) = s max ( W ) /s min ( W ) , the ratio between the largest and the smallest singular value of W . This implies that orthogonal weight matrices, for example, are optimally conditioned with respect to the L 2 metric. Thus, a notable and well-known consequence of Proposition 1 is that imposing orthogonality constraints on W improves the robustness of the network [13, 27, 38, 50].

While orthogonal constraints are widely studied in the literature, orthogonal matrices are not the only optimally conditioned ones. In fact, cond 2 ( W ) = 1 for any W with constant singular values. In the next section, we will use this observation to design a low-rank and low-cost algorithm that trains well-conditioned networks by ensuring cond 2 ( W ) ≤ 1 + τ , for all layers W and a desired tolerance τ > 0 .

4 Robust low-rank training

4.1 Instability of low-rank networks

Low-rank methods are popular strategies to reduce the memory storage and the computational cost of both training and inference phases of deep learning models [29, 55, 71]. Leveraging the intrinsic

0

2000 4000 6000 8000

# optimization step

10 1

layers condition numbers

Baseline

0

2000 4000 6000 8000

# optimization step

10 0

10 1

10 2

10 3

layers condition numbers

Low-rank gradient flow

0

2000 4000 6000 8000

# optimization step

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

layers condition numbers

DLRT

0

200

400

600

# optimization step

10 1

10 2

10 3

10 4

10 5

10 6

10 7

layers condition numbers

Layer decomposition

low-rank structure of parameter matrices [7, 17, 49, 59], these methods train a subnetwork with weight matrices parametrized as W = USV ⊤ , for 'tall and skinny' matrices U, V with r columns, and a small r × r matrix S . Training low-rank weight matrices has proven to effectively reduce training parameters while retaining performance comparable to those of the full model. However, while a variety of contributions have analyzed and refined low-rank methods to match the full model's accuracy, the robust accuracy of low-rank models has been partially overlooked in the literature.

Here we observe that reducing the rank of the layer may actually deteriorate the network's robustness. We argue that this phenomenon is imputable to the exploding condition number of the network. In Figure 1 we plot the evolution of the condition number cond 2 for the four internal layers of LeNet5 during training using different low-rank training strategies and compare them with the full model. While the condition number of the full model grows moderately with the iteration count, the condition number of low-rank layers blows up drastically. This singular value instability leads to poor robustness performance of the methods, as observed in the experimental evaluation of Section 5.

In the following, we design a low-rank training model that allows imposing simple yet effective training constraints, bounding the condition number of the trained network to a desired tolerance 1 + τ , and improving the network robustness without affecting training nor inference costs.

4.2 Low-rank gradient flow with bounded singular values

Let W ∈ R n × m be the weight matrix of a linear layer within f . For an integer r ≤ min { m,n } let M r = { W : rank( W ) = r } be the manifold of rankr matrices which we parametrize as

Obviously, the singular values of W = USV ⊤ ∈ M r coincide the singular values of S . For s, ε such that 0 < ε < s , define Σ s ( ε ) as the set of matrices with singular values in the interval [ s -ε, s + ε ] . Note that Σ s (0) is a Riemannian manifold obtained essentially by an s scaling of the standard Stiefel manifold (the manifold of matrices with orthonormal columns) and any A ∈ Σ s (0) is optimally conditioned, i.e. cond 2 ( A ) = 1 . Thus, ε can be interpreted as an approximation parameter that controls how close Σ s ( ε ) is to the 'optimal' manifold Σ s (0) . To enhance the network robustness, in the following we will constrain the parameter weight matrix S to Σ s ( ε ) . With this constraint, we get cond 2 ( W ) ≤ ( s -ε ) -1 ( s + ε ) = 1 + τ , with τ = 2( s -ε ) -1 ε , so that the tolerance τ on the network's conditioning can be tuned by suitably choosing the approximation parameter ε .

Given the loss function L , we are interested in the constrained optimization problem

To approach (3), we use standard arguments from geometric integration theory [31, 70] to design a training scheme that updates only the factors U, S, V and the gradient of L with respect to U, S, V , without ever forming the full weights nor the full gradients. To this end, following [55], we first recast the optimization of L with respect to each layer W as a continuous-time gradient flow

where 'dot' denotes the time derivative and where we write L as a function of W only, for brevity. Along the solution of the differential equation above, the loss decreases and a stationary point is

Input: Chosen compression rate, i.e. for each layer W choose a rank r ; Initial layers' weights parametrized as W = USV ⊤ , with S ∼ r × r ; Second singular value moment of S , s = √∑ k s k ( S ) 2 /r Conditioning tolerance τ > 0

1 for each iteration and each layer do (each block in parallel) 2 U ← one optimization step with gradient G 1 and initial point U 3 U ← project U on Stiefel manifold with r orthonormal columns 4 V ← one optimization step with gradient G 2 and initial point V 5 V ← project V on Stiefel manifold with r orthonormal columns 6 S ← one optimization step with gradient G 3 and initial point S 7 s ← √∑ k s k ( S ) 2 /r , squareroot of second moment of the singular values of S 8 ε ← τs/ (2 + τ ) 9 S ← project S onto Σ s ( ε )

approached as t →∞ . Now, if we assume W ∈ M r , then ˙ W ∈ T W M r , the tangent space of M r at the point W . Thus, to ensure the whole trajectory W ( t ) ∈ M r , we can consider the projected gradient flow ˙ W ( t ) = -P W ( t ) ∇ W L ( W ( t )) , where P W denotes the orthogonal projection (in the ambient space of matrices) onto T W M r . Next, we notice that the projection P W ∇ W L can be defined by imposing orthogonality with respect to any point Y ∈ T W M r , namely

where ⟨· , ·⟩ is the Frobenius inner product. As discussed in e.g. [31, 55], the above equations combined with the well-known representation of T W M r yield a system of three gradient flow equations for the individual factors

where P ⊥ U = ( I -UU ⊤ ) and P ⊥ V = ( I -V V ⊤ ) are the projection operators onto the space orthogonal to the span of U and V , respectively.

Based on the system of gradient flows above, we propose a training scheme that at each iteration and for each layer parametrized by the tuple { U, S, V } proceeds as follows:

update U and V by numerically integrating the gradient flows ˙ U = -G 1 ( U ) and ˙ V = -G 2 ( V )

project the resulting U, V onto the Stiefel manifold of matrices with r orthonormal columns

update the r × r weight S by integrating ˙ S = -G 3 ( S )

for a fixed robustness tolerance τ , project the computed S onto Σ s ( ε ) , choosing s and ε so that

s is the best constant approximation to S ⊤ S , i.e. s = argmin α ∥ S ⊤ S -α 2 I ∥ F

ε is such that the cond 2 of the projection of S does not exceed 1 + τ

Note that the coefficients s , ε at point 4 can be obtained explicitly by setting s = √ ∑ r j =1 s j ( S ) 2 /r , the second moment of the singular values s j ( S ) of S , and ε = τs/ (2 + τ ) . Note also that, in the differential equations for U, S, V in (5), the four steps above can be implemented in parallel for each of the three variables. The detailed pseudocode of the training scheme is presented in Algorithm 1. We conclude with several remarks about its implementation.

Remarks, implementation details, and limitations

Each step of Algorithm 1 requires three optimization steps at lines 2, 4, 6. These steps can be implemented using standard first-order optimizers such as SGD with momentum or ADAM. Standard techniques can be used to project onto the Stiefel manifold at lines 3 and 5 of Algorithm 1, see e.g. [4, 70]. Here, we use the QR decomposition. As for the projection onto Σ s ( ε ) at line 9, we

compute the SVD of the small factor S and set to s + ε or s -ε the singular values that fall outside the interval [ s -ε, s + ε ] . Note that, when τ = 0 , i.e. when we require perfect conditioning for the layer weight W = USV ⊤ , then the SVD of S can be replaced by a QR step or any other Stiefel manifold projection. Indeed, we can equivalently set s = √ trace( S ⊤ S ) /r , and then project onto Σ s (0) by rescaling by a factor s the projection of S onto the Stiefel manifold. In this case, the system (5) further simplifies, as we can replace ( SS ⊤ ) -1 and ( S ⊤ S ) -⊤ with the scalar 1 /s 2 .

Overall, the compressed low-rank network has r ( n + m + r ) parameters per each layer, where n and m are the number of input and output neurons. Thus, choosing r so that 1 -r ( n + m + r ) / ( nm ) = α can yield a desired compression rate 0 < α < 1 on the number of network parameters, i.e. the number of parameters one eliminates with respect to the full baseline. For example, in our experiments we will choose r so that α = 0 . 5 or α = 0 . 8 .

Computational complexity. Each pass of Alg.1 is done against a batch x batch . In order to obtain minimal computational costs for each step in the algorithm, we evaluate USV ⊤ x batch sequentially: first v = V ⊤ x batch , then u = Sv , and finally Uu . Assuming the size of the batch is negligible with respect to n and m , the cost of these steps is O ( rm ) , O ( r ) , O ( rn ) , respectively. Adding the bias term and evaluating the activation function requires O ( n ) operations. Hence, overall we have a cost per layer of O ( r ( n + m +1)) . Taping the forward evaluation to compute the gradient with respect to U, S, V does not affect the asymptotic costs. The QR decompositions used for U and V require O ( r 2 n ) and O ( r 2 m ) operations respectively, O ( r 2 ( n + m )) overall. Finally, computing the SVD in the projection step for S requires a worst-case cost of O ( r 3 ) . Hence the overall cost per layer is O ( r (1 + r )( n + m ) + r 3 ) as opposed to the dense network training, which requires O ( nm ) operations per layer. If r ≪ n, m then the low-rank method is cheaper than the full baseline. For example, if n = m , this happens provided r < √ n .

Limitations. As the rank parameter r has to be chosen a-priori for each layer of the network, a limitation of the proposed approach is the potential need for fine-tuning such parameter, even though the proposed analysis in Table 1 shows competitive performance for both 50% and 80% compression rates. Also, if the layer size n × m is not large enough, the compression ratio cr = 1 -r ( n + m + r ) / ( nm ) might be limited. Thus the method works well only for wide-enough networks ( n, m ≫ r , so that cr > 0 ). Finally, a standard way to obtain better adversarial performance would be to combine the proposed conditioning-based robustness with adversarial training strategies [16, 63, 73]. However, the cost of producing adversarial examples during training is not negligible, especially when based on multi-step attacks, and thus the way to incorporate adversarial training without affecting the benefits obtained with low-rank compression is not straightforward.

4.3 Approximation guarantees

Optimization methods over the manifold of low-rank matrices are well-known to be affected by the stiff intrinsic geometry of the constraint manifold which has very high curvature around points where W ∈ M r is almost singular [4, 31, 70]. This implies that even very small changes in W may yield very different tangent spaces, and thus different training trajectories, as shown by the result below:

Lemma 1 (Curvature bound, Lemma 4.2 [31]) . For W ∈ M r let s min ( W ) > 0 be its smallest singular value. For any W ′ ∈ M r arbitrarily close to W and any matrix B , it holds

where C > 0 depends only on B .

In our gradient flow terminology, this phenomenon is shown by the presence of the matrix inversion in (5). While this is often an issue that may dramatically affect the performance of low-rank optimizers (see also Section 5), the proposed regularization step that enforces bounded singular values allows us to move along paths that avoid stiffness points. Using this observation, here we provide a bound on the quality of the low-rank neural network computed via Algorithm 1, provided there exists an optimal trajectory leading to an approximately low-rank network. We emphasize that this assumption is well-aligned with recent work showing the existence of high-performing low-rank nets in e.g. deep linear models [7, 17, 25, 49].

Assume the training is performed via gradient descent with learning rate λ > 0 , and let W ( t ) be the full gradient flow (4). Further, assume that for t ∈ [0 , λ ] and a given ε > 0 , for each layer there exists

where s is the second moment of the singular values of W ( t ) and E ( t ) is a perturbation that has bounded variation in time, namely ∥ ˙ E ( t ) ∥ ≤ η . In other words, we assume there exists a training trajectory that leads to an approximately low-rank weight matrix W ( t ) with almost constant singular values. Because the value s is bounded by construction, the parameter-dependent matrix ˜ W ( t ) possesses singular values exhibiting moderate lower bound. Thus, W ( t ) is far from the stiffness region of (5) and we obtain the following bound, based on [31] (proof moved to Appendix C in the supplementary material)

Theorem 1. Let U k S k V ⊤ k be a solution to (5) computed with k steps of Algorithm 1. Assume that

The low-rank initialization U 0 S 0 V ⊤ 0 coincides with the low-rank approximation ˜ W (0) .

The norm of the full gradient is bounded, i.e., ∥∇ W L ( W ( t )) ∥ ≤ µ .

The learning rate is bounded as λ ≤ s -ε 4 √ 2 µη .

Then, assuming no numerical errors, the following error bound holds

Note that if f is smooth enough (e.g. Lipschitz) then by the previous theorem we directly obtain an equivalent bound for the functional distance ∥ f ( U k S k V ⊤ k ) -f ( W ( λk )) ∥ .

5 Experiments

We illustrate the performance of Algorithm 1 on a variety of test cases. All the experiments can be reproduced with the code in PyTorch available at https://github.com/COMPiLELab/CondLR . In order to assess the combined compression and robustness performance of the proposed method, we compare it against both full and low-rank baselines.

For all models, we compute natural accuracy and robust accuracy. Let { ( x i , y i ) } i =1 ,...,n be the set of test images and the corresponding labels and let f be the neural network model, with output f ( x ) on the input x . We quantify the test set robust accuracy as:

✶ where δ = ( δ i ) i =1 ,...,n are the adversarial perturbation associated to each sample. Notice that, in the unperturbed case with ∥ δ i ∥ = 0 , the definition of robust accuracy exactly coincides with the definition of test accuracy. In our experiments, adversarial perturbations are produced by both the fast gradient sign method (FGSM) [19] and the projected gradient descent attack (PGD) [47], with ∥ δ i ∥ ∞ = ϵ , and ϵ controls perturbation strength. As images in our set-up have input entries in [0 , 1] , the perturbed input is then clamped to that interval. Note that, for the same reason, the value of ϵ controls in our case the relative size of the perturbation.

Datasets. We consider MNIST , CIFAR10, and CIFAR100 [33] datasets for evaluation purposes. The first contains 60,000 training images, the second one contains 50,000 training images, while the third one contains 50,000 training images. All the datasets have 10,000 test images, first two have 10 classes and last one has 100 classes. No data-augmentation is performed.

Models. We use LeNet5 [35] for MNIST dataset, VGG16 [58] and WideResnet (WRN16-4) [78] for CIFAR10 and CIFAR100. The general architecture for all the used networks is preserved across the models, while the weight-storing structures and optimization frameworks differ.

Methods. Our baseline network is the one done with the standard implementation. Cayley SGD [39] and Projected SGD [3] are Riemannian optimization-based methods that train the network weights over the Stiefel manifold of matrices with orthonormal columns. Thus, both methods ensure cond 2 ( W ) = 1 for all layers. The former uses an iterative estimation of the Cayley transform, while the latter uses QR-based projection to retract the Riemannian gradient onto the Stiefel manifold. Both methods have no compression and use full-weight matrices. DLRT, SVD prune, and Vanilla are low-rank methods that ensure compression of the model parameters during training. DLRT [55] is based on a low-rank gradient flow model similar to the proposed Algorithm 1. SVD prune [75] is based on a regularized loss with a low-rank-promoting penalty term. This approach was designed to

compress the ranks of the network after training, but we imposed a fixed compression rate from the beginning for a fair comparison. 'Vanilla' denotes the obvious low-rank approach, that parametrizes the layers as W = UV ⊤ and performs alternate descent steps over U and V , as done in e.g. [29, 71]. All models are implemented with fixed training compression ratios α = 0 . 5 and α = 0 . 8 , i.e. we only use 50% and 20% of the parameters the full model would use during training, respectively. Finally, we implement CondLR as in Algorithm 1 for three choices of the conditioning tolerance τ ∈ { 0 , 0 . 1 , 0 . 5 } . We also implement a modified version of Algorithm 1 in which S is directly projected onto the Stiefel manifold Σ 1 (0) , i.e. the parameter s is fixed to one.

Training. Each method and model was trained for 120 epochs of stochastic gradient descent with a minibatch size of 128. We used a learning rate of 0.1 for LeNet5 and 0.05 for VGG16 with momentum 0.3 and 0.45, respectively, and a learning rate scheduler with factor = 0.3 at 70 and 100 epochs.

Results. For comparison, we measure robust accuracy for all chosen combinations of datasets, models, and methods using FGSM and PGD attacks with relative perturbation budget ϵ ∈ { 0 . 01 , 0 . 02 , 0 . 030 . 04 , 0 . 05 , 0 . 06 } for MNIST, ϵ ∈ { 0 . 001 , 0 . 002 , 0 . 003 , 0 . 004 , 0 . 005 , 0 . 006 } for CIFAR10 VGG16, ϵ ∈ { 0 . 0003 , 0 . 0006 , 0 . 001 , 0 . 0013 , 0 . 0016 } for CIFAR10 and CIFAR100 with WRN16-4.

The results are summarized in Table 1 for LeNet5 MNIST and VGG16 CIFAR10 with FGSM and a subset of the perturbation budget, and in Table 2 for WRN16-4 CIFAR10 with both FGSM and PGD. The additional results are presented in Tables 3-6 and are moved to Appendix A. In the tables, we highlight (in gray) the best-performing method for each range of compression rates α ∈ { 0% , 50% , 80% } , and the best method overall (in bold). The experiments show that the proposed method meaningfully conserves the accuracy as compared to the baseline for all datasets; moreover, the robustness is improved from the baseline for 50% compression rate and, additionally, in the case of MNIST for 80% compression rate. Compared to orthogonal non-compressed methods, CondLR with 50% compression rate also outperforms Cayley SGD and Projected SGD. At the same time CondLR is compressed by design, so we reduce memory costs alongside gaining robustness. As anticipated in Section 4.1, low-rank methods without regularization deteriorate the condition number and thus the robustness of the model. This is consistent with the results of Table 1, where we observe that compression without regularization does not work well in terms of robust accuracy, in particular DLRT, Vanilla and SVD prune exhibit a drop in the performance in comparison with baseline and, consequently, CondLR . Specifically, per each fixed compression ratio α , our method exhibits the highest robustness; moreover, the robustness of CondLR with α = 0 . 8 is higher than the robustness of any other method with an even lower compression ratio, α = 0 . 5 .

As a further experimental evaluation, we analyze the robustness of low-rank training models with respect to small singular values. As shown by Theorem 1, CondLR is not affected by the steep curvature of the low-rank manifold M r , i.e. accuracy and convergence rate of CondLR do not deteriorate when the conditioning of the weight matrices explodes. In contrast, small singular values may significantly affect alternative low-rank training models. In Figure 2 we show the behavior of loss, accuracy, and condition number as functions of the iteration steps, when the network is initialized with ill-conditioned layer matrices. Precisely, similar to what is done in [29], we randomly sample Gaussian initial weights, compute their SVD to define the initial U and V factors, and then force the singular values of the initial S factor to decay exponentially. We observe that CondLR and DLRT (which is also based on a low-rank gradient flow formulation) are the most robust, while the performance of the other methods deteriorates dramatically. This confirms that, as also observed in [29, 55], most low-rank approaches require specific fine-tuned choices of the initialization, while the proposed robust low-rank training model ensures solid performance independent of the initialization.

6 Conclusions

In recent years, extensive effort has been put into (a) studying different forms of implicit bias in deep learning, including bias towards low-rank parameter weights [7, 9, 18], and (b) designing modified training techniques that take advantage of the implicit low-rank structure to reduce training costs and compress the network [29, 55, 71, 75]. Based on the notion of condition number of a network, as a form of local Lipschitz constant, in this work we observe that training directly on the manifold of matrices with a fixed rank may lead to instabilities due to bad conditioning, going against the intuition that low-rank networks may be more robust to adversarial attacks. Thus, we propose an

algorithm able to mitigate this phenomenon in a computationally affordable way, allowing us to train directly on the manifold of matrices of a fixed rank while controlling the condition number.

0

200

400

600

800

1000 1200 1400

Optimization step

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

0.0150

0.0175

0.0200

Loss

0

200

400

600

800

1000 1200 1400

Optimization step

0.0

0.2

0.4

0.6

0.8

1.0

0

200

400

600

800

1000 1200 1400

Optimization step

10 3

10 8

10 13

10 18

10 23

10 28

10 33

10 38

Upper bound cond(f)

CondLR, =0

DLRT

SVD Prune

Vanilla

References

P. Ablin and G. Peyré. Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference on Artificial Intelligence and Statistics , pages 5636-5657. PMLR, 2022.

P.-A. Absil, R. Mahony, and R. Sepulchre. Optimization algorithms on matrix manifolds . Princeton University Press, 2008.

P.-A. Absil and J. Malick. Projection-like retractions on matrix manifolds. SIAM Journal on Optimization , 22(1):135-158, 2012.

P.-A. Absil and I. V . Oseledets. Low-rank retractions: a survey and new results. Computational Optimization and Applications , 62(1):5-29, 2015.

C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning , pages 291-301. PMLR, 2019.

M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning , pages 1120-1128. PMLR, 2016.

S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

A. Ashok, N. Rhinehart, F. Beainy, and K. M. Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning. In International Conference on Learning Representations , 2018.

B. Bah, H. Rauhut, U. Terstiege, and M. Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers, 2020.

N. Bansal, X. Chen, and Z. Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems , 31, 2018.

P. L. Bartlett, D. J. Foster, and M. J. Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

G. Ceruti and C. Lubich. An unconventional robust integrator for dynamical low-rank approximation. BIT Numerical Mathematics , 62(1):23-44, 2022.

M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning , pages 854-863. PMLR, 2017.

F. H. Clarke. Optimization and nonsmooth analysis . SIAM, 1990.

J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning , pages 1310-1320. PMLR, 2019.

J. Ding, T. Bu, Z. Yu, T. Huang, and J. Liu. Snn-rat: Robustness-enhanced spiking neural network through regularized adversarial training. Advances in Neural Information Processing Systems , 35:24780-24793, 2022.

R. Feng, K. Zheng, Y. Huang, D. Zhao, M. Jordan, and Z.-J. Zha. Rank diminishing in deep neural networks. arXiv:2206.06072 , 2022.

T. Galanti, Z. S. Siegel, A. Gupte, and T. Poggio. Sgd and weight decay provably induce a low-rank bias in neural networks, 2023.

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2015.

S. Gui, H. Wang, H. Yang, C. Yu, Z. Wang, and J. Liu. Model Compression with Adversarial Robustness: A Unified Optimization Framework . 2019.

Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. AMC: AutoML for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision , pages 784-800, 2018.

Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision , pages 1389-1397, 2017.

M. Hein and M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. Advances in neural information processing systems , 30, 2017.

D. J. Higham. Condition numbers and their condition numbers. Linear Algebra and its Applications , 214:193-213, 1995.

M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research , 2023.

Y. Idelbayev and M. A. Carreira-Perpinán. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8049-8059, 2020.

K. Jia, D. Tao, S. Gao, and X. Xu. Improving training of deep neural networks via singular value bounding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4344-4352, 2017.

A. Jordao and H. Pedrini. On the effect of pruning on adversarial robustness. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) . IEEE Computer Society, 2021.

M. Khodak, N. Tenenholtz, L. Mackey, and N. Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations , 2021.

H. Kim, M. U. K. Khan, and C.-M. Kyung. Efficient neural network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12569-12577, 2019.

O. Koch and C. Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications , 29(2):434-454, 2007.

A. Kolbeinsson, J. Kossaifi, Y. Panagakis, A. Bulat, A. Anandkumar, I. Tzoulaki, and P. M. Matthews. Tensor dropout for robust learning. IEEE Journal of Selected Topics in Signal Processing , 15(3):630-640, 2021.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

P. Langenberg, E. R. Balda, A. Behboodi, and R. Mathar. On the effect of low-rank weights on adversarial robustness of neural networks. arXiv preprint arXiv:1901.10371 , 2019.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

H. Lee, S. Han, and J. Lee. Generative adversarial trainer: Defense to adversarial perturbations with gan, 2017.

K. Leino, Z. Wang, and M. Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning , pages 6212-6222. PMLR, 2021.

M. Lezcano-Casado and D. Martınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning , pages 3794-3803. PMLR, 2019.

J. Li, L. Fuxin, and S. Todorovic. Efficient Riemannian optimization on the Stiefel manifold via the Cayley transform. 2019.

Q. Li, S. Haque, C. Anil, J. Lucas, R. B. Grosse, and J.-H. Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems , 32, 2019.

Y. Li, Z. Yang, Y. Wang, and C. Xu. Neural architecture dilation for adversarial robustness. In Advances in Neural Information Processing Systems , 2021.

N. Liao, S. Wang, L. Xiang, N. Ye, S. Shao, and P. Chu. Achieving adversarial robustness via sparsity. Machine Learning , pages 1-27, 2022.

H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representations for efficient architecture search. In International Conference on Learning Representations , 2018.

X. Liu, Y. Li, C. Wu, and C.-J. Hsieh. Adv-BNN: Improved adversarial defense through robust bayesian neural network. In International Conference on Learning Representations , 2019.

P. M. Long and H. Sedghi. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations , 2020.

D. Madaan, J. Shin, and S. J. Hwang. Adversarial neural pruning with latent vulnerability suppression. In ICML , 2020.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

K. D. Maduranga, K. E. Helfrich, and Q. Ye. Complex unitary recurrent neural networks using scaled cayley transform. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4528-4535, 2019.

C. H. Martin and M. W. Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research , 22(165):1-73, 2021.

E. Massart. Orthogonal regularizers in deep learning: how to handle rectangular matrices? In 2022 26th International Conference on Pattern Recognition (ICPR) , pages 1294-1299. IEEE, 2022.

L. Meunier, B. J. Delattre, A. Araujo, and A. Allauzen. A dynamical system perspective for lipschitz neural networks. In International Conference on Machine Learning , pages 1548415500. PMLR, 2022.

J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems , 30, 2017.

J. R. Rice. A theory of condition. SIAM Journal on Numerical Analysis , 3(2):287-310, 1966.

D. A. Roberts, S. Yaida, and B. Hanin. The Principles of Deep Learning Theory . Cambridge University Press, may 2022.

S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processing Systems , 2022.

V. Sehwag, S. Wang, P. Mittal, and S. Jana. Hydra: Pruning adversarially robust neural networks. NeurIPS , 2020.

U. Shalit, D. Weinshall, and G. Chechik. Online learning in the manifold of low-rank matrices. Advances in neural information processing systems , 23, 2010.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

S. P. Singh, G. Bachmann, and T. Hofmann. Analytic insights into structure and rank of neural network Hessian maps. In Advances in Neural Information Processing Systems , volume 34, 2021.

S. Singla and S. Feizi. Skew orthogonal convolutions. In International Conference on Machine Learning , pages 9756-9766. PMLR, 2021.

S. Singla, S. Singla, and S. Feizi. Improved deterministic l2 robustness on cifar-10 and cifar-100. In International Conference on Learning Representations , 2021.

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR) , 2014.

D. Terjék. Adversarial lipschitz regularization. In International Conference on Learning Representations , 2020.

L. N. Trefethen and D. Bau. Numerical Linear Algebra . SIAM, 1997.

A. Trockman and J. Z. Kolter. Orthogonalizing convolutional layers with the cayley transform. In International Conference on Learning Representations , 2021.

T. Tsiligkaridis and J. Roberts. On frank-wolfe adversarial training. In ICML 2021 Workshop on Adversarial Machine Learning , 2021.

D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations , 2018.

Y. Tsuzuku, I. Sato, and M. Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. Advances in neural information processing systems , 31, 2018.

M. Udell and A. Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science , 1(1):144-160, 2019.

A. Uschmajew and B. Vandereycken. Geometric methods on low-rank matrix and tensor manifolds. Handbook of variational methods for nonlinear geometric data , pages 261-313, 2020.

H. Wang, S. Agarwal, and D. Papailiopoulos. Pufferfish: communication-efficient models at no extra cost. Proceedings of Machine Learning and Systems , 3:365-386, 2021.

X. Wei, Y. Xu, Y. Huang, H. Lv, H. Lan, M. Chen, and X. Tang. Learning extremely lightweight and robust model with differentiable constraints on sparsity and condition number. In European Conference on Computer Vision , pages 690-707. Springer, 2022.

Y.-L. Wu, H.-H. Shuai, Z.-R. Tam, and H.-Y. Chiu. Gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6373-6382, 2021.

L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning , pages 5393-5402. PMLR, 2018.

H. Yang, M. Tang, W. Wen, F. Yan, D. Hu, A. Li, H. Li, and Y. Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 2899-2908, 2020.

S. Ye, K. Xu, S. Liu, H. Cheng, J.-H. Lambrechts, H. Zhang, A. Zhou, K. Ma, Y. Wang, and X. Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

T. Yu, J. Li, Y . Cai, and P. Li. Constructing orthogonal convolutions in an explicit manner. In International Conference on Learning Representations , 2022.

S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning , pages 7472-7482. PMLR, 2019.

A Additional results

In this section, we report the results on natural and robust accuracy for the broader range of perturbation budgets for Table 1 ϵ ∈ { 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 , 0 . 06 } for LeNet5 on MNIST, and ϵ ∈ { 0 . 001 , 0 . 002 , 0 . 003 , 0 . 004 , 0 . 005 , 0 . 006 } for VGG16 on CIFAR10. See Table 3 and Table 4 (standard deviation is shown in gray). Additionally we provide results for Projected Gradient Descent attack with L2 norm and 10 steps for VGG16 on CIFAR10 with perturbation budget ϵ ∈ { 0 . 1 , 0 . 13 , 0 . 16 , 0 . 2 , 0 . 23 , 0 . 26 , 0 . 3 } . See Table 5. The results confirm the same findings as the ones reported in the main paper. Table 6 provides results of comparison between baseline and Algorithm 1 for WideResnet model with depth 16 and width 4 on CIFAR100 for both FGSM and PGD attack with L ∞ norm with perturbation budget ϵ ∈ { 0 . 0003 , 0 . 0006 , 0 . 001 , 0 . 0013 , 0 . 0016 } .

B Proof of Proposition 1

Lemma 2. Let ϕ : ( Y, ∥ · ∥ Y ) → ( Z, ∥ · ∥ Z ) and ψ : ( X, ∥ · ∥ X ) → ( Y, ∥ · ∥ Y ) be continuous mappings between finite-dimensional Banach spaces, and assume cond( ψ ) and cond( ϕ ) are well defined. Then the following inequality holds:

Proof. To prove this lemma, we will pass through the definition of R ( ϕ ◦ ψ, x, δ ) . Let us recall that it is defined as

̸

Now, if ψ ( x + δ ) -ψ ( x ) = 0 for δ = 0 , then R ( ϕ ◦ ψ, x, δ ) = 0 . Since we're interested in the supremum, we can restrict to δ such that ψ ( x + δ ) -ψ ( x ) = 0 . Moreover, by multiplying and

̸

dividing by ∥ ψ ( x ) ∥ Y we obtain:

̸

Now, if we define cond( f, x, ε ) = sup δ =0: ∥ δ ∥ X ≤ ε R ( f, x, δ ) , we take the supremum both sides on the set { δ ∈ X | ∥ δ ∥ X ≤ ε } and we can upper bound it with the product of the suprema of the two blocks

̸

̸

Now, letting η = ψ ( x + δ ) -ψ ( δ ) , we can rewrite the second part of the last equation as a sort of restriction of cond( ϕ, ψ ( x ) , ε ) to the particular set of perturbation directions of the form η . Thus, we can lower-bound it as:

̸

By hypothesis, cond( ϕ ) and cond( ψ ) are finite, so we can take the limit ε ↓ 0 to get:

Finally, taking the supremum over x on both members of the last equation, we can upper bound it with the original condition number without constraint on the directions:

and thus conclude.

Now, the proof of Proposition 1 follows directly by applying the Lemma above recursively.

Proof of Proposition1. By defining T i ( z ) = W i z , we can write the neural network f as

for nonlinear activation functions σ i . Thus, for L = 1 , the thesis follows directly from Lemma2, as long as cond( σ 1 ) ≤ C < ∞ . For L > 1 , one can unwrap the compositional structure of f from the left, defining ϕ ( z ) = ( σ L ◦ T L )( z ) and ψ ( x ) = ( σ L -1 ◦ · · · ◦ T 1 )( x ) . Then by using Lemma2 we have that

Now, since ψ is a network of depth L -1 , we can proceed inductively to obtain that cond( f ) ≤ C ∏ i cond( T i ) , with C = Π L i =1 cond( σ i ) , and we conclude.

Note that the result above is of interest only if cond( σ ) = sup x ∈X cond( σ ; x ) < ∞ . When σ is Lipschitz, using the formula

CondLR

CondLR

with ∂ being the Clarke's generalized gradient operator [14], we observe below that this is the case for a broad list of activation functions σ and feature spaces X .

LeakyReLU. For x ∈ R , α > 0 , let σ ( x ) = max { 0 , x } + α min { 0 , x } . Then any ν x ∈ ∂σ ( x ) is such that ν x = 1 if x > 0 ; ν x = α if x < 0 ; ν x = [min( α, 1) , max( α, 1)] otherwise. Thus

̸

̸

Tanh. For x ∈ R , let σ ( x ) = tanh( x ) . Then σ ′ ( x ) = 1 cosh 2 ( x ) and thus

Since the maximum of cond( σ, x ) is reached at zero, where the function can be extended by continuity.

Hardtanh. For x ∈ [ -a, a ] and a > 0 , let σ ( x ) = a 1 x>a -a 1 x< -a + x 1 x ∈ [ -a,a ] . Then, we have that ∂σ ( x ) coincides with the derivative values in all points but x = ± a . In those two points, we have ∂σ ( ± a ) = [0 , 1] . Thus, for any ν x ∈ ∂σ ( x ) , we have

Therefore, when x ≥ 0 , we have | x | e -x ≤ 1 /e and (1 + e -x ) ≥ 1 , thus cond( σ ; x ) ≤ 1 /e .

Softplus. For x ∈ R , let σ ( x ) = ln(1+ e x ) . Then σ ′ ( x ) = S ( x ) = (1+ e -x ) -1 and cond( σ ; x ) = | x | S ( x ) σ ( x ) -1 . Thus, for x ≥ 0 , we have cond( σ ; x ) ≤ 1 .

SiLU. For x ∈ R let σ ( x ) = x (1 + e -x ) -1 = xS ( x ) . Then, σ ′ ( x ) = S ( x ) + xS ( x )(1 -S ( x )) and thus for any x ≥ 0 we have

C Proof of Theorem 1

In the following, the proof of the main approximation result is presented. We underline that the core part of the proof relies on [31, Theorem 5.2]. For completeness, we repropose here main elements of the argument. We refer the interested reader to [31] and references therein for further details.

Proof. Let Y ( t ) be the solution of (5) at time t ∈ [0 , λ ] . First, we observe that the projected subflows of W ( t ) = ˜ W ( t ) + E ( t ) and ˜ W ( t ) satisfy the differential equations

where P ( · ) denotes the orthogonal projection into the tangent space of the low-rank manifold M r . Next, we observe that the following identities hold

where P ⊥ ( · ) = I -P ( · ) represents the complementary orthogonal projection. The latter implies that

Let γ = 32 µ ( s -ε ) -2 . It follows from [31, Lemma 4.2] that

Further, we remind that

Hence, the error e ( t ) = ∥ Y ( t ) -˜ W ( t ) ∥ satisfies the differential inequality

The error e ( t ) for t ∈ [0 , λ ] admits an upper bound given by the solution of

The last differential initial-value problem admits a closed solution given by

where the last term is bounded by 2 tη for t √ γη ≤ 1 . The proof thus concludes as follows

where the last estimate arise by the integral identity E ( t ) = ∫ t 0 ˙ E ( s ) ds .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work demonstrates that conventional low-rank training methods suffer from poor adversarial robustness, which is hypothesized to stem from ill conditioning of weight matrices during training. The authors address this by constraining low-rank weight matrices to be approximately orthogonal.\n\nSTRENGTHS:\nI found this paper to be a solid contribution, although my lack of familiarity with low-rank approaches means I'm not exactly sure which parts of the paper are novel.\n\n1. The observation in Figure 1 that low-rank training methods lead to exploding conditioning numbers is quite interesting and is not something I have seen before in the literature.\n2. The empirical gains in the experimental section are substantial, even if only using a FGSM atatck.\n3. The paper is very well-written and organized, with clear explanations and visualizations.\n4. The approximation guarantee is a nice inclusion of theory to explain the numerical advantages of avoiding high-curvature stiffness points.\n\nWEAKNESSES:\n5. The observation behind S1 was left unexplained: why do low-rank constrained matrices lead to large condition numbers? This might be out of scope but felt somewhat unsatisfying.\n6. The empirical evaluations only used the fast gradient sign method, which is a weak attack. This is my biggest reason for not giving an full accept rating -- I'd like to at least see PGD on this problem, although the results for FGSM are reasonably convincing.\n\nQUESTIONS:\n7. Precisely which parts of the algorithm in 4.2 are original? My impression is that the main contribution is step (4), the projection onto $\\Sigma_s(\\epsilon)$.\n8. In proposition 1, X is referred to as the \"feature space\" -- is this a deliberate deviation from the more standard terminology of \"input space\"? Or am I missing something here.\n9. Related to question 2, I'm a little confused about the notation in Proposition 1. The $cond(f; x)$ definition boils down to the function $f$ taking $x$ as an input; proposition 1 uses $cond(\\sigma_i; x)$, even though $\\sigma_i$ is at an intermediate layer in the network and does not directly take $x$ as an argument. My interpretation would be that this should read $\\cond(\\sigma_i, W_i z_i)$. It doesn't really affect anything downstream as we're just taking the sup over the argument to the condition number anyways, but the notation seems a bit imprecise to me as on the LHS of the inequality we're taking the sup over caligraphic X representing the input space, and on the right hand side it's the sup over caligraphic X representing the feature space, I feel that each intermediate feature space should have its own notation $X_i$ or something of the sort.\n10. In 157 the condition number for logistic sigmoid requires nonnegative features, which does not seem to be satisfied in general for intermediate neural network layers.\n11. In theorem 1, what is the significance of the phrase \"assuming no numerical errors\" -- is this referring to rounding errors, or something more serious like non-invertibility of a matrix during execution?\n12. The definition of relative error in equation (1) doesn't sit perfectly right with me, as it is dependent on the normalization of the inputs. If all inputs were shifted by some large amount, the relative error would increase although nothing important about the data has changed. Also, the relative error at x=0 is always zero (or undefined?) which doesn't make a lot of sense to me. Would appreciate any clarity here.\n\nLIMITATIONS:\nLimitations are explicitly addressed in section 4.2.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training approach that can improve the robustness of low-rank DNN models from scratch. This method enforces the low-rank components onto the Stifel manifold and ensures well-conditioning to achieve robustness. Experimental results show that the proposed training method demonstrates robustness to the data perturbation.\n\nSTRENGTHS:\n+ The robustness of low-rank format DNN models is an important topic because it can improve the security of efficient DNN models.\n+ Training low-rank models without initializing from a pre-trained dense model is also important to save energy costs.\n+ This paper provides much theoretical stuff.\n\nWEAKNESSES:\n- The contribution is incremental. Improving the robustness of a compressed model using condition number is already proposed in [R1]. The main contribution of this paper is to re-apply it to low-rank formats. Improving the robustness of low-rank DNN models is also investigated in [R2].\n- The motivation why using conditioning numbers in low-rank scenarios is unclear.\n- This paper leverages Riemannian optimization to enforce the orthogonality of the left and right components, $U$ and $V$. However, this optimization algorithm contains a lot of matrix multiplications, which are extremely costly. The overall computational cost is overhead, so the proposed method is opposite to the meaning of low-rank training, which aims to save training FLOPs.\n- In terms of the compression approach, why should we choose low-rankness? The overall performance of robustness and compression ratio is better than sparsity or other trends of compression techniques? This paper does not compare with any other related works like [R1] in experiments.\n- The dataset and selected models in experiments are trivial. Lager datasets like ImageNet and advanced models like ResNet-50 and MobileNet should be tested.\n\n[R1] Learning Extremely Lightweight and Robust Model with Differentiable Constraints on Sparsity and Condition Number\n[R2] On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-13"}

comments: SUMMARY:
In this paper, the authors propose a low-rank training algorithm that incorporates approximate orthonormal constraints. Through a comparative analysis with previous methods for training orthonormal neural networks, the experimental results demonstrate the superiority of the proposed algorithm.

STRENGTHS:
1. An algorithm was introduced to ensure that the condition number of weight matrices remains close to $1$ during the training process.

2. Comprehensive experimental results were provided in the main paper.

WEAKNESSES:
1. Some terminologies are not defined.

2. While the theoretical analysis provides some insights, it may not fully capture the advantages of the algorithm.

QUESTIONS:
1. Some terminologies are not pre-defined. For instance, concepts like the Stiefel manifold and tangent space may be unfamiliar to readers who have not previously explored low-dimensional spaces. Moreover, it is crucial to establish definitions for significant notations beforehand.

2. Theorem 1 establishes that the estimated matrices are closely aligned with the noisy target matrices. Nevertheless, in practical implementations, our main concern lies in the error between the estimated output and the target output. While it is possible to prove that an individual estimated weight matrix is close to its target counterpart, it does not directly guarantee that the final error won't exponentially grow as a result of multiple products and layers. Therefore, it might be worthwhile to explore how the significance of this theorem can be introduced in light of addressing the overall error in the system, considering the cumulative effects of multiple products and layers.

3. There is a missing $.$ in Eqn.(1).

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

comments: SUMMARY:
Some work has used low-rank matrix factorizations in order to reduce training and inference costs. However, the authors demonstrate that these low-rank models are actually more brittle, i.e., susceptible to adversarial perturbations -- they tend to have large/exploding singular values. Consequently, the authors propose a technique to train low-rank networks while constraining the condition number, alleviating this problem.

STRENGTHS:
- The paper triangulates a well-defined problem and proposes an effective solution -- the paper was generally quite clear and well-motivated
- Demonstrates the method works in practical experience and also via an approximation theorem
- Investigation of exploding singular values in low-rank networks
- Comparisons for a variety of compression ratios / perturbation magnitudes
- Good explanation of background and related work

WEAKNESSES:
- Would be good to substantiate "improving the network robustness without affecting training nor inference costs" -- it's not obvious this is true to me since the method involves, e.g., projection onto the Stiefel manifold
- A conclusion would be nice -- I think you could save space by removing the definition of robust accuracy, which readers probably know

QUESTIONS:
This is minor, but why did you only produce perturbations with FGSM as opposed to, e.g., PGD? I'd be curious how your compressed models withstand somewhat more powerful adversaries.

LIMITATIONS:
The authors have adequately addressed limitations, e.g., noting that they introduce a new hyperparameter that may have to be tuned.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA brief summary of the paper is as follows:\n- Motivation: Make deep neural networks more efficient while maintaining robustness to adversarial attacks.\n- The authors propose a combined low-rank and approximate orthonormalisation constraint to help simultaneous model conditioning and compression during training.\n- The results show that both accuracy and adversarial robustness remain high.\n\nIt is well written and tackles an important problem, however it is missing references and comparisons to key existing works.\n\nSTRENGTHS:\nThe main strength of this paper lies in the writing and the novelty of the approach. Optimizing the low-rank decompositions alongside an orthonormal constraint is a very interesting approach that deserves an in-depth study. Preventing explosion of singular values is important and an interesting research direction that will be of value beyond low-rank neural networks and adversarial attacks. After addressing the weaknesses of this paper, the strengths can make this paper into a very complete piece of work.\n\nWEAKNESSES:\n- Paper does not mention previous studies that approach this same problem: adversarial robustness with low-rank constraints. [1] introduces tensor dropout, a method of increasing robustness (including robustness to adversarial attacks) using a stochastic dropout mechanism in the learned weight decompositions. It would be good to get a comparison to that baseline or at the very least a discussion of the similarities and differences between the approaches. \n- More discussion of the results and method would be good. There a number of questions that arise with this work that remain unanswered. Even though answering all of them might be out of scope for a single paper, it would be worth having the authors discuss more aspects of the approach they introduce here. I refer to my list of questions below, all of which would be interesting discussion points where the pros and cons of different decisions can be weighted.\n- The paper is missing a well-defined conclusion. That means the paper ends abruptly without comment on the impact of this work and what the authors see as possible next steps.\n\n[1] Kolbeinsson, A., Kossaifi, J., Panagakis, Y., Bulat, A., Anandkumar, A., Tzoulaki, I., & Matthews, P. M. (2021). Tensor dropout for robust learning. IEEE Journal of Selected Topics in Signal Processing, 15(3), 630-640.\n\nQUESTIONS:\nThese are questions that I have for the authors but I highly recommend they consider expanding the discussion and limitations part of their paper. They could touch on these points.\n- In the end of the abstract it is stated that \"This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.\" My question is: is it possible to check if a highly performing subnetwork exists, and does the orthonormal constraint affect the group of available subnetworks?\n- Related, how is the r parameter selected? Any suggestions for practice?\n- “Works only for wide enough networks” How wide is enough? \n- What is the added computational complexity of the new optimization? What is the time/memory cost relationship between the low-rank and the full network?\n- Why are VGG16 tested instead of the more common Resnets?\n- What attack is used for the adversarial experiments?\n- How sensitive is the method to choice of r?\n\nLIMITATIONS:\nThe paper has a brief discussion on the limitations but this should be expanded to greatly improve the paper. Getting more candid explanations and thoughts on the limitations from the authors themselves would be welcomed. Possible discussion points would be the  sensitivity to parameter r (related to compression ratio). This is a trade off between performance and computation and can be different for different applications, however it is important to know the scope of available r values. Another very interesting limitation is the additional time cost of this method, specifically in relation to the additional optimization steps that need to be done. What is the relative added computation of this? A discussion of what conditions are needed for this optimization is of value would be very welcome.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}