Robust low-rank training via approximate orthonormal constraints
Authors: Dayana Savostianova
Emanuele Zangrando
Gianluca Ceruti
Francesco Tudisco
Venue: NeurIPS 2023 poster (2023)
Keywords: low-rank neural networks
Stiefel manifold
orthogonal neural networks
pruning
adversarial robustness
neural network condition number
neural network singular values
TL;DR: We observe that low-rank layers may negatively affect network stability and we propose an algorithm that mitigates this phenomenon in a computationally affordable way, while maintaining the low-rank structure.

Abstract: With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy,  we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while  simultaneously enforcing approximate orthonormal constraints. The resulting model 
reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.

Robust low-rank training via approximate orthonormal constraints
Dayana Savostianova Gran Sasso Science Institute 67 100 L' Aquila (Italy) dayana. wavost aavoostiaoova@ges...t
Emanuele Zangrando
Gran Sasso Science Institute 67 100 L' A'quulla (Italy) emanuele zangrando@gesi. it
Gianluca Ceruti University of Innsbruck 6020 Innsbruck (Austria)
Francesco Tudisco
Gran Sasso Science Institute
67 67100 L' Aquila (Italy) francesco tudisco@gesi, it
gianluca. verutiiuubbe ac at
Abstract
With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, argue that this loss of robustness is due the exploding singular values of the low-rank weight matrices. Thus, we introduce robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultancously enforcing approximate orthonormal constraints. The resulting model reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness. without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-ap cell-approximates the ideal full model, provided highly performing low-rank sub-network exists.
Introduction
Deep learning and neural networks have achieved great success in variety of applications computer vision, signal processing, and scientific computing, name few. However, their robustness with respect to perturbations of the input data may considerably impact security and trustworthiness and poses major drawback to their real-world application. Moreover, the memory and computational requirements for both training and inferring phases render them impractical in application settings with limited resources. While broad literature on pruning methods and adversarial robustness has been developed to address these two issues in isolation, much less has been done to design neural networks that are both mergy--svingg and robust. Actually, in many approaches the two problems seem to compete against each other as most adversarial robustness caprovvng-techniqqee require even larger networks [36 thus more expensive training phases 13 23
79 or computationally more demanding loss functions, and
66
The limited work available so far on robust pruned networks mostly focused on reducing memory and computational costs of the inference phase, while retaining adversarial robustness [20] 28 42 76.. However, the inference phase amounts only very limited fraction of the of the
56
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

whole deep learning pipeline, which is instead largely dominated by the training phase. Reducing both inference and training costs a challenging but desirable goal, especially in view of a more accessible AT and its effective use on limited-resource and limited-connectivtty devices such as drones or satellites.
Some of the most effective techniques for the reduction of training costs so far have been based on low-rank weights parametrizations [29] 55 71]. These methods exploit the intrinsic low-rank structure of parameter matrices and large data matrices in general 17
69]. Thus, assuming a low-rank structure for the neural network's 's weights W USV! the resulting training procedures only use the small individual factors U, S. V. This results in training cost that scales linearly with the number of neurons, as opposed a quadratic scaling required by training full-rank weights. Despite significantly reducing training parameters, these methods achieve accuracy comparable with the original full networks. However, their robustness with respect to adversarial perturbations has been largely unexplored so far.
Contributions
In this paper, we observe that the adversarial robustness of low-rank networks may actually deteriorate with respect to the full baseline. By modeling the robustness of the network in terms of the neural network's 's condition number, we argue that this loss of robustness is due to the exploding condition number of the low-rank weight matrices, whose singular values grow very large in order to match the baseline accuracy and to compensate for the lack of parameters. Thus, to mitigate this growing instability, we design an algorithm that trains the network using only the low-rank factors U, V while simultaneously ensuring the condition number of the network remains small. To this end, we interpret the loss optimization problem as a continuous time gradient flow and use techniques from derive three separate projected gradient flows for U, S, V. individually, which ensure the condition number of the network remains bounded to desired tolerance 1 + T, throughout the epochs. For fixed small constant 0, this is done by bounding the singular values of the small rank matrices within a narrow band [s - E, + around a value s, chosen to best approximate the original singular values.
geometric integration theory manifolds 22] 31 55 70
We provide several experimental evaluations on different architectures and datasets, where the robust low-rank networks are compared against variety of baselines. The results show that the proposed technique allows us to compute from scratch low-rank weights with bounded singular values, significantly reducing the memory demand and computational cost of training while at the same time retaining or improving both the accuracy and the robust accuracy of the original model. On top of the experimental evidence, we provide key approximation theorem that shows that if high-performing low-rank network with bounded singular values exists, then our algorithm computes it up first-order approximation error.
This paper focuses on feed-forwer neural networks. However, our techniques and analysis apply
straightforwardly to convolutional filters reshaped in matrix form, as done in c.g. (29 [55] 77]. Other
ways exist to promote orthogonality of convolutional filters, e.g. (6) 65
which we do not
consider in this work.
Related work
Neural networks' robustness against adversarial perturbations has been extensively studied in the machine learning community. It is well-known that the adversarial robustness of a neural network see also Section
closely related to its Lipschitz continuity 13 23
62
68
3 Accordingly,
training neural networks with bounded Lipschitz constant is a widely employed strategy to address
the problem. A variety of works studied Lipschitz architectures (37 60, 655 68), and number
of certified robustness guarantees have been proposed [23 51 61. While scaling each layer to impose I-Lipschitz constraints is possibility, this approach may lead to vanishing gradients and it is known that a more effective way to reduce the Lipschitz constant and increase robustness is obtained by promoting orthogonality on each layer [3] On top of robustness, small Lipschitz constants and orthogonal layers are known to lead to improved generalization bounds [ 45 and more interpretable gradients [67]. Orthogonality was also shown to improve signal propagation in (very) deep networks [52 74

---

A variety of methods to integrate orthogonal constraints in deep neural networks have been developed over the years. Notable example approaches include methods based on regularization and landing
[ cheap parametrizations of the orthogonal group 6
38
39
48 50 Riemannian and projected
gradient descent schemes 2
In parallel to the development of methods to promote orthogonality, an active line of research has grown develop effective training strategies enforce low-rank weights. Unlike sparsity- promoting pruning strategies that primarily aim at reducing the parameters required for inference low-rank neural network models are designed to train directly on the low- parametric manifold of low-rank matrices and are particularly effective to reduce the number of parameters required by both inference and training phases. Similar to orthogonal training, methods for low-rank training include methods based on regularization (26 29], as well as methods based on efficient parametrizations of the low-rank manifold using the SVD, randomized tensor dropout or the
8
22
30
polar decomposition 32 71 75, and Riemannian optimization-bat -based training models 55 57
By combining low-rank training with approximate orthogonal constraints, in this work we propose a strategy that simultaneously enforces robustness while only requiring a reduced percentage of the network's 's parameters during training. The method is based on a gradient flow differential formulation of the training problem, and the use of geometric integration theory to derive the governing equations of the low-rank factors. With this formulation, we are able to reduce the sensitivity of the network during training at almost no cost, yielding well-conditioned low-rank neural networks. Our experimental findings are supported by an approximation theorem that shows that, if the ideal full network can be approximated by a low-rank one, then our method computes good approximation. This is well-aligned with recent work that shows the existence of high-pen highpperfrmming low-rank nets in e.g. deep linear models (7 17 25 49]. Moreover, as orthogonality helps in training really deep networks, low-rank orthogonal models may be used to mitigate the effect of increased effective depth when training low-rank networks (54).
3 The condition number of a neural network
The adversarial robustness of a neural network model f can be measured by the worst-case sensitivity of f with respect to small perturbations of the input data In an absolute sense, this boils down to measuring the best global and local Lipschitz constant of f with respect to suitable distances, as discussed in a variety of papers [13 13 23 62]. However, as the model and the data may assume arbitrary large and arbitrary small values in general, a relative measure of the sensitivity of f may be more informative. In other words, if we assume perturbation ō of small size compared to x, we would like to quantify the largest relative change in f(x + 5), as compared to fxx). This is well-known problem of conditioning, as we review next, and naturally leads to the concept of condition number of a neural network.
In the linear setting, the condition number of matrix is widely adopted relative measure of the worst-case sensitivity of linear problems with respect to noise in the data. For a matrix A and the matrix operator norm ||A|| sup=200 || the condition number of A is defined as (A) |||||||++||| where A+ denotes the pseudo-inverse of A. Note that it is immediate to verify that (A) 2 1. Now, if for example u and Uh are the solutions to the linear system Au = b, when A and b are exact data or when they are perturbed with noise ò & of relative norm || < and [|6% [66]l/[[]]] < E, respectively, then the following relative error bound holds
||u
uell [
S cAn
Thus, small perturbations in the data A, b imply small alterations in the solution if and only if A is well conditioned, i.e. (o)d(A) is close to one.
As in the linear case, it is possible to define the concept of condition number for general functions f. Let us start by defining the relative error ratio of a function f : Rdd - R. in the point a::
24 53
R(f. x; [ + 8) - f(x)) /
||6|| ||>||
[f(())]
In order to take into account the worst-case scenario, the local condition number of ʃ at x is defined by taking the sup of (I over all perturbations of relative size E, i.c such that ||o]| < E ||a]]] in the
3

---

limit of small E. Namely, cond(f:2) = li {[] R(f, z;; 6). This quantity is local measure of the "infinitesimal" conditioning of f around the point x. In fact, a direct computation reveals that
fl + - X co)
(2)
[[f(x))]
as long as ||+]| S S ||r]]] Thus, cond(f;; (xxx) provides a form of relative local Lipschitz constant for which in particular shows that, if ||0]|/||=]|| is smaller than cond ;z>=², we expect limited change in f when æ is perturbed with 6. A similar conclusion is obtained using an absolute local Lipschitz constant in e.g. [23]. Similarly to the absolute case, global relative Lipschitz constant can be obtained by looking at the worst-case over ε, setting cond(f)) = suprex cond(f; Clearly, the same bound (2) holds for cond(f). Note that this effectively generalizes the linear case, as when f(x) = Ax we have cond(f) = cond = cond(A).
When f is a neural network, cond(f) is a function of the network's ' weights and robustness may be enforced by reducing cond(f) while training. In fact, cond(/) is the relative equivalent of the network's & Lipschitz constant and thus standard Lipschitz-based robustness certificates [23 40 68 can be recast terms of cf") However, for general functions / and general norms || |l. cond may be (very) expensive to compute, it may be non-differentiable, and cond(f) > 1 can hold [24]. Fortunately, for weed-ro neural networks, holds (proof and additional details moved to Appendix B in the supplementary material)
Proposition 1. Let X be the input space and let f(x) = 2L+1 be a network with L linear layers 2+11 (WW2). i = l,...,L L. Then,
L.
L
cond(S) = sup cond(Ji =EX\(0)
II
sup cond( ox5;)) mEX)(o)
(II
cond(W.))
i=1
1=1
In particular, for typical X; and typical choices of os including o; E {leakyReLU, sigmoid, tanh, hardtanh, softplus, siLUJ, we have
suPzez cond(o;;)) .) SC < +00
for a positive constant C> that depends only on the activation function di-
Note that for entrywise nonlinearities a, the condition number cond(o; a:) can be computed straight- forwardly. In fact, when a is Lipschitz, the problem can be reduced to a one-dimensional function, and it follows directly from its definition that (see also (64]
cond(f; = sup
MM |x| (x)]
x PER R
where (a(x)) denotes Clarke's generalized gradient [4] of at the point Thus, for example, a is LeakyRelu with slope a, we have cond(ar = 1; a is the logistic sigmoid (1 +1++ e-3)-1 and the feature space X; is such that X; W W, then if |2-1| S C--1 entry-wise, we have lxy S maX W lev == c and cond(o) S sup |ale *(1 (1+e (1 S maxx(c,1/e))
From Proposition I we see that when f is feed-forward network, to reduce the condition number of f is enough to reduce the conditioning of all its weights. When ||- = || 1l2 the Euclidean L norm, we have cond((W), = Smax (Wn) /mii//iiii (W), the ratio between the largest and the smallest singular value of W. This implies that orthogonal weight matrices, for example, are optimally conditioned with respect to the 1 metric. Thus, a notable and well-known consequence of Proposition | is that imposing orthogonality constraints on W improves the robustness of the network [33
50
While orthogonal constraints are widely studied in the literature, orthogonal matrices are not the only optimally conditioned ones. In fact, cond2(W) = 1 for any W with constant singular values. In the next section, we will use this observation to design a low-rank and low-cost algorithm that trains well-conditioned networks by ensuring condg (W) S + r, for all layers W and desired tolerance T > 0.
Robust low-rank training
4.1 Instability of low-rank networks
Low-rank methods are popular strategies to reduce the memory storage and the computational cost of both training and inference phases of deep learning models [29 55 71. Leveraging the intrinsic

---

bywes condaton numbers
Bascline
byyirs conditinn numbers
Low rank gradient flow
layers tonddttion mumbers
DLRT
cenditon numbers
Layer decomposition
1.0
30"
10"
201
10-
10"
10
100
100%
10
10
10a
10"
10"
10
10
10
10
2000 4000 5000 8000
2000 4000 600D 8000
2000 4000 6000 8000
200
600 optimmization stop
400
optimization stop
optimization stop
optimization stop
Figure 1: Evolution of layers' condition numbers during training for LeNet5 on MNIST. From left to right: standard full-rank baseline model; [7T] vanilla low-rank training: [55] dynamical low-rank training based on gradient flow; [75] low-rank training through regularization. All low-rank training strategies are set to 80% compression ratio (percentage of removed parameters with respect to the full baseline model).
low-rank structure of parameter matrices 7 weight matrices parametrized as W USV! for "tall and skinny" matrices U, V with columns, and a small r x r matrix S. Training low-rank weight matrices has proven to effectively reduce training parameters while retaining performance comparable those of the full model. However, while a variety of contributions have analyzed and refined low-rank methods to match the full model's accuracy, the robust accuracy of low-rank models has been partially overlooked in the literature.
17
49 59], these methods train subnetwork with
Here we observe that reducing the rank of the layer may actually deteriorate the network's robustness. We argue that this phenomenon is imputable to the exploding condition number of the network. In Figure [] we plot the evolution of the condition number cond2 for the four internal layers of LeNet5 during training using different low-rank training strategies and compare them with the full model. While the condition number the full model grows moderately with the iteration count, the condition number of low-rank layers blows up drastically. This singular value instability leads to poor robustness performance of the methods, as observed in the experimental evaluation of Section
In the following, we design a low-rank training model that allows imposing simple yet effective training constraints, bounding the condition number of the trained network to desired tolerance 1 + r, and improving the network robustness without affecting training nor inference costs.
4.2 Low-rank gradient flow with bounded singular values
Let W E R=xm be the weight matrix of linear layer within f. For integer (m, let M, = { rank(W) = r be the manifold of rank-r matrices which we parametrize as
M, = {USVT U E R V E Rm with orthonormal columns, S E IR"** invertible].
Obviously, the singular values of W = USVT E _M, coincide the singular values of S. For s, E such that 0 < s. define (E) as the set of matrices with singular values in the interval [s - E, s + e]. Note that >, (0) is a Riemannian manifold obtained essentially by an s scaling of the standard Stiefel manifold (the manifold of matrices with orthonormal columns) and any A E (0) is optimally 1. Thus, can be interpreted as approximation parameter that controls how close (E) ) the "optimal" manifold (0). To enhance the network robustness, in the following we will constrain the parameter weight matrix S to (s).). With this constraint, we get condo (W)= S (s -) + 8) 1 T, with T 2(s - a) -Le, so that the tolerance T on the network's 's conditioning can be tuned by suitably choosing the approximation parameter 8.
conditioned, i.e. condy(A)
Given the loss function L, we are interested in the constrained optimization problem
min L
s.t.
W = USV M, and S E,(s), for all layers W.
To approach B), we use standard arguments from geometric integration theory 31 70 to design a training scheme that updates only the factors U, S, V and the gradient of L with respect to U, V. without ever forming the full weights nor the full gradients. To this end, following (55), we first recast the optimization of L with respect to each layer W continuous gradient flow
(() - V ((W),),
(4)
where "dot" denotes the time derivative and where we write L a function of W only, for brevity. Along the solution of the differential equation above, the loss decreases and a stationary point is
5

---

Algorithm 1: Pscudocode of robust well-Conditioned Low-Rank (CondLR training scheme
Input: Chosen compression rate, i.e. for each layer W choose rank r; Initial layers' weights parametrized as W = USVT, with S ~ P x
Second singular value moment of S, 8 = VL (r
Conditioning tolerance
for each iteration and each layer do (each block in parallel)
2
f one optimization step with gradient G1 and initial point U project U' Stiefel manifold with orthonormal columns
V one optimization step with gradient G2 and initial point V
V project V on Stiefel manifold with r orthonormal columns
S t one optimization step with gradient G3 and initial point S
(x5) squareroot of second moment of the singular values of S (2 T)
+ project onto (e)
approached as - oo. Now, if we assume W E Mr, then W E Tw.M,, the tangent space of M, at the point W. Thus, ensure the whole trajectory W(I) E My, we can consider the projected (w(() Vw (tW(,)), where Pw denotes the orthogonal projection (in the
gradient flow (tt) ambient space of matrices) onto TwM,, Next, we notice that the projection Pw V can be defined by imposing orthogonality with respect to any point Y Tw M,, namely
(Pw W VwC,Y) 0 for all Y E M,
where (. -) is the Frobenius inner product. As discussed in 3155 the above equations combined with the well-known representation of Mr yield a system three gradient flow equations for individual factors
= - Gi) Gi(U) = PJ V ((USVV))SSST
= GV(V), G2(V) = P. Vr
(5)
S Gs((), Gs(S)= V.S(USV))
where Pt = (I UUT) and P -1--V VVT) are the projection operators onto the space orthogonal the span of and V. respectively.
Based on the system of gradient flows above, we propose a training scheme that at each iteration and for each layer parametrized by the tuple {U, S, V) proceeds follows:
1. update U and V by numerically integrating the gradient flows ú (U) and V' -V((V) project the resulting U, V onto the Stiefel manifold of matrices with r orthonormal columns
update the r x r weight S by integrating Š = (S)
4. for fixed robustness tolerance т, project the computed S onto Ex),). choosing and that
s is the best constant approximation to S S, i.e. s = argmin, | 5 a²1]|r
is such that the condy of the projection of S does not exceed 1 +
Note that the coefficients 8, e at point can be obtained explicitly by setting 8 = &
the second moment the singular values sy of S, and 7s/(2 + 7). Note also that, in the differential equations for U, S. V in (5). the four steps above can be implemented in parallel for each of the three variables. The detailed pseudocode of the training scheme presented in Algorithm We conclude with several remarks about its implementation.
Remarks, implementation details, and limitations
Each step of Algorithm requires three optimization steps at lines implemented using standard first-order optimizers such as SGD with momentum or ADAM. Standard techniques can used project onto the Stiefel manifold lines $ and 5 Algorithm see e.g. 4. 70). Here, the QR decomposition. As for the projection onto (e at line g
These steps can be

---

compute the SVD of the small factor S and set to + e or 8 E the singular values that fall outside the interval [s E, s E]. Note that, when T i.. when we require perfect conditioning for the layer weight W USVT, then the SVD of S can be replaced by QR step or any other Stiefel manifold projection. Indeed, we can equivalently set v /ttrac(SST)// and then project onto (0) by rescaling by factor s the projection of S onto the Stiefel manifold. In this case, the system (5) further simplifies, as we can replace (SST) )-1 and -)= with the scalar 1/s2.
Overall, the compressed low-rank network has r(n + m r) parameters per each layer, where n and m are the number of input and output neurons. Thus, choosing r so that 1 1-r(n+ m +))) = a can yield desired compression rate 0 < on the number of network parameters, i.e. the number of parameters one eliminates with respect to the full baseline. For example, in our experiments we 0.8.
will choose that = 0.5
Computational complexity. Each pass of Alg Dis iss done against a batch Thatch- In order to obtain minimal computational costs for each step in the algorithm, we evaluate USV Thatch sequentially: first = V Thatch, then u Su, and finally Uu. Assuming the size of the batch is negligible with respect to n and m, the cost of these steps is (rm)), (r)), (rn)) respectively. Adding the bias term and evaluating the activation function requires (n) operations. Hence, overall we have cost per layer of Or((n + m + 1)). Taping the forward evaluation to compute the gradient with respect to S, V does not affect the asymptotic costs. The QR decompositions used for U and V require (rr2n) and O(r²m) operations respectively, O(r2(n n + m)) overall. Finally, computing the SVD in the projection step for S requires a worst-case cost of O Hence the overall cost per layer is (rr(1 + (n + m) + ) as opposed to the dense network training, which requires (nm) operations per layer. If r < n, m then the low-rank method is cheaper than the full baseline. For example, if m, this happens provided < vn.
Limitations. As the rank parameter r has to be chosen a-priori for each layer of the network, limitation of the proposed approach is the potential need for fine-tuning such parameter, even though the proposed analysis in Table [ shows competitive performance for both 50% and 80% compression rates. Also, if the layer size X m is not large enough, the compression ratio 1 - r(n + m + (rem might be limited. Thus the method works well only for wide-enough networks (n, >> r, that >> 0). Finally, standard way to obtain better adversarial performance would be to combine the proposed conditioning- based robustness with adversarial training strategies [66 63 73. However, the cost of producing adversarial examples during training is not negligible, especially when based on multi-step attacks, and thus the way to incorporate adversarial training without affecting the benefits obtained with low-rank compression is not straightforward.
4.3 Approximation guarantees
Optimization methods over the manifold of low-rank matrices are well-known be affected by the stiff intrinsic geometry of the constraint manifold which has very high curvature around points where W E M, is almost singular 31 70) This implies that even very small changes in W may yield very different tangent spaces, and thus different training trajectories, as shown by the result below:
Lemma 1 (Curvature bound, Lemma 4.2 (3I).. For W E M, let Smin (W) > 0 be its smallest singular value. For any W' E M, arbitrarily close to W and any matrix B, it holds
P - Pw S C' smin smmnnm) felllwWWWll W'
where C > O depends only on B.
In our gradient flow terminology, this phenomenon is shown by the presence of the matrix inversion in (5). While this is often issue that may dramatically affect the performance of low-rank optimizers (see also Section 55). the proposed regularization step that enforces bounded singular values allows us to move along paths that avoid stiffness points. Using this observation, here we provide a bound on the quality of the low-rank neural network computed via Algorithm|] T provided there exists an optimal trajectory leading to approximately low-rank network. We emphasize that this assumption is well-aligned with recent work showing the existence of nigh-performing low-rank nets in e.g. deep linear models
17 25 49
Assume the training is performed via gradient descent with learning rate À > 0, and let W(I) be the full gradient flow Further, assume that for t [0, A] and a given E > 0, for each layer there exists
7

---

E(t) and (t) E M, ni, (E) such that
W(t) W(t) + E(t),
where 8 is the second moment of the singular values of (1) and E(() is perturbation that has bounded variation in time, namely [[(()]] < In other words, we assume there exists training trajectory that leads to an approximately low-rank weight matrix (t) with almost constant singular values. Because the value is bounded by construction, the parameter-dependen matrix (()) possesses singular values exhibiting moderate lower bound, Thus, (()) far from the stiffness region of (5) and we obtain the following bound, based on [3 (proof moved to Appendix Cin the supplementary material)
Theorem 1. Let V be solution to (5) computed with l steps of Algorithm !
Assume that The low-rank initialization Vo coincides with the low-rank approximation W (O)).
The norm of the full gradient is bounded, i.e., VVV (t))l S p.
.
The learning rate is bounded as X S 4v/unjj
Then, assuming no numerical errors, the following error bound holds
V ((X)] 3Xm.
Note that if f is smooth enough (e.g. Lipschitz) then by the previous theorem we directly obtain an equivalent bound for the functional distance (VV(U) f(W())))
5 Experiments
We illustrate the performance of Algorithm|] T on a variety of test cases. All the experiments can be reproduced with the code in PyTorch available at httpp::// tigu........ Wamm/CCMPLEELb/CCma In order to assess the combined compression and robustness performance of the proposed method, we compare it against both full and low-rank baselines.
For all models, we compute natural accuracy and robust accuracy. Let {z y=)).. the set of test images and the corresponding labels and let f be the neural network model, with output f(x) on the input æ. We quantify the test set robust accuracy as:
robust_acce
Ei= 1 (J + &,))
where 5 =6+) (6+);=1....., are the adversarial perturbation associated to each sample. Notice that, in the unperturbed case with ||5%|| = 0, the definition of robust accuracy exactly coincides with the definition of test accuracy. In our experiments, adversarial perturbations are produced by both the fast gradient sign method (FGSM) [19] and the projected gradient descent attack (PGD) [47] with || e, and controls perturbation strength. As images in our set-up have input entries in [0, 1), the perturbed input is then clamped to that interval. Note that, for the same reason, the value of controls in our case the relative size of the perturbation.
We consider MNIST, CIFARIO, 10, and CIFAR100 33 datasets for evaluation purposes. The first contains 60,000 training images, the second one contains 50,000 training images, while the third one contains 50,000 training images. All the datasets have 10,000 test images, first two have 10 classes and last one has 100 classes. No data- aatiaaaammmntttion is performed.
Datasets.
Models. We use LeNet5 35 for MNIST dataset, VGG16 (58) and WideResnet (WRN16-4) 78 for CIFARIO and CIFAR 100. The general architecture for all the used networks is preserved across the models, while the weight-storing structures and optimization frameworks differ.
Methods. Our baseline network is the one done with the standard implementation. Cayley SGD 39] and Projected SGD 3 are Riemannian optimization-base methods that train the network weights over the Stiefel manifold of matrices with orthonormal columns. Thus, both methods ensure cond2(WW) for all layers. The former uses an iterative estimation of the Cayley transform, while the latter uses QR-based projection to retract the Riemannian gradient onto the Stiefel manifold. Both methods have no compression and use full-weight matrices. DLRT, SVD prune, and Vanilla are low-rank methods that ensure compression of the model parameters during training. DLRT 55 is based on low-rank gradient flow model similar to proposed Algorithm [ SVD prune [75
based on regularized loss with a low-rank www-rran-promotti promoting penalty term. This approach was designed to
8

---

compress the ranks of the network after training, but we imposed a fixed compression rate from the beginning for a fair comparison. "Vanilla" denotes the obvious low-rank approach, that parametrizes the layers as W = UV UVT and performs alternate descent steps over U and V. as done in e.g. [29 71
All models are implemented with fixed training compression ratios a = 0.5 and a 0.8, i.e. we only use 50% and 20% of the parameters the full model would use during training, respectively. Finally, we implement CondLR as in Algorithm| for three choices of the conditioning tolerance E {0, 0.1, 0.5}. We also implement a modified version of Algorithm in which S is directly projected onto the Stiefel manifold Σ (0), i.e the parameter & is fixed to one.
Training, Each method and model was trained for 120 epochs of stochastic gradient descent with a minibatch size of 128. We used a learning rate of 0.1 for LeNet5 and 0.05 for VGG16 with momentum 0.3 and 0.45, respectively, and a learning rate scheduler with factor = 0.3 at 70 and 100 epochs.
Results. For comparison, measure robust accuracy for all chosen combinations of datasets, models, and methods using FGSM and PGD attacks with relative perturbation budget c E (0.01, 0.02, 0.030.04, 0.05, 0.06} for MNIST, e E {0.001, 0.002, 0.003, 0.004, 0.005, 0.006) for CIFARIO VGGI6, e E {0.0003, 0.0006, 0.001, 0.0013, 0.00163 for CIFARIO and CIFAR 100 with WRN16-4.
The results are summarized in Table I for LeNet5 MNIST and VGG16 CIFARIO with FGSM and subset of the perturbation budget, and in Table 2 for WRN16-4 CIFARIO with both FGSM and PGD. The additional results are presented in Tables 3 and are moved to Appendix A In the tables, we highlight (in gray) the best-performing method for each range of compression rates a E (0%, 50%, 80%). and the best method overall (in bold). The experiments show that the proposed method meaningfully conserves the accuracy compared to the baseline for all datasets; moreover, the robustness is improved from the baseline for 50% compression rate and, additionally, in the case of MNIST for 80% compression rate. Compared orthogonal non-compressed methods, CondLR with 50% compression rate also outperforms Cayley SGD and Projected SGD. At the same time CondLR is compressed by design, so we reduce memory costs alongside gaining robustness. As anticipated in Section 4.1] low-rank methods without regularization deteriorate the condition number and thus the robustness of the model. This is consistent with the results of Table []] where we observe that compression without regularization does not work well in terms of robust accuracy, in particular DLRT, Vanilla and SVD prune exhibit a drop the performance in comparison with baseline and, consequently, CondLR Specifically, per each fixed compression ratio a, our method exhibits the highest robustness; moreover, the robustness of CondLR with a = 0.8 is higher than the robustness of any other method with an even lower compression ratio, = 0.5.
As further experimental evaluation, we analyze the robustness of low-rank training models with respect to small singular values. As shown by Theorem T CondLR is not affected by the steep curvature of the low-rank manifold Mp. i.c. accuracy and convergence rate of CondLR do not deteriorate when the conditioning of the weight matrices explodes. In contrast, small singular values may significantly affect alternative low-rank training models. In Figure 2 we show the behavior of loss, accuracy, and condition number as functions of the iteration steps, when the network is initialized with ill-conditioned layer matrices. Precisely, similar to what is done in [29], we randomly sample Gaussian initial weights, compute their SVD to define the initial U and V factors, and then force the singular values of the initial S factor to decay exponentially. We observe that CondLR and DLRT (which is also based on low-rank gradient flow formulation) are the most robust, while the performance of the other methods deteriorates dramatically. This confirms that, as also observed in 29 (55, most low-rank approaches require specific fine-tuned choices of the initialization, while the proposed robust low-rank training model ensures solid performance independent of the initialization.
6 Conclusions
In recent years, extensive effort has been put into (a) studying different forms of implicit bias in deep learning, including bias towards low-rank parameter weights 7 and (b) designing modified training techniques that take advantage of the implicit low-rank structure to reduce training costs and
compress the network 22 55 71 75]. Based on the notion of condition number of a network, as
a form of local Lipschitz constant, in this work we observe that training directly on the manifold of matrices with a fixed rank may lead to instabilities due to bad conditioning, going against the intuition that low-rank networks may be more robust to adversarial attacks. Thus, propose an

---

algorithm able to mitigate this phenomenon in computationally affordable way, allowing us to train directly on the manifold of matrices of fixed rank while controlling the condition number.
Table I: Method comparison results
LeNet5 MNIST
VGG16 Cifar10
(%)
Rel. perturbation
0.0
0.02
0.04
0.06
0.0
0.002
0.004
0.006
Baseline
0.9872
0.9793
0,9655
0.9407
0.9104
0.752
0,5822
0.4592
Cayley SGD
0.9874
0.9804
0.9688
0.9486
0.8962
0.7446
0.5816
0,4529
Projected SGD
0.9878
0.9807
0.968
0.9476
0.897
0.7455
0.5882
0.4574
T 0
0.9883
0.9825
0.9732
0,958
0,9099
0..7466
0.5711
0.4292
50
CondLKK R
0.1
0.9877
0.9828
0.9763
0.9677
0,9093
0.7411
0.5985
0.4878
50
T = 0.5
0.9867
0.9802
0.9724
0.9598
0.8997
0.7225
0.6019
0.5017
50
stief
0.986
0.9809
0.9721
0.9586
0.9138
0.7322
0.546
0.4054
50
DLRT Vanilla
0,967
0.9573
0.939
0.9078
0.8425
0.599
0.441
0.3691
0.9875
0.9773
0,9603
0.94
0.8997
0.6771
0.4886
0.3849
SVD prune
0,9883
0.9793
0.9639
0.9414
0.8992
0.673
0.4777
0.3698
50
0.0
0,9882
0.9801
0.9676
0.9452
0.9066
0.7263
0.541
0.4
80
CondL R
т 0.1 т == 0.5
0.9877
0.9795
0.966
0,9444
0.9048
0.7123
0.5262
0.4013
0,9858
0.9768
0.9613
0.9342
0.8933
0.6823
0.4854
0.3666
stief
0.9815
0.9729
0.9581
0.9399
0.9067
0.7184
0.5289
0.3861
80
DLRT
0.9649
0.9517
0,9281
0.8865
0.8092
0.5839
0.4178
0.3086
Vanilla
0.9862
0.972
0.9464
0,9194
0.881
0.6424
0.4266
0,299
80
SVD prune
0.9864
0.9737
0.9512
0,9281
0.8799
0.6357
0.4206
0.2927
80
Table 2: WRN16-4 Cifar10
Rel. perturbation
0.0
0.0003
0.0006
0.001
0.0013
0.0016
er (%)
Baseline, FGSM
0.9129
0.885
0.8531
0.8056
0.7667
0.7263
0
T 0.0
0.9254
0.8958
0.865
0,8092
0.7669
0.7236
50
CondLR
T = 0.1
[0.9271
00.8944
0.8612
[0.8117]
(0.7697
0.7223
0.0
0.9106
0.8807
0.844
0.7874
0.7413
0.697
80
0.1
0.9146
0.8833
0.8497
0.8004
0.7568
0.7142
80
Baseline, PGD10
0.9129
0.8898
0.8619
0.8188
0.7828
0.7435
T 0.0
0.9254
0.8997
0.8714
0.8241
0.7791
0.7375
CondLR
0.9271
0.9018
0.8684
0.8233
0.782
0.7374
50
T= 0.0
0.9106
0.8838
0.8512
0,8014
0.7565
0.7064
80
0.1
0.9146
0.8861
0.8582
0.813
0.7705
0.724
80
0.0200
1..
1p30
CondLR, r=
0.0175
0.8
Upper baund condin)
DLRT
0.0150
2030
SVD Prune
0.0125
Accuracy
0.6
Vanilla
0.0100
30:0
0.0075
10a
0.2
8.0025
10a
0.0000
200
400
650
12.0
1450
Optimization step
Optimi zation step
Optimization step
Figure 2: Evolution of loss, accuracy, and II, co,)(W,) for Lenet5 on MNIST dataset, for ill- conditioned initial layers whose singular values are forced decay exponentially with powers of two.
10

---

References
[1] Ablin and G. Peyré. Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference Artificial Intelligence and Statistics, pages 5636-5657, PMLR, 2022.
[2] P.-. Absil, R. Mahony, and R. Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2008.
[3] P.-.. Absil and J. Malick. Projection-lite like retractions on matrix manifolds. SIAM Journal on Optimization, 22(1):135-155 2012.
[4] P.-. Absil and Oseledets. Low-rank retractions: a survey and new results. Computational Optimization and Applications, 62(1):5-29, 2015.
[5] C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. International Conference on Machine Learning, pages 291-301. PMLR, 2019.
[6] M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120-1122 128. PMLR, 2016.
[7] S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. Advances Neural Information Processing Systems, 32, 2019.
[8] A. Ashok, N. Rhinehart, F. Beainy, and K. M. Kitani. N2n learning: Network to network com- pression via policy gradient reinforcement learning. In International Conference on Learning Representations, 2018.
[9] B. Bah, H. Rauhut, U. Terstiege, and M. Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers, 2020.
[10] N. Bansal, X. Chen, and Z. Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
[11] I.. Bartlett, J. Foster, and M. J. Telgarsky. Spectrally-normalizeed margin bounds for neural networks, Advances neural information processing systems, 30, 2017.
[12] G. Ceruti and C. Lubich. An unconventional robust integrator for dynamical low-rank approxi- mation. BIT Numerical Mathematics, 62(1):23 2022.
[13] M. Cisse, Bojanowski. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, pages 854-863, PMLR, 2017.
[14] H. Clarke.
Optimization and nonsmooth analysis. SIAM, 1990.
[15] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smooth- ing. In international conference on machine learning, pages 1310-1320. PMLR, 2019.
[16] J. Ding, T. Bu, Z. Yu, T. Huang, and J. Liu. Snn-rat: Robustness-enhanced spiking neural network through regularized adversarial training. Advances in Neural Information Processing Systems, 35:24780-24793, 2022.
[17] R. Feng, K. Zheng, Y. Huang, D. Zhao, M. Jordan, and Z.-J. Zha. Rank diminishing in deep neural networks. vrXiv:220606607. 2022.
[18] T. Galanti, Z. S. Siegel, A. Gupte, and T. Poggio. Sgd and weight decay provably induce low-rank bias in neural networks, 2023.
[19] J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2015.
[20] S. Gui, H. Wang, H. Yang, C. Yu, Z. Wang, and J. Liu. Model Compression with Adversarial Robustness: A Unified Optimization Framework. 2019.

---

[21] Y. He, J. Lin. Z. Liu, H. Wang, L.-J. Li, and S. Han. AMC: AutoML for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision, pages 784-800, 2018.
[22] Y. He, X. Zhang, and Sun, Channel pruning for accelerating very deep neural networks. In IEEF International Conference on Computer Vision, pages 1389-1397, 2017.
[23] M. Hein and M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. Advances in neural information processing systems, 30, 2017.
[24] D. J. Higham. Condition numbers and their condition numbers. Linear Algebra and its Applications, 214:193-213, 1995.
[25] M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola. The low-rank simplicity bias in deep networks. Transactions Machine Learning Research, 2023.
[26] Y. Idelbayev and M. A. Parreira-Perpinán. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8049-8059, 2020.
[27] K. Jia, D. Tao, Gao, and X. Xu. Improving training of deep neural networks via singular value bounding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4344 4352, 2017.
[28] A. Jordao and H. Pedrini. On the effect of pruning adversarial robustness. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE Computer Society, 2021.
[29] M. Khodak, N. Tenenholtz, L. Mackey, and N. Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations, 2021.
[30] H. Kim, M. U. K. Khan, and C.-M. Kyung. Efficient neural network compression. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12569-12577, 2019.
[31] O. Koch and C. Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434 454, 2007.
[32] A. Kolbeinsson, Kossaifi, Y. Panagakis, Bulat, A. Anandkumar, I. Tzoulaki, and P. M. Matthews. Tensor dropout for robust learning. IEEE Journal of Selected Topics in Signal Processing, 64000, 2021.
[33] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] Langenberg, E. R. Balda, A. Behboodi, and R. Mathar. On the effect of low-rank weights on adversarial robustness of neural networks. arXiv preprint 11Xiv: 2019.
[35] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basse learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233 1998.
[36] H. Lee, S. Han, and J. Lee. Generative adversarial trainer: Defense to adversarial perturbations with gan, 2017.
[37] K. Leino, Z. Wang, and M. Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning, pages 6212 2212-6222. PMLR, 2021.
[38] M. Carcano-Casadee and D. Martinez- Cheap orthogonal constraints in neural networks: simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pages 3794-3803. PMLR, 2019.
[39] Li, I.. Fuxin, and Todorovie, Efficient Riemannian optimization on the Stiefel manifold via the Cayley transform. 2019.
12

---

[40] Q. Li, S. Haque, C. Anil, J. Lucas, R. B. Grosse, and J.-H. Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances neural information processing systems, 32, 2019.
[41] Y. Li, Z. Yang. Y. Wang, and C. Xu. Neural architecture dilation for adversarial robustness. In Advances in Neural Information Processing Systems, 2021.
[42] N. Liao, S. Wang, L. Xiang, N. Ye, S. Shao, and P. Chu. Achieving adversarial robustness via sparsity. Machine Learning, pages 1-27, 2022.
[43] H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representa- tions for efficient architecture search. In International Conference on Learning Representations, 2018.
[44] X. Liu, Y. Li, C. Wu, and C.-J. Hsieh. Adv-BNN: Improved adversarial defense through robust bayesian neural network. In International Conference on Learning Representations, 2019.
[45] M. Long and H. Sedghi. Generalization bounds for deep convolutional neural networks, In International Conference on Learning Representations, 2020.
[46] D. Madaan, J. Shin, and S. Hwang. Adversarial neural pruning with latent vulnerability suppression. In ICML, 2020.
[47] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[48] K. D. Maduranga, K. E. Helfrrch,, and Q. Ye. Complex unitary recurrent neural networks using scaled cayley transform. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 4528-4535, 2019.
[49] C. Martin and M. W. Mahoney. Implicit reffraegulaiizttinn in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021.
[50] E. Massart, Orthogonal regularizers in deep learning: how to handle rectangular matrices? In 2022 26th International Conference on Pattern Recognition (ICPR), pages 1294-1299 1299. IEEE, 2022.
[51] L. Meunier, B. Delattre, A. Araujo, and A. Allauzen. dynamical system perspective for lipschitz neural networks. In International Conference on Machine Learning, pages 15484- 15500. PMLR, 2022.
[52] J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
[53] J. R. Rice. A theory of condition. SIAM Journal on Numerical Analysis, 3(2):287-310, 1966.
[54] D. A. Roberts, Yaida, and B. Hanin. The Principles of Deep Learning Theory. Cambridge University Press, may 2022.
[55] Schotthöfer, E. Zangrando, Kusch, G. Ceruti, and Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processing Systems, 2022.
[56] V. Sehwag, S. Wang, P. Mittal, and S. Jana. Hydra: Pruning adversarially robust neural networks. NeurIPS, 2020.
[57] U. Shalit, D. Weinshall, and G. Chechik. Online learning in the manifold of low-rank matrices. Advances in neural information processing systems, 23, 2010.
[58] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.156,, 2014.
13

---

[59] S. Singh, G. Bachmann. and T. Hofmann. Analytic insights into structure and rank of neural network Hessian maps. In Advances in Neural Information Processing Systems, volume 34, 2021.
[60] S. Singla and S. Feizi. Skew orthogonal convolutions. In International Conference on Machine Learning, pages 9756-9766. PMLR, 2021.
[61] S. Singla, S. Singla, and Feizi. Improved deterministic 12 robustness on cifar-10 and cifar- In International Conference on Learning Representations, 2021.
[62] C. Szegedy, ing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intrigu-
[63] D. Terjék. Adversarial lipschitz regularization. In International Conference on Learning Representations, 2020.
[64] L. N. Trefethen and D. Bau. Numerical Linear Algebra. SIAM, 1997.
[65] A. Trockman and Z. Kolter. Orthogonalizing convolutional layers with the cayley transform.
In International Conference on Learning Representations, 2021.
[66] T. Tsiligkaridis and J. Roberts. On frolk adversarial training. In ICML 2021 Workshop on Adversarial Machine Learning. 2021.
[67] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2018.
[68] Y. Tsuzuku, I. Sato, and M. Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. Advances in neural information processing systems, 31, 2018.
[69] M. Udell and A. Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144-160, 2019.
[70] A. Uschmajew and B. Vandereycken. Geometric methods on low-rank matrix and tensor manifolds. Handbook of variational methods for nonlinear geometric data, pages 261-313, 2020.
[71] H. Wang, S. Agarwal, and D. Papailiopoulos. Pufferfish: communication-effiiiee models at no extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 2021.
[72] X. Wei, Y. Xu, Y. Huang, H. Lv. H. Lan, M. Chen, and X. Tang. Learning extremely lightweight and robust model with differentiable constraints on sparsity and condition number. In European Conference on Computer Vision, pages 690-707. Springer, 2022.
[73] Y.-.. Wu, H.-H. Shuai, Z.-R. Tam, and H.-Y. Chiu. Gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6373-6382, 2021.
[74] L. Xiao, Y. Bahri, J. fohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pages 5393-5402. PMLR, 2018.
[75] H. Yang. M. Tang, W. Wen, F. Yan, D. Hu, A. Li, H. Li, and Y. Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsifica- tion. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2899-2908, 2020.
[76] S. Ye, K. Xu, S. Liu, H. Cheng, J. Lambrechts, H. Zhang, A. Zhou, K. Ma, Y. Wang, and X. Lin. Adversarial robustness vs. model compression, both? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
(77) T. Yu, J. Li, Y. Cai, and P. Li. Constructing orthogonal convolutions in an explicit manner. In International Conference on Learning Representations, 2022.
14

---

[78] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint 100.00666.. 2016.
[79] H. Zhang, Yu, Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482. PMLR, 2019.
15

---

A Additional results
In this section, we report the results on natural and robust accuracy for the broader range of per- turbation budgets for Table [ E {0.01, 0.02, 0.03, 0.04, 0.05, 0.06} } for LeNet5 on MNIST, and e E {0.001, 0.002, 0.003, 0.004, 0.005, 0.006} for VGGI6 on CIFARIO. See Table 3 and Table 4 (standard deviation is shown in gray). Additionally we provide results for Projected Gradient Descent attack with L.2 norm and 10 steps for VGG16 on CIFARIO with perturbation budget {0.1, 0.16, 0.2, 0.23, 0.26, 0.3). See Table 5 The results confirm the same findings as the ones reported in the main paper. Table ploviddss results of comparison between baseline and Algorithm I for WideResnet model with depth 16 and width on CIFAR 100 for both FGSM and PGD attack with Loo norm with perturbation budget c E {0.0003, 0.0006, 0.001, 0.0013, 0.00163.
B Proof of Proposition !
and àb (X, - (Y,11- be continuous mappings between finite-dimensional Banach spaces, and assume cond(w) and co) are well defined. Then the following inequality holds:
Lemma 2. Let Φ (Y, -ly)
cond @ o v) S con cond (+)
Proof. To prove this lemma, we will pass through the definition of R(o o 6, Let recall that it is defined as
R(6 o 10, x. 8) 1 o cxx + 5) - (z))||||=|
|8||x (<))ll
Now, (xx + - (()) 0 for + O, then Río v, 6) O. Since we're we're interested in the supremum, we can restrict to 8 such that (0 + 5) vxz) # O. Moreover, by multiplying and
Table 3: LeNet5 MNIST, FGSM
Rel. perturbation e 0.0
0.001
0.002
0.003
0.004
0.005
0.006
cr (%)
Baseline
0.9872
0.9836
0.9793
0.9735
0.9655
0.9543
0.9407
Rile-01
Sie01
6:
S.eeOl
1.7-00
Cayley SGD
0.9874
0.9843
0,9804
0.9753
0.9688
0.9602
0 0.94866
Projected SGD
0,9878
0.9845
0,9807
0.9753
0.968
0.9594
0.9476
0126-0
0000
1.1e-03
T = 0.0
0.9883
0.9957
0,9825
0.9783
0.9732
0.9661
0.958
50
01
0.
49c-01
9:9e-00
9:
4.8c
6.0c.0
CondLR
0.1
0,9877
0,9854
0.9828
0.98
0.9763
0.972
[0.9677
50
4.
04
84
13
535.04
4.
122
T 0.5
0.9867
0.9831
0.9802
0.9772
0.9724
0.9675
0.9598
50
4:40-04
3:1e:4
6:4-00
5:1e:00
0.
122403
1:2e-00
stief
0.986
0,9838
0,9809
0.9773
0.9721
0.9655
0,9586
50
2.6e-04
2:46004
246-04
288-0c
3.6.04
6.26444
1.4c-04
DLRT
0.967
0.9627
0,9573
0.9499
0.939
0.9253
0,9078
50
6.8c-04
64
7.5-00
1.0-00
6.6e04
L
1.5e-00
vanilla
0.9875
0.983
0,9773
0.97
0.9603
0.9503
0.94
50
5.8c-14
1.26-03
3.4033
4.1600
4.2c415
3
SVD prune
0,9883
0,9841
0,9793
0.9733
0,9639
0,9526
0.9414 2.6
50
9.6c-01
0:
1.le-00
L:4e00
1:2e000
1.5e-03
0.0
0,9882
0.9846
0,9801
0.9743
0.9676
0.958
(0.9552
80
CondLR
0.1
0.9877
7:54444
0.984
300--3
0,9795
0.9736
0.966
0.9564
0.9444
1:2-00
80
04
6.1e.01
Lie-0s
830.4
1.46-03
1:3e.00
0.5
0.9858
0.982
0,9768
0,97
0.9613
0.9487
0.9342
80
6.5c-44
8:26-04
L3
13e-0
35c411
4.0.00
stief
0.9815
0,9779
0,9729
0.9659
0.9581
0.9499
0.9399
80
3.8c004
4.6e-44
1.26-04
3.7e-04
5.7004
456404
1.04-00
DLRT
0.9649
0.9596
0.9517
0,9414
0.9281
0,9108
0.8865
80
7.6e-03
1e
vanilla
0.9862
0.9802
0.972
0.9602
0.9464
0.9331
0.9194
80
9:70-04
13e-08
3.46-1
4.1641
$:2000
SVD prune
0,9 0.9864
0,981 1
0,9737
0.9634
0,9512
0,9392
0.9281
80
8.
L.ee03
2.20-00
3.
4.5c.00
5.1e:03
5.9.-03
16

---

dividing by (1)(l we obtain:
1 (a + +) @ (x)))())222 (
(0 + +6) - (xx)) 1 (xxx)lz ( ||b( + +6) - (x)]|r||==11
||8||x [|<(())]ly
Now, if we define cond(f, x = sup
R(S.a x. 8). we take the supremum both sides on the set
8 Se
{8 x ||0||x = and we can upper bound it with the product of the suprema of the two blocks
(10 (f + 6) - o o (b) -
sup Se
flpow,c x. = 2,,
sup Ss
11/b(x + 6) ( <<)))>>
Now, letting 7 (x - 8) - (6)) we can rewrite the second part the last equation as a sort of
restriction of cond(o, (x), E) to the particular set of perturbation directions of the form 17. Thus, can lower-bound it as:
(1((b()) -(()))))
sup S cond v/, 7 Se
cond ar, e) cond (b,, (x)),e)
By hypothesis, (Ø) and cond(y) (b) are finite, cond( ( $ 2) I cond(o, (x)) cond x)
take the limit 4 0 to get:
Finally, taking the supremum over x on both members of the last equation, we can upper bound with the original condition number without constraint on the directions:
cond o vb) < sup cond(o, (xx)) cond x) I cond())cond(())) co)
and thus conclude.
Table 4: VGG16 Cifar10. FGSM
Rel. perturbation
0.0
0.001
0.002
0.003
0.004
0.005
0.006
(%)
Baseline
0.9104
0.8397
0.752
0.6631
0,5822
0,5154
0.4592
1-04m
216411
30c-0
6.6641
K0p
Cayley SGD
0.8962
0.8262
0,7446
0.6624
0.5816
0,5132
0.4529
1.fe00
136
335.00
4.
0:
4:2e-13
4.
Projected SGD
0.897
0.8271
0.7455
0,661
0.5832
0.5159
0.4574
138c-03
03
3.0e
4.1e-03
5.3c
4.2c
4.5c
0.0
0,9099
008366
0,7456
0.6564
0.5711
0.4955
0,4292
50
7:4e:03
0052
1.5e00
1.8e-02
1.
CondLR
0.1
0.9093
0.8291
0.7411 1
0.6678
0.5985
0,5386
0,4878
50
1.9c003
3.1e-3
6.000
122
13x00
8.4--0
6c
0.5
0.8997
0,8136
0,7225
0.6577
0.6019
0.5485
0.5017
50
10:00
195-03
5:3e400
7.5e-05
6.3000
1.7e-05
00
stief
0.9138
0,8331
0,7322
0.6337
0.546
0.4702
0.4054
50
1.3e003
3.3e-03
S:le-00
05
376-00
4.1e:03
3.9e-03
DLRT
0.8425
0.7179
0.599
0.5065
0.441
0.3979
0.3691
L.6e4ss
6.26-13
8:36-0
1:2e-12
Li6442
1.3c-02
vanilla
0.8997
0.7962
0.6771
0.5708
0.4886
0,4276
0,3849
50
3:2e-03
03
0.
3.3
9.
1.0e-02
1.
SVD prune
0.8992
0.7936
0.673
0.5643
0.4777
0.4161
0.3698 1.3e402
50
S.lea3
1.7008
7.66-03
1.34022
r 0.0
0,9066
0.825
0.7263
0.6289
0.541
0.4633
0.4
80
CondLR
0.1
0,9048
2.20-113
0,8152
1MC-11
0,7123
0,6112
0,5262
0.4574
0.4013
5.0440
80
Li6445
5:4e-00
9.1.0
9.9c-1
k6c-00
T 0.5
0.8933
0.7952
0.6823
0.5748
0.4854
0,4177
0.3666
1.3c4t
14c-0s
4.26-00
6.-03
S:5--08
5,3641
S.3c-00
stief
0.9067
0.8202
0.7184
0.6185
0.5289
0,4515
0.3861
80
4,30004
2.5443
536-08
5.56-8
7:6-48
9:3e-03
DLRT
0,8092
0.6916
0.5839
0.4936
0.4178
0.3577
0.3086
80
19-002
1.5e-0
2.3e
vanilla
0.881
0.7687
0.6424
0.523
0.4266
0.3533
0.299
80
2.42403
8.56-1i
122.00
SVD prune
0,8799
0.7634
0,6357
0,5168
0,4206
0,3474
0,2927
80
3.7e-03
0.
122e-00
Lite
1.2e-02
Liee02
1.
17

---

Now, the proof of Proposition Ceniiitiionoolloow directly by applying the Lemma above recursively.
Proof of Proposition T
By defining (2) = W, we can write the neural network f as
f(x) = (oz o T2 o LL-1 ..... (r)
for nonlinear activation functions o,. Thus, for L = 1, the thesis follows directly from Lemma 2 as
long as (o1) I C' < co, For L > 1, one can unwrap the compositional structure of f from the
left, defining 6(2) = (az o (2) and (xx) = (ØL-1 ..... (z). Then by using Lemma we
have that
cond(f) I cond Φ) cond( I cond(o,.) cond (Tr.) cond(
Now, since & is a network of depth 1, we can proceed inductively to obtain that cond(f) v CII, cond(T;), with C = II cond(a;), and we conclude.
Note that the result above is of interest only if con = supnex cond(o; < co. When o is Lipschitz, using the formula
cond(f:z)
sup v. (2)
| |=||||=||||
Table 5: VGG16 Cifar 10, PGD
Rel. perturbation e
0.0
0.1
0.13
0.16
0.2
0.23
0.26
0.3
cr (%)
Baseline
0.9104
0,6704
0,5799
0.5021
0.4222
0.3788
0.347
0.3188
0
8:1ads
9:0-08
Cayley SGD
0.8962
0.6768
0.5938
0,5109
0,4087
0.3446
0.2921
0.2362
0
8:4c-03
Projected SGD
0.897
0.6764
0.5925
0.5106
0.4086
0.3451
0.2919 7,5040
0.2364
0
L8c-0s
6.4045
5.6e-03
6.8005
9.e-33
T 0.0
0.9099
0.6608
0,562
0.4712
0.3665
0.305
0,2551
0.2078
50
CondLR R
T 0.1
0.9093
7:4e-0%
0.6703
Se-02
0.6208
4e-02
0,5915
2 3.8002
0.5692
2.2-00
0.5596
72200
0.5528
0,5458
5e:02
50
19-03
0.5
0,8997
0,6484
0.6025
0,5775
0,5603
0.5521
0,5475
0,5425
50
1.9g-02
3.0e-02
5e502
stief
0.9138
0.6318
0,5253
0,4266 7.0e-03
0,3118
0.2445
0.192
0,1388
50
1:3-03
7.8c-03
6:4e-03
7:2e-05
vanilla
0,8997
0,5574
0,4511
0.3686
0,2962
0.2628
0,2402
0,2184
50
3.2e-03
122e02
1.Se-02
1.5e02
1.Be-02
2.
2.1e-02
2.2e-02
SVD prune
0,8992
0,5487
0,4385
0.3523
0,2795
0.2438
0,2206
0.1994
50
3.1e-ds
S:4e-03
8.9345
1.5c-02
2.0e-02
2.5e-02
2.6-02
3.2e-02
0.0
0.9066
0.6324
0.5323
004457
0.325
0.2599
0.2096
0. 0.1603
80
CondLR R
0.1
0.9048
0.61
0,5084
0.4219
0.337
8.0e-03
0.2944
1:3g-02
0.2655
1.5eot
0.2397
5e-02
80
6.
9.7640
0.5
0,8933
0,5745
0,4635
0,372
0.283
0.2382
0.2065
0. 0.1784
80
1 03-00
9 9.1e-01
0.76403
00-00
1:3-402
1.-022
1.6-02
1.76-2
stief
0.9067
0.6153
0.5094
0,4082
0,2924
0,2236
0.1716
0.118
80
4.4e044
6.7e-03
9,5e-03
1.e002
9.4c-05
9.1e-03
8.7e-03
7.5e-03
vanilla
0,881
0.514
0.397
0.3012
0.2097
0.1629
0.1315
0.1037
80
2:46-115
1.3e-0
1.5022
1.4e-02
1.3e-02
1:4e-02
1.40-02
1.4e-12
SVD prune
0,8799
0,5064
0.3906
0.2957
0,2016
0,1549
0.1222
0.0957
80
3.2e-03
2.le-02
1.5c-02
1.2e-02
L.e-02
1.2e-02
Table 6: WRN16-4 C0
Rel. perturbation
0.0003 0.0006 0.001 0.0013 0.0016 er (%)
Baseline, FGSM
0.5845 0.5267 0.4606 0.4187
0.3841
0
CowiUR R
T 0.1
0.6492
0.5806 0.5009 0.4499
0.4011
50
== 0.5
0.6062 0.5442 0.4693 0.4199
0.3774
50
Baseline, PGD10
0.5925
0.5401 0.4713 0.4289
0.3878
0
RondUR
== 0.1
0.6552
0.5918 0.5127 0.4531
0.3972]
50
7 0.5
0.6160 0.5547 0.4823 0.4290
0.3810
50
18

---

with & being the Clarke's generalized gradient operator [1],, we observe below that this is the case for broad list of activation functions and feature spaces X'.
LeakyReLU. For x E R.,o > O, let (x) = max x) + a min æ). Then any Us E (x( is [min(a; 1), max((, 1)] otherwise. Thus
such that V 1 if x 2 a if 0: us
12>0 + a1x<0 + $1,=0 max(a, 1) max + min
co = sup cond = sup sup #0 (a)r(a)
z
Tanh. For x E R. let (a) tanh(x). Then (xx) cosh2(x) and thus
[r|
|
co)d(o) = sup Lanh(x))] cosh"("))
-uup 1e -- t
Since the maximum of cond(o, x) reached at zero, where the function can be extended by continuity.
Hardtanh. For a E [-a, a] and a > 0, let let = alzza sl-<-a =E(-a)a) Then, we have that (z)) coincides with the derivative values in all points but x = tta those two points, we have (a(+a) = [0, 1). Thus, for any E (r()), we have
cond(o)= sup
sup =E(-0,a
=E =E
(x)
Logistic sigmoid. For let (x)) (1 (1- Then o (z) ox) oxx)) and thus
cond(o;; x) = x (1 - (x)) = |x/e " *(1 -)
Therefore, when x 2 0, we have |a|e = < 1/e and (1 ") 2 1, thus cond(o; S L/e.
Softplus. For r E I, let (x) = an(1+e"), Then (z) S(z) (1+e and cond(a; [=]((r)a(z)- Thus, for x 2 0, we have cond( x1 < 1.
SiLU. For æ E let (x)) s01 + e =)= x((r). Then, (<x) = S(x) + x(()) S(x))
and thus for any 2 O we have
con = + x(1 S(x) 11
C Proof of Theorem 1
In the following, the proof of the main approximation result is presented. We underline that the core part of the proof relies on [i] Theorem 5.2). For completeness, we repropose here main elements of the argument. We refer the interested reader to [31 and references therein for further details.
Proof. Let (1) the solution of 5 time E [0, A). First, we observe that the projected subflows of W() = (1) + E()) and W (1) satisfy the differential equations
= P(Y) + P(Y)
W = P W W.
where P(-) denotes the orthogonal projection into the tangent space of the low-rank manifold M,. Next, we observe that the following identities hold
(P(Y) - P(W)) - (PP' (V) - p (W)) = -PpP(()) = -p WY)
where (-())= = - P(.) represents the complementary orthogonal projection. The latter implies that
(Y - W. (P(Y) - ((W)) = (P-(Y) - -(W))))
Let = =32p(8 =)- It follows from
31
Lemma 4.2] that
(Y W. y W (P)
P(W))W) + + W. $ <> WI + WIl
- P(Y))
19

---

Further, we remind that
~ d || w1| w d
(Y W. Y W)
l
wll.
Hence, the error e(t) ||Y (t)l satisfies the differential inequality
S r + 1 e(0)
The error e(t) for E [0, A] admits an upper bound given by the solution of
1 = 72 2 + 7 =00 0.
The last differential mitiil-valuu problem admits closed solution given by
z(t) = / tan (tv m7),
where the last term is bounded by 2tg for \ S The proof thus concludes as follows
(t W(t)l < (t (t)l ||((t)] || - 2tm + tnj 3t7,
where the last estimate arise by the integral identity E(t) = Je E(s)ds.
20


Reviews:
[{"review_id": "1NYnXeWRul", "reviewer": "Reviewer_t9ct", "summary": "A brief summary of the paper is as follows:\n- Motivation: Make deep neural networks more efficient while maintaining robustness to adversarial attacks.\n- The authors propose a combined low-rank and approximate orthonormalisation constraint to help simultaneous model conditioning and compression during training.\n- The results show that both accuracy and adversarial robustness remain high.\n\nIt is well written and tackles an important problem, however it is missing references and comparisons to key existing works.", "questions": "These are questions that I have for the authors but I highly recommend they consider expanding the discussion and limitations part of their paper. They could touch on these points.\n- In the end of the abstract it is stated that \"This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.\" My question is: is it possible to check if a highly performing subnetwork exists, and does the orthonormal constraint affect the group of available subnetworks?\n- Related, how is the r parameter selected? Any suggestions for practice?\n- “Works only for wide enough networks” How wide is enough? \n- What is the added computational complexity of the new optimization? What is the time/memory cost relationship between the low-rank and the full network?\n- Why are VGG16 tested instead of the more common Resnets?\n- What attack is used for the adversarial experiments?\n- How sensitive is the method to choice of r?", "limitations": "The paper has a brief discussion on the limitations but this should be expanded to greatly improve the paper. Getting more candid explanations and thoughts on the limitations from the authors themselves would be welcomed. Possible discussion points would be the  sensitivity to parameter r (related to compression ratio). This is a trade off between performance and computation and can be different for different applications, however it is important to know the scope of available r values. Another very interesting limitation is the additional time cost of this method, specifically in relation to the additional optimization steps that need to be done. What is the relative added computation of this? A discussion of what conditions are needed for this optimization is of value would be very welcome.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strength of this paper lies in the writing and the novelty of the approach. Optimizing the low-rank decompositions alongside an orthonormal constraint is a very interesting approach that deserves an in-depth study. Preventing explosion of singular values is important and an interesting research direction that will be of value beyond low-rank neural networks and adversarial attacks. After addressing the weaknesses of this paper, the strengths can make this paper into a very complete piece of work.", "weaknesses": "- Paper does not mention previous studies that approach this same problem: adversarial robustness with low-rank constraints. [1] introduces tensor dropout, a method of increasing robustness (including robustness to adversarial attacks) using a stochastic dropout mechanism in the learned weight decompositions. It would be good to get a comparison to that baseline or at the very least a discussion of the similarities and differences between the approaches. \n- More discussion of the results and method would be good. There a number of questions that arise with this work that remain unanswered. Even though answering all of them might be out of scope for a single paper, it would be worth having the authors discuss more aspects of the approach they introduce here. I refer to my list of questions below, all of which would be interesting discussion points where the pros and cons of different decisions can be weighted.\n- The paper is missing a well-defined conclusion. That means the paper ends abruptly without comment on the impact of this work and what the authors see as possible next steps.\n\n[1] Kolbeinsson, A., Kossaifi, J., Panagakis, Y., Bulat, A., Anandkumar, A., Tzoulaki, I., & Matthews, P. M. (2021). Tensor dropout for robust learning. IEEE Journal of Selected Topics in Signal Processing, 15(3), 630-640.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your useful feedback.  \nBelow we provide a detailed response to all the raised criticism. We notice that all raised weaknesses, except for one point concerning a missing comparison/reference with paper [1], are entirely related to the missing concluding section and some missing discussion of the results which (quoting) \"would be worth having\".   \n**We are quite surprised these seemingly minor weak points correspond to a borderline rejection evaluation and would really appreciate additional clarification and feedback from the reviewer on what should be improved.**\nAs highlighted below, we will complement the existing discussion with details re the missing points and we will add a concluding section. \n\nRe the specific questions:\n\n> Comparison with [1]\n\nWe provide comparison in the global rebuttal\n\n>  In the end of the abstract it is stated that [...] is it possible to check if a highly performing subnetwork exists, and does the orthonormal constraint affect the group of available subnetworks? \n\nThis is a very good point. Proving the existence of highly performing low-rank subnets is a challenging problem that is currently subject to intensive investigation in the literature. **As we state at the end of §2 (lines 104-106), there is a variety of (quoting) \"recent work that shows the existence of high-performing low-rank nets in e.g. deep linear models [7,16,23,45]\"**.  The extension to more general classes of NNs (which include nonlinearities) is of course very important as well as more challenging and still largely open to our knowledge.   We refer for instance to the recent work [Galanti et al, SGD and Weight Decay Provably Induce a Low-Rank Bias in Deep Neural Networks] for recent insights in this direction.\n\n> Related, how is the r parameter selected? Any suggestions for practice? **and** How sensitive is the method to choice of r?\n\nThe parameter $r$ is responsible for the compression rate of the method and thus its selection depends also on the memory and the computational budget provided.  As discussed at lines 242-246,  the number of parameters (per layer) used by the proposed model is $r(n+m+r)$ and the compression rate depends on $r$ as\n$$\ncr = \\frac{\\text{parameters full net}-\\text{parameters compressed net}}{\\text{parameters full net}}=1-\\frac{r(n+m+r)}{nm}.\n$$\nOne can use these formulas to select $r$, given the available resources.  \nOn the other hand, we agree that – connecting to the previous question – a theoretical analysis providing insight on the choice of $r$ based on the investigation of the rank properties of the full network would be ideal. However, this is a challenging question that is the subject of a broad effort by the community and has no clear answer yet.   \nFinally, we also mention that the parameter $r$ can be fine-tuned algorithmically and that there are some papers that address this point (e.g. [51,70] from our paper) however, the integration of these hyperparameter optimization steps into our proposed framework is not straightforward and is left to future investigations.   \nAs we highlight in the limitations section, while this is a limit of the proposed analysis, our current evaluation provides enough details to hint at the way the performance changes with $r$, as we show results for both moderate (50\\%) and intensive (80\\%) compression rate regimes, showing consistent performance.\n\n> \"Works only for wide enough networks\" How wide is enough?\n\nThanks for this question. Our point here was simply that, given the compression rate formula $cr = 1 -\\frac{r(n+m+r)}{nm},$ if the layer width parameters $m$ and $n$ are not large enough, the method would not perform compression. This is simply due to the fact that the numerator $r(n+m+r)$ is much smaller than the denominator only if $n,m\\gg r$. For example, if $n=m$, then we have a positive compression rate $cr>0$ only if $n>2.5*r$. We will adjust the text here to clarify this point in the revised version of the paper.\n\n> What is the added computational complexity of the new optimization? What is the time/memory cost relationship between the low-rank and the full network?\n\nThank you for pointing this missing information out. We detail the computational complexity of the method below and will add a paragraph with this info in the new version.  \n\nEach pass of Alg1 is done against a batch $x_{\\mathrm{batch}}$. In order to obtain minimal computational costs, in the algorithm we evaluate $USV^\\top x_{\\mathrm{batch}}$ sequentially: first $v=V^\\top x_{\\mathrm{batch}}$, then $u=Sv$, finally $Uu$. Assuming the size of the batch is negligible with respect to $n$ and $m$, the cost of these steps is $O(rm), O(r), O(rn)$, respectively. Adding the bias and evaluating the activation function requires $O(n)$ operations. Hence, overall we have a cost per layer of $O(r(n+m+1))$. Taping the forward evaluation to compute the gradient with respect to $U,S,V$ does not affect these costs as this is also done sequentially against the chosen batchsize. The QR decompositions used for $U$ and $V$ require $O(r^2n)$ and $O(r^2m)$ operations respectively, $O(r^2(n+m))$ overall. Finally, computing the SVD in the projection step for $S$ requires a worst-case cost of $O(r^3)$. Hence the overall cost per layer and per step is $O(r(1+r)(n+m)+r^3)$ as opposed to dense training, which requires $O(nm)$ operations. If $r\\ll n,m$ then the method is cheaper than the full baseline. For example, if $n=m$, this happens provided $r<\\sqrt n$. \n\n> Why are VGG16 tested instead of the more common Resnets?\n\nOur academic computational power did not allow us to scale to very large networks/datasets at the time of submission. We have now tested on Wideresnet on Cifar10 and Cifar100, **new results are in the global rebuttal**\n\n> What attack is used for the adversarial experiments?\n\nOur experiments are done using the FSGM (see lines 298-299 in the main paper). We have now added results for PGD attack. **new results are in the global rebuttal**"}, {"review_id": "UjkjcP1Pvh", "reviewer": "Reviewer_J9p9", "summary": "Some work has used low-rank matrix factorizations in order to reduce training and inference costs. However, the authors demonstrate that these low-rank models are actually more brittle, i.e., susceptible to adversarial perturbations -- they tend to have large/exploding singular values. Consequently, the authors propose a technique to train low-rank networks while constraining the condition number, alleviating this problem. ", "questions": "This is minor, but why did you only produce perturbations with FGSM as opposed to, e.g., PGD? I'd be curious how your compressed models withstand somewhat more powerful adversaries. ", "limitations": "The authors have adequately addressed limitations, e.g., noting that they introduce a new hyperparameter that may have to be tuned.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper triangulates a well-defined problem and proposes an effective solution -- the paper was generally quite clear and well-motivated\n- Demonstrates the method works in practical experience and also via an approximation theorem\n- Investigation of exploding singular values in low-rank networks\n- Comparisons for a variety of compression ratios / perturbation magnitudes\n- Good explanation of background and related work\n", "weaknesses": "- Would be good to substantiate \"improving the network robustness without affecting training nor inference costs\" -- it's not obvious this is true to me since the method involves, e.g., projection onto the Stiefel manifold\n- A conclusion would be nice -- I think you could save space by removing the definition of robust accuracy, which readers probably know", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your encouraging feedback. Below we provide our response to your weak points and questions. \n\n>  Would be good to substantiate \"improving the network robustness without affecting training nor inference costs\" -- it's not obvious this is true to me since the method involves, e.g., projection onto the Stiefel manifold\n\nThanks for this point - we agree that it is not obvious that the cost is not significantly affected and we will clarify this in the next iteration of the paper. As detailed in the response to Reviewer t9ct, we will add details about the computational cost of the algorithm. Specifically, the particular sentence at line 188 \"improving the network robustness without affecting training nor inference costs\" refers to a comparison with training using just the low-rank factors without the Stiefel manifold projection. Note that in our setting, the projection is done for the matrix $S$ which is $r\\times r$ so the cost of projecting has a worst-case complexity of $O(r^3)$. When $r\\ll n,m$ this cost is negligible and in this sense it does not affect training nor inference costs. We will support this sentence at line 188 with details. \n\n> A conclusion would be nice -- I think you could save space by removing the definition of robust accuracy, which readers probably know\n\n Thanks, we will definitely add a conclusion, as also indicated in the response to Reviewer t9ct\n\n> This is minor, but why did you only produce perturbations with FGSM as opposed to, e.g., PGD? I'd be curious how your compressed models withstand somewhat more powerful adversaries.\n\nWe have added results for PGD and also extended the evaluation to WideResNet on Cifar10 and Cifar100"}, {"review_id": "AKI4hR90hI", "reviewer": "Reviewer_8Zii", "summary": "In this paper, the authors propose a low-rank training algorithm that incorporates approximate orthonormal constraints. Through a comparative analysis with previous methods for training orthonormal neural networks, the experimental results demonstrate the superiority of the proposed algorithm.", "questions": "1. Some terminologies are not pre-defined. For instance, concepts like the Stiefel manifold and tangent space may be unfamiliar to readers who have not previously explored low-dimensional spaces. Moreover, it is crucial to establish definitions for significant notations beforehand.\n\n2. Theorem 1 establishes that the estimated matrices are closely aligned with the noisy target matrices. Nevertheless, in practical implementations, our main concern lies in the error between the estimated output and the target output. While it is possible to prove that an individual estimated weight matrix is close to its target counterpart, it does not directly guarantee that the final error won't exponentially grow as a result of multiple products and layers. Therefore, it might be worthwhile to explore how the significance of this theorem can be introduced in light of addressing the overall error in the system, considering the cumulative effects of multiple products and layers.\n\n3. There is a missing $.$ in Eqn.(1).", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. An algorithm was introduced to ensure that the condition number of weight matrices remains close to $1$ during the training process.\n\n2. Comprehensive experimental results were provided in the main paper.\n", "weaknesses": "1. Some terminologies are not defined.\n\n2. While the theoretical analysis provides some insights, it may not fully capture the advantages of the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, we provide below our response addressing the raised questions\n\n> Some terminologies are not pre-defined. For instance, concepts like the Stiefel manifold and tangent space may be unfamiliar to readers who have not previously explored low-dimensional spaces. Moreover, it is crucial to establish definitions for significant notations beforehand.\n\nThank you for pointing this  out. We did not realize that some geometric terminology might require further  detail. We will certainly add them to the revised version.\n\n> Theorem 1 establishes that the estimated matrices are closely aligned with the noisy target matrices. Nevertheless, in practical implementations, our main concern lies in the error between the estimated output and the target output. While it is possible to prove that an individual estimated weight matrix is close to its target counterpart, it does not directly guarantee that the final error won't exponentially grow as a result of multiple products and layers. Therefore, it might be worthwhile to explore how the significance of this theorem can be introduced in light of addressing the overall error in the system, considering the cumulative effects of multiple products and layers.\n\nThis is a fair point. Ideally, the Thm would provide a bound on the output of the network $f$ rather than its weights. On the other hand, while we agree that the error on $f(W)$ could blow up, even though the computed $\\bar W$ is close to the target matrix $W$, this depends on the regularity of $f$. For example, for a C-Lipschitz $f$  one has $\\|f(W)-f(\\bar W)\\|\\leq C \\|W-\\bar W\\|$. i.e., the variation of $f$ is controlled.\nInterestingly, one could obtain a relative version of such statement by looking into the condition number of $f$ with respect to the weights rather than the data and it would directly follow from Eq.(1)  that,\n$$\n    \\frac{\\|f(W)-f(\\bar W)\\|}{\\|f(W)\\|}\\leq cond_W(f)\\frac{\\|W-\\bar W\\|}{\\|W\\|} .\n$$\nA similar bound as the one in Prop1 holds for $cond_W$. We can add this type of analysis/discussion to the paper if the reviewer considers this would improve the quality of the contribution."}, {"review_id": "BuRDVGb0OS", "reviewer": "Reviewer_sVC2", "summary": "This paper proposes a training approach that can improve the robustness of low-rank DNN models from scratch. This method enforces the low-rank components onto the Stifel manifold and ensures well-conditioning to achieve robustness. Experimental results show that the proposed training method demonstrates robustness to the data perturbation.", "questions": "See Weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The robustness of low-rank format DNN models is an important topic because it can improve the security of efficient DNN models.\n+ Training low-rank models without initializing from a pre-trained dense model is also important to save energy costs.\n+ This paper provides much theoretical stuff.", "weaknesses": "- The contribution is incremental. Improving the robustness of a compressed model using condition number is already proposed in [R1]. The main contribution of this paper is to re-apply it to low-rank formats. Improving the robustness of low-rank DNN models is also investigated in [R2].\n- The motivation why using conditioning numbers in low-rank scenarios is unclear.\n- This paper leverages Riemannian optimization to enforce the orthogonality of the left and right components, $U$ and $V$. However, this optimization algorithm contains a lot of matrix multiplications, which are extremely costly. The overall computational cost is overhead, so the proposed method is opposite to the meaning of low-rank training, which aims to save training FLOPs.\n- In terms of the compression approach, why should we choose low-rankness? The overall performance of robustness and compression ratio is better than sparsity or other trends of compression techniques? This paper does not compare with any other related works like [R1] in experiments.\n- The dataset and selected models in experiments are trivial. Lager datasets like ImageNet and advanced models like ResNet-50 and MobileNet should be tested.\n\n[R1] Learning Extremely Lightweight and Robust Model with Differentiable Constraints on Sparsity and Condition Number\n[R2] On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer claims that the paper is incremental because: (a) condition number as a measure of the robustness of compressed models is used already in [R1], and (b) the robustness of low-rank NNs is investigated in [R2]. \nWe thank the reviewer for pointing these two papers out which we had missed. However, we strongly disagree our contribution is incremental with respect to [R1] and [R2] and we strongly disagree that (quoting) \"the main contribution of our paper is to re-apply [R1] to the low-rank format\". **The two papers are very different** as we detail below.\n\nRe [R1]:\n\nFirst, [R1] does not deal with low-rank weights but with sparse weights. Second, they observe that the condition number of the layers of VGG16 grows with **unstructured** sparsification of the weights. This is aligned with our observation of exploding singular values for **structured** low-rank nets, but it does not make our observation incremental. Moreover, we notice that they do not clarify\n- which dataset they used; \n- what condition number are they measuring (product of the layers perhaps?); \n- whether they have made the same observation on other datasets/networks.  \n\nThird, they propose a method based on Kronecker decomposition for *adversarial training*, where the weights are decomposed as $W = A_1\\otimes A_1$. Sparsity is promoted by adding additional regularization terms and further regularization is used to promote small condition number via a tailored weight-decay combined with a logarithmic barrier on the determinant of $A_i^TA_i$.   \n\nThis is different than the low-rank training discussed in our paper as \n1. we reduce the number of parameters during the entire training process by directly training matrices in low-rank factorized form; although the Kronecker format used in [R1] is low-parametric, penalty-based approaches as the ones they propose may in general make the training process more expensive than \"plain training\" and require tuning of many additional hyperparameters\n1. we do not  perform any adversarial training\n\nRe [R2]:  \n\nThe paper takes on a different point of view, showing that adversarial training alone tends to reduce the number of parameters. Thus, using their own words, they study \"whether it is possible to improve the adversarial robustness by promoting sparsity and low-rankness without adversarial training''. This second goal is more aligned with our work, however, they achieve this again by adding several penalty terms to the loss (three additional terms) which are based on the nuclear norm of the weight matrices and **do not reduce parameters during training.**  \nInterestingly, they seem to observe that simultaneously low-rank and sparse weights promote robustness against adversarial examples, which goes a different way than our observation about loss of robustness when **only** low-rank constraints are imposed. However, their evaluation is limited to quite small networks and we believe it would be worth further investigating whether adding sparsity on top of low-rankness can improve robustness. Moreover, they do not propose an algorithmic way to improve robustness. Overall, we find that contribution far from ours and do not see why our approach would be incremental over theirs.\n\nWe will properly reference both of these papers in the next version. \n\nConcerning your specific questions:\n\n> The motivation why using conditioning numbers in low-rank scenarios is unclear.\n\nAs we discuss in §3, using the condition number is a natural way to measure the sensitivity of a NN, in general. Concerning the low-rank framework, the advantage is that computing the SVD of the small $r\\times r$ matrix $S$ has a worst-case cost of $O(r^3)$ which is small when $r$ is small. Thus, the condition number of the network can be contained while training without prohibitive additional costs. This is what we state in lines 186--188. As pointed out in the response to Rev J9p9, we agree we are missing some clarification here and we will add details.\n\n> This paper leverages Riemannian optimization [...] However, this optimization algorithm contains a lot of matrix multiplications, which are extremely costly [...]\n\nWe agree that the additional operations required to impose $cond(W)\\leq 1 +\\tau$ add computational overhead, but these are implemented on small factors only. Thus we disagree: **these operations are not extremely costly.** As also detailed in the response to Rev t9ct, the overall cost per layer of our method is $O(r(1+r)(n+m)+r^3)$ as compared to $O(nm)$ of standard training. When $r$ is small enough (e.g. $r<\\sqrt n$ for $n=m$), the cost is fewer than the plain baseline.\n\n> In terms of the compression approach, why should we choose low-rankness? [..] This paper does not compare with any other related works like [R1] in experiments.\n\nLow-rank compression has the advantage that one can use the geometric properties of the matrix manifold to (a) provide theoretical analysis for the quality of the computed compressed net (see our thm 1) and (b) train directly on the manifold reducing the cost of the entire training phase. Moreover, as we point out in line 37-38, 104-105 & the response to Rev t9ct, there exists a variety of evidence showing that weights of NNs are (approximately) low-rank, which thus motivates the search of low-rank networks to start with. On the contrary, while certainly providing a number of advantages, unstructured pruning based on e.g. sparsification, does not rely on a manifold and lacks this type of supporting theory. For these reasons, we strongly believe the study of low-rank networks is a very relevant topic. \n\nRe comparison with other work, we stress that **we compare with 6 baselines**, one of which [70] uses a regularization strategy similar to [R1]\n\n> The dataset and selected models in experiments are trivial [..] \n\nWe added evaluation on WideResNet/Cifar100 and we additionally tested model robustness against PGD adversarial attacks. Please see the global rebuttal block for results"}, {"review_id": "PM9vMPwWF6", "reviewer": "Reviewer_UZkq", "summary": "This work demonstrates that conventional low-rank training methods suffer from poor adversarial robustness, which is hypothesized to stem from ill conditioning of weight matrices during training. The authors address this by constraining low-rank weight matrices to be approximately orthogonal. ", "questions": "7. Precisely which parts of the algorithm in 4.2 are original? My impression is that the main contribution is step (4), the projection onto $\\Sigma_s(\\epsilon)$.\n8. In proposition 1, X is referred to as the \"feature space\" -- is this a deliberate deviation from the more standard terminology of \"input space\"? Or am I missing something here.\n9. Related to question 2, I'm a little confused about the notation in Proposition 1. The $cond(f; x)$ definition boils down to the function $f$ taking $x$ as an input; proposition 1 uses $cond(\\sigma_i; x)$, even though $\\sigma_i$ is at an intermediate layer in the network and does not directly take $x$ as an argument. My interpretation would be that this should read $\\cond(\\sigma_i, W_i z_i)$. It doesn't really affect anything downstream as we're just taking the sup over the argument to the condition number anyways, but the notation seems a bit imprecise to me as on the LHS of the inequality we're taking the sup over caligraphic X representing the input space, and on the right hand side it's the sup over caligraphic X representing the feature space, I feel that each intermediate feature space should have its own notation $X_i$ or something of the sort.\n10. In 157 the condition number for logistic sigmoid requires nonnegative features, which does not seem to be satisfied in general for intermediate neural network layers.\n11. In theorem 1, what is the significance of the phrase \"assuming no numerical errors\" -- is this referring to rounding errors, or something more serious like non-invertibility of a matrix during execution?\n12. The definition of relative error in equation (1) doesn't sit perfectly right with me, as it is dependent on the normalization of the inputs. If all inputs were shifted by some large amount, the relative error would increase although nothing important about the data has changed. Also, the relative error at x=0 is always zero (or undefined?) which doesn't make a lot of sense to me. Would appreciate any clarity here.", "limitations": "Limitations are explicitly addressed in section 4.2.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I found this paper to be a solid contribution, although my lack of familiarity with low-rank approaches means I'm not exactly sure which parts of the paper are novel.\n\n1. The observation in Figure 1 that low-rank training methods lead to exploding conditioning numbers is quite interesting and is not something I have seen before in the literature.\n2. The empirical gains in the experimental section are substantial, even if only using a FGSM atatck.\n3. The paper is very well-written and organized, with clear explanations and visualizations.\n4. The approximation guarantee is a nice inclusion of theory to explain the numerical advantages of avoiding high-curvature stiffness points.", "weaknesses": "5. The observation behind S1 was left unexplained: why do low-rank constrained matrices lead to large condition numbers? This might be out of scope but felt somewhat unsatisfying.\n6. The empirical evaluations only used the fast gradient sign method, which is a weak attack. This is my biggest reason for not giving an full accept rating -- I'd like to at least see PGD on this problem, although the results for FGSM are reasonably convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging feedback. We have now implemented new experimental evaluation with PGD attack and on one additional network (wideresnet) on both cifar10 and cifar100. All new results are available in the pdf attached to the 'general rebuttal comment'.\n\nRe the specific questions:\n\n> The observation behind S1 was left unexplained: why do low-rank constrained matrices lead to large condition numbers? This might be out of scope but felt somewhat unsatisfying.\n\nThis is a good point. At the moment we don't have additional theoretical nor experimental insight on this matter other than our guess (stated in §1):  the singular values grow very large in order to match the baseline accuracy and to compensate for the lack of parameters, while some small sing values are also still there as the network is compressed in low-rank format. We will investigate this question further in the future as we agree it is an important question that needs to be further explored.\n\n> Precisely which parts of the algorithm in 4.2 are original? My impression is that the main contribution is step (4), the projection onto $\\Sigma_{s}(\\epsilon)$.\n\nThe equations that describe the gradient flow on the low-rank manifold (eq. (5) in the paper) are known in the geometric integration literature, but not broadly discussed in the ML community to our knowledge. Equations (5) often have instability issues, causing practitioners to avoid using them directly. The novelty of our approach lies in establishing a framework where the utilization of the provided formulation (5) is problem-free. This is thanks to the coupling with the projection onto a Stiefel-like manifold which has a dynamically changing barycenter. The projection (and the corresponding analysis of stability) and the structure of the chosen manifold are entirely novel.\n\n> In proposition 1, $X$ is referred to as the \"feature space\" --- is this a deliberate deviation from the more standard terminology of ``input space\"? Or am I missing something here.\n\nApologies, this is simply a typo. We will change this into 'input space'. \n\n> Related to question 2, I'm a little confused about the notation in Proposition 1. The $cond(f)$  definition boils down to the function $f$ taking $x$  as an input; proposition 1 uses $cond(\\sigma_i, x)$, even though $\\sigma_i$ is at an intermediate layer in the network and does not directly take as an argument. My interpretation would be that this should read $cond(f,W_iz_i)$. It doesn't really affect anything downstream as we're just taking the sup over the argument to the condition number anyways, but the notation seems a bit imprecise to me as on the LHS of the inequality we're taking the sup over calligraphic $X$ representing the input space, and on the right-hand side it's the sup over caligraphic $X_i$ representing the feature space, I feel that each intermediate feature space should have its own notation or something of the sort.\n\nWe totally agree and we will modify $X$ into $X_i$, thanks for pointing this out.\n\n> In 157 the condition number for logistic sigmoid requires nonnegative features, which does not seem to be satisfied in general for intermediate neural network layers.\n\nAgain, thank you. We agree. We will modify based on the following observation: If $\\mathcal X_i = W_i \\mathcal X_{i-1}$ then if $|z_{i-1}|\\leq c_{i-1}$ entry-wise,  we have $ |x_i| \\leq c_{i-1} \\max_{uv} |W_i|_{uv}=:c_i $. \nThus \n$$\ncond( \\sigma ) =  \\sup\\_{ x \\geq -c_i } |x|e^{-x}(1+e^{-x})^{-1} \\leq \\max\\{c\\_{i}, 1/e\\}\n$$\n\n> In theorem 1, what is the significance of the phrase \"assuming no numerical errors\" --- is this referring to rounding errors, or something more serious like non-invertibility of a matrix during execution?\n\nThanks for this question. The sentence refers to the fact that we assume that all the intermediate computed quantities (the updates and the projections) in Algorithm 2 are exact, i.e., no numerical error has been introduced.\n\n> The definition of relative error in Equation (1) doesn't sit perfectly right with me, as it is dependent on the normalization of the inputs. If all inputs were shifted by some large amount, the relative error would increase although nothing important about the data has changed. Also, the relative error at $x=0$ is always zero (or undefined?) which doesn't make a lot of sense to me. Would appreciate any clarity here.\n\nIf we understand your doubt correctly, the issue you are concerned about is the fact that if we look at the condition number on a point $x$ such that $||x||$ is very large while both $||f(x)||$ and $||\\delta||$ remain moderate (unif bounded for example), then the ratio $R(\\tilde f,x;\\delta)$ would grow as $||x||$. This is correct but it is part of what we want to measure. While we agree that if we shift the data by a large constant nothing important changes on the data while the cond number might be affected, it is also true that the sensitivity with respect to the noise should depend on the size of $x$ in the sense that we are allowed to make **relative** small perturbations: we want to look at $||\\delta||/||x||\\leq\\epsilon$ in practice, not just $||\\delta||\\leq \\epsilon$. If $||x||$ is large, also $||\\delta||$ can in principle be chosen larger. This is what we usually do when selecting the size of the noise of the attack as we set $\\epsilon = k/255$ for an RGB image, where $255=||x||_\\infty$.   \nAs for the points where $f$ is zero: usually one discards these points as the relative error is not defined there and the absolute notion $||f(x+\\delta)-f(x)||$ is used instead. This is standard in the theory of conditioning and it corresponds to the intuition that when a function is zero any variation of the function is simultaneously an absolute and a relative change.\nWe hope our answer provides sufficient clarification, but we are happy to elaborate further if not. "}]

Author rebuttal:
We thank all the reviewers for their valuable feedback and useful comments and criticism. We provide a detailed response to all weak points and questions in the individual rebuttal blocks below. We use this general rebuttal block to report additional evaluation in response to a number of reviewers' questions.

# Comparison with reference [1] from Rev t9ct

[1] proposes a singular value dropout approach where a number of singular values are removed at random during training. However, they only implement this form of randomized rank reduction in the final tensor regression layer. This is not comparable with our approach which instead reduces the rank of the entire network (all the layers). We implemented the approach of [1] on the entire network (instead of just the final layer as it is done in [1]), however the results are not competitive and we believe they are not worth being included in the paper. We will instead reference [1] in the related work section.

The tables 4 and 5 in the attached pdf show the performance of [1] when the dropout is performed on the entire network at different compression ratios. A similar performance is observed for zero compression rate.

# Additional evaluation on WideResNet and using PGD adversarial attack
In table 1, we provide robustness results using PGD attack with 10 steps and L2 norm. (This table is analogous to Tab.3 in supplementary material, just with PGD10-L2 instead of FGSM). As demonstrated, our model with 50% compression rate still outperforms baseline,  Riemannian methods and other low-rank methods.

Additionally we implemented CondLR for WideResnet 16-4(depth-width) and trained it on CIFAR10 and CIFAR100.
Table 2 (CIFAR10) provides comparison for two settings of our method ($\tau = 0.0$, $\tau = 0.1$) against baseline in the case of compression rates 50% and 80%. Performance of each model is measured in terms of the robustness against FGSM and PGD10 attacks with L$\infty$-norm. As demonstrated, for 50% compression rate our method gets better results than baseline model and 80% compression rate scores reasonably close to baseline.

Table 3 (CIFAR100) provides comparison for two settings of our method ($\tau = 0.1$, $\tau = 0.5$) against baseline for compression rate 50% (compared with FGSM and PGD10 attacks with L$\infty$-norm). Similarly, better performance of our method against baseline is observed.


Meta-review: This work focuses on the study of low-rank training. The authors demonstrate that conventional low-rank training methods suffer from poor adversarial robustness, primarily attributable to the ill-conditioning of weight matrices during training. The authors tackle this challenge by imposing constraints on low-rank weight matrices to make them approximately orthogonal.

The authors have successfully addressed the concerns raised by the reviewers, and all reviewers recommend acceptance. I concur with the reviewers and also recommend acceptance. The authors are advised to incorporate the discussion of comparison with existing research and include additional experimental results in the final version.
Decision: Accept (poster)