Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks
Authors: Steffen Schotthöfer
H. Lexie Yang
Stefan Schnake
Venue: NeurIPS 2025 oral ()
Keywords: Low Rank
Adversarial Robustenss
Adversarial Attacks
Rank Adaptive
Computer Vision
Compression
Primary area: deep_learning

Abstract: Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing clean accuracy. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94 compression while recovering or improving adversarial accuracy relative to uncompressed baselines.

Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks
Steffen Schotthöfer; H. Lexie Yang! and Stefan Schnake"
"Computer Science and Mathematics Division, 'Geospatial Science and Human Security Division
Oak Ridge National Laboratory Oak Ridge, TN 37831 USA
yang sca.ang..ccnaaeeerrrrr.....
Abstract
Deployment of neural networks on resource-constrained devices demands models that both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low- rank training scheme enhanced with novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models adversarial perturbations without sacrificing accuracy on clean data. The method is model- and data-agnostic computationally efficient, and supports rank adaptivity to automatically compress the network hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94% compression while recovering or improving adversarial accuracy relative to uncompressed baselines.
Introduction
Deep neural networks have achieved state-of-the-art performance across a wide range of tasks com- puter vision and data processing. However, their success comes and memory demands, which hinders deployment in resource-constraing environments. While sig- nificant progress has been made in scaling up models through data centers and specialized hardware, complementary and equally important challenge lies in the opposite direction: deploying accurate and robust models on low-power platforms such as unmanned aerial vehicles (UAVs) or surveillance sensors. These platforms often operate in remote locations with limited power and compute resources, and are expected to function autonomously over extended periods without human intervention.
of substantial computational
This setting introduces three interdependent challenges:
Compression: Models must operate under strict memory, compute, and energy budgets.
Accuracy: Despite being compressed, models must maintain high performance to support critical decision-making.
Robustness: Inputs may be corrupted by noise or adversarial perturbations, requiring models to be resilient under distributional shifts.
Recent work has shown that these three objectives are inherently at odds. Compression via low-rank 38 or sparsity techniques [14] often leads to reduced accuracy. Techniques to improve adversar- or regularization-bssee degrade clean accuracy. Moreover, it has been observed that low-rank compressed networks can exhibit increased sensitivity adversarial attacks [35). Finally, many methods increase adversarial robustness of the model impose additional computational burdens during training 43 g inference
ial robustness-
as data augmentation
defenses -frequenty
15 28 further complicating deployment on constrained hardware.
Our Contribution. We summarize our main contributions as follows:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Low-rank compression framework. We introduce a novel regularization and integration method modify class of low-rank training methods that yields low-rank compressed neural networks, achieving a more than 10x reduction in both memory footprint and compute cost, while maintaining clean accuracy and adversarial robustness on par with full-rank baselines.
Theoretical guarantees. We analyze the proposed regularizer and derive an explicit bound on the condition number A of each regularized layer. The bound gives further confidence that the regularizer improves adversarial performance.
Preservation of performance. We prove analytically-and verify empirically-that our regularizer neither degrades training performance reduces clean validation accuracy across variety of network architectures.
Extensive empirical validation. We conduct comprehensive experiments multiple architectures and datasets, demonstrating the effectiveness, robustness, and broad applicability of our method.
Beyond these core contributions, our approach is model- and data-agnostic, can be integrated seamlessly with existing adversarial defenses, e.g., adversarial training [3], and never requires assembling full-rank weight matrices- the last point guaranteeing low memory footprint during training and inference. Moreover, by connecting to dynamical low-rank integration schemes and enabling convergence analysis via gradient flow, we offer new theoretical and algorithmic insights. Finally, the use of interpretable spectral metrics enhances the trustworthiness and analyzability of the compressed models.
2 Controlling the adversarial robustness of neural network through the singular spectrum of its layers
We consider neural network f as a concatenation of L layers 2f+1 = of (W*2') with matrix valued parameters W E |Rax,, layer input z' E Raab and element-wise nonlinear activation of. For simplicity of notation, we do not consider biases, but they are included for the numerical experiments in Section 6 The data X constitutes the input to the first layer, i.e. 2" = X. We assume that the layer activations of are Lipschitz continuous, which is the case for all popular activations (35] The network is trained on a loss function L. which we assume to be locally bounded with I Lipschitz continuous gradient. Throughout this work, we call a network in the standard format a "baseline" network.
Low-rank Compression: The compression the network for training and inference is typically facilitated by approximating the layer weight matrices by a low-rank factorization W' S'S with Uº, E "R and S E Rrxr, where r < n is the rank of the factorization. In this work, we generally assume that U V are orthonormal matrices all times during training and inference. This assumption deviates from standard low-rank training approaches [17]. however recent literature provides methods that are able fulfill this assumption approximately (55) and even exactly 38 37. r < n. the low-rank factorization with storage and matrix -eetoo computation cost cost of (2nr + r²) is computationally more efficient than the standard matrix format W with a
computational cost of O(n)).
The adversarial robustness of neural network widely used trustwor- thiness metric, can be measured by its relative sensitivity S to small perturbations ö, e.g., noise, of the input data X [49 [1] i.e., S(f, X, = || f(X)l
Adversarial robustness:
|X]|| In this work, we consider the
[1(x)
sensitivity in the Euclidean (E²) norm, i.e., ||- 11-1111 -1-- 112- For neural networks consisting of layers
with Lipschitz, continuous activation functions d., can be bounded (35) by the product
S(f, S (II=)
k(O"))
where x(W)= = |[W]] || is the condition number matrix W. Wt is the pseudo-inverse of W. and k(a) is the condition number of the layer activation function a. The condition number of the element- wise non-linear activation functions of can be computed with the standard definitions (see [45] and (35] for condition numbers of several popular activation functions). Equation ① allows us to consider each layer individually, thus we drop the superscript e for brevity of exposition.
'We provide an extension to tensor valued layers, e.g. in CNNs, in Section 5.1
NNote that the difference between the baseline and low-rank singular spectrum may be less pronounced for other layers and architectures. However, we have observed in all test cases that regularization with R makes the singular spectrum of the low-rank network more benign.

---

The sensitivity of a low-rank factor- ized network can be readily deducted from Equation
orthonormality ( by leveraging U and V. i.e., )(USVT)= = x(S). Thus, we only consider the r x r coefficient matrix S to control the sensitivity of the network. The condition number x(S) can be determined via singular value decomposition (SVD) of S, which is computationally feasible when r < n.
2
RobusiDLRT
1
Uomegshattze
Baseline
10P
10F
10
indexi
Figure I: The singular values <:WW) of sequential layer 7 in VGG16 for baseline training, unregularized dynamical low- rank training. and RobustDLRT with condition number regularizer R with 3 = 0.075 (see Section 5 The matrix W is formed as the first-mode unfolding of the convolutional tensor. Conditioning of the regularized low-rank layer is significantly improved compared the non-regularized low- rank and baseline layer."
Adversarial robustness-awars low- rank training: Enhancing the adver- sarial robustness of the network dur- ing low-rank training thus boils down to controlling the conditioning of S. which is a non-trivial task. Moreover, the dynamics of the singular spectrum of S of adaptive low-rank training schemes as Dynamical Low-Rank Training (DLRT) (38] become more ill-conditioned than the baseline during training, even if S is always full rank. In Figure [I we observe that the singular values Ç of rank 64 factorization of a network layer compressed with DLRT range from = 2.7785 to S 0.8210 yielding condition number of x(S) = 3.3844. In comparison, the baseline network has singular values ranging from Sr=1 1.8627 Sr=128 0.9445 yielding lower condition number of x(S) 1.9722. As result, an F-FGSM attack with strength c = 0.3, reduces the accuracy of the baseline network to 54.96%, while the accuracy of the low-rank network drops 43.39%, see Table
3 Related work
Low-rank compression is a prominent approach for reducing the memory and computational cost of deep networks by constraining weights to lie in low-rank subspaces. Early methods used post- hoc matrix [12] and tensor decompositions [23], while more recent approaches enforce low-rank constraints during training for improved efficiency and generalization.
Dynamical Low-Rank Training [38] constrains network weights to evolve on low-rank manifold throughout training, allowing substantial reductions in memory and FLOPs without requiring full- rank weight storage. The method has been extended to tensor-valued neural network layers [53], and federated learning (36). Pufferfish 47 restricts parameter updates to random low-dimensional subspaces, while intrinsic dimension methods 2 argue that many tasks can be learned in such subspaces. Gal.ore [56] reduces memory cost by projecting gradients onto low-rank subspaces.
In contrast, low-rank fine-tuning methods like low-rank adaptation (LoRA) [17 inject trainable low-rank updates into a frozen pre-trained model, enabling efficient adaptation with few parameters. Extensions such as GeoLoRA (37], AdaLoRA (53) DyLoRA [4). and DoRA 31 incorporate rank
adaptation or structured updates, improving performance over static rank baselines. However, these
fine-tuning methods do not reduce the cost of the full training and inference, thus are not applicable
to address the need of promoting computational efficiency.
Pruning is another well studied approach reduce the number of parameters of trained neural net- work 18 26 40 (57 7 19 by either sparsifying weight matrices or layer output channels of a network. Typically sparsity pruning is performed after training a fully parametrized neural network and thus only reduces memory and compute load during inference, while treating training as an offline cost.
Improving adversarial robustness with orthogonal layers has been recently studied topic in 10 35). Many of these methods can be classified as either a soft approach, where orthogonality is imposed weakly via regularizer, hard approach, where orthogonality is explicitly enforced in training.
the literature 3 48
double soft or- Examples of soft approaches include the soft orthogonal (SO) regularizer 48
thogonal regularizer (4), mutual coherence regularizer 44, and spectral normalization (32]. These regularization-bii based approaches have several advantages; namely, they more flexible to many
3

---

problems/architectres and are amenable to transfer learning scenarios (since pertained models are admissible in the optimization space). However, influencing the spectrum weakly via regularization cannot enforce rigorous and explicit bounds on the spectrum.
Many hard approaches strongly enforce orthngonality/vellloonttoooooo constraints by training on a chosen manifold using Riemannian optimization methods [25] 35). A hard approach built for low-rank training is given in [35]; this method clamps the extremes of the spectrum to improve the condition number during training. The clamping gives a hard estimate on the range of the spectrum which enables a direct integration of the low-rank equations of motion with reasonable learning rates. However, this method requires a careful selection of the rank which is viewed as a hyperparameter in [35. If is chosen incorrectly, the clamping of the spectrum, a hard-ttreeholding technique, acts as strong regularizer which could affect the validation metrics of the network.
Our regularization method detailed below falls neatly into soft approach and our proposed regularizer can be seen as an extension of the soft orthogonality (SO) regularizer [48] to well-conditioned matrices in the low-rank setting. As noted in 4 the SO regularizer only works well when the input matrix is of size m x n with m S n. However, we avoid this issue since the regularizer is applied to the square r X matrix St extension to convolutional layers is discussed in Section 5.1 In the context of low-rank training, the soft approach enables rank-adaptivity of the method.
4 Improving conditioning via regularization
We design a computationally efficient regularizer R to control and decrease the condition number of each network layer during training. The regularizer R only acts on the small x r coefficient matrices S of each layer and thus has minimal memory and compute overhead over low-rank training. The regularizer is differentiable almost everywhere and compatible with automatic differentiation tools. Additionally, R has a closed form derivative that enables an efficient and scalable implementation of VR. Furthermore, R is compatible with any rank-adaptive low-rank training scheme that ensures orthogonality of U, V.V.V.g. e.g. (55] 36 35).
Definition 1. We define the robustness regularizer R. for any S E R.xx by
R(S) = 1 az/1],
where
11sp a² 7
(2)
and 1 = L. is the X identity matrix.
The regularizer R. can be viewed as an extension of the soft orthogonal regularizer 48 where we penalize the distance of ST to the well-conditioned matrix T² Here og is chosen such that ||| = ||as/]]. Moreover, R is also a scaled standard deviation of the squared singular values (S)( ie..
-
E (())))
(3)
See Appendix Cor the proof. Therefore, R is a unitarily invariant regularizer; namely, RUUSV RS) for orthogonal V. These two forms of R are useful in the properties shown below.
Proposition The gradient of R 2 is given by VR(S) = 2 S ()()).(S).
See Appendix for the proof. The gradient computation consists only of r X r matrix multiplications and Frobenius norm evaluation. Thus VR is computationally efficient for r < m. Further, its closed form enables a straight-forwall integration into existing optimizers such as Adam or SGD applied to S.
Proposition 2 (Condition number bound). For any S E xr xr there holds
Table I: VGG16 UCM data. Comparison of regularized LoRA and DLRT trained networks un- der the l F-FFSM attack. Orthogonality of U, V increases adversarial performance significantly.
<(S) S exp (v2.1 R(S))
1
(4)
Method
e.r. [%)
alean Acc [%)
P.GGMM r 0.1
See Appendix C for the proof. Thus, if (r(
Non-regularieed DLKT RobosDLRL,
95.30
93.92
73.41
78.68
not too small, we can use R(S) as a good measure for the conditioning of S. Note that the
LoRA, 5 0.075
95.83
83.57
73.81

---

10
e-
0.01
10
0-0.1 0.01
3.0
8 0.05
8 0.1
2.5
0.15
e 0.15
8 0.2
10A
0=0.01
0-0.1
0m0:5
10-3
1.0
250000
10a
250000
optimbzer
optimizer step
x(S(C))
(b) R(S())
(c) E(t)
Figure 2: UCM Dataset, (t)())) and R(S(t)) of layer 4 of VGGI6 for different regularizations strengths Each line is the median of training runs. Higher / values lead faster reduction of the layer condition k(S), which quickly approaches its minimum value 1, and faster decay of R. Unregularized training (3 0) leads to xsS)= 1000 after few iterations.
singular value truncation used rank-adaptive methods ensures that (,( is always sufficiently
large. Figures 2a and 2b show the dynamics of R(S(t)) and k(S(t)) during low-rank regularized training; we see that KcSS (t)()) decays as R(S(t)) decays, validating Proposition 2
Remark 1. When U, V the smallest r singular values of UST are often zero-valued; thus, the bound of Equation (4) not useful. Table shows that the clean accuracy and adversarial accuracy of regularized LoRA significantly lower than standard or regularized training with orthonormal U,, V.
orthonormal, e.g., in simultaneous gradient descent training (LoRA).
We now study the stability of the regularizer when applied to least squares regression problem. i.c., M|
given a fixed M e Rrxr we seek to minimize [(S) := (R(S) 2 MM| over S E R xr
Proposition 3. Consider the dynamical system generated by the gradient flow of J: namely, S(t) (V)()(()) + S(t) M. Then for any 2 0 we have the long- aome-time stability estimate
|1S(() M 2 +2B S. - (-(RR((()))d -
(5)
See Appendix C for the proof. We note that unlike standard ridge and lasso regularizations methods, R lacks convexity; thus long-time stability of the regularized dynamics is not obvious. However, VR possesses monotonicity properties that we leverage to show in (5) that the growth in J only depends on B, M, and the initial loss. Moreover, for large I, the change in the final loss by the regularizer only depends on 33 and the true solution M and not the specific path S(C). While training on the non-convex loss will not provide the same theoretical properties as the convex least-square loss used in Proposition 3 the experiments in Figure
confidence that adding our regularizer does not yield relatively large change the loss decay rate over moderate training regimes, Particularly, we observe empirically in Figure that the condition number x(S) of decreases alongside the regularizer value during training.
Remark We note R² can also be used in place of R. While R² is differentiable at R(S) we choose R as our regularizer due to the proper scaling in
5 A rank-adaptive and adversarial robustness increasing dynamical low-rank training scheme
In this section we integrate the regularizer R into a rank-adaptive, orthogonality preserving, and efficient low-rank training scheme. We are specifically interested in training method that 1) enables separation of the spectral dynamies of the coefficients from the bases U, V and 2) ensures orthogonality of U, V at all times during training to obtain control layer conditioning in compute and memory efficient manner. Popular schemes based upon simultaneous gradient descent of low-rank factors such as LoRA [1 are not suitable here. These methods typically do not ensure orthogonality of U and V. Consequently, TVUSVT * R(S), and this fact renders evaluation of the regularizer R computationally inefficient.
5

---

Thus we adapt the two-step scheme of [36] which ensures orthogonality of U, V. The method dynamically reduces increases the rank of the factorized layers depending on the training dynamies and the complexity of the learning problem at hand. Consequently, the rank of each layer is no longer a hyper-parameter that needs fine-tuning, c.f. 17 35,, but is rather an interpretable measure for the inherent complexity required for each layer.
To facilitate the discussion, we define = L + 3R as the regularized loss function of the training process with regularization parameter O. To construct the method we consider the (stochastic)
gradient descent-basse based update of a single weight matrix W = W++1 - W 2 for minimizing L with step size À > O. The corresponding continuous time gradient flow reads (t) = V [(W (t)).
which is high- fith-mmmeniionll dynamical system with a steady state solution. We draw from established dynamical low-rank approximation (DLRA) methods, which were initially proposed for matrix- valued dynamical systems [20]. DLRA was recently extended to neural network training [38 53 36
37
22
16
to formulate consistent gradient flow evolution for the low-rank factors U, S, and V.
The DLRA method constrains the trajectory of W to the manifold Mr, consisting of n x matrices with rank r, by projecting the full dynamics W onto the local tangent space of M, via orthogonal projection, see Figure The low-rank matrix is represented as USVT E M,, where E T and V R have orthonormal columns and S IR""" is full-rank (but not necessarily diagonal). An explicit representation of the tangent space leads to equations for the factors U, S, and V in [20] Proposition 2.1]. However, following these equations requires prohibitively small learning rate due to the curvature of the manifold [29]. Therefore, specialized integrators have been developed to accurately navigate the manifold with reasonable learning rates 22 5
Below we list the method of [36] with the changes introduced by adding our robustness regularizer. We call the resulting scheme RobustDLRT, and single iteration of RobustDLRT is specified in Algorithm
Basis Augmentation: The method first augments the current bases US, V at optimization step by their gradi- ent dynamics Cu VvL C via
V V
i orth((U* VV R
(6)
V = orth([V" V VV Rax2"
V&L.
to double the rank of the low-rank representation and subsequently creates orthonormal bases Ú, V. Here orth(A) denotes an orthonormal basis for the range of A and denotes horizontal concatenation of matri- ces. Since R(USVT) = R(S). (UUSV) VVR(USVT) 0: hence VoL V and Vv VvL are used in (6. The span of U contains U', which is needed ensure of the loss does not increase during aug- mentation, and a first-order approximation of span(U7++3 using the exact gradient flow for U, see [36] Theorem for details. Geometrically, the latent space
M,
M,
Figure 3: Geometric interpretation of Algorithm [ First, we compute the parametrization of the tangent plane TM,. Then we compute the projected gradient update with V VgL. Lastly, retract the updated coefficients back onto the manifold M,. The regularizer R steers training to regions of M, with lower curvature.
s ==1ŪZV Z 2rx22
can be seen as subspace of the tangent plane of M, at STT., see Figure]
Latent Space Training: We update the latent coeffi- cients š via Galerkin projection of the training dynam- ics onto the latent space S. The latent coefficients š are updated by integrating the projected gradient flow
'Technically the latent space contains extra elements not in the tangent space, but the extra information only helps the approximation.

---

Algorithm 1: Single iteration of RobustDLRT.
Input: Initial orthonormal bases U, V E guxr and diagonal E RTx;;
8: singular value threshold for rank truncation; X: learning rate.
1 Evaluate )
Forward evaluate
2 Gu + VvL(USV)) ); Gv + VVL(USTT)
/* Backprop on basis
0 orth([U Gul); V + orth([V Gvl)
/* augmentation in parallel
4 ÜTUSVTP
/* coefficient augmentation */
S coefficient. ( (, 8ez 1, U,S.V ttruncation (3,0, P)
/+ regularized coefficient training */
def coefficient. coefficient_updatee So: coefficient, 8a; # local steps, X: learning rate, 8: robustness regularization weight):
for 8 1.....8. 8, do
Gg t x VVs R(S.)
10
---1 optim(Gss)
/* optimizer update, e.g., SGD Adam */
return S..
def truncation(S: S: augmented coefficient, Ü: augmented basis, V. augmented co-basis ): Er. On + truncated svd(S) with threshold to new rank rı U- UPr - VQn.
13
/* Basis update */
15
St Σr, return U,S,V
/* Coefficient update with diagonal E, */
- Vw LV = - V using stochastic gradient
descent other suitable optimizer for number of s. local iterations, i.e.
R(((,,,, = O,..., 8, Equation (8 initialized with So VT 2rx22 and set S S..
3 = 3.
(8)
Finally, the latent solution ÜSVT retracted back onto the manifold M, The retrac- tion be computed efficiently by using a truncated SVD of Š that discards the smallest singular values. To enable rank adaptivity, the new rank instead of r can be chosen by variety of criteria, e.g., singular value threshold .l|sr....., s2rl1l2 < 9. Once suitable rank determined, the bases U and V are updated by discarding the basis vectors corresponding to the truncated singular values.
Truncation:
Remark 3. We note that R will likely increase the smallest singular values of $ to improve r(()). This could theoretically increase the truncated rank over non non-regularized DLRT and result in less compression. However, we find in the experiments in Section that RobustDLRT has similar compression rates to DLRT.
The computational cost of RobustDLRT is asymptotically the same as LoRA, since the reconstruction of the full weight matrix W is never required. The orthonormalization. computation of the regularizer R. and the SVD for accounts for (nrr²), Orr3). O(r3) floating point operations, respectively. When using multiple coefficient update steps 8, 1, the amortized cost
Computational cost:
lower than that of LoRA, since only the gradient with respect to required in most updates. While the regularizer may be applied to full-rank baseline models, its (n²²) computational scaling significantly increases training costs.
5.1 Extension to convolutional neural networks
The convolution layer map in 2D CNNs translates a W x H image with N, in-features to No out-features. Using tensors, this map is expressed Y C X where X N y Noxx x and C' N Sw xS# is the convolutional kernel with a convolution window size Sw SH. Neglecting the treatment of strides and padding, C X is given tensor contraction by
(o, w. h) Ex.or.n
=
C(o, c, Sw, 8h X + h + sn)
(9)
7

---

Table UCM and CifarI0 benchmark. Clean and adversarial accuracy means and std. devs. of the baseline and regularized low-rank networks for different architectures. We report the low-rank results for 3 = 0.0 (DLRT) and the best performing 3 that is given in Table 9 A Algorithm| (RobustDLL is able to match or surpass haseline adversarial accuracy values at compression rates of up to 94% in most setups. All runs where RobustDLRR surpasses the uncompressed baseline are highlighted.
UCM Dutu
Clea
Acc [S] for e c
Acc [S] For Jitter
Acc [R] Missp..
Method
ca. c2.
Acc. [S]
0.05
0.1
0.3
0.035
0.045
0,025
0.1
0.75
Honcline
a.
94.40-0:32
56 9: 50
26 26-00-2.00 64
54 14.9--2999
89 SK+2299
85.05-3440
72:72-160 61
39.2943666
25.5+
DLRT
95.30
30.92:0.225
$7.95 102
72.41:2.08
43.391..85
85.9911.22
67.41:1633
$5.79.1.51
40.42:2.89
20.12::.99 92
RobunDLRT
95.84
(4.81-0.55
89.124 1.
78.68-2.30
53.30+3.1 14
88:3341.200 1.00
79,81=0.38
90,33-0:00
70.1253.08
47.31
Beacline
0.0
94.23-0.71
59.03+1.5 33
78.66-2..66
39.45+2.96
90 66
K5.34-1.00
K3.00-145
40 00.54+4.58
2201+3.21
DLKT
94.89
90.70::0.71
86,582:122 23
07.55 27.56
28.923:265 65
83.9021.300
13.41::1.39
$7.15:1.18 1.18
40.173:4.56
14.1823.75
RubuaDERT
93.57:00.54
$7.90::0.91
72.9681.55
32.852:246 46
86.7769076
74.31=1.50
88:00:1:15
60.9784.188
26.5683.64
Beseline
00-22-0.00
93.02:0.35
10:018
89.718:026
05.7141:22
93.21=1.17
10.62=135
11
51,05-3.17
4..01:3.07
DLKF
66.7
96, 08:00
91 01:11:0044 46
10:00.2
62.4510441
86.611100
09.1
00,08:110
12
41.52.33.24
RobmatoLR RF
96.41 10.67
92,523:0354 54
85.67 10.444
69961:042
91.0310066
84.19:1.9 1.39
82.38-1 31
46. 26.79522.5
368
Cife Date
VOITTS
Rescline
0.0
80.22-0.45
76.22+1 38
68 (1.78-2.01
$4.97+22554
78.60+1.12
73.54-1.55
71.51-133
57. 37.56+2..00
16, 12
DLRT
94.37
60.23:0.62
74.07= 23
50.55:11.79
28.744221
72:31+11044
56.21:1411
79
56 15
50.58:2.36
38 98:95:111 94
KoboottL
94.18
89 00.59
76.06:1100
62.06:11.0
32.775:2004
75.531099
19.90:1:22
N7.65 67
00
14 94.0222.01
Beacline
0.0
59.34:0.49
75.99:145 42
64.21
31.7682.45
74.96:100 09
68.9911.3
74.77=1 26
40.58 00.5842..58
08.05:1 .9
DLRT
95.13
58.13:00.56
72.02:1.34 1.34
35.83
21.59::216 16
66.9521005
58.57=1.55
79.42=1 0.08
47.05:2..18
12.922:1.77
90/77+-2990 90
65.69-1.88
95.42:00.35 35
19.943:0.55 95
32.0992.005 05
84 04.65510.88 888
17.20:1.04
$2.17=1100 49
16:00:2.344
13.2942.01
13.42
95.39:0.41
79.501..911
61.62111448
30.323:190
83.333:0.80 90
78.16:00.95
59,32=1.44 1.4
1743322.8
14.4911.92 1.92
RubudDLRT
75.21
94,66:10.88
82:03:0:88
00.20:1144
38.05=1.99
88799760775
83,08:0:91
74,49=1
27.50:22.11
18.34:1.87
where 8u and 8) range from -Sw/2,... Sw and -Sn/2...., respectively, and o 1, 1...., No. w = 1,..., W, and h 1,.... H.
DLRT was extended to convolutional layers in [53] by compressing C' with a Tucker factorization. Little is gained in compressing the window modes they are typically small. Thus, only factorize C' in the feature modes with output and input feature ranks TO < No and Tı < Nr as
C(o, Sws sh.)
E =1 Uo (o) 91 (()0 91, Sw: 8h)-
(10)
Substituting (10p into and rearranging indices yields
Y(,, w. It) Eno Uolo, ( w. h), (go, w, h) E S(go qr, Sws sn) sn) X + Swo h + sh),
(11a)
(11b)
X(g,, w + sw, h + sn) = E. q1) { w + sw, h + sn).
(11c)
Remark 4. Aside from the prolongation Iaap and retraction 1lc) from/to the low-rank latent space,
the low-rank convolution map (11) features a convolution 11b
similar to (9) but in the reduced
dimension low-rank latent space.
Robustness regularization for convolutional layers. The contractions in (9) and (1b) show that the output channels arise from a tensor contraction of the input channel and window modes; hence, both 49) and (1bb can be viewed as matrix-vector multiplications where C' is matricised on the output channel mode; i.e., C - Mat(C) E R.No Sw and 5 - Mat( (S) E Sw Sw Therefore, we only regularize Mat(S) with our robustness regularizer. Moreover, we assume S Sy Sur which is almost always the case since To and rı are comparable and Sw SH >> 1, Then we regularize convolutional layers by R RMMa((S) ) so that SS is an TO X ro matrix, which is computationally efficient.
We remark that the extension of Algorithm to a tensor-valued layer with Tucker factorization only requires to change the truncation step; the SVD is replaced by a truncated Tucker decomposition of S. The Tucker bases Uo and Un can be augmented in parallel similarly to the matrix case.
6 Numerical Results
We evaluate the numerical performance of Algorithm I compared with non-regularized low-rank training, baseline training, and several other robustness-enhancin methods the VGG16, VGG11, and ViT architectures and University of California, Merced (UCM), Cifar10, and ImageNetlk datasets. Detailed descriptions of the models, datasets, pre-processing, training hyperparameters,
8

---

Table 3: Imagenet Benchmark, ViT-321 trained with baseline Adam, DLRT, and RobustDLRT. We
report the low-rank results for unregularized / = 0.0 and the best performing B, given in Table 2
Algorithm| | (RobustDLRT) is able to match or surpass baseline adversarial accuracy values in most
setups. All runs where RobustDLRT surpasses the uncompressed baseline are highlighted.
Topl/TopS Clean
Top1/Top5 Acc [%] for F-FG,M
Top1/Top5 Ace [%] for Jitter,
Method
c.r. [%]
Ace. [%]
0.05
0.1
0.3
0.035
0.045
Baseline
0
74.37/92.20 20
43 43.58/73.75
31.42/63.42 42.70/70.43
16.03/43.41
43.09/78.24
35.57/74.96
DLRT
58.02
72.27/90.06
30.32/60.90
15.47/40.58
43.98/74.49
38.44/ 71.31
RobustDLLR
57.98
72.25/90.03
43.17/71.58
33.11 /62.82
25.24/50.65
48.22 /77.35
43.51/75.14
and competitor methods are given in Appendix B A reference implementation is provided at https://githi: com/ScSteffen /oaeee/nnoobutDDT We measure the compression rate (c.r.) as the relative amount of pruned parameters of the target network, i.c. C.I. = (1 # params low-rank #parans baseline net x 100. The reported numbers in the tables represent the average over 10 stochastic training runs. We observe in Table thaat that clean accuracy results exhibit a standard deviation of less than 0.8%; the standard deviation increases with the attack strength e for all tests and methods. This observation holds true for all presented results; thus, we omit the error bars in the other tables for the sake of readability.
UCM dataset We observe in Table 2 that Algorithm D can compress the VGGII, VGG16 and ViT-16b networks equally well as the non-regularized low-rank compression and achieves the first goal of high compression values of up to 94% reduction of trainable parameters. Furthermore, the clean accuracy is similar to the non-compressed baseline architecture; thus, we achieve the second goal of (almost) loss-less compression. Noting the adversarial accuracy results under the f2-FGSM Jitter, and Mixup attacks with various attack strengths c, we observe that across all tests, the regularized low-rank network of Algorithm [ significantly outperforms the non-regularized low-rank network. For the (2-FGSM attack, our method is able recover the adversarial accuracy of the baseline network. For Mixup, the regularization almost doubles the baseline accuracy for VGGI6. By targeting the condition number of the weights, which gives bound on the relative growth of the loss w.r.t. the size of the input, we postulate that the large improvement could be attributed to the improved robustness against the scale invariance attack [27 Section 3.3] included in Mixup. We refer the reader to Appendix B.1.4 for a precise definition of the Mixup attack featuring scale invariance. However, this hypothesis was not further explored and is delayed work. Finally, we are able to recover half of the lost accuracy in the Jitter attack. Overall, we achieved the third goal of significantly increasing adversarial robustness of the compressed networks. We refer to Table | for the used values of / and Appendix A. for extended numerical results.
future
Ci dataset We repeat the methodology of the UCM dataset for Cifarl0, and observe similar computational results in Table 2 Fur- thermore, we compare our method in Table 4 to several methods of the recent literature, see Sec- tion 3 We compare the adversarial accuracy un- der the E'-FGSM attack, see Appendix 1.2 for details, for consistency with the literature results. We find that our proposed method achieves the highest adversarial validation accuracy for all attack strengths even surpassing the baseline adversarial accuracy. Additionally, we find an at least 15% higher compression ratio with Robust- DLRT than the second best compression method, CondLR [35). A similar experiment for the Pro- jected Gradient Descent (PGD) attack [30] is given in Appendix A.2
Table 4: Comparison to literature on CIFARIO with VGG16 under the F-GMMM attack. The first three rows list the computed mean over 10 random initializations. The values of all other methods, given below the double rule, are taken from 35 Table 1]. RobustDLRT has higher adversarial ac- curacy at higher compression rates than all listed methods.
P.FGSM, c
Method
[%]
0.0
0.002
0.004
0.006
Baseline DLRT
89.83
78.61
64.66
53.71
94.58
59.61
47.56
RobusDLRI / = 0.15
94.35
89.35
78.72
66.02
54.15
Cayley SGD
25
89.62
74.46
58.16
45.29
Projected SGD
1
89.70
74.55
58.32
45.74
CondLR
35
30
89.97
72.25
60.19
50.17
CondLR R
A51
0.5
80
89.33
68.23
48.54
36.66
LoRA
50
89.97
67.71
48.86
38.49
ImageNetlk dataset Finally we repeat the methodology for the ImageNetlk dataset, us- ing the ViT-321 vision transformer trained from an ImageNet21k checkpoint, and report the results in Table 3
LoRA
17
80
88.10
64.24
42.66
29.90
SVD prune
51
50
89.92
67.30
47.77
36.98
SVD prune
80
87.99
63.57
42.06
29.27
The hyperparameters are obtained by
9

---

an initial sweep and reported in Tables RobustDLRT consistently yields higher Top- 1/Top-5 accuracy across f FGSSM and Jitter attacks than DLRT, with especially pronounced gains at larger perturbations (e.g., +9 points in Top-1 accuracy under 2-FGSM = 0.3). These trends are con- sistent with our ViT experiments in Table demonstrating that adversarial regularization enhances robustness without compromising scalability. We benchmark the training runtime of one ImageNet epoch on an A100 80GB GPU. DLRT requires 26m 07s, while RobustDLRT (with the regularizer) requires 27m 51s, corresponding to overhead of approximately 3%. This overhead can likely be reduced with further implementation optimizations, indicating that our approach is computationally scalable.
Black-box attacks We investigate the scenario where an attacker has knowledge of the used model architecture, but not of the low-rank compression. We use the Imagenet- Ik pretrained VGG16 and VGGI1 and re-train it with Algorithm and baseline training on the UCM data using the same training hyperparameters. Then we generate adversarial examples with the baseline network and evaluate the performance on the low-rank network with and without regularization. The results are given in T In this scenario, the weights from low-rank training, being sufficiently far away from the baseline, provide an effective defense against the attack. Further, the proposed regularization significantly improves the adversarial robustness when compared to the unregularized low-rank network. Even for extreme attacks with <= 1, the regularized network achieves 84.76% and 87.33% accuracy for VGG16 and VGG11 respectively.
Adversarial Training We evaluate the performance of low-rank training for VGGI6 on the UCM dataset using adversarial training. Following [33, we use the (2FGGSM attack for different values of c and train on both 50% clean and attacked images per batch. The results reported in T that RobustDLRT is both compatible with and able to benefit from adversarial training. DLRT without regularization benefits from adversarial training, but exhibits a clear margin to RobustDLRT. Additionally, RobustDLRT is able to approximately match the non-compressed baseline.
Table 5: UCM dataset Black-box attack. Ad- versarial images with the F2-FGSM attack are generated by the baseline network for differ- ent values of c. The baseline, DLRT (8 = 0), and RobustDLRT (B = 0.075) networks are then evaluated on these images. Regularized low-rank compression achieves high adversar- ial accuracy, even under strong attacks.
Table 6: UCM dataset- Adversarial Training. VGG16 is trained on 50% clean images and 50% images attacked with F2-FGSM for vari- ous c. The displayed numbers are the mean of 5 repeated runs. RobustDLRT (B = 0.075) is superior DLRT (3 0) and able approx- imately match the non non-compressed baseline.
(R.GGSM.
(LFGSMM
cn
1.0
Method
[%]
VOGIl VGG16
Baseline
95.30
91.81
76.00
92.55
87.98
0.05
95.15
92.66
92.47
91.33
38.76
86.85
84.76
0.075
92.49
92.49
90.96
Duseline
0.0
89.93
78.66
60.76
45:23
38.38
35.52
99.82
92.76
91.81
89.25
34,09
80.57
77.71
01.05
96.12
92.95
92.66
92.00
91.94
38.56
87.33
7 Conclusion
RobustDLRT enables highly compressed neural networks with strong adversarial robustness by controlling the spectral properties of low-rank factors. The method is efficient. adak-adapivve, and yields an up to 94% parameter reduction across diverse suite of models and datasets. The method achieves competitive accuracy, even for strong adversarial attacks, surpassing the current literature results by significant margin. Therefore, we conclude the proposed method scores well in the combined metric of compression, accuracy and adversarial robustness.
The accomplished high compression and adversarial robustness advance computer vision models and enable broader applications on resource-constrainee edge devices. These achievements also enhance energy efficiency and trustworthiness, positively impacting society. The regularization and condition number bounds further improve interpretability, which is crucial for transparency and accountability in critical decision -aaonn when applying the proposed methods.
10

---

Acknowledgments and Disclosure of Funding
This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-ACU5-000R2272 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Wlan(http:/eneeet
gov/ htt////wmooldda..oom /oee http//dpeppppplcceeeeeeeeeeeeeee mublic-accesse
This material is based upon work supported by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory (ORNL), managed by UT- LLC for the U.S. Department of Energy under Contract No. Dc-AC05-000R22725
S. Schotthöfer, H. L. Yang, and S. Schnake were supported by the Artificial Intelligence Initiative of the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory (ORNL), managed by UT- LLC for the U.S. Department of Energy under Contract No. De- AC05-000R22725.
This research used resources of the Compute and Data Environment for Science (CADES) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-000R2271
References
[1] P-A. Absil and J. Malick. Projection-like retractions on matrix manifolds. SIAM Journal on Optimization, 22()):135-158, 2012.
[2] A. Aghajanyan, S. Gupta, and L. Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 7319-7328, 2021.
[3] C. Anil, J. Lucas, and R. Grosse. Sorting out Lipschitz function approximation. International conference machine learning, pages 291-301 PMLR, 2019.
[4] N. Bansal, X. Chen, and Z. Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
[5] G. Ceruti, J. Kusch, and C. Lubich. A rank-adaptive robust integrator for dynamical low-rank approximation. BIT Numerical Mathematics, pages 1-26, 2022.
[6] G. Ceruti and C. Lubich. An unconventional robust integrator for dynamical low-rank approxi- mation. BIT Numerical Mathematics, 62(1):233444 2022.
[7] T. Chen, H. Zhang, Z. Zhang. S. Chang, S. Liu, P.-Y. Chen, and Z. Wang. Linearity grafting: Relaxed neuron pruning helps certifiable robustness, 2022.
[8] G. Cheng, X. Sun, K. Li, L. Guo, and J. Han. Perturbation-seeking generative adversarial net- works: A defense framework for remote sensing image scene classification. IEEE Transactions on Geoscience and Remote Sensing, 60: 60:1-11, 2022.
[9] M. Cisse, P. Bojanowski. E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness adversarial examples. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 863. PMLR, 06-11 Aug 2017.
[10] M. Cisse, P. Bojanowski. E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. International Conference on Learning Representations (ICLR), 2017.
[11] W. Czaja, N. Fendley, M. Pekala, C. Ratto, and I.-J Wang. Adversarial examples in remote sensing. In Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances Geogruphic Information Systems, SIGSPATIAL '18, page 608-411, New York, NY, USA, 2018. Association for Computing Machinery.

---

[12] E. L. Denton, W. Zaremba. J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 2014.
[13] J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 141X.6::11 2014.
[14] Y. Guo, A. Yao, and Y. Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
[15] M. Hein and M. Andriushchenko. Formal guarantees the robustness of classifier against adversarial manipulation. Advances in neural information processing systems, 30, 2017.
[16] A. Hnatiuk, J. Kusch, L. Kusch, N. R. Gauger, and A. Walther. Stochastic aspects of dynamical low-rank approximation in the context of machine learning. Optimization Online, 2024.
[17] E. J. Hu, Y. Shen, P. Wallis, Z. Alu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: 2021.
Low-rank adaptation of large language models. arXiv preprint
[18] T. Jian, Z. Wang, Y. Wang. J. Dy, and S. loannidis. Pruning adversarially robust neural networks without adversarial examples, 2022.
[19] A. Jordao and H. Pedrini. On the effect of pruning on adversarial robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1-11,2021,
[20] O. Koch and C. Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434 454, 2007.
[21] A. Kurakin, J. Goodfellow, and S. Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations, 2017.
[22] J. Kusch, S. Schotthöfer, and A. Walter. An augmented hackward-corrected projector splitting integrator for dynamical low-rank training. arXiv preprint arXiv:2502.03000 2025.
[23] V. Lebedev, Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky. Speeding-up convolutional neural networks using fine-tuned CP-decomposition. In International Conference on Learning Representations, 2015.
[24] H. Lee, S. Han, and J. Lee. Generative adversarial trainer: Defense to adversarial perturbations with GAN. arXiv preprint arXiv:1705.03387, 2017.
[25] J. Li, F. Li, and S. Todorovic. Efficient Riemannian optimization the Stiefel manifold via the Cayley transform. In International Conference on Learning Representations, 2020.
[26] Z. Li, T. Chen, L. Li, B. Li, and Z. Wang. Can pruning improve certified robustness of neural networks?, 2022.
[27] J. Lin, C. Song, K. He, L. Wang, and J. E. Hoperoft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In International Conference on Learning Representations, 2020.
[28] X. Liu, Li, C. Wu, and C.-J. Hsieh. Adv-BNN: Improved adversarial defense through robust Bayesian neural network. In International Conference on Learning Representations, 2010.
[29] C. Lubich and I. V. Oseledets. A projector-spliting integrator for dynamical low-rank approxi- mation. BIT Numerical Mathematics, 54(1):171-188, 2014.
[30] A. Madry, A. Makelov, Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[31] Y. Mao, K. Huang, C. Guan, G. Bao, F. Mo, and J. Xu. DoRA: Enhancing efrfmettr-efficient fine-tuning with dynamic rank distribution. L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 11622-16775 Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics.
12

---

[32] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[33] J. Nagy. Über algebraische gleichungen mit lauter reellen wurzeln. Jahresbericht der Deulschen snathemaliike-Veriiiiienn 27:37-43, 1918.
[34] R. Nenov, D. Haider, and P. Balazs. (Almost) smooth sailing: Towards numerical stability of neural networks through differentiable regularization of the condition number, 2024.
[35] D. Savostianova, Zangrando, G. Ceruti, and Tudisco. Robust low-rank training via approximate orthonormal constraints. Advances in Neural Information Processing Systems, 36:66064-66083, 2023.
[36] S. Schotthöfer and M. P. Laiu. Federated dynamical low-rank training with global loss conver- gence guarantees. arXiv preprint arXiv:2406. 11661,7877,. 2024.
[37] S. Schotthöfer, E. Zangrando, G. Ceruti, F. Tudisco, and Kusch. GeoLoRA: Geometric integration for parameter efficient fine-tuning. In The Thirteenth International Conference Learning Representations, 2025.
[38] S. Schotthöfer, E. Zangrando, K. Jonas, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processessing Systems, 2022.
[39] L. Schwinn, Raab, A. Nguyen, D. Zanca, and B. Eskofier. Exploring misclassifications of robust neural networks to enhance adversarial attacks. Applied Intelligence, 53((7):19843- 19859, 2023.
[40] V. Sehwag, S. Wang, P. Mittal, and S. Jana. Hydra: Pruning adversarially robust neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19655-19666. Curran Associates, Inc., 2020.
[41] R. Sharma, M. Gupta, and G. Kapoor. Some better bounds on the variance with applications. Journal of Mathematical Inequalities, 4(3):355-366 2010.
[42] P. Singh, G. Bachmann, and Hofmann. Analytic insights into structure and rank of neural network Hessian maps. In Advances in Neural Information Processing Systems, volume 34, 2021.
[43] Y. Su, G. Zhang, S. Mei, J. Lian, Y. Wang, and S. Wan. Reconstruction-assee assisted and distance- optimized adversarial training: A defense framework for remote sensing scene classification. IEEE Transactions on Geoscience and Remote Sensing, 61:1-13, 2023.
[44] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint
2017.
[45] L. N. Trefethen and D. Bau. Numerical Linear Algebra. SIAM, Philadelphia, PA, 1997.
[46] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:22 10.00558. 2022.
[47] H. Wang, S. Agarwal, and D. Papailiopoulos. Pufferfish: Communication-efficcn models at no extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 2021.
[48] D. Xie, J. Xiong, and S. Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6176-6185, 2017.
[49] Y. Xu and P. Ghamisi. Universal adversarial examples in remote sensing: Methodology and benchmark. IEEE Transactions Geoscience and Remote Sensing, 60:1-15, 2022.
13

---

[50] Y. Xu and P. Ghamisi. Universal adversarial examples in remote sensing: Methodology and benchmark. IEEE Trans. Geos. Remote Sens., 60:1-15, 2022.
[51] H. Yang, M. Tang, W. Wen, F. Yan, D. Hu, A. Li, H. Li, and Y. Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 678-679, 2020.
[52] Y. Yang and S. Newsam. tng-oo-visual-wonee and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, GIS '10, page 270-279, New York, NY, USA, 2010. Association for Computing Machinery.
[53] E. Zangrando, S. Schotthöfer, G. Ceruti, J. Kusch, and F. Tudisco. Rank-adaptive spectral pruning of convolutional layers during training. In Advances in Neural Information Processing Systems, 2024.
[54] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning. pages 7472-7482, PMLR, 2019.
[55] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. AdaLoRA: Adaptive budget allocation for parameter-efficiet fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
[56] Zhao, Z. Zhang. B. Chen, Z. Wang. Anandkumar, and Y. Tian. GaLore: Memory-efficiett LLM training by gradient low-rank projection. In International Conference on Machine Learning, pages 6112--114 PMLR, 2024.
[57] Q Zhao, Königl, and C. Wressnegger. Non-uniform adversarially robust pruning. In I. Guyon, M. Lindauer, M. van der Schaar, F. Hutter, and R. Garnett, editors, Proceedings of the First International Conference on Automated Machine Learning, volume 188 of Proceedings of Machine Learning Research, pages 1/1-16. PMLR, 25-27 Jul 2022.
14

---

A Additional Numerical Results
A.1
UCM Dataset
The numerical results for the whitebox 2-FFGMM Jitter, and Mixup adversarial attacks on the VGGI6 and VGG11 architectures can be found in Figure Figure and Figure The regularizer confidently increases the adversarial validation accuracy of the networks.
In Table 10 observe that the regularizer R(W) applied to the full weight matrices (and flattened tensors) W in baseline format is able to increase the adversarial robustness of the baseline network in the UCM/VGG16 test case. However, the increased adversarial robustness comes at the expense of some the clean validation accuracy.
A.2 Cifar10 O Dataset
We the same experiment in Table but with the PCDGGD attack, which is iterative version of (2FFGSM with an random perturbation of the input image as the initial condition [30] Overall, that RobustDLRT is competitive with the other robustness-improving methods when the compression rate is taken into account.
Table 7: Comparison to literature CIFARIO with VGGI6 under the PPD attack. The first three rows list the computed mean over 10 random initializations. The values of all other methods, given below the double rule, are taken from 33 Table 5J. RobusIDLRT has competitive adversarial
accuracy to all methods with a compression rate 2 80%.
(-PGD, e
Method
c.r. [%]
0.0
0.1
0.13
0.16
0.2
0,23
0.26
0.3
RobustDLRT 3 0.15
94.18
88.80
62.58
53.47
44.95
34.75
28.33
22.64
16.59
DLRT
94.53
88.58
59.34
50.06
41.50
31.82
25.67
20.48
15.04
Baseline
0
90.48
63.01
54.66
47.87
40.77
36.75
33.51
29.93
Cayley SGD 225
0
89.62
67.68
59.38
51.09
40.87
34.46
29.21
23.62
Projected SGD
0
89.70
67.64
59.25
51.06
40.86
34.51
29.19
23.64
CondLR 35
T 0.1
50
90.93
67.03
62.08
59.15
56.92
55.96
55.28
54.58
CondLR
35
T = 0.5
50
89.97
64.84
60.25
57.75
56.03
55.21
54.75
54.25
CondLR
35
r 0.1
80
90.48
61.00
50.84
42.19
33.70
29.44
26.55
23.97
CondLR
35
0.5
80
89.33
57.45
46.35
37.20
28.30
23.82
20.65
17.84
LoRA 17
50
89.97
55.74
45.11
36.86
29.62
26.28
24.02
21.84
LoRA
80
88.10
51.40
39.70
30.12
20.97
16.29
13.15
10.37
SVD prune
51
50
89.92
54.87
43.85
35,23
27.95
24.38
22.06
19.94
SVD prune 5
80
87.99
50.64
39.06
29.57
20.16
15.49
12.22
9.57
B Details to the numerical experiments of this work
B.1 Recap of adversarial attacks
In the following we provide the defintions the used adversarial attacks. We use the implementation of (50) for the &² --FSMM Jitter, and Mixup attack. For the el L'-FGSM attack, we the implementation of httpps://g/ hgt..b.tthhttte/gg com/ COMMPLLELb/// /ConnLRR
B.1.1 P-FGSM attack
The Fast Gradient Sign Method (FGSM)[II single-step adversarial attack that perturbs an input in the direction of the gradient of the loss with respect to the input. Given neural network classifier So with parameters 0, an input v, and corresponding label y, the attack optimizes the cross-entropy loss Lce(fo fo (z), y) by modifying along the gradient's sign. The adversarial example is computed
15

---

as:
s' =
(12)
Ccece
where controls the perturbation magnitude. difference r' z is clamped by an bound, i.e.,
To ensure the perturbation remains bounded, the
s' + max(-e, min(1' I, e))
(13)
In this work we fix = 6. The attack can be iterated to increase its strength.
B.1.2 FFGSM attack
The e -FGM attack 44
is used in the reference work of 35 and uses the same workflow as (B.1.1)
where 12 is changed to
x' = sign V (C
(14)
where Σ denotes the standard deviation of the data-pont Mata-points in the training data-set and the sign of the gradient matrix is taken element wise.
B.1.3 Jitter attack
The Jitter attack 399 is an adversarial attack that perturbs an input by modifying the softmax- normalized output the model with random noise before computing the loss. Given a neural network classifier fa with parameters 0, an input æ, and its corresponding label y, the attack first computes the network output fe(x) and normalizes it using the ex norm:
Softmax 8-2
(15)
|=
where 8 is a scaling factor. A random noise term (N0, is added to 2, i.e.,
2 8 + 77.
(16)
The attack loss function is mean squared error between perturbed input and target, given by
C = || 31
(17)
The adversarial example is then computed using the gradient of L with respect
V.C
z'=rta
(18)
To ensure the perturbation remains bounded, the modification x' - x is clamped within an c bound: x' = x + max(-c, min(1' - xx x))
(19)
In this work, we fix a and set = 0.1. The Jitter attack can be performed iteratively. Then, for each but the first iteration k, the attack loss is normalized by the perturbation of the input image,
k>0
(20)
| x 1
In this work, 5 iterations of the Jitter attack for each image.
B.1.4 Mixup attack
The Mixup attack [49] an adversarial attack that generates adversarial samples that share similar feature representations with given virtual example. Inspired by the Mixup data augmentation technique, this attack aims to create adversarial examples that maintain characteristics original sample and its adversarial counterpart. Given neural network classifier fo with parameters 8, an input x, and its corresponding label y, the attack first computes a linear combination of cross-entropy and negative KL-divergence loss,
both the
Emixup
3
Σ km
LcB
So
I. CKL
3/
(21)
25
16

---

Table 8: Training hyperparameters for the UCM, Cifar10, and ImageNet Benchmarks. The first set hyperparameters apply to both DLRT and baseline training, and we train DLRT with the same hyperparameters as the full-rank baseline models. The second set of hyper-parametea DLRT. The DIRT hyperparameters are selected by initial parameter sweep. We choose the DLRT
specific to
truncation tolerance relative the Frobenius norm of S, i... 0 -||S|| |r suggested in [38]
Hyperparameter
VGG16
VGG11
ViT16b
VIT32I
Batch Size (UCM)
16
16
16
Batch Size (Cifar10)
128
128
128
n.a.
Batch Size (ImageNet)
256
Learning Rate Number of Epochs
0.001
0.001
0.001
0.001
20
20
5
10
1.2 regularization
0
0.001
0.0001
Optimizer
Adam W
A
Adam
AdamW
DLRT rel. truncation tolerance Coefficient Steps 8. Initial Rank
0.1
0.05
0.08
0.013
10
10
10
75
150
150
150
200
Parameters
138M
132M
86M
304M
V (J
5 =
(22)
Equation (1 features a scale invariance attack applied to the loss 27 Section 3.3].
The final adversarial example is computed as convex combination of the original input and its perturbed version:
x' = Aa (1 A) + 8),
(23)
where À Beta(B, (3) is sampled from Beta distribution with hyperparameter 3, controlling the interpolation between clean and perturbed inputs. The perturbation is further constrained within an e-ball ensure bounded adversarial modifications:
z' = + max( e,, min(x' - -x, xx))).
(24)
In this work, we fix and set 8 10 The attack can be iterated to increase its effectiveness, refining the adversarial perturbation at each step. We use 5 iterations of the Mixup Attack for each image.
B.2 Network architecture and training details
In this paper, we use the pytorch implementation and take pretrained weights from the imagenetl kk dataset as initialization. The data-loaded randomly samples a batch for each batch-update which the only source of randomness in our training setup. Below an overview of the used network architectures
VGG16 is deep convolutional neural network architecture that consists of 16 layers, including 13 convolutional layers and 3 fully connected layers.
VGG11 a convolutional neural network architecture similar to VGG16 but with fewer layers, consisting of 11 layers: convolutional layers and 3 fully connected layers. It follows the same design principle as VGG16, using small 3x3 convolution filters and 2x2 max- max-pooling layers. ViT16b is Vision Transformer with 16x16 patch size, deep learning architecture that leverages transformer models for image classification tasks.
VIT32I is Vision Transformer with 32x32 patch size, deep learning architecture that leverages transformer models for image classification tasks. We use the Imagenet2 lk weights from the huggingface endpoint boogleevit--ngg--acchh----2-2
weight initialization.
The full training setup described in Table g We train DLRT with the same hyperparameters the full-rank baseline models. It is known 37 that DLRT methods are robust w.r.t. common
17

---

Table Overview of the for best performing regularization strength for RobustDLRT of Table
UCM Dataset
Cifarl0
Dataset
ImageNet Dataset
Architecture
FGSM
Jitter
Mixup
FGSM
Jitter
Mixup
FGSM
Jitter Mixup
VGG16
0.075
0.2
0.15
0.05
0.05
0.05
n.a.
n.a.
n.a.
VGGI1
0.1
0.05
0.15
0.15
0.05
0.2
n.a.
n.a.
n.a.
ViT16b
0.1
0.15
0.15
0.01
0.01
0.05
n.a.
n.a.
n.a.
ViT32I
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
0.075
0.075
0.075
Table 10: UCM Data, VGG16, baseline training. Data is averaged over 10 stochastic training runs. The regularizer is able to increase the adversarial robustness of the baseline training network, at the cost of some reduction of its clean validation accuracy. The provided results averaged over iterations.
Acc [%]] under the 22-FGSM attack with
B
0
0.01
0.025
0.05
0.075
0.1
0.2
0.3
0.4
0.5
0
92.40
91.72
90.65
86.71
81.32
76.40
64.52
54.96
49.38
45.14
0.0001
91.69
91.69
91.10
87.73
83.14
78.43
63.21
53.31
47.18
42.99
0.001
88.81
88.78
87.90
84.40
80.00
76.34
62.61
53.77
48.09
44.38
0.01
88.22
88.19
87.12
82.78
77.52
72.72
58.32
48.89
42.83
38.61
0.05
90.45
90.43
89.63
87.23
84.11
80.55
68.66
59.29
52.62
46.61
0.1
92.51
92.51
92.11
90.45
88.43
86.32
76.91
68.01
61.29
55.52
0.2
89.20
89.18
88.85
86.66
84.36
81.96
73.25
65.20
58.61
53.29
hyperparameters learning rate, and batch-size, and initial rank. The truncation tolerance 7 is chosen between 0.05 and 0.1 per an initial parameter study. These values are good default values, as per recent literature [36 42 In general, there is trade-off between target compression ratio and accuracy, as illustrated e.g. in 38 for matrix-valuee and 42 for tensor-valued (CNN) layers.
B.3 UCM Test Case
The University of California, Merced (UCM) Land Use Dataset is a benchmark dataset in remote sensing and computer vision, introduced [52]. comprises 2,100 high-resolution aerial RGB images, each measuring 256x256 pixels, categorized into 21 land use classes with 100 images per class. The images were manually extracted from the USGS National Map Urban Area Imagery collection, covering various urban areas across the United States. The dataset contains images with spatial resolution approximately 0.3 meters per pixel (cquivalent to foot), providing detailed visual information suitable for fine-grinne scene classification tasks.
We normalize the training and validation data with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] for the rgb image channels. The convolutional neural neural networks used in this work are applied the original 256 x 256 image size. The vision transformer pinpllee resizes the image to resolution of 224 x 224 pixels. The adversarial attacks for this dataset are performed on the resized images.
B.4
Cifar10
The Cifar10 dataset consists of classes, with a total of 60000 rgb images with a resolution of 32 32 pixels.
We use standard data augmentation techniques. That is, for CIFAR10, we augment the train- ing data set by a random horizontal flip of the image, followed by a normalization using mean [0.4914, 0.4822, 0.4465) and std. dev. [0.2440,, 0.2435, 0.2616). The test data set only normalized. The convolutional neural neural networks used in this work are applied to the original 32 x 32 image size. The vision transformer data-pipeline resizes the image a resolution of 224 224 pixels. The adversarial attacks for this dataset are performed the resized images.
18

---

Low-Rank VGG16, FGSM whitebox attack
Full-Rank
92..40
91.72
90.65
86.71
81.32
76.40
64.52
54.96
45.14
90
93.92
93.86
93.00
87.95
80.14
72 41
52.12
43.39
39.07
35.81
95.30
80
0,001
94.03
93.94
93.04
88.70
81.51
74.10
53.70
43.94
38.04
34.26
95.51
Regularzation (8)
70
(96)
93.64
94.36
93.83
90.64
85.20
78.76
54.55
43.48
37.30
33.68
95.42
60
-50
0.05
93.01
92.78
92.25
88.83
84.23
78.92
61.16
51.46
44.67
40.03
95.88
-40
0.075
92.61
92.83
92.27
89.12
84.10
78.68
62.83
53.30
47.46
43.08
95.84
30
0.0
0.01
0.025
0.05
0.075
0.1
0.2
0.3
0.4
0.5
c.r. [%)
Attack (e)
Figure 4: UCM Dataset, VGG16 clean and adversarial accuracy under the FGSM attack. Data is averaged over 10 stochastic training runs. The top row displays the full baseline network with 0% c.r. and the matrix below displays the low-rank and regularized networks trained with Algorithm [ All numbers display the mean of 10 randomized training runs, where the randomness stems from shuffled batches. The initial condition of all runs is given by Imagenet- Ik pretrained weights. The regularized low-rank networks with 0.075 are able to recover the adversarial robustness baseline training while compressed by 95.84%. Results for VGG11 and Vbt16b are similar.
B.5 ImageNet-1k
The ImageNet dataset consists of 1000 classes and over 1.2 million RGB training images, with standard resolution of 224 224 pixels. We follow the standard data augmentation pipeline for ImageNet, which includes random resized crop to 224 x 224, and normalization using mean [0.5, 0.5, 0.5] and standard deviation [0.5, 0.5, 0.5). The test set is only resized and center-cropped to 224 x 224, followed by normalization. Adversarial attacks are generated on the normalized, resized images.
B.6 Computational hardware
All experiments in this paper are computed using workstation GPUs. Each training run used a single GPU. Specifically, we have used NVIDIA RTX A6000, NVIDIA RTX 4090, and 8 NVIDIA A-100 80G.
The estimated time for experimental depends mainly the data-set size and neural network architecture. For training, generation of adversarial examples and validation testing we estimate 30 minutes one GPU for one
C Proofs
To facilitate the proofs, we remark the definition L-continuity: A function f(x) is Lipschitz continuous on domain D if there exists constant L 2 0 such that for all x, D.
[f(()) f(y)l $ L| - y/1-
The smallest such L is called the Lipschitz constant.
19

---

Low-Rank VGG16, Jitter whitebox attack
0.0 Full-Rank
95
94.42
93.41
90.80
89.58
87.83
85.05
0.00
93.19
92.15
87.92
83.90 99
79.31
67.41
95.30
93.97
93.10
90.55
88.46
85.68
78.49
95.51
Regularzatin (8)
80
(%6
92.34
91.58
89.41
87.61
85.44
79.31
95.42
75
0.15
92.55
91.89
89.90
88.06
85.82
79.63
95.88
0.2
92.59
91.96
89.87
88.33
86.08
79.81
95.84
60
0.0
0.025
0.03
0.035
0.04
0.045
c.r. [%)
Attack (e)
Figure 5: UCM Dataset, VGG16 clean and adversarial accuracy under the Jitter attack. Data is averaged over 10 stochastic training runs. The top row displays the full baseline network with 0% c.r. and the matrix below displays the low-rank and regularized networks trained with Algorithm [ All numbers display the mean of 10 randomized training runs, where the randomness stems from shuffled batches. The initial condition of all runs is given by Imagenet- Ik pretrained weights. The regularized low-rank networks are able to recover most the adversarial robustness of the baseline network. Results for VGG11 I and Vitl 6b are similar.
For the following proofs, let
(A, B) trace((" A) Σ Aij
be the Frobenius inner product that induces the norm ||||| = v (A, By the cyclic property the trace, we have
(AB,C)] (B, (B,CDA)) (C A D).
(25)
matrices A, B. C, and D appropriate size.
Proof of
We calculate
R(S)² = - 0ss)
||STS|2 (8 S, + (T
||ST ||SS
Σ TS( - Σ si(S)2
(26)
s((S)2) 2 1=1
)
s((ST))2
Since S is symmetric positive semi-definite, <i(ST)) S.( Applying this substitution yields 3 The proof is complete.
20

---

Low-Rank VGG16, Mixup whitebox attack
Full-Rank
93.82
71.77
77.77
52.85
45.50
37.25
27.11
23.91
23.05
0.00
93.79
85.79
85.79
59.38
50.34
40.42 40.42
26.70
21.62
20.13
95.30
93.56
88.57
88.57
68.86
60.27
48.19
31.81
26.74
25.58
95.51
Regularzation (8)
70
Accuracy (%6
93.60
90.24
90.21
78.48
72.90
63.67
46.76
39.57
37.74
95.42
90.57
0.15
92.14
90.33
90.33
82.24
77.62
70.12
55.38
48.52
47.31
95.88
0.2
91.96
88.62
88.62
78.21
74.30
66.42
53.09
46.63
44.88
95.84
0.0
0.01
0.025
0.05
0.075
0.1
0.25
0.5
0.75
c.r[[%]
Attack (e)
Figure 6: UCM Dataset, VGG16 clean and adversarial accuracy under the Mixup attack. Data is averaged over 10 stochastic training runs. The top row displays the full baseline network with 0% c.r. and the matrix below displays the low-rank and regularized networks trained with Algorithm [ All numbers display the mean of 10 randomized training runs, where the randomness stems from shuffled batches. The initial condition of all runs is given by Imagenet- Ik pretrained weights. The regularized low-rank networks almost double the adversarial accuracy the baseline network at 95.84% compression rate. Results for VGG11 and Vit are similar.
Proof of Proposition 7 Given S E R"xr, the Fréchet derivative for = R² at S is linear operator + [Q(S) for Rrxr, Denote Ws S a which is symmetric. Since an inner product, we calculate [O(S) as
VQ(S) (Ws, Z5 ())) Z),
(Wss (Ws.5 (WS,))(ss.1)
(27)
(SW3, (SW -
2 ((S,2)(Ws.1)
-
Note by definition of az.
(S
= - -
(28)
Hence
VQ(S) = - aZ
(29)
Since R² = Q, therefore
VO(S) VR(S)=
(30)
2R(S)
The desired estimate follows. The proof is complete.
L
Proof of Proposition
From 26 there holds
( Si(S (iSTs)
R(S)²
(SST))
(31)
21

---

From 31. RS)2 is the variance of the sequence {s(ST)) The Von Szokefalvi Nagy
inequality [33 bounds the variance of a finite sequence of numbers below by the range of the
sequence (see 41 Applied to (31). this yields
R(S)2 2 (51(ST)) (T(
(5) - (5)())
(32)
2r
2r
Hence
(2R(S) 2 s1(5)² sr
(33)
An application of the Mean Value Theorem for logarithms (see 34 Proof Theorem 2.2]), gives
In(x(S)) - (I(S)² (8)
(34)
(5)
Combining 33 and (34) yields
In(x(S)) S (S)2 R(S),
(35)
which, after exponentiation, yields The proof is complete.
Proof of Proposition Since W is constant, we rewrite the dynamical system $ + SVR(S) +
W as
dr 36) by S W and rearranging yields
(s - W) + BVR(S) + (S - W) =
(36)
Testing
1
de || w|| +V) ||
B(VR()),
(37)
We calculate (VR(S). Note
- -
= -S
(38)
=IS =|s a3 (I, ||ST 2 |||s|| R(S)3
where the last equality is due to 26) Hence
(R(()) -) 2R(S).
(39)
R(S)
Using Hölder's inequality, the sub- uub-multiplicative property of) ||. and Young's inequality, we bound the right hand side (37) by
||SSSS - az
8(VR(S),W) s 28
| - S|][
R(S)
(40)
2 W ||]] < || w 23(1 + 2
Applying and 40
339
to
36
we obtain
|S 2 2 + 21 - 2 < 23 + 1/
(41)
2 at
An application Grönwall's 's inequality on [0. t] yields
/' *T
(42)
The proof is complete.
22

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA]
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked use checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] is perfectly acceptable to answer "[No] providee proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
.
Do not modify the questions and only use the provided macros for your answers.
1.
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
See contribution paragraph in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
23

---

We clearly discuss the assumptions of our propositions and discuss suitable applications of our method. Further, we point out in which applications the method is not suitable. We end the paper with conclusion section that reflects back on the proposed scope of the paper.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We state the global assumptions in the beginning of sections and 4, and clearly state all local assumptions of the propositions. The proofs do not use additional assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
24

---

Answer: [Yes]
Justification: We provide a full deseription of all used datasets, and neural network architec- ture details as well as the origin of pretrained weights. Furthermore, provide all training details and hyperparemeters that have been selected by our preliminary hyperparameter search. Detailed algorithmic descriptions allow the reader implement method based on the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [TODO]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips http:c/cwc..eeeeeeeee///e/aaaaaa
for more details.
25

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide full description of all used datasets, and neural network architec- ture details as well as the origin of pretrained weights. Furthermore, we provide all training details and hyperparemeters that have been selected by our preliminary hy hyperparameter search. Detailed algorithmic descriptions allow the reader to implement our method based on the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All numbers in the result tables of this paper, with the exception of Table IO present the mean over 10 stochastic training runs with the prescribed hyper-parameterr for the respective test cases. Table 10 presents results with 5 stochastic training runs. The lineplots in this paper show the metrics of the median run of 10 training runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
26

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computational hardware and experiment timing estimates are reported in Appendix B.6
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conforms with the conduct of Ethics and we have no reason to believe otherwise.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We point out the societal impact in the conclusion section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
27

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper is methodological research paper and we do not release certain data and model with potential risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The used data, models, and code are open source and properly credited.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpppsai thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
28

---

Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of L.MMs if is an important, original, or non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve L.MMs as any important, original, or non-standard components
29

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
30


Reviews:
[{"review_id": "Z4aX0fPwF2", "reviewer": "Reviewer_LKg2", "summary": "This paper proposes a training framework that considers compression, accuracy, and robustness. By controlling the number of conditions of the low-rank matrix spectrum,  compression and adversarial robustness are achieved during the training process. Extensive experiments have shown some improvement compared with some existing algorithms.", "questions": "Some small things: The authors should highlight the best performance in Table 2 to increase the readability of the results.\n\nIf the authors can solve some of my confusion in experiments, the practical significance of the paper will be greatly improved, and I will be happy to increase my score.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. I appreciate the theoretical part of this paper, which theoretically demonstrated that the regular term can control the spectral distribution, which in turn affects the robustness.\n2. The description of regular terms and algorithms is clear.\n3. RobustDLRT outperforms Cayley SGD, Projected SGD, CondLR, LoRA, SVD prune under $l_1$ FSGM attack.\n\nWeaknesses\n1. I have questions about the motivation of the paper, which claims that compression is contradictory to robustness, but a large body of work shows that compression improves robustness[1,2,3,4,5,6]. Thus, I think the authors should rethink their motivation and explain the difference from these existing works.\n2. I am wondering about the RobustDLRT's performance under pgd attack, which is stronger than FSGM. Can the authors provide the corresponding results?\n3. I realize that low-rank decomposition compression is not the same as traditional pruning, and that a lot of existing work is based on pruning. Are there any other benefits of low-rank over pruning, besides the possibility of hardware acceleration? For example, is it possible for RobustDLRT to use fewer FLOPs? Improvements to existing DLRTs limit the contribution of this paper, it would be nice to provide comparisons to the pruning-based works, especially if some experimental results can be used to demonstrate the necessity of RobustDLRT. (If as a reader who does not know much about low rank compression, one would certainly be curious about how such a compression differs from the others (e.g. pruning quantizaiton), even if the authors don't provide experimental comparisons, and simply add some discussion to illustrate the differences, I would still believe this weakness has been addressed.)\n\n[1] https://arxiv.org/abs/2210.04311\n\n[2] https://arxiv.org/abs/2206.07311\n\n[3] https://proceedings.neurips.cc/paper/2020/file/e3a72c791a69f87b05ea7742e04430ed-Paper.pdf\n\n[4] https://proceedings.mlr.press/v188/zhao22a/zhao22a.pdf\n\n[5] https://arxiv.org/abs/2206.07839\n\n[6]https://openaccess.thecvf.com/content/ICCV2021W/AROW/papers/Jordao_On_the_Effect_of_Pruning_on_Adversarial_Robustness_ICCVW_2021_paper.pdf", "was_revised": true, "final_justification": "The authors have addressed all of my concerns. I think those improvements increase the value of the paper, and I have changed the rating from 2 to 4.", "rebuttal": "We thank the reviewer for the constructive feedback and appreciate that they find the theoretical analysis and algorithmic description of the method clear. \n\n1. (Q about motivation) The cited papers [1,2,3,4,5,6] state - just like us - that adversarial robustness correlates with the parameter count of an otherwise untreated, pre-trained neural network. To the best of our understanding, the cited works [1-5] investigate the effects of sparsity pruning on models with robust defenses in mind, i.e., adversarial training.\nThe last reference [6] gives empirical evidence that a pretrained and then pruned network has increased adversarial robustness.\n\n    We state in line 30 \"Moreover, it has been observed that compressed networks can exhibit increased sensitivity to adversarial attacks [29]\", where [29] analyzes the degrading effect of raw low-rank compression on robustness, which we confirm in, e.g., Table 2, and explain in Figure 1.\n   However, we do not make an explicit statement about the direct effects of sparsity postprocessing, i.e. pruning, on the adversarial robustness of neural networks. \n\n    We acknowledge however the lack of clarity of the statement and will change it to \"Moreover, it has been observed that *low-rank* compressed networks can exhibit increased sensitivity to adversarial attacks [29]\"\n\n2. (Q about PDG attack)  In Table 2, we evaluate the baseline, DLRT, and RobustDLRT models with the FGSM, Jitter, and Mixup adversarial attacks.  Jitter and Mixup, described with references in Sections A.1.3 and A.1.4 respectively, are both iterative and gradient-based adversarial attacks that are quite strong.\n\n    To address the reviewers concerns, we also ran the same experiment in Table 3 but with the $\\ell^2$-PGD attack.  Overall, we see that robustDLRT is competitive with the other robustness-improving methods when the compression rate is factored in.\n\n    The first three rows list the computed mean over 10 random initializations.  \n    The values of all other methods, given below the double rule, are taken from [Savostianova 2023; Table 5].  We remark that RobustDLRT outperforms CondLR with τ=0.1 for $\\epsilon$<=0.2, and  CondLR with τ=0.5 for $\\epsilon$<=0.26. \n\n    We wish to remark that opposed to the experiments in Table2, we do not sweep over β but take the value that is obtained from the $l_1$ FGSM results of Table 2. \n\n**RobustDLRT** has competitive adversarial accuracy to all methods with a compression rate ≥ 80%.\n\n| Method                  | c.r. [%] | 0.0   | 0.1   | 0.13  | 0.16  | 0.2   | 0.23  | 0.26  | 0.3   |\n|-------------------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|\n| **RobustDLRT β=0.15**   | 94.18    | 88.80 | 62.58 | 53.47 | 44.95 | 34.75 | 28.33 | 22.64 | 16.59 |\n| DLRT                    | 94.53    | 88.58 | 59.34 | 50.06 | 41.50 | 31.82 | 25.67 | 20.48 | 15.04 |\n| Baseline                | 0        | 90.48 | 63.01 | 54.66 | 47.87 | 40.77 | 36.75 | 33.51 | 29.93 |\n|-------------------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|\n|-------------------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|\n| Cayley SGD              | 0        | 89.62 | 67.68 | 59.38 | 51.09 | 40.87 | 34.46 | 29.21 | 23.62 |\n| Projected SGD           | 0        | 89.70 | 67.64 | 59.25 | 51.06 | 40.86 | 34.51 | 29.19 | 23.64 |\n|-------------------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|\n| CondLR τ=0.1            | 80       | 90.48 | 61.00 | 50.84 | 42.19 | 33.70 | 29.44 | 26.55 | 23.97 |\n| CondLR τ=0.5            | 80       | 89.33 | 57.45 | 46.35 | 37.20 | 28.30 | 23.82 | 20.65 | 17.84 |\n|-------------------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|\n| LoRA                    | 80       | 88.10 | 51.40 | 39.70 | 30.12 | 20.97 | 16.29 | 13.15 | 10.37 |\n|-------------------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|\n| SVD prune               | 80       | 87.99 | 50.64 | 39.06 | 29.57 | 20.16 | 15.49 | 12.22 | 9.57  |\n\n3. (Q about sparsity vs low-rank) Pruning[1-6], i.e. sparsity postprocessing, of pretrained [6] or pretrained on adversarial examples [1] networks can maintain adversarial robustness. We remark that this requires the computational expense of (a) full network training, (b) sparsification and (c) finetunning.\n\n    The distinguishing factor of our work is that RobustDLRT is able to adaptively compress and influence the spectrum of a neural network during training.   Moreover, direct access to the spectrum during training is computationally viable when a low-rank factorization of the layer weights is used.\n    This puts the contributed work in the beneficial zone of low compute/memory cost during training and inference and high clean accuracy and high adversarial accuracy/robustness; these properties required for the target application of resource constrained edge devices. \n\n    You are correct in your assessment that on many hardware architectures, low-rank based compression increases throughput compared to sparsity compression formats. \n\n    We include a clarifying statement in the related work section of the revised manuscript to disambiguate the scope of this work from sparsity pruning scenarios, where (to the best of our knowledge) the full-model pretraining is treated as an offline cost. \n\n\n4. We will highlight all runs where RobustDLRT outperforms the full-rank baseline model.  We remark that these runs are surpassing the theoretical guarantees of DLRT of achievieng close-to-baseline performance, see [a]. In all runs the regularized method outperforms the unregularized low-rank baseline \"DLRT\".\n\n    [a] Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations; Schotthöfer et al. 2022 NeurIPS"}, {"review_id": "E3urdnLJ8R", "reviewer": "Reviewer_63AS", "summary": "This is an intriguing paper. It seems like a reasonable way to encourage low rank structure plus numerical stability during training of deep neural networks. There are posthoc compression/low-rank methods (i.e., compute the low-rank approximation after training), but this line of work in encouraging the low-rank + (near)-orthogonality of U,V through these gradient flow equations is a really interesting idea.", "questions": "Please see the other comment boxes for questions.", "limitations": "My main question regarding limitations is about the scalability of this training procedure to larger datasets. The main experimental results are on 2 datasets: CIFAR10 (60K images) and UCM (2K images), which are considered \"small\" nowadays; this leads me to ask how scalable this training method really is.  The authors discuss the computational cost as compared to LoRA in lines 256-262, but a clearer picture is warranted regarding how much overhead this type of low-rank regularization during training costs over standard training methods. Some context of this extra cost would be useful.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper has some good strengths and some issues that need to be clarified.  They are discussed more carefully in the other comments.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive review. We appreciate that they found our paper intriguing and the robust dynamical low-rank training idea interesting.\n\n\n1. (Q scalability) To eleviate the concerns of scalability we provide numerical results for the ImageNet-1k (1.2 Million images), where we compress the ViT-L32 vision transformer (304 Million parameters). We first conduct a hyperparameter sweep to determine standard hyperparameters as learning rate, weight-decay and batch size, which we report in Appendix C. \n\n    We compare then the baseline model, DLRT and the regularized RobustDLRT adversarial performance in the table below and observe similar trends as in the Vision Transformer experiments of Table 2 in the main manuscript. \n\n\n**Table: ImageNet, Top-1 Accuracy under various attacks**\n\n| Method      | c.r. [%] | Clean Acc. | FGSM ε=0.05 | FGSM ε=0.1 | FGSM ε=0.3 | Jitter ε=0.035 | Jitter ε=0.045 |\n|-------------|-----------|-------------|-------------|-------------|-------------|------------------|------------------|\n| Baseline    | 0         | 74.37       | 43.58       | 31.42       | 16.03       | 43.09           | 35.57           |\n| DLRT        | 58.02     | 72.27       | 42.70       | 30.32       | 15.47       | 43.98           | 38.44           |\n| RobustDLRT  | 57.98     | 72.25       | 43.17       | 35.11       | 25.24       | 48.22           | 43.51           |\n\n**Table: ImageNet, Top-5 Accuracy under various attacks**\n\n| Method                  | c.r. [%] | Clean Acc. | FGSM ε=0.05 | FGSM ε=0.1 | FGSM ε=0.3 | Jitter ε=0.035 | Jitter ε=0.045 |\n|-------------------------|-----------|-------------|-------------|-------------|-------------|------------------|------------------|\n| Baseline                | 0         | 92.20       | 73.75       | 63.42       | 43.41       | 78.24           | 74.96           |\n| DLRT                    | 58.02     | 90.06       | 70.43       | 60.90       | 40.58       | 74.49           | 71.31           |\n| RobustDLRT              | 57.98     | 90.03       | 71.58       | 62.82       | 50.65       | 77.35           | 75.14           |\n\nWe have measured the runtime of training one epoch on ImageNet of both DLRT and RobustDLRT on an A100 80GB. \nDLRT takes 26min 07 sec and RobustDLRT (with the regularizer) 27min 51 sec, so the overhead is roughly 3%. We remark that this overhead can be further reduced with an optimized implementation. We conclude that the method is scalable.\n\nFinally we wish to remark that the slightly lower compression rate is expected since the hidden dimensions of ViT-L32 of 1024 is close to the number of ImageNet classes (1000), thus there is less redundancy in the model compared to other reported benchmarks. \n\n2.  We highlighted the best performing method in each setup of Table 2 to improve readability in the revised manuscript.\n\n3.  We changed $\\hat{\\mathcal{L}}$ to $\\tilde{\\mathcal{L}} $ in Fig.2 in the revised manuscript."}, {"review_id": "1dbBDmOjGE", "reviewer": "Reviewer_HEm3", "summary": "This paper addresses the challenge of preserving adversarial robustness during low-rank model compression. The authors begin with a preliminary analysis showing that the compression process can alter the condition number of the model, potentially impacting robustness. To address this, they introduce a robustness-oriented regularization technique that explicitly controls the condition number of the low-rank decomposition. Both theoretical analysis and empirical evaluations are presented to support the effectiveness of the proposed method.", "questions": "The results in Table 1 show that RobustDLRT can improve the performance under a large perturbation budget, where the baseline performs very badly. I am wondering whether you are using an adversarial trained model or just a normal trained model as the baseline.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength #\n\n* The research topic of this paper is practical. It is always a concern about the trustworthiness of the model during model compression. This paper tries to solve the challenge, which is beneficial to the real model application.\n\n* The proposed method is easy but effective. The proposed method designs a regularization term about the condition number, which will not increase the training cost too much. In this case, the experimental results can outperform DLRT significantly, especially when the attack budget is large.\n\n# Weakness #\n\n* The outline of the paper is strange, making it hard to follow the logic of the paper. For instance, Section 2 shows some preliminary knowledge of low-rank compression. However, the related works are shown in the next section. I suggest to directly to the method part after the preliminary. Moreover, in Section 5, there is an individual subsection 5.1, but with all other contents in the main section.\n\n* Some definitions in the paper are not clear. \n\n-- The Lipschitz continuity should be explained in a formula. \n\n-- In line 78, the definition of the condition number of the activation function is referred to in related works. \n\n-- In equation (6), the definition of \"orth( | )\" is not clear. \n\n* The method part is messy, and it is hard to discriminate which part is the proposed part. Most of the content in Section 5 is similar to FeDLRT [1], including the * Basis Augmentation * and * Latent Space Training *. As I understand, the major difference is the change in the gradient computation after introducing the regularization term. \n\n* The baselines are not clear in the Experimental section. There should be a paragraph to explain the compared methods. I wonder which DLRT is applied for comparison. Besides, I expect FeDLRT is compared since the proposed regularization is applied to it.\n\n* Typo: line 62, the compression of the network\n\nIn general, I expect the authors can reorganize the paper and elaborate on which part is from existing works and which is the innovation part of this paper.\n\n[1] Schotthöfer, Steffen, and M. Paul Laiu. \"Federated dynamical low-rank training with global loss convergence guarantees.\" arXiv preprint arXiv:2406.17887 (2024).", "was_revised": true, "final_justification": "In the rebuttal, the authors solved my major concern about the baseline problem. Therefore, I increase my rating.", "rebuttal": "We thank the reviewer for their constructive review. We appreciate that they found the proposed method practical, efficient and easy to implement. \n\n\n1. (Q Lipschitz definition) We added the definition of Lipschitz continuity in the appendix. For completeness:  A function $ f(x) $ is **Lipschitz continuous** on a domain $ D $ if there exists a constant $ L \\geq 0 $ such that for all $ x, y \\in D $,\n$$\n\\|f(x) - f(y)\\| \\leq L \\|x - y\\|.\n$$\nThe smallest such $ L $ is called the **Lipschitz constant**.\n\n\n2. (Q orth operator) As described in line 235 of the original manuscript We need to check all line numbers before resubmission,   \\( Q=orth( Z )  \\) is defined as the $Q$ factor in the QR-factorization, i.e, $QR=qr(Z)$.\n Additionally, $[A|B]$ for $ A,B\\in\\mathbb{R}^{n\\times r}$ concatenates $A,B$ in the second dimension, i.e. $[A|B]\\in\\mathbb{R}^{n\\times 2r}$.\n\n\n3. (Q method part) We wish to remark that the main contribution is the construction of the Regularizer $\\mathcal{R}$ in Section 4. We state in line 153 that the method is compatible with any low-rank training scheme, e.g. [48, 30, 31, 29], that provides orthonormality of the basis $U,V$. \n \n    To facilitate the implementation we choose the scheme of [30] (\"FeDLRT\") as the backbone scheme. We do not claim invention of FeDLRT here, but incorporate our regularizer into it. The applicability of the regularizer to a wide range of low-rank training methods is an advantage, as a user does not need to change their implementation to use our regularizer. \n\n\n4. (Q Baselines) We denote by \"Baseline\" the default training of the full model (not low-rank factorized) with the training hyperparameters specified in Appendix A2; this is stated in Section 6, Line 293.  DLRT and RobustDLRT use the two-step scheme of [30], which is described in detail in section 5. As denoted in the caption of Table 2,  RobustDLRT with regularization parameter $\\beta=0$ equals DLRT. The same methods are used in all other experiments. All other compared methods are cited.\n\n5. (Q Adversarial training) In all experiments presented in this paper, we train solely on clean data without employing adversarial training. This choice is intentional: the proposed method acts as a regularizer and is conceptually orthogonal to data-driven approaches like adversarial training. To clearly isolate and illustrate the effect of the regularizer, we evaluate it in the unaltered, clean-data training setting.\n\n6. (Typos) We fixed the typos - Thank you for noticing!"}, {"review_id": "9wtBrmCwX1", "reviewer": "Reviewer_Gtfv", "summary": "This paper provides a theortical derivation on the relationship between weight condition number and the model robustness. The derivation leads to a novel regularization on the condition number during the dynamic low-rank training process. The paper claims that the proposed method can mitigate the sensitivity of low-rank decomposed model to adversarial attacks without sacrificing accuracy on clean data.", "questions": "1. How does the impact of compression appear in the derivation of Sec. 4? Given a similar R, does model of different ranks share a similar condition number bound? If that's the case, why compressed model tends to be less robust as shown in the experiment?\n2. As a follow up, experiments in Table 2 should be expanded to include additional results conducted with full rank trained model but regularized with the proposed regularization, so as to show the effectiveness of the regularization against baseline.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. This paper provides a novel perspective of connecting model robustness to weight condition number and makes important observation in Fig 1 on the impact of low-rank training on the condition number\n2. The paper proposes a novel and theortically sound regularization to regularize the condition number during low-rank training process\n\n## Weakness\n1. The paper claims to tackle efficiency and robustness together with the proposed method. However, the derivation in Sec. 4 only focus on the condition number's impact on robustness, but does not covers the impact of compression on the robustness or the impact of the regularization on the model comression rate.\n2. This lack of clarity between the impact of regualrization and compression also appears in the experiment results. For example, Table 2 shows that RobustDLRT, which should have a lower condition number, still suffers from a lower robustness compared to baseline model.\n3. The evaluation method is relatively weak in supporting the claim of the paper. Only weak adversarial attacks like FGSM is applied to show robustness improvement, which does not lead to a significant conclusion. Similarly, experiments are only applied on CIFAR-10, which lacks practical impact.", "was_revised": true, "final_justification": "During the rebuttal, the author resolves the concerns on the interaction between robsutness and compression, the scalability of the proposed method, and effectiveness under adversarial training. I believe the paper has good merits to be accepted to the coference.", "rebuttal": "We thank the reviewer for their constructive feedback.  We appreciate their assessment of the novelty of the method and the  soundness of the theory. \n\n\n1. (Q Cross influence of regularization and compression)   From an analytical perspective, we only target the spectrum to enable a well-conditioned low-rank matrix. \n However, we provide results on the effect of the regularizer on the compression rate in Figure 4,5,7 in the Appendix, which show a full result matrix of RobustDLRT-trained low-rank networks with different regularization strenghts $\\beta$ and their compression rate (and adversarial accuracy under different attacks). It becomes apparent that the compression rate of the networks is very similar and does not deepened on the regularization strength. \n\n     We show the impact of compression on robustness in all Tables in the main manuscript, where it is apparent that the adversarial robustness of non-regularized, compressed networks suffers. It can be recovered by application of the proposed regularizer as we show in multiple test cases, e.g. Table 1-4. This is the main contribution of this work.\n\n\n2. (Q Clarity of regularization vs compression)  We note that the case of Figure 1, where the DLRT spectrum is a widened version of the baseline spectrum and the regularizer straightens the spectrum, is not present across all layers.  In some layers, the spectrum of DLRT and robustDLRT is far larger than the baseline model; this fact could explain the lack of strong correlation between condition number and adversarial accuracy.  As an aside, a portion of the above comment is present in a footnote in the original work, but was not properly referenced in the caption of Figure 1.  \n\n    Additionally, in the cases where the baseline is acceptable already, RobustDLRT shares similar adversarial accuracy. \n\n    We finally comment that creating a robust network will require several defenses combined, and this paper is to demonstrate the capabilities of the proposed method. \n\n3. (Q Evaluation Method)  In Table 2, we evaluate the baseline, DLRT, and RobustDLRT models with the FGSM, Jitter, and Mixup adversarial attacks.  Jitter and Mixup, described with references in Sections A.1.3 and A.1.4 respectively, are both iterative and gradient-based adversarial attacks that are quite strong.  \n    \n    To eleviate the concerns of scalability we provide numerical results for the ImageNet-1k (1.2 Million images), where we compress the ViT-L32 vision transformer (304 Million parameters). We first conduct a hyperparameter sweep to determine standard hyperparameters as learning rate, weight-decay and batch size, which we report in Appendix C. \n\n    We compare then the baseline model, DLRT and the regularized RobustDLRT adversarial performance in the table below and observe similar trends as in the Vision Transformer experiments of Table 2 in the main manuscript. \n\n\n**Table: ImageNet, Top-1 Accuracy under various attacks**\n\n| Method      | c.r. [%] | Clean Acc. | FGSM ε=0.05 | FGSM ε=0.1 | FGSM ε=0.3 | Jitter ε=0.035 | Jitter ε=0.045 |\n|-------------|-----------|-------------|-------------|-------------|-------------|------------------|------------------|\n| Baseline    | 0         | 74.37       | 43.58       | 31.42       | 16.03       | 43.09           | 35.57           |\n| DLRT        | 58.02     | 72.27       | 42.70       | 30.32       | 15.47       | 43.98           | 38.44           |\n| RobustDLRT  | 57.98     | 72.25       | 43.17       | 35.11       | 25.24       | 48.22           | 43.51           |\n\n**Table: ImageNet, Top-5 Accuracy under various attacks**\n\n| Method                  | c.r. [%] | Clean Acc. | FGSM ε=0.05 | FGSM ε=0.1 | FGSM ε=0.3 | Jitter ε=0.035 | Jitter ε=0.045 |\n|-------------------------|-----------|-------------|-------------|-------------|-------------|------------------|------------------|\n| Baseline                | 0         | 92.20       | 73.75       | 63.42       | 43.41       | 78.24           | 74.96           |\n| DLRT                    | 58.02     | 90.06       | 70.43       | 60.90       | 40.58       | 74.49           | 71.31           |\n| RobustDLRT              | 57.98     | 90.03       | 71.58       | 62.82       | 50.65       | 77.35           | 75.14           |\n\n\n\nFinally we wish to remark that the slightly lower compression rate is expected since the hidden dimensions of ViT-L32 of 1024 is close to the number of ImageNet classes (1000), thus there is less redundancy in the model compared to other reported benchmarks. \n\n\n\n4. (Q Impact of the compression in derivation of Sec. 4)  \n    1) The compression of the model is largely achieved through the rank-augmentation and rank-truncation steps of Algorithm 1.  The regularizer influences the spectrum of the latent space map $\\hat{S}$ which affects the truncation criterion  of line 251.   The regularizer may negatively affect compression by raising small singular values that would be truncated if not regularizer were used; however, we find in our experiments that the compression rate is not affected in practice. \n\n        We feel the two follow up questions is addressed in the response to your second question (Q) above.\n\n5. (Q Full rank regularization) We have conducted this experiment in Table 8 in the original manuscript in the Appendix, where we explore the effects on regularization of a full-rank network, were we have observed: \"The regularizer is able to increase the adversarial robustness of the baseline training network, at the\ncost of some reduction of its clean validation accuracy.\"\n\n    We wish to remark, that using $\\mathcal{R}$ on the full-rank weight layer is computationally expensive, and that the regularizer is designed for low-rank networks."}]

Author rebuttal:

Meta-review: The authors investigate how to preserve adversarial robustness during low-rank model compression. An analysis shows that compression process can influence the condition number of the model, potentially impacting robustness. They then introduce a regularization technique that controls the condition number of the low-rank decomposition. They present theoretical analysis and empirical evaluations the proposed method.
Strengths:
- clear theoretical analysis
- good experimental results
- the method is simple and effective

The reviewers pointed to several weaknesses (scalability, unclear parts, inclusion of adversarially trained baselines) that were resolved within the discussion period. In conclusion, an interesting and sound study with good innovation. I propose acceptance.
Decision: Accept (oral)