On the Interaction of Compressibility and Adversarial Robustness
Authors: Melih Barsbey
Antônio H. Ribeiro
Umut Simsekli
Tolga Birdal
Venue: ICLR 2026 Poster (2026)
Keywords: compressibility
compression
adversarial robustness
generalization
safety
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We show that training methods that induce structured compressibility are likely to produce adversarial vulnerability.

Abstract: As demands for resource efficiency and safety in modern neural networks intensify, substantial research effort has gone into model compression and adversarial robustness. Yet despite progress on each in isolation, a systematic understanding of how compressibility shapes robustness remains elusive. In this paper, we develop a principled framework to analyze how different forms of structured compressibility - such as neuron-level and spectral compressibility - affect adversarial robustness. We show that structured compressibility can induce a small number of highly sensitive directions in the representation space, which adversaries can exploit to construct effective perturbations. Our analysis yields a robustness bound that reveals how neuron and spectral compressibility impact $\ell_\infty$ and $\ell_2$ robustness via their effects on the learned representations. Crucially, the vulnerabilities we identify arise irrespective of how compressibility is achieved - whether via regularization, architectural bias, or learning dynamics. Through empirical evaluations across synthetic and realistic tasks, we confirm our theoretical predictions, and further demonstrate that these vulnerabilities persist under adversarial training and transfer learning, and contribute to the emergence of universal adversarial examples. Our findings show a fundamental tension between structured compressibility and robustness and highlight new pathways for designing models that are efficient and safe.

Published as a conference paper at ICLR 2026
ON THE INTERACTION OF COMPRESSIBILITY AND ADVERSARIAL ROBUSTNESS
Melih Barsbey
Antônio H. Ribeiro
Department of Computing
Department of Information Technology
Imperial College London, UK
Uppsala University, Sweden
Umut Şimşekli
Tolga Birdal
INRIA, CNRS, Département d' d'Informatique ENS/ / PSLL France
Department of Computing Imperial College London, UK
ABSTRACT
As demands for resource efficiency and safety in modern neural networks intensify, substantial research effort has gone into model compression and adversarial robustness. Yet despite progress on each in isolation, a systematic understanding of how compressibility shapes robustness remains elusive. In this paper. develop a principled framework to analyze how different forms of structured compressibility such as neuron-level and spectral compressibil- ity affect adversarial robustness. We show that structured compressibility can induce a small number of highly sensitive directions in the representa- tion space, which adversaries can exploit to construct effective perturbations. Our analysis yields robustness bound that reveals how neuron and spectral compressibility impact Eo and l robustness via their effects on the learned representations. Crucially. the vulnerabilities we identify arise irrespective of how compressibility is achieved whether via regularization, architectural bias, or learning dynamics. Through empirical evaluations across synthetic and realistic tasks, we confirm our theoretical predictions, and further demonstrate that these vulnerabilities persist under adversarial training and transfer learning, and contribute to the emergence of universal adversarial examples. Our findings show fundamental tension between structured compressibility and robustness and highlight new pathways for designing models that are efficient and safe.
INTRODUCTION
Machine learning systems are increasingly deployed in safety-critical domains such as healthcare (Ra- jpurkar al.) 2022) and autonomous driving (Hussain & Zeadally 2019). where reliability is paramount. With their growing social impact, modern neural networks are now expected to not only be more resource-efficient, e.g. amenable to compression in favor of reduced memory footprint and latency, but to do so while retaining their safety properties. In this paper, we focus on primary safety concern in adversarial robustness, and investigate how it interacts with compressibility. While both topics have been studied extensively in isolation, a mature and unified understanding of how compressibility shapes adversarial robustness is still lacking.
As desirable as adversarial robustness and compressibility both are, the research has been equivocal regarding whether/when/hot their simultaneous achievement is possible (Guo al.) [0188] Balda et al., 2020 Li et al. 2020a| Merkle et al. 2022 Liao et al. 2022). This is even more pronounced for structured compressibility. which is alarming given its practical relevance (Blalock al.) 2020 Piras et al. 2025). However, recent research has started to provide mechanism-based explanations for this relationship, highlighting how compressibility impacts models' vulnerability to adversarial noise. For example, [Savostianova Saanple.Savossianova et al. (2023) demonstrate that low-rank parameterizations may inadvertently amplify local Lipschitz constants, increasing sensitivity to perturbations. Nern et al. (2023) connect adversarial transferability to layer-wise operator norms and their impact on representation geometry. Feng et al. (2025) further show that while moderate sparsity can enhance robustness, excessive sparsity causes ill-conditioning that reintroduces fragility and vulnerability. These results hint at

---

Published as a conference paper at ICLR 2026
Adverearil
Robust Accuracy
0.20
0.25
0.23
0.20
Image
0.28
10a
15
Latend space
Compressibility
Figure 1: visual preview of our findings. (Left) Sparsification expedites compression but creates sensitive latent directions. (Center) Adversaries exploit these sensitive directions to increase their potency. (Right) This leads decreased adversarial robustness.
delicate, regime-dependent relationship between compressibility and robustness but principled and general framework is still lacking.
In this work, we develop a framework to investigate the effect of structured compressibility on adversarial robustness through its effect parameter operator norms and network's Lipschitz constant. We jointly study how different forms of compressibility particularly neuron-level and spectral compressibility affect adversarial robustness. Our central result is instructive adversarial robustness bound that reveals how compressibility can lead to adversarial vulnerability by inducing small of highly sensitive directions the representation space. Empirically, we confirm this effect across architectures, datasets, and attack models, where ad adversarial attacks reliably identify and exploit these sensitive directions. Figure Trguureppret provides visual preview of our findings. Previous research tightly links compressibility to generalization (Arora et al. 2018 Barsbey et al.) (021); however, our findings imply that the very mechanisms that promote generalization can also introduce structural weaknesses. In summary, our contributions are:
We provide an adversarial robustness bound that decomposes into analytically interpretable terms, and predicts that neuron and spectral compressibility create adversarial vulnerability against loo and & attacks, through their effects on networks' Lipschitz constants.
2. Utilizing various compressibiliyyiinducii
interventions, we empirically validate our predictions regarding the emergence of adversarial vulnerability under structured compressibility with various datasets and models, including commonly used modern encoder architectures.
3. We demonstrate that the detrimental effects of compressibility persist under adversarial train- ing and transfer learning, and contribute to the appearance of universal adversarial examples.
4. We demonstrate and discuss our findings' implications for compression in practice, and highlight promising paths for designing models that reconcile efficiency and safety.
We provide our source code at
http://ttuub..oom//brreeeeee//// advcomm
2
SETUP
We denote scalars by lower case italic (k), vectors with lower case bold (æ), and matrices with upper case bold (W) characters respectively. Vector lp norms are denoted by || For matrices, [[][F, W| || correspond to Frobenius, spectral, and fo operator norms, respectively. We denote the 7th element of vector æ with v;, and row i of a matrix W with w4. Elements of sequence of matrices (c.g. layer matrices) are referred to by W [A], where for À E N let [A] = [L...., A}. Unless otherwise specified, we will be focusing on supervised classification problems, which will involve the input x E X' and label E y. A predictor g : X + IR > parametrized by 0 E θ produces output logits g(x; 0), the maximum of which is the predicted label arg maX $;. Predictions evaluated by loss function é: RDI x y - R., For brevity, define the composite loss function f(x, 0) E(g( (x. 0).
Notation.
Risk and adversarial robustness. Assuming a data distribution н on X' x V, we define the population and empirical risks as F(0) [f 0)], and F(e, S) := 1 Σn f(x; 0). where (z,, 31) denotes set of i.i.d. samples from æ. Adversarial attacks are small perturbations to input that dramatically disrupt model's predictions (Szegedy et al. 2014). In this paper, we focus on bounded p-norm attacks, which we define as
arg (x 0).
(1)
||a|| 58
2

---

Published as a conference paper at ICLR 2026
Given the adversarial loss fp (z. 0:6) == a², 0), we define adversarial risk and empirical ad- versarial risk as (0; :- Ex, En [ (xx 0; 0] and Fadyy (0, S; := - fadv (æ; 0: 6), respectively. The attack norm p chosen under the attack budger ó determines the type adversarial attack in question, with p = 2 and p oo as the most common choices. In this paper, we are primarily interested in what we call the adversarial robustness gap: Aadv := padv (0: 6) - F(0). where a small A is desirable for adversarial robustness.
of
Neural networks. Our analyses will focus on neural networks under classification. We define a fully connected neural network (FCN) with À hidden layers of h units as below:
(xz 0) = Co(W o(... ......Wz)) w))),,
where == (C, W ,..,, W W and C denote hidden layer and linear classification head parameters respectively, and ø is elementwise ReLU activation function. We omit 0 when it is obvious from the context for brevity. We can write as the composition of two functions, a linear classifier head e Rh + RD', and a feature encoder + : X + RRA, such that g( 0) := cl, C) o Φ(, W (z). When needed, we use ((x) or zadv (( to denote latent representations, where Tadv :- x + a". To expedite exposition and reduce notational clutter, throughout our analyses we assume that R and omit bias parameters.
Lipschitz continuity. Given two L" spaces X' and V. a function g : - y is called Lipschitz continuous if there exists constant Kp such that (((==))g(=)))))) )lp S K2llæ² x= 1p. V ax x² X. Said K2 is called the (global) Lipschitz constant. Any K, that is valid for a subset U C X is called a local Lipschitz constant on U. Although its computation is NP-hard for even the simplest neural networks (Scaman & Virmaux 2018); as notion of input- bpsue volatility, estimation, utilization. and regularization of the I Lipschitz constant have been a staple of robustness research (Cisse et al. 2017. (Bubeck et al., 2020; Muthukumar & Sulam] 2023 Grishina et al., (0025). Note that the FCN as defined in (2) is Lipschitz continuous in i, for p 2 1, along with other commonly used architectures such as convolutional neural networks (CNN) (Zühlke & Kudenko 2025).
Various prominent approaches to neural network compression exist, such as pruning, quantization, distillation, and conditional computing, (O' 2020). Here we focus on pruning and low-rank approximation, two of the most commonly used and researched forms of compression (Hohman et al. 2024). More specifically, we focus on inherent properties of network parameters that make them amenable to pruning We will first present a formal definition of a compressible vector, and then will show how this definition can be utilized to describe both structured prunability and (approximate) low-rankness,
Compressibility.
low-rank approximation, i.c. their compressibility.
Definition 2.1 ((g, k e) e>-compressiiiltt). Given a vector O E Rd and non-negative integer k S. d, let O& denote the compressed vector which contains the largest (in magnitude) k elements of 0 with all the other elements set to O. Then, 0 is (q, k e) commpesssible if and only if
10 Oulle/ |101|, S
In the case of equality, we call strictly (q, k. e) comppesssi. Complementarity, the spread variable 3 E [0, 1] can be used to characterize the dispersion of top-k terms, such that 10mzl = (1 B)]0 where m, indexes the i'th largest magnitude element in the vector.
Moving forward we will assume any vector denoted as compressible is strictly compressible, unless otherwise noted. See the Appendix for a more in-depth discussion of our compressibility definition and how it relates to other notions of approximate sparsity, where we show that our definition distin- guishes qualitatively different parameter configurations better compared to prominent alternatives.
Structured compressibility, Importantly, given that the O can be any vector, the above definition can be used flexibly to describe different notions of compressibility, including those of structured com- pressibility, where particular substructures in the model dominate the rest. More specifically, given a layer parameter matrix W E R from ( let v == (I|wal) || i............................. |/wlll)) denote É norms of rows of the matrix W. The compressibility of v would correspond to row/neuron compressibility, which is desirable property for neural network parameters it expedites pruning of whole neurons, with tan- gible computational gains. Note that this also would correspond to filter compressiblliy/pruabbii in CNNs with a matricization of the convolution tensor. Similarly, let := (2... denote the singular values of matrix W. Compressibility of a would correspond to spectral compressibility, serving as a notion of approximate/numericl low-rankness.
3

---

Published as a conference paper at ICLR 2026
3 NORM-BASED ADVERSARIAL ROBUSTNESS BOUNDS
Motivating hypothesis. Although structured (neuron, spectral) compressibility is desirable from a computational perspective, it also focuses the total energy of the parameters on a few dominant terms (rows/filters, singular values). This in turn creates few potent directions in the latent space and increases the operator norms of the parameters (f & operator norms respectively).
This increases their sensitivity worst-case perturbations: adversarial attacks exploiting these directions are amplified in the representation space, and can more easily disrupt the predictions of the model. For a more specific example using spectral compressibility, given a single layer neural network g(x:) Cøl Wa), assume that 01 > o iee. first singular value dominates the rest under high compressibility. Then, an adversarial perturbation a that aligns with the associated right singular vector V1 s.t. vf [a/||||l2 la R 1, will have scaled their post-layer representation by approximately 01- This turn would facilitate them to dominate the latent space against the original image. and ultimately change the prediction of the model. Taken from an experiment presented in full detail in Section 4 Figure
Input Space
Repr. Space
Baselline Model
Comp. Model
2 visualizes this phenomenon in reality. Here, we utilize PCA to visualize the input image, adversarial perturbation, and decision boundaries for a single sample under a baseline vs. compressible (low-rank) model. The top row visualizes the baseline model, where the minuscule adversarial perturbation fails move the perturbed image across class boundaries. The bottom row however, illustrates the compressible model under attack. Here, although attack budget identical in the input space, the adversarial perturbation is dramatically amplified in the representation space, leading to successful adversarial attack. Note that the decision boundaries in compressible model's s input space is much more contracted to reflect this vulnerability. In the Appendix, dedicate section to providing stronger, step-by-step intuition for our hypotheses.
Figure 2: Decision bound- aries under compressibility,
Compressibility-Dassee Lipschitz bounds. Our theory will relate structured compressibility to robustness through its effect the network's operator norms and Lipschitz constants. However, this brings about a particular conceptual challenge. Our notion of (g, k cocoprrrssibilltty like measure. Therefore, any direct relation between
others' (Diao et al. 2023), is compressibility and Lipschitz constants would be rendered void by the arbitrary scaling of the parameters. Therefore, we characterize foo and f2 operator norms of the parameters by upper bound that decomposes into (compressibility x Frobenius norm) terms. This "structure vs. scale" decomposition allows to meaningfully relate compressibility and robustness, and also allows us to develop concrete hypotheses regarding the effect of various interventions in neural network training.
Theorem 3.1. The following statements relate operator norms and structured compressibility.
(a) Neuron compressibility (i.e. row-sparsity): Let W,, i [h] denote the rows the matrix W. and let v-v = (I Ill..... (wlll)) denote E norms of its rows. Assuming v is (1, ku, Ev) compressible and each row W; is (2, ( (rr,,er--ooppesstiie -compressible implies:
| S (w- (vikkr + her Bu)
|
liv
(b) Spectral compressibility (Le. low-rankness): Leto == (01, .2....) denote the singular values of matrix W. Assuming a is (1, (1, ko; to) -o)-coppressible implies:
|W]|2 < (1 ())
hh
||
(5)
(1 3,)
kg
Intuitively, Theorem 3.1 deseribes how increasing compressibility affects layer operator norms: Neuron compressibility, i.e. small number of rows dominating the matrix increases for operator norm of the matrix, especially if the spread within these dominant rows are high. Similarly, increased spectral compressibility and spread increases the & operator norm. Note that the latter result is closely related to results from the literature that connect stable rank or condition number to robustness (Savostianova al. 2023 Feng et al., al,, 2025), see Section 5 Although Theorem 3.1 directly relates neuron and spectral compressibility to perturbations defined Ex and &g norms, standard norm inequalities couple these operator norms up dimension factors, vulnerability trends transfer
4

---

Published as a conference paper at ICLR 2026
across for and lg settings (see Appendix for further discussion and results). Lastly, while utilize the upper bounds for our following theoretical results, additional theoretical results in the Appendix characterize lower bounds the operator norm with similar implications.
As we move on to characterizing layers within a neural network, W will be used to denote the compressed version of the parameter matrix of layer 1. In the of row compression, this corresponds to setting the h trailing rows to O. In the case of spectral compression, given the singular value decomposition (SVD), W = U'V' the compressed matrix corresponds to W == U.E.V V where the h k smallest singular values are truncated.
Note that the sensitivity of the network not only relies on the characteristics of layer parameters, but also on the interactions between them. For example, it is possible upper bound the operator norm of two consecutive layers intericaved by a ReLU nonlinearity with || [|| However, this an overly pessimistic bound, as it accounts for the most potent directions of each layer perfectly lining up (unlikely in reality), and ignores the nonlinearity. This is why for our following theorem,
we first introduce the interlayer alignment terms Ap: These terms will help improve the operator norm bound by correcting for the said overly pessimistic assumption by using the "alignment" of the top-k terms in each layer- see Scaman & Virmaux (2018) for similar approach. With as the set of all diagonal binary matrices (for ReLU activations), we define Ap. for E {2, oo) as:
| W
& DD loc W
II
DU
Ax (1)
A max DED
+ RRo, A2(1)= max DED
+ R2.
V 2 W
(6)
where Roo, R2 are remainder alignment terms defined and shown be R2 0 as e 0 in the Appendix. We refer the reader to our proofs in the Appendix to explain the exact form the alignment terms take and a comparison to previous approaches, where we also dedicate section to provide more intuitive understanding for them. Having Theorem 3. help characterize the compressibility- based sensitivity of layers, and (6) help connect them, we now provide an upper bound to the Lipschitz constant of the complete encoder network.
Theorem 3.2. Let L be the Lipschitz constant of the encoder & defined following 2 Let DD denote the set of all diagonal binary matrices, corresponding to ReLU activation layers. Then:
(a) Neuron compressibility: The to Lipschitz constant of &b can be upper bounded by:
(1 -v) Vhk, + he, 11
L < 1
||
II 1=1
(1),
3x)
Ki
where (1) = A if l
Sept. and 1 otherwise. Sapt S
is the optimal
alignment partition set (See Definition A.4) that can be determined O()) time.
(b) Spectral compressibility: The Eg Lipschitz constant of t can be upper bounded by:
(1-6a) 30)
Vh Tio
12 5 12
II 1=1
II A2())
Note that for brevity and without loss of generality assume uniform compressibility across layers. These upper bounds can be directly used in conjunction with other results from the literature (Ribeiro) al. 2023) to characterize adversarial robustness gap, demonstrated in the next corollary. Here, given the binary classification context, we assume C E R² and ÿ E IR with slight abuse of notation.
Corollary 3.3. Under a binary classification task with logistic loss, Ely. Ú) = log (1 + e e-vi), given
neural network classifier described in 2 under the same assumptions with ( and (8) we have
paav (0:0 8) < (0) + 2
[|C|| and [aady (0:8) < F(0) 8 Si. CC|2, respectively.
See also results by Nern al. (2023) that connect encoder sensitivity to adversarial robustness. Note that although bounds provided in Theorem 3.2 are tighter than the pessimistic "product-of- norms" bounds. they deliberately trade off some tightness by utilizing Thcorem 3.1 However, in return, this results in bounds that decomposes into analytically interpretable and actionable terms. Such bounds have proven valuable in analyzing adversarial robustness in deep learning (Wen et al.)
5

---

Published as a conference paper at ICLR 2026
2020). Regardless, Figure 3 demonstrates the close correlation our bounds shows with the empirical robustness gap (p = 0.947). in a 2- 2-hidden-layer neural network with varying spectral compressibility (obtained through systematically varying the rank of factorized layer matrices). We provide full details in the Appendix, where we also explore the alignment terms' empirical behavior and estimation techniques, although detailed analysis thereof lies beyond our primary focus.
Given our focus on compressibility-driis threats to structural encoder safety under potential distribution shifts, we upper bound encoder's s global Lipschitz constant. While approaches that utilize local Lipschitz- ness are known to produce tighter bounds (Cisse et al.) 2017 Roth al.] 2020), results in Figure 3 and Section I show that for our theory this does not come at the cost of predictive power. Our results with univer- sal adversarial examples, to be presented within the latter, particularly highlight the relevance of global structural vulnerabilities. Lastly, in the Appendix we demonstrate strong correlation between global Lipschitz upper bounds and empirically estimated local Lipschitz constants.
1.40
Hob, Los
Std. Loss
1:20
Bou
0.00
0.40
-05
Layer (r)
Figure 3: Corollary 3.3 vs.
empirical robustness gap.
EXPERIMENTS
We now validate our theoretical findings through systematic experimentation. We first validate motivating hypothesis and then empirically show that (i) neuron and spectral compressibility- inducing interventions will reduce adversarial robustness against Eor and &2 adversarial attacks; (ii) persist under adversarial training, (iii) the compressibility-
the negative effects of compressibility related vulnerabilities induced on representations during pretraining will impact any downstream task in transfer learning; (iv) increasing compressibility creates vulnerable directions in the latent space, further enabling universal adversarial examples (UAEs), while increasing Frobenius norm will create vulnerability without leading to UAEs; and (v) compressed models will inherit the vulnerability of the original models, and conducting compression based on (g. k E) compressibiliity, reducing the spread of the dominant terms, or regularizing interlayer alignment will improve robustness.
Datasets, architectures, and training. We conduct our experiments in the most commonly used datasets and architectures in the literature on adversarial robustness and compression (Piras et al. 2025).
Datasets we use include MNIST (Deng) 2012), CIFAR- 10, CIFAR-100 (Krizhevsky & Hinton
2009). SVHN (Netzer et al. 2011), Flickr30k (Young et al. (014)). and ImageNet- 1k (Deng et al.
2009p. Architectures we utilize include fully connected networks (FCN), ResNet18 (He et al. 2016),
VGG16 (Simonyan & Zisserman 2014), WideResNNet-001- 111-11001222a-0o00 (Zagoruyko & (0modakis[2016] vision transformer (ViT)- both standalone classifier (Dosovitskiy et al. 2021) and as part of CLIP encoder (Radford et al. 2021), and Swin Transformer (Liu et al. 2021] ). Unless otherwise noted, we use softmax cross-entropy loss, the AdamW optimizer with a weight decay of 0.01, learning rate of 0.001, and use a validation set based model selection for early stopping.
Evaluating and training for adversarial robustness. When evaluating adversarial robustness, we utilize AutoPGD as the primary adversarial attack algorithm for evaluation (Croce & Hein 2020). as implemented by NNcollae et al. (20T8). When training for adversarial robustness, we utilize a PGD attack to generate adversarial samples at every iteration (Madry et al. 2018). Unless otherwise noted, we use a ratio of 0.5 for adversarial samples in a training minibatch. We use 8 = 8/255 and 8 0.5 for fx and lz attacks respectively for end-to-end adversarially trained models. We use 0.25x of these budgets for evaluating standard trained or adversarially fine-tuned models to allow a visible comparison (See Appendix for qualitatively identical results under different budgets and attack algorithms). By default, we present results for Eo and &2 attacks when evaluating robustness under neuron and spectral compressibility respectively, and defer the cross-norn results to the supplementary material, which also includes further details on our experiment settings and implementation.
Comparison across methods. Given that our theory is agnostic to the source of structured compress- ibility. we experiment and confirm our predictions with various methods to induce compressibility. Therefore, to retain the equivalence between these different methods and prevent confounding from specific compression procedures, we primarily compare uncompressed (e.g, unpruned) models while explicitly highlighting their different levels of compressibility. While some approaches such as low-rank factorization do not involve separate compression step, in approaches where a specific compression procedure is commonly utilized in practice (c.g. filter pruning after regularized training), show that our results apply the compressed models as well.

---

Published as a conference paper at ICLR 2026
0.1
8.3
0.55
25
2.05
Tial
8.4
0- 1.55
C20
Pruning
Figure 4: Model statistics under increasing strength of nuclear norm regularization (a).
4.1 RESULTS
Testing the motivating hypothesis. We start our empirical analysis with a demonstrative experiment to visually investigate the implications of our motivating hypothesis. For this, we train a single 400-width hidden layer FCN with ReLU activations on the MNIST dataset. We use nuclear norm regularization (NNR) to encourage spectral compressibility, adding the term ar||a||| to the training objective, with norms, we apply Frobenius norm normalization to W at every iteration (Miyato et al. 2018). While our following experiments will utilize more practically relevant currently apply normalization to fully isolate the effects of compressibility.
hyperparameter. To avoid confounding by NNR decreasing overall parameter
control mechanisms,
In Figure 4 (left) validate that in- tervention indeed increases spectral norm compressibility. As expected, Figure (center left) shows that spectral compress- ibility actually allows pruning: the more compressible models retain their perfor- mance under stronger spectral pruning. Figure (center right) shows that in- creased compressibility comes at the cost of adversarial robustness: as increases, adversarial accuracy dramatically falls. We further investigate whether this fall is due to our hypothesized mechanism. We let (ar) and 2ndv 4(xr a*) de- note the learned representations of clean and perturbed input images. If the adver- sarial attacks are taking advantage of the potent directions created by compressibility. then as compressibility increases: (1) The perturbations
0.3
Companbbl
0.42
0.08
0.30
0.00
3.50
2.40
Ac
2.30
Acc
2.20
0.50
12-
20
150
Row Sparsity Regularization (a)
Layer Rank (r)
Companibl by.
Corpessally
0.75
325
0.35
0.72
9.30
0.53
6.70
0.68
3.25
0.65
0.55
Acc
Arr Art
3.30
0.45
Acc
Jce Acc
12-4
2.20
Sparsity Regularization
Layer
Figure 5: Results with FCN (top) and ResNet18 (bottom) trained on CIFAR- 10 dataset.
a' should align more with the dominant singular directions, i.e., v'a" >> vJaa Vi E [k].j & [A]. (2) representations of adversarial perturbations should grow stronger relation to the original image's representation, i.e. Eavv = ||2/||=||a should increase. Results presented in (right)(righh)) confirm both predictions, further supporting our motivating hypothesis. Lastly, the previously presented Figure 2 visualizes the effect of compressibility in the input and representation space. We provide more detailed, step-by-step account of how potent leading directions are exploited by white box and black box adversaries in the Appendix for stronger intuition.
Adversarial robustness and compressibility under standard training. For implications of our analysis under more realistic settings, start by investigating the effects of compressibility adversarial robustness in fully connected networks (FCN). We induce neuron and spectral compress- ibility through group lasso regularization and low-rank factorization, respectively (latter avoids the excessive cost of nuclear norm regularization). As above, we conduct Frobenius norm normalization every iteration. (topp)55(top)) presents the results of these experiments: The reduction adversarial robustness as function of increasing compressibility is clear in both cases, confirming our main hypothesis. Note that we present robust accuracy (RA) / standard accuracy (SA) ratio alongside RA to highlight that the obtained results are not due to baseline SA being lower under compressibility.
We then investigate whether our hypotheses apply beyond the context of our theory, starting with convolutional neural networks (CNNs). We first test our predictions in ResNet18 8 models trained on CIFAR-10 datasets. Here we eschew Frobenius norm normalization for standard weight decay.
GGrupp lasso regularization penalizes the l norm of norms of cach layer, promoting row-sparsity.

---

Published as a conference paper at ICLR 2026
However, to prevent confounding from group lasso's 's effect on parameter scales, create scale-invariant version that regularizes row norms' f//82 norm ratio 2 Figure [5] (bottom) demonstrates that the above effects clearly translate to this set- ting as well, further solidifying the rela- tionship between structured compressibil- ity and adversarial robustness. We present similar results on two other architectures (VGG16, WideResNet-100 101) and two other datasets (CIFAR- 100, SVHN) in the Appendix. Going forward, for brevity we will focus on neuron compressibility results, and defer corresponding spectral compressibility results to the Appendix, where we also discuss unstructured compressibility and inductive bias-based emergent compressibility.
0:3
0.2
017
5.18
6.34
2.02
3.20
5.00
0.30
Rob. Aet.
Rub. A6L.506 AcC.
3.2
N.b. ALC
Rcb. Acc. (5od.
0.06
2.00
2.04
10 Row Spa Reguiarization
Sparsity Requiarization
Figure 6: Results with VIT (left) and CLIP (right).
Experiments with transformers. We next test our hypotheses under transformer architectures. Figure 6 (left) replicates our results under a ViT classifier model trained on CIFAR-IO dataset. Further, to test whether our hypothesis holds under zero-shot classification setting, we fine-tune pre-trained CLIP model on Flickr30k dataset under varying degrees of sparsification regularization, and conduct standard and adversarial zero-shot classification using ImageNet- k dataset. We find that our results (Figure [6, right) replicate here as well. That simply fine-tuning with sparsification can create this vulnerability with commonly repurposed encoder backbones highlights the safety implications of our results. See Appendix for further details and findings under other training settings.
Effects of compressibility on robustness under adversarial training. Given that adversarial training is the primary method for obtaining models that are robust against adversaries, we next investigate whether the effects we have observed will persist under this regime. To make this setting as close to practice possible, we also include learning rate annealing schedule (Cosine annealing) and basic data augmentation (random horizontal flip and crops), as well as attacks with standard budgets described above. The results almost identically replicate our observations under standard training (Figure 7 left). Although adversarial training increases adversarial robustness overall, the relative effect of compressibility remains as it is.
Universal adversarial examples. Examining the terms in Theorem we predict that while both compressibility and Frobenius norm are likely to increase vulnerability, only the former is likely to lead to universal adversarial examples (UAFs) (Moosavi-Dezfooli et al. 2017). due to the global vulnerable directions it creates. To test our hypothesis, modify the setting of FCN experiments presented above: In contrast to increasing row sparsity regularization under a fixed Frobenius norm, in an alternative set of experiments we systematically increase the constant to which Frobenius norm of the layers is fixed, without any row sparsity regularization. We utilize a FGSM based (Goodfellow et al.] 2015) UAE computation to develop adversarial samples. F (center left, center right) confirms our hypothesis: while increasing Frobenius norm only decreases standard adversarial robustness, increasing compressibility additionally creates vulnerability to UAES. In the Appendix, we replicate these results under ResNet18. Importantly, we also show that the converse relationship also holds: Training against UAEs vs. standard adversarial samples decreases top-k parameter spread B, providing further support for our arguments.
Adversarial vulnerability under transfer learning. Next, we investigate our hypothesis that the effects of compressibility should persist under transfer learning due to the structural effects created on representations. We train a ResNet18 8 model on CIFAR- dataset with increasing row sparsity regularization. After the training complete, we freeze the encoder parameters and train linear classifier head for prediction on CIFAR- 10 dataset and evaluate the robustness of the resulting model. Figure 7 (right) shows that the effects of compressibility observed above directly translate to the context of transfer learning, where increased compressibility in pretraining affects robustness performance in the downstream task, for which the network is fine-tuned.
Pruning and robustness. While we extensively investigated the effects of compressibility on robustness, for neuron compressibility we now focus on the behavior of models under down- stream layerwise filter pruning to ensure our insights transfer to practical pruning scenarios. Us-
In the Appendix, we show that standard group lasso creates tug-of-war between increasing compressibility and decreasing parameter scales; the former eventually wins, resulting in decreased robustness.
8

---

Published as a conference paper at ICLR 2026
224
Bub. Acc Sut. Acc.
Beh. Acc. Sua. ALc.
0.2
224
011
23
Unto
124
35
322
8.0% A/X
-33
Row Sparsity Rog dafiattion (e)
Trobenius Norm
Row Sparsity Regulatization (e)
Row Sparsity Regutarization
Figure 7: (Left) Effects of compressibility under adversarial training. UAEs under increasing (center
left) compressibility vs. (center right) parameter scale. (Right) Robustness under transfer learning.
ing the R and CIFAR-10 combination under adversarial training, in Figure g (left), compare the baseline model (a = 0.0) to a model regularized to be compressible (a 0.1).
We see that at no point do the com- pressed models surpass the uncom- pressed performance of the baseline model in terms of standard and ro- bust accuracy. However, as pruning ra- tio increases, the baseline model fails to retain its standard and robust per- formance, whereas the compressible (sparsified) model does considerably better, demonstrating the fundamental tension between robustness and com- pressibility. In the Appendix, we show that these results hold after post-pruning fine-tuning as well. Additionally, there we demonstrate that post-pruning fine-tuning can act as an additional source of vulnerability in and of itself, as under this procedure adversarial robustness deteriorates much faster than standard accuracy, confirming our results under yet another source of norm imbalance.
0.8
SA, Camp.
Accuracy
0.6
RA, Baseine Comp.
SA. LW SA. Glob.
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Pruning Ratio
Pruning Ratio
Figure 8: Robustness under compression. SA/RA: Stan- dard/Robust Acc. LW/Glob. Layerwise vs. global pruning.
In Figure 8 (right), we show that conducting pruning based on two simple interventions inspired by our bounds results in tangible improvements in standard and robust performance under pruning. Given the fact that layerwise pruning is known to produce harmful bottlenecks that lead layer collapse (Blalock et al. 2020), instead of targeting pruning ratio and pruning each layer accordingly. we set a target e for each layer, and for each compute & that satisfies this e level. Given a target global pruning ratio, scan over different levels of and determine the level that gets closest to the target ratio. Moreover, during training we control the spread of the dominant terms, 3, which our analyses show to be harmful for robustness, without decreasing compressibility. We accomplish this through regularizing the variance of the top 0.05 of each layer's filters' norms. Figure g (right) demonstrates that interventions create a tangible improvement in performance retention. In the Appendix, provide additional results showing that interlayer alignment can also be successfully used as regu- larization target for robust compressibility. We consider these interventions both as validations of our theory and promising directions for future robust compression research. However, we also highlight that it may not be possible to completely negate the dangers of concentrating parameter energy in few substructures, extensively demonstrated by our theory and experiments. Therefore, while pruning and low-rank approximation remain valuable compression methods, combining intermediate levels thereof with other compression methods such as quantization or knowledge distillation seems to be the most promising approach in reconciling safety and robustness, which is in line with recent findings in the literature (Pavlitska et al. (023).
RELATED WORK
Adversarial robustness. The susceptibility of the neural network models to adversarial examples created through small perturbations (Szegedy et al. 2014) engendered lot of research investigating the issue (Madry et al. 2018). To this day adversarial robustness remains one of the most important topics in machine learning safety (Malik et al.. 2024). The literature ranges from the development of new attacks and defenses (Moosavi-Deziine
et al.) 2016: Abdollahpoorrostam
al. 2024), to
investigating sources/mechanisms of adversarial vulnerability, to implications of AEs for the inductive biases of modern machine learning architectures (Ilyas et al. (019 Ortiz-Jimennt
al.) (22 Xu
jet al.) 2024), to developing strategies to retain model expressivity and generalization while defending against adversarial attacks (Tsipras al. 2019
Zhang al. 2024

---

Published as a conference paper at ICLR 2026
Pruning and low-rank approximation. Prominent compression approaches include pruning, quanti- zation, distillation, conditional computing. and efficient architecture development (O'Neill, (020). Out of these, pruning remains among the most actively researched compression approaches due to its versatility (Cheng et al. 2024). Inducing compressibility sparsity at training time is one of the easiest way to obtain prunable models (Hohman et al. 2024). Compressibility across different substructures, a.k.a group sparsity (Li et al. (2220)), allows for structured pruning (e.g. neuron/row, filter/channel, kernel pruning), which computationally efficient (Yang al., 2018), yet leads to sharp reduction in network connectivity, threatening performance (Blalock et al., 2020). Lastly, spec- tral compressibility relaxes the notion of low-rankness (Suzuki 20200 Schotthöfer et al., (022). While nuclear norm regularization is not a commonly utilized intervention due to the computational costs involved, low-rank factorization continues be a prominent architectural design choice due to its attractive theoretical and empirical properties (Savostianova et al. 2023).
Compressibility and robustness. Whereas some research argues that compressibility/spartttt is beneficial for adversarial robustness (Guo et al. 2018: Balda et al., 2020 Liao al. 2022). others indicate the relation is at best highly dependent on the degree and type of compressibility. as well
as attack type (Li et al. 2020a; |errle al. 2022 Savostianova et al. 2023 Feng al. 2025).
While a stream of new methods incorporate adversarial robustness in novel ways to pruning (a.k.a. adversarial pruning), recent systematic benchmarks reveal marginal benefits for such methods compared to weight-based pruning (Lee et al. 2020) Piras et al. 2025). Whereas some methods demonstrate benefits of adversarial traiming- aaware sparsification (Gui et al., 2019; SShwwag al. 2020: Pavlitska et al. 2023), adversarial training hampers standard generalization, transferability as well as computational feasibility especially for larger models, plaguing such methods (Tsipras et al..
2019 Wen et al., 2020 Yang et al. 2024). A comprehensive understanding of how compressibility and robustness interact, adversarial or otherwise (Barsbey et al. 2025), is still lacking.
Comparison to previous research. Our work addresses critical gap in the literature: paucity of research that establishes a principled, theoretical relationship between structured compressibility and adversarial robustness with extensive empirical confirmation. While doing so, we find that it produces complementary results to most closely related previous work. For example, Savostiannva et al.) (2023) and Feng et al. (2025) highlight the adversarial vulnerability created by increased condition numbers due to high unstructured sparsity or low-rank training, respectively. Our results complement and extend their conclusions by providing convergent theoretical results with a more fine-grat grained, source- agnostic notion of compressibility. and can naturally incorporate neuron which the cited work do not address. Lastly, in our Appendix we investigate two prominent structured adversarial pruning methods (Zhao & Wressnegger [203] Zhong et al. 2023 Piras et al.) [2025), and demonstrate that these implicitly control operator norms in a way that cannot be simply attributed to adversarial training. Our complementary findings highlight the design of theoretically informed robust pruning methods as a promising future research direction.
6 CONCLUSION AND FUTURE WORK
In this paper, we present a unified theoretical and empirical treatment of how structured compressibil- ity shapes adversarial robustness. Via novel analysis of neuron-level and spectral compressibility, uncover fundamental mechanism: compression concentrates sensitivity along small number of directions in representation space, rendering models more vulnerable even under adversarial training and transfer learning. Our bosed robustness bounds offer interpretable decompositions that predict both standard and universal adversarial vulnerability, and shed light on the trade-offs between efficiency and safety in modern neural networks. Empirically, we validate these insights across datasets, architectures, and training regimes, showing how compressibility determines adversarial susceptibility in various learning contexts. Inspired by our bounds, we outline simple, targeted strategies that can mitigate these vulnerabilities.
Future work. While our theory provides novel insights into structured compressibility adversarial vulnerability relationships, future work must focus on composite effects of practical compression approaches. A structured compression scheme might have multiple effects simultaneously: while harming robustness through increased structural imbalance, it can help it by reducing Frobenius norms or interlayer alignment, or closing off-data- datt--annfoll directions in the representation space. Achieving tighter local bounds, and incorporating other types of compression (c.g. semi-structurre pruning, quantization) and distribution shifts (e.g. other E, attacks, spurious correlations) are other important future directions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
MB was supported by the EPSRC Project GNOMON 1E6/X011364/11. US was partially supported by the French government under the management of Agence Nationale de la Recherche as part of the "Investissements a'vvenii" program, reference AAR-19-P3IA-000 (PRAIRIE 31A Institute) and by the European Research Council Starting Grant DYNASTY 101039676, TB was supported by a UKRI Future Leaders Fellowship Igrant number MR/Y018 8/1).
REPRODUCIBILITY STATEMENT
Experiment details are provided in the main paper and in the supplementary material. Repository at https //github. com/ httm///barbbby///eee hewessbeoyy///oopppaoooo adveomp includes code for reproducing main results.
REFERENCES
Alireza Abdollahpoorrostam, Mahed Abroshan, and Seyed-Mohsse Mohesnn Dezeoii-ee SuperDeepFool: A new fast and accurate minimal adversarial attack. In Advances in Neural Information Processing Systems, 2024.
Arash Amini, Michael Unser, and Farokh Marvasti. Compressibility of Deterministic and Random Infinite Sequences. IEEE Transactions on Signal Processing, 59(11):5193 5201, 2011.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square Attack: A Query- Efficient Black-Box Adversarial Attack via Random Search. European Conference on Computer Vision, 2020.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, 2018.
Emilio Balda, Niklas Koep, Arash Behboodi, and Rudolf Mathar. Adversarial Risk Bounds through Sparsity based Compression. In International Conference on Artificial Intelligence and Statistics, 2020.
Melih Barsbey, Milad Sefidgaran, Murat A. Erdogdu, Gaël Richard, and Umut Şimşekli. Heavy Tails SGD and Compressibility of Overparametrized Neural Networks. In Advances in Neural Information Processing Systems, 2021.
Melih Barsbey, Lucas Prieto, Stefanos Zafeiriou, and Tolga Birdal. Large Learning Rates Simultancously Achieve Rohustness to Spurious Correlations and Compressibility. In International Conference on Computer Vision, 2025.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the State of Neural Network Pruning? Stt::2003,030333 2020.
Sébastien Bubeck, Yuanzhi Li, and Dheeraj Nagaraj. A law of robustness for two-layers neural networks. arXiv:2009. 14444. 2020.
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A Survey on Deep Neural Network Pruning: Taxon- omy, Comparison, Analysis, and Recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10558-100 10578, 2024.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks:
Improving robustness adversarial examples. In International Conference on Machine Learning. 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2009.
Li Deng. The MNIST Database of Handwritten Digit Images for Machine Learning Research. IEEE Signal Processing Magazine, 19111144 2012.
Enmao Diao, Ganghua Wang, Jiawei Zhang, Yuhong Yang, Jie Ding, and Vahid Tarokh. Pruning Deep Neural Networks from Sparsity Perspective. In International Conference on Learning Representations, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2026
Yangqi Feng, Shing-Ho Lin, Baoyuan Gao, and Xian Wei. Lipschitz constant meets condition number: Learning robust and compact deep neural networks. arXiv:2503.2454, 2025.
A. Frank. Some Polynomial Algorithms for Certain Graphs and Hypergraphs. Utilitas Mathematica, 1976.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations, 2015.
Rémi Gribonval, Volkan Cevher, and Mike F. Davies. Compressible Distributions for High-Dimensional Statistics. IEEE Transactions on Information Theory, (8), 2012.
Ekaterina Grishina, Mikhail Gorbunov, and Maxim Rakhuba. Tight and Efficient Upper Bound Spectral Norm of Convolutional Layers. In European Conference on Computer Vision 2025.
Shupeng Gui, Haotao N Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model Compression with Adversarial Robustness: A Unified Optimization Framework. In Advances in Neural Information Processing Systems, 2019.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse DNNs with Improved Adversarial Rohustness, In Advances in Neural Information Processing Systems, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Fred Hohman, Mary Beth Kery, Donghao Ren, and Dominik Moritz. Model Compression in Practice: Lessons Leamedd from Practitioners Creating On-device Machine Learning Experiences. CHI Conference on Human Factors in Computing Systems, 2024.
Rasheed Hussain and Sherali 7eadally. Autonomous Cars: Research Results, Issues, and Future Challenges.
IEEE Communications Surveys & Tutooiils, 2172):1275-1313, 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021. httpss:s:://////m htt.///ii..rr/////// 10. /281/22nodo. zenodo. Stteeooo..511500 5143773
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box Adversarial Attacks with Limited Queries and Information. In International Conference Machine Learning, 2018.
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Brandon Tran, Dimitris Tsipras, and Aleksander Madry. Adversarial Examples are not Bugs, they are Features. In Advances in Neural Information Processing Systems, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009. URL hhtpps://www cs www.nto.ooooooooooo edu/-kriz/
learning- features 20leeeeess-000---0 15.09-TR..pFF.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwon Shin. Layer-adantive Sparsity for the Magnitude- based Pruning. In International Conference on Learning Representarions, 2020.
Fuwei Li, Lifeng Lai, and Shuguang Cui. On the Adversarial Robustness of Feature Selection Using LASSO. In International Workshop on Machine Learning for Signal Processing, 2020a.
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2020b.
Ningyi Liao, Shufan Wang, Liyao Xiang, Nanyang Ye, Shuo Shao, and Pengzhi Chu. Achieving adversarial robustness via sparsity. Machine Learning, 11 111(2):685-111, February 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In International Conference on Computer Vision, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Decp Learning Models Resistant to Adversarial Attacks. In International Conference Learning Representations, 2018.
Jasmita Malik, Raja Muthalagu, and Pranav M. Pawar. A Systematic Review of Adversarial Machine Learning Attacks, Defensive Controls, and Technologies. IEFF Access, 12.99382-994211, 2024.
12

---

Published as a conference paper at ICLR 2026
Florian Merkle, Maximilian Samsinger, and Pascal Schöttle. Pruning in the Face of Adversaries. In Image Analysis and Processing, 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations, 2018.
Seyed-Mohsen
Alhussein Fawzi, and Pascal Frossard. DeepFool: A Simple and Accu- rate Method to Fool Deep Neural Networks. In JEEE/CVF Conference on Computer Vision and Puttern Recognition, 2016.
Seyed-Mohsen
Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In IEEF/CVF Conference Computer Vision and Pattern Recognition, 2017.
Ramchandran Muthukumar and Jeremias Sulam. Adversarial Robustness of Sparse Local Lipschitz Predictors. SIAM Journal on Mathematics of Data Science, 544):920-948, 2023.
Laura F Nern, Harsh Raj. Maurice André Georgi. and Yash Sharma. On transfer of adversarial robustness from pretraining downstream tasks. In Advances Neural Information Processing Systems, 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. URL htt.:: /ttp::///ffdd wtt.lddd..ttaffaoaa.ram edu http///oousnemmbbb..ee
Maria-Irina Nicolae, Mathieu Sinn, Minh Tran, Beat Buesser, Anish Rawat, Martin Wistuba, Valerio Zantedeschi, Nathalie Baracaldo, Heiko Ludwig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v1.0.0. arXiv: 1oXiv:807.001065 2018.
James O'Neill. An Overview of Neural Network Compression. arXiv:2006.03666 2020.
Guillermo Ortiz-Jimenee Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Optimism in the Face of Adversity: Understanding and Improving Deep Learning Through Adversarial Robustness. Proceedings of the IEEE, (09(5):635-659 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. 7ach Vetooo Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch An Imperative Style, High- Perf-Peerfomaecc Deep Learning Library. arXiv: /et.:992200000 203,, 2019.
Svetlana Pavlitska, Hannes Grolig, and J. Marius Zollner. Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidence. In IEEE Symposium Series Computational Intelligence, 2023.
Giorgio Piras, Maura Pintor, Ambra Demontis, Battista Biggio, Giorgio Giacinto, and Fabio Roli. Adversarial pruning: A survey and benchmark of pruning methods for adversarial robustness. Pattern Recognition, 168: 11 111788, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, 2021.
Pranav Rajpurkar, Emma Chen, Oishi Banerjee, and Eric J. Topol. Al in health and medicine. Nature Medicine, 28:31-38, 2022.
Antonio Riheiro, Dave Zachariah, Francis Bach, and Thomas Schön. Regularization properties of adversarially- trained linear regression. Advances Neural Information Processing Systems, 2023.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. Adversarial Training is Form of Data-dependent Operator Norm Regularization. In Advances in Neural Information Processing Systems, 2020.
Dayana Savostianova, Emanuele Zangrando, Gianluca Ceruti, and Francesco Tudisco. Robust low-rank training via approximate orthonormal constraints. In Advances in Neural Information Processing Systems, 2023.
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: Analysis and efficient estimation. In Advances in Neural Information Processing Systems, 2018
Steffen Schotthffer, Emanuele Zangrando. Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Low-rank lottery tickets: Finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2026
Vikash Sehwag, Shiqi Wang. Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. In Advances in Neural Information Processing Systems, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 2014.
Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error. In International Joint Conference on Artificial Intelligence, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow. and Rob Fergus. Intriguing properties of neural networks. arXiv:1312.66 1316.6999, 2014,
Vischn maintainers and contributors. Torchvision: Pytorch's computer vision library, /attps://githut
wttplpytttcciiiiiiooooo 2016.
Dimitris Tsipras. Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness May Be at Odds with Accuracy, 0rXi: 12152, 2019.
Yijun Wan, Melih Barsbey, Abdellatif Zaidi, and Umut Simsekli. Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD. In International Conference on Machine Learning, 2024.
Yuxin Wen, Shuai Li, and Kui Jia, Towards understanding the regularization of adversarial robustness on neural networks. In International Conference on Machine Learning. 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State-of-the-ar Natural Language Processing. arXiv:1910003771 2020.
Tianlong Xu, Chen Wang, Gaoyang Liu, Yang Yang. Kai Peng, and Wei Liu. United We Stand, Divided We Fall: Fingerprinting Deep Neural Networks Adversarial Trajectories. Advances Neural Information Processing Systems, 2024.
Kelichiro Yamamura, Haruiki Sato, Nariaki Tateiwa, Nozomi Hata, Toru Mitsutake, Issa Oe, Hiroki Ishikura, and Katsuki Fujisawa. Diversified Adversarial Attacks hased on Conjugate Gradient Method, International Conference on Machine Learning, 2022.
Carl Yang, Aydın Buluç, and John D. Owens. Design Principles for Sparse Matrix Multiplication on the GPU. In International Conference on Parallel and Distributed Computing, 2018.
Sheng Yang, Jacob A. Zavatone- Veth, and Cengiz Pehlevan. Spectral regularization for adversarially-rooust representation lcarning. arXiv:2405.1718) 2024.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv: 2016.
Kaibo Zhang. Yunjuan Wang, and Raman Arora. Stability and Generalization of Adversarial Training for Shallow Neural Networks with Smooth Activation. Advances in Neural Information Processing Systems. 2024.
Qi Zhao and Christian Wressnegger. Holistic Adversarially Robust Pruning. In International Conference on Learning Representations, 2023.
Shaochen (Henry) Zhong, Zaichuan You, Jiamu Zhang, Sebastian Zhao, Zachary LeClaire, Zirui Liu, Daochen Zha, Vipin Chaudhary, Shuai Xu, and Xia Hu. One Less Reason for Filter Pruning: Gaining Free Adversarial Robustness with Structured Grouped Kemel Pruning. Advances in Neural Information Processing Systems, 2023.
Maxi--Maxinllan Zühlke and Daniel Kudenko. Adversarial Robustness of Neural Networks from the Perspec- tive of Lipschitz Calculus: A Survey, ACM Comput. Surv., (40:))::42:11122::: 2025.
14

---

Published as a conference paper at ICLR 2026
On the Interaction of Compressibility and Adversarial Robustness -Appendix-
Contents
A Proofs
2
Additional Technical Results and Analyses
B.I
cc.kppsicomyysssibbiilyy other notions of approximate sparsity
8.2
Lower bounds on operator norms
A.3
Relationships between operator norms
B.4
Empirical analyses of the robustness bound and related quantities
B.5
Approximating the interlayer alignment terms
10
Details of the Experimental Settings and Resources
11
Datasets
2
Models
11
Standard and Adversarial Training
12
Implementation Resources
12
Additional Empirical Results
12
D.I
Experiments with other datasets and architectures
12
D.2
Regularizing group sparsity
12
D.3
Adversarial training results for spectral compressibility
13
D.4
Compressibility through inductive bias
13
D.5
Unstructured compressibility
13
D.6
Results with alternative norms, budgets, attacks
................................
13
D.7
Fine-tuning results with transformers
14
D.8
Results with post-pruning fine-tuning
15
D.9
Exploitation of vulnerable latent directions
15
D.9.1
Attack alignment with latent directions
16
D.9.2
White box and black box attacks exploiting latent directions
16
D.9.3
Interlayer alignment
17
D.10 Comparison with adversarial pruning literature
18
Further experiments with UAEs
18

---

Published as a conference paper at ICLR 2026
A PROOFS
We with number of auxiliary results that used in the theorems and corollary presented in Section I1, = (1 - (1-
Lemma A.I. For any strictly (g, k c)compressible vector A and for all q ||0(*)|
Proof. |00 (0x) e !| follows from the definition compressibilityy Adding ||0 [3] | both sides leads | bا < + [ llg. with LHS due elements of ((k) populating disjoint sets of
coordinates. Result follows with simple algebraic manipulation.
Note that for the results in this section, we use ((4) and 0₂ equivalently to denote a vector that includes only the I dominant terms.
Lemma A.2. For p' < q. given the (2, k compressible vector 0 E IR", we have:
|0 S k
(10*)) ll d fell0lo.
(9)
Proef. We start by applying Minkowski's 's inequality to |001..::
| pT ||ا + |10 0 11r-
||
(10)
We now bound the terms on RHS separately. For the first term, since p" q by Hölder's inequality for k-sparse vectors we have
[10 11- s k - (10
the next term, we can write
0
o 1
11
S do- - 1 s d -el00lo...
with the left inequality due to Hölder's 's inequality, and the right due to (( (2, k. e) compressiiiiy Combining the expressions for both terms, have
|11|:- - ks - (100000)00000>>0000)))) + dit
bellolla. -
(II)
Proposition A.3. Given linear binary classifier and binary cross eoosooy loss function, assuming O RR we have the following bound:
Fadv (0:6) F(0) 51101|,-
(12)
Proof of Proposition A.3 For binary entsoyy loss we have:
pady (xr, 0: log + exp (-) +
We observe that pady (ax, 0: )) s f(x, 0, + 61|01|p> p- since
fady (x, 0: 5) = log [ + exp
(-v(x + +111011-
1+exp (-v(x² T0) + 01
= log (1 + exp
(-u(xTe)) log
exp( (-y(x v() 0))
exp (-y(x 0))
= [ 0) + log 1+ (6)
I exp 1y 0))
< f1x 0) + 01011p-
with the last incquality due to the fact that exp( exp(-v(x Tei) 1+epp yynn Ta))
Taking the expectation of the expression gives:
F puddy (0: < F(0, 0
Main results. We now present the proofs for Theorem 3.1 and 3.2 and Corollary 3.3
2

---

Published as a conference paper at ICLR 2026
Proof of Theorem 3.7
For brevity will omit as subscript, such that =cuk== B =B Bu.
For (a), we assume is in descending order w.l.o.g., g., and is the corresponding vector of f2 norms for each
Tow, We note that
1 (4) 111
2 kvk
(13)
> k(1 8)
(14)
c)|lp|aa k(1 8)v1
(15)
(1 6)
(1 B) || 2 m
(16)
(1 6)
(1 3) || N
W 00
(17)
with (13) being the smallest magnitude element in y (A)
14
due the definition of slack variable 3, and 15
due Lemma A.1 and (17) due the fact that || W| 00
P1. as is assumed to be magnitude-ordere.
then move on to characterizing ||=/||2- Notice that
h | = Σ vi S
Vhe
(18)
Tel @||:
(19)
h (Vir (1 112 vhllblla)
(20)
x V /hi, + vher) |/0122
(21)
< (v Whk + Vhe.)
|W|le
(22)
Note that [8] is due to standard norm inequality between & and la rows, 20 due to Lemma and 22
due lg norm of the vector of row rows equals the Frohenius norm. Plugging (22 back into gives the desired result.
For (b) the proof follows similarly through steps
B3
16
by replacing v with a. After that, we continue with
11 c)
||al|
||
2
a
(23)
(1 3) k
[1 E) 1
||al|] 2
[ ||
(24)
(1 (8 k
(1 c) Vh
||a||2 2
[ 2
(25)
(1 3) &
(1 - é) Vh
(1 3) k |||F 2 W
(26)
with (24 due to [[ ||2 = 02. 255 due to standard norm incquality between & and & norms, and 26 due to the fact that & norm of singular values equals Frobenius norm, i.c. || = ||a||2.
Proof of Theorem 3.2 Proofs for both conditions rely on an additive decomposition the layer matrices W
into dominant/lcading terms vs. remainder terms, i.c. W' = W + W!. In structured compressibility this takes the form of W and W! including & leading (largest & norm) rows and h k remaining rows, respectively, with the rest of the
to 0 both cases. In spectral compressibility. this takes the form of W + w u (V1)T + u (v!)² where the remaining & vs. leading singular values set respectively.
Let z' denote the poxt-activation representations of the network after layer I E [>]. The Jacobian of the network output zd with respect the input is given by:
(x) = D (x) w A "D'">""'''" D")
(27)
where D' is the diagonal binary matrix corresponding the ReLU activation after layer 1, i.c., (D'),: I[(="); > 0), with z' being the pre-activation representation at layer for input æ.
Letting I2 denote the p-norm Lipschitz constant of the encoder in the input domain, it can be computed as the maximum p + p operator norm of the Jacohian over the input space X:
3

---

Published as a conference paper at ICLR 2026
L sup
=Ex [J+(
sup
(D*())"""" ...'((e))'l
(28)
#ex
For brevity, we use the following notation:
P(D) = D'(=) D'eV
(x))
(29)
Note that optimization over can upper bounded by optimization over binary activation matrices D' E 'D for each layer whenever convenient. We replace the notation D (a) with D' when doing so.
Note that in this proof, for increased precision and brevity we introduce the following notation for the interlayer alignment terms:
Apr.t
Ap, DED
(30)
where Ap. stands for the inner RHS term optimized over (
(a) Row/neuron compressibility We aim to bound L$ as:
L max [P(D) D DA
(31)
We start by noting that we can upper bound this norm by partitioning inside terms based on the submulti- plicative property:
[(D) s DD'W D D' 1oc
(32)
S < " l D 2 1 | 1oc
I|
(33)
Note that any such parsing is valid as long as layer does not appear in two interlayer terms at once. Given valid parsing set {1, 21,,....... 13, have the interlayer alignment terms for E i.e. "W'"'''W" I| and standalone terms for all remaining layers {I & S,1+1 & S}: | We denote all such valid parsing layer subsets with S, where S does not include any consccutive indices for any S E S. We will first prove the bound for any valid parsing set, and then define the optimal alignment parsing set that would lead to the tightest bound.
We first analyze generic alignment term, using the additive decomposition into leading and remainder terms. Remember that for layer I we denote the row & norms with v' = (M/..... ......,,)) 14), and w.l.o.g. assume that the rows are ordered descending order according to vi. Also note that [ = | |0 = ví.
w D' D'W'l Ibe w Ilse + || D ww' 1lo
+ ! 1lso W D D'W'llo
(34)
I1 | /
+ +| |1x0 W 11 + || //
(35)
| D''''''', w/ |lse
|/
w |
+
W Ilo |
mn
v x
+ 1f41 ofoft1
(36)
[ D' D'w'll laxe
||
|
W Rm,(()
(37)
Since the remaining, standalone layer norms also contribute
[
we have
I |
1oo (ES
II
w
Roc(e)
(38)
W 1bo
Bounding the Lipschitz constant accordingly:
W D' Ioc
L
<
max D
DA
[
ooo
W 3o
W 1o
+ Roo Roo(e)
(39)
W ||
W Mas II
max DED
|
+ Roc
(40)
[=]
lES
A
[
lo II
A
(w W Rc)
(41)
[=]
FES
4

---

Published as a conference paper at ICLR 2026
Contributing alignment term of 1 for (1 & 5,1 E 5) gives the desired result if $ Sopt, which define below.
Given multiple valid parsing sets are possible whenever 2, we lastly define the optimal alignment parsing set, Sopt-
Definition A.4 (Optimal Alignment Parsing Set). The Optimal Alignment Parsing Set Sopt is set in S that achieves the minimum product of the corresponding maximum alignment factors:
Sopt arg min SES
II 1E8
A
(42)
Note that Sopt might not be unique, but min I A A is.
SCS
Complexity of finding Sope: Finding Supt E argmin Iles Abo,1 is equivalent to finding the independent
Bes
set S in the path graph G (V, E) with V = (I..... - 1} that maximizes Eies w7, where weights wi - log As. (assuming A',,1 > O; we handle A as a special case yielding I.e A O). This is the Maximum Weight Independent Set, which be solved in linear time chordal graphs, of which path graphs are subfamily [Frank] 1976).
(b) Spectral compressibility; We can upper bound L by considering all possible activation patterns (all possible binary diagonal matrices D'y:
L
S max
PP(D)
(43)
D DA....AA
DA
We modify the SVD decomposition for layers
w
U' VVVEVE((V)
(44)
= U. E + U
E (VL) + Ex (V!)²
(45)
B
Note that assume untruncated singular vector matrices for W and W. for the equation above to be valid. We then decompose the spectral norm using the submultiplicative property:
[P(D))la = ||D'''' D "W D
D' D'w'll 112
(46)
I |
B A 1A=111 "=='BB='' "BBB"""" AA A
[B IB'DDAA Al2.....
(47)
We then analyze the central term |[B' | A' and decompose using the submultiplicative and subadditivity properties. Remember that for layer we denote the singular values with a' = (of....,,ff Also note that
||
|| oi.
B t1 D'A'122 1/2
< Ily E (v D'U' D'U., IV E (V!*)) D'U! Ex
+ Ily E
E
(48)
$ ||y x
E1 /ot D'U oh
D'U'llay Tf
(49)
|l x D'UI Ehlle
of
o
$ / Vo
+ Vo
+
(50)
Volott
o.o 4+
|l (v D'U'' + Ra(e) Vototi l+1
S Viott Vol
oft
(51)
where we set all cross-alinnment terms other than dominant-dominan interaction to 1. This is made possible by the fact that they the multiplication of orthogonal matrices and ReLU matrix, all of which have spectral
5

---

Published as a conference paper at ICLR 2026
norms upper bounded by 1. Note that for all layers 1 1,..., Vo
will appear twice in the multiplication.
including the first and last layers due to the leading and final terms (47. leading to expression:
A-1 w
11
D'U
Exlla
PPD)l S
II
+ Ra(e)
(52)
Volot
1=1
Bounding the Lipschitz constant:
L
max Di DA....DA DA
[P(
(53)
1l Il+1 (V!+2) D'UR v E/il=
|
Ra(c)
(54)
Volott
1=1
1=1
E (V
D'U "
E Ra(c)
W lla
max DED
(55)
Volott
II
II
AS(W W
(56)
yielding the desired result.
Proof of Comolarr 3.3
Let a denote the adversarial perturhation on the input æ. where ||]|p S define
the effective perturbation budget in & norm for the feature encoder as o% == max [[0(z) 4bx z + a)llp- Note that by definition of the Lipschitz constant and by Theorem 3.2 we have
& max [[x( +xx a)ll
S || (x a) s ||lllL = SL
(57)
Plugging the result back into text, instead 0 in (12)
12 yields desired result. Note that for clarity, corollary uses the main
Lemma A.5. Under the conditions described in Theorem
3.2 R2
a 0 e 00)
Proof. p oc: Due to the definition of compressibility. for all E [A].
v |b S <||<'||
(58)
1 & chll c le.
(59)
by applying standard inequalities inequality applies to both vi.u and I+1
and columns. The result follows from noting that the final
p = 2: Similarly, due the definition of compressibility, for all e [A].
I|o' -o'lll e||o'|la Il2
(60)
o't1 S ev Vi |i
(61)
since ||a'||2 = [r. The result follows from noting that the final inequality applies both o and 1+1 1+t1
Lemma A.6. Under the conditions described in Theorem
3.2
A:(W1=1 w can be replaced with
min(1, (,,4, W for p E (2)
Proof. Forp 00.
W |W+1
DW 1oc
W
MaXDED
D
max DED
oo 1l
W
bo
(62)
W
W
1.
(63)
The proof follows identically for p

---

Published as a conference paper at ICLR 2026
B ADDITIONAL TECHNICAL RESULTS AND ANALYSES
B.1 (g, k, C)MPPEE-SEESSIIILLTY VS. OTHER NOTIONS OF APPROXIMATE SPARSITY
Further discussion (q. c)-coppressibility
Our concept of compressiiility can thought the generalization of sparsity, with the obvious advantage of being applicable to domains where true sparsity is rare, such as neural network parameter values. Note that our intuitive definition of compressibility is hased on foundational results in compressed sensing and is well exploited in the established machine learning literature (Amini et al. 2011 Gribonval et al. 2012 Barsbey (et al.. 2021 Diao al. 2023 Wan al. 2024p. More specifically, when & < d and c < Definition 2. is equivalent to Gribonval et al. d
definition of compressible vector. Inspired by desiderata from an ideal
metric of sparsity in the economics literature, Diao et al. (2023) recently introduced another scale-ine call-invariiat
notion of approximate sparsity:
Definition B.1 (PQ Index Diao et al. (023 ). For any 0 < < q. the PQ Index of a non-zero vector w E R is
(w)) - di ||
(64)
||]]g
Interestingly, is possible to directly relate this notion of sparsity to (q. k. compresssiii as shown in the following proposition.
Proposition B.2. Given O < p < q. for a vector 0, its (g, k, compressibillty implies the following lower bound for its PQ Index:
1 R² p. (0).
(65)
where K = k/dd and Φ = --1 Note that the constraints on p, g imply & > O.
Proof. Lety= = 1 Note that from 11 we know that |00|p S (k* + d' ||0|g. This implics
| 0
< No + d
(66)
Note that PQ Index from (64 can be written as (1 4p., (0))d' and simple algebraic manipulation gives the desired result.
101|. |0
Plugging this into the LHS of (66
Remark B.3. Assume that @ and 0' are (g, k, e) and (g. k', e') compressible respectively. If k'' and < é'; k/' and e é' implics larger lower bound PQI. That a larger (g, k, compressibility suggests larger PQI.
Dominance vs. spread. While (g, k, ecoppressibility quantifies how well vector can approximated using its top-k entries (e.g. top-k filters or singular values), it does not fully capture the internal structure among those dominant terms. Consider the vectors 21 = (10, 2, 1, 1) and 1, yet their dominant components differ markedly in structure. To
(6, 6, 1, 1): both yield the same 2-term
relative approximation error under formalize this distinction, we introduce the spread variable as a complementary descriptor. Given vector with elements sorted by magnitude, we define its spread | E [0, 1) via the relation [0s] = (1 3) Intuitively, 3 quantifies the relative decay from the largest to the k--th largest entry, capturing an additional degree of freedom in the geometry of compressibility, better describing and distinguishing compressible distributions beyond what is possible with approximation error alone.
Lastly, to provide numerical comparison, consider 21 (4.00, 4.00, 0.057, 0.057). The qualitative difference between the two vectors is obvious, and is casy to observe under our compressibility definition: with q 2, k = 2, we have = 0.169, B = 0.75 vs. e 0.014, f = 0.00, respectively. Note that this difference is captured neither by the classical notion of sparsity (neither vector includes any elements), the modern PQ Index, both vectors have PQI(2, 1) of 0.697.
(6.00, 1.50, 0.75, 0.75) and 22
B.2 LOWER BOUNDS ON OPERATOR NORMS
The following theorem characterizes the compressibility-asee lower bounds of operator norms, complementing the upper bounds presented in the main paper.
Theorem B.4. The following statements lower bound operator norms using compressibility and Frobenius norm.
7

---

Published as a conference paper at ICLR 2026
(a) Neuron compressibility (t.e. row-sparsity): Let wi, E [b] denote the rows of the matrix W, and let V == (llwall,....,) (.a...,,,,Wlllll (Walla) denote & norms of its rows. Assuming (1, ( and each row w, (2, kr,6,) compressible implies:
(-w)
| & ||
(67)
(L +)
Ky
(h) Spectral compressibilityy (i.e. low-rankness): Let == (0r, 02,...)
denote the singular values of matrix
W. Assuming a is (1, (o) compressible implies:
hee) W S [|]2-
(68)
Kg
Note that the minimum value this value take is || Vivlla/ku.. By the definition of strict compressibility, we know that | = (1 - e)]|[|||1- This gives us the incquality:
Proof. For (a) note that || |/ ||p/|
cu)
W oc.
(69)
We then turn to the components of v. and examine the relationship between ||w and ||w]] for row w. We will use wa. w, to refer to the dominant and remainder terms of respectively. We invoke Minkowski's inequality:
[ || - Wx ||2 1l.
(70)
We bound the leftmost term by ||we|12 vI 2 w||l vI ||wll2,, we observe that due to interpolation inequality:
c //l1 due to Lemma A.I. For the term
||w/112 I ||w~l1 |wr 1l.
(71)
Examining |/w.lls.. we note that the maximum magnitude w. can contain is less than or equal to the maximum value the lowest magnitude element of can take. This is the case when all elements of w. equal, therefore ||wrl/so I [| /kk Using this, the fact that wx/l2 S ||w||1, and that |/ I c/|w/]a by compressibility definition, we can write:
/w 1|2 w.
ed |wll
[wwl1 k
1/2
|w|1,
vTa
Plugging this back into the additive decomposition of [|w||2 above, we have:
Vi
||wla S ||]]1
(72)
VK( 22)
Let i denote the & norms W's rows. Then, plugging this back to the main incquality:
(v)
2
(73)
hv
Vk
(1 - cv)
2
|11|
(74)
A(1
22) + Ve
hi
Vk
(1 Ex)
|0112
(75)
Vk(1 2) Ve
hi
VT.
(1- Ev]
|||
(76)
Vk(1 -) +
Ve
liv
which gives use the desired inequality.
For (b), we will use x a to refer to the dominant and remainder terms of respectively, Note that W
||a||3 = l + ||-, We bound the norm of the dominant singular values by ||a 1 & ko? = k We bound the remainder singular values by noting that
13
| < (Ila,1112 < (ca [|ol)11) S {VTilloo))) = I WI
(77)
This gives us the inequality:
| < he ||
(78)
Rearranging the terms gives the desired lower bound.
8

---

Published as a conference paper at ICLR 2026
20
Le Op. Nomm
Norm
12 Op. Norm Fro. Norm
400
200
0
100
200
300
400
20
40
60
80
20
40
60
80
Iteration
Initial Matrix
Final Matrix
200
La Op. Norm
12 Op. Norm Fro. Norm
20
20
150
Norm
100
60
50
80
100
200
300
400
20
40
60
80
20
60 Final Matrix
Iteration
Initial Matrix
Figure 9: Optimizing for too (top) and l (bottom) operator norms.
x10
1.40
Rob. Loss Std. Loss
Bat
Op. Norm Prod.
1.20
Bound
25.00
Interlayer 0.68
0.85
0.80
0.60
0.10
622
0.8
15.00
Compeessbilly
Compresi
0..00
0.40
10.00
24.00
Layer Rank (r)
Layer Rank (r)
Layer Rank (r)
Figure 10: Empirically investigating the implications of Theorem 3.2
B.3 RELATIONSHIPS BETWEEN OPERATOR NORMS
Although Theorem 3.1 directly relates loo and l2 operator norms neuron and spectral compressibility, both the known inequality relationships and results two quantities are likely to strongly correlated under this context. conduct simple experiment lest this hypothesis: We optimize for either foc or Eu operator norm of a random i.i.d. Gaussian matrix A where
adversarial attacks imply that these
Aig Lidd. NNO0,), 1). We then conduct a gradient ascent based optimization of the matrix's 's either for or & operator norms, while normalizing the Frobenius random seeds, we show how &o and evolve while either (top) and & (bottom) are optimized. We note that both case both norms are strongly associated increasing simultancously. Note that given the incquality |||||a < || by the end of optimization the spectral norm reaches its limit in Frohenius norm. While the left column shows the norms across iterations, center and right columns portray the qualitative differences produced by optimizing for either columns. As expected, optimizing for foc collects all energy in single row, while optimizing &a produces a 1-rank matrix.
its initialization value. In Figure an average of 10
B.4 EMPIRICAL ANALYSES OF THE ROBUSTNESS BOUND AND RELATED QUANTITIES
In this section. we directly investigate how well our bound correlates with the adversarial robustness gap, as predicted in Corollary 3.3 In order fully conform the setting of Corollary 3. we convert the previously
introduced MNIST dataset binary classification task by converting its labels to 0-1, by assigning 0-4

---

Published as a conference paper at ICLR 2026
Baseline Modell
Rob.. Acc.
Compresiiilty
Robust Accursty
Low Rank Model
0.50
Pub.. Acc. Acc.
0.50
0.80
6.66-
0.48
0.60
0.70
5.64
0.58
0.00
0.62
0.42
Comprexibelty
Rob. Acc
0.50
5.60
0:54
0.40
5.0
Standard Accuracy
0.72
0.30
0.78
10'
Layer Rank (r)
16a
0.00
Learning Rate (n)
0.01
0.82
0.03
Figure 11: Adversarial fine-tuning (left) and training (center). Robust accuracy under increasing learning rate (right).
class O and 5-9 to class 1. We create fully connected network (FCN) with two hidden layers of width 300, with ReLU activations after each layer. We then networks with various spectral compressihility through varying the rank of the hidden layers, imposed through low-rank factorization. While computing the bound, we determine & (num. dominant terms), and compute and 3 as statistics. Note that if 3 = 1, this would make the bound undefined however, instead of being numerical problem. this implies that & should be selected lower. as dominant terms including 0 is undesired comer case. Figure TT demonstrates the results of our experiment. First, Figure (1eft) shows that our bound is closely correlated with adversarial robustness gap. This shows that although our bound is an order of magnitude above the empirical loss difference, it is still faithful indicator of adversarial robustness.
We then investigate whether local input sensitivity of the network tracks its global properties. As in the main paper, letting (xx) and zadv (xx a" denote the learned representations of clean and perturbed input images, we compute || 1 aadv = //2//|aa ||a || 1000 test samples. We take this metric as secant approximation of the local Lipschitz constant around input æ. We then use the maximum and the mean of this statistic over the samples as empirical lower bounds to the global and expected local Lipschitz constants respectively. Figure TO (center) shows that these two values are closely correlated with each other and with glohal bounds: An increase in the maximum sensitivity perturbation is reflected in similar increase in the average sensitivity. Lastly, Figure (right) investigates the effect of spectral compressibility on interlayer alignment, in parallel to product of spectral norms of the layers (to quantify the intra- vs. interlayer dynamics in our bound). Results show that while norms increase expected, interlayer alignment does not necessarily portray consistent pattern. We consider how and why interlayer alignment changes in response to various compressibility inducing sparsity and crucial future rescarch direction.
training dynamics
B.5 APPROXIMATING THE INTERLAYER ALIGNMENT TERMS
Note that the interlayer alignment terms used Theorem 3.2 lead a combinatorial optimization problem due to discreteness of ReLU gradients, i.c. {0, 1 A closely related precedent from the literature is SpqL.i by |caaman& & Virmaux (018). with the differences relating the normalization of the terms, and the k-term adaptation. However, since these differences not lead to any changes with respect to the optimization of these terms th.e, maxima), the authors' approximation methodology is an attractive choice for determining Aç. [Scaman& & Virmaux 22018 report that their gradient-ascent based greedy search algorithm 1% the analytical solution for cases where latter is computationally feasible. We adopt their solution to our case for both interlayer alignment terms.
Compressibility
0.35
0.25
0.30
0.25
0,20
Rob. Act.
Rob. ASt. Acc.
0.15
10-5
10
Row Sparsity Regularization (a)
Figure 12: Effects of standard group lasso on compressibility and adversarial rohustness.
10

---

Published as a conference paper at ICLR 2026
C DETAILS OF THE EXPERIMENTAL SETTINGS AND RESOURCES
C.I DATASETS
Our experiments are conducted using the most commonly utilized datasets chitectures in rescarch on adversarial robustness under pruning
and
ar-
PPira ct al.
2025
Our datasets include MNIST (Deng) 2012). CIFAR-10. CIFAR-100
(Krizhevsky & Hinton 2009). SVHN (Ntter al.
Compreacily
Compres Billy
0.80
0.50
0.73
0.78
2011). Flickr30k Young et al., 2014), and ImageNet- 1k (eng et al. 2009). As de- tailed in Appendix B convert MNIST into a binary classification task for empir- ically investigating how our bound corre- lates with adversarial robustness gap. In all datasets, use the canonical train-test splits. Whenever validation set-based model selection or early stopping is used, we uti- lize 10% of the training set for this task, and conduct carly stopping with patience of 10 epochs based on validation loss.
0.45
0.76
0.68
0.74
0.40
0.65
Rib Acc.
Rob ASc Acc
0.35
0.63
Rob Acc.
Rob Acc. Sld. Abc
10-1
Row Sparsity Reqularization
Layer Rank
0.05
Conprereciity
Comatoo Edl
0.1
0.35
0.10
0.00
0.25
0.08
Ro
0.20
0.00
Rbb. AcC
Acc
0.00
3.0
Soh. Acc
0.15
2012
40 Layer Rank (r)
C.2 MODELS
Row Sparsity Regularization (c)
Figure 13: Results with SVHN & Wide ResNet 101-2 (top).
Architectures we utilize include fully con-
CIFAR-100 VGG16 (bottom).
nected networks (FCN), ResNet18 (He et al.]
2016). VGG16 (Simonyan & Zisserman 2014). WideResNet-1011 101-2 Zagoruyko & Komodakis 2016 vision transformer (VTT)- both standalone classifier (Dosovitskiy
2021) and as part CL.IP encoder [Raffrrd al. (202I) and Swin Transformer (Liu et al. (0.1). Whenever needed, we apply modifications to standard architectures in question. For our visualization experiments at the beginning of Section we utilize a 1-hidden layer FCN with ReLU activation, with no bias nodes, and width of 400. For main results with CIFAR-10, utilize 2000-width FCN with 4 hidden layers, with the remaining architectural choices remain identical. Regarding the VGG16 architecture, due our datasets being size x 32, we remove the redundant 4096-width lincar layers (along with their interleaving dropout and ReLU layers). Lastly, when conducting the low-rank factorization experiments, we modify linear layers with factorized layer, and do the equivalent for 2D convolutional layers (Zhong et al. 2023)
For transformer models, utilize Base ViT architecture with 8 patch size. When tine-tuning a pre- gr-trrined version, we utilize version pretrained on ImageNet-211 and tunet on ImageNet- 1 hosted by the HuggingFace platform (Wolf et al. 2020). For the Swin Transformer we tiny version of the architecture, and utilize ImageNet-1K pretrained version hosted by torchvision (Torch maintainers contributors 2016). For CLIP experiments, we utilize a pre-trained CLIP model, CLiP VIT-B/32, trained on LAION 2B dataset, hosted by Open CLIP (Iharco al. 2021. To conduct the zero-shot classification with the tine-tuned CLIP,, we fine-tune the model with the Flickr30k dataset using a weight decay of 0.01 and a learning rate of le for 30 epochs. For the classification that follows, we present results with top-5 (standard and adversarial) accuracy, and utilize the following prompts to embed the text descriptions, which
(oranatire Rob (nrgndl A0c
0.5
0.
Stucturd aed Como.
unsouctured Coms. Buth
0.
Reg Regulorizaion Stength
0.85
0.80
Comp.
the class vectors:
Strengis
Figure Unstructured alongside struc- tured compressibility, for row/neuron (top) and spectral compressibility (bot- tom).
photo of
blurry photo of
photo of the
a close-up photo of a...
a black and white photo of a..
cropped photo ....
bright photo of
11

---

Published as a conference paper at ICLR 2026
C.3 STANDARD AND ADVERSARIAL TRAINING
Standard training. We normally use softmax cross-entropy loss, the AdamW optimizer with a weight decay of 0.01, learning rate of 0.001, and use validation set based model selection for carly stopping. For adversarial training tasks, we also include a cosine learning rate annealing schedule (epochs 60, min. lcarning rate O), basic data augmentation the form of random cropping and horizontal flips, and an adversarial validation set, again constituting 10% of the training set.
Evaluating and training for adversarial robustness. For evaluating adversarial robustness, we primarily employ the AutoPGD attack (Croce & Hein 2020). using the implementation from Nicolae et al. (018). During adversarial training, we generate adversarial examples at each iteration using the PGD attack (aadry et al. 2018). Unless stated otherwise, adversarial examples make up 50% of each training minibatch. For models trained end-to- tod with adversarial robustness, we set 8/255 for foo attacks and = 0.5 for & attacks. For standard adversarially fine-tuned tunet models, use 25% of these budgets enable clear comparison.
C.4 IMPLEMENTATION AND RESOURCES
Implementation. We utilize the Python programming language and PyTorch deep learning framework for implementation (Paszke et al. 2019). Whenever possible, we utilize the default torchvision (Torch Vision maintainers and contributors 2016) implementations of our models mentioned above. For adversarial training and evaluation, we use the Adversarial Robustness Toolbox (Nicolae et al. 2018). Our source code provides further details regarding implementation""
modify these baselines for the changes
Hardware. All experiments are conducted on the computational server of an institute, utilizing Nvidia L40S GPUs. The main paper experiments took total of 600 GPU hours to complete, including 2 3 seed replication for the main results. Total development time is estimated be 3.5x of the compute time for the final publication.
LLM usage. This work used LLMs to assist in literature search, phrasing, formatting, and implementation.
D ADDITIONAL EMPIRICAL RESULTS
D.1 EXPERIMENTS WITH OTHER DATASETS AND ARCHITECTURES
As mentioned the main paper, we now ex- tend our empirical findings to other datasets and architectures. Figure 13 demonstrates results with SVHN dataset and Wide ResNet 101-2 architecture (top), and CIFAR-100 dataset and VGG16 architecture (bottom). Our results replicate with novel datasets and architectures, as qualitatively identi- cal results obtained in these alterna- tive settings. Note that the slight initial in- crease under neuron compressibility seen with WideResNet 101-2 here and ResNet18 in the main paper cannot be seen with VGG16, highlighting the regime dependence of multiple inductive biases compressibility- inducing regularizations might have.
0.40
0.80
0.30
0.00
0.35
0.0
920
Rob
Rah Acc
0.40
Rob Acc 2t0
Rob. Acc Acc
0.0
10-1
40
100
Row Sparsity Reqularization
Layer Rank
0.25
0.70
0.2
0.30
0.50
011
0.45
Rob,
Bob
Ape
0..0
012
Rob: Acs, (50d. Acc
Rob,
0.43
10-1
10t4
2014
10
20
40 Layer Rank
Row Sparsity Regularization
D.2 REGULARIZING GROUP SPARSITY
Figure 15: Results with CIFAR-10, FCN (top) and ResNet18 (bottom), with alternttiv attack norms.
In the main paper, we highlight that we utilize a scale-invariant version of group lasso to disentangle the downstream effects of increasing compressibility vs. decreasing overall parameter scale. Figure 1 replicates main results ResNet18 and CIFAR- 10 while using standard group lasso regularization. While its effects are mostly similar to our version of group lasso, we note that Figure 12 presents a subtle difference, where group lasso first creates an increase in robustness at very low levels (error bars = I std. deviation). However, as indicated in the main text, these benefits are overtaken by the negative effects of row compressibility as regularization strength increases.
http:://gittub..comm/ttt
12

---

Published as a conference paper at ICLR 2026
D.3 ADVERSARIAL TRAINING RESULTS FOR SPECTRAL COMPRESSIBILITY
Figure [I (left, center) presents the spectral compressibility counterpart for adversarial fine-tuning and training results from the main paper, under & adversarial attacks. The patterns clearly mirror those presented in the main paper under row sparsity conditions.
D.4 COMPRESSIBILITY THROUGH INDUCTIVE BIAS
We now examine whether the results we have observed with explicit regularization methods also apply cases when compressibility is ob- tained through the inductive bias the learning algorithm. For this, we go back to the setting presented in Appendix [ and instead of increas- ing regularization hyperparameter, we increase initial learning rate (1) of the training algorithm. paint an intriguing picture. While initially increas- ing improves adversarial robustness under 30 attacks (perhaps paralleling its well-known ben- efits for standard generalization), soon as it starts to increase row compressibility. its bene- fits of quickly disappear. This highlights the fact that our results not only inform the adversar- ial robustness behavior under explicit regulariza- tion and architecture design, hut also inductive biases of the learning algorithm.
0.25
Cormeniityy
644
2.5
0.70
0.30
6.16
0.0
0.20
340
035
030
Hot Acc/5ol Mcc
Baz Act
235
Seb. Arc
4th AcC/Ht
10t1
10-1
op-r
10-1
12-h
10-7
10-1
Row Sparsity Regularization (4)
Rew Sparsity Regularization [o]
The results, presented Figure
Goomaed
1..0
0.c0
0.28
523
6.
6115
Act
0.54
10-1
Row Sparsity Regulartzation (e)
Row Spansity Regularization (o)
Figure 16: Results CIFAR-10 10, ResNet18 and attacks with FGSM (top left), AutoCG (top right), Square Attack (bottom left), AutoAttack (bottom right).
D.5 UNSTRUCTURED COMPRESSIBILITE
While unstructured compressibility is not the focus of our study, we note that it appears in the bound for L in Theorem 3.2 unlike that for L2. To investigate the significance of this result, we replicate pendix[B] but this time addition to increasing the group lasso/nuclear norm regularization, we run separate set of experiments where either solely increase I.1 regularization, or increase along with structured sparsity- inducing regularization. We then compare the performance of the resulting models under the corresponding adversarial attacks. The results are presented in Figure 14 Remember that bound implies positive effects of unstructured compressibility for L Indeed, in Figure we that LI regularization can compensate for the negative effects of structured compressibility (top), while it has no such benefits for spectral compressibiliity (bottom). We believe that understanding the intricate relationships among different types of compressibility is crucial future research direction.
0.5
Accuracy
setting presented Ap-
0.4
Glsb.
Comp.
0106.
0.0
0.0
0.2
0.4
0.5
1.0
02
0.4
0.5
Pruning Ratio
Pruning Ratio
Figure 17: Post-pruning fine fine-tuning and robustness.
D.6 RESULTS WITH ALTERNATIVE NORMS, BUDGETS, AND ATTACKS
While for brevity we presented our main results to include robustness against foo attacks under neuron sparsity, and &2 attacks under spectral compressibility. for completeness we provide our central results with the gross-norm attacks, Leef. attacks under spectral compressibility, and & attacks under neuron sparsity. The results are presented in Figure T5 and fully in line with the results presented in the main paper.
Model performance under varying attack budgets. described in the main paper, in order to investigate the effects of structural interventions on standard trained models' adversarial robustness, we utilize smaller attack budget to avoid floor effects from obscuring the effects we are investigating. Table aeboos[demosstrtee that our results not dependent various attack budgets: however standard trained models floor effects indeed prevent the observation of the results of our interventions, justifying our utilization of a reduced budget such cases.
specific attack budget, and the patterns that confirm our hypotheses hold across
Model performance under alternative attacks.. We investigate whether our results replicate under alternative attacks. We therefore repeat our experiments with ResNet18 and CIFAR-10 in the main paper with FGSM [Goddfellow al. 2015), AutoCG (Yamamura al. 2022), Square Attack (Andrisschhnkoo et al. 2020), and
13

---

Given that classification accuracy is the most commonly utilized and communicated metric in the literature adversarial robustness, the main paper reports these as primary metric. However, find that same hypothesized patterns can be observed when robust loss standard loss utilized as the main metric, instead fine-tuning experiments described above, replicating our Published as a conference paper at ICLR 2026
Table 1: Robust accuracy of VIT model trained CIFAR-I0, under increasing adversarial sample ratio in
training (p) vs. increasing too attack budgets (6).
0.0
0.05
p 0.1
p 0.25
0.5
0 = 2/255
0.111
0.333
0.479
0.519
0.510
0 4/255
0.002
0.061
0.263
0.371
0.390
0 8/255
0.000
0.002
0.032
0.113
0.179
0 = 16/255
0.000
0.000
0.000
0.005
0.019
the composite AutoAttack (Croce & Hein 2020); as opposed to the original AutoPGD. Results in Figure 16 confirm that our results are qualitatively identical under different attacks.
D.7 FINE-TUNING TUNING RESULTS WITH TRANSFORMERS
As described in the main text and above, we investigate whether we can replicate our results while fine-tuning ChaggeNet-pretraine transformer models, VIT and Swin Transformer, on CIFAR-I0 and SVHN respectively, while utilizing sparsification regularization. The results presented in Table and Table 3 and replicate hypotheses.
Table 2: Robust and standard accuracies of pretrained ViT models fine-tuned CIFAR-10 dataset under varying
neuron sparsification regularization strength (ox), i.e. group lasso.
0.0
0.001
0.005
0.01
0.05
0.1
Rob. Acc. 0.383
0.362
0.369
0.219
0.123
0.111
Std. Acc.
0.920
0.926
0.921
0.893
0.873
0.829
RA/SA
0.416
0.401
0.391
0.245
0.141
0.134
Table Robust and standard accuracies pretrained Swin Transformer models fine-tuned SVHN dataset under varying neuron sparsification regularization strength (a).
Q 0.0
0.001
0.005
0.01
0.05
0.1
Rob. Acc.
0.384
0.360
0.357
0.326
0.155
0.083
Std. Acc.
0.889
0.877
0.887
0.880
0.881
0.875
RA/SA
0.432
0.410
0.402
0.370
0.176
0.095
of accuracy.
these results findings with robust and standard accuracy.
Table Robust standard accuracies and loss differences for pretrained Swin Transformer models fine-tuned on SVHN dataset under varying neuron sparsification regularization strength (a).
0.0
a 0.001
0.005
a 0.01
a 0.05
a 0.1
Rob. Acc. Std. Acc.
0.384
0)360
0.357
0.326
0.155
0.083
0.889
0.877
0.887
0.880
0.881
0.875
Adv. Loss Test Loss
0.505
0.517
0.530
0.554
0.726
0.792
14

---

Published as a conference paper at ICLR 2026
D.8 RESULTS WITH POST-PRUNING PINE-TUNING
Utilizing a baseline model adversarially trained on CIFAR- dataset with ResNet18 architecture, instead regularizing compressibility, we prune and then fine tune our models to investigate 1- whether the main paper's s results will replicate under post-pruning fine-tuning, 2- whether fine-tuning procedure will be another source of vulnerability and itself. After layerwise structured pruning, we fine-tune the models until convergence on the standard validation set. Our results, presented in Figure[17] demonstrate that 1- results from our main paper replicate under post-pruning fine-tuning, and 2- fine-tuning procedure creates independent vulnerability after fine-tuning robustness deteriorates much faster compared to the standard accuracy vs. pre- fine-tuning results. [iguree[Sjjemmossrraal even when post-pruning fine-tuning is adversarial (conducted as defined above). These results significant for both strengthening the main paper's conclusions, and for showcasing another compressibiliy-induds intervention that leads to structure-
0.28
0.35
0.33
0.23
0.20
0.28
Rob. Acc. Acc.
0.25
0.2
0.4
0.5
Pruning Ratio
Figure 18:
Adversarial post post-pruning
that the same results apply
fine-tuning and robustness.
tructure-induced induced vulnerabilities.
D.9 EXPLOITATION OF VULNERABLE LATENT DIRECTIONS
Here we provide a more visual and intuitive walkthrough our proposed mechanisms. Let us consider an MLP with a single hidden layer,
g(x) = Co(Ww),
elementwise ReLU function, and we ignore bias nodes without loss of generality for a cleaner exposition.
where corresponds
Baseline Model (a = 0.0)
10
When two such networks have been trained on dataset with no regularization strong nuclear regularization, can expect latter's to have much more concentrated singular values (SV), i.c. more spectrally compressible.
2
Indeed, in Figure 19 we provide a comparison of two such networks trained on CIFAR-IO (regularization strength 0 vs. 0.05), with hidden layer size 400. We conduct singular value decomposition (SVD) W UEVT, and plot singular values of W for both networks σ := diag(E) = (01, 02, ......). As in the main paper (Figure 2, left), in the compressible model the singular values much more concentrated, creating the vulnerable directions in question.
30
Singular Value Index (I)
Compressible Model (a= 0.05)
10.0
6
5.0
2.5
But what exactly do we mean by attacks "aligning" with and "exploiting" these directions? For this, let us decompose an adversarial sample: aadv = x + where is the clean image and a is the adversarial perturbation. Examine the pre-activation representation of this attack:
Singular Value Index (i)
Figure 19: Comparing singular values of a baseline (top) vs. compressible (bottom) model.
W Bady = W(w + a) = Wæ + Wa.
Note that for given sample Wa and thus W are fixed. Having large |Wa|| (in relation [[Wal))) would make easier for attacker to dominate representation and change the downstream prediction.
So, how does the spikier a in the compressible case help the adversary achieve this? For this, note that for every singular value a, there exist the right and left singular vectors u, and v. constituting the columns of orthogonal matrices U and rows of VT respectively. So, based on the definition of SVD, can write:
Wa 0io of a + u2d2r.a + uzasnla +...
Without loss of generality, let assume a 1, and examine these terms, u a. Note that given both v, and a are unit vectors, v.o a corresponds to cosine similarity the two vectors, very intuitive notion of alignment.
15

---

Published as a conference paper at ICLR 2026
Why would "align" with that has large on (e.g. in the leading SVs of the compressible model)? To see this, let assume a z v1- Then, this would mean v.a a = 1, and vja = 0,Vj This in turn would imply that
How aligned is with Mis?
Let A via//a 0.13. 0.46, 0.20, 0.202
V2
|| |aal2 o u2o20la uzgzvfa +... 2|2
21-
[1 +0+0+ ... ||ui |
01
This means that after this layer got scaled by this large number d1 helping dominate the representation despite the small original attack budget:
Wall W
||a|| 2
V12
V12
This example makes clear why having a few, very large o, as result of compression can create large vulnerability. Note that Nem et al. (2023) also provide complementary theoretical justification regarding the dangers of encoders with such potent directions.
V2.
Figure 20: Examining alignment of single adversarial perturbation with first 20 singular directions.
D.9.1 ATTACK ALIGNMENT WITH LATENT DIRECTIONS
now we move on to question whether this actually happens in prac- tice. We already established that in- creased spectral compressibility ates vulnerable directions. How can we decide whether successful adver- sarial attacks are actually "exploiting" these directions? For any given and its perturbation a, can inves- tigate the "alignment" a with ev- ery singular direction i, we can com- pute = v! |alla where we are now normalizing since a does not have to be unit norm in general. Note that E [-1, 1] mcasure of alignment between v, and a; its absolute value lps can utilized as a notion of alignment strength. An intuitive way to plot how much a sample aligns with each of the first singular directions is to plot this on radar plot/spider plot. See Figure 20 for an example on a single From this graph, we can read that mostly aligns with 11, v4 and
Baseline Model 0.0)
Compressible (a- 0.05)
V11
913
Figure 21: Comparing singular directions exploited by adversaries in bascline (left) vs. compressible (right) model.
We can then such plot derstand overall patterns by plotting multiple samples. In Figure 21] we overlay this plot for 100 different samples for both models, for 7 20. The results strongly support our hy- potheses: While the attacks in the baseline model exploit (i.e. align with) all 20 directions, in the com- pressible model the attacks focus a few strong, vudnerable directions. Then, since the adversaries are us- ing these potent directions at their disposal in the compressible case, we would expect them to dominate the latent representations, compared that this is indeed the case, both for pre-activation ([[a)l aall//1ll W>]la) æ=|22) and post-activation (I) =al = >||2/ ||>[[2) representations. Note that these results replicate the results presented in the main paper's Figure 4, right.
Baseline
Compressible (a 0.05)
Bas - 1
z|p/ Z|: |/Wx)
300
[Wap:/Wx]:
30
200
20
100
10
0
0.0
0.1
0.3
0.5
0.1
Figure 22: Comparing pre-activation [/Wall// [a (22) and post- activation ([]zadv - - representations of baseline (left) vs. com- pressible (right) models.
the baseline model. Indeed, in Figure 22
D.9.2 WHITE BOX AND BLACK BOX ATTACKS EXPLOITING LATENT DIRECTIONS
We now can use this visualization technique to understand the process of adversarial attacks finding these directions. We choose two canonical, extensively cited white box and black hox attacks for this task respectively: PGD (Madry al. 2018) and NES (Ilyas 2018).
16

---

Published as a conference paper at ICLR 2026
Stea
Sep-
Stp4
Sie
Shap
Stea
Stee 10
Stea 23
S0w 30
238
5xx 1063
Figure 23: Utilization of singular directions by white box PGD (top) vs. black box NES (bottom) attack under a compressible model.
As a white box attack that assumes access to parameters, PGD is able to conduct iterative first order optimization on the image to find potent attack direction, projecting back to the ball after each iteration. Since in a compressible model, aligning with these sensitive directions would quickly increase the loss, the optimization algorithm very quickly finds these directions through its optimization objective. We present the iterates of PGD perturbation on a single image under the compressible model, see Figure 23 (top). Note that the perturbation quickly aligns with a very strong singular direction, Ut; so much so that by the 6th iteration, the algorithm converged on the attack already.
How can a black box attack make use of these exploitable directions? For this, let us take a closer look at NES. Being black box attack. NES assumes only access the logits, prohibiting the of standard backpropagation. Instead, at every step. NES creates N random Gaussian perturbations and evaluates the loss for all of them. It then calculates a weighted average of these directions (weighted by their impact on the loss) to estimate a prexy gradient, and update the adversarial perturbation accordingly. This means that whenever these random perturbations align, even slightly, with any of the exploitable latent directions, they would dominate the weighted average, effectively pulling the estimated gradient toward the vulnerability. So, although not as efficiently, without an explicit knowledge of the parameter space, NES can locate these adversarially exploitable directions, just by querying the input space. Indeed, the image in Figure 23 (bottom) presents a direct confirmation of this hypothesis: although it takes many more steps (~ 200) due to the randomness of its perturbations, NES can also converge exploiting the vulnerable directions in the latent space.
Note that we focused on the lg attacks in this exposition; however note that it is quite straightforward to apply a similar analysis to loo case, where the most vulnerable directions are rows wa in W with the largest [[
D.9.3 INTERLAYER ALIGNMENT
Following from previous example, let us now assume a two layer neural network g(z) = C of ((W")) we will use superscripts to denote the compoments of layers as well. As a simple example, assume that both layers have a single, very large SV, and rest of their SVs are ss O. This implies that both have potent singular direction that can potentially be exploited. However, these directions between layers will need to "align" for their impact to accumulate. More concretely, note that a unit perturbation a that aligns with the right singular vector of the first layer T in the input space a RJ vf will be "amplified" by oi. The resulting output will be in the direction of the left singular vector, i.c. uf Ignoring nonlincarity for now, as large as this intermediate representation can be, if it's not in the direction of the next layers' large SV, it will be effectively ignored. For example, at the extreme end, if (vl) uu z O, the attack will effectively disappear before reaches the final representation and can impact the prediction. This is because the second layer we will have || ufll2 Rs O. So, such theory will have to take into account how such signals are relayed between layers, while factoring in nonlinearity.
Note that Theorem 3.2 upper bounds LY the p-norm Lipschitz constant of the encoder. This can be computed as the maximum p + p operator norm of the Jacobian:
T.2 sup [[ sup | (<"" (t.":/..........(/W" =
D'
(79)
=ex
where the diagonal binary D' (xx terms stand for the ReLU nonlinearity. Notice that input dependence of these terms introduce a combinatorial complexity, making it infeasible to directly optimize this term. Our Theorem 3.2, like all other attempts the literature, utilizes an approximation of this monolith.
17

---

Published as a conference paper at ICLR 2026
Given that D 1, and submultiplicatiiity of the operator norm, it is possible to write:
L
S sup #Ex
(x) (=) (.) Ir.
(80)
| Il............... |..
(81)
While this is valid, notice that it corresponds to very pessimistic assumption: It looks how much every layer can maximally "stretch" an incoming vector, and multiplies this across layers. This assumes that all "worst case" directions in consecutive layers exactly line up.
Instead, in our bound, while layer operator norms appear (through their compressibilty--basde decomposition), interlayer alignment terms, (t) S 1, act correction term.
L < W 1 -1e,, Ap
Ap
(82)
It approximates and factors in how much dominant directions actually align in consecutive layers. Every Ap(!) consists of terms: the main term that computes the alignment and a remainder term that goes to as compressibility increases. See Appendix [B.5] how we approximate this (much more manageable) combinatorial computation. We refer the reader to our proofs for full
Adv. Robuet Accuracy
the dominant directions,
0.30
0.25
0.15
derivation of these terms. Note that while this term not the main focus of our paper, Figure includes empirical investigation of this term, and demonstrates that it does not have a strong directional relationship with compressibility.
10-3
10-2
Nuclear Norm Reguarization (o)
Figure 24: Effects regularizing in- terlayer alignment (ILA).
Utilizing interlayer alignment for regularization. In order provide more comprehensive examination this term, we conduct experiments that test whether this term can used as another theoretically inspired intervention for robust compressibility. With linear approximation to this term (regularizing (VV ([?), we this hypothesis. Results, presented in Figure 24 are directly in line with our predictions: regularizing interlayer alignment between layers lead to a tangible increase robustness under compressibility. While some mild computational hurdles need to be addressed for full practical utility, these results both provide new intriguing research direction for robust compression, as well as serving as yet another confirmation of our theory.
D.10 COMPARISON WITH ADVERSARIAL PRUNING LITERATURE
As discussed in the main paper, we consider our work to be complementary to those the field of adversarial pruning (Piras)
2. Operator Nomm
15-
HARP
et al. (025). More specifically, our theory implies that compress- ibility hurts robustness insofar as it increases operator norms and
Fiter fruning
creates adversarially vulnerable directions in the latent space; we thus investigate two structured adversarial pruning methods from the literature to see whether these successful adversarial pruning methods implicitly control operator norms. For this, we
10
5-
investigate HARP (Zhao & Wressnegger (023) and grouped kemee pruning (GKP) (Zhong et al.] We choose these two as they have distinct motivating hypotheses. neither of which is in common with ours in a meaningful way. We use both pa- pers' official repositories to conduct adversarial pruning with a ResNet18 8 CIFAR- 10. As baselines, for HARP we train uniform//ayerrise pruning algorithm with standard training set, for GKP we replace grouped kernel pruning with standard filter pruning.
10
15
Layer Index
22.5
Filter Pruning
17.5
25.0
10.0
7.5
After the training, we measure the lor operator norms for both methods and compare it their baselines. Intriguingly, shown Figure 25 although neither method conducts operator norm control explicitly, we find that both end up controlling operator norms indirectly. Note that this cannot just be by-product of adversarial training as GKP relies solely on filter restructuring, and does not involve any adversarial training. We find this to be a promising first finding towards comprehensive understanding of robust structured compression.
Layar
Figure 25: Operator norms of models under adversarial pruning vs. baselines.
D.11 FURTHER EXPERIMENTS WITH UAES
18

---

Published as a conference paper at ICLR 2026
We first replicate original results with a ResNet18 trained on CIFAR-IO Figure 26 Note that the z-axis this par- ticular figure represents the fixing of the Frobenius norms to z-times their initial- ization norms- this allows fix using a common value layers that have widely different widths (while for FCN we used a single constant). Our results qualitatively replicate those in the main paper.
0.70
0.60
0.80
Rob. Std
0.50
Bob. Acc. SStd.
0.40
Adv. Accuracy Unw, Adv. Acoursy
AdV Accurscy
Adk.
0.30
0.29
0.30
0.19
10tH
10-3
10-1
10-3
Frobenius Norm
Row Sparsity Regularization (a)
Figure 26: Robustness against standard vs. universal adversarial attacks under changing Frobenius coefficient (left) group lasso (right).
To further probe this causal relationship. we conduct adversarial training with ResNet18s under increasing compressibility. Importantly, we conduct the training either with standard adversarial examples vs. UAEs. Given the computational challenges of computing UAEs at every iteration, the chcaper FGSM attack for universal and standard adversarial samples, generated from 0.1 of the input batch. Results, presented Figure [27 illustrate the average spread (3) of the dominant terms in the networks under UAE standard adversarial training. Our findings show that UAE training dramatically reduces spread of the dominant terms compared to standard adversarial training, implying that just as creation vulnerable latent directions allow UAEs, training against them reduces the potency of such directions, providing convergent evidence for our hypotheses.
Tosmn) Usd Ad Teareny
Sprert
10-2
10"2
10"2
Row Sparsity Regularization (a)
Figure 27: and UAEs.
19


Reviews:
[{"review_id": "tqx9qAJzp7", "reviewer": "Reviewer_TtqN", "summary": "This paper develops a framework to investigate the effect of structured sparsity on adversarial robustness through its effect on parameter norms and the network's Lipschitz constant. Compressibility can induce a set of highly sensitive directions in the representation space.", "questions": "1. Can you please offer a motivation example of how compressibility might be taken advantage of by adversaries? Especially how would adversaries figure out the \"adversarial directions\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is in general well-written and presents results clearly.\n2. The motivating hypothesis is very interesting and described clearly in Figure 2.\n3. Abundant numerical results are provided to testify the paper's theoretical results.", "weaknesses": "1. One central claim of this paper is that the compressibility may result to a few potent direction that increases the sensitivity to perturbations, and the adversarial attacks might exploit these directions. However, I cannot picture when and how the advesaries might be able to figure out these directions. Is the neural network and the compressibility totally white-box to the adversaries, which could hardly happen?\n2. The evaluation of adversarially robustness of NN models seems to be dependent on the attack itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sfzcsMTFzb", "reviewer": "Reviewer_Tszb", "summary": "The paper investigates the fundamental relationship between network compressibility and adversarial robustness. The claim is that their interaction remains poorly understood. They show theoritical bounds and also empirical evaluation on architectures (FCNs, CNNs, VIT) and multiple datasets. Results show that Increased neuron or spectral compressibility consistently reduces adversarial robustness, even under adversarial training.", "questions": "- Why focus exclusively on structured compressibility (neuron/spectral)? Would unstructured or other forms behave differently?\n- In Fig1,2  did not understand how to interpret these new directions? How were they visualized?\n- The alignment equation notations could be explain bit better\n- Line 325, how does Fig4 prove the dominant singular directions claim?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Paper is well motivated and well written\n- Provides a well-explained theoretical contribution between compressibility and adversarial robustness, tying together concepts from pruning, low-rankness, and Lipschitz theory.\n- Empirical analysis covers diverse architectures and datasets, including FCN, convolutional and transformer families, and multiple attack settings", "weaknesses": "- The theory uses global operator norm–based Lipschitz bounds.\nand The bounds rely on scale-normalized parameters (using ∥W∥_F) and strict (q, k, ε)-compressibility. Can this reflect practical training dynamics with normalization layers or adaptive scaling or deep non-linear networks.\n- How does it position itself with other works exploring the same paradigm [1][2], As some works claim that some sparsity helps robustness \n- The claim that compressibility fosters universal adversarial examples is intriguing but briefly demonstrated\n\n[1] Lipschitz Constant Meets Condition Number: Learning Robust and Compact Deep Neural Networks\n[2] Robust low-rank training via approximate orthonormal constraints", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYVwZ6XN2w", "reviewer": "Reviewer_LZxs", "summary": "This paper proposes a systematic theoretical framework for analyzing how structured compressibility — specifically,  neuron-level compressibility and spectral compressibility, affect model’s adversarial robustness. The authors propose to characterize l∞ and l2 operator norms of the parameters by an upper bound that decomposes into (compressibility × Frobenius norm) terms. Building on this formulation, they further derive and analyze an upper bound on the network’s overall Lipschitz constant. They show that compression introduces a few highly sensitive directions that can significantly amplify perturbations, which can then be exploited by attackers, ultimately leading to degraded robustness. The experimental section covers a wide range of architectures, such as FCNs, CNNs and Transformers, validating the theoretical predictions that models with higher structured compressibility are more vulnerable to adversarial perturbations. The results also demonstrate that the vulnerabilities induced by compression persists even under adversarial training and transfer learning, and facilitates the emergence of universal adversarial perturbations.", "questions": "1. The main theoretical results seem to formalize known connections between compressibility, operator norms, and robustness. Could the authors further clarify the new theoretical insights or perspectives provided by their analysis beyond existing results?\n2. While the paper formalizes an upper bound on adversarial robustness in terms of compressibility and Frobenius norm, the bound appears relatively loose—being an order of magnitude above the empirical robustness gap as the authors describe in appendix. As such, its practical utility for predicting or guiding robust model design seems limited.\n3. The theoretical analysis appears focused on ℓ₂ and ℓ∞ perturbations. Does the same framework extend to other robustness notions (e.g., ℓ₁, distributional robustness, or label noise)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. Provides a unified norm-based framework connecting structured compressibility and adversarial robustness.\n2. Characterizing the l∞ and l2 operator norms of the parameters by decomposing the effects into compressibility and Frobenius norm terms, thereby further formalizing an upper bound on the model’s Lipschitz constant.\n3. The analysis shows that the impact of compressibility on robustness persists in adversarial training and transfer learning, and it can facilitate the emergence of universal adversarial perturbations.\n4. The theoretical analysis, mathematical derivation, and experimental process are relatively complete.", "weaknesses": "Although the theoretical analysis is mathematically sound and logically consistent with prior robustness theory, the overall reasoning builds on well-known intuitions (ideas such as model structured compression concentrates sensitivity along a small number of directions in representation space, which in turn results in decreased robustness), and the upper bound mainly formalize this intuition rather than uncover new mechanisms. The experiments, though thorough, largely confirm expected behaviors without surprising counterexamples or deeper causal probes. The theoretical results appear to just extensions, restating known connections in a more formalized way. In addition, the two interventions mentioned in the paper for improving robustness also seem to have been studied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ALB52sTUH", "reviewer": "Reviewer_1FKt", "summary": "To understand the impact of model compression on adversarial robustness, this paper presents an adversarial robustness bound that interprets how structured and spectral compression induce adversarial vulnerability through their effects on the model's Lipschitz constant. Extensive experiments further demonstrate the detrimental impact of model compression on adversarial robustness.", "questions": "**Q1: The rationale of interlayer alignment**\n\nI acknowledge the importance of the alignment between consecutive layers. However, in Lines 216–236, it is not clear what *interlayer alignment* exactly refers to. Moreover, I am curious about the rationale behind the formal definitions of $A^{\\ast}_{\\inf}$ and $A^{\\ast}_2$. How do these two terms represent interlayer alignment?\n\n**Q2: The impact on robustness improvement with small compressibility**\n\nFor CNNs (Figure 5) and Transformers (Figure 6), a relatively small spectral compressibility can result in a slight improvement in adversarial robustness. How can this positive effect be explained according to the proposed theory?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Valuable insight into model compressibility in terms of adversarial robustness**\n\nThis work provides a unified theoretical analysis that helps to clarify the relationship between model compressibility and adversarial vulnerability without being constrained to a specific norm-based perturbation.\n\n**Extensive experiments across various settings**\n\nThis paper presents extensive experiments across different model architectures, learning mechanisms, and adversarial example generation methods, offering a thorough and comprehensive analysis of the relationship between structured compressibility and adversarial robustness.", "weaknesses": "**W1: The scenario of generating model compression is unclear.**\n\nModel compression can be achieved either through a fine-tuning procedure using the full or partial training dataset, or in a data-free manner. In particular, for the former case, model compressibility is closely related to the feature representations of the training dataset in terms of model capacity. I would assume that the considered compressibility is restricted to a fine-tuning procedure; however, a clearer introduction to the model compression setup is needed before the discussion of compressibility.\n\n**W2: Representation of sensitivity along a small number of directions.**\n\nThe authors claim that structured compression induces high sensitivity to adversarial perturbations along a small number of directions. However, this conclusion cannot be directly drawn merely from the observed correlation between compressibility and adversarial robustness. Lines 216–223 explain that the potent attack directions are determined by interlayer alignment; however, these directions are neither visualized nor formally defined.\n\n**W3: Unclear explanation for the amplification of adversarial attacks in the representation space.**\n\nThe authors attribute the high adversarial vulnerability of compressed models to amplification along certain sensitive directions. However, it remains unclear how the adversarial attack is amplified and what mechanisms cause this amplification to occur.\n\n**W4: Lack of analysis on robustness-aware model pruning techniques.**\n\nThis work discusses compressibility mainly after model training. However, many prior approaches achieve better preservation of both standard accuracy and adversarial robustness simultaneously during structured model pruning [1,2,3]. Despite the observed proportional relationship between compressibility and adversarial vulnerability, an additional analysis is needed to investigate the impact of adversarially robust model pruning.\n\n---\n### References\n\n[1] Zhao and Wressnegger, \"Holistic Adversarially Robust Pruning\", ICLR 2023.  \n[2] Sehweg et al., \"Hydra: Pruning Adversarially Robust Neural Networks,\" NeurIPS 2020.  \n[3] Ye et al., \"Adversarial robustness vs. model compression, or both?\" ICCV 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)