Robustness Reprogramming for Representation Learning
Authors: Zhichao Hou
MohamadAli Torkamani
Hamid Krim
Xiaorui Liu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Adversarial Robustness
Robustness Reprogramming
Robust Representation Learning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: This work tackles an intriguing and fundamental open challenge in representation learning: Given a well-trained deep learning model, can it be reprogrammed to enhance its robustness against adversarial or noisy input perturbations without altering its parameters?
To explore this, we revisit the core feature transformation mechanism in representation learning and propose a novel non-linear robust pattern matching technique as a robust alternative. Furthermore, we introduce three model reprogramming paradigms to offer flexible control of robustness under different efficiency requirements. Comprehensive experiments and ablation studies across diverse learning models ranging from basic linear model and MLPs to shallow and modern deep ConvNets demonstrate the effectiveness 
of our approaches.
This work not only opens a promising and orthogonal direction for improving adversarial defenses in deep learning beyond existing methods but also provides new insights into designing more resilient AI systems with robust statistics. 
Our implementation is available at https://github.com/chris-hzc/Robustness-Reprogramming.

Published as a conference paper at ICLR 2025
ROBUSTNESS REPROGRAMMING FOR REPRESENTATION LEARNING
Zhichao Hou', MohamadAli Torkamani", Hamid Krim', Xiaorui Liu!
North Carolina State University "Amazon Web Services zhou4 @ncsu. edu, alitooramazee com, ahk@ncsu. edu, x1iu96encsu.
ABSTRACT
This work tackles an intriguing and fundamental open challenge in representa- tion learning: Given a well-trained deep learning model, can it reprogrammed to enhance its robustness against adversarial or noisy input perturbations without altering parameters? To explore this, we revisit the core feature transforma- tion mechanism in representation learning and propose a novel non-linear robust pattern matching technique as a robust alternative. Furthermore, we introduce three model reprogramming paradigms to offer flexible control of robustness der different efficiency requirements. Comprehensive experiments and ablation studies across diverse learning models ranging from basic linear model and MLPs to shallow and modern deep ConvNets demonstrate the effectiveness of our ap- proaches. This work only opens a promising and orthogonal direction for improving adversarial defenses in deep learning beyond existing methods but also provides new insights into designing more resilient AI systems with robust statis- tics. Our implementation is available at https://gthubbboom/////ss//////s RReprggrammig
INTRODUCTION
Deep neural networks (DNNs) have made significant impacts across various domains due to their powerful capability of learning representation from high-dimensional data (LeCun et al.] [015] Goodfellow et al. 2016). However, it has been well-documented that DNNs are highly vulnera- ble adversarial attacks (Szegedy) 2013] Biggio et al., 2013). These vulnerabilities are prevalent across various model architectures, attack capacities, attack knowledge. data modalities, and pre- diction tasks, which hinders their reliable deployment in real-world applications due to potential economic, ethical, and societal risks (Al 2023).
In light of the burgeoning development of AI, the robustness and reliability of deep learning models have become increasingly crucial and of particular interest. A mount of endeavors attempting to safeguard DNNs have demonstrated promising robustness, including robust training (Madry) [017]]
ZZhag et al. 2019 Gowal et al., 2021; Li & Liu (003). regularization (Cisse al. 2017 Zheng
et al.. 2016). and purification techniques (Ho & Vasconcelos 2022 Nie et al. 2022 Shi et al. 2021
Yoon et 2021, However, these methods often suffer from catastrophic pitfalls like cumbersome training processes or domain-specific heuristics, failing to deliver the desired robustness gains in an
efficient and adaptable manner.
Despite numerous advancements in adversarial defenses, an open challenge persists: Is it possible to reprogram well-trained model to achieve the desired robustness without modifying its parameters? This question is of particular significance in the current era of large-scale models. Reprogramming without training is highly efficient, as the pretraining-an and-nnntunnng paradigm grants access to sub- stantial open source pre-trained parameters, climinating the need for additional training. Moreover, reprogramming offers an innovative, complementary. and orthogonal approach to existing defenses, paving the way to reshape the landscape of robust deep learning.
To address this research gap, we firstly delve deeper into the basic neurons of DNNs to investigate the fundamental mechanism of representation learning. At its core, the linear feature transforma- tions serve as an essential building block to capture particular feature patterns of interest in most deep learning models. For instance, the Multi-Layer Perceptron (MLP) fundamentally consists of

---

Published as a conference paper at ICLR 2025
multiple stacked linear mapping layers and activation functions; the convolution operations in Con- volution Neural Networks (CNNs) (He et al.] 2016) execute a linear feature mapping over local patches using the convolution kernels; the attention mechanism in Transformers (Vaswami 2017) performs linear transformations over the contextualized token vectors. This linear feature mapping functions Linear Pattern Matching by capturing the certain patterns that are highly correlated with the model parameters. However, this pattern matching manner is highly sensitive to data perturba- tions, which explains the breakdown of the deep learning models under the adversarial environments.
To this end, we propose a novel approach, Nonlinear Robust Pattern Matching, which significantly improves the robustness while maintaining the feature pattern matching behaviors. Furthermore, we also introduce a flexible and efficient strategy, Robustness Reprogranminng, which can be deployed under three paradigms to improve the robustness, accommodating varying resource constraints and robustness requirements. This innovative framework promises to redefine the landscape of robust deep learning. paving the way for enhanced resilience against adversarial threats.
Our contributions can be summarized as follows: (1) We propose new perspective on represen- tation learning by formulating Linear Patten Matching (the fundamental mechanism of feature extraction in deep learning) as ordinary least-square problems; (2) Built upon our novel viewpoint, we introduce Nonlinear Robust Pattern Matching as an alternative robust operation and provide the- oretical convergence and robustness guarantees for its effectiveness; (3) We develop innovative and adaptable strategy, Robustness Reprogramming. which includes three progressive paradigms to enhance the resilience of given pre-traine pre-trained models; and (4) We conduct comprehensive experiments to demonstrate the effectiveness of our proposed approaches across various backbone architectures, using multiple evaluation methods and providing several insightful analyses.
2 RELATED WORKS
Adversarial Attacks. Adversarial attacks can generally be categorized into two types: white-box and black-box attacks. In white-box attacks, the attacker has complete access to the target neural network, including its architecture, parameters, and gradients. Examples of such attacks include gradient-based methods like FGSM (Goodfellow et al.) 20,4), DeepFool (Moosavi-Dezforl et al. 2016), PGD (Madry 2017), and C&W attacks (Carlini & Wagner (017). On the other hand, black- box attacks do not have full access to the model's 's internal information; the attacker can only use the model's s input and output responses. Examples of black-box methods include surrogate model-based method (Papernot et al.] (2017). zeroth-order optimization (Chen et al. (0177), and query-efficient metthoss/Andrusschhnnke et al. 2020 Alzantot et al., 2019). Additionally, Auto/ttack (Croce & Hein 2020b), an ensemble attack that includes two modified versions of the PGD attack, a fast adaptive boundary attack (Croce & Hein 2020a), and black-box query-efficient square attack (An-) driushchenko et al. 2020). has demonstrated strong performance and is often considered as reliable benchmark for evaluating adversarial robustness.
Adversarial Defenses. Numerous efforts have been made to enhance the robustness of deep learn- ing models, which can broadly be categorized into empirical defenses and certifiable defenses. Em- pirical defenses focus on increasing robustness through various strategies: robust training meth-
ods (Madry) 2017 Zhang et al., 2019 Gowal et al. 2021 Li & Liu 2023] introduce adversarial perturbations into the training data, while regularization- based approaches (Cisse et al. 2017 Zheng et al. 2016) stabilize models by constraining the Lipschitz constant or spectral norm of the weight matrix. Additionally, detection techniques Metzen et al. (2017); Feinman al. (2017) ) Grosse al.
2017) aim to defend against attacks by identifying adversarial inputs. Purification based approaches
seek eliminate the adversarial signals before performing downstream tasks (Ho Vasconcelos)
2022 Nie et al.) 2022] Shh et al. 2021] Yoon et al. 2021). Recently, some novel approaches have emerged by improving robustness from the perspectives of ordinary differential equations (Kang et al. 2021 Li et al., 2022 Yan et al.] 2019) and generative models (Wang et al., 2023) Nie et al.)
2022 Rebufhi et al. 2021). Beyond empirical defenses, certifiable defenses (Cohen et al. 2019
Gowal et al. 2018 Fazlyab et al. 2019) offer theoretical guarantees of robustness within specific regions against any attack. However, many of these methods suffer from significant overfitting issues or depend on domain-specifie heuristics, which limit their effectiveness and adaptability in achieving satisfying robustness. Additionally, techniques like robust training often entail high computational and training costs, especially when dealing with diverse noisy environments, thereby limiting their
2

---

Published as a conference paper at ICLR 2025
scalability and flexibility for broader applications. The contribution of robustness reprogramming in this work is fully orthogonal to existing efforts, and they can be integrated for further enhancement.
ROBUST NONLINEAR PATTERN MATCHING
In this section, we begin by exploring the vulnerability of representation learning from the perspec- tive of pattern matching and subsequently introduce a novel robust feature matching in Section 3.1 Following this, develop Newton-IRLS algorithm, which is unrolled into robust layers in Sec- tion 3.2 Lastly, we present a theoretical robustness analysis of this architecture Section 333
Notation. Let the input features of one instance be represented as æ (II,..., E ER, and
(a1...., aD) RR where D denotes the feature dimension. For simplicity, we describe our method in the case where the output is one-dimensional, i.e., 2 E IR.
the parameter vector
3.1
NEW PERSPECTIVE REPRESENTATION LEARNING
Linear Pattern Matching (LPM)
Nonlinear Robust Pattern Matching (NRPM)
D
Ed=1 Wa aaxa
Formula
7LPM
ad
(k+1) 2RPM
Σa Wd
(k)
d=1
Optimization
min
(z/D a
min 2
12/D aaxal
d=1
Algorithm
Closed-form Optimal Solution
Newton-IRLSS Algorithm
x1
Model
X2
X3
7,PM
ZNRPM
(One Layer) :
Ap
XD
Figure 1: Vanilla Linear Pattern Matching (LPM) vs. Nonlinear Robust Pattern Matching (NRPM).
Fundamentally, DNNs inherently function as representation learning modules by transforming raw data into progressively more compact embeddings (LeCun et al. 2013 Goodfellow et al.) 2016). The linear feature transformation, deep learning models to capture particular feature patterns of interest. Specifically, a certain pattern be captured and matched once it is highly correlated with the model parameter a.
a Ex=1 ag Ed. is the essentially building block of
Despite the impressive capability of linear operator in enhancing the representation learning of DNNs, the vanilla deep learning models have been validated highly vulnerable (Szegedy) [013] Biggio et al.] 2013). Existing approaches including robust training and regularization tech- niques (Madry] 2017 Cisse al. 20177; Zheng al., 2016) aim improve the robustness of feature transformations by constraining the parameters with particular properties. However, these meth- ods inevitably alter the feature matching behaviors, often leading clean performance degradation without necessarily achieving improved robustness.
Different from existing works, aim introduce a novel perspective by exploring how design an alternative feature mapping that enhances robustness while maximally preserving feature match- ing behaviors. First, we formulate the linear feature pattern matching the optimal closed-form solution of the following problem:
D
min C(x) = Σ
Od Id
#ER
D
d=1
3

---

Published as a conference paper at ICLR 2025
where the first-order optimality condition (C(=) 0 yields the linear transformation 2² =Ea=1 ad Dd- Since this estimation highly sensitive to outlying values due to the quadratic penalty, we propose derive robust alternative inspired by the Least Absolute Deviation (LAD) estimation:
min ((z) RER
Σ
Od Zd
By replacing the quadratic penalty with linear alternative on the residual z/D - Ogg Fia, the impact of outliers be significantly mitigated according to robust statistics (Huber & Ronchetti 2011).
Methodology. The NRPM architecture draws significant inspirations from optimization-iiieee inddued
deep learning architectures (Ma et al. 2022 Fan al. 2022 Hou al. bb Liu et 2021b Hou
et al. a), where robust alternatives are derived from robust optimization formulations. Specifically, reinterprets the linear feature pattern matching the backbone models as solution well-defined optimization objective, and corresponding optimization algorithm is developed to solve the problem induced by the nonlinear robust pattern matching. This approach highlights a promising direction for designing principled and inherently robust deep learning architectures.
3.2 ALGORITHM DEVELOPMENT AND ANALYSIS
Although the LAD estimator offers robustness implication, the non-smooth objective in Eg. (I) poses challenge designing an efficient algorithm to be integrated neural network layers. To this end, we leverage the Newton Iterative Reweighted Least Squares (Newton-IRLS) algorithm to address by optimizing an alternative smoothed objective function U with Newton method. In this section, we will first introduce the localized upper bound for in Lemma 3.1 and then derive the Newton-IRLS algorithm to optimize L.
the non-smooth nature of the absolute value operator
Lemma 3.1. Let L22) be defined Eq. 4 and for any fixed point 20, U(2 20) defined as
D
U( =0) = Σ
wa (aata - 2/ -f(2)))
(2)
d=1
where We
Aaazz- 20/D Then, for any 2, the following holds:
(1) (1), > ((2), (2),(z2)22) 20) L(20).
Proof. Please refer to Appendix B.I
The statement (1) indicates that Uf 20) serves upper bound for L(²), while statement (2) demonstrates that (2, 2n) equals [(2) at point zu. With fixed 20, the alternative objective 4(2 20) in Eq. (2) is quadratic and can be efficiently optimized. Therefore, instead of minimizing the non- smooth C(2) directly, the Newton-IRLS algorithm will obtain 2 by optimizing the quadratic upper bound Ud2z, 2(k) with second-order Newton method:
Ed Wy addd
(1+11) =D=
D
(k)
d== wi
where w/ (a.=a-2(4//D) Please refer Appendix B.2 for detailed derivation. As consequence
of Lemma 3.1 we can conclude the iteration (k) k= fulfill the loss descent of Z(2):
((2(++1) < ((+(++1) 2( < ((()()))2(0)) = ((2(k)).
This implics Eq. 3) can achieve convergence of L by optimizing the localized upper bound U.
Implementation. The proposed non-linear feature pattern matching is expected to improve the ro- bustmess against data perturbation in any deep learning models by replacing the vanilla linear feature transformation. In this paper, illustrate its cases through MLPs and convolution models. We provide detailed implementation techniques in Appendix [A] Moreover, we will demonstrate how to leverage this technique for robustness reprogramming for representation learning in Section 4
4

---

Published as a conference paper at ICLR 2025
3.3 THEORETICAL ROBUSTNESS ANALYSIS
In this section, we conduct a theoretical robustness analysis comparing the vanilla Linear Pattern Matching (LPM) architecture with our Nonlinear Robust Pattern Matching (NRPM) based on in- fluence function (Law 1986). For simplicity, we consider a single-step case for our Newton- IRLS algorithm (K 1). Denote the weighted feature random variable as and correspond- >6 4X=aaza>- Then we can represent LPM as 2LPM :- (F) = =E Odta and NRPM as 2NRPM == (WRPM(F) = D- ΣP Wanaad
ing empirical distribution as F(X)
2d wa
where wi -2 We derive their influence functions in Theorem 3.2 to demonstrate
their sensitivity against input perturbations, with proof presented in Appendix B.3
Theorem 3.2 (Robustness Analysis via Influence Function). The influence function is defined as the sensitivity of the estimate to a small contamination at A
T(F.) T(F)
IF T,F) = lim
c-0
where the contaminated distribution becomes F. = (1 - e) + cda. where 8 is the Diruc delta function centered at Ax and F is the distribution of x. Then, we have:
IF(Ax;TTPM:1 D(Ar ZLPM/D),
and
1F(Ac:TNRMM,H) F) = DWAK (Az /NRPM/D) where war =
1
Ed=1 Wa
A - 2LPM/
Theorem 3.2 provide several insights into the robustness of LPM and NRLPM models:
For LPM, the influence function is given by D(Ar y) D(Az 2LPM/D), indicating that the sensitivity of LPM depends on the difference between the perturbation Ar and the average clean estimation 2LPM/D D.
For NRPM, the influence function is DwAr Ar-snnarr))) where WAr AL-ELLPMDD Al- Σi wh
though the robustness of NRPM is affected by the difference Ar 2NRPM /D, D, the influence can be significantly mitigated by the weight waz, particularly when Ax deviates from the average estimation ELPM/ / D of the clean data.
These analyses provide insight and explanation for the robustness nature of the proposed technique.
ROBUSTNESS REPROGRAMMING
In this section, it is ready introduce the robustness programming techniques based on the non- linear robust pattern matching (NRPM) derived in Section 3.1] One naive approach is to simply replace the vanilla linear pattern matching (LPM) with NRPM. However, this naive approach does not work well in practice, and we propose three robustness reprogramming paradigms to improve the robustness, accommodating varying resource constraints and robustness requirements.
While NRPM enhances model robustness, it can lead decrease in clean accuracy, par- ticularly in deeper models. This reduction in performance may be due the increas- ing estimation error across layers. To balance the dobastrobsss performance trade-off be- tween LPM and NRPM, it is necessary to de- velop a hybrid architecture as shown in Algo- rithm where their balance is controlled by hyperparameters { and L is the number of layers in the entire model. Based on this hy- brid architecture, we propose three robustness reprogramming paradigms shown in Figure]
Algorithm 1 Hybrid Architecture
Require: { {ad) [ X.
Initialize =NRPM = 21.PM = =ED aa Ed
(0)
for k = 0.1..... K - do
w/ 10 (6)
(k)
Yd [D]
(k-1)
D En w
agza
2RPM
D w
end for
return X ZLPM + (1 - X) N
(K)
5

---

Published as a conference paper at ICLR 2025
Paradigm
Paradigm 2
Paradigm 3
1 0.7
1 0. 459
0. 925
42 0.7 2 = 0. 033
A 0. 119
A3 0.7
23 = 0. 131
A = 0.325
Frozen À Frozen Params
Learnable À Frozen Params
Learnable À Learnable Params
Pretrained
Fine-tuned
A LPM + (1-A) NRPM
Parameters
Parameters
Figure 2: Three Robustness Reprogramming Paradigms: (1) Paradigm freezes the model param- eters and treats {Ne as fixed hyperparameter; (2) Paradigm freezes the model parameters but allows {Ar) 1 be learnable; (3) Paradigm 3 enables both the model parameters and {Mr to be learnable.
Paradigm 1: without fine-tuning, good robustness with zero cost. As deep learning models be- come increasingly larger, critical to fully utilize pre-trained model parameters. Since the robust NRPM slightly refines the vanilla LPM through adaptive instance-wise reweighting scheme, en- suring that the pre-trained LPM parameters still well within NRPM architecture. Additionally, by adjusting the hyperparameters (Mr with pre- pre-trained parameters, between natural and robust accuracy. It worth noting that relying solely on pre-traines parameters with plug-and play paradigm significantly reduces computational costs, which is crucial in the of large-scale deep learning.
strike ideal balance
only fine-tuning {>r) strong robustness with slight cost. One drawback of Paradigm is that we specify the same for all the layers and need to conduct brute force hyper- parameters search obtain the optimal one. However, byper-parameters search time-consuming and computation-intensiiee Moreover, the entire model requires layer-wise {Ar 1 to balance the LPM and NRPM for different layers. To solve it, we propose Paradigm 2, which automatically learns optimal À with light-weight fine-tuning This paradigm only fine-tunes the hyperparameters (Nr)/= while keeping the model parameters frozen, which very efficient.
Paradigm 2:
Paradigm 3: overall fine-tuning, superior robustness with acceptable cost. To achieve best robustness, we can make both the model parameters {aa) and the hyperparameters {>r)/ learnalle. By refining these parameters based on the pre-trained model, we can prevent clean per- formance degradation. Moreover, with learnable {Ar]f=1 during adversarial training, the model can automatically select the optimal combination to enhance both clean and robust performance. This paradigm is still efficient since it only needs to tune-tun the model lightly with a few training epochs.
5 EXPERIMENT
In this section, we comprehensively evaluate the effectiveness of our proposed Robustness Repro- gramming techniques using wide range of backbone architectures, starting from basic MLPs, pro- gressing to shallow LeNet model, and extending to the widely-used ResNets architecture.
5.1 EXPERIMENTAL SETTING
Datasets. We conduct the experiments MNIST LeCun & Cortes (2005). SVHN Netzer al. 2011), CIFARIO (Krizhevsky al. 2009), and ImageNet10 (Russakovsky et al. 2015) datasets.
Backbone architectures. We select backbones ranging from very basic MLPs with 1, 2, layers, to mid-level architectures like LeNet, and deeper networks such as ResNet10, ResNet18, and ResNet34 (He al. 2016). In some experiments with ResNets, we chose the narrower version (with the model width reduced by a factor of 8) for the consideration for computation issue. Additionally, also choose popular architecture Mi.e-Mixer (Tolstikhin et al. 2021)

---

Published as a conference paper at ICLR 2025
Evaluation methods. We assess the performance of the models against various attacks under Loc norm, including FGSM (Goodfellow et al., 2014), PGD-20 (Madry] 2017). C&W (Carlini & Wag. ner 2017), and AutoAttack (Croce & & Hein 2020b). Among them, AutoAttack is ensemble attack consisting of three adaptive white-box attacks and one black-box attack, which is considered as a reliable evaluation method to avoid the false sense of security. In addition empirical robustness evaluation, we also evaluate certified robustness to further demonstrate the robustness of our pro- posed architecture.
Baselines & Hyperparameter setting. For backbone ResNets, we compare the baselines including PGD-AT (Madry, 2017), TRADES (Zhang al., 2019). MART (Wang al.] 2019), SAT (Huang et al. 2020), and AWP (Wu et al. (2220). We train the baselines for 200 epochs with batch size 128, weight decay 2e-5, momentum 0.9, and an initial learning rate of that is divided by the 100-th and 150-th epoch. For the backbone MLPs and LeNet, we train the vanilla models for 50 epochs. Our robustness reprogramming will fine-tune the pre-trained models for epochs.
5.2 ROBUSTNESS REPROGRAMMING ON MLPS
Table I: Robustness reprogramming under 3 paradigms on MNIST with 3-layer MLP as backbone.
Method / Budget
Natural
0.05
0.1
0.15
0.2
0.25
A1, A2, An
Normal-train
90.8
31.8
2.6
0.0
0.0
0.0
0.0
Ady-train
76.4
66.0
57.6
46.9
35.0
23.0
9.1
90.8
31.8
2.6
0.0
0.0
0.0
0.0
[1.0, 1.0, 1.0]
90.8
56.6
17.9
8.5
4.6
3.0
2.3
[0.9, 0.9, 0.9]
90.4
67.1
30.8
17.4
10.6
6.5
4.5
[0.8, 0.8, 0.8]
89.7
73.7
43.5
25.5
16.9
11.7
9.2
[0.7, 0.7,
Paradigm I
88.1
75.3
49.0
31.0
22.0
15.5
12.4
[0.6, 0.6, 0.6]
(without tuning)
84.1
74.4
50.0
31.9
22.8
18.1
14.3
10.5, O.5,
78.8
70.4
48.3
33.9
24.1
18.4
[0.4, 0.4]
69.5
62.6
45.2
31.5
23.1
19.0
15.5
[0.3, 0.3, 0.3]
58.5
53.2
38.2
27.6
22.2
16.4
12.9
[0.2, 0.2, 0.2]
40.7
38.3
29.7
22.8
16.8
12.9
11.1
[0.1, 0.1, 0.1]
17.6
16.4
14.6
12.4
10.7
9.4
[0.0, 0.0, 0.0]
Paradigm (tuning X)
81.5
75.3
61.2
44.7
33.7
26.0
20.1
[0,459, 0.033, 0.131]
Paradigm 3 (tuning all)
86.1
81.7
75.8
66.7
58.7
50.1
39.8
[0.925, 0.119, 0.325]
Comparison of robustness reprogramming via various paradigms. To compare the robustness of robustness reprogramming under paradigms well the vanilla normal/adversaria training. we evaluate the model performance under FGSM attack across various budgets with 3-layer MLP as backbone model. From the results in Table we can make the following observations:
terms of robustness, our Robustness Reprogramming across three paradigms progressively en- hances performance. In Paradigm 1, by adjusting {Ar)_ an optimal balance between clean and robust accuracy can be achieved without the need for parameter fine-tuning. Moreover, Paradigm 2 can automatically learn the layer-wise set {>r improving robustness compared to Paradigm 1 (with fixed {Ar H=)) Furthermore, Paradigm 3, by fine-tuning all the parameters, demonstrates the best performance among all the methods compared.
Regarding natural accuracy, we can observe from Paradigm 1 that increasing the inclusion of NRLM (i.e., smaller (x))))) results in decline in performance. But this sacrifice can be miti- gated by fine-tuning (Ar 1 the entire models as shown in Paradigm 2&33.
Behavior analysis on automated learning of of (Arl In Paradigm 2, we assert and expect that the learnable { across layers can achieve optimal performance under specified noisy environ- ment while maintaining the pre-trained parameters. To further validate our assertion and investigate the behavior of the learned { {Ar]/=1> we simulate various noisy environments by introducing ad- versarial perturbations (FGSM) into the training data at different noise levels, We initialize the {Mr' across all layers to 0.5. From the results shown in Table 2 we can make the following observations:

---

Published as a conference paper at ICLR 2025
Table 2: Automated learning of {Mr
under adversarial training with various noise levels.
Budget
Natural
0.05
0.1
0.15
0.2
0.25
0.3
Learned )
Adv-train 0.0)
91.3
75.7
45.0
24.9
14.9
10.5
8.0
[0.955, 0.706, 0.722]
Adv-train 0.05)
91.2
76.8
50.6
27.8
16.6
10.5
[0.953, 0.624, 0.748)
Adv-train (c 0.1)
91.0
82.6
62.5
41.7
26.4
19.8
14.9
[0.936, 0,342, 0.700]
Adv-train (c 0.15)
90.6
82.3
66.8
49.4
35.0
25.5
19.9
[0.879, 0.148, 0.599]
Adv-train 0.2)
90.3
82.5
67.5
49.3
35.8
27.0
21.3
[0.724, 0.076, 0,420]
Adv-train 0.25)
87.7
81.0
66.0
48.5
36.4
26.6
21.6
[0.572, 0.049, 0.243]
Adv-train 0.3)
81.5
75.3
61.2
44.7
33.7
26.0
20.1
[0.459, 0.033, 0.131]
The learned (Ar values are layer-specific, indicating that setting the same { for each layer in Paradigm is not optimal strategy. Automated learning of (Ar) enables the discov- ery of the optimal combination across layers.
As the noise level in the training data increases, the learned (Xr values tend to decrease, causing the hybrid architecture to resemble more robust NRPM architecture. This suggests that our proposed Paradigm can adaptively adjust { accommodate noisy environments.
Ablation studies. To further investigate the effectiveness of our proposed robust architecture, we provide several ablation studies on backbone size, attack budget measurement, additional backbone MLP-Mixer (Tolstikhin et al. 202T).the number of layers K in the Appendix C.2 Appendix C.3 and Appendix [C.4] respectively. These experiments demonstrate the consistent advantages of our method across different backbone sizes and attack budget measurements, Additionally, increasing number of layers K can further enhance robustness, though at the cost of a slight decrease in clean performance.
5.3 ROBUSTNESS REPROGRAMMIN ON CONVOLUTION
Robustness reprogramming on convolution. We evaluate the performance of our robustness re- programming with various weight {>r) in Figure 3 where
observe that incorporating more NRPM-induced embedding will significantly improve the robustness while sacrifice a little bit of clean performance. Moreover, by fine-tuning NRPM-model, can significantly improve robust performance while compensating for the sacrifice in clean accuracy.
Adversarial fine-tuning. Beyond natural training, also validate the advantage our NRPM architecture over the vanilla LPM under adversarial training. We utilize the pretrained parameter from LPM architecture, and track the clean/robust performance across 10 epochs under the adver- sarial fine-tuning tuning for both architectures in Figure The curves presented demonstrate the consistent improvement of NRPM over LPM across all the epochs.
100
100
95
Actursty (6)
Accuracy (%)
85
80
75
70
(Natural)
(Rabust)
NRPM (Natural)
fine-tuning
60
(Robust)
Epach
Figure 3: Robustness reprogramming on LeNet. The depth of color represents the size of budget.
Figure 4: Adversarial fine-tuning LeNet.
Hidden embedding visualization. We also conduct visualization analyses on the hidden embedding to obtain better insight into the effectiveness of our proposed NRPM. First, we quantify the differ- ence between clean embeddings (æ or 22) and attacked embeddings (x' or x/) across layers in 3a and visualize them in Fid Figure TT The results in Table show that NRPM-LeNet
show
8

---

Published as a conference paper at ICLR 2025
has smaller embedding difference across layers, indicating that proposed NRPM architecture in- deed mitigates the impact of the adversarial perturbation. Moreover, as demonstrated in the example in Figure [ the presence of adversarial perturbations leading to incorrect predictions in the case of vanilla LeNet. In contrast, our NRPM-LeNet appears to lessen the effects of such perturhations and maintain predicting groundtruth label. From the fig- ures, we can also clearly tell that the difference between clean attacked embeddings of LPM-LeNet much more significant than NRPM-LeNet.
disrupt the hidden embedding patterns,
Table 3: Embedding difference between clean and adversarial data 0.3) in LeNet. (MNIST)
LPM-LeNet
NRPM-Lete
1
93.21
27.54
0.30
90.09
26.67
0.30
2 271.20
116.94
1.56
21
167.52
58.55
1.19
Z2 79.52
62.17
1.89
22
22
19.76
4.27
0.56
23 18.77
22.56
2.32
3.63
0.98
0.50
21 9.84
18.95
3.34
21 21
2.21
0.79
0.57
4
4
(PM(())
NAPM(X)
LPM
44 NRPM(x)=
Visualization on LPM-LeNet.
(b) Visualization on NRPM-LeNeL.
Figure 5: Visualization of hidden embeddings. The LPM-LeNet is more sensitive to perturbation compared to the NRPM-LeNet: (a) When comparing z; (1st row) and 2 (2nd row), LPM (left)
shows more significant difference than NRPM (right). (b) When comparing the likelihood of pre- dictions, the perturbation misleads LPM from predicting to 8, while NRPM consistently predicts 4 in both clean and noisy scenarios.
Additional experiments. To further demonstrate effectiveness of our proposed method, we include experiments on two additional datasets, SVHN and ImageNet10, which are provided in Appendix D.2 All these experiments demonstrate consistent advantages of our proposed method.
5.4 ROBUSTNESS REPROGRAMMING ON RESNETS
In this section, will evaluate the robustness of robustness reprogramming ResNets across various attacks and further validate the effectiveness under diverse settings in the ablation studies.
Table 4: Robustness reprogramming CIFARIO with narrow ResNet18 as backbone.
Budget
Natural
8/255
16/255
32/255
[>r]
67.89
41.46
16.99
3.17
X 1.0
59.23
40.03
23.05
8.71
À = 0.9
40.79
28.32
20.27
13.70
À 0.8
Paradigm
24.69
18.34
15.31
13.09
À = 0.7
(without tuning)
17.84
14.26
12.63
11.66
À 0.6
15.99
12.51
11.42
11.07
À 0.5
10.03
10.02
10.0
10.0
À == 0.4
Paradigm 2 (tuning A)
69.08
44.09
24.94
12.63
Learnable
Paradigm 3 (tuning all)
71.79
50.89
39.58
30.03
Learnable
Robustness reprogramming on ResNets. We evaluate the robustness under PGD of our robust- From the results, Paradigm 1, adding more NRPM-based embeddings without fine-tuning leads to a notable drop in clean performance in ResNet18, which also limits the robustness improvement. (2) In Paradigm 2,
ness reprogramming via three paradigms in Table 4
observe that: (1) In

---

Published as a conference paper at ICLR 2025
by adjusting only the {ir values, improve the clean and robust performance, indicating the need of layer-wise balance across different layers. (3) In Paradigm 3, by tuning both (Nr) and parameters, we observe that both the accuracy and robustness can be further improved.
Adversarial robustness. To validate the effectiveness of our robustness reprogramming with ex- isting method, we select several existing popular adversarial defenses and report the experimental results of backbone ResNet18 under various attacks in Table From the results our robustness reprogramming exhibits excellent robustness across various attacks.
observe that
Table 5: Adversarial robsustness on CIFARIO with ResNet18 as backbone.
Method
Natural
PGD
FGSM
C&W
AA
DeepFool
SPSA
AVG
PGD-AT
80.90
44.35
58.41
46.72
42.14
14.81
62.92
44.89
TRADES-2.0
82.80
48.32
51.67
40.65
36.40
25.91
64.29
44.54
TRADES-0.2
85.74
32.63
44.26
26.70
19.00
12.98
57.79
32.23
MART
79.03
48.90
60.86
45.92
43.88
25.63
56.55
46.96
SAT
63.28
43.57
50.13
47.47
39.72
22.34
53.47
42.78
AWP
81.20
51.60
55,30
48.00
46.90
26.25
61.37
48.24
Consistency
84.37
45.19
53.84
43.75
40.88
21.27
65.91
45.14
DYNAT
82.34
52.25
65.96
52.19
45.10
28.72
67.97
52.03
Paradigm (Ours)
80.43
57.23
70.23
64.07
52.60
36.50
67.56
58.03
0.7
LPN o
c-110
60
certified accuracy
0.6
om 0.20
(IMM 0.20
Accuracy (%)
40
0.5
20
0.4
ResNet10
ResNet18
ResNet34
0.3
Backbone
LPM
8/253
16/255
0.0
0.2
0.4
0.6
NRPM 0/255
NRPM B/255
NRPM 16/255
radius
Figure Certified robustness via randomized smoothing with various levels.
Figure Ablation study on backbone size. The depth of color represents budget size.
Certified Robustness. Additionally, also evaluate the certified robustness using randomized smoothing with various a levels in Figure 6 The curves presented the figure demonstrate a significant advantage of our NRPM over vanilla LPM, further validating the effectiveness of our proposed architecture.
Different backbone sizes & budgets. Here, conduct ablation studies the backbone size & budget under AutoAttack Figure and leave the results under PGD in Appendix F The results show the evident advantage of our NRPM architecture.
CONCLUSION
This paper addresses a fundamental challenge in representation learning: how to reprogram a well- trained model to enhance its robustness without altering its parameters. We begin by revisiting the essential linear pattern matching in representation learning and then introduce an alternative non-linear robust pattern matching mechanism. Additionally, we present a novel and efficient Ro- bustness Reprogramming framework, which can be flexibly applied under three paradigms, making suitable for practical scenarios. Our theoretical analysis and comprehensive empirical evaluation demonstrate significant and consistent performance improvements. This research offers promising and complementary approach to strengthening adversarial defenses deep learning, significantly contributing to the development of more resilient AI systems.
10

---

Published as a conference paper at ICLR 2025
STATEMENT
Ethics statement. This paper proposes robustness reprogramming techniques to enhance the ro- bustness and safety of machine learning models. We do not identify any potential negative concerns. Reproducibility statement. This paper provides all necessary technique details for reproducibility, including theoretical analysis, algorithm details, experimental settings, pseudo code, implementa- tion, and source code of the proposed techniques.
ACKNOWLEDGMENT
Zhichao Hou and Dr. Xiaorui Liu are supported by the NSF National AI Research Resource Pilot Award, Amazon Research Award, NCSU Data Science Academy Seed Grant Award, and NCSU Faculty Research and Professional Development Award.
REFERENCES
NIST AI. Artificial intelligence risk management framework (ai rmf 1.0). 2023.
Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, Huan Zhang, Cho-Jui Hsieh, and Mani B Srivastava. Genattack: Practical black-box attacks with gradient-fret optimization. Proceed- ings of the genetic and evolutionary computation conference, pp. 1111-1119, 2019.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. In European conference on computer vision, pp. 484-501. Springer, 2020.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning time. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pp. 387-402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. Iece, 2017.
Pin-Ya Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artifccial intelligence and security, pp. 15-26, 2017.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In International conference on machine learning, pp. 854-863. PMLR, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. international conference on machine learning, pp. 1310-1320. PMLR, 2019.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference Machine Learning, pp. 2196-2205. PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020b.
Wenqi Fan, Xiaorui Liu, Wei Jin, Xiangyu Zhao, Jiliang Tang, and Qing Li. Graph trend filtering networks for recommendation. In Proceedings of the 45th international ACM SIGIR conference research and development in information retrieval, pp. 1 112-121, 2022.
Mahyar Fazlyab, Alexander Robey, Hamed Hassani. Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances neural information processing systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2025
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410, 2017.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue- sato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12775, 2018.
Sven Gowal, Sylvestre-Alviss Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems, 14:4218-4233 2021.
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. arXiv preprint DrXiv:1702.06260 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Chih-Hui Ho and Nuno Vasconcelos. Disco: Adversarial defense with local implicit functions. Advances in Neural Information Processing Systems, 35:23818 2--3337, 2022.
Zhichao Hou, Ruiqi Feng, Tyler Derr, and Xiaorui Liu. Robust graph neural networks via unbiased aggregation. In The Thirty- Annual Conference Neural Information Processing Systems, a.
Zhichao Hou, Weizhi Gao, Yuchen Shen, and Xiaorui Liu. Protransformer: Robustify transformers via plug-and-play paradigm. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, b.
Lang Huang, Chao Zhang, and Hongyang Zhang, Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33:19365-19376 2020.
Peter Huber and Elvezio M Ronchetti. Robust statistics. John Wiley & Sons, 2011.
Qiyu Kang, Yang Song, Qinxu Ding, and Wee Peng Tay. Stable neural ode with lyapunov-stable cquilibrium points for defending against adversarial attacks. Advances Neural Information Processing Systems, 34:14925-14993 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
John Law, Robust statistics approach based on influence functions, 1986.
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005. URL https:
/api.. wwwant seenatticsccoolar..rr//ooooooppp
ID: 6D:
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Boqi Li and Weiwei Liu. Wat: improve the worst-class robustness in adversarial training. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 37, pp. 14982-14990, 2023.
Xiyuan Li, Zou Xin, and Weiwei Liu. Defending against adversarial attacks via neural dynamic system. Advances in Neural Information Processing Systems, 35:6372-63883, 2022.
Xiaorui Liu, Jiayuan Ding, Wei Jin, Han Xu, Yao Ma, Zitao Liu, and Jiliang Tang. Graph neural networks with adaptive residual. Advances in Neural Information Processing Systems, 34:9720- 9733, 2021a.
12

---

Published as a conference paper at ICLR 2025
Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, and Jiliang Tang. Elas- tic graph neural networks. In International Conference on Machine Learning, pp. 6837-6849. PMLR. 2021b.
Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In Proceedings of the 30th ACM international conference on information & knowledge management, pp. 1202-1211, 2021.
Aleksander Madry. Towards deep learning models resistant to adversarial attacks, arXiv preprint 160.0100660088, 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprini arXiv:/702.04267 2017.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2574-2582, 2016.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint 2rXiv:2205.0744 2022.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference computer and communications security, pp- 506 519, 2017.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint 21X1v:2103.01946, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJV), 11(())211-2533 2015.
Changhao Shi, Chester Holtz, and Gal Mishne. Online adversarial purification based on self- supervision. arXiv preprint arXiv:2101.09387 2021.
Szegedy. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer. Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et Mlp-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey. Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning representations, 2019.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning, pp. 36246-36263, PMLR, 2023.
Dongxian Wu. Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in neural information processing systems, 33:2958-2969 2020.
Hanshu Yan, Jiawei Du, Vincent YF Tan, and Jiashi Feng. On robustness neural ordinary differ- ential equations. arXiv preprint 191.v:1900.055/.. 2019.
13

---

Published as a conference paper at ICLR 2025
Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning, pp. 12062-12072. PMLR, 2021.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Stephan Zheng, Yang Song. Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480-4488, 2016.
14

---

Published as a conference paper at ICLR 2025
A EFFICIENT IMPLEMENTATION OF ROBUST CONVOLUTION/MLPS
In the main paper, we formulate the pixel-wise algorithm for notation simplicity. However, not trivial to implement such theory into practice in convolution and MLPs layer. Here, will present the detailed implementation or techniques for each case.
A.1 ROBUST MIPS
In MLPs, we denote the input X PR, the output embedding Z R2, the model parameters as A xD>,, The pscudo code is presented Algorithm
Algorithm 2 Robust MLPs with NRPM
def RobustMI. (x, A, eps Te-3, 3)
2
X. unsquesze (-1)
3
D X. shapee[0]
4
Z torch. (x, A) # Initialization as LPM-estimation
For in range
DIST torch. (KX 2/D) Distance
7
W 1/IDIST + eps)
8
W normalize (W, p=1, dim=0)
Z = D* (X+AX) sum (dim=0) # Update
return
A.2 ROBUST CONVOLUTION
Unfolding. To apply robust estimation over each patch, we must first unfold the convolution into multiple patches and process them individually. While it's possible to use a for loop to extract patches across the channels, height, and width, this approach is not efficient. Instead, we can lever- age Py PyTorch's s torch.. nn. functional unfold function to streamline and accelerate both the unfolding and computation process.
B THEORETICAL PROOF
B. PROOF OF LEMMA
Proof. Since va S 2/b and the equlity holds when b, by replacemnet (aaD'
vb
=/D)² and h = (a.3. 20/ 2)2, then
1
lasaa 2/ s
(aaIa-z/D)) z/) Flazas - 20/
2 latAa 20/ D1
wa
- (
+ 2 lagod - 20/DI
Sum up the items on both sides, we obtain
D
((2)
Σ
aaza 2/DI S Σ wa (a + Σ
aada 20/D U(2,20)
d=1
d=1
and the equality holds at b== 20):
U(20, zn) = L(20)-
(4)
B.2 DERIVATION OF IRLS ALGORITHM
Proof.
D
4' (2(1))2000) = D d=
(k)
2k2 axZ2)
w
15

---

Published as a conference paper at ICLR 2025
D
(= 2(8)) = D
2
Σ wd
d=1
( -00 (1
(5)
(2 2(0))
D Ed= w (1) (2(2(k) agta)
(6)
2 Ex wid
D
2d Wd aazd
(K)
D
Ed wd
(k)
(k) where w/
Since the constant in w (k)
can be canceled in the updated formula-
odma-21k1/D)
tion, we have:
1
Wd
agg4 2(K)
B.3 PROOF OF THEOREM 3.2
Proof. For notation simplicity, we denote (FPM Ye = D [LPMM(F)) 2 JINRPM (F), = [TNRPM(FF) Since e is very small and D is large enough, we can assume BajTd yl laaxs ye for simplicity.
Influence function for LPM. The new weighted average under contamination becomes:
Ye (1 -cly cAr
Taking the limit as e + 0, we get:
D(y y)
De(Ar - D(Ar y) D(Ar - 2LPM D)
IF(Aa;TTPaC F)
lim c00
lim c0
c
c
This shows that the weighted average y is directly influenced by Ax. making it sensitive to outliers.
Influence function for NRPM. Next, we consider the reweighted estimate 2, when we introduce a
small contamination at Ar. which changes the distribution slightly. The contaminated reweighted estimate 2, becomes:
(1 - c) dE WhlgTd cwarAr
(1 - c) Ed Wt oWAr
We can simplify the expression for z, as:
Db Whlstd + - Db wadsta)
(1 (1-)) d=1 Wd + ewAr
ΣD
16

---

Published as a conference paper at ICLR 2025
First, expand the difference 2e
WalgEd + E(WA=Ar Ed-1 waasta) Σx walsed
(1 - 6) 2 Wa + EWAS
Edt Wd
11=1 Wald Ed=1 Wi + e(WAn Ed=1 wadsTa) Ed=
111 Ed=
EWAN) Ed=
+ (1 c) d Wd + EWAR Ed woosad
1(1 d= Wi + (WAZ) Ee= Wd
<(Wss Ar Ed wadsea) En
E(WAR - Ed 1d= wa) En WaagEa
[[1 - <) Ed=1 Wd + Ew Ed=
Aa Σd Wd WAr 1d=1 WadgTa)
[(1 ()) - Wd EwAn)
Ld Wd
Finally, divide the difference by e and take the limit as + 0:
D(z 2)
IF TNRPM, F) lim c00
e
WAr Ar E Wd WAr Ed= wadsEd
D
Eb We Ed=1 Wd
DEae (Ax 2NRPM/
Ea= Wd
Since War small for outliers (large Ar HLPM/DI. the influence of Ar on NRPM is dimin- ished compared to the influence LPM. Therefore, NRPM is more robust than LPM because the influence of outliers is reduced.
17

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTAL RESULTS ON MLPs
C.I MLP-MIXER
alternative model CNNs in computer vision. Building on the excellent performance of basic MLPs, we further explore the effectiveness of our NRPM with MLP-Mixer, presenting the results in Figure and Table 6 The results validate the effectiveness of our proposed methdod in MLP-Mixer with different blocks across various budgets.
MixerMixer (Tolstikhin al. 2021
80
LPM 0/255
LPM 2/255
Accuracy (%)
60
LPM 4/255
LPM 8/255
40
NRPM 0/255
NRPM 2/255
20
NRPM 4/255
NRPM 8/255
Blocks
Blocks
8 Blocks
Backbone
Figure 8: Robustness under PGD on MLP-Mixer. The depth of the color represents the size of the budget.
Table 6: Adversarial Robustness of MLP-Mixer CIFARIO
Model
Natural
1/255
2/255
3/255
4/255
8/255
L.M--MLP-MixeTT
M-Mixer-2
77.45
38.55
10.39
2.24
0.34
0.0
MEPM-MMPP-P Mixer-2
SEIM-MMP--ixxeE
76.86
64.27
49.62
37.22
27.99
15.73
LPM-MLP P
78.61
35,99
9.35
1.45
0.19
0.0
PRPM-MLP-Mixxee
77.51
66.72
51.73
39.43
30.88
18.31
POM-MLP-MixerT.
78.98
40.55
12.81
3.27
0.73
0.01
NRPM-MLP-Min
78.32
66.82
52.73
40.37
32.66
20.61
C.2 THE EFFECT OF BACKBONE SIZE.
Table 7: Robustness of MLPs with different layers MNIST.
Model Size
Arch / Budget
0.05
0.1
0.15
0.2
0.25
0.3
LPM
91.1
12.0
7.1
0.5
0.0
0.0
0.0
Layer
NRPM
87.6
14.0
13.6
12.9
12.1
11.5
10.3
LPM
91.6
32.9
2.6
0.2
0.0
0.0
0.0
2 Layers
NRPM
89.1
37.0
34.2
30.7
25.8
22.7
21.2
LPM
90.8
31.8
2.6
0.0
0.0
0.0
0.0
3 Layers
NRPM
90.7
47.2
45.1
37.8
30.8
24.9
21.1
We select the MLP backbones with different layers to investigate the performance under different model sizes. We report the performance with the linear models of (784 10), 2 (784 - 64 -10), (784 256 64 10) layers. We present the fine-tuning results for both the LPM model and NRPM model Table 7 from which we can make the following observations:
Our RosNet show better robustness with different size of backbones
Regarding clean performance, linear models with varying layers are equivalent and comparable.
However, our RosNet shows some improvement as the number of layers increases.
The robustness of RosNet enhances with the addition of more layers. This suggests that our robust layers effectively mitigate the impact of perturbations as they propagate through the network.
18

---

Published as a conference paper at ICLR 2025
C.3
ATTACK BUDGET MEASUREMENT
90
f model
-0- 1-made madee
m-moddl
Accuracy (%6
Accuracy (%)
Accuracy (%6
75
40
20
65
256 Budget Lerhrrm
128
512
0.0
1.0
2.0
4.0
5.0
6.0
0.0
0.05 0.1 0.15
0.2 0.25 0.3
Budget on Cy-norm
Budget on L--hrrm
(a) Lg-attack
(h) T-2-attcck
(c) To
Figure 9: Robustness under attacks with different norms.
In previous results of different backbone sizes, we notice the NRPM model is not effective enough under Loc To verify the effeet the attack budget measurement, we evaluate the robustness under LocLL attacks where the budget is measured under the Loo.no,,o-mooms that is, || x'Ipp < budget, where p = 0,2, co. We can make the following observations from the results in Figure
Our NRPM architecture consistently improve the robustness the LPM-bbccbooee across vari- ous attacks and budgets.
Our NRPM model experiences a slight decrease in clean accuracy. suggesting that the median retains less information than the mean.
Our NRPM model shows improved performance under the Lp and L2 attacks compared to the Loc. attack. This behavior aligns with the properties of mean and median. Specifically, under the Too attack, each pixel can be perturbed up to certain limit, allowing both the mean and the median to be easily altered when all pixels are modified. Conversely, under the L2 and Lo attacks, the number of perturbed pixels is restricted, making the median more resilient to disruption than the mean.
C.4 EFFECT OF DIFFERENT LAYERS K
To investigate the effect of the number of iterations K in the unrolled architecture, present the results with increasing K on combined models in the Table and Figure 10 in the appendix. As K increases, the hybrid model can approach the NRPM architecture, which makes the model more robust while slightly sacrificing little bit natural accuracy.
19

---

Published as a conference paper at ICLR 2025
Table 8: Ablation study number of layers K MLP- MNIST
K
0.05
0.1
0.15
0.2
0.25
0.3
À
0.9
K = 0
90.8
31.8
2.6
0.0
0.0
0.0
0.0
K 90.8
56.6
17.9
8.5
4.6
3.0
2.3
K = 2
90.7
76.1
44.0
26.8
18.0
13.4
9.3
K
90.7
82.1
56.7
37.4
26.3
19.9
16.3
K = 4
90.6
83.5
61.1
42.7
29.4
22.7
18.0
À =
0.8
K 0
90.8
31.8
2.6
0.0
0.0
0.0
0.0
K
90.4
67.1
30.8
17.4
10.6
6.5
4.5
K 2
90.4
81.1
56.2
36.2
26.0
19.3
15.5
K = 3
90.3
81.4
61.8
42.9
31.8
23.7
19.5
K 90.3
82.5
64.3
45.6
33.1
25.3
22.0
x= 0.7
K =
90.8
31.8
0.0
0.0
0.0
0.0
K 89.7
73.7
43.5
25.5
16.9
11.7
9.2
K = 2
89.1
80.3
56.4
39.7
28.4
21.4
18.5
K 3
88.9
80.3
61.8
44.0
33,2
24.4
19.2
K 4
88.4
79.6
61.3
43.6
33.9
26.3
21.4
0.6
K = 0 90.8
31.8
2.6
0.0
0.0
0.0
0.0
K 88.1
75.3
49.0
31.0
22.0
15.5
12.4
K 2 86.2
77.1
56.2
39.3
29.2
22.6
17.1
K =
85.5
75.9
55.8
40.2
30.7
23.1
18.8
K 4
84.0
74.7
56.2
39.6
29.4
22.1
18.7
À =
0.5
K 0 90.8
31.8
2.6
0.0
0.0
0.0
0.0
K K=1
84.1
74.4
50.0
31.9
22.8
18.1
14.3
K = 80.3
71.0
53.5
37.0
26.9
21.5
16.4
K 33 79.0
69.6
53.7
38.1
29.2
23.2
19.5
K 77.7
66.6
51.2
37.9
29.7
23.7
19.8
20

---

Published as a conference paper at ICLR 2025
Accuracy
Accuracy
000.0
6-0.1
200.0
0
Cw0.25
Layer
#ayer
(a) 0.5
(b) À 0.6
Accuracy
Accuracy
2-0.0
2-0.1
2-0.0
0
2w0.25
#Lyer
#Layer
(c) 0.7
(d) A 0.8
Accuracy
cm0.05
-0- 4-025
#Layer
(c) X 0.9
Figure 10: Effect of layers K
D ADDITIONAL EXPERIMENTAL RESULTS ON LENET
D.1 VISUALIZATION OF HIDDEN EMBEDDING
We put several examples of the visualizations of hidden embedding in Figure 11
21

---

Published as a conference paper at ICLR 2025
/
55
55
8
8
4
4
4
4
Figure 11: Visualization of hidden embedding.
D.2 ADDITIONAL DATASETS
Besides MNIST, we also conduct the experiment SVHN and ImageNet10, and show the results under PGD in Table and Table respectively.
22

---

Published as a conference paper at ICLR 2025
Table 9: Robustness of LeNet under PGD on SVHN.
Model / /Bdgee
0/255
1/255
2/255
4/255
8/255
16/255
32/255
64/255
LPM-LeNet
83.8
68.6
49.3
27.1
8.2
3.0
1.9
1.3
NRPM-LeNet
83.2
72.4
54.6
35.4
20.1
14.3
11.8
7.9
Table 10: Robustness of LeNet under PGD on ImageNet10.
Model / Budget e
0/255
2/255
4/255
8/255
LeM-LeNet 53.72
13.11
7.67
3.89
NRPM-LeNet
53.55
15.06
13.68
12.50
E ADDITIONAL EXPERIMENTAL RESULTS ON RESNETS
E. EFFECT OF BACKBONE SIZE
We evaluate the robustness under PGD with different backbone sizes including ResNet10, ResNet18, 88, and ResNet34. The results presented in Figure 12 demonstrate the consistent improvement of our NRPM over vanilla LPM.
60
Accuracy (%)
40
20
0
ResNet10
ResNet18 Backbone
ResNet34
LPM 0/255
LPM 8/255
LPM 16/255
0/255
NRPM 8/255
16/255
Figure 12: Ablation study on backbone size. (PGD)
23


Reviews:
[{"review_id": "1mFZS71uDX", "reviewer": "Reviewer_Pa5t", "summary": "This paper proposes a \"reprogramming\" approach to improve the robustness of neural networks with minimal or no alteration of the model parameters. The reprogramming approach is executed by incorporating additional set of parameters (one per input dimension) in each layer to reweight the importance of features at inference time. The approach is motivated from a perspective of looking at the function of each unit as a linear feature pattern matching that is proposed to be replaced by a nonlinear version. The final implementation of the approach combines the activity of both linear and nonlinear feature pattern matching methods.", "questions": "1. Related to weakness #1 above, on line 167 it is stated that by replacing the loss function with the proposed one the impact of outliers will be reduced, referring to Huber and Ronchetti's book Robust Statistics. The connection is unclear to me and I think deserves a more detailed explanation. Can you expand? \n\n2. Line 105, by \"robust training\" do you mean \"adversarial training\"? \n\n3. Line 244: \"One naive approach is to simply replace the vanilla linear pattern matching (LPM) with NRPM. However, this naive approach does not work well in practice\". Why doesn't this approach work? \n\n4. Line 231: according to the text, $x_o$ is not the perturbation but the initial input. Is that correct? \n\n5. In table1 and others, what are the numbers at the top of each column? Are they the epsilon values used for test? What attack was used? \n\n6. Adaptation of the approach to CNNs needs more explanation. Were separate $w_i$ parameters considered per filter or per activation? \n\n7. Fig 5: likelihood in the bar plots doesn't look like proper probability density functions. What is being shown? \n\n8. Fig 5 caption: 1 and 2 correspond to (a) and (b) in the figure?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Proposed idea of reprogramming is interesting as it would allow reusing the learned features by the original model. May be an important approach to explore in large models. \n2. Empirical results consider multiple datasets of various sizes and different perturbations", "weaknesses": "1. The approach is based on eq 1 and the precursor to this for what is named as \"linear feature pattern matching\". The first equation (unnumbered equation before eq 1) formulates the OLS solution as the minimum of \\mathcal{L}=\\sum_{d=1}^{D}{(\\frac{y}{D}-a_d.x_d)^2}. However, this formulation of the solution assumes that the prediction error is distributed uniformly across all dimensions which is not generally true and may indeed be very constraining in certain situations. Despite this possible issue, I didn't see any mention of this anywhere in the paper which in my opinion is a big weakness. \n\n2. Some of the key implementation details of the approach were unclear. What data is used to optimize the $w_i$ parameters? Are they optimized over the full training set of each dataset? Alg 1 contains a loop of K iterations for each $x_d$. Does this mean that the approach finds independent $w_d$ for each sample independently? In what order are different parameters/hyperparameters optimized? A step by step description of the approach or a pseudo-code would be helpful if not necessary. \n\n3. The approach requires computing additional activations for every layer that may be costly but no comparison of runtime, computation cost etc is provided. \n\n4. Details of Adv-train baseline is missing. Was this the result of adversarial training from scratch or fine-tuning? \n\n5. In Table5, the AA accuracy is substantially higher than that of PGD (64.60 vs. 57.23). This to me looks like a big red flag. AA is an ensemble attack made of 4 separate attacks including two effective variations of PGD. Because of this, AA accuracy is always lower than that of PGD. I'm doubtful of the validity of empirical evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MD5wj25FzO", "reviewer": "Reviewer_ktRd", "summary": "Many methods have been proposed to improve the adversarial robustness of neural networks, including adversarial training, purification, and regularization. However, these approaches don't take advantage of the learned parameters of models that were not robustly trained. This paper proposed robust reprogramming, which uses those learned parameters to construct a new model that is robust to adversarial perturbations by design. The authors identify linear transformations to essentially be linear pattern matching (LPM), with the pattern being specified by the parameters of the transformation which are capable of matching related features in the input. They further note that linear feature pattern matching is the optimal closed-form solution to the Optimized Least Squares problem, which is particularly sensitive to outliers due to its quadratic penalty. To reduce this sensitivity, this work introduces a new feature pattern matching term based on Least Absolute Deviation estimation, which is called Nonlinear Robust Pattern Matching (NRPM). Theoretical results show that adopting NRPM mitigates the influence of any datapoint on downstream predictions. Three paradigms of robust reprogramming are then introduced: one which only involves implementing the NRPM architecture, one which involves NRPM and fine-tuning hyperparameters, and one which involves NRPM and finetuning all parameters in the model. Experimental results show that these paradigms (and especially paradigm 3) can provide significant improvements over existing robustness techniques. For example, in ResNets trained on CIFAR10, paradigm 3 results in significantly higher robustness than state of the art adversarial training techniques, with only a modest drop in clean accuracy.", "questions": "* What is the difference in inference efficiency between a standard MLP and a reprogrammed MLP (in terms of memory usage/inference time)?\n* Similarly, how does execution time scale with the depth/width of the network?\n* Are there any adaptive attacks that could improve over AutoAttack against robust reprogramming? Does this approach just reduce to gradient masking?\n* Is the NRPM approach suitable for training from scratch? If so, do you think that would that further improve over the performance of paradigm 3?\n* Would an adversary with knowledge of the robust reprogramming scheme be able to design an adaptive attack to counteract these defenses?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* There is a great deal of novelty in this approach. Many defenses against adversarial examples atttempt to learn weights that are more robust, or to detect/purify adversarial inputs to a network. I am not aware of any prior work that modifies the model of computation to increase robustness in this way.\n* Code and model weights have been provided for replication purposes. \n* The experimental results are impressive. In particular, in certain cases the third paradigm can increase both robustness and clean accuracy.", "weaknesses": "* Theorem 3.2 is presented in a confusing manner. You refer to $x_0$ as the perturbation, but isn't it instead the location of a possible perturbation?\n* The theoretical justification for the robustness of NRPM largely relies on the influence functions that are derived for NRPM and LPM. However, there's no explicit connection that's made between influence functions and adversarial robustness. It's not immediately clear to me that the influence function on its own would immediately imply adversarial robustness. I think additional justification for this relationship would be necessary in the final version of this paper (or references to related works which establish this relationship).\n* On a similar note, there are settings that have been studied in the literature in which an adversary is capable of making unbounded changes to the input (i.e. [1,2]). It seems that the influence function would be less informative about the robustness of the model to adversaries that are unbounded in $\\ell_p$ space. I think it should be made more explicit if you are only envisioning this as a defense against $\\ell_p$ bounded adversaries.\n* The table and figure captions could use more information about the experimental setups. For example, its unclear which attacks/norms mentioned in section 5.1 are used in which table.\n\n[1] Xiao, Chaowei, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. \"Spatially transformed adversarial examples.\" arXiv preprint arXiv:1801.02612 (2018).\n[2] Kaufmann, Max, Daniel Kang, Yi Sun, Steven Basart, Xuwang Yin, Mantas Mazeika, Akul Arora et al. \"Testing robustness against unforeseen adversaries.\" arXiv preprint arXiv:1908.08016 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JshJodyrF2", "reviewer": "Reviewer_kryy", "summary": "This paper proposed nonlinear robust pattern matching algorithm (NRPM) that can improves the robustness without modifying its parameters while maintaining the feature pattern matching by reprogramming a pretrained model. Especially, this paper theoretically analyzed linear pattern matching and Newton-iterative reweighted least squares algorithm, and introduced robustness reprogramming paradigms under three different situations: parameter-free tuning, lightweight fine-tuning and full fine-tuning. The experimental results support its theoretical analysis and demonstrate the effectiveness of proposed method.", "questions": "- In table 5, robust accuracy against AutoAttack seems to be high, and it would be better to check whether the model suffers from obfuscated gradient or not. For example, following scenarios could be provided: 1) evaluation against adaptive attacks, 2) checking for gradient masking, or 3) evaluation against PGD attacks with extremely large attack strength where robust accuracy should be almost zero while PGD attacks using same strength but different step sizes and steps (e.g., 2 times smaller step size but 2 times larger attack steps) should achieve similar robust accuracy from original attack setting. \n- For the proposed three different paradigms, it would be better to explain when it is more effective to use which paradigm to achieve robustness efficiently based on the differences between three scenarios.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper provides comprehensive and detailed theoretical analysis and explanations.\n- This paper is well written to follow.", "weaknesses": "- This paper leveraged old baselines for ResNet18 backbone while recent baselines (e.g., [1], [2]) are missing and evaluation against stronger attack (e.g., LGV, SPSA, DeepFool) could be beneficial.\n   - [1] Consistency regularization for adversarial robustness\n   - [2] Dynamic Label Adversarial Training for Deep Learning Robustness Against Adversarial Attacks\n- This paper utilizes backbones that are too small and tasks that are too easy. For example, main table (Table 1, 2) used MLPs and MNIST. It would be beneficial to provide experimental results on CIFAR-100 and Tiny-ImageNet at least with backbone of ResNet18 and WideResNet28-10. \n- Under same computing resource, it would be better to directly compare the required resources (e.g., FLOPs) or training time (e.g., GPU Hours) between the proposed method (e.g., paradigm 3, full-tuning) and existing methods for ResNet18 backbone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "srIoua7DTG", "reviewer": "Reviewer_QnFV", "summary": "The paper proposes an adversarial robustness scheme that doesn't necessarily require altering the pretrained parameters. They propose a \"reprogramming\" scheme to adapt the representations learned by a pretrained network in order to enhance the network's robustness. The proposed approach is extensively evaluated on multiple datasets and simpler model architectures and is shown to enhance empirical performance.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The approach is thoroughly evaluated on multiple datasets and simpler models.\n- It empirically performs well under increasing attack strength/various attack types.", "weaknesses": "1) The readability of the paper is not good in its current form. The paper doesn't provide sufficient background/cite references to understand the terminology used or where ideas are derived from. Overall, I find it very hard to understand the ideas/contributions of this work, even though the results look promising.\n\n2) The notation is inconsistent, and several terms are used without introduction. For example,\n\n      a) What is X in the indicator function of F in line 212?\n\n      b) What is y in Line 223? Such missing details make it very hard to follow the ideas introduced in the paper.\n\n      c) What is the difference between {\\lambda} and \\lambda? Guessing from Table 1, the weights are learned/assigned per layer? The notation does not convey this correctly (it should either be a vector or otherwise properly subscripted in the notation).\n\n      d) What is N in algorithm 2?\n\n3) The figures lack sufficient description to understand what's happening. How are the visualizations of hidden embeddings created in Figure 5? What is meant by the \"size of the budget\" in Figure 3? (my intuitive guess is that refers to the attack budget, but under what norm? Which attack? The language needs to be a lot clearer and the captions/descriptions are too succinct and vague in the current draft).\n\n4) Please include an analysis of how exactly is this better than adversarial training. How do the training time and number of parameters compare for the two approaches?\n\n5) I also find it hard to appreciate the need for the 3 separate training paradigms. A discussion of when and how exactly would each of these be more applicable and suitable is lacking.\n\n6) What is the cost of scaling this approach to more practical models/datasets (e.g. ImageNet with ResNet50 or so)? This goes back to -4- as to if/how this is better than AT, besides the empirical gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)