Robustness Reprogramming for Representation Learning

## Abstract

Abstract This work tackles an intriguing and fundamental open challenge in representation learning: Given a well-trained deep learning model, can it be reprogrammed to enhance its robustness against adversarial or noisy input perturbations without altering its parameters?
To explore this, we revisit the core feature transformation mechanism in representation learning and propose a novel non-linear robust pattern matching technique as a robust alternative. Furthermore, we introduce three model reprogramming paradigms to offer flexible control of robustness under different efficiency requirements. Comprehensive experiments and ablation studies across diverse learning models ranging from basic linear model and MLPs to shallow and modern deep ConvNets demonstrate the effectiveness
of our approaches.
This work not only opens a promising and orthogonal direction for improving adversarial defenses in deep learning beyond existing methods but also provides new insights into designing more resilient AI systems with robust statistics.

## 1 Introduction

Deep neural networks (DNNs) have made significant impacts across various domains due to their powerful capability of learning representation from high-dimensional data (LeCun et al., ; Goodfellow et al., ).
However, it has been well-documented that DNNs are highly vulnerable to adversarial attacks (Szegedy, ; Biggio et al., ).
These vulnerabilities are prevalent across various model architectures, attack capacities, attack knowledge, data modalities, and prediction tasks, which
hinders their reliable deployment in real-world applications due to potential economic, ethical, and societal risks (AI, ).

In light of the burgeoning development of AI, the robustness and reliability of deep learning models have become increasingly crucial and of particular interest. A mount of endeavors attempting to safeguard DNNs have demonstrated promising robustness, including robust training (Madry, ; Zhang et al., ; Gowal et al., ; Li & Liu, ), regularization (Cisse et al., ; Zheng et al., ), and purification techniques (Ho & Vasconcelos, ; Nie et al., ; Shi et al., ; Yoon et al., ). However, these methods often suffer from catastrophic pitfalls like cumbersome training processes or domain-specific heuristics, failing to deliver the desired robustness gains in an efficient and adaptable manner.

Despite numerous advancements in adversarial defenses, an open challenge persists: *Is it possible to reprogram a well-trained model to achieve the desired robustness without modifying its parameters?* This question is of particular significance in the current era of large-scale models. Reprogramming without training is highly efficient, as the pretraining-and-finetuning paradigm grants access to substantial open source pre-trained parameters, eliminating the need for additional training. Moreover, reprogramming offers an innovative, complementary, and orthogonal approach to existing defenses, paving the way to reshape the landscape of robust deep learning.

To address this research gap, we firstly delve deeper into the basic neurons of DNNs to investigate the fundamental mechanism of representation learning. At its core,
the linear feature transformations serve as an essential building block to capture particular feature patterns of interest in most deep learning models. For instance, the Multi-Layer Perceptron (MLP) fundamentally consists of multiple stacked linear mapping layers and activation functions; the convolution operations in Convolution Neural Networks (CNNs) (He et al., ) execute a linear feature mapping over local patches using the convolution kernels; the attention mechanism in Transformers (Vaswani, )
performs linear transformations over the contextualized token vectors. This linear feature mapping functions as *Linear Pattern Matching* by capturing the certain patterns that are highly correlated with the model parameters.
However, this pattern matching manner is highly sensitive to data perturbations,
which explains the breakdown of the deep learning models under the adversarial environments.

To this end, we propose a novel approach, *Nonlinear Robust Pattern Matching*, which significantly improves the robustness while maintaining the feature pattern matching behaviors. Furthermore, we also introduce a flexible and efficient strategy, *Robustness Reprogramming*, which can be deployed under three paradigms to improve the robustness, accommodating varying resource constraints and robustness requirements.
This innovative framework promises to redefine the landscape of robust deep learning,
paving the way for enhanced resilience against adversarial threats.

Our contributions can be summarized as follows:
(1) We propose a new perspective on representation learning by formulating *Linear Pattern Matching* (the fundamental mechanism of feature extraction in deep learning) as ordinary least-square problems;
(2) Built upon our novel viewpoint, we introduce *Nonlinear Robust Pattern Matching* as an alternative robust operation and provide theoretical convergence and robustness guarantees for its effectiveness;
(3) We develop an innovative and adaptable strategy, *Robustness Reprogramming*, which includes three progressive paradigms to enhance the resilience of given pre-trained models;
and (4) We conduct comprehensive experiments to demonstrate the effectiveness of our proposed approaches across various backbone architectures, using multiple evaluation methods and providing several insightful analyses.

## 2 Related Works

Adversarial Attacks.
Adversarial attacks can generally be categorized into two types: white-box and black-box attacks. In white-box attacks, the attacker has complete access to the target neural network, including its architecture, parameters, and gradients. Examples of such attacks include gradient-based methods like FGSM (Goodfellow et al., ), DeepFool (Moosavi-Dezfooli et al., ), PGD (Madry, ), and C&W attacks (Carlini & Wagner, ). On the other hand, black-box attacks do not have full access to the model’s internal information; the attacker can only use the model’s input and output responses. Examples of black-box methods include surrogate model-based method (Papernot et al., ), zeroth-order optimization (Chen et al., ), and query-efficient methods(Andriushchenko et al., ; Alzantot et al., ). Additionally, AutoAttack (Croce & Hein, ), an ensemble attack that includes two modified versions of the PGD attack, a fast adaptive boundary attack (Croce & Hein, ), and a black-box query-efficient square attack (Andriushchenko et al., ), has demonstrated strong performance and is often considered as a reliable benchmark for evaluating adversarial robustness.

Adversarial Defenses.
Numerous efforts have been made to enhance the robustness of deep learning models, which can broadly be categorized into empirical defenses and certifiable defenses. Empirical defenses focus on increasing robustness through various strategies: robust training methods (Madry, ; Zhang et al., ; Gowal et al., ; Li & Liu, ) introduce adversarial perturbations into the training data, while regularization-based approaches (Cisse et al., ; Zheng et al., ) stabilize models by constraining the Lipschitz constant or spectral norm of the weight matrix. Additionally, detection techniques Metzen et al. (); Feinman et al. (); Grosse et al. () aim to defend against attacks by identifying adversarial inputs.
Purification-based approaches seek to eliminate the adversarial signals before performing downstream tasks (Ho & Vasconcelos, ; Nie et al., ; Shi et al., ; Yoon et al., ). Recently, some novel approaches have emerged by improving robustness from the perspectives of ordinary differential equations (Kang et al., ; Li et al., ; Yan et al., ) and generative models (Wang et al., ; Nie et al., ; Rebuffi et al., ). Beyond empirical defenses, certifiable defenses (Cohen et al., ; Gowal et al., ; Fazlyab et al., ) offer theoretical guarantees of robustness within specific regions against any attack.
However, many of these methods suffer from significant overfitting issues or depend on domain-specific heuristics, which limit their effectiveness and adaptability in achieving satisfying robustness. Additionally, techniques like robust training often entail high computational and training costs, especially when dealing with diverse noisy environments, thereby limiting their scalability and flexibility for broader applications.
The contribution of robustness reprogramming in this work is fully orthogonal to existing efforts, and they can be integrated for further enhancement.

## 3 Robust Nonlinear Pattern Matching

In this section, we begin by exploring the vulnerability of
representation learning
from the perspective of
pattern matching
and subsequently introduce a novel robust feature matching in Section 3.1. Following this, we develop a Newton-IRLS algorithm, which is unrolled into robust layers in Section 3.2.
Lastly, we present a theoretical robustness analysis of this architecture in Section 3.3.

Notation.
Let the input features of one instance be represented as ${\bm{x}}=\left(x_{1},\dots,x_{D}\right)^{\top}\in\mathbb{R}^{D}$, and the parameter vector as ${\bm{a}}=\left(a_{1},\dots,a_{D}\right)^{\top}\in\mathbb{R}^{D}$, where $D$ denotes the feature dimension.
For simplicity, we describe our method in the case where the output is one-dimensional, i.e., $z\in\mathbb{R}$.

### 3.1 A New Perspective of Representation Learning

Figure: Figure 1: Vanilla Linear Pattern Matching (LPM) vs. Nonlinear Robust Pattern Matching (NRPM).
Refer to caption: extracted/5905404/figure/model_comparison.png

Fundamentally, DNNs inherently function as representation learning modules by transforming raw data into progressively more compact embeddings (LeCun et al., ; Goodfellow et al., ).
The linear feature transformation, $z={\bm{a}}^{\top}{\bm{x}}=\sum_{d=1}^{D}a_{d}\cdot x_{d}$, is the essentially building block of deep learning models to
capture particular feature patterns of interest. Specifically, a certain pattern ${\bm{x}}$ can be captured and matched once it is highly correlated with the model parameter ${\bm{a}}$.

Despite the impressive capability of linear operator in enhancing the representation learning of DNNs, the vanilla deep learning models have been validated highly vulnerable (Szegedy, ; Biggio et al., ). Existing approaches including robust training and regularization techniques (Madry, ; Cisse et al., ; Zheng et al., ) aim to improve the robustness of feature transformations by
constraining the parameters ${\bm{a}}$ with particular properties. However, these methods inevitably alter the feature matching behaviors, often leading to clean performance degradation without necessarily achieving improved robustness.

Different from existing works, we aim to introduce a novel perspective by exploring how to design an alternative feature mapping that enhances robustness while maximally preserving feature matching behaviors.
First, we
formulate the linear feature pattern matching as the optimal closed-form solution of the Ordinary Least Squares (OLS) problem:

$$ $\min_{z\in\mathbb{R}}\mathcal{L}(z)=\sum_{d=1}^{D}\left(\frac{z}{D}-a_{d}\cdot x _{d}\right)^{2},$ $$

where the first-order optimality condition $\frac{\partial{\mathcal{L}}(z)}{\partial z}=0$ yields the linear transformation $z^{*}=\sum_{d=1}^{D}a_{d}\cdot x_{d}$.
Since OLS estimation is highly sensitive to outlying values due to the quadratic penalty, we propose to derive a robust alternative
inspired by the Least Absolute Deviation (LAD) estimation:

$$ $\min_{z\in\mathbb{R}}\mathcal{L}(z)=\sum_{d=1}^{D}\left|\frac{z}{D}-a_{d}\cdot x _{d}\right|.$ (1) $$

By replacing the quadratic penalty with a linear alternative on the residual $z/D-a_{d}\cdot x_{d}$, the impact of outliers can be significantly mitigated according to robust statistics (Huber & Ronchetti, ).

### 3.2 Algorithm Development and Analysis

Although the LAD estimator offers robustness implication, the non-smooth objective in Eq. (1) poses a challenge in designing an efficient algorithm to be integrated neural network layers.
To this end,
we leverage the Newton Iterative Reweighted Least Squares (Newton-IRLS) algorithm to address the non-smooth nature of the absolute value operator $|\cdot|$ by optimizing an alternative smoothed objective function ${\mathcal{U}}$ with Newton method. In this section, we will first introduce the localized upper bound ${\mathcal{U}}$ for ${\mathcal{L}}$ in Lemma 3.1, and then derive the Newton-IRLS algorithm to optimize ${\mathcal{L}}$.

###### Lemma 3.1 .

Let ${\mathcal{L}}(z)$ be defined in Eq. (1), and for any fixed point $z_{0}$, ${\mathcal{U}}(z,z_{0})$ is defined as

$$ ${\mathcal{U}}(z,z_{0})=\sum_{d=1}^{D}w_{d}\cdot(a_{d}x_{d}-z/D)^{2}+\frac{1}{2 }{\mathcal{L}}(z_{0}),$ (2) $$

where $w_{d}=\frac{1}{2|a_{d}x_{d}-z_{0}/D|}.$
Then, for any $z$, the following holds:

$$ $(1)\;{\mathcal{U}}(z,z_{0})\geq{\mathcal{L}}(z),\quad(2)\;{\mathcal{U}}(z_{0}, z_{0})={\mathcal{L}}(z_{0}).$ $$

###### Proof.

Please refer to Appendix B.1.
∎

The statement (1) indicates that ${\mathcal{U}}(z,z_{0})$ serves as an upper bound for ${\mathcal{L}}(z)$, while statement (2) demonstrates that ${\mathcal{U}}(z,z_{0})$ equals ${\mathcal{L}}(z)$ at point $z_{0}$.
With fixed $z_{0}$, the alternative objective ${\mathcal{U}}(z,z_{0})$ in Eq. (2) is quadratic and can be efficiently optimized.
Therefore, instead of minimizing the non-smooth ${\mathcal{L}}(z)$ directly, the Newton-IRLS algorithm will obtain $z^{(k+1)}$ by optimizing the quadratic upper bound ${\mathcal{U}}(z,z^{(k)})$ with second-order Newton method:

$$ $\displaystyle z^{(k+1)}$ $\displaystyle=D\cdot\frac{\sum_{d=1}^{D}w^{(k)}_{d}a_{d}x_{d}}{\sum_{d=1}^{D}w ^{(k)}_{d}}$ (3) $$

where $w^{(k)}_{d}=\frac{1}{|a_{d}x_{d}-z^{(k)}/D|}.$
Please refer to Appendix B.2 for detailed derivation.
As a consequence of Lemma 3.1, we can conclude the iteration $\{z^{(k)}\}_{k=1}^{K}$ fulfill the loss descent of ${\mathcal{L}}(z)$:

$$ ${\mathcal{L}}(z^{(k+1)})\leq{\mathcal{U}}(z^{(k+1)},z^{(k)})\leq{\mathcal{U}}( z^{(k)},z^{(k)})={\mathcal{L}}(z^{(k)}).$ $$

This implies
Eq. (3) can achieve
convergence of ${\mathcal{L}}$ by optimizing the localized upper bound ${\mathcal{U}}$.

Implementation.
The proposed non-linear feature pattern matching is expected to improve the robustness against data perturbation in any deep learning models by replacing the vanilla linear feature transformation. In this paper, we illustrate its use cases through MLPs and convolution models. We provide detailed implementation techniques
in Appendix A.
Moreover, we will demonstrate how to leverage this technique for robustness reprogramming for representation learning in Section 4.

###### Lemma 3.1 .

###### Proof.

### 3.3 Theoretical Robustness Analysis

In this section, we conduct a theoretical robustness analysis comparing the vanilla Linear Pattern Matching (LPM) architecture with our Nonlinear Robust Pattern Matching (NRPM) based on influence function (Law, ).
For simplicity, we consider a single-step case for our Newton-IRLS algorithm ($K=1$).
Denote the feature distribution as $F(X)=\frac{1}{D}\sum_{d=1}^{D}\mathbb{I}_{\{X=a_{d}x_{d}\}}$.
Then we can represent
LPM as $z_{LPM}:=T_{LPM}(F)=\sum_{d=1}^{D}a_{d}x_{d}$ and NRPM as $z_{NRPM}:=T_{NRPM}(F)=D\cdot\frac{\sum_{d=1}^{D}w_{d}a_{d}x_{d}}{\sum_{d=1}^{D
}w_{d}},$ where $w_{d}=\frac{1}{|a_{d}x_{d}-z_{LPM}/D|}$.

We derive
their influence functions
in Theorem 3.2 to demonstrate their sensitivity against input perturbations, with a proof presented in Appendix B.3.

###### Theorem 3.2 (Robustness Analysis via Influence Function) .

The influence function is defined as the sensitivity of the estimate to a small contamination at $x_{0}$:

$$ $IF(x_{0};T,F)=\lim_{\epsilon\to 0}\frac{T(F_{\epsilon})-T(F)}{\epsilon}$ $$

where the contaminated distribution becomes $F_{\epsilon}=(1-\epsilon)F+\epsilon\delta_{x_{0}}$, where $\delta_{x_{0}}$ is the Dirac delta function centered at $x_{0}$ and $F$ is the distribution of $x$. Then, we have:

$$ $IF(x_{0};T_{LPM},F)=D(x_{0}-y)=D(x_{0}-z_{LPM}/D),$ $$

and

$$ $IF(x_{0};T_{NRPM},F)=\frac{Dw_{0}\left(x_{0}-z_{NRPM}/D\right)}{\sum_{d=1}^{D} w_{d}}\text{ where }w_{0}=\frac{1}{|x_{0}-z_{LPM}/D|}.$ $$

Theorem 3.2 provide several insights into the robustness of LPM and NRLPM models:

- •
For LPM, the influence function is given by $D(x_{0}-y)=D(x_{0}-z_{LPM}/D)$, indicating that the sensitivity of LPM depends on the difference between the perturbation $x_{0}$ and the average clean estimation $z_{LPM}/D$.
- •
For NRPM, the influence function is $\frac{Dw_{0}\left(x_{0}-z_{NRPM}/D\right)}{\sum_{d=1}^{D}w_{d}}$, where $w_{0}=\frac{1}{|x_{0}-z_{LPM}/D|}$. Although the robustness of NRPM is affected by the difference $x_{0}-z_{NRPM}/D$, the influence can be significantly mitigated by the weight $w_{0}$, particularly when $x_{0}$ deviates from the average estimation $z_{LPM}/D$ of
the clean data.

These analyses provide insight and explanation for the robustness nature of the proposed technique.

###### Theorem 3.2 (Robustness Analysis via Influence Function) .

## 4 Robustness Reprogramming

In this section, it is ready to introduce the robustness programming techniques based on the non-linear robust pattern matching (NRPM) derived in Section 3.1. One naive approach is to simply replace the vanilla linear pattern matching (LPM) with NRPM. However, this naive approach does not work well in practice, and we propose three robustness reprogramming paradigms to improve the robustness, accommodating varying resource constraints and robustness requirements.

Figure: Algorithm 1 Hybrid Architecture

While NRPM enhances model robustness, it can lead to a decrease in clean accuracy, particularly in deeper models. This reduction in performance may be due to the increasing estimation error between OLS and LDA estimations across layers. To balance the clean-robustness performance trade-off between LPM and NRPM, it is necessary to develop a hybrid architecture as shown in Algorithm 1, where their balance is controlled by hyperparameters $\{\lambda\}$.
Based on this hybrid architecture, we propose three robustness reprogramming paradigms
as shown in Figure 2:

Figure: Figure 2: *Three Robustness Reprogramming Paradigms* : (1) Paradigm 1 freezes the model parameters and treats $\{\lambda\}$ as fixed hyperparameter; (2) Paradigm 2 freezes the model parameters but allows $\{\lambda\}$ to be learnable; (3) Paradigm 3 enables both the model parameters and $\{\lambda\}$ to be learnable.
Refer to caption: extracted/5905404/figure/paradigm.png

Paradigm 1: without fine-tuning, good robustness with zero cost.
As deep learning models become increasingly larger, it is critical to fully utilize pre-trained model parameters. Since the robust NRPM slightly refines the vanilla LPM through an adaptive instance-wise reweighting scheme, ensuring that the pre-trained LPM parameters still fit well within NRPM architecture.
Additionally, by adjusting the hyperparameters $\{\lambda\}$ with pre-trained parameters, we can strike an ideal balance between natural and robust accuracy.
It is worth noting that relying solely on pre-trained parameters with plug-and play paradigm significantly reduces computational costs, which is crucial in the era of large-scale deep learning.

Paradigm 2: only fine-tuning $\{\lambda\}$, strong robustness with slight cost. One drawback of Paradigm 1 is that we specify the same $\lambda$ for all the layers and need to conduct brute force hyper-parameters search to obtain the optimal one. However,
hyper-parameters search is time-consuming and computation-intensive. Moreover, the entire model requires layer-wise $\{\lambda\}$ to balance the LPM and NRPM for different layers. To solve it, we propose Paradigm 2, which automatically learns optimal $\lambda$ with light-weight fine-tuning.
This paradigm only fine-tunes the hyperparameters $\{\lambda\}$ while keeping the model parameters frozen, which is very efficient.

Paradigm 3: overall fine-tuning, superior robustness with acceptable cost.
To achieve best robustness,
we can make both the model parameters $\{a_{d}\}_{d=1}^{D}$ and the hyperparameters $\{\lambda\}$ learnable. By refining these parameters based on the pre-trained model, we can prevent clean performance degradation. Moreover, with learnable $\{\lambda\}$ during adversarial training, the model can automatically select the optimal combination to enhance both clean and robust performance. This paradigm is still efficient since it only needs to fine-tune the model lightly with a few training epochs.

## 5 Experiment

In this section, we comprehensively evaluate the effectiveness of our proposed *Robustness Reprogramming* techniques using a wide range of backbone architectures, starting from basic MLPs, progressing to shallow LeNet model, and extending to the widely-used ResNets architecture.

### 5.1 Experimental Setting

Datasets. We conduct the experiments on MNIST LeCun & Cortes (), SVHN (Netzer et al., ), CIFAR10 (Krizhevsky et al., ), and ImageNet10 (Russakovsky et al., ) datasets.

Backbone architectures. We select backbones ranging from very basic MLPs with 1, 2, or 3 layers, to mid-level architectures like LeNet, and deeper networks such as ResNet10, ResNet18, and ResNet34 (He et al., ). In some experiments with ResNets, we chose the narrower version (with the model width reduced by a factor of 8) for the consideration for computation issue. Additionally, we also choose one popular architecture MLP-Mixer (Tolstikhin et al., ).

Evaluation methods.
We assess the performance of the models against various attacks, including FGSM (Goodfellow et al., ), PGD-20 (Madry, ), C&W (Carlini & Wagner, ), and AutoAttack (Croce & Hein, ). Among them, AutoAttack is an ensemble attack consisting of three adaptive white-box attacks and one black-box attack, which is considered as a reliable evaluation method to avoid the false sense of security. For basic MLPs, we also examine attacks under different norms, including $L_{0}$, $L_{2}$, and $L_{\infty}$ norms.
In addition to empirical robustness evaluation,
we also evaluate certified robustness to further demonstrate the robustness of our proposed architecture.

Hyperparameter setting.
For the backbone MLPs and LeNet, we train the vanilla models for 50 epochs, followed by fine-tuning for 5 epochs.
For ResNet backbones, we train the models over 200 epochs with an adjusted learning rate, then fine-tune them for an additional 5 epochs. For all methods, the learning rate is tuned within {0.5, 0.01, 0.005} and the weight decay is tune within {5e-4, 5e-5, 5e-6}.
In adversarial training, we apply FGSM attack for MNIST and PGD attack with 7 iterations and 2/255 step size for CIFAR10.

### 5.2 Robustness Reprogramming on MLPs

**Table 1: Robustness reprogramming under 3 paradigms on MNIST with 3-layer MLP as backbone.**
| Method / Budget | Natural | 0.05 | 0.1 | 0.15 | 0.2 | 0.25 | 0.3 | $[\lambda_{1},\lambda_{2},\lambda_{3}]$ |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Normal-train | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 | \ |  |
| Adv-train | 76.4 | 66.0 | 57.6 | 46.9 | 35.0 | 23.0 | 9.1 | \ |  |
|  | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 | [1.0, 1.0, 1.0] |  |
|  | 90.8 | 56.6 | 17.9 | 8.5 | 4.6 | 3.0 | 2.3 | [0.9, 0.9, 0.9] |  |
|  | 90.4 | 67.1 | 30.8 | 17.4 | 10.6 | 6.5 | 4.5 | [0.8, 0.8, 0.8] |  |
|  | 89.7 | 73.7 | 43.5 | 25.5 | 16.9 | 11.7 | 9.2 | [0.7, 0.7, 0.7] |  |
| Paradigm 1 | 88.1 | 75.3 | 49.0 | 31.0 | 22.0 | 15.5 | 12.4 | [0.6, 0.6, 0.6] |  |
| (without tuning) | 84.1 | 74.4 | 50.0 | 31.9 | 22.8 | 18.1 | 14.3 | [0.5, 0.5, 0.5] |  |
|  | 78.8 | 70.4 | 48.3 | 33.9 | 24.1 | 18.4 | 14.6 | [0.4, 0.4, 0.4] |  |
|  | 69.5 | 62.6 | 45.2 | 31.5 | 23.1 | 19.0 | 15.5 | [0.3, 0.3, 0.3] |  |
|  | 58.5 | 53.2 | 38.2 | 27.6 | 22.2 | 16.4 | 12.9 | [0.2, 0.2, 0.2] |  |
|  | 40.7 | 38.3 | 29.7 | 22.8 | 16.8 | 12.9 | 11.1 | [0.1, 0.1, 0.1] |  |
|  | 18.8 | 17.6 | 16.4 | 14.6 | 12.4 | 10.7 | 9.4 | [0.0, 0.0, 0.0] |  |
| Paradigm 2 (tuning $\lambda$) | 81.5 | 75.3 | 61.2 | 44.7 | 33.7 | 26.0 | 20.1 | [0.459, 0.033, 0.131] |  |
| Paradigm 3 (tuning all) | 86.1 | 81.7 | 75.8 | 66.7 | 58.7 | 50.1 | 39.8 | [0.925, 0.119, 0.325] |  |

Comparison of robustness reprogramming via various paradigms. To compare the robustness of our robustness reprogramming under 3 paradigms as well as the vanilla normal/adversarial training, we evaluate the model performance under FGSM attack across various budgets with 3-layer MLP as backbone model.
From the results in Table 1, we can make the following observations:

- •
In terms of robustness, our Robustness Reprogramming across three paradigms progressively enhances performance. In Paradigm 1, by adjusting $\{\lambda\}$, an optimal balance between clean and robust accuracy can be achieved without the need for parameter fine-tuning. Moreover, Paradigm 2 can automatically learn the layer-wise set $\{\lambda\}$, improving robustness compared to Paradigm 1 (with fixed $\{\lambda\}$). Furthermore, Paradigm 3, by fine-tuning all the parameters, demonstrates the best performance among all the methods compared.
- •
Regarding natural accuracy, we can observe from Paradigm 1 that increasing the inclusion of NRLM (i.e., smaller $\{\lambda\}$) results in a decline in performance. But this sacrifice can be mitigated by fine-tuning $\{\lambda\}$ or the entire models as shown in Paradigm 2&3.

**Table 2: Automated learning of $\{\lambda\}$ under adversarial training with various noise levels.**
| Budget | Natural | 0.05 | 0.1 | 0.15 | 0.2 | 0.25 | 0.3 | Learned $\{\lambda\}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Adv-train ($\epsilon=0.0$) | 91.3 | 75.7 | 45.0 | 24.9 | 14.9 | 10.5 | 8.0 | [0.955, 0.706, 0.722] |
| Adv-train ($\epsilon=0.05$) | 91.2 | 76.8 | 50.6 | 27.8 | 16.6 | 10.5 | 8.1 | [0.953, 0.624, 0.748] |
| Adv-train ($\epsilon=0.1$) | 91.0 | 82.6 | 62.5 | 41.7 | 26.4 | 19.8 | 14.9 | [0.936, 0.342, 0.700] |
| Adv-train ($\epsilon=0.15$) | 90.6 | 82.3 | 66.8 | 49.4 | 35.0 | 25.5 | 19.9 | [0.879, 0.148, 0.599] |
| Adv-train ($\epsilon=0.2$) | 90.3 | 82.5 | 67.5 | 49.3 | 35.8 | 27.0 | 21.3 | [0.724, 0.076, 0.420] |
| Adv-train ($\epsilon=0.25$) | 87.7 | 81.0 | 66.0 | 48.5 | 36.4 | 26.6 | 21.6 | [0.572, 0.049, 0.243] |
| Adv-train ($\epsilon=0.3$) | 81.5 | 75.3 | 61.2 | 44.7 | 33.7 | 26.0 | 20.1 | [0.459, 0.033, 0.131] |

Behavior analysis on automated learning of $\mathbf{\{\lambda\}}$. In Paradigm 2, we assert and expect that the learnable $\{\lambda\}$ across layers can achieve optimal performance under a specified noisy environment while maintaining the pre-trained parameters. To further validate our assertion and investigate the behavior of the learned $\{\lambda\}$, we simulate various noisy environments by introducing adversarial perturbations into the training data at different noise levels, $\epsilon$. We initialize the $\{\lambda\}$ across all layers to 0.5. From the results shown in Table 2, we can make the following observations:

- •
The learned $\{\lambda\}$ values are layer-specific, indicating that setting the same $\{\lambda\}$ for each layer in Paradigm 1 is not an optimal strategy. Automated learning of $\{\lambda\}$ enables the discovery of the optimal combination across layers.
- •
As the noise level in the training data increases, the learned $\{\lambda\}$ values tend to decrease, causing the hybrid architecture to resemble a more robust NRPM architecture. This suggests that our proposed Paradigm 2 can adaptively adjust $\{\lambda\}$ to accommodate noisy environments.

Ablation studies. To further investigate the effectiveness of our proposed robust architecture, we provide several ablation studies on backbone size, attack budget measurement, additional backbone MLP-Mixer (Tolstikhin et al., ),the number of layers $K$ in the Appendix C.2, Appendix C.3, and Appendix C.4, respectively.
These experiments demonstrate the consistent advantages of our method across different backbone sizes and attack budget measurements. Additionally, increasing the number of layers $K$ can further enhance robustness, though at the cost of a slight decrease in clean performance.

### 5.3 Robustness Reprogramming on Convolution

Robustness reprogramming on convolution.
We evaluate the performance of our robustness reprogramming with various weight $\{\lambda\}$ in Figure 4
,
where we can observe that incorporating more NRPM-induced embedding will significantly improve the robustness while sacrifice a little bit of clean performance. Moreover, by fine-tuning NRPM-model, we can significantly improve robust performance while compensating for the sacrifice in clean accuracy.

Adversarial fine-tuning. Beyond natural training, we also validate the advantage of
our NRPM architecture over the vanilla LPM under adversarial training. We utilize the pretrained parameter from LPM architecture, and track the clean/robust performance across 10 epochs under the adversarial fine-tuning for both architectures in Figure 4. The curves presented demonstrate the consistent improvement of NRPM over LPM across all the epochs.

Figure: Figure 3: Robustness reprogramming on LeNet. The depth of color represents the size of budget.
Refer to caption: extracted/5905404/figure/lenet_diff_lambda.png

Hidden embedding visualization. We also conduct visualization analyses on the hidden embedding to obtain better insight into the effectiveness of our proposed NRPM. First, we quantify the difference between clean embeddings (${\bm{x}}$ or ${\bm{z}}_{i}$) and attacked embeddings (${\bm{x}}^{\prime}$ or ${\bm{z}}^{\prime}_{i}$) across all layers in Table 3, and visualize them in Figure 5
and Figure 11.
The results in Table 3 show that NRPM-LeNet has smaller embedding difference across layers, indicating that our proposed NRPM architecture indeed mitigates the impact of the adversarial perturbation. Moreover,
as demonstrated in the example
in Figure 5, the presence of adversarial perturbations can disrupt the hidden embedding patterns, leading to incorrect predictions in the case of vanilla LeNet.
In contrast, our NRPM-LeNet appears to lessen the effects of such perturbations and maintain predicting groundtruth label. From the figures, we can also clearly tell that the difference between clean attacked embeddings of LPM-LeNet is much more significant than in NRPM-LeNet.

**Table 3: Embedding difference between clean and adversarial data ($\epsilon=0.3$) in LeNet - MNIST**
| LPM-LeNet | $||\cdot||_{1}$ | $||\cdot||_{2}$ | $||\cdot||_{\infty}$ |
| --- | --- | --- | --- |
| $|{\bm{x}}-{\bm{x}}^{\prime}|$ | 93.21 | 27.54 | 0.30 |
| $|{\bm{z}}_{1}-{\bm{z}}_{1}^{\prime}|$ | 271.20 | 116.94 | 1.56 |
| $|{\bm{z}}_{2}-{\bm{z}}_{2}^{\prime}|$ | 79.52 | 62.17 | 1.89 |
| $|{\bm{z}}_{3}-{\bm{z}}_{3}^{\prime}|$ | 18.77 | 22.56 | 2.32 |
| $|{\bm{z}}_{4}-{\bm{z}}_{4}^{\prime}|$ | 9.84 | 18.95 | 3.34 |

Figure: (a) Visualization on LPM-LeNet.
Refer to caption: extracted/5905404/figure/lenet_visual_l2.png

Additional experiments.
To further demonstrate the effectiveness of our proposed method, we include experiments on two additional datasets, SVHN and ImageNet10, which are provided in Appendix D.2.
All these experiments demonstrate consistent advantages of our proposed method.

### 5.4 Robustness Reprogramming on ResNets

In this section, we will evaluate the robustness of robustness reprogramming on ResNets across various attacks and further validate the effectiveness under diverse settings in the ablation studies.

**Table 4: Robustness reprogramming on CIFAR10 with narrow ResNet18 as backbone.**
| Budget $\epsilon$ | Natural | 8/255 | 16/255 | 32/255 | $\{\lambda\}$ |
| --- | --- | --- | --- | --- | --- |
|  | 67.89 | 41.46 | 16.99 | 3.17 | $\lambda=1.0$ |
|  | 59.23 | 40.03 | 23.05 | 8.71 | $\lambda=0.9$ |
|  | 40.79 | 28.32 | 20.27 | 13.70 | $\lambda=0.8$ |
| Paradigm 1 | 24.69 | 18.34 | 15.31 | 13.09 | $\lambda=0.7$ |
| (without tuning) | 17.84 | 14.26 | 12.63 | 11.66 | $\lambda=0.6$ |
|  | 15.99 | 12.51 | 11.42 | 11.07 | $\lambda=0.5$ |
|  | 10.03 | 10.02 | 10.0 | 10.0 | $\lambda=0.4$ |
| Paradigm 2 (tuning $\lambda$) | 69.08 | 44.09 | 24.94 | 12.63 | Learnable |
| Paradigm 3 (tuning all) | 71.79 | 50.89 | 39.58 | 30.03 | Learnable |

Robustness reprogramming on ResNets.
We evaluate the robustness of our robustness reprogramming via three paradigms in Table 4. From the results, we can observe that: (1)
In Paradigm 1, adding more NRPM-based embeddings without fine-tuning leads to a notable drop in clean performance in ResNet18, which also limits the robustness improvement.
(2) In Paradigm 2, by adjusting only the $\{\lambda\}$ values, we can improve the clean and robust performance, indicating the need of layer-wise balance across different layers.
(3) In Paradigm 3, by tuning both $\{\lambda\}$ and parameters, we observe that both the accuracy and robustness can be further improved.

Adversarial robustness.
To validate the effectiveness of our robustness reprogramming with existing method, we select several existing popular adversarial defenses and report the experimental results of backbone ResNet18 under various attacks in Table 5. From the results we can observe that our robustness reprgramming exhibits excellent robustness across various attacks.

**Table 5: Adversarial robsustness on CIFAR10 with ResNet18 as backbone.**
| Method | Natural | PGD | FGSM | C&W | AA | AVG |
| --- | --- | --- | --- | --- | --- | --- |
| PGD-AT | 80.90 | 44.35 | 58.41 | 46.72 | 42.14 | 47.91 |
| TRADES-2.0 | 82.80 | 48.32 | 51.67 | 40.65 | 36.40 | 44.26 |
| TRADES-0.2 | 85.74 | 32.63 | 44.26 | 26.70 | 19.00 | 30.65 |
| MART | 79.03 | 48.90 | 60.86 | 45.92 | 43.88 | 49.89 |
| SAT | 63.28 | 43.57 | 50.13 | 47.47 | 39.72 | 45.22 |
| AWP | 81.20 | 51.60 | 55.30 | 48.00 | 46.90 | 50.45 |
| Paradigm 3 (Ours) | 80.43 | 57.23 | 70.23 | 64.07 | 64.60 | 64.03 |

Figure: Figure 6: Certified robustness via randomized smoothing with various $\sigma$ levels.
Refer to caption: extracted/5905404/figure/vary_noise_cifar10.png

Certified Robustness.
Additionally, we also evaluate the
certified robustness using randomized smoothing with various $\sigma$ levels in Figure 7. The curves presented in the figure demonstrate a significant advantage of our NRPM over vanilla LPM, further validating the effectiveness of our proposed architecture.

Different backbone sizes & budgets. Here, we conduct ablation studies on the backbone size & budget
under AutoAttack in Figure 7 and leave the results under PGD in Appendix E.1. The results show the evident advantage of our NRPM architecture.

## 6 Conclusion

This paper addresses a fundamental challenge in representation learning: how to reprogram a well-trained model to enhance its robustness without altering its parameters. We begin by revisiting the essential linear pattern matching in representation learning and then introduce an alternative non-linear robust pattern matching mechanism. Additionally, we present a novel and efficient Robustness Reprogramming framework, which can be flexibly applied under three paradigms, making it suitable for practical scenarios.
Our theoretical analysis and comprehensive empirical evaluation demonstrate significant and consistent performance improvements.
This research offers a promising and complementary approach to strengthening adversarial defenses in deep learning, significantly contributing to the development of more resilient AI systems.

## Appendix A Efficient Implementation of Robust Convolution/MLPs

In the main paper, we formulate the pixel-wise algorithm for notation simplicity. However, it is not trivial to implement such theory into practice in convolution and MLPs layer. Here, we will present the detailed implementation or techniques for each case.

### A.1 Robust MLPs

In MLPs, we denote the input as ${\mathbf{X}}\in\mathbb{R}^{D_{1}}$, the output embedding as ${\mathbf{Z}}\in\mathbb{R}^{D_{2}}$, the
model parameters as ${\mathbf{A}}\in\mathbb{R}^{D_{1}\times D_{2}}$. The pseudo code is presented in Algorithm 2.

Figure: Algorithm 2 Robust MLPs with NRPM
[Uncaptioned image]: x1.png

### A.2 Robust Convolution

Unfolding.
To apply robust estimation over each patch, we must first unfold the convolution into multiple patches and process them individually. While it’s possible to use a for loop to extract patches across the channels, height, and width, this approach is not efficient.
Instead, we can leverage PyTorch’s torch.nn.functional.unfold function to streamline and accelerate both the unfolding and computation process.

## Appendix B Theoretical Proof

### B.1 Proof of Lemma 3.1

###### Proof.

Since $\sqrt{a}\leq\frac{a}{2\sqrt{b}}+\frac{\sqrt{b}}{2}$ and the equlity holds when $a=b$, by replacemnet as $a=(a_{d}x_{d}-z/D)^{2}$ and $b=(a_{d}x_{d}-z_{0}/D)^{2}$, then

$$ $\displaystyle|a_{d}x_{d}-z/D|$ $\displaystyle\leq\frac{1}{2}\cdot\frac{1}{|a_{d}x_{d}-z^{(k)}/D|}\cdot(a_{d}x_ {d}-z/D)^{2}+\frac{1}{2}|a_{d}x_{d}-z_{0}/D|$ $\displaystyle=w_{d}\cdot(a_{d}x_{d}-z/D)^{2}+\frac{1}{2}|a_{d}x_{d}-z_{0}/D|$ $$

Sum up the items on both sides, we obtain

$$ $\displaystyle{\mathcal{L}}(z)=\sum_{d=1}^{D}|a_{d}x_{d}-z/D|\leq\sum_{d=1}^{D} w_{d}\cdot(a_{d}x_{d}-z/D)^{2}+\frac{1}{2}\sum_{d=1}^{D}|a_{d}x_{d}-z_{0}/D| \frac{1}{2}={\mathcal{U}}(z,z_{0})$ $$

and the equality holds at $a=b$ ($z=z_{0}$):

$$ ${\mathcal{U}}(z_{0},z_{0})={\mathcal{L}}(z_{0}).$ (4) $$

∎

###### Proof.

### B.2 Derivation of IRLS algorithm

###### Proof.

$$ ${\mathcal{U}}^{\prime}(z^{(k)},z^{(k)})=\frac{1}{D}\sum_{d=1}^{D}w^{(k)}_{d} \cdot 2(z^{(k)}/D-a_{d}x_{d})$ $$

$$ ${\mathcal{U}}^{\prime\prime}(z^{(k)},z^{(k)})=\frac{2}{D^{2}}\sum_{d=1}^{D}w^{ (k)}_{d}$ $$

$$ $\displaystyle z^{(k+1)}$ $\displaystyle=z^{(k)}-\frac{{\mathcal{U}}^{\prime}(z^{(k)},z^{(k)})}{{\mathcal {U}}^{\prime\prime}(z^{(k)},z^{(k)})}$ (5) $\displaystyle=z^{(k)}-\frac{\frac{1}{D}\sum_{d=1}^{D}w^{(k)}_{d}\cdot 2(z^{(k) }/D-a_{d}x_{d})}{\frac{2}{D^{2}}\sum_{d=1}^{D}w^{(k)}_{d}}$ (6) $\displaystyle=D\cdot\frac{\sum_{d=1}^{D}w^{(k)}_{d}a_{d}x_{d}}{\sum_{d=1}^{D}w ^{(k)}_{d}}$ (7) $$

where $w_{d}^{(k)}=\frac{1}{2}\cdot\frac{1}{|a_{d}x_{d}-z^{(k)}/D|}.$
Since the constant in $w_{d}^{(k)}$ can be canceled in the updated formulation, we have:

$$ $w_{d}^{(k)}=\frac{1}{|a_{d}x_{d}-z^{(k)}/D|}.$ $$

∎

###### Proof.

### B.3 Proof of Theorem 3.2

###### Proof.

For notation simplicity, we denote $y=\frac{1}{D}T_{LPM}(F)$, $y_{\epsilon}=\frac{1}{D}T_{LPM}(F_{\epsilon})$, $z=\frac{1}{D}T_{NRPM}(F)$, $z_{\epsilon}=\frac{1}{D}T_{NRPM}(F_{\epsilon})$. Since $\epsilon$ is very small and $D$ is large enough, we can assume $|a_{d}x_{d}-y|\approx|a_{d}x_{d}-y_{\epsilon}|$ for simplicity.

Influence function for LPM.
The new weighted average $y_{\epsilon}$ under contamination becomes:

$$ $y_{\epsilon}=(1-\epsilon)y+\epsilon x_{0}$ $$

Taking the limit as $\epsilon\to 0$, we get:

$$ $IF(x_{0};T_{LPM},F)=\lim_{\epsilon\to 0}\frac{D(y_{\epsilon}-y)}{\epsilon}= \lim_{\epsilon\to 0}\frac{D\epsilon(x_{0}-y)}{\epsilon}=D(x_{0}-y)=D(x_{0}-z_{ LPM}/D)$ $$

This shows that the weighted average $y$ is directly influenced by $x_{0}$, making it sensitive to outliers.

Influence function for NRPM.
Next, we consider the reweighted estimate $z$, when we introduce a small contamination at $x_{0}$, which changes the distribution slightly. The contaminated reweighted estimate $z_{\epsilon}$ becomes:

$$ $z_{\epsilon}=\frac{(1-\epsilon)\sum_{d=1}^{D}w_{d}a_{d}x_{d}+\epsilon w_{0}x_{ 0}}{(1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}}$ $$

We can simplify the expression for $z_{\epsilon}$ as:

$$ $z_{\epsilon}=\frac{\sum_{d=1}^{D}w_{d}a_{d}x_{d}+\epsilon(w_{0}x_{0}-\sum_{d=1 }^{D}w_{d}a_{d}x_{d})}{(1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}}$ $$

First, expand the difference $z_{\epsilon}-z$:

$$ $\displaystyle z_{\epsilon}-z=$ $\displaystyle\frac{\sum_{d=1}^{D}w_{d}a_{d}x_{d}+\epsilon(w_{0}x_{0}-\sum_{d=1 }^{D}w_{d}a_{d}x_{d})}{(1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}}-\frac{ \sum_{d=1}^{D}w_{d}a_{d}x_{d}}{\sum_{d=1}^{D}w_{d}}$ $\displaystyle=$ $\displaystyle\frac{\sum_{d=1}^{D}w_{d}a_{d}x_{d}\cdot\sum_{d=1}^{D}w_{d}+ \epsilon(w_{0}x_{0}-\sum_{d=1}^{D}w_{d}a_{d}x_{d})\cdot\sum_{d=1}^{D}w_{d}}{[( 1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}]\cdot\sum_{d=1}^{D}w_{d}}$ $\displaystyle+\frac{[(1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}]\cdot\sum_ {d=1}^{D}w_{d}a_{d}x_{d}}{[(1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}] \cdot\sum_{d=1}^{D}w_{d}}$ $\displaystyle=$ $\displaystyle\frac{\epsilon(w_{0}x_{0}-\sum_{d=1}^{D}w_{d}a_{d}x_{d})\cdot\sum _{d=1}^{D}w_{d}-\epsilon(w_{0}-\sum_{d=1}^{D}w_{d})\cdot\sum_{d=1}^{D}w_{d}a_{ d}x_{d}}{[(1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}]\cdot\sum_{d=1}^{D}w_ {d}}$ $\displaystyle=$ $\displaystyle\frac{\epsilon(w_{0}x_{0}\cdot\sum_{d=1}^{D}w_{d}-w_{0}\cdot\sum_ {d=1}^{D}w_{d}a_{d}x_{d})}{[(1-\epsilon)\sum_{d=1}^{D}w_{d}+\epsilon w_{0}] \cdot\sum_{d=1}^{D}w_{d}}$ $$

Finally, divide the difference by $\epsilon$ and take the limit as $\epsilon\to 0$:

$$ $\displaystyle IF(x_{0};T_{NRPM},F)$ $\displaystyle=\lim_{\epsilon\to 0}\frac{D(z_{\epsilon}-z)}{\epsilon}$ $\displaystyle=D\cdot\frac{w_{0}x_{0}\cdot\sum_{d=1}^{D}w_{d}-w_{0}\cdot\sum_{d =1}^{D}w_{d}a_{d}x_{d}}{\sum_{d=1}^{D}w_{d}\cdot\sum_{d=1}^{D}w_{d}}$ $\displaystyle=\frac{Dw_{0}\left(x_{0}-z_{NRPM}/D\right)}{\sum_{d=1}^{D}w_{d}}$ $$

Since $w_{0}$ is small for outliers (large $|x_{0}-z_{LPM}/D|$), the influence of $x_{0}$ on NRPM is diminished compared to the influence on LPM. Therefore, NRPM is more robust than LPM because the influence of outliers is reduced.

∎

###### Proof.

## Appendix C Additional Experimental Results on MLPs

### C.1 MLP-Mixer

MLP-Mixer (Tolstikhin et al., ) serves as an alternative model to CNNs in computer vision. Building on the excellent performance of basic MLPs, we further explore the effectiveness of our NRPM with MLP-Mixer, presenting the results in Figure 8 and Table 6. The results validate the effectiveness of our proposed methdod in MLP-Mixer with different blocks across various budgets.

Figure: Figure 8: Robustness under PGD on MLP-Mixer. The depth of the color represents the size of the budget.
Refer to caption: extracted/5905404/figure/mlp_mixer.png

**Table 6: Adversarial Robustness of MLP-Mixer - CIFAR10**
| Model | Natural | $\epsilon=1/255$ | $\epsilon=2/255$ | $\epsilon=3/255$ | $\epsilon=4/255$ | $\epsilon=8/255$ |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LPM-MLP-Mixer-2 | 77.45 | 38.55 | 10.39 | 2.24 | 0.34 | 0.0 |  |
| NRPM-MLP-Mixer-2 | 76.86 | 64.27 | 49.62 | 37.22 | 27.99 | 15.73 |  |
| LPM-MLP-Mixer-4 | 78.61 | 35.99 | 9.35 | 1.45 | 0.19 | 0.0 |  |
| NRPM-MLP-Mixer-4 | 77.51 | 66.72 | 51.73 | 39.43 | 30.88 | 18.31 |  |
| LPM-MLP-Mixer-8 | 78.98 | 40.55 | 12.81 | 3.27 | 0.73 | 0.01 |  |
| NRPM-MLP-Mixer-8 | 78.32 | 66.82 | 52.73 | 40.37 | 32.66 | 20.61 |  |

### C.2 The effect of backbone size.

We select the MLP backbones with different layers to investigate the performance under different model sizes. We report the performance with the linear models of 1 (784 - 10), 2 (784 - 64 -10), 3 (784 - 256 - 64 -10) layers. We present the fine-tuning results for both the LPM model and NRPM model in Table 7, from which we can make the following observations:

- •
Our RosNet show better robustness with different size of backbones
- •
Regarding clean performance, linear models with varying layers are equivalent and comparable. However, our RosNet shows some improvement as the number of layers increases.
- •
The robustness of our RosNet enhances with the addition of more layers. This suggests that our robust layers effectively mitigate the impact of perturbations as they propagate through the network.

**Table 7: Robustness of MLPs with different layers on MNIST.**
| Model Size | Arch / Budget | 0 | 0.05 | 0.1 | 0.15 | 0.2 | 0.25 | 0.3 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 Layer | LPM-model | 91.1 | 12.0 | 7.1 | 0.5 | 0.0 | 0.0 | 0.0 |
| NRPM-model | 87.6 | 14.0 | 13.6 | 12.9 | 12.1 | 11.5 | 10.3 |  |
| 2 Layers | LPM-model | 91.6 | 32.9 | 2.6 | 0.2 | 0.0 | 0.0 | 0.0 |
| NRPM-model | 89.1 | 37.0 | 34.2 | 30.7 | 25.8 | 22.7 | 21.2 |  |
| 3 Layers | LPM-model | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 |
| NRPM-model | 90.7 | 47.2 | 45.1 | 37.8 | 30.8 | 24.9 | 21.1 |  |

### C.3 Attack budget measurement

Figure: (a) $L_{0}$-attack
Refer to caption: extracted/5905404/figure/mlp_L_0.png

In previous results of different backbone sizes, we notice the NRPM model is not effective enough under $L_{\infty}$-attack. To verify the effect the attack budget measurement, we evaluate the robustness under $L_{\infty}$,$L_{2}$,$L_{0}$ attacks where the budget is measured under the $L_{\infty}$,$L_{2}$,$L_{0}$-norms, that is, $\|x-x^{\prime}\|_{p}\leq\text{budget},$ where $p=0,2,\infty$. We can make the following observations from the results in Figure 9:

- •
Our NRPM architecture consistently improve the robustness over the LPM-backbone across various attacks and budgets.
- •
Our NRPM model experiences a slight decrease in clean accuracy, suggesting that the median retains less information than the mean.
- •
Our NRPM model shows improved performance under the $L_{0}$ and $L_{2}$ attacks compared to the $L_{\infty}$ attack. This behavior aligns with the properties of mean and median. Specifically, under the $L_{\infty}$ attack, each pixel can be perturbed up to a certain limit, allowing both the mean and the median to be easily altered when all pixels are modified. Conversely, under the $L_{2}$ and $L_{0}$ attacks, the number of perturbed pixels is restricted, making the median more resilient to disruption than the mean.

### C.4 Effect of Different Layers K 𝐾 K italic_K

To investigate the effect of the number of iterations $K$ in the unrolled architecture, we present the results with increasing $K$ on combined models in the Table 8 and Figure 10 in the appendix. As $K$ increases, the hybrid model can approach to the NRPM architecture, which makes the model more robust while slightly sacrificing a little bit natural accuracy.

**Table 8: Ablation study on number of layers $K$ - MLP - MNIST**
| $K$ / $\epsilon$ | 0 | 0.05 | 0.1 | 0.15 | 0.2 | 0.25 | 0.3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $\lambda=0.9$ |  |  |  |  |  |  |  |
| $K=0$ | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 |
| $K=1$ | 90.8 | 56.6 | 17.9 | 8.5 | 4.6 | 3.0 | 2.3 |
| $K=2$ | 90.7 | 76.1 | 44.0 | 26.8 | 18.0 | 13.4 | 9.3 |
| $K=3$ | 90.7 | 82.1 | 56.7 | 37.4 | 26.3 | 19.9 | 16.3 |
| $K=4$ | 90.6 | 83.5 | 61.1 | 42.7 | 29.4 | 22.7 | 18.0 |
| $\lambda=0.8$ |  |  |  |  |  |  |  |
| $K=0$ | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 |
| $K=1$ | 90.4 | 67.1 | 30.8 | 17.4 | 10.6 | 6.5 | 4.5 |
| $K=2$ | 90.4 | 81.1 | 56.2 | 36.2 | 26.0 | 19.3 | 15.5 |
| $K=3$ | 90.3 | 81.4 | 61.8 | 42.9 | 31.8 | 23.7 | 19.5 |
| $K=4$ | 90.3 | 82.5 | 64.3 | 45.6 | 33.1 | 25.3 | 22.0 |
| $\lambda=0.7$ |  |  |  |  |  |  |  |
| $K=0$ | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 |
| $K=1$ | 89.7 | 73.7 | 43.5 | 25.5 | 16.9 | 11.7 | 9.2 |
| $K=2$ | 89.1 | 80.3 | 56.4 | 39.7 | 28.4 | 21.4 | 18.5 |
| $K=3$ | 88.9 | 80.3 | 61.8 | 44.0 | 33.2 | 24.4 | 19.2 |
| $K=4$ | 88.4 | 79.6 | 61.3 | 43.6 | 33.9 | 26.3 | 21.4 |
| $\lambda=0.6$ |  |  |  |  |  |  |  |
| $K=0$ | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 |
| $K=1$ | 88.1 | 75.3 | 49.0 | 31.0 | 22.0 | 15.5 | 12.4 |
| $K=2$ | 86.2 | 77.1 | 56.2 | 39.3 | 29.2 | 22.6 | 17.1 |
| $K=3$ | 85.5 | 75.9 | 55.8 | 40.2 | 30.7 | 23.1 | 18.8 |
| $K=4$ | 84.0 | 74.7 | 56.2 | 39.6 | 29.4 | 22.1 | 18.7 |
| $\lambda=0.5$ |  |  |  |  |  |  |  |
| $K=0$ | 90.8 | 31.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.0 |
| $K=1$ | 84.1 | 74.4 | 50.0 | 31.9 | 22.8 | 18.1 | 14.3 |
| $K=2$ | 80.3 | 71.0 | 53.5 | 37.0 | 26.9 | 21.5 | 16.4 |
| $K=3$ | 79.0 | 69.6 | 53.7 | 38.1 | 29.2 | 23.2 | 19.5 |
| $K=4$ | 77.7 | 66.6 | 51.2 | 37.9 | 29.7 | 23.7 | 19.8 |

Figure: (a) $\lambda=0.5$
Refer to caption: extracted/5905404/figure/mlp_diff_num_layers_lmbd0.5.png

## Appendix D Additional Experimental Results on LeNet

### D.1 Visualization of hidden embedding

We put several examples of the visualizations of hidden embedding in Figure 11.

Figure: Figure 11: Visualization of hidden embedding
Refer to caption: x2.png

### D.2 Additional Datasets

Besides MNIST, we also conduct the experiment on SVHN and ImageNet10, and show the results under PGD in Table 9 and Table 10, respectively.

**Table 9: Robustness of LeNet under PGD on SVHN.**
| Model / Budget $\epsilon$ | 0/255 | 1/255 | 2/255 | 4/255 | 8/255 | 16/255 | 32/255 | 64/255 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LPM-LeNet | 83.8 | 68.6 | 49.3 | 27.1 | 8.2 | 3.0 | 1.9 | 1.3 |
| NRPM-Lenet | 83.2 | 72.4 | 54.6 | 35.4 | 20.1 | 14.3 | 11.8 | 7.9 |

**Table 10: Robustness of LeNet under PGD on ImageNet10.**
| Model / Budget $\epsilon$ | 0/255 | 2/255 | 4/255 | 8/255 |
| --- | --- | --- | --- | --- |
| LPM-LeNet | 53.72 | 13.11 | 7.67 | 3.89 |
| NRPM-LeNet | 53.55 | 15.06 | 13.68 | 12.50 |

## Appendix E Additional Experimental Results on ResNets

### E.1 Effect of backbone size

Figure: Figure 12: Ablation study on backbone size. (PGD)
Refer to caption: extracted/5905404/figure/resnet_diff_backbone_pgd.png

**Table 11: Ablation study on backbone size with narrow ResNet - CIFAR10 (PGD)**
| Model/PGD | Datural | 8/255 | 16/255 | 32/255 |
| --- | --- | --- | --- | --- |
| LPM-ResNet10 | 64.51 | 37.26 | 15.41 | 2.71 |
| NRPM-ResNet10 | 67.00 | 40.02 | 27.52 | 19.18 |
| LPM-ResNet18 | 67.89 | 41.46 | 16.99 | 3.17 |
| NRPM-ResNet18 | 71.79 | 50.89 | 39.58 | 30.03 |
| LPM-ResNet34 | 72.54 | 43.7 | 16.76 | 3.16 |
| NRPM-ResNet34 | 74.55 | 50.12 | 36.29 | 25.54 |

**Table 12: Ablation study on backbone size with narrow ResNet - CIFAR10 (AutoAttack)**
| Model/AA | Datural | 8/255 | 16/255 | 32/255 |
| --- | --- | --- | --- | --- |
| LPM-ResNet10 | 64.51 | 28.70 | 5.50 | 0.20 |
| NRPM-ResNet10 | 67.00 | 40.40 | 30.10 | 12.80 |
| LPM-ResNet18 | 67.89 | 30.50 | 7.20 | 0.40 |
| NRPM-ResNet18 | 71.79 | 51.60 | 43.40 | 26.40 |
| LPM-ResNet34 | 72.54 | 33.90 | 9.20 | 0.30 |
| NRPM-ResNet34 | 74.55 | 54.20 | 45.00 | 28.90 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Related to weakness #1 above, on line 167 it is stated that by replacing the loss function with the proposed one the impact of outliers will be reduced, referring to Huber and Ronchetti's book Robust Statistics. The connection is unclear to me and I think deserves a more detailed explanation. Can you expand? 

2. Line 105, by "robust training" do you mean "adversarial training"? 

3. Line 244: "One naive approach is to simply replace the vanilla linear pattern matching (LPM) with NRPM. However, this naive approach does not work well in practice". Why doesn't this approach work? 

4. Line 231: according to the text, $x_o$ is not the perturbation but the initial input. Is that correct? 

5. In table1 and others, what are the numbers at the top of each column? Are they the epsilon values used for test? What attack was used? 

6. Adaptation of the approach to CNNs needs more explanation. Were separate $w_i$ parameters considered per filter or per activation? 

7. Fig 5: likelihood in the bar plots doesn't look like proper probability density functions. What is being shown? 

8. Fig 5 caption: 1 and 2 correspond to (a) and (b) in the figure?
soundness: 3
strengths: 1. Proposed idea of reprogramming is interesting as it would allow reusing the learned features by the original model. May be an important approach to explore in large models. 
2. Empirical results consider multiple datasets of various sizes and different perturbations
summary: This paper proposes a "reprogramming" approach to improve the robustness of neural networks with minimal or no alteration of the model parameters. The reprogramming approach is executed by incorporating additional set of parameters (one per input dimension) in each layer to reweight the importance of features at inference time. The approach is motivated from a perspective of looking at the function of each unit as a linear feature pattern matching that is proposed to be replaced by a nonlinear version. The final implementation of the approach combines the activity of both linear and nonlinear feature pattern matching methods.
weaknesses: 1. The approach is based on eq 1 and the precursor to this for what is named as "linear feature pattern matching". The first equation (unnumbered equation before eq 1) formulates the OLS solution as the minimum of \mathcal{L}=\sum_{d=1}^{D}{(\frac{y}{D}-a_d.x_d)^2}. However, this formulation of the solution assumes that the prediction error is distributed uniformly across all dimensions which is not generally true and may indeed be very constraining in certain situations. Despite this possible issue, I didn't see any mention of this anywhere in the paper which in my opinion is a big weakness. 

2. Some of the key implementation details of the approach were unclear. What data is used to optimize the $w_i$ parameters? Are they optimized over the full training set of each dataset? Alg 1 contains a loop of K iterations for each $x_d$. Does this mean that the approach finds independent $w_d$ for each sample independently? In what order are different parameters/hyperparameters optimized? A step by step description of the approach or a pseudo-code would be helpful if not necessary. 

3. The approach requires computing additional activations for every layer that may be costly but no comparison of runtime, computation cost etc is provided. 

4. Details of Adv-train baseline is missing. Was this the result of adversarial training from scratch or fine-tuning? 

5. In Table5, the AA accuracy is substantially higher than that of PGD (64.60 vs. 57.23). This to me looks like a big red flag. AA is an ensemble attack made of 4 separate attacks including two effective variations of PGD. Because of this, AA accuracy is always lower than that of PGD. I'm doubtful of the validity of empirical evaluations.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * What is the difference in inference efficiency between a standard MLP and a reprogrammed MLP (in terms of memory usage/inference time)?
* Similarly, how does execution time scale with the depth/width of the network?
* Are there any adaptive attacks that could improve over AutoAttack against robust reprogramming? Does this approach just reduce to gradient masking?
* Is the NRPM approach suitable for training from scratch? If so, do you think that would that further improve over the performance of paradigm 3?
* Would an adversary with knowledge of the robust reprogramming scheme be able to design an adaptive attack to counteract these defenses?
soundness: 3
strengths: * There is a great deal of novelty in this approach. Many defenses against adversarial examples atttempt to learn weights that are more robust, or to detect/purify adversarial inputs to a network. I am not aware of any prior work that modifies the model of computation to increase robustness in this way.
* Code and model weights have been provided for replication purposes. 
* The experimental results are impressive. In particular, in certain cases the third paradigm can increase both robustness and clean accuracy.
summary: Many methods have been proposed to improve the adversarial robustness of neural networks, including adversarial training, purification, and regularization. However, these approaches don't take advantage of the learned parameters of models that were not robustly trained. This paper proposed robust reprogramming, which uses those learned parameters to construct a new model that is robust to adversarial perturbations by design. The authors identify linear transformations to essentially be linear pattern matching (LPM), with the pattern being specified by the parameters of the transformation which are capable of matching related features in the input. They further note that linear feature pattern matching is the optimal closed-form solution to the Optimized Least Squares problem, which is particularly sensitive to outliers due to its quadratic penalty. To reduce this sensitivity, this work introduces a new feature pattern matching term based on Least Absolute Deviation estimation, which is called Nonlinear Robust Pattern Matching (NRPM). Theoretical results show that adopting NRPM mitigates the influence of any datapoint on downstream predictions. Three paradigms of robust reprogramming are then introduced: one which only involves implementing the NRPM architecture, one which involves NRPM and fine-tuning hyperparameters, and one which involves NRPM and finetuning all parameters in the model. Experimental results show that these paradigms (and especially paradigm 3) can provide significant improvements over existing robustness techniques. For example, in ResNets trained on CIFAR10, paradigm 3 results in significantly higher robustness than state of the art adversarial training techniques, with only a modest drop in clean accuracy.
weaknesses: * Theorem 3.2 is presented in a confusing manner. You refer to $x_0$ as the perturbation, but isn't it instead the location of a possible perturbation?
* The theoretical justification for the robustness of NRPM largely relies on the influence functions that are derived for NRPM and LPM. However, there's no explicit connection that's made between influence functions and adversarial robustness. It's not immediately clear to me that the influence function on its own would immediately imply adversarial robustness. I think additional justification for this relationship would be necessary in the final version of this paper (or references to related works which establish this relationship).
* On a similar note, there are settings that have been studied in the literature in which an adversary is capable of making unbounded changes to the input (i.e. [1,2]). It seems that the influence function would be less informative about the robustness of the model to adversaries that are unbounded in $\ell_p$ space. I think it should be made more explicit if you are only envisioning this as a defense against $\ell_p$ bounded adversaries.
* The table and figure captions could use more information about the experimental setups. For example, its unclear which attacks/norms mentioned in section 5.1 are used in which table.

[1] Xiao, Chaowei, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. "Spatially transformed adversarial examples." arXiv preprint arXiv:1801.02612 (2018).
[2] Kaufmann, Max, Daniel Kang, Yi Sun, Steven Basart, Xuwang Yin, Mantas Mazeika, Akul Arora et al. "Testing robustness against unforeseen adversaries." arXiv preprint arXiv:1908.08016 (2019).

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - In table 5, robust accuracy against AutoAttack seems to be high, and it would be better to check whether the model suffers from obfuscated gradient or not. For example, following scenarios could be provided: 1) evaluation against adaptive attacks, 2) checking for gradient masking, or 3) evaluation against PGD attacks with extremely large attack strength where robust accuracy should be almost zero while PGD attacks using same strength but different step sizes and steps (e.g., 2 times smaller step size but 2 times larger attack steps) should achieve similar robust accuracy from original attack setting. 
- For the proposed three different paradigms, it would be better to explain when it is more effective to use which paradigm to achieve robustness efficiently based on the differences between three scenarios.
soundness: 2
strengths: - This paper provides comprehensive and detailed theoretical analysis and explanations.
- This paper is well written to follow.
summary: This paper proposed nonlinear robust pattern matching algorithm (NRPM) that can improves the robustness without modifying its parameters while maintaining the feature pattern matching by reprogramming a pretrained model. Especially, this paper theoretically analyzed linear pattern matching and Newton-iterative reweighted least squares algorithm, and introduced robustness reprogramming paradigms under three different situations: parameter-free tuning, lightweight fine-tuning and full fine-tuning. The experimental results support its theoretical analysis and demonstrate the effectiveness of proposed method.
weaknesses: - This paper leveraged old baselines for ResNet18 backbone while recent baselines (e.g., [1], [2]) are missing and evaluation against stronger attack (e.g., LGV, SPSA, DeepFool) could be beneficial.
   - [1] Consistency regularization for adversarial robustness
   - [2] Dynamic Label Adversarial Training for Deep Learning Robustness Against Adversarial Attacks
- This paper utilizes backbones that are too small and tasks that are too easy. For example, main table (Table 1, 2) used MLPs and MNIST. It would be beneficial to provide experimental results on CIFAR-100 and Tiny-ImageNet at least with backbone of ResNet18 and WideResNet28-10. 
- Under same computing resource, it would be better to directly compare the required resources (e.g., FLOPs) or training time (e.g., GPU Hours) between the proposed method (e.g., paradigm 3, full-tuning) and existing methods for ResNet18 backbone.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: Refer to weaknesses.
soundness: 2
strengths: - The approach is thoroughly evaluated on multiple datasets and simpler models.
- It empirically performs well under increasing attack strength/various attack types.
summary: The paper proposes an adversarial robustness scheme that doesn't necessarily require altering the pretrained parameters. They propose a "reprogramming" scheme to adapt the representations learned by a pretrained network in order to enhance the network's robustness. The proposed approach is extensively evaluated on multiple datasets and simpler model architectures and is shown to enhance empirical performance.
weaknesses: 1) The readability of the paper is not good in its current form. The paper doesn't provide sufficient background/cite references to understand the terminology used or where ideas are derived from. Overall, I find it very hard to understand the ideas/contributions of this work, even though the results look promising.

2) The notation is inconsistent, and several terms are used without introduction. For example,

      a) What is X in the indicator function of F in line 212?

      b) What is y in Line 223? Such missing details make it very hard to follow the ideas introduced in the paper.

      c) What is the difference between {\lambda} and \lambda? Guessing from Table 1, the weights are learned/assigned per layer? The notation does not convey this correctly (it should either be a vector or otherwise properly subscripted in the notation).

      d) What is N in algorithm 2?

3) The figures lack sufficient description to understand what's happening. How are the visualizations of hidden embeddings created in Figure 5? What is meant by the "size of the budget" in Figure 3? (my intuitive guess is that refers to the attack budget, but under what norm? Which attack? The language needs to be a lot clearer and the captions/descriptions are too succinct and vague in the current draft).

4) Please include an analysis of how exactly is this better than adversarial training. How do the training time and number of parameters compare for the two approaches?

5) I also find it hard to appreciate the need for the 3 separate training paradigms. A discussion of when and how exactly would each of these be more applicable and suitable is lacking.

6) What is the cost of scaling this approach to more practical models/datasets (e.g. ImageNet with ResNet50 or so)? This goes back to -4- as to if/how this is better than AT, besides the empirical gains.

Meta-review: This paper presents Robustness Reprogramming, a novel method for enhancing the adversarial robustness of pre-trained models without modifying their core parameters. By introducing Nonlinear Robust Pattern Matching (NRPM) and three reprogramming paradigms, the authors propose an approach that is orthogonal to traditional adversarial training. Their method combines theoretical rigor with comprehensive experimental validation, demonstrating consistent robustness improvements across various models and datasets.

The reviewers unanimously recommend acceptance, recognizing the paper's novelty and solid empirical results. While some initial concerns were raised about the theoretical framing, notation clarity, and scalability, the authors effectively addressed these through detailed responses, revisions, and additional experiments on CIFAR-100, Tiny-ImageNet, and WideResNet28-10. These efforts strengthened the submission significantly for more real-world settings.

Overall, this paper is a strong submission, offering a new direction for robust machine learning and practical utility for adapting pre-trained models. I recommend acceptance as a spotlight, given its innovative approach and potential impact.

Decision: Accept (Spotlight)