Robust Feature Inference: A Test-time Defense Strategy using Spectral Projections
Authors: Anurag Singh
Mahalakshmi Sabanayagam
Krikamol Muandet
Debarghya Ghoshdastidar
Venue: Accepted by TMLR ()

Abstract: Test-time defenses are used to improve the robustness of deep neural networks to adversarial examples during inference. However, existing methods either require an additional trained classifier to detect and correct the adversarial samples, or perform additional complex optimization on the model parameters or the input to adapt to the adversarial samples at test-time, resulting in a significant increase in the inference time compared to the base model. In this work, we propose a novel test-time defense strategy called Robust Feature Inference (RFI) that is easy to integrate with any existing (robust) training procedure without additional test-time computation. Based on the notion of robustness of features that we present, the key idea is to project the trained models to the most robust feature space, thereby reducing the vulnerability to adversarial attacks in non-robust directions. We theoretically characterize the subspace of the eigenspectrum of the feature covariance that is the most robust for a generalized additive model. Our extensive experiments on CIFAR-10, CIFAR-100, tiny ImageNet and ImageNet datasets for several robustness benchmarks, including the state-of-the-art methods in RobustBench show that RFI improves robustness across adaptive and transfer attacks consistently. We also compare RFI with adaptive test-time defenses to demonstrate the effectiveness of our proposed approach.

Published in Transactions on Machine Learning Research (08/2024)
Robust Feature Inference: A Test-time Defense Strategy using Spectral Projections
Anurag Singh'
wwwngg.singhhhiisp.... de
CISPA Helmholtz Center for Information Security, Soarbrücken, Germany
Mahalakshmi Sabanayagam'
sabemaya@ciil.umm www.eiitttmmm...mm
School of Computation, Information and Technology
Technical University of Munich, Germany
Krikamol Muandet
muandet@@vispo.de
CISPA Helmholtz Center for Information Security, Saorbrücken, Germany
Debarghya Ghoshdastidar
ghoshdas@cit.c tum..
School of Computation, Information and Technology Technicol University of Munich, Germany
Reviewed on OpenReview: https: // openreview. met/forumt forum? id= 90HA #WFBB
Abstract
tist-time defenses are used to improve the robustness of deep neural networks to adversarial examples during inference. However, existing methods either require an additional trained classifier detect and correct the adversarial samples, or perform additional complex optimization the model parameters the input to adapt to the adversarial samples at test-time, resulting in significant increase in the inference time compared to the base model. In this work, we propose novel test-time defense strategy called Robust Feature Inference (RFI) that is easy to integrate with any existing (robust) training procedure without additional test-time computation. Based on the notion robustness of features that we present, the key idea is to project the trained models to the most robust feature space, thereby reducing the vulnerability adversarial attacks in non-robust directions. We theoretically characterize the subspace of the eigenspectrum of the feature covariance that is the most robust for generalized additive model. Our extensive experiments on CIFAR-10, CIFAR-100, liny ImageNet and ImageNet datasets for several robustness benchmarks, including the state-of-the- methods in Renuc show that RFI improves robustness across adaptive and transfer attacks consistently. We also compare RFI with adaptive test-time defenses to demonstrate the effectiveness of our proposed approach.
1 Introduction
Despite the phenomenal success of deep learning in several challenging tasks, they are prone to vulnerabilities such as the addition of carefully crafted small imperceptible perturbations to the input known as adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014). While adversarial examples are semantically similar the input data, they cause the networks make wrong predictions with high confidence. The primary focus of the community in building adversarially robust models is through modified training procedures. One of the most popular and promising approaches is adversarial training (Madry el al., 2018), which minimizes the maximum loss on the perturbed imput samples. Extensive empirical and theoretical studies on the robustness of deep neural networks (DNNs) using adversarial training reveals that adversarial examples are inevitable (Ilyas et al., 2019; Athalye et al., 2018b; Shafahi et al., 2019; Tsipras et al., 2018) and developing
"Equal contribution. This work was partly done when AS was master's student TU Munich.

---

Published in Transactions on Machine Learning Research (08/2024)
Training
Stage 1: Post-training computation of robust features
B
B
diasWoy
x(-)
Maa a(a)
(
T RRx
Trein input
3x
Compute Covariance
Stage 2: Robust Feature Inference
Denan == {(x): 86 Bil
Evaain
4 Udiag(A)U"
Projection onto robust feature subspace.
Compute classwise robustnces score for u, Vc E y as sc(u:) =- A, uz)
<(xx,)
0 0
Test Input æ;
Select lopk robust features Ve E y Up == (u s.f. s.()) is in top-K}.
((.)
A(x,) = (1d()))
Union U. Ve }}
3 E R
Figure 1: Illustration of our test-time defense mechanism. Given any trained model h(x), first post-process the penultimate layer features o(x) to get the top most informative and robust features in eigenspace Ü using the training data. During inference of the test data xt, <(x,) is projected onto the robust. feature space using p(x)) ÜÜT equivalently changing 3 to B = U.TB.
robust deployable models with safety guarantees require a huge amount of data and model complexity (Nie et al., 2022; Wang et al., 2023; Carmon et al., 2019). For instance, the current stale-of-the-arl methods (Wang et al., 2023; Peng et al., 2023; Gowal et al., 2021; Rebuffi et al., 2021) use additional one million to 100 million synthetic data along with the original 50000 training samples of CIFAR-10 and CIFAR-100. Although the improvement in robust performance convincing with this approach, there is an evident tradeoff with huge computational costs both in terms of data and model.
While the deep learning community has focused achieving robustness through different training paradigms such as adversarial training, little attention has been on improving the robustness of trained models at test-time. Test-time defenses refer to methods that improve the robustness of any trained model at test time. This is typically achieved through two main strategics: (i) Static test-time defenses update the model parameters or input stochastically independent of the Lest data (Cohen el al., 2019) or introduce fixed mechanisms to detect and correct the adversarial input (Guo et al., 2018; Nayak et al., 2022). Although the stochastic static defense based on randomized smoothing gives cortifiable defense, there is no thcoretically well founded deterministic static defense. Most such defenses are based heuristics. (ii) Dynamic/ Drnamic/Adaptive Adaptive test-time defenses adapt the input (Alfarra et al., 2022; Wu et al., 2021) or the model parameters (Kang et al., 2021; Chen et al., 2021) to the test data before making prediction. While the dynamic defenses seem promising as it can adapt to the adversary, the inference is computationally more demanding as adapts to every single input at test-time. Moreover, the existing adaptive defenses do not necessarily improve the robustness of the underlying model (Croce al., 2022). Thus, efficiently improving the adversarial robustness of the trained models lest-time without additional data or computation and with theoretical guarantees remain challenging problem.
Our contribution. In this work, we develop a novel test-time defense strategy with the same inference cost as the underlying model and no additional data or model complexity. We define robust features, inspired by Athalye et al. (2018b); Ilyas et al. (2019) in Definition 3.1, and subsequently describe the proposed method, RFI, in Algorithm that relics on the idea of retaining the most robust fcatures of the trained model. Notably, RFI is easy to integrate with any existing training paradigm. We provide a theoretical justification for our method by analyzing the robustness of features in generalized additive model (GAM)
2

---

Published in Transactions on Machine Learning Research (08/2024)
setting (Corollary 3.4). We conduct extensive experiments using different architectures such ResNet-18 18, ResNet-50, WideResNett-88-10 10, WideResNNe-34-10, WideResNNet50-22 and PreActResNet-188 on CIFAR-10, CIFAR-100, tiny ImageNet and ImageNet where RFI yields consistent robustness gains over base models and as well as other adaptive test-time defenses across datosets without additional cost at test time. Thus, we provide the first theoretically guided method with xx inference time as the base model, outperforming the adaptive test-time defenses of comparable computation overhead. An interesting by-product of our analysis is the learning dynamics of GAM showing that the features with large variation aligning with the original signal are more robust and learned early during training (Proposition 5.1). This phenomenon has been observed empirically for Neural Tangent Kernel (NTK) features (Tsilivis & Kempe, 2022) without theoretical proof. As a supplementary analysis, we prove it for NTK features (Proposition 5.2).
Illustration of our method (Figure 1). The proposed method abstracts any deep neural network as a feature extractor ((x) and a linear output layer 3 ((x) that consists of class prototypes. We compute the covariance of the features Eerain obtained from the training examples of the feature extractor. We define a robustness measure for the eigenvectors of Strain as sc(uk) and for each class protolype, we retain only the top most eigenvectors with respect to the robustness measure. This choice of the robustness metric well as the principle of sorting the cigenvector are mathematically justified through Corollary 3.4, where we consider generalized additive models and compute the robustness score of features (Defnition 3.1) showing that the top eigenvectors of the feature matrix are more robust. For a finite-width network, the above theoretical argument essentially corresponds to projecting the weights of only the last layer 3 onto the space spanned by the most robust features 3 = ÜÜ B, thereby improving the robustness of the underlying model at cest-time. Our method does not increase the inference time because the selected robust eigenbasis can be used to simply transform the lincar layer weights into the cigenbasis resulting in exactly the same number of parameters in the network at inference time.
2 Related Works
In recent years, there has been a significant amount. of research on generating adversarial examples and simultancously improving the robustness of DNNs against such examples. We review the most relevant works below along with static and adaptive test-time defenses.
Adversarial robustness. Szegedy et al. (2013) first observed that the adversarial examples, which are small imperceptible perturbations to the original data, can fool the DNN casily and make incorrect predictions. To generate adversarial examples, Fast Gradient Sign Method (FGSM) is proposed by Goodfellow et al. (2014). Madry et al. (2018) introduced an effective defense against adversarial examples known as adversarial training, where the network is trained by minimizing the maximum loss on the adversarially perturbed inputs. Adversarial training remains a promising defense to significantly improve the robustness of DNNs against adversarial attacks (Rice et al., 2020; Carmon el. al., 2019; Engstrom et al., 2019; Wang el al., 2023; Pang et al., 2022). However, sophisticated attacks are developed to break the defenses such as Carlini- Carlini-Wagner Wagner (C&W) attack Carlini & Wagner (2017), method for generating adversarial examples; 'obfuscated gradicnts' hypothesis (Athalye et al., 2018b) posits that the vulnerability to adversarial examples is due to the presence of easy to manipulate gradients in the model; 'feature collision' hypothesis Πlyas et al. (2019) postulates that the vulnerability is due to the presence of features in the data that are correlated with the labels, but loses the correlation when perturbed. As an advanced counter defense, methods to constrain the Lipschitzness of the model (Wang et al., 2019) are developed.
Static test-time defenses. Static defenses change the model parameters or inputs after training without the knowledge of the test data and remains fixed during inference. A theoretically guaranteed approach to update the model parameters is through randomized smoothing (Cohen et al., 2019; Liu et al., 2018). Another approach is to first detect the adversarial input and correct it using a trained classifier, and input the corrected sample to the base model for prediction. Guo et al. (2018) suggest model agnostic image transformation such as total variance minimization and image quilting for the test data as an effective defense against any adversary. Other works detect the adversarial inputs using a separate trained network and corrects it either by removing the high frequency component in Fourier domain (Nayak al., 2022) or by a trained masked autoencoder (Chao et al., 2023).
3

---

Published in Transactions on Machine Learning Research (08/2024)
Adaptive test-time defenses. Adaptive defenses update model parameters and inputs at inference to defend against the attack. One strategy of adaptive test-time defenses is inpul purification, in which the inputs to model (usually pre-trained with a robustness objective) is optimized with test-time objective. This test-time optimization can be hand crafted Alfarra et al. (2022); Wu et al. (2021) or learned Mao et al. (2021); Hwang et al. (2023) with the help of an auxiliary network Nie et al. (2022). Another stralegy for building adaptive test-time defenses is model adaptation, where model parameters are augmented with activations Chen al. (2021), implicit representations Kang et al. (2021); Qian et al. (2021) and additional normalization layers Wang et al. (2021). Although several methods are developed for adaptive test-time defenses, all of them increase the inference cost at least 2x (Kang et al., 2021) and sometimes 500x (Shi et al., 2021) compared to the underlying model. More importantly, most of the existing adaptive test-time defenses results in a weaker adversary than the base model, hence overestimated the robustness to adaptive attacks and are not really competitive with the static defenses as categorically shown in Croce et al. (2022).
3 Robust Feature Inference: A Test-time Defense Strategy using Spectral Projections
We consider multi-class classification problem, where we aim to learn a predictor h X + 3' where X' S IR and y c {0, 1) is the set of one-hot encodings of C classes. We assume that the data is independent and identically distributed (i.id) according to an unknown joint distribution D over instance-labels (x. y) E X x V. The goal of the paper is to develop a test-time defense that can be integrated with any training procedure. Hence, we assume that there exists learned predictor h X' + y that we aim to make robust against adversarial attacks. We aim to achieve this by decomposing the predictor into two components, h(x) = Probust (x) + Anonrabust (x) such that Arobus X' + Y/ corresponds to the robust component of the predictor while Anonrobust X + y represents the remaining (non robust) component of h. In this section, we formally characterize this problem by proposing a notion of robustness of features, inspired by llyas et al. (2019), and an algorithm based on pruning less robust features. We also show that the more robust features are more informative.
3.1 Robust and Non-Robust Features
The additive decomposition of a predictor in the form h = hrobust + Anonrobust is difficult in general for predictors with non-lincarities in the output, for instance, softmax in multi-class classifiers. Hence, we relax the setup to a multivariate regression problem, that is, y RC, We further assume that the trained model h : X + y is given by a generalized additive model (GAM) of the form h(x) = B Φ(x), where Φ : - H is a smooth function that maps the data into a feature space H and / are weights learned in the feature space. The above form of h may represent the solution of kernel regression (with H. being the corresponding reproducing kernel Hilbert space) or h could be the output layer of neural network, where = R², Φ(x) denotes the representation learned in the last hidden layer and B BBxC are learned weights of the output layer.
Features and their robustness. To identify the robust component of h, we aim to approximate o as sum of K robust components (o that is, h(x) s Σk S (x). We refer to each o; : X + H as a feature X + H). We now define the robustness of a More generally, we define the set of all features as = { x feature as follows.
Definition 3.1 (4g-Robustness of features). Given a distribution D on X x RC and a trained model h(x)= = (=Tx((x)) we define (f) = B(xx))
inf [|| x
v BT f(x) as the robustness of feature f F as the rosust of f with respect to the e-th class component.
and (D) = Bix,y\D inf
|||8~x|ass م25 yo [xx)
of y E RC, CE (1,.... ...,..,,, C}, where Be is c-th column of B.
The above definition is based on the notion of robust features introduced by lyas et al. (2019) as robobustly useful features, specialized to GAM model. While the robouusy useful feature in llyas et al. (2019) is defined on the network output, we define it for the penultimate feature f with a new class-specific definition sf)),c(f). Based on Definition 3.1, the goal is to approximate h using the most robust features. Searching over all f E F is difficult, hence, we focus on features that are linear maps of &, that is, f(x) M T ((x) for
4

---

Published in Transactions on Machine Learning Research (08/2024)
some M H. + H. (or M RP For such features, we bound the robustness score from below, under independent noise model.
Theorem 3.2 (Lower bound on robustness). Given h(x) = Br oxx). Assume that the distribution D is such that y = hxx) + c, where c E IC has independent coordinates, each satisfying B[c,] = 0, E[(2] S o² for all
C (l....,Cy) C}. Farther, assume that the map & is L-Lipschilz, that is, ll((x) - x - L|| - x||- Then, for any f = MΦ and every c E (I,...,,)),
(D. 2 B.MB. 1pp B.E.Be
where == Ex [o(x))(x)]
] and || op is operator norm.
Remark 3.3 (Lower bound is light up to constants). For linear models d(x) and Ex [x] 0, $D,8,o(f) is equal to the lower bound with L = 2 (proved in Appendix A.2).
Theorem 3.2 (proved in Appendix A.1) suggests that if we search only over f E J that are linear transforma- tions f M T Φ such that |M = 1, then the most robust feature is the one that maximizes the first term B. 3. If the search is further restricted to projections onto K dimensional subspace, M PPT with P being the orthonormal basis, then show that optimizing over such features corresponds to projecting onto the top K eigenvectors of Σ sorted according to specific robustness score.
Corollary 3.4. Fir any K and Σ Ex [x(x)x(x)] Consider the problem of mazimizing the lower bound in Theorem 3.2 over all fealures I E F that correspond to projection of á onto K dimensional subspuce. Then the solution is f= =UU.0e & where U. is the matria of the K top eigenvectors of closs- class-specific matrix B. = (3.03 I3
The above result, proved in Appendix A.3, leads to the principle idea of our test-time defense algorithm. The robust output can be defined as h(x) = [8] Ü, ÜJ o(x). U [[(()) where U, is computed from B. for every class c E fl....,
.....,,), C}. This results in the most robust projections theoretically, but suffers
computationally since requires (C (C++) 1) cigendccompositions.
Efficient version. To improve the computation time, we restrict the search space of M ÜÙT to the eigenvectors of 2, then Ú is the matrix of union of K eigenvectors for which the robustness score su:) = Ax are the largest for every class c. This method retains and leverages only the robust features of the trained model at Lest-time elficiently by projecting the output of the trained model to the eigenspace with higher robustness score, One may naturally ask how much error is incurred by retaining only the robust features. Later, in Corollary 3.6, we discuss that the most robust features also contain most of the information, and hence, drop in performance due the projection is low.
3.2 Our Algorithm: Robust Feature Inference (RFI)
Let Durain == ((x,, (1)) CX y be a training dataset with n samples, and h: X -> y a trained model such that 1((x) = T d(x) for all x E X where Φ X -> RP the feature map defined by the hidden layers RPxC is the weight matrix defined by the last fully-connected layer with p as the
of the model h and dimension of the feature space (refer Figure 1). From the previous analysis, we propose method operating on the feature space of & that projects the features in the robust directions, hence improving robustness by reducing the chance of attacks using the non-robust feature directions. To this end, we first compute the corresponding covariance matrix Etrain of the hidden-layer features based on the input data from Dirain: that
is,
E Etrain = -- ΦΦ with + = (o(x1), ((,,,,))(()))) o(xx)] E R
Next, as presented in Figure 1, we compute the cigendecomposition of the covariance Etrain Udiag(A) T where U e Rexx is the matrix whose columns consist of eigenvectors of Eurain denoted by u; E IR, X E Rp is a vector of corresponding eigenvalues such that a 2 A2 2 ..., and diag(A) is a diagonal matrix with eigenvalues as its diagonal entrics. The idca of our algorithm to retain only robustly useful features, i.c,, top K cigenvectors, when making predictions on unseen data. For each class e E y, we define the c-th column of B as Be class prototype for E y. The classwise robustness score of each feature is computed according
5

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm Robust Feature Inference (RFI)
Require:
The model h trained on Duran == {(x,, Bi) such that h(x) = B xx(x where Ø : - R² and E RPxC and the number of top robust features to select K.
1: Compute the covariance Etrain 6 # where + : (=((()),.... (xx)) E RRxx.
Compute eigendecomposition of Strain Udiag(A)U where columns of U are i R., {D Top K most
robust features U Roxk
(3 - 7))
Ü (): Be < e-th column of
4: for <- 1 to C do
5: For all i, compute robustness score se( + (B uj)²
6: U f- Union(U, (U, u;) if sc();) is in top K scores sc(.)
7: end for
(D. Robust Feature Inference on test set Xtest
(8-9))
Vx, E Xtest, h(x:) = BT (xx())
to Definition 3.1, that is, so(u)= == (a u.)² where (A;, u,) is the i-th pair of eigenvalue and eigenvector. We then select the sop-K most robust features for each class e E y based on the robustness score denoted by U. == {u)()) se (e() 2 se(ur()), Vi,j E , K]}. The global robust features for the model Ú is obtained as a union of the sets of classwise robust features U.. Finally, the prediction on a test data X; is subsequently obtained as
h(xx) (Tx())) B:=ū0T B, Ü: U .cy
where U denotes union of sets. Therefore, the new prediction is based on the updated parameters 3 instead of the original B. It is not difficult to see that this corresponds to applying the original parameters B on the robustly useful features, i.c., h(xx)= = 3 ((x)) g ŌÜ ((x)) B ((x)) where ((x,) UÜ Φ(xx). Figure 1 and Algorithm 1 summarize the proposed Lest-time defense.
3.3 Robustness vs information of features
We show that the robust features are also the informative features by defining a notion of informative features, inspired by usefulness property in llyas et al. (2019).
Definition 3.5 (Informative features). Given distribution D on X' RCC and trained model h(x) dxx), we define the information in a feature f with respect to e-th class component of y E RCC (D)
E(x.))-D [ fxx)], where e E {1...., C} and Be is e-th column of B.
While the informative feature is similar to the p-useful feature defined in llyas et al. (2019), it is important to note that define it class-specific for the penultimate feature f. Additionally, our definition also includes useful, non-robust features defined in llyas et al. (2019).
Corollary 3.6. Let (Ai, u;);=1,2.... denote the eigenpairs I = Ex [ox))x(x)T] For any feature J E F of the = ÚÜT ds where Ü = [ ux] is malrix of any K orthonormal eigenveclors of E, then information in feature f with respect to e-th component is given by
K
K
PD.a.(()) = Σ Aa u.)² > sc(u.)
Hence, the set of features selected in Algorithm 1 by sorting the robustness score so(u:) also correspond the eigenvectors with the most information. However, note that to maximize ( the full eigenspace has to be chosen, that is K p. We also provide visualizations of the defined features in B.13 of Appendix.

---

Published in Transactions on Machine Learning Research (08/2024)
4 Experimental Results
We present the following experimental analysis of RFI in this section: (1) evaluation of RFI against adaptive attacks resulting in consistent improvement in the robust performance in Section 4.1; (2) transfer attack evaluation of RFT showing the strength of RFT as well as establishing that RFI does not result in gradient obfuscation in Section 4.2; (3) in Section 4.3 we adapt static RFI a dynamic adaptive test-time defense and show that static RFI is better than dynamic RFI. Consequently,
compare static RFI to other test-time defenses in Section 4.4 showing RFI outperforms other dynamic test-time defenses;
(4) we discuss the abalations on RFT in Section 4.5. Additionally, we present the performance of RFI on calibrated models using temperature scaling (Guo et al., 2017) in the appendix since it has been shown to improve robustness (Qin et al., 2021; Grabinski et al., 2022; Stutz et al., 2020; Tao et al., 2023).
Datasets & Resources. We evaluate RFI on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), Robust CIFAR-10 (Ilyas et al., 2019), tiny ImageNet (Le & Yang, 2015) and ImageNet (Russakovsky et al., 2015) datasets. We use Pytorch Paszke et al. (2019) for all our experiments & a single Nvidia DGX A100 to run all of our experiments. We also open-source
implementation at https://github.. com/ http://ggttttoooooo///mrggg///// Anurag14/FII.
Adversarial Attacks. We evaluate RFI on different white and black-box adversarial attacks namely, Projected Gradient. Descent (PGD) Madry et al. (2018), a white-box attack that perturbs the input within a small lp radius e, so that it maximizes the loss of a model. We perform both foo and lp PGD attack with standard perturbation c, attack step size and iteration for cach datasot. AutoAttack (Croce et al., 2020), suite of while-box and black-box attacks including Auto PGD-Cross Entropy (APGD-CE), Auto PGD-Difference Logit Ratios (APGD-DLR) (Croce & Hein, 2020b), Fast Adaptive Boundary Attack (FAB) (Croce & Hein, 2020a), and Square Attacks (Andriushchenko et al., 2020). APGD-CE and APGD-DLR are parameter-free white-box attacks that are extensions of PGD attack with no step size parameter and stronger than PGD. FAB is a white-box attack that minimizes the norm of the adversarial perturbation. Square Attack is an efficient black-box attack that is score based and uses random search without gradient approximations. While adaplive allacks generate the adversarial images using the target model, transfer attacks generate adversarial images using surrogate model and attack the target model.
Benchmarking on SoTA defenses. We evaluate RFI on various architectures trained diffcrently: ResNet- 18 and ResNel-50 with standard training and the popular adversarial training methods such as PGD Madry et al. (2018), Interpolated Adversarial Training (LAT) Lamb et al. (2019), Carlini- Wogner (C&W) loss Carlini & Wagner (2017) and TRADES (Zhang et al., 2019). We select different state-of-the-art adversarially trained models from RobustBench upon which at the lest time we integrate RFI (Carmon et al., 2019; Engstrom et al., 2019; Rice et al., 2020; Wang et al., 2023; Pang et al., 2022). These methods either use additional data (Carmon et al., 2019; Wang et al., 2023), informed adversarial prior (Engstrom et al., 2019) or early stopping (Rice et al., 2020) to improve the robustness of models. We detail each training method in appendix.
Evaluation measures. We measure the performance of models with and without RFT by the accuracy of predictions to both clean/origin samples and adversarial samples averaged over 5 runs. We use 'Clean' to denote accuracy of models to original samples. Note that there are no standard deviation in our evaluation of models from Robust Bench as we are directly loading the models without training, hence no stochasticity. Details of the model evaluation are Appendix B.2. We further remark that our defense strategy does not circumvent gradient. based allacks due to gradient masking (Athalye et al., 2018b) since we simply project the last layer feature in its covariance eigenspace, hence the network remains differentiable with active gradients.
Comparison to adaptive test-time defenses. We compare RFI with two adaptive test-time defenses: SODEF (Kang et al., 2021) and Anti-adv (Alfarra et al., 2022). The choice of SODEF and Anti-ady is due to their relatively faster inference costs 2x and 8x, respectively, and are representative of model adaptation and input modification strategies for adaptive test-time defenses, respectively.
4.1 RFI improves adversarial robustness consistently
We evaluate RFI for adaptive attacks by generating adversarial samples to specifically target our de- fense (Tramer et al., 2020). We obtain clean and robust accuracy for standard and adversarially trained models before and after integrating RFI and setting K to the number of classes. The results for different.
7

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: Adaptive attack performance of RFI. We consider loo and & PGD attack on CIFAR-10 with Resnet-18 and b attack with step size c/4 and 40 iterations. lz attack with size c/5 and 100 iterations. RFI improves the performance on an average by 2%.
Clean
fa(t = 0.5)
Training
255
Method
RRFI
% Cain
Method
|RFI
% Cain
Method
IRFI
% Gain
Standard
95.28 0.04
88.53 oos
-6.75
1.02+ 0.12
4.35 o.0a
+3.33
0.39 s.s
9.73 0.39
49.34
Robust CIFAR-10
78.69 oos
78.75 wa2
+0.06
1.30, aos
7.01 0.30
+5.71
9.63 sus
11.00. 0.14
+1.37
PGD
83.53; co..
83.22; 0.e
-0.31
42.20 ose
43.29 o.00
+1.09
54.61 0.00
55.03: 0.00
+0.42
IAT
91.86g co..
91.26g n.w
-0.60
44.76z osa
46.95 0.00
+2.19
62.53 oax
64.31 : o.a
+1.78
C&W
85.16 0.12
84.91 0.00
-0.25
40.12 oxe
42.33 o.s
+2.21
55.18 0.2
56.68 0.20
+1.50
TRADES
81.22 0.
80.68 nas
-0.54
51.93 n.s
53.50 0.7
+1.57
59.87 mas
61.27 na
+1.40
training procedures on CIFAR-10 with ResNet-15 18 are presented in Table 1 (CIFAR-100 with ResNet-18 and tiny ImageNet with ResNet-50 Tables 11 and 12, respectively, in Appendix). Robust CIFAR-10 denotes standard training using Robust CIFAR-10 dataset. We observe that our method consistently improves the robust performance of adversarially trained models, performance as we choose only a subset of the informative Teatures that are also robust. (K o p), hence a loss in information to achieve the best possible clean performance as derived in Corollary 3.6. Nevertheless, the gain in robust performance is with almost no computational overhead The seemingly small improvement in performance is mainly due to the fact that we are adapting the trained model without any further learning, as well as the adaptive attacks on RFT results stronger adversary than the base model as we discuss transfer attack evaluation subsequently. Additional experiments showing the effectiveness of RFI on Expectation Over Transformation attack (Athalye el al., 2018b) is presented in Table 9 of Appendix. Furthermore, RFI improves the robustness of calibrated models by 4% - 8% as shown in Tables 10, 11 and 12 of Appendix.
average by 2%. There is minor drop in the clean
4.2 Transfer Attack Evaluation: RFI is stronger than base model
Many defences show remarkable robustness to adaptive attacks by obfuscating gradients, thereby circumventing gradient-based attacks and offering a false sense of sccurity (Athalye et al., 2018a; Huang et al., 2021). Therefore to validate the true effectiveness of a defense, evaluating transfer atlack is crucial. Hence, we expand our evaluation from Table to transfer attacks, where we assess the performance with and without RFI against adversarial samples generated from the base and base model+RFI. The results in Table 2 shows that RFI is more robust to attacks from base model whereas the base model loses considerable robustness when attacked with the adversory from RFI demonstrating that RFI is stronger adversary than the base model. It is interesting to note that the robustness of C&W trained model is completely lost when tested against adversarial examples from H&W+RFF model. This clearly establishes that the RFI is nol resulling in gradient obfuscation as C&W is not a gradient based attack. Contrastingly, TRADES results in a more robust model that withstands attack from TRADES+RET While the performance of TRADES almost the same for adversarial atlacks generated from TRADES and TRADES+REI, RFI results in more robust. models in both cases. We present the results calibrated models in Tables 15 and 16 in Appendix.
Table 2: Transfer attack on ResNet-18 for CIFAR-10. Setting same as Table 1. RF1 results in much stronger adversary than the base method.
Adversarial Examples are generated from base model
Adversariel Examples are generated from base model RFI
Training
2
E
0.5)
Training
fx
Egle 0.5)
Method
+RFT
Method
+RFT
Method
+RF1
Method
RRF
Standard
1.02
10.36
0.39
12.09
Standard
0.00
4.35
0.01
9.39
Robust CIFAR-10
1.30
15.41
9.63
17.38
Robust CIFAR-10
0.03
7.01
1.05
11.00
PCD
42.20
46.02
54.61
58.81
PGD
31.80
43.29
49.90
55.03
IAT
44.76
49.06
62.53
66.67
IAT
35.78
46.95
55.42
64.31
C&W
40.01
45.48
55.02
58.95
C&W
3.92
42.56
13.50
56.79
TRADES
51.98
54.33
60.03
65.23
TRADES
51,70
53.45
58.09
61.39
8

---

Published in Transactions on Machine Learning Research (08/2024)
4.3 Static RFI is better than Dynamic/Adaptive RFI
The principle of RFI can be effectively used to adapt the model at test-time every imput by computing the transformation matrix U using the robust feature score s(u) of cigenvectors of test set feature covariance Stest- The results for this adaptive strategy is in Table 3 evaluated for the robust training settings of Table 1. We consider transfer attack using the base model for fair comparison and observe that static RFI is better than dynomic RFI. This reinforces the theoretical result that the eigendirections of the training set feature covariance determines the most robust fcatures (Corollary 3.4). Morcover, adaptive attacks in dynamic RFI needs further information on when to adapt since the model should be static until the allacker creates an adversarial sample, and the adaptive transformation using U should be done only in the case of defender. Details of the challenges in deploying dynamic RFI when the use case is unknown, and the results for adaptive attacks are in Table 17 in Appendix B.10.
Table 3: Comparison of static and dynamic/ /adattive RFI. Setting same Table Adversarial examples are generated from the base model for fair comparison.
Clean
255 )
E 0.5)
Training
Static
Dynamic
Static
Dynamic
Static
Dynamic
PGD
83.22
82.86
46.02
46.83
58.81
59.23
IAT
91.26
91.35
49.06
48.53
66.67
66.28
C&W
84.97
83.01
45.48
43.98
58.95
57.82
TRADES
80.76
78.98
54.33
53.58
65.23
65.00
4.4 Static RFI outperforms adaptive test-time defenses
As result of the static vs dynamic RFT evaluation Section 4.3, we compare the effectiveness of static RFI on both white-box and black-box attacks with other adaptive test-time defenses such as SODEF and Anti-adv. In Table 4, we present the result for APGD-CE, APGD-DLR, FAB, Square and AutoAttack for different SOTA methods. We observe that adding slatic RFT improves the performance across all the methods. Importantly, RFI despite being non-adaptive improves the robust performance (at least marginally) for all the SOTA methods, even though each one of them achieves robustness by incorporating very different strategies like additional data, early stopping or informed prior. While this shows the strength and effectiveness of RFI, it also raises a fundamental question of whether in order to improve the robustness in the adaptive test-time defense strategy. Evaluation of other SOTA models for CIFAR-10, CIFAR-100 and ImageNet showing similar observations in Table 13 in Appendix.
necessary to adapt the model to individual test samples
Table 4: Robust performance evaluation of RFI on state-of-the-art methods. We evaluate APGD- CE, APGD-DLR, FAB, Square and AutoAttack under (e = 8/255) on CIFAR-10 and CIFAR-100. RFI consistently improves the performance of the base model, whereas Anti-ady and SODEF results in slight decrease in the performance to AutoAttack. The inference time of RFI is I 1x, whereas Anti-ady (Alfarra et al., 2022) and SODEF (Kang et al., 2021) are 8x and 2x, respectively. Additional results are in Table 13 in Appendix. There is no standard deviation as the trained models are from RenustBench.
Base Method
Defense
Clean
APGD-CE
APGD-DLR
FAB
Square
AutoAttack
CIFAR-10
None
89.69
61.82
60.85
60.18
66.51
59.53
Carmon et al. (2019)
Anti-adv
89.69
61.81
60.89
60.11
66.58
58.70
WideResNNet2255
SODEF
89.68
60.20
60.72
58.04
65.28
57.23
RFI (K- 10)
89.60
62.38
61.58
60.21
66.59
60.72
RFI (opt. 20)
89.60
62.45
61.60
60.38
66.90
61.02
CIPAR-100
None
63.66
35.29
31.71
31.32
35.70
31.08
Pang et ul. (2022)
Anti-ady
63.41
32.50
30.32
31.30
35.76
30.10
WideResNNe-228-00
SODEF
63.08
30.96
29.54
31.44
32.27
30.56
RFI (K 100)
63.01
36.03
31.95
31.88
35.79
31.29
RFI (opt. K 115)
63.10
36.07
31.95
31.96
35.88
31.91

---

Published in Transactions on Machine Learning Research (08/2024)
We compare the algorithmic time complexity for different test-time defenses in Tables 4 and 13 and provide the average time to infer a single sample on Nvidia DGX-A100 in the Table 5. Note that the average time closely follows the time complexity. RFI does not add additional computation overhead. However, Anti-adv and SODEF lead to 8x and 2x computation compared to the base model.
Table 5: Time comparison for RFI, Anti-ady, SODEF. RFI: 1x, Anti-adv: 8x, SODEF: 2x.
Time Comparison in (ms)
Model
Base
RFI
Anti-adv
SODEF
PreActResnet-18
0.2760
0.2777
1.5127
0.5133
ResNet-50
0.3692
0.3703
2.7684
0.6877
WideResnet-22 10
0.3780
0.3763
2.9735
0.7338
WideResnet-344
0.8619
0.8654
6.8380
1.6599
4.5
Abalation Studies
4.5.1 Effect of adversary strength
We study the effect of adversary strength on our method, RFI, by taking the adversarially trained ResNet-18 CIFAR-10 using PGD (c 8/255 for lo and 0.5 for 82) the base model. Table 6 shows the evaluation of RFI with foo PGD attack for attack for c= {0.25, 0.75, }} and 100 iterations. The results show that the underlying model augmented with RFI consistently improves over baseline across the perturbations of various strengths, especially by over 1% for adversary that is stronger than the base model (e = (12/255, 16 16/255 for loo and e = 75, 1.00} for lg.
(2/255, 4/255, 12 255, 16/255) and 40 iterations, and &
Table 6: RFI consistently improves over baseline across the perturbations of various strengths. Evaluation of RFI for foe and & on ResNet-18 adversarially trained with CIFAR-10 and PGD.
to attack
8 attack
Method
255
255
c
255
12
255
16
c 0.25
6 0.75
= 1.00
PGD
74.60
64.02
23.34
11.66
71.34
40.91
28.25
RGD-RFI
74.99
64.91
24.32
12.55
71.48
41.95
29.24
Further empirical analysis of the effect of step size in PGD attack is provided in Table 19 in Appendix.
4.5.2 Choice of K
To study the effect of parameter K in detail, we vary K for the ad adversarial training methods on CIFAR-10 with ResNet-18 under (== = 8/255) threat model, same setting as in Table 1 setting, Figure 2 (left plot) shows that the adversarial training methods (PGD, IAT, C&W and TRADES) behave similarly in their accuracy profile as compared to standard training even on Robust CIFAR-10 dataset. Moreover, the best performance is for K = 10 for all the robust training methods. The corresponding eigenvalue spectrum exhibits knee drop after top-10 eigenvalues (right plot), which motivates our choice of K as top-10 features for each class, equivalent to the number of classes. As a complementary explanation for our choice of K, neural collapse phenomenon observes that the penultimate feature of each class collapses to its mean after the training error is almost zero (Papyan et al., 2020). This implics that there is principally only C' number of feature vectors, one for each class, justifying our choice. Further ablation sludes on the ellect of parameter K for SoTA models are in Figures 4 and in Appendix. While we set K to be the number of classes, we also report the best performance of RFI by finding the optimal K using grid scarch for SoTA models in Table in Appendix. Although K number of classes is not the optimum for the SoTA models, is still better than SODEF and Anti-adv.
10

---

Published in Transactions on Machine Learning Research (08/2024)
Accuracy t e ( 8/255)
10
Standard
40-
PGD
IAT
Robust CIFAR 10
10'
C&W
10-
TRADES
107
top-K
Index of Eigenvalue
Figure 2: Effect of K in RFI. Robust accuracy and eigenvalue profile in ascending order of all the methods in Table 10.
4.5.3 Comparison of RFI to similar conceptual methods
The conceptual counterparts to RFI include performing the projection of intermediary layers to a low dimensional space instead of the last layer, enforcing low dimensional last layer directly. In Table 7, we evaluate effectiveness of performing RFI on intermediate layers by truncating the last but one hidden layer of ResNet-18 and evaluate the PGD trained model considered in Table 1. This hidden layer has 512 x 4 x 4 convolution which we project to 10 x 4 x 4 using RFT procedure. While it is clear that performing RFI on the last layer as derived theoretically improves the robust performance, RFI on intermediary layers harm the robustness. We provide the result for enforcing low dimension last layer in appendix (Table 20) which also demonstrates the superiority of RFI.
Table 7: RFI on last layer outperforms intermediate layer. Evaluation of PGD trained ResNet-18 on CIFAR-10.
None
RFI on last layer RFI on last but one layer
42.20
43.29
36.06
5 Discussion
The simplicity and effectiveness of RFI at test-time is impressive as the robustness gain is achieved with zero additional computation overhead for inference. While RFI on smaller models like ResNet demonstrate more improvement in robustness than larger SoTA models like WideResNet, it is important to note that these SoTA models are alroady optimized to their full potential, hence even small improvement is significant in these cases. Furthermore, RFI is well-founded theoretically. Consequently, the idea of RFI can also be used to develop a robust training procedure by incorporating the projection onto the robust feat ure space during training. We leave the experimental analysis for future study as the current work focuses on test-time defenses. However, it intriguing thcorctically analyze the robustness of features during training understand the RFT's potential an idea and the cause of vulnerability adversarial examples. Therefore, derive the learning dynamics of full batch gradient descent population squared error loss of GAM (stated informally in Proposition 5.1 and proved in Appendix A.5).
Proposition 5.1 (Learning dynamics of GAM). Given h(z) = I (T(()) and Σ Ex [ Let
(Ae, u.) be the eigenpair of Σ. Then full batch grudient descent learns fealures in the direction of u; with large eigenvalues X₂ first during the training and those directions are robust only if they align with the original signal direction B.
This result further strengthens the idea and suggests that truncating the non-robust directions during training, which is one of the plausible causes for the existence of adversaries, could improve the robustness of the model.
Connection to Neural Tangent Kernel (NTK) features. One of the related results Proposition 5.1 is using the NTK features. Tsilivis & Kempe (2022) defined features using NTK gram matrix and empirically
11

---

Published in Transactions on Machine Learning Research (08/2024)
observed that the features corresponding to the top spectrum of NTK are more robust and learned first during training. Our theoretical framework enables us to establish the equivalence of NTK features to the robust feature definition and more importantly prove that the robust NTK features indeed correspond to the top of the spectrum. The NTK gram matrix @ E R R is between all pairs of datapoints. NTK features of input x defined using the eigendecompoostionn function g. We state the result the following proposition and prove along with empirical verification Appendix A.6 (Figure 3).
θ = Ei= Av I as (x)) == g(A,, 14, æ:) for a specific
Proposition 5.2 (NTK feature robustness lies at the top). Let feature fher be Lipschitz continuous in grudient of NTK with respect to and an adversarial perturbalion ö such that ||01|p S A. Then, [| + 8) - (x)ll I (
Although we prove that the robust NTK features correspond to the top of the spectrum, we leave the challenge to establish its connection to the DNN for future analysis. Overall, our work develops a guaranteod algorithm to improve adversarial robustness at lest-Lime along with possibilities to improve the robust training procedures. Additionally, we note that while the current work focused on evaluating SOTA convolution-bass models, the effectiveness of RFI on bransformer-based models an interesting future direction.
6 Conclusion
In this paper, present novel test-time defense that can be scamlessly integrated with any method at the time of deployment to improve the robust of the underlying model. While the adaptive test-time defense as an approach offers promise to improve the robustness of models at the deployment stage, the general criticism of available methods is that they significantly increase the inference time of the underlying model. Our method, Robust Feature Inference (RFI), has no effect on the inference time of the underlying model which makes it a practical alternative for adaptive test-time defense. We also present a comprehensive theoretical justification for our approach describing the motivation behind retaining features in the top eigenspectrum of the feature covariance. In addition, we show that these top features are more robust and informative, and validate our algorithm through extensive experiments. In conclusion, we propose the first theoretically guided adaptive test-time defense algorithm that has the same inference time as the base model with significant experimental results. Our findings contribute to the ongoing efforts to develop robust models that can resist adversarial examples and improve the security and reliability of DNNs.
References
Sravanti Addepalli, Samyak Jain, et al. Efficient and effective augmentation strategy for adversarial training. Advances in Newral Information Processing Systems, 15:1488-1501, 2022.
Motasem Alfarra, Juan C Pérez, Ali Thabet, Adel Bibi, Philip HS Torr, and Bernard Ghanem. Combating adversaries with anti-adversaries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 5992-6000, 2022.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, pp. 484-501. Springer, 2020.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyua Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advonces in newral information processing systems, 32, 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradicnts give a false sense of sccurity: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. PMLR, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. Tn International conference on machine learning, pp. 284-293. PMLR, 2018b
12

---

Published in Transactions on Machine Learning Research (08/2024)
Yoshua Bengio. Practical recommendations for pradient-based training of deep architectures. In Newral Networks: Tricks of the Trade: Second Edition, pp. 437-478. Springer, 2012.
Alon Brutzkus and Amir Globerson. Why do larger models generalize better? a theoretical perspective via the xor problem. International Conference on Machine Learning, pp. 822 830. PMLR, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 feee sympostum security and privacy (sp), pp. 39-57. Icee, 2017.
Yair Carmon, Aditi Raghumathan, Ludwig Schmidt,, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances in neural information processing systems, 32, 2019.
Yun-Yin Tsail Ju-Chin Chao, Albert Wen, Zhaoyuan Yang, Chengzhi Mao, Tapan Shah, and Junfeng Yang. Test-time detection and repair of adversarial samples via masked autoencoder. 2023.
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Towards robust neural networks via close-loop control. arXiv preprint 0rXi::2102.01862 2021.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pp. 1320- PMTR, 2019.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference Machine Learning, pp. 2196 2205. PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020b.
Francesco Croce, Maksym Andriushchenko, Vikash Schwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint or Xiv:2011 09670, 2020.
Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. In International Conference on Machine Learning, pp. 1121-445.. PMLR, 2022.
Logan Engstrom, Andrew Dyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations. arXiv preprint arXi::1966.000 000000000045... 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arxiv preprinl ar Xrx: 03635, 2018.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint XrXio:1422.657 2014.
Sven Gowal, Pivrestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy Mann. Improving robustness using generated data. In Advances in Neural Information Processing Systems, 2021.
Julia Grabinski, Paul Gavrikov, Janis Keuper, and Margret Keuper. Robust models are less over-confident. Advances in Neural Information Processing Systems, 35:39059 39075, 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp- 1321-1330 PMLR, 2017.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018.
Yifei Huang, Yaodong Yu, Hongyang Zhang, Yi Ma, and Yuan Yao. Adversarial robustness of stabilized neural ode might be from obfuscated gradients. Proceedings of Machine Learning Research vol, 145:1-19, 2021.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Duhun Hwang, Eunjung Lee, and Wonjong Rhee. Aid-purifter: A light auxiliary network for boosting adversarial defense. Neurocomputing pp. 126251, 2023.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in newral information processing systems, 32, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Qiyu Kang, Yang Song, Qinxu Ding, and Wee Peng Tay. Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks. Aduances in Newral Information Processing Systems, 34: 14925- 14937, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Lamb, Vikas Verma, Juho Kannala, and Yoshua Bengio. Interpolated adversarial training: Achieving robust neural networks without sacrificing too much accuracy. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Securily, pp. 95 103, 2019.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Chengzhi Mao, Mia Chiquier, Hao Wang, Junfeng Yang, and Carl Vondrick. Adversarial attacks are reversible with natural supervision. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 661-671, 2021.
Gaurav Kumar Nayak, Ruchit Rawal, and Anirban Chakraborty. Dad: Data-free adversarial defense at test time. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3562 3571, 2022.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning, pp. 16805 -6827. PMLR, 2022.
Tianyu Pang, Min Lin, Xiao Yang, Jun Zhu, and Shuicheng Yan. Robustness and accuracy could be reconcilable by (proper) definition. In International Conference on Machine Learning, pp. 17258 17277. PMLR, 2022.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the Notional Academy of Sciences, 117(40):24652 24663, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in newral information processing systems, 32, 2019.
ShengYun Peng, Weilin Xu, Cory Cornelius, Matthew Hull, Kevin Li, Rahul Duggal, Mansi Phute, Jason Martin, and Duen Horng Chau. Robust principles: Architectural design principles for adversarially robust cnns. 2023.
Zhuang Qian, Shufei Zhang, Kaizhu Huang, Qiufeng Wang, Rui Zhang, and Xinping Yi. Improving model robustness with latent distribution locally and globally. Xin preprint arXiv:2107.04400 2021.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Yao Qin, Xuezhi Wang, Alex Beutel, and Ed Chi. Improving calibration through the relationship with adversarial robustness. Advances in Neural Information. Processing Systems, 34:14358-1M 14369,, 2021.
Sylvestre-Alvise Rebufi,, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint wrXi::208.009460 2021.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8108 8104. PMLR, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenct large scale visual recognition challenge. International journal of computer vision, 115:211 252, 2015.
Hadi Salman, Andrew Hyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenel models transfer better? Advances in Newral Information Processing Systems, 33:3533 3345, 2020.
Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In International Conference on Learning Represenlations, 2019.
Changhao Shi, Chester Holtz, and Gal Mishne. Online adversarial purification based on self-supervised learning. In International Conference on Learning Representations, 2021.
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-calibratee adversarial training: Generalizing to unseen attacks. In International Conference on Machine Leurning, pp. 9155-9166 PMI.R, 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint Xrxiv:1312) 6199, 2013.
Linwei Tao, Younan Zhu, Haolan Guo, Minjing Dong, and Chang Xu. A benchmark study calibration. arXiv preprint arxvv:230..1 11 11838, 2023.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33:1633-1645, 2020.
Nikolaos Tsilivis and Julia Kempe. What can the neural tangent kernel tell us about adversarial robustness? arXiv preprint arXi::210000577 2022.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint Xrv: 12152, 2018.
Dequan Wang, An Ju, Evan Shelhamer, David Wagner, and Trevor Darrell. Fighting gradients with gradients: Dynamic defenses against adversarial attacks. arXiv preprint arXiv:2105.0811 2021.
Yu Wang, Wotao Yin, and Jinshan Zeng. Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing, 78:29 63, 2019.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning, 2023.
Boxi Wu, Heng Pan, Li Shen, Jindong Gu, Shnai Zhao, Zhifeng Ti, Deng Cai, Xiaofei He, and Wei Liu. Attacking adversarial attacks as a defense. arXiv preprint arXiv:2106.04993 04998, 2021.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXi::1902.0447 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trado-off between robustness and accuracy. In International conference on machine learning, pp. PMLR, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. XrXiv preprint 1rXiu:1710.094 2017.
15

---

Published in Transactions on Machine Learning Research (08/2024)
A Proofs of the Main Results
In this section, we prove Theorem 3.2, and related results, Corollaries 3.4 3.6 and Remark 3.3.
A.1 Proof of Theorem 3.2
Proof. Recall that we assume y h(x) + 3T (xx) + where e E RC has independent coordinates, cach satisfying E|ea] = 0, E|ea] I o² for all c C (l....,0) CJ. The features for which we wish to compute robustness are the form f = Mό where M is a linear map.
We are interested in robustness with respect to the e-th component, which is computed as
inf 11x-x|1250
yeB. f(x)
= (x)a) [voST f(x)] + E (x.x)=D
inf
y=S. (f(x)) -(x))
(1)
1
We compute the first term exactly as
Exxy)D ((x)) = Exx,, ca)
(since B(ee] = 0),
= Ex [3] (xx)((x) MB.
= EM..
( = Ex [o(x)o(x) T]).
For the second term in (1), we aim to derive lower bound. Observe that
yaBk (f(x) - f(x)) = y. M (0(x) - x(xx)
2 - -lell |1 l/llx: || (M(x)) llx
2 - lyel B H || 110(x) - 6(x)]1
Using L-Lipschitzness of o, have |1ő(x) d(x)lx - L|| x1|2 LA. Hence, the second term in (1) can be bounded from below
(xx))
inf -| vo (f(x)-
| 11.3.11x LA Ex.eo [[yel]]
2 - 1 11, LA Ex.re [137 oxx) + eall
Finally, using Jensen's incquality, can write
Ex,t. [B x(x) + cel] S / Ex,e o(x) +c V 02 +2 B3.
Combining the above computation leads to
(D, 2 MAl 0² B..
which proves Theorem 3.2.
A.2 Proof of Remark 3.3
The proof requires assumption of Gaussian model, i.e., -N(0 E) and e NN(0 Since the feature map assumed to be linear, Φ(x) x, it follows that ye A__ ca is also Gaussian He (0,, + 3. ΣB.) and hence, |yel is half-normal distributed.
Proof.
Now recall that the first term in (1) can be computed exactly as B. B.M... To compute the second term in (1), note that
B. (f(x) f(x)) xx=*|
inf
v.B. M x) inf ||*-
16

---

Published in Transactions on Machine Learning Research (08/2024)
and the infimum is achieved when the difference is aligned with M T Be, that is, = x sign depends on the sign of yle, which leads to the second term in (1) compute to
M A [ Bellx
The
B(x,y>=
inf |
3o (f(x) f(xx)) -Exee [Ivel]
A
|| Bellze.
Since Wel is half-normal, E[lyal] = 2 /02 8: EB while || B. S M opp Bells with the inequality being tight when Be is the eigenvector of M, corresponding the largest eigenvalue.
A.3 Proofs of Corollary 3.4 and Corollary 3.6
Proof- In what follows, restrict the linear map M M = Ü-T= i where Ú= [ui 1 uk
is orthonormal matrix of basis for K-dimensional subspace. Since the operator norm || op for projection matrix, the problem of finding the most robust subspace corresponds to maximising B E. B.
E A Eu,u T B.-
Note that if (A, u) is eigenpair of Σ, then B Euu B. X(B u)a. Hence, if we restrict the choice of u1,...,1Ig to the eigenvectors of Σ, the optimal projection is obtained by choosing the K eigenvectors for which the robustness score su( A(B_ u)² are largest. So the claim of Corollary 3.4 holds only if the projections are restricted to eigenspaces of E. The claim of Corollary 3.6 follows along the same line as the information of the feature f = Má can be computed as (D) -. E. Br. For the case of M = ÜÙT where U is matrix of K eigenvectors of E, we have (D) = Σ (B u.)². Hence, if the search is restricted to eigenspaces of Σ, the most robust features also correspond to the most informative ones. R
Robust and informative features over all possible K-dimensional subspaces. If we consider M ÜÜ for any Ü [u...... uk] with orthonormal columns, as assumed Corollary 3.4, then
B. EMM Σ Em,u B. B. Trace (ŪB.A.E0T) Trace (UEA,3.
where the last equality follows from taking transpose. Hence, the resulting maximisation problem can be written as
max EMB. max Trace (ŪB.3..0)) = max Trace (ÜB.Ü"). Ú
(2)
where B. = (B.3.. -
). The above trace maximisation problem corresponds to finding the dominant cigenvectors of the matrix B This leads an alternative to Algorithm for finding robust projections for the test-time defense. The approach comprises of computing the dominant eigenvectors Ü. of the matrix B. for every class component E {1...., CJ and defining the robust output as h(x) The approach would result in thcoretically more robust projections, but
[ Ü, é(x)...., Ü. suffers computationally since it requires (C+1) + 1) eigendecompositions instead of only one eigendecomposition in Algorithm 1. Hence, it has O(C) more one-time computation than Algorithm 1, but with identical inference time. The conclusion of Corollary 3.6 that the most robust features, obtained from the maximisation in (2), are also the most informative features still holds in this case.
A.4 Dynamics of robust feature learning under GAM
In this short analysis, we argue that the trained model is a Generalized Additive Model (GAM), h(x) = TT d(x), the test-time defense of Algorithm 1 could also be replicated through early stopping of the training process. In other words, argue that the components of 8 Φ(x) along the robust features the eigen directions for which s(u) = XB are higher-aree are learned carlier.
For simplicity of analysis, we consider only the learning for e-th components, which corresponds to the following regression problem under GAM: Given training sample Durain = ((x,))) S X x R, minimize the squared loss
minimize bERP
l - D ((
2n
1=1
17

---

Published in Transactions on Machine Learning Research (08/2024)
A.5 Proof of Proposition 5.1
Proof. The optinal solution for b for the above problem when population squared loss is minimized is given by Be ( ) by, where - = [Ø(x),..... o(x,]] and y = [31... yb]" Furthermore, if the above optimisation is solved using gradient descent with learning rate > and initialisation b(0) = 0, the paramcters b(0) are learned over the iterations as
(1
L 0Φ T)
((-
2 Dy
Σ
1-1
1
! 1
& 7 bby
nt
n
1-1
&
7 ΦΦ T Se,
(3)
n
with ((0) + B. = =($b -1oy as I + co. Suppose the eigen decomposition Exrain n
1 - T
given by
E Etrain Udiag(A)UT Σ Auuu Hence, (3) bocomes
(UUT - =Udiag(A)UT) "yUdiag(A)U" Be
= Σ n - rydiag diag (A) B.
600 Σ (1-(11n))))uuuu)
(4)
From (4), it is clear that u, directions are learnt in the order of A₁ That is large X; learned carly during the training since (1 - - ())))))) is decreasing and al fixed 6, the eigendirection u; with the largest X is learned first. This proves that the direction with maximum variance is learned first. When the top eigendirection u aligns with the true signal Be, u, will be the most robust direction as well. Hence, the top directions basod on descending order of X is more robust if the directions align with the true underlying signal.
A.6 Connection to Neural Tangent Kernel features
We first briefly discuss NTK and the NTK features before proving the Proposition 5.2.
Neural Tangent Kernels (NTKs) and NTK features. Jacot al. (2018); Arora et al. (2019); Yang (2019) show the equivalence of training large width neural network by gradient descent to deterministic kernel machine called Neural Tangent Kernel. In the context of adversarial attacks and robustness, Tsilivis & Kempe (2022) propose a method to generate adversarial examples using NTK and show transferability of the attack to the finite width noural network counterpart successfully. Additionally, the authors define NTK the NTK gram matrix and observe that the robust features correspond
Teatures using the eigenspect the top of the eigenspectrum and learned first during training. In the following, we define the NTK and NTK features and show its equivalence to our robust fcature definition along with the proof that the robust NTK features correspond to the top of the spectrum. The NTK gram matrix θ E Baxn is between all pairs of datapoints and the NTK between X; and Xj for network that outputs f(w, x) at data point x E Rd parameterized by w E R² is defined by the gradient of the network with respect to w as
@ xj) Ew-.V(,11) V Jww, x;) TV (w,
(5)
For an extremely large width network, gradient descent optimization with least square loss equivalent to kernel regression, the kernel being the NTK. Formally, for a data x, the converged network output in the large width limit is f(w, x) = Oxx x) (X, X)-IY Tsilivis & Kempe (2022) define NTK features using the eigendecomposition of (X, = EL X
18

---

Published in Transactions on Machine Learning Research (08/2024)
- 0x YY= x O viv
vivi T Y = (x)
ker
(6)
where (x) RC is the i-th NTK feature of Note that fee is in accordance to our feature definition.
We prove the empirical observation that the top spectrum-induced NTK features fker are more robust in the following.
A.7 Proof of Proposition 5.2
Proof. Suppose that the NTK feature feer T-Lipschitz continuous in gradient of NTK with respect to x.
Then, we have
Oxx |2 L|0012
(7)
Recall that we can write the i-th NTK feature as (x) := X vivi Y. Bounding (ffe"((x + 8) fhee (x)]l2 by Taylor's expansion and applying (7) yield
| (x 5) 1hee (x)
1 Vx0(x.
XIvi Y A Rviv; Y1l2
(Where R remainder)
(
1T - 11012 TY
X)viv TY +
(from (7))
2
Vx0(x X) / 10112
vivi TY
0
where (a) follows from the Taylor's expansion of fler (x+ + 6) where is the remainder terms and (b) follows from (7), i.e. R (T/22)|/5[2.
Empirical validation: Top NTK features indeed robust. To verify Proposition 5.2, construct sanity experiment using simple 1-layer NN f(x) 1/w'' with parameters N/0, La). Let the data dimension d be 100, the number of training samples be 1000 and the data is sampled from a Gaussian ((0, Σ) where the covariance is diagonal spiked matrix, that is, Σn == 1 + dd/n and Eg = 14i 1. We then construct NTK features from the spectral decomposition of the exact NTK. Plot of Figure shows the norm of difference the original, and adversarially perturbed NTK features with respect to the eigenvalues of the NTK spectrun for different perturbation strengths of = (0.01, 0.05, 0.1). This validates our theory that the NTK features corresponding the large eigenvalues are more robust and hence remain closer to the original feature when perturbed.
Hd initialized from
B Experiments
B.1 Parameters for different algorithms
We set the parameters to the standard values in the literature. Refer to RobustBench for most of the attack parameters.
PGD: We perform PGD with the standard parameters in Table 8 to have an overall high strength PGD attack.
19

---

Published in Transactions on Machine Learning Research (08/2024)
fleer (x)]l
10
A 0.01
A 0.05
100
0.1
109
10-1
2.
10-3
10-1
10
10
10-1
10-2
10:1
Index of Eigenvalue
Figure 3: NTK feature robustness for X and the corresponding eigenvalue profile in ascending order.
Table 8: Parameters for PGD. We use these parameters for both training and altack.
Dataset
6p
step size
iteration
tx 8/255
e/1
40
CIFAR-10, CIFAR-100
f2 0.5
e/5
100
tiny ImageNet
fx 4/255
c/4
40
APGD-CE, APGD-DLR: We perform standard 6o perturbation with the budget = 8/255.
3. For Adversarial training in Table 10 we use same parameters for PGD, IAT, CW and TRADES used in PGD attack from Table 8
B.2 Details of benchmarking baseline methods
We perform benchmarking of our test-time defense on multiple SOTA methods that achieves adversarial robustness in the model. For our analysis of RFI on CIFAR-10 in table 10 we used PGD Madry et al. (2018), Interpolated Adversarial Training Lamb et al. (2019), Carlini- Carlini-Warger Loss Carlini & Wagner (2017) and TRADES Zhang et al. (2019) to adversarially train the baseline model. In the case of Robust CIFAR-10 Πlas et al. (2019), we only replaced the standard CIFAR-10 dataset with the robust dataset. In general for PGD, IAT and C&W attacks the adversarial training works as generating an adversarial example using the underlying attack and the objective is to minimize loss on these adversarial examples. PGD attack uses gradient descent to iteratively maximize the classification loss with respect to the input while projecting the perturbed example inlo the norm ball defined for the attack. IAT uses joint objective that minimnizes the classification loss of perturbed examples generated from PGD or any other attack along with classification loss on clean data with MixUP Zhang et al. (2017). We use Robust CIFAR-10 proposed in Ilyas et al. (2019), although is not an adversarial training method but rather the final dataset from procedure to only retain robust features in the dataset. Tlyas et al. (2019) disentangle the robust and non-robust features by creating a one-to-one mapping of an input to its robustified image. From an adversarially pretrained backbone (ResNet-18 using PGD & norm and < 0.25) linear layer features are extracted for the natural image and also from noise input. Then by minimizing the distance between these two representations the input space over the noisc, an image that only retains robust features of the original input is obtained.
For training using all these baseline adversarial training methods, set the batch size 128. We use SGD with momentum as the optimizer where we set the momentum to 0.1, also set the weight decay to 0.0002. We run our training for 200 epochs and set the lcarning rate schedule as starting with 0.1 for the first 100 epochs and then we reduce the learning rate by 90 percent every 50 epochs. For calibration using temperature scaling (Guo et al., 2017), we take the trained model and optimize for the temperature parameter. The standard deviation in all the cases of calibratod models is reported by loading the pretrained models and 5 runs of calibration. Hence, there no standard deviation for the non-calibrated models, and we also do not report the standard deviation for the SoTA models directly loaded from Robust Bench.
20

---

Published in Transactions on Machine Learning Research (08/2024)
B.3 Adaptive attack performance of RFI on Expectation Over Transformation (EOT) attack using ResNet-18 for CIFAR-10
Expectation Over Transformation (EOT) is procedure to synthesize examples that are adversarial over a chosen distribution of transformations (Athalye et al., 2018b). This procedure is shown to generate adversarial examples that are more robust to noise, distortions and affine transformations, and are consistently adversarial to the neural networks. FOT as an adversarial attack is observed be stronger (Tramer al., 2020) where randomized transformation is applied to an input x before being fed into classifier. RF1 can be easily integrated into the neural network classifier in such settings by computing the transformation matrix Ü in RFI by applying random transformations to the training samples to ensure similar distribution of the train and test sets.
We evaluate ResNet-18 using all the training settings considered in Table on CIFAR-10 for Expectation Over Transformation (EOT) as an adaptive attack. The hyperparameters are the same as considered for adaptive attack evaluation in Section 4.1. We evaluate Eo and &z attacks with 8/255 budget, c/4 step size and 40 iterations, and 0.5 budget, e/5 step size and 100 iterations, respectively. For RFI, we set K 10. We observe that RFII improves the performance by 1 to 2% consistently for EOT attack well.
Table Adaptive attack performance of RFI on Expectation over Transformation (EOT) attack. We consider foo (step size e/4, 40 iterations) and lg (step size c/5, 100 iterations) attack CIFAR-10 with ResNet-18. RFT improves robustness by 1 to 2% as shown in % Gain column.
Clean
8oo
$
f2(r = 0.5)
Training
266
Method
+RFT
Gain
Method
+RFI
% Gain
Method
+RFI
% Gain
PGD
81.08
80.49
-0.59
36.01
37.85
+1.84
35.52
36.65
+1.13
LAT
90.32 0.01
89.89 0.00
-0.43
26.92 0.00
0.01 28.30
+1.38
30.30 0.00
31.47
0.02
+1.17
C&W
77.55 1 0.03
77.50
-0.05
22.51 4
23.88
1 0.08
+1.37
25.71
a 6.00
26.95 0.00
+1.24
TRADES
79.17
+ 0.00
79.02 0.01
-0.15
47.20 0.01
47.98
a 0.01
40.778
48.55 0.09
49.61
0.00
+1.06
B.4
Adaptive attack performance of RFI on calibrated ResNet-18 for CIFAR-10
We evaluate ResNet-18 using all the training settings considered in Table 1 on CIFAR-10 for calibrated models. The hyperparameters are the same as non-calibrated setting. We evaluate foo and & attacks with é= 8/23 3/255 budget, e/4 step size and 40 iterations, and 0.5 budget, c/5 step size and 100 iterations, respectively. For RFI, we set. = 10. We observe that RFI improves the performance by to 9% for calibrated models.
Table 10: Adaptive attack performance of RFI on calibrated models using temperature scaling. Wo consider to (step size e/4, 40 iterations) and l2 (step size e/5, 100 iterations) atlack on CIFAR-10 with ResNet-18 18. RFI improves robustness by 4 9% as shown in % Gain column.
Training
Clean
255
Eaee 0.5)
Method
[RFI
% Cain
Method
|RPI
% Cain
Method
|RF
% Gain
Standard
95.20 + c.s
88.20 6.10
-7.00
2.01 uas
6.83
0.22
+4.82
2.58 0.2
10.21 0.1
+7.63
Robust CIFAR-10
78.70 + 0.0
78.73
0.00
+-0.33
3.81 + m.s
8.03
0.21
+4.22
9.10 + 692
11.21 0.s
+2.11
PGD
83.11 c.m
82.32 c.
-0.79
42.96 a a.s
50.08
u.s
+7.12
56.48 : 0.40
62.13
+5.65
IAT
91.24 a c.30
90.83 0.08
-0.41
46,22 a 0.30
51.34
0.50
+-5.2
63.48 = 0.96
71.12
0.20
47.64
C&W
84.36 0.10
63.32 0.05
-1.03
41.62 0.90
50.48
1:07
+-8.6
56.63 : 0.60
63.21
0.72
4.6.58
TRADES
81.11 c.oc
79.38
0.04
-1.73
o.sa 53.67
58.20
0.01
+-4.53
62.12 0.29
68.47
0.32
46.35
B.5 Adaptive attack performance of RFI for CIFAR-100 and tiny ImageNet
We evaluate both calibrated and non-calibrated ResNet-13 18 using all the adversarial training setting considered in Table 10 on CIFAR-100 since standard training would not result robust model. We also consider tiny ImageNet dataset that has 100, 000 training and 10, 000 validation samples with 200 classes and ResNet-50 pretrained adversarially on ImageNet. We evaluate for attack with 8/255 and 4/255 for CIFAR-100
21

---

Published in Transactions on Machine Learning Research (08/2024)
and tiny ImageNet, respectively. The attack budget is standard, taken from RobustBench. For RFI, we set K = 100 and 200 (number of classes) for CIFAR-100 (Table 11) and tiny ImageNet (Table 12), respectively. % Gain in tables is between Calibration-RFl and the base method.
Table 11: Adaptive attack performance of RFI on non-calibrated and calibrated models. Robust performance evaluation of RFI on CIFAR-100 with ResNet-18 (step size e/4 and 40 iterations). RFI improves the performace an average by 4%.
Clean
Ly (c
Training
Mechod
+RFI
+Calibration
Calibration+
% Gain
Method
+RFI
+Calibration
+Cllbration+REE
% Gain
PGD
55.30
55.27
55.82
55.08
-0.22
20.08
20.91
21.86
25.96
+5.88
IAT
58.94
58.88
58.86
58.09
-0.85
22.56
23.58
23.01
26.72
+4.16
C&W
49.36
49.31
49.30
49.02
-0.34
10.44
11.86
11.28
14.72
+4,28
TRADES
55.17
55.11
55.17
$5.10
-0.07
28.25
28.56
28.43
30.91
+2.66
In the case of tiny ImageNet, we subsampled 100 training samples per class instead of using the full training set for computing the transformation matrix U of the feature covariance due to the computation time, and evaluated the clean and robust performances on the 10, 000 validation samples. The results are given in Tables 11 and 12. We observe that RFI consistenly improves the adversarial performance on the dalasels with very small drop in the clean performance. Thus this shows RFI generalizes to larger datasets well. Furthermore, we would like to draw the attention that our method improves the performance even with small subsample of the dataset.
Table 12: Adaptive attack performance of RFI on non-calibrated and calibrated models. Robust performance evaluation of RFI on tiny ImageNet with ResNet-50 (step size e/4 and 40 iterations). RFI improves robustness even on large datascts.
Clran
B 265
Training
Method
+RFI
+Calibration CCalibratin++FI % Gain
Method
+RFI
CCalibration
+Calibration+ RFI % Gain
PGD
62.42
62.39
62.40
62.32
-0.10
33.38
33.50
33.43
34.27
+0.89
B.6 Adaptive attack performance of RFI on state-of-the-art models from RobustBench
For table 4 we benchmark our lest-time defense on multiple recent SoTA methods for CIFAR-10, CIFAR-100 and IeageNet. For all our baseline methods we obtain the model weights from RobustBench Croce & Hein (2020b). We update the weights of the last linear layer of the models using RFI and benchmark the updated models. We also report the performance for optimal K in RFI. We note that the Expected Calibration Error (ECE) for the SoTA models are very small as shown in Table 14 (already well calibrated), hence we do not explicitly calibrate in Table 13. Moreover, the results in Table 4 show that calibration will only further improve robustncss with RFI. Therefore, we do conservative analysis of RFI on the SoTA models. For Salman et al. (2020) on ImageNet we compute with and without dynamic RFT and not Anti-Ady and SODEF since it increase the inference costs of the evaluation such that we could no longer run experiments with our computational resources. Also we do not report AutoAttack since it requires all 4 attacks i.e. APGD-CE, APGD-DLR, FAB and Square to be executed sequentially which is outside the scope of max runtime of our resources. Nevertheless, we observe that. RFI improves the robustness reliably ~ 1.5% on average on non- non-calibrated SoTA models. Importantly, SODEF and Anti-ady reduces the robustness performance especially on AutoAttack which is inline to the findings of Croce & Hein (2020b).
B.7 Transferability Study
We conduct a more detailed transferability of allack analysis on CIFAR-10 using ResNet-18 and on CIFAR-100 using PreActResNNtt-13. Here, we generated adversarial examples with respect to the base model and all the defences and evaluated the robustness of different adaptive defences under all the adversary cases (Transfer attacks). Then we present the results for calibrated ResNet-18 on CIFAR-10 in Table 16 which completes the analysis together with the results from Table 2. We observe that the robustness of the calibrated model
22

---

Published in Transactions on Machine Learning Research (08/2024)
Table 13: Adaptive attack performance evaluation of RFI on state-of-the-art methods. We apply APGD-CE, APGD-DLR, and RobustBench attacks on CIFAR-10 and CIFAR-100. The inference time for RFI 1x, whereas Anti-adv and SODEF are 8x and 2x, respectively. There no standard deviation as the trained models directly from RobustBench. While RFI improves the robustnoss to AutoAttack upto 1.5% without calibration, SODEF and Anti-adv results in no (< 0.1%) decrease in robustness consistently.
Base Method
Defense
Clean
APGD-CE
APGD-DLR
FAB
Square
AutoAttack
None
89.69
61.82
60.85
60.18
66.51
59.53
Carmon et al. (2019) WideResNet-255
Anti-adv
89.69
61.81
60.89
60.11
66.58
58.70
SODEF
89.68
60.20
60.72
58.04
65.28
57.23
RFI [K 10)
89.60
62.38
61.58
60.21
66.59
60.72
RFI (opt. K 20)
89.60
62.45
61.60
60.38
66.90
61.02
None Anti-adv
87.03
51.75
60.10
49.90
58.00
49.25
CIFAR-II
Engstrom al. (2019)
87.00
51.62
59.95
49.84
58.06
49.20
ResNet-50
SODEF
86.95
50.01
58.20
48.64
56.68
47.92
RFI (K 10)
87.01
51.86
61.84
51.28
58.07
50.75
RFI (opt. K = 15)
87.03
51.94
61.90
51.46
58.12
50.98
None
85.34
50.12
56.80
53.87
56.88
53.42
Rice et al. (2020)
Anti-ady
85.40
50.10
57.50
53.90
57.00
50.98
WideResNet-34-10
SODEF
85.10
50.60
56.50
53.72
56.21
50.09
RFI(K =10)
85.30
51.19
58.55
53.98
57.13
54.64
RFI (opt. K=
85.30
51.62
58.97
54.12
57.13
54.86
None
92.44
70.23
67.82
67.41
73.13
67.31
Wang al. (2023)
Anti-adv
92.44
68,90
65.91
67.55
73.20
66.52
WideResNet-28-10
SODEF
92.01
67.53
65.08
65.93
73.01
64.20
RFI (K 10)
92.33
70.32
67.86
67.82
73.52
67.29
RFI (opt. K 20)
92.34
70.36
67.90
67.82
73.54
67.50
None
63.66
35,29
31.71
31.32
35.70
31.08
Pang et al. (2022)
Anti-adv
63.41
32.50
30.32
31.30
35.76
30.10
WideResNet-22--
SODEF
63.08
30.96
29.54
31.44
32.27
30.56
RFI (K- 100)
63.01
36.03
31.95
31.88
35.79
31.29
RFI (opt. K = 115)
63.10
36.07
31.95
31.96
35.88
31.91
CIFAR-10X 100
None
65.45
33.49
28.55
28.00
33.70
27.67
Addepalli al. (2022)
Anti-adv
65.38
30.92
26.61
27.92
33.61
26.01
ResNet-18
SODEF
65.23
29.37
26.90
24.62
29.60
26.53
RFI (K = opt. K= 100)
65.41
34.09
29.18
28.10
33.79
27.80
None
53.83
20.83
20.46
23.82
19.29
18.95
Rice al. (2020)
Anti-ady
53.83
20.78
20.06
23.49
19.27
18.97
PreActResNet-I1
SODEF
53.83
18.50
19.20
19.66
16.05
16.92
RFI [K 100)
53.70
21.10
20.98
20.93
18.13
19.23
RFI (opt. K 150)
53.75
21.18
21.10
21.03
19.53
19.46
None
72.58
44.04
39.78
39.19
44.46
38,83
Wang al. (2023)
Anti-adv
72.57
42.98
38.10
36.85
44.49
34.01
WideResNet-28-10
SODEF
72.34
38.10
36.95
34.82
44.42
32.29
RFI [K = 100)
72.55
44.37
39.91
39.68
44.50
39.10
RFI (opt. K = 115)
72.55
44.51
39.96
39.81
44.53
39.13
ImageNet
Salman al. (2020)
None
64.02
38.32
34.02
34.35
49.52
ResNet-50
Dynamic REI
63.91
38.48
34.68
34.68
49.98
Salman al. (2020)
None
68.46
40.67
37.09
37.81
54.61
WideResNeet-0-2
Dynamic REI
68.41
40.84
37.56
38.12
54.78
with RFI is on par with the base calibrated model. Moreover, when allacked with examples from RFI integrated model, the base model performs worse. Notably, the decrease in robust performance of the base method much more than the decrease of the performance of RFI when evaluated on adversary from the base method. This shows RFT's goodness and further confirms the absence of an obfuscated gradient in RFL. Similar observation using SoTA model CIFAR-100 are in Table 15 (Appendix). In rast, transfer
23

---

Published in Transactions on Machine Learning Research (08/2024)
Table 14: Expected Calibration Error (ECE) of the SoTA models are very small, hence already well calibrated.
CIFAR-10
CIFAR- 100
Method
ECE
ECE after Calibration
Method
ВCE
ECE after Calibration
Carmon et al. (2019)
4.310
0.328
Pang et (2022)
0.364
0.142
WideResNet-255
WideResNet-255
Engstrom et al. (2019)
0.091
0.065
Addepalli et al. (2022)
0.418
0.347
ResNet-50
ResNet-18
Rice et al. (2020)
0.074
0.037
Rice et al. (2020)
0.138
0.074
WideResNett34-10
PreActResNet-II
Wang et al. (2023)
0.145
0.039
Wang et al. (2023)
0.366
0.290
WideResNet 28-18
WideResNet
attacks from base model on SODEF show a significant drop in robustness (Section 4.6.2 of Kang et al. (2021)) and on Anti-adv render the defense ineffective (Section 3.8 Croce et al. (2022)). These results further highlight the soundness of RFI.
B.7.1 RFI results in stronger adversary against transfer from other defenses
In the set of experiments, we evaluate all combinations of transfer attacks on CIFAR-100 and PreActResNet- 18 Rice et al. (2020) in Table 15. We compare the transforability of all adaptive test-time defenses to base model and within themselves by using adversarial examples generated with one defense altacking another defense. The general observation and expectation is that the model performance is affected the most when the adversarial examples are created using the same model, i.c., adaptive attack. This observation holds in our experiments too. The most interesting and impressive observation is that RFI outperforms all other methods in almost all the cases, even when adversarial ezamples are generated from base model + RFT. Notice that SODEF and Anti-adv suffer the most when adversarial examples are generated from the respective models, unlike RFI showing the impressive robustness of our method.
Table 15: Transfer attack on non-calibrated PreActResNet-188 for CIFAR-100. RFI outperforms in all the cases and also generates the strongest adversary for the base model.
Adversacial Examples are generated from Method (Rice et al)
Adversorial Examples are generated from Method+SODEF
Attack
Method
AAulAAdv
-SODEF
+RFI
Attack
+AnriAdv Method
+SODEF
-RFI
APGD-CE
20.83
20..6
27.13
27.30
APGD-CE
32.99
37.32
18.50
37.30
APGD-DLR
20.46
20.52
29.33
29.53
APGD.DLR
33.65
38.34
19.20
38.30
FAB
19.29
19.28
35.38
35.90
FAB
39.67
18.11
16.05
48.12
Squire
23.58
Square
39.59
48.20
19.06
48.22
AutoAttack
18.95
18.97
26.09
26.43
AuroAtrack
32.76
33.16
15.69
37.23
Adversarial Examples generated from Method AntiAdy
Adversarial Examples are generated from Method RFI
Attack
Method
+AniiAdv
+SODEF
+RFI
Attack
Method
AutiAdv
SODEE
RFI
APGD-CE
20.59
20.58
27.31
26.65
APGD-CE
14.70
18.31
18.40
21.18
APGD-DLR
20.39
20.49
28.92
28.53
APGD-DLR
14.12
18.30
19.21
21.10
FAR
19.27
19.27
35.80
38.69
FAB
12.76
14.12
14.70
18.13
Square
23.60
23.49
37.41
39.04
Square
16.29
18.95
19.50
20.93
AutoAttack
18.98
18.96
25.61
26.15
AutoAttack
12.55
16.50
16.92
19.46
B.8 Transfer attack: RFI with calibration is on par with the base model
Results transfer attacks, where we assess the performance of RFI against adversarial samples generated from the basc, for calibrated and on CIFAR-10 with Resnet 18 backbone are in Tables 16. Notably, RFI demonsirates comparable robustness to the base model, ensuring that gradient obfuscation is not al play in RFI and affirming that it reliably improves the model robustness. Moreover, the transferability from RFI leads to degradation in robustness for the base model, suggesting that RFI acts as an on-par adversary to the base (refer +RFI rows of the left subtable in the Table 16). We hypothesize that the attack from base model and attack from base model RFT affect diffeerntt semantics examples such that.
adversary
24

---

Published in Transactions on Machine Learning Research (08/2024)
average both are on par post-calibration. As expected the adversarial samples from the base method + RFT are more powerful and reduce the robustness of the base method to greater extent than vice versa.
Table 16: Transfer attack performance of RFI calibrated models. RFI is par with the base, ensuring reliable robustness improvement without gradient obfuscation. Setting same as Table 10. The decrease in robustness of the base model is much more than the robustness of RFI when evaluated on the adversary from the base.
Adversary generated from base model+RFI
(c 255
8
E2 0.5)
Training
Method
+RFI
Method
+RFI
PGD
42.85 1 0.12
50.08
0.88
57.18
0.54
62.13 1 0.92
IAT
47.92 + 0.31
51.34
+ 0.93
64.38
0.33
71.12 + 0.29
C&W
40.73 4 0.64
50.48
è 1.07
55.96
0.88
63.21
+ 0.72
TRADES
55.43 + 0.42
58.20
0.01
64.34 0.40
68.47
+ 0.32
Adversary generated from base method.
(e 205
l2(e 0.5)
Training
Method
+RFI
Method
+RFI
PGD
42.96 = 0.75
41.38 + 0.48
56.48
# 0.42
54.28 0.62
IAT
46.22 0.10
43.44 +
63.48
+
62.19 0.09
C&W
41.62
39.10
56.63
55. 19
TRADES
53.67 0.48
52.88 0.38
62.12
0.28
59.85 0.97
B.9
Static vs Dynamic RFI on calibrated model
We extend the study of static vs dynamic RFI to calibrated models in this section using the same setup Section 4.3, where we consider pretrained ResNet-18 on CIFAR-10 by applying PGD (fx, = 8/255) and (l2, 0.5) in transfer attack setting i.e. generate adversarial examples from the base method. For the dynamic setting we compute the covariance batch-wise to compute U with the input. Table 17 shows static is betler than dynamic RFI similar to non-calibrated selling.
Table 17: Additional Comparison of static and dynamic/adaptiin RFI on calibrated model showing static RFI is better than dynamic RFI. Setting same as Table 10. Adversarial examples are generated from the base model for fair comparison.
Clean
fac (c
fafe 0.5)
Training
255
Static
Dynsmic
Static
Dyuamic
Static
Dynamic
Standard
10.36
20.08
11.64
12.45
Robust CIFAR-10
78.78
75.23
15.41
12.89
17.38
16.32
PGD
83.22
82.86
46.02
46.83
58.81
59.23
IAT
91.26
91.35
49.06
48.53
66.67
66.28
C&W
81.97
83.01
45.48
43.98
58.95
57.82
TRADES
80.76
78.98
54.33
53.58
65.23
65.00
B.10 Static RFI is Optimal
In the case of dynamic RFI implementation, one needs to know when to apply the transformation as the model should be static for the altakr and adapled only for the defender. This poses implementation difficulty as the situation is mostly unknown in practice. Hence, we explore different variants of RFI in a dynamic setting where we compute the covariance matrix and eventually the transformation matrix ò using the full validation set or single test input. We observe that the dynamic RFI is only maryinally better than the static RFI when full validation set is used in Table 18 (a). Similarly, we present the result. for single test.
25

---

Published in Transactions on Machine Learning Research (08/2024)
input in Table 18 where the method shows improvement in clean performance since it is only normalization of the feature representation. We perform these comparisons to highlight the fact that these hypothetical variants of dynamic RFI which work with information of validation set are also not significantly better than static RFI, thereby implying that static RFI is indeed the optimal way of selecting Ü as indicated by our theory.
Table 18: Static RFI is the optimal approach. RFI with covariance matrix calculated using different approaches.
RFI with covuriance matrix cxcenlated complete validatton
(b) RFI with covarinnce colenlated using single ispat
Cufc
$
fg/e
253
(2fe 0.5)
Tradinn
Training
Method
RFI
Method
IRFT
Method
[RF
Method
RPI
Method
[RFT
Method
RRP
Standard
95.28
Stancard
0.30
12.16
Robast CIFAR-10
78
78.80
1.30
11.21
12.56
Robust CIFAR-10
78
78.70
1.301
11.88
9.63
12.87
PGD
K3.53
83.29
42.20
43.82
54.61
56.13
PGD
83.53
83.52
42.20
44.08
54.61
14..5
CRW
85.11
85.06
40.01
43.48
65.02
57.83
C&W
85.11
85.11
40.01
43.48
55.02
68.09
TRADES
61.79
TRADES
$1.00
64.78
00.03
62.17
B.11
Ablation study
B.11.1 Effect of K
Neural Collapse phenomenon which the penultimate feature of each class collapses to its mean after the training error reaches zero. This implics that there principally only C= #classes number of feature vectors, one for each class. Hence, we suggest setting K to number of classes. We also justify it experimentally in Figure 2. Additional experiments for large-scale models used in Tables 4 and 13 with respect to CIFAR- 10 and CIFAR-100 also show drop in cigenvalues at the number of classes across models, justifying our choice for K. We also exlend the ablation study on K to report the best performance in Figure 2 and the optimal row in 13. Note that the optimal K for robust performance is not the best for standard performance as we are choosing only the nopt informative features (Corollary 3.6).
CIFARIO
CIFARIO0
Carmon et al.
Engstrom et al.
Rice et al.
Wang el al.
Addepalli el al.
Pang el al.
Rice el al.
Wang et al.
robust
no
Accuracy
k
clean
130
110
C10
832
400
7272
7.11
k
Figure 4: Ablation of performance with K for all SoTA models for CIFAR-10 and CIFAR-100.
CIAAR1O
CIFAR190
10
10
104
10a
Nayoa Coretal
Jug mill
10
10
10"
10
10
Figure 5: Eigenspectrun showing sharp drop at K number of classes for all SoTA models on CIFAR-10 and CIFAR-100.
26

---

Published in Transactions on Machine Learning Research (08/2024)
B.11.2 Effect of step size in PGD
We chose c/4 and c/5 for step sizes Eo and f2, respectively, following the benchmarks in several works RobustBench. The other common choice for the step size proportional the iterations, that is, 2c/40 and 2c/100 for lx and l2, respectively. We reevaluated the models in Table 10 with and without RFI for these step sizes and the results are in Table 19, showing that RFT is beller than the base. model, in line with the observations in the previous experiments.
Table 19: RFI is more robust than the base model irrespective of the step size in PGD. 2c/40 for for and 100 for lz.
lo
E2(c 0.5)
Training
255
Method
+RFI
Method
+RFI
Standard
0.03
9.73
3.67
14.13
PGD
44.44
45.48
57.77
58.97
IAT
45.91
48.26
66.26
67.73
Robust CIFARIO
7.14
15.57
12.94
17.15
CW Attack
38.89
41.53
51,20
54.45
TRADES
52.90
54.10
61.66
63.35
B.12 Conceptual ideas similar RFI
Low dimensional last layer. Similar to comparing RFI on last layer on intermediate layer in Sec- tion 4.5.3, here we compare RFI and directly training a network with K neurons in the last layer. We consider two ResNet-18 models with an additional fully connected hidden layer of size 512 and 10, respectively, and are trained with PGD. We apply RFI only to the larger model with 512 neurons and reduce the dimension to 10, and compare the performances terms of clean and robust accuracies in both cases. The results are presented Table 20, showing that RFI more robust compared to imposing low dimensional last layer.
Table 20: RFI is more robust compared to imposing a low dimensional last layer. ResNet-18 with last hidden layer size 10 and 512. RFI done on model with 512 hidden layer.
+ Hiidenn layer=10
+hidden layer=512 +hidden layer=512 + RFI
Clean Robust (lx0)
83.71
84.13
84.05
42.43
42.73
43.53
We further argue qualitatively why setting low dimension layers is not equivalent to RFl as follows. Firstly, overparameterizaation is shown empirically to be the key for both generalization Brutzkus & Globerson (2019) and robustness Madry et al. (2018). Especially in the case of CNN, there is an empirical understanding build the network with more than one fully connected layer after the convolution layers starting with larger widths to generalize well Bengio (2012). These findings oppose the idea of having low dimension for the last hidden layer. Secondly, there similar insights from the sparsity of neural networks a smaller subnetwork with similar performance can be obtained by sparifying the network, called lottery ticket Frankle & Carbin (2018). Once known, lottery tickets can be trained from scratch to reach similar performance the original network. However, it is not possible to obtain the ticket simply by setting hyperparameters for a smaller network from the beginning. Finally, we emphasize that with RFI the last hidden-layer dimension is reduced by a large amount in comparison to the actual model. For example, in CIFAR-10, ResNet-50 with 2048 dimensions is reduced to 10(= K). So, the network with 10 dimension conventionally would not help generalization, which is conclusively established in the above experiment.
27

---

Published in Transactions on Machine Learning Research (08/2024)
B.13 Visualization of robust and non-robust features
We oblain the visualizations of robust and non-robust features for input x by solving
arg min [x (x))UU"|
where U is top K eigenvectors based on se(.) for robust features and all eigenvectors except top K for non-robust features. The objective is solved using gradient descent. Figure shows the visualizations of features for a few classes in CIFAR-10 using the PGD adversarially trained ResNet-18 model. 'Robust K 10' and 'Non-robust K = 10' columns are obtained by setting U to the top K eigenvectors and everything except the top K eigenvectors based on sc(.), respectively. The columns top and bottom 100 eigenvectors are obtained by setting to the top and bottom 100 eigenvectors based on the cigenvalues. The feature visualizations show that robust and top eigenvectors result in more similar features. The interesting observation that the non-robust and bottom eigenvectors are equally noisy and might have some useful information that reflects the drop in clean performance. Nevertheless, is not possible to argue based on the visual interpretation of the features since the difference is primarily coming from the eigenspace of the feature covariance.
Robust
Top 100 Eigenvectors
Non-Robust K 10
Bottom 100 Eigenvectors
Original
K = 10
airplane
truck
dog
horse
Figure 6: Robust and non-robust features visualization. The features obtained using the PGD adversarially trained ResNet-18 model, and the original images are from CIFAR-10. The columns robust K = 10 are the robust features by fixing U to top K eigenvectors based on the score function se(-), whereas top 100 eigenvectors is based on the largest 100 eigenvalues. Likewise, non-robust K = 10 are obtained by fixing U to all eigenvectors except the ones robust K = 10 and bottom 100 eigenvectors are obtained using the smallest 100 eigenvalues, respectively.
28


Reviews:
[{"review_id": "YlfBAItt9n", "reviewer": "Reviewer_DgBb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[S1] Their method is simple to adopt on any deep learning model and easy to combine with other defense methods. It could be highly effective in large-scale deep learning models.\n\n[S2] They compare their method to various existing methods based on extensive experiments. Furthermore, it shows higher performance compared to other methods.\n\n[W1] In [1], the authors suggest the expectation over transformation (EOT) method that can successfully attack models with any transformation of images. How robust is the RFI method against EOT-combined attack methods? Although the proposed method does not change the input, the EOT attack method can be modified for the RFI method.\n\n[W2] Is the proposed method effective for transformer-based classification models? The motivation of this work, the idea of robust features, seems to target CNN-based classification models, so it may not work well on transformer-based models. However, it would be very beneficial if the authors show that this method works well on various models.\n\n[1] Synthesizing Robust Adversarial Examples, ICML (2018)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Defense methods against adversarial attacks are important for developing reliable AI systems. However, most existing methods require more computational resources, huge datasets, or longer test times for inference. The authors provide a novel defense method that does not require more computational resources, datasets, or test time. They suggest the RFI method, motivated by the idea of robust features.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(Major) I strongly recommend that the authors test their method on EOT-combined attack methods. This would make their method more robust (see W1).\n\n(Minor) I recommend that the authors show the experimental results on transformer-based models. If the proposed method does not work well, it would be better if the authors explain the reason for that.", "broader_impact_concerns": "It seems that there is no concern about ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4PtDSl1FUA", "reviewer": "Reviewer_Rc5R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* This paper proposes a new test-time defense algorithm, which can be seamlessly integrated with any robust learning algorithms in the training process. \n\n* This paper develops a theoretical analysis for the intuition and the developed algorithm, which justifies the validity of the proposed algorithm.\n\n* Experimental results highlight the superior performance of the developed algorithm, improving baseline robust accuracy by 1%-2%.\n\nWeaknesses:\n\n* The study primarily focuses on the output layer of the neural network, leaving the hidden layers and their representations, i.e., the representation $\\Phi(x)$  unchanged. However, in practice, $\\Phi(x)$ cmay also be sensitive to perturbations for some inputs, it would be beneficial to consider the sensitivity of  $\\Phi(x)$ as well.\n\n* Additional visualizations or experimental discussions regarding the robust and unrobust features could enhance understanding. These terms are widely used in literature and might differ from their usage in this paper, thus a more detailed discussion should be added.\n\n* The impact of the defense algorithm on clean accuracy is noted, but not theoretically justified. Including such justification could strengthen the paper's conclusions.\n\n* Considering the test-time defense approach presented, exploring whether a corresponding adversarial attack method could be designed—specifically targeting the space of robust features with all available perturbation budget—would be an interesting extension of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel test-time defense method designed to counter adversarial attacks during inference. Initially, it identifies the concept of robust features, subsequently developing a comprehensive theory and an algorithm to determine these features. Furthermore, leveraging these robust features, the authors have crafted efficient algorithms for classifying test data, resulting in enhanced robust performance. Experimental results back up the proposed algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness section above.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rOGFGTfL5J", "reviewer": "Reviewer_kcT8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and the designed method is presented well. The empirical study is extensive and corroborates the effectiveness of the proposed RFI.\n\n\nWeaknesses:\n\nAccording to the guidelines of TMLR, the paper does not have obvious weaknesses as the empirical evidence is sufficient and strong. \n\nIf I have to say some weakness, the effectiveness of RFI is not quite strong on CIFAR100 when AutoAttack is used, according to Table 4..", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper designs a robust feature inference (RFI) approach based on principal component analysis of the training features. The top-K principal components are selected as more robust features according to the theoretical analysis in Sec. 3.1. The proposed RFI is tested on five datasets using both white-box and black-box attacks and is demonstrated to improve the original model's adversarial robustness across different experimental settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It looks like the RFI is more effective when using the $l_{\\infty}$-norm attack than the $l_2$ attack, which is a bit of weird to me as the theoretical part defines the robustness with respect to $l_2$ norm. \n\n2. Please make Fig. 2 high-resolusion using the pdf/svg format.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is recommended for acceptance due to its strong theoretical basis, practical RFI method, and extensive empirical validation demonstrating improved adversarial robustness. Reviewers found the research interesting and well-presented, though noted a limitation in scalability, as the method is primarily evaluated on CNN-based models. It is suggested that the authors discuss this limitation in the camera ready version. The findings are relevant and valuable to the TMLR audience interested in adversarial robustness in AI.
Decision: Accept (poster)