Test-time Adaptation for Better Adversarial Robustness
Authors: Zhichao Huang
Chen Liu
Mathieu Salzmann
Sabine Süsstrunk
Tong Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Standard adversarial training and its variants have been widely adopted in practice to achieve robustness against adversarial attacks. However, we show in this work that such an approach does not necessarily achieve near optimal generalization performance on test samples. Specifically it is shown that under suitable assumptions, Bayesian optimal robust estimator requires test-time adaptation, and such adaptation can lead to significant performance boost over standard adversarial training. Motivated by this observation, we propose a practically easy to implement method to improve the generalization performance of adversarially-trained networks via an additional self-supervised test-time adaptation step. We further employs a meta adversarial training method to find a good starting point for test-time adaptation, which incorporates the test-time adaptation procedure into the training phase and it strengthens the correlation between the pre-text tasks in self-supervised learning and the original classification task. Extensive empirical experiments on CIFAR10, STL10 and Tiny ImageNet using several different self-supervised tasks show that our method consistently improves the robust accuracy of standard adversarial training under different white-box and black-box attack strategies.

Under review as a conference paper at ICLR 2023
TEST-TIME ADAPTATION FOR BETTER ADVERSARIAL ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Standard adversarial training and its variants have been widely adopted in prac- tice to achieve robustness against adversarial attacks. However, we show in this work that such an approach does not necessarily achieve near optimal general- ization performance on test samples. Specifically. it is shown that under suitable assumptions, Bayesian optimal robust estimator requires test-time adaptation, and such adaptation can lead to significant performance boost over standard adver- sarial training. Motivated by this observation, we propose a practically easy-to- implement method to improve the generalization performance of adversarially- trained networks via an additional self-supervised test-time adaptation step. We further employ meta adversarial training method to find good starting point for test-time adaptation, which incorporates the test-time adaptation procedure into the training phase and it strengthens the correlation between the pre-text tasks in self-supervised learning and the original classification task. Extensive empirical experiments on CIFAR 10, STL10 and Tiny ImageNet using several different self- supervised tasks show that our method consistently improves the robust accuracy of standard adversarial training under different white-box and black-box attack strategies.
INTRODUCTION
Adversarial Training (AT) (Madry et al. 2018) and its variants (Wang et al., 2019 Zhang et al.,
2019) are currently recognized as the most effective defense mechanism against adversarial attacks. However, AT' generalizes poorly; the robust accuracy gap between the training and test set in
much larger than the training-test gap standard training of deep networks (Neyshabur al. 2017
Zhang ot al. 2017). Unfortunately, classical techniques to overcome overfitting in standard training, including regularization and data augmentation, only have little effect in AT (Rice et al.) 2020).
Theoretically, as will be shown in Section 3, the loss objective of AT does not achieve optimal robust- Instead, under suitable assumptions, the Bayesian optimal robust estimator, which represents the statistical optimal model that can be obtained from training data, requires test-time adaptation. Compared with the fixed restricted Bayesian robust estimators, the test-time adapted estimators largely improve the robustness. Therefore, we should perform the test-time adaptation for each test input to boost the robustness.
To this end, we propose to fine-tune the model parameters for each test mini-batch. Since the labels of the test images are not available, we exploit self-supervision, which is widely used in the standard training of networks (Chen et al.) 2020b; Gidaris et al., 2018; He et al., 2020). Fine-tuning the self-supervised tasks has high gradient correlation with fine-tuning the classification task so that it forms a substitute of fine-tuning the classification loss at the inference time. Thus, we expect minimizing this self-supervised loss function yields better generalization on the test set.
To make our test-time adaptation strategy effective, we need to search for good starting point that achieves good robust accuracy after fine-tuning. As will be shown in our experiments, AT itself does not provide the optimal starting point. We therefore formulate the search for such start point as a bilevel optimization problem. Specifically, we introduce a Meta Adversarial Training (MAT) strategy dedicated to self-supervised fine-tuning inspired by the model- model-agnostic meta-learning (MAML) framework (Finn et al.] 2017). To this end, we treat the classification of each batch of adversarial images one task and minimize the corresponding classification error of the fine-tuned

---

Under review as a conference paper at ICLR 2023
network. MAT strengthens the correlation between the self-superrised and classification tasks so that self-supervised test-time adaptation can further improve robust accuracy.
In order to reliably evaluate our method, follow the suggestions of (Tramer et al. 2020 and design an adaptive attack that is fully aware of the test-time adaptation. Using rotation and ver- tical flip the self-supervised tasks, we empirically demonstrate the effectiveness of our method on the commonly used CIFARIO 10 (Krizhevsky et al., 2009), STL10 (Coates et al.] [201] and Tiny
ImageNett (Le & Yang, 2015) datasets under both standard (Andriushchenko et al. 2020 Croce &
Hein Madry et al. (0188) and adaptive attacks in both white-box and black-box attacks. The
experiments evidence that our method consistently improves the robust accuracy under all attacks.
Our contributions can be summarized as follows:
1. We show that the estimators should be test-time adapted in order to achieve the Bayesian optimal adversarial robustness, even for simple models like linear models. And the test-time adaptation largely improves the robustness compared with optimal restricted estimators.
2. We introduce the framework of self--uper test-time fine-tuning for adversarially-traiee networks, showing that it improves the robust accuracy of the test data.
3. We propose meta adversarial training strategy based on the MAMI. framework to find good starting point and strengthen the correlation between the self-supervised and classification tasks.
4. The experiments show that our approach is valid on diverse attack strategies, including an adaptive attack that is fully aware of our test-time adaptation, in both white-box and black-box attacks.
RELATED WORK
Adversarial Training. In recent years, many approaches have been proposed to defend networks against adversarial attacks (Guo et al., 2018 Liao et al., 2018 Song et al.] 2018). Among them, of the most robust and popu- lar methods, even under various strong attacks (Athalye et al. 2018 Croce & Hein| 2020a). AT optimizes the loss of adversarial examples find parameters that robust adversarial attacks. Several variants of AT (Wang et al. 2019: Zhang et al. 2019) also achieved and similar performance to AT (Rice al.] 2020). One important problem that limits the robust accuracy of AT is overfitting. Compared with training on clean images, the gap of robust accuracy between the training and test set much larger in AT (Rice et al. (2200). Moreover, traditional techniques to prevent overfitting, such as regularization and data augmentation, have little effect. Recently, some methods have attempted to flatten the weight loss landscape to improve the generalization of AT. In partieular, Adversar- ial Weight Perturbation (AWP) (Wu et al., (020)) achieves this by designing a double-perturbation mechanism that adversarially perturbs both inputs and weights. In addition, learning-base based smooth- ing can flatten the landscape and improve the performance (Chen et al. 20216).
Adversarial Training (AT) (Madry al. 2018) stands out
Self-supervised Learning. In the context of non-adversarial training, many self-supervised strate- gies have been proposed, such rotation prediction (Gidaris et al.) 2018). region/component fill- ing (Criminisi et al., (2044,, patch-base base spatial composition prediction (Trinh et al.) (2019) and con- trastive learning (Chen
2020b He et al. 2020). While self- superviiiii has also been employed in AT (Chen et al. 2020a] Kim et al. [0200] Yang & Vondrick 2020 [Hendrycks et al.] 2019). their methods only use self-supervised learning training time to regularize the parameters and improve the robust accuracy. By contrast, we propose to perform self-supervised fine-tuning at test time, which we demonstrate to significantly improve the robust accuracy on test images. As will be shown in the experiments, the supervised test-time adaptation has larger and complementary improvements over the training time supervupervi
Test-time Adaption. Test-time adaptation has been used in various fields, such as image super- resolution (Shocher et al.) 2018) and domain adaption (Sun et al. 2020, Wang et al. 2021). While our work is thus closely related to Test-Time Training (TTT) in (Sun et al. 2020), we target a sig- nificantly different scenario. TTT assumes that all test samples have been subject to the same distri- bution shift compared with the training data. As a consequence, it incrementally updates the model parameters when receiving new test images. By contrast, in our scenario, there is no systematic distribution shift, and it is therefore more effective to fine-tune the parameters of the original model for every new test mini-batch. This motivates our MAT strategy, which searches for the initial model parameters that can be effectively fine-tuned in self-supervised manner.
2

---

Under review as a conference paper at ICLR 2023
3 THEORY OF TEST-TIME ADAPTATION
In this section, we study the relationship between the test-time adaptation and Bayesian optimal robustness, which represents the optimal robustness that can be achieved from the training data, showing that the test-time adaptation can extend the function classes and improve the robustness of the model.
Definition 3.1. For a model F(x) with Ez adversarial constraint ||x* -xl1 define its natural risk and adversarial risk as at point x
1 (F) (x Elylx))?
(FF() Hix (F) (F)
max -x
(F(x)) - F()))
Remarks. We use the MSE loss to define the natural risk A (F) and adversarial R (F)) at point x. Similar to TRADES (Zhagg et al.) 2019) The adversarial risk is defined as the sum of natural risk and the loss changes under adversarial attack, and it can be bounded by the maximum MSE loss within the adversarial budget
Ex Rix padd
(F) S Ex max xx -x-=x
(F(x*) - x < 21Ex 12 (FF)
small R (F) guarantee small test error on x". Small (R(((() and R (F) represents good clean performance and high adversarial robustness respectively
Therefore, for the adversarial input x' with ||x* -x-
x||
In the following definitions, we define three algorithms to obtain adversarially robust functions and compare their adversarial risks.
Definition 3.2 (Adversarial Training with TRADES). We define FAT as
FAr arg min Σ (Js ( (xx)))
1
+ max
(f(x;) F(x)))
F Ti
|[]] -x-xx Ise
where represents the i-th clean training data and yı represents the clean training label.
Remark. Empirically, adversarial training is a very popular method to achieve robustness. It mini- mizes the adversarial risk the training data. Then consider the Bayesian optimal robustness. Let F represent a function class. We assume the response is generated by y F- (x) & with prior distribution F. I ~ PF and & Pg. Denote X E Rn Y R² the training data and training response generated by yı = F. (xx:)
) + Ç. For problems like Bayesian linear regression Bishop & Nasrabadi 2006), function f E F is able to achieve the Bayesian optimal natural risk Ex,Y, (F(x) 3)². However, the function class F is not enough achieve the Bayesian optimal adversarial risk. The adversarial risk depends on the local Lipschitz of function F, in order to better trade-off between the Lipschitz and natural risk of the function F, much more complicated func- tion classes than F are needed to achieve the optimal adversarial robustness. We defined the two Bayesian functions FRB and FAB that minimize global adversarial risk and adversarial risk at the specific point x. FAB extends the function class beyond and achieves better robustness.
Definition 3.3 (Restricted Bayesian Robust Function FRB). The restricted Bayesian robust function FRB minimizes the global adversarial risk inside the function class F
min Ex PEF
(f(x) - y)² + max [x=-x|<<e
(0(x))f(xx))
Remark. The Bayesian function represents the best robust function inside the function class I For any F F. no training algorithms can achieve better average adversarial risk than FRB-
Definition 3.4 (Adaptive Bayesian Robust Function FAR). The adaptive Bayesian robust function FAB inside the function class I that minimizes the adversarial risk at point x is
(f(xx) -0)² + max (f(x*) F(x))2
min Ey x Feg
||x x||Ce
3

---

Under review as a conference paper at ICLR 2023
Remark. Instead of minimizing the global average (F), FAS minimizes the adversarial risk in the given input point x. The function depends on the input x so that the model extends the function class beyond F. For different test inputs, can use different functions to achieve the optimal adversarial risk. Therefore, we refer to FAB as the test-time adapted function.
In the following theorem, show the difference between three functions in the model, where the test-adapted function FAB significantly improves the robustness.
Theorem 3.1 (Linear Models), We consider linear function classes Itin { (FLin FLin 0) x 0.0 E R. The output is generated by y = x 00 + E. where 0,, is independent of x with O. N(0, 12), and the noise & ~ N(0, )²). Let X E ggad Y E R" denote the training data and
the responses respectively. For linear model FLin (x: 0), three estimators in Definition 3.2 to 3.4 are
0// = - X + Ne II,
OLin RB
@hin mar
@ = (xx² + xx
@Lin
= Ed
where 0ui X(X X + ) YY d o² 12/73. Furthermore, if each dimension of is i...d
with Ex- 0, Cor(x) La/d d and dd < . for some universal constant M < co, denoting A = (1 + 2.)²² )² - 4c, then when nt d + co with n/d = =CE (0,
Re (0/m) = 7² 1 1+ A, VA
Ba = 1 (0kz) -2,
2( +1)
And when c + 1 with SNR= o² S 1, Te (O nea (O
2
31 M
Remarks. In this theorem, provide the form of the three estimators and their adversarial risks. The gap of adversarial risk between Okin and OLin vanishes when d - oc. The estimator Okin achieves the optimal robust risk among all linear models. However, for an arbitrary ratio = n/d, (0kin)) < 12 (Okin), , indicating that adaptation to each test data x can improve the robustness of the model even when compared with the best linear model.
Theorem 3.1 provides the optimal test-time adapted estimator in the linear function classes glin which depends on the clean input x. In Figure we plot the adversarial risk of three estimators for different adversarial budgets, which clearly shows that our adaptation the robustness. When the input is corrupted with adversarial noise, the same form of the test-time adapted estimator also significantly improves the adversarial risk shown in the following theorem.
significantly increase
Theorem 3.2 (Corrupted Input). We assume the oracle parameter 0. is independent of x and has the prior distribution 0.. NN(,=²1), and the noise & N(0, Furthermore, each dimension of is i.i.d. with Ex = O, Cov(x) La/d d and Er/d S M for some universal constant M < 00, then when n. d - oc with n/d = c. E (0, 1). Given corrupted input x* = x + =0/ with e < 1, the adversarial risk of ABB, (x*x* + X(X X
+ +,
-1 is
Red AB - (1 (1- c² + (1+E)) (1 (1+c+). + A
28a
2c
Raddy (0/m)
(
Adversarial Risk E 0.10
Remarks. The theorem shows that when the given input is adversarial, the test-time adaptation can still lower the ad- versarial risk of the model as Radv (@Lin) Rady (@lin AB.
1.0
Rady (Qkin).
In the statistical Bayesian model, we show that the test- time adaptation can extend the function classes and
0.4
0//
0%
achieve the significantly lower adversarial risk than the fixed model. In the practical non- non-Bayesian classification task, explicit calculation of the optimal model is difficult. Nev- ertheless, the test-time adaptation also helps to improve the robustness of the model. As will be shown in the following section, we perform the self-supervised test-time fine-tuning to adapt the model to each input, and largely improves the robust accuracy of the test-time adapted model.
0.2
0%
0.44595
203570
203620
50000 100000
150000
200000 250000
Figure The comparison of Radv for @ni Ozig. 0zig. Osig.. We set 72 = 1, 0² = 0.2 and d = 250000.
4

---

Under review as a conference paper at ICLR 2023
4 METHODOLOGY
We follow the traditional multitask learning formulation (Caruana 1997) and consider a neural net- work with a backbone {x; 0g) and K + heads. One head f2; f outputs the classification result while the other K heads 91 (z; 011 ) ..., 9K (z; 8 0yk) correspond to auxiliary self-supervised
tasks, (0E, Oz Og., 000x) encompasses all trainable parameters, and we further define
F F= G2 g E, 1,2....
Furthermore, let D gt) denote the training set, and D {(2,, (+), Ti=1 be the test set. For further illustration, the labels the test set are shown. However, they are unknown to the networks at test time. We denote the adversarial examples of x as x". It satisfies ||x -xll S E, and e is the size of the adversarial budget. For any set S, we represent its average loss as
L(S)
Σ ((s)) S
1
where |S] is the number of elements in S. The general classification loss, such as the cross-entropy, is denoted by Ceis. We use the superscript "AT" to denote the adversarial training loss. For example,
ATT =
Σ
max Sc
Lata(F(x)
S
||x
X..8t
ES
4.1 SELF-SUPERVISED TEST-TIME FINE-TUNING
Our goal is to perform self-supervised learning on the test examples to mitigate the overfitting problem of AT and adapt the model for each data point. To this end, let us suppose that an adversarially-traiee network with parameters O receives mini-batch of b adversarial test ex- not available, propose to fine-tune the backbone parameters 0g by optimizing the loss function
amples B- ()), (8%, }6)} As the labels {i% )
Ass Lss - Co Cx E (Gx( Ogk),
-
which encompasses self-supervised tasks. Here, Lss,k represents the loss function of the k-th task and (C+) are the weights balancing the contribution of each task. our experiments, the Kss,K is the cross-entropy loss to predict rotation and vertical flip.
The number of images b may vary from 1 to m. = corresponds to the online setting, where only one adversarial image is available at time, and the backbone parameters Og are adapted to every new image. The online setting is the most practical one, as it does not make any assumptions about the number of adversarial test images the network receives. By contrast, b
corresponds to the offline setting, where all adversarial test examples are available at once. It is similar to transductive learning (Gammerman al. 1998 Vapnik 2013). Note that online setting differs from the online test-time training described in TTT (Sun et al. 2020);; we do not incrementally update the network parameters as new samples come, but instead initialize fine-tuning from the same starting point 8 for each new test image.
Eqn (4) encourages Og to update in favor of the self-supervised tasks. However, the classification head was only optimized for the old backbone E(:; 0%)). ). it will typically be ill-adapted to the new parameters 0% resulting
degraded robust accuracy. Furthermore, for a small the model tends to overfit to the test data, reducing Lss to 0 but extracting features that are only useful for the self-supervised tasks. To overcome these problems, we add an additional loss function acting on the training data that both regularizes the backbone E and optimizes the classification head f so that remains adapted to the fine-tuned backbone E(: Specifically, let D denote a subset of the training set. We then add the regularizer
(R) C
AT (B)
Σ X.BIEB
max |xx -x
Lesa (F()), v:
(5)
to the fine-tuning process. In short, Eqn (5) evaluates the AT loss on the training set to fine-tune the parameters O, of the classification head. also forces the backbone F. to extract features that can
5

---

Under review as a conference paper at ICLR 2023
be used to make correct predictions, ie.. to prevent 0g from being misled by Las when b is small.
Combining Equ and Equ ( our final test-time adaptation loss is
Leest B) = Lss(B*) (B)
(6)
where C' sets the influence of R. The algorithms that describe our test-time self-supervised learning are deferred to Appendix [] As SGD is more efficient for larger amount of data, we use SGD to optimize when b is large (e.g. offline setting).
4.2 META ADVERSARIAL TRAINING
To make the best out of optimizing Lrest at test time, we should find suitable starting point 0", ie., a starting point such that test-time self-supervised learning yields better robust accuracy. We translate this into meta learning scheme, which entails a bilevel optimization problem.
Specifically, we divide the training data into small exclusive subsets = = and let B3 to be adversaries of Bj. We then formulate meta adversarial learning as the bilevel minimization of
meta (D:0) =
CAT (B); where 0 = min ( BjCD
Σ
(7)
where Lss is the self-supervised loss function defined in Equ and & A is the loss function of AT defined in Eqn (3). As bilevel optimization is time-consuming following MAML (Finn et al.,
2017 we use a single gradient step of the current model parameters l to approximate 0z.
0j ~ VoLss((B:0)
In essence, this Meta Adversarial Training (MAT) scheme searches for starting point such that fine-tuning with Ess will lead to good robust accuracy. this holds for all training subsets, then we expect the robust accuracy after fine-tuning at test time also to increase. Note that, because the meta learning objective Eqn 7 already accounts for classification accuracy, the regularization by R is not needed during meta adversarial learning.
Accelerating Training. To compute the gradient VoLmeta (D:0) 0), we need to calculate the time- consuming second order derivatives -a V2 [B;: 0) V 0; LAT (By; 0j) Considering that AT is already much slower than standard training (Shafahi al.) 2019),
cannot afford another signif- icant training overhead. Fortunately, as shown in (Finn et al. (017), second order derivatives have little influence the performance of MAML. We therefore ignore them and take the gradient be
(DF ~
Σ B,CD
Vor CT (By; (Bz;07)
However, by ignoring the second order gradient, only the parameters on the forward path of the classifier F, F.e.e 0g and f. will be updated. In other words, optimizing Eqn (7) in this fashion will update {0 1 k-1- To nonetheless encourage each self superviee head G3 output the correct prediction, we incorporate an additional loss function encoding the self-supervised tasks,
UST( Σ CRUST (D) - Σ C
max -x
(ss,
(10)
|x
Note that which, as shown in (Chen et al. [2020a] Hendrycks et al. 2219:) Yang & Vondrick 2020), is beneficial for the classifier. The final meta adversarial learning objective therefore
the adversarial version of Las to provide robustness the self-supervised tasks,
Lirain (D) &mota (D) + C' A (D)
(11)
where C' balances the two losses. Algorithm shows the complete MAT algorithm.
5 EXPERIMENTS
Experimental Settings. Following previous works (Cui et al. 2020 Huang et al. (020).. we con- sider loc- -orm attacks with an adversarial budget 0.031 8/255). We evaluate method

---

Under review as a conference paper at ICLR 2023
Algorithm Meta Adversarial Training
Input: Training set D; Learning rate a, 9; Iterations T; Weights Ck and C' Output: Starting parameters 0g for the test-time fine-tuning
1: for :1 to 7 do
2: Sample exclusive batches of training images B1, B2, BB C.D
3: Using PGD to find the adversaries B3: x5,x = arg max -s, Rots (x(2,)) v5,))
4: for batches B1, B2_ ... B, do
5:
0j 0 a o (B)
6: Imeta.j CA4 [By: 00)
7: end for
8: 0 = 0 EB, motaa C' V
9: end for
return Trained parameters 00
on three datasets: CIFARIO (Krizhevsky et al. 2009), STL10 (Coates et al. 2011) and Tiny Im-
ageNet (Le & Yang 2015). We also use two different network architectures: WideResNet-34.
10 (Zagoruyko & Komodakis 2016 for CIFARIO, and ResNet18 (He et al., 2016) for STL10 and
Tiny ImageNet. The hyperparameters are provided in the Appendix D
Self-Supervised Tasks. In principle, any self-supervised tasks be used for test-time fine-tuning, as long as they are positively correlated with the robust accuracy. However, for the test-time fine- tuning to remain efficient, aim to support the fully online setting, where only one image is available at a time, we cannot incorporate contrastive loss (Chen et al., 2020b; He al., 2020 Kim al.] (020)) to Lss- In our experiments, we therefore use two self-supervieed tasks that have been shown to be useful to improve the classification accuracy: Rotation Prediction and Vertical Flip Prediction.
should not too many self-supervised tasks. Furthermore,
Attack Methods. In the white-box attacks, the attacker knows every detail of the defense method. Therefore, we need assume that the attacker is aware of our test-time adaptation method and will adjust its strategy for generating adversarial examples accordingly. Suppose that the attacker is fully aware of the hyperparameters for test-time adaptation. Then, finding adversaries D* of the clean subset B can be achieved by maximizing the adaptive loss
arg max Lattack (F(x;) yr 0" O'""))) | -x
(12)
where Lettaac refers the general attack loss, such the cross- cross-entropy the difference of logit ratio (DLR) (Croce & Hein 2020a). We call this objective in Equ (12) adaptive attack, which can be either performed in white-bon box black-box attacks. We consider four common white-box and black-box attack methods: PGD-20 (Madry et al. 2018), AutoPGD (both cross-entropy and DLR loss) loss (Croce& & Hein 2020a). FAB (Croce & Hein| 2020b) and Square Attack (Andriushchenko ctt al. 2020). We apply both the standard and adaptive versions of these methods. Particularly, PuDoPGD we use is a strong version that maximizes the loss function that continues when finding adversarial examples (Croce et al., (022). More details are provided in the Appendix E
Baselines. We compare our method with the following methods: 1) Regular AT, which uses AA in Eqn B.. Regular AT with additional self-supervised loss, i.e., using LATT' + C' & for AT, where LAT given in Eqn (10). This corresponds to the formulation of (Hendrycks et al. 2019 3) MAT (Algorithm T without test-time fine-tuning.
5.1 ROBUST ACCURACY
CIFARIO. Table La shows the robust accuracy for different attacks and using two different tasks for fine-tuning. The adaptive attack is not applicable to models without fine-tuning. As we inject dif- ferent self-supervised tasks into the AT stage, and as different self-supervised tasks may impact the robust accuracy differently (Chen et al. 2020a),, the robust accuracy without fine-tuning still varies. The vertical flipping task yields better robust accuracy before fine-tuning but its improvement after fine-tuning is small. By contrast, rotation prediction achieves low robust accuracy before fine-tuning.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Robust accuracy on CIFARIO, STLI0 and Tiny ImageNet of the test-time fine-tuning on
both the online and the offinn settings. We use an foc budget & = 0.031. FT stands for fine-tuning We underline the accuracy of the strongest attack and highlight the highest accuracy among them.
(a) CIFARIO with WideRessNet33-10.
Square Attack
PGD-20
AutoPGI
FAB
Tasks Methods
Worst
Standard Standard
GMSA Standard
None
AT
62.51%
55.74%
52.14%
51.34%
$1.30%
AT w/o FT 63.54%
56.64%
52.57%
51.87%
51.85%
MAT w/o FT 63.96%
57.35%
53.09%
53.09%
53.04%
Rotation
Online FT 65.52%
65.85%
59.52%
59.50%
57.93%
56.96%
57.60%
75.58%
77.69%
56.62%
Offline FT 67.05% 65.75%
61.17% 59.71%
58.77%
57.63%
78. 18%
68. 68.60%
57.21 %
AT w/o FT 62.09%
55.50%
52.79%
51.24%
51.23%
MAT w/o FT 65%
59.73%
53.41%
53.02%
52.98%
VFlip
Online FT 66.91%
66. 16%
61.47%
59.40%
58.74%
56.799 79%
58.53%
75.68%
80.57%
55.98%
Offline FT 67.23%
65.60%
61.82% 59.69%
59.26%
58.06%
75.60% 72.24%
57.01 %
AT w/o FT 65.64%
59.19%
53.16%
53.05%
52.95%
Rola w/o FT 65.75%
59.51%
53.99%
53.85%
53.76%
VFlip Online FT 67.34%
66.80%
61.79%
60.46%
59.23%
57.70%
59.60%
76.39%
79.80 79.80%
57.21 %%
Offline FT 68.50%
66.05%
62.87%
60.54% 60.25% 58.26%
76.89%
71.58%
57.88% %
(b) STLIO with ResNet18.
Square Attack
PGD-20
AutoPGD
FAB
Tasks
Methods
Worst
Standard Adaptive Standard Adaptive Standard Adaptive Standard Adaptive
None AT
44.83%
37.89%
35.78%
35,64%
35.58%
AT w/o 44.00%
36.92%
33.72%
33.73%
33.65%
Rotation
MAT w/o FT 44.75%
38.66%
35.60%
35.38%
35.31%
+
VFlip
Online FT
45.07%
46. 19%
40.31
40.24%
39.53%
40.85%
51.25%
51.08%
39.21%
Offline FT
47.86%
48.03%
45.21%
43.33%
43.78%
43.20%
58.49%
54.13%
42.57%
(c) Tiny ImageNet with ResNet18.
Square Attack
PGD-20
AutoPGD
FAB
Tasks
Methods
Worst
Standard Adaptive Standard Adaptive Standard Adaptive Standard Adaptive
None AT
28.5%
20.6%
17.5%
17.2%
17.2%
AT w/o FT 29.5%
22.2%
17.1%
16.7%
16.7%
Rotation MAT w/o FT 29.3%
23.1%
16.9%
16.8%
16.7%
VFlip Online FT 30.2%
30.2%
24.0%
23.2%
18.9%
18.1%
33.7%
31.6%
17.7%
Offline FT 32.4%
31.0%
25.6%
24.1%
23.7%
20.6%
36.5%
27.7%
20.1 %
Table 2: The statistics p(X*) and two self-supervised tasks. The dataset is CI- FAR10 and the network is WideResNet- 34-10. Adversarial budget E = 0.031
Rotation
Vertical Rli
1.0
1.5
06-
0.-
02
02-
Tasks B(p(x*)) P(p(x*) > 0)
-2
-25
Rotation 0.15
68.51%
Figure 2: Empirical edf of p(X;)
) on CIFAR10 and
VFlip 0.22
72.16%
W4deRes Adversarial budget 0.031
but its improvement after fine-tuning is the largest. Using both tasks together combines their effect and yields the highest overall accuracy after test-time adaptation. Note that our self-supervised test- time fine-tuning, together with meta adversarial learning, consistently improves the robust accuracy under different attack methods. Under the strongest adaptive AutoPGD, test-time fine-tuning using both tasks achieves robust accuracy of 57.70%, significantly outperforming regular AT.
STL10 and Tiny ImageNet. As using both the rotation and vertical flip prediction led to the highest overall accuracy CIFARIO, 10, we focus this strategy for STL10 and Tiny ImageNet. Table Tb and Ic shows the robust accuracy on STL10 and Tiny ImageNet using a ResNet18. 8. Our approach also significantly outperforms regular AT on these datasets.
Offline Test-time Adapattion. As shown in Table Ia Ib Ic the robust accuracy over the online version.
the offline fine-tuning further improves
8

---

Under review as a conference paper at ICLR 2023
Diverse Attacks. Recommended by (Croce et al. 2022), in Appendix C.1
evaluate method
on diverse attacks including transfer attack, expectation attack and boundary attack, where test-time adaptation all improves the robustness of the model.
5.2 METHOD ANALYSIS
We observe the significant positive correlation between the gradient of self supervvse loss L and the classification loss Lels. Define
V D Css 1 Cel Dil Ve, (
p(x;)
(13)
and approximate Lds by the Taylor expansion (X De: 0g (os((ss())))
0E) V Cos
As OF contains millions of parameters, its gradient descent w.r.t. Lss should act as a good substitute for optimizing Lds when p(x;) is significantly larger than O. We further confirm this empirically. For all adversarial test inputs x* ~ D* regard p(x*) as random variable and calculate its empirical statisties on the test set. Table 2 shows the empirical statistics of
is typically large. Therefore, gradient
model on CIFAR1O. 10, and Figure shows the c.d.f. of p(X*). The mean of p(x*) ) is indeed significantly larger than O and P(p(x*) > 0) is larger than the robust accuuray of the ddderssrialll-trainee network (50%-60%), which implies that self-supervised test-time fine-tuning helps to correctly classify the adversarial test images.
We further provide the theoretical analysis in linear model in Theorem B.1 which shows that the correlated gradient significantly strengthens the robustness and lowers natural risk. Besides, the correlated gradient also helps the model to move closer to the Bayesian robust estimator OAB
5.3 ABLATION STUDY
Meta Adversarial Training. To show the effectiveness of MAT, we perform an ablation study to fine-tune the model with regular AT (i.e., setting = 0 in line 5 of Algorithm Table7/shone that the robust accuracy and the improvements of fine-tuning are consistently worse without MAT.
Accuracy Improvement Inputs with Different Adversarial Budget. As shown in Table X set & = 0.015 to perform the online test-time fine-tuning, showing that our method is also able to improve the robust accuracy of inputs with different adversarial budgets.
Lss or LR To study the effect of Lss and L in Ltest. we report the robust accuracy after online fine-tuning using only L and Lss in Table While, as expected, removing L tends to reduce more accuracy than removing &R. It shows the benefits of our self-supervised test-time fine-tuning strategy. Nevertheless, the best results are obtained by exploiting both loss terms.
Removing
Improvement on Clean Images. As predicted by Theorem 3.1 and B.I our method is able to shown in Table IO our
improve not only the robust accuracy also the natural accuracy. approach increases the clean image accuracy by test-time adaptation. This phenomenon further strengthens our conjecture that the improvement of robust accuracy is due the improvement of generalization instead of gradient masking.
CONCLUSION
In linear models and two-layer random networks, we theoretically demonstrate the necessity of test- time adaptation for the model to achieve optimal robustness. To this end, we propose self-supervised test-time fine-tuning on adversarially-o seeersarially-trainee -trinned models to improve their generalization ability. Further- more, we introduce a MAT strategy to find a good starting point for our self-supervised fine-tuning process. Our extensive experiments on CIFAR10, STL10 and Tiny ImageNet demonstrate that our method consistently improves the robust accuracy under different attack strategies, including strong adaptive attacks where the attacker is aware of test-time adaptation technique. In these periments, we utilize three different sources of self-supervision: rotation prediction, vertical flip prediction and the ensemble of them.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense curity; Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283, PMLR, 2018.
Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Jiefeng Chen, Xi Wu, Yang Guo, Yingyu Liang, and Somesh Jha. Towards evaluating the robustness of neural networks learned by transduction. In International Conference on Learning Represen- tations, 2021a.
Jinghui Chen and Quanquan Gu. Rays: A ray searching method for hard-label adversarial attack. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1739-1747, 2020.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustmess: From self-supervised pre-training to fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 699-708, 2020a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang. and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 3b. URL
wwt http://iww....eeooooomooaaaaaa/e
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMIR, 2020b.
Chen Cheng, John Duchi, and Rohith Kuditipudi. Memorize to generalize: on the necessity of interpolation in high dimensional linear regression. arXiv preprint arXiv:2202.09889, 2022.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-bas based image inpainting. IEEE Transactions on image processing, 120001220-11 2004.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fres free attacks. In International Conference on Machine Learning, pp. 2206- 2216. PMLR, 2020a.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference on Machine Learning, pp. 2196-2205. PMLR, 2020b.
Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. arXiv preprint arXiv:2202.13711, 2022.
Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia. Learnable boundary guided adversarial training. arXiv preprint 1..11:20 2020.
10

---

Under review as a conference paper at ICLR 2023
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1135. PMLR, 2017.
A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. In Proceedings the Fourteenth the Conference on Uncertainty in Artificial Intelligence, UAI'98, pp. 148-155, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL net/ forum
htttps://openrewe
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018. URL httptttt////eeeeet
net. wtt.://wweee.rtoeeeeeeeemee wtt.://orummmoommm dws.mmmdddSSSyySccS
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learn- ing can improve model robustness and uncertainty. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL http:: htt.t:::s/p/oocceei.ii neerips. cc wc.ppppr//// 2019 9/ff/1111 htd/22bb55555 7edac1 5df 90 9072191899261771118
pdf
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. In Advances in Neural Information Processing Systems, volume 33, 2020.
Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 81 199-881 2021.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. In Advances Neural Information Processing Systems, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)33 2015.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018.
Behnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nati Srebro. Exploring gener- alization in deep learning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan. and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpttststt//pptttteaee httpttpppp/pppccceeeieee caper/2017/F11
www.:iinnnnnnuuuuuueeee neurips CC/

---

Under review as a conference paper at ICLR 2023
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-81000 8104. PMLR, 2020.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin
(eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16282-16292. Cur- ran Associates, Inc., 2020. URL / www.riiiiinnnnnnnn.............e http://pppp.cccppppp///0000000 oetpaa/eeeeeeeeeeeeeeeeeeeeaaaaa Paper.
fil
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 13505acc223300066eeceee222222222
pdf
Changhao Shi, Chester Holtz, and Gal Mishne. Online adversarial purification based on self- supervised learning. In International Conference on Learning Representations, 2020.
Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3118-316,, 2018.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In Interna- tional Conference on Learning Representations, 2018. URL http::tt/////ct /ttpp//eee/per...e.... net/
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning. pp. 9229-9248 PMLR, 2020.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry, On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems, volume 33, 2020.
Trieu Trinh, Minh-Thang Luong, and Quoc Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint
2019.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference Learning Repre- sentations, 2021. URL
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2019.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in Neural Information Processing Systems, 33, 2020.
Junfeng Yang and Carl Vondrick. Multitask learning strengthens adversarial robustness. In European Conference on Computer Vision. Springer, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (cds.). ). Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.122 BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C. 30.87. URL htttl:://///// //t::///dxxxoo doi. ww..orgg//0/ 5204/2444232 30.87
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
12

---

Under review as a conference paper at ICLR 2023
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference Machine Learning, pp. 7472-7482. PMLR, 2019.
13

---

Under review as a conference paper at ICLR 2023
A PROOFS OF THEOREMS
A. PRELIMINARY: MARCHEENKO-PASTI LAW AND TRANSFORMATION OF EIGENVALUES
Before the proof linear models, we first give the asymptotic spectrum of the matrix K = X X, and some useful results of the trace of the transformation of K.
Lemma A.1 (Marchenko-Paste Pastur Law, Theorem 3.4 in (Bai & Silverstein 2010). We define the eigenvalues of X X: A > A2 ... Xn has distribution with c.d.f.
(s)
1=1
each dimension of i...d with Ex = 0, Cov(x) Ia/d and B B[vdx' S M for universal constant M < oc, then when n, + oc with n/d = c E (0, (0,1), for any bounded function 9. when n, d + 20 with n/dd [0, 00)
f g(0)d(lm(s) + s(s)dn((s).
with p.d.f. (s)
v(+ -8) (s
dH(ss) 2
1 LE 221ds,
where x_ (1- and X+ (1 + v
Lemma A.2. If each dimension of x is i.i.d with Ex = Cov(x) Ls/d and E[Vdr']* s M for some universal constant M < JO, then when n,, d with n/d E (0,
1 + VA
Tr(K(K + AI,)
2VA
Tr(K2(K + (I,) = (1 c A) VA
2
VA
Tr((K + XIn) d (1+c)(1 +c+ c A) 4e - (1-
2
where = (1+ 4c.
Proof. We define three transformations of the eigenvalue of dH (s)
(1(z)=/ ( (H)), 62(===/ - (s (s+2) dH()), la(=)= / (s+2p dH)),, (14)
s²
for E oc). They can be calculated by the Stieltjes transformation of Marchenko-Pastuu law.
By Marchenko- Pasttur semicircular law, the Stieltjes transformation of d.H is (Lemma 3.11 in
(Bai & Silverstein 2010) and Lemma 4.4 in (Cheng et al. 2022))
m(2)=
/
dH(s)
c-
(1 + 2) 4c
2cz
Let A (1 c1 2)²
((x) = m((=)= 1 v (1 2)2 4e
-2c2
and
/ (8 18+
((ti))))((s) =
ddd(ss)
dz /
d
1
dH
= dt(2)
5
dz
14

---

Under review as a conference paper at ICLR 2023
Then
t2(2) = 5+2
(s) - -f
(s (s+ dH(s) t(z) + dl(z)
2
1 2- - VA
(15)
dz
20v A
(=(2) =1 - /
2z
dH((s) + /
dH(()))s (8+2)2 1 221(2) 22)(2)
2²
8 8+2
(8+
dz
(16)
(1+0+2 2)
VA
(1
2c
V
d
/
1
dl(z)
[1+ (1+ - 4c (1-clv4
tz(2) =
dH
(17)
dz
+
dz
2e
The trace operation can be translated into:
Tr(((K + 22 =n
(8 + X)2 ds)
Tr(K + 2) = /
52
(s A)2 dH)
Tr((K + (I,) (s 1)2 (s).
Therefore, when n, d + with c == n/d E (0.1),
Tr(K(K + +AI) -1 1+c+ VA
1+c+ VA
20v
2VA
(1+c+ A) -V
VA
A ) dd
(1 (1+
- VA
A
Tr(K + - +n
(1-
(1 VA
)
2c
VA
2
(1 (1+c) A) 1)-48
4c c)vA
Tr((K + ALn) -n
20
(1 (1+c)(++c++ A) 4c 10- (1- -1-
2 A
where A = =1++ A)2 4c.
A.2 PROOF OF THEOREM
We separate the proof into three parts for three estimators separately.
Proof of 0l. For @AT taking gradient with respect to the objective function
[1 x, 0) xTO)
+ +
n
obtain
- YY - x 0) + 2n = 0.
Therefore,
in AT
= x(X X + nE I Y
Its natural risk is
1
R 0 .in
Eg.o. x(x (0 in 0,))
/
LL
0.12
AT
1 E..o0. 10. 10--X(X- x + - 0, + 12
1
(18)
2272
0²
=7²(1 - 11
) + Tr((K + (I
Tr(K + 2I
d
15

---

Under review as a conference paper at ICLR 2023
And the Lipschitz constant is
)( =100
-1E X TO 0. + E)I
(19)
Tr (KK AI,.) - + o Tr (KK AI,)
When A = no2 oc, by Lemma A.2
Tr(K(K+ + +II,) 0, Tr(K2 (K + 2 *TT(((K 1I,)
(20)
Therefore,
R (Oh)) = Rxx (0 + + ((Ok(( +
(21)
Lemma A.3. If the oracle parameter 0. is independent of and has the prior distribution O. N(, 7 T)) and the noise & ~ N(0, o²). Furthermore, if we assume each dimension of x is i.i.d.
with Ex = 0, Cov(x) = Ia/d and E[ E[vāx')* I M for some universal constant M < co, when (0,1), 1), then for X. 0²2 /72 and O 1 X.I,,,Y
n, d -> 0o with n/d
0²2/²²
7²
R ( 72
((1 +c - VD)(1 2.A"
A
d7
2
((1 [c +.)) A.) -VD),
2
where = (1+ 4c.
Proof.
1 XX + X 0. +
Raall (
BB.o. B.....1.. 11.
A
Th
1 (
Ia
to11) +x(K+Aon)11
+
0²
( + (.I
L
d
(22)
72
2n.
27
?2
(d
Tr (KK + I.II))
Tr +
d
A
dA
dA
a²
Tr
(KK
+ A.I..) -k)_
dA2
According to Lemma
A.2
when n, d with (0,1),
1 1-c+ X, VA
Tr((K A.I,)
-2h,
1 1+c+ VA
((K((+ALI))
2/A
(1 + +++c+ Aw) VA
d
(K
where X, = =2//72 and = (1+c+ 2.)2- 2-c. Therefore,
Rhat @ Piiyyes) =² 72
(1+c+t)) VA) 1
2A"
And
1
1(00) = =Eg.o. ||
0 +11
7- Tr X.IIIIII))) -.,,K+XX,--x'
0² Tr (1
XL +XI,)
(23)
A?
1²
A?
72
Tr (K*(K
(K+ +1 +
Tr (KK + X.I
A2
A2
16

---

Under review as a conference paper at ICLR 2023
Therefore,
dr² L(
2.42 +c (1+ VD).
(24)
Proof of Ozin. is well known that the posterior distribution of 0.
0 X, Y N(X X A.I,.) ),I
)m 1)
where o² /72 According the definition of Bayesian estimator,
Olin arg min Eg, xE ((x (x 0.)) =
As the linear model only allows fixed for each x, we obtain
Eo,1x (210- 0x11 + ==1001 =
(
- 1 0. + +=110112
d0
d0
d
(25)
=Eg_1x IX
0 0.) + 26 = 0.
d
Therefore,
ØLin
Rg 00
X(X X I.I
c2d+1 1
edd +
Using Lemma A.3 when
00
2e +
Rx
RB
=T
82 +
(1+ A.) -.) VA)
2 +1) + +73
(26)
d7²
@@Lin L L(kin
(1+c+ Xx) vA +
RB
2(
1)2
Summarizing the results,
Raall
Okin)
1xx (Oki) + +s ²
(27)
Proof of Ql. Bayesian robust estimator of each x, which optimizes (x (x- 0.))² + =
(x- L arg min E,,1x (xx - 0.)) + =²11013).
where it is well known that the posterior distribution of 0, is
0. X, Y N(X
A.I.)) X (XX A.I.))-)).
Taking the gradient w.r.t gives the solution
X(X X X Liin xx) [xxx] + 1
(28)
Xx
LLin
22
where Lin X + A.I.. -1 X 0. + E) with X, 02/ is the Bayesian estimator for natural risk.
For its natural risk Reaa @Lin) (
R (0lim)
Eg,,eExxx (0 0.))²
As
x 0
x xx
Olin
= x T
x x
0lin .in
2
+ x
nat
c
+
x
nat
17

---

Under review as a conference paper at ICLR 2023
then
x x
R AB
0
Eg. Ex
Ouin 0.))².
E2 + x x
When d + 00, xxx in probability. Therefore,
x X
(29)
62 +
1 1
Then when d - oo,
R (hin)
Eo. &
ØLi
02
c2 1
By Lemma A.3
72
1
Rix
(0)m)
72
(1+ - VAI(1
(30)
E2 +1
2 + 1)
As x x + in probability when d + oc,
x x
7 (##+1)2 Then
(c=+xTx)²
L(O
Eo. ||00iin
(31)
d(E2 + 1)2
From Lemma A.3
Ep. in 1 d7 (1+ + No) - VA
2
Summarizing two parts, the adversarial risk is
Rea .i =7² 72
2(e² + 1) ((1 (1+ A.) 1.)- VA) s 5
c
(s2 + 1)(1 +c+
A.3 PROOF OF THEOREM 3.2
Proof. As Llin
xx @ + the adversarial input
in AB
X
x +
10
|||||
AB
As d + oc, ||||| + 1 in probability. Therefore, x* E)x. Taking it into
AB = xx*x*T + ee
x x A.I.))
obtain
(1+ 2
OLin
nail
AB,
+ (1 + 8) 5) x
From Eqn
30 and 31
nnnat 0
AB. = 1
+ ++ vV))
(1 61+
=1++
(1+ -VD)
2( + (1+
(1)
L ) )2
(1++ VA).
2( + (1+
Therefore,
2 1 1+ VA
pady @Lin (l),,) =R (ii,,)) + e2 LL @Lin 1 y² 72 1- 11-(1-
AB,
(1 2( (1 +52 +
18

---

Under review as a conference paper at ICLR 2023
B CORRELATED GRADIENTS
In the following Theorem, we show that with correlated gradient, one gradient descent step like our method largely improves the natural and adversarial risk of the model.
Theorem B.1. We assume the oracle parameter O, is independent of and has the prior distribution 0. N(0, T)), and the noise & N(O, o2). Let 0 OAT be the estimator of adversarial training of the linear model FLin(x; (x;0) 0):
TO = X(X X + nE I.).
When receiving new test data point (x*, and taking one gradient descent step with cor- is an adversarial example of x. Corr(g, Vo ( (x, 0¹), u)) = p> 0 and rj is the learning rate. Let x = ( .xx)) where x² the i-th element of x. We further assume that {x Ji=1 are i..d with E[r"] = 0, Var[z'] = 1/d. And Blvdx' S. M for some universal constant M <0. When d + co with c = n/d E (0, 1), with the optimal learning rate,
related gradient g: 01 0 1/8 where ||x- -xl1
r2p² + + +))()+))3))
Rual (8)) Ra (01) 2 [[(1 + 8)2 (
2T2p²
((1 +)² + 0² +0² (1+ (1+}} +5)²
((1 (1+ + 72
Ruchy (0)
Badly (0') 2
+))
Remarks. With correlated gradients, improvements R and Rod are both positive when having p O. By taking correlated gradient descent on the parameter, we get large improvements of both natural and adversarial risks.
Theorem B.I shows that fine-tuning with correlated gradient largely improves both clean perfor- mance and robustness of the models. In addition, for linear models, the Bayesian optimal estimator is Ohin x. And 0² g with Corr(x", g) p. As x is close to with proper learning rate, can get close to Bayesian robust estimator with correlated gradient descent.
B.1 PROOF OF THEOREM B.1
Proof. For a new test input (x*, v), where ||x x|| S is an adversarial example near the input, its MSE loss is
1 _ (xx TT. 0. + T0)2
L [( y. 0)
Taking the gradient w.r.t
L y/, 0) = (xxT x 0, E)x²*
Suppose the self- supervised task gives a correlated version of gradient and updates 0 with one step of gradient descent
0 00 v 00 T 0. g)
where Corr(g, (g, x*) P and Eg T gl = E[x From Theorem 3.1 when ne² - co.
Rmmal (0)) and |00112 can be simplified as:
Ee. 1 -2 +c) +
1 + ol 7 + 0, R ( = Eg,.. -0,1 + +
s4d
Therefore, when d + x
θ + 11[(x* T 0. + E)
Then when + xc.
Ex 0 1|2 2 R gx xxxxxx + 22 [g + 2n Ex Fx [xx].
19

---

Under review as a conference paper at ICLR 2023
Therefore,
xx.0..1100) -1² (r'Exl8" [2"xxl"'xxTT
"x""x""'"] xx1+xx+++ o ox gl)
By decomposing g = px + v1 - -2.,, we can obtain,
Ex 110 112 =1²² (T (+)) + 0xxxxx xx(]xx
As ||x* x|| S.
Ex Ex.00.110011 Sn (T+ +
+ + =))²)
For natural risk
Raal (0)) = _. 0. ||x (0. - 02
2 + - [x* x (x 9) B
[(x [(xx))(] 2n X
x'x g).
By |xx - x|| S E, Corr xxxx = and Elg = xx²
R ( - 2) 0 + e)² - 21 - =E)a
Therefore,
Ra )+ + + 02 (1++2) (1+ 22/ p(1 - 5)2.
Optimizing get
pr (1
1/ (+ + 02 8 (1
(32)
With 17..
p1 c)2
Rady 00 S r²(
(33)
((1+)) + 02/ (1
and
720²
2-
R 00) ST²
()3 + 02/ (1 + +))
((1+ +) + 0² 022 +
(34)
Compared with @o
R (0') -
1 (") O 72,
we have improvements of
( Reat 00)
(((1 2) 02 T²p² p²
2r2p²
(35)
+ 0² (1,1))++02//111033((++
and
72p² e)²
( Radly (0
(36)
(1+ =
C ADDITIONAL EXPERIMENTS
C.I DIVERSE ATTACKS
Transfer Attack. Table 3 we perform transfer attack from the static adversarial defense. We use the robust networks with the same architecture the substitute model, and the test-time adaptation also improves the robust accuracy.
Expectation Attack. In Table we show the results of the expectation attack. We modify the adap- tive attack and average the gradient from 10 fine-tuned models, whose training batches are different. We evaluate the model using the ensemble of rotation and vertical flip as the self-supervised task on CIFAR 10. We evaluate the model with Adaptive-Autoo Auto-GDttottattt EOT and Adaptive-SquareeAteee Square EOT. One is the strongest attack in our method and the other is black-box attack that is less likely to be
20

---

Under review as a conference paper at ICLR 2023
Table 3: Accuracy on transfer attack on CIFARIO.
Methods
Rotation
VFlip
Rotation + VFlip
Without FT
84.77%
86.55% 87.00
86.36%
Online FT
86.10%
87.10%
Table Accuracy on expectation attack on CIFAR1O with the ensemble of rotation and vertical flip task.
Attacks
w/o Finc-tuning
w/ Fine-tuning
Adaptive-AutoPGD
53.99%
57.70%
Adapiive-AuoPiD--EOT
53.99%
57.65%
Adapivv-SSquareAttaek
65.75%
66.80%
Adapiie-SquarreAttcckFFoT
65.75%
66.73%
affected by gradient masking. The experiment shows that the expectation attack has little influence on the improvement of our test-time adaptation.
Boundary Attack. We use one of the SOTA decision-based attacks: RayS (Chen & Gu 2020). We test it CIFARIO with the ensemble of rotation and vertical flip. Table 5 shows that method also improves the robust accuracy of the decision-basee attack.
GMSA (Chen et al. 2021a with AutoPGD. GMSA is recently proposed attack algorithm targeted at the test-time model adaptation. We use the GMSA with AutoPGD to attack our method, and the results shown in Table 6 Under GMSA, our test-time adaptation still significantly improves the robust accuracy. Moreover, Talso demonstrates the strength of our adaptive attack strategy as it achieves higher success rate than GMSA.
C.2 ABLATION STUDY
Meta Adversarial Training. Our meta training strategy in A aart strengthen the cor- relation between the self-supervised tasks and classification. To show its effectiveness, we perform an ablation study where we fine-tune the model with regular AT (i.e., setting a in line 5 of Algorithm [] We then perform the same test-time fine-tuning on the model without MAT, using the same hyperparameters as in the MAT case. As shown Table 7 the robust accuracy and the improvements of fine-tuning are consistently worse without MAT.
Accuracy Improvement on Inputs with Different Adversarial Budget. Our method is also able to improve the robust accuracy of inputs with different adversarial budgets. As shown in Table B we set for budget of the adversarial inputs to be 0.015 to perform the online test-time fine-tuning. The robust accuracy is ed improved.
Removing Lss or RR. In our previous experiments, test-time fine-tuning was achieved using a combination of two loss functions: Lss and & To study the effect of each of these terms sep- arately, we remove either one them from Ltest- In Table 9 we report the robust accuracy after online fine-tuning using only La and only Las. While, as expected, removing Lss tends to reduce more accuracy than removing R. It shows the benefits of our self-supervised test-time fine-tuning strategy. Nevertheless, the best results are obtained by exploiting both loss terms.
Accuracy Improvement on Clean Images. As shown in Eqn (30 and Theorem B.1 our method able improve not only the robust accuracy but also the natural accuracy of clean images on adversarially-traiee models. To evidence this, we maintain all the components of our model and simply replace the adversarial input images with clean images (Lee. replacing B* with clean inputs B in Algorithm and perform the same self-supervised test-time fine-tuning.
As shown in Table 10 our approach increases the clean image accuracy. This phenomenon further strengthens conjecture that the improvement of robust accuracy is due to the improvement of generalization instead of perturbing the model parameters, because randomly perturbing the param- eters usually lowers the natural accuracy of the model.
21

---

Under review as a conference paper at ICLR 2023
Table 5: Accuracy on RayS on CIFARIO with the ensemble of rotation and vertical flip task.
Attacks
w/o Fine-tuning
w/ Fine-tuning
RayS
65.61%
77.38% 75.03%
Adaptive-RayS
Table Accuracy under GMSA on CIFARIO.
Tasks
Rotation 53.90%
VFlip
Rotation+ VFlip
Meta AT w/o FT
52.79%
53.16%
Meta AT w/ FT + Adaptive AutoPGD
56.96%
56.79%
57.70%
Meta AT w/ FT + GMSA AutoPGD 57.60%
58.53%
59.63%
Attacking Objectives. The improvement of the test-time adaptation is not affected by the attack objectives. Even no information of the ground truth label is incorporated in the attack, the test- time adaptation improves the robust accuracy. When the attacker randomly lowers the of the false label to perform the adversarial attack, our method uses the information of the leaked label to improve the robust accuracy, it will predict the false label and reduce the accuracy. However, as shown in Table [T] the self-supervised test-time line-tuning improves the robust accuracy on these "adversarial" images. Besides, previous experiments on clean images already show that test-time fine-tuning is effective even if there is no information of the ground truth label.
C.3 ADDITIONAL COMPARISON
Comparison with SOAP (Shi et al., 2020). Our method is different from SOAP tuning the model to adapt to new examples instead of purifying the input. We apply SOAP-RP to the adversarially- trained model and find that its improvement is marginal. Under AutoPGD, the accuracy is improved from 53.09% to 53.57%. %. This improvement much smaller than our method, whose improvement is from 53.09% 57.93%. SOAP only has little effect when combined with the commonly used AT.
fine-
Combination with (Gowal al. 2020). We combine our test-time adaptation with AT using ad- ditional data (Gowal et al. 2020). We apply our Meta AT to it with the ensemble of rotation and vertical flip. Using WideResNet-288 10, achieves robust accuracy of 62.07% under AutoPGD. With our time adaptation, the robust accuracy is improved to 64.34%. The improvement of robust accuracy 2.27%.
Robust Accuracy v.s Fine-tuning Steps. Figure 3 shows the robust accuracy at each step of the test-time fine-tuning for different self-supervised tasks and attack methods. When using the standard version of attacks, the robust accuracy gradually increases as fine-tuning proceeds. When using our adaptive attacks, the adversarial examples are generated to attack the network with 0² (T' = 10) instead of 0", Thus, when the parameters gradually change from 0 to 0" the accuracy drops.
Inference Time. Table shows the inference time for different methods. While the inference time for method is larger than SOAP and the normal method when the batch size is the inference time gets closer when using a larger batch size. And the batch size of 20 or more is a common scenario of the inference. In order achieve the statistical optimal adversarial risk, additional time
Table 7: Ablation study the online test-time fine-tuning, The dataset is CIFARIO and the task is the "Rotation + VFlip". All attacks are standard attacks. SA stands for Square Attack.
Methods Regular AT 65.64%
SA
PGD-20
AutoPGD
FAB
59.19%
53.16%
53.05%
Online FT
66.26%
60.18%
56.86%
75.26%
Improvement 0.62%
0.99%
3.70%
22.21%
Meta AT 65.75%
59.51%
53.99%
53.85%
Online FT 67.34%
61.79%
59.23%
76.39%
Improvement
1.59%
2.28%
5.24%
22.54%
22

---

Under review as a conference paper at ICLR 2023
Table 8: Robust test accuracy CIFARIO of the online test-time fine-tuning. We use the same WideResNeet34-11 in Table la which is trained with &oo budget 0.031. The inputs are in the loc ball of 0.015. The self self-suuer task the ensemble of rotation and vertical flip.
Square Attack
PGD-20
AutoPGD
FAB
Methods
Standard Adaptive Standard Adaptive Standard Adaptive Standard Adaptive
Meta AT w/o FT 78.01% %
75.34%
72.72%
72.58%
Online
80.50% 79.87%
77.14%
76.75%
77.25%
74.93%
82.04%
83.76%
Table 9: Ablation study on the online test-time fine-tuning tuning. The dataset is CIFARIO and the task is the "Rotation VFlip". All attacks are standard attacks. Removing the Lss or L results in lower robust accuracy than the full method. SA stands for Square Attack.
Methods
SA
PGD-20
AutoPGI
FAB
Before FT
65.75%
59.51%
53.99%
53.85%
Online FT
67.34%
61.79%
59.23%
76.39 %
Removing &R 66.83%
60.45%
57.32%
75.63%
Removing Las
65.44%
60.24%
55.64%
75.08%
Table 10: Accuracy on clean images. training.
Networks are trained with corresponding meta adversarial
Methods Without FT
Rotation
VFlip
Rotation + VFlip
84.77%
86.55%
86.36%
Online FT
86.10%
87.00%
87.10% %
of test-time adaptation is necessary. Reducing the inference time is an important future work of these kinds of methods.
Combination with TRADES (Zhang et al. 2019). Table 13 shows the robust accuracy combin- ing our test-time adaptation with TRADES. Our test-time adaptation improves the robust accuracy by about 4%, which shows our approach can improve various types of robust training methods.
C.4 VISUALIZATION
In Figure g we show the visualization of several examples that our test-time adaptation successfully corrects the misclassified examples. The input examples are generated by AutoPGD CIFAR 10, and we fine-tune the network with the ensemble of Rotation and VFlip tasks. shows our test- time adaptation reduces the loss for the whole neighbourhood of the input examples to increase the accuracy of the model.
In Figure 5 we show the histograms of the loss values for the successful and unsuccessful test-time adapted models. For each input instance, if the test-time adaptation corrects the wrong prediction, count it as successful. And if the misclassified instance is not correctly predicted after our test- time adaptation, is counted as an unsuccessful one. The figure illustrates that our method can adapt the model to correctly classify the instances close the decision boundary (with medium loss value). However, for the highly misclassified instances (with large loss value), which are far away from the decision boundary, our test-time adaptation cannot make the model change so much to predict correct labels for them.
D DETAILS OF OUR EXPERIMENTAL. SETTING
D.1 HYPERPARAMETERS
Meta Adversarial Training. The algorithm Meta Adversarial Training is shown in Algorithml I We consider lo norm with adversarial budget 0.031. We also two different net- work architectures: WideResNett34-11 for CIFAR 10 and ResNet18 for STL10 and Tiny ImageNet. Following the common settings for AT, we train the network for 100 epochs using SGD with
23

---

Under review as a conference paper at ICLR 2023
Table 11: Experiments rule out the possibility of label leaking. We use the - trained with for budget ε 0.031 and show the robust test accuracy on CIFARIO of the online test-time fine-tuning. The self self-supervised task the ensemble of rotation and vertical flip.
Square Attack
PGD-20
AutoPGD
FAB
Methods
Standard Adaptive Standard Adaptive Standard Adaptive Standard Adaptive
Meta AT w/o FT
85.43%
85.60%
85.00%
86.22% 87.61 86.97%
Online
86.56%
87.63%
86.29%
86.68%
86.
85.90%
Standard Square Attack
Standard PGD-20
Standard AutoPGD
Standard FAB
Accuracy
Accuracy
0.52
Accuracy
Accuracy
Vertical Fip
Ersemble
6:0
Stops
Steps
Stops
Steps
Adaptive Square
Adaptive PGD-20
Adaptive
Adaptive FAB
640
Accuraoy
6.000
Accuracy
Accuracy
Aocurray
0.00
cais-
Vertical Fip
6668
Ensemble
Steps
Steps
Steps
Figure 3: Robust accuracy at different steps of the online test-time fine-tuning on CIFAR10.
Table 12: Average inference time for each instance using different methods.
Batch Size
5
10
20
40
Normal SOAP (hi et al. 2020) Ours
17. Ims
14.5ms
13.2ms
12.8ms
11.7ms
163ms
91.2ms
75.3ms
73.1ms
72.5ms
545ms
168ms
118ms
83.9ms
82.9ms
momentum factor of 0.9 and a weight decay factor of 5 x 10 10-4, The learning rate 3 starts at 0.1 and is divided by a factor of 10 after the 50-th and again after the 75-th epochs. The step size in Eqn (8 is the same as 3. The factor C' in Eqn (11) is set to 1.0. We use 10-iteration PGD (PGD-10) with step size of 0.007 to find the adversarial image B3 at training time. The weight
of each self-supervised task set to Cs = 1- We set [By| = 32 and sample 8 batches B Bg in each iteration. Furthermore, save the model after the 51-st epoch for further evaluation, as the model obtained right after the first learning rate decay usually yields the best performance (Rice let al. 2020).
We use PGD with the standard cross-entropy loss to generate adversarial examples at training time in line 3, line 6 and line 8 of Algorithm 1 The hyperparameters of the attacks are follows:
Line 3: PGD-10 with step size 0.007.
Line 6: As 03 is similar to 0, the adversarial examples at this step are similar to those at Line 4. To save training time, therefore choose the starting point of the attack as the adversarial examples in Line 4 and use PGD-2 with a step size of 0.005.
Line 8: PGD-3 with step size 0.02.
Online Test-time Fine-tuning. The algorithm for online fine-tuning is shown in Algorithm 2 We fine-tune the network for T 10 steps with a momentum of 0.9 and a learning rate of = 5 10 44 We set C₁ = x and C 15.0. In line of Algorithm 2 we sample batch B c D containing 20 training images. line 3, we use PGD-10 with a step size of 0.007.
Offline Test-time Fine-tuning. The algorithm for offline fine-tuning is shown in Algorithm As stochastic gradient descent is more efficient for a large amount of data, we use stochastic gradient descent in the offline fine-tuning. This is the main difference between Algorithm 2 (online fine-
24

---

Under review as a conference paper at ICLR 2023
Before Fine-tuning
After Fine-tuning
Loss Change
0.0
0.03
1.4
0.03
14
0.03
0.02
1.2
0.02
1.2
0.02
0.2
1.0
0.01
1.0
-0.4
-0.01
-0.01
-001
0.4
0.4
-0.02
-0.02
-0.02
-0.8
-02
0.2
-0.03
-0.03
-0.33
0.02
-0.02
0.02
0.02
Adversarial Direction
Adversarial Direction
Adversarial Direction
0.0
1.4
14
0.02
1.2
0.02
1.2
0.02
0.2
0.01
1.0
0.01
1.0
0.01
0.8
0.85
0.6
0.6
0.6
-0.4
-0.02
-0.02
-0.02
00.0
-0.03
-0.03
-0.03
-0.02
0.00
0.02
0.0
-0.02
0.00
0.02
0.0
-0.02
0.00
0.02
110
0.0
1.4
1.4
122
1.2
0.01
1..
0.01
1.0
0.01
004
0.00
0.8
0.00
0.85
0.00
0.6
0.6
0.66
0.01
000
-0.03
-0.03
-0.3
-0.02
0.00
0.02
0.0
-0.02
0.00
0.02
0.0
-0.02
0.00
0.02
1.0
Direction
Direction
Diraction
0.0
0.03
1.4
0.03
1.4
0.03
1.0
1.0
0.4
0.00
0.8
0.00
0.85
0.00
0.6
00.11
0.01
0.01
0.4
0.02
000
0.0
0.2
0.2
0.02
0.00
0.02
0.0
0.02
0.00
0.02
0.0
0.02
0.00
0.02
1.0
Adversarial Direction
Adversanal Direction
Adversarial Direction
0.03
0.03
14
0.03
0.02
1.2
0.02
1.2
0.02
0.2
1.0
1.0
0.4
0.0
0.08
0.01
0.01
0.01
0.4
0.4
0.02
.0.22
-002
008
0.2
0.2
0.02
0.0
-0.02
0.0
-0.02
Adversarial Direction
Adversurtal Direction
Adversartal Diraction
Figure 4: Visualization of several examples that test-time adaptation successfully changes the wrong prediction. Each row represents an example of loss surfaces before fine-tuning, after fine- tuning and the loss changes of our fine-tuning. The origin point represents the clean example. Following (Kim et al.) 202T). -r-xis represents the direction of the adversarial example and y-axis random direction. The white line the decision boundary. As the fine-tuned model correctly classifies the input example, the decision boundary does not exist in the neighbourhood of the clean input for the fine-tuned model.
25

---

Under review as a conference paper at ICLR 2023
Table 13: Combination with our test-time adaptation with TRADES on CIFARIO with the ensemble of rotation and vertical flip tasks.
Standard AutoPGD Adaptive AutoPGD GMSA AutoPGD
Meta AT w/o FT
54.06%
Online FT
59.63%
57.74%
59.39%
VFlip
Rotation+VFiip
2.0
1.0
Densly
Denslty
0.0
Density
0.6
0.6
0.4
0.4
0.4
0.2
Successful
0.0
0.0
0.0
Croes Entropy Before Fine-tuning
Crass Entropy Before Rinetuuinng
Cross Entropy Before Fine-tuning
Figure 5: Histograms of the loss values for the successful and unsuccessful test-time adapted models.
tuning) and Algorithm E (offline fine-tuning). We also fine-tune the network for 10 epochs. The batch size of each Bj is 128. The other hyperparameters are the same in the online version.
Algorithm 2 Self- Test-time Fine-tuning
Input:
Initial parameters 0º; Adversarial test images D = (x; 18 J==1 Training data D; Learning rate 7/. Steps T; Weights C and C
Output: Prediction of X 7/
for 1 to T do
2: Sample batch of training images B C D
3: Find adversarial x; of training image x; E B by PGD attack.
4: Calculate Lrest in Eqn 66
5: 0¹ p==66 0 V testt B" BB,B, 01-)
6: end for
7: return Prediction arg max, F(x;;0)))
Attacks. The detailed settings of each attack are provided below:
PGD-20. We use 20 iterations of PGD with step size = 0.003. The attack loss is the cross-entropy.
AutoPGD. We use both the cross-entropy and the difference of logits ratio (DLR) as the attack loss. The hyperparameters are the same as in (Croce & Hein 2020a).
FAB. We use the code from (Croce& & Hein 2020a and keep the hyperparameters the same.
Square Attack. We set 2000 and the initial fraction hyperparameters the same in (Andriushchenko et al. 2020).
elements p = 0.3. The other
For the adaptive versions, the interval [T/5].
D.2 SELF-SUPERVISED TASKS
Rotation Prediction is widely used self-supervision task proposed in (Gidaris et al. (018)) and has been employed in AT as an auxiliary task to improve the robust accuracy (Chen et al., 2020a Hendrycks al. 2019). Following (Gidaris al. 2018), create 4 copies of the input image by rotating it with 12 (0", 90", 180" 270" J. The task then consists a 4-way classification problem,
where the head Orotate aims predict the correct rotation angle. The loss for image is the average
26

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Self-supervised Test-time Fine-tuning with SGD
Input:
Initial parameters 0% Adversarial test images B {x J== Training data D; Learning rate
7); Steps T; Weights C₁ and C
Output: Prediction of xt: 1/4
1: for t= 1 to 7 do
2: Divide B" into subsets Bi B;
3: for Bj in Bi.... B do
4: Sample batch of training images B C D
5:
Find adversarial x training image x; E by PGD attack. Lress )
6:
-'=0"
7: end for
8: end for
9: return Prediction ÿ arg max, F(x;; 0T);
cross-entropy over the copies, given by
Lrotate (x)
Σ WEll
log (orotate ( (xw)w)
(37)
where x. the rotated image with angle Ω Grotate Grotaee E denotes the classifier for rotation prediction, and Gmnate()) is the predicted probability for the w angle. The head froatee is a fully-connected layer followed by softmax layer.
Vertical Flip (VFlip) Prediction is a self-supervised task similar to rotation prediction and has also been used for self-supervised learning (Saito al., 2020). In essence, we make two copies of the imput image and flip one copy vertically. The head Svnip then contains 2-way fully-connected layer followed by a softmax layer and predicts whether the image is vertically flipped not. The corresponding loss for an image is
1 Σ 2
Lviip (x)
log( Log(Grnp(Xx))) (x.)))
(38)
NEV
where V = (flipped, not flipped] | is the operation set and Gvmipp Ivilip E. denotes and transformed input and G- is the probability of operation v. Note that we do not flip the image horizontally as it is common data augmentation technique and classifiers typically seek to be invariant to horizontal flip.
E ADAPTIVE ATTACKS
In the white-box attacks, the attacker knows every detail of the defense method. Therefore, we need to assume that the attacker is aware of our test-time adaptation method and will adjust its strategy for generating adversarial examples accordingly. Here, we discuss one such strong adaptation strategy targeted to our method.
Suppose that the attacker is fully aware of the hyperparameters for test-time adaptation. Then, finding adversaries 3* of the clean subset can be achieved by maximizing the adaptive loss
arg max Lattack (F yr 0J (B*)),
(39)
| Sx
where Lattack refers to the general attack loss, such as the cross-entropy or the difference of logit ra- tio (DLR) (Croce & Hein 2020a). Let 9 be the fine-tuned test-time parameters using Algorithm At the k-th step of the attack, it depends on the input (k) {(x))) K,)} via the update
g++1 0*
Vo Lrest ((k) B)
(40)
where Frest and B are the loss function and subset of training images mentioned in Eqn (6) As
0 is function of the input (kk), caleulate the end-to-end gradient of x(4) B as
27

---

Under review as a conference paper at ICLR 2023
V x(()) Lattack (F (F(;
(8))) 0 (B(h))) However, 0² goes through T gradient descent steps, and thus calculating the gradient V "x 0 (B())) requires T-th order derivatives of the backhone E, which is virtually impossible if T or the dimension of 0p large. We therefore approximate the gradient
Grad(x (*) V V Cattack (F(x 0²)
(41)
which treats 0" as a fixed variable so that high-order derivatives from ("((B()(x(*)) can be avoided. Although this approximation makes Grad(x inaccurate, common white-box attacks use projected gradients, which are robust to such inaccuracies. For example, PGD only uses the sign of the gradient under an &o adversarial budget. Note that solving the maximization Eqn (12) does
not necessarily require calculating the gradient Grad(x; (k)) For instance, we will also use Square Attack (Andriushchenko et al. 2020), strong score-based black-box attack, to maximize Eqn (2) and generate adversaries for B.
As another approximation to save time, one also fix 0 for several iterations. This leverages the intuition that attack strategies often make small changes to the input X, and thus, for the intermediate images in the k-th and (k+1)-th steps, 0" (B(k)) and 0² 0 (B((+1) should be close. Therefore, general version of adaptive attacks only updates 0² every u iterations, with hyperparameter.
In Algorithm/ Square Attack and FAB, respectively. The main difference between the original and adaptive ver- sions is the target loss function for maximization. The reader may refer to (Andriushchenko et al., 2020 Croce Hein (2020a]) b) for a more detailed description of the steps in these algorithms (e.g.. the condition for decreasing the learning rate in AutoPGD).
5 and 7 we show the algorithms for for norm-based adaptive PGD, AutoPGD,
Algorithm Lo Norm Adaptive PGD Attack
Input: Test images (&;, }; Attack loss Lattack: Step size "r; Iterations T; Intervals u; Adversarial budget Trained parameters of the network 0".
Output: Adversarial images B {x;}
1: Add random noise to X; in B and get B'
2: for t = 1 to T do
3: if mod = then
4: Get final parameters 0 by taking B' input image for Algorithm 2 0 = 0
5: end if
6:
for x in B' do
7: Grad(X)) (X))
Te: 0) Cottac (F(X)), "e:)),:::
8: X = =Clipp -exxtts - (8 + +Sign Grad
9: end for
10: end for
11: return Adversarial image X; X
28

---

Under review as a conference paper at ICLR 2023
Algorithm 5 to Norm Adaptive AutoPGD
Input: Test images
{ :; Attack loss Lattack: Step size "i Iterations T; Intervals u;
Adversarial budget E; Parameter the adversarially- -rainnd network 0%; Decay iterations W {Wo,... ] Momentum
Output: Adversarial image B
{x;}
1: Get final parameter 0 by taking as input image for Algorithm
2:
0 -
3: for X;
in B do x X;
4:
5:
Gra = V attack (F Ts: 0)
6:
x Clippe,-c.a
(Sign(Grad((x.))
7:
1 = Luttack F Fxx))
Ys; 0)
8:
Pattack F x
Vi 0)
9: 1 max
11, 11]
x; x if 17 10 else x x
11: end for
for 1 to - do
13: if mod и = 0 then
Get final parameter 8T by taking B* {x²] as input image for Algorithm 0= 0"
16:
end if
for 1,... do
18:
Grad(x!) = V x aattack (F(x)), Ne; 0)
19:
o Clippx- [ + Sign Grad(x))))
20:
x Clipx,-e (x} + E(2 - 25) + (1- ()x x
1+1 Lattack (x
( T65
& x and 13 11+1 ir >
22:
23:
if & E W and satisfy the condition of dropping learning rate then
= 7/2 and x X
24:
25:
end if
26:
end for
27:
end for
28: return Adversarial image x; = x
29

---

Under review as a conference paper at ICLR 2023
Algorithm lo Norm Adaptive Square Attack
Input: Test images B ((x, }; Attack loss Lattacki Step size s; Iterations T:: Intervals u; Image size w;; Color channels e; Adversarial budget e; Parameter of the adversariilly-trained network 00
Output: Adversarial image D = }
1: Add noise to in B and get B'
2: for = 1 to T do
3: if' mod = then
4:
Get final parameter 0 by taking B' as input image for Algorithm) 2 5: 0 9"
6:
end if
7:
for x in B' do
8:
h' +- side length of the square to modify (according to some schedule)
9:
+ array of zeros of size w x w c
10:
Sample uniformly
E {0, ..., h+) CN
11:
for j I,.... e do
12:
Uniform( -22, 2x)
13:
8, 1 1: 11
14:
end for
15:
xnew
Clip -. (X + 5) Lattack (F ), 0)
16:
new
if Inew
1" then
18:
X
xhew
Jnow
20:
end if
end for
22:
end for
23:
return Adversarial image x X
30

---

Under review as a conference paper at ICLR 2023
Algorithm 7 to Norm Adaptive FAB
Input: Test images B {(,, V:) Step size Iterations T; Intervals u; Adversarial budget E; Trained parameters of the network 0% Omax: 1), S.
Output: Adversarial images D* {x;}
1: Add random noise to X; in and get B'
2: +00
3: for -= 1 to T do
4: if mod
then 5: Get final parameters 0 by taking 33' as input image for Algorithm 2 0 0
6:
end if
7:
for X in B' do (rad(xx)= Vx FX; (X;0); 0):
8:
-
9:
8 = arg millyys yiy x
10:
proj (X,, Te C)
11:
8 proj (X,, T. C)
a min
|| + bigg Il.
a Omax
E [0, 1]
& projc (1 (x{ + 178)) + a(x; + 1/8 orig
13:
X is not classified as yi then
15:
if X
x, 11 then
x
17:
| 8:11,0
end
19:
X,
8)
+ 93
20: end if
21: end for
22:
end for
23:
return Adversarial image x
31


Reviews:
[{"review_id": "uXwIihLC5C", "reviewer": "Reviewer_CHte", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows that under suitable assumptions, the Bayesian optimal robust estimator requires test-time adaptation, and such adaptation can lead to a significant performance boost over standard adversarial training. It then proposes self-supervised test-time fine-tuning on adversarially-trained models to improve their generalization ability. A MAT strategy is introduced to find a good starting point for the self-supervised fine-tuning process. Extensive experiments on CIFAR-10, STL10, and Tiny ImageNet demonstrate that the method consistently improves the robust accuracy under different attack strategies, including strong adaptive attacks where the attacker is aware of the test-time adaptation technique. ", "summary_of_the_review": "Overall, I am positive about this paper. I only have some concerns about the adaptive attack evaluation. It is possible that there are stronger adaptive attacks than the ones considered in the paper. I give a weak rejection for now. If the authors could address my concerns, I am willing to raise my scores. \n\n--------POST REBUTTAL--------\n\nThe authors have addressed my concerns. Thus, I raise my score from 5 to 6. ", "strength_and_weaknesses": "I think this paper has the following strengths: \n\n1. The idea of using a meta adversarial training method to find a good starting point for test-time adaptation is novel. The ablation study experiment shows that the meta adversarial training is effective in improving robust accuracy. \n\n2. It theoretically shows that the estimators should be test-time adapted in order to achieve the Bayesian optimal adversarial robustness, even for simple models like linear models and the test-time adaptation largely improves the robustness compared with optimal restricted estimators. The theoretical results are significant, though I don't check the correctness of the proofs carefully. \n\n3. The experiments show that the approach is valid on diverse attack strategies, including an adaptive attack strategy that is fully aware of the test-time adaptation, in both the white-box and black-box attacks. \n\n4. The paper is well-written. The proposed defense method is described clearly. It provides enough details of the adaptive attacks used. \n\nHowever, I think this paper has the following weaknesses: \n\n1.  It doesn't evaluate the Greedy Model Space Attack (GMSA) proposed in [1], which is designed for the transductive-learning based defenses (or the test-time adaptation defenses). I think it is important to include GMSA in the attack evaluation of the proposed defense. \n\n2. It doesn't report the point-wise worse-case robust accuracy under all the attacks considered. For each test data point x, if any of the attacks considered could find an adversarial example x' that fails the defense, then the robust accuracy on x is 0; otherwise, the robust accuracy on x is 1. If we aggregate it over the entire test set, we can get the point-wise worse-case robust accuracy under all the attacks considered. It would be good to include this metric in the experimental results. \n\n\n[1] Chen, Jiefeng, et al. \"Towards Evaluating the Robustness of Neural Networks Learned by Transduction.\" International Conference on Learning Representations. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rJAeG6-xTBx", "reviewer": "Reviewer_SHJC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a test-time fine-tuning to boost the adversarial robustness of the classifier. Namely, the new method updates the parameter of the underlying deep network based on self-training (and potentially assuming the network is trained with a meta-learning algorithm). To evaluate the robustness, the paper places the model under a regular white-box adversary and a smarter one aware of the fine-tuning and shows that even with a smarter adversary the fine-tuned model is still more robust. ", "summary_of_the_review": "In summary, I am inclined to reject it at this moment because I am not sure if the proposed method actually produces a new network $\\theta$ that is more robust with more obviously vulnerable parts exposed to the adversary. The writing of the paper can be improved as well.", "strength_and_weaknesses": "### Strength\n\nThe paper has a solid motivating theorem that shows why a test-time fine-tuning can improve the robustness. Even though the theorem does not directly help the design of the final algorithm, the theoretical contribution is still important. In the empirical part, the paper uses a standard set of adversaries, e.g. AutoAttack and the improvement of robustness is significant compared to its baselines. Overall, the structure of the paper is easy to follow and I know what to expect when I finish reading one section. \n\n\n### Weakness\n\nMy major concern is the empirical evaluation. I also have some minor concerns on the motivating theorem and the writing. Please see the details below. \n\n**The setup of an empirical adversary may not be strong enough.** Firstly let me restate the setup of the problem. If my read on the paper is correct, given a batch of input $X$, this work proposes to fine-tune the parameter of the model $\\theta_0 \\rightarrow \\theta$ and the update $\\Delta \\theta = \\theta - \\theta_0$ is a function of the batch $X$ (and some training batch). \n\nTo evaluate the robustness, the paper uses two adversaries: a standard and an adaptive one. I am not able to find the definition of a standard adversary but by reading the description of an adaptive one I think the paper assumes that a standard one targets on a model parameterized with $\\theta_0$ while an adaptive one targets $\\theta$ (please correct me if I am wrong). \n\nThe paper assumes the adaptive one is a strong adversary by fully leveraging the knowledge of the fine-tuning process. My comment for this setup is that: \n\n(1) the adaptive one may be just as strong as a standard adversary who is faced with a model without fine-tuning. This is because the white-box adversary has access to the parameters (i.e. $\\theta$) of the model that makes the inference instead of some parameters (i.e.$ \\theta_0$) that have nothing to do with the inference; \n\nand (2) an adversary who is smarter should be targeting the fine-tuning process. One example I can think of is that the adversary carefully constructs the test batch sent to your system such that these inputs sit evenly on two sides of the decision boundary and are both less than $\\epsilon$ away from the boundary. How would the fine-tuning behave? Will it almost make no update to $\\theta_0$ (so the adversary can attack the original model again) or it gives up one half the points so after fine-tuning $\\theta$ can be robust on the rest? In practice, the adversary can jointly optimize the noise added to each input. For example, if the adversary only cares about the inference result on $x_i$, it can accompany another input $x_j$, together with $x_i, that targets only on making the fine-tuning doing nothing or worse. I drew a picture [here](https://ibb.co/yXPS71D) for a linear classifier to illustrate the case. Also, the training set used in fine-tuning is also exposed to the attacker and a realistic attacker should take advantage of it. In general, I don't think the current adaptive attacker is adaptive enough to the potentially vulnerable parts in the proposed defense. \n\nTwo additional questions regarding the experiment:\n\n1. a strictly stronger attacker should always have lower accuracy than a standard one but in Table 1, some adaptive attacker is even worse than a standard one. For example, on the intersection of the row Probation-OnlineFT and the column Square Attack. Can the authors give some explanation to these results? \n\n2. It seems that there are a lot of hyper-parameters to be tuned. Can the author provide some recommendations of ways to find these parameters in the main body of the paper. \n\n**Soft labels used in Theorem 3.1.**  Will the theorem fail to hold if considering hard labels like [0, 1] instead of soft labels generated from Gaussian noise? I think in a classification setup using a soft label might be okay if well-explained motivation is given. \n\n**Typos.** I found many typos in the paper. I list some examples here:\n\n1. Near Theorem 3.1: “However, for ^ arbitrary ratio c = n/d” (an is missing)\n\n2. Near Theorem 3.1: “we plot the of adversarial risk of three estimators for different adversarial budgets, which clearly shows that adaption can significantly …” (missing “our” or “the” before “adaptation”) \n\n3. Near Theorem 3.2: “The theorem shows that when the given input is the adversarial, the test-time adaptation can still ^ lower the adversarial risk” (“is adversarial”)\n\n4. In Figure 1, it should be $\\theta_AB$ not $\\theta_BA$ from the caption (or I mis-understand the picture). \n\n5. Near Eq 6, “more efficient for more large amount of data” (larger instead of more large) \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "idYjdb3oq19", "reviewer": "Reviewer_1EpH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a test-time adaptation method to improve the adversarial robustness of a neural network by proposing (a) a test-time objective based on existing self-supervised training methods (e.g., RotNet [Gidaris et al., 2018]) and (b) a pre-training objective based on incorporating gradient-based meta-learning (e.g., MAML [Finn et al., 2017]) and adversarial training (AT). Experimental results on CIFAR-10, STL10, and Tiny ImageNet demonstrate the effectiveness of the proposed method in improving adversarial robustness under several adversarial attacks.\n\n[Gidaris et al., 2018] Unsupervised Representation Learning by Predicting Image Rotations, ICLR 2018\n\n[Finn et al., 2017] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, ICML 2017\n", "summary_of_the_review": "The paper tackles a challenging and under-explored problem of test-time adaptation for adversarial robustness. However, I slightly feel a lack of technical novelty, and that the overall clarity could be improved. For the evaluation side, I am still not fully convinced that the adaptive attacks considered here would faithfully assess the proposed model, which would affect the correctness of defense evaluation. In these respects, I am currently on a slight negative side.", "strength_and_weaknesses": "**Strength:**\n\n* The experiments are performed under various datasets to validate the method.\n* The paper tackles an under-explored problem of improving adversarial robustness via additional test-time adaptation.\n* The method is easy-to-implement, yet shows a considerable improvement in robust accuracy.\n* The paper performs a theoretical study that motivates the method\n\n**Weakness:**\n\n* Although I appreciate that the paper includes a discussion (and results) on adaptive attacks (Eq. 12), but I personally doubt that the attack could indeed faithfully find a true (hidden) adversarial examples: (a) the threat model considered in this paper is quite challenging, in a sense that now the attacker should find a sample that minimize the chance of correct adaptation (with a high-dimensional optimization); (b) empirically, the results from adaptive attacks are sometimes worse than the standard attack, which signals that the adaptive attack can be actually worse sometimes. \n* The theoretical analysis in Section 3 yet does not fully justify how the proposed method in Section 4 helps to improve the robustness, e.g., to motivate the use of the self-supervision based test-time training objective. Instead, all the explanation in Section 3 deals with a direct AT-like objective; how can such a self-supervised task replace AT as the surrogate objective? Any intuitions/high-level explanations would be welcome. Moreover, at least, for better clarity to readers, I recommend mentioning Section 5.2 in Section 4 that the gradient of the self-supervised objective empirically shows a highly-correlation to a gradient of the AT objective.\n* Given that the main claim of the paper is “AT without adaptation cannot be optimal for the best robustness”, the paper should show the superiority of the method by providing a comparison with state-of-the-art AT methods (e.g., TRADES [Zhang et al., 2019]), not limited to regular AT [Madry et al., 2018].\n* The technical novelty may be limited: both the pre-training objective (meta adversarial training) and the adaptation objective in the proposed method are an extension of prior works using adversarial examples.\n* The paper is generally hard to follow, and the writing could be improved for better clarity. For instance, in the second paragraph in Section 1, “Theoretically, AT does not achieve the optimal robustness. Under suitable assumptions, the Bayesian optimal robust estimator requires test-test adaptation.” appears without detailed explanation or references. It would be much better if the authors can provide several prior relevant works (if it exists) on investigating why the optimal robustness cannot be achieved only with AT and test-time adaptation is a necessary choice for the robust estimator. Otherwise, the paper may include more insights/intuitions on these arguments, as this is the very first part of the paper.\n\n**Questions:**\n\n* The test-time adaptation objectives include the regularization term, which requires the training set: how does the training batch is selected, and is the proposed method robust to the choice of training mini-batches?\n* Can the proposed method be combined with other AT objectives (e.g., TRADES), not limited to the vanilla AT objective [Madry et al., 2018]?\n\n**Minor:**\n\n* Typo: test-test adaptation → test-time adaptation (Section 1, second paragraph)\n* Typo: Definition 3.2 to 3.2 → Definition 3.2 to 3.4 (Section 4, Theorem 3.1)\n* Typo: more large → larger (Section 4.1, last paragraph)\n\n\n[Madry et al., 2018] Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018\n\n[Zhang et al., 2019] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FPo51XrJIKg", "reviewer": "Reviewer_J4z7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes test time training for the task of learning adversarially robust models. The idea is to finetune the model for the test samples using a combination of self-supervised loss on the test samples and a memory-based loss on a small subset of training samples. Since the original adversarially trained model might not provide a good initialization for finetuning, a meta-learning based objective is proposed that learns a model that gives best finetuning robustness. Experimental results are shown on some benchmark datasets.\n", "summary_of_the_review": "I think the approach is interesting. But I feel the paper lacks experimental rigor. Also, some visualizations could have really helped. I feel if authors spend a bit more time with more rigorous experiments and visualizations, this paper would get strong.", "strength_and_weaknesses": "Strengths:\n\nThe idea of using test time training in the context of adversarial robustness is new. The paper advocates for the need for sample-specific decision boundaries as they claim that it is needed to get Bayesian optimal adversarial robustness. To do this, the paper uses a simple self-supervised learning objective on the test samples.\n\nThe objective function used is simple and intuitive. The fact that the test time objective also supports a batch size of 1 is nice.\n\nResults clearly show that using test time training improves the accuracy compared to standard adversarial training. The authors also show results on adaptive attacks, which is nice.\n\n\nWeaknesses:\n\nI think one of the main weaknesses of the paper is that it doesn’t compare with the SOTA approaches such as TRADES, MART, etc. Comparison with SOTA approaches would be nice. The authors can use some benchmarks like RobustBench to compare their models with the best ones out there.\n \nThe other thing that is concerning me is the difficulty in training meta-learning based objective. Training models with meta-learning objectives are not straightforward, so it might be hard to scale this to large datasets like Imagenet.\n\nThe next issue is the increase in the inference time. From Figure 3, it looks like more adaptation steps are needed to improve the accuracy. This would mean inference time would rise, which could be a concern.\n\nThe method proposed by the authors does not depend on what type of adversarial robustness algorithm used. We can simply replace the AT loss with any type of loss used. So, it would be interesting if this method can improve over the best SOTA results. The authors have one experiment in this regard - comparison with Gowal et al.\n\nOne of the things I would have liked to seen in this paper is more visualizations. Why is this approach helping? Does the classifier decision boundary change for samples near the decision surface? For the samples far from the decision boundary, do things change? Such visualizations would help improve the understanding of what is happening.\n\nAnother interesting experiment would be to see if the approach improves more if the test distribution has more domain shift. I know that the approach was not proposed with this objective in mind, but it might be an additional benefit the authors would get.\n\nThe authors use rotation prediction and vertical flip prediction as the self-supervised objective because it doesn’t have dependence on the batch size which makes sense. But these objectives are quite simple ones, and there are other objectives that could be better too. One example is teacher-student objective for self-supervised learning. I am wondering if these approaches would yield better results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BJqmZrXY1F5", "reviewer": "Reviewer_wAba", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about adversarial training for robust accuracy against adversarial attacks. The authors introduce a new procedure to improve the generalization performance of adversarially-trained networks by using self-supervised test-time ﬁne-tuning. In addition, to determine a good starting point for test-time adaptation, a meta-adversarial training strategy based on the MAML framework is proposed that integrates the test-time adaptation procedure during training. This also strengthens the correlation between the self-supervised and classiﬁcation tasks. This new method is validated for different self-supervised tasks on the CIFAR10, STL10, and Tiny ImageNet datasets tasks, and indicates that their method can improve the accuracy of standard adversarial training under diverse white-box and black-box attack strategies.", "summary_of_the_review": "Overall this is good quality submission.  The proposed method appears to be novel and well-motivated, although the experimental validation could be improved. \n ", "strength_and_weaknesses": "Strengths:\n+ The key concepts and motivations are described in enough detail to understand the paper.   This approach seems easy to implement and employ in practice. \n+ Theoretical explanation for the necessity of the strategy and showing that it improves the robust accuracy of the test data. 1. We show that the estimators should be test-time adapted in order to achieve the Bayesian optimal adversarial robustness, even for simple linear models. And the test-time adaptation largely improves the robustness compared with optimal restricted estimators.\n+ Extensive empirical results show the benefit of the method, and ablation studies are provided to characterize the approach.\n+ The supplementary material provides additional proofs of theorems, implementation details, adaptive attacks, and experimental results that help support the paper.  Also, since the codes are also provided in supplementary material, there is less concern that the results in this paper would be difficult for a reader to reproduce.   \nWeaknesses:\n- The theoretical analysis in Section 3 is dense and somewhat difficult to read. \n- The experimental validation is lacking in some respects. The authors should justify their choice of baselines, datasets, and CNN backbones for experiments. Their proposed method should be also compared with SOA methods in terms of time and memory complexity.  There should be further analysis to assess the impact on the performance of class imbalance and batch size. \n- It is unclear why offline FT sometimes yields lower accuracy than online FT with the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject