Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off
Authors: Futa Kai Waseda
Ching-Chun Chang
Isao Echizen
Venue: ICLR 2025 Poster (2025)
Keywords: computer vision
adversarial defense
adversarial robustness
robustness-accuracy trade-off
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We identify novel key challenges in applying invariance regularization to adversarial training for improving the robustness-accuracy trade-off, and provide novel solutions.

Abstract: Adversarial training often suffers from a robustness-accuracy trade-off, where achieving high robustness comes at the cost of accuracy.
One approach to mitigate this trade-off is leveraging invariance regularization, which encourages model invariance under adversarial perturbations; however, it still leads to accuracy loss.
In this work, we closely analyze the challenges of using invariance regularization in adversarial training and understand how to address them.
Our analysis identifies two key issues: (1) a "gradient conflict" between invariance and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem arising from diverged distributions between clean and adversarial inputs.
To address these issues, we propose Asymmetric Representation-regularized Adversarial Training (ARAT), which incorporates asymmetric invariance loss with stop-gradient operation and a predictor to avoid gradient conflict, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem.
Our detailed analysis demonstrates that each component effectively addresses the identified issues, offering novel insights into adversarial defense.
ARAT shows superiority over existing methods across various settings. Finally, we discuss the implications of our findings to knowledge distillation-based defenses, providing a new perspective on their relative successes.

Published as a conference paper at ICLR 2025
RETHINKING INVARIANCE REGULARIZATION IN AD- VERSARIAL TRAINING TO IMPROVE ROBUSTNESS- ACCURACY TRADE-OFF
Futa Waseda+ Ching-Chun Chang" Isao Echizen 1.2
The University of Tokyo National Institute of Informatics
futa- uuta-wasedaag.een ecc. u-tokyo.ac. jp, (ccchang, iechizen @nii. aaa.aac..... ac. jp
ABSTRACT
Adversarial training often suffers from a robustness-accuracy trade-off, where achieving high robustness comes at the cost of accuracy. One approach to mitigate this trade-off is leveraging invariance regularization, which encourages model in- variance under adversarial perturbations; however, it still leads to accuracy loss. In this work, we closely analyze the challenges of using invariance regularization ad- versarial training and understand how to address them. Our analysis identifies key issues: (1) "gradient conflict" between invariance and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem arising from diverged distributions between clean and adversarial inputs. To address these issues, we propose Asymmetric Representaaioo--rggllaaa (AR-AT), which incorporates asymmetric invariance loss with stop-grade noo-ggaddeett opera- and predictor to avoid gradient conflict, and split- BarchNorm (BN) structure to resolve the mixture distribution problem. Our detailed analysis demonstrates that each component effectively addresses the identified issues, offering novel insights into adversarial defense. AR-AT shows superiority over existing methods across various settings. Finally, we discuss the implications of our findings to knowledge distillation-ban based defenses, providing a new perspective their relative successes.
Adversarial Training
INTRODUCTION
Computer vision models based on deep neural networks (DNNs) are vulnerable adversarial examples (AEs) SZegedy et al. 2014; Goodiellow et al., (201)), which are carefully perturbed inputs to fool DNNs. Since AES can fool DNNs without affecting human perception, they pose potential threat to real-world DNN applications. Adversarial training (AT) (Goodfellow et al. 2015 Madry al. 2018) has been the state-of-the-art defense strategy, however, AT-based methods suffer from a significant robustness-accuracy trade-off (Tsipras et al., 2019): to achieve high robustness, they sacrifice accuracy on clean images. Despite the extensive number of studies, this trade-off has been huge obstacle to their practical use.
One of the promising approaches to mitigate this trade-off leverage invariance regularization. such as TRADES (Zhang et al., 2019) and LBGAT (Cui et al. 2021). which encourage the model to be invariant under adversarial perturbations. However, these methods still face trade-offs, and their limitations require further understanding.
To this end, we investigate the following research question: "How can a model learn adversarially invariant representations without compromising discriminative ability? "This work carefully analyzes the challenges of applying invariance regularization in AT to improve the robustness-act accuraccy trade- off. We identify novel issues and propose novel solutions to address them, offering novel insights into adversarial defense.
Our investigation identifies two key issues in applying invariance regularization (Fig la (1) "gradi- ent conflict" between invariance loss and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem within Batch Normalization (BN) layers when the same BNs are used for both clean and adversarial inputs. "Gradient conflict" suggests that minimizing invariance loss may push the model toward non-diseriminative directions and conflict with the classification

---

Published as a conference paper at ICLR 2025
objectives. Furthermore, we find that using the same BNs for both clean and adversarial inputs causes the mixture distribution problem: the BNs are updated to have a mixture distribution of clean and adversarial imputs, which can be suboptimal for both clean and adversarial inputs.
To address these issues, wC propose a novel method, Asymmetric Representation pnnresenttaiin-reglaaiiii regularized Adversarial Training (AR-AT), incorporating asymmetric invariance regularization with stop- gradient operation and a predictor, and a split-BN structure, as depicted in Fig. Tr Our step-by -y---pp analysis demonstrates that each component effectively addresses the identified issues, offering novel insights: stop-gradient operation resolves "gradient conflict," and split-BN structure resolves the mix- ture distribution problem. With both components combined, AR-AT improves the abustness-accurrac trade-off, outperforming existing methods across various settings.
Furthermore, we discuss the relevance of our findings to existing knowledge distillation (KD)- based defenses (Cui et al.) 2021; Suzuki et al., (023). pointing out the underlying factors of their effectiveness that have not yet been investigated. We attribute their effectiveness resolving "gradient
conflict" and the mixture distribution problem.
Our contributions are summarized as follows:
We focus on understanding the challenges of invariance regularization in adversarial training to improve the cobustness-accuraae trade-off. We identify novel issues and propose novel solutions.
We reveal two key issues in using invariance regularization: (1) "gradient conflict" between invariance loss and classification objectives, and (2) the mixture distribution problem within Batch Norm (BN) layers.
We propose a novel method, AR-AT, which incorporates an asymmetric invariance regular- ization and split- Biit-BN structure. Our detailed analysis shows that each component effectively adresses the identified issues, offering novel insights into adversarial defense.
AR-AT outperforms existing methods across various settings.
We present new perspective on KD-based defenses, which have not been well understood.
PRELIMINARIES
2.1 ADVERSARIAL ATTACK
Let E Rd be input image and y [I....,K] K} be a class label from data distribution D. Let fo Rd > R be a DNN model parameterized by 9. Adversarial attacks aim to find a perturbation that fools the model fo by solving the following optimization problem:
x'
8, where 8 arg max L(fo(x + 8).
(1)
8ES
where is an adversarial example, S is a set of allowed perturbations, and L is loss function. In this paper, we define the set of allowed perturbations S with L.g S {ō IR [|o]ooo I c). where represents the size of the perturbations. The optimization of Eq [ is often solved iteratively based on the projected gradient descent (PGD) (Madry et al. 2018)
2.2 ADVERSARIAL TRAINING
Adversarial training (AT), which augments the training data with AEs, has been the state-of-the-ar approach to defend against adversarial attacks. Originally, Goodfellow et al. (2015) proposed to train the model with AEs generated by the fast gradient sign method (FGSM); contrast, Madry et al.) (2018) proposed to train model with much stronger AEs generated by PGD, which is iterative version of FGSM. Formally, the standard AT (Madry et al. 2018) solves the following optimization:
min E (R
max C(Solx + 5).y)
dES
where the inner maximization problem is solved iteratively based on PGD.
2

---

Published as a conference paper at ICLR 2025
(elassification)
(classfiication)
classification
classification
Invariance
Invariance
Invariance
encoder
encoder
encoder
encuder f
encoder
encoder f
framework AT (aadry et al. 2018) with
(a)
General
(b) MART Wang et al.
2019
(c) TRADES
Zhang et
2019
invariance regularization
classification
classification
(classffication)
(classification)
Invariance
trained
Invariance
Invariance
predictor
encoder f
encoder g
stop-grad
encoder
encoder g
encoder
Split
encoder f
+6
ARREST
Suzuki
al.
(d) LGBAT Cui et al. 2021
2023
(f) AR-AT (ours)
Figure I: Comparison of invariance regularization-beee based adversarial defense methods. Our approach employs an asymmetric structure for invariance regularization with stop-gre nop-gradient and predictor, and (BN) to maintain consistent batch statistics during training.
2.3 INVARIANCE REGULARIZATION -ASSD
BASED DEFENSE
While AT only inputs AEs during training, invariance regularization-base adversarial defense methods (Zhang et al. 2019 Wang et al. 2019) input both clean and adversarial images to ensure adversarial invariance of the model. TRADES (Zhang et al. 2019) introduces a regularization term on the logits to encourage adversarially invariant predictions; although it allows trade-off adjustment by altering the regularization strength, still suffers from a trade-off. MART (Wang et al. 2019) further improved TRADES by focusing more the misclassified examples to enhance robustness, still sacrificing clean accuracy. Unlike these logit-based invariance regularization methods, our method leverages representation invariance to mitigate the trade-off generalized form of invariance regularization.
Another line of research is employ knowledge distillation (KD) (Hinton al., 2014 based regularization, which has been shown effective in mitigating the trade-off. Cui et al. (2021) proposed LBGAT, which aligns the student model's predictions on adversarial images with the standard model's predictions on clean images, encouraging similarity of predictions between a student and a standardly trained teacher network. More recently, accently,/uzuuki (Suzuki al. (2023) proposed ARREST, which performs representation-bast based KD so that the student model's representations are similar to the standardly trained model's 'ss representations. In contrast these KD-based methods, which enforce adversarial invariance implicitly, we investigate output invariance within a single model, potentially more effective and more memory-efficied during training. Furthermore, we provide a new perspective on the relative success of KD-based methods, which had not been well understood.
Figure. summarizes the loss functions of these methods and compares them with our method.
3 ASYMMETRIC (AR-AT)
ADVERSARIAL TRAINING
In this section, we introduce a novel approach to effectively learn adversarially invariant representation without sacrificing discriminative ability clean images.
straightforward approach to learn adversarially invariant representation is to employ a siamese structured invariance regularization, depicted in Fig. Tal follows:
Lvo a -(So(x') 3 ((fo(x) + Dist(2,2')
3

---

Published as a conference paper at ICLR 2025
where z' is AE generated from using Cross- Cross-Entropy Loss, and s' are the normalized latent representations of z and z', respectively. L is a classification loss and Dist is a distance metric. Although existing approaches typically rely either adversarial or clean classification loss, begin with this naive method to systematically identify and address the underlying issues step by step. Here, we focus on representation-basel eggulaiizatinn since we found that it can be more effective in mitigating the trade-off than logit-based regularization, discussed in Appendix [
However, we identify two key issues in this naive approach: (I) "gradient conflict" between invariance loss and classification objectives, and (2) the mixture distribution problem arising from diverged distributions of clean and adversarial inputs. To address these issues, we propose novel approach incorporating (1) dlop-gradient operation and predictor MLP to asymmetrize the invariance regularization, and (2) a split- Delit-BatchNorm (BN) structure, explained in the following subsections.
3.1
ASYMMETRIZATION VIA STOP-GRADIENT FOR ADDRESSING GRADIENT CONFLICT
Although naive invariance regularization (Eq straightforward approach for learning adver- sarially invariant representation, we observed a "gradient conflict" between invariance loss and classification objectives, as shown Fig. 2 "Gradient conflict" oceurs when the gradients of multi- ple loss functions oppose each other during joint optimization (i.e., for loss functions A and LB. VoLA VeLB < 0). Resolving this issue has been shown to enhance performance in various fields, including multi-task learning (Yu al. 2020) and domain generalization (Mansilla al. (02T). The observed "gradient conflict" suggests that minimizing the invariance loss may push the model toward con-diseriminative directions, conflicting with the classification objectives, thereby degrading the classification performance.
In this work, reveal that asymmetrizing the invariance regularization with stop-gradient operation can effectively resolve "gradient conflict." " The asymmetric invariance regularization is defined as follows:
=v = -(fo=')))))) + 8 [(fo(x).)) + y Dist(z', ,s(()))
where sg(-) stops the gradient backpropagation from z' to z, treating a constant.
Our motivation of employing the stop-gradient operation to eliminate the unnecessary gradient flow from the adversarial representation z' to the clean representation 2, which can lead to the degradation of classification performance. This can be understood by decomposing the invariance loss into two components, as follows:
(Di(( sg(=)) + Dis)(ss(='),=))
(5)
Minimizing the first term, Dist(=', Dist()',sg())), encourages to bring the corrupted adversarial representa- tion z' closer to the clean representation 2 by treating "purification" of representations. In contrast, minimizing the second term, Dist(sg(z') ((((((((()))))))))))()))) z). attempts to bring the clean representation z closer to the potentially corrupted adversarial representation z', encouraging the "corruption" of representations. This can be harmful for learning discriminative representations. Therefore, we hypothesize that minimizing the second term can conflict to the classification objectives, leading to the "gradient conflict."
constant, which can be interpreted as the
3.2 LATENT PROJECTION FOR ENHIANCING TRAINING STABILITY
We employ a predictor MLP to the latent representations to predict 2, inspired by recent self-supervised learning approach (Chen & He (021), following:
Lv2 = a ((fo(z'), y) + [(fo(z),)) v) + 7 Dist(h(2z),
(6)
The predictor MLP h is trained to predict the clean representation from the corrupted adversarial representation z'_ Our intuition is that the adversarial representation varies with each training iteration, as adversarial perturbations depend on both the model parameters and the randomness inherent in the attack process. Therefore, introducing an additional predictor head may help stabilize the training process by handling the variations of adversarial representation through updates of the predictor, preventing large fluctuations of the classifier's parameters caused by variations in a'. In other words. the predictor MLP h functions "stabilizer", preventing the model from being overly distracted by variations in adversarial representations while achieving invariance, thus preventing compromise of classification performance.
4

---

Published as a conference paper at ICLR 2025
3.3 SPLIT BATCH NORMALIZATION FOR RESOLVING MIXTURE DISTRIBUTION PROBLEM
We point out that using the same Batch Norm (BN) (loffe & Szegedy 2015 layers for both clean and adversarial inputs can lead difficulty in achieving high performance due "mixture distribution problem." BNs are popularily used for accelerating the training of DNNs by normalizing the activa- tions of the previous layer and then adjusting them via learnable linear layer. However, since clean and adversarial inputs exhibit diverged distributions, using the same BNs on a mixture of these inputs can be suboptimal for both inputs, leading to reduced performance.
To address this issue, we employ split-BN structure, which uses separate BNs for clean and adversarial inputs during training, inspired by Xie & Yuille (2019); Xie et (2020). Specifically, Eq. is rewritten as follows:
LV3 [(fo(='). + B C(fo (x). v) + y Dist(h(z(() (g(2((()N))
where fauxBN shares parameters with fe but employs auxiliary BNs specialized for clean images, F(6man)) and z(0) are the latent representations of x and x' from fauaBN and So, respectively. In this way, the BNs in fe exclusively process adversarial inputs, while the BNs fanxBN exclusively process clean inputs, avoiding the mixture distribution problem.
Importantly, the split-BN structure is exclusively applied during training: During inference, the model fe is equipped to classify both clean and adversarial inputs with the same BNs. Therefore, in contrast to MBN-AT (Xie & Yuille 2019) and AdvProp (Xie et al. (020) that require test-time oracle selection of clean and adversarial BNs for optimal robustness and accuracy, approach is more practical.
3.4 BLANCB FOR REDUCING HYPERPARAMETERS
Furthermore, employ dynamic adjustment rule for the hyperparameters and 3, which automat- ically controls the balance between adversarial and clean classification loss. Specifically, we adjust and (9 based on the training accuracy of the previous epoch, inspired by Xu et al. (203):
1 N
((1) = AcC( =
(argmax (JoLnaN (x)) = v), B(e) = Acc(e-1)
(8)
1=1
where ((()) and Bity are the hyperparameters at the current epoch t, and Acc(e-1) is the clean accuracy of the model f fanxBN at the previous epoch (t 1). This adjustment automatically forces the model to focus more on adversarial images as its accuracy on clean images improves. We empirically demonstrate that this heuristic dynamic adjustment strategy works well in practice, successfully reducing the hyperparameter tuning cost. We provide an ablation study in Sec. 5.5
3.5 MULTI-LEVEL REGULARIZATION OF LATENT REPRESENTATIONS
Finally, our method is extended to regularize multiple levels of representations. Specifically, we employ multiple predictor MIPs hi...., hy to enforce invariance across multiple levels of repre- sentations 21, 2y
follows: 1, E Dist(hr(2/g (=(0),))))))))))))))( sg( (2(8maN)) We found that regularizing
multiple layers in the later stage of network is the most effective, as demonstrated in our ablation study in Sec. 5.5
4 EXPERIMENTAL SETUP
Models and datasets. We evaluate our method on CIFAR-10, CIFAR-100 (Krizhevsky & Hinton 2009), and Imagenette (Howard 2019) datasets. We use the standard data augmentation techniques of random cropping with pixels of padding and random horizontal flipping. We use the model architectures of ResNet-18 (He al. 2016) and Net-33e (WRN-34-10 (Zagoruyko & Komodakis 2016), following the previous works (Madry et al.) 2018 Zhang et al. 2019 Cui et al.
2021). Implementation details of baselines provided in Appendix A.3
Evaluation. We use 20-step PGD attack (PGD-20) and AutoAttack (AA) CCroce & Hein 2020) for
evaluation. The perturbation budget is set 8/255 with 7 -nrm (Appendix B.4 shows that
5

---

Published as a conference paper at ICLR 2025
Table 1: Effectiveness of Stop-grad and Split-BN on naive invariance regularization (Lvo). Split- BN consistently improves the robustness-accuracy trade-off. Stop-grad improves this trade-off, particulary when Split-BN addresses the "mixture distribution problem."
Stop-grad
Split-BN
Clean
AA
Grad-sim.
82,93
46.50
0.06
ResNet-18
82.47
45.21
0.68
CIFAR10
84,35
47.96
0.14
85.51
49.30
0.59
86.25
41.17
0.03
86 60
42.0
0.31
WRN-34-10
87.13
47.31
0.01
88.27
47.98
0.23
60.23
19.66
0.00
61.55
20.15
0.58
CIFAR100
ResNet-18
62.41
22.45
-0.03
67.04
22.25
0.40
61.70
20.72
-0.01
61.82
19.76
0.38
WRN-34-10
62.72
24.32
0.00
67.87
24,46
0.22
AR-AT is also effective for lz-bounded scenarios). The step size is set to 2/255 for PGD-20. We compare our method with the standard AT, and existing regularization-basee methods TRADES, MART, LBGAT, and ARREST, Comparisons with other methods are provided in Appendix [B.]
Training details. We use a 10-step PGD for adversarial training. We initialized the learning rate to 0.1, divided it by factor of 10 at the 75th and 90th epochs, and trained for 100 epochs. We use the SGD optimizer with a momentum of 0.9 and weight decay of 5c-4, with a batch size of 128. Cosine Distance is used as the distance metric for invariance regularization. The predictor MLP has two linear layers, with the hidden dimension set to 1/4 of the feature dimension, following SimSiam (Chen & He (02T). The latent representations to be regularized are spatially average-pooled to obtain one-dimensional vectors. The regularization strength is set to 30.0 for ResNet- 18 and 100.0 for WRN-34-10. We regularize all ReLU outputs in "layer4" for ResNet- 18, and "layer3" for WRN-34-10. Computational cost of AR-AT is smaller than TRADES and LBGAT (Appendix G
EMPIRICAL STUDY
5.1 ANALYSIS AND UNDERSTANDING OF EACH COMPONENT OF AR-AT
In this section, we analyze and understand the effectiveness each component of our method, focusing on the stop-gradient operation, the split-BN structure, and the predictor MLP. Based on the naive invariance loss (Eq. 3). we add each component step by step to identify the underlying issues and address them systematically. Here, for simplicity, omit the auto-balas aatonbalance heuristic and the multi-level regularization of latent representations, fixing the regularized layer to the last layer of the network for this analysis.
stop-gradient operation resolves "gradient conflict." " In Fig. 2 we plot the cosine similarity between the gradients of the classification loss (the sum of clean and adversarial classification loss) and the invariance loss with respect to 0 during training. The proportion of parameters experiencing gradient conflict in Fig. [p. We observe that with the naive invariance regularization (Cvo: Eq. 3). the gradient similarity fluctuates near zero, and over 40% of parameters experience gradient conflict, suggesting suboptimal convergence of both classification and invariance loss. To verify our hypothesis that the second term of Eq. 5] D(sg(='), z). conflicts with the classification loss, we isolate and minimize either the first or second term. We observe that minimizing only ((sg(="), results in strong gradient conflict. In contrast, minimizing only D(2', sg())) (Cv1; Eq. 4 alleviates gradient conflict. This demonstrates the effectiveness of eliminating unnecessary gradient flow from the adversarial clean representations, preventing their corruption and mitigating "gradient conflict."
However, Tab. shows that mitigating "gradient conflict" does not necessarily improve the robustness and accuracy. Only in the cases of CIFAR-10/WRNY 10//000///--A 34-10---00 10 and 1 18, using the stop- -aaddrradentt operation alone improved the robustness-accuracy trade-off. This suggests that the stop-gradient operation by itself is insufficient, which we address next.

---

Published as a conference paper at ICLR 2025
Lyu Naive] D) (D(2 D()g(
Lyyi wi stop-grat grad]
Di sg()
(sgg(F))
11
04.3
Lyai Naive
Lyp1 stop-grad
Grd--im.
0.25
Conflctrate rato
Avfe "
0.1
0.7
10.20
00.00
10000
20000
3006
40000
10003
20000
30000
4KC0
Training
Training stops
(b) The proportion of parameters
(1) Gradient similarity
30
30
106
experiencing gradient conflict
Epoch
Figure 2: Gradient conflict between classification loss and different invariance losses, w.r.t. 0. With the naive invariance loss (Lvo), (a) gradient similarity is near zero, and (b) over 40% of parameters experience gradient conflicts. The term D(sg(z'), 2) causes strong conflit (orange line). In contrast, our approach (Ivv1) to only minimize D(z', sg(2)) effectively resolves the conflict (blue line).
Figure 3: Mixture distribution prob- lem, indicated by the L2 distance be- tween adversarial and clean feature, ||z 2/112. While Lvo already suffers from this problem, the use of stop-grad exacerbates this issue by weakening variance regularization.
prod.
ARAT
5.0-
GooDisttrg
Table 2: Effectiveness predictor MLP head the naive invariance regularization (Cvo). using ResNet-18 (C10/100 refers CIFAR-10/100).
16-
ARAT (wole prod. B.)
ARAT
1090
Epech
Training stage
Stop-grad Split-BY
Pred.
Clean
AA
(a) Cosine distance between (b) The gradient norm the loss adversarial representations at function (11)).
C100 C10
85.51
49.30
epochs f and + 1.
v
87.23
49.30
67.04
22.25
Figure 4: Predictor MLP head improves the training stabil- ity. We observe that the predictor MLP stabilizes the updates of the representations where invariance loss is applied, and reduces the gradient norm.
67.64
22.42
Split-BN resolves the mixture distribution problem. Another issue in invariance regularization stems from using the Batch Norm (BN) layers for both clean and adversarial inputs. Tab. demonstrates that using split-BN structure consistently improves the robustness and accuracy. Impor- tantly, Tab. demonstrates that simultaneously resolving issues of "gradient conflict" and the mixture distribution problem leads to substantial improvements.
To understand the mixture distribution problem, analyze the similarity between adversarial and clean features. In Fig. we plot the L2 distance between the adversarial and clean features, ||z-2'|la for the same input x. The increasing distance over time indicates that adversarial and clean features become more dissimilar during training, illustrating a robustness-accural trade-off. This suggests that shared BN layers between clean and adversarial inputs may fail to accurately estimate adversarial feature statistics (i.e., mixture distribution problem). Appendix F.2 demonstrates that the estimated batch statistics become more stable with the use of split-BN.
Notably, observe that the stop-graad moopgradientt operation exacerbates this issue by weakening the invari- ance regularization, as removes the second term in Eq. 5 This explains why the stop-grad alone does necessarily improve the vobustness-accuraan trade-off, but does when combined with split-BN, as shown in Tab.
Latent projection with predictor MLP head enhances performance by improving training stability. Finally, we analyze the effectiveness of the predictor MLP head. Tab. 2 shows that employing the predictor ML. head (Lv1 Lv2) leads further improvements in the robustness
7

---

Published as a conference paper at ICLR 2025
Table Comparison with invariance regularization-basee based defense methods on CIFAR datasets. We report clean and robust accuracy (AutoAttack; AA).
Defense
CIFARIO
CIFAR 100
Clean
AA
Sum.
Clean
AA
Sum.
83.77
42,42
126.19
55.82
19.50
75.32
ResNet- 18
TRADES
81.25
48.54
129.79
54.74
23.60
78.34
MART
82.15
47.83
129.98
54.54
26.04
80.58
LBGAT
85,00 =
48.85
133.86
65.87 = 0.74
23.19 +
89.07 0.73
ARREST
86.63
46,14
132.77
AR-AT (ours)
87.82 =
49.02
136.84
67.51 = 0.13
23.38 +
90.89
AR-AT+SWA (ours)
86.44 =
50.28
136.72 0.19
67.17 = 0.18
24.36 +
91.53=
AT
86.06
46.26
132.32
59.83
23.94
83.77
WRN-34-11 10
TRADES
84,33
51.75
136.08
57.61
26.88
84.49
MART
86.10
49.11
135.21
57.75
24.89
82.64
LBGAT
88. 19 = 0.11
52.56 + 0.34
140.75 0.34
68.17 0.56
26.92 + 0.32
95.09 0.64
ARREST
90,24
50.20
140.44
73.05
24.32
97.37
AR-AT (ours)
90.89 = 0.22
50.77 + 0.50
141.66 + 0.51
72.51 + 0.5L
24.18 + 0.46
96,70 0.56
AR-AT+SWA (ours)
90.06 =: 0.08
54.03 + 0.31
144.09 0.39
72.37 + 0.13
27.31 + 0.13
99.69 0.26
Table 4: Comparison with defense methods Imagenette. (The learning rate of LBGATT is lowered from default due to gradient explosion.)
Table 5: Representation invariance analysis.
"Cos. sim.": cosine similarity between adversarial and clean features.
Defense
Imnugeettle AA
Defense
CIFARIO
Clean
Sum.
Sum.
Cos. sim.
AT
84.58
52.25
136.83
AT
126.19
0,9423
RosNet-18
TRADES
79.21
53.98
133.19
ResNet-18 18
TRADES
129.79
0.9693
MART
84.07
59.89
143.96
MART
129.98
0.9390
LBGAT
80.80
50.42
131.22
LBGAT
133.86
0.9236
AR-AT (ours)
88.66
59.28
147.94
AR-AT (ours)
136.84
0.9450
and accuracy. In Fig- analyze the training stability. Fig. shows the cosine distance between the adversarial representations at epochs t and t + 1. where the predictor MLP stabilizes the updates of the representations where invariance loss applied. This suggests that the predictor MLP head helps stabilize the optimization of the invariance regularization by smoothing the updates of the representations. Fig. 4 reveals that, indeed, the predictor MLP reduces the gradient norm of the loss function ( ( L 2)), implying more stable optimization.
5.2 COMPARISON WITH SENTE-OF-TEEAAEE INVARIANCE REGULARIZATOIN-BE DEFENSES
In this section, evaluate the complete version of AR-AT, which employs all components outlined in Sec. 3 The ablation study for the auto-balance heuristic and the multi-layer invariance regularization provided in Sec. 5.5
Tab. 3 shows the results of ResNet- 18 and WRN-34-10 trained on CIFAR-10 and CIFAR-100 compared with existing invariance regularization-bssed defense methods. Following common practice in existing works (Sitawarin et al.) 2021 Suzuki et al. (2023), we also report the sum of clean and robust accuracy against AA (i.c., Sum. for the trade-off metric. We include the trade-off plots in Appendix B.2. We observe that our method achieves much better performance than the baselines of regularization-be based defenses TRADES and MART on both datasets, highlighting the effectiveness of our methodology in employing invariance regularization. Moreover, our method achieves the best trade-off in most cases, outperforming LBGAT and ARREST, the saate-of-hhe-an invariance coallarization-baseee defenses. Additionally. we found that combining AR-AT with stochastic weight averaging (SWA) (Izmailov et al. 2018) enhances its performance, achieving the best results. Tab, shows the results Imagenette, dataset with high-resolution images, and demonstrate that our method achieves the best performance. These results highlight that AR-AT not only introduces novel
Results with are directly copied from original papers. We report the mean and standard deviation of the results over five for LBGAT and AR-AT.
8

---

Published as a conference paper at ICLR 2025
Cleen (%)
(baselin AT;
Naive Inc. Loss [6q.3]
AR-AT SplicUN (Eq
PGD-20 ACC. (%0
(None)
convt.
layer?
layer?
leyer3 sayers (M0))
(a) Naive invariance
(b) AR-AT (Cv3; Eq.
Shared layers
regularization (Cvo:
Eq. 3
Figure 6: Comparison of "sharing parameters vs. separate networks" in AR-AT for adversar- ial and clean branches. The x-axis represents the depth until which parameters are shared: the leftmost corresponds to completely separate net- works for adversarial and clean branches, while the rightmost shares all parameters (default).
Figure 5: Visualization of clean representa-
tions for (a) naive invariance regularization (Lvo.
Eq. 3 and (b) AR-AT (Cy3. Eq. 7 using UMAP MMelnnes et al., 2018). 2000 images are
randomly sampled from the CIFARIO test set.
perspectives to invariance regularization but also achieves state-of-the-an performance in mitigating the robustness-accure trade-off.
Note that the trade-off can be adjusted by simply changing the perturbation size the adversarial training: In Appendix [33] we show that AR-AT with 9/2555 achieves both higher clean and robust accuracy than LBGAT at the same time.
5.3 ANALAYSIS OF LEARNED REPRESENTATIONNS: ADVERSARIAL INVARIANCE AND DISCRIMINATIVE ABILITY
Quantitative analysis. Tab. 5 presents representation invariance measured by cosine similarity between adversarial and clean features extracted from the penultimate layer. While TRADES achieves high adversarial invariance, it sacrifices discriminative ability, as shown by its low accuracy. Conversely, LBGAT, KD-based method, achieves high performance but low adversarial invariance, demonstrating that using separate teacher network for regularization does not ensure invariance within model. On the other hand, AR-AT achieves both high adversarial invariance and high accuracy simultaneously. We attribute this to effectively addressing the "gradient conflict" and the mixture distribution problem invariance regularization, while imposing the invariance regularization directly the model itself, which is different from KD-based methods.
Visualization of learned representations. Fig. 5 visualizes the clean representations for (a) naive invariance regularization (Cvo. Eq. 3 and (b) AR-AT (Lv3, Eq. 7. using UMAP (Mcinnes et al.) 2018 Naive invariance regularization (Cvu) leads to relatively non-diseriminative representations, likely due to the model's pursuit of adversarial invariance; In fact, the cosine similarity between adversarial and clean features was 0.9985, highest among all methods in Tab, 5 In contrast, AR-AT learns more discriminative representations than the naive invariance regularization.
5.4 ON THE RELATION OF AR-AT TO KNOWLEDGE DISTILLATION (KD)-BASED DEFENSES
Here, we offer novel perspective on the effectiveness of KD-based defenses, such as LBGAT and ARREST. In this section, the hyperparameters a and set to 1.0, and the penultimate layer is used for regularization loss for simplicity.
Using separate networks can relieve the "gradient conflict" and the mixture distribution problem. Fig. 6 shows the impact of using "shared weights vs. separate networks" between adversarial and clean branches. We observe that "Naive Inv. Loss (Eq. ()q.3)" and "AR-AT w/o Split- BN (Eq. 6) struggle to attain high robustness when weights are shared but perform better with separate networks. We hypothesize that using separate networks (1) acts similarly to the stop- gradient operation by eliminating gradient flow between branches. and also (2) resemble split- BpNit-BNs BNs in avoiding the mixture distribution problem. In contrast, AR-AT (Eq. 7 maintains performance when layers are shared: Importantly, it improves performance when layers shared, indicating its

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of AR-AT "with vs. with- Table Choice of regularized layers in AR-AT out auto-balance (Eq. 8 and / on CIFAR-10.
of hyperparameter
on CIFAR-10.
Regularized layers
Clean
PGD-20
(a,, B)
ResNet-18
WRN-34-10
Penultimate layer ("kayerd")
86.28
51.78
Clean
PGD-20
Clean
PGD-20
WR ReM
"Layerr", "layens"
87.35
52.27
(1.0, 1.0)
88.91
51.75
91.39
49.17
All BasicBlocks in "layerd"
87.16
52.27
All ReLUs in "layerd"
87.93
52.13
0.50
88.55
52.12
91.48
49.01
(1.0.0.1)
87.55
52.27
90.80
49.46
Penultimate layer ("Kayer3")
88.87
52.50
(0.5. 1.0)
89.01
51.37
91.71
50.35
"laver2", "layer3" All BasicBlocks in "layerd"
88,95
50.74
1.01
89.08
46.73
91.85
48.51
89.50
51.76
auto-bal.
87.93
52.13
90.81
52.72
All ReLUs in "layer3"
90.81
52.72
superiority over KD-based regularization. This advantage of sharing layers may arise from more explicit invariance regularization compared to using a separate teacher network.
5.5 ABLATION STUDY
Asymmetrizing logit-based regularization. We also explore the benefits of asymmetric structure for logit-based regularization, detailed in Appendix D We find that TRADES also faces the "gradient conflict" and the mixture distribution problem, and asymmetrizing TRADES leads significant improvements in robustness and accuracy, demonstrating the consistency of our findings.
Effectiveness of hyprrarraeeeer auto-balance. Tab. 6 shows an ablation study of AR-AT "with vs. without auto-balance (Sec. 3.4) of the hyperparameters and 88 which controls the balance between adversarial and clean classification loss. The results demonstrate that auto-balancing performs comparable to the best hyperparameter manually obtained. Additionally, in the case of WRN-34-10. 10, auto-balancing outperforms the best typerparameter setting in Tab. suggesting that it can discover superior hyperparameter configurations beyond manual tuning.
Layer importance in invariance regularization. Tab. shoows the ablation study of AR-AT on regularizing different levels of latent representations. ResNet-18 and WRN-34-10 have four and three "layers", ,, respectively. "Layers" are composed of multiple "BasicBlocks" ", which consist of multiple convolutional layers followed by BN and ReLU. We show that regularizing multiple latent representations the later stage of the network can achieve better performance. In practice, we recommend regularizing all ReLU outputs in the last stage of the network, which is the default of AR-AT. However, we note that the choice of layers for invariance regularization is not critical as long as the later-stage latent representations are regularized.
Table 8: Importance of BatchNorm. We compare AR-AT on CIFARIO using ResNet-18, using main BNs auxiliary BNs during inference.
Importance of batchNorm for robustness. Tab, g presents the performance of AR-AT when the auxiliary BNs are used during inference. Intriguingly, despite sharing all the layers except BNs between fo and f faixBN using the auxiliary BNs during inference notably en- hances the clean accuracy while compromising robust- ness significantly. This indicates that the auxiliary BN are exclusively tailored for clean inputs, highlighting the critical role of BNs in adversarial robustness.
Defense
Clean
PGD-20
AR-AT (fo)
87.93
52.13
AR-AT ( (PUEAN)
92.84
1.98
6 CONCLUSION
We explored invariance regularization-basse adversarial defenses to mitigate trade-off. By addressing two challenges of the "gradient conflict" and the "mixture distribution prob- lem." our method, AR-AT, achieves the state-of-the gradient conflict between classification and invariance loss is resolved using stop-gradient operation, while the mixture distribution problem is mitigated through split-BN structure. Additionally, provide a new perspective on the success of KD-based methods, attributing it to resolution of these challenges. This paper provides a novel perspective to mitigate the robustness-accuracy trade-off.
accuracy
performance of mitigating the trade-off. The
Limitations. While AR-AT demonstrates the effectiveness of representation-llvel regularization, determining the appropriate layers for regularization remains ambiguous, especially for novel model architectures. Additionally, the proposed method may not be directly applicable to models without BNs, such as Transformers, which use Layer-Norm (Ba et al. 2016), and whether split-BN structure can be extended to Layer-Norm is interesting future direction.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. This work was partially supported by JSPS KAKENHI Grants JP2 JP21H04907 and JP24H00732, by IST CREST Grants JPMJCR 18A6 and JPMJCR20D3 including AIP challenge program, by JST AIP Acceleration Grant JPMJCR24U3, and by JST K Program Grant JPMJKP24C2 Japan.
Ethic statements. We only use publicly available datasets, and do not involve any human subjects or personal data. Our work does not include harmful methodologies or societal consequences, but rather, aims improve the robustness of machine learning models. We do not have any conflicts of interest or sponsorship to disclose. We have followed the ethical guidelines and research integrity standards in our work.
Reproducability. We provide the hyperparameters used in our experiments in Sec. 4 We also provide the additional implementation details of our method and the baseline methods in the appendix (Appendix A.2 and Appendix A.3). The code will be made available upon publication.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15750-15758, 2021.
Francesco Croce and Matthias Hein, Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-221111 PMLR, 2020.
Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia. Learnable boundary guided adversarial training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 15721-15730 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Workshop on Advances in neural information processing systems, 2014.
Jeremy Howard. A smaller subset of 10 easily classified classes from imagenet, and a little more 2019.
french. http:s/////gttuub..o/ffaaaasaaa
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 pmlr, 2015.
Pavel Izmailov. Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint arXiv: / 1803,800000 2018.
Gaojie Jin, Xinping Yi, Wei Huang, Sven Schewe, and Xiaowei Huang. Enhancing adversarial training with second-order statistics of weights. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 15273-15283, 2022.
Gaojie Jin, Xinping Yi, Dengyu Wu, Ronghui Mu, and Xiaowei Huang. Randomized adversarial training via taylor expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16447-16457, 2023.

---

Published as a conference paper at ICLR 2025
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Qian Li, Yuxiao Hu, Yinpeng Dong, Dongxiao Zhang, and Yuntian Chen. Focus on hiders: Exploring hidden threats for enhancing adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24442-24451, 2024.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.
Lucas Mansilla, Rodrigo Echeveste, Diego H Milone, and Enzo Ferrante. Domain generalization via gradient surgery. In Proceedings of the IEEF/CVF international conference computer vision, pp. 6630-6638, 2021.
Leland McInnes. John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rXiv:1802.03422 2018.
Chawin Sitawarin, Supriyo Chakraborty, and David Wagner. Sat: Improving adversarial training via curriculum-based loss smoothing. In Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security, pp- 25-36, 2021.
Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, et al. Towards efficient and effective adversarial training. Advances Neural Information Processing Systems, 15::1881-11833 2021.
Satoshi Suzuki, Shin'ys ya Yamaguchi, Shoichiro Takeda, Sekitoshi Kanai, Naoki Makishima, Atsushi Ando, and Ryo Masumura. Adversarial finetuning with latent representation constraint to mitigate rccuracy-robusttess tradeofT. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4367-4378. IEEE, 2023.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumne, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning representations, 2019.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. Advances in Neural Information Processing Systems, 2-299999299, 2020.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. arXiv preprint arXiv:1906.03787 2019.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF conference on computer vision and putlern recognition, pp. 819-828, 2020.
Xilie Xu, Jingfeng Zhang, and Mohan Kankanhalli. Autolora: A parameter-free automated robust fine-tuning framework. urXiv preprini arXiv:2310.018) 2023.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 475 2016.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2025
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pp. -1287711887 PMLR, 2020.
Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2 04928, 2021.
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENTS DETAILS
A.1 DATASETS
Here, we provide the details of the datasets we used in our experiments, CIFARIO and CI- FARIO0 (Krizhevsky Hinton 2009) are standard datasets for image classification with 10 and 100 classes, respectively, with resolution of 32 x 32, Imagenette (Howard 2019) is subset of 10 easily classified classes from Imagenet (Deng al. 2009p. In our experiments, we used the version with the resolution of 160 x 160.
Table 9: The details of datasets we used in our experiments.
Dataset
Resolution
Class Num.
Train
Val
CIFARIO
32 32
10
50,000
10,000
CIFAR100
32 x
100
50,000
10,000
Imagenette
160 x 160
10
9,469
3,925
A.2 EXPERIMENTS COMPUTE RESOURCES
In this work, we use NVIDIA A100 GPUs for our experiments. Training of AR-AT on CIFARIO using R8 takes approximately 2 hours, and CIFARIO using WideResNet-34-10 takes approximately 10 hours.
A.3 IMPLEMENTATION DETAILS OF BASELINE METHODS
Here, we describe the implementation details of the baseline defense methods.
Adversarial Training (AT) (Madry et al. 2018): We use the simple implementation by
Dongbin Na 3 We aligned the hyperparameter setting described in the official GitHub
repository
TRADES Zhang et al. hyperparameter setting.
2019): We use the official implementation and used the default
MART Wang et al. hyperparameter setting.
2019 We use the official implementation and used the default
LBGAT (Cui et al. 2021): We use the official implementation and used the default hyperparameter setting. For Imagenette, we observed that the default learning rate causes gradient explosion, so we lowered the learning rate from 0.1 to 0.01. Note that LBGAT uses a ResNet-18 teacher network for both ResNet-18 and WRN-34-10 student networks.
2023): They do not provide an official implementation, so we directly copied the results in Tab. from their paper.
ARREST (Suzuki al.,
The comparison of loss functions is summarized in Tab, 10
Stwpeesarrrrrraiainn-nnCC-- Training-CIFAR
"https://gttub..co http://hhhttttbooooavaaaaaaaaaas
http:://gthub..cm/yaaddoogguuu
14

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of loss functions defense methods. fa(.) the predictions of the trained model, and 2(0) and {()) is the latent representation of the model fo on clean and adversarial inputs, respectively. LBGAT and ARREST are knowledge distillation-basee methods using teacher model Is- Our method AR-AT uses both clean and adversarial classification losses and employs representation-basee regularization (Details in Sec.
Method
Classification Loss
Regularization Loss
Adversarial
Clean
AT
CE((o(e'),))
TRADES
CE(J(()).))
KL(((=))\l<(aa') [L((((=))) (1- fofo()),))
MART
BCE(fo(2").M)
LBGAT
CE(ga(a).v)
MSE(Jo ). (g((a))
ARREST
CE(fo(z'),)
Cos (z(0) ((0)), where Jo is fixed std. model
AR-AT (ours)
C'((J(e'))))
CE (z).))
CosSim(h(2//0 sg(=(amuNN))
B ADDITIONAL PERFORMANCE COMPARISONS
B. COMPARISON WITH STATE-OF- THE-ARY DEFENSE METHODS
Table 11:
Comparison of clean accuracy and robust accuracy against AutoAttack (AA) for WideResNet-34-10 trained on CIFAR1O. The compared methods are sorted by the sum of clean and robust accuracy.
Method
Clean
AutoAttack
Sum.
Reference
AT Madry et al.
2018
86.06
46.26
132.32
Reproduced
FAT (Zhang et al 2200
89.34
43.05
132.39
Copied from Suzuki et al.
2023
MART Wang
86.10
49.11
135.21
Reproduced
TRADES (Zhang
2019
84.33
51.75
136.08
Reproduced
NuAT2 [Srramanan et all 2021
84.76
51.27
136.03
Copied from the original paper
IAD (Zhu el al. 2021
85.09
52.29
137.38
Copied from the original paper
TRADES Rand Jin el al.
2023
85.51
54.00
139.51
Copied from the original paper
AWP Wu et all
2020
85.57
54.04
139.61
Copied from the original paper
S Uin et al.
2022
85.67
54.10
139.77
Copied from the original paper
ARREST Suzukki et al.
2023
90.24
50.20
140.44
Copied from the original paper
LBGAT (Cui
2021
88.28
52.49
140.77
Reproduced
NuAT2+WA (Sirrmanna al.
2021
86.32
54.76
141.08
Copied from the original paper
AT+HF (Li al. 20244)
87.53
55.58
143.11
Copied from the original paper
TRADES#AWP-HR Rand (Jin et all
2023
86.10
57.10
143.20
Copied from the original paper
AR-AT (ours)
90.81
50.77
141.66
(Ours)
AR-AT+SWA (ours)
90.06
54.03
144.09
(Ours)
15

---

Published as a conference paper at ICLR 2025
Table 12:
Comparison of clean accuracy and robust accuracy against AutoAttack (AA) for WideResNeet34-100 trained on CIFAR100. The compared methods are sorted by the sum of clean and robust accuracy.
Method
Clean
AutoAttack
Sum.
Reference
MART (Wang al.
2019
57.75
24.89
82.64
Reproduced
AT Madry al. 201
59.83
23.94
83.77
Reproduced
TRADES (Zhangg
a
2019
57.61
26.88
84.49
Reproduced
FAT (Zhang et al.
2020
65.51
21.17
86.68
Copied from (Suzuki et al. 2023
AT+HI (Li et al
2024
58,99
28.65
87.64
Copied from the orgginal paper
IAD (Zhu al. 2021
60.72
27.89
88.61
Copied from the original paper
TRADES+HF
2024)
58.70
30,29
88.99
Copied from the original paper
AWP (uu et al. 2020)
60.38
28.86
89.24
Copied from the original paper
TRADES+ Rand
Jmn of al.
2023
62.93
27.90
90.83
Copied from the original paper
S (Jin al. 2022
63.40
27.60
91.00
Copied from the original paper
TRADES+AW Rand (Jin et al.
2023
64.71
30,20
94.91
Copied from original paper
LBGAT CCui et al. 2021
69.26
27.53
96.79
Reproduced
ARREST (Suzuki et al.
2023
73.05
24.32
97.37
Copied from the original paper
AR-AT (ours)
72.23
24.97
97.20
(Ours)
AR-AT+SWA (ours)
72.41
27.15
99.56
(Ours)
We provide more comparison with state-of-the-art defense methods Tab. II and Tab. 12 We observe that AR-AT achieves high clean accuracy while maintaining high robustness. These results demonstrate the effectiveness of AR-AT in mitigating the robustness-accure trade-off.
B.2 TRADE-OFF PLOT WITH ARDIST
In this section, visualize the trade-off between robustness and accuracy, we plot them for AR-AT and the baseline methods. We also added URDist(Suzuks et al. (2023), which uses existing methods to approximate a curve representing the accuracy-robustnes trade-off. Methods positioned towards the top-right are better, they achieve both high robustness and accuracy. Our method, ARAT, consistently appears on the top-right side across most scenarios. The absence the ARDist curve in the ResNet-18 18 plots is due it being positioned far in the top-right corner, exceeding the plot's range. This occurs because the ARDist curve calculated using the best models for each method, typically from the Wede family.
16

---

Published as a conference paper at ICLR 2025
CIFARIO: ResNet-18
CIFARIO WRN-34-10
56
(ours) AR-AT+SWA
(ours) SR-AT+SWA
(ours) AR-AT
54-
(AA)
MARI
Robnst Accunayy (AA)
TRADES
48
52
(ours) AR-AT ARRES
ARREST
MANT
44
46
AT
46
ARD
Clean Accuracy
Clean Accuracy
Figure 7: Trade-off plot CIFARIO with ResNet- 18.
Figure 8: Trade-off off plot CIFAR1O with WideResNet-344
CIFARTO0 ResNet-18
CIFARIO0: WRN:34-10
29
MART
28
(ours) AR-AT+SWA
(ours) AR-AT SWA
YFRADES
LBGAT
(AA)
Robust Accuracy (AA)
27-
24
TRADES
LBGAT (ours) AR-AT
ARREST
24
(ours) AR-AT
20
23-
ARDist
22
Clem Accunccy
Clean Accunecy
Figure 9: Trade-off plot on CIFARI00 with ResNet-18 18.
Figure 10: Trade-off plot on CIFAR100 with WideResNNtt34-10.
B.3 ADJUSTING TRADE-OFF IN AR-AT
To adjust the robusness-accuracy trade-off in AR-AT, the streightforward way is change the perturbation size of AEs (c) in the adversarial training. By adjusting the perturbation size, in Fig. [3] show that the off between clean accuracy and robust accuracy be adjusted. We show that by adjusting the perturbation size, AR-AT achieves both higher clean accuracy and robust accuracy than LBGAT, and ARAD
17

---

Published as a conference paper at ICLR 2025
Table 13: Trade-off between clean accuracy and robust accuracy by adjusting the perturbation size for adversarial training (c) in AR-AT. In the REFARRO/WideRsNNer. LBGAT and TRADES-ANE AWP for both clean and robust accuracy.
setting, we outperform
Dataset
Model
Method
eps (x/255)
Clean
AA
LBGAT
(default)
88.28
52.49
89.36
50.67
87.76
52.63
CIFARIO
WRN-34-10
TRADES-AWP
86.79
53.92
(default)
84.88
55.35
(default)
90.81
51.19
ARAT
9
89.83
52.60
B.4 L2- L2-BOUNDED ADVERSARIAL TRAINING
We provide evidences that AR-AT also works in T.2-bounded adversarial training scenarios. The perturbation size is set to = 0.5 for Tzz-bounded adversarial training. Compared with the baseline L2-bounded adversarial training (AT*), our proposed method, ARAT, shows better clean accuracy and robust accuracy against AA on both CIFARIO and CIFAR 100 datasets.
Table 14: Comparison of La-bbuudee adversarial training on CIFARIO and CIFAR100 datasets. We compare the clean accuracy and robust accuracy against AutoAttack (AA).
Dataset
Model
Method
Clean
AA
CIFAR1O
PreAct ResNet-18
AT*
88.91
65.93
ResNet-18
AR-AT
92.10
68.25
CIFARTOO
PreAct ResNet-18
RT
60.50
35.27
ResNet-18
AR-AT
73.86
37.92
B.5 ABLATION STUDY: CONTRIBUTIONS OF EACH COMPONENT IN AR-AT
Tab. shows the abltation study of three components in AR-AT: stop-gradient operation, predictor MLP, and split-BN. The results demonstrate that Split-BN consistently improves the robustness and accuracy compared the naive invariance regularization (i.e.. (1) in Tab. 15 by mitigating the mixture distribution problem. On the other hand, we observe that mitigation of "gradient conflict" does not necessarily improve the robustness and accuracy (i.e., (2) and (4) in Tab. [5]: it is explicitly effective when the mixture distribution problem is resolved by split-BN (i.e., (8) in Tab. 15). Therefore, we conclude that resolving both issues of "gradient conflict" and mixture distribution problem important to achieve high robustness and accuracy.
Table 15: Contributions of each component in AR-AT, for ResNet- 18 trained on CIFARIO. "Grad- sim." represents the average gradient similarity between the classification loss and invariance loss with respect to 0 during training. Here, the penultimate representation is regularized.
Method Pred.
Stop-grad
Split-BN
Clean
PGD-20
Grad-sim.
(1) Lva
82.93
48.89
0.06
(2) Lvi
82.47
48.57
0.68
(3)
83.49
48.22
0.00
(4) Lv2
83.00
49.14
0.63
(5)
84.35
51.34
0.14
(6)
85.51
51.46
0.59
(7)
84.07
50.79
0.00
(8) Lv3
V
86.29
52.40
0.52
18

---

Published as a conference paper at ICLR 2025
B.6 HYPERPARAMETER STUDY: STRENGTH OF INVARIANCE REGULARIZATION
Tab. 16 shows the ablation study of AR-AT on the strength of invariance regularization We observe that the performance is not too sensitive to the strength of invariance regularization 7. Nevertheless, the optimal y depends on the architecture. For example, 7 30.0 is optimal for ResNet-18, while
y = 100.0 is optimal for WRN-34-10.
Table 16: AR-AT's sensitivity to the strength of invariance regularization
CIFARIO
CIFAR100
Y
Clean
PGD-20
Clean
PGD-20
10.0
87.90
51.53
66.89
26.79
WRN-344 14N-34-10 ResNet- 18
30.0 (default)
87.93
52.13
68.07
26.76
50.0
87.54
51.27
67.72
26.52
100.0
87.06
50.14
66.19
26.02
120.0
86.46
50.15
65.31
25.71
10.0
89.21
48.71
67.46
26.86
30.0
90.93
49.68
70.69
50.0
91.36
50.51
71.52
26.46
100.0 (default)
91.29
52.24
71.58
28.06
120.0
91.39
51.94
71.92
27.31
C ADDITIONAL EXPLANATION ON GRADIENT CONELICT
In addition to the main text, we provide step-by-step explanation on why the second term of Eq. Dist(sg(2'). 2)), causes gradient conflict.
1. Adversarial Representations are Perturbed: Adversarial representations 2' are perturbed from clean representation due input perturbations 8. For example, with single linear layer of weights W.
2 WT x, = WT (x+0)= W W
(9)
(10)
The gap between z and z' tends to increase with deeper layers.
2. Role of Stop-Gradient (sg): The stop-gradient operation treats representations constant in the invariance regularization loss.
3. Minimizing the first term, (ist(2(,sg(2)) This aligns adversarial representations z' closer to clean representations z. Since is corrupted with W minimizing this term "purifies" the representation by reducing W 8 to zero, mitigating the impact of adversarial noise d on the learned representations. This ensures that meaningless input-space perturbations do not alter the underlying semantics.
Minimizing the second term, Dist(sg(z'), 2))) This aligns clean representations 2 closer to corrupted adversarial representations z'. Specifically, is pulled toward z' + W where W 5 unpredictable from z due to dependence on unknown This "contaminates" 2, resulting in the unnecessary corruption of learned representations.
Conclusion: Minimizing Dist(sg(2)) 22)) 2) can harm classification objectives by corrupting learned representations. This leads to conflict between clean and adversarial objectives.
19

---

Published as a conference paper at ICLR 2025
D ASYMMETRIZING LOGIT-BASED REGULARIZATION: IMPROVING TRADES AND ITS RELATION TO LBGAT
1.0
0.8
Gradient similarity
0.2
-0.4
TRADES Asym-TRADES
-0.6
5000
10000
15000
20000 25000 30000 35000 40000
Training steps
Figure 11: Comparison of gradient similarity between "TRADES vs, Asymmetrized-TRADES (Asym-TRADES) We compare gradient similarity between the classification loss and invariance loss with respect to 0 during training.
The main text mainly focused employing invariance regularization latent representations to mitigate the robustness-accuracy trade-off. Thus, this section discusses logit-based invariance regularization, such as TRADES (Zhang al. 2019) and LBGAT (Cui et 2021), and shows that similar discussion to representation-basee regularization methods is applicable for logit-based regularization.
TRADES also suffers from "gradient conflict." In Fig, we visualize the gradient similarity between the classification loss and invariance loss with respect to 9 during training. TRADES also suffers from "gradient conflict," where the gradient similarity between the classification loss is nega- tive for many layers. This can be explained by the difficulty achieving prediction invariance while maintaining discriminative ability, which is the same reason why the naive invariance regularization (Eq. suffers from "gradient conflict" (Sec. 3.T.
Table 17: Comparison loss functions between TRADES, Asymmetrized-TRADD and LBGAT. LBGAT is similar to TRADES, except that it employs separate teacher network regularize the student network.
Method
Classification Loss
Regularization Loss
Adversarial
Clean
TRADES
CE((o(x).)) CB(AaUu (x). CE(gs(z),
(L)((())) KL(fe ([]ff(z')]|s((e feuin (x))) (
Asym-TRADES CE(fo(z',,) LBGAT
Table 18: Comparison of TRADES vs. Asym-TRA TRADER for ResNet18 trained on CIFAR1O.
Method
Clean
AutoAttack
TRADES
81.25
48.54
ASym-TRADES
85,62
48.58
w/o Stop-grad
82.13
48.97
w/o Split-BN
80,90
49.93
LBGAT
85.50
49.26
20

---

Published as a conference paper at ICLR 2025
Aymmetrizing TRADES with stop-gradient operation and split-BN. To further validate findings, we consider asymmetrizing TRADES based on AR-AT's strategy. Specifically, we replace the regularization term of AR-AT with TRADES's regularization term, which call "Asym- TRADES," as shown in Tab. 17 Here, we did not employ the predictor MLP Asym-TRADEs since having phenomenon in representation-base regularization, conflict" (Fig. and achieves higher robustness and accuracy than TRADES (Tab. 18
predictor MI.P does not make sense for logits. Similar to the
does not suffer from "gradient
Asymmetrized TRADES achieve approximately the same performance as LBGAT. Intriguingly, Tab. 18 demonstrates that Asym-TRe TRADES achieve approximately the same performance as LBGAT. This validates our hypothesis that the effectiveness of the KD-based method LBGAT can be attributed resolving "gradient conflict" and the mixture distribution problem, as discussed in Sec. 5.4 LBGAT employs separate teacher network to regularize the student network, which implicitly resolves the "gradient conffict" and the mixture distribution problem.
Representation-based regularization can outperform logit- based regularization By comparing Tables [8] 3 we observe that AR-AT with representation- regularization can outperform AR-AT with logit-based regularization, "Asym-TRADES" We hypothesize that, since early layers have more diverse levels of information than logits, representation-buee regularization can more effective than logit-based regularization.
E GRADIENT CONFLICT BEYOND MINI-BATCH LEVEL: ANALYSIS ACROSS BATCH SIZES.
To investigate whether the gradient conflict observed at the mini-batch level reflects conflicts in the full input distribution, we provide empirical results showing that gradient conflict persists across different batch sizes.
In Figure 12
we plot the proportion parameters experiencing gradient conflict across varied batch sizes using ResNet-18 trained on CIFAR-10. With the naive invariance loss (Lvo), the conflict ratio remains consistently high, with around 50% of parameters experiencing gradient conflict, regardless of the batch size. Actually, the conflict ratio appears increase with the larger batch size. This indicates that the gradient conflict is not limited to mini-batch levels but is prevalent throughout the entire input distribution.
Thus, we conclude that the gradient conflict can be the fundamental issue affecting the robustness- accuracy trade-off when using invariance regularization.
(Naive: Lvp) bs- 128
(Naive: Lvo) bs-256
Conflict Ratio
(Naive: Lvo) bs=512
(Naive: Lvo) 1024
0.4
(w/ stop-grad: Lv1) bs=128
0.3
0.2
Training epochs
Figure 12: Proportion of parameters experiencing gradient conflict across varied batch sizes ("bs"). With the naive invariance loss (Lyo). the conflict ratio remains consistently high across different batch sizes. indicating that the gradient conflict is not limited to mini-batch levels but is prevalent throughout the entire input distribution.
21

---

Published as a conference paper at ICLR 2025
F ADDITIONAL ANALYSIS ON MIXTURE DISTRIBUTION PROBLEM
F.I FURTHER ANALYSIS OF 1,2 norm(z'-
In Figure 13
show the L2 distance between adversarial and clean features (ll=' - =1|2) for ResNet- 18 trained on CIFAR-10. In addition to Figure from the main text, we include results for standard training and adversarial training (AT) without invariance regularization for comparison.
All methods show increasing in || - 2|12 over time, reflecting the robustnes--accuracy trade-off. As the model accuracy improves, adversarial perturbations become easier find, leading to higher | This poses a challenge for methods using both clean and adversarial inputs during training: the diverging distributions of clean and adversarial samples make it difficult for BatchNorm (BN) layers to correctly estimate batch statistics, leading to the mixture distribution problem. Note that standard training and AT avoid this issue by using only clean or adversarial samples, respectively.
Stop-grad increases || - =2l1 compared to AR-AT without it. While sop-ggrad resolves gra- dient conflict, weakens the invariance regularization by removing the second term in Eq. [ ((Dist(z', sg(z)) + Dist(2, sg(z")))/2) This leads to an increase in || z||2. exacerbating the mixture distribution problem. Our split-BN strategy effectively mitigates this issue, enhancing the utility of stop-grad.
14
12
Std. Training
1.50
1.25
ARAT
L.00
ARAT (w/o stop-grad)
12nomm2z'
0.75
0.25
0.00
60
100
Fpoccs
Figure 13: L2 distance between adversarial and clean features - All training methods show increase in ||2/ =|12 over time, reflecting the robustness-accuracy trade-off. Using stop-grad
increases || 2|| compared to AR-AT without stop-grad, exacerbating the mixture distribution problem.
F.2 SPLIT-BN STABILIZES THE BATCHNORR STATISTICS
To further evaluate the effectiveness of Split-BN, we analyze the stability of the BatchNorm (BN) statisties during training.
BN normalizes each input x; using the running estimates of the mean and variance from mini-batches, follows:
3; 1
(11)
V22
where the moving averages of the mini-batch mean y and variance updated with momentum term:
1 m
A momentum + (1 momentum) PB.
MB
(12)
m
2² momentum * 0² + (1 momentum)
o
og
(x
1g)
(13)
22

---

Published as a conference paper at ICLR 2025
plot the variance of jt of the BN layers computed each epoch. With the naive invariance loss (Lya), the variance of the μ remains high throughout training. In contrast, split-BN stabilizes the updates of BN statistics by separating clean and adversarial BN statistics. Specifically, (1) the variance of adversarial BN statistics is comparable or lower than that of the shared BN without Split-BN, and (2) the variance for clean BN statistics is consistently lower. This aligns with our intuition, as clean samples are unaffected by the dynamic nature of adversarial perturbations during training. These results confirm that the split-BN stabilizes the estimation of BN statisties, effectively addressing the mixture distribution problem.
In Figure 14
Naiva (chnal INN) Sylt BS (clen BX)
SyttRR
Syil-atN (ade Ihho
30000
(a) L..br1.0.bn2 .0.bn2
(b) Layer 1..22
1000
Navc (danad HN]
008199
Nave (dand [N!
4000
Sylt-R (cian Bh)
Syst SS Inde BN
Sylt BN tade BNS
SylkdN IS)
2000
2003
A000115
2002
A000000
0001
0080
Qecck
Spot
(c) Layer2.0.brr
(d) Layer2.1.bb2
000111
Nave (dhmal [N]
0.005
300130
Syn Sin-NNicke BN)
Spli-NN tnde thN
Sylt (nd, BN)
3080
0000775
0000000
(c) Layer3.0.brr
(f) Layer3. 1.bn2
Nava (dandd DNN
Nave (dand DN)
Sytt-RN BN)
SytvRN (okon mS)
Splt-8N Inde BN)
S28t-8N Indo BB)
10012
10000
00
00012
0000
0000
00000
(g) Layer4.0.brr bm2
(h) Layer4.1.bbl
Figure 14: Variance of running means of the BatchNorm (BN) layers of ResNet-18. We compute the variance at each epoch and plot the results. With the naive invariance loss (Lvo), the variance of the running means remains high throughout training. In contrast, applying Split-BN stabilizes both the clean and adversarial BN statistics, leading to lower variance.
G COMPUTATIONAL TIME
In Table 19 we report the computational time of the baseline methods and ARAT, trained on CIFAR- 10 with ResNet- 18 using single NVIDIA A100 GPU and a batch size 128.
AR-AT is faster than TRADES and LBGAT. AR-AT is faster than TRADES and LBGAT, because these methods use the KT. divergence (fTT (z)))) for adversarial example generation,
23

---

Published as a conference paper at ICLR 2025
requiring forward passes for both clean and adversarial images. while AR-AT uses the cross-entropy loss (fo(fo(z'). y)), which only requires forwarding the adversarial images.
Compared to AT, AR-AT introduces 15% increase in time. This is due to (1) the need for forward passes of both clean and adversarial images for loss calculation (please refer Table I0 and (2)
additional updates for separate BatchNorm layers and the predictor MLP head.
Table 19: Computational time of AR-AT and baseline methods, trained on CIFAR-10 using ResNet- 18, along with clean and robust AtutoAtrack; AA) accuracies.
Method
Clean
AA
Time (sec/batch)
Time (rel. to AT)
AT
83.77
42.42
0.122 + 0.045
TRADES
81.25
48.54
0.184 + 0.112
1.51
LBGAT
85.00
48.85
0.183 + 0.200
1.50
(ours) AR-AT
87.82
49.02
0.140 + 0.039
1.15
24


Reviews:
[{"review_id": "v8tKv1czCR", "reviewer": "Reviewer_daZ8", "summary": "This paper proposes a novel method called Asymmetric Representation-regularized Adversarial Training (AR-AT), aimed at improving the robustness-accuracy trade-off in adversarial training. AR-AT achieves this goal by addressing two key issues in invariance regularization: (1) the “gradient conflict” between invariance loss and classification objectives, and (2) the mixed distribution problem caused by the differences in input distributions between clean and adversarial samples. AR-AT introduces asymmetric invariance regularization, a stop-gradient operation, a predictor, and a split BatchNorm (BN) structure. Experimental results show that AR-AT outperforms existing methods across various settings and provides new insights into knowledge distillation-based defenses.", "questions": "(1) The novelty of this paper may be somewhat limited, as the proposed stop-gradient operation and predictor are commonly utilized in self-supervised learning.\n\n(2) Additional evidence and experiments addressing the mixed distribution problem should be provided.\n\n(3) There are a few minor errors, such as the notation \"clean representation Z^' \" in line 191, which should be corrected.\n\n(4) The ImageNet dataset should be incorporated into the experiments to enhance the robustness of the results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) This paper is well written and easy to follow.\n\n(2) The paper validates the effectiveness of AR-AT through experiments on multiple datasets and model architectures, demonstrating superior performance in the robustness-accuracy trade-off.\n\n(3) The paper not only presents a new method but also provides an in-depth analysis of the “gradient conflict” and mixed distribution problems in invariance regularization, offering new theoretical insights for the field of adversarial defense.", "weaknesses": "(1) The novelty of this paper maybe limited. The proposed stop-gradient operation and predictor are often adopted in self-supervised learning \n\n(2) More evidence and experiments on mixed distribution problem should be claimed. \n\n(3) There are a few minor errors, such as \"clean representation z^' \" in line 191.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0MEJboQi1J", "reviewer": "Reviewer_Kdc6", "summary": "This paper proposed a bag of tricks to address two issues in siamese-based adversarial training, i.e., ``conflict gradients'' and the mixture distribution problem.", "questions": "1. Since the proposed method applies invariance regularization to multiple layers, I wonder if the baseline methods also adopt such multi-layer regularization.  \n\n2. Does your method introduce additional computation costs compared to baseline methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.  The paper is well organized. The target issues are clear and each step of the solution is motivated and well represented.\n2.  Experiments clearly show the effectiveness of each technique they proposed.", "weaknesses": "In Table 3, some baseline methods provide error bars, but others do not.\n \nMoreover, considering the huge body of literature on adversarial training, I have concerns about whether the presented comparison with baseline methods is sufficient and comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gvcItNDHzo", "reviewer": "Reviewer_JoTP", "summary": "The authors propose AR-AT, which uses asymmetric invariance loss with stop-gradient operation and a predictor to avoid gradient conflict, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem, to improve robustness-accuracy trade-off. The experiments demonstrate the effectiveness of this method.", "questions": "1. Where is Figure 4?\n\n2. What is the computational overhead of your method compared to other baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy-to-follow\n\n2. The motivation of the proposed method is clear", "weaknesses": "1. In Figure 3, can you explain why $||z-z'||_2$ increases over time even though you add a regularization term? and why the use of stop-grad exacerbates this issue?\n\n2. Line 200-201 & Figure 2: Plotting curves of only minimizing the second term can enhance your claim.\n\n3. Table 17 indicates that solely resolving the mixture distribution problem can already improve the performance to a large extent. However, addressing gradient conflict alone does not contribute much to robustness, especially for (1)vs(2) and (5)vs(6). Thus, you should also report the standard deviations to make your results more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzEF7PPKkT", "reviewer": "Reviewer_zuTF", "summary": "This work addresses the fundamental robustness-accuracy trade-off in adversarial defenses through a novel invariance regularization approach. They proposed the method, AR-AT, achieves state-of-the-art performance by systematically addressing two key challenges: the gradient conflict between classification and invariance losses, and the mixture distribution problem in adversarial training. They resolved the gradient conflict through a strategic stop-gradient operation, while implementing a split batch normalization structure to handle the mixture distribution challenge.", "questions": "Ref Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is well-written and easy to understand. The perspective on gradient conflict is particularly interesting, connecting previously disparate threads in adversarial robustness research.", "weaknesses": "First, the paper's central premise about gradient conflicts requires deeper theoretical examination. Similar gradient conflicts arise in various scenarios, such as when optimizing the same loss across different mini-batches, yet these conflicts don't necessarily impact model generalization (or adversarial robustness). The paper doesn't provide sufficient theoretical justification for why resolving gradient conflicts specifically improves adversarial robustness.\n\nSecond, there's a critical gap in the analysis between local optimization dynamics and global distributional properties. The gradient conflicts observed at the mini-batch level may not accurately reflect the underlying conflicts in the full input distribution. Without this theoretical bridge, it's unclear whether the proposed solution addresses the fundamental cause of the robustness-accuracy trade-off.\n\nThird (minor), the experimental results may not be the SOTA. Ref https://github.com/wzekai99/DM-Improves-AT", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)