Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off

## Abstract

Abstract Although adversarial training has been the state-of-the-art approach to defend against adversarial examples (AEs), it suffers from a robustness-accuracy trade-off, where high robustness is achieved at the cost of clean accuracy.
In this work, we leverage invariance regularization on latent representations to learn discriminative yet adversarially invariant representations, aiming to mitigate this trade-off.
We analyze two key issues in representation learning with invariance regularization: (1) a “gradient conflict" between invariance loss and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem arising from diverged distributions of clean and adversarial inputs.
To address these issues, we propose A symmetrically R epresentation-regularized A dversarial T raining ( AR-AT ), which incorporates asymmetric invariance loss with stop-gradient operation and a predictor to improve the convergence, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem.
Our method significantly improves the robustness-accuracy trade-off by learning adversarially invariant representations without sacrificing discriminative ability.
Furthermore, we discuss the relevance of our findings to knowledge-distillation-based defense methods, contributing to a deeper understanding of their relative successes.

## 1 Introduction

Computer vision models based on deep neural networks (DNNs) achieve remarkable performance in various tasks.
However, they are vulnerable to adversarial examples (AEs) [, ], which are carefully perturbed inputs to fool DNNs.
Since AEs can fool DNNs without affecting human perception, they pose potential threat to real-world DNN applications.

Among the defense strategies against AEs, adversarial training (AT) [, ] has been the state-of-the-art approach, which augments the training data with AEs to enhance robustness.
However, AT-based methods suffer from a significant robustness-accuracy trade-off []: to achieve high robustness, they sacrifice accuracy on clean images.
Despite the extensive number of studies on AT-based methods, this robustness-accuracy trade-off has been a huge obstacle to their practical use.

To this end, we investigate the following research question:
“How can a model learn adversarially invariant representations without compromising discriminative ability?."
Specifically, we focus on invariance regularization on latent representations for AT-based methods, in contrast to most of the existing defenses [, ] that employ invariance regularization on the predicted outputs.

Our investigation addresses two key issues in naively applying invariance regularization (Fig.1(a)): (1) a “gradient conflict" between invariance loss and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem within Batch Normalization (BN) layers when the same BNs are used for both clean and adversarial inputs.
“Gradient conflict" suggests that minimizing invariance loss may push the model toward non-discriminative directions and conflict with the classification objectives.
We hypothesize that it arises from unnecessary gradient flow from the adversarial representation to the clean representation in the invariance loss, which can lead to the degradation of classification performance.
Furthermore, we find that using the same BNs for both clean and adversarial inputs causes the mixture distribution problem: the BNs are updated to have a mixture distribution of clean and adversarial inputs, which can be suboptimal for both clean and adversarial inputs.

To address these issues, we propose a novel method, Asymmetrically Representation-regularized Adversarial Training (AR-AT), incorporating an asymmetric invariance regularization with a stop-gradient operation and a predictor, and a split-BN structure, as depicted in Fig. 1(f).
We show that asymmetric invariance regularization effectively resolves the “gradient conflict" by eliminating unnecessary gradient flow from the adversarial representation to the clean representation.
Additionally, the split-BN structure plays a key role in achieving high robustness and accuracy by resolving the mixture distribution problem.
With both components combined, our method learns both adversarially invariant and discriminative representations, leading to high robustness and accuracy.

Furthermore, we discuss the relevance of our findings to existing knowledge distillation (KD)-based defenses [, ], identifying the underlying factors of their effectiveness that have not yet been investigated.
We attribute their effectiveness to resolving “gradient conflict” and the mixture distribution problem.
Furthermore, we show that split-BN structure can outperform KD-based defenses in terms of robustness-accuracy trade-off and reduces memory consumption, highlighting that split-BN can be viable alternative to KD-based structure.

Our contributions are summarized as follows:

- •
We focus on the invariance regularization on latent representation in adversarial training to achieve high robustness without sacrificing accuracy.
- •
We reveal two key issues in representation learning with invariance regularization: (1) a “gradient conflict" between invariance loss and classification objectives, and (2) the mixture distribution problem within Batch Norm (BN) layers.
- •
We propose a novel method AR-AT to incorporate an asymmetric invariance regularization, and a split-BN structure. Our method significantly improves the robustness-accuracy trade-off by effectively learning discriminative yet adversarially invariant representations.
- •
We present a new perspective on KD-based defenses and show that split-BN can be an alternative, potentially outperforming KD in terms of the robustness-accuracy trade-off while reducing memory consumption.

## 2 Preliminaries

### 2.1 Adversarial attack

Let $x\in\mathbb{R}^{d}$ be an input image and $y\in\{1,\dots,K\}$ be a class label from a data distribution $\mathcal{D}$.
Let $f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{K}$ be a DNN model parameterized by $\theta$.
Adversarial attacks aim to find a perturbation $\delta$ that fools the model $f_{\theta}$ by solving the following optimization problem:

$$ $x^{\prime}=\mathop{\rm arg~{}max}\limits_{\delta\in\mathcal{S}}\mathcal{L}(f_{ \theta}(x+\delta),y)$ (1) $$

where $x^{\prime}$ is an adversarial example, $\mathcal{S}$ is a set of allowed perturbations, and $\mathcal{L}$ is a loss function.
In this paper, we define the set of allowed perturbations $\mathcal{S}$ with $L_{\infty}$-norm as $\mathcal{S}=\{\delta\in\mathbb{R}^{d}\mid\|\delta\|_{\infty}\leq\epsilon\}$, where $\epsilon$ represents the size of the perturbations.
The optimization of Eq.1 is often solved iteratively based on the projected gradient descent (PGD) [].

### 2.2 Adversarial training

Adversarial training (AT) [, ], which augments the training data with adversarial examples (AEs), has been the state-of-the-art approach to defend against adversarial attacks.
Originally, Goodfellow et al. [] proposed to train the model with AEs generated by the fast gradient sign method (FGSM); in contrast, Madry et al. [] proposed to train a model with much stronger AEs generated by PGD, which is an iterative version of FGSM.
Formally, the standard AT [] solves the following optimization:

$$ $\min_{\theta}\mathop{\mathbb{E}}_{(x,y)\sim\mathcal{D}}\left[\max_{\delta\in \mathcal{S}}\mathcal{L}(f_{\theta}(x+\delta),y)\right]$ (2) $$

where the inner maximization problem is solved iteratively based on PGD.

### 2.3 Invariance regularization-based defense

Figure: (a) General framework of AT [] with invariance regularization
Refer to caption: extracted/5628063/figures/illust/general_inv_reg_grey.png

While AT only inputs AEs during training, invariance regularization-based adversarial defense methods [, ] input both clean and adversarial images to ensure adversarial invariance of the model.
TRADES [] introduces a regularization term on the logits to encourage adversarially invariant predictions; although it allows trade-off adjustment by altering the regularization strength, it still suffers from a trade-off.
MART [] further improved TRADES by focusing more on the misclassified examples to enhance robustness, still sacrificing clean accuracy.
Unlike these logit-based invariance regularization methods, our method leverages representation invariance to mitigate the trade-off.

Another line of research is to employ knowledge distillation (KD) []-based regularization, which has been shown effective in mitigating the trade-off.
Cui et al. [] proposed LBGAT, which aligns the student model’s predictions on adversarial images with the standard model’s predictions on clean images, encouraging similarity of predictions between a student and a standardly trained teacher network.
More recently, Suzuki et al. [] proposed ARREST, which performs representation-based KD so that the student model’s representations are similar to the standardly trained model’s representations.
In contrast to these KD-based methods, which enforce adversarial invariance implicitly, we investigate output invariance within a single model, potentially more effective and more memory-efficient during training.
Furthermore, we provide a new perspective on the relative success of KD-based methods, which had not been well understood.

Figure. 1 summarizes the loss functions of these methods and compares them with our method.

## 3 Asymmetric Representation-regularized Adversarial Training (AR-AT)

In this section, we propose a novel approach to effectively learn adversarially invariant representation without sacrificing discriminative ability on clean images.

A straightforward approach to learn adversarially invariant representation is to employ a siamese structured invariance regularization, depicted in Fig. 1(a), as follows:

$$ $\displaystyle\mathcal{L}_{V0}=\alpha\cdot\mathcal{L}(f_{\theta}(x^{\prime}),y) +\beta\cdot\mathcal{L}(f_{\theta}(x),y)+\gamma\cdot Dist(z,z^{\prime})$ (3) $$

where $x^{\prime}$ is an AE generated from $x$, $z$ and $z^{\prime}$ are the normalized latent representations of $x$ and $x^{\prime}$, respectively. $\mathcal{L}$ is a classification loss and $Dist$ is a distance metric.
Although existing approaches typically rely on either adversarial or clean classification loss, we begin with this naive method to systematically identify and address the underlying issues step by step.
Here, we focus on representation-based regularization since we found that it can be more effective in mitigating the trade-off than logit-based regularization, discussed in Appendix E.

However, we identify two key issues in this naive approach: (1) a “gradient conflict" between invariance loss and classification objectives, and (2) the mixture distribution problem arising from diverged distributions of clean and adversarial inputs.
To address these issues, we propose a novel approach incorporating (1) a stop-gradient operation and a predictor MLP to asymmetrize the invariance regularization, and (2) a split-BatchNorm (BN) structure.

### 3.1 Stop-gradient for addressing gradient conflict

Although naive invariance regularization (Eq.3) is a straightforward approach for learning adversarially invariant representation, we observed a “gradient conflict" between invariance loss and classification objectives, as shown in Fig. 2.
“Gradient conflict" occurs when gradients for multiple loss functions oppose each other during joint optimization, and resolving it has been shown to enhance performance in various fields, including multi-task learning [] and domain generalization [].
The observed “gradient conflict" suggests that minimizing the invariance loss may push the model towards non-discriminative directions, conflicting with the classification objectives, thereby degrading the classification performance.

In this work, we reveal that asymmetrizing the invariance regularization with a stop-gradient operation can effectively resolve “gradient conflict.”
The aymmetric invariance regularization is defined as follows:

$$ $\displaystyle\mathcal{L}_{V1}=\alpha\cdot\mathcal{L}(f_{\theta}(x^{\prime}),y) +\beta\cdot\mathcal{L}(f_{\theta}(x),y)+\;\gamma\cdot Dist(z^{\prime},\text{ stop-grad}(z))$ (4) $$

where $\text{stop-grad}(\cdot)$ stops the gradient backpropagation from $z^{\prime}$ to $z$, treating $z$ as a constant.

Our motivation of employing the stop-gradient operation is to eliminate the unnecessary gradient flow from the adversarial representation $z^{\prime}$ to the clean representation $z^{\prime}$, which can lead to the degradation of classification performance.
This can be understood by decomposing the invariance loss into two components, as follows:

$$ $\displaystyle Dist(z^{\prime},z)=Dist(z^{\prime},\text{stop-grad}(z))+Dist( \text{stop-grad}(z^{\prime}),z)$ (5) $$

Minimizing the first term, $Dist(z^{\prime},\text{stop-grad}(z))$, encourages to bring the corrupted adversarial representation $z^{\prime}$ closer to the clean representation $z$ by treating $z$ as a constant, which can be interpreted as the “purification" of representations.
In contrast, minimizing the second term, $Dist(\text{stop-grad}(z^{\prime}),z)$, attempts to bring the clean representation $z$ closer to the potentially corrupted adversarial representation $z^{\prime}$, encouraging the “corruption” of representations. This can be harmful for learning discriminative representations.
Therefore, we hypothesize that minimizing the second term can conflict to the classification objectives, leading to the “gradient conflict."

### 3.2 Latent projection for enhancing training stability

Furthermore, we employ a predictor MLP $h$ to the latent representations $z^{\prime}$ to predict $z$, inspired by recent self-supervised learning approach [], as following:

$$ $\displaystyle\mathcal{L}_{V2}=\alpha\cdot\mathcal{L}(f_{\theta}(x^{\prime}),y) +\beta\cdot\mathcal{L}(f_{\theta}(x),y)+\;\gamma\cdot Dist(h(z^{\prime}),\text {stop-grad}(z))$ (6) $$

The predictor MLP $h$ is trained to predict the clean representation $z$ from the corrupted adversarial representation $z^{\prime}$.
Our intuition is that the adversarial representation $z^{\prime}$ varies with each training iteration, as adversarial perturbations depend on both the model parameters and the randomness inherent in the attack process.
Therefore, introducing an additional predictor head may help stabilize the training process by handling the variations of adversarial representation through updates of the predictor, preventing large fluctuation of the classifier’s parameters caused by variations in $z^{\prime}$.
In other words, the predictor MLP $h$ functions as a “stabilizer", preventing the model from being overly distracted by variations in adversarial representations while achieving invariance, thus preventing compromise of classification performance.

### 3.3 Split batch normalization for resolving mixture distribution problem

In addition, we point out that using the same Batch Norm (BN) [] layers for both clean and adversarial inputs can lead to difficulty in achieving high performance due to a “mixture distribution problem."
BNs are popularily used for accelerating the training of DNNs by normalizing the activations of the previous layer and then adjusting them via a learnable linear layer.
However, since clean and adversarial inputs exhibit diverged distributions, using the same BNs on a mixture of these inputs can be suboptimal for both inputs, leading to reduced performance.

To address this issue, we employ a split-BN structure, which uses separate BNs for clean and adversarial inputs during training, inspired by Xie et al. [, ].
Specifically, Eq. 6 is rewritten as follows:

$$ $\displaystyle\mathcal{L}_{V3}=\alpha\cdot\mathcal{L}(f_{\theta}(x^{\prime}),y) +\beta\cdot\mathcal{L}(f_{\theta}^{\text{auxBN}}(x),y)+\;\gamma\cdot Dist(h(z^ {\prime}_{(\theta)}),\text{stop-grad}(z_{(\theta^{\text{auxBN}})}))$ (7) $$

where $f_{\theta}^{\text{auxBN}}$ shares parameters with $f_{\theta}$ but employs auxiliary BNs specialized for clean images, $z_{(\theta^{\text{auxBN}})}$ and $z^{\prime}_{(\theta)}$ are the latent representations of $x$ and $x^{\prime}$ from $f_{\theta}^{\text{auxBN}}$ and $f_{\theta}$, respectively.
In this way, the BNs in $f_{\theta}$ exclusively processes adversarial inputs, while the BNs in $f_{\theta}^{\text{auxBN}}$ exclusively processes clean inputs, avoiding the mixture distribution problem.

Importantly, the split-BN structure is exclusively applied during training: During inference, the model $f_{\theta}$ is equipped to classify both clean and adversarial inputs with the same BNs.
Therefore, in contrast to MBN-AT [] and AdvProp [] that require test-time oracle selection of clean and adversarial BNs for optimal robustness and accuracy, our approach is more practical.

### 3.4 Auto-blance for reducing hyperparameters

Furthermore, we employ a dynamic adjustment rule for the hyperparameters $\alpha$ and $\beta$, which automatically controls the balance between adversarial and clean classification loss.
Specifically, we adjust $\alpha$ and $\beta$ based on the training accuracy of the previous epoch, inspired by Xu et al. []:

$$ $\displaystyle\alpha_{(t)}=Acc_{(t-1)}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}( \text{argmax}(f^{\text{auxBN}}_{\theta_{t-1}}(x))=y),\;\;\;\;\beta_{(t)}=1-Acc _{(t-1)}$ (8) $$

where $\alpha_{(t)}$ and $\beta_{(t)}$ are the hyperparameters at the current epoch $t$, and $Acc_{(t-1)}$ is the clean accuracy of the model $f_{\theta}^{\text{auxBN}}$ at the previous epoch $(t-1)$.
This adjustment automatically forces the model to focus more on adversarial images as its accuracy on clean images improves.
We empirically demonstrate that this heuristic dynamic adjustment strategy works well in practice, successfully reducing the hyperparameter tuning cost.
We provide an ablation study in Sec. 5.5

### 3.5 Multi-level regularization of latent representations

Finally, our method is extended to regularize multiple levels of representations.
Specifically, we employ multiple predictor MLPs $h_{1},\dots,h_{L}$ to enforce invariance across multiple levels of representations $z_{1},\dots,z_{L}$ as follows:

$$ $\displaystyle\frac{1}{L}\sum_{l=1}^{L}Dist(h_{l}(z^{\prime}_{(\theta),l}), \text{stop-grad}(z_{(\theta^{\text{auxBN}}),l}))$ (9) $$

We found that regularizing multiple layers in the later stage of a network is the most effective, as demonstrated in our ablation study in Sec. 5.5.

## 4 Experimental setup

Models and datasets.
We evaluate our method on CIFAR-10, CIFAR-100 [], and Imagenette [] datasets.
We use the standard data augmentation techniques of random cropping with 4 pixels of padding and random horizontal flipping.
We use the model architectures of ResNet-18 [] and WideResNet-34-10 (WRN-34-10) [], following the previous works [, , ].

Evaluation.
We use 20-step PGD attack (PGD-20) [] and AutoAttack (AA) [] for evaluation. The perturbation budget is set to $\epsilon=8/255$. The step size is set to $2/255$ for PGD-20.
We compare our method with the standard AT [], and existing regularization-based methods TRADES [], MART [], LBGAT [], and ARREST [].

Training details.
We use a 10-step PGD [] for adversarial training.
We initialized the learning rate to 0.1, divided it by a factor of 10 at the 75th and 90th epochs, and trained for 100 epochs.
We use the SGD optimizer with a momentum of 0.9 and a weight decay of 5e-4, with a batch size of 128.
Cosine Distance is used as the distance metric for invariance regularization.
The predictor MLP has two linear layers, with the hidden dimension set to 1/4 of the feature dimension, following SimSiam [].
The latent representations to be regularized are spatially average-pooled to obtain one-dimensional vectors.
The regularization strength $\gamma$ is set to 30.0 for ResNet-18 and 100.0 for WRN-34-10.
We regularize all ReLU outputs in “layer4” for ResNet-18, and “layer3” for WRN-34-10.

## 5 Empirical study

### 5.1 AR-AT resolves “gradient conflict" and mixture distribution problem

Stop-gradient operation resolves “gradient conflict."
In Fig. 2, we plot the cosine similarity between the gradients of the classification loss (the sum of clean and adversarial classification loss) and the invariance loss with respect to $\theta$ during training.
We observe that with the naive invariance regularization ($\mathcal{L}_{V0}$; Eq. 3), the gradient similarity fluctuates around zero, indicating that some neurons encounter “gradient conflict" during training.
This indicates that minimizing invariance loss may push the model towards non-discriminative directions, conflicting with the classification objectives.
In contrast, with the stop-gradient operation ($\mathcal{L}_{V1}$; Eq. 4), the gradient similarity becomes consistently positive.
This demonstrates the effectiveness of eliminating unnecessary gradient flow from the adversarial representation to the clean representation, which can lead to the corruption of representations, thereby preventing “gradient conflict.”

However, Tab. 1 shows that mitigating “gradient conflict” ($\mathcal{L}_{V0}$ vs. $\mathcal{L}_{V1}$) does not improve the robustness and accuracy, indicating that the stop-gradient operation alone is not sufficient.

**Figure 2: Gradient similarity between classification loss and invariance loss with respect to $\theta$ during training, with and without the stop-gradient operation. The light areas show the standard deviation of cosine similarity per layer, while the dark line represents the mean.**
|  | Method | Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | stop-<br>grad. | pred.<br>$h$ | split-<br>BN | Clean | PGD-<br>20 | Grad-<br>sim. |
| (1) $\mathcal{L}_{V0}$ |  |  |  | 82.93 | 48.89 | 0.06 |
| (2) $\mathcal{L}_{V1}$ | $\checkmark$ |  |  | 82.47 | 48.57 | 0.68 |
| (3) |  | $\checkmark$ |  | 83.49 | 48.22 | 0.00 |
| (4) $\mathcal{L}_{V2}$ | $\checkmark$ | $\checkmark$ |  | 83.00 | 49.14 | 0.63 |
| (5) |  |  | $\checkmark$ | 84.35 | 51.34 | 0.14 |
| (6) | $\checkmark$ |  | $\checkmark$ | 85.51 | 51.46 | 0.59 |
| (7) |  | $\checkmark$ | $\checkmark$ | 84.07 | 50.79 | 0.00 |
| (8) $\mathcal{L}_{V3}$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 86.29 | 52.40 | 0.52 |

Split-BN resolves the mixture distribution problem.
Another issue in representation learning with invariance regularization stems from using the same Batch Norm (BN) layers for both clean and adversarial inputs.
Tab. 1 shows the comparison of invariance regularization with and without split-BN ($\mathcal{L}_{V3}$ vs. $\mathcal{L}_{V2}$, (6) vs. $\mathcal{L}_{V1}$, (5) vs (1)).
We observe that using separate BNs for clean and adversarial inputs ($\mathcal{L}_{V2}$) significantly improves the robustness and accuracy compared to using the same BNs for both clean and adversarial inputs ($\mathcal{L}_{V1}$).

Importantly, Tab. 1 demonstrates that only resolving either “gradient conflict" or the mixture distribution problem alone does not significantly improve robustness and accuracy, but resolving both issues simultaneously leads to substantial improvements.

Additionally employing the latent projection further improves the performance.
Tab. 1 shows that employing the predictor MLP head ($\mathcal{L}_{V1}$ vs. $\mathcal{L}_{V2}$, (6) vs. $\mathcal{L}_{V3}$) leads to substantial improvements in the robustness and accuracy.
This demonstrates that the predictor MLP head may stabilize the training process and improve the convergence of the two loss functions of classification and invariance regularization, leading to better performance.

### 5.2 Comparison with state-of-the-art invariance regularizatoin-based defense methods

**Table 2: Comparison with invariance regularization-based defense methods on CIFAR datasets. We report clean and robust accuracy (AutoAttack; AA). (^2^22Results with ^∗ are directly copied from original papers.)**
|  | Defense | CIFAR10 | CIFAR100 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Clean | AA | Sum. | Clean | AA | Sum. |
| ResNet-18 | AT | 83.77 | 42.42 | 126.19 | 55.82 | 19.50 | 75.32 |
| TRADES | 81.25 | 48.54 | 129.79 | 54.74 | 23.60 | 78.34 |  |
| MART | 82.15 | 47.83 | 129.98 | 54.54 | 26.04 | 80.58 |  |
| LBGAT | 85.50 | 49.26 | 134.76 | 67.33 | 23.94 | 91.21 |  |
| ARREST^∗ | 86.63 | 46.14 | 132.77 | - | - | - |  |
| AR-AT (ours) | 87.93 | 49.19 | 137.12 | 67.72 | 23.54 | 91.26 |  |
| WRN-34-10 | AT | 86.06 | 46.26 | 132.32 | 59.83 | 23.94 | 83.77 |
| TRADES | 84.33 | 51.75 | 136.08 | 57.61 | 26.88 | 84.49 |  |
| MART | 86.10 | 49.11 | 135.21 | 57.75 | 24.89 | 82.64 |  |
| LBGAT | 88.28 | 52.49 | 140.77 | 69.26 | 27.53 | 96.79 |  |
| ARREST^∗ | 90.24 | 50.20 | 140.44 | 73.05 | 24.32 | 97.37 |  |
| AR-AT (ours) | 90.81 | 51.19 | 142.00 | 72.23 | 24.97 | 97.20 |  |

**Table 3: Comparison with defense methods on Imagenette. (The learning rate of LBGAT^† is lowered from default due to gradient explosion.)**
|  | Defense | Imagenette |  |  |
| --- | --- | --- | --- | --- |
|  |  | Clean | AA | Sum. |
| ResNet-18 | AT | 84.58 | 52.25 | 136.83 |
| TRADES | 79.21 | 53.98 | 133.19 |  |
| MART | 84.07 | 59.89 | 143.96 |  |
| LBGAT^† | 80.80 | 50.42 | 131.22 |  |
| AR-AT (ours) | 88.66 | 59.28 | 147.94 |  |

Tab. 2 shows the results of ResNet-18 and WRN-34-10 trained on CIFAR-10 and CIFAR-100.
We observe that our method achieves much better performance than the baselines of regularization-based defenses TRADES and MART on both datasets, highlighting the effectiveness of our methodology in employing invariance regularization.
Moreover, our method achieves state-of-the-art performance on CIFAR10 in mitigating the trade-off, outperforming LBGAT and ARREST, which employs implicit invariance regularization with knowledge distillation.
On CIFAR-100, our method achieves comparable results to LBGAT and ARREST.
Tab. 4 shows the results on Imagenette, a dataset with high-resolution images, and demonstrate that our method achieves the best performance.
These results strongly support AR-AT’s effectiveness in mitigating the robustness-accuracy trade-off.
Note that AR-AT is more memory-efficient than LBGAT, which trains two separate networks for KD-based regularization.

### 5.3 Analaysis of representations: adversarial invariance and discriminative ability

Quantitative analysis.
Tab. 4 presents representation invariance measured by cosine similarity between adversarial and clean features extracted from the penultimate layer.
While TRADES achieves high adversarial invariance, it sacrifices discriminative ability, as shown by its low accuracy.
Conversely, LBGAT, a KD-based method, achieves high performance but low adversarial invariance, demonstrating that using a separate teacher network for regularization does not ensure invariance within a model.
On the other hand, AR-AT achieves both high adversarial invariance and high accuracy simultaneously.
We attribute this to effectively addressing the “gradient conflict" and the mixture distribution problem in invariance regularization, while imposing the invariance regularization directly on the model itself, which is different from KD-based methods.

Figure: (a) Naive invariance regularization ($\mathcal{L}_{V0}$; Eq. 3)
Refer to caption: x1.png

Visualization of learned representations.
Fig. 4 visualizes the clean representations for (a) naive invariance regularization ($\mathcal{L}_{V0}$, Eq. 3) and (b) AR-AT ($\mathcal{L}_{V3}$, Eq. 7), using UMAP [].
Naive invariance regularization ($\mathcal{L}_{V0}$) leads to relatively non-discriminative representations, likely due to the model’s pursuit of adversarial invariance; In fact, the cosine similarity between adversarial and clean features was 0.9985, highest among all methods in Tab. 4.
In contrast, AR-AT learns more discriminative representations than naive invariance regularization.

### 5.4 On the Relation of AR-AT to Knowledge Distillation (KD)-based defenses

Here, we offer a novel perspective on the effectiveness of KD-based defenses, such as LBGAT and ARREST, in adversarial robustness.
In this section, the hyperparameters $\alpha$ and $\beta$ are set to 1.0, and the penultimate layer is used for regularization loss for simplicity.

Using separate networks can relieve the “gradient conflict” and the mixture distribution problem.
Fig. 4 shows the impact of using “shared weights vs. separate networks" between adversarial and clean branches.
We observe that “Naive Inv. Loss (Eq. 3)" and “AR-AT w/o Split-BN (Eq. 6)” struggle to attain high robustness when weights are shared but perform better with separate networks.
We hypothesize that (1) using separate networks acts similarly to the stop-gradient operation by eliminating gradient flow between branches, and also (2) resemble split-BNs in avoiding the mixture distribution problem.
In contrast, AR-AT (Eq. 7) maintains performance when layers are shared: Importantly, it even improves performance when layers are shared, indicating its superiority over KD-based regularization.
This advantage of sharing layers may arise from more explicit invariance regularization compared to using a separate teacher network.

### 5.5 Ablation study

Asymmetrizing logit-based regularization.
We also explore the benefits of an asymmetric structure for logit-based regularization, detailed in Appendix E.
We find that TRADES also faces the “gradient conflict” and the mixture distribution problem, and asymmetrizing TRADES leads to significant improvements in robustness and accuracy, demonstrating the consistency of our findings.

Effectiveness of hyperparameter auto-balance.
Tab. 6 shows an ablation study of AR-AT “with vs. without auto-balance (Sec. 3.4)" of the hyperparameters $\alpha$ and $\beta$, which controls the balance between adversarial and clean classification loss.
The results demonstrate that auto-balancing performs comparable to the best hyperparameter manually obtained.
Additionally, in the case of WRN-34-10, auto-balancing outperforms the best hyperparameter setting in Tab. 6, suggesting that it can discover superior hyperparameter configurations beyond manual tuning.

**Table 5: Comparison of AR-AT “with vs. without auto-balance (Eq. 8)" of hyperparameter $\alpha$ and $\beta$ on CIFAR-10.**
| $(\alpha,\beta$) | ResNet-18 | WRN-34-10 |  |  |
| --- | --- | --- | --- | --- |
|  | Clean | PGD-20 | Clean | PGD-20 |
| $(1.0,1.0)$ | 88.91 | 51.75 | 91.39 | 49.17 |
| $(1.0,0.5)$ | 88.55 | 52.12 | 91.48 | 49.01 |
| $(1.0,0.1)$ | 87.55 | 52.27 | 90.80 | 49.46 |
| $(0.5,1.0)$ | 89.01 | 51.37 | 91.71 | 50.35 |
| $(0.1,1.0)$ | 89.08 | 46.73 | 91.85 | 48.51 |
| auto-bal. | 87.93 | 52.13 | 90.81 | 52.72 |

Layer importance in invariance regularization.
Tab. 6 shows the ablation study of AR-AT on regularizing different levels of latent representations.
ResNet-18 and WRN-34-10 have four and three “layers", respectively. “Layers” are composed of multiple “BasicBlocks", which consist of multiple convolutional layers followed by BN and ReLU.
We show that regularizing multiple latent representations in the later stage of the network can achieve better performance.
In practice, we recommend regularizing all ReLU outputs in the last stage of the network, which is the default of AR-AT.
However, we note that the choice of layers for invariance regularization is not critical as long as the later-stage latent representations are regularized.

Importance of batchNorm for robustness.

**Table 7: Importance of BatchNorm. We compare AR-AT on CIFAR10 using ResNet-18, using main BNs or auxiliary BNs during inference.**
| Defense | Clean | PGD-20 |
| --- | --- | --- |
| AR-AT ($f_{\theta}$) | 87.93 | 52.13 |
| AR-AT ($f_{\theta}^{\text{auxBN}}$) | 92.84 | 1.98 |

Tab. 7 presents the performance of AR-AT when the auxiliary BNs are used during inference.
Intriguingly, despite sharing all the layers except BNs between $f_{\theta}$ and $f_{\theta}^{\text{auxBN}}$, using the auxiliary BNs during inference notably enhances the clean accuracy while compromising robustness significantly.
This indicates that the auxiliary BN are exclusively tailored for clean inputs, highlighting the critical role of BNs in adversarial robustness.

## 6 Conclusion

We explored invariance regularization-based adversarial defenses to mitigate robustness-accuracy trade-off.
By addressing two challenges of the “gradient conflict" and the “mixture distribution problem," our method, AR-AT, achieves the state-of-the-art performance of mitigating the trade-off.
The gradient conflict between classification and invariance loss is resolved using a stop-gradient operation, while the mixture distribution problem is mitigated through a split-BN structure.
Additionally, we provide a new perspective on the success of KD-based methods, attributing it to resolution of these challenges.
This paper provides a novel perspective to mitigate the robustness-accuracy trade-off.

Limitations.
While AR-AT demonstrates the effectiveness of representation-level regularization, determining the appropriate layers for regularization remains ambiguous, especially for novel model architectures.
Additionally, the proposed method may not be directly applicable to models without BNs, such as Transformers, which use Layer-Norm [], and whether split-BN structure can be extended to Layer-Norm is an interesting future direction.

## Appendix A Datasets

Here, we provide the details of the datasets we used in our experiments.
CIFAR10 and CIFAR100 [] are standard datasets for image classification with 10 and 100 classes, respectively, with a resolution of $32\times 32$.
Imagenette [] is a subset of 10 easily classified classes from Imagenet []. In our experiments, we used the version with the resolution of $160\times 160$.

**Table 8: The details of datasets we used in our experiments.**
| Dataset | Resolution | Class Num. | Train | Val |
| --- | --- | --- | --- | --- |
| CIFAR10 | 32 $\times$ 32 | 10 | 50,000 | 10,000 |
| CIFAR100 | 32 $\times$ 32 | 100 | 50,000 | 10,000 |
| Imagenette | 160 $\times$ 160 | 10 | 9,469 | 3,925 |

## Appendix B Experiments Compute Resources

In this work, we use NVIDIA A100 GPUs for our experiments. Training of AR-AT on CIFAR10 using ResNet-18 takes approximately 2 hours, and on CIFAR10 using WideResNet-34-10 takes approximately 10 hours.

## Appendix C Implementation details of baseline methods

Here, we describe the implementation details of the baseline defense methods.

- •
Adversarial Training (AT) []: We use the simple implementation by Dongbin Na (^3^33https://github.com/ndb796/Pytorch-Adversarial-Training-CIFAR). We aligned the hyperparameter setting described in the official GitHub repository (^4^44https://github.com/MadryLab/robustness).
- •
TRADES []: We use the official implementation (^5^55https://github.com/yaodongyu/TRADES) and used the default hyperparameter setting.
- •
MART []: We use the official implementation (^6^66https://github.com/YisenWang/MART) and used the default hyperparameter setting.
- •
LBGAT []: We use the official implementation (^7^77https://github.com/dvlab-research/LBGAT) and used the default hyperparameter setting. For Imagenette, we observed that the default learning rate causes gradient explosion, so we lowered the learning rate from 0.1 to 0.01. Note that LBGAT uses a ResNet-18 teacher network for both ResNet-18 and WRN-34-10 student networks.
- •
ARREST []: They do not provide an official implementation, so we directly copied the results in Tab. 2 from their paper.

The comparison of loss functions is summarized in Tab. 9.

**Table 9: Comparison of loss functions of defense methods. $f_{\theta}(\cdot)$ is the predictions of the trained model, and $z_{(\theta)}$ and $z^{\prime}_{(\theta)}$ is the latent representation of the model $f_{\theta}$ on clean and adversarial inputs, respectively. LBGAT and ARREST are knowledge distillation-based methods using a teacher model $g_{\phi}$. Our method AR-AT uses both clean and adversarial classification losses and employs representation-based regularization (Details in Sec. 3).**
| Method | Classification Loss | Regularization Loss |  |
| --- | --- | --- | --- |
|  | Adversarial | Clean |  |
| AT | $CE(f_{\theta}(x^{\prime}),y)$ |  |  |
| TRADES |  | $CE(f_{\theta}(x),y)$ | $KL(f_{\theta}(x)||f_{\theta}(x^{\prime}))$ |
| MART | $BCE(f_{\theta}(x^{\prime}),y)$ |  | $KL(f_{\theta}(x)||f_{\theta}(x^{\prime}))\cdot(1-f_{\theta}(x)_{y})$ |
| LBGAT |  | $CE(g_{\phi}(x),y)$ | $MSE(f_{\theta}(x^{\prime}),g_{\phi}(x))$ |
| ARREST | $CE(f_{\theta}(x^{\prime}),y)$ |  | $CosSim(z^{\prime}_{(\theta)},z_{(\phi)})$, where $g_{\phi}$ is a fixed std. model |
| AR-AT (ours) | $CE(f_{\theta}(x^{\prime}),y)$ | $CE(f_{\theta}^{\text{auxBN}}(x),y)$ | $CosSim(h(z^{\prime}_{(\theta)}),\text{stop-grad}(z_{(\theta^{\text{auxBN}})}))$ |

## Appendix D More Comparison with State-of-the-Art Defense Methods

**Table 10: Comparison of clean accuracy and robust accuracy against AutoAttack (AA) for WideResNet-34-10 trained on CIFAR10. The compared methods are sorted by the sum of clean and robust accuracy.**
| Method | Clean | AutoAttack | Sum. | Reference |
| --- | --- | --- | --- | --- |
| AT [] | 86.06 | 46.26 | 132.32 | Reproduced |
| FAT [] | 89.34 | 43.05 | 132.39 | Copied from [] |
| MART [] | 86.10 | 49.11 | 135.21 | Reproduced |
| TRADES [] | 84.33 | 51.75 | 136.08 | Reproduced |
| IAD [] | 85.09 | 52.29 | 137.38 | Copied from the original paper |
| AWP [] | 85.57 | 54.04 | 139.61 | Copied from the original paper |
| S^2O [] | 85.67 | 54.10 | 139.77 | Copied from the original paper |
| ARREST [] | 90.24 | 50.20 | 140.44 | Copied from the original paper |
| LBGAT [] | 88.28 | 52.49 | 140.77 | Reproduced |
| AR-AT (ours) | 90.81 | 51.19 | 142.00 | (Ours) |

**Table 11: Comparison of clean accuracy and robust accuracy against AutoAttack (AA) for WideResNet-34-10 trained on CIFAR100. The compared methods are sorted by the sum of clean and robust accuracy.**
| Method | Clean | AutoAttack | Sum. | Reference |
| --- | --- | --- | --- | --- |
| MART [] | 57.75 | 24.89 | 82.64 | Reproduced |
| AT [] | 59.83 | 23.94 | 83.77 | Reproduced |
| TRADES [] | 57.61 | 26.88 | 84.49 | Reproduced |
| FAT [] | 65.51 | 21.17 | 86.68 | Copied from [] |
| IAD [] | 60.72 | 27.89 | 88.61 | Copied from the original paper |
| AWP [] | 60.38 | 28.86 | 89.24 | Copied from the original paper |
| S^2O [] | 63.40 | 27.60 | 91.00 | Copied from the original paper |
| LBGAT [] | 69.26 | 27.53 | 96.79 | Reproduced |
| ARREST [] | 73.05 | 24.32 | 97.37 | Copied from the original paper |
| AR-AT (ours) | 72.23 | 24.97 | 97.20 | (Ours) |

We provide more comparison with state-of-the-art defense methods in Tab. 10 and Tab. 11
We observe that AR-AT achieves high clean accuracy while maintaining high robustness.
These results demonstrate the effectiveness of AR-AT in mitigating the robustness-accuracy trade-off.

## Appendix E Asymmetrizing Logit-based Regularization: Improving TRADES and Its Relation to LBGAT

Figure: Figure 5: Comparison of gradient similarity between “TRADES vs. Asymmetrized-TRADES (Asym-TRADES)". We compare gradient similarity between the classification loss and invariance loss with respect to $\theta$ during training.
Refer to caption: x3.png

The main text mainly focused on employing invariance regularization on latent representations to mitigate the robustness-accuracy trade-off.
Thus, this section discusses logit-based invariance regularization, such as TRADES [] and LBGAT [], and shows that similar discussion to representation-based regularization methods is applicable for logit-based regularization.

TRADES also suffers from “gradient conflict."
In Fig. 5, we visualize the gradient similarity between the classification loss and invariance loss with respect to $\theta$ during training.
TRADES also suffers from “gradient conflict," where the gradient similarity between the classification loss is negative for many layers.
This can be explained by the difficulty in achieving prediction invariance while maintaining discriminative ability, which is the same reason why the naive invariance regularization (Eq. 3) suffers from “gradient conflict" (Sec. 3.1).

**Table 12: Comparison of loss functions between TRADES, Asymmetrized-TRADES, and LBGAT. LBGAT is similar to TRADES, except that it employs a separate teacher network to regularize the student network.**
| Method | Classification Loss | Regularization Loss |  |
| --- | --- | --- | --- |
|  | Adversarial | Clean |  |
| TRADES |  | $CE(f_{\theta}(x),y)$ | $KL(f_{\theta}(x)||f_{\theta}(x^{\prime}))$ |
| Asym-TRADES | $CE(f_{\theta}(x^{\prime}),y)$ | $CE(f_{\theta}^{\text{auxBN}}(x),y)$ | $KL(f_{\theta}(x^{\prime})||\text{stop-grad}(f_{\theta}^{\text{auxBN}}(x)))$ |
| LBGAT |  | $CE(g_{\phi}(x),y)$ | $MSE(f_{\theta}(x^{\prime}),g_{\phi}(x))$ |

**Table 13: Comparison of TRADES vs. Asym-TRADES for ResNet18 trained on CIFAR10.**
| Method | Clean | AutoAttack |
| --- | --- | --- |
| TRADES | 81.25 | 48.54 |
| Asym-TRADES | 85.62 | 48.58 |
| w/o Stop-grad | 82.13 | 48.97 |
| w/o Split-BN | 80.90 | 49.93 |
| LBGAT | 85.50 | 49.26 |

Aymmetrizing TRADES with stop-gradient operation and split-BN.
To further validate our findings, we consider asymmetrizing TRADES based on AR-AT’s strategy.
Specifically, we replace the regularization term of AR-AT with TRADES’s regularization term, which we call “Asym-TRADES," as shown in Tab. 12.
Here, we did not employ the predictor MLP in Asym-TRADES since having a bottleneck-structured predictor MLP does not make sense for logits.
Similar to the phenomenon in representation-based regularization, Asym-TRADES does not suffer from “gradient conflict" (Fig. 5), and achieves higher robustness and accuracy than TRADES (Tab. 13).

Asymmetrized TRADES achieve approximately the same performance as LBGAT.
Intriguingly, Tab. 13 demonstrates that Asym-TRADES achieve approximately the same performance as LBGAT.
This validates our hypothesis that the effectiveness of the KD-based method LBGAT can be attributed to resolving “gradient conflict" and the mixture distribution problem, as discussed in Sec. 5.4.
LBGAT employs a separate teacher network to regularize the student network, which implicitly resolves the “gradient conflict" and the mixture distribution problem.

Representation-based regularization can outperform logit-based regularization
By comparing Tables 13 and 2, we observe that ARAT with representation-based regularization can outperform ARAT with logit-based regularization, “Asym-TRADES".
We hypothesize that, since early layers have more and diverse level of information than logits, representation-based regularization can be more effective than logit-based regularization.

## Appendix F Additional Ablation Study

### F.1 Strength of Invariance Regularization γ 𝛾 \gamma italic_γ

Tab. 14 shows the ablation study of AR-AT on the strength of invariance regularization $\gamma$.
We observe that the performance is not too sensitive to the strength of invariance regularization $\gamma$.
Nevertheless, the optimal $\gamma$ depends on the architecture.
For example, $\gamma=30.0$ is optimal for ResNet-18, while $\gamma=100.0$ is optimal for WRN-34-10.

**Table 14: AR-AT’s sensitivity to the strength of invariance regularization $\gamma$.**
|  |  | CIFAR10 | CIFAR100 |  |  |
| --- | --- | --- | --- | --- | --- |
|  | $\gamma$ | Clean | PGD-20 | Clean | PGD-20 |
| ResNet-18 | 10.0 | 87.90 | 51.53 | 66.89 | 26.79 |
| 30.0 (default) | 87.93 | 52.13 | 68.07 | 26.76 |  |
| 50.0 | 87.54 | 51.27 | 67.72 | 26.52 |  |
| 100.0 | 87.06 | 50.14 | 66.19 | 26.02 |  |
| 120.0 | 86.46 | 50.15 | 65.31 | 25.71 |  |
| WRN-34-10 | 10.0 | 89.21 | 48.71 | 67.46 | 26.86 |
| 30.0 | 90.93 | 49.68 | 70.69 | 26.86 |  |
| 50.0 | 91.36 | 50.51 | 71.52 | 26.46 |  |
| 100.0 (default) | 91.29 | 52.24 | 71.58 | 28.06 |  |
| 120.0 | 91.39 | 51.94 | 71.92 | 27.31 |  |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) The novelty of this paper may be somewhat limited, as the proposed stop-gradient operation and predictor are commonly utilized in self-supervised learning.

(2) Additional evidence and experiments addressing the mixed distribution problem should be provided.

(3) There are a few minor errors, such as the notation "clean representation Z^' " in line 191, which should be corrected.

(4) The ImageNet dataset should be incorporated into the experiments to enhance the robustness of the results.
soundness: 3
strengths: (1) This paper is well written and easy to follow.

(2) The paper validates the effectiveness of AR-AT through experiments on multiple datasets and model architectures, demonstrating superior performance in the robustness-accuracy trade-off.

(3) The paper not only presents a new method but also provides an in-depth analysis of the “gradient conflict” and mixed distribution problems in invariance regularization, offering new theoretical insights for the field of adversarial defense.
summary: This paper proposes a novel method called Asymmetric Representation-regularized Adversarial Training (AR-AT), aimed at improving the robustness-accuracy trade-off in adversarial training. AR-AT achieves this goal by addressing two key issues in invariance regularization: (1) the “gradient conflict” between invariance loss and classification objectives, and (2) the mixed distribution problem caused by the differences in input distributions between clean and adversarial samples. AR-AT introduces asymmetric invariance regularization, a stop-gradient operation, a predictor, and a split BatchNorm (BN) structure. Experimental results show that AR-AT outperforms existing methods across various settings and provides new insights into knowledge distillation-based defenses.
weaknesses: (1) The novelty of this paper maybe limited. The proposed stop-gradient operation and predictor are often adopted in self-supervised learning 

(2) More evidence and experiments on mixed distribution problem should be claimed. 

(3) There are a few minor errors, such as "clean representation z^' " in line 191.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Since the proposed method applies invariance regularization to multiple layers, I wonder if the baseline methods also adopt such multi-layer regularization.  

2. Does your method introduce additional computation costs compared to baseline methods?
soundness: 3
strengths: 1.  The paper is well organized. The target issues are clear and each step of the solution is motivated and well represented.
2.  Experiments clearly show the effectiveness of each technique they proposed.
summary: This paper proposed a bag of tricks to address two issues in siamese-based adversarial training, i.e., ``conflict gradients'' and the mixture distribution problem.
weaknesses: In Table 3, some baseline methods provide error bars, but others do not.
 
Moreover, considering the huge body of literature on adversarial training, I have concerns about whether the presented comparison with baseline methods is sufficient and comprehensive.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Where is Figure 4?

2. What is the computational overhead of your method compared to other baselines?
soundness: 4
strengths: 1. The paper is easy-to-follow

2. The motivation of the proposed method is clear
summary: The authors propose AR-AT, which uses asymmetric invariance loss with stop-gradient operation and a predictor to avoid gradient conflict, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem, to improve robustness-accuracy trade-off. The experiments demonstrate the effectiveness of this method.
weaknesses: 1. In Figure 3, can you explain why $||z-z'||_2$ increases over time even though you add a regularization term? and why the use of stop-grad exacerbates this issue?

2. Line 200-201 & Figure 2: Plotting curves of only minimizing the second term can enhance your claim.

3. Table 17 indicates that solely resolving the mixture distribution problem can already improve the performance to a large extent. However, addressing gradient conflict alone does not contribute much to robustness, especially for (1)vs(2) and (5)vs(6). Thus, you should also report the standard deviations to make your results more convincing.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Ref Weaknesses.
soundness: 3
strengths: This work is well-written and easy to understand. The perspective on gradient conflict is particularly interesting, connecting previously disparate threads in adversarial robustness research.
summary: This work addresses the fundamental robustness-accuracy trade-off in adversarial defenses through a novel invariance regularization approach. They proposed the method, AR-AT, achieves state-of-the-art performance by systematically addressing two key challenges: the gradient conflict between classification and invariance losses, and the mixture distribution problem in adversarial training. They resolved the gradient conflict through a strategic stop-gradient operation, while implementing a split batch normalization structure to handle the mixture distribution challenge.
weaknesses: First, the paper's central premise about gradient conflicts requires deeper theoretical examination. Similar gradient conflicts arise in various scenarios, such as when optimizing the same loss across different mini-batches, yet these conflicts don't necessarily impact model generalization (or adversarial robustness). The paper doesn't provide sufficient theoretical justification for why resolving gradient conflicts specifically improves adversarial robustness.

Second, there's a critical gap in the analysis between local optimization dynamics and global distributional properties. The gradient conflicts observed at the mini-batch level may not accurately reflect the underlying conflicts in the full input distribution. Without this theoretical bridge, it's unclear whether the proposed solution addresses the fundamental cause of the robustness-accuracy trade-off.

Third (minor), the experimental results may not be the SOTA. Ref https://github.com/wzekai99/DM-Improves-AT

Meta-review: This paper tackles the long-standing challenge of balancing robustness and accuracy in adversarial training by introducing a method called ARAT (Asymmetric Representation-regularized Adversarial Training). The authors focus on two key problems: gradient conflicts between invariance and classification objectives, which can hinder training, and the mixture distribution problem that arises from differences between clean and adversarial inputs. Their solution combines an asymmetric invariance loss with a stop-gradient operation and a split BatchNorm structure. The results show clear improvements over existing methods, backed by solid experiments and insightful analysis.

The paper is well-written, and particularly the use of asymmetric designs in AT is new and insightful. Reviewers note its clarity and strong empirical results and each component of the proposed method is carefully evaluated. The authors provided detailed explanations during the rebuttal to address questions about novelty and baseline comparisons. This effort significantly strengthened the reviewers' confidence in the work.

That said, some areas for improvement remain. A deeper theoretical exploration of the link between gradient conflict resolution and the robustness-accuracy trade-off would have made the paper even stronger. The lack of large-scale experiments, like those on ImageNet, was another point of critique.

Overall, the paper stands out for its  novel insights and solutions to adversarial training. It’s a solid contribution to the field and well-deserving of acceptance.

Decision: Accept (Poster)