Modeling Adversarial Noise for Adversarial Defense
Authors: Dawei Zhou
Nannan Wang
Bo Han
Tongliang Liu
Venue: ICLR 2022 Submitted (2022)

Abstract: Deep neural networks have been demonstrated to be vulnerable to adversarial noise, promoting the development of defense against adversarial attacks. Motivated by the fact that adversarial noise contains well-generalizing features and that the relationship between adversarial data and natural data can help infer natural data and make reliable predictions, in this paper, we study to model adversarial noise by learning the transition relationship between adversarial labels (i.e. the flipped labels used to generate adversarial data) and natural labels (i.e. the ground truth labels of the natural data). Specifically, we introduce an instance-dependent transition matrix to relate adversarial labels and natural labels, which can be seamlessly embedded with the target model (enabling us to model stronger adaptive adversarial noise). Empirical evaluations demonstrate that our method could effectively improve adversarial accuracy.

Under review as a conference paper at ICLR 2022
MODELING ADVERSARIAL NOISE FOR ADVERSARIAL DEFENSE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have been demonstrated to be vulnerable to adversarial noise, promoting the development of defense against adversarial attacks. Motivated by the fact that adversarial noise contains well -ell-generalizing features and that the relationship between adversarial data and natural data can help infer natural data and make reliable predictions, in this paper, study model adversarial noise by learning the transition relationship between adversarial labels (i.e. the flipped labels used to generate adversarial data) and natural labels (i.e. the ground truth labels of the natural data). Specifically, we introduce an mnstance-dependent transition matrix relate adversarial labels and natural labels, which can seamlessly embedded with the target model (enabling us to model stronger adaptive adversarial noise). Empirical evaluations demonstrate that our method could effectively improve adversarial accuracy.
INTRODUCTION
Deep neural networks have been demonstrated to be vulnerable to adversarial noise (Goodfellow
al. 2015 Szegedy et al. 2014
Jin et al.
2019 Liao et al. 2018 Ma et al. 2018 Wu et al. (020a))
The vulnerability of deep neura
networks seriously threatens many
deep learning
applications (LeCun al. 1998
He al. 2016; Zagoruyko & Komodakis 2016 Simonyan &
Zisserman
2015 Kaiming et al.
2017 Ma et al. (021).
To alleviate the negative affects caused by adversarial noise, number of adversarial defense methods have been proposed. A major class of adversarial defense methods focus exploiting adversarial examples to help train the target model (Madry et al.) 2018 Ding et al.) 2019 Zhang et al.) 2019 Wang et al. 2019), which achieve the state-of-the-an performance. However, these methods do not explicitly model the adversarial noise. The relationship between the adversarial data and natural data has not been well studied yet.
Studying (or modeling) the relationship between adversarial data and natural data is considered to be beneficial. If we model the relationship, the adversarial data and the relationship. Previous researches have made some explorations on this idea. Some data pre- proceesociing based methods (Jin al. 2019 Liao al. 2018 Naseer al.) 2020; Zhou et al. 2021), which try to recover the natural data by removing the adversarial noise, share the same philosophy. However, those methods suffer from the high dimensionality problem because both the adversarial data and natural data are high dimensional. The recovered data is likely have camman-observallee loss (i.e., obvious inconsistency between processed examples and natural examples) (Xu et al. 2017) or contain residual adversarial noise (Liao et al. 2018).
infer natural data information by exploiting
To avoid the above problems, in this paper, we propose model adversarial noise in the low- dimensional label space. Specifically, instead of directly modeling the relationship between the adversarial data and the natural data, we model the adversarial noise by learning the label transition from the adversarial labels (i.e., the flipped labels used to generate adversarial data) to the natural labels (i.e., the ground truth labels of the natural data). Note that the adversarial labels have not been well exploited in the community of adversarial learning, which guide the generation of adversarial noise and thus contain valuable information for modeling the well-generalizing features of the adversarial noise (lyyss et al. 2019).

---

Under review as a conference paper at ICLR 2022
It is well-known that the adversarial noise depends many factors, e.g., the data distribution, the adversarial attack strategy, and the target model. The proposed adversarial noise modeling method is capable to take into account of the factors because that the label transition is dependent of the adversarial instance (enabling us to model how the patterns of data distribution and adversarial noise affect label flipping) and that the transition can be seamlessly embedded with the target model (enabling us to model stronger adaptive adversarial noise). Specifically, we employ deep neural network to learn the complex instance-dependen label transition.
By using the transition relationship, we propose a defense method based on Modeling Adversarial Noise (called MAN). Specifically, we embed a label transition network (i.e., a matrix- matrix-valued function parameterized by deep neural network) into the target model, which denotes the transition relationship from mixture labels (mixture of adversarial labels and natural labels) to natural labels (i.e.. the ground truth labels of natural instances and adversarial instances). The transition matrix explicitly models adversarial noise and help us to infer natural labels. Considering that adversarial data can be adaptively generated, we conduct joint adversarial training the target model and the transition network to achieve an optimal adversarial accuracy, We empirically show that the proposed MAN based defense method could provide significant gains in the classification accuracy against adversarial attacks in comparison to ofate-of-thh-ar defense methods.
valued transition
The rest of this paper is organized follows. In Section and briefly review related work on attacks and defenses. In Section 3 we discuss how to design an adversarial defense by modeling adversarial noise. Experimental results are provided in Section 4 Finally, we conclude this paper in Section 5
introduce some preliminary information
2 PRELIMINARIES
In this section, we first introduce some preliminary about notation and the problem setting. We then review the most relevant literature on adversarial attacks and adversarial defenses.
Notation. We use capital letters such as X and Y to represent random variables, and lower-case letters such as x and y to represent realizations of random variables X and Y, respectively. For norms, we denote by | a generic norm. Specific examples of norms include || the L norm of xr and ||a||| 2, the T,² norm of x. Let B(x, c) represent the neighborhood of x: { : where is the perturbation budget. We define the classification function as f X + {1, 2, C}. can be parametrized, e.g., by deep neural networks.
xl| S c.
Problem setting. This paper focuses on a classification task under the adversarial environment, where adversarial attacks are utilized to craft adversarial noise to mislead the predictions of the target classification model. Let X and Y be the variables for natural instances and natural labels (i.c., the ground truth labels of natural instances) respectively. We sample natural data {(), y1) Vi according the distribution of the variables (X, Y), where (X, Y) E X x 21, C). Given deep learning model based classifier f and a pair of natural data (z, y), the adversarial instance ă satisfies the following constraint:
(o) +
s.t. ||x ž|| S.
The generation of the adversarial instance is guided by the adversarial label ÿ which is different with the natural label y. Adversarial labels can be set by the attacker in target attacks, or be randomly initialized and then found by non-target attacks. Let X and Y be the variables for adversarial instances and adversarial labels respectively. We denote by [ } the adversarial data drawn according to distribution of the variables (X, Y). Our aim is to design an adversarial defense correct the adversarial label i into natural label y by modeling the relationship between adversarial data (i.e., T. or ğ) and natural data (i.e., 3: or y)-
A Adversarial attacks. Adversarial examples are inputs maliciously designed by adversarial attacks to mislead deep neural networks (Szegedy et al. 2014). They are generated by adding imperceptible but adversarial noise natural examples. Adversarial noise can be crafted by multi-step attacks following the direction of adversarial gradients, such as PGD (Madry et al. 2018) and AA (Croce & Hein S c, that their adversarial examples can be perceptually similar to natural examples. Optimization based attacks such as CW (Carlini & Wagner 7b) and DDN (Rony et al. 2019) jointly minimize the perturbation L22 and
2020). These adversarial noise is bounded by small nolm-ball
2

---

Under review as a conference paper at ICLR 2022
differentiable loss based on the logit output of classifier. Some attacks such as FWA (Wu et al. 2020b) and STA (Xiao et al. 2018) focus on mimicking non-suspicious vandalism by exploiting the geometry and spatial information.
Adversarial defenses. The issue of adversarial examples promotes the development of adversarial defenses. A major class of adversarial defense methods is devoted to enhance the adversarial robustness in an adversarial training manner (Madry et al.) 2018 Ding et al. 2019 Zhang et al.) 2019; Wang et al. 2019). They augment training data with adversarial examples and use min-max formulation to train the target model (Madry et al.] 2018). However, these methods do not explicitly model the adversarial noise. The relationship between the adversarial data and natural data has not been well studied yet.
In addition, some data pre- pre-processing based methods try to remove adversarial noise by learning denoising function or feature-squeeziin
function. For example, denoising based defenses (Liao jet al., 2018: Naseer et al.) 2020) transfer adversarial examples into clean examples, and feature squeezing based defenses (Guo et al. 2018) aim to reduce redundant but adversarial information. However, these methods suffer from the high dimensionality problem because both the adversarial data and natural data are high dimensional. For examples, the recovered data is likely to have human-observable loss (Xu et al. 2017) (i.e., the obvious inconsistency between processed examples and natural examples), and the recovered data may contain residual adversarial noise (Liao et al.) 2018), which would be amplified by the target model and mislead the final prediction. To avoid the above problems, we propose model adversarial noise in the low-dimensional label space.
3 MODELING ADVERSARIAL NOISE BASED DEFENSE
In this section, we present the proposed Modeling Adversarial Noise (MAN) based defense method, to improve the adversarial accuracy against adversarial attacks. We first illustrate the motivation of the proposed defense method (Section p.Tpp. Next, introduce how to model adversarial noise (Sec- tion 3.2h. Finally, we present the training process of the proposed adversarial defense (Section 3.3)
3.1 MOTIVATION
Studying the relationship between adversarial data and natural data is considered to be beneficial for adversarial defense. If we can model the relationship, we can infer clean data information by exploiting the adversarial data and the relationship. Processing the input data to remove the adversarial noise is a representative strategy to estimate the relationship. However, data pre-processing based methods may suffer from high dimensionality problems. To avoid the problem, would like to model adversarial noise in the low-dimensional label space.
Adversarial noise can be modeled because it have imperceptible but well-generalizing features. Specifically, classifiers are usually trained to solely maximize accuracy for natural data. They tend to any available signal including those that are well-generalizing. yet brittle. Adversarial noise can arise as result of perturbing these features (Ilyas et al., 2019). and it controls the flip from natural labels to adversarial labels. Thus, adversarial noise contains well-generalizing features which can be modeled by learning the label transition from the adversarial labels to natural labels.
Note that the adversarial noise depends on many factors, such as data distribution, the adversarial attack strategy, and the target model. Modeling adversarial noise in label space is capable to take into account of these factors. Specifically, since that the label transition dependent of the adversarial instance, we can model how the patterns of data distribution and adversarial noise affect label flipping, and exploit the modeling to correct the flipping of the adversarial label. In addition, we can model adversarial noise crafted by the stronger white-box adaptive attack, because the label transition can be seamlessly embedded with the target model. By jointly training the transition matrix with the target model, it can also be adaptive adversarial attacks. We employ deep neural network to learn the complex label transition from the well- well-generalizing and misleadingly predictive features of irttace-ddeendenn adversarial noise.
Motivated by the value of modeling adversarial noise for adversarial defense, we propose a defense method based Modeling Adversarial Noise (MAN) by exploiting label transition relationship.
3

---

Under review as a conference paper at ICLR 2022
Transition network Su
122
Mitrootd Nakuel
Input instance
Transitio
x'
matris
(b)
Figure 1: (a). Infer the natural label by utilizing the transition matrix and the adversarial label. (b). The transition network yo parameterized by w takes the instance x' as input, and output the label transition matrix :w); (1').
3.2 MODELING ADVERSARIAL NOISE
We exploit transition network to learn the label transition for modeling adversarial noise. The transition network can be regarded as a matrix-valued transition function parameterized by a deep neural network. The transition matrix, estimated by the label transition network, explicitly models adversarial noise and help us to infer natural labels from adversarial labels. The details are discussed below.
Transition matrix. To model the adversarial noise, we need to relate adversarial labels and natural labels in a explicit form (e.g., matrix). Inspired by recent researches in label-noise learning (Xia
eet al. 2020 Liu & Tao 2013] Xia et al. [2199] Yang et al.) 2021 Wu et al., 2021] Xia et al. 2021). we design label transition matrix, which can encode the probabilities that adversarial labels flip
into natural labels. We then can infer natural labels by utilizing the transition matrix and adversarial
data (see Figure I(a) Note that the transition matrix used in our work is different from that used in label-noise learning. The detailed illustration can be found in Appendix A
Considering that the adversarial noise is instance-dependeet, the transition matrix should be designed to depend on input instances, that is, we should find a transition matrix specific to each input instance. In addition, we not only need to focus on the accuracy of adversarial instances, but also the accuracy of natural instances. An adversarially robust target model is often trained with both natural data and adversarial data. We therefore exploit the mixture of natural and adversarial instances (called mixture instances) as input instances, and design the transition matrix to model the relationship between the mixture natural and adversarial labels (called mixture labels) to the ground truth labels of mixture instances (called natural labels).
Specifically, let X' denote the variable for the mixture instances, Y' denote the variable for the mixture labels, and Y denote the variable for the natural labels. We combine the natural data and adversarial data as the mixture data, i.e., {< km) = ((x,,))) {(1r, 2 is the data drawn according a distribution of the random variables (X', Y"). Since we have the ground-truth labels of mixture instances (i.e., natural labels), we can extend the mixture data ((a), )2 into triplet form { 8/. y/ ) We utilize the transition matrix T e [o, 1 to model the relationship between mixture labels Y' and natural labels Y. The transition matrix T is dependent of the mixture instances X'. It defined as:
U ((E
where
(X' = x') = P(Y
i,'' x'),
where Tæj denotes the (i.j) - th element of the matrix T(X' = ='). It indicates the probability of the mixture label flipped to the natural label j for the input x'.
The basic idea that given the mixture class posterior probability (i.e., the class posterior probability of the mixture labels Y') P(Y" X' = x') = [P(Y" = X' = ='),').-- ........ P(y = C. X" = x')] the natural class posterior probability (i.e., the class posterior probability of natural labels Y) P(Y X' = x') could be inferred by exploiting P(Y'| X' x') and T(X' =x)::
P(Y X x') T(X'= r') P(Y" X' x')=)).
(3)
We then obtain the natural labels by choose the class labels that maximize the robust class posterior probabilities. Note that the mixture class posterior probability P(Y' X' = x') can be estimated by exploiting the mixture data.
Transition network. we employ a deep neural network (called transition network) to estimate the label transition matrix by exploiting the mixture data ((1,, vl, vi) 2- Specifically, as shown in
4

---

Under review as a conference paper at ICLR 2022
Target bg
Adverariaa nedoe
Masintio cros- catrapy loss
Transition marris P(c):0)
Mixture Instance
Target model fe
Transition network
Transition network
training
Natural
instance
Figure 2: An overview of the training procedure for the proposed defense method. ÿ' and ý denote the probability of the estimated mixture label v' and the probability of the inferred natural label respectively, i.e., ğ' = ((h)(z²)) and ÿ = - T(x'; y y in the form vector.
Figure 1(b), the transition network (.(() (.) parameterized by takes the mixture instance x' as input, and output the label transition matrix (a):w) = (w)) We then can infer the natural class posterior probability P(Y X = x') according to Eq. and thus infer the natural label. To optimize the parameter w of the transition network, we minimize the difference between the inferred natural labels and the ground-truth natural labels. The loss function of the transition network define as:
2n
Σ Ely! T((hw), y.). (l(W) = 2n i=1
(4)
where v Yi are v/. yi the form vectors. e(-) is the cross-entropy loss between the inferred natural labels and the ground-truth natural labels, i.e., l(v) (w)::),, yı) = yi log(v/ (r//w)).
3.3 TRAINING
Considering that adversarial data can be adaptively generated, and the adversarial labels are also depended on the target model, we conduct joint adversarial training the target model and the transition network to achieve the optimal adversarial accuracy. We provide the overview of the training procedure in Figure
For the transition network, we optimize its model parameter w according to Eq. For the target model, considering that the final inferred natural class posterior probability is influenced by the target model, we also use the cross-entropy loss between the inferred natural labels and the ground-truth natural labels optimize the parameter 0. The loss function for the target model he is defined as:
2n
Cear( = 2n 1=1
yv log(0/ T(rhw))) Øf =(ha(a)))
(5)
where 6(.) denote the softmax function. // denote the mixture label the form vector predicted by the target model.
To demonstrate the effectiveness of joint adversarial training, we conduct an ablation study by independently training the transition network with fixed pre-trained target model (see Section 4.3) In addition, to prove that using mixture instances can achieve better adversarial accuracy compared to only using adversarial instances, conduct ablation study by training the proposed defense using only adversarial data (see Section 4.3
The details of the overall procedure presented in Algorithm Specifically, for each mini-batch { sampled from natural training set, we first generate adversarial instances 03 { ) via a strong adversarial attack algorithm, and obtain the mixture mini-batch B' = {-]) (i.c.. mixture of B and [3) with mixture labels 1mm
! (i.e., mixture of adversarial labels {ū. )m and natural labels { 1m Then, we input the mixture instances 1m into the target model (.).)
and output (0) ==1- We also input the mixture instances (x:) 2m into the transition network g.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Training of Modeling Adversarial Noise (MAN) based defense
Input: Target model h-)+) parametrized by 0, transition network (-(+) parametrized by w, batch size m, and the perturbation budget 6;
1: repeat
2: Read mini-batch B = {2 )2 from training set;
3: Craft adversarial instance {2:] mm at the given perturbation budget for each instance æs; in B;
4: Obtain mixture mini-batch B' [=]]== with mixture labels {vi]==
5: for to 2m (in parallel) do
6: Forward-pat 2 through (-) and obtain V6:
7: Forward-pass x through (.) to estimate T(d;;
8: Infer final prediction label by exploiting of and ((dlsw);
9:
end for
10: Calculate (r(w) and Ltar (0) using Eq. 4 and Eq.
11: Back-pass and update w and 0:
12:
until training converged.
to estimate the label transition matrices {)) We next infer the final prediction labels by exploiting ) and (T'; ) Finally, we compute the loss functions (r(w) and Cear(0), and update the parameters w and 6. By iteratively conducting the procedures of adversarial example generation and defense training, and 0 are expected to be adversarially optimized.
EXPERIMENTS
In this section, we first introduce the experiment setup including datasets, attack settings and defense settings in Section 4.1 Then, we evaluate the effectiveness of our defense against representative and commonly used L norm and L norm adversarial attacks in Section 4.2 4.2 Finally, conduct
ablation studies in Section 4.3
4.1 EXPERIMENT SETUP
Datasets. We verify the effective of our defense method on two popular benchmark datasets. i.e., CIFAR-10 (Krizhevsky et al.) (009) and Tiny-mageNNrr (Wu et al. 2017]. CIFAR- has 10 classes of images including 50,000 training images and 10,000 test images. Tiny-ImageNel has 200 classes of images including 100,000 training images, 10,000 validation images and 10,000 test images. Images in the two datasets are all regarded as natural instances. All images are normalized into [0,1], and are performed simple data augmentations in the training process, including random crop and random horizontal flip. For the target model, we use ResNet-18 (He et al. 2016) for CIFAR-1O and Tiny-mmageNet.
Attack settings. Adversarial examples for evaluating defense models are crafted by applying state- of-the-art attacks. These attacks include - norm PGD (Madry et al.) (2018). L norm AA (Croce & Hein 2020), 0 norm TI-DIM (Dong et al. 2019 Xie et al. 2019) 7, norm FWA (Wu et al. 2020b). T, norm CW (Carlini & Wagner 201 7b) and 1,² norm DDN (Rony et al. 2019). Among them, the AA attack algorithm integrates three non-target attacks and a target attack. Other attack algorithms belong to non-target attacks. The iteration number of PGD is set to 40 with step size 0.007. The iteration number of FWA is set to 20 with step size 0.007. The iteration numbers of CW, and DDN are set to 200 and 40 respectively with step size 0.01. For CIFAR=10 and Tin-ImageNeer, the perturbation budgets for 2 norm attacks and L norm attacks are e 0.5 and 8/255 respectively.
Defense settings. We use three representative defense methods as the basclines: standard adversarial training AT (Madry al., 2018), TRADES (Zhang et a (209)) and MART (Wang et al. 2019). We use the ResNet-18 18 as the transition network for CIFAR-TO and Tiny-ImageNer.. The perturbation budget e is set to 8/255 for CIFAR-IO and defense, and 100 for baselines. For fair comparisons, all the methods are trained using SGD with momentum 0.9 and an initial learning rate of 0. The weight decay is 2 x 10 4 for CIFAR- 10, and is 5 x 10 for -ing-enaeeNe.. For TRADES and MART, set À 6. We use the - norm non non-target PGD-10 (i.c., PGD with iteration number of 10) with random start and step size e/4 to craft adversarial training data for all defenses
The epoch number is set to 150 for our

---

Under review as a conference paper at ICLR 2022
4.2 DEFENSE EFFECTIVENESS
Defending against adaptive attacks. A powerful adaptive attack strategy has been proposed to break defense methods (Athalye et al. | Carlini & Wagner 22]7a). In this case, the attacker can access the architecture and model parameters of both the target model and the defense model (i.e., the transition network). We study the following attack scenarios for evaluating our defense.
Scenario (i). Considering that the models in baselines are completely leaked to the attacker, for fair comparison, we utilize white-box adversarial attacks against the combination of the target model and the transition matrix. Similar to attacks against baselines, the goal of the non-target attack in this scenario is to maximize the distances between final predictions of our defense and the ground-truth natural labels. The adversarial instance èt is crafted by solving the following optimization problem:
max C(ÿ T(; w)) y). subject to: || - 2 S
where ý (+)) and C()) denote the specific loss function used by each attack. Similarly, we can craft adversarial examples via the target attack. The details are presented in Appendix B
We combine this attack strategy with six representative adversarial attacks introduced in section atlack setfings to evaluate defenses. For training our defense, We use adversarial examples crafted by adaptive PGD-10 10 attack as adversarial training data. The average natural accuracy (i.e., the results in the second column) and the average adversarial accuracy of defenses are shown in Table
The results show that our defense achieves superior adversarial accuracy compared with AT. This presents that defense is effective. In addition, our method achieves significant improvements against more destructive FWA, CW2 and DDN attacks (e.g., the adversarial accuracy is increased from 0.00% 46.36% against CW2). Note that the training mechanism in method can be regarded as the standard adversarial training on the combination of the target network and the transition matrix, our method thus is applicable to different adversarial training methods. To avoid the bias caused by different adversarial training methods, we apply our method to TRADES and MART. As shown in Table I the results show that our method improves the adversarial accuracy. We find that using TRADES and MART would affect the performance against L2 Lo-no based attacks. We will further solve this issue in future work. The results on Tiny-lmageNer are shown in Appendix(B..
In addition, we evaluate the adversarial accuracy of defense methods against white- box adaptive attacks on CIFAR-IO by using the VggNet- 19 as the target model and the transition network. The results are also shown in Appendix B.1
Table Adversarial accuracy (percentage) of defense methods against white-box adaptive attacks on CIFAR-10. The target model is ResNet-18. We show the most successful defense with bold.
Attack
None
PGD-40
AA
FWA-40
TI-DIM
CW2
DDN
AT
83.39
42.38
39.01
15.44
57.12
0.00
0.09
MAN
82.71
44.84
39.31
29.70
58.88
46.36
10.73
TRADES
80.68
49.38
44.43
23.92
57.45
0.00
0.16
MAN TRADES
76.28
49.94
44.49
30.51
57.53
7.69
1.18
MART
76.15
50.93
44.37
25.63
59.23
0.00
0.02
MAN MART
75.29
50.97
44.45
31.24
59.28
6.08
1.06
Scenario (ii). In this scenario, we design an adversarial attack destroy the crucial transition matrix of our defense. If the transition matrix is destroyed, the defense would immediately become ineffective. Since the ground-truth transition matrix is not given, we use the target attack strategy to craft adversarial examples. We choose an anti-diagonal identity matrix (see Figure as an example of the target transition matrix in the target attack. The optimization goal designed as:
4 in Appendix B.2)
max - -mse (T (i;w), T*), subject to: || all <
where Lise denote the mean square error loss. We just showed two examples for non- ttrgeteeee and targeted attacks designed to the transition network in the above two scenarios. Note that different attacks can be designed, which however is beyond the scope of our work.
7

---

Under review as a conference paper at ICLR 2022
We use - norm PGD-40 with this target attack strategy to evaluate the MAN based defense trained in scenario (i). The adversarial accuracy is 70.26% on CIFAR-1O, which represents that our defense is effective against this adaptive target attack. This may be because that the attack in scenario (i) craft adversarial data for reducing the accuracy by destroying the transition matrix as well. The transition network adversarially trained on such adversarial data thus have good robustness against the attack designed for the transition matrix.
Scenario (iii). We explore another adaptive attack scenario. In this scenario, the attack not only disturbs the final prediction labels, but also disturbs the output of the target model. This attack is called dual adaptive attack. The optimization goal of the non-target attack can be designed as:
max [C(ū- - (i) v) + Cee y)), subject to: || - -1| c.
(8)
where ğ - (T),, (-)) denote the specific loss function used by each adversarial attack and Lon the cross-entropy loss of the target model, i.e. Lee (ū. y) = - Σ Y: log Similarly, ly, we can craft adversarial examples via the target attack. The details are presented in Appendix [B.
We combine this dual adaptive attack strategy with above six attacks to evaluate the proposed defense trained in scenario (i). As shown in Table 2 the results in the second row are the adversarial accuracy of our MAN based defense in this scenario, the results in the third row (Modely:) show that accuracy of the target model for adversarial instances. The results demonstrate that defense method can infer natural labels by exploiting the instance-dependent transition matrix and adversarial labels, it provides effective protection against multiple attacks in this scenario.
We note that the adversarial accuracy of MAN in Table is higher than that in Table | This may be because, for some input instances, the dual adaptive attack mainly uses the gradient information of the attack loss only against the target model, to craft adversarial noise for breaking the target model. Our MAN based defense can model the adversarial noise (i.e., learn the transition relationship) for the ad adversarial label predicted by the target model. The final natural label can be inferred, and the final adversarial accuracy thus can be improved.
Table 2: Adversarial accuracy (percentage) of our MAN based defense against dual white- box adaptive attacks (the second row) and against general attacks (the third row) on CIFAR-1O. The target model is R8
Attack
None
PGD-40
AA
FWA-40
TI-DIM
CW
DDN
MAN
82.71
70.91
39.41
64.18
72.87
46.37
38.74
Modely
8.64
0.34
2.50
0.64
0.63
2.04
0.64
MAN"
89.01
81.07
79.90
80.02
61.93
77.89
77.82
Model
88.98
0.00
0.00
0.00
10.45
0.00
0.00
Defending against general attacks. We also evaluate the effectiveness of the proposed defense by using general attacks, i.e., the adversarial attacks against the target model. We utilize above six adversarial attacks to evaluate the proposed MAN based defense. We use the adversarial instances crafted by non-target _ norm PGD-10 attack against the target model as the adversarial training data, to train our defense model. The adversarial accuracy of our MAN based defense is shown in Table 2 (MAN*). The adversarial accuracy of the target model (Model,,) also shown on the fifth row in Table 2 It can be seen that our defense method achieve a great defense effect against adversarial examples which are crafted against the target model.
Defense transferability. The work in Ilyas et al. (2019) shows that any two target models are likely to learn similar mixture pattern, so modeling the adversarial noise which manipulate such features would apply to both target models for improving adversarial accuracy. We apply the MAN based defense which is trained for the ResNet- 18 (ResNet), to two naturally pre-trained target models VGG-19 (VGG) (Simonyan& & Zisserman| 2015) and Wide-ResNet 28 10 (WRN) (Zagoruyko & Komodakis 2016) for evaluating the transferability of our defense. The performance on CIFAR-1O is reported in Table 3 It can be seen that our defense method has effective transferability for providing cross-model protections against adversarial attacks. In addition, we transfer our defense model to VGG-19 for defending against adaptive attacks. The details can be found in Appendix B.4
8

---

Under review as a conference paper at ICLR 2022
Table 3: Adversarial accuracy (percentage) of our defense method for different target models on CIFAR-10 against general adversarial attacks.
Attack ResNet
None
PGD-40
AA
FWA-40
TI-DIM
CW2
DDN
89.01
81.07
79.90
80.02
61.93
77.89
77.82
VGG
89.17
72.05
71.86
71.25
61.01
77.03
77.80
WRN
91.19
70.08
69.66
69.40
57.58
77.05
77.70
4.3 ABLATION STUDY
To demonstrate that the joint adversarial training on the target model and the transition network could achieve better defense compare only training the transition network, we conduct an ablation study on our MAN based defense. Specifically, we use a naturally pre-trained ResNet-18 18 target model, and train the transition network independently. The model parameters of the target model are fixed in the training procedure, The defense model is called as MAN-. We use the adversarial examples crafted by L norm PGD-10 against the target model as adversarial training data. Compared with the above MAN", the results shown in Figure 3(a) demonstrated the joint adversarial training manner can provide positive gain.
In addition, to show the effect of using mixture instances defense performance, only utilize the adversarial instances to retrain the defense model (called MAN'). The adversarial training data are crafted by _ norm PGD-10 with adaptive attack strategy. We use white-box adaptive attacks (i.e., 6) to evaluate the defenses. The results as shown in Figure (bb)) that the using mixture data can achieve better performance in defending adaptive attacks.
Moreover, to show that the improvement of our method has little relationship with the fact that the transition network introduces model parameters, conduct experiment by using two parallel R8 as the target model. The details can be found in Appendix
200
10
+MAN-
#MAN KMAW
(a)
(b)
Figure 3: (a). Ablation study by independently training the transition network. The adversarial examples crafted against the target model. (b). Ablation study by training the proposed defense using only adversarial data. The adversarial examples are crafted by white-box adaptive attacks.
5 CONCLUSION
Traditional adversarial defense methods typically focus on directly exploiting adversarial examples to remove adversarial noise or train adversarially robust target model. In this paper, motivated by that the relationship between adversarial data and natural data can help infer clean data from adversarial data to obtain the final correct prediction, study model adversarial noise by learning the label transition relationship for using adversarial labels to improve adversarial accuracy. We propose defense method based on Modeling Adversarial Noise (called MAN). Specifically, we embed label transition network into the target model, which denote the transition relationship from mixture labels (i.e., mixture of adversarial labels and natural labels) to robust. labels (i.e., natural labels). The transition matrix explicitly models adversarial noise and help us to infer robust labels. Considering that adversarial data can be adaptively generated, we conduct joint adversarial training on the target model and the transition network to achieve optimal adversarial accuracy. The empirical results demonstrate that our defense method can provide effective protection against white-box general attacks and adaptive attacks. Our work provides a new adversarial defense strategy for the community of adversarial learning. In future, we will further optimize the MAN based defense method to improve its performance against white-box adaptive attacks.

---

Under review as a conference paper at ICLR 2022
Ethics Statement This paper doesn't 't raise any ethics concern.. This study doesn't 't involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discriminnaioo/iissaaaans concerns, privacy and security issues, legal compliance, and research integrity issues.
Reproducibility Statement To ensure the reproducibility of this work, we use the codes from the baseline method. We then conduct our experiments on this basis. Note that we report the "last" adversarial accuracy (i.e.. the adversarial accuracy of the last epoch models obtained during training) in this work, instead of the "best" results.
REFERENCES
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference Machine Learning, 2018.
Nicholas Carlini and David Wagner. Magnet and" efficient defenses against adversarial attacks" are not robust to adversarial examples. arXiv preprint arXiv: 1rXiv:1711.08441 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 Ieee Symposium on Security and Privacy (sp). pp. 39-57. IEEE, 2017b.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Percy Liang. Unlabeled data improves adversarial robustness. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buse Bucé-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp. 11190-11201, 11201, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- attacks. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang, On the sensitivity of adversarial robustness to input data distributions. In ICLR (Poster), 2019.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses transferable adversarial ex- amples by translation-invariant attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4312-4321, 2019.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van Maaten. Countering adversarial images using input transformations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv: arXiv:1905.000 1905.02/75 2019.
Guoqing Jin, Shiwei Shen, Dongming Zhang. Feng Dai, and Yongdong Zhang, APE-GAN: adversarial perturbation elimination with GAN. In International Conference on Acoustics, Speech and Signal Processing, pp. 3842-3846, 2019.
He Kaiming, Gkioxari Georgia, Dollar Piotr, and Girshick Ross. Mask r-cnn. IEEE Transactions on Pattern Analysis & Machine Intelligence, PP:1-1, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2022
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 96(11))2278-233 1998.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. Conference on Computer Vision and Pattern Recognition, pp. 1778-1788 2018.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on paltern analysis and machine intelligence, 3873)4447-46,, 2015.
Xingjun Ma. Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi N. R. Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E. Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations, 2018.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Understanding adversarial attacks on deep learning based medical image analysis systems. Puttern Recognition, 110:107332, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, 2018.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self-supervised approach for adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 262-271, 2020.
Jérôme Rony, Luiz G. Hafemann, Luiz S. Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based L2 adversarial attacks and defenses. In Conference on Computer Vision and Pattern Recognition, pp. 4322 4330, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference Learning Representations, 2019.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. Advances in Neural Information Processing Systems, 33, 2020a.
Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical Report, 2017.
Kaiwen Wu, Allen Houze Wang, and Yaoliang Yu. Stronger and faster wasserstein adversarial attacks. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 10377-10387, 2020b.
Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In International Conference Machine Learning, pp. 11285-11 1295. PMLR, 2021.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? arXiv preprint arXiv:1906.0011 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33, 2020.

---

Under review as a conference paper at ICLR 2022
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust carly-learning: Hindering the memorization of noisy labels. International Conference Learning Representations, 2021.
Chaowei Xiao, Jun-Yan Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In 6th International Conference on Learning Representations, 2018.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L. Yuille. Improving transferability adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2730-2739, 2019.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint 1r0iv: 155, 2017.
Shuo Yang, Erkun Yang. Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent label-noise transition matrix using dnns. arXiv preprint arXiv:2105.13001,
2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. Smith (eds.), Proceedings of the British Machine Vision Conference 2016, 2016.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference Machine Learning, pp. 7472-7482. PMLR, 2019.
Dawei Zhou, Tongliang Liu, Bo Han, Nannan Wang, Chunlei Peng, and Xinbo Gao. Towards defending against adversarial examples via attack-invariall features. In Proceedings of the 38th International Conference on Machine Learning, pp. 12835-12845 2021.
12

---

Under review as a conference paper at ICLR 2022
Appendices
A THE DIFFERENCE OF THE TRANSITION MATRIX BETWEEN OUR METHOD AND LABEL-NOISE LEARNING
In label-noise learning, the transition matrix is used to infer clean labels from given noisy labels. The transition matrix denotes the probabilities that clean labels flip into noisy labels (Xia et al. 2020: Liu & Tao 2015 Yang et al. 2021 The label-noise learning methods utilize the transition matrix to correct the training loss on noisy data (i.e., the clean instance with the noisy label). Given the noisy class posterior probability, the clean class posterior probability can be obtained, i.e., ) "py"" (2), where y" denotes the noisy label in the form of a vector. Differently,
p(y] (T(x)) ) our method, the transition matrix used infer natural labels from observed adversarial labels. The transition matrix denotes the probabilities that adversarial labels flip into natural labels. We utilize the transition matrix help compute the training loss on the adversarial instance with the natural label. For the observed adversarial class posterior probability, the natural class posterior probability can be obtained, i.c., p ( = TT) p(ğ addition, our method for the first time utilizes the transition matrix to explicitly model adversarial noise for improving adversarial robustness.
B DEFENDING AGAINST ADAPTIVE ATTACKS
B.1 SCENARIO (I)
In scenario (i), the target attack solve the following optimization problem: max- - Lÿ T(ū:w) w), y²), subject to: | - - S e,
where y is the target label in the form of vector set by the attacker. The adversarial accuracy of defense methods against white-box adaptive attacks CIFAR- 10 and Tiny-maggeNet are shown in Table[aand Table respectively.
Table 4: Adversarial accuracy (percentage) of defense methods against white-box adaptive attacks on CIFAR-IO. 10. The target model ResNet-18. We show the most successful defense with bold. 'MAN T denotes the MAN TRADES, and "MAN M' denotes the MAN MART.
Attack
None
PGD-40
AA
FWA-40
TI-DIM
CW2
DDN
AT
8399
42.38-10.56
39.01
15.4411.32
57.12-40.71
0.00 46.36-11.32
0.09-10.03
MAN
82.71+10.62 0.48
39.31
29.70-
58.88-+0.49
10.73-10.67
TRADES
8.66810063 49.3810.56
44.4310.45
23.92=0.51
57.45+0.30
0.00
0.16100.2
MAN_T
76.28+-0.59
49.94+0.37
4..49-0.63
00.51
0.35
7.69
1,18++0.20
MART
76.15:10.91 50.9310.72
44.3710.83
25.6310.88
59.23:1..99
0.00:10.00
0.02+10.01
MAN.M
75.29110.76
50.97-0.63
44.45:0.79
31.2410.57
59.28-10.37
6.08:11.69
1.06:10.23
Table 5: Adversarial accuracy (percentage) of defense methods against white-box adaptive attacks on Tmny-lmageNet. The target model is ResNet-18. We show the most successful defense with bold. "MANT"" denotes the MAN.TRADES and 'MAN M' denotes the MART.
Attack
None
PGD-40
FWA-40
TI-DIM
CW2
DDN
AT
48.4111.08
17.3811.68
11.30
10.3820.81
29.02-0881
0.00110.00
0.3110.03
MAN
43.80++1661
18.17+1.67
12.46+1.34
13.14-1.78
29.41-2.36
17.31+1.70
4.03--0.21
TRADES
44.61110.4 19.2111.95
12.80+1.43
10.78=1.54
30.36=0.84
0,00110.00
0.0910002
MAN.T
43.66110.83
20.141:.55
12.87-1.21
14.94=1.67
3..40-0.69
1.89-0.56
0.8710.14
MART
42.21 20.93+1.21
15.79+1.41
12.99-0.55
30.81=0.88
0.00-50000
0.04
MAN M
42.20.11.07
2..2311.166
15.85
1.41
31.34-1.03
1.69.10.77
0.63.10.12
In addition, evaluate the adversarial accuracy of defense methods against white-bot box adaptive attacks on CIFAR-10 10 by using the VggNet- 19 as the target model and the transition network. As shown in Table 6 our method still achieves better performance.
13

---

Under review as a conference paper at ICLR 2022
Table 6: Adversarial accuracy (percentage) of defense methods against white-box adaptive attacks CIFAR-IO. The target model is VggNet-19.
Attack
None
PGD-40
AA
FWA-40
TI-DIM
CW₂
DDN
80.91
29.83
26.00
7.55
49.47
0.10
0.15
MAN
80.22
37.18
33.20
17.88
49.75
38.13
6.10
B.2 SCENARIO (II)
In scenario (ii), we design an adversarial attack destroy the crucial transition matrix of our defense. Since the ground-truth transition matrix is not given, we use the target attack strategy to craft adversarial examples. We choose an anti-diagonal identity matrix as an example of the target transition matrix in the target attack. The target transition matrix 7 is shown in Fig 4 Note that there may be other attacks that can designed, this is beyond the scope of our work, and we would not explore further in this paper.
Target transition
Figure 4: Target transition matrix T for the target adversarial attack.
B.3 SCENARIO (III)
In scenario (iii), the optimization goal of the target attack can be designed as:
();:w), max L( - 3²) + Loe (ø. y)], subject to: || - S
(10)
where y the target label in the form of vector set by the attacker.
Note that we report the "last" adversarial accuracy (i.e., the adversarial accuracy of the last epoch models obtained during training) in Table [ 2 and foollooin the discussion in Carmon et al.
(201)), instead of the "best" results.
B.4 DEFENSE TRANSFERABILITY AGAINST ADAPTIVE ATTACKS
We transfer our defense model VggNet-19 19 for defending against adaptive attacks. The original target model is ResNet-18 18. As shown in Table 7 the results show that our method has some transferability against adaptive attacks. We find that the adversarial accuracy of VggNet- than that of ResNet-18, However, this shows the importance and effectiveness of the joint adversarial training on the target model and the transition network in our method.
lower
Table 7: Adversarial accuracy (percentage) of our defense method for different target models on CIFAR-IO against adaptive adversarial attacks.
Model
PGD-40
AA
FWA-40
TI-DIM
CW2
DDN
R8
44.84
39.31
29.70
58,88
46.36
10.73
VggNet-19
19.67
13.73
11.60
13.51
17.32
5.19
14

---

Under review as a conference paper at ICLR 2022
C THE INFLUENCE OF MORE MODEL PARAMETERS
We think that the improvement the adversarial robustness has little relationship with the fact that the transition network introduces more model parameters. The transition network is only used to learn the transition matrix, and does not directly learn the logit output for the instance to predict the class label. The main reason why our method can improve the robustness is that we explicitly model adversarial noise in the form of the transition matrix. We can use this transition matrix to infer natural label. In addition, considering that the model parameters our method are indeed larger than those of AT, we conduct the following experiment to compare AT and our method in the case of similar number of model parameters.
Considering that both the target model and the transition network in our method are mainly based on ResNet-18 on CIFAR-IO, we use two parallel ResNet-18 as the target model to classify the input. We use the average of their logit outputs the final logit output. We use the training mechanism of AT to train the new target model. In this way, The number of model parameters of our method similar to that of AT. As shown in T the results show that method still has higher adversarial accuracy. This demonstrates that the improvement of the adversarial robustness is not due to the increase of model parameters.
Table 8: Adversarial accuracy (percentage) of defense methods against white-box adaptive attacks CIFAR-10. The defense models have a similar number model parameters.
Attack
None
PGD-40
AA
FWA-40
TI-DIM
CW2
DDN
AT
83.57
42.10
38.70
13.22
56.66
0.00
0.03
MAN
82.71
44.84
39.31
29.70
58.88
46.36
10.73
15


Reviews:
[{"review_id": "rG1O2uYDSmU", "reviewer": "Reviewer_9XAu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The defense strategy that embedding an instance-dependent transition matrix into the target model is novel.\n2. The experiments show that the performance of the proposed framework is better than normal adversarial training (AT).\n\nWeakness:\n1. The authors claim that the proposed framework can be viewed as adversarial training and the transition matrix can be embedded into the target model. Thus, the authors should conduct experiments by embedding the transition matrix into current adversarial training strategies, e.g., [a,b]. Such experiments are necessary to demonstrate the effectiveness of the proposed method, and should not be viewed as the future work.\n[a] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" International Conference on Machine Learning. PMLR, 2019.\n[b] Zheng, Yaowei, Richong Zhang, and Yongyi Mao. \"Regularizing neural networks via adversarial model perturbation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n2. Experiments in Table 1 and 2 should also be conducted with other network structures besides ResNet18.\n\n3. The trained model with MAN will lose the performance on the clean samples without perturbations. Especially, the distance is obvious for tiny-ImageNet as shown in Table5. This lead me to doubt the effectiveness of MAN.\n\n4. The size of transition matrix will increase a lot with the increase of class number. Thus, this method would cause greater cost for the dataset with larger class number, e.g., ImageNet.\n\n5. Why the training epoch is different for MAN and the baseline (150 vs 100)?\n", "summary_of_the_paper": "This paper proposes to learn a transition relationship between adversarial labels and natural labels for achieving adversarial defense. Especially, the authors introduced an instance-dependent transition matrix, which can be seamlessly embedded with the target model, to relate the adversarial labels and natural labels. Experiments on CIFAR and tiny-ImageNet demonstrate the proposed defense framework can outperform normal adversarial training.", "summary_of_the_review": "Although the proposed method is novel by embedding a transition matrix to the target model, there are several concerns: \n1) the experiments with more kinds of AT baselines, \n\n2) the decrease of performance on clean samples, \n\n3) the applicability of the proposed method for the dataset with large class number, \n\n4) the comparison fairness.\n\nThus, I think this paper is not enough for publication until these concerns are solved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LZxwpGiYx1K", "reviewer": "Reviewer_aEYo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1.\tOverall speaking the paper is well written and the method is clearly presented.\n2.\tThe proposed approach seems novel to me. The method is empirically validated by using several attack methods that not only attack the classifier but also the transition network as well.\n\nWeaknesses\n1.\tI am not fully convinced by the motivation. I would like to see supports on the claim that data pre-processing based methods suffer from high dimensionality problem and addressing the dimensionality problem in modeling adversarial noise is good for robustness. In addition, supposed that modeling perturbations in low-dimensional space is beneficial, it is still not clear that the proposed low-dimensional transition matrix actually models the adversarial noise. It would be helpful if the authors can address these concerns.\n2.\tIt seems like the proposed transition network works as a kind of post-processing. Instead of parameterizing $p(y|x)$ using a neural network $f(x)$, the proposed method parameterizes $p(y|x)$ using $g_{\\omega}(x, h_{\\theta}(x))$. Following this lens, why cannot we take $g_{\\omega}(x, h_{\\theta}(x))$ as a whole and expect a single function $f(x)$ to learn and perform as the same? Please kindly elaborate if I am mistaken.\n3.\tIn experiments, the comparison with SOTAs may not be fair. Does the method improve robustness merely because it uses a more complicated neural network architecture with more capacity (as it introduces an additional network $g_{\\omega}$ )? It would be great if the authors can align the number of network parameters used by each method in the comparison.\n\nOther questions: How does the proposed method perform combined with TRADES[1]? \n\n[1] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.\n", "summary_of_the_paper": "This paper proposes a defense method based on Modeling Adversarial Noise. The authors introduce an additional transition network to capture the transition matrix from adversarial predictions to vanilla predictions. The final output is the rectified prediction using this transition matrix. The proposed method is novel to me and is empirically validated by the authors under several adaptive attacks.\n", "summary_of_the_review": "Given the points listed, I give the current rating here. It would be helpful if the questions listed can be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HVEr7vmK1N", "reviewer": "Reviewer_Ee6e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. In the experiments part, the authors clearly evaluate the robustness of the models under three adaptive attack scenarios and the empirical results are technically convincing.\n\nWeaknesses: \n1. The idea of using the transition matrix seems to come from the widely used methodology in noisy label learning. The authors should discuss those literatures in the related work and compare the difference between the proposed method and those methods. Otherwise, the novelty of the paper can not be well justified.   \n2. In Section 3.2, the use of mathematical symbols is a bit confusing, and there are situations where symbols are not defined. For example, the sentence $Y'$  denotes the variable for the natural label, and $Y$ denotes the variable for the natural labels” is confusing. The symbols such as $\\widetilde{x_i}$, $\\widetilde{y_i}$ are not defined. The authors should well organize this part. \n3. For the robustness evaluation, In Table 1, the comparison with only AT is not enough. The transition matrix can be a general module combined with more adversarial training methods such as Trades, MART or ect. to conduct more experiments to benchmark the robustness improvement with different adversarial training methods. \n4. For  the experiments in Defense transferability, can the authors provide more results against adaptive attacks? Can the transferability exist under the stronger adaptive attacks?\n5. For the effectiveness of the transition matrix, I am concerned with the generalization of the learned transition matrix. From the results in Table 1, we can observe that the improvement of the adversarial accuracy is diverse under different attacks, the improvement under AA is just 0.3 and under TI-DIM is 1.76 which is smaller than 2.46 under PGD40 (the test attack is the same with training attack). \n\n\n", "summary_of_the_paper": "This paper proposes an adversarial defense method MAN by learning an instance-dependent transition matrix exploiting the transition relationship of adversarial labels and natural labels.  MAN generates the transition matrix by training a parameterized deep neural network which takes the mixture data (adversarial and natural data) as input and outputs the instance-dependent transition matrix. During inference, we can infer the natural labels with the adversarial labels and the transition matrix.  The authors evaluate MAN under three adaptive attacks scenarios and show the improved adversarial accuracy with the proposed method. ", "summary_of_the_review": "\nOverall, from my point of view, I think the novelty of the methodology and experiments evaluation of the paper in current form is not enough. I suggest rejection. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to model adversarial noise by learning the transition relationship between adversarial labels and natural labels. In particular, an instance-dependent transition matrix to relate adversarial labels and natural labels. Reviewers agreed that the paper is well motivated and well written, and the proposed method is novel. Meanwhile, reviewers raised some concerns about experiments and paper presentation. During discussion, the authors provided a lot of additional results that partially addressed the reviewers' concerns. However, the reviewers still think the experimental part of this paper should be further strengthened before acceptance. 

Thus, I recommend to reject this paper. I encourage the authors to take the review feedback into account and submit a future version to another venue.
Decision: Reject