Removing Undesirable Feature Contributions Using Out-of-Distribution Data
Authors: Saehyung Lee
Changhwa Park
Hyungyu Lee
Jihun Yi
Jonghyun Lee
Sungroh Yoon
Venue: ICLR 2021 Poster (2021)
Keywords: adversarial training
adversarial robustness
generalization
out-of-distribution

Abstract: Several data augmentation methods deploy unlabeled-in-distribution (UID) data to bridge the gap between the training and inference of neural networks. However, these methods have clear limitations in terms of availability of UID data and dependence of algorithms on pseudo-labels. Herein, we propose a data augmentation method to improve generalization in both adversarial and standard learning by using out-of-distribution (OOD) data that are devoid of the abovementioned issues. We show how to improve generalization theoretically using OOD data in each learning scenario and complement our theoretical analysis with experiments on CIFAR-10, CIFAR-100, and a subset of ImageNet. The results indicate that undesirable features are shared even among image data that seem to have little correlation from a human point of view. We also present the advantages of the proposed method through comparison with other data augmentation methods, which can be used in the absence of UID data. Furthermore, we demonstrate that the proposed method can further improve the existing state-of-the-art adversarial training.

Published as a conference paper at ICLR 2021
REMOVING UNDESIRABLE FEATURE CONTRIBUTIONS USING OUT-OF-DISTRIBUTION DATA
Saehyung Lee, Changhwa Park, Hyungyu Lee, Jihun Yi, Jonghyun Lee, Sungroh Yoon" Electrical and Computer Engineering, AlIS, ASRI, INMC, and Institute of Engineering Research Scoul National University Seoul 08826, South Korea [halo8218, onega 64 64, rucy 1,, sryoon/
ABSTRACT
Several data augmentation methods deploy to bridge the gap between the training and inference of neural networks. How- ever, these methods have clear limitations in terms of availability of UID data and dependence of algorithms pseudo-labels. Herein, we propose data augmen- tation method to improve generalization in both adversarial and standard learning by using -itt----iittiittiion (OOD) data that are devoid of the abovementioned issues. We show how to improve generalization theoretically using OOD data in each learning scenario and complement our theoretical analysis with experiments on CIFAR-10, CIFAR- 100, and a subset of ImageNet. The results indicate that un- desirable features are shared even among image data that seem to have little corre- lation from a human point of view. We also present the advantages of the proposed method through comparison with other data augmentation methods, which can be used in the absence of UID data. Furthermore, we demonstrate that the proposed method can further improve the existing state-of-the-ar adversarial training.
(UID) data
INTRODUCTION
The power of the enormous amount of data suggested by the empirical risk minimization (ERM) principle (Vapnik & Vapnik 1998) has allowed deep neural networks (DNNs) to perform outstand- ingly on many tasks, including computer vision (Krizhevsky et al., 2012) and natural language processing (Hinton et al.) 2012). However, most of the practical problems encountered by DNNs have -igh-dimensional input spaces, and nontrivial generalization errors arise owing to the curse of dimensionality (Bellman 1961). Moreover, neural networks have been found to be easily de- ceived by adversarial perturbations with high degree of confidence (Szegedy et al. 2013). Several studies (Goodfellow et al. 2014; Krizhevsky et al. 2012) have been conducted to address these generalization problems resulting from ERM. Most of them handled the generalization problems by extending the training distribution (Madry al. 2017 Lee et al. 2020). Nevertheless, it has been demonstrated that more data are needed to achieve better generalization (Schmidt et al. 2018). Re- cent methods (Carmon al.] 2019 Xie al. 2019) introduced un labeled-in-distrbbuttte to compensate for the lack of training samples. However, there are limitations associated with these methods. First, obtaining suitable UID data for selected classes is challenging. Second, when apply- ing supervised learning methods on pseudo-labeled data, the effect of data augmentation depends heavily on the accuracy of the pseudo-label generator.
(UID) data
In our study, in order to break through the limitations outlined above, we propose approach that promotes robust and standard generalization using out-of-distribution (OOD) data, Especially, moti- vated by previous studies demonstrating the existence of common adversarial space among different images or even datasets (Naseer et al., 2019 Poursaeed et al., 2018), we show that OOD data can be leveraged for adversarial learning. Likewise, if the OOD data share the same undesirable features as those the in-distribution data in terms of standard generalization, they can be leveraged for standard learning. By definition, in this work, the classes of the OOD data differ from those of the on-distribution data, and our method do not use the label information of the OOD data. Therefore the
"Correspondence Sungroh Yoon sryooo kr

---

Published as a conference paper at ICLR 2021
proposed method is free from the previously mentioned problems caused by UID data. We present theoretical model which demonstrates how to improve generalization using OOD data in both adver- sarial and standard learning. In our theoretical model, we separate desirable and undesirable features and show how the training on OOD data, which shares undesirable features with in-distribution data, changes the weight values of the classifier. Based the theoretical analysis, we introduce out-of- distribution data augmented training (OAT), which assigns a uniform distribution label to all the OOD data samples remove the influence of undesirable features in adversarial and standard learn- ing. In the proposed method, each batch is composed of training data and OOD data, and OOD data regularize the training so that only features that are strongly correlated with class labels are learned. We complement our theoretical findings with experiments on CIFAR-10 10 (Krizhevsky et al.) 2009). CIFAR- 100 (Krizhevsky et al. 2009). and a subset of ImageNet (Deng et al. 2009). In addition, we present the empirical evidence for the transferability of undesirable features through further studies on various datasets including Simpson Characters (Attial 2018). Fashion Product Images (Aggarwal
2018). SVHN (Netzer et al. 201T). Places365 (Zhou a.. 2017). and VisDA-17 (Peng al. 2017).
Contributions (i) We propose a simple method, out-of-distribution data augmented train- ing (OAT), to leverage OOD data for adversarial and standard learning, and theoretical analyses demonstrate how our proposed method can improve robust and standard generalization. (ii) The re- sults of experimental procedures on CIFAR-10. 10, CIFAR- 100, and subset of ImageNet suggest that OAT can help reduce the generalization gap in adversarial and standard learning. (iii) By applying OAT using various OOD datasets, it is shown that undesirable features are shared among diverse image datasets. It is also demonstrated that OAT can effectively extend training distribution by com- parison with other data augmentation methods that can be employed in the absence of UID data. (iv) The state-of-the-art adversarial training method using UID data is found to further improve by incorporating the proposed method of leveraging OOD data.
2 BACKGROUND
Undesirable features in adversarial learning Tsipras et al. (2018) demonstrated the existence of a trade-off between standard accuracy and adversarial robustness with the distinction between robust and non-robust features. They showed the possibility that adversarial robustness is incompatible with standard accuracy by constructing a binary classification task that the data model consists of input- label pairs (x. y) E IRd+1 x {+1] sampled from a distribution as follows:
a.0.T +1,, 21 (- w.p. y
+y w.p.
.2..1 a.d. 1).
(I)
Here, is a robust feature that is strongly correlated with the label, and the other features F2. .... Fd++ are non-robust features that are weakly correlated with the label. is small but suf- ficiently large such that a simple classifier attains a high standard accuracy, and p 2 0.5. To charac- terize adversarial robustness, the definitions of expected standard loss 33 and expected adversarial loss B, for a data distribution D are defined as follows:
B.
E [C(x, yı 0)], Ba = E max L(x + 8, y: BE9
(w.9)~-
(2.v)~D
Here, L(: is the loss function of the model, and S represents the set of perturbations that the adversary can apply to deceive the model. For Equation (I. Tsipras et al. (2018) showed that the following classifier can yield a small expected standard loss:
0 2
favg(æ) sign(w 2T) where Wanit
(3)
They also proved that the classifier is vulnerable to adversarial perturbations, and that adversarial training results in classifier that assigns zero weight values to non-robust features.
Transferability of Adversarial Perturbations Naseer et al. 2019) produced domain aagaosicc ad- versarial perturbations, thereby showing common adversarial space among different datasets. They showed that an adversarial function trained on Paintings, Cartoons or Medical data can deceive the classifier on ImageNet data with a high success rate. The study findings show that even datasets from considerably different domains share non-robust features. Therefore a method for supplementing the data needed for adversarial training is presented herein.
2

---

Published as a conference paper at ICLR 2021
Undesirable features in standard learning Wann et al. (2020) noted that convolutional neural networks (CNN) can capture nigh-frequency components in images that are almost imperceptible to human. This ability is thought to be closely related to the generalization behaviors of CNNs, especially the capacity in memorizing random labels. Several studies (Geirhhos et al. 2018 Bahng eet al., 2019) reported that CNNs are biased towards local image features, and the generalization performance can be improved by regularizing that bias. In this context, method of regularizing undesirable feature contributions using OOD data is proposed, assuming that undesirable features arise from the bias of CNNs or insufficient training data and are widely distributed in the input space.
METHODS
3.1
THEORETICAL MOTIVATION
In this section, we analyze theoretically how OOD data can be used to make up for the insuffi- cient training samples in adversarial training based on the dichotomy between robust and non-robust features. The theoretical motivation of using OOD data to reduce the contribution of undesirable features in standard learning can be found in Appendix B
Setup and overview We denote in-distribution data as target data. Given a target dataset (('',")) >))) C X x {=1} sampled from a data distribution D, where X' is the input space, we suppose that a feature extractor & : X' + Z C IRd and a linear classification model are trained on ((6', , ))) and target feature-label pairs {(((#"), y") respectively, to yield the small ex- pected standard loss of the classification model. We then define an OOD dataset {# > C X' sampled from a data distribution Ó that has the same distribution of non-robust features as that of δ with reference to the preceding studies (Naseer et al. 2019 Moosavi-Dezforo et al. 2017). After fixing & to facilitate the theoretical analysis on this framework, we demonstrate how adversarial training on the OOD dataset affects the weight values of our classifier.
Our data model The feature extractor & can be considered consist of several feature ex- tractors @ X' + Z
C R. Hence, we can set the distributions of the target feature-label pair (f()). v) = (ž.8) E Rd {+1} and the OOD feature vector (()) = E R as follows:
" {-1,
žσ N(y, u²),
82 , 1.d. (ny, 1),
(4)
(-1,+1}, 2 ~ NO 2 2141 iid. N 1),
where u.a.r stands for uniformly at random. From here on, we will only deal with the OOD data, therefore the accents (tilde and caret) that distinguish between target data and OOD data are omitted. In Equation 44. the feature 21 Φ1 (a)) is the output of a robust feature extractor Φ1, and the other features 22,.... 2d+1 are those of non-robust feature extractors Φ2,-... dd+1- Since the OOD input vectors do not have the same robust features as the target input vectors, 21 has zero mean and a small variance. Furthermore, because the OOD data have the same distribution of non-robust features as the target data, ø2. 2d+1 have a non-zero mean and a larger variance than the robust features. In addition, q represents the unknown label associated with the non-robust features, and is a non- negative constant which represents the degree of correlation between the non-robust features and the unknown label. Please note that the input space of our classifier is the output space of & in our data model. Therefore, 7) is not limited to a small value even in the context of the bp-unudded adversary. Rather, the high degree of confidence that DNNs show for the adversarial examples (Goodfellow ct al. 2014) suggests that 7) is large.
Our linear classification model According to Section 2. we know that our linear classification model (logistic regression), defined as follows, yields a low expected standard loss while demon- strating high adversarial vulnerability.
1 d
1
pvv = +1 =) = aw 2), p(y 1 z) = aww z), where 0.
(5)
d
To observe the effect of adversarial training on the OOD dataset, we train our classifier by applying the stochastic gradient descent algorithm to the cross-entropy loss function L(; un).
3

---

Published as a conference paper at ICLR 2021
Firstly, we construct the adversarial feature vector 2 4bx 6 E S against classifier for adversarial training.
Theorem 1. Let t E [0 be the given target value of the feature vector z in our classification model, and X be a non-negative constant. Then, when 1 = 0.5, the expectation of the adversarial feature vector is
Ex 1 [21),
Ex [xx] 2 Eg [2k] + x..9. where k E f2.....d. d 13.
(6)
(All the proofs of the theorems in this paper can be found Appendix A Here, we assume the -o -ouudddd adversary. In Theorem [ we can observe that the adversary pushes the non-robust features farther in the direction of the unknown label q. which coincides with our intuition. When the given target value is 0.5, the adversary will make our classification model output equal to zero or one to yield a large loss.
Our classification model is trained on the adversarial features shown in Theorem Theorem 2. When = 0.5, the expected gradient of the loss function [(2, t; w) with respect to the weight vector w of our classification model is
ac
dc
Eg
Eg
x where & E {2..... .d...,,++1., 1
Ow
Owk_
Thus, the adversarial training with & 0.5 on the OOD dataset leads the weight values cor- responding to the non-robust features converging to zero while preserving wi from the gradient update. This shows that we can reduce the impact of non-robust features using the OOD dataset. We, however, should not only reduce the influence of the non-robust features, but also improve the classification accuracy using the robust feature. This can be achieved through the adversarial train- ing on the target dataset. Accordingly, dataset when W1 > 0 in our example.
show the effect of the adversarial training on the OOD
Theorem 3. When t 0.5 and u1 0, the expected gradient of the loss function L(z, t; w) with respect to the weight vector u of our classification model is
OL
Ez
2 X,
Ez
2 (n
where k E (2,..., 13.
(8)
Ow
Owk
Theorem 3 shows that when wi 0, the adversarial training with t = 0.5 on the OOD dataset reduces the influence of all the features in Z. However, we can see that the expected gradients for the weight values associated with the non-robust features are always greater than the expected gradient for the weight value associated with the robust feature. In addition, the greater the value of 7, the faster the weight value associated with it converges to zero. This means that the contribution of non-robust features with high influence decreases rapidly. In the case of multiclass classification, it is straightforward that & = 0.5 corresponds uniform distribution label funit ........ d where e is the number of the classes. Intuitively, the meaning of (z, bunit) is that the input lies on the decision boundary. To reduce the training loss for (I, lunit), the classifier will learn that the features of x do not contribute to specific class, which can be understood as removing the contributions of the features of z.
3.2 OUT-OR-DISTRIBUTION DATA AUGMENTED TRAINING
Based on our theoretical analysis, we introduce Out-of-diste distrribbiiiino data Augmented Training (OAT). OAT the training on union of the target dataset D, and the OOD dataset D.. When applying our proposed method, we need to consider the following two points: 1) Temporary labels associated with OOD data samples are required for supervised learning, and 2) the loss functions corresponding to D, and D, should be properly combined.
First, we assign a uniform distribution label tunif to all the OOD data samples as confirmed in our theoretical analysis. This labeling method enables to leverage OOD data for supervised learning at no extra cost. Moreover, it means that our method is completely free from the limitations of the methods using UID data (see Section
Second, although OOD data can be used to improve the standard and robust generalization neu- ral networks, the training target data is essential enhance the classification accuracy of neural
4

---

Published as a conference paper at ICLR 2021
networks. In addition, according Theorem adversarial training on the pairs of OOD data sam- ples and funir affects the weight for robust features as well as that for non-robust features. Hence, the balance between losses from D, and D. is important in OAT. For this reason, we introduce hyperparameter a E R into our proposed method and train neural networks as follows:
OAT-A:
min (m))))
max C(x: + 8: 8ES
E m.
max L(x. + e, tumit 00)
LEES
(9)
OAT-S:
min (D
[ y: 0) +
E E.DD
[C(x uumir: 0)]
Here, OAT-A and OAT-S represent OOD data augmented adversarial and standard learning, respec- tively, The pseudo-code for the overall procedure of our method is presented in Algorithm T
Algorithm
augmented Training (OAT)
Require: Target dataset Di, OOD dataset Do, uniform distribution label funif. batch size n, training iterations T, learning rate т, hyperparameter a, adversarial attack function
1: for = 1 to I do
2: (X,, Y) =
= Dr, size = Do, size 2)
3: X, SAMPPE
4: if Adversarial Learning then
5:
[x,XX] + [[[Xe, x X.] [Y, (unis] 0)
6:
else if Standard Learning then
7:
Xx,XX, + Xo]
8:
end if
9:
model update:
10:
0 -- - Y lunit: 0))
11:
end for
12: Output: trained model parameter 0
4 RELATED STUDIES
Adversarial examples Many adversarial attack methods have been proposed, including the pro- jected gradient descent (PGD) and Carlini Wagner (CW) attacks (Madry et al.] [2077; Car- lini & Wagner, 2017). The PGD attack employs iterative procedure of the fast gradient sign method (FGSM) (Goodfellow et al.) (2044) to find worst-case examples in which the training loss is maximized. CW attack finds adversarial examples using CW losses instead of cross-entropy losses. Recently, Croce& & Hein (020) proposed autoattack (AA), which is a powerful ensemble attack with two extensions of the PGD attack and two existing attacks (Croce Hein 2019 Andriushchenko et al. 2019). To defend against these adversarial attacks, various adversarial defense methods have been developed (Goodiellow et al., 2014] Kannan et al. 2018). Madry et (2017) introduced ad- versarial training that uses adversanal examples as training data, and Zhang et al. (2019) proposed TRADES to optimize a surrogate loss which sum of the natural eтror and boundary error.
OOD detection Lee et al. (2017) and Hendryeks et al. (018) dealt with the overconfidence prob- lem of the confidence score- based OOD detectors. They used uniform distribution labels as in our method to resolve the overconfidence issue. They did not, however, address the generalization prob- lems of neural networks. Specifically, our theoretical results allow us to explain the classification performance improvement of classifiers that were only considered as secondary effects in the above- mentioned studies. Further related works can be found Appendix C
5 EXPERIMENTAL RESULTS AND DISCUSSION
EXPERIMENTAL SETUP
OOD datasets We created OOD datasets from the 80 Million Tiny Images dataset Torralba et al. 2008) (80M-TI), using the work Carmon et al. (2019) for CIFAR-10 and CIFAR-100, respectively. In addition, resized (using bilinear interpolation) ImageNet to dimensions of 64 x 64 and
5

---

Published as a conference paper at ICLR 2021
Table 1: Accuracy (%) comparison of the OAT model with Standard. PGD, and TRADES on CI- FAR10, CIFARI 100, and ImgNet10 (64x x64 under different threat models. We show the improved results compared the counterpart of each model in bold.
Model
Target
OOD
Clean
PGD100
CW100
AA
Standard
95.48
0.00
0.00
0.00
PGD
87,48
49.92
50.80
48.29
PGD+CutMix TRADES
CIFAR1O
89.35
53.39
52.35
49.05
85.24
55.69
54.04
52.83
OAT
80M-TI
86.63
56.77
52.38
49.98
OATTRADES
80M-TI
86.76
59.66
55.71
54.63
Standard
78.57
0.02
0.00
0.00
PGD
61.37
24.66
24.68
22.76
TRADES
CIFARIO0
58.84
30.24
27.97
26.91
OATPGp
80M-TI
61.54
30.02
27.85
25,36
TRADES
80M-TI
63.07
34.23
29.02
27.83
Standard
86.03
0.11
0.06
0.00
1mgNet10 (64 x 64)
PGD
82.80
48.77
48.86
48.34
OATPGD
ImgNe1990
81.91
59.03
54.69
53.83
160 160 and divided it into datasets containing 10 and 990 classes, respectively; these called ImgNet10 and ImgNe1990. Furthermore, we resized Places365 and VisDA-17 for the experiments on Im and cropped the Simpson Characters (Simpson), and Fashion Product (Fashion) datasets to dimensions of 32 x 32 for the experiments on CIFARIO and CIFARIO0 The details in sourcing the OOD datasets can be found in Appendix D
Implementation details Implementation details including the hyperparameter a, architectures, batch sizes, and training iterations are summarized in Appendix All models compared in the same target dataset are trained in the same training batch size and iterations to ensure fair comparison. In other words, OAT have a batch size of target OOD which would be compared with a model that is normally trained with batch size of n. To evaluate the adversarial robustness of the models in our experiments, we apply several adversarial attacks including PGD, CW, and AA. Note that we denote PGD and CW attacks with 7' iterative steps PGD7' and CWT, respectively, and the original test set as Clean. We compare the following models in our experiments
1. Standard: The model which is normally trained the target dataset.
PGD: The model trained using PGD-based adversarial training on the target dataset.
3. TRADES: The model trained using TRADES the target dataset.
PAT The model which is adversarially trained with OAT based on a PGD approach.
TATTRADES: The model which adversarially trained with OAT based on TRADES.
6. OAT The model which is normally trained with OAT using the OOD dataset D..
5.2 STUDY ON THE EFFECTIVENESS OF OAT IN ADVERSARIAL LEARNING
Table 2: Comparison of OAT using various OOD datasets for improving robust generalization on CIFAR10 and 1mgNet10 (64 x 64). The None represents the baseline model (PGD).
CIFAR10
ImgNet10 (64 x 64)
OOD
None
SVHN
Simpson
Fashion
None
Places365
VisDA17
Clean
87.48
86.16
86.79
85.84
82.80
82.37
82.46
PGD20
50.41
53.70
53.88
53.27
49,00
59.86
55,34
CW20
51.11
52.21
52.15
51.70
48.91
56.23
53,80
Our codes available

---

Published as a conference paper at ICLR 2021
Evaluating adversarial robustness The improvements in the robust generalization performances of the PGD and TRADES models through the application of OAT are evaluated against PGD 100, CW100. and AA with -bound of 253 and 0.031. The results are summarized in Table and indi- cate that OAT improves the robust generalization of all adversarial training methods tested regardless of the target dataset. In particular, from the results against AA it be seen that the effectiveness of OAT does not rely on obfuscated gradients (Athalye et al. 2018). This is because AA removes the possibility of gradient masking through the application of a combination of strong adaptive tacks (Croce & Hein 2019, 2020) and a black-box attack (Andriushchenko et al. 2019).
However, while OAT brings significant improvement robustness against PGD attacks, it is rel- atively less effective against CW attacks. According to our theoretical analysis, these results imply that the OOD data have relatively few non-robust features used in the CW attacks. In other words, powerful targeted attacks, such as CW attacks, are constructed using gradient information more se- lectively than untargeted attacks, such as PGD attacks. To gain insight into this phenomenon, we train models with various hyperparameter values. The results suggest that the greater the influence of OOD on the training process is, the higher the robustness against PGD attacks and the lower the robustness against CW attacks are (see Appendix for more details).
In the absence UID data, various data augmentation methods other than OAT can be employed. CutMix (Yun et al. 2019), method of amplifying training data by cutting and pasting patches between training images, was recently proposed and exhibited excellent performance in classifica- tion and transfer learning tasks. By applying CutMix adversarial training, the advantages of OAT over other data augmentation methods are assessed. Specifically, OAT brings a higher level of ro- bustness than CutMix, as deduced from Table [ Because adversarial examples are closely related to the high-frequency components of images (Wang et al.) 2020) and adversarial training reduces the model's s sensitivity to these components, CutMix is an ineffective data augmentation method. Although CutMix extends the global feature distribution, there is no significant difference in terms of local feature distribution. Contrarily, OAT can effectively regularize the classifier by observing various undesirable features held by additional data in the learning process.
OAT with diverse OOD datasets Non-robust features are widely shared among different datasets, as demonstrated by applying OAT using various OOD datasets. Table[]indien iindicates that OAT improves robust generalization for all OOD datasets, including those that have little correlation with the tar- get dataset from a human perspective. In addition, given that relatively simple datasets with no background (Fashion and VisDA-17) are less effective than others, we can suppose that non-robust features arise from the high complexity (Snodgrass & Vanderwart] 1980) of images for natural image datasets, such as CIFAR and ImageNet.
When UID data are available In adversarial training, in in-distribution data reduce the sensitivity of neural networks to non-robust features and provide robustly generalizable features. On the other hand, the proposed theory shows that OAT takes advantage of the data amplification effect only for non-robust features using OOD data. Therefore, the effectiveness of OAT is expected to decrease as more in-distribution data are augmented. To observe such trend empirically. the effect of OAT is investigated according to additional in-distribution data size using previously published pseudo-
labeled data (Carmon et al. 2019).
64
60.84
Accuracy (%)
70
CAT + RST
60
RST
Accuracy
53
56.19
58.1
$7.62
65.
65.46
63.89
66.89
64.10
62.43
52-
49.92
label
60.2460.00
Pseude label
48
48.02
PSD
60
5K 30K
125K
500K
PGralrr
PG
PGtT
AA
Pseudo-labeled data size (a)
(b)
Figure 1: (a) The effectiveness of OAT with increasing pscudo-labeled data size under PGD100 and (b) the results of the state-of-the-ar model improved by OAT under PGTRADES. PG PGRST (Car- mon et al. 2019, and AA CIFAR-10.

---

Published as a conference paper at ICLR 2021
Surprisingly, Figure a shows that OAT still improves robust generalization pseudo-labeled data are used. OAT is also combined with RST (Carmon et al.) 2019). which has re- cently recorded state-of-the-an adversarial robustness using UID data; in fact, Figure Tbb demon- strates that OAT can further improve the state-of-the-ar adversarial training method. This is pre- sumably because the additional data include noisy labeled data, in which induce memorization in the learning process and thus impair the generalization performance (Zhang et al. 2016). OAT seems achieve higher level of robust generalization by effectively suppressing the effects of noisy data. Additional details on the experiments illustrated in Figure I are described in Appendix H
when many
5.3 STUDY ON THE EFFECTIVENESS OF OAT IN STANDARD LEARNING
Randomization test The effect of OAT is analyzed based on the randomization test (Zhagg et al. 2016). The randomization test is experiment aiming observe the effective capacity of neural networks and the effect of regularization by training the model on copy of the data where the true labels are replaced by random labels. In Figure aa), it can be seen that the Standard model
Training error
1.0
0.15
OAT
Test error
Standard
0.10
0.5
0.05
QAT
Standerd
0
40000
80000
0.00
40000 Training steps (b)
80000
Training steps (a)
Figure 2: (a) The results of the randomization test and (b) the test error curves for the Standard and OAT models along their optimization trajectories on CIFAR-10.
memorizes all training samples to obtain a low training error, whereas the OAT model continues to have high training error. These results show that OAT effectively regularizes neural networks so that learns only features with strong correlation with the class labels. Figure 2 indicates that the OAT model learns more slowly than the Standard model owing to the influence of strong regularizer the early stages of training; however, achieves a better generalization performance at the end of training.
Evaluating classification performance The effect of OAT on the classification accuracy is shown in T When the number of training samples (N) is small, the influence of undesirable features expected to be large. Therefore, the experiments classified depending on the number of train- ing samples (N). Table 3 indicates that the effect of OAT is large when the amount of training data
Table 3: Accuracy (%, mean over 5 runs) comparison of the OAT model using various datasets with the baseline models. We show the improved results compared the counterpart of each model in bold. Pseudo-label is the model trained using pseudo-labeled data (from 80M-TI) in conjunction with the target dataset. Fusion the model that applies OATsoM-TI to the Pseudo-label model. A detailed description can be found in Appendix D
Dataset N
CIFAR10
CIFAR100 2,500 / Full
ImgNet10 (64 x 64) 100 / Full
ImgNet10 (160 x 160)
2,500/ Full
Dataset
Standard
65.44/94.46
24.41 / 74.87
100 / Full
SAHN
08.56/94.44
24.82 / 75.65
Standard
37.90 86.93
33.36 / 90.91
OAT Simpson
70.08 94.43
27.04 / 76.03
OAT ViTiDA117
36.21
86.71
35.93 91.23
OAT 80T-IM-11 SUM-11
72.49 /95.20
26.13 / 76.30
OAT OATpiiess36 P1aces365
41.84 / 88.37
40.11 / 91.42
Pseudo-label
/95.28
777.24
OAT ImgNer990
42.18 / 87.88
40.41 91.87
Fusion
/95.53
777.36
small, predicted. Additionally, OAT enhances the generalization performance using 80M-TI, ImgNet990, and Places365, which have input distributions similar to the target datasets, more than when using other OOD datasets. This proves empirically that in theoretical analysis, the OOD
8

---

Published as a conference paper at ICLR 2021
data, following the same undesirable feature distribution the target data, improve generaliza- tion through OAT. In addition. the results of Pseudo-label and Fusion models show that even when pseudo-labelee data are available, OOD data can be leveraged to further improve the standard gener- alization performance. Moreover, the proposed method, which does not require complex operations and is very simple to implement, can lead higher performance by combining with existing data augmentation methods, As an example, the effectiveness of Mixup (Zhang et al. (017)) is enhanced by applying OAT; the experimental results are provided in Appendix
Finally, OAT in a standard learning scheme using the entire target dataset is generally less effective than OAT in an adversarial training scheme (see Appendix For more details). Therefore, it can be inferred that the transferability of undesirable features is greater in adversarial settings than in stan- dard settings. In other words, these results experimentally show that the number of training samples required for robust generalization is large compared to that required for standard generalization.
CONCLUSIONS AND FUTURE DIRECTIONS
In this study, a method proposed to compensate for the insufficient training data by using OOD data, which are less restrictive than UID data. It is theoretically demonstrated that training with OOD data can remove undesirable feature contributions a simple Gaussian model. Experiments performed various OOD datasets, which surprisingly demonstrate that even OOD datasets that apparently have little correlation with the target dataset from the human perspective can help standard and robust generalization through the proposed method. These results imply that com- mon undesirable feature space exists among diverse datasets. In addition, the effectiveness of the proposed method is evaluated when extra UID data are available, and the results indicate that OAT can improve the generalization performance even when substantial pseudo-labeled data are used.
Nevertheless, some limitations need to be acknowledged. First, it is challenging predict the effec- tiveness of the proposed method before applying it to specific target- OOD dataset pair. Second, our method is less effective against strong targeted adversarial attacks, such as CW attacks, as is diffi- cult to generate deliberate adversarial attacks on in-distribution in the process of OAT. Therefore, as a future research direction, we aim to quantify the degree to which undesirable features are shared between the target and OOD datasets and construct strong adversarial attacks using OOD data.
Acknowledgements: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (Ministry of Science and ICT) [2018RIA2]] 3223001622)) the BK21 FOUR program of the Education and Research Program for Fu- ture ICT Pioneers, Seoul National University in 2020, and AIR Lab (AI Research Lab) in Hyundai & Kia Motor Company through HKMC-SNU AI Consortium Fund.
REFERENCES
Param Aggarwal, Fashion product images (small), 2018. data retrieved from Kaggle, https: www. kaggle.
wrophiitto-roouctt---- -images-------- small
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. arXiv preprint arXiv:1912.00049, 2019,
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of se- curity: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.004200 2018.
Alexandre Attia. The simpsons characters data, 2018. data retrieved from Kaggle, https://www.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning biabe representations with biased representations. arXiv preprint arXiv: 1910:209, 2019.
Robert Bellman. Curse of dimensionality. Adaptive control processes: a guided tour. Princeton, NJ, 3:2, 1961.

---

Published as a conference paper at ICLR 2021
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Sci- ence, 59(2):341-357, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEBE, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves udversarial robustness, In Advances in Neural Information Processing Systems, pp. 11190-1 1201, 2019.
Alvin Chan, Yi Tay, and Yew-Soon Ong. What thinks important is important: Robustness transfers through input gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. arXiv preprint arXiv:1907.0200 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fiee free attacks. arXiv preprint urXiv:2003.01690 2020.
Zihang Dai, Zhilin Yang, Fan Yang. William W Cohen, and Russ Salakhutdinov. Good semi- supervised learning that requires a bad gan. In Advances in neural information processing systems, pp. 6510-6520, 2017.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint 1.Xiv:8811.122331 2018.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual
networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 10l...82204000 2018.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre- pre-training can improve model robustness and uncertainty. arXiv preprini MXXi::9901.0900 2019.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.
Jeremy Howard. Imagenette. /RL/ttts:://iitiuubc http:///twbb.coc///aataa/aaaaags
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint
arXiv:1803.06377 2018
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrace classifiers for detecting out-of-distributiin samples, arXiv preprint arXiv: 171 1.09255, 2017.
10

---

Published as a conference paper at ICLR 2021
Sachyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adver- sarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: (:XXvx/706.06085 2017.
Seyed-Mohsen
Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 765-1773, 2017.
Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. In Advances in Neural Information Processing Systems, pp. 5542-5552, 2019.
Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. In Advances in Neu- ral Information Processing Systems, pp. 12885-12895, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge, 2017.
Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial pertur- bations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4422-4431, 2018.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. Advances in Neural Information Processing Systems, pp. 501455044500 5026, 2018.
Joan G Snodgrass and Mary Vanderwart. A standardized set of 260 pictures: norms for name agree- ment, image agreement, familiarity, and visual complexity. Journal of experimental psychology: Human learning and memory, 6(2):774, 1980.
Robert Stanforth, Alhussein Fawzi, Pushmeet Kohli, et al. Are labels required for improving adver- sarial robustness? arXiv preprint 1rXiv:1905.133: 13725, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13X2vv1312.66 1312.6199, 2013.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 10:11)1:958-9700 2008
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. arXiv preprint arXiv: 12152, 2018
Vladimir Vapnik and Vlamimir Vapnik. Statistical learning theory wiley. New York, 1, 1998.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. Proceedings of the IEFF//CV Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoe V Le. Self-triining with noisy student improves imagenet classification. arXiv preprint wrXivv1911.04255 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the IEEE International Conference Computer Vision, pp. 6023-6032. 2019.
11

---

Published as a conference paper at ICLR 2021
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide residual
networks.
arXiv preprint
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: /6/ 1.03530, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7472-7482, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL lttp:://sroceeennggg mll.. press/ v97 v97/zhangn htt//hhagg99pp ntm.
https: http:///ttaa...cccaa/coo/ooooae/
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint urxiv: 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
A PROOFS
Theorem 1. Let t E [0 be the given target value of the feature vector z our classification model, and À be a non-negative constant. Then, when t = 0.5, the expectation of the adversarial feature vector is
Eg [=]= E. [2],
Eg [xx]
Ez [ax] + À where k E (2...., 1}.
(10)
Proof. Let de be the adversarial perturbation in the input space. Then,
_ arg max L( + 8). w) arg max ((((x) +8 - D, w)
(11)
= arg max C(2 + 5 V v t; w) = arg max l(z + 62, w), where ó= = =5vaaz
Equation 11 suggests that constructing the adversarial perturbation 5n in the input space can be ap- proximated by finding the adversarial perturbation 8g in the feature space. These have a relationship of 8 ò = by linear approximation. Hence, without loss of generality, the perturbed feature vector z can be approximated by sign (V V C(z, t; w)) with an 0. Then,
+ À sign(w) (a(w" T 2) ]
Eg [21] Ez [2] + X sign( V C(z, t; w)) = Eg
(12)
Ex |21| + XE, (signssiin 0)] Ez [21].
Because classification model trained minimize the expected standard loss, Eg siign wn (aww" 2) } can be approximated by q. Then,
w))) = Ex 24 sign(wx(o(w)) }
E2 [Ex] = Ez [Ek + A sign DV C(2, 1;
(13)
Eg
[a A
Theorem 2. When = 0.5, the expected gradient of the loss function L(z, t; w) with respect to the weight vector un of our classification model
ac
ac
Eg
z 0,
Eg
z
(1
x),
where & E (2,..., 2....d+11. 1}.
(14)
Ow
Owi
12

---

Published as a conference paper at ICLR 2021
Proof. Based on the adversarial vulnerability of our classifier, a(w 2) can be approximated by (1 + q). Therefore,
Oc
1
Ez
Ez
22)
-
2
Eg
[1 ooooow (uuT2) 2)] 3 Ez [21]
Own
Es
[11] (1 + q)
1
Eg
[21]
E [Z1] = 0,
2
2
(15)
0C
Ez
Eg
Zk(o(w" T 2)
2
=
E2
"(w" T 2)] E2 [Zx]
Omnk
Ex
[Ex]
+ q)
[xx]
2
Eg
[3
2 A).
2
Theorem 3. When t 0.5 and W1 O, the expected gradient of the loss function L(z, t; w) with respect to the weight vector an of our classification model is
OC
aC
Es
2 X
Eg
=
2( A),
where E (2...., .d. 13.
(16)
Ow
Owk
2
Proof.
21 + X sign(wi 2) 21
Eg
[21]
Ez
[2] + X- sign( v., L(z, 1; w))] E
EE [21] + q,
Eg
Oc
Eg
ト EE
| -.w Tz)]
1 B [21]
(17)
Ow
! Eg [21] = A.
~
Eg [21] (1 - -2 Eg [2]
2
2
B THE THEORETICAL MOTIVATION OF OAT IN STANDARD LEARNING
Based on the same setup described in Section 3 of the main manuscript, we construct the data model for OAT in the following manner:
~ {-1,+1},
A((, 22 ~ of), where E E of <<< 0z
(18)
For simplicity, we consider only one desirable feature extractor Φ1 and one undesirable feature extractor Φ2. In Equation (18). features 21 (xx) and 22 represent the output of feature extractors Φ1 and d2, respectively. As the OOD input vectors do not have the same desirable features as the target input vectors, the mean of 21 is zero. On the other hand, because the OOD data have the same distribution of undesirable feature as the target data, the mean of 22 is non-zero. In addition, considering that the feature extractors are trained respond sensitively the target input vectors, the output variance of the undesirable feature extractor is much larger than that of the desirable feature extractor for OOD data, especially in the high-dimensional case. q represents the unknown label associated with the undesirable feature, and K is a non neganeietie constant that represents the degree of correlation between the undesirable feature and the unknown label.
We can prove the following theorems for a logistic regression model that is not strongly dependent on the desirable feature.
Theorem 4. The OOD data barely affect the gradient update of the weight associated with the desirable feature.
Proof. Because we assumed linear classifier that is not strongly dependent on the desirable feature owing the bias of CNNs or insufficient training data, w wi 21 + W222 be approximated
13

---

Published as a conference paper at ICLR 2021
by W222- Then, for the target value t, specified the feature as shown in the following equation:
JC
Eg
= E= [[o 2) 1)=1]
Own
(19)
Eg ((0()2))))
t)21] Ex (o(2222) t Eg [21] = 0.
Note that in Equation 19 the gradient of the loss function with respect to the weight value Wi is zero regardless of t. The optimal is then a value that makes W2 converge to zero, thereby removing the influence of the undesirable feature.
Theorem 5. When 0.5, the standard learning on the OOD data leads to the weight value corresponding to the undesirable feature converging to O.
Proof.
0C
1 ) = Ez (0(1222)) )2 = E2 (a Tu)
E
2)
(20)
iw:z
When w2 > 0, (2w222) > j/22 with high probability, Hence,
1 ))²² (0(w22))
E
> 0, where W2 > 0.
(21)
Similarly,
1 ] Eg (a(w222)
< 0,
where w2 < 0.
(22)
C FURTHER RELATED WORKS
Using Unlabeled Data to Improve Adversarial Robustness Stanforth et al. (2019) and Carmon et al. (2019) analyzed the requirement of larger sample complexity for adversarially robust general- ization (Schmidt et al. 2018). Based on model described in prior work (Schmidt et al.) 2018).
they theoretically proved that unlabeled data can alleviate the need for the large sample complexity of robust generalization. Therefore, they proposed semi-supervised learning technique by aug- menting the training dataset with extra unlabeled data. They used pseudo-labels obtained from model trained with the existing training dataset. They experimented on the CIFAR-10 dataset by us- ing the 80 Million Tiny Images dataset (Torralba et al. (2088) (80M-TI) as an extra training dataset, resulting improvements of adversarial robustness of the model. Najafi et 2019) also used unla- beled data for adversarial robustness by extending the distributionally robust learning (Ben- Tal et al.) 2013) to seni-sueer learning scenarios. Instead of using pseudo-labels, they used soft-labels, which are chosen from set of labels and softened according the loss values of data. Their results include experiments on the MNIST, CIFAR-10, 10, and SVHN datasets.
Comparison with Bad GAN Dai 2017) theoretically showed that a perfect generator cannot able enhance the generalization performance, and good semi-supervised learning actually requires a bad generator. Through theoretical analysis, they proposed an empirical formulation to generate samples with low-density in the input space. Compared our method, there are two main differ- ences between Bad GAN and OAT. First, Dai et al. (2017) only considered semi-supervised learning settings, whereas OAT can also be applied to adversarial settings. Second, data augmentation using GAN has clear limitations. Dai et al. (017)) penalized high-density samples generate low-density samples in the input space, but this method is inefficient a high- digh-dimensional imput space. On the other hand, OAT uses various and efficient OOD directly to regularize the model, and the research direction that develops from the use of synthetic data to the use of real data can also found for OOD detection (Hendrycks et al. 2018).
14

---

Published as a conference paper at ICLR 2021
Table Implementation details for the experiments of OAT-A
Target
architecture
training steps
batch size
CIFAR WRN-34-10 Zagoruyko & Komodakis 2016
1.0
80K
128
ImgNet10
ResNet18 He et al.
2016
1.0
15,4K
128
Table 5: Implementation details for the experiments of OAT-S
Target
N
architecture
Cx
training steps
batch size
2500
4000
128
CIFAR
ResNer18
1.0
50K
78K
128
1mgNet10
100
1.0
200
100
WRN-22-10
(64 x 64)
9894
0.2
15.4K
128
ImgNet10
100
1.0
400
100
ResNet18
(160 160)
9894
0.1
38.5K
128
D SOURCING OOD DATASETS
We created OOD datasets from 80M-TI, using the work of Carmon et al.) (2019) for CIFARIO and CIFAR 100, respectively. In other words, trained a 11-way (I more class for OOD) classifier on a training set consisting of the CIFARIO dataset and 1M randomly sampled images from 80M-TI with keywords that did not appear in CIFARIO. We then applied the classifier on 80M-TI and sorted the images based on confidence in the OOD class. The 1M and 5M images selected in the order of the highest confidence were used for OAT-A and OAT-S, respectively. In Table Fusion has batch size of target pseudo-labeledd OOD" 4. which would be compared with the Pseudo-label model that is trained with batch size target it + pseudo-labeled In addition, we resized (using bilinear interpolation) ImageNet to dimensions of 64 64 and 160 x 160 and divided it into datasets containing 10 and 990 classes, respectively; these are called ImgNet10 (train set size 9894 and test set size 3500) and ImgNet990. respectively. The classes were divided based on the Imagenette dataset (Howard), and the experimental results for the differently divided dataset (Imagewoof) can be in Appendix F We increased the number of images of ImgNet990 by 10 times through random cropping and created the OOD datasets in the same process as 80M-TI. Furthermore, we resized (bilinear interpolation) Places365 (Zhou et al., (017)) and VisDA17 (Peng et al., (017) for the experiments on ImgNet10 and cropped Simpson Characters (Simpson) (Attia 2018) and Fashion product (Fashion) (Aggarwal 2018) to dimensions of 32 32 for the experiments on CIFAR.
E IMPLEMENTATION DETAILS
For all the experiments except TRADES and IRAERS the initial learning rate is set to 0.1. The learning rate is multiplied by 0.1 at 50% and 75% of the total training steps, and the weight decay factor is set to 2e-4. We use the same adversarial perturbation budget = 8, as in Madry et al. (017).. We recorded the maximum adversarial robustness of the models the test after the first learning rate decay in adversarial training and the empirical upper bound of the test accuracy of the models during the standard learning. The other details are summarized Tables 4 and [ For TRADES and OATTRADES, we deploy a batch size of 64 and train the models using the same configurations as Zhang al. (2019).
F FURTHER DISCUSSION ABOUT THE EFFECTIVENESS OF OAT
We created a new ImgNet10 dataset with reference to Imagewoof (Hward)) to learn more about the effects of OAT. Imagewoof is subset of ImageNet, and all classes dog breeds. We tested
15

---

Published as a conference paper at ICLR 2021
Table Accuracy (%) comparison of the OAT model with Standard and PGD on ImgNet10 (64 x64) under different threat models. We show the improved results compared to the counterpart of each model bold.
Model
OOD
Clean
PGD20
CW20
Standard
75.77
PGD
53.64
16.18
14.81
ImgNet990
48.24
22.09
17.1
OAT
Places365
50.45
22.57
17.37
Table 7: Accuracy (%, mean over 5 runs) comparison of the OAT models with the baseline models. We show the improved results compared to the counterpart each model in bold.
Target
N
100
250
500
1250
2500
Full
Standard
19.42
25.48
31.49
47.04
57.12
75.77
ImgNet10 (64 x 64)
OAT ImgNec9900
22.74
30.07
35.16
48.84
58.14
75.17
OAT Places365
OATPhces365
22.36
30.15
33.57
48.16
57.15
75.63
Standard
20.03
25.36
30.69
50.69
64.82
81.22
ImgNet10 (160 x 160)
OAT ImgNec900
23.64
30.05
37.09
60.01
70.39
83.16
OAT OAT Places365
23.09
31.74
37.22
59.45
69.90
83.04
whether OAT is effective for the newly constructed ImgNet10 dataset, and the experimental results of OAT-A and OAT-S are shown in Tables 6 and 7 respectively. We can see from Tables and that for the ImgNet10, OAT is largely useful for generalization. However, from the results of ImgNet1 x4) in Table 7 we can speculate that OAT will help in training as a regularizer only when an excessive number of features are involved in the classification. This because OAT shows no performance improvement for the new ImgNet SigNet10((644) x(64 dataset. The new ImgNet1 1064et1000xxxxx 64) is believed to have lost much of the categorical features, since all the classes are dog breeds, in addition to being overly downscaled. In other words, OAT is beneficial for the overfitting problem, but not for the underfitting problem. We can confirm this by artificially reducing the number of training samples to increase the features that can distinguish between each categorical sample distribution, and then observe the improvement in generalization performance by applying OAT.
G ABLATION STUDY
Our experiments indicate that is possible enhance adversarial robustness by removing the contri- butions of non-robust features existing in additional data through OAT and transfer the consistency to in-distribution data. However, the increase the robustness against targeted attacks appears smaller than that against untargeted attacks in our method. To gain insight into this phenomenon, the OAT models for various a values and investigate the difference between untargeted and targeted attacks. In Table it can be that the greater the influence of OOD the training process is, the higher the robustness against PGD20 and the lower the robustness against CW20 are. According to our theoretical analysis, in OOD data, whereas relatively few features exist for targeted attacks. Because targeted attacks are
train
be understood that there are many features used for untargeted attacks
Table 8: Adversarial robustness (%) as hyperparameter changes
α
PGD20
CW20
1.0
57.45
52.65
2.0
58.25
52.16
3,0
58.31
52.02
4.0
59.04
51.71
5.0
59.48
51.24
16

---

Published as a conference paper at ICLR 2021
Table 9: Error rate (%) comparison of OAT with Mixup on CIFARIO, CIFARI00, and ImgNet10.
We show the best result for each target dataset in bold.
Model
Target
Ratio
Error rate
Standard
5.54
S0T-T
4.80
CIFARIO
Mixup
0.0
4.11
OAT Mixup
0.6
3.88
Standard
25.13
80M-T
24.35
CIFAR
Mixup
0.0
22.63
OAT KOM-TI Mixxp
0.6
21.60
Standard
13.07
OAT 10g
12.12
Places366
ImgNet10 (64 64)
11.63
Mixup
0.0
11.60
OAT ImgNe1990 Img Mixxup
0.7
10.77
OAT Placesto6 Mixupp
0.7
10.39
Standard
9.09
OAT ImgN
8.13
OAT Places365
ImgNet10 (160 160)
8.58
Mixup
0.0
7.04
OAT ImgNess90 +Mixuu
0.3
6.46
OAT Places365 Mixup
0.3
6.90
stronger than untargeted attacks (Carlini Wagner 2017). the results of Table 3 provide empirical evidence for the trade-off between the transferability of adversarial perturbations and the strength of adversarial attacks.
A similar trend was reported in the study of CCan et al. (2020). The authors showed that input gradient adversarial matching (IGAM) can transfer robustness across different tasks. Their results show that IGAM-trained models have similar higher robustness than baseline models against weak attacks, such as FGSM or low-step PGD attacks, but are vulnerable to strong attacks. The finetuning- based method can also be regarded as attempt to increase the robustness by using other datasets, but the abovementioned phenomenon is not observed in the method proposed by Hendrycks al. (2019). This is because the method does not transfer the robustness learned from other datasets to the target dataset but rather uses a function learned from a dataset that has large sample complexity and a data distribution similar to that of the target data with little modification.
This can be confirmed from the small number of training iterations and small learning rate involved by the fine-tur men-tuning process. Moreover, applying the same method dataset with small sample size or far from the target data distribution has no effect (Chan et al. 2020).
H
WHEN UID DATA ARE AVAILABLE
We train the OAT models in conjuction with UID data as follows:
min Oin
max C(x, + 8, 0) + an dES
max L(x, kunif 0)
(EL
D.ED.
eES
(23)
max L(x + ypscudoi 0) CES
+QUID
E
20
DuD
In the experiment of Figure aa) the Pseudo-label models trained via PGD-based approach, and every batch consists of 64 CIFAR-10 data and 64 pseudo-labeles data. The OAT+Pseudo-Be models are also trained via PGD-based approach, and every batch consists of 64 CIFAR-10 data, 32 pseudo-labele data, and 32 OOD data (80M-TI). The hyperparameters (ain: a., aum) are set (3. }). All other conditions the same described in Appendix E In the experiment of
17

---

Published as a conference paper at ICLR 2021
Figure [b), we train the models with the same configurations Carmon al. (2019). but every batch for the OAT+RST model comprises 128 pseudo-labeled data, 64 CIFAR-I0 data, and 64 OOD data (the RST model has batch size of 256). The hyperparameters (Cin: Ogr (tt))) are set to (0.25, 0.25, 0.5).
I EXPERIMENTAL RESULTS FOR MIXUP
Zhang et al. 2017) proposed data augmentation method named Mixup. Mixup generates training examples 2 Qz; (1 a.2;, and ay; + a3)y,, where (ri, Jk) and (rzz Mj) two examples drawn at random from the training data, and a E [0, 1J. Here, we show that even outside of the convex combination of training data can be effectively regularized using OOD data. Table shows that the models trained with the combination of OAT and Mixup always output the highest accuracy. The MAT+Mixup models are trained with the following algorithm:
Algorithm 2 OAT+Mixup
Require: Ratio y Target dataset D,, OOD dataset Do, uniform distribution label fnnir. batch size n. training iterations T, learning rate т
1: for 1 to T do
2:
(Xe, Y2) = SAMPLE (dttasetttttttt = Dr. size = n) X, SAMPLE
3:
DD, size [n 7
4:
(X,, Y.) + PERMUTE(X; Yi)
5:
(Xe [0: []* Y, [0: [n [n* (Xor tunit)
6:
(X, Y) f MIXUP X,, Y,, X2,
7:
model update:
8:
④ - VOAVERAGE(CIX, Y;
9: end for
10: Output: trained model parameter 0
18


Reviews:
[{"review_id": "Wnj4fkb9zxo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new data augmentation method that utilizes out-of-distribution data for enhancing generalizability for both supervised and adversarial learning. While most of existing data augmentation methods explore auxiliary unlabeled in-distribution data, this paper tries to leverage out-of-distribution data for enhancing performance. Theoretical analysis is first presented and explains why out-of-distribution data can help. Then, a simple method motivated from the analysis is proposed and later verified by extensive experiments including both supervised and adversarial learning experiments. \n\nThe proposed method is technically sound and extensive experiments are conducted to verify the effectiveness of the proposed method in different datasets for both the supervised learning and adversarial learning tasks. The paper is well motivated and well written. \n\nI am not familiar with the topic. My main concern is the technical contribution of this paper. It is straightforward that labeled data, though are out-of-domain, can enhance the capability of the feature extractor as the low layers of deep neural networks extract low-level semantics that are shared across images. There is no doubt about this. The paper provides some analysis on this and proposes a simple method which combines in-distribution and out-of-distribution data to train the model. The combination approach is very straightforward as well. I am not an expert in this area, but from an educated guess. I do not think this paper is good enough for acceptance.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PaYewOuqTdI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**1. Summary and contributions: Briefly summarize the paper and its contributions**\n In this work, authors looked at Out of Distribution (OOD) data from the data augmentation and regularization perspective and introduced Out-of-distribution data Augmented Training (OAT) based on their theoretical analysis which demonstrated that training with OOD data can remove undesirable feature contributions in a simple Gaussian model. The authors conducted experiments on both standard learning and adversarial learning and showed the effectiveness of OAT with strong results. Experimental results also imply that a common undesirable feature space exists among diverse datasets.  \n\n##########################################################################\n\n**2. Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the community.**\n \nVery strong experimental results, it is clear from the results that OAT is effective for both standard learning and adversarial learning. \n\nA sound theoretical analysis demonstrating that training with OOD data can remove undesirable feature contributions in a simple Gaussian model. \n\nNeat randomization test that analyzed the effect of OAT for standard learning. It verified the claim that OAT regularizes the model to learn only features with a strong correlation with class labels, even though the generalization gap is rather small.  \n\n##########################################################################\n\n**3. Weaknesses: Explain the limitations of this work along the same axes as above.**\n \nIn the middle of page 6: “In particular, from the results against AA it can be seen that the effectiveness of OAT does not rely on obfuscated gradients (Athalye et al., 2018).” It would be helpful to explain this a bit more.\n\nAdversarial training on OOD data could be more clearly described before the theoretical analysis. How exactly are you using the OOD data? Maybe switch 3.1 and 3.2? Then it will be easier for the readers to follow the theoretical analysis. I think it will improve readability if Equation 9 is introduced earlier in the paper. Why not introduce OAT earlier?\n\nFor the rest of the writing issues, see the next section. \n ##########################################################################\n\n**4. Clarity: Is the paper well written?**\nTypos: \\\nPage 2: a high success rates\\\nPage 2: a human-an ability thought to be\\\nAmbiguity: \\\nTitle: The word “remove” first appeared on page 4, the middle of the paper. Even though the paper conveyed the idea in the title, consider either change the title or more explicitly introducing the ideas in page 4 in the introduction section to improve readability. \\\nPage 3, The space X was never explicitly defined. \\\nPage 3, it’d be good to remind readers what is u.a.r\\\nPage 8, “can lead to higher performance by implementing existing data augmentation methods.” Did you mean when mixed with existing data augmentation methods?\\\n##########################################################################\n\n**5. Reasons for score**\nIn conclusion, the ideas are very interesting and the strengths outweigh weakness by a large margin, so I would recommend accept. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GGnQB44chM7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper seeks to remove \"non-robust\" features by first hypothesizing that out-of-distribution image datasets share the same spurious features as the target data distribution, then performing adversarial training on the out-of-distribution data (against uninformative labels) to remove these spurious features. They apply the method to PGD and TRADES adversarial training and show empirically that it improves the clean and robust accuracies. When unlabeled in-distribution data is available, the method can be combined with robust self-training to get further gains.\n\nStrengths:\n- The method seems to be reasonably robust to the choice of OOD dataset, which makes the method quite flexible.  \n- The method gives consistant and substaintial gains beyond PGD and TRADES adversarial training, as well as robust self-training, when OOD data is used. This is useful particularly since robust self-training (RST) doesn't seem to work well with OOD data (filtering unlabeled TinyImages data to be more like CIFAR was crucial to RST's success).\n- They test against a variety of attacks, including PGD with 100 steps, CW, and AA attacks. \n\nWeaknesses: Given the generality of using OOD data, there should be some things to check to make sure the given intuitions are correct.\n\n- To perhaps give a bit more supporting evidence that the OOD datasets help due to sharing the same non-robust features, some ablations could be done - for example, what if you use a synthetically generated OOD set (like Gaussian noise)? I think that random OOD examples could also help because it can provide a regularizing effect - recent works [1,2] show that generic regularization such as dropout or early stopping improve adversarial training methods. It's unclear to me that the benefit is necessarily from removing non-robust features. \n- The toy theoretical model seems to rely on the fact that the OOD dataset specifically only contains nonzero mean features for non-robust components, essentially allowing the algorithm to differentiate the non-robust features and robust features. However, it's unlikely that real-world OOD datasets have \\emph{only} the non-robust features. \n- How much does the choice of OOD dataset matter - why was SVHN, Fashion etc used for CIFAR while Places, Visda only used for ImageNet? Does the method work for all pairs of datasets? \n- If you combine all the OOD datasets, do we get even more gains? Why do we choose a particular OOD dataset rather than just using whatever images we can find?\n- From Table 1 and Table 2, it seems that OAT tends to further degrade the clean accuracy of PGD, although it slightly improves the clean accuracy of TRADES. \n\nOther: \n- It's pretty unclear throughout what is mean by the notion of \"undesirable features\", which also seems to be used synonymously with \"non-robust features\".\n- Can you use in-distribution unlabeled data with the OAT loss, and if so do we get a similar benefit?  \n\n\n[1] Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, Kamalika Chaudhuri. A Closer Look at Accuracy vs. Robustness, 2020.\n[2] Leslie Rice, Eric Wong, J. Zico Kolter. Overfitting in adversarially robust deep learning, 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DgcQLZepC6v", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose to use out-of-distribution (OOD) data to improve the generalization of deep neural networks, especially against adversarial attacks. Theoretic analysis and experimental results demonstrate the effectiveness of such method. The idea is interesting and the paper is easy to follow. The experiments are also thorough and support their theoretic analysis.\n\nHowever, I still have some concerns below:\n\n1.\tUnlabeled data [1,2,3] seems to be related to your work more than you claimed. And you missed two related works of unlabeled data. IPlease explain more on the difference between your method and previous works on unlabeled data and provide a more detailed comparison.\n\n2.\tYou adopt OOD data and assign each data with uniform distribution label. I am curious on what would happen if your OOD data contain some data that is overlapped with the data in training set. Would it still work well?\n\n[1] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, pp. 11190–11201, 2019.\n\n[2] J. Uesato, J. Alayrac, P. Huang, R. Stanforth, A. Fawzi, and P. Kohli. Are labels required for improving adversarial robustness? In Advances in Neural Information Processing Systems (NeurIPS), 2019.\n\n[3] Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John E. Hopcroft, Liwei Wang. Adversarially Robust Generalization Just Requires More Unlabeled Data. arXiv Preprint arXiv: 1906.00555.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the effect of using unlabelled out-of-distribution (OOD) data in the training procedure to improve robust (and standard) accuracies. The main algorithmic contribution is a data-augmentation based robust training algorithm to train a loss which is carefully designed to benefit from the additional OOD data. What's also interesting is that the OOD data is fed with random labels to the training procedure. As demonstrated in the theoretical results, this way of feeding OOD data helps to remove the dependency to non-robust features and hence improves robustness.  

As pointed out by all the reviewers (which I agree with), the idea of using unlabelled OOD data at training is novel/interesting, and the paper also shows how this can be done algorithmically. The numerical results also confirm the effectiveness of the proposed methods. 
Decision: Accept (Poster)