Title: On the Adversarial Robustness of Out-of-distribution Generalization Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial Robustness

Out-of-distribution Generalization
Abstract: Out-of-distribution (OOD) generalization has attracted increasing research attention in recent years, due to its promising experimental results in real-world applications. Interestingly, we find that existing OOD generalization methods are vulnerable to adversarial attacks. This motivates us to study OOD adversarial robustness. We first present theoretical analyses of OOD adversarial robustness in two different complementary settings. Motivated by the theoretical results, we design two algorithms to improve the OOD adversarial robustness. Finally, we conduct experiments to validate the effectiveness of our proposed algorithms.

On the Adversarial Robustness of Out-of-distribution Generalization Models

Abstract

Out-of-distribution (OOD) generalization has attracted increasing research attention in recent years, due to its promising experimental results in real-world applications. Interestingly, we find that existing OOD generalization methods are vulnerable to adversarial attacks. This motivates us to study OOD adversarial robustness. We first present theoretical analyses of OOD adversarial robustness in two different complementary settings. Motivated by the theoretical results, we design two algorithms to improve the OOD adversarial robustness. Finally, we conduct experiments to validate the effectiveness of our proposed algorithms. Our code is available at https://github.com/ZouXinn/OOD-Adv.

1 Introduction

Recent years have witnessed the remarkable success of modern machine learning techniques in many applications. A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution. However, this assumption is always violated in many practical applications. The test environment is influenced by a range of factors, such as the distributional shifts across the photos caused by different cameras in image classification tasks, the voices of different persons in voice recognition tasks, and the variations between scenes in self-driving tasks [48]. Therefore, there is now a rapidly growing body of research with a focus on generalizing to unseen distributions, namely out-of-distribution (OOD) generalization [56].

Deep neural networks (DNNs) have achieved state-of-the-art performance in many fields. However, several prior works [59, 25] have demonstrated that DNNs may be vulnerable to imperceptibly changed adversarial examples, which has increased focus on the adversarial robustness of the models. Adversarial robustness refers to the invariance of a model to small perturbations of its input [54], while adversarial accuracy refers to a model's prediction performance on adversarial examples generated by an attacker. However, the adversarial robustness of OOD generalization models (OOD adversarial robustness) is less explored, despite its importance in many systems requiring high security such as self-driving cars. We evaluate the adversarial robustness of the models trained with the current OOD generalization algorithms (the detailed experimental settings can be found in Appendix C.3), and present the results in Table 1. Surprisingly, under the PGD-20 [44] attack, the algorithms achieve nearly 0 % adversarial accuracy on RotatedMNIST [21], VLCS [15], PACS [38], and OfficeHome [62], and no more than 10 % adversarial accuracy on ColoredMNIST [4]. These results show that even if the OOD generalization algorithms generalize well in different scenes, they remain highly vulnerable to adversarial attacks.

Motivated by these limitations of existing algorithms, we provide theoretical analyses for OOD adversarial robustness in two different but complementary OOD settings. We then design two

baseline algorithms to improve the OOD adversarial robustness, based on the implications of our theory, and validate the effectiveness of our proposed algorithms through experiments.

Our contributions can be summarized as follows:

We evaluate the adversarial robustness of current OOD generalization algorithms and experimentally verify that the current OOD generalization algorithms are vulnerable to adversarial attacks.

We present theoretical analyses for the adversarial OOD generalization error bounds in the average case and the limited training environments case. Specifically, our bounds in limited training environments involve a 'distance" term between the training and the test environments. We further use a toy example to illustrate how the 'distance" term affects the OOD adversarial robustness, which is verified by the experimental results in Section 5.

Inspired by our theory, we propose two algorithms to improve OOD adversarial robustness. Extensive experiments show that our algorithms are able to achieve more than 53% average adversarial accuracy over the datasets.

The remainder of this article is structured as follows: §2 introduces related works. §3 presents our main theory. §4 shows our two theory-driven algorithms. §5 provides our experimental results. Finally, the conclusions are presented in the last section.

2 Related Work

2.1 Adversarial Robustness

[59] show that DNNs are fragile to imperceptible distortions in the input space. One of the most popular methods used to improve adversarial robustness is adversarial training (AT) . The seminal AT work, the fast gradient sign method [25, FGSM], perturbs a sample towards its gradient direction to increase the loss, then uses the generated sample to train the model. Following this line of research, [44, 47, 34, 11] propose iterative variants of the gradient attack with improved AT frameworks. [70, 30, 41] investigates the adversarial robustness from the perspective of ordinary differential equations. Recently, [50, 66] utilize the data from generative models as data augmentation to improve adversarial robustness. Besides, [43] analyze the trade-off between robustness and fairness, [37] study the worst-class adversarial robustness in adversarial training.

For the theoretical perspective, [46] study the PAC learnability of adversarial robust learning, [69] extend the work of [46] to multiclass case, [72, 31, 5, 18, 67] give theoretical analyses to adversarial training by standard uniform convergence argumentation and giving a bound of the Rademacher complexity, and [65, 78] study adversarial robustness under self-supervised learning.

2.2 Out-of-distribution generalization

OODgeneralization aims to train a model with data from the training environments so that it is capable of generalizing to an unseen environment. A large number of algorithms have been developed that aim to improve OOD generalization. One series of works focuses on minimizing the discrepancies between the training environments [40, 17, 42, 58, 1]. The most related work among them is [1], which measures the discrepancy between the domains by d H ( S, T ) , while we focus on adversarial robustness and use d B glyph[lscript] ( H ) ( S, T ) . Meta-learning domain generalization [39, MLDG] leverages the meta-learning approach and simulates train/test distributional shift during training by synthesizing virtual testing domains within each mini-batch. [53, GroupDRO] studies applying distributionally robust optimization (DRO) [19, 36, 77, 13] to learn models that instead minimize the worst-case training loss over a set of pre-defined groups. Another line of works [68, 64] conducts adversarial training to improve the OOD generalization performance. In this work, we focus on improving the OOD adversarial robustness.

From the theoretical perspective, [10] introduce a formal framework and argue that OOD generalization can be viewed as a kind of supervised learning problem by augmenting the original feature space with the marginal distribution of feature vectors. In [51], OOD generalization is cast into an online game where a player (model) minimizes the risk for a 'new" distribution presented by an adversary at each time-step. [14] propose a probabilistic framework for domain generalization called Probable Domain Generalization, wherein the key idea is that distribution shifts seen during training should inform us of probable shifts at test time. Notably, all these works focus on OOD generalization performance, while we present theoretical results for OOD adversarial robustness.

2.3 Works relating domain shifts and adversrial robustness

[8] focuses on improving the adversarial robustness of the models by regarding the adversarial distribution as the target domain and then applying the domain adaptation methods, while we focus on improving the model's adversarial robustness on the OOD distribution. [71] studies the relationship between adversarial robustness and OOD generalization, it shows that good adversarial robustness implies good OOD performance when the target domain lies in a Wasserstein ball. While we study the OOD adversarial robustness and propose algorithms to improve OOD adversarial robustness. [2] empirically analyzes the transferability of models' adversarial/certified robustness under distributional shifts. It shows that adversarially trained models do not generalize better without fine-tuning and that the accuracy-robustness trade-off generalizes to the unseen domain. Its results for adversarial robustness can also be found in our experimental results. [29] investigates how to improve the adversarial robustness of a model against ensemble attacks or unseen attacks. [29] regards the adversarial examples for each type of attack (such as FGSM, PGD, CW, and so on) as a domain, and utilizes the OOD generalization methods to improve the models generalization performance under different (maybe unseen) attacks.

[45, 22] study the relationship between the dependence on spurious correlations and the adversarial robustness of the models and show that adversarial training increases the model's reliance on spurious features. [60] studies the relationship between fairness and adversarial robustness and shows that models that are fairer will be less adversarially robust. However, they do not consider the adversarial robustness of the model on the unseen target domain, which is the topic of this paper.

3 Theoretical Analysis for OOD Adversarial Robustness

In this section, we present two theorems in two different settings, each of which inspires an algorithm designed to improve the OOD adversarial robustness. The proofs of all results in this section can be found in Appendix A. We first introduce some notations and basic setups.

Notations. We define [ n ] := { 1 , 2 , · · · , n } . We denote scalars and vectors with lowercase letters and lowercase bold letters respectively. We use uppercase letters to denote matrices or random variables, and uppercase bold letters to denote random vectors or random matrices. For a vector x ∈ R n , we define the glyph[lscript] p -norm of x as ‖ x ‖ p := ( ∑ n i =1 | x i | p ) 1 /p for p ∈ [1 , ∞ ) , where x i is the i -th element of x ; for p = ∞ , we define ‖ x ‖ ∞ := max 1 ≤ i ≤ n | x i | . For a matrix A ∈ R m × n , the Frobenius norm of A

is defined as ‖ A ‖ F := ( ∑ m i =1 ∑ n j =1 A 2 ij ) 1 2 , where A ij is the entry of A at the i -th row and j -th column. We define the determinant of A as det ( A ) . N ( µ, Σ) represents the multivariable Gaussian distribution with mean vector µ and covariance matrix Σ . Given f, g : R → R + , we write f = O ( g ) if there exist x 0 , α ∈ R + such that for all x > x 0 , we have f ( x ) ≤ αg ( x ) . We use sign ( · ) to denote the sign function [57].

Setups. Let X ∈ R m be the input space and Y be the label space. We set Y = {± 1 } , Y = { 1 , 2 , · · · , K } (where K is the number of classes), and Y = R for the binary classification problem, the multi-class classification problem, and the regression problem, respectively. We use glyph[lscript] : Y × Y - → R + as the loss function. We consider learning with the hypothesis class H ⊆ { h : X → Y} . Given a distribution D on X × Y , the error of h ∈ H with respect to the loss function glyph[lscript] under the distribution D is R D ( glyph[lscript],h ) = E ( x,y ) ∼D [ glyph[lscript] ( h ( x ) , y )] , where x ∈ X and y ∈ Y . We further define B ( · ) : X → 2 X

as a perturbation function that maps an input x to a subset B ( x ) ⊆ X , where 2 X is the power set of X . The adversarial error of the predictor h under the perturbation function B ( · ) is defined as

3.1 The Average Case

In this section, following [20, 14, 52], we consider the average case, i.e., the case in which the target environment follows a distribution. In this case, we aim to minimize the average target adversarial error of the hypothesis h ∈ H , where the average is taken over the distribution of the target environment.

Suppose that P ( X × Y ) is the set of all possible probability measures (environments) on X × Y for the task of interest. Assume there is a prior p on P ( X × Y ) , which is the distribution of the environments. Moreover, suppose the process of sampling training and test data is as follows:

(1). We generate the training data according to the following two steps: (i) we sample t training environments from p , i.e., D 1 , · · · , D t ∼ p ; (ii) the examples ̂ D i = { ( x i 1 , y i 1 ) , · · · , ( x in i , y in i ) } ∼ D n i i are drawn independent and identically distributed (i.i.d.) from D i , where n i is the size of ̂ D i and i ∈ [ t ] . To simplify the notations, we also use ̂ D i to denote the empirical distribution of the dataset { ( x i 1 , y i 1 ) , · · · , ( x in i , y in i ) } . Let ̂ D = 1 t ∑ t i =1 ̂ D i and ˆ p = 1 t ∑ t i =1 D i . To simplify the problem, we assume n 1 = n 2 = · · · = n t = n , but note that with a more careful analysis, our result in the average case can be extended to the case in which n 1 , · · · , n t are not necessarily the same.

(2). For each test example, we first sample an environment from p , i.e., D ∼ p , then sample an example ( x, y ) ∼ D .

Let L p ( glyph[lscript],h ) = E D∼ p [ R D ( glyph[lscript],h )] be the average risk of h on prior p . For the perturbation function B ( · ) , we define L B p ( glyph[lscript],h ) = E D∼ p [ R B D ( glyph[lscript],h ) ] as the average adversarial risk of h . The empirical Rademacher complexity [55, Chapter 26] of the hypothesis class H is defined as follows:

where σ is a Rademacher random vector with i.i.d. entries, and { x 1 , · · · , x n } is a set of data points. The following theorem presents an upper bound for the average adversarial risk of h .

Theorem 3.1. Suppose the loss function glyph[lscript] is bounded, i.e., glyph[lscript] ∈ [0 , U ] . Then with probability of at least 1 -δ over the sampling of ̂ D 1 , · · · , ̂ D t , the following bound holds for all h ∈ H :

where

Theorem 3.1 presents a bound for all hypotheses h ∈ H . We now consider the convergence property of the adversarial empirical risk minimization ( AERM ) algorithm in the average case. We define the output of the AERM algorithm as ˆ h ∈ arg inf h ∈H L B ̂ D ( glyph[lscript],h ) . We then define the hypothesis with the best adversarial generalization performance as h glyph[star] ∈ arg inf h ∈H L B p ( glyph[lscript],h ) .

The next corollary shows an upper bound for the excess risk [63, Chapter 4] of AERM:

Corollary 3.2. Suppose the loss function glyph[lscript] is bounded, i.e., glyph[lscript] ∈ [0 , U ] . Then with probability of at least 1 -δ over the sampling of ̂ D 1 , · · · , ̂ D t , the following bound holds:

Remark 1. The convergence rate for both Theorem 3.1 and Corollary 3.2 is O ( t -1 2 ) , which prompts us to ask: can we derive a tighter bound that has a faster convergence rate than O ( t -1 2 ) ? The next section provides an affirmative answer to this question.

3.2 Theory with Limited Environments

In this section, we provide the theoretical analysis for the limited training environment case, i.e., in the case in which t = O (1) . Suppose that we have t training environments D 1 , · · · , D t and one unknown target environment T . Our goal is to find a predictor that obtains good adversarial robustness on the target environment T . Assume we obtain n examples from each training distribution D i .

First, we introduce a discrepancy between the distributions. Based on the hypothesis class H , the seminal work for unsupervised domain adaptation (UDA), [9], defines the discrepancy between two distributions as follows:

[9] analyze the generalization error bound of the target domain for UDA by d H ∆ H ( · , · ) , where g ∈ H ∆ H ⇐⇒ g = h ⊕ h ′ for some h, h ′ ∈ H ; here, ⊕ is the XOR operator. However, the above definition is limited to the binary classification. This paper extends d H to the multi-class adversarial case. Given the loss function glyph[lscript] , distributions P , Q on X × Y , and hypothesis class H , we define the adversarial discrepancy as follows:

When B ( x ) = { x } , d B glyph[lscript] ( H ) ( P, Q ) becomes the standard case. It can be easily verified that d B glyph[lscript] ( H ) ( · , · ) is symmetric and satisfies the triangle inequality (the proof can be found in Appendix A.3); thus, d B glyph[lscript] ( H ) ( · , · ) is a pseudometric.

Comparison of d H ( · , · ) and d B glyph[lscript] ( H ) ( · , · ) . The theory outlined in [9, Theorem 2] presents an upper bound with a d H ∆ H ( · , · ) term. To align the feature distributions of the source and target domain, we need to calculate d H ∆ H ( P, Q ) , which takes the supremum over two hypotheses h, h ′ ∈ H :

glyph[negationslash]

glyph[negationslash]

However, the definition of d B glyph[lscript] ( H ) ( · , · ) shows that: d B glyph[lscript] ( H ) ( P, Q ) takes the supremum over one hypothesis h ∈ H , which is easier to optimize [76] and can thus significantly ease the minimax optimization in Section 4.2.

Theorem 3.3. For a given but unknown target distribution T , let ∆ t -1 := { ( λ 1 , · · · , λ t ) | λ i ≥ 0 , ∑ t i =1 λ i = 1 } be the ( t -1) -dimensional simplex, and Conv ( D ) := { ∑ t i =1 λ i D i ∣ ∣ λ ∈ ∆ t -1 } be the convex hull of D = {D 1 , . . . , D t } . Let T P ∈ arg inf D∈ Conv ( D ) d B glyph[lscript] ( H ) ( D , T ) be the 'projection" of T

onto Conv ( D ) , and λ glyph[star] be the weight vector where T P = ∑ t i =1 λ glyph[star] i D i . Assume glyph[lscript] ∈ [0 , U ] . Then: with probability of at least 1 -δ , for all h ∈ H ,

Moreover, R tn ( ˜ G ) and R n ( ˜ G ) measure the capacity of the model, and can be regarded as an implicit regularizer on the model. There are many existing works that present the upper bounds of the empirical Rademacher complexity for neural networks [49, 6, 7, 23] and the adversarial Rademacher complexity [31, 73, 18, 5, 67]. Their results can be summarized as follows: for bounded X , with proper weak assumptions on the loss function glyph[lscript] , R n ( ˜ G ) can be upper-bounded by O ( C √ n ) , where C is a constant that is related to some norm of the parameters of h and increases with the norm. Thus, we consider constraints on the norm of the parameters of h in the algorithm designing part of Section 4.

Last but not least, different from the results in Theorem 3.1, the convergence rate is O (√ ln 1 /δ tn + √ ln t/δ n ) . When t = O (1) , O (√ ln t/δ n ) in Theorem 3.3 converges much faster

The d B glyph[lscript] ( H ) ( T , T P ) term in Theorem 3.3 is determined by the distance between the source and target environments. A larger d B glyph[lscript] ( H ) ( T , T P ) may lead to worse adversarial robustness of the model on the target domain. Next, we present a toy example to illustrate this case.

Example 1. We consider the input space X = R d ⊆ R m and label space Y = { +1 , -1 } . For simplicity, we consider only three distributions on X × Y , i.e., D (0) , D (1) and D (2) . The marginal distributions for Y satisfy D (0) Y ( {-1 } ) = D (0) Y ( { +1 } ) = D (1) Y ( {-1 } ) = D (1) Y ( { +1 } ) = D (2) Y ( {-1 } ) = D (2) Y ( { +1 } ) = 1 2 . For D ( i ) , let the conditional distribution of X ( i ) given Y be X ( i ) | Y = y ∼ N ( yµ ( i ) , σ 2 I ) , where µ ( i ) ∈ R d is a non-zero vector and σ 2 I ∈ R d × d is the covariance matrix, i.e., the glyph[negationslash]

µ

Q

µ

Q

2

µ

α

α

R

class 1, domain 0

class 1, domain 1

class 1, domain 2

class -1, domain 0

class -1, domain 1

class -1, domain 1

elements of X ( i ) are independent. Let Q be a rotation transformation on R d , which is a special orthogonal transformation. Suppose that Q = I , where I is the identity transformation. Let Q ∈ R d × d be the matrix of Q under some orthonormal basis. We then know that Q is a rotation matrix and Q T Q = QQ T = I, det ( Q ) = 1 , Q = I . To model the distributional shift among the environments, we apply the transformation Q to µ (0) and use µ (1) = Qµ (0) , µ (2) = Q 2 µ (0) as the mean vectors of D (1) , D (2) respectively. Here, we consider the glyph[lscript] 2 -norm adversarial attack with radius glyph[epsilon1] , i.e., B ( x ) = { x ′ : ‖ x -x ′ ‖ 2 ≤ glyph[epsilon1] } . We use two environments as the training environments and the remainder as the test environment. Theorem 3.4 shows that a larger distance between the training and test environments leads to worse OOD adversarial robustness.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Theorem 3.4. Consider the setting in Example 1, and suppose that µ lies in the 2-dimensional subspace R in Theorem A.4 (see Appendix A.1 for details about R , and see Figure 1 for an intuitive visualization). Let glyph[lscript] 01 ( x, y ) = ✶ [ x = y ] , where ✶ [ · ] is the indicator function and B r p ( x ) = { x ′ : ‖ x -x ′‖ p ≤ r } . Consider training with hypothesis class H = { h w : h w ( x ) = sign ( w T x ) , w ∈ R d } , and denote D ( ij ) = 1 2 D ( i ) + 1 2 D ( j ) , i, j ∈ { 0 , 1 , 2 } , i = j . Consider the glyph[lscript] 2 -norm adversarial attack with radius glyph[epsilon1] ; for notation convenience, let ˜ R ij ( w ) = R B glyph[epsilon1] 2 D ( ij ) ( glyph[lscript] 01 , h w ) and ˜ R i ( w ) = R B glyph[epsilon1] 2 D ( i ) ( glyph[lscript] 01 , h w ) . Let Φ( · ) be the distribution function of the standard normal distribution. Let α denote the angle between µ and Qµ , which is the rotation angle of Q in the subspace R . Furthermore, suppose that 0 < α ≤ arccos glyph[epsilon1] ‖ µ 1 ‖ 2 . We then have:

If we train with D (0) and D (1) , let w (01) = µ + Qµ , which achieves the minimum of ˜ R 01 ( w ) . Then the adversarial accuracy of w (01) on the test distribution D (2) is ˜ R 2 ( w (01) )=Φ ( glyph[epsilon1] σ -( µ + Qµ ) T Q 2 µ σ ‖ µ + Qµ ‖ 2 ) .

If we train with D (0) and D (2) , let w (02) = µ + Q 2 µ , which achieves the minimum of ˜ R 02 ( w ) . Then the adversarial accuracy of w (02) on the test distribution D (1) is ˜ R 1 ( w (02) )=Φ ( glyph[epsilon1] σ -( µ + Q 2 µ ) T Qµ σ ‖ µ + Q 2 µ ‖ 2 ) .

If we train with D (1) and D (2) , let w (12) = Qµ + Q 2 µ , which achieves the minimum of ˜ R 12 ( w ) . Then the adversarial accuracy of w (12) on the test distribution D (0) is ˜ R 0 ( w (12) )=Φ ( glyph[epsilon1] σ -( Qµ + Q 2 µ ) T µ σ ‖ Qµ + Q 2 µ ‖ 2 ) .

Remark 3. Let { i, j, k } = { 0 , 1 , 2 } . For task i , we use D ( j ) , D ( k ) as the training environments and D ( i ) as the test environment. ˜ R i ( w ( jk ) ) is the target adversarial error of the learned classifier w ( jk ) in task i .

glyph[negationslash]

We now consider the distance between the training and test environments for each task. Intuitively, we regard the angle as the 'distance" between two distributions. We define angle ( i, j ) as the angle between the mean vector of D ( i ) and D ( j ) , i = j . For task i , we define the average 'distance" between the test environment and each training environment as d avg ( i ) := angle ( i,j )+ angle ( i,k ) 2 . We use d avg ( i ) as a measure of the distance between the training environments and the test environment for task i .

From the settings in Example 1, it can be clearly seen that d avg (0) = angle (0 , 1)+ angle (0 , 2) 2 = α +2 α 2 = 3 α 2 = d avg (2) and d avg (1) = angle (1 , 0)+ angle (1 , 2) 2 = α + α 2 = α , which implies that d avg (1) < d avg (0) = d avg (2) . Theorem 3.4 tells us that ˜ R 1 ( w (02) ) < ˜ R 0 ( w (12) ) = ˜ R 2 ( w (01) ) , and thus implies that a smaller distance between the training and test environments leads to better OOD adversarial robustness .

Moreover, the assumption of the angle between µ and Qµ in Theorem 3.4 is reasonable. Consider glyph[epsilon1] = ‖ µ ‖ 2 2 ; in this case, the attack is strong and perceptible to human eyes. Then, α ≤ arccos glyph[epsilon1] ‖ µ 1 ‖ 2 = arccos 1 2 = π 3 . In this case, the maximal angle between the environments is 2 π 3 , which leads to a strong distribution shift. When glyph[epsilon1] < ‖ µ ‖ 2 2 , the allowed rotation angle can be further enlarged, and when glyph[epsilon1] = 0 , it becomes the standard case.

Furthermore, the data distribution here can be regarded as a simplified data model for RotatedMNIST. Moreover, our experimental results in Section 5 are consistent with our analysis here in both the standard and adversarial cases. Please refer to the observation part of Section 5.2 and the table in Appendix D.1 for further details.

4 Algorithms

In this section, based on our theory, we present two algorithms that can improve the adversarial robustness of the model on the target environment.

4.1 Adversarial Empirical Risk Minimization (AERM or AT)

Based on Theorem 3.1, which shows an upper bound of the average adversarial risk over the environments, we propose our first algorithm: adversarial empirical risk minimization (AERM, which corresponds to applying AT to multiple source domains). The bound in Theorem 3.1 consists of the average adversarial empirical risk L B ̂ D ( glyph[lscript],h ) = 1 t ∑ t i =1 R B ̂ D i ( glyph[lscript],h ) and two empirical Rademacher complexity terms R t ( ˜ G ) + R tn ( ˜ G ) . As outlined in Remark 2, the empirical Rademacher complexity implies an implicit regularizer on the model capacity. We choose ‖ · ‖ F as a regularizer on the model parameters. The optimization objective of AT is as follows:

where λ is a trade-off hyper-parameter, and W is the parameter matrix of the model. From Remark 1, we know that AT may not generalize well in the case where the training environments are limited. Next, we propose another algorithm for the limited environment case.

4.2 Robust DANN (RDANN)

Theorem 3.3 shows that R n ( ˜ G ) , 1 t ∑ t i =1 R B ̂ D i ( glyph[lscript],h ) , and 1 t ∑ i ∑ j λ glyph[star] j d B glyph[lscript] ( H ) ( ̂ D i , ̂ D j ) play the key roles in designing the OOD adversarial training methods. However, the weights λ glyph[star] 1 , · · · , λ glyph[star] t are unknown, since we have no information about the target distribution T . Since λ glyph[star] j ∈ [0 , 1] , ∀ j , it is evident that 1 t ∑ i ∑ j λ glyph[star] j d B glyph[lscript] ( H ) ( ̂ D i , ̂ D j ) ≤ 1 t ∑ i ∑ j d B glyph[lscript] ( H ) ( ̂ D i , ̂ D j ) . We therefore turn to optimize the average discrepancy 1 t 2 ∑ i ∑ j d B glyph[lscript] ( H ) ( ̂ D i , ̂ D j ) . To improve the OOD adversarial robustness, we minimize the following:

where λ 1 , λ 2 are two hyper-parameters, and W is the parameter matrix of the model. However, the term 1 t 2 ∑ t i =1 ∑ t j =1 d B glyph[lscript] ( H ) ( ̂ D i , ̂ D j ) is a constant. Motivated by [17], we minimize the discrepancy of the training environments in the feature space of a feature extractor f .

Specifically, we consider H = { c ◦ f | f ∈ F , c ∈ C} ; this means that the predictor h consists of a classifier c and a feature extractor f , where F = { f : X → Z} , C = { c : Z → Y} and Z ⊆ R l is the feature space. Any feature extractor f ∈ F determines a hypothesis class H f = { c ◦ f | c ∈ C} . Given a feature extractor f , we apply Theorem 3.3 to the hypothesis class H f . Then with high probability, for any c ∈ C , the target error of h = c ◦ f can be controlled mainly by 1 t ∑ t i =1 R B ̂ D i ( glyph[lscript], c ◦ f ) and 1 t 2 ∑ t i =1 ∑ t j =1 d B glyph[lscript] ( H f ) ( ̂ D i , ̂ D j ) . We then aim to find c and f such that h = c ◦ f has good OOD adversarial robustness. Thus, we aim to minimize the following:

To minimize L ( c ◦ f ) , we need to solve a minimax problem. For simplicity, we fix B ( · ) , glyph[lscript] and define the following:

Since D ( h, P, P ) = 0 and D ( h, P, Q ) = D ( h, Q, P ) for any P, Q, h , our optimization problem can be formulated as follows:

To solve the minimax optimization problem, we adopt an idea similar to that of adversarial neural networks [24] and refer to c ij as the discriminator. However, there are t ( t -1) 2 discriminators in this case, and training with many discriminators may make the optimization process unstable [3]. We therefore opt to use the same discriminator for all ( ̂ D i , ̂ D j ) pairs. Note that:

such that optimizing over a shared discriminator c ′ is equivalent to optimizing a lower bound of the original objective. Our final optimization problem then becomes:

where w,w ′ , θ are parameters for c, c ′ , f respectively. We call our method robust adversarial-domain neural network ( RDANN ), the pseudo-code of which is presented in Algorithm 1.

Algorithm 1 RDANN

Input: the training data ̂ D 1 , . . . , ̂ D t , the number of iterations T , the number of iterations for the inner maximization problem k , the learning rate η , the inner max learning rate α , the tradeoff hyper-parameters λ 1 , λ 2 .

Output: the parameters w T , θ T for c, f .

1:

, θ

, w

0

2:

3:

4:

5:

6:

7:

8:

9:

10:

11:

initialize for

i

w

0

set

w

0

to

′

for

j

T

0

0

w

′

(

j

+1)

end for

i

1

L

L

i

L

2

θ

i

+1

c

(

i

L

1

w

i

+1

end for

5 Experiments

To verify the adversarial robustness of our algorithms, we conduct experiments on the DomainBed benchmark [26], a testbed for OOD generalization that implements consistent experimental protocols across various approaches to ensure fair comparisons. Our code is attached in the supplementary material.

5.1 Experimental Setup

Datasets. The datasets we use are RotatedMNIST [21], ColoredMNIST [4], VLCS [15], PACS [38], and OfficeHome [62]. There are several different environments in each dataset. For the DomainBed benchmark, we choose one environment as the test environment and use the others as training environments. We report the average accuracy over different choices of the test environment. Further details about the selected datasets can be found in Appendix C.2.

Backbone network. Following [26], we use a small CNN architecture for RotatedMNIST, ColoredMNIST, and ResNet-50 [27] for VLCS, PACS, and OfficeHome.

w

k

′

1

-

′

k

to

k

-

w

w

i

′

randomly do

1

do

j

, θ

+

θ

λ

i

+

i

α

∇

) +

1

-

w

η

i

-

L

λ

d

(

2

w

∇

θ

′

w

′

L

L

k

reg

, θ

i

L

η

∇

2

w

L

1

i

d

(

(

w

i

)

w

i

′

j

, θ

i

, θ

)

i

)

Hyper-parameters for adversarial attack and adversarial training. We use glyph[lscript] ∞ -norm attack for both adversarial attack and adversarial training. We use glyph[epsilon1] = 0 . 1 for ColoredMNIST and RotatedMNIST, and glyph[epsilon1] = 4 / 255 for VLCS, PACS, and OfficeHome; moreover, following [74, 75], we use PGD-10 to generate adversarial examples at the training stage and PGD-20 at the evaluation stage to avoid overfitting. The step size used to generate adversarial examples is set to be glyph[epsilon1]/ 4 .

More details of the experimental settings can be found in Appendix C.4.

5.2 Results

Table 2 presents the results of our experiments. As is clear from the table, our proposed algorithms significantly improve the OOD adversarial robustness of the model. For example, under the attack PGD-20, all other algorithms achieve no more than 3% average adversarial accuracy, while our proposed algorithms achieve more than 53% average adversarial accuracy. Moreover, in our datasets, the number of environments ( t ) is small. From Table 2, we can see that the overall performance of RDANN is superior to that of AT. RDANN achieves better or comparable adversarial accuracy on most datasets (except OfficeHome). The results are consistent with our claim in Remark 2: when t = O (1) , the bound in Theorem 3.3 converges faster than that in Theorem 3.1. The detailed results for each test environment are attached in Appendix D.

Observations. According to the detailed results for RotatedMNIST and ColoredMNIST in Appendix D.1 and Appendix D.2, we can make the following observations (these phenomena occur in both the adversarial training setting and the standard training setting):

glyph[negationslash]

For RotatedMNIST, we have six environments, each of which corresponds to a rotation angle of the original image. The rotation angle of the i -th environment is i × 15 ◦ , i ∈ { 0 , 1 , 2 , 3 , 4 , 5 } . For task i , we use the i -th environment as the test environment and the remaining environments as the training environments. Following Remark 3, we define d avg ( i ) := 1 5 ∑ j = i angle ( i, j ) as the average distance between the test environment and each training environment for task i . Then, d avg (0) = 45 ◦ , d avg (1) = 33 ◦ , d avg (2) = 27 ◦ , d avg (3) = 27 ◦ , d avg (4) = 33 ◦ and d avg (5) = 45 ◦ . We define the PGD-20 adversarial accuracy of the model trained by RDANN for task i as a ( i ) ; then, a (0) = 90 . 8 , a (1) = 94 . 7 , a (2) = 95 . 3 , a (3) = 95 . 6 , a (4) = 95 . 5 and a (5) = 89 . 0 . As i increases, d avg ( i ) first decreases and then increases, while a ( i ) first increases and then decreases. The result indicates that d avg ( i ) is anticorrelated with a ( i ) , which is consistent with the analysis in Remark 3. Note that this phenomenon occurs in all algorithms.

glyph[negationslash]

For ColoredMNIST, we have three environments, each of which corresponds to a correlation between the additional channel and the label. For task i , i ∈ { 0 , 1 , 2 } , we define the correlation as cor ( i ) and cor (1) = 0 . 9 , cor (2) = 0 . 8 , cor (3) = -0 . 9 . Here, we define d avg ( i ) = ∣ ∣ ∣ 1 2 ∑ j = i cor ( j ) -cor ( i ) ∣ ∣ ∣ , then d avg (1) = 0 . 95 , d avg (2) = 0 . 8 , d avg (3) = 1 . 75 . Similarly, we can define a ( i ) for a given algorithm. The detailed results in Appendix D.2 imply that d avg ( i ) is anticorrelated with a ( i ) . To further understand this phenomenon, we present another toy example for ColoredMNIST with a different data model in Appendix B.

6 Conclusion

In this paper, we focus specifically on out-of-distribution adversarial robustness. First, we show that existing OOD generalization algorithms are easily fooled by adversarial attacks. Motivated by this, we then study the theory of the adversarial robustness of models in two different but complementary OOD settings. Based on our theory, we propose two algorithms, AT and RDANN. Extensive experiments show that our proposed algorithms can significantly improve the OOD adversarial robustness of the model.

References

Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H. Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching, 2021.

Kumail Alhamoud, Hasan Abed Al Kader Hammoud, Motasem Alfarra, and Bernard Ghanem. Generalizability of adversarial robustness under distribution shifts. CoRR , abs/2209.15042, 2022.

Anonymous. Fairness and accuracy under domain generalization. In Submitted to The Eleventh International Conference on Learning Representations , 2023. under review.

Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR , abs/1907.02893, 2019.

Pranjal Awasthi, Natalie Frank, and Mehryar Mohri. Adversarial learning guarantees for linear hypotheses and neural networks. In ICML , volume 119, pages 431-441, 2020.

Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. In NeurIPS , pages 6240-6249, 2017.

Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3:463-482, 2002.

Pouya Bashivan, Reza Bayat, Adam Ibrahim, Kartik Ahuja, Mojtaba Faramarzi, Touraj Laleh, Blake A. Richards, and Irina Rish. Adversarial feature desensitization. In NeurIPS , pages 10665-10677, 2021.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Mach. Learn. , 79(12):151-175, 2010.

Gilles Blanchard, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. Journal of Machine Learning Research , 22:2:1-2:55, 2021.

Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In SP , pages 39-57, 2017.

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML , pages 2206-2216, 2020.

John C. Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. CoRR , abs/1810.08750, 2018.

Cian Eastwood, Alexander Robey, Shashank Singh, Julius von Kügelgen, Hamed Hassani, George J. Pappas, and Bernhard Schölkopf. Probable domain generalization via quantile risk minimization. CoRR , abs/2207.09944, 2022.

Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV , pages 1657-1664, 2013.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , volume 70, pages 1126-1135, 2017.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 17:59:1-59:35, 2016.

Qingyi Gao and Xiao Wang. Theoretical investigation of generalization bounds for adversarial learning of deep neural networks. Journal of Statistical Theory and Practice , 15:1-28, 2021.

Rui Gao and Anton J. Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics if Operations Research , 48(2):603-655, 2023.

Vikas K. Garg, Adam Tauman Kalai, Katrina Ligett, and Zhiwei Steven Wu. Learn to expect the unexpected: Probably approximately correct domain generalization. In AISTATS , volume 130, pages 3574-3582, 2021.

Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In ICCV , pages 2551-2559. IEEE Computer Society, 2015.

Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Singh Sachdeva, and Chitta Baral. Generalized but not robust? comparing the effects of data modification methods on out-ofdomain generalization and adversarial robustness. In ACL , pages 2705-2718, 2022.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In COLT , volume 75, pages 297-299, 2018.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , pages 2672-2680, 2014.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves crossdomain generalization. In ECCV , volume 12347, pages 124-140, 2020.

Adam Ibrahim, Charles Guille-Escuret, Ioannis Mitliagkas, Irina Rish, David Krueger, and Pouya Bashivan. Towards out-of-distribution adversarial robustness. CoRR , abs/2210.03150, 2022.

Qiyu Kang, Yang Song, Qinxu Ding, and Wee Peng Tay. Stable neural ODE with lyapunovstable equilibrium points for defending against adversarial attacks. In NeurIPS , pages 1492514937, 2021.

Justin Khim and Po-Ling Loh. Adversarial risk bounds for binary classification via function transformation. CoRR , abs/1810.09519, 2018.

Hoki Kim. Torchattacks : A pytorch repository for adversarial attacks. CoRR , abs/2010.01950, 2020.

David Krueger, Ethan Caballero, Jörn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distribution generalization via risk extrapolation (rex). In ICML , volume 139, pages 5815-5826, 2021.

Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In ICLR , 2017.

Yann LeCun. The mnist database of handwritten digits. http://yann.lecun.com/exdb/ mnist/ , 1998.

Jaeho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. In NeurIPS , pages 2692-2701, 2018.

Boqi Li and Weiwei Liu. WAT: improve the worst-class robustness in adversarial training. In AAAI , pages 14982-14990, 2023.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV , pages 5543-5551, 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI , pages 3490-3497, 2018.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In CVPR , pages 5400-5409, 2018.

Xiyuan Li, Xin Zou, and Weiwei Liu. Defending against adversarial attacks via neural dynamic system. In NeurIPS , 2022.

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV , volume 11219, pages 647-663, 2018.

Xinsong Ma, Zekai Wang, and Weiwei Liu. On the tradeoff between robustness and fairness. In NeurIPS , 2022.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

Mazda Moayeri, Kiarash Banihashem, and Soheil Feizi. Explicit tradeoffs between adversarial and natural distributional robustness. CoRR , abs/2209.07592, 2022.

Omar Montasser, Steve Hanneke, and Nathan Srebro. VC classes are adversarially robustly learnable, but only improperly. In COLT , pages 2512-2530, 2019.

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In CVPR , pages 2574-2582, 2016.

Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. In ICLR , 2021.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In COLT , volume 40, pages 1376-1401, 2015.

Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy A. Mann. Fixing data augmentation to improve adversarial robustness. CoRR , abs/2103.01946, 2021.

Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. An online learning approach to interpolation and extrapolation in domain generalization. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, AISTATS , volume 151, pages 2641-2657, 2022.

Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal representations for covariate shift. In ICLR , 2022.

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. CoRR , abs/1911.08731, 2019.

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In NeurIPS , 2020.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning - From Theory to Algorithms . Cambridge University Press, 2014.

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. CoRR , abs/2108.13624, 2021.

Sign function. Sign function, 2023. [Online; last edited on 16-January-2023].

Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In ECCV , volume 9915, pages 443-450, 2016.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR , 2014.

Cuong Tran, Keyu Zhu, Ferdinando Fioretto, and Pascal Van Hentenryck. Fairness increases adversarial vulnerability. CoRR , abs/2211.11835, 2022.

Vladimir Vapnik. Statistical learning theory . Wiley, 1998.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , pages 5385-5394, 2017.

Martin J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Qixun Wang, Yifei Wang, Hong Zhu, and Yisen Wang. Improving out-of-distribution generalization by adversarial training with structured priors. CoRR , abs/2210.06807, 2022.

Zekai Wang and Weiwei Liu. Robustness verification for contrastive learning. In ICML , pages 22865-22883, 2022.

Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In ICML , pages 36246-36263, 2023.

Jiancong Xiao, Yanbo Fan, Ruoyu Sun, and Zhi-Quan Luo. Adversarial rademacher complexity of deep neural networks. CoRR , abs/2211.14966, 2022.

Shiji Xin, Yifei Wang, Jingtong Su, and Yisen Wang. Domain-wise adversarial training for out-of-distribution generalization, 2022.

Jingyuan Xu and Weiwei Liu. On robust multiclass learnability. In NeurIPS , 2022.

Hanshu Yan, Jiawei Du, Vincent Y. F. Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. In ICLR . OpenReview.net, 2020.

Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang, Qun Liu, and Zhiming Ma. Improved OOD generalization via adversarial training and pretraing. In ICML , pages 1198711997, 2021.

Dong Yin, Kannan Ramchandran, and Peter L. Bartlett. Rademacher complexity for adversarially robust generalization. In ICML , volume 97, pages 7085-7094, 2019.

Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John E. Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. CoRR , abs/1906.00555, 2019.

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML , volume 97, pages 7472-7482, 2019.

Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan S. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In ICML , volume 119, pages 11278-11287, 2020.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I. Jordan. Bridging theory and algorithm for domain adaptation. In ICML , volume 97, pages 7404-7413, 2019.

Zhengyu Zhou and Weiwei Liu. Sample complexity for distributionally robust learning under chi-square divergence. Journal of Machine Learning Research , 24(230):1-27, 2023.

Xin Zou and Weiwei Liu. Generalization bounds for adversarial contrastive learning. Journal of Machine Learning Research , 24:114:1-114:54, 2023.

A Proofs

In this section, we show the proofs of the results in the mainscript.

A.1 Proofs of the Toy Example

Lemma A.1. If X ∼ N ( µ, Σ) where X ∼ R n , then for any A ∈ R m × n , we have:

Proof of Lemma A.1. We prove the lemma by a powerful tool, the characteristic function of random variables. The characteristic function of a random vector X is defined as:

Let Y = AX , we have:

Since X ∼ N ( µ, Σ) , we have:

so we have:

Since the characteristic function and the distribution is one-to-one, we have: Y obeys the multivariable Gaussian distribution and Y ∼ N ( Aµ,A Σ A T ) .

Lemma A.2. Let D Y ( Y = 1) = D Y ( Y = -1) = 1 2 , X ∈ R d and D X | Y = y = N ( yµ, Σ) where µ ∈ R d and Σ ∈ R d × d , then for any linear classifier h w ∈ H :

where 1 p + 1 p ∗ = 1 . Further more, if p = 2 and Σ = σ 2 I , then w glyph[star] = arg min w : h w ∈H R B glyph[epsilon1] 2 D ( glyph[lscript] 01 , h w ) , the weight of the most robust h w ∈ H under adversarial attack B glyph[epsilon1] 2 ( · ) is:

Proof of Lemma A.2. Let h w ( x ) = sign ( w T x ) , then we have:

glyph[negationslash]

where a is the result of Lemma A.1 when we regard w T as A in Lemma A.1.

If p = 2 and Σ = σ 2 I , then we have:

Since Φ( · ) is increasing, we need to minimize glyph[epsilon1] σ -w T µ σ ‖ w ‖ 2 , i.e., to maximize w T µ σ ‖ w ‖ 2 , it's easy to see that the maximum is achieved if we choose w = µ , so we have w glyph[star] = µ .

Lemma A.3. Let D 1 , D 2 to be defined as: D 1 ,Y ( Y = 1) = D 1 ,Y ( Y = -1) = D 2 ,Y ( Y = -1) = D 2 ,Y ( Y = -1) 1 2 , X ∈ R d and D 1 ,X | Y = y = N ( yµ 1 , σ 2 I ) , D 2 ,X | Y = y = N ( yµ 2 , σ 2 I ) where µ 1 , µ 2 ∈ R d , ‖ µ 1 ‖ 2 = ‖ µ 2 ‖ 2 and σ ∈ R + , let D = 1 2 D 1 + 1 2 D 2 and suppose α is the angle between µ 1 and µ 2 , if 0 < α ≤ 2 arccos glyph[epsilon1] ‖ µ 1 ‖ 2 , then w glyph[star] = µ 1 + µ 2 achieves the minimal robust risk.

Proof of Lemma A.3. To simplify the notation, we denote R B glyph[epsilon1] 2 D ( glyph[lscript] 01 , h w ) by R ( w ) . Then by Lemma A.2 we have:

where Φ( u ) = 1 √ 2 π ∫ u -∞ e -y 2 2 dy . The proof is divided into two steps. In the first step, we prove that for any weight w , we have R ( µ P ) ≤ R ( µ ) , where µ P is the project of µ onto the space span 〈 µ 1 , µ 2 〉 . Then we can reduce our analysis to the weights in the space span 〈 µ 1 , µ 2 〉 .

Step 1. For any w ∈ R d , we take a direct sum decomposition as w = w s + w t where w s ∈ span 〈 µ 1 , µ 2 〉 and w t ∈ span 〈 µ 1 , µ 2 〉 ⊥ , the orthogonal complement space of span 〈 µ 1 , µ 2 〉 , then we have:

Similarly, we have w T µ 2 = w T s µ 2 . Since w s ⊥ w t , then we have ‖ w ‖ 2 2 = ‖ w s ‖ 2 2 + ‖ w t ‖ 2 2 , so for any w ∈ R d , we have:

So it is obvious that R ( w s ) ≤ R ( w ) .

Step 2. According to the above results, we consider w ∈ span 〈 µ 1 , µ 2 〉 . Suppose the angle between µ 1 , µ 2 is α , note that R ( w ) does not depend on the length of w , so without loss of generality, we assume ‖ w ‖ 2 = 1 here. Suppose the angle between w and µ 1 is θ , then the angle between w and µ 2 is α -θ , then w is uniquely determined by θ , we denote R ( θ ) = R ( w ) , then we have:

By the rule of derivation for composition function, we have:

It is easy to see that when θ = α 2 or θ = α 2 + π , R ′ ( θ ) = 0 , since:

Similarly we have R ′ ( α 2 + π ) = 0 . Next we prove R ( θ ) achieves the minimum at θ = α 2 , taking the second order derivation of θ , we have:

Now, for simplicity, let k = ‖ µ 1 ‖ 2 , x = cos α 2 , then sin 2 α 2 = 1 -x 2 , then we consider f ( x ) = σ 2 x -k ( glyph[epsilon1] -kx )(1 -x 2 ) = σ 2 cos α 2 -‖ µ 1 ‖ 2 ( glyph[epsilon1] -‖ µ 1 ‖ 2 cos α 2 ) sin 2 α 2 , where α ∈ (0 , π ) , so x ∈ (0 , 1) . We have:

Since x ∈ (0 , 1) , then σ 2 x > 0 and k (1 -x 2 ) > 0 , by the assumption, 0 < α ≤ 2 arccos glyph[epsilon1] ‖ µ 1 ‖ 2 = 2 arccos glyph[epsilon1] k , so x = cos α 2 ≥ glyph[epsilon1] k , so kx -glyph[epsilon1] ≥ 0 , so we have f ( x ) > 0 , which means that R ′′ ( α 2 ) > 0 , so we know that θ = α 2 achieves the minimum of R ( θ ) , which means that the direction w glyph[star] = µ 1 + µ 2 achieves the minimum of R ( w ) . Similarly we can show that R ( θ ) achieves its maximum at θ = α 2 + π , which means that the direction -( µ 1 + µ 2 ) achieves the maximum of R ( w ) in span 〈 µ 1 , µ 2 〉 .

glyph[negationslash]

Theorem 3.4. Consider the setting in Example 1 and suppose that µ lies in the 2-dimensional subspace R in Theorem A.4 (see Appendix A.1 for details about R and see Figure 1 for intuitive illustration). Let glyph[lscript] 01 ( x, y ) = ✶ [ x = y ] , where ✶ [ · ] is the indicator function and B r p ( x ) = { x ′ : ‖ x -x ′‖ p ≤ r } , consider training with hypothesis class H = { h w : h w ( x ) = sign ( w T x ) , w ∈ R d } , and denote D ( ij ) = 1 2 D ( i ) + 1 2 D ( j ) , i, j ∈ { 0 , 1 , 2 } , i = j . Consider the glyph[lscript] 2 -norm adversarial attack with radius glyph[epsilon1] , for notation convenience, let ˜ R ij ( w ) = R B glyph[epsilon1] 2 D ( ij ) ( glyph[lscript] 01 , h w ) and ˜ R i ( w ) = R B glyph[epsilon1] 2 D ( i ) ( glyph[lscript] 01 , h w ) , let Φ( · ) be the distribution function of the standard normal distribution, let α denote the angle between µ and Qµ , which is the rotation angle of Q in subspace R , suppose that 0 < α ≤ arccos glyph[epsilon1] ‖ µ 1 ‖ 2 , then we have:

glyph[negationslash]

If we train with D (0) and D (1) under H , let w (01) = µ + Qµ , which achieves the minimum of ˜ R 01 ( w ) , then the robust accuracy of w (01) on the test distribution D (2) is:

If we train with D (0) and D (2) under H , let w (02) = µ + Q 2 µ , which achieves the minimum of ˜ R 02 ( w ) , then the robust accuracy of w (02) on the test distribution D (1) is:

If we train with D (1) and D (2) under H , then w (12) = Qµ + Q 2 µ , which achieves the minimum of ˜ R 12 ( w ) , then the robust accuracy of w (12) on the test distribution D (0) is:

What's more, we have ˜ R 1 ( w (02) ) < ˜ R 2 ( w (01) ) = ˜ R 0 ( w (12) ) .

Proof of Theorem 3.4. The values of ˜ R 0 ( w (12) ) , ˜ R 1 ( w (02) ) , ˜ R 2 ( w (01) ) is a direct result of Lemma A.2 and Lemma A.3. Next we compare the three adversarial risks.

Let C (01) = ( µ + Qµ ) T Q 2 µ σ ‖ µ + Qµ ‖ 2 , C (02) = ( µ + Q 2 µ ) T Qµ σ ‖ µ + Q 2 µ ‖ 2 , C (12) = ( Qµ + Q 2 µ ) T µ σ ‖ Qµ + Q 2 µ ‖ 2 , i.e., the subtractor in ˜ R 2 ( w (01) ) , ˜ R 1 ( w (02) ) , ˜ R 0 ( w (12) ) respectively. Since Φ( u ) = 1 √ 2 π ∫ u -∞ e -y 2 2 du is monotonely increasing, to judge which of the three robust errors in Theorem 3.4 is the smallest, we need only to judge which of the corresponding C ( ij ) is the largest.

Firstly, let Q e be the matrix of Q under the orthonormal basis e 1 , · · · , e d where ( e i ) j = δ ij and δ ij = 1 if i = j , δ ij = 0 otherwise. Then we have Qµ = Q e µ and we can see that C (12) = ( Q e µ + Q 2 e µ ) T µ σ ‖ Q e µ + Q 2 e µ ‖ 2 = µ T Q T e µ + µ T Q T e Q T e µ σ ‖ Q e ( µ + Q e µ ) ‖ 2 a = µ T Q e µ + µ T Q 2 e µ σ ‖ µ + Q e µ ‖ 2 = µ T Q T e Q e Q e µ + µ T Q 2 e µ σ ‖ µ + Q e µ ‖ 2 = ( µ + Q e µ ) T Q 2 e µ σ ‖ µ + Q e µ ‖ 2 = C (01) (where a is from the fact that ‖ Q e x ‖ 2 = ‖ x ‖ 2 , ∀ x and x T Ax = x T A T x, ∀ x, A ). So ˜ R 2 ( w (01) ) = ˜ R 0 ( w (12) ) .

To compare which one of ˜ R 1 ( w (02) ) and ˜ R 2 ( w (01) ) is smaller, we note that µ lies in the 2 -dimensional subspace R ∈ R d in Theorem A.4 as specified in Theorem 3.4, so there exists a scalar k ∈ R such that µ + Q 2 µ = k Qµ , and we know that Qµ ∈ arg min w : h w ∈H ˜ R 02 ( w ) (by Lemma A.2),

inf w : h w ∈H ˜ R 2 ( w ) (which is obvious according to Lemma A.2). And following the same direction of the proof of Theorem A.4, it's easy to see that w (01) = µ + Qµ is not in the same direction of Q 2 µ (the weight attains smallest robust error on D (2) ), so ˜ R 2 ( w (01) ) > inf w : h w ∈H ˜ R 2 ( w ) , so we have ˜ R 1 ( w (02) ) < ˜ R 2 ( w (01) ) = ˜ R 0 ( w (12) ) .

glyph[negationslash]

Theorem A.4. Let Q be a rotation transformation on R d and Q = I , then there exists at least one 2 -dimensional subspace R ∈ R d such that there exists a scalar k s.t. r + Q 2 r = kr for any r ∈ R .

Proof of Theorem A.4. By the results of matrix theory, we know that since Q is an orthonormal transformation, then there exists an orthonormal basis η 1 , . . . .η d under which the corresponding matrix of Q , Q , has the form:

glyph[negationslash]

Since Q is a rotation transformation, Q is a rotation matrix, so λ 1 = · · · = λ r = 1 . Once given the basis, linear transformations on R d and matrices in R d × d has a one-to-one correspondence, so Q = I tells us that Q = I , where I is the matrix of I under the orthonormal basis η 1 , . . . .η d . So we have m ≥ 1 in our case, without loss of generality we consider the subspace R = span 〈 η 1 , η 2 〉 be the subspace spanned by the vectors { η 1 , η 2 } .

glyph[negationslash]

For any α ∈ R , the coordinates of α under the basis η 1 , . . . .η d is then [ α 1 , α 2 , 0 , · · · , 0] T and the coordinates of Qα,Q 2 α is Q [ α 1 , α 2 , 0 , · · · , 0] T , Q 2 [ α 1 , α 2 , 0 , · · · , 0] T respectively. We now prove that there exists k = 2 cos θ 1 such that [ α 1 , α 2 , 0 , · · · , 0] T + Q 2 [ α 1 , α 2 , 0 , · · · , 0] T = kQ [ α 1 , α 2 , 0 , · · · , 0] T . We have:

So the coordinates of α + Q 2 α is:

which is just k times of the coordinates of Qα . So we get α + Q 2 α = kQα for any α ∈ R .

A.2 Proofs of the Average Case

Theorem 3.1. Suppose the loss function is bounded, i.e., glyph[lscript] ∈ [0 , U ] , then with probability at least 1 -δ over the sampling of ̂ D 1 , · · · , ̂ D t , for all h ∈ H , we have:

where

Proof of Theorem 3.1. Our proof consists of 2 steps, our 2-step uniform convergence analysis is different from (but based on) the standard Rademacher complexity generalization error bounds.

Step 1, upper bound the difference between L B p ( glyph[lscript],h ) and L B ˆ p ( glyph[lscript],h ) for any h ∈ H .

Let D = {D 1 , · · · , D t } , we define:

Since glyph[lscript] ∈ [0 , U ] , changing one of the D i in D will lead to at most U t change in Φ B ( D ) . So by McDiarmid's inequality we have:

so with probability at least 1 -δ 4 over the choice of D , we have:

Then we give an upper bound for E (Φ B ( D )) , by the symmetric technique , let S = {S 1 , · · · , S t } be a set of independent copy of D , we have:

where: a uses Jensen's inequality; b comes from the fact that the random variables σ i ( R B S i ( glyph[lscript],h ) -R B D i ( glyph[lscript],h ) ) and R B S i ( glyph[lscript],h ) - R B D i ( glyph[lscript],h ) are identically distributed, where σ i ∼ Uniform ( {± 1 } ) and σ ∈ {± 1 } t has independent elements σ 1 , · · · , σ t ; c is from the property of sup that sup( a + b ) ≤ sup( a ) + sup( b ) and the fact that σ has the same distribution as -σ ; d uses Jensen's inequality.

Since glyph[lscript] ∈ [0 , U ] , change in one data point ( x i , y i ) will cause at most U t difference in R t ( ˜ G ) , we then use McDiarmid's inequality to get: with probability at least 1 -δ 4 ,

Combine (A.1) with (A.2) we get: with probability at least 1 -δ 2 ,

Step 2, upper bound the difference between L B ˆ p ( glyph[lscript],h ) and L B ̂ D ( glyph[lscript],h ) for any h ∈ H .

Let ̂ D = ̂ D i ∪ · · · ∪ ̂ D t and then define:

Since glyph[lscript] ∈ [0 , U ] and there are tn examples in ̂ D , changing the j -th example of ̂ D i , ( x ij , y ij ) , will cause at most U tn change Ψ B ( ̂ D ) . So by McDiarmid's inequality we know: with probability at least 1 -δ 4 over the choice of ̂ D ,

Next we upper bound E [ Ψ B ( ̂ D ) ] , by the symmetric technique, let ̂ S = { ̂ S 1 , · · · , ̂ S t } be a set of independent copies of ̂ D and let (˜ x ij , ˜ y ij ) be the j -th example in ̂ S i , we have:

where: a is from Jensen's inequality; b is from the fact that Σ ij ( sup ˜ x ′ ij ∈B (˜ x ij ) glyph[lscript] ( h (˜ x ′ ij ) , ˜ y ij ) -sup x ′ ij ∈B ( x ij ) glyph[lscript] ( h ( x ′ ij ) , y ij ) ) has the same distribution as sup ˜ x ′ ij ∈B (˜ x ij ) glyph[lscript] ( h (˜ x ′ ij ) , ˜ y ij ) -sup x ′ ij ∈B ( x ij ) glyph[lscript] ( h ( x ′ ij ) , y ij ) , where Σ ij ∼ Uniform ( {± 1 } ) and Σ has independent entries; c is from the property of sup that sup( a + b ) ≤ sup( a ) + sup( b ) and the fact that Σ has the same distribution as -Σ .

Similar as before, if we change one of the { ( x ij , y ij ) } n i,j =1 , the change of R tn ( ˜ G ) is at most U tn , so by McDiarmid's inequality, with probability at least 1 -δ 4 , we have:

Combine (A.4) and (A.5) we have: with probability at least 1 -δ 2 ,

Combine (A.3) and (A.6) we have: with probability at least 1 -δ :

By the property of supremum, we have: with probability at least 1 -δ ,

Proof of Corollary 3.2. The main idea of the proof is to use risk decomposition to reduce L B p ( glyph[lscript], ˆ h ) -L B p ( glyph[lscript],h glyph[star] ) to L B p ( glyph[lscript],h ) -L B ̂ D ( glyph[lscript],h ) for some h . Then we have: with probability at least 1 -δ ,

where: a is from the fact that L B ̂ D ( glyph[lscript], ˆ h ) ≤ L B ̂ D ( glyph[lscript],h glyph[star] ) by the definition of ˆ h ; b is from Theorem 3.1, although in Theorem 3.1, we only have bounds for sup h ∈H ( L B p ( glyph[lscript],h ) -L B ̂ D ( glyph[lscript],h ) ) , we can use a similar argumentation as in Theorem 3.1 to get the same bound for sup h ∈H ( L B ̂ D ( glyph[lscript],h ) -L B p ( glyph[lscript],h ) ) , both with probability at least 1 -δ ′ , using union bound for these two cases we get a bound for sup h ∈H ∣ ∣ ∣ L B p ( glyph[lscript],h ) -L B ̂ D ( glyph[lscript],h ) ∣ ∣ ∣ with probability at least 1 -2 δ ′ , take δ = δ ′ 2 , we get the bound for sup h ∈H ∣ ∣ ∣ L B p ( glyph[lscript],h ) -L B ̂ D ( glyph[lscript],h ) ∣ ∣ ∣ with probability at least 1 -δ , this is why we get ln 4 /δ in Theorem 3.1 but ln 8 /δ here.

A.3 Proof of the Limited Environment Case

Proof d B glyph[lscript] ( H ) ( · , · ) is a pseudometric. It is obvious that d B glyph[lscript] ( H ) ( · , · ) is symmetric, now we proof that d B glyph[lscript] ( H ) ( · , · ) satisfies the triangle inequality. By definition, for distribution P, Q, R :

where: a is from the subadditivity of the supremum operator. So d B glyph[lscript] ( H ) ( · , · ) is a pseudometric.

Before prove Thorem 3.3, we first give a useful lemma.

Lemma A.5. For any h ∈ H , any distribution P, Q on , we have:

Proof of Lemma A.5. From the definition of d B glyph[lscript] ( H ) ( P, Q ) , we can know that:

which is just what we want.

Theorem 3.3. For a given but unknown target distribution T , let ∆ t -1 := { ( λ 1 , · · · , λ t ) | λ i ≥ 0 , ∑ t i =1 λ i = 1 } be the t -dimensional simplex and Conv ( D ) := { ∑ t i =1 λ i D i ∣ ∣ λ ∈ ∆ t -1 } be the convex hull of D = {D 1 , . . . , D t } , define T P ∈ arg inf D∈ Conv ( D ) d B glyph[lscript] ( H ) ( D , T ) be the 'projection" of T

onto Conv ( D ) and λ glyph[star] be the weight vector where T P = ∑ t i =1 λ glyph[star] i D i , glyph[lscript] ∈ [0 , U ] , then we have: with probability at least 1 -δ , for all h ∈ H ,

Proof of Theorem 3.3. Our proof is divided into 2 steps, the first step gets results for the population distributions and the second step uses finite sample approximation to get results for empirical distributions.

Step 1. Get the relationship between the population distributions .

Lemma A.5 tells us that: for all h ∈ H ,

where the last inequality is from the triangle inequality of d B glyph[lscript] ( H ) ( · , · ) and ¯ D = 1 t ∑ t i =1 D i . Recall that ̂ D = 1 t ∑ t i =1 ̂ D i , then we have:

where: a is from the triangle inequality of d B glyph[lscript] ( H ) ( · , · ) ; b follows from the linearity of R B D ( glyph[lscript],h ) and the subadditivity of the supremum operator. So we have, for any h ∈ H :

Step 2. We now show the bound of the finite sample approximation error d B glyph[lscript] ( H ) ( T P , ̂ T P ) and d B glyph[lscript] ( H ) ( ̂ D , ¯ D ) . Since we have no access to the population distribution ¯ D , we also give a finite sample approximation of R B ¯ D ( glyph[lscript],h ) and bound the corresponding approximation error.

The empirial distribution of ¯ D is ̂ D = 1 t ∑ t i =1 ̂ D i , then we have:

where Ψ B ( ̂ D ) is defined in the proof of Theorem 3.1. Then we have, with probability at least 1 -δ 4 , for all h ∈ H ,

Note that d B glyph[lscript] ( H ) ( ̂ D , ¯ D ) = sup h ∈H ∣ ∣ ∣ R B ¯ D ( glyph[lscript],h ) -R B ̂ D ( glyph[lscript],h ) ∣ ∣ ∣ , similar as the bound of Ψ B ( ̂ D ) = sup h ∈H ( R B ¯ D ( glyph[lscript],h ) -R B ̂ D ( glyph[lscript],h ) ) , we have that with probability at least 1 -δ 4 :

so with probability at least 1 -δ 2 :

Now we bound d B glyph[lscript] ( H ) ( T P , ̂ T P ) , by the definition of T P , we have:

For each i , we define:

Since glyph[lscript] ∈ [0 , U ] , changing one example in ̂ D i will lead to at most 1 n chang in Γ B ( ̂ D i ) , so by McDiarmid's inequality we know: with probability at least 1 -δ 16 t ,

Next we upper bound E [ Γ B ( ̂ D i ) ] , by similar analysis as in the proof of Theorem 3.1, we have: with probability at least 1 -δ 16 t ,

Then with probability at least 1 -δ 8 t ,

With similar argument, we know that with probability at least 1 -δ 8 t ,

So with probability at least 1 -δ 4 t ,

So with probability at least 1 -δ 4 ,

,

B Toy Example to Model the ColoredMNIST

Example 2. Consider data point ( x, y ) , where x ∈ X ⊆ R d +1 consists of invariant feature x inv ∈ X inv ⊆ R d and spuriously correlated feature x sp ∈ {± s } glyph[subsetnoteql] R , to simplify the example, let x =

( x inv , x sp ) be the feature observed. Consider the case there is a linear classifier h ( x inv ) = w inv · x inv such that y · h ( x inv ) > 0 , i.e., x inv is invariant for all distributions on X inv × {± 1 } . Suppose the dataset S = { ( x i , y i ) } n i =1 is linearly separable, and the dataset S is induced into two disjoint groups: a majority group S maj where x sp · y > 0 and a minority group S min where x sp · y < 0 , let p = P [ x sp · y > 0] > 0 . 5 , which means that the spurious feature is positively related to the label y .

The next theorem considers the degree of dependence on the spurious feature of the trained linear classifier.

Theorem B.1 (Theorem 4 of [48]) . Let H be the set fo linear classifiers, h ( x ) = w inv x inv + w sp x sp . Consider any task that satisfies all the constraints in Section 3.1 of [48]. Consider a dataset S drawn from D such that the empirical distribution of x inv given x sp · y > 0 is identical to the empirical distribution of x inv given x sp · y < 0 . Let w inv ( t ) x inv + w sp x sp be initialized to the origin, and trained with an infinitesimal learning rate gradient descent to minimize the exponential loss on a dataset S . Then, for any ( x, y ) ∈ S , we have:

where:

p denotes the empirical level of spurious correlation, p = 1 | S | ∑ ( x,y ) ∈ S 1[ x sp · y > 0] which without generality is assumed to satisfy p ∈ [0 . 5 , 1) .

M denotes the maximum value of the margin of the max-margin classifier on S , i.e., M = max x ∈ S ˆ w · x where ˆ w is the max-margin classifier on S .

Remark 4. In the proof of Theorem B.1 (please refer to [48]), the loss function is:

We denote L 1 = E x inv ∼ ˆ D inv [ e -( w inv x inv + w sp s ) ] and L 2 = E x inv ∼ ˆ D inv [ e -( w inv x inv -w sp s ) ] , so L = pL 1 + (1 -p ) L 2 . To model the case in the ColoredMNIST dataset, we let S 1 denote the dataset with corruption rate 0 . 1 , S 2 the dataset with corruption rate 0 . 2 , S 3 the dataset with corruption rate 0 . 9 . For training environment with corruption rate p train, we have L train = p train L 1 +(1 -p train ) L 2 and L test = p test L 1 +(1 -p test ) L 2 if p train > 0 . 5 . otherwise we have L train = p train L 2 +(1 -p train ) L 1 and L test = p test L 2 +(1 -p test ) L 1 . What's more, since the lower bound > 0 for p > 1 2 , we have L 1 < L 2 . So in the standard training case:

When we train with S 1 , S 3 , then the overall p train = 0 . 1+0 . 9 2 = 1 2 . So according to Theorem B.1 we know that w sp ( t ) = 0 , which means that the classifier just depends on the invariant feature, so the performance on S 2 is good. Since w sp ( t ) = 0 , L 1 = L 2 , so we have L train = p train L 1 +(1 -p train ) L 2 = E x inv ∼ ˆ D inv [ e -( w inv x inv ) ] = L test. So the mixed training distribution here is the same as the test distribution, which means that we can generalize well in this case, which is consistent with our experimental results.

When training with S 2 , S 3 , then the overall p train = 0 . 2+0 . 9 2 = 0 . 55 , which is quite close to 0 . 5 , which means that the correlation between x sp and y is not very strong. It means that the upper bound and the lower bound in Theorem B.1 is small, so w sp is small, the classifier mainly depends on the invariant feature and slightly depends on the spurious feature, which is just a little negatively related to y when p test = 0 . 1 in the test environment. In this case, L train = p train L 1 +(1 -p train ) L 2 = 0 . 55 L 1 +0 . 45 L 2 and L test = p test L 1 +(1 -p test ) L 2 = 0 . 1 L 1 + 0 . 9 L 2 , and L test -L train = -0 . 45 L 1 + 0 . 45 L 2 = 0 . 45( L 2 -L 1 ) > 0 , but in this case w sp is small, so we have L 2 -L 1 is small according to (B.11) , so the difference between training error and test error is not too large, so the generalization performance for this case is not bad, which is consistent with our experimental results.

Finally, it comes to the failure case in our experimental results, i.e., using S 1 , S 2 as the training environments and S 3 as the test environment. In this case, the overall p train =

0 . 1+0 . 2 2 = 0 . 15 , so the classifier depends more on the spurious features compared with the above two cases, what's more, since p test = 0 . 9 , the relationship between x sp and y is almost reversed when we switch to the test stage, so we have L train = p train L 2 +(1 -p train ) L 1 = 0 . 85 L 1 + 0 . 15 L 2 and L test = p test L 2 + (1 -p test ) L 1 = 0 . 1 L 1 + 0 . 9 L 2 and L test -L train = -0 . 75 L 1 +0 . 75 L 2 = 0 . 75( L 2 -L 1 ) > 0 , and the w sp here is much larger than that in case 2, which means that L 2 -L 1 here is much larger than that in case 2, so the gap between training error and test error here map be much larger than that in case two, leading to the poor generalization performance in practice.

We can see that the analysing results here are consistent with our experimental results for ColoredMNIST in Appendix D.2.

C Experimental Settings

We run each algorithm 20 times and 1 trial (since adversarial training is time-consuming, we just run 1 trial rather than 3 trials as done in DomainBed). We use part of the training data as the validation set to select the best model of the 20 runs according to the adversarial robustness of the training environments. Following DomainBed, we use random hyper-parameters in the 20 runs.

C.1 Attacked Algorithms

Empirical Risk Minimization ( ERM , [61]) minimizes the errors across domains.

Meta-Learning for Domain Generalization ( MLDG , [39]) leverages MAML [16] to metalearn how to generalize across domains.

Class-conditional DANN ( C-DANN , [42]) is a variant of DANN [17] matching the conditional distributions P [ φ ( X d ) | Y d = y ] across domains, for all labels y .

Risk Extrapolation ( VREx , [33]) approximates IRM with a variance penalty.

Representation Self-Challenging ( RSC , [28]) learns robust neural networks by iteratively discarding (challenging) the most activated features.

Domain-wise Multiple-perturbation Adversarial Training ( MAT , [64]) use an universal adversarial perturbation (UAP) with low rank along the dimension of examples (for n examples, use the convex combination of k ( k < n ) perturbations as the universal perturbation) to conduct universal adversarial training (UAT) to improve the OOD generalization.

Adversarial Training with Low-rank Decomposed perturbations( LDAT , [64]) shares similar idea with MAT, but it uses an UAP with low rank along the input space (for N × N images, it constrains the N × N UAP matrix has rank l < N ) to conduct universal adversarial training (UAT) to improve the OOD generalization.

C.2 Datasets

We use the following datasets provided by the DomainBed [26]:

ColoredMNIST [4] is a variant of the MNIST handwritten digit classification dataset [35]. Domain d ∈ { 0 . 1 , 0 . 3 , 0 . 9 } contains a disjoint set of digits colored either red or blue. The label is a noisy function of the digit and color, such that color bears correlation d with the label and the digit bears correlation 0 . 75 with the label. This dataset contains 70000 examples of dimension (2 , 28 , 28) and 2 classes.

RotatedMNIST [21] is a variant of MNIST where domain d ∈ { 0 , 15 , 30 , 45 , 60 , 75 } contains digits rotated by d degrees. Our dataset contains 70000 examples of dimension (1 , 28 , 8) and 10 classes.

PACS [38] comprises four domains d ∈ { art, cartoons, photos, sketches } . This dataset contains 9991 examples of dimension (3 , 224 , 224) and 7 classes.

VLCS [15] comprises four photographic domains d ∈ { Caltech101, LabelMe, SUN09, VOC2007 } . This dataset contains 10729 examples of dimension (3 , 224 , 224) and 5 classes.

OfficeHome [62] includes domains four d ∈ { art, clipart, product, real } . This dataset contains 15588 examples of dimension (3 , 224 , 224) and 65 classes.

C.3 Settings for attacking standard OOD algorithms

We evaluate the adversarial robustness of some of the algorithms in Table 1. For adversarial attacks, we use glyph[lscript] ∞ adversarial perturbation upper bound glyph[epsilon1] = 0 . 1 for RotatedMNIST and ColoredMNIST, glyph[epsilon1] = 4 / 255 for others (it is because glyph[epsilon1] = 4 / 255 is use usually used in 224 × 224 colored images); we use the classical FGSM and PGD-20 as the attack methods, for PGD-20, the step size α = glyph[epsilon1]/ 4 , we realize the attacks according to the package 'torchattacks' [32].

We train the models in the same hyper-parameter random search setting (except that we just train the model for one trial) in DomainBed, and the we attack the models trained with the best hyperparameters choosed by the training-domain validation set model selection method in DomainBed with the same seed as used during training. For MAT and LDAT, we use the same hyper-parameter random search setting in [64] except that we train with 20 random hyper-parameter for 1 trial and they train with 8 random hyper-parameter for 3 trials.

C.4 Settings for OOD adversarial algorithms

In this subsection, we introduce the settings in our experiments in Section 5. We use the trainingdomain validation set model selection method in the experiment and train the model with 20 random hyper-parameter for 1 trial.

Basic setting for training hyper-parameter random search.

For AT, we use the same basic random search setting as ERM.

For RDANN, we use the same basic random search setting as DANN.

Setting for adversarial training. We use PGD-10 as the adversarial attack to generate adversarial examples for adversarial training, with glyph[epsilon1] = 0 . 1 for RotatedMNIST and ColoredMNIST and glyph[epsilon1] = 4 / 255 for the other 224 × 224 datasets, and we use the attack step size α = glyph[epsilon1]/ 4 as usually done in the adversarial training community.

Setting for evaluating adversarial robustness. We consider two attack methods, FGSM and PGD20, to evaluate the OOD adversarial robustness of the trained models. We use the same glyph[epsilon1] as that used for training but we do 20 iterations for PGD in the evaluation stage rather than just 10 iterations in the training stage to avoid overfitting, which is also a strategy used by the adversarial robustness community.

D Additional Experimental Results

In this section, we show the detailed adversarial robustness of the current OOD generalization algorithms and our proposed algorithms, here we show the adversarial robustness in each test environment, for the ColoredMNIST dataset and RotatedMNIST dataset, we set glyph[epsilon1] = 0 . 1 , and for other 224 × 224 datasets, we set glyph[epsilon1] = 4 255 .

In Appendices D.1 to D.3, we use FGSM and PGD-20 as the attacks to evaluate the adversarial robustness and the results are shown in the form of triple tuple (clean accuracy / accuracy under FGSM attak / accuracy under PGD-20 attack) . Each column represents the results for a test environment. The tables show that compared with existing methods, our methods (AT and RDANN) significantly improve the adversarial robustness of the model on the target domains.

In Appendix D.6, we use AutoAttack as the attacks to evaluate the adversarial robustness. Each column represents the results for a test environment. The results show that under AutoAttack, all the existing methods fails ( no more than 1% adversarial accuracy), and our methods significantly improves the adversarial robustness.

D.1 RotatedMNIST

D.2 ColoredMNIST

D.3 VLCS

D.4 PACS

D.6 Results for AutoAttack

D.6.1 RotatedMNIST

D.7 OfficeHome

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the adversarial robustness of models on the unseen target domain. Firstly, exhaustive experiments show that existing OOD generalization methods are vulnerable to adversarial attacks. Then, a theoretical analysis for the OOD adversarial robustness is presented. Finally, this paper proposes two methods (AERM and RDANN) based on the theoretical results.\n\nSTRENGTHS:\n- The topic of this paper is novel and the paper considers two perspectives of trustworthy machine learning, which is very useful in practice. I agree with the authors that the adversarial robustness of the model on the OOD domain is very important since in the real-world task, the data we face always has a different style from what we use during the training stage.\n- The theoretical analysis is strong and interesting. The authors provide theoretical analyses in two different settings (the average setting and the agnostic setting where the distance between the training domains and the test domain is unknown), which cover the practical cases according to how many domains we can sample from. I think the theoretical analysis of this paper is very comprehensive.\n- According to the experimental results, the proposed methods significantly improve the OOD adversarial robustness. Furthermore, the observations on RotatedMNIST and ColoredMNIST are quite interesting and are consistent with the theoretical results of the two examples, which motivates researchers to consider the hardness of the tasks in OOD generalization. \n- The paper is well-written. The details of the methods are adequately explained, the proofs of the theorems are presented step-by-step, and the meaning of the theoretical results is discussed in detail.\n\nWEAKNESSES:\n- There are other datasets in the DomainBed benchmark, however, the experiments are conducted only in three datasets (RotatedMNIST, ColoredMNIST, and VLCS). Although AT is time-consuming and DomainBed involve running many groups of random parameters, I think the authors show add at least one more dataset in DomainBed.\n\nQUESTIONS:\nAs I said in the weaknesses part, would you please provide additional experimental results for other datasets (at least one) in DomainBed?\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper first evaluates the adversarial robustness of current OOD generalization algorithms and shows that existing OOD generalization algorithms are vulnerable to adversarial attacks. Then, the paper presents a theoretical analysis of the adversarial robustness (of a model) on the unseen test domain. After that, the paper designs two algorithms to improve the OOD adversarial robustness inspired by the theoretical implications. Finally, extensive experiments are conducted and the improvement is significant.\n\nSTRENGTHS:\n(1) This paper takes the first step to study the adversarial robustness of a model in the out-of-distribution generalization setting. As the extensive experiments show, existing out-of-distribution generalization methods fail under adversarial attacks, which means that it is urgent to study this problem. So, I believe that the studied problem is meaningful for the machine learning reliability community.\n\n(2) The writing is good and the paper is easy to follow. On the one hand, the proof steps are so detailed that it is easy to check the correctness of the theorems. On the other hand, after the theorems, the paper provides some remarks to show the implications of the theorems to help the readers understand the messages that the theorems convey.\n\n(3) The empirical results are strong. The improvement on the out-of-distribution adversarial robustness is significant. The experiments are extensive, and the details of the experimental settings are described in detail.\n\n(4) The proposed algorithms are inspired by the theory, which means that they are guaranteed by the theory. The theoretical analyses are technically solid. This paper not only provides theoretical results in general settings (Theorem 2.1, Corollary 2.2, and Theorem 2.3) where the data distributions and the hypothesis class are not specified but also studies an intuitive example (Example 1, Theorem 2.4 and Example 2, Theorem C.1) that presents many intuitive understandings.\n\nWEAKNESSES:\n(1) The paper [1] seems to be very relevant to this paper, but the discussions are missing.\n\n(2) The clean out-of-distribution generalization accuracy on VLCS decreases for both AERM and RDANN, however, this does not happen for ColoredMNIST and RotatedMNIST. The authors do not provide adequate discussion on this phenomenon.\n\n[1] Ibrahim, Adam, et al. \"Towards Out-of-Distribution Adversarial Robustness.\" arXiv preprint arXiv:2210.03150 (2022).\n\nQUESTIONS:\n(1) Could you provide a discussion about the differences between this paper and [1]?\n\n(2) Please provide a discussion about why the clean accuracy of the proposed methods decreases on VLCS but not on ColoredMNIST and RotatedMNIST.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the adversarial robustness of OOD generalization algorithms. Authors claimed that existing OOD generalization algorithms fail in adversarial attacks (FGSM and PGD). To this end, authors first theoretically analyze the problem definition and generalization bound of OOD algorithms in adversarial settings. Then, they proposed two algorithms: adversarial ERM and robust DANN to solve the problem. Experiments on several datasets show the effectiveness of the algorithms.\n\nSTRENGTHS:\n1. The problem of adversarial OOD generalization is interesting and of great importance in real scenarios.\n2. The theoretical parts offer valuable insights in the algorithm design.\n3. The experimental results are showing the advantages of the algorithms.\n\nWEAKNESSES:\nI think this paper can be seen as a pioneering work in exploring adversarial robustness of OOD generalization algorithms, but unfortunately has many flaws:\n- Lack of related work discussion. Although the discussion of adv. and OOD is rare, there are certainly some existing works on this intersection. For instance, [Yi et al'2021, ICML] theoretically studied the bounds of adversarial and OOD generalization, and they pointed that at mild conditions, adversarial robustness can guarantee OOD generalization (Thm 3.1). And [Alhamound'2022, arXiv] studies the generalization of adv. algorithms, which is of high interest to your work.\n- The algorithms are intuitive and not novel. I highly appreciate the efforts in deriving the theories and then lead to the algorithm design. But these two algorithms are very simple and intuitive to design. For instance, the average ERM over several adv. distributions and average domain-adversarial training are easy to propose and can even be seen as baselines when one tries to solve the adv. OOD generalization problems. The only reason is that no one has explored this before. But to me, the two algorithms are very simple and not novel.\n- The theoretical part is dense, which I give my appreciation. But this part lacks enough motivation and is hard to read. I strongly encourage authors to revise this to better align with the main topic. In addition, most of the theories are based on Ben-David et al's distribution gap theory and do not present much technical novelty.\n- The experiments are lacking sufficient backbones and comparison methods:\n1. Lack of various backbones in addition to ResNet50. Have you tried Vision Transformer? ViTs are known to be better cope with adv. robustness than CNNs.\n2. Lack of other datasets. The largest dataset in this paper is VLCS, which is not enough for OOD generalization. You should at least utilize the DomainBed datasets.\n3. Lack of adversarial comparison methods. Adversarial robustness is an active area and I'm sure most of them can be applied to OOD algorithms. Authors did not compare with any existing efforts.\n4. Lack of interpretatbility. Why do the algorithms in this paper work? Or why do existing algorithms not work?\n\n\n\nReferences:\n\n[Yi et al'2021, ICML] Improved OOD Generalization via Adversarial Training and Pre-training.\n\n[Alhamound'2022, arXiv] Generalizability of Adversarial Robustness Under Distribution Shifts.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors use statistical learning theory tools and the HdH divergence to motivate methods to improve the performance of a domain generalizing model dealing with adversarially perturbed, out-of-distribution domains. The authors evaluate two methodologies in that setting, and show how this can indeed defend models.\n\nSTRENGTHS:\n1. While OoD generalization is mostly concerned with improving performance on out-of-distribution domains, there is not much work on evaluating the performance of OoD-trained models on adversarially attacking an out-of-distribution domain.\n2. The two approaches studied are motivated with theory, and are shown to work well in that setting.\n\nWEAKNESSES:\nFixing the following issues would lead me to accept the paper. They are related to toning down some claims and missing references.\n1. Missing references and comparisons. First, the relevant work of “Adversarial Feature Desensitization” by Bashivan et al. 2021. E.g., Bashivan et al. already appear to apply HdH theory to estimate adversarial generalization error bounds, etc. Your setup has some differences in that you’re using pairwise HdH distances on perturbed domains, but it still needs to mention previous HdH approaches to adversarial robustness (and potentially factor that in when making the claims). Moreover, RDANN also echoes “Generalizing to unseen domains via distribution matching” by Albuquerque et al. 2019, e.g., they train on non-perturbed domains while you do, but the logic is quite similar; RDANN would be applying Albuquerque et al.’s approach to adversarially train the model. \n2. Please correct me if I am wrong, but AERM appears to just be adversarial training as done by Madry et al. 2018 (and Goodfellow et al. 2014 before). Adversarial training is done over all samples seen during training; it’s orthogonal to having multiple domains or a single one. I value that the authors provide both theoretical motivation for it and experimental results in the particular setting of adversarial robustness of a model trained for OoD generalization. I would however avoid calling the method itself “your” contribution, or giving it a new name; it’s just adversarial training = training on the adversarially perturbed training dataset (agnostically to whether it corresponds to one or multiple domains).\n3. The benchmark/discussion of results/claims re: performance are quite strange, as they effectively compare undefended models with two defended models, so seeing an improvement in robustness is quite trivial. I suggest framing the results a bit better to avoid making it about improving over existing methods (of course it does, you’re using adversarial training), but rather discussing how adversarial training (your AERM) compares with the RDANN approach, and how on their own (instead of compared to undefended models), they provide robustness on OoD datasets that are adversarially perturbed. Interesting things with your results are that on RotatedMNIST and ColoredMNIST you do not suffer much from the usual drop in clean (= unperturbed) accuracy as you defend a model, compared to undefended models. \n\n\n**Typos and suggestions:**\n1. I believe the phrase “OOD adversarial robustness” is more indicative/suggestive of robustness against diverse perturbations, which is an existing problem, than “adversarial robustness of models trained for OoD generalization” (in fact, a google/bing search appears to confirm this view, with e.g. “Towards Out-of-Distribution Adversarial Robustness” Ibrahim et al. 2022). For minimal effort, I suggest rephrasing to “adversarial robustness of OoD” if you want to keep it short, to clear up any confusion.\n2. I suggest adding equation numbers. \n3. The equation below L250, $c_{ij}$ are introduced. I would write in the text what they correspond to (classifiers for pairs of distributions of features of perturbed domains).\n\nQUESTIONS:\nNo.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}