Title: Transformation-Invariant Learning and Theoretical Guarantees for OOD Generalization
Venue: NeurIPS 2024
Accepted: yes
Keywords: pac learning guarantees

theory for distribution shifts

sample complexity

ood generalization

vc dimension
Abstract: Learning with identical train and test distributions has been extensively investigated both practically and theoretically. Much remains to be understood, however, in statistical learning under distribution shifts. This paper focuses on a distribution shift setting where train and test distributions can be related by classes of (data) transformation maps. We initiate a theoretical study for this framework, investigating learning scenarios where the target class of transformations is either known or unknown. We establish learning rules and algorithmic reductions to Empirical Risk Minimization (ERM), accompanied with  learning guarantees. We obtain upper bounds on the sample complexity in terms of the VC dimension of the class composing predictors with transformations, which we show in many cases is not much larger than the VC dimension of the class of predictors. We highlight that the learning rules we derive offer a game-theoretic viewpoint on distribution shift: a learner searching for predictors and an adversary searching for transformation maps to respectively minimize and maximize the worst-case loss.

Transformation-Invariant Learning and Theoretical Guarantees for OOD Generalization

Abstract

Learning with identical train and test distributions has been extensively investigated both practically and theoretically. Much remains to be understood, however, in statistical learning under distribution shifts. This paper focuses on a distribution shift setting where train and test distributions can be related by classes of (data) transformation maps. We initiate a theoretical study for this framework, investigating learning scenarios where the target class of transformations is either known or unknown. We establish learning rules and algorithmic reductions to Empirical Risk Minimization (ERM), accompanied with learning guarantees. We obtain upper bounds on the sample complexity in terms of the VC dimension of the class composing predictors with transformations, which we show in many cases is not much larger than the VC dimension of the class of predictors. We highlight that the learning rules we derive offer a game-theoretic viewpoint on distribution shift: a learner searching for predictors and an adversary searching for transformation maps to respectively minimize and maximize the worst-case loss.

1 Introduction

It is desirable to train machine learning predictors that are robust to distribution shifts. In particular when data distributions vary based on the environment, or when part of the domain is not sampled at training such as in reasoning tasks. How can we train predictors that generalize beyond the distribution from which the training examples are drawn from? A common challenge that arises when tackling out-of-distribution generalization is capturing the structure of distribution shifts. A common approach is to mathematically describe such shifts through distance or divergence measures, as in prior work on domain adaptation theory [e.g., Redko et al., 2020] and distributionally robust optimization [e.g., Duchi and Namkoong, 2021].

In this paper, we put forward a new formulation for out-of-distribution generalization. Our formulation offers a conceptually different perspective from prior work, where we describe the structure of distribution shifts through data transformations. We consider an unknown distribution D over X × Y which can be thought of as the 'training' or 'source' distribution from which training examples are drawn, and a collection of data transformation maps T = { T : X → X} which can be thought of as encoding 'target' distribution shifts, hence denoted as { T ( D ) } T ∈T . We consider a covariate shift setting where labels are not altered or changed under transformations T ∈ T , and we write T ( D ) for notational convenience. Our goal, which will be formalized further shortly, is to learn a single predictor ˆ h that performs well uniformly across all distributions { T ( D ) } T ∈T .

We view this formulation as enabling a different way to describe distribution shifts through transformations T = { T : X → X} . The collection of transformations T can be viewed as either: (a) given to the learning algorithm as part of the problem, or (b) chosen by the learning algorithm.View (a) represents scenarios where the target distribution shifts are known and specified by some downstream application (e.g., learning a classifier that is invariant to image rotations and translations). View (b)

Emmanuel Abbe

EPFL and Apple

represents scenarios where there is uncertainty or there are no pre-specified target distribution shifts and we would like to perform maximally well relative to an expressive collection of transformations. We highlight next several problems of interest that can be captured by this formulation. We refer the reader to Section 7 for a more detailed discussion in the context of prior work.

Covariate Shift & Domain Adaptation. By Brenier's theorem [Brenier, 1991], when X = R d , then under mild assumptions, for any source distribution P over X and target distribution Q over X , there exists a transformation T : X → X such that Q = T ( P ) . Thus, by choosing an expressive collection of transformations T , we can address arbitrary covariate shifts.

Transformation-Invariant Learning. In many applications, it is desirable to train predictors that are invariant to transformations or data preprocessing procedures representing different 'environments' (e.g., an image classifier deployed in different hospitals, or a self-driving car operating in different cities).

Representative Sampling. In many applications, there may be challenges in collecting 'representative' training data. For instace, in learning Logic or Arithmetic tasks [Abbe et al., 2023], the combinatorial nature of the data makes it not possible to cover well all parts of the domain. E.g., there is always a limit to the length of the problem considered at training, or features may not be homogeneously represented at training (bias towards certain digits etc.). Choosing a suitable collection of transformations T under which the target function is invariant can help to model in such cases.

Adversarial Attacks. Test-time adversarial attacks such as adversarial patches in vision tasks [Brown et al., 2017, Karmon et al., 2018], attack prompts in large language models [Zou et al., 2023], and 'universal attacks' [Moosavi-Dezfooli et al., 2017] can all be viewed as instantiations constructing specific transformations T .

̸

Our Contributions. Let X be the instance space and Y = {± 1 } the label space. Let H ⊆ Y X be a hypothesis class, and denote by vc( H ) its VC dimension. Consider a collection of transformations T = { T : X → X} , and some unknown distribution D over X × Y . Let err( h, T ( D )) = Pr ( x,y ) ∼D [ h ( T ( x )) = y ] be the error of predictor h on transformed distribution T ( D ) .

Given a training sample S = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } ∼ D m , we are interested in learning a predictor ˆ h with uniformly small risk across all transformations T ∈ T . Formally,

This objective is similar to that considered in prior work on distributionally robust optimization [Duchi and Namkoong, 2021] and multi-distribution learning [Haghtalab et al., 2022]. The main difference is that in this work we are describing the collection of 'target' distributions { T ( D ) } T ∈T as transformations of the 'source' distribution D . This allows us to obtain new upper bounds on the sample complexity of learning under distribution shifts based on the VC dimension of the composition of H with T , denoted vc( H◦T ) (see Equation (3)). We describe next our results (informally):

In Section 2 (Theorem 2.1), we show that, given the knowledge of any hypothesis class H and any collection of transformations T , by minimizing the empirical worst case risk, we can solve Objective 1 with sample complexity bounded by vc( H◦T ) . Furthermore, in Theorem 2.2, we show that the sample complexity of any proper learning rule is bounded from below by Ω(vc( H◦T )) .

In Section 3 (Theorem 3.1), we consider a more challenging scenario in which H is unknown. Instead, we are only given an ERM oracle for H . We then present a generic algorithmic reduction (Algorithm 1) solving Objective 1 using only an ERM oracle for H , when the collection T is finite. This is established by solving a zero-sum game where the H -player runs ERM and the T -player runs Multiplicative Weights [Freund and Schapire, 1997].

In Section 4 (Theorem 4.1), we consider situations where we do not know which transformations are relevant (or important) for the learning task at hand, and so we pick an expressive collection T and aim to perform well on as many transformations as possible. We then present a different generic learning rule (Equation (4)) that learns a predictor ˆ h achieving low error (say ε ) on as many target distributions in { T ( D ) } T ∈T as possible.

In Section 5 (Theorems 5.1 & E.1), we extend our learning guarantees to a slightly different objective, Objective 7, that can be favorable to Objective 1 when there is heterogeneity in the noise

across different transformations. This is inspired by Agarwal and Zhang [2022] who introduced this objective.

2 Minimizing Worst-Case Risk

If we have access to, or know, the hypothesis class H and the collection of transformations T , then the most direct and intuitive way of solving Objective 1 is minimizing the empirical worst-case risk. Specifically,

̸

We highlight that this learning rule offers a game-theoretic perspective on distribution shift, where the H -player searches for a predictor h ∈ H to minimize the worst-case error while the T -player searches for a transformation T ∈ T to maximize the worst-case error. For instance, both predictors H and transformations T can be parameterized by neural network architectures, which is an interesting direction to explore further. We note that similar min-max optimization problems have appeared before in the literature on adversarial examples and generative adversarial networks [e.g., Madry et al., 2018, Goodfellow et al., 2020].

We present next a PAC-style learning guarantee for this learning rule which offers the interpretation that solving the min-max optimization problem in Equation (1) yields a predictor ˆ h ∈ H that generalizes to the collection of transformations T . We show that the sample complexity of this learning rule is bounded by the VC dimension of the composition of H with T , where

Theorem 2.1. For any class H , any collection of transformations T , any ε, δ ∈ (0 , 1 / 2 ) , any distribution D , with probability at least 1 -δ over S ∼ D m ( ε,δ ) where m ( ε, δ ) = O ( vc( H◦T )+log(1 /δ ) ε 2 ) ,

Proof. The proof follows from invoking uniform convergence guarantees with respect to the composition H ◦ T (see Proposition A.1 in Appendix A) and the definition of ˆ h described in Equation (2). Let h ⋆ ∈ H be an a-priori fixed predictor (independent of sample S ) attaining OPT ∞ = inf h ∈H sup T ∈T err( h, T ( D )) (or ε -close to it). By setting m ( ε, δ ) = O ( vc( H◦T )+log(1 /δ ) ε 2 ) and invoking Proposition A.1, we have the guarantee that with probability at least 1 -δ over S ∼ D m ( ε,δ ) ,

Since ˆ h, h ⋆ ∈ H , the inequality above implies that

Furthermore, by definition, since ˆ h minimizes the empirical objective, it holds that

By combining the above, we get

We show next that vc( H◦T ) can be much higher than vc( H ) and the dependency on vc( H◦T ) is tight for all proper learning rules, which includes the learning rule described in Equation (2) and more generally any learning rule that is restricted to outputting a classifier in H .

Theorem 2.2. ∀ k ∈ N , ∃X , H , T such that vc( H ) = 1 but vc( H◦T ) ≥ k , and the sample complexity of any proper learning rule A : ( X × Y ) ∗ →H solving Objective 1 is at least Ω(vc( H◦T )) .

A proof is deferred to Appendix C. We remark that the sample complexity cannot be improved by proper learning rules and this leaves open the possibility of improving the sample complexity with improper learning rules. There are many examples in the literature where there are sample complexity gaps between proper and improper learning [e.g., Angluin, 1987, Daniely and Shalev-Shwartz, 2014, Foster et al., 2018, Montasser et al., 2019, Alon et al., 2021]. In particular, it appears that we encounter in this work a phenomena similar to what occurs in adversarially robust learning [Montasser et al., 2019]. Nonetheless, even at the expense of (potentially) higher sample complexity, we believe that there is value in the simplicity of the learning rule described in Equation (2), and exploring ways of implementing it is an interesting direction beyond the scope of this work.

2.1 Examples and Instantiations of Guarantees

To demonstrate the utility of our generic result in Theorem 2.1, we discuss next a few general cases where we can bound the VC dimension of H composed with T , vc( H◦T ) . This allows us to obtain new learning guarantees with respect to classes of distribution shifts that are not covered by prior work, to the best of our knowledge.

Linear Transformations. Consider T being a (potentially infinite) collection of linear transformations. For example, in vision tasks, this includes many transformations that have been widely studied in practice such as rotations, translations, maskings, adding random noise (or any fixed a-priori arbitrary noise), and their compositions [Engstrom et al., 2019, Hendrycks and Dietterich, 2019].

Interestingly, for a broad range of hypothesis classes H , we can show that vc( H ◦ T ) ≤ vc( H ) without incurring any dependence on the complexity of T . Specifically, the result applies to any function class H that consists of a linear mapping followed by an arbitrary mapping. This includes feed-forward neural networks with any activation function, and modern neural network architectures (e.g., CNNs, ResNets, Transformers). We find the implication of this bound to be interesting, because it suggests (along with Theorem 2.1) that the learning rule in Equation (2) can generalize to linear transformations with sample complexity that is not greater than the sample complexity of standard PAC learning. We formally present the lemma below, and defer the proof to Appendix B.

Lemma 2.3. For any collection of linear transformations T and any hypothesis class of the form H = { f ◦ W : R d →Y | W ∈ R k × d ∧ f : R k →Y } , it holds that vc( H◦T ) ≤ vc( H ) .

Non-Linear Transformations. Consider T being a (potentially infinite) collection of non-linear transformations parameterized by a feed-forward neural network architecture, where each T = W L ◦ ϕ ◦ · · · ϕ ◦ W 2 ◦ ϕ ◦ W 1 and ϕ ( · ) = max { 0 , ·} is the ReLU activation function. Similarly, consider a hypothesis class H that is parameterized by a (different) feed-forward neural network architecture, where each h = sign ◦ ˜ W H ◦ ϕ ◦ · · · ϕ ◦ ˜ W 2 ◦ ϕ ◦ ˜ W 1 . Observe that the composition H◦T consists of (deeper) feed-forward neural networks, where h ◦ T = sign ◦ ˜ W H ◦ ϕ ◦· · · ϕ ◦ ˜ W 2 ◦ ϕ ◦ ˜ W 1 ◦ W L ◦ ϕ ◦ · · · ϕ ◦ W 2 ◦ ϕ ◦ W 1 . Thus, we can bound vc( H◦T ) by appealing to classical results bounding the VC dimension of feed-forward neural networks. For example, according to Bartlett et al. [2019], it holds that vc( H◦T ) ≤ O (( H + L ) P H◦T log( P H◦T )) , where H + L is the depth of the networks in H◦T and P H◦T is the number of parameters of the networks in H◦T (which is P H + P T ). In this context, Theorem 2.1 and Equation (2) present a new learning guarantee against distribution shifts parameterized by non-linear transformations induced with feed-forward neural networks.

Transformations on the Boolean hypercube. The Boolean hypercube has also received attention recently as a case-study for distribution shifts in Logic or Arithmetic tasks[Abbe et al., 2023]. We show next that when the instance space X = { 0 , 1 } d , we can bound the VC dimension of H ◦ T from above by the sum of the VC dimension of H and the VC dimensions of {T i } d i =1 where each T i = { x ↦→ T ( x ) i : T ∈ T } is a function class resulting from restricting transformations T : { 0 , 1 } d →{ 0 , 1 } d ∈ T to output only the i th bit. The proof is deferred to Appendix B

Lemma 2.4. When X = { 0 , 1 } d , for any hypothesis class H and any collection of transformations T , vc( H◦T ) ≤ O (log d )(vc( H ) + ∑ d i =1 vc( T i )) , where each T i = { x ↦→ T ( x ) i : T ∈ T } .

In this context, Theorem 2.1 and Equation (2) present a new learning guarantee against arbitrary distribution shifts parameterized by transformations on the Boolean hypercube, where the sample complexity (potentially) grows with the complexity of the transformations as measured by the VC

dimension. We note however that this learning guarantee does not address the problem of length generalization, since we restrict to transformations that preserve domain length.

̸

Adversarial Attacks. In adversarially robust learning, a test-time attacker is typically modeled as a pertubation function U : X → 2 X , which specifies for each test-time example x a set of possible adversarial attacks U ( x ) ⊆ X that the attacker can choose from at test-time [Montasser et al., 2019]. The robust risk of a predictor ˆ h is then defined as: E ( x,y ) ∼D [ sup z ∈U ( x ) ✶ [ ˆ h ( z ) = y ]] . On the other hand, the framework we consider in this paper can be viewed as restricting a test-time attacker to commit to a set of attacks T = { T : X → X} without knowledge of the test-time samples, and the risk of a predictor ˆ h is then defined as: sup T ∈T E ( x,y ) ∼D ✶ [ ˆ h ( T ( x )) = y ] . While less general, our framework still captures several interesting adversarial attacks in practice which are constructed before seeing test-time examples, such as adversarial patches in vision tasks [Brown et al., 2017, Karmon et al., 2018] and attack prompts for large language models [Zou et al., 2023] can be represented with linear transformations.

̸

3 Unknown Hypothesis Class: Algorithmic Reductions to ERM

Implementing the learning rule in Equation (2) crucially requires knowing the base hypothesis class H and the transformations T , which may not be feasible in many scenarios. Moreover, in many applications we only have black-box access to an off-the-shelve supervised learning method such as an ERM for H . Hence, in this section, we study the following question:

Can we solve Objective 1 using only an ERM oracle for H ?

We prove yes, and we present next generic oracle-efficient reductions solving Objective 1 using only an ERM oracle for H . We consider two cases,

Realizable Case. When OPT ∞ = 0 , i.e., ∃ h ⋆ ∈ H such that ∀ T ∈ T : err( h ⋆ , T ( D )) = 0 , there is a simple reduction to solve Objective 1 using a single call to an ERM oracle for H . The idea is to inflate the training dataset S to include all possible transformations T ( S ) = { ( T ( x ) , y ) : ( x, y ) ∈ S ∧ T ∈ T } (similar to data augmentation), and then run ERM on T ( S ) . Formal guarantee and proof are deferred to Appendix D. It is also possible, via a fairly standard boosting argument, to achieve a similar learning guarantee using multiple ERM calls (specifically, O (log |T ( S ) | ) ≤ O (log( | S | |T | )) ), where each ERM call is on a sample of size O ( vc ( H )) . So, we get a tradeoff between the size of a dataset given to ERM on a single call, and the total number of calls to ERM.

Agnostic Case. When OPT ∞ > 0 , the simple reduction above no longer works. Specifically, the issue is that running a single ERM on the inflation T ( S ) effectively minimizes average error over transformations T ∈ T as opposed to minimizing maximum error over transformations T ∈ T . So, OPT ∞ > 0 , by definition, implies there is no predictor h ∈ H that is consistent (i.e., zero error) on every transformation T ( S ) , T ∈ T , thus minimizing average error over transformations can be bad.

To overcome this limitation, we present a different reduction (Algorithm 1) that minimizes Objective 1 by solving a zero-sum game where the H -player runs ERM and the T -player runs Multiplicative Weights [Freund and Schapire, 1997]. This can be viewed as solving a sequence of weightedERM problems (with weights over transformations), where Multiplicative Weights determines the weight of each transformation.

Theorem 3.1. For any class H , collection of transformations T , distribution D and any ε, δ ∈ (0 , 1 / 2 ) , with probability at least 1 -δ over S ∼ D m ( ε,δ ) , where m ( ε, δ ) ≤ O ( vc( H◦T )+log(1 /δ ) ε 2 ) , running Algorithm 1 on S for R ≥ 8 ln |T | ε 2 rounds produces ¯ h = 1 R ∑ R r =1 h r satisfying

̸

Remark 3.2 . When T is a finite collection of transformations, we can bound vc( H◦T ) from above by O (vc( H ) + log |T | ) using the Sauer-Shelah-Perels Lemma [Sauer, 1972]. See Lemma B.1 and proof in Appendix B.

Algorithm 1: Reduction to Minimize Worst-Case Risk Input: Black-box ERM H , dataset S = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } , and transformations T . 1 For each T ∈ T , set Q 1 ( T ) = 1 |T | . 2 Set number of rounds R = 8 ln |T | ε 2 . 3 for 1 ≤ r ≤ R do 4 Run ERM H on m ERM i.i.d. samples drawn from the distribution induced by Q r over T and Unif( S ) , and let h r denote its output. 5 For each T ∈ T , update Q r +1 ( T ) = Q r ( T ) exp( -η (1 -err( h r ,T ( S )))) Z r , where Z r is a normalization factor such that Q r +1 is a distribution. Output: 1 R ∑ R t =1 h r .

Proof of Theorem 3.1. Let S = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } be an arbitrary dataset. By setting R ≥ 8 ln |T | ε 2 and invoking Lemma D.2, which is a helpful lemma (statement and proof in Appendix D) that instantiates the regret guarantee of Multiplicative Weights in our context, we are guaranteed that Algorithm 1 produces a sequence of distributions Q 1 , . . . , Q R over T that satisfy

At each round r , observe that Step 4 in Algorithm 1 draws an i.i.d. sample from a distribution P r over X × Y that is defined by Q r over T and Unif( S ) , and since ERM H is an ( ε, δ ) -agnostic-PAC-learner for H , Step 5 guarantees that

̸

Finally, by appealing to uniform convergence over H◦T (Proposition A.1), with probability at least 1 -δ over S ∼ D m ,

On finiteness of T . We argue informally that requiring T to be finite is necessary in general when only an ERM oracle for H is allowed. For example, consider a distribution supported on a single point ( x, -) on the real line where x = 5 , and transformations T i ( x ) = x + i for all i ≥ 1 induced by some collection { T i } i ∈ N . Calling ERM on a finite subset of these transformations T i 1 , . . . , T i k could return a predictor that labels x, x + i 1 , x + i 2 , . . . , x + i k with a label -and labels x + i k +1 , . . . with + (e.g., if H is thresholds) which fails to satisfy Objective 1. But it would be interesting to explore additional structural conditions that would enable handling infinite T , and leave this to future work.

4 Unknown Invariant Transformations

When we have a large collection of transformations T and there is uncertainty about which transformations T ∈ T under-which we can simultaneously achieve low error using a base class H , the learning rule presented in Section 2 (Equation 1) can perform badly. We illustrate this with the following concrete example:

Example 1 . Consider a class H = { h 1 , h 2 , h 3 } , a collection of transformations T = { T 1 , T 2 , T 3 } , and a distribution D with risks (errors) as reported in the table.

Then, solving Objective 1 may return predictor h 3 where ∀ T ∈ T : err( h 3 , T ( D )) = 49% , since we only need to compete with the worst-case risk OPT ∞ = 49% . However, predictor h 1 is arguably better since it achieves a low error of 1% on at least two out of the three transformations.

To address this limitation, we switch to a different learning goal-achieving low error under as many transformations as possible. We present next a different generic learning rule for any class H and any collection of transformations T , that enjoys a different guarantee from the learning rule presented in Section 2. In particular, it can be thought of as greedy since it maximizes the number of transformations under which low empirical error is possible, but also conservative since it ignores transformations under which low empirical error is not possible. Specifically, given a training dataset S , the learning rule searches for a predictor ˆ h ∈ H that achieves low empirical error on as many transformations T ∈ T as possible, say err( ˆ h, T ( S )) ≤ ε .

Another way of thinking about this learning rule is that it provides us with more flexibility in choosing the collection of transformations T , since the learning rule is not stringent on achieving low error on all transformations but instead attempts to achieve low error on as many transformations as allowed by the base class H . Thus, this is useful in situations where there is uncertainty in choosing the collection of transformations. We present next the formal learning guarantee for this learning rule,

Theorem 4.1. For any class H , any countable collection of transformations T , any distribution D and any ε, δ ∈ (0 , 1) , with probability at least 1 -δ over S ∼ D m , where m = O ( vc( H◦T ) log(1 /ε )+log(1 /δ ) ) , then

Furthermore, it holds that ∀ T ∈ T : err( ˆ h, T ( D )) ≤ err( ˆ h, T ( S )) + err( ˆ h, T ( S )) ε 3 + ε 3 .

Remark 4.2 . We can generalize the result above to any prior over the transformations T , encoded as a weight function w : T → [0 , 1] such that ∑ T ∈T w ( T ) ≤ 1 . By maximizing the weighted version of Equation (4) according to w , it holds that ∑ T ∈T w ( T ) ✶ [ err( ˆ h,T ( D )) ≤ ε/ 3 ] ≥ max h ⋆ ∈H ∑ T ∈T w ( T ) [err( h ⋆ ,T ( D )) ≤ 3 ε ] with high probability.

✶ Proof. The proof follows from the definition of ˆ h and using optimistic generalization bounds (Proposition A.2). By setting m ( ε, δ ) = O ( vc( H◦T ) log(1 /ε )+log(1 /δ ) ε ) and invoking Proposition A.2, we have the guarantee that with probability at least 1 -δ over S ∼ D m ( ε,δ ) , ( ∀ h ∈ H )( ∀ T ∈ T ) :

Since ˆ h ∈ H , inequality (4) above implies that ∀ T ∈ T if err( ˆ h, T ( S )) ≤ ε then err( ˆ h, T ( D )) ≤ 3 ε . Thus,

Furthermore, by definition, since ˆ h maximizes the empirical objective, it holds that

Since h ⋆ ∈ H , inequality (5) above implies that ∀ T ∈ T if err( h ⋆ , T ( D )) ≤ ε / 3 then err( h ⋆ , T ( S )) ≤ ε . Thus,

By combining the above three inequalities,

5 Extension to Minimizing Worst-Case Regret

When there is heterogeneity in the noise across the different distributions, Agarwal and Zhang [2022] argue that, in the context of distributionally robust optimization, solving Objective 1 may not be advantageous. Additionally, they introduced a different objective (see Objective 7) which can be favorable to minimize. In this section, we extend our guarantees for transformation-invariant learning to this new objective which we describe next.

For each T ∈ T , let OPT T = inf h ⋆ T ∈H err( h ⋆ T , T ( D )) be the smallest achievable error on transformed distribution T ( D ) with hypothesis class H . Given a training sample S = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } ∼ D m , we would like to learn a predictor ˆ h : X → Y with uniformly small regret across all transformations T in T ,

We illustrate with a concrete example below how solving Objective 7 can be favorable to Objective 1. Example 2 (Risk vs. Regret) . Consider a class H = { h 1 , h 2 } , a collection of transformations T = { T 1 , T 2 , T 3 , T 4 } , and a distribution D such that with errors as reported in the table:

Thus, solving Objective 1 may return predictor h 2 where ∀ T ∈ T : err( h 2 , T ( D )) = 1 / 2 , since we only need to compete with the worst-case risk OPT ∞ = 1 2 . However, solving Objective 7 will return predictor h 1 where ∀ T : T : err( h 1 , T ( D )) ≤ OPT T .

More generally, as highlighted by Agarwal and Zhang [2022], whenever there exists h ⋆ ∈ H satisfying ∀ T ∈ T : err( h ⋆ , T ( D )) -OPT T ≤ ε , solving Objective 7 is favorable. We present next a generic learning rule solving Objective 7 for any hypothesis class H and any collection of transformations T ,

̸

We present next a PAC-style learning guarantee for this learning rule with sample complexity bounded by the VC dimension of the composition of H with T . The proof is deferred to Appendix E.

Theorem 5.1. For any class H , any collection of transformations T , any ε, δ ∈ (0 , 1 / 2 ) , any distribution D , with probability at least 1 -δ over S ∼ D m ( ε,δ ) where m ( ε, δ ) = O ( vc( H◦T )+log(1 /δ ) ε 2 ) ,

Algorithmic Reduction to ERM. Using ideas and techniques similar to those used in Section 3, we develop a generic oracle-efficient reduction solving Objective 7 using only an ERM oracle for H . Theorem, proof, and algorithm are deferred to Appendix E.

0

200

400

600

800

1000

Epochs

50%

60%

70%

80%

90%

100%

Test Accuracy

Baseline

Invariant Permutations

0

25

50

75

100

125

150

175

200

Epochs

50%

60%

70%

80%

90%

Test Accuracy

Baseline

Invariant Permutations

6 Basic Experiment

We present results for a basic experiment on learning Boolean functions on the hypercube {± 1 } d . We consider a uniform distribution D over {± 1 } d and two target functions: (1) f ⋆ 1 ( x ) = Π d i =1 x i , the parity function, and (2) f ⋆ 2 ( x ) = sign( ∑ 2 j =0 (Π d/ 3 i =1 x j ( d/ 3)+ i )) , a majority-of-subparities function. We consider transformations T 1 , T 2 under which f ⋆ 1 , f ⋆ 2 are invariant, respectively (see Section 2). Since D is uniform, note that for any ˆ h : sup T ∈T err( ˆ h, T ( D f ⋆ )) = err( ˆ h, D f ⋆ ) .

Algorithms. We use a two-layer feed-forward neural network architecture with 512 hidden units as our hypothesis class H . We use the squared loss and consider two training algorithms. First, the baseline is running standard mini-batch SGD on training examples. Second, as a heuristic to implement Equation (2), we run mini-batch SGD on training examples and permutations of them. Specifically, in each step we replace correctly classified training examples in a mini-batch with random permutations of them (drawn from T ), and then perform an SGD update on this modified mini-batch. We set the mini-batch size to 1 and the learning rate to 0 . 01 . Results are averaged over 5 runs with different seeds and are reported in Figure 1. We ran experiments on freely available Google CoLab T4 GPUs, and used Python and PyTorch to implement code.

7 Related Work and Discussion

Covariate Shift, Domain Adaptation, Transfer Learning. There is substantial literature studying theoretical guarantees for learning when there is a 'source' distribution P and a 'target' distribution Q [see e.g., survey by Redko et al., 2020, Quinonero-Candela et al., 2008]. Many of these works explore structural relationships between P and Q using various divergence measures (e.g., total variation distance or KL divergence), sometimes incorporating the structure of the hypothesis class H [e.g., Ben-David et al., 2010, Hanneke and Kpotufe, 2019]. Sometimes access to unlabeled (or few labeled) samples from Q is assumed. Our work differs from this line of work by expressing the structural relationship between P and Q in terms of a transformation T where Q = T ( P ) .

Distributionally Robust Optimization. With roots in optimization literature [see e.g., Ben-Tal et al., 2009, Shapiro, 2017], this framework has been further studied recently in the machine learning literature [see e.g., Duchi and Namkoong, 2021]. The goal is to learn a predictor ˆ h that minimizes the worst-case error sup Q ∈P err( ˆ h, Q ) , where P is a collection of distributions. Most prior work adopting this framework has focused on distributions P that are close to a 'source' distribution D in some divergence measure [e.g., f -divergences Namkoong and Duchi, 2016]. Instead of relying on divergence measures, our work describes the collection P through data transformations T of D : { T ( D ) } T ∈T which may be operationally simpler.

Multi-Distribution Learning. This line of work focuses on the setting where there are k arbitrary distributions D 1 , . . . , D k to be learned uniformly well, where sample access to each distribution D i is provided [see e.g., Haghtalab et al., 2022]. In contrast, our setting involves access to a

single distribution D and transformations T 1 , . . . , T k , that together describe the target distributions: T 1 ( D ) , . . . , T k ( D ) . From a sample complexity standpoint, multi-distribution learning requires sample complexity scaling linearly in k while in our case it is possible to learn with sample complexity scaling logarithmically in k (see Theorem 3.1 and Lemma B.1). The lower sample complexity in our approach is primarily due to the assumption that the transformations T 1 , . . . , T k are known in advance, allowing the learner to generate k samples from a single draw of D . In contrast, in multi-distribution learning, the learner pays for k samples in order to see one sample from each of D 1 , . . . , D k . Therefore, while the sample complexity is lower in our setting, this advantage arises from the additional information/structure provided rather than an inherent improvement over the more general setting of multi-distribution learning. From an algorithmic standpoint, our reduction algorithms employ similar techniques based on regret minimization and solving zero-sum games [Freund and Schapire, 1997].

Invariant Risk Minimization (IRM). This is another formulation addressing domain generalization or learning a predictor that performs well across different environments [Arjovsky et al., 2019]. One main difference from our work is that in the IRM framework training examples from different environments are observed and no explicit description of the transformations is provided. Furthermore, to argue about generalization on environments unseen during training, a structural causal model is considered. Recent works have highlighted some drawbacks of IRM [Rosenfeld et al., 2021, Kamath et al., 2021]. For example, how in some cases ERM outperforms IRM on out-of-distribution generalization, and the sensitivity of IRM to finite empirical samples vs. infinite population samples.

Data Augmentation. A commonly used technique in learning under invariant transformations is data augmentation, which involves adding transformed data into the training set and training a model with the augmented data. Theoretical guarantees of data augmentation have received significant attention recently [see e.g., Dao et al., 2019, Chen et al., 2020, Lyle et al., 2020, Shao et al., 2022, Shen et al., 2022]. In this line of research, it is common to assume that the transformations form a group, and the learning goal is to achieve good performance under the 'source' distribution by leveraging knowledge of the invariant transformations structure. In contrast, our work does not make the group assumption over transformations, and our goal is to learn a model with low loss under all possible 'target' distributions parameterized by transformations of the 'source' distribution.

Multi-Task Learning. Ben-David and Borbely [2008] studied conditions underwhich a set of transformations T can help with multi-task learning, assuming that T forms a group and that H is closed under T . Our work does not make such assumptions, and studies a different learning objective.

Acknowledgments

We thank Suriya Gunasekar for insightful discussions at early stages of this work. This work was done in part under the NSF-Simons Collaboration on the Theoretical Foundations of Deep Learning. OMwas supported by a FODSI-Simons postdoctoral fellowship at UC Berkeley. HS was supported in part by Harvard CMSA. This work was conducted primarily while HS was at TTIC and supported in part by the National Science Foundation under grants CCF-2212968 and ECCS-2216899, and by the Defense Advanced Research Projects Agency under cooperative agreement HR00112020003. The views expressed in this work do not necessarily reflect the position or the policy of the Government and no official endorsement should be inferred.

References

Abbe, S. Bengio, A. Lotfi, and K. Rizk. Generalization on the unseen, logic reasoning and degree curriculum. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 31-60. PMLR, 2023. URL https://proceedings.mlr.press/v202/abbe23a.html .

Agarwal and T. Zhang. Minimax regret optimization for robust machine learning under distribution shift. In P.-L. Loh and M. Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 2704-2729. PMLR, 02-05 Jul 2022. URL https://proceedings.mlr.press/v178/agarwal22b.html .

Alon, S. Hanneke, R. Holzman, and S. Moran. A theory of PAC learnability of partial concept classes. In 62nd IEEE Annual Symposium on Foundations of Computer Science, FOCS 2021, Denver, CO, USA, February 7-10, 2022 , pages 658-671. IEEE, 2021. doi: 10.1109/FOCS52979. 2021.00070. URL https://doi.org/10.1109/FOCS52979.2021.00070 .

Alon, A. Gonen, E. Hazan, and S. Moran. Boosting simple learners. TheoretiCS , 2, 2023. doi: 10.46298/THEORETICS.23.8. URL https://doi.org/10.46298/theoretics.23.8 .

Angluin. Queries and concept learning. Mach. Learn. , 2(4):319-342, 1987. doi: 10.1007/ BF00116828. URL https://doi.org/10.1007/BF00116828 .

Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. CoRR , abs/1907.02893, 2019. URL http://arxiv.org/abs/1907.02893 .

L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. J. Mach. Learn. Res. , 20:63:1-63:17, 2019. URL http://jmlr.org/papers/v20/17-612.html .

Ben-David and R. S. Borbely. A notion of task relatedness yielding provable multiple-task learning guarantees. Machine learning , 73:273-287, 2008.

Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Mach. Learn. , 79(1-2):151-175, 2010. doi: 10.1007/S10994-009-5152-4. URL https://doi.org/10.1007/s10994-009-5152-4 .

Ben-Tal, L. E. Ghaoui, and A. Nemirovski. Robust Optimization , volume 28 of Princeton Series in Applied Mathematics . Princeton University Press, 2009. ISBN 978-1-4008-3105-0. doi: 10.1515/9781400831050. URL https://doi.org/10.1515/9781400831050 .

Blumer, A. Ehrenfeucht, D. Haussler, and M. Warmuth. Learnability and the Vapnik-Chervonenkis dimension. Journal of the Association for Computing Machinery , 36(4):929-965, 1989a.

Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the vapnikchervonenkis dimension. J. ACM , 36(4):929-965, 1989b. doi: 10.1145/76359.76371. URL https://doi.org/10.1145/76359.76371 .

Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Communications on pure and applied mathematics , 44(4):375-417, 1991.

B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer. Adversarial patch. arXiv preprint arXiv:1712.09665 , 2017.

Cesa-Bianchi and G. Lugosi. Prediction, learning, and games . Cambridge University Press, 2006. ISBN 978-0-521-84108-5. doi: 10.1017/CBO9780511546921. URL https://doi.org/ 10.1017/CBO9780511546921 .

Chen, E. Dobriban, and J. H. Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research , 21:1-71, 2020.

Cortes, S. Greenberg, and M. Mohri. Relative deviation learning bounds and generalization with unbounded loss functions. Ann. Math. Artif. Intell. , 85(1):45-70, 2019. doi: 10.1007/ S10472-018-9613-Y. URL https://doi.org/10.1007/s10472-018-9613-y .

Daniely and S. Shalev-Shwartz. Optimal learners for multiclass problems. In M. Balcan, V. Feldman, and C. Szepesvári, editors, Proceedings of The 27th Conference on Learning Theory, COLT 2014, Barcelona, Spain, June 13-15, 2014 , volume 35 of JMLR Workshop and Conference Proceedings , pages 287-316. JMLR.org, 2014. URL http://proceedings.mlr.press/v35/ daniely14b.html .

Dao, A. Gu, A. Ratner, V. Smith, C. De Sa, and C. Ré. A kernel theory of modern data augmentation. In International Conference on Machine Learning , pages 1528-1537. PMLR, 2019.

C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics , 49(3):1378-1406, 2021.

Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant. A general lower bound on the number of examples needed for learning. Information and Computation , 82(3):247-261, 1989.

Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. Exploring the landscape of spatial robustness. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 1802-1811. PMLR, 2019. URL http://proceedings.mlr.press/v97/engstrom19a.html .

J. Foster, S. Kale, H. Luo, M. Mohri, and K. Sridharan. Logistic regression: The importance of being improper. In S. Bubeck, V. Perchet, and P. Rigollet, editors, Conference On Learning Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018 , volume 75 of Proceedings of Machine Learning Research , pages 167-208. PMLR, 2018. URL http://proceedings.mlr.press/ v75/foster18a.html .

Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. , 55(1):119-139, 1997. doi: 10.1006/JCSS.1997.1504. URL https://doi.org/10.1006/jcss.1997.1504 .

J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial networks. Commun. ACM , 63(11):139-144, 2020. doi: 10.1145/3422622. URL https://doi.org/10.1145/3422622 .

Haghtalab, M. I. Jordan, and E. Zhao. On-demand sampling: Learning optimally from multiple distributions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022 , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 02917acec264a52a729b99d9bc857909-Abstract-Conference.html .

Hanneke and S. Kpotufe. On the value of target data in transfer learning. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 9867-9877, 2019. URL https://proceedings.neurips.cc/paper/2019/ hash/b91f4f4d36fa98a94ac5584af95594a0-Abstract.html .

Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/ forum?id=HJz6tiCqYm .

Kamath, A. Tangella, D. J. Sutherland, and N. Srebro. Does invariant risk minimization capture invariance? In A. Banerjee and K. Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 4069-4077. PMLR, 2021. URL http://proceedings.mlr.press/v130/kamath21a.html .

Karmon, D. Zoran, and Y. Goldberg. Lavan: Localized and visible adversarial noise. In International Conference on Machine Learning , pages 2507-2515. PMLR, 2018.

Lyle, M. van der Wilk, M. Kwiatkowska, Y. Gal, and B. Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00178 , 2020.

Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=rJzIBfZAb .

Montasser, S. Hanneke, and N. Srebro. Vc classes are adversarially robustly learnable, but only improperly. In A. Beygelzimer and D. Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2512-2530, Phoenix, USA, 25-28 Jun 2019. PMLR.

Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturbations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 86-94. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017.17. URL https://doi.org/10.1109/CVPR.2017.17 .

Namkoong and J. C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 2208-2216, 2016. URL https://proceedings.neurips.cc/paper/2016/ hash/4588e674d3f0faf985047d4c3f13ed0d-Abstract.html .

Quinonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning . Mit Press, 2008.

Redko, E. Morvant, A. Habrard, M. Sebban, and Y. Bennani. A survey on domain adaptation theory. CoRR , abs/2004.11829, 2020. URL https://arxiv.org/abs/2004.11829 .

Rosenfeld, P. K. Ravikumar, and A. Risteski. The risks of invariant risk minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=BbNIbVPJ-42 .

Sauer. On the density of families of sets. J. Comb. Theory, Ser. A , 13(1):145-147, 1972. doi: 10.1016/0097-3165(72)90019-2. URL https://doi.org/10.1016/0097-3165(72)90019-2 .

Shao, O. Montasser, and A. Blum. A theory of pac learnability under transformation invariances. Advances in Neural Information Processing Systems , 35:13989-14001, 2022.

Shapiro. Distributionally robust stochastic programming. SIAM J. Optim. , 27(4):2258-2275, 2017. doi: 10.1137/16M1058297. URL https://doi.org/10.1137/16M1058297 .

Shen, S. Bubeck, and S. Gunasekar. Data augmentation as feature manipulation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 19773-19808. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr. press/v162/shen22a.html .

Vapnik and A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications , 16(2):264-280, 1971.

Vapnik and A. Chervonenkis. Theory of Pattern Recognition . Nauka, Moscow, 1974.

Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. CoRR , abs/2307.15043, 2023. doi: 10.48550/ARXIV.2307.15043. URL https://doi.org/10.48550/arXiv.2307.15043 .

A Uniform Convergence

We can use tools from VC theory [Vapnik and Chervonenkis, 1971, 1974] to obtain uniform convergence guarantees that will allow us to establish our learning guarantees and sample complexity bounds in the remainder of the paper. The starting point is a simple but key observation concerning the hypothesis class H and the collection of transformations T . Specifically, consider the composition of H with T defined as:

We next apply VC theory to the class H◦T to obtain our uniform convergence guarantees. Formally,

Proposition A.1. For any class H , any collection of transformations T , any distribution D over X × Y , and any m ∈ N , with probability at least 1 -δ over S ∼ D m : ∀ h ∈ H , ∀ T ∈ T ,

Proof. Since the composition H ◦ T is a hypothesis class consisting of functions h ◦ T where h ∈ H , T ∈ T , the claim follows from the definition of VC dimension and uniform convergence guarantees for VC classes [Vapnik and Chervonenkis, 1971, 1974].

Proposition A.2 (Optimistic Rate) . For any class H , any collection of transformations T , any distribution D , and any m ∈ N , letting B ( m,δ ) := 1 m ( vc( H◦T ) log ( 2 em vc( H◦T ) ) +log( 4 / δ ) ) , with probability at least 1 -δ over S ∼ D m : ∀ h ∈ H , ∀ T ∈ T ,

Proof. The claim follows from applying relative deviation bounds, or optimistic rates, for the composition class H◦T [see e.g., Corollary 7 in Cortes et al., 2019].

B Bounding the VC dimension of H composed with T

Given the relevance of vc( H◦T ) in our theoretical study, in this section we explore the relationship between vc( H ◦ T ) and vc( H ) which we believe can be helpful in interpreting our results and comparing them with the sample complexity of standard PAC learning [which is controlled by vc( H ) Blumer et al., 1989a, Ehrenfeucht et al., 1989]. To this end, we consider below a few general cases and prove bounds on vc( H◦T ) in terms of vc( H ) and some form of capacity control for T . These results may be of independent interest.

Finitely many transformations. When T is a finite collection of transformations, we can bound vc( H◦T ) from above by O (vc( H ) + log |T | ) using the Sauer-Shelah-Perels Lemma [Sauer, 1972],

Lemma B.1. For any class H and any finite collection T , vc( H◦T ) ≤ O (vc( H ) + log |T | ) .

Proof. Consider an arbitrary set of points P = { x 1 , . . . , x m } ⊆ X . To bound vc( H◦T ) from above, it suffices to bound the number of behaviors when projecting the function class H◦T on P , defined as

Observe that

where the first inequality follows from the definition of Π H◦T ( P ) and the second inequality follows from the Sauer-Shelah-Perels Lemma [Sauer, 1972]. Solving for m such that the above bound is less than 2 m , implies that vc( H◦T ) ≤ O (vc( H ) + log |T | ) .

Linear transformations. Consider T being a (potentially infinite) collection of linear transformations. For example, in vision tasks, this includes transforming images through rotations, translations, maskings, adding random noise (or any fixed a-priori arbitrary noise), and their compositions. Surprisingly, for a broad range of hypothesis classes H (including linear predictors and neural networks), we can show that vc( H◦T ) ≤ vc( H ) without incurring any dependence on the complexity of T . Specifically, the result applies to any function class H that consists of a linear mapping followed by an arbitrary mapping. This includes feed-forward neural networks with any activation function, and modern neural network architectures (e.g., CNNs, ResNets, Transformers).

Lemma B.2. For any collection of linear transformations T and any hypothesis class of the form H = { f ◦ W : R d →Y | W ∈ R k × d ∧ f : R k →Y } , it holds that vc( H◦T ) ≤ vc( H ) .

Proof. By definition of the VC dimension, it suffices to show that H◦T ⊆ H . To this end, consider an arbitrary h = f ◦ W ∈ H where W : R d → R k is a linear map and f : R k →Y is an arbitrary map (see definition of H in the lemma statement), and consider an arbitrary linear transformation T ∈ T . Then, observe that for each x ∈ R d ,

where the last equality follows from Riesz Representation theorem and T ∗ is the adjoint transformation of T . Thus, we have shown that there exists ˜ W = T ∗ ( W ) such that ( f ◦ W )( T ( x )) = ( f ◦ ˜ W )( x ) for all x ∈ R d . Therefore, H◦T ⊆ H .

Transformations on the Boolean hypercube. When the instance space X = { 0 , 1 } d is the Boolean hypercube, we can bound the VC dimension of H◦T from above by the sum of the VC dimension of H and the sum of the VC dimensions of {T i } d i =1 where each T i = { x ↦→ T ( x ) i : T ∈ T } is a function class resulting from restricting transformations T : { 0 , 1 } d →{ 0 , 1 } d ∈ T to output only the i th bit.

Lemma B.3. When X = { 0 , 1 } d , for any hypothesis class H and any collection of transformations T , vc( H◦T ) ≤ O (log d )(vc( H ) + ∑ d i =1 vc( T i )) , where each T i = { x ↦→ T ( x ) i : T ∈ T } .

Proof. Every function h ◦ T ∈ H ◦ T can be viewed as x ↦→ h ( T ( x ) 1 , . . . , T ( x ) d ) , which is a composition of h with d Boolean functions T ( · ) 1 , . . . , T ( · ) d : X → { 0 , 1 } where each T ( · ) i is the restriction of transformation T to the i th coordinate. The claim then follows from a direct application of Proposition 4.9 in Alon et al. [2023], which itself generalizes a classical result due to Blumer et al. [1989b] bounding the VC dimension of composed function classes.

C Proofs for Section 2

Proof of Theorem 2.2. We note that the proof follows a standard no-free-lunch argument for VC classes, where the game will be to guess the support of a distribution.

Let x 1 , . . . , x 3 k be arbitrary points. For each P ⊆ [3 k ] where | P | = k , define a transformation T P that maps x 1 , . . . , x 3 k to some new and unique points T P ( x 1 ) , . . . , T P ( x 3 k ) . Then, define classifier h P to be positive everywhere, except on the points { T P ( x i ) } i ∈ P which are labeled negative. Let X = { x 1 , . . . , x 3 k } ⋃ P { T P ( x 1 ) , . . . , T P ( x 3 k ) } , H = { h P : P ⊆ [3 k ] , | P | = k } , and T = { T P : P ⊆ [3 k ] , | P | = k } .

It is easy to see that vc( H ) = 1 , since classifiers in H operate in different parts of X . Furthermore, vc( H ◦ T ) ≥ k where we can shatter x 1 , . . . , x k with H ◦ T as follows: for each y 1 , . . . , y k , let I = { i ∈ [ k ] : y i = -1 } and P = I ∪ { j : k +1 ≤ j ≤ 2 k -| I |} , then ( h P ◦ T P )( x i ) = h P ( T P ( x i )) = y i for all i ∈ [ k ] .

Consider now a family of distributions {D P : P ⊆ [3 k ] , | P | = k } over X × Y where each D P is uniform over 2 k points { ( x i , +1) } i/ ∈ P . For each P ⊆ [3 k ] where | P | = k , observe that by definitions of D P , H , T , sup T ∈T err( h P , T ( D P )) = 0 since h P only labels the points { T P ( x i ) } i ∈ P negative and { x i } i ∈ P are not in the support of D P . That is to say, our lower bound holds in the realizable setting where OPT ∞ = 0 (see Equation (1)).

Next, consider an arbitrary proper learning rule A : ( X × Y ) ∗ →H . For a distribution D P chosen uniformly at random and upon receiving a random sample S ∼ D k P , A needs to correctly guess which points from { x 1 , . . . , x 3 k } \ S lie in the support of D P in order to choose an appropriate h ∈ H with small error. However, since the support is chosen uniformly at random, A will most likely incorrectly guess a constant fraction of the support, leading to a constant error. This is a standard argument [see e.g., Montasser et al., 2019, Alon et al., 2021], but we repeat it below for completeness.

Fix an arbitrary sequence S ∈ { ( x 1 , +1) , . . . , ( x 3 k , +1) } k . Denote by E S the event that S ∈ supp( D P ) for a distribution D P that is picked uniformly at random. Next,

̸

̸

where the last inequality follows from the fact that A ( S ) ∈ H and that the remaining (at least k ) points that are not in S but in supp( D P ) are chosen uniformly at random, because D P is chosen randomly. From the above, by law of total expectation, we have

By the probabilistic method, this means there exists P ∗ such that E S ∼D k P ∗ [sup T ∈T err( A ( S ) , T ( D P ))] ≥ 1 4 . Using a variant of Markov's inequality, this implies that Pr S ∼D k P ∗ [ sup T ∈T err( A ( S ) , T ( D P )) > 1 8 ] ≥ 1 7 .

D Proofs for Section 3

Proposition D.1. For any class H , any ERM for H , any collection of transformations T , any distribution D such that OPT ∞ = 0 , and any ε, δ ∈ (0 , 1 / 2 ) , with probability at least 1 -δ over S ∼ D m ( ε,δ ) , where m ( ε, δ ) = O ( vc( H◦T ) log(1 /ε )+log(1 /δ ) ε ) ,

where ˆ h is the output predictor of running ERM on the inflated dataset T ( S ) = { ( T ( x ) , y ) : ( x, y ) ∈ S ∧ T ∈ T } .

Proof of Proposition D.1. Since OPT ∞ = 0 , i.e., there exists h ⋆ ∈ H such that ∀ T ∈ T : err( h ⋆ , T ( D )) = 0 , and since ˆ h is the output predictor of running ERM on the inflated dataset T ( S ) = { ( T ( x ) , y ) : ( x, y ) ∈ S ∧ T ∈ T } , it follows that ∀ T ∈ T : err( ˆ h, T ( S )) = 0 . Thus, by invoking the optimistic generalization guarantee (Proposition A.2), with probability at least 1 -δ over S ∼ D m ( ε,δ ) : ( ∀ h ∈ H )( ∀ T ∈ T ) : err( h, T ( S )) ⇒ err( h, T ( D )) ≤ ε . Since ˆ h ∈ H , it follows that ∀ T ∈ T : err( ˆ h, T ( D )) ≤ ε .

Lemma D.2. Let S = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } be an arbitrary dataset. For any distribution Q over T and any predictor h , define the loss function ℓ S ( h, Q ) = 1 -E T ∼ Q err( h, T ( S )) . Then for any sequence of predictors h 1 , . . . , h R , running Multiplicative Weights with η = √ 8 ln |T | /R (see Algorithm 1) produces a sequence of distributions Q 1 , . . . , Q R over T that satisfy

Proof of Lemma D.2. The proof follows directly from considering a two-player game, where the T -player plays mixed strategies (distributions over T ) Q 1 , . . . , Q R against predictors h 1 , . . . , h R played by an arbitrary learning algorithm A , and in each round the T -player incurs loss ℓ S ( h r , Q r ) = 1 -E T ∼ Q r err( h r , T ( S )) . Then, the regret guarantee of Multiplicative Weights [see e.g., Theorem 2.2 in Cesa-Bianchi and Lugosi, 2006] implies that

E Proofs for Section 5

Proof of Theorem 5.1. The proof follows from the definition of ˆ h (Equation (8)) and using uniform convergence bounds (Proposition A.1). Let h ⋆ ∈ H be an a-priori fixed predictor (independent of sample S ) attaining

or is ε -close to it. By setting m ( ε, δ ) = m ( ε, δ ) = O ( vc( H◦T )+log(1 /δ ) ε 2 ) and invoking Proposition A.1, we have the guarantee that with probability at least 1 -δ over S ∼ D m ( ε,δ ) ,

Since ˆ h, h ⋆ ∈ H , the inequality above implies that

Furthermore, by definition, since ˆ h minimizes the empirical objective, it holds that

By combining the above, we get

This concludes the proof by definition of h ⋆ .

Similar to Section 3, we develop in this section a generic oracle-efficient reduction solving Objective 7 using only an ERM oracle for H . This reduction may be favorable in applications where we only have black-box access to an off-the-shelve supervised learning method. The techniques used are similar to those used in Section 3, and Agarwal and Zhang [2022] who developed a similar reduction when having access to a collection of importance weights instead of a collection of transformations (which is the view we propose in this work).

Theorem E.1. For any class H , collection of transformations T , distribution D and any ε, δ ∈ (0 , 1 / 2 ) , with probability at least 1 -δ over S ∼ D m ( ε,δ ) , where m ( ε, δ ) ≤ O ( vc( H◦T )+log(1 /δ ) ε 2 ) , running Algorithm 2 on S for R ≥ 8 ln |T | ε 2 rounds produces ¯ h = 1 R ∑ R r =1 h r s.t.

Algorithm 2: Reduction to Minimize Worst-Case Regret

Input: Black-box learner ERM H , dataset S = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } , and transformations T .

1 For each T ∈ T , run learner ERM H on T ( S ) and denote its output by ˆ h T .

2 For each T ∈ T , set Q 1 ( T ) = 1 |T | .

3 Set R = 8 ln |T | ε 2 .

4 for 1 ≤ r ≤ R do

5 Draw m ERM i.i.d. samples ( X 1 , Y 1 ) , . . . , ( X m ERM , Y m ERM ) , where each ( X i , Y i ) is drawn by randomly drawing a transformation T according to Q t and randomly drawing ( X,Y ) from Unif( S ) , and letting ( X i , Y i ) = ( T ( X ) , Y ) .

6 Run learner ERM H on ( X 1 , Y 1 ) , . . . , ( X m ERM , Y m ERM ) , and let h r denote its output.

7 For each T ∈ T , update Q r +1 ( T ) = Q r ( T ) exp ( η (err( h r ,T ( S )) -err( ˆ h T ,T ( S ))) ) Z r , where Z r is a normalization factor such that Q r +1 is a distribution.

Output: 1 R ∑ R r =1 h r .

Proof of Theorem E.1. Let S = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } be an arbitrary dataset, and let A be an ( ε, δ ) -agnostic-PAC-learner for H . Then, by setting R ≥ 8 ln |T | ε 2 and invoking Lemma D.2, we are guaranteed that Algorithm 2 produces a sequence of distributions Q 1 , . . . , Q T over T that satisfy

At each round r , observe that Step 3 in Algorithm 1 draws an i.i.d. sample from a distribution P r over X × Y that is defined by Q r over T and Unif( S ) , and since ERM H is an ( ε, δ ) -agnostic-PAC-learner for H , Steps 5-6 guarantee that

̸

Finally, by appealing to uniform convergence over H and T , with probability at least 1 -δ over S ∼ D m , we have

Thus,

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The problem setup and the main theoretical results of the paper are described in the introduction, under 'Our Contributions'. They match the scope and the claims made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our results immediately after the statements or at the beginning of the next section. E.g., for the limitations of the algorithm proposed in Section 2, we state its limitation immediately after Theorem 2.2 (i.e., the sample complexity cannot be improved by such proper learning rules) and at the beginning of Section 3 (i.e., the algorithm requires knowledge of H and T ). We also mention the limitation of Algorithm 1in the introduction, i.e., the algorithm is limited to finite transformations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theoretical results clearly state the required assumptions. Full proofs are provided either in the main text or the supplemental material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a full description of the simple experiment that we perfomed in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data used in our experiment is synthetic and we provide a full description of how to generate it in Section 6.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All is specified in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Reported in Section 6 and Figure 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We see no violation of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper contributes theoretical/foundational results addressing the problem of statistical learning under distribution shifts. Our work will potentially lead to further theoretical study, as well as practical methodological development with direct impact on downstream applications. There are many potential societal consequences of our work, none of which we think must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study a new framework for learning under distribution shift caused by a family of `invariance maps’ $\\mathscr{T}: X \\to X$. Informally, one may think of this as generalizing e.g. settings such as learning under invariance to rotation, where the maps may be rotation matrices and the learner should perform well at test time regardless of whether the raw data has been rotated.\n\nMore formally, the learner is given a hypothesis class $H$, a family of maps $\\mathscr{T}$, and sample access to a distribution $D$ over $X \\times {0,1}$, and wishes to train a classifier that minimizes the test error over all possible shifts $T(D)$ for $T \\in \\mathscr{T}$, either globally in the sense of outputting $h$ satisfying\n\n$$sup_T \\ \\{ err(h,T(D))\\} \\leq OPT + \\varepsilon$$\n\nWhere $OPT=min_{h_* \\in H} sup_T err(h^*,T(D))$, or where error is measured with respect to the best possible over each individual T (the authors call this regret minimizing):\n\n$$sup_T \\ (err(h,T(D))-OPT_T) \\leq min_{h^* \\in H} \\ (sup_T \\ (err(h^*,T(D))-OPT_T)) + \\varepsilon$$\n\nThis model captures a number of settings considered in the literature. In particular, under reasonable assumptions on distributions $D,D’$ over $X$, there is always a map $T$ such that $D=T(D’)$ (Breiman’s theorem), so the setup does not lose substantial generality over classical study of covariate shift. It may also model transformation invariances as mentioned above, as well as certain types of adversarial attacks (where the adversarial shift is chosen before the adversary sees the test example).\n\nThe main results concern the sample complexity of learning in this framework in the case were 1) H is known to the learner 2) H is unknown and the learner only has ERM access to H.\n\nIn the first setting, the authors show the sample complexity of (proper) agnostic PAC learning in this model is controlled by the VC dimension of the composed class VC(H \\circ T). In other words, it takes at least $VC(H \\circ T)$ samples to properly learn the class, and for any $k$ there exists a VC 1 class H and maps T with $VC(H \\circ T)=k$ which requires $\\Omega(k)$ samples to learn.\n\nTo complement their result, the authors also investigate $VC(H \\circ T)$ for a few special cases, including when $H$ is a composition $f \\circ W$ of an arbitrary linear map W and any fixed $f: X \\to \\{0,1\\}$ and $T$ is a family of linear maps (in which case $VC(H \\circ T) \\leq VC(H)$), some basic non-linear $T$ coming from neural net architectures, and transformations on the hypercube.\n\nWhen $H$ is unknown, the authors give a similar result in the realizable case using data augmentation, and a weaker variant for the agnostic setting for finite $\\mathscr{T}$ which ensures low error across a probabilistic mixture of hypotheses from H via an MWU type strategy.\n\nFinally the authors also consider a related setting in which the learner is in some sense allowed to ignore transforms with bad error, and the learner tries to return a hypothesis with low error across as many of the transforms $T \\in \\mathscr{T}$ as possible.\n\nAt a technical level, most of the results are the consequence of the basic duality of transforming input data and transforming the hypothesis class, allowing the authors to move between $T(D)$ and $H \\circ T$.\n\nSTRENGTHS:\nThe authors introduce a clean theoretical framework for various types of distribution shift and transformation invariance studied in the literature, capturing several interesting problems in the literature. The proposed framework should be of interest to the theory audience at NeurIPS, and is particularly nice in that it allows easy application of classical methods in VC theory to the problem.\n\nThe authors give a (weak) characterization of its statistical complexity in terms VC(H \\circ T), a natural combinatorial parameter of the concept and transformation class.\n\nThe authors discuss several natural scenarios in which their upper bound gives new sample complexity bounds on transformation invariant learning, including settings occurring in modern neural net architectures (namely the composition of linear transformations with activation functions).\n\nThe authors study several natural variants of the framework (global risk, regret, maximizing #transformations with low error) with similar corresponding upper bounds.\n\nWEAKNESSES:\nOn the other hand, beyond the main upper bound and examples given by the authors, many of the further results feel somewhat incomplete. For instance:\n\nThe lower bound based on VC(H \\circ T) is weaker than what one typically hopes for in a combinatorial characterization. Namely while the authors show there *exist* classes for which this bound is tight, a strong lower bound (as holds for standard learning) would show this is tight for *every* class. Is there a parameter characterizing the model in this stronger sense? The lower bound also only holds for proper learners.\n\nIn the setting where H is unknown, the use of data augmentation is a bit troubling in the sense that the size of the augmented sample could be extremely large (indeed infinite for infinite T). This isn’t captured by the model the authors introduce which simply assumes the ERM oracle can handle a call of any size, but this does not really seem to capture the computational cost of learning even at an oracle level. Furthermore, the agnostic guarantees in this setting only hold for finite $T$. POST REBUTTAL: The authors have included some discussion re: the first point that helps ameliorate the issue (e.g. using boosting one can lower the batch size to ERM at the cost of log(T) calls).\n\nCombined with the fact that the authors main techniques are fairly standard (note in some cases, as mentioned in the strengths, it is an *advantage* of the framework one can use standard techniques! But this is less the case when these techniques do not seem to give the desired results), these issues prevent me from a stronger recommendation of the paper.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a theoretical model of out-of-distribution (OOD) generalization, where the possible distribution shifts are encoded by a collection $\\mathcal{T}$ of transformations on the domain. The goal in this setting is given a distribution $D$ to learn a hypothesis $h$ from some hypothesis class $\\mathcal{H}$, such that $h$ achieves the best possible risk on the distribution $T(D)$ under the worst-case shift $T\\in\\mathcal{T}$. The paper gives algorithms achieving this goal in a variety of settings including when $\\mathcal{H}$ and $\\mathcal{T}$ are both known, and when given an ERM oracle for $\\mathcal{H}$ and $\\mathcal{T}$ is finite. The bounds are given in terms of the VC-dimension of the composition $\\mathcal{H}\\circ\\mathcal{T},$ consisting of all function of the form $h\\circ T$.\n\nSTRENGTHS:\n1. The model introduced of a class of transformations corresponding to distribution shifts gives very mathematically clean results for sample complexity in terms of the VC-dimension of the composed class $\\mathcal{H}\\circ\\mathcal{T}$.\n\n2. The explanations of how the model can be applied to different practical settings involving distribution shift are clear and show broad applicability.\n\nWEAKNESSES:\nThe primary weakness is in distinguishing these results from prior work, particularly that on multi-distribution learning. In particular,  when the class of transformations $\\mathcal{T}$ is finite, then every problem instance from this paper's model is also an instance of multi-distribution learning with $D_1,...,D_k$ given by $T_1(D),...,T_k(D)$. \n\nFurthermore, as pointed out on line 41, if the domain of the learning problem is $\\mathbb{R}^d$ then Brenier's Theorem implies that for any instance of a multi-distribution learning problem $D_1,...,D_k$ with sufficiently \"nice\" distributions, there is a distribution $D$ and set of $k$ transformation $T_1,\\dots,T_k$ such that $D_i = T_i(D)$. Thus, in the case of finite $\\mathcal{T}$ on domain $\\mathbb{R}^d$ these two models are equivalent so long as only \"nice\" distributions are considered. The setting of finite $\\mathcal{T}$ is particularly important, as it is the setting for which the main results in this paper give oracle-efficient algorithms (given an ERM oracle for $\\mathcal{H}$).\n\nThe above discussion leads to what may be an issue with Theorem 3.1 as stated. As far as I can tell, Theorem 3.1 combined with Remark 3.2 would imply a $O(d + \\log k)$ upper bound on sample complexity in the case that the VC-dimension of $\\mathcal{H}$ is $d$ and $|\\mathcal{T}|=k$. However, prior work on multi-distribution learning [1] suggests that in the setting of $k$ distributions $D_1,...,D_k$ and a hypothesis class of VC-dimension $d$ there is a lower-bound on sample complexity of $\\Omega(d + k\\log k)$. Perhaps I have missed something here, but I don't see any obstruction to instantiating the lower-bound family of instances from [1] as a family of instances in the model of this paper. So there seems to be some discrepancy between the known lower bounds and Theorem 3.1.\n\n[1] Haghtalab, Nika, Michael Jordan, and Eric Zhao. \"On-demand sampling: Learning optimally from multiple distributions.\" Advances in Neural Information Processing Systems 35 (2022): 406-419.\n\nQUESTIONS:\n1. Could you explain the main differences between this model and that of multi-distribution learning? In particular, are there some natural instances of multi-distribution learning which do not fall into this model?\n\n2. As described above under weaknesses, why do the lower bounds for multi-distribution learning not apply to the results of Theorem 3.1? Have I misunderstood something about the model in this paper, or perhaps the lower bound instances from [1]?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate transform-invariant binary classification problems in which the learner observes transformed features during the prediction phase and aims to construct a classifier robust against test-time transformations. Their first contribution is the derivation of a generalization error bound on the worst-case risk of transform-invariant classification, characterized by the VC dimension of the composite class. Subsequently, they propose an algorithm that achieves this generalization bound without direct access to the hypothesis class, relying solely on the empirical risk minimization (ERM) oracle associated with the hypothesis class. The second contribution is the establishment of a generalization error bound on the number of high-risk transformations, which is also characterized by the VC dimension of the composite class. The third result presents a generalization error bound on the worst-case regret, similarly characterized by the composite class's VC dimension. Empirical evaluations were conducted to demonstrate the behavior of the transform-invariant learning algorithm.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. It addresses a crucial topic for machine learning: robustness against transformations. This focus on developing classifiers that remain effective under various input transformations is highly relevant to real-world applications and aligns well with the conference themes.\n\nThe authors provide generalization error bounds for various accuracy measures, each corresponding to potential practical scenarios. This comprehensive approach enhances the applicability of their work.\n\nThe methodology to minimize worst-case accuracy solely relying on the ERM oracle is innovative. While it adapts the Multiplicative Weights method to the transform-invariant classification problem, the application of this method to this specific domain represents a novel contribution.\n\nWEAKNESSES:\nThe theoretical results appear to be direct applications of existing uniform bounds on empirical processes and VC dimension theory. The introduction of transform-invariance does not significantly impede the application of these uniform bounds, as they represent worst-case bounds for functions to be optimized. Specifically, Theorem 2.1 is derived by applying the original VC theory (by Vapnik and Chervonenkis) to the composite class. Theorem 4.1 is a straightforward consequence of the optimistic rate by Cortes et al. for the composite class. Theorem 5.1 is also obtained by directly applying the original VC theory to the composite class. Consequently, the technical novelty underlying the theoretical results is not clearly evident.\n\nThe authors do not sufficiently elucidate the implications of their experimental results. The significance and benefits of the empirical evaluation are not adequately explained, leaving the reader unclear about the practical impact of the proposed approach.\n\nQUESTIONS:\n- Can you provide more insight into the practical implications of your empirical evaluations? What specific benefits does your transform-invariant learning algorithm offer in real-world scenarios?\n\nLIMITATIONS:\nThe authors adequately address the limitations and potential impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work focuses on theoretical aspects of distribution shift, considering various frameworks that expand the standard PAC model of learning. \nThe paper considers a hypothesis class $\\mathcal{H}$ and a class of transformations of the domain $\\mathcal{T}$. The learning algorithm is given some data drawn i.i.d. from some distribution $D$ and the overall goal is to produce a classifier that does well not only on fresh samples from the source distribution $D$ but also when a transformation $T$ from $\\mathcal{T}$ is applied to $D$. \n\nOverall, the makes assumptions on the VC dimension of the class of functions obtained from composition of $\\mathcal{H}$ and $\\mathcal{T}$. In some parts of the paper it is further assumed that $\\mathcal{T}$ is finite. Various parts of the paper consider different loss functions that depend on the data distribution and the class of transformations $\\mathcal{T}$. The goal is to find a hypothesis $\\hat{h}$ for which the value of a loss function is not more by $\\epsilon$ greater than the best (i.e. smallest) value of the loss function among all functions in the hypothesis class $\\mathcal{H}$. (Note that this optimum value also depends on the choice of the loss function.)\n\nThe paper considers a number of specific goals in this broad setup:\n- In Section 2 the loss of a classifier $\\hat{h}$ equals to the worst-case population risk that $\\hat{h}$ will have as a result of applying a transformation $T$ from $\\mathcal{T}$ to the data distribution. Note that this loss function quantifies the worst-case impact that a transformation from $\\mathcal{T}$ can have on the population loss. Section 2 considers the classifier $\\hat{h}$ that has the smallest \n introduces and analyzes the classifier obtained by minimizing the empirical worst-case risk that a hypothesis will have as a result of applying a transformation $T$ from $\\mathcal{T}$ to the training dataset.\n- In Section 3 the class of transformations $\\mathcal{T}$ is assumed to be finite and the algorithm has access to an ERM oracle for the hypothesis class $\\mathcal{H}$. First, the realizable setting is considered (i.e. the setting in which there is some function in $\\mathcal{H}$ that perfectly desribes the data even if it is transformed by an arbitrary transformation in $\\mathcal{T}$). The paper shows that the classifier shown to exist in Section(2) can be obtained by (i) augmenting the training dataset by applying various transformations in $\\mathcal{T}$ and (ii) applying the ERM oracle to the resulting dataset.\n- In Section 3 the paper also considers the agnostic setting. In this setting there can potentially be  a transformation in $\\mathcal{T}$ that causes the empirical risk to be non-zero. The paper gives an efficient algorithm that combines calls to the ERM oracle with the multiplicative weight update method to obtain a classifier shown to exist in section 2. Note that the run-time of the algorithm does not include the time to actually minimize the empirical risk when the ERM oracle is called, which can potentially be slow.\n- Section 4 considers the case in which there is potentially no classifier whose error stays small when **any** transformation in $\\mathcal{T}$ is applied. Instead, Section 4 considers a loss function equal to the **number** of transformations in $\\mathcal{T}$ that make the population loss of the classifier greater then epsilon. The paper gives a statistical generalization bound with respect to this loss. \n- Section 5 again considers a different loss function. The loss of a hypothesis $\\mathcal{h}$ is defined as the worst-case difference (over transformation $T$ in $\\mathcal{T}$) between (i) the population risk of $\\mathcal{h}$ after transformation $T$ is applied (ii) the lowest population risk among all hypotheses in $\\mathcal{H}$. In other words, if there is some transformation $T$ makes the error of $\\mathcal{h}$ large, we do not penalize the algorithm if the worst-case error in the hypothesis class  $\\mathcal{H}$ is likewise large. With respect to this loss function as well, the paper again gives generalization bounds (similar to Sections 2 and 4) and an algorithmic reduction (Similar to Section 3).\n\nAdditionally, an experiment is presented in which a Boolean function is learned from synthetic data, and the function is invariant under a class of permutations.\n\nSTRENGTHS:\n- The question of mitigating distribution shift is central for modern machine learning.\n- The paper is written clearly and numerous examples are given.\n\nWEAKNESSES:\n- The technical results follow by a fairly direct use of standard generalization bounds, together with standard tools from convex optimization such as multiplicative weights.\n- One could argue that one of the most important use cases is when $\\mathcal{T}$ form a group of symmetry transformations. As discussed on page 9, in this setting the technique of data augmentation was studied previously. However, the previous work does not enjoy the worst-case error guarantees across transformations.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nLimitations are discussed adequately. However, I believe it would be helpful to additionally have a dedicated short section titled limitations that briefly summarizes the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper proposed a transformation-based invariant learning framework for OOD generalization. The authors initiated a theoretical study for this framework, investigating learning scenarios where the target class of transformations is either known or unknown. Upper bounds on the sample complexity in terms of the VC dimension of the class composing predictors with transformations were obtained.

STRENGTHS:
Nice theoretical results, with various scenarios considered.

WEAKNESSES:
When proposing a new learning rule, it is better to demonstrate its utility by comparing it to existing and similar ones, such as ERM (at least) and DRO (similar).

As the authors mentioned in Related Work, IRM is another popular learning framework for OOD generalization, but some works concluded that ERM may outperform IRM in some cases. Some other works showed that ERM may outperform DRO. Will the learning framework proposed in this paper be underperformed by ERM under some scenarios?

Consider a more realistic definition of $\mathcal{T}$: T(X)=X+R(X), as in the ResNet, because Q and P should not be too far away. In such a case, what will the learning rule, the theoretical guarantee, and the algorithm be like?

What's the connection between this work and the importance weighting for distributional shift (such as [1])?

[1] T. Fang, etc. Rethinking Importance Weighting for Deep Learning under Distribution Shift. NeurIPS 2020.

QUESTIONS:
Please see the questions mentioned above.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-23