Question: What is the relationship of this work to prior theory [1,2]? [1] Towards a Theoretical Framework of Out-of-Distribution Generalization [2] Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
Evidence: Theoretical Analysis of the DG Setting and Algorithms The DG problem setting was first analysed in. Since then there have been some attempts to analyse DG algorithms from a generalisation bound perspective. However these studies have theoretical results that are either restricted to specific model classes, such as kernel machines, or make strong assumptions about how the domains seen during training will resemble those seen at test time-e.g., that all domains are convex combinations of a finite pre-determined set of prototypical domains. In contrast, our Rademacher complexity approach can be applied to a broad range of model classes (including neural networks), and makes comparatively milder assumptions about the relationship between domains-i.e., they are i.i.d. samples from another arbitrary distribution over domains.
The majority of the existing work investigating the theoretical foundations of DG follow the initial formalisation of the domain generalisation problem put forth by, where the goal is to minimise the expected error over unseen domains. However, several recent works have also explored the idea of bounding the error on a single unseen domain with the most pathological distribution shift. This type of analysis is typically rooted in methods from causal inference, rather than statistical learning theory. As a consequence, they are able to make stronger claims for the problems they address, but the scope of their analysis is necessarily limited to the scenarios where their assumptions about the underlying causal structures are valid. For example, Janzing (2019) provides bounds that assume problems conform to a specific class of structural equation models, and the analysis is performed under the assumption that infinite training data is available within each of the observed training domains. Throughout the work we address the stan-dard DG formalisation given by, where one is concerned with the expected performance of a model on domains sampled from some distribution over domains.
Others rely on trying to link between domain adaptation objectives (where target domains are observable for alignment to source domains) and domain generalisation (where target domains are not observable and thus cannot correctly be used in a learning objective). proceed by making assumptions on the structure of the distribution over possible domains (i.e., that it has support determined by the convex hull of a finite set of prototypical domains), which allows them to upper bound the domain alignment metric. provide a bound that depends on an unobservable domain distance quantity, which they then approximate in experiments using kernel density estimates. is another piece of work that theoretically investigates the generalisation of ERM in a DG setting. They deal with online DG, where each time-step corresponds to observing a new domain, and the learner must produce a new model capable of generalising to novel domains. Another point of difference between their work and the standard DG problem setting of is that the domain at each time-step is chosen by an adversary. They analyse this game for a finite number of time-steps, but they assume each domain has an infinite amount of data. They also put some limitations on the adversary: e.g., it must choose a domain that is a convex combination of a finite number of pre-determined domains. In contrast, our theoretical analysis is in the more realistic setting where one has a finite amount of data per domain, and the domains we consider are not limited to convex combinations of a set of prototypical domains. Possibly the most similar work to our theoretical contributions is due to, who also provide learningtheoretic generalisation bounds for DG. However, their analysis only applies to finite hypothesis classes (which does not include, e.g., linear models or neural networks), whereas ours can be applied to any class amenable to analysis with Rademacher complexity.
The main existing empirical analysis on DG is, who compared several state of the art DG methods under a common evaluation and hyper-parameter tuning protocol called DomainBed. They ultimately defend Empirical Risk Minimization (ERM) over more sophisticated alternatives on the grounds that no competitor consistently beats it across the benchmark suite. We also broadly defend ERM, and build on the same benchmark, but differently we provide a much deeper analysis into when and why ERM works. More specifically: (i) We provide a new theoretical analysis of ERM's generalisation quality unlike the prior purely empirical evaluation, (ii) We re-use the DomainBed benchmark to directly corroborate this theory under controlled conditions using linear models where model complexity can be tractably and accurately tuned. (iii) We use our complexity-based analysis to explain the previously erratic results of prior DomainBed competitors in terms of model complexity. (iv) We identify, and empirically validate, the preferred model selection criterion for DG, a point which was inconclusive in.
Theoretical Analysis of the DG Setting and Algorithms The DG problem setting was first analysed in. Since then there have been some attempts to analyse DG algorithms from a generalisation bound perspective. However these studies have theoretical results that are either restricted to specific model classes, such as kernel machines, or make strong assumptions about how the domains seen during training will resemble those seen at test time-e.g., that all domains are convex combinations of a finite pre-determined set of prototypical domains. In contrast, our Rademacher complexity approach can be applied to a broad range of model classes (including neural networks), and makes comparatively milder assumptions about the relationship between domains-i.e., they are i.i.d. samples from another arbitrary distribution over domains.
The majority of the existing work investigating the theoretical foundations of DG follow the initial formalisation of the domain generalisation problem put forth by, where the goal is to minimise the expected error over unseen domains. However, several recent works have also explored the idea of bounding the error on a single unseen domain with the most pathological distribution shift. This type of analysis is typically rooted in methods from causal inference, rather than statistical learning theory. As a consequence, they are able to make stronger claims for the problems they address, but the scope of their analysis is necessarily limited to the scenarios where their assumptions about the underlying causal structures are valid. For example, Janzing (2019) provides bounds that assume problems conform to a specific class of structural equation models, and the analysis is performed under the assumption that infinite training data is available within each of the observed training domains. Throughout the work we address the stan-dard DG formalisation given by, where one is concerned with the expected performance of a model on domains sampled from some distribution over domains.
Others rely on trying to link between domain adaptation objectives (where target domains are observable for alignment to source domains) and domain generalisation (where target domains are not observable and thus cannot correctly be used in a learning objective). proceed by making assumptions on the structure of the distribution over possible domains (i.e., that it has support determined by the convex hull of a finite set of prototypical domains), which allows them to upper bound the domain alignment metric. provide a bound that depends on an unobservable domain distance quantity, which they then approximate in experiments using kernel density estimates. is another piece of work that theoretically investigates the generalisation of ERM in a DG setting. They deal with online DG, where each time-step corresponds to observing a new domain, and the learner must produce a new model capable of generalising to novel domains. Another point of difference between their work and the standard DG problem setting of is that the domain at each time-step is chosen by an adversary. They analyse this game for a finite number of time-steps, but they assume each domain has an infinite amount of data. They also put some limitations on the adversary: e.g., it must choose a domain that is a convex combination of a finite number of pre-determined domains. In contrast, our theoretical analysis is in the more realistic setting where one has a finite amount of data per domain, and the domains we consider are not limited to convex combinations of a set of prototypical domains. Possibly the most similar work to our theoretical contributions is due to, who also provide learningtheoretic generalisation bounds for DG. However, their analysis only applies to finite hypothesis classes (which does not include, e.g., linear models or neural networks), whereas ours can be applied to any class amenable to analysis with Rademacher complexity.
The main existing empirical analysis on DG is, who compared several state of the art DG methods under a common evaluation and hyper-parameter tuning protocol called DomainBed. They ultimately defend Empirical Risk Minimization (ERM) over more sophisticated alternatives on the grounds that no competitor consistently beats it across the benchmark suite. We also broadly defend ERM, and build on the same benchmark, but differently we provide a much deeper analysis into when and why ERM works. More specifically: (i) We provide a new theoretical analysis of ERM's generalisation quality unlike the prior purely empirical evaluation, (ii) We re-use the DomainBed benchmark to directly corroborate this theory under controlled conditions using linear models where model complexity can be tractably and accurately tuned. (iii) We use our complexity-based analysis to explain the previously erratic results of prior DomainBed competitors in terms of model complexity. (iv) We identify, and empirically validate, the preferred model selection criterion for DG, a point which was inconclusive in.
Ye et al. (2021) provide a bound that depends on an unobservable domain distance quantity, which they then approximate in experiments using kernel density estimates.****Possibly the most similar work to our theoretical contributions is due to Ahuja et al. (2021), who also provide learningtheoretic generalisation bounds for DG. However, their analysis only applies to finite hypothesis classes (which does not include, e.g., linear models or neural networks), whereas ours can be applied to any class amenable to analysis with Rademacher complexity.
Answer: [1] Does not have a sample complexity generalisation bound which we provide here. They analyse generalisation in terms of distribution distance, but this is not observable. In their experiments they approximate it with kernel density estimation, but the quality of their approximation is not well-justified. (Note also that our theory does not depend on the distance between the source and target distributions.) [2] Provides sample complexity bounds but they only apply to finite hypothesis classes. This means that it is not applicable to real-world model classes. In contrast, we are able to analyse models such as linear classifiers and neural networks with our Rademacher complexity based theory. We have added discussion of these papers to our related work section.