Title: How Many Domains Suffice for Domain Generalization? A Tight Characterization via the Domain Shattering Dimension
Venue: NeurIPS 2025
Accepted: yes
Keywords: Domain generalization

PAC learning

sample complexity

shattering dimension

partial concepts
Abstract: We study a fundamental question of domain generalization: given a family of domains (i.e., data distributions), how many randomly sampled domains do we need to collect data from in order to learn a model that performs reasonably well on every seen and unseen domain in the family? We model this problem in the PAC framework and introduce a new combinatorial measure, which we call the domain shattering dimension. We show that this dimension characterizes the domain sample complexity. Furthermore, we establish a tight quantitative relationship between the domain shattering dimension and the classic VC dimension, demonstrating that every hypothesis class that is learnable in the standard PAC setting is also learnable in our setting.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how many domains are needed to train a model that generalizes well to unseen domains from a theoretical perspective? This problem is formulated as a PAC learning setting and using a new combinatorial complexity measure called the domain shattering dimension, extensions analogous to classic PAC learning are offered. The domain shattering dimension captures how complex a hypothesis class is in relation to a family of domains, and it determines the number of domains required to ensure generalization. The central insight is that while VC dimension explains learning on a single domain, domain sample complexity quantifies how many domains a learner must see to perform well on unseen domains. The domain shattering dimension measures how well a hypothesis class can separate domains with respect to error thresholds. The authors provide tight upper and lower bounds on domain sample complexity using this new dimension. They design a min-max ERM algorithm and prove that the number of required domains depends logarithmically on the domain shattering dimension. Moreover, they show this measure is tightly related to the VC dimension. The paper also explores connections to domain adaptation, showing that when domains are similar (as measured by a variant of H-divergence), the domain shattering dimension is small.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well written and can be followed without the need to check the details of proofs.\n\n- Introduction of domain shattering dimension for characterizing domain generalization sample complexity is novel and new.\n\n- The results and algorithms apply to binary classification, multi-class classification, and regression.\n\nWeaknesses:\n\n- The practical value of this work to the current trend in AI is unclear.\n\n- There is no empirical verification. \n\n- The domain shattering dimension is hard or even impossible to compute or estimate for practical deep networks, similar to how VC dimension is often intractable.\n\n- A core assumption is that the hypothesis class contains a single model that performs well across all domains. But in diverse domain distributions. this assumption is probably wrong.\n\n-\n\nQUESTIONS:\n1. While this work might be an interesting work for a venue like COLT, it will not be very interesting in the absence of any empirical validation for the general audience of NeurIPS. Why there is not empirical experiments to validate the theoretical results or at least show their applicability in some cases?\n\n2. What are the benefits of this work and the results for modern ML models?\n\n3. The theory assumes the hypothesis class H is given and contains a suitable hypothesis, but how can this be verified in practice?\n\nLIMITATIONS:\nI would like to highlight that while I am familiar with the broad concepts of PAC-learnability and understand the paper, theoretical ML is not my research area and it is not something that I work about on a day to day basis. For this reason, my confidence in rating  is low because it is challenging for me to judge the novelty level of this work from theoretical perspective. I hope the rest of reviewers can compensate for it. Despite this shortcoming, I am quite familiar with empirical aspects domain generalization and domain adaptation.\n\n1. Similar to PAC-learning, this work provides guarantees when the hypothesis class has finite domain sample complexity. Deep neural networks, however, typically have extremely large or even infinite VC dimension, making the standard PAC bounds uninformative in this context. As a result, the sample complexity guarantees become unhelpful.\n\n2.  The theory is built around ERM. In deep learning, however, practical optimization cannot search the entire hypothesis space and often only finds local minima, not the global one required by ERM. This means the theoretical guarantees do not directly translate to the way deep learning models are actually trained.\n\n3. While deep learning models can be trained efficiently in practice, the theoretical guarantees about computational tractability provided by PAC-learning are not tight or meaningful for modern neural architectures.\n\n4. The classic PAC model assumes the data can be generated by some function within the hypothesis class which is rarely true for deep learning.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an important problem in domain generalization(DG): how many randomly sampled domains are needed to perform well on unseen domains? To this end, the papers develops a new measure, domain shattering dimension and also establish a relationship between this dimension and the classic VC dimension.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses an important problem that is highly relevant in practical scenarios.\n\n2. It is always valuable to see theory in DG: the paper develops a new combinatorial measure characterizing domain sample complexity and establishes its relationship with VC dimension.\n\nWeaknesses:\n\n1. Weak motivation: In Lines 71–74, the paper claims that for a different threshold \\tau' the fat shattering dimension grows large, but provides no detailed justification for this assertion.\n\n2. Missing practical experiment: Seminal theoretical papers in domain adaptation and domain generalization (Ben-David et al., 2010; Blanchard et al., 2011) always accompany their theory with a practical experiment demonstrating its usefulness. This paper provides no such experiment.\n\n3. The section 6, where the paper connects the developed domain shattering dimension to domain adaptation is not explained clearly. In lines 312-314, it is mentioned that the domain shattering dimension upper bounds the error. But in the paper it only establishes the relationship between relaxed H-divergence and H-divergence in Line 321. It is not clear if this relaxed H-divergence can upper bound the target error? Also in Ben-David et al. [2010] the upper bound that is finally developed uses H \\delta H divergence. Will a similar relaxed version of H \\delta H divergence exist?\n\n4. There are several typographical errors: Lines 143, 213, 313 (it should be domain adaptation and not domain generalization), 81 (before this line there is no reference or explanation about what is partial concept class) and 165 (\\gamma is not defined)\n\nQUESTIONS:\nThe following are the questions/suggestions:\n\n1. The motivation lines in 71-74 need detailed justification.\n\n2. It is not sure how will the developed theory be used in practical scenarios? Atleast a toy experiment is needed to establish the significance of the developed theory.\n\n3. The connection to domain adaptation section is not well explained. How will the relaxed H-divergence upper bound the target error? Can the same idea be extended to H \\delta H divergence? This divergence is actually used in the final upper bound developed in Ben-David et. al [2010]\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles a \"meta-learning\" problem of understanding domain generalization in the same vein as how sample complexities are derived in PAC learning to give statistical guarantees. In this setting, there exists a family of domains $\\mathcal{G}$, where every domain is a distribution on instance-label pairs, and a concept class $\\mathcal{H}$ and the goal is to characterize the number of domains needed to be observed in the training data such that it is possible to select an $h \\in H$ that yields low error on unseen domains. The idea is that there is a distribution $\\mathcal{P}$ over $\\mathcal{G}$ and using PAC-like statistical guarantees, to produce a classifier $h$, with high probability, that has low expected error on all domains in $\\mathcal{G}$ with respect to $\\mathcal{P}$.\n\nThe authors construct a new combinatorial quantity called the domain shattering dimension that is defined in somewhat similar fashion to the fat shattering dimension. It essentially measures the \"capacity\" of $\\mathcal{G}$ in the sense that it captures the largest set $S \\subseteq \\mathcal{G}$ such that for any subset $S' \\subseteq S$, there exists a classifier $h$ with error below a certain threshold on domains in $S'$ and error above a threshold for all domains in $S \\setminus S'$. The authors use the domain shattering dimension to construct both upper and lower bounds on the number of domains needed to achieve successful domain generalization and also explore connections with the VC-dimension.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles a new problem of domain generalization that hasn't been explored before. \n- The paper is quite well-written and the technical details of their constructions are accessible.\n- For the upper bound, the technique of casting the domain learnability problem as a partial concept class and then using machinery established by Alon et al. to produce a uniform convergence guarantee is quite clever. \n\nWeaknesses:\n- My primary concern is the way the lower bound is constructed to show that the sample complexity using the domain shattering dimension is indeed tight up to logarithmic factors. As explained in the first paragraph of Section 4.2, the domain shattering dimension doesn't necessarily provide a lower bound for the domain sample complexity in all cases. For the specific mentioned by authors in Section 4.2, it seems that even the domain sample complexity generated from the domain shattering dimension provides a pretty loose upper bound (i.e. much larger than the true domain sample complexity). Generally, one tries to show the validity of a combinatorial quantity by showing that it measures the hardness of a learning instance through lower and upper bounds that work in the same exact learning setting. Here, the family $\\mathcal{G}$ is augmented to the family of $\\mathcal{G}'$ to prevent certain degenerate cases and the lower bound is on this $\\mathcal{G}'$. Altering the learning setting for the lower bound seems a bit concerning because the lower bound no longer certifies the hardness of the original problem instance.\n\nQUESTIONS:\n- If my understanding is correct, is it true that the VC dimension of the partial concept class generated from $\\mathcal{G}$ is equivalent to the domain shattering dimension?\n- This is a follow-up to my point in the weakness section. Is there a somewhat altered notion of the domain shattering dimension that additionally takes into account the structure of domains in $\\mathcal{G}$ with respect to the instance space? To me (I could be wrong), it seems to make sense that the domain sample complexity can also be tied to the inherent structure of the domains itself because one could leverage this information in learning regardless of the different domains seen by the learner.\n- In $(\\tau, \\alpha, \\gamma, \\delta)$-domain learnability, is there a size on what $m$ is required to be? Is the size of $m$ inferred from VC theory on the number of samples needed to obtain uniform convergence guarantees on $\\mathcal{H}$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies domain adaptation, focusing on quantifying how many domains are sufficient to learn a model that performs well across both seen and unseen domains within a family. Under the assumption that there exists an estimator that achieves low error across all domains, the authors derive upper and lower bounds on the domain sample complexity, introducing a new combinatorial quantity, the \"domain shattering dimension.\" The upper bound is established on Min-Max ERM. \nInterestingly, the authors also construct counterexamples demonstrating that this bound is not instance-optimal, indicating that further improvements may be possible in specific settings. Additionally, the paper draws connections between the proposed framework and existing measures in domain adaptation, such as the H-divergence.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a serious theoretical contribution. The authors formalize the domain adaptation problem in a rather general setting—without assumptions on domain similarity—and provide tight upper and lower bounds applicable to arbitrary hypothesis classes and classes of domains. \nThe results connect the new notion of domain shattering dimension with established concepts from the literature, offering a unified perspective on domain generalization and adaptation.\nOne potential drawback is that, in practice, it can be challenging to clearly specify or identify the class of domains relevant to a given problem. Computing the domain shattering dimension would then be difficult.\n\nQUESTIONS:\nPersonally, I would have also been interested in connections to the underlying data generating process, e.g., whether causal assumptions could help (sparse mechanism shift; independent causal mechanisms) - cf. https://arxiv.org/abs/1206.6471, https://proceedings.mlr.press/v48/gong16.html, https://ojs.aaai.org/index.php/AAAI/article/view/9542. But don't take this as a criticism of the paper.\n\nLIMITATIONS:\nLimitations are discussed in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyses the domain sample complexity—that is, how many domains must be randomly sampled to learn a model that performs reliably on both seen and unseen domains from a given family—within the PAC learning framework. Specifically, the authors derive both upper and lower bounds on the domain sample complexity using a novel combinatorial measure they introduce, called the domain shattering dimension. Furthermore, the authors establish a tight quantitative relationship between the domain shattering dimension and the classical VC dimension. This connection implies that any hypothesis class that is learnable under the standard PAC framework is also learnable under their proposed setting.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\nThe authors derive both upper and lower bounds on domain sample complexity using a combinatorial measure they introduce, termed the domain shattering dimension, which appears to offer some degree of novelty, although its practical or theoretical implications are not immediately clear.\n\n## Weaknesses:\n\n- The paper does not provide any clear practical or theoretical implications for the domain generalization (DG) problem. In DG, we typically assume access to a limited number of source domains, and the key challenge is how to leverage these to generalize to unseen target domains. A more relevant formulation would focus on to what extent a model trained on a small number of domains can generalize well. In contrast, the question this paper raises—how many domains are needed to guarantee generalization to any unseen domain—feels somewhat unrealistic and disconnected from the practical challenges in DG. This formulation does not appear to offer useful insights for designing algorithms for DG.\n- Moreover, the authors seem to treat domains as individual samples and then define a new loss function at the domain level, comparing it to 0/1 loss on samples via a threshold $\\tau$. They then apply sample complexity bounds analogously at the domain level. However, this domain complexity upper bound does not seem to provide any additional insight beyond standard sample complexity bounds in the context of DG.\n- The paper states: \"domain sample complexity can still be very small if the domains in \\mathcal{G} are very similar.\" However, domain similarity is difficult to quantify, and even if we had a metric, it would only apply to the observed training domains. This observation is only meaningful if the number of training domains is large and diverse enough to represent a meaningful meta-distribution. If the training domains are limited and highly similar, it's unclear what conclusion we can draw—do we need more diverse training domains, or have we already captured enough information?\n- Regarding the connection to domain adaptation, the authors suggest defining a new divergence measure. However, it is unclear what the benefit of this new divergence is, or how it contributes to our understanding or improvement of domain adaptation techniques. This section lacks a clear motivation and purpose.\n\nQUESTIONS:\nPlease see the strengths and weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-06"}