Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapolation

## Abstract

Abstract Out-of-distribution (OOD) generalization is a favorable yet challenging property for deep neural networks. The core challenges lie in the limited availability of source domains that help models learn an invariant representation from the spurious features. Various domain augmentation have been proposed but largely rely on interpolating existing domains and frequently face difficulties in creating truly “novel” domains. Humans, on the other hand, is capable of extrapolating novel domains, thus, an intriguing question arises: How can neural networks extrapolate truly “novel” domains and achieve OOD generalization? We introduce a novel approach to domain extrapolation that leverages reasoning ability and the extensive knowledge encapsulated within large language models (LLMs) to synthesize entirely new domains. Starting with the class of interest, we query the LLMs to extract relevant knowledge for these novel domains. We then bridge the gap between the text-centric knowledge derived from LLMs and the pixel input space of the model using text-to-image generation techniques. By augmenting the training set of domain generalization datasets with high-fidelity, photo-realistic images of these new domains, we achieve significant improvements over all existing methods, as demonstrated in both single and multi-domain generalization across various benchmarks. With the ability to extrapolate any domains for any class, our method has the potential to learn a generalized model for any task without any data. To illustrate, we put forth a much more difficult setting termed, data-free domain generalization , that aims to learn a generalized model in the absence of any collected data. Our empirical findings support the above argument and our methods exhibit commendable performance in this setting, approximating the supervised with synthetic data only and even surpassing the supervised setting by approximately 1-2% on datasets such as VLCS. † † Preprint. Under review. † † We will release our code upon acceptance. † † Email: yli556@jhu.edu

###### Keywords:

## 1 Introduction

Deep neural networks have demonstrated remarkable achievements in various fields and applications [, , , , ], yet their effectiveness heavily depends on the assumption that the training and testing environments are subject to independent and identically distributions [, ].
Out-of-distribution (OOD) generalization aims to learn model from some training distribution that can generalize well to unseen testing domains, usually with distribution or label shifts []. A typical scenario is domain generalization (DG) where multiple source domains are available and these available source domains aid the training of generalizable models that learn invariant features and discard spurious ones.
However, a significant challenge arises. The availability of these source domains often becomes a limiting factor, hindering the success of current OOD approaches in more challenging scenarios [, , , ], which can be attributed to the difficulty and high expenses to collect, not just, data but data in diverse domains with annotations, which is sometimes impossible in critical areas such as healthcare or extreme conditions (e.g. deep sea or space).
Motivated by these challenges, domain augmentation is straightforward and multiple methods have been proposed to generate novel domains and images through mixup [], mixing of statistics [], uncertainty modeling [, ] and convex combination []. However, these methods generally interpolate the existing training domains to generate novel domains that still fall within the convex hall of available domains []. Consequently, the constrained number of source domains hampers the expressiveness of these methods, continuing to act as a performance bottleneck. On the other hand, Humans harness the innate ability of the human brain to create novel domains as illustrated in [, ] where a pre-defined set of novel domains and styles are utilized. However, this also requires human labor which fails to scale to larger sizes. Naturally, an intriguing question arises: How can neural networks extrapolate truly “novel” domains and achieve OOD generalization?

Large language models (LLMs) [] have been shown to encapsulate a vast wealth of knowledge and simulate human cognitive processes.
Thus, a pertinent question emerges: Can one harness the power of LLMs to produce novel domains and relevant knowledge, thereby replacing the human in the above training process?
Stemming from this primary query, we investigate how we can extract knowledge of a specific task and produce novel domains from LLMs.
A subsequent research question is: How can we leverage this text-centric knowledge from LLMs to instruct an image system that processes pixel input?
State-of-the-art text-to-image generation models such as Imagen [], Stable Diffusion [] and GLIDE [] exhibit promosing capability to synthesize photo-realistic images positioning them as the optimal conduit between textual and visual realms. Finally, we seek to answer to what extent the synthesized images based on knowledge can serve as Out-of-distribution learners that can generalize to unseen testing domains. Following these problems, we are the first study to design a new paradigm that leverages the knowledge of LLMs to extrapolate novel domains for training better generalizable and sample-efficient models. With the ability to extrapolate any domains for any class, our method has the potential to learn a generalized model for any task without any existing data.

In addition, we present data-free domain generalization. Data-free generalization endeavors to enable a model across unseen testing domains based solely on task specifications (for example, distinguishing between dog and cat classes) without the need for gathering or utilizing any pre-existing datasets. In the era of large foundation models, data-free domain generalization is formulated as OOD problem with inaccessible meta distribution and domain distribution (detailed in Section 2.1) – essentially, devoid of any real-world data. This scenario presents a significantly more complex challenge than that encountered in multi-domain or single-domain generalization efforts. Moreover, it holds pragmatic significance in democratizing machine learning, by urging the community to develop methodologies that are viable under stringent resource constraints. Such an initiative paves the way for wider access to, and application of, machine learning technologies. For instance, our proposed method provides a viable solution by leveraging LLM as a knowledge base to extrapolate domains and scenarios domains and scenarios where specific classes may be represented. Then synthetic data is generated via a text-to-image generation model. These synthetic data only are used to train a model that can generalize well to the given task and fulfill its requirements. Our method not only addresses the challenge of data scarcity in DG problems but also underscores the potential of synthetic data in overcoming traditional barriers to machine learning implementation.

Extensive experiments on single, multi-domain and data-free evaluations demonstrate the effectiveness of our proposed method. In both single and multi-domain configurations, we demonstrate that synthetic data in the extrapolated novel domains markedly outperforms baseline results across various datasets. On the more challenging data-free setting, our proposed method exhibits near-supervised performance in this setting, even surpassing the supervised baseline by approximately 1-2% on VLCS. Data synthesized via the knowledge from LLMs excels compared to the synthetic data directly generated from text-to-image generation models. This demonstrates the ability of LLMs to effectively extrapolate like humans and integrate this prior knowledge into the model.

We also underscore the scalability of our approach by highlighting that as the number of domains escalates, the performance correspondingly improves. Intriguingly, this trend diverges from the outcomes observed when augmenting with synthetic data directly produced by text-to-image models reported in [, ]. This further demonstrates the pivotal role of the knowledge derived from LLMs in mitigating overfitting to synthetic data.

The remainder of this paper is organized as follows: In Section 2, we will first motivate our method from the perspective of the theoretical error bound for out-of-distribution (OOD) generalization. Then we will detail our method design and specifications.
Section 3 introduces the data-free generalization and its potential usage in the era of large foundation models.
Section 4 describes our experiment design, results and the implications of our findings.
Section 5 introduces related work.
Section 6 concludes our paper and potential limitation of our work.

## 2 Method

We motivate our method from the perspective of the theoretical error bound for out-of-distribution (OOD) generalization. We will first provide the notation for the theoretical framework. Then we motivate our research problem from the OOD generalization error bound, i.e. limited number of source domains leading to a larger error bound. Then we propose a proxy method that approximates the meta-distribution with a proxy distribution. We give a new error bound on this method. Lastly, we propose one realization of our method by using LLMs to approximate the meta-distribution and text-to-image generation models to bridge the text-centric knowledge with the input pixel space.

### 2.1 Theoretical Bound

Notation. Let $\mathcal{X}$ denote the observation space and $\mathcal{Y}=\{1,-1\}$ the output space. Denote $P_{XY}$ as the joint probability of the joint space of $\mathcal{X}\times\mathcal{Y}$ and assume a meta distribution $\mu$ and n domains $P^{(1)}_{XY},\cdots,P^{(i)}_{XY},P^{(n)}_{XY}$ are i.i.d realizations from $\mu$. A decision function is a function $f\in\mathcal{F}:\mathcal{X}\rightarrow\mathcal{Y}$ predicts $\hat{y}_{i}=f(x_{i})$. We denote $\mathit{l}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}_{+}$ a loss function and define the generalization error of a decision function as

$$ $\displaystyle\small\mathcal{L}^{\mu}(f)=\mathbb{E}_{P_{XY}\sim\mu}\mathbb{E}_{ (x,y)\sim P_{XY}}[\mathit{l}(f(x),y)]$ (1) $$

Since we have no access to $\mu$ and all the realizations $P^{(1)}_{XY},\cdots,P^{(i)}_{XY},P^{(n)}_{XY}$ but sampled images from these realizations, we can derive an empirical error:

$$ $\displaystyle\small\hat{\mathcal{L}}^{\mu}(f)=\sum_{i=1}^{n}\sum_{j=1}^{m} \mathit{l}(f(x_{ij},y_{ij})$ (2) $$

where $(x_{ij},y_{ij})\sim P^{(j)}_{XY}$ denotes the $i$th sample drawn from $P^{(j)}_{XY}$.
It’s easy to see that when $n\rightarrow\infty,m\rightarrow\infty$, $\hat{\mathcal{L}}^{\mu}(f)$ converges to $\mathcal{L}^{\mu}(f)$, which gives the intuitive sense that increasing $m$ and $n$ gives us better-approximated solutions.
This motivates us to increase $n$ and $m$, i.e.increasing the number of domains and training images per domain, which is difficult due to the inaccessible $\mu$ and $P^{(1)}_{XY},\cdots,P^{(i)}_{XY},P^{(n)}_{XY}$. Prior arts have proposed various methods to generate novel domains but the majority falls in the interpolation of existing domains, failing to effectively increase $n$.
How can to approach this problem? We can approximate $\mu$ by new distribution $\mu^{\prime}$ sufficiently close to $\mu$ that can be sampled.

###### Definition 1

We define the distance between the two distributions as

$$ $\displaystyle\small D(\mu,\mu^{\prime})=\sup_{f\in\mathcal{F}}|\mathcal{L}^{ \mu^{\prime}}(f)-\mathcal{L}^{\mu}(f)|$ $$

With the following assumption,

###### Assumption 1

We assume the distance $D(\mu,\mu^{\prime})\leq\epsilon$.

we can derive a bound through the approximated $\mu^{\prime}$.

###### Theorem 2.1

With confidence at least $1-2\delta$ and for all $f\in\mathcal{F}$, we have

$$ $\displaystyle\small\mathcal{L}^{\mu}(f)\leq\hat{\mathcal{L}}^{\mu^{\prime}}(f) +2\mathcal{R}_{mn}(\mathcal{F})+2\mathcal{R}_{n}(\mathcal{F})+3\sqrt{\frac{\ln (2/\delta)}{2mn}}+3\sqrt{\frac{\ln(2/\delta)}{n}}+\epsilon$ $$

Proof in Appendix 0.A.
By replacing $\mu$ with $\mu^{\prime}$, we now have control over $\hat{\mathcal{L}}^{\mu^{\prime}}(f)$, $m$ and $n$ as we can sample as many domains and images from $\mu^{\prime}$ as possible. This is obtained at the cost of $\epsilon$, which we assume to be small.

###### Remark 1

We also note that as $n$ and $m$ increase, the upper bound of the generalization error decreases, which gives us better generalization errors.

With sufficiently large $n$ and $m$, the decrease part of the generalization error will cancel out the cost of $\epsilon$, leading to a lower generalization error.

Figure: Figure 1: Overall pipeline of our paradigm: Extrapolation of novel domains via the knowledge of LLMs, a novel learning paradigm where knowledge from LLMs assists the training of generalizable models via text-to-image models in a completely data-free fashion.
Refer to caption: x1.png

###### Definition 1

###### Assumption 1

###### Theorem 2.1

###### Remark 1

### 2.2 Domain Extrapolation with LLMs

Given the aforementioned theoretical bound, our objective is to approximate $\mu$ with $\mu^{\prime}$. Humans, as evidenced in [, ], usually can efficiently extrapolate novel domains (by imagination), which is a good approximation of $\mu$. Nonetheless, human intervention is expensive and not scalable to arbitrary datasets. Conversely, LLMs not only embody a vast expanse of knowledge [] and exhibit comparable reasoning capabilities [], but they also present the benefit of being amenable to extensive sampling. To this end, we propose to query LLMs, in place of human, to extrapolate novel domains.

After sampling from meta distribution $\mu^{\prime}$, we need to further sample from the domain distribution to generate images in this particular novel domain. As discussed in Section 1, this leads to a gap between the text-based knowledge output by the LLMs and the input pixel space of vision systems. Text-to-image generation models (e.g. stable diffusion []) exhibit the great capability to output photo-realistic images through inputting texts positioning them as the optimal bridge between textual and visual realms. The synthetic images of extrapolated novel domains are used to augment the original dataset or train the models solely in a data-free fashion. An overall illustration of our paradigm can be seen in Figure 1.

Extracting Knowledge from LLMs. The objective is to approximate $\mu$ via LLMs as close as possible. This introduces a constraint whereby the generated novel domains must reside within the high-density regions of distribution $\mu$. To ensure adherence to this criterion, we purposefully instruct the LLMs to conceive the most plausible and reasonable domains where a particular class would realistically exist. To better guide LLMs to understand the instruction and generate the required response accordingly, we craft system prompts that include role description ([Role]) and task description ([Task Description]), as illustrated by the example in Figure 2. Numerous strategies exist to solicit knowledge and novel domains from LLMs.

- •
Dataset-wise query. The most direct approach entails querying the LLMs with comprehensive dataset information (i.e. all of the class names) and instructing the model to produce $n$ novel domains. However, as the marginal distribution for each class might exhibit minimal overlap (worse when the number of classes grows), it becomes considerably intricate to sample novel domains that are both plausible and likely for all classes.
- •
Class-wise query. Thus, we propose to query the LLMs for novel domains of specific classes. For each class in the task, we query the LLMs for knowledge and $n$ novel domain information specific to that class. We repeat the process one class after another until all of the classes are iterated. We provide a example prompt in Figure 2.

Figure: Figure 2: Knowledge extraction pipeline. We first employ various SOTA prompting methods: e.g. "Chain of Thought []" (CoT) prompting, role prompting to extract domains from LLM (Step 1) and automatically generate prompt for a Text-to-Image model. (Step 2)
Refer to caption: x2.png

Bridging text and pixel with text-to-image generation models. After obtaining a number of the most plausible and reasonable domains of a specific class, we transform the text-centric knowledge from LLMs to pixel space by text-to-image generation models. This process is exactly the realization of sampling $X$ from $P_{X}^{(i)}$ where $P_{X}^{(i)}$ is the $i$th domain generated by $\mu^{\prime}$ (i.e. the LLM). Numerous strategies exist to prompt text-to-image generation models conditioned on class and domain information.

- •
Template prompt.
The most immediate strategy involves employing templates as prompts (e.g., "an image of [CLASS_NAME] in the domain of [DOMAIN_NAME]"). However, the limitation lies in its lack of diversity: utilizing the identical prompt to produce multiple images results in images bearing resemblance to one another.
- •
LLM generated prompt. Thus, we propose to query the LLMs for prompts conditioned on the class name and domain information acquired in the previous step. As illustrated in Figure 2, we craft system prompts that specifically tailor the LLM to generate prompts for text-to-image generation models and generate multiple prompts for each of the novel domains of each class.

## 3 Data-free Domain Generalization

We present Data-free Generalization, a new formation of generalization in the era of large foundation models. Given a task with detailed description and requirements (e.g. the classes to be classified and the definition of each class), Data-free Generalization endeavors to learn a model that can generalize to this specific task and fulfill the requirement without collecting any data or utilizing any existing datasets. Formally, this problem is formulated as follows. Task description and requirements specify the decision function $f\in\mathcal{F}:\mathcal{X}\rightarrow\mathcal{Y}$ and the meta distribution $\mu$. The problem then turns to minimizing Equation 3, as detailed in Section 2.1. The difference is that now the meta distribution $\mu$ cannot be sampled and thus we have no access to any training domains $P^{(1)}_{XY},\cdots,P^{(i)}_{XY},P^{(n)}_{XY}$ or images that are sampled from these domains. However, in the era of large foundation models, the meta distribution $\mu$ can be approximated by LLMs while the domain distribution can be approximated by image generation models. Consequently, we can provide a guarantee on the learning with Theorem 2.1. We provide one such method in Section 2.

Data-free generalization can not only serve a more difficult setting to push the limits of current OOD methods but also holds pragmatic significance in democratizing machine learning. It does so by mitigating or potentially eliminating the necessity for data collection and annotation within the machine learning pipeline, which facilitates a broader access to and application of machine learning technologies, particularly for entities facing resource constraints.

Envision a modest-sized enterprise incapable of investing in the training of large foundational models, nor possessing the necessary time and funding to collect and label an extensive dataset for particular tasks. This situation aligns with the concept of Data-free Generalization, characterized by the availability of only task specifications in the absence of accessible data. Our methodology offers an ideal resolution for such organizations. Initially, they can leverage LLMs’ APIs for a limited number of queries to derive extrapolated domains and scenarios. Following this, they may engage text-to-image models for data synthesis. This synthetic data can then be utilized to either develop new models or enhance existing ones, thereby circumventing the limitations posed by resource constraints.

## 4 Experiments

The objective of our experiments is to (i) demonstrate that knowledge from LLMs successfully extrapolates novel domains and leads to performance benefits grounded by theoretical bounds. (ii) Investigate the most efficient and effective approach for extracting knowledge and sampling from text-to-image models. (iii) Analyze to what extent the synthetic images generated condition on LLMs’ knowledge can serve as good out-of-distribution learners that lead to generalization on unseen testing domains.

### 4.1 Experiment Setup

Datasets. We evaluate generalization to domain shift using four multi-domain datasets in DomainBed [], namely PACS, VLCS, OfficeHome and DomainNet. We follow the train-validate-test split of each dataset as in [] and use the training-domain validation set to perform the hyperparameter search.

Evaluation. To comprehensively evaluate our method, We experiment on both the leave-one-out evaluation protocol and single-domain generalization protocol. In addition, we propose the data-free domain generalization to evaluate whether it is possible to train a generalizable model in a data-free fashion with only task information, the knowledge from LLMs and text-to-image models that bridge the text space to pixel space.

Baseline. We set two baselines for our experiments, namely empirical risk minimization (ERM) and ERM with exponential moving average (ERM + EMA). ERM with exponential moving average is demonstrated to be more stable and effective than ERM []. It is thus adopted to perform ablation study and analysis since its stable performance and more correlated to the validation accuracy [].

Implementation. All experiments use ResNet50 pretrained on ImageNet1k as the image encoder unless otherwise stated. We remove the dropout and follow the rest of the implementation as in [] since dropout is reported to have a negative impact on some of the DG methods [], e.g. RSC []. We adopt GPT-4 to extract novel domain knowledge and leverage Stable Diffusion 2 [] as the text-to-image generation model. We use one A100 GPU to generate synthetic images. All experiments of training ResNet50 and CLIP ViT-B16 model can be run on 1 RTX3090 GPU.

**Table 1: Leave-one-out Evaluation on DomainBed Benchmark. CLIP adopts ViT-B16 as the backbone. ${\dagger}$ denotes reproduced results. MixStyle result is taken from []**
| Algorithm | VLCS | PACS | OfficeHome | DomainNet | Avg |
| --- | --- | --- | --- | --- | --- |
| ERM [] | 77.5 $\pm$ 0.4 | 85.5 $\pm$ 0.2 | 66.5 $\pm$ 0.3 | 40.9 $\pm$ 0.1 | 67.6 |
| IRM [] | 78.5 $\pm$ 0.5 | 83.5 $\pm$ 0.8 | 64.3 $\pm$ 2.2 | 33.9 $\pm$ 2.8 | 65.1 |
| GroupDRO [] | 76.7 $\pm$ 0.6 | 84.4 $\pm$ 0.8 | 66.0 $\pm$ 0.7 | 33.3 $\pm$ 0.2 | 65.1 |
| MLDG [] | 77.2 $\pm$ 0.4 | 84.9 $\pm$ 1.0 | 66.8 $\pm$ 0.6 | 41.2 $\pm$ 0.1 | 67.5 |
| CORAL [] | 78.8 $\pm$ 0.6 | 86.2 $\pm$ 0.3 | 68.7 $\pm$ 0.3 | 41.5 $\pm$ 0.1 | 68.8 |
| Mixup [, , ] | 78.1 $\pm$ 0.3 | 86.8 $\pm$ 0.3 | 68.0 $\pm$ 0.2 | 39.6 $\pm$ 0.1 | 68.1 |
| MMD [] | 77.9 $\pm$ 0.1 | 87.2 $\pm$ 0.1 | 66.2 $\pm$ 0.3 | 23.5 $\pm$ 9.4 | 63.7 |
| RSC [] | 77.8 $\pm$ 0.6 | 86.2 $\pm$ 0.5 | 66.5 $\pm$ 0.6 | 38.9 $\pm$ 0.6 | 67.4 |
| VREx [] | 78.1 $\pm$ 0.2 | 87.2 $\pm$ 0.6 | 65.7 $\pm$ 0.3 | 30.1 $\pm$ 3.7 | 65.3 |
| SWAD [] | 79.1 $\pm$ 0.4 | 88.1 $\pm$ 0.4 | 70.6 $\pm$ 0.3 | 46.5 $\pm$ 0.2 | 66.9 |
| MIRO [] | 79.0 $\pm$ 0.2 | 85.4 $\pm$ 0.4 | 70.5 $\pm$ 0.4 | 44.3 $\pm$ 0.2 | 65.9 |
| MixStyle [] | 77.9 | 85.2 | 60.4 | 34.0 | 64.4 |
| \hdashlineERM ${}^{\dagger}$ [] | 77.2 $\pm$ 1.0 | 84.4 $\pm$ 0.8 | 64.8 $\pm$ 0.4 | 43.6 $\pm$ 0.1 | 67.5 |
| + ours | 78.5 $\pm$ 0.4 | 88.0 $\pm$ 0.3 | 70.0 $\pm$ 0.1 | 45.2 $\pm$ 0.1 | 70.4 |
| $\Delta$ | + 1.3 | + 3.6 | + 5.2 | + 1.6 | + 2.9 |
| \hdashlineERM + EMA | 78.8 $\pm$ 0.6 | 87.8 $\pm$ 0.3 | 70.5 $\pm$ 0.1 | 46.0 $\pm$ 0.1 | 70.8 |
| + ours | 80.2 $\pm$ 0.3 | 90.3 $\pm$ 0.4 | 74.6 $\pm$ 0.2 | 47.5 $\pm$ 0.3 | 73.2 |
| $\Delta$ | + 1.4 | + 2.5 | + 4.1 | + 1.5 | + 2.4 |
| \hdashlineCLIP Zero-shot | 80.1 | 96.2 | 83.0 | 58.5 | 79.5 |
| CLIP Finetune | 82.4 $\pm$ 0.1 | 95.3 $\pm$ 0.2 | 84.8 $\pm$ 0.1 | 59.9 $\pm$ 0.1 | 80.6 |
| + ours | 82.7 $\pm$ 0.3 | 96.5 $\pm$ 0.3 | 86.5 $\pm$ 0.2 | 61.3 $\pm$ 0.0 | 81.8 |
| $\Delta$ | + 0.3 | + 1.2 | + 1.7 | + 1.4 | + 1.2 |

### 4.2 Main Results

We perform two existing evaluations on the four datasets in DomainBed benchmarks. Additionally, we propose a more challenging evaluation to further investigate to the synthetic images generated condition on LLMs’ knowledge can serve as good representation learners.

Leave-one-out evaluation. Leave-one-out Evaluation leaves one domain as the testing domain and uses the rest as training domains. For our method, all of the synthetic images are treated as an additional domain to the source domains. As per Table 1, augmenting with the novel domain synthetic images leads to a consistent improvement (as large as 5.2%) over the ERM and ERM + EMA baselines. On average, we achieve a 2.9% and 2.4% improvement over ERM and ERM + EMA baselines respectively. Our method also achieved a significant improvement (1.2% on average) over the CLIP fine-tuned baseline. This improvement is remarkable, given the already high performance of the CLIP model.

**Table 2: Single-domain Evaluation on DomainBed Benchmark. CLIP adopts ViT-B16 as the backbone.**
| Algorithm | VLCS | PACS | OfficeHome | Avg |
| --- | --- | --- | --- | --- |
| ASA [] | - | 67.0 | - | - |
| Pro-RandConv [] | - | 67.0 | - | - |
| CPerb [] | - | 73.3 | - | - |
| RSC [] | 59.2 $\pm$ 0.7 | 60.9 $\pm$ 1.7 | 46.9 $\pm$ 1.7 | 55.7 |
| \hdashlineERM (Multi-domain) | 77.2 $\pm$ 1.0 | 84.4 $\pm$ 0.8 | 64.8 $\pm$ 0.4 | 75.5 |
| ERM [] | 59.2 $\pm$ 0.8 | 64.6 $\pm$ 0.6 | 51.5 $\pm$ 0.3 | 58.4 |
| + ours | 76.3 $\pm$ 0.2 | 83.9 $\pm$ 0.9 | 64.7 $\pm$ 0.2 | 75.0 |
| $\Delta$ | + 17.1 | + 19.3 | + 13.2 | + 16.5 |
| \hdashlineERM + EMA (Multi-domain) | 78.8 $\pm$ 0.6 | 87.8 $\pm$ 0.3 | 70.5 $\pm$ 0.1 | 79.0 |
| ERM + EMA | 64.2 $\pm$ 0.7 | 67.9 $\pm$ 1.1 | 58.2 $\pm$ 0.1 | 62.7 |
| + ours | 78.0 $\pm$ 0.1 | 87.6 $\pm$ 0.6 | 69.4 $\pm$ 0.3 | 78.3 |
| $\Delta$ | +13.1 | +21.7 | +12.0 | +15.6 |

Single Domain Generalization. Single-domain generalization Evaluation leverages a single domain for training and subsequently assesses the outcomes on the remaining domains. This scenario presents a greater challenge when juxtaposed with the Leave-one-out setting due to the model’s exclusive exposure to just one domain during its training phase. Such a setting accentuates the issue of restricted availability of source domains. Considering our methodology does not impose assumptions on either the source domains or the model, but instead extrapolates novel domains via LLMs to augment the training set, it is optimally more suited for this specific context. Empirical evidence underscores its exceptional efficacy and with merely one source domain of real images, our results closely mirror, and at times even surpass, those obtained in a multi-domain configuration, as per Table 2. Specifically, we achieve the highest of 78.0%, 87.6%, 69.4% on the three datasets, outperforming the ERM with multiple source domains by margins of 0.8%, 3.2% and 4.6% respectively. Compared to baselines, our method achieves a remarkable improvement of over 10% across all datasets and baselines. This evidences that our methodology substantially mitigates the challenges associated with restricted source domains, rendering it particularly optimal and effective in scenarios where source domains are unavailable, such as single-domain generalization.

**Table 3: Comparison with two baselines and current SOTA augmentation-based DG methods. All models are equipped with EMA for fair comparison.**
| Algorithm | VLCS | PACS | Avg |
| --- | --- | --- | --- |
| MixStyle [] | 78.7 $\pm$ 0.1 | 87.7 $\pm$ 0.1 | 83.2 |
| DSU [] | 77.7 $\pm$ 0.0 | 87.6 $\pm$ 0.2 | 82.7 |
| AutoAug [] | 78.6 $\pm$ 0.3 | 88.6 $\pm$ 0.1 | 83.6 |
| RandAug [] | 79.1 $\pm$ 0.0 | 87.5 $\pm$ 0.3 | 83.3 |
| ERM + EMA | 78.8$\pm$ 0.6 | 87.8 $\pm$ 0.3 | 83.3 |
| +larger batch-size | 78.1 $\pm$ 0.1 | 87.4 $\pm$ 0.1 | 82.7 |
| + class-template | 79.3 $\pm$ 0.1 | 88.0 $\pm$ 0.3 | 83.7 |
| + class-prompt | 79.3 $\pm$ 0.0 | 88.5 $\pm$ 0.2 | 83.9 |
| + ours | 80.2 $\pm$ 0.3 | 90.3 $\pm$ 0.4 | 85.3 |

Comparison with augmentation-based DG methods. We compared with SOTA augmentation methods in Table 3 including MixStyle [], DSU [], AutoAug [] and RandAug [], where our method demonstrates an improvement of more than 2% on average.

### 4.3 Data-free Generalization.

Data-free Generalization Evaluation serves as a more difficult setting to evaluate our proposed methods. Data-free Generalization aims to generalize to unseen testing domains with only knowledge of the task, i.e. the classes and definition of each class are available and no available data of any kind. To simulate Data-free Generalization with existing benchmarks, we use all the domains in existing DG datasets as testing domains. To evaluate our method, we directly train models on the synthetic images generated conditioned on novel domain knowledge. Then the model is tested on all the available real images of the domains for evaluation. Results are illustrated in Table 4 where we achieve the highest performance of 79.9%, 86.9%, 67.4% with only less than 1% gap between its multi-domain counterparts and largely surpasses single-domain counterparts. Notably, data-free ERM + EMA presents an accuracy of 79.9% on VLCS outperforming the multi-domain supervised counterparts by more than 1%. With the knowledge injected and novel domain extrapolated, this empirical result illustrates the promise of achieving generalization in a completely data-free fashion free of laborious data collection and annotation.

**Table 4: Data-free generalization on DomainBed Benchmark.**
| Algorithm | VLCS | PACS | OfficeHome | DomainNet | Avg |
| --- | --- | --- | --- | --- | --- |
| ERM |  |  |  |  |  |
| Multi-domain | 77.2 $\pm$ 1.0 | 84.4 $\pm$ 0.8 | 64.8 $\pm$ 0.4 | 43.6 $\pm$ 0.1 | 67.5 |
| Single-domain | 59.2 $\pm$ 0.8 | 64.6 $\pm$ 0.6 | 51.5 $\pm$ 0.3 | - | - |
| Data-free (ours) | 73.9 $\pm$ 0.3 | 82.5 $\pm$ 0.9 | 62.1 $\pm$ 0.1 | 25.9 $\pm$ 0.2 | 61.1 |
| ERM + EMA |  |  |  |  |  |
| Multi-domain | 78.8 $\pm$ 0.6 | 87.8 $\pm$ 0.3 | 70.5 $\pm$ 0.1 | 46.0 $\pm$ 0.1 | 70.8 |
| Single-domain | 64.2 $\pm$ 0.7 | 67.9 $\pm$ 1.1 | 58.2 $\pm$ 0.1 | - | - |
| Data-free (ours) | 79.9 $\pm$ 0.6 | 86.9 $\pm$ 0.1 | 67.4 $\pm$ 0.2 | 30.3 $\pm$ 0.1 | 66.1 |

### 4.4 Ablation Study and Analysis

To fully understand the performance of our method, we perform an ablation study by first providing three baselines building upon ERM + EMA with minor modifications. First, we provide larger batchsize baseline, which is used to ablate the influence of larger batch sizes incurred by the additional augmentation data. Then, we provide class template baseline, which prompts the text-to-images generation model to generate synthetic images with the template "An image of [CLASS]". Then we will provide a third baseline, termed class prompt that will prompt LLMs to give a diffusion-style prompt (without explicitly instructing it to extrapolate novel domains) and use the generated prompts to query text-to-image models for synthetic data. Comparison is shown in Table 3. We can see that a larger batch size in fact has a negative effect while both template and prompt baseline underperform our method. This ablates the influence brought by text-to-image models and further underscores the importance of LLMs’ knowledge regarding the novel domain.

Comparison between different knowledge extraction. We provide three approaches to extract knowledge regarding the novel domains of particular classes. Comparison can be seen in (b) of Figure 4, where we show that, overall, class-wise combined with LLM-generated prompt leads to better performance than class-wise query only and data-wise query. This is because class-wise query provides more plausible and reasonable novel domains given some class and LLM-generated prompt further extracts knowledge regarding this novel domain and increases diversity in generation.

Scaling to larger synthetic dataset. It has been widely reported that data generated by generation models negatively impacts the model, especially when the number of synthetic images grows at scale [, ]. To this end, we investigate whether the performance increases scales with more synthetic data from more extrapolated novel domains. We perform scaling by prompting LLMs to extrapolate more novel domains and generate 64 image per domain. We can see in Figure 3
that with more domains (larger $n$ in Section 2.1), performance keeps increasing, which is consistent with our theoretical framework. We also make a comparison with class-template and class-prompt baselines and scale the two baselines by increasing the synthetic images to the corresponding size. However, these two methods both suffer from performance saturation and degradation when synthetic data increases, which is consistent with previous studies [, ]. This demonstrated that our method can scale better to larger sizes of synthetic data and underscore the importance of new knowledge injected by LLMs that benefits generalization.
variance measure on
PACS
LLMs extrapolation
89.87 $\pm$ 0.4
text-to-image generation
89.72 $\pm$ 0.2
model training
90.3 $\pm$ 0.4
Table 5: Variance analysis over the three modules to measure how stable our method performs.

Figure: Figure 3: Scaling the training dataset by adding more novel domains. Each novel domain consists of 64 images. To facilitate fair comparison, we scale the class template method by the same amount of images.

Figure: Figure 4: (a) Effectiveness of CLIP filtering. (b) Comparison between different knowledge extraction methods.

Variance Analysis. We aim to measure how stable our method is by decomposing the variance into three parts, i.e. LLMs extrapolation, text-to-image generation and final model training. We repeat each experiment three times and report the average and standard deviation in Table 5. For instance, to conduct variance anlysis on text-to-image generation, we use the same set of novel domains generated by LLMs, can generate synthetic datasets with the same text-to-image model three times. As per the table, we can see that all three parts contribute to a relatively small variance, suggesting that our method is stable.

Additional CLIP filtering. Text-to-image generation models are essentially noisy and might generate images of distortion or without the main class of interest. We experiment with CLIP filtering before the training process. As shown in (a) of Figure 4, we can observe an increase with additional filtering techniques by 1 %. To further illustrate the effectiveness of filtering, we visualize some filtered failure cases in Appendix 0.C.

Different LLMs. To make sure that our method does not reply on specific LLMs, i.e. ChatGPT-4, we conduct experiments with LLMs from different families, e.g Llama and Mixtral in table .

**Table 6: Performance with different LLMs. We experiment with GPT4, Llama-13B, Llama-70B and Mixtral-8x7B models.**
| LLM | A | C | P | S | Avg |
| --- | --- | --- | --- | --- | --- |
| GPT-4 | 94.4 $\pm$ 0.2 | 85.0 $\pm$ 0.5 | 98.5 $\pm$ 0.1 | 83.3 $\pm$ 1.7 | 90.3 |
| Llama-13B | 92.6 $\pm$ 0.5 | 83.2 $\pm$ 0.5 | 98.2 $\pm$ 0.1 | 80.9 $\pm$ 0.7 | 88.7 |
| Llama-70B | 93.0 $\pm$ 0.4 | 83.6 $\pm$ 0.4 | 98.5 $\pm$ 0.2 | 81.9 $\pm$ 0.4 | 89.3 |
| Mixtral-8x7B | 92.4 $\pm$ 0.0 | 84.6 $\pm$ 0.3 | 98.8 $\pm$ 0.0 | 81.1 $\pm$ 0.6 | 89.2 |

Visualization. We provide visualization to validate that our method do extrapolate novel domains and generate the desired class. We demonstrate generated images from three different novel domains of the PACS dataset in the last four columns of Figure 5 and compare them with the real images in the PACS dataset (first two columns). We can see that the generated novel domains are by no means an interpolation of the real domains and are different from the existing training domains by a large margin. This illustrates that our method takes one step further toward "truly" extrapolation of novel domains without human labor. We provide more visualization in the Appendix.

Figure: Figure 5: Examples of synthetic images conditioned on novel domain knowledge from LLM. The first two columns (i.e. art painting and cartoon) are selected from PACS datasets while the rest four columns are images generated based on the novel domains (i.e. cityscapes, etc) provided by LLMs.
Refer to caption: x5.png

## 5 Related Work

Domain Generalization. Various approaches have been proposed to solve this problem, such as domain alignment [, ], meta-learning [, ], ensemble learning [, ] and augmentation-based [, , , , , ]. Augmentation-based methods are closely related to this work, both with the intention of generating more source domains to approximate the expected generalization error. However, these methods resort to interpolation of existing domains and fail to extrapolate the "truly" novel domains. For instance, MixStyle [] mixes the statistics of two samples by linear interpolation. More recently, with the advent of vision-language models such as CLIP [] and Stable Diffusion [], researchers propose to utilize Stable Diffusion to identify and cure shortcuts [] or CLIP to generate novel domain augmentation []. However, they all require some form of human labor to pre-define a set of domains or styles, which makes them laborious and not scalable. Our work aims to solve this problem and achieve genuine domain extrapolation.

Language scaffolded vision aims to develop better and more robust vision systems with the help of language. Our method also falls within this category. Clipood [] proposes to fine-tune a CLIP model to adapt the downstream DG tasks by a text similarity aware loss. [] utilize an RNN as an explanation network enforcing the model to self-explain, thereby increasing the robustness. [] utilize language models to produce a comprehensive set of bottleneck features and leverage CLIP to classify. With the help from LLMs, [] has pushed the performance of the bottleneck network to SOTA. Despite many works proposed, this research, to the best of our knowledge, is the first endeavor to investigate the potential of a Large Language Model (LLM) in facilitating the training of a robust and generalizable vision model.

Large Language Models. Recent advances in NLP, as evidenced by  [, ]) highlight the impressive capabilities of Large Language Models like ChatGPT, GPT4 [], and Llama 2 []. These models glean diverse knowledge from vast training data sourced from the Internet, positioning LLMs as next-generation knowledge bases for various tasks. Motivated by studies showcasing the vast knowledge [, ] and the exceptional reasoning ability [, , ] within LLMs, we aim to harness this knowledge for the training of robust vision models.

## 6 Conclusion

The limited availability of domains has been a prevailing problem in Domain Generalization. In this work, we propose the first data-free learning paradigm that leverages the knowledge and reasoning of LLMs to extrapolate novel domains. By bridging the text-centric knowledge and pixel input space by sampling from text-to-image generation models, we are able to train generalizable models with task information only. The synthetic images can be used to augment the existing dataset or train a model in a data-free fashion. Extensive experiments have demonstrated that our method achieves significant improvements over baselines and the state-of-the-art by a significant margin. We also demonstrate a promising learning paradigm where LLMs’ knowledge combined with text-to-image generation models are sufficient to train a generalizable model to any task. However, it’s important to acknowledge the inherent biases present in foundational models like LLMs and text-to-image generators, which our vision models may inherit. This includes a bias towards common object representations, exacerbating the long-tail distribution challenge by privileging common entities in generation processes. Furthermore, our method faces limitations in domain specificity, with current text-to-image models excelling in generating natural images but underperforming in specialized fields such as medical imaging, highlighting a critical area for future improvement and adaptation.

## Appendix 0.A Proof of Theorem 2.1

Notation. Let $\mathcal{X}$ denote the observation space and $\mathcal{Y}=\{1,-1\}$ the output space. Denote $P_{XY}$ as the joint probability of the joint space of $\mathcal{X}\times\mathcal{Y}$ and assume a meta distribution $\mu$ and n domains $P^{(1)}_{XY},\cdots,P^{(i)}_{XY},P^{(n)}_{XY}$ are i.i.d realizations from $\mu$. A decision function is a function $f\in\mathcal{F}:\mathcal{X}\rightarrow\mathcal{Y}$ predicts $\hat{y}_{i}=f(x_{i})$. We denote $\mathit{l}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}_{+}$ a loss function and define the generalization error of a decision function as

$$ $\displaystyle\small\mathcal{L}^{\mu}(f)=\mathbb{E}_{P_{XY}\sim\mu}\mathbb{E}_{ (x,y)\sim P_{XY}}[\mathit{l}(f(x),y)]$ (3) $$

Since we have no access to $\mu$ and all the realizations $P^{(1)}_{XY},\cdots,P^{(i)}_{XY},P^{(n)}_{XY}$ but sampled images from these realizations, we can derive an empirical error:

$$ $\displaystyle\small\hat{\mathcal{L}}^{\mu}(f)=\sum_{i=1}^{n}\sum_{j=1}^{m} \mathit{l}(f(x_{ij}),y_{ij})$ (4) $$

It’s easy to see that when $n\rightarrow\infty,m\rightarrow\infty$, $\hat{\mathcal{L}}^{\mu}(f)$ converges to $\mathcal{L}^{\mu}(f)$, which gives the intuitive sense that increasing $m$ and $n$ gives us better-approximated solutions.

To prove Theorem 2.1, we use a modified version of the standard empirical Rademacher complexity bound that weakens the i.i.d assumption to an independence assumption [].

###### Theorem 0.A.1

For distribution $P^{(1)},\cdots,P^{(n)}$ independent sampled from meta-distribution $\mu$, and 1-Lipschitz loss $l(\cdot,\cdot)$ taking values in $[0,1]$, the following holds with confidence at least $1-\delta$,

$$ $\frac{1}{n}\sum_{j=1}^{n}\mathcal{L}_{P^{(j)}}(f)\leq\frac{1}{n}\sum_{j=1}^{n} \mathcal{\hat{L}}_{P^{(j)}}(f)+2\mathcal{R}_{mn}(\mathcal{F})+3\sqrt{\frac{\ln (2/\delta)}{2mn}}$ (5) $$

where $\mathcal{\hat{L}}_{P^{(j)}}(f)$ is losses on empirical set $S_{P^{(j)}}$ i.i.d. drawn from $P^{(j)}$.

###### Proof

Let $S=\cup_{i=1}^{n}S_{P^{(i)}}$ and

$$ $\Phi(S)=\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{j=1}^{n}(\mathcal{L}_{P^{(j)}}( f)-\mathcal{\hat{L}}_{P^{(j)}}(f))$ (6) $$

which satisfies the bounded differences property required by McDiarmid’s inequality, which implies that with confidence at least $1-\frac{1}{2}\delta$ that

$$ $\Phi(S)\leq\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\left[\Phi(S)\right]+\sqrt{ \frac{\ln(2/\delta)}{2mn}}$ (7) $$

Then we can bound the expected value of $\Phi(S)$

$$ $\displaystyle\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\left[\Phi(S)\right]$ (8) $\displaystyle=\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\left[\sup_{f\in\mathcal {F}}\frac{1}{n}\sum_{j=1}^{n}(\mathcal{L}_{P^{(j)}}(f)-\mathcal{\hat{L}}_{P^{( j)}}(f))\right]$ (9) $\displaystyle=\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\left[\sup_{f\in\mathcal {F}}\frac{1}{n}\sum_{j=1}^{n}\left(\mathbb{E}_{S_{P^{(j)}}^{\prime}\sim P^{(j) }}\left[\frac{1}{m}\sum_{i=1}^{m}l(f(x^{\prime}_{ij}),y^{\prime}_{ij}\right]- \frac{1}{m}\sum_{i=1}^{m}l(f(x_{ij}),y_{ij})\right)\right]$ (10) $\displaystyle\leq\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\mathbb{E}_{S^{\prime }_{P^{(1:n)}}\sim P^{(1:n)}}\left[\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{j=1}^ {n}\frac{1}{m}\sum_{i=1}^{m}l(f(x^{\prime}_{ij}),y^{\prime}_{ij})-l(f(x_{ij}), y_{ij})\right]$ (11) $\displaystyle=\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\mathbb{E}_{S_{P^{(1:n)} }^{\prime}\sim P^{(1:n)}}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\frac{ 1}{n}\sum_{j=1}^{n}\frac{1}{m}\sum_{i=1}^{m}\sigma_{ij}(f(x^{\prime}_{ij}),y^{ \prime}_{ij})-l(f(x_{ij}),y_{ij})\right]$ (12) $\displaystyle\leq\mathbb{E}_{S_{P^{(1:n)}}^{\prime}\sim P^{(1:n)}}\mathbb{E}_{ \sigma}\left[\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{j=1}^{n}\frac{1}{m}\sum_{i =1}^{m}\sigma_{ij}(f(x^{\prime}_{ij}),y^{\prime}_{ij})\right]$ (13) $\displaystyle\quad\quad\quad\quad+\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}} \mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{j=1}^{n}\frac{ 1}{m}\sum_{i=1}^{m}-\sigma_{ij}(f(x_{ij}),y_{ij})\right]$ (14) $\displaystyle=2\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\mathbb{E}_{\sigma} \left[\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{j=1}^{n}\frac{1}{m}\sum_{i=1}^{m} \sigma_{ij}l(f(x_{ij}),y_{ij})\right]$ (15) $\displaystyle=2\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\left[\mathcal{R}_{mn}( \mathcal{F})\right]$ (16) $$

Following McDiarmid’s inequality, we know that with confidence at least $1-\frac{1}{2}\delta$,

$$ $2\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\left[\mathcal{R}_{mn}(\mathcal{F}) \right]\leq 2\mathcal{R}_{mn}(\mathcal{F})+2\sqrt{\frac{\ln(2/\delta)}{2mn}}$ (17) $$

Finally, we have

$$ $\displaystyle\Phi(S)$ $\displaystyle=\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{j=1}^{n}(\mathcal{L}_{P^{ (j)}}(f)-\mathcal{\hat{L}}_{P^{(j)}}(f))$ (18) $\displaystyle\leq\mathbb{E}_{S_{P^{(1:n)}}\sim P^{(1:n)}}\left[\Phi(S)\right]+ \sqrt{\frac{\ln(2/\delta)}{2mn}}$ (19) $\displaystyle\leq 2\mathcal{R}_{mn}(\mathcal{F})+2\sqrt{\frac{\ln(2/\delta)}{2 mn}}+\sqrt{\frac{\ln(2/\delta)}{2mn}}$ (20) $\displaystyle=2\mathcal{R}_{mn}(\mathcal{F})+3\sqrt{\frac{\ln(2/\delta)}{2mn}}$ (21) $$

Thus,

$$ $\frac{1}{n}\sum_{j=1}^{n}\mathcal{L}_{P^{(j)}}(f)\leq\frac{1}{n}\sum_{j=1}^{n} \mathcal{\hat{L}}_{P^{(j)}}(f)+2\mathcal{R}_{mn}(\mathcal{F})+3\sqrt{\frac{\ln (2/\delta)}{2mn}}$ (22) $$

which completes the proof.

Then we can derive the generalization bound with standard empirical Rademacher complexity bound [].

###### Theorem 0.A.2

For a 1-Lipschitz loss $\mathit{l}$, with confidence at least $1-2\delta$ and for all $f\in\mathcal{F}$, we have

$$ $\displaystyle\small\mathcal{L}^{\mu}(f)\leq\hat{\mathcal{L}}^{\mu}(f)+2 \mathcal{R}_{mn}(\mathcal{F})+2\mathcal{R}_{n}(\mathcal{F})+3\sqrt{\frac{\ln(2 /\delta)}{2mn}}+3\sqrt{\frac{\ln(2/\delta)}{n}}$ $$

where $\mathcal{R}(\mathcal{F})$ standard empirical Rademacher complexity on function class $\mathcal{F}$.

Now we show that both the number of domains $n$ and the number of images observed from each domain $m$ is negatively correlated to the upper bound of generalization error.

###### Proof

Let $P=\{p^{(1)},\cdots,p^{(n)}\}$ be a set of n domain distribution i.i.d. sampled from $\epsilon$. Define

$$ $\Phi(P)=\sup_{f\in\mathcal{F}}\mathcal{L}^{\epsilon}(f)-\frac{1}{n}\sum_{j=1}^ {n}\mathcal{L}_{p^{(j)}}(f)$ (23) $$

We construct $P^{\prime}$ by replacing any $p^{(j)}\in P$ with $p^{\prime}\sim\mu$, then we have $|\Phi(P)-\Phi(P^{\prime})|\leq 1/n$. Thus, McDiarmid’s inequality
tells us that with confidence at least $1-\frac{1}{2}\delta$

$$ $\Phi(P)\leq\mathbb{E}_{P^{(1:n)}\sim\mu}\left[\Phi(P)\right]+\sqrt{\frac{\ln(2 /\delta)}{2n}}$ (24) $$

Following the proof techniques in Theorem 0.A.1, we bound the expected value of $\Phi(P)$

$$ $\displaystyle\mathbb{E}_{P^{(1:n)}\sim\mu}\left[\Phi(P)\right]$ (25) $\displaystyle\quad\quad\quad=\mathbb{E}_{P^{(1:n)}\sim\mu}\left[\sup_{f\in \mathcal{F}}\left(\mathbb{E}_{q\sim\mu}[\mathcal{L}_{q}(f)]-\frac{1}{n}\sum_{j =1}^{n}\mathcal{L}_{p^{(j)}}(f)\right)\right]$ (26) $\displaystyle\leq 2\mathbb{E}_{P^{(1:n)}\sim\mu}\mathbb{E}_{(x_{j},y_{j})\sim p ^{(j)}}\left[\mathcal{R}_{n}(\mathcal{F})\right]$ (27) $$

McDiarmid’s inequality can be used to say with confidence $1-\frac{1}{2}\delta$ that

$$ $2\mathbb{E}_{P^{(1:n)}\sim\mu}\mathbb{E}_{(x_{j},y_{j})\sim p^{(j)}}\left[ \mathcal{R}_{n}(\mathcal{F})\right]\leq 2\mathcal{R}_{n}(\mathcal{F})+2\sqrt{ \frac{\ln(2/\delta)}{2n}}$ (28) $$

Thus, we have

$$ $\displaystyle\Phi(P)$ $\displaystyle=\sup_{f\in\mathcal{F}}\mathcal{L}^{\epsilon}(f)-\frac{1}{n}\sum_ {j=1}^{n}\mathcal{L}_{p^{(j)}}(f)$ (29) $\displaystyle\leq\mathbb{E}_{P^{(1:n)}\sim\mu}\left[\Phi(P)\right]+\sqrt{\frac {\ln(2/\delta)}{2n}}$ (30) $\displaystyle\leq 2\mathcal{R}_{n}(\mathcal{F})+3\sqrt{\frac{\ln(2/\delta)}{2n}}$ (31) $$

With Theorem 0.A.1, we have with confidence at least $1-\delta$ that,

$$ $\frac{1}{n}\sum_{j=1}^{n}\mathcal{L}_{p^{(j)}}(f)\leq\mathcal{\hat{L}}^{\mu}(f )+2\mathcal{R}_{nm}(\mathcal{F})+3\sqrt{\frac{\ln(2/\delta)}{2nm}}$ (32) $$

Finally, we have

$$ $\displaystyle\sup_{f\in\mathcal{F}}\mathcal{L}^{\epsilon}(f)\leq\frac{1}{n} \sum_{j=1}^{n}\mathcal{L}_{p^{(j)}}(f)+2\mathcal{R}_{n}(\mathcal{F})+3\sqrt{ \frac{\ln(2/\delta)}{2n}}$ (33) $\displaystyle\leq 2\mathcal{R}_{nm}(\mathcal{F})+3\sqrt{\frac{\ln(2/\delta)}{2 nm}}+2\mathcal{R}_{n}(\mathcal{F})+3\sqrt{\frac{\ln(2/\delta)}{2n}}$ (34) $$

which completes the proof.

Then we prove our Theorem 2.1.

###### Proof

With confidence at least $1-2\delta$ and for all $f\in\mathcal{F}$, we have

$$ $\displaystyle\mathcal{L}^{\mu}(f)-\hat{\mathcal{L}}^{\mu^{\prime}}(f)$ $\displaystyle=\mathcal{L}^{\mu}(f)-\mathcal{L}^{\mu^{\prime}}(f)+\mathcal{L}^{ \mu^{\prime}}(f)-\hat{\mathcal{L}}^{\mu^{\prime}}(f)$ (35) $$

With Theorem 0.A.2, we have

$$ $\displaystyle\mathcal{L}^{\mu}(f)$ $\displaystyle-\mathcal{L}^{\mu^{\prime}}(f)+\mathcal{L}^{\mu^{\prime}}(f)-\hat {\mathcal{L}}^{\mu^{\prime}}(f)$ (36) $\displaystyle\leq 2\mathcal{R}_{mn}(\mathcal{F})+2\mathcal{R}_{n}(\mathcal{F}) +3\sqrt{\frac{\ln(2/\delta)}{2mn}}+3\sqrt{\frac{\ln(2/\delta)}{n}}+\mathcal{L} ^{\mu}(f)-\mathcal{L}^{\mu^{\prime}}$ (37) $\displaystyle\leq 2\mathcal{R}_{mn}(\mathcal{F})+2\mathcal{R}_{n}(\mathcal{F}) +3\sqrt{\frac{\ln(2/\delta)}{2mn}}+3\sqrt{\frac{\ln(2/\delta)}{n}}+\sup_{f}| \mathcal{L}^{\mu}(f)-\mathcal{L}^{\mu^{\prime}}|$ (38) $$

With the assumption that $D(\mu,\mu^{\prime})=\sup_{f}|\mathcal{L}^{\mu}(f)-\mathcal{L}^{\mu^{\prime}}|\leq\epsilon$, we have

$$ $\displaystyle\mathcal{L}^{\mu}(f)$ $\displaystyle-\hat{\mathcal{L}}^{\mu^{\prime}}(f)$ (39) $\displaystyle\leq 2\mathcal{R}_{mn}(\mathcal{F})+2\mathcal{R}_{n}(\mathcal{F}) +3\sqrt{\frac{\ln(2/\delta)}{2mn}}+3\sqrt{\frac{\ln(2/\delta)}{n}}+\epsilon$ (40) $$

which finishes the proof.

###### Theorem 0.A.1

###### Proof

###### Theorem 0.A.2

###### Proof

###### Proof

## Appendix 0.B Visualization

We provide more examples of synthetic images conditioned on novel domain knowledge from LLM. We present in Figure 6 the synthetic images of VLCS datasets.

Figure: Figure 6: Examples of synthetic images conditioned on novel domain knowledge from LLM. The first two columns (i.e. Caltech101 and VOC2017) are selected from VLCS datasets while the rest three columns are images generated based on the novel domains (i.e. fairytale, etc) provided by LLMs
Refer to caption: x6.png

## Appendix 0.C Pitfall of Text-to-image Generation Models

Text-to-image generation models are by nature noisy as no strict control can be achieved. We present some pitfalls (commonly reported by the community) that will insert noise and influence the training of a generalizable model. We show in Figure 7 where each row is a type of problem and below each image is the corresponding class and domain.

Figure: Figure 7: Examples of pitfalls of synthetic images.
Refer to caption: x7.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Despite its strengths, this work has several weaknesses that I encourage the authors to address:
1. **Lack of Technical Details**: The authors omit several crucial details. For instance, it is unclear what specific domains are generated per class, how many such domains are created, and the total number of final data points. Additionally, how is the "novelty" of these domains ensured? The generated domains could potentially overlap with those in the benchmark datasets, thereby limiting the ability to truly measure domain generalization.

2. **Distinction Between Table 2 and Table 3**: As I understand, Table 2 presents results where the generated data is added to a single existing domain for training, with performance measured on all other domains. In contrast, Table 3 reports results after training on synthetic data alone, with performance measured on all other domains. This distinction is somewhat unclear, and I encourage the authors to clarify this in the manuscript.

3. **Unfair Comparison in Tables 1 and 2**: It is unclear if the results across rows are directly comparable, as the proposed method uses significantly more data than other methods. Wouldn’t a fairer comparison involve keeping the dataset size fixed and sampling accordingly from both existing and synthetic data? 

4. **Overclaiming Out-of-Distribution (OOD) Capabilities**: The proposed method, at best, generates “novel” domain data relative to existing benchmarks. However, as text-to-image models are trained on specific data distributions, it remains an open question how well they can create data that truly diverges or extrapolates significantly beyond their training distribution. The authors claim at multiple points that “truly novel data” can be created for any task with only task specifications, yet provide no supporting evidence. I encourage the authors to either substantiate this claim or temper their language accordingly.

I am willing to adjust my score if the authors adequately address these concerns.
soundness: 2
strengths: 1. The theoretical framework for modeling data distributions that span multiple domains, along with the provided theoretical bounds, could be of value to the community. 
2. The approach of leveraging LLMs to generate class-specific domains and using them to design prompts for text-to-image models, ultimately creating domain data, is both useful and, to the best of my knowledge, novel.
summary: This paper introduces a method for generating novel domain data to enhance domain generalization in models. The authors first leverage large language models (LLMs) to create relevant novel domains for each class. For each class and domain, they use LLMs to design prompts tailored for text-to-image models, generating images that represent these new domains. Leveraging the synthetic data from these novel domains, the authors demonstrate improved generalization in both single- and multi-domain settings across benchmarks. Notably, even without real data, models trained solely on the generated dataset achieve competitive performance on a few benchmarks.
weaknesses: Please see the questions section for more information.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: * How does Definition 1 relate to ideas from optimal transport? Why did you define this as your measure of distance as opposed to more traditional OT measures?
 * How much data does each method see in your experiments?
 * How do prompts compare between your LLM generated prompts (your method) and class template/class prompt? I wonder if some of your improved performance comes from better data diversity due to better diversity of prompts when generated by an LLM as opposed to using a template.
soundness: 2
strengths: * The authors create an interesting pipeline for generating synthetic data in novel domains.
* The authors provide some theoretical analysis of the domain generalization problem and provide classical complexity results in the domain generalization setting.
 * The authors provide empirical evidence that their pipeline trains models that generalize better in terms of accuracy, and provide ablation studies of various components of their pipeline. They also analyze components of the pipeline independently, such as the choice of LLM and knowledge extraction approach.
 * I liked your variance analysis experiment.
 * You position your work well in the context of prior literature (such as interpolation failing to effectively increase the number of domains seen)
summary: In this work, the authors propose a novel technique to address challenges in the field of domain generalization. They use an LLM to imagine novel domains over which they generate synthetic data with a diffusion model. They then train models on this synthetic data and find that these models generalize better to new domains in multiple settings. Finally, they even extend to "data free" settings where they train a model from scratch on synthetic data and find that it applies well to real-world data.

While I appreciate the theoretical and empirical results, I am concerned with how realistic these results are and how fair are the author's comparisons. Specifically, I am not convinced that the performance does not come from training models on more data when adding in synthetic domains. I am also unsure how fair it is for large models that may have been trained on the domain's data to then be used in a "data free domain adaptation" setting. Finally, I would love to see some analysis or study of Assumption 1.
weaknesses: * A big weakness of this paper is the key assumption, that $\mu$ and $\mu'$ are close for a language model and the true domain. The authors are able to completely attribute generalization loss to $\epsilon$ and ignore it for the rest of the paper. However, assuming that an LLM does a good job of approximating $\mu$ is a large claim to make. I think it would be good to see some sort of attempt at measuring $\epsilon$, such as even seeing how much this distance changes between two estimated $\mu', \mu''$ generated by two independent LLMs.
 * The authors then say this error can be recovered by increasing $m$, and $n$, but by how much? Would this be a realistic or computationally-feasible amount of data? From an experiment perspective, the baseline methods must be fairly compared with similar values of $m$ and $n$. This is another question I had with the results, but how much more data did your method see than baselines? How would results change if you fixed the same amount of data for baselines, ERM and your method? Otherwise, I do not feel these are fair comparisons.
 * In practice, the LLM and DM are trained on large internet data. It is highly likely that they have seen the image datasets used in this paper (for the DM) or a textual description of these domains. This may provide evidence that the $\epsilon$ is actually quite small, but then raises another question: is it fair to leverage the knowledge of these models for domain generalization given they have seen the generalizing domains?
 * I disagree that data-free generalization can democratize machine learning. Eliminating real-world data collection just results in our representation and learning of certain groups more biased and creates feedback loops where our generative models create biased data that we use to train biased models. Consider, for instance [1], where the authors find that diffusion models generate stereotyped images of various objects, traits, and ethnicities (See Fig. 1 for an illustrative example). In this case, relying on a diffusion model to generalize to a new domain (ie. images from Africa) will not improve performance for users in Africa but instead create harmful stereotypes and potentially not improve model performance.
 * When you increase your domains, do you keep the total number of samples seen constant? Otherwise, it seems difficult to say that it was the domains that led to more performance, rather than just training a model on more data.
 * In your analysis you mention "LLM-generated prompt further extracts knowledge" and later that "the importance of knowledge injected by LLMs that benefits generalization". Can you elaborate on this? Its a pretty vague sentence to me and I would like to understand more clearly what your main thesis is with regards to the importance or contribution of LLMs to domain generalization. 

Minor suggestions:
 * You should bold the best scores in your tables, especially Table 1 as it is so large.
 * Line 317, Evaluation is capitalized
 * When you use quotes, make sure you use ``text'' so that latex formats the quotes properly. 

[1]: Bianchi, Federico, et al. "Easily accessible text-to-image generation amplifies demographic stereotypes at large scale." Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 2023.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - What happens when the images are from more niche datasets like from the healthcare domain? Do you expect that a diffusion model would generate accurate x-ray images, for example?

- What kind of performance is expected for classes which are underrepresented in the image generator?

- Is there any basis for the claim that you can estimate $\mu$ with an LLM?
soundness: 2
strengths: - The idea is interesting, augmenting datasets with generated examples may be useful in small sample regimes

- The procedure seems to work well for some real-world image datasets
summary: The work pertains to domain generalization/OOD generalization in that it seeks to generate new samples from novel domains using existing generative models.

The proposed pipeline entails:1. input a task description with requests for output format into an LLM. 2. The LLM will then generate prompts that will then be used in an image generation model. 3. The image generation model will generate new images based on these prompts and the images can then be used to train the model on the task. 

The work further suggests that this can be done without any previous data. I.e. the task description will be used to prompt the pipeline and no other data than the one generated is used for training the model. 

The results of the empirical investigation show that on DomainBed the method performs well and outperforms other augmentation baselines on VLCS and PACS. There are also some additional small ablation studies of e.g. changes to the query strategy and choice of LLM.
weaknesses: - The theoretical argument is fairly weak, the assertion that an LLM can approximate the meta distribution over samples is unfounded

- Unclear how many samples are generated and how many samples the model has access to. Does the model using your method have access to more samples than the other baselines?

- Generating more samples may help the model if the samples are of sufficient quality. The method does not guarantee this and seems entirely dependent on the generation performing well. 

- Unclear how many times the experiments are run, it is also difficult to compare results as the models have seen different samples

- Experiment on the scaling benefits somewhat unclear, which dataset is being considered?

- Variance analysis on PACS only does not really say much of the variance we can expect in other tasks

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: The paper is well written. Ablation studies are good. The idea of using LLMs for DG is somewhat interesting.
summary: The paper proposes a method leveraging LLMs to synthesize new domains to aid domain generalization (DG). Experiments are shown to demonstrate benefits.
weaknesses: 1. Lots of missing baselines to compare against. Please compare and cite the following relevant works:

- P. Teterwak et al, ERM++: An Improved Baseline for Domain Generalization, https://arxiv.org/abs/2304.01973, ICML 2023 Workshop on Spurious Correlations, Invariance and Stability
- C. Liao et al, Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning, CVPR 2024
- Vishaal Udandarao et al., SuS-X: Training-Free Name-Only Transfer of Vision-Language Models, ICCV 2023
- L. Dunlap et al, Using Language to Extend to Unseen Domains, ICLR 2023

Namely, ERM++ reports an average accuracy of 81.4% in your experimental setup Table 1 (see Table 1 in ERM++ paper). The Descriptor/Word soup paper has been shown to outperform WaffleCLIP and other GPT descriptor baselines, and is currently SoTA. The SuS-X paper is another missing baseline that uses task information to retrieve data and train models. The LADS paper performs domain extension using language to improve accuracy on target domain using language description as input.

2. Lack of key datasets. The authors have not provided results on DG methods on ImageNet and its variants (Adversarial, Sketch, Rendition, V2), and WILDS distribution shifts. The datasets presented are not challenging for DG and saturated in the literature.

3. It's not clear that such a complex method is warranted to obtain these marginal gains. The method uses large LLMs (GPT4) in addition to text to image generation models (Stable Diffusion 2).

4. The authors need to consider the training data used to train the LLM and text-to-image generation model. Namely, addressing important limitation such as what happens when the data is so different, such as a task on satellite image classification for example, or biology problems, when the LLM and image generation is on Internet-scale datasets. It is well known that on such fine grained specific problems, CLIP-trained models are quite poor.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. It is better to put results other than art or cartoon-like image domains to highlight the robustness of the approach. I think improving the performance on such domains using stable diffusion does not make a much impact due to the reasons I described in the weaknesses 1.
soundness: 3
strengths: 1. The idea of using LLM and text2image models to augment the image domain is reasonable to boost the performance in generalizable image classification. 
2. Their approach shows improvements in experiments.
summary: They propose a new approach for domain generalizable image recognition, specifically, leveraging LLM and stable diffusion to expand the knowledge in novel domains. They first propose to generate prompts for novel image domains using LLM and feed the prompts to stable diffusion to generate diverse images. In their experiments, their proposed approach shows superior performance over baseline methods.
weaknesses: 1. They claim that their approach is "data-free". But, I disagree with this point since they employ a stable diffusion model to augment data, and the stable diffusion model is trained on a large amount of data. I think their approach shows high performance, especially in the domains in which the stable diffusion is good at generating. Therefore, I think the data to train the stable diffusion includes their target domain. In this sense, I think their approach is still "data-dependent". 
2. Their way of generating images from the text2image model is reasonable, yet a bit straightforward and is not very novel, considering the existing work like [1]. 
3. Compared to the approach like [1], their prompt design relies only on the heuristic knowledge of the dataset. This is because their approach does not feed any training images into their framework. Therefore, if the training images look totally different from images generated from stable diffusion, their approach can fail. 


[1] Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation