Provably Improving Generalization of Few-shot models with Synthetic Data
Authors: Lan-Cuong Nguyen
Quan Nguyen-Tri
Bang Tran Khanh
Dung D. Le
Long Tran-Thanh
Khoat Than
Venue: ICML 2025 poster (2025)
Keywords: Synthetic data
Few-Shot learning
Robustness
Generalization bounds
Primary area: deep_learning->algorithms
TL;DR: The paper provides a theoretical-guided method for few-shot learning.

Abstract: Few-shot image classification remains challenging due to the scarcity of labeled training examples. Augmenting them with synthetic data has emerged as a promising way to alleviate this issue, but models trained on synthetic samples often face performance degradation due to the inherent gap between real and synthetic distributions. To address this limitation, we develop a theoretical framework that quantifies the impact of such distribution discrepancies on supervised learning, specifically in the context of image classification. More importantly, *our framework suggests practical ways to generate good synthetic samples and to train a predictor with high generalization ability*. Building upon this framework, we propose a novel theoretical-based algorithm that integrates prototype learning to optimize both data partitioning and model training, effectively bridging the gap between real few-shot data and synthetic data. Extensive experiments results show that our approach demonstrates superior performance compared to state-of-the-art methods, outperforming them across multiple datasets.

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Lan-Cuong Nguyen" 12
Quan Nguyen-Tril Bang Tran Khanh 3
Dung D. Le Long Tran-Thanh" Khoat Than
Abstract
tasks (Van Breugel et al., 2023). The main reason comes from the distribution gap, which is the discrepancy between the real and synthetic data distributions, due to the imperfect nature of the generator for downstream task. Overcom- ing this gap is crucial for ensuring effective utilization of synthetic data in this training scenario.
Few-shot image classification remains challeng- ing due to the scarcity of labeled training exam- ples. Augmenting them with synthetic data has emerged as a promising way to alleviate this is- sue, but models trained on synthetic samples often face performance degradation due to the inher- ent gap between real and synthetic distributions. To address this limitation, we develop theoreti- cal framework that quantifies the impact of such distribution discrepancies on supervised learning. specifically in the context of image classification. More importantly, our framework suggests practi- cal ways to generate good synthetic samples and to train a predictor with high generalization abil- ity. Building upon this framework, propose a novel cetoretical-basee algorithm that integrates prototype learning to optimize both data partition- ing and model training, effectively bridging the gap between real few-shot data and synthetic data. Extensive experiments results show that our ap- proach demonstrates superior performance com- pared to state-of-the of-the-art methods, outperforming them across multiple datasets.
Some recent methods minimize this gap by fine-tuning gen- erative models. Notable examples include RealFake (Yuan et al., 2024) and DataDream (Kim et al., 2024) which focus on full dataset and few-shot image classification, respec- tively. These approaches emphasize pixel-space distribution matching. The importance of feature-space alignment has been demonstrated dataset distillation (Wang et al., 2018), where condensed datasets are created to approximate the performance of models trained on full datasets (Wang et al., 2022; Zhao & Bilen, 2023; Zhao al., 2023). An issue of distribution matching is the potential for incorrect associa- tions between samples from different classes. DataDream (Kim et al., 2024) partially mitigates this by fine-tuning gen- erators for each class independently, yet its results remain suboptimal. promising direction to address this issue is prototype learning, which emphasizes local class-specific behavior. Prototype-bas based approaches have been shown to enhance performance in various applications, such as robust classification (Yang et al., 2018), data distillation (Su et al., 2024), and dataset representation (Tu et al., 2023; van No- ord, 2023). While such recent methods have shown promise in narrowing the distribution gap, they predominantly rely heuristics, lacking theoretical guarantees.
1. Introduction
Deep learning models often require extensive annotated datasets to achieve excellent performance. However, ere- ating such datasets is both labor-intensive and costly. To address this challenge, many recent studies has explored synthetic data as an alternative approach to training deep learning models.
In more detail, for a downstream task, we are interested in these questions:
1. What properties can indicate the goodness of a synthetic dataset?
In this paper, we focus on a specific scenario: How to effi- ciently use a synthetic dataset along with real samples to train predictor in downstream task? A naive way of us- ing synthetic data can degrade performance on downstream
2. How to generate a good synthetic dataset?
3. How to efficiently train a predictor from a training set of both real and synthetic samples?
4. How can the quality of a generator affect the generaliza- tion ability of the trained predictor?
'FPT Software AI Center Hanoo University of Science and Technology Vinnniivrrsity ' 'University of Warwick. Correspon- dence to: Khoat Than <khoattq @ wus....ust..d.....>......
From the theoretical perspective, existing studies typically have preliminary answers to simple models only. Indeed, Yuan et al. (2024) proposed framework that generates synthetic samples by training/finetuning generator to min- imize the distribution gap, addressing the second question.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Räisä & Honkela (2024) and Zheng et al. (2023) investi- gated the first question and suggested that synthetic data should be close to real data samples. Recently, Ildiz et al. (2025) provided a systematic study about the role of surro- gate models which create the synthetic labels for training linear models. However, as they only focus on linear mod- els, their study cannot provide a reasonable answer to the four above-mentioned questions of interest.
2. Related Works
2.1. Generative Data Augmentation
With the rapid advancements in generative models, train- ing with synthetic data has gained significant attention. In the context of image classification, recent studies have em- ployed text-to-image models, aligning conditional distri- butions through text-prompt engineering. These efforts have explored class-level descriptions (He et al., 2023), instance-level descriptions (Lei et al., 2023), and lexical definitions (Sariyildiz et al., 2022). While text-based con- ditioning offers flexibility, it often overlooks intrinsic vi- sual details, such as exposure, saturation, and object-scene co-occurrences. To address these shortcomings, RealFake (Yuan et al., 2024) attempts to reduce the distribution mis- alignment with a model-agnostic method that minimizes the maximum mean discrepancy between real and synthetic distributions by fine-tuning the generator. This discrepancy naturally becomes a lower bound for the training loss of diffusion models under certain conditions. GennataAgent (Li et al., 2025) further improves diversity by perturbing image captions and quality by filtering out with the Variance of Gradient (VoG) score.
Against this background, in this paper make systematic step towards understanding the role of synthetic data and distribution for training a predictor in a few-shot setting. Specifically, our contributions are as follows:
Theory: We analyze the generalization ability of pre- dictor, trained with both synthetic and real data. A novel bound on the test error of the predictor is presented. It suggests that a good synthetic set not only should be close to the real samples, but also should be diverse so that the trained predictor can be locally robust around the training samples. This bound also suggests a theoretical principle generate good synthetic samples, and a practical way train a predictor that generalizes well on unseen data.
Beyond empirical findings, theoretical frameworks have been also developed to validate the use of synthetic data. Zheng et al. (2023) employed algorithmic stability to derive generalization error bounds based on the total variation dis- tance between real and synthetic image distributions. Addi- tionally, Gan & Liu (2025) analyzed the impact of synthetic data on post-training large language models (LLMs) using information theory. However, none of these results can be adopted to the few-shot learning domain.
We further present two novel bounds on the test error. Those bounds encloses the discrepancy between the true and synthetic distributions into account. It theoretically shows that the closer the synthetic distribution, the bet- ter the trained predictor becomes. Hence our analyses provide theoretical answers the four questions above.
Methodology: Guided by our theoretical bounds, we intro- duce novel loss function and training paradigm designed to jointly optimize data partitioning and model training, effectively minimizing generalization errors.
2.2. Few-shot Image Classification with Vision-Language Models
The rising performance of vision-language models, such as CLIP (Radford ct al., 2021), has sparked interest in apply- ing them to few-shot image classification. Initial methods focused on leveraging visual or textual prompting (Jia et al., 2022; Zhou et al., 2022; Khattak et al., 2023; Li et al., 2024; Yao et al., 2024; Zheng et al., 2024). Another simultaneous line of research focused on leveraging parameter-efficenn fine-tuning methods with shared Adapter modules (Yang et al., 2024; Yao et al., 2025). With the advent of more powerful generative models, attention shifted toward syn- thesizing additional data complement real few-shot sam- ples, forming joint data pool for model fine-tuning. CaFo (Zhang et al., 2023) enhances diversity of training dataset by combining prior knowledge of four large pretrained models.
Empirical Validation: We evaluated our method in the context of few-shot image classification using synthetic data. Experimental results demonstrate that our method consistently outperforms state-of-te oftthe-arttttt methods across multiple datasets.
The next section summarizes the related work. In Section we investigate the theoretical benefits of synthetic data and the quality of a synthetic distribution to train model. Those investigations suggest explicit an- swers to the four questions mentioned above. Section 4 discusses how to train a good few-shot model, while Sec- tion 5 reports our main experimental results. Mathematical proofs and additional experimental results can be found in the appendices.
Organization:
The central challenge for this approach lies in guiding the synthesis process to generate data closely aligned with the real sew-shot samples. IsSynth (He al., 2023)
2

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
generates images by adding noise to real samples, while DISEF (da Costa et al., 2023) extends this approach by pro- moting diversity through image captioning and fine-tune CLIP models using LoRA (Hu et al., 2022) to reduce com- putational overhead. Both methods employ CLIP filtering to remove incorrectly classified samples. DataDream (Kim et al., 2024) advances this line of work by fine-tuning the generator with few-shot data, further aligning synthesized images with the real data distribution.
discrepancy between real distribution Po and synthetic one Pg. Let du (P, Po) Bu--P, [I((u) h())] be the h-based discrepancy for the whole data space, and d, be the discrepancy for a local area A.
.owPolll(()) - -((s]l u. E A
=
This concept is closely related to distribution matching, which has become prominent technique for generating informative data for supervised learning. The idea of dis- tribution matching has been applied extensively on dataset distillation (Zhao & Bilen, 2021; Zhao et al., 2023) where we generate small dataset from a larger one, and has been used in large-scale image classification (Yuan et al., 2024).
3. The Theoretical Benefits of Synthetic Data for Few-Shot Models
Let ((2) = - Z, be a partition of into K disjoint nonempty subsets. Denote Tss = {e E [K] : S S 2, * 0). In more details, Tss is the collection of all valid areas (i.e.. local areas which contain real data samples from S, Sn and n; |S:| as the number of samples falling into Z, meaning that n Σ5=1 nz.
In this section, theoretically analyze the role of synthetic data when being used to train a predictor. We derive novel bound on the test error of a model, that explicitly reveals the role of a synthetic distribution, the discrepancy of the true and synthetic distribution, and the robustness of a predictor around the training (real and synthesized) samples. This bound provides novel insights and idea to train a predictor with synthetic data.
S). Denote
KK
Definition 3.2. The local robustness of a predictor at a data instance s ~ P in the area A is defined as RA (s. = Ez=p(ll((z)= - h())ll = E A].
3.1. Preliminaries
By definition, R, measures how robust a model is at a spe- cific data point. A small local robustness suggests that the model should be robust in a small area around a point. Other- wise, the model may not be robust. Note that this definition uses the outputs from a model to define robustness.
Notations: A bold character (e.g.. z) denotes vector, while bold capital (e.g., S) denotes set. We denote by || || the fg-norm. |S| denotes the size/cardinality of S, and [K] denotes the set 21, .... K} for K 2 1. We will work with a model (or hypothesis) class H. an instance set Z. and a loss function é: 7(x 2- + R. Given a distribution P defined on Z. the quality of model E H can be measured by its expected loss F(P, h) = Eg... (((h,
(s, We denote R, (5, Po) |s] EeEs RA Po) and
(G, = G Egeg (g, P,)p,) as the local ro- bustness of model h on real dataset S and synthetic dataset G, respectively. Those quantities can be understood as a measurement of a model robustness with respect to a dataset.
=)]. In practice, we typically collect a training set S {21, 3 C Z and work with the empirical loss F(S, h) [S] Exes 2).
Problem setting: Let S denotes a real dataset consisting of n independent and identically distributed samples from the true data distribution Po. Denote G as generator that induces a synthetic data distribution Pg. We can use G to generate a synthetic dataset G with g samples. We are inter- ested in training a classification model from the union SUG of the real and synthetic datasets. This training problem is particularly important in many applications, especially for the cases that only few real samples can be collected.
3.2. Main Theorem
Given these two concepts, we now have the technical tools to provide a theoretical analysis for generalization ability of a model h trained on both real and synthetic data (S U G). The following theorem presents an upper bound on the test error of a model, whose proof appears in Appendix A.1.
Theorem 3.3. Consider model h, a dataset S containing n i..dd. samples from a real distribution Po. and a synthetic dataset G which contains i.i.d. samples distribution Pg. so that g; = |G.| > 0.ror each i E Ts. where G; = Gr Z.. Let C₁ = supzez [(,, z). 9 == EieTs It. Assume that the loss function ((h, 2) is Ln-Lipschitz continuous w.rt h. For any 6 > 0, with probability at least 1 1 8, we have:
To understand the role of synthetic data for training a pre- dictor, we will use the following concepts:
Definition 3.1 (Model-based discrepancy). Let S and G be two datasets and h be a predictor. The h-based dis- crepancy between S and G is denoted as dz S) = Eueg, [[h(s) h(u)ll-
F(Po. S La
Σ 1ET
9i dd (G S.) Z, Pa) +A
This quantity can be seen as the model-dependent distance (through h) between two sets of real and synthetic samples S). One can easily extend this concept to measure the
(1) where A = h) + Eiery 1 $) h) +
3

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Ta Σ teTs
a (S. + + + Jo) 22 2 z.
3.3. Asymptotic Case
Next we consider the asymptotic cases, which can help us to understand the roles of increasing the size of synthetic datasets and the quality of synthetic distribution. The fol- lowing theorem reveals some new insights.
This theorem implies that can bound the population loss F(Po,h) h) with synthetic data. This bound demonstrates that synthetic data can be leveraged to enhance the model's overall generalization performance. In particular, various implications can be inferred from (1):
Theorem 3.4. Using the same notations and assumptions of Theorem 3.3, and let p¹ = (2( as the measure of area
(2)
2 according to distribution Po. for each i E Ts. For any
8 > O, with probability at least 1 - 6, we have:
The discrepancy term Eiets 2t di, S.) theoretically reveals that the quality of the synthetic samples plays a crucial role for h. When those synthetic samples are close to the real ones, each di (G,, S.) would be small, suggesting that those synthetic samples can help improve generalization ability of h. Otherwise, if there are some samples of G that are far (different) from the real samples, the bound (1) will be high, and not be optimal to train h.
F(Po, h) S La Σ sET's
(s,
Rs. 2.11b)]] +A
n
where A
F(P,,)) h) + +iETs 1ttt F(S, +
Co
22v2Khhz In - 2 V
This result suggests that by using a sufficiently large num- ber of synthetic samples to train h. we are making model h locally robust (w.r.t. both distributions) in every region containing the real samples, and hence improving general- ization ability of the trained models. It also further explains the benefits of scaling the number of synthetic samples that have been observed empirically in prior studies.
2. The local robustness of h is also important to the gen- eralization ability of h. Indeed, decrease in robustness quantities of synthetic and real data (R,, (G,, Zi| Po) and (S, Z, Po)) can lead to a decrease in the population loss, leading to better generalization. Furthermore, they also reveal a connection between the local behavior and the generalization capability of the model.
Finally, we consider how well the quality of the synthetic distribution Pg can estimate the quality of model h. To this end, we assume that - 00. In this case, it is
3. The upper bound (1) suggests an amenable way to gener- ate synthetic samples. Assume that we only have the real samples S which may be small. We can use a generator G to generate a synthetic set G that minimizes the upper bound (1). This amounts to optimize the synthetic samples to minmmize the test error of predictor h. It suggests that the generated samples not only need to be close to the reul ones, but also need to ensure better robustness of the trained model different local areas of the input space.
easy to see that (S,, Pg) - dP. Pa (2)) and
+ Pt Po.))). This suggests [ - F( h) + P: p F. h). where F. h) = Ez-ro [e(h, 2)
E 2,Jis is the expected loss of h in the area Z, Further- more, (S Po) + da (Po. Po Z.) as + co, and (Po. P3) = Pf Pg 2.). Therefore, the follow- ing result is consequence of Theorem 3.4.
Corollary 3.5. Given the notations and assumptions from F. (P h) S F(Pg. h) + Theorem 3.4, we have Σ P.
4. Theorem 3.3 also indicates that, for strong generaliza- tion on unseen data, the model h must not only accurately predict both real and synthetic samples, but also keep the discrepancy between real and synthetic predictions low and maintain local robustness for both data types. This reveals novel ways to train h from both real and synthetic data.
LA E 1pd Py 2,) +p: P.) Moreover,
K
(Po, < F(Pg, (Po. d Po)
This corollary provides theoretical support for the intuition that better synthetic distribution P, should be closer to the true one Po. This can be derived from the discrepancy du (Po. P3) in our bound: The smaller this discrepancy is, the tighter bound on the test error of h we will get. Note that the left-hand side of the bound in Corollary 3.5 represents the macro-level average loss of h.
Remark 3.1 (Tightness). Though providing interesting in- sights, our bound (1) is not very tight for some aspects. For instance, it is O(VK) which can be not optimal when K is large. Luckily, the few-shot setting does not allow us to choose a large K, since bound (1) mostly concerns on areas containing real samples. Furthermore, for the extreme cases with only one real sample, the "local" behavior of model h cannot be captured in our bound anymore, and hence our bound can be loose. On the other hand, the sample com- plexity of our bound seems to be optimal for both real and synthetic data. The reason is that the error of the best model is at least Ogg 0.5) + const for a hard learning problem, according to Theorem in (Than et al., 2025).
It is worth highlighting an important implication of Corol- lary 3.5: it suffices for the model h to perceive a small distance between Po and P,- This perspective differs fun- damentally from traditional assumptions, which require the two distributions to be objectively close in some metric space. Our result suggests that even if Po and P, are far
4

---

By minimizing the classification losses on real and synthetic data (Ag and A4), we fine-tune the pretrained models on the target datasets. Optimizing the discrepancy term (Ai) then aligns the model's predictions between real and syn- thetic samples within each region, reducing any mismatch. Finally, minimizing the robustness term (A2) ensures that the model maintains stable predictions on synthetic data, Provably Improving Generalization of Few-Shot Models with Synthetic Data
apart in reality, strong generalization can still be achieved as long as the model h treats them as similar and incurs low synthetic loss.
empirical distributions of the real and synthetic data can approximate well the true real and synthetic distributions Po and P3 on that region, then the partition optimization objectives can be upper bounded by K-means clustering optimization objectives on prediction space. It serves as motivation for our choice of K-means clustering to solve the partitioning optimization. The proof appears in Ap- pendix A.4.
4. Algorithmic Design
This section presents how to use our theoretical bound on the generalization error to design an efficient algorithm to train few-shot model, using synthetic data. We first discuss the main idea, and then the implementation details. While our discussion here focuses on classification problems, we believe that it is general enough to be applied to many other settings including regression.
4.1.2. MODEL OPTIMIZATION
Utilizing the few-shot learning setting. The other com- ponent that heavily affects our generalization bound is the classifier h itself. Because the classifier affects all terms mentioned above except A4, so we can formulate them as minimizing the sum of the remaining expression. Moreover, since we are focusing on few-shot real data, the intra-region distance term for real data (Rx Po)) could be ig- nored as it becomes negligibly small in this setting, and may not affect much to the model optimization.
4.1. Minimizing the Generalization Bound
To guarantee the performance of models trained with syn- thetic data on real data distribution to be small, we aim to minimize the R.H.S. of the bound (I). If we use a Lipschitz loss function for supervised learning problem such as abso- lute loss, the Lipschitz constant La will not change. So we can ignore it from the bound above (e.g.. by rescaling) from optimization perspective. For the sake of clarity, we rewrite the bound as the sum of the following terms:
A1
Σ iETs
Ji dr 24((G S.)
(2)
g
A2
Σ
Ji Rr 211,))
+ T4 (S., PP)o
(3) thereby enhancing overall predictive quality and improving
n
iETs
generalization.
Li F(S,, h))
A3 ====F =F(Sh)) h)
(4)
TETss
g
4.2. Algorithmic Details
A4 CC(( 0.5
+ g
0.5 / 2K In 2 + 22/()
(5)
We propose an algorithm (Algorithm 1) to address the two interrelated optimization problems discussed earlier through a two-phase optimization process. In the first phase, the algorithm optimizes data partitioning address the initial optimization problem. In the second phase, it refines the classifiers to tackle the subsequent problem. This two-phase strategy is designed to minimize the loss function L, which is formulated as a combination of distribution matching, robustness terms, and classification loss. Figure visually demonstrates this general pipeline. The convergence of the loss function and the associated regularization terms empirically demonstrated in Section 5.4.
As F))
(6)
F(S. h) here. Three components play a central role in our optimization problem: the partition I; the synthetic distribution P. and generated data G; and the classifier h. Because the gener- ated data components will be partly addressed with a fine- tuning step for the generator (see Section 5.4 for empirical evidence), we focus on formalizing the other two.
Note that use EieTs 1 h)
1. PARTITION OPTIMIZATION
In the previous bound, given fixed real dataset S and fixed generated dataset G, we will minimize over the set of partitions A₂ (the quantity of distance between data point and its own regions) which depends on the partition the most in all of the above quantities. This optimization problem can be formalized as:
Loss function:
L XF(S,h) h) F(G, h)
+ x1 Σ Σ
Fi
1
[h(s) - h(gg)l
g
|.
TE + SGG.E
min
Ji
R. (G. 2IP.) +
Σ ni R, (S,, 2())
+ X2 3
1 [h((gg) h(g2)) (7)
TT
ET
iET 91
9:
If we assume that in each region i, the amount of real and synthetic data (n, and gi) are sufficiently large so that the
This loss function is directly inspired by the R.H.S. of Eq. 1
5

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Algorithm 1 Fine-tuning few-shot models with synthetic data
components for loss computation (the terms h(s) with real data S and h(g) with synthetic data g E G These terms are computed as the softmax outputs of clas- sification models.
Input: Real dataset 5, number of synthesis samples, (conditional) Pretrained generator models G
1: Initialize centroids for every local area
Loss Computation and Gradient Descent: We calculate the loss function L based on computed components. We update model h via gradient descent.
2: Fine-tuning generator 9 by real dataset S with LoRA
3: Generate 9 symthetic images from generator G
4: Use K-means clustering on both real and synthetic im- ages to obtain partition I(z)
Although this implementation achieves impressive re- sults-thanks to the high-quality data produced by the fine- tuned generator- it also incurs computational costs, both for fine-tuning and for generating large volumes of synthetic data. As an alternative. we introduce a lightweight version that requires generator fine-tuning phase and uses only about one-eighth as much synthetic data. The corresponding pseudocode and detailed description appear in Appendix C (Algorithm 2). As shown in the next section, even this streamlined approach provides competitive performance, compared to the state of the art.
5:
6: for each mini-batch A do
7: Assign datapoints to their nearest clusters
8: Train the model h using the loss function L on the combined dataset SA L GAA that includes both real data > Refer to equation 7.
and synthetic data.
9:
end for
and model optimization problem. To minimize A3 and As. we substitute them with the empirical loss on real and synthetic data. To address A₁ and A2, we include them as regularization components in the loss function. Note that the expectation part robustness term A2 was calculated with its empirical version, utilizing other synthetic data in the same region to compute.
5. Experiments
In this section, we validate the effectiveness of our pro- posed algorithm on few-shot image classification problems. First, we describe the experimental settings, including the baselines, datasets, and implementation details. Then, the main results of fine-tuning models are provided, followed by ablation studies and some additional analysis.
Our loss function is then defined the sum of the empirical loss in real and synthetic data along with two regularization terms. We introduced hyper-parameters X, λ1 and X2 to control the influence of these terms in the optimization process. The empirical loss is modeled using the traditional cross-entropy loss, while the discrepancy and robustness terms calculated using the f--norm to ensure smoother optimization.
5.1. Experimental Settings
Baselines. We compared our solutions with other state-of-the-art methods in Few-shot Image classification: DataDream (Kim al., 2024), DISEF (da Costa al., 2023), and IsSynth (He et al., 2023). All of the results of the base- line methods were obtained from the DataDream paper, except for the DTD (Cimpoi et al., 2014) dataset, where we reproduced the results due to an erroneous implementation in their training/evaluation data split. Datasets. Similar to baselines methods, evaluate our method 10 common datasets for few-shot image clas- sification: FGVC Aircraft (Russakovsky et al., 2015) and Caltech101 (Li et al., 2022) for general object recognition, FGVC Aircraft (Maji et al., 2013) for fine-grained aircraft data, Food101 (Bossard et al., 2014) for common food ob- jects, EuroSAT (Helber et al., 2019) for satelittle images, Oxford Pets (Parkhi et al., 2012) for discrimination of cat and dog types, DTD (Cimpoi et al., 2014) for texture images, SUN397 (Xiao et al., 2010) for scene understanding. Stan- ford Carse et al., 2013) for cars data, and Flowers (Nilsback & Zisserman, 2008) for flower classes.
We now describe the details of our algorithm:
1. Initialization: We begin with initializing the partitions and noise vectors essential for the iterations. Each iteration one iteration of training the classifier. We fine-tune the generator with LoRA (Hu et al., 2022), with the same loss and procedure in DataDream (Kim et al., 2024). Then, we use this generator attached with LoRA module to generate synthetic data.
2. Main Optimization process:
Partition Optimization: In part due to reasons mentioned in Section 4.1 and due to the simplicity of the K-means clustering algorithm, we decided to use it as partition optimization algorithm. Furthermore, to save computa- tion, we decided to perform clustering on data space avoid recomputing the clustering at each iteration.
Model Optimization: With a stabilized partition, we opti- mize the classifier based on loss function C. Forward real and synthetic data through models and compute necessary
Experimental details. We fine-tuned the CLIP ViT-B/16 image encoder with LoRA (Hu al., 2022). To be consis-
6

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Real Images
Clusters
Got labels
Prompt
LoRA
Symthetic Images
Loss
Classiflers
Generator
mm
Learnable parameters
LoRA
Frozen parameters
Figure 1. Illustration of the overall algorithm pipeline outlined in Algorithm 1. First, we generate synthetic images using the labels of real images. Subsequently, both real and synthetic images are clustered. Finally, the images are fed into classifier. The final calculated based on the model's predictions for the samples that belong to the same cluster, thereby reducing the prediction discrepancy between real and synthctic images and between synthetic images themselves within the same cluster.
real shots and 500 synthetic images per-class, except our lightweight version, where only 64 synthetic images per class were utilized. R/S columns denoted whether to use real or synthetic data, respectively. Each result a method was averaged from random seeds, except our full version, where we fixed the same seed 0 for all datascts.
Table Few-shot image classification with CLIP VIT-B/16 experiments are conducted
Method
S
IN
CAL
DTD
EuSAT
AirC
Pets
Cars
SUN
Food
FL.
Avg
CLIP (zero-shot)
70.2
96.1
46.1
38.1
23.8
91.0
63.1
72.2
85.1
71.8
64.1
Real-finetune
73.4
96.8
73.9
93.5
59.3
94.0
87.5
77.1
87.6
98.7
84.2
IsSynth
v
73.9
97.4
75.1
93.9
64.8
92.1
88.5
77.7
86.0
99.0
84.8
DISEF
v
73.8
97.0
74.3
94.0
64.3
92.6
87.9
77.6
86.2
99.0
84.7
DataDream
v
73.8
97.6
73.1
93.8
68.3
94.5
91.2
77.5
87.5
99.4
85.7
DataDream
V
v
74.1
96.9
74.1
93.4
72.3
94.8
92.4
77.5
87.6
99.4
86.3
Ours (lightweight)
V
73.7
97.9
75.5
94.2
71.5
94.5
90.2
77.6
90.0
99.0
86.4
Ours (full)
<
73.8
97.3
74.5
94.7
74.3
94.6
93.1
77.7
90.4
99.3
87.0
tent with the baselines, the generator used is Stable Diffu- sion (SD) (Rombach et al., 2022) version 2.1. Similarly to the baselines, the guidance scale of SD is set to be 2.0 to enhance diversity. In the lightweight version, only 64 im- ages per class were generated without the need fine-tune the generator, and in the full version 500 images per class synthesized from LoRA-attached fine-tuned Stable Diffusion. For the lightweight version, inspired by Real- Fake (Yuan et al., 2024), we improved the quality of the synthesized data using negative prompts "distorted, unrealistic, blurry, out of frame".
0.2, respectively. For the number of clusters, we generally choose it twice as the number of classes of each dataset, ex- cept for ImageNet, where we set to half of them. Analysis on the optimal choice of this hyperparameter can found in the next section of Ablation Studies. More details of the hyperparameter settings can be found in Appendix B.
5.2. Main Results
Table presents the main experimental results in ten datasets. Our method (in both lightweight and full variants) con- sistently outperforms existing state-of-the-- oott----tttt--tttt- approaches, ranking first on 7 of the 10 datasets and second on 2 others. In the datasets where we do not achieve the top score, our results are within 0. 0.1.0.3% of the best-performing method. On average, our lightweight variant performs on par with the strongest baseline (DataDream), while our full variant surpasses by an additional 0.6%. The most notable gains are datasets Food101 and challenging FGVC Aireraft, where method improves performance by more than 2%.
The clustering phase was performed with the FAISS library (Douze et al., 2024). The hyperparameters be tuned are: d1, > control the discrepancy and robustness terms, num- ber of clusters, learning rates, and weight decay. The values of X1, A2 vary between the data sets, but consistently main- tain the ratio of 1/10, since we observe that this ratio brings best balance between them and yields the best results. The hyperparameter was chosen at for all datasets except Stanford Cars, where we it 1. This choice resembles the choice in DataDream, where they select the weight for cross-entropy loss of real and synthetic data to be 0.8 and

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
5.4. Loss Convergence, Influence of Fine- Tone-Tuning, and Correlation with Diserepancy and Robustness
Table 2. Ablation of the loss function components.
Discre.
Rob.
EuroSAT
DTD
AirC
Cars
To further investigate the behavior of algorithm and empirically validate the conrectness of our generalization error bound, we compare four settings: using either our proposed loss function or the DataDream cross-entropy loss, each with or without a generator fine-tuning step. We focus on two challenging datasets: DTD, where fine-tuning the generator (as in DataDream and our full approach) unexpect- edly degrades performance compared methods without fine-tuning (IsSynth, DISEF, and our lightweight variant), and FGVC Aircraft, where Stable Diffusion is known to produce lower-quality images. We track the discrepancy and robustness terms (two regularization terms without multi- plying by hyperparameters) throughout the training process of full version (measured twice per epoch for a total of 100 steps) and record the accuracy of each method. Our key observations from Figure 2 are:
93.5
74.1
72.5
92.6
94.6
74.4
73.1
93.1
94.3
74.3
74.8
93.0
/
94.7
74.5
74.3
93.1
Table 3. Methods performance on CI.IP-Resnete
Methods
AirC
Cars
Food
CAL
Real fine-tune
61.57
78.86
63.52
93.29
IsSynth
70.94
90.82
68.77
94.54
DISEF
65.99
79.18
70.10
94.34
DataDreamele
79.21
92.99
66.70
94.37
DataDreamdset
81.46
93.30
66.63
94.62
Ours
82.67
93.71
70.35
94.17
Effective optimization and performance gains. Our proposed algorithm successfully minimizes both the dis- crepancy and robustness terms, showing smoother and lower values compared to settings that do not incorporate these terms. Moreover, it achieves superior accuracy in both scenarios (with and without generator fine-tuning!
5.3. Ablation Studies
Effectiveness of Regularization Terms. We present results of ablation studies on the regularization terms in Table 2. There are four settings: no regularization, adding two terms of discrepancy and robustness subsequently, and adding both of them. As we introduced regularization terms has positive effects on the results, with most of them have lead to increased perfor- mance.
from the results, adding the
Impact of generator fine-tuning, Fine-tuning the gener- ator can substantially boost performance when the initial generative model is poorly suited to the domain (e.g., im- proving accuracy by up to 7% on FGVC Aircraft), Across most other datasets, methods that include this fine-tuning step (DataDream and full version) also demonstrate improved results comparing ones without them. Results validate our claim that fine-tuning can partially reduce the discrepancy and robustness terms in Section 4.1.
Analysis of partitioning. We investigate how the number of clusters affects performance by varying it from a single cluster (where all data lie in one partition) to multiples of the total number of classes. We observe that the optimal choice is typically around twice the number of classes. We hypothesize that fewer clusters produce ambiguous decision boundaries, while too many clusters overly disperse the data, weakening regularization and degrading results. Based on these findings, we set the number of clusters to twice the number of classes in all main experiments, except for Ima- geNet, where we choose half of the classes as the number of clusters to reduce computational overhead. The detail of experiments can be found in Figure 3, Appendix D.
Empirical support for our theoretical framework. Models that achieve smaller discrepancy and robustness values generally exhibit better accuracy, indicating that these metrics serve as reliable indicators of performance and generalization ability. Our ablations suggest that the robustness term, which has been overlooked in prior studies, seem to be really important to ensure high gener- alization of the trained model.
Results on different architectures. We provide additional results of our methods compared to other different methods in Table 3. We select 4 datasets of FGVC Aireraft, Stanford Cars, Food101, and 11 as in additional experiments in (Kim et al., 2024). Our method outperforms others on 3 out of 4 datasets, while being competitive on the last one. This additional experiment shows the robustness of our method in different architectures, further demonstrating its superiority over existing current approaches.
6. Conclusion
on fine-tuning of the pre- MeainndCLIP-Resneee
In this paper, we introduce a theoretically guided method for training few-shot models with synthetic data. We begin by deriving a generalization bound that reveals how the misalignment between real and synthetic data, as well as model robustness, affects performance and generalization. Building directly on these thcoretical insights, we propose the first algorithm with theoretical guarantees to minimize
8

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
DTD Discrepancy Terms
DTD Robustness Terms
Ours fine-tune (Ace 74.4%%
Ours fine-tune (Acc 74.4%)
DataDream fine tune (Acc 73.8%)
73 888
DataDream fine tune (Acc 73.8%)
73.8%
Ours fine tune (Ace 74.8%)
Ours no fine-tune (Acc 74.8%)
0.0250
DataDream no tine-tune (Acc 74.1%)
0.009
DataDream no tine-tune (Acc 74.1%))
74.1%
Discrepancy
0.0225
Robustness
0.008
0.0175
0.006
0.0150
0.005
14.8%
0.0125
0.004
74.8%
Steps
Steps
FGVC Discrepancy Terms
FGVC Robustness Terms
fine-tune
fine tune 74.4%%
0.05
DetaDream fine tune (Acc 73.5%)
446633%
DataDream fine tune (Acc 73.5%)
Ours no fine tunc (Ace 67.1%
Ours no fine tune (Acc 67.1%)
DataDream no fine-tune (Acc 66.3%)
DataDream no fine -une (Acc 66.3%)
0.0035
0.04
Discregancy
Robustness
0.0030
0.03
7.55%
0.0025
466.3%
0.0020
67.1%
0.02
AA6 6..%%
0.0015
100
Steps
Steps
Figure 2. Visualization of Discrepancy and Robustness Terms Across Settings for DTD and FGVC Aircraft Datasets. The top row shows results for the DTD dataset, with discrepancy (left) robustness (right) terms, while the bottom row shows results the FGVC Aircraft dataset. Accuracy values are annotated at the end of each corresponding line.
this bound, thereby maximizing the performance of few-shot models. Extensive experiments ten benchmark datasets demonstrate that our method consistently outperforms the state-of-theeart.
expanding machine learning applications in real-world narios with limited labeled data. In most cases, by reducing reliance on large-scale annotated datasets, our approach can potentially democratize access to high-performance models. However, when dealing with synthetic data, the method, being misused, may facilitate malicious activities such as model poisoning or creation of deceptive content. Further- more, part of the ideas can be extended to related topics in Machine Learning such adversarial training and domain adaptation, thus sharing the same societal impact on those fields.
Future works can come from additional analysis to fully solve the partition and model selection problems, and find an efficient way to directly use this theoretical framework for optimizing synthetic data through fine-tuning generator filtering by using our loss function thanks to the versatility and simplicity of this generaliza- tion bound, one can use them for enhancing performance of related domains such as adversarial training or domain adaptation with or without additional synthetic data.
criterion. Moreover,
References
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101--01- mining discriminative components with random forests. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, (eds.), Computer Vision ECCV 2014, pp. 446-461, Cham, 2014. Springer International Publishing.
Impact Statement
This work aims to leverage synthetic data for improving few-shot classification models, which holds promise for

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 3606-3613, 2014.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine-grained categorization. In 2013 IEFE International Conference on Computer Vision Workshops, pp. 554-561, 2013.
da Costa, V. G. T., Dall' Asen, N., Wang, Y., Sebe, N., and Ricci, E. Diversified in-domain synthesis with ef- ficient fine-tuning for few-shot classification. ArXiv, abs/2312.03046, 2023.
Lei, S., Chen, H., Zhang, S., Zhao, B., and Tao, D. Image captions are natural prompts for text-to-image models. ArXiv, abs/2207.08826, 2023.
Li, F.F., Andreeto, M., Ranzato, M., and Perona, P. Caltech 101, Apr 2022.
Douze, M., Guzhva, A., Deng, Johnson, J., Szilvasy, G., Mazar'e, e, P.-E., Lomeli, M., Hosseini, L., and J' J'egou, H. The faiss library. ArXiv, abs/2401.000 108281, 2024.
Li, Z., Li, X., Fu, X., Zhang, X., Wang, W., Chen, S., and Yang. J. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Gan, Z. and Liu, Y. Towards theoretical understanding of synthetic data in LLM post-training: A reverse-bottleneck perspective. In The Thirteenth International Conference on Learning Representations, 2025.
pp.
Li, Z., Chen, L., Andrews, J., Ba, Y., Zhang, Y., and Xiang, A. Gendataagent: On-the-fly dataset augmentation with synthetic data. In The Thirteenth International Confer- ence on Learning Representations, 2025.
He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., Bai, S., and QI, X. IS SYNTHETIC DATA FROM GEN- ERATIVE MODELS READY FOR IMAGE RECOG- NITION? In The Eleventh International Conference on Learning Representations, 2023.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference Learning Representations, 2019.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Maji, S., Rahtu, E.,, Kannala, J., Blaschko, M., and Vedaldi, Fine-grained visual classification of aircraft, 2013.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Im- age Processing, pp. 722-729, 2008
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Parkhi, O. M., Vedaldi, A., Zis.esmaaa, A., and Jawahar, C. V.
Ildiz, M. E., Gozeten, H. A., Taga, E. O., Mondelli, M., and Oymak, S. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. In The Thirteenth International Conference Learning Representations, 2025.
Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 2012.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference Computer Vision (ECCV), 2022.
Räisä, O. and Honkela, A. A bias-variance decomposition for ensembles over multiple synthetic datasets. arXiv preprint arXiv:2402.03985 2024.
Khattak, M. U., Wasim, T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. S. Self-regulating prompts: Founda- tional model adaptation without forgetting. In Proceed- ings of the IEEE/CVF International Conference on Com- puler Vision (ICCV), pp. 15190-15200, October 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. Hegh-Resolution Image Synthesis with Latent Diffusion Models. In 2022 IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, Los Alamitos, CA, USA, June 2022. IEEE Computer Society.
Kim, J. M., Bader, J., Alaniz, S., Schmid, C., and Akata, Z. Datadream: Fhw-shot guided dataset generation. In Com- puter Vision - ECCV 2024: 18th European Conference, Milan, Itly, September 29- October 4, 2024, Proceed- ings, Part LXXI. pp. 252 268, Berlin, Heidelberg, 2024. Springer-Verle
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh S., Ma, S., Huang, Z.,, Karpathy, A., Khosla, A., Bernstein,
10

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 115 (3):211-252, December 2015. ISSN 0920-5691.
Yao, H., Zhang, R., and Xu, C. TCP: textual-based class- aware prompt tuning for visual-language model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 23438-23444 IEEE, 2024.
Sariyildiz, M. B., Karteck, A., Larlus, D., and Kalantidis, Y. Fake it till you make it: Learning transferable representa- tions from synthetic imagenet clones. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8011-8021, 2022.
Yao, H., Zhang, R., and Xu, C. Mmrl: Multi- mudti represen- tation learning for vision-language models. In IEEF/VVF Conference on Computer Vision and Pattern Recognition, CVPR 2025. IEEE, 2025.
Su, D., Hou, J., Gao, W., Tian, Y., and Tang, B. Dr4m: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pp- 5809-5818, June 2024.
Yuan, J., Zhang, J., Sun, S., Torr, P., and Zhao, B. Real- fake: Effective training data synthesis through distribu- tion matching. In The Twelfth International Conference on Learning Representations, 2024.
Than, K., Phan, D., and Vu, G. Gentle local robustness implies generalization. Machine Learning, 1 114(6):142, 2025.
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., and Choe, J. Cutmix: Regularization strategy to train strong clas- sifiers with localizable features. In 2019 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pp. 6022-6031. IEEE, October 2019.
Tu, W., Deng, W., Gedeon, T., and Liang, Z. A bag-of- prototypes representation for dataset-level applications. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2023.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In Interna- tional Conference on Learning Representations, 2018.
Van Breugel, B., Qian, Z., and Van Der Schaar, M. Synthetic data, real errors: how (not) to publish and use synthetic data. In International Conference on Machine Learning, pp. 34793-34808, 2023.
Zhang, R., Hu, X., Li, B., Huang. S., Deng, H., Qiao. Y., Gao, P., and Li, H. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 15211-15222. IEEE, 2023.
van Noord, N. Prototype- based dataset comparison. In ICCV, 2023.
Wang, K., Zhao, B., Peng, X., Zhu, Z., Yang. S., Wang, S., Huang, G., Bilen, H., Wang, X., and You, Y. Cafe: Learning to condense dataset by aligning features. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12186-12195, 2022.
Zhao, B. and Bilen, H. Dataset condensation with distribu- tion matching. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6503-6512, 2021.
Zhao, B. and Bilen, H. Dataset condensation with distribu- tion matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6514-6523, January 2023.
Wang, T., Zhu, J.-Y., Torralba, A., and Efros, A. A. Dataset distillation. arXiv preprint arXiv: 1. 109999, 2018
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEFE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485- 3492, 2010.
Zhao, Li, G., Qin, Y., and Yu, Y. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2023.
Recognition, pp.
Yang, H.-M., Zhang, X.-Y., Yin, F., and Liu, C.-L. Robust classification with convolutional prototype learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3474-3482, 2018. doi: 10.1 CVPR.2018 STRR2011..00330 8.00666.
Zheng, C., Wu. G., and Li, C. Toward understanding gen- erative data augmentation. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
Zheng, Z., Wei, J., Hu, X., Zhu, H., and Nevatia, R. Large language models are good prompt learners for low-shot image classification. In CVPR, 2024.
Yang. L.,, Zhang, R.-Y., Wang, Y., and Xie, X. Mma: Multi- modal adapter for vision-language models. In Proceed- ings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. -2837,, June 2024.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for lision-languaee models. International Jour- nal of Computer Vision (IJCV), 2022.

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
A. Proofs
A.1. Proof of Theorem 3.3
Proof. In the following analysis, we will denote F(0, h)) = O. We first observe that:
F(Po.h) = F(Pg. ( F F(S, F(S,h) Σ
9i
F(S,, h))
1ETs g
9i Σ
- (((,,)))F(GGn))) -(G F(P h)
(8)
1E 9
Next, we consider
Σ
9i
F(S, h) F(G,h)
9i F(S, h) - E Σ E(h,u)
(9)
9
9
9 SET's uEG.
iETs
Σ
[F(S h)) e(h, u))
(10)
9
TET's wEG
1
Σ
Σ
Σ
[8(h, 8) E(h, u)l
(11)
9
T4
iETs wEG.
#ES;
1
Ln)lh(s) h(u)l
(12)
9 TETs DEG
T4 #ES;
LA
[h(s) h(w)l
(13)
TK sES;
NETss uEG.
LA
Σ
du (u, S.)
(14)
9 NET's uEG:
La
dn (G., S.)
(15)
NETss
where we have used the fact that (h,, 8) - E(h, u) < h(s)
h(w)ll. due to the Lipschitz continuity of
By Theorem A.I, for any O, we have each of the followings with probability at least 8/2:
F(G,h) h) S
LA Σ
9i Rr Rr (G,, 2,1,) + +C
2K In + (2/()
(16)
9
9
iETs
Thi Rr (S,, 2.)P) +CV
2K In 2 + (2/8)
F(Po.h) h) - F(S,h)
Lx
(17)
n
NTTs
Combining (8) with (15,16,17) will complete the proof.
A.2. Proof of Theorem 3.4
Proof. Using the same arguments as before, we first observe that:
h) = F(Po, -(( F(S,h) F(SS))
Σ 9t F(S., h)++))++
+
Σ
9h
F(S. h) F(G,h) + FG, (18)
g
iET's
iTTs
12

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Next, we consider
Σ T 9
9i F(S, h) - h)
9h F( h)
1 E
Σ
E(h, u))
(19)
9
9 SET'S uEG.
i
Σ
h)) u))
(20)
g
iETs uEG.
1 Σ
[6(h, 8) E(h, u)]
(21)
9
iETs GEG TH #ES;
1
Ln h(u)ll
(22)
9 TETs uEG,
T4 #ES;
LA
(h(8) h(w)l
(23)
g iETs WEG,
T4 sES.
LA
Σ
Σ
dn, S.)
(24)
iETss uEG.
La
Σ
9t dr (G., S.)
(25)
ETSS
where we have used the fact that (h,, 8) E(h, u) < L h+)) h(u)ll
[. due to the Lipschitz continuity of
By Theorem A.I, for any > O, we have the following with probability at least - 6:
" R. (S, Po) + +CV
2K 2K In + 21n(1/5)
F(Po, h)- - F(S,h) <
LA
(26)
n
TE
Combining (18) with (25,26), we have the following with probability least 1 1-8: 8:
h) S
Lp
Σ
T Rr
(,
+ CV
2K In 2 +2m((1)) + F(S,h)
Σ
9i F(S., h)
n
iETs
iETs
+T,,
Σ
Di (G,, h(
(27)
g
'ETs
As g oc, observe that di (G,, S.) R. ((P.)) and F(G,h) - F(Pg. h) and 9 - pp. Combining those facts with (27) completes the proof.
A.3. Some necessary bounds
Theorem A.1. Consider model h learned from dataset S with n i.id. samples from distribution P. Let C sup 2), Assume that the loss function l(h, 2) is Ln-Liccchitz continuous w.r.t h. For any d > 0, the following holds with probability at least 1 6:
F(P,h) S F(S,h) + LA Σ
T4 Rr (S, Z.PP) + +C
2K In - lnd
(28)
iE
F(S,h) S F(P,h)+LI Lm
Th/ Rr, (S., 2. + +0V
2K In 2 2 In
(29)
iET
Proof. Firstly, we make the following decomposition:
K Σ
n Egn p(eh,, >)|2 E 2,]
=(Ph))
(30)
n
1=1
K
n
2,| F(S,h) + F(S.h)
(31)
13

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Secondly, observe that
F(P.h) Σ
n4 E x p p(((h,
Z,]
E Z,1
n
74
Exn. ple(h, 2)|2 2dl
(32)
7
K
Σ
Be / 2,] P(Zz) R2
(33)
==1
K
n4
C
P(Z,)
(34)
n
Note that (ni...., R&) is ii.d multinomial random variable with parameters and (P(Z1), ..... P(Zx)). Bretag- nolle Hae---HuHee-aae Carol inequality, shows the following for any > 0:
K
Pr
Σ
P(Z))
n 2 2e
I 2" exp(
In other words, for any 0, the following holds true with probability at least
K
F(P,h)
Σ T n
21),
2n] S P(Z,) 74
Σ
72
2K In
22
< Cy
(35)
Furthermore,
K
K
E B 1(h, 2) =) 2, - h)
Σ
E Eg [[(h, 2)| = 2,] 2,1-1
1 Σ ((h,)) s)
(36)
sES
ni
E E 11 2)|= E 2,]
Σ l(h, 8)
(37)
n
TETs
TE BES,
1
n& By (((, 2,] Σ ((h,s) 8)
(38)
n
TETs
BES,
1
Σ
Σ
Ex~ 16(h, | 2) E(h, 8) 2 2,]
(39)
n
ieTs sES,
1
<
E Ex L [1((2) h(s))l 12 Z,]
(40)
"
TETs sES,
LA
E P [|(((2) h(s)] E
(41)
n
TETs dES,
= L (S Z|P)
(42)
ET
Where (40) comes from (39) since is L, Lipschitt continuous w.r.t h. Combining the decomposition (30) with (35) and (42) will arrive at (28). One can use the same arguments as above with reverse order of empirical and population loss to show (29), completing the proof.
14

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
A.4. Proof of property in partition optimization 4.1.1
Lemma A.2. æi and æj are vectors from the same clusters, is the mean of that cluster, and is the number of data points in that cluster have the following expression
|z: - - 1.j
Σ (1x, - 2) 171
-0 x)12
2n Σ
x0 012
Proof: If we extract the inside of the 2nd expression above:
( Φb (z, 2 = || - x + |0, - - 2(x; - ( )))
Due to symmetry, we can evaluate the first two expressions easily:
Σ #
22 - ক Σ
|ay 21 = (n 1) Σ c - =
The last expression is:
(Note that
Σ
[z) x)-
[ +
(xg ②)
n1 11 (x; ②) =-(x; x))
j=1
Σ
2(x; 2)7
- Σ
Ft ) T
Σ
ang 5) = Σ
=2
ar ())((-axs x))
itj
171
0i 2
Finally, combining all, we have (2(n 1 2) - -
22-
Main Proof Now, coming back the main results, rewrite here the partition optimization problem:
min T(2)
Ji 2
Σ
(s,
(43)
n
TETs
TT
min
9:
Σ
Rg (g.g,,2P))) Pg) + Σ
n
Σ
RA Po)
(44)
I(2)
9 9 gEG;
nt #ES;
TETs
iETs
min T(2)
Σ
Σ
E Ez~p, [[()) h(g) : 2,]
Σ
E Ez P. [1)(z)) h(s)]] = E 2.]]
(45)
11
SET's gEG.
LET's
sES.
Next we deal with each term in above expression separately. We approximate the expectation term by its empirical version as follows:
E - r g TET's gEG, gET ZEG, g
min min-
[h(z) h()))
(46)
For easier derivation, we will minimize the sum of squares of distance. Note that the square of the sum of positive number is always smaller than or equal to sum of their respective square multiply with constant. So basically we are minimizing an upper bound of the problem. Denote 14 as the average of all the output of generated samples in the region (average of all (==) with z G.), and our optimization problem becomes:
1
Σ
min T(2) 9
[1(()) - h(gg)l2
(47)
iETs g zEG."
Fi
= min T(2) g NET's zEG;
Σ
Σ
[[((z) - 1ill2
(48)
15

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
Note that the quantity inside the sum of (47) is called the within-cluster variation of the K-means problem and is equivalent the traditional variation in (48). The proof is provided in Lemma A.2. Our bound now becomes the lower bound of the K-means optimization problem, and if we ignore the term of summing only over the region containing real samples, it becomes K K-means problem and can be solved easily. The same argument can be applied for the second term.
B. Hyperparameters settings
We adopt the same data preprocessing pipeline as the baseline DataDream (Kim et al., 2024), applying standard augmen- tations including random horizontal flipping, random resized cropping, random color jitter, random grayseale, Gaussian blur, and solarization. Our main difference is to CutMix (Yun et al., 2019) and Mixup (Zhang et al., 2018) by default on all datasets to reduce the amount of hyperparameter tuning. We train our models using AdamW (Loshchilov & Hutter, 2019), searching the learning rate in {2e-4, 1e-4, 1e-5, 1e-6] and the weight decay in {le-3, 54,4, |e-4] for the full approach. For the lightweight approach, we adopt the learning rate and weight decay settings from DISEF(da Costa et al., 2023). We run the K-means clustering step for 300 iterations using the FAISS library (Douze al., 2024) in the full approach. For the classifier tuning phase, we train for epochs for the full approach and 150 epochs for the lightweight approach. The A1 22 values are fixed to be (0.1,1) for the lightweight version and search inside {00.1, 1), ; (2, 20), (20, 200), (50, 500) for the full version. In general, find that our method is quite robust with this choice of values, no substantial performance difference observed when changing these values inside the defined set.
C. Lightweight version
Algorithm 2 Lightweight version
Input: Real dataset S., number g of synthesis samples, (conditional) Pretrained generator models G, Learning rate schedule 7
Output: Set of g generated data samples G Initialize centroids z for every local area
2: Initialize noise vectors (u,, U,,..., Ug) randomly
3: for each iteration do
> In each epoch
4: for mini-batch A do
5: Sample real data set SA, and take their labels as conditional inputs for generator
6: if iteration = then
D. Start optimize partition
7:
Generate set G = (uu) based on labels condition of SA
8:
else
9:
Take set G. from stored generated set.
10:
end if
11:
Assign data points G' to their nearest clusters, indexed by i and centered z; Update learning rate + 1
12:
|=.] 1) GTG
Update the center: 2; + Update Ts and the counts 9: Use own generated data to compute second term of loss function
14:
15:
16:
Compute loss function L Train model h with loss function L and set of real data S and GA
17:
18:
end for
The loss function differs slightly from the full version, where we compute regularization terms all data instead of each batch. The reason for this is because the number of generated data is much smaller, so this computation ensures that the regularization is big enough be meaningful. Initially, the algorithm optimizes with respect data partitions (addressing the partition optimization problem) and subsequently refines classifiers. This is achieved using alternate optimization strategy in each epoch to minimize the loss function L which was constructed as a combination of distribution matching, robustness terms, and classification loss. Note that in this version, the LoRA modules are also attached to the generator, without tuning them.
Algorithm Overview:
16

---

Provably Improving Generalization of Few-Shot Models with Synthetic Data
EuroSAT
Pets
94.6
94.5
94.5
Accuracy
94.0
Accuracy
94.4
93.5
93.0
94.2
92.5
94.1
10
20
40
37
148
Number of Clusters
Number of Clusters
DTD
FGVC Aircraft
75.50
74.4
75.25
Accuracy
75.00
Accuracy
74.2
74.75
74.0
74.50
73.8
74.25
73.6
47
188
100
200
400
Number of Clusters
Number of Clusters
Figure 3. Results with increasing number of clusters on datasets
1. Initialization:Begl with initializing the partitions and noise vectors essential for the iterations. Each iteration is one epoch of training classifier.
2. Iterative Optimization:
Utilize the MiniBatch K-means algorithm to optimize the partition (lines 11-13). Following partition updates, recalibrate dependent quantities (line 14) and monitor changes in regions to optimize memory usage when calculating L.
Partition Optimization:
Model Optimization: Forward real and synthetic data through models and update by loss function
Loss Computation and Gradient Descent: Calculate the function based on computed components: output of real and synthetic data and their discrepancy. Finally, update model h via gradient descent by backpropagating.
D. Partitioning Experiments
In this section, show the detailed figure of the analysis number of clusters (Figure 3). The experiments conducted on 4 datasets: AT, Oxford-IIIT Pets, DTD, and FGVC Aircraft, with the number clusters increased from 1 (all data belong to the same partition) to 1,2 and 4 times the number of classes in each dataset. The results validate our claim in the main text that the optimal choice of number of clusters typically about twice as the number of classes.
E. More extreme few-shot scenarios
We conduct experiments on 3 datasets that were also used for DataDream (DD) (Kim al., 2024). The results shown in Table 4.
Overall, our method underperforms the baseline in the extreme I-shot scenario. With only one real sample, the regularization terms our loss function become small, reducing model robustness and possibly causing performance drops. However, our method significantly outperforms the baseline 4-shot and 8-shot settings. Thus, extremely limited real data case remains a limitation of our approach.
17

---

Table The impact of number of synthetic samples per class. Results of only 16-shot real data Provably Improving Generalization of Few-Shot Models with Synthetic Data
No. of real shots
AirC
Cars
FLO
DD
Ours
DD
Ours
DD
Ours
1
31.1
25.3
72.9
72.1
88.7
86.1
4
38.3
51.6
82.6
86.9
96.0
96.9
8
54.6
63.9
87.4
91.3
98.4
98.7
Table Results more extreme few-shot conditions
F. Adaptation to synthetic data only
In this section, we investigate possible adaptation of our method to the case of synthetic data only. In order do it, one remove the discrepancy term and loss real data from the loss function and compute the robustness loss all regions that contains at least 2 synthetic samples. Overall, the loss function can be rewritten as follows: F(G, h) + 1 E; E Zg..g2EG gi _ (11)((91) h( We conduct experiments to test the effectiveness of this loss function in some small and medium-sized datasets. The results are shown in Table 5.
Dataset
DD
Ours
EuSAT
80.3
80.6
Pets
94.0
94.0
AirC
71.2
70.6
CAL
96.2
96.8
Food
86.7
89.2
Table 5. Results if only synthetic data were used.
Our method outperforms the baseline on 3 out of 5 datasets, comparable in 1 and worse in 1 dataset. On average, our methods still perform better than the baseline, showcasing the necessity of the robustness regularization. However, these increases are marginal, and much less significant compared to our full method, which takes into account both discrepancy and robustness terms.
G. Varying the number of synthetic data
To further investigate the effect of the number of synthetic samples, we conduct more experiments varying the number of them in Table 6. The results confirmed that our method consistently outperform baselines when varying synthetic data sizes.
No. synth. samples
EuSAT
DTD
AirC
DD
Ours
DD
Ours
DD
Ours
100
93.4
94.2
73.4
73.9
68.5
69.6
200
93.5
94.5
73.1
74.0
69.3
71.9
300
93.7
94.4
73.5
73.8
70,9
73.0
400
93.8
94.4
74.1
74.2
70.8
73.3
500
93.5
94.7
74.1
74.5
72.3
74.3
18


Reviews:
[{"review_id": "Xk2x7RU6bU", "reviewer": "Reviewer_hvE3", "summary": "The paper addresses the challenge of improving few-shot image classification by augmenting real data with synthetic data. It identifies the distribution gap between real and synthetic data as a key obstacle. The paper presents a theoretical framework to quantify the impact of this distribution discrepancy on supervised learning and proposes a theoretically-driven algorithm that integrates prototype learning to optimize data partitioning and model training. The algorithm aims to bridge the gap between real and synthetic data. The authors claim that their method outperforms state-of-the-art approaches across multiple datasets. The key idea is to minimize a generalization error bound that accounts for both the discrepancy between the real and synthetic distributions and the robustness of the predictor around training samples. They achieve this through a novel loss function and a training paradigm that jointly optimizes data partitioning and model training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your positive evaluation and valuable suggestions. We address your concerns below.\n\n*1. Key assumptions made in deriving the generalization bounds, and their realisticity*\n\nThe only assumptions are (1) data samples are i.i.d. and (2) loss function is bounded and Lipschitz w.r.t. the model. Note that for classification problems, many common losses satisfy our assumptions, such as absolute loss, square loss, Hinge loss. Therefore, those assumptions are realistic and reasonable, widely used in the literature.\n\n*2. Tightness and practical implications of the bounds*\n\n*Tightness:* We consider several factors regarding the tightness of our bounds:\n\n- Our bound scales with $O(\\sqrt{K})$, which can be sub-optimal for large $K$ (partition size). Therefore, we should not choose a large $K$. Luckily, for few-shot problems, our bound does not allow a large $K$ which goes beyond the number of real samples.  \n- For fixed $K$ and number $n$ of real samples, our bound scales with $O(g^{-0.5})+const$. This indicates *our bound is optimal w.r.t. the number of synthetic samples.* The reason is that the error of the best model is at least $O(g^{-0.5})+const$, for a hard learning problem, according to Theorem 8 in [Than et al. (2024). Gentle local robustness implies Generalization].\n- Finally, since $n$ can be very small, our bound can be far from the true error of a model. This is a common limitation of any theoretical error bound for limited data. To the best of our knowledge, *limited data poses a big (open) challenge* for estimating the model's true error.\n\n*Practical implications:* We have discussed various implications after Theorem 3.3 in the paper. We point out some useful roles of the main terms. We also have Sections 5.3 and 5.4 to investigate empirically the influence of robustness and discrepancy terms.\n\n*3. Ablation study limited to 3 datasets*\n\nWe conducted ablation study on 4 datasets of varying sizes,  which we believe sufficiently illustrate our method's effectiveness. We will add more results on the remaining datasets in the Appendix.\n\n*4. Hyperparameter selection process*\n\nFirst, we balance the discrepancy ($\\lambda_1$) and the robustness ($\\lambda_2$), using coordinate descent to determine this optimal ratio. We then chose a suitable value of $\\lambda$ associated with the loss on real data. The results are reported below:\n|||DTD||EuSAT||Pets||\n|-|-|-|-|-|-|-|-\n|||light|full|light|full|light|full\n|**dis/rob ratio** |1/1|69.2|72.6|89.9|**93.9**|93.1|94.3\n||1/5|72.8|72.8|93.0|93.8|**94.5**|94.3\n||1/10|**73.7**|**73.1**|**93.3**|**93.9**|**94.5**|94.3\n||1/20|73.2|72.8|92.6|**93.9**|94.3|**94.4**\n|$\\lambda$|1|73.7|73.1|93.3|93.9|94.5|94.3\n||2|74.9|73.4|93.3|94.1|94.3|94.5\n||4|**75.2**|**74.5**|**93.8**|**94.7**|**94.8**|94.6\n||6|75.1|74.4|92.7|94.3|94.5|**94.8**\n\nWe then selected the actual values of ($\\lambda_1$/$\\lambda_2$) from a set of values that follows the 1/10 ratio. We chose their values to maintain a good ratio between the regularization and loss.\n\nFor lightweight version, we directly adopted the learning rate and weight decay from the DISEF paper, as our synthetic sample sizes matched theirs. For the full version, we employed grid search with early stopping to choose them. \n\nThese hyperparameter value sets are presented in Appendix B. The others follow exactly as in the baselines.\n\nNote that hyperparameter selection was performed only during classifiers tuning, separately from generator tuning and synthetic data generation, minimizing additional computational overhead.\n\n*5. Relation to existing theoretical work on domain adaptation or generalization with synthetic data*\n\nWe thank the reviewer for useful suggestion. We have discussed the existing theoretical works of generalization with synthetic data in the last paragraph of section 2.1. We will add more related references.\n\n*6. Why select parameters from DataDream (DD), DISEF, and IsSynth as baselines? Were hyperparameters tuned for fair comparison?*\n\nWe chose hyperparameters to be consistent with DD. Since DD's authors reran DISEF and IsSynth on the same number of synthetic data and generative models, it serves as a good baseline for us. Additionally, experiments confirmed that directly using baselines' learning rate and weight decay yielded optimal results across datasets.\n\n*7. In ablation studies, why not include results on the 4 datasets used in Table 3?*\n\nWe followed the Resnet50 setup from DD in table 3 for fair comparison. However, in the ablation study, we aim to evaluate our method's performance across datasets of varying scales, whereas the datasets in Table 3 are similar in size.\n\n*8. Lightweight version give better performance for some datasets*\n\nThe lightweight version performs better on 2 datasets: Caltech and DTD. For these two datasets, we observed that the methods with generator fine-tuning (DD and our full version) perform equal or worse than the others without it (DISEF, IsSynth, and our lightweight).", "claims_and_evidence": "The overall claim that the proposed method improves few-shot learning with synthetic data is supported by the experimental results on several datasets. The tables presented show superior performance compared to several baselines, including DataDream, DISEF, and IsSynth. \n\nThe individual contributions of different components (data partitioning, discrepancy minimization, robustness) are somewhat validated through ablation studies. However, the ablation study in Table 2 only shows the effect of regularization on a few datasets. A more comprehensive ablation study on more datasets might be better to support the effectiveness of each component.", "methods_and_evaluation_criteria": "The proposed method combines theoretical analysis with a practical algorithm. The K-means clustering for partitioning and the prototype learning aspect appear reasonable for the problem. The evaluation uses standard datasets for few-shot image classification (e.g., FGVC Aircraft, Caltech101, Food101), which is appropriate. Comparing against DataDream, DISEF, and IsSynth seems adequate given the focus on synthetic data augmentation.", "theoretical_claims": "While the authors derive generalization error bounds, the practical implications of these bounds depend on the tightness of the bounds and the validity of the assumptions made in their derivation. \n\nWhat are the key assumptions made in deriving the generalization bounds (e.g., concerning the loss function, the data distributions, the model class)? Are these assumptions realistic for the problem of few-shot image classification with synthetic data? \n\nGeneralization bounds are often loose. How tight are the derived bounds in practice? Is there any discussion of the practical implications of the bounds (e.g., how the different terms in the bound influence the performance of the algorithm)?", "experimental_designs_or_analyses": "The experimental designs seem generally sound.\n\nAblation Studies: The ablation studies in Table 2 are helpful for understanding the impact of different components of the algorithm. However, the ablation study only focus on three datasets.\n\nHyperparameter Sensitivity: The paper mentions tuning hyperparameters. A more detailed discussion of the hyperparameter selection process would be useful. It would be beneficial to see a sensitivity analysis that shows how the performance of the algorithm varies with different hyperparameter values. A plot of each hyperparameter in relation to the accuracy/generalization bound should be shown.", "relation_to_broader_scientific_literature": "The paper relates to several areas of research, namely, Few-Shot Learning, Synthetic Data Augmentation, Domain Adaptation and Dataset Distillation\n\nThe paper makes explicit connections to DataDream, DISEF, and IsSynth. It would be useful to discuss how the proposed theoretical framework relates to existing theoretical work on domain adaptation or generalization with synthetic data.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n-The combination of theoretical analysis and a practical algorithm is a strength.\n\n-The experimental results demonstrate promising performance.\n\nWeaknesses:\n\n-The tightness and practical implications of the theoretical bounds need further clarity.", "other_comments_or_suggestions": "The paper could benefit from a clearer statement of the assumptions made in the theoretical analysis.\n\nA more detailed discussion of the hyperparameter selection process would be helpful.", "questions_for_authors": "What are the key assumptions made in deriving the generalization bounds in Theorems 3.3 and 3.4? \n\nWhy do you choose the parameters in DataDream, DISEF and IsSynth as your baseline? Have you tuned the hyper-parameters to have a fair comparison?\n\nIn ablation studies, why not include results on the four datasets used in Table 3?\n\nWhy does lightweight version give you better performance for some datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wgo23dG20V", "reviewer": "Reviewer_L3yv", "summary": "The paper tackles the problem of few-shot image classification, where limited labeled data restricts model generalization, by proposing a theoretically grounded approach that augments real data with synthetic data, such as that produced by Stable Diffusion. It introduces a novel test error bound for models trained on mixed real and synthetic datasets, highlighting the need to minimize the distribution gap between data types and ensure local robustness around training samples. The authors develop a new loss function and training method that incorporates prototype learning via K-means clustering for data partitioning and regularization to align predictions and boost robustness. Experiments across 10 diverse datasets demonstrate that this method surpasses state-of-the-art baselines like DataDream, DISEF, and IsSynth, achieving an average performance boost of 0.6% and exceeding 2% on challenging datasets like Food101 and FGVC Aircraft, with ablation studies validating the effectiveness of the proposed components.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for praising our work and your positive evaluation. We would like to reiterate the key novelties of our work:\n1. We introduced a novel generalization bound for a model trained with synthetic data augmentation. It suggests an effective way to generate synthetic samples, and to train a model by minimizing the distribution gap between real and synthetic data and ensuring local robustness around real samples. \n2. Guided by this theory, we designed an algorithm to jointly optimize models and data partitioning to enhace models' performance.\n3. Experimental results show that our method outperforms state-of-the-art baselines across multiple datasets. Our more efficient lightweight version also shows comparable performance with the baselines.\n\nShould you have any more questions or concerns, we would be very happy to address them.", "claims_and_evidence": "Supported Claim1 : The method outperforms state-of-the-art baselines across multiple datasets\n\nSupported Claim2 : The lightweight version of the algorithm provides competitive performance with reduced computational costs\n\nSupported Claim3 : The proposed algorithm, guided by the theoretical bound, effectively minimizes generalization errors by optimizing data partitioning and model training", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem of few-shot image classification with synthetic data. The theoretical framework tackles the critical issues of distribution gaps and data scarcity, and the algorithm builds on this with practical clustering and loss design. The implementation offers flexible options for performance and efficiency. Meanwhile, the evaluation uses diverse datasets, strong baselines, a suitable metric, and thorough ablation studies to demonstrate the method’s effectiveness. While minor enhancements—like more data on computational savings—could strengthen the case, the overall approach is well-tailored to the problem at hand.", "theoretical_claims": "Did not check", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are largely sound and valid, supported by diverse datasets, strong baselines, and thorough evaluations.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eM51uHv8mB", "reviewer": "Reviewer_dDLg", "summary": "The paper presents a theoretical framework and corresponding algorithm to enhance the generalization of few-shot learning models by leveraging synthetic data.  Guided by our theoretical generalization error bounds, the authors introduce a novel loss function and training paradigm designed to jointly optimize data partitioning and model training. Experiment results demonstrate superior performance on the few-shot classification task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for your positive evaluation and valuable feedback to our work. We would like to address your remaining concerns as follows:\n\n*1. Results for more extreme few-shot conditions*\n\nThanks for your insightful suggestion. We conduct experiments on 3 datasets that was also used in the DataDream (DD) paper. The results are shown in the following table:\n|No. of real shots|AirC||Cars||FLO||\n|-|-|-|-|-|-|-|\n||DD|Ours|DD|Ours|DD|Ours|\n|1|**31.1**|25.3|**72.9**|72.1|**88.7**|86.1|\n|4|38.3|**51.6**|82.6|**86.9**|96.0|**96.9**|\n|8|54.6|**63.9**|87.4|**91.3**|98.4|**98.7**|\n\nOverall, our method underperforms the baseline in the extreme 1-shot scenario. With only one real sample, the regularization terms in our loss function become small, reducing model robustness and possibly causing performance drops. However, our method significantly outperforms the baseline in 4-shot and 8-shot settings. Thus, extremely limited real data case remains a limitation of our approach.\n\n*2. Results if only synthetic data were used*\n\nWe want to gently point out that our method are designed to take into account both real and synthetic data. Our loss function was designed to match between prediction of real and synthetic samples and regularization are computed on regions containing real data only. In order to adapt our method into the case of only synthetic data, one can remove the discrepancy term and loss on real data from the loss function and compute the robustness loss on all regions that contains at least 2 synthetic samples. Overall, the loss function can be rewritten as: $F(\\mathbf{G},\\boldsymbol{h})+\\lambda_2\\frac{1}{g}\\sum_{i}\\sum_{\\boldsymbol{g_1},\\boldsymbol{g_2}\\in \\mathbf{G_i}}\\frac{1}{g_i}\\\\|\\boldsymbol{h}((\\boldsymbol{g_1})-\\boldsymbol{h}(\\boldsymbol{g_2})\\\\|$. \n\nWe conduct experiments to test the effectiveness of this loss function in some small and medium-sized datasets. The results are shown below:\n\n|Dataset|DD|Ours\n|-|-|-\n|EuSAT|80.3|**80.6**\n|Pets|**94.0**|**94.0**\n|AirC|**71.2**|70.6\n|CAL|96.2|**96.8**\n|Food|86.7|**89.2**\n\nOur method outperforms the baseline on 3 out of 5 datasets, comparable in 1 and worse in 1 dataset. On average, our methods still perform better than the baseline, showcasing the necessity of the robustness regularization. However, we admit that these increases are marginal, and much less significant compared to our full method, which takes into account both discrepancy and robustness terms.\n\n*3. The impact of the number of synthetic samples per class. Results of only the 16-shot real data*\n\nWe chose 500 to be consistent with the main experiment in the DataDream paper, ensuring a fair comparison in Tables 1 and 3.\n\nTo further investigate the effect of the number of synthetic samples, we conduct more experiments summarized below:\n\n|No. synth. samples|EuSAT||DTD||AirC||\n|-|-|-|-|-|-|-|\n||DD|Ours|DD|Ours|DD|Ours|\n|100|93.4|**94.2**|73.4|**73.9**|68.5|**69.6**|\n|200|93.5|**94.5**|73.1|**74.0**|69.3|**71.9**|\n|300|93.7|**94.4**|73.5|**73.8**|70.9|**73.0**|\n|400|93.8|**94.4**|74.1|**74.2**|70.8|**73.3**|\n|500|93.5|**94.7**|74.1|**74.5**|72.3|**74.3**|\n\nThe results confirmed that our method consistently ourperform baselines when varying synthetic data sizes.\nThe results of only 16-shot real data was reported in the \"Real-finetune\" row in Table 1 in our paper.\n\n*4. Better solution for generation of synthetic data for the FGVC Aircraft datasets*\n\nStable Diffusion is infamously bad on aircraft data. In practice, we can choose a better generative model on aircraft data. A future direction for our method involves using the theoretical framework for optimizing synthetic data by fine-tuning generator or filtering synthetic data by using our loss function as a criterion.\n\n*5. Detailed hyperparameter analysis*\n\nWe want to gently redirect the reviewer's attention to our response to Reviewer hvE3 (part 4) to see our detailed discussion about the hyperparameter selection. [https://openreview.net/forum?id=L6U7nYc4ah&noteId=sANtK8HtxS]\n\n*6. Essential References Not Discussed*\n\nWe thank the reviewer for highlighting missing references. We will include the following discussion of the mentioned papers and add further references on prompt and adapter tuning in the revised version:\n\nRecently, methods such as prompt and adapter tuning have become prominent for enhancing few-shot classification performance, including [r1], [r3], etc. Moreover, [r2] uses synthetic images to improve performance but relies on combining four large pretrained models, significantly increasing implementation complexity.\n\n*7. Not visually appealing figures and awkward sentence*\n\nThanks for suggesting these issues, we will improve the presentation in the new version.\n\n*8. Stable Diffusion selection*\n\nWe chose it to be consistent with the baseline for fair comparison. DataDream also uses Stable Diffusion of the same version as the generator.", "claims_and_evidence": "The main claims regarding the benefits of minimizing distributional discrepancy and enhancing local robustness to improve generalization are convincingly supported through theoretical analysis and empirical evaluations.", "methods_and_evaluation_criteria": "The proposed ProtoAug method is suitable for few-shot learning and the experimental setup is fair.", "theoretical_claims": "I thoroughly examined the theoretical results, especially Theorems 3.3 and 3.4.", "experimental_designs_or_analyses": "The experimental design is reasonable, using public datasets.  However, I have some questions in the **Weaknesses**.", "relation_to_broader_scientific_literature": "Synthetic data and prototype learning have advanced the broader development of computer vision, providing valuable insights.", "essential_references_not_discussed": "Although the paper compares several related methods, many recent studies are still not discussed or compared, such as [r1], [r2], and [r3]. Thus, the related work section remains relatively weak.  [r2] also generates synthesis images.\n\n[r1] Large Language Models are Good Prompt Learners for Low-Shot Image Classification. CVPR 2024.\n\n[r2] Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners . CVPR 2023.\n\n[r3] Self-regulating Prompts: Foundational Model Adaptation without Forgetting. ICCV 2023.", "other_strengths_and_weaknesses": "**Strengths：**\n1. Synthetic data benefits few-shot learning by improving generalization.\n2. The theoretical proofs provided are clear.\n\n**Weaknesses：**\n1. The experiments are insufficient, including the selection of comparative methods and additional experiments that require further validation. See the \"Questions\" section.\n2. The authors do not provide a detailed hyperparameter analysis, such as the selection of weighting factors for the loss terms. How these hyperparameters were set is not clearly described, and the supplementary material does not seem to clarify this sufficiently.\n3. The layouts of Figures 1 and 2 are not visually appealing.\n4. The prior knowledge about the choice of Stable Diffusion (SD) is missing. Why was it selected specifically?", "other_comments_or_suggestions": "The sentence \"From the theoretical perspective, existing studies typically have preliminary answers to simple models only: E.g., Yuan et al. (2024) proposed a framework that generates synthetic samples by training/finetuning a generator to minimize the distribution gap, addressing the second question.\" sounds awkward, particularly the use of \"E.g.\" Consider revising the sentence structure.", "questions_for_authors": "1. Why didn't the authors report results for more extreme few-shot conditions (e.g., 1-shot, 4-shot, 8-shot)? These settings are commonly evaluated in few-shot learning literature.\n2. What would the results be if only synthetic data were used?\n3. How was the number of synthetic samples per class set to 500? Would using more or fewer samples have an impact? This seems to have a significant effect. Additionally, what would be the results if only the 16-shot real data were used?\n4. Regarding the generation of synthetic data for the FGVC Aircraft dataset, is there a better solution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The final ratings are 3 weak accept. 

Reviewer hvE3 is mainly concerned about the practical implications of the error bound. The authors have addressed those issues in the rebuttal. Reviewer L3yv generally appreciates the idea of this work. Reviewer dDLg, though gave a positive rating, requested more experimental comparison on extreme settings. The authors provided such experiments in the rebuttal and concluded that the proposed method is not suitable for the extreme 1-shot scenario.

After checking the review, the manuscript, and the rebuttal, the AC notices that the performance gain brought by the proposed method is not that significant, especially considering the experiments requested by the reviewers. However, the AC believes that the authors did a good rebuttal and the work is worth sharing with a broader audience.
Decision: Accept (poster)