Do Generated Data Always Help Contrastive Learning?
Authors: Yifei Wang
Jizhe Zhang
Yisen Wang
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive Learning
Diffusion Model
Representation Learning
Self-supervised Learning
Deep Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Contrastive Learning (CL) has emerged as one of the most successful paradigms for unsupervised visual representation learning, yet it often depends on intensive manual data augmentations. With the rise of generative models, especially diffusion models, the ability to generate realistic images close to the real data distribution has been well recognized. These generated high-equality images have been successfully applied to enhance contrastive representation learning, a technique termed ``data inflation''. However, we find that the generated data (even from a good diffusion model like DDPM) may sometimes even harm contrastive learning. We investigate the causes behind this failure from the perspective of both data inflation and data augmentation. For the first time, we reveal the complementary roles that stronger data inflation should be accompanied by weaker augmentations, and vice versa. We also provide rigorous theoretical explanations for these phenomena via deriving its generalization bounds under data inflation. Drawing from these insights, we propose **Adaptive Inflation (AdaInf)**, a purely data-centric strategy without introducing any extra computation cost. On benchmark datasets, AdaInf can bring significant improvements for various contrastive learning methods. Notably, without using external data, AdaInf obtains 94.70% linear accuracy on CIFAR-10 with SimCLR, setting a new record that surpasses many sophisticated methods. Code is available at https://github.com/PKU-ML/adainf.

Published as a conference paper at ICLR 2024
Do GENERATED DATA ALWAYS HELP CONTRASTIVE LEARNING?
Yifei Wang Jizhe Zhang?* Yisen Wang
I School of Mathematical Sciences. Peking University
2 Institute of Artificial Intelligence and Robotics, Xi'an an Jiaotong University
National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University
4 Institute for Artificial Intelligence, Peking University
ABSTRACT
Contrastive Learning (CL) has emerged as one of the most successful paradigms for unsupervised visual representation learning, yet often depends intensive manual data augmentations. With the rise of generative models, especially diffu- sion models, the ability generate realistic images close to the real data distribu- tion has been well recognized. These generated high-equality images have been successfully applied to enhance contrastive representation learning, technique termed "data inflation"'. However, we find that the generated data (even from a good diffusion model like DDPM) may sometimes even harm contrastive learning. We investigate the causes behind this failure from the perspective of both data infla- tion and data augmentation. For the first time, we reveal the complementary roles that stronger data inflation should be accompanied by weaker augmentations, and vice versa. We also provide rigorous theoretical explanations for these phenomena via deriving its generalization bounds under data inflation. Drawing from these in- sights, we propose Adaptive Inflation (AdaInf), a purely data-centric strategy without introducing any extra computation cost. On benchmark datasets, AdaInf can bring significant improvements for various contrastive learning methods. No- tably, without using external data, AdaInf obtains 94. 70% linear accuracy CIFAR-I0 with SimCLR, setting new record that surpasses many sophisticated methods. Code is available hntp::: htttps://qithut httptttttubb....m wwt.coo////.... PLtp/PKK-MMMaaa1
INTRODUCTION
Contrastive learning has to be the state-of-the-art method for self-supervised representation learning across many domains (Chen et all 2020a; He et all 2020; Wang et ali, 202T: Guo et all 2023: Zhang et al., 20233t6). However, there still remains a noticeable gap in performance compared to its supervised counterparts (Chen et all 202I; Cu all 2023). Among many attempts to close this gap, recent surge of interest lies in leveraging high-quality generative models to boost con- trastive learning all 2023: Wang et all, 20224; Azizi et all, 2023; [Tan all 2IZ)). Given an unlabeled dataset, e.g., CIFAR-I0, one can train generative model on it (e.g.. DDPM (Hoctall 2020)) generate a lot of synthetic samples, and then perform contrastive learning on the combi- nation of the real and generated data. This simplest way for using generated data is called "data inflation". Notably,
orthogonal the data augmentation process, wherein an image-be be it raw or generated-is is subjected to manual augmentations (e.g., cropping) to yield positive and negative pairs used contrastive learning (see Figure [[a] for an illustration the pipeline).
Though one may expect that generated data will benefit contrastive learning with more diverse data, we find that it is not always the case. As shown in Figure [[b]. simply inflating CIFAR-10 10 with IM images generated by DDPM (the vanilla setting) leads to even worse linear probing accuracy (91.33% v.s. 90.27%). We investigate this unexpected performance degradation from two aspects: data inflation (how inflated data constructed) and data augmentation (how to craft augmented
Equaa Contribution. Yifei Wang has graduated from Peking University, and is currently a postdoc at MIT. Corresponding Author: Yisen Wang (yisen.wang wang@@kkr @ku.edu.cn).

---

Published as a conference paper at ICLR 2024
Infiation Adaint (ours)
Inflation
Prsilive
Linear Acturary
Generated dama
92
Diflusine Model
Training Data
Contrastive Learinng
Data Augmentation
89
Rew data
Negative
DOPM
Diffusian Models
STF
(a) Data Inflation Pipeline
(b) Performance on CIFAR-10
During data inflation, the real data and the generated data (usually with a larger size) combined together the training data for contrastive learning. where two random augmentations are drawn from each sample to compute the contrastive loss. bb Linear accuracy of contrastive learning (Chen all 20208) with different data inflation strategies CIFAR-10. The generated data are M samples drawn from DDPM (with 3.04 FID) or STF (with 1.94 FID).
Figure 1: a
samples with inflated data). For the former, find that better generation quality is of limited help, while reweighting real and generated data can attain larger gains. For the latter, we discover an intriguing phenomenon that weaker data augmentation, although harmful in standard contrastive learning (Tian et all 2020. Luo et all 2023),, can be very helpful with data inflation. To uncover mysteries behind these observations, we establish the first generalization guarantees for inflated contrastive learning, and explain the benefits of weak augmentations by revealing the complementary roles between data inflation and data augmentation. Based on these insights, we propose an Adaptive Inflation (AdaInf) strategy that adaptively adjusts data augmentation strength and mixing ratio for data inflation, which can bring significant improvements in downstream performance without any introducing computation overhead. We summarize contributions follows:
We discover a failure mode of data inflation for contrastive learning, and reveal the causes of this failure from the perspective of both data inflation and data augmentation. particular, we find that data reweighting and weak augmentation contribute significantly to improving final performance.
To understand these phenomena, establish the first theoretical guarantees for inflated contrastive learning, which not only rigorously explain previous phenomena, but also reveal the complementary roles between data inflation and data augmentation.
We propose the Adaptive Inflation (AdaInf) strategy to adaptively adjust data augmentation strength and mixing ratio for data inflation. Extensive experiments show that the proposed approach improves downstream accuracy significantly no extra cost, and it is particularly beneficial for data-scarce scenarios.
2 PRELIMINARY & RELATED WORK
Self-Supervised Learning. Given an unlabelled dataset that contain raw samples T E IRd, the goal of self-supervised learning is pretrain feature extraction ʃ RRd RRa on D such that the learned representations generalize well to downstream tasks. In this paper, we mainly consider contrastive learning as representative example. For each sample D, we draw two randomly augmented samples v. x ~ A(- as positive pair. The general learning objective of contrastive learning is to align the representations of positive samples while pushing negative samples apart, as in the following widely adopted InfoNCE loss (Oord et aC, 2018 Wang and Isola, 2020):
exp (f(x) ( f(x+))
CIntONCE(f, = -E, Lx; HM, log
(1)
exp (f(x) + EN exp (()TT((77))
where {x, M, are M negative samples drawn independently from D with data augmentation A(-).
Besides, some variants propose to drop negative samples and adopt asymmetric modules to encoder positive pairs to avoid feature collapses (Grill atl 2020: Chen and He, 202I: Caron all, 2020
2

---

Published as a conference paper at ICLR 2024
2021: Zhuo et M(23). Some propose to regularization terms replace the negative samples and obtain similar performance (Zhontar at I02 Recent theoretical analyses show there exists deep connection between contrastive learning and these variants (Tia et all 202T; Garndo et all
2023 Wang et all, 2023). Therefore, we regard them as general contrastive learning methods.
Generative Models. Generative models refer to a broad class of models that learn the data distribu- tion P(z). Popular generative models include GANs (Lroodtellow el al. 2014 Wang et all 202TE), VABS (Kingma and Welling 2014), diffusion models (Ho 2020), etc. In this paper, we mainly take diffusion models for an example due their superior generation quality. During training time, we add random Gaussian noise of scale t E [0. T] an image E D, and train a denoising network Eg Rd + Rd (typically a U-Net) to reconstruct the ground noise added to the image T. ie..
LSM (g. D) = Ee,eED.e ( + /1 o,C,, - =ell"
(2)
where ā₂ is the mixing coefficient at time t (H 20200) In this paper, to enhance contrastive learning unlabeled data, train an unsupervised diffusion model with real data (e.g., CIFAR-10), sample one million generated samples from the diffusion model, and append them to the real data. During this process, we can inflate the training data from 50k samples more than 1M samples, so we call it data inflation. Remarkably, we do not use any external data or model since the diffusion model is also trained on the same training dataset. See Appendix A for more related work on learning with generated data.
3 UNCOVERING REASONS BEHIND THE FAILURE OF DATA INFLATION
As shown in Figure [(b] we discover that directly adding IM images generated by DDPM HHl 2020) may yield minimal or even negative improvements on contrastive learning. this section, we explore the reasons behind this failure from two aspects, the generated data and data augmentation, and design effective strategies to mitigate these failures.
3.1 CAUSES IN DATA INFLATION: DATA QUALITY AND DATA REWEIGHTING
First, we investigate whether the failure lies in design of data inflation. Denote the distribution of the real data Da as Pa. and that of the generated data D, as Pg. After inflation, the overall training distribution becomes P/ PPP + (1 P) where 3 [Pal/[[a] + (Dal) denotes the proportion of the real data when equally mixing them together. The distribution gap between real and generated data can be characterized by the following Theorem B.J (proof in Appendix D.III:
Theorem 3.1. DTv(P,, Pa) (1 DTv(P'p Pa), where DTV denotes the TV distance.
From the above, we can see that there are two factors influencing the distribution gap: the generated data Pg and the mixing ratio B.
Generated Data Quality. A straightforward reason for the failure is that the generative model, DDPM, is not good enough. Since the generative model is not perfect, the distribution gap between real and generated data will be large. Thus, there will be a large mismatch between training and testing data, which prevents generalization. In turn, as long as Pg = Pa. generated data will always be helpful (with more training examples). Thus, direct solution the degradation is to use better generative model with a smaller gap to real data. To validate this point, we compare four diffusion models with different generation qualities (measured by FID). Figure Ka shows that indeed, diffu- sion models with lower FID, such as STF eX et all 2023), consistently bring better downstream accuracy. However, we also notice two drawbacks. First, better generative quality often requires larger models and/or slower sampling (e.g., more denoising steps) BBond-Taylon et alj, 2022). which detriments the efficiency. Second, the improvement over the baseline (91.333 (91.33%) is marginal, as using the best diffusion model STF only gains +0.02% accuracy, which is less worth the effort. Therefore, in the rest of the discussion, we fix the generative model to be STF and explore how to improve downstream performance with other techniques.
Data Reweighting. Beside generated data quality, Theorem G suggests another useful strategy, data reweighting. We can upweight the real data (or downweighting the generated data) with larger mixing ratio 3 which can lead to a smaller gap (Pv,P,, Pa). In practice, we upweight the real data by replicating N times during the mixing (equivalent N NN. (P,)++D,)))
3

---

Published as a conference paper at ICLR 2024
94
93.5
91.33
91.35
Linear Accuracy (89)
91.05
Linear Accuracy
Augmentation
92.5
Standard
Weak
88
92.0
87
86.05
86
No Intationn
EDM
00PM
EDM
STF
20
Diffusion Models
Replicating times (N)
(a) Data Quality
(b) Reweighting
Figure 2: Performance of contrastive learning with 1M generated data CIFAR-IO.
a Linear
accuracy using four diffusion models for generation: DDPM (Ho 2020), EDM KKrras et all
2022). STF (Xueetal 2023) (two EDM models differ in their training time). with data reweighting (real data: generative data = N 1).
b
Linear accuracy
94
95
Augmentation
92
94
Standard
Linear Accuracy (%)
90
Linear Accuracy (%)
93
Weak
91.24
88
92
91.9
91.92
86
91-
84
CIFAR-10
Half CIFAR-10
82
CIFAR-10-0.1M
CIFAR-10-1M
88.63
80
0.05
0.10 0.15 0.20 0.25
0.35 0.40
No Inflation
EDM
DDPM
EDM MD=1.
STF
RAC Min Scale
Diffusion Models
nD=1.a4
nD=1.99
Weaker Augmentation
(a) Random Resized Crop
(b) Inflation Strategy
Figure 3: Linear accuracy with different augmentation strengths, by changing the min scale of random resized cropping (lower value represents stronger augmentation). [b]: Linear accuracy of different inflation strategies on CIFAR-10 (with 10 data reweighting).
Figure 2b shows that after data reweighting, the linear accuracy reaches 92.15 (N = 20) under standard augmentation, indicating that real sample is worth multiple generated samples. Notably, more replication beyond N = 10 leads to worse performance under weak augmentation, as the low weight of generated data (with higher replication of real data) hinders the benefits in data diversity brought by generated data.
3.2 CAUSES IN DATA AUGMENTATION
Aside from the data inflation strategy, we also wonder whether the current training protocol of con- trastive learning should also be adjusted for much larger size of data (20x bigger). Data augmen- tation is arguably the most important part of contrastive learning (Wang et all 20226).. The seminal work SimCLR (Chen et all, 2020a) shows that different data augmentations dramatically influence performance (much larger than learning objectives). Therefore, might wonder how different choices of data augmentation affect the performance of data inflation.
Among commonly used augmentations, random resized crop (RRC) is the most important one (Chen atall 2020a). Therefore, we adjust the augmentation strength by varying the minimal cropped (rel- 0.08), and keeping others fixed. Smaller indicates
ative) area size, denoted as (by default, a stronger augmentation that can crop the image to smaller size, and vice versa. We compare four scale of training data: CIFAR-10, Half CIFAR-10 (50% random split), CIFAR-10 10 O. 1M generated
4

---

Published as a conference paper at ICLR 2024
data (with STF), and CIFAR-10 10 + 1M generated data (with STF). Figure Baa shows clear trend that the optimal augmentation (marked by *) is consistently weaker for larger training data (0.02 for Half CIFAR-10, 0.08 for CIFAR-10, 10, 0.20 for CIFAR-I0 + O.IM, and 0.30 for CIFAR-10 + IM). Therefore, more training data (especially with data inflation) requires an adaptive adjustment of aug- mentation strength fully unleash its benefits. Guided by this principle, we propose a weak version of data augmentation (detailed in Section DD, and Figure B(b) shows that this weak augmentation can consistently bring significant gains for generative data of different FIDs.
3.3 PROPOSED STRATEGY: ADAPTIVE INFLATION
Following these lessons far, arrive at general guideline for data inflation: 1) we should put different weights on real and generated data, and worse quality data should have lower weights; 2) should adopt milder data augmentations with to emphasize that we should adapt the training configurations according to the quality and size of generated data. In practice, to avoid exhaustive search, adopt default choice (called Simple AdaInf, or AdaInf for short) with 10: : 1 mixture of real and generated data (Figure 2bb shows that 10 1 yields the best linear accuracy under weak augmentation.), and a weak data augmentation strategy designed following the AdaInf principle (details in Section H. This default choice, baseline strategy for AdaInf training, works surprisingly well across multiple datasets and generated data. A preview performance of simple AdaInf is shown in Figure
data. We call it Adaptive Inflation (AdaInf)
[bb With no downstream data, one may rely on surrogate metrics for finding the adaptive strategy (e.g. ARC Wang el al. (20226)).
4 THEORETICAL CHARACTERIZATION OF DATA INFLATION
In Section E. we show that different strategies in data inflation have large effects on downstream performance. In this section, we provide in-depth theoretical explanations of these phenomena.
4.1 MATIEMATICAL FORMULATION
To analyze the influence of data augmentation, we adopt the standard augmentation graph frame- work (HaoChen al, 202T: Wang el al, 20226), where data augmentations induce interactions (as edges) between training samples (as nodes). Different from their original setting dealing with in-domain generalization on population distribution, now we need to characterize the influence of adopting more training data and mismatched training-test distribution downstream generalization.
Raw Data as Subgraph. To describe the difference between using raw data and inflated data, key insight is that when the two have the same population distribution (perfect generation), the raw data can be seen as a random subset of the inflated data. This allows us to analyze their difference through a subsampled graph perspective in the augmentation graph framework. Denote the inflated dataset as X' and its augmented dataset as X. We can define an augmentation graph over all augmented training samples in X, and its adjacency matrix A E Rax= represents the joint probability of positive samples under data augmentation, Ag,x' (normalized) graph Laplacian is -- ADD 2, where D is a diagonal degree matrix with the (ar, rt)th diagonal element as DIS = Ar.''' Denote the eigenvalues of L as 0 x < Ag S .... AN < 2. Ignoring the difference between real and generated data, we can regard the raw dataset X (real data) random subset of X' and accordingly, the augmentation graph of raw data is a random subgraph of A. This view allows us to use random graph theory to characterize the influence of data inflation.
(z). The
Following common practice, we evaluate learned features with linear probing as the downstream task, where we leam linear classifier with weights B E kg (r denotes the number of classes) as gf.B on top of pretrained features fxx E & to predict the labels E y of augmented data. Then we define majority voting classifier (T) arg max Prz-A(())) (9),8(z) i) to predict real data. A smaller classification error, denoted E(S, B), indicates better feature separability.
5

---

Published as a conference paper at ICLR 2024
neek segrecstaiinn aee difficllt to crente sinsdar viens
IORA, Caili, DRoyle
blyck
(a) Labeling Error
(b) Graph Connectivity
Figure 4: Illustrative examples on the effect of data augmentation on the labeling error (a) (i.e., aug- mentated samples belonging to different classes) and graph connectivity (b) (i.c., different samples bridged together after augmentation).
4.2 GUARANTEES ON INFLATED CONTRASTIVE LEARNING
Given the formulation above, we establish a formal guarantee for contrastive learning with data inflation. Compared to the original result in HaoChen et all (2021). our guarantees accommodate the discrepancies between the pretraining a downstream distribution (i.e., OOD generalization).
Theorem 4.1. With probability at least 1 6, for the optimal encoder f on inflated data and learned linear head B", its linear probing error has the following upper bound,
8a
((ftBB))
16a + 2 33DDrv
AR+1
where a Aa a) [y(()) + y(²) denotes the labeling ervor caused by data augmentation.
Ak+1 denotes the 1-th smallest eigenvalue of the inflated Laplacian matrix L., Dry (a, a Pe) 1/2 J_ PP((x) Pg (d) denotes the total variation (TV) between real and generated data." m
Based on the generalization upper bound in Eq. B (proof in Appendix D.2), we can provide rigorous explanations for the AdaInf strategy proposed in Section RL
(1) Explaining the Data Inflation Strategy (Section [
First of all, by involving the generated data, Theorem I has additional error term that accounts for the distribution gap between the real and the generated data Drv(P,, Pa). which naturally ex- plains why utilizing better generative model (with lower FID) brings consistently better down- stream performance (Figure (a)) Similarly, a larger weight of raw data 3 also helps close the distribution gap, which aligns well with our analysis (Figure 2(b). Thus, the distribution gap term rigorously justifies the cause from the data inflation side (Section B.I.). For case of analysis, below we assume that two distributions are roughly the same, i.e., Pa R Pg.
(2) Explaining the Data Augmentation Strategy (Section 3.2)
Influence on Labeling Error a. Second, one would wonder how data inflation affects the labelling етrог a, which, intuitively, means the probability that augmentations produce samples belonging to different classes. As shown in Figure Hay stronger augmentations often lead to a larger labeling error. Because a is calculated as the expectation, inflating the data size does not affect it.
Influence on (Algebraic) Graph Connectivity. The most critical and interesting part of this analy- sis, is that we find that data inflation plays an important role in affecting the graph connectivity A From spectral graph theory, we know that Laplacian eigenvalues can serve as algebraic measures of graph connectivity. Loosely speaking, larger eigenvalues indicate better connectivity (and complete graphs have the largest). Data augmentation plays a positive role in improving graph connectivity, since stronger augmentations create more overlap between different training samples (illustrated in Figure Abb). Meanwhile, we also notice that using only raw data (Le., a subset of inflated data) generally has worse connectivity, as there generally fewer edges when restricted to the subgraph. The following lemma from Chung and Horn (2012)) shows that a subsampled graph has a smaller
In practice, although only have finite samples, with the inductive bias of neural networks (Saunshi et RU2A: HaoChen and Ma. (D2E), will not meet Xc+1 0 that renders the bound vacuous.
In practice, since we only use limited real data and training epochs, too large mixing ratio 9 will render the generated data almost useless during training, which also hurt model performance with reduced data diversity. Thus, the optimal reweighting is usually smaller than 0.5 but O.

---

Published as a conference paper at ICLR 2024
075
Dea Sa.
5
2300
1200
120
331
3251
3833
2893
Dala Size
Aigmakiann Stanghh
Augmcilation Seorghi
(a) Ap+1 V.S. Data Size
(b) Ax+1 E.S. Augmenta-(() (C) v.s. Augmentation (d) Error Bound V.S. Aug- tion Strength
Strength
mentation Strength
Figure 5: Analysis the influence of data size (i.e., inflation) and augmentation strength (r) on two crucial factors in the generalization error, the labeling and the graph connectivity λ the synthetic dataset (Section ...)) The optimal augmentation strengths are marked red dots.
spectral gap (usually equals to the second smallest eigenvalue X2, known as algebraic graph con- nectivity (Chung, (197)). Experiments in Appendix E show that other N 's also decrease with larger sampling ratio. Since the no-inflation graph can be seen as subgraph of the inflation graph, it means that inflation will increase the eigenvalues of the non-inflation graph and thus bring better graph connectivity than the raw data.
Lemma 4.2 (Theorem 1.1 in Chung and Horn (ςιлт)). Suppose G is a graph vertices with spectral gap À = min X2, 2 AN and minimum degree dmin- A random subgraph H of G with edge-selection probability p almost surely has a spectral gap dh satisfying
An = - (v log n pdmin
(logn)3/ 3333
+ pdmin (log log n. 3//3
The Complementary Roles between Inflation and Augmentation. Given the analysis above, we know two important facts: 1) data augmentation has a conflicting effect on downstream performance. since stronger augmentation improves graph connectivity (larger (x+1) but also improves labeling error (larger a); data inflation only has one-way effect, it improves graph connectivity and does not change labeling error. Therefore, when data inflation can bring enough graph connectivity, order to further minimize the generalization error, we can accordingly adopt weaker augmen- tation in the sake of smaller labeling error. In turn, if the data size is too small, we need to adopt stronger augmentation to gain better connectivity. Therefore, inflation and augmentation have com- plementary roles for generalization, increasing one of them will decrease the need for the other, and vice versa. As result, with more inflated data, the optimal augmentation strength will shift to a lower one, which explains why weak augmentations lead to better performance in Section B.2
4.3 VERIFICATION EXPERIMENTS
In Section we theoretically characterize the influence of data inflation and data augmentation on the generalization error through two crucial factors: labeling error a and connectivity A Since the augmentation graph is hard to construct for real-world data, we now validate this analysis with synthetic experiment designed following Wang et al, (20226))
Setting. We sample data from the isotropic Gaussian distribution with means (-1, 0) and (1, 0) (two classes). and variance 0.7. The augmentation here to apply uniform noise in a circle of radius r. Thus, can be seen as measure of augmentation strength. With this toy model, we can construct the augmentation graph and explicitly compute the labeling error a and the Laplacian eigenvalues like A4+1 (k by default), allowing to closely examine their changes.
Results. Figure DI shows the influence of data size and augmentation strength, which verifies our analysis from the following aspects. First, Figures D(al & 5(b) show that large data size and stronger augmentations are indeed complementary since they both bring better connectivity (larger Ax+1). Second, Figure p(c] shows that stronger augmentations indeed bring larger labeling error a. At last, Figure ((d) verifies that when combined, the optimal augmentation (marked by red dots) indeed decreases when the increase of data size, which aligns well with our observation in Section 3.2
7

---

Published as a conference paper at ICLR 2024
Table Comparison linear probing accuracy (mean stdev) different contrastive learning meth- ods and different Datasets. As for the generative models, we adopt STF for CIFAR-10 (1.94 FID) and CIFAR-100 (3.14 FID), and DDPM for Tiny ImageNet (18.61 FID).
(a) Different Methods
(b) Different Datasets (w/ SimCLR)
Inflation
SimCLR
MoCo V2
BYOL
Barlow Twins
Inflation
CIFAR-10
CIFAR-I00
Tay ImageNet
No
91.56::0.29
02.75:0043
92.46=00.06
91.2410.30
No
91.56-100.99
66.81+00.66
47.21++0.86
Vanilla
91.38=0.11
92.51+0.40
92.9-0.21
92.09++0.10
Vanilla
91.3844.11
65.52-40.73
41.0334-.39
AdaInf
93,42=0.20
94.19:0.19
92.87=0.26
93.64:0.38
AdaInf
93.42:10.35
0.26
0..46
95.0
96
No Inflation
94
Vanilla
92.5
92
Adainf (ours)
Linear Accuracy 196
90.0
90
88
85.0
86
82.5
84
80.0
No Inflation
Vanilla
77.5
Adainf (ours)
75.0
10k
30k
100k
500K
100k
200k
300k
400k
500k
Training Stops
Training
(a) Different Total Training Steps
(b) Training Process (500k steps in total)
Figure 6: CIFAR-10 results. aat linear accuracy (%) with total different training steps (10k steps R 100 epochs of no-inflation training). [bk linear accuracy along the 500k-step training process.
EXPERIMENTS
Setup. We conduct experiments on three benchmark datasets: CIFAR-10, C0FAR-100, and Tiny ImageNet. . By default, we use 1M synthetic data for CIFAR- and CIFAR-100 generated by high- quality diffusion model STF (Xu et all 2023) (FIDs are 1.94 (CIFAR-10) and 3.14 (CIFAR-100)). Due to the limit of computation resource, we adopt DDPM (18.61 FID) for Tiny ImageNet. These diffusion models are unconditional since we only assume access to unlabeled data for pretraining. We evaluate the pre-trained contrastive models using the linear probing method, where only the raw dataset is used. The model training relies on the solo-learn repository (ia Costa et all 2022). We include SimCLR (Chen et all, 20208) (default choice), MoCo V2 (Chen et all 20205), B YOL (GTill at 2020) and Barlow Twins (Zhontar et all 202T) in this part. For fair comparison of inflated and non-inflated training, we train the model for 100k steps in all cases, which amounts to 1,000 training epochs without inflation. We compare three inflation methods: 1) No Inflation, which serves as bascline for our study; 2) Vanilla Inflation, which equally mixes real and generated data and adopts default augmentations: 3) our AdaInf' strategy, which adopts a mixing ratio of 10: : and weaker augmentations. Specifically, we weaken two most important augmentations: the min scale of random resized cropping improves from 0.08 0.2; the ColorJitter strength decreases from to 0.5; and the probability of applying ColorJitter decreases from 0.8 to 0.4.
Results. We summarize the benchmark results in Table MI where we run each experiment for 3 ran- dom trials. Table [Ta]] shows that compared to the no-inflation baseline, vanilla inflation sometimes leads to even worse performance (e.g., on MoCo V2), while AdaInf has consistent improvements on all datasets. Mcanwhile, AdaInf outperforms vanilla inflation significantly (with more than I 1% gain in accuracy) on most methods (SimCLR, MoCo V2, and Barlow Twins), while the two perform comparably on BOL (potentially because the BYOL augmentation needs specific weakening strat- egy). Table [b] shows that AdaInf brings consistent improvements across datasets having different scales and different numbers of classes.
To gain further understanding of data inflation, we further take closer look at its behaviors. Exper- iments in this part conducted CIFAR-10 with SimCLR unless specified.
8

---

Published as a conference paper at ICLR 2024
Table [a The impact of weak augmentation and reweighting real data when data inflation. Both of them contribute together to make performance of CL model better. {b} By randomly sampling 5000 images from CIFAR- 10 as the original dataset and applying AdaInf for data inflation, the linear accuracy significantly improves.
(a) Ablation Study of AdaInf
(b) Data-Scarce Scenario
Generated Data
X
/
V
Inflation
Linear Accuracy
Data Reweighting
x
x
X
v
No Inflation
74.83
Weak Augmentation
X
X
/
Vanilla
76.95
Linear Accuracy
91.33
91.35
91.92
93.21
93.57
AdaInf (ours)
79.15
Training Steps. We examine whether data inflation is also effective under different total train- ing steps. As shown in Figure b[a], AdaInf brings very large improvement under short training (80.45 84.58 with 10k training steps), and there still remains clear advantage even if train for 500k steps (around 5,000 epochs in standard training). Remarkably. SimCLR attains 94.7% linear accuracy under this setting, setting a new SSL record on CIFAR-10 with simply the Sim- CLR method. It shows that even the simplest method has the potential to match state-of-the-ar performance by simply inflating the dataset with generated data, and combining data inflation with advanced methods may lead to further improvements.
Learning Curve. We further examine the learning process with and without data inflation in Figure 66by. Interestingly, we observe that vanilla inflation is inferior to non-inflated training in most time and only achieves small improvement last (when non-inflated training saturates). In compari- son, AdaInf can consistently outperform the standard non non-inflated training across the entire training process, and continue to bring improvements when non-inflated training saturates.
Ablation Study. We study the influence of the three components of AdaInf in Table aa gener- ated data, data reweighting, weak augmentation. We can observe that while all three components contribute the final performance, their importance is: weak augmentation > data reweighting > generated data. In particular, the major improvement is brought by weak augmentation that solely brings around +2% accuracy. It shows that the interplay between inflated data and the learning al- gorithm has a large influence on the final performance and the adaptive co-design is very important.
Application to Data-scarce Scenarios. As generative models can provide a large amount of syn- thetic data, the proposed data inflation strategy can be particularly helpful when facing data scarcity issues. To show this benefit, we construct small dataset consisting of 5,000 images (1/10 1/10 size of CIFAR-101 10) by randomly sampling 500 images from each class of CIFAR- 10. Subsequently, we train an STF model on this subset (with 18.27 FID) and use the generated 1M samples for data inflation. As shown in Table 2 AdaInf obtains much higher linear accuracy than standard training (+4.32% accuracy), and also achieves better performance than vanilla inflation (+2.2%). Thus, AdaInf indeed simple and effective approach for data-scarce scenarios.
6 CONCLUSION
In this work, contrary to the common belief that generated data help representation learning, we show that they can be harmful when improperly used for contrastive learning. Upon our investigation, we have identified two sources of failure from the data inflation and data augmentation perspectives. To gain better understanding of these phenomena, we have provided rigorous theoretical explanations with careful examination of the generalization bounds. Based these observations, propose an adaptive data inflation strategy, Adaptive Inflation (AdaInf), that combines data reweighting and weak augmentations for inflated contrastive learning. Experiments show that this simple data-cente centrac strategy brings significant improvements over the standard training and the vanilla inflation method without any additional cost, especially on data-scarce scenarios.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Yisen Wang was supported by National Key R&D Program China (2022D0110030), Na- tional Natural Science Foundation of China (62376010, 92370129), Beijing Nova Program (20230484344), and CCF-BaiChuan-Fitee. Foundation Model Fund.
REFERENCES
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08846, 2023.
Dmitry Baranchuk, Andrey Voynow, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. In ICLR, 2022.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: Training models from generated images. In ICASSP, 2020.
Sam Bond-Taylor Adam Leach, Yang Long, and Chris G. Willcocks. Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. Transactions Pattern Analysis and Machine Intelligence, 44( 44(11):7327-7347, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-vuue vision transformers. In ICCV, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI., 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Fan Chung and Paul Horn. The spectral gap random subgraph of graph. Internet Mathematics, 4(2-3):225-244, 2007.
Fan RK Chung. Spectral graph theory. volume 92. American Mathematical Soc., 1997.
Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision helping contrastive learning. In ICML, 2023.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23:56:1-56:6, 2022.
Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, 2015.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self superviee learning. In ICLR, 2023.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In NeurIPS, 2014.
Grill, Florian Strub, Florent Altché, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, C. Doersch, Bermardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: approach self-supervised learning. In NewrIPS, 2020.
10

---

Published as a conference paper at ICLR 2024
Xi Guo, Wei Wu, Dongliang Wang, Jing Su, Haisheng Su, Weihao Gan, Jian Huang, and Qin Yang. Learning video representations of human motion from synthetic data. CVPR, 2022.
Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning per- spective on oversmoothing and beyond. In ICLR, 2023.
Jeff Z HaoChen and Tengyu Ma. theoretical study of inductive biases in contrastive learning. In ICLR, 2023.
J.ff HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurlPS, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In CVPR, 2020.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, and Xiaojuan Qi. synthetic data from generative models ready for image recognition? ICLR, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Ali Jahanian, Xavier Puig. Yonglong Tian, and Phillip Isola. Generative models as data source for multiview representation learning. In ICLR, 2022.
Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking noise. In NeurlPS, 2021.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurlPS, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Rundong Luo, Yifei Wang, and Yisen Wang. Rethinking the effect of data augmentation adver- sarial contrastive learning. In ICLR, 2023.
Jianxin Ma, Shuai Bai, and Chang Zhou. Pretrained diffusion models for unified human motion synthesis. arXiv preprini arXiv:22/2.02887, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748. 2018.
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning using synthetic imagery. In CVPR, 2018.
Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa. Learning from synthetic data: Addressing domain shift for semantic segmentation. In CVPR, 2018.
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In ICML, 2022.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NewrIPS, 2020.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. arXiv preprint arXiv:2306.0m 00984, 2023.
Yuandong Tian, Xinlei Chen, and Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICMI, 2021.
11

---

Published as a conference paper at ICLR 2024
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In JCML, 2020.
Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurIPS, 2021a.
Yifei Wang, Yisen Wang. Jiansheng Yang. and Zhouchen Lin. Reparameterized sampling for gener- ative adversarial networks. In ECML-PKDD, 2021b.
Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. A unified contrastive mergy-bbssdd
model for understanding the generative ability of adversarial training. ICLR, 2022a.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022b.
Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang. Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023.
Yawen Wu, Zhepeng Wang, Dewen Zeng, Yiyu Shi, and Jingtong Hu. Synthetic data can also teach: Synthesizing effective data for unsupervised visual representation learning. In AAAI, 2023.
Yilun Xu, Shangyuan Tong, and Tommi S. Jaakkola. Stable target field for reduced variance score estimation in diffusion models. In ICLR, 2023.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML, 2023a.
Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable contrastive learning with automatic feature importance discovery. In NeurlPS, 2023b.
Zhijian Zhuo, Yifei Wang, Jinwen Ma. and Yisen Wang. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. In ICLR, 2023.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RELATED WORK
Learning with Generated Data. Using synthetic data from generative models has been explored in many scenarios (Sankaranarayanan et all 20IS: Dosovitskiy et al, 2015; Ma 2022; Guo et all, 2022: Baranchuk etal 2022). To address the distribution shift between generated data and real data, many works have focused on sampling methods of generative models (Hesmer all 2020 Wang caall 20216; He et all, 2023). Jurjo ct all (202TT) proposed that the color, image coherence, and FID of the generative model are important factors influencing the learning representation from generated data. Moreover, |Ren and Les (2018) proposed learning representations by predicting information (e.g., surface normal, depth, and instance contour) of generated data through self-supervised learn- ing. Recently, due to the rise of diffusion models that are able to synthesize high-quality images, generated data have also been extensively studied for enhancing representation learning. Datanian at (2022) and al at (2I22) show that using generated data alone can achieve comparable per- formance to real data for contrastive learning with proper configurations of the generative models. While major drawback of these methods is that they often require problem-specific designs of the sampling process which can be costly. Instead, we explore whether generated data with a standard sampling process can help contrastive learning by enlarging the dataset. AzizI et al (2123) recently show that this kind of generated data can significantly improve supervised learning by around I 1% accuracy on ImageNet. In view of these successes, the goal of this work is to deeply investigate how generated data influence contrastive learning, and provide theoretical explanations for these effects.
B ADDITIONAL RESULTS
Influence of Subsampling on Laplacian Eigenvalues. We randomly generate 10k datapoints from two-dimensional uniform distribution [0, 112 as overall dataset and sample the dataset with different sample ratios. We then construct the augmentation sub-graph by drawing edged A.j = for any data pair satisfying ||> æzyl22 0.05. Figure / shows the Laplacian eigenvalues of the subsampled graphs. We can observe a consistent trend that a smaller sampling ratio leads to a larger decrease of eigenvalues. Therefore, equivalently speaking, using stronger data inflation (i.e., increasing from a smaller sampling ratio (the original data) to 1 (inflated data)) can lead to a larger increase of eigenvalues, and thus bring larger improvements in graph connectivity.
Sampling Ratio
7%
10%
15%
Egenvaluss
0.10
20%
50%
0.08
1
0.06
0.64
0.02
20
30
40
50
Figure 7: The Laplacian eigenvalues A Aso under different sampling ratios.
Influence of the Scale of Generated Data. Figure E illustrates the impact of different scales of generated data on the linear accuracy SimCLR. Consistent with the results shown in Figure 2bb the comparison of results between "No Replication" and "10:1 Replication" in Figure & further high- lights the significance of replication (Data Reweighting) in mitigating the distribution shift between real data and generated data. Additionally, the results indicate that using 1M generated data samples the optimal scale when employing the AdaInf strategy. Notably, using a larger size of generated data beyond 1M when 10:1 replication actually leads to worse performance. This is because the same replication leads to a smaller mixing ratio 3 with a larger size of generated data, leading to a larger distribution gap between the real and generated data, explained in Section 4. Therefore,
13

---

Published as a conference paper at ICLR 2024
if we want utilize larger amount of generated data than 1M, should use larger replication to maintain the same mixing ratio B.
93.5
Linea Accuracy (%)
92.5
92.0
No Replication
10:1 Replication
0.2M
1M
2M
SM
Generated
Figure 8: Linear accuracy on inflated dataset with different scale generated data.
Augmentation Matters. Figure a(a investigates the optimal minimal cropped size for different- scale datasets. Based on Figure 5(a
Table investigates the effects of both the minimal cropped size and color jitter which play crucial role in contrastive learning when inflating the original dataset. The results demonstrate that the standard augmentation (the first row) is suitable for no inflation dataset (baseline). However, the weak augmentation method proves be more effective compared to the standard augmentation. Specifically, when the crop minimum scale is set to 0.2 and the color jitter probability is 0.4 with strength of 0.5 (relative). SimCLR achieves the best performance on inflated data. Once again, these results highlight the complementary effects between data inflation and data augmentation.
Table Linear accuracy (%) of SimCLR with diffcrent augmentation strength.
Color Jitter
Dataset
RRC Min Scale
CIFAR-10 (Baseline)
Inflated
Prob
Strength
CIFAR-10
0.08
0.8
1x
91.33
91.92
0.2
0.8
1x
90.81
92.98
0.3
0.8
1x
88.27
92.68
0.2
0.8
0.5x
91.01
93.43
0.2
0.4
1x
90.89
93.11
0.2
0.4
0.5x
90.96
93.57
The Optimal Mixing Ratio Depends the Quality of Generated Data. Figure El shows the impact of data quality on the optimal mixing ratio. For generated data from STF (FID = 1.94). the optimal replication
which means a real sample roughly worth 10 generated samples under weak augmentation. However, for lower quality generated data from DDPM (FID=3.04), the optimal replication 15 which means real sample is roughly worth 15 generated samples. The result is consistent with Theorem I.ll which suggests adjusting the mixing ratio / based on the magnitude of the distribution gap.
Performance on High- Righ-Resoluttin Images. To further validate the relationship between data scale and data augmentation on high-resolution images, we compared the optimal augmentation strength between two different scale datasets: ImageNet100 (1300 images for each class) and 10% Ima- geNet100 (randomly sampled 10% of from each class of ImageNet100, 130 images for each class). The Table gl shows that for 10% ImageNet100, the model performs better with RRC min scale of 0.04 compared to 0.08. However, for ImageNet100, which scale is larger than 10% ImageNet100 weaker augmentation (RRC min scale of 0.08) achieved better performance.
The Influence of Augmentation under Diffferent Training Steps. Figure IG shows the influence of augmentation under different training steps. We see that under different training steps, using
14

---

Published as a conference paper at ICLR 2024
93.50
93.25
Linerr Accuracy (S3)
93.00
Difusion Models
92.50
DDPM13.04FID)
92.00
10
15
20
Replicating times
Figure 9: Linear accuracy on different inflated datasets under different data reweighting.
Table Linear accuracy with different augmentation strengths on ImageNet100 (1300 images for each class) and 10% ImageNet100 (randomly sampled 10% of from each class of ImageNet 130 images for each class), by changing the min scale of random resized cropping (low value represents stronger augmentation).
RRC min scale
10% ImageNet100
ImageNet100
0.04
47.24
71.26
0.08
45.34
72.76
weak augmentation after data inflation effective. Additionally, we can see that when training is insufficient (fewer steps), generated data tend to help than hurt the performance. Instead, as training converges (100k steps 1000 epochs). "Inf & Standard Augmentation" underperforms no inflation but our AdaInf still performs well.
Inf Standard
92
Inf Weak Aug
Accuracy
90
88
86
04
82
$9.4559
80
10k
30k
100k
Steps
Figure 10: The influence of augmentation under different trainging steps. The experimental configu- ration for "Inf & Standard Aug" involves applying standard augmentation the inflated CIFAR- 10. The same applies to "Inf & weak augmentation". The generated data is from DDPM (FID=3.04).
Comparison between GAN and Diffusion Model. There are mainly two types of generated models with SOTA performance CIFAR-I0: 10: diffusion model and GANs currently. Table E provides a comparison data inflation on CIFAR-10 with generated data from StyleGAN2-ADS (GAN) Karrasettal 222) and STF (diffusion model). We find that similar DDPM. vanilla inflation leads to worse performance, while AdaInf can bring significant improvements. Comparably, diffusion models with better generation quality (e.g., STF with lower FID) can achieve better accuracy.
Pretraining Cost of the Generative Model. We conduct a time test for pretraining diffusion models with 4 NVIDIA GTX 3090 GPUs CIFAR-10, and the total training time is shown in Table E. We
15

---

Published as a conference paper at ICLR 2024
Table 5: Linear accuracy (%) of SimCLR with generated data from GAN and diffusion model.
Model
FID
No Inflation
Vanilla Inflation
AdaInf(ours)
StyleGAN2-ADA
2.50
91.33
90.06 91.35
91.93 (+1.87)
STF (diffusion)
1.94
91.33
93.57 (+2.02)
can see that models with better quality (EDM, STF) generally require longer training. In practice, since these models have public checkpoints, we do not need train on our own for CIFAR-10.
Table 6: Pretraining cost of generative models. All of the models are pretrained with NVIDIA
GTX 3090 GPUs
Model
DDPM (3.04FID)
STF (1.94FID)
EDM (1.96FI)
Training Time
71h
87h
91h
C EXPERIMENTAL DETAILS
Unless otherwise specified, experiments are conducted using the default configuration in the code- base of the solo-learn library (daCCosaa et aL, 2022).. The CIFAR-10 dataset served as the original dataset and 1M generated data is generated by STF (1.94 FID). We adopt SimCLR with backone Resnet-18 for contrasive learning and pre-train train for 100,000 steps. In AdaInf, weak augmentation used with a minimal cropped (relative) area size of 0.2 and a color jitter probability of 0.4. The values for brightness, contrast, and saturation of color jitter is 0.4 and the hue value is 0.1. Other hy- perparameters of weak augmentation remained consistent with the standard augmentation provided by the solo-learn library.
For the experiment in Figure 2a the dataset consisted of CIFAR-10 and 1M generated data. The generated data from STF was generated using the provided checkpoint from Xu all (2023). The generated data from EDM (1.96 FID) and DDPM was generated using the provided checkpoint from Karras et all (2022). Additionally, the generated data for EDM (7.29 FID) was generated using shorter training time checkpoint, where only 5017k images were trained.
For the experiment in Figure B(a), the augmentation method modified the minimal cropped (rela- tive) area size based on the standard augmentation. The dataset "Half CIFAR- 10" was created by randomly selecting 2,500 images from each class of CIFAR-10. The dataset "CIFAR-10" + 1M" con- sisted of CIFAR-I0 data and I million generated data from STF and "CIFAR-10 10 0.IM" consisted of CIFAR- 10 and 0. million generated data from STE.
For the experiment in Figure 3(b), the dataset for the Vanilla method consisted of CIFAR-10 data and I million generated data, with standard augmentation. The dataset for the AdaInf method consisted of CIFAR- 10 data replicated 10 times and I million generated data, with weak augmentation.
For the experiment in Table [[] the generated data for CIFAR- 100 is from STF, while the generated data for Tiny-Imaged ImageNet is from DDPM. The configuration of SimCLR trained on ImageNet followed the settings provided by the solo-learn library for imagenet-100
For the experiment in Table 2b sampled 5,00 images from each class of CIFAR-10 10 create small-scale dataset. We then generated 100,000 images by STF trained on this small-scale dataset, which is used for data inflation. The training steps is 10,000.
16

---

Published as a conference paper at ICLR 2024
D OMITTED PROOF
D.1 PROOF OF THRORFM
Proof. Since P. 3Pa + 111-B),P and 0 0 8 S 1, we have
(rr(Pe, Pa)
/ [P((z)) - Pa(z)|dr
/ [8Pa +(1+11111111111 (1- - Pa(x))dz
(4)
-(1 3) / (Pal)()))dd dz
(1 3) D Drv Drv(Pg Pa): Pa).
which completes the proof.
D.2 PROOF OF THEOREM
Proof. We define linear function with weights B Rex (r represents the number of dataset
classes) as fF.8 R$ + y on top of pretrained features f(x) E R to predict the labels y E y of augmentated data, And we define
(T. == arg max Pr
(gg,B(r)= i)
(5)
EE(r] z~d((
to predict the labels y of real data.
To describe the difference between labels of two augmented data of the same natural datapoint, we define function:
ΦV
Σ
AZz 18((x) + ((x)))
(6)
We have
Σ Agd' 1[v(z) y(0')]
222' Ex
EENP,
A(x()))A<<|bb) 1p((x) # v(x')
< SERPP
Σ [A(r|T)A(<<1) (8) (1b()) # v()) + 1b((2') # s(a))
(7)
2,2'EX
2. =
EE P. Σ [(zz()) 1bx()) [())]
ZEX
= 2 Eg EE 11
12) (g(x) # v())
2a
Lemma D.I (Theorem B.3 in HaoChen et all (2022)) Assume the of augmented data X is finite. Let f E arg xxx be minimizer of the population spectral contrastive loss L(f) with k E zt. Then, for any labeling function X + [r] there exists linear probe B B with norm |B* F I 1/(1 Aτ) such that
OV 44(y 9)
B" (²) S
8
dit1
where T(7) is the one-hot embedding of v(σ) and Ayy 1)
2- (u(x)) + ((x)) denotes
Pr
the average disagreement between ÿ and the ground-truth labeling y- Furthermore, the error can be bounded by
200 dR+1
Pr A((1)
(B (x) # y(z)) S
+ 8a.
(9)
17

---

Published as a conference paper at ICLR 2024
According to the definition of (Ø), BB* (ã) * y(ū) happens only if more than half of the augmentations of I predicts differently from y()). Formally, for any ā Pt with -y- (ā)) + (z), according to the definition of "r-,, (I), we must have
Pr (g)-BB(x)) # v())) 2 0.5.
Thus we have
8a A&+1
B-P. (z) (())) 2
Pr
Pr A-Pr)
(gr- (x)) + v())) -
+ 16a.
(10)
At last, we notice that the following result obtained when the training and test data are from the same distribution. When considering generalization from the training distribution P. to the testing distribution Pa. we have
Pr (-))) + v()) B~Pa
Pa(c) (f)) + v())ddz
(PP(t)) P.(2) P. + y())ddz
$ / [Pa(s)) P. (- v / P((x)llgr-BB-33 (8)) + (())dda
(11)
$ / [Pa(x) - + / P((z)110r B- (T)) + v())ddE
-2D Pt) Pr (ğr- B (5) {
&a 16a.
22 - D Py) +
AR+1
In the last equation, combine the result of Theorem B and the generalization bound on training data (Eq. IT))
E EXAMPLES OF GENERATED DATA
For concrete understanding, provide examples of the generated data with different diffusion models on different datasets below. can seen that the generated data used in our experiments indeed look very similar to the real data, and models with lower FID indeed have fewer artifacts.
Row Data
EDM
(FID=8.85)
DDPM (FID-3.04)
EDM (HD
STF
(.94)
ADA
(FID-2.50)
Figure 11: CIFAR- 10 examples.
18

---

Published as a conference paper at ICLR 2024
STF (HID=3.14)
Figure 12: CIFAR-100 100 examples.
Raw DaEa
DDPM (F10=18661)
Figure 13: Tiny ImageNet examples.
19


Reviews:
[{"review_id": "S7TjW6kcvw", "reviewer": "Reviewer_gCqn", "summary": "The paper challenges the common belief of generated data being always helpful to contrastive learning and empirically show that this need not always be the case. The paper identifies, two issues that control if generated data would be helpful or not to learn good representations, namely the discrepancy between generated and real data distributions and the mixing ratio of real and generated data. Based on these insights, the authors propose AdaInf which does a grid search over different mixing ratios and augmentation levels to find the best combination for downstream tasks. The authors also give theoretical justifications as to the empirical trends they observe.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The insights and results provided by this paper seem novel, creative and significant. I believe the results would greatly help in our understanding of contrastive learning both theoretically and empirically.\n\nThe paper is very well-written in general except some proofs in the appendix which I will describe in detail in the weaknesses section.", "weaknesses": "The proposed Adainf method seems to rely on the downstream task to find the optimal weighing factor and augmentation strength. This might limit it's use in practice since the goal of self supervised learning is to learn a good representation from training data that can be useful in future downstream tasks, I do not know a priori. Some discussion on this would be useful, either as a limitation or clarifying the exposition in case I misunderstood something.\n\nEq (7) in proof of theorem 4.1: I think the steps need to be explained. It is not clear to me how the indicator function is swapped from I(y(x) \\neq y(x')) to I(y(x) \\neq y(\\bar{x})) in the second equality, why is the first inequality true, and how the sum disappeared in the third equality. \n\nWhat is the difference between the definition of \\alpha and equation 8? Shouldn't they be equal?\n\nThe statement of theorem 4.1 should be qualified with what exactly is the optimal encoder? What is the objective here? Lemma D.1 indicates this is some population spectral contrastive loss. The precise definition of this loss should be made clear in the paper. Similarly, eq 5 should be in the main text since it defines the prediction function using the linear classifier on top of Pretrained features. The paper makes no mention that this is really majority voting over prediction made on all possible augmentations of the sample point. \n\nI would suggest the authors use Markov's inequality to obtain equation 11 to make the proof rigorous. \n\nMinor Typo: Eq (12) second equality, shouldn't it be P_t instead of P_g?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wHAsuBkZ5u", "reviewer": "Reviewer_17xG", "summary": "The paper studies whether synthetic data generated with a diffusion model can help self-supervised learning. By reweighting the ratio of generated and real data, and re-tuning the augmentation strength used on the positive pairs, the authors are able to show that gains are possible on CIFAR10, CIFAR100 and Tiny Imagenet.", "questions": "- could you add some samples from the diffusion models to the supplement for a better visual impression at different FID levels? ideally compared to a set of samples from the real datasets.\n- do you use different diffusion models (with different training datasets) for each of the considered subtasks? in particular, is e.g. the CIFAR model only trained on CIFAR, etc?\n- Sec 4.2, how is $P_d \\approx P_g$ a reasonable assumption in the light of your argument for stating Theorem 3.1?\n- How do you connect Lemma 4.2 to your observed empirical results, what is it signifiance?\n- In Fig 6., Vanilla data inflation *always* helps or keeps results the same. How does this relate to the statements in Figure 3(b)? It seems like the number of training steps is an important confounder we need to control for?\n- datasets like ImageNet have a \"photographer\"/object centric bias. Do the generated samples also have such a bias? If not, could this be a possible explanation for the difference in the optimal augmentation strategy?\n- Is the augmentation strategy in the base setup optimal with respect to the selection metric? If so, how was this confirmed?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is generally well written, and the results are interesting (albeit limited in scale, see below, which limits signficance). Some theoretical intuition is given, although clarity in the connection of theory and empirical results could be improved.", "weaknesses": "Theory and empirical experiments are not well connected, in my opinion (I would be happy to get corrected on this during the rebuttal). Here is how I read the paper: Leveraging synthetic data requires to define how it will be mixed with real data. Also, as image statistics of synthetic data might be different from real data*, the existing augmentations applied to contrastive learning might be suboptiomal --- hence, re-tuning both aspects of the model could help to boost performance. If the authors would have a better way to connect theory and empiricaly observations, this could be interesting --- but in the current outline of the paper, both tracks seem pretty independent to me.\n\nAnother weakness are the empirical results: These results are fairly small scale with limited evaluation of different generative models. A lot of evaluations are performed on CIFAR-10 where one would expect the possibility to run extensive experiments and analysis of various factors. To make the statements in the paper more convincing, it would help to extend these experiments, compute error bars, study the influence of other parameters, and extend the most promising models towards full ImageNet-scale.\n\nAdditional weaknesses/concerns:\n\n- The InfoNCE loss on (1) is incorrect. It misses a \"-\" in front of the positive loss, and also typically in InfoNCE, there is a sum vs. a mean over the negative examples, as the number of negatives influences embedding quality (this only yields a $\\log N$ difference in the value of the loss but does not influence the gradients) --- I think it is still better to re-state the loss as common in the literature.\n- \"significantly\" is used at different places in the paper. Please check the usage, and remove where it is not backed up by a statistical test. I would strongly encourage the authors to run multiple (I would suggest at least 5) seeds of their models and report error bars on every CIFAR-scale experiment reported in the paper.\n- in Table 1, the FID scores of the used generative models on the respective dataset should be added, this seems to be a major influence on experiment outcome.\n\nMinor\n\n- Theorem 3.1, was the total variation introduced before? If not, clarify the abbreviation (TV) where it first appears in the text.\n- for the theorems, it might be useful to link to the respective section where the proof is stated, for easier navigation through the document\n- The notation in 4.1 should be double checked. E.g. $\\mathcal{\\overline{X}} = {\\overline{x}}$, is that correct? or $D_{xx}$.\n- Sec 4.2, \"an downstream\" -> \"a downstream\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgG1DeniJt", "reviewer": "Reviewer_pGvU", "summary": "This paper discovered the failure mode of data inflation, a technique that augments real data with synthetic data, for contrastive learning. The finding was that data reweighting and weak augmentation can help improve the performance of contrastive learning on the combined dataset. Based on that, they proposed a method, called Adaptive Inflation (AdaInf), to train contrastive learning by adaptively adjusting augmentation strength and mixing ratio for data inflation. In addition, the authors provided the first theoretical guarantees for inflated contrastive learning. Empirically, AdaInf improved downstream accuracy significantly.", "questions": "- Eq. (1) is wrong. In addition, the authors should cite this paper for Eq (1): “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere”.\n- Eq. (2) is also wrong. Again, the citations are not chosen well. Ho et. al 2020 should be a better reference here.\n- Why do we need to sample 1M data while CIFAR-10 has only 60k samples? Speaking of which, instead of increasing $\\beta$ we can decrease $|\\mathcal{D}_g|$. \n- Section 3.1: Is the ratio $10:1$ true for all datasets? And is it universal for different diffusion models?\n- Section 3.2: Are there any intuitions or reasonings why larger data size needs weaker data augmentation? Or do generated images need weaker data augmentation and they dominate the inflated data set?\n- Section 3.3: How to choose the correct strength of data augmentation in AdaInf? The word \"adaptive\" seems misleading here because the hyperparameters are chosen manually rather than auto adjusted or learned.\n- Section 5.1: Why is the min scale of random resized cropping (0.2) different from the optimal value (0.3) in Figure 3?\n- The result in Table 3 is slightly different from that in Figure 3 where 0.3 yields the best performance.\n- Figure 8: Is the replication N fixed to 10? Again, the question is whether N = 10 is universal.\n- The labeling error definition seems weird. It is better to define it as Eq (8). Then, the term $\\phi^y$ in Eq (6)  is less than $2 \\alpha$. If the definition is kept as in Theorem 4.1, we have $\\phi^y = \\alpha$. Note that $\\alpha$ in HaoChen et al. 2021a has a different meaning.\n\n**Minors**:\n- The usage of $B^\\star$ can be safely removed.\n- There are a lot of typos: Diffusion models in Section 2, Theorem 4.1, Lemma 4.2, Eq (7), Eq (9), and Eq (12) just to name a few.\n- Notation:\n  - N is used for replication times in Section 3.1 but later used in Section 4.1 with different meanings. What is the meaning of N in Section 4.1?\n  - Eq (6), what is $\\mathcal{X}$?\n  - Is $w_{x x'}$ defined as $A_{x, x'}$ in the main text?\n  - Overall, the notation is similar to HaoChen et al. 2021a. However, the way the authors use them is not good and sufficient.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper is well-structured.\n- The theoretical results and findings are interesting.\n- The experimental design is good. Experiments and ablation studies successfully verify theoretical results.", "weaknesses": "- The preliminary section is not well-written.\n- There is a concern about the generality of AdaInf. It seems that the method is hand-crafted rather than adaptive.\n- There is a lack of motivation for considering diffusion models in this paper. All theoretical results can be generalized to any generative model.\n- Some notation and experiment results are not consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IA0tW2lc6d", "reviewer": "Reviewer_umDF", "summary": "This paper studies the problem of using of synthetic data from generative models (e.g. diffusion models) for contrastive learning (CL). Firstly it finds that naively inflating CIFAR-10 with 1M generated images can in fact hurt linear probe performance of CL. The paper delves deeper in the role of (a) weight of inflated data, (b) augmentation strength. Interestingly the paper finds two simple fixes, (a) upweighting real data by 10x, (b) using weaker augmentations with more data. Based on this finding, the paper proposes adaptive Inflation (AdaInf) strategy that adjusts augmentation strength based on dataset size. Experiments on CIFAR and TinyImageNet evaluate AdaInf data strategy with SimCLR, MoCo V2, BYOL, Barlow Twins. Overall AdaInf does better than no data inflation and data inflation w/ standard augmentation. It achieves accuracy as high as 94.7% on CIFAR-10 with SimCLR. Ablations on smaller training horizon and smaller training data further shows the benefit of AdaInf.\n\nThe paper provides theoretical analysis using augmentation graph framework introduced in HaoChen et al. and shows a complementary benefit of inflation and augmentation. Stronger augmentation improve \"graph connectivity\" at the cost of \"labeling error\". More data also improves graph connectivity, thus we can get away with weaker augmentation that has smaller labeling error. Synthetic example with gaussian distribution verifies this insight.", "questions": "- Role of reweighting: Theorem 3.1 and 4.1 do not fully explain that there’s an optimal weighting. In light of mismatch between $P_s$ and $P_d$ Theorem 3.1 basically suggests that \\beta=1 is best. I believe the argument in Theorem 4.1 is that $\\lambda_{k+1}$ is larger when $P_g$ has larger support. It would be nice to make this point more clear. Overall it would help to more carefully deal with the case of $P_s \\neq P_g$ and why would one pick $\\beta < 1$.\n\n- Lemma 4.2 uses random sub-graph with probability $p$ of selecting each edge. Is random edge selection reflective of subsampling data? Random node selection seems more appropriate.\n\n- Thm 4.1 argues about connectivity and non-zero $\\lambda_{k+1}$ for the training set, but Saunshi et al. 2022 show lack of connectivity in the training set for standard augmentations, implying $\\lambda_{k+1}=0$. This would make the bound in Theorem 4.1 vacuous without any . Does that contradict the claims? The results in HaoChen et al. are non-vacuous because that uses the eigenvalue of the population distribution in the bound rather than eigenvalue of training data.\n\n\nSaunshi et al. 2022. Understanding Contrastive Learning Requires Incorporating Inductive Biases", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Significance: The paper studies an interesting problem of role of synthetic data from generative models on contrastive learning. Given the increasing of procuring high quality data, effectively using synthetic data is a important problem to study\n\nNovelty: The perspective on interplay between larger data and augmentation strength is interesting and novel to my knowledge. The value of real data being more than synthetic data is intuitive but not surprising. The theory, although a simple application of HaoChen et al., does provide interesting insights.\n\nClarity: The paper is overall well written and easy to follow. Experimental results are also presented well.", "weaknesses": "- It would have been useful to have some more discussion on the issues of using synthetic data, e.g. bias\n\n- Some clarification and cleanup of the theory parts would be useful. See comments below\n\nOverall the paper makes a positive contribution in studying the interplay between synthetic data and augmentation for contrastive learning. I have currently assigned weak accept, but I'd be happy to raise the score after clarification on the theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HwklvR4vtl", "reviewer": "Reviewer_XrzN", "summary": "This paper explored the reason for the performance degradation when utilizing synthetic data from a generative model to enhance contrastive learning (CL). The authors found that the strengths of the data augmentation and the mixing ratio or the size of data inflation are the two critical factors that affect performance through theoretical analysis and empirical results. Based on the observation, the author proposed AdaInf, which adapts the weight of synthetic data and the strengths of the data augmentation to achieve the best performance. Experimental results on multiple datasets showed that AdaInf can outperform the plain CL method and the CL method with naive data inflation.", "questions": "The questions of the paper are listed as follows.   \n1. For equation (1), should there be a negative sign before the first term?   \n2. Are there any experiments to show that the two tricks in AdaInf can also improve the performance of CL with data generated by more generative models like GANs?   \n3. On page 3, there is a statement that better generative quality often requires larger models and/or slower sampling. What does the slower sampling mean?   \n4. Are there any experiments to show the trade-off between the size of generated data for inflation and the strength of augmentation on real-world datasets? It looks like all the experiments used the same size of generated data.   \n5. The purpose of Figure 4 is a little bit confusing to me. Why can this figure be used to show stronger augmentations create more overlap between different training samples?  \n6. For the experiments of Figure 6, is the batch size for CL, CL with vanilla data augmentation and CL with AdaInf the same? If this is the case, does it mean the training cost is the same when the total training steps are the same for the three methods?   \n7. What is the pertaining cost of the diffusion model on the evaluated datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are listed as follows.   \n1. This paper is well-motivated. Data inflation is an important method to improve the performance of CL. The reason for the unexpected performance degradation is worth exploring.   \n2. The conclusions about the data inflation and data augmentation are analyzed theoretically and proved with a simple synthetic dataset empirically.    \n3. The paper is well-written, where the problem is clearly defined and the analysis is easy to follow.", "weaknesses": "The weaknesses of the paper are listed as follows.   \n1. It would be better if the authors could compare AdaInf with the other existing data inflation methods for CL.   \n2. The description of AdaInf is in qualitative way. For example, it claims to use different weights on real and generated data and adopt milder data augmentation. However, how to get an appropriate weighted and the strengths of the data augmentation remains unknown.  \n3. It would be better if the authors could provide some experiments on more generative models like GAN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)