Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences
Authors: Damien Ferbach
Quentin Bertrand
Joey Bose
Gauthier Gidel
Venue: NeurIPS 2024 spotlight (2024)
Keywords: retraining
curating
generative model
self-consuming
Primary area: generative_models
TL;DR: We theoretically show that iteratively retraining a generative model on its own generated samples with human curation optimizes human preferences

Abstract: The rapid progress in generative models has resulted in impressive leaps in generation quality, blurring the lines between synthetic and real data. Web-scale datasets are now prone to the inevitable contamination by synthetic data, directly impacting the training of future generated models.
    Already, some theoretical results on self-consuming generative models (a.k.a., iterative retraining) have emerged in the literature, showcasing that either model collapse or stability could be possible depending on the fraction of generated data used at each retraining step.
    However, in practice, synthetic data is often subject to human feedback and curated by users before being used and uploaded online. For instance, many interfaces of popular text-to-image generative models, such as Stable Diffusion or Midjourney, produce several variations of an image for a given query which can eventually be curated by the users.
    In this paper, we theoretically study the impact of data curation on iterated retraining of generative models and show that it can be seen as an implicit preference optimization mechanism. However, unlike standard preference optimization, the generative model does not have access to the reward function or negative samples needed for pairwise comparisons. Moreover, our study doesn't require access to the density function, only to samples. We prove that, if the data is curated according to a reward model, then the expected reward of the iterative retraining procedure is maximized. We further provide theoretical results on the stability of the retraining loop when using a positive fraction of real data at each step. Finally, we conduct illustrative experiments on both synthetic datasets and on CIFAR10 showing that such a procedure amplifies biases of the reward model.

Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences
Damien Ferbach 2.2 Quentin Bertrand', Avishek Joey Bose 1.3 Gauthier Gidel Mila, Université de Montréal Eoole Normale Supérieure de Paris University of Oxford, 4 Canada CIFAR Al Chair
Abstract
The rapid progress in generative models has resulted in impressive leaps in genera- tion quality, blurring the lines between synthetic and real data. Web-scale datasets are now prone to the inevitable contamination by synthetic data, directly impact- ing the training of future generated models. Already, some theoretical results on self-consuming generative models (a.k.a., iterative retraining) have emerged in the literature, showcasing that either model collapse or stability could be possible depending the fraction of generated data used at cach retraining step. However, in practice, synthetic data often subject to human feedback and curated by users before being used and uploaded online. For instance, many interfaces of popular text-to-image image generative models, such as Stable Diffusion or Midjourney, produce several variations of image for given query which can eventually be curated by the users. In this paper, we theoretically study the impact of data curation iterated retraining of generative models and show that it can be implicit preference optimization mechanism. However, unlike standard preference optimization, the generative model does not have access the reward function negative samples needed for pairwise comparisons. Moreover, our study doesn't ' require access the density function, only samples. We prove that, if the data is curated according to reward model, then the expected reward of the iterative retraining procedure maximized. We further provide theoretical results on the stability of the retraining loop when using positive fraction of real data at each step. Finally, we conduct illustrative experiments on both synthetic datasets and on CIFARIO showing that such procedure amplifies biases of the reward model.
Introduction
Today state-of-the-arr generative models can produce multi-modal generations virtually indistin- guishable from human-created content, like text (Achiam al., 2023), images (Stability AI, 2023), audio (Borsos et al., 2023), or videos (Villegas et al., 2022; Brooks et al., 2024). The democratiza- tion these powerful models by open-sourcing their weights (Stability AI, 2023; Jiang et al., 2023; Touvron et al., 2023) allowing public inference (Ramesh al., 2021; Midjourney, 2023; Achiam al., 2023) paves the way to creating synthetic content at an unprecedented scale-the results in- evitably populate the Web. In particular, existing datasets are already composed of synthetic data such as JourneyDB (Pan et al., 2023) and SAC (Pressman et al., 2022). Moreover, Alemohammad et al. (2024, Fig. showed LAION-5B (Schuhmann et al., 2022), large-scale Web-crawled dataset used to train future generative models, already contains synthetically generated images.
There is strong evidence that the synthetic data on the web has been, to large extent, curated by human users. For instance, the LAION-Aestheticc datasets contains synthetically generated
Corresponding author: ferbach. dami enw.main....mmmmmmmmmmmmmmmmm..m com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1. User picks prompts Proeo())
Train the model P++1 on dataset D
Only the upscaled samples are in the dataset 2
2. Sample: x1 11.X2. K3.K, p)(x(x)
REPEAT
Sample x is upscaled by User with probubility: (15)
En (AA
Figure 1: Illustration of the curation phenomenon: 1. User proposes prompts such as "butterfly going to the bathroom". Four images generated with Midjourney, 3. User only upscale (e.g. the top left image) image, 4. Solely upscaled images are incorporated into the JourneyDB dataset (Pan et al., 2023). Samples from other diffusion models can found in Figures 12a 12b.
images that have been curated using a reward model learned from human feedback on the Simulacra Aesthetic Captions dataset (Pressman et al., 2022). Additionally, the JourneyDB dataset, contains human-picked images from the Midjourney (Midjourney 2023) discord server, that have been upscaled, i.e., images that have been requested in higher resolution (see Figure 1). More generally, the user interface of the most popular state-of-the-are text-to-image models (e.g., Midjourney and Stable Diffusion 2.1 Huggingface implementation) proposes four alternatives for single prompt for the user to pick their favorite.
While the consequences of iterative retraining of generative models on synthetically generated data have raised a lot of attention in the community (Alemohammad al., 2024; Shumailov al., 2023; Bertrand et al., 2024; Dohmatob et al., 2024a), previous works do not consider that generated data could be curated. This subtle nuance may of major importance. Indeed, in numerous applications, augmenting the datasets with curated synthetically generated data is found to enhance the performance of the downstream task (Azizi et al., 2023; Wang et al., 2023; Gillman et al., 2024) and even generative models themselves (Hemmat et al., 2023; Gulcehre et al., 2023), hinting that quality might not be the biggest problem. On the other hand, recent works Wyllie et al. (2024); Chen et al. (2024b) showed that this might lead new issues, such as bias amplification.
This is why, in this work, we aim theoretically understand how the process of curation of synthetic data is connected with the reward model underlying human preferences and what distribution is learned by generative models trained on such curated synthetic data.
Main contributions. We summarize our core contributions as the following:
We first focus on the self-consuming loop on (only) curated synthetic samples: we show that the expected reward gets maximized in Lemma 2.2 and that its variance vanishes. We further provide a convergence result in Theorem 2.
We additionally study the iterative retraining loop when real data re-injected at each step: we first improve previous results of stability using synthetic samples by Bertrand et al. (2024) and show convergence in Kullback-Leible (KL) divergence to the optimal distribution Theorem 2.2. When using curated synthetic samples, show that the KL divergence with the optimal distri- bution remains bounded Theorem 2.4, as well as an improvement on the expected reward Theo- rem 2.3, enlightening connections with Reinforcement Learning from Human Feedback (RLHF). We finally illustrate our theoretical results on synthetic datasets (mixtures of Gaussians and two moons) well as natural images CIFARIO in Section 4. We highlight how curation based on the confidence of a classifier can lead to the emergence of biases (Wyllie et al., 2024).
2 Iterative retraining with curated synthetic data
We now study the fully synthetic self- self-consuming loop with curated samples. Unlike previous works that do not take curation into account (Alemohammad et al., 2024; Shumailov et al., 2023) and focused on stability of the process (Bertrand et al., 2024), we show that retraining with curated samples both maximizes an underlying reward whose variance collapses, and converges to maximum reward regions. In Section 2.1 first specify explicitly the distribution induced

---

by discrete choice model and highlight connections with RLHF. We additionally show that the expected reward increases and that its variance vanishes Lemma 2.2. Finally, inspired by stability results of Bertrand et al. (2024), in Section 2.2 we extend our study to settings when real data is injected. More precisely, we improve previous results of stability of Bertrand et (2024) without curation and provide novel theoretical bounds when the synthetic data is curated.
Notation and conventions. Lowercase letters p denote densities while uppercase letters I indicate their associated probabilities. We denote Pdata E P(R")) the real data distribution and for t E N, we denote p. E ") the model distribution at step t the iterative retraining loop. Analogously, 0, is the corresponding parameters of the learned parametric model p:- We write Po indicate the initial model learned using maximum likelihond on Pdasa. this includes many modern generative model families such diffusion models (Ho et al., 2020; Song et al., 2021) and flow-matching methods (Lipman et al., 2022; Tong et al., 2023b).
Discrete K KCchoice model. We propose using discrete choice model for the curation phenomenon illustrated in Figure 1, where users pick their preferred image that will be upscaled in the next dataset. Modeling human preferences is usually done via the Luce choice rule model (Shepard, 1957; Luce et al., 1963; Dumoulin et al., 2023), where the human modeled as rational Bayesian subject. The probability that 21 is preferred to 22 is formulated using underlying reward function r(z) and Plackett-luuc model (equivalently Bradley-Terr model) (Bradley and Terry 1952). Under Luce's choice axiom (Luce, 1959), it is possible to generalize this formula to K > samples in Equation 2. For given samples 21...., IK drawn from Pt, the random curated sample denoted is chosen according to this Plackett-Luce model & ,, as in Equation 2 (Bradley and Terry, 1952; Luce, 1959; Plackett, 1975). In particular, the curation procedure can be summarized follows
1) Sample 2 ~ Pt...., 2K Pr. independently.
(1)
(
2) Pick IK( i.e. P@ I TK.,
xK)
1 K.
Σド= er(z,)
Self-consuming loop. After generating and curating a synthetic dataset according to Equations and 2, the next generation of generative models is trained either solely on the distribution of curated samples (A - xi), or on mixture of reference samples (that either comes from real data Pdara or reference generative model po) and synthetic curated samples (A 20) depending on the studied setting
Pi+1 arg maax
BEeeperr [log(()) +
A + X
E
21 "loeegge SnwP m.))))
[log(())
(3)
1 1+1
PEP
where P is the set of achievable distributions with our model. This work aims to study the retraining dynamics of the distribution defined Equation 3. First, in Section 2.1 we study the simplified dynamics of Equation 3 in the regime X - co, i.e, when solely retraining data and show convergence of the process but variance collapse. Section 2.2 we study the exact
curated synthetic
dynamics given in Equation 3 and the impact on the stability of retraining on mix of real data synthetic curated data.
2.1 Iterative retraining only on the curated synthetic samples
In this section, we study the dynamics of the density learned through iterative discrete K-choice curation in the sully-synthetic setting (i.e., + 20): Equation boils down
P141
arg max
E a Bimn xpp PEP a~PE(s
log p(t)
As a warm-up, we first consider the limit of K + do in Lemma 2.1 and draw explicit connections with RLHF. This simplification yields closed-form form formula form for the solution of Equation and provides intuitions for the dynamics learning on curated samples.
3

---

Lemma 2.1. Let p++1 be defined as in Equation 4. P (IR") is the set of probability
distributions on R.. and we we assume that E Eyup. [e < oc, then we have for all a E R
en(e)
Pt+1(z) K pc(x) Bž-pe [er(#)]
(5)
Dependency on K and connection to RLHE.. The proof of Lemma 2.1 relies the fact that can obtain closed-form formula for the density P++1 induced from discrete K-choice curation on Pt (Equation 4). This is done in Appendix A.4.1 where show that its density can be written
K ke) (x(,)) + E cr(os)
(x) = P(()) H (x), with H (x) = ExI. -EII..RRIIM
(6)
The latter directly implies that for all K 2 1, H (x) (0, K). In particular, small values of K act as a regularization which prevents the density from blowing up too much high rewards areas. On the other hand, the higher the number of samples used for curation, the more it can affect the induced distribution. In the limit K co, Lemma 2.1 shows an interesting connection between iterative retraining curated data and reward maximization via RLHE. Given hupervised-finetunee model distribution SFT and a regularization parameter 3, the goal of RLHF to find a policy that maximizes a reward r(z) fitted on human preferences:
RLHF arg max Exws [r(x)] - 3DL (H|[TSPT), which has closed form formula,
RLHF (x) x π
SPT (x)e "0/// (Go al., 2023; Rafailov al., 2024).
Therefore, in the limit K + oc, Equation 5 shows that performing iterative retraining with human curation for iterations equivalent to performing RLHF with hyperparameter 3 from the initial distribution xSFT := Pn- The corresponding regularization parameter B inversely proportional to the number of retraining steps. This connection is surprising since performing maximum likelihood on a curated distribution (Equation 3) a priori different than directly maximizing reward with Kullback-Leibler (KL. regularization.
To prove that curation both increases the expected reward and reduces the variance, we will need an additional assumption that the reward is bounded at initialization. We decompose this assumption into three sub-assumptions of increasing thrength:
Assumption 2.1. and Eple"(a) ((x) < co.
The distribution p P(TR)) has a density w.r.t. Lebesgue measure
B. The distribution E (((R)) has a density w.r.t. Lebesgue measure and there exists T. E R such that: (a) p-almost surely, r(x) S r. and (b) p puts positive mass in a neighborhood of r. i.e., Ve > 0, 0,P(r(x) 2 =) >>>
C. Assum. 2.1.B and P(r(x) r*) > 0.
In particular, Assumption 2.1 A and B are satisfied if we suppose that the reward bounded, which reasonable we suppose it continuous given that the set of images [0, 1]d compact. Note that assuming (a), that almost surely bounds the reward at initialization. On the other hand (b) imposes that r. the smallest value that a.s. upper-bounds the reward. In nutshell, r. should be thought as the smallest number that upper-bounds the random variable Po with probability 1. For example, a Uniform distribution on the interval [0, 10), r. 10 whereas for unbounded distributions such as N/(0, 1), r. does not exist. In other words, r. = inf(r e R, (o(r(() < r.)) 1}. This shows that r. is uniquely defined which is important point set r(z) = r,, in Lemma 2.2 and Theorem 2.1.
always choose r. such that (b) is satisfied by picking the smallest value
will show convergence of p, towards the level
Lemma 2.2 states the reward expectation increases proportionally to the reward variance.

---

Lemma 2.2. Let p4t1 be the distribution induced from a discrete choice model on Pt (Equa- tion 4). Suppose Assumption 2.1 B holds, then the expected reward increases proportionally its variance each retraining iteration:
Epics [erle)] 2 Epp
[erle)]
(crto)]
K-1
Varp, [e)(x)]
(7)
K
e²
Especially the expected reward converges to the maximum reward and its variance vanishes:
Ep. [x(a]] 100, e"-
and
Varp.
1-00 0
Discussion. Lemma 2.2 shows that the reward augmentation is directly proportional the reward variance at each retraining step. In other words, the more heterogeneous the reward is, the more its expectation increases the next step. Lemma 2.2 further shows that the expected reward converges towards the reward maximizers. We can additionally deduce that the variance is doomed to vanish. This is detailed in Appendix A.4.3 which additionally states that the reward variance decreases fast enough to have finite sum. Finally, we note that Lemma 2.2 helps of this process: due to the variance term in Equation 7, fixed point of the retraining loop must put mass on single level set of the reward function. The reciprocal is obviously true as detailed in the appendix (Lemma A.3).
understand the fixed points
We can finally show a stronger result of convergence for the Kullback-Leibler divergence. We will need to assume that at initialization, the probability density puts positive mass on the level set r(x) T.. This corresponds to Assumption 2.1 C. Without this assumption, the probability density support would consecutively vanish towards the maximizer of the reward preventing KL conver- gence. Under assumption 2.1 C, we can denote the probability density at initialization restricted the domain that maximizes the reward and renormalized: p. (x) = P
Po(r(a)
Theorem 2.1. Let all t 2 0, Pt-1 be the distribution induced from discrete choice model on Pt. (Equation 4) where = P(Rd) is the set of probability distributions on Rdd If po satisfies Assumption 2.1 C, then loop curated samples Pt converges to p+:
define pa (x) po(x))l. and the self-consuming Po(r(z)=r_)
DKL (au(ppllp)) 1000 O.
Theorem 2.1 proved in Appendix A.4.5 shows that the process of retraining with curation Equation eventually converges to the highest level set of the reward reached al initialization. In particular, in the limit of large number of retraining steps, the probability of all smaller rewards vanishes. This can have strong implications when retraining the next generation of generative models on curated Web-scaled dataset: the learned distribution will lose diversity and collapse to the highest reward samples.
2.2 Stability of iterative retraining on a mixture of real and synthetic data
After showing convergence but variance collapse of the self-consuming loop on curated synthetic samples, we now study the impact on the stability of injecting real data each step. This setting motivated by the recent work of Bertrand et al. (2024) that showed stability of the iterative retraining loop with real and synthetic data around local maximizer t of the training distribution likelihood. This setting is furthermore relevant since Web-scrolled datasets will presumably keep containing mixture of real data and human-curated synthetic data. In Section 2.2. we first improve previous results on retraining on mixed datasets which underlines the beneficial impact of real data on stability and in Section 2.2.2, we prove both stability and reward augmentation the setting of mixed real and curated synthetic data.
5

---

2.2.1 Iterative retraining without curation
To motivate the impact of real data on the stability of the retraining loop with curation, we focus first on its impact without curation and improve previous results in that setting in Theorem 2.2.
Setting. In this section only, following the approach of Bertrand et al. (2024), we will not assume infinite capacity for our distribution (i.e., + P(R)) and hence adopt parametric approach Pe = (pe E ).. Given the current generative model distribution Pa, must at the next iteration maximize the combined log- ligeiikel of real and generated data with hyperparameter X, l.., Equation 3 becomes:
pa. arg max
E Byaar [ogpo())) + 1 Eps, [log
X
Pe EPa
We finally denote po.
arg max Epaaia log maximizer of the data distribution log- PeEPe
likelihood. We also make the following assumption taken from Bertrand et al. (2024):
Assumption 2.2. For 0 close enough 0., the mapping
Vilog is L-
Lipschitz and the mapping 0 - Epana po(]] is continuously twice differentiable with Epaas V log pa(x)] -al O. Further suppose Wi Pdaaaa) S E, iee pa. is close to the data distribution Pdata-
Bertrand et al. (2024) proved stability of the retraining loop provided À is sufficiently small. How- A their proof is restricted to X < 2.2 preventing the use of fraction of synthetic data I+X bigger than one-third which they left as future work. In Theorem 2.2, we extend their proof to any fraction of synthetic data provided the best model distribution is sufficiently close to Pdata. in Wasserstein distance (Villani et al., 2009) i.e. Wi (pg., Pdata) S < f. Additionally, express the result in distribution, while they expressed parameter space.
Theorem 2.2. Under Assumption 2.2, if Le and X a then there exists neighbor- hood of the optimal distribution parameters 0, such that for any initial parameters 9g in that neighborhood, pa, converges pe. exponentially fast:
((-Nott2)) EL) {
21
DKL (1o.)))))))
a -el.)
2.2.2 Iterative retraining on a mixture of real and curated samples
Interestingly when curating the synthetic samples we cannot expect stability around the optimal distribution (f, in Theorem 2.2) since it is longer fixed point of the retraining loop. We will instead show a closeness result in KL divergence combined with an increasing property of the expectation of the reward, which bears close connections to RLHF. We therefore now study the setting described in Equation 3 where the synthetic samples are curated using a discrete K-choice model and real data is reused each step () co). In other words, suppose that the retraining step uses mixture of a reference distribution and a curated distribution as
(a) (x) +
A (x)) H (z)
(H K is defined in Equation 6)
In Theorem 2.3, we prove that when retraining on a mixture of curated samples and samples from the reference distribution, the reward increases with respect to the reference distribution:

---

Theorem 2.3. Let À 0 and consider the process (pt) defined in eq. 8, with Pu = Pref. If Prot satisfies Assumption 2.1 B, then for all 1:
(K- - Var [er(x)]
E Pr
(crte)
2 E Prot
(erta)]
(1+
Ker
Discussion. A first interesting case is taking the reference distribution Peef equal Pdata- In that case, we recover the fact that Pdata is not a fixed point of the retraining loop as soon as different reward values have non-zero probabilities to happen (we recover the result from Lemma A.3). In fact, Theorem 2.3 shows that such a process initialized at Pdata will increase the reward expectation. The second interesting retraining on a mixture of samples from the initial model and curated samples from the current model improves the reward expectation with respect initialization.
is taking Pref Po the generative model at initialization. In that case,
After showing that such retraining loop improves the expected reward, we can conversely show that this process does not deviate too much from Pref.
Theorem 2.4. Let X >> O and Pret & P(RA) with a density with respect to Lebesgue measure. Consider the process (pt) defined in Equation 8, with po Pret- Suppose that X < k-P, then, for all 2
(RI(pellPrett) S log (1 A(K - -))
Applying Theorem 2.4 with Pref = Pdata shows that retraining on a mixture of real and curated synthetic samples does not deviate too much from the data distribution. On the other hand, when setting Prer to be any initial model distribution, we see that reusing samples from the initial model stabilizes the retraining loop around initialization.
Connection with RLHF, Theorem 2.3 and Theorem 2.4 together emphasize that retraining on a mixture of reference and filtered synthetic data bears important connections with RLHF. Indeed, the RLHF objective is composed of both a reward maximization term and a KL regularization between the current and initial model. In turn, Theorem 2.3 states that the expected reward increases and Theorem 2.4 shows that the KI. divergence with respect to initialization remains bounded. The upper bound on the KI. divergence further indicates that setting K small, i.e., using fewer samples for comparison acts as a regularizer, as previously noticed.
3 Related work
Iterative retraining on synthetic data and model collapse. The study of the retraining loop of a generative model on synthetic data has witnessed a recent surge of interest. Alemohammad et al. (2024); Shumailov al. (2023) first evidenced catastrophic degradation of the generated data in the fully synthetic loop. Bertrand et al. (2024) mitigate these conclusions in the setting where the model retrained on mixture of synthetic and real data and they show the stability of the process around the data distribution. Briesch et al. (2023) specifically focus on large langage models and Hataya al. (2023); Martínez al. (2023) study large scale datasets. A recent theoretical push by Dohmatob et al. (2024a.b) provides bounds on the performance degradation in the regression setting as well as modified scaling laws. Finally recent works, Wyllie al. (2024); Chen et (2024b) study the emergence or amplification of biases in self-consuming loops.
Aligning models with human preferences. With the urgent and critical safety concerns of public deployment, the need to align models with human preferences has gained significant importance. RLHF is popular reinforcement learning technique to align an already pretrained and finetuned model on human preferences (Christiano et al., 2017; Stiennon et al., 2020; Lee et al., 2021; Ouyang al., 2022: Shin et al., 2023). It consists of two steps: first fitting a reward r(z) on human pref- erences using dataset of pairwise human comparisons and then, maximizing the expected reward over the model distribution. Kullback-Leibler regularization to the initial model further used during the maximization step to avoid reward hacking (Skalse et al., 2022; Chen et al., 2024a) or catastrophic forgetting (Korbak et al., 2022). Variants of RL.FF have recently been proposed such Direct Preference Optimization (DPO) which maximizes the reward directly without modeling
7

---

it (Rafailov et al., 2024), Identity Preference Optimization (IPO) (Azar et al., 2024) or Kahneman- Tversky Optimization (KTO) (Ethayarajh et al., 2024).
4 Experiments
This section aims to empirically illustrate our previous theoretical results on how curation impacts the self-consuming loop. In Algorithm 1, we recall and detail the different steps performed in our experiments.
Synthetic datasets. We first focus on two synthetic datasets: a mixture of Gaussians and the two moons dataset. For both datasets, we study the settings of solely retraining on curated synthetic samples (A = oc) and mixed datasets (A = 1). In Figure 4, we iteratively retrain a denoising diffusion probabilistic model (DDPM, Ho et al. 2020) on mixture of 8 Gaussians. The reward r(x) used for the discrete choice model is the clipped negative Euclidean distance to one of the centers of the Gaussians x., i.e., r(x) := -7 max(0, || x. Tmin where we choose 7 10,77m T'in 1. Clipping the distance is used to ensure that the process does not collapse to a single point. Indeed applying Theorem 2.1, we know that the density will converge to a renormalized Gaussian distribution restricted to the ball centered at 2, of radius T'in- In Figure 5, we plot the retraining compute the reward, we use an MLP classifier with 2 hidden layers of width 512 which yields probabilities qo (3)) 91 (Ir) for each class. The reward is then defined as r(x)= == "To ; 7 > O. Both Figure 4 and Figure illustrate that retraining on solely curated samples induces collapse to regions that maximize the reward: respectively one mode of the MoG or one single moon. On the other hand, the use of real data results at the same time both in stability and higher density in high reward regions. Further experimental details are provided in Appendix A.5.
curves on the two moons dataset:
4.1 Natural images on CIFAR10
Setting. We train normalizing flow using optimal transport conditional flow matching (Lipman et al., 2022; Shaul et al., 2023; Tong et al., 2023b) with the torchefm library Tong et al. (2023a, 2024). The initial model has been pretrained on the 50000 train images of the CIFAR-10 dataset (Krizhevsky et al., 2009). At each iteration, we generate 5 10 samples using the current model from which we keep 2.5 - 10s samples filtered by discrete K-choice comparisons. The reward r(x) is computed using the class probabilities (z))....999( from pretrained VGG11 classifier (Simonyan and Zisserman, 2014) with 92.39% test accuracy. Due to the expensive compute cost of retraining generative model for multiple iterations (c.f. Appendix A.5.4), plot only one run on each figure. To ensure the reproducibility of our results, we plot the retraining curves for 3 independent runs in Figure 11 in the appendix, illustrating that they have small variance.
Using probability of one class as reward. As a first experiment, we filter samples following the probability of the classifier a predefined class. We arbitrarily chose the class 0 corresponding to planes. The reward is then defined as r(z) = y (z). y >>> O. We plot the evolution of the class proportions well as the averaged reward across 10 retraining steps in Figure 2 with 5. Figure shows collapse to the single plane class as the reward increases monotonically. illustrating Lemma 2.2.
Using the confidence of the classifier as a reward: the emergence of bias. As a second experi- ment, we use the confidence of the classifier reward, i.e., r(x) = 7 maxogico (xx)), >>>>>>>>>> 0. As written, the reward therefore uncorrelated from the class but, remains implicitly correlated to it by the fact that the classifier statistics are class dependent. In Figure 3 we plot the evolution of the class proportions as well as the average reward. As expected by our theoretical results in Section 2, the average reward increases monotonically. On the other hand, clearly see that the class propor- tions become more and more heterogeneous throughout the retraining loop. While confirming our theoretical study this plot therefore additionally shows that retraining on filtered samples increases bias, in a setting where the reward is implicitly correlated to diversity. Taking a step back, this has strong societal and ethical implications as it may imply that in a filtered internet biases may emerge or strengthen as we explain in Section 6.
Reusing real samples: stability and reward augmentation. Finally, we illustrate our results from Section 2.2.1 by mixing real and filtered synthetic samples with hyperparameter À 3 Figure shows that the process remains stable as the proportion of classes remains approximately uniform
8

---

Airplane Class
Other Classes
1.0
Avg. Reward
Proportion
0.5
0.0
0
10
10
# retraining
# retraining
Figure 2: CIFAR-10. Evolution of the proportion of the class 'Airplane" and of the 9 other classes when filtering on curated synthetic samples with reward r(z) = 7 qo())
(as suggested by Theorem 2.3). On the other hand, the average reward increases before stabilizing as predicted by Theorem 2.3.
Airplane
Bird
Dear
Frog
Ship
Automobile
Cat
Dog
Horse
Truck
Only Curated Synthetic
Real & Curated Synthetic
Avg. Confidence Proportion
0.5
0.0
0.975
0.950
0.925
10
15
20
0
10
15
20
# retraining
retraining
Figure 3: CIFAR-10. Evolution of the proportion each class and the average reward r(x) when filtering based on the confidence of classifier. On the left, retraining is done solely the curated synthetic samples which results in the emergence of proportion biases. On the right, retraining performed on mixture of real and curated synthetic samples which results in both increased stability and still reward augmentation.
Conclusion and open questions
We study the impact of data curation on the training of generative models in the self-consuming loop. We provide theoretical results demonstrating that the expected reward underlying the curation process increases and its variance collapses (Lemma 2.2) as well as convergence result (Theo- rem 2.1) for the generative model. We additionally provide stability guarantees when reusing real data at each step (Theorem 2.3 and Theorem 2.4) establishing close connections with RLHF and preference optimization. Our work sheds light and theoretically grounds novel phenomenon: in- creasing the proportion of curated synthetic data on the Web automatically optimizes preferences for future trained large models. A limitation is that we not propose an algorithm to address emerging issues like bias amplification as we feel it goes beyond the scope of our paper and is substantially complex field already intensively explored (Grover et al., 2019; Wyllie et al., 2024; Chen et al., 2024b). We believe, however, that it should be research priority and constitutes an interesting

---

avenue for future work. Another interesting direction is to study the impact of the particular reward function underlying filtering (confidence, quality, diversity...) on the emerging bias amplification.
Broader impacts
Training and aligning large generative models are prone to substantial ethical concerns regarding their alignment objective (Shen et al., 2023), representational disparities of the training datasets (Clemmensen and Kjærsgaard, 2022), or the presence of harmful images in the datasets (Birhane et al., 2021; Schramowski et al., 2023; Birhane et al., 2024). Our work mostly focuses on the impact of the curation of synthetic datasets which itself heavily depends on the agent performing the cura- tion and its underlying reward function. In particular the documentation of the Simulacra Aesthetic Captions dataset (Pressman et al., 2022) alerts that the human-based curation step is performed by a group of individuals that lacks diversity, mostly from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) individuals (Henrich et al., 2010). similar bias is likely occurring in the JourneyDB (Pan et al., 2023) dataset and, more generally, in the synthetic data appearing the web. However, our work mostly revolves around a theoretical analysis and raises awareness of the implicit alignment and potential bias amplification of self-consuming generative models. We there- fore firmly believe that the potential benefits of this awareness outweigh the potential unforeseen negative consequences of this work.
7
Acknowledgements
We sincerely thank Sophie Xhonneux for providing feedback and corrections the paper. We further thank Mats L. Richter and William Buchwalter for fruitful discussions about the experiments. We thank Mila Cluster for access to computing resources, especially GPUs. AJB is partially funded by an NSERC Post-doc fellowship and an EPSRC Turing AI World-Leading Research Fellowship.
References
Hugging face Stable Diffusion 2.1. ctable-diffusion Accessed: 2024-05-17.
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidr,
S. Altman, Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
S. Alemohammad, J. Casco-Rodriguez, L. Luzi, A. I. Humayun, H. Babaei, D. LeJeune, A. Siahkoohi, and R. G. Baraniuk. Self-consuming generative models go MAD, International Conference on Learning and Representations, 2024.
M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, 2024.
S. Azizi, S. Kornblith, C. Saharia, M. Norouzi, and J. Fleet. Synthetic data from diffusion models improves imagenet classification. TMLRR 2023.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 22X1:2220.005662 2022.
Q. Bertrand, A. J. Bose, A. Duplessis, M. Jiralerspong, and G. Gidel. On the stability of iterative retraining of generative models on their own data. International Conference on Learning and Representations, 2024.
A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.0196 2021.
A. Birhane, S. Han, Boddeti, S. Luccioni, et al. Into the LAIONS Den: Investigating hate in multimodal datasets. Advances in Neural Information Processing Systems, 2024.
10

---

Z. Borsos, R. Marinier. D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, al. Audiolm: language modeling approach to audio generation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 2023.
R. A. Bradley and M. E. Tenry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
M. Briesch, D. Sobania, and F. Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv preprint urXiv:2311.1682. 2023.
Brooks, B. Peebles, C. Holmes, W. DePue, Guo, L. Jing, D. Schnurr, Tay- lor, т. Luhman, E. Luhman, C. Ng. R. Wang, and A. Ramesh. Video gener- ation models as world simulators. 2024. URL /ttps://openai.oo htap:://opaaa..ccm/reecchh com/research/ latee--geeeraciooomddeel--------
L. Chen, C. Zhu, D. Soselia, J. Chen, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catan- zaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXi1:2402.07319, 2024a.
Chen, Hirota, M. Otani, N. Garcia, and Nakashima. Would deep generative models amplify bias in future models? arXiv preprint 2xXiv22444002444 2024b.
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 2017.
L. H. Clemmensen and R. D. Kjærsgaard. Data representativity for machine learning and Al sys- tems. arXiv preprint (xXiv:2203.04700 2022.
E. Dohmatob, Y. Feng, and J. Kempe. Model collapse demystified: The case regression. arXiv preprint 220.:::20222..71.2.. 2024a.
E. Dohmatob, Y. Feng, P. Yang. F. Charton, and J. Kempe. A tale of tails: Model collapse change of scaling laws. International Conference on Machine Learning. 2024b.
V. Dumoulin, D. D. Johnson, P. S. Castro, H. Larochelle, and Y. Dauphin. A density estimation perspective on learning from pairwise human preferences. arXiv preprint vrXiv:2311.14115. 2023.
K. Ethayarajh, Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
Y. Feng, E. Dohmatob, P. Yang. F. Charton, and J. Kempe. Beyond model collapse: Scaling up with synthesized data requires reinforcement. arXiv preprint 1100::2400.07515, 2024.
L. Gao, Schulman, and J. Hilton. Scaling laws for reward model overoptimization. International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
M. Gerstgrasser, R. Schaeffer, A. Dey, R. Rafailov, H. Sleight, J. Hughes, T. Korbak, R. Agrawal, D. Pai, A. Gromov, et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, arXiv preprint arXiv:2404. 02444.1.111, 2024.
N. Gillman, M. Freeman, Aggarwal, C. H. Hsu, Luo, Tian, and C. Sun. Self-correcting self-consuming loops for generative model training. arXiv preprint arXiv:2402.07087 2024.
D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.08215, 2023.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
A. Grover, J. Song, A. Kapoor, K. Tran, A. Agarwal, E. J. Horvitz, and S. Ermon. Bias correction of learned generative models using likelihood-free importance weighting. Advances neural information processing systems, 2019.

---

C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, Ahem, A., Wang, M., C. Gu, and W. Macherey. Reinforced self-training (rest) for language modeling. 2023.
arXiv preprint
A. Gupta and J. Zou. Feedback gan for dna optimizes protein functions. Nature Machine Intelli- gence, 1(2):105-11, 2019.
R. Hataya, H. Bao, and H. Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
R. A. Hemmat, M. Pezeshki, F. Bordes, M. Drozdzal, and A. Romero-Soriano. Feedhack-guided data synthesis for imbalanced classification. arXiv preprint 3rXiv:2310.0011 2023.
J. Henrich, S. J. Heine, and A. Norenzayan. The weirdest people in the world? Behavioral and brain sciences, 2010.
Ho, Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in neural infor- mation processing systems, 33, 2020.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. 1. Casas, F. Bressand, G. Lengyel, G. Lample, Saulnier, et al. Mistral 7B, arXiv preprint arXi::3310006225, 2023.
R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu. Understanding the effects of rlhf Ilm generalisation and diversity. arXiv preprint arXiv:2310.06452 2023.
L. Kong, Y. Du, W. Mu, K. Neklyudov, V. De Bortol, H. Wang, D. Wu, A. Ferber, Y.-A. Ma, C. P. Gomes, al. Diffusion models as constrained samplers for optimization with unknown constraints. arXiv preprint arXiv:2402.18001 2024.
T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distri- bution matching for fine-tuning language models with no catastrophic forgetting. Advances in Neural Information Processing Systems, 2022.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
K. Lee, L. Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint 1xXi:22/6600509,, 2021.
Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative model- ing. arXiv preprint 10.00:2210002747, 2022.
R. Luce, R. R. Bush, and E. E. Galanter. Handbook of mathematical psychology: I. 1963.
R. Luce. Individual choice behavior, volume Wiley New York, 1959.
G. Martínez, L. Watson, Reviriego, A. Hernández, Juarez, and Sarkar. Combining generative artificial intelligence (Al) and the internet: Heading towards evolution or degradation? arXiv preprint arXiv:2303.01255 2023.
Midjoumey. https: //www//wwwww.. http://wwiioonneee/r/..ss com/home/, 2023. Accessed: 2023-09-09.
R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, et al. Nash learning from human feedback. arXiv preprint
arXiv:2312.00880 2023.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 2022.
J. Pan, K. Sun, Y. Ge, H. Li, H. Duan, X. Wu, R. Zhang, A. Zhou, Z. Qin, Y. Wang, J. Dai, Y. Qiao, and Li. JourneDD:: benchmark for generative image understanding, 2023.
E. Perez, S. Huang, F. Song, Cai, R. Ring, Aslanides, Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.032588 2022.
12

---

R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 14222293-202, 1975.
J. D. Pressman, K. Crowson, and S. C. Contributors. Simulacra aesthetic captions. Technical Report Version 1.0, Stability AI, 2022. url
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 2024.
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. 2021.
Schramowski, M. Brack, B. Deiseroth, and K. Kersting. Safe latent diffusion: Mitigating in- appropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. LAION-5B: An open large-scale dataset for training next genera- tion image-text models. Advances in Neural Information Processing Systems, 2022.
N. Shaul, R. T. Chen, M. Nickel, M. Le, and Y. Lipman. On kinetic optimal probability paths for generative models. In International Conference on Machine Learning, 2023.
Shen, Jin, Huang, Liu, W. Dong. Guo, Wu. Liu, and D. Xiong. Large language model alignment: A survey, arXiv preprint arXiv:2309. 15025, 2023.
R. Shepard. Stimulus and response generalization: stochastic model relating generalization to distance in psychological space. psychometrika22: 32545. [5545_ddwm((198) stimulus and response generalization: Deduction of the generalization gradient from trace model. Psychological Re- view, 1957.
D. Shin, A. D. Dragan, and D. S. Brown. Benchmarks and algorithms for offline preference- -assed reward learning. arXiv preprint arXiv:2301.01992 2023.
I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papemnt, and R. Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305. 17493, 2023.
K. Simonyan and Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409.1556 2014.
Skalse, N. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 2022.
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based gener- ative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
Stability AI. http:://sabbllyyyaasstaaledddd
2023. Accessed: 2024-05-05.
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Pro- cessing Systems, 33:3008 3021, 2020.
A. Tong, N. Malkin, K. Fatras, L. Atanackovic, Y. Zhang, G. Huguet, G. Wolf, and Y. Bengio. Simulation-free schrödinger bridges via score and flow matching. arXiv preprint 036.03672, 2023a.
A. Tong, N. Malkin, G. Huguet, Y. Zhang, J. Rector-Brooks, K. Fatras, G. Wolf, and Y. Ben- gio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:2302.00488, 2(3), 2023b.
13

---

A. Tong, K. Fatras, N. Malkin, G. Huguet, Y. Zhang, J. Rector-Brooks G. Wolf, and Y. Ben- gio. Improving and generalizing flow-based generative models with minibatch optimal trans- port. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //operrviiew net/forunfld-CDSScc/77
Expert Certification.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint M3XV::3302.1377,, 2023.
C. Villani et al. Optimal transport: old and new. Springer, 2009.
R. Villegas, M. Babaeizadch, P.-J Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, Kunze, and D. Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
7. Wang. T. Pang, C. Du, M. Lin, W. Liu, and S. Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning. 2023.
Wyllic, I. Shumailov, and N. Papernot. Fairness feedback loops: Training on synthetic data amplifies bias. arXiv preprint arXiv:2403.078577 2024.
Y. Yao, Y. Pan, I. W. Tsang, and X. Yao. Differential-critte gan: Generating what you want by cue of preferences. IEEE Trunsactions on Neural Networks and Learning Systems, 35(3):3554-3768, 2022.
Zhao, R. Brekelmans, A. Makhzani, and R. Grosse. Probabilistic inference in language models via twisted sequential monte carlo. arXiv preprint arXiv:2404.17546, 2024.
14

---

A Appendix/ supplemental material
A.1 Extended related work
On retraining with data curation to align on preferences. In Gupta and Zou (2019), the authors tackle the problem of generating synthetic DÑA sequences using Generative Adversarial Networks (GAN, Goodfellow et al. (2014)). They introduce an external function analyzer to rate synthetic samples from the generator and add the highest-scored ones into the discriminator training set. In Yao et al. (2022), the authors propose a new GAN framework to incorporate users preferences in the training. They show state of the art results in generating the user-desired data distribution and theoretically prove the convergence of their method. The key difference to our work is that they aim to generate diversity of samples that are desired by users and their focus is therefore not on the collapse of their method to maximal reward set.
On the impact of alignment on diversity. In Kirk et al. (2023), the authors investigate how the different stages of alignment affect models gencralization capabilities and output diversity. They empirically show that the output diversity of the RLHF policy is decreased w.r.t. the supervised finetuned policy, which is consistent with our theoretical insights (e.g. lem. 2.2, thm 2.1). However there are major differences with our setting as their contribution is empirical and we investigate the impact of iteratively retraining a model several times synthetic samples while they study single training round using RLHE In Perez et al. (2022), the authors investigate how a language model can be used to generate prompts that lead to a harmful behavior of another language model (red teamnig).. This is useful to prevent such behavior before public deployment of a large language model. They find that LM-red teaming is a powerful tool that successfully unveils harmful behaviors of language models and help mitigate them. However, such a method can suffer from a diversity problem in the generated prompts if the red-teaming language model is itself biased. In Gao et al. (2023), the authors study how over-optimizing an imperfect proxy of a reward model affects the average reward of the ground-truth reward. They uncover for both reinforcement learning based and best-of-n methods, functional forms of the underlying reward model scores as a function of the KL between the optimized policy and the initial policy that they confirm empirically. In Bai et al. (2022), the authors propose to use iterative online learning for RLHF, where the preferences are updated on weekly basis using human feedhack. In particular each iteration step they use their best RLHF policy to produce comparisons submitted to crowd-workers. The comparisons are then mixed with existing data and the process continues. The intuition for such methods is that diversity could be improved compared to RLHF since the dataset is generated using different states of the reward models. They additionally empirically show benefits of this practice on metries such as Elo scores as evaluated by wowddworkers.
On model collapse. Briesch et al. (2023) investigates experimentally the self-consuming loop specifically in the case of LLMs and evidences model collapse in that setting. Feng et al. (2024) show that pruning unwanted samples or selecting the best ones from multiple synthetic samples can prevent model collapse in the self-consuming loop.
Iterative finetuning of language models and rejection-sampling While there is already large literature on RLHF to iteratively finetune LLMs, rejection-samplig is one way to optionally see finetuning sampling problem amenable to probabilistic inference. Indeed, recent works Zhao et al. (2024); Kong et al. (2024) frame iterative finetuning as drawing samplesusing rejec- tion sampling, Twisted Sequentional Monte Carlo etefrom the unnormalized posterior distribution p(x) x po()) (z where po(x) is an initial generative model trained on real data. From this per- spective, our framework studies the case where we curate data by using human reward and obtain ~ p(z) which are samples from the posterior, without access to the density. This allows to then finetune (0( to approximate the posterior p(x), which in our notation is a step of iteratively finetuning on curated data.
Accounting for the accumulation of data. In Gerstgrasser et al. (2024), the authors show that model collapse is evitable, in the case where a model is retrained on the accumulation of all its previous iteraions (and not only data generated from the last iteration). We believe that their setting could be adapted in order to show that accumulating data provides additional stability to the retrain- ing loop and avoids collapse of the reward variance. We leave this exciting research direction as future work.
15

---

A.2 Extension to using a mixture of rewards
Frameworks going beyond a single reward model are especially relevant in practical scenarios in LLM alignment. An interesting reference on this topic is the recent work Munos et al. (2023) which addresses such extension by learning a preference model of samples given a prompt P(x > x zly) (as a function of the two variables r, -) instead of the Plackett-Lacc reward model r(x:) (less general when preferences are non-transitive) which they refer to as Nash Learning from Human Feedback.
We now outline how to derive an extension to a mixture of reward to our setting: First, we can introduce new latent variable z that describes the randomness in the reward used, which leads to the following expression of the curated distribution after one step of curation:
K . (
(x)) = (z) H (x) with H (z) = E 2K-MPP (T((x)) + EE1 er(mm)))
In our setting, we were able to prove that the expected reward increases and the distribution con- verges to the maximum level set of unique reward (Lem 2.2, Thm 2.1). However, in the presence of multiple rewards, it is not straightforward that the rewards have the same maximal level sets. Therefore this may yield interesting dynamics and the convergence of Pt may differ. We believe such an extension of our results is outside of the scope of this work and think that it is a fascinating avenue for future work. For example, it may be interesting to study a reward component in the mixture dominates, thereby dictating the convergence, e.g. if it gets large differences between two samples. In that case, the distribution may converge to only one maximal level set introducing a new model collapse behavior as the mixture of rewards would be dictated by single reward.
A.3
Comparison of our results against other results from the litterature
A.3.1 On refraining from scratch iterative fine-tuning
1. Experiments. All retraining step in experiments on mixture of Gaussians and two moons are performed from scratch, whereas in the case of CIFAR dataset, due to the high compute cost of retraining the model from scratch (20 hours on an A100 GPU) we performed fine-tuning at each step. Fine-tuning is always performed on 10 images which corresponds to 20 epochs on the orig- inal dataset using batch size 128 (i.e. 7.8 10 gradient steps). We always use the same amount of images for fair comparison between different proportions of real data injected. In contrast, in Alemohammad et al. (2024), the collapse is shown when the model retrained from scratch each iteration. In Shumailov et al. (2023), the experiments are performed using retraining from scratch for Variational AutoEncoders and Gaussian Mixture Models and sequential fine-tuning for Large Lan- guage Models. In Bertrand et al. (2024), toy experiments on two moons and mixture of Gaussians are performed by retraining from scratch while experiments on CIFARIO, FFHQ are performed us- ing iterative fine-tuning. It is worth noting that stability using real data in the setting of iterative fine-tuning is easier to obtain than when retraining from scratch, since the model parameters are initialized around a good potential set of parameters. However, we point out that model collapse occurs also in the setting of iterative fine-tuning as shown in Figure 2 of Bertrand et al. (2024) (red curves).
Theory. Finally regarding our theoretical results, only theorem 2.2 happens the setting of iterative fine-tuning (since it uses the same setting as in Bertrand et al. (2024)). However, all our other results and in particular theorem 2.1, 2.3, 2.4 do not explicitly assume a special learning algorithm in parameter space. Instead, we consider having perfect learning model and consider the evolution of the expected reward for such a learning model. In that sense, it applies to learning from scratch with the additional assumption that the model attained perfectly fits the curated distribution.
A.3.2 On fresh real vs fixed real data
Alemohammad al. (2024) studies the self-consuming loop in three different settings where the model is retrained a) only on synthetic data b) on a mixture of synthetic data and fixed set of real data samples b) on synthetic data and a fresh set of real data samples at each step. In setting (a), the retraining loop collapses. In (b) it collapses too but with some delay related to the amount of fixed real data. In (c) the retraining loop does not degrade performances provided there enough fresh real data at each step. Bertrand et al. (2024) proved stability in the setting (b) under some theoretical
16

---

assumptions and in the iterative finetuning framework. Comparatively, our experiments on mixtures of Gaussians are performed using fresh real data at each step while the CIFAR experiments are performed in the fixed real data framework.
A.4 Proofs
A.4.1 Proof of Lemma 2.1
Lemma 2.1. Let Pt+1 be defined in Equation 4. IfP = P(R)) is the set of probability distribu- tions on Rd, and we assume that Eyg-p [e"(v) oc, then have for all IR
er(x)
Pr+1(x) K-00, p(()) Ez-pe [er(a)]
(5)
Proof. First, by minimization of the cross-entropy, we know that for any distribution %. arg max, E [og[p(s))] = q. Therefore, if p++1 the solution of Equation 4, then we have directly that P++1 has the law of é, where è is defined in Equations 1 and 2. We can now spec- ify explicitly the distribution p++1- Let pt be the current distribution time We first sample
" JK Pt. and then independently sample an index ig following the Plackett-Luce model:
Lid.
(x(x)
P(ix i|.,,, ... ,KK
Ek
er(z))
By noting that the events fix = i) are disjoint, we can write the resulting density:
1t+1(2) = Σ F=1 Mン
Pe (1111 1, 3, UK Pii i1, Uj.J +
II dyj-
By independence since the K samples are drawn i..dd and since the Plackett-Luce formula is sym- metric, all K terms in the sum are equal. This leads to rewriting:
(1))(x)) K
Pe(U1
(K-1 1K)1, K yi. 1K-1,x)dyy dyk --
UUL" WK-1
(xx)
=p.(x)K
Pr(vi) ... Pr(yk- dyk-
Jo WK-1 er(a) + K er(v.)
=pe(x) HBK((.. (I)
where
/
(xr)
H (x) Pi
K-1 pranc Prly1) -elyk--)ddy dyk-1 2i=1
(10)
(r))
UI.. MK
We now can study the limit co. Consider the random variable en(ar) as Pt. By as- sumption, [[[] < co. We can therefore apply the law of large numbers. Namely, if X1, Xx-1 are sampled i.i.d.:
1
(X1 + + X 3 [|X]
(11)
K
Furthermore, for all H, we have <
e (x)
S and
(r(x)
K K-000 0.
pr(e)
R- crine)
K
Rewriting Equation 10:
[
(()
H K (x) p.
(e() PP --dyk-1
yy. UK
erlr)
+
K k- r(u)
K
K K
17

---

we get that:
(x)
H (x)
K-000
E Bynur [er(M))
which directly implies
cn(ax)
P(41(x)
K4000
(1(
E Bywpe (er(v))
A.4.2 Additional lemma: the reward expectation is increasing
Without assuming that the reward is bounded, show using Jensen inequality that the reward expectation increases at each retraining step.
Lemma A.1. When performing K-wise filtering, the expected reward increases, ie., 11 2 0:
(etao] 2 By,
(e((e)]
Bpi
(12)
D (x())
Proof. Consider the random variable Y = K-1
when V1... .9K
K
K-1
For a, > 0, the function x +
is convex on RT. Hence by Jensen inequality, for any x b+z
a
er(2)
en(x)
(r(a)
Pc
(x) = By
e>()) K
+
Y
er(r) K +
E[y
er(r) K
K- +
Ep, [er(m)
K
Finally, write:
Epes1
E
[em(e]]
(1(0 (x)) (r)drr
/ pe(x))
2r()))
+ K-1 Eynoe [er(v]] dx
K
K
E
Exppe
(erla)]2
Ezepp
[*(())
+ K- Eyyps [er(a]]
(e((s))
E Enpp
where we have used again Jensen inequality on the convex function
* on R, where
K
[er(o)] >
c:
Eynpe
K
A.4.3 Proof of Lemma 2.2
Lemma 2.2. Let Pt+1 be the distribution induced from a discrete choice model on Pt (Equation Suppose Assumption 2.1 B holds, then the expected reward increases proportionally to its variance each retraining iteration:
K 1 Varp, [e)(x)]
Epecs (er((e) 2 Ey, [e((e]
r.
18

---

Especially the expected reward converges to the maximum reward and its variance vanishes:
Ep, (x(x)
1 et-
and Varp.
((()
1-000 0
Proof. By symmetry, we can write:
2r(21) + .... + (r((x)
KB pet1
[erte)]
K
II Pe(x)ddz
(r((s)) + ....4 (r(xx)
k=1
K
en(z,) (((x)) en((1) + e>(zx)
er(z)) +... + er(xx) + t(()) (K - 1)e Eiit, e*(x²)
et(z1) +... ...... et(zk)
MK j=l
Il pe(xx)do
Eiey (e e
2 er(2a) +... ex(ex)
))(e)))
KEp, (cr(x)
II (( k=1
+
EI....
2arp. (er(a)
(e(())
< KEE,
Ker
K(K 1) 2Vrr,, [()()))
< KlEp (e(())
2
Ke
(K Var [e)(x)) 6r
(e(())
S Klp
This brings finally,
(e(z))
(r(za)
K -
Varp, (e)
E P
E Eptt1
K
We now prove that the expected reward converges and we will first show the following lemma:
Lemma A.2. Ve 1, O,
(1+( 2r, r.
2,( rr
(13)
Proof. Consider (#1...., σK)
iid.
Pt and denote B (x,r(x) 2 r. e). Then,
K
P((((2) 2 =) = -
L
K
2
2l.... 3K
On the other hand,
(x()))
P+t 2 ro -8) Emr...
Σ
n EB
Σk=1 er(or)
Proving Lemma A.2 is then equivalent, by permutation symmetries to showing that Vk S K, if
((Ii)...., 2 r. and ((844))...,,(x7. < r. - 8, then K
Ei 5 etaze
priaz)
19

---

We can then write:
k
e (
Σ i= ((,))
k= (r((x)
Σ k=1 er(())
kym
kju1 + (K - k)pz
z
E d((a))
Eisy
cn(o))
Where =
K-k
Let E > 0. By assumption on r. (Assummption 2.1 B), we know that there exists 5 > 0) such that
Po(r(z) 2 To E) ô and hence using Lemma A.2, Vi 0,1p((((x) r. E) 2 6. Therefore,
while Ey. [e)(a)] S e 28, we know that
Varps [e 2 e'PP(((())))) r. -) 20
Therefore, while Epp. [ente]] S er 2s, we have using Lemma 2.2 that [e)()]
[erle)]
K - 1
Eepet1
Eppta
K e*
Since K-1 228 >> O, this can happen for only a finite number of steps and hence we know that K c. there exists a time T. 2 such that (remind that the expectation of the reward increasing by Lemma 2.2):
Vt Te, Epp [er(w] c 2=
Since, the expected reward is obviously recursively bounded by ex at any iteration t. we just have proved that it converges.
We now prove that the variance has finite sum. Indeed, just notice that using Lemma 2.2 that VT 2 0:
T
Varp.
S c
(Bprzs
[erwe]] Epo (er(s)))
K
1=0
K
$ K
2r.
This proves that ΣT Varp, [e)(x) is positive, it converges to O.
< co. Especially since the reward variance has finite sum and
A.4.4 Fixed points of the retraining loop with filtering
Lemma A.3. probability density p a fixed point of Equation if and only if it puts all its mass on a single level set of the reward function. In other words, there exists r. E R. such that P(r(z) r.) 1.
Proof. Given the density p. denote IP the corresponding probability function and FK (p) the curated distribution using Equations I and 2. When the reward is p-a.s. bounded, this is a direct conse- quence of Lemma 2.2. When this is not the case, we know the existence of two disjoint interval J C R such that P(r(x) T) and P(((x) J) > 0. From the proof of Lemma 2.2, have seen that, taking : pt
(er(o)] = KEp [erteo] + / 21....KKK
Eies - - ) -((z))) (a(m1) +... + er(ax) II p(zx)dx
2 K
KE
KE (())
using that J have strictly positive mass and disjoint rewards. Therefore, p cannot be fixed point.
Conversely, p puts mass on single level set of r, it is straightforward that a fixed point of the filtering operator because H, (x) almost surely constant.
D
20

---

A.4.5 Proof of Theorem 2.1
Theorem 2.1. Let for all t 0. pttz be the distribution induced from discrete choice model on Pe (Equation 4) where P ((R) is the set of probability distributions on R If po satisfies
pol=)) and the celf-consuming loop curated Assumption 2.1 C, then can define p. (x) Po(r(a)=r.)
samples Pt converges to p.i
(pel(p)lpe)) - O.
Proof. Recall p. px(x) po(e)).(a)=rr. Po(r(z)=n_)
Furthermore, notice that for any 0,
Pet1 ( X r())=.,
by recursion because H. (x) depends only on r(r). From that we deduce:
(KL (ppe) = log(P.(((2)) = r-)). We therefore only have show that Pe(r(x)= r.) - 1.
We will first show the following lemma:
Lemma A.4. AE 2 0t vt 2 O,
P++(((()== = r.) - (r(x)) = 2 Po((())) (P(r))))))))(((((()))))))))))))) 8) - rr. -6)
rr.
Proof. We will actually show:
Pi+i((()) = r.) -. r.) 2 P((r(2) =r.)* (Pr-(((()) 2 r. c) - P((r(x) 2 r. - =)) (14)
from what directly deduce Lemma A.4 by using Lemma A.2.
To prove Equation 14, just notice that for any x, 3, r(x) 2 r(y) then (x))) 2 H (v) by pe
increasing monotonicity of 2 + + on R for positive constant > O. Therefore we know the
existence of constant C' such that Vz, y, if r(x) = r. and r(y) S r., then H 2 2 H (v).
For example, take C = inf st. r(w)=r. HK (I)). Then we can write:
P141((2)==1) = =r.) -..r--PPrrllee P.(((() = r.) =
(Ptt))) - pe(x)))x
/ Po)) (x) 1)dr
/ [L()))= 1)dr
rr)
and:
(r+1(1(x) 2 =) -.( 2r, -( pe (x)))
/ 1e (x) 1)dr
< / |/a))n---P( 1)dx
Pr(r(x) 2r. - 2 - 1)
$ (C-
21

---

where in the last step we have used C- 2 because P241(1()) 2 2 -P) 2 - by Lemma A.2 and (r(x)) r. E)
Combining the last two equations we get:
(Pr-1( 2
n -)) P((r(x) 2 rx - =))
We can now prove P,(r(x)= =T.) و 1. Let 6 > 0, suppose that at time t,
(2(r(z) r.) S 1
Denote for e > 0, 0, far R r(z) 2 r. e). We know that TexoUs Ø. Therefore, 3e > 0 such that P,(Ue:) < 1. Furthermore, for any t' 2 t, we know that
(r(x())
E²) d'0000 1
(15)
by convergence of the expectation (Lemma 2.2) and Markov property. We therefore know that Et' such that (v(((() I r. =²) 2'
By using the preceding Lemma A.4, we get:
= = - -.p)) = 2 po (Pr(((x) 2 r. - Po(r(()) = r.) ((1 / 8 + ()
2 -(()-( - -E)))
2 Po((())=
and (r(z) r.) hence increases by at least Therefore, the condition (r(x)) =r. must become invalid some point. Since we have shown this for any 5 0, this shows that (.(r()) r.) + 1.
A.4.6 Proof of Theorem 2.3
Theorem 2.3. Let X > and consider the process (pt) defined in eq. 8, with po Pref- If Pref
satisfies Assumption 2.1 B, then for all 1:
(K 1) [er(a))
E
(r(x)
2 E Eest
(er(x)
(1+ A)3
Ker
Proof. We proceed by recursion.
First, we know that Vt 2 1, Varp, [er(a)]
1+X Varpaet [er(z]]. Furthermore straightforward using Lemma A.1 and recursion that Vt 2 0, Ep. [en(x]] 2 Epter [er(x)].
This brings that
1 Epeer [er(x]]+ + A I+x Eper [erl)]] A (K (arrpe (x))
Vt 2 1, Ep, [e^(x]] 2 1+X the result.
which brings
(1+2)² Ker.
We can actually show the following lower bound on the limit:
Lemma A.5. Consider the process ( = Then,
1 Pre (x) (x) HK (1)) with po = Pref-
[r(=)] 2 Epet (x(z)) +
A
(K 1a [e)(x)]
lim inf (-000
(1+
Ker.
22

---

Proof. Using the proof of Theorem 2.3 we show the following more precise lower bound at each
step: denote A = I+X and B 1
(K-
[
then for all 2 1:
Ke
Ep: [ertao] 2 Exier (er(())
A'B A'
+ AB
This directly bring that:
liminf in 10000
[erte)] 2 Beer [x(x]]
+ AB
Σ
A
i=0
Byrer [x)x))
AB
+ 1-
-epeet (x(aa)
A
(K 1a prot [c)(2)
1
+
1 À (1 (1+ X)2
Ke²
1
1+A
(K-1)Varpa [er(o]]
[er(as]
A
Bpret
+
(1+
Ker-
A.4.7 Proof of Theorem 2.4
Theorem 2.4. Let X and Pref E P(RR) with density with respect Lebesgue measure. Consider the process (pt) defined in Equation 8, with po = Pret- Suppose that X < K-1 then, for all t 21
(KI(PelPPee) S log (1 A(K - 1))
Proof. We know that AK 2 2, Va E Rd HK (x) S K.
We can then show by recursion that Vt 2 1, Va, 2z(2) tion and if true at time 4, then at time + 1:
Indeed, it is true at initializa-
Partan 1->(K-1)
Pe+ (I))
1
A 1+\1- -(K-1)
K 1 A(K 1)
(x)
1+λ
We then just replace this bound in the expression of the DKL(PrlPeeg
(pu(Pelleeet Ep. log( (x()
< log
-1) (1-XK-1))
Pret Pret(x)) (x)
A.4.8 Additional lemma: retraining on convex combination previous iterations
We study here the impact of retraining on combination of all previous iterations and show that the process remains constant. This motivates and enlightens previous works that consider only retraining on the distribution the last iteration. Let ag, Q1; Q2 a fixed non-negative sequence and consider
a retraining process using maximum likelihood: 0,+1 = arg maxę Ei a; &;Ep,, log(pe(z)) We will assume for this lemma that the solution of this optimization problem unique. Otherwise the lemma remains valid but for carefully chosen solution when there are multiple possibilities.
23

---

Lemma A.6. Suppose we start with the first T iterations predefined, i.c., by fixing Po. 1r Then starting t T. the learned distribution is constant, i.e., Vt T. Pr pr.
Discussion. As example, suppose that take Pu Pdata and P1 an initial generative model trained on Pdata- Then, Lemma A.6 states that starting = 2, the learned distribution at each step will be constant equal to p2- In other words, cannot expect the process converge maximizer of the data log-likelihood. More generally, Lemma A.6 shows that if the respective pro- portion of previous iterations remains constant throughout the retraining loop, the process remains constant and hence cannot converge towards the data distribution. These considerations have in- teresting links with previous work by Gerstgrasser et al. (2024) which experimentally showed that accumulating data with fixed relative ratios breaks the curse of recursion. However, note that the focus is different since they are in the finite sample setting while we study the infinite sample setting. Finally Lemma A.6 implies that to ensure convergence, we need to relatively decrease the propor- tion of previous iterations and comparatively increase the relative proportion of the data distribution or only use the distribution of the current iteration. This has been done in Bertrand et al. (2024) for parametrized generative models under some assumptions
global
Proof. We prove the result by recursion starting t = T. By definition:
0r arg max
a Byp log(po(x:)
1=0
Then suppose that for all j such that T S s 0j 0r. Then write:
01+1 arg max
or log(po (z))
But we know by eross-entropy minimization that
fr = arg max Eyor log(po(z)) = arg max
E pe, log log(po(z:))
Furthermore, by definition,
T-1
0r = arg max
Ep log(pe(z))
In particular it maximizes the sum of both previous terms and hence 0,+1 = OT
A.4.9 Additional lemma of convergence in parameters
Lemma A.7. AX E R+, if rate of convergence:
2L. then for dg in neightoroood of 9.. we have the following
110, - 001l= 8.11=0 10--B_l=0((AA(tt (
(16)
Proof. We follow the same steps and notations as in Bertrand et al. (2024). The main idea is to get another bound the operator of the Jacobian 0.: [[Tg(())] (their lemma E.1 (ii)). We begin with their intermediate result (lemma E.1 (ii)):
JG(0") (I XA - -A-
However we will bound this term differently. First note that || A S Le.

---

From this, we deduce by sub-multiplicatiiity of the matrix norm that:
-A- All $ L
and by triangular inequality:
L
||A = ||A A) + -1I S | II - +||+151+
Now use the triangular inequality again to write:
| S -|
But,
(I AA- - + X(A 1))
1
B-
1+\
1+
1
1 X
|A-1B-
I#X
1
1+\- LE
where we have used that Le < 1. Finally,
1
L
Le
[
(1+
1+ - X Le
a
and sufficient condition for having ||](()))| i
A 1
1 \ 1 (1+ Le
<1
or equivalently,
a
A <
2
2he which ensures that the operator norm of the Jacobian smaller than 1. L.e. [[]](")|| < 1, we can unroll the remaining steps of their proof to get Equation 16
With this new bound À a
A.4.10 Proof of Theorem 2.2
Theorem 2.2. Under Assumption 2.2, if LE and X < a. then there exists neighborhood of the optimal distribution parameters 6., such that for any initial parameters dg in that neighborhood, pe, converges to pe, exponentially fast:
M(x+EL) (a))et))
21
DKL(po) (po.oo.l)p.)) llpo,)
Maax)a=EL)
25

---

Algorithm Iterative retraining with curated synthetic data
input Dreal { A // True data, learning procedure,
param: T, X, B // Number of retraining iterations, proportion of gen. data, reward multiplicative factor
Po = AA(reeal) // Learn generative model true data
for 1..... T do
for i in 1 [ nJ do
31. .1...., FK Pt-1 // Sample K synthetic data points
Tg is selected by user with probability EK
1sks K. // Luce's model
34 3x
Diltered = { [A-el // New filtered dataset A(Dreal U Diltered) // Generative model is learned synthetic and true data return PT
Proof. We know that 0, locally maximizes 0 Egwpo, (x)( and hence locally minimizes 0 (RI(pe, [po). Hence, V DaDii(po, (pe.) 0. Furthermore we know that
DiDKL (pe, (pe) = = po. (z) V log(pa)dr
For fixed parameters 8, denote for s E [0, 1], 0, = st + +11 s)0. and f(s) = (po.l///p We have f0()) 0 and
-0 (-f po. (c)) (og(po))d)) 10-03 0.)
Using Taylor expansion with explicit remaining, we know the existence of s E [0, 1] such that f(1) f(0) + f'(0) + _²(" There remains to bound the spectral norm of (- (o, (x)V3, (dz). Since by assumption the mapping 9 + Epasts V2 (og((p(<))) is lo- cally continuous, and that the spectral norm is itself continuous, we know that we can bound on neighborhood of 0., Epp.. V (o & 2/IBpa... Va log(po_ (=))
2C co. Further- more, using that + v log(po is L-Lipschitz (Assumption 2.2) and that W(pe, Pdata) by assumption, using Kantorovitch-Rubinee
duality we know that
D/ (z) (z) log(po,))dceeoo
- Paaaaa (x) Log(s, S EL
Putting all things together. we know the existence of a constant such that for in a neighborhood of 0_ (that hence DKL(Po. llp)) S C² 12 for C' < ao on a neighborhood of 0.. Using the previous
particular choose convex), have for S 1, (f) & 2C" 0. I1 and
Lemma A.7, we deduce the convergence rate:
( EL))
(O(
( Ma+
20
(po.))
A.5
Experiments
We recall and detail the general set-up of iteratively retraining on mixture of real data and curated synthetic samples in Algorithm
A.5.1 MoG and two moons datasets DDPM
Experimental details. For both experiments, the learned vector field parametrized by an MLP of 2 hidden layers and hidden width 128. We use time discretization in 250 steps. Finally, we retrain the model for multiple iterations for MoG, 5 for two moons), first only on real data and then filtered synthetic samples from the previous iteration using pairwise comparisons. We use
26

---

5 10 initial samples from the real data distribution and 5- 10 generated samples filtered from 10* generated initial samples. When mixing, we use equal fractions of real and filtered samples. For the two moons we add a Gaussian noise with standard deviation 1.10-1.
Ground Truth
Iteration 0
Iteration 2
Iteration 4
Iteration 6
Iteration 8
Real & Synth. Only Synth.
Figure 4: Mixture of Gaussians. Iterative retraining on the two moons dataset for 8 iteeaaiios. On the top row, display the fully filtered synthetic loop, and below we mixture of real and filtered data.
Ground Truth
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration
Only Synth.
Real & Synth.
Figure 5: Two moons. Iterative retraining the two moons dataset for iterations. On the top row, we display the fully filtered synthetic loop, and below we use a mixture of real and filtered data.
A.5.2 Difference between collapse of the reward variance and overall variance
It is crucial to note the difference between collapse of the reward variance and collapse of the overall distribution variance. To highlight this difference, in appendix A.5.2 show heat-maps of respec- tively a reward with four different modes, the density of the mixture of Gaussians Po. and the limit density of theorem 2.1 as defined (x) po(=)Irco)=. In appendix A.5.2, we show that the reward variance collapses to 0 while the variance of the overall distribution density does not seem to collapse.
Fo(r(a)=rr.)
27

---

Hhut Mus Lovdl no)
Denaity P
Notur ot Cauollrne Demity
(a) Level sets of the reward function r(x) max (0, d(x,, D) Tmin where Tmin
2
(c) Limit density as defined by po(=)I-(a)=r> Fg(r(=)=rr)) where
1 and d(x, D) minyeD || ylI and
set of points. In particular, the reward is constant on the 4 balls of radius Tmm around D
(=( =
Pu is the density of the mixture
(b) Density of the mixture of of Gaussians Iearned at initializa- Gaussians
tion
Figure 6: Plots of respectively a) the level sets of the reward b) the density of the mixture Gaus- sians c) The limit density of the fully synthetic retraining loop with curation predicted by theory
0.5
Symerr bach and Sydthit
Ground Truth
Iteration o
Iteration 1
Ieeattio 2
Iteration
lteration
0.4
0.2
Reall &
0.0
Iteratioo
(a) In the fully synthetic loop with curation, the model distribution converges the predicted optimal distribution shown in figure 6c. When mixing with data, limit distribution's density on the four other centroids does not vanish, as predicted hy the KI. regularization in theorem 2.4
(b) Evolution of the reward's variance when iteratively retraining the model either fully synthetic curated data (blue) or on a mix of synthetic and real curated data (red)
Figure 7: Iteratively retraining a diffusion model on a mixture of Gaussians. For curation, we use reward r(x) max(0, d(x, Fminn where Tmin 1 and d(x, D) minyeD || yll with D set of 4 points.
A.5.3 FID, precision, recall
We measured FID, precision and recall for the three different settings on CIFAR10 presented in Section 4, i.e., a) filtering based on the probability of classifier on class of planes (Figure 8), b) filtering based on the confidence of the classifier (Figure 9) and c) filtering based on the confidence of the classifier and using mixture of real data and filtered synthetic samples each retraining step (Figure 10).
In the first two settings, we observe that the FID dramatically increases during retraining. We want to point out that it is not only due to a degradation in quality of the generated samples but also and mostly from the inequalities of the class proportions emerging during retraining. A clear indicator of this is the correlation between the FID behavior in Figure 8 and the behavior of the proportion of class shown in Figure the FID stabilizes at the end of the retraining loop when the proportion of class O reaches its maximum. A second interesting fact is that in all three settings, the precision increases, which hints that filtering does not necessarily degrades the quality of generated samples in our case. Additionally, we can clearly see the impact on stability of real data on Figure 10 where the FID witnesses much smaller variations compared Figure and Figure 9. Interestingly, we see on Figure that using the confidence of the classifier reward function implies bigger increase
28

---

of the precision than on Figure 8 or Figure 10, which correlates with the intuition that confidence is linked precision. Finally, notice that the three runs on Figure 9 have small variance, as we have already highlighted.
100
0.70
0.6
Precision
Recall
0.4
FID
60
0.68
40
0.2
0.66
20
0
10
10
0
10
Step
Step
Step
Figure 8: FID, precision and recall when retraining with filtering and r(z) -Y9u(x)), 7 5
0.6
0.800
80
Precision
0.775
0.5
Recall
FID
60
0.750
0.4
40
0.725
0.3
20
0.700
0.2
0.675
0.1
10
20
20
Step
Step
Step
Figure 9:
FID, precision and recall when retraining with filtering and r(x)
arg maxoo Pi(x), 7 15
5.25
0.695
0.575
Precision
5.00
0.690
WW
Recall
0.570
FID
4.75
0.685
0.565
4.50
0.680
0.560
4.25
10
20
10
20
10
20
Step
Step
Step
Figure 10:
FID, precision and recall when retraining with filtering and r(x)
arg maxugisy Px(r), 7 15 and reusing real data at each step
29

---

Airplane
Bird
Deer
Frog Horse
Ship Truck
Automobile
Cat
Dog
Run 0
Run 1
Run 2
Proportion
0.5
0.0
Avg. Confidence
0.975
0.950
0.925
10
20
10
20
0
10
20
retraining
# retraining
# retraining
Figure 11: CIFAR-10. Evolution of the proportion of the classes and the average reward when filtering based on the confidence of classifier for three independent runs. The curves have small variance which supports our results when only one run was reported due the high compute costs of retraining a generative models multiple times.
A.5.4
Compute Cost
Experiments on synthetic data (mixture Gaussians and two moons) ran on a single GPU in a few minutes. However, retraining with filtering CIFAR10 more costly. On A100 GPU of 40GB RAM and using 4 workers with total 32 GB RAM, retraining for 20 iterations with generation of 50000 samples took about 22 hours.
A.6 Examples of sets of four generated images Mid.Journey and Stable Diffusion 2.1
We show in Figure 12 two sets of four images generated respectively with Midjourney and Stable Diffusion. both cases, users can choose their preferred image out of the proposed and more specifically in the case of Midjourney, upscale it.
30

---

(a) Midjourney. Images from Midjournyy discord, generated with the prompt "Modemn and white bath- room, elean and shiny, high resolution, real scene".
(b) Stable Diffusion. Four images were generated us- ing Stable Diffusion 2.1 Hugging Face implementation (Hug), with the prompt "a bathroom".
Figure 12:
Two sets of four images were generated using two different generative models. For Midjourney (Figure 12a), users select which image to upscale. The upscaled images are then incorporated into the JourneyDB dataset (Pan et al., 2023). For Stable Diffusion, users can choose the preferred generated image.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show theoretical results regarding iterative retraining with filtering in Section and illustrate it on synthetic data and natural images in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

We provide required assumptions in Assumption 2.1 and Assumption 2.2 and all proofs are explicited Appendix A. Guidelines:
Justification:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details on the model architectures, datasets and experimental hyperparameters Section 4.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: did not release the code in the submission, since the experiments are mainly illustrative of our theoretical results and were not required by the reviewers.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips... httpps.cc/pubbbl ptcc/pubicc// wwwp///gg////ee/eeeeeeee/ceess Cotee//odeubbbiee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide all experimental parameters especially number retraining iter- ations and pprar-parameters for the reward and number of generated samples in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the expensive compute cost of retraining a generative model for sev- eral iterations chose not include error bars. However, reported in Figure 11 three independent runs in one setting of our experiments and noticed small variance in the differ- ent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computer ressources details Appendix A.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attpp://neurr cc. cctppblicccccccccccccccccccccclt Etthi caw.ehhhhhceeeeeeeeeeeee
Answer: [Yes]
Justification:
We read and followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Ys]]
Justification:
We discuss potential implications of our work in Section 6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release high risk models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all original owners of code and data in Section 4.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
36

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "C0JLyE2ylS", "reviewer": "Reviewer_WZgs", "summary": "Self-consuming generative models are known to have collapse or stability problems, and the curation of synthetic data is often ignored. This paper theoretically studies the impact of data curation and proves that it optimizes the expected reward.", "questions": "See weakness.\n\nOne related work: [4] propose to use a correction function on the synthetic data to mitigate the model collapse. \n\n[4] Gillman, Nate, et al. \"Self-Correcting Self-Consuming Loops for Generative Model Training.\" Forty-first International Conference on Machine Learning.", "limitations": "The limitations and societal impacts are well discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, and works on synthetic data are extremely important to the field.\n2. The connection between retraining with a mixture of curated data and original data and RLHF (Reinforcement Learning with Human Feedback) with KL regularization is novel.", "weaknesses": "1. The theoretical results are applied in a simplified setting, focusing on distribution and reward on $x$ only, instead of considering a joint distribution of $x$ given $y$ mimicking text-to-image generation. The theoretical results apply only to learning the distribution directly, without considering finite samples and optimization, and there is only one reward function. In recent works combining RLHF and diffusion models [1], multiple rewards are considered. How would the results hold with a (random) weighted sum of multiple rewards?\n2. On the connection with model collapse: The authors motivate the results from previous literature on synthetic data leading to model collapse but do not discuss how and whether curation with a reward model will avoid model collapse. The reviewer thinks that improvements can occur when inconsistencies between text and image and implausible images are discarded through curation. However, other problems persist in model collapse, such as the inherent lack of diversity in synthetic data [2], and existing results show that human interaction with GPTs still produces data lacking diversity [3]. In this case, curation does not address the loss of diversity in synthetic data. The authors should discuss this clearly, especially since the paper is motivated by model collapse and spends considerable time discussing it.\n3. Could the authors explain the contribution of Section 2.2.1? Comparing Sections 2.2.1 and 2.2.2, there seems to be no direct comparison to be made.\n\n### Reference\n\n[1] Liang, Youwei, et al. \"Rich human feedback for text-to-image generation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[2] Guo, Yanzhu, et al. \"The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text.\" *Findings of the Association for Computational Linguistics: NAACL 2024*. 2024.\n\n[3] Padmakumar, Vishakh, and He He. \"Does Writing with Language Models Reduce Content Diversity?.\" *The Twelfth International Conference on Learning Representations*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and review of our paper. We are glad that the reviewer finds our paper \"well-written\" and our connection with RLHF to be \"novel\" in the \"extremely important\" area of work on synthetic data. We address below the key points raised by the reviewer:\n\n## Extension of the theory\n\n### Theory for conditional distribution $p(x|y)$   \nAs remarked by the reviewer, our theoretical results focus only on an unconditional distribution $p(x)$. However we believe that our theory can be easily applied to a conditional distribution $p(x|y)$. Indeed for any sampling of $x_1, \\dots, x_K$ given a prompt $y$, and a subsequent curation by humans of these samples, we can apply our theoretical results conditioned on $y$ (especially because the reward $r(x,y)$ also depends on the prompt $y$). In particular, our results state that the expected reward conditioned on $y$, i.e. $E_{p_t(x|y)}[r(x,y)]$ will be maximized and converge to a variable $r^*(y)$ where $r^*(y)$ is given as in our Assumption 2.1 (but now conditionally on $y$). Note that this is valid for any $y$, hence our framework extends to conditional generation.\n\n### Finite samples and optimization: \nWe agree with the reviewer that we work under the assumption of perfect learning. We believe that accounting for finite sample optimization is a crucial extension, but that it would necessitate a more involved framework beyond the scope of our paper, especially in the case of deep generative models optimization. However, we note that a starting point to such analysis could be similar to the assumption 3 made by [5] to include finite sample optimization to their framework.\n\n### Mixture of reward  \nWe thank the reviewer for referring us to an interesting related work [1] which investigates the use of rich human feedback to improve text-to-image models. This enhanced human feedback is no longer a single scalar score but consists of multiple evaluation such as delimiting part of an image with implausibility content, or labels on prompts words that are misrepresented in the image. While we proved convergence of $p_t$ to the maximal level set in the presence of a single reward, this raises an interesting question on what dynamics would arise when the maximal level sets of each reward from the mixture are disjoints. Please refer to our global response for more details on mixtures of rewards. \n\n## Can curation prevent collapse? \nThe reviewer highlights an important point related to the difference between collapse of the distribution (i.e. the model only generates a single sample $x$ given $y$), and collapse of the expected reward to its maximum level set. \n- Retraining with curated samples can avoid the former but not later as mentioned L71 of our paper: \n> Retraining with curated samples both maximizes an underlying reward whose variance collapses and converges to maximum reward regions. \n- We also refer the reviewer to the new experiments of the global response, especially Figs.1 and 2 that illustrate how reward’s variance may vanish but not the overall distribution’s variance.\n- However, such a collapse of the expected reward will induce **a decrease of diversity** as the samples with high but non-maximal reward will not be generated. This is additionally shown in our Figs. 7 where we show on CIFAR that, even if the average reward increases, the **FID consistently increases**. \n\nFinally, we mention that we believe that **both the reward and the quality are related since it is expected that low-quality images would have low reward* following human standards and understanding to what extent is an interesting avenue for future works. We will make sure to clarify this point in the updated draft and discuss the two important works [2, 3] mentioned by the reviewer as follows:\n\nIn [2], the authors empirically investigate the linguistic diversity of generated text in the self-consuming loop. They consistently observe a decrease in diversity across the retraining iterations. However, unlike our work, they don’t study the impact of curation. In [3], the authors compare the diversity of essays written by humans, assisted either by GPT3, a feedback-tuned LLM (InstructGPT) or without LLM assistance. They found that when using InstructGPT, the overall diversity decreased. As pointed out by the reviewer, this shows that feedback-tuned models have a diversity decrease with respect to the original distribution and that human supervision is not sufficient to compensate for it. It is consistent with our theoretical insights: we show how retraining with curation incurs a collapse of the reward’s variance similar to feedback fine-tuning. \n\n## Sec. 2.2.1\nAs rightfully pointed out by the reviewer, Sec. 2.2.1 revisits the framework proposed by [5] which differs from the rest of the section. We chose to incorporate this section as a preliminary to the rest of the section for the following reasons: \n- It introduces the reader to how reusing real samples in the retraining loop can provide stability.\n- It formulates the stability results of [5] using a Kullback-Leibler divergence term, which is new and is the same formulation as in Thm 2.4.\n- It yields **a significant improvement on prior work** for the upper bound on the parameter $\\lambda$ which [5] left as future work. \n\n[5] Bertrand, Q., et. al. On the Stability of Iterative Retraining of Generative Models on their own Data. In ICLR 2024\n\nWe will make the setting distinction clearer in the updated draft to avoid confusion. \n\n## Related work\nWe thank the reviewer for this reference to [4]. We note that this work is presently cited on line 47 but are happy to provide more discussion. \n\nWe are grateful to the reviewer for their great questions and hope that our answers in conjunction with the global response clarified them. We politely encourage the reviewer to ask any further questions they may have and, if they are presently satisfied with our responses, to consider a fresher evaluation of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L33XM8CLaw", "reviewer": "Reviewer_cUrK", "summary": "This paper explores the scenario where generative models are iteratively trained on self-generated data curated by human users with some implicit reward. The key idea is that each iteration of training on the self-generated data reweights the previous distribution based on the implicit reward, which converges to reward maximization as the iteration approaches infinity. The paper also studies the scenario where the curated self-generative data is mixed with natural data and analyzes its implication on the stability of iterative training. Experiments on synthetic data and CIFAR 10 validate the insights from the theoretical analysis.", "questions": "See Weaknesses.", "limitations": "I think the authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper studies an interesting problem of generative models being iteratively trained on self-generated data curated by human users with some implicit reward. This is arguably an accurate description of what happens when new generative models are trained nowadays.\n2. The theoretical framework analyzed the convergence and stability of iterative training with and without reference data.\n3. Experiments on synthetic data and CIFAR 10 are interesting, especially the one on CIFAR with replay, which shows how bias amplification can be mitigated with natural data.", "weaknesses": "1. Human preference can be heterogeneous, so in some cases, Eq. (2) does not hold. It would be interesting to see if the theoretical analysis can be extended to a mixture of rewards.\n2. This paper didn't provide experiments on realistic datasets and large models such as LAION and Stable Diffusion (SD). Thus, it's hard to map the theoretical insights to how we should train the next generation of SD. It would be really interesting if the experiments on CIFAR were reproduced on SD to see what realistic biases are picked up, how much replay is needed to mitigate that, etc.\n3. Conceptually, retraining with data curation is related to iterative finetuining in language models (e.g., rejection-sampling-based SFT). Would like to see more discussion on that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding the problem we tackle “interesting” and to be “an accurate description of what happens when new generative models are trained nowadays”. We also appreciate that they find our experiments “interesting”. We now address below the key points raised in the review:\n\n## Extension to mixture of reward \nExtending our results to hold for a mixture of rewards is a very interesting point that cannot be straightforwardly tackled in our framework. We believe it is an exciting avenue for future work and refer the reviewer to our global response for a more detailed discussion. The crucial difference with our framework is that while we showed that the learned distribution concentrates around the maximal level set of the reward, in the presence of multiple rewards it is not clear to which level set the learned distribution will concentrate. \n\n## Larger scale experiment\nWe acknowledge the reviewer's remark that experiments on larger scales dataset would be interesting to shed light on practical implications of such a collapse in terms of the expected reward. However due to the computational cost of such an experiment, we couldn't perform during the rebuttal period. We would however like to point out that existing studies already show how the style of samples of a generated model evolves when fine-tuned on synthetic and real data with preference optimization. For example, in [1] the authors retrain a generative model by iteratively fine-tuning using Direct Preference Optimization on a mixture of synthetic and real data and where the reward model favors real data. It is demonstrated in Figure 5 [1] that the style of large scale vision models can be shifted by retraining on synthetic data using a preference model. In particular, the model’s interpretation of “a very cute boy” drastically changes, illustrating how the reward influences the realistic biases that are picked up.\n\n## Relationship with rejection sampling fine-tuning\nWe thank the reviewer for encouraging us to strengthen the discussion between iterative finetuning of language models and retraining with data curation. While there is already a large literature on RLHF to iteratively finetune LLMs the reviewer highlights rejection-sampling as one way to optionally see finetuning as a sampling problem amenable to probabilistic inference. Indeed, recent works [2,3] frame iterative finetuning as drawing samples—using rejection sampling, Twisted Sequentional Monte Carlo etc…—from the unnormalized posterior distribution $p(x) \\propto e^{r(x)} p_0(x)$,where $p_0(x)$ is an initial generative model trained on real data. From this perspective, our framework studies the case where we curate data by using human reward and obtain $x \\sim p(x)$ which are samples from the posterior, without access to the density. This allows us to then finetune $p_0(x)$ to approximate the posterior $p(x)$, which in our notation is a step of iteratively finetuning on curated data. We will include a small discussion on this connection in our updated paper.\n\nWe thank the reviewer again for their valuable feedback and great questions. We hope our answer here and in the global response allows the reviewer to consider potentially upgrading their score if they see fit.  We are also more than happy to answer any further questions that arise.\n\n[1] H. Yuan, Z. Chen, K. Ji, and Q. Gu. Self-play fine-tuning of diffusion models for text-to-image generation, 2024.\n\n[2] Zhao, Stephen, et al. \"Probabilistic inference in language models via twisted sequential monte carlo.\", 2024.\n\n[3] Kong, Lingkai, et al. \"Diffusion models as constrained samplers for optimization with unknown constraints.\", 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "es8VgIlKQR", "reviewer": "Reviewer_ej3h", "summary": "This paper extends Bertrand et al. 2024’s analysis of model collapse to study settings where data is filtered (based on a particular preference model) before being used for training the next iteration of generative models.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper extends prior work studying self-consuming generative models to integrate preference learning, which is highly sensible\n- The paper is well written", "weaknesses": "- I do not think that Equation (3) faithfully models reality. Specifically, it assumes that the $t+1$-th model is fit to a mixture of (1) real data and (2) preference-filtered data from the $t$-th model. But realistically, synthetic data should amass over time, as I believe is the case with the datasets that the authors mention in their introduction, e.g., LAION-5B. This is a point made by \"Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data.\" and I agree with them.\n\n- I might be missing something, but intuitively, Theorem 2.1 seems wrong, specifically the claim “eventually converges to the highest level set of the reward reached **at initialization**”. For the simplest possible counter example, suppose p_0 is a discrete Uniform distribution on integers 0 to 10 and the reward function is $r(x) = x$. Defining $r_* := 10$, we see that Assumption 2.1 is satisfied, including the variant where $\\epsilon \\geq 0$. It is hard to imagine that the iterative loop would concentrate on $10$ rather than something greater than $10$. The intuition is that sampling and filtering from the first model iteration might shift the distribution towards regions where it is possible to sample higher rewards.\n\n- I think the experiments corresponding to Theorem 2.1(Figures 4 and 5) could be improved to better connect with the maths. Specifically: (1) Provide a heatmap of the level set of $r_*$ to show what $p_*$ is. (2) As I understand, the theorem doesn’t say that the distribution’s variance collapses, but rather, that the _reward variance_ collapses. In the figures, the reward functions are unimodal, and so we see the distributions converge towards unimodal behavior. You should modify the reward functions to be multimodal to demonstrate this distinction between the distribution's variance collapsing versus the reward's variance collapsing. One way to do this might be to define the reward function as 4 of the 8 MoGs (i.e. the reward is the max of the set of negative distances to the 4 chosen centroids). Then, we should see the model concentrate on those 4 chosen centroids.\n\n- I feel like I don’t understand Theorems 2.3 or 2.4. In Theorem 2.3, the lower bound on the right hand side does not appear to depend on the model fitting iteration $t$, as best as I could tell, nor does the upper bound in Theorem 2.4. Here, I’m expecting the answer to depend heavily on the model fitting iteration. I read the adjacent discussion but didn't receive any clarity. I'm consequently not sure how to evaluate the significance of Section 2.2. Perhaps the authors could clarify?\n\n- I think there are many additional papers you might want to cite. Some may be concurrent with yours (I intentionally did not search for a preprint in order to preserve double blind reviewing), and if so, that’s fine. Here are some suggestions on several different topics:\n\n**On Model Collapse:**\n\nBeyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement. https://arxiv.org/abs/2406.07515\n\nIs Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data.\nhttps://arxiv.org/abs/2404.01413\n\n**On mode collapse in RLHF:**\n\nUnderstanding the Effects of RLHF on LLM Generalisation and Diversity. https://openreview.net/forum?id=PXD3FAVHJT\n\nA Distributional Approach to Controlled Text Generation.\nhttps://openreview.net/forum?id=jWkw45-9AbL\n\nRed Teaming Language Models with Language Models\nhttps://arxiv.org/abs/2202.03286\n\nImproving alignment of dialogue agents via targeted human judgements\nhttps://arxiv.org/abs/2209.14375\n\nAligning Language Models with Preferences through f-divergence Minimization\nhttps://arxiv.org/abs/2302.08215\n\n**On filtering data using reward models - often known by multiple names in the RLHF literature including “Best of N” or “rejection sampling” or “reranking” in the RLHF literature:**\n\nScaling Laws for Reward Model Overoptimization\nhttps://arxiv.org/abs/2210.10760\n\nTraining a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\nhttps://arxiv.org/abs/2204.05862\n\nThere are many more", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their nuanced review and constructive feedback. We appreciate that the reviewer has found our paper \"well-written\" and the integration of preference learning to the model collapse literature to be \"highly sensible\". We took note of the reviewer's concerns and provide clarifications:\n\n## Accounting for the accumulation of data  \nWe thank the reviewer for referring us to the paper [1] and pointing out this interesting extension. We believe, together with the reviewer, that the setting of [1] could be adapted to show that accumulating data provides additional stability to the retraining loop and avoids collapse. However, we believe such a study is out of the scope of our work whose aim was to introduce and theoretically develop a new research question of model collapse from the view of preferences. We will, therefore, update our draft to provide additional clarification of this aspect and mention it as an exciting future direction. \n\n## Clarification on Thm 2.1\nWe understand the reviewer's natural concern. The crucial point is that the curated distribution at time $t+1$, $p_{t+1}$ is issued from learning a modified distribution from $p_t$ constructed by **sampling from** $p_t$ and curating samples using preferences. This implies that if a set has probability $0$ for $p_t$, it will never be sampled and therefore never be preferred over other samples. This means that this set will also have probability $0$ for $p_{t+1}$.\n\nFinally, we note that the reviewer’s intuition is valid when the retraining step is not perfect (for example due to bounded expressivity of the class $\\mathcal{P}$ involved in equation 4), or when noise is injected in the process. Then the support of $p_{t+1}$ is not necessarily included in the support of $p_t$ anymore. In that case, the self-consuming loop iteratively explores regions that had probability $0$ at initialization and converges to the maximal reward possible, validating the reviewer’s intuition.\n\n## Improvement of the experiments on MoGs\nWe thank the reviewer for their suggestion which will help clarify the theoretical results. We provided in the additional pdf for figures a heat map of the level set of the reward when using $4$ centroïds (Fig 1a). We additionally provided a heat map of the raw mixture of Gaussians (MoG) distribution (Fig 1b) and the corresponding limit distribution defined L154, as the renormalized MoG restricted to the set of maximal reward at initialization (Fig 1c).  We also re-ran our experiments for this reward model and observed that the learned distribution converges as expected to $4$ Gaussian restricted to $4$ balls around the designed centroïds (Fig 2a). We additionally plotted the reward variance and showed that it vanishes in the purely synthetic setting. This demonstrates that the reward variance can vanish independently of the overall distribution variance. We will clarify this in the updated manuscript as it is central to our analysis.\n\n## Clarification of Thm 2.3, 2.4 \nThe goal of these two theorems is to respectively provide a lower bound on the expected reward and an upper bound on the KL divergence of a self-consuming generative model loop **with respect to a reference distribution $p_{ref}$**. These theorems formally connect our results to the KL regularization in RHLF ensuring that the KL divergence between the aligned model and the reference model is not too large. This is why only $p_{ref}$ and not $p_t$ appear on the right-hand side. The reviewer’s observation is, however, insightful as our proof proceeds by an induction argument in which each induction step uses Lemma A.1 (see line 563) and hence a right-hand side dependent on $t$. We finally achieve to make this term not dependent on $t$ by induction.\n\n## Related work \nWe thank the reviewer for pointing us to these important references that we will cite and discuss in our revision. We will especially provide more discussion on the relationship between RLHF and rejection sampling fine-tuning. We now discuss the most relevant works mentioned by the reviewer (because of the space constraint we do not have space to discuss them all extensively here):\nNote that [2,3] are concurrent work according to NeurIPS guidelines:\n>papers that appeared online within two months of a submission will generally be considered \"contemporaneous\"\nThe key difference between [2] and our work is that they show the benefit of using feedback on the quality of a sample to prevent model collapse.\n\nIn [3], the authors investigate how the different stages of alignment affect a model’s generalization capabilities and output diversity. They empirically show that the output diversity of the RLHF policy is decreased w.r.t. the supervised finetuned policy, which is consistent with our theoretical insights (e.g. lem. 2.2, thm 2.1). However, there are major differences with our setting as their contribution is empirical, and we investigate the impact of iteratively retraining a model several times on synthetic samples while they study a single training round using RLHF. \n[4] experimentally investigates the self-consuming loop, specifically in the case of LLMs, and evidences model collapse in that setting. \n\nWe thank the reviewer again for their review and detailed comments that helped strengthen the paper. We hope our answer here and in the global response allows the reviewer to consider potentially upgrading their score if they see fit. We are also more than happy to answer any further questions.\n\n[1] Gerstgrasser, Matthias, et al \"Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data\" 2024\n\n[2] Feng, Yunzhen, et al \"Beyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement\" 2024\n\n[3] Kirk, Robert, et al \"Understanding the effects of RLHF on LLM generalisation and diversity\" 2023\n\n[4] Briesch, Martin, et al \"Large language models suffer from their own output: An analysis of the self-consuming training loop\" 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrtawk2M91", "reviewer": "Reviewer_cgiY", "summary": "This paper studies the impact of data curation on iterated retraining of generative models. Theoretical results are derived for the convergence state of the retraining loop when using a fraction of curated synthetic data or a mixture of real data and curated synthetic data at each step. Empirical experiments on both synthetic datasets and CIFAR-10 demonstrate that the proposed approach can bias the generative model to generate samples with higher reward.", "questions": "1. Eq.3: The term $p_{ref}$ is not explained. What distinguishes $p_{ref}$ from $p_{data}$?  \n2. In Theorem 2.2, deriving $\\lambda < 0.5$ based on the assumption $L \\varepsilon < \\alpha$ and $\\lambda < \\frac{\\infty}{2 D \\varepsilon}$ appears to conflict with the claim in line 181.  \n3. The explanation of $r_{\\ast}$ in lines 134-137 is somewhat complex. Adding some equations would clarify its meaning.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem of iterated retraining of generative models using curated data is important and interesting. The authors have indeed made some achievements in this direction. The theoretical results presented in this paper are interesting and reasonable, especially in their connection to preference optimization. The toy experiments are consistent with the theoretical claims.", "weaknesses": "The writing of this paper needs further improvement. Some notations are unsuitable, and certain mathematical notations are introduced without explanation. Some literature references are missing.  \n1. In Eq.2, $\\mathcal{BT}\\left(x_1, \\ldots, x_K\\right)$, where \"BT\" refers to the Bradley-Terry model, is typically used to model pairwise preferences, whereas the Plackett-Luce (PL) model is proposed for preferences involving more than two items.  \nEq.5: If $p_{t+1}(x)$ is not a normalized density function, the same applies to $p_{t+1}(x)$ in Eq.6.  \n2. There are already studies on deep generative models (GANs) that investigate the convergence state of iterative retraining on curated data from the perspective of preference [1, 2]. It should include a discussion of these works.\n3. The theoretical proofs especially the part in Section 2.2 are heavily inherited from the previous work [3], which somehow makes the theoretical contributions marginal.\n\n[1] Gupta, A., Zou, J. Feedback GAN for DNA optimizes protein functions. Nat Mach Intell. 2019.  \n[2] Yao, Y., Pan, Y., et. al. Differential-Critic GAN: Generating What You Want by a Cue of Preferences. IEEE Transactions on Neural Networks and Learning Systems, 2022.  \n[3] Bertrand, Q., Bose, et. al. On the Stability of Iterative Retraining of Generative Models on their own Data. In The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and constructive comments. We appreciate that the reviewer finds the problem we tackle \"important and interesting\" and that our theory in connection to preference optimization “interesting and reasonable”. We now address the key points raised in the review: \n\n## Plackett-Luce model\nWe thank the reviewer for referring us to the Plackett Luce model. We would like to mention that we were aware of the Luce choice rule model that we cited on lines 87-90. We will change our notation from $BT$ to $PL$ referring to the Plackett-Luce model since it provides a more unified framework consistent with the literature. \n\n## Normalization of $p_t$\nWe agree with the reviewer on the fact that **if** $p_t$ is not normalized, then $p_{t+1}$ is not normalized either. However we do not foresee a problem since in practice we apply our results to an **initial normalized probability distribution**. We hope this answers the reviewer’s question and are happy to clarify further. \n\n## Related Work\nWe are grateful to the reviewer for referring us to the two important references [1, 2] that we will add to our related work section. In [1], the authors tackle the problem of generating synthetic DNA sequences using GANs. They introduce an external function analyzer to rate synthetic samples from the generator and add the highest-scored ones into the discriminator training set. *This work [1] is mostly experimental and tied to the GAN architecture while we adopt a more theoretical framework to understand the self-consuming loop without specifying any architecture*. \n\nFurthermore our study takes the point of view of the recently developed model collapse literature and relies only on $K$-wise preferences as in [2]. In [2], the authors propose a new GAN framework to incorporate user’s preferences in the training. They show state of the art results in generating the user-desired data distribution and theoretically prove the convergence of their method. *The key difference to our work is that they aim to generate a diversity of samples that are desired by users and their focus is therefore not on the collapse of their method to a maximal reward set.* \n\n## Closeness with [3]\nWe acknowledge the reviewer’s comment that our proof of theorem 2.2 is adapted from the proof of [3], since our goal was to improve their paper’s main result in the same setting. Note that despite the similarity in the proof technique **the theoretical improvement we provide is significant**. \n\nMoreover, we believe that it does **not** consist in our main theoretical contributions, and was serving as an introduction to the rest of the section on how real data provides stability to the retraining loop. Instead, our major theoretical results, i.e. **theorem 2.1, 2.3 and 2.4 are proved in a different setting** and with proof techniques that completely differ from [3].\n\n## Clarification of the $p_{ref}$ notation \nWe acknowledge the reviewer's concern about the notation $p_{ref}$ and we will clarify this notation in the updated draft. Namely, we denote $p_{ref}$ any probability distribution with density with respect to Lebesgue measure. It may indeed be of interest to apply thm 2.3 to other cases than retraining on a mixture of synthetic data at iteration $t$ and the data distribution. For example, it may be of interest to retrain on a mixture of synthetic data at iteration $t$ and of the synthetic data at initialization. In such a case, $p_{ref}$ would be identified to $p_0$. It bridges the gap with RLHF, as the KL regularization in the RLHF objective is done with respect to the supervised fine-tuned policy (see equation 3 in the DPO paper [5]). This makes our result more general as we show that stability occurs around any reference distribution that is injected in the retraining loop (either real data samples or samples from the initial model). It also provides an avenue to study the retraining process over a mixture of all previous iterations of the generative models.\n\n## Clarification on the assumption on $\\lambda$\nWe respectfully disagree with the reviewer that our assumptions $$L\\epsilon < \\alpha \\quad \\text{and} \\quad \\lambda < \\frac{\\alpha}{2L\\epsilon}$$ are in conflict with the claim that previous work was restricted to $\\lambda < \\frac{1}{2}$. Indeed, the necessary conditions for theorem 1 of [3] are $$\\lambda \\leq \\frac{1+ \\frac{L\\epsilon}{\\alpha}}{2}\\quad \\text{and}\\quad L\\epsilon< \\alpha$$  which necessarily requires $\\lambda < \\frac{1}{2}$. These conditions are more restrictive than ours. In particular, we see that for fixed $\\alpha$, if $L\\epsilon\\rightarrow 0$ then both our conditions are satisfied (in particular $\\lambda$ bigger than $\\frac{1}{2}$ will be allowed).\n\n## Clarification of $r_*$ notation \nWe agree upon the reviewer's remark that our definition of $r_*$ could be clarified and we will improve this paragraph in the updated paper. In a nutshell, $r_*$ should be thought as the smallest number that upper-bounds the random variable $p_0$ with probability $1$. For example, a Uniform distribution on the interval $[0, 10]$, $r_*=10$ whereas for unbounded distributions such as $\\mathcal{N}(0, 1)$, $r_*$ does not exist. As suggested by the reviewer, an equation that could clarify this is $r_* = inf \\\\{r \\in \\mathbb{R}, \\mathbb{P}(r(x)\\leq r_*) = 1\\\\}$.\n\n\nWe thank the reviewer for their valuable feedback and great questions. We believe we have answered to the best of our ability all the great questions raised by the reviewer and we kindly ask the reviewer to potentially upgrade their score if they are satisfied with our responses. We are also more than happy to answer any further questions that arise.\n\n[4] Gerstgrasser, Matthias, et al. \"Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data.\" arXiv 2024\n[5] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oI3NZyWAkb", "reviewer": "Reviewer_orKH", "summary": "The authors investigate the properties of self-consuming loops that arise in the training of generative models. In particular, they investigate the impact that data curation has on the iterative retraining performance of these models. The paper contains theoretical and empirical analysis of how model performance is affected by various data curation assumptions (only synthetic examples, or real data being injected at each step, or human-preference curated synthetic samples, etc).", "questions": "I'm not sure I understand how the experiment supports the theory here, please help clarify this for me--at line 250, the authors say \"applying theorem 2.1, the density will converge to a renormalized Gaussian distribution restricted to the ball centered at x_* of radius r_min\". But this isn't what I see in Figure 4, it looks like there are a bunch of different Gaussian balls with different densities, corresponding to different intensities. Should these gaussian balls all be the same density, or did I misunderstand something?\n\nSome misc points that didn't affect my score/judgment of the paper, but the authors should consider fixing in the next draft:\n- improper formatting in equation 8 (spacing/parentheses)\n- awkward/grammatically incorrect wording in the sentence immediately after equation (7)", "limitations": "The two main things preventing me from giving a higher score are the following two limitations. I'm looking forward to hearing the author's rebuttal to these points (and hopefully seeing an updated draft, if possible).\n\n1. Presentation of statements of theorems (what are the specific hypotheses? eg take a look at the latest version of the bertrand et al paper on the arXiv for a good way to do this, since that paper is structured in a similar way. They number their assumptions and make them more clear. See \"Weaknesses\" section for more details.) This is the primary limitation, from my perspective, as I think the usefulness of the paper is very limited by unclear theorem statements.\n\n2. Proper contextualization of these results relative to the literature--namely, there's a difference between the work in Bertrand et al and Alemohammad et al, but the authors seems to be comparing them in an \"apples to apples\" way. Namely, the former work considers the case of iterative fine-tuning, whereas the latter considers the case of retraining from scratch. In the former case, it is strictly easier to avoid model collapse, since the model parameter update is \"local\", and in the latter case, the updates are \"global.\" Specifically, on line 218 it says: \"Alemohammad et al. (2024); Shumailov et al. (2023) first evidenced catastrophic degradation of the generated data in the fully synthetic loop. Bertrand et al. (2024) mitigate these conclusions in the setting where the model is retrained on a mixture of synthetic and real data and they show the stability of the process around the data distribution.\" I think this is a false statement, because Alemohammad et al considered re-training from scratch at each iteration, which wasn't considered in the Bertrand et al paper--but please correct me if I'm misunderstanding. And in that same vein, I think it is important to properly contextualize the present paper in the presence of that dichotomy--does this paper consider iterative fine-tuning, or iterative re-training from scratch? That is important information for the readers/the literature.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Originality: I believe this work is highly original, largely because it considers a new problem formulation--\"what happens to self-consuming generative models when the synthetic data that they re-train one has been curated via human preferences?\". To my knowledge, nobody has considered this problem, and it is an excellent and timely problem to consider.\n\nClarity: The authors motivated the problem very clearly.\n\nSignificance: many previous works have investigated self-consuming loops, but the area is still burgeoning, and right now, the area seems like \"the wild west\"--there are lots of papers out there with different assumptions, different results, no agreed-upon benchmarks tasks, etc. This paper is significant because it considers a more realistic setup than previous papers; it considers the setting where the web-scale data contamination happens because of human-curated data. This is an important case to consider, and represents a large step towards modeling the data contamination issue more rigorously. This more realistic setup comes with more mathematical overhead, which is challenging to deal with.", "weaknesses": "Clarity: in my opinion, the presentation of statements of the theorems in the paper needs improvement in order to be useful to the community. Consider for example Theorem 2.1. The assumptions for this theorem are distributed in the section preceding it, which makes it much harder to understand and contextualize that theorem. I would strongly suggest summarizing the assumptions and key notations in the statement of that theorem (and likewise for all the other results.) An excellent model for this would be the paper that the authors cited the most, Bertrand et al's \"On the Stability of ...\" . That paper's Theorem 1 (from the latest arXiv version) begins--\"Given theta^\\star as defined in Equation (7) that follows assumptions 1 and 2....\". Given that a large part of this paper's contribution is its theorems, I would be likely to raise my score, and champion this paper, if I could first see an updated draft which makes explicit every assumption in the statement of each theorem.\n\nSee also the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and are pleased that they find the question raised \"highly original\" and \"very clearly\" motivated. We further appreciate that they find it \"significant\" and \"more realistic\" than previous works. We now address the key clarification points raised by the reviewer.\n\n## Presentation of statements of the theorems \nWe acknowledge the reviewer's concern that the dissemination of all of our assumptions may not have been sufficiently clear. Below, we present updated assumptions and theorems that we hope address the clarity issue.\n\nWe propose to restructure assumption 2.1 to sub-assumptions of increasing strength:\n\n**Assump. 2.1-A:** The distribution $p \\in \\mathcal{P}(\\mathbb{R}^d)$ has a density w.r.t. Lebesgue measure and $E_p[e^{r(x)}] < \\infty$.\n\n**Assump. 2.1-B:** The distribution $p\\in \\mathcal{P}(\\mathbb{R}^d)$ has a density w.r.t. Lebesgue measure and there exists $r_* \\in \\mathbb{R}$ such that: (a) $p$-almost surely, $r(x)\\leq r_*$ and (b) $p$ puts positive mass in a neighborhood of $r_*$ i.e., $\\forall \\epsilon > 0, P_0(r(x)\\geq r_* - \\epsilon) > 0$.\n\n**Assump. 2.1-C:** Assum. 2.1.B and $P(r(x) = r^*) > 0$.\n\nUsing these assumptions, our results of section $2$ will have the following overheads:\n\n> **Thm. 2.1:** Let for all $t\\geq 0$, $p_{t+1}$ be the distribution induced from a discrete choice model on $p_t$ (4) where $\\mathcal{P}= \\mathcal{P}(\\mathbb{R}^d)$ is the set of probability distributions on $\\mathbb{R}^d$. If $p_0$ satisfies Assump 2.1-C, then we can define $p_*(x):=\\frac{p_0(x)1_{r(x)=r_*}}{P_0(r(x)=r_*)}$ and the self-consuming loop on curated samples $p_t$ converges to $p_*$: $KL(p_*||p_t) \\xrightarrow{t\\rightarrow \\infty} 0$\n\nThe other results have been updated similarly. If the reviewer would like to see the other exact statements, we can share them in another response.\n\n## Contextualization of our results in the literature on model collapse \nWe agree with the reviewer that better contextualization of our results on whether the retraining is performed from scratch or via fine-tuning can improve the clarity. **We will update the paper to clarify this and present below how we would proceed.**\n\n## On retraining from scratch vs iterative fine-tuning: \n\n**1.) Experiments:** All retraining step in experiments on mixture of Gaussians (MoG) and two moons are performed from scratch, whereas in the case of CIFAR dataset, due to the high compute cost of retraining the model from scratch (20 hours on an A100 GPU) we performed fine-tuning at each step. Fine-tuning is always performed on $10^6$ images. We use the same amount of images for fair comparison between different proportions of real data injected. In contrast, in [1], the collapse is shown when the model is retrained from scratch at each iteration. In [3], the experiments are performed using retraining from scratch for VAEs and GMMs and sequential fine-tuning for LLMs. In [2], toy experiments on two moons and MoG are performed by retraining from scratch, while experiments on CIFAR10 and FFHQ are performed using iterative fine-tuning. We agree with the reviewer's remark that stability using real data in the setting of iterative fine-tuning is easier to obtain than when retraining from scratch since the model parameters are initialized around a good potential set of parameters. However, we would also like to point out that model collapse occurs also in the setting of iterative fine-tuning as shown in Figure 2 of [2] (red curves). \n\n**2.)Theory:** Finally regarding our theoretical results, only thm 2.2 happens in the setting of iterative fine-tuning (since it uses the same setting as in [2]). However, all our other results and in particular thm 2.1, 2.3, 2.4 do not explicitly assume a special learning algorithm in parameter space. Instead, we consider having a perfect learning model and consider the evolution of the expected reward for such a learning model. In that sense, it applies to learning from scratch with the additional assumption that the model attained perfectly fits the curated distribution. We will make this point clear in the updated draft.\n\n## On fresh real vs fixed real data:\n[1] studies the self-consuming loop in three different settings where the model is retrained a) only on synthetic data b) on a mixture of synthetic data and a fixed set of real data samples b) on synthetic data and a fresh set of real data samples at each step. In setting (a), the retraining loop collapses. In (b), it collapses, too, but with some delay related to the amount of fixed real data. In (c) the retraining loop does not degrade performances provided there is enough fresh real data at each step. [2] proved stability in setting (b) under some theoretical assumptions and in the iterative finetuning framework. Comparatively, our experiments on MoG are performed using fresh real data at each step while the CIFAR experiments are performed in the fixed real data framework.\n\n## Experiment clarification:\nWe thank the reviewer for highlighting a relevant point on the presence of multiple spots with different densities of fig. 4 at iteration 4. These are remaining errors due to the non-fully convergence. To improve clarity, we ran the experiment for more iterations (global response fig 3) in which case it is clearer that the final distribution is a unique renormalized Gaussian distribution restricted to the ball centered at $x_*$ of radius $r_{min}$.\n\nIf the reviewer finds our updates agreeable, we would appreciate it if the reviewer considers championing/upgrading their score for this paper, as mentioned in the original review. We are also happy to answer any further questions that the reviewer may have.\n\n[1] Alemohammad et al Self-consuming generative models go mad. ICLR 2024\n\n[2] Bertrand et. al On the Stability of Iterative Retraining of Generative Models on their own Data. ICLR 2024\n\n[3] Shumailov et al. The curse of recursion: Training on generated data makes models forget. Nature 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their detailed feedback which has allowed us to strengthen the updated manuscript. In particular, we are heartened that all reviewers (WZgs, cUrK, ej3h, cgiY, orKH) found our research question to tackle an interesting, highly-sensible, and timely problem. We are also glad that reviewers (WZgs, ej3h) appreciated the writing of the paper. We below address some key points that were raised in multiple reviews:

**Extension of our framework to using a mixture of rewards (reviewers WZgs, cUrK):**

We thank the reviewers for raising a very interesting point regarding the use of mixed rewards. Frameworks going beyond a single reward model are especially relevant in practical scenarios in LLM alignment. An interesting reference on this topic is the recent work [1] which addresses such extension by learning a preference model of samples given a prompt $P(x ≻ x’|y)$ (as a function of the two variables $x, x’$)--- instead of the Bradley-Terry reward model $r(x)$ (less general when preferences are non-transitive) which they refer to as Nash Learning from Human Feedback. 

We now outline how to derive an extension to a mixture of reward to our setting:
First, we can introduce a new latent variable $u$ that describes the randomness in the reward used, which leads to the following expression of the curated distribution after one step of curation:

$$
p_{t+1}(x) = p_t(x) \cdot H^K_{p_t}(x) \quad  \text{with} \quad  H^K_{p_t}(x):= E_{x_1,\ldots,x_{K-1} \sim p_t, u} \left[\frac{K \cdot e^{r(x; u)}}{e^{r(x; u)} + \sum_{i=1}^{K-1}e^{r(x_i; u)}} \right]
$$

In our setting, we were able to prove that the expected reward increases and the distribution converges to the maximum level set of a unique reward (Lem 2.2, Thm 2.1). However, in the presence of multiple rewards, it is not straightforward that the rewards have the same maximal level sets. Therefore this may yield interesting dynamics and the convergence of $p_t$ may differ. We believe such an extension of our results is outside of the scope of this work and think that it is a fascinating avenue for future work. For example, it may be interesting to study if a reward component in the mixture dominates, thereby dictating the convergence, e.g. if it gets large differences between two samples. In that case, the distribution may converge to only one maximal level set introducing a new model collapse behavior as the mixture of rewards would be dictated by a single reward. We added this discussion on the multiple rewards setting and [1] our revised manuscript.

**Extended related work and contextualization of our results in the literature** We thank the reviewers for inviting us to develop our discussion of related works. We will include in the updated paper an in-depth discussion on rejection sampling finetuning (cUrK, ej3H, cGiY), RLHF (ej3H) and comparison of our setting with respect to the previous model collapse literature (orKH).

**Clarification and additional experiments** We thank reviewers orKH and ej3h for suggesting improvements to our experiments which we believe will help clarify them and illustrate the theory better. Especially, our new experiment in Figures 1 and 2 of the figures pdf shows that the reward variance may vanish independently of the overall distribution variance. This underlines the crucial difference between model collapse from the view point of previous works and collapse of the reward variance, that we introduce and develop in our work. 

## References

[1] Munos, Rémi, et al. "Nash learning from human feedback." 2023.

Meta-review: This paper performs a theoretical analysis of the setting in which a generative model is trained on a curated dataset of its own samples.  It shows that the implicit reward that guides such curation can successfully be learned by the generative model, thus motivating the soundness of the practice.  The reviewers were unanimously in favor of acceptance, with the paper's strengths being that it performs a novel, rigorous theoretical analysis of a topic of great modern importance.  Moreover, the results are experimentally supported in interesting experiments (Reviewer cUrK).  The reviewers primary concerns were if the analysis extends to a mixture of reward functions, relationship to prior work, and on some of the underlying conditions / assumptions of the results.  I believe the authors have either successfully addressed  these concerns (e.g. the underlying conditions) or rebutted their significance (e.g. mix of rewards is a nice extension but not essential, some of the related works are concurrent or specialized to specific generative model classes).
Decision: Accept (spotlight)