Self-Consuming Generative Models with Adversarially Curated Data
Authors: Xiukun Wei
Xueru Zhang
Venue: ICML 2025 poster (2025)
Keywords: generative model
self-consuming loop
human feedback
adversarial curation
attack algorithms
Primary area: social_aspects->alignment

Abstract: Recent advances in generative models have made it increasingly difficult to distinguish real data from model-generated synthetic data. Using synthetic data for successive training of future model generations creates “self-consuming loops,” which may lead to model collapse or training instability. Furthermore, synthetic data is often subject to human feedback and curated by users based on their preferences. Ferbach et al. (2024) recently showed that when data is curated according to user preferences, the self-consuming retraining loop drives the model to converge toward a distribution that optimizes those preferences. However, in practice, data curation is often noisy or adversarially manipulated. For example, competing platforms may recruit malicious users to adversarially curate data and disrupt rival models. In this paper, we study how generative models evolve under self-consuming retraining loops with noisy and adversarially curated data. We theoretically analyze the impact of such noisy data curation on generative models and identify conditions for the robustness and stability of the retraining process. Building on this analysis, we design attack algorithms for competitive adversarial scenarios, where a platform with a limited budget employs malicious users to misalign a rival’s model from actual user preferences. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithms.

Self- Self-Consuming Generative Models with Adversarially Curated Data
Xiukun Wei Xueru Zhang'
Abstract
the internet, is inevitably used for training future gener- ations of models, creating "seelf-onsuuing training loop." A line of research has focused on examining the impact of this self- celf-consuming training loop on generative models' outputs, both theoretically (Bertrand et al., 2024; Taori & Hashimoto, 2023) and empirically (Gerstgrasser et al., 2024; Alemohammad et al., 2024a; Shumailov al., 2024). These studies demonstrate that such a conf-ccnnu training loop may lead to model collapse (Gerstgrasser al., 2024; Ale- mohammad et al., 2024a; Shumailov al., 2024), training instability (Bertrand et al., 2024), and possibly bias am- plification (Taori & Hashimoto, 2023; Wyllie et al., 2024; Xie Zhang, 2024). Several solutions have also been pro- posed to mitigate these issues, such as integration of real data (Bertrand al., 2024), leveraging cumulative datasets (Gerstgrasser et al., 2024), and employing Self- Self-Improving Diffusion Models with Synthetic Data (SIMS) (Alemoham- mad et al., 2024b).
Recent advances in generative models have made it increasingly difficult to distinguish real data from model-generated synthetic data. Using synthetic data for successive training of future model generations creates "self-consuming loops." which may lead model collapse or training in- stability. Furthermore, synthetic data is often sub- ject human feedback and curated by users based on their preferences. Ferbach et al. (2024) re- cently showed that when data is curated according to user preferences, the self- consuming retraining loop drives the model converge toward a distri- bution that optimizes those preferences. However, in practice, data curation is often noisy or adver- sarially manipulated. For example, competing platforms may recruit malicious users to adversar- ially curate data and disrupt rival models. In this paper, we study how generative models evolve under self-consuming retraining loops with noisy and adversarially curated data. We theoretically analyze the impact of such noisy data curation generative models and identify conditions for the robustness of the retraining process. Building on this analysis, we design attack algorithms for competitive adversarial scenarios, where plat- form with a limited budget employs malicious users to misalign a rival's model from actual user preferences. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithms.
In contrast to these works, a recent study (Ferbach et al., 2024) explores a more practical scenario in which synthetic data is curated by human users. To improve safety, user trust, and the relevance and quality of generated outputs, mod- ern generative models are increasingly trained with human participation and feedback. For example, platforms such as JourneyDB (Pan et al., 2023) and Pika Labs (Labs, 2025) provide multiple variations of outputs for users to choose from, with only selected outputs being upscaled and used to train next-generation models. As shown in Ferbach et al. (2024), when synthetic data is curated based a reward model representing user preferences, the generative models trained iteratively this curated data tend to converge to an output distribution that maximizes the expected reward.
However, in practice, data curated from users are likely to be noisy, biased, or even maliciously manipulated. Consider scenario where multiple platforms compete for the same target user population with similar preference distributions (e.g., ChatGPT (OpenAI, 2025) and Claude (Anthropic, 2025), Stable Diffusson (Diffusion, 2025) and MidJourney (MidJoumey, 2025)). To compete for market share, plat- forms may leverage their collected datasets, which contain rich information about actual user preferences, to design attack algorithms targeting their competitors. For example, as shown in Fig. 1, platform may employ malicious users with limited budgets to deliberately select outputs on ri-
1. Introduction
The latest generative models can produce highly realistic texts (OpenAL, 2024), images (Diffusion, 2025), audio (AI, 2025), and videos (MI., 2025). As synthetic data proliferates
DDpaarmeent of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, USA. Correspondence to: Xueru Zhang <zhang. 1 12807 osu..du>>
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Self- Belf-Consuming Generative Models with Adversarially Curated Data
Target Platform
Adversarial Platform
3. Sample &3 upscaled by the ranking
Only the upsealed samples are in the dataset Dr
4. Only the upsealed samples are in the dataset Dr
5. Train the model Pt-1 on the dataset Dr
Train the model P 244 the dataset Dj
U.eer picks prompts y- -Paer ueer (y)
2.Sample: xs, x2z x3, X4 -P, (xly)
Malicious curation using attack algorithms with limited budgets
Figure 1. An example of adversarial data curation in competitive setting: the adversarial platform and the target platform serve the same population of users. The adversarial platform can access the data generated by the target platform and obtain real user preferences through its own preference collection mechanisms. Using the attack algorithm. the adversarial platform employs malicious users to adversarially curate data on the target platform, preventing its model from aligning with genuine user preferences.
val's platform that significantly deviate from genuine user preferences, creating dataset on the competing platform that no longer reflects true user preferences. Over time, this adversarially curated data degrades the competitor's 's ability to train models that align with user preferences, ultimately reducing their capacity to produce content that attracts users.
work is Wu al. (2025), which investigates the vulnera- bility of reward model learning to preference poisoning. However, unlike our work that focuses on a self-consuming training loop, where the attacker aims to gradually misalign the target model with human preferences, Wu et al. (2025) considers static setting where the attacker aims to flip pref- erence labels promote or demote specifie target outcomes under the learned reward model. In Appendix A, we discuss more related works.
This paper takes first step in examining the impact of adver- sarially curated data the iterative retraining of generative models. We provide a theoretical analysis to understand how the output distribution of generative models evolves under adversarially curated data and assess the robustness of the self-consuming training loop to such manipulations. Our results demonstrate that under specific conditions, the self- consuming generative model trained from such adversarially curated data remains robust that it still converges to output distribution that optimizes user preferences. However, we also identify conditions (on the fraction of adversarial data and its associated reward function) under which this robust- ness guarantee fails to hold.
Our contributions are summarized as follows:
In Section 3, we theoretically analyze the long-term per- formance of self-consuming generative models under ad- versarial data curation, considering both pure synthetic data and mixtures of synthetic and real data.
In Lemma 3.3, we prove that the convergence of the gen- erative model toward maximizing user-expected rewards is governed by the covariance of adversarially curated synthetic data. This result establishes conditions where convergence remains robust and conditions where adver- sarial data curation hinders model alignment with human preferences.
Building on this theoretical understanding, we design attack algorithms for adversarial data curation aimed at disrupting the alignment of self-consuming generative models with user preferences. Specifically, we consider a competitive scenario in which platforms compete for market share by employing malicious users to curate adversarial data on a rival's platform. Given dataset reflecting genuine user preferences (collected from the platform's own users), our attack algorithms strategically flip limited number of pref- erence labels. The modified dataset then guides malicious users in curating adversarial data on the rival platform. Over time, the generative models on the targeted platform, when iteratively trained on such adversarially curated data, fail to align with genuine user preferences, reducing their ability to remain appealing to users. To the best of our knowledge, this is the first attack algorithm for deviating self-consuming generative models from user preference. The most related
In Section 4, we model a competitive situation where an adversarial platform aims to disrupt competitor's 's model alignment through adversarial data curation, and propose brasee and heuristic attack algorithms.
In Section 5, we validate the theorems and proposed algo- rithms through experiments on synthetic and real datasets.
experimne
2. Problem Formulation
Consider a platform that iteratively trains generative models from data curated by users. Denote Pdata E (((dd) as the real data distribution and for t E N, let Pt E P(Rd) be the data distribution of the generative model at f-th round of iterative retraining loop. Throughout the paper, we use
2

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
lowercase letters p to denote densities and uppercase letters IP indicate the associated probabilities.
Ezopl [er(a)) converges the maximum reward, and its vari- ance Vargop, [ex())] vanishes. However, in the presence of malicious users, it remains unclear how the model evolves and whether ps can align with actual user preferences in the long run. This paper explores this problem and we first ex- amine the long-term impact of adversarially curated data on self-consuming models (Section 3) and then design attack algorithms that platforms can use to disrupt competitors' model training processes and misalign their models from user preferences (Section 4).
Adversarially curated data. every round & E N, the plat- form presents synthetic data 21.- , . randomly sampled from current model P: to users, and users select their pre- ferred outputs that will be upscaled in the dataset for training -exeereenrttion of models. Following Ferbach al. (2024), we adopt the generalized Bradley-Terry model (Bradley & Tenry, 1952) to model user's choice. Specifically, let r(x:) be an underlying reward function that captures user preference of one data I3 over another zy, then the probability that a data & E {I1,' "1 ] } is curated by the user is as follows:
3. Impact of adversarially curated data
(x(x)) R Σ er(o,))
Next, we examine the evolution of self-consuming genera- tive models with adversarially curated data. We begin with the case of purely synthetic data (A + oo) and then gener- alize to a mixture of synthetic and real data (A < cc). All proofs can be found in Appendix B.
P (6 = 3y F1, -1 - xK)
EE
[K] (1)
Because curated data in practice may be noisy and adver- sarially manipulated, we use a mixture model account for such adversarial behavior. Specifically, assume another competing platform employs malicious users to curate noisy data deviating from actual user preference. Let d, E 1) be the fraction of malicious users at round t and Fz (x) the un- derlying reward function. Then the probability è is curated by a mixture of malicious and benign users is:
Iterative retraining only on curated synthetic data, With- out real data, the iterative retraining process reduces to:
Pc+1 arg max E "1 SxxPP PEP BOBT BT (156))))
[log (p (f))
(4)
Lemma 3.1. Consider the asymptotic case where the num-
ber of samples users select from satisfies K + co. Suppose
e (ma)))
Ezapi (e)t))] 0o and p1+1 follows Eq. (4). Then, we have
F(i = I& Ii,...., ..k.....k)=_. =(1 - Φ1)
(2)
Σ 2j=1 (*(ay)
((t)
((c)) 0e Ezmp, [eFite)]
(+)) + pu(x)) (1 (z) Eempi [er(e)]
Φ1
[K]
K Σj=1 (x)))
Lemma 3.1 characterizes the relation between Pt++1 and pt in self-consuming loop with adversarially curated data. Next, we analyze the evolution of Ep, [er())], which quantifies the expected reward users experience from interacting with the generative model. We first introduce some technical assumptions similar to Ferbach et al. (2024).
We use à ~ BT (e....., )x) to denote è is sampled from II, IK according to probability (2).
Self-consuming training loop. Given adversarially curated data, the platform updates its model at round t+1 according to the following, either solely the distribution of curated synthetic data (A- - oo) or on a mixture of synthetic and real data (A < oo):
Assumption 3.2. There exist finite constants Fr.mins Tr,max- Tt,min Ft.max E R. such that: Pe-almost surely, Vz ~ Pastas Tt,iin = inf, r(x). Tt,max = sup,, r(x), Tt.mia inf, (r)),, Fi,max = sup, (x)).
1
Pt+1 arg max PEP
E
Exppon [log (p (=))
1
1+X
In most realistic scenarios, this assumption holds because user preferences typically have finite support or are bounded in a probabilistic sense. Under this assumption, Lemma 3.3 below examines the impact of adversarially curated data on Ep [<)( and presents its upper and lower bounds.
A
I+X
E
2 2
Dog (p (+))
(3)
where À E [0, oo) controls the fraction of real data and P is the set of achievable distributions with the platform's 's model.
Lemma 3.3. Let p1+1 be the distribution induced from discrete choice model in Eq. (4). Suppose Assumption 3.2 holds, then the following holds:
Objectives. A recent study (Ferbach et al., 2024) fo- cused special case without malicious users (d. 0). They showed that if data are curated based on the be- nign users' reward function r(z) and the generative model is updated solely using curated synthetic data, the self- consuming training loop can result in Pt converging to a distribution that optimizes user preferences, i.e., as f -> 0o,
Epeci [or(e]] 2 Ep. [er(c]]
Var
Cov
max
Epez1 [er(s)]
S Ep.
[er(e)]
Var
Cov
minn
3

---

Self- Belf-Consuming Generative Models with Adversarially Curated Data
where
and Ep, [ertx)] > (eta)) still holds. However, when Covmin < O, this is no longer the case, suggesting that simply adding real data is not sufficient to defend against adversarial curation.
Var = (1- 61) (K -61 1) Varp, [ertar]
K
Cov = Øi (K- -1) Co pe (erte) c
K
4. Attack algorithms
According Lemma 3.3, when e²(x) and (T are pos- itively correlated, i.c., Co 2 O, the expected reward increases, pp.,, [e"(00]] 2 Ep. [er(=)], allowing the gener- ative model to align with user preferences despite adversari- ally curated data. In this case, the expected reward converges to the maximum, highlighting the model's s inherent robust- ness against noise and adversarial attacks. However, when e*(x) and (*)()) are negatively correlated, i.e., Covp, < O, the convergence is no longer guaranteed. Instead, the ex- pected reward may oscillate and deviate from the maximum value. This shows the model's potential vulnerability in scenarios where adversarially curated data introduces nega- tive correlations with the user reward function. The rationale for the upper bound is discussed in Appendix B.3.
As shown in Lemma 3.3, adversarially curated data may result in Ey-ps [c](*)] deviating from maximum reward in the long run. Next, we consider a competitive scenario illustrated in Fig. 1, where an adversarial platform em- ploys malicious users to curate data on a target platform to compete for market share. Our goal is to design attack algorithms for the adversarial platform that guide malicious users to act in the most effective way to disrupt the target platform's 's model.
Learning reward model from pairwise comparisons. Un- like Ferbach et al. (2024) that assumes user reward model r(x) is known, we consider a more realistic setting that the adversarial platform does not have access to r(x) but must learn it from a dataset indicating user preferences. Here, we consider learning from pairwise comparisons (Wu et al., 2025; Liu al., 2024; Zhou et al., 2025) detailed below.
Iterative retraining mixed real and synthetic data. Prior works such as Ferbach et al. (2024); Bertrand et al. (2024); Gerstgrasser et al. (2024); Alemohammad et al. (2024a) explored the role of real data in self-consuming generative models. Without data curation, Bertrand et al. (2024); Gerstgrasser et al. (2024); Alemohammad et al. (2024a) showed that retraining models with a mix of real and synthetic data help stabilize the algorithm and prevent Pe from deviating too much from Pdata- When synthetic data is curated by users based on their preferences, Pdata is no longer fixed point of the retraining loop, as different re- ward values can occur with positive probability; however, [e((r) still increases compared to Epan [e^(=)] (Ferbach et al., 2024). Next, we study whether incorporating real data can help defend against adversarially curated data during iterative model retraining.
Let D {(x;, 2i, ,.) be the dataset adversarial plat- form collects from benign users, where I₁, 2g E R are i-th pair of data samples acquired from the target platform and o E 0.,0.5, 13 indicates the user's preference among them'. Specifically, o, = if x; is preferred to 2j (x; > z2), o = if 2; is preferred to Iy (I; < 24), and 0 = 0.5 if Yi and =; are equally preferred. In this paper, we assume adversaria and target platforms face users with identically distributed preferences, so that the adversarial platform can learn r(z) from D.
Let Rg be a parametric reward model of ((x) learned from preference data D, where 0 E θ is the parameter. A typical method Maximum Likelihood Estimation (MLE) which minimizes the following loss function:
Lemma 3.4. Let P++1 be defined as in Eq. (3), with po Paata and Φe w., Vt. The following holds for all t:
((D:0)=-X [11 o.) log fzz > 21 Ro}
[er(e)]
E Epaa [e)(c)]
E Epec1
(5)
+o, log Pr (zi > xx RaHl
(7)
(1((A)))
A
+ (1+
1-
Covinn
where preference label o, is generated according to
((..))
where Covmin = min Covp. (er(a), eFilee) (E(1)
0, ~ Prf2, > 111r)= ('())) + en(e)
(8)
Ast + oc.
Eptt1 [ertes]
Ep Epsn [orte]] +
Objective and constraint of the adversarial platform. To compete against the target platform, the adversarial platform
>
(6)
'To obtain this dataset, the adversarial platform can first ac- quire K data samples (.1,-.., ,IK] } from target platform and then present them to its users select. Suppose K = and the user selects x2, the data pair {(11, x2, 1), (222, 23, 0)} is added to D.
Lemma 3.4 implies that when Covmin > 0, i.e., the ad- versarial reward values are positively correlated with the true rewards, model can align with genuine user preferences

---

Self-Consuming Generative Models with Adversarially Curated Data
employs malicious users to adversarially curate data on the target platform, aiming cause the generative models itera- tively trained on this curated data to deviate from user pref- erences. The behavior of malicious users can be formalized as "flipping the preference label og of data pairs." Specifi- cally, given D= = {(z;; Zi o)){: malicious users can flip preference 0; to o; + 8, resulting in an adversarial dataset D(6) { 24, o + 6,));=1, where 6, {-1,0, 1} presents label perturbation and 6- == {6. )2= In practice, there is possibility that malicious users fail to curate data adversarially on the target platform, or the curated data is not selected for training the future model. To account for this, we impose a constraint on the total number of label flips Σ |6:] S K n,, where x E (0, 1) represents the success rate of perturbing the data on the target platform.
then solve for ( in optimization (10). The resulting (() can then guide the malicious users in curating data on the target platform at t. Such adversarially curated data is subse- quently used by the target platform to retrain its model P-+1- Over time, this iterative interaction may cause the target platform to deviate significantly from user preferences.
Challenges to solve optimization (10). The solution to (10) is difficult to find because it a bi-level optimization prob- lem. Moreover, the variables to be optimized are subset of preference comparison labels, which involves solving a combinatorial optimization problem over discrete space. Next, we tackle these challenges and introduce two methods for finding approximated solutions to optimization (10).
4.1. Gradient-based methods
The goal of the adversarial platform is to find 8 such that the resulting perturbed dataset D(5), when mixed with benign users' preference data (via malicious users' adversarial data curation), disrupts the target platform's alignment with user preferences. Formally, let Pt+1 be the self- consuming model of the target platform trained from such adversarially curated data, the goal is reduce the expected reward Ex-pp [er(=)] such that it can deviate from the maximum reward in the long run, i.c.
To tackle discrete decision space, we relax the action space 8, E (-1,0,1) 1 to interval ó; E [-1, 1]. If reward models are differentiable with respect to 0, then we can compute the gradient of the objective function in (10) as follows:
V8J(6) V8 (Corm. Rolt), e BRatr)] + a dist (Ro. Ra)
=Vo (Covpn [cRo(o), , Ro(o)] + a dist (Ro. Rop)
d0
dā
[ [B((x)) < Ep, [o(()))
Epeci
(9)
Recall that Rg is the reward model trained from perturbed dataset D(8), parameter 0 is function of 8. Similar to Wu et al. (2025), we can leverage the implicit function theorem (Mei & Zhu, 2015; Koh & Liang, 2017) to compute the implicit derivative do- 25.
where Ro is parametric reward model learned from benign users. Based on Lemma 3.3, we formulate the following optimization for adversarial platform:
minJ(() == =Covpe [e (e Raloo] + a dist (Ro. Ri)
dō = - [H -1 d5
VV d8
(11)
s.t. Ō arg min C (D(8),0)
(10)
where HyC is the Hessian of L with respect to A. Given the gradient, we can then apply projected gradient descent to find optimal 8*. Since original action space is 6, e {-1, 0, 1} and the total number of flips is constrained by Σ [6 = R n. we select the the top K n among 8* based magnitude |6%] and only flip o associated to them. The complete procedure is shown in Algorithm 1.
where Ri is parametric reward model learned from per- turbed preference data D(8), which may belong to a differ- ent function family than Ro. To achieve the objective in Eq. (9), we aim to make (e (g())) as negative as possible. Meanwhile, to prevent the adversarial behav- ior from being easily detected as anomalous, we impose penalty on the difference between R₂ and Rg, quantified by dist Rg, Rig) and can be defined as Ep, [d[ [o Rx(x), Rg(x))] for some distance metric d (e.g.. lp norm).
The efficiency of this algorithm is primarily influenced by two key factors: batch size and model structure. Batch size plays a critical role in the accuracy of covariance estimation, which is essential for aligning the computed covariance with the true data distribution. While larger batch sizes gener- ally yield more precise covariance estimates and reduce the discrepancy between theoretical and practical outcomes, they also incur greater memory requirements and computa- tional overhead. Meanwhile, the structure of the initialized model directly influences the size of the implicit Hessian matrix, which emerges during gradient computations in- volving second-order derivatives. These calculations are
Dynamic attack during iterative training. Since the target platform dynamically updates its model over time, the adversarial platform must repeatedly solve optimization (10) as Pt evolves. In practice, the adversarial platform can pe- riodically interact with the target platform to acquire data (1) (1) Pt and collect user preference 0f
pairs x. .22
(8) from its
own customers. The dataset (tt) = {(a)") (4) o (0)] )R
can be first used to fine-tune benign reward model Rg and
5

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Algorithm 1 Gradient- attack
the user, making the preference flip most effective.
Input: Benign preference data D. parameter it
Multi-objective heuristic method. Since optimization problem (10) simultaneously considers two competing ob- jectives, we also propose a heuristic method that finds the Pareto front (Ngatchou et al., 2005) for multi-objective op- timization. Specifically, a solution is considered Pareto optimal if no other solution exists that can improve one objective without degrading at least one other objective; the set of all Pareto optimal solutions forms the Pareto front.
Train the reward model Rg of benign users on D: Randomly initialize 6, ensuring (o; + d,) E [-1,1] for m to M do
Create a perturbed dataset D(6);
Train new reward model Rg on the dataset D(6);
Compute the gradient [6))
8 - 8 - 77 so];;
Clip 5 such that (o + 6,) E [11,,]] e
end for
Our method begins by generating K- n. random perturbations ò to flip data D, resulting in D(0). For each perturbation, we record the empirical performances of Cov (e RBaav) and dist Ro: Rg). Using a Pareto optimization algorithm, such as NSGA-II (Deb et al., 2002), we compute the Pareto front of non-dominated solutions, which represents the best trade-off's between objectives. Finally, we select the optimal solution from the Pareto front based specific prioritized objective, and apply the corresponding flip.
Select the top K- n indices based on 16:|:
Flip the preference label of the corresponding data pair; Output: Label perturbations 5, reward model Rg
computationally expensive. Although approximate meth- ods can be used to estimate the Hessian matrix and reduce the computational burden, models with complex architec- sure as deep neural networks with many layers and parameters- subsiantiill increase both the computational time and memory requirements for these calculations.
5. Experiments
In this section, we conduct experiments on both synthetic and real data to validate our theorems and proposed algo- rithms. We first evaluate the evolution of generative model Pt and user reward Ep, [r(=)] under self-consuming training loop (Section 5.1). Then, we demonstrate the effectiveness of proposed attack algorithms (Section 5.2).
4.2. Heuristic methods
The high computational costs associated with gradient-base -based calculations for complex models present significant chal- lenge. To mitigate this issue, we propose leveraging heuris- tic methods as an alternative. These methods eliminate the need for explicit gradient computation, providing a more computationally efficient approach, as detailed below.
Datasets. Similar to Ferbach et al. (2024), we conduct experiments on three datasets:
Reward- based heuristic method. Instead of directly opti- mizing perturbations ō to minimize Covp, p. [e Ro()), e Ro(e))
1. Synthetic Gaussian: A dataset following 8-mode Gaus- sian mixture model, the details are in Appendix C.2.
cistt(R. RR, R2)), we adopt a heuristic approach by flipping the preference label o; based on the rewards of data samples.
2. CIFAR-10 (Krizhevsky, 2009): It contains 60,000 im- ages from 10 classes {airpine := 0, automobile ==
Specifically, given preference data D= = {II,, 2g, ),));= we first learn the reward model Rg from D. The idea is to identify K n data pairs (I,, 2,) based on their rewards Ro (a:)), Ro (2:) such that flipping their preference label o has the greatest impact on the underlying reward model.
1, bird == 2, cat = 3, deer == 4, dog == 5, frog ==
6, horse := 7, ship == 8, truck == 9}.
3. CIFAR-100 (Krizhevsky, 2009):
contains 100
classes, each with 600 images. Class labels from 0 to 99 are assigned according to the alphabetical or-
We propose two methods for finding such data pairs: (i) finding (Ira, 25) based on dissimilarity between Ro (ara) and Ra (z.); ; (ii) finding (zi, x) based on maximum of [x:)] and RR (2.).. Specifically, define f : Rdd x Rd
der of class names, i.e., {aquatic mammals-beaver := O...., vehicles 2-tractor := -99].
+ R as
We present the results for CIFAR-10 10 and CIFAR-100 below, while the results for the Gaussian data are provided in the Appendix C.2.
f(z, 2) = [R) - Ro(z)) or max Ro m (x)1. Rx(2))
We select the K data pairs (z,, 2)) with the high- est f(z;, 2;) values to flip their preference label. Intu- itively, a larger [R((;;)= - Ro(=)) indicates greater differ- ences in user preferences for the data pair (2,, mak- ing the preference flip more impactful. Similarly, a larger max{ RR ). [2))] suggests that the pair includes a sample that is either highly favored or strongly disliked by
Reward functions and user preference labels. Using Gaussian, CIFAR-10, and CIFAR- 100 datasets, we can construct user preference dataset D = ((r,, zi, ox) us- ing reward function r(z). Specifically, given data pair (I;, 2;) sampled from Gaussian, CIFAR-10 or CIFAR-100 100 datasets, we generate the corresponding preference label oi ~ Pr(z, > It r) based on Eq. (8). r(x) for each
6

---

Self- Celf-Conuuming Generative Models with Adversarially Curated Data
airplane
automobile
-
bird
cat
deec
6og
Frog
honse
ship
truck
0.6
0.6
0.6
Renign Curatio
Adversarial Caration (Gradientt)
Advensarial Curation (Random)
0.5
0.5
Propurtion
0.4
Proporties
0.4
Propurton
0.4
0.3
0.3
0.3
0.1
0.1
0.1
5 6
8
10
Iteration
lseration
Heration
Figure The proportion of each class generated by self-consumption model retrained with different data curation methods on CIFAR-10: henign curation based on actual user preferences (left), adversarial curation using the proposed gradient-based attack algorithm (middle), and adversarial curation via a random attack (right). results show that the proposed gradient-basee attacks are the most effective in deviating the model from preferences.
#T0-519
A40-449
Benign Caration
Mallclous22155)
#70-079
#90u599
0.3
00.4
85
Benign Caration
Adversurial Curation (Gmdieen)
0.7
0.35
0.6
75
0.5
Proportionn
0.25
70
0.4
0.3
0.15
60
0.2
0.1
35
0.1
0.05
9
10
lteration
Beration
Figure 3. The proportion of each ten classes generated by the colf-consumption model retrained with different data curation methods CIFAR- -00:: benign curation based on actual preferences (left), adversarial curation using the proposed gradient basdd attack algorithm (middle). And empirical estimate of Ep, [r(x)] from samples generated by the model over iterations (right).
dataset is defined as follows:
probabilistic model (DDPM) (Ho al., 2020), a generative framework known for its ability model complex data dis- tributions through a reversible diffusion process. In addition to the generative model, the target and adversarial platforms leverage user preference data D = {(x,, Yi, o;) to train reward model Rg- For the adversarial platform, it also trains Rg using perturbed preference data D(6). The reward models R and f may or may not have the same architecture, we discuss details in Appendix C.I.
Synthetic Gaussian: r(z) : : -y max [|c--,-11-7 where || - Hell the distance from one Gaussian cen- ter je., is the minimum clipped radius, and ry controls the scaling of the reward (details are in Appendix C.2). Intuitively, this function captures user preferences by as- signing higher rewards to samples farther from Gaussian center within a threshold.
CIFAR-10: First identify the label (xx) E {0.-..,9) 9} of image by pretrained VGGI1 (Simonyan & Zisser- man, 2015) classifier with 92.79% test accuracy. Sup- pose users prefer classes with smaller indices and define r(x) = 10 - [x)). reflects user's preference ordering by assigning higher rewards to images classified closer to the most preferred class (class 0) in the hierarchy.
Iterative retraining process. At each iteration, the genera- tive model produces 50,000 data samples, of which 25,000 are selected (after curation by the reward model) for retrain- ing the next-generation model. In Appendix C, we provide details the process of (adversarial) data curation and their interaction with the generative model training. The complete process is presented in Algorithm 2.
3. CIFAR-100: Similar CIFAR-10, the label [(x) E ,99] of image v is first identified by pretrained ResNet56 (He et al., 2016) classifier with 72.63% test ac- curacy. Suppose users prefer classes with smaller indices [xx).
5.1. Evolution of model under adversarial data curation
First, we examine the impact of adversarial data curation on the self-consuming generative model pt and the respective reward Ep, [r()). Specifically, we employ both the gradient- based attack (Algorithm I) and random attack (i.e., flip
and define r(xr) = 100
Generative model and reward model. Following Ferbach (2024), we iteratively retrain denoising diffusion

---

Self- Celf-Consuming Generative Models with Adversarially Curated Data
Benign Cunition
Gradient
Random
10
Table 1. Fffeecivveness of one-round attack under different methods on the same CIFAR- 10 dataset: The results are empirical estimates of Ep [r(=)] for generated samples at 1: method with a lower reward is more effective.
Avg. Rewaads
8
6
METHOD BENIGN
GRADIENT #1
GRADIENT #2
AvG. R
6.3606
5.6460
5.5959
METHOD PARETO
R-BASED #1
R-BASED #2
0
2
6
7
8
10
lteration
AVG. R
5,4740
5,5982
5,5612
Figure 4. Empirical estimate of Bp, [=(] from model-generated samples over iterations on CIFAR-10: it increases (resp. decreases) over iterations under benign (resp. adversarial) data curation.
ences over time. In contrast, under adversarial curation, the model generates more samples from less preferred classes (i.e., classes 60 to 99). The average rewards shown on the right further highlight this difference: benign curation leads to a steady increase in expected rewards, whereas adversar- ial curation causes a continuous decline, indicating growing misalignment between the model and user preferences.
preference labels uniformly at random) algorithms to target 20% of data pairs.
Fig. 2 shows the evolution of the proportion of each class generated during the iterative retraining process on CIFAR- 10 and demonstrates the impact of three types of data curation on self-consuming models: benign curation, adver- sarial curation via gradient-based based attacks, and adversarial curation via random attacks. Without adversarially curated data (left), the generative model gradually aligns with hu- man preferences, producing an increasing number of sam- ples from class 0 (airplane), the most preferred class in C0. This observation is consistent with Ferbach et al. (2024). Under random attacks, while the model does not align well with user preferences as in benign curation, it still generates a reasonable proportion of samples from the most favored classes, O (airplane) and 1 (automobile). In contrast, with our proposed attack algorithm, the model becomes highly misaligned with user preferences, generat- ing samples predominantly from the least favored classes: (horse), 8 (ship), and 9 (frog).
We also conducted iterative model retraining on a mixture of ad adversarially curated synthetic and real data on CIFAR-10 10, where adversarial data generated through gradient-baser attacks is combined with real data at varying proportions. Fig. 5 shows the class proportions of the generated samples over iterations. The results demonstrate that incorporating real data helps align the distribution of generated samples with the real data distribution. However, it does not steer the model toward the user- see--preeeree distribution. This suggests that adding a limited amount of real data is insufficient and fails to defend against adversarially curated data.
To examine model quality, we present synthetic images generated by self-consuming models under both adversarial and benign data curation in Fig. 7 (Appendix C.I). The results show that the image quality does not significantly degrade during the iterative retraining process.
Fig. 4 shows the empirical estimate of Ep, [r(x)] for gen- erated samples over iterations on CIFAR-10, 10, with observa- tions consistent with Lemma 3.3. Under benign curation, the expected reward steadily increases as the model aligns with user preferences. In contrast, adversarial curation using the gradient-base based attack continuously lowers the reward, indicating a significant deviation from the optimal reward distribution. Random attacks initially lead to slight in- crease in rewards, suggesting some robustness. However, as attacks progress, the average reward fluctuates due to the inherent randomness of this method. Indeed, the out- comes for random attacks vary significantly across different runs of experiments, and we provide additional examples in Appendix C.I.
5.2. Effectiveness of attack methods
To evaluate the effectiveness of various attack algorithms, we applied different attack methods to the same CIFAR-10 dataset, flipping 20% of the data pairs. Table summa- rizes the empirical estimates of Ey [r(=)] for generated samples at = 1. Comparisons include benign curation (BENiGN), gradient-based attacks that target platform and adversarial platform using different (GRADIENT # #1) or iden- tical (GRADIEN #2) reward models, reward- based heuris- tic methods with f(z, 2) = [Ro(z) - Re(z)) (R-Ba (R-BASED #1 ) or max [R (x)], [=)))]] (R-BASED #2), and multi- objective heuristic method (PARETO), where we select the solution from the Pareto front with the lowest sum of the two metrics. We also show the class proportions of generated samples over iterations in Fig. 8 (Appendix C.
We extend our experiments to CIFAR- 100, which contains a larger number of classes. Fig. 3 illustrates the different be- haviors under benign curation and adversarial curation using the gradient- based attack. With benign curation, the model progressively generates more samples from user-pre useferrefr classes, reflecting an increasing alignment with user prefer-
Overall, the results demonstrate the effectiveness of all at- tack methods in this experimental scenario. Despite employ- ing different reward model architectures, gradient-basse
8

---

Self- Celf-Consuming Generative Models with Adversarially Curated Data
sirplane
automobiie
-- bird
deec
Frog
homs
truck
0.6
0.6
Mixed (Real Data: Advensarial Curation 1:4)
Mixed (Real Date: Advensriial Curation 1:])
Mixed (Real Date: Adversarial Curntion 4:1)
0.5
0.5
0.5
Ptoporton
0.4
Proportonn
0.4
Proportion
0.4
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
1
3
6
8
10
0
2
6
7
9
10
0
2
5 Iteration
9 10
Ieeration
Beration
Figure 5. The proportion of each class generated by the self- consumptonn model retrained with a mix of adversarially curated synthetic and real CIFAR-I0 10 data. shows that adding real data only helps the model align with the real data distribution Piuza but does not defend against adversarial data curation.
attack methods consistently maintained high efficacy. In contrast, the relatively weaker performance of heuristic methods is reflected in their higher average rewards. As shown in Fig. 8, this is primarily due to heuristic methods failing to account for all classes comprehensively, resulting in limited effects on certain classes. The multi-objective heuristic method performs best by exploring broader range of potential solutions, yielding the lowest average reward. However, this performance comes at the cost of significantly higher computational time and resource requirements.
biases. Additionally, our attack algorithm already considers such defense mechanisms: when formulating in Eq. (10), we impose a penalty term dist Ro, R₂) to prevent the adver- sarial behavior from being easily detected as anomalous.
Limitations. Our theoretical analysis assumes that each model update converges to the global optimum of the train- ing objective. While our experiments validate the effec- tiveness of the proposed attack under this assumption, such convergence may not always hold in practice due to opti- mization noise, local minima, or limited training budgets.
It is important to note that the effectiveness of these attack methods may depend heavily on the specific context, and no method can be considered "optimal". Future research could focus on developing solutions that are more general and universally applicable.
Additionally, our experiments rely on a known success ratio K to control the adversarial curation. In real-world scenar- ios, however, the attacker may not have direct access to or precise knowledge of the effective success rate, which could affect the practical impact of the attack.
6. Discussion
7. Conclusion
In this section, we discuss the robustness of existing defense strategies and analyze the practical challenges of defending against our proposed attack. We also discuss key assump- tions made in our analysis and identify limitations that may affect applicability in real-world scenarios.
This paper examines the evolution of self-consuming gen- erative models under adversarially curated data. We theo- retically analyze the impact of adversarial data curation on these models and identify conditions for the (in)stability of the iterative retraining process. Building on theoretical in- sights, we develop attack algorithms that effectively disrupt model training and prevent alignment with user preferences.
Defense. A common strategy proposed in prior work to stabilize the self-consuming retraining loop of generative models is to regularly inject real data during training (Ale- mohammad et al., 2024a; Bertrand et al., 2024). However, as shown in the experiments in Fig. 5, adding real data only partially mitigates adversarial effects by driving the model closer to the true data distribution Piata- It does not effectively prevent model misalignment under targeted ad- versarial curation attacks.
The findings highlight the potential vulnerability of self- consuming generative models to adversarial data curation, suggesting that developing effective defense mechanisms could be promising direction for future work.
A Acknovledgement
This material is based upon work supported by the U.S. National Science Foundation under award IIS2202699 and USS22416855, by OSU President's Research Excellence Ac- celerator Grant, and grants from the Ohio State University's Translational Data Analytics Institute and College of Engi- neering Strategic Research Initiative.
Although outlier detection may assist in identifying and fil- tering adversarially curated samples, they can inadvertently remove genuine preferences. When users are heterogeneous and come from multiple groups, removing genuine pref- erences from minority groups may potentially introduce

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Impact Statement
and Tramèr, F. Poisoning web-scale training datasets is practical. In 2024 IEEE Symposium Security and Privacy (SP), pp. 407-425, 2024, doi: 10.1109/SP54263, 2024.00179.
This paper advances the field of Machine Learning by exam- ining the long-term impact of the self-consuming retraining loop on generative models, well as the role of (adversar- ial) data curation in this process. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Chen, T. Hirota, Y., Otani, M., Garcia, N., and Nakashima, Y. Would deep generative models amplify bias in future models? In 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 10833-10843, 2024. doi: 10.1 100///CV52233..0222..oo
References
AI, R. Resemble AI AI voice generation and cloning plat- form. https:/ /www www.reeeemle....... 2025. Ac- cessed: 2025-01-10.
Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2): 182-197, 2002. doi: (0.1109/425599600
Alemohammad, S., Casco-Rodriiuue, J., Luzi, L., Hu- mayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. Self-consuming generative models go MAD. In The Twelfth International Conference on Learning Representations, 2024a. URL https: // //openreview wwwpeeervviee.nt//oooeeeeee
Diffusion, S. Stable diffusion open-source for creating images from text. https: / /tssaaaaledddeee fffffffifooo... com/, 2025. Accessed: 2025-01-10.
Ferbach, D., Bertrand, Q., Bose, A. J., and Gidel, G. Self- consuming generative models with curated data provably optimize human preferences. In Advances in Neural Information Processing Systems, volume 37, pp. 102531- 102567. Curran Associates, Inc., 2024.
iidiiihhh SMHEMPPO..
Alemohammad, S., Humayun, A. I., Agarwal, S., Col- lomosse, J., and Baraniuk, R. Self-improving diffu- sion models with synthetic data, 2024b. URL https: //arxiv. wttp:////a.//....133333
Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Kor- bak, T., Sleight, H., Agrawal, R., Hughes, J., Pai, D. B., Gromov, A., Roberts, D., Yang, D., Donoho, D. L., and Koyejo, S. Is model collapse inevitable? Break- ing the curse of recursion by accumulating real and syn- thetic data. In First Conference on Language Modeling, 2024. URL https:/ oppnrevieweeeeeeeeeeeee net forum? i 1d=5B2K4LRgm2.
Anthropic. Claude A next-generation Al assistant by thropic https: /ttp:::://wwww.. www.rpirtoooiccccoo com/ index/ claude-introduutioon 2025. Accessed: 2025-01- 10.
Baumgärtner, T., Gao, Y., Alon, D., and Metzler, D. Best- of-venom: Attacking RLHF by injecting poisoned pref- erence data. In First Conference on Language Modeling, 2024. URL httpt::t//// wwwp://eeeeeeeeeeeeeeeee net/ forum? 1d=v74mJURD11.
Gillman, N., Freeman, M., Aggarwal, D., HSU, C.-H., Luo, C., Tian, Y. and Sun, C. Self-correcting self-consuming loops for generative model training. In Scaling Self- Improving Foundation Models without Human Super- vision, 2025. URL https: wpwp/eveeeee..eee/e/ee net/ forum? Brtumidd=0BB9aaI11
Bertrand, Q.. Bose, J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data. In The Twelfih International Conference on Learning Representations, 2024. URL https: http///peeeeveieeeeeeeeeeeeeeeee wttp:/iwwwwweee.rrrrreeee id=JoRAfH2xFd.
He, K., Zhang. X., Ren, S., and Sun, J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770-778, 2016. doi: 10. 1119/CVPR.2016.900
Biggio, B., Nelson, B., and Laskov, P. Support vector ma- chines under adversarial label noise. In Proceedings of the Asian Conference on Machine Learning, volume 20, pp. 97-112, South Garden Hotels and Resorts, Taoyuan, Tai- wain, 1.1 PMLR. URL https:h////c/ http::/:///cccenggsss... mlr. mlr.press wrwssprssss.. h2////2/////// http20//iggiol111 html.
Ho, J. Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.
Jiang, S., Kadhe, S., Zhou, Y., Cai, L.,, and Baracaldo, N. Forcing generative models to degenerate ones: The power of data poisoning attacks. In NeurIPS 2023 Workshop on Backdoors in Deep Learning The Good, the Bad, and the Ugly, 2024. URL https: / openreview net/ forum 8=
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39:324-345, 1952.
Carlini, N., Jagielski, M., Choquette- Chhoo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K.,
10

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Koh, P. W. and Liang, P. Understanding black-box pre- dictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, vol- ume 70, pp. 1894, PMLR, 2017. URL https:// proceedi hgs ml www.ppress/v.omm / .tmml.
Paudice, A., Muñoz-Gonzzless L.., and Lupu, C. La- bel sanitization against lahel flipping poisoning attacks. In ECML PKDD 2018 Workshops, pp. 5-15. Springer International Publishing, 2019.
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., An- derson, R., and Gal, Y. AI models collapse when trained on recursively generated data. Nature, 63 (31(2)2)) 759, Jul 2024. URL https: http//....g/.....0/0/0.00......0 s s41586 024 07566-y.
Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. URL https:://wwt cs. toronto. edu/ekrizzz wtrii/ciiaiaaaaaaaaaaaa htt./ifaaaarrrre
Labs, P. Pika labs- Al-generated videos from text. https: //www, pika abss com/, 2025. Accessed: 2025-01- 10.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. In The 3rd International Conference on Learning Repre- sentations, 2015. URL 1409.
Liu, Li, B., Vorobeychik, Y., and Oprea, A. Robust linear regression against training data poisoning. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec '17, pp. 91-102, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10. 1145/3128572.3140442
abs/
Suya, F., Zhang, X., Tian, Y., and Evans, D. What distri- butions are robust to indiscriminate poisoning attacks for linear learners? In Advances in Neural Information Pro- cessing Systems, volume 36, pp. Associates, Inc., 2023.
Liu, Y., Zhou, H., Guo, Z., Shareghi, E., Vulié, I., Korhonen, A., and Collier, N. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In First Conference on Language Modeling, 2024. URL https: hwtpp//////ee.eiee....o. met/fooun; id= id=Sgd2I7c6yr.
Curran
Taori, R. and Hashimoto, T. Data feedback loops: Model-driven amplification of dataset biases. In Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202, pp. 33883-33920. PMIR,, 2023. URL httptt:s: http://s//pppceeeieneee mlr hillsssss/// v202/taori23a. html.
Mei, S. and Zhu, X. Using machine teaching to identify optimal training-set attacks on machine learners. In Pro- ceedings of the Twenty-Ninth AAAI Conference on Artifi- cial Intelligence, AA pp. 2871-2877. AAAI Press, 2015.
Vorobeychik, Y. and Kantarcioglu, M. Data Poison- ing Attacks, pp. 77-98. Springer International Pub- lishing, 2018. URL https: htts://ddi...gg///// ore..orrg10.1077 978 - 580-9_6.
https:///o..rrg//0.1000.
MidJourney. Midjoumey Al-generated art platform. ht https: //tw://wwwww. midjjurnny.....mmmommmmm com/, 2025. Ac- cessed: 2025-01-10.
Wu, J., Wang. J., Xiao, C., Wang. C., Zhang, N., and Vorobeychik, Y. Preference poisoning attacks on reward model learning. In 2025 IEEE Symposiun on Security and Privacy (SP), pp. 1622-1640, 2025. URL https: /do:::///ooii/ wwwp::eeeeeeeeeeeeeeeeeoeeeee 10. T..1009/SP61557 202...00250.0000 00094.
ML, R. Runway MI. Al tools for creators. https: // //runwayml com/, 2025, Accessed: 2025-01-10.
Ngatchou, P., Zarei, A., and El-Sharkawi, Pareto Proceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems, pp. 84-91, 2005. doi: 10.1 10.1109/AAPP222555599999
multi objective optimization.
Wyllie, S., Shumailov, I., and Papernot, N. Fairness feedback loops: Training synthetic data amplifies bias. In Proceedings of the 2024 ACM Conference Fairness, Accountability; and Transparency, FAceT '24, pp. 2113-2147. Association for Computing Ma- chinery, 2024. URL https: pttpp:://doo. http///i..oggg/////// 3630106. 3659029.
OpenAl. GPT-4 technical report, 2024. URL https: //arxiv. hwwlvoorg/aaae httgg//bs//300 23003 08774.
OpenAL. ChatGPT Conversational by OpenAL https: //openai. com/ chmm/hhhttt, 2025. Accessed: 2025- 01-10.
Xie, T. and Zhang, X. Automating data annotation un- der strategic human agents: Risks and potential solu- tions. The Thirty-eghtth Annual Conference on Neural Information Processing Systems, 2024. URL https: / oppnrevvl ew net fwr.mmormmmmmm 3=P00I..3KGG...
Pan, J.. Sun, K., Ge, Y., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y., Dai, J., Qiao, Y., and Li, H. Journeydb: A benchmark for generative image understanding, 2023.

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Zeng, Y., Pan, M., Jahagirdar, H., Jin, M., Lyu, L., and Jia, R. Meta-Sift: How to sift out clean subset in the presence of data poisoning? In 32nd USENIX Security Symposium, pp. 1667-1684, USENIX Association, 2023.
Zhang, H., Li, Y., Ding, B., and Gao, J. Practi- cal data poisoning attack against next-item recommen- dation. Proceedings of The Web Conference 2020, 2020. URL https: htppppp.seee..... sem.ssssaanniicool.... org/CorrpsII::211 :
6122.
Zhou, E., Zheng. G., Wang, B., Xi, Z., Dou, S., Bao, R., Shen, W. Xiong, L.. Fan, J., Mou, Y., Zheng, R., Gui, T., Zhang. Q., and Huang, X. RMB: comprehensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations, 2025. https: / net/ / forum? 1d=kmgr1G9TR0.
/penreview.
12

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Appendix
A. Related work
A.1. Self-consuming generative models
Generative models have demonstrated remarkable success in synthesizing high-quality data. Recent research has focused on the challenges faced by generative models trained iteratively on their own synthetic data. Despite slight differences in the definition of collapse, Alemohammad et al. (2024a); Shumailov al. (2024); Gerstgrasser et al. (2024) all confirm that the model collapses when trained exclusively synthetic datasets, leading to degradation in quality diversity over successive generations. Bertrand et al. (2024); Taori & Hashimoto (2023) theorizes that this collapse arises due to the nature of the iterative training process. Another potential consequence of this iterative training is bias amplification, where the generative model amplifies specific features while neglecting other equally important data characteristics (Chen et al., 2024; Taori & Hashimoto, 2023; Wyllie et al., 2024).
There are two main solutions to this problem. One effective approach to stabilizing generative models is introduce real data into the training process at each iteration. Alemohammad et al. (2024a) provided empirical evidence that injecting fresh real data mitigates model collapse. Bertrand et al. (2024) further substantiated this observation with theoretical proofs, demonstrating that maintaining a sufficient proportion of real data ensures the stability of iterative training. Gerstgrasser et al. (2024) leverages cumulative data, where previously generated samples are stored and reused alongside new real data to stabilize the performance of the generative model. This approach aligns with practical data accumulation strategies. An alternative strategy is to introduce corrective mechanisms to prevent model collapse. Alemohammad et al. (2024b) proposed self-improvemmnt framework for generative diffusion models (SIMS), which mitigates model collapse by employing a negative guidance mechanism during the generation process. Gillman et al. (2025) introduced a self-correcting generative model training loop, where synthetic data undergoes transformation through expert-informet correction functions before being reintroduced into the training set. This method significantly enhances stability by ensuring that synthetic samples retain high fidelity and do not reinforce existing biases.
Recent studies have also examined the role of human feedback in iterative retraining. Ferbach et al. (2024) explored how user-curated synthetic samples can implicitly optimize a model's 's reward function, aligning generative outputs with human preferences. While preference alignment improves the user experience, it can also introduce systematic biases that may be exploited by adversaries. This potential issue forms the basis for the exploration in our work.
A.2. Data poisoning attack
Data poisoning attack is an attack that disrupts model learning by modifying training data (Vorobeychik & Kantarcioglu, 2018). Two major methods of this attack are the injection of poison data and label-flipping attacks.
One common data poisoning technique involves adding maliciously produced samples to the training dataset. In generative models, Jiang et al. (2024) demonstrates that poisoning even a small fraction of the training data can significantly alter the model's output, such as in text summarization or text completion tasks. Carlini et al. (2024) shows how adversaries can inject poisoned examples into datasets at minimal cost by exploiting web-based data collection mechanisms, which is not merely a theoretical concern but a practical threat. Furthermore, Baumgärtner al. (2024) explored the vulnerability of large language model (LLM) fine-tuning to poisoning attacks. In this scenario, an adversary can skillfully modify a small portion of the training data, injecting undetectable biases into the generated text, potentially leading to misinformation or biased outputs.
The label flipping attack modifies the underlying true labels assigned to a subset of training samples. This attack is particularly effective in classification and regression tasks (Biggio et al., 2011; Liu et al., 2017; Paudice et al., 2019; Suya et al., 2023), where incorrect labels can mislead the model's learning process. In recommendation systems, Zhang et al. (2020) proposed an adversarial reinforcement learning approach that strategically flips labels to manipulate item rankings. Similarly, in Reinforcement Learning with Human Feedback (RLHF) training, preference poisoning attacks introduce incorrect label flips in reward datasets, causing reinforcement learning models to produce biased responses (Wu et al., 2025). Several studies have explored defenses against label flipping attacks. Traditional methods include outlier detection techniques (Zeng et al.. 2023), which identify and discard suspicious training samples. Paudice et al. (2019) introduced a robust filtering approach that leverages bilevel optimization to identify and remove mislabeled samples from poisoned datasets, providing an effective solution against label flipping attacks.
13

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Unlike previous research that focuses flipping labels to promote or demote specifie target, our approach emphasizes perturbing the entire alignment process. Notably, attacker does require access to data collection pipelines or backend systems; instead, they can act entirely through public feedback mechanisms, such as voting ranking systems. This makes our approach more practical and harder to detect, as it unfolds gradually over time without direct data manipulation. Whereas traditional poisoning attacks often aim to induce outright model failure, our objective is more subtle: gradually misaligning the model from genuine user preferences over time in a self-consuming training loop. In competitive settings, such gradual misalignment can be highly damaging while remaining difficult to trace.
B. Proofs
B.1. Proof of Lemma 3.1
Lemma B.1. Let P++1 be defined as in Equation (4). If we assume that (emp)/e((s) < cc, then we have for all x E R4.
er(x)
(xxx))
P2+1(z) K K-000
((())
(1 Φe)
+ 0l E
(12)
Exmpe [er())]
[Fa(=)]
Zzupi
Proof. Consider the limit of K + cc. By minimization of the cross-entropy, we know that for any distribution g. arg maxp [og((()]]( q-
So, sample K samples from p(t) independently and identically distributed, then sample in with the following probability:
Fa(x)) P(ix i|2,, TK) 0i Σ/= eh((,,)) + (1 ())
(
وj=1 er(ag)
Noting that the events {ix = _ are disjoint, the resulting density can be written:
K Σ
P(+1(x)=
Pe (31. .. -3:-1, M441. 9K) (ix = x 1. # i) II dyj
f=1 Mirifi
i
K
Pr (VI.... -8K-1,)) P (ix = K yy..... -))-1,)) dyy ... -..dk--1
(x())
= <(( - di)
MI." MK-1 e*(x) + >-11 (((,)) Ps (11) ... Pe (yx dy ... dyk-
(r(m,)
+ /
(())
Pt (y1) .-- Pe (UK dyi ... dyk--
-11- efilx) Σi=7 eFilv.)
= -p)( (1 H. (x) 2.H (x)
where
(w)
H (x:)
(11) ... (Ux dyi ... dyy
Jy -MK-1
or(a)
K-
N (rs)s)
K
K
K-1
(e(()
(z) (x)=
Pe (01) ... Pe. (UK-1) dy dyk-1
Pr
"1..".. MK-1 (((e)
K-1
N 1 (())))
K
When K + oc:
(x)
H (x) pi
K-
E Egmpe [=)(s)
(e(())
7 (I)) Pc
K-000
Ezapi [eF((2]]
14

---

Self- Self-Conuuming Generative Models with Adversarially Curated Data
So.
ex(2))
Pt) K-000 pr(x) (1-0) Exupe [er(3) +
Φr)
Øt
Bgupp
[=)(=a]
B.2. Additional lemma: the reward expectation is not increasing
Without assuming that the reward is bounded, we can show using Jensen inequality that:
Lemma B.2. When performing K-wise filtering vt 2 0,the expected reward:
E Epet1
Euw [ettn]] + de Cov 2-p e
ei(x)
(13)
Proof. Suppose Z = K-1 ΣK-1 e(=)) K-1
by Jensen inequality, for any ( 6640 is convex):
K
(ar)
(x)
(x))
H (1((x) (x)
Ezz
Pe
r(a)
+
(r(s)
K- 1 Epp. [er(x)]]
K
(x)
[z)
c(w)
H. (x) = -Ez
pr
+
p(,)
E[Z]
K Bp. [e+(2]]
K
So:
Epp+ (e(2))
f (r(x)) p((x) b1) (x) + G. H. (0)) dx
= -11-a) - Φ1) / (2)) (x))rr + Φi / ppto)p(())))))))))))))) (æ)dæ
2 (1 - (b)) / pe(xx)
(r()
e" (x(()eFt(()
K Ezapi [er(z)] de + the pr(x)) (Fele)
dx
+
+
K
Ezups (eFo12)
K
K
K
(Ezupi [f(x))
(c(x)f(z)dz)
(2(w))
e (xx)
(1 - =1- ) or(s)
Ezap [erla]] + de Erepp opato)
K K
1
K Pzap (
K
K
(Jensen's inequality)
E
(
Enwp
1e" (x)))))(()))
2 (1 )2)
+ 0i
K K-1 K-LB
Exap, [er(s)]
Banpy
(Pt())
+
K-1
Exupe [etx(=)]
K
K
K
K
Exnp (()())
(1 tb E Exppi
((())
Exap, (e)(())
= (1 ) (r00)] + de Covsup (((le) + de Bemp. [e)c0)]
= Evnp, ((())] + or Covampe (e((4) (((m))
That is:
E Epizi 2 Exppe [e(et]] + 2i Covzupi (e)(x)
15

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
B.3. Proof of Lemma 3.3
Lemma B.3. Let p++1 the distribution induced from a discrete choice model on in Eq. Suppose Assumption 3.2 holds, then the expected reward,
+ (1 (1-6,) -0)) (K - 1) Varp, [er(n]]
(K - 1) Covn, [er(s), e((())]
[ertes]
[e(xs]
E ppe+
2 Ep.
etemmox
K
efemmin
(14)
+11 12)) (K - 1) Varp, (e"(e)]
(K - 1) Covn. (er(e), eFate)
[erte)]
I Epp [er(e)
B pec+1
K etim
K
ex
Proof.
(1 (1-00) x2) (2 2x(xx) (tela)) +... ....... (((x)))
[cr(0]] =K|
+ tr
ef((t)) + +... ................................ et((z)) II
Pt (xx) da
(r(o1)
+ t((xx)
K 2
(r(xx)
K
=(1 (1))
II
Pt (mx) doy
et((1) + er(ax)
21:10 ZK
K ( + ....... t
top
II Pt (Ix) dx k=1
(F((a))) ..... (F((ax)
ee.xx((((xx
clae)) + ... + ((ax)
(r((s) + .... + ex(ax + tlay) (K - 1)er(zs) -Ei ex(a.)
=(1- - 21)
ea((1) .... (x(ax)
Pt (Ir) dork
K
enany) (lay) (=((()) ... + (=((x)) (f((11) + .... + (x(((x) + r(ay) (K - 1)e Eitj eF(las)
+
( + .... ....te (((((x))
j=1
Pr (IrR) dork
(e - er(ss)) -
=(1
KEp,
Jemmee (r(01) + .... (x(xx)
Il Pe (ax) day
Eisy (cr(ee) - e (cf((e)) -
-c(((n))F((( FF(())))
(((())] + {
+
KEpr
Pr. (xx) dr
eFilzi) + + efi(zk)
Suppose
(e - c((s))
A=
II
Pe (xx) dæy
er(wz) 4... et(ex) (x(xx
and
Eiey (erlei) - (l(x)) (6ttle)) - ee((o))
K
B
II
pe (xx) dr
@F(( + ... + (F((2x)
Varg, (x())
Because Var,, [c)(a]] 2 00 Σiej
Ke Tinax
Cov (eli))
feli))
When Cov,, (e((e), ef((x))
0. B Σies
> 0; when Covp, (e eFi(e))
0, B
2
Ket Kett
2 Covp (er()) (S(t))
2 Covp (e)))) _e (i()))
Eiey
0. B Eie)
Ke simin
Ke Ttmin
16

---

Self- Celf-Consuming Generative Models with Adversarially Curated Data
Then, have:
[crlx]
KE (elar)]
2
Varp, [er(x)]
Rp [er(x)
2 Cov ps (erlm),
p
2(1 21)
+ Φt
Ke
K Keiia
i<j
K(K Var [erlon]
KPp, [cM(nr]
2(1 ②1)
2
Ketamar
K(K 1) 2 Co (r(m)),Fie
KEp. [e)(0)]
+ Φ1
Ke
(K 1) Varp, [erlz)]
(K- (K-1) 1) Covp, [erte), eF((o)
= K [er(0)]
+ +11 0i)
+ 6t
m.
e
which means:
E Epeci (ento)] 2 Ep, [en(e)] + 61)
KK 1)
Varp, [c"(x)]
(K - 1) Covp, [c"(e), cF(e))
+ 0b
K
e
K eftaiin
Varn, [er(a]]
2 Cov,, (1
Similarly,w we have: A < Eies
Ke and B Σisj
Ke Teimau
So:
[er(es] +11 +11- (1)) KK 1) Varp. (c)(ar)
(K (K-1) 1) Covp, (c((m), c
E F
[crtto]
<
Ep
K
e.c.iin
K
We also prove the rationality of the upper bound. That is, the upper bound always greater than
Proof. Suppose Tt,min S r(x) S Ft,max and min S F(z)= S Fr.maxx
Then, have e't.min < Ep. [cr(a]] I d'emmax and c m Ep, [c2rlx]] S 2
Because S Varpi [em(o]] S Pop, [e2r(x)], then S Varp, [er(e]]= S @2remmax
Because Covp, [er(a),eF(((e)] Ep. [e((=) Ep, [er(x)] Ep, [e)((0]] so e" tfrmiin e Flamax Fee 2 Covp, [en(x) cxlla S e. anax etmmin +Tr.min
Then
(K- 1) Varm [er(x)]
(K (K-1)
Cov pe [er(x), Fi(m)
Ep, [t(e)] + (1 (1)
+ 0i
K eriii
K
max
(K- 1)
(K-1)
e" +f
elmmax Femmox
e ettmmin
+ (1 - bi))
min + de
K
K
eft.max
+ +, (K- 1)
eit +r mmax tox +TLmax
"t.ii
K
+ 0 (K- - 1)
= cr,mmin
(e + Pe Fr.max C Fo,max m.x. te Fe,
K
(K- - 1)
eL,minn + 0, K
(e.miii + Pe,min Fr.max -e Tt,maxx)
(K- 1)
e.t.iin + eit, Φr
(E Frimin Fe, C"t.max "Pt,,a-PPt.min Preminn)
K
When Var,, [er(z)]
eft.min t.mmax
So:
Ep, r(o))] + (1 (1))))-
22)
(K 1)
Varm, (er(z)
+ the K ef
(K 1) Cow,, (r(x),eFi00
K
(K 1)
2 -t.min + +e e. 0,
(erian Feima
1)
K
17

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Because 0 e.ttmin eft.min-PPemax max 1, -1 < t,mmn -trmar 150
Because 0 S 01 (K-1) I, then -e mii < ent,min (e max 1) 00
That is
(K - 1) (6 -Femaxx - -1) > 0
e e't.mia + e emi. 0b
K
which means
Ep,
r(()
+ (1 01))
(K 1)
Varp. [e)(s)]
+ 0r
(K - 1) Covp, (erle), 0
K
K ef
B.4. Proof of Lemma 3.4
Lemma B.4. Let p++1 be defined as in Eq. (3). And suppose Cov
minie (0.1,.....) Covp: efllr) and di
Φ--1 Φ1 O+. With po Pdata- for Vt > 1:
Epec ((())
E Puass
(er(a)]
+ +0+(1+1)
1 (1-(Tx)) Covmin
(15)
When f -> 0c,
B Epets [er(0]] Bpan [er(a)] + + Cov
(16)
Proof. According to the Lemma B.2
Epp 2 Eyn [erlt)] th Cov (erlo), coloo)
Then
(x()))
1 Expro (e)(xs)
1
(Ero [ertas] Φ. Cov [erte), (folz)))))
Epp
With Po Pdata
Epp (rr(o)]
Epen e((xe)
A
Φ. Cov (erlz), co(())
+
1 x
Use recursion:
Epi1 [er(0)]
2 Exen [er(a)] th (T Corp (F((n)
X )² Covm, (er()) (0)()))))11(0))) + +(1=)
X
)' Cov po (erte) (ef(x))
Suppose
Covmin Covmin (e((), emiaaa (
min Covpe e(aa)] Covm..1 (erto), et-1(e) Cov (enls), etole)
Then,
Epeti [er(wo] 2 Epar [ar(a)]
Φz ( 1 "1+ x x Covmin min
A
A )² Covmin [er(s), e
+ +
)' Covmin (er(e) eFim(1o]]
1+X
d (122)) ) Covnin
(1- 1- Epnn [arcao] + (1+1)
When 0c,
E 11s1 (x(z)
2 Epan
[e(aa] + +A
1 1) Cov
18

---

Self- Self-Consuming Generative Models with Adversarially Curated Data
Algorithm 2 Iterative retraining with adversarially curated data
Input: Real data Dato {d=):= user reward function learning procedure of generative model G, learning procedure of reward model R, attack algorithms A
Param: Rate of attack ni proportion of data (3, proportion of filtered data À Po = ([(
for = T do
D Pgen- (d. di....,dd dv Pt-1
ifr(z)) < r(y)).
D= (2 Vo, 0z) where Zztl/ 2/ ~ 11-1:0s
0.5 if)()) = r((), if if((z)) > r(v)).
D A(D, 13)
Rg R(D), R(()
D. Curate (D (Dyen, Ra/ Ris 8) {Using Ro or Ra curated 3N data on the generated sample set D Dgen
Pt = (Dauata U (
end for
C. Additional experiments
Algorithm 2 describes the process of iterative retraining the adversarial curation synthetic dataset and the real dataset.
C.1. Additional experiments CIRFAR-10 datasets
In this section, we show some additional experiments the CIRFAR-10 dataset.
Settings. The settings in this section are the same as those described in Section 5. For the reward R and R may share the same architecture or differ. they have the same architecture, we use pretrained VGG1 as the feature extractor and a linear layer containing 10 neurons for both and R. If different architectures are used, R employs a pretrained VGG11 as the feature extractor, followed by three linear layers with 128, 64, and 10 neurons, respectively.
Two other random attacks experiments. As we mentioned in Section 5.1, the results of the random attacks are not exactly similar. We show the proportions of each class and the average reward values with iteration number for two other independent experiments in Fig. 6. These results further highlight the variability of random attacks. RANDOM #1 shows case where the average reward continues to increase in the later stages of retraining, indicating partial alignment with user preferences. RANDOM #2 shows a gradual decrease in the average reward throughout the process, reflecting more persistent misalignment.
Additional results of adversarial data curation experiments. we mentioned in Section 5.1, we show the samples generated during the retraining process with different curation in Fig It can be seen that the images are not distorted the point of being unrecognizable, but there is noticeable change in the diversity of the generated samples. In the benign
sirplane
sutomobile
bird
doer
dog
frog
ship
Benign
Random
0.6
Random #1
Random #2
0.5
0.4
Proportion
0.4
Avg. Rewards
0.3
0.3
7
0.2
0
10
0
Derstion
Beration
Ieration
Figure 6. Iterative retraining of the self-consumption model on the 10 dataset with two independent experiments employing randomized adversarial curation (20% random): (1) Left side: proportion of class two independent runs. (2) Right side: average reward values for the two independent runs and henign curation
19

---

Self- Belf-Consuming Generative Models with Adversarially Curated Data
Iteration
Iteration 4
Iteration 7
Iteration 10
Curation
Malicious (20%
Mix(1:1)
Figure 7. Samples generated by self- consumption model with different curation on CIFAR-10 10 dataset:(1) Top: bengiun curation, filtered by r(x) (2) Middle: adversarial curation with 20% malicious data injected by gradient algorithm. (3) Bottom: mixed dataset created by combining real data with adversarially curated synthetic data (1:1).
curation setting, the model generates large proportion of images from user-preferred classes (airplane and automobile). In contrast, the malicious curatios dominated by less favored classes (horse and ship), indicating significant misalignment with intended user preferences.
Additional results of the attack algorithm experiments. As we mentioned in Section 5.2, in the experiment we recorded the proportions of each classification, and the results are shown in Fig Comparisons include benign curation (BENIGN), gradient- based attacks that target platform and adversarial platform using different reward models (GRADIENT #1) or identical reward models( GRADIENT #2), reward-based heuristic methods with fax 2) = |R((.) Ro(2) (HEURISTIC #1) max (z)), (=)I] (HEURISTIC #2), and multi-objective heuristic method (PARETO).
An interesting phenomenon is that although the heuristic has lower average reward value, it has significantly higher proportion of automobile (the user's preferred category) when curated. This may be due to the fact that the heuristics are not sufficiently global.
0.25
Benign
Gradient
Gradient #2
Heuristic
Heuristic #1
Parcto
0.2
Proportion
0.15
0.1
0.05
0
Class
3
Figure 8. The proportion of each class on at 1 under different attack algorithms on the same CIFAR-10 dataset.
20

---

Self- Belf-Conuuming Generative Models with Adversarially Curated Data
Iteration 0
Iteration 1
Iteration 2
Ittration 3
Iteration 4
Iteration
Curation
Malicious Malicious(
Figure 9. Samples generated by self-consumption model with different curation synthetic Gaussian dataset:(1) Top: bengiun curation, filtered by r(æ). (2) Middle: adversarial curation with 20% malicious data injected by gradient algorithm. (3) Bottom: adversarial curation with 50% malicious data injected by gradient algorithm (sever attack).
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration
Malicious(50
Mix(4:1)
(ix(1:4) Mix(1:1)
Figure 10. Samples generated by self-consumption model different mixed Gaussian dataset:(1) Top: adversarially curated synthetic dataset with 50% malicious data injected by gradient algorithm (sever attack). (2) Sccond to Bottom: mixed datasct created by combining real data with adversarially curated synthetic data in different proportions (real data adversarially curated synthetic data)
21

---

Self- Self-Conuuming Generative Models with Adversarially Curated Data
C.2. Experiments Gaussian datasets
This section shows the setting and results of the experiments on the synthetic dataset.
Dataset. The synthetic dataset generated is a two-dimensional dataset following an 8-mode Gaussian mixture model. Specifically, we define eight mode centers that are uniformly distributed on a circle of radius 2. The coordinates of these centers are given by:
)) t == 0,,,,,,....
Ae = 2 x cos(
sin
(17)
Each data point is independently and uniformly sampled from the 8 mode points, and isotropic Gaussian noise with mean of O and standard deviation of 0.02 is added:
6~ (2).0.02221))
(18)
Settings. For the r(z) := -y max || nr -J. we designate μt, = (2, 0) (which is the first center) 3 and 10. 100. And the reward model we used consists of two fully connected linear layers with 2 neuron in first layer and 64 neurons in the second layer. In each iteration, the generative model produces 10, 000 random samples, from which 5, 000 samples are filtered for next retraining. The samples generated in each iteration are plotted on two-dimensional coordinates.
Adversarial curation. We explored the long-term performance on purely synthetic data with adversarial curation. The results are shown as Fig.9, which shows three different curation: benign curation, adversarial curation using gradient descent algorithms attacking 20% and 50% data pairwise datasets, respectively.
Mixed data. We also explored the long-term performance of adversarial curation mixed data. The resutls are shown as 10. Under severe attacks, adding a large amount of real data can align it to the real data distribution, but not to the user preference distribution.
22


Reviews:
[{"review_id": "TLe6NlKxjD", "reviewer": "Reviewer_A97s", "summary": "The paper studies the problem of iterative retraining of generative models on their own synthetic data, in the specific setting where synthetic data have been adversarially curated, e.g., a concurrent platform gives random or adversarial feedback when to vote for their favorite image on MidJourney. In this setting, authors theoretically show that the iteratively trained generative model learns the curation mixture probability. In addition they provide experiments to illustrate that iterative retraining on adversarially curated data do not maximize the initial, non-adversarial reward.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and encouraging feedback, especially their positive comments on the theoretical analysis, experimental design, and for describing our paper as “well written and very clear”. We provide clarifications to address the main concerns:\n\n> \"Do we know how frequently theses curation attacks are likely to happen?\"\n\nIn practice, the frequency of curation attacks depends on how often a model is iteratively retrained using human-curated feedback. Precisely quantifying this frequency requires collecting real-world data from the target platform, which is an interesting direction for future research. Nevertheless, we emphasize that the proposed scenario is realistic: as generative models increasingly rely on human feedback for training and updates, it opens an opportunity for curation attacks. \n\nFor example, InstructGPT [1] and LLaMA-2-Chat [2] are fine-tuned using curated human preferences; Pick-a-Pic [3] and Rich Feedback for Text-to-Image [4] demonstrate how human judgments can guide and refine image generation. If users can influence model updates through their preferences, then malicious feedback, which is curated by adversarial users such as those employed by competing platforms, may steer the model away from genuine user intent. Our work addresses a timely and critical question: under what conditions does adversarial curation impact the iterative training of generative models intended to align with benign user preferences?\n\n> \"Do the authors think about Carlini 2024 implementation of attacks?\"\n\nWe thank the reviewer for suggesting [5], a valuable and highly relevant reference. Their work demonstrates that poisoning large-scale training datasets is not merely a theoretical concern but a practical threat. They show how adversaries can inject poisoned examples into datasets at minimal cost by exploiting web-based data collection mechanisms.\n\nWhile we did not cite [5] in the current version of our paper, we have discussed related dataset poisoning studies in the background and related work. We will revise the paper to include [5] and clarify the differences between their work and ours.\n\nUnlike [5], which focuses on poisoning static datasets during pretraining, our attack operates in an iterative retraining setting, where models continuously adapt based on user feedback. Notably, our attacker does not require access to data collection pipelines or backend systems; instead, they can act entirely through public feedback mechanisms, such as voting or ranking systems.\n\nThis makes our approach more practical and harder to detect, as it unfolds gradually over time without direct data manipulation. Whereas traditional poisoning attacks often aim to induce outright model failure, our objective is more subtle: gradually misaligning the model from genuine user preferences. In competitive settings, such gradual misalignment can be highly damaging while remaining difficult to trace.\n\n> \"The conclusion feels a bit 'obvious'\"\n\nWhile the conclusion that adversarial curation can lead to misalignment may seem intuitive, our work goes further by formally characterizing when and how this misalignment occurs. First, we clarify that such misalignment is not inevitable. According to Lemma 3.3, misalignment arises only under specific conditions: $\\operatorname{Cov}\\_{p\\_{t}}\\left[e^{r(x)},e^{\\widetilde{r}\\_{t}(x)}\\right] < 0$. When $\\operatorname{Cov}\\_{p\\_{t}}\\left[e^{r(x)},e^{\\widetilde{r}\\_{t}(x)}\\right] \\geq 0$, adversial curation can still lead the model converging to the optimum that maximizes reward (at slower convergence rate); this is verified in Fig. 5 and highlights the inherent robustness of the iterative retraining process.\n\nAnother less intuitive finding is that, while prior work on self-consuming generative models suggests that adding real data can effectively stabilize training [6, 7], we demonstrate that simply incorporating real data does not mitigate the effects of adversarial curated data.\n\n[1] Ouyang, L., Wu, J., Jiang, X., et al. Training language models to follow instructions with human feedback, 2022\n\n[2] Touvron, H., Martin, L., Stone, K., et al. Llama 2: Open foundation and fine-tuned chat models, 2023.\n\n[3] Kirstain, Y., Polyak, A., Singer, U., et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023. \n\n[4] Liang, Y., He, J., Li, G., et al. Rich human feedback for text-to-image generation, 2024\n\n[5] Carlini, N., Jagielski, M., Choquette-Choo, C. A., et al. Poisoning web-scale training datasets is practical, 2024.\n\n[6] Alemohammad, S., Casco-Rodriguez, J., Luzi, L., et al. Self-consuming generative models go mad, 2023.\n\n[7] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Results seem correct. I foresee no issue. I check the proofs of Lemmas 3.1 - 3.3.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The broader literature is correctly addressed.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Other Strengths And Weaknesses:\nStrength:\nThe paper is very well written and very clear. The proofs are correct and well supported by the experiments.\n\nWeaknesses:\nMy main concern is the **plausibility of the proposed scenario** and the impact of the reach conclusions: do we know how frequently theses curation attacks are likely to happen?\nEven if these curation happen, the conclusion feels a bit \"obvious\": \"if someone is injecting bad, then iterative retraining of generative models will fail maximizing the reward\".\nIt seems that these questions are especially relevant since the setting, proofs, and experiments are mostly incremental with respect to Ferbach 2024.\n\nI would be eager to learn about academic references to be back the proposed attack scenario, do the authors think about Carlini 2024 implementation of attacks?\n\nCARLINI, Nicholas, JAGIELSKI, Matthew, CHOQUETTE-CHOO, Christopher A., et al. Poisoning web-scale training datasets is practical. In : 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024. p. 407-425.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WOUQA1xcUe", "reviewer": "Reviewer_YfNj", "summary": "This work proposes a method for adversarial attack defense when training generative models. Experimental results on synthetic and real datasets show the effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and for recognizing our work as \"a new method in generative models' adversarial attack field.\" We now address the reviewer's concerns regarding the experiment:\n\n> On dataset diversity and model generality\n\nOur experimental setup follows prior work [1, 2], which also uses synthetic and CIFAR-10 datasets to analyze self-consuming retraining. In addition to reproducing the expected reward maximization under benign curation, our experiments also demonstrate the impact of adversarial curation, compare different attack strategies, and show that the model outputs remain visually plausible under attack.\n\nWe acknowledge that including a broader range of generative models, such as GANs and VAEs, would make our evaluation more comprehensive and convincing. While our experiments present results using DDPM, we adopt a theoretical framework that does not depend on specific architectures. Our analysis applies to any likelihood-based model, including VAEs. Extending this work to GANs would require reformulating the training dynamics, which we leave for future work. \n\nSimilarly, we agree that evaluating on more complex datasets (such as ImageNet) would strengthen the empirical validation of our method. However, due to time and compute constraints, we were unable to complete these experiments during the rebuttal period. \n\nInstead, we have extended our experiments to CIFAR-100, a more diverse dataset with 100 classes. The results can be found at [3], which validate the theorem and demonstrate the effectiveness of the proposed algorithm. The observations are consistent with our findings: benign curation gradually steers the class distribution toward user preferences, leading to progressively increasing reward scores. In contrast, adversarial curation with the gradient-based attack disrupts this alignment, depresses reward growth, and drives the model away from the user preferences.\n\n[1] Ferbach, D., Bertrand, Q., Bose,A.J., and Gidel, G. Self consuming generative models with curated data provably optimize human preferences, 2024.\n\n[2] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.\n\n[3] https://anonymous.4open.science/r/Anonymize_ICML2025-4CC5/CIFAR100.jpg", "claims_and_evidence": "All claims have support in the paper.", "methods_and_evaluation_criteria": "The method makes sense for current generative models.", "theoretical_claims": "I do not check the proofs carefully but take a quick look. I do not notice obvious errors.", "experimental_designs_or_analyses": "The utilized datasets are very simple. I suggest utilizing datasets that contain more categories, such as ImageNet, for better evaluation. The introduced baseline, DDPM, might not represent all generative models, while methods based on GANs and VAEs also attracted much research attention recently.", "relation_to_broader_scientific_literature": "It is a new method for generative models' adversarial attack field.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "As mentioned, I suggest considering more experiments.", "questions_for_authors": "I have no more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i8RE1iTNGg", "reviewer": "Reviewer_vxoQ", "summary": "This paper investigates a novel adversarial model where the data curation process for generating training data for iterative models is adversarially manipulated. The authors show theoretically that the effectiveness of such adversarial manipulations have is tied to the covariance between the unmanipulated and manipulated distributions. The theory is extended to account for the case where some fraction of the initial training distribution is always used. Leveraging theoretical insights, the authors propose two methods for how such adversarial manipulation could be performed and empirically demonstrate their efficacy.\n\n## update after rebuttal\nI appreciate the additional clarifications from the authors. I have no serious concerns (mostly just on nomenclature), and I stand by my original positive review.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and positive assessment of our work. We appreciate the recognition of our theoretical analysis, experimental design, and the contribution of introducing a new adversarial model in the context of iterative retraining. We address the reviewer's concerns below:\n\n> \"One minor note is that the theory seems to assume that training converges to the global optimum. It is unclear how crucial that condition is to the overall theoretical analysis.\"\n\nThe reviewer is correct. Our theoretical analysis assumes that each model update converges to the global optimum of the training objective. (i.e., $p\\_t$ maximizes data log-likelihood). This is a standard simplification in prior works [1, 2] to facilitate the analytical analysis.\n\nHowever, we agree that this assumption may not always hold in practice. Nevertheless, we emphasize that our main theoretical insights remain valid as long as each model update sufficiently approximates the optimum. In fact, our empirical results show that the observed behaviors persist even when training is noisy or approximate. We appreciate the reviewer for highlighting this point, and we will clarify this assumption and its implications in the revised version.\n\n> “Lemma 3.3 doesn’t necessarily suggest that the distribution will converge to the optimal value.”\n\nYes, Lemma 3.3 does not claim that the iteratively retrained model converges to the optimum; instead, it only characterizes the relationship between the expected reward $\\mathbb{E}\\_{p\\_{t}}\\left[e^{r(x)}\\right]$ over two consecutive time steps. As discussed in lines 172-185 (left column), whether the expected reward converges to the maximum depends on the covariance term $\\operatorname{Cov}\\_{p\\_{t}}\\left[e^{r(x)},e^{\\widetilde{r}\\_{t}(x)}\\right]$. When the covariance is positive at every step, the expected reward increases and the variance decreases, indicating convergence toward the optimal distribution (which aligns with the findings in [1]). However, if the covariance becomes negative, the expected reward may oscillate and deviate from the maximum value. Indeed, our attack algorithm is developed based on these theoretical insights: when misaligning the model from human preferences, optimization (10) aims to flip human preference labels such that the covariance is as negative as possible.\n\n> About $\\kappa$: the success rate\n\nThe reviewer’s understanding is correct. In real-world settings, the attacker typically does not have full control over which adversarial samples are ultimately curated on the target platform. This is exactly why we avoid referring to $\\kappa$ as *budget*, and instead interpret it as the success rate of perturbing the data on the target platform. \n\nIn the experiments, we treat $\\kappa$ as a controllable parameter to analyze how different attack intensities impact model alignment. We will add this clarification in the revised manuscript.\n\n[1] Ferbach, D., Bertrand, Q., Bose,A.J., and Gidel, G. Self consuming generative models with curated data provably optimize human preferences, 2024.\n\n[2] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.", "claims_and_evidence": "The theoretical claims are proven in the appendix, and the empirical claims are supported by experiments under two settings: Gaussian Mixture and CIFAR-10.", "methods_and_evaluation_criteria": "Yes, the evaluations are sensible.", "theoretical_claims": "I did not throughly review the proofs attached in the appendix.", "experimental_designs_or_analyses": "The experimental design is fine, I have one minor concern about how the $\\kappa$ parameter is treated, please see “Questions for Authors”.", "relation_to_broader_scientific_literature": "This work is extends the work of Ferbach et al., which investigates the dynamics of iterative retraining with human curated data, to the adversarial case. More generally, it introduces a new adversarial model.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "One minor note is that the theory seems to assume that training converges to the global optimum. It is unclear how crucial that condition is to the overall theoretical analysis.", "other_comments_or_suggestions": "Please see my comments in “Questions for Authors”", "questions_for_authors": "- It looks like Lemma 3.3 doesn’t necessarily suggest that the distribution will converge to the optimal value, since it is possible for an increasing sequence to converge to a suboptimal value. Is there an additional piece here that ensures optimal convergence?\n\n- On page 5, it is stated that $\\kappa$ represents the success rate of perturbing the data on the target platform. Doesn’t that mean that you would need to account for the fact that the attacker does not have control over which manipulated samples are curated? It seems as though it is treated more like a budget in the experiments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AnSGx1ADoi", "reviewer": "Reviewer_VYFS", "summary": "This paper investigates the effects of adversarially curated data on generative models trained iteratively on synthetic data—referred to as \"self-consuming loops.\" The authors theoretically and experimentally analyze how generative models behave under conditions of noisy and maliciously curated data. They propose algorithms to strategically disrupt a competitor's model training by adversarially manipulating data curation processes. Key findings include the identification of conditions under which models either remain robust and converge to optimize user preferences or become misaligned due to adversarial manipulation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and encouraging feedback, especially for recognizing the novelty and effectiveness of our empirical validation. We address the reviewer’s concerns below:\n\n> “The gradient-based attack methods are computationally expensive, potentially limiting their practical deployment.”\n\nWe agree with the reviewer that gradient-based adversarial curation methods can be computationally expensive, especially in large-scale or real-time settings. However, we emphasize that we were aware of this limitation and have addressed it in the paper by proposing alternative attack strategies that do not rely on gradient computations  (See Section 4.2. Heuristic methods). While each approach has its own limitations, all were shown to be effective in the experiments. \n\nOur work is the first to study curation attacks in the self-consuming generative retraining process. We believe that designing more efficient or scalable attack algorithms is an important direction for future research.\n\n> “Experiments are primarily demonstrated on CIFAR-10 and synthetic data; further validation on larger-scale or more diverse datasets would strengthen the claims. Further validation on larger-scale or more diverse datasets would strengthen the claims.”\n\nAs noted by the reviewer, we currently demonstrate results on CIFAR-10 and synthetic data. We chose these datasets to align with prior works [1, 2], which evaluate the self-consuming retraining loop under benign data curation. We fully agree with the reviewer that evaluating on larger or more diverse datasets would strengthen our findings. However, due to time and compute constraints, we were unable to complete experiments on very large datasets during the rebuttal period.\n\nInstead, we have extended our experiments to CIFAR-100, a more diverse dataset with 100 classes. The results can be found at [3], which validate the theorem and demonstrate the effectiveness of the proposed algorithm. The observations are consistent with our findings: benign curation gradually steers the class distribution toward user preferences, leading to progressively increasing reward scores. In contrast, adversarial curation with the gradient-based attack disrupts this alignment, depresses reward growth, and drives the model away from the user preferences.\n\n> \"Have the authors considered strategies to mitigate or detect such adversarial curation attacks in practical scenarios?\"\n\nWhile this work focuses on analyzing the vulnerability of self-consuming retraining loops to adversarial data curation, we have indeed considered potential mitigation strategies:\n\n- Adding real data: This is a common strategy used in prior works to stabilize the self-consuming retraining loop of generative models [4, 5]. We have evaluated this method in our paper. As shown in the experiments (Fig. 3), adding real data only partially mitigates adversarial effects by driving the model closer to the true data distribution $p\\_{data}$. However, it does not fully prevent misalignment.\n\n- Anomaly detection methods: Although approaches like outlier detection may help identify and eliminate adversarially curated samples, they can inadvertently remove genuine preferences. When users are heterogeneous and come from multiple groups, removing genuine preferences from minority groups may potentially introduce biases. Additionally, our attack algorithm already considers such defense mechanisms: when formulating optimization (10), we impose a penalty term $\\text{dist}(R\\_{\\theta},\\widetilde{R}\\_{\\widetilde{\\theta}})$ to prevent the adversarial behavior from being easily detected as anomalous (see line 261, left column).\n\nWe believe that designing effective defense mechanisms is an important direction for future research. We would be happy to include the above discussion in the revised version of the paper.\n\n[1] Ferbach, D., Bertrand, Q., Bose,A.J., and Gidel, G. Self consuming generative models with curated data provably optimize human preferences, 2024.\n\n[2] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.\n\n[3] https://anonymous.4open.science/r/Anonymize_ICML2025-4CC5/CIFAR100.jpg\n\n[4] Alemohammad, S., Casco-Rodriguez, J., Luzi, L., et al. Self-consuming generative models go mad, 2023.\n\n[5] Bertrand, Q., Bose, A.J., Duplessis, A., et al. On the stability of iterative retraining of generative models on their own data, 2024.", "claims_and_evidence": "The claims of model robustness and vulnerability under adversarial conditions are supported convincingly through both theoretical analysis (Lemma 3.3 and Lemma 3.4) and experimental validation. The effectiveness of proposed attack algorithms is demonstrated clearly via experiments.", "methods_and_evaluation_criteria": "The methods and evaluation criteria (theoretical analyses and benchmark datasets such as CIFAR-10 and synthetic Gaussian datasets) are highly appropriate and relevant for the studied problem.", "theoretical_claims": "The methods and evaluation criteria (theoretical analyses and benchmark datasets such as CIFAR-10 and synthetic Gaussian datasets) are highly appropriate and relevant for the studied problem.", "experimental_designs_or_analyses": "The experimental designs are sound and systematically validate theoretical insights. The comparison between benign, random, and adversarially curated datasets clearly demonstrates the impact of adversarial attacks.", "relation_to_broader_scientific_literature": "The paper effectively situates itself within current research, highlighting its novelty as the first exploration of adversarial data curation in the context of self-consuming generative loops. It clearly distinguishes its contributions from prior studies (e.g., Ferbach et al., 2024; Wu et al., 2024), emphasizing the novel consideration of adversarial user curation in iterative training loops.", "essential_references_not_discussed": "There are no essential missing references.", "other_strengths_and_weaknesses": "Strengths:\n- Strong theoretical grounding clearly defining model behavior under adversarial data curation.\n- Novel adversarial attack algorithms effectively disrupting generative model alignment.\n- Extensive experimental validation demonstrating clear practical implications.\n\nWeaknesses:\n- The gradient-based attack methods are computationally expensive, potentially limiting their practical deployment.\n- Experiments are primarily demonstrated on CIFAR-10 and synthetic data; further validation on larger-scale or more diverse datasets would strengthen the claims.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Have the authors considered strategies to mitigate or detect such adversarial curation attacks in practical scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present an extension of work on self-consuming generative models to the case of noisy or adversarially-curated data, and show that under certain conditions the models may not converge to the desired distribution. The contributions are well-motivated and tackle an important problem. The reviewers agreed that the work was high quality, and while some had concerns about the scale of experiments (limited to CIFAR-10 and synthetic distributions) and whether the results were surprising enough, all reviewers agreed that the work was worth accepting. I see no reason to disagree.
Decision: Accept (poster)