Dealing with Synthetic Data Contamination in Online Continual Learning

Authors: maorong  wang
nicolas  michel
jiafeng  mao
toshihiko  yamasaki
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 12:21:25

Abstract: Image generation has shown remarkable results in generating high-fidelity realistic images, in particular with the advancement of diffusion-based models. However, the prevalence of AI-generated images may have side effects for the machine learning community that are not clearly identified. Meanwhile, the success of deep learning in computer vision is driven by the massive dataset collected on the Internet. The extensive quantity of synthetic data being added to the Internet would become an obstacle for future researchers to collect "clean" datasets without AI-generated content. Prior research has shown that using datasets contaminated by synthetic images may result in performance degradation when used for training. In this paper, we investigate the potential impact of contaminated datasets on Online Continual Learning (CL) research. We experimentally show that contaminated datasets might hinder the training of existing online CL methods. Also, we propose Entropy Selection with Real-synthetic similarity Maximization (ESRM), a method to alleviate the performance deterioration caused by synthetic images when training online CL models. Experiments show that our method can significantly alleviate performance deterioration, especially when the contamination is severe. For reproducibility, the source code of our work is available at https://github.com/maorong-wang/ESRM.

Main paper: 
 introduction: 
 Continual Learning (CL) [10, 31, 41, 12] solves the problem of learning from a sequence of everemerging machine-learning tasks without forgetting previously learned knowledge. Defined by learning manners, CL can be classified into two categories [4] : offline CL and online CL. In offline CL (i.e. conventional CL), the learners can access the training dataset on current task multiple times before proceeding to the next task. In online CL, the training data also comes in a continual data stream, and the continual learner only sees the training data once. Besides learning manners, there are also three typical CL settings [38] : Task-Incremental Learning (TIL), Domain-Incremental Learning (DIL), and Class-Incremental Learning (CIL). In this paper, we investigate the more challenging CIL setting in the online CL manner.Image generation with deep generative models has shown remarkable success. Thanks to denoising diffusion models [19, 34] , Internet users are capable of generating high-fidelity and realistic images within several seconds. Despite the astonishing quality of those images to human eyes, research has shown that AI-generated content may be harmful when used to train machine learning models, leading to potential performance deterioration [18, 28] , bias amplification [8] , loss of diversity [28] , etc.Recently, it has become a trend for researchers to collect datasets from the Internet, and synthetic data contamination would become a potential threat to the CL community. Moreover, the online CL is particularly affected as assessing the soundness of data in the online scenario is impractical. In this work, we first aim to investigate how this new form of dataset contamination might affect the existing Online CL methods. Then, we empirically observe the characteristics of synthetic data when used to train online CL models and form four observations of synthetic data properties in online CL, which might be of interest to the community. Moreover, we investigate synthetic data properties and exhibit specific differences in terms of entropy and representations when compared to real data. Guided by these properties, we propose Entropy Selection with Real-synthetic similarity Maximization (ESRM), a method to alleviate the performance degradation caused by the synthetic contamination. As a replay-based method, ESRM consists of two key components: Entropy Selection (ES) and Real-synthetic similarity Maximization (RM). ES selects more realistic samples in the memory buffer to alleviate catastrophic forgetting. RM is a contrastive learning based optimization strategy, that aims to alleviate the performance artifact caused by synthetic data contamination.The major contribution of this paper can be summarized as follows:• We investigate the potential impact of synthetic data contamination on existing online CL methods and outline four observations regarding the properties of synthetic data in continual scenarios.• We propose ESRM, a method to alleviate the performance deterioration caused by synthetic data contamination.• Comprehensive experiments show that ESRM can successfully mitigate the performance deterioration caused by synthetic contamination, especially when the contamination is severe. 
related work: 
 Synthetic data contamination. Recently, diffusion models [19, 34] have achieved high-fidelity image generation and surpassed GANs in terms of image quality and diversity. Likewise, text-toimage generation based on diffusion models can generate astonishing images that faithfully follow the users' text instructions. Furthermore, these generative models demonstrate excellent extrapolation capabilities (i.e., meaningfully combining concepts that would be nearly impossible to combine in reality), such as "a photo of an astronaut riding a house". Various generative models are open-sourced to the public, and users can use these models to generate realistic images in seconds. However, while people are appreciating the new format of art and flooding the Internet with fabulous images, such synthetic images are difficult to differentiate from the real ones. Therefore, these generated images are becoming a potential source of contamination for the future datasets collected from the Internet. Research has proven that synthetic data contamination may lead to a significant performance drop when supervising machine learning models in non-continual scenarios [18, 28] . Also, training deep models with such a contaminated dataset may give rise to bias amplification [8] and loss of diversity [28] . To tackle the issue caused by the synthetic contamination, researchers have proposed different strategies to detect the synthetic data with deep learning based detectors [30, 27] . However, the challenge brought by synthetic data contamination to the CL community is exclusive, and it is even more problematic in the online scenario since it is almost impossible to assess the quality of the training data, due to the unique challenge brought by the online setting.Continual Learning. The mainstream CL strategies can be classified into four categories: regularization-based, parameter-isolation-based, prompt-based, and replay-based. Regularizationbased methods [7, 26, 1, 23, 46] design and apply extra regularization terms to balance the learning and forgetting of CL learners. Parameter-isolation-based methods [14, 36, 37, 35, 3] tackle the CL problem by allocating task-specific parameters. Prompt-based methods [44, 43] take the idea of prompt learning and use prompt pools against catastrophic forgetting. Replay-based methods [33, 5, 6, 16, 17, 45] store a small portion of historical data with a memory buffer. Among all of the strategies, replay-based methods have prevailed in Online CL with better performance and simplicity. Early work [33] proposed Experience Replay (ER), suggesting using a random replay buffer to alleviate catastrophic forgetting. Dark Experience Replay (DER++) [5] proposes to store the logits in the memory buffer and leverage the stored logits as dark knowledge to extend ER. ER-ACE [6] is a variant of ER with asymmetric cross-entropy loss. OCM [16] alleviates catastrophic forgetting by maximizing the mutual information between current and past representations. GSA [17] addresses the cross-task class discrimination problem with gradient self-adaption. OnPro [45] solves the shortcut learning problem with online prototype learning. In this paper, these methods are used as baselines to assess the impact of synthetic data contamination.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitation section is included in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
synthetic dataset generation: 
 To simulate a dataset contaminated with synthetic data, we employed five diffusion-based models to generate synthetic counterparts of the original datasets. These twin datasets contain the same number of images and the same classes, while all images are synthetic. The models used in generation include Stable Diffusion XL [34] , Stable Diffusion v1.4, Stable Diffusion v2.1, VQDM [15] , and GLIDE [29] .The synthetic data contamination was simulated across four benchmark datasets used in online CL, including CIFAR-10 [24] , CIFAR-100 [24] , TinyImageNet [25] , and ImageNet-100 [13, 20] .The generation of synthetic twin datasets is guided by the category names of the original dataset. For each class, we devised a simple yet effective prompt. For instance, for the class "helicopter", we employed the prompt "an image of a helicopter". After the generation by the diffusion model, we adjusted the image size to match that of the original dataset.In the experiments, we design two different settings: a) all the synthetic twin datasets are generated from Stable Diffusion XL, one of the state-of-the-art generative models; and b) Synthetic images are generated with the aforementioned five diffusion models, with each model contributing 20% to the synthetic dataset.For setting a), we denote the generated dataset as SDXL-C10, SDXL-C100, SDXL-Tiny, and SDXL-In100, respectively, while for setting b), we denote the generated dataset as Mix-C10, Mix-C100, and Mix-Tiny. Some examples of generated images and more detailed information about the synthetic twin dataset generation can be found in Appendix D.2. Notably, the generation results in Fig. 10 reveal the lack of diversity for synthetic data compared with the real data.To simulate the contamination by the synthetic data, we substitute a portion P of the original dataset with its synthetic twin, where P is the contamination ratio. We designate these contaminated datasets using specific notations. For example, we denote the CIFAR-100 contaminated by SDXL-C100 as C100/SDXL. More details about the simulation of contamination are included in Appendix D. 3 . In the following sections, we will investigate the effect of synthetic contamination with these simulated datasets.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
synthetic data contamination in online cl: 
 In this section, we explore the potential impact of synthetic data contamination on the existing online CL methods and exhibit specific properties of synthetic data.When dealing with synthetic data contamination, it is crucial to measure the impact of such contamination on existing methods. In that sense, we train ResNet-18 models in the online continual setting on the contaminated dataset C10/SDXL, C100/SDXL, Tiny/SDXL, and In-100/SDXL with representative online CL methods. We observed that as the contamination ratio P increases, the performance of all existing methods drops significantly, as shown in Table 2 . Detailed information about the experiment settings can be found in Sec. 6 . With such experiments, we can form the following observation:Observation 1 As a source of potential contamination, synthetic data is harmful to the performance of existing online CL methods. Performance degradation increases as contamination becomes more severe. Replay-based methods are characterized by the existence of a replay buffer, which helps alleviate forgetting and implicitly improves network plasticity [42] . The quality of data in the replay buffer intuitively influences network performance. Motivated by Obs. 1, we hypothesize that the presence of synthetic data in the replay buffer might degrade performance. We trained ER on the C100/SDXL dataset, with extra information on the samples' synthetic status (i.e., whether the image is real or synthetic). We employed two memory strategies: storing only real data in the replay buffer and storing only synthetic data. The results showed in Table 1 indicate that knowing the synthetic status and storing only real data in the replay buffer can achieve performance on par with the no-contamination scenario, even at high contamination ratios. For instance, at a contamination ratio of P = 80%, ER achieves an accuracy of 38.13% on C100/SDXL when only real images are stored in the replay buffer. This result is comparable to the accuracy achieved when training on the clean CIFAR-100 dataset (38.70%), which contains four times more real data. These findings also demonstrate the potential of synthetic models to enhance performance through data generation and augmentation in an online continual learning scenario.The memory buffer plays a key role in replay-based methods, and storing real samples in the memory buffer is effective against performance degradation caused by contamination.Lower entropy distribution. One noteworthy characteristic of synthetic data is its lower entropy distribution compared to real data, as observed from the perspective of continual learners. Fig. 2 shows the entropy distribution produced by a representative method ER and a state-of-the-art method OnPro when trained on the contaminated dataset. The values in the histogram are calculated at the end of the training on the whole training dataset (In-100/SDXL, P = 50%). From the figure, we can spot a salient distribution difference between synthetic data and real data. Moreover, the synthetic samples have a peak in entropy distribution close to 0. Extra entropy histogram with other baselines can be found in Appendix C.1. Thus, we conclude with another finding:Observation 3 Compared with real data, synthetic data entropy tends to be lower.Feature gap in the embedding space. We find another intriguing property of synthetic data in the feature embeddings. Fig. 3 shows the t-SNE [39] visualization of the memory data produced by ER and OnPro on the In-100/SDXL dataset. For clarity, we only visualize the embeddings of the first 10 classes. With the synthetic data contamination, the features of the synthetic samples are better clustered than the real data. The pattern of the clustering of synthetic data indicates the ease of classification, which is on par with the limited diversity of synthetic data (cf. Fig. 10 ), and the low entropy distribution (cf. Obs. 3). Moreover, the embeddings of real data are inferior and fail to align with the superior embeddings of synthetic samples, which explains the performance degradation of inference on real test datasets. Extra visualization produced by other baselines is illustrated in Appendix C.2.Observation 4 With the limited diversity of synthetic data, the synthetic data are better clustered than the real data, leading to a misalignment in the embedding space between synthetic samples and real samples. Such misalignment likely contributes to performance deterioration.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The source code of our work, along with the result of the hyperparameter search are included in the supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
proposed method: 
 Fig. 1 presents the main framework of our proposed ESRM to alleviate the performance degradation caused by synthetic data contamination. In this section, we introduce the two components of ESRM: Entropy Selection (ES) and Real-synthetic similarity Maximization (RM). Then, we explain the whole ESRM framework. The introduced ES aims to select more representative samples in the memory buffer. For replay-based methods, having high-quality samples in the memory buffer helps alleviate forgetting and achieve better overall performance. As per Obs. 2, selecting real images into the memory buffer can provide more representative and reliable features and therefore lead to better performance. Motivated by this, we propose ES, a memory management strategy. Guided by Obs. 3 (synthetic data has lower entropy distributions), the core idea of ES is to select more real samples in the memory buffer based on the entropy distribution of the current batch.At the beginning of the training, ES initializes an empty buffer. When a new batch comes, ES drops 50% of the batch samples with lower entropy, and stores the remaining samples, along with their entropy of the prediction. Once the buffer is full, as shown in Fig. 4 , ES takes four steps to replace the elements in the buffer. Firstly, ES drops 50% of low entropy samples in the incoming batch. Then, ES uses Reservoir Sampling [21, 40] to decide whether to keep or discard the remaining incoming samples. If Reservoir Sampling decides to keep the incoming sample, it will nominate a buffer sample. After that, ES checks the class of the nominated sample and chooses a sample of the same class with the lowest entropy in the memory buffer. And finally, ES replaces the chosen memory sample with the incoming sample, along with its entropy. Moreover, all entropy values in the memory buffer are updated by the current model's prediction at the end of each task. The pseudo-code of ES is given in Appendix B.Prior to introducing the loss function of RM, we present the network structures of our ESRM. As shown in Fig. 1 , the continual learner consists of three components: a feature extractor f , a projection head g, and a classifier ϕ. The output dimension of the projection head g is set to 128. For each sample x from incoming data stream X new , the projected embedding z can be formulated as:EQUATIONAs per Obs. 4, the gap in the embedding space might be a cause for the performance degradation in the contamination setting. To tackle this issue, we propose RM. The main idea of RM is to maximize the cosine similarity between the features of real and synthetic data. Motivated by the seminal supervised contrastive loss [22] , we propose our variant RM loss.RM aims to maximize the similarity between two groups of data X 1 and X 2 . We define the loss to match the similarity of samples in group X 1 to group X 2 as:EQUATIONwhereI 1 = {i : x i ∈ X 1 }, I 2 = {d : x d ∈ X 2 }are the set of the indices of X 1 , X 2 , respectively. And P (i) = {p ∈ X 2 : y p = y i } is the set of the indices of positive samples in group X 2 , which share the same class with x i . τ is the temperature hyperparameter which is set to 0.07. Since L M (X 1 , X 2 ) only optimize samples in the group X 1 , in the optimization, it is used together with L M (X 2 , X 1 ). Due to RM aims to maximize the inter-group similarity between the X 1 group and the X 2 group, we do not need to perform augmentations as shown in similar work, such as SimCLR [9] and SupCon [22] . Intuitively, the way to handle the similarity matrix is illustrated in Fig. 1 .For an incoming batch X new , we use entropy criteria to split it into two groups X new + and X new of the same size. Group X new + includes 50% of the samples with the highest entropy, and as per Obs. 3, tend to contain more real images. On the contrary, group X new contains low entropy samples which tend to be synthetic. To alleviate the gap in feature embeddings mentioned in Obs. 4, we maximize the inter-group similarity between X new + and X new with L M (X new + , X new -) and L M (X new -, X new + ). Moreover, to align the holistic feature embeddings between the stream data X new and memory data X mem , we also applied L M (X new , X mem ) and L M (X mem , X new ) in the loss function.Thus, the proposed RM to alleviate the feature gap in Obs. 4 can be achieved by employing L RM :EQUATIONThe overall framework of ESRM is shown in Fig. 1 . Besides ES and RM, following [42] , ESRM employs a self-distillation technique to alleviate the overconfidence problem of the replay-based methods. For the combined batch X = (X new , X mem ), we apply the self-distillation as:EQUATIONwhere D KL (•) is the Kullback-Leibler divergence, t is another temperature hyperparameter which is set to 4, ϕ(f (aug(X))) is the fixed copy of ϕ(f (aug(X))), without gradient propagation, and aug(•) is the data augmentation used in the training process, with detailed information in Appendix D.5.Thus, the total loss of ESRM can be formulated as:EQUATIONwhereL CE = CE(ϕ(f (X)), y) + CE(ϕ(f (aug(X))), y) is the cross-entropy loss, and λ 1 , λ 2 are the balancing hyperparamteres. We set λ 1 = 1 and λ 2 = 0.5 after a small hyperparameter search as illustrated in the Appendix D.6. Baselines. We evaluate the effectiveness of ESRM against six representative and state-of-the-art baselines, including ER [33] , DER++ [5] , ERACE [6] , OCM [16] , GSA [17] , and OnPro [45] .Implementation details. We use full-width ResNet-18 (not pre-trained) as the backbone for all experiments. For a fair comparison, we conduct a hyperparameter search on CIFAR-100 (Memory Size = 5K) and apply the same hyperparameter to all settings. Stream batch size is set to 10 and memory batch size is set to 64. We do not use multiple updates trick for incoming batches as detailed in [2] . Detailed information about task allocation, hyperparameter search protocol, and data augmentation can be found in Appendix D.Buffer size. For CIFAR-10 and CIFAR-100 experiments, the buffer size is set to 1,000 and 5,000, respectively. For the harder TinyImageNet experiments, the buffer size is set to 10,000. The buffer size of ImageNet-100 is set to 5,000. Appendix C.5 demonstrates more experiments with different memory buffer sizes. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code is included in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
results and analysis: 
 Final Average Accuracy. Table 2 shows the final average accuracy of learners trained with four datasets, including C10/SDXL, C100/SDXL, Tiny/SDXL, and In-100/SDXL, with different contamination ratios P . More results on C10/Mix, C100/Mix, and Tiny/Mix are given in Appendix C.3. For the six baselines, we can notice a significant performance drop when synthetic data contamination appears. Notably, when the contamination is severe (contamination ratio P ≥ 70%), the performance degradation is significant. Also, it can be observed that ESRM is less impacted by synthetic data contamination. For most datasets and contamination ratio P , the ESRM performance drop is the lowest of the compared methods. This is remarkably true for large values of P .Apart from the robustness against synthetic data contamination, the absolute performance of ESRM is also attractive. In most cases, ESRM outperforms the baseline methods by a large margin. More interestingly, for some datasets, such as CIFAR-100 and ImageNet-100, even with an extreme contamination ratio, ESRM can still achieve a substantial performance, while the performance of most baseline methods is unsatisfactory.Plasticity and Stability Metrics. We measure the model's plasticity and stability with Learning Accuracy (LA) [32] and Relative Forgetting (RF) [42] , respectively. As shown in Table 3 Table 7 : Final Average Accuracy (%; higher is better) on DIL-C20/SDXL dataset. Numbers in parentheses indicate the performance degradation due to synthetic contamination compared to the clean setting. The average and deviation over 10 runs are reported. Table 7 shows the final average accuracy with different contamination ratios. Notably, we adapted the CIL-specifically designed components in OnPro and GSA to the DIL scenario, and the performance suffered a decent loss. We did not report ERACE results because its Asymmetric Cross Entropy (ACE) loss converges to standard cross-entropy loss in the DIL scenario, making it equivalent to vanilla ER. The experimental results show that ESRM can yield robust performance against domain shift in the DIL setting, under different synthetic contamination situations, which validates the efficiency of ESRM under DIL settings.Effect of ES. To evaluate the effect of ES, we substitute ES with three different memory strategies: random sampling, storing real data only, and storing synthetic data only. Note that storing real or synthetic data requires knowing the ground truth of an image's synthetic status, which is not practical in realistic settings. Additionally, storing only real data is an idealized case for memory management, while storing only synthetic data represents the worst-case scenario. As shown in Table 5 , both ES and random reservoir sampling [21, 40] outperform the worst-case scenario by a large margin. Moreover, ES outperforms the random sampling significantly.Effects of loss terms. We also conduct experiments to verify the effects of loss terms in Eq. 5. As shown in Table 6 , Both L SDC and L RM can benefit the final average accuracy of the classification. Furthermore, the combination of the two loss terms can further improve the final accuracy, validating that both terms complement each other. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The main experimental setting is included in the paper, and all of the details are included in the source code in the supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Error bars are reported in the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 With the widespread availability of advanced generative models, the prevalence of AI-generated images appears inevitable, posing a potential challenge for researchers attempting to collect datasets devoid of AI-generated content from the Internet. In this paper, we examine the potential side effects of AI-powered image generation on the continual learning community. First, we experimentally demonstrate that synthetic data has become a potential source of data pollution. We spot a catastrophic performance loss when the contaminated datasets are used to train continual learning models. Based on our experiments, we identify and summarize four typical characteristics of synthetic data when involved in the training of continual learners. Additionally, we propose ESRM, a method designed to alleviate performance deterioration, maintaining satisfactory performance even with highly contaminated datasets. Lastly, we hope our work highlights the need for improved regulation and systematic control over generated data, such as watermarking AI-generated content before publication. Internet data is a valuable resource accumulated over decades. We believe ensuring the integrity of Internet data is crucial for the future soundness of AI development. Secondly, the way we produce the generated dataset is simple: we use prompts like "an image of a <class_name>" in the generation. In reality, the users' prompts are usually more diverse. Some works use LLMs to simulate more realistic prompts, where we leave a further in-depth analysis for future research.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Detailed information is included in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
 
 Appendix: The PyTorch-like pseudo code showing how the ES updates the memory buffer is shown in Alg. 1. As mentioned in Sec. 4, we show the entropy distribution produced by other baselines, when trained on In-100/SDXL (P = 50%). Fig. 8 illustrates the entropy distribution histogram, which is calculated at the end of the training on the whole contaminated dataset. Similar to the result in Fig. 2 , the entropy distribution of the synthetic data is saliently lower than the entropy distribution of the real data. We show additional experiments of t-SNE visualization of memory data produced by other baseline methods. As shown in Fig. 9 , we visualize the memory embeddings of different baselines on the In-100/SDXL (P = 50%) dataset. For clarity, we only visualize the first 10 classes. Similar to the results in Fig. 3 , the synthetic data are better clustered compared with the real data. This proves the Obs. 4.As mentioned in Sec. 6, we would like to include the experiment results on C10/Mix, C100/Mix, and Tiny/Mix datasets. The final average accuracy of different methods on the dataset is included in Table 8 . Similar to the results in Table 2 , ESRM has less performance degradation and better performance in most cases.As mentioned in Sec. 6.2, the plasticity and stability metrics of methods on other datasets are demonstrated in Table 9 and 10. Similar to the results in Table 3 and 4 , in most settings, the plasticity metric (LA) and stability metric (RF) of baseline methods drop with an increased contamination ratio P . For the model plasticity, ESRM alleviates the issue with a larger plasticity. From a stability perspective, ESRM addresses the issue of stability degradation with the presence of contamination.C.5 The impact of buffer size.In Sec. 6.2, we evaluate the effectiveness of ESRM in a limited buffer size setting. To evaluate our methods' scalability against different buffer sizes M and contamination ratio P , we compare the accuracy of different methods on C100/SDXL with different buffer sizes M , as shown in Table 11 . Similar to the results in As discussed in Sec. 3, we used four benchmark datasets in evaluation, including CIFAR-10/100, TinyImageNet, and ImageNet-100. In the experiments, all of the datasets are split into tasks containing non-overlapping classes. The details about the task split are as follows:CIFAR-10 [24] has ten classes with 50,000 training images and 10,000 test images. The image is 32 × 32 in size. The dataset is split into five disjoint tasks with two classes per task.CIFAR-100 [24] has 100 classes with 50,000 training samples and 10,000 test samples. Image size is 32 × 32. It is split into 10 disjoint tasks with 10 classes per task.TinyImageNet [25] has 200 classes, 100,000 training samples, and 10,000 test samples. Image size is 64 × 64. The dataset is split into 100 non-overlapping tasks with two classes per task.ImageNet-100 [20] is a subset of ImageNet-1k [13] dataset. It consists of 100 classes. We follow [11] for the class selection. We do not perform image resizing in generating ImageNet-100 from the original ImageNet-1k dataset. The dataset is split into 10 disjoint classes with 10 classes per task.Image size of SDXL-IN100. In the synthetic dataset generation, we manually adjust the size of generated images to match that of the original dataset. For the ImageNet-100 dataset, since the image size is not fixed, we resize the generated images to 224 × 224, to align with the training protocols. Samples from the generated datasets. Fig. 10 shows some samples from class "n01558993" (Robin) in the SDXL-In100 dataset along with the samples from the original dataset. We can notice a significant loss of diversity in the samples from SDXL-In100.As mentioned in Sec. 3.2, we generate synthetic twins of benchmark datasets and substitute a fixed portion P of the original datasets with their synthetic counterparts. Similar to the class-wise distribution of Mix-C10/C100/Tiny, we also conduct the mixture class-wise. For datasets in setting a), we assure that the contamination ratio in each class is also P . For datasets in setting b), while maintaining a consistent class-wise contamination ratio, we also ensure that each individual synthetic model contributes 20% of the contamination in each class.In some work, the authors use a fixed task sequence for fair comparison. However, the final performance is largely affected by the task order. For fair comparison, we randomly assign the class to tasks and shuffle the sequence of tasks with 10 fixed random seeds. This ensures the evaluation is not biased to task difficulty.Data augmentation is effective in boosting the training of online continual learners. Methods may benefit differently from different augmentation intensities, and some methods may favor simpler augmentations instead of complicated ones. For a fair comparison, it is vital to ensure all methods are in their optimal performance. Thus, we introduce two different augmentation strategies:1. Partial strategy. The partial augmentation is a weaker version of augmentation, consisting of random cropping with p = 0.5, followed by random horizontal flip with p = 0.5.2. Full strategy. The full augmentation strategy is a stronger version of augmentation. The full augmentation strategy is a superset of its partial counterpart, which consists of random cropping, random horizontal flip, color jitter, and random grayscale. The parameters for color jitter are set to (0.4, 0.4, 0.4, 0.1) with p = 0.8. The probability of random grayscale is set to p = 0.2.We define the data augmentation strategy of each method with a hyperparameter search, as detailed in Appendix D.6.For hyperparameters in all of the methods (except DER++ on TinyImageNet), we conduct a hyperparameter search on the clean CIFAR-100 dataset with a memory size of 5,000, and apply the same hyperparameter to all of the other settings. The exhaustive list of the hyperparameter search is shown in Table 13 .Special treatment for DER++. DER++ encounters a catastrophic performance defect (close to 0) when trained on the TinyImageNet dataset using an optimizer with momentum. Thus, we applied another hyperparameter search for DER++ on TinyImageNet and found the SGD optimizer (without Momentum) gives reasonable performance. All the experiments of DER++ on TinyImageNet are conducted using these new hyperparameters. For continual learning baselines, we use the codebase listed in Table 14 to reimplement baseline methods. For image generation methods, we use the Diffuser library from Hugging Face for Stable Diffusion and VQDM experiments, and we use the codebase in Table 14 for GLIDE. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The Code of Ethics is fully respected and obeyed in our research.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: All reviewers found this paper very novel and interesting.  The authors presented a strong response to the comments raised by reviewers.

Reviews: review: paper_summary: paper_summary: This paper investigates the negative impact of synthetic data contamination on existing online continual learning methods. An entropy selection with the real-synthetic similarity maximization method is proposed to alleviate the performance deterioration.
strength_weakness: strength_weakness: 1. The creation of simulated data is the cornerstone of this research, but there is a lack of detailed explanation on how to generate these data in the main paper.
2. For Observation 4 "With the limited diversity of synthetic data", how about mixing the synthetic data from different generation models to increase the diversity, since the synthetic images on the internet also form different models? While the observation remains unchanged in this case?
3. In order to simulate real data collection more realistically, in addition to synthetic data, new open-domain real data should also be incorporated.
questions: questions: Please refer to the weaknesses.
limitations: limitations: The limitations are discussed while the potential negative societal impact is not discussed. However, for this work, I think it is not necessary to discuss this.
paper_summary: paper_summary: Image generation has been showing promising achievements in the last few years, but the generative models may not be able to keep up with the distribution of the real samples. Due to low diversity, synthetic images perform poorly in downstream learning tasks. This paper tackles this problem by diversifying the sample selection using the proposed ESRM framework, which prioritizes the high-entropy samples in the experience replay memory with an RM loss function. The authors first performed experimental analysis to build their motivation in section 4 and proposed their solution to the problem in section 5. The experiments are diversified with several datasets, benchmarks, and extensive ablation studies.
strength_weakness: strength_weakness: 1. The paper only deals with diffusion generative models. I understand that diffusion models are the most powerful image generators but it would be good to experiment other generative models.

2. The accuracy scores are overall low in CL. Does the proposed method 

3. Why is a fixed 50% dropping rate selected for ES? 

4. Downstream ML tasks have been explored with synthetic data to validate synthetic datasets, and related works clearly miss that angle. The authors should also discuss them and maybe add some of them to benchmarks.
questions: questions: Authors can refer to the weaknesses section for major questions. Here, I listed minor points for authors reference:

1. It seems there is a typo in the line 231. 
2. It could be good to highlight the best results in Tables 3 and 4, the same as Table 2.
limitations: limitations: I agree with the limitation the authors brought up.
paper_summary: paper_summary: This paper  investigates the impact of AI-generated images on the performance of online continual learning (CL) models. It introduces a novel method called Entropy Selection with Real-synthetic similarity Maximization (ESRM) to mitigate the negative effects of synthetic data contamination. ESRM leverages entropy-based sample selection and a contrastive learning approach to align the feature embeddings of real and synthetic data, thereby enhancing the robustness of CL models against the degradation caused by synthetic data.
strength_weakness: strength_weakness: - The paper evaluates the impact of synthetic data using a limited set of generative models. And the experiments are primarily conducted on image classification datasets.  Expanding this to include a broader range of tasks, particularly truly text-to-images, text-to-text, could strengthen the findings.

- The method for generating synthetic datasets is straightforward, using simple prompts. Incorporating more complex and diverse prompts, potentially using large language models to simulate user queries, could better reflect real-world synthetic data contamination.

- The reliance on entropy as a key metric for distinguishing real from synthetic data might not be universally applicable. Further exploration of this metric's effectiveness across different domains and its theoretical underpinnings could bolster the method's credibility.

- While the paper provides insights into the method's effectiveness, there is limited discussion on its computational efficiency and scalability, especially when dealing with large-scale datasets or high contamination ratios. Addressing these aspects could be crucial for practical applications where computational resources and time are critical constraints.
questions: questions: Please refer to the weaknesses above.
limitations: limitations: The authors have discussed the limitations in the paper.

Comments: title: Thanks for the  rebuttal.
Thank you for your rebuttal
comment: Thank you for your feedback and insightful comments.
The authors have addressed most of my concerns. I decide to keep my initial positive rating.
Thank you for your feedback and valuable comments.
Thank you for the rebuttal.  This paper is very insteresting and promising. After considering the comments from other reviewers and the rebuttal, I decided to raise my score.
Dear Reviewers,
Authors have carefully prepared their rebuttal addressing the concerns you have raised. Please check the rebuttals and join the discussion about the paper. 
Thanks,
Your AC