Title: Dealing with Synthetic Data Contamination in Online Continual Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Continual Learning

Image Generation

Replay-based method

Entropy Selection
Abstract: Image generation has shown remarkable results in generating high-fidelity realistic images, in particular with the advancement of diffusion-based models. However, the prevalence of AI-generated images may have side effects for the machine learning community that are not clearly identified. Meanwhile, the success of deep learning in computer vision is driven by the massive dataset collected on the Internet. The extensive quantity of synthetic data being added to the Internet would become an obstacle for future researchers to collect "clean" datasets without AI-generated content. Prior research has shown that using datasets contaminated by synthetic images may result in performance degradation when used for training. In this paper, we investigate the potential impact of contaminated datasets on Online Continual Learning (CL) research. We experimentally show that contaminated datasets might hinder the training of existing online CL methods. Also, we propose Entropy Selection with Real-synthetic similarity Maximization (ESRM), a method to alleviate the performance deterioration caused by synthetic images when training online CL models. Experiments show that our method can significantly alleviate performance deterioration, especially when the contamination is severe. For reproducibility, the source code of our work is available at https://github.com/maorong-wang/ESRM.

Dealing with Synthetic Data Contamination in Online Continual Learning

Abstract

Image generation has shown remarkable results in generating high-fidelity realistic images, in particular with the advancement of diffusion-based models. However, the prevalence of AI-generated images may have side effects for the machine learning community that are not clearly identified. Meanwhile, the success of deep learning in computer vision is driven by the massive dataset collected on the Internet. The extensive quantity of synthetic data being added to the Internet would become an obstacle for future researchers to collect 'clean' datasets without AI-generated content. Prior research has shown that using datasets contaminated by synthetic images may result in performance degradation when used for training. In this paper, we investigate the potential impact of contaminated datasets on Online Continual Learning (CL) research. We experimentally show that contaminated datasets might hinder the training of existing online CL methods. Also, we propose E ntropy S election with R eal-synthetic similarity M aximization (ESRM), a method to alleviate the performance deterioration caused by synthetic images when training online CL models. Experiments show that our method can significantly alleviate performance deterioration, especially when the contamination is severe. For reproducibility, the source code of our work is available at https://github.com/maorong-wang/ESRM .

1 Introduction

Continual Learning (CL) [10, 31, 41, 12] solves the problem of learning from a sequence of everemerging machine-learning tasks without forgetting previously learned knowledge. Defined by learning manners, CL can be classified into two categories [4]: offline CL and online CL. In offline CL ( i.e. conventional CL), the learners can access the training dataset on current task multiple times before proceeding to the next task. In online CL, the training data also comes in a continual data stream, and the continual learner only sees the training data once. Besides learning manners, there are also three typical CL settings [38]: Task-Incremental Learning (TIL), Domain-Incremental Learning (DIL), and Class-Incremental Learning (CIL). In this paper, we investigate the more challenging CIL setting in the online CL manner.

Image generation with deep generative models has shown remarkable success. Thanks to denoising diffusion models [19, 34], Internet users are capable of generating high-fidelity and realistic images within several seconds. Despite the astonishing quality of those images to human eyes, research has shown that AI-generated content may be harmful when used to train machine learning models, leading to potential performance deterioration [18, 28], bias amplification [8], loss of diversity [28], etc.

Recently, it has become a trend for researchers to collect datasets from the Internet, and synthetic data contamination would become a potential threat to the CL community. Moreover, the online CL is particularly affected as assessing the soundness of data in the online scenario is impractical. In this work, we first aim to investigate how this new form of dataset contamination might affect the existing Online CL methods. Then, we empirically observe the characteristics of synthetic data when used to train online CL models and form four observations of synthetic data properties in online CL, which might be of interest to the community. Moreover, we investigate synthetic data properties and exhibit specific differences in terms of entropy and representations when compared to real data. Guided by these properties, we propose E ntropy S election with R eal-synthetic similarity M aximization (ESRM), a method to alleviate the performance degradation caused by the synthetic contamination. As a replay-based method, ESRM consists of two key components: Entropy Selection (ES) and Real-synthetic similarity Maximization (RM). ES selects more realistic samples in the memory buffer to alleviate catastrophic forgetting. RM is a contrastive learning based optimization strategy, that aims to alleviate the performance artifact caused by synthetic data contamination.

The major contribution of this paper can be summarized as follows:

We investigate the potential impact of synthetic data contamination on existing online CL methods and outline four observations regarding the properties of synthetic data in continual scenarios.

We propose ESRM, a method to alleviate the performance deterioration caused by synthetic data contamination.

Comprehensive experiments show that ESRM can successfully mitigate the performance deterioration caused by synthetic contamination, especially when the contamination is severe.

2 Related Work

Synthetic data contamination. Recently, diffusion models [19, 34] have achieved high-fidelity image generation and surpassed GANs in terms of image quality and diversity. Likewise, text-toimage generation based on diffusion models can generate astonishing images that faithfully follow the users' text instructions. Furthermore, these generative models demonstrate excellent extrapolation capabilities ( i.e. , meaningfully combining concepts that would be nearly impossible to combine in reality), such as 'a photo of an astronaut riding a house'. Various generative models are open-sourced to the public, and users can use these models to generate realistic images in seconds. However, while people are appreciating the new format of art and flooding the Internet with fabulous images, such synthetic images are difficult to differentiate from the real ones. Therefore, these generated images are becoming a potential source of contamination for the future datasets collected from the Internet. Research has proven that synthetic data contamination may lead to a significant performance drop when supervising machine learning models in non-continual scenarios [18, 28]. Also, training deep models with such a contaminated dataset may give rise to bias amplification [8] and loss of diversity [28]. To tackle the issue caused by the synthetic contamination, researchers have proposed different strategies to detect the synthetic data with deep learning based detectors [30, 27]. However, the challenge brought by synthetic data contamination to the CL community is exclusive, and it is even more problematic in the online scenario since it is almost impossible to assess the quality of the training data, due to the unique challenge brought by the online setting.

Continual Learning. The mainstream CL strategies can be classified into four categories: regularization-based, parameter-isolation-based, prompt-based, and replay-based. Regularizationbased methods [7, 26, 1, 23, 46] design and apply extra regularization terms to balance the learning and forgetting of CL learners. Parameter-isolation-based methods [14, 36, 37, 35, 3] tackle the CL problem by allocating task-specific parameters. Prompt-based methods [44, 43] take the idea of prompt learning and use prompt pools against catastrophic forgetting. Replay-based methods [33, 5, 6, 16, 17, 45] store a small portion of historical data with a memory buffer. Among all of the strategies, replay-based methods have prevailed in Online CL with better performance and simplicity. Early work [33] proposed Experience Replay (ER) , suggesting using a random replay buffer to alleviate catastrophic forgetting. Dark Experience Replay (DER++) [5] proposes to store the logits in the memory buffer and leverage the stored logits as dark knowledge to extend ER. ER-ACE [6] is a variant of ER with asymmetric cross-entropy loss. OCM [16] alleviates catastrophic

Projector

Time step

Entropy

Feature

Real-syntheticsimilarity!

Entropylabelingi

Selection (ES)

Extractor

maximization (RM)

M

Classifier

Entropy Feedback

Cross-entropyLoss

ProbabilityDistribution

High Entropy samples (Likely to be real)

Pushaway

Pulltogether

Ignored

LowEntropysamples(Likelytobegenerated)

forgetting by maximizing the mutual information between current and past representations. GSA [17] addresses the cross-task class discrimination problem with gradient self-adaption. OnPro [45] solves the shortcut learning problem with online prototype learning. In this paper, these methods are used as baselines to assess the impact of synthetic data contamination.

3 Preliminary

3.1 Synthetic dataset generation

To simulate a dataset contaminated with synthetic data, we employed five diffusion-based models to generate synthetic counterparts of the original datasets. These twin datasets contain the same number of images and the same classes, while all images are synthetic. The models used in generation include Stable Diffusion XL [34], Stable Diffusion v1.4, Stable Diffusion v2.1, VQDM [15], and GLIDE [29]. The synthetic data contamination was simulated across four benchmark datasets used in online CL, including CIFAR-10 [24], CIFAR-100 [24], TinyImageNet [25], and ImageNet-100 [13, 20].

The generation of synthetic twin datasets is guided by the category names of the original dataset. For each class, we devised a simple yet effective prompt. For instance, for the class 'helicopter', we employed the prompt 'an image of a helicopter'. After the generation by the diffusion model, we adjusted the image size to match that of the original dataset.

In the experiments, we design two different settings: a) all the synthetic twin datasets are generated from Stable Diffusion XL, one of the state-of-the-art generative models; and b) Synthetic images are generated with the aforementioned five diffusion models, with each model contributing 20% to the synthetic dataset.

For setting a) , we denote the generated dataset as SDXL-C10, SDXL-C100, SDXL-Tiny, and SDXLIn100, respectively, while for setting b) , we denote the generated dataset as Mix-C10, Mix-C100, and Mix-Tiny. Some examples of generated images and more detailed information about the synthetic twin dataset generation can be found in Appendix D.2. Notably, the generation results in Fig. 10 reveal the lack of diversity for synthetic data compared with the real data.

3.2 Simulation of synthetic data contamination

To simulate the contamination by the synthetic data, we substitute a portion P of the original dataset with its synthetic twin, where P is the contamination ratio. We designate these contaminated datasets using specific notations. For example, we denote the CIFAR-100 contaminated by SDXL-C100 as C100/SDXL. More details about the simulation of contamination are included in Appendix D.3. In

(a) ER

(b) OnPro

(a) ER

(b) OnPro

Real data

Synthetic data

29000

Real

1000

Real

28000

Synthetic

Synthetic

27000

800

Count

5000

Count

600

4000

3000

400

2000

200

1000

0

0

1

Entropy

2

3

4

0

0

1

Entropy

2

3

4

the following sections, we will investigate the effect of synthetic contamination with these simulated datasets.

4 Synthetic Data Contamination in Online CL

In this section, we explore the potential impact of synthetic data contamination on the existing online CL methods and exhibit specific properties of synthetic data.

4.1 Contamination as a cause of performance degradation

When dealing with synthetic data contamination, it is crucial to measure the impact of such contamination on existing methods. In that sense, we train ResNet-18 models in the online continual setting on the contaminated dataset C10/SDXL, C100/SDXL, Tiny/SDXL, and In-100/SDXL with representative online CL methods. We observed that as the contamination ratio P increases, the performance of all existing methods drops significantly, as shown in Table 2. Detailed information about the experiment settings can be found in Sec. 6. With such experiments, we can form the following observation:

Observation 1 As a source of potential contamination, synthetic data is harmful to the performance of existing online CL methods. Performance degradation increases as contamination becomes more severe.

4.2 Detecting synthetic data matters

Replay-based methods are characterized by the existence of a replay buffer, which helps alleviate forgetting and implicitly improves network plasticity [42]. The quality of data in the replay buffer intuitively influences network performance. Motivated by Obs. 1, we hypothesize that the presence of synthetic data in the replay buffer might degrade performance. We trained ER on the C100/SDXL dataset, with extra information on the samples' synthetic status ( i.e. , whether the image is real or synthetic). We em-

ployed two memory strategies: storing only real data in the replay buffer and storing only synthetic data. The results showed in Table 1 indicate that knowing the synthetic status and storing only real data in the replay buffer can achieve performance on par with the no-contamination scenario, even at high contamination ratios. For instance, at a contamination ratio of P = 80% , ER achieves an accuracy of 38.13% on C100/SDXL when only real images are stored in the replay buffer. This result is comparable to the accuracy achieved when training on the clean CIFAR-100 dataset (38.70%), which contains four times more real data. These findings also demonstrate the potential of synthetic models to enhance performance through data generation and augmentation in an online continual learning scenario.

Observation 2 The memory buffer plays a key role in replay-based methods, and storing real samples in the memory buffer is effective against performance degradation caused by contamination.

4.3 Synthetic data properties

Lower entropy distribution. One noteworthy characteristic of synthetic data is its lower entropy distribution compared to real data, as observed from the perspective of continual learners. Fig. 2 shows the entropy distribution produced by a representative method ER and a state-of-the-art method OnPro when trained on the contaminated dataset. The values in the histogram are calculated at the end of the training on the whole training dataset (In-100/SDXL, P = 50% ). From the figure, we can spot a salient distribution difference between synthetic data and real data. Moreover, the synthetic samples have a peak in entropy distribution close to 0. Extra entropy histogram with other baselines can be found in Appendix C.1. Thus, we conclude with another finding:

Observation 3 Compared with real data, synthetic data entropy tends to be lower.

Feature gap in the embedding space. We find another intriguing property of synthetic data in the feature embeddings. Fig. 3 shows the t-SNE [39] visualization of the memory data produced by ER and OnPro on the In-100/SDXL dataset. For clarity, we only visualize the embeddings of the first 10 classes. With the synthetic data contamination, the features of the synthetic samples are better clustered than the real data. The pattern of the clustering of synthetic data indicates the ease of classification, which is on par with the limited diversity of synthetic data (cf. Fig. 10), and the low entropy distribution (cf. Obs. 3). Moreover, the embeddings of real data are inferior and fail to align with the superior embeddings of synthetic samples, which explains the performance degradation of inference on real test datasets. Extra visualization produced by other baselines is illustrated in Appendix C.2.

Observation 4 With the limited diversity of synthetic data, the synthetic data are better clustered than the real data, leading to a misalignment in the embedding space between synthetic samples and real samples. Such misalignment likely contributes to performance deterioration.

5 Proposed Method

Fig. 1 presents the main framework of our proposed ESRM to alleviate the performance degradation caused by synthetic data contamination. In this section, we introduce the two components of ESRM: Entropy Selection (ES) and Real-synthetic similarity Maximization (RM). Then, we explain the whole ESRM framework.

5.1 Entropy selection

The introduced ES aims to select more representative samples in the memory buffer. For replay-based methods, having high-quality samples in the memory buffer helps alleviate forgetting and achieve better overall performance. As per Obs. 2, selecting real images into the memory buffer can provide more representative and reliable features and therefore lead to better performance. Motivated by this, we propose ES, a memory management strategy. Guided by Obs. 3 (synthetic data has lower entropy distributions), the core idea of ES is to select more real samples in the memory buffer based on the entropy distribution of the current batch.

At the beginning of the training, ES initializes an empty buffer. When a new batch comes, ES drops 50% of the

batch samples with lower entropy, and stores the remaining samples, along with their entropy of the prediction. Once the buffer is full, as shown in Fig. 4, ES takes four steps to replace the elements in the buffer. Firstly, ES drops 50% of low entropy samples in the incoming batch. Then, ES uses Reservoir Sampling [21, 40] to decide whether to keep or discard the remaining incoming samples. If Reservoir Sampling decides to keep the incoming sample, it will nominate a buffer sample. After that, ES checks the class of the nominated sample and chooses a sample of the same class with the lowest

entropy in the memory buffer. And finally, ES replaces the chosen memory sample with the incoming sample, along with its entropy. Moreover, all entropy values in the memory buffer are updated by the current model's prediction at the end of each task. The pseudo-code of ES is given in Appendix B.

5.2 Real-synthetic similarity maximization

Prior to introducing the loss function of RM, we present the network structures of our ESRM. As shown in Fig. 1, the continual learner consists of three components: a feature extractor f , a projection head g , and a classifier ϕ . The output dimension of the projection head g is set to 128. For each sample x from incoming data stream X new , the projected embedding z can be formulated as:

As per Obs. 4, the gap in the embedding space might be a cause for the performance degradation in the contamination setting. To tackle this issue, we propose RM. The main idea of RM is to maximize the cosine similarity between the features of real and synthetic data. Motivated by the seminal supervised contrastive loss [22], we propose our variant RM loss.

RMaims to maximize the similarity between two groups of data X 1 and X 2 . We define the loss to match the similarity of samples in group X 1 to group X 2 as:

where I 1 = { i : x i ∈ X 1 } , I 2 = { d : x d ∈ X 2 } are the set of the indices of X 1 , X 2 , respectively. And P ( i ) = { p ∈ X 2 : y p = y i } is the set of the indices of positive samples in group X 2 , which share the same class with x i . τ is the temperature hyperparameter which is set to 0.07. Since L M ( X 1 , X 2 ) only optimize samples in the group X 1 , in the optimization, it is used together with L M ( X 2 , X 1 ) . Due to RM aims to maximize the inter-group similarity between the X 1 group and the X 2 group, we do not need to perform augmentations as shown in similar work, such as SimCLR [9] and SupCon [22]. Intuitively, the way to handle the similarity matrix is illustrated in Fig.1.

For an incoming batch X new , we use entropy criteria to split it into two groups X new + and X new -of the same size. Group X new + includes 50% of the samples with the highest entropy, and as per Obs. 3, tend to contain more real images. On the contrary, group X new -contains low entropy samples which tend to be synthetic. To alleviate the gap in feature embeddings mentioned in Obs. 4, we maximize the inter-group similarity between X new + and X new -with L M ( X new + , X new -) and L M ( X new -, X new + ) . Moreover, to align the holistic feature embeddings between the stream data X new and memory data X mem , we also applied L M ( X new , X mem ) and L M ( X mem , X new ) in the loss function.

Thus, the proposed RM to alleviate the feature gap in Obs. 4 can be achieved by employing L RM :

5.3 Overall framework of ESRM

The overall framework of ESRM is shown in Fig. 1. Besides ES and RM, following [42], ESRM employs a self-distillation technique to alleviate the overconfidence problem of the replay-based methods. For the combined batch X = ( X new , X mem ) , we apply the self-distillation as:

where D KL ( · ) is the Kullback-Leibler divergence, t is another temperature hyperparameter which is set to 4, ϕ ( f ( aug ( X ))) is the fixed copy of ϕ ( f ( aug ( X ))) , without gradient propagation, and aug ( · ) is the data augmentation used in the training process, with detailed information in Appendix D.5.

Thus, the total loss of ESRM can be formulated as:

where L CE = CE ( ϕ ( f ( X )) , y ) + CE ( ϕ ( f ( aug ( X ))) , y ) is the cross-entropy loss, and λ 1 , λ 2 are the balancing hyperparamteres. We set λ 1 = 1 and λ 2 = 0 . 5 after a small hyperparameter search as illustrated in the Appendix D.6.

6 Experiments

6.1 Experiment setup

Datasets. In the experiments, we used four benchmark datasets in evaluation, including CIFAR10/100, TinyImageNet, and ImageNet-100. All of the datasets are split into tasks containing nonoverlapping classes. The details about the task split are available in Appendix D.1.

Baselines. We evaluate the effectiveness of ESRM against six representative and state-of-the-art baselines, including ER [33], DER++ [5], ERACE [6], OCM [16], GSA [17], and OnPro [45].

Implementation details. We use full-width ResNet-18 (not pre-trained) as the backbone for all experiments. For a fair comparison, we conduct a hyperparameter search on CIFAR-100 (Memory Size = 5K) and apply the same hyperparameter to all settings. Stream batch size is set to 10 and memory batch size is set to 64. We do not use multiple updates trick for incoming batches as detailed in [2]. Detailed information about task allocation, hyperparameter search protocol, and data augmentation can be found in Appendix D.

Buffer size. For CIFAR-10 and CIFAR-100 experiments, the buffer size is set to 1,000 and 5,000, respectively. For the harder TinyImageNet experiments, the buffer size is set to 10,000. The buffer size of ImageNet-100 is set to 5,000. Appendix C.5 demonstrates more experiments with different memory buffer sizes.

6.2 Results and analysis

Final Average Accuracy. Table 2 shows the final average accuracy of learners trained with four datasets, including C10/SDXL, C100/SDXL, Tiny/SDXL, and In-100/SDXL, with different contamination ratios P . More results on C10/Mix, C100/Mix, and Tiny/Mix are given in Appendix C.3. For the six baselines, we can notice a significant performance drop when synthetic data contamination appears. Notably, when the contamination is severe (contamination ratio P ≥ 70% ), the performance degradation is significant. Also, it can be observed that ESRM is less impacted by synthetic data contamination. For most datasets and contamination ratio P , the ESRM performance drop is the lowest of the compared methods. This is remarkably true for large values of P .

Apart from the robustness against synthetic data contamination, the absolute performance of ESRM is also attractive. In most cases, ESRM outperforms the baseline methods by a large margin. More interestingly, for some datasets, such as CIFAR-100 and ImageNet-100, even with an extreme contamination ratio, ESRM can still achieve a substantial performance, while the performance of most baseline methods is unsatisfactory.

Plasticity and Stability Metrics. We measure the model's plasticity and stability with Learning Accuracy (LA) [32] and Relative Forgetting (RF) [42], respectively. As shown in Table 3 and 4, for baseline methods, both plasticity and stability performance are hindered by synthetic data contamination. From the model plasticity perspective, ESRM alleviates the problem with a larger plasticity. For the model stability, ESRM solves the problem of stability degradation with the presence of contamination. It is notable that with RM loss and self-distillation loss, ESRM implicitly trades off some model stability in favor of plasticity. Plasticity and stability performance on other datasets are available in Appendix C.4.

Domain-Incremental Learning Results. While Class-Incremental Learning (CIL) setting is the standard evaluation protocol in online CL, we also evaluated the performance of ESRM in DomainIncremental Learning (DIL) scenarios. We conducted the experiment with the 20 coarse labels of the

Realdata

Y

Syntheticdata

Buffer

1.0

0.9

0.8

.=

0.7

Synthetic Image i

0.6

P=0.95

P=0.9

0.5

P=0.8

0.4

P=0.7

0.3

P=0.5

0.2

0.1

0.0

0

3000

6000

9000

12000

Iteration

CIFAR-100 dataset. Since the 100 classes in CIFAR-100 are grouped into 20 superclasses with 5 finegrained classes for each superclass, we split the CIFAR-100 dataset with 5 domain increment steps. For each step, we feed the model with the training data of a fine-grained class from each superclass. Because the model only classifies 20 coarse labels, we refer to this dataset as DIL-CIFAR20. Similar to the simulated CIFAR100/SDXL dataset, we replace the images in the DIL-CIFAR20 dataset with its Stable Diffusion XL generated counterpart with a contamination ratio P, as per the protocol in Sec. 3.2.

Table 7 shows the final average accuracy with different contamination ratios. Notably, we adapted the CIL-specifically designed components in OnPro and GSA to the DIL scenario, and the performance suffered a decent loss. We did not report ERACE results because its Asymmetric Cross Entropy (ACE) loss converges to standard cross-entropy loss in the DIL scenario, making it equivalent to vanilla ER. The experimental results show that ESRM can yield robust performance against domain shift in the DIL setting, under different synthetic contamination situations, which validates the efficiency of ESRM under DIL settings.

6.3 Ablation studies

Effect of ES. To evaluate the effect of ES, we substitute ES with three different memory strategies: random sampling, storing real data only, and storing synthetic data only. Note that storing real or synthetic data requires knowing the ground truth of an image's synthetic status, which is not practical in realistic settings. Additionally, storing only real data is an idealized case for memory management, while storing only synthetic data represents the worst-case scenario. As shown in Table 5, both ES and random reservoir sampling [21, 40] outperform the worst-case scenario by a large margin. Moreover, ES outperforms the random sampling significantly.

Effects of loss terms. We also conduct experiments to verify the effects of loss terms in Eq. 5. As shown in Table 6, Both L SDC and L RM can benefit the final average accuracy of the classification. Furthermore, the combination of the two loss terms can further improve the final accuracy, validating that both terms complement each other.

7 Discussions

The alleviation of feature misalignment. As mentioned in Obs. 4, baseline methods suffer performance degradation due to the misalignment between the inferior feature embedding of real images and the superior feature embedding of synthetic samples. Fig. 5 presents the t-SNE visualization of memory data at the end of training of ESRM on the In-100/SDXL dataset. Similar to Fig. 3, only the first 10 classes are visualized for clarity. Compared to ER and OnPro, the embeddings of synthetic and real samples in ESRM are better aligned, facilitated by the RM.

Training dynamics of memory buffer. To intuitively demonstrate the effect of ES, we visualized the percentage of synthetic data in the memory buffer throughout the whole training process. Fig. 6 displays the curve of the percentage of synthetic data in the memory buffer when the model is trained with ESRM on In-100/SDXL with different contamination ratios P . To generate this curve, we checked the memory buffer every 10 iterations. As shown in the figure, the percentage of synthetic data is close to the contamination ratio P in the early stages of training. As training progresses, the amount of synthetic data decreases. This trend intuitively illustrates the effect of ES in selecting real samples. Furthermore, we take a model trained with ESRM at the end of training and use the model's entropy

as the criterion to categorize the synthetic status of samples in the training dataset and generate an ROC curve, as shown in Fig. 7. We regard real data as positive and use models trained with ESRM on In-100/SDXL ( P = 50% ) to categorize the samples in the training set. The AUC of the ROC curve is 0.7098, showing the effect of the entropy criterion in discriminating real and synthetic samples.

8 Conclusion

With the widespread availability of advanced generative models, the prevalence of AI-generated images appears inevitable, posing a potential challenge for researchers attempting to collect datasets devoid of AI-generated content from the Internet. In this paper, we examine the potential side effects of AI-powered image generation on the continual learning community. First, we experimentally demonstrate that synthetic data has become a potential source of data pollution. We spot a catastrophic performance loss when the contaminated datasets are used to train continual learning models. Based on our experiments, we identify and summarize four typical characteristics of synthetic data when involved in the training of continual learners. Additionally, we propose ESRM, a method designed to alleviate performance deterioration, maintaining satisfactory performance even with highly contaminated datasets. Lastly, we hope our work highlights the need for improved regulation and systematic control over generated data, such as watermarking AI-generated content before publication. Internet data is a valuable resource accumulated over decades. We believe ensuring the integrity of Internet data is crucial for the future soundness of AI development.

References

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV , 2018.

Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In NeurIPS , 2019.

Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In CVPR , 2017.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurIPS , 2019.

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NeurIPS , 2020.

Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025 , 2021.

Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV , 2018.

Tianwei Chen, Yusuke Hirota, Mayu Otani, Noa Garcia, and Yuta Nakashima. Would deep generative models amplify bias in future models? arXiv preprint arXiv:2404.03242 , 2024.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607, 2020.

Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning , 12(3):1-207, 2018.

Yubei Chen Chun-Hsiao Yeh. IN100pytorch: Pytorch implementation: Training resnets on imagenet-100. https://github.com/danielchyeh/ImageNet-100-Pytorch , 2022.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. TPAMI , 44(7):3366-3385, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 , 2017.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR , pages 10696-10706, 2022.

Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In ICML , pages 8109-8126, 2022.

Yiduo Guo, Bing Liu, and Dongyan Zhao. Dealing with cross-task class discrimination in online continual learning. In CVPR , pages 11878-11887, 2023.

Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? In ICCV , pages 20555-20565, 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , volume 33, pages 6840-6851, 2020.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR , pages 831-839, 2019.

David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI , volume 32, 2018.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , 2020.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS , 114(13):3521-3526, 2017.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In NeurIPS , 2017.

Huan Liu, Zichang Tan, Chuangchuang Tan, Yunchao Wei, Jingdong Wang, and Yao Zhao. Forgery-aware adaptive transformer for generalizable synthetic image detection. In CVPR , pages 10770-10780, 2024.

Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet. In International Workshop on Epistemic Uncertainty in Artificial Intelligence , pages 59-73, 2023.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML , volume 162, pages 16784-16804, 2022.

Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In CVPR , pages 24480-24489, 2023.

German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks , 113:54-71, 2019.

Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 , 2018.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NeurIPS , 2019.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022.

Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. TPAMI , 42(3):651-663, 2018.

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016.

Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML , 2018.

Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 , 2019.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR , 9(11), 2008.

Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS) , 11(1):37-57, 1985.

Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 , 2023.

Maorong Wang, Nicolas Michel, Ling Xiao, and Toshihiko Yamasaki. Improving plasticity in online continual learning via collaborative learning. arXiv preprint arXiv:2312.00600 , 2023.

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV , pages 631-648, 2022.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR , pages 139-149, 2022.

Yujie Wei, Jiaxin Ye, Zhizhong Huang, Junping Zhang, and Hongming Shan. Online prototype learning for online continual learning. In ICCV , pages 18764-18774, 2023.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML , 2017.

Algorithm 1 PyTorch-like pseudo-code of ES.

A Limitations

The paper investigates the potential impact of synthetic data contamination on online CL research and proposes a method to alleviate side effects caused by the contamination. Nevertheless, our research has some limitations. Firstly, in the evaluation, we only use five generative models, including Stable Diffusion v1.4, Stable Diffusion v2.1, Stable Diffusion XL, VQDM, and GLIDE. Other excellent commercial generative works, such as Midjourney and DALL-E, are not included in the data generation. With limited computation/resources, we could not exhaust all generative methods.

Secondly, the way we produce the generated dataset is simple: we use prompts like 'an image of a <class_name>' in the generation. In reality, the users' prompts are usually more diverse. Some works use LLMs to simulate more realistic prompts, where we leave a further in-depth analysis for future research.

B Pseudo code for ES

The PyTorch-like pseudo code showing how the ES updates the memory buffer is shown in Alg. 1.

(a) ER

(b) DER++

(c) ERACE

(d) OCM

(e) GSA

(f) OnPro

(g) ESRM

29000

28000

Real

Synthetic

8000

Real

Synthetic

25000

26000

Real

Synthetic

6000

5000

Real

Synthetic

27000

24000

ung

4000

5000

3000

ount

8

4000

6000

ount

5000

4000

3000

ung

4000

2000

3000

2000

2000

2000

1000

1000

1000

0

0

1

Entropy

2

3

4

0

0

1

Entropy

2

3

4

0

0

1

Entropy

2

3

4

0

0

1

Entropy

2

3

4

34000

Real

1000

Real

Synthetic

2000

Real

Synthetic

Synthetic

800

1500

Count

6000

Count

600

unc

1000

4000

400

2000

200

500

0

0

1

Entropy

2

3

4

0

0

1

Entropy

2

3

4

0

0

1

Entropy

2

3

4

(a) ER

(b) DER++

(c) ERACE

(d) OCM

(e) GSA

(f) OnPro

(g) ESRM

Realdata

Y

Syntheticdata

整

C Extra Experiments

C.1 Entropy distribution of other baselines

As mentioned in Sec. 4, we show the entropy distribution produced by other baselines, when trained on In-100/SDXL ( P = 50% ). Fig. 8 illustrates the entropy distribution histogram, which is calculated at the end of the training on the whole contaminated dataset. Similar to the result in Fig. 2, the entropy distribution of the synthetic data is saliently lower than the entropy distribution of the real data.

C.2 T-SNE visualization of other baselines

We show additional experiments of t-SNE visualization of memory data produced by other baseline methods. As shown in Fig. 9, we visualize the memory embeddings of different baselines on the In-100/SDXL ( P = 50% ) dataset. For clarity, we only visualize the first 10 classes. Similar to the results in Fig. 3, the synthetic data are better clustered compared with the real data. This proves the Obs. 4.

C.3 Performance on C10/Mix, C100/Mix and Tiny/Mix Dataset

As mentioned in Sec. 6, we would like to include the experiment results on C10/Mix, C100/Mix, and Tiny/Mix datasets. The final average accuracy of different methods on the dataset is included in Table 8. Similar to the results in Table 2, ESRM has less performance degradation and better performance in most cases.

C.4 Plasticity and stability performance on other datasets

As mentioned in Sec. 6.2, the plasticity and stability metrics of methods on other datasets are demonstrated in Table 9 and 10. Similar to the results in Table 3 and 4, in most settings, the plasticity metric (LA) and stability metric (RF) of baseline methods drop with an increased contamination ratio P . For the model plasticity, ESRM alleviates the issue with a larger plasticity. From a stability perspective, ESRM addresses the issue of stability degradation with the presence of contamination.

C.5 The impact of buffer size.

In Sec. 6.2, we evaluate the effectiveness of ESRM in a limited buffer size setting. To evaluate our methods' scalability against different buffer sizes M and contamination ratio P , we compare the accuracy of different methods on C100/SDXL with different buffer sizes M , as shown in Table 11. Similar to the results in Table 2, ESRM can obtain better performance when the dataset is contaminated with synthetic data.

SDXL-In100

Original

In-100

D Implementation Details

D.1 Dataset.

As discussed in Sec. 3, we used four benchmark datasets in evaluation, including CIFAR-10/100, TinyImageNet, and ImageNet-100. In the experiments, all of the datasets are split into tasks containing non-overlapping classes. The details about the task split are as follows:

CIFAR-10 [24] has ten classes with 50,000 training images and 10,000 test images. The image is 32 × 32 in size. The dataset is split into five disjoint tasks with two classes per task.

CIFAR-100 [24] has 100 classes with 50,000 training samples and 10,000 test samples. Image size is 32 × 32 . It is split into 10 disjoint tasks with 10 classes per task.

TinyImageNet [25] has 200 classes, 100,000 training samples, and 10,000 test samples. Image size is 64 × 64 . The dataset is split into 100 non-overlapping tasks with two classes per task.

ImageNet-100 [20] is a subset of ImageNet-1k [13] dataset. It consists of 100 classes. We follow [11] for the class selection. We do not perform image resizing in generating ImageNet-100 from the original ImageNet-1k dataset. The dataset is split into 10 disjoint classes with 10 classes per task.

D.2 Details about synthetic dataset generation.

Image size of SDXL-IN100. In the synthetic dataset generation, we manually adjust the size of generated images to match that of the original dataset. For the ImageNet-100 dataset, since the image size is not fixed, we resize the generated images to 224 × 224 , to align with the training protocols.

Class-wise distribution of sources for Mix-C10/C100/Tiny dataset. In the main manuscript, we mentioned that the dataset of setting b) in Sec. 3.1 is generated from five synthetic models: Stable Diffusion v1.4, Stable Diffusion v2.1, Stable Diffusion XL, VQDM, and GLIDE. Each method contributes 20% of the dataset in setting b) . In our implementation, we ensure that this distribution is consistent across all classes in the dataset, so that each class has an equal number of images from each generation model.

Hyperparameters used in image generation. For Stable Diffusion and VQDM, we use source code and model snapshots from huggingface, as mentioned in Table 14. For Glide experiments, we use the official implementation and the released model snapshots. Following the recommendation, we use the refiner in Stable Diffusion XL and the upsampler in GLIDE. The diffusion steps and guidance scale hyperparameters we used are shown in Table 12. For other hyperparameters, we follow the recommendations from Huggingface and GLIDE's official implementation. We use the prompt "An image of a class_name." as the text guidance to generate the image and interpolate the generated image to the size of the target dataset (32 for CIFAR, 64 for TinyImageNet, and 224 for ImageNet-100).

Samples from the generated datasets. Fig. 10 shows some samples from class 'n01558993' (Robin) in the SDXL-In100 dataset along with the samples from the original dataset. We can notice a significant loss of diversity in the samples from SDXL-In100.

D.3 Details about synthetic contamination simulation.

As mentioned in Sec. 3.2, we generate synthetic twins of benchmark datasets and substitute a fixed portion P of the original datasets with their synthetic counterparts. Similar to the class-wise distribution of Mix-C10/C100/Tiny, we also conduct the mixture class-wise. For datasets in setting a) , we assure that the contamination ratio in each class is also P . For datasets in setting b) , while maintaining a consistent class-wise contamination ratio, we also ensure that each individual synthetic model contributes 20% of the contamination in each class.

D.4 Task sequence.

In some work, the authors use a fixed task sequence for fair comparison. However, the final performance is largely affected by the task order. For fair comparison, we randomly assign the class to tasks and shuffle the sequence of tasks with 10 fixed random seeds. This ensures the evaluation is not biased to task difficulty.

D.5 Data augmentation.

Data augmentation is effective in boosting the training of online continual learners. Methods may benefit differently from different augmentation intensities, and some methods may favor simpler augmentations instead of complicated ones. For a fair comparison, it is vital to ensure all methods are in their optimal performance. Thus, we introduce two different augmentation strategies:

Partial strategy. The partial augmentation is a weaker version of augmentation, consisting of random cropping with p = 0 . 5 , followed by random horizontal flip with p = 0 . 5 .

Full strategy. The full augmentation strategy is a stronger version of augmentation. The full augmentation strategy is a superset of its partial counterpart, which consists of random cropping, random horizontal flip, color jitter, and random grayscale. The parameters for color jitter are set to (0 . 4 , 0 . 4 , 0 . 4 , 0 . 1) with p = 0 . 8 . The probability of random grayscale is set to p = 0 . 2 .

We define the data augmentation strategy of each method with a hyperparameter search, as detailed in Appendix D.6.

D.6 Hyperparameter search protocol.

For hyperparameters in all of the methods (except DER++ on TinyImageNet), we conduct a hyperparameter search on the clean CIFAR-100 dataset with a memory size of 5,000, and apply the same hyperparameter to all of the other settings. The exhaustive list of the hyperparameter search is shown in Table 13.

Special treatment for DER++. DER++ encounters a catastrophic performance defect (close to 0) when trained on the TinyImageNet dataset using an optimizer with momentum. Thus, we applied another hyperparameter search for DER++ on TinyImageNet and found the SGD optimizer (without Momentum) gives reasonable performance. All the experiments of DER++ on TinyImageNet are conducted using these new hyperparameters.

D.7 Hardware and computation.

All of the experiments are conducted on NVIDIA A100 GPUs. The average training time of each method on CIFAR-100 (Memory size = 5k), ImageNet-100 (Memory size = 5k), and TinyImageNet (Memory size = 10k) is shown in Fig. 11. The training efficiency is much faster than OCM and OnPro, while on par with the most efficient method.

D.8 Useful source code links.

For continual learning baselines, we use the codebase listed in Table 14 to reimplement baseline methods. For image generation methods, we use the Diffuser library from Hugging Face for Stable Diffusion and VQDM experiments, and we use the codebase in Table 14 for GLIDE.

CIFAR-100

Training Time (Minutes)

IN-100

Tiny-lmageNet

102

101

OCM

GSA

DER++

ER-ACE

OnPro

ESRM

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the main claims made in both the abstract and introduction can accurately reflect the paper's contribution.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation section is included in the appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The source code of our work, along with the result of the hyperparameter search are included in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is included in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The main experimental setting is included in the paper, and all of the details are included in the source code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are reported in the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Detailed information is included in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The Code of Ethics is fully respected and obeyed in our research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the used assets in the appendix. The licenses of existing assets are properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have a Readme file along with our source code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper  investigates the impact of AI-generated images on the performance of online continual learning (CL) models. It introduces a novel method called Entropy Selection with Real-synthetic similarity Maximization (ESRM) to mitigate the negative effects of synthetic data contamination. ESRM leverages entropy-based sample selection and a contrastive learning approach to align the feature embeddings of real and synthetic data, thereby enhancing the robustness of CL models against the degradation caused by synthetic data.\n\nSTRENGTHS:\n- The authors clearly articulate the problem, methodology, and results, enhancing the paper's accessibility and understanding for a broad readership.\n\n- The paper uniquely identifies the issue of synthetic data contamination in online continual learning, a significant challenge for the future of this field. The work has substantial implications for the ML community, offering a pioneering approach to maintaining the integrity of continual learning models in the presence of synthetic data.\n\n- This paper proposes ESRM, an innovative method that combines entropy selection and contrastive learning to mitigate the negative effects of synthetic data, demonstrating creativity in addressing this new problem.\n\n- The paper is underpinned by robust technical approaches, ensuring the quality and reliability of the proposed solution through comprehensive experimental validation.\n\nWEAKNESSES:\n- The paper evaluates the impact of synthetic data using a limited set of generative models. And the experiments are primarily conducted on image classification datasets.  Expanding this to include a broader range of tasks, particularly truly text-to-images, text-to-text, could strengthen the findings.\n\n- The method for generating synthetic datasets is straightforward, using simple prompts. Incorporating more complex and diverse prompts, potentially using large language models to simulate user queries, could better reflect real-world synthetic data contamination.\n\n- The reliance on entropy as a key metric for distinguishing real from synthetic data might not be universally applicable. Further exploration of this metric's effectiveness across different domains and its theoretical underpinnings could bolster the method's credibility.\n\n- While the paper provides insights into the method's effectiveness, there is limited discussion on its computational efficiency and scalability, especially when dealing with large-scale datasets or high contamination ratios. Addressing these aspects could be crucial for practical applications where computational resources and time are critical constraints.\n\nQUESTIONS:\nPlease refer to the weaknesses above.\n\nLIMITATIONS:\nThe authors have discussed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nImage generation has been showing promising achievements in the last few years, but the generative models may not be able to keep up with the distribution of the real samples. Due to low diversity, synthetic images perform poorly in downstream learning tasks. This paper tackles this problem by diversifying the sample selection using the proposed ESRM framework, which prioritizes the high-entropy samples in the experience replay memory with an RM loss function. The authors first performed experimental analysis to build their motivation in section 4 and proposed their solution to the problem in section 5. The experiments are diversified with several datasets, benchmarks, and extensive ablation studies.\n\nSTRENGTHS:\n1. The paper is nicely written, with clear language and sufficient details.\n\n3. Section 4's experimental analysis of existing CL methods clearly motivates the proposed model. \n\n4. The authors propose a novel loss function to address the issue of low-entropy samples in CL. This approach builds upon existing supervised constructive loss techniques. Their method aims to maximize the cosine similarity between embeddings of real and synthetic samples while simultaneously minimizing the impact of low-entropy samples on the learning process.\n\n4. In the ablation study, the authors studied different components of the proposed framework that show the superiority of the overall ESRM.\n\nWEAKNESSES:\n1. The paper only deals with diffusion generative models. I understand that diffusion models are the most powerful image generators but it would be good to experiment other generative models.\n\n2. The accuracy scores are overall low in CL. Does the proposed method \n\n3. Why is a fixed 50% dropping rate selected for ES? \n\n4. Downstream ML tasks have been explored with synthetic data to validate synthetic datasets, and related works clearly miss that angle. The authors should also discuss them and maybe add some of them to benchmarks.\n\nQUESTIONS:\nAuthors can refer to the weaknesses section for major questions. Here, I listed minor points for authors reference:\n\n1. It seems there is a typo in the line 231. \n2. It could be good to highlight the best results in Tables 3 and 4, the same as Table 2.\n\nLIMITATIONS:\nI agree with the limitation the authors brought up.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
This paper investigates the negative impact of synthetic data contamination on existing online continual learning methods. An entropy selection with the real-synthetic similarity maximization method is proposed to alleviate the performance deterioration.

STRENGTHS:
1. Detailed analysis of synthetic data contamination and its influence on continual learning.
2. This paper is technically clear and easy to follow.

WEAKNESSES:
1. The creation of simulated data is the cornerstone of this research, but there is a lack of detailed explanation on how to generate these data in the main paper.
2. For Observation 4 "With the limited diversity of synthetic data", how about mixing the synthetic data from different generation models to increase the diversity, since the synthetic images on the internet also form different models? While the observation remains unchanged in this case?
3. In order to simulate real data collection more realistically, in addition to synthetic data, new open-domain real data should also be incorporated.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
The limitations are discussed while the potential negative societal impact is not discussed. However, for this work, I think it is not necessary to discuss this.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15