Rainbow Generator: Generating Diverse Data for Name Only Continual Learning

Authors: Minhyuk Seo, Seongwon Cho, Minjae Lee, Diganta Misra, Hyeonbeom Choi, Seon Joo Kim, Jonghyun Choi
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual learning, generative model

Abstract:
Requiring extensive human supervision is often impractical for continual learning due to its cost, leading to the emergence of ‘name-only continual learning’ that only provides the name of new concepts (e.g., classes) without providing supervised samples. To address the task, recent approach uses web-scraped data but results in issues such as data imbalance, copyright, and privacy concerns. To overcome the limitations of both human supervision and webly supervision, we propose Generative name only Continual Learning (GenCL) using generative models for the name only continual learning. But naïve application of generative models results in limited diversity of generated data. So, we specifically propose a diverse prompt generation method, HIerarchical Recurrent Prompt Generation (HIRPG) as well as COmplexity-NAvigating eNsembler (CONAN) that selects samples with minimal overlap from multiple generative models. We empirically validate that the proposed GenCL outperforms prior arts, even a model trained with fully supervised data, in various tasks including image recognition and multi-modal visual reasoning. Data generated by GenCL is available at https://anonymous.4open.science/r/name-only-continual-E079.

Decision: reject
Reviewer scores: 6, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730239843885

thread_type:
official_review

<thread_object>
creator: reviewer_z3yn

<content>
summary:
This paper presents a synthetic data augmentation approach for name-only continual learning, eliminating the need for web scraping. The authors propose diverse prompt generation and a post-generation sample selection method to create a more informative coreset for new concepts. The resulting dataset achieves state-of-the-art performance in both the class continual learning setting and multimodal visual recognition tasks.

strengths:
The paper is well-written and easy to follow. The authors have done an excellent job in presenting the figure and explaining the proposed method. Additionally, the experimental ablation study (both in the main paper and in the appendix) is extensive, providing a thorough analysis.

weaknesses:
1. The claim (figure 1 and line 188) that synthetic data generation does not have privacy issues is debatable. Text-to-Image generative models can suffer from memorization issues, potentially reproducing its own training data, which also violates privacy. Can you address this potential limitation in the paper and discuss any measures you could take to mitigate privacy risks associated with generative models.

2. The baselines discussed in the paper are not sufficient. Numerous synthetic data augmentation methods for image classification, like [1] and [2], share similar techniques such as prompting language models, text-to-image generation, and ensemble/filtering approaches. These methods, which can be easily adapted to the name-only continual learning setting, should be compared to evaluate if the proposed method offers any unique advantages for the name-only continual learning setting. I request that the authors either include comparisons with these methods or provide a detailed explanation of why such comparisons would not be appropriate for their specific setting. It's unclear whether this name-only continual learning setting is truly distinct from normal image classification with an expanding label space, and whether it requires a dedicated solution. 

3. The MVCIL setup is a little strange, as the LLM pretraining would likely have exposed it to the new concepts and do not fully align with the continual learning setting (where the introduced concepts need to be disjoint from the previous training data). It is perhaps a different setting rather than continual learning.  

I'll be willing to raise the score if authors provide satisfactory response.

[1] https://arxiv.org/abs/2310.10402
[2] https://arxiv.org/pdf/2302.14051

questions:
1. Can you discuss what is the sample efficiency for your proposed method? Can you report the number of synthetic data samples generated and the GPU time required for generation and training? Without these details, it is hard to assess the method's sample efficiency and computational requirements. 

2. Instead of CIL, an alternative approach to address name-only continual learning could be continual open-world representation learning [3]. This involves training a generalizable embedding model in a continual learning fashion. During testing, labels can be generated by retrieving them from a gallery that associates images with their corresponding labels, leveraging the learned representations. Would the proposed method still work in this setting? 

[3] https://arxiv.org/pdf/2409.05312
</content>
</thread_object>

</thread>
<thread>
creation date:
1730571672684

thread_type:
official_review

<thread_object>
creator: reviewer_cbvu

<content>
summary:
This paper introduces GenCL, a generative approach to address the challenge of continual learning with minimal reliance on labeled data. The authors propose a hierarchical recursive prompt generation (HIRPG) method to create diverse prompts, enabling generative models to produce varied and representative samples that cover a broad range of visual styles and contexts. Additionally, they introduce a complexity navigation method (CONAN) that selects the most representative samples from the generated data based on sample complexity. These samples are then used to train a continual learning model on class-incremental learning (CIL) and multi-modal visual-concept incremental learning (MVCIL) tasks. The authors evaluate GenCL on standard domain generalization datasets like PACS and DomainNet, demonstrating superior performance in both in-domain (ID) and out-of-domain (OOD) scenarios compared to other methods. The results highlight GenCL’s ability to generalize to new tasks while mitigating catastrophic forgetting.

strengths:
1. The paper introduces a novel approach to continual learning that minimizes reliance on labeled data through effective generative data augmentation.

2. HIRPG and CONAN methods enhance data diversity and representative sample selection, contributing to improved model generalization in incremental learning.

3. Experimental results demonstrate GenCL’s superior performance in both in-domain and out-of-domain tasks, highlighting its robustness and practical applicability.

weaknesses:
1. The choice of the K-ary tree structure in the HIRPG method appears arbitrary, lacking theoretical or empirical justification. In the paper, K is set to 7 and the depth D to 2, yet there is no discussion on how these hyperparameters affect the quality of generated prompts and, consequently, the diversity of generated data. Without exploring the impact of different values for K and D, it’s unclear if these settings are optimal or if they generalize well across different tasks. This lack of analysis raises concerns about the adaptability and robustness of the proposed method in other continual learning scenarios where different prompt structures may be required.

2. The paper says that "overlap between generated prompts from different nodes is rare," but it doesn’t give any theoretical reasoning or experimental proof for this. 

3. The paper doesn’t provide a hyperparameter analysis for the parameters in Equation 4, which would help clarify how these settings impact experimental results and generalization. A sensitivity analysis on these hyperparameters would strengthen the claims of robustness and adaptability across tasks. Additionally, visualizing real sample features to demonstrate the effectiveness of Relative Mahalanobis Distance (RMD) could provide clearer evidence of RMD’s role in selecting complex samples.

questions:
For detailed questions and suggestions, please refer to the Weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730637685205

thread_type:
official_review

<thread_object>
creator: reviewer_qykq

<content>
summary:
This paper tackles the continual learning with the idea of data generation. Two additional techniques are proposed. The first idea is to generate diverse samples with a hierarchical prompt generation approach. The second idea is to select samples from multiple generative models with minimal overlap. The experiments are conducted on domain generalization and multi-modal visual reasoning benchmarks.

strengths:
The idea of using generative models for generating useful samples for downstream tasks is interesting. It is important to explore the way of generating diverse samples with diverse prompts and generative models. Generally, it is quite easy to follow the main idea of the paper.

weaknesses:
(1) The evaluation of CIL setup is very limited to a few classes (7 classes for PACS). The conventional CIL setup considered more classes (e.g. ImageNet with 1000 classes), It seems unclear how the proposed method behave when it the data scales up, which can be evaluated on CIFAR-100 and ImageNet-1K.

(2) There is no discussion of the computational cost or extra resources needed for the proposed method, although the performance is better than the competing baselines. It would be good to include a section to discuss the computational cost, such as training time, storage usage, extra resources for generating images.

(3) What about the classes of fine-grained species, such of different breeds of dogs or different kinds of birds. Does it still work with the proposed framework? It can be evaluated on fine-grained benchmarks, such as CUB, Flowers and Cars.

(4) I agree that the data generation is crucial for CL but it is difficult to scale up in its current form with text-to-image generative models, it would be good to discuss the limitations of proposed method and possible solutions.

questions:
(1) The evaluation is based on pre-trained ImageNet-1k ViT-Base models, can you explain why this is the case because there are other settings for CIL?

(2) Why for the results on Tab. 3, a CLIP-pretrained ResNet-50 is used, which is different from other results?

(3) Regarding copyright and privacy concerns, I am wondering if it exists as well for generative models used in the paper? 

(4) Why difficult samples are preferred for section 4.3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702729433

thread_type:
official_review

<thread_object>
creator: reviewer_sa3c

<content>
summary:
This paper introduces the Generative name-only Continual Learning (GenCL) framework, designed to enhance CL but without requiring manually annotated data. The GenCL framework take care of generating diverse prompts and managing diverse data generated from multiple models. GenCL addresses privacy concerns, costs, and limitations associated with manually curated and web-scraped data. Empirical results demonstrate its effectiveness across several image recognition and multi-modal visual reasoning tasks.

strengths:
The main strength of this work lies in the following presepectives:
1) It introduces an interesting and innovative approach to use generative models in name-only continual learning as a novel solution to limitations associated with labeled data which typically quite expansive in practice.
2) The introduce two techniques, HIRPG and CONAN, are important components for enhancing intra- and inter-diversity, which are essential for effective continual learning, I believe it is could also be useful in other type of learning problems.
3) The experiments are extensive and comprehensive, covering in-distribution and out-of-distribution evaluations, with comparisons against various baselines, shows the effectiveness and robustness of the proposed method.

weaknesses:
1) Though the proposed method achieved promising results, it still have a large gap compared to manually annotated data. A more practical scenario worth exploring is how GenCL would perform if supplemented with a limited number (1–2) of manually annotated examples. 
2) The proposed method still relies heavily on generative models that were trained on extensive datasets, potentially including names and concepts from the test data used in this paper. It remains unclear how the model would perform in domains entirely disjoint from any related training data, such as in the medical domain.

questions:
1) Are there any potential biases in the generated data? How might these biases impact continual learning?
2) Have you used any mechanism to select hard negative examples and how this selection impacts model performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732472730443

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely thank the reviewers for their valuable feedback and encouraging comments including  interesting and innovative approach (**sA3C, cBvu**), applicability (**sA3C**), useful approach (**QykQ**), extensive and comprehensive experiments (**sA3C, cBvu, Z3yn**), and clear presentation (**QykQ, Z3yn**).   

We have uploaded the first revision of the manuscript (changes are highlighted by red color).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces the generative name-only continual learning setting, where generative models are used to create data for training based on label names. The key methods introduced are how to generate appropriate prompts such that they are diverse, as well as selection of the data from multiple prompts/models. While the reviewers appreciated the paper's overall clarity of the idea and experimental results, a number of strong concerns were raised. First, the usage of such generative models that have been trained on a large amount of prior data raise many concerns including potential biases (sA3C), inability to address truly novel categories that are not in-distribution in the sense of new categories or fine-grained (or heavily covered) (sA3C, QykQ, Z3yn), existence of biases (sA3C), and unclear advantages in terms of privacy/copyright (sA3C, Z3yn, QykQ). There were other concerns about the method itself including computational cost, ad-hoc nature of the method, and importantly lack of thorough experimental results covering in-distribution, out-of-distribution, fine-grained, and truly novel categories as well as comparison to well-known state-of-art baselines such as Internet Explorer (IE), which the paper is situated against in the main figure. While the rebuttal provided a number of results including ImageNet-1K, fine-grained datasets, and comparisons to methods such as IE, three of the reviewers overall recommended not accepting this paper due to these limitations. 

  After reviewing all of the materials, including the paper, reviews, rebuttals, and discussion, I agree with the reviewers that significant effort is needed to improve the paper and do not recommend acceptance at this time. While the idea is certainly interesting, the large amount of both practical and conceptual issues that were raised cannot be addressed without significant revision. For example, methods such as IE should have been compared to across all datasets (including ImageNet, etc.) and this is not possible to thoroughly do in the rebuttal period. Further, the underlying idea of using generative models raises a number of concerns including whether the categories are truly "new categories"; this does not in and of itself mean that such a method cannot be accepted, but there should be thorough experiments demonstrating exactly where such methods work and don't work (e.g. medical imagery as mentioned by the authors) to really understand the level of generalization involved. I encourage the authors to significantly address all of the reviewers concerns for a future resubmission.