Title: Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual learning

Model inversion

Vision-Language models.
Abstract: Continual learning (CL) aims to incrementally train a model to a sequence of tasks while maintaining performance on previously seen ones. Despite effectiveness in mitigating forgetting, data storage and replay may be infeasible due to privacy or security constraints, and are impractical or unavailable for arbitrary pre-trained models. Data-free or examplar-free CL aims to continually update models with new
tasks without storing previous data. In addition to regularizing updates, we employ model inversion to synthesize data from the trained model, anchoring learned knowledge through replay without retaining old data. However, model inversion in predictive models faces two key challenges. First, generating inputs (e.g., images) solely from highly compressed output labels (e.g., classes) often causes drift between synthetic and real data. Replaying on such synthetic data can contaminate and erode knowledge learned from real data, further degrading inversion quality over time. Second, performing inversion is usually computationally expensive, as each iteration requires backpropagation through the entire model and many steps are needed for convergence. These problems are more severe with large pre-trained models such as Contrastive Language-Image Pre-training (CLIP) models. To improve model inversion efficiency, we propose Per-layer Model Inversion (PMI) approach inspired by the faster convergence of single-layer optimization. The inputs optimized from PMI provide strong initialization for full-model inversion, significantly reducing the number of iterations required for convergence. To address feature distribution shift, we model class-wise feature distribution using a Gaussian distribution and preserve distributional information with a contrastive model. Sampling features for inversion ensures alignment between synthetic and real feature distributions. Combining PMI and feature modeling, we demonstrate the feasibility of incrementally training models on new classes by generating data from pseudo image features mapped through semantic-aware feature projection. Our method shows strong effectiveness and compatibility across multiple CL settings.


Peer reviews:
comments: SUMMARY:
This paper proposes PMI, a method designed to improve model inversion by providing better initialization, thereby accelerating the inversion process. In addition, the authors introduce CFS, a sampling strategy that selects informative and diverse class-wise features from Gaussian distributions. Experiments conducted on both ResNet and CLIP models demonstrate the effectiveness of the proposed methods and validate the claimed improvements.

STRENGTHS AND WEAKNESSES:
Pros:

(1) The paper is clearly written, and the proposed method effectively addresses the stated motivation.

(2) Leveraging model inversion to generate data for unseen classes is a novel and potentially promising direction in the field of continual learning.

(3) The proposed method demonstrates satisfactory improvements over existing baselines.

Cons:

(1) The comparison of inversion time cost is not sufficiently comprehensive. Table 8 only reports results for the Full-model and SMI. Including additional baselines would better justify the claimed efficiency of PMI. Moreover, the number of inversion iterations should be controlled across methods to fairly assess performance under limited iterations.

(2) While PMI accelerates inversion via better initialization, methods like ABD and DCMI use random-initialized generators to improve speed and reduce memory usage. A comparison with these methods in terms of inversion time and GPU efficiency is missing. Discussing the differences between image-noise-based and generator-based approaches would better support the motivation.

QUESTIONS:
see weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed two techniques to improve the efficiency and effectiveness of the model inversion-based method in data-free continual learning. Specifically, they leverage a Per-layer Model Inversion (PMI) approach inspired by the faster convergence of single-layer optimization, which reduces the number of iterations required for convergence. Moreover, they model class-wise feature distribution using a Gaussian distribution and sampling features from them for inversion. By combining PMI and feature modeling, the proposed method demonstrates efficiency and effectiveness in multiple incremental settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Comprehensive experiments have been conducted to verify the effectiveness and efficiency of the proposed method (different incremental settings including ResNet- and CLIP-based CIL, ablation studies, etc.).\n2. Although the work follows the branch of model inversion-based data-free continual learning, which is not as vibrant as Non-Exemplar CIL probably due to the extra time for synthesizing images and rehearsal, the proposed techniques have certain novelty. \n3. The paper is generally well written with a clear logic flow. The methodology and the findings are presented in a well-structured manner.\n\nWeaknesses:\n1. Since synthesizing images via model inversion may incur much time, a runtime analysis (comparison with other methods such as ABD, and R-DFCIL) should better be included. In Sec. F of the supplementary, there is a runtime analysis that does not cover the aforementioned methods.\n2. Although not strongly related to model inversion, recent works in Non-Exemplar CIL should be discussed due to its similar data-free formulation (e.g. [A1][A2]).\n[A1] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, Zheng-Jun Zha. Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. CVPR 2022\n[A2] Wuxuan Shi, Mang Ye. Prototype Reminiscence and Augmented Asymmetric Knowledge Aggregation for Non-Exemplar Class-Incremental Learning. ICCV 2023\n\nOthers that may not affect the rating:\n1. “ci” and “cj” are not clear indices for classes. Better use one character.\n2. In Fig. 1, the order of the steps is arranged from right to left, which affects the readability.\n3. In Table 4, a value of “-0.15” is in bold, which is not supposed to be that way according to my understanding.\n4. Result tables use different types of visualization to highlight the best performance. Better use the same.\n\nQUESTIONS:\nApart from the concerns in the “Weakness” part, In Eq. (6), there is no temperature \\tau inside the \\exp function as the contrastive loss. I was wondering whether it may affect the convergence.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to improve data-free continual learning based on model inversion. It tackles two key challenges: the high computational cost of inversion and the feature drift between synthetic data coming from model inversion and real data. To improve efficiency, the authors introduce Per-layer Model Inversion (PMI), which generates a strong initialization for the inversion process by optimizing layer-by-layer, thus reducing the total number of required iterations. To mitigate feature drift, they model the class-wise feature distributions using Gaussian and contrastive models to guide the generation of more realistic synthetic samples. The paper uses CLIP as the backbone, using the original pre-trained model as a generator to create synthetic training data for classes that are new to the continual learning task.\n\nSTRENGTHS AND WEAKNESSES:\nWeaknesses:\n- The paper argues that the problems of data-free continual learning (i.e. synthetic / real drift + expensive model inversion) are even more severe in models like CLIP (line 14). This is not clear to me why that would be the case. Why would CL be worse in CLIP models compared to any other model ? There's no empirical nor theoretical argument given by the authors and so the motivation to work with CLIP is weak.\nAlso CLIP is already a powerful general-purpose model that has seen data similar to imagenet or CIFAR during its pretraining. It already has some knowledge about the \"unseen\" categories (as we can see from the strong zero-shot capabilities) and so we cannot really say that these categories are \"unseen\".\n\n- Experimental results are not very convincing. The results shown in Table 1 for example are only marginally above the results from R-DFCIL, which is a method which is 3 years old.\n\n- To generate new synthetic data for replay, the method uses the original, frozen pre-trained model. This is less an \"inversion\" of the learning model and more akin to using a powerful, external generative model. This setup blurs the lines of the contribution and deviates from the core data-free continual learning problem where knowledge should be extracted from the model being trained. If one can use the original frozen CLIP, why not leveraging any other foundation models (i.e. text to image generative models) to generate data for continual learning ?\n\n- The method's reliance on storing \"input statistics from real data\" for each layer fundamentally challenges its \"data-free\" claim. The assumption that one cannot store raw data but can compute and store detailed, layer-wise statistics derived from it seems unrealistic. Can the authors comment on this ? Is there a way to make the method work without this assumption ?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper addresses the inefficiency issue of model inversion for data-free continual learning. It proposes a PMI technique to efficiently generate synthetic data by performing model inversion layer by layer, thus reducing computational cost and improving convergence. To mitigate feature drift between synthetic and real data, they model class-wise feature distributions and use these for sampling features in inversion. Experimental results on some datasets validate the effectiveness of the method, which improves the classification accuracy and training efficiency.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	The proposed PMI presents an elegant idea of decomposing the inversion process, leading to faster convergence and lower computational cost, which is also justified theoretically and empirically.

2.	The combination of Gaussian modeling and contrastive feature selection improves the quality of synthetic data.

Weaknesses:

1.	Though PMI reduces the number of update steps, the extra step of per-layer inversion adds overhead. It will be better to compare the training time of the proposed method with baselines.

2.	The effectiveness of CFS seems minor, as shown in Table 1.

3.	The limitations of the proposed method are not discussed comprehensively.

QUESTIONS:
See wesknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03