Title: A Minimalistic Unified Framework for Incremental Learning across Image Restoration Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-task image restoration

new baseline
Abstract: Existing research in low-level vision has shifted its focus from "one-by-one" task-specific methods to "all-in-one" multi-task unified architectures. However, current all-in-one image restoration approaches primarily aim to improve overall performance across a limited number of tasks. In contrast, how to incrementally add new image restoration capabilities on top of an existing model — that is, task-incremental learning — has been largely unexplored. To fill this research gap, we propose a minimalistic and universal paradigm for task-incremental learning called MINI. It addresses the problem of parameter interference across different tasks through a simple yet effective mechanism, enabling nearly forgetting-free task-incremental learning. Specifically, we design a special meta-convolution called MINI-Conv, which generates parameters solely through lightweight embeddings instead of complex convolutional networks or MLPs. This not only significantly reduces the number of parameters and computational overhead but also achieves complete parameter isolation across different tasks. Moreover, MINI-Conv can be seamlessly integrated as a plug-and-play replacement for any convolutional layer within existing backbone networks, endowing them with incremental learning capabilities. Therefore, our method is highly generalizable. Finally, we demonstrate that our method achieves state-of-the-art performance compared to existing incremental learning approaches across five common image restoration tasks. Moreover, the near forgetting-free nature of our method makes it highly competitive even against all-in-one image restoration methods trained in a full-supervised manner. Our code is available at https://github.com.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose MINI (Minimalistic Incremental Network for Image Restoration), a novel and lightweight framework tailored for task-incremental learning across various image restoration tasks. To enable effective hard parameter isolation, they introduce MINIconv, along with a lightweight degradation query module and an embedding consistency regularization (ECR) strategy.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. This work is the first attempt to explore task-incremental learning in the field of image restoration.\n2. The idea is performed well.\n3. The paper is well-organized and easy to read.\n4. The limitations are clearly and well mentioned.\n\n[Weaknesses]\n1. In line 258, the authors state that \"its use of hard parameter isolation causes the total number of parameters to scale by a factor of T.\" If this is the case, isn’t there a significant difference in terms of forward computation overhead or the number of parameters compared to multi-head methods?\n\n2. In the same context as the previous question, I am curious about the performance differences and differences in parameter counts between MINI and a scenario where a learned query function selects task-specific (one-for-one) methods for each task.\n\n3. What is the performance when only the Degradation Classification MLP is used in the query function, excluding the Global MLP and Local MLP?\n\n4. I am curious about the actual number of parameters in each case listed in Table 2. Additionally, what is the parameter count and inference time of the MLP-based query function? \n\n5. Minor error.\n-- L17: overll >> overall\n-- Table 2: It would be visually more effective to use color for emphasis rather than bold text.\n\nQUESTIONS:\nPlease refer to the weaknesses section. My current rating could change significantly depending on the authors’ responses. While I have some knowledge of image restoration, I do not have specialized expertise in the specific field of this study (e.g. meta-convolution, All-in-One IR, ...). I am aware of this and will actively refer to other reviewers’ questions and the authors’ responses as well.\n\nLIMITATIONS:\nThe limitations and future work are clearly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes MINI, a minimalistic framework for task-incremental learning in image restoration, addressing the challenge of catastrophic forgetting and parameter conflict in existing multi-task models. MINI introduces a novel Meta Convolution module (MINIconv) that generates task-specific parameters via lightweight embedding pools, enabling complete parameter isolation across tasks without additional computational overhead. This design allows seamless integration into existing backbones, transforming them into incremental learning systems. A degradation query mechanism classifies input degradations to select appropriate embeddings, while embedding-consistency regularization enhances robustness against misclassification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.This work represents the first systematic attempt to explore task-incremental learning in the field of image restoration.\n\n2.The proposed architecture is highly minimalistic, requiring only the replacement of standard convolutional layers with MINIconv and the addition of a lightweight degradation query module. It can be seamlessly integrated into existing mainstream backbone networks.\n\n3.The method enables efficient and nearly forgetting-free task-incremental learning, while avoiding the parameter explosion issues typical of multi-head approaches.\n\n4.The introduction of Embedding Consistency Regularization (ECR) effectively enhances the system's robustness and fault tolerance, particularly when the degradation classification is inaccurate.\n \nWeaknesses:\n1.The maximum number of tasks supported is inherently constrained by the pre-defined size of the embedding pool (T), making it difficult to accommodate scenarios where the total number of tasks is unknown or grows dynamically.\n\n2.Although the method avoids additional inference-time computation, hard parameter isolation causes the total number of model parameters to grow linearly with the number of tasks, which may present challenges for deployment in resource-constrained environments.\n\n3.The current work primarily focuses on CNN-based architectures, and its generalization to other popular frameworks such as Vision Transformers or Diffusion Models remains unexplored.\n\nQUESTIONS:\nQ1:The MINI architecture requires pre-defining the embedding pool size T to accommodate different tasks. However, real-world applications may face an unknown or unbounded number of tasks. Can the authors clarify whether MINI can dynamically extend T post-deployment? If not, what practical upper bound do the authors recommend for T considering memory and computational trade-offs?\n\nQ2:The degradation query module relies on accurate task classification. While the paper discusses robustness via ECR, in practice, real-world degradation can be ambiguous or unknown. Can the authors elaborate on MINI's performance when encountering degradations outside the predefined set?\n\nQ3:The paper mentions potential future extensions to Transformer-based backbones but lacks preliminary results or analysis. Can the authors provide insights or experiments indicating MINI's compatibility or limitations with Transformer or hybrid architectures?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of task-incremental learning for image restoration, where models struggle to sequentially add new capabilities without catastrophic forgetting. The authors propose a lightweight framework MINI built around MINIconv, a novel meta-convolution module that replaces standard convolutions in existing backbones. MINIconv uses task-specific embedding pools to achieve hard parameter isolation: each task exclusively activates its own embeddings during inference, preventing interference with prior tasks. This design requires no complex meta-networks, introduces negligible computational overhead during inference, and enables near-forgetting-free incremental learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper systematically address task-incremental learning for image restoration (IR), filling a critical gap between \"one-by-one\" and \"all-in-one\" paradigms.\n2. MINI mechanism replaces complex meta-networks with lightweight embedding pools, achieving hard parameter isolation without inference overhead. The design is simple yet effective.\n3. Plug-and-play compatibility with existing IR backbones (e.g., NAFNet, Restormer) significantly boosts practicality. Experiments show consistent multi-task performance gains.\n4. Extensive experiments across 5 diverse IR tasks (deraining, dehazing, deblurring, etc.) using multiple backbones.\n\nWeakness:\n1. Potential limited task scalability: The fixed embedding pool size T requires predefining the maximum number of tasks, which may be impractical for open-ended incremental learning.\n2. Despite ECR, performance still depends on the query module’s accuracy. The paper reports a 6% misclassification rate, and ECR only mitigates, not eliminates errors. Meanwhile, the lightweight query design (global/local branches) may struggle with complex hybrid degradations (e.g., rainy + hazy images).\n3. Non-convolutional components (e.g., LayerNorm) are not adapted to MINI’s isolation paradigm. The authors note this has \"limited impact,\" but no data supports this claim.\n4. For experiments, this paper lacks comparison against prompt-based continual learning methods (e.g., L2P, DualPrompt), which also tackle task-incremental settings. The \"all-in-one\" joint training baselines (Table 2) are not state-of-the-art (e.g., missing recent works like PromptIR).\n\nQUESTIONS:\n1. How would MINI scale to open-ended task sequences (e.g., >10 tasks) without retraining? Could you propose strategies (e.g., embedding pruning/generation) to dynamically expand capacity while preserving efficiency?\n2. How does MINI handle hybrid degradations (e.g., rainy + hazy images)? Could you test on blended datasets (e.g., RainCityscapes) or modify the query for multi-label classification?\n3. Could you ablate MINI’s performance when these layers are made task-specific versus shared? Would adapting them (e.g., via embeddings) further boost multi-task gains?\n4. Can MINIConv integrate into transformer backbones (e.g., Restormer)? Does replacing MHSA/MLP weights with embeddings require architectural changes? Please discuss feasibility.\n5. Why omit comparisons against prompt-based continual learning (e.g., L2P, DualPrompt) that also tackle task-incremental settings? Could you add these to Table 3 to clarify MINI’s advantages over parameter-efficient alternatives?\n\nLIMITATIONS:\nPlease see the Weakness and Question parts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an incremental learning paradigm for image restoration tasks, which I think is meaningful. But I remain skeptical about the novelty of the method proposed in this paper. However, despite this, since incremental learning has a wide range of application scenarios in the field of image restoration, especially in the real-world scenes, I think this paper can reach the acceptance line of NeurIPS after reasonable discussion and modification.\n\nSTRENGTHS AND WEAKNESSES:\nThe outstanding contribution of this article is the introduction of incremental learning for image restoration tasks. I will mainly list here what I think are the weaknesses of this paper:\n\n1. The MINI-Conv is very similar to the Dynamic Conv [1] and also the Prompt Block in PromptIR [2]. Perhaps the authors could discuss the difference between these two?\n2. I am quite confused by the comparative experimental setup of this paper. Taking Table 2 as an example, this paper compares the all-in-one (joint learning) method with the incremental learning-based method. The all-in-one method trains on five degradation jointly, and the training process is blind. However, the incremental learning based method proposed in this paper can get the correct degradation type embedding during training, which is not blind. Will this difference affect the final restoration performance?\n3. Why not use the image restoration network itself as the degradation classifier, such as DCPT [3]? Is an external degradation classifier necessary? As I understand, MINI-Conv requires additional degradation classification during inference. If the degradation classification result is wrong, can the MINI-Conv-based method still output a clean image?\n4. I am very interested in the ablation experiment results of the Task sequence order. Why do different Task sequence orders lead to different results? The PSNR difference between 1→2→3→4→5 and 1→3→2→5→4 is 0.2 dB, which cannot be called \"little impact\".\n5. Does the data imbalance problem affect the effect of incremental learning? I noticed that the proposed method trains 400 epochs for each dataset. If this training epoch is redistributed according to the actual data volume of the dataset, can better results be achieved? For example, train more epochs for the small dataset (LoL) and train fewer epochs for the large dataset (RESIDE).\n6. For networks that do not use convolution, such as SwinIR, is MINI-Conv still useful?\n7. Is MINI-Conv applicable to mixed degradation or more degradation types? Can MINI-Conv be generalized to real degradation scenarios?\n\n[1] Dynamic Convolution: Attention over Convolution Kernels. CVPR 2020.\n\n[2] PromptIR: Prompting for All-in-One Blind Image Restoration. NeurIPS 2023.\n\n[3] DCPT: Universal Image Restoration Pre-training via Degradation Classification. ICLR 2025.\n\nQUESTIONS:\nPlease see `Strengths And Weaknesses`.\n\nLIMITATIONS:\nPlease see `Strengths And Weaknesses`.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}