A Minimalistic Unified Framework for Incremental Learning across Image Restoration Tasks
Authors: Xiaoxuan Gong
Jie Ma
Venue: NeurIPS 2025 poster ()
Keywords: multi-task image restoration
new baseline
Primary area: applications
TL;DR: A Minimalistic multi-task image restoration architecture based on adaptive convolutional kernels, well-suited for task-incremental learning.

Abstract: Existing research in low-level vision has shifted its focus from "one-by-one" task-specific methods to "all-in-one" multi-task unified architectures. However, current all-in-one image restoration approaches primarily aim to improve overall performance across a limited number of tasks. In contrast, how to incrementally add new image restoration capabilities on top of an existing model — that is, task-incremental learning — has been largely unexplored. To fill this research gap, we propose a minimalistic and universal paradigm for task-incremental learning called MINI. It addresses the problem of parameter interference across different tasks through a simple yet effective mechanism, enabling nearly forgetting-free task-incremental learning. Specifically, we design a special meta-convolution called MINI-Conv, which generates parameters solely through lightweight embeddings instead of complex convolutional networks or MLPs. This not only significantly reduces the number of parameters and computational overhead but also achieves complete parameter isolation across different tasks. Moreover, MINI-Conv can be seamlessly integrated as a plug-and-play replacement for any convolutional layer within existing backbone networks, endowing them with incremental learning capabilities. Therefore, our method is highly generalizable. Finally, we demonstrate that our method achieves state-of-the-art performance compared to existing incremental learning approaches across five common image restoration tasks. Moreover, the near forgetting-free nature of our method makes it highly competitive even against all-in-one image restoration methods trained in a full-supervised manner. Our code is available at https://github.com.

A Minimalistic Unified Framework for Incremental Learning across Image Restoration Tasks
Xiaoxuan Gong
School of Artificial Intelligence and Automation, HUST Huazhong University Science and Technology
Wuhan, China
gongxiaaxuan22860gmaa com
Jie Ma*
School of Artificial Intelligence and Automation, HUST
Huazhong University Science and Technology Wuhan, China majie edu. cn
Abstract
Existing research in low-level vision has shifted its focus from "one-by-one" task-specific methods "all-in-one" multi-task unified architectures. However, current all-in-one image restoration approaches primarily aim to improve overall performance across limited number of tasks. In contrast, how incrementally add new image restoration capabilities incremental learning has been largely unexplored. To fill this research gap, we propose minimalistic and universal paradigm for nask-incremental learning called MINI. addresses the problem of parameter interference across different tasks through a simple yet effective mechanism, enabling nearly forgetting- free task- incremental learning. Specifically, we design special meta-convolution called MINIconv, which generates parameters solely through lightweight embeddings instead complex convolutional networks or MLPs. This not only significantly reduces the number of parameters and computational overhead but also achieves complete parameter isolation across different tasks. Moreover, MINIconv can be seamlessly integrated as plug- aagg-and-payy replacement for any convolutional layer within existing backbone networks, endowing them with incremental learning capabilities and boosting their multi-task overall performance. Therefore, method highly generalizable. Finally, we demonstrate that our method achieves state-of-the-arr performance compared to existing incremental learning approaches across five common image restoration tasks. Moreover, the near forgetting- free nature of our method makes it highly competitive even against all-in-one image restoration methods trained under joint learning. Our code is available at hhtts: //github.com
top of an existing model - that is, task-
Introduction
The core challenge of all- All-in-one(AAO) image restoration lies in the need to accomplish diverse feature extraction and image manipulation tasks using a single set of fixed parameters, which inevitably leads to parameter conflicts between different tasks. To alleviate this issue, most existing methods introduce additional information (prompts) to guide the model in handling differennt types of degradations (as Figure Ta), such
Alternatively,
recent trend is to leverage the priors from large-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

cetwork Jayer
weight
Selective mechanism
promnota prices
(a) prompt/rrio- based
(b) multi head
(c) ours
Illustration of different multi-task image restoration paradigms. Colors indicate distinct image restoration tasks such as deraining, dehazing, and raindrop removal etc. (a) prompt/prior-base methods: although additional prompt information provided, the parameters are isolated-shared shared weights are still responsible for multiple tasks; (b) muiti-head methods: parameter isolation is achieved by parallelizing multiple task-specific subnetworks, but the overall structure becomes bloated and difficult to deploy; (c) MINI: parameter isolation is implemented at each layer within a single network, resulting lightweight architecture that easier to deploy and transfer.
Figure 1:
scale models as guidance, such as 6 10 Although these methods have achieved impressive results, they do not fundamentally address the issue of parameter conflict across different tasks. As result, they often suffer from significant performance degradation compared to task-specific models. Moreover, when performing incremental learning over new tasks, these methods are prone to severe catastrophic forgetting. leading unacceptable overall performance. This demonstrates that parameter conflict is shared bottleneck for both all-in-one image restoration and task-incremental learning.
To address the parameter conflict issue, a natural idea is to train multiple models, each responsible for different task, and then select different pathways through some selection mechanism, thereby achieving parameter isolation across tasks (as Figure However, such multi-head approach means that complete new pathway must be added for each new task, resulting in increased parameters and computational overhead. In addition, some eggularization-basee approaches have been proposed in incremental learning to alleviate catastrophic forgetting [I] 12 [3]. However, these methods are often limited in effectively addressing parameter conflicts. A more elegant solution is to employ dynamic parameters within single model instead of fixed ones, allowing the parameters adapt based on the model input. Such approaches are known as dynamic convolution or meta convolution 17. These methods aim to use a small network, referred to a meta-network, to generate the convolutional weights of the main network, thereby avoiding the parameter conflicts caused by fixed weights. Nevertheless, these overhead and are notoriously difficult to optimize during training. Furthermore, paradoxically, since meta-networks themselves rely on fixed parameters, they are also prone to parameter conflicts when handling diverse inputs.
14
networks introduce significant computational
For addressing the above issues, we propose the Minimalistic Incremental Network for Image Restoration(MINND novel and lightweight universal architecture. MINI's core component is specialized Meta Convolution module, which call MINIconv. Unlike vanilla meta convolutions, it does not introduce any additional computational overhead. Instead, it achieves complete parameter isolation between different tasks through a selective embedding mechanism, thus possessing an extremely simple structure. Moreover, this embedding based structure is naturally well-suited for task-incremental learning, and when combined with a simple query mechanism, it can achieve near- forgetting-free task expansion. Finally, we design specialized embedding regularization method to enhance the robustness of MINI. Extensive comparative experiments demonstrate that our method surpasses all existing incremental learning approaches in insseinnremettll settings, achieving state-

---

Table 1: Qualitative comparison among different types of multi-task image restoration methods
Mrtpods/Property
paramcter-isolatd
low-overhead
incrementally-ddapiic
optimization-fricn friendly
Prompd/rrior-ased
Multi-head
Meta-Conv
MINI (Ours)
of-the-art performance. Remarkably, under MINI architecture, the overall multi-task performance of most existing image restoration methods is significantly improved. Overall, the main contributions of this paper are follows:
To the best of our knowledge, MINI is one of the first attempts to explore task-incremental learning the field of image restoration. It serves strong and minimal baseline that can inspire future research in this emerging sub-task.
We propose MINI with a minimalistic design that achieves almost complete parameter isolation across different tasks. Moreover, the proposed MINIconv can be seamlessly integrated into any existing image restoration backbone, endowing it with task-incremental learning capability and boosting their multi-task overall performance, while introducing negligible additional computational cost.
To further enhance the robustness of MINI, we introduce task-aware embedding consistency regularization tailored to its structure.
MINI achieves state-of-the-art performance compared to existing eesk-incremental learning methods, and is even competitive with AIO image restoration approaches trained under joint learning.
Preliminary: Meta Convolution in Image Restoration
Meta convolution (MetaConv) is class of parameter-adaptive techniques that dynamically gen- erate convolutional weights based on task-specific or input-conditioned information. First intro- duced in the context of dynamic filter networks [8] and further popularized by approaches such as HyperNetworks and Dynamic Convolution [9] MetaConv has recently been applied to a range
of low-level vision tasks, such as super-resolution 20 The core idea is to replace fixed convolutional
weights with weights produced by a small auxiliary network- known as meta netwoke which allows the model to adapt to varying tasks or degradations by conditioning on additional embeddings or features.
While conceptually appealing, existing MetaConv methods face several significant limitations:
Increased computational complexity: The meta-network itself often comprises multi-layer perceptrons or lightweight convolutional sub-networks. These introduce considerable overhead during both training and inference I7].
Optimization difficulties: MetaCony introduces nested dependency between the generated weights and the meta-network's 's parameters, which complicates gradient flow and frequently results in unstable slow convergence 21
Static retaapprrmettes: Paradoxically, while MetaCony aims to mitigate task interference by generating dynamic weights, the meta-network itself typically fixed once trained. This means the meta-network may still suffer from parameter conflicts when facing multiple tasks or distribution shifts.
These issues limit the scalability and robustness of MetaConv, particularly in the context of incre- mental image restoration, where tasks arrive sequentially and require both parameter isolation and computational efficiency. Addressing these challenges is the motivation behind our proposed archi- tecture. Table pressnts a qualitative comparison of various types of multi-task image restoration methods.
3

---

(a) Overall MINI A rchitecture
Query function g
Imputxx
outguty y
Any bockbone
with MINI-ConV
(b) MINI-Conv
(c) Query function g())
Mese ambeeckingg
Chandiictio KLB
(x)
IV'
Glibbal
soforar
wesghtt embeedlings
pelching
Loral
(Xx)
reshape
convolution
NLP
hidden state
@ Element: wise addition
Switch
An overview of the MINI architecture. MINI can be built upon any existing image restora- tion backbone by simply replacing the original convolutional layers with our proposed MINIconv and introducing a lightweight degradation query mechanism. Subfigure (b) shows the structure of MINIconv, which consists of only two parallel embedding pools without any complex components, as detailed in Section 3.1 Subfigure (c) illustrates our lightweight degradation query module, which provides specific task IDs for MINIconv to select the corresponding embeddings for use, as detailed in Section 3.2]
Figure 2:
3 Minimalistic multi-task image restoration architecture
Our proposed MINI architecture is illustrated in Figure 2 a). As shown, the design is remarkably simple and intuitive, and can be built upon any existing image restoration backbone. The only modifi- cations required are to replace the standard convolutions in the original backbone with MINIconvs and to introduce lightweight query function. "MINIconv + query + any backbone." It is worth emphasizing that our approach does not focus on designing complex novel structures or sub-modules. In contrast, we aim address the problem of parameter conflicts across multiple tasks using the most minimalistic paradigm possible.
other words, the MINI framework essentially
3.1 MINIconv: embeddings are all your need
Classical regularization based methods the model to update parameters that are more relevant to the current task, while minimizing changes to those deemed less important. We consider this essentially form of "soft" parameter isolation. Inspired by these methods, we seek a hard parameter isolation mechanism, where only subset of parameters is deterministically updated during incremental learning, while the remaining parameters are completely excluded from training. In this way. when training the model on new tasks, the weights associated with previously learned tasks remain completely unaffected.
22 in incremental learning focus on constraining
Unfortunately, is difficult to implement the aforementioned hard isolation mechanism in a standard convolutional layer, convolution kernel is overly compact and the convolution operation itself is inherently continuous and sliding. To address this, we propose MINIconv, a specialized and flexible convolutional structure, illustrated in Figure 2(.). The core components of MINIconv only consist of two embedding pools, each composed of a fixed number of embeddings with desirable separability, In the PyTorch framework, they can be conveniently implemented using nn. Embedding. Each weight embedding pool is set to have size of (C'n. x Cout x K2)/G, where Cin is the number of input channels of the hidden state h, Cout is the number of output channels, K the kernel size, and G is the number of groups in the grouped convolution. And each bias embedding pool is set to have a size of Cout. Each pool consists of embeddings of the same size, representing the maximum number of tasks the model can accommodate. During training and inference, one embedding is selected from each of the two pools via the query function (detail Section 3.2). Then after a simple reshaping, these two embeddings used to perform standard convolution operations on the hidden state h,

---

and the remaining embeddings are excluded from both forward and backward propagation, thereby achieving complete parameter isolation. The complete computation process of MINIconv can be formulated as follows:
i g(x),
W = reshape(e ) E
bt (1) E IR Cour,
h' We & h br,
where "@"" denotes convolution operation, tw c denote 1-th embedding of weight embedding pool and bias embedding pool respectively, q(-) is the query function.
Intuitively, specific task (c.g., deraining or dehazing) exclusively uses its assigned weight and bias embeddings. When a new task arrives, it only needs to be allocated an unused embedding pair as its dedicated parameters. Therefore, this mechanism is highly suitable for task-incremental learning. Interestingly and perhaps surprisingly, although MINIconv increases the total number of convolutional parameters by a factor of T, its minimalist select-and-use mechanism incurs almost no additional computational cost. This is because only the parameters corresponding to single standard convolution kernel are actually involved during each forward pass. Therefore, MINIconv be extensively applied throughout the network, unlike traditional MetaConv approaches that rely on auxiliary networks to generate dynamic kernels, which can lead to considerable computational overhead when used widely.
3.2 Degradation query mechanism
As described in the previous section, MINIconv requires query mechanism to determine the task type of the input image and select the corresponding embedding accordingly - essentially serving as a degradation classifier. Some promising related works already be proposed, such as DA- CLIP(6). However, to adhere to the principle of minimalism, we propose a lightweight degradation classification module, illustrated in Figure Ce) We observe that most image degradations can be roughly categorized into global (c.g., low-light, blur) and local (e.g., rain, raindrops, fog). Therefore, we first divide the input image into patches and feed them into global/local classification MLP, which outputs a two-dimensional vector. The degradation type is then determined via an argmax operation. Based this result, the patched image is further routed to either the global or local MLP branch for more fine- grained degradation classification. The final output vector is passed through a softmax function, and the task ID is obtained via argmax operation. The entire process can be described as follows:
(x:) N Patching(x))
2 MLP global local Aggregate( ((x;))) R
mode argmax(o (2))) E [1,2].
(2)
MLP global (xa) H), is Imndd
RTT
0
MT.P Ptoote (xx)), if Imode = 2
tdd (rgmax(((())
where N is the number of image patches, (-) denotes the softmax operation, and tia denotes the final task ID obtained. Before training the main network, the entire query module can be simply pretrained using the following loss function:
e
Equery log
Aquerylog
Σ e
where n (1, is the ground-truth label indicating whether the degradation global or local, m E (1,2 is the ground-truth task ID, and Aquery is a balance coefficient. In our early experiments, we attempted build the degradation classifier using either single MLP series of MLPs in sequence. However, we found that the classifier consistently struggled to distinguish certain types of degradations, such as blur and fog. address this issue, we proposed a two-stage degradation classification mechanism grained classification which led to the design of our current query module. This simple yet effective change improved the classification accuracy by approximately 15
performing coarse classification first, followed by fine-
5

---

3.3 Embedding-consisterr regularization
Although the above MINIconv and query mechanism are sufficient construct complete multi- task image restoration framework, we observe in practice that the overall performance of MINI is highly sensitive the accuracy of the query mechanism. Once the degradation type sample is misclassified - even for a small number of samples - it can significantly degrade the overall performance. Therefore, in training phase, to enhance the fault tolerance and robustness of MINI, we introduce a specialized embedding-coosiistenn regularization (ECR) method, as formulated below:
W
Inow -=
WOO
know
80
(1)
know
Leer
W0 E
11 Unone 1
1=
where I. is the total number of MINIconv layers in the model, W:O 60 denote the t-th weight embedding and bias embedding in the (-th MINIconv layer respectively, and tnow denotes the ID of the newly introduced task currently being trained, and W B(t)) denote the mean of the first fnow weight embeddings and bias embeddings in 1-th MINIconv layer, respectively.
Intuitively, we expect the embedding of the new task tasks, encouraging the embeddings responsible for different tasks to remain as consistent as possible. In this way, even if the query mechanism makes an incorrect degradation prediction and selects the embedding of another task, the overall performance will not be significantly degraded. Overall, the total training loss of MINI is follows:
differ significantly from those of existing
Call Lmain decr Leor:
(5)
where Aeer is the ECR regularization coefficient, Lmain refers to common reconstruction losses such as L1, L2, perceptual loss etc.
4 Experiments
4.1
Comparative experiment
Table 2: Quantitative comparison of several image restoration baselines trained on five datasets using one-for-one, joint learning, and incremental learning (MINI) strategies. The incremental learning is conducted in sequential manner following the task order of rain - haze + blur - raindrop + low light. In all-in-one manner, the symbol indicates that the data is reported from the original paper. "4" indicates that higher value is better for the metric, while "4"" indicates that a lower value preferable. In the MINI framework, the metric values that show improvement compared to the all-in-one setting are highlighted in bold.
denncti
RI008L (586)
RESIDE AL (3a2c)
Cofoo (slan) SS0M
Kaisaro (ainskop)
LOLIZ (lw lish
PSNR
SSIKT
PSNR
LIOS.
PSNRT
SSIM
TPUNT
MAXIM
30.61
0.501
20.12
0.902
33.8m
0:061
23.45
0.853
2.11
NAFNo
(onb-pezili))
IR-SDE
31.65
01511
0/967
30.70
0.901
0.064
DA-CLIMG
33.91
0.925
0.021
30.16
0.936
0.030
30.68
0.963
0
31.50
0.964
0.056
23.77
0,830
0083
AidNet
27.54
27.13
hompull1
$1.02
091.4
24.5
0.911
0.64
38.05
0.061
0.063
28.55
01912
0.014
21.36
0.886
0.118
guer leamnin)
all iss
NAFNo
20.54
01386
0.375
0.065
0.055
21
20.15
0.914
0.941
0.19
35.05
30.51
0.055
0061
0.074
22.90
01895
0.916
0.51
0.841
21.35
0.875
0.121
IRSDE
0.592
0/67
29.33
0.950
0.065
0.525
0.022
21.34
DA-CLIPG
31.51
0.923
0.022
24.58
0.955
0.36
20.39
0.902
0.000
30.44
0.500
0.076
22.15
0.887
0.101
0,775
29.75
21.31
Propuli2
31.51
01917
0194
38.64
(U952
0.617
38.00
0.565
30.7
1L915
0.289
31.44
0.885
MINI
MAXIM
31.32
6.00
11,89
38.55
IL941
0,37
32.53
6.557
0.899
31.94
IU927
0.843
23.81
0.896
8.991
NAFNo [3
30.90
0.91
0.442
29333
0.960
0.08
29.56
0.593
0.095
30.45
0.912
0.846
22.60
0.873
leaming)
0,848
30.15
S.E
23.45
DA-CLIF5 15
31
0.927
0
38.15
0.949
6.822
36.15
0.514
0.083
31.00
0U921
0.841
23.15
01,98
0.895
4.1.1 Experiment setup
To demonstrate the effectiveness of our proposed MINI architecture for task-incremental image
restoration, we conduct detailed comparisons on five datasets (five tasks) based on several existing im- age restoration methods, the five datasets are [2],H RESIDE-6K (8), GoPro[29] Raindrop 30).
and L3Lv We evaluate image restoration performance using the following metrics: PSNR.

---

SSIM LPIPS We compare the overall performance of these methods under three training paradigms: one-for-one task-specific training, all-in-one joint training, and incremental training using the proposed MINI framework. For the all-in-one training setting, we train for 2000 epochs on mixed dataset composed of the five aforementioned training sets. The learning rate follows a cosine annealing schedule with warm-up. peaking at 0.0002. The loss function consists of an 1.2 loss combined with a VGG16-based perceptual loss. It is important to NOTE that during actual training, the embeddings need to be manually initialized using He initialization 34]; otherwise, the model may struggle to converge. The training is conducted on two NVIDIA 2080ti And due to the imbalance in the sizes of the five datasets, we adopt common resampling strategy during training to ensure that each training batch contains a balanced number of images from each dataset. Specifically, methods that require more training iterations, they are trained for 3000 epochs. For MINI, we adopt an incremental learning strategy following the task sequence: R100H - RESIDE-6K - GoPro Raindrop - I.OL22. For each baseline, we train on each dataset for 400 epochs under the same settings, while IR-SDE and DA-CLIP are trained for 600 epochs. During the training phase, the degradation query module and the main network trained separately. The MINIconv layers in the main network take the ground-truth task ID as input instead of g(x). The hyperparameter settings are as follows: Aquery 1, Aeer 0.001, T 5. During inference, ((x) is used as the input to MINIconv.
since IR- SDE(6) [6] and [ are diffusion-
Table 3: Final performance comparison of model-agnostic generic incremental learning methods under NAFNet (4] backbone. L2PB5) and DualPrompt (36] use the backbone networks from their original papers, and their results are provided as reference for comparison. The training task sequence
is: rain- haze- + blur + raindrop + low light. The best results are highlighted in bold.
Dataness
R1000 (aan)
RESIDE (haze)
Katndrop Lalidropp
(lw lipha)
metbods / metrics
PSNKT
SSSM!
LPIPS.
PSNRT
SS3MI
LPIPS)
PSNK1
SS1M
LP1P5
PSNR1
SSIMI
LPIPS
PSNR
SSIMM
LPIPS
1wf
16.23
EWCLT
13.36
0.512
0.584
19.51
0.271
0.623
0.255
01.440
018771
11.36
0.496
0.612
18.24
0.597
0.365
22.34
0.603
0
0.719
0.165
21.24
01809
0.113
MAS 13
14.37
0.684
0.509
20.77
0.205
24.81
0.753
0.154
0.383
0.067
32.75
0.890
35
17.06
0.694
22.73
0.137
24.53
0.788
0.108
24.61
0.908
21.39
IL907
DualPhoupe 56
16/03
0.682
0.124
20,02
0.733
0.156
24.48
0.792
0.116
27.04
0.874
0.071
21.90
0.904
0.073
MINI
30.90
23.00
4.1.2 Analysis
As shown in Table under our MINI framework, the overall performance of various baseline methods on multi-task image restoration has been significantly improved. In some tasks, the performance even rivals that of their corresponding task-specific training versions. More importantly, MINI endows these methods with excellent incremental learning capabilities. Meanwhile, we take NAFNet (24 as baseline methods and compare MINI with existing model-agnostic generic incremental learning approaches, as shown in Table 3 The results show that our MINI design, based on "hard parameter isolation," effectively eliminates catastrophic forgetting and achieves state-of-the-art performance in task-incremental learning. In contrast, other methods suffer increasingly from catastrophic forgetting as the task sequence grows longer, ultimately leading to poor overall performance. The visual comparison is shown Figure
In addition, we compared meta-conv with our proposed MINI-conv under the same training settings in FLOPs and Rarams, and the results are shown in Table 4 We adopted the same NAFNet as the backbone, replaced its original standard convolutions, and conducted tests on images with a resolution of 256x256. It can be observed that, compared with standard 2D convolutions, meta-cony increases both FLOPS and Params, with a particularly large increase in Params, which is impractical for real-world applications. This limits the scalability of MetaConv within backbone networks. In contrast, our MINIcony has no impact on FI.OPs, and its Params increase only linearly with the number of tasks, allowing replace standard 2D convolutions in the backbone on a large scale.
Table 4: Comparison between MetaConv and MINIconv. MetaConv generates convolution parameters
using a single-layer MLP. The backbone network is NAFNet.
methods
FLOPs]
Params]
standard 2D-conv
22.56G
19.30M
MetaConv
30.22G
1814.IM
MINIconv
22.56G
96.5M
7

---

task sequence order
Input
GT
SI
EWC
LWF
MAS
MINI
Rin
hoze
blur
rainrro
low light
Figure 3: Visual comparison model cedlosagnostic generic incremental learning methods. Zoom for details.
4.2 Ablation study
Embedding--onniisennyy regulation. As described in Section 3.3 the accuracy of the query mech- anism significantly affects the overall performance of MINI. To enhance the robustness and fault tolerance of MINI, we introduce an Embedding- Consistency Regularization (ECR). To validate its ef- fectiveness, intentionally inject certain proportion of random incorrect degradation classifications into the model and compare its overall performance with and without the proposed regularization. Similarly, train the MINI-based NAFNet the five aforementioned datasets. The results shown in Figure 4 Under different degradation classification еттог rates, ECR consistently leads to better overall performance compared the case without ECR. Visual results under several misclassi- fication cases are shown in Figure 5) It can be observed that even when degradation is misclassified, the model trained with ECR still maintains certain level of image restoration capability, whereas the model trained without ECR exhibits almost no fault tolerance.
Our pretrained query module achieves error rate of approximately 6%. While it is possible adopt more advanced image classification models-such as the powerful DA-CLIP further reduce the rate, this would come at the cost of inereased structural complexity and computational overhead. Therefore, applying ECR on top of a lightweight degradation classifier can be viewed as a better trade-off between performance and efficiency.
Task sequence order. To explore the impact of task training order on MINI, we train the MINI-based NAFNet under four different task sequence orders and compare the final performance, shown in Table 5 The results show that, thanks to MINI's strong parameter isolation capability, changing the task sequence order has little impact its final performance. The variation in average PSNR is within 0.3 dB, SSIM within 0.01, and LPIPS within 0.002. This demonstrates the robustness of the MINI architecture to task order.
worth noting that, model with complete parameter isolation should, in theory, achieve consistent performance regardless of the training order. Although MINI achieves near-complete parameter

---

ave PSNR, 11
3ve SSIMI
ave LPIPS(4)
10
With ECR
ie whthout ECR
0.9
with ECR
Without ECR
0.5
With ECR
Without EGR
25
0.7
20
0.6
0.2
75
0.5
0.4
30
15
30
30
Errorittte 186)
Race
Emor (56)
Figure 4: Overall performance comparison with and without ECR under different degradation classification error rates. The baseline architecture is the MINI-based NAFNet, with Aeer set to 0.001.
Connect
Rain
Reindhp
hage
low Light
Impul
GT
BCR
Musceesifedd Lnc
Reinthop
Rain
low light
Jaase
Figure 5: Visual comparison with and without ECR under incorrect degradation classification, based on the MINI-based NAFNet.
isolation, as discussed in Section 5 (Discussion), most baseline architectures still contain small number of shared parameter components beyond standard convolutions such as LayerNorm and nn. Parameter elements- whichh which may introduce minor parameter conflict. This residual overlap is the primary reason behind the slight performance differences observed under different training orders. However, we found in practice that these shared parameters rarely lead to catastrophic forgetting. Therefore, in favor of architectural simplicity, we did not propose replacements like "MINI-LayerNorm" to fully isolate these components.
Table 5: Final overall performance of the MINI-based NAFNet under different training orders. The task IDs and corresponding datasets are as follows: 1: derain (R (R100H); 2: dehaze (RESIDE-6K); 3:
deblur (RESIDE-6K); 4: raindrop removal (Raindrop); 5: low-light enhancement (LOLv2).
orders/ metrics
PSNR
SSIM1
LPIPSJ
1-2 +-
28.754
0.911
0.0632
4-44-3-42-1
28.612
0.912
0.0614
1-33-2-45-4
28.518
0.908
0.0619
3-22-4+11-5
28.625
0.914
0.0628
Discussion
Other parameter components except for convolutional layers.
In most CNN-based backbones, in addition to convolutional layers, there are also some smaller parameter components such as LayerNorm and nn. Parameter etc. Although these components can also suffer from parameter conflict across different tasks, we find in practice that converting them to IINIconv-does not lead to significant improvements
into smmeeding-basele

---

This is mainly because the parameter norms of these components do not vary substamtially across tasks, meaning that even without explicit parameter isolation, they have limited impact on the overall performance. The application of the backbones will be explored in our future work. In addition, a
in multi-task performance (detailed in Appendix B.
MINI architecture to transformer-basee preliminary test result on SwinlR (37 can be found in Appendix C
Limitations of MINI.
Despite the strong performance and efficiency demonstrated by the proposed MINI architecture in task-incremental image restoration, there remain several limitations worth noting.
First, the number of tasks that MINI can support is inherently limited by the size of the embedding pool T' in each MINIconv layer. This value must be predetermined during model design and cannot be extended dynamically afterward, which may pose challenges in scenarios where the total number of tasks is unknown or incrementally growing over time. Second, although MINIconv introduces no additional computational overhead during inference and remains equivalent to standard convolution in terms of forward computation, its use of hard parameter isolation causes the total number of parameters to scale by factor of T. While this expansion the trade-off for achieving interference- free learning across tasks, it may impose memory burdens in resource-constrained environments.
In future work, more flexible or compression aware embedding strategies may be explored to enhance the scalability and deployability of MINI.
6 Conclusion
In this paper, we propose MINI (Minimalistic Incremental Network for Image Restoration), a novel and lightweight framework designed for task-incremental learning across multiple image restoration tasks. Unlike traditional all-in-one models that suffer from parameter conflict, MINI adopts a hard parameter isolation strategy through the introduction of simple yet effective module called MINIconv. By leveraging embedding pools instead of dynamic meta-networks, MINIconv achieves task-level parameter decoupling without introducing additional computational overhead. Importantly, MINI is a aong-and-paay design that can be seamlessly integrated into a wide range of existing image restoration backbones (e.g., NAFNet, Restormer), enabling them to acquire incremental learning capabilities with minimal modification. Moreover, MINI consistently improves the overall multi-task performance of these baselines, while preserving strong performance on each individual task. To further support robust task adaptation, we introduce lightweight degradation query module and an regularization (ECR) strategy, which together enhance MINI's fault tolerance and reliability. Extensive experiments across five diverse image restoration tasks demonstrate that MINI achieves strong task-incremental performance with minimal forgetting, significantly outperforming existing generic continual learning methods. Notably, MINI also retains competitive performance compared to fully joint training baselines, while offering the flexibility of sequential task adaptation. We believe that MINI provides practical and generalizable solution for continual learning in low-level vision. Future work may explore more dynamic embedding mechanisms, better task discovery under unknown settings, and extensions to wansformer-basell architectures.
References
[1] Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17452-17462. 2022.
[2] Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz. Khan. Promptir: Prompting for all-in-one image restoration. Advances in Neural Information Processing Systems, 36, 2024.
[3] Jiaqi Ma, Tianheng Cheng, Guoli Wang, Xinggang Wang, Qian Zhang, and Lefei Zhang. Prores: Exploring degradation-awarc visual prompt for universal image restoration. arXiv preprint arXiv:2306. 13653, 2023.
[4] Zilong Li, Yiming Lei, Chenglong Ma, Junping Zhang, and Hongming Shan. Prompt-in-prompt learning for universal image restoration. arXiv preprint arXiv:2312.05032 2023.
10

---

[5] Xingyu Jiang, Xiuhui Zhang, Ning Gao, and Yue Deng. When fast fourier transform meets transformer for image restoration. In European Conference on Computer Vision, pages 381-402. Springer, 2024.
[6] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Controlling Vision-language models for multi-task image restoration. International Conference on Learning Representations, 2024.
[7] Jun Cheng, Dong Liang, and Shan Tan. Transfer clip for generalizable image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25974-25984, 2024.
[8] Xiaogang Xu, Shu Kong, Tao Hu, Zhe Liu, and Hujun Bao. Boosting image restoration via priors from pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2900-2909, 2024.
[9] Yuhao Liu, Zhanghan Ke, Fang Liu, Nanxuan Zhao, and Rynson WH Lau. Diff-plugin: Revitalizing details for diffusion-based low-level tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4197-4208, 2024.
[10] Yuang Ai, Huaibo Huang, Xiaoqiang Zhou, Jiexiang Wang, and Ran He. Multimodal prompt perceiver: Empower adaptiveness generalizability and fidelity for all-in-one image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1-432-254444 2024.
[11] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3511-3526 2017.
et al.
[12] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995. PMLR, 2017.
[13] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( (9:25)2993-2247, 2017.
[14] Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Ming-Hsuan Yang. Decoupled dynamic filter networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6647-6656, 2021.
[15] Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. Advances neural information processing systems, 32, 2019.
[16) Ningning Ma, Xiangyu Zhang, Jiawei Huang, and Jian Sun. Weightnet: Revisiting the design space of weight networks. In European Conference on Computer Vision, pages 776-792. Springer, 2020.
[17] David Ha, Andrew Dai, and Quoe V Le. Hypemetworks. arXiv preprint arXiv: 1609.0916. 2016.
[18] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. Advances in neural information processing systems, 29, 2016.
[19] Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11030-11039 2020.
[20] Jae Woong Soh, Sunwoo Cho, and Nam Ik Cho. Meta-transfer learning for zero-shot super- resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3516-3525, 2020.
[21] Sebastian Flennerhag, Andrei Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. arXiv preprint 190vv11009.0000 2019.
11

---

[22] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[23] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mip for image processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5769-5780, 2022.
[24] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European conference computer vision, pages 17-33. Springer, 2022.
[25] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5728- 5709,55732220 2022.
[26] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mean-reverting stochastic differential equations. International Conference on Machine Learning, 2023.
[27] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying I. Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1357-1366, 2017
[28] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang. Benchmarking single-image dehazing and beyond. IEEE Transactions on Image Processing, 28(1):492-505, 2019.
[29] Tae Hyun Kim, Seungjun Nah, and Kyoung Mu Lee. Deep multi-scale convolutional neural net- work for dynamic scene deblurring. In Conference on Computer Vision and Puttern Recognition, pages 1-21. IEEE, 2017.
[30] Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2482-2491, 2018
[31] Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing, 022202222086. 2021.
[32] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (3(4):00 612, 2004.
[33] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceeuiiggs of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[35] Zifeng Wang, Zizhao Zhang, Chen-Y Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2022.
[36] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference on computer vision, pages 631 631-648. Springer, 2022.
[37] Jingyun Liang, Jiezhang Cao. Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEF/CVF international conference computer vision, pages 1833-1844, 2021.
12

---

A Potential societal impact
This work focuses on the development of minimalistic and general framework for task-incremental learning in image restoration. The proposed method can enhance the adaptability and longevity of vision systems deployed in dynamic real-world environments, such autonomous driving, surveillance, and medical imaging, by enabling them to incrementally learn new restoration tasks without forgetting previous ones. This contributes to more sustainable and upgradable systems.
Since the method is task-agnostic and does not rely on sensitive or personal data, we do not foresee direct negative societal impacts. However, with many vision enhancement technologies, poten- tial misuse in image manipulation or surveillance scenarios should be considered. We encourage responsible deployment aligned with ethical guidelines and privacy regulations.
B Fully embedded vs. Only MINI-conv
To further validate the conclusions discussed in the Discussion section, we compared the final performance of "embedding parameter components" and "embbedding only the convolutional layers" (i.e., MINI-conv). The results are shown in Table
Table 6: Comparison between fully-embedded and
paradigm.
methods
ave PSNRT
ave SSIM
ave LPIPS.
Fully embedded
28.844
0.915
0.0629
Only MINI-conv
28.754
0.911
0.632
can be seen that, the average performance improvement was only around 0.1 dB in PSNR. We believe this is because these parameters primarily perform affine and scaling transformations among features, and such transformations tend to exhibit limited variation across tasks within the same network architecture. Therefore, even without explicit parameter isolation, these components do not lead to severe catastrophic forgetting.
C MINI-based transformer architecture
To preliminarily evaluate the performance of the MINI architecture transformer-basee models, we conducted the same experimental tests on SwinIR[37], specifically, we embedded the parameter components within each transformer block to replace the original parameter components. and the results are shown in Table 7 The results indicate that our MINI architecture can also be applied to transformer-basee backbone networks to enhance incremental learning capability. However, it is important to ensure that the parameter initialization of the embedding pool remains consistent with that of the original parameter components.
Table 7: Final performance comparison of model-agnostic generic incremental learning methods under SwinIR (37 backbone. The training task sequence is: rain - haze - blur - raindrop - low light. The best results are highlighted in bold.
RETICE (rann)
RESIDE-6: (laze]
CiPro [har]
Rainlny (ranntrop))
LOL.22 [low light
metrcc
PSNRT
55IM1
LPIPS.
550M1
LPEPS)
LPIPS
SSIM
LPIPS]
133
14.01
0.501
0.636
19.21
0.595
0.263
23.96
0.623
0.159
27.04
0.316
0.095
22.05
11873
0.305
EwC [1]
14.55
0.565
0.525
20.00
0.511
0.233
20.02
0.634
0.249
24.07
0.789
0.128
22.83
0.880
0.100
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In Section 1 and Abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section 3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Described in Section 3.1 3 and 3.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Our code will made publicly available after the camera-ready stage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
ily for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwtc//gguie///eeeeceoocc CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/ httpr//...../pp...ttttccccceeeea
Answer:
[Yes]
Justification:
We comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: In Appendix A
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The methods and datasets mentioned this paper are publicly available and open-source. They are properly cited in the article, and there is plagiarism misuse.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
18

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided a detailed description of how to use our code in and commit making our code publicly available on GitHub after the camera-ready stage.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
19

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standadd components.
Please refer to our LLM policy /tttps://neurips cc. cc//om/feremccs/222551... for what should or should not be described.
20


Reviews:
[{"review_id": "aLoKUjuITk", "reviewer": "Reviewer_svUr", "summary": "This paper introduces an incremental learning paradigm for image restoration tasks, which I think is meaningful. But I remain skeptical about the novelty of the method proposed in this paper. However, despite this, since incremental learning has a wide range of application scenarios in the field of image restoration, especially in the real-world scenes, I think this paper can reach the acceptance line of NeurIPS after reasonable discussion and modification.", "questions": "Please see `Strengths And Weaknesses`.", "limitations": "Please see `Strengths And Weaknesses`.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The outstanding contribution of this article is the introduction of incremental learning for image restoration tasks. I will mainly list here what I think are the weaknesses of this paper:\n\n1. The MINI-Conv is very similar to the Dynamic Conv [1] and also the Prompt Block in PromptIR [2]. Perhaps the authors could discuss the difference between these two?\n2. I am quite confused by the comparative experimental setup of this paper. Taking Table 2 as an example, this paper compares the all-in-one (joint learning) method with the incremental learning-based method. The all-in-one method trains on five degradation jointly, and the training process is blind. However, the incremental learning based method proposed in this paper can get the correct degradation type embedding during training, which is not blind. Will this difference affect the final restoration performance?\n3. Why not use the image restoration network itself as the degradation classifier, such as DCPT [3]? Is an external degradation classifier necessary? As I understand, MINI-Conv requires additional degradation classification during inference. If the degradation classification result is wrong, can the MINI-Conv-based method still output a clean image?\n4. I am very interested in the ablation experiment results of the Task sequence order. Why do different Task sequence orders lead to different results? The PSNR difference between 1→2→3→4→5 and 1→3→2→5→4 is 0.2 dB, which cannot be called \"little impact\".\n5. Does the data imbalance problem affect the effect of incremental learning? I noticed that the proposed method trains 400 epochs for each dataset. If this training epoch is redistributed according to the actual data volume of the dataset, can better results be achieved? For example, train more epochs for the small dataset (LoL) and train fewer epochs for the large dataset (RESIDE).\n6. For networks that do not use convolution, such as SwinIR, is MINI-Conv still useful?\n7. Is MINI-Conv applicable to mixed degradation or more degradation types? Can MINI-Conv be generalized to real degradation scenarios?\n\n[1] Dynamic Convolution: Attention over Convolution Kernels. CVPR 2020.\n\n[2] PromptIR: Prompting for All-in-One Blind Image Restoration. NeurIPS 2023.\n\n[3] DCPT: Universal Image Restoration Pre-training via Degradation Classification. ICLR 2025.", "was_revised": true, "final_justification": "The author's response resolved all my concerns.", "rebuttal": "Thank you very much for your recognition and positive feedback on our work. Your response is truly encouraging to us. We will carefully and thoroughly address your questions to resolve your concerns.\n\n$\\textbf{For question 1:}$\n\nAs you mentioned, Dynamic Conv and PromptIR share certain similarities with our proposed MINI. In fact, during the early stages of our investigation into incremental learning for image restoration, we conducted an in-depth study of the design principles behind these two methods to assess their potential benefits. However, we ultimately found that both approaches have certain inherent limitations that are difficult to overcome, which in turn inspired us to design MINI.\n\n(1) Dynamic convolution methods such as Dynamic Convolution and MetaConv introduce additional subnetworks to generate adaptive convolution kernels based on the input, thereby partially alleviating the issue of parameter conflict. However, as discussed in the Introduction (lines 46–51), replacing a large portion of the original backbone with such dynamic convolutions incurs considerable computational overhead. Moreover, these methods do not explicitly isolate task-specific parameters at the architectural level. Instead, they rely on the model to “implicitly” learn task differentiation during training, which is often difficult to optimize and inherently uncertain in practice. In contrast, our MINIconv does not require the introduction of additional subnetworks, thus avoiding extra computational overhead. This enables it to replace a large number of standard convolution units in the backbone, leading to more effective parameter isolation. Moreover, we enforce hard parameter isolation directly at the architectural level, which is not only effective but also highly optimization-friendly. To highlight these advantages, we will add a comparison with Dynamic Convolution methods in terms of performance, FLOPs, and Params in Section 4.1.\n\n(2) PromptIR, as a representative prompt/prior-based all-in-one method (as illustrated in Figure 1(a)), typically introduces learnable parameter components (Prompt Blocks) into the main network to guide the model in adapting to multiple tasks. While this strategy can enhance multi-task performance to some extent, it does not address the issue of parameter interference within the main network, thus leading to performance bottlenecks. In contrast, MINI explicitly addresses this issue by replacing standard convolutions in the main network with MINIconv, thereby further improving multi-task performance. In fact, as shown in Table 2, integrating MINIconv into the main network of PromptIR significantly boosts its performance and also equips it with incremental learning capability.\n\n$\\textbf{For question 2:}$\n\nThese all-in-one methods typically rely on different task data to “implicitly” learn degradation type classification during training, and subsequently learn how to perform restoration. In contrast, MINI adopts an incremental learning paradigm that decouples these two steps: we explicitly train task-specific embeddings for each task, and then train a lightweight query module to perform degradation classification. \n\nOur core motivation is to achieve parameter isolation across different tasks. Therefore, during the embedding training phase, we intentionally train embeddings for different tasks separately and explicitly. It is important to emphasize that the performance improvement should not be attributed solely to the change in training strategy, but rather to the structural parameter isolation enabled by our design.\n\n$\\textbf{For question 3:}$\n\nOur goal is to design a plug-and-play, general-purpose multi-task incremental learning paradigm, which requires us to introduce as little additional computational overhead as possible into the original model. In the MINI architecture, the only extra computation comes from the degradation classifier, so we are very cautious in selecting its design.\n\n(1) The approach in DCPT, which uses the decoder as a degradation classifier to enhance the encoder’s multi-task capability, is indeed elegant. However, such a design goes against our minimalist philosophy. First, training the DCPT decoder requires intermediate features from the restoration model (encoder) as input. Yet, different models have different feature resolutions in their intermediate layers, making it difficult to deploy a DCPT-style decoder in a plug-and-play manner across various architectures. Additionally, it introduces considerable computational overhead. In contrast, our proposed query module is structurally independent of the main network and achieves effective degradation classification with minimal computational cost. We will include a detailed comparison with DCPT in Section 4.1 and evaluate whether a MINI-based variant of DCPT can achieve even better performance.\n\n(2) Misclassification by the degradation classifier is almost inevitable. To minimize its impact on the final performance, we propose the Embedding-Consistency Regularization (ECR) strategy (Section 3.3). Its effectiveness is demonstrated in the ablation study (Figures 4 and 5 in Section 4.2). The results show that ECR significantly improves the model’s robustness under misclassification scenarios.\n\n$\\textbf{For question 4:}$\n\nThis is indeed an interesting observation. A model with complete parameter isolation should, in theory, achieve consistent performance regardless of the training order. Although our MINI achieves near-complete parameter isolation, as discussed in Section 5 (Discussion), most baseline architectures still contain a small number of shared parameter components beyond standard convolutions—such as LayerNorm and nn.Parameter elements—which may introduce minor parameter conflict. This residual overlap is the primary reason behind the slight performance differences observed under different training orders. However, we found in practice that these shared parameters rarely lead to catastrophic forgetting. Therefore, in favor of architectural simplicity, we did not propose replacements like “MINI-LayerNorm” to fully isolate these components.\n\nWe will address this point in our revised version. Specifically, we will add an explanatory note at line 239 to clarify this issue.\n\n$\\textbf{For question 5:}$\n\nAs you pointed out, selecting appropriate numbers of training iterations based on the scale of each dataset can indeed further improve the final performance. We are happy to explore this in future work. In this paper, however, our primary focus is on the comparison between models before and after applying MINI (Table 2), rather than on performance optimization. Therefore, we adopted a unified number of training epochs across tasks as a rough setting to validate the improvements brought by MINI in both multi-task and incremental learning scenarios.\n\n$\\textbf{For question 6:}$\n\nAs mentioned in the Discussion section, our future work will extend MINI to transformer-based architectures. We also hope that our work can inspire future researchers to design interesting modules such as “MINI-MLP” or “MINI-Transformer” etc.\n\n$\\textbf{For question 7:}$\n\nThis paper demonstrates the effectiveness of the proposed MINI on five common image restoration tasks. However, this does not imply that MINI's generalizability is limited to these five tasks. With appropriate settings of the task ID (T) values, MINI can also be applied to other image restoration tasks (e.g., glare removal, image denoising, etc.) with only minimal impact on previously learned tasks.\n\nAs a plug-and-play approach, the generalization ability of MINI largely depends on that of the underlying backbone model. Therefore, it is difficult to evaluate the generalization of MINI in isolation. The design of MINI aims to enhance the task-incremental and multi-task capabilities of existing methods.\n\n$\\textbf{For Paper Formatting Concerns:}$\n\nThank you for your suggestion. We will revise the description in Section 4 by separating the experimental setup into a standalone subsection."}, {"review_id": "1tTPqzRfTE", "reviewer": "Reviewer_MAFh", "summary": "This paper addresses the challenge of task-incremental learning for image restoration, where models struggle to sequentially add new capabilities without catastrophic forgetting. The authors propose a lightweight framework MINI built around MINIconv, a novel meta-convolution module that replaces standard convolutions in existing backbones. MINIconv uses task-specific embedding pools to achieve hard parameter isolation: each task exclusively activates its own embeddings during inference, preventing interference with prior tasks. This design requires no complex meta-networks, introduces negligible computational overhead during inference, and enables near-forgetting-free incremental learning.", "questions": "1. How would MINI scale to open-ended task sequences (e.g., >10 tasks) without retraining? Could you propose strategies (e.g., embedding pruning/generation) to dynamically expand capacity while preserving efficiency?\n2. How does MINI handle hybrid degradations (e.g., rainy + hazy images)? Could you test on blended datasets (e.g., RainCityscapes) or modify the query for multi-label classification?\n3. Could you ablate MINI’s performance when these layers are made task-specific versus shared? Would adapting them (e.g., via embeddings) further boost multi-task gains?\n4. Can MINIConv integrate into transformer backbones (e.g., Restormer)? Does replacing MHSA/MLP weights with embeddings require architectural changes? Please discuss feasibility.\n5. Why omit comparisons against prompt-based continual learning (e.g., L2P, DualPrompt) that also tackle task-incremental settings? Could you add these to Table 3 to clarify MINI’s advantages over parameter-efficient alternatives?", "limitations": "Please see the Weakness and Question parts.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper systematically address task-incremental learning for image restoration (IR), filling a critical gap between \"one-by-one\" and \"all-in-one\" paradigms.\n2. MINI mechanism replaces complex meta-networks with lightweight embedding pools, achieving hard parameter isolation without inference overhead. The design is simple yet effective.\n3. Plug-and-play compatibility with existing IR backbones (e.g., NAFNet, Restormer) significantly boosts practicality. Experiments show consistent multi-task performance gains.\n4. Extensive experiments across 5 diverse IR tasks (deraining, dehazing, deblurring, etc.) using multiple backbones.\n\nWeakness:\n1. Potential limited task scalability: The fixed embedding pool size T requires predefining the maximum number of tasks, which may be impractical for open-ended incremental learning.\n2. Despite ECR, performance still depends on the query module’s accuracy. The paper reports a 6% misclassification rate, and ECR only mitigates, not eliminates errors. Meanwhile, the lightweight query design (global/local branches) may struggle with complex hybrid degradations (e.g., rainy + hazy images).\n3. Non-convolutional components (e.g., LayerNorm) are not adapted to MINI’s isolation paradigm. The authors note this has \"limited impact,\" but no data supports this claim.\n4. For experiments, this paper lacks comparison against prompt-based continual learning methods (e.g., L2P, DualPrompt), which also tackle task-incremental settings. The \"all-in-one\" joint training baselines (Table 2) are not state-of-the-art (e.g., missing recent works like PromptIR).", "was_revised": true, "final_justification": "I have carefully read the authors’ response. Overall, I am relatively satisfied, especially with the extended explanation and comparisons on MINI. I believe my initial positive rating has already reflected the quality of the paper.", "rebuttal": "Thank you for your careful reading and positive feedback on our work. We will address your questions thoroughly to resolve your concerns.\n\n$\\textbf{For weakness 1:}$\n\nIndeed, as discussed in the “Limitations of MINI” section of the Discussion, the current MINI framework is designed to address incremental learning with a limited number of tasks. Open-ended incremental learning is a more challenging problem and be the focus of our future work. Nevertheless, we believe that the proposed MINI architecture remains a pioneering and practically meaningful contribution to the current research on incremental learning in the field of image restoration. Since research on incremental learning in the field of multi-task image restoration is still very limited, MINI serves as a ready-to-use solution that can be easily integrated into existing image restoration methods to effectively enhance their incremental learning capabilities. At the same time, it can also serve as a heuristic approach to inspire future researchers and provide useful insights for further exploration in incremental learning for image restoration.\n\n$\\textbf{For weakness 2:}$\n\n(1) Completely eliminating degradation classification errors is a highly challenging task. In fact, even the state-of-the-art degradation classifier DA-CLIP [1] still exhibits around a 2% error rate. Further reducing this error typically requires significantly larger network architectures, which in turn introduces substantial additional computational cost. In this context, our proposed ECR offers a highly cost-effective solution, enabling us to maintain robust overall performance (as shown in Figures 4 and 5) while using only a lightweight degradation classifier.\n\n(2) The original intention of MINI is to address the problem of incremental learning in multi-task image restoration, rather than to solve all-in-one image restoration under complex mixed degradation scenarios. Therefore, the query module is not specifically designed for handling mixed degradations. Moreover, whether models trained on single-degradation datasets can generalize well to mixed-degradation cases remains a challenging open problem in the image restoration community. This issue is more related to model design or training strategy optimization, rather than the core incremental learning problem that this paper focuses on.\n\n$\\textbf{For weakness 3:}$\n\nThank you for your valuable suggestion. In our revised version, we will include an additional evaluation in the supplementary materials to assess the impact of non-convolutional components on the final incremental learning performance. Similarly, we conducted experiments on five tasks (deraining, dehazing, raindrop removal, deblurring, and low-light enhancement) under the same training setup. The detailed experimental results are as follows:\n\n| Method       | ave PSNR ↑ | ave SSIM ↑ | ave LPIPS ↓ |\n|------------|--------|--------|--------|\n| Fully embedded   | 28.844  | 0.915  | 0.0629 |\n| Only MINI-conv | 28.754  | 0.911  | 0.0632 |\n\n\nIt can be seen that, the average performance improvement was only around 0.1 dB in PSNR. We believe this is because these parameters primarily perform affine and scaling transformations among features, and such transformations tend to exhibit limited variation across tasks within the same network architecture. Therefore, even without explicit parameter isolation, these components do not lead to severe catastrophic forgetting.\n\n$\\textbf{For weakness 4:}$\n\nIn fact, we have already included comparison results with PromptIR in Table 2, as it is a well-established all-in-one image restoration method. In addition, we compared our approach with several more recent and advanced image restoration methods, such as DA-CLIP [1] and IR-SDE [2]. The results demonstrate that MINI consistently improves the overall performance of these methods.\n\nThe methods you mentioned, L2P [3] and DualPrompt [4], are incremental learning approaches based on transformer architectures, whereas our MINI is currently designed for CNN-based architectures. Therefore, it is unfortunately not feasible to conduct comparison experiments under the same backbone. Nevertheless, we still compared the MINI-based NAFNet with the models reported in the original papers of these two methods as a reference. Similarly, we conducted comparison experiments on the five tasks considered in this paper, and the results are as follows:\n\n| Method       | ave PSNR ↑ | ave SSIM ↑ | ave LPIPS ↓ |\n|------------|--------|--------|--------|\n| L2P [3] | 22.067  | 0.812  | 0.117 |\n| DualPrompt [4] | 21.897  | 0.795  | 0.124 |\n| MINI | 28.754  | 0.911  | 0.0632 |\n\nThe incremental training order is (rain → haze → blur → raindrop → low light). It can be clearly observed that, thanks to our hard parameter isolation design, our MINI achieves significantly better final performance. Following your suggestion, we will incorporate the comparison results with the aforementioned transformer-based incremental learning methods into Table 3.\n\n$\\textbf{For question 1:}$\n\nMINI is specifically designed to effectively address the problem of parameter interference in multi-task image restoration. As discussed in the Discussion section, it requires predefining the maximum number of tasks $T$. In contrast, open-ended incremental learning presents a much greater challenge. Within the MINI framework, supporting such a setting would likely require a dynamically adjustable embedding pool. Based on insights from previous studies, such dynamic adjustments often necessitate the introduction of additional subnetworks (e.g., Dynamic Convolution [1]), which inevitably increases computational overhead. If such subnetworks are extensively integrated into the model, the additional cost becomes non-negligible, violating MINI’s original design goal of lightweight efficiency. Furthermore, the inclusion of such subnetworks would break the “hard parameter isolation” principle that MINI is built upon. Therefore, it is important to clarify that MINI is designed specifically for incremental learning in a fixed-task setting, rather than in open-ended scenarios.\n\n$\\textbf{For question 2:}$\n\nThis is a very interesting and insightful suggestion—to extend the current single-task classification-based query module to a multi-task weighting mechanism, thereby generalizing MINI’s single-degradation restoration capability to handle mixed degradations. However, it is important to clarify that MINI is designed to address the functional challenge of incremental learning in multi-task image restoration, rather than the performance-oriented problem of restoring images with mixed degradations. That said, as per your suggestion, we will include experimental results on the RainCityscapes dataset in the supplementary materials of our revised version, in order to clearly communicate the scope and limitations of MINI to readers.\n\n$\\textbf{For question 3:}$\n\nThe response and supplementary experimental results can be found in the section \"$\\textbf{For weakness 3}$\".\n\n$\\textbf{For question 4:}$\n\n(1) In fact, we have already integrated MINI into Restormer [6] and further improved its multi-task performance, as shown in Table 2. Although Restormer draws inspiration from the self-attention mechanism in transformer architectures, its core components are still convolutional layers. Therefore, the current MINI can be directly applied in a plug-and-play manner.\n\n(2) As mentioned in the Discussion section, extending MINI to transformer-based architectures is a key direction for our future research. In theory, task specialization of simple structures such as MLPs and multi-head self-attention (MHSA) can be achieved by properly handling the computation logic between embeddings and features. However, it is important to note that in practice, the initialization of embedding parameters plays a critical role—it can significantly affect both training efficiency and final performance. Each embedding should be initialized in a way that closely aligns with the module it is intended to replace (e.g., convolution, MLP, etc.). This sometimes requires manual modification of the initialization logic in PyTorch’s $nn.Embedding$ implementation.\n\n$\\textbf{For question 5:}$\n\nAs you suggested, we will include experimental comparisons with L2P and DualPrompt in Table 3 of the revised version. The response and supplementary experimental results can be found in the section \"$\\textbf{For weakness 4}$\".\n\n[1] Controlling Vision-Language Models for Universal Image Restoration, ICLR 2024\n\n[2] Image Restoration with Mean-Reverting Stochastic Differential Equations, ICML 2023\n\n[3] Learning to Prompt for Continual Learning, CVPR 2022\n\n[4] DualPrompt: Complementary Prompting for Rehearsal‑free Continual Learning, ECCV 2022\n\n[5] Dynamic Convolution: Attention over Convolution Kernels. CVPR 2020\n\n[6] Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022"}, {"review_id": "VsFGvTEcDN", "reviewer": "Reviewer_PGor", "summary": "The paper proposes MINI, a minimalistic framework for task-incremental learning in image restoration, addressing the challenge of catastrophic forgetting and parameter conflict in existing multi-task models. MINI introduces a novel Meta Convolution module (MINIconv) that generates task-specific parameters via lightweight embedding pools, enabling complete parameter isolation across tasks without additional computational overhead. This design allows seamless integration into existing backbones, transforming them into incremental learning systems. A degradation query mechanism classifies input degradations to select appropriate embeddings, while embedding-consistency regularization enhances robustness against misclassification.", "questions": "Q1:The MINI architecture requires pre-defining the embedding pool size T to accommodate different tasks. However, real-world applications may face an unknown or unbounded number of tasks. Can the authors clarify whether MINI can dynamically extend T post-deployment? If not, what practical upper bound do the authors recommend for T considering memory and computational trade-offs?\n\nQ2:The degradation query module relies on accurate task classification. While the paper discusses robustness via ECR, in practice, real-world degradation can be ambiguous or unknown. Can the authors elaborate on MINI's performance when encountering degradations outside the predefined set?\n\nQ3:The paper mentions potential future extensions to Transformer-based backbones but lacks preliminary results or analysis. Can the authors provide insights or experiments indicating MINI's compatibility or limitations with Transformer or hybrid architectures?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.This work represents the first systematic attempt to explore task-incremental learning in the field of image restoration.\n\n2.The proposed architecture is highly minimalistic, requiring only the replacement of standard convolutional layers with MINIconv and the addition of a lightweight degradation query module. It can be seamlessly integrated into existing mainstream backbone networks.\n\n3.The method enables efficient and nearly forgetting-free task-incremental learning, while avoiding the parameter explosion issues typical of multi-head approaches.\n\n4.The introduction of Embedding Consistency Regularization (ECR) effectively enhances the system's robustness and fault tolerance, particularly when the degradation classification is inaccurate.\n \nWeaknesses:\n1.The maximum number of tasks supported is inherently constrained by the pre-defined size of the embedding pool (T), making it difficult to accommodate scenarios where the total number of tasks is unknown or grows dynamically.\n\n2.Although the method avoids additional inference-time computation, hard parameter isolation causes the total number of model parameters to grow linearly with the number of tasks, which may present challenges for deployment in resource-constrained environments.\n\n3.The current work primarily focuses on CNN-based architectures, and its generalization to other popular frameworks such as Vision Transformers or Diffusion Models remains unexplored.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our work. We will do our best to address your questions and concerns.\n\n$\\textbf{For weakness 1:}$\n\nThe core motivation behind the design of MINI is to address parameter interference across different tasks through hard isolation, thereby enabling strong incremental learning capabilities within a known set of tasks—especially under imbalanced training data conditions. The scenario you mentioned, involving an unknown number of tasks, actually falls under a much more challenging setting: open-ended incremental learning, which is not the primary focus of this paper. Nonetheless, it remains a practically important problem and will be one of the key directions in our future research.\n\n$\\textbf{For weakness 2:}$\n\nYes, this viewpoint has already been discussed in the “Limitation of MINI” section. However, we believe that this trade-off is entirely acceptable compared to the benefits brought by MINI—achieving nearly forgetting-free incremental learning with almost no additional computational cost. Moreover, for most models, the majority of GPU memory consumption comes from the size and number of intermediate feature maps, with only a small portion attributed to model parameters themselves. \n\nFor example, the well-known ResNet50 [1] has only 25.6M parameters (approximately 100MB of memory in fp32), yet when training with 224×224 images and a batch size of 32, the total GPU memory usage reaches around 1.5–2GB. This indicates that the vast majority of memory is used to store intermediate features (around 95%), and this proportion increases further as the input image resolution grows. \n\n$\\textbf{For weakness 3:}$\n\nWe are pleased to see that most reviewers seem interested in the potential extension of MINI to transformer-based architectures. Therefore, in the revised version, we will apply MINI to a representative transformer-based image restoration method, SwinIR [2], and include the experimental comparison results in the appendix (due to the limited space in the main paper). This will help evaluate the scalability of MINI within transformer-based frameworks. We are currently conducting related experiments. If you remain interested during the rebuttal phase, we would be happy to share the experimental results with you.\n\n$\\textbf{For question 1:}$\n\n(1) In practical deployment scenarios, if the embedding pool needs to be extended to accommodate new tasks after the model has already been deployed, this can be easily implemented at the code level. For example, if we want to extend a trained MINI-based model from $T=5$ to $T=7$ tasks, we only need to initialize a $T=7$ model and load the pretrained weights for the first 5 embeddings. This can typically be achieved with a simple line of code:\n\nmodel.load_state_dict(state_dict, $\\textbf{strict=False}$).\n\nSubsequently, the newly added embeddings can be trained while keeping the existing ones fixed.\n\n(2) Since the setting of $T$ does not affect the inference-time computational cost, we only need to consider the choice of the main model and the number of tasks. For scenarios where the number of tasks is uncertain, we recommend setting a relatively large $T$ (within the limits of hardware capacity) to accommodate potential future tasks. In contrast, if the number of tasks is relatively fixed, $T$ can simply be set to the required number of tasks.\n\n$\\textbf{For question 2:}$\n\nThe issue you raised is a common challenge in the field of image restoration—how to improve the model's out-of-distribution (OOD) robustness for better generalization to complex real-world scenarios. As a plug-and-play method, MINI is designed to enhance the incremental learning capability of the backbone model. Therefore, its generalization ability largely depends on the architecture, capacity, and training strategy of the backbone itself. Therefore, we believe that the OOD robustness of MINI mainly depends on the training strategies used when learning each individual embedding (e.g., regularization), as well as the inherent generalization ability of the backbone model.\n\n$\\textbf{For question 3:}$\n\nSame as response to $\\textbf{For Weakness 3}$.\n\n[1] Deep Residual Learning for Image Recognition, CVPR 2016\n\n[2] SwinIR: Image Restoration Using Swin Transformer, ICCV 2021"}, {"review_id": "lpeUIuV6Ru", "reviewer": "Reviewer_G6kR", "summary": "The authors propose MINI (Minimalistic Incremental Network for Image Restoration), a novel and lightweight framework tailored for task-incremental learning across various image restoration tasks. To enable effective hard parameter isolation, they introduce MINIconv, along with a lightweight degradation query module and an embedding consistency regularization (ECR) strategy.", "questions": "Please refer to the weaknesses section. My current rating could change significantly depending on the authors’ responses. While I have some knowledge of image restoration, I do not have specialized expertise in the specific field of this study (e.g. meta-convolution, All-in-One IR, ...). I am aware of this and will actively refer to other reviewers’ questions and the authors’ responses as well.", "limitations": "The limitations and future work are clearly addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n1. This work is the first attempt to explore task-incremental learning in the field of image restoration.\n2. The idea is performed well.\n3. The paper is well-organized and easy to read.\n4. The limitations are clearly and well mentioned.\n\n[Weaknesses]\n1. In line 258, the authors state that \"its use of hard parameter isolation causes the total number of parameters to scale by a factor of T.\" If this is the case, isn’t there a significant difference in terms of forward computation overhead or the number of parameters compared to multi-head methods?\n\n2. In the same context as the previous question, I am curious about the performance differences and differences in parameter counts between MINI and a scenario where a learned query function selects task-specific (one-for-one) methods for each task.\n\n3. What is the performance when only the Degradation Classification MLP is used in the query function, excluding the Global MLP and Local MLP?\n\n4. I am curious about the actual number of parameters in each case listed in Table 2. Additionally, what is the parameter count and inference time of the MLP-based query function? \n\n5. Minor error.\n-- L17: overll >> overall\n-- Table 2: It would be visually more effective to use color for emphasis rather than bold text.", "was_revised": true, "final_justification": "I believe the paper has sufficient merit for publication, and I encourage the authors to address the concerns raised in a revised version.", "rebuttal": "We sincerely appreciate your thorough review and insightful comments. We will make every effort to address all of your concerns and questions.\n\n$\\textbf{For weakness 1:}$\n\nAs we stated in the Introduction (lines 37–41), although multi-head methods achieve complete parameter isolation across tasks, their computational cost (FLOPs), number of parameters, and the amount of intermediate features grow linearly with the number of tasks. This ultimately results in a linear increase in inference time and a significant rise in computational resource consumption, which is often unacceptable in real-world applications.\n\nIn contrast, our proposed MINI maintains constant FLOPs and the amount of intermediate features regardless of the number of tasks, meaning that the inference time remains unchanged as new tasks are added. Although the number of model parameters still grows linearly with the number of tasks, it is well known that the majority of GPU memory consumption in modern deep models is dominated by the storage of intermediate features, while model parameters occupy only a small fraction of the memory. Therefore, we believe that the slight increase in memory usage brought by MINI is a worthwhile trade-off for achieving near “no-forgetting” performance in incremental learning.\n\n$\\textbf{For weakness 2:}$\n\nWe are glad that this question was raised, as the “query-to-model” approach you mentioned was indeed considered during the early stages of our research and partly inspired the development of MINI! Below, we provide a detailed explanation of the differences between this strategy and our proposed MINI:\n\n(1) Although the \"query-to-model\" approach also achieves parameter isolation and does not introduce additional inference overhead, it intuitively resembles a multi-head architecture. As a result, the overall pipeline becomes less elegant, which goes against our minimalist design philosophy.\n\n(2) Although the \"query-to-model\" strategy enables complete isolation of all parameter components, as we discussed in the Discussion section (lines 242–249), it is in fact unnecessary to strictly isolate non-convolutional components (e.g., LayerNorm) whose learnable parameters have minimal impact on incremental learning performance. Replacing these components with task embeddings may not introduce a large number of additional parameters, but it would make the overall architecture less streamlined and violate our minimalist design principles.\n\n(3) In most practical applications, image restoration—being a low-level vision task—typically serves as a single functional module within a larger system (e.g., autonomous driving), providing high-quality inputs for subsequent high-level tasks such as object detection or semantic segmentation. In such scenarios, practitioners often prefer to use a single model, like our proposed MINI, to accomplish this functional block. This approach facilitates more efficient system management and optimization, rather than deploying multiple sub-models within a single functional module. Similarly, for scenarios where the algorithm needs to be deployed on edge devices (e.g., smartphones), deploying a single “one-for-all” model at once is a more practical choice compared to deploying multiple sub-models, as it better aligns with real-world engineering considerations.\n\nOverall, we ultimately chose to design the MINI architecture and abandon the “query-to-model” approach out of considerations for algorithmic simplicity and engineering practicality. At the same time, we believe that the minimalistic nature of MINI also facilitates future improvements and upgrades based on this framework.\n\n$\\textbf{For weakness 3:}$\n\nIn our early experiments, we attempted to build the degradation classifier using either a single MLP or a series of MLPs in sequence. However, we found that the classifier consistently struggled to distinguish certain types of degradations, such as blur and fog. To address this issue, we proposed a two-stage degradation classification mechanism — performing coarse classification first, followed by fine-grained classification — which led to the design of our current query module. This simple yet effective change improved the classification accuracy by approximately 15%.\n\nWe will include the above content in Section 3.2 of the revised version to better clarify our design motivation, and we will also add an ablation study of the query module in Section 4.2. The detailed results are as follows:\n\n|  method  | degration classify acc.|\n|------|----------|\n| Sequential MLPs |     78.3%     |\n| Our query module |     94.6%     |\n\nIt is worth noting that the Sequential MLP maintains approximately the same number of parameters as our query module.\n\n$\\textbf{For weakness 4:}$\n\nThe parameter counts of the all-in-one baseline methods listed in Table 2 are as follows:\n\nAirNet 8.93M\n\nPromptIR 36M\n\nMAXIM 14.1M\n\nNAFNet 19.3M\n\nRestormer 26.09M\n\nIR-SDE-Unet 137.14M\n\nDA-CLIP-Unet 137.14M\n\nThe corresponding MINI versions are constructed by setting the number of task-specific branches to $T=5$ in our experiments. As a result, the total parameter count of each MINI-based variant is approximately $T$ times that of its original backbone. Our lightweight query module contains only 0.11M parameters, with an inference time of few millisecond on an NVIDIA 2080Ti GPU.\n\n$\\textbf{For weakness 5:}$\n\nThank you for your careful reading and valuable suggestions. In the revised version, we will thoroughly review and correct all typos. As you suggested, we will also change the bold fonts in Tables 2 and 3 to red highlights to improve readability for the readers."}]

Author rebuttal:

Meta-review: The manuscript investigates the problem of task-incremental learning for image restoration. To this end, the manuscript introduces a lightweight framework built around a meta-convolution module that substitutes standard convolutions in existing backbones. The proposed module leverages task-specific embedding pools to obtain hard parameter isolation. The proposed design introduces negligible computational overhead at inference, while enabling near-forgetting-free incremental learning. The manuscript received ratings of four borderline accept. Reviewers appreciated the problem setting (addressing the gap between one-by-one and all-in-one paradigms), effectiveness of the proposed design, plug-and-play design compatibility with existing image restoration backbones, and extensive experiments  (e.g., deraining, dehazing, deblurring, etc.).  Reviewers also raised several questions including, task scalability and comparison with prompt-based continual learning methods. Post-rebuttal, reviewers maintained their positive rating with reviewers mentioning that most of the concerns are addressed. Given that all reviewers are positive, rebuttal from the authors, and author-reviewer discussions, the recommendation is accept.
Decision: Accept (poster)