Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Authors: Fengqiang Wan
Yang Yang
Venue: ICML 2025 poster (2025)
Keywords: Incremental Learning
Primary area: general_machine_learning->everything_else
TL;DR: We propose Probabilistic Group Mask selection (PGM), an incremental learning method that captures parameter dependencies through group-wise mask selection, improving sub-network allocation and mitigating catastrophic forgetting.

Abstract: Incremental learning (IL) aims to sequentially learn new tasks while mitigating catastrophic forgetting. Among various IL strategies, parameter-isolation methods stand out by using mask techniques to allocate distinct parameters to each task, explicitly addressing forgetting. However, existing approaches often disregard parameter dependencies, resulting in an over-reliance on newly allocated parameters. To address this issue, we propose Probabilistic Group Mask selection (PGM), a group-wise approach that captures parameter dependencies by exploring candidate masks within each group. Specifically, PGM partitions parameters into groups with multiple candidate masks, assigning probabilities to these masks and leveraging Gumbel-Softmax for differentiable sampling, enabling efficient optimization of the discrete mask selection process. Our theoretical analysis demonstrates that incorporating parameter dependencies enhances sub-network selection. Experiments conducted on standard benchmarks confirm its superior effectiveness compared to existing IL approaches. The source code is available at: \url{https://github.com/njustkmg/ICML25-PGM}.

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Fengqiang Wan
Yang Yang'
Abstract
00655
Ds
Incremental learning (IL) aims to sequentially learn new tasks while mitigating catastrophic for- getting. Among various II strategies, parameter- isolation methods stand out by using mask tech- niques to allocate distinct parameters to each task, explicitly addressing forgetting. However, existing approaches often disregard parameter dependencies, resulting in an over-reliance on newly allocated parameters. To address this is- sue, propose Probabilistic Group Mask se- lection (PGM), group-wing wise approach that cap- tures parameter dependencies by exploring can- didate masks within each group. Specifically, PGM partitions parameters into groups with mul- tiple candidate masks, assigning probabilities to these masks and leveraging Gumbel-Softmn for differentiable sampling, enabling efficient opti- mization of the discrete mask selection process. Our thcoretical analysis demonstrates that incor- porating parameter dependencies enhances sub- network selection. Experiments conducted on standard benchmarks confirm its superior effec- tiveness compared to existing approaches. The source code is available at: https: // github. com/njustan 25-
0193
Deperalnt
.0
.0.08
nets matai 9o9 8o
Tadk
Capacity
Lmased Parureter
Deperdent Parader
(a) Isolation Dependency
Performance
Figure I. Illustration of different parameter selection strategies. (a) collatio-basee methods evaluate parameters independently and select the top-ranked ones, while dependency-awarg methods consider parameter interactions, enabling joint selection based their collective contributions. (b) The dependency proach, exemplified by PGM, demonstrates enhanced ACC and CAP compared to isolation-hasee haased methods like WSN, underscor- ing the advantages incorporating parameter dependencies.
ap-
ing in major challenge in called Catastrophic Forgetting (CF) (Lange al., 2022: Zhou al., 2022), which refers to the drastic performance drop on past knowledge after learning new knowledge.
Several promising approaches have been proposed for IL that focus on mitigating CF (Hung et al., 2019; Kumar et al., 2021; Yoon et al., 2018a; Liu et al., 2024; Yang et al., 2019; 2023b). Among these, parameter isolation achieves complete prevention of catastrophic forgetting (Wortsman et al., 2020; Serrà al., 2018; Hu et al., 2024), by masking distinct sub-networks for each task within shared model architecture. However, a key challenge lies in determining the optimal allocation of parameters for each task, balanc- ing the preservation of prior knowledge with adaptation to new tasks. Several approaches rely on handcrafted crite- to assess parameter importance, such as the absolute magnitude of the weights (Mallya & Lazebnik, 2018) gradient information (Konishi et al., 2023), but these prox- ies lead to suboptimal allocations due to not considering their true contributions to the task. Alternatively, parame- ters can be assigned learnable weights (Mallya et al., 2018; Kang et al., 2022; Hu al., 2024), which are updated via
1. Introduction
Incremental Learning (IL) enables models incrementally acquire new knowledge, making it highly applicable to do- mains such as autonomous driving (Fang et al., 2024a) and healthcare (Lesort et al., 2020). A good IL model is ex- pected to keep the memory of all seen tasks upon learning new knowledge (Hocquet, 2021; Yang et al., 2023a). How- ever, due to the limited access previous data, the learning phase is naturally sensitive the current task, hence result-
NNajjin University Science and Technology, Correspon- dence to: Yang Yang <yyang @ njust.edu.en>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
gradients during training to more effectively capture their true importance, enabling more precise parameter alloca- tion. However, existing parameter selection methods, as illustrated in the left part of Figure la, independently score and rank each parameter, offering simplicity but neglect- ing critical dependencies among them. As established in Theorem 3.2, neglecting parameter dependencies and their collective contributions leads to suboptimal sub-network. In contrast, dependency-awann -awaee methods, shown in the right part of Figure la, explicitly account for parameter dependencies, facilitatingg joint selection that optimizes their combined con- tributions. According to Definition 3.1, incorporating these dependencies throughout the learning sequence enhances parameter reuse, reducing the need for newly introduced parameters in each task. As demonstrated in Figure lb, this dependency-awarc reuse enables each task to achieve comparable or superior performance with fewer additional parameters, resulting in more compact sub-networks. This reduction in model expansion preserves more capacity for future tasks, facilitating better adaptability to unseen tasks while maintaining scalability.
In TIL, parameter isolation methods have proven to be highly effective in mitigating catastrophic forgetting. These methods leverage the over-parameleriiztton of neural net- works to create task-specific subnetworks, isolating the pa- rameters dedicated to each task. This approach minimizes interference between tasks and preserves performance on previously learned tasks, while still allowing for flexibil- ity in learning future tasks. Mathematically, for given set of model parameters 0, a binary mask m, defines the subnetwork for task 4, constrained by ||m'|| S c, where represents the subnetwork's 's capacity. The optimization problem is formulated as:
arg min 1 The m,E(0.1) Tig
m
C(F(1)0 mf).
- -(F((::)). s.t. || S c,
where L denotes the task-specifie loss function, denotes elementwise multiplication. The goal is to find the optimal mask m that isolates the task-specific subnetwork, ensuring that it achieves task performance comparable to the full network, while adhering to the capacity constraint.
Therefore, we propose Probabilistic Group Mask selection (PGM), which models the mask selection process a prob- abilistic sampling process. Specifically, PGM organizes parameters into groups, with each group containing mul- tiple candidate masks, and assigns a probability to each candidate mask. This probabilistic sampling process is repa- rameterized using the Gumbel-Softmax technique (Jang et al., 2017), enabling a deterministic approximation of the selection process and facilitating end-to-end optimization through gradient. Our theoretical analysis demonstrates that incorporating parameter dependencies improves sub- network selection by capturing synergies between parame- ters, leading to more effective utilization of model capacity. Extensive experiments on widely-used datasets demonstrate that our method achieves competitive performance.
3. Method
3.1. Theoretical Understanding
To enhance subnetwork selection, propose group-wise strategy that incorporates parameter dependencies, improv- ing both parameter reuse and error reduction. By structuring the parameter selection process into groups, this approach facilitates the joint evaluation of parameter dependencies within each group, leading to more accurate and effective subnetwork optimization.
Definition 3.1 (Parameter Reuse with Dependency). Let K be the set of parameters, and let parameter dependencies that influence their reuse across tasks. The total parameter reuse R across N tasks is defined as:
2. Preliminaries
[0]
N
iask-incremental learning (TIL) is a key paradigm in incre- mental learning, where the task identity í is available during the inference phase. This enables the model to specialize its processing to each task as it learns sequentially from stream of tasks {"i, T2,... .N..
Σ
m4,i > >0)
where II()) is the indicator function, and m,, denotes the ef- fective mask for parameter 0g in task f, which determined by the joint contribution of K parameters:
J. For each task T,, the model F(.,0) 0), with parameters 0, learns from the dataset D, (Xe,, Yi). where X, is the input feature set and Y, is the corresponding label set. The objective for task t is to minimize the task-specific loss, expressed as:
((
$1,1 + Σ Pos
>7) )
my,i =I
JEAR
1 ((F(2)))) 0) v/).
The
In this definition, St.i represents the independent score of parameter 0, for task t, Pj. quantifies the dependency be- tween parameter 6; and f,, and A²K is the set of K - 1 parameters most strongly interacting with A,. The threshold T determines the selection criterion for parameter reuse.
0- = arg min nt
where L represents the loss function and ny is the number of samples in D,,
2

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
According to Definition 3.1, when K 1, the dependency term is absent, resulting in parameter selection based solely on St,i, which corresponds to independent selection. In contrast, when K > 1, the joint evaluation of dependencies with K - additional parameters increases the probability of reusing 0g across tasks, thereby achieving a higher R.
model comprises numerous parameter groups, denoted as Ot, each necessitating the selection of corresponding mask Mr. To identify an optimal subnetwork, Equation (I) can be reformulated as:
{Mi)+ = arg min
C o (Mi)).
Theorem 3.2 (Error Reduction via Dependency). Divid- ing N parameters into N/K groups of size K reduces the uncertainty in parameter selection by leveraging local inter- actions within each group. The probability of an incorrect selection for parameter combination C under the group- wise strategy is given by:
(MIMESKK Th
(F((a)))). vi)
(2)
where the operator @ represents elementwise multiplication.
Identifying the optimal set of masks {Mr} poses signifi- cant challenges due to the nondifferentiable nature of mask selection and the high dimensionality of model parameters. To address these difficulties, the mask selection problem is reformulated as a sampling-basee process, thereby en- abling efficient optimization while preserving the capacity for task-specific adaptability.
N/K
(Lies; TS( - Eiec, TS(O,)
Parron (C) = II
VKo
b=1
where S denotes the ideal parameter combination within group b, Co represents the selected parameter combination, TS(.) is the ideal score, a is the standard deviation of the noise, and (.) denotes the cumulative distribution Iistributionfuunctinn function.
Sampling Process: Let W E RIIK denote a parameter groups containing K parameters. Identifying the exact op- timal mask for such a group is inherently complex due to interdependencies with other parameter groups. To address this, an independent sampling approach employed for each group, providing tractable and efficient means of eval- uating overall model quality (Fang et al., 2024b). Empirical studies further demonstrate that model performance tends to stabilize when K surpasses a certain threshold, highlighting the scalability of the proposed sampling strategy.
By explicitly modeling parameter dependencies, the group- wise approach reduces evaluation error by capturing interac- tions that contribute to task-specific performance. Theorem 3.2 demonstrates that incorporating dependencies into the selection process lowers the likelihood of errors by ensur- ing a more informed evaluation. Structuring parameters into groups facilitates this joint consideration, avoiding the limitations of independent evaluations and leading more accurate subnetwork selection. The detailed proof can be found in the Appendix A.
To enable efficient sampling of M, two categorical distri- butions are defined: pl, P1, PT. which determine the number of nonzero elements, and qj1. 9/2; ................................ lyk. which select specific mask from the candidate set. These distribu- tions are normalized such that Σ, Pj and Σx 9j* 1. Masks that demonstrate strong performance are assigned higher probabilities, guiding subsequent sampling toward promising candidates. Through iterative sampling and prob- ability updates, the process converges to distribution that prioritizes nigh-performing masks, thereby increasing the likelihood of discovering optimal subnetwork. This refor- mulates Equation (2) as probabilistic sampling problem:
3.2. Probabilistic Group Mask Selection
Building the theoretical results above, the challenge of dividing K parameters into groups is reframed as the task of selecting optimal subsets within each group to achieve spe- cific objectives. This method introduces parameter grouping, reformulates the problem as a sampling process, and utilizes Gumbel-SSfftnt for efficient optimization. Furthermore, an adaptive initialization leverages the transferability of prior task distributions facilitate task-specific adaptations.
The Σ (())e( Th 2m1
{q" (M, 15)h = arg min
Consider a parameter group consisting of K parameters, denoted as W E RR x K The goal is to determine the optimal binary mask M E BIXK of the same dimension. This leads to a discrete candidate set Sk which contains Σ K Σ (5) possible masks, expressed as:
Parameter Group:
[(FF(aus0 (Ma,)), CFF(r, 0), vi.e)
Here, p(j) defines the probability distribution over the num- ber of nonzero elements in a mask, and g(M, j) rep- resents the conditional probability of selecting a specific mask MGs given j. While gradient descent can be applied to optimize objectives when gradients are available, the in- herently nondifferentiable nature of categorical sampling
K-1 U [Mab}, f=1
S = (M E BIxK)=
where {M denotes the set of masks with exactly j nonzero elements, and indexes the specific subsets. The
3

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
presents significant challenges. This issue arises because sampling from categorical distribution involves discrete operations, which lack the gradient information required for optimization, thereby impeding the direct use of standard optimization techniques.
efficiency and flexibility, this approach facilitates precise mask selection while maintaining scalability. For task t, the optimal mask is obtained by selecting the combination of and i that maximizes the joint probability of p(j) and q Mak j), formally expressed as:
Differentiable Sampling of Masks: To address the two- stage sampling problem involving p(j) and (M,k j)) the Gumbel-Softmax reparameterization technique employed. This method facilitates differentiable sampling by providing a continuous approximation to the discrete process. Specif- ically, for p(i), the discrete sampling is relaxed into the following differentiable form:
M;- = [Mzx arg max ( ())-
(5)
By design, the mask selection process adapts dynamically to task-specific requirements, focusing on the most relevant parameter subsets. This adaptability not only enhances inter- pretability but also reduces the risk of suboptimal selections, ensuring effective utilization of model capacity across tasks.
Insk-enf Mask Initialization: To effectively ini- tialize the probability distributions p(f) and q(Mss j) for task t, this method incorporates the similarities be- tween the current task mask and those of all previous tasks (M1 My -1) By leveraging these similarities, the initialization process allows previously learned knowledge to directly influence the sampling process. This approach not only facilitates efficient parameter reuse but also ensures that the model retains adaptability to task-specific require- ments. The contribution of each prior task is weighted based on its similarity to the current task, thereby shaping the prob- ability distributions in a task aware manner. To quantify this influence, the similarity between current task mask M, and prior task mask M, (n E {I....,t 1}) is computed as:
exp((log((())++ gj)/T)) p(i) ES exp((log(((i)) + my m/m)
where gj are independent samples from the Gumbel dis- tribution - -oog(- log(U)), with U Uniform(O, 1), and T is temperature parameter that controls the degree of approximation. As Tı 0, P(i) approaches one-hot vector. Similarly, the sampling of specific mask M₁ for {(Msk | ()) can be relaxed using the same reparameteriza- tion technique. The relaxed form is expressed as:
exp(()ogg(MMja 15)) + 92) {M, (0) En ( ()) + )/m2))
where 9k introduces stochasticity, and T2 controls the smoothness of mask selection. These reparameterized forms ensure that the sampling process remains differentiable, en- abling efficient optimization of p(j) and (M,jk () using gradient-based methods.
sim(M,, M.) = M, |S| (..M))
In this formulation, the similarity is measured using the inner product between M,, and M,, with adjustment to remove the mean similarity across all candidate masks. This adjustment is critical as it normalizes the similarities, re- ducing the influence of global variations across the mask space and focusing on meaningful relationships between the current and prior tasks. Building on this similarity measure, the initialization process prioritizes candidate masks with higher similarity to previous tasks by adjusting their prob- abilities accordingly. Specifically, the logits for p(j) and (MMa j) are updated using the aggregated similarity:
Using these reparameterized distributions, the final differ- entiable mask M is constructed by integrating the soft se- lection over j with the conditional distribution over masks. This results in a weighted combination of the candidate masks, expressed as:
M -ME=ad (UI Σ a(Myu M5)
(4)
The reparameterization ensures that the sampling process is fully differentiable, allowing p(j) and g( MMx []) to be opti- mized through standard gradient- methods. Instead of directly optimizing these probabilities, their corresponding logits n, and の are learned. Sealing factors Kn and K2 are applied to adjust the sharpness of the distributions, with the probabilities computed as follows:
n. = me
sim (Ma, M.) ((x)) .01:
8. = 01 sim (M,, M.) - a(d))) .2. a2.
Here, (TT)) and (Φ))) denote the standard deviations of the logits for task n, and the hyperparameters an and a: determine the extent to which prior task information influ- ences the current initialization. By incorporating both the similarity and variability of prior task distributions, the ini- tialization process achieves a balance between knowledge transfer and task-specific differentiation.
eXP(TT - K1) p(i) = Σy exp(TTy - k1) ((M)x1)) = Σx exploX) K2)
exp(0 K2)
This formulation enables the construction of the differen- tiable mask M, efficiently addressing the sampling prob- lem outlined in Equation (3). By balancing computational
4

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Table 1. Performance comparison of PGM with other II. methods across three datasets. Under the same conditions, ACC is prioritized over BWT, as it reflects the overall balance between network stability and plasticity, whereas BWT solely quantifies the degree of forgetting.
CIFAR-100 Split
CIFAR- 110 Superclass
TinyImageNet
Method
ACC (%) f
CAP ($))
BWT f
ACC (%) f
CAP (S))
BWT 1
ACC (%) f
CAP (%)
BWT (%) 1
Multitask
79.75 (+00388
100.0
61.00 (=0.200
100.0
77.10 (#11 (#:1.0)
100.0
La-MaML
71.37 (=0067)
100.0
(20.53)
54.44 (=1.36)
100.0
-6.65 (40.85)
66.90 (#1166)
100.0
99.13 (#0.99)
GPM
73.18 (=0,52)
100.0
-1.17 (70227)
57.33 (=0.37)
100.0
-0.37 (20.12)
67.39 (#0.47)
100.0
-1.45 (#0.22)
PS-DGPM
74.33 (=10.41)
100.0
-2.71 (1:0.11)
58.81 (=0.34)
100.0
-2.97 (+0035)
70.41 (+7)
100.0
-2.11 (#0.84)
RP2F
75.89 (1:0455
100.0
-0.82 13)
60.11 (=0.23)
100.0
-0.32 (+0.02)
70.32(+0.29)
100.0
-1.05 (#0.4)
PackNet
72.39 (=00377 96.38 (20.38)
0.0
58.78 (=0.52) 126.65 (+0.00)
0.0
$5,46 (+ 1.22) 188.61 67 (10.00)
0.0
SupSup
75.47 (+00333)
129.00 (+H0.00)
0.0
61.70 1.31)
162.49 (+0000)
0.0
59.60 (+ 1 005
214.52 (+0.89)
0.0
WSN
76.38 (=3.33)
99.13 (8))48)
0.0
61.79 (=23)
80.93 (+1558)
0.0
69.06 (+ 82)
92.03 (#:1.0))
0.0
SPG
77.12(+0:42) 90.24 (+0.89) 1.02 (4:0.38)
62.04 (=0.39)
75.93 (+0097) -0.53 (#:017)
70.16 (=0074)
85.53 (+1.19) -1.26 (=0.17)
PGM
77.29(+0.31)
(3.51
0.0
62.56( + 0.71)
50.38( 1.05)
0.0
70.99( + 0.54)
80.261 + 0.86)
0.0
4. Experiments
regularization techniques such as La-MAML (Joseph & Gu, 2021), GPM (Saha et al., 2021), FS-DGPM (Deng et al.,
We validate our method across multiple benchmark datasets, comparing its performance against relevant incremental learning baselines. For all experiments presented in this paper, we utilize TIL with a multi-head configuration. The experimental setups are carefully aligned with those em- ployed in recent works (Kang et al., 2022). The evaluation encompasses comprehensive performance comparisons, ab- lation studies examining module effectiveness, group size variations, and adaptability to different training paradigms, as well as an in-depth analysis of computational efficiency, parameter dependencies, and parameter distribution.
2021), RP2F (Sun et al., 2024), and MTD (Wen et al., 2024); and (3) the naive sequential training strategy, referred to as FINETUNE, in addition to multitask learning (MTL), which serve to establish lower and upper bounds for performance.
Implementation Details: To ensure a fair comparison, we follow the experimental protocols from (Kang et al., 2022), maintaining consistent backbone architectures across datasets. For instance, we use a modified AlexNet for Split CIFAR-100 and a customized LeNet for CIFAR-100 Su- perclass. Training employs the Adam optimizer with mo- mentum of 0.9, with each task trained for fixed number of epochs to ensure convergence. Experiments are conducted using Py PoTrch on a high-perffrmance computing platform with NVIDIA 4090 GPUs Additional hyperparameter set- tings are provided in the Appendix B.
4.1. Setup
Datasets and Evaluation Metries: We use three different popular datasets, including Split CIFAR-100 (Krizhevsky & Hinton, 2009), CIFAR- 100 Superclass (Yoon et al., 2018b), Split TinylmageNer (Krizhevsky et al., 2017). To evaluate IL methods, we evaluate all methods on three metries: ACC, CAP, BWT. ACC measures the average classification per- formance across all tasks, defined ACC = 1 ΣL Ar. where Ar, represents the test accuracy for task i after train- ing on task T. CAP quantifies the proportion of nonzero weights and prime masks used for all tasks, calculated as CAP- = (1 - S) + (1 - T)T x 32. where S is the sparsity of Mr and a (approximately 0.78) is the aver- age compression rate achieved through 7- Huffman en- coding. Finally, BWT assesses the degree of forgetting in incremental learning scenarios, expressed as BWT T-1 EL' -A.)
4.2. Main Results
The performance of the proposed PGM is rigorously compared with several sfate-of-the th-the- methods across three key evaluation metrics on three promi- nent benchmark datasets, as presented in Table 1. PGM con- sistently outperforms all existing approaches, achieving the highest ACC of 77. 77.29%, 62.56%, and 70.99%, respectively. These results demonstrate the efficacy of PGM in address- ing the challenges associated with IL. In particular, as high- lighted in Table 1, (1) In comparison to parameter regulariza- tion methods, both PGM and parameter isolation approaches exhibit zero forgetting, effectively preventing catastrophic forgetting across tasks. Notably, PGM achieves superior ACC, suggesting that it not only mitigates forgetting but also enhances overall model performance; (2) When com- pared to parameter isolation methods, PGM achieves better ACC with a smaller set of parameters, indicating a more effi- cient use of model resources. As shown in Figure 2a, PGM outperforms WSN in terms of accuracy across most tasks,
Overall Performance:
1.
1
Baselines: To thoroughly evaluate the performance of the proposed PGM, a comparison is conducted with several prevalent incremental learning baselines. Specifically. (1) parameter isolation methods, including PackNet (Mallya &
Lazebnik, 2018), SupSup (Wortsman et al., 20d WSN (Kang et al., 2022), SPG (Konishi et al., 2023); (2) parameter
5

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Table Results of ablation studies highlighting the contributions of the Group and Mask Initialization modules across CIFAR-100 Split,
CIFAR-100 100 Superclass, and TinylmageNet datasets. The table reports accuracy (ACC) and capacity (CAP) metries, demonstrating the impact each component on performance and resource utilization.
CIFAR-100 Split
CIFAR-100 Superclass
TinylmageNet
Group
Mask Initialization
ACC (%) f
CAP (%) 4
ACC (%) t
CAP (%) J
ACC (%) 1
CAP (%)
x
x
76.38
99.13
61.79
80.93
69.06
92.03
x
76.98
65.97
62.04
54.25
70.13
82.39
/
77.29
63.51
62.56
50.38
70.99
80.26
further reinforeing the effectiveness of the approach; (3) Fur- thermore, Figure 2b illustrates that, throughout the entire training process, PGM exhibits a higher degree of overlap in model parameters between tasks, leading to more efficient knowledge transfer. This results in reduction in the depen- dence on newly introduced parameters, in contrast to WSN, which shows greater reliance on task-specific parameters. Such behavior suggests that PGM is more adept at retaining and reusing previously learned knowledge, thereby improv- ing its ability to generalize across tasks while minimizing the cost of learning new tasks.
mance. Theoretically, as outlined in 3.2, larger group sizes tend to yield better results. However, empirical evidence reveals that beyond a certain value of K, performance tends to plateau, as shown in Figure 2c. Furthermore, it is not req- uisite for each group to select fixed number of parameters. When K = ix, the model allows for dynamic selection of parameter sizes across different group combinations, lead- ing to enhanced performance. This flexibility contributes to more effective utilization network resources, optimizing both specifc adaptation and overall model efficiency.
Adaptability to Diverse Training Paradigms: Parameter isolation methods for mask selection can be broadly catego- rized into two primary approaches. The three-stage training paradigm, represented by PackNet (Mallya & Lazebnik, 2018), consists of sequential steps: training the entire net- work, selecting an optimal mask, and subsequently fine- tuning the selected sub-network. In contrast, the iterative training paradigm, as employed in WSN (Kang et al., 2022), integrates sub-network updates directly within the training process, enabling a more dynamic adaptation to task- requirements. As summarized in Table 3, PGM exhibits strong adaptability to both training paradigms. Neverthe- less, the three-stage training approach exhibits limitations in knowledge reuse across tasks, resulting in increased overall parameter overhead and suboptimal resource efficiency. In contrast, the iterative training approach not only achieves superior accuracy but also significantly reduces capacity utilization, underscoring its effectiveness in optimizing pa- rameter efficiency and computational resources.
4.3. Ablation study
Effectiveness of Each Component: Table 2 provides detailed summary of the ablation studies conducted across multiple datasets, highlighting the individual and combined contributions of the Group and Mask Initialization mod- ules to the overall performance of PGM. The Group mod- ule plays a pivotal role in optimizing parameter allocation, thereby enhancing both the model's capacity and accuracy by efficiently utilizing available network resources. This module is particularly effective in balancing model size with performance, making it well-suited for scenarios with resource constraints. In contrast, the mask initialization module plays a crucial role in enhancing accuracy, espe- cially in the context of subsequent tasks, as illustrated in Figure 2d. By offering an adaptive and structured starting point for mask optimization, ensures efficient parameter usage for task-specific adaptations, minimizing task inter- ference and reducing the risk of catastrophic forgetting. The consistent performance improvements observed across riety of datasets highlight the complementary nature of these two components. Together, they enable PGM to achieve both high accuracy and efficient resource utilization, estab- lishing a robust framework for incremental learning. This highlights the essential role of integrating both modules effectively tackle the dual challenges of preserving task performance while managing model capacity.
Table 3. Comparison of training modes on the CIFAR-100 Super- class dataset terms of ACC and CAP, where higher ACC and lower CAP indicate better performance and efficiency.
Training Mode
ACC (%) 1
CAP (%) /
Threc-stage training
62.42
76.26
Iterative training
62.56
65.97
Different Group Size: In the PGM framework, the size of each group plays a critical role in determining model perfor-
6

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
5.0
0.5-
50
ACC
60
50
0.3
0.2
6.3
WSN
10-
0.0
FGM Cap
Emit
10
15
20
13
20
35
18
Task ID
Task ID
Task ID
Task ID
(a) Comparison of accuracy
(h) Comparison of capacity
(c) Different group size
(d) Ablation of component
Figure& 2. Performance comparison across tasks. (a) Accuracy comparison across tasks, showing the performance differences in accuracy between WSN and our proposed method across different tasks. (b) Capacity comparison across tasks, illustrating the capacity used after training different tasks by WSN and our proposed method, as well as the overlap between the current and previous tasks. (c) Performance and compressed capacity variations with different K values, showing that larger group improve accuracy until performance platcaus. (d) Comparison of PGM with bate-of-the--ee methods, highlighting PGM's superior accuracy and efficiency.
4.4. Further Analysis
A
Computational Efficiency: The proposed method empha- sizes a balance between computational efficiency and perfor- mance, offering practical advantages without significantly increasing time consumption. As depicted in Figure 4a, PGM consistently outperforms some methods such as Pack- Net, SupSup, and WSN in terms of convergence speed various benchmark datasets. This result highlights PGM's ability to deliver superior performance with only marginal increase in computational time. Furthermore, Fig- ure 4a illustrates the time consumption for different values of K. Notably, even as K' increases, the time required grows only slightly, reflecting the method's capability to evaluate a larger number parameter combinations without imposing significant computational burden. These findings under- line PGM's scalability and its ability optimize parameters effectively while maintaining time efficiency, making it robust and feasible solution for tasks.
4.00
0.00
Parametor IT Grop
Gr
(a) Task Fel layer
(b) Task 1 Conv1 layer
R
6.13
4:35
4:500
5:08
Parameter Dependency: Figure 4b provides an in-depth analysis of parameter dependency patterns in convolutional layers. The results demonstrate that these dependencies predominantly follow a localized structure, with interac- tions primarily concentrated among adjacent parameters. Although broader dependencies occasionally manifest, the overall trend indicates that parameter interactions are largely confined to local neighborhoods. This observation suggests that, in most scenarios, global parameter dependencies may not require extensive consideration, thereby simplifying op- timization processes and facilitating more efficient model design strategies. Moreover, the localized nature of de- pendencies implies that leveraging localized optimization techniques can be highly effective, reducing computational complexity without compromising performance. Figure 2c further corroborates this insight by illustrating that as the
(c) Task Fel layer
(d) Task 5 Convil layer
Figure 3. Visualization of MASK distributions across tasks in dif- ferent layers of the CIFAR 100uperclasss dataset.
value of K increases beyond a certain threshold, the model's performance stabilizes and exhibits diminishing sensitivity to additional increases. This trend underscores the effec- tiveness of optimizing local dependencies without the need for exhaustive exploration of global interactions, highlight- ing the scalability and efficiency of the proposed approach. Analysis of other layers is obtained in Appendix D.
Task Differentiability: Transforming task incremental learning (TI.)) into a class incremental learning frame-

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
$4864
1.00
150
230
250
30
350
400
0.505
TOP3
340
0.50
TOP
62.4
Accursey
Task
200
$3.5
250
41.4
150
103
400
611 K3 E32
2:34 Computational
Parameters
Predicted Task
Predicted Task
(a) Computational efficiency
(b) Dependency
(c) WSN
(d) PGM
Figure 4. Comprehensive Evaluation of PGM. (a) Computational efficiency comparison highlights PGM's scalability. (b) Dependency patterns within layer showing localized correlations. (c, d) Confusion matrices illustrating task ID) prediction using OOD scores for WSN and PGM, respectively, showing improved accuracy with an optimal sub-network.
work presents notable challenges, with effective Out-of- Distribution (OOD) detection (Liu al., 2020; Xu & Yang, 2025) being crucial for accurate task identification during inference (Lin et al., 2024). Figures 4c and 4d illustrate the confusion matrices for task ID prediction, where OOD scores are computed by feeding input samples into each model and selecting the one with the highest energy score from class logits, as proposed in (Liu et al., 2020). Com- pared WSN, PGM demonstrates clearer diagonal struc- ture in the confusion matrix, indicating improved task dif- ferentiation. This result suggests that PGM's sub-network selection strategy enhances task ID prediction accuracy by better capturing task-specific features and reducing task interference. These findings highlight the importance of optimizing sub-network selection to improve task differenti- ation in class incremental learning scenarios. More results can be found in the appendix E.
5. Related Work
5.1. Incremental learning
To mitigate catastrophic forgetting (CF) in Incremen- tal learning (IL), various methods have been proposed. Regularization-ben based approaches penalize changes to impor- tant parameters to preserve knowledge from previous tasks (Kirkpatrick et al., 2020; Lee al., 2020), while rehearsal- based methods retain and replay key samples to support the learning of new tasks without forgetting prior ones (Yoon et al., 2022; Chaudhry et al., 2019). Distinct from these approaches, parameter isolation methods allocate distinct parameters to each task, ensuring that updates for new tasks do not interfere with those assigned prior tasks, thereby effectively mitigating CF. For example, PackNet (Mallya & Lazebnik, 2018) and CLNP (Golkar et al., 2019) use I regularization to identify and freeze important neurons while reinitializing unselected neurons for future tasks. Sim- ilarly, PathNet (Fernando et al., 2017) partitions each layer into multiple submodules and selects the optimal pathway for each task. However, these methods rely on handcrafted criteria assess the importance of individual parameters, which often results in suboptimal allocations due to their inherent limitations. Alternatively, learnable importance measures have been introduced, enabling dynamic evalua- tion of parameter relevance during training and providing a more precise approach to parameter allocation. For instance, SPG (Konishi et al., 2023) promotes plasticity by employ- ing parameter-level soft masking, selectively constraining updates to critical parameters based on their relevance to prior tasks. Similarly, WSN (Kang et al., 2022) learns task- specific subnetworks by jointly optimizing binary masks and reusing weights from prior tasks, enabling the identification of compact and effective subnetworks for each task.
Different Layer Parameter Distribution: We visualize the parameter distribution across two tasks (Task1 and Task5) and different layers (Convl I and Fcl) in the CIFARI00 Su- perclass dataset. The probabilities, including p(j), which denotes the likelihood of selecting a specific number of pa- rameters within a group, and the probability of selecting particular combination pattern, are defined in the meth- ods section. As shown in Figure 3, different layers exhibit distinct distribution patterns. Convolutional layers, which share parameters within local receptive fields, tend to have more parameters within each group, whereas the dense con- nectivity of fully connected layers results in fewer corre- lated parameters per group. Additionally, during incremen- tal learning, convolutional layers exhibit higher parameter reuse compared to fully connected layers, leading to fewer parameters retained per group in later tasks. Additional visualizations of distributions for other tasks are provided in the appendix C.
8

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
6. Conclusion
models. In Advances in Neural Information Processing Systems, 2024b.
In this paper, we improve parameter selection IL by ac- counting for parameter dependencies. Unlike traditional methods that independently score parameters, PGM models the selection process probabilistic sampling process, grouping parameters and allowing for joint optimization. PGM utilizes Gumbel-Softma to enable efficient mask se- lection. Our theoretical demonstrates that incorporating pa- rameter dependencies enhances subnetwork selection, lead- ing to more effective utilization of model capacity. The experimental results on datasets validate the superiority of PGM over existing IL techniques. Future work could explore further optimizations of mask selection processes, including the integration of more complex dependency struc- tures or alternative optimization strategies.
Fernando, C.,, Banarse, D., Blundell, C., Zwols, Y., Ha, D.,
Rusu, A. A., Pritzel, A., and Wierstra, D. Pathnet: Evolu- tion channels gradient descent in super neural networks. CoRR, 2017.
Golkar, S., Kagan, M., and Cho, K. Continual learning via neural pruning. CoRR, 2019.
Hocquet, G. Class Incremental Continual Learning in Deep Neural Networks. (Apprentissage continu classe par classe pour les réseaux de neurones profonds). PhD thesis, University of Paris-Saclay, France, 2021.
Hu, Y., Cheng, D., Zhang, D., Wang, N., Liu, T., and Gao, X. Task-aware orthogonal sparse network for exploring shared knowledge in continual learning. In International Conference on Machine Learning, 2024.
Acknowledgements
National Key RD Program of China (2022YFF0112100) NSFC (62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20240081), the Fun- damental Research Funds for the Central Universities 1N0.30922010317, No.3092301 1007), Key Laboratory of Target Cognition and Application Technology (2023-XXPT- LC-005).
Hung, S. C. Y., Tu, C., Wu, C., Chen, C., Chan, Y., and Chen, C. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, 2019.
Jang, E., Gu, S., and Poole, B. Categorical reparameteriza- tion with gumbel-softmax. In International Conference on Learning Representations, 2017.
Impact Statement
Joseph, J. and Gu, A. Reproducibility report: La-maml: Look-ahead meta learning for continual learning. CoRR, 2021.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Kang, H., Mina, R. J. L.., Madjid, S. R. H., Yoon, J., Hasegawa-Johnson, M., Hwang, S. J., and Yoo, C. D. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning. 2022.
References
Arandjelovic, and Zisserman, A. Look, listen and learn. In International Conference on Computer Vision, 2017.
Kim, G., Xiao, C., Konishi, T., Ke, Z., and Liu, A theo- retical study solving continual learning. In Advances in Neural Information Processing Systems, 2022.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with A-GEM. In Interna- lional Conference on Learning Representations, 2019.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N. C., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ra- malho, T.. Grabska-Barwine C., Kumaran, D., and Hadsell, R. Overcoming catas- trophic forgetting in neural networks. In International Conference on Learning Representations, 2020.
Deng, D., Chen, G., Hao, J., Wang, Q., and Heng, Flatten- ing sharpness for dynamic gradient projection memory benefits continual learning. In Advances in Neural Infor- mation Processing Systems, 2021.
A., Hassabis, D., Clopath,
Fang, C., Liu, S., Zhou, Z., Guo, B., Tang, J., Ma, K., and Yu, Z. Adashadow: Responsive test-time model adaptation in non-stationary mobile environments. In Embedded Networked Sensor Systems, 2024a.
Konishi, T., Kurokawa, M., Ono, C., Ke, Z., Kim, G., and Liu, B. Parameter-level soft-masking for continual learn- ing. In International Conference on Machine Learning. 2023.
Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., and Wang, X. Maskllm: Learnable remi-structured sparsity for large language
Krizhevsky, A. and Hinton, G. Learning multiple layers features from tiny images. Handbook of Systemic Autoimmune Diseases, 2009.

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017.
Wen, H., Pan, L., Dai, Y., Qiu, H., Wang, L., Wu, Q. and Li, H. Class incremental learning with multi-teacher distillation. In Computer Vision and Pattern Recognition, 2024.
Kumar, A., Chatterjee, S., and Rai, P. Bayesian structural adaptation for continual learning. In International Con- ference on Machine Learning, 2021.
Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A.. Rastegari, M., Yosinski, J., and Farhadi, A. Supermasks in superposition. In Advances in Neural Information Processing Systems, 2020.
Lange, M. D., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G. G., and Tuytelaars, T. A continual learning survey: Defying forgetting classifi- cation tasks. IEEE Trunsuctions on Pattern Analysis and Machine Intelligence, 2022.
Xu, H. and Yang, Y. ITP: instance-aware test pruning for out-of-distributtio detection. In Association for the Ad- vancement of Artificial Intelligence, 2025.
Lee, J., Hong, H. G.. Joo, D., and Kim, J. Continual learning with extended kronecker-fie factoord approximate curvature. In Computer Vision and Pattern Recognition, 2020.
Yang, Y., Zhou, D., Zhan, D., Xiong, H., and Jiang, Y. Adap- tive deep models for incremental learning: Considering capacity scalability and sustainability. In Knowledge Dis- covery and Data Mining, 2019.
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., and Rodríguez, N. D. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information Fusion, 2020.
Yang. Y., Sun, Z., Zhu, H., Fu, Y., Zhou, Y., Xiong, H., and Yang. J. Learning adaptive embedding considering incremental class. IEEE Transactions on Knowledge and Data Engineering, 2023a.
Lin, H., Shao, Y., Qian, W., Pan, N., Guo, Y., and Liu, B. Class incremental learning via likelihood ratio based task prediction. In International Conference on Learning Representations, 2024.
Yang. Y., Zhou, D., Zhan, D., Xiong, H., Jiang, Y., and Yang, J. Cost-effective incremental deep model: Match- ing model capacity with the least sampling. IEEE Trans- actions Knowledge and Data Engineering, 2023b.
Liu, R., Diao, B., Huang, L., An, Z., and Xu, E2net: Resource-efficient continual learning with elastic expan- sion network. In Association for the Advancement of Artificial Intelligence, 2024.
Yoon, J., Yang, E., Lee, J., and Hwang, S. Lifelong learning with dynamically expandable networks. In Inter- national Conference on Learning Representations, 2018a.
Liu, W., Wang, X., Owens, D., and Li, Y. Energy-based detection. In Advances in Neural In- formation Processing Systems, 2020.
Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learn- ing with dynamically expandable networks. In Interna- tional Conference on Learning Representations, 2018b.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Computer Vision and Pattern Recognition, 2018
Yoon, J., Madaan, D., Yang, E., and Hwang. S. Online coreset selection for rehearsal- continual learning. International Conference on Learning Representations, 2022.
Mallya, A., Davis, D., and Lazebnik, S. Piggyback: Adapt- ing single network to multiple tasks by learning mask weights. In European Conference on Computer Vision, 2018.
Zhou, D., Yang, Y., and Zhan, D. Learning to classify with incremental new class. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Saha, G., Garg, I., and Roy, Gradient projection memory for continual learning. In International Conference Learning Representations, 2021.
Serrà, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, 2018.
Sun, W., Li, Q.. Zhang, S., Wang, W., and Geng, Y. Incre- mental learning via robust parameter posterior fusion. In ACM International Conference on Multimedia, 2024.
10

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
A. Detailed Derivation of Error Combination Selection Method
To provide more detailed derivation of the error differences between the Top-k selection and combination selection methods, we start with the fundamentals of model selection, constructing mathematical model step by step derive the error difference. By analyzing the sources of error, can better understand how each method behaves in the selection process.
Scoring Model and Error
For each parameter 9g .,,,,..,NN, there exists true score TrueScore(O.). Due to the presence of scoring errors, the actual score of each parameter represented as:
Score(0,) = TrueScore(O.) + A;
where A; is the emoor associated with the true score, assumed to follow Gaussian distribution A; NN., with zero mean and variance o².
Error Analysis of Top-k Selection
In the Top-k method, we select parameters based on their independent scores. Let the set of selected parameters be Stop.k> chosen by ranking the scores:
Stop.k TopK Score(O))............. Score(()))))
We then compute the set of incorrectly selected parameters Swrong: i.e., those that were chosen but do not belong to the ideal set set S*. Due to the scoring errors, parameters 0; and 0g may be misselected. The condition for 0g to be misselected is:
Score(O,) > Score(0,)
Substituting the scoring error model, we get:
TS(O,) + A, > TS(O,) + A,
Simplifying:
A - A > OS) - TS(0,)
Assuming A, A; ~ 20²2 the probability of error selection P Pwrong (i, j) is:
Pwrang (i,j) = Pr (A, A, > (A,) TS(0,))
This probability can be expressed using the cumulative distribution function of the standard normal distribution:
P Pwrong (i,j) +b (- TS((,)
(-TS(8)) - TS(O,)
V2o
Error Analysis of Combination Selection
In the combination selection method, we evaluate all possible combinations of l parameters. Let the selected parameter combination be C, and its score is:
Seumbo arg max Score(O,) CEOK
IEC
In combination selection, we consider the errors of all parameters. For each combination, we have:
Score(0.) = Σ (TS(0:) + A:)
IEC
iEC

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
This combined score follows a Gaussian distribution:
Σ
Score(Os) ~ Σ TT((B)),
IC
sEC
Thus, the probability of error selection for the combination is:
P Perron (C) = Φ (0-- (-Eies TS(O,) - -iec TS(O;)
Vko
This indicates that, due to the combination of parameters, the error variance is reduced by factor of Vk, leading to a smaller overall error.
B. Implementation Details
For fair comparison, we strictly adhere to the experimental settings outlined in (Kang et al., 2022), including the use of identical backbone networks for the corresponding datasets. Specifically, we employ modified version of AlexNet, as proposed by (Serrà et al., 2018), for the Split CIFAR-10K (0 dataset, and a modified LeNet architecture, as described by (Saha et al., 2021), for the CIFAR-100 Superclass dataset. For Split TinylmageNet, we adopt the backbone introduced by (Kang et al., 2022), which comprises 4-layer convolutional structure followed by 3 fully connected layers. In line with (Kang al., 2022), we use the Adam optimizer with momentum of 0.9 for initial model training. Each task trained for 50 epochs CIFAR-10 100 and 40 epochs Split TinylmageNet. All experiments are implemented using PyTorch on system equipped with four NVIDIA 4090 GPUs.
C. Different Layer and Parameter Distribution
Here, we visualize all layers of LeNet on the CIFAR Superclass across different tasks. The results confirm our previous observation: convolutional layers exhibit greater parameter sharing compared to other layers.
3.13
M
3.19
A
3.55
3125
Protabblyy
0.100
Protabblly
5.13
0.00
2.90
0.90
3.0%
6.00
Number
Sumberal Parameterss Sumberofframetres
Mumberaa Peramttere
a
(a) Task 1 c
(b) Task 1 conv2
(c) Task 1 fel
(d) Task fc2
0.14
8.15
3.12
4.25
3.00
0.35
0.13
$.00
0.05
4050
5.29
8:50
170%
5.0
28
27 13
(e) Task 5- convl
(1) Task 5 conv2
(g) Task 5- fel
(h) Task 5 fc2
Figure 5, Weight distributions across multiple tasks
12

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
D. Parameter Dependency Across Different Layers
This section presents the parameter dependencies in different layers. As illustrated in the figure below, the convolutional layer exhibits stronger parameter correlations compared the fully connected layer, where dependencies are more sparse and dispersed. This observation highlights the structured nature of convolutional filters, which inherently capture local patterns and share spatial correlations, whereas fully connected layers tend to have less structured parameter interactions.
Fisher Information Matrix tept
Fisher Information Metris Heet
0.00012
3300225
140
345
001
13000075
-00
21000000
STD
0,000000
500
0,000025
3.000000
300
0,000000
1656
000005
1140
13:0
407
2.9 2 B2
Tarumatara
(a) Convolutional Layer (Conv)
(b) Fully Connected Layer (FC)
Figure 6. Visualization of parameter dependency across different layers. The left subfigure illustrates dependencies within the convolutional layer, while the right subfigure shows those in the fully connected layer.
E. Extend to Class-Incremental Learning
The proposed method is developed under the task-incremental learning setting, where parameter isolation methods prevents forgetting via task- task-specific subnetworks. However, noted in (Kim et al., 2022), forgetting may re-emerge in the more challenging class maassinncrmental learning (CIL) setting due to the absence of task identity during inference. To address this, (Kim et al., 2022) introduces OOD detection for implicit task inference, and (Lin et al., 2024) further enhances it with rehearsal samples. Following the protocol in (Lin et al., 2024), we replace their mask selection strategy with ours, and compare against parameter-isolation baselines without additional memory. As shown in Table 4, our method achieves competitive performance the CIL setting.
Table 4. CIL performance comparison CIFAR100 10 using DeiT as the backbone. "Last" denotes the accuracy after learning the final task, and "AIA" denotes the average incremental accuracy.
Method
WSN
PGM
Last
AIA 1
Last 1
AIA 1
Energy
62.21 67.89
75.19
64.43
77.06
TPL
80.93
69.55
81.78
F. Generalization to Non- Vision tasks
To ensure fairness and comparability, we adopt the same task setup as in (Kang et al., 2022; Hu et al., 2024), which are widely used as baselines for evaluating incremental learning performance. To further assess the generalization capability of our method beyond the vision domain, we extend it to an audio classification task using the KineticsSounds dataset (Arandjelovic & Zisserman, 2017). The dataset is divided into five incremental tasks, denoted as KS-5. As shown in Table 1, PGM outperforms WSN in both accuracy and parameter capacity when using the ResNet18 architecture.
13

---

Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Table Performance evaluation the KS dataset.
Method
WSN
PGM
Acc 1
CAP 4
Acc 1
CAP
KS-5
69.43
76.44
70.44
57.41
G. Generalization across Model Architectures
To further assess the generalization capability of PGM across different architectures, evaluate it both DeiT and ResNet18 following the settings in (Lin et al., 2024; Kang et al., 2022), and compare it with parameter-isolation methods that incur no additional storage cost and exhibit no forgetting. As shown in Table 6, PGM consistently demonstrates robust performance across all evaluated configurations. Notably. the greater parameter reduction observed on ResNet architectures can be attributed to the larger number of convolutional layers, where modeling parameter dependency tends to be more effective. contrast, Transformer-batt architectures contain more linear layers, where parameter dependency are inherently weaker, leading comparatively smaller gains.
Table 6. Comparative performance evaluation across different model architectures CIFARI00-10.
Method
WSN
PGM
Acc f
CAP
Ace f
CAP 1
ResNet18
73.51
90.66
75.37
71.59
DeïT
93.78
69.46
94.21
60.65
14


Reviews:
[{"review_id": "irYXAvkeWm", "reviewer": "Reviewer_Vqy2", "summary": "In this paper, the author proposed probabilistic group mask selection, which aims to group parameters and explore the dependencies between them. In addition, the author used gumbel-softmax to make the sampling differentiable. To verify the method, the author conducted experiments on multiple datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.\n\n**Clarifying the Role of Dependency in Our Method:** Modeling parameter dependency during subnetwork selection is the key contribution of this work, enabling more effective parameter allocation in incremental learning. Specifically, this involves two key aspects: (1) **Dependency** refers to the notion that the importance of a parameter should be evaluated in the context of its interactions with other parameters [1], in order to better reflect the collective contribution of parameter subsets. (2) To implement this idea, we partition parameters into groups and learn a categorical distribution over all possible selection combinations within each group, enabling joint evaluation of parameter subsets and thus capturing intra-group dependency. \\\nIn addition to dependency modeling, we introduce **task-informed mask initialization** to initialize task-specific mask distributions by leveraging similarities with prior task masks, thereby promoting efficient parameter reuse while preserving task-specific adaptability.\n\n**Clarifying the Novelty of Dependency-Aware Subnetwork Selection:** Parameter-isolation based incremental learning methods aim to assign compact yet effective subnetworks to individual tasks, thereby reducing capacity overhead and mitigating forgetting. However, existing approaches typically assume parameter independence, selecting parameters based on weight magnitude [2] or learnable importance scores [3], without considering how the importance of one parameter may depend on others. This assumption can lead to inaccurate parameter importance estimation, ultimately resulting in suboptimal subnetworks. To address this limitation, we propose a dependency-aware approach, Probabilistic Group Masking (PGM), which explicitly models parameter dependency during subnetwork selection. Specifically, (1) At the methodological level, PGM partitions parameters into multiple groups and evaluates all possible combinations within each group to capture intra-group dependency. Based on this modeling, PGM performs probabilistic sampling within each group to generate task-specific masks, with the entire process made differentiable via Gumbel-Softmax reparameterization. This design enables more dependency-aware subnetwork construction by jointly evaluating parameter combinations, thereby facilitating the selection of higher-quality subnetworks and enhancing the reuse of previously activated parameters. (2) At the theoretical level, we provide a theoretical analysis showing that modeling parameter dependency significantly reduces the risk of selecting suboptimal parameters. (3) At the empirical level, we conduct extensive experiments, including comparisons with state-of-the-art baselines, ablation studies of key components, and analysis of group size. The results consistently demonstrate that our dependency-aware approach reduces overall parameter usage while maintaining or even improving task performance.\n\n**References**:  \n[1] Denis Kuznedelev, et al. CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models. *NeurIPS*, 2023.  \n[2] Arun Mallya, et al. PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning. *CVPR*, 2020.  \n[3] Haeyong Kang, et al. Forget-free Continual Learning with Winning Subnetworks. *ICML*, 2022.", "claims_and_evidence": "The author's claim is consistent with his technique and motivation. The main purpose is to explore the application of parameter isolation in the field of continual learning. More precisely, it is to explore the application of pruning in continual learning.", "methods_and_evaluation_criteria": "Yes, the proposed methods and techniques are all related to conditional masks, including mask selection, optimization and initialization.  However, I don’t see the specific application of dependency in the whole method. The more obvious point is that the final differentiable mask is generated by weighting different masks and task-informed mask initialization. But I’m not sure if this is the dependency discussed by the author, and I hope the author can explain this part.", "theoretical_claims": "Since it is an urgent review, I took a quick look at it. If there are any questions later, I will add them.", "experimental_designs_or_analyses": "The experimental datasets selection is more in line with the measurement standards in the field. But the experimental effect does not seem to be very significant? As the sparsity index is significant, so such experimental results are acceptable. My question is that it seems not uncommon to generate a group of different masks in continual learning, so I am a little skeptical about the novelty of doing so.", "relation_to_broader_scientific_literature": "It is helpful for parameter isolation methods in continual learning.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Ablation experiments are quite sufficient.", "other_comments_or_suggestions": "As mentioned above, the issues of novelty and dependency need to be explained by the author.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eIHJI3RNgN", "reviewer": "Reviewer_MRKd", "summary": "The paper introduces Probabilistic Group Mask (PGM) for incremental learning (IL), addressing catastrophic forgetting by modeling parameter dependencies during sub-network selection. Unlike prior methods that independently score parameters, PGM partitions parameters into groups and optimizes task-specific masks via probabilistic sampling within each group, using Gumbel-Softmax to enable differentiable optimization. Theoretical analysis shows grouping reduces selection errors by leveraging local parameter interactions, while task-aware initialization enhances reuse by aligning mask probabilities with prior task similarities. Experiments on Split CIFAR-100, CIFAR-100 Superclass, and Split TinyImageNet demonstrate PGM's superiority, achieving state-of-the-art accuracy and near-zero forgetting, underscoring its effectiveness in balancing stability and plasticity for scalable IL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n\n**Training Overhead and Parameter Sensitivity:** While probabilistic sampling and Gumbel-Softmax reparameterization may increase implementation complexity, this design enables differentiable and learnable mask selection, which is essential for effective parameter grouping. Larger group sizes $K$ can incur longer training time, and we observe that the performance improvement tends to plateau as $K$ increases (see Figure 4a of the original paper). Regarding hyperparameter sensitivity, the optimal value of $K$ may vary across datasets due to differences in task difficulty and parameter dependency. More detailed hyperparameter settings will be given in the final version.\n\n**Generalization across Model Architectures.**: To ensure the fairness and comparability of results, we adopt the same model architectures as in [1,2], which are widely recognized as baselines for evaluating incremental learning performance. To further assess the generalization capability of PGM across different architectures, we evaluate it on both DeiT and ResNet18 following the settings in [3,4], and compare it with parameter-isolation methods that incur no additional storage cost and exhibit no forgetting. As shown in Table 1, PGM consistently demonstrates robust performance across all evaluated configurations. Notably, the greater parameter reduction observed on ResNet architectures can be attributed to the larger number of convolutional layers, where modeling parameter dependency tends to be more effective. In contrast, Transformer-based architectures contain more linear layers, where parameter dependency are inherently weaker, leading to comparatively smaller gains.\n\n**Table 1.** Comparative performance evaluation across different model architectures. \n|Architecture|Method|CIFAR100-10||\n|-|-|-|-|\n|||Acc↑|CAP↓|\n|ResNet18|WSN|73.51|90.66|\n||PGM|**75.37**|**71.59**|\n|DeiT|WSN|93.78|69.46|\n||PGM|**94.21**|**60.65**|\n\n**Generalization across Different Task Types:** To further evaluate the generalization capability of our method beyond the vision domain, we extend it to an audio classification task using the KineticsSounds dataset [5]. The dataset is partitioned into five incremental tasks, referred to as KS-5. As shown in Table 2, PGM outperforms WSN in both accuracy and parameter capacity when using the ResNet18 architecture.\n\n**Table 2.** Comparative performance evaluation on the KS dataset.\n|Architecture|Method|KS-5||\n|-|-|-|-|\n|||Acc↑|CAP↓|\n|ResNet18|WSN|69.43|76.44|\n||PGM|**70.44**|**57.41**|\n\n**References**:  \n[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. *ICML*, 2022.\\\n[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. *ICML*, 2024.\\\n[3]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. *ICLR*, 2024.\\\n[4]. Md. Sazzad Hossain, et.al. Rethinking Task-Incremental Learning Baselines. *ICPR*, 2022.\\\n[5]. Relja Arandjelovic, et.al. Look, listen and learn. *ICCV*, 2017.", "claims_and_evidence": "The claims made in the submission are well-supported by both theoretical analysis and empirical results.", "methods_and_evaluation_criteria": "The paper presents a novel theoretical framework for understanding parameter selection in incremental learning. The authors introduce the concept of parameter reuse with dependency (Definition 3.1) and demonstrate through Theorem 3.2 that group-wise selection reduces evaluation errors by capturing local parameter interactions. This theoretical foundation justifies the group-wise approach and provides insight into why considering parameter dependencies leads to better sub-network selection.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "The experimental evaluation is comprehensive and rigorous. The authors test PGM on three standard benchmark datasets: Split CIFAR-100, CIFAR-100 Superclass, and Split TinyImageNet. They compare PGM against multiple state-of-the-art methods, including parameter isolation approaches (PackNet, SupSup, WSN), parameter regularization techniques (La-MAML, GPM, FS-DGPM), and other relevant baselines. The evaluation metrics include ACC (average classification performance), CAP (parameter capacity usage), and BWT (backward transfer). The results consistently show that PGM outperforms existing methods across all metrics.", "relation_to_broader_scientific_literature": "The paper situates itself within the broader incremental learning literature, building upon and advancing parameter isolation methods. It acknowledges previous work on mask-based approaches while highlighting the novel contribution of incorporating parameter dependencies through group-wise selection. The approach aligns with recent trends in efficient machine learning that seek to optimize model capacity while maintaining performance.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Advantages:\n1. Theoretical Contributions: The paper provides a novel theoretical framework for understanding parameter selection in incremental learning, which is a significant contribution to the field.\n2. Practical Efficiency: The method balances computational efficiency with performance, making it feasible for real-world applications.\n3. Scalability: The group-wise approach allows the method to scale effectively with increasing numbers of tasks and parameters.\n\nDisadvantages:\n1. Implementation Complexity: The probabilistic sampling and Gumbel-Softmax reparameterization may increase implementation complexity compared to simpler mask selection methods.\n2. Hyperparameter Sensitivity: Performance may be sensitive to the choice of group size (K) and other hyperparameters, requiring careful tuning for optimal results.\n3. Limited Generalization Analysis: While the method performs well on the tested datasets, the paper could benefit from more extensive analysis of its generalization capabilities across different types of tasks and model architectures.", "other_comments_or_suggestions": "No", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHd3VfFSIP", "reviewer": "Reviewer_uc5U", "summary": "This paper introduces Probabilistic Group Mask (PGM), a novel parameter-isolation method for incremental learning (IL) that addresses catastrophic forgetting by incorporating parameter dependencies during sub-network selection. PGM groups parameters, assigns probabilistic masks via Gumbel-Softmax Sampling, and dynamically initializes masks based on prior task similarities. Theorem 3.2 states that grouping parameters minimizes selection error by utilizing dependencies. Comprehensive experimental results on Split CIFAR100, CIFAR-100 Superclass, and Split TinyImagenet demonstrate PGM’s superior accuracy and parameter efficiency compared to existing methods like WSN [ICML’22], SPG [ICML’23], RP2F [ACMMM’24], with reduced reliance on new parameters and improved task differentiation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.\n\n**Scalability to Modern Architectures:** To ensure the fairness and comparability of results, we adopt the same model architectures as in [1,2], which are widely recognized as baselines for evaluating incremental learning performance. To further assess the generalization capability of PGM across different architectures, we evaluate it on both DeiT and ResNet18 following the settings in [3,4], and compare it with parameter-isolation methods that incur no additional storage cost and exhibit no forgetting. As shown in Table 1, PGM consistently demonstrates robust performance across all evaluated configurations. Notably, the greater parameter reduction observed on ResNet architectures can be attributed to the larger number of convolutional layers, where modeling parameter dependency tends to be more effective. In contrast, Transformer-based architectures contain more linear layers, where parameter dependency are inherently weaker, leading to comparatively smaller gains.\n\n**Table 1.** Performance evaluation across different model architectures. \n|Architecture|Method|CIFAR100-10||\n|-|-|-|-|\n|||Acc↑|CAP↓|\n|ResNet18|WSN|73.51|90.66|\n||PGM|**75.37**|**71.59**|\n|DeiT|WSN|93.78|69.46|\n||PGM|**94.21**|**60.65**|\n\n**On Correlation Quantification and Adaptive Grouping Strategies:** Figure 6 of the original paper shows that parameter interactions are predominantly local, with each parameter mainly influenced by its immediate neighbors, while the impact of distant parameters is considerably weaker. This observation supports the assumption that grouped parameter blocks can be treated as approximately independent, suggesting that a simplified grouping strategy is sufficient to capture essential dependencies without significant loss in modeling capacity. Incorporating correlation-aware analysis and adaptive grouping mechanisms remains a promising direction for future research.\n\n**Huffman Encoding in CAP Calculation:** To ensure fair comparison with prior work [1], we adopt Huffman encoding for consistency. The binary nature of the mask (i.e., 0/1 values) aligns well with Huffman's optimal prefix coding scheme, allowing for efficient compression and significant reduction in mask storage.\n\n**References**:  \n[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. *ICML*, 2022.\\\n[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. *ICML*, 2024.\\\n[3]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. *ICLR*, 2024.\\\n[4]. Md. Sazzad Hossain, et.al. Rethinking Task-Incremental Learning Baselines. *ICPR*, 2022.", "claims_and_evidence": "Yes. Claim “incorporating parameter dependencies” is supported by Theorem 3.2 and empirical results. Claim “Group-wise mask selection” is supported by derivation of differentiable sampling and ablation study.", "methods_and_evaluation_criteria": "Yes. The proposed PGM effectively addresses the challenges, i.e., catastrophic forgetting by incorporating parameter dependencies, in incremental learning.", "theoretical_claims": "Yes. The theory proposed in the paper is sound. Theorem 3.2 establishes a mathematical basis, demonstrating how group-wise selection reduces error rates through variance reduction. Although based on a simplified assumption (Gaussian errors), this analysis is consistent with empirical findings (Fig. 2c) and highlights the group size K as an adjustable parameter balancing accuracy and complexity.", "experimental_designs_or_analyses": "The author presents the extensive experiments including main comparison with recent state-of-the-art baselines of incremental learning, ablation study for effectiveness of key components, influence of group size, adaptability to diverse training paradigms.", "relation_to_broader_scientific_literature": "Yes. This paper focuses on core issue, i.e., catastrophic forgetting by incorporating parameter dependencies, in incremental learning.", "essential_references_not_discussed": "The references have been thoroughly discussed. All key references have been incorporated, particularly those concerning the most recent advancements in incremental learning.", "other_strengths_and_weaknesses": "Strengths:\n1. Novel integration of probabilistic group masking and parameter dependencies. PGM introduces a group-wise strategy that explicitly models parameter interactions, a significant departure from existing methods that treat parameters independently.\n2. Strong empirical results across metrics and datasets. The method achieves state-of-the-art results on Split CIFAR-100, CIFAR-100 Superclass, and Split TinyImageNet, outperforming parameter-isolation baselines like WSN and regularization methods like GPM. Furthermore, the authors analyze the computational cost, parameter dependency, task differentiability.\n3. Effective use of Gumbel-Softmax for differentiable optimization. The reparameterization of discrete mask selection via Gumbel-Softmax (Eq. 4) enables end-to-end training while preserving task-specific adaptability.\n4. Comprehensive ablation study validating design choices. The paper rigorously validates individual contributions of the Group and Mask Initialization modules (Table 2), showing that their combination drives performance gains.\nWeaknesses:\n1. Limited scalability analysis for large models: Experiments focus on smaller architectures (e.g., AlexNet, LeNet), leaving scalability to modern networks (e.g., ResNet, ViTs) unaddressed. \n2. Superficial treatment of dependency patterns: Although Figure 6 visualizes localized dependencies in convolutional layers, the work does not quantify their impact (e.g., correlation strength) or explore adaptive grouping strategies.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "How does PGM handle scenarios where parameter dependencies are non-local (e.g., in transformers)?\nWhy is CAP defined with Huffman encoding (Sec. 4.1), and how does this affect interpretation?\nFurthermore, please refer to weakness for more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hHWu1ByMYc", "reviewer": "Reviewer_uCBE", "summary": "The authors propose PGM, a dependency-aware parameter-isolation framework for IL that optimizes task-specific subnetworks via probabilistic group masking. By grouping parameters and sampling masks with Gumbel-Softmax, PGM improves parameter reuse and reduces capacity overhead. Theoretical proofs link dependency modeling to improved sub-network selection. Extensive experiments are conducted to show the SOTA results on standard benchmarks. The work includes ablation studies, efficiency analysis, and visualizations of parameter distributions. The author has thoroughly substantiated the paper’s arguments from multiple perspectives, including methodology, theoretical analysis, and experiments, thereby strongly supporting the proposed viewpoints.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n\n**Extending to Longer Task Sequences:** To ensure fairness and comparability, we adopt the same task configurations as in [1,2] (i.e., 10, 20, and 40 tasks), which are widely recognized as baselines for evaluating incremental learning performance. To further assess performance under longer task sequences, we extend the evaluation on CIFAR-100 by increasing the number of tasks to 10, 20, and 50 under the DeiT architecture [4]. As shown in Table 1, PGM consistently achieves strong performance across all settings and demonstrates better scalability than WSN, particularly with respect to CAP.\n\n**Table 1.** Performance evaluation on long task sequences using DeiT architecture.\n|Architecture|Method| CIFAR100-10||CIFAR100-20||CIFAR100-50||\n|-|-|-|-|-|-|-|-|\n|||Acc↑|CAP↓|Acc↑|CAP↓|Acc↑|CAP↓|\n|DeiT|WSN|93.78|69.46|97.68|72.84|98.40|76.25|\n||PGM|**94.21**|**60.65**|**98.22**|**65.33**|**98.91**|**70.64**|\n\n**Stability under Conflicting Dependency across Tasks and Layers:** Parameter dependency are modeled at the intra-layer levels, reflecting the fact that the importance of a parameter can be influenced by other parameters within the same task. To facilitate knowledge transfer while reducing task interference, we employ a similarity-based mask initialization strategy that modulates the influence of prior tasks according to their similarity to the current task. Greater task discrepancy results in lower similarity scores, which suppress the transfer of conflicting knowledge. While the current framework focuses on modeling intra-layer dependency, incorporating cross-layer dependency may further improve performance and generalization. This remains a promising direction for future work, and we plan to explore it in subsequent research.\n\n**Different Performance Gains on TinyImageNet Compared to CIFAR-100:** The difference in ACC improvement can be attributed to variations in dataset characteristics and model configurations. TinyImageNet uses images with a resolution of 64×64, includes 40 tasks, and is trained with a network composed of four convolutional layers and three fully connected layers. In contrast, CIFAR-100 uses images with a resolution of 32×32 and includes 10 and 20 tasks, with AlexNet used for the 10-task setting and LeNet for the 20-task setting. These differences result in performance variation across the two benchmarks.\n\n**Impact of Group Size K on Complexity and Training Time:** While the theoretical search space for mask combinations grows exponentially with group size $K$, our Gumbel-Softmax sampling reduces the practical complexity to linear time (i.e., $\\mathcal{O}(K)$). This is achieved by directly learning parameter activation distributions, thereby avoiding exhaustive combinatorial search. As shown in Figure 4a of the original paper, performance gains tend to plateau beyond K = 8, suggesting that further increasing the group size yields limited benefit while introducing additional computational overhead.\n\n**Extending to Class-Incremental Learning with Rehearsal Sample:** The proposed method is developed under the task-incremental learning setting, where parameter isolation methods prevents forgetting via task-specific subnetworks. However, as noted in [3], forgetting may re-emerge in the more challenging class-incremental learning (CIL) setting due to the absence of task identity during inference. To address this, [3] introduces OOD detection for implicit task inference, and [4] further enhances it with rehearsal samples. Following the protocol in [4], we replace their mask selection strategy with ours, and compare against parameter-isolation baselines without additional memory. As shown in Table 2, our method achieves competitive performance in the CIL setting.\n\n**Table 2.** CIL performance comparison using DeiT architecture. AIA is the average incremental ACC. Last is the ACC after learning the final task.\n|Architecture|Method|CIFAR100-10||\n|-|-|-|-|\n|||Last↑|AIA↑|\n|Deit|WSN+TPL|67.89|80.93|\n||PGM+TPL|**69.55**|**81.78**|\n\n**References**:  \n[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. *ICML*, 2022.\\\n[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. *ICML*, 2024.\\\n[3]. Gyuhak Kim, et.al. A Theoretical Study on Solving Continual Learning. *NeurIPS*, 2022\\\n[4]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. *ICLR*, 2024.", "claims_and_evidence": "Yes. The main claims in this paper are supported from methodology, theoretical analysis and experiments.", "methods_and_evaluation_criteria": "Yes. The PGM is well-designed and effectively address the identified challenges in Incremental learning.", "theoretical_claims": "Yes. The proposed theory, i.e., Error Reduction via Dependency, is rigorously proven to be correct and strongly supports the viewpoints of the paper.", "experimental_designs_or_analyses": "Main experimental results demonstrate the superiority of the PGM. And the authors provide the ablation study, Computational Efficiency, Parameter Dependency, Different Layer Parameter Distribution to support the claims of the paper.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the incremental learning.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n1. Practical Impact: The efficient parameter reuse mechanism in PGM significantly reduces memory and computational overhead, making it particularly well-suited for deployment in resource-constrained environments.\n2. Differentiable Optimization: The Gumbel-Softmax reparameterization elegantly handles discrete mask selection, enabling end-to-end training. The learning algorithm is well-designed and novel.\n3. Task-Informed Initialization: Leveraging prior task masks for initialization is an innovative strategy that enhances transferability by effectively retaining relevant knowledge from previous tasks.\n4. The paper rigorously validates its claims from multiple perspectives, including methodological soundness, theoretical justification, comprehensive ablation studies, efficiency analysis, and insightful visualizations. This multifaceted evaluation ensures a thorough understanding of the proposed approach, reinforcing its reliability and effectiveness.\nWeaknesses\n1. Long-Task Scalability: The experiments are limited to sequences of 5–10 tasks, which may not fully capture the scalability of the approach. Extending the evaluation to longer task sequences would provide deeper insights into its long-term performance, stability, and adaptability in more complex scenarios.\n2. How does PGM ensure stability when dependencies conflict across tasks (e.g., Task 1’s critical parameters are Task 2’s noise\n3. For the TinyImageNet results, why is the ACC improvement over SPG smaller compared to CIFAR-100?", "other_comments_or_suggestions": "1. The dependency analysis (Fig. 5b) is insightful but could be expanded—e.g., how do dependencies vary across layers/tasks?\n2. The impact of group size K on training time (Fig. 6a) is underdiscussed. A complexity analysis would help.", "questions_for_authors": "Could the method integrate rehearsal or generative replay to further reduce forgetting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TmAHS17wzd", "reviewer": "Reviewer_km4w", "summary": "This paper concerns the parameter-isolation methods in incremental learning. However, existing approaches often disregard parameter dependencies, resulting in an over-reliance on newly allocated parameters. To address this issue, this paper proposes Probabilistic Group Mask selection (PGM), a group-wise approach that captures parameter dependencies by exploring candidate masks within each group. Specifically, PGM partitions parameters into groups with multiple candidate masks, assigning probabilities to these masks and leveraging Gumbel-Softmax for differentiable sampling, enabling efficient optimization of the discrete mask selection process. The theoretical analysis demonstrates that incorporating parameter dependencies enhances sub-network selection. Experiments conducted on standard benchmarks confirm its superior effectiveness compared to existing IL approaches.\n\n## update after rebuttal", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n\n**Generalization to Non-Vision tasks:** To ensure fairness and comparability, we adopt the same task setup as in [1,2], which are widely used as baselines for evaluating incremental learning performance. To further assess the generalization capability of our method beyond the vision domain, we extend it to an audio classification task using the KineticsSounds dataset [3]. The dataset is divided into five incremental tasks, denoted as KS-5. As shown in Table 1, PGM outperforms WSN in both accuracy and parameter capacity when using the ResNet18 architecture.\n\n**Table 1.** Performance evaluation on the KS dataset.\n|Architecture|Method|KS-5||\n|-|-|-|-|\n|||Acc↑|CAP↓|\n|ResNet18|WSN|69.43|76.44|\n||PGM|**70.44**|**57.41**|\n\n**Analysis of the Plateau Effect at Larger Group Sizes:** While the theoretical search space for mask combinations grows exponentially with group size $K$, our Gumbel-Softmax sampling reduces the practical complexity to linear time (i.e., $\\mathcal{O}(K)$). As shown in Figure 4a of the original paper, while computational overhead increases only marginally with larger \\(K\\), the average accuracy begins to plateau beyond a certain threshold. More detailed hyperparameter settings will be given in the final version.\n\n**Extended to Class Incremental Learning**: Extending task-incremental learning to class-incremental learning requires accurate task identity recognition. Prior works [4,5] have shown that accurate task-ID prediction in the CIL setting relies on strong out-of-distribution (OOD) detection capabilities. To this end, we integrated the Energy-based OOD detection method [6] into our framework. As shown in Table 2, PGM achieves higher accuracy than WSN on both the Last and AIA metrics, suggesting that the subnetworks selected by PGM demonstrate stronger OOD detection capability.\n\n**Table 2.** CIL performance comparison using DeiT architecture. AIA is the average incremental ACC. Last is the ACC after learning the final task.\n|Architecture|Method|CIFAR100-10||\n|-|-|-|-|\n|||Last↑|AIA↑|\n|Deit|WSN+Energy|62.21|75.19|\n||PGM+Energy|**64.43**|**77.06**|\n\n**Training Time on Large Scale Models**: To ensure fairness and comparability, we adopt the same model architectures as in [1,2], which are widely recognized as baselines for incremental learning. To further assess the training time of PGM across different architectures, we follow the settings in [5,7] and evaluate it on both ResNet18 and DeiT, comparing against parameter-isolation methods that require no additional storage and exhibit no forgetting. As shown in Table 3, the joint evaluation and optimization of grouped parameters introduce noticeable computational overhead as model size increases. While this results in longer training time, it helps reduce the risk of suboptimal parameter selection by explicitly modeling parameter dependency.\n\n**Table 3.** Performance evaluation across different model architectures. \n|Architecture|Method|CIFAR100-10|||\n|-|-|-|-|-|\n|||Acc↑|CAP↓|Time(h)↓|\n|ResNet18|WSN|73.51|90.66|0.32|\n||PGM|**75.37**|**71.59**|0.43|\n|DeiT|WSN|93.78|69.46|0.55|\n||PGM|**94.21**|**60.65**|0.76|\n\n**References**:  \n[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. *ICML*, 2022.\\\n[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. *ICML*, 2024.\\\n[3]. Relja Arandjelovic, et.al. Look, listen and learn. *ICCV*, 2017.\\\n[4]. Gyuhak Kim, et.al. A Theoretical Study on Solving Continual Learning. *NeurIPS*, 2022\\\n[5]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. *ICLR*, 2024.\\\n[6]. Weitang Liu, et.al. Energy-based out-of-distribution detection. *NeurIPS*, 2020.\\\n[7]. Md. Sazzad Hossain, et.al. Rethinking Task-Incremental Learning Baselines. *ICPR*, 2022.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes. It makes sense.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes. The soundness and validity of experimental designs or analyses are make sense.", "relation_to_broader_scientific_literature": "This paper is related to class-incremental learning, which is widely researched.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. Innovative Approach: This paper proposes Probabilistic Group Mask selection (PGM), a group-wise approach that captures parameter dependencies by exploring candidate masks within each group. The group-wise probabilistic masking strategy effectively captures parameter dependencies, a novel contribution to parameter-isolation methods. \n\n2. Theoretical Grounding: The theoretical analysis demonstrates that incorporating parameter dependencies enhances sub-network selection. The formal analysis of parameter reuse and error reduction via dependency modeling strengthens the method’s motivation.\n\n3. Empirical Results: PGM outperforms baselines like WSN across metrics, particularly in reducing parameter capacity while maintaining accuracy. Meanwhile，the further analysis reveal the effectiveness of each module.\n\n4. Good Analysis: The ablation studies, computational efficiency tests, and visualizations (e.g., dependency patterns, mask distributions) provide valuable insights.\n\nWeaknesses:\n\n1. Limited Task Diversity: Experiments focus on image classification; testing on non-vision tasks (e.g., NLP) would strengthen generalizability claims.\n\n2. Group Size Sensitivity: While larger groups improve performance, the plateau effect at higher K is not thoroughly analyzed (e.g., computational trade-offs).", "other_comments_or_suggestions": "1. The writing is clear, but the appendix could better explain implementation details (e.g., hyperparameters for Gumbel-Softmax).", "questions_for_authors": "1. Could the method be extended to class-incremental learning (CIL) without task IDs?\n\n2. What is the overhead of maintaining group-wise masks for large-scale models (e.g., Transformers)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers saw both strengths and weaknesses in the submitted version of the paper. After some discussion, the overall verdict was that this submission could be accepted in ICML-2025 while the concerns raised by reviewers should be addressed in the final version.
Decision: Accept (poster)