Concept-Driven Continual Learning
Authors: Sin-Han Yang
Tuomas Oikarinen
Tsui-Wei Weng
Venue: Accepted by TMLR ()

Abstract: This paper introduces two novel solutions to the challenge of catastrophic forgetting in continual learning: Interpretability Guided Continual Learning (IG-CL) and Intrinsically Interpretable Neural Network (IN2). These frameworks bring interpretability into continual learning, systematically managing human-understandable concepts within neural network models to enhance knowledge retention from previous tasks. Our methods are designed to enhance interpretability, providing transparency and control over the continual training process. While our primary focus is to provide a new framework to design continual learning algorithms based on interpretability instead of improving performance, we observe that our methods often surpass existing ones: IG-CL employs interpretability tools to guide neural networks, showing an improvement of up to 1.4% in average incremental accuracy over existing methods; IN2, inspired by the Concept Bottleneck Model, adeptly adjusts concept units for both new and existing tasks, reducing average incremental forgetting by up to 9.1%. Both our frameworks demonstrate superior performance compared to exemplar-free methods, are competitive with exemplar-based methods, and can further improve their performance by up to 18% when combined with exemplar-based strategies. Additionally, IG-CL and IN2 are memory-efficient as they do not require extra memory space for storing data from previous tasks. These advancements mark a promising new direction in continual learning through enhanced interpretability.

Published in Transactions on Machine Learning Research (09/2024)
Concept-Driven Continual Learning
Sin-Han Yang
608202029@nuu. edu.
National Tahwon University
Tuomas Oikarinen
toikarinen edu
UC San Diego
Tsui- Weng
luerig Grasdd. edu
UC San Diego
Reviewed on OpenReview: https: // openreview. net/ forum? id= HSW,
Abstract
This paper introduces two novel solutions to the challenge of catastrophic forgetting in continual learning: Interpretability Guided Continual Learning (IG-CL) and Intrin- sically Interpretable Neural Network (IN2). These frameworks bring interpretability into continual learning, systematically managing human-understandaale concepts within neural network models to enhance knowledge retention from previous tasks. Our methods are designed to enhance interpretability, providing transparency and control over the continual training process. While our primary focus is to provide a new framework to design contl learning algorithms based on interprotability instead of improving performance, we observe that our methods often surpass existing ones: IG-CL employs interpretability tools guide neural networks, showing an improvement of up to 1.4% in average incremental accuracy over existing methods; IN2, inspired by the Concept Bottleneck Model, adeptly adjusts concept units for both new and existing tasks, reducing average incremental forgetting by up to 9.1%. Both our frameworks demonstrate superior performance compared to exemplar-free methods, are competitive with exemplar-basee methods, and can further improve their performance by up to 18% when combined with exemplar- based strategics. Additionally, IG-CL and IN2 are memoory-fffcient as they do not require extra memory space for storing data from previous tasks. These advancements mark promising new direction in continual learning through enhanced interpretability!
1 Introduction
Continual learning is a crucial aspect of machine learning, empowering models to adapt and improve as they encounter new data over time. This dynamic learning process, however, faces a significant hurdle known 'catastrophic forgetting." This phenomenon, where model forgets previously learned information upon acquiring new knowledge, is largely attributed to the shift in input distribution with changing tasks. According to van de Ven et al. (002)]; De Lange et al. (2021). there are three primary settings in continual learning: class incremental, task incremental, and domain incremental. Our focus in this paper is on the class incremental setting, identified as the most challenging setting among three due to its pronounced susceptibility
to catastrophic forgetting, as detailed in van de Ven et al. (2022); Chaudhry et al. (2018 De Lange et al. (2011)
Current. strategies in continual learning fall into the following main categories: Exemplar-free and Exemplar- based methods. Exemplar-fres methods do not rely data from old tasks, like regularization--ern methods that introduce constraints to preserve old knowledge; architecture-bosed methods that adapt the model's structure for new lasks. On the other hand, Exemplar-based methods, also known as replay-based Quu code available

---

Published in Transactions on Machine Learning Research (09/2024)
Finetune
LwF
IG-CL-freeze-al (Ours)
EWC
GEM
IG-CL-freeze-pe (Ours)
51
MIR
preserved concept unit ratio
preserved concept unit ratio
0.6
0.6
0.4
0.4
0.2
0.0
0.0
3
4
2
3
5
task index
(a) CIFAR-100
(b) TinylmageNet
Figure 1: IG-CL's concept evolution in (a) CIFAR-100 and (b) TinyImageNet. Tasks' classes are in Appendix D.3
The results show that existing work perform poorly on preserving the learned concepts hence suffers from catastrophic forgetting problem, while the proposed IG-CL can preserve the concepts from previous classes. The values are in Table 5 of Appendix A.1
methods, involve revisiting previous data. Replay-based methods generally outperform others but require extra memory for data storage. Despite existing methods show some promise, they still lack in performance and many of them still have serious catastrophic forgetting problems. This gap highlights the need for more systematic and understandable approaches continual learning.
In parallel, there growing body of research dedicated to deciphering the role of neurons in neural networks.
Pioncering studies, such as (Bau al., 2017 2020 Oikarinen & Weng 2023 Hernandez al. 2022 Bai et al.
2024 Oikarinen & Weng. 2024b, have highlighted 'concept units' neurons that closcly align with concepts easily understood by humans.. Our research reveals critical gap in current continual learning algorithms: While previous continual learning algorithms generally fail to effectively preserve these concept units, our research reveals the advantage of leveraging intorpretability for mitigating catastrophic forgetting. This key finding, detailed in Figure has shaped our approach to developing new continual learning frameworks.
In this paper, we present novel frameworks, Interpretability Guided Continual Learning (IG-CL) and Intrinsically Interpretable Neural Network (IN2), to address the issue of catastrophie forgetting in continual learning. Unlike existing approaches that lack interpretability, our methods stand out by incorporating interpretability, enabling systematic management of human-understandable concepts within the model, which enhances knowledge retention from prior tasks. Our contributions are the following:
Our first method, IG-CL, uses interpretability tools (Bau et al. 2020 Oikarinen & Weng 2023 to guide neural networks in continual learning. IG-CL has proven to be effective in our extensive experiments, outperforming existing methods by achieving up to 1.4% higher average incremental accuracy.
Our second method, IN2, is a novel framework based on the Concept Bottleneck Model (CBM) (Koh et al, 2020; Oikarinen al.) 2023), tailored for continual learning. IN2 is engineered to maintain and adapt concept units corresponding to both old and new tasks, further reducing average incremental forgetting by up 9.1% compared to existing methods. Different from IG-CL, IN2 modifies the model's architecture and inherit interpretability from CBM.
To our best knowledge, IG-CL and IN2 are the first general approach to make the learned knowledge and the training process transparent and interpretable in the continual learning sctting.
Our primary aim is leverage interpretability provide transparency and control over retained knowledge and the training process, addressing gaps in the current literature. IG-CL and IN2 bridge interpretability and
2

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Comparison of our method against existing methods for continual learning.
(T) Senlahility
(IT) Flexibility
(III) Interpesability
Only need
Without
For aay
Method:
one backbone
memoory buffers
architectare
Exemplar-free
EWC (Karkpatrick et
2017
Yes
Yes
Yes
No
(Zenke xJ 201Y
Yes
Yes
No
LuF
Horm 2017
Yes
Yes
Yes
No
Adon-NSCL Wang
2021
Yes
Yes
Yes
No
SSRE
el al.
Yes
Yes
No
DEN
Yoon
20
Yes
Yes
Yes
No
ICICLE Rymarcyy x el al
2023
Yes
Yes
No
A (part-based prototype)
Exemplar- hased
GEM
(Loney-Poz & Ranzato
2117
Yes
No
Yes
No
MIR Ajundi ot al 2015
Yes
No
Yes
No
DEM
Yan
IG-CL (Ours)
Yes
Yes
Yes
Yes, text-based concept (more general) Yes, text-based concept (more general)
IN2 (Ours)
Yes
Yes
Yes
continual learning from two perspectives. IG-CL brings interpretability to continual lcarning by using external interpretability tools to guide models. On the other hand, IN2 tailor interpretable models like CBM for continual learning by leveraging their own interpretability. While our focus is on proposing new framework to design continual learning algorithms based on interpretability, we observe that our methods often surpass existing ones. For instance, our approach outperforms exemplar-fres methods average incremental accuracy and competitive with exemplar-based methods. When combined with exemplar-based strategies, our methods enhance their performance by up to 18%. Additionally, our methods are memory--ffiiiet,, requiring extra memory space for storing data from previous tasks. The benefits of our methods summarized in Table Interpretability is key distinguishing feature of our approach compared to existing methods.
2 Background and Related Work
2.1
Continual Learning
To mitigate catastrophic forgetting in continual learning, several methods have been proposed. They can be categorized into Exemplar-free and Exemplar-based methods. For Exemplar-free methods, it methods and architecture-based method. The key idea of
consists of two streams: regularizationn-asse egguaaizattion-asee methods is to add additional terms in the loss function to constrain model parameters
to not change too much from previous tasks (Kirkpatrick et al.
2017 Zenke et al., 2017 Li & Hoiem 2017
Jung et al. 2016 Dhar et al., 2019 Castro et al. 2018 Hu et al. 2019 Lee et al. 2019 Aljundi et al. 2018
Chaudhry et al. 2018: Lee el al.] 2017 Schwarz el al.
2018). On the other hand, archilecture-baser methods
modify the model's architectree or parameters when learning new tasks, by dynamic expansion or pruning
Ruusu et al.
2016: Yoon et al. 2018 Xu & Zhul 2018 Yan et al. 2021 Li ct al.) 2019 Serra et al. 2018
Wang et al. 2021 Zhu et al. 2022) For Exemplar-based method, it is also known replay-based methods,
whose spirit is to store previous tasks' information and train the model with new tasks jointly [Lopez-aaz &
Ranzato 2017 Rebulli et al. 2017] Ch7uddryy et al.
2019: Rolnick et al. 2019
Hou et al. 2019 Wu et al.
2019 Buzzega et al. 2020 Wang et al. 2022: Guo et al. 2022 Liu et al.
2021
Aljundi et al.
2019 More
details of these methods is Appendix
D.2
Meanwhile, some works focus on the theoretical aspect of continual learning (Pnng et al. 2023 Peng &
Risteski 2022 Cao et al.) 2022 Ruvolo & Eaton 2013) Pentina & Urner 2016 Chen et al. 2022 Kim et al. 2022). However, none of these methods
able to control buman-interpreett
concepts directly, which
makes them lack interpretability. Indeed, interpretability is one of the main differences between our methods and existing methods. There are some recent works (Marconato el al.) 2023 Rymarczyk el. al. 2023) connects interpretability with continual learning. They focus on part-based prototype concepts or neuro-symbolic concepts. Our work focuses on text-based concepts instead, which allows more general interpretability. Mcanwhile, they are only suitable for particular model architectures whereas our methods are compatible for general DNN.
3

---

Published in Transactions on Machine Learning Research (09/2024)
2.2
Neuron-Level Interpretation and Concept Bottleneck Models
Several works (Bau et al.) 2017 2020; Oikarinen & Weng 2023 Hernandez et al. 2022 Mu & Andreas 2020; Bai ct al. 2024; Oikarinen & Weng, 2024) provide automated descriptions of the roles of individual neurons in deep vision models, and do extensive studies for their methods' interpretability. Typically these methods generate a description by analyzing what kinds of inputs result in high activations for the given neuron. For example, Network Disscction (Bau et al.) 2020) identifies the concepts of individual neurons by comparing the neuron's activation map to concept annotated data. A more recent work CLIP-Dissect (Oikarinen & Weng, 2023) eliminates the need of concept annotated data by leveraging the Contrastive Language-Image Pre-training (CLIP) model (Radford et al. 2021) and designing several similarity functions. Detailed introduction of CLIP-Dissect is in Appendix D.2.2 In this paper, we explore a departure from the typical emphasis on refining interpretability tools, instead utilizing them to guide continual learning processes. Our work focuses on the computer vision domain since the interpretability tools in the domain have made significant progress. It can be applied to other domains like NLP with their interpretability tools like Sajjad et al.
2022); Lee et al. (2023) potentially in the future.
Concept Bottleneck Model (CBM) (Koh et al. 2020) has a layer called Concept Bottleneck Layer (CBL) where each neuron corresponds to a human interpretable concept. The idea of CBL is to create an meaningful concept mapping from black-box representations to a layer that contains a huuman-nterppettbble concept set C, allowing effective intervention and explaining the model predictions. Recent works (Oikarinen et al. 2023 Yuksekgonul et al., 2023 Srivastava et al., 2024) try to address it since concept annotations expensive and hard to collect. Specifically, Label-Free CBM (LF-CBM) (Oikarinen et al. 2023) generates concepts from task information. Detailed introduction of LF-CBM is in Appendix D.2.2 The concept set C' generation depends on the CBM. For LF-CBM (Oikarinen & Weng 2023), concept sets are generated by asking large language models like GPT-3 (Brown et al. 2020) about the concepts related to the classes. For original CBM (Koh et al. 2020), concept sets has to be predefined and provided in the dataset. The primary focus of these CBM works is only on single task and does not consider the challenging continual learning setting. In contrast, our IN2 is tailored for continual learning with a new learning procedure that enables it to learn a series of tasks, which generalizes CBM-based methods to continual learning.
3 Our first method: IG-CL
Overview. To address catastrophic forgetting, we aim to manage concepts learned by models in an interpretable manner. We introduce two frameworks: Interpretability Guided Continual Learning (IG-CL. in this section and Intrinsically Interpretable Neural Network (IN2) in the next section. IG-CL uses an interpretability tool to steer the learning process, identifying and preserving previously learned concepts to prevent forgetting. This approach enhances model comprehension and has shown to improve incremental accuracy by up to 1.4%. Building on this, IN2 incorporates interpretable neurons that are systematically integrated and controlled, further reducing forgetting by up to 9.1% in our experiments. Figure illustrates the key steps in IG-CL, and the details of each step are described below. IG-CL's full algorithm is summarized in Algorithm I in Appendix C.2
2
3.1 Step 1: Decipher network
Initially, we use a neuron-interpretablliy tool to decipher the interpretable neurons in the model. Interpretable neurons are defined as those whose activations correspond closely to human-understanddabl concepts (e.g. 'red', 'stripes', "windy', etc). A model, denoted as 0, consists of feature extractor (Wi)= also known backbone, and a prediction layer Wr. Our goal is to preserve the important concepts learned from the previous tasks. Previous work (Bau el al., 2020) analyzes the concept neurons in each layer, and finds out that high-level concept neurons emerge in the deep layers. Low-level concepts are basic descriptions like colors, while high-level concopts are objects or components of the images. The high-level concepts are specific to the classes in the previous tasks, which needed be preserved. Therefore, we apply the interpretability tool the last layer of the feature extractor Wz. to decipher the neurons. The discovered interpretable
4

---

Published in Transactions on Machine Learning Research (09/2024)
Step 3: train the NN for Task while retaining Task t-1's knowledge:
C(0' D.)
|
Al
wo'l
@ W.....
Step 4: Repeat until finish all tasks
(trainable) (trainable)
tree
tree
W WF
W WF bus
bus Step 1: decipher neurons
Pretrained or training from scratch
Step 2: freeze (#$) learned
knowledge from Task t-1
Figure 2: IG-CL's procedure.
neurons are called concept units. Existing neuron interpretabilityy tools (Bau al. 2017 Oikarinen & directly applicable to detect the concepts of neurons in framework.
Weng 2023
3.2 Step 2: Freezing the learned knowledge
Second, we want freeze the concept units the model preserve the learned knowledge. These concept units emerged in the network during learning the current task t 1. To prevent catastrophic forgetting, it is desired to preserve these concept units when learning new task t. To achieve this, we propose to find and freeze these task-related neurons when learning the new task by finding a sub-network which links all related neurons. In practice, this can be implemented by using Breadth First Search (BFS) from the concept unit in WL to the input layer W1. Define two neurons are connected when the weight exceeds threshold т. We propose two ways freeze the subnetworks to preserve the learned knowledge by different degrees:
Forward Pass Direction
Frozen Subnetwork
Trainable Weights freeze-all
freeze-part
freeze-all Freeze all weights connected to neurons in any concept units' subnetwork.
freeze-part
Only freeze input weights to all neurons in the subnetworks,
leaving output weights trainable.
The illustra- tion of two freezing sub- network methods. Red nodes and lines stand for trainable unit and weights.
Figure 3:
We describe Step 2 mathematically in Line Line 19 Algorithm Appendix
The freezc-part method leaves the output weights of the neurons in the subnet- work trainable, which may have a beller ability to learn new lasks. For freeze-all method, it isolates the subnetworks from the rest of the model which is expected to promote less forgetting. An illustration of these two methods in Figure 3
The experiment results Section 5 show that the freeze-all method forgets less, while the freeze-part
method has better learnability for longer series of tasks as expected. This step summarized Algorithm Line 8-19.
3.3 Step 3: Learning without forgetting
Next, we train the model on new task with Eq. 11. Let D, be the data for task t. We have two goals when learning task regularize (i) the size of subnetworks and (ii) WF. To regularize the size of subnetworks described in Step 2 we choose 2,4 1 and I to regularize the number of activated units. Second,
5

---

Published in Transactions on Machine Learning Research (09/2024)
inspired by the regularization-basee methods, train the model with regularization / 3= 2 on the Wp, which aims to prevent Wr from changing too much. But different from previous regularization-bass preserved knowledge in our method is interpretable (as they are detected as interpretable concept units in Step I which can be easily understood through the text description e.g. "tree", "bus" in Fig 2
methods, the
min C(0"; ( Σ
Will A/W3 - W '133
(1)
6r
Then, we will repeat Step 1
to
for task 2, ... untll the last task T, illustrated in Figure
3.4 Further Discussion
Interpretability. Unlike provious methods in continual learning, which lack interpretability due to their inability to clearly identify retained knowledge in a human--undesstandablle manner, IG-CL simplifies the knowledge preservation process, ensuring clarity and ttraightforwardnees compatible with existing neuron interpretability tools (Bau al.) 2017; Oikarinen & Weng 2023), allowing it to integrate scamlessly with multiple frameworks and enhance the transparency of the learning process. Nonetheless, the selection of interprelability tools will affect IG-CL. The generalizability of interprelability tools are the key to guide and improve continual learning. Therefore, it's crucial that the interpretability tools we selected to be flexible (e.g. open-vocabulary concepts). Meanwhile, they have to be efficient so the continual learning process will not be too time-come cone-consuning Based on these criteria, in this work we use CI.P-DDisoct (Oikarinen & Weng 2023 as the tool detect neuron concepts.
Notably, IG-CL is designed to be
Ability to Share Concepts Among Tasks. IG-CL is designed to keep and reuse the learned concepts from old tasks. Experiment in Section 5.3 Appendix[A.I and B.6 study its ability and show that it outperforms baselines by up to 9% in forward transfer metric.
Efficiency. IG-CL framework is light since it does not need extra memory storage to store previous tasks' samples. Previous tasks' knowledge stored within the model. Experiment on GPU usage in Appendix C.1 shows the efficiency of IG-CL.
4 Our second method: IN2
In the previous Section 3 we introduced IG-CL as a novel interpretable method to identify and manage concept units in models for reducing catastrophic forgetting, marking it as a pioneering approach in the regularization-based category. Building on this, we now present the Intrinsically Interpretable Neural Network (IN2), a new architecttue--based method that transforms any neural network into an interpretable framework designed for continual learning. IN2 uniquely maintains previously learned concept units while integrating new ones for upcoming tasks, demonstrating reduction in average incremental forgetting by up to 9.1% over existing methods according to our experiments. Compared with IG-CL, IN2 does not need external interpretability tools since CBM provides interpretability. Figure 4 illustrates the procedure of IN2.
4.1 Step 0: Set up CBM
For the first task, the learning procedure the same as training a Concept Bottleneck Model (CBM) (Koh et al. 2020) on the first single task. In the continual learning setting, after learning the task t t- 1, the CBM has concept set Cr-1 and the concept mapping W in between the backbone and the prediction layer. We consider the setting that backbone f(x) is frozen for reduced training cost, but our method also allows backbone being trained from scratch or finetuned. For task t, we create task-relevant concept set cj from its class labels k. Creation of the lask-relevant concept sets depends on the Concept Bottleneck Model (CBM). For Label-free CBM (Oikarinen et al.) 2023), concept sets are generated by asking GPT-3 about the concepts related to the classes For CBM (Koh et al. 2020), concopt sets are predefined and provided in the dataset. In the following paragraphs of IN2's learning procedure, we consider the case that the model has learned tasks, and it is going to learn new task l.
6

---

Published in Transactions on Machine Learning Research (09/2024)
create task-relevant concetts:
repeat for new Task t+1 until finish all tasks
Fon
Task classes: ((1: car).., horse))
For 1,-,k:
List the things rost commonly seen around car: {road, house, -]
W
W
road
[Froznn] W
W
Pretrained backbone
learn Task t-I concept mapping W add Task itss concept & learn concept and predictor W.F"
mapping W. and predictor W
Figure 4: IN2's procedure.
4.2 Step 1: Concept set expansion
In this step, we expand the concept set based on classes in new task t. Given the concept set from t- tasks as Ct-1, we form a new concept set C by adding all concepts from de and Ct-1, If certain concepts in dt are already present in C²-1, we still include them nonetheless. This is because identical textual concepts across different tasks may exhibit distinct attributes, such as variations in color and shape. For example, concept 'ship' might refer to "vessel" or 'cargo ship' in different tasks. After the expansion, there are [C_| = [ + |ck concepts in C The analysis of duplicate concepts and numbers are in Appendix C.3
4.3
Step 2: Learning the concept mapping
To preserve learned concepts, we keep the weights of the existing concepts from the concept mapping of the previous task. Given a new concept mapping for learning the task t is W. e RlC'Ixd we first inherit the concept mapping's weights from the previous model:
: = w i ii--iiiii 1 1,:] ViE
(2)
where :i means the i-th row of the Wt. The second step is to learn W using the procedure of CBM. Take LF-CBM Oikarinen et al. (2023) as an example. LF-CBM aims to align the concept mapping with CLIP's (Radford et al. 2021) representation of the concepts. Therefore, the procedure is to maximize the similarity between concept mapping's fz( = f()) and CLIP's activation matrix. The second step can be done in two ways:
1. Finetune: We learn Wt task without additional constraints. We only initialize W with old concepts but these concepts will be changed on the new task.
2. IN2:
We freeze the previous concepts' corresponding mapping in Wt to prevent them from changing. This stratogy enables us to learn the mapping of new concepts without affecting the previously learned ones. This strategy makes IN2 light and interpretable. eliminates the need for additional memory storage for previous tasks' samples, and provides transparency regarding the retained knowledge.
4.4 Step 3: Learning the prediction layer
Finally, we aim to learn the prediction layer without forgetting. First, we inherit prediction weights in W to Wt before the learning process. The idea is similar to Step where |Ce-1| concept output weights in W inherit the same weights from Wt-1, Again, the Finetune strategy is to learn W without any constraints. On the other hand, the IN2 strategy follows the similar idea as IG-CL's Stop 3 learn the prediction layer with regularization y and 8 Meanwhile, we control the sparsity of Wp by setting a 1 and X. Given the dataset D. (X' ))) the training goal is to minimize the loss in Eq. where L could be cross-entropy
7

---

Published in Transactions on Machine Learning Research (09/2024)
loss:
min f(W"; AllWblla >/l wl''lls
(3)
4.5 Further Discussion
Interpretability. IN2 is a versatile framework that supports various approaches for constructing CBMs (Koh et al., 2020 Yuksekgonul et al. 2023 Oikarinen et al. 2023), enhancing interpretability by allowing terms, as illustrated in Figure 5 This interpretability also makes the progression of knowledge acquisition in continual learning transparent, a significant improvement over previous architecture- based methods (Yoon al. 2018 Wang al.) 2021), which lack clarity and interpretability on the knowledge integrated into the extended architeclure. Similar to IG-CL, generalizability, efficiency and accuracy are the selection criteria for CBMs. Because the CBMs for IN2 have to be accurately applied on different data streams, and be efficient for smooth continual learning processes. Based on these criteria, in this work we build IN2 upon LF-CBM (Oikarinen et al. 2023
analysis of predictions in human-understnndaan
Ability to Share Concepts Among Tasks. IN2 designed to keep the learned concepts from old tasks. Figure 5 Appendix A.2 and B.6 study IN2's ability, and show that it can improve GEM (Lopez-Paz Ranzato 2017 by up 7.8% forward transfer metric when combining with GEM.
5
Experiments
In Section we introduce the datasets, evaluation metrics, baselines and other details in the experiment. In Section 5.2 compare methods with existing methods in standard metrics. In Section 5.3 unveil the learned knowledge and the training process by leveraging IG-CL and IN2 interpretability. In Section 5.4
we conduct ablation studies for our methods to analyze the contributions to their superior performance, Due to page limit, additional experiments and discussions are in Appendix A
B
and
C
An overview of additional
experiments is provided in the first page of Appendix as Appendix Outlinc
5.1 Experiment setup
Datasct. To evaluate our methods, we perform experiments on two datasets: CIFAR-100 Krizhevsky et al. 2009) and TinylmageNet (Le & Yang 2015). Experiments on CIFAR-10 (Krizhevsky et al. 2009) and
CUB-200 (Wah al. (201]) are discussed Appendix B and C We consider 5, 10, 20 tasks scenario
class incremental setting. We use ResNet18 (He et al. 2016 as the experiment model. Experiment, dataset,
result report and hyperparameter selection details in Appendix D.1
Evaluation Metrics. Following (Mirzadch et al. 2022b Chaudhry et al. 2018 we use the standard evaluation metrics to evaluate our methods. Define ai as model's accuracy on j-th task after learning i-th task, 2 j. When testing the performance on t-th task, the metrics' definitions are as follows:
Average Accuracy: Measures the average model performance A₁ = Ei 0r,i
Average Forgetting: Measures model performance drop on previous tasks Fr
max (a,,s - -4,5)
JE(1...1-1)
However, these standard metrics only reflect models' performance the final stage. Following recent works
(Zhu al. 2022 Soutil-Cormerais et. al. 2023 Zhou al. 2023 Caccia et al. 2020 2022 Koh et. al. 2022
also evaluate models throughout the stream. Specifically, also report:
Average Incremental Accuracy: A --1 ΣT, A
Average Incremental Forgetting: F ET F.
Baselines. We perform experiments the following continual learning baselines:

---

Published in Transactions on Machine Learning Research (09/2024)
Finetune: the standard method where models are updated continuously series of tasks
Exemplar- free methods
(regularization-bssel EWC
(Kirkpatrick et al.
2017 ST Zenke et al. 2017], LwF (Li & Hoiem
2017
(architecture-basee))
Adam-NSCL (Wang et al.
2021
DEN Yoon et al.
2018
in Appendix
B.5.2
SSRE [Zhu al.
2022 in Appendix B.5.1
ICICLE Rymarczyk et al.
2023
in Appendix
B.5.3
Ekemplar-based methods
GEM (Lopez-Pas Paz & Ranzato 2017) and MIR (Aljundi al. 2019
Notations and Details. For IG-CL, IG-CL-frreeeex' means IG-CL with implementation freeze-all/ freeze- part in Step 2 For CBM based methods, 'Finctuno-CBM means using Finetunc strategy in Step 2 and Step 3 while 'IN2" means using IN2 strategy instead. 'Imovremvve means removing duplicate concepts in concept set expansion. "-GEM" and "-MIR" means combining our methods with GEM and MTR respectively. For baseline strategies, we use the implementations from continual learning library Avalanche [Lmmnnac
ct al. 2021). We also use Avalanche to implement our methods.
Here we discuss 7- experiment results in (År, Fr). For different metrics (Ar, Fr], please see Appendix B.3 for experiment results of {T' 10, 20), please Appendix [.4 The comparisons with SSRE (Zhu al. 2022), DEN (Yoon al. 2018) and ICICLE (Rymarczyk et al. 2023
in Appendix B.5 We were not able to compare with DER (Yan et al. 2021 as their official code is incomplete. Due to scalability and efficiency, we use CLIP-Dissed (Oikarinen & Weng 2023) as the interpretability tool for IG-CL, and
LF-CBM OOikarinen et al. 2023 for CBMs in IN2. Memory usage experiment is in Appendix C.1
Table 2: Accuracy comparison for IG-CL. t means larger values are better, while 4 means smaller values are better. Underline means the strongest baseline, and boldface means the best number. In the Ours Improvement rows, +/- measures best number minus the strongest baseline, and the value is in blue if our improvement is statistically significant. Our methods outperform the baselines on both Ây and Fr.
CIFAR 100, 5T
Tinylmagene 5T
År
Fr
Art
Fy
Exemplar-free baselines
Finetune
20.47 0.67
63.02 0.45
16.82
2.18
49.80 + 0.57
EWC
20.97 0.55
62.56 0.29
16.19
2.65
48.42
0.39
SI
17.75 1.37
59.31 + 1.93
13.07
+ 2.57
44.71
+ 2.28
LwF
12.74 2.15
63.66 + 2.40
16.09 1 3.25
49.43 1 1.26
Adam-NSCL
17.45 2.35
59.54 + 3.04
17.90 + 2.57
44.98 + 0.74
Ours
IG-CL-freeze-al
22.37 + 1.20
58.75 + 0.26
18.19 + 0.76
43.39 + 0.92
IG- IG-CI..freze-part
21.73
0.79
60.51 + 0.85
18.08 + U.56
46.00 + 0.66
Ours Improvement
+1.40
+0.56
+1.37
+1.22
Exemplar-based baselines
GFM
23.02
1.65
60.63 + 4.12
11.29 + 2.62
41.30 + 1.67
Ours
IG-CI-frrezoaal-CENN
28.18
2.36
42.61 +: 2.12
12.49 + 1.87
37.78 + 1.04
fr-Cee-rreeee part- GEM
2.08
3.24
1.58 +1.20
Ours Improvement
+5.16
+18.02
+3.52
Exemplar- baselines
MIR
27.26
0.93
53.02 + 2.84
17.81 + 0.82
41.60 +: 2.13
Ours
IG- CG-CI-freeze-al-Bee -II-
27.04 1.08
51.66 + 3.11
19.95 + 2.41
43.08 + 0.95
IC-CIl-frezze-art-NIi
27.59 + 0.62
52.77 + 2.75
20.02 + 2.58
43.87 + 2.25
Ours Improvement
+0.33
+1.36
+2.21
+1.52

---

Published in Transactions on Machine Learning Research (09/2024)
5.2 Quantitative Results
5.2.1 Comparison results for IG-CL
For IG-CL, the accuracy comparisons with existing works are in Table 2 Compared with the Exemplar-free methods, our method outperforms existing works by up to 1.4% in Ar and up to 1.3% in Fr. Most of the time IG-CL performs better using freeze-all than with freeze-part. Mcanwhile, the performance is comparable to even better than Exemplar-based methods in some benchmarks. When combining IG-CL with Exemplar-based methods, both freeze-all and freeze-part tend to improve Exemplar-based method to achieve higher Ar by up to 5.1% and lower Fr by up to 18.0%. The experiment results show that IG-CL can make models more effective and forget less, either standalone or combining with other methods. Considering the experiment results of 10-task and 20-task scenario in Appendix [3.4] we observe that freeze-part has better performance when task number is bigger. Overall, the freeze-all approach suitable for a limited mumber of tasks and emphasizes preserving learned knowledge, while the freeze-part approach more appropriate for scenarios involving a large number of tasks.
Table 3: Accuracy comparison for IN2. All models are pre-trained on the Place365 dataset (Zhou et al. 2017] means larger values are better, while means smaller values are better. Underline means the strongest bascline, and boldface mcans the best number. In the Ours Improvement rows, +/- mcasures our best number minus the strongest baseline, and the value is in blue if our improvement is statistically significant. Our methods clearly outperform the baselines on both Ay and Fr.
CIFAR-100, 5T
Tinyhnagenet, 5T
Ay
Pr
Ar t
Fr !
Exemplar-free baselines
Finetune
22.68 0.88
75.69
+ 2.25
20.50 + 0.45
66.52 + 2,10
EWC
22.20 1.56
74.93
+ 2.34
19.68 + 1.82
65.26 + 2.54
SI
22.70 1.20
74.44
+ 2.45
18.87 + 1.71
62.44 + 0.68
LwF
24 2.29
74.11
+ 3.33
20.48 + 2.36
64.17 + 3.08
Adam-NSCA
23.37 2.14
53.87 + 3.19
20.17 + 2.04
58.28 + 3.02
Ours
Finetune-CDD
24.99 0.79
59.29 + 1.40
21.13 +: 1.66
61.57 + 1.42
IN2
0.77
1.37
0.43
1.05
IN2
24.25 0.86
47.62 + 1.52
21.30
1 1.55
49.11 - 1.74
Ours Improvement
+0.79
+6.25
+0.80
+99.17
Exemplat-based baselines
GEM
24.27
1.57
74.28 + 2.54
10.34 + U.73
41.49 + 1.62
Ours
IN2-GEM
25.41 + 2.06
66.91 + 2.16 +7.37
12.14 + 0.53 +1.60
38.23 + 0.42 +3.26
Ours Improvement
+1.14
Exemplar-basse based baselines
MIR
24.32
2.09
61.22 + 1.69
11.58 +: 2.08
44.17 + 2.06
Ours
INR-MIR
31.03 + 2.23
60.39
+ 1.06
14.86
+ 1.10
46.24 +: 2.81
Ours Improvement
+6.71
+0.83
+3.28
-2.07
5.2.2 Comparison results for IN2
Table g compares the results of IN2 and baselines. It can be seen that our Finetune-CBM already outperforms existing exemplar-free methods balanced average accuracy Ar with often better balanced average forgetting Fr, which indicates the impact of CBM in continual learning. Notably, our IN2 even has better performance both metrics by up to 0.8% in Ar and 9.1% in Fr. Additionally, combining IN2 with exemplar-based methods can further yields more improved performance for the gcemplar-basedd methods by up to 6.7% in Ar and 7.3% in Fr. However, the tradc-off between interpretability and accuracy causes a minor performance drop for MIR. CBM transforms a neural network's 's architecture inlo a more interpretable one, which causes a slight decrease in accuracy as LF-CBM (Oikarinen et al., 2023) and Post-hoc CBM (Yuksekgonul et al. 2023 shows. However, IN2 increases interpretability significantly compared with the previous continual learning methods. When remove duplicate concepts in concept set expansion, both Ay and Fr are worse than original IN2, but still better than the baselines in Fr with similar År. Since the same text concepts represent.
10

---

Published in Transactions on Machine Learning Research (09/2024)
different visual concepts, as demonstrated in Appendix C.3 adding duplicate concepts help IN2 achieves
better performance.
Remark. Our methods, IG-CL and IN2, introduce interpretability to continual learning for the first time. While often achieving superior performance on two standard metrics compared to baselines, it's important to emphasize that our primary goal extends beyond mere performance enhancement. Our approach may sometimes yield accuracy comparable to or slightly worse than existing methods in 5-task scenario and different settings in Appendix B yet it stands out by significantly boosting interpretability This innovation marks pivotal step in making continual learning not just more effective, but also more understandable.
5.3 Discussion on Concept Evolution
In this section, we visualize the learned concept give more insights on the proposed IG-CL and IN2.
IG-CL. For IG-CL, we study the evolution of the concepts represented by neurons as we train across different tasks. We analyze the case which we group similar classes into the same task, so we can recognize which task a concept belongs to easier. For CIFAR-100 and Tinylmagenet the class distributions are in appendix D.3 We try to maximize the diversity between tasks, which makes tasks share less concepts and become more challenging in continuous learning regime. First, we use CLIP-Dissect (Oikarinen & Weng 2023) to analyse how many units are still detecting the same concept after learning new task. The results are in Figure and Table 4 The values are in Table 5 in Appendix A.I Compared with existing methods, our method has better ability to retain knowledge of concepts, with by up 62.5% improvement in the preserved ratio as Table 5 shows. We also do a case study to understand how well our method preserves the concept units from the previous tasks. Table 4 shows results for some example neurons. Some concepts are preserved after learning unrelated new tasks, which gives us insight on how our method helps avoid catastrophic forgetting. As Table 2 shows, IG-CL outperforms baselines by up to 1.3% in forgetting metric Fr.
Table 4: Concept evolution for IG-CL in freeze-all implementation. We analyse the concept evolution in the layer 4 of 18 The blue concept means the concept is related to the current task, while the green concept means it is unrelated. "x" stands for non-interpretable units. The results show that IG-CL can preserve the concepts from previous classes.
Tinylmagenet, 5T
task 1
task 2
task 3
task
task 5
Hunsn-made
Big animals
Small animals
Food & Clothes & Others
Classes
Big objects
small objects
& Natural seenes
& Sea animals
Unit 85
Kitchen
Kitchen
Kitchen
Kitchen
Unit 123
x
Bedroom Bus
Bedroom Bus
Bedroom
Bedroom
129
Bus
Dus
Bus
CIFAR-100, 5'T
task 1
task 2
task
task
task
Suall aninuls
Natural sceues
Classes
Big animals
Big objects
Others
& Sea animals
& Plants
Unit 307
Kitchen
Kitchen
Unit 310
Highway
Higlway
IN2. For IN2, also studied the evolution of concepts across different tasks. Figure 5 shows example for CIFAR-100 under 5-tasks scenario, with more results in Appendix A.2 As discussed in the previous section that though our Fine-tune CBM results is already better than many Exemplary-free baselines, our IN2 is even better than Fine-tune CBM, especially on Fr, with up to 12.4% improvement. This can be seen in Figure 5 that IN2 is much better at retaining and using knowledge of concepts learned from previous tasks, which is the key to help to combat against catastrophic forgetting. As Table B shows, IN2 outperforms baselines by up to 9.1% in forgetting metric Fr.
11

---

Published in Transactions on Machine Learning Research (09/2024)
Task
Task 2
Concept
Class
Concept
Class
family
alimy body
black and yellow strppe
fowers
plants
pllar
caterpiitar
rocka
small round eyea
water
lseves
not brown fur
Nature trail
(a)
Finetune CBM
Task
Task
Concept
Class
Concept
Class
cocoon
not family beavers
plant
plan
clack and yellow stripes
back and
not brown
celer pillar
brown fur
caterpiller
grass
larve
grass
leaves
reeds
trees
bress
(b) IN2
Figure 5: Final weight visualization for random classes in (a) Finetune- CBM and (b) IN2 trained on CIFAR- 100 under 5-tasks scenario. For class 'caterpillar' from task 1, we show its significant weight (> 0.16) after training on task and task Concepts generated from the caterpillar class itself are colored blue, and other concepts from task 1 are colored gray. The class distribution is in Table 38 final layer while Finetune-CBM loses most significant weights.
We can see IN2 keeps similar
5.4 Ablation Studies
To understand the contributions IG-CL and IN2's continual learning abilities, we do ablation studies analysis each part of the methods. The experiment results show that utilizing all steps in Section 8 and Section | leads to best. performances in continual learning. For IG-CL, freezing concept. subnetworks improves Ar by up to 3.1% and 4.8% in Fr, and regularizing We improves Ar by up to 3.6% and 7.3% in Fr. For IN2, regularizing WF improves A7 by up to 6.2% and 2.5% in Fr.
Appendix B.1
6 Conclusion
In this work, introduced two novel interpretable continual learning frameworks: IG-CL and IN2. The main goal of our research is to integrate interpretability into continual learning, enhancing transparency and control throughout the learning process. Although our primary focus is not on performance enhancement, our findings demonstrate that. IG-CL and IN2 frequently outperform existing methods. Specifically, these frameworks improve upon previous continual learning methods by up to 1.4% in average incremental accuracy, and up to 9.1% in average incremental forgetting. Furthermore, they effectively preserve learned knowledge without the need for storing past data. The clear and interpretable nature of IG-CL and IN2 sets a strong foundation for future advancements in the field of continual learning.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Broader Impacts
Our methods make black-box continual learning process become interpretable, which is beneficial to modcls' performance. However, the improvement of miligating forgetting by controlling concepts in models might have some potential negative impact in terms of privacy. For example, if an adversary only has access to model checkpoints but not model's training data, they can analyse the concept units in the model. This will give the attacker some information about how the model was trained, and may allow them to extract some private information from the models without access to training data.
Acknowledgement
This work is supported in part by National Science Foundation (NSF) awards CNS-1730158, ACI-1540112, ACT1541349, OAC-1826967, OAC-2112167, CNS-2100237, CNS-2120019, the University of California Office of the President, and the University of California San Diego's California Institute for Telecommunicatonn and Information (echnology/Qualcomm
Institute. Thanks to CENIC for the 100Gbps networks. The authors thank the anonymous reviewers for valuable feedback on the manuscript. Oikarinen and T.-W. Weng are supported by National Science Foundation awards CCF-2107189, TIS-2313105, ITS-2430539. T.-W. Weng also thanks the Hellman Fellowship for providing research support.
References
Rahal Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuylelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in neurol information processing systems, 2019.
Nicholas Bai, Rahul A Iyer, Tuomas Oikarinen, and Tsui- Wei Weng. Describe-and-dissed. Interpreting neurons vision networks with language models. ar preprint arXiv:2403.1 137711 2024.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolci Zhou, and Antonio Torralba. Under- standing the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple bascline. Advances in neural information processing systems, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccial Issam Hadj Laradji, Irina Rish, Alexandre Lacosto, David Vázquez, et al. Online fast adaptation and knowledge accumulation (osaka): new approach continual learning. Advances in Neural Information Processing Systems, 2020.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Xinyuan Cao, Weiyang Liu, and Santosh Vempala. Provable lifelong learning of representations. In International Conference on Arlificial Intelligence and Stalistics, 2022.
Francisco M Castro, Manuel Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End- to-end incremental learning. Tn Proceedings of the European conference on computer vision (ECCV), 2018.
Arslan Chaudhry, Punect K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the Europeon conference computer vision (ECCV), 2018.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019.
Xi Chen, Christos Papadimitriou, and Binghui Peng. Memory bounds for continual learning. In 2022 IEEE 63rd Anmual Symposium on Foundations of Computer Science (FOCS), 2022.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisol, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pottern analysis and machine intelligence, 2021.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In International Conference on Machine Learning, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob An- dreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2022.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVE conference on Computer Vision and Pallern Recognition, 2019.
Linmei HU, Tianchi Yang, Luhao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, and Ming Zhou. Compare to the knowledge: Graph neural fake news detection with external knowledge. In Proceedings of the 59th Annual Meeting of the Association for Computationll Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao Tao, Dongyan Zhao, Jinwen Ma, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In International conference on learning representations, 2019.
Hecchul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetting learning in deep neural networks. arXiv preprint arXie:1607.0011 2016.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. A theoretical study on solving continual learning. Advances in Newral Information Processing Systems, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinsk,, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Hyunsco Koh, Dahyun Kim, Jung- Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In International Conference on Learning Representations, 2022.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Justin Lee, Thomas Oikarinen, Arjun Chantha, Keng-Chi Chang, Yilan Chen, and Tsui- Wei Weng. The importance of prompt tuning for automated neuron explanations. arXiv preprint arXiv:2310. XX0::0300000200, 2023.
Kibok Loe, Kimin Lee, Jinwoo Shin, and Honglak Lec. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
Sang-Was Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment. matching. Advances in neural information processing systems, 2017.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pottern analysis and machine intelligence, 2017.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2021.
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido M Van de Ven, al. Avalanche: an end-to-end library for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2021.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in newral information processing systems, 30, 2017.
Max Losch, Mario Fritz, and Bernt Schiele. Interpretability beyond classification output: Semantic bottleneck networks. arxiv preprint [xXic:1907.10888 2019.
Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea. Passerini, and Stefano Teso. Neuro-symbolic continual Icarning: Knowledge, reasoning shortcuts and concept rehearsal. In International Conference on Machine Learning, 2023.
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. In International Conference on Machine Learning, 2022a.
Scyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mchrdad Farajtabar. Architecture matters in continual learning. arXiv preprint Xrxiv:2202.000 00275, 2022b.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 2020.
Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. International Conference Learning Representations, 2023.
Tuomas Oikarinen and Tsui- Wei Weng. Linear explanations for individual neurons. In International Conference on Machine Learning, 2024.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wel Wei Weng. Label-free concept bottleneck models. In International Conference on Learning Representations, 2023.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Binghui Peng and Andrej Risteski. Contimual learning: feature extraction formalization, an efficient algorithm, and fundamental obstructions. Advances in Neural Information Processing Systems, 2022.
Liangzu Peng, Paris Giampouras, and René Vidal. The ideal continual learner: An agent that never forgets.
In International Conference on Machine Learning, 2023.
Anastasia Pentina and Ruth Urner. Lifelong learning with weighted majority votes. Advances in Newral Information Processing Systems, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
Sylvestro-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference Learning Representations, 2019.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.00 606.6066..0 04671, 2016.
Paul Ruvolo and Eric Eaton. Ella: An efficient lifelong learning algorithm. In International conference machine learning, 2013.
Dawid Rymarczyk, Joost van de Weijer, Bartosz Zieliński, and Bartlomiej Twardowski. Icicle: Interpretable class incremental continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-lovel interpretation of deep nlp models: survey. Transactions of the Association for Compulational Linguistics, 2022.
Jonathan Schwarz, Wojeicch Czarnecki, Jelena Luketina, Agnieszka Barakkaa,, Yee Whye Tch, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continnal learning. In International conference on machine learning, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, 2018.
Albin Sormif-Cormerais, Antonio Carta, and Joost Van de Weijer. Improving online continual learning performance and stability with temporal ensembles. In Conference on Lifelong Learning Agents, 2023.
Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng. Vlg-cbm: Training concept bottleneck models with vision-language guidance. arXiv preprint, 2024.
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning, Nature Machine Intelligence, 2022.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-ucsd birds-200-2011 dataset. 2011.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing HONG, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations, 2022.
Shipeng Wang; Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in mull space of feature covariance for continual learning. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognilion, 2021.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedlings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognition, 2019.
Ju Xu and Zhanxing Zhu, Reinforced continual learning. Advances in Neural Information Processing Systems, 2018.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVE International Conference on Computer Vision, 2023.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021.
Sin-han Yang. Chung- Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. Entity-aware dual co-attention network for fake news detection. In Findings of the Association for Computational Linguistics: FACL 2023, 2023a.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- and Mark Yatskar. Language bottle: Language model guided concept bottlenecks for interpretable image classification. Proceedings of the IEEF/CVF Conference on Computer Vision and Pallern Recognition, 2023b.
Haiyan Yin, Ping Li, et al. Mitigating forgetting in online continual learning with neuron calibration. Advances
Neural Information Processing Systems, 2021.
Jaehong Yoon, Funho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Fleventh International Conference on Learning Representations, 2023.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In
International conference on machine learning. PMLR, 2017.
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2017.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Compuler Vision (ECCV), 2018.
Da-Wei Zhou, Qi- Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-incremental learning: A survey, arXiv preprint 23x.:2302.00645 2023.
Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zueng-Jun Zha. Self-sustaining representation expansion for non-exemplar class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Appendix Outline
In this section we provide a brief overview of the Appendix contents.
A
Interpretability Analysis
IG-CL's Concept Evolution
Discussion of Concept Evolution for IN2
B
Additional Experiments
B.
Ablation Study
Ablation on IN2 Sparsity
B
More 5 task Results
B
10, 20 tasks Experiment Results
B.
Comparison with SSRE, DEN and ICICLE
Foward Transfer Metric
B.7
Experiment Results on ImageNet-10
B.8
Impact from the order of tasks
Further Analysis and Details
Computational Efficiency
Interpretability Guided Continual Learning Algorithm
Details of IG-CL and IN2
Others
D.1
Experiment Setup and Details
Extended Previous Works Introduction
Class Distribution
18

---

Published in Transactions on Machine Learning Research (09/2024)
A Interpretability Analysis
A.1 IG-CL's Concept Evolution
Figure and Table show the IG-CL's concept evolution when grouping similar classes together. The analysis is in Section 5.3 Meanwhile, we visualize the classification head's weight. for IG-CL in Figure G IG-CL can preserve classes' related concepts contribution to prediction. Compared with LN2 that analyzed Figure 5 IG-CL's perserved concepts are more general and less task-specific since IG-CL can not control the emergence and lypes of related concepts.
Besides grouping similar classes into the same task, also analyse the cases where class labels are distributed randomly. Following the same procedure as in section 5.3 we analyse how many units are still detecting the same concept after learning new task. The average results for three datasets are in Table O Similar to Table 5 our methods outperform existing methods to retain knowledge of concepts learned from previous tasks. In general the concepts are more stable in random class distribution since different tasks might share more overlapped concepts.
Table 5: The ratio of units that still detect the same concepts they detected in the last task. Tasks' classes Underline means the strongest baseline, and boldface means the best number. Our methods outperform existing exemplar-free methods for preserving concepts, and are even better than exemplar-base methods without replaying buffer.
in Appendix D.3
Method
CIFAR100, 5T
TinyImagenet, 5T
Task 2
Task 3 Task 4 Task 5
Task 2
Task 3
Task 4
Task 5
Exemplar-free baselines
Finetune
0
0
0
0
0
0.086
EWC
0.030
0.037
SI
0
0.028
0.191
0.200
0.250
LwF
0.019
0.011
0.176
O
0.114
0.191
0.393
0.365
Exemplar- based baselines
GEM
0.125
0.076
0.153
0
0
0
MIR.
0
0,250
0.028
0
0
Ours
(G-CL-froeze-al (no replay)
0.500
0.500
0.571
0.700
0.739
0.771
0.812
0.812
IG-CL CL---recrzz-prt (no replay) 0.125
0.133
0.727
0.695
0.515
0.488
0.544
0.606
Table 6: The ratio of units which still detect the same concepts they detected in the last task for CIFAR-10 and CIFAR-100. Class labels are distributed randomly, and the results are average over three runs. Underline means the strongest baseline, and boldface means the best number. Our methods outperform the existing works for preserving learned concepts.
Method
CIFARIO,
CIFAR100, ST
Tiyllsggeet, ST
Tack 2 Task Task 4 Tisk Task
Task
Taek
Task 5
Task
Tuok
Task
Thek
Exemplar-free baselines
0.005
0.010
(L030
BWC
0
0.065
0.010
0
01
0.015
0.010
DX011
LwF
0
0
0.020
0.030
0
0
0.015
0
Exemplar- baselines
GEM
0
0.033
0
0.118
0.042
0.008
0L023
D
0.041
0.012
0.029
Ours
CL-CI-Ereeseal
1G- OL-Crewse-part
0.023
0,150
0.090
0.150
0.210
0.166
0.687
0.869
0.586
0,739
0.686
0.821
19

---

Published in Transactions on Machine Learning Research (09/2024)
Task 1
Task 2
Concept
Class
Concept
Class
stripes
textile
hardwood
grounds
beige
magenta
yellow
caterpillar
orange
caterpillar
landscape
dotted
field
zebra
magenta
textile
dotted
mesh
countryside
brown
meadow
green
pink
field
pink
granite
(a) caterpillar
Task 3
Task 4
Concept
Class
Concept
Class
dorm
grounds
latice
logo
ground
mesh
snake
snake
lattice
dotted
dotted
townhouse
renovated
stripe
(b) smake
Figure 6: Final weight visualization for random classes in IG-CL CIFAR-100 under 5-tasks scenario. For class 'caterpillar' from task 1, we show its significant weight (> 0.05) after training task 1 and task 2. For class 'snake' from task 3, we show its significant weight after training on task 3 and task 4. Concepts related to caterpillar, and preserved from task to task 2 are marked in blue. Same for concepts related to snake and preserved from task 3 to task The class distribution is in Table 38 preserve related concepts' appearance and contribution.
We can see IG-CL has ability to
20

---

Published in Transactions on Machine Learning Research (09/2024)
A.2 Discussion of Concept Evolution for IN2
In this section, study the evolution of the concepts represented by neurons and final layer weights as train IN2 across different tasks under 5-tasks scenario. We analyse a classes' final layer weights after learning its task, and after learning a new task. We visualize the final layer weights of Finctune-CBM and IN2 by Sankey diagrams, only including weights with absolute value greater than predefined threshold. Negative weights are reported as "NOT {concept)'. The visualization for random classes in CTFAR- 100 are in the main text Figure 5 The visualizations for random classes in other two datasets in Figure 7 Compared IN2 is much better at retaining and using knowledge of concepts learned
with Finetune- CBM, from previous tasks. This helps explain why IN2 performs better in Pr and Fr.
Task 4
Task 5
Concept
Class
Concept
Class
a road
reddish-brown coat
taillights
field
grass
green or brown body a cab for the driver
grass
truck
truck
a flatbed for carrying cargo a highway tailgate
large, brown eyes
whiskers
trailer
(a) Finetune- CBM
Task 4
Task 5
Concept
Class
Concept
Class
a road taillights
a road
not boat not bow and stern a cab for the driver not a captain a flatbed for carrying cargo a tailgate trailer a truck stop not sails not the sea
truck
truck
tailgate
(b) IN2
Figure 7: Final weight visualization for random classes (a) Finet and (b) IN2 trained on CIFAR-10 under 5-tasks scenario. For class "truck' from task 4, we show its significant weight (> 0.15) after training on task and task 5. Concepts generated from the truck class itself colored blue, and other concepts from task are colored gray. Concepts from task 5 are colored green. The class distribution is in Table 36 We can see both models change significantly, but IN2 keeps more concepts from original task.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Concept
Class
Concept
Class
digital
pressure
stren
blades for cutting grass
emergency suppless
grass
plenty of space for storago
measuring
stopwatch
wheel
second hand or digitl time
devico
label with product information
headphonee jack
timeploce
tmoplece
(a) Finetune-CBi CBM
Concept
Class
Concept
Class
adal
pressure gauge
pressure gauge
second digital timer
stopwalch
second digital
meter
meter
tmoplece
timepisce
(b) IN2
Figure 8: Final weight visualization for random class in (a) Finetune-CBM and (b) IN2 trained TinylmageNet under 5-tasks scenario. For class "stopwatch' from task 2, we show its significant weight (> 0.05) after training on task 2 and task 3. Concepts generated from the stopwatch class itself are colored blue, and other concepts from task are colored gray. The class distribution is in Table 40 We can see IN2 weights stay almost identical, while Finetune- CBM changes lot.
22

---

Published in Transactions on Machine Learning Research (09/2024)
B Additional Experiments
B.1 Ablation Study
Besides comparing the results with cxisting mothods, we perform ablation study to analyze the impact of the key components in our methods. In Table 7 we compare IG-CL with (i) not freezing subnetwork step 3 which denoted as 'IG-CL w/o freeze', and (ii) not regularizing parameter Wr in step | which is w/o reg'. The experiment results show that
denoted as 'IG-CL-freeze-all w/o reg" or "IG-CL-freeze-pee combining two of them results in less forgetting and better average accuracy. We believe it's because they are necessary to each other. If the concept units' subnetworks are not frozen, the concepts might change and the regularization term may not work. On the other hand, if the WF changes when learning new tasks, then the concopt units learned from previous tasks may not contribute to the final prediction.
For IN2, we do a similar ablation study with (i) not freezing previously learned concepts when learning CBL in step 2 which denoted as 'IN2 w/o froeze', and (ii) not regularizing parameter Wp in step 3 which is denoled as 'IN2 w/o reg". The result Table shows similar result. In conclusion, these ablation studies demonstrate the importance of both freezing subnetworks and regularizing the WF to achieving better performance and mitigating forgetting.
Table 7: Experiment results of IG-CL's ablation study. 'IG-CL w/o freeze': not freezing subnetwork. IIG-L--freze-all w/o reg' and 'IG-CL-freeze-pe paret w/o reg': not regularizing parameter Wp. means larger values are better, while & means smaller values are better. Preserving two components turns out have the best performances.
CIPAR-10,
CIFAR-100 S1
Tlyytaggeee, ST
Fr
A,
Ay
w/la
=
48:25
wyjo
05:30 1:7
10.32
36.77
50.09
froms par:
137
0201
2:00
51.25
4,21
frocs-all
81.55
22.97
0.20
18.19
0.70
0.92
IC-CL Brose- par:
00.56
94.47 1.12
21773
00.51
0.35
0.50
40.00
Table 8: Experiment results of IN2's ablation study. "IN2 w/o freeze": not freezing previous concepts. 'TN2 w/o reg": not regularizing parameter WF. Preserving both has the best performance.
CIFAR-1O,
CIFAR-100, ST
Inapurt, ST
Fy
Fy
A,
F
IN2 w/o Givoe
20.78 2.45
87.76 1.07
18.01 5-42
49.50 + 2.58
17.30 + 2.9
51.64 4.0
IN2 w/o
28.76
1.02
91.10
220
20.42
2.88
58.72 # 1.04
19.76 2.26
62.24 # 288
IN2
88.88
47.02
B.2 Ablation on IN2 Sparsity
In this section experimented with some modifications to WF in our CBM and report their results. In our main results for IN2 and FinetuneCBBM, we used dense final layer Wp instead of the sparse final layer used by OOikarinen et al.) 2023) as we found that to have best performance, and we are less focused on interpretable final decisions, Below we compare the results for CBM between dense We, and sparse We (-S). For the sparse WF, we used À 10-6 in Eq. 3 The experiment results are in Table 9 We found that dense We had the best performance most cases.
Table 9: Experiment results of CBM based methods on 3 datasets. 'S' sparse Wp. Dense WF results in best performance in average.
CIEAR-1OST
CIFAR-IBL
HaylingeNe. ST
Ap
Fr
Ay
Fr
Cin
0.67
08.09
24.90 0.79
38.20 1.10
1.66
61.57
IN2
32.25 :: 0.76
88.58
0.30
24.25
0.86
47.02
+: 1.52
21.30 :: 1.55
40.11
1 1.74
Finetor--CMM
28.84 + 1.35
01.74 2.22
20.23 2.79
58.20
+ 3.12
10.50 + 2.53
61.57
2.57
IN2S
3.04
10:20
2.62
IN2
1.74
25.41
12.14
0.53
IN2-
NN-GEMMS
36-32-1 1. 2.00
45.28 4.17
26. 2: 4.15
65.21
3.28
5.32 1: 1.05
30.23
23

---

Published in Transactions on Machine Learning Research (09/2024)
B.3 More 5 task Results
In Table 10 and 11
we report Ar and Fr for CIFAR-100 and TinyImageNel, and also experimenl results for
CIFAR-I 10 in 4 metrics.
Table 10: Accuracy comparison for IG-CL. t means larger values are better, while + means smaller values are better. Underline means the strongest baseline, and boldface means the best number. In the Ours Improvement rows, +/- mcasures our best number minus the strongest bascline, and the value is in blue if our improvement is statistically significant.
CIFAR-10,
CIFAR-10,
CIFAR-100,
TisyImagenet,
Ag
Ay
Pr
Ay
Fy
Ay 1
Fy
Exemplar-fren baselline
Finelume
27.46 0.89
95.86
1.28
15.89
0.76
78.51 +: 0.87
16.48
2,63
05 3:24
13:20 :: 2.25
43,50 +: 0:45
EWC
3005 0.79
94.13
2.26
18.72
1.82
76.06
3.01
15.70
1.88
3.87
13:07
3.55
43.04
SI
29.64 + 0.35
94.21
1.06
19.06
1.80
14.19
+ 1.48
13.30 + 2.62
50.59
3,50
10.51
+ 3.01
30.19 + 3.91
LeF
1.30
19:01
0.49
75.00
17.92
2.54 72.35
13:47
ASCm-
30,23 1.02
94.82
0.50
20.00
0.71
76.82
1.72
17.70 + 0.85
1,25
14.15
2.19
62.71 + 2.03
IG-CL (G-CI-faeese-ll all
$1.55 =: 0.13
92.69 + 0.81
21.14
1: 1.90
70.68 :: 1.72
14:60 1.81
61.17 2,48
11.20 1: 0.48
37.85 = 0.83
JG.CI.-rreee-pear part
1.12
20.71 2.29
2.85
3.67
Ours Improvementt
+1.32
+1.44
+0.54
+3.51
-2.75
-0.58
-2.81
+1.31
Exemplac-osed baselimes
GEM
34.59 + 0.05
90.00 + 3.80
22.51
0.22
78.50 +: 3.61
19:07 + 0.62
62 2:74
11.12 4: 1.79
37 37.08 1.15
I0-CI-freoe-lll --ress--ll-GGMM C...
35.60 + 0.62
94.93
0.66
24.66 +: 1.63
68.96 + 0.55
25.66 35.68 + 1.72
36.88 0.26
9:89 +: 2.56
$3.45 = 3.05
37.00 0.96
92.22
2.00
20.22 1.48
72.18 2.73
23.70 3.84
43.73
3.81
10.91 2.26
36:17 2.35
Ours Improvement
+-2.11
-2.22
22115
+44.54
+-6.61
+ +16.04
-0.21
+-4.13
Exemplac-bused baselines
MIR
28.97
2.34
89.80 + 0.99
29. 18 1.70
47.71 # 1.82
9:75 + 3:39
43.09 0.68
10.91 #: 3.80
38.42 + 1.57
Ours
IG-CI-freese-lee -ll---rsee-all---- all-MilR- MIR
30.23 1.51
91.01 2.57
33.16
3.52
42.00 1: 2.92
14.49 0.05
13.82 0.12
12:29 1: 0.60
38.42 = 1.78
IG-CL-freese MIR
30.75 # 1.37
91.07
2.04
34.88 1.48
42.11 + 2.11
14.37 à 2.99
44.41 1.38
12.47
38.50 3.76
Ours Improvementt
+1.78
-1.21
+6.70
+5.71
+4.82
-0.73
+1.5
0.00
Table 11: Accuracy comparison for IN2. All models are pre-trained on the Place365 dataset (Zhou el al. 2017]. means larger values are better, while & means smaller values are better. Underline means the strongest baseline, and boldface means the best number. In the Ours Improvement rows, +/- measures our best number minus the strongest baseline, and the value is in blue if our improvement is statistically significant.
CIFAR- 10, ST
CIFAR ST
CIFAR 100, ST
TisyInagenet, 5T
Ap
Fr
Ap
Pr
Ap
Fy
Ay
Fy
Exemplar-free basellines
Finetune
20.13 + 0:28
97.78
0.78
17.99
2.26
79:66 +: 0.72
17:65
+ 2.55
64.94
2.15
15:34
+ 3.3
54.81 + 0.71
EWC
30.79
97.92
0.78
19.00
1,87
76.92
0.77
17.44
3.98
64.43
1.84
15.07
2.23
54.77
0.54
SI
30.25 2: 0.72
96.55
1.70
19.31
0.62
74.78
1.55
17.12
3.89
63.11
1.03
14.47
4 1.02
53.69
4: 1.32
LWF
30.76
97.75
0.84
19.11
0.44
76.36
19.98
0.64
63.79
2.65
17.08
55.40
1.85
ASCL
30.78 4: 0.82
96.82
2.35
21:00
0.48
77.73 4: 0.86
16.23
3.39
66.26
3.18
13.44 1: 1.70
58.86 4: 0.70
Ours
Fieetue-CBM
30.30 0.67
93.00
2.22
21.14
0.83
69:47 + 1.29
16:43
2.82
66. 76
3.53
14.91 :: 0.65
31.39 3.78
IN2
$2.25 : 0.76
68.58
0.30 +7.97
22.16
1.73
67.92 1 0.72
16.37
3.82 -3.65
45.71
2.55
15.08 1: 1.84
39.40 = 2.86
Ours Improvement
+1.46
+0.56
+-6.86
+17-40
-1.40
+14.29
Exemplar-bnsed baselines
1.86
69.38 2.88
1.75
63.49 3.15
5.64
0.38
37.52
Ours
IN2-GEM
36.37
44.72
2.71
28.93
23.64
3.82
58.54
3.75
1.67
29.06
Ours Improvement
+0.10
+24.66
+44822
-2.02
+2.99
-4.95
-0.27
+8.46
Exemplar-bose baselines
MIR
33.20 + 2.92
11 1.94
33.99 2.00
48.50 + 2.32
8.33 +: 0.64
46.33
2.06
2.97 +: 3.69
32.09 0.53
IN2-MIR
36.61 :: 1.77
70.45 : 1.89
33.91 1.25
38.93 :: 1.74
13.81 1: 1.94
51.76 1: 1.59
5.07 3.66
31.75 4: 1.54
Improvement
+33.41
+5.06
+5.48
22.66
24

---

Published in Transactions on Machine Learning Research (09/2024)
B.4 10. 20 tasks Experiment Results
Table 12 reports comparison for IG-CL under 10-tasks scenario for CIFAR-100 and TinyImagenet. Compared with exemplar-free baselines, IG-CL generally forgets less as Fr and Fr shown. However, IG-CL sometimes is worse in Ar and Ar, which is the limitation of our methods. We view this as the future works to be improved, since this paper's goal to retain intepretable concepts in models. Nevertheless, IG-CL generally improves exemplar-hased baselines when combined with them, which is same as 5-tasks scenario.
Table 13 reports comparison for IN2 under 10-tasks scenario for CIFAR-100 and Tiny Imagenet. Similar to CL, IN2 performs well in forgetting metries Fr and Fr. but performs worse in Ar and Ar when comparing with exemplar-free baselines. We believes this is partially because LF-CBM's training procedure is different than standard end-to-end training. As LF-CBM Oikarinen et al. (2023) paper shown, LF-CBM's classification accuracy is usually woise than standard model's. Again, we view this as future works to be improved. Besides, IN2 still can improve exemplar- based baselines when combine with them.
Table 14 and 15 report comparison for IG-CL and IN2 under 20-tasks scenario in CIFAR-100 respectively. Similar 10-askss scenario, our methods can improve exemplar-based baselines when combine with them, while the comparison with exemplar-free methods is still under the same trend.
We observe that both of our proposed methods can improve exemplar- based methods when combined with them. When compared with exemplar-free methods, our methods forget less while slightly worse than the strongest baselines in terms of Ar. Nonetheless, we would like to highlight that our work is the first to bridge continual lcarning and neuron interpretability with competitive results, which opens up a new and promising direction transform continual learning from black-box process to more transparent.
Table 12: Accuracy comparison for IG-CL in 10 tasks scenario. means larger values better, while 4 means smaller values are better. Underline means the strongest baseline, and boldface means the best number. In the Ours Improvement rows, +/- mcasures our best number minus the strongest bascline, and the value is in blue if our improvement is statistically significant.
CIFAR-100,
TinyInmgenet, 10T
Ap
Fr.
Ap
Fy
Ay
Exemplar-froe baselines
Finetune
6.49 1.20
65.51 + 1.32
13.87
0.83
64.20
1.52
5.15 4 2.13
48.07
0.57
10.23
1.35
45.83 + 0.21
EWC
6.71 0.63
65-32
+ 1.12
13.88
0.45
63.90
0.94
5.48
+ 2,35
47.42
1.47
10.11
2.12
45.19
1.29
SI
6.74 1.17
66.07
1.91
14.01
1.37
64.39
1.53
5.30
4: 1.57
44.68
1.40
10.33
1.32
46.21
1.36
LwF
7.85
61.13
2.19
15.02
à: 2.85
62.68
1.25
5.95
4: 2.35
52.17
1.26
10.82
2.35
15.57 1: 1.36
AdCm-NSCL
7.95 2.77
62.54
1.80
13.00
2.15
61.54
2.40
6.63
2.23
49.43
1.45
10.63
1.33
49.43
1.45
Ours
IG-CL-fresseal
6.99 1.20
62.23
0.36
13.81
1.45
62.79
1.70
5.76 4: 2.67
48.89
2.17
11.24
0.34
49.39 #: 1.92
IG-CL CL-Eresze-per
6.97 1.20
64.03 1 1.32
13.71 1: 1,36
63.25
1.04
5.86 1: 0.56
19.99
2.16
11.25
1.28
45.00 1: 1.20
Ours Improvement
-0.96
+0.31
-1.21
-1.74
.0.77
-4,21
+0.43
+0.19
Exemplar- baselines
GEM
8.25 1.07
60.51 1: 1.25
15.71 4: 1.35
58.77 2.42
6.00 2.34
47.13 1.25
10.74 2.02
47.68 1.67
Ours
IG-CL- (G-CI--reese-all Ereese-al--GEr.
15.61 1.02
52.01
1.34
29.83 1: 2.15
52.43
2.36
3.03 1: 3.05
22.39 1: 3.91
7.75 2:90
31.75 1: 2.37
VO-CI--recse-part--C
7.65 1.85
80.12 +: 1.57
14.84 #: 1.39
54.12 2.73
2.97 4: 2.34 -2997
24.91 4: 2.02
5.52 4: 2.40
30.44 4: 2.59
Ours Improvement
+7.36
+8.30
+11.12
+0.31
+24.74
-2.99
+17.24
Exemplar-hase baselines
MIR
4.64 0.37
59.37 4: 1.98
12.61 1.02
60.51 1.72
4.54 4 0.82
46.19 2.43
10.11 1.34
46.19 2.17
Ours
5.43 à: 1.54
52.22 1.65
12.86 1.08
56.19
2.30
5.09 4: 3.21
41.34 4: 1.28
9.97 2.41
43.21 4: 1.32
LE-CC-Eresze pari--IR
5.12 1.31
52.34 1: 1.37
12.89 1: 0.62
56.72 1.84
5.17 :: 2.58
12.87 0.99
9.99 1.28
13.25 1: 1.41
Ours Improvement
+0.79
+7.15
+0.28
+-.33
+0.63
++4.85
.0.12
+2.98
25

---

Published in Transactions on Machine Learning Research (09/2024)
Table 13: Accuracy comparison for IN2 in 10 tasks scenario. All models are pre-trained on the Placc365 dataset (Zhou et al, 2017 t means larger values are better, while & means smaller values are better. Underline means the strongest baseline, and boldface means the best number. In the Ours Improvement rows, +/- measures our best number minus the strongest baseline, and the value is in blue if our improvement is statistically significant.
CIFAR-100, 10T
Tiny
1UT
Ay
Fr
Ay
Fr 4
Ar 1
Fr
Ar
Fr L
Exemplar-free baselines
Fiaetune
8.00 +: 1.82
80.43
1.42
17.34 = 1.87
81.23 = 2.30
4.61
0.18
06.40
3.42
14.62 + 0.58
70.70 + 1.56
EWC
7.59 à: 1.52
80.99
1.28
17:23
0.96
81.08
2:34
5.62
2.38
67.27
1.30
14.88
4: 1.28
71.25 4: 1.38
SI
8.51 1: 1.29
81.13
1.83
17.53
0.65
81.60 = 2.45
7:90
1.40
71.50
0.82
15.65
+: 1.19
72.33
2.08
LooF
13.83 + 1:26
77.21
1.53
24.99
2.29
71.19
3.33
10.40 2.35
71:40
1.59
20.42
1.86
06,78 + 3.95
ASCL
14.29 1.19
09.75
3.15
23.19
1.38
76.21
1.29
9.74
2.23
66.87
1.45
18.63 1: 2.45
09.97 1 1.21
Ours
CIaetune-CMM
7.31
69.88
1.40
14.91
1.28
66.72
2.73
7:18 1.26
65.76
1.42
14.41
1.89
IN2
7.34 d: 1.68
37.90 d: 1.59
14.98
1.00
61.96 4: 1.73
7:37 +: 2.54
56,80
1.27
14.98 4: 2.53
60.02 = 3.09
Ones Improvement
-6.95
1-31.85
-10.01
9.23
-3.03
19.00
-5.44
1 6.76
Exemplar-hased baselines
GEM
11.47 = 1.32
3.14
22.21
1.20
70.54 =
4.12 1.79
54.29 1.62
13:44 1.33
03.58
Ours
IN2-GEM
8.06 -2.79
53.13
3.71
22.46
3.07
47.95
1.48
57.94 1.20
Ours Improvement
+13.31
+0.25
-11.18
-2.08
+0.31
+1.52
+5.61
Exemplac- baselimes
MIR
3.24 +: 2:00
68.77 # 1.69
14.19 2221
78.32 = 0.313
3:37 +: 0.84
58.83 +: 2.06
9-90 + 3.20
59.29
4:20
Ours
IN2-MIR
8.38 + 1.35
71.02 2.06
16.65 + 2.23
70.49 + 1.90
7.13 = 2.15
60.98 4: 2.38
14.07 :: 2.31
61.14 + 2.36
Ours Improvement
1.6.4
12.25
2.46
2.83
1.3.76
-2.15
1.6.7
-1.85
Table 14: Accuracy comparison for IG-CL in 20 tasks scenario. t means larger values are better, while + means smaller values are better. Underline means the strongest baseline, and boldface means the best num- ber. In the Ours Improvement rows, +/- measures our best number minus the strongest baseline, and the value in blue if our improvement is statistically significant.
Table 15: Accuracy comparison for IN2 in 20 tasks scenario. All models are pre-trained on the Place365 dataset (Zhou et al. 2017). means larger values are better, while + means smaller values are better. Underline means the strongest baseline, and bold- face means the best number. In the Ours Improve- ment rows, +/- mcasures our best number minus the strongest baseline, and the value is in blue our improvement is statistically significant.
CIFAR-100, ATT
Ap
CIPAR-120
Excmmplorffee beselines Finetize
S80-2 2.26
1816 2
9.72 3.54
70.25 2 0.85
SE FWC
336 0.16
08 4 0.71
077 277
70.11 4-3.5
Essmplar- dree baseiins
5.0 2:57
T1.8 2 2.17
1000 1.05
76.19 2:000
Firecene
110 5 0.00
7:17 -085
50:35 1.35
17.07 4 3,00
1.1 2.18
77 06 18
10.51 1.
75:35
EWC
242 2 3:2
71.00 = 2.41
8045
1.20
75.75 2 5.67
NSCL
10.09
19.45
S1
373 0.45
75:93
0039
11:31
2.45
8237 3.67
3.18
05.31
50.45
IC Ci----ls
8:35 1.18
05.12 9.02
8.80
66.31
IG CL-CL-oonz-pr
3.70-2 252
01 1101
MMM 2.26
7252 1.12
Ours
Oure Improvement
-U30
+-.20
-0.86
+3..00
SlutteeCCod
416 2:38
77.40 8:
50087
0.55
74.b 2 5,00
Dicemplae-hoee howeltree
TV2
401 9-87
66.55 3.23
11:45 7.15
T3.06 0.20
CEM
Our Improvement
40.4
+2.97
-4.45
+4.001
Escmplar-bone baseluus
VC-CL-Snos--e all-GEM
GESt
10-05 D..CI--nose.pott.oo CRM
68
Ours
Oura Empresveent
+3.82
+.3..0
+2.85
+1.06
EN2.GEM
12.87
2.97
0..2 0 1
20,02
2.85
68:64 015
Excempar-- hooedd hownituee
Emppeseeent
+0.55
-3.11
+22.04
-1.50
2.15
45:31 0.00
17.01 1310
37.71
Esemplar based baselines
1.01
0.41
15-CL-Soos-all all--E
10 1G-CCL-twor MrR
15.05
1.71
02 17
14,02
3.50
53.00
INS-MIR
19.53 4 0.08
35.31
1.58
19.53
1.00
50.79
Oura Imprevement
+3.7
$5.05
10.55
+1.07
Ours Improvenaant
+1.32
-0.02
+2.87
-4.45
26

---

Published in Transactions on Machine Learning Research (09/2024)
B.5
Comparison with More baselines
In this section, we do individual comparisons with SSRE (Zhu el al. 2022 DEN Yoon et al. 2018 and ICICLE [Rymarczyk et al. 2023).
B.5.1 Comparison with SSRE
We compare our methods with SSRE Zhu et al.
Since we can not access the pre-training details described in SSRE paper, compare our IG-CL with SSRE that is trained from scratch. Besides metrics described in Section 5.1 we also mcasure Learning Accuracy (Fr 1 E 0z,i) (Mirzadeh et al. 2022b; 2022a) to understand model's ability to learn new tasks. IG-CL forgets more as the performace in Fy is worse. However, it has much better learning ability as LT shown.
2022
Riemer et al. 2019 Yin et al.
2021
Mirzadeh et al.
Experiment results are Table 16
Table 16:
Comparison between IG-CL and SSRE Zhu et al. (002 in CIFAR-10 and CIFAR-100. 1 mcans larger values are better, while & means smaller values are better. Even though IG-CL forgets more it has better learning ability.
CIFAR-10, 5F
CIFAR-100, ST
CIFAR-100, 10T'
CIFAR-TO0, 20T
Tiny Inogpmet, 5T
Tinghaapent, 10T
1
Lr
1
Fy
Ly
Ar 1
Fr
1
Ap
Fr
SSRE
28.44
15.96
36.18
31.35
7.97
32.73
17.72
7.40
11.29
8.94
6.73
4.51
13.76
6.06
13.59
11.44
7.33
15.83
16.66
63.22
62.23
64.25
3.35
63.12
66.46
13.20
47.92
5.76
48.80
49.81
B.5.2
Comparison with DEN
We compare our methods with DEN YYoon et al. 2018) in the CIFAR-10 and CIFAR-100 dataset. We combine a pretrained model with a 2-layer DEN, and compare it with Finetune- CBB and IN2. The experiment results are Table IN2 and Finetune- CBM outperform DEN by up to 20.01% and 18.15% in Ar respectively.
B.5.3 Comparison with ICICLE
We compare IN2 with ICICLE (Rymarczyk et al. 2023 in the CUB200 (Wah et al. 2011 dataset. CUB200 is a bird species classification dataset that has 200 classes with 5994 training examples and 5794 testing examples. The experiment results are in Table 18 Our method outperforms ICICLE by 31.68% Fr.
Comparison between DEN Yoon et al. 2018) and our methods, the 5-task scenario of CIFAR-10 and CIFAR-100. The experiment results shows that our methods outperform DEN in Är by large margins.
Table 17:
Table 18: Comparison between IN2 and ICICLE Ry
marczyk et al. 2023 and other baselines in CUB200 5-task scenario. IN2 outperform ICICLE both År and Fr.
CUB200, 5T
CIFAR-1O, ST
CIFAR-100, 5T
Äy t
Fr. +
Ar 1
Fr
Ar 1
Fr
DEN
12.24
19.20
9.18
10.50
Exemplar-free baselines
Finetune- CBM (Ours)
30.39
93.09
24.99
59.29
Finetune
10.16
60.73
IN2 (Ours)
32.25
88.58
24.25
47.62
EWC
11.02
62.43
LwF
12.50
64.50
Adam-NSCL
13.27
40.73
ICICLE
11.40
53.66
Ours
IN2
9.42
21.98
27

---

Published in Transactions on Machine Learning Research (09/2024)
B.6 Forward Transfer Metric
We use Forward Transfer Metric (FWT) Lopez-Paz & Ranzalo 2017 to measure our methods' ability to reuse old knowledge on new tasks. We did experiments on CUB200, 5 tasks scenario since CUB200 shares several concepts among classes. Table 19 and show that our methods have better abilitics to share knowledge from old tasks to new tasks compared with baselines.
Table 19: Forward Transfer (FWT) Metric compari- son for IG-CL. Our methods outperform the baselines FWT by large margins.
Table 20: Forward Transfer (FWT) Metric compari- son for IN2. Our methods outperform the baselines on FWT by large margins.
CUB200, 5T
CUB200, 5T
FWT f
FWT t
Exemplar-free baselines
Exemplar-free baselines
Finetune
0.76
Finetune
1.43
EWC
3.83
EWC
6.88
SI
13.50
SI
11.67
LwF
10.20
LwF
10.36
Adam-NSCL
21.04
Adam-NSCL
17.63
Ours
Ours
IG-CI-freeze-al
28.79
Finetune-CBM
16.71
IG- IG-CL-freeze-patt
30.07
IN2
12.66
Exemplar-based baselines
Exemplar-based baselines
GEM
1.35
GEM
19.98
Ours
Ours
CG-CI-freez--al--GGM IG-CI-Ifrewx--patt-GGMM
26.76
IN2-GEM
27.78
26.68
Exemplar- based baselines
Exemplar-based baselines
MIR
3.20
MIR
-12.44
Ours
Ours
IG-CI-Ireezeall-MIRR
9.84
IG- IG-CL-freeze-all-MII IC-CI--frcoze-part-MMI
6.89
8.22
B.7 Experiment results on ImageNet-10
We conduct experiments on large scale dataset ImageNet-10, and the results are Table and 22 The experiment results still under the same trend other datasets: Our methods outperform other exemplar- free methods by up to 0.87% in Ar and up to 5.77% in Fr. Meanwhile, can improve exemplar exemplar-based methods by combining them by up to 1.83% in Ar and up to 2.87% in Fr
Table 21: Experiment results on ImageNet-10 10 in the 5-task scenario.
Table 22: Experiment results on ImageNet-10 in the 5-task scenario, the backbones are pretrained on Place365 dataset.
Fy
Finctune
23.33
81.20
Är 1
Fr +
EWC
24.14 0.09
78.46
IG Ci-freeze (Ours)
25.01 + 0.51
77.20 4 0.95
Finetune
23.83 + 0.93
81.92 4 1.38
GEM
31.31 0.82
61.71
0.82
EWC
25.40 + 0.39
78.98 + 1.51
C.---frezz--l--GEE (Ours)
33.17 0.50
59.65 1.29
IN2 (Ours)
26.12 1 0.84
73.21 1. 1.73
GEM
32.66 1 0.85
60.27 1 1.32
IN2-GEM (Ours)
34.37 + 0.43
57.40 + 0.99
28

---

Published in Transactions on Machine Learning Research (09/2024)
B.8 Impact from the order of tasks
To understand the impact from the order of lasks, conducted experiments on CIFAR-100 using the class distribution outlined in Table 38 where similar classes are grouped into the same task. We then swapped the order of task 1 and task and compared the results with the original task order. The continual learning performance for IG-CL is Table 23 and IN2 is in Table 24 In both task orders, our methods consistently preserve learned knowledge effectively. TG-CL and TN2 outperform other exemplar-free methods by up 0.14% in Ar and up to 26.39% in Fr. Additionally, they improve upon exemplar methods by up 1.45% in Ar and up to 11% in Fr. These results demonstrate that methods robust to changes in task order. Meanwhile, it shows that our methods don't 't have constant performances in Ar when changing class distribution and task order, yet it boosts interprotability a lot.
We also analyze the concept evolution of two methods like Section 5.3 For IG-CL, the concept evolution analysis in Table 25 Same as the original order that analyzed in Table 4 IG-CL can preserve concepts even Same the original order that analyzod in Figure 5 IN2 can preserve concepts while uinetune CBM loss all of the siguificant weights. Overall, the concept evolution analysis show that our methods can preserved learned concepts regardless of the task order.
after learning a unrelated task. For IN2, the analysis is Figure !
Table 23: Continual learning performance CIFAR-100 5-task scenario. Original order is the class distribution in Table 38 and Swap order exchanges the task 1 and task 2 of it. IG-CL has better Fr and similar or better Ar.
Original order
Swap order
Ar f
Fr 4
Ay f
Fr 1
Naive
12.83
45.01
12.50
47.94
ST
13.10
41.10
12.87
47.03
LwF
14.47
41.83
14.36
47.00
IG- IC-CI-freez--ll
13.93
36.85
13.63
43.36
IG-CI-firezz-part
14.61
39.19
13.33
46.29
GEM
16.57
40.31
16.01
45.92
IG-CI-frrezz-al--CEM
17.23
29.31
16.43
40.41
IC-CCL-freeze-part-Cie
15.88
34.54
15.08
41.91
Table 24: Continual learning performance on CIFAR-100 5-task scenario. Original order is the class
distribution in Table 38 and Swap order exchanges the task and task 2 of it. IN2 has better Fy and similar or better Ay.
Original order
Swap order
Ar t
Fr +
Äy t
Fr +
Naive
14.64
56.16
14.27
59.95
SI
15.20
55.98
15.58
59.12
LwF
18.32
55.26
18.74
52.13
Enetune_cbre
13.95
37.51
11.41
42.29
IN2
15.34
28.87
16.23
33.32
GEM
17.02
55.35
20.61
53.22
IN2-GEM
18.15
48.50
22.06
52.71
29

---

Published in Transactions on Machine Learning Research (09/2024)
Table 25: Concept evolution for TG-CL in freeze-all implementation, testing on class distribution 38 but swap task 1 and task 2. We analyse the concept evolution in the layer 4 of ResNet18. The blue concept means the concept related the current lask, while the green concept means it is unrelated. "x" stands for non non-interpretable units. The results show that IG-CL can preserve the concepts from previous classes.
CIFAR-100, 5T
task 1
task 2
task 3
lask 4
task
Natural scenes
Small animals
Classes
Big animals
Big objects
Others
Plants
Sea animals
Unit 275
landscapes
landscapes
landscapes zebra
Unit 140
x
X
zebra
x
Task 1
Task
Concept
Class
Concept
Class
branch
donaal
booe
long, thin abdomen
Leaves
Chridmas
maple tree
maple free
artrroped
park
a tal, stardy trunk
Insect
sough bark
simple, untroken
relatively size
(a) Finetune-CBN) CBM
Task
Task 2
Concept
Class
Concept
Class
branch
bush
tre
not back and yellow stripe
not Bowers
not on oocan
yellww strpee
Christmaa tree
brown for
maple bre
green.
not fowers
not large, cup shaped blcom
leaves at the top
gracehu, drooping branchoc
found in garden
rough bark
unbroken
treee
(b) IN2
Figure 9: Final weight visualization for random classos in (a) Finetune-CBa CBM and (b) IN2 trained on CIFAR- 100 under 5-tasks scenario. The class distribution is in Table 38 but swap task 1 and task 2. For class 'maple tree" from task 1, we show its significant weight (> 0.16) after training on task and task Concepts generated from the maple tree class itself are colored blue, and other concepts from task are colored gray. We can see IN2 keeps a similar final layer while Finetune-CBM loses most significant weights.
30

---

Published in Transactions on Machine Learning Research (09/2024)
C Further Analysis and Details
C.1
Computational efficiency
Table 26 shows the maximum GPU usage of Finetune, EWC, GEM and our methods on CIFAR-10 under 5-lasks scenario. Even integrated with the interpretable tool CLIP-Dissecl (Oikarinen & Weng 2023) IG-CL's maximum GPU usage is still lower than EWC and GEM. Meanwhile, IN2's memory usage efficient even though the CBL layer increases when learning new tasks. The experiment results show our methods are efficient.
Table 26: Maximum GPU usage comparison for our methods, under CIFAR-10 5-tasks scenario. IG-CL's maximum GPU usage is smaller than EWC and GEM, which shows IG-CL's computational efficiency, IN2 is more efficient than existing methods.
Method
Finetune
EWC
GEM IG-CL-freezo-all/ IG-CL-freezo-part
IN2
Max GPU usage (GB)
5.51
5.65
5.68
5.61
4.00
C.2
Interpretability Guided Continual Learning Algorithm
The algorithm summarizes the procodure of Interpretability Guided Continual Learning.
Algorithm IG-CL: Freeze the subnetworks of the concept units
Require: Dataset D; regularization coefficient μi; connection threshold т; regularization factor X; Neural
network parameters 0
for 11,,..,,TT do
2:
if is 1 then
3: Train 0² on Dj by solving min (() + El=1 [Willa
Ø=
4: else
5: Train 6t on D, by solving Eq.
ConceptUnit CIIP-Dissect(W')
Prev-active ConceptUnit
8:
for layer 1 < L....,1 do for Unit + U,...UU do if Prev-active(u) is True then for Unit u111 +1,..., if [ Active[u-i]+ True
b Find the subnetwork of the concept units
10:
b uy is in subnetwork
11:
12:
then
weight exceeds threshold
13:
14:
if Using freeze-all then
15:
A Prev-activeluj] is True, Freeze W Active[u-ai] is Truc, Freeze W
16:
17:
else if Using freeze-part then Prev-activelu] is True, Freeze W
18:
19:
Prev-active Active
31

---

Published in Transactions on Machine Learning Research (09/2024)
C.3 Details of IG-CL and IN2
In this section, study the technical details of the IG-CL and IN2. ImageNel-10 is the 10-class subset of ImageNet (Russakovsky et al., 2015). For IG-CL, Figure 10 shows the subnetwork sizes in different datasets. Generally, a more complicated dataset has a bigger subnetwork, but the size adjustable by tuning hyperparameters.
10
0.8
ratio of freezing units
0.6
CIFAR-10
CIFAR-100
0.2
CUB200
ImageNet-10
0.0
TinylmageNet
50
52
54
56
58
60
layer deeper direction: -+)
Figure 10: The subnetwork sizes in the last residual block after learning two tasks in the 5-task scenario. A more complicated dataset will have bigger subnetworks.
For IN2, Table 2 and 30 show the number of concepts added per tasks. Overall, larger and more complicated datasets have more concepts. Around 10% of concepts are the same during the concept set expansion 4.2 Table 27 shows some example concepts that are related to different classes. Table 28 shows some complex concepts in IN2 from different. datasets.
Concept
Related Classes
ship
vessel, cargo ship apple, rose baby, fish wolf, fish, beetle bicycle, train, streetcar
Icaf small body
animal
vehicle
Table 27: Some concepts that generated from the different classes in IN2.
Dataset
Complex concepts
CIFAR-10
a flatbed for carrying cargo; four round, black tires; long hind legs for jumping tractor trailer; soft, round head; a crew of astronauts second hand digital timer; seat that is low to the ground; double-reed mouthpiece
CIFAR-100
TinyImageNet
Table 28: Some complex concepts that generated from the different datasets in IN2.
We build LN2 based on LF-CBM Oikarinen et al. (023 Therefore, we would like discuss two properties related to LF-CBM. First, following LF-CBM's procedure, we use CLIP calculate the activation matrix P
described in Section 2.2 We can replace CLIP with other vision language aligned (VL-aligned) models as long as they have text encoder and an image encoder to calculate matrix P. One future work will be replacing CLIP with other suitable VL-aligned models.
32

---

Published in Transactions on Machine Learning Research (09/2024)
Second, using GPT-3 to general concept sets is firstly proposed from an earlier work LF-CBM, which has many concurrent and follow-up works that use language models to create concept bottleneck layers (Yang) et al. 2023b; Yan et al. 2023). Introducing additional information is widely exploited when designing CBM Yuksekgonul et al., 2023 Zhou et al. 2018 Losch et al.) 2019). and is reasonable task-relovant information that we can leverage. Meanwhile, using additional information ion to enhance a model's performance common in other fields as well (Hu et al., 2021 Yang et al., 2023a). Even though LF-CBM has a potential leakage problem when using addtional text information, our method is still under the setting of continual learning. This is because during the training phase, the information for each class only appears in one task. One future work is to handle the potential leaking of CBM.
Table 29: Average concepts added per task in the 5-task scenario.
CIFAR-10
CIFAR-100
CUB200
ImageNet-10
TinylmageNet
Number of Concept 37.4
133.1
276.6
35.6
301.8
Table 30: Average concepts added per task in the 10-task scenario. TinylmageNer
CIFAR-100
Number of Coucept
154.6
284.3
D Others
D.1
Experiment Setup and Details
D.1.1
Training and computing details
All models are trained single NVIDIA V100s (32 GB SMX2). The hyperparameters used for our methods are in Table 31 We tune the hyperparameters for best performance in Ar. The hyperparameters tuning results are in Table 32
and 35 The hyperparameter selection of À in Eq. (3) is discussed in Appendix B.2 The buning process ensures that our methods are fairly compared to other baselines by optimizing their performance under the same conditions. When combine IG-CL or IN2 with GEM (Lopez-Paz & Ranzato 2017) or MIR (Aljundi et al.] 2019),, we use the memory buffer size: 150 samples per task. Mcanwhile, we experiment GEM and MIR with the same memory buffer size for fair comparisons. We split each dataset by 3 different random seeds, and run each class distribution for 3 times. The code and full training details will be released to public upon acceptance.
33
34
Table 31: Hyperparamotors for our mothods.
f in Eq.
10
X in Eq.
0.4
X in Eq.
3
0
in Eq.
3
0.4
T in IG-CL's Step 2
0.15
D.1.2 Dataset details and result report
CIFAR-10/ CIFAR-100 and TinyImageNet are standard image classification benchmarks. Both CIFAR-10 and CIFAR-100 have 50k training examples and 10000 testing examples with 10 classes and 100 classes respectively. TinyTmageNet has 200 classes with 500 training examples and 50 testing examples per class.
33

---

Published in Transactions on Machine Learning Research (09/2024)
Table 32: IG-CL's performance in different μ of Eq. under CIFAR-100 5-task scenario. The À of Eq. is set to 0.4 and connection threshold is set to 0.15 during the experiments.
Table 33: IG-CL's performance in different of Eq.
under CIFAR-100 5-task scenario.. The j of Eq-
is set to 10 and connection threshold 7 is set
to 0.15 during the experiments.
AT
AT
/ = 10-5 20.65
= 0.1
19.28
1e = 1 22.37
À 0.4
22.37
μ = 10- 20.19
À = 4
17.17
Table 34: IN2's performance in different of Eq. under CIFAR-100 5-task scenario. The À Eq. is set to and connection threshold T set to 0.15 during the experiments.
13
Table 35: IG-CL's performance in different connec- tion threshold T under CIFAR-100 5-task scenario.. The j and of Eq. ① is set to 10-6 and 0.4 spectively during the experiments.
[1
AT
AT
T = 0.125
19.24
7
0.1
22.91
T = 0.15
22.37
7 0.4
24.99
T = 0.2
17.61
γ = 4
18.26
In experiment results, we perform the independent t-test. We calculate the p-value with confidence interval 95% when comparing our best performance and the strongest baseline. The improvement is mark in blue if the p-value is less than 0.05.
D.2 Extended Previous Works Introduction
In the following sections, provide detailed introductions to previous continual learning methods, and the details of interpretability tools and models.
D.2.1 Previous Continual Learning Methods
The representative method regularization- methods category (i) Elastic Weight Consolidation (Kirkpatrick et al., 2017). Elastic Weight Consolidation (EWC) is regularized-based method. The loss function in this strategy L(0*)- = LCe(O") Σ, (0, 1 has quadratic penalty term which is related to the difference between the parameters of the old task and the new task. Meanwhile, this penalty term is proportional to the diagonal of the Fisher information matrix F. is a hyperparameter, and 0³ 0 stand for model parameters before and after training on a new task / respectively. Similarly, Zenke et al. (2017) adds a quadratic penalty term in the loss function, while estimating the importance of parameters during training. Li & Hoiem (2017) trains models with knowledge distillation loss for old tasks and a regularization loss to outperform joint training.
The classic method in architecture based methods category (ii) is Adam-NSCL (Wang et al., 2021). For each task, it learns model parameters in the null space of all previous tasks, which is based on Adam. The approximate null space is obtained by using singular value decomposition to the uncentered covariance matrix of input representations of all previous tasks. Yoon et. al. (018)) expands and prumes the neural network's architecture when learning new tasks.
The classic method in replay-based methods category (iii) Gradient Episodic Memory (Lopez-Paz Ranzato, 2017). Gradient Episodic Memory (GEM) is replay-based method. Tt. stores subset. of the
34

---

Published in Transactions on Machine Learning Research (09/2024)
observed examples from previous tasks, which is described episodic memory. When training new tasks, it regularizes the projection of the estimated gradient descent 9 the gradient descent of episodic memory 9k. The optimization goal is formalized as (g.9k) 2 0, vk < t when training on the task t. This regularization prevents the loss of episodic memory from increasing. (Rebuffi et al., 2017 stores a subset of samples for each class and uses the nearest-mean classifier on the data representation space.
D.2.2 Detailed introduction interpretability tools and models
CLIP-Disscct Oikaarinn & Weng 2023) use the Contrastive Language-Iman Pre-training (CLIP) model Radford et al. 2021) to decipher the neurons in larget model. CLIP is composed of an image encoder Er and a text encoder ET. Given a probing dataset Pprobe = { 24 E IR and a set concepts {c])= the first step is to generate the representation by the encoders. The results are I [x; R and C2 = Er()), C2 E RT Next, they compute concept-activation matrix P E RNXM where pa,= -1, C E R. Third, ncuron k's activation map is defined as (x;);) for input image T₁- The mcan of activation map is described as ((,)(z)))
and the activation vector is q = [g(AA (xi)) (())((()))) E IRN. Finally, the concept label of neuron k On where calculated from the Eq. (4)
argmax,, sigmmm,, qk: P)
(4)
where sim is similarity function.
Label-Free Concept Bottleneck Model (LF-CBM) (Oikarinen et al., 2023) transforms neural networks into interpretable CBM without labeled concept data. The procedure in LF-CBM is as follows: First, it uses GPT-3 (Brown et al. 2020) to generate set C' of text concepts important for the task based class labels, which is then filtered to improve quality. Second, LF-CBM learns CBL where each neuron corresponds to one of these concepts, by aligning the neurons with CLIP's (Radford et al. 2021) representation of the concepts. Given M concepts generated from the provious step, the CBL is a lincar transformation of the pretrained NN backbone f(xx), expressed as W. E do x (C). Here do the output dimension f(x). W. is learned maximize the similarity between CBL's output fc(x) (x) and CLIP's activation matrix P. This incentivizes the k-th neuron to have activation pattern similar to CLIP with the k-th concept. We denote k-th neuron's, activation pattern as 9 (x)) (ZN)] We is then optimized by minimizing P.-2.
the following objective: M
where q means the vector normalized to have mean and 11 1
standard derivation Finally, LF-CBM learns sparse linear prediction layer with weight. Wp and bias bp. The optimization goal is in Eq. 5
min WW WFFfoXXnain by -112 (WF)
(5)
W
where R. 0.5(1 W/Wls
W 11 a Wrlli
35

---

Published in Transactions on Machine Learning Research (09/2024)
D.3 Class Distribution
Table 36: Classes distribution of CIFAR-10 separaled by random seed 3456.
Task automobile, dog
Task deer, horse
Task bird, frog
Task ship, truck
Task airplane, cat
Table 37: Classes distribution of CIFAR-100 separated by random seed 3456. bear, bee, butterfly, camel, caterpillar, chair, clephant, forest, hamster, lion, motorcycle, otter, plates, sea, shark, shrew, spider, tank, train, willow beaver, bowls, boy, bridge, castle, cockroach, couch, dinosaur, house, keyboard, lawn mower, mushrooms, pears, pickup truck, poppies, possum, ray, skyscraper, wardrobe, whale
Task
Task 2
beetle, cloud, crocodile, lamp, leopard, lizard, palm, pine, porcupine, snail, streetcar, sweet peppers, table, telephone, television, tiger, tulips, turtle, woman, worm
Task 3
maple, baby, bed, bicycle, bottles, cans, chimpanzee, crab, lobster, man, mouse, oak, orchids, plain, road, rocket, roses, skunk, squirrel, trout apples, aquarium fish, bus, cattle, clock, cups, dolphin, flatfish,, fox, girl, kangaroo, mountain, oranges, rabbit, raccoon, seal, snake, sunflowers, tractor, wolf
Task 4
Task 5
Table 38: Classes distribution of CIFAR-100 separated by grouping similar classes together.
beaver, dolphin, otter, seal, whale, aquarium fish, flatfish, ray, shark, trout, bee, beetle, butterfly, caterpillar, cockroach, crab, lobster, snail, spider, worm
Task 1
maple, oak, palm, pine, willow, orchids, poppies, roses, sunflowers, tulips, apples, mushrooms, oranges, pears, sweet peppers, cloud, forest, mountain, plain,
Task 2
hamster, mouse, rabbit, shirew, squirrel, fox, porcupine, possum, raccoon, skunk, crocodile, dinosaur, lizard, snake, turtle, bear, leopard, lion, tiger, wolf
Task 3
bicycle, bus, motoreyele, pickup truck, train, lawn mower, rocket, streetcar, tank, tractor,
Task 4
bed, chair, couch, table, wardrobe, bridge, castle, house, road, skyscraper baby, boy, girl, man, womann, camel, cattle, chimpanzee, elephant, kangaroo, bottles, bowls, cans, cups, plates, clock, keyboard, lamp, telephone, television
Task 5
36

---

Published in Transactions on Machine Learning Research (09/2024)
Table 39: Classes distribution of TinyImageNet separated by random seed 3456.
Persian cat, cliff, plunger, German shepherd, teddy, American lobster, hourglass, seashore, dumbbell, ice cream, nail, convertible, orangutan, coral reef, go-kart, king penguin, sulphur butterfly, lesser panda, kimono, comic book, cockroach, projectile, lakeside, chimpanzee, bannister, bucket, gondola, koala, Iffeboat, teapot, police van, pill bottle, hog, crane, cash machine, mushroom, water tower, black stork, ice lolly, scorpion sewing machine, lemon, barn, Yorkshire terrier, stopwatch, lawn mower, thatch, pizza, barbershop, organ, computer keyboard, bighorn, cardigan, baboon, snail, syringe, spider web, Labrador retriever, pretzel, pomegranate, tarantula, pop bottle, trilobite, poncho, remote control, European fire salamander, altar, obelisk, binoculars, CD player, ladybug, miniskirt, cannon, wok, potter's wheel, cougar, chest, sunglasses, water jug, picket fence rugby ball, steel arch bridge, refrigerator, espresso, dining table, monarch, brown bear, confectionery, beach wagon, scoreboard, flagpole, potpie, brass, bow tie, brain coral, backpack, chain, bison, pole, beer bottle, grasshopper, tailed frog, lion, torch, abacus, magnetic compass, standard poodle, goose, bullet train, African elephant, gazelle, triumphal arch, iPod, beacon, jinrikisha, fly, dugong, suspension bridge, ox, wooden spoon
Task
Task
Task 3
Egyplian cat, volleyball, rocking chair, bullfrog, apron, swimming trunks, lountain, bikini, school bus, plate, guinva pig, oboe, maypole, goldfish, orange, drunstick, centipede, mashed potato, viaduct, military unlform, banana, sock, bathtub, guacamnle, walking stick, pay-phone, alp, lampshade, bell pepper, meat loaf, tabby, tractor, sombrero, gasmask, frying pan, spiny lobster, jellyfish, sandal, vestment, snorkel
Task 4
reel, basketball, parking meter, black widow, umbrella, trolleybus, Arabian camel, space heater, American alligator, albatross, sea cucumber, sea slug, cliff dwelling, boa constrictor, mantis, freight car, Chihuahua, fur coat, beaker, moving van, barrel, acorn, caullflower, birdhouse, academic gown, golden retriever, neck brace, candle, desk, bee, dam, punching bag, butcher shop, slug, dragonfly, limousine, sports car, turnstile, Christmas stocking, broom
Task
Table 40: Classes distribution of TinyImageNet separated by by grouping similar classes together. school bus, maypole, projectile, freight car, pay-phone, moving van, bullet train, birdhouse, tractor, triumphal arch, camnon, police van, crane, cash machine, jinrikisha, water tower, limousine, sports car, suspension bridge, turnstile, picket fence, refrigerator, barn, lawn mower, barbershop, beach wagon, scoreboard, flagpole, trolleybus, convertible, go-kart, viaduct, pole, bathtub, altar, obelisk, bannister, gondola, Ilfeboat, bucket
Task 1
reel, volleyball, rocking chair, balk plunger, parking meter, dining table, umbrella, oboe, hourglass, computer keyboard, space heater, backpack, pop bottle, beer bottle, remote control, lampshade, torch, abacus, barrel, CD player, teapot, candle, desk, frying pan, iPod, wok, potter's wheel, pill bottle, snorkel, sunglasses, water jug, broom, wooden spoon, rugby ball, sewing machine, stopwatch, plate, teddy, drumstick Egyptian cat, bullfrog, German shepherd, brown bear, guinea pig, Arabian camel, baboon, Labrador retriever, king penguin, lesser panda, chimpanzee, tabby, goose, koala, gazelle, golden retriever, hog, cougar, black stork, Persian cat, Yorkshire terrier, bighorn, orangutan, American alligator, bison, boa constrictor, Chilmahma, lion, standard poodle, African elephant, cliff, seashore, lakeside, alp, dam, steel arch bridge, fountain, cllff dwelling, magnetic compass monarch, snail, albatross, spider web, sulphur butterfly, tarantula, cockroach, mantis, European fire salamander, ladybug, bee, slug, dragonfly, fly, scorpion, black widow, centipede, grasshopper, tailed frog, goldfish, coral reef, sea cucumber, spiny lohster, jellyfish, American lobster, brain coral, sea slug, trilobite, lemon, banana, guacamole, mushroom,, thatch, orange, mashed potato, pomegranate, bell pepper, acorn, cauliflower, binoculars cardigan, sock, fur coat, academic gown, miniskirt, neck brace, sombrero, gasmask, punching bag, sandal, Christmas stocking, apron, swimming trunks, bikini, bow tie, military uniform, kimono, poncho, espresso, pizza, organ, potpie, ice cream, nail, pretzol, beacon, butcher shop, vestment, chest, dugong, lolly, confectionery, brass, comic book, meat loaf, dumbbell, syringe, chain, walking stick, heaker
Task 2
Task 3
Task 4
Task 5
37


Reviews:
[{"review_id": "Fq1uqivuhr", "reviewer": "Reviewer_8Gdw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* Improving the interpretability of continual training is important. The paper provides some interesting discussions by identifying concept units or reusing the concept sets constructed by CBM to preserve learned knowledge and increase interpretability in continual learning.\n* Technically correct. The author reported some gains on two image classification sets.\n* Overall the paper is written clearly. \n\n### Weaknesses:\n* The first method does not suggest globally stable strategies (freeze-part or freeze-all), as shown in Table 2.\n* The second method's assumption that adding all new concepts by treating instances from the same concept, albeit with different attributes, is the optimal approach is questionable. There's an opportunity to consider mappings to previously learned concepts when integrating new concepts. This would enable continual learning to better leverage learned knowledge and reduce redundancy in concepts.\n* The paper is more a proof-of-concept and lacks convincing evidence for the general effectiveness of the proposed changes. The limited number of experimented tasks hinders a comprehensive evaluation. The authors should consider experimenting with more tasks and conducting further discussions, such as ablation studies for the order of tasks in continual training. Moreover, it's unclear how the proposed methods would handle complex compound concepts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors aimed to incorporate interpretability into continual learning while mitigating catastrophic forgetting, two methods were proposed. The first method employs an external tool to decipher neuron activations, subsequently freezing them within a regularization-based framework. The second method adapts the concept bottleneck model (CBM), expanding existing concepts into new ones, and either fine-tuning all model weights or selectively freezing previous concept weights to learn those associated with the new concepts. The authors evaluated their approach on CIFAR-100 and Tiny Images for image classification tasks, reporting modest accuracy improvements over standard continual learning methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Enhanced Clarification of Proposed Methods:\n* To better clarify the relationship and differences between the proposed methods, the authors should provide more detailed explanations.\n* For instance, adding similar weight visualization for the first method would enable clearer comparisons between concept units and CBM concept sets.\n\n### Comprehensive Experiments for Effectiveness Validation:\n* While validating effectiveness with an image classification task may suffice for preliminary exploration, more comprehensive experiments are required to convince readers of the proposed concept-driven continual learning approach.\n* These experiments should encompass a wider range of tasks and include a deep analysis of important variables, such as the order of tasks presented in continual pre-training.", "broader_impact_concerns": "Controlling concepts in models might have some potential negative impact in privacy, which might be used in adversary attacks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OdMm5hflAO", "reviewer": "Reviewer_9mSG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nInterpretability is a property that should always be valued. This paper addresses the problem of continual learning, which is a very useful task, in an interpretabe manner. In my opinion both the proposed methods are needed in this paper: IG-CL is useful for models that were not trained initially for continual learning, while IN2 has this knowledge from the begining and it strategizes the continual learning process. Both consepts are useful and the paper shows good performance in both cases. \n\nWeaknesses\n\n1) Minor syntactical errors like \"fall into below main categories:\" in Introduction.\n\n2) The column \"Flexibility - For any dataset\" in table 1 seems obsolete given that every method is noted as \"yes\"\n\n3) Overstatement: \"IG-CL and IN2 represent significant advancements in continual learning\" page 2 last paragraph. For something to be a significant advancement it must be judged by the scientific community and to pass the test of time", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes two methods (namely IG-CL and IN2) to imbue interpretability to the continual learning task. IG-CL finds the concept units in the network and freezes them to preserve the initial knowledge, while IN2 uses a \"concept bottleneck model\" approach to inject the knowledge from each task to predetermined regions in the network. Both methods seem reasonable and they show strong numbers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please consider removing the overstatement", "broader_impact_concerns": "The paper addresses well enough potential privacy ethics violations", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oGDWNQ3wSk", "reviewer": "Reviewer_XJSC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is very important to consider interpretability when designing algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes two frameworks to solve  the challenge of catastrophic forgetting in continual learning.\nThese frameworks bring interpretability into continual learning, which can surpass existing methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In this paper, authors did not clarify or explain details of designing the proposed frameworks, which make this paper hard to understand. Besides, this paper is not self-contained, such as clip-dissect, just introducing a name without introducing any details. More crucially, experimental results are not convincing enough, since authors just conduct experiments on just one model.\n\n1. For better understanding, authors should explain how to decipher the neurons detailedly, and visualize these \"concept units\". Now, these detailed are missing.\n\n2. Why only decipher the neurons in the last layer? How about decipher neurons in other layers? I suggest authors clarify this selection. \n\n3. For the step 2 of IG-CL, I wonder when the model is trained for the task 3, whether concept units of both task 1 and task 2 are frozen?\nMoreover, I think authors should describe this step 2 in mathematical equations.\n\n4. What is the relation between the proposed IG-CL and IN2? Why not proposing a framework that can combine IG-CL and IN2?\n\n5. How to create a  task-relevant concept set? Details are missing. \n\n6. What is the definition of M^{t-1} and N^{t-1}? Can authors introduce their definitions?\n\n7. Authors should explain the reason why there exists a \"minus\" performance in Table 3.\n\n8 More experiments should be performed to verify the effectiveness of the proposed method, especially on more models.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper meets the standards of claims and evidence and audience for TMLR, and makes interesting connections between interpretability and continual learning, thus I recommend accepting the paper. There were a few remaining concerns about clarity and density of the paper. In preparing the camera-ready version, I would encourage the authors to further clarify/emphasize some of the details that they added in the revision; for example, promoting some of the methods details from Appendix D.2.2 to the main text—such as how the concepts are generated—would help the reader to understand the work without needing to flip back and forth between the main text and the appendix.

The reviewers also highlighted some lingering concerns, for example about the novelty and especially the limited scope of experiments (i.e., restricted to small image datasets with simple structures, that may lend itself to simple concept-driven methods like these); I concur and therefore I am not recommending the paper for further certification.
Decision: Accept (poster)