Does Continual Learning Meet Compositionality? New Benchmarks and An Evaluation Framework
Authors: Weiduo Liao
Ying Wei
Mingchen Jiang
Qingfu Zhang
Hisao Ishibuchi
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: continual learning
TL;DR: We evaluate continual learners' compositional generalization capability with proposed evaluation protocol and two benchmarks, i.e., CGQA and COBJ.

Abstract: Compositionality facilitates the comprehension of novel objects using acquired concepts and the maintenance of a knowledge pool. This is particularly crucial for continual learners to prevent catastrophic forgetting and enable compositionally forward transfer of knowledge. However, the existing state-of-the-art benchmarks inadequately evaluate the capability of compositional generalization, leaving an intriguing question unanswered. To comprehensively assess this capability, we introduce two vision benchmarks, namely Compositional GQA (CGQA) and Compositional OBJects365 (COBJ), along with a novel evaluation framework called Compositional Few-Shot Testing (CFST). These benchmarks evaluate the systematicity, productivity, and substitutivity aspects of compositional generalization. Experimental results on five baselines and two modularity-based methods demonstrate that current continual learning techniques do exhibit somewhat favorable compositionality in their learned feature extractors. Nonetheless, further efforts are required in developing modularity-based approaches to enhance compositional generalization. We anticipate that our proposed benchmarks and evaluation protocol will foster research on continual learning and compositionality.

Does Continual Learning Meet Compositionality? New Benchmarks and An Evaluation Framework
Weiduo Liao!?2 Ying Wei",, Mingehen Jiang', Qingfu Zhang", Hisao Ishibuchi"
'Department of Computer Science, City University of Hong Kong DDeparmeent of Computer Science and Engineering, Southern University of Science and Technology weiduliao2- comy. cityu. .k,, {yingwei, qingfu Langfu.zhang)0cce zhang) edu.hk, jiangmingchen0 wat.001199mmmmmmmm com, hisaodsustech. edu. cn
Abstract
Compositionality facilitates the comprehension of novel objects using acquired concepts and the maintenance of a knowledge pool. This particularly crucial for continual learners prevent catastrophic forgetting and enable compositionally forward transfer of knowledge. However, the existing state-of-the-art benchmarks inadequately evaluate the capability of compositional generalization, leaving an intriguing question unanswered. To comprehensively assess this capability, we introduce two vision benchmarks, namely Compositional GQA (CGQA) and Com- positional OBJects365 (COBJ), along with a novel evaluation framework called Compositional Few-Shot Testing (CFST). These benchmarks evaluate the system- aticity, productivity, and substitutivity aspects of compositional generalization. Experimental results five baselines and modularity-basee methods demon- strate that current continual learning techniques do exhibit somewhat favorable compositionality in their learned feature extractors. Nonetheless, further efforts are required in developing modularity-bane eralization. We anticipate that our proposed benchmarks and evaluation protocol will foster research on continual learning and compositionality.
approaches to enhance compositional gen-
Introduction
Human understanding of the world relies abstraction of concrete objects, allowing for the disentan- glement of high-level understandings into low-level concepts. This ability, known as compositionality, crucial aspect of human intelligence as it addresses the stability-plastiice
dilemıma [II) and introduces a computationally efficient learning paradigm 3 (7). By leveraging previously acquired concepts while acquiring new ones, individuals faced with a new task. For instance, accurate comprehension of varied bird wing shapes enables rapid identification of new bird species by focusing its wings. This compositional approach allows continual learners to achieve efficient knowledge reuse, benefiting not only in preventing catastrophic forgetling (37] of previous tasks (i.e., stability) but also in rapidly adapting to tasks (i.e., plasticity).
effectively build and maintain knowledge when
There have been various approaches to continual learning (CL) that be roughly categorized introduce regularization techniques to control the gradient update process for the current task. Secondly, replay- methods [6] [] store past tasks for later replay during training. Lastly, parameter-isolation based methods, specifically modularity-base based methods 28 36 41, employ static or dynamic model architectures and allocate distinct parameters to each task. Nonetheless, it remains largely unclear
into three directions. Firstly, regularization based methods 24
33 35]
"Corresponding authors.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

continual
(b) few-shot
Label
Srakk TANk
Helmet Swass
Serees thet
Boer Stirt.. ESrast Table
HaiL Helinet Bonich
Task 1
Task 7
Ome-hot
Fitgy
sub
New
Learned
Frozen
Label: Hidders
(ivvn
(c) CZSL
non
Concept Label
Compositional Label
One-hot Label
Figure Process for CFST to test compositional generalization capability and comparison with CZSL. Taking CGQA as an example, after continually training T tasks in (a), the feature extractor is frozen and we assign a new classifier specifically for each few-shot task. We use five types of few-shot tasks (b), which introduced detailedly in section [ to evaluate three aspects (i.e., systematicity, productivity, substitutivity) CFST aims challenge the compositional generalization ability of SOTA continual learners. Note that this protocol is suitable for class-IL. and task-IL. settings and we take the task-IL. setting for illustration. Different from the CZSL setting in (c), setting does not provide any concept-level supervision.
compositionality, with the references of non and noc.
whether these methods truly satisfy compositionality. The current works have two main drawbacks. Firstly, the evaluation mainly focuses on catastrophic forgetting and overlooks forward transfer [35] The most relevant work to study compositionality is [41]. which evaluates compositionality a toy bolored-MNIST dataset but only concentrates on color-shape composition. Secondly, the majority of existing CL benchmarks, such Split- Milit-CIFAROOO 10/100 and Split-MNIST, comprehensively evaluate compositionality. One potential alternative could be adapting benchmarks from the Compositional Zero-Shot Learning (CZSL) community for the CL setting. However, many of these benchmarks are either toy, synthetic, and lacking systematicity (e.g., CLEVR (21). or restricted simple object-attribute composition (e.g., UT-ZZaposs (56) and MIT-States 19
insufficient to
To address the aforementioned drawbacks and stimulate research on compositionality, we have devel- oped two benchmarks: synthetic yet systematic benchmark called CGQA, and one real-world benchmark called COBJ, both which involve vision classification tasks. Additionally, we have designed evaluation protocol called Compositional Few-Shot Testing (CFST), illustrated in Figure D comprising a continual training phase where a learner accumulates knowledge from a se- quence of tasks, followed by five few-shot testing phases. CFST aims evaluate three compositional capabilities: 1) Systematicity (sys) - the ability to correctly predict outcomes for few-shot tasks with novel recombinations of familiar concepts: 2) Productivity (pro) the ease of understanding combi- nations involving an increasing number of concepts; 3) Substitutivity (sub) - the speed at which the learner acquire knowledge about slightly different concepts from those learned previously. Two additional evaluation aspects are considered: 1) Non-novel (non) assessing the prediction accuracy for seen combinations of concepts: 2) non-compositional (noc) establishing lower bound of accuracy for few-shot tasks by evaluating the learner with unseen concepts. CFST challenges the compositional generalization capability of saute-of-the-ar continual learners, with primary focus on the feature extractor's i ability to capture compositional concept features after continual training. Drawing inspiration from the field of natural language [8]. we summarize the basic principles of compositional testing in vision. Firstly, the testing targets are novel recombinations of concepts found the training set, ensuring that the continual learner has the opportunity to learn meanings of these concepts (solvability). Secondly, while providing supervision for the testing task, it is crucial not overwhelm the learner with excessive information, as they are expected to "quickly" understand the testing targets show-shor), Finally, after acquiring sufficient knowledge during training, the feature

---

extractor is frozen to prevent further learning and overfitting during the testing phase (frozen feature extractor). Consequently, the continual learner must possess compositional generalization ability to bridge the systematic gap between training combinations and testing recombinations.
We assess the performance of five baseline methods and two parameter-isolatioo--asss (modularity- based) methods on our benchmarks using CFST. The results indicate that the continual learning (CL) paradigm does introduce a decline in compositional performance on the concept level compared to the multi-task learning paradigm. Additionally, the efficiency of learning compositionality is highly influenced by the number of classes in each continual task. Furthermore, the sate-of-the-arr CL methods employing ResNet-18 18 backbone demonstrate some degree of compositional generalization capability in terms of Systematicity and Productivity, but struggle with Substitutivity. Conversely, methods utilizzing the ViT backbone exhibit preference for Substitutivity. Remarkably, we observe that the reusable knowledge acquired by modularity-ban the underlying concepts. It is clear that there is still a significant journey ahead for continual learners to develop interpretable compositionality.
methods does not precisely correspond to
2 Related Works
Compositional benchmarks Compositionality is widely considered to be one of the most crucial elements in achieving human-level understanding of the world. While compositionality has been extensively studied in the field of natural language, particularly in the context of local and global understanding 77. there is need to evaluate the compositional capability of models. To address this, researchers have focused on creating datasets with specific split methods for training and testing purposes 7 46 18
23 22 57 10 [6]. However, the exploration of compositionallty in the field of computer vision is still in its early stages. While some proposed benchmarks prioritize the use of toy or synthetic images 4 41 58], there is lack of comprehensive evaluation on
26
datasets since most of them only test a limited aspect of compositionality (i.e., systematicityy [
15 14 60
Continual learning Continual learning typically addresses the issue of catastrophic forgetting through various approaches, which can be broadly categorized into four families. Regularization- based methods (35 33 24) incorporate regularization techniques into the gradient updates to mitigate 59 [2] involve retraining on past task data or storing memory-efficien features to retain knowledge from previous tasks. carameter-isolation based, particularly hierarchical modularity-based methods 43
forgetting of previous tasks. Replay-based methods 66 48
30 50 9. focus on modularizing learners to enable compositional learning of tasks (38). Building on the advancements of Vision
Transformers [ several prompt-based methods [33 52 5T aim to extract task-specific knowledge
from pre-trained backbone by using prompts. However, comparing these pre-trained methods to others may be somewhat unfair as they might have prior exposure to certain concepts or their combinations. Furthermore, only a limited number of studies (e.g.. [1 evaluate their compositional capabilities, especially in the field of computer vision. For instance,
50
explicitly
41 uses a toy dataset (color-MNIST) to evaluate simple color-shape compositions. [47 introduces vision-and-langee languug benchmark for continual concept learning tasks. In other works
132 49 [I
[5] 16, 25), including studies on compositionality outside the continual learning community, the datasets themselves possess compositional characteristics, but their evaluation may not rigorously test novel combinations.
3 Preliminaries
Vision continual classification tasks Continual learning focuses on sequence of I' vision classi- fication tasks, each involving an N-way classification. For each task t E T = [1 ..,,, there is corresponding training set D. {(1,, v1)). where 2; E X' represents the input image and y E y the corresponding class label. Here, N denotes the number of classes within each individual task. A continual leamer is a neural network comprised of two main components. Firstly, there is feature extractor ((x) : + L, which maps input images to an extracted feature space. Secondly, a multi-task classifier m)(((x),)) (): L x T y is present. This classifier takes the extracted features from the feature extractor, f(xx), as well as the task identifier, 4, as inputs and produces a prediction for the corresponding task. Here, y c represents the label space, which will be discussed further in
3

---

the subsequent paragraph. It is important to note that tasks are sequentially presented to the continual learner, thereby deviating from the assumption of independent and identically distributed (i.i.d.) data. Remark 3. I. This setting corresponds to the task-incremental (task-IL) scenario. Additionally, we also investigate the casseinnrremetal (class-IL) setting, where the task identifier t is not provided. Consequently, the multi-task classifier mff()), t) simplifies to m(f(x)).
Concept-based categories From human learning perspective, it is essential for representations to be compositional, meaning that a global understanding of an object is formed by combining all its semantic sub-parts [9]. For instance, people learn to recognize others by memorizing characteristics such as their hair color and mouth size. To assess whether the model is capable of learning compositional representations in images, we initially define the components, referred to as concepts, that we want the model to learn explicitly. Specifically, we create a pool of candidate concepts, denoted as C {ef).- where M represents the number of concepts. The vector J=11
cl:M = (c'...., ..)..,,.. E 10, indicates the presence of these concepts, where {0, 1)MM M denotes the M-fold Cartesian product (0, 1} x .... x (0.1) 1), and e² = indicates the presence of the i-th concept, while 0 denotes its absence. Further details regarding the continual training stage and evaluation stages will be discussed in section
Remark 3.2. Although we define multi-hot concept space for data generation, it is important to note that y remains single image-level label, and no concept label is provided to the candidate learner. This distinction is illustrated in Figure [ where categories such as (Door, Shirt) and (Grass, Shirt) are considered as two completely separate categories despite sharing the concept Shirt. As a result, the learner must deduce the underlying compositional concepts within an image. This represents the key difference between our label space definition and compositional zero-shot learning (CZSL), as depicted in Figure Dcc
4 Compositional Few-Shot Testing Protocol: CFST
We will now elucidate the precise methodology for assessing the compositionality of a learner through the utilization of the Compositional Few-Shot Testing (CFST) protocol, as illustrated in Figure T
Concept factorization We adopt attribute factorization process by using latent variable since it accurately captures the different concept distributions be- tween training and testing tasks. Denoting the joint distribution of the image x and the corresponding hidden concepts as p(c ,x), we can write
54 29 to model the data generation
2~ p(z), d' - p(d())) x - p(pl)) p(c .x) = =p(lliM) / pl=l2)p(2|c) daz
We split the label space Ver: I'm C Vir n. = 0 using different marginal distributions, p(c # Per (cl & Pno (c ). and keeping the same conditional generative model p(z]c = p(zlz)p(((()
( c. )dz for different phases, thus, artificially introducing compositional distribution shifts. Note that, .1: cliM are the corresponding concepts in Vtr. Y respectively. Without loss of generality, we bound the maximal number of activated concepts Ver. such that
Pi Pi-norm
|c! [| I Mr. The principles guiding the evaluation are summarized as follows:
1. Few-shot tasks: After continually training with T N-way tasks with Per (ct:A)) ), we expect this learner should have acquired sufficient knowledge in the concept pool C and be able to recognize novel concept combinations Yoo E Yan with Paw (ct:M) ) using only small number of support samples. Thus, we few-shor tasks, in which each concept combination Une has a small support set D.. Proper recognition of underlying compositional concepts crucial to avoid severe overfitting.
2. Solvability: All M concepts in C should be encountered at least once in the continual training tasks ( (Tr Additionally, the learner should have sufficient exposure to learn the concepts in various combinations. We take the label { [Grass, Shirt) with concept Grass in Figure aas an example, Grass should be seen in at least one training task, e.g., (Grass, Table). This assumption aligns with the general idea that standard gradient-based models cannot generalize to nonlinear functions without observing enough diverse examples (55)
Novelty: Few-shot tasks should only contain novel recombination of concepts not seen in the training set, i.c., Vne Ner 0. The presence of any seen label would negatively impact the

---

evaluation of the learner's compositional ability. In the above example, {Di) should not contain any Grasss Shirt) sample.
4. Frozen feature extractor: We recommend not further updating the feature extractor f using the support samples D₂ in the few-shot evaluation task. Only the classifier m is updated to assess whether f extracts the expected features related to the underlying concepts in the few-shot task. Empirically, a non-frozen feature extractor attempting to learn from a few-shot task leads to a noticeable reduction in performance, as shown in Appendix.
Based on the mentioned principles, we present five different few-shot testing schemes for the continual leamer:
Systematicity Novel Testing (sys) To examine the comprehension of learners on novel combi- nations of concepts in C, we design Tay N-way K-shot tasks. Each task includes N concept combinations Yaye randomly selected from Veys. with K samples for each Yays- Here, Veys C Vnv and the corresponding ||c2pe || V Mer. It should be noted that Vn. does not include any previously seen combinations in Vtr. ensuring the fulfillment of the novelty
recutreeuuuueeeee requirement.
Productivity Novel Testing (pro) The acquired knowledge should be able generalize effectively to complex images that contain a greater number of previously encountered concepts. After learning about (Door, Shirt), (Grass, Table), and (Hat, Leaves), the learner should have no trouble recog- nizing (Door, Leaves, Shirt, Table). To assess this ability, we construct few-shot tasks that involve more concepts per task compared to the sys testing. Specifically, consider another label space Ppro C Yan where the candidate label Uyro E Yrro satisfies the condition c1oppod ||| >> M
Substitutivity Novel Testing (sub) Concepts can possess specific attributes, such as the color of a Shirt being either red or green. Given knowledge acquired from images of red shirts (or shirts that not green) and understanding of Grass being green, learners expected to quickly recognize that a green shirt is still Shirt, as it combines the attribute of green color with the object of a shirt. In order to assess compositional understanding at the attribute level, we maintain the same distribution of concepts Pant((e (1:M) ) (cype() ). but utilize different conditional generative models Paub (x (2 ) 3f p(z]c!:M)) ). It is important to note that this attribute-based testing should be observed in relation to other concepts during continual training, allowing learners the opportunity to recognize it (solvability). This distinction is the primary difference between our sub approach and the
setting.
The aforementioned three novel few-shot schemes provide comprehensive evaluation of composi- tionality. In addition, we will introduce two reference schemes for the purpose of comparison and analysis.
Non-novel Testing (non) To assess the few-shot testing performance of learners on trained combi- nations, we create few-shot tasks where the label is taken from Ver rather than Veys- This particular setting intentionally violates the novelty assumption. The average test accuracy on these non tasks serves as complement to the average test accuracy on the tasks involving continually trained combi- nations. Additionally, comparing the results with other few-shot novel methods (such as sys) allows us to determine if the learner possesses the specific compositionality.
Non-compositional Testing (noc) We introduce noc tasks evaluate learners' few-shot testing performance on combinations that involve unseen concepts. In this case, the label Ynoe is selected from Vnoe. and the condition | 1| < Mar applies, with Cnoe Nc = 0. These tasks follow the normal continual learning scheme with T + 1 tasks, but the last task is few-shot task and the feature extractor remains frozen. This particular setting intentionally violates the solvability assumption. The average test accuracy on noc tasks serves as useful metric to indicate the lower limit of few-shot testing performance.
5 Proposed Benchmarks
The existing continual learning (CL.) community lacks comprehensive benchmark specifically designed to evaluate compositionality. In order to address this, we can adapt existing benchmarks for Compositional Zero-shot Learning (CZSL) for this purpose. However, directly transforming CZSL
5

---

benchmarks into CL versions, as done with Split- is not appropriate. This is because some CZSL benchmarks are considered to be toy benchmarks, such as CLEVR [2]] Furthermore, most CZSL benchmarks do not emphasize systematic compositionalityy as they primarily focus simple
object-attribute compositions, such as UT-Zaapps [56) and MITT-States [19]. Simply combining multiple benchmarks is also not suitable, as the concepts involved in different benchmarks might vary, resulting in lack cross-benchmark concept combination.
Our objective in developing benchmarks is to incorporate wide range of concept combinations alongside substantial number of instances for each concept. Thus, we first introduce a synthetic benchmark known Compositional GQA (CGQA), derived from GQA [[7] (License CC BY 4.0).
CGQA offers comprehensive combinations of concepts and substantial number of concept instances. Then, to evaluate continual learners using real-world images, we also propose real-world benchmark called Compositional Objects365 (COBJ), sourced from Objects365 to the limitations of space, please refer to the Appendix for image examples and additional benchmark statistics.
(License CC BY 4.0). Due
Label: Car, Shirt
Object: Car
Label
Crop
Rossiel
246 x 115
Resolution
Combine
Object: Shirt
Label
Restee
Bowl,
Pensn, shirt
Recolution
Resolution
(a) Original Image
(b) Selecting & Cropping
(c) Resizing & Combining
Figure Example of CGQA construction.
Label
Label
Streetlight
Finding Gosest Combleation
Crog
Person
Crappiid 36544
Streetligtt
Resolution: 1024 x768
Resolution:
357 235
Resolution: 224x224
(a) Original Image
(b) Selecting & Cropping
(c) Resizing
Figure 3: Example of COBJ construction.
Construction CGQA Since the available number of annotations in GQA is inadequate to construct the compositional benchmark using the original images, we have chosen to artificially combine suitable objects and create new images for our purposes. The process illustrated in Figure Initially, crop the image using the bounding box of the selected object and then resize the cropped region to a resolution of 98 x 98 pixels. Next, we randomly position the processed object image within a x 2 grid. Any unoccupied grid spaces are filled with black pixels to ensure complete composition.
Construction of COBJ The images in Object365 are of high resolution and contain multiple instances of classes, annotated with bounding boxes (as shown in Figure 3.). To conveniently utilize these images, we need to resize them. However, resizing high-resolution images directly can lead to loss object resolution, which may have detrimental effect on the model's performance. Therefore. we employ strategy where we select a specific bounding box within each object set to crop and generate a new image. The process explained in Figure Subsequently, we perform the resizing operation, allowing the object to occupy a larger proportion within the image and providing more pixels for analysis. The selection of bounding boxes follows the principle minimizing the total distance between their respective center points. This process ensures that the closest bounding boxes are chosen. By using the cropped region shown in yellow Figure 3 we reduce the proportion of irrelevant parts and maximize the preservation of the original object's information.

---

Table I: Results on CGQA. Accuracy (%) 955 confidence intervals (%) over 300 tasks are reported for few-shot testing phases. Task-IL settings denote with a postfix "*"
Methods
Aroa
Auga
Aprn
Aaub
H,
Ano
Aque
H
Нa
Muluiaakk
83.61
88.1 = 0.62
85.94 + 0.65
69.67 + 0.75
80.35
91.55 + 0.51
40.04 +: 0.99
55.71
68.28
Finetune
8.38
64.73 = 0.78
65.43 d: 0.73
61.26 +: 0.67
63.75
68.54 + 0.80
40.32 4: 0.72
50.77
57.84
ER
19.78
71.38 + 0.75
70.11 + 0.64
64.32 + 0.69
68.46
77.27 + 0.67
40.96 + 0.72
53.56
61.60
GEM
8,56
66.56 = 0.77
73.47 + 0.61
62,80 + 0.71
67.33
72.65 + 0.73
41.64 + 0.72
52.94
60.72
LwF
9.11
71.22 = 0.74
68.74 0.66
71.03
76.56 -: 0.66
48.69 1: 0.77
59.52
65,93
KWC
8.22
64.99
0.78
73.47 + 0.64
63.25 + 0.66
66.95
69.03 + 0.74
41.38 + 0.71
51.75
59.91
RPSnet
33.45
59,80
+ 0.83
60.26 + 0.72
59.75 + 0.74
59.94
64.22 +: 0.81
45,09
0.64
52,98
56.95
Mult'lask*
92.17
82.16 + 0.68
84.22 + 0.60
71.82 + 0.75
79.01
86.44 + 0.69
44.07 + 0.95
58,38
69.23
Finetune*
72.46
70.32 = 0.73
72.62 + 0.63
66,33 + 0.69
69.66
75.32 + 0.70
43.26 + 0.73
54.95
62,92
ER
76.05
71.37
0.70
72.67 : 0.69
66,80 + 0.63
70.19
76.28 - 0.66
45.61
+ 0.77
57.09
64.29
GEM
21.60
69,44
0.75
73.14 0.67
66.61 0.66
69.63
73.31 0.71
45.97 :: 0.74
56.51
63.71
LWFS
73.19
69.61 + 0.77
74.22 + 0.63
65.00
+ 0.65
69.40
75.25 + 0.69
42.52 + 0.72
54.34
62.48
EWC
71.10
69.40 2 0.79
72.37
+: 0.65
65.10
68.83
74.99 0.72
42.82
+ 0.77
54.51
62.29
MNTDP
68.98
47.27
0.91
47.41
0.86
47.50 0.85
47.39
52.28
0.1
32.29
1 0.83
39.93
44.09
6 Experimental Studies
In this section, we conducted experiments using SOTA continual learning algorithms on our proposed compositional benchmarks, namely CGQA and COBJ. The purpose of these comprehensive computa- tional experiments is to address the following research questions: (1) Can our proposed benchmarks accurately evaluate the learners' ability to extract compositional features from a sequence of tasks? (2) Can five different few-shot testing schemes provide insightful perspectives on the learners' com- positionality? (3) Do modularity- -baeed approaches demonstrate superior few-shot testing accuracy, indicating their efficacy in extracting compositional features?
6.1 Experiment Settings
Compared Algorithms We evaluate five baseline algorithms, including: Finetune, which trains a continual learner sequentially for all tasks; regularization-bass methods: GEM (35), LwF (33 and
EWC 24] replay-based method: ER [6]. Additionally, we also evaluate two SOTA hierarchical compositional modularity approaches: MNTDP (50) (MNTDP-D), which dynamically incorporates new modules or reuses existing frozen modules to improve performance on the current task; RP- Snet [43), which randomly selects modules in each layer and employs regularization and experience replay for assistance. When evaluating these methods in the class-incremental learning (class-IL) setting, where the task identifier is unknown, we use a single-head classifier. In the task iiskiincrementl learning (task-IL) setting, where the task identifier is known, we employ multi-hean head classifier (denoted by postfix "*"). All the algorithms mentioned above are evaluated using a ResNet- 18 backhone [13]. For detailed hyper-parameter settings and additional results using VIT backbone [ please refer to the Appendix.
Evaluation Metric During the continual training phase, we report the Average test accuracy Acon on all seen tasks after completing T = 10 tasks. For the five different few-shot testing phases, namely sys, pro, sub, non, and noc, we report the Average test accuracy for each test (i.e.. Asys: Apro: Arub, Anons and Anoc) on Tue 300 tasks. It is worth noting that the Asub metric not applicable for COBJ due the absence of attribute information in the source benchmark Objects365. We denote the prediction accuracy of the multi-task classifier m as A (fm((()), t), v). The corresponding testing dataset for task denoted as Df=. The Average test accuracy is calculated as T 1 Ein (( D/e (AaiiiDDe(AAMMiiiA (mm({())) u)]- Additionally, we employ the Harmonic mean H, = 3/(1/Agg + 1/AApro pro + 1/Agub) among sys, pro, and sub, H, = 2/(1/Anon + +//Ano)) between non and noc, and H. 5/(1/Aays + 1/ 1/Apro + 1/Asub + 1/Anon + 1/Anae) to show the compositionality. reference capability, and overall few-shot testing capability, respectively. The Harmonic mean places emphasis on achieving balanced performance across all few-shot cases simultaneously, measuring the model's ability perform well across different scenarios.
7

---

Table 2: Results on COBJ. Accuracy (%) 95% confidence intervals (%) over 300 tasks are reported for few-shot testing phases. Task-IL settings denote with a postlix "*"
Methods
Acon
Aaya
Apro
Hn
Anon
Anoe
H
H
MultiTask
39.73
72.70 + 1.55
67.11
1.86
69.79
83.38
1.36
57.52
1.81
68.08
68.93
Finetune
6.37
43.06 + 1.28
41.89
+ 1.29
42,46
46.96 + 1,51
46.12 - 1.36
46.54
44,41
ER
7.67
46.51 + 1.48
45.22
+ 1.38
45.86
57.58 + 1.85
42.81 = 1.32
49.11
47.43
GEM
5.57
46.11 10.11+10 1.41
43.99
+ 1.33
45.03
49.51 + 1.57
46.07 + 1.47
47.73
46.34
LwF
6.73
55.10 1.54
+
52.49
59.37 + 1.61
51.40 1.55
55.10
53.76
EWC
6.53
42.73
1.19
41.00
+ 1.32
41.85
46,69
1.48
41.21
1.34
43.78
42.79
RPSnet
21.93
73.26
1.24
67.89
+ 1.44
70.47
1.27 78.82
68.22
1.24
73.14
71.78
MuliiTas**
75.73
55.72 + 1.75
49.84 + 1.81
52.62
59.21 + 1.89
53.43 + 1.74
56.17
54.34
Finetune*
44.37
45.67 1.39
44.18 + 1.38
44.91
47.92 1.54
47.80
1.57
47.86
46.34
ER*
64.80
54.40 1.73
51.33
1.69
52.82
58.91
1.73
48.83
1.54
53.40
53.11
GEM
36.53
45.57
1.39
45.50 + 1.39
45.53
51.93
1.64
46.40 1.36
49.01
47.21
LwFs
61.03
53.77 = 1.63
50.06 + 1.76
51.84
60.54
1.58
54.53
1.61
57.38
54.47
BWC
53.87
45.63
1.46
46.12
1.47
45.88
51.16 1.59
48.86
1.48
49.98
47.84
*
71.80
51.89 + 1.41
49.16 + 1.56
50.49
57.20
1.59
1.36 48.52
52.50
51.48
6.2 Results
We report the performance results of our CGQA benchmark in Table and the results of the COBJ benchmark Table For detailed analysis of the COBJ benchmark results under different settings of T' please refer to the Appendix.
Multi-task baselines outperform CL methods in both continual tasks and compositional gener- alization performance In both class IL and task-IL settings, MultiTask(*) demonstrates the highest performance on Acon and H,, which is as expected since this method benefits from i.i.d. training task sequences. To visualize the learned concept region, employ the CAM [1]. The results in the Appendix for both CGQA and COBJ demonstrate that the proposed H, accurately evaluates learners' performance in extracting compositional features. Notably, CL methods generally do not outperform MultiTask(*)) on Ha revealing that forgetting also contributes to the degradation of compositionality. This observation highlights the urgent question of how to equip learners with compositionality for the continual learning community.
Continual tasks prefer modularity-based methods Among the baselines, RPSnet achieves a significantly larger margin on Acons particularly on COBJ, indicating that the reused knowledge the modules can effectively mitigate forgetting. While MNTDP performs poorly on CGQA, outperforms the other methods on the real-world benchmark COBJ. This difference can be attributed to the fact that forgetting is not as significant on CGQA as is on COBJ, especially in the task-IL setting. Therefore, simply freezing modules does not provide efficient reusable (compositional) knowledge for the novel recombination of seen concepts. The second- best method on Acon is ER(*), particularly in the class-IL setting, demonstrating the effectiveness of the memory buffer.
Modularity- -base methods show no superiority on compositionality In terms of H.. another winning method alongside ER* is LwF, although its performance Aron is not outstanding. This suggests that the feature extractor learned by LwF excellent at extracting compositional features, but the classifier fails to make correct predictions. The phenomenon of prediction bias is further discussed in the Appendix, with CAM visualization. Regularization-base mechanisms like GEM, LwF, and EWC struggle to solve this classifier issue. As result, only replay-based methods (such ER and RPSnet) show improved performance. Unfortunately, RPSnet and MNTDP* do not demonstrate superiority on CGQA, indicating that the reused knowledge is not sufficiently compositional for handling novel combinations of seen concepts. Freezing the modules for old tasks may not be appropriate for new tasks unless the modules have enough compositionality. Further efforts are required to enhance modularity-basee on COBJ, these two methods show competitive performance compared to others, with RPSnet performing the best. This indicates their superiority in dealing with real-world tasks.
methods to bring benefits to compositionality. Interestingly,

---

Five few-shot tests exhibit a consistent order: Anon > Apro > Asys > Asab > Anoe Generally, the performance on non is the best, which is expected because models tend to favor the seen combinations and may struggle with compositional gaps to some extent. The performance gap between sys and pro acceptable, but it is larger for sub. This is because sub requires the composition of attributes of concepts to form unseen recombination, demanding higher-level compositional ability compared to sys and pro. Interestingly, when models have a ViT backbone (in the Appendix), we observe contrary results: Asub > Ayro Aays- It can be attributed to the fact that the selected attributes for concepts in sub primarily pertain to texture attributes (e.g., color). This observation aligns with the analysis in [2] that the multi-head attention layer in ViT has a bias towards shape attributes, while the convolution layer has a bias towards texture attributes. Therefore, models with 18 backbone are more sensitive to the texture difference between continual and sub. Regarding accuracy on pro, it is generally slightly better than on sys, except for MultiTask and ER, indicating that having more concepts can somewhat aid in understanding the images. The overall lower accuracy on the single-head baselines (class-II. setting) compared to their multi-head counterparts (task-IL setting), except for LwF, demonstrates that the single-head classifier makes it difficult for the feature extractor to retain compositional knowledge.
Concept-level forgetting We conducted a case study to investigate the phenomenon of forget- ting with regard to concepts. Following continual training phase, evaluated the learned models on four few-shot task sets: non(f), which consists of seen combinations of freshly learned concepts in the last task; non(o), which consists of seen combinations of old con- cepts from previous tasks; sys(f), which consists of novel combinations of freshly learned con- cepts in the last task; and sys(o), which consists of novel combinations of old concepts from previous tasks. If the model experiences forgetting on the concept level, we would expect a small gap between non(f) and sys(f) and a large gap between non(o) and sys(o). The con- struction of these two task sets is detailed in the Appendix. We also report the related scores S(o)= = (Asgs(o) - Anonto)) //Ano(()) and S(f) = (Arys(f) ())on())/Anonn to clearly show the performance gaps. The average test accuracy and the related score are shown in Figure 4 Our observations reveal that Anon(s)
Anon(o) and f > Asys(o), which is as expected since fresh concepts are newly
Test Accuracy
Score
98.27
96. 35
97.87
95.67
FT
87.10
70.25
87.63
67.65
95
0.61
-3.70
updated in the last continual task. Addi-
ER
67.92
78.77
87.52
75.60
00.5
4.00
tionally, S()) > S(o), indicating that the
GEM
65.62
71.52
84.15
67.70
90
477
-5.34
-2
LwF
79.05
86.22
74.45
0.92
.5.82
learner has better systematicity composi- tional generalization capacity for fresh con- cepts than for old concepts. These obser- vations demonstrate that the learner indeed learns concepts in composition-wis man- ner and forgetting occurs primarily with old concepts. On the other hand, Multi- Task, which is regarded as the upper bound, experiences the smallest performance drop on both fresh and old tasks, confirming our expectation that the learner jointly learns all continual tasks. This observation also suggests that the continual learner experiences forgetting not only on old tasks but also on old concepts. To visualize the learned concepts, we used CAM; please refer to the Appendix for more details
EWC
86.68
71.70
85.97
66.40
85
.083
-7.39
MT*
98.02
95.92
97.63
94.57
80
0.39
FT+ 87.15
77.23
86.38
73.30
-0.88
-5.09
ER*
87.23
80.73
86.
74.77
.0.61
-7.39
GEM
88.47
75.92
87.65
70.75
-0.92
-6.8
LinF
37.40
77.28
85.
73.63
70
EWC
68.42
77.90
86. 8.
72,80
-1.73
65
non(f)
non(a)
sys(h
sys(o)
S(()
S(o)
Figure 4: Fresh tasks vs old tasks on CGQA. MT stands for MultiTask and FT stands for Finetune. Take away: fresh concepts suffer little performance drop when test- ing on novel combinations (sys(f)) while old concepts (sys(o) on the contrary.
Sample effiiciecyy for learning compositionality To investigate the number of training samples required for continual learner to acquire the ability of compositional generalization, we vary the number of training samples for each class in the CGQA continual training tasks. It is intuitive that a small number of training samples not only hinders novel combinations (i.e., sys, pro. sub), but also affects the seen combinations (i.e., non). Examples of sys and non are depicted in the left and middle figures of Figure 5], respectively. The complete results can be found in the Appendix. Moveover, we utilize the related score, denoted as S(sys). It is calculated as (Aays larger positive S(sys) indicates better compositional generalization ability on Systematicity, while smaller negative value indicates poor handling of novel combinations of seen concepts by the learner. The results are presented in the right figure of Figure 1. For almost all methods, S(sys) reaches
Anon)/Anons where a
9

---

0.025
finetune
0.000
Finetune
60
ER
50
S(sys)
-0.025
Anon
GEM
40
-0.050
GEM
30
LwF
LuF*
20
20
-0.100
FWC
10
0
250
500
750
1000
0
250 500
750
1000
1000
Number of Sample
Number of Sample
Number Sample
Figure 5: Varying number of training samples CGQA. Take away: Accuracy is improved with more training samples for both sys and non. Related scores become smooth and stable more than 300 training samples for each class. This fact shows that 300 samples needed for learning compositionality.
convergence when trained with more than 300 samples for each class. Beyond this threshold, the ratio of improvement for sys to non remains unchanged. Consequently, we can conclude that learners need 300 samples for each class learn compositionality in CGQA.
MuhiTakk
Finetune
ER
GEM
LwF
EWC
RPSNSt/MNNTP"
Class-l
Task-IL
40
30
20
20
10
10
Tasks
Tasks
Figure 6: Number of continual tasks vs Hn 10-way COBJ testing tasks. Take away: A larger number of continual tasks (smaller ways for each task) hinders compositionality.
Varying number of continual training tasks As the number of training tasks increases, the number of classes in each task decreases while the total number of classes remains constant. Consequently, compared single task that includes all the classes, the learner requires a smaller number of compositional features to distinguish classes in each individual task. However, these features may be insufficient for future compositional tasks. To evaluate the learners, five (6-way), and ten (3-way) COBJ tasks respectively. and then assess their performance on 300 10-way few-shot sys and pro tasks. We visualize the results using Ha, as shown in Figure 6 The observation aligns with the aforementioned understanding. For more detailed information, please refer the Appendix.
train them three (10-way),
7 Conclusion
In this paper, we present two benchmarks, CGQA and COBJ, which offer ample samples for CL. Additionally, we introduce CFST, novel evaluation protocol that enables comprehensive assessment of compositionality in three key aspects: systematicity, productiviry, and substitutiviry. The construction of our benchmarks is based on publicly available sources, ensuring the absence of sensitive data and ethical suitability for research purposes. Experimental results demonstrate that our protocol provides valuable insights for SOTA CL methods. However, modularity- approaches do not exhibit superiority when subjected to compositional testing, highlighting the need for significant advancements in this area. We believe that our benchmarks and evaluation protocol will inspire researchers to prioritize compositionality during the development of continual learners.
Limitation and future work While our work specifically focuses on local compositionality, future studies will explore the global scopes of compositionality. extensively examined in other fields such as natural language processing. Although our candidate compositions of concepts are limited, they sufficiently serve the purpose of testing compositionality in continual learners. Moving forward, the combination of multiple benchmarks will be considered form more comprehensive evaluation framework.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation of China (Grant No. 62250710163, 62250710660 10682), Guangdong Provincial Key Laboratory (Grant No. (020B121201001) the Program for Guangdong Introducing Innovative and Enterpreneurial Teams (Grant No. 20172107X386), The Stable Support Plan Program of Shenzhen Natural Science Fund (Grant No. 20200925174447003) Shenzhen Science and Technology Program (Grant No. KQTD20161 12124335112)14
References
[1] Amit Alfassy, Leonid Karlinsky, Amit Aides, Joseph Shtok, Sivan Harary, Rogerio Feris, Raja Giryes, and Alex M Bronstein. Laso: Label-set operations networks for multi-label few-shot learning. In Pruceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 6548-6557, 2019.
[2] Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: Scan both left and right. arXiv preprint
009.04640, 2018.
[3] Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey. arXiv preprint arXiv:2012.09822 222222 2020.
[4] Gail A. Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37(1):54-115, 1987.
[5] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-GEM. International Conference on Learning Representations, 2019.
[6] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 902. 10486, 2019.
[7] Verna Dankers, Elia Bruni, and Dicuwke Hupkes. The paradox of the compositionality of natural language: a neural machine translation case study. arXiv preprint arXiv:2 (0l::200..0500 108.05885, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[9] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: /to::/701.000 008334. 2017.
[10] Catherine Finegan- Dollak,, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. Improving text-to-SQL. evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 351-360, Melbourne, Australia, July 2018. Association for Computational Linguisties.
[11] Stephen T Grossberg. development, cognition, and motor control, volume 70. Springer Science & Business Media, 2012.
Studies of mind and brain: Neural principles of learning, perception,
[12] Tyler I. Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Re- mind your neural network prevent catastrophic forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
11

---

[14] Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, 110:107383, 2021.
[15] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human- object interaction detection. European Conference on Computer Vision, pages 584-600. Springer, 2020.
[16] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[17] Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[18] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
[19] Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations image collections. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 1383-1391, 2015.
[20] Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference Machine Learning, pages 4804 PMLR, 2020.
[21] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017.
[22] Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashu- bin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method realistic data. In International Conference Learning Representa- lions, 2020.
[23] Najoung Kim and Tal Linzen. COGS: compositional generalization challenge based on semantic interpretation. In Proceedingy of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105. Association for Computational Linguistics, November 2020.
[24] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bart Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-335 2017.
[25] Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. Advances in neural information processing systems, 31, 2018.
[26] Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to pruuence-to-sqquenee recurrent networks. In International conference on machine learning, pages 2873-2883 PMLR, 2018.
[27] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 35006266) 2015.
[28] Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for fask-free continual learning. International Conference on Learning Representations, 2020.
12

---

[29] Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang. Jiawei Ren, Tong Che, Jun Zhang. and Ziwei Liu. Sparse mixture-of-exxie are domain generalizable learners. In The Eleventh International Conference on Learning Representations, 2023.
[30] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pages
PMLR, 2019.
[31] Zeqian Li, Michael Mozer, and Jacob Whitehill. Compositional embeddings for multi-label one-shot learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 296-304, January 2021.
[32] Zeqian Li and Jacob Whitehill. Compositional embedding models for speaker identification and diarization with simultaneous speech from 2+ speakers. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7163- 7167. IEEE, 2021.
[33] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transuctions pullern analysis and machine intelligence, 40(12):2935-2947, 2017.
[34] Weixin Liang and James Zou. MetaShift: A dataset of datasets for evaluating contextual distri- bution shifts and training conflicts. In International Conference on Learning Representations, 2022.
[35] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[36] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruming. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
[37] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[38] Jorge A Mendez and Eric Eaton. How reuse and compose knowledge for a lifetime of tasks: A survey on continual learning and functional composition. arXiv preprint arXiv:2207.077300 2022.
[39] Richard Montague and Richmond H Thomason. Formal philosophy. selected papers of richard montague. Erkenninis, 9(2), 1975.
[40] Tushar Nagarajan and Kristen Grauman. Attributes as operators: Factorizing unseen attribute- object compositions. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
[41] Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298 30312, 2021.
[42] Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations, 2022.
[43] Jathushan Rajasegaran, Munawar Hayat, Salman H Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[44] Lukas Schott, Julius Von Kügelgen, Frederik Träuble, Peter Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. In International Conference on Learning Representations, 2022.
[45] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pages 8430-8439, 2019.
13

---

[46] Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? arXiv preprini arXiv:2010. 12725, 2020.
[47] James Seale Smith, Paola Cascante-Bonin Assaf Arbelle, Donghyun Kim, Rameswar Panda, David Cox, Diyi Yang, Zsolt Kira, Rogerio Feris, and Leonid Karlinsky. Construct-vl: Data-free continual structured vl concepts learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14994-15004, 2023.
[48] Morgan B Talbot, Rushikesh Zawar, Rohil Badkundri. Mengmi Zhang, and Gabriel Kreiman. Lifelong compositional feature replays beat image replays in stream learning. arXiv preprint arXiv:2104.02206 2021.
[49] Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprint
2019.
[50] Tom Veniat, Ludovic Denoyer, and Mare' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint urXiv:2012.12633 2020.
[51] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's s razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695. 2022.
[52] Zifeng Wang, Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal continual learning. In European Conference Computer Vision, pages 631-648. Springer, 2022.
[53] Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[54] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishna- murthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2022.
[55] Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolei Du, Ken-Ichi Kawarabayashi, and Stefanic Jegelka. How neural networks extrapolate: From feedforward graph neural networks. In International Conference on Learning Representations, 2021.
[56] Aron Yu and Kristen Grauman. Fine- Fine-grained visual comparisons with local learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 192- 199, 2014.
[57] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale numan--lbblled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[58] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. Advances in Neural Information Processing Systems, 35:29776- 2022.
[59] Mengmi Zhang, Tao Wang, Joo Hwee Lim, Gabriel Kreiman, and Jiashi Feng. Variational prototype replays for continual learning. arXiv preprint arXiv:1905.09444 2019.
[60] Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Nico++: Towards better benchmarking for domain generalization. arXiv preprint arXiv:2204.08044) 2022.
14

---

[61] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
15


Reviews:
[{"review_id": "K8pTLY6DQB", "reviewer": "Reviewer_xdij", "summary": "", "questions": "", "limitations": "The authors discussed the limitation of the work from the technology point of view in Section 7. Nonetheless, I don't find an explicit discussion about the negative societal impact. The risk of misusing the dataset may be discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The CGQA and COBJ benchmarks are designed to test the learner's ability to handle novel combinations of seen concepts, providing a comprehensive evaluation of the compositional generalization capability of continual learning methods.\n\nThe paper evaluates three compositional capabilities: systematicity, productivity, and substitutivity.\n\nThe benchmarks do not contain any sensitive data, such as medical data, which makes them ethically sound for use in research.\n\nThe baseline methods are all public and widely used, which means they are accessible to researchers and can be used to compare the performance of different methods.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "klIaFzBUqb", "reviewer": "Reviewer_RpQ2", "summary": "", "questions": "", "limitations": "The main limitation is that all concepts are pertaining to visual entities, but in reality, there are concepts about things like motion, human mood, or anything that is not directly describable. I believe those will offer a quite different challenge for generalization.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is well-written and easy to follow. The proposed benchmark is reasonable and what it tests, to my knowledge, cannot be found in the existing literature.\n\n2. The way the authors test what the agent learns during the CL phase is interesting, i.e., by testing its downstream performance without finetuning.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "P8SqMEMFxv", "reviewer": "Reviewer_9Qe4", "summary": "", "questions": "", "limitations": "The discussion of limitations is quite short and superficial.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed benchmark is well-designed and addresses an interesting problem combining compositional learning and continual learning. There is a notable lack of such benchmarks in the literature.\n2. The experimental evaluation is thorough and well-explained. The chosen metrics cover a spectrum of aspects of interest: \"standard\" continual learning performance (on tasks seen during training), three \"compositional\" abilities, and two \"reference\" or non-compositional abilities. \n3. The details of the benchmark creation process are covered sufficiently in Appendix C.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BIABUamELW", "reviewer": "Reviewer_fvLh", "summary": "", "questions": "", "limitations": "1. The main motivation of the paper, as indicated in the Introduction, is \"compositionality addresses the stability-plasticity dilemma\" (Line 17-18). However, the experiments to support this claim are insufficient. First, from Table 1, LwF has the best compositional ability, but low continual learning accuracy. It contradicts with the aforementioned motivation of this paper. From the analysis, I speculate that compositionality and continual learning ability are the two independent aspects of machine learning. Then, why do need to measure the compositionality if it doesn't improve the continual learning ability of the model?\n2. To verify the claim in the Abstract \"compositionality helps continual learning\", the paper should also conduct a truly continual learning setting, i.e., none of the previous concepts are present in the next task. Current continual learning setting randomly groups a pairs of \"concepts\", which results in overlapping concepts between tasks, i.e., leading to knowledge leaking. However, the standard continual learning setting only evaluates on novel classes, where no overlapping and knowledge leaking are present\n3. The evaluations of 3 compositional capabilities lack motivations. Why are we interested in model's systematicity, productivity and substitutivity?\n4. The adoption of few-shot learning and freezing feature extractor in evaluating compositionality also lacks motivations. How does few-shot learning (Principle 2 in the introduction) and frozen feature extractor (Principle 3) support compositional testing.\n5. The literature review misses several recent works. For example, NeurIPS2022 [1] and CVPR2023 [4] works also proposes a benchmark for testing mode's compositionality. Additionally, the benchmark should include more recent continual learning works [2,3] to give a more meaningful conclusion about the current model's compositionality. Especially, a recent method [4] also explores compositionality in continual learning, which should be included in the benchmark comparisons and the literature review.\n6. Some experimental findings are unclear. Why is forgetting is not as suffered as that in the class-IL setting on CGQA (Line 244-245)?\n7. The explanation of high $A_{sub}$ on ViT and low $A_{sub}$ on conv-based methods is unclear. Conv-based methods are more sensitive to texture, and selected attributes for `sub` are mainly texture attributes. Hence, should conv-based methods be better than Transformer for $A_{sub}$? \n8. From Fig. 3, why does $S(sys) have positive values when using around 0 to 100 samples? My understanding is that A_non is always better than A_sys as A_non is tested only on seen concept combination. Could you please explain why S(sys) are positive when few samples (between 0 to 100 samples) are present?\n9. From Fig. 4, it can be seen that having smaller number of classes with more tasks reduce the accuracy. It contradicts with the observation on Line 320-321: \"the learner needs a smaller number of compositional features necessary for distinguishing classes in each task\"\n\nReferences:\n[1] Zerroug et al., \"A Benchmark for Compositional Visual Reasoning\", NeurIPS, 2022.\n\n[2] Wang, Zifeng, et al., \"Dualprompt: Complementary prompting for rehearsal-free continual learning\", ECCV, 2022.\n\n[3] Wang et al., \"S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning\", NeurIPS, 2022\n\n[4] Smith, James Seale, et al., \"Construct-vl: Data-free continual structured vl concepts learning\", CVPR, 2023.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The evaluation of model's compositional generalizability is useful for the research community. \n2. The proposed evaluation procedure is comprehensive with 5 different protocols to assess model's compositionality from different angles. \n3. The finding that modularity-based methods lack compositional generalization is interesting.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "KxAtVzAA3P", "reviewer": "Reviewer_6pHG", "summary": "", "questions": "", "limitations": "See the limitations and suggestions below.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is generally well-written; but it still needs some minor improvements (see below).\n2. The compositional generalization problem in continual learning is an important and interesting problem.\n3. Multiple aspects of compositional generalization are covered and assessed.\n4. The dataset design is well-structured and well-organized (some concerns are expressed below).", "weaknesses": "", "quality": null, "clarity": 1, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper underwent a thorough review process by five reviewers, with most of them showing a positive inclination toward accepting the paper after the authors' rebuttal. Based on the positive feedback from the majority of reviewers, I am inclined to recommend accepting this paper. I encourage the authors to carefully incorporate the suggested revisions and clarifications from the reviewers (especially for the concern of how CGQA and COBJ reflect practical applications raised by the reviewer xdij) to further enhance the paper's quality in the final version.
Decision: Accept (Poster)