SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Construction of COBJ The images in Object365 are of high resolution and contain multiple instances of classes, annotated with bounding boxes (as shown in Figure 3a). To conveniently utilize these images, we need to resize them. However, resizing high-resolution images directly can lead to a loss of object resolution, which may have a detrimental effect on the model's performance. Therefore, we employ a strategy where we select a specific bounding box within each object set to crop and generate a new image. The process is explained in Figure 3. Subsequently, we perform the resizing operation, allowing the object to occupy a larger proportion within the image and providing more pixels for analysis. The selection of bounding boxes follows the principle of minimizing the total distance between their respective center points. This process ensures that the closest bounding boxes are chosen. By using the cropped region shown in yellow in Figure 3b, we reduce the proportion of irrelevant parts and maximize the preservation of the original object's information.
Table 1: Results on CGQA. Accuracy  $(\%)\pm 95\%$  confidence intervals  $(\%)$  over 300 tasks are reported for few-shot testing phases. Task-IL settings denote with a postfix \*\*.
Methods	\( A_{con} \)	\( A_{sys} \)	\( A_{pro} \)	\( A_{sub} \)	\( H_n \)	\( A_{non} \)	\( A_{noc} \)	\( H_r \)	\( H_a \)
MultiTask	83.61	88.14 ± 0.62	85.94 ± 0.65	69.67 ± 0.75	80.35	91.55 ± 0.51	40.04 ± 0.99	55.71	68.28
Finetune	8.38	64.73 ± 0.78	65.43 ± 0.73	61.26 ± 0.67	63.75	68.54 ± 0.80	40.32 ± 0.72	50.77	57.84
ER	19.78	71.38 ± 0.75	70.11 ± 0.64	64.32 ± 0.69	68.46	77.27 ± 0.67	40.98 ± 0.72	53.56	61.60
GEM	8.56	66.56 ± 0.77	73.47 ± 0.61	62.80 ± 0.71	67.33	72.65 ± 0.73	41.64 ± 0.72	52.94	60.72
LwF	9.11	71.22 ± 0.74	73.28 ± 0.61	68.74 ± 0.66	71.03	76.56 ± 0.66	48.69 ± 0.77	59.52	65.93
EWC	8.22	64.99 ± 0.78	73.47 ± 0.64	63.25 ± 0.66	66.95	69.03 ± 0.74	41.38 ± 0.71	51.75	59.91
RPSnet	33.45	59.80 ± 0.83	60.26 ± 0.72	59.75 ± 0.74	59.94	64.22 ± 0.81	45.09 ± 0.64	52.98	56.95
MultiTask*	92.17	82.16 ± 0.68	84.22 ± 0.60	71.82 ± 0.75	79.01	86.44 ± 0.69	44.07 ± 0.95	58.38	69.23
Finetune*	72.46	70.32 ± 0.73	72.62 ± 0.63	66.33 ± 0.69	69.66	75.32 ± 0.70	43.26 ± 0.73	54.95	62.92
ER*	76.05	71.37 ± 0.70	72.67 ± 0.69	66.80 ± 0.63	70.19	76.28 ± 0.66	45.61 ± 0.77	57.09	64.29
GEM*	21.60	69.44 ± 0.75	73.14 ± 0.67	66.61 ± 0.66	69.63	73.31 ± 0.71	45.97 ± 0.74	56.51	63.71
LwF*	73.19	69.61 ± 0.77	74.22 ± 0.63	65.00 ± 0.65	69.40	75.25 ± 0.69	42.52 ± 0.72	54.34	62.48
EWC*	71.10	69.40 ± 0.79	72.37 ± 0.65	65.10 ± 0.67	68.83	74.99 ± 0.72	42.82 ± 0.77	54.51	62.29
MNTDP*	68.98	47.27 ± 0.91	47.41 ± 0.86	47.50 ± 0.85	47.39	52.28 ± 0.81	32.29 ± 0.83	39.93	44.09
6 Experimental Studies
In this section, we conducted experiments using SOTA continual learning algorithms on our proposed compositional benchmarks, namely CGQA and COBJ. The purpose of these comprehensive computational experiments is to address the following research questions: (1) Can our proposed benchmarks accurately evaluate the learners' ability to extract compositional features from a sequence of tasks? (2) Can five different few-shot testing schemes provide insightful perspectives on the learners' compositionality? (3) Do modularity-based approaches demonstrate superior few-shot testing accuracy, indicating their efficacy in extracting compositional features?
6.1 Experiment Settings
Compared Algorithms We evaluate five baseline algorithms, including: Finetune, which trains a continual learner sequentially for all tasks; regularization-based methods: GEM [35], LwF [33], and EWC [24]; a replay-based method: ER [6]. Additionally, we also evaluate two SOTA hierarchical compositional modularity approaches: MNTDP [50] (MNTDP-D), which dynamically incorporates new modules or reuses existing frozen modules to improve performance on the current task; RP-Snet [43], which randomly selects modules in each layer and employs regularization and experience replay for assistance. When evaluating these methods in the class-incremental learning (class-IL) setting, where the task identifier is unknown, we use a single-head classifier. In the task-incremental learning (task-IL) setting, where the task identifier is known, we employ a multi-head classifier (denoted by a postfix “*”). All the algorithms mentioned above are evaluated using a ResNet-18 backbone [13]. For detailed hyper-parameter settings and additional results using a ViT backbone [8], please refer to the Appendix.
Evaluation Metric During the continual training phase, we report the Average test accuracy  $A_{con}$  on all seen tasks after completing  $T = 10$  tasks. For the five different few-shot testing phases, namely sys, pro, sub, non, and noc, we report the Average test accuracy for each test (i.e.,  $A_{sys}, A_{pro}, A_{sub}, A_{non}$ , and  $A_{noc}$ ) on  $T_{nv} = 300$  tasks. It is worth noting that the  $A_{sub}$  metric is not applicable for COBJ due to the absence of attribute information in the source benchmark Objects365. We denote the prediction accuracy of the multi-task classifier  $m$  as  $\Delta(m(f(x), t), y)$ . The corresponding testing dataset for task  $t$  is denoted as  $D_t^{te}$ . The Average test accuracy is calculated as  $\frac{1}{T_{nv}} \sum_{t=1}^{T_{nv}} \mathbb{E}_{(x,y)} \sim D_t^{te}[\Delta(m(f(x), t), y)]$ . Additionally, we employ the Harmonic mean  $H_n = 3/(1 / A_{sys} + 1 / A_{pro} + 1 / A_{sub})$  among sys, pro, and sub,  $H_r = 2/(1 / A_{non} + 1 / A_{noc})$  between non and noc, and  $H_a = 5/(1 / A_{sys} + 1 / A_{pro} + 1 / A_{sub} + 1 / A_{non} + 1 / A_{noc})$  to show the compositionality, reference capability, and overall few-shot testing capability, respectively. The Harmonic mean places emphasis on achieving balanced performance across all few-shot cases simultaneously, measuring the model's ability to perform well across different scenarios.
Table 2: Results on COBJ. Accuracy  $(\%)\pm 95\%$  confidence intervals  $(\%)$  over 300 tasks are reported for few-shot testing phases. Task-IL settings denote with a postfix \*\*.
Methods	Acon	Asys	Apro	Hn	Anon	Anoc	Hr	Ha
MultiTask	39.73	72.70 ± 1.55	67.11 ± 1.86	69.79	83.38 ± 1.36	57.52 ± 1.81	68.08	68.93
Finetune	6.37	43.06 ± 1.28	41.89 ± 1.29	42.46	46.96 ± 1.51	46.12 ± 1.36	46.54	44.41
ER	7.67	46.51 ± 1.48	45.22 ± 1.38	45.86	57.58 ± 1.85	42.81 ± 1.32	49.11	47.43
GEM	5.57	46.11 ± 1.41	43.99 ± 1.33	45.03	49.51 ± 1.57	46.07 ± 1.47	47.73	46.34
LwF	6.73	55.10 ± 1.54	50.12 ± 1.54	52.49	59.37 ± 1.61	51.40 ± 1.55	55.10	53.76
EWC	6.53	42.73 ± 1.19	41.00 ± 1.32	41.85	46.69 ± 1.48	41.21 ± 1.34	43.78	42.79
RPSnet	21.93	73.26 ± 1.24	67.89 ± 1.44	70.47	78.82 ± 1.27	68.22 ± 1.24	73.14	71.78
MultiTask*	75.73	55.72 ± 1.75	49.84 ± 1.81	52.62	59.21 ± 1.89	53.43 ± 1.74	56.17	54.34
Finetune*	44.37	45.67 ± 1.39	44.18 ± 1.38	44.91	47.92 ± 1.54	47.80 ± 1.57	47.86	46.34
ER*	64.80	54.40 ± 1.73	51.33 ± 1.69	52.82	58.91 ± 1.73	48.83 ± 1.54	53.40	53.11
GEM*	36.53	45.57 ± 1.39	45.50 ± 1.39	45.53	51.93 ± 1.64	46.40 ± 1.36	49.01	47.21
LwF*	61.03	53.77 ± 1.63	50.06 ± 1.76	51.84	60.54 ± 1.58	54.53 ± 1.61	57.38	54.47
EWC*	53.87	45.63 ± 1.46	46.12 ± 1.47	45.88	51.16 ± 1.59	48.86 ± 1.48	49.98	47.84
MNTDP*	71.80	51.89 ± 1.41	49.16 ± 1.56	50.49	57.20 ± 1.59	48.52 ± 1.36	52.50	51.48
We report the performance results of our CGQA benchmark in Table 1 and the results of the COBJ benchmark in Table 2. For a detailed analysis of the COBJ benchmark results under different settings of  $T$ , please refer to the Appendix.
Multi-task baselines outperform CL methods in both continual tasks and compositional generalization performance. In both class-IL and task-IL settings, MultiTask(*) demonstrates the highest performance on  $A_{con}$  and  $H_{n}$ , which is as expected since this method benefits from i.i.d. training task sequences. To visualize the learned concept region, we employ the CAM [61]. The results in the Appendix for both CGQA and COBJ demonstrate that the proposed  $H_{n}$  accurately evaluates learners' performance in extracting compositional features. Notably, CL methods generally do not outperform MultiTask(*) on  $H_{n}$ , revealing that forgetting also contributes to the degradation of compositionality. This observation highlights the urgent question of how to equip learners with compositionality for the continual learning community.
Continual tasks prefer modularity-based methods Among the baselines, RPSnet achieves a significantly larger margin on  $A_{con}$ , particularly on COBJ, indicating that the reused knowledge in the modules can effectively mitigate forgetting. While MNTDP performs poorly on CGQA, it outperforms the other methods on the real-world benchmark COBJ. This difference can be attributed to the fact that forgetting is not as significant on CGQA as it is on COBJ, especially in the task-IL setting. Therefore, simply freezing modules does not provide efficient reusable (compositional) knowledge for the novel recombination of seen concepts. The second-best method on  $A_{con}$  is ER $(^*)$ , particularly in the class-IL setting, demonstrating the effectiveness of the memory buffer.
Modularity-based methods show no superiority on compositionality In terms of  $H_{n}$ , another winning method alongside ER* is LwF, although its performance on  $A_{con}$  is not outstanding. This suggests that the feature extractor learned by LwF is excellent at extracting compositional features, but the classifier fails to make correct predictions. The phenomenon of prediction bias is further discussed in the Appendix, with CAM visualization. Regularization-based mechanisms like GEM, LwF, and EWC struggle to solve this classifier issue. As a result, only replay-based methods (such as ER and RPSnet) show improved performance. Unfortunately, RPSnet and MNTDP* do not demonstrate superiority on CGQA, indicating that the reused knowledge is not sufficiently compositional for handling novel combinations of seen concepts. Freezing the modules for old tasks may not be appropriate for new tasks unless the modules have enough compositionality. Further efforts are required to enhance modularity-based methods to bring benefits to compositionality. Interestingly, on COBJ, these two methods show competitive performance compared to others, with RPSnet performing the best. This indicates their superiority in dealing with real-world tasks.
Five few-shot tests exhibit a consistent order:  $A_{non} > A_{pro} > A_{sys} > A_{sub} > A_{noc}$ . Generally, the performance on non is the best, which is expected because models tend to favor the seen combinations and may struggle with compositional gaps to some extent. The performance gap between sys and pro is acceptable, but it is larger for sub. This is because sub requires the composition of attributes of concepts to form unseen recombination, demanding a higher-level compositional ability compared to sys and pro. Interestingly, when models have a ViT backbone (in the Appendix), we observe contrary results:  $A_{sub} > A_{pro} > A_{sys}$ . It can be attributed to the fact that the selected attributes for concepts in sub primarily pertain to texture attributes (e.g., color). This observation aligns with the analysis in [42] that the multi-head attention layer in ViT has a bias towards shape attributes, while the convolution layer has a bias towards texture attributes. Therefore, models with a ResNet-18 backbone are more sensitive to the texture difference between continual and sub. Regarding accuracy on pro, it is generally slightly better than on sys, except for MultiTask and ER, indicating that having more concepts can somewhat aid in understanding the images. The overall lower accuracy on the single-head baselines (class-IL setting) compared to their multi-head counterparts (task-IL setting), except for LwF, demonstrates that the single-head classifier makes it difficult for the feature extractor to retain compositional knowledge.
Concept-level forgetting We conducted a case study to investigate the phenomenon of forgetting with regard to concepts. Following a continual training phase, we evaluated the learned models on four few-shot task sets:  $non(f)$ , which consists of seen combinations of freshly learned concepts in the last task;  $non(o)$ , which consists of seen combinations of old concepts from previous tasks;  $sys(f)$ , which consists of novel combinations of freshly learned concepts in the last task; and  $sys(o)$ , which consists of novel combinations of old concepts from previous tasks. If the model experiences forgetting on the concept level, we would expect a small gap between  $non(f)$  and  $sys(f)$  and a large gap between  $non(o)$  and  $sys(o)$ . The construction of these two task sets is detailed in the Appendix. We also report the related scores  $S(o) = (A_{sys(o)} - A_{non(o)}) / A_{non(o)}$  and  $S(f) = (A_{sys(f)} - A_{non(f)}) / A_{non(f)}$  to clearly show the performance gaps. The average test accuracy and the related score are shown in Figure 4.
Our observations reveal that  $A_{\text{non}(f)} > A_{\text{non}(o)}$  and  $A_{\text{sys}(f)} > A_{\text{sys}(o)}$ , which is as expected since fresh concepts are newly updated in the last continual task. Additionally,  $S(f) > S(o)$ , indicating that the learner has better systematicity compositional generalization capacity for fresh concepts than for old concepts. These observations demonstrate that the learner indeed learns concepts in a composition-wise manner and forgetting occurs primarily with old concepts. On the other hand, Multi-Task, which is regarded as the upper bound, experiences the smallest performance drop on both fresh and old tasks, confirming our expectation that the learner jointly learns all continual tasks. This observation also
Figure 4: Fresh tasks vs old tasks on CGQA. MT stands for MultiTask and FT stands for Finetune. Take away: fresh concepts suffer little performance drop when testing on novel combinations (sys(f)) while old concepts (sys(o)) on the contrary.
suggests that the continual learner experiences forgetting not only on old tasks but also on old concepts. To visualize the learned concepts, we used CAM; please refer to the Appendix for more details.
Sample efficiency for learning compositionality To investigate the number of training samples required for a continual learner to acquire the ability of compositional generalization, we vary the number of training samples for each class in the CGQA continual training tasks. It is intuitive that a small number of training samples not only hinders novel combinations (i.e., sys, pro, sub), but also affects the seen combinations (i.e., non). Examples of sys and non are depicted in the left and middle figures of Figure 5, respectively. The complete results can be found in the Appendix. Moreover, we utilize the related score, denoted as  $S(sys)$ . It is calculated as  $(A_{sys} - A_{non}) / A_{non}$ , where a larger positive  $S(sys)$  indicates better compositional generalization ability on Systematicity, while a smaller negative value indicates poor handling of novel combinations of seen concepts by the learner. The results are presented in the right figure of Figure 1. For almost all methods,  $S(sys)$  reaches
Figure 5: Varying number of training samples on CGQA. Take away: Accuracy is improved with more training samples for both sys and non. Related scores become smooth and stable if more than 300 training samples for each class. This fact shows that 300 samples are needed for learning compositionality.
convergence when trained with more than 300 samples for each class. Beyond this threshold, the ratio of improvement for sys to non remains unchanged. Consequently, we can conclude that learners need 300 samples for each class to learn compositionality in CGQA.
Figure 6: Number of continual tasks vs Hn on 10-way COBJ testing tasks. Take away: A larger number of continual tasks (smaller ways for each task) hinders compositionality.
Varying number of continual training tasks As the number of training tasks increases, the number of classes in each task decreases while the total number of classes remains constant. Consequently, compared to a single task that includes all the classes, the learner requires a smaller number of compositional features to distinguish classes in each individual task. However, these features may be insufficient for future compositional tasks. To evaluate the learners, we train them on three (10-way), five (6-way), and ten (3-way) COBJ tasks respectively, and then assess their performance on 300 10-way few-shot sys and pro tasks. We visualize the results using  $H_{n}$ , as shown in Figure 6. The observation aligns with the aforementioned understanding. For more detailed information, please refer to the Appendix.
In this paper, we present two benchmarks, CGQA and COBJ, which offer ample samples for CL. Additionally, we introduce CFST, a novel evaluation protocol that enables comprehensive assessment of compositionality in three key aspects: systematicity, productivity, and substitutivity. The construction of our benchmarks is based on publicly available sources, ensuring the absence of sensitive data and ethical suitability for research purposes. Experimental results demonstrate that our protocol provides valuable insights for SOTA CL methods. However, modularity-based approaches do not exhibit superiority when subjected to compositional testing, highlighting the need for significant advancements in this area. We believe that our benchmarks and evaluation protocol will inspire researchers to prioritize compositionality during the development of continual learners.
Limitation and future work While our work specifically focuses on local compositionality, future studies will explore the global scopes of compositionality, extensively examined in other fields such as natural language processing. Although our candidate compositions of concepts are limited, they sufficiently serve the purpose of testing compositionality in continual learners. Moving forward, the combination of multiple benchmarks will be considered to form a more comprehensive evaluation framework.
This work was supported by National Natural Science Foundation of China (Grant No. 62250710163, 62250710682), Guangdong Provincial Key Laboratory (Grant No. 2020B121201001), the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (Grant No. 2017ZT07X386), The Stable Support Plan Program of Shenzhen Natural Science Fund (Grant No. 20200925174447003), Shenzhen Science and Technology Program (Grant No. KQTD2016112514355531).
[1] Amit Alfassy, Leonid Karlinsky, Amit Aides, Joseph Shtok, Sivan Harary, Rogerio Feris, Raja Giryes, and Alex M Bronstein. Laso: Label-set operations networks for multi-label few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6548-6557, 2019.
[2] Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: Scan both left and right. arXiv preprint arXiv:1809.04640, 2018.
[3] Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey. arXiv preprint arXiv:2012.09823, 2020.
[4] Gail A. Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37(1):54–115, 1987.
[5] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019.
[6] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019.
[7] Verna Dankers, Elia Bruni, and Dieuwke Hupkes. The paradox of the compositionality of natural language: a neural machine translation case study. arXiv preprint arXiv:2108.05885, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[9] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017.
[10] Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[11] Stephen T Grossberg. Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control, volume 70. Springer Science & Business Media, 2012.
[12] Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[61] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.