SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weiduo Liao $^{1,2}$ , Ying Wei $^{1*}$ , Mingchen Jiang $^{1}$ , Qingfu Zhang $^{1*}$ , Hisao Ishibuchi $^{2*}$
$^{1}$ Department of Computer Science, City University of Hong Kong
$^{2}$ Department of Computer Science and Engineering, Southern University of Science and Technology weiduliao2-c@my.cityu.edu.hk, {yingwei, qingfu.zhang}@cityu.edu.hk, jiangmingchen0129@gmail.com, hisao@sustech.edu.cn
Compositionality facilitates the comprehension of novel objects using acquired concepts and the maintenance of a knowledge pool. This is particularly crucial for continual learners to prevent catastrophic forgetting and enable compositionally forward transfer of knowledge. However, the existing state-of-the-art benchmarks inadequately evaluate the capability of compositional generalization, leaving an intriguing question unanswered. To comprehensively assess this capability, we introduce two vision benchmarks, namely Compositional GQA (CGQA) and Compositional OBJECTs365 (COBJ), along with a novel evaluation framework called Compositional Few-Shot Testing (CFST). These benchmarks evaluate the systematicity, productivity, and substitutivity aspects of compositional generalization. Experimental results on five baselines and two modularity-based methods demonstrate that current continual learning techniques do exhibit somewhat favorable compositionality in their learned feature extractors. Nonetheless, further efforts are required in developing modularity-based approaches to enhance compositional generalization. We anticipate that our proposed benchmarks and evaluation protocol will foster research on continual learning and compositionality.
Human understanding of the world relies on abstraction of concrete objects, allowing for the disentanglement of high-level understandings into low-level concepts. This ability, known as compositionality, is a crucial aspect of human intelligence as it addresses the stability-plasticity dilemma [11], and introduces a computationally efficient learning paradigm [3, 27]. By leveraging previously acquired concepts while acquiring new ones, individuals can effectively build and maintain knowledge when faced with a new task. For instance, accurate comprehension of varied bird wing shapes enables rapid identification of a new bird species by focusing on its wings. This compositional approach allows continual learners to achieve efficient knowledge reuse, benefiting not only in preventing catastrophic forgetting [37] of previous tasks (i.e., stability) but also in rapidly adapting to new tasks (i.e., plasticity).
There have been various approaches to continual learning (CL) that can be roughly categorized into three directions. Firstly, regularization-based methods [24, 33, 35] introduce regularization techniques to control the gradient update process for the current task. Secondly, replay-based methods [6, 4] store past tasks for later replay during training. Lastly, parameter-isolation-based methods, specifically modularity-based methods [28, 36, 43, 41], employ static or dynamic model architectures and allocate distinct parameters to each task. Nonetheless, it remains largely unclear
(c) CZSL
Figure 1: Process for CFST to test compositional generalization capability and comparison with CZSL. Taking CGQA as an example, after continually training  $T$  tasks in (a), the feature extractor  $f$  is frozen and we assign a new classifier specifically for each few-shot task. We use five types of few-shot tasks (b), which are introduced dually in section 4, to evaluate three aspects (i.e., systematicity, productivity, substitutivity) of compositionality, with the references of non and noc. CFST aims to challenge the compositional generalization ability of SOTA continual learners. Note that this protocol is suitable for class-IL and task-IL settings and we take the task-IL setting for illustration. Different from the CZSL setting in (c), our setting does not provide any concept-level supervision.
Concept Label
Compositional Label
whether these methods truly satisfy compositionality. The current works have two main drawbacks. Firstly, the evaluation mainly focuses on catastrophic forgetting and overlooks forward transfer [35, 5]. The most relevant work to study compositionality is [41], which evaluates compositionality on a toy colored-MNIST dataset but only concentrates on color-shape composition. Secondly, the majority of existing CL benchmarks, such as Split-CIFAR10/100 and Split-MNIST, are insufficient to comprehensively evaluate compositionality. One potential alternative could be adapting benchmarks from the Compositional Zero-Shot Learning (CZSL) community for the CL setting. However, many of these benchmarks are either toy, synthetic, and lacking systematicity (e.g., CLEVR [21]), or restricted to simple object-attribute composition (e.g., UT-Zappos [56] and MIT-States [19]).
To address the aforementioned drawbacks and stimulate research on compositionality, we have developed two benchmarks: one synthetic yet systematic benchmark called CGQA, and one real-world benchmark called COBJ, both of which involve vision classification tasks. Additionally, we have designed an evaluation protocol called Compositional Few-Shot Testing (CFST), as illustrated in Figure 1), comprising a continual training phase where a learner accumulates knowledge from a sequence of tasks, followed by five few-shot testing phases. CFST aims to evaluate three compositional capabilities: 1) Systematicity (sys) — the ability to correctly predict outcomes for few-shot tasks with novel recombinations of familiar concepts; 2) Productivity (pro) — the ease of understanding combinations involving an increasing number of concepts; 3) Substitutivity (sub) — the speed at which the learner can acquire knowledge about slightly different concepts from those learned previously. Two additional evaluation aspects are considered: 1) Non-novel (non) — assessing the prediction accuracy for seen combinations of concepts; 2) Non-compositional (noc) — establishing a lower bound of accuracy for few-shot tasks by evaluating the learner with unseen concepts. CFST challenges the compositional generalization capability of state-of-the-art continual learners, with a primary focus on the feature extractor's ability to capture compositional concept features after continual training. Drawing inspiration from the field of natural language [18], we summarize the basic principles of compositional testing in vision. Firstly, the testing targets are novel recombinations of concepts found in the training set, ensuring that the continual learner has the opportunity to learn the meanings of these concepts (solvability). Secondly, while providing supervision for the testing task, it is crucial not to overwhelm the learner with excessive information, as they are expected to "quickly" understand the testing targets (few-shot). Finally, after acquiring sufficient knowledge during training, the feature
extractor is frozen to prevent further learning and overfitting during the testing phase (frozen feature extractor). Consequently, the continual learner must possess compositional generalization ability to bridge the systematic gap between training combinations and testing recombinations.
We assess the performance of five baseline methods and two parameter-isolation-based (modularity-based) methods on our benchmarks using CFST. The results indicate that the continual learning (CL) paradigm does introduce a decline in compositional performance on the concept level compared to the multi-task learning paradigm. Additionally, the efficiency of learning compositionality is highly influenced by the number of classes in each continual task. Furthermore, the state-of-the-art CL methods employing ResNet-18 backbone demonstrate some degree of compositional generalization capability in terms of Systematicity and Productivity, but struggle with Substitutivity. Conversely, methods utilizing the ViT backbone exhibit a preference for Substitutivity. Remarkably, we observe that the reusable knowledge acquired by modularity-based methods does not precisely correspond to the underlying concepts. It is clear that there is still a significant journey ahead for continual learners to develop interpretable compositionality.
Compositional benchmarks Compositionality is widely considered to be one of the most crucial elements in achieving a human-level understanding of the world. While compositionality has been extensively studied in the field of natural language, particularly in the context of local and global understanding [7], there is a need to evaluate the compositional capability of models. To address this, researchers have focused on creating datasets with specific split methods for training and testing purposes [7, 46, 18, 26, 23, 2, 22, 57, 10, 16]. However, the exploration of compositionality in the field of computer vision is still in its early stages. While some proposed benchmarks prioritize the use of toy or synthetic images [44, 41, 58], there is a lack of comprehensive evaluation on real-world datasets since most of them only test a limited aspect of compositionality (i.e., systematicity) [1, 31, 15, 14, 60, 34, 7, 20, 47].
Continual learning Continual learning typically addresses the issue of catastrophic forgetting through various approaches, which can be broadly categorized into four families. Regularization-based methods [35, 33, 24] incorporate regularization techniques into the gradient updates to mitigate forgetting of previous tasks. Replay-based methods [6, 48, 59, 12] involve retraining on past task data or storing memory-efficient features to retain knowledge from previous tasks. Parameter-isolation-based, particularly hierarchical modularity-based methods [43, 30, 50, 9], focus on modularizing learners to enable compositional learning of tasks [38]. Building on the advancements of Vision Transformers [8], several prompt-based methods [53, 52, 51] aim to extract task-specific knowledge from a pre-trained backbone by using prompts. However, comparing these pre-trained methods to others may be somewhat unfair as they might have prior exposure to certain concepts or their combinations. Furthermore, only a limited number of studies (e.g., [41, 40, 50, 47]) explicitly evaluate their compositional capabilities, especially in the field of computer vision. For instance, [41] uses a toy dataset (color-MNIST) to evaluate simple color-shape compositions. [47] introduces a vision-and-language benchmark for continual concept learning tasks. In other works [32, 49, 1, 31, 15, 16, 25], including studies on compositionality outside the continual learning community, the datasets themselves possess compositional characteristics, but their evaluation may not rigorously test novel combinations.
Vision continual classification tasks Continual learning focuses on a sequence of  $T$  vision classification tasks, each involving an  $N$ -way classification. For each task  $t \in \mathcal{T} = [1 \dots T]$ , there is a corresponding training set  $\bar{D}_t = \{(x_t, y_t)\}$ , where  $x_t \in \mathcal{X}$  represents the input image and  $y_t \in \mathcal{Y}$  is the corresponding class label. Here,  $N$  denotes the number of classes within each individual task. A continual learner is a neural network comprised of two main components. Firstly, there is a feature extractor  $f(x): \mathcal{X} \to \mathcal{L}$ , which maps input images to an extracted feature space. Secondly, a multi-task classifier  $m(f(x), t): \mathcal{L} \times \mathcal{T} \to \mathcal{Y}$  is present. This classifier takes the extracted features from the feature extractor,  $f(x)$ , as well as the task identifier,  $t$ , as inputs and produces a prediction for the corresponding task. Here,  $\mathcal{V} \subset \mathbb{R}$  represents the label space, which will be discussed further in
the subsequent paragraph. It is important to note that tasks are sequentially presented to the continual learner, thereby deviating from the assumption of independent and identically distributed (i.i.d.) data. Remark 3.1. This setting corresponds to the task-incremental (task-IL) scenario. Additionally, we also investigate the class-incremental (class-IL) setting, where the task identifier  $t$  is not provided. Consequently, the multi-task classifier  $m(f(x),t)$  simplifies to  $m(f(x))$ .
Concept-based categories From a human learning perspective, it is essential for representations to be compositional, meaning that a global understanding of an object is formed by combining all its semantic sub-parts [39]. For instance, people learn to recognize others by memorizing characteristics such as their hair color and mouth size. To assess whether the model is capable of learning compositional representations in images, we initially define the components, referred to as concepts, that we want the model to learn explicitly. Specifically, we create a pool of candidate concepts, denoted as  $\mathcal{C} = \{c^i\}_{i=1}^M$ , where  $M$  represents the number of concepts. The vector  $c^{1:M} = (c^1, \ldots, c^M) \in \{0,1\}^M$  indicates the presence of these concepts, where  $\{0,1\}^M$  denotes the  $M$ -fold Cartesian product  $\{0,1\} \times \dots \times \{0,1\}$ , and  $c^i = 1$  indicates the presence of the  $i$ -th concept, while 0 denotes its absence. Further details regarding the continual training stage and evaluation stages will be discussed in section 4.
Remark 3.2. Although we define a multi-hot concept space for data generation, it is important to note that  $y$  remains a single image-level label, and no concept label is provided to the candidate learner. This distinction is illustrated in Figure 1, where categories such as {Door, Shirt} and {Grass, Shirt} are considered as two completely separate categories despite sharing the concept Shirt. As a result, the learner must deduce the underlying compositional concepts within an image. This represents the key difference between our label space definition and compositional zero-shot learning (CZSL), as depicted in Figure 1c.
4 Compositional Few-Shot Testing Protocol: CFST
We will now elucidate the precise methodology for assessing the compositionality of a learner through the utilization of the Compositional Few-Shot Testing (CFST) protocol, as illustrated in Figure 1.
Concept factorization We adopt attribute factorization [54, 29] to model the data generation process by using a latent variable  $z$  since it accurately captures the different concept distributions between training and testing tasks. Denoting the joint distribution of the image  $x$  and the corresponding hidden concepts as  $p(c^{1:M}, x)$ , we can write
$$
z \sim p (z), \quad c ^ {i} \sim p (c ^ {i} | z), \quad x \sim p (x | z), \quad p (c ^ {1: M}, x) = p (c ^ {1: M}) \int p (x | z) p (z | c ^ {1: M}) \mathrm {d} z.
$$
We split the label space  $\mathcal{Y}_{tr},\mathcal{Y}_{nv}\subset \mathcal{V},\mathcal{Y}_{tr}\bigcap \mathcal{Y}_{nv} = \emptyset$  using different marginal distributions,  $p(c^{1:M})\neq p_{tr}(c_{tr}^{1:M})\neq p_{nv}(c_{nv}^{1:M})$  and keeping the same conditional generative model  $p(x|c^{1:M}) = \int p(x|z)p(z|c^{1:M})\mathrm{d}z$  for different phases, thus, artificially introducing compositional distribution shifts. Note that,  $c_{tr}^{1:M},c_{nv}^{1:M}$  are the corresponding concepts in  $\mathcal{Y}_{tr},\mathcal{Y}_{nv}$ , respectively. Without loss of generality, we bound the maximal number of activated concepts in  $\mathcal{Y}_{tr}$ , such that  $p_1$ -norm  $\| c_{tr}^{1:M}\| _1\leq M_{tr}$ . The principles guiding the evaluation are summarized as follows:
1. Few-shot tasks: After continually training with  $T$ $N$ -way tasks with  $p_{tr}(c_{tr}^{1:M})$ , we expect this learner should have acquired sufficient knowledge in the concept pool  $\mathcal{C}$  and be able to recognize novel concept combinations  $y_{nv} \in \mathcal{Y}_{nv}$  with  $p_{nv}(c_{nv}^{1:M})$  using only a small number of support samples. Thus, we adopt few-shot tasks, in which each concept combination  $y_{nv}$  has a small support set  $D_s$ . Proper recognition of underlying compositional concepts is crucial to avoid severe overfitting.
2. Solvability: All  $M$  concepts in  $\mathcal{C}$  should be encountered at least once in the continual training tasks  $\{D_t\}_{t=1}^T$ . Additionally, the learner should have sufficient exposure to learn the concepts in various combinations. We take the label  $\{\text{Grass, Shirt}\}$  with concept Grass in Figure 1 as an example, Grass should be seen in at least one training task, e.g.,  $\{\text{Grass, Table}\}$ . This assumption aligns with the general idea that standard gradient-based models cannot generalize to nonlinear functions without observing enough diverse examples [55].
3. Novelty: Few-shot tasks should only contain novel recombination of concepts not seen in the training set, i.e.,  $\mathcal{V}_{nv} \cap \mathcal{V}_{tr} = \emptyset$ . The presence of any seen label would negatively impact the
evaluation of the learner's compositional ability. In the above example,  $\{D_t\}_{t=1}^T$  should not contain any  $\{\text{Grass}, \text{Shirt}\}$  sample.
4. Frozen feature extractor: We recommend not further updating the feature extractor  $f$  using the support samples  $D_{s}$  in the few-shot evaluation task. Only the classifier  $m$  is updated to assess whether  $f$  extracts the expected features related to the underlying concepts in the few-shot task. Empirically, a non-frozen feature extractor attempting to learn from a few-shot task leads to a noticeable reduction in performance, as shown in Appendix.
Based on the mentioned principles, we present five different few-shot testing schemes for the continual learner:
Systematicity Novel Testing (sys) To examine the comprehension of learners on novel combinations of concepts in  $\mathcal{C}$ , we design  $T_{nv}$ $N$ -way  $K$ -shot tasks. Each task includes  $N$  concept combinations  $y_{sys}$  randomly selected from  $\mathcal{V}_{sys}$ , with  $K$  samples for each  $y_{sys}$ . Here,  $\mathcal{V}_{sys} \subset \mathcal{V}_{nv}$  and the corresponding  $\|c_{sys}^{1:M}\|_1 \leq M_{tr}$ . It should be noted that  $\mathcal{V}_{nv}$  does not include any previously seen combinations in  $\mathcal{V}_{tr}$ , ensuring the fulfillment of the novelty requirement.
Productivity Novel Testing (pro) The acquired knowledge should be able to generalize effectively to complex images that contain a greater number of previously encountered concepts. After learning about {Door, Shirt}, {Grass, Table}, and {Hat, Leaves}, the learner should have no trouble recognizing {Door, Leaves, Shirt, Table}. To assess this ability, we construct few-shot tasks that involve more concepts per task compared to the sys testing. Specifically, we consider another label space  $\mathcal{Y}_{pro} \subset \mathcal{Y}_{nv}$  where the candidate label  $y_{pro} \in \mathcal{Y}_{pro}$  satisfies the condition  $\|c_{pro}^{1:M}\|_1 > M_{tr}$ .
Substitutivity Novel Testing (sub) Concepts can possess specific attributes, such as the color of a Shirt being either red or green. Given knowledge acquired from images of red shirts (or shirts that are not green) and an understanding of Grass being green, learners are expected to quickly recognize that a green shirt is still a Shirt, as it combines the attribute of green color with the object of a shirt. In order to assess compositional understanding at the attribute level, we maintain the same distribution of concepts  $p_{sub}(c_{sub}^{1:M}) = p_{sys}(c_{sys}^{1:M})$ , but utilize different conditional generative models  $p_{sub}(x|c_{sub}^{1:M}) \neq p(x|c^{1:M})$ . It is important to note that this attribute-based testing should be observed in relation to other concepts during continual training, allowing learners the opportunity to recognize it (solvability). This distinction is the primary difference between our sub approach and the domain-incremental setting.
The aforementioned three novel few-shot schemes provide a comprehensive evaluation of compositionality. In addition, we will introduce two reference schemes for the purpose of comparison and analysis.
Non-novel Testing (non) To assess the few-shot testing performance of learners on trained combinations, we create few-shot tasks where the label is taken from  $\mathcal{Y}_{tr}$  rather than  $\mathcal{Y}_{sys}$ . This particular setting intentionally violates the novelty assumption. The average test accuracy on these non tasks serves as a complement to the average test accuracy on the tasks involving continually trained combinations. Additionally, comparing the results with other few-shot novel methods (such as sys) allows us to determine if the learner possesses the specific compositionality.
Non-compositional Testing (noc) We introduce noc tasks to evaluate learners' few-shot testing performance on combinations that involve unseen concepts. In this case, the label  $y_{noc}$  is selected from  $\mathcal{Y}_{noc}$ , and the condition  $\| c_{noc}^{1:M} \|_1 \leq M_{tr}$  applies, with  $\mathcal{C}_{noc} \cap \mathcal{C} = \emptyset$ . These tasks follow the normal continual learning scheme with  $T + 1$  tasks, but the last task is a few-shot task and the feature extractor remains frozen. This particular setting intentionally violates the solvability assumption. The average test accuracy on noc tasks serves as a useful metric to indicate the lower limit of few-shot testing performance.
5 Proposed Benchmarks
The existing continual learning (CL) community lacks a comprehensive benchmark specifically designed to evaluate compositionality. In order to address this, we can adapt existing benchmarks for Compositional Zero-shot Learning (CZSL) for this purpose. However, directly transforming CZSL
benchmarks into CL versions, as done with Split-CIFAR100, is not appropriate. This is because some CZSL benchmarks are considered to be toy benchmarks, such as CLEVR [21]. Furthermore, most CZSL benchmarks do not emphasize systematic compositionality, as they primarily focus on simple object-attribute compositions, such as UT-Zappos [56] and MIT-States [19]. Simply combining multiple benchmarks is also not suitable, as the concepts involved in different benchmarks might vary, resulting in a lack of cross-benchmark concept combination.
Our objective in developing benchmarks is to incorporate a wide range of concept combinations alongside a substantial number of instances for each concept. Thus, we first introduce a synthetic benchmark known as Compositional GQA (CGQA), derived from GQA [17] (License CC BY 4.0). CGQA offers comprehensive combinations of concepts and a substantial number of concept instances. Then, to evaluate continual learners using real-world images, we also propose a real-world benchmark called Compositional Objects365 (COBJ), sourced from Objects365 [45] (License CC BY 4.0). Due to the limitations of space, please refer to the Appendix for image examples and additional benchmark statistics.
Figure 2: Example of CGQA construction.
Figure 3: Example of COBJ construction.
Construction of CGQA Since the available number of annotations in GQA is inadequate to construct the compositional benchmark using the original images, we have chosen to artificially combine suitable objects and create new images for our purposes. The process is illustrated in Figure 2. Initially, we crop the image using the bounding box of the selected object and then resize the cropped region to a resolution of  $98 \times 98$  pixels. Next, we randomly position the processed object image within a  $2 \times 2$  grid. Any unoccupied grid spaces are filled with black pixels to ensure a complete composition.