Continual Learning in Open-vocabulary Classification with Complementary Memory Systems
Authors: Zhen Zhu
Weijie Lyu
Yao Xiao
Derek Hoiem
Venue: Accepted by TMLR ()

Abstract: We introduce a method for flexible and efficient continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in human cognition. Specifically, we propose to combine predictions from a CLIP zero-shot model and the exemplar-based model, using the zero-shot estimated probability that a sample's class is within the exemplar classes. We also propose a ``tree probe'' method, an adaption of lazy learning principles, which enables fast learning from new examples with competitive accuracy to batch-trained linear models. We test in data incremental, class incremental, and task incremental settings, as well as ability to perform flexible inference on varying subsets of zero-shot and learned categories. Our proposed method achieves a good balance of learning speed, target task effectiveness, and zero-shot effectiveness. Code is available at https://github.com/jessemelpolio/TreeProbe.

Published in Transactions on Machine Learning Research (10/2024)
Continual Learning in Open-vocabulary Classification with Complementary Memory Systems
Zhen Zhu
zhenzku, wilknoiss edu
Siebel School of Computing and Data Science University of Ilinois Urbena- Champaain
Weijic Lyu'
www.:3uucmeeedd.co edu
University of California Merced
Yao Xiao
yooz @illinois. edu
Siebel School of Computing and Data Science University of Illinois Urbana- Champaign
Derck Hoiem
dhoiem@@llinois.. edu
Siebel School of Computing and Data Science University of Blinois Urbana-Cher Channpag
Reviewed OpenReview: https:// https:///penreeveeu ope.r/vee.. net/ per/formm? httpu///wwa.f.e////oo/oooooooooa
Abstract
We introduce method for flexible and efficient. continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in human cognition. Specifically, we propose to combine predictions from a CLIP zero-shot model and the exemplar-based model, using the zero-shot estimated probability that sample's class is within the exemplar classes. We also propose "tree probe" method, an adaption of lazy learning principles, which enables fast learning from new examples with competitive accuracy to batch-trained linear models. We test in data incremental, class incremental, and task incremental settings, well ability to perform flexible inference on varying subsets of zero-shot and learned categories. Our proposed method achieves good balance of learning speed, larget Lask effectiveness, and zero-shot effectiveness. Code is available at littps// brnthub.com/jjsseennee preeseeeneeplllloTTeeeeeeeee
TreeProbe
1 Introduction
A major machine learning goal to create flexible learning systems, which we investigate in the context of image classification, with the following goals:
Flexible inference: classify an image within any label set, testable via zero-shot classification
Continual improvement: accuracy improves as new data received for related tasks without detriment. to previous tasks
Efficient incremental learning: can efficiently update the model with new training examples
Flexible inference and continual improvement lead robust and widely usable systems. Efficient incremental learning reduces cost of training and facilitates responsive and interactive learning approaches. Imagine interacting with nature classification app: begins with open-vocabulary classifiers, providing reasonable predictions for plant and animal categories. Whenever expert user correct mistakes, the app learns from their input to improve on the labeled species without detriment to its existing capabilities, as shown in Fig.
*Co-first author. Work is done while being student UTUC

---

Published in Transactions on Machine Learning Research (10/2024)
Update model in real time
Improve related predictions
Blue Jay x (45%)
Blue Jay
Indigo Bunting (88%)
(45%)
Indigo Bunting (32%)
Indiso Bunting (32%)
Blue Grosbeak (7%)
Don't forget existing classes
Blue Grosh (17%)
Blue Grosbeak (17%)
Blue Jay (3%)
Figure 1: Our work proposes a method to continually expand and improve classification ability, updating the model quickly with each new labeled example. This is especially helpful in long-tailed classification problems, like the depicted nature classification app. Given a user-provided image, the system predicts the likely classes and immediately updates given any corrections. In this way, the system becomes increasingly capable without any costly offline retraining.
However, to our knowledge, no solution exists that meets all these goals. Open vocabulary methods, such as CLIP (Rafford et al. 2021), attain flexible inference but cannot easily learn and improve with new examples. Continual learning methods, such as LwF (Li & Hoiem 2018), iCaRL (Rebuffi et al. 2017), EWC (Kirkpatrick
ct al. 2016). and GDumb (Prabhu et al. 2020). lack flexible inference and are not casily extended to open vocabulary models. For example, LwF requires maintaining classification heads for previous lasks, and GDumb maintains examples for every possible label; neither strategy directly applics when the label set is unbounded. Recent prompt-based approaches Wang et al. (2022c bla:: [mithh et al. (2023) share the same disadvantages. In fact, one homogeneous system cannot easily accommodate all goals. A deep network can aggregate information over enormous datasets to achieve flexible inference, but updates from small targeted subsets can disrupt other tasks. Instanco- based methods can incrementally learn and contimually improve, but cannot generalize to novel labels.
To build such a flexible learning system, we are inspired by the flexibility of human learning and inference, enabled by complementary learning systems (CLS) (O'Reilly et al. 2014). The exemplar-based system forms new memories and associations with sparse connections, enabling non-disruptive storage of individual experiences. The consolidated system gradually consolidates information stored in the exemplar- system to a denscly connected network, encoding general knowledge and regularities across experiences. The interplay between these two complementary systems enables the brain to balance the rapid encoding of novel information and gradual generalization of knowledge. How can we make computer learning systems that likewise benefit from both consolidated and exemplar-based memory systems to continually learn with zero-shot inference ability?
We investigate in the context of open-vocabulary image classification, using frozen CLIP as the consolidated system. Our exemplar-based system stores individual representations of learning exemplars, and the challenge is to learn from them in way that is performant in both accuracy and learning time (Sec. 3.2) To balance speed and performance, achieve good accuracy but Iearns from new example in ((n) time, where represents the total number of exemplars. K-nearest neighbor (KNN) can learn in O(1) time but tends to be less accurale than linear probe. Based on local linear models from the lazy learning literature (Bontempi et al.) 1999), we propose a "tree probe" method that hicrarchically clusters examples and trains linear models for each cluster. The time learn from new example is Olog n) (in practice, essentially constant time), and the accuracy is close linear probe.
consider classic learning approaches. One option is linear probe, which
2

---

Published in Transactions on Machine Learning Research (10/2024)
A second challenge is to predict using both CLIP and exemplar models (Sec. 3.3) to achieve good performance new tasks while maintaining zero-shot performance. The exemplar model tends to perform well for test. samples with labels that are in the exemplar set ("exemplar covered"), while the CLIP model can potentially predict any label. At test time, we may not know whether the label of a given test image is exemplar-coven Our idea is to use CLIP to estimate the probability that an image's s label exemplar-covered and then use that probability to weight the predictions of the two models.
Our unified system achieves good accuracy, efficient learning, and flexible inference, which meets all three goals for flexible learning systems. We evaluate our method in the forms of data-incremente class- incrementall and task-incremental learning, as well as flexible inference with categorization tasks that involve some, all, or none of the sxemplar-covered labels. Additionally, we compare on an existing benchmark (Zheng et al., 2023). outperforming all compared approaches with the additional benefit of efficient learning. In summary, the reader may benefit from the following paper contributions:
Tree-probe exemplar model: Our locally linear models using a hicrarchical clustering can be considered a member the long-studied lazy learning approaches (Bontempi el al. 1999). but we are not aware of this specific method being proposed or used. Tree-probe has practically constant training time number of training samples and achieves better accuracy than KNN approaches. This is attractive for interactive scarch, active learning, and other applications where annotated examples are received in a trickle and fast. learning is required.
Exemplar and consolidated model combination with embeddings: Our approach, to use the con- solidated model to estimate applicability of the exemplar model and to combine model predictions in the label embedding space, enables effective continual open-vocabulary learning and performs significantly better than alternatives tested.
Flexible dearning/inferent /inference: Our proposed new benchmark evaluates both the ability to continually learn from new samples in various learning scenarios and to flexibly apply those learnings to various category sets. This may provide a useful test framework to further improve open- spen-vocabulary continual learning.
2
Related works
2.1 Open-vocabulary image classification
Open-vocabulary image classification aims to categorize images without constraints of predefined labels. CLIP [Radford et al., 2021) achieves this by learning embedding spaces of images and text, trained via a contrastive objective that corresponding image/ text pairs will be more similar than non-corresponding pairs. A new image can be classified into an arbitrary set of labels based on the similarity of the image embedding features to the texl embedding fealures corresponding to each label. An alternative approach is to jointly encode the image and text information and decode into text. Following this approach, vision- vision-language models, such as GPV-1 (Gupta et al., (002)), GPV-2 (Kamath et al.] 2022], VI-BBRRT (Su al.) 2020). VL-T5 (Cho et al. 2021), Unified-IO (Lu et al. 2022), Gato (Reed et al. 2022), and Flamingo (Alayrac al. 2022). can solve broad range of tasks that includes open-vocabulary image classification, but typically are larger and more complex than CLIP. We use CLIP effectiveness in a broad range of tasks (Wortsman et al.) 2022b Gu et al. 2021 Ghiasi et al.) 2021 Lin et al.
consolidated model, due to its simplicity and verified
2022 According to the data overlap analysis in (Radford et al.] 2021 though trained on Internet-scale
data, CLIP under-performs models trained for particular benchmarks. Our method contimuall and efficiently improves CLIP's capability for new concepts, enabling users to add customized expertise while maintaining general capability.
2.2 Continual learning
Wang et al. 2023) provides a comprehensive survey of continual learning techniques, which aim to acquire new
knowlodge over time while preserving previously learned knowledge (MeCloskey & Cohen 1989). Approaches
continual learning can be broadly categorized into regularization (Li & Hoiem
2016
Kirkpatrick et al.
2016: Zenke et al. 2017), parameter isolation (Aljundi et al.] 2017 Rusu al.
2016
Serrå et. al.] 2018
3

---

Published in Transactions on Machine Learning Research (10/2024)
Mallya & Lazebnik 2018: Zhang et al., 2020 and rehearsal methods (Rebuffi et al. 2017 Yan et al 2021
Lopez-Paz. & Ranzato, 2017: Shin et al. 2017). Regularization techniques generally impose constraints on the learning process to alleviate forgetting. Parameter isolation methods maintain learning stability by fixing subsets of parameters (Mallya & Lazebnik 2018 Serrà et al., 2018) or extending model with new
parameters (Rusu el. al.; 2016 Yoon et al. 2017 Zhang et al. 2020p. Recently, inspired from the Prompt Tuning technique (Lester et al., 2021], prompt-based approaches (Wang et al., 2022c a. Smith et al. 2023
Samadh et al., 2023) continually learn task-specific prompts to condition frozen ViTs (Dosovitskiy al,, 2021) for image classification. The learned prompts can be considered as expanded parameters of task-specific knowledge but tend to overfit as the training progresses, therefore losing the generalization capability of the frozen model. Khattak et al. (2023) propose to use a self-regulating mechanism to learn both task-specific and lask-agnostic ic prompls but the more effective deep prompling technique requires appending prompts at each transformer layer, impacting the training efficiency. Rehearsal methods involve storing and replaying past data samples during training (Shin et al.) 2017 Bang et al.] 2021: Zhang ct al.) (022). Some approaches (Tiwari et al. 2022 Yoon et. al.) 2022] Hao et al.) 2023) consider to maintain a coreset when concerning the memory capacity, but simple sampling methods work better if memory capacity is not concern (Prabhu et al. 2023
Many approaches highlight the effectiveness of incorporating neuro-inspired adaptability to balance memory stability and learning plasticity in artificial intelligence systems (Parisi et al. 2018 Kemker & Kanan 2018
Ayub & Wagner 2023 Jung et al. 2023 Liang et al. 2023 Madaan et al. 2023 For instance, the CLS theory is exemplified in DualNet (Pham et al. 2021), which integrates supervised and self-supervised learning to facilitate robust continual learning by emulating the hippocampus and neocortex functions. Similarly, Sarrraz et al. 2023) uses sparse coding techniques to mimic the brain's memory retention capabilities, and Parisi et al. [2018] leverages dual-memory recurrent self-organization for spatiotemporal representation. CBCL-PR (Ayub) & Wagner 2023) applies cognitively inspired models to improve the class-incremental learning in robotics. A
recent advancement (Khan et al. 2024) leverages a muxtture-of-ddapee architecture with generative routing to mitigate forgetting while maintaining parameter efficiency. FearNet. (Kemker & Kanan 2018) creates a brain-inspired dual-memory system containing a hippocampal network for recent memories and a prefrontal cortex nefwork for long-term storage, and a basolateral amygdala network for model selection. We share the inspiration from CLS theory, but our work is distinguished in its focus on efficient incorporation of new examples, ability for open vocabulary prediction, and demonstration on larger scale datasets.
To generalize to unseen categories, a line of meth (Skorokhodov & Elhoseiny 2021 Gautam et al. 2021
continually learn attributes (Farhadi et al.) 2009 Lampert et al. 2014) and use attribute-class relations as a way to generalize from seen to unseen categories. Despite using extra attribute annotations, these methods usually under-perform on unseen categories as compared to CLIP (Radford el al. 2021). We are not aware of many works that address continual learning the context of open-vocabulary image classification. WiSE-FT (Wortsman et al. 2022a) finetunes CLIP encoders on target tasks and averages finetuned weights with original weights for robustness to distribution shifts. Likewise, CLS-ER (Arani et al. averages model weights in different paces for its plastic and stable model to balance learning and forgetting, drawing inspirations from CLS. Though not specifically designed for open-vocabulary continual learning,
2022) exponentially
WiSE-FT and CLS-ER can be modified lo counter forgetting. Recently, ZSCL (Zheng et al. 2023) explores this setting but mainly focuses on finetuning consolidated CLIP encoders. It combines LwF (Li & Hoiem, 2016) and weight ensemble idea similar to WiSE-FT as the remody to reduce forgetting during finctuning. Rather than refining or expanding a single model, our approach is to create a separate exemplar-based model that complements a pre-trained and non-updated more general consolidated model. In our tree-probe method, the exemplar-base model continually learns using local rehearsal, such that incorporating a new example requires only storing the new image and text embeddings and tuning a linear model using a subset of visually similar examples. This requires little storage or computation and preserves the generality of the consolidated model.
2.3 Instance- based learning
Instanco-basod learning (IBL) (Aggarwal 2014) a family of learning algorithms that construct a decision boundary using a memory of training instances, allowing for efficient and flexible adaptation to new data points. This learning paradigm relies the principle of local approximation, where predictions are made
4

---

Published in Transactions on Machine Learning Research (10/2024)
based the stored instances that most similar to the query. One of the most well-known IBL methods is the k-Nearest Neighbors (KNN) algorithm, which has been extensively studied for its simplicity and effectiveness in various domains, including classification and regression tasks (Guo et al. 2003 Zhang et al, Song et al., 2017).
2006
2017
Closely related IBL is lazy learning, in which training data is organized for prediction at inference time instead of training time. This approach can be more practical than batch or "eager" learning in applications like online recommendation systems, when the training data is constantly evolving. Lazy learning approaches include KNN locally RBF network (Bottou & Vapnik 1992). linear regression (Atkeson et al 1997 or classification models [Aggarwal 2014), where models are trained based on neighbors to the query.
We investigate KNN, a form of locally linear classifiers and globally linear classifiers, exploring their trade-offs for training and inference time and accuracy on a growing set of exemplars, as well as how to combine their predictions with static imago-languaee foundation model.
3 Method
Exemplar-based System
Non-leaf
Leaf modes
Exemplar Set
TreeProbe
Exemplar
Train
Training flow
Classifier
Inference flow
Store
Pe vill)
CLIP Image Encoder (Consolidated System)
Image
VI
Weighting (AIM)
ply = vi(l)
exp(I(vI, Vt,)) j=1 exp(l(vi, V1,))
[apple, bear, bee, chair,
CLIP Text Encoder
Vt
Ps (
DilT)
Figure Method overview. (a) Our model integrates exemplar-based and consolidated systems. Final results made by fusing the predictions from both systems using weighting method such as AIM. (b) Illtstration of TreeProbe, which incrementally adds and hierarchically clusters examples. TreeProbe trains logistic regression classifiers using examples in updated leaf nodes. Colors of exemplars indicate different. categories.
Given an image 1 and label set Y, our system's inference task is to assign the correct label E Y. Each training example (or "exemplar") consists of an image, and a ground-truth label. Our problem setting is or "zero-shot", as the labels can be represented and related through text, and the task at inference time may contain labels or candidate label sets not observed during training. The training goal to efficiently update the model with cach new example, such that accuracy improves for related labels while maintaining zero-shot, performance.
called
Analogous to complimentary learning systems (CLS) theory (O'Reilly et al. 2014 our approach (Fig. uses two models: CLIP image/text encoder as the consolidated system (Sec. 3.1). and an exemplar-based system that stores and encodes image and label embeddings (Sec. 3.2 The CLIP model can assign confidence to any label from an arbitrary set. The exemplar-base consolidated model, but scope is limited to exemplar labels. In Soc. 3.3 we propose simple mechanisms combine the predictions of each model retain zero-shot capability while improving in tasks related received examples.
model can acquire expertise that complements the CLIP
5

---

Published in Transactions on Machine Learning Research (10/2024)
3.1 Consolidated system
Deep networks, such as CLIP (Radford al. 2021 relate to consolidated systems in humans in that they repeatedly process examples to iteratively consolidate experience and improve predictive ability in dense weight connections. We use CLIP as the consolidated system, taking advantage of its representations and open-vocabulary predictive ability learned from batch (raining on massive datasets. We wish to enable fast learning and retain zero-shot ability; therefore, we do not retrain or fine-tune the CLIP encoders. Adapters and prompt-based methods cannot meet the goal of efficient training because they require relatively large computation to run through a part or the whole CLIP encoders. Instead, we train complementary exemplar-based models to be used in combination with CLIP.
CLIP encodes an input image I using an image encoder fimg to an image embedding v, - fing (T), and encodes an input collection of text labels T = f1, , Im using a text encoder fix to text embeddings Vt, = (tx). Following (Radford et al. 2021 a text label 4 is framed as a sentence or caption contextualizing the image label yi, such as "a photo of a F-16A/B, a type of aireraft" to represent the image label "F-16A/B" in an airplane classification lask. The model compules logits for each label as cosine similarity between the image and text label, weighted by temperature (=100 in CLIP):
Vi Vi,
l(VI, Vo.) =
(1)
Logils can be converted to probabilities using a sofumax function:
exp(I(vI,
Pa = vill) ΣラT exp(l(vr, v(.))
(2)
The label with maximum probability is predicted.
3.2 Exemplar-based system
For the exemplar-based system, given one or more exemplars, our goal is to maximize classification performance with minimal training time and acceptable inference time. We consider two approaches: instance- based and model-based prediction. Instance-based prediction leverages the exemplar set directly by retrieving and comparing samples. Model-based prediction seeks to capture the underlying structure of the data through a parameterized model. In both cases, our approach leverages the embeddings from CLIP encoders to reduce storage and improve sample efficiency of learning.
Our exemplar memory module M stores encoded image embeddings and their labels. Each entry in the memory can be denoted by M, = [Vi,, by} where j represents the entry index, Vı, represents the image embedding of Lj, and y₁ is the corresponding label.
Given set of target labels, the exemplar- based model can produce a probability for each label Pe y4 vi) Alternatively, the prediction can be represented as an embedding vector v. of composition of one or several text embeddings. Prediction in terms of Ve enables the exemplar model to support zero-shot prediction for labels similar to exemplar labels.
Given VI, the KNN memory module finds its most similar k entries the memory through cosine similarities between V, and all Vi, in the memory. Let (2 be the set of indices of the k highest cosine similarity scores to Vr. KNN classification for V, can be performed by majority voling from the values: arg max, E)EN((v)) 12) y). Here, 11()) is an indicator function. The probability of v being label y is Pe = vllv7) = EvEN(( 1 [1;; = y))
KNN:
To predict the embodding Ve, we can use the text embedding of the most likely label. But we find that computing a similarity-weighted average of the retrieved text embeddings gives better performance: V. Ejer( By (ty), where 33 x exp(I(v), v1,)) and vi, is the image embedding the jth neighbor.
KNN takes virtually no time to train (O(1)), and reasonably fast retrieval is possible with optimized libraries and parallel computing. However, accuracy tends to be lower than model-based methods.
6

---

Published in Transactions on Machine Learning Research (10/2024)
Linear probe: Parameterized models offer an alternative approach, learning fixed set of parameters
that exploit the underlying structures of the data to maximize classification performance. The linear probe (LinProobe) method runs on extracted image embeddings and trains linear classifiers on all accumulated exemplars when new exemplars are added. The output probability can be written as: Pe (V = yalvi; = exp (0", vz)
2yjer. exp (0 vr) where Oy, represents the learned model parameters for label ys and Y. is the union of exemplar labels. The predicted embedding is the text embedding of the label 9 with maximum probability: )va()y)
Compared to KNN, the LinProbe model is much slower to train (n) for n training samples assuming a constant number of epochs, which may be prohibitive when the model needs to be updated quickly based on lew examples. However, classification accuracy tends to be higher.
Tree probe: In a continual learning setting, we would ideally have fast training time of KNN with the relatively good accuracy of LinProbe. We take inspiration from the instance-based and lazy learning literature (Aggarwal, 2014), particularly locally linear models (Domeniconi & Gunopulos) 2002 Atkeson ct al. 1997). Those methods classify test sample by finding its k-nearest neighbors and applying a linear classifier trained on the neighbors. This achieves ((1) training time but may be impractical for inference, since a new classifier may need to be trained for each test sample.
Instead, we propose an approximation, building a clustering tree from the training data and training linear classifier in each leaf node, as shown in Fig. 2 Starting from root node, we search for the nearest leaf node for new data point and insert it if the node has not reached the predefined node capacity th. If a leaf node reaches v, it splits into two child nodes and becomes non-leaf node. The attached data points are distributed to their children by KMeans clustering. In experiments, when receiving new data, samples are added into the cluster tree one by one. Only classifiers in affected leaf nodes need to be retrained. When fixing the number of linear model training epochs and KMeans iterations, the complexity to incorporate a new exemplar in training is Olp + log n) with vi >> log n; the training time stays limited even when the total number of exemplars is very large.
The simplest inference method would be to assign test sample to a leaf node in the cluster tree and classify it within the corresponding linear model, but this may lead to non-smooth predictions for samples near the cluster boundaries. In our experiments, we ensemble the classifiers from leaf nodes corresponding to the k nearest neighbors, such that the final output probability (v = y1lv1) is the average of the probabilities predicted by each neighbor's classifier.
Similar to KNN, the exemplar embedding V. can be computed as the text embedding of the most likely label from all neighbor classifiers. However, we obtain the best performance using weiihtte average embedding from the most likely label 8, of each classifier in the retrieval set: V. Ez 3, fia ), where 83 is similarly defined as in the KNN classification. The tree-probe method, denoted TreeProbe, achieves similar accuracy to Lioerobe in our continual learning experiments with sufficiently large v. but with much faster training time.
3.3 Fusing predictions of the two systems
We want to integrate predictions from the zero-shot and exemplar-based models to retain good open-vocabulary classification performance and improve accuracy in predicting exemplar labels. This can be tricky, especially when some labels in the task are not contained within the exemplar labels.
One approach is to simply average the probability or embedding predictions of the two models:
p(y vill)= ap. vlvv) + (1 aa)p. vrl
(3)
Vout ov.. (1 avyi
(4)
When setting a = 0.5 for an unweighted average, this approach denoted as Avg-Prob or Avg-Emb, respectively. Using an unweighted average presumes that the zero-shot and exemplar models are equally reliable for all samples. However, if the test sample's label is within the exemplar's domain, the exemplar model will tend to
7

---

Published in Transactions on Machine Learning Research (10/2024)
be accurate; otherwise, the zero-shot model will almost certainly outperform. The test label is unknown, but an educated guess could enable better prediction.
Addressing this issue, we devise an adaptive weighting mechanism, named Adaptive Instance Marginalization (AIM), that estimates the likelihood of a test sample's label being in the exemplar set and balances the predictions from both models accordingly. The target label set. divided into exemplar y Y. and non-exemplar y Y, subsets, where Ye the union of all exemplar labels.
The likelihoods ply Y.I and ply & Yel subsets: p(y E T) = E,ey, P=(y y: 7) and p(y & YelT) Σiey, P=(y y: f) 1 p(u E Y.IT), with the summation in the denominator of Eq. 2 over all candidate labels for the current task. Naturally, we set a = p(y E Y) in Eq. 4 which capitalizes on the strengths of both models, improving overall performance.
obtained by summing the zero-shot probabilities over these
Our final AIM-Prob method further encodes that both zero-shot and exemplar models are predictive for exemplar classes, while only the zero-shot model can be trusted for non-exemplar classes:
= (y vvlv))
ply : plu Yell) x Poly.Poly 1) (y = v,lvi) + ply Yell) p yI
(5)
4 Experimental setup
4.1 Tasks
Target tasks
-Zero-shot tasks
TOVEArcnat
X530
Bosinn 207
Erbtr evarnal
Sa MAOIE
Ice CTAUT
Task incremental.
Stage
Stage
Stage
Stage StanfordCars
FOVCArceaft
OxfordiliPetss
Food101
A39J
A393
BUeT AUY
Samvoyed
Haydeagey
Nasccapp
Suan
Class incremental
Data incremental
Stage
Stage
Stage
Figure 3: The upper row illustrates several randomly selected samples in target tasks and zero-shot tasks. In the middle and the lower row, we illustrate how data samples are organized in task, class and data incremental learning scenarios. Borders of images with different colors indicates the source of the data. Rod class names in data and class incremental learning are used to highlight differences of the two settings.
We evaluate our system on target and zero-shot classification tasks, A "task" is an assignment an image into label from particular label set. The exemplar model has received examples in the context of "target" tasks but "zero-shot" tasks. We utilize general tasks such ImageNet (Russakovsky et al.) 2015). SUN397 (Xiao) et al. 2010), CIFAR100 (Krizhevsky & Hinton 2009). and fine-grained tasks like FuroSAT (Helber al.
8

---

Published in Transactions on Machine Learning Research (10/2024)
2019
Oxfors (Prrkhi et al. 2012). DTD Cimpoi et al., 2014). Flower102 NNilsback & Zisserman
2008), FGVCAircraft (Maji et al., 2013), StanfordCars (Krause et al. 2013), Food101 Bossard et al. 2014).
CCF101 Soomro et al. 2012).
We report main results using target tasks CIEAR100, SUN397, FGVCAircraft, EuroSAT, OxfordllITPets, StanfordCars, Food101 and Flowers102 and zero-shot tasks ImageNet, UCF101, and DTD. All target tasks collectively provide > 220k samples in total, which is sufficient to give a reliable assessment of different methods. Several randomly selected samples of some tasks are displayed at Fig. 3 The supplemental material contains dataset descriptions, prompt templates, zero-shot performance of tasks, and additional experimental results and analysis.
4.2 Evaluation scenarios
We consider several continual learning scenarios for receiving data: 1) Data incremental: A fraction of the training data, randomly sampled without enforcing class balance, is added in each stage; 2) Class incremental: All training data for a randomly sampled subset of classes are added in each stage; 3) Task incremental: All data for single task, i.c. a datasct of examples assigned to a set of target labels, are added in each stage. Data incremental learning includes seven stages, comprising 2%, 4%, 8%, 16%, 32%, 64%, and 100% of task data, respectively. Class incremental learning divides task into five stages, each containing 20% of classes. In task incremental learning, each task is considered a stage. In data and class incremental experiments, models are built separately for each target task. A target task is fully evaluated if there is at least one training sample for that lask, even if there are no training samples for some classes. Tn task incremental, one model is built spanning all accumulated labels in each stage. In all cases, results are reported as the average accuracy of target and zero-shot tasks cach stage. We additionally compute the averaged accuracy on seen classes and unseen classes of each larget task for class incremental learning to give more detailed result analysis. An intuitive demonstration of these continual learning scenarios are shown in Fig. 3
Flexible Inference: Zero-shot
Flexible Inference: Union Zero-shot
Flexible Inference: Mix Zero shot
Conchace Miei Ieei matic
Conflaane lahess
Union target labels roacarooal of Octoed Part trel Rood IC1 Lebate
laaeb
Uneon bobcc
Union www. lebels OTO label LCT101
latel
Sanpled zoro-chet task
Figure 4: Demonstration of the three inference scenarios of flexible inference. Images and visual elements are consistent with Fig. 3. Bold text in green shows the final accuracy obtained from each scenario.
Flexible inference: In the task incremental setting, after all training data for target tasks is received, we will evaluate each method's performance in three inference scenarios:
Zero-shot: evaluate cach zero-shot task and then average the accuracy all tasks. Under this scenario, higher performance of model than CLIP would indicate positive transfer from learned tasks to zero-shot tasks, while lower performance would suggest forgetting has occurred.
Union + Zero-shot: we first create a union of target task labels that is considered as candidate labels for each target task. Then, we randomly sample 100 labels from the union of zero-shot task labels. We separately compute the average accuracies the target and zero-shot tasks. The final performance is the average of the target and zero-shot accuracy.
Mix + Zero-shot: we randomly split the union of target labels into five splits. For each split, we additionally add a random sample of 100 labels from the zero-shot task. To balance evaluation across classes, draw 100 test samples per class. Performance is the average across all splits.
9

---

Published in Transactions on Machine Learning Research (10/2024)
Fig. 1 gives more intuitive demonstration of these three inference scenarios. These scenarios are especially challenging for most continual learning methods because there is no task identifier at inference time, and the label of a test sample will sometimes be among the trained classes and sometimes not.
5 Experimental results
5.1 Effectiveness of complementary system and AIM
Hatbods
Drazt
Mellies
Mathods
Unssen
Fieeus
CUP Zenoathtt
PiPaaaa Jugy Cna
CuPte shrobe E18
CL CLPv (A M-Ori
Poccnagee of Andl abcc Duta
"escolagg: of Leamed Cuasea 151
torget
lraming
Figure 5: (a) Results comparing CLIP zero-shot, LinProbe and LinpoeinProbe with Avg-Emb and AIM-Emb on target tasks under the data incremental learning scenario. (b) Results of corresponding models on target. tasks under class incremental learning. We visualize curves for seen and unseen classes including the overall performance identified with different markers.
(c) Results of corresponding models on target tasks under task incremental learning, along with the performance of fine-tuning the whole CLIP network (CLIP Fine-tune) and ZSCL (Zheng et al. 2023). (d) Flexible inference results after task incremental learning on all tasks. Note LinProbe is hard to see in (a) and (b) because it has similar results as CLiP+LinProoe (Arg-Emb).
We first investigate the importance of complementary learning systems and the effectiveness of AIM. We design experiments comparing CL.P Zero-shot model, exemplar model using Lin and then complementary models that incorporate CLIP Zero-shot and LinProbe using the Avg-Emb and AIM-Emb fusing operations, denoted by CinPooinProb (Avg-Emb) and CLip+LinProbe (AIM-Emb).
The results on target tasks under data, class, and task incremental learning scenario are shown in Fig. (a-c)- For data incremental, except for CLIP zero-shot, all other methods constantly improve target tasks, with CLIP+LinProd (AIM-Emb) being the best on all stages. When the percentage of available data is larger than 20%, all other models surpass the zero-shot model. LinProbe and CLip+LinProbe (Avg- Bmng have similar performances on target tasks, which can also be observed in the class incremental lcarning results in Fig. (b).
Results under class incremental learning are crucial as this setup simulates scenarios where exemplar labels do not encompass all candidate labels. In such cases, we cannot simply decide between using the exemplar model zero-shot model for prediction by checking if exemplar labels fully overlap with candidate labels. In Fig. 5 (b), we plot the accuracies on seen and unseen classes along with the overall accuracy on all classes. The zero-shot model underperforms other methods on seen classes but excels in unseen classes, LinProbe achieves high accuracy on seen classes but performs poorly on unseen ones, with accuracy close to O. ELIP+LinProd (Avg-Fanb) shows similar results. The ATM-Fabb version has slightly lower accuracy than the Avg- version on soen classes in early stages (<60% classes) but maintains reasonable performance on unseen classes, particularly in earlier stages. Its unseen accuracy decreases as more classes are learned because AIM gradually favors the exemplar model. Overall, CLIP+LinProbs AAM-Emb) demonstrates the highest overall accuracy among all methods, highlighting the effectiveness of AIM.
In Fig. (c) & (d) under the lask-incremental learning and flexible inference scenario, in addition to the models mentioned above, we also compare with ZSCL (Zheng et al., 2023). , For the implementation, we try to maintain identical training techniques and hyperparameters as the original codes of ZSCL, and we will release codes for sanity chock with written details in the supplemental material. Furthermore, we train CLIP by fine-tuning all its parameters except. for the logit scale on target datasets, resulting in stronger exemplar
10

---

Published in Transactions on Machine Learning Research (10/2024)
model- Fine-tune. As side-to-side comparison with ZSCL, we only use the data of the current stage to train this model but without additional reference datasets used in 7SCL.
From the results in Fig. 5 (c), the AIM approach steadily shows better performance than the zero-shot model at all stages. CLIP Fine-tune, LinProbe and CLIP+LinProbe (Avg-Emb) can only beat the zero-shot model after the 6-th stage (FGVCAireraftt. However, CLIP Fine-tune better than LinProbe and CI.IP+LinProbe Avgg-Em) in early stages, implying that the loss of generalization to other tasks is less severe. ZSCL achieves relatively close performance as
(AIM-Emb) in early stages but lags behind quickly in latter (~ -6% off in the final stage). Overall, CLIP+LinProbs (AIM-Emb) has the best performance across all stages and show better performance than exemplar-nny models, demonstrating the benefits of complementary systems.
As shown in Fig. (d), exemplar-only models, being biased towards learned classes, are generally incompetent on zero-shot tasks. LinProbe extremely bad on zero-shot tasks, leading to poor performance on both Zero-shot and Z scenarios where zero-shot lasks are evaluated. However, CLIP+LinProbe (Avg-Emb) shows considerable improvement over LinProbe in Zero-shot and Mer--sho scenarios while remaining comparable in Uzion+Zero-sho.. CLIP Fine-tune received less severe impact of biased learning to target classes but still has a decent decrease under Zero shot. When replacing the fusing operation with ATM-Emb, performances across all scenarios significantly improve. Note that 7SCL, also has good Zero-shot performance, as mentioned in its paper, and obtains the best -nion+Zero-sll performance in our evaluation. Our method is inferior to ZSCL under the ZeronoZeo--sho scenario mainly because AIM likely assigns more weights to the exemplar model due to the large portion of in-exemplar labels (around 91%) of all candidate labels. Overall, our system shows decent flexible inference ability with the effectiveness of the complementary systems well validated.
5.2
Comparison of Efficiency and Accuracy for campplar-based Approaches
CLIP-KNN
Mathods
Time (poate teeery (ecordo (hg soal)
Unfooage
Accustty on tarpe cas the fmal stage
UnProee
CLP Trachole [3k]
30
Trcohobc
75
CUP+ PreeProbe
CUP- Trechooee
Testrate
Trachole
Accurscy
10*
TreeProee (50k)
Thee
treenobe (100K)
10t-
igo
ecohrab (2
65
220
mecehooe
00
S0N097
BussdA
Oetonaaite bets FocMuiccaat
Rewenss07
Number of Instances Memory llog scalet
10a
10a
10
Trining Time 10gg scales
207
Taskinncrmenntal leamning results on target tasks
(b) to one example TreaProbse and LinProbe
(c) Alccuraayy vs. training time for exemplan models
Figure 6: (a) Results of different complementary models under the inssemnncremtttll learning scenario. Numbers included in a bracket are the node capacity of corresponding TreeProbe models. (b) We perform an efficiency analysis of TreeProbe with different node capacities and LinProbe by estimating how much time needed to incorporate one example into the exemplar model with different number of instances carried in the exemplar set. To minimize randomness, each method is run five times. At each data collection point, we sample an example from the exemplar set and fit it to memory five times. Thus, cach data point averages unique simulations. and y-axes are visualized in log scale. (c) The accuracies of different complementary models on target tasks after learning the final stage (g-axis) us. the total training time in log scale (z-axis).
In this part, we perform a thorough analysis of different complementary models using AIM on their perfor- mances and efficiency: CLIP+KNN, CLIP+LinProie and capacities of TreeProbe to create different versions of
We can also vary the node models.
We compare their performance under the task incremental learning scenario. Results in Fig. (a) show that node capacity can be used to control the performance of CLIP TreePrreero When node capacity exceeds 5k, TroeProbe steadily outperforms KNN, and when it reaches 100k, TreeProbe slightly surpasses LinProbe Probe.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Fig. 6 (b) illustrates that LinProbe adheres to linear learning complexity the mumber of instances in the exemplar set increases. When node capacity is not reached, TreeProbe only has one classifier, thus identical to LinProbe in learning complexity, After reaching node capacity, TreeProbe exhibits constant learning complexity. TreeProbe (100k) and (50k) achieve similar performance to LinProbe but with 6x and 25x improvement in learning speed. This corroborates our analysis the time complexity of learning for different.
exemplar models in Sec. 3.2 Fig. (c) combines the results of Fig. 0 (a) and Fig. G (b), showing the trade-off
of learning efficiency and accuracy. The TreeProbe method, therefore, offers flexibilitty in selecting the node capacity that appropriately balances responsiveness with performance for specific application.
We also compare the learning efficiency of our methods with CLIP Fine-tune and ZSCL (Zheng et al. 2023 and summarize the training time in Tab. [ Since ZSCL and CLIP Fine-tune require GPU accelerated training, we also implement GPU-versions of TreeProbe and LinProbe for better comparison. Since our methods only operate on the image embeddings, we can cache the embeddings for new samples and store for expedited replay or multi-cpoch training. For both our models, we train the linear classifiers for 20 epochs, using hyperparameters selected from hyperparameter sweep. The evaluation is taken using the task incremental learning scenario and the training time is accumulated for all eight target tasks. From the table, after taking the processing time to get image embeddings in account, our approaches are still much faster than CLIP Fine-tue tune and ZSCL. Note that to mitigate forgetting, ZSCL needs to refer to the outputs of frozen CL.IP to calculate distillation losses, making it around two times slower than CLIP Fine-tune. Besides training efficiency, our approaches are more effective through the final averaged accuracy over all target tasks. This test further reinforces the training efficiency and effectiveness of TreeProbe.
Method
Training Time (s)
A
Avg. Acc. (%)
A
TreeProbe (50k)
239.1
0.0
80.2
0.0
LinProbe
256.6
+17.5
80.4
+0.2
CLIP Fine-tune
2542.7
x10.6
74.9
-5.3
ZSCL
5409.3
x22.6
74.3
-5.9
Table 1: Efficiency comparison of different methods. Training Time is mcasured in seconds, and Avg. Acc. is the averaged accuracy over all larget lasks evaluated at the end of inkkinncreeetlll learning. A shows the difference from the baseline method TreeProbe (50k) ("x" means times). For equal comparison using the same GPU, the linear layers in TreeProbe and LinProbe are implemented using PyTorch's 's linear layer (i.c, torch. nn. Linear) to support GPU accelerated training.
5.3 Comparison to previous methods
Method
Transfer
A
Avg.
A
Last
A
CLIP Zero-shot
69.4
0.0
65.3
0.0
65.3
0.0
LwF (Li & Hoiem 2016
56.9
-12.5
64.7
-0.6
74.6
+9.3
iCaRL. RRebulfi et al.
2017
50.4
-19.0
65.7
+0.4
80.1
+14.8
WiSE-F Wortsman et al.
2022a
52.3
-17.1
60.7
-4.6
77.7
+12.4
7SCL, (Zheng et al.; 2023
68.1
-1.3
75.4
+10.1
83.6
+18.3
TreeProbe (50k)
69.3
-0.1
75.9
+10.6
85.5
+0022
Table 2: Comparison of different methods on MTIL Order T from 7SCT. Zheng al. 2023 All results are taken from other ZSCL's paper except for TreeProbe. "Transfer" evaluates the model's 's performance on zero-shot tasks; "Last" is the averaged accuracy all target tasks after finishing the final task while "Avg." computes the average task performance on all training stages.
As introduced in Sec. 2.2 ZSCL Zheng et al. 2023) explores a similar setting to open- vocabulcyy contimual learning and mainly evaluated under the ins-eineremental learning scenario. To ensure a fair comparison, we evaluate our approach using their evaluation protocols, employing the same prompt ensemble technique, data split ratio and seed, backbone network (ViT-B/16) /6)), and other relevant parameters. The results are
12

---

Published in Transactions on Machine Learning Research (10/2024)
presented in Tab. Our TreeProbe (50k) model achieves the best performance across all metries. Compared to the CLIP vero-shot model, our method barely compromises Transfer, demonstrating robust performance on zero-shot tasks. Notably, our method is more lightweight in GPU consumption than ZSCL. Additionally, we do not perform hyperparameter selection for cach task, as we assume no prior knowledge of future tasks in practice, whereas 7SCL, used different learning rates for different lasks oplimize performance.
5.4 Scaling to larger zero-shot models
ViT-B/32
VIT-L/140336px
ViT-H/14
Method
Target
Zero-shot
Target
Zero-shot
Target
Zero-shot
CLIP Zero-shot
60.2
54.0
72.1
65.6
79.2
71.7
TreeProbe (50k)
80.5
53.8
86.6
65.4
90.2
71.6
Table 3:
Accuracies of CLIP Zero-shot and TreeProbe target and zero-shot tasks with pretrained image encoders of different capacities.
We use the CLIP VIT/B-33 model as our default zero-shot model but can easily adapt to other models. We experiment with the pre- trairraied CLIP model VIT-L/140336ps which boasts approximately 4x the capacity of ViT-B/32, and ViT-H/14 which is double the size of ViT-L/144336pp We present our findings in Tab. where both methods are evaluated under the task incremental learning scenario. Performance improves for larger models, and TreeProbe consistently outperforms CLIP Zero-shot on the larget lasks with nearly identical performance on zero-shot tasks. This demonstrates that our approach is beneficial across wide range of zero-shot model sizes.
6 Conclusion and limitations
In this work, we present an elficient and performant "tree probe" exemplar-based model and Adaptive Instance Marginalization to combine zero-shot and exemplar models for open-vocabuse voca-vocyy continual learning. Our method is able to efficiently incorporate new samples, improving performance for related tasks without negatively impacting zero-shot task performance, providing simple way to continually improve large-scale pretrained image-text classification models. We believe this work is step towards more flexible, efficient, and effective strategies in open-vocabulary continual learning.
Our work has several limitations. First, we do not consider constraints how many exemplars can be stored. Since each exemplar requires storing up to 4KB for the image and text encodings (using the base CLIP ViT-B/32 model without any compression limiting precision), roughly one million exemplars can be stored per 4GB of memory. Second, we do not investigate how to improve the consolidated zero-shot model. This is a promising direction for future work. Finally, we do not explore structured prediction problems like semantic sogmentation or visual question answering, which likely require more complex image representations than a single embedding vector.
Acknowledgments
This work is supported in part by ONR award N00014-21-1-2705, ONR award N00014-23-1-2383, and U.S. DARPA ECOLE Program No. #HR00112390060. The views and conclusions contained herein those of the authors and should not be interpreted necessarily representing the official policies, either expressed or implied, of DARPA, ONR, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Broader Impact Statement
This paper presents an efficient and flexible continual learning system for open-vocabulary image classification. Our method may reduce the computational cost of extending and improving large foundation models, which could reduce the carbon footprint of development and facilitate new applications. As with other image
13

---

Published in Transactions on Machine Learning Research (10/2024)
classification approaches, method is dependent training data and annotations, which may be biased non-representative of deployed use cases.
References
Charu C. Aggarwal. Instance-basee learning A survey. In Data Classification: Algorithms and Applications, chapter 6. CRC Press, 2014.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob Mcnick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14195 2022.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In CVPR, pp. 7120-7129, 2017.
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based complementary learning system. In ICLR, 2022.
Christopher G. Atkeson, Andrew W. Moore, and Stefan Schaal. Locally weighted learning. Intelligence Review, 11:11 73, 1997.
Artificial
Ali Ayub and Alan R. Wagner. CBCT,PR: A cognitively inspired model for class-incremental learning robotics. IEEE Trans. Cogn. Dev. Syst., 15(4):2004 2013, 2023.
Jihwan Bang, Heesu Kim, Youngjoon Yoo, Jung- Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with memory of diverse samples. In CVPR, pp. 8218 8227, 2021.
Gianluca Bontempi, Mauro Birattari, and Hugues Bersini. Lazy learning for local modelling and control design. International Journal of Control, 72(7-8)6643 658, 1999.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
Léon Bottou and Vladimir Vapnik. Local learning algorithms. Newral Comput., 4(6):888 900, 1992.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying iision-and-languaan tasks via text generation. 2021.
ArXiv,
M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proc. CVPR, 2014.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In ICCV, pp. 695-704, 2021.
C. Domeniconi and D. Gunopulos. Adaptive nearest neighbor classification using support vector machines. In NewrIPS, 2002.
Alexey Dosovilskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenRevicw.ng nett 2021.
Ali Farhadi, Ian Endres, Derck Hoiem, and David Forsyth. Describing objects by their attributes. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 1778-1785. IEEE, 2009.
Chandan Gautam, Sothupathy Parameswaran, Ashish Mishra, and Suresh Sundaram. Online lifelong generalized zero-shot learning. Neural networks, 155:487-497, 2021. BRRLhttps://// //app. semanticschollar htt.atppps/ppp...mmmnniccccaa
orr:CoorpusID::22297794
14

---

Published in Transactions on Machine Learning Research (10/2024)
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2113 12.221111111 12143,, 2021.
Xiuye Gu, Tsung Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arx preprint orXin:2104 13921, 2021.
Gongde Guo, Hui Wang, David Bell, Yaxin Bi, and Kieran Greer. Knn model-based approach in classification. In On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CooplS, DOA, and ODBASE 2003, Calania, Sicily, Haly, November 3-7, 2003. Proceedings, pp. 986 996. Springer, 2003.
Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture. In CVPR, 2022.
Jie Hao, Kaiyi Ji, and Mingrui Liu. Bilevel coresct selection in continual learning: A new formulation and algorithm. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), NeurIPS, 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remole Sensing, 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Dala, 2019.
Minsoo Jung, Seongmin Hong, and Jinwoo Shin. Generating instance-level prompts for rehearsal-free continual learning. In ICCV, 2023.
Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly supervised concept expansion for general purpose vision models. In ECCV, 2022.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In ICLR. OpenReview.nng net. 2018.
Hikmat Khan, Nidhal Carla Bouaynaya, and Ghulam Rasool. Remembering transformer for continual learning. arXiu, 2024.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsual Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, pp. 15144-15154, 2023.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming capastrophic forgetting in neural networks. CoRR,
2016.
Jonathan Krause, Michael Stark, Jia. Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), 2013.
Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Muster's thesis, Deportment of Computer Science, University of Toronto, 2009.
Christoph H. Tampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot. visual object categorization. IEEE PAMI, 36(3):453-465, 2014. doi: 10.109/TPAMI:20111400
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-cfficiet prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punla Cana, Dominicon Republic, 7-11 November, 2021, pp. 3045-3059, 2021.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Z. Li and D. Hoiem. Learning without forgetting. PAMI, 40(12):2935-29)) 2947, 2018.
Zhizhong Li and Derck Hoiem. Learning without forgetting. In Proc. ECCV, 2016.
Xuefeng Liang, Yixin Hu, and Yang Feng. Adaptive plasticity improvement for continual learning. In CVPR, 2023.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avit, Israel, October 23-27, 2022, Proceedings, Part XXXV, , pp. Springer, 2022.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In CVPR, pp. 2537-2546, 2019.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proc. CVPR, 2022.
David Lopez-Pas Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Proc. NeurIPS, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR. OpenReview.nee net, 2019.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. ArXiv, abs/2206.0890 08916, 2022.
Divyam Madaan, Hongxu Yin, Wonmin Byeon, Jan Kautz, and Pawlo Molchanov. Heterogeneous continual learning. In CVPR, pp. 15985-15999 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rX6511306.515,, 2013.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In CVPR, 2018
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109 165. Elsevier, 1989.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sizth Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
Randall C. O'Reilly, Rajan Bhattacharyya, Michael D. Howard, and Nicholas Ketz. Complementary learning systems. Cogn. Sei., 38(6):1229 1248, 2014.
German Ignacio Parisi, Jun Tani, Cornelius Weber, and Stefan Wormter. Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization. Fronliers Neurorobotics, 12:78, 2018
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Proc. CVPR, 2012.
Quang Pham, Chenghao Liu, and Steven C. H. Hoi. Dualnet: Continual learning, fast and slow. In Marc' Aurelio Ranzato, Alina Beygekimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (cds.), NewrIPS, pp. 16131-16144, 2021.
Ameya Prabhu, Philip Torr, and Puncet Dokania. Gdumb: A simple approach that questions our progress in continual learning. In The Europeon Conference on Computer Vision (ECCV), August 2020.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puncet K. Dokania, Philip H. S. Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? CoRR, abs/2303.1 1112003.116,, 11165, 2023.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In Proc. CVPR, 2017.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Foeles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv, hls//205.001175 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 115(3):211 252, 2015.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. 166.::6066.4000 2016.
Jameel Abdul Samadh, Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fa- had Shahbaz Khan, and Salman H. Khan. Align your prompts: Test-time prompting with distribution aligmment for zero-shot generalization. In NewrIPS, 2023.
Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. Sparse coding in a dual memory system for lifelong learning. Tn Brian Williams, Yiling Chen, and Jennifer Neville (eds.), AAAT, pp. 9714 97222 AAAT Press, 2023.
Joan Serrå, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proc. ICML, pp. 4555-4564, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay, In NewrIPS, pp. 2990-2999, 2017.
Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (continual)? generalized zero-shol learning. In International Conference on Learning Representations, 2021. URL
net/
forum?id=7ppfL22Dyy)
James Seale Smith, Loonid Karlinsky, Vyshnavi Gutta, Paola Cascanto- Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogério Ferís, and Zsolt. Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, pp. 11909 11919, 2023.
Yunsheng Song, Jiye Liang, Jing Lu, and Xingwang Zhao. An efficient instance selection algorithm for k
nearest neighbor regression. Newrocompuling, 251:26 34, 2017.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: dataset of 101 human actions classes from videos in the wild. CoRR, hhs/1212.0402 2012.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic reprosentations. In International Conference on Learning Representations, 2020. URL
Rishabh Tiwari, KrishnaTery Teja Killamsetty, Rishabh K. Tyer, and Pradeep Shenoy. GCR: gradient coreset.
based replay buffer selection for continual learning. In CVPR, pp. 99 19-108. IEEE, 2022.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. CoRR, abs/ hbs/2302.00487, 2023.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. In NewrIPS, 2022a.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pp. 631 648, 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Raoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pp. 139-149, 2022c.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR, pp. 7949-7961, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine- luning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 7959-79777 7971, 2022b.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proc. CVPR, 2010.
Shipeng Yan, Jiangwei Xie, and Xuming He. DER: dynamically expandable representation for class incremental learning. In Proc. CVPR, 2021.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arxiv preprint arv: 708.700.1111 01547, 2017.
Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal- bassed continual learning. In ICLR. OpenReview.net, 2022.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proc. ICML, pp. 3987-3995, 2017.
Hao Zhang, Alexander C Berg, Michael Maire, and Jitendra Malik. Svm-knn: Discriminative nearest neighbor classification for visual category recognition. Tn 2006 IEFE Computer Sociely Conference on Computer Vision and Pattern Recognition (CVPR '06, volume 2, pp. 2126 2136. IEEE, 2006.
Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tuning: A bascline for network adaptation via additive side networks. In ECCV, pp. 698-714, 2020.
Shichao Zhang, Xuelong Li, Ming Zong, Xiaofeng Zhu, and Debo Cheng. Learning for knn classification.
ACM Transactions on Intelligent Systems and Technology (TIST), 8(3):1-19, 2017.
Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin Scan Lim, and Yunzhe Jia. A simple but strong baseline for online continual learning: Repeated augmented rehearsal. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), NewrIPS, 2022.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. CoRR, abs/2303.06622 2023.
Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba, Places: A 10 million image database for scene rocognition. IEEE TPAMI, 4066):1452-1464, 2018
18

---

Published in Transactions on Machine Learning Research (10/2024)
A Appendix
S-1 Algorithmic descriptions of TreeProbe
A Algorithm Training Procedure of TreeProbe
Require: Training set X, Tree T, Leaf capacity vb
Ensure: Trained classifiers in each leaf node of T
1: for all V, E X do SVEREETTEAPPE 7)
if COUNT(() & then
1= INSERIDATA(V; I)
TRAINCIASSIFEER()
6: else
7:
SPLITNODE(L, v:)
8:
= NEARESTLEAE(V. T)
INSERT (V,
10:
TRAINCLASSIFIER()
11: end if
12: end for
Algorithm 2 Inference Procedure of TreeProbe
Require: Image embedding Vı, Tree T, Number of nearest nodes k, Exemplar set M
Ensure: Exemplar embedding Ve for Vy
1: Vк FNNDNEARETTSSSPPPESYVY... list.)
for all E VV do
= NEARESTLEAF(V,.2 T)
GETCLASSIFIER()
6:
v CLASSIFY(VI,C)
7: end for
8: V. = COMPUTEEABEDDING(Y, vy)
Algorithm D and Algorithu 2 contain psuedocode for the training and inference of our TreeProbe method.
Definitions of the involved functions are provided below:
Near T):
Returns the nearest leaf node the data point o; tree T.
Count((): Returns the current number of data points in leaf node (.
InscrtData(v,, I): Inserts data point v, into leaf node I and returns the updated node.
SplitNode(l, vz): Splits leaf node into two child nodes when it reaches capacity, distributes data points using KMeans clustering, and adds new data point to the appropriate child node.
TrainClassiffer(() Trains linear classifier on the data points in leaf node I.
FindNearesttSamllesvvy M): Finds the nearest neighbors in the exemplar set to the image embedding Vi.
Classifrer) Return the classifier for node 1.
Classify(v,, c): Classifies v, using the linear classifier c, returning the label embedding of the most likely label.
19

---

Published in Transactions on Machine Learning Research (10/2024)
ComputeEmbedding/w vi): Computes the exemplar embedding for Vi by applying similarity- weighted average of the text. embeddingg of the most likely class labels from temporal list
The notation of Algorithm | and Algorithm 2 may differ from the main paper.
S-2 Implementation details
We conduct our experiments on a setup featuring an RTX 3090 GPU and an AMD Ryzen 9 5950X CPU, using Py'Torc as our primary framework. We adhere the CLIP code example, using sklearn LogisticRegression. to implement linear classifiers and setting the sklearn regularization strength to 0.316. The maximum iteration is set to 5k. Our tree probe's node capacity is set at 50k. For efficient retrieval from large-scale exemplar sets, use FAISS [Johnson et al. 2019 specifically using the IndexFlat class for its precision and performance. Model performances are gauged via Top-1 accuracy, with the officially released VIT-B/32 CLIP checkpoint serving as our memory or zcro-shot model. We select ho = 9 based on a hyp pyperarramcter sweep. Our approach is not sensitive to k; with very similar performance in a range from to 30.
Additional implementation details of CLIP Fine-tune and ZSCL. Since we do not assumee have additional knowledge about data distributions, we choose the learning rate for both methods as the most frequent used learning rate in ZSCL experiments, i.e,, 1e-5. For each lask, following ZSCL, warmup training for 100 iterations and proceed to train 1K iterations. For the weight ensemble technique used in ZSCL, we also use an update interval of 100 iterations. Batch size is kept to 64 for both methods, with AdamW (Loshchilov Hutter 2019) optimizer and the beta set to 0.9. The base network backbone we use for both methods CLIP VBT-B/32, identical our approach.
Additional implementation details of GPU-version CeeeProbe/LinProbie After sweeping a good set of hyperparameters that lead to smaller number of training epochs while maintaining good performance held-out classification datasets (Caltech101), set learning rate to 0.005, optimizer to SGD, weight decay to 0.1, and use 20 epochs for training each stage. We use the cross-entropy loss for training the probes.
S-3 More experiment setting details
Class Incremental Learning
20%
40%
60%
80%
100%
Proportion of olasses
8%
32%
Incremental Leariing
6456
Numbor darasets
Proportion of data
Dataset
Datesel 2
Dainsot
Task Incremental Leaminng
Figure S1: Ilustration of continual learning scenarios. Data incremental learning includes seven stages, cach comprising 2%, 4%, 8%, 16%, 32%, 64%, and 100% of task data respectively. Class incremental learning divides a task into five stages, each containing 20% of classes. Tn task incremental learning, each task is considered a stage.
We present an illustration of data, class, and task incremental learning scenarios in Fig, ST When evaluating different methods in data and class incremental learning scenarios, we ensure fairness by randomly selecting an identical portion of data/clas, for all methods, achieved by setting the same seed. Models are separately built for each target task. The performance of each stage is averaged across all target tasks. In task-incremental
20

---

Published in Transactions on Machine Learning Research (10/2024)
learning, each stage is embodied by distinct task. The task order is randomly arranged CIFAR100, SUN397, FGVCAircraft, EuroSAT, OxfordIITTPPts, StanfordCars, Food101, and Flowers102, consistent for all methods for fair comparison. As shown in ZSCL (Zheng et al., 2023), task order has little impact on the relative performance comparison of different models. For all learning scenarios, we make the assumption that training data accumulates across all stages in all scenarios, with a similar spirit to (Prabhu et al. 2023). This assumption is based on the fact that real-world applications are often more limited by computational and time budgets than by storage. If data privacy is not concern, building a continual learning system without losing data access is more effective than assuming past data is non-accessible. Furthermore, enhance storage efficiency by saving samples condensed feature vectors, a significant improvement over some earlier works.
S-4 Descriptions of tasks
We perform experiments various commonly used visual datasets to demonstrate the generalization capabilities of our method. These datasets encompass broad range of image categories and application scenarios, including both fine-grained and generalized datasets. We briefly introduce all used tasks in this paper in the following.
S-4.1 General tasks
2015) contains 1,281,167 training images, 50,000 validation images and 100,000 test images. The categories represent a wide variety of objects, animals, scenes, and even abstract concepts. This dataset has served as a fundamental dataset to evaluate performances of classification models, or as pretraining dataset.
ImageNet Image.Net (Russakovsky et al.
CIFAR100 The CIFAR100 dataset (Krizhevsky & Hinton 2009) consists of object images and is a subset of the 80 million tiny images dataset. It contains 60,000 x2 32 color images from 100 object categories, with 600 images per category. The dataset has 100 fine-grained classes, grouped into 20 coarso-grained classes.
SUN397 The SUN397 dataset (Xiao et al. 2010 consists of scene images, containing 108,754 754 images across 397 scene categories, with each category having between 100 and 500 images. This dataset commonly used for scene understanding tasks. Since there is no official dataset split for this dataset, we randomly select 60% of images as training data, 20% as validation data, and the rest as test data. We use NumPy random permutation split with the seed set to 0.
S-4.2 Fine-grained tasks
FGVCAircraft The FGVCAireraft dataset Maaji al. 2013) serves as a benchmark for fine-grained visual categorization of aircraft. It contains 10,200 images from 102 distinct categories. Each category includes approximately 100 images, annotated with the aircraft model, variant, and manufacturer.
DTD The Describable Textures Dataset (DTD) Cimpoi et al. 2014 consists of 5,640 images across 47 texture categories, with each category featuring 120 real-world texture images such as fabrics, rocks, and surfaces. The dataset poses a challenge for texture classification due to subtle differences between textures within the same category and large variations in texture appearance caused by scale, orientation, and lighting.
Food101 The Food-101 dataset (Bossard et al. 2014) comprises 101,000 images across 101 food categories, each with 1,000 images. This dataset challenges fine- tne-ggrined image classification due to high intra-class variation and visual similaritics across categories. It serves as rigorous benchmark for evaluating computer vision models in food recognition and provides a robust platform for training machine learning models in understanding culinary aesthetics and preferences.
StanfordCars The StanfordCars dataset (Krause al. 2013 is a benchmark dataset containing 16,185 images from 196 different car classes, divided into 50-50 training and testing split. The classes correspond to specific car makes, models, and years, such the 2012 Tesla Model S or 2012 BMW M3 coupe.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Flowers102 The 102 Category Flower Dataset Nilssbcc & Zisserman 2008) is compilation of flower images. It includes 8,189 images across 102 flower categories, with each category containing between 40 and 258 images. The dataset's 's images vary in size and aspect ratio, captured using different cameras, lighting conditions, and backgrounds.
OxfordIlIPPets The OxfordllITPets dataset (Parkhi al. 2012) is a collection of pet images, featuring 7,349 images from 37 different cat and dog breeds. Each breed has between 100 and 200 images. The dataset is challenging because the appearance of the same breed can vary significantly, and different breeds may have similar-looking features.
EuroSAT The EuroSAT dataset (Helber et al. 2019) is a remote sensing image dataset comprising Sentinel-2 satellite data. It contains 27,000 images that cover 13 spectral bands and consist of 10 different land use and land cover categories, including forests, urban areas, and water bodies. This dataset is commonly employed for remote sensing and land cover classification tasks. Since there is no official datasct split for this datasct, randomly select 70% of images as training data and the rest validation data. We use NumPy random permutation to perform splitting with the seed set to 0.
UCF101 The UCF101 dataset (Soomro et al. 2012 is commonly used benchmark for action recognition. It consists of 13,320 videos from 101 action categories, with each category containing at least 100 videos. The actions include a wide range of human activities such as basketball shooting, horse riding, and juggling. The dataset is unique in its focus on complex, naturalistic action sequences, with videos varying in length from a few seconds to a minute. Since there is no official dataset split for this dataset, we randomly select 70% of images as training data and the rest validation data. We use NumPy random permutation to perform splitting with the seed set to 0.
S-4.3 Long-tailed task
Placcs365LT Places365LT (Lu et al. 2019) a synthetic long-tail derivative of Places2 datasct (Zhou et al., 2018). The image resolution is 256 256. It contains 365 scene classes with at least. 5 samples each. The classes are not uniformly distributed, forming long-tailed distribution. It contains some label noises, making classification even harder on this dataset.
S-5 Prompt templates for tasks
Tssk(s)
Prompt template
ImageNet, CIFARIOO, SUN397
"a photo of a [label)."
FGVCAircraft
"a photo of a {label), a type of aircraft."
DTD
"a photo of a (label} texture."
StanfordCars
"a photo of a (label), a type of car."
101
"a photo of (label), a type of food."
Flowers102
"a photo of a {label}, a type of flower."
OxfordllITPets
"a photo of a {label), a type of pet."
EuroSAT
"a centered satellite photo of (label)."
UCF101
"a video of a person doing (label)."
Places365LT
photo of the (label), a type of place."
Table S1: Prompts of tasks
CLIP (Radford et al.] 2021) suggests utilizing sentence template (e.g., "A photo of a (label).") "), as input to the text decoder instead of a plain text label, due to its training data being primarily full sentences describing images. Consistent with this paper's focus, we employ a simple prompt template for each task. Most of these templates are based on CLIP's recommendation and are sumnariied in Tab. S1
22

---

Published in Transactions on Machine Learning Research (10/2024)
S-6 Zero-shot performances on different tasks
Task
Zero-shot Acc (%)
Official ZS Acc (%)
ImageNet Russakovsky et al.
2015
59.7
63.2
CIFAR100 (Krizhessky & Hinton
2009
62.3
65.1
SUN397 (Xiao et al.] 2010
59.2
63.2
FGVCAírcraft (Maji et al.
2013)
18.1
21.2
DTD (Cimpoi al.] 2014)
42.0
44.5
StanfordCars (Krause al.
2013
58.6
59.4 4
Food101 (Bossard et al. 20014
82.6
84.4
Flowers 02 (Nilsback & Zisserman,
2008
67.9
66.7
Oxford (Parkhi et al. 2012
87.5
87.0
EuroSAT (Helber et al. 20199
45.4
49.4
UCF101 (Parkhi et al. 2012
60.1
64.5
Places365IT (Liiu et al.
2019)
40.0
Table S2: Zero-shot performances CLIP ViT-B/32 pretrained model on different tasks. "ZS" for zero-shot and " "Ace" for accuracy. Results of column "Official ZS Acc" are taken from the CLIP original paper (Radfodd et al. 2021). */" represents lack of official results.
Tab. S2 shows the zero-shot performance of our implementation in different tasks. We conjecture that the main difference of official zero-shot performances comes from the ensemble prompt trick mentioned in CLIP Radford al. 2021) and randomness in dataset splits of several tasks (e.g., SUN397).
S-7
Additional results
40
40-
Accurccy (%)
30-
20
CLP Zero-shot
Mechoch
LiProbe
Zero shot
Zero-thor
Methods
10
LinProbe
Linttaee (Avg-Emb)
UiPooe (tug Embi Lirtrobe (Alk-am))
CLIP LinProbe (Avg
LinPrabe Embl)
20
LinPoLn (AIM.Emh)
80
100
20
Percentage
50
80
OFARLO EuroSAT
mndordiiins focvMrecalt FOC df Aheeoo
Percentage Avallable
1%))
shot parformance
leaming
Zaro performance incremental leamning
(c) Zero-shat periormance
leaming
Figure S2: Zero-shot performances on data (a), class (b), and task (c) incremental learning scenarios. r-axis represents stages when the evaluation is performed.
S-7.1 Zero-shot task performances of every stage
We further append the results of zero-shot tasks evaluated after each stage on the data, class, and task incremental learning in Fig. S2 Across all scenarios, AIM-Emb helps maintain the zero-shot performances on all stages.
23

---

Published in Transactions on Machine Learning Research (10/2024)
Caltech101
CIFAR100
OxfordPet
Aireraft
EuroSAT
Flowers
MNIST
SUN39F
DTD
Food
Cars
Transfer
87.90
68.22
45.32
54.61
71.08
88.86
59.45
89.07
64.61
64.05
69.3
Aircraft
52.45
87.90
68.22
45.32
54.61
71.08
88.86
59.45
89.07
64.61
64.05
Caltech101
52.48
96.89
68.22
45.32
54.61
71.08
88.86
59.45
89.07
64.61
64.05
CIFAR100
52.48
96.89
68.22
45.32
54.61
71.08
88.86
59.45
89.07
64.61
64.05
DTD
52.42
96.83
81.98
70.32
54.61
71.08
88.86
59.45
89.07
64.61
64.05
EuroSAT
52.45
89.92
81.99
66.65
95.74
71.08
88.86
59.45
89.07
64.61
64.05
Flowers
52.51
90.55
81.93
66.44
95.74
54.12
88.86
59.45
89.07
64.61
64.05
Food
52.48
90.78
81.94
67.23
95.78
65.49
92.25
59.45
89.07
64.61
64.05
MNIST
52.09
93.95
81.96
69.73
94.33
95.69
92.27
98.59
89.07
64.61
64.05
OxfordPet
52.63
95.10
82.05
70.05
95.63
95.69
92.29
98.58
92.91
64.61
64.05
Cars
52.54
95.22
81.94
67.98
94.15
95.59
92.29
98.58
93.02
86.27
64.05
SUN397
52.48
95.56
81.94
66.91
95.59
95.59
92.21
98.60
93.19
86.15
81.76
85.5
Avg.
52.45
93.59
79.47
61.93
80.49
77.96
90.40
73.68
90.15
68.53
65.66
75.9
Table S4: Accuracy (%) of our TreeProbe (50k) model the MTIL benchmark with order-1. Each row represents the performance every dataset of the model trained after the corresponding task. Transfer, Avg., and Last metrics are shown in color. We follow the same table arrangement as in ZSCL (Zheng et al. 2023
S-7.2 More results of the comparison to previous methods
Method
Transfer
Avg.
Last
A
CLIP Zero-shot
69.4
0.0
65.3
0.0
65.3
0.0
LwF (Li & Hoiem 2016
56.9
-12.5
64.7
-0.6
74.6
+9.3
iCaRL Rebuffi et al. 2017
50.4
-19.0
65.7
+0.4
80.1
+14.8
WiSE-FIT Wortsman et
2022a
52.3
-17.1
60.7
-4.6
77.7
+12.4
ZSCL (Zheng et al. 2023
68.1
-1.3
75.4
+10.1
83.6
+18.3
KNN
69.3
-0.1
72.7
+7.4
78.4
+13.1
LinProbe
69.3
-0.1
77.1
+11.8
86.0
+20.7
TreeProbe (50k)
69.3
-0.1
75.9
+10.6
85.5
+20.2
Table S3: Comparison of different methods MTIL in Order from ZSCL (Zheng et al. 2023
KNN,
LinProbe, and TreeProbe (50k) are complementary methods with AIM-Emb as the fusing approach.
We also supplement the results obtained by KNN and LinProbe while comparing to previous methods in Tab. 53 As shown, all of the approaches of complementary systems with AIM-Emb achieve good Transfer, reiterating the effectiveness of AIM. LinProbe excels al all metrics with the cost of efficiency, which is predictable from the results shown in our main manuscript.
To give more details on the accuracies we achieve on every task under all stages for TreoProbe (50k), we follow ZSCL. (Zheng al. 2023 give all numbers Tab. S4 for further reference.
S-8 Additional ablation experiments
Effect different fusing operations. We describe several forms of the fusing operations in Sec. 3.3 including: Avg-Prob, AIM-Prob, Avg-Emb, and AIM-Emb. We compare using TrecProbe (50k) under the task incremental learning scenario. Fig. [S3 shows the results on target and zero-shot tasks. The figure shows that ATM-Fmb and ATM-Prob have similar performance target tasks, surpassing the other two
24

---

Published in Transactions on Machine Learning Research (10/2024)
55
80
50
70
45
Accuracy (%)
60
Accuracy (%)
40
35
Methods
Methods
30
Avg-Prob
Avg-Prod
Avg-Emb
Avg-Emb
25
30
AIM-Pro Pro-
AIM-Prob
AIM-Emb
AIM-Emb
20
CIFAR100 SUN397
EUroSAT
OxfordilITPets
Flowers102
FGCVAircraft
CIFAR100
SUN397
EuroSAT
OxfordilITPets
Flowers102
FGCVAircraft
StanfordGafsd10r
Figure S3: Target (left) and zero-shot (right) accuracies of different fusing operations.
fusing operations. Combined with the zero-shot performance, the results suggest the effectiveness of AIM in adaptively choosing the better prediction model. The probabilistic prediction is better than the embedding prediction when performing averaging both target tasks and zero-shot tasks. But when combined with AIM, the embedding version has a reasonably better performance zero-shot lasks. Therefore, we choose ATM-Fmb the default fusing operation.
50
45
40
35
Accuracy
25
Methods
Target
20
Zero-shot
15
MV-KNN
AVG-KNN
10
WAVG-KNN
5
10
30
Figure 84: Results of different. versions of KNN on larget lasks after finishing all stages under the task incremental learning scenario. We further ablate on k selection so choose & the æ-axis. The vertical dashed line represents k 9.
Different versions of KNN and choice of k. As indicated in Sec. 3.2, the attached to the most likely label (MV-XNN), or averaging the text embeddings of the k nearest neighbors (AVG-KNN), or performing weighted-averaging over the embeddings of all & ncarest neighbors where weights come from the sinilarities between the image embedding and the k neighbors' image embeddings (WAVG-KNN). We also compare these approaches under different ks to further choose a suitable k for experiments. As indicated Fig. S4 from the curve, and zero-shot tasks. Compared to AVG-KNN, larger & is more beneficial for MV-XNN since larger more stable for MV-KNM, and it more likely to include more mismatches from the nearest neighbors forAVG-KNN
get the embedding by taking
gives reasonable performances of all approaches on both the target
25

---

Published in Transactions on Machine Learning Research (10/2024)
From the plot, clearly read that WAVG-KNN is consistently better than AVG-KNN and MV-KNN across different ks, making it our default option of the prediction approach for fast learning system.
76
53.90
74-
53.85
72
53.80
Accuracy (%)
70-
Accuracy (%)
53.75
68
53.70
66
53.65
64
53.60
Methods TreeProbe w/o ensem. cls.
Methods
62
53.55
TreeProbe w/o ensem, cls.
60
TreeProbe
53.50
TreeProbe
CIFAR100 SUN397
EuroSAT
OxfordilTPPes
Flowers Flowers102
FGCVAircraft Caro
Stanford
1ood101
CIFAR100 SUN399
SUN397
EuroSAT
OxfordilIPPets
Flowers Flowers102
FGCVAircraft
Nannorddarra
Figure $5: Target (left) and zero-shot (right) task performance comparison w.r.t. ensemble classifiers. "TreeProbe w/o ensem. cls." is the version of TreeProbe by finding the cluster most similar to the input and using the corresponding classifier to predict labels. The reported accuracy is the average across lasks after cumulatively receiving training data from the datasets shown on the r-axis.
Effect of ensemble classifiers in TreeProbe inference. Referencing Sec. 3.2, we observe that ensemble predictions from multiple classifiers associated with & retrievals slightly enhance performance. Fig. S5 presents these results under the task incremental learning setting for eight target tasks. Our final model, TreeProbe, is better than its variant without the ensemble classification function in both target and zero-shot performance. The additional inference cost. for ensemble predictions is negligible, so we choose it as the default setting for better performance.
S-9 Evaluation on long- tailed classification
Method
CLIP Zero-shot
KNN
LinProbe
TreeProbe
KNN*
LinProbe*
TreeProbe"
PaCo
RAC
Accuracy (%)
40.0
35.5
37.0
30.5
40.4
42.7
41.3
41.2
47.2
Table S5: Comparison of long-tailed classification Places3651TT (Liu et al. 2019). For this experiment, we use CLIP ViI-L/140336px as the backbone network.
means + ATM-Fmb.
In long-tailed classification, some test labels are rare or unobserved in training, so blending exemplar-based based models with consolidated models can be beneficial, as shown by (Long al. 2022). To accommodate this setting, adjust our AIM-Emb method by considering the 2/3 rarest labels not being present in the exemplar set and the remainder being present, to calculate Vout. In this experiment, the node capacity of TreeProbe methods is 10k. In Tab. S5 present results the Places365LT dataset (Liu et al. 2019). Our AIM-Emb method with LinProbe and TreeProbe outperform the zcro-shot bascline. We also compare to PaCo Cui et al. 2021) and RAC (Long el al., 2022),, which are specifically designed for long-tail classification. PaCo incorporates learnable class centers to account for class imbalance in a contrastive learning approach. RAC trains an image encoder augmented with label predictions from retrieved exemplars. Although not
specifically designed for long-tailed classification, our method performs similar PaCo, but RAC performs best of all.
26

---

Published in Transactions on Machine Learning Research (10/2024)
S-10 Detailed time analysis
Tab. S6 compares the time complexity and actual times of three prediction approaches in a rapid learning system: KNN, LinProbe, and TreeProbe. Please refer to Sec. S-2 for soflware and hardware enviromments for this comparison. The actual times are calculated in a task incremental learning scenario, with set to
KNN has a constant time complexity for both training and inference, with actual times of 9.8 and 416.1 seconds respectively. LinProbe has linear training time complexity and constant inference time complexity. The actual training time is considerably long at 30971.4 seconds (~8.6 hours), and the inference time is 449.1 seconds. Our most recommended algorithm, TreeProbe, has logarithmic training time complexity and linear inference time complexity related to the mumber of retrieved classifiers (k). In practice, it exhibits significantly shorter training time than LinProbe at 2082.0 seconds (~0.6 hour) while having a slight increase in inference time. The table illustrates that TreeProbe strikes a balance between accuracy and efficiency, being more accurate than KNN and more efficient than LinProbe. Note that numbers may vary depending on software and hardware situations. This number is collected from the same PC we used run all experiments.
Methods
Train time
Inference time
Actual TT (s)
Actual IT (s)
KNN
(())
(nn)
9.8
416.1
LinProbe
((n)
((1)
30971.4
449.1
TreeProbe
(loo n + 30)
O(k)
2082.0
614.1
Table S6: Time analysis of different prediction approaches in rapid learning system, namely KNN, LinProbe, TreeProbe. Here, k represents the number of retrieved classifiers. "TT" means training time while "TT" means inference time, Actual train and inference time is calculated with task incremental learning scenario by summing up lime spend on training and inference across lasks. For TreeProbe, number of exemplars that are stored, and the log is due cluster assignment, which is negligible compared to v, the capacity of each cluster node.
n the total
27


Reviews:
[{"review_id": "0ABvrBz6Rg", "reviewer": "Reviewer_FUce", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n\n-How to quickly adapt pre-trained embeddings for downstream tasks is an important topic of frontier research. \n-This work draws inspiration from cognitive science to design an architecture that resembles human cognition that has two different systems for fast and slow learning.\n-The experiments contain multiple datasets and include ablations of CLIP and the exemplar learning system.\n\n\n\nweakness:\n\n-The proposed approach seems to assume the visual-textual representation (CLIP) contains all necessary information for future/downstream tasks since it does not update or finetune the encoders. This is a major limitation for applications with larger downstream domain gaps. \n\n-Using CLIP and learning a smaller network on top of the pretrained embeddings for continual learning is not a novel idea itself, and TreeProbe is an incremental contribution on top of linear probe. Therefore the major contribution of this work comes from the empirical results. However, the performance gain compared with the prior work ZSCL (Zheng et al., 2023) seems marginal and therefore the contribution isn't convincing enough. It would be useful to compare the learning and inference speed with this prior work.\n\n-At the same time, while the authors ablated multiple different versions of CLIP, it is desirable to also ablate different vision-language models (possibly stronger ones) to gain a comprehensive understanding of the usefulness of the proposed system. \n\n-The experiments section of this paper is not easy to follow with too many different settings. It would be useful to include some qualitative results and example evaluation data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method that leverages pretrained multimodal embeddings for continual/incremental learning of image classification. The proposed method leverages a TreeProbe algorithm to learn a small and fast model on top of the frozen CLIP embeddings. Experiments ablated different design choices of the system and performance of the proposed system is compared with results in prior work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "more baselines/ablations needed (see weaknesses above)\n- ablate different VLM embeddings\n- some finetuning version of the encoders\n\nimproving the performance margin compared with ZSCL (Zheng et al., 2023), maybe with a different metric (speed) or tailored dataset?\n\nit would be useful to separate evaluation of datasets released before and after CLIP", "broader_impact_concerns": "the broader impact statement is present and sufficient", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5HGAAukowq", "reviewer": "Reviewer_avc7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- This paper studies an interesting problem, i.e., continual open-vocabulary classification. A comprehensive analysis is provided in this task.\n\n- The paper is well-written and easy to follow.\n\n- The paper proposes a simple yet effective method, which achieves consistent improvement over CLIP baseline.\n\n- Extensive experiments are provided, demonstrating the effectiveness of the proposed method.\n\n\n\n**Weakness**\n\n- In the introduction, it would be better to show a figure to illustrate the motivation of the method, the task definition of continual open-vocabulary classification, as well as the benefit of the proposed method.\n\n- Figure 2 is hard to understand. For example,  what is AIM in the figure? Examplars in different classes should be in different colors.\n\n- Using CLIP as incremental learning is not novel. This paper lacks of comparison of more recent CLIP-based methods.\n\n- Instead of combining with the proposed examplars-based method, I think it can also be combined with others, such as prompt-based methods. To demonstrate the effectiveness of the proposed method, the authors should compare with the combinations with other types of incremental learning methods.\n\n- I think there may be a drawback when there is a domain gap in the testing data. For example, when evaluating the generalization ability on ImageNet-V2-Sketch-A-R, the examplar-based module will produce low results on these target domains. In this situation, combining with examplar-based module may bring negative impact to the whole system.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of continual open-vocabulary classification. To solve this problem, the authors propose a simple, yet effective approach, by combining the predictions of CLIP baseline and examplar-based model. Experiments are conducted on various datasets, showing the improvements of the proposed method over previous methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please solve the Weaknesses.", "broader_impact_concerns": "The authors have already discussed broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3bIlsgp5Yq", "reviewer": "Reviewer_LQD8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe concept of extending continual learning to open-vocabulary learning is rather interesting. This approach broadens the scope of continual learning, making it applicable to a wider range of vocabulary.\n\nThe combination of a visual-language foundation model like CLIP with an exemplar model vividly simulates the human cognitive process. By leveraging the strengths of both models, the authors have created a robust system.\n\nThe tree probe method stands out as an impressive innovation. It integrates incremental hierarchical clustering into continual learning, thereby enhancing efficiency. \n\nThe authors conducted extensive experiments to validate their proposed method. These experiments were carried out under various settings, including data incremental, class incremental, and task incremental scenarios. \n\n# Weaknesses\nThe improvement of the model appears to stem primarily from the adaptation of the CLIP model, resulting in performance only slightly better than that of ZSCL, another model utilizing CLIP. \n\nThe reliance on a frozen CLIP model constrains the potential performance gains after fine-tuning. \n\nAdditionally, this method faces challenges in generalizing to more complex tasks, such as semantic segmentation, limiting its broader applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel method for open-vocabulary continual learning by leveraging inspiration from the complementary learning systems observed in human cognition.\n\nThe approach integrates predictions from a CLIP zero-shot model with an exemplar-based model, using zero-shot probability estimates to determine class membership. \n\nThe author introduces the tree probe method, which uses hierarchical clustering for nearly constant training time and better accuracy than KNN approaches.\n\nThe authors conduct numerous experiments to validate the efficacy of their proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Problem 1:** Why do the LinProbe and CLIP + LinProbe (Avg-Emb) curves overlap in Figure 2(a) and (b)? Could the authors provide a more detailed explanation?\n\n**Problem 2:** Additionally, it is recommended that the authors include more baseline methods for comparison and use a column to specify whether a vision-language model is used in the table.\n\n**Problem 3.**  When set $\\alpha=p\\left(y \\in \\mathbf{Y}_e \\mid I\\right)$, equation(3) is different from equation(5).   Could the authors give more explanations how to get AIM(Adaptive Instance Marginalization) equation(5) from equation(3)?", "broader_impact_concerns": "The authors discussed broader impact concerns in their paper. Moreover, this article primarily considers common publicly available image datasets and does not involve personal privacy, thereby presenting no ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers provided a mix of positive and constructive feedback for the paper. They appreciated the extensive experiments across different incremental learning settings, noting that the method achieves a strong balance between learning speed, task performance, and zero-shot effectiveness. Reviewers also highlighted the significant 20x improvement in training efficiency compared to ZSCL as a major strength.

Overall, the paper was considered solid, though some reviewers felt that additional experiments and baseline comparisons could further strengthen its contributions. Two of the reviewers leaned towards acceptance, while one suggested exploring combinations with other methods and conducting experiments on datasets like ImageNet-V2-Sketch-A-R. This reviewer expressed concerns about the exemplar-based model's ability to handle domain gaps, which is a valid and important point for future work. Taking these factors into account, the Action Editor recommended accepting the paper.
Decision: Accept (poster)