Memory-Modular Classification: Learning to Generalize with Memory Replacement
Authors: Dahyun Kang
Ahmet Iscen
Eunchan Jo
Sua Choi
Minsu Cho
Cordelia Schmid
Venue: Accepted by TMLR ()

Abstract: We propose a novel memory-modular learner for image classification that separates knowledge memorization from reasoning. Our model enables effective generalization to new classes by simply replacing the memory contents, without the need for model retraining. Unlike traditional models that encode both world knowledge and task-specific skills into their weights during training, our model stores knowledge in the external memory of web-crawled image and text data. At inference time, the model dynamically selects relevant content from the memory based on the input image, allowing it to adapt to arbitrary classes by simply replacing the memory contents. The key differentiator that our learner meta-learns to perform classification tasks with noisy web data from unseen classes, resulting in robust performance across various classification scenarios. Experimental results demonstrate the promising performance and versatility of our approach in handling diverse classification tasks, including zero-shot/few-shot classification of unseen classes, fine-grained classification, and class-incremental classification.

Published in Transactions on Machine Learning Research (03/2025)
Medory-Modular Classification: Learning to Generalize with Memory Replacement
Dahyun Kang POSTECH
babggnn.ongg@pastech.ooo
Ahmet Iscen
iscen iscen&ggoogle.co com
Google DeepMindd
Eunchan Jo
cunchann9229ppostell oc. swachoi@postech. oc. oc.kr mscho muscho@posecch. kr
Sua Choi
Minsu Cho
POSTECH
Cordclia Schmid
cordelias crrtliaaa@goggle.ccom
Google Deed
Reviewed on OpenReview: https: // openreview. net/ ht///forur forum? id= DcIWWidrrgg8
Abstract
We propose a novel memory-modular learner for image classífication that separates knowl- edge memorization from reasoning. Our model enables effective generalization new classes by simply replacing the memory contents, without the need for model retraining. Unlike tra- ditional models that encode both world knowledge and task-specific skills into their weights during training, our model stores knowledge in the external memory of web-crawled im- age and text data. At inference time, the model dynamically selects relevant content from the memory based on the input image, allowing to adapl to arbitrary classes by sim- ply replacing the memory contents. The key differentiator is that our learner meta-learns to perform classification tasks with noisy web data from unseen classos, resulting ro- bust performance across various classification scenarios. Experimental results demonstrate the promising performance and versatility of our approach in handling diverse classification pero-shot/few-shot
tasks, including zero-shot /fow-shot classification of unseen classes, fine-grained classification, and class-incrementee classification.
1 Introduction
Large-scale neural models have achieved remarkable results when fine-tuned and applied to downstream tasks in computer vision (Kolesnikov et al.] [2020] Yuan et al. 2021 Alayrac et al. (022) and natural language processing (Brown et al.) 2020 Touvron et al. 2023). These models are trained on massive datasets using immense computational resources, resulting a vast number model parameters that encapsulate both world knowlodge and task-specific skills. This complexity poscs two challenges. First,
difficult determine which knowledge in the training data learned skills contributes to the model output for specific task. Second, models cannot directly reflect changes in the ever-growing real world, such updates to data sources relevant to the target task, without undergoing additional training.
To flexibly adapt to the external world knowledge, recent zero-shot image recognition models (Guu el al. 2020 Hu et al. 2023b enhances image representations with their relevant data retrieved from an external knowledge source. Such lcarning method often callod retrieval-augmennee Icarning. This approach allows models leverage external knowledge sources and efficiently allocate model parameters focus on reasoning tasks. Although these models have shown promising results in knowledge-intensive applications, such

---

Published in Transactions on Machine Learning Research (03/2025)
frozen encoder
frozen encoder
frozen encoder
retrieval
retrieval
retrieval
baguette
azalea
azalea
q
Pantheon
Jelly cat
Q. van cat
van
Bym
nectar peach
Q Garfield
Garfield
knww juice
(a) training with external image & text memory (b) testing with new image & text memory
[c] testing with incremental image & text memory
Figure 1: Training and evaluation stages of MML for web-assisted zero-shot classification. MML constructs image/text memory with text keyword search on the internet given target classes. The memory provides relevant image/text features which are integrated via a trainable knowledge integration module (a). On evaluation, the memory can be replaced or detached from the model such that MML joins the new knowledge as memory, while the rest of the model remains unchanged. Once trained, MML handles zero- shot classification on unseen classes with memory replacement (b) and incremental classes with memory expansion (c) using the new knowledge collected from web to solve zero-shot classification.
question answering (Gao et al. 2022) and long-tailed classification (Long et al. 2022), their capabilities are limited to a specific target task. Morcover, they assume that the memory content is retained throughout training and testing; the generalizability of the learned models when faced with substantial memory updates or replacements remains unexplored.
In this paper, we introduce a novel learning architecture, the memory-modular learner (MML), for image classification. MML leverages an external memory to perform input-adaptive reasoning during the clas- sification process. A key advantage of MML is its ability generalize with memory replacement, i.c., memory-modulat generalization. By simply plugging in new-class content into memory, MML can adapt to novel classification tasks without requiring any architectural modifications (Fig. D The external memory used by MML is populated by web-crawled images and text obtained by keyword search of the target class names. This approach facilitates the incorporation of up-to-date world knowledge into the memory, ensuring that MML remains applicable as external knowledge evolves. Despite the potential introduction of data noise from web crawling, MML demonstrates robust classification performance in practice. This remarkable robustness allows MML to effectively leverage the noisy memory contents for accurate image classification.
One critical observation of our work is that by representing classification as metric learning (Vinyals et al. 2016 Snell et al.) 2017), MML becomes less susceptible to overfitting on the specific content of the mem-- ory. This allows it to learn more effectively how to perform classification reasoning with arbitrary memory contents. Specifically, we represent the classifier weight vectors, or the class prototypes, as the average of representative memory items rather than as learnable parameters. When a new set of classes is given, the class prototypes are immediately computed with the average of the memory items of the highest cross-moda similarity. The imput query is classified by the class of the closest class prototype. This design choice allows us to update the memory and adapt to new classes without retraining the entire model. Due to its inherent flexibility, our meta-learned model can handle zero- to multi-shot samples, as well as variable number of classes with the knowledge collected from w Experimental results in various scenarios, including zero- shot thot/few-shoo classification of previously unseen classes, fine-grained classification, and class inss-incremmenal classification, demonstrate the promising performance of MML.
Our contributions can be summarized follows.
We introduce a gemory-modulle learner (MML) for image classification, that performs adaptive reasoning using external and replaceable memory.
We clarify our sero-shot classifcattion approach that accesses to unlabeled web data as web-assisted zero-shot classification.
2

---

Published in Transactions on Machine Learning Research (03/2025)
We investigate the generalizability in adapting to new classes by replacing the memory with related content, without tuning the model weights.
We provide in-depth analyses on the memory-modular generalization to unseen classes in realistic setups, i..,, using noisy web-crawled memory.
We show that MML achieves promising gains in various scenarios such as zero-shot, few-shot, fine- grained, and class-incremental classification by leveraging target-class knowledge collected from web.
2 Related work
2.1
Few-shot and zero-shot classification with the assistance of external web data
Few-shot image classification (Fii-Fei et al.
2006) aims to generalize to arbitrary unseen classes given The conventional experimental setup of few-shot classifica-
few support images from largel class set.
tion Vinyals et al. 2016 Allen et al. 2019
Triantafilluu et al. 2020: Doersch et al.) 2020 Zhang et al.
2020 Kang et al. 2021 h assumes at least a few hundred labeled images used for (meta- fontta-ttrining before the actual few-shot inference stage. We, however, adopt a more label-efficient and realistic approach for this task; we train a model with even fewer labeled training samples 6.g-, < 16; instead we assume retrieval access to external unannotated data. Zero-shot classification (Larochelle et al. 2008 Yu & Aloimonos 2010) aims for generalization beyond scen classes without the use of few-shot support images for the target classes. Instead, classification is conducted based non-visual clues such textual information of the images Fuu ct al., 2015 Akata et al. 2016),, , yes-or-no attributes (Lampert et al.; 2013) or the class name in text (Socher et al.] 2013) of arbitrary classes. The conventional zero-shot tasks have assumed no use of images from target classes during training, but with the advent of web-driven pretrained models, recent "zero-shot. methods (Iscen et al.] 2024: Liu et al. 2023) started use the expression in a more relaxed annotated images from target classes, thus allowing access to noisy web data. For example, a Dision-and-language foundation model named CLIP (Radford et al.] 2202]] trains image and text encoders with 400 million madg--nn--aapton pairs from internet which likely overlap with standard zero-shot classification benchmark categories. We follow this usage in our paper and leverage web data to leverage the external world knowledge for zero-shot classification. We thus clarify that our approach as web- assisted zero-shot classification with the terminology of "shot" denoting the number of class-annotated images for each target class.
way, meaning no use of
2.2 Image recognition with memory retrieval
One of the carliest works of using an external memory in machine learning is the k-ncarest neighbor (kNN) classifier (Hart, 1968). which retrieves k-nearest neighbors from memory for class prediction. Re- cent work constructs memory from large-scale pre-trained models and performs kNN retrieval for class prediction (Khandelwal et al., 2020 Nakata et al. 20022). This straightforward method revisits the potential of external memory for class reasoning, being decoupled from encoder learning (Graves et al.) 2014). Image recognition models have also been trained using external image-text paired memory (Jia et al.
2021b Long
et al. 2022 Iscen et al.] 2023). Our approach assumess a more weakly-supervised type of memory, collecting image and text memory contents separately. Other external memory- image recognition work focuses on training multi-modal feature encoders (Wei et al. 2023 Hu et al. 2023b or training CLIP models with external image-text, paired data. (Iscen et al. 2024 Liu et al. 2023). One common theme among the existing memory-basee based models is that they are either trained for a specific task (Long al. 2022; Hu et al. 2023b;
Iscen et al. 2023) static memory (Iscen et al. 2024). Among them, REVEAL (Hu et al. 2023b) is perhaps learning architecture of REVEAL and MML is indeed similar
most similar to ours. The memory-augmentt in terms of architecture but different in terms of the role of memory. The memory of REVEAL serves as a general knowledge bank to assist VQA and captioning tasks. On the other hand, the memory of MML contains specifically related contents of the target classes for classification, crawled from web. Therefore, memory contents can be completely replaccable when the target classes are updated the memory is mod- ular. Note that any other previous models do not replace memory contents completely. Also, the difference of general and specific memory also leads to the size difference. REVEAL contains 20.3M memory items
3

---

Published in Transactions on Machine Learning Research (03/2025)
image prototype
image memory
query
azalea
ImageNN cross-attn
k
NN retrieval
van cat
+
Garfield
image NN
frozen CLIP
encoder
query
query
Q
textNN cross-attn
K
azalea
query
NN retrieval
van cat
text NN
Garfield
text prototype
text memory
L access front-loaded external memory
retrieval-based attentive knowledge integration
Figure 2: Memory-modular learner (MML) constructs image/text memory by web-crawling with text keyword search. Given a query image, its kNN features are retrieved from each memory and used for altentive knowledge integration. The class prototypes are constructed with the average of the memory elements of the highest cross-modal similarity. MML derives class reasoning with the nearest neighbors (NNs) from the external memory. This modular memory enables MML perform web-assisted zero- pero-/few-shot classification on unseen classes by memory replacement and class- mass-ineremental classification by memory expansion.
of general image-text pairs. MML requires only 0.7M image and 0.2M text memory items for 1K classes of ImageNetIK, (see Sec. 4.1 which is the 4.4 % size of REVEAL. In contrast to the previous related work, MML aims to generalize beyond seen class set and modular memory that can be updated at any time. To the best of our knowledge, MML is the first to investigate the memory replacement with new memory contents to tackle unscen-class generalization.
2.3 Class itasemincrementll classification
Class-incremental classification (Rebuffi et al. 2017 Zhu et al. 2023 assumes that set of unseen classes arrives at each stage and aims to classify the input into all known classes given limited access to the old class data. The most critical challenge of this task is catastrophic forgetting, i.e., directly training neural networks with the new-class data leads significant performance drops old classes. To address this challenge, recent work (Yan et al. 2021: Wang et al., 2022a; Zhou et al. 2023b Douillard et al. 2022 Wang et al. 2022b) introduces memory to store data. from previously seen classes knowledke lnowledge into model. In contrast, the memory in MML plays the role of replaceable and extensible world knowlenn reference. The purpose of memory in these two models are different: the memory of class- incremental learners helps not to forget the previously seen classes (Belouadah & Popescu 2019 [scen et al. 2020), however, the memory MML assists the current classes of interest, which might have not been seen.
training source to compile
3 modularmodular learner
We address the problem of classifying an input image into target classes that are represented by a class name in text, ie., zero-shot classification, or additional few support images, f.e., few-shot classification. To this end, we introduce a memory- modular learner that performs adaptive rcasoning using an external memory that updatable and replaceable. Our memory-modular learner takes advantage of both vision and language modalities using the CLIP encoder (Radford et al. 2021
base feature extractor for image and text. Since our method is not restricted to CLIP, any other image-text model, e.g. ALIGN (Jia et al. 2021a) LiT Zhai et al. 2022 can also be adopted. Figure illustrates the overall architecture of our approach.
The memory- learner starts by loading the knowledge memory and generating class prototypes for target classes (Sec. 3.1] These front-loaded memory items and prototypes are all stored as frozen features from a pre-trained image-text encoder. They are replaceable whenever the target classes change the external knowledge sources updated. Given input image, the memory-modular learner accesses
4

---

Published in Transactions on Machine Learning Research (03/2025)
the knowledge memory, retrieves k-nearest-neighhor (kNN) items, and predicts the corresponding class via cosine-similarity with class prototypes (Sec. 3.2). Since class prototypes are generated immediately from the memory items, the prototype- prototype-based classifier can adapt to new target classes of updated memory contents without additional training.
3.1 Memory construction and prototype generation
Given target class names or descriptions, we construct the knowledge memory based on available image and text data and generate class prototypes using the memory. As the world knowledge is updated, these memory items can be added or deleted, and even completely replaced, without updating the model weights.
Knowledge memory
The image memory is constructed using images obtained from keyword searches the internet. For each larget class images are collected using the class name as the search keyword search engine, e.g., Google or Flickr (Kim et al. 2023 Hou et al. 2018). We follow a similar strategy for text memory. In this work, textual information relevant to cach target class name is retrieved by querying Wikipedia (Tian et al. 2022 Hu et al. 2023a Nacem et al. 2023). These web-crawled images and texts may be noisy, but consist of scalable memory contents that reflect the world knowledge. After collecting the relevant images and texts for each target class e, we extract their d-dimensional features with the image-text encoder, and then store Nime
them in the image and text memory: M {w,)== and M = {tز
No respectively.
Class prototypes
For zero-shot classification, we construct class prototypes based on cross-modal consensus between image and text memory items. For each target class e, we first compute the cross-modal cosine similarity cos(;;) from each image to all text items of the same class and then select the top-M images with the highest similarity to the texts, i.e., images with high cross cross-moda modal consensus. The image prototype for class e is then set to be the average of the M features:
pl
ing 1
Σ v, T= argmax M M'E
Σ cos (on,ttt))) t))
(1)
IT
#ET
LEMba
where argmaxWss() denotes the top-M operator that returns the best M items from the set S maximizing the operand function. Based on image-text consensus, this process constructs robust and representative class prototypes from noisy data in the absence of human annotation. Likewise, the text prototype is obtained using the average lext-lo-image similarity. This zero-shot protolype construction resembles Prototypical Networks (Snell et al. 2017), which averages the image examples for each class, On the other hand, we build multi-modal prototypes by averaging the representative M samples collected without given annotations. For few-shot classification, f.e., when few support image samples are available for the target class name, we simply construct class prototypes by averaging the given samples as done in Snell et al. (217
Memory update for adapting to unseen classes
The knowledge memory contents and class prototypes modular and replaceable. When target classes are updated, e.g., classification of unseen classes or incremental classes, new memory contents are collected to pertain to the new classes. Subsequently, the prototypes for the classes are updated accordingly using Eq.
3.2 Reasoning with memory access
Given an input image for classification, we incorporate memory knowledge into reasoning. Items relevant the input are retrieved from image/text memory and integrated with the input feature through cross- attention. The input is then correlated with the image and text class prototypes. Finally, the predictions from the image and text branches merged at the logit level for class prediction.
5

---

Published in Transactions on Machine Learning Research (03/2025)
Memory retrieval
For an input image feature f extracted from the image encoder, its k-nearest-neighhor image items are retrieved based on cosine similarity with all image memory items of all target. classes:
f f.v [f
Nims argmax DE Mins
(2)
where Mime = UeMasm The text &NNs are also retrieved by querying the image feature to the text memory.
Attentive knowledge integration
The knowledge of the retrieved memory items Aimg [vw]km is aggregated by cross-attention (Vaswani ct al. 2017 Jaegle et al. 2021) and then integrated with the input embedding f. The cross-attention learns
to integrate the nearest neighbor (NN) features into the input feature:
pimx= f (2L)) ( (( (V(()))
Vd
where Q, K. V are projection layers with non-linearity, softmax over & items, and []] concatenation. Sim- ilarly, the same step with the text NN features is performed in parallel. This process can be viewed as a learnable soft NN integration in contrast to the hard majority voting with NNs [Nakata et al. 2022
Classification inference
The resulting embedding is matched against the multi-modal prototypes for all C target classes with cosine similarity cos(;,)) to produce classification score. The e-th class logit ze is obtained with:
2. cos(p' txt f + cos(p (m fims).
(4)
Final class prediction is conducted simply by taking the class with the highest score.
3.3 Training
Our model is trained with cross-entropy loss with one-hot ground-truth class label y and the logit z:
C= yo log exp(Ze/T)
(5)
exp(ze</T)
where r is a temperature for scaling. Note that we freeze the pre-trained image-text encoder and train the remaining parameters only, (.e., those of attention layers on the image and text branches. The number of training parameters and the frozen CLIP-B/32 is 6.3M and 151M, respectively. Using the frozen pre-trained encoder has three advantages. 1) The pre-trained features provide more reliable similarity for ANN retrieval and prototype construction than scratch features, encouraging stable training. 2) Retaining the general pre-trained knowledge, the knowledge integration part converges elficiently with a small amount of data. 3) Most importantly, if the encoder is trained or fine-tuned, then all memory features should be synchronized regularly, while the frozen pre-trained encoder allows us to avoid such extensive computation.
4 Experiments
4.1
Experimental setup
Training details
For the image/ter feature extractor, we use the pre-trained CLIP (Radford al. 2021 and ALIGN (Jia et al. 2021a Unless specified, CLIP-B/32 used. For training, we batch size of 256, learning rate

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Zero-shot cross-dataset transfer. MML is trained with 1 or 4 samples from ImageNet coarse-
grained classes and tested on 10 fino-grained datasets with zero shot, which is roughly a domain shift scenario.
method
ImgNes1K
Caltech101 OxtordPets Cars Plowers Food Aircraft SUN
DTD
EuroSAT
UCF
avg.
objects
pets
cars
flowers
food airplanes scones textures
land
actions
pero-shot CLIP
Radford es al 2021
66.7
75.9
63.6
62.9
84.7
74.5
18.2
55.3
33.3
43.0
58.7
55.2
RNN classifier
Nakata et al. 2022
55.7
87.6
72.7
68.6
75.2
75.6
29.6
56.2
33.2
37.3
63.2
59.5
MML KI1) MML (ImageNet\K..
48.3
92.6
86.1
68.1
76.2
81.8
26.2
60.0
11.6
15.6
61.2
62.8
69.0
93.5
86.7
68.9
77.5
84.2
26.3
64.7
42.8
48.2
66.5
66.2
Table 2: Comparison on zero-shot classification on CUB
Wah et al.
2011 with different backbones. Note
that RECO [Iscen et al. 2024 trained with CC12M.
method
backbone
accuracy (%)
CLIP RRdford al.
2021
ResNet-101
68.8
method
backbone
accuracy (%)
Yu al.
2020
ResNet-101
72.4
CLIP
Radford
2021
CLIP-B/32
70.3
al.
2020
ResNet-101
73.8
RECO
Iseen
2024
CLIP-R/32
75.2
ResNet-101
76.1
MML
CLIP-B/32
76.7
MML
ResNet-101
78.8
of 1 and weight decay of 54 on single 2080 Ti or an RTX 3090 GPU for all training and testing, We retrieve 32 NNs from both the image and text memory. We use M = 16 for prototype construction and set the logit temperature 7 = 16, which is chosen via hyperparameter search. We use three random seeds for drawing few-shot samples randomly and report the average.
Memory and data
To construct the external image memory for ImageNet derivatives, we employ a readily available web-crawled image dataset, WebVision ver. 2 (Li et al.] 2017 WibVision is collected from Google and Flickr by the keyword search of the 1000 class names of ImageNetIK (Russakovsky et al., 2015). We use the image subset crawled from Google unless htheewise specified. To construct image memory for the other 10 datasets used in Table D as no public web-crawled datascts for the corresponding classes are available, we crawl a maximum of 100 images per class from Google with an auto crawler. For lext memory, we query Wikipedia for each class name and retrieve the correspouding article text by web crawling. In such a way, the modest length of memory is obtained, e.g., 0.7M images and 0.2M texts for the 1K classes of ImageNet of which ANN search is feasible with the Py (Paszke et al. 2017 built-in topk module. The dataset details used for zero-/few-shor fine-grained, and class incremental classificat ion are specified in the corresponding paragraph.
4.2 Web-assisted zero-shot classification
First of all, we evaluate our method on zero-shot classification setup, where no labeled images are provided for the target classes. The only information given for the task is phrased class label for cach class, e.g., "van cat", which is used as the search keyword to collect the web-crawled memory.
Datasets: MML is evaluated on single-dataset and cross-dataset zero-shot classification benchmarks. For single-dataset zero-shot classification, ImageNet-S and CUB are used, where the classes of each dataset are split into disjoint sets for few-shot training and zero-shot testing. We adopt the existing zero-shot classification CUB benchmark (Wah et al., 2011: Akata et al. 2013) of which classes are split into 150/50 bird species classes for train/valitn /validaaion.. Similarly, we introduce an ImageNet. (Russakovsky et al. 2015) split such that it comprises 600/200/200 classes for train het/vvvaalaiiontteee and call it ImageNet-S (S stands for class split). We use 16 images per class for training, é.e., 9.6K training images. For testing target classes, cither zero or a few shots are used for zero- or few-shot classification scenarios. For the cross-dataset setting, we adopt a cross-datasel zero-shot transfer scenario (Zhou et al.) 2022), where a model is trained with a few samples from ImageNetIK, e.g., 1 or 4 training samples per 1000 classes, and transferred to 10 fino-grained datasots: Caltoch101, OxfordPets, StanfordCars, Flowers102, Food101, FgveAircraft, SUN397, DTD, EuroSAT, and UCF101. The total classes of these datasets amount to 1,310 classes and their details including the references are found in Table 10
7

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Ablation study on MML components
Table 4: Effect of different class prototypes
learnable integration
prototype
ImageNet-S
CUB
model
ImageNet-S
CUB
retrieval
avg of all memory items
81.2
74.4
80.1
61.9
avg of random memory items
81.9
70.4
(b)
76.4
67.9
text only
82.7
69.1
(c)
75.6
60.0
image only
82.3
76.8
MML
83.0
75.6
MML (imaaee & text)
83.0
75.6
Baselines:
The kNN classifier (Nakata al. 2022 retrieves kNN of the imput from memoryyande and immediately extracts text embeddings of the text class
predict the class by majority voting. Zero-shot CLIP/ALICE names in the predefined templates, e.g., a photo of a van cat, and matches them against the input image embedding. Three stale-oftthe-al zero-shot models (Yu et al. 2020 Xu et al. 2020 Chen el al. 2022) are also compared, which are trained with the total 8885 annotated images and text attributes of CUB.
Results: Table compares zero-shot baselines and MML on cross-dataset transfer. MML is trained with few ImageNet samples but exhibits great performance on other datasets with extreme domain shifts, e.g., from classifying general objects koussakvvsky et al. 2015) to land (Helber et al. 2019), by simply replacing the memory with the web-crawled domain-related knowledge. In particular, compared with the kNN classifier, which is uni-modal and non-learnable, our method meta-learns to integrate the multi-modal kNNs and effectively transfers to unseen classes. Table S compares MML and other zero-shot models on the zero-shot CUB benchmark (Akata et al. 2013) with 150 training classes and 50 test classes, where MML demonstrates its outstanding effectiveness compared to other models. While the existing models train with full training images and ground-truth attribute annotations (e.g., eye colors), MML learns with zero human- annotated attributes, but shows great performance based integrating the retrieved knowledge from the external memory. Comparing CLIP and ours examines the signeificant advanlage of external memory access and knowledge integration where ours obtains a 7.6-11.0 % point accuracy improvement. Plus, we examine the efficacy of different backbones in Table 15 in Appendix, where MML consistently outperforms the others. The following paragraphs continue with more analyses and ablation studies on zero-shot classification.
4.3 Model analyses
We present the analyses of the model components. All experiments are based on CLIP ViT-B unless specified.
Ablation study on model components
Table 3 presents the ablation study of the main model components of MML. The first model (a) is zero-shot prototype classifier. When the kNN retricval is added without the learnable kNN integration, the model (b) corresponds to the &NN classifier (Nakata et al. 2022), which is beneficial on CUB compared to the model (a). The model (c) examines the learnable integration of the cross-attention module without kNN retrieval, thus transforming the input feature with the learnable self-attention. The worst result of (c) implies that the additional cross-attention is even harmful without the proper source of ANN knowledge integration. The last with the two components (MML.) achieves the highest performance the two datasets.
Effect of different class prototypes
Table I compares different methods to build class prototypes. We first try to naïvely average all the contents in each memory to obtain class prototypes without using the top-M operator (Eq. D This average aggrega- tion is likely to include plenty of unfiltered data noise, resulting in poor performance on both datasets. Next, "avg of random memory items" randomly selects the same number of items with the proposed cross-modal protolype method and averages them per class. This noisy and unrepresentative protolype leads poor clas- sification. We also attempt to use the mingle-modality class prototype. The image prototype is more helpful than the text prototype on CUB and the reverse on ImageNet-S,, suggesting that the efficacy of the image and text prototype can be dependent on target dataset characteristics. All these methods do not use image-text
2The original work Nakata et 2022 leverages annotated datasets such as Tet as image memory, which is expensive be used memory. We thus replace it with the noisy web-crawled memory for reproduction.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Table 5: Effect of different memory types
F Figure Effect of memory size on ImageNet--
retrieval from
ImageNet-S
CUB
Accuracy (%)
ANN
75.6
60.0
text memory
$2.3
69.4
70
image memory
76.2
71.2
MML
unified memory
76.8
71.3
60
kNN classifier
MML (separate memoryy
83.0
75.6
10
100
500760
Number of memory items per class
Table 6; Memory content robustness of MML. Memory replace- ment at testing time from memory content to another.
Table 7: Memory content robustness com- parison with different noise levels on clas- sification (class) and retrieval
imnage meen at train image mem at test
WebVissoo (WV) Google WV Google + WV Flickr
WebVision (WV) Flickr
WV Flickr + WV Google
memory
&NN retrieval
class
IetgeNet-S
83.0 + 83.1 (+0.1)
832 83.2 (-0.0)
noiseness
rec81
roc016
acc.
text train text mem at test
Wikipedia
text thumbnails text thumbnails - Wikipedia
memory
75.6
Wikipedia + text thumbnails
noisy (Web
65.5
90.3
83.0
ImageNet-S
83.0 + 82.7 (-0.3)
83.2 83.0 (-0.2)
clean (ImageNet11K)
66.4
93.5
84.7
consensus while our method carefully selects memory items that exhibits the high rross-modal similarity for constructing class prototype. In this way, the prototype is comprised of the representative class data and also avoids potential data noise. Using the multi-modal prololypes, our model achieves robust performance.
Effect of different memory types
Table 5 validates our dual-branch image and text memory. The "no KNN" baseline has the same architecture as the proposed model, but instead, feeds the input feature for the key and value inputs replace of the kNNs, i.e., the query feature is shared with Q, K, in F This baseline exhibits the lowest performance and signifies the importance of the &NN knowledge integration. Next, we ablate either image or text memory. It is noticed that the model using only the image memory is more effective than the one using the text memory CUB, while this trend is reversed visual knowledge collected from the internet is beneficial for fine-grained image classification, on the other hand, textual information is useful for coarse-gras garsee-raiied classification of general objects. Lastly, we merge the image and text memory contents and then retrieve the modality- ANN features, which are then passed to a single knowledge intogration branch. We observe that the majority of kNNs are from the image memory, thus closely matching the performance of the image-menory model. To effectively interact with multi-modal ANNs, we choose to separate the image/text memories. This result signifies that the maal-branch multi-modal knowledge integration is crucial in zero-shot unseen class gencralization.
ImageNet-S. The opposite trend suggests that the vast and detailed
Effect of memory size
To validate the size of the memory, we set the memory size per class from to full for two searivval-based models, the ANN majority voting classifier (Nakata et al. 2022) and MML, and verify the performance growth in Figure 3 and Table 12 While the larger memory is more helpful, the performance reaches platean with the abundant memory size. We thus claim that MML requires a moderated size of the retrieval pool as the retrieval and knowledge integration effectively incorporate the useful data from the noisy data source.
Robustness of memory
Table 0 shows that MML does not overfit to certain memory contents and performs robustly to different memory contents with little loss performance. Note that MML already makes unseen class predictions with completely new memory contents of the new classes the zero-shot testing phase (Tables 1 This experiment further examines whether the model performs robustly when different instances of the same classes are plugged into the memory. We equip pair of image/text data collection from two different
9

---

Published in Transactions on Machine Learning Research (03/2025)
Query image
Retrieved images
Retrieved texts
GT
boa consitictor
3-4 bea constrictors become sexuall marure and reach the adult size 6-10 foet (1.8-3.0 m).
Zero-
castern diamond rattlesnake
CLIP
2010, constrictor shown to have reproduced asexually via parthnnogenesis.
MML
boa constrictor
campestris: prairic sharp arouse lives
Saskatchewan. southeastern Manitohn. southwestern
GT
prairic grouse
Ontario, and Upper Peninsula of Michigan Minnesota and northern Wisconsin
Zem- CLIP
partridge
The greater prairic chicken, losser prairic chicken, and sharp-taii tailed grouse make up the genus Tympunuchus, a genus of grouse found only in North America.
MML
prairie grouse
These vehicles are widely used in South Asia and Asia, where rickshaw driving provides essential emppleyment for recent immigranis from tural arvas, generally impoverished men.
GT
Zero- CLIP
rickshaw
Perhape seated rickshaw passenger back of the laboring driver, who, besides, metsphorically draught animal harnessed between shafts.
MML
rickshaw
Figure 4: Examples of query, image 2NNs and text 2NNs Human faces are anonymized for visualization.
sources of the same target classes. For image memory, we use the two disjoint sets from the WebVision (WV) dataset; one set is obtained through Google crawling and the other from Flickr, where both of them are from TmageNet-S classes. For text. memory, we use the text articles from Wikipedia and the text thumbnails of Google text keyword search. Once MML is trained with one source, we test it by plugging the two different memory sources. The results show that the model exhibits a marginal performance gap when replacing the test memory from one to another and also proves the modular property of the memory.
Table presents the robustness of the memory when compared with the clean (human-annotated) and the noisy (web-crawled) memory. The of web data inherently introduces the trade-off between avoiding additional human annotation and data noise. Resse learning of MML is also for noise reduction. The retrieval-based knowledge integration remedies such problem by selecting the nearest, i.e,, the most relevant, samples to integrate them to establish the data and classifier prototype representation. To quantify the noisiness the memory data pool, we also present the recall of retrieved items with the standard retrieval metric of recall@K K (rec@K) (Jégou et al., 2011]. The recall@ K returns if any instances from the ground- truth class are included in the ANN and 0 otherwisc. Although the clean image memory enhances mid-level retrieval and end-task classification accuracy, the noisy memory model achieves comparable results to the upper bound. This experiment supports modeling choice utilizing web-crawled images as external memory is reasonably effective and label-efficient compared to fully annotated memory.
Visualization of multi-modal ANN
Figure 4 visualizes the retrieved two image nearest neighbors (NNs) and two text NNs of the given input as well as the zero-shot CLIP prediction. Note that the images and texts in the example are independently retrieved from each memory. We notice that the image NNs often contain the query's noticeable visual patterns. From the text NNs, we observe that retrieved texts often contain synonymous keywords, e.g., the scientific names of animals. The last example with cows contains multiple objects hence ambiguously class-labeled. In this case, MML is able to retrieve semantically related images and predicts a reasonable class than the ground truth.
10

---

Published in Transactions on Machine Learning Research (03/2025)
Table 8: Few-shot classification on ImageNet-S
Table 9: Few-shot to many-shot classification. TTT stands for time training with the given data (shots).
method
4-shot
16-shot
methods
TTT
16
61
128
256
linear-prob CLIP
[Radford et
2021
72.1
80.6
(a) linear prob
72.1
80.6
85.5
86.9
87.3
ProtoNet (Seell et al. 2017
76.4
76.5
(b) MML
82.8
83.5
85.7
85.7
85.8
RAC [Long ct al. 2022
66.8
78.1
(c) MML."
83.4
87.0
87.4
87.9
88.3
ANN classiter Nakata ct 2022
77.2
77.2
(b) (a) gap
10.8
+2.9
+0.22
-1.2
-1.5
MML
82.8
83.5
(c) (a) gap
+11.4
+6.4
+1.9
+1.0
+1.0
4.4 Application to other classification setups
Few-shot to many-shot image classification
Problem setup: Few-shot classification (Fei- Fei et al. 2006; Vinyals et al. 2016 represents unseen classes with few-shot image samples for each target class during testing. We reuse imageNet-S to make it few-shot classification scenario by allowing access to additional or 16 labeled images during validation and testing.
Baselines: Linear prob is the simplest few-shot classification baseline (Chen et al. 2019), where we add a class-length linear layer on top of the frozen backbone and train it with the given target class few-shot examples. ProtoNet (Snell et al. 2017) is another few-shot classification baseline, where the few-shot samples are averaged and used as a class prototype. We also compare ours with another memory-bast based classification model, Retrieval-Augmeenter Classification (RAC) (Long et al., 2022). RAC first retrieves the nearest images from an image memory and feeds their corresponding class text labels to the subsequent text encoder to obtain auxiliary textual feature, which is then added to the input image feature. RAC was originally designed to be trained with abundant training data for long-tailes classification (Huang et al. 2016). We adapt RAC for few-shot classification and keep the text encoder frozen; otherwise, few-shot training fails to converge. All methods use the CLIP-B/33 backbone.
Results on few-shot classification: Table X compares MML and the aforementioned baselines on few-shot classification. While our MML outperforms the other methods, we observe that the performance gap between MML and the linear prob CLIP is bigger with fewer shots. This result implies that the knowledge relrieval from ezternal memory is especially effective when limited supervised data are available as the external memory access can compensate for the lack of supervised data.
Extended results on many-shol classification: In addition, we attempt to increase the few shots from the target classes to many shots and demonstrate the performance trend in Table g While the linear prob (a) is directly trained with 4 to 256 shots from the target classes, our method (b) is trained on the nou-target classes and tested without additional training with the 256 shots. MML outperforms linear prob by a significant. margin with 4 shots, and the gain gradually diminishes with increasing training data for linear prob. The results of MML with the test-time training (c) with the 4 to 256 shots show that our model recovers the diminished gap and further improves performance. Note that this work primarily focuses on leveraging external knowledge with zero or minimal supervision, in addition, we also show that additional test-time training benefits MML orthogonally to rotrieval-based reasoning.
Class-incremental classification
Problem setup: A class-incremental learning model assumed to receive a set of new class data sequentially and is asked classify a test image into the accumulated classes. As the model is not assumed to access to the previously seen data, the key challenge is not to forget the old classes.
Details on reproduction: For fair comparison, we reproduce existing class-incremental learning methods LLi
& Hoiem 2017 Ratelifi 1990 Rebulfi et al. 2017 Wang et al. 2022a) with the CLIP-B/32 backbone as well as ours on unified codebase (Zhou et al. 2023a by following the standard constraints.
Benchmark: We adopt a public benchmark, ImageNet100 Saaect000--aaOO-neiii (Rebuffi et al. 2017). where 10 unseen classes and their annotated samples are sequentially given for 10 consecutive stages. For each stage, a model is required to classify image into all the known classes, resulting in the accumulation of 100-class
11

---

Published in Transactions on Machine Learning Research (03/2025)
90
68
66
80
Accurccy (%)
70
62
60
lineur prob
50
LWF
RAC
40
Replay-
FOSTER MML
MML
iCARL-
30
10
20
40
10
20
30
40
50 60 70
80 90 100
Training label noise (%)
Figure 5: Class-incremental classification on ImageNct100
Figure 6: Result with training label noise
classification at the end. Across all stages, the size of the memory for each stage is always restricted to 2000 elements for all methods. For each stage, MML manages the memory length by dropping some old-class data from the memory and storing the new-class data such that the remaining memory elements are closest to the average of the memory contents, following (Rebuffi ct al., 2017]. Accordingly, MML updates the class protolypes with the updated memory elements al each slage. For evaluation, input images are classified inlo all the seen classes without stage-specific information.
Results: As seen in Figure 5 MML outperforms performs par with the dass-inccrmmentl learning specialist models, without using specific techniques for the task such as distillation of old class knowledge in model weights (Rebuffi et al. 2017) or storing the heavy model weights to the model memory (Wang et al. 2022a). For CIFARIOO, please see Figure in Appendix.
Training label noise robustness
We showcase that the reasoning procedure via memory retrieval is robust against the training data label noise. To simulate the label noise, randomly permute from 10% to 40% of the class labels of training queries with a wrong class and train the architecture with the corrupted labels. This comparison validates the effectiveness of reasoning sources for classification: reasoning from the relevant external knowledge us. reasoning from the memorized parameters. Figure D presents the comparison of the baselines and ours on ImageNetlI with the increasing portion of incorrect class labels. The memory-based models, RAC and MML, show robustness and powerful performance against training data noise. As MML predicts classes assisted by retricving input-adaptive ANN from the frozen memory, MML can avoid directly compiling the wrong training data into parameters, particularly being more robust the more incorrect label noise is injected in training. We hypothesize that retrieval-based reasoning encourages robust learning against the training label noise as the kNNs provide interactive reasoning with the neighborhood embeddings.
4.5 Feasibility on real-world scenarios
MML is lightweight and introduces little computational overhead; feature extraction, KNN retrieval, and knowledge integration take 1129.9, 58.3, and 10.7 GFLOPs (94.2, 4.9, and 0.9 %), respectively. MMT. is efficient in that training with batch size 256 consumes only 2.2 GB GPU memory on a 2080Ti thanks to the frozen backbone and memory features. We verify that classification inference with MML scales up to 1000 classes at once which consumes only 3.4 GB memory on single GPU. As MMT. is scalable with the increasing number of classes with the managcable size of memory, MML is expected to handle the dynamic nunber of classes for classification tasks in the real world.
The frozen pre-trained encoders significantly contribute to the little computation overhead and are considered the prerequisite of MML. Conversely, MML is implausible to be trained without such pre-trained encoders. We have allempled to Irain MML from scratch and achieved nearly random accuracies of 2.4% and 5.8% on ImageNet-S and CUB, respectively. This is due to the unreliable contents in the memory and the lack of training data to train imago-text encoders. This reliance pre-trained encoder might hinder application on specialized target domains such as medical, industrial vision, or domains where CLIP not applicable. Such domains necessitate specialized image-text encoders pre-trained on each domain to aim higher precision.
12

---

Published in Transactions on Machine Learning Research (03/2025)
5 Conclusion
We have presented the memory-modular learner and demonstrated its efficacy in various scenarios, investigat- ing the memory-modular generalization for unseen classes. The experiments show that our memory-modular reasoning effortlessly generalizes to unseen classes with memory replacement and exhibits robustness to noisy memory data. We also frame our retrieval-basol zero-shot classification as web-assisted zero-shot classifica- tion, which is believed be more realistic in the future research with the growth of web-trained foundation models. We believe that memory-modular learning benefits various tasks in the areas of artificial intelligence beyond classification, leaving them for future work.
Acknowledgments
This work was supported by Samsung Electronics Co., Ltd. (I0201208-07822-01) and the ITTP grants (2022-0-00290: Visual Intelligence for Space-time Understanding and Generation based Multi- Layered Vi- sual Common Sense (50%), 2022-0-00113: Sustainable Collaborative Multi-modal Lifelong Learning (50%))
funded by Ministry of Science and ICT, Korea.
References
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute- based classification. In Proc. IFFE Conference on Computer Vision and Pollern Recognition (CVPR), 2013.
Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele. Multi-cue zero-shot learning with strong supervision. In Proc. IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In Proc. International Conference on Machine Learning (ICML), 2019.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory, In Proc. IEEE International Conference on Computer Vision (ICCV), 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In Proc. European Conference on Computer Vision (ECCV), 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmu Peng, Kai Wang, Jian Zhao, and Xinge You. Msdn: Mutually semantic distillation network for zero-shot learning. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations (ICLR), 2019.
M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
Carl Docrsch, Ankush Gupta, and Andrew Zisserman. Crosstransformors: spatially-aware few-shot transfer. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthicu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern. Recognation Workshop, 2004.
Li Fei-Fei, Rob Fergus, and Pictro Perona. Ono-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2006.
Zhenyong Fu, Tao Xiang, Elyor Kodirov, and Shaogang Gong. Zero-shot object recognition by semantic manifold distance. In Proc. IEEE Conference on Compuler Vision and Pollern Recognition (CVPR), 2015.
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Transform- retrieve-generater Natural language-centric outside-knowledd visual question answering. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. ar preprint arXio:1410.5401, 2014.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In Proc. International Conference on Machine Learning (ICML), 2020.
Peter Hart. The condensed nearest neighbor rule (corresp.). IEFE transactions on information theory, (3):515 516, 1968.
Yangji Hc, Weihan Liang, Dongyang Zhao, Hong- Yu Zhou, Weifeng Ge, Yizhou Yu, and Wenqiang Zhang, Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning, In Proc. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Qibin Hou, Ming-Ming Cheng, Jiangjiang Liu, and Philip HS Torr. Webseg: Learning semantic segmentation from web searches. Xir preprint or Xiv: 1803: 180.009809,, 2018.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. orXiv preprint ar arXiv:2302.11154, 2023a.
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M Hospodales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: annrieeel-nngmeeneed visual-language pre-training with multi-source multimodal knowledge memory. Proc. IEEE Conference on Computer Vision and Pattern Recognation (CVPR), 2023b.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalancer classification. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In Proc. European Conference on Computer Vision (ECCV), 2020.
Ahmet Iscen, Alireza Fathi, and Cordelia Schmid. Improving image recognition by retricving from web-scale image-text data. In Proc. IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), 2023.
Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Retrieval-enhanced contrastive vision- text models. Proc. International Conference Learning Representations (TCLR), 2024.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Proc. International Conference on Machine Learning (ICML). PMLR, 2021.
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proc. International Conference on Machine Learning (ICML), 2021a.
Menglin Jia, Bor-Chun Chen, Zuxuan Wu, Claire Cardic, Serge Belongie, and Ser-Nam Lim. Rothinking nearest neighbors for visual classification. arXiv preprint arXiv:2112 08459, 2021b.
Herve Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. 1EEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2011.
Dahyun Kang, Heesoung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classifica- tion. In Proc. IEEE International Conference on Compuler Vision (ICCV), 2021.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre- Pre-tirining of deep bidirectional transformers for language understanding. In Proceedings of MAACL-HLT, 2019.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. Proc. International Conference on Learning Represen- tations (ICLR), 2020.
Namyup Kim, Taeyoung Son, Jaehyun Pahk, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Wedge: web- image assisted domain generalization for semantic segmentation. In 2023 IEEE International Conference on Roboties and Automation (ICRA), pp.
IEEE, 2023.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigeerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby, Big transfer (bit): General visual representation learning. In Proc. European Conference Computer Vision (ECCV), 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained cate- gorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Attribute-based classification for zero-shot visual object categorization. IEFE Transactions on Pattern Analysis and Machine Intelligence (TPAMT), 2013.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new lasks. In AAAI, 2008
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint Xiv: 1rXix:170.00880 008.02 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and (TaIiienceeePMMMI (TPAMI), (0(12):2935-299) 2017.
Machine Intelligence
Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning customized visual models with retrieval-augmmetee knowledge. In Proc. IEEE Conference on Computer Vision and Pattern Recognétion (CVPR), 2023.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recog- nition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-graincl visual classification of aircraft. ar Xiv preprint Xr0 2013.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Muhammad Ferjad Nacem, Muhammad Gul Zain Ali Khan, Yongqin Xian, Muhammad Zeshan Afzal, Didier Stricker, Lue Van Gool, and Federico Tombari. 12mvformer: Large language model generated multi-view document supervision for zero-shot image classification. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Kengo Nakata, Youyang Ng, Daisuke Miyashila, Asuka Maki, Yu-Chieh Lin, and Jun Deguchi. Revisiting knn- image classification system with high-capacity storage. In Proc. European Conference on Computer Vision (ECCV), 2022.
M-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Proceed- ings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Newral Information Processing Systems (NewrIPS) Workshop AulodijJ, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. International Conference on Machine Learning (ICML), 2021.
Roger Rateliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2))285, 1990.
Sylvestre-Alve Alvise Rebulli, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremen- tal classifier and representation learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Newral Information Processing Systems (NeurIPS), 2017.
Richard Sochor, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. Advances Neural Information Processing Systems (NeurIPS), 2013.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiw:1212.0400 2012.
Mingxing Tan and Quoe Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. International Conference on Machine Learning (ICML). PMLR, 2019.
Changyao Tian, Wenhai Wang, Xizhou Zhu, Jifeng Dai, and Yu Qiao. VI-Itr: Learning blasswwise visual- linguistic representation for long-tailed visual recognition. In Proc. European Conference on Computer Vision (ECCV). Springer, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ane Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXi::2302.11900 2023.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierro-Antoine Manzagol, et al. Mota-datasct: A datasct of datasets for learning to learn from few examples. In Proc. International Conference on Learning Representations (ICLR), 2020.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informalion Processing Systems (NeurIPS), 2017.
Oriol Vinyals, Charles Bhundell, Timothy Lillicra,, and Daan Wicrstra. Matching networks for one shot learning. In Advances in Newrul Information Processing Systems (NeurIPS). 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd birds- 200-2011 dataset. Technical report, 2011.
Fu-Yun Wang. Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In Proc. European Conference on Computer Vision (ECCV), 2022a.
Yaqing Wang, Quamming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 2020.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proc. IEEE Conference on Computer Vision and Pultern Recognition (CVPR), 2022b.
Yixuan Wei, Yue Cao, Zheng Zhang, Houwen Peng, Zhuliang Yao, Zhenda Xie, Han Hu, and Baining Guo. iclip: Bridging image classification and contrastive language-inage pre-training for visual recognition. Tn Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Yongqin Xian, Bernt Schicle, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. Proceedings of the IEEE conference on computer vision and paltern recognilion, pp. 4582 4591, 2017.
Jianxiong Xiao, Krista Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring large collection of scene categories. International Journal of Computer Vision (IJCV), 2016.
Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. Advances in Neural Information Processing Systems (NeurIPS). 2020.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class in- cremental learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Xiaodong Yu and Yiannis Aloimonos. Attribute-based transfer learning for object categorization with zero/one training example. In Proc. European Conference on Compuler Vision (ECCV), 2010.
Yunlong Yu, Zhong Ji, Jungong Han, and Zhongfei Zhang. Episode-based prototype generaling network for zero-shot learning. In Proc. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfong Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint xrXie:2111.11432 2021.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proc. IEEE Conference Computer Vision and Pallern Recogrition (CVPR), 2022.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable carth mover's distance and structured classifiers. In Proc. IEEE Conference on Computer Vision and Pattern Recogrition (CVPR), 2020.
Da-Wei Zhou, Wei Wang. Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class- incremental learning: A survey. Xix preprint ar urXim22302.06648 2023a.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Da-Wei Zhou, Qi- Wii Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-elficient class-incremental learning. In Proc. International Conference on Learning Represen- tations (ICLR), 2023b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Zhen Zhu, Weijie Lyu, Yao Xiao, and Derek Hoiem. Continual learning in open-vocabulary classification with complementary memory systems. orXiv preprint orXi::2307.014 01400, 2023.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Table 10: Class split and the numbers of memory items per class of the datasets used in this paper
number classes
avg. memory items per class
dataset
train/val /aa//vll////e /tstt
image
text
dotasets for single-dataset zero-shot transfer
ImageNet-S (Russakovsky et al.
2015
600/200/200 150/50
760.0
209.0
CUB (Wah et al. 2011 Akata et al. 2013
65.1
147.6
datasets for cross-dalasel zero-shot transfer
ImageNet1K 1K
RRussakovsyy et al.
2015
1000
760.0
209.0
Caltech101
Fei-Fei et al.
2004
100
73.2
292.0
OxfordPetss
Parkhi et al.
2012
70.6
240.1
Cars196
Krause al.
2013
196
80.9
219.0
Flowers1 02 NNilsback & Zisserman
2008
102
82.7
153.2
Food101 [Bosaadd et al.
2014
101
64.7
157.3
FgveAircraft [Maji et 2013
100
76.7
295.2
SUN397 (Xiao et al. 2016
397
75.4
209.9
DTD (Cimpol et al. 2014
71.1
169.5
EuroSAT (Helber et al.
2019
10
70.4
208.1
UCF101 (Soomro
2012
101
74.9
226.2
A
Appendix
In this appendix, we provide additional experimental details and results of our method. We will make our attached code and data publicly available once accepted.
A.1 Clarification of the term "zero-shot" classification using web data
Here we continue the discussion with the zero-shot classification from Sec. P and justify the task of web- assisted zero-shot classification. In our "zero-shot" classification experiments, MML uses the web-crawled images in memory, which are collected by web search with the class names in text and thus are inevitably noisy. The use of such images may appear to mismatch with the term "zero-shot", However, note that we use the term "shot" to refer to buman-annotaten oracle images as conventionally used in the zero- zero-/few-sher /few-shot learning literature (Xian et al. 2017 Wang et al. 2020 Zhai et al. 2022 Liu et al. 2023). In this context, MML is indced a zero-shot learner assisted with web data, which takes advantage of external momory data in contrast to conventional zero-shot methods without it. This zero-shot approach generalizes to arbitrary classes by replacing memory without training and shows strong robustness to noisy memory contents (cf.,
Table 6
A.2 Additional implementation details
All input images are normalized and resized 224 224 before being fod to the CLIP encoder following the official implementation. F All lexts are truncated with 75 words before being fed the CLIP text encoder. For image and text ANN attention layers, use single layer for each image and text branch and implement with the transformer encoder implementation of CLIP. Please refer to the attached code for the precise implementation details.
A.3 Details web-crawled datasets
Table 10 shows the composition of the datasets used for experiments. For the image memory for ImageNet benchmarks, we use publicly available dataset,
(Li et al. 2017). which consists of 1,000 class images crawled from Google and Flickr. For the other datasets, there are no such web-crawled image datasets of corresponding to the classes of each datasct. We thus actually crawl images by using an automatic image
hhttps//ggtthub. wttpetttuu.ccmoopaatttt com/openai/CL...
19

---

Published in Transactions on Machine Learning Research (03/2025)
Table 11: Cross-dataset zero-shot transfer results with and without duplicated images in the memory with the test set
MML
shot 4 shots
test images containing duplicates with memory 62.80
66.20
test images not containing duplicates with memory 62.79
66. 19
Table 12: Comparison of retrieval- based classifiers on Table 13: Comparison of retrieval-based classifiers on ImageNet-S with different values of k in kNN
ImageNet-S with varying length of memory
# mem elements per class
10
100
500
full (== 760)
k KNN
16
32
ANN classifier Nakata et al. 2022
55.3
66.9
73.6
76.5
76.4
KNN classifier Nakata al. 2022
75.2
76.3
76.4
76.0
MML
80.4
82.0
83.0
83.1
83.0
MML
83.0
83.1
83.0
82.7
crawler f that searches a text keyword on Google and downloads the images. For text memory, we search the class text name on English Wikipedia Sonnddii@ssnnngg [usin Wikipedia crawler p to retrieve the texts of the article, where example is shown in Table [8 We also add 80 text phrases such as "a photo of [class]", which is provided by CLIP as done in Tian et al. (2022). Each sentence in the retrieved articles corresponds to an item in the text memory. In total the crawled images amount to 89,970 images for 11 datasets and 465,496 text sentences from 2,141 Wikipedia articles for text memory of 12 datasets. We inspect that 6 images among all the web-crawled images turn out to be the same the test images. We thus remove the 6 images from the memory and evaluate the model in the exact same sctup with that of Table 1 As shown in Table the accuracy is not affected by the negligible amount of duplicated images from the Lest set.
A.4 Additional experimental results
Effect of memory length
Table 12 shows the numerical results of Figure 3 in the main manuscript.
Effect of in ANN
We vary & for the two retrieval-based models and compare their accuracy in Table 13 based models reach sweet spots at a certain k; in this example at around 16, and continue to drop with 2 32 perhaps because more irrelevant kNNs negatively affect attentive integration. Note that MML outperforms the &NN classifier with all difference choices of k.
Both the retrieval-
MML with another image-text encoder backbone
Table 14 compares the baselines and MML that use ALIGN (Jia al. 2021a) as an alternative image-text encoder of CLIP. A baseline "zero-shot "LIGN" selects the class of the highest text similarity with class text labels in text prompts such as "a photo of [class]". ". ALIGN is trained with a web-crawled image and text pairs as CLIP but on a slightly heavier image and text encoder (EfficientNet-L2 (Tan & Le. 2019) and BERT-Large (ennonn & Toutanova 2019)), which amount, to 172M parameters in total, compared to 151M parameters of CLIP. As the official model checkpoints are not available, we adopt the released checkpoints and the data preprocessor provided by a third party. | In this experiment, except for the backbone, all the experimental settings remain the same those in /hoou et al. (2023a]. We observe that MML exhibits more outstanding performance than baselines and show that MML. not specific to certain imago-text encoder.
httta://o..wkkpppeiaoor/////.//. Fage
20

---

Published in Transactions on Machine Learning Research (03/2025)
Table 14: Cross-dataset zero-shot transfer with ALIGN (Jia et al. 2021a] image-text encoder. MML is learned with to samples from 1000 ImageNet1K classes and transferred 10 other datasets with zero shot.
ImgNet. Caltech101 OxfordPets Cars Flowers Food Aircraft
SUN
DTD
EuroSAT
UCF
method
awg.
objects
pels
cars
Dowers food airplame scenes texcuures
land
actions
zero-shot ALIGN
65.9
79.7
62.5
69.5
53.0
76.1
8.3
41.5
51.4
23.1
61.1
51.4
ANN classifer Nakata et al. 2022
62.2
87.0
68.6
77.5
67.8
71.8
24.2
39.1
37.9
28.2
61.3
58.7
MML (etage
67.2 2
94.2
76.1
77.3
67.9
77.7
22.0
67.2
47.3
23.8
66.1
62.4
MML (ImageNet-4)
69.1
93.3
76.2
77.9
00.7
78.2
21.6
67.9
19.5
218
66.3
62.9
Table 15: Single-dataset zero-shot classification with CLIP Radford et al.
2021 and ALIGN
Jia et al.
2021a) image-text encoders.
ImageNet-S
CUB
CLIP B/32
CLIP L/14
ALIGN base
CLIP B/32
CLIP L/14
ALIGN
backbone
base
zero-shot CLIP CLII/ALIGN
82.8
90.1
85.1
61.9
73.8
55.9
ANN classifier Nakata et al. 2022 MMT.
76.4
86.5
82.4
67.9
82.6
64.1
83.0
91.1
86.1
75.6
87.8
73.7
100
EWC +
LWF +
Foster
100
iCARL + SLCA
Replay
- MML
80
75
60
50
40
25
20
0
10
15
2
6
8
Figure 7: Class-ineremental classification on CIFAR100 Basc5-inc5 (left), Bascl0-inc10 (right)
Class incremental classification on CIFAR100
We conduct experiments on CIFAR100 and compare MML with the other methods dedicated class- incremental classification in Fig. To reproduce the class-incremental learning baselines with CLIP, we adopt the unified code base from survey paper (Zhou et al. 2023a) and replace the backbone from ResNet18 to CLIP-B/32 Note that MML cannot be implemented with ResNet18 as CLIP-ResNet18 is unvaiilable. As the backbone is switched, we tune learning rates and epochs for each method and present their best results for fair comparison. The results show that MML performs comparably to the class-incremental learning models on CIFAR100 as well, but achieves less gain compared to the performance on ImageNet 100. We hypothesize that this is due to the majority of web-crawled images in the wild displaying a distribution that is markedly different from the x 32 objoct-centric images CIFAR.
Few-shot classification on mini-ImageeNet
In Table 16 we provide additional experimental result using CLIP-B/32 public few-shot image classification benchmark:
(Vinyals et al. 2016). We also include the two existing methods evaluated on the benchmark (He et al.] 2022 Hu et al. 2022), which leverage vision foundation model via fine-tuning it. The seminal sho- classification work based on deep learning (Vinyals et al., 2016) proposes this benchmark, thus it configures a toy experimental setup the nowadays perspective. Derived from ImageNet (Russakovsky et al. 2015), mini-ImageNNe consists of 84 84 sized downsampled images from 24/16/20 object classes for train/val/ten val/test splits, respectively. Compared to ImageNet-S with 600/200/200
21

---

MML can Published in Transactions on Machine Learning Research (03/2025)
Table 16: Comparison on few-shot classification
Table 17: Single-dataset zero-shot trans- fer on ImageNet-S with varying the tem- perature hyperparameter, т. CLIP-B/32 is used.
mini- EeageNet
ImageNet-S 200 classes shots
# target classes
classes shot
# shot
shots
Eicarrprob CLIP
Radford ct al.
2021
61.3
72.1
80.6
4
8
16
32
641
He et al.
2022
74.7
Hu eb al.
2022
95.3
MML 80.0 82.3 83.5
83.6
83.5
82.9
RACC
Long et al
2022
69.1
66.8
78.1
ANN classifier
Nakata al. 2022
96.6
77.2
77.2
MML (ours)
96.6
82.8
83.5
class split with the original image size, the benchmark is less challenging in terms of the smaller image size and the easier categorization difficulty. As shown in Table [6] kNN classifier and MML show comparable performance on mini-ImageNet probably because the casier problem sctup makes &NN classifier relatively powerful than the method involving learning. However, when it comes to the more complected benchmark, kNN classifier no longer powerful as MML on ImageNet-S. The nemory-modulas approach, which meta- learns to generalize unscen classes, is shown substantially more helpful on the more complex real-world classification scenarios than toy few-shot classification problem.
Supervised image classification
We also signify the efficacy of MML on the slandard supervised image classification in Table 19 also tackle this classification by setting the memory content and the prototype unchanged between training and testing to handle the known closed-set target classes.
Problem setup: In contrast to the unseen-class classification of the previous paragraphs, the more primitive definition of image classification assumes that the target classes are identical across training, validation, and testing. In other words, the generalization beyond the seen classes during training is not the focus of this classification task.
Baseline: Note that there exist no prior zero-shot methods that use dynamic memory, directly comparable to ours. A similar method with external memory retrieval, RAC (Long et al. 2022], is compared. RAC tackles seen class problems, whereas MML focuses on generalizing beyond seen classes with memory replacement.
Benchmark: The 11 datascts in Table I are used. For cach datasot, 4 random images per class are usod for training. All methods are evaluated on each dataset using the CLIP-RN50 backbone.
Results: The results are shown in Table 19 MML performs more accurately than other baselines on average. In particular, our model demonstrates greater effectiveness when supplemented with external web-crawled data that provide relevant features for classification. For instance, the diverse viewpoints and color variations of car images on the internet benefit. for car model categorization (Cars196). Overall, the proposed MMT. signifies its effectiveness also on the standard supervised and fine-grained image classification.
Logit temperature hyperparameter T.
Table 17 demonstrates the effect of the lemperature hyperparammeter 7 of Eq. 5 We empirically observe that adjusting the smoothness of the logit plays an important role in effective training. Note that higher value for encourages the logit value more indistinguishable to each other, i.c., smoothing effect. We observe that the loss for training hardly converges with - = 1 as the loss is not big enough to penalize the model. We thus increase the temperature value from to 64 and notice the trend; the performance reaches the highest peak at certain point, e.g., around 16 on ImageNet-S, and continues to drop afterward. This experiment shows that 7 controls the degree of the logit smoothness and is crucial hyperparamoter for effective training.
22

---

Published in Transactions on Machine Learning Research (03/2025)
Table 18: The first five sentences from the Wikipedia article of the "tench" class
The tench or doctor fish (Tinca tinca) is a fresh- and brackish- waterr fish of the order Cypriniformes found throughout Eurasia from Western Europe including the British Isles east into Asia as far the Ob and Yenisei Rivers. It is also found in Lake Baikal. It normally inhabits slow-moving freshwater habitats, particularly lakes and lowland rivers. The tench was formerly classified in the subfamily Leuciscinae with other Eurasian minnows, but more recent phylogenetic studies have supported it belonging to its own family Tincidae. The tench is most often found in still waters with a clay or muddy substrate and abundant vegetation. This species is rare clear waters across stony substrate, and is absent altogether from fast-flowing streams. It tolerates water with low oxygen concentration, being found in waters where even the carp cannot survive.
Table 19: Supervised classification results trained with 4 shots from the targel (seen) classes from each dataset
method
ImgNetiK Caltech101 OxfordPets Cars Flowers Food Aireraft SUN
DTD EuroSAT UCF avg.
zero-shot CLIP [Radford et al.
2021
58.2
86.3
85.8
55.6
66.1
77.3
17.3
58.5
42.3
37.6
61.5
58.8
linear prob Radtord et al. 2021
41.3
84.3
56.4
48.4
84.8
55.2
23.6
54.6
50.1
68.3
62.2
57.2
ProtoNet (uell et al. 2017
38.3
81.6
59.0
45.9
81.3
52.2
21.6
54.0
48.0
65.3
61.0
55.6
ENN classiter NNakata et al.
2022
55.7
79.7
59.9
55.8
65.6
60.4
20.5
50.2
29.0
19.4
45.2
49.2
RAC Longg et al. 2022
37.0
83.6
69.4
52.9
78.9
59.4
20.6
53.0
49.8
57.4
59.0
56.5
MML
66.2
90.2
64.8
84.1
76.7
21.3
65.8
52.4
44.1
70.1 65.6
Table 20: Performance with the increasing noise level in the memory to construct class protolypes on ImageNet-S
Table 21: Positive and negative similarity statis- Lies of the protolype classifier and MML
noise level
20
%
10
%
%
%
prototype classifier MML
of random instances
79.7
80.8
80.6
83.0
classification accuracy
80.1
83.0
avg of highest cross-modal consensus (purs)
83.0
82.7
82.7
83.0
avg positive class similority avg negative classes similarity
0.654
0.388
0.361
0.090
positive negative similarity gap
0.293
0.298
Robust performance with prototype construction from noisy data pool
Table 20 shows the robustness the proposed cross- moosllmodal consensus prototype generation method to the
data noise. As described in Section 3.1 the class prototypes are built by averaging the memory items that have the highest cross-modal similarity of the same class from the other modality memory. This prototype construction effectively ignores the noise in the memory by leveraging the cross-modal memory consensus.
We experiment with two copics of the memory. For one memory to construct zero-shot class prototypes, we randomly shuffle the memory items with other classes such that a certain percentage of memory items are from the web-crawled instances of other class names. For the other memory for ANN retrieval being fed to the subsequent knowledge integration, the memory is set unchanged.
Experimental setup:
Results: The consensus- based class prototype construction shown to be robust to the data noise. Even with the 20 % of mislabeled memory items, the class prototypes are constructed as accurately the clean prototype within the error bound. If the prototypes are the average of random memory instances of each class, the performance drops with the increasing noise level. However, the proposed consensus- protolype construction is able to ignore the unrelated noise as it uses only the most similar image memory items to the text memory, and from image memory to the text prototypes also
Analysis of class prototype similarity
Table 21 provides the average distance between all evaluation images and the prototypes of each model. This experiment investigates the behavior of the learnable integration of MML: how it updates the input image embedding to be close or distant to the class prototypes. Note that the class prototypes static, Le., both
23

---

Published in Transactions on Machine Learning Research (03/2025)
bald eagle imput image
bald eagle
bald eagle
bald eagle
bald cagle
bald cagle attn:
bald cagle 0.03
bald eagle
bald cagle
loggerhead turtle input Image
whole shoek
whole logperhead turtle redibreasted loggerhead turtle loggerhend turtle loggerhead turtle
Figure 8: An input query (leftmost) from ImageNet and its 8NN images retrieved from the WebVision memory using CLIP ViT-B/32. "attn" denotes the attention weight value, and the red frame denotes the image which gained the highest attention among 8NNs.
model uses exactly the same class prototypes, but MML updates the embodding through the knowledge integration layer. We observe that MML pushes the embedding to be far away especially from the negative class prototypes, achieving the strong final classification result.
Visualizations of retrieved kNNs.
Figure provides examples the image retrieval results. The leftmost image the input query from ImageNet, and its 8NNs retrieved from the WebVision image memory are presented on the right. Note that the retrieved 8NNs exhibit the superficially similar appearance of the query but often belong to different classes, e.g,, turtle floating on the sea. The subsequent attentive knowledge integration process then reweights the NNs with soft attention weight values. In the second example, the 1NN appears similar to the query but is a different class, however, its attention value is down-weighted and contributes insignificantly to the attentive aggregation. The attentive aggregation meta-learns to function independently of memory contents and able to perform effectively with unseen memory contents. We also present the assigned attention weight values in attentive knowledge aggregation, which are the similarity of the query and the NNs after Q and K projection in Eq. 3 The attention weight value is translated the learned aggregation weight for the NNs to what eztent each element is contributed for aggregation among the NNs to represent the query. The attentive knowledge integration process reweights the NN elements with soft attention weight values based on the attentive convex-combination similarity such that the semantically relevant NN elements are more attended.
24


Reviews:
[{"review_id": "veLTD4l3UL", "reviewer": "Reviewer_1vVW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper investigates an important setting in classification where each data point consists of multiple modalities (here is image-text pair). In classification, such a setting has not been widely studied, and hence, makes the problem significant and interesting. Given the additional modalities, classification performance could be improved further.\n- The proposed method in the paper is simple but effective. Its main idea replies on prototype-based classification with an additional step \"augmenting\" features of a sample of interest before measuring the similarity to each target prototype. Such a simplicity allows the proposed method to quickly adapts to new settings without the need to re-train the whole model.\n\n**Weaknesses**\n\nThe expecation after the introduction of the paper is how to fuse multiple-modalities to improve classification performance. However, when seeing Eq. (4) where the fusion happens by simply adding those modality embeddings, there is no \"wow\", but a slight disappointment.\n\nFurthermore, the paper, in a loose sense, can be considered as an extension of Prototypical Networks (Snell et al. 2017) applied on multi-modality data. The main difference is at \"augmenting\" features of the sample of interest by using the features of its k-nearest-neighbours as shown in Eqs (2) and (3). Despite its empirical effectiveness, it lacks the explanation or intuition on why such steps improve the results compared to Prototypical Networks.\n\nAnother concern is that the proposed method heavily relies on models that are well-trained to extract features. This could be useful in some common settings, such as vision-language models on text and natural images. However, when working on uncommon domains, such as medical or space exploration, where there is no pre-trained models, extracting good features to measure their similarity leads to a severe issue and could break the proposed method. Thus, claiming that there is no need to re-train the whole model might not be exactly correct, especially when using pre-trained ones.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies a multi-modality in terms of input data where each datum is a pair of an image and its text description to perform classification. In particular, the paper proposes a modular memory based method that can modify or replace its modular memory without re-training the whole model to perform classification and zero-shot learning. The main idea is to employ k-nearest neighbours to retrieve some samples in each target classes to integrate them into the features of the sample of interest before calculating its (the sample of interest) similarity to each prototype for classification. Due to such modularity, the proposed method is quite flexible, especially, it only needs to learn the cross-attention to integrate the nearest neighbours into the (image and text) features of a sample of interest. In terms of empirical performance, the proposed method shows promising results in several benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In the current form, the paper is quite simple and does not clearly differentiate itself from previous studies, especially compares to Prototypical Networks and CLIP. The main request is to understand more about the differences, explain why additional steps proposed in Eqs. (2) and (3) help to improve classification results, as well as ablation studies when there is no such step involved or when one of those two steps has high errors.\n\nAnother request is to minorly revise the section of Related Work, where one relevant study \"Infinite Mixture Prototypes for Few-Shot Learning\" by Allen et. al presented at ICML 2019.\n\nSome minors to improve the clarity of the paper:\n- The abbreviation \"NN\" is mentioned early in the paper, but its full-form is only explained at page 8.\n- The abuse of notations in Eq. (4) should be avoided and each element of the similarity vector z should be used. What I mean is as follows:\n\n$$\nz_{c} = \\operatorname{cosine\\\\_sim}(p\\_{c}^{\\mathrm{img}}, f) +  \\operatorname{cosine\\\\_sim}(p\\_{c}^{\\mathrm{txt}}, f).\n$$", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ptkrYOXGap", "reviewer": "Reviewer_7Wbr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- Innovative Use of Modular Memory: The concept of leveraging replaceable external memory for flexible adaptation to unseen tasks is innovative and potentially impactful.\n\n- Robust Experimental Results: The authors provide a wide range of experiments to validate the model's performance across different tasks, showcasing robustness against noisy data and generalization to unseen classes.\n\n- Efficiency Considerations: The paper addresses computational overhead effectively, showing the method's scalability and lightweight implementation.\n\n\n**Weaknesses:**\n\n1. Introduction Lacks Method Details:\nThe introduction outlines the motivation and general approach of MML but fails to discuss its specific workings. Readers unfamiliar with the concept may struggle to connect the proposed framework with the results presented later. For example, the introduction should briefly touch on key ideas such as prototype-based classification and the retrieval-augmented learning process.\n\n2. Insufficient Related Work on Class-Incremental Learning:\nThe discussion on class-incremental learning (CIL) is incomplete and somewhat misaligned with prior research. Although the authors differentiate MML from CIL approaches reliant on distillation, they overlook works that similarly incorporate external memory or modular components. A more comprehensive exploration of CIL strategies and their limitations would strengthen the paper.\n\n3. Overreliance on External Factors:\nWeb-based Data Collection: MML's dependence on web-crawled data introduces potential biases and limitations. The reliance on keyword-based retrieval may result in noisy or irrelevant samples, especially when the search engine model has domain-specific calibration issues.\nZero-shot Class Prototypes: The method generates class prototypes using zero-shot predictions, which are inherently noisy. This reliance may propagate errors, particularly in domains where text-image alignment is miscalibrated.\n4. Model Dependency on Pre-trained Encoders:\nThe proposed architecture is heavily reliant on pre-trained encoders like CLIP for both memory construction and feature extraction. This dependency raises concerns about the generality of the approach if applied to domains not well-supported by such models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel architecture, Memory-Modular Learner (MML), designed for image classification tasks. MML introduces the idea of external, replaceable memory populated by web-crawled image and text data to enable generalization to unseen classes without retraining. It demonstrates applications in zero-shot, few-shot, fine-grained, and class-incremental classification tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-  While the authors demonstrate robustness to noisy data, they could provide more detailed analyses on the quality of web-crawled memory contents. For instance, how do different noise levels or retrieval methods affect the generated prototypes?\n- Addressing how MML handles domain shifts or misaligned retrievals would be valuable. Could the system incorporate mechanisms to detect or filter irrelevant memory items dynamically?\n- While the modular memory concept is compelling, the paper could include more detailed ablations to assess how memory size and content diversity impact performance.", "broader_impact_concerns": "The method provides the opportunity to easily adapt a pre-trained foundation model to different downstream tasks, which is an important research direction for the present time. The solution holds the potential to revolutionize the foundation model adaptation literature.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x8aD6qutxV", "reviewer": "Reviewer_9m1F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: (1) A new image classification framework using replaceable memory. (2) Comprehensive evaluation across multiple downstream image classification tasks. (3) The experiments show promising performance with robustness to noisy web data, and computational efficiency in terms of memory usage. \n\nWeakness: (1) While the work targets the image classification task, the framework is still similar to in REVEAL (Hu, et al. CVPR2023), it would be better to put more emphasis on the difference and also which proposed component really makes this framework unique and effective. (2) There is limited discussion about the size of external memory given the concern when deploying it to very large real-world tasks. (3) The experiments should include additional ablation studies to validate the use of some key components in the framework such as the use of cross-attention for knowledge integration and the NN retrieval. (4) The comparison with different downstream tasks could be unfair as the proposed method use web-crawled data while the baseline use curated data only. In addition, for class-incremental learning, did all the compared methods use the pre-trained CLIP model or not?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a memory-modular learner (MML) for image classification that separates knowledge storage from model reasoning. The key innovation is using modular external memory that can be updated/replaced without model retraining. The memory consists of web-crawled images and text data collected through keyword searches. The model uses cross-attention to integrate memory with input and prototype-based classification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) emphasize the specific technical differences from REVEAL (Hu, et al. CVPR2023), especially the design of the framework such as the use of K-NN instead of learnable modules for retrieval or so. \n\n(2) The experiments need more details to ensure the fairness of comparisons, and also include more details when conducting on different image classification problem setups.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were in agreement about recommending acceptance for this paper. The clear writing and comprehensive experiments were recognized as strengths while similarity to previous work and dependence on pretrained features were raised as weaknesses. The authors clearly addressed each of the reviewers' concerns during the rebuttal phase.

In the camera-ready version, the authors are urged to ensure that the appendix is contained inside the main pdf (after references).
Decision: Accept (poster)