Towards flexible perception with visual memory

## Abstract

Abstract Training a neural network is a monolithic endeavor, akin to carving knowledge into stone: once the process is completed, editing the knowledge in a network is nearly impossible, since all information is distributed across the network’s weights. We here explore a simple, compelling alternative by marrying the representational power of deep neural networks with the flexibility of a database. Decomposing the task of image classification into image similarity (from a pre-trained embedding) and search (via fast nearest neighbor retrieval from a knowledge database), we build a simple and flexible visual memory that has the following key capabilities:
(1.) The ability to flexibly add data across scales: from individual samples all the way to entire classes and billion-scale data;
(2.) The ability to remove data through unlearning and memory pruning;
(3.) An interpretable decision-mechanism on which we can intervene to control its behavior.
Taken together, these capabilities comprehensively demonstrate the benefits of an explicit visual memory. We hope that it might contribute to a conversation on how knowledge should be represented in deep vision models—beyond carving it in “stone” weights.

### 1 Introduction

In the pretty diagrams on "Intro to Machine Learning” slides, an ideal ML workflow looks like this: Data collection, preprocessing, choosing a model, training, evaluation, deployment. Happy ending—the model is deployed, the users love it, and one can finally go on that well-deserved vacation and catch up on the latest AGI memes.

Until, of course, the enemy of any ideal world sets in: reality. The real world constantly keeps changing, and so do data requirements. New data and datasets become available, and existing ones become deprecated for a variety of reasons, including concerns around fairness, biases or unsafe content. Knowledge changes, and concepts drift [; ]: Phones and cars look different today than they did a few years ago, and different from how they will look in the future. When it comes to data, the only constant is change [; ; ; ]. Consequently, from a modeling perspective, in order to keep up with this change one would ideally want to constantly re-train or fine-tune models, which is of course not feasible. In short, as anyone who has ever deployed a model has experienced firsthand, one is constantly battling the symptoms of a single underlying cause: the fact that deep learning models have a static knowledge representation entangled in millions or billions of model parameters. We, among many others working on memory [e.g. ; ; ; ; ; ; ; ], believe that this is not a great way to represent visual knowledge for deep learning. Instead, we argue that we should build models that cleanly separate representation (*how* things are represented, for instance through feature embeddings) from visual memory (*what* is known). In short, deep learning models need a flexible visual memory: a way to explicitly utilize and edit knowledge.

In this work, we build a simple visual memory for classification and show that it has seven desirable capabilities, including the ability to flexibly add data across scales (from individual samples to classes and even billion-scale data), the ability to remove data from our model’s classification process through machine unlearning and memory pruning, and a simple, interpretable decision-mechanism on which we can intervene to control its behavior. Our main goal is to provide a compelling idea of how beneficial a flexible visual memory for deep learning can be from a variety of perspectives and capabilities. From a technical standpoint, we aim for simplicity: retrieving $k$ nearest neighbors (in an embedding feature space) along with their labels to classify a query image. This approach allows us to investigate where a simple visual memory mechanism helps, where its limitations may be, and where there might be opportunities for improvement through a more complex system. We hope that by demonstrating clear benefits from a simple visual memory, this article might contribute to a conversation on how knowledge ought to be represented in deep vision models.

Here are some highlights of this article:

- 1.
Improved aggregation of retrieved memory samples: we introduce *RankVoting*, a power-law weighting function that surpasses previous state-of-the-art (SoftmaxVoting).
- 2.
Re-ranking samples using a vision-language model achieves 88.5% top-1 ImageNet validation accuracy, improving over both DinoV2 ViT-L14 kNN and linear probing.
- 3.
Flexible perception: the visual memory achieves perfect unlearning, scales to billion-scale data without additional training, and enables controlling sample influence via *memory pruning*.

Code and data are available here: [https://github.com/google-deepmind/visual-memory](https://github.com/google-deepmind/visual-memory)

Related work. The concept of a visual memory has a long history in ML, neuroscience and psychology. In psychology, *exemplar theory* posits that humans recognize objects by comparing them to existing examples in visual memory [; ; ; ; ], like the ALCOVE model []. In ML, prior to deep learning, *instance-based learning* (also known as memory-based learning) was a popular alternative to *model-based learning* [; ]. In recent years, hybrid approaches have started to combine the benefits of both approaches. For instance, deep neural network variants (model-based since they learn generalized abstractions of data) of $k$-nearest neighbor algorithms (instance-based since they compare new data to existing exemplars in memory) have been proposed with various motivations, including few-shot learning [; ; ], improving adversarial robustness [; ; ], medical image classification [], confidence calibration [], interpretability [; ; ; ], image denoising [], retrieval-augmented learning [; ], anomaly and out-of-distribution detection [; ]. Recently, [] tested a kNN-based visual memory up to ImageNet-scale (1.28M images), and [; ] applied kNN-based approaches to neural language models.

### 2 Building a retrieval-based visual memory for classification

Given a dataset $\mathcal{D}_{\mathsf{test}}:=\{(\tilde{\boldsymbol{x}}_{1},y_{1}),\cdots,
\tilde{\boldsymbol{x}}_{n},y_{n}\}$, we want to classify each image $\tilde{\boldsymbol{x}}_{i}\in\mathcal{D}_{\mathsf{test}}$. Our classification approach consists of two steps: (i) building a visual memory, and (ii) fast nearest neighbor based inference using the visual memory.

#### 2.1 Building a visual memory

Our visual memory retrieves (image, label) pairs from an image dataset when a query is made by directly retrieving those images that are considered similar to a test image according to a model. The model is a fixed pre-trained image encoder, meaning that no training takes place when adding information to visual memory. No copies of the dataset are stored in the visual memory. Instead, feature maps are extracted from the model based on a set of images related to the downstream classification task at hand, such as a standard training set. For our experiments, our visual memory comprises of features extracted from a dataset like the ImageNet-1K [] training set using different encoders like DinoV2 [] and CLIP []. Thus, given a pretrained image encoder, $\Phi$, and a dataset of (image, label) pairs $\mathcal{D}_{\mathsf{train}}:={(\boldsymbol{x}_{1},y_{1}),(\boldsymbol{x}_{2},
y_{2}),\cdots,(\boldsymbol{x}_{N},y_{N})}$, we obtain features $\boldsymbol{z}_{i}:=\Phi(\boldsymbol{x}_{i}),\forall\boldsymbol{x}_{i}\in
\mathcal{D}_{\mathsf{train}}$. Subsequently, the feature maps and corresponding label pairs are put in a database thereby creating $\mathsf{VisualMemory}:=\{(\boldsymbol{z}_{1},y_{1}),(\boldsymbol{z}_{2},y_{2})
,\cdots,(\boldsymbol{z}_{N},y_{N})\}$ for classification. For both DinoV2 and CLIP, we use the last image embedding layer as a feature space.

#### 2.2 Retrieval-based classification using visual memory

Given a query image $\tilde{\boldsymbol{x}}\in\mathcal{D}_{\mathsf{test}}$, we extract its feature map, $\tilde{\boldsymbol{z}}=\Phi(\tilde{\boldsymbol{x}})$. We then query $\mathsf{VisualMemory}$ to extract $k$ feature vectors, $\mathsf{Neighbors}({\tilde{\boldsymbol{x}}}):=\{(\boldsymbol{z}_{[1]},y_{[1]})
,(\boldsymbol{z}_{[2]},y_{[2]}((,\cdots,(\boldsymbol{z}_{[k]},y_{[k]})\}$, that are closest to the query features $\tilde{\boldsymbol{z}}$ using the cosine distance, which is the default retrieval similarity measure for SSL models like DinoV2 []. The extracted neighbors, $\mathsf{Neighbors}({\tilde{\boldsymbol{x}}})$, are ordered by distance i.e.

$$ $\displaystyle\mathsf{dist}(\tilde{\boldsymbol{z}},\boldsymbol{z}_{[i]})\leq \mathsf{dist}(\tilde{\boldsymbol{z}},\boldsymbol{z}_{[j]}),\leavevmode\nobreak \ \forall i\leq j.$ $$

We then assign a weight, $w_{i}$, to each neighbour $(\boldsymbol{z}_{[i]},y_{[i]})$ and aggregate the scores for each neighbour with the same label. Finally, we assign that label to the query image with the highest aggregate score. We implemented $\mathsf{VisualMemory}$ retrieval based classification using one of the following two approaches:

1. Fast inference using matrix multiplication on GPUs/TPUs: For smaller datasets like ImageNet, we saved $\mathsf{VisualMemory}$ as a matrix of size $\mathsf{num_images}\times\mathsf{num_dims}$. During inference, for an encoded query image of size $1\times\mathsf{num_dims}$, we computed the dot product of this encoded image with every entry in $\mathsf{VisualMemory}$ getting a matrix of size $\mathsf{num_images}\times 1$. We then computed the $k$ nearest neighbors using the $\operatorname*{arg\,max}$ operation.

2. Fast and scalable nearest neighbor search: We used ScaNN [] for accelerating nearest neighbor search at scale. Specifically, we saved the $\mathsf{VisualMemory}$ as a database and used ScaNN for fast lookup of nearest neighbors during inference. This method scales easily to billion-scale memory (cf. Section 3.3).

Figure: (a) ImageNet
Refer to caption: x1.png

We mentioned earlier that we retrieve a set of neighbors, $\mathsf{Neighbors}(\tilde{\boldsymbol{x}})$ and aggregate information across them to make a classification decision. In order to understand how reliable (i.e., accurate) retrieved memory samples are from the first to the 100th neighbor, we systematically analyze neighbor reliability in Figure 1. As expected, reliability decreases as the neighbor index $k$ increases, but even at large $k$ the neighbors contain above-chance information about the ground truth class. This suggests that aggregating information across different neighbors may be beneficial to decision-making, leading to the question: *What is the best aggregation strategy?* We empirically study this by testing different weighting strategies for aggregation:

Plurality voting: Each neighbour in $\mathsf{Neighbors}({\tilde{\boldsymbol{x}}})$ is assigned an equal weight of 1.0. This is the classic, most simple voting method and used for instance by [].

Distance voting: Each neighbour in $\mathsf{Neighbors}({\tilde{\boldsymbol{x}}})$ is assigned a weight based on its Cosine distance to the query image $\tilde{\boldsymbol{x}}$ i.e. $w_{i}=\exp\big{(}-\mathsf{dist}(\tilde{\boldsymbol{z}},\boldsymbol{z}_{[i]})
\big{)}$. This approach has been used by [] for nearest neighbor language models.

Softmax voting: Each neighbour is assigned a weight based on the softmax function i.e. $w_{i}=\mathsf{softmax}\big{(}\mathsf{dist}(\tilde{\boldsymbol{z}},\boldsymbol{
z}_{[i]}\big{)},\tau)$ where $\tau$ is the temperature. This voting method is considered state-of-the-art; for example nearest neighbor accuracies of self-supervised models are reported using this method. A temperature of $\tau=0.07$ frequently appears in literature [; ; ] and is reported as a parameter “which we do not tune” in the Dino paper [, p. 18]. We observe that performance is sensitive to this parameter; other temperatures perform worse. We therefore follow the literature in using $\tau=0.07$.

Rank voting (ours): We propose a new, simple aggregation approach wherein each neighbour is assigned a power-function weight based on its rank in the ordered set $\mathsf{Neighbors}({\tilde{\boldsymbol{x}}})$ i.e. $w_{i}=1/(\alpha+\mathsf{rank}_{i}$) where $\mathsf{rank}_{i}$ is $i$ and $\alpha$ is an offset to avoid division by zero that is set to 2.0.

Figure: (a) Existing aggregation methods, DinoV2 ViT-L14
Refer to caption: x3.png

In Figure 2(a), we compare the top-1 ImageNet validation accuracy of different ranking methods as a function of number of neighbours, with the ImageNet-1K training set as the visual memory using the DinoV2/ViT-L14 model as the featurizer. Paradoxically, existing aggregation methods like plurality voting, distance-based voting, and softmax voting show *decaying* performance as the provided information (number of nearest neighbors) *increases*. This suggests that the methods are overconfident in distant neighbors, assigning them too much weight. Our simple, parameter-free rank based voting method, however, leads to an increase in performance with more neighbors until a certain $k$ after which the performance plateaus, which is the ideal scenario (Figure 2(b)). Furthermore, our rank-based voting also outperforms baselines in absolute terms; quantitative comparisons can be found in the Appendix (Tables 4, 5, 6, 7, and 8) where we also study the influence of hyperparameters (Figure 9). This indicates that a simple, power-function based method can reliably integrate information across retrieved memory samples.

Gemini re-ranking. Our results above demonstrate that different aggregation strategies have a large impact on downstream performance. How far can we push the upper limit on aggregating information from different neighbors? We perform a controlled experiment using the Gemini 1.5 Flash model [] to test this: We add the 50 nearest neighbors from DinoV2 ViT-L14 for a query image along with their labels into Gemini’s context. We then query Gemini to predict the query image’s label. This achieves 88.5% ImageNet validation accuracy, a substantial improvement over both DinoV2 ViT-L14 kNN (83.5%) and linear probing (86.3%) performance. Interestingly, the performance of the Gemini model is mainly driven by the neighbor information since it only achieves 69.6% accuracy without neighbors (when just the query image is provided to the model). The performance improvement highlights the potential of using vision-language models as a visual memory re-ranker. Given that our main goal is to explore a simple visual memory system, we mostly focus on non-Gemini ranking methods throughout our analysis.

### 3 Capabilities of a visual memory

Our primary goal is to motivate the concept of a machine *visual memory* from a variety of different perspectives. To this end, we investigate how such a memory can benefit the following capabilities:

- 3.1
Flexible lifelong learning: adding novel OOD classes
- 3.2
Flexibly trading off compute and memory
- 3.3
Flexibly adding billion-scale data without training
- 3.4
Flexible removal of data: machine unlearning
- 3.5
Flexible data selection: memory pruning
- 3.6
Flexibly increasing dataset granularity
- 3.7
Interpretable & attributable decision-making

#### 3.1 Flexible lifelong learning: adding novel OOD classes (both data and labels)

Standard classifiers, whether trained end-to-end (supervised models) or with a linear classifier (self-supervised models), are not able to handle new information without re-training. For instance, adding new classes or changing labels in an existing model usually involves either re-training or fine-tuning parts of the model. A retrieval-based visual memory, in contrast, is able to process such information in a natural and flexible way, aligning with the requirements of lifelong learning []. We tested this by adding data for 64 new classes, along with their new labels, to the visual memory of a pre-trained DinoV2 ViT-L14 model (in addition to the ImageNet train set, which is in-distribution for the model). We took the new classes from the NINCO dataset [], a dedicated OOD dataset that is designed to have no overlap with existing ImageNet labels and samples. This requires the model to transfer what it has learned to new, unseen concepts. The new task is therefore harder, as the model has to retrieve images from both in-distribution and OOD classes. The resulting visual memory has 1064 classes (1K from ImageNet and 64 from NINCO). Table 1 shows that with a visual memory it is possible to add new classes such that the in-distribution accuracy is maintained without catastrophic forgetting (the new classes only change ImageNet validation performance by 0.02–0.04% depending on the aggregation method), while at the same time reaching very high accuracy on the new OOD classes (approx. 87% top-1) without any training. Figure 11 in the appendix confirms that the samples are indeed OOD for the model, as demonstrated by larger distances to nearest neighbors. This highlights that a visual memory is capable of flexibly adding new information—an important capability since the world is not static.

**Table 1: Flexible lifelong learning: adding novel OOD classes. A visual memory of DinoV2 ViT-L14 with ImageNet-train (IN-train) as the memory database is able to handle a simple “insert into memory” operation for 64 out-of-distribution classes (data and labels) from the NINCO dataset [], leading to high performance on the new classes without affecting top-1 ImageNet validation accuracy.**
| memory $\rightarrow$ | IN-train | IN-train-and-NINCO |  |
| --- | --- | --- | --- |
| query $\rightarrow$ | IN-val | IN-val | NINCO |
| no aggregation | 81.1 | 81.1 | 86.4 |
| PluralityVoting | 83.2 | 83.2 | 86.9 |
| DistanceVoting | 83.3 | 83.3 | 87.1 |
| SoftmaxVoting | 83.6 | 83.5 | 87.5 |
| RankVoting | 83.6 | 83.6 | 87.4 |

#### 3.2 Flexibly trading off compute and memory

Next, we turn our attention to studying the scaling behaviour of visual memory with increasing memory model size. We hypothesize that bigger models will be able to attain similar performance as smaller models with lesser amount of visual memory. This is because, all else being equal, a bigger model should be a better featurizer that requires fewer examples in memory to represent different concepts. We empirically study the scaling behaviour of visual memory based retrieval systems in Figure 3(a) using models of different sizes like DinoV2 ViT models of sizes S/14 (21M params), B/14 (86M params), and L/14 (300M params), as well as CLIP ViT models of sizes B/16 and L/14. We plot the top-1 error rate as a function of number of images in visual memory. The plot demonstrates that for each model, the error rate consistently decreases as we increase the visual memory size. Notably, already with a single exemplar per class in memory, ImageNet validation performance is far beyond chance (41% top-1 error for DinoV2 ViT-L14). It also visualizes the possibility of a flexible trade-off between model size and memory size: e.g. for the different DinoV2 models, the S/14, B/14, and L/14 variant achieve similar performance at 1.28M, $\sim$150K, and $\sim$70K memory capacity respectively. In line with [], this indicates that a smaller model with large memory can match the performance of a larger model with smaller memory.

Figure: (a) ImageNet (million-scale)
Refer to caption: x5.png

#### 3.3 Flexibly adding billion-scale data without training

Billion-scale dataset with pseudo labels.
As demonstrated in Section 3.2, performance systematically improves with increased memory size across both small and large models. We here test how far this trend holds beyond relatively small-scale, well-curated settings like ImageNet-1K by scaling visual memory to the billion-scale unlabeled data regime. We study this by obtaining a large-scale dataset from the union of the ImageNet-1K train set and a subset of the JFT-3B dataset []. To this end, we treat JFT as an unlabeled dataset by ignoring its original labels and instead obtaining *pseudo labels* by running them through ViT-22B-224px [], a highly performant classifier. We excluded images whose labels do not have a correspondence with the ImageNet labels.

Scaling. In Figure 3(b), we show the downstream performance of two DinoV2-ViT model variants on ImageNet validation examples as a function of visual memory size. The plot demonstrates that even in the billion-scale data regime, validation error decreases when increasing memory size without any training. The gain from more data is most prominent when having fewer samples in memory (e.g., going from 1 to 10 samples per class). In log-log space, a logarithmic function fits the empirical scaling trend well. In the literature, simple scaling trends such as the one we observe are powerful predictors of scaling behaviour for different model and dataset sizes [; ; ].

Out-of-distribution performance. In order to understand whether the benefits of increased memory size transfer to out-of-distribution (OOD) data, we compared DinoV2 ViT-L14 once with ImageNet-train in memory and once with JFT pseudo-labels in memory. The models are evaluated on the ImageNet-A [], ImageNet-R [], ImageNet-Sketch [], ImageNet-V2 [], and ImageNet-ReaL [] datasets. As an additional well-performing yet “inflexible” baseline, we report linear probing accuracies from the DinoV2 paper []. Table 2 shows that visual memory scaled with JFT data improves OOD performance across all datasets compared to an ImageNet-based visual memory. Gemini re-ranking again improves leads to performance gains. Overall, the finding that memory scale transfers to OOD improvements is important in the context of continual learning, where a flexible visual memory can easily incorporate newly available data that the model was not trained on and improve performance both in- and out-of-distribution.

**Table 2: OOD evaluation. Out-of-distribution performance improves with larger visual memory size. Across all OOD datastes, a visual memory of DinoV2 ViT-L14 with JFT-3B (JFT memory) outperforms ImageNet memory demonstrating advantages of scaling visual memory for OOD performance.**
| Model | Method | IN-A | IN-R | IN-Sketch | IN-V2 | IN-ReaL |
| --- | --- | --- | --- | --- | --- | --- |
| DinoV2 ViT-L14 | linear probe | 71.3 | 74.4 | 59.3 | 78.0 | 89.5 |
| DinoV2 ViT-L14 | ImageNet memory | 58.8 | 62.8 | 61.5 | 75.6 | 87.1 |
|  | + Gemini re-ranking | 68.4 | 72.3 | 72.5 | 81.7 | 89.9 |
| DinoV2 ViT-L14 | JFT memory | 61.1 | 73.7 | 68.0 | 77.6 | 88.2 |
|  | + Gemini re-ranking | 69.6 | 81.4 | 75.0 | 82.3 | 90.5 |

#### 3.4 Flexible removal of data: machine unlearning

The world is not static. Thus, in addition to the need to flexibly add novel data, there is often a need to remove the influence of specific training data from a model’s decision-making process after it has been trained [; ; ; ]. A range of intricate methods are being developed to remove or reduce the influence of certain training samples [; ; ; ; ]—a challenging endeavour if knowledge is embedded in millions or billions of model weights. In contrast, for models with an explicit visual memory, machine unlearning becomes as simple as removing the dataset sample from the visual memory. For instance, after adding the NINCO dataset [] into visual memory, we can remove any NINCO sample with outstanding performance on all three key unlearning metrics reported by []: *Efficiency*: How fast is the algorithm compared to re-training? (Lightning fast.) *Model utility*: Do we harm performance on the retain data or orthogonal tasks? (Not at all.) *Forgetting quality*: How much and how well are the ‘forget data’ actually unlearned? (Completely and entirely.) Can machine unlearning therefore be solved with a visual memory? If the embedding model is trained on data that needs to be unlearned, machine unlearning remains challenging. If, however, the embedding model is trained on a safe, generalist dataset (e.g., a publicly available image dataset) and data that may need to be considered for unlearning later is simply put into the visual memory, then machine unlearning indeed becomes as simple as deleting a datapoint from the visual memory. This can be particularly helpful for tasks that may require private or confidential data—a model can be trained on publicly available datasets to learn general and information features and the private data can be added to a visual memory on local devices for downstream tasks to preserve privacy.

#### 3.5 Flexible data selection: memory pruning

The ability to flexibly remove the influence of certain datapoints is not just desirable in the unlearning sense, but also advantageous in the context of dataset pruning, an emerging field that analyzes the quality of individual data points. The goal of dataset pruning is to retain only *useful* samples, while removing those that have a neutral or harmful effect on model quality. The key challenge is that in standard black-box models, it is entirely unclear whether any given sample is helpful or harmful. The gold standard is leave-one-out-training (for ImageNet, this would consist of training 1.28 million models); current methods seek to approximate this extremely costly approach with various heuristics [; ; ; ; ]. By contrast, the contribution of a data sample to decision-making in a visual memory based system is straightforward. For any given query image $\tilde{\boldsymbol{x}}$, the neighbor set $\mathsf{Neighbors}(\tilde{\boldsymbol{x}})$ clearly reveals which samples contributed to the decision. Furthermore, this information also highlights whether the samples were helpful (correct label) or harmful (wrong label) for the decision. We, therefore, transfer the concept of dataset pruning to memory, and propose visual *memory pruning*. To this end, we estimate sample quality by querying the ImageNet training set against a visual memory consisting of the exact same dataset (IN-train, discarding the first neighbor which is identical to the query). This approach requires no more compute than a single forward pass over the training set. We then record the number of times any given neighbor contributed to a wrong decision, resulting in a sample quality estimate. This enables us to exclude low-quality neighbors from the decision-making process by either removing them from the visual memory entirely (“hard memory pruning”) or by reducing their weight compared to higher-quality neighbors (“soft memory pruning”). Method details can be found in Appendix G. In Table 3, we show that both memory pruning variants improve ImageNet validation accuracy, with soft pruning leading to larger gains than hard pruning. Figure 4 visualizes the decision-making process for a randomly selected sample where estimating sample reliability improves decision quality. Given that observing the outcome of an intervention is many orders of magnitude faster in visual memory models (as opposed to traditional leave-out-training), we are optimistic that the visual memory pruning gains we observed with two simple strategies can be improved further in the future.

Figure: Figure 4: Visualization of memory-based decision-making with and without memory pruning. Given a query image, nearest neighbors are retrieved from memory via Cosine similarity in the embedding space of a model (here: five closest neighbors from the ImageNet train set, embedded via DinoV2 ViT-L14). The model’s prediction is based on the weighted aggregation of the neighbor class labels. The rank-based weight decreases with the rank of the neighbor. For soft memory pruning, those weights are adjusted by the reliability of their neighbors. In the specific example here, all five neighbors appear sensible, but they have four different labels. Since the first two neighbors contributed to wrong decisions on the training set, they are downweighted via soft memory pruning, and the prediction changes to the correct class.
Refer to caption: extracted/5860709/figures/pruning/pruning_figure.png

**Table 3: Flexible data selection: memory pruning. ImageNet validation accuracy improves when removing low-quality samples (hard pruning) or downweighting them (soft pruning). In contrast to standard black-box models, memory models (here: using DinoV2 ViT-L14) offer a strikingly simple way to estimate sample quality since their decisions are based on a few retrieved memory samples.**
| Pruning | PluralityVoting | DistanceVoting | SoftmaxVoting | RankVoting |
| --- | --- | --- | --- | --- |
| no pruning (standard) | 83.2 | 83.3 | 83.6 | 83.6 |
| hard pruning (ours) | 83.3 | 83.4 | 83.6 | 83.7 |
| soft pruning (ours) | 83.6 | 83.6 | 83.9 | 84.1 |

#### 3.6 Flexibly increasing dataset granularity

In contrast to static classification, where a model is trained once without further updates, a visual memory model should be able to flexibly refine its visual understanding as more information becomes available. We test this using DinoV2 ViT-L14 embeddings on the iNaturalist21 dataset [], a large-scale imbalanced dataset of animal and plant images containing 10,000 species spanning seven taxonomic levels, from coarse (kingdom) to fine-grained (species). In a leave-one-out fashion, we simulate the discovery of a new species by putting 50 exemplars for each of the 9,999 species into memory and then step by step adding more data for the remaining “newly discovered” species—starting from zero exemplars all the way to 50 exemplars (see Algorithm 1 for an algorithmic description). In Figure 5 we observe the following: (1.) Already before a single example of the new species is added, it can already be placed in the right part of the taxonomic tree well beyond chance (35.2% accuracy at the genus level compared to $\sim$0% chance accuracy). (2.) Accuracy at the species level improves substantially by adding just a handful of images of the target species (e.g., 5–10 images); a regime where training a classifier would typically fail due to data scarcity. (3.) Interestingly, adding more samples of the discovered species not only improves species-level accuracy, but also leads to a “rising tide lift” of improvements across all levels of the taxonomic hierarchy. Overall, this indicates that a visual memory is well-suited for hierarchical classification tasks and settings where data for new concepts is initially scarce but becomes more abundant over time—which is often the case in applications like fraud detection, personalized recommender systems, and scientific discovery.

Figure: Figure 5: Impact of memory bank size on top-1 accuracy across taxonomic levels on iNaturalist. Top-1 accuracy for a target species across different taxonomic levels as the number of exemplars in the memory bank for that species increases from 0 to 50. Each line represents the average accuracy over all 10,000 species in the iNaturalist 2021 dataset, while the number of examples in visual memory is fixed at 50 exemplars for all other species. The black dotted line indicates baseline accuracy from predicting the majority class.
Refer to caption: x7.png

#### 3.7 Interpretable & attributable decision-making

##### Attributing model decisions to training data.

Unlike a black-box deep learning model, a visual memory system offers a natural way to understand a model’s specific predictions by attributing them to training data samples [e.g. ]. In Figure 6, we visualize misclassified validation set examples from the ImageNet-A dataset [] using a memory of the ImageNet-1K training set. These randomly selected samples illustrate that many seemingly strange errors (e.g., predicting a type of fence instead of a teddy bear, or a unicycle instead of a bow tie) do in fact appear sensible given the data, raising questions about label quality of ImageNet-A—in a similar vein as label issues identified for ImageNet [; ; ]—rather than about model quality.

##### Understanding compositionality of representations.

A flexible visual memory also provides a path to analyze representations of various models, particularly, how different models represent multiple concepts in an image. We study this for an ImageNet-train visual memory of DinoV2 ViT-L14 and CLIP ViT-L14. We use manually selected query images from outside the ImageNet dataset that have multiple objects from the ImageNet labels. We query the visual memory for nearest neighbors of the query image. Subsequently, we obtain the *residual image* by subtracting the features of the nearest neighbor from the features of the query image. We, then, obtain the nearest neighbors for the residual image from the visual memory. We plot the results in Figure 14 which shows that DinoV2 ViT-L14 and CLIP ViT-L14 represent concepts in their features in a different manner. The nearest neighbors for DinoV2 are mostly images with a single concept (or object) from the query image. The residual image, subsequently, leads to nearest neighbors dominated by another single object in the query image. In contrast, CLIP often results in nearest neighbors that are generally a blend of concepts from the query image. These qualitative explorations are simple demonstrations of the advantages of an interpretable decision-making process provided by a flexible visual memory.

Figure: Figure 6: Interpretable decision-making. A retrieval-based visual memory enables a clear visual understanding of why a model makes a certain prediction. Here, we show four randomly selected misclassified query images from ImageNet-A [] along with five nearest neighbors from DinoV2 ViT-L14 using the ImageNet-1K training set as visual memory. All labels are from the respective datasets (ImageNet-A for query and ImageNet-train for neighbors). While all neighbors visually look reasonable, not all labels do.
Refer to caption: extracted/5860709/figures/interpretability/interpretability.png

### 4 Discussion

##### Summary.

Typical neural networks are trained end-to-end: perfect for static worlds, yet cumbersome to update whenever knowledge changes. This is limiting their potential in real-world settings since the world is constantly evolving. Incorporating a visual memory, in contrast, enables a range of flexible capabilities that embrace change: lifelong learning through incorporating novel knowledge, being able to forget, remove and unlearn obsolete knowledge, flexible data selection through memory pruning, and an interpretable decision-making paradigm on which one can intervene to control its behavior. We systematically explored a simple visual memory that decomposes the task of image classification into two primitives, image *similarity* (from a pre-trained embedding representation) and *search* (via fast, scalable nearest neighbor search from a vector database). Our results demonstrate that technical improvements like RankVoting improve kNN accuracies for both DinoV2 and CLIP over the widely used SoftmaxVoting method, the previous state-of-the-art, that is sensitive to two hyperparameters (temperature $\tau$ and number of neighbors $k$). Our approach also narrows the accuracy gap between a nearest neighbor memory (best flexibility, perfect unlearning, improved interpretability) and a fixed linear probe (highest accuracy on static image classification). More importantly, we show that visual memory enables *flexible* perceptual capabilities.

##### Limitations and future work.

First, we only considered the task of image classification across a broad range of datasets. It will be interesting to extend the approach to other visual tasks, such as object detection, image segmentation, instance recognition and to image generation where a visual memory would also be desirable. Secondly, our approach relies on a fixed, pre-trained embedding model; strong distribution shifts may require updating the embedding. Self-supervised models are a particularly flexible choice, but it is an open question whether one could train smaller models that excel at their task with the help of a larger memory database. Conceptually, if a model needs to save less information in its weights, it might be possible to reduce the computational footprint of such a model. Furthermore, we sometimes observe a trade-off between flexibility and accuracy.

##### Outlook.

Deep learning is increasingly becoming a victim of its own success: the more widely it is deployed, the stronger its limitations are felt. While the static nature of end-to-end trained networks can easily be forgotten when focusing on fixed, static academic benchmarks, the real world is anything but static. Data is constantly evolving, leading to the dreaded “model drift” where once-optimal models gradually become less effective []. Incorporating an explicit visual memory—however it may be instantiated—appears to be a promising way forward, a natural solution for real-world tasks where flexibility is key. While the specific approach we employ here might well be improved through more complex systems, we hope that the flexible capabilities we demonstrated might inspire and contribute to a conversation on how knowledge ought to be represented in deep vision models.

#### Acknowledgements

We would like to thank Kevin Swersky, David Fleet, Been Kim, Hagen Soltau, Majid Hadian, Kaifeng Chen, Mojtaba Seyedhosseini and Sung Jin Hwang for feedback and discussions, Mike Mozer for helpful comments on our article draft, as well as Rebecca Bland and Phoebe Kirk for legal advice.

*

## Appendix

We here provide the following supplemental information:

- Appendix A
Aggregation method comparison on ImageNet-1K
- Appendix B
Aggregation method comparison on iNaturalist
- Appendix C
Hyperparameter sensitivity analysis
- Appendix D
Hit rate analysis as an upper bound on aggregation accuracy
- Appendix E
Scaling law details
- Appendix F
OOD analysis for NINCO dataset
- Appendix G
Memory pruning details
- Appendix H
Algorithm for hierarchical label prediction
- Appendix I
Calibration analysis
- Appendix J
Additional errors on ImageNet-A visualized
- Appendix K
Compositionality analysis

### Appendix A Aggregation method comparison (ImageNet-1K)

**Table 4: Benchmarking different aggregation variants at different $k$ thresholds, DinoV2 ViT-L14.**
| Aggregation | @10 | @20 | @30 | @40 | @50 | @60 | @70 | @80 | @90 | @100 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PluralityVoting | 83.2 | 82.9 | 82.6 | 82.4 | 82.1 | 82.0 | 81.8 | 81.6 | 81.5 | 81.4 |
| DistanceVoting | 83.3 | 83.0 | 82.7 | 82.4 | 82.2 | 82.1 | 81.9 | 81.7 | 81.6 | 81.5 |
| SoftmaxVoting | 83.5 | 83.5 | 83.4 | 83.3 | 83.2 | 83.1 | 83.1 | 83.0 | 82.9 | 82.9 |
| RankVoting (ours) | 83.5 | 83.6 | 83.6 | 83.5 | 83.5 | 83.4 | 83.3 | 83.3 | 83.3 | 83.3 |

**Table 5: Benchmarking different aggregation variants at different $k$ thresholds, DinoV2 ViT-B14.**
| Aggregation | @10 | @20 | @30 | @40 | @50 | @60 | @70 | @80 | @90 | @100 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PluralityVoting | 81.8 | 81.4 | 81.1 | 80.9 | 80.7 | 80.4 | 80.2 | 80.0 | 79.8 | 79.6 |
| DistanceVoting | 81.9 | 81.5 | 81.2 | 81.0 | 80.8 | 80.5 | 80.3 | 80.0 | 79.9 | 79.7 |
| SoftmaxVoting | 82.0 | 82.0 | 81.9 | 81.8 | 81.7 | 81.7 | 81.6 | 81.5 | 81.3 | 81.3 |
| RankVoting | 82.1 | 82.2 | 82.1 | 82.0 | 82.0 | 82.0 | 81.9 | 81.9 | 81.9 | 81.9 |

**Table 6: Benchmarking different aggregation variants at different $k$ thresholds, DinoV2 ViT-S14.**
| Aggregation | @10 | @20 | @30 | @40 | @50 | @60 | @70 | @80 | @90 | @100 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PluralityVoting | 78.6 | 78.2 | 77.8 | 77.4 | 77.1 | 76.8 | 76.5 | 76.3 | 76.1 | 75.9 |
| DistanceVoting | 78.8 | 78.4 | 77.9 | 77.5 | 77.2 | 76.9 | 76.6 | 76.4 | 76.2 | 76.0 |
| SoftmaxVoting | 78.9 | 78.9 | 78.7 | 78.6 | 78.5 | 78.3 | 78.1 | 78.0 | 77.9 | 77.7 |
| RankVoting | 78.9 | 79.1 | 79.0 | 78.9 | 78.9 | 78.9 | 78.9 | 78.8 | 78.8 | 78.8 |

**Table 7: Benchmarking different aggregation variants at different $k$ thresholds, CLIP ViT-L14.**
| Aggregation | @10 | @20 | @30 | @40 | @50 | @60 | @70 | @80 | @90 | @100 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PluralityVoting | 79.0 | 78.7 | 78.3 | 78.0 | 77.8 | 77.6 | 77.4 | 77.4 | 77.2 | 77.0 |
| DistanceVoting | 79.2 | 78.9 | 78.5 | 78.2 | 78.0 | 77.8 | 77.6 | 77.5 | 77.3 | 77.1 |
| SoftmaxVoting | 79.3 | 79.3 | 79.1 | 78.9 | 78.8 | 78.7 | 78.5 | 78.5 | 78.4 | 78.2 |
| RankVoting (ours) | 79.3 | 79.6 | 79.7 | 79.7 | 79.7 | 79.7 | 79.7 | 79.7 | 79.7 | 79.7 |

**Table 8: Benchmarking different aggregation variants at different $k$ thresholds, CLIP ViT-B16.**
| Aggregation | @10 | @20 | @30 | @40 | @50 | @60 | @70 | @80 | @90 | @100 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PluralityVoting | 72.8 | 72.6 | 72.3 | 72.0 | 71.7 | 71.4 | 71.2 | 70.9 | 70.8 | 70.5 |
| DistanceVoting | 73.1 | 72.9 | 72.6 | 72.3 | 71.9 | 71.6 | 71.4 | 71.1 | 70.9 | 70.6 |
| SoftmaxVoting | 73.3 | 73.3 | 73.1 | 72.9 | 72.7 | 72.5 | 72.3 | 72.1 | 71.9 | 71.7 |
| RankVoting | 73.0 | 73.7 | 73.8 | 73.8 | 73.8 | 73.8 | 73.7 | 73.7 | 73.7 | 73.7 |

Figure: (a) DinoV2 ViT-B14
Refer to caption: x8.png

**Table 9: Benchmarking different aggregation variants on ImageNet-1K.**
| Model | Aggegation | IN-val acc (%) |
| --- | --- | --- |
| CLIP ViT-L14 | CLIP paper (zero-shot) | 75.3 |
| CLIP ViT-L14 | no aggregation | 76.0 |
| CLIP ViT-L14 | PluralityVoting | 79.2 |
| CLIP ViT-L14 | DistanceVoting | 79.4 |
| CLIP ViT-L14 | SoftmaxVoting | 79.6 |
| CLIP ViT-L14 | RankVoting (ours) | 79.9 |
| DinoV2 ViT-L14 | DinoV2 paper (kNN Softmax) | 83.5 |
| DinoV2 ViT-L14 | no aggregation | 81.1 |
| DinoV2 ViT-L14 | PluralityVoting | 83.2 |
| DinoV2 ViT-L14 | DistanceVoting | 83.3 |
| DinoV2 ViT-L14 | SoftmaxVoting | 83.6 |
| DinoV2 ViT-L14 | RankVoting (ours) | 83.6 |

### Appendix B Aggregation method comparison (iNaturalist)

Figure: (a) Comparing aggregation methods, DinoV2 ViT-L14
Refer to caption: x12.png

### Appendix C Hyperparameter sensitivity analysis

Figure: (a) RankVoting ($\alpha$)
Refer to caption: x14.png

### Appendix D Hit rate analysis

Figure: Figure 10: Hit rate. This plot shows the probability of the true label being contained in list of labels of the first $k$ retrieved neighbors on ImageNet-1K, for five different models and $k\in[1,100]$. With 100 neighbors, the hit rate approaches 98% for the best model. Conceptually, this is a very high upper bound on the performance that can be achieved by a given featurizer via nearest neighbor retrieval.
Refer to caption: x17.png

### Appendix E Scaling law

As we mentioned in Section 3.3, we found that a logarithmic form fits the data well between $\log_{10}$(memory size) and $\log_{10}$(error rate). Specifically, we found the following functional forms for DinoV2 ViT S14 and DinoV2 ViT L14 respectively via np.polyfit(x, y, dim=1):

$$ DinoV2 ViT L14: $\displaystyle y=-0.9434\cdot\log_{10}(x)+2.0704$ DinoV2 ViT S14: $\displaystyle y=-1.0942\cdot\log_{10}(x)+2.3187$ $$

where $x=\log_{10}$(memory-size) and $y=\log_{10}$(error-rate), where memory-size $\in[10^{3},10^{9}]$ and error-rate in $[0,100]$.

### Appendix F NINCO dataset

Figure: (a) Mean
Refer to caption: x18.png

### Appendix G Memory pruning

For memory pruning from Section 3.5, we implemented two pruning methods: removing unreliable neighbors from memory entirely (“hard memory pruning”), and reducing their weight (“soft memory pruning”). We report results on the ImageNet validation set with a (potentially pruned) ImageNet-train set in memory. For hard pruning, we excluded images from memory that contributed to a wrong decision at least 128 times (this meant excluding 26,257 images for DinoV2 ViT-L14), based on querying the ImageNet-train set against a memory consisting of the ImageNet-train set and querying 100 neighbors for each sample. In order to obtain a fair comparison, instead of reporting accuracies for an arbitrary choice of $k$ (the number of neighbors) we instead evaluate accuracy for each $k$ in $[1,100]$ and report the maximum accuracy obtained in Table 3. This ensures that differences in observed accuracy can indeed be attributed to memory pruning, as opposed to a choice of $k$. For soft pruning, instead of excluding unreliable neighbors entirely as in hard pruning, the neighbor weights (1.0 for PluralityVoting, or a rank-based weight in case of RankVoting) are instead multiplied by a reliability factor $\gamma$ with $\gamma=\frac{d}{c+v}$ where $v$ is the number of times the image contributed to a wrong decision on the ImageNet-train set, $c=1$ to avoid division by zero, and $d=1.75$. This results, for instance, in $\gamma=0.88$ for images that only contribute to a single wrong decision; in $\gamma=0.16$ for images that contribute to ten wrong decisions, and in $\gamma=0.02$ for images that contribute to 100 wrong decisions on the training set. Images that never contributed to any wrong decision are assigned $\gamma=1.0$, i.e. their default weight remains unchanged.

### Appendix H Hierarchical label prediction algorithm for flexibly increasing dataset granularity

Figure: Algorithm 1 Hierarchical Label Prediction

### Appendix I Calibration analysis

Figure: (a) DinoV2 ViT-L14, linear classification
Refer to caption: x20.png

### Appendix J Visualizing errors on ImageNet-A

Figure: Figure 13: Interpretable decision-making. A retrieval-based visual memory enables a clear visual understanding of why a model makes a certain prediction. Here, we show four randomly selected misclassified query images from ImageNet-A Hendrycks et al. [] along with five nearest neighbors from DinoV2 ViT-L14 using the ImageNet-1K training set as visual memory. All labels are from the respective datasets (ImageNet-A for query and ImageNet-train for neighbors). While all neighbors visually look reasonable, not all labels do.
Refer to caption: extracted/5860709/figures/interpretability/interpretability_appendix.png

### Appendix K Compositionality analysis

Figure: Figure 14: Compositionality of representations. The first column indicates a query image; the next three columns are the three nearest neighbors from the training set. The last three columns are the *residual* images, obtained by subtracting the features of the nearest neighbor (2nd column from the left) from the features of the query image (1st column from the left). The nearest neighbors for DinoV2 are mostly images with a single concept (or object) from the query image. The residual image, subsequently, leads to nearest neighbors dominated by another single object in the query image. In contrast, CLIP often finds neighbors that are a blend of concepts from the query image.
Refer to caption: extracted/5860709/figures/composition/compositionality.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: In Table 2, for the ImageNet-A dataset, the performance achieved through linear probing surpasses even that of the JFT memory combined with Gemini reranking. This is not the case for the other datasets. Initially, I considered that this anomaly might indicate some form of overfitting. However, later in the paper you mention potential issues with the ground-truth labels in the ImageNet-A dataset, which could affect the trend.

Did you observe similar issues with any of the other datasets used in your experiments? Moreover, do you think it is possible to quantify this problem—for instance, by assessing how reassigning or correcting problematic labels influences the performance?
soundness: 4
strengths: * __Presentation.__ The paper is written in a highly engaging and accessible manner, capturing the reader's interest from the outset. It is well-positioned within the context of related works, providing a clear understanding of how it contributes to the field. The novelty of the research is evident, and the authors effectively communicate the motivation behind each experiment.

* __Experimental design.__ The authors have conducted a comprehensive set of experiments that excellently demonstrate the advantages of memory-based image classification over traditional model-based approaches. They show how their method easily scales to accommodate new data, illustrating its effectiveness in scenarios requiring continual learning. Furthermore, they highlight the flexibility of their approach in machine unlearning and memory pruning, showcasing how it can efficiently remove or adjust the influence of specific data points. The interpretability of their method is also emphasized, providing clear insights into the decision-making process of the model.

* __Novelty.__ The main highlight of the paper is the introduction of the novel neighbor aggregation method called RankVoting. This method is clearly and thoroughly described, with an intuitive rationale that makes its underlying principles easy to grasp. The authors present solid empirical evidence of its advantages over state-of-the-art distance-based softmax voting methods, demonstrating superior performance, especially as the number of considered neighbors increases. This contribution significantly advances the field by providing a more effective way to aggregate information from nearest neighbors in neighbor-based classification.
summary: The authors address the memory-based image classification task by utilizing large, unsupervised pretrained visual models to extract image descriptors. They perform classification based on nearest neighbor retrieval in the embedding space of these models. A key contribution is their novel aggregation approach called _RankVoting_, which outperforms state-of-the-art methods. Unlike previous methods, _RankVoting_ improves performance as the number of considered neighbors increases. Through various experiments, the authors demonstrate the advantages of visual memory-based classification over traditional model-based classification, highlighting benefits such as flexibility, scalability, and interpretability.
weaknesses: While some may argue that the methodological novelty of the paper is modest—introducing only a single, simple formula to slightly improve kNN classification—I believe that this advancement is substantial in its own right. The elegance and simplicity of the _RankVoting_ method support its effectiveness, providing a meaningful enhancement to nearest neighbor classification. Moreover, the authors contribute significantly to the broader discussion of neighbor-based versus model-based image classification. They present strong, well-articulated arguments that underscore the advantages of visual memory-based approaches. The paper not only provides a practical improvement but also stimulates important discourse that could shape future research directions.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the framework extend to tasks like segmentation or object detection, and if so, how would visual memory be adapted?
2. How does RankVoting handle adversarial or noisy samples in memory retrieval? Would further methods mitigate this?
3. Would memory retrieval latency impact real-time performance, especially in billion-scale databases?
4. What would be the computational cost for re-ranking with Gemini, particularly on constrained devices?
5. Could the proposed system be combined with incremental learning techniques to improve distribution shift resilience?
6. How might RankVoting's performance vary with different embedding models?
7. What measures could prevent sensitive or incorrect data from being memorized, impacting model reliability?
8. Are there optimizations to manage storage costs of billion-scale datasets without affecting retrieval accuracy?
9. Could memory pruning methods be customized for specific tasks, improving model performance further?
10. How does the model handle cases where similar but incorrect neighbors dominate the nearest neighbors?
soundness: 3
strengths: 1. Combines the strengths of deep neural networks with database principles, enabling flexible memory operations such as adding, removing, and unlearning data at scale.
2. Achieves high accuracy and flexibility, outperforming baseline voting methods, especially with RankVoting and Gemini-based reranking.
3. Supports continual learning, efficient memory management, and interpretability, addressing practical needs like model drift and data privacy through machine unlearning.
summary: The paper introduces a "visual memory" framework for image classification, aiming to make perception more adaptable by decoupling representation from memory. By combining image similarity from pretrained embeddings with fast nearest-neighbor retrieval, this framework allows flexible handling of knowledge updates, enabling data addition, memory pruning, and unlearning. The proposed RankVoting aggregation method surpasses previous methods and achieves high accuracy across ImageNet and billion-scale datasets. The work illustrates the potential of visual memory in achieving continual learning without re-training while supporting interpretability and control in deep learning models.
weaknesses: 1. Limited application focus on other visual tasks (e.g., object detection, segmentation) beyond classification. How might the visual memory approach be adapted for tasks like object detection or segmentation? What unique challenges would arise in those domains?
2. Reliance on a fixed pretrained embedding model may limit adaptability under significant distribution shifts. Have you considered approaches to update the embedding model incrementally as new data becomes available? This could help address potential limitations in adaptability under distribution shifts.
3. Sensitivity of SoftmaxVoting to hyperparameters could impact performance stability.
4. Limited ablation studies on the impact of visual memory size across various neural network architectures.
5. No detailed analysis on memory retrieval latency and computational overhead for large-scale memory databases. Could you provide more information on the memory retrieval latency and computational overhead, especially for the billion-scale experiments? This would help readers understand the practical implications of scaling up the visual memory approach.
6. Lack of performance benchmarking across diverse dataset distributions.
7. Absence of discussions on handling adversarial attacks or noise within the visual memory framework. How robust is the visual memory approach to adversarial examples or noisy data? It would be valuable to see an analysis of how the system performs when the memory contains corrupted or adversarially crafted samples.
8. Limited interpretability assessment for users to control specific decision boundaries. Could you provide more details on how users might interact with or control the decision-making process in your visual memory framework? Specifically, how might one adjust decision boundaries or influence the model's predictions in practice?
9. Potential limitations in scalability for different embedding models with high memory footprints.
10. Insufficient exploration of non-image applications (e.g., text-based memory) within the flexible memory paradigm.

---

confidence: 5
contribution: 1
initial_rating: 6
presentation: 3
questions: As per my comment above, have the authors considered the effect of class-imbalance in the training data and what this means with respect to $k$ and potential accuracy? When you get improvements in performance where do these come from?

What is the set-up for the linear probes? How was this validated?

Can you comment on the computational efficiency aspects? how does it scale with instances? How long does a query take? How much storage is required?
soundness: 2
strengths: Overall, the paper is well written and easy to read if somewhat whimsical in style. The paper does highlight well some of the problems with current approaches to image classification, particularly in the context of deployment of models where the training data might change (for various reasons) regularly. The proposed approach of using weighted KNN seems to work reasonably, and the use of a scalable NN search engine helps with scaling to large numbers of instances in the memory. 

A wide range of different aggregation methods are tested, and there are comparisons against standard linear probing baselines. I like the experiments that look at OOD performance and this is particularly interesting.
summary: This paper makes the argument that performing classification through nearest-neighbour search (in an embedding space) has many more desirable properties than more common approaches to classification found today - namely: that its much easier to add new data (and new classes) as this doesn't require retraining, that things (both images and classes) can easily be unlearned/removed, and the decisions might be more interpretable. The authors demonstrate that embeddings from recent pre-trained (vision-language) models are effective for use in a knn context and with appropriate weighting functions can outperform linear probing.
weaknesses: I think the biggest problem I have is that the contribution of the paper is not really clear - almost everything presented is rather obvious and based on well studied approaches; use of NN methods for classifying images based on features dates at least back to the days of Turk & Pentland (https://sites.cs.ucsb.edu/~mturk/Papers/mturk-CVPR91.pdf) in 1991 (if not before), and research on fast nearest neighbour approaches for solving KNN problems based on image features also has a long history (e.g. Sivic's VideoGoogle from the early 2000s was using inverted indexes for object matching/detection - https://www.robots.ox.ac.uk/~vgg/publications/2003/Sivic03/sivic03.pdf). Similarly, there is much research on KNN aggregation methods, including many that utilise rank (e.g. http://www.jcomputers.us/vol6/jcp0605-01.pdf & there are even articles suggesting reciprocal rank: https://visualstudiomagazine.com/articles/2019/04/01/weighted-k-nn-classification.aspx), so the contribution of the proposed RankVoting scheme is not clear. That new "memories" can be added/removed seems "obvious".

Experimentally, I feel there are quite a few problems. Firstly, it's not clear how the extreme class-imbalance in datasets like imagenet are taken into account; its non-obvious to me that it even makes sense to use a fixed value of $k$ in such cases. Secondly, Many of the presented results also do not show a clear win for any particular method (e.g. table 1 - difficult to claim RankVoting is better than softmax; table 2 - quite a mixed bag).

In the cases where a linear probe has been trained its not clear what procedure was adopted and how optimal the learned decision boundary is; was this a linear SVM or just a linear layer trained with gradient descent? does that not make a difference? 

The pruning experiments are interesting, but are they not rather highlighting problems with the data (labels)? Presumably, more generally, this could be an approach for improving data quality rather than specifically improving the approach presented (although of course if does reduce computational complexity).

The paper doesn't seem to give any thought to the complexity of the proposed approaches - how big is the database of embeddings? how long does retrieval take? How does this compare to state of the art classification models (e.g. inference time, and perhaps comparing number of model weights to feature-extractor model weights + database size)?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: The result of using Gemini re-ranking is quite strong and interesting. I wonder if this is due to VLM's in-context learning capacity. A discussion on this can be helpful.
soundness: 3
strengths: 1. The paper is well-written and easy to follow.
2. Investigation into different voting strategies looks interesting.
3. The idea of using visual memory database + retrieval is a promising strategy for visual recognition, given its flexibility, OOD robustness, and adaptation capacity to ever-growing concepts.
summary: This paper introduces a flexible and interpretable visual memory system that integrates deep neural networks with a database-like structure to improve image classification. By separating knowledge representation from the model’s weights, the system enables easy addition, removal, and modification of data without retraining, addressing the limitations of static knowledge in traditional models. Key features include scalable data management, machine unlearning, and transparent decision-making processes. The system uses a k-nearest neighbour retrieval mechanism and RankVoting to enhance sample aggregation for improved performance, highlighting the potential of using visual memory in deep learning models to adapt to dynamic data and evolving knowledge.
weaknesses: 1. One limitation of retrieval and search engines is that the cost of storage and inference increases and the accuracy of recognition decreases as the memory database size grows. Although computing budget and accuracy can be traded off, this remains a limitation. Additionally, regarding the computation cost, I wonder if there is a rough estimate of the inference time when the database size increases to 1 billion images.
2. The adaptation capacity of the proposed visual memory to accommodate ever-growing concepts assumes that the image encoder can produce meaningful embeddings for new concepts. While for geometrically distinctive concepts, I believe this is less of a concern for DINO representations, as they are observed to contain rich geometric information, for concepts where class label correlates more with semantics rather than geometry, I wonder if the adaptation capacity still holds.

Meta-review: This work introduces a flexible and interpretable visual memory system that combines deep neural network embeddings with database-like capabilities, enabling scalable data addition, unlearning, and intervention for controlled decision-making.

The paper is generally well-written, presenting an intriguing idea of utilizing external memory to enable flexible and adaptive modifications to AI models. This approach holds significant potential for various applications, including data addition, unlearning, and intervening in decision-making.

However, the paper received mixed review scores: 8, 6, 6, and 3.

After reviewing all the feedback, the AC identified key unresolved issues raised by Reviewer AsSs, which ultimately led to the paper's rejection. Despite the rebuttal, Reviewer AsSs maintained two primary concerns:

[unclear novelty] Many works have presented similar claims or methods (for example, the "simple scales well" claim => https://arxiv.org/pdf/1702.08734 did billion-scale k-nearest-neighbour search with image features in 2017). It is unclear what is the technical novelty or contribution of this paper.

[professional writing] The claims in the paper are not written in a scientific manner (claims like the algorithm being "Lightning fast." (line 356-357) are not statements with any scientific meaning).

Decision: Reject