ICML 2025

Towards ﬂexible perception with visual memory
Robert Geirhos * 1 Priyank Jaini * 1 Austin Stone * 1 Sourabh Medapati * 1
Xi Yi 1 George Toderici 1 Abhijit Ogale 1 Jonathon Shlens 1
Abstract
Training a neural network is a monolithic en-
deavor, akin to carving knowledge into stone:
once the process is completed, editing the knowl-
edge in a network is hard, since all information is
distributed across the network’s weights. We here
explore a simple, compelling alternative by mar-
rying the representational power of deep neural
networks with the ﬂexibility of a database. De-
composing the task of image classiﬁcation into
image similarity (from a pre-trained embedding)
and search (via fast nearest neighbor retrieval
from a knowledge database), we build on well-
established components to construct a simple and
ﬂexible visual memory that has the following key
capabilities: (1.) The ability to ﬂexibly add data
across scales: from individual samples all the way
to entire classes and billion-scale data; (2.) The
ability to remove data through unlearning and
memory pruning; (3.) An interpretable decision-
mechanism on which we can intervene to control
its behavior. Taken together, these capabilities
comprehensively demonstrate the beneﬁts of an
explicit visual memory. We hope that it might
contribute to a conversation on how knowledge
should be represented in deep vision models—
beyond carving it in “stone” weights.
1. Introduction
In the pretty diagrams on ”Intro to Machine Learning” slides,
an ideal ML workﬂow looks like this: Data collection, pre-
processing, choosing a model, training, evaluation, deploy-
ment. Happy ending—the model is deployed, the users love
it, and one can ﬁnally go on that well-deserved vacation and
catch up on the latest AGI memes.
Until, of course, the enemy of any ideal world sets in: reality.
1Google DeepMind. Correspondence to: Robert Geirhos <last-
name@google.com>, Priyank Jaini <pjaini@google.com>.
Proceedings of the 42 nd International Confe...