No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations
Authors: Walter Simoncini
Andrei Bursuc
Spyros Gidaris
Yuki M Asano
Venue: NeurIPS 2024 poster (2024)
Keywords: self-supervised
gradients
computer vision
transformers
k-nearest neighbor
classification
in-context learning
clustering
retrieval
Primary area: machine_vision
TL;DR: We propose using self-supervised gradients to enhance pretrained embedding features and achieve significant improvements in k-nearest neighbor classification, in-context scene understanding, linear probing and clustering.

Abstract: This paper introduces FUNGI, **F**eatures from **UN**supervised **G**rad**I**ents, a method to enhance the features of transformer encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These gradients are projected to a lower dimension and then concatenated with the model's output embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification, clustering and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training. Code is available at https://github.com/WalterSimoncini/fungivision.

No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations
Walter Simoncini !M
Spyros Gidaris?
Andrei Bursuc²
Yuki M. Asano
QUVA Lab, University of Amsterdam; avaleo.ai, Paris, France
Abstract
This paper introduces FUNGI, Features from UNsupervised GradIents, method enhance the features of transformer encoders by leveraging self- supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These gradients are projected to lower dimension and then concatenated with the model's s output embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification, clustering and image retrieval, and that they significantly improve the retrieval-basee in-context scene understanding abilities of pretrained models, for example improving upon DINO by + +17% for semantic segmentation without any training. Code is available https:/ //g thub. haoom//aateetii www.iiiiinmmmmiii////mmmmmmi /inggiiviiiii.nn.
Introduction
The k-nearest neighbor algorithm (kNN) (Fix, 1985) is a fundamental non-parametric machine learning tool, and can be scaled to datasets with billion of examples thanks advances in quantization (Jegou et al., 2010; Guo et al., 2020) and efficient GPU implementations (Johnson et al., 2019). This simple and versatile algorithm has shown potential in multiple applications well before deep neural networks became relevant (Efros & Leung, 1999; Hays & Efros, 2008; Torralba et al., 2008). Its recent applications include fast and robust image classification with Vision Transformers (Caron et al., 2021; Chen & He, 2021). unlabeled data selection (Yalniz et al., 2019), relevant text-retrieval (Lewis al., 2020), and visual in-context learning (Balazevic et al., 2024), where context of data samples with their annotations (e.g.. a semantic segmentation map) are used to make dense predictions.
Devising powerful and expressive features for recognition and image understanding has long history in computer vision. Feature engineering strategies range from simple local features (Lowe, 2004; Dalal & Triggs, 2005; Van De Sande et al., 2009) extracting gradient, boundary color information, to various mid-level (Boureau et al., 2010) or global pooling (Oliva & Torralba, 2001; Sivic & Zisserman, 2003; Jégou et al., 2010) techniques. It also possible couple off- aff-the-shelf pretrained backbones as feature extractors with such pooling strategies (Gong et al., 2014; Kulkarni et al., 2015; Gidaris et al., 2020) to improve performances. While these approaches demonstrate the utility of using a neural network's 's learned embedding space, they still require specific expertise and tuning for each backbone and task with only limited guidance from the data itself.
Work partially done as an intern at valco.ai. Datasets were solely downloaded and evaluated the University of Amsterdam. walterdashita .nl.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SimCLR
*
V DINO
Project Concat
Gradlent Enhanced Features Database
for (x)
V
FUNGI
Figure 1: Gradient-augmenter features: given pretrained backbone fo- and its embeddings, apply a family of SSL. losses, extract their gradients, and project and concatenate them. These new features used to build k-nearest neighbor index, which can be used for classification retrieval.
We depart from this line of work and aim to attain strong representations without training and feature engineering, yet still exploiting information cues from data. In particular, we suggest to enhance the neural network's embeddings by incorporating FUNGI (Features from Unsupervised Gradlents). FUNGI are obtained from self-supervised loss functions, as these do not require any human annotations and allow for simple enhancement to embedding-only kNN. The losses computed on top of pretrained backbones (with randomly initialized linear layers if needed), which permits method to be "plug-and-play" and-play" and benefit from the diverse set of pretraining objectives put forth by the community.
We explore gradients from various learning objectives, such as contrastive learning (Chen et al., 2020a) and self-distillation (Caron et al., 2021) thereby integrating complementary information that mitigates the weaknesses of individual losses. The gradients layers, making them computationally cheap. Finally, these are projected to smaller dimensions, and concatenated with the neural network embeddings, yield new inputs to the classic kNN algorithm.
obtained from late hidden
Using kNN with FUNGI can be regarded as non- non-paraettric transfer learning approach: the gradient information encodes first-order learning signals that are specific to the downstream data, yet no param- eters need be updated. Despite this simplicity, we achieve consistent performance improvements across multiple models and benchmarks. Overall, our main contributions are summarized as follows:
We introduce FUNGI, a novel method that combines neural network features and gradients to enhance representations.
We demonstrate that the gradients from self-supervised losses have predictive abilities and offer complementary information to model embeddings.
We validate the generality and utility of our method by achieving consistent gains across 11 image, 5 text, and 2 audio classification benchmarks, plus in-context image segmentation and image retrieval tasks, utilizing a total of 20 backbones.
2 Related Work
Fast Adaptation There is a broad range of approaches to quickly adapt models to newly specified tasks and data. Inspired by early learning-to-lean work (Hochreiter al., 2001), meta-learning methods (Finn et al., 2017; Nichol et al., 2018) learn to initialize the parameters of a learner such that becomes faster to fine-tune with small number of gradient steps and data. Alternative approaches leverage external memory modules to store relevant training samples to learn to match query examples (Santoro et al., 2016; Vinyals et al., 2016), learn to produce and compare class-based prototypes (Snell et al., 2017) or learn to generate the weights of a classifier (Gidaris & Komodakis, 2018) or even of an entire neural network (Bertinetto et al., 2016) from only a few labeled examples. The advent of Vision Transformers (Dosovitskiy et al., 2021) enable new parameter- and data- efficient strategies adapt pretrained models through visual prompts (Jia al., 2022) and in-context learning (Zhang et al., 2023). In contrast to this line of works, our method does not require specialized training and can be applied to any frozen pretrained backbone. FUNGI can also be related test-time training (Sun et al., 2020; Hardt & Sun, 2024), where the parameters of predictive model are

---

CSA Flowers)
CKA (ESAT)
C6A (Cara)
3000
523
Clr-Class Accuracc Deltas (Flowers 102)
60.1
106.0
65.2
99-1
199
512
BNO
12.3
03
120
53
04.3
05.2
0280
415
05
85
220
580
SKCCU
48.
53.5
$60,0
10000
51.5
51.2
50.3
100.0
0.5
0.0
-05
Dmh.
العلممم sny س ملاييسسس
0.5
0.0
-0.5
03
202
10.1
245
25.3
Sm/
68.0
25.2
10
670
K.4
11.
66.5
11.8
16.7
18.4
110
simmu
A اo Iy الال
0.5
0.0
-0.5
Figure 2: Combining diverse features (em- beddings, gradients) leads to large improve- ments. Pairwise CKA similarity of features (top) and the KNN accuracy of their combina- tion (bottom).
Figure 3: Gradients encode different informa- tion. Delta per-class kNN accuracy of gradi- ents from different objectives compared to the embeddings, indicated as "Emb." in the plot.
updated over test samples with a self-supervised objective to reduce the gup between the training and test distributions. While we also use self-supervised objectives and gradients, our approach does not update model parameters and is not limited predictive models, as it can be applied to any task that can be solved with retrieval.
Self-Supervised Learning Objectives In recent years, self-supervised learning (SSL) has made tremendous progress in computer vision. SSL aims to learn good representations from unlabeled data by leveraging supervision from different signals in the data itself via pretext objectives, thus foregoing human supervision. Models pretrained with self-supervision are subsequently finetuned to downstream tasks of interest with few labeled samples. The crux of SSL is in the pretext learning objective. A wide and diverse collection of pretext objectives have been proposed in the community relying on contrastive learning (Chen al., 2020a; He et al., 2020; Chen et al., 2020b), clustering (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020), self-distillation (Caron et al., 2021; Grill al., 2020; Chen & He, 2021; Gidaris et al., 2021), feature (Zhou et al., 2022; Assran et al., 2023) or input reconstruction (He et al., 2022).
We hypothesize that the gradients induced by these objectives encapsulate different information from the input data, and that this information can be combined to produce more information-rich representations. Here, we do rather focus on pretext objectives and data augmentation strategies to compute representations from a frozen pretrained model.
self-supervision in the usual way, i.e, to pretrain an encoder, but
Feature Engineering A long-standing research area for pattern recognition and image understand- ing before the advent of deep neural networks that brought the paradigm of end-to-end representation learning. In contrast, classic feature extraction methods are devised without labeled data and of- ten from only few data samples. They range from local features, such SIFT (Lowe, 2004), HOG (Dalal & Triggs, 2005), to global pooling, such as GIST (Oliva & Torralba, 2001), Bag-of- Visual- (Sivic & Zisserman, 2003), Fisher vectors (Perronnin al., 2010), VLAD (Jégou et al., 2010), selective match kernels (Tolias et al., 2013), etc. These pooling strategies can be easily plugged to intermediate output neural network activations (Gong et al., 2014; Kulkarni al., 2015; Gidaris et al., 2020), harnessing data-driven learned representations. Other modern examples of feature engineering include Head2Toe (Evci et al., 2022), which augments the model embeddings using intermediate activations, -NN-poompiing (Xu et al., 2023), which uses the next token probabilities of language model to perform few shot nearest neighbor classification and LOST (Siméoni et al., 2021) which uses patch features from self-supervise vision transformers for object localization. Closer to our line of work, Wei al. (2022) shows that kernel regression using the empirical neural tangent kernel (eNTK), which corresponds to the model Jacobian, can achieve performance similar to fine-tuning in vision, and Mu et al. (2020) shows that features obtained from the Jacobian of the top layers of convolutional neural network can be used to enhance a linear classifier. In contrast, our method does not require any annotations and is computationally cheaper, as we only compute the gradient for single layer rather than the full Jacobian.
3

---

(2) Compute Loss & Backpropagate
h
Attract
(1) Patchify and Forward
Repel
(8) Per-Sample
Gradients
(4) Project
Fixed Nogative Batch
Figure 4: Gradients extraction using a SimCLR loss. Given a pretrained backbone f and a randomly initialized projection head h, first patchify image, obtain the latent representations of patches (1), calculate the SimCLR loss by maximizing the pairwise cosine similarity patches, and minimizing their similarity a fixed negatives batch and backpropagate (2), extract the per-sample gradients (3) and finally project the gradients to the same dimensionality as the embeddings (4).
3 Gradients as Features
Gradients encode information how the weights of neural network should change to solve a given task. Thus, they contain information about the current state of the network and its relation to the data point(s) used to compute it. Therefore, should perform better than the embeddings, as they are adapted to the dataset at hand. Empirically, the second row in Figure shows this to be accurate, e.g., gradients from SimCLR loss improve the accuracy of k-nearest neighbor classification by 10.1 % on Flowers102.
hypothesize that features from supervipd gradients
If we plot the per-class accuracy distribution as in Figure 3, we notice that gradient features encode different information depending on the loss and that they can perform significantly worse on some classes, possibly because they are estimated using single data point, and are thus dependent on the local loss curvature. These findings suggest that the information in embeddings and gradient features could be complementary. We show that this holds in the second row of Figure 2, as feature pairs perform better. Moreover, the first row of the figure suggests that more diverse feature pairs, as measured via their centered kernel alignment (CKA) (Kornblith et al., 2019) score, lead to better overall performance.
Method
Our method, FUNGI, enhances k-nearest neighbor search by incorporating features from unsupervised gradients. We extract gradients from self-supervised loss functions, project them to smaller dimen- sions, and concatenate them with neural network embeddings. The extraction of self-supervised gradients is illustrated in Figure 4, while Figure shows how FUNGI features constructed.
Definitions Throughout this section, we define L2 normalization as z' 2/||>| 22. vision backbone as f, a linear projection head as l and vectorization as vec(-).
4.1 FUNGI: Features from Unsupervised Gradients
Gradients Extraction. Given an arbitrary vision backbone f, our case vision transformer (ViT) (Dosovitskiy et al., 2021), we attach randomly initialized linear projection head h and obtain a latent representation 2 = h (x)) of the input images, which we use to compute the loss for one of our self-supervised objectives. We then to the weights and biases of an arbitrary hidden linear layer within f. Unless specified otherwise, we use the attention output projection of the last transformer block as our gradient's 's source.
backpropagation and extract the gradients with respect
From Gradients Retrieval-Friendde Features. Gradients are high dimensional and thus im- practical for nearest-neighhor retrieval due to speed and storage considerations and the curse of dimensionality. tackle these issues, we downsample the gradients to the dimensionality of original model embeddings d using the binary random projections method introduced by Achlioptas (2003). For this, we first vectorize the gradients by flattening them to a m-dimensional vector and then multiply them by a matrix R e {-1, 1-1,1)d. whose entries are the realizations of Bernoulii random

---

variable with p = 0.5. The gradient 9ø with respect to loss Lg is then defined as
ga ( R vec (VLa(a))
Combining with Embeddings. To produce our FUNGI features 6, we concatenate one or more gradients the model embeddings. As gradient magnitudes can vary widely across losses, and we want gradients to be equally considered as the embeddings, we Ly-normalize each gradient, as well the embeddings and compute
(xx) = cat (2), (o) (x),,,.)
where cat denotes concatenation. Finally, we reduce the dimensionality of these combined features Via PCA to d-dimensional vector. This allows the combination of multiple losses iso-storage cost. Our final FUNGI features for sample x are thus obtained as:
OCAA (z) PCAa (o(z)).
4.2 Self-Supervised Objectives
We consider losses representing three families of self-supervised objectives: DINO (Caron al., 2021), SimCLR (Chen et al., 2020a) and KL-divergence based loss inspired by the out-of- distribution detection literature (Huang et al., 2021). ) In this section we briefly describe the objectives and our adjustments to them, and in Appendix B.6, we also briefly discuss clustering and masked image modeling- -based losses.
DINO. DINO is distillation and implicit clustering- based learning method. We use the standard DINO loss, which, given an image, enforces global and local crop correspondence between teacher and student models using a cross-entropy loss. In our case, both models share the same parameters, but have independent heads h, and for student and teacher respectively, thus we have 2; h4 (f(()) E {s, t). The DINO objective can be expressed as:
LINO Cross-Entropy (zx,22)
SimCLR. SimCLR is noise-contrastive method. Given a batch of images, SimCLR generates two views for each image and aims to minimize the distance between views belonging to the same image and maximize their distance to all other views. Instead, we generate set of 49 overlapping patches for each image, which we call the positive set. This set is then contrasted against a fixed comparison batch of 49 256 negative examples. Our objective is the expectation of the pair InfoNCE (Oord et al., 2018) loss for each pair of positive views. we define the positive set of latent view representations Z, where 2, % (x:)) for a view wi, the comparison batch size as N and the temperature as r, the LSICLLR objective is then defined as:
exp(sim(2;; 2,)/T)
SimLLR (( [c..,
l,,,,, log 49(N+1)
(5)
Ek=1
[ats]] exp(sim(2,, 2x)/))
KL Divergence. The KL objective is calculated as the KL divergence between the softmaxed logits of the latents and a uniform distribution U4:
LK = KL((sftmax
We hypothesize two reasons as for why this loss produces predictive gradients: first, receives non-augmented image, with a higher signal-to-noise ratio compared to other objectives, and second, we assume that similar images (e.g., the ones that belong to the same class) produce similar activations, then maximizing their entropy by forcing the output distribution to match uniform should produce similar intra-class gradients and help separability. This hypothesis supported by the fact that while the KL gradients are discriminative, they have chance performance in other tasks, such as in-context scene understanding.
4.3 In-Context Scene Understanding
Balazevic et al. (2024) introduced method for retrieval-based in-context scene understanding, where, for semantic segmentation, they first build a memory bank containing training image patches and their labels, and at test time, for each image patch, retrieve its nearest neighbors and use them to predict its
5

---

Few Shot
11 6%
0.55
Enbeddings
Accuracy
0.62
+4.00
+2
6:8
660
0.33
CUP
025
Figure 6: FUNGI works across backbones. Accuracy in k-nearest neighbor classification using embeddings and FUNGI features from various VIT backbones, both for full dataset and few shot setups, averaged over 11 datasets. For the FUNGI features we chose the best performing combination across datasets. "AR" indicates backbones trained with the AugReg strategy (Steiner al., 2022).
Table 1: FUNGI features are better several datasets. Accuracy of embeddings and FUNGI features in KNN classification over 11 datasets, for two AugReg (Dosovitskiy et al., 2021; Steiner et al., 2022)
VIT-B/16 models from timm (Wightman, 2019) pretrained on IN1K and IN21K.
Pretrain Cars CUB DTD ESAT C100 C10 Pets Food INIK FGVC Flowers Mean
Full dataset
Embeddings INIK
21.3
42.0
54.3
89.0
66.3
89.4
87.3 52.3
77.2
17.9
53.8
59.2
FUNGI
INIK
27.2
50.1
58.6
93.4
69.7
90.7
89.5
58.9
78.8
21,4
61.6
63.6
*4.4
Embeddings
IN21K
21.0
74.0
58.4
91.8
58.4
82.9
83.6
70.6
72.1
23.0
95.0
66.4
FUNGI
IN21K
25.1
74.2
65.0
94.7
63.5
85.7
85.7
73.4
74.5
24.3
96.6
69.3 12.9
5-Shot
Embeddings
INIK
9.4
23.7
32.5
38.6
36.9
48.8
57.5
20.1
55.7
8.3
41.2
33.9
FUNGI
INIK
11.4
26.6
33.9
42.2
38.6
50.2
59.4
24.1
58.6
9.2
49,8
36.7 1228
Embeddings IN21K
7.6
50.0
33.7
47.7
23.2
39.7
53.3
32.0
40.3
10.7
86.2
38.6
FUNGI
IN21K
9.2
48.5
36.3
54.5
28.2
41.7
51.0
37.8
45.4
12.2
85.8
41.0
12.4
label using an attention mechanism. Images are first resized to 512 x 512, and then encoded as a set of 322 1024 patch features using ViT with patch size 16.
We enhance the patch features using SimCLR gradients, obtained by contrasting the input patch tokens against their nearest neighbors from a support index built with ScaNN (Guo et al., 2020). We use the reproduction of this evaluation protocol by Pariza et (2024) to run our experiments.
5 Experiments
In this section, we evaluate the performance of FUNGI in k-nearest neighbor image, text and audio classification and retrieval-band -aased in-context scene understanding. Further ex- periments, including image retrieval, clustering and linear probing. are provided in Appendix B.
0.8
Arcurapy
0.6
0.4
Embeddings
5.1 Image Classification
KL SimCLR Gradients
Following Caron et al. (2021), evaluate FUNGI fea- tures using the task kNN classification. To show the generalizability of our method, we evaluate our features across ViT backbones (Dosovitskiy et al., 2021) with vary- ing model sizes and pretraining strategies, including both supervised and self-supervised methods.
Figure 5: Better data-efficiency. kNN
accuracy of embeddings and FUNGI (us- ing only KL and SimCLR gradients) on ImageNet- 100 using DelT-B/16 back- bone when only shots are used.
We conduct our experiments on 11 diverse downstream datasets, described in Appendix D. Unless otherwise specified, we report the average accuracy across these datasets. We evaluate our features using the kNN implementation of scikit-learn (Pedregosa

---

Table 2: Performance improves as more gradients are used. Accuracy in image classification using kNN with embeddings and FUNGI features, averaged across 11 datasets for backbones, for standard and few shot setups. Results for additional backbones are shown Table "K", "D" and "S" stand for KI., DINO and SimCLR, respectively.
DINOv2
DINO
DefT
MoCov3 VIT-B/16
DelT VIT-B/32
AugReg INIK
AugReg INIK
VIT-B/14
VTT-B/16
VIT-B/16
VIT-S/16
VIT-B/16
Embeddings
79.9
69.0
65.3
63.2
61.7
60.8
59.2
Jull
80.6 10.7
10.4
66.3 f1.0
63.4 10.2
63.3 11.6
60.3 10.5
58.9 J0.3
K+D
81.3
70.1 11.1
68.1
64.7 11.5
65.7 14.0
62.6113
61.1
+ s
81.7 11.8
70,9
70.1 14.8
65.8 12.6
67,3 15.6
64,3 13.5
63.6 14,4
Embeddings
47.6
39.3
38.4
34.7
36.2
34,9
33,9
Sew shor
48.1 10.5
39.4 10.1
38.7 10.3
35.8 11.1
36,6 10.4
34,2 J0.7
33.5 J0.4
+
49.1 11.5
39.7 10.4
39.1 10.7
36.6 11.9
37.6 11.4
35.0 10.1
34,5 10.6
S
50.3 12.7
40.5 11.2
41.1 12.7
38,2 13.5
39.0 12.8
36.5 11.6
36.7 12.8
Table 3: FUNGI features improve in-context semantic segmentation. mloU for retrieval-based semantic segmentation on Pascal VOC 2012, comparing a DINO baseline against FUNGI features and the self-supervised HummingBird model. Results from Balazevic et al. (2024) marked with t.
We resize each image to 512 x 512 and extract 32" 1024 patch features.
Memory Bank Size
Backbone
Features
1024
10
1024 10
1024 x 10"
DINO V/T-S/16
Embeddings
37.2
43.1
46.6
DINO VIT-S/16
FUNGI
50.7 113.5
56.3
113.2
58.0 111.4
DINO ViT-B/16
Embeddings
44.9
50.8
55.7
DINO VIT-B/16
FUNGI
62.1 117.2
66.1 115.3
67.0 11.3
HummingBird ViT-B/16
Embeddings
70.5
DINO
FUNGI
GT
DINO
PUNGI
GT
Figure 7: FUNGI produces sharper and more complete segmentation masks. Segmentation masks produced via nearest neighbor retrieval using DINO features (left), FUNG1 (center) and the ground truth (right). Both methods use a memory bank of 1024 x 10a patches.
et al., 201 with majority voting over 20 neighbors, for full dataset and few shot scenarios, the latter using five examples per class, to analyze the efficacy of our approach in low-data scenarios.
Our results, presented Figure show that FUNGI consistently improves the kNN performance of all VIT models, regardless of model size or pretraining strategy, both for the full dataset and in few shot scenarios. We further investigate data-efficient settings in Figure 5, where FUNGI shows significant improvement when 3 to 6 shots are used, highlighting the potential of FUNGI in low-data regimes.
In Table 1, we show that, with some exceptions, FUNGI provides consistent improvements across datasets for two AugReg (Steiner et al., 2022) ViT-B/16 backbones, pretrained IN1K and IN21K, with FUNGI providing better results on the former. We further discuss these results in Section 6.
Lastly, in Table 2 we show that performance improves more gradients from different self-supervised objectives are used.
5.2 In-Context Scene Understanding
In this section, we assess the effectiveness of our approach in the task retrieval-based semantic segmentation Pascal VOC 2012 (Everingham et al., 2010) and ADE20K (Zhou et al., 2019,
7

---

Table 4: Data-efficient semantic segmentation. mloU scores for cata-efficient retrieval- based seman- tic segmentation on Pascal VOC 2012 and ADE20K, using DINO backbones and their FUNGI features and embeddings. We also compare FUNGI to end-to-end fine-tuning and find our method to perform best for VOC. Results from Balazevic et al. (2024) are marked with f-
Dataset Size
Pascal VOC
ADE20K
Backbone
Features
Decoder
1/128 (n = 83)
1/64 (n 165)
1/128 (n 158)
1/64 (n = 316)
VIT-B/16*
E2E FT
36.1
44.3
11.7
14.4
VIT-S/16
Emb.
NN
26.3
31.8
8.8
10.0
VIT-S/16
FUNGI
NN
29.1 12.8
34.0 12.2
10.2
11.4
12.3 f2.3
VIT-B/16
Emb.
NN
32.2
39.0
9.3
11.3
VIT-B/16
FUNGI
NN 38.0 15.8
46.8
17.8
11.7
12.4
13.7 12.4
Table 5: FUNGI features are useful for the text modality. Top-1 accuracy in kNN text classification for the full dataset and few shot setups. "K" and "S" stand for KL and SimCLR, respectively.
TREC
Banking-77 77
SST (Fine Grained)
AG News
Tweet Eval
Full
5-shot
Full
5-shot
Full
S-shol
Full
10-shot
Full
5-shot
BERT Base
Embeddings 83.6
20.0
55.4
14.5
40.0
20.4
88.8
45,8
23.8
13.6
+K
85.6 12.0
27.6
77.6
67.1
111.7
22.2
17.7
40.7
10.7
23,2
91.0
61.4
115.6
24.4
10.6
13.8 10.2
S
86.8 13.2
23.0
"3.0
67.9
112.5
23,8
19.3
41.8
11.8
18.4
(2.0
89.6
10.8
61.9
116.1
24.8
14.5 10.9
15 Small
Embeddings
88.6
25.6
29.7
5.2
30.0
25,9
71.8
37.4
23.4
8.4
+ 88.6
23.6
.2.0
33.3
13.6
5.6 10.4
32.7
12.7
24.1
JL88
74.3
12.5
40.6
13.2
24.2
10.8
9.5 f11
K
88.4 J0.2 23.6
.2.0
29.1
J0.6
6.1 10.9
32.0
12.0
24.2
JL.
74.8
13.0
41.0
13.6
24.4
9,9 11.
2017). We use the trainaug and train splits to build the memory banks for Pascal VOC and ADE20K, respectively, and report the mean intersection over union (mloU) on the validation set.
We apply FUNGI to DINO VIT-S/16 and VIT-B/16 models. Our results, presented in Table 3 and Table 7, demonstrate that FUNGI significantly enhances DINO's performance across all memory bank sizes, with the most substantial improvements observed in smaller memory banks for Pascal VOC. Qualitatively, FUNGI produces sharper and more complete segmentation masks, as shown in Figure 7. Notably, the DINO ViT-B/16 model, when enhanced with our FUNGI approach, achieves competitive results against the current state-of-the-art Humming model (Balazevic et al., 2024), with difference of only 3.5% on Pascal VOC and 3.1% % on ADE20K, without any training. This is particularly promising result, as HummingBirt employs supf-suuee pretraining strategy that is specialized for retrieval-basee dense prediction tasks, which are the focus of our evaluation in this study.
In addition, we evaluate the efficacy of FUNGT in a data-efficient setup, and report the results in Table 4. Our findings indicate that our method outperforms DINO in this scenario, even when compared to end-to-end fine-tuning of DINO on the downstream task for Pascal VOC.
5.3 Other Modalities
Natural language. We evaluate FUNGI in the text domain using five datasets, described Ap- pendix D, and two transformer language models: BERT base uncased (Devlin et al., 2019) and T5-small (Raffel et al., 2020). We use the LKL and SimLLR losses and obtain the SimCLR views by randomly deleting words with 10% probability. The results are presented in Table 5 and show that FUNGI achieves improvements in the text domain. However, SimCLR gradients struggle with some datasets. Different data augmentation strategies, such back-translation (Sennrich et al., 2016), or language-specific self-supervised losses, e.g., masked language modeling (Devlin et al., 2019), may yield more discriminative gradients. We leave this investigation for future work. Furthermore, in Appendix B.5, we investigate the potential of FUNGI in language in-context learning.
Audio. We demonstrate gains for the audio modality in Table 12, where we improve the ESC-50 KNN classification accuracy from 42.8% to 47.0% and ComennCo from 27.4% to 29.9% with an SSAST backbone (Gong et al., 2022). Further details provided in Appendix B.2.
8

---

Table Impact of the projection head configuration. Top-1 accuracy of gradients on ImageNet- 100 in k-nearest neighbor classification versus the projection head configuration for KL, DINO and SimCLR gradients. "norm" indicates whether the features are Lz-normalized before being projected. As features are always Iz-normalized for the SimCLR objective, the "empty" head configuration is not applicable. The default setup marked in cyan.
VkL
VINO
V VimCLR
Norm
Projection
Acc.
Norm
Projection
Acc.
Norm Projection
Acc.
88.3
79.3
N/A
87.3
87.8
88.7
88.8
84.7
V
88.8
89.1
90.1
88.7
5.4 Ablation Studies
Projection head. To compute superviid losses, first Ly-normalize the model em- beddings (except for SimCLR) and then project them using a randomly initialized linear head. We motivate this choice empirically by ablating these components, and the results in Table show that this configuration produces the most predictive gradients for ImageNet-100.
KL
DINO
SimCLR
0.8
Accuracy
embeddings
attn.qicv
attn,proj
0.2
mlp.ft1
mlp.fc2
0.
Layer
Layer
Layer
Figure 8: Gradients from deeper layers are more predictive. Top-1 accuracy of gradients obtained from every layer of a supervised DelT ViT-B/16 in k-nearest neighbor classification on ImageNet- for the KL, DINO, and SimCLR objectives. The default setup (last layers) is marked cyan
Gradients source layer. Throughout the paper, we extract gradients from the self-attention output projection of the last transformer block. Intuitively, deeper layers provide more predictive features, and thus, their gradient should display the same behavior. This assumption is confirmed by our results Figure 8, where, for all losses, deeper layers consistently produce more predictive gradients. Regarding the choice of layer within a transformer block, for shallower blocks, the second MLP layer is significantly more predictive, but the performance gap becomes insignificant as we move towards deeper blocks, favoring (by small margin) the attention output projection, which is also more memory efficient, as it has fewer parameters compared other layers.
Discussion and Conclusion
Broader impact. Our method improves the features used for the kNN algorithm. As such, it is fundamental contribution to Machine Learning. Given the ubiquitous use of kNN, our method could have positive consequences, such as improving reliability and factuality in Retrieval Augmented Generation (RAG) systems, where Language Models are grounded in retrieved pieces text before generating answer. We do not foresee any direct negative consequence caused by method.
Impact of pretraining dataset. Our method works with various backbones, model sizes, and pretraining strategies. However, we have observed that the benefits diminish the size of the pretraining dataset increases: in Table 1, FUNGI provides smaller relative improvements on backbone pretrained with IN21K compared to one pretrained on INIK, and similarly, in Table 16 the relative improvement over EVA-CIA (Sun et al., 2023) smaller compared to the improvement over CLIP (Radford et al., 2021), as they pretrained on 2B and 400M text-image pairs respectively.

---

Computational efficiency. Computing FUNGI features introduces an overhead, which we measure in Table 27 by comparing the throughput of a DeIT ViT-B/16 when extracting gradients and embed- dings. The DINO and SimCLR losses have the largest overhead, as they forward 12 and 49 views per image, respectively. As shown in Appendix C.I, this number can be reduced, at a performance cost. However, thanks to our dimensionality reduction, the speed of kNN retrieval is not impacted.
7 Conclusion
We have shown that gradients from self-supervised objectives have predictive abilities and encode complementary information the model embeddings. Building on those findings, introduced FUNGI, which effectively combines embeddings and gradients into powerful features for retrieval- based tasks. Specifically, we have shown that FUNGI enhance the performance of kNN-based image and text classífication across models, pretraining strategies, and downstream datasets, both for full dataset and few shot setups. Moreover, have shown that FUNGI significantly boost the performance of DINO features for retrieval-ban semantic segmentation tasks.
A scknowledgements. We acknowledge the use of the Dutch national supercomputer Snellius to the experiments presented in this paper. YMA thanks Tengda Han for the initial discussions on using self-supervised gradients for tasks other than learning. WS thanks the Amsterdam ELLIS Unit for their generous funding, which allowed him to visit the Valeo laboratory in Paris. The presentation of this paper at the conference financially supported by the Amsterdam ELLIS Unit and Qualcomm.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXi::2303.08774, 2023. 19
Dimitris Achlioptas. Drtehlse-friendly random projections: Johnson-lindenstraus with binary coins. JCSS, 2003. 4, 22
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020. 3
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In ICCV, 2023. 3
Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelović, and Olivier Henaff. Towards in-context scene understanding. NeurlPS, 2024. 1, 5, 7, 8, 17, 24
Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. Semeval 2018 task 2: Multilingual emoji prediction. In /WSE, 2018. 25, 26
Francesco Barbieri, Jose Camacho- Collados,, Luis Espinosa Anke, and Leonardo Neves. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In EMNLP, 2020. 25, 26
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. NeurlPS, 2016.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mol--ing discriminative compo- nents with random forests. In ECCV, 2014. 25, 26
Y-Lan Boureau, Francis Bach, Yann LeCun, and Jean Ponce. Learning mid-level features for recognition. In CVPR, 2010.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018. 3, 20, 21
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, NeurIPS, 2020. 3
10

---

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. ICCV, 2021. 1, 3, 6, 26
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. In NLP4ConvAI, 2020. 25, 26
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a. 2, 3,5
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021. 1,
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04220 042909 2020b. 3
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021. 26
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014. 25, 26
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005. 3, 3
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In NACL, 2019. 8, 26
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2021. 2, 4,
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Berre-Emmauuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024. 19
Alexci A Efros and Thomas K Leung. Texture synthesis by non paramarrettric sampling. In ICCV, 1999.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing intermediate representations for better transfer learning. In ICML, 2022. 3
M. Everingham. L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. ICML, 2017.
Evelyn Fix. Discriminatory analysis: nonparametric discrimination, consistency properties. USAF school of Aviation Medicine, 1985.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In CVPR, 2020. 1,3
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Obow: Online word generation for self-supervised learning. In CVPR, 2021. 3
Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Interspeech, 2021. 18, 19, 26
Yuan Gong, Cheng-T Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. In AAAI, 2022. 8, 18, 19, 26

---

Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik. Multi-scale orderless pooling of deep convolutional activation features. In ECCV, 2014. 1,
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NewrlPS, 2020.
Antonio Gulli. Ag's corpus of news articles. http:: http://groups di. unipi itteiit/////// gullli//__ooopuu. sf__nes_aariclle html, 2005. Accessed: 2020-05-21. 25, 26
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In ICML, 2020. 1, 6, 24
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In ICLR, 2024. 2
James Hays and Alexei A Efros. Im2gps: estimating geographic information from single image. In CVPR, 2008.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022. 3, 17, 26
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JST-AEORS, 2019. 25, 26
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In ICANN, 2001. 2
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts the wild. NewrlPS, 2021. 5
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. TPAMI, 2010.
Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into a compact image representation. CVPR, 2010. 1,3
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022. 2
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. T-BD, 2019. 1, 19
Simon Kornblith, Mohammad Norouzi. Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. JCML, 2019. 4
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine- ane-grained cars. 2013. 25, 26
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 25, 26
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955. 19
Praveen Kulkarni, Joaquin Zepeda, Frederic Jurie, Patrick Perez, and Louis Chevallier. Hybrid multi-layer deep cnn/aggregator feature for image classification. In ICASSP, 2015. 1,3
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-augmented genera- tion for mnwledge-intensive nlp tasks. In NewrIPS, 2020.
12

---

Ping Li, Trevor J Hastie, and Kenneth W Church. Very sparse random projections. In KDD, 2006. 22 Xin Li and Dan Roth. Learning question classifiers. In COLING, 2002. 25, 26
W Johnson J Lindenstrauss and Johnson. Extensions of lipschitz maps into a hilbert space. Contemporary Mathematics, 1984. 21
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In DeeLIO, 2022. 19
Stuart Lloyd. Least squares quantization in pem. 7-/7, 1982. 19
David G Lowe. Distinctive image features from scale-invarian keypoints. IJC,, 2004. 1,3
Julien Mairal. Cyanure: An open-source toolbox for empirical risk minimization for python, c++, and soon more. arXiv preprint arXii::112..08//0 2019. 18
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aireraft. Technical report, 2013. 25, 26
Fangzhou Mu, Yingyu Liang, and Yin Li. Gradients as features for deep representation learning. In ICLR, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.029900 2018. 2
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008. 25, 26
Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. IJC,, 2001. 1,3
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:/807.03740 /807.03748, 2018. 5
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinoy2: Learning robust visual features without supervision. TMLR, 2023. 26
Valentinos Pariza, Mohammadreza Salehi, and Yuki Asano. Hummingbird evaluation for vision encoders, 2024. URL https: http:::://ttttbbooo/ wtwps/hww....oo/ppaaaaaaa/aaa//e httpm//aaaizaaa/aaaa sapeen-humminnnirdeeraaaae 6
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012. 25, 26
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. JMLR, 2011. 6
Florent Perronnin, Jorge Sánchez, and Thomas Mensink. Improving the fisher kernel for large-scale image classification. ECCV, 2010.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In CVPR, 2007. 17, 19
James Philbin, Ondrej Chum, Michael Isard. Josef Sivic, and Andrew Zisserman. Lost in quantization: Improving particular object retrieval large scale image databases. In CVPR, 2008. 17, 19
Karol Piezak. ESC: Dataset for Environmental Sound Classification. In ACMMM, 2015. 18, 25, 26
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In CVPR, 2018. 17
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, 2021. 9, 17, 26
13

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020. 8, 26
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 25, 26
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augment neural networks. In ICML, 2016. 2
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL, 2016.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self self-supe transformers and no labels. In BMVC, 2021. 3
Sivic and Zisserman. Video google: A text retrieval approach object matching in videos. In ICCV, 2003. 3,3
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NeurIPS, 30, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality sentiment treebank. In EMNLP, 2013. 25, 26
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. In TMLR, 2022. 6, 7, 17, 20, 26
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15382 2023. 9, 17,26 26
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with supervisoei for generalization under distribution shifts. In ICML, 2020.
Giorgos Tolias, Yannis Avrithis, and Hervé Jégou. To aggregate or not to aggregate: Selective match kernels for image search. In ICCV, 2013.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. TPAMI, 2008. 1
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. ICML, 2021. 26
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In ECCV, 2022. 17
Koen Van Sande, Theo Gevers, and Cees Snoek. Evaluating color descriptors for object and scene recognition. TPAMI, 2009.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. NeurlPS, 2016. 2
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 25, 26
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint 03200, 2018. 18, 25, 26
Alexander Wei, Wei Hu, and Jacob Steinhardr. More than toy: Random matrix models predict how real-world neural representations generalize. In ICML, 2022. 3
14

---

Ross Wightman. Pytorch image models.
httpp:://gataabb.comm///ggg
pytorch- image-maann models, 2019. 6, 26
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In EMNLP, 2020. 26
Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. nn prompting: Beyond-context learning with calibration-free nearest neighbor inference. In ICLR, 2023. 3
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv:/90S.00546 2019. 1
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutinall networks for text classification. In NewrIPS, 2015. 26,26
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? In NeurlPS, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017. 8
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019. 7
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. ICLR, 2022. 3, 20
15

---

Appendix
Table of Contents
A Algorithm
16
B Additional Experimental Results
16
Image Retrieval
B.2
Audio Classification
18
B.3
Linear Classification of Image Features
18
B.4
Clustering
19
B.5 Language In-Context Learning
19
Additional Supervised Objectives
20
B.7
Additional Ablations
21
C Experimental Details
23
C.I Vision Nearest Neighhor Classification Experimental Details
23
C.2 In-Context Scene Understanding Experimental Details
24
Text Classification Experimental Details
24
D Data and Models
25
E Compute Resources
26
A
Algorithm
Algorithm provides pytorch-style pseudocode for the computation of LKL, the gradient extraction, and the computation of FUNGI features (without PCA).
Algorithm PyTorch pscudocode for the KL FUNG1 features.
backbone, head proj randeh.randffestttCe grad_dim) 0.5) uniform torch, onech (feat_dim) feat_dim
model, head, proj:
projection used to dosnsample gradients
for x in dataset:
feature and projection
mxdel(x)
head(y)
xl_div(log aL_div(lo__ofttaxxda (z), softmar (uniform)) backvard() Calculate the loss and backpropagata 11.ck.. # Select the target: layer
layer
project gradients
gradients torch.catt(laas veight.grad, layer bias gradients proj gradtents.vtexx-1)
layer.bias.g...........bba... bww...iad......................a wrw.ggradd.aa imqquozza(dimm-11) dimm-1) riee(((1)
feature torch.cat [normalizaay), cormaliza dimm-1) Build the final feature
B Additional Experimental Results
In this section we first illustrate additional results that solidify the findings discussed the main paper, including the evaluation of FUNGI in image retrieval, k-nearest neighbor audio classification, linear classification and clustering and a brief investigation of the performance of gradients from clustering and masked image modeling self-supervised objectives.
16

---

In Table 8, we report the performance of embeddings and FUNGI features for some additional backbones, including CLIP and EVA-CLIP models, for which, as explained in the main text, we experience diminishing returns as the pretraining dataset size grows. Moreover, for both models, the SimCLR loss does not produce predictive gradients, which we hypothesize is due to models being saturated to a contrastive loss, as they use similar objective for pretraining.
In Table 10, we report the mean accuracy and one standard deviation (computed via numpy. std) across three seeds for a subset of our datasets, using a DeIT ViT-B/16 backbone, showing that the performance improvements of FUNGI are consistent across seeds.
In Figure we evaluate the effectiveness of FUNGI across different ViT model sizes. The findings show that FUNGI improves the results for all three ViT models (ViT-S, VIT-B, and ViT-L), with the most significant improvements observed the ViT-B model. In Table 9, we provide further evidence for the scalability of FUNGI by evaluating it on several VIT-L and VIT-H backhones.
In Table 7, we report the performance of FUNGI for in-context retrieval etrrevval-base semantic segmentation on ADE20K, and show that our method outperforms DINO across all memory bank sizes and is competitive against HummingBird.
Table FUNGI features improve in-context semantic segmentation on ADE20K. We report the mloU for retrieval-beit based semantic segmentation ADE20K, comparing DINO baseline against FUNGI features and the self-supervised HummingBird model. Results from Balazevic et al. (2024) marked with f. We resize each image 512 x 512 and extract 32 1024 patch features.
Memory Bank Size
Backbone
Features
1024
10a
1024
10a
1024 x 10"
DINO VIT-S/16
Embeddings
11.4
17.0
DINO S/16
FUNGI
16.1 14.7
20.0
15.5
22.3 f5.3
DINO ViT-B/16
Embeddings
14.5
18.3
20.8
DINO VIT-B/16
FUNGI
19.2 14.7
23.5
15.2
25.2 14..4
BurdingBird ViT-B/16"
Embeddings
28.3
Table 8: Additional backbones. Average accuracy of embeddings and FUNGI features in k-nearest neighbor classification across 11 datasets for CLIP (Radford et al., 2021; Sun et al., 2023), AugReg (Steiner et al., 2022), DelT III (Touvron al., 2022) and masked autoencoder (He et al., 2022) models. "K", "D" and "S" stand for KL, DINO and SimCLR, respectively.
EVA CLIP
CL.P
AugReg IN21K
AugReg IN21K ViT-S/16
Deff VIT-B/16
MAE VIT-B/16
ViT-B/16
ViT-B/16
ViT-B/16
Full Dataset Emheddings
83.2
73.7
66.4
65.6
64,2
24.0
K
83.8 10.6
76.9 13.2
67.1 10.7
65.3 J0.3
10.6
44.4 120.4
K
84.1 10.9
77.7 14.0
68.6 f2.2
67.1 11.5
67.3 f3.1
45.4 121.4
S
83.4 1022
74.6 10.9
69.3 12,9
67.6 12.0
68.2 14.0
38.8 114.8
Few Shot
Embeddings
53.1
43.0
38.6
37.4
34,9
7.5
K
54.0 10.9
47.2 14.2
40.2 f1.6
37.7 10.3
36,2 11.3
18.5 111.0
+ K
54.1 11.0
47.9 14.9
40.3 11.7
38.7 11.3
37.2
19,2 111.
K
53.7 10.6
14.4
41.0 12.4
39.5 12.1
39.6 14.7
14.0 16.5
B.1 Image Retrieval
We evaluate the performance of FUNGI features in image retrieval using the revisited (Radenović et al., 2018) Oxford (Philbin et al., 2007) and Paris (Philbin al., 2008) landmarks datasets. We report the mean average precision (mAP) for both the medium (M) and hard (H) splits.
For this task, we use FUNGI features built with DINO and KL gradients, as the SimCLR gradients did not result good retrieval performance. The results displayed in Table show that FUNGI features improve the retrieval abilities of all backbones, except for DINOv2. Our method is particularly effective when applied on CLIP backbones: on the Paris hard split, we improve by 12.4% and 7.2% for CLIP and EVA-CLIP respectively.
17

---

Table Scalability experiments. Average accuracy of embeddings and FUNGI features in k-nearest neighbor classification across 11 datasets (7 for ViT-H) for ViT-L and H backbones. "K", "D" and "S" stand for KI., DINO and SimCLR, respectively.
DINOv2
CLIP
DelT VIT-H/14
AugReg IN2IK
ViT-L/16
ViT-L/14
ViT-L/16
Full Dataset
Embeddings
80.5
80.2
77.2
74.7
K
81.2 10.7
82.4 12.2
77.3 10.1
75.0 10.3
+K+D
81.6 f1.1
82.9 12.7
78.0 10.8
76.2 11.5
+
82.3 11.
81.1 10.9
78.8 11.6
76.5 11.8
Few Shot
Embeddings
47.1
48.5
48.3
47.7
+ K
48.6 1.5
51.8 13.3
47.7 J0.6
48.3 10.6
D
49.0 11.9
52.9 14.4
46.9 11.4
48.5 10.8
+ +
51.4 14.3
50.3
11.8
50.4
12.1
50.2 12.5
Table 10: FUNGI is consistent across seeds. Average accuracy in kNN classification and one standard deviation for FUNGI features on 8 datasets, measured across three seeds using a DelT ViT-B/16 backbone. "K", "D" and "S" stand for KL, DINO and SimCLR, respectively.
Cars
CUB
DTD
ESAT
Pets
Food
FGVC
Flowers
Embedddings
32,3
56.0
58.6
90.7
90.8
60.5
23.5
56.9
+ K
33.5 : 0.2
57.9
1: 0.1
60.4
: 0.2
91.6 0.2
91.3 0.2
61.5 0.1
22.9 = 0.1
60.7 0.6
60.9 0.2
63.1
9.0
6.22
0.4
64.3
K
D
39,3 = 0.3
63.6 +
64.1 0.2
95.0
92.1
6.1
28.3 0.6
69.3 + 0.5
B.2 Audio Classification
We evaluate FUNGI on the audio modality using SSAST (Gong et al., 2021, 2022), self-supervised audio spectrogram transformer trained for audio and speech classification, the backbone. We construct FUNGI features using KI. and SimCLR gradients, and test their performance in k-nearest neighbor classification on ESC 50 (Piczak, 2015) and SpeechCommands V2 (Warden, 2018).
We use the same formulation as the vision experiments for the LKL. and LSiBLL objectives. However, for the latter, we obtain 16 views of the same clip by adding uniform noise following Gong et al. (2022). If we define the filter bank of an audio clip as c E h the noise-augmented clip è is computed
D2
UOO, A)
UOO 1).
10
Finally, è is shifted by factor sampled from a discrete uniform distribution U(-10, 10). The complete list of hyperparameters used for the audio classification experiments is reported in Table 13.
The results are reported in Table 12, and show that FUNGI features built using KI. gradients yield promising results, improving by up 4.2% the baseline. On the other hand, using SimCLR gradients does not consistently yield improvements. It rather often causes a performance drop when combined with KL gradients. As with text classification, further research is needed to determine the optimal self-supervised objectives and data augmentation to extract predictive gradients.
B.3 Linear Classification of Image Features
We evaluate FUNGI features in logistic regression, using the implementation from the cyanure library (Mairal, 2019). We train each classifier for a maximum of 300 epochs (30 in the case of ImageNet- IK) using Le regularization. For each dataset and feature combination (i.e., embeddings, embeddings + VKL, etc.), we pick the best regularization parameter between linearly spaced values in the interval [5 x 10-6 5 10 10-4] using the validation set. For datasets without validation set, generate one using an 80/20 stratified split. The final model is trained using the entire training dataset.
We report the results in Table 15 and Table 16 and find that, in linear classification, FUNGT features are most effective for backbones pretrained using a supervised objective. In contrast, self-supervised backbones do not benefit much. This is especially evident for DINO and DINOv2, where
18

---

retrieve in-context Table 11: FUNGI improves image retrieval. Mean average precision (mAP) of embeddings and FUNGI for retrieval on the Paris (Philbin et al., 2008) and Oxford (Philbin et al., 2007) landmarks datasets, for both medium (M) and hard (H) splits. "K" and "D" stand for KI. and DINO, respectively.
DINDv2 B/14
DINO
CLIP
EVA CLIP
DeIT
VIT BT-6/16
VIT-B/16
VIT B/16
VIT-B/16
Oxford
M
H
M
H
H
M
M
H
Embeddings
69.7
42.0
39.2
11.0
31.4
10.8
36.7
12.7
36.6
12.6
K
70.4 10.7
42.6 10.6
38.6 10.6
11.6 70.6
40.4
19.0
14.5 13.7
39.9
13.2
13.9 11.
36,9 10.3
12.6
10.0
69.4 10.3
41.2 10.8
40,4 f1.2
12.
41.4
710.0
15.3
14.5
40.9
14.2
12.1
38.8 12.2
12.7 10.1
70.1 10.4
42.0 10.0
39.8 10.6
13.0 2200
42.6 111.2
14.7
13.9
41.2
14.5
14,9 72.2
39.1 12.5
12.8 10.2
Paris
M
H
M
M
H
Embeddings
89.4
63.8
37.6
64.6
40.4
69.8
46.7
63.0
37.2
K
88.7 j0.7
76.2
64.5
38.4 10.8
74.7
52.8
1124
75.2
134
53,9 17.2
10.6
37.7 10.5
+D
89.0 )0.4
76.2 [1.3
65.6
11.8
39.0 114
72.0 17.4
47.8
17.4
72.9
13.1
50.2 13.5
65.7 12.7
40.2 13.0
+
88.7 10.7
11.6
65.8
12.0
39.5 119
710.5
52.5
112.1
74.9
15.1
53.0
16.3
65.6
12.6
40.0 12.8
Table 12: FUNGI works for audio. Top-1 accuracies in k-nearest neighbor audio classification of embeddings and FUNGI features obtained from a SSAST backbone (Gong et al., 2022, 2021). "K" and "S" stand for KL. and SimCLR, respectively.
ESC 50
SpeechCommands V2
Full
5-shot
Full
5-shot
Embeddings
42.8
20.0
27.4
5.3
+ K
47.0 14.2
21.2 11.2
12.5
6.1 f0.8
+ S
45.2 12.5
19.0 11.0
42.0
5.5 f02
+ S
45.8 +3.0
21.0 +1.0
27.3 J0.1
5.8 10.5
Table 13: Audio classification experimental details. Parameters used to extract audio encoder gradients for the CKL (left) and CimCLR (right) objectives.
Parameter
Value
Parameter
Value
(Positive, Negative) Views
16, 2
Temperature
15
Projection Dim
768
Projection Dim
768
Negatives Batch Size
64 x 2
Temperature
0.07
FUNGI often yields worse results, especially in a few shot scenarios. Contrary to the k-nearest neighbor classification results, the best feature combination is backbone specific, and in Figure we show that significant performance improvements can be achieved by picking the best feature combination for each backbone.
B.4 Clustering
We evaluate the performance of FUNGI features built with KL and DINO gradients in k-means clustering (Lloyd, 1982) of image features using faiss (Johnson et al., 2019; Douze et al., 2024). Given a dataset with C' classes, we compute C' clusters via k-means and match clusters to classes using the Hungarian algorithm (Kuhn, 1955). We then measure the average overlap between clusters and classes. The results in Table 14 show that, on average, FUNGI features built with KL and DINO gradients improve the overlap between clusters and classes. In particular, FUNGI can improve the clustering performance by up to 15.8% on Oxford-II7 Pets for CLIP ViT-L/14 backbone.
B.5 Language In-Context Learning
Liu al. (2022) has shown that selecting examples for in-context learning using retrieval outperforms random baseline, and that using encoders fine-tuned on task similar to the target one further improves performance. Thus, we hypothesize that using FUNGI features examples can improve performance, as they contain adaptation signal to the task at hand. We test this hypothesis by measuring the in-context classification performance of GPT 4o mini (Achiam et al., 2023) using examples retrieved either using embeddings or FUNGI features built with KL and
19

---

Table 14: FUNGI improves clustering. Overlap between classes and clusters found via k-means clustering of embeddings and FUNGI features. "K" and "D" stand for KL and DINO respectively.
Cars CUB
DTD
ESAT
C100 C10
Pets
Food
INTK PGVC Hlowers Mcan
CLIP ViT-L/14
Emheddings
49.4
52.8
44.7
52.4
44.5
65.0
55.6
72.9
50.9
29.1
69.4
53.3
+K
55.0
57.9
53.1
55,9
56.1
68.5
67.5
71.0
55.3
33,8
66.8
58,3 f5.0
+ +K+D
56.2
57.0
55.4
55.4
55.9
66.5
71.4
72.7
55.7
35,1
66.6
58.9 15.6
DINOv2 VIT-L/14
Emheddings
28.6
63.7
50.6
55.8
70.3
77.6
79.1
68.8
61.4
21.5
79.2
59.7
+K
29.3
64.0
48.5
55.9
67.2
84.5
74.2
67.9
59.9
20.3
80.2
59.2 L0.5
+K+D
28.9
64.9
48.3
66.6
70.5
85.2
75.2
66.5
60.3
22.0
81.2
60.8 11.1
ViT-B/16
Embeddings
15.8
43.9
55.7
50.0
83.9
76.0
28.9
78.5
14.9
41.3
47.5
+K
16.3
34.5
44.3
65.1
49.2
82.7
79.6
28.6
77.5
14.5
43.0
48,7 f1.2
+ +K+D
17.8
36.7
48.7
61.2
49.0
82.1
77.1
31.8
76.6
15.4
49.2
49.6 12.1
Dataset
Few Shot
Embeddiingg
FUMGI
0.30
32%
Accuraty
0.75
0.02
12.00
+22
+22%
0.99
+1.0
+226
0.55
6.00
WRLCE
2007
Figure 9: FUNGI works across backbones for linear probing, Accuracy in logistic regression- based image classification of embeddings and FUNGI features on various ViT backbones, both for full dataset and few shot setups, averaged over 11 datasets. For the FUNGI features, we chose the best performing combination across datasets. "AR" indicates AugReg backbones (Steiner al., 2022).
SimCLR gradients using a BERT backbone. We retrieve the top 20 most similar training examples for each test sample and ask the model to predict its label using prompt template similar to the one shown in Listing 1. For a fair evaluation, we set the model temperature O. The results listed in Table 17 show that examples retrieved using FUNGI features improve the in-context learning accuracy by up to 2.5% on banking-77.
banking relsted You nust chooss single class following corna-ssparated liat:
have
appropriate
(liet possible labels, comna separated)
You nust reply only with
naze, nothing nore. Hore's sone examplea:
((text, label) paira froz. the training aut)
The quary sample Equary text.)
Listing 1: The prompt used for the in-context learning evaluation of embeddings and FUNGI features on banking-77 using a GPT 4o mini backbone. The labels are given as strings, e.g. exchange_rate
B.6 Additional Self-Supervised Objectives
In this section, we study the performance of gradients obtained by two additional self-supervieed objectives, DeepCluster (Caron et al., 2018) and iBOT (Zhou et al., 2022) in k-nearest neighbor self-distillation classification on ImageNet- using DelT ViT-B/16 backbone. DeepCluster and explicit custering-based self-supervised method that alternates between clustering image features and training a model predict cluster assignments. iBOT an extension of DINO that combines image and patch-level objectives, the latter implemented as latent-space masked image modeling (MIM) objective, where learnable patch token replaces subset of patches, and the model must reconstruct them.
20

---

Table 15: The best gradients for linear probing are backbone-specific for the main backbones. Average accuracy across 11 datasets for logistic regression praesssio--bbeedd image classification of embeddings and FUNGT features. "K", "D" and "S" stand KI., DINO and SimCLR, respectively.
DINOv2 DINO DefT
MoCov3
DefT
AugReg INIK
AugReg IN1K
ViT-B/14 ViT-B/16 ViT-B/16
ViT-B/32
ViT-B/32
ViT-B/16
ViT-S/16
Full Dataset
Embeddings
88.3
81.0
78.2
77.3
73.1
71.6
71.4
KK
88.3 10.0
80.4 [0.6
78.5 10.3
77.7 10.4
73.4 10.3
70,9
70.5 J0.9
+
88.8 10.5
81.2 10.2
80.7 12.5
79.4 12.1
76.2 13.1
73.0 414
73.0 116
+ +K+D+S
88.7 10.4
80.7 J0.3
80.5 12.3
78.7 11.4
76.4 13.3
73.1 11.5
73.0 116
Few Shot
Embeddings
76.7
62.9
61.0
58.0
57.2
54.8
54.4
76.3 10.4
62.2 J0.7
10.7
57.6 J0.4
57.8 10.6
54.4 J0.4
53.5 J0.9
+ K D
76.7 10.0
62.4 [0.5
63.5
12.5
59.2 11.
60.2 13.0
56.5
11.7
55.9 11.
K D S
76.6 J0.1
61.6 L1.
61.6
63.4
12.4
12.4
58.7
10.7
60.1
12.9
57.0
12.2
56.0 116
Table 16: The best gradients for linear probing are backbone-specifie for the additional back- bones. Average accuracy across 11 datasets for logistic regression-based image classification of embeddings and FUNGI features. "K", "D" and "S" stand for KI. DINO and SimCLR, respectively.
EVA CLIP AugReg IN2IK CLIP DeIT-III
AugReg IN2IK
AugReg IN2JK
MAE
VTT-B/16
VIT-L/16
VIT-B/16
VIT-B/16
VIT-B/16
VIT-S/16
ViT-B/16
Dutuset
Bibheddings
86.9
82.9
81.8
77.1
76.8
75.6
38.6
+
87.2 10.3
82.0 10.9
82,6 10.8
78.1 11.0
76.1 J0.7
74.4 11.2
63.4 124.8
+K+D
87.8 10.9
10.4
83,9 f2.1
80,2 13.1
77,9 11.1
76.6 $1.0
66.2 127.6
+
87.7 10.8
83.2 10.3
83.0 11.2
79.9 12.8
77.8 11.0
76.8 11.2
63.1 124.5
Few Shot
Embeddings
78.0
68.9
66.2
60.5
58.4
57.6
23.9
KK
78.6 70.6
69.0 10.1
69.3 13.1
60.3 [0.2
59.1 10.7
$7.6 10.0
36.0 f12.1
+
78.9
70.1
70.4
62.5
12.3
59.3
37.3 113.4
+K+D+S
77.5
69.5 10.6
65.9 10.3
61.7
59.8 11.4
58.6 11.0
32.3 18.4
The results are displayed in Figure 10, and show that the objectives used this work achieve similar performances the model embeddings, time, iBOT and DeepCluster instead produce gradients with relatively poor predictive performance. For the former, possible reason is that it incorporates dense loss, whose gradients may not help to discriminate examples on the image level. Regarding DeepCluster, models pretrained using this strategy had worse performance in retrieval tasks compared to supervised pretraining (Caron et al., 2018), 8), which may explain the poor retrieval abilities of its gradients.
surpassing them in the case of DINO. At the same
B.7 Additional Ablations
DINO data augmentation and head. To maximize the signal-to-noie ratio, only use local and global crops for the DINO data augmentation. We validate this choice empirically, and the results in Table 18 show that random crops produce standard data augmentation policy. Moreover, we also empirically validate the choice using two independent heads for the DINO loss in Table 18, showing that this choice beneficial for kNN classification.
discriminative gradients compared the
Random Projections. In order to reduce the dimensionality of the gradients we use random projections, an unsupervised dimensionality reduction technique based on the lemma by Lindenstrauss
Table 17: FUNGI improves language in-context learning. Classification accuracy GPT 4o mini in language in-context learning with examples retrieved using embeddings or FUNGI features, both extracted from a BERT backbone.
Banking-77
SST
Embeddings
88.7
52.5
+ KI. + SimCLR
91.2 12.5
52.9
10.4
21

---

Gradients Accuracy ImageNet-100
Im improvement vs Model Size
0.94
features
0.7b
Embeddings
+1.8 11.8%%
FUNGI
0.90
Accuracy
0.88
0.72
0.70
42.95%
0.84
+1.93%
0.82
0.68-
Guaierr
Deep
DINO
(No NM
IBOT
(MIM)
SImCUR
VIT S
VIT
VIT
Figure 10: Not all objectives produce good predictive gradients. Top-1 accuracy in k- nearest neighbor classífication of gradients obtained from different self-supervised ob- jectives using a DelT ViT-B/16 backbone. "MIM" stands for masked image modeling.
Figure 11: Gains across backbone sizes. Ac- curacy in k-nearest neighbor image classifi- cation averaged across 11 datasets using the model embeddings and FUNGI features ex- tracted from AugReg backbones of increasing size.
Table 18: DINO head configuration and data augmentation. Top-1 accuracy on ImageNet-100 100 in k-nearest neighbor classification for the DINO gradients using shared or independent teacher and student heads (left) and with respect to the data augmentation policy (right).
Independent Heads
Accuracy
Data Augmentation
Accuracy
88.4
DINO
88.9
89.1
Random Crops
90.1
& Johnson (1984), which states that for set of N d-dimensional points and projection subspace of dimension & 2 Kg- = ( loog [)) there exist a mapping f IR + I that preserves euclidean distances with distortion of most = 1. This mapping can be either implemented as Gaussian, binary (Achlioptas, 2003) or sparse (Li et al., 2006) projection. Our method uses a binary projection, as it's more memory-efficient than Gaussian matrix, and in Table 19, compare its performance to the other initializations. The results show that the initialization has little impact on downstream performance, favoring binary projections by small margin.
Table 19: The random projection initialization has little impact on performance. Comparison of the downstream accuracy of FUNGI features built with gradients projected using matrices with different initializations. We report the mean and one standard deviation measured across three runs using the Flowers 102 dataset and a DelT ViT-B/16 backbone. No further dimensionality reduction was applied to the concatenated features.
Binary
Gaussian
Sparse
Embeddings
56.9
56.9
56.9
K
59.7 + 0.4
60.2 - 0.2
60,0 + 0.1
K
+
63.5 + 0.1
63.2 - 0.5
63.1 0.2
+
69.2 + 0.6
69.0 + 0.7
68.9 + 0.9
PCA. We use Principal Component Analysis (PCA) to combine data from multiple losses at an iso-storage and retrieval speed cost. Given dataset of FUNGI features, split and use it to transform training and test splits. Table 20 lists the PCA dimensionalities used for each model architecture and shows that they do not cause decrease performance but rather provide minor improvement average. Moreover, we compare PCA against binary, Gaussian, and sparse random projections in Table 21 and find that all random projection- -aased methods result in drop in accuracy compared to the original features, while PCA consistently improves performance.
the PCA on the training
22

---

Table 20: PCA does not degrade performance. PCA oulput dimensionalities with respect to the backbone architecture (left) and its impact on k-nearest neighbor image classification accuracy averaged across 11 datasets using a DelT ViT-B/16 backbone (right).
Architecture
PCA Dim
No PCA
PCA
VIT-S/16
384
Embeddings
65.1
65.3
10.2
ViT-B/16, ViT-L/16
512
KI.
66.0
66.3
10.3
BERT, T5
512
+ KL DINO
67.8
68.1
10.3
SSAST
512
KL DINO + SimCLR
69.8
70.1 10.3
Table 21: PCA is the best dimensionality reduction method. The mean-pert per-class accuracy of embeddings and FUNGI features from DeIT ViT-16/B backbone Flowers 122, transformed with PCA and random projections. We report the mean and one standard deviation across three seeds.
No Reduction
PCA
Proj. (Binary)
Proj. (Gaussian)
Proj. (Sparse)
Embeddings
57.2 + 0.0
61.6 + 0.0
55.5 = 0.4
55.8 + 1.0
56.0 + 0.5
+ K
59.4 0.0
64.0 0.0
59.0 0.8
58.2 0.8
58.7 + 0.3
+ K D
64.3 + 0.0
68.3 + 0.0
62.9 + 0.4
62.2 + 0.3
61.9 + 0.5
+ +
0.0
70.9 + 0.0
67.7 = 0.5
67.0 + 0.4
66.9 + 0.7
C Experimental Details
C.1 Vision Nearest Neighbor Classification Experimental Details
Hyperparameters. We use three losses to extract gradients from vision backbones: CKL. DINO and LSmCLL.. The parameters used for each loss are shown in Table 22. This of parameters is used consistently across backbones and datasets. While LKL. and LDINO are robust to the choice of hyperparameters, LSimCLR is particularly sensitive the number positive views, as shown in Figure 12, where performance increases in logarithmic fashion as more positive views are used, at the cost of gradient extraction speed. While this behavior consistent across datasets, it has the most significant impact in datasets with many classes, e.g., Flowers
SimCLE data augmentation and loss details. Given image, we patchify it in 49 overlapping views as follows: we first resize the input image to (224, 224), and then extract 49 patches of size 112 x 112, using stride corresponding 1/6 of the patch size. No other style or color augmentation is used. As the number of patches increases, so does the memory required to compute the loss and the gradients. This problem can be partially tackled by precomputing the negative batch, which in our experiments is picked randomly from the training dataset and kept constant for every input. Moreover, we can observe that the SimCLR loss only defined for positive pairs, so we only need compute the similarity of positive pairs to all other pairs, significantly reducing the size of the similarities matrix and memory usage.
Table 22: Image gradients setup. Data augmentation and loss parameters used to extract gradients from vision encoders for LKL, LSimCL and LDNOO (left to right).
Parameter
Value
Parameter
Value
Proj. Dim.
2048
Parameter
Value
Pos. Views
49
Global Crops
2 @ 224 x 224
Neg. Views
Global Crop Scale
0.25, 1.0
Proj. Dim.
768
Proj. Dim.
96
Local Crops
10 @ 224 x 224
Temp. 15
Neg. Batch
256
49
Local Crop Scale
0.05, 0.25
Temp.
0.07
Teacher Temp.
0.07 0.1
Student Temp.
23

---

SimCLR Accuracy vs Number of Patches
SimCLR Num Patches vs images/s
67
Accuracy
70
Images/s
66
50-
Accuracy
65
40
64
4
16
25
25
36
Number of Patches
Number of Patches
Figure 12: SimCLR is sensitive to the number of views. The SimCLR gradients mean-per-class accuracy Flowers 102 with respect to the number of patches (left) and the images/s versus the number of patches (right) using supervised DelT ViT-B/16 backbone.
C.2 In-Context Scene Understanding Experimental Details
For the evaluation of the in-context scene understanding abilities of FUNGI features we closely replicate the setup described by Balazevic al. (2024) for both the full and few shot setups, with two minor exceptions: (1) we use single augmentation epoch for the full dataset evaluation and (2) we an anisotropic quantization threshold of 0.2 for the nearest neighbor index, this parameter was not specified in the paper. The full set of parameters for the evaluation, loss computation and data augmentation reported in Table 23. As for data augmentation, use the same policy of Balazevic et al. (2024), and apply each augmentation independently.
In order to construct FUNGI features for this task, implement SimCLR loss that contrasts patch tokens from an input image to their nearest neighbors retrieved from a supporting memory bank. In practice, we:
Construct a memory bank of image patches of the same size the one used for evaluation and its nearest neighbor index with ScaNN (Guo et al., 2020) following the procedure by Balazevic al. (2024). We call this our support index.
Then, for each image, we:
1. Resize it to 512 x 512, compute its [CLS] and patch tokens and project them with linear head. Excluding the [CLS] token, each image is mapped to 32 1024 features, as all our backbones use patches of size 16.
For each token, retrieve its two nearest neighbors from the support index and randomly drop 50% of them.
3. Compute the SimCLR loss, where the patch tokens constitute the positive and the neighbors the negative batch. This allows us to compute a per-patch gradient.
4. Drop the [CLS] token, as it does not correspond to real image patch.
Construct FUNGI features as Equation 8, where f(x) maps an image to patches of dimension d, L2 normalization is defined as z' z/||=||a and cat indicates concate- nation.
F = cat' (V SSmLLR, fx))) f(z) : 3x512x5112 + 1024xd
(8)
C.3 Text Classification Experimental Details
The parameters used to extract gradients from text encoders for LKL and LSinCLR are shown in Table 24. The gradient source layer is always the attention output projection of the last transformer encoder block. We use the same parameters across backbones. No data augmentation is used for the LKL. while for LSSCCLR the views are obtained by randomly deleting words independently, where each word has 10% probability of being deleted.
24

---

Table 23: In-context scene understanding setup. Parameters (left) and data augmentation (right) used for the in-context scene understanding task for both full dataset and few shot setups. For the computation of LSmCLR we use 1025 views as we include the [CLS] token, which discarded afterwards. The retrieved negatives indicate the number of neighbors retrieved from the support index, while the loss negatives the number of neighbors used for the loss computation. The color data augmentations are applied independently. in the order shown here.
Parameter
Full Dataset
Pew-Shots
Memory Bank Size
See Table
2048 x 10
Nearest Neighbors k Temperature Augmentation Epochs
30
90
0,02
0.1
I
8
Parameter
Value
ScaNN Index
Random crop p
1.0
Num Leaves
512
512
Scale factor
[0.5, 2.0]
Num Leaves to Search
32
256
Brightness jitter p
0.5
Reordering Num Neighbors
120
1800
Contrast jitter p
0.5
Dimensions per Block
4
4
Saturation jitter p
0.5
Anisotropic Quantization
0.2
0.2
Hue jitter
0.5
CSImCLR
Max brightness
0.1
Max contrast A
0.1
Support Index Size
See Table 3
2048 x 10*
Max saturation A
0.1
Projection Dim
96
96
Max hue A
0.1
Positive Views
1025
1025
Negatives Batch Size
1025
1025
SimCLR Temperature
0.07
0.07
Retrieved Negatives per Patch
2
2
Loss Negatives per Patch
1
1
Table 24: Text classification experimental details. Parameters used to extract text encoder gradients for the LxI. (left) and LSmCLLR (right) objectives.
Parameter
Value
Positive Views
12
Parameter
Value
Negative Views
Temperature
15
Projection Dim
256
Projection Dim
768
Negatives Batch Size
256 x 2
Temperature
0.07
Word Deletion p
0.1
D Data and Models
We investigate the performance of our gradient-enhanced features on 11 image classification datasets, namely CIFAR 10 and CIFAR 100 (Krizhevsky et al., 2009). Oxford Flowers 102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), ImageNet- (Russakovsky et al., 2015), FGVC Aircraft (Maji et al., 2013), CUB 200-2011 (Wah et al., 201 I), Oxford-IIT Pets (Parkhi et al., 2012), Stanford Cars (Krause et al., 2013), DTD Textures (Cimpoi et al., 2014) and EuroSAT (Helber et al., 2019), 5 text classification datasets: TREC (Li & Roth, 2002) in its coarse version, banking-77 (Casanueva et al., 2020), Stanford Sentiment Treebank (SST) (Socher et al., 2013) in its fine-grained version, AG news (Zhang al., 2015; Gulli, 2005) and tweet eval (emoji) (Barbieri et al., 2018, 2020) and 2 audio classification datasets: ESC 50 (Piczak, 2015), an environmental sound classification dataset, and Speech ComechCo V2 (Warden, 2018), keyword spotting task, where the goal is to classify utterances into a predefined set of words. The datasets, their license and citations are also listed in Table 25.
We follow the evaluation protocol for each individual dataset and report the top-1 accuracy for CIFAR 10 and 100, Food101, ImageNet-1 K, Stanford Cars, EurOSAT, DTD Textures, CUB 200-2011, TREC, banking-77, SST, AG news, tweet eval (emoji), ESC and SpeechCommands V2, and the mean-per-class accuracy for Flowers 102, FGVC Aireraft and Oxford-IIT IIT Pets.
25

---

We use the default splits defined by torchvision or the dataset authors where possible. As EuroSAT does not explicitly define a test split, we use an 80/20 stratified split as indicated by the dataset paper. We always report metrics on the test splits, with the exception of ImageNet, for which we use the validation split.
Table 25: Datasets. Summary table of all datasets used in this paper, their license and citation.
Dataset
Type
License
Citation
CIFAR
Image
Unknown
Krizhevsky al. (2009)
CIFAR 100
Image
Unknown
Krizhevsky al. (2009)
Stanford Cars
Image
Custom (Non Commercial)
Krause al. (2013)
DID Textures
Image
Custom (Research Only)
Cimpoi et al. (2014)
EuroSAT
Image
MIT
Helber al. (2019)
CUB 200 (2011)
Image
Custom (Research Only, Non Commercial)
Wah al. (2011)
Oxford-IIT Pets
Image
CC BY-SA 4.0
Parkhi al. (2012)
Food101
Image
Unknown
Bossard al. (2014)
FGVC Aircraft
Image
Custom (Research Only, Non Commercial)
Maji al. (2013)
Flowers102
Image
Unknown
Nilsback & Zisserman (2008)
ImageNet IK
Image
Custom (Research Only, Non Commercial)
Russakovsky et al. (2015)
ImageNet 100
Image
Custom (Research Ouly. Non Commercial)
Russakovsky et al. (2015)
TREC
Text
Unknown
Li Roth (2002)
Banking-77
Text
CC BY 4.0
Casanueva et al. (2020)
SST
Text
Unknown
Socher et al. (2013)
News
Text
Custom (Non Commercial)
Zhang et al. (2015); Gulli (2005)
Tweet Eval
Text
Unknown
Barbieri et al. (2018, 2020)
ESC 50
Audio
CC B.C BC 4.0
Piczak (2015)
SpeechCommands V2
Audio
Warden (2018)
We evaluate FUNGI features across several architectures, pretraining strategies and model sizes. These are listed in Table 26, alongside their license, data type and citation.
E Compute Resources
The gradient features were extracted using machine with a single NVIDIA A100 GPU with 40GB of VRAM. Considering the inference times listed in Table 27, replicating the k-nearest neighbor image classification results would require approximately 27 GPU hours per backbone using float16, which we use throughout all our experiments. As we evaluate our method across 17 vision backbones, reproducing these results would require 459 GPU hours. As for text and audio classification experiments, they require around GPU hours per backbone, for a total of 9 hours. The extracted gradient features were reused for the linear probing and clustering experiments, the former requiring 168 hours machine with single AMD EPYC 7H12 CPU and the latter requiring 18 hours machine with a single NVIDIA A100 GPU with 40GB of VRAM.
Table 26: Models used in the paper. Summary table of all architecturrs/ppetraiinn strategies evaluated in the paper. along with their license, citation, and implementation, if applicable.
Model
Type
License
Citation
Masked Autocncoder
Image
CC BY-NC 4.0
He et al. (2022); Wightman (2019)
AugReg
Image
Apache 2.0
Steiner et al. (2022); Wightman (2019)
DelT
Image
Apache 2.0
Touvron et al. (2021)
DINO
Image
Apache 2.0
Caron al. (2021)
DINOv2
Image
Apache 2.0
Oquab et al. (2023)
CLIP
Image
MIT
Radford al. (2021); Wightman (2019)
EVA-CLIP
Image
MIT
Sun et al. (2023); Wightman (2019)
MoCov3
Image
CC BY-NC 4.0
Chen et al. (2021)
BERT
Text
Apache 2.0
Devlin et al. (2019); Wolf et al. (2020)
T5
Text
Apache 2.0
Raffel al. (2020); Wolf et al. (2020)
SSAST
Audio
BSD 3- 3-Clause
Gong et (2022, 2021)
26

---

Table 27: FUNGI introduces speed overhead. Embeddings and gradients extraction speed measured in images/second on an NVIDIA A100 GPU for DelT ViT-B/16 backbone. The gradients speed include the random projection step. The performance column reports the aceuracy averaged across 11 datasets for the combination of a single gradient with the model embeddings. f indicates k-nearest neighbor inference on CPU.
Features Source
Images/s
Inference Speed (samples/s)
Performance
Embeddings
479
2700
67.3
VKL
344
2700
68.2 10.9
DINO
32
2700
70.1 12.8
V SimCLR
12
2700
70.9 13.6
Finally, additional experiments such image retrieval, in-context learning, and ablation studies re- quired approximately 84 hours, while preliminary experiments for this paper required negligible amount of compute.
27

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: we analyze the predictive abilities of gradients and show that they are comple- mentary the model embeddings Section 3. We show that our method improves k-nearest neighbor classification for vision, text and audio in Sections 5.1 and 5.3. We evaluate FUNGI in linear probing in Appendix B.3, image retrieval Appendix B.1 and visual in-context learning in Section 4.3.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: discuss limitations concerning the backbone pretraining datasets and the computational efficiency in Section
Guidelines:
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: we do not provide any theoretical result in the paper.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: we describe all our self-supervised objectives and data augmentation in Section 4. Additional details regarding the self-supervised objectives are described in Appendices C.1, C.2 and C.3. The complete set of hyperparameters used for all our experiments are listed in Table 22, Table 23 and Table 24. Furthermore, Appendix contains pytorch-like pseudocode that illustrates how to perform gradient extraction for one of our objectives. Finally, we open sourced the code used to run our experiments at htttps://githie
wwticiiiiioontttiiiiieeoe no-train-aan train-all-gain
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: we evaluate method using only publicly available datasets and backbones, and we release the code to reproduce our experimental results at this URL https: // github. CoomWWalerSSmonn. mani/nootrrri tooo--rinnnalleeeee -aii.. The code documentation and the supplementary materials contain all the information needed to replicate our experiments.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: to the best of our knowledge, we have provided all the necessary hyperparame- ters, dataset splits, and experimental details to fully reproduce our results.
7. Experiment Statistical Significance
28

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Table 10, we show that our method is consistent across three seeds by reporting the average performance and standard deviation. The results support claim that FUNGI features provide consistent improvements over the model embeddings. We calculate the standard deviation using numpy. std. We only report the standard deviation for this experiment, as computing it for all backbone and dataset combinations would require a significant amount of compute.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We fully disclose the hardware and the run time of all our experiments in appendix E.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://neurpppppe http://eurrrpp.cppppp.epeee cc/public/ heroollc//hhhheeee csGuidelines"
Answer: [Yes]
Justification: we have read the code ethics and made sure the paper conforms to in every aspect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: in the main paper, discuss potential broader impacts, but make a fundamental contribution to machine learning, we do not foresee any direct negative consequences from our method.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
12. Lieenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we properly cite all models, datasets, and code implementations used to run our experiments. A summary of all models and datasets used in the paper are available in Table and Table 25, along with their license and citation. We made sure to adhere to usage guidelines and license of each individual asset.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]

---

Justification: we release two new assets: (a) the code to replicate our experimental results, available https: /tt/tggttttbb.. com woom//aaterimmocnii///aeeeeeeeee library to extract FUNGI features from ViT backbones, available at WalterrSinoncinn/ffnnn fungivviiiin... Both repositories include proper documentation and are released under the MIT license.
train- all-gain and (b) thub. com/
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
30


Reviews:
[{"review_id": "udwCsuc7D2", "reviewer": "Reviewer_gxYR", "summary": "The paper introduces an approach to augment the feature representations from ViTs, by utilizing the gradients from self-supervised losses. The gradients from the attention output projection of the last transformer block is extracted and projected into the output embedding space using random-projections and PCA to compute FUNGI. Exhaustive experimental evaluation across vision and NLP datasets brings out the efficacy of the approach.", "questions": "- Intuitively, the proposed approach \"encodes first-order learning signals that are specific to the downstream data\" (line 46). If we use task specific loss functions, instead of self-supervised loss, would it further help with the adaptation of the latent representation towards the downstream task?\n- How much is the method dependent on the PCA that is performed to reduce the dimensionality (line 141)? Would there be any other alternative to try?\n- FUNGI has been found effective is augmenting features used for k-NN based classification and scene understanding task. Can FUNGI features be consumed by downstream discriminative classifiers that are trained with softmax?\n- Would be good to clarify whether the reported numbers are the mean of multiple runs.", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ Novel methodology\n+ Simple approach\n+ Exhaustive experimental analysis", "weaknesses": "- The method to generate features from unsupervised gradients seems very empirical. Any theoretical backing for the good results would further improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Can task specific loss functions help with adaptation?\n\nIn this work, we only use self-supervised losses that do not need human labels. If we understand correctly the reviewer's question, task-specific losses such as cross-entropy or a ranking loss are supervised and would require us to have access to the labels. Moreover, if that were the case, we could easily achieve perfect accuracy, as the label information would \"leak\" into the gradient features, as they are calculated on a per-sample basis.\n\nOn the other hand, different losses exhibit different properties, e.g. DINO, being a clustering loss, is particularly helpful in improving the k-means clustering performance, as shown in Table 21 of the uploaded PDF. In particular, using FUNGI features from a CLIP ViT-L/14 backbone, we can improve the cluster/class overlap by up to $+15\\\\%$.\n\n### How much is the method dependent on the PCA? Can other alternatives be used?\n\nThe primary role of PCA in our method is to produce features that have the same storage and compute requirements (for retrieval) as the model embeddings. In Table 11 of the appendix (reported here as Table 22 for convenience) we compare the performance of PCA-transformed features to raw features, and find that, on average across 11 datasets, PCA provides a consistent minor performance boost (up to $+0.3\\\\%$), but it's not essential for good performance of FUNGI features.\n\n|                              | No PCA   | PCA           |\n| ---------------------------- | -------- | ------------- |\n| Embeddings                   | 65.1     | 65.3 ↑0.2     |\n| $\\quad$ + KL                 | 66.0     | 66.3 ↑0.3     |\n| $\\quad$ + KL + DINO          | 67.8     | 68.1 ↑0.3     |\n| $\\quad$ + KL + DINO + SimCLR | **69.8** | **70.1 ↑0.3** |\n\n_**Table 22. PCA slightly improves performance, but it's not essential:** performance of FUNGI features and embeddings from a DeIT ViT-B/16 backbone transformed (or not) using PCA, averaged over 11 vision datasets._ \n\nAs for the strategy used for dimensionality reduction, in Table 23 we evaluate the performance of using no reduction, PCA and random projections with different initializations, and find PCA to be the best across the board.\n\n|                      | No Reduction    | PCA                 | Rand Proj (Binary) | Rand Proj (Gaussian) | Rand Proj (Sparse) |\n| :------------------- | :-------------- | :------------------ | :----------------- | :------------------- | :----------------- |\n| Embeddings           | 57.2 $\\pm$ 0.0 | **61.6 $\\pm$ 0.0** | 55.5 $\\pm$ 0.4    | 55.8 $\\pm$ 1.0     | 56.0 $\\pm$ 0.5   |\n| + KL                 | 59.4 $\\pm$ 0.0 | **64.0 $\\pm$ 0.0**  | 59.0 $\\pm$ 0.8   | 58.2 $\\pm$ 0.8      | 58.7 $\\pm$ 0.3    |\n| + KL + DINO          | 64.3 $\\pm$ 0.0 | **68.3 $\\pm$ 0.0** | 62.9 $\\pm$ 0.4   | 62.2 $\\pm$ 0.3      | 61.9 $\\pm$ 0.5   |\n| + KL + DINO + SimCLR | 69.2 $\\pm$ 0.0 | **70.9 $\\pm$ 0.0** | 67.7 $\\pm$ 0.5    | 67.0 $\\pm$ 0.4     | 66.9 $\\pm$ 0.7   |\n\n_**Table 23. PCA is the best dimensionality reduction method:** mean per-class accuracy on Flowers102 of embeddings and FUNGI features from a DeIT ViT-B/16 backbone transformed with different dimensionality reduction methods. We report the mean performance and one standard deviation across three seeds._\n\n### Can FUNGI features be consumed by downstream discriminative classifiers that are trained with softmax?\n\nYes, we conducted a thorough evaluation of FUNGI features in logistic regression. See the global response for details.\n\n### Are the reported numbers the mean of multiple runs?\n\nThe numbers reported in the paper refer to a single run, except for Table 16 in the appendix, where, for 8 datasets and a DeIT ViT-B/16 backbone, we report the mean accuracy over three runs and one standard deviation, which is generally low, falling below $0.3$ in most cases, and being $0.6$ at most, indicating that our method provides consistent results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tnZvpkNOlX", "reviewer": "Reviewer_8XpZ", "summary": "- The draft introduces a feature enhancement technique called FUNGI (Features from Unsupervised GradIents) for vision transformers. - \n- The core idea is to leverage the un/self-supervised loss gradients at an arbitrary hidden layer within a vision backbone (the default option being the attention output projection of the last transformer block) to enhance the feature embeddings\n- These gradients, along with the pre-trained embeddings, are shown to achieve higher accuracy in kNN classification across various datasets and transformer backbones.\n- Comprehensive experimentation evaluated this technique in three activities (image classification, in-context scene understanding, and text classification). Consistent gains in accuracy are reported across model backbones, tasks/datasets, learning scenarios, etc.", "questions": "1. It would be interesting to understand (if not already) the role of the pre-trained embedding. In other words, can a half-trained or untrained (random) embedding also extract discriminative information from the self-supervised objectives similar to the pre-trained one? Authors may clarify this.\n\n2. Authors may clarify the second weakness mentioned in the above section.", "limitations": "The authors addressed the limitations and potential societal impact of the proposed work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The core contribution is to show that gradients w.r.t. self-supervised objectives have complementary information to the pre-trained model embeddings. This information can be combined to achieve better performance. In my opinion, this can be claimed to be a novel representation. \n\nIt is well-known that pretraining on self-supervised objectives results in moderately powerful features. Contrary to this notion, the draft presents a method to enrich the trained embeddings with more information from multiple such self-supervision objectives. However, the clever thing the draft does is to avoid costly fine-tuning by concatenating the (one-step) gradients for the learned embeddings toward the self-supervision tasks.\n\n2. The authors have introduced a \"generic\" technique to enhance the performance of the kNN classifier built on top of a transformer backbone.\n3. The experimental evaluation of the proposed technique is sound, with several backbones, various unsupervised objectives, different tasks/datasets, ablations, initial verifications (Figures 2 & 3), etc.", "weaknesses": "1. Although complementary information is extracted as gradients, it is computationally demanding - it requires one (partial) backpropagation for each self-supervision objective, total backpropagation operations are linear with respect to the number of objectives, the added linear projection head (h), and finally, the dimensionality reduction to match the dimension of the embedding. One may opine that the additional cost is not worth performance gains (poor returns). Overall, the gains may not be significant to be deployed in practice.\n\n2. The draft provides no sound reasoning regarding the performance enhancement resulting from the self-supervised gradients. Section 3 (figures 2 & 3) empirically proves that these gradients complement the embeddings but doesn't discuss how/why.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Computational cost\n\nOur method does indeed introduce a computational overhead, but we believe that in a retrieval setting, where the embeddings for the database are computed once and only query samples are encoded on the fly, the performance improvement may be very well worth the added computational cost.\n\nIn particular, on retrieval on the revisited Paris landmarks dataset [B], our KL loss improves the mean average precision (mAP) of a CLIP ViT-B/16 backbone by $+10.1\\\\%$ and $+12.4\\\\%$ on the medium and hard splits respectively, at the cost of a ${\\sim}28\\\\%$ reduction in throughput, including the random projection. The results are also shown in Table 24.\n\n|          | Medium         | Hard           |\n| -------- | -------------- | -------------- |\n| Features | 64.6           | 40.4           |\n| $\\quad$ + KL     | **74.7 ↑10.1** | **52.8 ↑12.4** |\n\n_**Table 24. FUNGI improves retrieval:** retrieval mAP on the Paris landmarks dataset of embeddings and FUNGI features built with KL gradients using a CLIP ViT-B/16 backbone._\n\nBeyond that, we note that the main scope of this work was to demonstrate that gradient-enhanced features can improve performance across several tasks, modalities, and backbones. We acknowledge that our method, particularly with the DINO and SimCLR losses, introduces a computational overhead due to the requirement of multiple views for each individual sample. However, we believe that it is possible to make the computation of these gradient features significantly more efficient, which we leave for future work.\n\n### Why are gradients and embeddings complementary?\n\nIn the global response we built some further intuition on why SSL-gradient features can improve the pretrained embeddings. Moreover, the second row of Figure 2 in the main paper shows that gradient alone can already provide a significant performance improvement, thus it's not strictly necessary to combine them with the embeddings.\n\nOn the other hand, gradient features may be \"brittle\", as gradients are estimated using a single data point and depend on the local curvature of the loss landscape. Thus, we hypothesize that combining gradients and embeddings may address this issue and provide more stable yet discriminative features, which also result in better performance overall.\n\n### What is the role of pre-trained embeddings? Can random or half-trained embeddings be used to extract discriminative information?\n\nThank you for this interesting idea and question, that can bring new insights into the utility of gradient features. Indeed, it is possible to extract discriminative information from both a half-trained and random embeddings. For the former, as we do not have access to a half-trained backbone, we consider a Masked Autoencoder ViT-B/16, which is known to have bad performance in retrieval and requires (almost) full-fine tuning for peak performance, while for the latter we consider a randomly initialized ViT-B/16.\n\nIn Table 25 we show the performance of embeddings and FUNGI features for these backbones, averaged over 11 datasets. The results show that it's indeed possible to extract discriminative information from such embeddings, in particular we improve over a random embedding by $6.25\\times$ and by $1.89\\times$ on MAE embeddings.\n\n|                   | MAE ViT-B/16       | Random ViT-B/16    |\n| ----------------- | -------------- | -------------- |\n| Embeddings        | 24.0           | 2.5            |\n| $\\quad$ + KL       | 44.4 ↑20.4     | 12.8 ↑10.3     |\n| $\\quad$ + KL + DINO    | **45.4 ↑21.4** | 14.0   ↑11.5   |\n| $\\quad$ + KL + DINO + SimCLR | 38.8 ↑14.8     | **15.6 ↑13.1** |\n\n_**Table 25. FUNGI significantly improves retrieval for random or half-trained backbones:** average accuracy, over 11 datasets, in full dataset k-nearest neighbor classification for a MAE and a randomly initialized backbone._\n\n\n**References**\n\n[B] Revisiting Oxford and Paris: Large-scale image retrieval benchmarking. CVPR 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qQTHgkinGa", "reviewer": "Reviewer_EjYW", "summary": "The paper proposes a simple method named FUNGI - Features from Unsupervised Gradients, to improve the representations of Vision Transformers (ViTs). Specifically, FUNGI uses gradients from self-supervised objectives to augment the embeddings from pre-trained models. FUNGI involves three straightforward steps - (i) compute gradients at a specific layer for selected self-supervised objectives (ii) project the gradients to a lower dimension (iii) concatenate these projected features with the model’s embeddings for kNN-based classification. Extensive experiments across various datasets, pre-trained models, and tasks demonstrate the effectiveness of the proposed approach.", "questions": "1. Does FUNGI also work for CNNs and CNN-ViT hybrid architectures? For example, MobileCLIP mentioned above consists of convolutional layers and attention-based layers. Would FUNGI work for CNNs such as CLIP ResNet-50 and CNN-ViT hybrids such as MobileCLIP?\n\n2. Since FUNGI is a generic gradient-based method, does it extend beyond classification tasks? Would FUNGI work for generative models such as LLMs, VLMs such as Llava, and BLIP? The application of FUNGI to any such model would greatly improve the impact of the work.", "limitations": "The authors have adequately addressed the limitations of the work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Presentation**: The paper is presented well and well-structured overall. The use of gradients for feature enhancement in ViTs is motivated well through simple analytical experiments. All the experimental setups have been clearly explained and the results are presented well, covering a wide range of pre-trained models and datasets.\n \n- **Simplicity**: The proposed method is simple and easy to understand and implement. Moreover, the authors have provided the PyTorch pseudocode in the supplementary, which makes it easier to replicate the results of the proposed method.\n\n- **Results**: The results from Fig. 5, Tables 1, 2, 3, and 5 demonstrate the effectiveness of FUNGI across various datasets, pre-trained models, and tasks (visual and text-based). Given that the improvements come with no training, they are noteworthy.", "weaknesses": "- **Missing backbones**: While the authors have presented results on a comprehensive list of backbones, a few recent backbones are missing, which are listed below:\n    - Touvron, Hugo, Matthieu Cord, and Hervé Jégou. \"Deit iii: Revenge of the vit.\" ECCV 2022.\n    - Vasu, Pavan Kumar Anasosalu, et al. \"Mobileclip: Fast image-text models through multi-modal reinforced training.\" CVPR 2024.\n\n- **Scalability**: There is no discussion on how FUNGI scales with larger ViT models. All the analyses are performed with ViT-B and ViT-S. The authors should also demonstrate the scalability of FUNGI with larger ViT models across the various pre-training schemes that they have presented results on for ViT-B and ViT-S.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Missing backbones\n\nWe extend the evaluation of FUNGI to DeiT-III, MobileCLIP and CLIP and MoCov2 ResNet-50 backbones. For MobileCLIP, we extract gradient features from the attention output projection of the last token mixer, and for ResNet-50 models we use the last convolutional layer, i.e. `layer4.2.conv3`. For ResNet-50 models we only evaluate KL and DINO gradients, as SimCLR gradients did not seem particularly promising.\n\nAlthough we find that our method works consistently with ViTs, the results on CNNs or ViT-CNN hybrids are less consistent. While our method improves on MoCov2 ResNet-50, it does not improve on the CLIP ResNet-50 and MobileCLIP-S1. We would like to stress that these are preliminary results, and different data augmentation strategies or a more throughout evaluation of the possible gradient sources (i.e. layers) may yield better results for CNNs or ViT-CNNs. The results, averaged over 11 datasets, are reported in Table 26 and Table 27 for full dataset and few shot setups respectively.\n\n|                      | DeiT III ViT-B/16 | CLIP ResNet50 | MoCov2 ResNet50 | Mobile CLIP S1 |\n| -------------------- | ----------------- | ------------- | --------------- | -------------- |\n| Embeddings           | 64.2              | **65.7**      | 52.7            | **79.8**       |\n| + KL                 | 64.8  ↑0.6        | 63.9   ↓1.8   | 52.6 ↓0.1       | 78.2 ↓1.7      |\n| + KL + DINO          | 67.3  ↑3.1        | 63.7  ↓2.0    | **53.6 ↑0.9**   | 78.1 ↓1.8      |\n| + KL + DINO + SimCLR | **68.2 ↑4.0**     | --            | --              | --             |\n\n_**Table 26. FUNGI works primarily for ViTs:** accuracy in full dataset k-nearest neighbor evaluation averaged over 11 datasets for FUNGI and embeddings._\n\n|                      | DeiT III ViT-B/16 | CLIP ResNet50 | MoCov2 ResNet50 | Mobile CLIP S1 |\n| -------------------- | ----------------- | ------------- | --------------- | -------------- |\n| Embeddings           | 34.9              | **34.7**      | 26.4            | **47.9**       |\n| + KL                 | 36.2  ↑1.3        | 33.4 ↓1.3     | 26.8 ↑0.4       | 46.2 ↓1.7      |\n| + KL + DINO          | 37.2  ↑2.3        | 32.5 ↓2.2     | **27.6 ↑1.2**   | 46.6 ↓1.3      |\n| + KL + DINO + SimCLR | **39.6 ↑4.7**     | --            | --              | --             |\n\n_**Table 27. FUNGI works primarily for ViTs:** accuracy in few shot k-nearest neighbor evaluation averaged over 11 datasets for FUNGI and embeddings._\n\n### Scalability\n\nIn Table 18 of the uploaded PDF we report results for AugReg (labelled as \"AR\"), CLIP and DINOv2 ViT-L models, and a DeiT ViT-H/14 model, for full dataset and few shot setups. The results are averaged over 11 datasets, except for DeiT ViT-H/14, whose results exclude CIFAR 10 and 100, Food-101 and ImageNet-1K due to computational reasons.\n\nThe results show that FUNGI works also for larger backbones, especially in few-shot setups, where it improves by $+4.4\\\\%$, $+4.3\\\\%$ and $+2.1\\\\%$ for CLIP ViT-L/14, DINOv2 ViT-L/14 and DeiT ViT-H/14 respectively.\n\n### Does FUNGI extend beyond classification tasks? Can it be used for generative models such as LLMs, VLMs such as Llava, and BLIP?\n\nWe're thankful for this suggestion, and yes, in the global response we show that FUNGI can be used to improve the performance of linear probing, clustering and that it can be used to retrieve better example for in-context learning with LLMs.\n\nAs for integrating FUNGI in the pipeline of generative models such as BLIP or LLaVA we believe it should be possible, as they both use a frozen vision encoder. But, taking BLIP as an example, the vision features are used by the text decoder via cross-attention, and thus we expect FUNGI features to be out-of-distribution without at least a partial re-training or fine-tuning. We did not have enough time to setup such an experiment during the rebuttal, but we agree that it would be an interesting direction to explore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VsGfdyOjJ2", "reviewer": "Reviewer_98WT", "summary": "This paper presents an unsupervised method to enhance the representations of pretrained models. The authors observe that gradient features from various self-supervised objectives are helpful to enhance k-nearest neighbor (KNN) retrieval. The proposed method, FUNGI, combines the embeddings of a pretrained model with gradients from different self-supervised objectives, followed with dimensionality reduction from PCA. Experiments on Image classification, semantic segmentation and text classification show the effectiveness of proposed algorithm.", "questions": "1. What's the dimension of the gradient feature? What's the time cost of random projection and PCA?\n\n2. What's the advantage of random projection with matrix sampled from {-1, 1}, compared to random Gaussian matrix? \n\n3. For text classification, nearest neightbors are used to select examples for in-context learning [1]. Would FUNGI be helpful in this process?\n\n\n[1] What makes good in-context examples for gpt-3? ACL 2022.", "limitations": "The author have discussed some of the limtations in the paper. For others, please refer to \"Weakness\"", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is well written and eazy to follow, with nice charaterization of background and related works. The algorithm is well motivated. The plots and tables are also well-structured with clear information. The idea to combine the gradient features from multiple objective is simple and effective. The proposed algorithm is evaluated on different pretrained models and different tasks.", "weaknesses": "1. My biggest concern is that the evaluation of proposed method is limited to nearest neighbor retrieval (though I agree that this task is important). As a method of feature engineering, it'd be good to test the performance of (few-shot) linear prob or K-Means clustering.\n\n2. Lack of analysis on random projection. For example, [6] uses random Gaussian matrix to project the gradient features to low-dim and shows that it perseves inner-product with high probability. In FUNGI, the projection matrix is sampled in {-1, 1} by Bernoulli random variables. What properties would it have?\n\n3. Some related works are not covered. There've been a number of works using gradient as features for downstream tasks, for example [1-3]. It'd be good to metion them and discuss the intuition why gradient could be used as features for KNN retrieval. The same also holds for the analysis on self-supervised learning objectives, for example [6].\n\n[1] Gradients as Features for Deep Representation Learning, ICLR 2020.\n\n[2] More than a toy: Random matrix models predict how real-world neural representations generalize. ICML 2022. \n\n[3] Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning. ICML 2022.\n\n[4] A Kernel-Based View of Language Model Fine-Tuning. ICML 2023.\n\n[5] Trak: Attributing model behavior at scale. ICML 2023.\n\n[6] Learning by Reconstruction Produces Uninformative Features For Perception. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Other evaluations (linear probing, k-means, in-context learning)\n\nWe conducted further evaluations of FUNGI features for logistic regression, clustering and text classification via in-context learning. The results are discussed in the global response.\n\n### What's the dimension of the gradient feature? What's the time cost of random projection and PCA?\n \nAssuming a ViT-B/16 backbone, the weight gradients from the last attention output projection have shape $(768, 769)$, including the bias. We project them to the same dimensionality as the embeddings, in this case $768$. The final FUNGI features (before PCA) will then have shape $768 \\times (N + 1)$, where $N$ is the number of gradient features used.\n\nPerforming the random projection on the GPU for a batch of $8$ samples results in an overhead of $0.017$ seconds, or $0.0002$ seconds per sample.\n\nAs for the PCA, fitting it over the entire training set of EuroSAT ($18,000$ samples) requires $5.18$ seconds, while transforming the training set requires $0.17$ seconds, or $0.00001$ s/sample. For larger datasets an incremental version of PCA can be used.\n\nThe reported runtimes were averaged over $1000$ runs ($100$ for the PCA fitting).\n\n### What properties does a binary random projection have? What is the advantage?\n\nTRAK [98WT-5] uses a random projection sampled from $N(0, 1)$ to reduce the dimensionality of gradients, which, according to the Johnson-Lindenstrauss lemma, preserves inner products with high probability, given a large enough projection subspace $k$. The main result of [C] is demonstrating that this lemma holds also for matrices whose entries are sampled from either $\\\\{-1, 1\\\\}$ or $\\\\{-1, 0, 1\\\\}$.\n\nEmpirically we show that this holds true by evaluating the k-NN classification accuracy on Flowers102 of raw and projected KL gradients from a DeIT ViT-S/16 backbone.\n\n| Features            | Dimensionality | Accuracy |\n| ------------------- | -------------- | -------- |\n| Raw Gradients       | 147456         | 53.53    |\n| Projected Gradients | 768            | 53.17    |\n\n_**Table 28. Binary random projections preserve euclidean distances:** k-NN classification using raw and projected gradients from a DeIT ViT-S/16 backbone. Projected gradients perform marginally worse, showing that relative euclidean distances are mostly preserved._\n\nOne advantage of a random matrix with $\\\\{-1, 1\\\\}$ entries is that it can be efficiently stored as a boolean matrix. For example, the projection matrices used for ViT-B models have a shape of $(768 \\times 769, 768)$, equivalent to ${\\sim}1.8$ GB when encoded as `float32` and to $0.45$ GB when encoded as booleans.\n\nWe also evaluate the difference in performance given by random projection matrices using different initializations, shown in Table 29, which displays the mean per-class accuracy on Flowers102 of FUNGI features extracted from a DeIT ViT 16/B backbone with respect to the projection initialization. The results show that a binary matrix has a slight advantage over Gaussian and sparse matrices, and has a lower standard deviation as more sub-features are used, but the performance gap is negligible.\n\n|                   | Binary $\\\\{-1, 1\\\\}$   | Gaussian            | Sparse           |\n| ----------------- | -------------------- | ------------------- | ---------------- |\n| Embeddings        | 56.9                | 56.9               | 56.9            |\n| $\\quad$ + KL       | 59.7 $\\pm$ 0.4     | **60.2 $\\pm$ 0.2** | 60.0 $\\pm$ 0.1 |\n| $\\quad$ + KL + DINO    | **63.5 $\\pm$ 0.1** | 63.2 $\\pm$ 0.5    | 63.1 $\\pm$ 0.2  |\n| $\\quad$ + KL + DINO + SimCLR | **69.2 $\\pm$ 0.6** | 69.0 $\\pm$ 0.7    | 68.9 $\\pm$ 0.9 |\n\n_**Table 29. The random projection initialization has little impact on performance:** comparison of the downstream accuracy of FUNGI features built with gradients projected using random matrices with different initializations. We report the mean and one standard deviation measured across three runs using the Flowers102 dataset and a DeIT ViT-B/16 backbone._\n\nPS: We assumed that for this question you were referencing [98WT-5] (TRAK) rather than [98WT-6] (Learning by Reconstruction..), as the former has the exact sentence mentioned in the review. Please correct us in case.\n\n### Missing related works\n\nWe highly appreciate the suggestion of these works, we discussed them and used some of their results to build an intuition in the global response on why/how gradient features work. We will be include them in the updated version of the paper.\n\nBuilding on that discussion, our work is also related to [98WT-2], which used the empirical neural tangent kernel (eNTK), i.e. the per-sample Jacobian, to benchmark an estimator of generalized risk, and also shows that kernel regression on the eNTK can achieve a performance close to fine-tuning in vision. Their work is extended by [98WT-4], which shows that eNTK can be used for prompt-based fine-tuning of masked language models, if the task exhibits kernel behavior. In comparison to their work, our method does not require the downstream task to show kernel behavior, and we expect their method to be more computationally intensive, as they compute the full model Jacobian for each sample. [98WT-3] takes inspiration from NTK and propose using gradients from task-specific losses as features for representation learning. Given a pre-trained network, they first train a linear classifier in a supervised manner and subsequently learn a linear model using both activation and gradient-based features. FUNGI differs from this work by not requiring any human labeled samples and not running any training as it simply uses the gradients as features for k-NN classification.\n\n**References**\n\n[C] Database-friendly random projections: Johnson-Lindenstrauss with binary coins. Journal of computer and System Sciences (2003).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the constructive feedback. In this general response, we address comments, shared by multiple reviewers, regarding the extension of FUNGI evaluation beyond k-NN classification and providing further insight into why FUNGI improves performance. Then, reviewer-specific responses address individual comments. The uploaded PDF includes tables with additional experimental results requested by the reviewers.

We mark references to papers cited by reviewers as `[reviewer-index]` and ours using letters, e.g. `[A]`.

### Beyond K-NN classification evaluation (gxYR, EjYW, 98WT)

**Linear probing.** We evaluate FUNGI features with logistic regression using the cyanure library [A], for all vision (backbone, dataset) pairs. Each classifier is trained for up to 300 epochs (30 in case of ImageNet) using $L_2$ regularization. For each feature combination we pick the best regularization hyper-parameter between 5 linearly spaced values in the interval $[5 \times 10^{-6}, 5 \times 10^{-4}]$ on the validation set. For datasets without a validation set, we generate one from the training set using an $80/20$ stratified split. The results are shown in the Tables 19 and 20 of the uploaded PDF. FUNGI improves linear classification in nearly all cases, especially with supervised backbones. The only exception is with DINO and DINOv2 ViT-B backbones in the few-shot setting, where FUNGI decreases performance.

**Clustering.** We evaluate FUNGI features from DeiT ViT-B/16, CLIP ViT-L/14 and DINOv2 ViT-L/14 backbones in k-means clustering. For this task, we only use the DINO and KL losses, as the SimCLR gradients did not yield good performance improvements. The clusterings are evaluated by matching clusters and classes via the Hungarian algorithm, and measuring the average (cluster, class) overlap. The results are reported in Table 21 of the uploaded PDF, and show that FUNGI can significantly improve performance, by up to $+10.8\\%$ in the case of DINOv2 ViT-L/14 and by up to $+15.8\\%$ on CLIP ViT-L/14.

**In-Context Learning.** We thank reviewers `EjYW` and `98WT` for suggesting the application of FUNGI to generative models and LLMs. In particular, reviewer `98WT` suggested that FUNGI could be used to enhance the example selection for language in-context learning (ICL). Thus, we ran a small experiment for intent classification using the banking-77 dataset (where a banking-related intent must be classified within $77$ possible classes) using $\texttt{gpt-4o-mini}$ as the LLM backbone.

For each test sample, we retrieve the top-20 most similar training set examples, and append them to the prompt alongisde their label. We then ask the model to predict the label for the test sample. For a fair evaluation, we set the model temperature to $0$.

Using the model embeddings to retrieve the ICL examples results in an accuracy of $88.7\\%$, while using FUNGI features (built with KL and SimCLR gradients) results in a $91.2\\%$ accuracy, an improvement of $+2.5\\%$. 

For reference, we used the following prompt template:

```
You have to annotate banking-related queries with an appropriate intent.
You must choose a single class in the following comma-separated list:

{list of possible labels, comma-separated}

You must reply only with the class name, nothing more. Here's some examples:

{(text, label) pairs from the training set}

The query sample is: {query text}
```

labels are given as strings, e.g. `exchange_rate`.

### Theoretical backing and intuition on gradient contribution (gxYR, 8XpZ)

While we cannot provide a sound proof that theoretically explains why our method works, we can build some intuition using some papers that reviewer `98WT` pointed out, for which we're thankful for.

Similarly to our approach, [98WT-1] utilizes gradients, obtained by calculating the Jacobian for multiple deep layers, alongside the model activations to train a linear classifier with improved performance. They frame their method as a linear approximation around the model parameters, and argue that it approximates fine-tuning. Similarly, one of the core assumption that motivates [98WT-3] is that fine-tuning can be approximated as a Taylor expansion of the form:

$$
F(x,w*) \approx F(x, w) + \sum_{i,j} \frac{\delta F(x, w)}{\delta w_{i,j}} \Delta w_{ij}.
$$

Considering this, our method can be interpreted as a form of update-free fine-tuning, that gives rise to different model abilities depending on the loss being used, e.g. SimCLR, an instance discrimination objective, excels in retrieval, resulting in performance improvements of up to $+11\\%$ in k-NN classification (DeiT ViT-B/16, Flowers102), and DINO, a clustering loss, can improve the matching between k-means clusters and the original classes by up to $+15.8\\%$ (CLIP ViT-L/14, Pets).

On the other hand, pixel reconstruction losses such as masked autoencoding (MAE), which have little alignment with semantics [98WT-6], produce uninformative gradients. For example, using a MAE loss on a MAE ViT-B/16 backbone results in a ${\sim}30\\%$ top-1 accuracy on CIFAR-10, compared to an accuracy of $51.43\\%$ using activations.


**References**

[A] Cyanure: An open-source toolbox for empirical risk minimization for python, c++, and soon more. arXiv preprint 2019.

Meta-review: Overall, the paper is well-received with three positive reviews and one borderline reject. The reviewers commend the paper's clarity, novelty, and effectiveness. While some concerns were raised regarding missing backbones, scalability, and theoretical explanations, the paper's strengths outweigh these weaknesses. The authors' ability to demonstrate the effectiveness of their proposed method across various datasets and pre-trained models is particularly noteworthy. The paper's contribution of combining self-supervised gradients with pre-trained embeddings is novel and promising.
Decision: Accept (poster)