Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation
Authors: Justus Westerhoff
Golzar Atefi
Mario Koddenbrock
Alexei Figueroa
Alexander Löser
Erik Rodner
Felix Alexander Gers
Venue: Accepted by TMLR ()

Abstract: The capacity of foundation models allows for their application to new, unseen tasks. The adaptation to such tasks is called transfer learning. An efficient transfer learning method that circumvents parameter optimization is imprinting. The conceptual differences between studies on imprinting form the basis for our systematic investigation. In this work, we propose the general $\texttt{IMPRINT}$ framework, identifying three main components: generation, normalization, and aggregation. Through the lens of this framework, we conduct an in-depth analysis and a comparison of the existing methods. Our findings reveal the benefits of representing novel data with multiple proxies in the generation step and show the importance of proper normalization. Beyond an extensive analytical grounding, our framework enables us to propose a novel variant of imprinting  which outperforms previous work on transfer learning tasks by $4\%$. This variant determines proxies through clustering motivated by the neural collapse phenomenon -- a connection that we draw for the first time. We publicly release our code at \url{https://github.com/DATEXIS/IMPRINT}.

Erik Rodner Published in Transactions on Machine Learning Research (12/2025)
Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation
Justus Westerhoff
justus. hettsuussettrhfffttthheennnnn.
DATEXIS, Berliner Hochschule für Technāk (BHT), Germany
Golzar Atefi
DATEXIS, Berliner Mochschule für Technik (BHT), Germany
Mario Koddenbrock
Werkstatt, Hochschule für Technik und Wirtschaft Berlin (HTW), Germany
Alexei Figueroa
DATEXIS, Berliner Hochschule für Techniik (BHT), Germany
Alexander Löser
DATEXTS, Berliner Hochschule für Technäk (BHT), Germany
KI Werkstatt, Hochschule für Technik und Wirtschaft Berlin (HTW) 8 Merantia Momentwn, Cermany
Felix A. Gers
DATEXIS, Berliner Hochschule für Technik (BHT), Germany
Reviewed OpenReview: https: openreveeeenr.eveeee. net/ forum? id= duU11BmQ3Y
Abstract
The capacity of foundation models allows for their application to new, unseen tasks. The adaptation such tasks is called transfer learning. An efficient transfer learning method that circunvents parameter optimization is imprinting. The conceptual differences between studies on imprinting form the basis of our systematic investigation. propose the general IMPRINT framework, identifying three main components: generation, normalization, and aggregation. Through the lens this framework, we conduct an in-depth analysis and comparison of the existing methods. Our findings reveal the benefits of representing novel data with multiple proxies in the generation step and show the importance of proper normalization. Beyond an extensive analytical grounding, our framework enables us to propose novel variant of imprinting which out performs previous work transfer learning tasks by 4%. This variant determines proxies through clustering motivated by the neural collapse phenomenon connection that we draw for the first time. We publicly release our code at attps://githut com www.eooooccciieee DATEXXI/IIPPRIITT
In this work, we
1 Introduction
In machine learning applications, training models from scratch is often not viable due to limitations in data and compute. A popular solution is to apply transfer learning (Bengio 2012 Yosinski el al. 2014) based on foundation models (FMs) [oommasani et al.) 2021) that are pre-trained on a large amount of data. To tune a to a novel task, e.g., for classification, a common procedure to freeze the model parameters and replace the output layer with new head. A particularly simple method for implementing such a new head proposed by Qi et al. (2018) and coined imprinting.

---

Published in Transactions on Machine Learning Research (12/2025)
Neural collapse in embeddings
Training
Weight Generator
Imprinted
class
Weights
NORN gr
GEN
NORM goot
class c
Imprinted
Training data from novel task
Weights
Foundation Model FM
Inference
output
NORM
AGG
Test data from novel task
Our IMPRINT Framework
Overview of our IMPRINT framework. The foundation model FM is frozen and shows neural collapse. The weight generator (GEN) uses training data from novel task to consecutively generate one or more weight vectors (proxies) per class 1, I,...,C in T. In inference, the final output for the test data in T is computed by an aggregation (AGG) mechanism. Embeddings and generated weights are normalized according NORMpre and NORMpost, respoctively. During inference, embeddings are normalizod according NDRM
Figure 1:
Imprinting. the original work by Qi et al. 2018), the last-layer weight vector of a novel class is set the normalized average of its scaled embedding vectors, i.e., its class mean. These class means are representatives of the classes, which we denole prozies. In general, we refer to imprinting efficient learning methods without the need for cross-class statistics or gradient-based optimization. A plethora of studies have emerged surveying this technique by adding complexity and adaptability (Gidaris & Komodakis
2018:
Siam et al.)
2019: Passalis et al. 2020 Khan et al. 2021 Li et al.) 2021 Cristovao et al. 2022; Yan et al. 2023 Despite its many adaptations, imprinting lacks a systematic comparison that unifies them. Understanding its variations could unlock greater efficiency across many fields, making the method even more versatile and impactful.
Applications. In particular, while in some practical applications, accuracy prioritized over computational efficiency, the latter does become critical requirement in scenarios where computational resources are severely constrained, e.g., in the chemical and polymer processing industries. Here, battery-powered edge compute is essential and frequent retraining or large-scale optimization is infeasible. Imprinting has proven particularly effective in these contexts of edge-embedded devices (Passalis et al. 2020). For instance, Zhu el al. (2022) implement vision-based robotic force grasping with variable-stiffness gripper that can safely handle both fragile and heavy objects by rapidly adapting to novel categories without retraining (Contimual Learning (CL) setting). Industrial adoption also underscores this trend with Google's Coral Edge TPU including an ImprintingEngine API (Coral), that allows users to add new classes from a few examples without recompiling the model. Very recently, (Belal et al. 2025 apply imprinting to spectrogram embeddings from LMU/EME gait data, achieving efficient classification in low-data Human Activity Recognition (HAR) tasks. Apart from that, Janson et al. (2022) extend imprinting CL, establishing simple bascline with competitive performance compared to more sophisticated statiiaaaaassseeeeaaa state-of-the-art CL algorithms.
Framework. We present IMPRINT, framework that enables comprehensive analysis of existing imprinting techniques. More precisely, generalize prior work by decomposing imprinting into three principal steps (sce fig. [ During generation (GEN) of weights, the method seleets representative data samples and generates one or more weight vectors (proxies) per class. Normalization (NORM) is crucial, as the generated weights need to be balanced. Aggregation (AGG) entails the computation of the final output, e.g., a class label. The computational efficiency imprinting allows us to perform a large number of experiments. Through IMPRINT, we are able to propose novel, rest-performing imprinting strategy using multi-proxy weight imprinting combination with 1,2 normalization, outperforming previously studied methods, depicted in fig.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Neural Collapse. When neural networks are trained to reach near-zero loss, their pemultmmate-laeer embeddings collapse to the class means (Papyan et al. 2020 Zhu et al., 2021]. We investigate this phenomenon as a potential explanation for when and why imprinting works. Our analysis proves that there exists a relationship between a measure of noural collapse and the success of imprinting. Since quantification of this phenomenon possible a post-hoc Tashion over the FM features, we believe that these insights will contribute to further development of imprinting methods, as well as, training regimes of FMs that are more suitable for transfer learning via imprinting.
Contributions. In summary, main results and contributions are:
We deconstruct weight imprinting into the IMPRINT framework composed of generation, normalization, and aggregation, and discuss variations for each of them, while identifying prior work special cases (section 3.1. To the best of our knowledge, we are the first to conduct a comprehensive analysis of imprinting to this scale (section 4
We present a new, outperforming imprinting method utilizing k-means clustcring for weight generation
(section 5.1 and show its benefits in certain low-data regimes (section 5.2)
As far as we are aware, we are the first to identify a relationship between the degree of neural collapse and imprinting success (section 5.3)
2 Related Work
Imprinting and Low-Data Regimes. Weight imprinting is implemented by setting the final layer weights for the novel classes to the scaled average of the embedding vectors of their training samples and was first introduced by Qi et al.] (2018) for the few-shot learning scenario. The authors find that for up to 20 samples, using combination of imprinting and fine-tuning outperforms other saate-of-the-ars methods, including ncarest neighbor algorithms. In contract, in our work we do not limit the number of samples and perform no fine-tuning on the imprinted weights to maintain efficiency.
Imprinting has been applied to object detection (Li et al. 2021 Yan et al. 2023 ) multi-label classification KKaa et a 2021), scmantic segmentation (Siam et al., 2019), and combined with an attention mechanism generale weights for the novel classes in few-shot classification lask (Gidaris & Komodakis 2018).
Hosoda et al. (2024) apply imprinting using quantile normalization to ensure statistical similarity between new and existing weights. We consider this as one normalization scheme in our framework. Zhang et al. [2021) apply imprinting in chest radiography for detection of COVID-19 and find that it yields better results than joint gradient descent training of all classes when only few samples are available. They speculate whether normalization is a constraint in their imprinting model.
Before the era of deep learning, Mensink et al. (2013) analyze the transferability of hand-crafted image features. They use "nearest class multiple centroids" (NCMC) classifier with multiple proxies generated from a k-means clustering algorithm. In combination with metric learning, they compare favorably against the m-nearest neighbor algorithm. Our work, on the other hand, highlights efficient transfer learning provided by foundation models.
Transfer Learning. Using embedding vectors from pre-trained models is a simple and widely used transfer learning approach, established in the seminal works on computer vision (Donahue et al., 2014) and natural language processing (Devlin et al. 2019). Kornblith al.] (2019) show that pre-training performance of a model is highly correlated with the performance of the resulting embedding vectors in downstream tasks. In addition, Huh et al. (2016) provide insights into the required quality of pre-training data. Our work is orthogonal to these studies, since we focus on studying weight generation, normalization, and aggregation techniques applied later on for new lask adaptation.
Continual Learning (CL). Although we investigate transfer learning scenarios, we review the imprinting applications and results from CL. Rebuffi et al. (2017) dynamically select a subset of examples for each class and update internal representations via gradient descent. They use a nearest mean classifier (NMC)
3

---

Published in Transactions on Machine Learning Research (12/2025)
Table & Figure 2: Previously studied imprinting strategies are special cases within IMPRINT. We evaluate 12 different classification tasks Ts derived from MNIST, FashionMNIST,, and CIFAR-10. 10, cach with 10 classes or subsets thereof, and 4 pre-trained models FMs (resnet18, resnet50, vit_b sit_b_16, swin_ b). The proposed configuration ("Ours") derived from IMPRINT outperforms previous work across FMs and Ts by large margin with statistical significance, accuracies vary substantially across tasks and models (as reflected by the large standard deviations (std)), this rank-based aggregation is used for fair comparison. Here, h = 20 is used, highlighting the gain of using multiple proxies per class. For reference, the gray row reports an oracle method that uses cross-class feature statistics to generale weights (see appendix A..5.. is not an imprinting method and therefore not directly comparable to the imprinting-basse based approaches above. Nonetheless, the results indicate that our method substantially narrows the gap between single-proxy mean imprinting and this oracle basclinc.
confirmed by the critical difference (CD) diagram below. Since absolute
Work
NORM pm
GEN
NORMpost
NORM
AGG
Avg. acc. % +
Qi et al. 2018
L2
mean
L2
L2 max
86.79 #7.83
Hosoda et al.
2024
none
mean
quantile
none max
82.90 #12.87
Janson et al.
[2022)
none
mean
none
none 1-nn
86.64 #7.88
Ours
L2
k- x-means
L2
max
91.06 16221
Oracle
none
Least-squares
none
none
max
94.54 #5.00
Average rank
Hosoda et al. 3.27
Ours
Jason et al.
2:00
2.67
Qi et al.
with respect to the saved examples. Janson et al. (2022) use an NMC classifier as well and achiove good performance on CL benchmarks without any minittnnig of the embeddings. However, they do not investigate the effect of normalization and using multiple proxies.
Findings of Prabhu et al. (0023) show that simple, approximate m-nearest neighbor classifier outperforms existing methods in Online CL setting when all data can be stored. In work, however, we compare imprinting all data to a limited number of more representative proxies striving for efficiency.
Neural Collapse (NC). The phenomenon of NC identified by Papyan et al. (0200) and refers to the convergence of the last-layer weight. vectors to class means. It was shown that, regardless of the loss function, optimizer, batch-normallzation. or regularization, NC will eventually occur (provided the training data has a balanced distribution) (Zhu et al., 2021 Han et al. 2022 Kothapalli 2023). Nevertheless, complete neural collapse is practically unrealistic (Tirer et al. 2023). In transfer learning, Galanti el al. (2022) show that NC occurs on new samples and classes from the same distribution as the pre-training dataset, highlighting the usability of foundational models in such scenarios. In our work, we expand the survey on NC by experimenting with diitt--iistti classes belonging to different datasets and linking their degree of collapse to the success of certain imprinting strategies.
3 Methods
3.1 IMPRINT
In order to find out how to best and efficiently set the classifier weights foundational model FM new, previously unscen tasks T, we create the IMPRINT framework (see fig. 1 that generalizes previous work, specifically all the methods that work without access to cross-class statistics and gradient- based training. Thereby, we unify all the existing imprinting strategies described in section
4

---

Published in Transactions on Machine Learning Research (12/2025)
Overview. We analyze the effect of weight generation (GEN), normalizations (NORM (NORMpre: post, NORM and aggregation (AGG). The IMPRINT framework depicted in fig. con- sists of three main building blocks: foundation model FM, a weight generator GEN, and extendable classifier weights that are imprinted. The FM remains frozen throughout the experiments. It receives data from T as inputs and produces embedding vectors. The training process generates weight. vectors for each of the C classes T. Hereby, embeddings from the FM are normalized before the generation (GEN) step according NORM pre. The generated weight vectors per class are referred
prozies, protolypes, or representatives [Movshvviz--teer Attias et al. 2017; Snell et al., 2017 Yang et al. 2018 Yu al. 2020 These proxies are normalized according to NORM As in the work by Qi et al. (2018), we do not use bias values. To classify the test data in during inference, it is first embedded by the FM, normalized according to NORM and finally aggregated by AGG, resulting in a predicted class label.
Generalization of Previous Methods. Previously proposed imprinting methods can be defined special cases of our framework. We design IMPRINT such that every method can be defined by a single combination of GEN, NORM, and AGG, of which we inspect all possible combinations.
Weight Generation (GEN). The purpose of GEN is to determine how the embeddings of the training data in T used to form the weights. In contrast to Qi et al. (2018) which only incorporates one proxy per class (the mean), we add flexibility by allowing each class to have multiple proxies as in Mensink et al. (2013) and enable non-linear classification. We denote the number of proxies per class as k, ranging between and the number of samples, and investigale the following operations conducted per class to generate those:
all: All embeddings (denoted = all).
k k-random: k random embeddings.
mean: The mean of all embeddings.
m-mea:s: k-means cluster centers using KMeanS from sklearn (Pedregosa et al. 2011 =1 is the same as mean.
k- medoiiss: k-medoids cluster centers using KMedoids from sklearn.
k-cov-max
(covariance-maximiztioov)
Top k embeddings by covariance (in code: proxies suw (torch. cov (emboddings), dim-0), descending- (:k]].
embeddings [torch. argsort (torch. sum
k-fps (farthest-poirs sampling): Iteratively selecting & embeddings, such that it maximizes the distance from already selected ones (starting with a random sample).
We choose this diverse list of methods to cover wide range of approaches, ranging from heuristics (e.g., k-fps) to more complex algorithms (e.g., k-means). Note that only mean and k-means generate proxies beyond existing samples by producing synthetic cluster centers in embedding space. In comparison, k-medoids must choose actual samples as cluster centers (analogous to a median). None of these methods use cross-class statistics or gradient-based optimization. We also note an analogy to associative memory models, interpreting imprinting as memory update process following a covariance rule, detailed in appendix A.3
Normalization (NORM). The main reason for applying normalization is to allow each embedding and weight vector to contribute equally on the same scale. The modes we allow are no normalization (none), L normalization (L2), and quantile normalization (quantile).
L2 normalization can be applied to embeddings before GEN via NORM the generated weights via NORM and to embeddings in inference via NORM In any case, the vector is Le-normalized by dividing it by its Euclidean length || 112-
quantile normalization (Amaratunga & Cabrera 2001 Bolstad et al. 2003) can only be applied to generated weights. This non-linear operation distributes weights equally. Recall that more than one class contained in (c > 1), GEN is performed for each class consccutively, and the reference distribution changes accordingly. In particular, for the first class there is no reference distribution to map to. This is different from Hosoda et al. (2024), where weights are matched to the distribution of the original classifier weights of the FM.
5

---

Published in Transactions on Machine Learning Research (12/2025)
Since we do not consider the classes used for pre-training the FM and especially do not assume access to their last-layer weights, this is not possible in our scenario.
A Aggregation (AGG). There are various ways to use the generated weights (proxies) per class during inference, especially when & > 1. We focus on two different modes, max and m-nearest neighbor (m-nn). The former, max, computes the inner product of the input embedding and the imprinted weights and outputs the class label with the maximum activation. The latter, m-nn, uses the class weights as keys and the embeddings values, and chooses the final winning output class via the m-nearest neighbor algorithm. The m-nn voting is weighted by the inverse of the Euclidean distances to their nearest neighbor, turning it into weighted majority voting.
We also experimented with alternative distance functions (Cosine, Manhattan) and uniform weighting, but found differences well within statistical noisc. Using the Mahalanobis distance proved computationally prohibitive. More elaborate voting mechanisms, such as learned lop-k attention or entropy-based filtering, are conceivable, but extend well beyond the minimalist imprinting paradigm and therefore left for future work.
Note that max is the same 1-nn in the case of L2 for NORMposts since for any fixed embedding vector v and variable proxy w, the argmin of ||v w||2 ||v] - 2(v, w) + ||w/|", calculated by 1-nn, is the same as the argmax of the inner product (v, w) calculated in max.
3.2 Quantifying Neural Collapse
Neural collapse (NC) Paayyan et al. 2020) characterizes the state of the features produced by classification neural network after training to near zero training loss. Namely, the learned embeddings of cach class converge, i.e., collopse, to their class means. These globally centered class means and classifier weights form a simplex equiangular tight frame (ETF)- a collection of equal length and maximally equiangular vectors, that maximize the between-class variability. This results in an optimal linearly separable state for classification. In fig. 3 (left), we illustrate the collapse of a FM on its pre-training data. The newly arrived data from a different dataset is distributed more unevenly across the embedding space (right).
Pre to--training data containing classes o7,
New task T containing classes 01 02
Collapsed embeddings
Non-collapsed new embeddings
Figure 3: Left: The emboddings of the pro-training data, after being used to train the foundation model FM, show neural collapse, each class (oI,...,04) 04) is evenly separated in space and accumulates around their respective class means. Right: For novel task with classes C1, C2 (pink and brown) scatter around the collapsed pre-trained classes (gray).
Two important characteristics of NC are variability collapse, i.e., the within-class variability of the penultinate-layer embeddings collapses to zero, and convergence to pcarest-meancllasficcaee locus on variability collapse (NC,) as in Zhu et al. (2021). Given a foundation model FM and a finite dataset containing C' classes with (for simplicity) N samples per class, we use its I-dimensional embeddings 1 D ΣN he, class means (ssse, ISiEN given by FM to define the global feature mean hG = CN
We
h. = 1/4 EN bag the within-class covariance matrix Σw = 1
CN E En (h - T.)( -he) and
the between-class covariance matrix ΣB == à IC (h. ha) (T. ha) to finally compute
NC, tracel Ew Et)
(1)
6

---

Published in Transactions on Machine Learning Research (12/2025)
where symmbllizs the pseudo-inverse.
Based on eq. [ an NC1 score closer to zero sigmifies a higher collapse. In contrast, an increase in multi- modality of data leads to a higher WC1 score (as analyzed in fig. Iaab Note that this measurement is not independent of the embedding dimension 1 and the number of classes C. According to NC, imprinting the mean, as originally done by Qi et al. (2018), is best when NC1 small. We claim that when the data not. fully collapsed (as is often the case in practice), the scale of NC₁ could guide the proxy generation method, e.g.: having multiple proxies & > per class. We investigate this in section 5.3
3.3 Significance Testing with Critical Difference Diagrams
Our experiments compare large numbers of imprinting configurations across tasks with substantially different achievable accuracy levels. To obtain scale-independent statistical comparisons, base our analysis on ranking daiie)aagremeenss rather than raw accuracy, Non-parametrie rank-based tests are well suited for this setting. Following Demšar (2006). we use the Friedman test and Wilcoxon signed-rank tests because they do not assume normality or homoscedastiiitty, and can be applied uniformly to any evaluation metric. Empirical evidence in Demšar (2006) further suggests that these tests have good power in classifier comparison sconarios.
Concretely, for each evaluation instance (i.e., foundation model FM and task Т) we compute the classification accuracy of all configurations and assign ranks (rank 1 = highest accuracy). This yields a matrix of ranks with one row per configuration and one column per ovaluation instance. On this matrix we first apply the Friedman Lest to assess whether there are any overall differences between configurations. If the Friedman test rejects the global null hypothesis at significance level = 0.05, we proceed with a post-hoc, pairwise analysis. Specifically, for every pair of configurations we run a two-sided Wilcoxon signed-rank test across evaluation instances. The resulting p-values are corrected for multiple comparisons using the Holm step-down procedure [Holm 1979).
Critical difference (CD) diagrams summarize both the average performance and the significance structure. The horizontal axis shows the average rank of each configuration across all evaluation instances, with lower ranks indicating better performance. Two configurations are connected by a thick horizontal line if their Wilcoxon Holm comparison does not show a significant difference.
4 Experimental Setup
Foundation Models FM. We use reanet18, resnet50 (H et al. 2021], and swin_b b (Liu et al., 2021) as FMs, two CNN-based and two Transformer-basse architectures. All four models are pre-trained on ImageNet-IKK (ILSVRC 2012) (Dngg et al. 2009 To generate the embeddings, we use PyTorch's torchwision models.
2016). vit_b_16 (Dosovitskiy et al.)
Tasks T. We analyze multi-class classification scenarios without separating base and new classes, instead focusing on all classes within novel T at the same time. To investigate the effect of the number of samples given, we look at n-shot (n N) scenarios. For that, we randomly pre-sample the training data of T to n samples per class transitioning into the low-data regime.
To find out the best imprinting strategy within our IMPRINT framework, we focus on tasks T created from the datasets MNIST (Deng, 2012), PashionMMIST (Xiao et al. 2017), and CIFA R-10 (Krizhessky et al., 2009), each containing 10 classes. We mainly focus on the three T containing all ten classes. Furthermore, we look at smaller tasks only containing classes {0, 1, 2}, and the two tasks containing classes 5,7,, resp. {0, 2, 4, 6, 8). This random selection of 3 8-44 4 = 12 tasks adds variation to evaluations.
Neural Collapse. In the analysis of neural collapse (NC), we also look at the FMs' pre-training data (emageNel). As its test set is not available, we use its validation set in NC,, computations. Furthermore, for ImageNet, we relabel data by combining multiple classes into one label to simulate multi-modal class distributions for an in-depth NC analysis. These tasks are called "d 1", d = =,.... .....,10, 10, each containing different labels. More precisely, we take 100 random classes from ImageNet and sequentially map the first d to label the second d to label 2, etc., until we reach 10 distinct labels. See fig. 4 for simplified illustration.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Moreover, construct new out-of-distribution non-collapsed dataset by merging all classes from four digit datasets: MNIST, MNIST-M (Ganin et al. 2016), SVHN (Netzer et al. 2011), and USPS (Hull 1994). The resulting dataset, which we refer to as CombiDigits, exhibits reduced collapse due to the greater distributional diversity within cach class. To ensure scale invariance in covariance- based NC mcasurements, all emboddings are I." no-normalizedd before computing NC1.
Scale. In total, we ran approximately 500 000 experiments, varying imprinting components, foundation models, tasks, and seeds. This feasible with minimal effort as imprinting a highly efficient method: core steps such as weight generation (GEN), normalization (NORM), and aggregation (AGG) are linear in dataset size number of proxies and parallelize efficiently within each step. Details experimental infrastructure and parallelization are provided in appendix A.6
class labels
1
2 in
0
in
Figure Combining multiple classes into one create tasks with multi- class distributions. Simplified example for "d in 1", 1,2,3, with only six (instead of 100) original classes.
Evaluation. Unless stated otherwise, we report the median test accuracy over three different seeds. In sections 5.1 and 5.2 we evaluate the imprinting performance by varying the FM (4) and T (12) and report average accuracy and standard deviation (std) across these. Due to the heterogeneity of models and tasks, large std are expected. Therefore, overall method comparisons are based on ranks rather than absolute accuracies. Methods are ranked by their median accuracy, yielding 4 12 48 potentially different ranks, We report the average rank and assess statistical significance of ranking (dis-)aggeen) (dis-)agrrements using critical difference (CD) diagrams as explained in section 3.3 Ismail Fawaz et al. )019).
The code used to generate these diagrams is inspired by
In experiments with neural collapse (section 5.3 we investigate the four FMs on 100 random ImageNet tasks with remapped labels (as explained above) and the four tasks containing all MNIST, Fashion.MISTT, CIFAR-10. and CombiDigils, respectively.
5 Results
Our main experimental insights are: 1. Our IMPRINT framework generalizes previous methods, and we find a new superior imprinting strategy (section 5.1 2. We show that our novel strategy is even beneficial in low-data regimes with as little as 50 samples per class (section 5.2p. 3. We identify a correlation between imprinting success utilizing multiple proxies and a measure of neural collapse (section 5.3).
5.1 Best Imprinting Strategy
We provide comparison between existing memory-constrainee methods used for imprinting on foundation models in fig. 2 namely, the work by Qi et al. (2018); Hosoda et al. (2024); Janson et al. (2022). as well as a novel, berf-rmrn configuration ("Ours") that results from IMPRINT. Focusing on = 20, we find that our method, consisting of k-means weight generation, L2 normalizations, and max aggregation, outperforms all previously studied approaches by margin of 4% on average with statistical significance. For reference, we additionally report oracle baseline that cross-class feature statistics to generate its
8

---

Published in Transactions on Machine Learning Research (12/2025)
weights, representing upper bound that is not constrained by imprinting. The results indicate that new imprinting method significantly narrows the gap between single-proxy mean imprinting and this oracle baseline.
The impact of using the unconstrained m-nn aggregation on all data is investigated at the end of this section. Next, we analyze each of the components of IMPRINT separately.
Table & Figure 5: Benchmarking GEN mechanism for I across FMs and Ts. Best NORM combination for each row used implicitly. AGG is fixed to max. CD diagram proves that k-means weight generation is significantly better than all other methods.
GEN
k
Avg. % std
Average rank
meann
20
91.04 +6.26
k-medoids
20
87.01 18.23
mean
86.84 17.80
k-random (k 20)
k-fps (k= 20) J.M
4.2
k-means (K 20) 2.45 k-modoids 20) mean
k-cov-max
20
83.98
+9.56
k-cov-max (k 20)
4.00
k-random
20
82.14 1:00.1
k-fps
20
65.56 112.32
Weight Generation (GEN). To assess the impact of GEN, first focus the max aggregation and do not fix NORM, but simply show the run with the best NORM combination, if not otherwise specified. The m-nn aggregation and different values for NORM are analyzed later in this section.
Initially, we limit the number of generated proxies to k < 20, Results in fig. show how k-means, using as many proxics as possible (in this case; 20) outperforms by 4% on average accuracy compared to all the other GEN methods. The CD diagram illustrates its statistical significance in ranking. Furthermore, while k-medoids with 20 proxies (which necessarily have to be among the given samples, computationally expensive, it is par with mean, and covariance maximization, furthest-point sampling and random selection show even weaker performances. We find similar results for i S 5, where k-means oulperforms the other hods as well (see fig. A.1
section 3.1 is
80
60
k-means
acc
k-medoids
k-cov-max
50
k-random
k-fps
40
80
number of proxies
Figure 6: Benchmarking different. GEN methods with resnet18 all of CIFAR-10 shows superiority of k-means proxies. All combinations employ L2 for all NORM and max AGG.
As the number of proxies l: increases, k-means continues to be the best GEN method. An example for resnet18 and C-FAR-10 can be found in fig. 6 All methods converge towards the point of imprinting (saving) all data (k- = all), even surpassing it in the case of k-means. Due to its superior performance, we mainly focus on k-means the remainder of the analysis.
Learned Weights. Employing gradient- praseent-based methods such as setting weights via the analytical least- squares initialization proposed by Harun & Kanan (2025). which uses data and labels from all classes jointly

---

Published in Transactions on Machine Learning Research (12/2025)
can be seen upper bound ("Oracle" in fig. 2 for the unsupervised weight imprinting approaches discussed here. We extend this method by combining it with k-means, forming k-least-squares and observe in appendix A.5 that using multiple proxies per class instead of a single weight vector can still improve performance, particularly on datasets with high NC1.
Normalization (NORM). We compare all the different NORM methods, focusing on k k-means as GEN. For k 1 and varying NORM (while taking best values for NORMpre and NORM implicitly), fig. shows that L2 is the best choice for weight normalization. quantile and none normalization both perform worse.
Table & Figure 7: Benchmarking NORM mechanism across FMs and Ts. The best NORM and NORM combinations for each The CD diagram shows the statistical significance of L2 as the best weight normalization NORM post.
used implicitly. GEN is fixed to mean (that is, k = 1) and AGG is fixed to
NORM
Avg. acc. % = std
Average rank
3
2
L2
86.84 #7.80
quantile
82.90 12287
none 2,87 quantile
123
L2
none
83.26 #9.18
Table 4: Benchmarking NORM and NORMay mechanisms across FMs and Ts. NORMpost is fixed to L2, GEN to mean, and AGG max. No stalistically significant. differences were found.
NORMpre
NORM
Avg. acc. % + std
none
L2
86.84 +7.80
none
none L2
86.84 1.780
L2
86.79 47.83
Keeping L2 for NORMpost fixed, we find no statistical differences between the different combinations of NORM and NORM (sce 4) For larger values of k the differences among NORM become However, the performance of NORMpre and NORM remains statistically indifferent for L2 weight normalization (see fig. A.2 for all combinations at once with à = 1, and fig. A.3 for k = 20)-
pronouneed.
We restrict all the subsequent experiments to using L2 across all NORM following Qi ct al. 2018 This combination normalizations chosen to specifically model cosine similarity within max aggregation.
Table 5 & Figure 8: Benchmarking AGG mechanism across FMs and Ts. GEN is fixed to all (k all), that is, imprinting (saving) all data to weights. L2 normalization is used for all NORM. The CD diagram shows statistical significance of 3-nn, 5-nn, and 20-nn over max aggregation.
AGG
Avg. acc. % std
Average rank
5-nn
93.74 14.97
3-nn
93.50 +5.22
20-nn
93.56 +4.93
50-nn
5-nn
1-nn
2.31
3-nn
1-nn
92.81 +5.69
max 398
2:48
20-nn
max
92.81 +3.69
50-nn
92.91 45.25
Aggregation (AGG). In addition to max, we study the effect of using m-nearest neighbor (m-nn) as an aggregation method. Recall that max is a special case of m-nn when m = (as NORM is set. to L2). We investigate different values for m E (1,3, 5, 20, 50).
When all data is imprinted, fig. shows that using m-nn aggregation for E 73,5,20) slightly better than With 20 and k-means GEN, (=1-nn) aggregation becomes the top performing combination
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table & Figure 9: Benchmarking AGG mechanism across FMs and Ts. GEN is fixed to k-means with 20. normalization is used for all NORM. The CD diagram shows that max is the best-performing aggregation method.
AGG
Avg. acc. % std
Average rank
1-nn
91.06 16.21
5
2
91.06 16.21
3-nn
90.59 16.16
20-nn 2.88
1.0 max
5-nn
90.13 12 16.21
5-nn
2.80
1-nn
3-nn
20-nn
87.05 47.46
(see fig. 9
Furthermore, the reduction of proxies (from all ( 6000 per class) to k 20) leads decrease in accuracy of less than 3%.
resnet18, MNIST
resnet18, FashionMNIST
resnet18, CIFAR-10
90
80
80
60
800
70
60
50
40
40
vit_b 1r_b_166. 16, MNIST
b b_16, FashionMNIST
vit_b_16 1_ CIFAR-10
90
80
80
800
70
60
80
50
50
10
10
10'
10
10'
10"
102
#samples n
k=n k= 1
k=
20
Figure 10: k-means with different values for k in n-shot scenarios. Shaded areas indicate 95% confidence intervals (CIs) over five different seeds. Other variables are fixed to previously described best method of using L2 normalizations and max as AGG. Note that only data for the meaningful case of & S n is shown. For MNIST and FashionMNISTT. mean ceases to be the best strategy already at only roughly 50 samples.
5.2 low-Data Regime
We analyze the n-shot scenario with our method (k-means GEN, L2 NORM, and max as AGG). Furthermore, we focus on the large tasks T containing all ten classes of MNIST, FashionMNIST. resp. CIFAR-10 at once. In this scenario, due to sampling only a few examples n, we average over five (instead of three) different seeds.
From the results in fig. I0 we find that as the number samples n increases, k-means starts to outperform mean imprinting. The use of more proxies further amplifics this performance gain. The shift occurs becomes prominently better around 200 samples per class (see fig. A.4 for display of all FMs focused 10 & n S 400).
roughly 50 samples per class for MNIST and FashionMNIST while for CIFAR-10,
11

---

Published in Transactions on Machine Learning Research (12/2025)
5.3 Neural Collapse and Number of Proxies
Figure 11 depicts the neural collapse measurement NC1 (see eq. 1 for the 100 random ImageNNl tasks with remapped labels as explained in section | as well as the four tasks containing all of MNIST, FashionMNISTI, CIFAR-10, resp. CombiDigits. It can be inferred that ImageNet has a close-to-zcro NC1 score, which increases linearly when adding more classes to each label (i.e., increasing multi-modality)). As for the other datasets, CTFA R-10 generally more collapsed according to its low value of NCC, which even falls below for the Transformer-basees FMs. We hypothesize that this due to the similarity of its categories to those appearing in ImageNet. By design, the synthetic CombiDigits dataset, introduced in section 4 has very high NC1- Apart from that, NC1 of the ImageNel data for the Transformer-basee architect are much lower and therefore they are more collapsed compared to the CNN-based FMs. These architectural differences are further investigated in appendix A.4
ImageNet ("d in 1")
resnet18
resnet50
10
100
resnat18 vit b_16 16
resnet50
swin b
58
30
vil b.bbu.5 16
swin
100
85
10'
-10"
10
15
MNIST FashionMNIST CIFAR-10 CombiDigits
proxies
1in
10
in (ImageNet)
(a) A clear linear relationship between neural collapse measure NC, and d can be inferred for FMs, i.e., increased multi-modality implies less collapse. The NVC, of other datasets (CombiDigits in particular) is much higher across all FMs, while only vit b__ 16 and swin get an NC, of less than one on CIFAR-1O.
MNIST --FashonMMIST
CIFAR-10 CombiDigits
(b) Accuracy over number of proxies A used in k-means together with
NORM max as aggregation. peaks in accuracy at & = d can be inferred. This confirms the connection between the effect of using multiple proxies and the collapse of the data.
subplots,
Figure 11: Averaged NC1 resp. accuracy of ten random ImageNel label remappings (*d in 1") for every 1, 1,..,, 10. Shaded areas indicate 95% CIs over three different seeds. Values for the tasks containing all of MNIST, PasHionMNISTT, CIFAR-10, resp. CombiDigits at once are shown in dotted styles.
For the same data, fig. 11b depicts accuracy over varying number of proxies i inferred from k-means. A prominent peak at = can be inferred for every FM and reflects that d class proxies lead to the best result for d-modal class distributions. Furthermore, increasing for the ImageNet sets has a much larger effect on the CNNd FMs. We argue that this bocause of their higher values of NC,, indicating less neural collapse. In appendix A.4 we analyze architectural differences and training setups to explain these variations. The fact that CTPA R- 10 has the lowest NC₁ scores (see fig. 11a is reflected by flat green curves over k.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Figure 12 renders this connection more precise: For datasets with large NC1, i.c. where intra-class variability
surpasses inter-class variability, using more than single proxy per class (k: >> 1) yields a clear performance gain over mean imprinting (k = 1).
resnet18
resnet50
17.5
vit vit_b_16
15.0
acc Increase (k vs
10.0
R0,0.36
7.5
2.5
ImageNet
0.0
Non-ImageNet
10
10a
10¹
NC
Figure 12: Accuracy (%) gain when using instead of proxies per class with k-means, plotted against the neural collapse measure NC₁, (log-scaled axis). Colors indicate different FMs. Outlined markers correspond to non-ImageNeet datasets (MNIST, FashionMNIST, CIFAR-10, and CombiDigits), while filled markers denote averaged results from our random ImageNNl label remappings. The dashed gray line shows a least-squares fit (R² = 0.36), summarizing an approximate log-lines log-linear dependence between NC, and the gains from using k > proxies. Across all points, the Spearman rank correlation is P p= 0.82 (p < 0.0001), indicating a statistically significant positive association. Consistent with this trend, for regimes with NC1 single-proxy mean imprinting (k: = 1) appears become substantially suboptimal compared to using multiple proxies (k 1).
6 Conclusion
We present a new framework, IMPRINT, to analyze three main components relevant to weight imprinting, namcly, weight generation, normalization, and aggregation. Within IMPRINT, statc-of-the-art imprinting strategies become special cases. This allows for a comprehensive analysis of different approaches through systematic experiments and leads us to generalize to a new, best-performing imprinting variant. That is, using k-means weight generation with L2 normalizations and max aggregation, which outperforms all previously studied methods (see fig. 2
k-means generates better weights than mean. In particular, we find that the mean weight generation (GEN) method, despite its prominence in previous work, falls short compared to k-means even when the number of proxies very small. Remarkably, with as little as 50 samples per class, k-means can already outperform the original imprinting method proposed by Qi al. (2018), highlighting its advantage in low-data regimes.
L2 weight normalization is essential for strong performance. The max aggregation directly scales with the magnitude of the weights. Normalization (NORM ensures that all proxies have the same magnilude, preventing differences vector norms from disproportionately affecting classifier predictions. Nearest neighbor (1-nn) aggregation is not as affected by the lack of normalization, since it uses Euclidcan distance. Although still part of common procedure, normalizations for embeddings (NORM and NORM,))) appear have minimal impact performance.
13

---

Published in Transactions on Machine Learning Research (12/2025)
With aggregation, there is need to store all data. While nearest neighbor (m-nn) aggregation (AGG) performs well when all data is saved (e-g., when there are no storage constraints), max aggregation with limited number of representative proxies (e.g., through k-means) is an efficient alternative without a substantial loss in performance.
Neural collapse proves the efficacy of imprinting. During training, the last-layer weights of a FM tend to collapse to their respective class means. This proves the success of mean imprinting on known classes. New, out-of-distribution data, however, often shows less collapse, making it beneficial to imprint more than one proxy. Our results show that the advantage of using multiple proxies is strongly coupled to the degree of neural collapse: the accuracy gain from using k >> 1 proxies increases approximately log-linearly with NCC, In particular, multi-proxy imprinting yields substantial and predictable improvements on datasets with NC1 exceeding 1. While from a practical perspective, choosing the exact number of proxies & based on pure greedy search with validation data (or as part any AutoML pipeline) is still a valid option, our analysis provides insights into the underlying mechanism, turning NC1, into an indicator for when mulli-proxy imprinting is highly benefitial.
Limitations. Our experiments are limited to foundation models for image classification and do not cover other data modalities such as audio, text, or sensor imputs. While IMPRINT is agnostic to modality, empirical validation of our results outside of vision is needed. Although imprinting alone provides an efficient solution to transfer learning, when compared to purely gradient-based learning, a gap still remains. We do not investigate the benefit combining it with optimization methods such as gradient- learning, and the choice of k still roquires heuristic or empirical selection rather than direct prediction.
Future Work. The usage of both weight and activation sparsity as Shen et al. (023) could change the within- and between-class variability in favor of using higher number of proxies. Besides only using the penultimate layer embeddings for generating the classifier weights, an interesting area of study could be extracting embeddings from previous layers of the FM for this purpose. Recently, the study by Marczak et al.. (2025) showed that adding a multi-layer perceptron projector between the penultimate and classification layers results in representations that are more transferable. Another avenue of research is to thoroughly analyze imprinting the weights of other layers as well (Siam al. 2019
Acknowledgments
Our work is funded by the Deutsche Forschungsgemmensscaatt (DFG, German Research Foundation) FIP-12 Project-ID 528483508, as well as the European Union under the grant project 101079894 (COMFORT: Improving Urologic Cancer Care with Artificial Intelligence Solutions). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union European Health and Digital Executive Agency (HADEA). Neither the European Union nor the granting authority can be held responsible for them. We thank Viet Anh Khoa Tran for initial discussions about the neural collapse phenomenon. We further thank the reviewers and the Action Editor for their detailed and constructive feedback, which substantially improved this work.
Author Contributions
JW contributed to the development of the framework, conducting experiments and evaluated the findings. GA was responsible for investigating NC measures and overall contribution to the project. MK contributed to extending the framework and handling data preparation. AF provided critical feedback on the presentation the results and contributed to refining the manuscript. AL, ER, and FG provided supervision, contributed to the overall concepts presented, and to refining the mamuscript.
14

---

Published in Transactions on Machine Learning Research (12/2025)
References
Dhammika Amaratunga and Javier Cabrera. Analysis of data from viral dna microchips. Journal of the American Statistical Association, 161445) 2001.
James A Anderson. A simple neural network generating an interactive memory. Mothematical biosciences, 14(3-4):
197-220 220, 1972.
James A Anderson, Jack W Silverstein, Stephen A Ritz, and Randall S Jones. Distinctive features, categorical perception, and probability learning: Some applications of a neural model. Psychologioal review, 84(5):413, 1977.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (uds.), Advances Newral Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrIPS 2024, Vencouver, BC, Canada, December 10 15, 2024, 2024. URL http://paperrr nips cc/paper Stappppr_fllss/////
Sale---bsrrrttttee Centtraac-Cooffeeee Coff--oooeeeeeeeee Confereenee.hml html
Mohammad Belal, Taimur Hassan, Abdelfatah Hassan, Divya Velayudhan, Nourekdin Elhendawi, Ahmad Aljarah, and Irfan Hussain. Fsid: novel approach to human activily recognition using few-shot weight imprinting. Scientific Reports, 15(1):20865, 2025.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning, pp. 17 36. JMLR Workshop and Conference Proceedings, 2012.
Benjamin M Bolstad, Rafael A Irizarry Magnus Astrand, and Terence P. Speed. A comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics, 19(2):185 193, 2003.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bolig, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprind xxX::21088.02255 2021.
Google Coral. Retrain classification model on-device with weight imprinting. http:://ccoral.aa//occ////////
retrain-dlaasfitttiooooooo ondevice
Paulino Cristovao, Hidemoto Nakada, Yusuke Tanimura, and Hideki Asoh. Few shot model based on weight imprinting with multiple projection head. In 2022 16th International Conference on Ubiquitous Information Management and Communication (IMCOM), pp. 1 7. IEEE, 2022.
Peter Dayan and David J Willshaw. Optimising synaptic learning rules in linear associative memories. Biological cybernetics, 65)):2553-22) 1991.
Janez Demsar. Statistical comparisons of classifiers over multiple data sels The Journal of Muchine learning research, 7:1-30, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pallern recognilion, pp. 248 225, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web). IEEE signal processing magazine, 1946)114 2012.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT, pp. 4171-4186, 2019.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In ICML, pp. 647-655, 655, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaolua Zhai, Thomas Unterthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning, In International Conference on Learning Representations, 2022.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Dommin-adversarial training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference compuler vision and pollern recognition, pp. 4367-4377 2018
X. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In The Tenth International Conference Learning Representations, ICLR, 2022.
Md Yousuf Harun and Christopher Kanan. good start matters: Enhancing continual learning with data-driven weight initialization. arXiv preprind er erx:::2003 06385, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pallern recograitin,, pp. 770-7777 2016.
Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pp. 65 70, 1979.
Kazufumi Hosoda, Keigo Nishida, Shigeto Seno, Tomohiro Mashita, Hideki Kashioka, and Izumi Ohzawa. single fast hebbian-like process emabling one-shot class addition in deep neural networks without backbone modification. Frontiers in Neuroscience, 18:1344114, 2024.
Minyoung Huh, Pulkit Agrawal, and Alexei A. Efros. What makes ImageNet good for transfer learning?, 2016. arXiv:1600.08614.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pottern analysis and machine intelligence, 16(5):550-554, 1994.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917 963, 2019.
Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. In NeurIPS 2022 Workshop on Distribution Shiftss Connecting Methods and Applications, 2022.
Mina Khan, P Srivatsa, Advait Rane, Shriram Chenniappa, Asadali Hazariwala, and Pattie Maes. Personalizing pre-trained models. orXiv preprint orxi::2106.014 01499, 2021.
Teuvo Kohonen. Correlation matrix memories. JEEE transactions on computers, 100(1):353 359, 2009.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do Better ImageNet Models Transfer Better? In CVPR, pp. 2656-2666, Long Beach, CA, USA, 2019. IEEE.
Vignesh Kothapalli, Neural collapse: A review modelling principles and generalization. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
Alex Krizhevsky, Geolfrey Hinton, et al. CIFAR-10. https://uww.cr cs
http://wwww...toooooooooooo tooottorooo..... sew.entto.edu/-kiiciiifaaae html 2009.
Yiting Lí, Haiyue Zhu, Jun Ma, Sichao Tian, Chek Sing Teo, Cheng Xiang, Prahlad Vadakkepa, and Tong Heng Lee. Classification weight imprinting for data efficient object delection. In 2021 IEEE 30th International Symposium Industrial Electronics (ISIE), pp. 1-5. IEFE, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, 10012-10022, 2021.
Daniel Marczak, Sebastian Cygert, Tomasz Trzcińskii and Bartlomiej Twardowski. Revisiting supervision for continual representation learning. In Earopean Conference on Computer Vision, pp. 181-197. Springer, 2025.
Thomas Mensink, Jakob Verbeek, Florent Perromnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes al near-zero cost. 1EEE transactions on pallern analysis and machine intelligence, 35 (11):2624 2637, 2013.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Yair Movshovitz-AAtias, Alexander Toshev, Thomas K Leung, Sergey lofle, and Saurabh Singh. No fuss distance metric learning using proxies. In Proceedings of the IEEE international conference on computer vision, pp. 360-363- 368, 2017.
Kaoru Nakano. Associatron-a model of associative memory. IEEE Transactions on systems, mon, and cybernetics, pp. 380-388, 2007.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsuperoised feature learning, volume 2011, pp. 4. Granada, 2011.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training, Proceedings of the National Academy of Sciences, 117(40):24632 24652-2463,, 2020.
Nikolaos Passalis, Alexandros Iosifidis, Moncef Gabbouj, and Auastasios Tefas. Hypersplerr--used weight imprrinting for few-shot learning embedded devices. IEEE Transactions on Neurul Networks and Learwing Systems, 32(2): 925-930, 2020.
F. Pedregosa,
G. Varoquaux, A. Gramsort, V. Michel, B. Thirion, O. Crisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 12:2825-2830, 2011.
Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. Xrv preprint arXiv:2305.0 002553, 2023.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5832, 2018.
Sylvestre-Alvise Rebulfi, Alexander Kolesuikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning, In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2000- 2017.
Terrence J Sejnowski. Storing covariance with nonlinearly interacting neurons. Journal of mathematical biology, 4(4): 303-30 001- 1977.
Yang Shen, Sanjoy Dasgupta, and Saket Navlakha. Reducing catastrophic forgetting with associative learning: lesson from fruit flies. Neural Computation, 35(11):197-1181, 2023.
Mennatullah Siam, Boris N Oreshkin, and Martin Jagersand. Amp: Adaptive masked proxies for few-shot segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. -248, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advonces in neurul information processing systems, 30, 2017.
Tom Tirer, Haoxiang Huang, and Jonathan Niles Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning, pp. 34301 34329. PMLR, 2023.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. Xixiv preprint Xiv: Xww.::10..00000 708.07747, 2017.
Dingtian Yan, Jitao Huang, Hai Sun, and Fuqiang Ding. Few-shot object detection with weight imprinting. Cognitive Computation, 15(5):1725-1777 1735, 2023.
Hong-Ming Yang, Xu-Yao Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolutional prototype learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3474-3482, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for blass-nncrement learning. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 6982 6991, 2020.
Jianxing Zhang, Pengcheng Xi, Ashkan Ebadi, Hilda Azimi, Stéphane Tremblay, and Alexander Wong. Covid-19 detection from chest x-ray images using imprinted weights approach. arXiv preprint X05:2105001710 2021.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Haiyue Zhu, Xiong Li, Wenjie Chen, Xiaocong Li, Jun Ma, Chek Sing Teo, Tal Joo Teo, and Wei Lin. Weight imprinting classification- based force grasping with a variable-stiffness robotic gripper. IEEE Trunsactions on Automation Science and Engineering, 19(2):969 981, 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29832 2021.
18

---

Published in Transactions on Machine Learning Research (12/2025)
A Appendix
A.1 Additional Results
We provide additional tables and critical difference (CD) diagrams that expand the results of section
Table A.1 & Figure A.1: Benchmarking GEN mechanism for S across FMs and Ts. Best NORM combination for each row is used implicitly. AGG fixed to max. CD diagram depicts statistical significance of k-means as GEN. See fig. for 20.
GEN
k
Avg. acc. %
Average rank
k-means
5
89.15 16.96
mean
1
86.84 17.80
k x-medoids
5
84.87 28.76
k-random k-cav-max
k-cov-max
5
82.11 +10.39
k-modaids (k
k x-random
5
75.93 +11.2
k-fpa
5
63.64 1:2.2
Table A.2 & Figure A.2: Benchmarking NORM across FMs and Ts shows crucial effect of L2 normalization. GEN is fixed to mean and AGG to max. CD diagram depicting statistical significance of L2 for NORM Combinations are listed as "NORMing & NORM pre & NORMpost"
post.
NORM
NORM
NDRMpost
Avg. acc. % + std
L2
none
L2
86.84 17.80
none
none
L2
86.84 17.80
Average rank
L2
L2
L2
86.79 +7.83
L2
none
quantile
82.90 =: 12.87
manu new
L2 12 & quardie
L2
none
none
quantile
82.90 12.87
L2 L2 nome
quentle
natu
L2
L2
quantile
82.83 412.86
L2
L2
83.26 19.18
L2
none
none
67.66 1:22.12
67.66 1222.12
Table A.3 & Figure A.3: Benchmarking NORM across FMs and Ts. GEN is fixed to k-means with k 20 and AGG to max. CD diagram depicting statistical significance of L2 for NORMpost. Combinations listed "NORM & NORM & NORM post.
N
NORM
NORMpost
Avg. acc. % std
L2
none
L2
91.04 16.26
none
none
L2
91.04 16.26
Average rank
L2
L2
L2
91.06 #6.21
L2
L2
quantile
90.51 +6.40
none none none
L2 nono & L2
L2
L2
89.55 16.70
12 none & mannie
L2 quandie
none qaniil
L2
none
quantile
79.21 415.20
none
quantile
79.21 +15.20
L2
none none
73.53 421.05
none
none
none
73.53 +21.05
A.2 Datasets
We brielly describe the datasets used in our experiments.
ImageNet (Deng al. 2009). We use the ILSVRC 2012 version (commonly called ImageNet-1K)
containing 000 classes and 1.2M training images. Since the test set is not publicly available, we use the
19

---

Published in Transactions on Machine Learning Research (12/2025)
resnet18, MNIST
resnet18, FashionMNIST
resnet18, CIFAR-10
82
90
80
78
70
85
800
74
66
80
72
70
68
resnet50, MNIST
resnet50, FashionMNIST
resnet50, CIFAR-10
80
85
75
88
80
76
800
74
72
70
70
68
vit b_16__1______ MNIST
vit 1_16, 16, FashionMNIST
vit___6, CIFAR-10
92.5
82
90.0
90-
80
87.5
85.0
78
88
82.5
76
86
74
84
77.5
72
75.0
70
swin_ b, MNIST
swin FashionMNIST
swin b, CIFAR-10
82
90
80
80
78
86
76
900
75
74
65
70
10'
10
#samples
k=n k=
k=5
20
Figure A.4: k-means with different values for & in n-shot scenarios with focus on 10 < n < 400. Shaded areas indicate 95% confidence intervals (CIs) over five different seeds. Other variables are fixed to our previously described best method of using L2 for NORM and max as AGG. Note that only data for the meaningful case of le < n is shown. For MNIST and FoshionMNISTT mean coases to be the bost strategy already at only roughly 50 samples. See fig. 10 for more values of n.
20

---

Published in Transactions on Machine Learning Research (12/2025)
validation set (50000 images) stand-in. For neural collapse investigations, construct tasks by randomly grouping d classes into one label, producing "d in 1" tasks as explained in section 4
MNIST (Deng) 2012). A benchmark dataset of handwritten digits (0 (0-9), consisting of 60 000 training and 10 000 test grayscale images of size 28 x 28.
FashionMNIST (iaao et al., 2017). A drop-in replacement for MNIST with the same format and number of samples, containing grayscale images fashion items across 10 classes.
CIFA (Krizhevsky et al., 2009). -
A dataset of 32 x 32 RGB images covering 10 object classes,
with 50 000 training and 10000 test samples.
MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), USPS (Hull, 1994). Digit classification datasets each containing digits 0-9 with domain-specific visual characteristics. MNIST-M applies color and texture transformations to MNIST digits, yielding 60 000 training and 10 RGB images of size 28 x 28. SVHN consists of digit crops from house numbers in Google Street View, totaling 73 257 training and 26 032 test images of 32 32 in RGB. USPS contains scanned and normalized handwritten digits in 16 x 16 grayscale, split into 7219 training and 2007 test images.
CombiDigits (Ours). A synthetic dataset constructed by merging all classes from MNIST, MNIST-M, SVHN, and USPS. Each class label corresponds to digit (0-9) and fig. A.5 shows example images of class "6". The dataset includes significant visual heterogeneity across sources and thus simulates multi-modal, non-collapsed class distributions.
4
6
6
6
6
6
6
6
6
Figure A.5: Twenty sample images from class "6" of the CombiDigits dataset. Each row corresponds to one of the four source datasets: MNIST, MNIST-M, SVHN, and USPS (top to bottom), illustrating the intra-class heterogeneity across domains.
A.3
Imprinting as Memory
We revisit the classical idea of Bidirectional Associative Memories (BAMs) (Khhnnen 2009 Anderson 1972
Nakano 2007 Anderson et al.) [977) and the associated update rule (Sejnowski 1977 Dayan & Willshaw 1991h for storing key-value pairs as in
W W v
where W E g is a matrix and k, v E R2d are key and value vectors, respectively, to be stored therein.
Throughout this work, imprinting can be interpreted inserting such key-value associations into the classifier weights. More precisely, the generation (GEN) component. extends the linear classification head by setting
21

---

Published in Transactions on Machine Learning Research (12/2025)
one-hot vector representing the class, and k as the corresponding proxy previonsly computed. While we focus on this simple linear setting, imprinting is not limited to it, as discussed in section 2
Notably, this form of direct memory update has seen renewed attention in modern architectures beyond standard query-key-value attention. In particular, the xLSTM model (Beck et al. 2024) implements this mechanism within its mLSTM memory blocks, where the matrix memory cell is updated by gated key-value associations, closely following this classical covariance rule, indicating a broader resurgence of associative memory principles in contemporary sequence modeling.
A.4 Differences between Foundation Models
While an in-depth comparison of foundation models is beyond the scope of this work, we believe it is important to highlight key observations we have made. In particular, fig. 1la shows significantly lower NC1 scores for vit_b, f_____i 16 and swin_b b on their pre-training ImageNel data compared to the resnet models. We hypothesize that this difference is primarily due to model size and training regimes. The Transformer- based architectures (vit_b, b________ 16 and swin_b) b) have a considerably higher parameter count (~s 87M) than the resnet models (11.7M and 25.6M, respectively). Additionally, vit_b_16 and swin_b were trained for more than three times as many epochs (300 vs. 90) while using a substantially lower learning rate (0.003 and 0.01 vs. 0.1). Notably, the embedding dimensions of these models are comparable, meaning that the observed differences in NNC1 scores cannot be attributed to differences in representation dimensionality. Instead, we argue that the combination of larger model size, extended training duration, and lower learning rates likely contributes to greater overfitting, leading to more pronounced collapse. As discussed in section 5.3 this enables the Transformer--aset FMs to handle the ImageNet tasks with remapped labels more effectively and to achieve significantly better performance on the similarly distributed CIFAR-10 10 dataset.
Figure A.6a similar to fig. 11b illustrates the impact of varying the number of proxics on imprinting accuracy across different foundation models (FMs). The key difference in this figure is the use of none for NORMpre instead of L2. This seemingly minor change reveals striking contrast between CNN- and Transformer-basee architectures: a distinct and consistent dip between A: = 1 and k = d appears in Transformer- based models, whereas this dip is absent in fig. 11b. where L2 is used as NORMpre. and does not occur at all in the resnet models. We hypothesize that this difference arises from the distinct. embedding distributions of CNN- and Transformer-basee based architectures (see, e.g.. Hosoda et al. (2024 Figure S2)).
A.5 Learned Weights (with Multiple Proxies) and Comparisons
Harun & Kanan 2025 recently studied the initialization of classifier weights for novel categories in a Continual Learning (CL) setting using the least squares algorithm, comparing it to random initialization and class mcan imprinting across various loss functions. In contrast, our work centers on imprinting-baser approaches, which avoid using gradient-based optimization and cross-class statistics, operating instead on a per-class basis with immediate availability of data. Nonetheless, we include the least-squares method as a supervised oracle baseline- exxlicitly not an imprinting method fine-tuned for classification accuracy.
We evaluate the performance of least-squares on the same lasks as defined in section 4 Additionally, we introduce a multi-proxy extension, k-least-squares, which combines least squares with k-means clustering. As in the case of weight generation GEN in imprinting (where k-means outperforms mean), we find that using multiple proxies per class improves performance in settings with high NC, as well, indicating that the benefits of multi-prototype representations persist even in this non-imprinting, supervised context.
All experiments use no normalization (none as NORM), based on ablations confirming that additional normal- ization impairs performance. This matches expectations, since least squares outputs are already calibrated, and normalization distorts them.
Least Squares Weights. For all data contained in task T, define H E RIXN as the collection of all feature vectors of the N training samples in T obtained by applying a fixed FM, i.c., h E R the feature vector of the i-th sample in the e-th class. Recall from section 3.2 that the within-class covariance matrix Ew is given Be, (he, (hg - T] and additionally define the total covariance matrix Σr and the mlass-means
22

---

Published in Transactions on Machine Learning Research (12/2025)
resnet18
resnet50
reshet18
resnet50
100
100
95
%
800
acc
80
68
60
40
50
vit b 16
swin b
vit b 16
swin b
100
100
90
820 %
90
80
atc %
28
30
15
15
of proxles
number of proxies k
1
in
10in
(ImageNed)
(ImageNet)
MNIST FashlonMNIST CIFAR-10 CombiDigits
MNIST FashionMNIST
CIFAR-10 CombiDigite
(a) k-means used as GEN. NORMgest and NORMmf are set L2, and NORM to none. Besides the prominent peaks in accuracy
(b) k-least-squares (see appendix A.5 used as GEN and all NORM are set to none. Here, the CNN-based FMs exhibit clear accuracy peak at & d, with consistent drop for intermediste values 1 < & < d. For the Transformer-haseed FMs, however, accuracy is typically highest only improves slightly around pronounced peak. For all FMs, performance declines with k indicating that excessive proxy splitting harmss gen- cralization. This degradation does not occur in k-means GEN. which remains robust with large k (see figs. 11b and A.6a
(as already observed fig. [Ib]. consistent dip between & = 1 and & d appears in Transformer- models, which was not NORM,re set to L2 well. In appendix A.4 we hypothesize that this is due to the distinct embeddings distributions of CNN- and mausformeer-assed archilectures.
seen with
d but does not show
Figure A.6: Averaged accuracy ten random ImageNet label remappings ("d in 1") for every d 1, 10 over number proxies k. Shaded areas indicate 95% Cls over three different seeds. Values for the tasks containing all of MNIST, FashionMNIST, CIFAR-10, resp. CombiDigits at once are shown in dotted lines.
matrix M as
M = [h,..., hc] E c
Σr = Ea, ho) hG) -ho Rix
From these feature statistics, we can obtain the least- weights via
1 M (Er + hab AAI-1
WLs r.s = C
(A.1)
Here, I the identity matrix and À is the weight decay. We to match the value used during the original training of each respective model. Table A.4 lists the specific À values used for all models considered in our experiments.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Table A.4: Weight decay values used for each model.
Model
resnet18, resnet50 (He et al. 2016)
0.0001
vit vit_b_tg 16 Dosovitskiy et al. 2021)
0.1
swin_b b (Liu ct al. 2021
0.05
Equation A.1 reveals that least-squares shares structural similaritics with mean imprinting, but applics more sophisticated normalization scheme that includes both scaling and rotation. Crucially, it relies on cross-class statistics computed across the entire dataset, and thus is incompatible with imprinting scenarios, which operate by directly constructing class weights from the data of single class without supervision access to other classes.
Algorithm k --leass-squares
Input: Class data {H.)a- number of proxies
Output: Weights {Woe with shape [k. I] per class
1: if == then
2:
return standard least squares weights WLS for each class (see eq.
A.1
3: end
4: for each class do
Cluster H. into & clusters via k-means
Assign each cluster j to proxy class (c,j) Collect proxy samples 1 (
8: end for
Compute least squares weights W(ej) jointly for all proxy classes (c,j)
10: Assemble W = (W...)...., W(ck)) for each original class e
1C 11: return (Wo)er
Combining k-means and least-aquares into k-least-squares. To allow for multiple proxies per class, propose s-least--quares, generalization of standard least squares (least-squares that integrates clustering. Instead of computing single weight vector per class using all class samples, we first partition each class's feature set H. into & clusters using k-means. Each cluster is treated as a separate proxy class, effectively expanding the classification task from to k proxy classes. We then solve a single regularized least squares problem over all proxy classes at once, assigning a distinct target vector to each proxy. The resulting weights are grouped by their original class to yicld k weight vectors per class. The complete procedure is given in algorithm
The results in fig, A.6b extend previous findings from mean and k-means for the CNN-based FMs, accuracy increases with increasing proxy count k, peaks at h d, and reveals distinct dip between k = and & = d, confirming the weakness of using only one proxy per class. For Transformer-based FMs, by contrast, accuracy is often highest at = 1, improves slightly near k= d, but lacks a pronounced peak. Beyond > d, accuracy generally declines for x-least-squares, indicating dininishing returns from excessive proxy splitting, contrast, k-means GEN maintains stable performance even as grows, demonstrating robustness to increasing proxy counts.
Comparison with k-means. We now compare the performance of k-means and k-least-squares proxy generation methods, noting again that k-least-squares is not an imprinting scheme as it does not operate on a class-yy-llass basis as the imprinting methods presented in section 3.1 do.
Upon proper comparison of the best performing k-means and k-least-squares configurations, as depicted in fig. A.7 we observe that on our synthetic ImogeNet tasks, --laast-suuares does not consistently offer substantial improvement. In fact, it is only reanet50, in highly multi-modal setting (d 10), that
24

---

Published in Transactions on Machine Learning Research (12/2025)
k-least-squares reaches better accuracy. For the Transformer-basee models, k-means even performs better in these scenarios.
While the Transformer-baee models generally achieve superior performance overall (note the different y-axis scales for CNN- and Transformer-basee models in fig. A.7], a consistent trend across all models is observed: with increasing multi-modalityy (increasing d), using more than one proxy, whether through k-means or x-least-squal square, begins to outperform the single-proxy methods (mean and least-squares). Still, is striking to see how effective single weights with least eqastesquares can be. However, it is also noteworthy how easily k-means can bridge this gap, particularly considering that least= is not actual imprinting schome, while k-means is.
resnet18
resnet50
vit. b. 16
swin b
100
100
98
90
96
80
94
%
acc
70
92
90
60
50
86
10
10
10
least-squares
k-means
k-least-sqaares
Figure A.7: Averaged accuracy of ten random ImageNet label remappings (nd in 17) for every d = 1,..., 10 for mean and least-squares, and optimal k-means and k-least-squares (k: (O...., 15)) weights. Shaded areas indicate 95% Cls over three different seeds. Note the differing y-axis scales for CNN-based (reanet18, resnet50) and Transformer-base multi-modality d, multi-proxy methods (k-means, E-least-squares generally outperform single-proxy methods (mean, least sqquress and that the k-means imprinting scheme is competitive with equaresquares approaches.
(vit_b _61 swin_b) models. The figure illustrates that with increasing
Furthermore, evaluate the k-least-squares weight generation method (using none for all NORM) across the 48 tasks proposed in the main part of our paper (sce section 4 least-squares reaches an average accuracy of 94.54%, while 20-least-squares drops to 91.20%. The observed decrease in accuracy can be explained by the dynamics shown in fig. A.6b In comparison to the numbers presented in fig, 2 this shows that our k-means imprinting scheme significantly bridges the gap between single-proxy imprinting (I: = 1) and the optimal least squares weights.
A.6 Computational Efficiency
Clustering-based imprinting. Our k-means inplementation in GEN uses sklearn cluster. .Means with its default parameters. The cost is O(Nktt) for N samples, & clusters, feature dimension 4, and t Lloyd iterations. While k-mcans not computationally negligible, both assignment and update steps in cach iteration parallelize naturally, and convergence typically requires only a few iterations. Similarly, the other steps in our imprinting framework (NORM, ACG) scale linearly with dataset size or proxy count and parallelize efficiently, resulting in no practical scalability bottlenecks even for larger or more complex datasets.
Gradient-based optimization. The closod-form least-squares costs O(NI + 13 + NIC) from the covariance computation and matrix inversion. If solved through stochastic gradient descent, the cost is O(NICI) with epochs.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Empirical runtime. Table shows it processes all data in single closed-form step rather than iterating class-by-class. Nevertheless, it should be clarified again that least-squares represents the analytic, non-iterative optimum derived from gradient minimization and lacks sequential class handling precisely where imprinting excels (c.g., in continual or edge-learning scenarios).
shows that least-squares is faster than k-means in practice, likely because
Table A.5: Average runtime in scconds for different GEN variants across all 48 tasks (sce scction 4p. To ensure transparency, all timing measurements were obtained on identical hardware (8 vCPUs on Intel Xeon Gold 6438Y+ nodes (2 sockets, 64 physical cores/128 threads)). Cores were not pinned to processes, which may introduce minor variance; however, we observed less than 10% variation across seeds.
GEN variant
Runtime (s)
0.0029
5-means
1.2249
20-means
1.7321
least-squares
0.1593
5-least-Squares
1.3724
10-Least-squares
2.3717
26


Reviews:
[{"review_id": "2YmxmjKCdY", "reviewer": "Reviewer_cfTm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work introduces a framework to analyze weight imprinting in transfer learning, splitting it into generation, normalization, and aggregation steps. The proposed framework is conceptually clear and provides a common view for previous methods. Experimentation is comprehensive: different architectures, datasets and scenarios. They also link the success of imprinting with the neural collapse phenomenon, which provides a theoretical angle.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well supported by extensive experiments and improvements over prior methods, although broader baselines and scalability analysis would strengthen the evidence.", "audience_yn": "Yes", "audience_explanation": "The proposed framework unifies weight imprinting methods and connects them to neural collapse, which are relevant for researchers in transfer learning and related areas.", "requested_changes": "- Try or discuss the computational cost and scalability to larger and more challenging datasets, including potential bottlenecks in the framework steps.\n- Consider at least a discussion of extensions to other modalities and tasks.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "In5XYWWOYB", "reviewer": "Reviewer_81QZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper examines \"imprinting\", a branch of transfer learning that avoids parameter optimization (even for the classification head). Imprinting works in two stages. To train, we obtain representative \"proxy\" (or \"prototype\") vectors for each class by aggregating embeddings of a training set. To make predictions, a test image's embedding is compared to each class' proxy, and a nearest neighbor (or m-nearest neighbors) approach converts these distances to a predicted class. \n\nThis paper suggests a general framework for imprinting, focusing on 3 key design decisions: generation of proxies (at training), normalization of embeddings (at training and test time), and aggregation (at test time to make a prediction). The recommended pipeline involves k-means to obtain proxies, L2 normalization, and m-nearest neighbor aggregation when possible.\n\nThe paper also makes a connection to neural collapse. A previous quantitative measure of collapse (variability collapse, see Eq 1), which gives smaller scores when within-class (co)variance is smaller and when between-class (co)-variance is larger, is said to coorelate with the need for many proxies.\n\nExperiments on several image datasets evaluate lots of different instances of the IMPRINT framework, identifying the GEN, AGG steps that give on average the best performance.", "claims_and_evidence_yn": "No", "claims_explanation": "I'll answer \"no\" about the *present* draft, but I look forward to reading an improved version... I do think a \"Yes\" might be possible here.\n\nStrengths:\n\n* Framework seems to cover a swath of existing literature\n* Coverage of many datasets feels comprehensive\n* Coverage of many architectures (2 resnets, 2 transformers) feels comprehensive\n* Use of \"critical difference\" diagrams and attention to significance of experimental results is welcome\n\nAreas to improve:\n\n* within aggregation, the choice of *distance function* used for prediction should be bigger part of framework\n* neural collapse ideas seem disconnected\n* math details of *variability collapse* are missing\n\n\n## C1: distance function should be part of AGG framework\n\nTo claim that IMPRINT is a general framework, I think you need more attention to aggregation step.\n\nThe 'max' aggregation focuses on maximum inner product, which seems equivalent with Euclidean distance *when L2 normalized*. However, in general, it seems like the two key axes in the design space are:\n\n* what distance function is used (Euclidean, Mahalonobis, manhattan, etc.) \n* how you aggregate votes from the top m neighbors (uniform, weighted by distance, etc.)\n\nIn short, the current AGG step is a bit limited in scope. More comprehensive discussion seems warranted.\n\n## C2: Neural collapse ideas seem disconnected\n\nThe abstract claims that the method \"determines proxies through clustering motivated by the neural collapse phenomenon\". The intro later says there \"exists a relationship between a measure of neural collapse and the success of imprinting.\"\n\nHowever, I don't see any figure directly connecting the NC numerical measure to classifier success. Fig 11 just shows NC as a function of d, an artificial way of assembling a dataset to get more modes within a class. Fig 12 shows accuracy as a function of k, but doesn't directly connect to NC.\n\nMore importantly, it seems a bit tautological to say that if a given class is embedded in more well-separated clumps, then k-means with larger k will be more effective for a nearest-proxy classifier. I don't feel that the \"neural collapse\" angle adds too much to the discussion here... what am I missing? To ask another way, suppose I measure a specific NC value for a new dataset .... can I avoid a grid search over the k in k-means by directly guessing what k is useful based on the NC value? \n\n\n\n## C3: variability collapse metric under-defined\n\nLooking at Eq 1, how do you compute $\\Sigma_W$ (within-class covariance) and $\\Sigma_B$ (between-class covariance) given a finite dataset of images and class labels?\n\nI know how to compute a within-class covariance matrix for a specific classes (e.g. all dog images) ... but how do I get one $\\Sigma_W$ that represents all classes? Do you just average over the each class's matrix? sum? \n\nThe between-class covariance matrix calculation should also be clarified, ideally with a concrete formula.", "audience_yn": "Yes", "audience_explanation": "I've answered Yes above because there does seem to be a sub-population of ML researchers interested in imprinting. There seems to be enough past literature cited here to indicate that.\n\nHowever, my worry here is that imprinting has a rather narrow use case, when you really need efficiency and you'll tolerate a performance hit to get it. I don't think the present paper does enough to remind readers how this method would compare to *actually adapting parameters to the target task*. \n\nFor example, in Fig 6, the ceiling of all imprinting methods for CIFAR-10 with resnet18 is 80% accuracy. Brief scouting online suggests that fine-tuning resnet18 on CIFAR-10 would get closer to 95% accuracy, a notable leap (https://huggingface.co/edadaltocg/resnet18_cifar10). I'd imagine similar gains are possible on other datasets tested here.\n\nI think readers deserve to see explicitly what the accuracy-efficiency tradeoff is for these methods. I'm sure this present method is less expensive, but how much so? Won't most application be tolerant of a few hours of training time? \n\nI'd also appreciate more citations and discussion of real applications the authors have in mind, where the efficiency of imprinting is needed.", "requested_changes": "* 1) better help reader understand the applications you have in mind, where efficiency of imprinting is needed because last-layer or all-layer fine-tuning is not possible, and even storing 'all' training embeddings may be difficult. Surely if you can store all the training images, a little extra space isn't too hard to find? Compared to an image, the embedding might be just 5-10% of the storage, or less!\n\n* 2) help reader understand tradeoffs (acc vs efficiency) of imprinting vs. fine-tuning, at least somewhere in the paper\n\n* 3) Edits to address C1, C2, and C3 issues above\n\n\n### Presentational requested changes\n\nIn Fig 1, I suggest separating the \"training time\" and \"test/prediction time\" aspects of the framework into separate diagrams... it is tempting to read the current diagram left to right as what happens at prediction time, which is conceptually inaccurate.\n\nMore substantial limitations section in Sec. 6 is needed, addressing differences between gradient-based transfer learning and this method (e.g. accuracy limitations). There are probably several other key limitations that could be listed here.", "broader_impact_concerns": "Nothing specific is concerning here.", "additional_comments": "## Minor: poor phrasing: \"all weights contribute equally to the output\"\n\nI'm not a fan of this phrase, found in Conclusion. Maybe instead, you want to say that by ensuring the magnitude of each class-proxy vector is the same, you avoid problems where differences in vector magnitudes make big differences in classifier predictions. \n\n\n## Minor: how many examples is \"few shot\"?\n\nMy reading of \"few shot\" literature is that they often focus on just 5 or 10 examples per class. The present paper suggests (e.g. bottom of page 11) that 50 or 200 examples is where the existing framework really shines... but is that \"few shot\"? I'm open to a big-tent definition, just wanted to say the numerical value of \"few\" seems to vary a lot across the literature", "extra_scores": {}, "extra_text": {}}, {"review_id": "BQ8FVicdO9", "reviewer": "Reviewer_n2pB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed the Imprint framework to address the transfer learning for unseen classes. The framework consists of three steps: GEN, NORM, and AGG, which are based on the embeddings of pre-trained models. It does not require any cross-class statistics and gradient optimization. The details of the three steps are sufficiently explained, and the experiment demonstrates that the K-means GEN and L2 normalization NORM, along with the appropriate k-NN AGG, can provide state-of-the-art results. Also, the comprehensive analyses were conducted, which show that the proposed algorithm can avoid the Neural Collapse problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There are ablation results to be used for the validation of the claim, especially for the neural collapse probelms. The disadvantages are that 1) there is no comparison with the gradient optimization algorithm, 2) It is not clear why the best option in GEN is k-means and others.", "audience_yn": "Yes", "audience_explanation": "The algorithm is beneficial in many areas, and the representational power of deep networks can be analyzed well. It can enrich the study concerning the representative learning.", "requested_changes": "First, the more intensive experiments compared to gradient-based or other type of algorithms. The process to find the optimal hyperparameters is well explained and sufficient. However, the comparison over various types is limited. \nSecond, more analytic results can be required. Especially, the difference between k-means and k-medoids should be highlighted, as their performances are significantly different.", "broader_impact_concerns": "There are no critical ethical issues. Also, the study can be used in the more elaborate transfer learning, which can have practical ways to various domains and industries.", "additional_comments": "Minor comments:\n\nk-cov-max: \n\nIf you can provide a simple description for this algorithm, the reader can gain a more comprehensive understanding. \n\neffect of $d$: \n\nIn the neural collapse, the effect of $d$ is interesting. Can you provide a more insightful explanation, and provide a criterion?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)