On the Dynamics & Transferability of Latent Generalization during Memorization
Authors: Simran Ketha
Venkatakrishnan Ramaswamy
Venue: Accepted by TMLR ()

Abstract: Deep networks have been known to have extraordinary generalization abilities, via mechanisms that aren't yet well understood. It is also known that upon shuffling labels in the training data to varying degrees, deep networks, trained with standard methods, can still achieve perfect or high accuracy on this corrupted training data. This phenomenon is called memorization, and typically comes at the cost of poorer generalization to true labels. Our recent work has demonstrated, surprisingly, that the internal representations of such models retain significantly better latent generalization abilities than is directly apparent from the model. In  particular, it has been shown that such latent generalization can be recovered via simple probes (called MASC probes) on the layer-wise representations of the model. However, the origin and dynamics over training of this latent generalization during memorization is not well understood. Here, we track the training dynamics, empirically, and find that latent generalization abilities largely peak early in training, with model generalization. Next, we investigate to what extent the specific nature of the MASC probe is critical for our ability to extract latent generalization from the model's layerwise outputs. To this end, we first examine the mathematical structure of the MASC probe and show that it is a quadratic classifier, i.e. is non-linear. This brings up the question of the extent to which this latent generalization might be linearly decodable from layerwise outputs. To investigate this, we designed a new linear probe for this setting.  We find cases where this linear probe outperforms MASC, and other cases, where the opposite happens, notably for many instances of ResNet-18 trained on CIFAR-10. Next, we consider the question of whether it is possible to transfer latent generalization to model generalization by directly editing model weights. To this end, we devise a way to transfer the latent generalization present in last-layer representations to the model using the new linear probe. This immediately endows such models with improved generalization in many cases, i.e. without additional training. We also explore training dynamics, when the aforementioned weight editing is done midway during training. Our findings provide a more detailed account of the rich dynamics of latent generalization during memorization, provide clarifying explication on the specific role of the probe in latent generalization, as well as demonstrate the means to leverage this understanding to directly transfer this generalization to the model. Our code is available at: https://github.com/simranketha/Dynamics_during_training_DNN .

Published in Transactions on Machine Learning Research (03/2026)
On the Dynamics & Transferability of Latent Generalization during Memorization
Simran Ketha
p20200021 1 wit.dddiitttttaiit.....o.o
Anuradha & Proshanth Palakurthi Centre for Artificial Intelligence Research, Deportment of Computer Science Information Systems, Birla Institute of Technology & Science Pilani, Hyderabod 500078, India.
Venkatakrishnan Ramaswamy
venkutGhyderabau. bits-pilanei oc.
Anaradha Proshanth Palakurthi Centre for Artificial Intelligence Research, Department of Computer Science & Information Systems, Birla Institute of Technology Science Pilani, Hyderabad 500078, India.
Reviewed on OpenReview: https: // openreview. net/ forum? id= t024Zm0 tKF
Abstract
Deep networks have been known to have extraordinary generalization abilities, via mech- anisms that aren't 't yet well understood. It is also known that upon shuffling labels in the training data to varying degrees, deep networks, trained with standard methods, can still achieve perfect or high accuracy on this corrupted training data. This phenomenon called memorization, and typically comes at the cost of poorer generalization to true labels. Our recent work has demonstrated, surprisingly, that the internal representations of such models retain significantly better latent generalization abilities than is directly apparent from the model. In particular, it has been shown that such latent generalization can be recovered via simple probes (called MASC probes) on the layer-wise represent attions of the model. However, the origin and dynamics over training of this latent generalization during mem- orization is not well understood. Here, we track the training dynamics, empirically, and find that latent generalization abilities largely peak early in training, with model generaliza- tion. Next, we investigate to what extent the specific nature of the MASC probe is critical for our ability to extract latent generalization from the model's layerwise outputs. To this end, we first examine the mathematical structure of the MASC probe and show that it is a quadratic classifier, iee. is non-linear. This brings up the question of the extent to which this latent generalization might be linearly decodable from layerwise outputs. To investi- gate this, we designed new linear probe for this setting. We find cases where this lincar probe outperforms MASC, and other cases, where the opposite happens, notably for many instances of ResNet-18 trained on CIFAR-10. Next, it is possible to transfer latent generalization to model generalization by directly editing model weights. To this end, we devise way to transfer the latent generalization present in last-layer representations to the model using the new linear probe. This immediately endows such models with improved generalization in many cases, i.c. without additional training. We also explore training dynamics, when the aforementioned weight. editing is done mid- way during training. Our findings provide more detailed account of the rich dynamics of latent generalization during memorization, provide clarifying explication the specific role of the probe in latent generalization, well as demonstrate the means leverage this understanding to directly transfer this gemeralization to the model. Our code is available at: httts://githuu com http://www.......aaaaaeeaaaaaaaa during traininngniit DNN
consider the question of whether

---

Published in Transactions on Machine Learning Research (03/2026)
1 Introduction
Overparamcterized deep neural networks have seen widespread deployment in many fields, due their remarkable generalization abilities. However, we still don't have clear understanding of the mechanisms underlying their ability to generalize so well to unseen data. It has also been shown (Zhang et al. 2017 2021 that overparameterized deep networks are capable of achieving high or even perfect training accuracy on datasets, wherein a subset of training data have their labels randomly shuffled. Such models typically have poor generalization performance, i.e. poorer accuracy on test data with correct labels phenomenon that has been called memorization. It known (Arpit al., 2017) that during training, models trained with such corrupted datasets exhibit, better generalization during the initial phases of training; however generalization progressively deteriorates as training accuracy improves subsequently.
Label noise is also a practical issue in Deep Learning. The success of Deep Learning has depended on large, carefully annotated datasels, which are costly to oblain. To reduce labeling effort, non-expert sources such as Amazon Mechanical Turk and web-derived annotations are often used, but they frequently introduce unreliable labels (Paolacci et al., 2010 Cothey 2004; Mason & Suri 2012
Scott et al., 2013). Labeling can also be difficult even for domain experts (Frénay & Verleysen 2013 RV 2004), and may be intentionally
corrupted through ad attacks (Xiao et al. 2012). Such corrupted annotations, referred to as noisy labels, have been estimated to constitute 8.0% 38.5% of real-world datasets (Xiao et al. 2015 Li et al, 2017 Lee et al. 2018 Song et al. (209)) and are known to degrade performance more severely than other noise lypes, such as input noise (Zhu & Wu 2004 Consequently, a better understanding of learning under label noise and the representations that drive it, not only has foundational relevance, but could also have downstream implications for our ability to design more effective techniques for learning in the presence of label noise.
Our recent study (Ketha & Ramaswamy having corrupted labels tend to exhibit poor generalization, their intermediate--ayee representations retain a surprising degree of latent generalization ability. This ability can be recovered from such trained networks by using a simple probe - the Minimum Angle Subspace Classifier (MASC) - that utilizes the subspace geometry of the corrupted training dataset representations, to this end. Our findings suggest that generalizable features are present in the layer-wise representations of such networks, even when the model fails to utilize them sufficiently. However, the origin and evolution of this latent generalization ability during training are not well understood. In particular, it is unclear whether model generalization and latent generalization arisc from the same underlying mechanisms, and whether the specific nature of the probe (MASC) is critical for extracting latent generalization from layer-wise representations. Moreover, it is not even clear whether this latent generalization can be directly transferred to the model. That is, whether it is, in principle, possible to directly modify model weights, so it overtly acquires this latent generalization performance, is not known. On the one hand, this question is conceptually important because establishes conclusively that the latent gencralization manifested by the probe is also within reach of the model, with exactly the information that the model was provided during training, namely the corrupted training data. On the other hand and more pragmatically - it also suggests the possibility of "repairing" trained model that has memorized, without requiring expensive retraining from scratch. Here, we address these questions.
2026 has shown that while deep networks trained datascts
Our main contributions are listed below.
We empirically characterize the evolution of latent generalization as manifested by MASC over epochs of training, in comparison to model generalization over training.
We observe that MASC is a nou-linear classifier. This brings up the question of the extent to which the improved generalization performance of the probe (i.e. MASC) attributable to the effectiveness of the non-linear nature of the probe itself that may not casily be linearly decodable. To address this point, we introduce a simple linear alternative Vector Lincar Probe Intermediato-layer Classifier (VeLPIC). On the one hand, we find cases where VeLPIC achieves superior latent, generalivation performance in comparison to MASC, with this happening especially for higher degrees of label
We provide precise operational definitions of latent, generalization in Subsection 3.1
2

---

Published in Transactions on Machine Learning Research (03/2026)
corruption. On the other hand, there cases, e.g. with ResNet- 1 where the best-performing layers with MASC outperform those of VeLPIC.
As a baseline, we analyze latent generalization during training using a linear probe (logistic regres- sion) trained by minimizing a crossentropy loss & compare its performance with MASC & VeLPIC.
By leveraging the linear probe (VeLPIC), we devise a way to directly edit the pre-softmax weights of such deep networks, that immediately transfers to the model, the latent generalization performance of VeLPIC (as applied to the last layer).
We study the dynamics of model generalization and linear probe behavior (VeLPIC) during training by introducing a targeted weight intervention at a specific training epoch.
2 Related work
In influential work, 7Zhng et al. 2017 2021 showed that deep networks can achieve perfect training accuracy even with randomly shuffled labels, accompanied by poor generalization. In follow-up work, (Arpit et al.] 2017] find that in the memorization regime, networks learn simple patterns first during training. Their work provides a detailed account of the early dynamics of training. More recently, our work (Ketha & Ramaswamy, 2026) in fact show that spite of the fall in model generalization later on in training, the layerwise representations of the model retain significant latent generalization ability. (Ketha et al.) used probes from (Ketha & Ramaswamy
2026 to investigate the role of internal representations in adversarial
setting.
Analyzing intermediate representations in deep networks has been previously explored using kernel-PCC (Montavon et al., 2011) and linear classifier probes (Alain & Bengio 2018). Notably, (Alain & Bengio, 2018) state that they deliberately did not probe deep networks the memorization setting since they thought that such probes would inevitably overfit. On the contrary, (Ketha & Ramaswamy 2026) demonstrate that probes on deep networks in the memorization setting, can have enhanced generalization. (Stephenson et al., 2021) show evidence suggesting that memorization occurs in the later layers. Li et al.
(2200) show that in the memorization regime, gradient descent with early stopping is provably robust to label noise and that
there is substantial deviation from initial weights after the early stopping point, which drives overfitting.
Several training paradigms have been proposed to enhance generalization performance when learning from corrupted datasets. For example, MentorNet (Jiang et al.) 2018) introduces framework wherein a mentor network guides the learning process of a student network by guiding the student model to focus on likely clean labels. Likewise, Co-Teaching (Han et al.) 2018) trained two peer networks simultancously, each selecting small-loss examples to update its counterpart. Early-Learning Regularization (ELR) (Liu et al., 2020) augmented the training objective with regularization term, which implicitly prevents learning of incorrect labels.
Gradient descent on separable data has been shown to exhibit an implicit bias toward specific solutions, even the absence of explicit regularization, helping to explain generalization in overparameterized models (Soudry al. 2018). This view was further refined by characterizing implicit bias through optimization geometry, linking solution selection to the structure of the loss landscape (Gunasckar et al. 2018).
Saxe et al. 2013) offer theoretical explanations on generalization for deep linear networks and Lampinen & Ganguli (2018) offer theoretical explanations in the memorization regime. Methodologies such as Canonical Correlation Analysis (Raghu et al. 2017 Morcos el al. 2018) and Centered Kernel Aligmment (Kornblith
et al. 2019 have been used to characterize training dynamics and network similarity. Representational
geometry and structural metrics provide further insights into learned representation properties (Chung et al.,
2016
Cohen et al.] 2020 Sussillo & Abbott, 2009 Farrell et al. 2019 Bakry et al. 2015 Cayco-Gajie &
Silver
2019
Yosinski et al. 2014). The Neural Tangent Kernel (NTK) framework provides kernel-based
PResNet-18 is the most modern uetwork tested here and, in particular, for the best performing layers, MASC has significantly better accuracy than VcLPIC. Furthermore, there multiple carly layers where MASC that layer outperforms VeLPIC the same layer for many degrees label corruption. However, even with ResNet-18, find that, for later layers, VeLPIC outperforms MASC later in training, when there is high label noise.
3

---

Published in Transactions on Machine Learning Research (03/2026)
perspective learning dynamics in infinitely wide networks, connecting neural network training to kernel regression and offering insights into generalization (Jacot. et al. 2018
Recent advances have highlighted limitations of classical generalization theory. In particular, the double descent phenomenon challenges the traditional dias-variane trade-off by showing that model test risk can decrease even after achieving zero training error as model capacity grows, thereby extending the classical U-shaped risk curve (Belkin al., 2019). This insight has important implications for understanding gen- eralization in high-capacity models that are often trained large, imperfect datasets. In parallel, robust learning in the presence of label noise has emerged as critical research direction, since mislabeled data significantly impair generalization performance and is pervasive in real-world applications. The survey in (Song et al., 2022) systematically reviews existing approaches for learning under noisy labels and highlights the remaining challenges in achieving robustness to label corruption.
3
Preliminaries & experimental setup
3.1 Preliminaries
We study C-class classification problem defined over unknown data distribution D X x V, where X I= and y C}.
Let fo : X + RC denote a model, i... neural network. The corresponding classifier is defined as
ha arg max (a( cey
We assume that the network admits decomposition
fo go 60
where Φ8 : X + Rd denotes the map from the input to the latent representation at a chosen hidden layer, and go T2d + R.C denotes the mapping performed by the remainder of the network.
To study generalization properties inherent the representation from Φe, consider probe flat Rdd that operates solely on latent representations and the corresponding latent classifier is
(a) arg max plat(x)). (x)
In practice, let Tiest = (2x mm denote a Lest datasel of m samples, where E y are the corresponding true labels. For each test input I; E X, we define its latent representation as
= Φo(x;) E Rd.
The empirical latent generalization of the chosen hidden layer is the test accuracy of faat evaluated on the latent representations with respect to their true labels Vi-
1 Σ m
Ama (0)
14 (zx) = (z:)= =21).
The empirical model generalization is the test accuracy of so evaluated on with respect to their true labels
Ā(Ø) = m Σ 1 (ho(x:) = y/ --
4

---

Published in Transactions on Machine Learning Research (03/2026)
Let T B.)l ~ D' be an i.id. training dataset drawn from D. For corruption degred p. we construct a modified training dataset T3 by changing the labels uniformly at random with probability p. Tp is used to train fo; we call such models memorized model4] The corresponding 2 and labels are used to construct probes on the layer in question. In this paper, we study the latent generalization of specific classes of probes during model training, and examine their empirical latent generalization relative to the empirical model generalization of the models at the corresponding epoch of model training.
3.2 Experimental setup
We demonstrate results for the same set of models and datasets as presented in (Ketha. & Ramaswamy 2026).
Specifically, we use Multi-Layer Perceptrons (MLPs) trained on the MNIST (Deng. 2012) and CIFAR-10
Krrzhevsky
2009) datasets; Convolutional Neural Networks (CNNs) trained on MNIST, PasHIon-MNIST
(iao et al. 2017], and CIFAR-10; AlexNet (Krizhevsky et al.] 2012) trained on Tiny ImageNet dataset
[Moustafa 2017) and ResNet-18 (He et al. 2016) trained on CIFAR-10. Additional dctails on the model
architectures and training are available in Section A
Each model was trained under two distinct schemes: (i) using training data with true labels, referred to
"gencralized modcls," and (ii) using training data with labels randomly shuffled to varying degrees (referred to as "memorized models" Zhang et al.] (202T). Similar to Ketha & Rarnaswamy (2026), we train the aforementioned models using corruption degrees of 0%, 20% , 40%, 60%, 80%, and 100%. Training with a corruption degree с implies that, with probability c, the label of a training datapoint is changed with
a randomly selected label drawn uniformly from the set of possible classes. This may result in the label remaining the same after the change as well. All models were trained either until achieving high training accuracy (99% or 100%) or for a maximum of 500 epochs, whichever occurred first.
To study the dynamics of the training process, we conducted the experiments on model checkpoints saved at various stages of training. Specifically, we began with the randomly initialized model (corresponding to epoch 0), followed by chockpoints saved at every second epoch up to the 20th epoch. Beyond epoch 20, results are shown al intervals of five epochs for the MLP, CNN and ResNet-18 models, and al intervals of ten epochs for the AlexNet model. The reported results are averaged over three independent training runs, with shaded regions in the plots indicating the range across instances.
Ketha & Ramaswany (026)) investigale the organization of class-conditional subspaces using the training data at various layers of deep networks. These subspaces are estimated via Principal Components Analysis (PCA), specifically, ensuring that they pass through the origin. To probe the layerwise geometry without relying on subsequent layers, we propose a new probe- - the Minimum Angle Subspace Classifier (MASC). For a given test input, MASC projects the layer output onto each class-specific subspace, and computes the angles between the original and projected vectors, for each subspace. The label predicted by MASC corresponds to the class whose subspace yields the projected vector with the smallest such angle. We provide a detailed summary of the working of MASC in Section B We have used 99% the percentage of variance explained by the principal components that form the class-specific subspaces used by MASC, similar to experiments conducted in Ketha & Ramaswamy (2026).
4 Training dynamics of latent generalization using MASC
As shown in Ketha & Ramaswamy 2026), for most models trained with corrupted labels, there exists at least one layer where MASC exhibits better generalization than the corresponding trained model. However, the origin& & evolution of this latent. generalization across training isn't well understood.
Here, we empirically study the behavior of latent generalization, as manifested by MASC, during training. MASC testing accuracy during training for MLP trained on MNIST, MLP trained on CIFAR-10, CNN trained MNIST, CNN trained on MaShion-MNIST, CNN trained on CIFAR-10, AlexNet trained Tiny
3 Please see Scction 3.2 (2ud paragraph) for the definition of corruption degree.
4 'Following convention of prior work. The phenomenon of such models having poor empirical model generalization has been called memorization convention we continue to follow here.
5

---

Published in Transactions on Machine Learning Research (03/2026)
ImageNet and ResNet-18 trained CIFAR-10 are shown in Figure
Results with 0% and 100% corruption
degrees are shown in Figure T0 in Section
Model accuracy
(2048) FC2 (250%/ Alexkeet-FC (4096)) ResNet-18-12 (8192)
Input
MLP-FC4 (2048)/ CMN-FC3 (250y RedNet-18-13 (4096)
(1281) (576/1024) AlexNet- (256)/ Bel (16384)
MLP-FC2 (512)/ CNN-FC1 (250)/ AlexNes-FC1 (4006) ResNat-18-L1 (16384)
RosNet-18-Ave BarNett1--Avg_Paee pool (512)
20% corruption degree
40% corruption degree
60% corruption degree
80% corruption degree
MLD MNISTTT
0.2
0.2
0.2
022
0.0
0.0
0.0
0.0
24
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Minimum Angle Subspace Classifier (MASC) Test Accuracy
0.6
0.6
0.6
0.6
MLP
0.4
0.4
0.4
122
200
CNN MNVTT
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
400
129
148
430
141
320
CNN Fastion Maiion-MMISTT MNST
1.0
1.0
1.0
1.0
0.8
0.8-
0.8
0.8
CNN CIFAR 10
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
252
124
165
201
630
115
261
125
116
482
MediesTinn Imageeet geVee
9.15
0.125
0.025
0.100
0.06
0.020
0.075
0.04
0.015
0.05
0.050
0.02
0.010
0.00
0.000
0.005
Recke:18 CFAR10 10
1.0
10
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
02
16
4
Epoch
Figure Minimum Angle Subspace Classifier (MASC) test accuracy over epochs of training for multiple models/datasets, where test data is projected onto class-specific subspaces constructed at each epoch from corrupted training data with the indicated label corruption degree. The plots display MASC accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training also shown. FC denotes fully connected layers with ReLU activation, and Flat refers to the flatten layer without ReLU.
It known Arpit et al. 2017) that for models trained with label noise for corruption degrees less than 100%, there early rise in model test. accuracy that culminates in peak, subsequent to which the

---

Published in Transactions on Machine Learning Research (03/2026)
model test accuracy falls. For various non-zero degrees of corruption, in most cases (except those with 100% degrees of corruption), the MASC test accuracy largely follows the rise in the model's test accuracy up to its peak. However, beyond the peak, while the model's test accuracy declines significantly, the drop in MASC accuracy for the best-performing layers is usually less steep and platcaus at a higher levenover the epochs. A notable exception to these observations occurs in the early layers of ResNet-18 trained on CIFAR-10, which we discuss next.
An interesting point about the training dynamics is indeed what occurs prior to the start of training, i.c. just at the random initialization, which we plot as Epoch 0 in Figures and T0 At Epoch 0, the model always has roughly chance-level test accuracy. However, MASC demonstrates test accuracies that are significantly above chance; there is variation in the values depending on the layer in question for each model. However, for most models (except ResNet-18 trained on CIFAR-10), the rise of test accuracies over the initial epochs happens at rates comparable to that of MASC. For this reason, in those models, the model test accuracy as well as the MASC test accuracies appear to be largely overlaid on each other early in training, prior to the peak in test accuracy. For ResNet-18 trained on CIFAR-10 however, MASC test accuracies seem to improve somewhat slower over early epochs. Furthermore, in case of some early layers (i.e. LO, L1), for higher corruption degrees, MASC in fact beats the best carly stopping accuracy of the model at Epoch O, and the MASC test accuracies end up being at levels markedly above model test accuracies all through training. This may have to do with the significantly higher ambient dimensionality (16,384) of those layers. Also, LO MASC test accuracy appears largely flad over training, whereas LI MASC test accuracy shows a slight rise early in training before being largely flat.
One question that arises is about why there is marked difference in MASC test accuracies across layers of the network. A definitive answer likely requires deeper understanding of the principles used by deep networks to organize layerwise representations and their evolution across layers - an understanding that we don't yet have, as a field. That said, one observes that MASC layer performance appears to be correlated with the ambicnt dimensionality of the layer, in most cases. That is, the best-performing layers tend to be those with the highest dimensionality. degrees, where there exist layers that outperform the CNN-Flat (1024) layer. In many models (MLPs, AlexNet, ResNet-18), multiple layers have the highest dimensionality. Here, layer performance differs, albeit not significantly. Furthermore, it does not seem to be determined by their relative position of those layers in the deep network.
notable exception is with CNN-CIFAR-10 for lower corruption
Our results represent progress in clarifying the origin & evolution of latent generalization by MASC, during training. In particular, given that model generalization & latent generalization often show a concurrent initial rise, it suggests the possibility of common mechanisms that drive both in the early phases of training, for those models. The subsequent divergence between model generalization & latent generalization is an intriguing phenomenon, whose mechanisms merit future investigation.
5 Non-linearity of MASC
Classically (Alain& & Bengio 2018). , linear probes have been used to probe layers of deep networks. However, [Ketha & Ramaswany, 2026) do not use the standard linear probe from (Alain & Bengio, 2018). Lincar probes are simple and interpretable, due to which they have been widely deployed. Furthermore, a linear probe demonstrating good performance indicates that the representations of the layer that the probe operates on, contain information that is linearly decodable, This implies that such information is, in principle easily decodable by downstream layers. While MASC is a clegant probe with a nice geometric interpretation, (Ketha & Ramaswany (0066) do not consider the question of whether it is a linear probe. Below, we prove that MASC (Ketha & Ramaswamy 2026) is in fact a non-linear classifier. In particular, it is quadratic in the layerwise output of the layer that it is applied to. This mcans that it uses quadratic dccision surface.
Indeed, as reported in (Ketha & Ramaswamy 2026), ; there are layers whose MASC performance is worse than the model performance.
Indeed, the performance of MASC at Epuch 0 for subset of models has also been reported in Ketha & Ramaswamy
2026
We nate that L.0 does not have residual connections, unlike subsequent layers, although it is unclear if this contributes to this phenomenan.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Proposition 1. MASC is quadratic classifier.
Proof. Let æ denote the output of the layer I of the deep network when it is given imput æ. Let P2.p2..... P% be an orthonormal basis of the subspace S. corresponding to class Let of be the projection of æn on Se. We have
कं = (α) pi) pi + + (a) ph) Pk.
(1)
Now, MASC on layer predicts p the label of x as
arg max (n af) = arg max pi)² +... + (a) ph)))
(2)
which is quadratic in 24. This establishes that MASC is a quadratic classifier.
6 Vector Linear Probe Intermediate-layer Classifier (VeLPIC): A new linear probe
Given that MASC is inherently a non-linear classifier as proved above, natural question is if ils remarkable ability to decode generalization from hidden representations of memorized networks is largely a consequence of its non-linearity. In other words, if the quadratic nature of MASC is indeed responsible for its effectiveness, then a corresponding linear probe would be expected to perform substantially worse. It raises the question of the extent. which the latent generalization reported (Ketha & Ramaswamy 2026 linearly decodable from the layerwise representations of the network.
To investigate this, we build linear probe analogous to MASC. We sought to retain the same broad idea, namely determine an instance of a mathematical object per class and measure closeness of the layerwise output of an incoming datapoint to these objects with the prediction corresponding to the class whose object was closest in this sense. In contrast to (Alain & Bengio, 2018), where parameters of their linear probe are learned iteratively by minimizing a cross-entropy loss, we seek to determine the linear probe parameters directly via the geometry of the class-conditional training data. We choose to simply use vector To as this mathematical object and measure closeness in the angle sense. We call this probe the Vector Linear Probe Intermediate-layee layer Classifier (VeLPIC). As we discuss subsequently, we find, surprisingly, that this choice is often more effective than MASC. Secondly, we show that we can use the parameters of the probe as applied to the last layer, to modify the model weights to immediately confer the corresponding gencralization to the model.
We now discuss how the vector corresponding to each class in VeLPIC constructed. Each class vector is determined using only the top principal component from PCA run on augmented" |T| class-conditional corrupted training data. However, the first principal component can manifest in two opposite directions (i.e. the vector or its negative). This is important here because incoming data vectors can be "close" to this class vector, even though their angles are obtuse and closer to 180". VcLPIC resolves this directional issue by aligning the class vector based on the sign of the projection of the training data mean; if the mean of the training data projected on this principal component is negative, the direction of the principal component is flipped to obtain the class vector; otherwise, it is retained as is.
Formally, for a given test data point x. let कi denote its activation at layer obtained in the forward pass of x through the deep network until the output of layer 1. For layer 1, let } be the top principal component vectors, one each per class, of the class-conditional corrupted training data and {Tm J.M be its corresponding
13 projection means, where M is the number of classes. Let ) be unit vectors
"which is typically estimated via PCA, where & is the number of principal components. "This is equivalent to the formulation of MASC in (Ketha & Ramaswamy 2026), where maximize cosine similarity.
See Section B.1 for a proof of equivalence.
10 note that previous work (Das et al. 2007 has used the idea of using classwise-PAA for classification in the setting of EEG data, although they use Bayes classifier to perform the classification.
to obtain 1-D subspace, rather than 1-D affine space, along the lines of the subspacc construction procodure for MASC.
We augment class training data points with their nogative,
12Qbserve that. this isn't an issue with MASC, since a quadratic classifier.
13.... T: is the mean of projecting training data points on P..
8

---

Published in Transactions on Machine Learning Research (03/2026)
representing VcLPIC class vectors. VcLPIC uses (Vn Jo= to predict the label of æ based on its maximum projection among these class vectors as outlined in Algorithm 1
Model test accuracy
MLP-FC3 (2048)/ CMN-FC2 (250% Aloxket-FCZ (4096) ResNet-1B-22 (8192)
(2048)/ (250V/ ResNet-18. (4095)
MIP-FCI (1281/ CNN-Fial (576/1024)/ Aleshint-Fat (756)) ResNet-1B-LD (16384)
RosNet-18-kB 18-14 (2048)
MLP-F22
AloxNes-FC1 ResNet-18-L1
RosNet: poolopooollloo
20% corruption degree
40% corruption degree
60% corruption degree
80% corruption degree
1.0
1.0
100
1.0
MUP NNNST
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
Vector Linear Probe Intermediate-bn laee Classifier (VeLPIC) Test Accuracy
120
150
120
10
LLP-CIAAR11
0.4
0.2
0.2
0.2
02
0.0
0.0
0.0
0.0
40
120
126
166
200
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
119
CNN Fazhinn MNST
0.8-
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
180
30
121
180
201
431
761
128
481
1.0
1.0
1.0
1.0
-0
ON--FAR:30
0.0
0.0
0.0
0.0
AlcsNet Tiny Imagetet
0.125
0.025
0.100
0.30
0.075
0.04
0.015
0.05
0.050
0.025
0.02
61
124
100
0.000
66
12
166
30
06
166
240
353
49
0.005
30
144
20
33
460
48
Resler- 18 C338-33 10
1.0
0.0
0.0
0.0
Epoch
Figure 2: Vector Linear Probe Intermediaaee-ayer Classifier (VeLPIC) test accuracy during training of the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display VeLPIC accuracy across different layers of the network for various model-datasee combinations. For reference, the test. accuracy of the models (blue dotted line) over epochs of training is also shown. FC denotes fully connected layers with ReLU activation, and Flat refers to the flatten layer without ReLU.
14This is equivalent minimum angle the VeLPIC class

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm Vector Linear Probe Intermediate-layer Classifier (VeLPIC)
Input: Principal component vectors {P m b projection training mcans {T m mm=1, layer output æt, class
}M
labels (Cm ) M
m=1-
Output: Predicted label y(æ)). for each class m ,....,,M M do
if Tm 0 then
3: Vm P
4: else
5: Vm + Pm
6: end if
7: end for
for each class m ....,, M do
3tm f Projection of æı onto Vm 10: end for
11: y(m)) t Cj where arg maxm Zim
12: Return: y(œ1)
6.1 Training dynamics of the linear probe
Here, we examine the performance of VeLPIC, in contrast that of MASC during training, assess the extent of lincar docodability of latent generalization. It is worth mentioning that favorable performance of VeLPIC implies high linear decodability; however, poor performance of VeLPIC does not rule out the possibility of the existence of another linear probe with superior performance.
VeLPIC test accuracy during training for MLP-MNIST, MLP-CIFAR-10, CNN-MNIST, CNN-Fashion- MNIST, CNN-CIFAR- AlexNet-Tiny ImageNet and MesNee--8-CIFAN
are shown in Figure 2 The results with 0% and 100% corruption degroes are shown Figure 11 in Scction D The difference between VeLPIC test accuracy and MASC test accuracy are shown in Figure Results for 0% and 100% corruption degrees are available in Section D.I
Unexpectedly, the performance of VeLPIC is often, but not always, better than that of MASC. For represen- tations from many layers, VeLPIC is able to extract significantly better latent generalization performance than MASC and our results show that, for these layers, VeLPIC's performance plateaus at significantly higher levels than MASC, There also exist many cases, where MASC does indeed outperform VeLPIC. particular, in ResNet-1s trained on CIFAR-10, MASC substantially outperforms VeLPIC's test accuracy in layers LO and L1, which were the best. performing layers for MASC in this model. This suggests the possi- bility that indeed in these layers, MASC utilizes its nonlincarity in ways that VeLPIC is not able to. There is also the possibility of ot linear probes that may be able to do better for this model. Similarly, in multiple other cases (e.g. early layers of CNN-MNIST, MoN-Faahhion-MNISS later layers of MLP-MNIST, MIP-CIFAR-10), MASC outperforms VeLPIC.
One factor that appears be correlated with poor linear decodability as manifested by VeLPIC, is the ambient dimensionality of layerwise outputs. Recall that, in the previous section, we observed that MASC appears to be quite effective when the ambient dimensionality of the layer in question is high. The results in this section suggest that VeLPIC performance is concomitantly lower in such layers, in comparison to MASC. This hypothesis is especially supported by results from layers Lo and L1 of ResNet-18, which have the highest layerwise dimensionality of 16,384. However, results from CNN-CIFAR- 10 run contrary to this hypothesis. Here, VeLPIC outperforms MASC in the CNN-Flat layer, with dimensionality 1024, which has the highest dimensionality of all layers lested in this model. Therefore, while ambient dimensionality may
Which is arguably, the most modern uetwork tested. We do note, however, that AlexNet larger model, by the measure of mumber of trainable paramcters.
16 show the next section, the only other baseline linear probe tested, fact shows worse performance than VeLPIC these two layers, for higher corruption degrees.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy
MUAFOI (2048)/ CNW-F (256) AlexNet-FC2 (4096) Res Net-8811 (8192)
MLP-FCA (2048V CMVFF33 ResNet-
MLP-FCI (128y CNN-fiat 15 15/0/1024 Nexheet- fltt (256 Reslict-B 18-10 136384;
ResNet- 18-0--18-44--- (2048)
MLP-FC2 (512// CNN (250y NexNet-FC1 (4096) ResNet- 1a--t-1--111 (16384)
ResNet-18-Avg pool (512)
20% corruption degree
40% corruption degree
60% carruption degree
80% corruption degree
0.05
0.8
0.00
0.05
0.1
0.6
0.4
-0.05
-0.10
0.0
-0.15
-0.15
0.15
0.2
0.05
0.10
0.1
0.1
0.1
0.00
0.0
Difference Test Accuracy (VeLPIC Acc. MASC Acc.)
-0.10
-0.1
-0.1
0.15
0.3
0.4
0.025
0.10
0.05
0.1
0.2
-0.050
-0.225
-0.05
0.00
0.0
0.0
142
141
0.050
0.15
0.10
0.2
0.3
0.2
0.000
0.1
0.00
0.0
-0.050
0.05
-0.1
-0.075
16
162
165
0.3
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
-0.1
-0.1
0.11
144
26
38
246
144
321
240
Imagetet
0.04
0.04
0.005
0.01
0.02
0.00
0.00
-0.02
0.1
0.1
0.1
0.0
0.0
-0.1
-.2
-0.3
0.2
-0.1
Epoch
Figure 3: Difference in test accuracy (VeLPIC Accuracy MASC Accuracy) during training of the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various model-dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be
be an important factor, the extent of linear decodability or non-decodability of latent generalization during memorization might involve an interplay of multiple factors that require careful investigation.
Interestingly, with some CNN models (o.g. CNN-Pashion-MNIST)) MASC decodes good generalization early layers but later layers; however, VeLPIC is able to extract comparable generalization from later layers well.
11

---

on Published in Transactions on Machine Learning Research (03/2026)
We compare the test accuracy of the model, MASC, and VeLPIC for both randomly initialized and trained models across varying corruption degrees in Section H Furthermore, we study the impact of dropout as a regularizer in latent generalization (MASC & VeLPIC) during memorization in Section [
In closing, our results with the new linear probe indicate that there are multiple cases where one can linearly decode latent generalization to a greater extent than reported with MASC. Here, as well, it is unclear if there exist other quadratic classifiers that can extract better latent generalization than VeLPIC On the other hand, there are important exceptions where MASC still significantly outperforms this linear probe, especially for ResNet--1, 18, which is the most modern network architecture tested. It is unclear if the underperformance of VeLPIC is because a significant mcasure of latent generalization in these cases is not lincarly decodable. Alternatively, there could exist other linear probes that could match or outperform MASC in these cases.
7 Comparing MASC & VeLPIC with a baseline linear probe
While VeLPIC is indeed linear probe, it differs from standard lincar probes (Alain & Bengio 2018 which are trained by iteratively minimizing suitable loss function, unlike VeLPIC whose parameters are directly derived from the class-conditinaal geometry of internal representations. In order to compare the relative performance of MASC and VeLPIC with a classic lincar probe, here, we study the evolution of latent generalization during training using a logistic regression probe (LR probe).
For given model and network layer, we train a logistic regression probe on the outputs of the layer for 20 epochs using cross-entropy loss with the Adam optimizer (for learning rate 1 x 10-3). The test accuracy of the logistic regression probe during training of the memorized models, for all models and corruption degrees ranging from 20% to 80%, is shown in Figure 4 Corresponding results for 0% corruption degree is presented
in Figure 13 For the non-zero corruption degrees, we find interestingly that in most cases, there is at least a layer or two, whose LR probe tends to largely follow the dynamics of the model's test accuracy over training, including after the early peak, with ResNet-18 being a notable exception. Secondly, in many cases where the test accuracy of the LRR probe does not follow that of the model (o.g. CNN-CIFAR-10 for higher corruption degrees), the test accuracies of the LR probe after the early peak tend to plateau a lower level than is the case with MASC and VeLPIC probes. Thirdly, for many layers, the accuracy of the LR probe tends to underperforn[ 18 both MASC and VeLPIC.
We compare the performance of the logistic regression probe with that of the MASC probe in Figures 5 and 14 We find that MASC perforos LR for many layers in multiple models; however, there are several layers where the opposite effect is true. There is also some regularity in the identity of layers where this happens. For some models (MLPs, CNN-CIFAR-10 and AlexNet- Tiny ImageNet) the LR probe outperforms MASC in the early layers and vice versa in the later layers. For other models (CNN-MNIST, CNN-Fashion- MNIST, ResNet- -I-CAR--) one sees the opposite effect.
We further compare the logistic regression probe results with the VeLPIC probe in Figures 0 and 15
Here, we find that especially for higher corruption degrees and later during model training, VeLPIC almost always outperforms the LR probe. Even for low corruption degrees, in most layers (with 20% corruption degree being a notable exception), VeLPIC outperforms the LR probe.
A natural question that arises here is about why the logistic regression probe underperforms VeLPIC with respect to generalization to true labels, even though both of them are lincar classifiers. We speculate that this might have to do with the fact that classic linear probes seek to minimize a loss which corresponds to doing well on the corrupted training set, which can be anthithetical to doing well on the test set which has true labels. However, techniques such as MASC and VcLPIC have an unsupervised flavor to them that does not seek to optimize training set performance explicitly, even though they operate on the same corrupted training set. This view requires careful future investigation.
A general quadratic classifier, in principle, will do as well VeLPIC, by keeping only its lincar terms and setting its quadratic terms to zero. 18eThe corresponding differences are plotted in Figures and
12

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy
MLP-FC4 (2048) CNN-FC3 12500/ Restot-1--13
(576/1024V
MLP-FC2 (512/ CNN-FC1 1250// AlexNur-FC] (4006)/ Redtet-18-11
MLP-FC3 (2048) (250N AlexNet-FC2 (4096) Resivat-18.12
20% corruption degree
40% corruption degree
60% corruption degree
80% corruption degree
1.0
1.0
100
1.0
0.8
0.8
0.8
0.8
MLP
0.6
0.6
0.6
0.6
150
120
ML CIFAR0
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
Logistic Regression Probe Test Accuracy
43
139
126
166
290
1.0
1.0
1.0
1.0
CNN NNN
0.8
0.8
0.8
0.8
0.6
0.6
5
CNN Faghien MNI
0.8-
0.6
0.6-
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
180
200
30
791
121
180
201
261
128
485
1.0
1.0
1.0
1.0
CIFA33
CNN
0.0
0.0
0.0
0.0
Rcsle-18 CIB3R:10 AlexNet Thy Imagettt
9.25
0.125
0.025
0.100
0.06
0.020
0.04
0.015
0.05
0.050
0.025
0.010
0.005
120
106
164
206
20
148
240
300
489
0
141
30
30
426
400
1.0
1.0
1.0
1.0
02
0.0
0.0
0.0
0.0
Epoch
Figure 4: Logistic regression probe's test accuracy over epochs of training for multiple models/datasets The plots display logistic regression probe's accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training is also shown.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy
MLP-TC4 (204B) CNN-FCI (2501/ ResNet--B-L3
MLP-FC2 (512M OVN-CI (2501) Alexheet FC1 (4095) ResNet 18-11
ResNet 18-Avg_po
MLP-FC3 (2048) CNN-FC2 (250H/ AlexNet (4096)/ ResNNet18-122
20% corruption degree
40% corruption degree
60% corruption degree
80% corruption degree
0.0
0.1
0.25
-0.1
0.0
-0.25
-0.1
-0.50
-0.2
-0.75
0.1
0.1
0.14
Difference Test Accuracy (MASC Acc. Logistic regression Acc.)
0.0
0.0
0.0
-0.1
-0.1
-0.1
-0.1
-0.2
-0.2
0.0
0.6
CHN-NNNST
0.4
0.2
0.2-
0.0
0.0
-0.2
24
0.4
0.4
0.4
0.2
0.1
0.0
-0.1
140
26
402
14
14
10
0.1
0.1
0.1
0.0
-0.1
-0.1
-0.1
CI-CA-100 Aloxtet- tny Imagotet
0.02
0.02
0.00
0.02
0.01
0.02
-0.02
0.00
0.00
-0.01
-0.04
-0.03
0.3
0.2
0.2
-0.1
0.0
0.0
0.0
-02
0.2
-0.1
Epoch
Figure 5: Difference in test accuracy (MASC Accuracy Logistic regression probe Accuracy) during training of the network, where for MASC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference accuracy across different layers of the network for various model dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be
14

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy
MLP-TC4 (204B) CNN-FCI (2501/ ResNett-B-L3
MLP-FC2 (512M OVN-CI (2501) Alexheet FC1 (4095) ResNet 18-11
ResNet 18-Avg_pod
MLP-FC3 (2048) CNN-FC2 (250H/ AlexNet (4096)/ ResNNet18812
20% corruption degree
40% corruption degree
60% corruption degree
80% corruption degree
0.05
0.00
0.2
0.2-
-0.10
-005
0.0
0.0
-0.1
0.2-
-0.2
0.05
0.15
Difference Test Accuracy (VeLPIC ACC. Logistic regression Acc.)
0.1
0.00
0.00
0.05
-0.55
-0.05
-0.10
-0.10
-0.1
-0.05
-0.15
-0.10
0.6
0.6
CNNNNN
0.4
02
0.2
248
362
CNN Fashion MNIST
0.6
0.4
0.4
02-
0.2
0.2
00.
480
144
16
402
48
14
248
162
CITAR-10
0.2
0.2
0.1
0.1
0.1
0.1-
0.0
-0.1
-0.1
-0.1
-0.1
Imagotet.
0.04
0.02
0.015
0.02
0.00
0.00
0.00
-0.01
0,000
0.02
00.02
-0.07
-0.005
10-0000-FAAM AK-10
0.05
0.15
0.15
-0.05
0.05
-0.10
0.05
0.05
0.05
-.20
0.00
Epoch
Figure 6: Difference in test accuracy (VELPIC Accuracy Logistic regression probe Accuracy) during training of the network, where for VELPIC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference accuracy across different layers of the network for various model dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be
15

---

Published in Transactions on Machine Learning Research (03/2026)
8 Transferring latent generalization to model generalization
Given that latent generalization often exceeds the models' generalization ability, natural question is whether this hidden latent generalization can be transferred directly to the model by appropriately modifying its weights. An affirmative answer to this question has implications that the phenomenon of latent generalization is not mainly driven by the remarkable effectiveness of the probe (MASC) used in our prior work (Ketha) & Ramaswamy| 2026). Conceptually, this question is significant because it demonstrates that the probe's latent generalization is inherently accessible to the model using only the corrupted data it was trained on. Yet, the model during the act of standard training does not end up generalizing to the same extent, for reasons that we don't understand well. Practically, it raises the possibility of repairing memorized model using the latent representations, without the cost of retraining from scratch.
Here, we ask if the latent generalization in models that memorize, can be directly transferred to the model, in order to immediately improve its generalization. To this end, it turns out that the class vectors of VeLPIC applied to the last layer can be directly substituted in the pre-softmax layer of the model as an intervention that transfers VeLPIC's generalization performance to the model, without further training. We elaborate below on how this so.
Consider model whose last layer (i.e. the layer preceding the pre-softmax layer) consists of d units. Let 0j E Rd be the VeLPIC class vector for class j. The new pro-softmax weight matrix W pre-softmax E RRM Mxd is constructed as:
W pre-softmax ([01
U2
UM
(3)
This weight matrix W pre-softmax replaces the original pre-softmax weights, and all biases are set to zero. It is straightforward to see that this substitution results in the model making the same predictions as VeLPIC applied to the last layer. While this substitution is fairly straightforward, it needs the linear probe, i.e VeLPIC, for it to work. In particular, we note that it is unclear if transferability can happen with MASC alone and therefore the development of VcLPIC was an enabling factor in helping us answer this question.
During model training, we replace the pre-softmax weights with VeI.PCC vectors, as indicated above and evaluate the model's performance on the test dataset at each epoch. Figure 7 presents these results for MLP- MNIST, MLP-CIFAR-10 CNN-MNIST, LNN-Fasshion-MNIST. CNN-CIFAR-10, AlexNet- Tiny ImageNet and 1esNNe--18CIIAAR-II Results for models with 0% and 100% corruption levels, for all model-dataset pairs are presented in Figure 16 in Section F
We observe that, in most cases, the weight intervention that replaces pre-softmax weights with the VeLPIC vectors leads to an immediate & significant improvement in generalization performance in every epoch of the latter phase of training, matching that of the lincar probe, & in particular, without any further training. A notable exception is for Res.Net-18, where, especially for lower corruption degrees the edited model has worse generalization performance. For this model, for none of the probes tested, is it the case that later layers manifest better latent generalization than the model, due to which this weight editing technique does not lead to better model generalization either. An interesting question is if one could develop weight editing techniques that can transfer exactly or approximately probe performance on early layers well, to the model. In closing, we establish in this section that the latent generalization in memorized models can be directly harnessed in many cases to enhance their test performance, even in the presence of label noise.
9 Weight intervention during training using VeLPIC
Having shown that latent generalization can be directly transferred into the model by editing its weights, natural question arises: What happens if we inject/ this latent generalization to the model at a specific epoch and continue standard training thereafter?
We ask whether intervening during training - by updating the model weights at a specific epoch using information derived from its latent generalization can enhance model generalization. Additionally, we investigate how latent generalization (VeLPIC) evolves across different layers throughout training, when we do so.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Weight using
20% corruption degree
40% corruption degree
60% corruption degree
80% corruption degree
MLB
220
140
283
120
120
129
330
CNN
0.4
Model Test Accuracy
CNN Fashion MNIST
0.8-
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.0
1.0
1.0
1.0
CWN CIFAR.10
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.
02
0.0
0.0
0.0
0.0
Resie:3 1...e.3B.CI. CIFAR: 10 Alenlee Tiry Imagefet
0.15
0.125
0.100
0.06
0.20
0.075
0.025
0.00
0.000
0.005
126
129
16
20
2/c
06
166
144
20
1.00
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0.
0.0
0.0
0.0
i 0.0
Epoch
Figure Model test accuracy when the weight intervention applied to the epoch in question during training. The test accuracy of the model with standard training without weight intervention (blue dotted line) is overlaid for comparison.
To address these questions, the model is trained with corrupted data for the first 40 epochs using standard training. The intervention is performed at the 40th epoch by replacing the pro-softmax weights with VeLPIC vectors (last layer), as done in the previous section. Standard training is performed for the next 60 epochs using corrupted training data. Model test accuracy on true labels shown in Figure S overlaid with the case of no intervention, for comparison. VeLPIC test accuracy during training when this intervention applied the 40th epoch for different corruption degrees in Figure and specifically for 0% corruption degree Figure 19
17

---

Published in Transactions on Machine Learning Research (03/2026)
CNN MNIST
CNN Fashion-MNIST
CNN CIFAR-10
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.4
0.4
0.4
Model Test Accuracy
0.2
0.2
04
0.2
0.6
0.
0.0
20
40
60
40
60
60
100
MLP MNIST
MLP CIFAR-10
1.0
1.0
0.6
0.6
0.4
0.4
0.0
0.2
0.2
40
60
Epoch
Figure Model test accuracy with true labels during training when a weight intervention that involves replacing pro-softmax weights with the VeLPIC vector, is performed at the 40th epoch and standard training is performed thereafler for 60 epochs. The corresponding test accuracies for the model with standard training (dotted) without intervention is overlaid for comparison. The results correspond to single run in each case.
As expected from results in Section 6.1 upon applying the intervention at the 40th epoch, there is typically rise in model generalization (i.e. test accuracy of the model). In many cases, this model generalization degrades subsequently over training, albeit not typically to the degree that model generalization would have degraded without the intervention; there are exceptions. One exception to this is MLP-CIFAR-10 at 80% corruption degree, wherein the model generalization is ultimately worse post-intervention than in the no-intervention case. On the contrary, CNN-MNIST and CNN--aashion-MNND intervention for low and modest corruption degrees. Also, e.g. with CNN-CIFAR-10, barring the transient rise in model generalization immediately after the intervention, the model generalization with without the intervention seem to follow largely similar trajectories. We also plotted accuracies when VeLPIC was applied at each epoch over training for models during the same interventions. Here again results are mixed and largely follow trends of the dynamics of model generalization, post-intervention. However, by-and-large, it appears that applying VeLPIC with standard training, and without intervention, on a suitably chosen layer, usually yields the best generalization at any epoch over training, in comparison to alternatives considered here.
show little degradation post-
10 Discussion
The notion of memorization, where deep networks are able to perfectly learn noisy training data at the ex- pense of gencralization has posed challenge to traditional notions of generalization from Statistical Learning Theory (Zhang et al.) 2017 2022). Our recent work (Ketha & Ramaswany 2026)) demonstrating improved latent generalization in such models a new development in our understanding of memorization and the nature of representations that drive it. Our goal here was take deeper dive into this phenomenon, to in- vestigate the origin and dynamics of latent generalization and examine the possibility of directly transferring it to the model. We showed that early-on in training, latent generalization and the model's generalization
18

---

Published in Transactions on Machine Learning Research (03/2026)
NUMICE GIV E3421200
KU-G 12366 COM 2531
NUMPCE SSTH OWSECE (290)
IOF CS 2511
1.0
63
3.0
03
34-
Vector Linear Probe Intermediatelln layeee-llyyrraee Classifter (VeLPIC) Test Accuracy
044
5.4
6.22
92
1.0
3.0
0.3
1.0
30
1.2
3.6
3.6
92
00
Epoch
Figure Vector Linear Probe Intermediate layer Classifier (VeLPICC test accuracy during training on models when intervention is performed 40th epoch and standard training is performed thereafter for 60 epochs. The VeLPIC test accuracy models trained with standard training (dotted) without intervention is overlaid for comparison. Model test accuracy for models trained with and without intervention is overlaid for comparison. The results correspond to a single run in each case.
closely follow each other in many cases, suggesting the possibility of common mechanisms that contribute to both. However, later in training, there is a divergence, with the model oflen retaining signíficant latent generalization ability in one more layers, while sacrificing overt model generalization Next, showed that MASC (Ketha & Ramaswamy] 2026 is quadratic classifier and built new lincar probe (VeLPIC). We found cases where this linear probe outperforms MASC and also found other cases, in particular, with ResNet-18, where MASC appears to be able to extract significantly more latent generaliza- tion from early layers than VeLPIC. We also ran comparisons of MASC and VeLPIC to a baseline linear logistic regression (LR) probe trained using standard techniques and find that MASC and VeLPIC outper- form the LR probe in many layers. We were also interested in examining if the latent generalization could readily be translated to model gencralization by directly editing model weights. We utilized the VeLPIC probe to derive a new set of model pre-softmax weights to make this so. We also briefly examined the effect of further training upon editing the weights in the 40th epoch, and find that this yields mixed results.
greater degree.
There are multiple limitations in this work. While we have carried out extensive experiments on number of models and datasets, how the phenomenon of latent generalization and its dynamics over training depends on model architecture and nature of the dataset isn't yet clear. Secondly, we have explored in detail the dynamics of MASC, when the class-conditioned subspaces explain 99% variance of training data. MASC performance for other class-conditioned subspaces remains be explored. Thirdly, an advantage of MASC lies in its ability to capture generalization when it occurs within subspace of dimension greater than one.
19

---

Published in Transactions on Machine Learning Research (03/2026)
This behavior is particularly evident in the early layers of ResNet-18, where MASC generalizes significantly better than both VeLPIC and the model. In contrast, VeLPIC provides a practical technique that its generalization performance at the final layer can be directly transferred to the model, an extension that is not yet available for MASC. Finally, we haven't explored the phenomenon of latent generalization in transformer models. Memorization has been reported Zhang et al. (2017) cases where input. is corrupted instead of class labels. This is a memorization setting that we have not studied here.
This work brings up multiple detailed mechanisms governing latent generalization during memorization remain to be investigated. It is also an open question, whether there exist other probes that can extract better latent generalization from layerwise representations, in comparison to MASC and VeLPIC. Next, it is unclear if latent generalization from representations of layers other than the last layer can be transferred towards model generalization. This can be useful to do, cases where early or middle layers exhibit better latent generalization than the last layer. Furthermore, what causes latent generalization during memorization to be or not be lincarly decodable is open question. To what exlent. does model architecture, e.g. ResNet-like vs. AlexNel- like influence the answer. Do need fundamentally different ways to remove effects of memorized labels in these differing network architecturos? Answering these questions might need a deeper understanding of representations used by deep networks during memorization. More generally, in light of these results, whether an understanding of generalization in the memorization regime can inform a better understanding of generalization for models trained with uncorrupted labels a worthwhile direction for future investigation.
directions for investigation. While we have made some progress, the
In closing, our results highlight the rich role of representations in driving generalization during memorization and how their understanding can be utilized, in many cases, to directly improve model generalization.
Acknowledgments
Simran Ketha was supported by an APPCAIR Fellowship, from the Amuradha & Prashanth Palakurthi Centre for Artificial Intelligence Research. The work was supported in part by an Additional Competitive Research Grant from BITS to Venkatakrishnan Ramaswamy. The authors acknowledge the computing time provided on the High Performance Computing facility, Sharanga, at the Birla Institute of Technology and Science Pilani, Hyderabad Campus.
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. 2018. arXiv preprint arX:::660.00000 2018.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference machine learning, pp. 233-242. PMLR, 2017.
Amr Bakry, Mohamed Elhoseiny, Tarek El-Gaaly, and Ahmed Elgammal. Digging deep into the layers of cnns: In search of how enns achieve view invariance. arXiv preprint arXiv:1508.01955 2015.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849- 15854, 2019.
N Alex Cayco-Gajic and R Angus Silver. Re-evaluating circuit mechanisms underlying pattern separation. Newron, 101(4):584 602, 2019.
Yue Chung, Daniel D Lee, and Haim Sompolinsky. Linear readout of object manifolds. Physical Review E, (3)6000033 2016.
Uri Cohen, Yeon Chung, Daniel Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):746, 2020.
Viv Cothey.
Web-crawling reliability. Journal of the American Society for Information Science and Tech- mology, 55(14):1228- 1238, 2004.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Koel Das, Sergey Osechinskiy, and Zoran Nenadic. A classwise pea-based recognition of neural data for brain-computer interfaces. In 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 6519 6522. IEEE, 2007.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
Matthew S Farrell, Stefano Recanatesi, Guillaume Lajoic, and Eric Shea-Brown. Dynamic compression and expansion in classifying recurrent network. bioRxiv, pp. 564476, 2019.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural nelworks and learning systems, 25(5):845 869, 2013.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias terms of optimization geometry. In International Conference on Machine Learning, pp. 1832 1841. PMLR, 2018.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama, Co-teaching: Robust training of deep neural networks with extremoly noisy labels. Advances in neural information processing systems, 31, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern vecogailion, pp. 770-778, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural langent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304 2313. PMLR, 2018
Simran Ketha and Venkatakrishnan Ramaswamy. Decoding generalization from memorization in deep neural networks. Trunsactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https:
//openreview.net.eee ptt.:/eeeee//oooom////e/eeeee
Simran Ketha, Nuthan Mummani, Niranjan Rajesh, and Venkatakrishnan Ramaswamy, Adversarially-rouus probes for deep networks. In NewrIPS 2025 Workshop: Reliable ML from Unreliable Data.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint orXiv: 11..44422.68888 2014.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519 3529. PMLR, 2019.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprint Xrx:::1899. 10374, 2018
Kuang- Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1444-5456 2018.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pp. 4313-4322 PMLR, 2020.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint or Xrxi 178.. 02862, 2017.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Semnandz--Grande Early-learning regular- isation prevents memorization of noisy labels. Advances in newral information processing systems, 33: 003312 2020.
Winter Mason and Siddharth Suri. Conducting behavioral research on amazon's mechanical turk. Behavior research methods, 44(1):1-23, 2012.
Grégoire Montavon, Mikio L Braun, and Klaus-Rober' Müller. Kernel analysis of deep networks. Journal of Machine Learning Research, 12(9). 2011.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances neural information processing systems, 31, 2018.
Mohammed Ali Moustafa. tiny-immi imagenet
Tiny imagenet, 2017.
URL httpes///aaglee.ccocoooottttttte
Gabricle Paolacci, Jesse Chandler, and Panagiotis G Ipcirotis. Running experiments on amazon mechanical turk. Judgment and Decision making, 5(5):411 419, 2010.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145-151, 1999.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
LLOYD RV. Observer variation in the diagnosis of follicular variant of papillary thyroid carcinoma. Am J Sury Pathol, 28:1336-1340, 2004.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning deep linear neural networks. arXiv preprint arXi::1312.612) 2013.
Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Conference learning theory, pp. 489 511. PMLR, 2013.
Hwanjun Song, Minscok Kim, and Jac-Gil Lec. Selfic: Refurbishing unclean samples for robust deep learning. PMLR, 2019.
In International conference on machine learning, pp.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey, IEEE transactions neural networks and learning systems, 34(11): 8135-8153, 2022.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasckar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1 57, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Cory Stephenson, Abhinav Ganesh, Yue Hui, Hanlin Tang, SueYeon Chung, et al. On the geometry of generalization and memorization deep neural networks. In International Conference on Leorning Rep- resentations, 2021.
David Sussillo and Larry F Abbott. Generating coherent patterns of activity from chaotic neural networks. Newron, 63(4):544 557, 2009.
Lina M Tran, Adam Santoro, Lulu Liu, Sheena A Josselyn, Blake A Richards, and Paul W Frankland. Adult neurogenesis acts neural regularizer. Proceedings of the National Academy of Sciences, 119(45): e2206704119. 2022.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial label flips attack on support vector machines. In ECAT 2012, pp. 870 875. IOS Press, 2012.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Rashion-mnist novel image dataset for benchmarking machine learning algorithms, 2017.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699, 2015.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in nearal information processing systems, 27, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. iclr 2017. arXiv preprint 1xXiv:661.00555 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study. Artificial intelligence review, 22(3):177-22) 210, 2004.
23

---

Published in Transactions on Machine Learning Research (03/2026)
A Model architectures and training details
MLP Model. The MLP architecture consists of four hidden layers with 128, 512, 2048, and 2048 units, respectively. Each layer is followed by a ReLU activation, and a soflmar layer is used for classification. Models were trained SGD Qian 1999) with a learning rate of 1 x 10-3 and momentum 0.9. A batch size of 32 was used across all experiments. Input dataset was normalized by dividing pixel values by 255.
composed of three convolutional blocks, each containing two convolu- tional layers followed by a max pooling layer. The convolutional layers use 16, 32, and 64 filters, respectively, with kernel size 3 3 and stride 1. The max pooling layers have kernel size of 2 2 and stride 1. These blocks are followed by three fully connected layers with 250 units each. ReLU activation is used after all layers except pooling, and softmax is used at the output for classification. The CNN was trained using Adam optimizer Kingma (2014) with a lcarning rate of 0.0002. For MNIST and Fashion-MNIST, a batch size of 32 was used, while for CIFAR-10, a batch size of 128 was used. Input data was normalized by subtracting the mean and dividing by the standard deviation of each channel.
CNN Model. The CNN mode
ResNet-18 was slightly adapted for CIFAR-10 dataset and trained using SGD (learning rate=0.001, momen- (um=0.9) with batch size of 32. Inpuls were normalized using channel-wise mean and standard deviation. We analyze six layers: LO-L4, and the average-pooling (avg_poo) layer. LO denotes the activation immedi- ately before the first residual block, while L1 L4 correspond to the outputs of successive residual blocks.
The experiments were conducted on servers and workstations equipped with NVIDIA GeForce RTX 3080, RTX 3090, Tesla V100, and Tesla A100 GPUs. The server runs on Rocky Linux 8.10 (Green Obsidian), while the workstation uses Ubuntu 20.04.3 LTS. Memory requirements varied depending on the specific experiments and models. All model implementations were developed in Python using the PyTorch library, with torch.manul seed set to 42 to ensure reproducibility. Accuracy served as the primary evaluation metric throughout this work.
B Minimum Angle Subspace Classifier (MASC)
We summarize below the Minimum Angle Subspace Classifier (MASC) from (Ketha & Ramaswamy 2026 in order to keep the exposition here largely self-contained.
For a given deep network, MASC leverages the class-specific geometric structure of network's latent repre- sentations. For an input data point æ, let its activation vector at layer be denoted by ær. The objective is to classify æt by leveraging a set of
subspaces, {S, ) estimated from a training dataset To predict the class label y(œz), MASC Algorithm 2 (reproduced verbatim from (Ketha)
D { v:) )
& Ramaswany 20260) assigns 0t to the class whose training subspace forms the smallest angle with it.
The class-conditional subspaces {S*] are estimated from the training dataset D = {(x,( y)) where each æ; E Rd is paired with a label yi E (C+) For a given layer I, those subspaccs are constructed following Algorithms 3 and 4 (reproduced verbatim from (Ketha & Ramaswany (026))) In practice, each subspace Sg represented by its principal components, which provide a compact basis for capturing the underlying class-condittonal structure.
19 convolution network were implemented following design principals outlined in
Tran al.
2022
24

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 2 Minimum Angle Subspace Classifier (MASC) (reproduced verbatim from (Ketha &
Ramassamy 2026))
Input: Training subspaces {Sk 1 layer output data point æι from layer when input is passed through the network and classes {Ca (k
Output: MASC prediction class label ((r) according to layer L.
for each class C₁ do
3itk compute the projection of ay onto subspace Sk-
Compute the angle ((1, Tike) between æ and Ztk
end for
5: Assign the label y(ι)) Cj where is = arg ming ((1 Tbk)
6: Return: label y(11)
Algorithm 3 Subspaces Estimator for MASC
(reproduced verbatim from (Ketha & Ramaswamy 2026
Input: Training dataset D( Wi) where each 2; E Rd and E {C are input-label pairs, neural network, and layer I. Output: Subspaces { for classes K and given layer
=
2: for each input pair (æq, Mi) in D do
3: Pass ay through the network layers to obtain the output of layer I, denoted as æ E RRdd
4: DiU ( y/))
end for
Estimated subspaces [5 1
PCA-Based Subspace Estimation(D:)
Return: Subspaces {Sx 1==
Algorithm 4 PCA-Based Subspace Estimation
(reproduced verbatim from (Ketha & Ramnaswayy 2026))
Input: Layer output D; {(x,, w)) where I4 Rd and y E (C+)
Output: Subspaces { for classes K.
1: Dnow + Di
2: for each (Iig E Di do
3: Dnew + Deew ((-x,, y:)}
end for
for each A: (1,...,K K} do
Extract the subset of data Dnew.k (x0 y1 C
S₁ :cCA((Deew.))
8: end for
9: Return: Subspaces K
25

---

Published in Transactions on Machine Learning Research (03/2026)
B.1 Proof of equivalence of MASC and classifier used in Proposition 1 of Section 5
In Proposition of Section different formulation of MASC in Proposition 1 Here, we show that, that formulation is equivalent to the formulation of MASC from (Ketha & Ramaswamy 2026).
show that MASC is quadratic classifier. However, we used slightly
Let 21 denote the output of the layer I of the deep network when it given input æ. Let p .s..... an orthonormal basis 20 of the subspace Se corresponding class c. Let of be the projection i on Se. We have
............ Ph be
of (m pi) pi + ... + (ar pie) Pk
(4)
zi
(5)
MASC MASC(Ketha Ketha
Ramaswamy 2026) on layer I predicts the label æ as
arg max (8).22)
(6)
The formulation in Proposition of Scction 5 predicts the label of a as
arg max (211-08)
(7)
Proposition 2. arg max, (5))
arg max, (2))
Proof. As pi,.... Pi are orthonormal, therefore
ı V(a ps)2 + +... + (on p
(8)
After expanding the dot product and noting that /pil..... lpAl = 1, we have
b b1 cos? or +... cos2 05
(9)
where of is the angle between and pf:
Ketha Ramaswamy 2026) on layer / predicts the label of æ as
(x) pf) ..... + (x) p))
arg max ( = arg max
(10)
[ x
We now substitute g in 10
((1) (x) 2 +... ...... + (x) 2) |mall |a coos 0f +... cos2 A
arg max
(11)
1/f = 1, we have After expanding the numerator and noting that (pf),
| cos2 0f
x cos? 2 0 + cos 0
arg max
(12)
| 2 cos"
0 c
which typically estimated PCA, where the number of principal components.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Simplifying, now have
arg max (f).0f)) arg max (Veus? 0f + ... cos? 0g
(13)
Now,
arg max (2g a() arg max ((ag p9)2 +...4 (a) - pp))))
(14)
After expanding and noting that lpfl...., I = 1, we have
arg max (|ar) cos? of +... | cos? 0g)
(15)
| is taken common and is not dependent on due to which, equivalently, we have
arg max (x). = arg max ((cos? 0f + ... + cos? 0g))
(16)
From 13 and 16 it follows that
arg max (6..0))) arg max (2)
(17)
C Training dynamics of latent generalization using MASC
MASC testing accuracy during training with 0% and 100% corruption degrees are shown in Figure 10
Training dynamics of the linear probe: VeLPIC
A linear probe VeLPIC test accuracy during training for all models with 0% and 100% corruption degrees are shown in Figure
D.1 Difference between VeLPIC and MASC
Here, we present the difference between test accuracy of VeLPIC and MASC during training and for differ- ent layer of the networks. For MLP-MNIIT.MMLPIIIEEEE CIFAR-10, AlexNet-Tiny ImageNet and ResNet-18-CTFAR-10,, these results for 0% and 100% corruption degrees are shown in Figure 12
CNN-MNIST, CNN-Fashion-MNIIS CNN-
E
Additional results with linear probe (logistic regression)
For all models with 0% corruption degree, the test accuracy of the logistic regression probe during training of the memorized models, are shown in Figure 13 The results comparing the performance of logistic regression
probe with MASC probe is shown in Figures 14 and with VeLPIC probe is shown Figures 15
F Transferring latent generalization to model generalization
For all models with 0% and 100% corruption degroes, model test accuracy during training when we replace the pre-softmax weights with VeLPIC vectors are shown in Figure 16 Model corrupted training accuracy for few modelss-lataset-commmppp
plotted in Figure 17 and Figure
27

---

Published in Transactions on Machine Learning Research (03/2026)
-.. Model test acturacy
MLP-FC3 (2048y CNN-FG2 (2501) AlexNet-FCC (40P6)/ Restvet-18-1 (8192)
MLP-FC4 (2048Y CNN-FC3 (250M ReaNNet-18-L3 (4096)
MLP-FC1 (128)/ OWN flat 1576/1024/ Alexhlet-Fid Fiat 12561/ RedVet: 18-L0 (16384)
ResNict 18:L4 (2048)
MLP.FC2 (512)/ CVN-FC1 (250)/ AlexNet (4036)/ ResNet--8 18 (16384)
RestNet-18-Avg pool (512)
0% corruption degree
100% corruption degree
1.0
1.0
0.4
0.2
0.0
0.0
1.0
1.0
MIACIPAR110
0.8
0.8
0.6
0.0
CONNNIST
0.8
0.8
0.6
0.6
0.4
0.4
MASC Test Accuracy
0.2
391
OWN Fadiionioonnoo WNISMNISTT
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
120
340
400
465
1.0
1.0
OWN DWUCIFFRR100
0.4
0.2
0.0
0.0
BesNe-18.C 10.118.PP55 CIFAR80 Alattel Thny Imagetet
0.006
0.10
0.004
0.003
0.00
0.8
0.6
0.4
0.2
Epoch
Figure 10: Minimum Angle Subspace Classifier (MASC) Test accuracy for 0% and 100% corruption degrees during training of the network, where test data is projected onto class-specific subspaces constructed from training data with the indicated label corruption degrees. The plots display MASC accuracy across different layers of the network for various model-ddtasee combinations. For reference, the test accuracy of the models (dotted line) is also shown. Each row corresponds to a specific corruption degree, while columns represent different models, labeled. FC denotes fully connected layers with ReLU activation, and Flat refers to the flatten layer without ReLU.
G Experiment results with weight intervention
Here, we present results of the latent generalization of VeLPIC evolves across different layers during training for models trained with 0% corruption degree. The results for intervention performed at the 40th epoch is shown in Figure 1g
28

---

Published in Transactions on Machine Learning Research (03/2026)
-.. Model test acturacy
MLP-FC3 (2048y CNN-FG2 (2501) AlexNet-FCC (40P6)/ Resivet-184 18-11 (8192)
MLP-FC4 (2048Y CNN-FC3 (250M ReaNNet-18-L3 (4096)
MLP-FC1 (128)) OWN flat 1576/1024/ Alexhlet-Fid Fiat 12561/ RedWet 18-L0 (16384)
ResNict 18:L4 (2048)
MLP.FC2 (512)/ CWN-FC1 (250)/ AlexNet (4036)/ ResNet- 1a-eet-1--112 11 (16384)
RestNet-18-Avg pool (512)
0% corruption degree
100% corruption degree
1.0
1.0
0.4
0.2
0.0
0.0
1.0
1.0
MIACIPAR11
0.8
0.8
0.6
0.0
CONNNIST
0.8
0.8
0.6
0.6
VeLPIC Test Accuracy
0.4
0.4
0.2
391
OWN Fastion WNISMNIST
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
220
340
465
1.0
1.0
OWN DHUCIFAR-10
0.4
0.2
0.0
0.0
ResNe-18.c 10.118.PPP CB-AR8 Alattel Thny Imaqetet
0.15
0.006
0.10
0.00
0.002
0.8
0.6
0.4
0.2
Epoch
Figure 11: Vector Linear Probe Intermediate-laver Classifier (VeLPIC) test accuracy for 0% and 100% corruption degrees during training of the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display VeLPIC accuracy across different layers of the network for various model-dataser combinations. For reference, the test accuracy of the models (blue dotted line) over epochs training is also shown. FC denotes fully connected layers with ReLU activation, and Flat refers to the flatten layer without ReLU.
H Comparison between randomly initialized and trained model
We compare the test accuracy of the model, MASC, and VeLPIC for both randomly initialized and trained models across varying corruption degroes. Results for all model-datasct pairs are presented in Figure 20 For MASC and VeLPIC, report accuracies corresponding the best-performing layer, defined as the layer achieving the highest MASC VeI.PIC test accuracy for each model-datased pair.
29

---

Published in Transactions on Machine Learning Research (03/2026)
Medel test accuracy
MLP-CI 12048y CMN-TC2 (250V) AlexNet- FC2 14090) ResNet- 18-LL (8192)
MLP-FG4 (2048y CNN-FC3 ResNet-1B-LL3
(128y CMN-Flat (S76/1 (2/4// Alezteet-Fiat 12566) ResNett-8:ll BoLD (16384)
Res1 (2048)
MLP-FC2 1512 CMN 12501/ MlssNet-rC1 C1 (4096)/ Resflet-19-11 (16.884)
ResWett 18 18-Avg_pooo (5112)
0% corruption degree
100% corruption degree
0.00
0.06
-0.05
0.04
0.10
0.02
0.00
180
24
289
0.10
0.02
0.05
0.00
-0.10
Difference in Test Accuracy Vc Acc. MASC Acc.)
131
311
MNNSSSTT
0.025
0.000
0.10
-0.025
-0.050
0.05
-0.075
0.050
0.10
0.025
-0.025
0.00
CMN
10
19
OWN GWN-CIFAR.30
0.2
0.02
0.1
0.0
303
329
320
ResNer--1-CIFAM.1 Aladta:Tiny inapetet
0.06
0.94
0.001
0.02
0.00
0.001
0.02
320
0.1
-0.1
-02
Epoch
Figure 12: Difference test accuracy (VeLPIC Accuracy MASC Accuracy) during training the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be O.
At lower corruption degrees, we observe consistent trend across all model-datasot combinations: MASC and VeLPIC evaluated on trained models outperform their counterparts evaluated on randomly initialized models when measured at the respective best. layers.
At higher corruption degrees, however, a small number of exceptions emerge. Specifically, for MASC, performance on randomly initialized models exceeds that on trained models in the following cases: AlexNet- Tiny ImageNet at 60% corruption, and MIPP-MNIST, MIPP-IFAAR-10 and AlexNet- ImageNet at 80%
30

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy
MUP-FC4 (2048)/ CNN-FC3 1250// Resivet-18-1)
MUP-FC1
MLP-FC2 (532V CNN-FC) 1250// AlexNat-FC1 (4096) ResNet-18-111
ResNet-18-44 Baattt---Avggpoo
MUP-FC3 (2048) CNN-FC2 Q500 MexxNett-F2 (4096) ResNet-18-122
0% corruption degree
1.0
MLP MISST
0.8
0.6
MLP-0T-8:10
0.6
0.4
0.
0.0
Logistic Regression Probe Test Accuracy
1.0
CNN MNNS
CNK: Feshion MNST
0.8
0.6-
0.4
0.2
36p
1.0
CNN car A9:10
0.0
Alette-Tio Tiny Imageeeet
0.10
0.05
0.00
Restee- 10 00
0.4
0.2
Epoch
Figure 13: Logistic regression probe's test accuracy over epochs of training for multiple models/datasets. The plots display logistic regression probe's accuracy across different layers of the network. For reference, the evolution of Lest. accuracy of the corresponding model (blue dotted line) over epochs of training also shown.
corruption. In contrast, VeLPIC exhibits a single exception. At 80% corruption on AlexNet-Tiny ImageNet, VeI.PIC evaluated the randomly initialized model outperforms the trained model.
31

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy
MLP-F44 (204BV CNN-FCJ (2501/ ResNet-1B-L3
MLP-FCL (128M CNN- 1576/1024) AlcxNet-Fiat (256)/ ResNet-1B
MLP-FC2 (51ZM FIN 12501/ Alexhiet FC1 (4090) ResNet- 18-11
ResNet 18-Avg_pod
MLP-FC3 (2048) ENN-FC2 (250H AlexNot (4096)/ Res
0% corruption degree
-0.025
-0.000
-0. 100
-0.125
0.00
Alnc Difference Test Accuracy (MASC Logistic regression
CWN-NNST
0.0
CNN Fashion MNIST
0.4
0.0
0.0
-0.1
-02
320
0.000
-0.025
00.075
0.1
-01
-02
Epoch
Figure 14: Difference in test accuracy (MASC Accuracy- Logistic regression probe Accuracy) during training of the network, where for MASC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference accuracy across different layers of the network for various model-dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be
Impact of Dropout as a Regularizer
We conducted some preliminary experiments to study the effect of dropout (Srivastava et al. 2014) as a regularization technique on latent generalization during memorization. Both the CNN and ResNet- models were slightly modified incorporate dropout layers. For dropout version of CNN models, a dropout layer (p=0.2) used after every fully connected layer. The CNN models trained MNIST, Fashion-
32

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy
MLP-F44 (204B) CNN-FCJ (2501/ ResNet-1B-L3
MLP-FCL (128M ONN 1576/1024) AlcxNet-Fiat Flat (256)/ ResNet-18444
MLP-FC2 (51ZM OVN-FCI 12501/ Alexhiet FCI (4090) ResNet- 18-11
ResNet 18-Avg_pod
MLP-FC3 (204B) ENN-FC2 (250H AlexNot (4096)/ Res
0% corruption degree
0.00
-0.025
-0050
-0075
-0.000
-0.125
CNNANNST
0.4
Difference Test Accuracy (VeLPIC AcC. Logistic
0.0
CNN Fashion---iee MNIS
0.4
0.0
0.0
-0.1
10-AIAR-110 Imagetet
0.02
0.00
-0011
0.00
0.15
025
Epoch
Figure 15: Difference in test accuracy (VELPIC Accuracy Logistic regression probe Accuracy) during training of the network, where for VELPIC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference accuracy across different layers of the network for various model-dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be
MNIST, and CIFAR-10. For dropout version of ResNet-18, dropout layer (p-0.2) was added before the final classificattin layer.
MASC test accuracy on models trainod with and without dropout layers are shown in Figure 21 and 22 For models trained with dropout (WD), we plot the results after the dropout layer. For models trained with dropout, the MASC dynamics remain largely similar those of models trained without dropout. However,
33

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Weight transfer using
0% corruption degree
100% corruption degree
120
J88
MIPCIAAR
CUNMNST
0.4
Model Test Accuracy
CNN Feshiiiiinnnn MNIST
9.8
0.4
0.4
02
0.0
12
0.0
1.0
1.0
CWN CITAR 10
0.8
0.8
0.6
0.6
0.4-
0.4
0.2
02
0.0
0.0
Resher 1eshee:.-CCAAe CI-AAA0:1 AlesNe--Ting Imageeet
0.007
0.10
0.005
0.004
0.05
0.00
303
430
1.0
0.8
0.6
0.4
0.2
0.0
Epoch
Figure 16: Comparing model test accuracy with VeLPIC transferred accuracy when the weight intervention is applied to the model at the epoch in question during training for corruption degrees 0% and 100%. The test accuracy of the model with standard training without weight intervention (blue dolted line) is overlaid for comparison.
in the later layers of CNN-WD-Fashion-NNIST and CNN-WD-MNIST, we observe a noticeable deviation, where the layer-wise behavior no longer aligns with the model's test accuracy in the early epochs of training: 2a
interestingly the corresponding VeLPIC plots Figure 23 do not show this deviation. In most cases, MASC performance shows an initial decline followed by rise toward peak. In contrast, for ResNet-18 models, significant differences observed between the dropout and non-dropout models.
34

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Standard
Model: Weight transfer using VeLMC
MLP-MNIST
CNN-Fashhion
0.6-
04
0.4
0%
12
1.
1.0
20% coppetee
0.4
0.2
1.0
Model corrupted training accuracy
40h conuutio 56
0.8-
0.2
0.2
0.0
60% ciquttie degee
0.
0.5
0.6
0.0
0.0
0166 commotaaoooo 040
0.8
0.
0.5
0.4
18
16
80
80
1.0
1.0
0.6
100% cap
0.4
0.4
Epoch
Figure 17: Model train accuracy corrupted dataset when the VeLPIC weight intervention applied to pre-softmax weights at the epoch in question during training. The training accuracy corrupted dataset of the model with standard training without weight intervention (blue dotted line) is overlaid for comparison. Observe that, except for 100% corruption degree, the transferred training accuracy tends to saturate at a level largely consistent with the fraction of true training labels in the corrupted dataset.
VeLPIC test accuracy on models trained with and without dropout layers are shown in Figure 23 and 24 There no significant difference in the dynamics of VeLPIC performance between models trained with dropout and those trained without dropout.
35

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Standard
Model: Weight transfer using VeLPIC
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10 10
1.0
1.0
1.0
0%% cortuption degree
0.8
0.8
0.8
0.6
0.6
0.6
0.4
18
100
131
20% comuption degiee
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
189
0.0
1.0
1.0
1.0
Model corrupted training accuracy
60% comppttinn degiee
0.8
0.8
0.8-
0.6
46
80
380
100
90
139
36
200
24
286
306
400
40
40c
130
180
200
261
300
390
60% comppttin degree
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
1.0
1.0
1.0
80% cotruption degree
0.8
0.8
0.8
0.6
0.6
0.4
66
116
26
48
80
120
168
200
290
284
104
400
443
120
180
230
245
289
300
390
465
100% corruption degree
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Epoch
Figure 18: Model train accuracy on corrupted dataset when the VeLPIC weight intervention applied to pre-softmax weights at the epoch question during training. The training accuracy on corrupted dataset of the model with standard training without weight intervention (blue dotted line) is overlaid for comparison.
36

---

Published in Transactions on Machine Learning Research (03/2026)
RTS FIIFFE 1291
MEP (2500) 239)
VeLPIC Test Accuracy
HUP MLP-CIFAR-10
CNK-MNIST
ChN--F San--aashoo--M5555
CHN-DFAR-10
22
Epoch
Figure 19: Vector Linear Probe Intermediate layer Classifier (VeLPIC) test accuracy during training
models when intervention is performed at 40th epoch and standard training is performed thercafter for 60 epochs for 0.0 corruption degree. The VeLPIC test accuracy on models trained with standard training (dotted) without intervention is overlaid for comparison. The results correspond to a single run in each case.
Random nitiolized Mood
Trained Modd
MASC intiaaizedd Model
MAEC Trained Modal
VOLPIC on iniiaizedd Modal
VeLPIC Trained Modal
corruption
corruption degree
corruption degree
1.0
62-
Test Accuracy
MLP-ONST
MLP OFMILL
Chivaniss
Cheli ChM--CPAR-1
Chilk Fashion
Aboutes Thyy imunedte
Rodket- AN-10
corruptionn degree
comruption degree
1.0
0.0
0.2
MUPMINSTT
M.P.CFARLL
ChN MNNST
3IN-CFAA-11
CoN- Pestoos
Aln Tryy Imagetes
Reciett Sectiet-/--OFAA 1B-OFA4-110
MLP MLPAINIST
MUP W.P-UFAS-10
CNN-MAUST
CHN-CIFAR
Pachenn-siusst
Aindivel Imagote Reciea J.-CAAR
Models-datasee
Figure 20: Model, MASC and VeLPIC test accuracy for randomly initialized and trained models across different corruption degrees and model-dataset pairs. For MASC and VeLPIC, results are reported for the best performing layer, defined as the layer achieving the highest MASC/VeLPPC test accuracy for the corresponding model dataset pair. The test accuracies are averaged over three runs.
37

---

Published in Transactions on Machine Learning Research (03/2026)
CIN-FCC--Drppeui
CMN Flat (576/1024)
CNNFF33 (250W CNN-FC3 Dropout/ 11--18 (4096)
ResNet-- Drapout
CNN FCN 1250/ CNN-FCI-Drooout
ResNet-1B-44 12048)
20% cornuption degree
40% computtion degree
60% corruption degroe
80% computtion degree
02
02
13
38
04-
24
06-
MASC Test Accuracy
369
130
90
18
74
680
144
30
106
400
180
240
06
CHRINO
1.2
120
14
20
185
468
480
14
290
480
140
240
241
499
WRIST
06-
1.-
3.
1.
28c
120
14
420
120
384
493
82
Epoch
Figure 21: Minimum Angle Subspace Classifier (MASC) test accuracy over epochs of training for RosNet- 18 and CNN models having trained with and with dropout, where test data is projected onto class-specific subspaces constructed at each epoch from corrupted training data with the indicated label corruption degree. The plots display MASC accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training is also shown. WD corresponding to models trained with dropout.
38

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy CNN--flt (57N1024)
CNN--F2 1250y CMN-FC2-Dropout RestNes-8-422 (8192)
monllet-18.. 1a-ivtt--1ggoea pool (512)
CNN--G3 1250Y ChRreCC3-rropou Restett-B--L3 (4091)
Resf et 18--1---0000 18-Dropout
FCI (250)) CNN Dropout
(2048)
0% corruption degree
MASC Test Accuracy
150
20
18
4
Epoch
Figure 22: Minimum Angle Subspace Classifier (MASC) test accuracy for 0% corruption degrees over epochs of training for ResNet-18 and CNN models having with and with dropout, where test data is projected onto class-specific subspaces constructed at each epoch from corrupted training data. The plots display MASC accuracy across different layers of the network. For reference, the evolution of accuracy of the corresponding model (blue dotted line) over epochs of training is also shown. WD corresponding to models trained with dropout.
39

---

Published in Transactions on Machine Learning Research (03/2026)
CIN-FCC--Drppeui
CMN Flat (576/1024)
CNNFF33 (250W CNN-FC3 Dropout/ 11--18 (4096)
ResNet-- Drapout
FCN 1250/ CNN-FC1-Drooout
ResNet-1B-44 12048)
20% corruption degree
40% computtion degree
60% corruption degroe
80% computtion degree
02
02
13
38
04-
24
VeLPIC Test Accuracy
369
130
80
18
74
680
144
30
400
780
240
06
CAR
1.5
ONN Fushio
120
14
24
488
480
144
290
365
480
140
240
499
WRSST
20-
06-
16
1.-
1.
1.
120
14
420
926
144 493
82
Epoch
Figure 23: Vector Linear Probe Intermediate-layer Classifier (VeLPIC) test accuracy during training of ResNet-18 and CNN models with and with dropout, where test data is projected onto class vectors con- structed at each epoch from training data with the indicated label corruption degrees. The plots display VeLPIC accuracy across different layers of the network for various model-dataset combinations. For ref- erence, the test accuracy of the models (blue dotted line) over epochs of training is also shown. WD corresponding to models trained with dropout.
40

---

Published in Transactions on Machine Learning Research (03/2026)
Model test accuracy CNN--fat (57N1024)
CNN--F2 1250y CMN-FC2-Dropout RestNes-8-422 (8192)
monllet-18.. Ma-evtt----ggeaa pool (512)
CNN--G3 1250Y ChWreCC3rrropuu Resteet-B--L3 (4091)
Resf et 18-Dropout
FCI (250)) CNN Dropout
(2048)
0% corruption degree
VeLPIC Test Accuracy
150
24
20
18
Epoch
Figure 24: Vector Linear Probe Intermediate--aver Classifier (VeLPIC) test accuracy for 0% corruption degrees during training of ResNet-18 and CNN models with and with dropout, where lest data is projected onto class vectors constructed at each epoch from training data. The plots display VeLPIC accuracy across different layers of the network for various model-datasct combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training also shown. WD corresponding to models trained with dropout.
41


Reviews:
[{"review_id": "k4Plxme4N1", "reviewer": "Reviewer_8di8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the topic of memorization -- it is focused around the data representations built by the networks which are trained on the partially corrupted datasets (part of the labels during training are shuffled). The authors confirm previous results [1] showing that despite poor (model) generalization when trained on corrupted data, the model still learns well-generalizing representations at earlier layers. The phenomenon called latent generalization is analysed using the probing technique (MASC) which is later developed by the authors and used for further experiments (VeLPIC).\n\nThe authors have three major contributions:\n1. Confirm the results from previous work.\n2. Develop a simplified probing mechanism VeLPIC that results in comparable or better generalization than MASC.\n3. Based on VeLPIC, the authors propose a mechanism for improving the model's generalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments are clearly introduced and explained, the results confirm the findings of the authors. With that being said, **I would like to encourage the authors to broaden the scope of the experiments.** While I understand that the authors tried to recreate the experimental setup from the work [1], I'm afraid that the results might not hold on different architectures and datasets.", "audience_yn": "Yes", "audience_explanation": "The topic of generalization/memorization is still an unsolved problem in deep learning community and the work offers an interesting perspective on the topic which can be further improved by the authors with additonal results and increased clarity of the text.", "requested_changes": "1. Editorial changes -- please consider rewriting abstract and contributions part (make them shorther!) You may consider this piece for an inspiration: https://www.easterbrook.ca/steve/2010/01/how-to-write-a-scientific-abstract-in-six-easy-steps/\n2. Broader set of experiments -- as mentioned earlier, it would strengthen the work substantially.\n3. Further experiments:\n     3.1. Adding an analysis with the Linear Probes trained with Cross Entropy would be an informative experiment for the readers. \n     3.2. Using VeLPIC vectors for pre-softmax layer as a warm-start to further training. I.e. Use these vectors as a trainable parameters and see if the model degrades the generalization or somehow the bias is strong enough so that the model keeps it?\n4. Clarifications:\n    4.1. I'm quite confused with the VeLPIC mechanism. As far as I understand, you compute $\\mathcal{P}_m$ and $T_m$ on the **train dataset** according to the class-split with random labels? And then build $\\mathcal{V}_m$ based on $\\mathcal{P}_m$ and $T_m$ and use it to assign the label for the samples from the **test dataset** which have correct labeling. Then I'm not really sure how this should work. Imagine that the training dataset is 100% corrupted, but in a specific way where all the samples with true label 0 shift to label 1, all with label 1 shift to label 2 and so on. So in some sense, the training dataset works as a normal dataset with labels shifted by one. Do I understand correctly that given a test image with correct label 0, using the VeLPIC method we most probably would see it assigning label 1 due to the shift in the training data or am I missing something?\n\n[1] https://arxiv.org/pdf/2501.14687", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0c0UaH4btT", "reviewer": "Reviewer_8TP4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the phenomenon of \"latent generalization\" within neural networks that have achieved high training accuracy on data with corrupted labels. The authors look into a few fundamental questions: the origin of this latent knowledge, the architecture of \"probes\" capable of extracting that knowledge, and the feasibility of \"repairing\" a corrupted model via weight transfer . Key findings include a demonstration that latent generalization is linearly decodable via a new probe (VeLPIC) and that replacing the pre-softmax weights of a model with these probe vectors can significantly recover test performance without additional training.\n\nStrengths\n\nS1: Geometrical perspective: The proof that MASC is a quadratic form serves as a good starting point to pivot toward VeLPIC, demonstrating that the latent \"generalizable\" signals in deep networks are not only extractable but are actually linearly decodable.\n\nS2: Practical repair mechanism: Weight transfer offers a concrete mechanism for model recovery that requires no further training. By replacing the final layer's weights with vectors derived from latent probes, the authors show that a \"memorized\" model can be immediately repaired to utilize its internal representations.\n\nS3: Dynamics analysis: The paper moves beyond static analysis by tracking the evolution of latent generalization throughout the training process. The authors provide an empirical study of the \"early learning\" window and the temporal point \nwhere internal features and output heads diverge.\n\nWeaknesses\n\nExperimental\n\nW1: Limited models: The authors perform experiments on a few simple architectures including small MLPs and CNN, and AlexNet. At this point in time, this cannot be considered sufficient coverage. I understand that access to advanced hardware may at times be limited, but looking into latent learning in ResNets/ DenseNets, and perhaps small ViTs should be quite possible. \n\nW2: Limited tasks/ domains: The authors consider MNIST, CIFAR-10 and Tiny ImageNet. As with W2 - this is simply not enough to draw conclusions. Are the observed phenomena general - or a quirk of small models on small visual classification tasks? How would the dynamics behave in other visual domains? NLP? Regression tasks? \n\nW3: Limited questions: Precisely because generalization is such an interesting topic, there is much that one would wish to see explored here. What happens as we move between layers (and here eg ResNets would be a great candidate)? What about noise other than corrupted labels? Does dropout impact the findings? Does an auxiliary semi/self-supervised mechanism? Are there forms of regularization that encourage latent generalization and make it more \"retrievable\"? Obviously any given paper cannot explore all the questions one might ask - but the list of questions that do get explored in this work really could be enriched.\n\nW4: Limited benchmarks: The authors prove VeLPIC's efficacy mostly by comparing it to MASC. This is not entirely compelling - one would wish to see comparisons to alternative common approaches. There are entire multiple bodies of literature - masked autoencoders, semi-supervised learning, feature robustness, adversarial learning, and much more. MASC is not a widely used \"gold standard benchmark\", so just beating it cannot be considered strong evidence.\n\nTheoretical\n\nW5: Related literature: The paper does not engage with numerous fields and approaches that could shed much light on latent generalization. To name a few - PAC-Bayes theory could contribute to the discussion of linear vs quadratic classifiers, as well as clarify issues pertaining to sharp minima (for instance, does the proposed repair mechanism \"move\" the model into flatter, more robust areas in hypothesis space?). Work in gradient dynamics and implicit bias (e.g. [1], [2]) surely is relevant to the learning dynamics the paper explores. The Neural Tangent Kernel framework ([3]  seems like a good fit for the authors' kernel-based approach. Double descent ([4]) has findings that similarly challenge the \"naive narrative\" of classical ML. Learning with noisy labels (see [5] for a survey) cannot be ignored in this context. I want to stress here that I'm not merely complaining about the paper missing this or that favorite reference - ideally the paper would explore potential connections with some of the sub-fields above (or others) much more vigorously. \n\nW6: Mathematical rigor: The paper is light on formal definitions and statements. To give a prominent example, the paper does not give a clear definition of latent generalization. When analyzing these phenomena, one should distinguish between the test-training performance gap (sometimes called the generalization gap), just the test error itself, and notions like the ability of the learned features to perform downstream tasks. All of those might be reasonably called generalization, and the claims sound different in each of these different contexts. Careful mathematical notation would be helpful to disambiguate.\n\nW7: Mathematical \"ambition\": Viewed as a theoretical contribution, the paper does not attempt to establish any independently significant theoretical claims. The proof that MASC is quadratic is solid, but seems highly similar to existing SVM theory and QDA. \n\nStylistic\n\nW8: Redundancy and imbalanced length: The abstract is substantially longer than is common, and importantly, contains much that is repeated (almost verbatim) in the introduction and discussion. \n\nW9:MASC vs VeLPIC: Given that a key message is simply that VeLPIC is better than MASC, the amount of \"real estate\" given to MASC in the main body could perhaps be reduced. This include the proof MASC is quadratic (could be simply stated, with the proof relegated to the appendix) as well as the experimental setup (the MASC paper already provided this information, and is extensively cited and referred to in the current paper, and this is discussed in the appendix anyway).\n\nW10: Motivation: the problem of learning with corrupted labels is presented \"as-is\", with no motivation, no discussion of real-life scenarios and the error regime that characterizes them, adversarial poisoning etc.\n\nW11: Tone: On occasion, the paper uses language that this reviewer found too \"dramatic\" and rich in superlatives. Examples: MASC's \"extraordinary ability\" and the overstated (in my personal belief) emphasis on the surprising nature of latent generalization. It may or may not have been deeply surprising in 2017-18 (though anecdotally and personally I was deeply interested in such topics precisely then, and do not find the paper's description of the community consensus fully accurate), but since then we have had many results in that vein. \n\nReferences: \n\n[1] - Soudry et al., 2018, The Implicit Bias of Gradient Descent on Separable Data\n\n[2] - Gunasekar et al., 2018, Characterizing Implicit Bias in Terms of Optimization Geometry\n\n[3] - Jacot et al., 2018, Neural Tangent Kernel: Convergence and Generalization in Neural Networks\n\n[4] - Belkin et al., 2018, Reconciling modern machine learning practice and the bias-variance trade-off\n\n[5] - Song et al., 2020, Learning from Noisy Labels with Deep Neural Networks: A Survey", "claims_and_evidence_yn": "No", "claims_explanation": "As detailed above, the experimental evidence is limited in a number of ways (overly simple and outdated models, a narrow range of domains, a restricted set of investigated questions). The proposed repair method is not benchmarked against strong literature baselines. The engagement with most of the relevant up-to-date research literature that could corroborate the paper's approach is minimal. Mathematical work that would lend credence to the generality of the findings is missing.", "audience_yn": "No", "audience_explanation": "Readers who want to see strong empirical performance will need much more extensive experimental results. The more theoretically inclined members of the audience will not see a sufficiently rigorous or mathematically sophisticated investigation. Those who want to learn about the domain will not get a clear view of the state of literature. The most pragmatic readers in search of solutions to real-life problems will not see a discussion of noisy labels - or strong evidence that VeLPIC should be in their toolbox.", "requested_changes": "I listed above what I see as key weaknesses in the paper. Not every single one of those *absolutely has* to be addressed - but the authors need to choose at least one aspect and strengthen it substantially. If it's to be an observational paper, then W1 and at least two out of the other three of W2-4 (and ideally all of them) would need to be greatly improved.\n\nIf the authors decide to emphasize concepts and theory, then a rigorous mathematical treatment (W6), as well as extensive work relating the paper to some of the existing bodies of research (W5) are \"must-haves\". \n\nA more detailed coverage of related literature (W5) should be given however the paper evolves.\n\nAs for stylistic recommendations - I feel less strongly about those. Some form of motivational discussion (W10) does seem to be critical.\nI feel confident that taking W8 and W9 into account will be beneficial, but would keep those as recommendations. Finally, the tone (W11) is a matter of personal taste.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZrzQytOoaF", "reviewer": "Reviewer_YWzM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines the phenomenon of generalization in deep neural networks with corrupted training labels. It has been observed that DNNs trained with corrupted labels can achieve perfect training accuracy (necessarily memorizing the spurious labels), at the cost of generalization performance. However, even in this regime the model's internal representations may still contain useful, generalizable structure, which is measured by fitting a simpler predictor (a probe) to hidden representations of the trained network. The contribution of this paper follows in the tracks of recent work based around a subspace-based yet nonlinear probe to propose a linear probe, which is claimed to achieve similar performance.", "claims_and_evidence_yn": "No", "claims_explanation": "There are two main points of the paper's contribution which demand experimental evidence: (i) that VeLPIC is an effective method of obtaining a functional classifier from a deep neural network trained with corrupted labels; and (ii) that VeLPIC is competitive with MASC (Ketha & Ramaswamy 2025). The most convincing evidence for both of these points is relegated to the loosely-structured appendices. To point (i), the issue here is that the story is relatively clear for the easy benchmarks (MNIST and Fashion-MNIST) but there seems to be deterioration for more difficult datasets (Tiny ImageNet and CIFAR10), where even the uncorrupted models are not achieving good generalization. To argue convincingly that the performance of VeLPIC itself generalizes to more complex tasks, experiments on larger models and datasets are necessary -- the largest network examined here is AlexNet. To point (ii), the main body contains no numerical comparison with MASC at all (although the material in the appendix does convincingly demonstrate that VeLPIC is comparable or better than MASC on the evaluations examined here).", "audience_yn": "No", "audience_explanation": "The phenomenon that this paper investigates is interesting and important to understanding how deep learning works. However, I'm skeptical that its findings in the current state have much to offer, since the loose approach they explore is not new (see e.g. Li et al 2020, \"Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks\" which examined the behavior of the classwise principle components under corruption) and it is unclear whether the findings will generalize. The clearest contribution is a conceptual improvement on MASC (Ketha & Ramaswamy 2025), so those who were aware of that work may be interested in this one, but given that was a pre-preprint with a single citation by the same authors I am not convinced that demographic is very substantial.", "requested_changes": "* The abstract reads more like an introduction and should be trimmed down by about half to better focus on the paper's contribution.\n\n* As I understand it, VeLPIC is needlessly convoluted by the augmentation step. Adding the negatives of a set of vectors only changes the empirical covariance matrix by an overall scalar and so does not affect the PCs or their ordering. It would then appear that VeLPIC is just linear classification with classwise PCs as class vectors, which is an old idea and should be properly referenced (e.g. Das et al, \"A Classwise PCA-based Recognition of Neural Data for Brain-Computer Interfaces\", IEEE 2009).\n\n* The experiments that directly compare MASC and VeLPIC should be included in the main body.\n\n* Toy datasets like MNIST and Fashion-MNIST should not be prioritized in the results since the results there do not seem to generalize to the more complex datasets. At minimum I recommend moving the CIFAR10 experiments to the main body.\n\n* High-accuracy generalization is not being obtained in the 0%-noise setting for the harder datasets (Tiny ImageNet and CIFAR10). I recommend modifying models and training protocols as necessary to improve baseline generalization and then re-run the corruption experiments in that setting.\n\n* Comparison with the baseline of VeLPIC or MASC applied to representations drawn from a randomly initialized network would also strengthen the argument that it is some generalization property of the representations rather than the strength of the probes.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SVBcxk0wLS", "reviewer": "Reviewer_StSB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper extends the results of Ketha & Ramaswamy (2025) in two directions. First, the paper considers the dynamics of probe accuracy throughout training rather than focusing on the final weights. Second, the authors propose new, more accurate VeLPIC probes instead of MASC probes.\n\n**Strengths**\n1. The work investigates an important memorization phenomenon, relevant for understanding deep learning mechanisms.\n2. The paper offers a clear improvement of Ketha & Ramaswamy's (2025) methodology.\n\n**Weaknesses**\n1. The paper mostly offers technical improvements and extensions without providing deeper insights.", "claims_and_evidence_yn": "No", "claims_explanation": "1. I am not sure why the analysis of dynamics suggests that model generalization and latent generalization use largely the same underlying mechanisms. From what we can see, the dynamics is indeed similar at the beginning of training. However, at the end of training, the results diverge, suggesting different mechanisms for these phenomena.\n2. I am not sure what authors mean by a quadratic classifier and why this distinction is important. MASC projection could easily be rewritten in a matrix form without the tensor product of PCA vectors.\n3. I do not think that the inclusion of negative datapoints is a theoretically grounded way to perform PCA. In particular, in our case, this augmentation will arguably inflate a direction in which the class-conditional vector space is shifted from the origin.", "audience_yn": "Yes", "audience_explanation": "The research question seems important. However, the significance of contributions is modest.", "requested_changes": "1. Please explain in more detail the conclusions of the experiments and their limitations.\n2. Please clarify what a quadratic classifier means in the context of this paper.\n3. Please explain the goal of applying PCA in this setting.\n4. I think the paper would benefit from a discussion about why different layers produce different results. Generally, the discussion in Section 4 seems rather short.\n5. (minor) I think the paper would benefit from the inclusion of more modern vision architectures (e.g., ResNets) for FashionMNIST and CIFAR-10 experiments.", "broader_impact_concerns": "I do not feel a need for a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)