Decoding Generalization from Memorization in Deep Neural Networks
Authors: Simran Ketha
Venkatakrishnan Ramaswamy
Venue: Accepted by TMLR ()

Abstract: Overparameterized deep networks that generalize well have been key to the dramatic success of deep learning in recent years. The reasons for their remarkable ability to generalize are not well understood yet. When class labels in the training set are shuffled to varying degrees, it is known that deep networks can still reach perfect training accuracy at the detriment of generalization to true labels -- a phenomenon that has been called memorization. It has, however, been unclear why the poor generalization to true labels that accompanies such memorization, comes about. One possibility is that during training, all layers of the network irretrievably re-organize their representations in a manner that makes generalization to true labels difficult. The other possibility is that one or more layers of the trained network retain significantly more latent ability to generalize to true labels, but the network somehow “chooses” to readout in a manner that is detrimental to generalization to true labels. Here, we provide evidence for the latter possibility by demonstrating, empirically, that such models possess information in their representations for substantially-improved generalization to true labels. Furthermore, such  abilities can be easily decoded from the internals of the trained model, and we build a technique to do so. We demonstrate results on multiple models trained with standard datasets. Our code is available at: https://github.com/simranketha/MASC_DNN .

Published in Transactions on Machine Learning Research (02/2026)
Decoding Generalization from Memorization in Deep Neural Networks
Simran Ketha
p20200021 1 @hyderabad hiits-illaii.o
Anuradha and Prashanth Palakurthi Centre for Artificial Intelligence Research, Department of Computer Science Information Systems, Birla Institute of Technology & Science Pilani, Hyderabad 500078, Indio.
Venkatakrishnan Ramaswamy
venkut@hyderabud. bits-pilanei
Anaradha and Prashanth Palakurthi Centre for Artificial Intelligence Research, Department of Computer Science & Information Systems, Birla Institate of Technology Science Pilani, Hyderabad 500078, India.
Reviewed on OpenReview: https: // openreview net/ forum? id=
Abstract
Overparameterized deep networks that generalize well have been key to the dramatic success of deep learning in recent years. The reasons for their remarkable ability to generalize are not well understood yet. When class labels in the training set are shuffled to varying degrees, is known that deep networks can still reach perfect training accuracy at the detriment of generalization to true labels phenomenon that has been called memorization. It has, however, been unclear why the poor generalization to true labels that accompanies such memorization, comes about. One possibility is that during training, all layers of the network irretrievably re-organize their representations in a manner that makes generalization to true labels difficult. The other possibility is that one or more layers of the trained network retain significantly more latent ability to generalize to true labels, but the network somehow "chooses" to readout in a manner that is detrimental to generalization to true labels. Here, we provide evidence for the latter possibilityy by demonstrating, empirically, that such models possess information in their representations for gubstantially-improvee generalization to true labels. Furthermore, such abilities can be easily decoded from the internals of the trained model, and we build a technique to do so. We demonstrate results on multiple models trained with standard datasets. Our code is available al: httpp:// github. www.m//immaanetaaaaaano..aa DNN
1 Introduction
Prior to the advent of deep learning, the conventional wisdom for long was that in building a predictive model, the model should have as few parameters as possible and this number should certainly be less than the number of training samples that one was fitting. The dogma was that, otherwise, the model would exactly fit the training points, but invariably generalize poorly to unseen data, i.e. overfit. This intuition was also largely borne out by the models of the day. Modern deep learning, however, has gone on show the opposite, namely that overparameterizel models not only don't 't necessarily overfit, but that they can generalize remarkably well to unseen data. However, understand why this is so. Interestingly, it has been shown (Zhang et al. 2017 2021 that when one randomly shuffles class labels of data points from standard training datasets to varying degrees, deep networks can still have high perfect training accuracy when trained on such corrupted training data; however, this appears to typically be accompanied by poor performance on unseen lest. data (that have true labels). This phenomenon
decade later, still do not satisfactorily
von Neumann famously said, "With four parameters I can an elephant, and with five make him wiggle his trunk."
Dyson al. 2004

---

Published in Transactions on Machine Learning Research (02/2026)
has been called memorization since it is thought that the model rote-learned the training data without acquiring the ability to generalize to true labels. It has been suggested that progress on understanding memorization could enable a better understanding of generalization to true labels in deep networks trained on real-world data (Zhang et al., 2017 2021) and indeed that a detailed understanding of mochanisms of generalization to true labels should also be able to explain the phenomenon of memorization.
An open question arising in this context is about the detailed mechanisms that lead poor generalization to true labels in models trained with shuffled labels, i.c. models that memorize. A natural hypothesis governing such mechanisms, stated informally, is that, during training, the network organizes its internal reprosentations in all layers, in manner suited to doing well on the (corrupted) training data. Since this data is siguificantly noisy, on being given unseen data with true labels, it fundamentally lacks the ability to have good prediction performance, leading to poor generalization to true labels. An alternative hypothesis is that layerwise representations on a subseffoff the layers in the network retain significantly more ability to gencralize to true labels, than the model, but that the network somchow chooses to readout in favor of high training accuracy in a manner that incidentally causes poor generalization to true labels. A consequence of this alternative hypothesis is that one ought to be able to construct decoder (i.e. a probe) for the outputs of such layers that has better gencralization performance on truc labels.
Here, surprisingly, we show evidence for this alternative hypothesis. In particular, we study the organization of subspaces of class-condin conditioned training data on layerwise outputs, in deep networks. We estimate these subspaces using Principal Components Analysis (PCA). In order to remain oblivious to the information decoded by subsequent layers, we build a simple probe that leverages the geometry of the present layer's output of an incoming datapoint, relative to these class-conditioned subspaces. Specifically, we measure the angle between this output vector and its projection on each of these class-conditioned subspaces and the probe predicts the datapoint's ' class to be the class whose subspace has the minimum such angle. We call this probe the Minimum Angle Subspace Classifier (MASC). Notably, unlike probes used conventionally (e.g. in (Alain & Bengio 2018)) whose paramcters are determined by iteratively minimizing a crossentropy loss, the parameters of MASC are directly determined from the subspace geometry of the training data. A schematic illustrating the geometry of MASC is presented in Figure
We train a number of deep networks with standard datasets in the memorization sctting. Here, a randomly- chosen fraction of training data points have their labels changed lo a randomly-chosen label from the available labels in the dataset. We do so for differing fractions of the training dataset and - consistent with previous work (Zhang et al., 2017 2021 Arpit et al., 2017) see that training with such corrupted training datasets causes correspondingly poor test accuracies in the model. However, MASC which uses the internals of the network to predict the class label tends to do significantly better than the model on the test set.. A schematic illustration of the memorization setting with MASC is shown in Figure
We outline a more detailed summary of our main contributions below.
1. For models trained with standard methods & datasets with training data corrupted by label noise to varying degrees, we demonstrate (with one exception) that MASC applied on at least one layer, when using subspaces corresponding to such corrupted training data, has significantly better test ac- curacy than the model. For example, MASC outperforms the model test accuracy by upto 159.93%, 189.00%, 64.86% and 119.16% on MIP-MNIST, ResNNet-1-CCIAAR respectively. A more detailed account of these numbers is in Table 1
and
2. For the models discussed above, we perform a comparison study evaluating five probes namely Logistic regression, K-Nearest Neighbor (KNN), Linear Discriminant Analysis(T.DA),, Quadratic Discriminant Analysis (QDA), and Nearest Class Mean (NCM) over the layers of multiple deep networks under varying corruption degrees. While many of these probes exhibit similar overall accuracies and show no consistent trends across corruption degrees, their computational costs differ considerably, which we have empirically analyzed.
2e direct the reader Section 3.1 for formal description of our setting, Sit indoed possible that certain layers of the network have representations that are significantly more generalizable to true labels than others and these layers may early or later layers. We find that few other probes do comparably well.
2

---

Published in Transactions on Machine Learning Research (02/2026)
a)
Standard training dataset
b)
Corrupted training dataset
Class 1
Class 2
Class
Class
c)
Model
Corrupted training dataset
Input layer
Hiddenn layers
Output layer)
Class
Class
Training accuracy
accuracy on true labels
Training accuracy
Minimum Angle Subspace Classifier (MASC) Our technique
Test accuracy
true labels
Figure 1: A schematic of the memorization setting used in our work and the application of the MASC classifier in it. a) Illustration of standard training dataset. b) A corrupted training dataset is created by changing the labels of the standard training dataset with specific probability (due to which few of the colors are changed, representing the changed labels). Changing the labels happens uniformly at random for the whole dataset. c) A deep network trained with this corrupted training dataset to achieve ~100% training accuracy, which is usually accompanied by poor test accuracy (as measured on true labels from the test set). We have shown that the Minimum Angle Subspace Classifier (MASC) our technique which uses the internals of the deep network, tends to have significantly better generalization to true labels (tost accuracy) than the deep network itself.
3. For the aforementioned models, if the true training class labels are known post hoc, i.c. after the model is trained, we can build MASC using subspaces corresponding to true class labels. These MASC classifiers usually have better generalization to true labels than in (1). For example, MASC using true labels outperforms the model by upto 198.43%, 212.42%, 337. 337.51% and 228.64% on MLP- MNIST, LNN-FaahionnMNNSSS detailed account of these numbers is in Table 7 This demonstrates that the layers of the memorized network maintain representations in manner that is amenable to straightforward generalization to true labels to a degree not previously recognized.
and ResNet- 1eae--8-CCFFFA CIFAR-10 respectively. A more
4. Conversely, we asked if a model trained on true training labels similarly retained internal representa- tions that have the capability to memorize easily, as manifested by MASC. Adapting our technique to this setting, we create corrupted training sets which we use to build MASC. In this setting, we find that we can extract a high degree of memorization, in some cases. The results are presented in Section H
3

---

Published in Transactions on Machine Learning Research (02/2026)
5. Finally, leveraging the MASC classifiers built in (1) and (3), we ask, if we can retrain the memorized model for a few epochs to achieve better model generalization to true labels. We find that indeed, in many cases, there is an improvement in the generalization to true labels of the model.
2 Related work
The idea of probing intermediate layers of deep networks isn't new. For example, kernel-PCA (Montavon et al.] 2011) with RBF kernels has been used to analyze layerwise evolution of representations of deep networks. In that work, they quantify the quality of layerwisc representations and find that the last layers of the network tend to have representations that are more simple and accurate than previous layers. Likewise, linear classifier probes (Alain & Bengio 2018) have been used to study the roles and dynamics of intermediate layers in deep networks. There, they show that the degree of linear separability increases over the layers of the network. However, they explicitly avoid examining memorized networks (Zhang et al. 2017) because they thought such probes would inevitably overfit. Our results are therefore especially surprising in this context, because we demonstrate, on the contrary, that intermodiate representations, in fact, tend to resist overfitting, to a degree not previously recognized.
The authors in (Zhang et al. 2017) investigate the question of whether deep networks can perfectly learn noise by training randomly relabeled data, showing that although such models achieve ncar-perfect training accuracy, they exhibit, poor generalization to the true labels. Subsequent work by (Arpit et al.) 2017 shows that early on in training, memorized networks (Zhang et al. 2017) start off by having better generalization to true labels; however generalization to true labels worsens as training accuracy increases across epochs of training. Despite these observations, there have been limited efforts to analyze the internal representations of models in the memorization regime (i.e. one where the labels of a subset of training data points are shuffled randomly) using probing methods. It was assumed that the cause of poor generalization to true labels in memorization scttings directly manifests from poor reprosentations. For example, in a probe-based analysis [Alain & Bengio. 2018) argue that probe measurements would be "entirely meaningless" in memorization settings. They say
"It was recently demonstrated that very large models can fit random labels on ImageNet (Zhang et al., 2016). This is a situation that we want to avoid because the probe measure- ments would be entirely meaningless in that situation."
This view aligns with our mull hypothosis that, during training, all layers of the network irretrievably re- organize their representations in manner that makes generalization to true labels difficult. However, the alternative hypothesis that we articulate here has not been explicitly examined in past work, presumably because it was not considered plausible: that one or more layers of the trained network retain significantly more latent ability to generalize to true labels, but the network somehow "chooses" to readout in manner that is detrimental to generalization to true labels.
There have been efforts to build training algorithms that are designed to extract better generalization to true labels in the case where the data is known to be noisy (Jiang et al., 2018 Han et al. 2018 Liu et al., 2020). Stephenson et al (Stephenson et al. (2021) investigate memorized models, suggesting that memorization predominantly occurs in the later layers. This is based, in part, on the observation that rewinding early- layer weights to their carly-stopping values can recover generalization to true labels, whereas rewinding later-layer weights does not yield the same effect. In general, the thinking in the field has been that while there is an initial peak in generalization to true labels, it is lost during further training, although one can mitigate some of this loss by modilying training (Jiang el al. 2018) or by rewinding a subset of weights to their early values. On the contrary, our results suggest that layerwise outputs of deep networks retain significant ability generalize after training and we demonstrate that this generalization to truc labels can be extracted without modifying the weights of the trained network that are obtained via standard training methods.
4

---

Published in Transactions on Machine Learning Research (02/2026)
An important line of theoretical research deep linear models has explored the question of generalization to true labels (Saxe et al. 2013). Here, a theoretical explanation for the phenomenon of memorization in networks trained with noisy labels has been proposed (Lampinen & Ganguli 2018).
Studies have investigated training dynamics across layers using varions forms of Canonical Correlation Anal- ysis (Raghu et al.] (017), including analyses in both generalized and memorized networks (Morcos et al. 2018). Centered Kernel Alignment has been employed to examine the effects of different random initializa- tions (Kornblith et al.) 2019), as well as to study network similarity between models trained the same data with different initializations (Korbblihh et al. 2019). Additionally, experiments have explored the use of representational geometry measures to understand the dynamics of layerwise outputs (Chung et al. 2016 Cohen et al.] 2020), along with other structural measures such as curvature dimensionality (Hénalf et al. 2019 which aim to capture underlying properties of learned representations (Sussillo & Abbott 2009: Far-
rell et al.] 2019 Gao & Ganguli 2015 Litwin-Kumar et al. 2017 Bakry et al. 2015 Cayco- Gajic & Silver 2019 Yosinski et al.
2014 Stringer et al. 2019).
To address label noise,
various heuristic approaches have been proposed
Khetan et al 2017
Scott et al.
2013 Reed et al. 2014 Zhang & Sabuncu 2018 Malach & Shalev-Shwartz 2017). particularly in the context
of classification tasks Frénay et al.) 2014 Ren et al. 2018 Menon et al. 2018 Shen & Sanghavi 2019)
In the case of overparameterized models, Li et al (Li et al.
2020) demonstrate that memorization requires
the network weights to deviate significantly from their initial random state in order to overfit noisy labels.
Additionally, in a theoretical model of epochwise double descent (Stephenson & Leel 2021), it has been suggested that for smaller models, moderate levels of label noise can lead to a reduction in generalization error at later stages of training.
3 Methods
3.1 Preliminaries
In this subsection, we state preciscly the setting that treated in this paper.
We study classification task defined over data distribution D, with an i.i.d. training dataset T drawn from D. For given corruption degree + generate modified training set T, by randomly relabeling certain expected fraction of the training samples uniformly at random. The resulting label corrupted training data distribution is denoted by Dy-
A series of prior studies (e.g., Zhang et al. (0177) 1 Arpit et al. (2017) have demonstrated that deep networks can successfully fit such corrupted datasets Tpp even when substantial fraction of the labels are random- ized. These findings highlight the remarkable expressive power of the hypothesis class H associated with contemporary deep models. In particular, they show that H is sufficiently rich to learn training data arising from highly perturbed label distributions such as Dp, thereby enabling the model to achieve near perfect training accuracy despite the presence of significant label noise.
As the corruption degree p increases, the distribution of the corrupted data D, diverges progressively from the true distribution D. Conseqquetlyy, one would expect that any network h trained to fit samples Êp drawn from Dd becomes increasingly misaligned with the task defined by D, and therefore performs poorly on test dataset T' drawn from distribution D. This phenomenon has been called memorization in past work (e.g. Arpit et al. 2017
(hhang et al. 2017 2021:
'When we say the training datasct has corruption degree p. mean that with probability p. attempt changing the label for each training datapoint. Changing the labels happens uniformly at random to any of the class labels. Note that this may result in the label remaining the same; therefore the expected fraction of datapoints whose labels changed are p p/ where K is the number of class labels. So, e.g. for dataset with 10 classes, this would mean that for corruption degrees of 20%, 40%, 60%, 80% and 100%, the expected percentage of training datapoints with changed labels 18%, 36%, 54%, 72% and 90% respectively. We have run experiments for values of p being 0% (generalized model), and memorized models with p being 20%, 40%, 60,, 80% and 100%.
6This phenomenon could also be viewed as learning under label naise. However, given the usage of the term memorization to refer this phenomenon in past work, we choose continue to so here.
5

---

Published in Transactions on Machine Learning Research (02/2026)
For network trained on T. drawn from Dp, generalization would, usually, refer to network's performance on D after learning from T.. In this work, however, we examine whether network trained on F, (using the network's internal representations) can perform well on test. data drawn i.i.d. from the true distribution test set T' sampled from D. Throughout the rest of the paper, we refer to this notion simply as generalization to true labels.
D. Accordingly, define good generalization in terms of high performance
3.2 Experimental setup
We have used multiple models and datascts, namely Multi-layer Perceptron (MLP) trained on MNIST
(Deng 2012) and CIFAR-10 (Krizhevsky 2009) datasets, Convolutional Neural Networks (CNN) X trained
on MNIST, Fashion-MNIST (Xiao et al. (017), and CIFAR-10 10, AlexNet (Krizhevsky al.) 2012) trained on
CIFAR-100 (Krizhevsky 2009) and Tiny ImageNct (Moustafa 2017) and ResNet-18(He et al., 2016) trained
CIFAR-10. We have trained these models with training data having true labels ("generalized models")
well as separately using training data with labels shuffled to varing degrees ("memorized models") (Zhang
et al. 2017 2021).
A summary of the models, datasets, training set sizes, and number of parameters is provided in Table 2
Tables 3 and report the average training and test accuracies of all models over three runs. Additional details on the models, hyperparameters, and training procedures are also included in the Appendix. The general terminology used in this work is also explained in the Appendix.
Following standard practice in studying memorized models (e.g. (Stephenson et al. 2021 ) we do not use explicit regularizers such as dropout or batchnorm, or early stopping, unless otherwise mentioned, as a result of which our baseline test accuracy mumbers are often much lower than what is usually found with standard training of these models, All the models are trained to either reach very high training accuracies (i.c. 99% - 100%) or trained until 500 epochs. Some modcls did not reach such high accuracies, in which case, results have been shown on the model obtained at epoch 500. We trained instances of each model and results displayed are averaged over these instances with the shaded region indicating the range of results also indicated in the plots.
3.3 Minimum Angle Subspace Classifier Algorithm (MASC)
For a given data point æ from the training or test set, a layer output data point It from layer when imput x is passed through the network and its corresponding training subspaces { with K classes, we use Minimum Angle Subspace Classifier (MASC) Algorithm for predicting class labels y(æ1).
Algorithm Minimum Angle Subspace Classifier (MASC)
Input: Training subspaces (m layer output data point æt from layer when input æ is passed through the network and classes {C*
Output: MASC prediction class label ((x) according layer L.
1: for each class Ck do
2: Vik -- compute the projection of æ onto subspace Sy.
3: Compute the angle ((1,, IiI)) between In and aikk
end for
Assign the label y(m) Cs where h arg ming 0(11, Fmk)
Return: label y(x1)
Given training dataset DI( y1) 1/) where each xs E Rd and ye E {C*] are input-label pairs, we 1K estimate training subspaces { for all classes K, for a given layer of the neural network using Algorithm and] In practice, S4 is represented via its principal components, which form a basis for the subspace.
We have used 99% the percentage of variance explained by the principal components, unless otherwise mentioned. While the subspaces are estimated using the training data alone, accuracy of the MASC is
The CNN models were built along the lines of Tran al. 2022

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm 2 Subspaces Estimator for MASC
Input: Training dataset D((2s, vi) where each 2ú E R2 and E {Ca are input-label pairs, neural network, and layer I. Output:
Subspaces {Sk] for classes K, for given layer I.
1: =
2: for each input pair (æt, vi) in D do
3: Pass 2; through the network layers to obtain the output of layer I, denoted as æ E Rdd,
4: Di= Dr ((11, y6)}
end for
Estimated subspaces {Su]a= PCA-Based Subspace Estimation(P:)
Return: Subspaces {5k]k=1
Algorithm PCA-Based Subspace Estimation
Input: Layer output D; {(xs, u) W)) where æı Rdd and E {Ck)
Output: Subspaccs {Sp k== for classes K.
Dnew f- Di
for each (xi; E Di do
Dnnw Dnww U vi( (1(-ri; y:))
end for
5: for cach h E f1,.... K] do
6: Extract the subset of data Dnww, = {ats yi = Ck
7: S =PCAA(D ne),,)
end for Return: Subspaces {
determined for the training data and the test data separately. This process is followed for all the layers in the network independently. MASC is using labels of the dataset while creating the class-specific subspaces. The process of creation and use of subspaces with MASC for new data point are shown schematically in Figure 2
We apply MASC on each layer of the network with respect to different subspaces. For MLP models, all the MASC experiments were performed for all the layers in the network including on the input (after it is pre-processed). For CNN models and AlexNet models, the experiments were performed on flatten layer (Flat) and fully connected layers (FC). For ResNet-18 18 model, we evaluated nine layers LO, L0-1, L0-2, LO-3, L1, L2, L3, L4, and the average-pool (avg pool) layer containing 16,384; 16,384; 16,384, 16,384; 16,384; 8,192; 4,096; 2,048; and 512 neurons respectively. Here, LO denotes the layer immediately before the block; T.01, T0-2, and L0-3 are intermediate outputs within the L1 block; and L1-I4 correspond to the outputs of successive residual blocks. All layer outputs were flattened prior to analysis. While we ran the experiments on the input layer for CNNs, we did not do so for AlexNet or ResNet-18.
3.4 Leveraging MASC to retrain the model
Here, the idea is use one of the layerwise MASC classifiers in order to relabel the corrupted training This relabeled training set is then used to retrain the existing model. To determine the layer whose MASC classifier will use, find the layer whose MASC classifier generalizes best. To this end, we first split the test data set into 80%-20%. We use the MASC accuracy on the corrupted subspaces in the validation set (created from 20% of the test dataset) to identify the model's best-layer. Then, using the best-layer MASC predictions, we relabel the corrupted training dataset. We train with the relabeled corrupted training dataset for upto 30 epochs and perform early stopping with pationce of by considering the 20% test dataset as a validation dataset; this validation dataset was not used in reporting test accuracy. A similar process was followed while working with subspaces corresponding to true labels. The test. accuracy the models is

---

Published in Transactions on Machine Learning Research (02/2026)
Training dataset
Model
Adding negative datapoint in order zero-center the dataset
Input laer
Hizden layers
Subspace
Closs
PCA
Input layer
Hidden layers
New datapoint
Datapoint ambient
Class
Project datapoint onto classwise subspaces individually and find the angle
Class subspace
Class subspace
Class subspace
Class subspace
90'
ON
90
0
Class the minimum angle the predicted class
Figure 2: A schematic of the Minimum Angle Subspace Classifier (MASC) constructed for specific hidden layer. a) Illustration of the process of fitting subspace (i.e. linear space that passes through the origin) corresponding to single class, for the outputs of specific. hidden layer. For a specific layer, class-wise training dataset (Class 1) is passed through the model till the hidden layer in question. For every output datapoint (activation values) of the layer, a negative data point added to the point set in order to zero- center the outputs dataset before performing Principal Components Analysis (PCA).
b) Such subspaces for the hidden layer are constructed for every class. When new (e.g. unseen) datapoint needs be classified MASC, its output from the hidden layer is computed, which is a datapoint in the ambient space of the hidden layer. This datapoint is projected onto the individual class-conditional subspaces and the angle between the data point and its respective projections, 0,.... Oz, are determined. MASC predicts the datapoint's 's class to be the one whose subspace the datapoint has the smallest such angle with i.e arg min)(0;}}
calculated with respect to the 80% test dataset, obtained in the aforementioncl split. schematic of the retraining process using MASC is shown in Figure
4 Enhanced innate generalization to true labels in memorized models
Models trained with corrupted labels have high training accuracy (on corrupted labels) while also having low accuracy on the test set with true labels (Zhang et al. 2017 2021). We ask if we can decode the representations of the hidden layers of these memorized models to obtain better generalization true labels.

---

Published in Transactions on Machine Learning Research (02/2026)
Model Training Accuracy
MASC Accuracy on Comuutte Training
MASC Accuracy en Test
Model Testing Accuracy
Accuracy on Original Training
MLP-MNIST
CNN-Fashion-MNSS
AlexNet-CIFAR-100
ResNec18-CIFAR-1
coruption
0.0
20% corruttin 86
0.8
0.6
0.4
0.2
0.0
40% conrpttin degres
1.0
0.8
0.4
0.2
Accuracy
compptiin degree
degres
0.6
0.4
0.2
0.0
1.0
0.8
100% cortuption
0.6
0.4
0.2
0.0
wan
sc co00)
Pardon
coom
Rercoe
PC
Network layers
Figure 3: Minimum Angle Subspace Classifier (MASC) accuracy over the layers of the network when the data is projected onto corrupted training subspaces with the indicated corruption degree, for multiple mod- els/datasets. Rows corresponds to plots with the same corruption degree the columns correspond to the models, as noted. Training accuracy (dashed line) & test accuracy (dotted line) of the model shown. FC corresponds to fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation. The number of class- wise PCA components of these models are shown in Figure 30 SGD
optimizer (Qian 1999) was used for training MLP models, whereas Adam optimizer (Kingma 2014] was
used for other models. ResNet-11 has layer outputs of size 16,384 for LO-L1, followed by 8,192 (L2), 4,096 (L3), 2,048 (L4), and 512 (avg pool) layer.

---

Published in Transactions on Machine Learning Research (02/2026)
To do so, build a probe that call Minimum Angle Subspace Classifier (MASC) using class-conditioned corrupted training subspaces obtained from the memorized models' hidden layer outputs. MASC is performed layer-wise for the layers of the network independently. More details on MASC are available in the Methods section. MASC accuracy on corrupted training data, MASC accuracy on original training data (with true labels), and MASC accuracy on lest data (with true labels) over the layers of MLP trained on MNIST, CNN trained on Fashion-MNIST, AlexNet trained on CTFAR-100, and ResNet-18 trained on CIFAR-10 for various randomly-chosen fractions of label corruption in training data (i.e. corruption degrees) are shown in Figure g Likewise, results for MLP trained on CIFAR-10, CNN trained MNIST & CIFAR-10 and AlexNet trained on Tiny ImageNet are presented in Figure 27
Importantly, for every corrupted model we have (with non-zero corruption degree), except those with 100% corruption degree, we find that our Minimum Angle Subspace Classifier (MASC) in at least one layer (with one exception D has better test accuracy than the corresponding model itself. Table [ reports by what percentage the MASC classifier outperformed the model for the best such layer, for each model. In Table [ we also report the accuracy difference between the MASC classifier and the model for the best such layer, for each model. In many cases, the MASC test accuracy is dramatically better than that of the model. This is remarkable, because, in addition to the layerwise outputs, MASC used preciscly the same information (including the same corrupted training dataset) that was available to the model itself, and yet is able to extract better generalization to true labels. This suggests that the model retains significant latent generalization to true labels, which is not captured same MASC, especially on the later layers, also approaches perfect accuracy on the corrupted training set, indicating that this improved generalization to true labels happen concurrently with memorization of training data points with shuffled labels. Below, we make specific observations the performance of the models.
its own test-set performance. In many models, the
Table I: Percentage by which the MASC classifier (run on the best layer) purporrrd the model's test accuracy when the data is projected onto corrupted training subspaces. The best layer corresponds to the one that has the highest measured MASC test accuracy among the layers for the said model/datasee The accuracies in each case are averaged over three runs and are rounded to the second decimal place. Some of the detailed results are available in Appendix, as indicated.
Corruption degree
20%
40%
60%
80%
MLP-MNIST
10.93%
31.63%
75.04%
159.93%
MLP-CIFAR-10 (Appendix)
9.90%
24.42%
46.97%
64.75%
CNN-MNIST (Appendix)
9.81%
37.03%
98.69%
201.06%
CNN-Pashion-MNNIST
7.49%
33.50%
84.93%
189.00%
CNN-CIFAR-10 (Appendix)
2.29%
6.26%
27.03%
60.17%
MexNNet-CFFAR-100
33.58%
53.10%
64.86%
45.00%
AlexNet-Tiny ImageNet (Appendix)
27.50%
53.46%
45.38%
14.16%
ResNett-I-CIFAR-
-0.41%
20.34%
67.28%
119.16%
With generalized models i.c. those with 0% corruption degree, at the later layers of the network, it is observed that most of the cases MASC accuracy on training data approaches the models training accuracy. Similarly, MASC accuracy on test dataset is comparable to or performed better than the models' test accuracy, with the exception of the ResNet-18 model.
Even for high corruption degrees, we find that MASC performs well. For example, with 80% corruption degree, which implies that approximately 72% of the training labels have been changed, we observed good MASC test accuracy in many cases. Notably, the MASC test accuracy on the later layers is over 80% on HLP--NIST.. in comparison to 34% test accuracy by the model. Similarly, MASC test accuracy on one of the layers about 75% for CNN-Fashion-MNISTT in contrast. to 25% model test. accuracy. Even for larger
R trained CIFAR-10 with 20% cormpttin degree is the lone exception. See Figure and Table for the corresponding results.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Testing Accuracy
Model Testing Accuracy
Accuracy on
MLP-MNIST
CNN-Fashion-MNIST
AlexNet-CIFAR-100
1.0
0.8
corrustion
0.6
0.4
0.2
1.0
0.8
20% compptio
0.6
0.4
0.2
0.0
1.0
0.8
40% comupttin
0.6
0.4
0.2
corruption Accuracy
1.0
0.8
0.6
0.4
0.2
0.0
1.0
80%
degree
1.0
0.8
0.6
0.4
100%
0.2
0.0
Input
FC (128)
FC (512)
PC (2048)
FC (2048)
Input
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure MASC test accuracy over the layers of the network when the data is projected onto corrupted training subspaces with the indicated corruption degree. Bost model test accuracy corresponds accuracy of the test data of the model if early stopping was used.
11

---

Published in Transactions on Machine Learning Research (02/2026)
models/datasets such as AlexNet-CIFAR-ION MASC test accuracy outperforms the model test accuracy by 45%, for training sets with 80% corruption degree. Likewise, for ResNVet188-CFAA--11 several layers exhibit MASC test accuracies that exceed the model's test accuracy.
Not only does MASC have better accuracy than the model on the test data but, when applied to some layers, it also does well on the training data with the true labels. Although the model has memorized the training data with corrupted labels, outputs from certain layers have the ability to predict the trained true labels. For example, in MLP-MNIST, for low to moderate degrees of corruption, MASC the middle layer (FC (512)) has good accuracy the true training labels, while also retaining good accuracy the test set. With 40% corruption degree, approximately 36% are changed labels and yet the model has good accuracy on the true training labels in at least one layer of the network. e.g. MLP-MNIST has over 90% true training accuracy at layer FC((1)), WNNN-Fashion-MNEE & AlexNet-CTFAR-100 has approximately 60% in FC (4096) layer. This means that almost 20% of those labels are predicted correctly even though the model was trained for 500 epochs or has reached high training accuracy on corrupted labels. In the process of doing this, the model does not have any direct information about the true labels and neither does MASC.
has approximately 85% in Flat (576) layer
For subset of models, we also compare the results of test accuracy of MASC on trained model with early stopping model accuracy. MASC accuracy over the layers of the network when the data is projected onto corrupted training subspaces is shown in Figure Best model test accuracy and trained model test accuracy are shown for reference. Best model test accuracy corresponds to the accuracy of the test data of the model if early stopping was used.
For MASC when the data is projected onto corrupted training subspace, in AloxNot-CIFAR-I00, the MASC in at least one layer shows better performance than the best model test accuracy for less than 60% corruption degree. For MLP- -NIST,, the best model (early stopping) maintains over 90% accuracy even when the data is corrupted up to 80%. Despite the increase in corruptions (except 100% corruption), the accuracy of the last layer remains close to that of the best model accuracy. For CNN-Pashionn-MNSST (except 100% corruption), in at least one layer MASC performance is near to that of best model test accuracy.
One way to think about deep network, is one that successively transforms input representations in manner that aids in good prediction performance. Therefore, performance of MASC on the input is good baseline measure to assess if subsequent layers have favorable accuracies. Naively, for models trained with corrupted data, one would expect layered representations that enable the model to do well on the corrupted training data, but not do well on the test cestitrainnn data that have true labels. While this expectation seems to hold with respect to the model itself, we find that the layer-wise representations do not necessarily follow this expectation. That is, MASC applied to subsequent layers, often have better true training accuracy and test accuracy than MASC applied to the input, suggesting that the deep network does indeed transform the data in manner amenable to better generalization to true labels, even if its labels are dominated by noise.
5 Generalization to true labels comparison: MASC versus other probes
Given that MASC is a probe on layerwise outputs, it is natural to ask how a few other probes might perform in the memorization setting. Accordingly, we have used five different probes on the layer of the deep neural networks namely, Logistic Regression (LR). K-Nearest Neighbor (KNN), Linear Discriminant Analysis (LDA),, Quadratic Discriminant Analysis (QDA), ), and Nearest Class Mean (NCM). Figure 5 reports the test accuracies of all probes across the layers of MLP-MNIST, CNN-FashionnMMNISIT and Alex-ee-CIFAA--D Model and MASC test accuracies are overlaid for comparison. Results for additional models are provided in We also report computational cost (GFLOPS) for different probes over the layer of the network in Section
Section C.2
We find that, although all probes achieve broadly comparable test accuracies with no consistent pattern regarding which probe performs best at different corruption levels - their computational costs (GFLOPs) differ substantially. For AlexNet, the probes ranked from highest to lowest computational cost are: QDA, LDA, KNN, MASC, LR, and NCM. For the CNN models, the ordering is KNN, followed by QDA and LDA
9 Experimental setup is provided in Section
C.1
12

---

Published in Transactions on Machine Learning Research (02/2026)
Model
NCM probe
LDA probe
MASC prabe
Logistic regression probe
k-NN probe
QDA probe
MLP-MNIST
CNN-Fashion-MNIST
AlexNet-CIFAR-10
1.0
0% corruption degroz
0.8
0.6
0.4
0.2
0.0
1.0
20% corruption degree
0.8
0.6
0.4
0.2
0.0
40% corruption degree
Test Accuracy
0.4
0.0
1.0
60% corruption degroe
1.0
80% corruption degree
0.8
0.6
0.2
FC (128)
FC (512)
FC (2048)
FC (2048)
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure Test accuracies for different probes over the layers of the network. Rows corresponds to plots which have the same corruption degree and the columns correspond to the models as noted. Test accuracy of the model and MASC are shown for comparison. FC corresponds fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation.
13

---

Published in Transactions on Machine Learning Research (02/2026)
(which have identical cost), MASC, LR. and NCM. For the MLP models, the sequence is KNN, QDA, LDA, MASC, LR, and NCM. Across all models, MASC consistently exhibits lower computational cost compared to the KNN, QDA, and LDA probes, although its cost remains higher than that of LR and NCM.
Given this computational profile, we focus on results comparing the test accuracy of MASC with LR and NCM; the corresponding results are detailed Section C.4 The three probes display distinct trends across layers and corruption degrees. On AlexNet CIFAR-100 and MLP MNIST at 20% 40% corruption, MASC surpasses both NCM and LR on most layers. For CNN-based models, NCM generally performs best, with MASC typically ranking above LR. Notably, MASC matches or exceeds NCM in specific cases such CNN-MNIST and ONN-Fashion-MNIST at 20%, 40%, and 60% corruption, at the F576( layer.
While our initial focus had been on MASC, it is interesting that other probes also have comparable per- formance, and indeed, this performance isn't always correlated with that of MASC. That is, in some cases, these probes perform better than MASC some layers and worse than MASC other layers. This suggests that latent representations that contain information useful for generalization to true labels may manifest in different forms, which result in different probes decoding them with differing accuracies. This phenomenon requires deeper investigation, which has been beyond the scope of the present paper.
6 Generalization to true labels via true training label subspaces in memorized models
While Section ! demonstrated improved generalization to true labels by MASC, here, we investigate if there exist subspaces that can offer even botter generalization to true labels. To this end, we consider the setting where the true label identities of the training set are known, after training with corrupted labels is complete. Can extract significantly high training as well test. performance in this case from the layerwise outputs of the network? To do so, we build MASC using subspaces obtained from training data with true labels. It is a priori unclear if MASCs trained in this manner will have high accuracy. Since the network trained assurning different labels for many of the datapoints, it is conceivable that class-wise subspaces corresponding to true labels lack structure and predictive power. We find, however, that these possibilitics do not bear out.
MASC accuracy on original training data and on test data projected on true training label subspace over the layers of the same networks is shown in Figure and Figure 28 For comparison, MASC accuracy on corrupted training data and test data projected on corrupted training subspace is also shown. We find that, in many cases, accuracies on the true training labels, as well as the test set are dramatically better here than with the experiments where subspaces were determined for the corrupted training data. In Table 7 we show by what percentage the MASC classifier outperformed the model for the best layer for corruption degrees 20%, 40%, 60% and 80%. In Table S we also report the accuracy difference between the MASC classifier and the model for the best such layer, for each model. In fact, the MASC test accuracies for the corrupted models (with non-zero corruption degree) are sometimes fairly close the test accuracy of the uncorrupted model.
Notably, even for models trained with 100% corruption degree, in most cases, MASC retains significant accuracy on the true training labels as well as the test set. This is in spite of the fact that the model itself has chance-level test-set accuracy. For example, MASC classifier has 95% test accuracy in the last FC(2048) layer for MLP-MNIST, 69% test accuracy for Flat(576) layer in for Flat(256) layer in AlexNet-CIFAR-100.
MNNIS,, and 4% test accuracy
The results here are proof of principle that suggest the existence of subspaces which allow one extract significantly high generalization to true labels on models trained with datapoints whose labels are shuffled to remarkably high degree. This has two implications. On the one hand, it demonstrates that models trained with very high label noise, surprisingly, retain the latent ability to generalize very well. On the other hand, it suggests that development of new techniques to identify favorable subspaces could help markedly boost generalization to true labels of models, whose training data is known to have label noise.
The results comparing MASC test accuracy over the layers of the network when the data projected onto true training subspaces with early stopping model accuracy is shown in Figure 7
14

---

Published in Transactions on Machine Learning Research (02/2026)
Model Tax Acousty
MAG Accussy Comuited Lubd Outspeca)
MAGC (Tre Labe Suturaci
ML P-NNIST
CANN-Fahhio--oo MNIST
AlexNet-CIFAR CIFAR 100
AlexNet
ResNet18 MesNet18-CIFAR-D CIFAR-10
3.0
0.0
corruption
0.6
0.4-
0.2
0.0
20% compution degree
2.0
0.2
60% coruption, Accuracy 40% corupption degree
2.0
0.8
0.6
0.4
0.2
0.0
80% comppttin degres
1.0
0.8
0.6
0.4
0.2
0.0
100% corruption degree
rect1))
Pe c200
PC crn
rerzem
Network layers
Figure 6: Minimum Angle Subspace Classifier (MASC) accuracy the layers of the network when the data set is projected onto corrupted subspace and subspace corresponding to true training labels. Rows corresponds to plots which have the same corruption degree and the columns correspond to the models noted. Training and test accuracy of the model is shown. FC corresponds to fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation. The respective number of class-wise PCA components for true training label subspaces of the models is shown in Figure 32 SGD optimizer was used for training MLP models, whereas Adam optimizer was used for other models. ResNet-18 has layer outputs of size 16,384 for LO-L1, followed by 8,192 (L2), 4,096 (L3), 2,048 (L4), and 512 [ovg_pool) layer.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy on 0% corrustion
Testing Accuracy
Model Testing Accuracy
MLP-MNIST
CNN-Fashion-MNIST
AlexNet-CIFAR-100
1.0
0.8
0.6
0.4
0.2
1.0
20% comppsio degres
0.8
0.6
0.4
0.2
0.0
1.0
0.8
40% comuption
0.6
0.4
0.2
corruption Accuracy
1.0
0.8
0.6
0.4
0.2
0.0
1.0
80%
1.0
0.8
0.6
0.4
100%
0.2
0.0
Input
FC (128)
FC (512)
PC (2048)
FC (2048)
Input
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 7: MASC test accuracy over the layers of the network when the data set is projected onto subspace corresponding to true training labels. Best model test accuracy corresponds accuracy of the test data of the model if early stopping was used.
16

---

Published in Transactions on Machine Learning Research (02/2026)
The Appendix presents MASC comparison results across different PCA variance thresholds, along with an analysis of MASC's computational and time complexity. The Appendix also describes a control experiment with MASC accuracies on a random initialization of the network. We have results corresponding to MLP trained on CIFAR-10, CNN trained on MNIST and CIFAR-10, and AlexNet trained on Tiny ImageNet in the Appendix for all the experiments. We also have a section comparing MLP models trained on MNIST and CIFAR-10 with SGD and Adam optimizer.
In Section H also investigating the latent memorization capabilities of uncorrupted models. Here, con- versely, we ask how well a network trained on true labels can manifest memorization of an arbitrary relabeling of its training data. More specifically, we built MASC classifier on model trained on true training labels, with the goal of memorizing training data whose labels are corrupted to varying degrees post hoc. Interest- ingly, we find a dichotomy in model behavior here, with some models trained on specific datasets having the propensity to memorize a high degree, whereas others not demonstrating such ability.
7 Leveraging MASC to retrain the base memorized model for improved generalization to true labels
Taking into account the better generalization to true labels using MASC on memorized models, in this section, we ask the following question. Can we use the MASC classifier to retrain the existing model to achieve better generalization to true labels?
Details of the pipeline for retraining existing models, leveraging MASC are already presented before. For different corruption degrees, test accuracy 11 before and after retraining with relabeled data using MASC (corrupted and true subspaces) for MLP--NIST. MLP-CIFAR-10, CNN-MNIST, CNN-Fashion-MNNIST
CNN-CIFAR-10 models are shown in Figure g Similar results for AlexNet-CIFAR--UI AlexNet-Tiny Ima- geNet models are shown in Figure 42
In order to study the dynamics of accuracy during retraining, unencumbered by the carly stopping criterion, we also performed similar experiment without using early stopping, for 10 epochs. The results for model before training, model after retraining MASC corrupted subspace, and model after retraining on MASC subspace with true labels over the 10 epochs for all model-dataset pairs with various corruption degrees are shown in Figure 46
In Figure and 42
we find that for some models MNIST, CNN-MNIST, CNN-Fashion-MNNIS
CIF-RIFA with non-zero corruption degrees, there an improvement in the test accuracy of models retrained using relabolling with MASC on corrupted subspaces, in comparison to the models' test accuracy before retraining (existing models). Indeed, in some cases, the improvement is quite significant, especially for larger corruption degrees (that are below 100% corruption degree).
However, for some models IMLP-CIFAR- appear marginal. Indeed, in some cases (MLP-CIFARE) AlexNet-Ting ImageNet for 20% corruption degree), there is a decrease in the test accuracy with such retraining. In order to study why, we checked the fraction of incorrect labels in the relabeled training dataset and compared it with the same measure for the existing corrupted training dataset. For the corruption degrees 20%, 40%, 60%, and 80%, these results are plotted in Figure 43 Table lists the exact values of the same.
CIeAR10-- the accuracy gains due to such retraining
In particular, it lurns out that for MLP-CIFAR-10 and AlexNeL-CIPAR-100 this fraction is almost equal to the fraction on the existing corrupted dataset; for MLP- 1L the fraction is marginally higher for the relabeled dataset and for AlexNet-CIFAR-100 it is marginally lower. This simply implics that the MASC classifier on the best layer that uses corrupted subspaces does roughly as well on the training set with true labels as the existing model, while surprisingly, the same MASC classifier is able to perform significantly better than the existing model on the test-set with true labels (See light orange bar in Figure S and 42) With regard to retraining, it would seem that the relabeled training isn't more effective than the existing corrupted training set in training the model, which possibly reflects in the lack of significant inprovement in test accuracy. More broadly, this suggests that MASC's better generalization to true labels isn't 't necessarily
Note that the test accuracy on the models calculated with respect to the 80% test. dataset.
17

---

Published in Transactions on Machine Learning Research (02/2020)
Model retraining
Model
Model retraining MASC subspace
accurccy layer :Comupted
accurncy
0% corruption degree
20% corruption degree
40% corruption degree
0.8
0.6
0.4
Test Accuracy
60% carruption degree
80% corruption degree
100% corruption degree
0.4
0.2
MLRMMIST
MLA.CAA-IO
CAN ANNSI
Fachien MNIST
OAN CRAR-10
MUR-MNIST
MLRCHARR 1O
CRN CRA-MMIST
CWN Woi-Fashiion.aa HW-FRshion-HNSTT
Oity CHAR-C
MLP.ANIST
MLR. M.R.CRRA1
CAN MNIST
OWN Frr--Rahhn-maa MRIST
CRA CRA-CIPAR CIRAR
Model-dataset
Figure 8: Test accuracies averaged over three runs on the 80% test dataset is plotted for different model- dataset pairs for various corruption degrees and for various models/MASC, MASC classifiers. Model before retraining corresponds to the existing memorized model. Model after retraining on MASC corrupted subspace corre- sponds to model trained with training dataset relabeled using MASC corrupted subspace predictions on the best layer. Model after retraining on MASC true subspace corresponds to model trained with training dataset relabeled using MASC subspacos corresponding to true label predictions the best layer. MASC test accuracy on the best layers for corrupted and true label subspaces existing corrupted models (before retraining) are shown for comparison. The best layer was identified using validation set that was carved out of the test dataset, for this experiment. Error bar represents the range on three different runs.
accompanied by better training set performance on true labels. This phenomenon requires a more detailed
future investigation.
Secondly, for AlexNet-Tiny ImageNet with 20% corruption degree, the relabeled training set has significantly fewer fraction of correct labels (92.46% lower) than in the existing corrupted training set. We wanted to determine to what extent this lower fraction is driven by previously incorrect label predictions (in the existing corrupted training dataset) being predicted correctly (in the relabeled set), versus previously correct predictions being relabeled incorrectly. For all models, these numbers are visualized in Figures 44 & 45 and Tables 10 & 11 list. corresponding numbers. We find for the case of AlexNet-Tiny ImageNet with 20% corruption degree that there a small fraction (5.94%) of previously incorrect labels that are correctly relabeled and large fraction (24.46%) of correctly labeled points that are incorroctly labclod by the MASC classifier trained on the corrupted label subspaces. Notably, even though this MASC classifier while doing significantly worse the training data than the model happens to do markedly better than the model in
18

---

Published in Transactions on Machine Learning Research (02/2026)
test accuracy. As before, we think that the poor performance of the retrained model is driven by the fact that relabeling results in dataset with larger fraction of incorrect labels.
Thirdly, in some cases (e.g,, AlexNet-Tiny ImageNet with 40%, 60%, 80% corruption degree), we observe that even though relabeling results in a somewhat larger fraction of incorrect labels, the test accuracy of the retrained model is slightly better. This suggests that retraining performance not simply driven by fraction of correct labels, but that specifics of which points are relabeled can drive retraining performance in ways that remain to be investigated.
With respect to models retrained on MASC true label subspaces, it was observed that the test accuracy of such models usually performed significantly better than the models before retraining.
By-and-large, we find that the retrained models that use MASC corrupted subspace relabeling don't have better test performance than the corresponding MASC classifiers. However, it would be interesting to apply MASC on the retrained models to see if that would further improve generalization to true labels performance.
8 Discussion
In this work, we investigated the phenomenon of memorized networks not generalizing well, asking why the ability to generalize is apparently diminished due to the act of memorizing. We find, surprisingly, that the intrinsic ability to generalize remains present to degree not previously recognized, and this ability can be decoded from the internals of the network by forward means. On the one hand, we design probes that use the subspace geometry of the corrupted training data to decode such better generalization to true labels. We also demonstrate using true labels post hoc that there exist subspaces that allow for an even more improved decoding. Furthermore, we show that such decoding can be leveraged to retrain the models to have better generalization to true labels. We also show (Appendix) that the internal representat ions of some deep networks trained on true labels, possess the ability to substantially memorize relabelings of its training data. Moreover, the new type of probe - MASC - that we use here, is relatively lightweight computationally while being easy to implement and lending itself to a straightforward geometric interpretation.
In building MASC, we were motivated by the manifold hypothesis in machine learning Goodfellow et al. (2016) that posits that high-dimensional data typically reside on a low-dimensional manifold. It has also been suggested (Brahma et al. 2015) that such manifolds in layerwise representations flatten across layers of deep networks. Fitting manifolds can be computationally expensive, so were interested in examining the organization of classwise data in subspaces, even if such subspaces might be somewhat higher dimensional than the corresponding manifolds. Indeed, this view leads to the natural idea of classifying unseen data points by determining which class manifold it is closest to. MASC is simply a formalization of this idea. In particular, this classifier lends strong geometric motivation and intuition, in contrast to c.g. training a standard linear probe that iteratively minimizes a crossentropy loss. However, the reasons for the success of MASC in this setting are still largely unclear to us. The difficulty is that the principles that underlie the nature of layerwise representations in deep networks trained with standard techniques are not well understood at this time and it appears that such representations play a significant role in the success of MASC in the memorization setting. Indeed, it is even a bit surprising that the deep network does not directly leverage this structure to obtain better generalization to true labels, although that may also be because its loss function aims to maximize training accuracy which might run counter to the act of bettering generalization to true labels.
An interesting question is about why this phenomenon even occurs; naïvely one would expect that deep networks, on being trained with noisy data, discard the ability to generalize in favor of learning noise. Are there specific inductive biases that promote such generalization to true labels? And, do such mechanisms also promote generalization to true labels in networks whose training data isn't corrupted significantly by such noise? It would also be instructive to study the dynamics of this form of generalization to true labels during training. It is known (Arpit et al. 2017) that the model's test accuracy transiently peaks in the early epochs of training with corrupted data, before dropping while training accuracy of the corrupted training data rises. It is unclear whether this transient rise in model generalization to true labels is caused by the subspace organization seen here, & if so, why such subspace organization isn't degraded as much as the model's test
19

---

Published in Transactions on Machine Learning Research (02/2026)
accuracy over further epochs of training. Additionally, certain deep networks trained on specific uncorrupted datasets seem to possess internal representations that are amenable to significant. memorization, whereas others aren't. The mechanistic basis of this ability is unclear & its possible connections to generalization to true labels in the such models merit further investigation.
Tt interesting that probes other than MASC also often have generalization to true labels comparable to MASC, and this performance often manifests in layers different from those that have best accuracies of MASC. The reasons for this are unclear and remain to be investigated.
The work has a number of implications. On the one-hand, it suggests that the ability to memorize and generalize may not be antithetical. Indeed, in multiple cases, we are able to construct single MASC classifiers that perform well both on the shuffled training labels as well as on the held-out test data that has true labels. Secondly, theories proposed to explain generalization to true labels in deep networks have traditionally argued for the setting where the data distribution is well-bchaved, i.c. corresponding to real-world data, but not for data with shuffled labels. We suggest, in light of the present results, that such theories also ought to be able to explain why networks retain the ability to generalize even in the face of noisy training data. That is, a satisfactory understanding of generalization to true labels in deep networks should also cover settings where the training data is noisy and its distribution is not well-behaved. Thirdly and more pragmatically, techniques such as the MASC classifier might suggest a way of boosting generalization to true labels in trained deep networks, whose training data intrinsically contains varying degrees label noise. While this has been beyond the scope of the present paper, possibilities of designing new techniques for learning subspaces that have good generalization to true labels could be explored. Indeed, it is possible that significantly better subspaces exist than the ones uncovered here, & it would be interesting to see how much the generalization to true labels accuracy can be improved by pursuing this direction. Relatedly, it is possible that other classifiers operating on layerwise outputs have better performance than MASC a possibility that merits further exploration. Fourthly, it would be interesting to formulate a measure to study roprosentational similarity between memorized & generalized networks to see if they use similar mochanisms. Does the answer depend on the particular class of networks (e.g. MLPs vs. CNNs)?
Finally, the results here are reminiscent of a puzzling phenomenon observed in Neuroscience. In multiple scttings (Miura et al., 2012 Stringer et al.] 2021), in the rat olfactory system and the mouse visual system, it has been shown that decoder using data from subset of neurons from specifie areas in the brain of a well-trained behaving animal has accuracy significantly better than the behavioral accuracy of the animal on novel trials, even though the animal is motivated to do well on the task. This implies that those animals have better innate generalization to true labels ability on that task - which can be easily decoded from a subset of their neurons than is manifested by their behavior. may therefore be that this is a phenomenon shared between brains and machines, whose underlying mechanisms and potential trade-offs remain to be investigated.
Broader impact statement
The Minimum Angle Subspace Classifier (MASC) is primarily an analytical probing method, and its direct societal risks are therefore limited. However, caution is advised when applying it to models trained sen- sitive proprictary data. In addition, probe-derived labels should not be repurposed for model retraining without appropriate safeguards, as this may reinforce existing biases propagate systematic mislabeling; robust validation procedures, uncertainty thresholds, and human oversight are recommended. Finally, the diagnostic performance of MASC should not be interpreted as cquivalent to the underlying model's gen- eralization to true labels ability. MASC's results are about understanding representations and not about predicting how the model will perform when deployed.
Acknowledgments
Simran Ketha was supported by an APPCAIR Fellowship, from the Amuradha Prashanth Palakurthi Centre for Artificial Intelligence Research. The work was supported in part by an Additional Competitive Research Grant from BITS to Venkatakrishnan Ramaswamy. The authors acknowledge the computing time provided on the High Performance Computing facility, Sharanga, at the Birla Institute of Technology and
20

---

Published in Transactions on Machine Learning Research (02/2026)
Science Pilani, Hyderabad Campus. We thank Harsha Varun Marisetty for compute assistance for training some of the models used here.
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. 2018. arXiv preprint arXic:1610.01644 2018.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pp. 233 242. PMLR, 2017.
Amr Bakry, Mohamed Elhosseny, Tarek El-Gaaly, and Ahmed Elgammal. Digging deep into the layers of cmns: In search of how cmns achieve view invariance. arXiv preprint arXiv: 1508. 1500.0188,,,, 2015.
Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: A manifold disen- langlement perspective. IEEE transactions neural nelworks and learning systems, 27(10):1997 1900-00000-- 2015.
N Alex Cayco-Gajie and R Angus Silver. Re-evaluating circuit mechanisms underlying pattern separation. Newron, 101 (4):584-602 2019.
Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Linear readout of object manifolds. Physical Review E, (666000,, 2016.
Uri Cohen, YueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geomelry of object manifolds in deep neural networks. Nature communications, 11(1):746, 2020.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141 142, 2012.
Freeman Dyson et al. meeting with enrico fermi. Nature, 427(6972):297 297, 2004.
Matthew Farrell, Stefano Recanatesi, Guillaune Lajoie, and Eric Shea-Brow Brown. Dynamic compression and expansion in classifying recurrent network. bioRriv, pp. 564476, 2019.
Benoît Frénay, Ata Kabán, et al. A comprehensive introduction to label noise. In ESANN. Citeseer, 2014.
Peiran Gao and Surya Ganguli. On simplicity and complexity in the brave new world of largo-scale neuro- science. Currenl opinion in neurobiology, 32:148 155, 2015.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://wwa.
deeplearningbook.oo org.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust. training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference computer vision and pattern recognition, pp. 770-778, 2016.
Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, :284):984 991, 2019.
Kevin Jarrett, Koray Kavukcuoglu, Marc' Aurelio Ranzato, and Yann LeCun. What the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference on computer vision, pp. 2146 2153. IEEE, 2009.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304-2313 PMLR, 2018.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. arXiv preprint or 1ix::111220000 04577,, 2017.
Diederik P Kingma. Adam: A method for stochastic optimization. arviv prepriul 1. 2014.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519-3529. PMLR, 2019.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprint Xix:::1809. 10374, 2018.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pp. 4313-4324. PMLR, 2020.
Ashok Litwin-Kuman Kameron Decker Harris, Richard Axel, Haim Sompolinsky, and LF Abbott. Optimal degrecs of synaptic connectivity. Neuron, 93(5):1153-111 1164, 2017.
Sheng Liu, Jonathan Weed, Narges Razavian, and Carlos Fernandez-Granda Early-learning regular- ization prevents memorization of noisy labels. Advances in newral information processing systems, 33: 20331 20331-20342, 2020.
Eran Malach and Shai Shalev-Shwartz. Decoupling' when to update' from' how to update". Advances in neural information processing systems, 30, 2017.
Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. Learning from binary labels with instance-dependent noise. Machine Learning, 107:1561-1595, 2018.
Keiji Miura, Zachary F Mainen, and Naoshige Uchida. Odor representations in olfactory cortex: distributed rale coding and decorrelated population activily. Newron, 74(6):1087 1098, 2012.
Grégoire Montavon, Mikio Braun, and Klaus-Robert Müller. Kernel analysis of deep networks. Journal of Machine Learning Research, 12(9), 2011.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
Mohammed Ali Moustafa.
Tiny imagenet, 2017.
URL atttss//aaggee.coo//mppttttttttt
tiny-imagenet]
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145-151, 1999.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in newral information processing systems, 30, 2017.
Scott. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping, arXiv preprint orXiv:1,12. 2.996, 2014.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pp. 4334 4343. PMLR, 2018.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. Xiv preprint arXio:1312. 6120, 2013.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Conference on learning theory, pp. 489-511 PMT.R, 2013.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. Tn International conference machine learning, pp. 5739-5748. PMT.R, 2019.
Cory Stephenson and Tyler Lee. When and how epochwise double descent happens. or preprint arXiv:2108. 12006, 2021.
Cory Stephenson, Abhinav Ganesh, Yue Hui, Hanlin Tang, SueYeon Chung, et al. On the geometry of generalization and memorization in deep neural networks. In International Conference on Learning Rep- resenlations, 2021.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High- dimensional geometry of population responses in visual cortex. Nature, 571 (3766))333611 2019.
Carsen Stringer, Michalis Michaelos, Dmitri Tsyboulski, Sarah E Lindo, and Marius Pachitariu. High- precision coding in visual cortex. Cell, 184(10):2767- 2778, 2021.
David Sussillo and Larry F Abbott. Generating coherent, patterns of activity from chaotic neural networks. Newron, 63):5444 557, 2009.
Lina M Tran, Adam Santoro, Lulu Liu, Sheena A Josselyn, Blake A Richards, and Paul W Frankland. Adult neurogenesis acts as neural regularizer. Proceedings of the National Academy of Sciences, 119(45): 22206704119, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017. URL htttps://arrtt http:///wwwwwwwwwaaaa/ee trrg/bbs/6611.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
23

---

Published in Transactions on Machine Learning Research (02/2026)
A
Model details
Multilayer Perceptron (MIP) model has 4 hidden layers with 128, 512, 2048 and 2048 units respectively, ReLU activation was used after every layer and for classification softmar activation was applied. We have trained the models with two different optimizers namely, SGD and Adom. Learning rate of 1 x 10-3 and momentum = 0.9 was used with SGD optimizer. Learning rate = 1x100 was used for Adam in experiments. Batch size of 32 was used in all the models. The dataset was normalized by dividing each pixel value with 255.
Convolutional Neural Network (CNN) network has 3 blocks, each consisting of two convolutional layers, one max pooling layer. These blocks are followed by three fully connected layers. Convolutional layers have 16, 32, and 64 filters, respectively with stride=1 and filter size = 3 x 3. Max pooling layer has stride of and filter size of 2 x 2. The fully connected layers at the end has 250 units cach. was trained with Adam optimizer with learning rate 0.0002. For MNIST and Fashion-MNIST batch size 32 whereas for CIFAR-10 10 batch size of 128 were used. The dataset was normalized by subtracting the mean and diving by the standard deviation for each channel. ReLU activation was used after every layer except pooling and softaa activation for classification.
AlexNet model was slightly modified for the use of each dataset. Adam optimizer with learning rate of 0.0001 was used. For CIFAR-100, batch size of 128 and for Tiny ImageNet, batch size of 500 was used. All the results with respect to test on AlexNet trained on Tiny ImageNet are shown with the validation dataset. CIFAR-100 dataset. before training was normalized by subtracting the mean and diving by the standard deviation for each channel. No data normalization was performed on Tiny ImageNet dataset.
ResNet-18 model slightly modified for the use of CIFAR-10 dataset. SGD optimizer with learning rate of 0.001 and momentum of 0.9 was used. Batch size of 32 was used for training. The dataset was normalized by subtracting the mcan and diving by the standard deviation for each channel.
The experiments were performed on workstations/set sitiaaatiionsseversssssss servers with a variety of GPUs, including Nvidia GeForce RTX3080s, GeForce RTX3090s, Tesla V100s and A100s.
A summary of the models and datasets with training set size and number of parameters is in Table 2 The average training and test accuracies of the models over three different runs used in the paper are shown in Tables 3 and
Table 2: Training set size of the datasets and the number of parameters of the models.
Training set size
Number of parameters
Model
Dataset
MLP
MNIST
60,000
5,433,994
CIFAR-10
50,000
5,726,858
CNN
MNIST
60,000
344,042
Fashion MNIST
60,000
344,042
CIFAR-10
50,000
456,330
AlexNet
CIFAR-100
50,000
88,7389952
Tiny ImageNet
100,000
39,776,466
R8sNet-18
CIFAR-10
50,000
11,173,962
A.1 General terminology
The general terminology used in this work is as follows:
Model Training Accuracy: The accuracy of the model on the training set with corrupted labels.
Model Test Accuracy: The accuracy of the model the test dataset with true labels.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Average training accuracy in percentages of all the models over three runs over different corruption degrees (indicated in the last six columns). The values are rounded to the second decimal place.
Model
Dataset
Parameter
0%
20%
40%
60%
80%
100%
MLP
MNIST
SGD
99.99
99.99
99.99
99.99
100.0
100.0
Adam
100.0
99.87
99.73
99.77
99.73
99.66
CIFAR-10
SGD
99.99
99.99
99.99
99.99
99.99
99.99
Adam
99.63
99.53
99.43
99.61
99.52
30.21
CNN
MNIST
Adam
99.90
99.32
98.62
97.25
95.11
94.92
Fashion-MNIST
Adam
99.15
99.14
97.90
96.25
91.65
83.14
CIFAR-10
Adam
99.70
99.29
99.26
99.03
99.02
39.69
AlexNet
CIFAR-100
Adam
99.19
99.15
99.11
99.16
99.14
97.88
Tiny ImageNet
Adam
99.92
99.90
99.91
99.93
87.71
85.95
ResNet-18
CIFAR-10
SGD
100.0
100.0
100.0
100.0
100.0
Table 4: Average test accuracy in percentages of all the models over three runs over different corruption degrees (indicated in the last six columns). The values are rounded to the second decimal place.
Model
Dataset
Parameter
0%
20%
40%
60%
80%
100%
MLP
MNIST
SGD
97.87
87.38
73.28
54.16
32.09
9.81
Adam
98.31
90.49
76.78
55.59
31.85
9.77
CIFAR-10
SGD
56.37
48.62
40.35
30.55
19.68
9.80
Adam
52.24
44.11
35.55
25.24
15.18
10.07
CNN
MNIST
Adam
99.15
87.51
69.44
47.10
28.30
9.85
MaSIIO
Adam
90.74
77.74
61.35
43.26
25.57
10.08
CIFAR-10
Adam
74.95
60.48
46.15
30.96
18.32
9.89
AlexNet
CIFAR-100
Adam
36.75
28.44
20.53
9.64
3.43
0.96
Tiny ImageNet
Adam
15.88
9.74
5.44
2.02
0.73
0.43
ResNet-18
CIFAR-10
SGD
80.02
67.39
51.53
34.48
20.02
Minimum Angle Subspace Classifier (MASC) Accuracy Corrupted Training: Using Algorithm 1 (main paper), the accuracy of MASC predicted class labels with respect to corrupted labels of training datasct.
Minimum Angle Subspace Classifier (MASC) Accuracy Original Training: Using Algorithm (main paper), the accuracy of MASC predicted class labels with respect to true labels of training dataset.
Minimum Angle Subspace Classifier (MASC) Accuracy Test: Using Algorithm 1 (main paper), the accuracy of MASC predicted class labels with respect to true labels of test dataset.
B Minimum Angle Subspace Classifier
A schematic illustrating the retraining process using Minimum Angle Subspace Classifier (MASC) is shown in Figure
25

---

Published in Transactions on Machine Learning Research (02/2020)
80% Test dataset
Stop 2: Use the 20% test datatset to identify the best layer (MASC accuracy)
1: the test dataset
Class
Class
Class
Clsss
Testing
Best layer MASC
Class
20% Test datasot
Minimum Angle Subspace Classifier
Step 3; Relabel the corrupted training dataset using the best layer MASC predictions
Relabelied training dataset
Corrupted training dataset
Best layer
Class
Clasa
Class
Class 3
Clss
using relabelled training dataset
Minimum Angia Subspace Classifler
label predictions
Relebelled training dataset
Class
Class
Class
Clocss
Model
20% Test dataset used for validstion
Class
Class
Class
Clss
Step
model
Model
lnye
layers
Curput layer
80% Test dataset
Test accuracy
Figure schematic of the memorized model retraining pipcline that leverages MASC. Step 1: We first split the lest dataset into 80%-20%. Step 2: We the MASC accuracy the corrupted subspaces on the 20% of the test dataset to identify the model's best-layer. Step 3: Using the best-layer MASC predictions, relabel the corrupted training dataset. Step 4: We train with the relabeled corrupted training dataset for upto 30 epochs and perform carly stopping with patience of 3 by considering the 20% test dataset as a validation dataset. similar process was followed while working with subspaces corresponding to true labels. Step 5: The test accuracy on the models calculated with respect to the 80% test dataset.
26

---

Published in Transactions on Machine Learning Research (02/2026)
C Generalization to true labels comparison: MASC vs. other probes
C.1 Experiment setup
For comparison, we evaluated five probes across all network layers: Logistic Regression (LR), K-Nearest Neighbors (KNN), Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), and Near- est Class Mean (NCM).
For LR, we load the full corrupted training and test layer outputs along with labels, construct respectively data loaders with batch size 128, and train PyTorch logistic regression model for 20 epochs using the Adam optimizer (learning rate 1 x 10 10-3) with cross-entropy loss. The test loader is used for inference.
For NCM, we load the corrupted training and test layer outputs along with labels, compute class-wise mean vectors from the training data, and classify each test example by cosine similarity to these class means.
For LDA and QDA, we load the same training and test layer outputs along with labels. We use scikit-learn library to build and train the corresponding models the corrupted training layer outputs. Evaluating is performed using the test layer outputs.
For KNN, we use scikit-learn library to build the model with neighbors=5 and cosine distance, following the same training and evaluation protocol as LDA.
C.2 Results on additional models
Test accuracy of various probes over the layers of the MLP trained on CIFAR-10, CNN trained on MNIST and CIFAR-10, and AlexNet trained on Tiny ImageNet are shown in Figure 10 MASC test accuracy was overlaid for comparison.
C.3 Computational cost across layers
The computational costs (in GFLOPs) for the various probes over the layers of the network are presented in Figure I for MLP -NIIS, CNN Pashion-MNIST, and AlexNet CIFAR-100, and in Figure 12 for the remaining models.
C.4 Comparing generalization to true labels of MASC with logistic regression and nearest class mean
Test accuracy of Logistic Regression (LR), Nearest Class Mean (NCM) and MASC over the layers of the network for MLP-MNIST, CNN-Fashion-MNNSST and AlexNet-CIFAR-100 are shown in Figure 13 and for MLP-CIFAR-10, CNN-MNIST, CNN-CIFAR-10, and AlexNet-Tiny ImageNet are shown in Figure 14
The three probes exhibit distinct behaviors across network layers and corruption degrees. For AlexNet-CTFAR-1D and MIP--NNIST at 20% and 40% corruption, MASC outperforms both NCM and LR most layers. In contrast, for CNN-based models, NCM generally achieves higher accuracy than MASC across the majority of layers, while MASC lypically performs belter than LR. However, there are in- stances where MASC matches or surpasses NCM for example, in CONNMNIIS and Fashion-MNISS at 20%, 40%, and 60% corruption degroes, at the FLAT (576) layer.
D MASC comparison results with various PCA variance thresholds.
In this section, we compare MASC performance using subspaces constructed with different variance thresholds: 99%, 75%, 50%, 25%, and a single principal component (1PC). Results for MLP MNIST, NexNet-CIFAAR-100
CNN-Fashion- Fashion-MNIST, and AlexNet- CTFAR-100 are presented in Figure 15 with additional model results
shown in Figure 16
For these experiments, a single trained model was used, and the MASC computations for the variance comparison repeated across different. seeds. To assess robustness with respect to randomness,
27

---

Published in Transactions on Machine Learning Research (02/2026)
Model
NCM k-NN probe
LDA probe
MASC probe
Logistic regression probe
QDA probe
MLP-CIFAR-10
CNN-MNIS
CNN-CIFAR-10
AlexNet-Timg ImageNet
1.0
0% corruption dogree
0.8
0.6
0.4
0.2
0.0
1.0
20% coruuttinn degrac
0.8
0.6
0.4
0.2
0.0
1.0
40% corruption degrec
0.8
Test Accuracy
0.6
0.4
0.2
0.0
1.0
60% corruption degree
0.8
0.6
0.4
0.2
0.0
1.0
80% corruttion degroe
0.8
0.6
0.4
0.2
0.0
FC (128)
FC (512)
FC (2048)
FC (2048)
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (1024)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 10: Test accuracies for different probes over the layers of the network. Rows corresponds to plots which have the same corruption degroe and the columns correspond the models as noted. Test accuracy of the model and MASC are shown for comparison. FC corresponds fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Logistic regression probe
LDA probe
K-NN probe
NCM probe
QDA probe
MASC probe
MLP-MNIST
CNN-Fashion-MNIST
MlexNet-CIFAR-100
5000
1250
4000
1000
30000
3000
750
20000
Computational cost (GFLOPs)
2000
1000
250
10000
0
0
40000
1000
80
30000
800
60
600
40
20000
400
200
20
10000
0
0.12
0.4
0.10
0.3
0.08
0.1
0.06
2
FC (120)
FC (512)
FC (2048)
FC (2048)
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 11: Computational costs (GFL.OPS) for different probes over the layers of the network. Computational costs (GFLOPS) of MASC are shown for comparison. FC corresponds to fully connected layer with ReLC activation whereas Flat corresponds to flatten layer without ReLU activation.
performed five runs using seeds 10, 20, 30, 40, 50, and report the results as the mean along with the 95% confidence interval (CI).
For most models, we observed that under non-zero corruption degrees, several variance thresholds across many layers outperform the model's generalization to true labels. However, no consistent. pattern emerges across all model-datase pairs the variance thresholds vary.
For MIP-MNIST and CNN- subspaces capturing variance of 75,, 50%, 25%, and even 1-PC often outperformed the subspace capturing 99% variance across multiple layers. In contrast, for AlexNet-CIFAR-UU CIFARR-00,, the 99% variance threshold performed better than the other thresholds at 20% and 40% corruption degrecs. Additionally, we found that the 1-PC subspace exceeded the performance of other thresholds several models.
E
Computational costs of MASC
In this section, we report total time taken (in seconds) and computation cost (GFLOPS) of MASC for building the subspaces and inference over the layers of the network, for multiple models/ models/datasen datasets shown in Figure 17 and Figure [8] The computational cost (GFLOPS) for building per-class subspaces are shown
in Figure 19
It was observed that, although inferened requires more computational operations (in GFLOPs) than sub- space construction, the total time taken for inference is nevertheless lower. This is likely because inference
By Inference in the of MASC, (s).
using the already determined subspaces classify incoming data
29

---

Published in Transactions on Machine Learning Research (02/2026)
Logistic regression probe
LDA probe
K-NN probe
NCM probe
QDA probe
MASC probe
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
4000
2000
80000
3000
1000
1500
60000
Computational cost (GFLOPs)
2000
750
500
1000
40000
1000
20000
0
80000
1000
250
800
60
200
60000
600
40
150
40000
100
200
20
20000
0
0
0
0.12
0.4
0.20
15
0.3
0.10
0.2
0.08
0.10
5
0.06
0.05
FC (128)
FC (512)
FC (2048)
FC (2048)
Rlat (576)
FC (250)
FC (250)
FC (250)
Flat (1024)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 12: Computational costs (GFL.OPS) for different probes over the layers of the network. Computational costs (GFLOPS) of MASC are shown for comparison. FC corresponds to fully connected layer with ReLC activation whereas Flat corresponds to flatten layer without ReLU activation.
computations are highly optimized and benefit more from hardware acceleration than the covariance and PCA steps involved in building subspaces.
We also find clear scaling trend with respect to layer dimensionality. Layers with higher feature dimen- sionality such as FC(2048), FC(4096), Flat(1024/576) - consistently require higher computational cost during subspace construction. In contrast, lower-dimensional layers (e.g., FC(128) or FC((50)) remain com- paratively inexpensive. This behavior aligns with the expected quadratic dependence of PCA based subspace construction on the feature dimension, and highlights that computational cost grows steadily as the layer dimensionality increases.
F
Experiments with AlexNet model trained on Tiny ImageNet
MASC test accuracy over the layers of AlexNet trained on Tiny ImageNet when the data projected onto corrupted training subspaces with the indicated corruption degree is shown in Figure 20 Test accuracy of the model and best model test accuracy is shown for comparison. Best model test accuracy corresponds accuracy of the test data of the model if early stopping was used.
Even for AlexNet-Tiny ImageNet corrupted model (with non-zero corruption degree), except those with 100% corruption degree, we find that our Minimum Angle Subspace Classifier (MASC) in at least. one layer has better test accuracy than the corresponding model itself.
MASC test accuracy over the layers of AlexNet trained on Tiny ImageNet when the data set projected onto corrupted training and true training subspace is shown in Figure 21
30

---

Published in Transactions on Machine Learning Research (02/2026)
Model
Logistic regression probe
NCM probe
probe
MASC probe
MLP-MNIST
CNN-Fashion-MNIST
AlexNet-CIFAR-100
1.0
1.0
0% corruption dogros
0.8
0.8
0.42
0.40
0.6
0.6
0.38
0.4
0.4
0.36
0.2
0.2
0.34
0.0
0.0
0.32
1.0
1.0
0.38
20% corruption degree
0.8
0.8
0.36
0.6
0.6
0.34
0.4
0.4
0.30
0.2
0.2
0.28
0.26
0.0
0.0
1.0
1.0
0.32
40% corruption degree
0.8
0.8
0.30
Test Accuracy
0.28
0.6
0.6
0.26
0.4
0.4
0.24
0.22
0.2
0.2
0.20
0.18
0.0
0.0
1.0
1.0
60% corruption degree
0.8
0.8
0.16
0.6
0.6
0.14
0.4
0.4
0.12
0.2
0.2
0.10
0.0
0.0
1.0
1.0
0.08
80% corruption degret
0.8
0.8
0.07
0.6
0.6
0.06
0.4
0.4
0.05
0.2
0.2
0.04
0.03
0.0
0.0
FC (128)
FC (512)
FC (2048)
FC (2048)
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 13: Test accuracies for different probes over the layers of the network. Rows corresponds to plots which have the same corruption degree and the columns correspond to the models as noted. Test accuracy of the model and MASC are shown for comparison. FC corresponds fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation.
31

---

Published in Transactions on Machine Learning Research (02/2026)
Model
Logistic regression probe
NCM probe
MASC probe
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
1.0
0.19
0.18
0% corruption dogree
0.8
0.8
0.16
0.6
0.6
0.6
0.15
0.4
0.4
0.4
0.14
0.13
0.2
0.2
0.2
0.11
0.0
0.0
0.0
1.0
1.0
1.0
0.13
20% cor cerruption degrac
0.8
0.8
0.8
0.12
0.11
0.6
0.6
0.6
0.10
0.4
0.4
0.4
0.09
0.08
0.2
0.2
0.2
0.07
0.0
0.0
0.0
0.06
1.0
1.0
1.0
40% corruption degrec
0.8
0.8
0.8
Test Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.07
0.2
0.2
0.2
0.06
0.0
0.0
0.05
1.0
1.0
1.0
0.045
60% com corruption degree
0.8
0.8
0.8
0.040
0.6
0.6
0.6
0.035
0.4
0.4
0.4
0.025
0.2
0.2
0.2
0.020
0.0
0.0
1.0
1.0
1.0
0.014
80% corruptien dogree
0.8
0.8
0.8
0.013
0.012
0.6
0.6
0.6
0.011
0.010
0.4
0.4
0.4
0.009
0.008
0.2
0.2
0.2
0.007
0.0
0.0
0.006
FC (128)
FC (512)
PC (2048)
FC (2048)
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (1024)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 14: Test accuracies for different probes over the layers of the network. Rows corresponds plots which have the same corruption degree and the columns correspond the models as noted. Test accuracy of the model and MASC are shown for comparison. FC corresponds to fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation.
32

---

Published in Transactions on Machine Learning Research (02/2026)
G Random initializations of control models versus trained models
This section covers a set of control experiments to show MASC performance on random initialized model and contrast this with the trained models presented in the main text. We have verified that, for every such control model, the model training and test accuracies for the randomly initialized models is at chance level, for the corresponding dataset in question.
MASC accuracy on test for randomly initialized model and trained model when data is projected on corrupted training subspaces is shown in Figure 22 and 24 Trainod model training and test accuracics are shown for reference.
We find that indeed accuracies of the MASC classifier on the random initialization outperforms the network, except for low corruption degrees (i.c. <= 20% corruption degree). However, in the experiments where subspaces are trained on corrupted training data from corrupted models, by-and-large, the MASC classifier usually, and on at least one layer outperforms the MASC classifier trained on the random initialization with exceptions being the 80% corruption degree models on MLP-MNIST, AlexNet-Tiny ImageNet and 100% corruption degree on CNN-Fashion-MNTE
MASC accuracy on test for random initialized model and trained model when data is projected on sub- spaces corresponding true training labels is shown in Figure 23 and 25 Notably, for the experiments where subspaces are constructed with true labels on corrupted models, the MASC classifier these models outperforms the MASC classifier on random initializations usually and certainly in at least one layer on every model tested. These results are consistent with the main message of the paper, namely that even with memorized models, the layerwise representations of the models are organized in a manner that they develop significant ability to generalize over and above that bestowed by a random initialization, and in particular, they do not lose this ability, as one might have naively expected, due to label noise. If they were losing this ability, then the MASC classifier on the subspaces would end up performing significantly worse than the MASC classifier
randomly initialized models.
Although it is interesting that random projection have good generalization to true labels, it is not surprising as this has been shown (Alain & Bengio 2018 and also studied (Jarrelt el al. 2009
H Investigating latent memorization capabilities of uncorrupted models
Conversely, we ask how well network trained on true labels can manifest memorization of an arbitrary relabeling of its training data, More specifically, we built MASC classifier on a model trained on true training labels, with the goal of memorizing training data whose labels are corrupted to varying degrees post hoc.
To do this, we shuffle the labels of the training set to some corruption degree and construct the corresponding class-specific subspaces with respect to the layerwise outputs of the generalized models, i.e. models trained with uncorrupted training data. We then build a MASC classifier corresponding to these subspaces.
MASC accuracy on corrupted training data, original training data and MASC accuracy on test data over the layers of the networks are shown in Figure 26 We have results on additional models, namely MLP trained on CIFAR-10, CNN trained on MNIST and CIFAR-10, and AlexNet trained on Tiny ImageNet; see Figure 29 Tn Table 5 we show by what percentage the MASC classifier outperformed the model train accuracies on the corrupted labels for the best layer for corruption degrees 20%, 40%, 60% and 80%.
Interestingly, we find dichotomy in model behavior here, with some models trained on specific datasets hav- ing the propensity to memorize to high degree, whereas others not demonstrating such ability. Specifically, we observe that for uncorrupted models with low/ modest model test accuracies (i.c. AlxxNet-CIFAR- 100 and AlexNel-Tinn ImageNet), the MASC classifiers described above have high accuracies on the corrupted training set (i.e. appear to be able to memorize with high accuracies). Conversely, in most uncorrupted models with high model test accuracies (i.c. MIIST and CNN--Fsshion-MNTSTT we find that these MASC classifiers have more modest accuracies on the training set with corrupted labels (i.e. do not memorize to high degree). This suggests the hypothesis that. high generalization to true labels inhibits the innate
33

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: Percentage by which the MASC classifier (run on the best layer) outperformed the model's train accuracy on corrupted labels. The model is trained on images with true training labels. MASC classifier is operating on subspaces corresponding to corrupted training data. The best layer corresponds to the one that has the highest measured MASC train accuracy among the layers for the said model The accuracies in cach case are averaged over three runs. The values are rounded to the second decimal place.
Corruption degree
20%
40%
60%
80%
MLP-MNIST
-2.74%
-2.35%
-1.81%
3.11%
MLP-CIFAR-10 (Appendix)
18.4%
51.28%
112.81%
255.93%
CNN-MNIST (Appendix)
-0.24%
0.40%
3.50%
26.76%
CNN-Fashion-MNNTST
-5.33%
-1.01%
7.79%
50.05%
CNN-CIFAR-10 (Appendix)
13.17%
42.65%
101.38%
243.12%
AlexNet-CIFAR,
27.91%
70.03%
152.78%
394.89%
AlexNet-Ting ImageNet (Appendix)
23.19%
63.18%
144.52%
389.85%
ability of models to memorize and conversely that low hww/meeerate generalization to true labels is accompanied by representations that offer greater propensity memorize, at least as manifested by the MASC classifier, However, more work is needed to study this phenomenon to understand the tradc-offs and the mechanisms underlying them, which has been beyond the scope of the present work. Also, surprisingly, we find that MASC classifiers often have test accuracies that approach excoed uncorrupted model test accuracios, even though they correspond to corrupted subspaces (see e.g. AlexNet-CIFAR-100 and Alex ImageNet)..
Experimental results on additional models (MLP-CIFAR-10, CNN-MNIST, CNN-CIFAR-10, AlexNet- Tiny ImageNet and ResNet-18-CIFAR-
In Table 6 we report the accuracy difference between the MASC classifier - when the data is projected onto corrupted training subspaces and the model for the best such layer.
Table 6: Accuracy difference between the MASC classifier (run on the best layer) and the model's test accuracy when the data is projected onto corrupted training subspaces. The model's test accuracy are reported in Table accuracy among the layers for the said runs. The values are rounded to the second decimal place. The values are rounded to the second decimal place.
The best layer corresponds to the one that has the highest measured MASC test The accuracies in each case averaged over three
Corruption degree
20%
40%
60%
80%
MLP-MNIST
9.55
23.18
40.65
51.33
MLP-CIFAR-10 (Appendix)
4.81
9.85
14.35
12.75
CNN-MNIST (Appendix)
8.59
25.72
46.49
56.91
CNN-Pashion-MMNIST
5.82
20.55
36.74
48.33
CNN-CIFAR-10 (Appendix)
1.39
2.89
8.37
11.02
AlexNet-CIFATE
9.55
10.90
6.26
1.54
AlexNet-Tiny ImageNet (Appendix)
2.68
2.91
0.92
0.10
ResNee-18-CIIAA--10
-0.28
10.48
23.20
23.86
In Table 7 we show by what percentage the MASC classifier (subspace corresponding to true labels) outper- formed the model for the best layer for corruption degroes 20%, 40%, 60% and 80%. In fact, the MASC test accuracies for the corrupted models (with non-zero corruption degree) are sometimes fairly close to the test accuracy of the uncorrupted model. Tn Table g also report the accuracy difference between the MASC
34

---

Published in Transactions on Machine Learning Research (02/2026)
classifier when the data is projected onto training subspaces corresponding to true labels and the model for the best such layer.
Table 7: Percentage by which the MASC classifier (run on the best layer) outperformed the model's test accuracy when the data projected onto training subspaces corresponding to true labels. The best layer corresponds to the one that has the highest measured MASC test accuracy among the layers for the said model/dataset. The accuracies in each are averaged over three runs. The values are rounded to the second decimal place.
Corruption degree
20%
40%
60%
80%
MLP-MNIST
10.96%
32.01%
77.77%
198.43%
MLP-CIFAR-10 (Appendix)
11.00%
30.27%
66.92%
146.09%
CNN-MNIST (Appendix)
10.69%
39.50%
104.97%
239.44%
CNN-PashionnMMNNST
8.37%
33.09%
88.97%
212.42%
CNN-CIFAR-10 (Appendix)
3.29%
13.43%
58.23%
138.65%
AlexNet-CTFAR-100
38.78%
68.91%
133.76%
337.51%
AlexNet-Tiny ImageNet (Appendix)
33.31%
83.85%
157.10%
212.33%
ResNee--8-CIIAR-110
2.92%
32.23%
93.55%
228.64%
Table Accuracy difference between the MASC classifier (run on the best layer) and the model's test accuracy when the data projected onto training subspaces corresponding to true labels. The model's test accuracy reported in Table The best layer corresponds to the that has the highest measured MASC test accuracy among the layers for the said model/dataset. The accuracies in each over three runs. The values are rounded to the second decimal place.
averaged
Corruption degree
20%
40%
60%
80%
MLP-MNIST
9.57
23.46
42.13
63.69
MT.P-CIFAR-10 (Appendix)
5.35
12.21
20.44
28.76
CNN-MNIST (Appendix)
9.36
27.43
49.45
67.77
CNN-Fashion-MNIITT
6.51
20.30
38.49
54.32
CNN-CTFAR-10 (Appendix)
1.99
6.20
18.03
25.40
AlexNetCCIFAR-100
11.03
14.15
12.90
11.58
AlexNet-Tiny ImageNet (Appendix)
3.25
4.57
3.17
1.55
ResNet-18-CIFAR.1
1.97
16.61
32.26
45.77
All the experimental results on additional models i.e, MLP-CIFAR-10, CNN-MNIST, CNN-CIFAR-10 and AlexNet-Tinn ImageNet are shown in this section.
MASC accuracy the layers of the MLP trained CIFAR-10, CNN trained MNTST, CNN trained CIFAR-10, and AlexNet trained on Tiny ImageNet when the data is projected onto corrupted training subspaces is shown in Figure 27 MASC accuracy over the layers of the MLP trained on CIFAR-10, CNN trained on MNIST, CNN trained on CIFAR-10, and AlexNet. trained on Tiny ImageNet when the data set is projected subspace corresponding to true training labels is shown in Figure 28 MASC accuracy over the layers of the generalized MLP network trained on CIFAR-10, CNN trained on MNIST, CNN trained on CIFAR-10 and AlexNet network trained on Tiny ImageNet when the data is projected onto corrupted training subspaces is shown in Figure 29
J Number of PCA components
This section covers the number of class-wise PCA components used in all the experiments.
35

---

Published in Transactions on Machine Learning Research (02/2026)
Number of class-wise PCA components of corrupted training subspace over the layer of MLP-MNIST, Fashion-MNIST, and AlexNet-CFFAR-100 is shown in Figure 30 and CAA.CCAAR--.,CCNNNNTIII
CNN- CIFAR-10 and Alex.Net-Tiny ImageNet is shown in Figure 31 For generalized models, it is observed that for 99% variance captured, the number of PCA components is significantly smaller in comparison to the ambient dimensionality of the layer (number of units in that layer). Over corruption, is observed that for MLP- MNIST, MLP-CIFAR-10, and CNN-Fashion-MNNSTT, the number of class-wise PCA components increase. And the variance between the number of dimensions decrease. For AlexNet-CIFARR-O0 and AlexNet-Tiny ImageNet, it is the opposite case, wherein the number of PCA components over corruption decreases.
Number of class-wise PCA components of original training subspaces over the layer of networks is shown in Figure 32 and Figure 33 We find that for original training subspaces, although the dimensionality has increased with corruption degree, the variance has remained the approximately similar.
Number of class-wise PCA components of original training subspaces over the layer of networks of the generalized model is shown in Figure 34 and Figure 35
SGD vs Adam K
We have also trained MLP networks with Adam optimizer on MNIST and CIFAR-10 with various degrees of corruption. The results for MASC accuracy using corrupted subspaces is shown in Figure 36 and its respective average number of PCA components is shown in Figure 37 With both optimizer choices, even with high corruption degrees, we find that the MASC have better accuracy than the model on the test data. MASC on corrupted training accuracy in most cases reaches the models training accuracy at the latter layers of the network with an exception of MLP trained on MNIST. In most cases, at the initial FC (128) layer of the network, there is a drop in accuracy observed in comparison to the corresponding value for the input and then an increase in latter layers of the network; the MLP trained on CIFAR-10 with SGD is an exception, however.
For 40% corruption degree, although approximately 36% of labels are flipped, with MLP trained on MNIST, model trained on Adam has MASC test accuracy of around 95%, and around 50% for CIFAR-10. This is better than the MASC accuracy at the input layer and models test accuracy. For 60% corruption degree with MLP trained on MNIST, model trained on Adam has MASC test accuracy of around 90% whereas for model trained on SGD is around 85%. Although the networks are trained with 56% of label corruption, yet in FC (512) layer, the MASC training original accuracy is about 70% for model trained on Adam whercas it is 85% for model trained on SGD. MASC on original training data does unfavorably on model trained with SGD rather than with Adam, although further investigation is required.
The results for MASC accuracy using original subspaces are shown in Figure 33 and its respective average number of PCA components are shown in Figure 39 MLP models trained with Adam have qualitatively similar results. The results for MASC accuracy using corrupted subspaces of generalized models are shown in Figure 40 and its respective average nunber of PCA components are shown in Figure 41 MLP models trained with Adam have qualitatively similar results.
L Retraining experiment results on AlexNet models and additional analysis results.
Retraining the model using MASC experiment results on AlexNet models i.e, AlexNet-CIFAR-100 and AlexNet-Tiny ImageNet for different corruption degrees are shown in this seetion. Test accuracies averaged over three runs on the 80% test dataset is plotted for different AlexNet-datasel pairs for various corruption degrees and for various Models/MASCC classifiers is shown in Figure 42
Decrease in fraction of incorrectly labeled data points resulting from relabeling the training set expressed as a fraction of size incorrectly labeled points in the existing corrupted data, using the corresponding best-layer MASC classifier (corrupted subspace and true subspace) is shown in Figure 43 for all models-dataset pairs and corruption degree 20%, 40%, 60%, & 80%. The average values for reference are available in Table D The percentage of incorrect labels that are correctly relabeled using MASC (corrupted subspace and true subspace) for all models-dataset and corruption degrees 20%, 40,, 60%, 80% is shown in Figure 44 The
36

---

Published in Transactions on Machine Learning Research (02/2026)
average values for reference are available in Table 10
The percentage of incorrect labels that correctly relabeled using MASC (corrupted subspace and true subspace) for all models-dataset and corruption degrees 20%, 40%, 60%, 80% is shown in Figure 45 The average values for reference are available in Table
The average decrease in fraction of incorrectly labeled data points resulting from relabeling the training set expressed fraction of size incorrectly labeled points in the existing corrupted data, using the corresponding best-layer MASC classifier. The values are rounded to the second decimal place.
Table 9:
Subspace
Corrupted labels
True labels
Corruption degree
20%
40%
60%
80%
20%
40%
60%
80%
MLP-MNIST
76.93%
38.63%
19.90%
7.75%
97.83%
99.66%
99.93%
99.95%
M.P-CIFAR-110
3.45%
2.38%
1.10%
0.22%
95.49%
97.91%
98.75%
99.14%
CNN-MNIST
88.15%
89.19%
86.44%
71.12%
92.42%
96.46%
96.93%
97.12%
CNN-Pashion-MNIIST
40.69%
64.96%
69.38%
62.61%
35.96%
66.08%
79.26%
80.59%
CNN-CIFAR-10
6.25%
3.30%
9.18%
3.79%
16.44%
40.22%
81.75%
80.09%
-0.09%
0.26%
-0.20%
-0.32%
99.61%
98.96%
90.20%
95.13%
AlexNet-Tiny ImageNet.
-92.46%
-26.50%
-3.95%
-11.55%
-91.32%
-20.24%
2.89%
-12.38%
The average percentage of incorrect labels that correctly relabeled using MASC (corrupted subspace and true subspace) for all models and for corruption degrees 20%, 40%, 60%, 80%. The values rounded to the second decimal place.
Table 10:
Subspace
Corrupted labels
True labels 40%
Corruption degree
20%
40%
60%
80%
20%
60%
80%
MLP-MNIST
77.62%
38.65%
19.90%
7.75%
99.23%
99.80%
99.95%
99.96%
MLP-CIFAR-10
5.59%
2.82%
1.20%
0.24%
99.08%
99.12%
99.23%
99.37%
CNN-MNIST
93.71%
91.34%
87.64%
72.08%
98.27%
98.54%
98.26%
97.88%
CNN-Fashion--MNITT
81.42%
79.18%
76.16%
66.23%
83.82%
86.80%
88.76%
86.08%
CNN-CIFAR-10
17.70%
11.54%
12.30%
5.51%
41.98%
55. 55.63%
89.85%
85.41%
AlexNet-CIIFRR-II
0.15%
0.43%
0.19%
0.06%
99.84%
99.32%
91.06%
95.34%
AlexNet-Tiny ImageNet
5.94%
3.96%
0.51%
0.41%
25.01%
24.43%
18.77%
3.26%
The average percentage of correct labels that incorrectly relabeled using MASC (corrupted subspace and true subspace) for all models and for corruption degrees 20%, 40%, 60%, 80%. The values are rounded to the second decimal place.
Table 11:
Subspace
Corrupted labels
Truc labels 40%
Corruption degree
20%
40%
60%
80%
20%
60%
80%
MLP-MNIST
0.15%
0.01%
0%
0%
0.31%
0.08%
0.03%
0.03%
MLP-CIFAR-10
0.47%
0.25%
0.13%
0.06%
0.79%
0.69%
0.58%
0.59%
CNN-MNIST
1.22%
1.20%
1.40%
2.49%
1.28%
1.17%
1.55%
1.97%
CNN-Fashion-MMIIST
8.86%
8.06%
8.04%
9.30%
10.42%
11.76%
11.26%
14.12%
CNN-CIFAR-10
2.52%
4.67%
3.69%
4.48%
5.62%
8.73%
9.60%
13.82%
0.06%
0.11%
0.57%
1.44%
0.06%
0.23%
1.26%
0.81%
AlexNet-Tiny ImageNet
24.46%
20.10%
6.60%
46.97%
28.92%
29.48%
23.51%
61.39%
37

---

Published in Transactions on Machine Learning Research (02/2026)
M Retraining experiment: epoch-wise results
To study the dynamics of accuracy during retraining, uncncumbered by the carly stopping criterion, we also performed & similar experiment without using early stopping, for 10 epochs. The results for model before training, model after retraining on MASC corrupted subspace, and model after retraining on MASC original subspace for all model-dataset pair and for various corruption degrees are shown in 46 The following steps are performed to retrain the model for corrupted and original subspaces independently.
1. Test dataset is split into 80%-20%
The model's best layer identified using MASC accuracy on the validation set (carved out from 20% of the test dataset).
3. The corrupted training dataset was relabeled using the best layer MASC predictions.
4. The existing corrupted model loaded for retraining purpose. Using the relabeled corrupted training dataset, the model was further trained for 10 epochs.
The remaining 80% test dataset was used to calculate generalization to true labels.
38

---

Published in Transactions on Machine Learning Research (02/2026)
Madel
MASC (75%)
MASC (25%)
MASC (99%)
MASC (50%)
MASC (1 PC)
MLP-MNIST
CNN-F Soll-Fashion-Mae -MNIST
AlexNet-CIFAR-100
1.0
0.44
0.42
0% corruption dogrer
0.8
0.8
0.40
0.6
0.6
0.38
0.36
0.4
0.4
0.34
0.2
0.2
0.32
0.30
0.0
0.0
1.0
0.38
20% corrupten degree
0.8
0.36
0.6
0.4
0.28
0.2
0.26
0.0
0.0
0.24
0.32
40% corruption degree
0.8
Test Accuracy
0.6
0.28
0.26
0.4
0.24
0.2
0.22
0.0
0.0
0.16
60% curruption degree
0.15
0.14
0.6
0.13
0.4
0.11
0.2
0.10
0.0
0.0
0.045
80% corruption degret
0.8
0.040
0.035
0.4
0.2
0.025
FC (128)
FC (512)
FC (2048)
(2048)
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
(4096)
Network layers
Figure 15: Minimum Angle Subspace Classifier (MASC) test accuracy over the layers the network when the data is projected onto corrupted training subspaces with the indicated corruption degree, for multiple models/datased Rows corresponds to plots with the same corruption degree & the columns correspond the models, noted. Test accuracy (dotted line) of the model is shown. FC corresponds to fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation.
39

---

Published in Transactions on Machine Learning Research (02/2026)
Model
MASC (75%)
MASC (25%)
MASC (99%)
MASC (50%)
MASC (1 PC)
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
1.0
0.19
0.18
0% corruption dogres
0.8
0.8
0.17
0.6
0.6
0.6
0.16
0.15
0.4
0.4
0.4
0.14
0.2
0.2
0.2
0.13
0.12
0.0
0.0
0.0
1.0
1.0
1.0
0.12
20% corruption degrec
0.8
0.8
0.8
0.11
0.6
0.6
0.6
0.10
0.4
0.4
0.4
0.09
0.2
0.2
0.2
0.08
0.07
0.0
0.0
0.0
1.0
1.0
1.0
40% corruption degrec
0.8
0.8
0.8
0.09
Test Accuracy
0.6
0.6
0.6
0.08
0.4
0.4
0.4
0.2
0.2
0.2
0.06
0.0
0.0
0.0
1.0
1.0
1.0
0.045
60% co corruption degree
0.8
0.8
0.8
0.040
0.6
0.6
0.6
0.035
0.4
0.4
0.4
0.030
0.2
0.2
0.2
0.025
0.0
0.0
0.020
1.0
1.0
1.0
0.013
80% corruptinn dogree
0.8
0.8
0.8
0.012
0.011
0.6
0.6
0.6
0.010
0.4
0.4
0.4
0.009
0.2
0.2
0.2
0.008
0.007
0.0
0.0
FC (128)
FC (512)
PC (2048)
FC (2048)
Flat (576)
FC (250)
FC (250)
FC (250)
Flat (1024)
FC (250)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 16: Minimum Angle Subspace Classifier (MASC) test accuracy over the layers of the network when the data is projected onto corrupted training subspaces with the indicated corruption degree, for multiple datasets. Rows corresponds to plots with the same corruption degree & the columns correspond to the models, noted. Test accuracy (dotted line) of the model is shown. FC corresponds to fully connected layer with ReLU activation whercas Flat corresponds to flatten layer without ReLU activation.
40

---

Published in Transactions on Machine Learning Research (02/2026)
Build subspaces
Inference
MLP-MNIST
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR- 10
140
120-
60
70
120
100-
50
60
100
80
80
40
40
50
Time taken (in seconds)
60
30
40
40
20
20
10
10
0
PC
(51))2)
FC (2048)
FC (2048)
(128)
(512)
FC (2048)
FC (2048)
Flat (576)
PC (250)
(250)
FC (250)
Rat. (1024)
FC (250)
(250)
FC (250)
CNN-Fashion-MNIST
AlexNet-CIFF --FN-e--IFAR-0000
MexNet-Tiny ImageNet
3000
6000
50
2500
5000
40
2000
4000
1500
3000
20
1000
2000
10
500
1000
0
0.
Rat (576)
(250)
FC (250
(250)
Rat (256)
FC (4096)
FC (4096)
Flat (256)
FC (A096)
FC (4096)
Network layers
Figure 17: Total time taken in seconds by MASC for building the subspaces and inference over the layers of the network, for multiple datasesss FC corresponds to fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation.
Build subspaces
Inference
MLP-MNIST
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
175
500
150
12.5
125
Computational cost (GFLOPs)
10.0
300
100
10
7.5
200
5.0
5
2.5
100
25
0.0
0
(C2))
(512)
PC (2048)
FC (2048)
FC(128)
FC(5)2)
FC (2048)
FC (2048)
Pat (576)
(23)00)
F21250)
(250)
Flat. (1024)
(250)
FC
(250))
CNN-Fashion-MNISS
MlexNet-CIFAR-10
MexNet-Tiny ImageNet
35
3000
3500
30
2500
3000
25
2000
2500
2000
1500
1000
1000
500
500
Flat (576)
(250)
PC (250)
(250)
Flat (256)
(A096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 18: Total computational cost (GFLOPS) by MASC for building the subspaces and inference over the layers the network, for multiple models datasets. FC corresponds to fully connected layer with ReLU activation whercas Flat corresponds to flatten layer without ReLU activation.
41

---

Published in Transactions on Machine Learning Research (02/2026)
Computational cost (GFLOPs) building classwise subspaces
MLP-MNIST
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
0.05
0,040
0.020
0.035
0.018-
0.04
0.012
0.03
0,025
0.014
0.020
0.010
0.012
0.02
0.015
0.008
0.010
0,010
0.008
0.01
0.005
0.006
0.006
(128)
(512)
(2048)
(2048)
FC2
FC (312)
FC (2048)
FC (2048
Flat((576)
FC (250)
FC (250)
FC (250)
Flat (1024)
FC (250)
FC (250)
FC (250)
CNN-Fashion-MNIST
MlexNet-CIFAR-LL
AlexNet-Tiny ImageNet
0.014
0.013
0.008
0.008
0.012
0,007
0.007
0.011
0,006
0.006
0,005
0.005
0.009
0,004
0.004
0.008
0.003
0.003
0,002
0.002
0.007
0,001
0.001
0.006
Flat (576)
(25)
Fc (250)
FC (250)
Plat (256)
(4096)
(4096)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 19: Computational cost (GFLOPS) by MASC for building per-class subspace over the layers of the network, for multiple models/datasets. FC corresponds to fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation.
42

---

Published in Transactions on Machine Learning Research (02/2026)
Model Testing Accuracy
Model Testing Accuracy
Accuracy on Tlest
0% corruption degree
20% corruption degree
40% corruption degree
0.20
0.16
0.13
0.18
0.14
0.12
0.11
0.16
0.12
0.10
0.09
0.14
0.10
0.08
0.07
0.12
0.06
Accuracy
0.06
0.05
60% corruption degree
80% corruption degree
100% corruption degree
0.08
0.022
0.020
0.0070
0.07
0.018
0.0065
0.06
0.016
0.0060
0.05
0.014
0.0055
0.04
0.012
0.0050
0.010
0.03
0.0045
0.008
0.02
0.0040
0.006
Flat (256)
FC (4096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 20: MASC Lest accuracy the layers of AlexNet trained Tiny ImageNel when the data is projected onto corrupted training subspaces with the indicated corruption degree. Test accuracy of the model and best model test accuracy is shown for comparison. Best model test accuracy corresponds accuracy of the test data of the model if early stopping was used.
43

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
MASC Accuracy on (Corrupted Label Subspace) Accuracy on (True Subspace)
Accuracy
0% corruption degree
20% corruption degree
40% corruption degree
0.20
0.16
0.13
0.18
0.12
0.11
0.16
0.12
0.10
0.09
0.14
0.10
0.08
0.07
0.12
0.06
Accuracy
0.06
0.05
60% corruption degree
80% corruption degree
100% corruption degree
0.08
0.0250
0.0200
0.07
0.0225
0.0175
0.0200
0.06
0.0150
0.0175
0.05
0.0150
0.0125
0.04
0.0125
0,0100
0.03
0.0100
0.0075
0.0075
0.0050
0.02
Flat (256)
FC (4096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 21: MASC test accuracy over the layers of AlexNet trained on Tiny ImageNet when the data set is projected onto corrupted training and true training subspace. Test accuracy of the model and best model test accuracy is shown for comparison. Best model test accuracy corresponds accuracy of the test data of the model if early slopping was used.
44

---

Published in Transactions on Machine Learning Research (02/2026)
Model Training Accuracy inai Model)
MASC Accuracy Test (Trained Model)
Model festraccuacy (Trined Model)
MASC Accuracyy on Test (Random Initialized Model)
MLP-MNIST
CNN-Fashion-MNIST
1.0
0.8
0% corruption
0.6
0.4
0.2
0.0
1.0
0.8
20% corruption
0.4
0.2
0.0
1.0
40% comuptoon degree
0.8
0.6
0.4
0.2
0.0
Accuracy
1.0
60% comuption degree
0.8
0.6
0.4
0.2
0.0
1.0
0.8
degres 80% corruction
1.0
0.8
0.6
0.4
100%
0.2
0.0
Input
FC (128)
FC (512)
FC (2048)
PC (2048)
Input
Flat (576)
FC (250)
FC (250)
FC (250)
Network layers
Figure 22: MASC accuracy over the layers trained and random initialized network when the data is projected onto corrupted training subspaces with the indicated corruption degree.
45

---

Published in Transactions on Machine Learning Research (02/2026)
Model Training Accuracy (Trained Modell
MASC Accuracy on Test (Trained Madel)
Accuracy (Trained Modell
MASC Accuracy on Test (Random Initialized Model)
MLP-MNIST
CNN-Fashion-MNIST
1.0
0.8
0% corrupiion
0.6
0.4
0.2
0.0
1.0
0.8
20% corruption
0.6
0.4
0.2
0.0
1.0
40% comupption degree
0.8
0.6
0.4
0.2
Accuracy
0.0
1.0
corruption degree
0.8
0.6
0.4
0.2
0.0
0.6
0.4
80%
0.2
0.0
1.0
100% on degres
0.8
0.6
0.4
0.2
0.0
Input
FC (128)
FC (512)
FC (2048)
FC (2048)
Input
Flat (576)
FC (250)
FC (250)
FC (250)
Network layers
Figure 23: MASC accuracy over the layers of trained and random initialized network when the data set is projected onto subspace corresponding to true training labels. Test accuracy the trained model is shown for comparison.
46

---

Published in Transactions on Machine Learning Research (02/2026)
Model Tast Accuracy (Trained Model)
MASC Accuracy on Rest (Trained Model)
MASC Accuracy Text (Random Initialized Model)
0% corruption degree
20% corruption degree
40% corruption degree
0.18
0.12
0.08
0.16
0.10
0.07
0.06
0.12
0.08
0.10
0.05
0.08
0.06
0.04
0.06
0.03
0.04
Accuracy
60% corruption degree
corruption degree
coruuttion degree
0.0250
0.034
0.00575
0.0225
0.032
0.00550
0,0200
0.030
0.00525
0.028
0.0175
0.00500
0.026
0.0150
0.00475
0.024
0.0125
0.00450
0.022
0.0100
0.00425
0.020
0.0075
0.00400
Flat (256)
FC (4096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Flat (2.56)
FC (4096)
FC (4096)
Network layers
Figure 24: MASC accuracy over the layers of trained and random initialized AlexNet-Ting ImageNet. when the data is projected onto corrupted training subspaces with the indicated corruption degree.
..... Model Test. Accuracy (Trained Model)
MASC Accuracy on Test (Trained Modell
MASC Accuracy Test (Randor Initialized Model)
0% co corruption degree
20% corruption degree
40% corrpption degree
0.10
0.18
0.16
0.12
0.09
0.10
0.12
0.07
0.10
0.08
0.06
0.06
0.06
0.04
0.04
Accuracy
60% corruption degree
corruption degree
corruption degree
0.040
0,040
0.050
0.035
0.035
0.045
0.030
0.030
0.040
0.025
0.025
0.035
0.020
0.020
0.030
0.015
0.015
0.025
0.010
0.010
0.020
0.005
0.005
Flat (256)
FC (4096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 25: MASC accuracy over the layers of trained and random initialized AlexNet-Ting ImageNet when the data set is projected onto subspace corresponding to true training labels.
47

---

Published in Transactions on Machine Learning Research (02/2026)
Model Training Accuracy
MASC Accuracy on Comupted Trining
MASC Accuracy on Test
Madel Testing Accuracy
MASC: Accuracy on Original Training
MLP-MNIST
CNN-Fashion-MNIST
AlexNet-CIFAR-100
1.0
0% conuptiin deg
0.8
0.6
0.2
0.0
1.0
20% compution degiee
0.8
0.6
0.2
0.0
1.0
40% corruption degree
0.8
0.6
0.4
0.2
Accuracy
0.0
1.0
60% comuptiin degige
0.8
0.6
0.2
0.0
1.0
80% comrptiin degree
0.8
0.6
0.4
0.2
0.0
1.0
100% corrutiion degree
0.8
0.4
0.2
0.0
input
FC (128)
FC (512)
FC (2048)
FC (2048)
Input
FC (250) (576)
FC (250)
FC (250)
Flat (256)
FC (4096)
FC (4096)
Network layers
Figure 26: Minimum Angle Subspace Classifier (MASC) accuracy over the layers of the generalized network when the data set projected onto corrupted training subspaces with the indicated corruption degree. Rows corresponds to plots which have the same corruption degree & the columns correspond to the generalized models noted. Training & test accuracy of the generalized model is shown. FC corresponds to fully connected layer with ReLU activation whereas Flat corresponds to flatten layer without ReLU activation. The respective number of class-wise PCA components of the models is shown in Figure 34 SGD optimizer was used for training MLP models, whereas Adam optimizer was used for other models.
48

---

Published in Transactions on Machine Learning Research (02/2026)
Model Training Accuracy Testing Accuracy
MASC Accuracy en Corrupted Training Accuracy en Original Training
MASC Accuracy on Test
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
0% corruption degree
1.0
20% comuption degree
0.8
0.6
0.4
0.2
1.0
40% comuption degree
0.6
0.4
0.2
60% comppsion Accuracy
0.0
degree
1.0
80% corupttio degree
1.0
100% corruption degree
0.8
0.6
0.4
0.2
Inpil
FC
(264))
(240)
(ar(570)
PC (250)
PC (250)
PC (250)
Inser
ParitoZa
PC (Sa)
PC (so))
Fiat (250)
(409))
(40000
Network layers
Figure 27: MASC accuracy over the layers of the network when the data projected onto corrupted training subspaces with the indicated corruption degree. The number of class-wise PCA components of these models are shown in Figure 31
49

---

Published in Transactions on Machine Learning Research (02/2026)
Latel
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
1.0
0% comuution degree
0.8
0.6
0.4
20% computonn degree
0.6
0.4
0.2
0.0
40% comuption degree
0.2
Accuracy
0.0
1.0
60% corruption degree
0.8
0.6
80% corruption degree
0.8
0.6
0.4
0.2
0.0
1.0
100% corruption degree
0.8
0.2
0.0
lpid
PC (.26)
RC (C(S)))
PC (286))
Gooal
Apput
PC (25)
FC (c.2
PC (350
loput
(1024)
Pc (300
cse)
PC caaa
Fist (56)
PC (4888)
PC
Network layers
Figure 28: MASC accuracy the layers of the network when the data set is projected onto corrupted subspace and subspace corresponding to true training labels. The respective number of class-wise PCA components for true training label subspaces of the models is shown in Figure 33
50

---

Published in Transactions on Machine Learning Research (02/2026)
Model Training Accuracy
Accuracy on Comuupted Training Accuracy on Original Training
MASC Accuracy Test
Testing Accuracy
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
1.0
0.8
0% comugtion
0.6
0.4
0.2
0.0
1.0
0.8
20% corrpttion
0.2
1.0
0.8
0.6
0.4
40%
0.2
0.0
Accuracy
60% corruction
1.0
0.8
80% comuption
0.6
0.4
0.2
0.0
1.0
100% comuption degree
0.6
0.4
Input
(128)
FC (512)
FC (2048)
FC (2048)
Input
Rlat (576)
FC (250)
FC (250)
FC (250)
Input
Flat (1024)
FC (250)
FC (250)
FC (250)
Flt (256)
FC (4096)
FC (4096)
Network layers
Figure 29: MASC accuracy over the layers of the generalized network when the data set projected onlo corrupted training subspaces with the indicated corruption degree. The respective number of class-wise PCA components of the models is shown in Figure 35
51

---

Published in Transactions on Machine Learning Research (02/2026)
Class Class
Class 4
MLP-MNIST
CNN-Fashion-MNIST
AlexNet-CIFAR-100
0% corruption degree
350
400
200
300
350
250
250
100
150
200
50
100
150
100
Input
FC1512) FC
(2
FC (2048)
Inport
Flar (576)
FC (250)
FC (200)
FC (250)
input
Fist (250)
FC (4096)
FC(4090)
20% corruption degree
1000
400-
350
600
300
300
400
250
200
200
150
200
100
150
50
Input
FC1)522) FC(12))
FC (2048)
Fc (2048)
Inpad
Flxt (578)
FC (25)
(200)
FC (250)
Input
Plat (258)
FC (4096)
FC (4096)
Number of PCA components degres
1200
400
450
1000
350
350
40% corrugtion
800
600
300
250
250
400
150
200
50
Input
(512))) FC2))))
FC (2048)
FC (2049)
Inpal
Flat (578)
G50)
(250)
(250)
Input
(256)
FC (409g)
FC (4096)
60% corruption degree
12000
400
300
1000
350
250
600-
200
400
250
175
150
200
200
Input
FC()28)
(C FC1552)
PC (2048)
Input
Flat (576)
FC (250)
FC (50)
FC (250)
Input
Flat (258)
(4096)
FC
80% corruction degree
375
300
350
325
250
300
200
275
150
250
100
200
175
Input
(2048) (S12) (C2228)
FC (2048)
Input
Fixt (578)
FC (250)
(250)
FC (250)
Input
Flat (256)
FC (4096)
FC (4096)
100% corruction degree
375
300
1400
350
1200
325
1000
300
200
275
150
250
600
100
400
200
200
175
InpiR
FC (128)
(C1512)
PC (2048)
PC (2048)
Inpid
Flst (575)
FC (50)
PC ase)
(250)
Inpit
(256)
PC (4096)
FC (4096)
Network layers
Figure 30: Class-wise number of PCA components of the corrupted training subspace over the layers of multiple networks with various corruptions degrees. Although it not mentioned in the legend, all the 100 classes of CIFAR-100 are plotted.
52

---

Published in Transactions on Machine Learning Research (02/2026)
Class
Class
Class
Class
Class
Class
Class
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
0% corruption degree
250
300
1000
200
150
600
200
600
150
100
400
50
200
50
0
Impis
FC
(512)
FC (2048)
FC (2048)
Inpit
Flat (578)
FC (250)
Fc (250)
FC (250)
Inpli
(102))
(250)
(252
(250))
Flat (256)
FC (4096)))
FC (4096)
20% comuption degree
1200
350
250
1000
300
600
200
800
150
600
200
100
150
200
200
100
100
input
F2.0220)
Fc (912)
(2048)
(2048)
inpit
(S77))
FC FC
Fc (250)
FC (50)
Input
Flat (1024)
FC (250)
FC (25))0)
FC F256)
Flat C36)
(4096)
FC (4096)
Number of PCA components degree
1400
250
2200
400
800
1000
350
200
40% corruption
800
300
150
500
600
250
100
400
300
50
200
150
200
Inpit
PC2))
PC (512)
FC, (2046)
FC (2048)
Inpit
(576)
FC (56)
rc (25))
FC (25)
Input
Flat (laz()
PC (200)
FC (2(0)))
(256)
FC (4096)
FC (4096)
PC
60% comupptin degree
14000
120
1200
350
100
800
300
250
200
Input
P2
(512)
FC (2048)
PC (2048)
Impal.
(576))
(250)
FC (250)
G250)
Input
Flat (1024)
(250)
FC (250)
FC
Flat (256)
FC (4096)
FC (4096)
80% corruction degree
1400
400
900
20.0
1200
350
1000
800
300
600
125
15.0
600
250
500
10.0
400
300
75
200
2.5
Input
(128)
(51))2)
FC (2048)
FC (2048)
Inpat
Plat (876)
FC (250)
FC (250)
FC (250)
Input
Part (1024)
PC (250)
FC (250)
(250)
Plat (256)
FC (4098)
FC (4096)))
100% corruction degree
3400
400
700
1200
600
1000
500
8
300
6
400
100
200
200
Input
(128)
FC (512)
2048)
(2048)
Impat
(576)
(250)
(20))
(250)
Inpui
Flat Flut(1024) (1024)
FC (250)
FC (250)
PC (250)
Plat (256)
(4096)
FC. (4096)
FC
Network layers
Figure 31: Class-wise number of PCA components of the corrupted training subspace over the layers of multiple networks with various corruptions dogrecs. Although it not mentioned in the legend, all the 200 classes of Tiny ImageNet plotted.
53

---

Published in Transactions on Machine Learning Research (02/2026)
Class Class
Class 4
MLP-MNIST
CNN-Fashion-MNIST
AlexNet-CIFAR-100
0% corruption degree
350
400-
200
300
350
250
250
100
150
200
50-
100
150
100
Input
FC
FC1122)
FC (2008)
FC (2048)
Inpst
Flst (576)
FC cso)
FC (280)
FC (250)
Inpit
Flet (256)
FC (4096)
FC
20% corruption degree
450
8000
350
400-
350
250
400
150
200-
100
Input
FC1)12 FC
(2648)
FC (2048)
Inpot
(578)
PC (50)
FC (250)
FC (250)
Input
Flat. (258)
FC (4096)
FC (4098)
Number of PCA components degree
400
400-
1000
350
300
40% comuuttin
250
250
300
200
400
150
150
200
100
100
50
50
Input
FC1(52) FC
FC (2046)
Fc (2049)
Inpul
Flat (575)
(50)
(250)
(250)
Input
(256)
FC (4090)
FC (4096)
60% corruption degree
1200
350
300
800
250
250
200
200
400
150
Inpit
FC(512) FC(222))
(2
FC (2048)
Inpst
Flat (976)
FC (50)
FC (250)
FC (250)
Input
Flat (256)
(4036)
FC
80% corruption degree
1200
350
1000
300
250
250
200
200
400
150
200
100
100
50
Input
(22))
FC (512
FC (2048)
FC (2048)
Inpat
Flat (578)
FC (20)
(200)
FC (250)
Input
Flat
FC (4096)
FC (4096)
100% corrption degree
350
1000
200
400
150
100
100
50
50
0
Inpit
FC (28)
(C1512)
PC (2048)
PC (2048)
Inpid
Flat (575)
FC (5o)
ase)
(250)
Input
(256)
PC (4096)
PC (4096)
Network layers
Figure 32: Class-wise number PCA components the subspace corresponding to true training labels over the layers of multiple networks with various corruptions. Although is not mentioned the legend, all the 100 classes of CIFAR- 100 are plotted.
54

---

Published in Transactions on Machine Learning Research (02/2026)
Class Class
Class
Class
Class
Class
Class
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNet-Tiny ImageNet
0% corruption degree
250
800
300
1000
200
150-
600
200
600
150
100
200
0
Inpit
FC20))) (512)
FC (2048)
FC (2048)
Inpat
(578
(250)
(250)
(250)
Inputt
Flat (2024)
(250)
(25200)
(250)
Flat (256)
FC (4096)
FC (4096)
20% coruption degree
350
250
1000
300
700
200
800
250
600
150
600
200
150-
400
100
300
100
100
Input
FC (120)
FC(512))
FC (2048)
FC (2048)
Inpul
Flat (S7a)
FC (250
FC (250)
FC (250)
Input
Flat (1024)
PC (250)
FC (250)
FC (25n)
Flat (256)
FC (4096)
FC (409))))
Number of PCA components degree
400
900
1200-
350-
800
250
1000
300
200
800
600
600
200
40%
400
200
150-
100
300
400
100
50
Input
FC (128)
FC (912)
FC (2048)
(2048)
Input
(S7
FC (230
Fc (250)
PC (250)
Input
Flat (1024)
FC (250)
FC (250)
FC (250)
Plat (256)
FC (4036)
FC (4096)
60% coruuption degree
1400
200
12000
350-
175
1000
300
150
800
200
250
500
100
125
600
150
400
100
300
50
200
5o.
200
0
Input
PC
(51512)
FC (2048)
FC (2648)
Input
Flar (576)
(230
FC1250)
eC (250)
Inpur
Flxt (1024)
PC (20)
FC1250
(250)
Flat (256)
FC (4096)
FC (409))
80% Corruption degree
1440
350
2200
1000
300
250
600
700
50-
800
200
600
400
150
100
300
400
20
200
50
Imput
(12))
(512)
FC (2048)
FC (2048)
Inpar
Plat (576)
FC c250)
PC (250)
(250)
Input
Fixt (1024)
FC (250)
(250)
FC (250)
Plat (256)
FC (4096)
(4096)))
100% coruuption degree
1400
1200
1000
300
500
250
400
200
150
200
200
100
100
10
Input
(128)
FC (512)
(2048)
(2048)
Impat
(576)
FC (250)
FC (20))
FC (250)
Inpuit
Flat (1024)
FC (250)
FC (250)
PC (25g)
Plat (256)
(4096)
PC (4096)
Network layers
Figure 33: Class-wise number PCA components the subspace corresponding to true training labels over the layers of multiple networks with various corruptions. Although is not mentioned the legend, all the 200 classes of Tiny ImageNet are plotted.
55

---

Published in Transactions on Machine Learning Research (02/2026)
Class
Class
MLP-MNIST
CNN-Fashion-MNISS
AeexNet-IIFA--10
0% cerruption degree
400
200
250
350
200
250
100
150
100-
50
50
100
Input
FC
(512)
FC (208))
(C
Flat Fl71()77))
FC (230)
FC (C
(250)
Flat (256)
FC (4036)
FC
20% corruption degree
350
450
250
300
200
350
150-
300
100-
200
100
50-
50-
100
Input
(22)
FC (512)
FC (2048)) (204))
(S7
FC (250)
FC (250)
FC (250)
(256)
FC (4036)
FC (4098))
Number of PCA components degres
250
300
400
200
250
40% comuptiinn
150
300
100
150
200
100.
100
Imput
(128)
(512)))
FC (20s)
FC (2048)
Plat (576)
(230)
(250)
(250)
Flat (256)
FC. (4096)
FC (4096)
60% corruption degree
350
250-
450
200
250
150
100-
Input
(22(22)
FC (512)
(C
(C
Plat (978
FC (250)
(C(220)
(250)
Plat (256)
(4036)
(4099))
80% conuption degree
350.
450
300-
250
350
150
250
100
50
150
Input
(128)
FC (512)
FC (2045)
FC (2048))
(1766)
FC (250)
FC (250)
FC (250)
(256)
FC (4006)
FC (4099))
100% coruption degree
350
250
400
200
300
200
200
150
100
Input
FC (128)
FC (512)
FC (208))
PC (2048)
Pat (576)
(25))
(250)
(25))
Flal (256)
FC (4036)
FC (4096)
Network layers
Figure 34: Class-wise wise number of PCA components of the corrupted training subspace over the layers of multiple generalized networks with various corruption degrees. Although it not mentioned in the legend, all the 100 classes of CIFAR-100 100 are plotted.
56

---

Published in Transactions on Machine Learning Research (02/2026)
Class
Class
Class
Class
Class
MLP-CIFAR-10
CNN-MNIST
CNN-CIFAR-10
AlexNel-Tiny ImageNet
0% corruption degres
250
800
300
1000
200
800
400
150
400
100
200
50
50-
0
Inpus
(128)
(512)
FC (D48)
FC (2048)
Flat (578)
(25)
(250)
F259)
Flat (1024)
(250)
PC (250)
(256)
Flatten (256)
RelU (4096)
Rew (4096)
20% corruption degree
1200
300
700
200
600
600
150
400
300
200
50
200
100
Inpit
FC2822
(912)
FC GD48)
FC (2040)
Fat (578)
(c.)20)
FC (250)
(250
FC (250)
Flat (1024)
FC (50)
FC,(20)) (250)
Flatten (256)
RedUU RelUU (4096)
ReUU (4096)
40% corruption degree
1200
300
1000
250
800
700
300
250
Number of PCA components
800
150
600
500
400
200
150
300
100
200
200
50
100
Input
FC (22)
FC (512)
FC Qo44))
PC (2045)
Far (578)
FC (250)
(200)
FC (250)
Flat (1024)
FC (50)
FC (250)
Flatten (256)
ReLU (4096)
ReLL (4096)
FC
60% corruption degree
1200
350
1000
250
800
700
300
200
600
250
200
600
400
400
100
300
200
50
200
100
Input
(128)
(512)
FC (248)
FC (2048)
Flat (576)
PC (250)
(250)
FC (250)
Flat (1024)
(50)
FC (250)
(250)
Pletten (256)
RaLU (4096)
ReLu (4096)
80% corruption degres
1200
300
350
800
300
700
800
600
600
150
500
200
150
300
100
200
100
Inpit
(220)
FC (912)
G04m
FC (24m)
Far (578)
(2.
FC (50)
FC (250)))
Flat (1024)
PC (05)
FC.(250) (2502
Platten (256)
BelU (4096)
RelU (4096)
100% corrption degree
300
800
1000
700
300
800
200
600
250
200
150
400
300
100
200
50
200
50
100
Input
PC (128)
FC (512)
FC Q048)
FC (2048)
Flat (576)
(252
FC (25)
FC (250)
Rat (024))
FC (250)
PC (2(250)
FC (250)
Flatten (256)
RelU (4096)
Bell (4096)
Network layers
Figure 35: Class- wise number of PCA components of the corrupted training subspace over the layers of multiple generalized networks with various corruption degrees. Although it is not mentioned in the legend, all the 200 classes of Tiny TetgeNet plotted.
57

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Accuracy Original Training
MLP-SGD-MNIST
MLP-Adam-Mer -NNIST
MLP-SGD-CIFAR-10
MLF-Adam-CIFAR-10
0% corruption degret
0.6
0.4
0.2
20% comrutiio degree
0.6
0.4
0.2
40% corrutoin degree
0.6
0.4
0.2
Accuracy
60% corrutton degree
0.6
0.4
0.2-
80% corruption degree
0.6-
0.4
0.2
0.0
100% corruption degres
0.4
0.2
Inpur
FC(612) (128)
(208)
FC (048)
Input
(1282
(.))12)
FC (242))
FC (2ms)
Imput
FC (128)
PC (3)))))
FC (2085)
FC (248))
Inpur
FC (128)
PC (CB))))
FC (2948)
FC (2048)
Network layers
Figure MASC accuracy over the layers of the MLP network when the data projected onto cor- rupted training subspaces with the indicated corruption degree, for MLP models with MNIST and CIFAR-10 datasets. Rows corresponds to plots which have the same corruption degree and the columns correspond to the models with SGD and Adam optimizer noted. Training and test accuracy of the model is shown.
58

---

Published in Transactions on Machine Learning Research (02/2026)
MLP-SGD-MMN MNGDMMIIST
MLP Adam MNIST
Adam-MNIST
MLD-SGD CIADDCIFFR-LLLI 10
MLP-Adam -Iaamm---FFR1O0
0% corupption degroe.
600
1500
200
1000
150
400
1000
250
0-
Input
PC.0
FCB12)
PC DD48)
FC (2024
Inpor
FC
(113))
FC (248)
FC 2048)
Ingut
PC (220
PC PC112
FC PC:204e
FC (o02
Inpos
FC
RC (112)
PC (048)
FC (2048)
20% corruption degree.
1250
1000
1000
600
750
400
500
500
200
250
Input
FC (C2022))
FC/012)
Fc (2048)
PC (C12966)
Inpur
FC2(128
FC (512)
PC (2048)
FC (248))
Ingut
FC (128)
FC
FC p244s
(298)
Imput
FC2(228)
FC (333)
PC (DMB))
PC (248)
40% corruption degree
1500
1250
1500
1000
750
1000
1000
Number of PCA companents
500
500
500
250
250
0
Inpot
FC1512) FC2228)
FC (048)
FC12046)
Inputt
FC2222)
Fc (512)
FC (2048)
(248)
Insut
Fc (128)
FC (112)
FC (245))
FC Co48)
Impos
FC
PC (512)
FC Cosa)
FC
60% corruption degree
1250
1500
1250
1500
1000
750
750
500
500
250
Inpat
(515522) (128)
FC (248))
FC (204))
Inpet
(C.))8))
(522)
FC coea)
FC (248))
Ingut
FC (120)
FCB1
Fc (2048)
FC (200
Inse
FC(22))
(513)
FC Conm
PC (204))
80% corruption degree
1500
1250
1000
750
1000
750
500
500
500
0
Inpot
FC (226)
FC11112
(288)
(264))
Input
FC2(22))
FC (512)
(204))))
(2648)
Ingut
(128)
FC FC12
(2845)
FC (C
Imput
FC(22))
FC (513)
(208)
100% corruption degree
1500
600
1500
1250
3000
1000
400
1000
750
500
500
200
250
Inpat:
FC (120)
PC (240)) (512)
PC (204))
Inpat
120)
Ft (512)
PC (240)
FC (240)
Ingat
Fc. (120)
FC (312)
Fc (20A8)
Pc (200))
Inpat
(22)
R. (512)
PC (200))
PC (2040)
Network layers
Figure 37: Class-wise number of PCA components of the corrupted training subspace over the layers of MLP networks trained with MNIST and CIFAR-10 datasets with various corruption degree. Rows corresponds to plots which have the same corruption degree and the columns correspond to the modcls with SGD and Adam optimizer noted.
59

---

Published in Transactions on Machine Learning Research (02/2026)
Takking Accuracy
Accuracy Original
Hodel Accuracy
MASC Accuracy
MLP-SGD-MNIST
MLP-Adam-MMN
MLP-SOD-CIFAR-10
MLF-Adam-CIFAR-10
0% comuption degree
0.8
0.6
0.4
20% corruption degree
0.8
0.6-
0.4
40% corruption degree
0.8
0.6
0.4-
Accuracy
60% comruttion degree
0.8
0.6
0.4-
80% corruption degres
0.8
0.6
0.4-
100% corruption degree
0.6
0.4
Insu
PCB12) (128)
(248)
FC (048)
Input
(C012) (222
Ro4s)
PC CoMM)
Input
Rc (120)
PC (C())
(24e)
FCD244
Inpuit
PC (C1112)
(2048)
FC (200)
Network layers
Figure 38: MASC accuracy over the layers of the MLP network when the data set is projected onto subspace corresponding to true training labels. Rows corresponds to plots which have the same corruption degree and the columns correspond the models with SGD and Adam optimizer as noted. Training and test accuracy of the model is shown. FC corresponds fully connected layer with ReLU activation.
60

---

Published in Transactions on Machine Learning Research (02/2026)
Class
Gass
Class
Class
Cas
MLP-SGD-MNIST
MLP-Adam-MNER
MLP-SGD-CIFAR10
MLP-Adam-CIFARI
0% corruption degres
600
2500
200
150
400
730
300
200
500
250
Inputt
FC(512) FC
FC (2049)
FC (212446)
Inpur
FC
(S1512)
FC (206)
FC (248)
Ingett
FC FC
FC (2048)
Fc (2046)
Input
FC
FC (51))))
FC (206)
PC (248)
20% corruption degros
1250
800
1000
730
1000
400
500
500
500
200
250
250
Input
FC (228)2
FC/1112)
FC (048)
(2(228))
Input
FC.(128)
Fc (512)
PC (2088)
(248))
Ineut
PC (128)
PC FC2)
(2045)
(2988)
Impst
FC2222)
RC (513)
(2088)
K. (208)
40% corruption degree
1000
1500
1250
1500
800
1000
1000
750
1000
Number of PCA companents
400
500
500
500
200
0.
Ingat
FC222))
FC(512)
(248)
FC(2045)
Inpst
FC
(122))
PC (048)
FC (018)
Ingest
FC (128))
FCB12
FC (2(2205)
Inpet
FC(28)
(512)
(208)
FC (048)
60% corruptin degree
1250
1500
1000
1000
1000
500
250
Inputt
FC (12))
FC1512)
Fc (244)
FC (208)
hin
FC FC2012))
FC (12)
PC coen
FC (248)
Ineut
FC FC201
F121
rc (2048)
FC (2044)
last
FC FC2028)
FC (513)
PC Qo4a
PC (208))
80% corruption degree
1250
1250
1000
1000
1000
750
500
500
500
250
250
Inpot
FC (120)2)
(515512
(040)
(242)
Input
FC (C2)
FC (112))
FC (2048)
(248)
Ingut
PC (120)
FC (112)
PC (249)
(204)
Input
FC (128)
FC (512)
PC (200)
(C0224)
100% corruption degree
800
1250
1500
1250
600
1000
750
500
500
200
250
250
Inpet
PC1512) (128)
FC (2048)
FC (2)4))
Inse
FC
(12))
PC (204))
PC (248)
Ingot
(129)
PC (C812
FC (2048)
FC (208))
Inees
(C28))
(512)
FC (208))
FC (2048)
Network layers
Class-wise number of PCA components of the subspace corresponding to true training labels over the layers of MLP networks with various corruption degrees. Rows corresponds to plots which have the same corruption degree and the columns correspond to the models with SGD and Adam optimizer as noted.
Figure 39:
61

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Accuracy Original Training
MLP-SGD-MNIST
MLP-Adam-Men -NNIST
MLP-SGD-CIFAR-10
MLF-Adam-CIFAR-10
0% corruption degret
0.6
0.4
0.2
20% comruttion degree
0.6-
0.4
0.2
40% corrppdin degree
0.4
0.2
Accuracy
60% corruttinn degree
0.6
0.4
0.2
80% corruption degree
0.6-
0.4
0.2
0.0
100% corruption degres
0.4
0.2
Insur
FC/612) (128)
(248)
(048)
Input
(128)
(.)))))
FC (242))
FC (2ms)
Imput
FC (128)
PC (c)))))
FC (2065)
FC (248))
Inpur
FC (128)
PC (1)))
FC (2048)
FC (2048)
Network layers
Figure 40: MASC accuracy over the layers of the generalized ML.P network when the data set is projected onto corrupted training subspaces with the indicated corruption degrec. Rows corresponds to plots which have the same corruption degree & the columns correspond the generalized models with SGD and Adam noted. Training & test accuracy of the generalized model with SGD and Adam is shown.
62

---

Published in Transactions on Machine Learning Research (02/2026)
Class
Gass
Class
Class
Casss
MLP-SGD-MNIST
MLP-Adam-MMO
MLP-SGD-CIFAR-10
MLP-Adam-CIFAR-10
0% corruption degree
600
1200
1300
200
500
1250
250
400
1000
300
600
750
50
200
Impot
PC (133)
(St2)
FC c24m
FC (20a))
lapir
FC
F612
(244244)
FC (204)
Inpur
FC1220)
FC15Sa)
FC (048)
Inpit
FC (23
FC (512)
FC
FC (200)
20% comuption degree.
250
1200
1500
200
150
800
1000
750
100
400
50-
200
impat
FC PC
FC(sta)
Fc (2048)
FC (2048)
rsput
FC(22)
FC112
FC (2146)
FC (2049)
Inpurt
PC(120)
FC(5ta)
FC (048)
FC (240)
Imput
FC (22)
(S12)
FC Go4a)
PC (208)
40% corrupton degree
1200
1500
1000
200
600
800
1250
250
Number of PCA components
400
750
100
400
500
200
50-
250
Input.
FC
FC(51a)
c248)
ic (208))
loput
FC(28)
FC
(2448)
FC (208)
Inpilt
#C1120)
FC1512)
FC R648)
Inpat
FC
FC/5122
FC Goan)
FC
60% corruption degree
1200
250
1500
1000
200
600
1250
1000
150
600
400
300
50-
200
Input
(Sta))) (C.0)0))
Ft (208))
Fc (208))
Inpit
FC028)
FC012)
(2022)))
Fc (204))
Inpoit
FC
FC(51a)
rc ceein) Fc coos)
Inpo
FC (C2)
PC (5122)
Fc coaa)
Fc (20R)
80% corruption degree
1200
250
800
1250
200
600
800
1000
150-
750
200
400
500
50
200
250
Inpst
FC (138)
FC (312)
(2048)
(2065)
Insut
(128)
FC 0122
(2022
(048)
Inputt
FC (128)
PC (S13)
(048)
c048)
Input
FC (128)
PC (512)
Go4d)
(2048)
100% corruption degree
800
1500
1000
200
600
1250
350
400
600
200
200
200
Inpa
PC (138)
(512)
FC (204))
FC (2045)
Input
(C28)2)
Fc1112
(2048)
(204))
Inpat
PC(129) (229)
FC(552)
FC grus)
FC c148)
Inpa
(328)
(512)
FC G048)
FC (248)
Network layers
Figure 41: Class-wise number of PCA components of the corrupted training subspace over the layers of generalized MLP network with various corruption degrees. Rows corresponds plots which have the corruption degree and the columns correspond to the models with SGD and Adam optimizer noted.
63

---

Published in Transactions on Machine Learning Research (02/2026)
retraining
after retraining
comuptte subspace
Model after retraining MASC subspace
MASC accuracy on best layer: Corrupted
MASC accuracy on best layer TTue
0% corruption degree
20% corruption degree
40% corruption degree
0.40
0.35
0.25
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
Test Accuracy
0.0
0.00
0.00
60% corruption degree
80% corruption degree
100% corruption degree
0.175
0.150
0.10
0.08
0.075
0.04
0.050
0.00
0.000
0.00
AlexNel -CI-A-AA---- 100
Mestiet:TTy Imagettet
AlextNet CIFAR 100
AlexNet- ImageNet
NestNel-CFFN 1000
NexNes Tiny ImageNet
AlexNet-dataset
Figure 42:
Test accuracies averaged over three runs on the 80% test dataset is plotted for different AlexNet- dataset pairs for various corruption degrees and for various models/ models/MASCC MASC classifiers. Model before retraining corresponds to the existing memorized model. Model after retraining on MASC corrupted subspace corre- sponds to model trained with training dataset relabeled using MASC corrupted subspace predictions the best layer. Model after retraining on MASC true subspace corresponds to model trained with training dataset relabeled using MASC subspaces corresponding to true label predictions on the best layer. MASC test accuracy on the best layers for corrupted and true label subspaces on existing corrupted models (before retraining) are shown for comparison. The best layer was identified using the validation set which was carved out of the test set for this experiment. Error bar represents the range on three different runs.
64

---

Published in Transactions on Machine Learning Research (02/2026)
MASC corrupted subspace
MASC subspace
80% corruption degres 60% corruption degree 40% corruption degree 20% coruption degrea
650
26
025
200
-625
-550
Decrease in fraction incorrect labels after relabelling
22
-12
:0
2.4
Oll tetoooontt
Aodde--S 250
199
Model-dataset
Figure 43: Decrease fraction of incorrectly labeled data points resulting from relabeling the training set expressed fraction of size incorrectly labeled points in the existing corrupted data, using the correspond- ing best-layer MASC classifier. That is if X is the number of data points incorrectly labeled in corrupted dataset and Y is the number of data points incorrectly labeled in relabeled datasct then each of these plots refer to XX-Y)/X. Rows corresponds to results with the same corruption degree. The first column correspond to results with MLP and CNN models and second column with AlexNet model, as noted. The fractions are averaged over three runs. Error bar represents the range on three different runs.
65

---

Published in Transactions on Machine Learning Research (02/2026)
MASC comupted subspace
MASC true subspace
80% corruption degree 60% corruption degree 40% corruption degrée 20% corruption degree
10-
Percentage of incorrect labels relabeled correctly
330
30-
20-
:00
320
60-
10057
ANuNELCSAS. 129
Model-dataset
Figure 44: The percentage of incorrect labels that correctly relabeled using MASC (corrupted subspace and true subspace) for all the models and for corruption degrees 20%, 40%, 60%, 80%. Rows corresponds to results with the same corruption degree. The first column correspond to results with MLP and CNN models and second column with AlexNet model, as noted. The percentage are averaged over three runs. Error bar represents the range on three different runs.
66

---

Published in Transactions on Machine Learning Research (02/2026)
NASC comupted subspace
MASC true subspace
20% corruption degree
20-
25-
30-
80% corruption degree 60% corruption degree. 40% corruption degree
Percent of correct labels relabeled incorrectly
25-
10
20-
40-
20
10W57
MAY?
CoN De 06-8-15
Ancdee.cANN.O 500
Model-dataset
Figure 45: The percentage of correct labels that are incorrectly relabeled using MASC (corrupted subspace
and true subspace) for all the models and for corruption degrees 20%, 40%, 60%, 80%. Rows corresponds to results with the same corruption degree. The first column correspond to results with MLP and CNN models and second column with AlexNet model, as noted. The percentage are averaged over three runs. Error bar represents the range on three different runs.
67

---

Published in Transactions on Machine Learning Research (02/2026)
Model before retraining
Model after retraining on MASC corrupted supspace
Model after retraining on MASC original pupspace
MLP.MNIST
MLP-CIFARID
CNN-MNIST
CNN-Fashion- MNIST
CNN-CIFAR-10
AlexNet MexxNet-CIFAR. 100AexxNet-iingg ImageNet
0% compupti Stgres
0.00
6.00
317
516
036
0.5
0.5
Test Accuracy
60% comppption Scores
90% conuetto dagies
200% computionn Scopec
Epochs
Figure 46: Test accuracy on 80% dataset is plotted for different model-dataset pairs for various corruption degrees over 10 epochs. The experiment was performed without using early stopping. Model before re- training corresponds to the existing memorized model. Model after retraining MASC corrupted subspace corresponds to model trained with training dataset relabeled using MASC corrupted subspace predictions on the best layer. Model after retraining on MASC true subspace corresponds to model trained with training datasot relabeled using MASC subspaces corresponding to true label prodictions on the best layer. Error bar represents the range on three different runs.
68


Reviews:
[{"review_id": "JzrD02do1g", "reviewer": "Reviewer_Fq5y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n\nConsider a classification task with a data distribution $D$ and an iid training dataset $T$ of size $m$ sampled from $D$. For a fixed percentage $p$, let $\\hat{T}_p$ denote a modified training, where $p$ percentage of inputs are relabeled with a uniformly random labeling function. Denote the distribution of the corrupted dataset $\\hat{T}_p$ by $\\hat{D}_p$. \n\nPrior work (Zhang et. al., Arpit et. al) demonstrates the ability of neural networks to fit the noisy dataset $\\hat{T}_p$. This phenomena suggests that the hypothesis class $H$ pertaining to a specific neural architecture is highly expressive and capable of classifying for learning tasks with complex distributions (e.g. $\\hat{D}_p$). \n\nSomewhat confusingly, prior work and this article refer to the above phenomena as “memorization” a term that is both ill-suited and imprecise but repeatedly misused in this thread of literature (including Arpit et. al., Stephenson et. al etc). \n\nNaturally as the percentage of noise $p$ increases, the distribution of the noisy training data $\\hat{D}_p$ is further apart from the true distribution $D$ and thus one expects that predictors such as $h$ learned to fit samples $\\hat{T}_p$ from $\\hat{D}_p$ are ill-suited for labeling inputs as per $D$. In this article, the authors investigate whether a network $h$ trained to fit noisy data $\\hat{T}_p$ contains layer-wise representations with predictive power for the true noiseless distribution $D$. \n\nThe authors, suggest that, from the “internals of $h$” -- layer-wise representation embeddings computed by $h$-- one can identify subspaces $S_{l,c}$ for each class $c$ and layer $l$ that could inform a variant of nearest neighbour classifier $h_{MASC, l}$ where for each input $x$, the predicted label is determined as,\n$$\\text{prediction}(h_{MASC, l}, x) =  \\min_c ~\\text{angle}\\left(x_l, \\text{projection}(x_l, S_{l,c})\\right).$$ \nHere $x_l$ is the l-th layer representation internal to $h(x)$. \n\nExperiments on MNIST, Fashion-MNIST and CIFAR100 show that the modified classifier $h_{MASC,l}$ often exhibits improved performance on test data from the original noise-less data distribution $D$. This procedure requires no retraining and suggests a way for learning accurate classifiers from noisy data. The article also suggests that a version of self-training where all labels (including the noisy labels) are re-labeled using $h_{MASC,l}$ and the neural weights that are fine-tuned on this relabeled dataset $T_{p,MASC,l}$ exhibits better performance on $D$. \n\n\n## Strengths\n1. **Experimental evidence**\n\nThe empirical study is systematic. The authors evaluate multiple networks, corruption levels, and both corrupted-label and true-label subspace constructions. The results generally affirm the utility of MASC as a tool for recovering performance on noise-less data. \n\n2. **Lightweight method**\n\nMASC is straightforward to implement, computationally inexpensive, and easy to interpret. This is a valuable property compared to conventional probes that require additional training.\n\n## Weaknesses\n1. **Usage of the term “Memorization”**\n\nThe poor performance of a network $h$ fit to training dataset $T_p \\sim D_p$ when evaluated on $D$ is not memorization. It is perhaps more appropriate to refer to this as “overfitting” if $h$ attains low error on $T_p$ but high error on $D_p$. Otherwise, this is simply learning under noise. \n\n\nAchieving low training error on $T_p$ is an outcome of 3 factors: (1) expressivity of $H$, (2) level of noise $p$, and (3) sample size $m$.  In the extreme, when $p = 100$, for any input $x$ in the domain, all labels are equally likely and thus *all* deterministic functions $h \\in H$ exhibit the same test error: $\\frac{C-1}{C}$ where $C$ is the number of labels. When the sample size $m$ is small such that for a training input, there is only one observed pair $(x,c)\\in T_p$ then the learning algorithm can learn a neural network that predicts the label $c$ for $x$ and this way, can attain low training error on $T_p$ while exhibit low error on $D_p$, i.e. overfit w.r.t $D_p$.  Thus, achieving low training error on $T_p$ shows that the hypothesis class $H$ contains a sufficiently complex labeling function $h$ capable of labeling as per $T_p$ and such a phenomena requires low noise levels and small sample sizes. Even for $p=10$ with sufficiently large sample size, if one observes $(x,c)$ for all $c$ in the training data then a deterministic labeler cannot achieve low training error. \n\nThe continual insistence on terming this phenomena memorization rather than simply learning under noise is puzzling. Poor learning w.r.t $D$ is not the automatically the same as memorization.. especially if the training input are not i.i.d from $D$! \n\n2. **Ambiguity of the usage of the term “generalization”**\n\nAfter multiple reads, it appears the authors consider good or bad generalization to always mean “performance on D” regardless of the training distribution. This is not the precise usage of the term! If a network is trained on $T_p \\sim D_p$ then generalization should technically mean performance on $D_p$ after learning from $T_p$. \nWhile I understand the intention of the authors: to investigate whether $h$ learned on $T_p$ can be modified to perform well on $D$, I would strongly suggest that the authors be very precise about each usage of the term generalization. Some of the ambiguity is perhaps easily resolved if the authors set down more formal notation and identify the exact training and testing setup with symbols for the datasets and distributions (as in this review) and to rewrite instances of the word generalization with appropriate precision. \n\n\n3. **The two motivating hypotheses are not mutually exclusive**\n\nThe paper contrasts two explanations for poor test accuracy under label corruption: (i) degraded representations and (ii) intact representations with poor readout. In practice, both mechanisms may be at play, especially given evidence that later layers overfit while earlier layers retain structure. A revised exposition should acknowledge this and situate MASC as quantifying the extent to which useful signal persists. Once again, the framing of this article is unclear and could be more precise. This is a serious issue as it impacts readability and either prevents other researchers in the field from building on the authors’ contributions or amplifies the existence of vague terminology in the literature. \n\n4. **The main contribution is underemphasized**\n\nThe discussion in the last page briefly mentions that the experiments “suggest that MASC enables leveraging weights of models learnt on noisy data,” but treats this as future work. In fact, this capability appears to be the real (and only) contribution of this article. Elevating this point—namely, that noisy-trained weights can be repurposed effectively via subspace decoding—would greatly strengthen the manuscript. I would strongly recommend that the authors invest time/experiments to bolster this argument. \n\n5. **Reported percentage improvements exaggerate effect sizes**\n\nThe manuscript frequently highlights percentage improvements measured relative to somewhat small baselines (e.g., 198%). These numbers can be misleading and should be replaced with absolute accuracy differences (e.g. model learnt on noisy data attains error A on noise-less test data, while model modified with MASC prediction rule attains error B > A), which would give a more transparent sense of improvement.\n\n6. **Stylistic and terminological issues**\n\nThe manuscript occasionally uses informal terminology (e.g. words such as agnostic, corollary, memorization) without regard to their protected meanings (e.g. agnostic in the context of generalization!). On a minor stylistic note, there is inconsistent and over-capitalization of certain words (e.g., “Deep Networks”). Revising these phrases for clarity and style-consistency would greatily improve readability.", "claims_and_evidence_yn": "No", "claims_explanation": "The empirical findings are interesting, and MASC is a clean methodological contribution. However, the article's framing of contributions in the title, abstract and main summaries are vague and imprecise translation of their empirical finding.", "audience_yn": "Yes", "audience_explanation": "Understanding the ability of neural networks to learn under noisy data is an important and relevant direction of inquiry.", "requested_changes": "This article would greatly benefit from improved conceptual framing and clearer articulation of its central message. With revisions emphasizing the utility of representation learned under noisy settings, the paper could make a stronger contribution. Please address the weakness listed above, by rewriting sections wherever appropriate, and being formal and precise using more mathematical notation.", "broader_impact_concerns": "I don't foresee any broader impact concern.", "additional_comments": "I apologize to the authors and the action editors for the delay in the review process.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZRMr1AybQA", "reviewer": "Reviewer_mKtr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a simple **Minimum Angle Subspace Classifier (MASC)** to “decode” latent generalization from trained deep networks—even when those networks have memorized noisy labels. For each layer, the authors estimate **class-conditioned PCA subspaces** from the (possibly corrupted) training features, then classify a test point by the class whose subspace yields the **smallest angle** to the point’s feature vector. Across MLP/CNN/AlexNet models and MNIST/Fashion-MNIST/CIFAR-10/100/Tiny-ImageNet, MASC at **at least one layer** often **substantially exceeds the model’s own test accuracy** under moderate–high label corruption. If true training labels are known post hoc, building subspaces with true labels yields even larger gains. The authors also explore **retraining** with labels relabeled by the best-layer MASC and report mixed improvements. The results support the view that **memorization does not destroy internal structure**; trained networks can retain **linearly decodable, class-structured geometry** in hidden layers despite noisy supervision. \n\n**Key strengths**\n\n* Conceptually clean, training-free probe with **no weight updates**; easy to implement and reason about.\n* Consistent empirical trend: **large test-accuracy improvements** over the base model at one or more layers under label noise.\n* Clear experimental setup spanning **multiple datasets, architectures, and corruption levels**.\n* Interesting **dual result**: true-label subspaces post hoc can unlock even higher generalization; plus a retraining pipeline that leverages the probe.\n\n**Key weaknesses**\n\n* **Theory is minimal**: no analysis of when/why class subspaces preserve angles under label noise; no guarantees for layer selection.\n* **Baselines are thin**: lacks comparisons to standard probes (linear probe/logistic regression, LDA/NCM, k-NN in feature space, cosine-to-class-means, ridge/SVM), or other subspace/classifier geometry methods.\n* **Hyperparameter transparency**: PCA variance thresholding (99%), sign-flip augmentation, and layer selection appear **heuristic**; sensitivity/ablation is limited.\n* **Methodological concern**: best-layer selection and early stopping use a **20% split of the test set** as validation in retraining, which risks evaluation bias.\n* Mixed **scalability and practicality** questions: per-class PCA for high-dim features/layers may be costly; unclear guidance for which layer(s) to use in practice.\n* Evaluation on **modern backbones** (e.g., ResNet-18/50, ViT) is limited; AlexNet is dated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The empirical evidence is **consistent and compelling within the reported scope**: layer-wise MASC repeatedly outperforms the trained model’s test accuracy under label corruption, often by large margins (e.g., >100% relative improvements at higher noise on several model/dataset pairs). The paper reports multiple corruption levels, multiple architectures, and both **corrupted-label** and **true-label** subspace variants, and shows that the phenomenon persists broadly. The retraining study—though mixed—adds credibility by probing **downstream utility** of the decoded signal. However, support would be strengthened with (i) **stronger baselines**, (ii) **statistical tests** across seeds, and (iii) avoiding **test-set splits for model selection**.", "audience_yn": "Yes", "audience_explanation": "The work addresses a **core question** in modern deep learning—how generalization and memorization coexist—and proposes a **lightweight, interpretable probe** that reveals **structure in hidden layers under heavy label noise**. This is directly relevant to readers interested in **representation analysis, robust learning under noisy labels, diagnostics/probing**, and practical techniques to **salvage generalization** from existing models. The results may also stimulate **theoretical work** on subspace geometry and **practical tools** for auditing noisy datasets.", "requested_changes": "**Critical (affect acceptance):**\n\n1. **Add stronger baselines:** Compare MASC to (a) **linear probes** (logistic regression / ridge / linear SVM) on frozen layer features; (b) **nearest class mean** with cosine; (c) **LDA/QDA**; (d) **k-NN** in feature space; (e) cosine-angle to **class mean subspaces** vs. PCA subspaces. Report accuracy and compute cost across layers.\n2. **Eliminate test-set leakage in selection:** Replace the 20% test split used for **best-layer selection/early stopping** with a **proper validation set** carved from training (or a held-out set). Re-run retraining results accordingly and discuss any changes.\n3. **Sensitivity/ablation studies:** Vary PCA variance thresholds, the **sign-flip augmentation**, number of components per class, distance metrics (principal angles vs. first-component angle), and **layer choice** strategies (e.g., earliest layer passing a threshold). Quantify robustness across **seeds** (≥5) with **mean ± 95% CI** and simple significance tests.\n4. **Modern backbones:** Include **ResNet-18/50** (and, ideally, a small **ViT**) to demonstrate that the effect holds on contemporary architectures.\n5. **Clarify computational complexity:** Provide **time/memory** costs for building per-class subspaces and for inference across layers; discuss scaling to high-dimensional layers.\n\n**Important but not strictly critical (would strengthen the paper):**\n\n6. **Theory sketch or intuition:** Provide analysis (even stylized) explaining why **class-conditioned subspaces** preserve **angle-based separability** under label noise and why later layers often help. Connect to **representational geometry** and known results on **early-learning** vs. memorization.\n7. **Failure analysis:** Systematically report **where MASC fails** or brings marginal gains (datasets, layers, corruption levels); examine **when true-label subspaces** help the most.\n8. **Corruption types:** Beyond uniform label shuffling, test **instance-dependent** or **class-dependent** noise; discuss domain shift vs. label noise.\n9. **Code and reproducibility:** Release code and configs; document PCA implementation details, preprocessing, and exact layer taps.\n10. **Fair baseline training:** Include baselines with **regularization/early stopping** typical for each dataset to ensure model test accuracies are not artificially deflated.", "broader_impact_concerns": "The method is primarily an **analysis/probing tool**; direct societal risks are limited. Potential concerns:\n\n* **Data leakage or privacy:** If subspaces can **reconstruct or expose** structure from models trained on sensitive data, this may aid **information extraction** beyond intended usage. Clarify what MASC reveals about **individual examples** vs. class structure.\n* **Relabeling risks:** Using probe-generated labels to retrain could **amplify biases** or entrench mislabeled patterns if not validated on clean data. Recommend guardrails (hold-out validation, uncertainty thresholds, human oversight).\n* **Misinterpretation:** Emphasize that improved probe accuracy **does not imply** the base model generalizes well; MASC is a **diagnostic** and should not be conflated with deployed performance.\n\nAdding a concise **Broader Impact** paragraph noting these points would be sufficient.", "additional_comments": "* Writing is clear and figures are helpful. Consider **re-ordering** experiments: first establish the core MASC effect vs. strong baselines on modern backbones, then show **true-label subspaces** and finally the **retraining** study.\n* Report **per-layer dimensionalities** of PCA subspaces (already touched in the supplement) alongside accuracy to illuminate the complexity–performance trade-off.\n* Consider evaluating **principal angles** (full spectrum) or **subspace affinity** metrics instead of/min addition to the single-projection angle; this might further stabilize performance.\n* Explore a **layer aggregation** variant (e.g., voting or stacking simple probes across 2–3 adjacent layers) to avoid brittle best-layer choice.\n* Minor: define all notations once (Sk, angles, projection operator); ensure consistent treatment of **activation vs. pre-activation** features.", "extra_scores": {}, "extra_text": {}}, {"review_id": "d4Ll2zaJG4", "reviewer": "Reviewer_fBaQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n\nThis paper explores the intriguing question of why overparameterized deep networks trained on corrupted labels tend to generalize poorly, despite achieving perfect training accuracy. The authors provide compelling evidence that such networks still retain **latent generalization capability** within their internal representations. To reveal this hidden potential, they propose the **Model Activation Subspace Classifier (MASC)**—a simple yet effective decoding framework that extracts generalizable information from the learned representations of memorized networks. Through extensive experiments, the authors demonstrate that MASC can dramatically boost test accuracy relative to the original models, even under substantial label noise. Overall, this work offers fresh empirical insights into the nuanced interplay between memorization and generalization in deep learning.\n\n## Strengths\n- The paper is clearly written, and largely self-contained, making it accessible to a broad audience.  \n- The empirical evidence is thorough and convincingly supports the central claim that memorization does not necessarily destroy generalization.  \n- The results challenge prevailing assumptions in the deep learning literature and introduce a novel interpretability tool (MASC) with strong practical implications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the submission are generally supported by accurate and convincing empirical evidence. The authors conduct comprehensive experiments across multiple architectures (MLP, CNN, and AlexNet) and datasets (MNIST, Fashion-MNIST, CIFAR-100), demonstrating substantial and consistent performance gains achieved by MASC. The quantitative improvements—often exceeding several hundred percent over the baseline—clearly validate the central claim that memorizing networks retain latent generalization ability. The presentation of results is clear, and the comparisons are fair and reproducible. However, while the empirical evidence is strong, the lack of theoretical analysis or experiments on large-scale modern architectures somewhat limits the depth of the conclusions. Overall, the evidence convincingly supports the primary claims but would benefit from additional theoretical or large-scale validation.", "audience_yn": "Yes", "audience_explanation": "The issues of poor generalization and memorization in deep networks are critical in the literature, and understanding how to resolve them and what are the underlying mechanisms is highly essential. Those insights can be further adapted to foundation models, thus boosting the development of modern AI.", "requested_changes": "## Suggestions\n- The central role of **PCA** in constructing MASC is mentioned but not conceptually well explained. Providing more intuition would help readers better understand its necessity and contribution to the method’s success.  \n- While the experiments are well executed, they are limited to relatively **small-scale models and datasets**. Evaluating MASC on larger architectures (e.g., ResNet, ViT) and providing **theoretical justification** for its effectiveness would greatly strengthen the paper’s impact and generality.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)