Empirical Studies on the Convergence of Feature Spaces in Deep Learning
Authors: Haoran Liu
Haoyi Xiong
Yaqing Wang
Haozhe An
Dongrui Wu
Dejing Dou
Venue:  (2021)

Abstract: While deep learning is effective to learn features/representations from data, the distributions of samples in feature spaces learned by various architectures for different training tasks (e.g., latent layers of AEs and feature vectors in CNN classifiers) have not been well-studied or compared. We hypothesize that the feature spaces of networks trained by various architectures (AEs or CNNs) and tasks (supervised, unsupervised, or self-supervised learning) share some common subspaces, no matter what types of DNN architectures or whether the labels have been used in feature learning. To test our hypothesis, through Singular Value Decomposition (SVD) of feature vectors, we demonstrate that one could linearly project the feature vectors of the same group of samples to a similar distribution, where the distribution is represented as the top left singular vector (i.e., principal subspace of feature vectors), namely $\mathcal{P}$-vectors. We further assess the convergence of feature space learning using angles between $\mathcal{P}$-vectors obtained from the well-trained model and its checkpoint per epoch during the learning procedure, where a quasi-monotonic trend of convergence to small angles has been observed. Finally, we carry out case studies to connect $\mathcal{P}$-vectors to the data distribution, and generalization performance. Extensive experiments with practically-used MLP, AE and CNN architectures for classification, image reconstruction, and self-supervised learning tasks on MNIST, CIFAR-10 and CIFAR-100 datasets have been done to support our claims with solid evidences.

Under review as a conference paper at ICLR 2021
EMPIRICAL STUDIES ON THE CONVERGENCE OF FEA- TURE SPACES IN DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
While deep learning is effective to learn featureedrepresentttions from data, the distributions of samples in feature spaces learned by various architectures for dif- ferent training tasks, e.g., latent layers of Autoencoders (AEs) and feature vectors in Convolutional Neural Network (CNN) classifiers, have not been well-studied or compared. We hypothesize that the feature spaces of networks trained by var- ious architectures (AEs or CNNs) and tasks (supervised, unsupervised, supervised learning) share some common subspaces, no matter what types of ar- chitectures or whether the labels have been used in feature learning. To test our hypothesis, through Singular Value Decomposition (SVD) of feature vectors, we demonstrate that one could linearly project the feature vectors of the same group of samples to similar distribution, where the distribution is represented as the top left singular vector (i.e., principal subspace of feature vectors), namely Pevector. We further assess the convergence of feature space learning using angles between P-vectors obtained from the well-trained model and its checkpoint per epoch dur- ing the learning procedure, where quasi-monotonic converging trend from nearly orthogonal to smaller angles (c.g.. 10") has been observed. Finally, we carry out case studies to connect P-vectors to the data distribution, and generalization performance. Extensive experiments with practically-uset Multi- Percep- tron (MLP), AE and CNN architectures for classification, image reconstruction. and self self-supervised learning tasks on MNIST, CIFAR- 10 and CIFAR- 100 datasets have been done to support our claims with solid evidences.
self-
INTRODUCTION
Blessed by the capacities of feature learning, deep neural networks (DNNs) (LeCun et al. 2015) have been widely used to perform learning tasks, ranging from classification, generation Goodffellow et al. 2014 Radford et al.) 2015), in various settings (e.g., supervised, unsupervised, and self- better analyze the features learned by deep models, numerous works have studied on interpreting the features spaces of the well-trained models (Simonyan et al. White 2016 Zhu et al. 2016 Bau et al. 2017 2019 Jahanian et al. 2020: Zhang & Wu, 2020).
supervised learning).
Invariance beyond the use of architectures and labels. While existing studies primarily focus on the interpolation of a given model to discover mappings from the feature space to outputs of the model (e.g., classification (Bau al., (017) and generation (Jahanian et al.) (0200) the work is so few that compares the feature spaces learned by deep models of varying architectures (c.g.. MLP/CNN classifiers versus Autoencoders) for different learning paradigms (Chen al. 2020)] Khosla et al.] 2020 Spinner et al. 2018). More specifically, we are particularly interested in whether there exists certain "statistical invariance" " the feature space, no matter what type of architectures or whether label information (e.g., supervised vs. unsupervised vs. self-supervised (Chen et al.)
2020) learning) are used in feature learning with the same training dataset.
Hypotheses. is not difficult to imagine that the feature spaces of well-trained DNN classifiers in
supervised learning setting might share some linear subspace /Vaswani et al. 2018). When models well fitted to the same training set, the feature vectors of training samples should be projected to the ground-truth labels after a Fully-Comnecter Layer (i.e., a linear transform), while such lin- subspace are supposed to distribute samples in discriminative manner. We doubt that such

---

Under review as a conference paper at ICLR 2021
(a) Cosine (Train)
(b) Cosine (Test)
(c) Supervised (Train)
(d) Supervised (Test)
Won' Sath Moo
(e) Unsupervised (Train) (I) Unsupervised (Test)
(g) Self- (Train)
(h) Self-Sup. (Test)
Figure 1: The Common Feature Subspace and Converging Trends of P-vector Angles with CIFAR-10. Figure 1. (a)-(b) present cosine (in the range of [0,1]) of angles between the P-vectors of well-trained models of various architectures under different learning paradigms, using training and testing datasets respectively. A well-trained model here is the one trained under the suggested settings after 200 epochs for supervised/selff sfupeersessssssseeeeieeee CNN classifiers and 100 epochs for umsu- pervised AEs. Figure 1. (c)-(h) present angles of PP-vectors between the well-trained model and its checkpoint per training epoch of three learning paradigms, where the converging trends of P--vector angles from nearly-orthogonal to smaller ones have been observed in all models, no matter whether the feature extractors of these models are trained with / without labels. Note that we carried out ex- periments with different random seeds independent trials to obtain the averaged results above. More discussion provided in Section 4.
subspace might be not only shared by supervised learners but also with AEs which are trained to reconstruct input data without any label information in unsupervised manner, even shared with self-supervised DNN classifiers (e.g., SimCIR (Chen et al., 2020)) which train (1) CNN fea- ture extractor (using contrastive loss without labels) and (2) linear classifiers (using discriminative loss based on labels) separately in an ad-hoc manner. More specifically, we hypothesize that (H.I:) there exists certain common feature subspaces shared by well-trained deep models using the same training dataset, even though the architectures (MLPs, CNNs, and AEs) and the learning paradigms (supervised, unsupervised, and supfrsuper are significantly different. Further, as the training procedure usually initializes the DNN models from random weights and learns features from the training set step-by-step, we hypothesize that (H.II:) the training procedure gradually shapes the feature subspace over training iterations and asymptotically converge to the common subspace in certain statistical measure. Finally, we hypothesize that (H.III) the convergence to the common fea- ture subspace would connect to the data distribution and performance of models, as such behavior indicates how well the features are learned from data. This hypothesis is motivated by the observa- tion that when the DNN model tends be linear the DNN feature subspace should be close the data subspace, while the well-trained DNN models should be locally linear (Zhang & Wu 2020) or piecewise linear Arora al. 2018).
Contributions. To test above three hypotheses, this work makes contributions in proposing new measures to the DNN features, namely P-vectors, and conducting extensive experiments for empir- ical studies. We train deep models using various DNN architectures, multiple learning paradigms, and datasets, with the checkpoint restored per epoch. Then, we extract the feature vectors for either training or testing sample sets, from the model (Please see Section 3 for details) and discover some interesting relationships associations discussed below.
P-vector and Convergence: Given the matrix of feature vectors (#samples i x #features) for ei- ther training testing samples, perform the singular value decomposition (SVD) to obtain left and right singular vectors, characterizing the subspaces that samples distribute and the projection of features to subspaces respectively. We observe that deep models well-trained using the same
We follow the convenience that denotes # as the term "the number of for short.
2

---

Under review as a conference paper at ICLR 2021
dataset share similar top left-singular vectors (referring to the principal subspace of feature vectors), namely P-vector in this study. For example, Figure (b) show the high cosine similarity (close to 1) between the P-vectors of any two well-trained DNNs for eight different deep learning archi- tectures/tasks. This observation well backups H.I, where the common subspace in feature spaces shared by models trained with different architectures/prrddiins has been observed. With check- points obtained along the training procedure, we estimate the angle that compares P-vectors of the checkpoint per epoch and the well-trained one. We find such angle decrease over the number of epochs (in an overall manner) and would converge to smaller ones. For example, Figure 1 (c)-(h). demonstrate the consistent converging trends of angles between P-vectors of the well well-trained model and its checkpoints in progress of training for eight different deep learning architecturesstask.. This observation supports H.II, where convergence to the common feature subspace are expected.
II. Data Distribution and Performance: feature vectors and original data vectors of samples using P-vectors. In addition to the P-vector es- timated from the feature vectors of deep model (namely "model P-vector"), We simply form the sample vectors (either for training or testing) into a #samples x SVD to obtain the top left singular vector as the P-vector of samples (namely a "data P-vector"). " We estimate the angles between the model and data P-vectors, and find a trend of convergence (from nearly orthogonal to relatively small angles), where we can see the well-trained models would in- corporate smaller angles than ones in the early stage of training processes. We further correlate such angles with training and testing accuracy of the models, where we observe significant negative cor- relations in most cases of experiments. The trends show that the model with smaller angle between the model and data P-vectors would enjoy better performance. The evidences backup IIIII.
To connect to the data distributions, we intend to compare
matrix and perform
2 RELATED WORK
In this section, we first present the preliminaries in understanding feature learning of DNNs, then discuss the most relevant works to our studies.
As early as 2013, (Simonyan al., 2013) proposed to visualize the features learned by deep volutional neural networks (CNN) and made sense of discriminative learning via deep feature ex- traction. For generative models, (White 2016: Zhu al. 2016) studied the interpolation of latent spaces while (Zhu et al.] 2016) discovered an user-controlless way to manipulate the images gener- ated through the surrogation of latent spaces via manifolds. Later, (Bau et al.) 2017) presented the visual concepts learned in the feature spaces of discriminative models through network dissection on specific datasets while the same group of researchers also proposed GAN dissection (Bau al., 2019) an interactive way to manipulate the semantics and style of image synthesis. (Richardson & Weiss 2018) compared GAN and Gaussian Mixture Models (GMMs) to understand the capacity of distribution learning in GAN. (Berthelot al. 2019) proposed to improve understanding and interpolation of Autoencoders using adversarial regularizer while (Spinner et al., 2018) compared AEs with its variational derivatives to interpret the latent spaces. More recently, (Saxe et al. 2019) mathematically analyzed the process of neural representation construction from the perspectives of learning dynamics of deep neural networks. (Jahanian et al., 2020) studied the "steerability" of GAN, where it discovered point-to-point editing paths for content/style manipulation. (Zhang & Wu 2020) uncovered the phenomena that DNN classifiers with piecewise linear activation tend to map the input data to linear subregions. Other impressive studies in this line of research include
Nguyen al. 2016 Arvanitidis et al. 2018 Sereu et al. 2019).
Discussion. The most relevant studies to our work are (Bau et al., 2017 Zhang & Wu, 2020 Saax
et al. 2019 Lee et al. (211)). For discriminative models, (Bau et al. (2117) recovered visual features learned by CNN classifiers with a priorly labeled dataset, and quantified then compared the feature learning capacities (namely "interpretability"" in the work) of different DNN models through patterns matching with the ground truth. Compared to (Bau et al.] 2017). we carry out the empirical studies on their features and observe consistent phe-
wide range of datasets without any prior information nomena in the distribution of samples in the feature spaces. Furthermore, while (Zhang & Wu 2020)
In our research, we name convergence as the decreasing trend of P-vector angles from a larger one to a smaller one (e.g., 10" for supervised CNN classifiers and SimCI., < 10" for Fss//nns// and 30 for SupCon) over training epochs. For reference, (0sine(10))00995 is close 1.0
3

---

Under review as a conference paper at ICLR 2021
#fratures
nxck look kxd
Druader
#ssmplns
Nemple
Temalc
P.oector
Classilers and AF
Feature Extraction for Every Sample
Randomized SVD to obiain Toc Stogulor Verter the .P.vecoor
Figure 2: Obtain P-vectors of deep neural networks using a group of samples
studied the properties of regions where supervised DNN classifier with piecewise linear activation behaves linearly, our work observes the common linear subspaces shared by the features learned by the networks that trained with different architectures (e.g., MLP/CNN classifiers and AEs with ReLU activation) and paradigms (c.g., supervised, unsupervised, and self-supervised learning). Fur- thermore, both work and (Saxe al. 2019) propose to compare feature/rerresentatinn learning by various models through SVD DVD, while we perform SVD to investigate the distribution of samples in feature space and (Saxe et al. 2019 uncovered the latent structures in input-and gouut-and-ous and-output of neurons.
To understand the latent space of generative models, some statistical/mathematiie tools (Richardson & Weiss 2018, Lee al., 2019) have been introduced. Through comparisons between GAN and GMM, (Richardson & Weiss 2018) uncovered GAN's superiority in feature/texture reconstruction and in the meanwhile incompetence in distribution learning, while (Lee al.) 2019) provides some analytical insights the structure of latent spaces. In our work, we propose P-vectors to model the distribution samples the feature/latent spaces DNN classifiers and AEs. To the best of our knowledge, make unique contributions compared to the above work.
METHODOLOGIES: FEATURE VECTORS, SINGULAR VALUE DECOMPOSITION, AND P-VECTORS
As mentioned in Section. 1, we carry out extensive experiments on analyzing the feature vectors of DNN models with various architectures and different training paradigms through the newly- proposed P-vectors. this section, we present the design of experiments details, where we cover three key procedures of the experiment as shown in Figure 2
Feature Vector Extraction. Given a model, either the well-trained one or checkpoint obtained during the training process, we extract the feature vector for every sample, with respect to the archi- tectures. For DNN classifiers (either under pneervised/sel-ssueere of CNN feature extractor (i.e., the input to the Fully-Connected Layer) as the feature vector of the given sample, while we vectorize the output bottleneck layer as the feature vector for AFs. Note that, in our research, we consider AEs with symmetric architectures of encoders and decoders only.
learning), we use the output
Singular Value Decomposition with Feature Vectors. Given the feature vector for every sample, we form #samples #features matrix and perform SVD to obtain the top left singular vector as the P--vector. Furthermore, there no need to solve the singular vectors of the complete spectrum, as only the top singular vector is requested for P-vector estimation. In this way, we propose to use Randomized SVD (Halko et al. 2011I) that compresses the feature domain and approximate the low-rank structure of SVD for acceleration purpose. Actually, compare the numerical solution of Randomized SVD and Common SVD for ResNet-50 on CIFAR-10 dataset (#features=256 and #samples= 50,000), where need to perform SVD #samples #features matrix and the P. vector/top left singular vector should be with dimensions. Compared to the vanilla SVD, around 109x (from 44.88 seconds 0.41 seconds) speedup has been achieved by Randomized SVD while no significant numerical errors having been found in the results.
P-vectors, Principal Subspace, and the Distribution of Samples in Feature Spaces. The top left singular vector of #samples #feature matrix represents the principal subspace (Jolliffe 2002) that samples are distributed into a one- mee-dimensional space (with the maximal variances) by the pro- jection of principal component (Abi & Williams 2010) of features. In this way, can compare
4

---

Under review as a conference paper at ICLR 2021
and measure the divergence between the feature spaces of two DNNs through assessing the angle between their P-vectors based on the same set of samples.
Discussion. We plan to discuss the properties or P- vectors from the aspects as follow.
Significance of P-vectors. possible threat to valid- ity of using P-vectors to analyze the feature space of DNN is that P-vectors might fail to capture the necessary information to represent the features learned. To validate the capacity of P-vectors to represent the feature space, we carried out Singular Value Decomposition on the matrix feature vectors, using CIFAR-10 and CIFAR- 100 datasets both based on ResNet-50 models, and obtain the distribution of singular values over indices. More specifically, the distributions of singular values for the feature matrices obtained in the 1st, 60th 120th 160th and 200th epochs monitor the change of singular value distributions throughout the training pro- cedure. It has been observed in Figures 3|a) and (b) that "cliff" pattern in the distribution becomes more and more significant after epochs of training for both CIFAR-10 and CIFAR- datasets very small number (less than 10) of top singular values might dominate the whole distribution. In Figure c)), we plot the curve of explained variance ratio of / j== of for every pair of singular vectors, using well-trained models of 200 epochs based on CIFAR-10 10 and CIFAR-100, where on refers to the hth singular value and d the rank of matrix. The explained variance ratio of the top-1 singular vectors (i.c., the P-vector and the top-1 right singular vector) is more than 50% while the second top singular vectors are less than 10%. Results show the use of P-vectors could represent the features learned. In addition to the use of top-1 singular vectors (the P-vector), in Appendix (A.7), we also discuss the results of including more singular vectors in analysis, where no consistent observations have been obtained.
(a) CIFAR-10
(b) CIFAR-100
(c) Ratios
Figure Singular Value Distribution and Explained Variance Ratios
of the Matrix of Feature Vectors
compute
Distribution of Values in a P-vector: As a singular vector, a P-vector should be a unit-vector, where the sum of squared value on every di- mension of the vector should be 1. To take closer look at the P-vector, we first retrieve the value on every di- mension of the vector and count the frequency that each value appears. We normalize the frequency with the total counts and plot the smoothed probability density of the distributions (of the values in the P. vector) in Figure 4 (a) and (b) for CIFAR-10 and CIFAR- datasets. Specifically, we plot the distribution for P-vectors obtained in different epochs throughout the training procedure, where clear "concentration" process could be observed. In the beginning, probably due to the random ini- tialization, the values are flatten in a wider range. With the training epochs, the distributions in the both figures would be "concentrated" into narrow ones with reduced ranges. We could observe the peak shifts over training epochs in both figures, while the distributions based on two datasets are sig- nificant different from both magnitudes and ranges' perspectives. Examples on the raw frequency of values in P-vector is included Appendix (A.6), where same trends could be observed. Note that two P--vectors are not necessary to be close, when their distributions (of values) are close. Because the specific value assigned to every sample in the P-vector could be significantly different. Thus, analyzing the distribution of values in P-vectors with respect to the distribution of data might be part of future work.
4.7
Boeh d
Li
0:08
0:0
(a) CIFAR-10
(b) CIFAR-100
Figure 4: Distribution of Values in the P- P-Vector
Ways to compare P-Vector. Given the same set of samples, P-vectors of the two DNNs should be in the equal length, as they are both the top singular vectors in the sample side. Furthermore, we can easily measure their divergence via Cosine of two vectors. A larger cosine of two vectors (e.g., close to 1.0) usually refers to the evidence that the two networks share subspace in their features. Note that, in high-dimensional spaces, the chance of orthogonality between two random vectors
5

---

Under review as a conference paper at ICLR 2021
(a) MNIST (Train)
(b) MNIST (Test)
(c) CIFAR-100 (Train)
(d) CIFAR-100 (Test)
Figure 5: Cosine of angles between the P-vectors of well-trained models
appears more frequently, due to the curse of dimensionality (Pestov) 1999). Thus, given sample set such as CIFAR-10 with more than 60,000 samples, when the cosine measure close to 1.0 or the angle between the two P-vectors (with 60,000 dimensions) is small, we can conclude that the two networks would share a subspace in the feature spaces in high confidence. Of-course, there might exist other ways perform analysis using P-vectors. In our future work, we plan leverage advanced numerical tools Björck & Golub (1973) to estimate the angles between the subspaces of DNN models in general dimensions.
UNCOVERING COMMON FEATURE SUBSPACE WITH P-VECTORS
Common Subspaces. To demon- strate the existence of common sub- spaces, we propose to measure the angles between the P-vectors tracted from well-trained models us- ing the same datasets. For any two models, a small angle between the two P-vectors refers to the facts that the two models would project these samples to similar principal subspaces in feature spaces, as the P-vectors are estimated as the top left singular vector of the feature ma- trix (i.e., #samples #features) #ffeaturs SVD based on the same group of samples. Figure. [ (a)-(b presents the co- sine of angles between the P-vectors of well-trained models (trained with various architectures and tasks using CIFAR- dataae)0dattsee)., To generalize our observations, we propose to compare the P-vectors of models using the feature matrices based on the training and testing samples in Figures. (a)
(a) Sup. vs. Sup.
(b) Sup. vs. Unsup (c) Sup. vs. Self-Sup.
(d) Unsup vs. Sup. (c) Unsup vs. Unsup. Unsup vs Self-Sup
(g) Self. Sup. (h) Self. Unsup (i) Self. Self-Sup. Figure 6: P-vector angles between checkpoint per epoch and well-trained models using CIFAR-10.
and (b) respectively. We carry out the same experiments on CIFAR-10 and CIFAR-100 datasets, and
present the cosine of angles between the P-vectors of well-trained models in Figure 5 All experi- ments demonstrate that, with a relatively large cosine (close to 1), the models well-trained using the same dataset, matter what types of architectures or whether labels has been used in the feature learning of various tasks, share a common subspace.
Converge to the Common Subspace To understand the dynamics of feature learning that shapes the common subspace from scratch, we measure the change of angles over epochs between the P- vectors of training models (i.e., checkpoint in the epoch) and well-trained ones in comparisons. In Figure D (c)-(h, for every model of various tasks, we present angles between the P-vectors of the well-trained model and itself's 's training checkpoint per epoch. A consistent convergence could be observed. As the P-vectors of well-trained models close to each others (see in Figures and (5), we can conclude the feature vectors extracted by these models would evolve on time and gradually converge to share the common subspace during the learning procedure.
6

---

Under review as a conference paper at ICLR 2021
We carry out more comprehensive model-to-modee comparison using CIFAR-10 dataset. Fig- ures. (a)-(c) present the convergence of P-vector angles between the well-trained supervised models and the checkpoints per epoch of supervised, unsupervised, and self-supervised learning models, where we use the well-trained Wide-ResNer28 (trained with 200 epochs under suggest set- tings) as the reference of supervised models. Figures. [ (d)-(f) present P-vector angles between the well-trained ConvAE (trained with 100 epochs under suggest settings as the reference of unsu- pervised learning) and the checkpoints per epoch of supervised, unsupervised, and self-supervised learning models. Figures. (g)-() present the P-vector angles between the well-trained SimCLR representations (trained under suggest settings (Chen et al. (020) as the reference of self-supervised learning) and the checkpoints per epoch of supervised, unsupervised, and self-supervised learning models. The trends convergence all comparison further validate our hypotheses.
Discussion. To better visualize ev- the maxi- mal angles achieved during the first epoch to represent the angles between P-vectors corresponding to the first epoch in Figures Jaaures[andd and Actually, in the first epoch, there would incorpo- rate some -o-moonttonic trends for the angles varying over the number of iterations. Figure 7 presents the angles between the P-vectors of the training and well well-trained models over the number of iterations in the first epoch in three settings of learning, where dure that the learning procedure shapes the feature space - with the random weight initialization, the angles of P-vetoors between the initial models and the well-trained models large, the angles drop down and rebound a zigzag trend quickly in all three learning paradigms. In the rest of learning procedure, the angles between the P-vectors drop down and converge smaller ones over the number of iterations. (H.Il)
ery comparison,
(a) Supervised
(b) Unsupervised
(c) Self-Sup.
Figure 7: Angles between the P-vectors of the training and well-trained models over the number of iterations in the first epoch using CIFAR-10.
CIFAR-10 10 dataset for the experiments. In this way, we recover the proce-
Note that in Appendix (A.7), we include the comparison between the top singular vectors other than the P-vectors, i.c., the angles of top-2, 3, 4, 5 and 6 left singular vectors between models, where we cannot observe the converging trends.
5 CASE STUDIES ON P-VECTORS WITH DATA DISTRIBUTIONS AND MODEL PERFORMANCE
In this section, we intend to ex- tend our observations to connect P. vector with the (raw) data distribu- tions and the generalization perfor- mance of models.
Correlation to Data P-vectors. Given the raw data matrix, i.e.. a #samples commplesxddatt. #data dimensions matrix, we obtain the Data P- of these samples using the top left singular vector of the raw data matrix, which represents the principal subspace of data (or the position of every sample projected by the principal component of the data). To understand the con- nection between models and data, We
(a) Supervised C10
(b) Unsup. C10
(c) Self-Sup. C10
(d) Supervised C100 (e) Unsup C100
(I) Self- C100
Figure 8: Angles between the model and data P-vectors per training epoch. C1O: CIFAR-10, C100: CIFAR-100
We use the term "model "--vector" to represent the P-vector estimated using the feature vectors of deep model, while using "data P-vector" as the top left singular vector of the data matrix.
7

---

Under review as a conference paper at ICLR 2021
Gent- Ninaman's pu $4
codfi en: Sparncess 010111
1006-2
Toal AC
Teat ACC
Towae
Ter
(a) VN16-C10 (Train)
(b) VNI6-C10 (Test)
(c) RN20-C10 (Train)
(d) RN20-C10 (Test)
Spoarran confeirt GLRL
Spea dart prila Liteeds
Spcaman's pIArUE 2011
Spearar's prodic 6776-44
11.0
(e) RN56-C10 (Train)
(I) RN56-C10 (Test)
(g) RN20-C100 (Train)
(h) RN20-C100 (Test)
Swarmer's codftiint
535033
Speertee's pundls
Jesamaat ii coete
0.827
Sprormart cotficat
6.3125
444
Textace
Trsht Ace
Teetace
(i) RR56-C110 (Train)
(i) RN56-C100 (Test) (k) RNI 10-C100 (Train)
()) RNI10-C100 (Test)
Figure 9: Strong and consistent correlations between the model performance (training and test- ing accuracy) and the angels between model and data P-vectors using CIFAR- 10 and CIFAR-100 datasets. VN: VggNet, RN: ResNet, C10: CIFAR-10, 10, and C100: CIFAR-100.
carry out the case study using CIFAR- 10 dataset, so as to measure the angles between the model P- vector and the data P-vector and how the angles change over the number of training epochs.
Figure & shows the convergence of P-vector angles between the raw data and the checkpoints per epoch of supervised, unsupervised, and self-superrised learning models. The angles between the data and model PP-vectors start from about orthogonal and generally decrease and converge to about 50 degree to 60 degree. Note that the angle points corresponding to the first epoch on the curves are the largest P-vector angles during the training procedure of the first epoch from the random scratch. The results indicate that the principal subspace of the well-trained models are more close to the principal subspace of the data, matter what type of architectures is used for what kind of learning tasks and whether the labels are used for training.
Correlation to Deep Learning Performance. We further explore the relationship between the performance of the various models on different datasets and the angles between P-vectors and data P-vectors. Experiments are delivered using ResNett-0056/110 10 and VGG-16 on CIFAR-10/100 datasets. As shown in Fig 9 there exists strong and consistent correlation between the train- ing/testing accuracy of models and the angles between the model and data P-vectors. Note that in these experiments, we use the samples in the training dataset to estimate the model and data P. vectors while avoiding the use of validation information, so as the demonstrate the connection of P--eectors all based on training samples the generalization performance. For CIFAR-100 dataset, we use ResNets with pre-activation enabled.
To avoid the dominance of some outliers, conduct correlation analysis between the rank of model performance and the rank of angles by the use of Spearman's coefficients and p-values. With 0.05 as the threshold for significance, we find significance in the correlations between the angles be- tween model and data P-vectors and the training/testing accuracy for all above cases, except the correlation between the angles and training accuracy on ResNet-100 using CIFAR- dataset (p- valuc=0.09277 In Appendix (A.8), we also include an additional correlation analysis based on log-log plots, where we can further validate our observation. In this way, we could conclude that (1) both training and testing accuracy are correlated to the angles between the model and data P- vectors, (2) the strong correlations between angles and the testing accuracy might not be caused by the correlations between the angles and training accuracy, as the earlier ones are even stronger, (3) the angles between the model and data P-vectors would be a reasonable performance indicator, as they are strongly, consistently, and significantly correlated to the testing accuracy. This observation coincides our intuition that a significant (locally) linear term exists in the rell-trained model (Zhang & Wu 2020), which makes DNN feature principal subspace correlate to the data principal subspace.
8

---

Under review as a conference paper at ICLR 2021
Methods
Prediction Score
VC Dimension |Vapnik 2013)
0.020
Jacobian norm w.E. intermediate layers (Jiang et al.)
2.061
Angles between the model and data P-vectors
3.325
Distance to Initialization (Nagarajan & Kolter 2019)
4,921
Distance from Initialization + Angles between the model and data P-vectors
4.971
Sharpness of the convergence point (Jiang al. 2019)
10,667
Pseudo validation accuracy
13.531
Pseudo validation accuracy + Angles between the model and data P-vectors
15.618
Table 1: Scores of different method to predict the generalization gap using CIFAR-10 and SHVN.
Applications to Generalization Performance Prediction. To further demonstrate the feasibility of using the P-vector "validation- frree""iiio"""" measure of generalization performance, use the ex- periment settings of "Predicting Generalization in Deep Learning Competition" at NeurlPS 2020 to evaluate P-vectors on predicting the generalization performance of wide range of models using the training dataset. The competition offers a large number of deep models trained with various hyper- parameters and DNN architectures, while the official evaluator for the competition first predicts the generalization performance of every model using the proposed measure, then verify the prediction results through the mutual information (the higher the better) between the proposed measures and the (observable) ground truth of generalization gaps.
In the experiments, we propose use the angles between the model and data P-vectors using the training dataset as the metrics of generalization performance. In the comparisons with the proposed P-vectors, we include several baseline measures in generalization performance predictors, including VC Dimension (Vapnik] 2013), Jacobian norm w.r. intermediate layers (Jiang et al.). Distance from the convergence point to initialization (Nagarajan & Kolter, 2019), and the Sharpness of conver- gence point (Jiang al.) 2019). In addition to these methods, Accuracy" as a measure for comparisons, where this measure first uses random data augmentation apply to the original set of training data to generate set of "pseudo validation samples", then tests the accuracy of the model using "pseudo validation samples".
also propose "Pseudo Validation
Dooseettssnetts the comparisons between the proposed measures and baselines. It shows that when the proposed measure- angles between the model and data P-vectors stands alone, the measures significantly outperform the baseline methods including Jacobian norm w.r.t intermediate layers and the VC dimensions. However, through complementing with other metrics, the metrics based on P-vector angles could be further improved in predicting the generalization performance and finally outperform all bascline methods when combining with "Pseudo validation accuracy". " Note that we combine the results of two metrics through weighted aggregation (Pihur al. (2009), with constant weight 0.05) of two ranking lists that are sorted according to the two metrics respectively.
All above experiment results in three case studies backup our hypothesis that the P-vectors of the model feature spaces would connect to the data distribution and performance of models (H.III).
6 CONCLUSION
In this work, propose the P-vector, defined the top left singular vector of the feature ma- trix (the #samples x #features matrix) through SVD, to characterize the principal subspace of the feature space founded by the samples. We observe that, no matter what type of DNN architec- tures or whether the labels have been used to train the models, the angle of P--vectors between any two models would decrease smaller ones (e.g., around 10a for most models in this study and cosine( (00)=00.85) when the models are well trained with the same dataset. We conclude that the feature spaces all well-trained DNN models using the same training dataset would share a prin- cipal subspace. Furthermore, during the training procedure from the random scratch, the model P-vector would slowly approach to the data P-vector (defined as the top left singular vector of the raw data matrix, i.c., #samples #data dimensions), where the data and model P-vectors start from an altoos-orthogonal status while the angles between the data and model P-vectors overall decrease and converge to smaller ones (c.g., 50"-60") Finally, find that angles between the model and data P-vectors are strongly correlated to the performance of models (i.e., both training/testing ac- euracy) while they are capable of predicting generalization performance, even when the model and data P-vectors are all estimated using training dataset only. As was discussed, we believe the em- pirical observations obtained here are partially due the local lincarity of DNN models Zhang & Wu 2020); our future work may focus on the theoretical understanding to these phenomena.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433 459, 2010.
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. International Conference on Learning Representations, 2018.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: On the curvature of deep generative models. In International Conference on Learning Representations, 2018.
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Conference on Computer Vision and Pattern Recognition, 2017.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works. In International Conference on Learning Representations, 2019.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving inter- polation in autoencoders via an adversarial regularizer. In International Conference on Learning Representations, 2019.
Åke Björck and Gene H Golub. Numerical methods for computing angles between linear subspaces. Mathematics of computation, 27(123):579-594, 1973.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 2rXiv:2002.05709 2020.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53 (2):217-288, 2011.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations, 2020.
Yiding Jiang, Pierre Foret, Scott Yak, Behnam Neyshabur, Isabelle Guyon, Hossein Mobahi, Gintare Karolina, Daniel Roy, Suriya Gunasekar, and Samy Bengio. Predicting generalization in deep learning, competition NeurIPS 2020. URL htthhhtt://// httthttsss//steess http://sstsssggese gowgle.ooggee...ooooooooo. com/view//
|pgd12020 home authuser=0
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912.0277,, 2019.
It Jolliffe. Principal component analysis. Springer Series in Statistics, 2002.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11362 2020.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436444, 2015.
Dennis Lee, Christian Szegedy, Markus N Rabe, Sarah M Loos, and Kshitij Bansal. Mathematical reasoning in latent space. arXiv preprint 118.71190..1187,, 2019.
Vaishnavh Nagarajan and Zico Kolter. Generalization in deep networks: The role of distance from initialization. arXiv preprint arXiv: 901002900 2019.
10

---

Under review as a conference paper at ICLR 2021
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems, pp. 3387-3395, 2016.
Vladimir Pestov. On the geometry of similarity search: Dimensionality curse and concentration of measure. arXiv preprini cx/9901004, 1999.
Vasyl Pihur, Susmita Datta, and Somnath Datta. RankAggreg, an R package for weighted rank aggregation. BMC Bioinformatics, 10(1):62, 2009.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:15/1.066 2015.
Eitan Richardson and Yair Weiss. On GANs and GMMs. In Advances in Neural Information Processing Systems, pp. 5847-5858, 2018.
Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development deep neural networks. Proceedings of the National Academy of Sciences, 116 (23):1 2019.
Tom Sereu, Sebastian Gehrmann, Hendrik Strobelt, Payel Das, Inkit Padhi, Cicero Dos Santos, Kahini Wadhawan, and Vijil Chenthamarakshan. Interactive visual exploration latent space (IVELS) for peptide auto-encoder model selection. In ICLR DeepGenStruct Workshop, 2019.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop, 2013.
Thilo Spinner, Jonas Körner, Jochen Görtler, and Oliver Deussen. Towards interpretable latent space: an intuitive comparison of autoencoders with variational autoencoders. In IEEE VIS, 2018.
Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer Science & Business Media, 2013.
Namrata Vaswani, Thierry Bouwmans, Sajid Javed, and Praneeth Narayanamurthy. Robust subspace learning: Robust PCA, robust subspace tracking, and robust subspace recovery. IEEE Signal Processing Magazine, 35(4):32-55, 2018.
Tom White. Sampling generative networks. arXiv preprint urXiv:1609.04463 2016.
Xiao Zhang and Dongrui Wu. Empirical studies on the properties of linear regions in deep neural networks. In International Conference Learning Representations, 2020.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation on the natural image manifold. In European Conference on Computer Vision, pp. 597-613, 2016.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 COMPARISON OF ANGLES BETWEEN MODEL AND WELL-TRAINED P-VECTOR USING DIFFERENT ARCHITECTURES ON CIFAR-100
In the main text, we presented the result on CIFAR-10 dataset. To generalize the observations, we repeated the experiments on CIFAR-I00 dataset to validate our hypothesis of the convergence of the angles between model checkpoints and well-trained model P-vectors. We investigate the change of angles over the P-vectors of training model checkpoints per epoch with comparison the P. vectors of well-trained models (model of epoch 200 in our case). As shown in Fig gradually decreasing manner of the curves for the angle between P-vectors and angles between P-vectors cross models with different supervisory manners generally converge to value that smaller than 10 degree. We can conclude that the hypothesis of the existence of common subspace during the learning procedure also stands on the experiments with CIFAR- 100 dataset.
(a) Supervised (Train)
(b) Supervised (Test)
(c) Unsupervised (Train)
160
(d) Unsupervised (Test)
(e) Self-Sup. (Train)
(1) Self-Sup. (Test)
Figure 10: Convergence to the Common Feature Subspace with CIFAR-100. Curves of angles of P-vectors between the well-trained model and its checkpoint per training epoch of three learning supervisory manners. The trends of convergence for the angles can be observed in all models.
A.2 THE MON-MONOTONIC TREND WITHIN THE FIRST EPOCH
The variation of angles between P-vectors for the well-trained model and its checkpoint per training epoch of each iteration in the first epoch. The non-monotonic trends within the first epoch also incorporate in the experiments on CIFAR-101 datasets. Fig [T] shows the curves indicating the variation of angles between the training model P--ectors and the well-trained model P-vectors in the iterations in the first epoch. As we use 128 as the batch size in training procedure, the number of iterations for updates is 391 per epoch. We obtain the observation of non-monotonic trend that the angle first rises with the random initialization and drop down. And the rest of training process, the angles keeps the approximately monotonically decreasing and converging to small values. The experiments shows consistent result and conclusion on CIFAR-100 dataset with the discussion in section 4.
A.3 MODEL-TO-MODEL COMMON SUBSPACE
We also test and verify the model- poddel-to-mont co-model common subspace shared by models trained with dif- ferent supervisory manners 1IFAR-100 dataset. Experiments carried out evaluate the angles between P--vectors for checkpoints of all models and P-vectors for well-trained supervised, unsu- pervised and self-supervised models, where we use the well-trained Wide-ResNer2/Connoohn
12

---

Under review as a conference paper at ICLR 2021
(a) Supervised(Train)
(b) Unsupervisse(Trai))
(c) Self-Sup( Train).
Supervised(Test).
Unsupervised(Test
(f) Self-Sup(Test).
Figure 11: Angles between the P-vectors of the training and well-trained models over the number of iterations in the first epoch using CIFAR-100.
SINGO
40002
4000
(a) Sup. vs. Sup.
(b) Sup. vs. Unsup
(c) Sup. vs. Self-Sup.
(d) Unsup. vs. Sup.
(e) Unsup. vs. Unsup
Unsup Self-Sup
Self. Sup.
(h) Self. Unsup
(i) Self. vs Self-Sup.
Figure 12: Convergence of P-vector angles between checkpoint per epoch and well-trained models using CIFAR-100.
13

---

Under review as a conference paper at ICLR 2021
Auto-encoder and SimCLR model (trained with 200 epochs under suggest settings) the reference of supervised, unsupervised and self--uuprvvsed models, respectively. As shown in Fig 12 a con- sistent convergence for the curves of the angles can be observed and support our hypothesis that the dynamics learning procedure construct the common subspace gradually.
A.4 THE NON-MONOTONIC TREND IN THE FIRST EPOCI OF COMPARISON OF ANGLES BETWEEN MODEL AND RAW DATA P-VECTORS
We also explore the construction procedure for the common subspace share between feature vectors and the raw data during the training process. Experiments are carried out to compare the space of models and raw data P-vectors on the training dataset. As shown in Fig[33 we observe non- monotonic trend that the angle first rises with the random initialization and drop down. The an- gles keeps the approximately monotonically decreasing and converging to small values in following training epochs. The experiments shows consistent result on both CIFAR-10 10 and 100 dataset. Note that we follow the default random data augmentation policy to pre-process the training dataset.
Broke-t3 CSA
Bockects
150
140
100
150
(a) Supervised C10
(b) Unsupervised C10
(c) Sclf-Sup C10.
125
22.0
eensone
(d) Supervised C100
(e) Unsupervised C100
Self-Sup C100.
Figure 13: Angles between the P-vectors of the training and well-trained models over the number of iterations in the first epoch using CIFAR-10 /CIFAR-100.
A.5 CASE STUDIES ON THE ANGLE VARIETY BETWEEN MODEL AND RAW DATA FOR EACH LAYER
To explore the dynamic variation process by zooming in every layer variation in each epoch, perform the case study using Resnet-18 structure on CIFAR-10 dataset. As shown in Fig 14] we give the angles according layers of different epochs, where the x-axis indicating the indices of residual blocks in the network structure and y-axis refers to the angles between the model checkpoint and raw data P-vectors. We observed that early training stage, the angles between the model checkpoint and raw data P--vectors keeps an increasing manner when the features passing though layers and turn into a decrease trend towards the stacked layers in the late training stage. This set of experiments further support hypothesis that the dynamics learning procedure construct the common subspace gradually through training process.
A.6 DISTRIBUTION OF VALUES IN THE P-VECTOR
Please refer to Figure D for the results of experiments carried out on CIFAR-10 and CIFAR-100 datasets using ResNet-50.
14

---

Under review as a conference paper at ICLR 2021
50.00
40.75
49.75
50.5
51.5
49.50
49.50
50.0
49.25
50.5
49.00
50.0
48.75
48.75
49.0
49.0
48.50
48.50
48.50
48:25
48.5
48.25
48.25
48.0
48.00
48.0
48.00
layar
layer
Tayer
layer
layar
(a) Epoch 5.
(b) Epoch45
(c) Epoch85
(d) E2och125.
(e) Epoch165.
Figure 14: Angles changes through layer between the P-vectors of the training and raw data over the number of iterations the first epoch using CIFAR-10.
C10 Epoch 0
(b) C10 Epoch 60
C10 Epach120
C10 Epoch 160
(e) C10 Epoch 200 teo-de
Beck))
C100 Epoch
(g) C100 Epoch 60
(h) C100 Epoch120
(i) C100 Epoch 160 (j) C100 Epoch 200
Figure 15: Frequency distributions of the P-vector values over training epochs.
15

---

Under review as a conference paper at ICLR 2021
Tops Lth drgulrr Nette
Tce Vetter
6:35
2554
033
03:5
0121
8pa0k
(a) Top singular vector.
(b) Top singular vector.
(c) Top singular vector.
lumpide Vaia
ossse
80.30
(d) Top singular vector.
(c) Top singular vector.
(f) Top 6 singular vector.
Figure 16: Angles between the top-k left singular vector (k 1 is the P-vector) of the training and well-trained models over the number epochs training process (Resnet-50, CIFAR-1O), Note the first plot point refers to the feature matrix after trained for one epoch.
VGS.16 CIFAR110
VOG-16 OIFAR.10
ReaNet-20 CIFAR-10
ResMett20
Chmrrean' cwe.teett 4.30
100
140
140
cnd deeert 429
Trush acc logt
That 6cc
Thank adc Jog)
Tex acc
(a) VN16-C10 (Train)
(b) VN16-C10 (Test)
(c) RN20-C10 (Train)
(d) RN20-C10 (Test)
ResNet55 CIFAR-10
RezNet55 CIFAR-10
180 Spearor's pedlls 2032
(c) RN56-C10 (Train)
(f) RN56-CI0 (Test)
Strong and consistent correlations between the model performance (train- ing and testing accu racy in log range) and the angels (in log range) between model and data P. vectors using CIFAR-10 datasets. VN: VggNet, RN: ResNet, C10: CIFAR-I0.
Figure 17: Log-Log Plots:
A.7 No CONVERGENCE FOUND IN COMPARISONS BETWEEN THE TOP SINGULAR VECTORS OTHER THAN THE P-VECTORS
Please refer to Figure 16 for the results of experiments carried out on CIFAR-10 datasets using ResNet-50.
A.8 LOG-LOG PLOTS THAT CORRELATE THE P-VECTOR ANGLES AND THE MODEL PERFORMANCE
Please refer to Figure 17 for the log-log plot of the results based on on CIFAR-10 datasets.
16


Reviews:
[{"review_id": "lRdMrcaXwmR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the top singular vector of the feature space learned by supervised and unsupervised deep learning models on CIFAR datasets. The hypothesis of converging feature spaces is interesting (converging both in terms of different models, and in terms of training epochs), but the conclusion from the current experiment results is overstretching.\n\n1. While the authors emphasize the convergence of subspaces, the P-vector defined in the paper is actually the top singular vector of the feature space, so it's actually about the convergence of the 1-dimensional principal subspace. A subspace refers to an arbitrary dimensional space in general. In the context of SVD, the literature often studies the top-$k$ dimensional subspace, which is represented by the $k$ top singular vectors, and the approximation error of the top-$k$ dimensional subspace: $E=\\|X - U_k \\Sigma_k V_k^T\\|_F^2$, where $X$ would be the feature matrix in this paper, and $U_k, V_k$ are the first $k$ columns in the result of SVD. The authors didn't measure $E$, so the readers won't know how well the top-1 dimensional subspace represents the feature matrix. I recommend looking at $E$ as a function of $k$, and use some criteria to determine how closely you want the subspace to approximate the feature matrix. For example, we can say we want to keep the top-$k$ dimensional subspace such that $E < 0.1 \\|X\\|_F^2$. This way, you can rule out the possibility that the P-vector is a trivial vector that every model will converge to.\n(As an analogy for a trivial vector, we can consider the top-1 eigenvector of the similarity matrix defined in the classical spectral clustering method called Normalized Cut. No matter how the edge weights in a graph is defined, the similarity matrix used in Normalized Cut always has an all-one vector as the top-1 eigenvector.)\nAnd to measure the angle between general subspaces, many methods are available including classical ones (e.g. Åke Björck and Gene H. Golub, Numerical Methods for Computing Angles Between Linear Subspaces, 1973).\n\n2. This paper tries to emphasize the P-vectors found in the features from different deep learning models are very close (for example, \"no matter what type of DNN architectures or whether the labels have been used to train the models, the P-vectors of different models would converge to the same one\"). Actually it seems the angle typically converges to 10 to 20 degrees. It may be better to lower the tone, or quantify better (compared to the angles obtained by ..., the angles between P-vectors are smaller).\n\n3. The data in Fig. 7 looks quite noisy, though p-value shows statistical significance of the correlation. p-value can guide our findings but is not always meaningful. For example, comparing Fig.7(e) and Fig.7(l), we may argue the latter has a better correlation but the former has a much smaller p-value. It seems the very small p-value in Fig.7(e) results from some outliers. Intuitively I don't quite understand why the raw data and the features should have a correlated linear principal subspace, given that the neural network layers that generate the feature from the data are highly nonlinear.\n\nThe only convincing data I found is in Table 1, which shows P-vectors can serve as an indicator of the model performance. But overall the readers would need more evidence as explained in #1 above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3qH-tbIEh4N", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors identify an interesting empirical phenomenon: across a range of network architectures and training approaches (supervised, unsupervised, auto-encoders), the feature spaces identified by these networks are similar. The authors introduce a specific way to summarize the feature space of a network as a vector (the top-left singular vector of the num_examples x num_features matrix) and show that these vectors are highly correlated across networks. In addition, the authors show that the features spaces become more similar throughout training and are predictive of the generalization performance of a neural network.\n\nThe paper presents purely experimental findings, but the experiments are sufficiently broad (e.g., covering different training approaches and datasets) so that this is not a shortcoming. Investigating potential theoretical models or more models and datasets could be fruitful directions for future work.\n\nAspects of the presentation in the paper could be improved (see the comments below). Overall I still recommend accepting the paper.\n\n\nAdditional comments:\n\n- Many plots have labels that are too small to read. I strongly encourage the authors to produce more readable plots.\n\n- Have the authors explored visualizations of the P-vectors? For instance, what is the ordering of training examples induced by the P-vectors?\n\n- What is the \"SupCon\" method? Do the authors have a hypothesis for why it behaves different from the other methods w.r.t. P-vector angles?\n\n- Have the authors experimented with training approaches that explicitly encourage small angles between model and data P-vectors?\n\n- The paper would benefit from a thorough editing pass to fix typos and improve clarity. The structure of the paper is well-organized, just some sentences are hard to parse.\n\n- When abbreviations like \"AE\" or \"CNN\" are used for the first time, it is generally good to write them out.\n\n- Why does the paper sometimes use angle and sometimes use cosine of the angle? It could be better to use one of the two consistently.\n\n- Should the x-axis labels in Figure 5 be \"training steps\" instead of \"epochs\"?\n\n- There is too little vertical space separating the caption of Figure 7 from the text below.\n\n- Typos:\n  * Introduction: \"To better euclid\"\n  * Section 3: \"various architectures amd different training paradigms\"\n  * Section 5: \"an data, We carry out\"  (capitalization)\n  * Section 5: \"expect\" -> \"except\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "-aZ-tFnQdRp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper has a closer look at the distributions of samples in the feature space by utilizing P-vector to analyze principal subspace. According to their empirical studies, the authors concluded that the feature spaces learned by different deep models with the same dataset would share common principal subspaces for the same dataset. It will not be affected by DNN architectures or the usage of labels in feature learning. Only the training procedure gradually shapes the feature subspace to the shared common subspace. \n\n-----------------------------------------------------------------------\n\nReasons for score: \n \nThe paper explores a new question and gives some interesting conclusions. But my major concern is its empirical studies cannot support the findings well. Besides, there are few discussions to provide the readers with some insights. I hope the authors carefully consider how to enhance this paper and make the conclusions more convincing.\n \n-----------------------------------------------------------------------\n\nPros: \n \n1. The paper explores a new question and gives some interesting conclusions.\n \n2. The proposed metric is simple and easy to follow. The authors also attached the source code for reference.\n\n3. The usage of the P-vector for predicting generalization achieves promising results.\n\n-----------------------------------------------------------------------\n\nCons: \n \n1. Why can the similarity of P-vectors be used to indicate the similarity of two distributions in the feature space?\n2. I would like to know how many trials it takes to plot similarity figures (e.g., Figure 1 (a)-(b)). It would be better to try many times and give the mean and variance to avoid coincidence. Besides, will other parameters such as the number of samples and dimensions of features affect Hypothesis I?\n3. The authors mentioned that the reference model used in Figure 4 (a)-(c) is Wide-ResNet28 trained with 200 epochs under suggest settings. But, the plot of Wide-ResNet28 in Figure 4 (a) is weird. It cannot converge to zero. \n4. I would like to know why most models (such as Figure 4) cannot converge to zero after about 200 epochs training, and the angles are approximately 10 degrees. In other words, is there exists a threshold after which we can think the compared two models have a common subspace? \n5. Why the P-vector can be used to predict the generalization?\n\n-----------------------------------------------------------------------\n\nQuestions during the rebuttal period: \n \nPlease address and clarify the cons above. \n \n-----------------------------------------------------------------------\n\nSome typos: \n(1) Figure 5, Figure 9, the xaxis's titile should be iterations rather than epochs.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an intriguing empirical phenomenon in deep learning. They train a variety of architectures for different tasks using different datasets and study the relationship between the learned representations. In particular they collect the representations into a large matrix and take the top left singular vector and measure the cosine of the angle. They show that it is much smaller than one might expect, about 10 degrees or so, and has an approximate monotonicity property as the network is being trained although it does not seem to converge to zero. Moreover this measure also correlates with performance. 

The reviewers had divided opinions on this paper. On the one hand, the range of experiments is impressive and truly demonstrates that this is a pervasive phenomenon. On the other hand, it is not so clear what it means. In particular, suppose we have a collection of graphs which have close to the same degree distributions. If we take the top left singular vectors of all the adjacency matrices, they would also have low angles between them. While this is a very different setting and there is no analogy between the experiments in this paper and this toy model, it does raise philosophical questions about whether the phenomenon is meaningful or is a byproduct of something else about the data. This may be a challenging question to answer, but one reviewer brought up a natural next step: One could measure the principal angle between the subspace of the top k left singular vectors across experiments for larger values of k. The authors do bring up the point that the spectrum decays very quickly, so it could be that beyond a certain point the singular vectors behave somewhat randomly. 
Decision: Reject