Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors identify an interesting empirical phenomenon: across a range of network architectures and training approaches (supervised, unsupervised, auto-encoders), the feature spaces identified by these networks are similar. The authors introduce a specific way to summarize the feature space of a network as a vector (the top-left singular vector of the num_examples x num_features matrix) and show that these vectors are highly correlated across networks. In addition, the authors show that the features spaces become more similar throughout training and are predictive of the generalization performance of a neural network.

The paper presents purely experimental findings, but the experiments are sufficiently broad (e.g., covering different training approaches and datasets) so that this is not a shortcoming. Investigating potential theoretical models or more models and datasets could be fruitful directions for future work.

Aspects of the presentation in the paper could be improved (see the comments below). Overall I still recommend accepting the paper.


Additional comments:

- Many plots have labels that are too small to read. I strongly encourage the authors to produce more readable plots.

- Have the authors explored visualizations of the P-vectors? For instance, what is the ordering of training examples induced by the P-vectors?

- What is the "SupCon" method? Do the authors have a hypothesis for why it behaves different from the other methods w.r.t. P-vector angles?

- Have the authors experimented with training approaches that explicitly encourage small angles between model and data P-vectors?

- The paper would benefit from a thorough editing pass to fix typos and improve clarity. The structure of the paper is well-organized, just some sentences are hard to parse.

- When abbreviations like "AE" or "CNN" are used for the first time, it is generally good to write them out.

- Why does the paper sometimes use angle and sometimes use cosine of the angle? It could be better to use one of the two consistently.

- Should the x-axis labels in Figure 5 be "training steps" instead of "epochs"?

- There is too little vertical space separating the caption of Figure 7 from the text below.

- Typos:
  * Introduction: "To better euclid"
  * Section 3: "various architectures amd different training paradigms"
  * Section 5: "an data, We carry out"  (capitalization)
  * Section 5: "expect" -> "except"
review_title: Interesting experiments into neural networks behaving similarly

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper has a closer look at the distributions of samples in the feature space by utilizing P-vector to analyze principal subspace. According to their empirical studies, the authors concluded that the feature spaces learned by different deep models with the same dataset would share common principal subspaces for the same dataset. It will not be affected by DNN architectures or the usage of labels in feature learning. Only the training procedure gradually shapes the feature subspace to the shared common subspace. 

-----------------------------------------------------------------------

Reasons for score: 
 
The paper explores a new question and gives some interesting conclusions. But my major concern is its empirical studies cannot support the findings well. Besides, there are few discussions to provide the readers with some insights. I hope the authors carefully consider how to enhance this paper and make the conclusions more convincing.
 
-----------------------------------------------------------------------

Pros: 
 
1. The paper explores a new question and gives some interesting conclusions.
 
2. The proposed metric is simple and easy to follow. The authors also attached the source code for reference.

3. The usage of the P-vector for predicting generalization achieves promising results.

-----------------------------------------------------------------------

Cons: 
 
1. Why can the similarity of P-vectors be used to indicate the similarity of two distributions in the feature space?
2. I would like to know how many trials it takes to plot similarity figures (e.g., Figure 1 (a)-(b)). It would be better to try many times and give the mean and variance to avoid coincidence. Besides, will other parameters such as the number of samples and dimensions of features affect Hypothesis I?
3. The authors mentioned that the reference model used in Figure 4 (a)-(c) is Wide-ResNet28 trained with 200 epochs under suggest settings. But, the plot of Wide-ResNet28 in Figure 4 (a) is weird. It cannot converge to zero. 
4. I would like to know why most models (such as Figure 4) cannot converge to zero after about 200 epochs training, and the angles are approximately 10 degrees. In other words, is there exists a threshold after which we can think the compared two models have a common subspace? 
5. Why the P-vector can be used to predict the generalization?

-----------------------------------------------------------------------

Questions during the rebuttal period: 
 
Please address and clarify the cons above. 
 
-----------------------------------------------------------------------

Some typos: 
(1) Figure 5, Figure 9, the xaxis's titile should be iterations rather than epochs.
review_title: Marginally below acceptance threshold

final_score: rating: 3: Clear rejection
review_content: This paper studies the top singular vector of the feature space learned by supervised and unsupervised deep learning models on CIFAR datasets. The hypothesis of converging feature spaces is interesting (converging both in terms of different models, and in terms of training epochs), but the conclusion from the current experiment results is overstretching.

1. While the authors emphasize the convergence of subspaces, the P-vector defined in the paper is actually the top singular vector of the feature space, so it's actually about the convergence of the 1-dimensional principal subspace. A subspace refers to an arbitrary dimensional space in general. In the context of SVD, the literature often studies the top-$k$ dimensional subspace, which is represented by the $k$ top singular vectors, and the approximation error of the top-$k$ dimensional subspace: $E=\|X - U_k \Sigma_k V_k^T\|_F^2$, where $X$ would be the feature matrix in this paper, and $U_k, V_k$ are the first $k$ columns in the result of SVD. The authors didn't measure $E$, so the readers won't know how well the top-1 dimensional subspace represents the feature matrix. I recommend looking at $E$ as a function of $k$, and use some criteria to determine how closely you want the subspace to approximate the feature matrix. For example, we can say we want to keep the top-$k$ dimensional subspace such that $E < 0.1 \|X\|_F^2$. This way, you can rule out the possibility that the P-vector is a trivial vector that every model will converge to.
(As an analogy for a trivial vector, we can consider the top-1 eigenvector of the similarity matrix defined in the classical spectral clustering method called Normalized Cut. No matter how the edge weights in a graph is defined, the similarity matrix used in Normalized Cut always has an all-one vector as the top-1 eigenvector.)
And to measure the angle between general subspaces, many methods are available including classical ones (e.g. Åke Björck and Gene H. Golub, Numerical Methods for Computing Angles Between Linear Subspaces, 1973).

2. This paper tries to emphasize the P-vectors found in the features from different deep learning models are very close (for example, "no matter what type of DNN architectures or whether the labels have been used to train the models, the P-vectors of different models would converge to the same one"). Actually it seems the angle typically converges to 10 to 20 degrees. It may be better to lower the tone, or quantify better (compared to the angles obtained by ..., the angles between P-vectors are smaller).

3. The data in Fig. 7 looks quite noisy, though p-value shows statistical significance of the correlation. p-value can guide our findings but is not always meaningful. For example, comparing Fig.7(e) and Fig.7(l), we may argue the latter has a better correlation but the former has a much smaller p-value. It seems the very small p-value in Fig.7(e) results from some outliers. Intuitively I don't quite understand why the raw data and the features should have a correlated linear principal subspace, given that the neural network layers that generate the feature from the data are highly nonlinear.

The only convincing data I found is in Table 1, which shows P-vectors can serve as an indicator of the model performance. But overall the readers would need more evidence as explained in #1 above.
review_title: Interesting hypothesis but results are not convincing enough

Meta-review: This paper presents an intriguing empirical phenomenon in deep learning. They train a variety of architectures for different tasks using different datasets and study the relationship between the learned representations. In particular they collect the representations into a large matrix and take the top left singular vector and measure the cosine of the angle. They show that it is much smaller than one might expect, about 10 degrees or so, and has an approximate monotonicity property as the network is being trained although it does not seem to converge to zero. Moreover this measure also correlates with performance. 

The reviewers had divided opinions on this paper. On the one hand, the range of experiments is impressive and truly demonstrates that this is a pervasive phenomenon. On the other hand, it is not so clear what it means. In particular, suppose we have a collection of graphs which have close to the same degree distributions. If we take the top left singular vectors of all the adjacency matrices, they would also have low angles between them. While this is a very different setting and there is no analogy between the experiments in this paper and this toy model, it does raise philosophical questions about whether the phenomenon is meaningful or is a byproduct of something else about the data. This may be a challenging question to answer, but one reviewer brought up a natural next step: One could measure the principal angle between the subspace of the top k left singular vectors across experiments for larger values of k. The authors do bring up the point that the spectrum decays very quickly, so it could be that beyond a certain point the singular vectors behave somewhat randomly. 
Decision: Reject