Dominant Datapoints and the Block Structure Phenomenon in Neural Network Hidden Representations

Abstract: Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations. This block structure has two seemingly contradictory properties: on the one hand, its constituent layers have highly similar dominant first principal components (PCs), but on the other hand, their representations, and their common first PC, are highly dissimilar across different random seeds. Our work seeks to reconcile these discrepant properties by investigating the origin of the block structure in relation to the data and training methods. By analyzing properties of the dominant PCs, we find that the block structure arises from dominant datapoints — a small group of examples that share similar image statistics (e.g. background color). However, the set of dominant datapoints, and the precise shared image statistic, can vary across random seeds. Thus, the block structure reflects meaningful dataset statistics, but is simultaneously unique to each model. Through studying hidden layer activations and creating synthetic datapoints, we demonstrate that these simple image statistics dominate the representational geometry of the layers inside the block structure. We also explore how the phenomenon evolves through training, finding that the block structure takes shape early in training, but the underlying representations and the corresponding dominant datapoints continue to change substantially. Finally, we study the interplay between the block structure and different training mechanisms, introducing a targeted intervention to eliminate the block structure, as well as examining the effects of pretraining and Shake-Shake regularization.     

Reviews and author responses:

 We thank the reviewer for their comments. We agree that the relationship between the block structure and dominant datapoints is straightforward. To restate what is shown in the paper: Across many different layers, activations of dominant datapoints point in similar directions and have large norms. Kernels that are sensitive to either or both of these properties of dominant datapoints will find high similarity between layers that share dominant datapoints, and similarity will be reduced when the dominant datapoints are removed. So far, every kernel we have tried is at least somewhat sensitive to these properties. But again, our goal in this work is simply to explain the results with the linear kernel in Nguyen et al. (2021).

If we understand correctly, after our initial response, the reviewer is now convinced that the connection [block structure] -- [dominant datapoints] is robust. We are happy to add additional results regarding the sensitivity of different kernels to simulated dominant datapoints in Appendix J. We hope this would satisfy the reviewer. If not, it would help us immensely if the reviewer could clarify what their remaining concerns are, with regards to the significance of our findings.

 I would like the authors for the detailed feedback as well as additional results in Appendix.

I was also curious how the outliers affect CKA with cosine kernel.
I therefore conducted a synthetic experiment by myself (see code below), and found that the removal of outliers can reduce CKA with cosine kernel as well.
That is, CKA with cosine kernel can be also vulnerable to the existence of outliers as well.
If this is the case, even for other kernels, the connection [elimination of block structure] -- [dominant datapoints] can be perhaps explained by the simple fact that "dominant datapoints are kept being outliers across layers".

|    |   remove |   CKA-linear |   CKA-cos |
|---:|---------:|-------------:|----------:|
|  0 |       20 |     0.999889 |  0.834033 |
|  1 |       40 |     0.999855 |  0.832287 |
|  2 |       60 |     0.999786 |  0.830499 |
|  3 |       80 |     0.999578 |  0.82867  |
|  4 |      100 |     0.915296 |  0.826801 |
|  5 |      120 |     0.91276  |  0.82521  |
|  6 |      140 |     0.911354 |  0.823581 |
|  7 |      160 |     0.909763 |  0.821929 |
|  8 |      180 |     0.908688 |  0.820247 |
|  9 |      200 |     0.907845 |  0.818594 |
| 10 |      220 |     0.90724  |  0.81698  |
| 11 |      240 |     0.907075 |  0.815416 |
| 12 |      260 |     0.906788 |  0.813852 |
| 13 |      280 |     0.906135 |  0.812017 |
| 14 |      300 |     0.904944 |  0.810073 |
| 15 |      320 |     0.90408  |  0.808198 |
| 16 |      340 |     0.903655 |  0.806376 |
| 17 |      360 |     0.902987 |  0.804522 |
| 18 |      380 |     0.902607 |  0.802229 |
| 19 |      400 |     0.901987 |  0.80034  |
| 20 |      420 |     0.901795 |  0.798269 |
| 21 |      440 |     0.900919 |  0.79594  |
| 22 |      460 |     0.900212 |  0.793388 |
| 23 |      480 |     0.900124 |  0.791651 |
| 24 |      500 |     0.90013  |  0.789716 |

```
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

def pca(x):
    S = np.cov(x.T)
    D, U = np.linalg.eigh(S)
    idx = np.argsort(D)[::-1]
    return U[:, idx[0]]

def CKAlinear(x, y):
    Sxy = np.cov(x.T, y.T)[2:, :-2]
    Sxx = np.cov(x.T)
    Syy = np.cov(y.T)
    K = np.sum(Sxy**2)
    K = K / np.sqrt(np.sum(Sxx**2) * np.sum(Syy**2))
    return K

def CKAcos(x, y, eps=1e-8):
    x = x / (eps + np.sqrt(np.sum(x**2, axis=1))[:, np.newaxis])
    y = y / (eps + np.sqrt(np.sum(y**2, axis=1))[:, np.newaxis])
    return CKAlinear(x, y)

def gen_data(num_inlier=100, num_outlier=10, seed=0):
    np.random.seed(seed)
    x = 10 * np.random.randn(num_inlier, 2) # inliers
    x = np.concatenate([x, np.random.randn(num_outlier, 2) + 1000], axis=0) # outliers
    return x

# feature in the first layer
x = gen_data(num_inlier=1900, num_outlier=100, seed=0) # 5% outliers

# feature in the second layer
np.random.seed(1)
y = x + 3.0 * np.random.randn(*x.shape)

# remove outliers and measure kca
u = pca(y)
y_pca = np.abs((y - np.mean(y, axis=0)).dot(u))
cka = []
for num_remove in np.linspace(20, 500, 25).astype(int):
    idx = np.argsort(y_pca)[::-1][num_remove:]
    x_rm, y_rm = x[idx], y[idx]
    c1 = CKAlinear(x_rm, y_rm)
    c2 = CKAcos(x_rm, y_rm)
    cka.append((num_remove, c1, c2))
cka = np.array(cka)
plt.plot(cka[:, 0], cka[:, 1], 'bo-')
plt.plot(cka[:, 0], cka[:, 2], 'ro-')
plt.legend(['linear', 'cos'])
plt.xlabel('Number of Removed Data Points')
plt.ylabel('KCA')
plt.show()

df = pd.DataFrame(cka)
df.columns = ['remove', 'CKA-linear', 'CKA-cos']
print(df.to_markdown())
```

 Thank you for the elaboration!

**It appears that the trained networks seem to have similar classification performances, and getting rid of the block structures does not seem to lead to significant performance boosts**
First, the main focus of our paper is to resolve the discrepancies/missing pieces from previous work (Nguyen et al. 2021 [1]) with regards to the properties of the hidden representations of overparameterized models, and not to try to improve performance on any task. Given that the block structure phenomenon uncovered by [1] is prevalent in all overparameterized networks, our work seeks to rigorously characterize its cause and effects, and consequently provide a better understanding of the nuances in the way these networks learn, despite their similarly good performances.

Second, the motivation of the experiments with principle component regularization is not to boost performance. We simply believe it is surprising that even though a considerable set of layers in large-capacity networks exhibit highly similar representations, which may indicate the presence of redundant modules (see [1]), the block structure is a phenomenon that emerges naturally with standard training. As a result, we explore the implications of several training interventions, and demonstrate (in section 6) that it is possible to obtain good generalization accuracy in networks with and without the block structure. 

**Is the block structure phenomenon only relevant for relatively small training data?**
The block structure phenomenon is found in neural networks that have large capacity *relative to* the size of the training data, as demonstrated in prior work [1]. So it’s possible that as model size continues to increase over the years, it becomes more common to observe the block structure with large training datasets. In addition, as mentioned in our “Scope and Limitations” section, our experiment design is motivated by domains such as medical imaging where data is very difficult to collect and the high-stakes nature makes it important to understand the model’s internals. In this case, even with access to large, widely available natural image datasets, pretraining only yields limited benefits for medical imaging (see [2]). Although our primary focus is on vision tasks, many applications of machine learning beyond vision (such as those on Kaggle) involve applications/learning problems with small dataset sizes as well. 

References 

[1] Nguyen, T., et al. "Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth." International Conference on Learning Representations (2021).

[2] Raghu, Maithra, et al. "Transfusion: Understanding transfer learning for medical imaging." NeurIPS (2019).


 Thank you for the feedback! We appreciate your recognition of the strengths of the paper. 

**The paper lacks a rigorous theoretical analysis**
The paper is intended to be an empirical study of phenomena concerning the representational structures of large-capacity neural networks. We would be very interested in a theoretical explanation for this phenomenon, and we hope our empirical findings will assist in developing one.

**The idea of influential points is not very novel**
We would like to clarify the distinction between our “dominant datapoint” concept and influential points/support vectors/coreset/etc. that have been widely studied in existing literature. We define a dominant datapoint to be a datapoint that yields a large value when projected onto the first principal component of the layer activations, and that dominates the representational geometry of large-capacity neural networks in general. To our knowledge, no previous work has shown the existence of such points in neural network *representation spaces*. In our work, we also perform extensive analysis to study the implications of dominant datapoints on other properties of the internal representations, such as their high activation norms. In contrast, existing studies that uncover influential points focus on the effects of different training examples on the *output predictions* or the *decision boundary* of a classifier instead.

We hope our response has clarified the main concerns that you have, and ask if you might consider increasing your score to support the acceptance of the paper.

 Thank you for all the feedback! Please find the corresponding response to each point below:

**On the issue of surprise and arising of the block structure:**
1. It’s not surprising that subsequent layers have similar representations 
We respectfully disagree with this point. Successive layers are indeed expected to progressively refine the network’s representations towards something that could predict the output well. However, increasing network depth alone does not imply the existence of the block structure - a large set of contiguous layers with highly similar representations as well as dominant and similar first principal components. Furthermore, this phenomenon also emerges in the case of increasing network width, i.e. given the same depth, narrower models do not exhibit a block structure but wider models do. 
In general, we believe the nontriviality and significance of the block structure phenomenon itself is already sufficiently justified in previous work [1] and our work only seeks to understand its properties and implications.

2. **Try to vary the residual connections** 
In Appendix C of previous work [1], the authors removed the residual connections from ResNets while keeping the constituent layers the same, simulating a standard CNN architecture. They found that the block structure still emerged when the width was increased. Thus, we believe that this phenomenon does not depend on the presence of the residual connections.

3. **Can the block structure be seen from an information bottleneck viewpoint?**
This is a difficult question to answer in part because the claims regarding information bottlenecks in deep neural networks made by Shwartz-Ziv et al. [2] have been challenged in later work [3,4]. Like Goldfeld et al. [4], we believe that deterministic neural networks are likely to be injective or nearly so, and thus there is likely no true information bottleneck whether or not a block structure is present in representations. The block structure and dominant datapoint phenomena may nonetheless affect some estimators of mutual information, but characterization of those effects is beyond the scope of our present work.

**On the implications of block structure:**
1. **The authors don't address in their paper why this block structure "is bad"** 
Our paper makes no claim that the block structure “is bad”. We simply believe it is surprising that even though a considerable set of layers exhibit highly similar representations, which may indicate the presence of redundant modules in large-capacity networks (see [1]), the block structure is a phenomenon that emerges naturally with standard training. Consequently, we experiment with some training interventions, and demonstrate (in section 6) that it is possible to obtain good generalization accuracy in networks with and without block structure. 

2. **The authors have no results showing that it affects results or generalization**  
The goal of the paper is to reconcile seemingly contradictory properties of the block structure uncovered by previous work [1], and provide an answer to the fundamental question of whether the block structure picks up noisy artifacts in the data, or whether it actually captures relevant dataset characteristics. The block structure/dominant datapoint phenomenon that we explore explains most of the variance in the hidden representations of many layers in high-capacity networks, but it is not present at all in low-capacity networks. Thus, we are not studying an arbitrary phenomenon; it is the single largest representational change associated with overparameterization. We accept the reviewer’s criticism that our results relate to generalization only indirectly through overparameterization. However, we also believe that understanding the effects of overparameterization on neural network representations is important, and is likely a prerequisite for connecting representational properties of neural networks to generalization.  

3. **The paper has no experiments showing the ability to prune or reduce networks when they eliminate block structure** 
This experiment can be found in Section 5.2 of previous work [1]. The authors there already showed that some hidden layers exhibiting the block structure can indeed be pruned with minimal impact on performance. Our work instead focuses on the implications of different training mechanisms on the block structure representations.

**On the differences in block structure that arise:**
1. **There actually seems to be an order-dependence to the particular representation that arises** 
Even though the block structure and the dominant first PCs arise very early in training, the dominant datapoints can change over the course of training, as shown in Figures 8, F.1, and F.2, suggesting that it is not just the initial few minibatches that determine which datapoints become dominant.


 **On eliminating the block structure:**
1. **It seems that most attempts to eliminate block structure still leaves some of it intact** 
We respectfully disagree with this point. Figure 9 in our paper indicates that the principle component regularizer, which we specifically designed to eliminate the block structure, does actually get rid of it, as does Shake-Shake regularization. Transfer learning attenuates the block structure but does not eliminate it; how the interaction between the statistics of the pretraining dataset and the finetuning dataset affects the ability to remove the block structure is left to future work.

2. **Can it be removed by explicitly penalizing for between layers similarity? Would this affect generalization?** 
Penalizing the representational similarity between adjacent layers is potentially another way to eliminate the block structure. However, as we note above, a regularizer that considers only a single layer and regularizes the proportion of variance explained by its first principal component already suffices. We are willing to try the regularizer that the reviewer suggests if they strongly feels that those experiments should be run, but given that the PC regularizer already eliminates the block structure, we do not believe that our overall findings would change.


We have addressed most of the weaknesses/questions raised to the best of our ability, and ask if you might consider increasing your score to support the acceptance of the paper.

**References:**

[1] Nguyen, T., et al. "Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth." International Conference on Learning Representations (2021).

[2] Shwartz-Ziv, R., & Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810.

[3] Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2019). On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12), 124020.

[4] Goldfeld, Z., Van Den Berg, E., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., & Polyanskiy, Y. (2019, May). Estimating Information Flow in Deep Neural Networks. In International Conference on Machine Learning (pp. 2299-2308). PMLR.

 We would like to thank the reviewer for their time and feedback!

**“A neural network is rarely trained on small data sets”**
As mentioned in our “Scope and Limitations” section, our work is applicable to domains such as medical imaging where data is very difficult to collect and the high-stakes nature makes it important to understand the model’s internals. In addition, earlier work has shown that even with access to large, widely available natural image datasets, pretraining yields limited benefits for medical imaging (see [1]). Although our primary focus is on vision tasks, many applications of machine learning beyond vision (such as those on Kaggle) involve applications/learning problems with small dataset sizes. In general, understanding how the characteristics of neural network representations change with model capacity (relative to dataset size) is of both scientific and practical interest, as model size continues to increase over the years but there are many domains where dataset size does not.

**“The paper present too small a novelty”**
Our goal is not to develop a novel algorithm — we use CKA as an exploratory tool to study phenomena related to the internal representations of large-capacity networks. Besides showing the implications of different training mechanisms on the representational structures like the reviewer has mentioned, we have also characterized how the dominant first principal components (PCs) of the layer activations in very wide/deep networks arise from a small set of dominant datapoints, as well as how the first PCs and the corresponding dominant datapoints evolve over the course of training. We believe that these findings are empirically novel and help further understanding of the underlying mechanism behind this significant representational change associated with overparameterization. We are not aware of previous work that presents empirical findings similar to ours, but if the reviewer is aware of such work, we would greatly appreciate it if they could point us to it.

We hope our response has addressed the weaknesses raised, and ask if you might consider increasing your score to support the acceptance of the paper.

References 

[1] Raghu, Maithra, et al. "Transfusion: Understanding transfer learning for medical imaging." NeurIPS (2019).


 We would like to thank the reviewer for their time and feedback!

The main focus of our paper is to resolve the discrepancies/missing pieces from previous work (Nguyen et al. 2021 [1]). As the block structure phenomenon uncovered by [1] is prevalent in all overparameterized networks, rigorously characterizing its cause and effects is of great importance to understanding the nuances in the way these networks learn, despite their similarly good performances. It’s not our goal to explicitly improve performance on any task or propose a new algorithm. We understand that this makes our work appear less practical than other ML papers and makes impact harder to assess ahead of time, but curiosity-driven research is a fundamental part of science, and curiosity-driven research to understand neural network representations in particular has been historically important to the advancement of ML. A considerable portion of the Rumelhart et al. paper that popularized backprop was dedicated to analysis of properties of its hidden representations [2]. We believe that our findings on the properties of the hidden representations of overparameterized networks can potentially lead to improvements for techniques that make direct use of the internal representations, such as transfer learning and interpretability methods.

In light of our response, we ask if you might consider increasing your score to support the acceptance of the paper.


References

[1] Nguyen, T., et al. "Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth." International Conference on Learning Representations (2021).

[2] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by 
back-propagating errors. Nature, 323(6088), 533-536.


 We want to thank the reviewer for taking the time to read the paper and for their helpful feedback!
If we understand the reviewer's critique correctly, it is that the connection between the block structure and dominant datapoints may exist only when a linear kernel is used to measure similarity, and it is possible that a block structure exists when other kernels are used but it is not eliminated by eliminating dominant datapoints. In our updated manuscript, we have addressed this criticism in two ways.

First, at the reviewer's request, we now show the results of deleting dominant datapoints on CKA heatmaps computed with different kernel choices in Appendix Figure J.3. For the ResNet-38 10x model, removing the top 10-20% most dominant datapoints eliminates the block structure and also the differences among representational similarity heatmaps computed with different kernels. For the ResNet-164 1x model, which has fewer dominant datapoints, Appendix Figure J.1 in our original submission already showed that some kernels can reduce the appearance of the block structure, but our new results confirm that removing dominant datapoints eliminates the block structure regardless of the kernel. In other words, _across all choices of kernel we have tried_, when a CKA heatmap exhibits blocks resembling those observed by Nguyen et al. (2021), it can be eliminated by removing the dominant datapoints of the internal representations.

Second, we now explicitly describe the block structure as a phenomenon that is observed in linear CKA heatmaps. Our goal is to relate the dominant datapoint phenomenon to the phenomenon studied by Nguyen et al. (2021), and what they called the “block structure” is only demonstrated in linear CKA heatmaps. Our understanding is that the reviewer already accepts that there is a relationship between block structure in _linear_ CKA heatmaps and the presence of dominant datapoints, but to summarize our reasoning:
Eq. (2) implies that CKA will primarily reflect the first PC if it is dominant.
Nguyen et al. (2021) and Appendix B of our submission demonstrate that the first PC is dominant.
Figure 3 shows that there are a small set of dominant datapoints that have much larger projections on the first PC.
Figure 4 shows that removing these datapoints from the evaluation set eliminates the block structure in linear CKA heatmaps.

We hope these changes will convince the reviewer that there is indeed a robust relationship between “[block structure] -- [dominant datapoints]” that holds across different representational similarity metrics. Having spent several months immersed in this data, we are convinced that the phenomena we describe does in fact hold and we are happy to provide additional results if necessary, but we are also aware that we only have a few days left before we will no longer be able to update the paper. If the reviewer remains unconvinced, we hope they will get back to us as soon as possible.

 I would like to add the following points: 

(1) I think it is worthwhile to study the internal representations learned by deep neural networks, and how the internal representations are affected by influential data points as well as random seeds, and how they change over the training process. 

(2) On the other hand, it appears that the trained networks, regardless of whether they exhibit block structures, seem to have similar classification performances, and getting rid of the block structures via regularization or other methods does not seem to lead to significant performance boosts. 

(3) What if we consider the problem of pre-training, i.e., you train a deep network by a supervised learning task or a self-supervised learning task, and then use the layers of the trained network for downstream tasks. In that case, maybe getting more stable internal representations will be more important?

(4) Is the block structure phenomenon only relevant for relatively small training data? What about very big training data? 

In conclusion, I feel the paper is an insightful empirical study. However, its usefulness is not convincingly demonstrated. 


The study of (Nguyen et al., 2020) reported that wide and deep neural networks tend to have similar feature representations across layers (called "block structure").
(Nguyen et al., 2020) also revealed that the block structure is essentially induced by the first principal component (PC) of the representation in each layer.

This paper provides more detailed empirical analysis of the block structure.
To measure the similarity, the authors used KCA (with linear kernel) as the similarity measure, which was also used in (Nguyen et al., 2020).
This paper reported that
* The dominant datapoints, which lie in the direction of PC, share some common image patterns such as background colors.
* The dominant datapoints are different across different model initialization.
* Although the block structure emerges in the very first stage of the training, the dominant data points differ between the beginning and at the end of the training.
* The block structure can be suppressed by modifying training, e.g., by regularizing PCs, by using Shake-Shake regularization, and by using transfer learning. ## Strength

### [Strength1]
The authors conducted exhaustive experiments on standard datasets (CIFAR-10, CIFAR-10, Patch Camelyon) and on different model architectures (several levels of shallow/deep ResNets with narrow/wide layers).
According to (Nguyen et al., 2020), the block structure can emerge beyond ResNet as well (they conducted experiments on ResNet without skip connections).
These observations imply that the results reported in the paper will be ubiquitous (at least for images).
Revealing general phenomena underling deep models would be valuable to the community.

### [Strength2]
The observation that dominant datapoints tend to share some common image patterns seems to be nontrivial.
This phenomenon itself can be a topic of interest for further researches.
This phenomenon might also imply that deep models can be vulnerable to a slight change of the colors of the entire images.
That is, if we modify an input image to have a color close to that of dominant points, its intermediate representations might be large (e.g., Fig5), which will lead to different outputs (i.e., the model is vulnerable). 


## Weakness

In the paper, the authors reported that the block structure is induced by the dominant datapoints.
The authors then discussed the experimental results based on this connection.
The results are therefore connected to block structure through dominant datapoint, i.e., [block structure] -- [dominant datapoints] -- [results].
Because of this structure, there are two topics underlying the current paper: (i) the connection [block structure] -- [dominant datapoints] discussing the origin of the block structure, and (ii) the connection  [dominant datapoints] -- [results] discussing the emergence and dynamics of dominant datapoints.

My concern here is that these topics (i) and (ii) would be irrelevant (or, at least the connection would be far looser than expected).
The current paper claims that [block structure] and [results] are tightly connected through [dominant datapoints].
However, the validity of the connection (i) [block structure] -- [dominant datapoints] is questionable, as I describe below.
In particular, I suspect it can be an artifact induced by the use of KCA with linear kernel for the similarity measure.

### [Detail]
Essentially, KCA with linear kernel measures the squared Frobenius norm of the covariance matrix (with some normalization).
It is known that the covariance is vulnerable to the existence of outliers.
Indeed, as reported in (Nguyen et al., 2020, Fig3) as well as in Fig4 of the current paper, removing outliers (i.e., PCs or dominant datapoints) eliminate the similarity between layers.
These observations imply that "dominant datapoints are kept being outliers across layers ((ii) above)" rather than "the representations are similar across layers ((i) above)" because (at least some fraction of) the similarity is the consequence of the existence of such outliers.

In Appendix J, the authors reported the similarity under some other kernels.
There, we can find that linear kernel exhibits the strongest similarity (possibly because it is vulnerable to the existence of outliers).
This raises a question on the adequacy of the current study which rely solely on linear kernel.
The removal of outliers will lead to smaller KCA under linear kernel (and less blocks) as reported in the paper.
However, I wonder whether the similar can hold for KCA under some other kernels.
For example, we can reduce the effect of outliers with large activations for block structure by adopting the feature map $\phi(x) = \frac{x}{\\| x \\|}$ which induces the cosine kernel $k(x, y) = \cos(x, y)$.
Then, the question is whether the removal of dominant datapoints lead to reduction of KCA under cosine kernel (and some other kernels as well) and elimination of the block structure.
If the significant reduction of KCA is not observed in variety of kernels, the connection (i) [block structure] -- [dominant datapoints] would be an artifact induced by the use of linear kernel.

Here, I would like to remind that my comment above is all about the validity of the connection (i) [block structure] -- [dominant datapoints].
If the connection (i) still holds despite the selection of kernels, that would be far nontrivial, and would be an interesting phenomenon with further investigations needed.
I would also remark that the second connection (ii) [dominant datapoints] -- [results] can be of independent interest irrelevant to the validity of the connection (i). (unless there are related studies in this direction)

 The current paper mixes the two connections (i) [block structure] -- [dominant datapoints] and (ii) [dominant datapoints] -- [results].
To make the paper rigid, these two connections need to be distinguished.
The results in the current paper implies the connection (ii).
However, the validity of the connection (i) remains questionable.
I suspect it can be an artifact induced by the use of KCA with linear kernel for the similarity measure.
If this is the case, the paper needs to be reorganized with its main focus on (ii) but not (i).
If the authors are willing to connect (i) and (ii), it would be essential to confirm that the reported phenomena are not artifact induced by the use of KCA with linear kernel, and demonstrate they are ubiquitous irrelevant to the selection of similarity measures.

The authors investigate the block structure present in large deep networks and relate it to structure in the data and investigate trainings approaches that can reduce the presence of block structure.   The claims made by the authors in the paper seem correct and there is no obvious error I can see in the paper. The paper provides a number of experimental results that provide interesting insights into the appearance of block structures in large deep networks. The work builds on Nguyen et al. 2020 and therefore is not strikingly novel, the block structure itself and using PCA to investigate it are both coming from the original paper. There are some new insights coming out of the experiments carried out by the authors on this paper and therefore the paper has some novelty to it. The paper is very clearly written and the figures are informative and easy to parse. The relevance or impact of the work is in my mind small. Even though the paper provides interesting insights it is not clear to me what are the benefits of removing the block structure.    Interesting paper following up on Nguyen et al 2020. Some nice insights and overall a well-written paper, but probably impact is low. My recommendation is still leaning towards accepting the paper in the conference.

This paper studies the block structure phenomenon in training large capacity neural networks. The paper provides an explanation that the phenomenon is caused by dominant data points. The paper also investigates methods to mitigate this issue.  Strengths: 

(1) The paper uncovers the origin of the block structure phenomenon. 

(2) The experiments are insightful and thorough. 

(3) The paper's analysis of dominant data points can potentially be useful for training big networks on relatively small data. 

Weaknesses: 

(1) The paper lacks a rigorous theoretical analysis. 

(2) The idea of influential points is not very novel.  This paper provides a reasonable explanation to the block structure phenomenon. It is a useful empirical contribution. 

Authors study phenomenon in large-capacity neural networks which contain blocks of contiguous hidden layers with highly similar representations. The block structure has two seemingly contradictory properties: on the one hand, its constituent layers have highly similar dominant first principal components (PCs), but on the other hand, their representations, and their common first PC, are highly dissimilar across different random seeds. 

The authors investigated the origin of this block structure with the data and training methods. They found that the block structure arises from dominant datapoints - a small group of examples that share similar image statistics (e.g. background color).  The strengths of the paper:
- investigating the existing problem which is the block structure phenomenon in large-capacity networks trained on relatively small datasets,
- proposing a novel principal component regularization method.


The weaknesses of the paper:
- authors focus on the block structure phenomenon in large-capacity networks trained on relatively small datasets. In practice, a neural network is rarely trained on small data sets.
- the paper present too small a novelty, to eliminate the block structure, the authors studied different existing training mechanisms such as transfer learning and Shake-Shake regularization.
 The authors consider the problem of the block structure phenomenon uncovered in previous works but consider large-capacity networks trained on relatively small datasets. They try to answer the question: "is the block structure a sign of overfitting to idiosyncrasies of the data and training process, or does it pick up meaningful signals?" They tried to eliminate the block structure by a novel principal component regularization method, or alternative existing training mechanisms such as transfer learning and Shake-Shake regularization. In my opinion, the novelty of this paper is too small for the conference ICLR.

Building on the work by Nyugen et al. 2020  the authors examine the block structure phenomenon where subsequent layers of a neural network have very similar internal representations. They show that a small number of datapoints load very highly to the dominant principle components of the block structure.  They also examine the dynamics of the block structure and the fact that it changes through training. Finally the authors show that some regularizations such as Shake-shake improves the phenomenon.  While several facts mentioned in the paper are interesting, one of the key dissatisfying experiences of reading this paper is that the authors do not place this phenomenon in larger context or really examine its implication. 

On the issue of surprise and arising of the block structure:
1. It does not seem entirely surprising to me that subsequent layers have similar representations, after all this is why we need deep neural networks, to slowly evolve the representations towards something that correctly predicts the output. 
2. Since the authors mainly examine resnets, which have block structure architecture, this is even less surprising. Perhaps the authors could try to vary the residual connections. 
3. Can the block structure be seen from an information bottleneck viewpoint?


On the implications of block structure:
1. The authors don't address in their paper why this block structure "is bad". The authors have no results showing that it affects results or generalization. 
2.  The main issue with it seems to be that the layers have redundant representation and perhaps one could reduce depth if the representation evolved more quickly, but this is never mentioned or brought up in the paper. The paper has no experiments showing the ability to prune or reduce networks when they eliminate block structure. 


On the differences in block structure that arise:
1. There actually seems to be an order-dependence to the particular representation that arises, thats why different random seeds result in different representations. This could be because the network learns what to do on a the initial set of inputs first and then on other sets of inputs, so the authors could try batching groups of similar inputs which are shown first and which are shown later to the network. But having said that it is not clear why the order dependence is bad again. 
2. This would explain the transfer learning helping as well, because in transfer learning a different dataset is trained initially. 

On eliminating the block structure: 
1. It seems that most attempts to eliminate block structure still leaves some of it in tact. 
2.  Can it be removed by explicitly penalizing for betewen layers similarity? Would this affect generalization?


 The paper does not deeply examine the phenomenon at hand, it makes assumptions that are not explained, and does not explore implications well. 

Meta-review: This paper experimentally shows that the block-structure of similarities between layers typically appears for different models and such a structure is mainly induced by small set of dominant datapoints. Moreover, the dominant datapoints are not just noisy artifacts but represent some common image patterns such as background colors. The authors also found that the block structure can easily disappear by removing  the dominant datapoints, and the authors also proposed a method to suppress the block structure by regularizing PCs, Shake-Shake regularization, and transfer learning.

This paper gives thorough experiments that clarify the mechanism of appearance of a block structure. However, its significance is a bit minor. Indeed, the block structure does not affect the generalization ability very much, and it can be removed without changing the predictive performance. I agree that investigating the behavior of the internal representation is of scientific interest as the authors pointed out, but on the other hand, its significance would not be convincing. Indeed, this concern was pointed out by several reviewers. Next, the main focus of this study is about the setting of large model with small data size. It is not clear whether it is universal across different model size relative to the dataset size. There is no theoretical investigation (for example, the block-structure phenomenon could be explained by a high dimensional random matrix theory). 

In summary, this paper investigates a somehow interesting phenomenon but its significance is not convincing. Thus, it would be a bit below the threshold of the acceptance.

Decision: Reject