Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The study of (Nguyen et al., 2020) reported that wide and deep neural networks tend to have similar feature representations across layers (called "block structure").
(Nguyen et al., 2020) also revealed that the block structure is essentially induced by the first principal component (PC) of the representation in each layer.

This paper provides more detailed empirical analysis of the block structure.
To measure the similarity, the authors used KCA (with linear kernel) as the similarity measure, which was also used in (Nguyen et al., 2020).
This paper reported that
* The dominant datapoints, which lie in the direction of PC, share some common image patterns such as background colors.
* The dominant datapoints are different across different model initialization.
* Although the block structure emerges in the very first stage of the training, the dominant data points differ between the beginning and at the end of the training.
* The block structure can be suppressed by modifying training, e.g., by regularizing PCs, by using Shake-Shake regularization, and by using transfer learning.
main_review: ## Strength

### [Strength1]
The authors conducted exhaustive experiments on standard datasets (CIFAR-10, CIFAR-10, Patch Camelyon) and on different model architectures (several levels of shallow/deep ResNets with narrow/wide layers).
According to (Nguyen et al., 2020), the block structure can emerge beyond ResNet as well (they conducted experiments on ResNet without skip connections).
These observations imply that the results reported in the paper will be ubiquitous (at least for images).
Revealing general phenomena underling deep models would be valuable to the community.

### [Strength2]
The observation that dominant datapoints tend to share some common image patterns seems to be nontrivial.
This phenomenon itself can be a topic of interest for further researches.
This phenomenon might also imply that deep models can be vulnerable to a slight change of the colors of the entire images.
That is, if we modify an input image to have a color close to that of dominant points, its intermediate representations might be large (e.g., Fig5), which will lead to different outputs (i.e., the model is vulnerable). 


## Weakness

In the paper, the authors reported that the block structure is induced by the dominant datapoints.
The authors then discussed the experimental results based on this connection.
The results are therefore connected to block structure through dominant datapoint, i.e., [block structure] -- [dominant datapoints] -- [results].
Because of this structure, there are two topics underlying the current paper: (i) the connection [block structure] -- [dominant datapoints] discussing the origin of the block structure, and (ii) the connection  [dominant datapoints] -- [results] discussing the emergence and dynamics of dominant datapoints.

My concern here is that these topics (i) and (ii) would be irrelevant (or, at least the connection would be far looser than expected).
The current paper claims that [block structure] and [results] are tightly connected through [dominant datapoints].
However, the validity of the connection (i) [block structure] -- [dominant datapoints] is questionable, as I describe below.
In particular, I suspect it can be an artifact induced by the use of KCA with linear kernel for the similarity measure.

### [Detail]
Essentially, KCA with linear kernel measures the squared Frobenius norm of the covariance matrix (with some normalization).
It is known that the covariance is vulnerable to the existence of outliers.
Indeed, as reported in (Nguyen et al., 2020, Fig3) as well as in Fig4 of the current paper, removing outliers (i.e., PCs or dominant datapoints) eliminate the similarity between layers.
These observations imply that "dominant datapoints are kept being outliers across layers ((ii) above)" rather than "the representations are similar across layers ((i) above)" because (at least some fraction of) the similarity is the consequence of the existence of such outliers.

In Appendix J, the authors reported the similarity under some other kernels.
There, we can find that linear kernel exhibits the strongest similarity (possibly because it is vulnerable to the existence of outliers).
This raises a question on the adequacy of the current study which rely solely on linear kernel.
The removal of outliers will lead to smaller KCA under linear kernel (and less blocks) as reported in the paper.
However, I wonder whether the similar can hold for KCA under some other kernels.
For example, we can reduce the effect of outliers with large activations for block structure by adopting the feature map $\phi(x) = \frac{x}{\\| x \\|}$ which induces the cosine kernel $k(x, y) = \cos(x, y)$.
Then, the question is whether the removal of dominant datapoints lead to reduction of KCA under cosine kernel (and some other kernels as well) and elimination of the block structure.
If the significant reduction of KCA is not observed in variety of kernels, the connection (i) [block structure] -- [dominant datapoints] would be an artifact induced by the use of linear kernel.

Here, I would like to remind that my comment above is all about the validity of the connection (i) [block structure] -- [dominant datapoints].
If the connection (i) still holds despite the selection of kernels, that would be far nontrivial, and would be an interesting phenomenon with further investigations needed.
I would also remark that the second connection (ii) [dominant datapoints] -- [results] can be of independent interest irrelevant to the validity of the connection (i). (unless there are related studies in this direction)


summary_of_the_review: The current paper mixes the two connections (i) [block structure] -- [dominant datapoints] and (ii) [dominant datapoints] -- [results].
To make the paper rigid, these two connections need to be distinguished.
The results in the current paper implies the connection (ii).
However, the validity of the connection (i) remains questionable.
I suspect it can be an artifact induced by the use of KCA with linear kernel for the similarity measure.
If this is the case, the paper needs to be reorganized with its main focus on (ii) but not (i).
If the authors are willing to connect (i) and (ii), it would be essential to confirm that the reported phenomena are not artifact induced by the use of KCA with linear kernel, and demonstrate they are ubiquitous irrelevant to the selection of similarity measures.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors investigate the block structure present in large deep networks and relate it to structure in the data and investigate trainings approaches that can reduce the presence of block structure.  
main_review: The claims made by the authors in the paper seem correct and there is no obvious error I can see in the paper. The paper provides a number of experimental results that provide interesting insights into the appearance of block structures in large deep networks. The work builds on Nguyen et al. 2020 and therefore is not strikingly novel, the block structure itself and using PCA to investigate it are both coming from the original paper. There are some new insights coming out of the experiments carried out by the authors on this paper and therefore the paper has some novelty to it. The paper is very clearly written and the figures are informative and easy to parse. The relevance or impact of the work is in my mind small. Even though the paper provides interesting insights it is not clear to me what are the benefits of removing the block structure.   
summary_of_the_review: Interesting paper following up on Nguyen et al 2020. Some nice insights and overall a well-written paper, but probably impact is low. My recommendation is still leaning towards accepting the paper in the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the block structure phenomenon in training large capacity neural networks. The paper provides an explanation that the phenomenon is caused by dominant data points. The paper also investigates methods to mitigate this issue. 
main_review: Strengths: 

(1) The paper uncovers the origin of the block structure phenomenon. 

(2) The experiments are insightful and thorough. 

(3) The paper's analysis of dominant data points can potentially be useful for training big networks on relatively small data. 

Weaknesses: 

(1) The paper lacks a rigorous theoretical analysis. 

(2) The idea of influential points is not very novel. 
summary_of_the_review: This paper provides a reasonable explanation to the block structure phenomenon. It is a useful empirical contribution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Authors study phenomenon in large-capacity neural networks which contain blocks of contiguous hidden layers with highly similar representations. The block structure has two seemingly contradictory properties: on the one hand, its constituent layers have highly similar dominant first principal components (PCs), but on the other hand, their representations, and their common first PC, are highly dissimilar across different random seeds. 

The authors investigated the origin of this block structure with the data and training methods. They found that the block structure arises from dominant datapoints - a small group of examples that share similar image statistics (e.g. background color). 
main_review: The strengths of the paper:
- investigating the existing problem which is the block structure phenomenon in large-capacity networks trained on relatively small datasets,
- proposing a novel principal component regularization method.


The weaknesses of the paper:
- authors focus on the block structure phenomenon in large-capacity networks trained on relatively small datasets. In practice, a neural network is rarely trained on small data sets.
- the paper present too small a novelty, to eliminate the block structure, the authors studied different existing training mechanisms such as transfer learning and Shake-Shake regularization.

summary_of_the_review: The authors consider the problem of the block structure phenomenon uncovered in previous works but consider large-capacity networks trained on relatively small datasets. They try to answer the question: "is the block structure a sign of overfitting to idiosyncrasies of the data and training process, or does it pick up meaningful signals?" They tried to eliminate the block structure by a novel principal component regularization method, or alternative existing training mechanisms such as transfer learning and Shake-Shake regularization. In my opinion, the novelty of this paper is too small for the conference ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Building on the work by Nyugen et al. 2020  the authors examine the block structure phenomenon where subsequent layers of a neural network have very similar internal representations. They show that a small number of datapoints load very highly to the dominant principle components of the block structure.  They also examine the dynamics of the block structure and the fact that it changes through training. Finally the authors show that some regularizations such as Shake-shake improves the phenomenon. 
main_review: While several facts mentioned in the paper are interesting, one of the key dissatisfying experiences of reading this paper is that the authors do not place this phenomenon in larger context or really examine its implication. 

On the issue of surprise and arising of the block structure:
1. It does not seem entirely surprising to me that subsequent layers have similar representations, after all this is why we need deep neural networks, to slowly evolve the representations towards something that correctly predicts the output. 
2. Since the authors mainly examine resnets, which have block structure architecture, this is even less surprising. Perhaps the authors could try to vary the residual connections. 
3. Can the block structure be seen from an information bottleneck viewpoint?


On the implications of block structure:
1. The authors don't address in their paper why this block structure "is bad". The authors have no results showing that it affects results or generalization. 
2.  The main issue with it seems to be that the layers have redundant representation and perhaps one could reduce depth if the representation evolved more quickly, but this is never mentioned or brought up in the paper. The paper has no experiments showing the ability to prune or reduce networks when they eliminate block structure. 


On the differences in block structure that arise:
1. There actually seems to be an order-dependence to the particular representation that arises, thats why different random seeds result in different representations. This could be because the network learns what to do on a the initial set of inputs first and then on other sets of inputs, so the authors could try batching groups of similar inputs which are shown first and which are shown later to the network. But having said that it is not clear why the order dependence is bad again. 
2. This would explain the transfer learning helping as well, because in transfer learning a different dataset is trained initially. 

On eliminating the block structure: 
1. It seems that most attempts to eliminate block structure still leaves some of it in tact. 
2.  Can it be removed by explicitly penalizing for betewen layers similarity? Would this affect generalization?



summary_of_the_review: The paper does not deeply examine the phenomenon at hand, it makes assumptions that are not explained, and does not explore implications well.
review_title: null

Meta-review: This paper experimentally shows that the block-structure of similarities between layers typically appears for different models and such a structure is mainly induced by small set of dominant datapoints. Moreover, the dominant datapoints are not just noisy artifacts but represent some common image patterns such as background colors. The authors also found that the block structure can easily disappear by removing  the dominant datapoints, and the authors also proposed a method to suppress the block structure by regularizing PCs, Shake-Shake regularization, and transfer learning.

This paper gives thorough experiments that clarify the mechanism of appearance of a block structure. However, its significance is a bit minor. Indeed, the block structure does not affect the generalization ability very much, and it can be removed without changing the predictive performance. I agree that investigating the behavior of the internal representation is of scientific interest as the authors pointed out, but on the other hand, its significance would not be convincing. Indeed, this concern was pointed out by several reviewers. Next, the main focus of this study is about the setting of large model with small data size. It is not clear whether it is universal across different model size relative to the dataset size. There is no theoretical investigation (for example, the block-structure phenomenon could be explained by a high dimensional random matrix theory). 

In summary, this paper investigates a somehow interesting phenomenon but its significance is not convincing. Thus, it would be a bit below the threshold of the acceptance.
Decision: Reject