Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the collapse problem within nodes (tokens) of a sample if I correctly got the idea of it. The authors employ the effective rank to measure the quailty of the feature. Then, they borrow the idea of uniformity loss from the contrastive learning algorithm, and convert the gradient through back-propagation to the forward-propagation as the Contranorm algorithm. Finally in the experiments, the authors provide several results that the Contranorm boost the performance.
strength_and_weaknesses: Strength:

The paper presents a nice idea to convert the gradient to the form of normalization.


Weaknesses:

Instinctively, I would like to ask why it is not possible to apply uniformity loss straightforwardly unless it doesn't converge, the authors should provide a comparison between the proposed method and uniformity loss. Since, theoretically, the ContraNorm is equivalent to applying the loss, and appending the ContraNorm to the neural network does increase the computational costs during the inference time.

The authors did not provide an analysis of the computational costs of the proposed Contranorm, I wonder if the number of tokens grows, it will significantly decrease the training speed.

In the ImageNet1K experiment, the performance for #L=24 is still lower than that of #L=16, which means the proposed method does not actually address the mentioned over-smoothing problem, despite the effective rank depicted in figures 2 & 3 seems to be much higher than the reference model. Therefore, I do not consider the mechanism of feature collapsing discussed in this paper to be comprehensive or even correct, which challenges the motivation of this study.

I do not understand how the layer norm can be replaced by the factor 1+s in Eq.9, the layer norm should remove the bias which the factor 1+s cannot.

The clarity of this paper is not acceptable to this conference.
clarity,_quality,_novelty_and_reproducibility: Clarity:

This paper is hard to understand, and the main reason is the notations used in Section 3. I am heavily confused by the $\mathbf{H}$ in Eq. 6~12. It seems that the authors are updating the $\mathbf{H}$, however, I wonder if Eq. 10 still holds for $\mathbf{H}^2$ and $\mathbf{H}^1$, what does the superscript stand for? layers or what else?

Quality:

The overall quality of this paper is not satisfying because of the weaknesses mentioned above.

Novelty: 

The idea to convert the contrastive uniformity loss to the forward pass is very sophisticated, the method should be helpful to this community.

Reproducibility:

The paper lacks details on how the methods are actually implemented in the Transformers. For instance, how the stop-gradient is used? How the layer norm is configured in terms of the bias and scale factor? How is the former layer norm dealt with when you apply the contranorm before it?
summary_of_the_review: I did really spend a lot of time understanding this paper, and it is not because the technical details are difficult, it is simply because the paper is low in clarity and representation quality. I suggest the authors carefully revise this paper and submit it to another conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses an over smoothing issue of representation in Graph Learning and message-passing like computation unit (attention) as the depth of the models increase. The paper argues that attention maps are not a very good indictors of over smoothing, due to the fact that a feature can have low similarity while having high attendance score. Subsequently, an optimization based normalization unit is proposed to encourage large variance across features and decorrelation of dimensions. Results, especially for image and GCN, show good improvement.
 
strength_and_weaknesses: Comments and questions:
- The paper is very well written with good organization. Its topic of discussion is a well-motivated and focused problem with clear solution which makes it easy to follow. Results, especially for GCN and vision tasks, show good improvement as the depth of models increase. 
- Questions: 
    - In Subsection 3.1 and Figure 2, it was argued that similarity in attention does not imply feature collapse. The argument is based on using attention map, which is a cosine similarity between query and key, as a measure of representation collapse. The question is, what is referred as feature here? I am assuming the value embedding. If so, similarity between attention map naturally does not imply similarity between value pairs. What is new here or am I missing something?
    - The uniformity term in contrastive learning is between the representation in question and all its negative examples. However, $\dot{L}_{uniform}$ seem to measure similarity across all representation. Is that the case?  
     - The normalization layer Eq(7) and Eq(8), are designed as a single gradient step that updates the features in the direction of minimal similarity. Could this be amortized? as opposed to approximating a solution with single step. 
- Questions regarding experiment:
     -  What average performance in Table 1?
     -  The average performance for both BERT and ALBERT is reported inconsistently in the **Results** section and **Table 1**. Cloud you correct/clarify?
     -  Reported results in **Table 1** show tiny improvement. Are these the right models and experimental setup to show case your solution for the GLUE task? 

- Corrections:
    - missing fullstop in the second line form the last, in the second paragraph of the introduction section.
clarity,_quality,_novelty_and_reproducibility: Very clear, good quality. seems reproducible. 
summary_of_the_review: The paper motivated the problem attempts to solve, suggested a solution and provided insight into why it would work with theoretical justification. It further showed, empirically, performance improvement as the depth of the model increases.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the phenomenon of oversmoothing in GNNs and transformers; and presents the ContraNorm layer. The experiments on real-world dataset show the efficacy of this method.
strength_and_weaknesses: Strength:
It’s a nice method with good motivations and theoretical insights. 
The paper is generally well written.
The experiments show the efficacy of the model. It indeed can improve the performance in general.

weakness

1, I would ask one particular extreme case on any dataset: the authours can train a very deeper model that encounter the collapse issue, and the model performance is not good. However, with the help of ContraNorm layer, the issue can be solved. This may be important to show the efficacy.

2, Eq.(2) uniform loss part:  what’s the relation between symbol i^+ and x_k? seems some typos here.

3, could the authours give more reasons/motivations why directly discarding the second terms in Eq. (6)? And “Empirically, the two terms play a similar role in our method.” for any empirical evidence?

4, After reading the paepr, I feel that the ContraNorm is not necessarilty related to/address the collapse issues. As least, there is not enough evidence now. Some visualization of Fig. 1 in real world dataset and backbone is thus needed to directly show that the collapse issues happened when training the model; and ContraNorm can change the distribution of features. This can be shown on any dataset. Otherwise, we can just take ContraNorm as one typical type of contrast learning strategy to better train the model.
 And particularly, note that ContraNorm is built upon the layer norm.

clarity,_quality,_novelty_and_reproducibility: the paper is well written, and very clean. 

summary_of_the_review: please refer to the weakness part. I would ask for some direct evidence and visualization that ContraNorm can solve the collapse issue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the overs-moothing problem of graph neural network (GNN) and Transformers, and proposes to avoid the dimensional collapses in the representation, borrowing the idea from self-supervised learning. It thus proposes a normalization layer ContraNorm, which aims to learn a more uniform distribution in the embedding space that can ensures slighter dimensional collapse, motivated on the uniform loss from self-supervised learning. The proposed normalization layer can be inserted into GNNs and Transformers, and improves these baselines based on the experiments on three different scenarios: ViT for image classification, BERT for natural language understanding, and GNNs for node classifications.
strength_and_weaknesses: **Strengths:**

+ The motivation of the proposed method is clear. It is good this paper try to connect the designed normalization method with the uniformity loss, which provides stronger motivation/foundation for this paper, even though I have concerns for its connections and analyses.  

+ The effectiveness of the proposed method over baseline is empirically verified in several architectures (e.g., GNN, Transformer) on several datasets. 

**Weaknesses:** 

1.One severe problem of this paper is that it misses several important related work/baselines to compare[1,2,3,4], either in discussion [1,2,3,4]or experiments[1,2]. This paper addresses to design a normalization layer that can be plugged in the network for avoiding the dimensional collapse of representation (in intermediate layer). This idea has been done by the batch whitening methods [1,2,3] (e.g, Decorrelated Batch Normalization (DBN), IterNorm, etal.). Batch whitening, which is a general extent of BN that further decorrelating the axes, can ensure the covariance matrix of the normalized output as Identity (IterNorm can obtain an approximate one). These normalization modules can surely satisfy the requirements these paper aims to do. I noted that this paper cites the work of Hua et al, 2021, which uses Decorrelated Batch Normalization for Self-supervised learning (with further revision using shuffling). This paper should note the exist of Decorrelated Batch Normalization. Indeed, the first work to using whitening for self-supervised learning is [4], where it shows how the main motivations of whitening benefits self-supervised learning.

2.I have concerns on the connections and analyses, which is not rigorous for me. Firstly, this paper removes the $AD^{-1}$ in Eqn.6, and claims that “In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods (He et al., 2020; Grill et al., 2020). By throwing away some terms in the gradient, stop-gradient makes the training process asymmetric and thus avoids representation collapse with less computational overhead. It verifies the feasibility of our discarding operation”. I do not understand how to stop gradients used in SSL can be connected to the removement of $AD^{-1}$, I expect this paper can provide the demonstration or further clarification.  
Secondly, It is not clear why layerNorm is necessary. Besides, how the layer normalization can be replace with an additional factor (1+s) to rescale H shown in claims “For the convenience of analysis, we replace the layer normalization with an additional factor 1 + s to rescale H”. I think the assumption is too strong.   
In summary, the connections between the proposed contraNorm and uniformity loss requires: 1) removing $AD^{-1}$ and 2) add layer normalization, furthermore the propositions for support the connection require the assumption “layer normalization can be replace with an additional factor (1+s) to rescale H”. I personally feel that the connection and analysis are somewhat farfetched. 



**Other minors:**

1)Figure 1 is too similar to the Figure 1 of Hua et al, 2021, I feel it is like a copy at my first glance, even though I noted some slightly differences when I carefully compare Figure 1 of this paper to Figure 1 of Hua et al, 2021. 

2)The derivation from Eqn. 3 to Eqn. 4 misses the temperature $\tau$, $\tau$ should be shown in a rigorous way or this paper mention it. 

3)In page 6. the reference of Eq.(24)? 

**References:**

[1] Decorrelated Batch Normalization, CVPR 2018  
[2] Iterative Normalization: Beyond Standardization towards Efficient Whitening, CVPR 2019  
[3] Whitening and Coloring transform for GANs. ICLR, 2019  
[4]Whitening for Self-Supervised Representation Learning, ICML 2021  

clarity,_quality,_novelty_and_reproducibility: The clarity is clear but I have certain concerns on the claims, the quality is overall good, and the novelty is somewhat incremental. I believe the experiments can be somewhat reproduced based on the descriptions of this paper.
summary_of_the_review: This paper is a well-motivated paper with empirical success in experiments, but it misses several important related work for comparison and some claims are not well clarified. In summary, I am slightly negative to this paper. 

==================after rebuta==================

I look over the revised paper, and the authors almost address all my concerns. E.g., add the comparison to the missing related works, and re-organize the clarity. I still have concerns on the claims that “In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods”. I believe removing $AD^{-1}$ may improve the performance based on the authors experiments, but it does not correspond to the stop-gradient in contrastive learning (they are different mechanisms, e.g., stop-gradient is essential to prevent collapse in simsiam/BOYL, while I believe removing $AD^{-1}$ is not the essential part in the proposed method.). Even though this concern holds, the revised version is more rigorous than the previous one, and I tend to positive to this paper. I thus raise my score from 5 to 6.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This manuscript addresses the overs-moothing problem of graph neural network (GNN) and Transformers, and proposes to avoid the dimensional collapses in the representation, borrowing the idea from self-supervised learning. It proposes a normalization layer ContraNorm, which aims to learn a more uniform distribution in the embedding space that can alleviate dimensional collapse, motivated on the uniform loss from self-supervised learning. The proposed normalization layer can be inserted into GNNs and Transformers, and improves these baselines based on the experiments on three different scenarios: ViT for image classification, BERT for natural language understanding, and GNNs for node classifications. Reviewers unanimously suggest to accept the paper, so is the decision.
justification_for_why_not_higher_score: Three reviewers suggest marginally above the acceptance threshold, while one reviewer likes the idea to convert the gradient to the form of normalization and suggests acceptance. So it is a solid acceptance for poster. 
justification_for_why_not_lower_score: All reviewers unanimously suggest to accept the paper.
Decision: Accept: poster