ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond
Authors: Xiaojun Guo
Yifei Wang
Tianqi Du
Yisen Wang
Venue: ICLR 2023 poster (2023)

Abstract: Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance degenerates as the layer goes deeper. Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective dimensional collapse in which representations lie in a narrow cone. Accordingly, inspired by the power of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer ContraNorm. Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and slighter dimensional collapse. On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under some conditions. Our proposed normalization layer can be easily inserted into GNNs and Transformers with negligible parameter overhead. Experiments on various real-world datasets verify the effectiveness of our method.

Published as a conference paper at ICLR 2023
CONTRANORM: A CONTRASTIVE LEARNING PER- SPECTIVE ON OVERSMOOTHING AND BEYOND
Xiaojun Guo Yifei Wang Tianqi Du Yisen Wang
National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University
2 Schooo of Mathematical Sciences, Peking University
3 Institute for Artificial Intelligence, Peking University
ABSTRACT
Oversmoothing is a common phenomenon in a wide range of Graph Neural Net- works (GNNs) and Transformers, where performance worsens as the number of layers increases. Instead of characterizing oversmoothing from the view of com- plete collapse in which representations converge to a single point, we dive into a more general perspective of dimensional collapse in which representations lie in a narrow cone. Accordingly, inspired by the effectiveness of contrastive learning in preventing dimensional collapse, we propose novel normalization layer called ContraNorm. Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and a slighter dimen- sional collapse. On the theoretical analysis, we prove that ContraNorm can alle- viate both complete collapse and dimensional collapse under certain conditions. Our proposed normalization layer can be easily integrated into GNNs and Trans- formers with negligible parameter overhead. Experiments various real-world datasets demonstrate the effectiveness of our proposed ContraNorm. Our imple- mentation is available at plttps://qit.n:
INTRODUCTION
Recently, the rise of Graph Neural Networks (GNNs) has enabled important breakthroughs in vari- ous fields of graph learning (Ying et al. 2018 Senior et al. (020). Along the other avenue, although getting rid of bespoke convolution operators, Transformers (Vaswani et al.) 2017) also achieve phe- nomenal success in multiple natural language processing (NLP) tasks (Lan et al. 2020 Liu et al.] 2019 Rajpurkar et al., 2018) and have been transferred successfully to computer vision (CV) field (Dosovitskiy et 2021 Liu al. 202T: Strudel al. 2021). Despite their different model archi-
tectures. GŇNs and Transtormers are both hindered by the oversmoothing problem (Li et al.] 2018 Tang et al. 2021), where deeply stacking layers give rise to indistinguishable representations and sigmificant performance deterioration.
In order to get rid of oversmoothing, we need to dive into the modules inside and understand how oversmoothing happens on the first hand. However, we notice that existing oversmoothing analysis fails to fully characterize the behavior of learned features. A canonical metric for oversmoothing is the average similarity (Zhou et al.] 2021 Gong et al.) 2021 Wang et al.) 2022). The tendency of similarity converging to I indicates representations shrink to single point (complete collapse). However, this metric can not depict a more general collapse case, where representations span a low- dimensional manifold in the embedding space and also sacrifice expressive power, which is called dimensional collapse (left figure in Figure []. In such cases, the similarity metric fails quantify the collapse level. Therefore, we need to go beyond existing measures and take this so-called dimen- sional collapse into consideration. Actually, this dimensional collapse behavior is widely discussed in the contrastive learning literature (Jing al.) 2022 Hua et al. 2021 Chen & He 2021 Grill ct al. 2020). which may hopefully help us characterize the oversmoothing problem of GNNs and Transformers. The main idea of contrastive learning is maximizing agreement between different augmented views of the same data example (i.e. positive pairs) via a contrastive loss. Common
contrastive loss can be decoupled into alignment loss and uniformity loss (Wang & Isola 2020)
"Equal Contribution.
Corresponning Author: Yisen Wang (yisen, wang wang@@pk.eddu

---

Published as a conference paper at ICLR 2023
The two ingredients correspond to different objectives: alignment loss expects the distance between positive pairs to closer, while uniformity loss measures how well the embeddings are uniformly distributed. Pure training with only alignment loss may lead to a trivial solution where all represen- tations shrink to one single point. Fortunately, the existence of uniformity loss naturally helps solve this problem by drawing representations evenly distributed in the embedding space.
Given the similarities between the oversmoothing problem and the representation collapse issue, establish connection between them. Instead of directly adding uniformity loss into model training, we design a normalization layer that can be easily used out of the box with almost no parameter overhead. To achieve this, we first transfer the uniformity loss used for training to loss defined over graph node representations, thus it can optimize representations themselves rather than model parameters. Intuitively, the loss meets the need of drawing a uniform node distribution. Following the recent research in combining optimization scheme and model archi- tecture (Yang et al. [02T] Zhu al. 202T; Xie al. 2021 Chen al.) (022). we use the transferred uniformity loss as an energy function underlying our proposed normalization layer, such that descent steps along it corresoonds with the forward pass. By analyzing the unfolding iterations of the principled uniformity loss, we design new normalization layer ContraNorm. demonstrates
As a proof of concept, Figure [ that ContraNorm makes the features away from each other, which eases the dimensional col- lapse. Theoretically, we prove that ContraNorm increases both average variance and effective rank of representations. thus solving complete collapse and dimensional collapse effectively. We also conduct a comprchensive evaluation of ContraNorm on various tasks. Specifically, ContraNorm boosts the average performance of BERT (Devlin al.) (018)) from 82.59% to 83.54% on the validation set of General Language Understanding Evaluation (GLUE) datasets (Wang et al. 2019), and raises the test accuracy of DeiT (Touvron et al. 2021) with 24 blocks from 77.69% to 78.67% (Russakovsky et al. 2015) dataset. For GNNs, experiments are conducted on fully supervised graph node classification tasks, and our proposed model outperforms vanilla Graph Convolution Network (GCN) (Kipf & Welling 2017) on all depth settings. Our contributions are summarized as:
CettrNNer
raving Recion
pesh sway Dross eadl
Figure 1: An illustration of how our proposed ContraNorm solves the dimensional collapse. Left: Features suffer from dimensional collapse. Right: With the help of ContraNorm, features be- come more uniform in the space, and the dimen- sional collapse is eased.
ImageNet1K K
We dissect the limitations of existing oversmoothing analysis, and highlight the importance of incorporating the dimensional collapse issue into consideration.
Inspired by the techniques from contrastive learning measure and resolve oversmooth- ing. we propose ContraNorm as an optimization-indueee normalization layer to prevent dimensional collapse.
Experiments on a wide range of tasks show that ContraNorm can effectively mitigate di- mensional collapse in various model variants, and demonstrate clear benefits across three different scenarios: VIT for image classification, BERT for natural language understanding. and GNNs for node classifications.
BACKGROUND & RELATED WORK
Message Passing in Graph Neural Networks. In the literature of GNNs, message-passing graph neural networks (MP-GNNs) are intensively studied. It progressively updates representations by exchanging information with neighbors. The update of node i's representation in 1-th layer is for- malized as h ( UPDATE(h (GGREGATE(D ((-1) [h, [h(--11 e N(O))), where (iii) denotes the neighborhood set of node i, AGGREGATE() is the procedure where nodes exchange message, and UPDATE(-) is often a multi-layer perceptron (MLP). A classical MP-GNNs model is GCN (Kipf & Welling 2017), which propagates messages between 1-hop neighbors using an adjacency matrix.
Self-Attention in Transformers. Transformers encode information in global scheme with self- attention as the key ingredient (Vaswani et al. 2017). Self-attention module re-weights interme-
2

---

Published as a conference paper at ICLR 2023
diate representations by aggregating semantically near neighbors. Formally, it estimates similari- ties between key and query, namely self-attention matrix, as A = coftmax(QQ Q = XWa, and K = XWK k where X, Wo, and WK the input, weight matrices for query and key, spectively. A head self-attention module with a residual connection can be formularized as attn(X) = X + Σk= A₃ V where h is the number of heads, and V. W are weights for value and final output.
Connections between Message Passing and Self-Attention. Note that the self attention matrix can be seen as a normalized adjacent matrix of a corresponding graph (Shi et al. (002). Considering a weighted fully connected graph G with adjacency matrix denoted as A, we map nodes to token representations and set weights of the edge between node i and node j to exp Q, Kj). Then (i,j) entry of a normalized adjacency matrix is A.j A.//D,, = exp(Q, K2)/Z, Σx exp K&), where diagonal matrix Da - Σ; Arj. Apparently, Á is equal the form of self-attention matrix defined in Transformers. Simultaneously, A plays major part in the message- passing scheme by deciding which nodes exchange information with.
Oversmoothing in GNNS and Transformers. The term oversmoothing is firstly proposed by Li et al.] (2018) in research of GNNs. Intuitively, representations converge to constant after repeatedly exchanging messages with neighbors when the layer goes to infinity. Zhou et al. (2020) mathemat- ically proves that, under some conditions, the convergence point carries only information of the graph topology. Coincidentally, ers. Unlike CNNs, Transformers can not benefit from simply deepening layers, and even saturates with depth increasing. Early works empirically ascribe it to attention/feature collapse or patch/token
oversmoothing-lik phenomenon is also observed in Transform-
uniformity (Tang al. 2021 Zhou et al. 2021 Gong et al. 2021 Yan et al. (022). To be specific, the attention maps tend to be overly similar in later layers, whereby features insufficiently exchange information and lose diversity in the end. Outputs of pure transformers, i.e., attention without skip connections MLPs, also observed to converge to rank-1 matrix (Dong al.) 2021). For illustration proposes, we also refer to the degeneration issue Transformers as oversmoothing.
Whitening Methods for Dimensional Collapse. Whitening methods ensure that the covariance matrix of normalized outputs is diagonal, making dimensions mutually independent and implicitly solving dimensional collapse. Huang et al. (2018; 2019]; Siarohin et al. (2018); Ermolovv et al. (2021) all adopt the idea of whitening, but differ in calculating details of whiteming matrix and application domain. Compared with them, avoid complicated calculations on the squared root of the inverse covariance matrix and delicate design of backward pass for differentiabilityy Moreover, Huang et al. (2018 (219) are proposed for convolution operations,Siarohi et al. (018)) is for GAN, and Ermolov al. (2021) works in self-supervised learning. In contrast, we borrow the idea from contrastive learning and solve oversmoothing in completely different fields like Transformers and GNNs.
3. MITIGATING OVERSMOOTHING FROM THE PERSPECTIVE OF CONTRASTIVE LEARNING
In this section, we first empirically demonstrate that current similarity metric fails to characterize dimensional collapse, thus overlooking crucial part of oversmoothing. To address this problem, we draw inspiration from contrastive learning whose uniformity property naturally rules out dimen- sional collapse. Specifically, we transfer the uniformity loss to a loss that directly acts on represen- tations. By unfolding the optimization steps along this loss, we induce normalization layer called ContraNorm. Theoretically, we prove our proposed layer helps mitigate dimensional collapse.
3.1 THE CHARACTERIZATION OF OVERSMOOTHING
In this part, we begin by highlighting the limitations of existing metrics in characterizing over- smoothing. These limitations motivate us adopt the effective rank metric, which has been demon- strated to be effective in capturing the degree of dimensional collapse in contrastive learning.
Taking the oversmoothing problem of Transformers as an example without loss of generality, a prevailing metric is evaluating attention map similarity (Wang et al.) [022 Gong al. 2021| Shi al. 2022). The intuition is that as attention map similarity converges to one, feature similanty increases, which can result in a loss of representation expressiveness and decreased performance. However, by conducting experiments transformer structured models like VIT and BERT, we find
3

---

Published as a conference paper at ICLR 2023
ellariign
Tayer index
layer
layer index
(a) Similarity on BERT
(b) Similarity on VIT
Feature Distribution
Effective Rank
Figure 2: Comparison between different metrics of oversmoothing of 12-layer BERT on GLUE sub-task STS-B and 12-layer VIT on CIFARIO. Fig. 2a and Fig. 2b show feature and attention map cosine similarity along layer index. Fig. 2c and Fig. [d] show the singular value distribution of features and effective rank of BERT w/ and w/o ContraNorm.
that high attention map similarity does not necessarily correspond to high feature similarity. As shown in Fig. 2a and Fig, 2b] although attention map similarity is close to 0.8 or even higher, the feature similarity remains below 0.5 in most cases. It means the oversmoothing problem still occurs even with a low feature similarity. This finding suggests that similarity can not fully depict the quality of representations and the oversmoothing problem.
Intuitively, we can consider a case that representations in the latent embedding space do shrink to one single point but span low dimensional space. In such cases, feature similarity may be relatively low, but representations still lose expressive power. Such representation degeneration problem is known as dimensional collapse, and it is widely discussed in the literature of contrastive learning. In contrastive learning, common practice to describe dimensional collapse is the vanishing distribution of singular values (Gao et al. 2019 Ethayarajh| 2019 Jing et al. 2022). To investigate whether dimensional collapse occurs in Transtormers, we draw the singular value distribution of features in the last block of 12-layer BERT. As shown in Fig. [2 the insignificant (nearly zero) values dominate singular value distribution in the deep layer of vanilla BERT, indicating that representations reside a low-dimensional manifold and dimensional collapse happens. To show the collapse tendency along layer index, we simplify the singular value distribution to concise scalar effective rank (erank) (Roy & Vetterli 2007), which covers the full singular value spectrum.
Definition 3.1 (Effective Rank). Considering matrix X R whose singular value decomposi- tion is given by X = UEV, where Σ a diagonal matrix with singular values d1 2 02 2 σQ 2 0 with Q min(m, n). The distribution of singular values is defined Linormalized form p: 0 Σ =1 |oz| The effective rank of the matrix X, denoted as erank X), ) is defined as erank( X) exp( H P2,, -PQ)), where H Hp1,P2, ... -PQ) is the Shannon entropy given by H HPI.PP. ... PQ) = - Ei
L1
- Pk: log Pk.
Based on erank, we revisit the oversmoothing issue of Transformers Fig. 2d the effective rank descends along with layer index, indicating an increasingly imbalanced singular value distribution in deeper layers. This finding not only verifies dimensional collapse does occur in Transformers, but also indicates the effectiveness of using effective rank as metric to detect this issue.
We can see that
3.2 INSPIRATIONS FROM THE UNIFORMITY LOSS IN CONTRASTIVE LEARNING
The core idea of contrastive learning is maximizing agreement between augmented views of the same example (i.e.. positive pairs) and disagreement of views from different samples (i.e. negative pairs). A popular form of contrastive learning optimizes feature representations using a loss function with limited negative samples (Chen et al.) 2020). Concretely, given a batch of randomly sampled examples, for each example we generate its augmented positive views and finally get total of N samples. Considering an encoder function f. the contrastive loss for the positive pair (i, it) is
exp(f(x:TTf(z;)))>
C(i,it) -log
(1)
Σn
1==1 f(xx)/ f(cl ((oe))
f(x:)/T)
4

---

Published as a conference paper at ICLR 2023
where T denotes temperature factor. The loss be decoupled into two parts named alignment loss and uniformity loss:
Caligu =
Cuniform (i) log Σ exp(f(xx)) f(x))/T).
The alignment loss encourages feature representations for positive pairs be similar, thus being invariant to unnecessary noises. However, training with only the aligmment loss will result trivial solution where all representations are identical. In other words, complete collapse happens. While batch normalization (loffe & Szegedy 2015)) can help avoid this issue, it cannot fully prevent the problem of dimensional collapse, which still negatively impacts learning (Hua et al. 202I).
Thanks to the property of uniformity loss, dimensional collapse can be solved effectively. Reviewing the form of uniformity loss in Eq (2), it maximizes average distances between all samples, resulting in embeddings that are roughly uniformly distributed in the latent space, and thus more information is preserved. The inclusion of the uniformity loss in the training process helps to alleviate dimen- sional collapse. Intuitively, it can also serve sharp tool for alleviating oversmoothing in models such as GNNs and Transformers.
An alternative approach to address oversmoothing is to directly incorporate the uniformity loss into the training objective. However, our experiments reveal that this method has limited effectiveness (see Appendix D for more details). Instead, we propose a normalization layer that can be easily integrated into various models. Our approach utilizes the uniformity loss an underlying energy function of the proposed layer, such that a descent step along the energy function corresponds to a forward pass of the layer. Alternatively, we can view the layer as the unfolded iterations of an optimization function. This perspective is adopted to elucidate GNNs (Yang al. 2021 Zhu al. 2021). Transformers (Yang et al. 2022). and classic MLPs (Xie et al. 2021
Note that the uniformity loss works by optimizing model parameters while what the normalization layer directly updates is representation itself. So we first transfer the uniformity loss, which serves as a training loss, to a kind of architecture loss. Consider fully connected graph with restricted number of nodes, where node h, is viewed as representation of random sample I; Reminiscent of Luuiform, we define Luiiform over all nodes in the graph as
Luniform
Cuniforn (i) Σ log Σ CTThT/TT h//T.
This form uniformity loss is defined directly on representations, and we later use it as the under- lying energy function for representation update.
3.3 THE PROPOSED CONTRANORM
Till now, we able build connection between layer design and the unfolded iterations of descent steps used minimize uniformity loss. Specifically, we take the derivative of Luniform on node representations:
aunitorm
exp(h hz/hT h//TT + Σ exp(h, 12//T) En exp(h? hx/T)
hi/T,
(4)
Oh;
En exp(h) h2/T)
by denoting feature matrix as H with h as the i-th row, we rewrite Eq. into a matrix form for simplicity:
@unttfrrm 0H
AD-
(5)
where A = exp(H /T) and D deg( A In order to reduce the uniformity loss Luniform natural way to take a single step performing gradient descent, which is to update representations H in the way of
H, H, ALmiform
H, 3/7 A+
AD- A
OH,
'de((A) is diagonal matrix, whose element in the i-th and the i-th column is the of the i-th row of A.
5

---

Published as a conference paper at ICLR 2023
where H& and H2 denote the representations before and after the update, respectively, and is the step size of the gradient descent. By taking this update after a certain representation layer, we can reduce the uniformity loss of the representations and thus help ease the dimensional collapse.
In Eq.(6,, there exist two terms D-1 A and AD multiplied with H3. Empirically, the two terms play a similar role our method. Note that the first term is related to self-attention matrix in Transformers, so we only preserve it and discard the second one. Then Eq. becomes
H = Hb S/T x (D H)H - Ho - s/T x softmax (fftmax(HHH.... Hb-
In fact, the operation corresponds the sop-gradient technique, which is widely used in contrastive learning methods (He et al. 2020; Grill et al.) 2020; Tao et al. (2022). By throwing away some terms in the gradient, stop-gradient makes the training process asymmetric and thus avoids representation
collapse with less computational overhead, which is discussed detail in Appendix [A]
However, the layer induced by (q. still can not ensure uniformity on representations. Consider an extreme case where softmax softmax(H,HH,,
is equal to identity matrix I. 7 becomes H, H, - 8/T x H = (1 - 8/T) H which just makes the scale of H smaller and does not help alleviate the complete collapse. To keep the representation stable, propose two methods:
Regularization. We add a regularization term -; ||4: to the uniformity loss. When the regularization term becomes smaller, the norm of hú becomes larger. Therefore, adding this term can help prevent the norm of representation hú from becoming smaller. In this way, the update form becomes
-1+
S/T soltmax( H, Hb.
Proposition 1. Lete (1,1, /VT. For attention matrix A softmax HH, ). lel min be the smallest eigenvalue of P (I ee (I A) (I A) (I eeT). For the update in E i.e. H, ((1 ([1+s) +)] Hb, > 0, we have Var (ar 2 (1 + so V Var (Hs). Especially,
Mmin Z 0, we have V (ar(H.) 2 Var(HH))
Proposition] D giives bound for the ratio of the variance after and before the update. It shows that the change of the variance is influenced by the symmetric matrix P = -I- (I-ee" -I- -I) (I- ee e) If P is a semi-positive definite matrix, we will get the result that V aar H.) Z Var (n), which indicates that the representations become more uniform. In Appendix B we will give out
some sufficient conditions for that P is semi-positive definite.
LayerNorm. We leverage layer normalization (Ba et al. 2016), which adjusts the representation according to its mean and variance. The update form of the layer normalization on representation h; is LayerNormhh,, (h,) = 7 ((h: mean h)))) / Var(h.)) (h.)) + + 8, where y and 3 are learnable parameters and & = 10-5 prevents the denominator from becoming zero. The learnable parameters and B can rescale the representation h, to help ease the problem. We append the layer normalization to Eq.(7) as
H. = LayerNorm (H, s/T softmax H, Ho)
where applying the layer normalization to a representation matrix H means applying the layer nor- malization to all its components h1...., h,, We empirically compare the two proposed methods and find their performance comparable, while the second one performs slightly better. Therefore, adopt the second update form and name it Contrastive Normalization (ContraNorm). The Con- traNorm layer can be added after any representation layer to reduce the uniformity loss and help relieve the dimensional collapse. We discuss the best place to plug our ContraNorm in Appendix E Given H, E Rx where n and is the number of samples in the batch and hidden dimension, the time complexity of ContraNorm is O(n2d), which is the same order as the self alff-attention operation in Transformer. For enhancing scalability, we propose modified version of ContraNorm with linear complexity in the number of samples. See Appendix F for more details.
3.4 THEORETICAL ANALYSIS
In this part, we will give theoretical result on our proposed ContraNorm method. We will show that with different form of the uniformity loss, the ContraNorm update can help alleviate the dimensional collapse.
Proposition 2. Consider the update form
H = (1 + H) s HH H
(10)

---

Published as a conference paper at ICLR 2023
let max be the largest singular value of H For 0 satisfying + (1 o max)) > 0, we have crank( H,) > cran
Proposition 2 gives a promise of alleviating dimensional collapse under special update form. De- note Ldim tr((T HH ///4 then Laim shares similar form with Barlow Twins (Zbnn- tar et al. (021). This loss tries to equate the diagonal elements of the similarity matrix HH to and equate the off off-diagonal elements of the matrix to O, which drives different features becoming orthogonal, thus helping the features become more uniform in the space. Note that (H.). which HH, implies that this update form is in fact a ContraNorm with different uniformity loss. Proposition 2 claims that this update can increase the effective rank of the representation matrix, when satisfies
ač.mm = (I H H, H therefore, Eq.910) can be rewritten as H, = H + @Cat.. (H.) OH
1+ (1 maxx Is > O. Note that no matter whether max > 0 or not, if is sufficiently close to 0, the condition will be satisfied. Under this situation, the update will alleviate the dimensional collapse.
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of ContraNorm by experiments including 1) lan- guage understanding tasks on GLUE datasets with BERT and ALBERT (Lan et al.) (020) as the backbones; 2) image classification task on ImageNet100 and ImageNetlk datasets with DeiT as the base model; 3) fully supervised graph node classification task on popular graph datasets with GCN as the backbone. Moreover, we conduct ablative studies on ContraNorm variants comparison and hyperparameters sensitivity analysis.
4.1 EXPERIMENTS ON LANGUAGE MODELS
To corroborate the potential of ContraNorm, we integrate it into two transformer structured models: BERT and ALBERT, and evaluate it GLUE datasets. GLUE includes three categories of tasks: (i) single-sentence tasks CoLA and SST-2: (ii) similarity and paraphrase tasks MRPC, QQP, and STS-B; (iii) inference tasks MNLI, QNLI, and RTE. For MNLI task, we experiment on both the matched (MNLI-m) and mismatched (MNLI-mm) versions.
Setup. We default plug ContraNorm after the self-attention module and residual connection (more position choices are in Appendix E). We use a batch size of 32 and fine-tune for 5 epochs over the data for all GLUE tasks. For each task, we select the best scale factor in (q among (0.005, 0.01, 0.05, 0.1, 0.2). We use base models (BERT-base and ALBERT-base) of 12 stacked blocks with hyperparameters fixed for all tasks: number of hidden size 128, number of attention heads 12, maximum sequence length 384. We use Adam (Kingma & Ba 2014) optimizer with the learning rate of 2e- - 5.
Results. As shown in Table 1 ContraNorm substantially improves results on all datasets compared with vanilla BERT. Specifically, our ContraNorm improves the previous average performance from 82.59% to 83.39% for BERT backbone and from 33.74% to 84.67% for ALBERT backbone. We also submit our trained model to GLUE benchmark leaderboard and the results can be found in Appendix G It is observed that BERT with ContraNorm also outperforms vanilla model across all datasets. To verify the de-oversmoothing effect of ContraNorm. We build models with/ aith/without ContraNorm on various layer depth settings. The performance comparison is shown in Fig. Bal Constant stack of blocks causes obvious deterioration in vanilla BERT, while BERT with ContraNorm maintains competitive advantage. Moreover, for deep models, we also show the tendency of variance and effective rank in Fig.3bland collapse and dimensional collapse, respectively.
3bland Fig. 3c which verifies the power of ContraNorm in alleviating complete
4.2 EXPERIMENTS ON VISION TRANSFORMERS
We also validate effectiveness of ContraNorm in computer vision field. We choose DeiT as back- bone and models are trained from scratch. Experiments with different depth settings are evaluated on ImageNet100 and ImageNetlk datasets. Based on the Timm (Wightman) 2019) and DeiT repos- itories, we insert ContraNorm into base model intermediately following self-attention module.
Setup. We follow the training recipe of Touvron et al. (202T), and make minimal changes to hy- perparameters. Specifically, we use Adam (Loshchilov & Hutter 2019) optimizer with cosine learning rate decay. We train each model for 150 epochs and the batch size set 1024. Aug-
7

---

Published as a conference paper at ICLR 2023
matthows correaation
varienca
221
Sonem Combullam
No norm
ContraNorm
20
Contruhorm
layer number
(a) Deep Model Performance
(b) Variance
(c) Effective rank
Figure
Comparison of BERT and BERT+ BERT+CCotraNorm on COLA with various layer depths.
Fig. 3a presents performances of models with block number=12, 12, 24, 36, 48, 60, 72. Fig. 3b and
Fig.
show the tendency of features variance and effective rank for models with 72 blocks.
Table I: Results comparison on validation of GLUE tasks. Following (Devlin al. 2018
report scores for QQP and MRPC, Spearman correlations for STS-B, and accuracy scores for the other tasks. ContraNorm* denotes the best model when varying plugging positions of ContraNorm
Avg denotes the average performance all the tasks and bold denotes the best performance.
Dutaset
COLA
SST-2
MRPC
QQP
STS-B
MNLI-m
MNLI-mm
QNLI
RTE
Avg
BERT-base
55.28
92.89
88.96
88.24
88.61
84.65
84.61
91.51
68.59
82.59
BERT* + ContraNorm
58.83
93.12
89.49
88.30
88.66
84.87
84.66
91.78
70.76
83.39
BERT ContruNorm*
59.57
93.23
89.97
88.30
88.93
84,87
84.67
91.78
70.76
83.54
ALBERT-base
57.35
93.69
92.09
87.23
90.54
84,56
84.37
90.90
76.53
83.74
ALBERT ContraNorm
58.51
92.89
92.86
87.45
90.56
84.33
84.62
91.76
79.06
84.67
ALBERT ContraNorm*
58.76
93.23
92.89
87.67
90.72
84.69
84.95
92,28
79.06
84.92
Table 2:
Test accuracy (%) comparison results. For ImageNet100 and ImageNet we use DeïT-
tiny and DeiT-small as the baseline respectively. The block number is set to 12, 16 and 24. The best result for each dataset bolded.
Dataset
Model
#L=12
#==16
#L=24
DeiT-tiny
76.58
75.34
76.76
ImageNet100
CContraNorm
79.34
80.44
81.28
DeiT-small 77.32
78.25
77.69
ImageNet1k
+ContraNorm
77.80
79.04
78.67
mentation techniques are used to boost the performance. For all experiments, the image size is set be 224 224. For Imagenet100, set the scale factor to 0.2 for all layer depth settings. For Imagenetlk, we set s to 0.2 for models with and 16 blocks, and raise it to 0.3 for models with 24 blocks.
Results. In Table DeiT with ContraNorm outperforms vanilla DeiT with all layer settings both datasets. Typically, our method shows a gain of nearly 5% on test accuracy for ImageNet100. For ImageNetlk, we boost the performance of DeiT with 24 blocks from 77.69% to 78.67%.
4.3 EXPERIMENTS ON GRAPH NEURAL NETWORKS
We implement ContraNorm simple normalization layer after each graph convolution of GCN, and evaluate it on fully supervised graph node classification task. For datasets, we choose two popular citation networks Cora (McCallum et al.) 2000) and Citeseer (Giles et al. 1998), and Wikipedia article networks Chameleon and Squirrel (Rozemberezki al.) 2021p. More informa- tion of datasets is deferred to Appendix H We compare ContraNorm against a popular normaliza- tion layer PairNorm IZhao & Akoglu 2020) designed for preventing oversmoothing. We also take LayerNorm as baseline by setting the scale
Setup. We follow data split setting in Kipf & Welling (017) with train/validationfeest splits of 60%, 20%, 20%, respectively. To keep fair comparison, we fix the hidden dimension to 32, and dropout rate to 0.6 as reported in Zhao & Akoglu (2020). We choose the best of scale controller s in range of (0.2, 0.5. 0.8, 1.0) for both PairNorm and ContraNorm. For PainNorm, we choose the best variant presented by /Zhao& & Akoglu (2020).
Results. shown in Table in shallow layers (e.g., two layer), the addition of ContraNorm reduces the accuracy of vanilla GCN by a small margin, while it helps prevent the performance from sharply deteriorating as the layer goes deeper. ContraNorm outperforms PairNorm and Lay-
8

---

Published as a conference paper at ICLR 2023
Table 3: Test accuracy (%) comparison results. We use GCN as backbone and apply LayerNorm, PairNorm and ContraNorm, respectively. We fairly tune the scale parameter in LayerNorm and ContraNorm. The layer number is set to 2, 4. 8, 16, 32. For every layer setting, the best accuracy is marked in blue background, and the second best is underlined. Results are averaged over runs.
Datasel
Model
#L=2
#L=4
#L=8
#L=16
#L=32
Vanilla GCN
81.75 + 0.51
72.61 + 2.42
17.71
+ 6.89
20.71 = 8.54
19.69 9.54
+LayerNorm
79.96 1:966000 0.73
77.45 0.67
39.09 1 4.68
7.79 + 0.00
7.79 + 0.00
Cora
+PairNorm
75.32
1.05
72.64 2.67
71.86
+ 3.31
54.11 = 9.49
36.62
2.73
+ContraNorm
79.75 0.33
77.02 + 0.96
74.01
+ 0.64
68.75 = 2.10
2,46 46.39
Vanilla GCN
69.18 0.34
55.01 4.36
19.65 + 0.00
19.65 = 0.00
19.65
0.00
+1 -LayerNorm
63.27 + 1.15
60.91 + 0.76
33.74 6.15
19.65 - 0.00
19.65
0.00
CiteSeer
+PairNorm
61.59 + 1.35
53.01 + 2.19
55.76 + 4.45
44.21 1.73
36.68 2.55
ConttaNorm
60.55 + 0.72
59.30 + 0.67
49.01 3.49
1.70 36.94
Vanilla GCN
45.79
1.20
37.85 1.35
22.37 + 0.00
22.37
0.00
23.37
0.00
+LayerNorm
63,95
1.29
55.79 1.25
34.08 + 2.62
22.37 0.00
22.37
0.00
Chameleon
+PairNorm
62.24 + 1.73
58.38
1.48 58.73
49.12 2.32
37.54 = 1.70
30.66
1.58
+ContraNorm
64.78 + 1.68
1.12
48.99 + 1.52
40.92 = 1.74
3.16 35.44
Vanilla GCN
29.47 0.96
19.31
0.00
19.31
0.00
19.31 = 0.00
19.31
0.00
Squirrel
+LayerNorm
43.04
1.25
5.50
19.63 + 0.45
19.96 = 0.44
19.40
0.19
+PairNorm
43.86 13.86:0.44
40.25
0.55
36.03 + 1.43
29.55 = 2.19
29.05
+ContraNorm
47.24 0.66
40.31 + 0.74
35.85
1.58
32.37 0.93
27.80 0.72
Table 4: Performance comparison among different variants of ContraNorm. SG, LN, LyN the abbreviations of stop gradient, layer normalization and L2 normalization, respectively. All exper- iments are conducted on GLUE tasks with the same parameter settings. Avg denotes the average performance on all the tasks. We bold the best result for each task.
Variants
Datasets
Avg
LN L
COLA
SST-2
MRPC
QQP
STS-B
MNLI-m
MNLI-mm
QNLI
RTE
58.80
93.12
89.60
88.35
88.97
84.81
84.67
91.47
68.23
83.11
59.82
93.00
89.64
88.37
88.92
84.72
84.58
91.58
68.95
83,29
59.57
93.12
89.97
88.30
88.93
84.84
84.67
91.58
68.95
83.33
erNorm in the power of de-oversmoothing. Here, show the staple in diluting oversmoothing is ContraNorm, and LayerNorm alone fails to prevent GCN from oversmoothing, even amplifying the negative effect on Cora with more than 16 layers.
4.4 ABLATION STUDY
Comparison of different ContraNorm variants. Recalling Section 3.3 with stop-gradient technique (SG) by masking the second term. For solving data representation in- stability, apply layer normalization (LN) to the original version, while for the convenience of theoretical analysis, layer normalization is replaced by L2 normalization (L2N). Here, we inves- tigate the effect of these tricks and the results are shown in Table 4 Compared with the variant with only LayerNorm, ContraNorm with both stop gradient and layer normalization presents better performance. As for the two normalization methods, they are almost comparable to our methods, which verifies the applicability of our theoretical analysis.
we improve ContraNorm
Hyperparameters sensitivity analysis. We conduct an ablation study regarding the gains of Con- traNorm while using different values of scale factor s. In Appendix [ robust in an appropriate range of
we show that ContraNorm is
5 CONCLUSION
In this paper, we point out the deficiencies of current metrics in characterizing over-smoothing, and highlight the importance of taking dimensional collapse the entry point for oversmoothing analy- sis. Inspired by the uniformity loss contrastive learning. we design an coamiizationinnduee nor- malization layer ContraNorm. Our theoretical findings indicate ContraNorm mitigates dimensional collapse successfully by reducing variance and effective rank of representations. Experiments show that ContraNorm boosts the performance of Transformers and GNNs on various tasks. Our work provides a new perspective from contrastive learning on solving the oversmoothing problem, which helps motivate the following extensive research.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
Yisen Wang is partially supported by the National Key R&D Program of China (20222D0160304) 60304), the National Natural Science Foundation of China (62006153). Open Research Projects of Zhejiang Lab (No. 2022RCOAB0S), and Huawei Technologies Inc. Xiaojun Guo thanks for Hanqi Yan for her kind guidance on the implementation of BERT models. We thank anonymous reviewers for their constructive advice this work.
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, al. Xeit: Cross-covariance image transformers. In NeurlPS, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning re- cover global and local spectral embedding methods. arXiv preprint arXiv:2205.11500 2022.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020.
Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-induceed graph implicit nonlinear diffusion. In ICML, 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
Yihe Dong, Jean- Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. ICML, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x scale. In ICLR, 2021.
words: Transformers for image recognition at
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In ICML, 2021.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geom- etry of bert, elmo, and gpt-2 embeddings. arXiv preprint 1rXi::1909.0000 2019.
William Fulton. Eigenvalues, invariant factors, highest weights, and schubert calculus. Bulletin of the American Mathematical Society, 37(3):209-2499 2000.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In ICLR, 2019.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality be- tween contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:220..02574 2022.
Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In DL, 1998.
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Vision transformers with patch diversification. arXiv preprint X1V::20114.175:: 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bermardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent-a approach to self-supervised leaminn. In NeurIPS, 2020.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In ICCV, 2021.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In CVPR, 2018.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardiza- tion towards efficient whitening. In CVPR, 2019.
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self- supervised learning, arXiv preprint arXiv:2111.00743 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint [tfi:::412..9900 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self supervised learning of language representations. In ICLR, 2020.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, 2018
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 190n::997.11 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In /CCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the
construction of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv: 1806. 03822, 2018.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In EUSIPCO, 2007.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding, Journal of Complex Networks, 9(2):cnab014, 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. In IJCV. 2015.
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):70-7710 2020.
Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen M. S. Lee, and James Kwok. Revisiting over-smoothing in bert from the perspective of graph. In ICLR, 2022.
Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening and coloring batch transform for gans. arXiv preprint arXiv: 1806. 00420, 2018.

---

Published as a conference paper at ICLR 2023
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
Ychui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, and Yunhe Wang. Augmented shortcuts for vision transformers. In NeurlPS, 2021.
Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via unified gradient framework. In CVPR, 2022.
Hugo Touvron,
Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In
ICLR, 2019.
Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory practice. arXiv preprint arXiv:2203.05962. 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICML, 2020.
Ross Wightman. Pytorch image models.
https://githin wttp:/..co///w//iiiaia.o Beanm/wwightnnn wtt..//wwwwaaaa/a/
gyttrrch-mmge-mmoon
2019.
Xingyu Xie, Qiuhao Wang. Zenan Ling, Xia Li, Yisen Wang, Guangean Liu, and Zhouchen Lin. Optimization induced equilibrium networks. arXiv preprint 1rXiv22105.13228, 2021.
Hanqi Yan, Lin Gui, Wenjie Li, and Yulan He. Addressing token uniformity in transformers via singular value transformation. In UAI, 2022.
Yongyi Yang, Tang Liu, Yangkun Wang, Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, and David Wipf. Graph neural networks inspired by classical iterative algo- rithms. In ICMI., 2021.
Yongyi Yang, Zengfeng Huang, and David Wipf. Transformers from an optimization perspective. arXiv preprint arXiv:2205. 13891 2022.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure I Leskovec. Graph convolutional neural networks for web-scale recommender systems. In ACM SIGKDD, 2018.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In ICLR, 2020.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 11886. 2021.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changeheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. Al open, 1:57-81, 2020.
Meiqi Zhu, Xiao Wang, Chuan Shi. Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with optimization framework. In WWW, 2021.
12

---

Published as a conference paper at ICLR 2023
A ILLUSTRATION ON THE STOP-GRADIENT OPERATION IN EQ.
In this section, we will illustrate the stop-gradient operation in Eq by using the framework pro- posed by (Tao et al. 2022). The original update form should be Eq.
H. H, aLuniform H, S/T (D A AD H).
HH,
We take SinCLR(Chen et al. 2020) as the contrastive learning framework. Tao et al. (2022) have studied the stop-gradien form of SimCLR and illustrated that the stop-gradient operation will make
similar performance with the original one. Based on this, we will elaborate on how AD is removed in the following part. In fact, we can directly illustrate how the second term in Eq can be omitted.
ahunfform Oh.
exp(h, h2ghT) Ex exp(h) hx/T)
h2/T T Σ exp(h, 1//TT hg/T. En exp(h) hx/T)
In SimCLR, by denoting the normalized features from the online branch as ui ......... the normalized features from the target branch (although the two branches have no differences in SimCLR) are uf, 1,2 2,,,..., nt and V = {uf, ub un uf. uu........................ "n the SimCIR loss can be represented as
exp(4 uT/T) Lee//u. exp(u u/T)
L L
Σ
log
2n
E
EVV
where μ and w/ are positive pairs and r is the temperature. Then the gradient of on uf can be calculated as
OL
+
Σ
8ill
uj +
Σ 1
Oug
2nn
2rn
VEV/uf
vEV/af
where
exp(ui V/T)
Everyng exp(uf v'/T)
the softmax results over similarities between uf and other samples, and
exp(v T u//T)
tu Evev/o exp(uT V'/T)
computed over similarities between sample v and its contrastive samples V/v. We can see that the first term of 0L/ôuţ comes from the part which takes uj as the anchor, and the second term comes from the part which takes the other feature the anchor. TTo al. (2022) proposes to stop the second term and verifies that stopping the second gradient term will not affect the performance empirically.
Note that the in term in the gradient is from the alignment loss. So the gradient of the uniformity loss on uf can be written as
Lumiform
1 27m
Σ VEV/up
8
Σ L0
(11)
Oug
2rm
"EV/ug
is noteworthy that by writing V {h1, hig..... hy Eqi4 shares the same form as Eq(ll) By
4
adopting the stop-gradiem method just as (Tao et al. 2022 Eq which is just the AD- term in Eq
takes, we remove the second term in
Empirically, we draw the singular value distribution of embeddings for vanilla BERT and +Contra- Norm with only AD-1 term or D-1 A term on RTE task. As shown in Fig. 4 compared with vanilla BERT with long-tail distribution (dimensional collapse). adding ContraNorm with D- A and AD- both reduce the number of insignificant (nearly zero) singular values and make a more balanced distribution. The similar singular value distributions mean that they play similar role in alleviating dimensional collapse.
13

---

Published as a conference paper at ICLR 2023
BERT
ContraNorm with D 'A
ContraNorm with AD -1
number
140
singular value
Figure 4: Singular value distributions with ContraNorm using only AD-1 term or D-1 A term. Experiments are conducted with 12-layer BERT on RTE task.
B PROOFS FOR PROPOSITIONS IN SECTION 4.3
We give out two lemmas.
Lemma B.1. Denote
[1,1,... 1 /n. For any update form X1 PX, and X > O, if the eigenvalues of (I - ee T) XP (I ce T) are all not larger than zero, then we have Var(X,) 2 X-1 Var(Xo).
Proof. We denote Xo = Y ddiag(w)...... Wn y for the eigen-decomposition of X 0 where Y 1811 V2...., ynl is the orthogonal basis and all wi 2 O. Note that (I ce =e ) = (I ee )) Therefore,
Var = |[I- - ee X
=Ir(1(T ee X (I (I-ee
tr((II ee Jo diiag(w)...,.u,))
e)
tr(diig((1) ..... wi) Y (I ee [[ ee ) tr(diag (w), wn) Y T (I ce \Y)
(12)
Σ way (I ce ')y
Similarly, we have
n Var(X1) Σ wiy P(I ee )
(13)
Therefore, have
Var(Xo) AVar(Xi) Σ
way. {(I (C -PP (I - ee PP
(14)
i=1
Thus, if the cigenvalues of (I ee XPT (I eee P = E are all not larger than zero, Σ is semi-negative definite, then we have
y ((I - ee -P (I- ( $0 0,
(15)
which implies that Var(X,) AVar(X)) < 0. Therefore, Var(X,) 2 --1, Var(X.).
The second lemma is from the (13( in (Fulton 2000
14

---

Published as a conference paper at ICLR 2023
Lemma B.2. Let A, B, C be symmetric matrices and C A + B. Suppose the eigenvalues of A
are 0 2 02 2 2 ans the eigenvalues of are 31 2 32 ... 2 Sn. and the eigenvalues of C
are 71 2 72 2 ... 2 7n. Then we have the inequality
max 04 + Bg mh S min a, Bj- i4j=nth
(16)
+4j=k+1
We can now start to prove Proposition 1
Proposition Let (1, (1. 1) /vn. For attention matrix A softmax(H,,H,, ), let min be the smallest eigenvalue of P (I ee ) A) + A) (I ee For the ContraNorm update H;- ((1 + s)I HA)H,, > 0. have Var (i) 2 (1 s0mia) (ar(H,)
Especially, if anin 2 0, we have Var(H. (H.) ) 2 Var (H)).
Proof. We denote
(I - ee ) X((1 s) sA) T (I ce T) ((1 s) sA). Then,
= (1-1-X1 -1- A)I
A(s(I A) - (e-ee 8 ee )[I A) 5 A) (I ee' ) A))
= (1- A)I AsP - As (I e A).
(17)
Let 01 2 02 2- ... 2 a2 be the cigenvalues of E. Since I - ee' has a eigenvalue of and n
eigenvalues 1, ee a semi-definite positive matrix. Thus, s² A) T)
(I A) is
also a semi-definite positive matrix. Notice that the largest eigenvalue of (1 -)- - sP is (1
ma and the largest eigenvalue of -5 - A) [I- ee ) (I
A) is 0. Therefore, by Lemma B.2
the largest eigenvalue of Σ is less or equal to (1 - X) min- Let À = 1 mim. then the largest
eigenvalue of Σ is less or equal to By Lemma B.1 we have V Yaar H() 2 (1 (1-s0mml
Var Hb).
Moreover, min 0, then (1 min) )-1 z 1, leading to Var (H)) V Var(H.).
Remark. Now we discuss some sufficient conditions for amin 2 0. If P is a diagonally dominant
matrix, then will have the result mii 2 0. Denote agj Aij. bi Osj asj and Q =
(I ee )(I A), where og 1 if i j and 0zg 0f * then we have
Qi = by
(bez).
(18)
If we have Ek=1 akj S 1 + nai for any i.j. then we will have
رزو k== bus 2 by:, i it j.
(19)
Notice that Ek buj O, we have
10pa1= = Σ
Qkyl
(20)
Since À is an attention matrix, we also have
1Qús1 = Σ
Uikl.
(21)
Therefore, we have
|Pal = 2|Q
= - Quyi + +1Σ gjal
Σ
Quj + 2
(22)
kjj
M
Payl
kyl
which indicates that P is a diagonally dominated matrix, thus amin 2 0. Therefore, Σk= akj S
1 + nay for any i. is just a sufficient condition. A special case for this is Σh=1 akj 1,4
We now move on to the proof of Proposition We first give lemma on the property of effective
rank.
15

---

Published as a conference paper at ICLR 2023
Lemma B.3. Let the eigenvalues of AA be A A2 2 An and the eigenvalues of BB" be 0 2 02 2 ... on. a, X; increasing as increases, then we have erank ()) 2 crank(A).
This lemma can be proved just by using the definition of the effective rank. Proposition 2. Consider the update form
H, = (1+ H)H, - (H,H, Hb:
(23)
let max be the largest singular value of Ho. For s >> satisfying + max > have erank( > (rank(Ha)).
Proof. We have
H, = ((1+ HHHHH H
(24)
Therefore,
H.H, = S 2s(1 (H (1 (H
(25)
Suppose A A2 2.... A, are the eigenvalues of H, then H,H? has the same eigenvectors
as HgH, and its eigenvalues are (A,s (1 +)) Since satisfies + (1 - max*) > 0, we
have 1 + > s,s. Therefore. (A,s +)) is increasing as increases, resulting the fact that
erank( H,) > erank(H,) by using Lemma 1ammmaBB.. B.3
C METRICS CALCULATING FEATURE AND ATTENTION MAP COSINE SIMILARITY
Following Wang et al. (2022), given feature map H E Nx and attention map of the h-th head A (1) E IR NxN with batch size N and hidden embedding size D, the feature cosine similarity and the attention cosine similarity is computed
(1)
(N)
2
Σ
h h
simy
simattn
N(N - EJSi
N(N 1)H
|la
8,jii
where h, denotes the i-th row of H. a is the i-th column of A, and is the number of attention heads.
D APPLYING THE UNIFORMITY Loss DIRECTLY
We conduct comparative experiment BERT model with straightforwardly applied uniformity loss and our proposed ContraNorm. Specifically, we the uniformity loss (lossuni) to clas- sification loss (MSELoss, CrossEntropyLoss or BCELoss depending on the task type, denoted by los8ews. Formally, the final loss is written
losStotal lossele + lossuni-
where lossuui = EN log Mn exp (x(, f(x)) and N is the number of samples in the batch. We tune in the range of 00.5, 0.8, 1.0, 1.2, 1.5] and choose the best one in terms of average performance. Other hyperparameters are kept the same as settings of ContraNorm. The results are shown in the T
We can see that +ContraNorm gets the best score in 8 tasks, while +Uniformity loss reaches the best only 1 1 tasks. ContraNorm also has the highest average score among all tasks. The reason is that updating the total loss is a combined process for objectives of correct classification and uniform distribution. Thus, lower losstotal may be only caused by lower classification loss while uniformity loss is kept the same, which cannot ensure more uniform distribution of representations. In contrast, ContraNorm acts directly on representations in each layer and enforces the uniform property.
In fact, there are many methods in GNNs such as Yang et al. (202T) and Zhu et al. (202T), which design the propagation mechanism under the guidance of the corresponding objective. The well- designed propagation mechanism is shown to be the most fundamental part of GNNs (Zhu al.
16

---

Published as a conference paper at ICLR 2023
Table 5: Results comparison validation of GLUE tasks. Following Devlin al. 2018 we report Fl scores for QQP and MRPC, Spearman correlations for STS-B, and accuracy scores for the other tasks. Avg denotes the average performance on all the tasks. For each task, the best performance bolded.
Dataset
COLA
SST-2
MRPC
QQP
STS-B
MNLI-m
MNLI-mm
QNLI
RTE
Avg
BERT-base
55.28
92.89
88.96
88.24
88.61
84,65
84.61
91.51
68.59
82.59
BERT + Uniformity Loss
58.08
93.00
89 46
88. 14
88.69
84.45
84.43
91.60
59
82.94
BERT ComtrNNomm
58.83
93.12
89,49
88.30
88.66
84.87
84,66
91.78
70.76
83,39
Table 6: Results comparison different plugging positions of ContraNorm. Experiments are eval- uated on the validation set of GLUE tasks. Avg denotes the average performance on all the tasks. The best result for each task bolded.
Dataset
COLA
SST-2
MRPC
QQP
STS-B
MNLI-m
MNLI-mm
QNLI
RTE
Avg
BERT-base
55.28
92.89
88.96
88.24
88.61
84.65
84.61
91.51
68.59
82.59
before-residual
59.57
93.12
89.97
88.30
88,93
84.84
84.67
91.58
68.95
83.33
after-residual
58.83
93.12
89.49
88.30
88.66
84.87
84.66
91.78
70.76
83.39
Instead of directly using the loss function, these methods transfer the loss function into specific propagation method and achieve superior performance, which indicates that changing the network may be more effective than directly adding the objective to the loss function.
2021
E
CHOICES OF CONTRANORM PLUGGING POSITION
We explore two ways to integrate ContraNorm into BERT and ALBERT. Concretely, consider the update of H1)) in 1-th block of Transformers
(+) = MultiHeadAttentionn/ (H²)),
(26)
H (1) (1) + X,
(27)
H (1) LayerNorm((H!)) (9),
(28)
where X HG is the input tensor.
We choose positions 1) between F.q.266 and Eq. 27 named before-residual; 2) between Eq. and Eq. 28). named as after-residual. The performance comparison between the two positions on GLUE datasets is shown in Table. (6). It is observed that putting ContraNorm after the residual connection slightly outperforms that before residual. Therefore, we choose the after-residual variant as our basic ContraNorm.
F TIME COMPLEXITY ANALYSIS AND A MODIFIED CONTRANORM
The most time-consuming operation of ContraNorm is the matrix multiplication. Given H Raxd where n and denote the number of samples in batch and feature size respectively, the time complexity of ContraNorm is O(n²d), which is the same order as the self-attention operation in Transformer. Empirically, we report the training time BERT with or without ContraNorm on GLUE tasks Table 7 All experiments are conducted on a single NVIDIA GeForce RTX 3090.
On average, raise the performance of BERT on GLUE tasks from 82.59% to 83.54% (see Table T with less than minutes overhead. We think the time overhead acceptable considering the benefits it brings.
Moreover, to enhance scalability propose modified version of ContraNorm inspired by Ali et al. (20T As already alluded to, ContraNorm resolves dimensional collapse by pushing samples away from each other with the uniformity loss as the energy function. A different choice of the underlying form of normalization layer. Here, to maintain the goal of settling dimensional collapse, we choose another contrastive loss named Barlow Twins (Zbontar et al. 2021
function generates
Lgr Σ
C
Σ
C
(29)
17

---

Published as a conference paper at ICLR 2023
Table 7: Estimated training time of BERT with or without ContraNorm GLUE tasks. All experi-
ments are conducted on a single NVIDIA GeForce RTX 3090. 8 is the abbreviation for second. Avg denotes the average training time across all the tasks.
Dataset
COLA
SST-2
MRPC
QQP
STS-B
MNLI-m (/mm)
QNLI
RTE
Avg
BERT 110s
8518 9838
748
6458s
110s
8186s 8941
2402I
74s
2283s
+ContraNorm
125s
80s
71500
122s
2594s
80s
2509g
(E where is a trading-off constant, b is the batch index, i index the vector dimension, and x xx denotes different augmented views of x. Dislike the sample-wise loss InfoNCE, Barlow Twins decorrelates the dimensions and can be seen as contrastive between the dimensions of the embeddings (Garrido et al.) 2022). Despite seem- ing different, recent works empirically show strong similarities between the two loss functions, and theoretically prove the equivalence between them under limited assumptions (Garrido et al. 2022 Tao et al., 2022; Balestriero & LeCun 2022 Huang et al.) (021). Therefore, the layer generated by unfolding iterations of Barlow Twins is believed to work equally well as ContraNorm. Specifically, we reorder the matrix multiplication from HH" E R to H E RRddd to enforce the channels of embeddings different from each other. Then the modified version of ContraNorm becomes
with C; (
ContraNorm-vv(HH) LayerNorm(HH H -8/T x H softmax H)).
(30)
The transposed alternative has a linear complexity in the number of samples, i.e. O(nd²). In the case of n >> d, the modified version greatly alleviates the computation burden.
To verify whether performs equally well as ContraNorm, we conduct experiments on the validation
set of GLUE tasks. The learning rate of BERT with ContraNorm-v2 is set to 4e - 5 and other experiment setups are the same in Section 4.1 As shown from Table for each task the performance of BERT with ContraNorm-v2 surpasses the vanilla BERT, and the average performance raised from 82.55 92.59% to 84.21%. The results imply effectiveness of this modified version of ContraNorm, which can also be explained with the relationship between Gram matrix (G HH" R=x" and covariance matrix ( H" E pdxd)
(Ai et al. 2021
Table 8: Results comparison of BERT with or without ContraNorm-v2 validation set of GLUE tasks. Following Devlin al. (2018), we report Fl scores for QQP and MRPC, Spearman correla- tions for STS-B, and accuracy scores for the other tasks. Avg denotes the average performance on all the tasks and bold denotes the best performance.
Dataset
COLA
SST-2
MRPC
QQP
STS-B
MNLI-m
MNLI-mm
QNLI
RTE
Avg
BERT
55.28
92.89
88.96
88.24
88.61 89,59
84.65 85,24
84.61
91.51
68.59
82.59
ContraNorm-v2
62.08
93.69
91.78
88.36
85.19
91.95
70.04
84.21
G RESULTS ON TEST SET OF GLUE DATASETS
We submit our trained model to GLUE benchmark leaderboard and the resultant feedback perfor- mance is shown in Table (9.
Table Results comparison on test set of GLUE tasks. Avg denotes the average performance on all the tasks.
Dataset
COLA
SST-2
MRPC
QQP
STS-B
MNLI-m
MNLI-mm
QNLI
RTE
Avg
BFRRThaase
$3.3
92.8
86.8
71.2
82.8
84.4
83.2
90.9
66.3
79.
BERT ContraNorm
54,5
93.0
87.9
71.4
83.0
84,5
83.4
91.0
66.9
79.5
18

---

Published as a conference paper at ICLR 2023
H INTRODUCTION OF GRAPH DATASETS
Citation Network. Cora, CiteSeer are three popular citation graph datasets. In these graphs, nodes represent papers and edges correspond to the citation relationship between two papers. Nodes are classified according to academic topics.
Wikipedia Network. Chameleon and Squirrel are Wikipedia page networks on specific topics, where nodes represent web pages and edges are the mutual links between them. Node features are the bag-of-word. representation of informative nouns. The nodes are classified into four categories according to the number of the average monthly traffic of the page.
Statics of datasets are shown in Table.
(10
Table 10: Graph datasets statics.
Category
Dataset
Nodes
# Edges
# Features
Degree
Classes
Cora
2708
5278
1433
4.90
Citation
CiteSeer
3327
4552
3703
3.77
Chameleon
2277
36101
500
5.0
Wikipedia
Squirrel
5201
217073
2089
154.0
ABLATION STUDY FULL RESULTS
We conduct experiments using BERT+ContraNomm with varying scaling factor GLUE datasets. For each dataset, we vary the normalization scale around the best choice, and other parameters remain consistent. As shown in (5g the results illustrate that model with ContraNorm is robust in appropriate range of normalization scaling factor.
matth 00T5
(a) RTE
(b) CoLA
(c) STS-B
accurscy
(d) MRPC
(e) SST2
(I) QNLI
ScCrroy
SCWIP
scale
scale
(g) MNLI-m
(h) MNLI-mm
(i) QQP
Figure 5: Performance when varying scale factor on GLUE datasets. We choose BERT as the base model. The varying range is interval containing the best choice of Here, fix it to [0.005, 0.1] for tasks.
19


Reviews:
[{"review_id": "wjbN32640_", "reviewer": "Reviewer_Zh7A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses an over smoothing issue of representation in Graph Learning and message-passing like computation unit (attention) as the depth of the models increase. The paper argues that attention maps are not a very good indictors of over smoothing, due to the fact that a feature can have low similarity while having high attendance score. Subsequently, an optimization based normalization unit is proposed to encourage large variance across features and decorrelation of dimensions. Results, especially for image and GCN, show good improvement.\n ", "summary_of_the_review": "The paper motivated the problem attempts to solve, suggested a solution and provided insight into why it would work with theoretical justification. It further showed, empirically, performance improvement as the depth of the model increases. ", "strength_and_weaknesses": "Comments and questions:\n- The paper is very well written with good organization. Its topic of discussion is a well-motivated and focused problem with clear solution which makes it easy to follow. Results, especially for GCN and vision tasks, show good improvement as the depth of models increase. \n- Questions: \n    - In Subsection 3.1 and Figure 2, it was argued that similarity in attention does not imply feature collapse. The argument is based on using attention map, which is a cosine similarity between query and key, as a measure of representation collapse. The question is, what is referred as feature here? I am assuming the value embedding. If so, similarity between attention map naturally does not imply similarity between value pairs. What is new here or am I missing something?\n    - The uniformity term in contrastive learning is between the representation in question and all its negative examples. However, $\\dot{L}_{uniform}$ seem to measure similarity across all representation. Is that the case?  \n     - The normalization layer Eq(7) and Eq(8), are designed as a single gradient step that updates the features in the direction of minimal similarity. Could this be amortized? as opposed to approximating a solution with single step. \n- Questions regarding experiment:\n     -  What average performance in Table 1?\n     -  The average performance for both BERT and ALBERT is reported inconsistently in the **Results** section and **Table 1**. Cloud you correct/clarify?\n     -  Reported results in **Table 1** show tiny improvement. Are these the right models and experimental setup to show case your solution for the GLUE task? \n\n- Corrections:\n    - missing fullstop in the second line form the last, in the second paragraph of the introduction section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kWgbqUwYM_o", "reviewer": "Reviewer_uxeV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the overs-moothing problem of graph neural network (GNN) and Transformers, and proposes to avoid the dimensional collapses in the representation, borrowing the idea from self-supervised learning. It thus proposes a normalization layer ContraNorm, which aims to learn a more uniform distribution in the embedding space that can ensures slighter dimensional collapse, motivated on the uniform loss from self-supervised learning. The proposed normalization layer can be inserted into GNNs and Transformers, and improves these baselines based on the experiments on three different scenarios: ViT for image classification, BERT for natural language understanding, and GNNs for node classifications.", "summary_of_the_review": "This paper is a well-motivated paper with empirical success in experiments, but it misses several important related work for comparison and some claims are not well clarified. In summary, I am slightly negative to this paper. \n\n==================after rebuta==================\n\nI look over the revised paper, and the authors almost address all my concerns. E.g., add the comparison to the missing related works, and re-organize the clarity. I still have concerns on the claims that “In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods”. I believe removing $AD^{-1}$ may improve the performance based on the authors experiments, but it does not correspond to the stop-gradient in contrastive learning (they are different mechanisms, e.g., stop-gradient is essential to prevent collapse in simsiam/BOYL, while I believe removing $AD^{-1}$ is not the essential part in the proposed method.). Even though this concern holds, the revised version is more rigorous than the previous one, and I tend to positive to this paper. I thus raise my score from 5 to 6. ", "strength_and_weaknesses": "**Strengths:**\n\n+ The motivation of the proposed method is clear. It is good this paper try to connect the designed normalization method with the uniformity loss, which provides stronger motivation/foundation for this paper, even though I have concerns for its connections and analyses.  \n\n+ The effectiveness of the proposed method over baseline is empirically verified in several architectures (e.g., GNN, Transformer) on several datasets. \n\n**Weaknesses:** \n\n1.One severe problem of this paper is that it misses several important related work/baselines to compare[1,2,3,4], either in discussion [1,2,3,4]or experiments[1,2]. This paper addresses to design a normalization layer that can be plugged in the network for avoiding the dimensional collapse of representation (in intermediate layer). This idea has been done by the batch whitening methods [1,2,3] (e.g, Decorrelated Batch Normalization (DBN), IterNorm, etal.). Batch whitening, which is a general extent of BN that further decorrelating the axes, can ensure the covariance matrix of the normalized output as Identity (IterNorm can obtain an approximate one). These normalization modules can surely satisfy the requirements these paper aims to do. I noted that this paper cites the work of Hua et al, 2021, which uses Decorrelated Batch Normalization for Self-supervised learning (with further revision using shuffling). This paper should note the exist of Decorrelated Batch Normalization. Indeed, the first work to using whitening for self-supervised learning is [4], where it shows how the main motivations of whitening benefits self-supervised learning.\n\n2.I have concerns on the connections and analyses, which is not rigorous for me. Firstly, this paper removes the $AD^{-1}$ in Eqn.6, and claims that “In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods (He et al., 2020; Grill et al., 2020). By throwing away some terms in the gradient, stop-gradient makes the training process asymmetric and thus avoids representation collapse with less computational overhead. It verifies the feasibility of our discarding operation”. I do not understand how to stop gradients used in SSL can be connected to the removement of $AD^{-1}$, I expect this paper can provide the demonstration or further clarification.  \nSecondly, It is not clear why layerNorm is necessary. Besides, how the layer normalization can be replace with an additional factor (1+s) to rescale H shown in claims “For the convenience of analysis, we replace the layer normalization with an additional factor 1 + s to rescale H”. I think the assumption is too strong.   \nIn summary, the connections between the proposed contraNorm and uniformity loss requires: 1) removing $AD^{-1}$ and 2) add layer normalization, furthermore the propositions for support the connection require the assumption “layer normalization can be replace with an additional factor (1+s) to rescale H”. I personally feel that the connection and analysis are somewhat farfetched. \n\n\n\n**Other minors:**\n\n1)Figure 1 is too similar to the Figure 1 of Hua et al, 2021, I feel it is like a copy at my first glance, even though I noted some slightly differences when I carefully compare Figure 1 of this paper to Figure 1 of Hua et al, 2021. \n\n2)The derivation from Eqn. 3 to Eqn. 4 misses the temperature $\\tau$, $\\tau$ should be shown in a rigorous way or this paper mention it. \n\n3)In page 6. the reference of Eq.(24)? \n\n**References:**\n\n[1] Decorrelated Batch Normalization, CVPR 2018  \n[2] Iterative Normalization: Beyond Standardization towards Efficient Whitening, CVPR 2019  \n[3] Whitening and Coloring transform for GANs. ICLR, 2019  \n[4]Whitening for Self-Supervised Representation Learning, ICML 2021  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ecrfK36JDW", "reviewer": "Reviewer_HpT5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the collapse problem within nodes (tokens) of a sample if I correctly got the idea of it. The authors employ the effective rank to measure the quailty of the feature. Then, they borrow the idea of uniformity loss from the contrastive learning algorithm, and convert the gradient through back-propagation to the forward-propagation as the Contranorm algorithm. Finally in the experiments, the authors provide several results that the Contranorm boost the performance.", "summary_of_the_review": "I did really spend a lot of time understanding this paper, and it is not because the technical details are difficult, it is simply because the paper is low in clarity and representation quality. I suggest the authors carefully revise this paper and submit it to another conference.", "strength_and_weaknesses": "Strength:\n\nThe paper presents a nice idea to convert the gradient to the form of normalization.\n\n\nWeaknesses:\n\nInstinctively, I would like to ask why it is not possible to apply uniformity loss straightforwardly unless it doesn't converge, the authors should provide a comparison between the proposed method and uniformity loss. Since, theoretically, the ContraNorm is equivalent to applying the loss, and appending the ContraNorm to the neural network does increase the computational costs during the inference time.\n\nThe authors did not provide an analysis of the computational costs of the proposed Contranorm, I wonder if the number of tokens grows, it will significantly decrease the training speed.\n\nIn the ImageNet1K experiment, the performance for #L=24 is still lower than that of #L=16, which means the proposed method does not actually address the mentioned over-smoothing problem, despite the effective rank depicted in figures 2 & 3 seems to be much higher than the reference model. Therefore, I do not consider the mechanism of feature collapsing discussed in this paper to be comprehensive or even correct, which challenges the motivation of this study.\n\nI do not understand how the layer norm can be replaced by the factor 1+s in Eq.9, the layer norm should remove the bias which the factor 1+s cannot.\n\nThe clarity of this paper is not acceptable to this conference.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AopFbYyZw2", "reviewer": "Reviewer_Q9GP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the phenomenon of oversmoothing in GNNs and transformers; and presents the ContraNorm layer. The experiments on real-world dataset show the efficacy of this method.", "summary_of_the_review": "please refer to the weakness part. I would ask for some direct evidence and visualization that ContraNorm can solve the collapse issue.", "strength_and_weaknesses": "Strength:\nIt’s a nice method with good motivations and theoretical insights. \nThe paper is generally well written.\nThe experiments show the efficacy of the model. It indeed can improve the performance in general.\n\nweakness\n\n1, I would ask one particular extreme case on any dataset: the authours can train a very deeper model that encounter the collapse issue, and the model performance is not good. However, with the help of ContraNorm layer, the issue can be solved. This may be important to show the efficacy.\n\n2, Eq.(2) uniform loss part:  what’s the relation between symbol i^+ and x_k? seems some typos here.\n\n3, could the authours give more reasons/motivations why directly discarding the second terms in Eq. (6)? And “Empirically, the two terms play a similar role in our method.” for any empirical evidence?\n\n4, After reading the paepr, I feel that the ContraNorm is not necessarilty related to/address the collapse issues. As least, there is not enough evidence now. Some visualization of Fig. 1 in real world dataset and backbone is thus needed to directly show that the collapse issues happened when training the model; and ContraNorm can change the distribution of features. This can be shown on any dataset. Otherwise, we can just take ContraNorm as one typical type of contrast learning strategy to better train the model.\n And particularly, note that ContraNorm is built upon the layer norm.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster