Title: On Separate Normalization in Self-supervised Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Transformer

Self-supervised Learning

Normalization
Abstract: Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average  2.7% performance improvement over the image, natural language, and graph domains.

On Separate Normalization in Self-supervised Transformers

Abstract

Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the class token [CLS] and the tokens. We propose in this paper a new yet simple normalization method that separately normalizes embedding vectors respectively corresponding to normal tokens and the [CLS] token, in order to better capture their distinct characteristics and enhance downstream task performance. Our empirical study shows that the [CLS] embeddings learned with our separate normalization layer better encode the global contextual information and are distributed more uniformly in its anisotropic space. When the conventional normalization layer is replaced with a separate normalization layer, we observe an average 2.7% performance improvement in learning tasks from the image, natural language, and graph domains.

1 Introduction

Transformer models [Vaswani et al., 2017] have revolutionized natural language processing (NLP) [Devlin et al., 2018, Liu et al., 2019] and demonstrated remarkable performances across a wide range of NLP tasks. The significance of transformer models lies in their ability to model context and capture complex linguistic patterns without being constrained by the sequential nature of data. Beyond NLP transformers have further found their successes in areas such as computer vision (CV) [Han et al., 2022], speech recognition [Karmakar et al., 2021], and recommendation systems [Sun et al., 2019, Gu et al., 2020, Wu et al., 2020]. Their flexible architecture and ability to capture dependencies have made them adaptable to diverse data modalities in these domains.

LN

CLS

Tokens

ShareNorm

BN

CLS

LN

Tokens

Transformer architectures have been studied extensively from various perspectives such as attention mechanisms, positional encoding [Devlin et al., 2018], and normalization techniques. Specifically, layer normalization [Ba et al., 2016] and batch normalization [Ioffe and Szegedy, 2015] are employed to enhance stability and speed up convergence during training. The literature on transformers also explores parameter initialization [Xu et al., 2019a], optimization algorithms [Huang et al., 2020], regularization techniques [Steiner et al., 2021, Zhou et al., 2020], and improved architectures [Han et al., 2021]. This collective research has advanced transformer architectures and their applications in NLP, CV, and other learning domains.

The study of normalization in transformer architectures is motivated by several factors [Xiong et al., 2020, Shen et al., 2020, Nguyen and Salazar, 2019]. For example, Xiong et al. [2020] emphasize the importance of the warm-up of the learning rate and the position of layer normalization layers for the purpose of stable training and faster convergence. Shen et al. [2020] investigates the disadvantage of using batch normalization in transformers and proposes power normalization. While most previous works focus on how the normalization layer can be modified to stabilize the training process, it is less understood how the normalization affects the encoding abilities of these embeddings.

In self-supervised transformers, the [CLS] symbol is frequently used as a global representation for various downstream tasks [Devlin et al., 2018, He et al., 2022]. Often, the normalization applied to the [CLS] symbol is shared with the rest of the tokens in the sequence, which we term it as Shared Normalization (ShareNorm). Given that the [CLS] symbol plays a special role in representation learning, a natural question is whether we should treat it separately in the normalization operation. Driven by this question, our research first scrutinizes the behavior of the current shared normalization in transformers, particularly the properties of the [CLS] embedding and its influence on downstream task performance. Subsequently, we propose a replacement of ShareNorm with Separate Normalization (SepNorm), the latter of which employs distinct normalization operations for the [CLS] symbol and the token features, as depicted in Figure 1. Through extensive analysis, we demonstrate that [CLS] embeddings learned using ShareNorm have the issue of dimensional collapse, which cannot be rectified even by enforcing uniformity [Wang and Isola, 2020]. However, the straightforward substitution of SepNorm for ShareNorm substantially mitigates this issue. We empirically validate the effectiveness of SepNorm in tasks from the image, text, and graph domains, demonstrating the universal advantage of the proposed SepNorm.

2 Background

2.1 Pretraining Transformers with the [CLS] symbol

Unsupervised pretraining of a transformer-based model [Vaswani et al., 2017] is widely investigated in many domains, including NLP, computer vision (CV), and graphs.

Pretraining BERT for NLP. In NLP, Devlin et al. [2018] first develop the BERT model by pretraining a transformer-based network by performing the following two tasks - masked language modeling and next sentence prediction. During pretraining, BERT takes a pair of sentences ( x , y ) ,

SepNorm

which are represented as a special sequence

Here [SEP] is a special token that separates the two sentences. A fraction (e.g., 15%) of the tokens in x and y are randomly replaced by a special symbol [MASK] . The first task in BERT is to predict the original tokens replaced by [MASK] with cross-entropy loss. The second task is to predict whether y is the next sentence following x , and the decision is made by classifying the final embedding of the [CLS] symbol. After pretraining, the representation in the [CLS] is usually used for sentence-level downstream tasks such as sentiment analysis [Medhat et al., 2014].

Pretraining MAE for CV. The Vision Transformer (ViT) [Dosovitskiy et al., 2020] applies the transformers to computer vision tasks. In ViT, an image is usually voxelized into 16 × 16 patches, which are then flattened into a sequence of 256 tokens and fed into the ViT. He et al. [2022] proposes a self-supervised training scheme, Masked Autoencoder (MAE), for the ViT architecture. A training image has 75% of its patches masked. The MAE feeds Tokens of unmasked patches as well as a [CLS] token into the encoder and gets the representations for these tokens. Then the decoder tries to reconstruct the original image by minimizing the mean square error (MSE). Only the encoder will be used for downstream tasks after pretraining. The [CLS] symbol is treated as the class token for linear probing and fine-tuning in the downstream tasks.

Pretraining Graphormer for molecule discovery. Graphormer [Ying et al., 2021] is a transformerbased model designed for graph representation learning tasks. It is used to predict the property of a graph rather than a node or edge. Specifically, Graphormer introduces a new symbol [VNode] as a node connecting to all original graph nodes. Then the vector learned for [VNode] represents the global information of the entire graph. The mechanism of [VNode] is similar to the [CLS] symbol in BERT and MAE.

In typical applications of transformers, the [CLS] symbol is not a natural data token. It summarizes other tokens to capture global information, which is especially useful in downstream tasks. For these reasons, we argue that it should be treated differently in normalization operations.

2.2 Normalization Layers in Transformers

Given that transformers are initially proposed for NLP tasks, layer normalization (LN) Ba et al. [2016] is typically the normalization method of choice [Xiong et al., 2020]. LN normalizes across feature dimensions and is independent of the sequence length and the batch size. For any features h ∈ R d , the LN has the following computation:

Here h i is the i -th dimension of h , ⊙ represents element-wise multiplication, and γ , β ∈ R d are scale and bias parameters, respectively. In a transformer, all tokens, including special tokens, such as [CLS] and [SEP] , are all treated equally and share the same LNs.

Batch Normalization (BN) [Ioffe and Szegedy, 2015] works by normalizing the input data to have zero mean and unit variance along the batch dimension, followed by an affine transformation to scale the result using gamma and beta parameters. BN normalizes a given vector h as:

Here µ B , σ 2 B ∈ R d are the running statistics (mean and variance) maintained by the BN. The running mean and variance are updated during training after each batch. They are usually calculated as an exponential moving average of the batch mean and variance. BN is widely adopted in CV but leads to significant performance degradation when naively used in NLP.

2.3 Uniformity of the Learned Representations

The dimensional collapse in self-supervised representation learning is a common phenomenon where the embedding vectors only span a lower-dimensional subspace [Jing et al., 2021] of the entire vector

space. This means that the model fails to capture data patterns with full power and instead collapses to a simpler representation. Contrastive methods [Oord et al., 2018, Chen et al., 2020b] have been one of the standard approaches to address this problem. Specifically, Wang and Isola [2020] propose the uniformity metric (loss) to quantify the degree of dimensional collapse. Given a set of representation vectors { h 1 , . . . , h N } from a dataset of size N , the uniformity metric L U is computed as follows:

If the distribution of the representation is perfectly uniform, then the numerical value of L U will converge to -4 as the dimension of h increases to infinity [Wang and Isola, 2020].

In self-supervised transformers, the uniformity of the representation is also taken into consideration by some works. For example, Gao et al. [2021] finetune the pretrained BERT model using the InfoNCE loss [Oord et al., 2018], and Zhang et al. [2022] jointly train the MAE loss along with uniformity loss.

3 Approach

3.1 Separate Normalization

We present SepNorm, a normalization scheme that separately normalizes embeddings of the [CLS] symbol and embeddings of other tokens. In this work, we focus on the exploration of combinations of BN and LN for the two separate normalization channels.

For instance, if we apply BN to the [CLS] symbol and LN to other tokens, the learnable parameters are structured as g 1 = ( γ 1 , β 1 ) and g 2 = ( γ 2 , β 2 ) . Let H ∈ R L × d represent the feature sequence, where L denotes the sequence length, and d is the feature dimension. Assume embedding H 0 in the first position corresponds to the [CLS] symbol. The normalization process is as follows:

where H ′ denotes the normalized features. We can also run separate normalization with one of the three other combinations:

Separate normalization allows the [CLS] features to be encoded distinctly from other tokens.

As a comparison, the [CLS] token's embedding and other tokens' embeddings interfere with each other in a shared normalization structure. With ShareNorm, the update directions of the LN parameters { γ , β } are primarily driven by the embeddings of normal tokens. Below is the gradient calculation for these parameters,

We see the summation in the gradient calculation is dominated by normal tokens given that the number of normal tokens is typically a large number. Given the potentially diverse characteristics (i.e., mean and scale) of feature distributions, it might be challenging for normalization parameters to accommodate both token types simultaneously. Moreover, mapping two types of token features into the same sphere may also mix the signal of [CLS] tokens with other tokens. Figure 2(a, b) demonstrates this phenomenon in the scenario where both token types utilize a ShareNorm and how using SepNorm mitigates this effect.

(a) Shared Normalization (ShareNorm)

(b) Separate Normalization (SepNorm)

3

-2-i

0

1

-2-1

-32i0i2332i0123

3.2 Encourage the Uniformity of the [CLS] Embeddings via a Constrastive Term

We further relate SepNorm with the uniformity of embeddings. Higher uniformity values indicate that embeddings can better exploit the space to store information. Contrastive methods often employ negative instances to encourage uniformity. In particular, we incorporate SepNorm into transformers trained with U-MAE [Zhang et al., 2022], which uses a constrastive term to promote uniformity of features.

The U-MAE explicitly adds a uniformity loss term L unif to the training objective to encourage uniformity of [CLS] embeddings.

Here L MAE is the MAE training objective. The two indices i and j represent two sequences within the same batch. [CLS] embeddings h CLS ,i and h CLS ,j , which are respectively for the two sequences, are obtained from our SepNorm during the transformer calculation. By minimizing L unif , [CLS] features tend to be different from each other.

4 Experiments

We examine the effectiveness of the proposed SepNorm component in three domains: CV, NLP, and graphs. We then further investigate how the ShareNorm and SepNorm affect the uniformity of the [CLS] embeddings.

4.1 Computer Vision

Datasets. We investigate the model performance on the four image datasets: STL10 [Coates et al., 2011], FGVC Aircraft [Maji et al., 2013], Street View House Numbers (SVHN) [Netzer et al., 2011], and Oxford 102 Flowers [Nilsback and Zisserman, 2008]. All four datasets are for classification tasks. We follow the train/test split provided in the papers introducing the datasets. We report top-1 and top-5 accuracy for all datasets.

Vision transformers (ViT) and MAE. We choose Vision Transformer (ViT) [Dosovitskiy et al., 2020] as our feature extractor for all datasets. To pretrain the ViT, we adopt the MAE training scheme [He et al., 2022]. We follow MAE and use a 75% masking ratio on input image. During the downstream tasks, we use the embeddings of the [CLS] token to predict the class labels.

Experiment setup. We follow the setup in He et al. [2022] to pretrain and evaluate the ViT. For pertaining, we train the ViT for 4000 epochs. For linear probing, we freeze the encoder's weight and train the last layer on the specific datasets for 2000 epochs. We use a batch size of 512 for pretraining and a batch size of 128 for linear probing.

ShareNorm(LN)

B

8

6

Experiment results. The results presented in Table 1 demonstrate the performances of our model and the baseline model. Our model consistently outperforms the baseline across multiple datasets, indicating its effectiveness in image classification tasks. In the STL-10 dataset, our approach achieves the top-1 accuracy of 93.84% and the top-5 accuracy of 99.7%, higher than the baseline's respective accuracies of 92.01% and 99.5%. Similar improvements are observed in the Aircraft, SVHN, and Flower datasets, where our model consistently outperforms the baseline in both top-1 and top-5 accuracies. These results demonstrate the effectiveness of SepNorm in enhancing image classification performance. We also visualize the embeddings of ShareNorm and SepNorm using t-SNE in Figure 3. Compared with ShareNorm, SepNorm provides embeddings that have better separation among different classes.

4.2 Natural Language Processing

Datasets. We evaluated our approach using the STS dataset, which comprises seven semantic textual similarity (STS) tasks. These tasks, including STS 2012-2016 [Agirre et al., 2012, 2013, 2014, 2015, 2016], STS Benchmark [Cer et al., 2017], and SICK-Relatedness [Marelli et al., 2014]. We also evaluate our method on multiple transfer tasks, including MR [Pang and Lee, 2005], CR [Hu and Liu, 2004], SUBJ [Pang and Lee, 2004], MPQA [Wiebe et al., 2005], SST-2 [Socher et al., 2013], TREC Voorhees and Tice [2000], and MRPC [Dolan and Brockett, 2005]. Following the evaluation settings of SimCSE [Gao et al., 2021], we use Spearman's correlation coefficient as the evaluation metric.

BERT and RoBERTa. We conduct our study with pretrained checkpoints of BERT (uncased) [Devlin et al., 2018] and RoBERTa (cased) [Liu et al., 2019], instead of training them from scratch. Using pretrained models is common in this research field [Gao et al., 2021] because the findings are compatible with the common practice of finetuning pretrained models in actual learning tasks. This strategy also saves significant training time and computational resources, allowing us to extend the study to more learning tasks.

Experiment setup. We follow the experiment setup in Gao et al. [2021] and further finetune the BERT and RoBERTa models on English Wikipedia. We evaluate the models using established STS tasks and employ standard evaluation metrics such as Spearman's correlation.

Experiment results. The experiment results presented in Table 2 highlight the performance of our model compared to the SimCSE baseline on NLP tasks. With the SepNorm layer, BERTbase and RoBERTabase achieve overall higher average accuracy compared to ShareNorm's average accuracy. Only in the transfer learning tasks, SepNorm works slightly worse than ShareNorm in BERTbase, but the difference is marginal.

4.3 Prediction of Molecule Properties

Datasets. We conducted experiments using the ZINC dataset [Irwin and Shoichet, 2005], which contains approximately 250,000 molecular graphs. The task is to predict the properties of molecules from their graphs. We use a subset of 12,000 molecular graphs, as recommended by the benchmarking methodology outlined in [Dwivedi et al., 2020], so that our results are comparable with other studies. Despite being smaller, the subset retained sufficient diversity and complexity for effective evaluation. We also the MolHIV dataset from the OGB [Hu et al., 2020a] collection, which is widely used for training and evaluating graph-based models in molecular property prediction tasks.

Graphormer. We use Graphormer [Ying et al., 2021] as the transformer backbone to construct the predicting model. To obtain graph-level information, Graphormer adds a special node [VNode] to the graph and connects it to all normal graph nodes. The embedding of [VNode] is a summary of the entire graph and will be used in downstream classification tasks. The special node [VNode] serves the same purpose as the [CLS] token in traditional Transformer models. Graphormer has used three encodings to enhance the transformer's learning ability: centrality encoding captures node importance, spatial encoding considers spatial relations, and edge encoding incorporates edge features.

Experiment setup. We strictly follow Graphormer [Ying et al., 2021] in terms of the model architecture, hyperparameters, and training strategies. We replaced the ShareNorm in Graphormer with SepNorm to investigate the effectiveness of the proposed component. We evaluate the pretrained model on a broad class of graph-level prediction tasks. We report the mean absolute error for

the ZINC and ZINC (subset) datasets and the area under the curve (AUC) for the MolHIV dataset.

0

1k

2k

3k

4k

Training epochs

0.49

0.51

0.53

0.55

0.57

Reconstruction loss (MSE)

LN

BN+LN

LN+LN

0

20

40

60

80

100

The index of sigular value

3.0

3.5

4.0

4.5

5.0

5.5

6.0

6.5

Log of singular value

LN

BN+LN

LN+LN

0

20

40

60

80

100

The index of sigular value

4.5

5.0

5.5

6.0

Log of singular value

LN

BN+LN

LN+LN

(a) Pretraining loss

(b)

[CLS]

embeddings

(c) normal token embeddings

Experiment results. Table 3 shows the performances of our model and the Graphormer baseline. For the ZINC datasets, Graphormer with SepNorm achieves a significantly lower mean absolute error compared to that with ShareNorm. On the MolHIV dataset, SepNorm also improves the AUC to 75.64%, compared with ShareNorm's AUC of 73.36%. These results are strong evidence that the embeddings of the [VNode] can better summarize the properties of the entire graph and thus give superior performance on downstream tasks.

4.4 Uniformity Analysis

In this section, we investigate how, under both non-contrastive and contrastive training methods, ShareNorm and SepNorm respectively affect the uniformity of learned embeddings and further classification performances.

Experiment setup. We pretrain MAE on the STL10 dataset via four different losses:

MAE loss L MAE without any L unif on [CLS] and token embeddings. This setting is a study with MAE training only.

MAE loss L MAE with L unif on the [CLS] embeddings. We treat all [CLS] embeddings within the same batch (except itself) as negative instances.

MAEloss L MAE with L unif on the token embeddings. We treat all token embeddings within the same batch or same images (except itself) as negative instances.

MAE loss L MAE with L unif on both [CLS] and token embeddings.

We choose λ = { 0 , 0 . 01 , 0 . 1 , 1 } . Note that the second loss with λ = 0 . 1 corresponds to the UMAE [Zhang et al., 2022]. We also replace the normalization layer of the ViT in MAE with one of the following: {LN, BN, BN+LN, BN+BN}. The combination of different losses, different λ 's, and different normalization layers yields 40 specifications of the experiments.

We first report our results with MAE training only. The uniformity of learned embeddings is first measured by singular values of the decomposition of an embedding matrix: we randomly choose 10k embeddings to form the matrix. We do this separately for [CLS] embeddings and normal token embeddings. Figrue 4 shows the results, which indicate that [CLS] features learned from SepNorm exhibit better representational power and thus can better encode the global information.

Then the uniformity is measured by the score in Eqn. 4. Table 6(a) shows the numerical value of the uniformity on the STL10 and Aircraft datasets [Coates et al., 2011, Maji et al., 2013]. Compared to ShareNorm, SepNorm significantly enhances the uniformity of [CLS] embeddings . Interestingly, the uniformity of normal tokens' embeddings remains comparable. We also empirically verify that better uniformity on the [CLS] embeddings results in better performance on the downstream task (Figure 6(b)). Another observation is that the uniformity of [CLS] embeddings is clearly improved when they are normalized by BN instead of LN. Our hypothesis is that BN tries to make each feature dimension useful by controlling its variance while LN may still neglect some feature dimensions.

3.5

2.5

1.5

0.5

Uniformity

92

93

94

LN + BN

LN + LN

LN

BN + LN

BN + BN

BN

Figure 6: Uniformity Analysis. (a) Under SepNorm, the uniformity of the [CLS] embeddings are better preserved on the STL10 and Aircraft datasets. (b) Uniformity is positively related to the downstream task performance - lower uniformity results in higher classification accuracy on the STL10 dataset.

We then report results from studies with U-MAE training. Figure 5 shows the uniformity metrics obtained using different λ 's. When using ShareNorm, the uniformity of the [CLS] embeddings is no better than -3.088, and even the explicit uniformity loss does not help much. On the contrary, embeddings learned from the proposed SepNorm can easily achieve better uniformity scores. The study with the contrastive approach further verifies the advantage of SepNorm in terms of encouraging uniformity of [CLS] embeddings.

The results provide strong evidence that the uniformity of the [CLS] embeddings is held down by ShareNorm even the minimization of an explicit contrastive loss cannot increase it . We

0

0.01

0.05

0.1

0.5

1

4

3

2

1

Uniformity

LN

LN+LN

BN

BN+LN

hypothesize that all features after LN will distribute in the same sphere, and [CLS] embeddings are squeezed to a small area of the sphere surface because they need to be different from embeddings of normal tokens.

̸

Table 4 reports the downstream performance (accuracy) on STL10 across 40 different settings. We summarize our observations: (1) In the non-contrastive method MAE, with proper configuration, the performance of SepNorm is superior to that of ShareNorm. (2) In contrastive methods ( λ = 0 ), SepNorms' advantages are further highlighted. For example, when λ = 1 , the performance of SepNorm (BN+LN) is improved by 1.6% compared to the non-contrastive method. The performance gain in SepNorm (BN+BN) is less obvious as the double BNs already impose implicit uniformity loss on both [CLS] and token embeddings.

In contrast to SepNorm, the performance of ShareNorm is less satisfactory when using contrastive methods. We believe it is very challenging to encourage the two types of embeddings to be uniformly distributed in the same sphere and keep them separable at the same time. (3) The uniformity of the token embeddings is also vital for learning a good [CLS] representation, as evident by SepNorm (BN+LN) gaining accuracy with increasing λ on the token embeddings. We hypothesize that by enforcing uniformity, the token embeddings are forced to contain less information about others, which encourages the [CLS] embedding to encode as much information as possible. Our empirical study also shows that, when contrastive loss [Oord et al., 2018] is used to encourage the uniformity of [CLS] features in self-supervised transformers, the difference between BN and LN on [CLS] features is not significant anymore.

5 Related Works

The training of transformer architectures with self-supervised learning has seen significant advancements in both contrastive and non-contrastive training. Among self-supervised learning methods,

non-contrastive ones do not rely on negative samples for learning. They have emerged as a powerful approach for training transformer models and demonstrated remarkable successes in various tasks. BERT [Devlin et al., 2018] and RoBERTa [Liu et al., 2019] were proposed in the NLP domain. Additionally, there are some works focus on the specific task, such as speech recognition [Wang et al., 2020], image generation [Chen et al., 2020a], and heterogeneous graph generation [Hu et al., 2020b].

Contrastive methods on the contrary train networks using positive and negative samples that are constructed without manual labeling. They have also been used to train transformer-based architectures. Gao et al. [2021] and Zhang et al. [2022] make significant strides in natural language processing tasks, while Chen et al. [2021] provide valuable insights into the pre-training of transformers. Meanwhile, the potential of contrastive methods in vision transformers has been demonstrated by Caron et al. [2021] and Radford et al. [2021]. These collective efforts underscore the versatility and efficacy of contrastive methods in self-supervised learning of transformers.

Normalization layers, including layer normalization and batch normalization, are essential to transformer architectures because they help stabilize the training procedure and accelerate convergence. Xiong et al. [2020] delve into the role of layer normalization in the transformer architecture and provide insights about how the layer improves the training stability and the performance of transformers. Similarly, Xu et al. [2019b] explores the intricacies of layer normalization and offers potential enhancements to its effectiveness. To address the limitations of traditional batch normalization in a transformer architecture, Shen et al. [2020] introduces a new normalization layer, Powernorm, which is a variant of batch normalization. Nguyen and Salazar [2019] focus on the normalization process in the self-attention mechanism of transformers and propose methods to optimize the normalization of self-attention. All the efforts above underscore the critical role of normalization layers in transformer models.

6 Conclusion

In this work, we have introduced SepNorm to separate the normalization of [CLS] embeddings from that of other tokens. Across three application domains (images, text, and graphs), SepNorm shows consistent performance improvement when it is incorporated into transformer models. Our analysis shows that SepNorm promotes uniformity of [CLS] embeddings and thus enhances the transformers' ability to encode information. As a valuable technique for improving the foundational transformer architecture, SepNorm has the potential to benefit a wide range of applications.

References

Agirre, D. Cer, M. Diab, and A. Gonzalez-Agirre. Semeval-2012 task 6: A pilot on semantic textual similarity. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pages 385-393, 2012.

Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre, and W. Guo. * sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics (* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity , pages 32-43, 2013.

Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014) , pages 81-91, 2014.

Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Maritxalar, R. Mihalcea, et al. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015) , pages 252-263, 2015.

Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez Agirre, R. Mihalcea, G. Rigau Claramunt, and J. Wiebe. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 1617; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511. ACL(Association for Computational Linguistics), 2016.

L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 , 2017.

Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International conference on machine learning , pages 1691-1703. PMLR, 2020a.

Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020b.

Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Dolan and C. Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005) , 2005.

Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. 2020.

Gao, X. Yao, and D. Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 , 2021.

Gu, Z. Ding, S. Wang, L. Zou, Y. Liu, and D. Yin. Deep multifaceted transformers for multiobjective ranking in large-scale e-commerce recommender systems. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 2493-2500, 2020.

Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang. Transformer in transformer. Advances in Neural Information Processing Systems , 34:15908-15919, 2021.

Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence , 45(1): 87-110, 2022.

He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Hu and B. Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining , pages 168-177, 2004.

Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020a.

Hu, Y. Dong, K. Wang, and Y. Sun. Heterogeneous graph transformer. In Proceedings of the web conference 2020 , pages 2704-2710, 2020b.

S. Huang, F. Perez, J. Ba, and M. Volkovs. Improving transformer optimization through better initialization. In International Conference on Machine Learning , pages 4475-4483. PMLR, 2020.

Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

J. Irwin and B. K. Shoichet. Zinc- a free database of commercially available compounds for virtual screening. Journal of chemical information and modeling , 45(1):177-182, 2005.

Jing, P. Vincent, Y. LeCun, and Y. Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348 , 2021.

Karmakar, S. W. Teng, and G. Lu. Thank you for attention: a survey on attention-based artificial neural networks for automatic speech recognition. arXiv preprint arXiv:2102.07259 , 2021.

Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Zamparelli, et al. A sick cure for the evaluation of compositional distributional semantic models. In Lrec , pages 216-223. Reykjavik, 2014.

Medhat, A. Hassan, and H. Korashy. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal , 5(4):1093-1113, 2014.

Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Q. Nguyen and J. Salazar. Transformers without tears: Improving the normalization of selfattention. arXiv preprint arXiv:1910.05895 , 2019.

M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Pang and L. Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cs/0409058 , 2004.

Pang and L. Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075 , 2005.

Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Shen, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer. Powernorm: Rethinking batch normalization in transformers. In International Conference on Machine Learning , pages 8741-8751. PMLR, 2020.

Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 , 2021.

Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management , pages 1441-1450, 2019.

Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

M. Voorhees and D. M. Tice. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval , pages 200-207, 2000.

Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 99299939. PMLR, 2020.

Wang, A. Mohamed, D. Le, C. Liu, A. Xiao, J. Mahadeokar, H. Huang, A. Tjandra, X. Zhang, F. Zhang, et al. Transformer-based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6874-6878. IEEE, 2020.

Wiebe, T. Wilson, and C. Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation , 39:165-210, 2005.

Wu, S. Li, C.-J. Hsieh, and J. Sharpnack. Sse-pt: Sequential recommendation via personalized transformer. In Proceedings of the 14th ACM Conference on Recommender Systems , pages 328-337, 2020.

Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning , pages 10524-10533. PMLR, 2020.

Xu, Q. Liu, J. van Genabith, D. Xiong, and J. Zhang. Lipschitz constrained parameter initialization for deep transformers. arXiv preprint arXiv:1911.03179 , 2019a.

Xu, X. Sun, Z. Zhang, G. Zhao, and J. Lin. Understanding and improving layer normalization. Advances in Neural Information Processing Systems , 32, 2019b.

Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems , 34: 28877-28888, 2021.

Zhang, Y. Wang, and Y. Wang. How mask matters: Towards theoretical understandings of masked autoencoders. arXiv preprint arXiv:2210.08344 , 2022.

Zhou, T. Ge, K. Xu, F. Wei, and M. Zhou. Scheduled drophead: A regularization method for transformer models. arXiv preprint arXiv:2004.13342 , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a simple modification to Transformer-based self-supervised learning by having distinct normalization (parameters) for the CLS token, which captures the global information for use in downstream tasks, and the remaining tokens. The motivation is that this allows the tokens to avoid dimension collapse as indicated by a lower uniformity score, which in turn correlates with higher downstream performance. The paper verifies in a case study that this is empirically the case.\nThe proposed method, SepNorm, is evaluated in a controlled setting against the standard approach, ShareNorm, on various datasets from different domains. SepNorm consistently achieves improvements. An additional ablation study confirms that this is also the case when uniformity is encouraged explicitly through an additional loss term.\n\nSTRENGTHS:\n* The paper proposes an extremely simple, yet consistently effective solution to mitigate the problem of dimensional collapse and thereby improve performance.\n* The paper is concerned with general self-supervised Transformer architectures, which are relevant to the majority of the NeurIPS audience.\n* The experiments are very clear and well executed, the controlled experimental setup leaves little room for doubt regarding the method's merits.\n* The paper is generally self-contained, all necessary pieces can be understood by a general ML-savvy audience.\n* The idea to use separate sets of learnable parameters across different components of the model is of course not new per se, but the application context and the resulting insights constitute sufficient novelty.\n\nWEAKNESSES:\nThe only main weakness is that the paper could be structured better. Sections 3.1. and 3.2 mix the description of the proposed method with experimental results that are supposed to motivate the method. However, those results already rely on the existence of the proposed method, and do thus not suit as motivation. Rather, they constitute useful analyses of the proposed method, which could just as well be discussed in the \"Experiments\" section. This has the added benefit that the datasets used for generating results in Figures 3 and 4 have already been introduced (which only happens in 5.1). In fact, section 3.1. and 3.2. never explicitly mention that these are computer vision datasets.\n\nQUESTIONS:\n* In section 4, alternative normalization methods are discussed. In how far do these address the dimensional collapse problem? How would your results change with different normalizations, i.e., in how far is your method orthogonal to these?\n* Table 2: It is unclear what you mean by BERT-base and RoBERTa-base. Do these refer to the original models, or are these finetuned in the same way as the \"+SepNorm\" models? If it's the former, how do we know that the improvement isn't merely a result of continued finetuning?\n\nLIMITATIONS:\nThe paper does not address limitations. I think it would be useful to discuss whether the results would extend to other types of normalization (see question above), and what potential avenues for future work are.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper introduces SepNorm, a normalization technique for Transformer models. SepNorm separates the normalization of the [CLS] token from the rest of the tokens in a sequence, a departure from the traditional ShareNorm method that normalizes all tokens together.

STRENGTHS:
1. By applying SepNorm, the embeddings of the [CLS] symbol effectively captures the characteristics of the entire graph, leading to improved results in downstream applications.
2. SepNorm achieves better uniformity by encouraging uniformity for both the [CLS] and token embeddings when used in contrastive methods.
3. SepNorm seems to do well in the evaluation.

WEAKNESSES:
No uncertainty/confidence/error bars on experimental results or significance testing.

QUESTIONS:
Would SepNorm be helpful for multilingual machine translation tasks?

LIMITATIONS:
No visualization of the learned representation from before and after applying SepNorm
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The authors propose to use a different normaliser to the [CLS] token and the rest of the tokens for masked autoencoders. 
They motivate this as an improvement on the standard normalisation and combine it with contrastive uniformity loss. 
They observe some improvements in classification tasks .

STRENGTHS:
The pape is proposing a new normalisation of MAE using 2 separated normalisers one for [CLS] token and another for other tokens. 
They provide evidence this helps the models to obtain better perfomance indepndently and in combination with uniformity loss.

WEAKNESSES:
The contribution while important doesn't seem relevant for a larger audience but they do per discussion below for several tasks.

QUESTIONS:
While the [CLS] token stands among others,  it looks like this can be an extreme of different normalisation depending on the frequency. 
Have authors compared how much of the improvement comes from the token frequency vs how much comes from the semantic full sentence encoding of the token ?

LIMITATIONS:
The authors have studied some ablation studies.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method called SepNorm, which utilizes separate normalization layers for the [CLS] token and the remaining tokens, replacing the conventional single normalization layer (ShareNorm) in transformers. Experiments demonstrate the importance of applying separate normalization to the [CLS] token and the remaining tokens, and show that SepNorm enhances the transformer's ability to encode the input.\n\nSTRENGTHS:\nThe paper propose to explore separate normalization for the [CLS] token in pretrained transformers. Experiments on three different domains  show the effectiveness of the proposed method.\n\nWEAKNESSES:\nThe paper mentions that \"Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles.\" It seems that this statement is not empirically validated in the paper. \n\nIn the ablation study, only experiments on a CV dataset have been conducted. It would be better if more experiments on NLP and Molecule Discovery datasets could be carried out.\n\nThe paper only selects one or two models for each domain in the experiments, such as evaluating the STS task in the NLP domain. However, this limited selection of tasks is not sufficient to demonstrate the generalizability of SepNorm. It would be more appropriate to include a wider range of experiments across multiple tasks to provide a comprehensive evaluation of the effectiveness and generalizability of the proposed SepNorm method.\n\nThe paper does not empirically compare the proposed method against previous works, such as Powernorm mentioned in the section of related work.\n\nQUESTIONS:\n1. Can the proposed method be scaled to large language models?\n\n2. How is the speed of SepNorm in comparison to ShareNorm?\n\nLIMITATIONS:\nThe paper does not discuss the limitations of the proposed method. It would be better to discuss the potential impact of the proposed normalization method on learned representations, training time in addition to the performance on the downstream tasks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper analyzes the normalization layer that is applied to both the context tokens and the [cls] token in an input. It argues that traditional normalzation that is applied to both type of tokens, dubbed ShareNorm, is not effective since they have different roles. The paper proposes to use separate normalization layers for these tokens, SepNorm, and conducted a series of analysis. Echoing prior work, they showed that a measure of uniformity of the representations correlates with downstream classification performance, and further that ShareNorm find hard to tackle this uniformity deficiency even with uniformity fixing methods, while SepNorm improves uniformity and hence downstream task performances.

STRENGTHS:
1. The paper is well written and easy to understand. It starts with sufficient background and explains the analysis clear.
2. The hypothesis intuitively makes sense to me, and the empirical analysis provides good support.
3. Experiments are conducted on a number of tasks in vision, nlp and graphs.

WEAKNESSES:
I don't have major concerns on this work.

QUESTIONS:
n/a

LIMITATIONS:
no limitations were discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-13