Dynamic Token Normalization improves Vision Transformers
Authors: Wenqi Shao
Yixiao Ge
Zhaoyang Zhang
XUYUAN XU
Xiaogang Wang
Ying Shan
Ping Luo
Venue: ICLR 2022 Poster (2022)
Keywords: classification
Normalization
transformer

Abstract: Vision Transformer (ViT) and its variants (e.g., Swin, PVT) have achieved great success in various computer vision tasks, owing to their capability to learn long-range contextual information. Layer Normalization (LN) is an essential ingredient in these models. However, we found that the ordinary LN  makes tokens at different positions similar in magnitude because it normalizes embeddings within each token. It is difficult for Transformers to capture inductive bias such as the positional context in an image with LN. We tackle this problem by proposing a new normalizer, termed Dynamic Token Normalization (DTN), where normalization is performed both within each token (intra-token) and across different tokens (inter-token). DTN has several merits. Firstly, it is built on a unified formulation and thus can represent various existing normalization methods. Secondly, DTN learns to normalize tokens in both intra-token and inter-token manners, enabling Transformers to capture both the global contextual information and the local positional context. Thirdly, by simply replacing LN layers, DTN can be readily plugged into various vision transformers, such as ViT, Swin, and PVT. Extensive experiments show that the transformer equipped with DTN consistently outperforms baseline model with minimal extra parameters and computational overhead. For example, DTN outperforms LN on small ViT by $1.1\%$ top-1 accuracy on ImageNet.

Published as a conference paper at ICLR 2022
DYNAMIC TOKEN NORMALIZATION IMPROVES VI- SION TRANSFORMER
Wenqi Shao 1.2, Yixiao Ge Zhaoyang Zhang Xuyuan Xu
Xiaogang Wang ' Ying Shan 2, Ping Luo
(weqishe Link, zhaoyangznange Link, xgwanggee. Jeunk edu.
(yixiacge, evanxyxu, yingsshan Sethaaanneennntt.comm eecee com
p ccmm
TTe Chinese University of Hong Kong 2 ARC Lab, Tencent PCG
3 AI Technology Center Tencent Video 4 The University of Hong Kong
ABSTRACT
Vision Transformer (ViT) and its variants (e.g.. Swin, PVT) have achieved great success in various computer vision tasks, owing to their capability to learn long- range contextual information. Layer Normalization (LNN) is an essential ingredient in these models. However, we found that the ordinary LN makes tokens at differ- ent positions similar in magnitude because it normalizes embeddings within each token. difficult for Transformers to capture inductive bias such as the posi- tional context in image with LN. We tackle this problem by proposing new normalizer, termed Dynamic Token Normalization (DTN), where normalization is performed both within each token (intra-token) and across different tokens (inter- token). DTN has several merits. Firstly, it is built on a unified formulation and thus can represent various existing normalization methods. Secondly, DTN learns to normalize tokens in both intra-token and inter-token manners, enabling Trans- formers to capture both the global contextual information and the local positional context. Thirdly, by simply replacing LN layers, DTN can be readily plugged into various vision transformers, such as VIT, Swin, PVT, LeViT, T2T-VIT, BigBird and Reformer. Extensive experiments show that the transformer equipped with DTN consistently outperforms baseline model with minimal extra parameters and computational overhead. For example, DTN outperforms LN by 0.5% 1.2% top-1 accuracy on ImageNet, by 1.2 1.4 box AP in object detection on COCO benchmark, by 2.3% 3.9% mCE in robustness experiments ImageNet-C, and by 015% 0.8% accuracy in Long ListOps on Long-Range Arena. Codes will be made public at https::/ https::/githuu. whw.bbtttttugggggg com wcon///wshal125/DD1
INTRODUCTION
Vision Transformers (ViTs) have been employed in various tasks of computer vision, such image classification (Dosovitskiy et al. 2020: Yuan et al.] (002)) object detection (Wang et af.] 20216] Liu al.) 2021) and semantic segmentation (Strudel et al.) [202I). Compared with the conven- tional Convolutional Neural Networks (CNNs), ViTs have the advantages in modeling long-range dependencies, as well as learning from multimodal data due the representational capacity of the
multi-head self-attention (MHSA) modules (Vaswani et al. 2017 Dosovitskiy et al., 2020). These appealing properties are desirable for vision systems, enabling VITs serve as a versatile backbone for various visual tasks.
However, despite their great successes, VITs often have greater demands on large-scale data than CNNs. due to the lack of inductive bias in ViT's such as local context for image modeling (Dosovit-) skiy et al. 2020). In contrast, the inductive bias can be easily induced in CNNs by sharing convolu- tion kernels across pixels in images (Krizhevsky al. 2012). Many recent advanced works attempt to train a data-efficient ViT by introducing convolutional prior. For example, by distilling knowl- edge from a convolution teacher such RegNet 16GF (Radosavovic al. 2020). DeiT (Touvron jet al., (20TT) can attain competitive performance when it is trained on ImageNet only. Another line of works alleviates this problem by modifying the architecture of ViTs. For example, Swin (Liu

---

Published as a conference paper at ICLR 2022
Intrattkken
Tokero
Inbe-ooken
Intra-
Posticn-anore
betore LN
Cambeds
after LN
befora IN
Presevve
Presens Differonce
Offerencc
Differnnse
Takkro
Tolerts
Tokkra
TolkeTs
10 DTM
Figure 1: The visualization of token difference in magnitude when different normalizers are em- ployed including LN (a), IN (b) and the proposed DTN (c). The results are obtained using a trained ViT-S with different normalizers on a randomly chose sample. The cube represent a feature of to- kens whose dimension is B T C, and each token is a vector with C-dimenssio embedding. We express IN, LN and DTN by coloring different dimensions of those cubes. We use a heatmap to vi- sualize the magnitude of all the tokens, i..., the norm of token embedding for each head. (a) shows that LN operates within each token. Hence, it makes token magnitude have uniform magnitude re- gardless of their positions. Instead, (b) and (c) show that IN and our DTN can aggregate statistics across different tokens, thus preserving variation between different tokens.
al., 2021) and PVT (Wang et al. 2021b) utilize a hierarchical representation to allow for hidden features with different resolutions, preserving the details of locality information. Although these works present excellent results in the low-data regime, they either require convolution or require tedious architectural design.
In this work, we investigate the problem of inductive bias in vision transformers from the perspec- tive of the normalization method. Specifically, it is known that layer normalization (LN) (Ba et al.) 2016) dominates various vision transformers. However, LN normalizes the embedding within each token, making all the tokens have similar magnitude regardless of their spatial positions shown in Fig
I [).. Although LN encourages transformers to model global contextual information (Doso- vitskiy et al.] 2020). we find that transformers with LN cannot effectively capture the local context in an image as indicated in Fig Figgaa, aa), because the semantic difference between different tokens has been reduced.
To tackle the above issue, we propose a new normalizer for vision transformers, termed dynamic to- ken normalization (DTN). Motivated by Fig [bb. where normalization in an inter-token manner like instance normalization (IN) (Ulannov et al. 2016) can preserve the variation between tokens, DTN calculates its statistics across different tokens. However, directly aggregating tokens at different positions may lead to inaccurate estimates of normalization constants due to the domain difference between tokens as shown in Fig To avoid this problem, DTTN not only collects motra-token statisties like LN, but also employs a position-aware weight matrix to aggregate tokens with similar semantic information, as illustrated Fig. DTN has several attractive benefits. (1) DTN is built on a unified formulation, making it capable of representing various existing normalization methods such as LN and instance normalization (IN). (2) DTN learns to normalize embeddings in both intra-token and inter-token manners, thus encouraging transformers to capture both global contextual informa- tion and local positional context as shown in Fig c). (3) DTN is fully compatible with various advanced vision transformers. For example, DTN can be easily plugged into recently proposed models such as PVT (Wang et al. (0221) and Swin (Liu et al. 2021) by simply replacing LN layers in the original networks.
The main contributions of this work are three-fold. (1) From the perspective of the normalization, observe that LN reduces the difference in magnitude between tokens regardless of their differ- ent spatial positions, making it ineffective for ViT's to induce inductive bias such as local context. capture
(2) We develop a new normalization technique, namely DTN, for vision transformers both long-range dependencies and local positional context. Our proposed DTN can be seamlessly plugged into various vision transformers, consistently outperforms its baseline models with vari- ous normalization methods such as LN. (3) Extensive experiment such as image classification on ImageNet (Russakovsky et al. 2015), robustness on ImageNet-C (Hendrycks & Dietterich 2019). self-supervised pre-training on VTTs (Caron et al., 2021). ListOps on Long-Range Arena (Tay al. 2021) show that DTN can achieve better performance with minimal extra parameters and marginal increase of computational overhead compared to existing approaches. For example, the variant of ViT-S with DTN exceeds its counterpart of LN by 1.1% top-1 accuracy on ImageNet under the same amount of parameters with only 5.4% increase of FLOPs.
2

---

Published as a conference paper at ICLR 2022
LN
DTN
Hean Mtnn Distance
Layers
Layers
Layers
Figure 2: The visualization of mean attention distance in multi-heade self-attention (MHSA) module of trained ViT-S with (a) LN, (b) IN, and (c) DTN. The mean attention distance denotes the average number of patches between the center of attention, and the query patch Dosovitskiy et al. (2020) (see details in Appendix Sec. B.I). A large mean attention distance indicates that the head would attend to most image patches, presenting excellent ability of modeling long-range dependencies. (a) shows that LN can perform excellent long-range contextual modeling while failing to capture local positional context. (b & c) show that multiple heads in ViT-S with IN and DTN have a small mean attention distance. Hence, IN and our DTN can capture local context because they preserves the variation between tokens shown in Fig R & c).
2 RELATED WORK
Vision transformers. Vision transformer models have been widely utilized recently (Dosovitskiy et al.] 2020 Yang et al., 2022). For example, the vision transformer (ViT) (Dosovitskiy et al.) 2020) splits the source image into a sequence of patches and fed them into transformer encoders. Although VIT presents excellent results
image classification, it requires costly pre-training on large-scale datasets (c.g., JFT-300M (Sun et al. 2017)) as VIT models are hard to capture inductive bias. Many works tackle this problem by introducing convolutional priors into ViTs because convolution inher- ently encodes inductive bias such as local context. For example, DeiT' Touvron et al. (2021) uses a token- strategy to distill knowledge from convolutional teachers. Moreover, PVT (Wang al.) 2021b), Swin (Liu et al. [021], T2T (Yuan et al. (021I, CVT (Wu et al. (22TT) and et al. intro- duces hierarchical representation to vision transformers by modifying the structure of transformers. Different from these works, our DTN trains data-efficien transformers from the perspective of nor- malization. We show that DTN enables vision transformers capture both global context and local positional locality. More importantly, DTN is generally functional module and can be easily plugged into the aforementioned advanced vision transformers.
Normalization methods. Normalization techniques have been extensively investigated in CNN (Toffe & Szegedy, 2015). Shao al.) (201)). For different vision tasks, various normalizers such as BN (loffe & Szegedy 2015). IN (Ulyanov al. 2016), LN (Ba et al., 2016), GN (Wu & Hel 2018). SN (Luo et al. 2019a) and et al. are developed. For example, BN is widely used in CNN for image classification, while IN performs well in pixel-level tasks such image style transfer. To our interest, LN outperforms the above normalizers in transformers and has dominated various mansformer-based models. Although ScaleNorm (Nguyen & Salazar 2019) and PowerNorm (Shen ett al. 2020) improves LN in language tasks such as machine translation, it does not work well in vision transformer as shown in Table 3 Instead, we observe that vision transformer models with LN cannot effectively encode local context in an image, as shown in 2ig 2 To resolve this issue, we propose new normalizer named DTN, which can capture both global contextual information and local positional context as shown in Fig c).
Dynamic Architectures. Our work is related to dynamic architectures such as mixture-of-expprrs (MoE) (Shazeer et al., (0188) Eigen et al. (013)) and dynamic channel gating and grouping networks (Hua et al. 2018 ZZhng al. 2019). Similar to these works, DTN also learns weight ratios to select computation units. In DTN, such a strategy is very effective to combine intra- and inter- token statistics. In addition, DTN is substantially different from DN (Luo et al., 2019b), where dynamic normalization can be constructed as the statistics in BN and LN can be inferred from the statistics of IN. However, it does not hold in the transformer because LN normalizes within each token embedding. Compared to SN (Luo et al. 2019a), we get rid of BN, which is empirically detrimental to ViTs. Moreover, we use a position-awart aware probability matrix to collect intra-token statistics, making our DTN rich enough contain various normalization methods.
3

---

Published as a conference paper at ICLR 2022
3
METHOD
This section firstly revisits layer normalization (LN) (Ba et al. 2016) and instance normalization (IN) (Ulyanov et al. 2016) and then introduces the proposed dynamic token normalization (DTN).
3.1 REVISITING LAYER NORMALIZATION
In general, we denote a scalar with a regular letter (e.g. "a') and a tensor with bold letter (e.g. a). Let us begin by introducing LN in the context of ViTs. We consider typical feature of kens E IR BxTx in the transformer where B, T, and C' denote batch size, token length, em- bedding dimension of each token respectively, as shown in Fig Since LN is performed in a data-independent manner, we drop the notation of "B" for simplicity.
LN standardizes the input feature by removing each token's mean and standard deviation and then utilizes an affine transformation to obtain the output tokens. The formulation of LN is written by
Tte 1
Etc 7/6
+ +.3c
Vi +
where t and e are indices of tokens and embeddings of a token respectively, e small positive constant to avoid zero denominator, and Ye: B. are two learnable parameters in affine transformation. In Eqn. [). the normalization constants LN f4 and (a²) are calculated in an intra-token manner
as shown in Fig [a). Hence, for all E (T), we have
14 In
1
Tte and (o²))m 5
Σ (Tt (
14 12)2
Previous works show that LN works particularly well with the multi-head self-attention (MHSA) module to capture long-range dependencies vision tasks, as can also be seen from Fig aaa a) where most heads in MHSA after LN attend to most of the image. However, we find that LN reduces the difference magnitude between tokens at different positions, preventing the MHSA module from inducing inductive bias such as local context. To see this, Eqn. shows that the mean and variance are obtained within each token, implying that each token would have zero mean and unit variance. Further, the standardized tokens then operated by the same set of affine parameters {Te Se ) through [q Therefore, all the tokens returned by LN would have a similar magnitude regardless of their positions in the image.
This fact can also be observed by visualizing token magnitude in Fig a). As we can see, the difference between tokens after LN is reduced. However, since tokens generated from image patches at different spatial locations, they should encode specific semantic information to embed the local context in image. As a result, MHSA module after LN cannot effectively capture local context as presented in 2(g where only few heads have small attention distance.
Recent works tackle the issue of inductive bias in ViTs by combining the design of convolutional neural network (CNN) and vision transformers (Wang et al.) [2021b] Liu et al. 2021). Although these methods have achieved good performance on various vision tasks, they still require compli- cated architecture design. In this paper, we aim to improve vision transformers by designing new normalization method.
Instance Normalization (IN). IN provides an alternative to normalizing tokens while preserving the variation between them. In CNN, IN learns invariant features to a pixel-level perturbation, such as color and styles. Hence, IN can be employed to learn features of tokens with the local context. The definition of IN is the same with LN in Eqn. except for the acquisition of normalization constants,
T
T
én MC
Σ 1=1
Tte (a²))n
Σ (Ite t=1
Since IN obtains its statistics in an inter-token manner, the tokens returned by IN still preserves the variation between tokens in each head shown in Fig (b. In addition, As we can observe from b), MHSA in the transformer with IN have more heads with a small mean attention distance than that of LN, showing that IN encourages MHSA to model local context.
However, IN has a major drawback in the context of vision transformers. From a(a,, the nor- malization constants in IN are acquired by considering all tokens in the image. However, tokens
4

---

Published as a conference paper at ICLR 2022
vartation Couftiiiet
80
'Tree'
60
Domain difference
Normalization constants 1²*, a
40
image patchee
"Soo
20
10
Layers
(4) with domain difference
(b) Statisties shift in different normalizers
Figure 3: (a) illustrates the normalization constants in IN. IN obtains its normalization constants by considering all patches from the image. Hence the semantic distribution shift from different image patches may lead inaccurate estimates of normalization constants in IN. (b) shows the relative variation between token features and the statistics (mean), which is measured by the variation co- efficient defined by the norm of (> /) where 4 the mean normalization layer. A larger variation coefficient indicates that the mean is further away from the token feature. From (b), IN may lead to inaccurate estimates due to the domain difference between tokens. LN and our proposed DTN can mitigate the problem of inaccurate estimates of statistics in IN.
encoded from different image patches in the vision transformer may come from different semantic domains. For example, the top patch presents a tree, while the bottom is snow picture. we take an average over all the tokens to calculate mean and variance, it would lead to inaccurate estimates of normalization constants as shown in ((). Empirically, we find that VIT-S with IN obtains much lower top-1 accuracy (77.7%) on ImageNet than its counterpart with LN (79.9%). Section 3.2 resolves these challenges by presenting dynamic token normalization.
3.2 DYNAMIC TOKEN NORMALIZATION (DTN)
DTN encourages attention heads in MHSA to model both global and local context by inheriting the advantages of LN and IN.
Definition. DTN defined by unified formulation. Given the feature of tokens E R TxC normalizes it through
DTN
T = 7 x Concatens [m"} +
(4)
ConcatenEsи; ((
where 7. B are two C-by-1 vectors by stacking all Te and Be into column, and n² RTx 4 2 E R & are normalization constants of DTN in head h where denotes the number of heads in transformer. The 'Concate' notation indicates that DTN concatenates normalization con- stants from different heads. This design is motivated by two observations in 2 First, attention heads attend patches in the image with different attention distances, encouraging diverse contex- tual modeling for different self-attention heads. Second, by obtaining statistics in two different ways (intra-token and inter-token), LN and IN produce different patterns of attention distance in MHSA. Hence, we design DTN by calculating normalization constants specific to each head.
Normalization constants in DTN. As aforementioned in See 3.1 LN acquires its normalization constants within each token, which is helpful for global contextual modeling but fails to capture local context between tokens. Although IN can achieve self-attention with locality, calculates the normalization constants across tokens, resulting inaccurate mean and variance estimates. To overcome the above difficulties, DTN obtains normalization constants by trading off intra- and inter- token statisties given by
1 Xa (μ Ingh +1 (1- (1-
(5)
= Xa ( + Xa> x²) (P" o ph*")]
where (H E T ((o2)lnge E T & are intra- token mean and variance obtained by stacking
all ut" in Eqn. into column and then broadcasting for C/H columns, x² T represents token embeddings in the head h of æ. In Eqn.(5). pd RTx X [p"("" @ x:h) - (p*a* R p* are expected to represent inter-token mean and variance respectively. Towards this goal, we define ph as a T-by-T learnable matrix satisfying that the sum of each row equals 1. For example, when ph #1 1 and 1 is a T-by-T matrix with all ones, they become mean and variance of IN respectively.
Moreover, DTN utilizes a learnable weight ratio d [0. to trade off intra-token and inter-token
5

---

Published as a conference paper at ICLR 2022
statistics. By combining intra-token and inter-token normalization constants for each head, DTN not only preserves the difference between different tokens as shown in [ig but also enables different attention heads to perform diverse contextual modelling in MHSA as shown in Fig In DTN, the weight ratios for fe and e can be different as the mean and variance in the normalization plays different roles in network's training Luo et al. (2018) Xu et al. (20T9), but they are shared in Eqn. to simplify the notations.
Representation Capacity. By comparing Eqn.(5) and Eqn. 3 we see that DTN calculates its normalization constant in a unified formulation, making it capable to represent a series of nor- malization methods. For example, when A = 1, we have μ² (urnth and (o²)* = ((o3) Hence, DTN degrades into LN. When A = 0 and ph = + 1, we have Fe¹ 1 and (o²)* 11x @ x") - 1+ @ $1x") which are the matrix forms of mean and variance in Eqn. 3 Therefore, DTN becomes IN in this case. Moreover, when A = 0 and ph is a nor- malized k- b--aanded matrix with entries in the diagonal band of all ones. DTN becomes local version of IN which obtains the mean and variance for each token by only looking at k neighbouring tokens. Some examples of k-banded matrix is shown Fig of Appendix.
The flexibility of X and ph enables DTN to mitigate the problem of inaccurate estimates of statis- ties in IN in two ways. Firstly, by giving weight ratio À a large value, DTN behaves more like LN, which does not suffer from inaccurate estimates of normalization constants. Secondly, by restricting the entries of Ph such as k-banded matrix, DTN can calculate the statistics across the tokens that share similar semantic information with the underlying token. As shown in Fig b), the relative variation between tokens and the corresponding statistics (mean) in DTN is significantly reduced compared with that of IN.
However, it is challenging to directly train a huge matrix with a special structure such as k-banded matrix through the vanilla SGD Adam algorithm. Moreover, the extra learnable parameters in- troduced by Ph in DTN are non-negligible. Specifically, every ph has T variables, and there are often hundreds of heads in transformer. Therefore, directly training P would introduce millions of parameters. Instead, we design bb exploring the prior of the relative position of image patches which leads to marginal extra parameters.
Construction of ph_ We exploit the implicit visual clue in an image to construct ph, As shown in FigBaa) a). the closer two image patches are each other, the more similar their semantic informa- tion would be. Therefore, we employ positional self-attention with relative positional embedding (d' d'AAscoli et al. 2021) to generate positional attention matrix pd as given by
ph = softmax (Ra")
(6)
where R e RT is a constant tensor representing the relative positional embedding. To embed the relative position between image patches, we instantiate R. as written by where 63, and 8y are relative horizontal and vertical shifts between Rg = [05)) + (8%) (2.)).... 88 6
patch é and patch j (Cordonnier et al. 2019), respectively. An example of the construction of R illustrated in 5f Appendix. Moreover, a" R3x1 are learnable parameters for each head. Hence, Eqn.(6) only introduces 3 parameters for each head, which is negligible compared to the parameters of the transformer model. In particular, by initializing a" equation below, ph gives larger weights to tokens in the neighbourhood of size v x VH relative to the underlying token,
a" [ 1, 1, TTAT
where A - (A= is each of the possible positional offsets of the neighbourhood of size VH x VH relative to the underlying token. It also denotes the center of attention matrix Ph, which means that P would assign the largest weight to the position A h. For example, there are 4 optional attention centres (denoted by red box) when H 4 as shown in Fig 6 Since the weights of each pd concentrates on a neighbourhood of size HH x /H Equ. 6 7) DTN can aggregates the statistics across tokens with similar semantic information through positional attention matrix ph.
3.3 DISCUSSION
Implementation of DTN. DTN can be inserted extensively into various vision transformer mod- els by replacing LN in the original network. In this work, we verify the effectiveness of DTN on VIT, PVT, and Swin. However, it can also be easily applied to recently- advanced vision

---

Published as a conference paper at ICLR 2022
Table 1: Performance the proposed DTN evaluated on VIT models with different sizes, DTN can consistently outperform LN with various ViT models. H and C' denote the number of heads and the dimension of embeddings of each token, respectively.
Model
Method
H
FLOPs
Params
Top-1 (%)
Top-5 (%)
VIT-T
LN
1.26G
5.7M
72.2
91.3
LN
4
192
1.26G
5.7M
72.3
91.4
VIT-T
DIN
4
192
1.40G
5.7M
73,2
91.7
LN
6
384
4,60G
22.1M
79.9
95.0
VIT-S
DIN
6
384
4,88G
22.1M
80.6
95,3
LN
9
432"
5.77G
2788
80.6
95.2
VIT-S
9
6,08G
27.8M
81.7
IN
12
768
17.58C
86.5M
81.8
95.9
VIT-B
DIN
12
768
18.13G
86.5M
82.3
96.0
LN
16
768
17.58G
86.5M
81.7
95.8
VIT-B"
DTN
16
768
18.13G
86.5M
82.5
96.1
Table 2: Performance of the proposed DTN on various vision transformers in terms accuracy and FLOPs on ImageNet. Our DTN improves the top-1 and top-5 accuracies by clear margin compared with baseline using LN with marginal increase of FLOPs.
PiT-Tiny
PVT-Small
Swin-T Top-5
Top-1
Top-5
FLOPs
Top-1
Top-5
FLOPs
Top-1
FLOPs
Baseline 75.1
92.3
1.90G
79.9
95.0
3.80G
81.2
95.5
4.5IG
DTN
76.3
93.1
2.05G
80.8
95.6
4.15G
81.9
96.0
5.09G
Table 3: Comparisons between DTN and other normalization methods, including LN, BN, IN, GN, ScaleNorm, and PowerNorm on ImageNet in terms of top-1 accuracy. Our DTN achieves competi- tive results over various normalizers.
LN
BN
IN
GN
SN
ScaleNorm
PowerNorm
DTN
VIT-S
79.9
77.3
77.7
78.3
80.1
80.0
79.8
80.6
A versus LN
-2.6
-2.2
-1.6
+0.2
+0.1
-0.1
+0.7
VIT-S*
80.6
77.2
77.6
79.5
81.0
80.6
80.4
81.7
A versus LN
-3.4
-3.0
-1.1
+0.4
0.0
-0.2
+1.1
transformers such as CVT at 2021).
As shown in Algorithm I of Appendix, DTN can be im- plemented in forward pass. The introduced learnable parameters by DTN are weight ratio X and a by Eqn. 77) which are initialized through line 2 of Algorithm Note that All the computations involving in DTN are differentiable.
Complexity analysis. The computation and parameter complexity for different normalization meth- ods are compared in Table 7b Note that the number of heads in the transformer is far less than the embedding dimension i.c., < C. DTN introduces almost the same Icarnable parameters com- pared with IN and LN. Morcover, our DTN calculates inter-token normalization constants for each token embedding through positional attention matrix ph in Eqn.(5), resulting O (BBTT² com- putational complexity. The computational overhead proportional to T is nonnegligible when the token length is large. To make the computation efficient, we adopt a global average pooling op- eration with pooling size the token level, which reduces the length of tokens to T/s". Hence, the computational complexity of DTN decreases to (BCT2//44 We also investigate the FLOPs of vision transformers with DTN in Table I and Table
4 EXPERIMENT
This section extensively evaluates the effectiveness of the proposed Dynamic Token Normaliza- tion (DTN) in various computer vision tasks, including classification ImageNet, robustness on ImageNet-C and ImageNet-R (.ec of Appendix), and self-supervised pre-training (Sec B.3 of Appendix). Besides, we also test DTN the task of ListOps using transformers with different efficient self-attention modules. An ablation study is provided to analyze our proposed DTN com- prehensively (see more details in B. Appendix). The training details can be found in Sec B.
7

---

Published as a conference paper at ICLR 2022
Table 4: Comparisons between DTN and other normalization methods on Long ListOps task of LRA in terms of accuracy. Our DTN is validated three SOTA transformers architectures and yields competitive results to other normalizations.
LN
BN
IN
GN
ScaleNorm
DTN
Transformer Vaaswani et al.
2017
36.4
29.2
22.7
28,2
36.6
37.0
versus LN
-7.2
-13.7
-8.2
+0.2
+0.6
BigBird /aheer al.
2020)
36.7
36.7
36.3
36.8
36.9
37.5
versus LN
0.0
-0.4
+0.1
+0.2
+0.8
Reformer Kitaev ct al
2020
37.3
37.2
37.0
37.4
37.3
37.8
A versus LN
-0.1
-0.3
+0.1
0.0
+0.5
4.1 RESULTS IMAGENET
DTN on ViT models with different sizes. We evaluate the performance of the proposed DTN on ViT models with different sizes, including VIT-T, VIT-S, and VIT-B. Note that DTN obtains inter-token normalization constants by positional attention matrix P in Eqn. which gives larger weights to entries in the neighbourhood of size VH x /H where H denotes the number of heads. To exploit this property, increase the number of heads from 6 to in order to produce 3 3 neighborhood and decrease the embedding dimension in each head from 64 to 48. The resulting models are denoted by ViT-T", ViT-S*, and ViT-B".
As shown in Table I. ViT models with DTN can consistently outperform the baseline models using LN with a marginal increase of parameters and FLOPs on the ImageNet dataset, demonstrating the effectiveness of our DTN. For example, DTN surpasses LN by a margin of 1.1% top-1 accuracy on ViT-S* with almost the same parameters and only 5.4% increase of FLOPs compared with LN. On the other hand, we see the margin of performance becomes larger when ViT models are instantiated with a larger number of heads. For instance, DTN exceeds LN by 0.8% top-1 accuracy on ViT-B" while DTN outperforms LN by 0.5% top-1 accuracy ViT-B.
DTN on various vision transformers. We investigate the effect of DTN on various vision trans- formers on ImageNet. Since normalization is an indispensable and lightweight component in trans- formers, it is flcxible to plug the proposed DTN into various vision transformers by directly replac- ing LN. In practice. two representative vision transformers, including PVT Wang et al. (202lb) and Swin Liu et al. (002)) are selected to verify the versatility of our DTN. We use their respective open-sourced training framework. As shown in Table 2 DTN improves the performance of PVT and Swin by a clear margin in terms of top-1 and top-5 accuracies compared with baseline using LN with a marginal increase of FLOPs. Specifically, PVT-Tiny, PVT-Small, and Swin-T with DTN surpass their baseline with LN by 1.2%, 0.9%, and 0.7% top-1 accuracy, respectively.
DTN versus other normalization methods. We show the advantage of our DTN over exiting representative normalizers that are widely utilized in computer vision models such as BN, IN, and GN, and in language models such as LN, ScaleNorm, and PowerNorm. The results are reported in Table 3 It can be seen that our DTN achieves competitive performance compared with other normalization methods on both VIT-S and VIT-S* In particular, we see that vanilla BN and IN obtain the worst performance, which is inferior to LN by 2.6% and 2.2% top-1 accuracy, respectively. This may caused by inaccurate estimates of mean and variance IN and BN. For example, PowerNorm improves BN by providing accurate normalization constant through moving average in both forward and backward propagation, achieving comparable top-1 accuracy compared with LN. Our proposed DTN further calculates the statistics in both intra-token and inter-token ways, consistently outperforming previous normalizers.
4.2 RESULTS ON LISTÖPS OF RANG-RANGE ARENA
We further evaluate our DTN on the ListOps task of Long-Range Arena benchmark (Tay al. 2021) to validate the proposed methods on non-image sources. The Long-Range Arena (LRA) benchmark is designed explicitly to assess efficient Transformer models under the long-context scenario. To demonstrate the effectiveness of DTN, we choose the Long ListOps task, one of the most challenging tasks in the LRA benchmark. The Long Listops task is a longer variation of standard ListOps task (Nangia & Bowman 2018) with up to 2K sequence lengths, which is considerably difficult. We compare DTN with other normalizations using vanilla Transformer (Vaswani et al. 2017) and
8

---

Published as a conference paper at ICLR 2022
two recently proposed efficient Transformer models including BigBird Zaheer al. 2020 and Reformer (Kitaev et al.) 2020). All of the experiments are conducted with default settings in (Tay et al. 2021 and the results are reported in Table. 4 As shown, DTN yields consistent performance gains (>> 0.5%) compared to other listed normalizations, including the SOTA method Reformer. observe significant accuracy drop vanilla Transformer models with IN. BN, and GN, which is possibly caused inaccurate estimates of statistics (as mentioned in Sec 3.1.
For the other normalizations,
4.3 ABLATION STUDY
Effeet of each component. As shown in Eqn.(5). DTN obtains its normal- ization constants by three crucial com- ponents including inter-token statistics from LN, intra-token statistics con- trolled by positional attention matrix ph, and learnable weight ratio X We investigate the effect of each compo- nents by considering five variants of DTN. (a) X 1. In this case, DTN becomes LN: (b) Xa
Table 5: The effect of each component in DTN. Each
component is crucial to the effectiveness of DTN.
(a) LN
(b) IN
(c)
(d)
(e) DTN
A
1.0
0.0
0.0
0.5
learnable
pa
11
learnable
learnable
learnable
Top-1
80.6
77.6
81.2
81.3
81.7
and p = 1. For this situation, DTN turns into IN; (c) X = 0 and ph is learnable. (d) A = 0.5
and ph is learnable; (c) Both X and ph are learnable which is our proposed DTN. The results reported in Table 5 By comparing (b) and (a & c), we find that both intra-token statistics from LN and positional attention matrix Pd can improve IN by large margin, showing the reasonableness of our design in Eqn. 5 It also implies that both intra-token and inter-token statistics are useful in nor- malization. By comparing (e) and (c & d), see that learnable X can better trade off inter-token and intra-token statistics for each head, which further improves the performance of DTN.
Different initilization of a We investigate the effect of different initializations of a using ViT-T" with DTN. To compare with the initialization in (7). we also use a truncated normal distribution initialize a². experiment, we find DTN with the initialization of a followed truncated normal distribution achieves 72.7% top-1 accuracy, which is worse than DTN with a initialized by Eqn. (73.2% top-1 accuracy). The effectiveness of initialization Eqn. 7 is further revealed by visual- ization in 6 We can see that p generated by Eqn. gives larger weights to its neighbouring tokens before training while P obtained by initialization with truncated normal distribution has uniform weights, indicating that Eqn. helps DTN aggregate statistics over tokens with similar local context. After training, the initialization in Eqn.(7) can also better capture local positional content than initialization using truncated normal as shown in Fig 7
CONCLUSION
In this work, find that layer normalization (LN) makes tokens similar to each other regardless of their positions in spatial. It would result in the lack of inductive bias such as local context for ViTs. We tackle this problem by proposing new normalizer named dynamic token normalization (DTN) where normalization constants are aggregated on intra- and inter-token bases. DTN provides a holistic formulation by representing a series of existing normalizers such as IN and LN. Since DTN considers tokens from different positions, it preserves the variation between tokens and thus can capture the local context in the image. Through extensive experiments and studies, DTN can adapt to ViTs with different sizes, various vision transformers, and tasks outperforming their counterparts. In particular, DTN improves the modeling capability of the self-attention module by designing a new normalizer, shedding light on future work on transformer-bast example, DTN could be combined with a sparse self-attention module (Tang et al.) 2022) because encourages self-attention with small attention distance, shown in Fig 2
architecture development. For
Acknowledgments. We thank anonymous reviewers from the venue where we have submitted this work for their instructive feedback. This work is supported in part by Centre for Perceptual and Interactive Intelligence Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 142031 18, 14208619), in part by Research Impact Fund Grant No. R5001-18. Ping Luo is supported by the General Research Fund of HK No.27208720 and 17212120.

---

Published as a conference paper at ICLR 2022
Ethics Statement. We improve the performance of vision transformers by the proposed dynamic token normalization (DTN). We notice that DTN can be plugged into various deep architectures and applied in wide range of learning tasks. Hence, tasks would have the same negative impact on ethics. Moreover, DTN may have different effects on different image sources, thus potentially producing unfair models. We will carefully study our DTN on the fairness of the model output.
work and AI applications in different
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450 2016.
arXiv preprint
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. arXiv preprint 111.5:111..03584, 2019.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.1069) 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929 2020.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4311 2013.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herve Jegou, and Matthijs Douze. Levit: A vision transformer in convnet's clothing for faster inference. In the Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12259-
12269, October 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. 122661,..222,... 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
Weizhe Hua, Yuan Zhou, Christopher De Sa, Zhiru Zhang, and G Edward Suh. Channel gating neural networks. arXiv preprint arXiv:1805. 12549, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
10

---

Published as a conference paper at ICLR 2022
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
Ping Luo. Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. arXiv preprint 1809: 2018.
Ping Luo, Ruimao Zhang, Jiamin Ren, Zhanglin Peng, and Jingyu Li. Switchable normalization for learning-to-normalin deep representation. IEEE transactions on pattern analysis and machine intelligence, 43(2):712-728, 2019a.
Ping Luo, Peng Zhanglin, Shao Wenqi, Zhang Ruimao, Ren Jiamin, and Wu Lingyun. Differentiable dynamic normalization for learning deep representation. In International Conference on Machine Learning, pp. 4203-4211. PMLR, 2019b.
Nikita Nangia and Samuel R Bowman. Listops: A diagnostic dataset for latent tree learning. NAACL HLT 2018, pp. 92, 2018.
Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895, 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436. 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))221-222, 2015.
Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. Ssn: Learning sparse switchable normalization via sparsestmax. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 443 451, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le. Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR, 2018.
Sheng Shen, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Powernorm: Re- thinking batch normalization in transformers. In International Conference on Machine Learning, pp. 8741-8751. PMLR, 2020.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv preprint arXiv:2105. 05633, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers.
arXiv preprint arXiv:2201.02760 2022.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. In International Conference on Learning Representations, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 160 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008. 2017.
11

---

Published as a conference paper at ICLR 2022
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. arXiv preprint arXiv:2106.13797, 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.2122,, 2021b
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp- 3-19, 2018.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. arXiv preprint 191.v:191..07000 2019.
Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. Improving visual ground- ing with visual-linguistic verification and iterative reasoning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2022.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv: 1986, 2021.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurlPS, 2020.
Zhaoyang Zhang, Jingyu Li, Wenqi Shao, Zhanglin Peng, Ruimao Zhang, Xiaogang Wang, and Ping Luo. Differentiable learning-to-growe channels via groupable convolutional neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2019.
12

---

Published as a conference paper at ICLR 2022
0
0
01
1
1
1
1
1
1
1
0
0
1
1
0
0
1
1
1
0
1
0
0
1
1
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
1
1
0
0
1
0
0
1
1
1
0
0
1
1
0
0
0
0
1 1
0
1
1
1
1
1
1
0
0
0
1
0
0
0
1
1
(a) Full-baaned matrix with ones
(b) banded matrix with binary entries
(c) Babaded matrik with binary entries
Figure 4: Illustration of different types of the banded matrix when token length T = 9. When
Ph in Eqn.(5) is matrix in (a), DTN aggregates its mean and variance by considering all tokens.
Hence, DTN becomes IN in this case. When pk in Eqn. 5 is a matrix in (b) or (c), DTN becomes local version of IN, which obtains the mean and variance for each token by only looking at neighboring tokens.
1
0
Li I1
0
1
-1
-1
0
-1
0
-
-1
0
0
(4) Vertical shift #r Figure 5: Illustration of construction of horizontal and vertical in relative positional embedding R.
(4) Index image
(b) Horizontal shift #t
Take T = 4 as an example; tokens with a length of are encoded from a 2 x 2 image patches. (a) index matrix indicating the relative position of each patch. The entries the index matrix are defined by i where and j are row and column index, respectively. Based on this definition, (b) and (c) show the relative horizontal and vertical shifts between patch and patch j. respectively.
Head
Hend
Wead
Head
Head
Figure Visualization of p initialized by (a) truncated normal distribution and (b-e) Eqn. before training. We visualize the weights corresponding to 91-th token, i.e. Ph
The lighter is the larger. yellow box highlights the underlying token. (a) shows that truncated normal initialization Ieads to uniform weights for all heads. On the contrary, initialization in Eqn. can give larger weights to tokens in the neighbourhood with size x Morcover, each head gives the largest weight to different neighboring token denoted by the red box.
A MORE DETAILS ABOUT APPROACH
A.1 ARCHITECTURAN DETAIL.
We integrate DTN into various vision transformers such as ViT, PVT and Swin which is instanti- ated by stacking multiple transformer encoders. Note that our proposed DTN utilizes the relative positional encoding, implying that the class token can not be directly concatenated with the tokens encoded from image patches. To tackle this issue, we employ DTN the first jotn transformer encoders and leave the last L dde transformer unchanged where L denotes the number of trans- former encoders. In this way, the class token can then appended to the output tokens of the pdtn-ph transformer encoder by following d' Ascoli et al. (202 I): Wang al. b2021b). By default, we set pdtn = 10 for ViTs and pdtn = 3 for PVT which indicates that DTN is used in the first 3 blocks of PVT. Since Swin do not utilize class token, insert DTN into all the blocks of Swin. Note that pdtn determines how many DTNs are plugged into the transformer. We investigate the influence of
the number of DTN used in transformers by the ablation study on Idie shown in Sec B.4
13

---

Published as a conference paper at ICLR 2022
Layer 10 Layer Layer Layer 7 Layer 6 Layer Layer Layer 3 Layer 2 Layer 1
Layer 3 Layer Layer
Layer
Layer
Layer
Layer
Layer Layer Layer
Figure 7: Visualization of ph initialized by (Left) truncated normal distribution and (Right) Eqn. after training. We visualize the weights corresponding to 91-th token, i.e. : view,111) for all layers. Lighter is larger.
Algorithm Forward pass of DTN.
1: Input: mini-batch inputs # E 2C where the b-th sample the hatch nh e RTXC E 21, B); lcarnable parameters " to generate X², h E [H]: relative positional embedding R e Txx and a² g Raxx ( in Eqnn.7px scale and shift parameters 8 f
gex
2: Initialization: initialize - and a" [-1, 24, 24 before training.
3: Hyperparameters:cc
4: Output: the normalized tokens {#., 1,2, .). N). sigmoil(u**.
calculate:
6: calculate: ph = softmax(Ra" (Ra))
7: for = 1 to B' do
calculate: sd!"
DC ain (a3) = EC (ate 1 r (T).
intra-token statistics.
9: calculate: (n p" (M, ((o2))i (P"((")) (x*)) (a") (*()))
& inter-token statistics.
10:
calculate: J² Xa (1'""A + -x ". be [H].
> mean of DTN.
11:
calculate: " = ("(("")" (1- (1) , & []]]
variance of DTN.
12: concatenate statistics in heads: == Concate (1t ). Concaten ( (102)
calculate DTN output: 7 (z #l/vo
14:
end for
B MORE EXPERIMENTAL RESULTS
B. IMPLEMENTATION DETAIL
ImageNet. We evaluate the performance of our proposed DTN using VIT models with different sizes on ImageNet, which consists of 1.28M training images and 50k validation images. Top-1 and
14

---

Published as a conference paper at ICLR 2022
Table 6: Mean Corruption Error (mCE) of ViT and ResNet models using different normalizers on ImageNet-C, which is designed to evaluate robustness when 'natural corruptions' are presented. The column of 'Norm.' indicates the normalizer. Res-50 is ResNet-50 model. The mCE is obtained by averaging corruption error across all corruption types. Our proposed DTN consistently improves the robustness of ViT-S and ViT-S* by large margin.
Noise
Blur
Weather
Digital
Norm.
Arch
mCE
Curs
Defocus
Glan
Moten
Zoam
Same
Frok
Fis.
Briats
Central
Fisl
JPEG
BN
Res-50
76.7
80
82
83
75
89
78
80
78
75
06
57
71
85
77
77
LN
Vit-S
42.8
40
42
50
60
46
57
43
38
39
25
44
42
38
DIN
Vir-S
40.6
39
40
40
49
58
44
54
40
36
34
24
32
43
39
35
IN
Vir.
41.9
40
42
51
39
45
56
42
37
36
25
44
39
36
DTN
Vit-S"
38.0
37
45
58
41
54
36
35
23
34
33
Top-5 accuracies are reported on ImageNet. By default, set the patch size of VIT models as 16. Moreover, we utilize the common protocols, i.c., number of parameters and Float Points Operations (FLOPs) to obtain model size and computational consumption. We train ViT with DTN by following the training framework of DeiT (Touvron et al. 2021) where the ViT models are trained with total batch size of 1024 on all GPUs. We use Adam optimizer with momentum of 0.9 and weight decay of 0.05. The cosine learning schedule is adopted with the initial learning rate of 0.0005. We use global average pooling for PVT and Swin, where pooling sizes for the first two blocks are set to 4 and 2, respectively.
proposed
ListOps on LRA. For all experiments on the LRA benchmark, we follow the open repository re- leased by (Tay et al. 202T) and keep all of the settings unchanged. Specifically, all evaluated models have an embedding dimension of 512, with heads and layers. The models are trained for 5K steps on 2K length sequence with batch size 32.
Definition of mean attention distance F The mean attention distance is defined by 1 EE di, de = E Aijōi where A, and 6.y indicate the self-attention weight and Euclidean T distance in 2D spatial between token and token j. respectively. We calculate the mean attention distance for each head by averaging batch of samples on the ImageNet validation set. When computing the attention weight between token i and other tokens, we deem token as the attention center. Since the over of A,j is 1, d; indicates the number of tokens between the attention center token and other tokens. Therefore, a large mean attention distance implies that self-attention would care more about distant tokens relative to the center token. In this sense, self-attention is thought to model global context. On the contrary, a small mean attention distance implies that self-attention would care more about neighboring tokens relative the center token. In this case, self-attention can better capture local context.
B.2 RESULTS ON IMAGENET-C AND IMAGENET-R
DTN can calculate the normalization constants for each token embedding. Thus, DTN is expected to be robust input perturbations the mean and variance specific to each token. To verify this, we compare the robustness to input perturbations rely different, specialized benchmarks ImageNet-C Henndrycks & Dietterich) 2019) and ImageNet-R (Hendrycks et al.) 2021). which contains images the presence of 'natural corruptions' and 'naturally occurring distribution shifts,' respectively. For the experiment on ImageNet-C, we can see from Table that DTN can surpass the baseline LN by 2.2% mCE metric on ViT-S and 3.9% mCE metric on ViT-S*. Moreover, DTN is also superior to LN in the presence of various natural corruptions such as 'Nosie" and 'Blur' as shown in Table 6 For the experiment on ImageNet-R, DTN obtains 30.7% and 32.7% top-1 accuracy on ViT-S and VIT-S* respectively, which are superior to LN by 1.5% and 3.0% top-1 accuracy. Therefore, our DTN is more robust to natural corruptions and real-world distribution shifts than LNN.
ViT models with LN (baseline) and our DTN.
To capture different aspects of robustness,
B.3 RESULTS ON SELF-SUPERVISION
A special way of improving a neural model's generalization is supervised contrastive learning [9, 11, 25. 31]. We couple DTN with the supervised contrastive learning on ViT-S" for 100 epoch pre-training, followed by lineal evaluation. We fine-tune the classification head by 100 epochs
15

---

Published as a conference paper at ICLR 2022
Table (a) self-superviee learning with DTN in DINO framework. We pre-train ViT-S* with DTN for 20, 40, 60, 80 and 100 epochs and report top-1 accuracy on linear evaluation. DTN is also effec- tive in learning representation complexity between different. B, T, C, H. are # samples, # tokens, embedding dimension of each token, and heads, respectively.
supervision.
(b) Comparisons of parameter and computation
(b)
Complexily
Epochs
20
40
60
80
100
Method
# parameters
computation
VIT-S (LN)
58.8
67.2
70.1
72.8
74.0
LN
2C
O
VIT-S' (LN)
58.5
68.3
71.3
74.0
74.7
IN ScaleNorm
2C
O
VIT-S" (DTN)
58.9
72.0
74.4
75.2
C
O(BCT)
DTN
20 + 3H
O(BCT
Table 8: performance.
Effect of the number of DTN layers on ViT-S"
More DTN leads to better recognition
ddn
0 (LN)
8
10 (DTN)
Top-1 ace
80.6
81.2
81.4
81.5
81.7
Head
Head
Head
300
300
300
300
360
Tradiins Eawcto
Traesing Tperie
Trakiin Baacss
Traliin Baads
Traeee taxch
Trandin Epncts
Traleno Epents
Tratning Eprcht
Tradning Eanth
Tranin Epedi
Figure 8: Training dynamics A for mean and variance in different layers of ViT-T*
for both VIT-S with LN. Please see the Appendices for more implementation details. Compared to the training procedure with LN, find considerable performance gain DTN can encour- age transformers to capture both the long-range dependencies and the local context, improving the ImageNet top-1 accuracy of VIT-S* from 74.7% 75.2%.
B.4 MORE ABLATION STUDIES
Learning dynamics of X In the implementation, treat the weight ratios X of mean and variance in Eqn. (5) differently because mean and variance in normalization play a different role in network's training. For DTN layers of ViT-T*, we plot the learning dynamics of X for (a) mean and (b) variance. As implied by Eqn. the smaller X is, the more important inter-token statistics would be. As shown in Fig variance have distinct learning dynamics. A of the mean for different heads are more diverse than that of variance. Second, different DTN layers have different learning dynamics, which are smoothly converged in training. Third, multiple heads in shallow layers prefer toter-token statics. Whereas larger A are typically presented in higher layers. is consistent with Fig 2 where some heads in shallow layers have a small attention distance while most heads in higher layers have large attention distance.
have three observations. First, the weight ratio of mean and
Effect of the number of DTN layers. As discussed in Sec A.1 Idta determines how many DTNS are plugged into the transformer. We investigate the influence of the number of DTN used in trans- formers by the ablation study on fdtn We set jotn 4, 6, 8, 10 for VIT-S* which means LN in the first 4, 6, 8, 10 transformer encoders are replaced by our DTN while leaving the remaining trans- former encoder layers unchanged. The results obtained by training VIT-S" with different pdta' on ImageNet. As shown Table, more DTN layers bring greater improvement on classification accuracy.
16

---

Published as a conference paper at ICLR 2022
Table Performance of DTN on larger models.
PVT-Large
Swin-S Params.
PVTv2-B3
Top-1
Params.
FLOPs
Top-1
FLOPs
Top-1
Params.
PLOPs
Baseline DTN
81.7
61.4M
9.8G
83.0
50M
8.7G
83.2
45.2M
6,9G
82.3
61.4M
10.5G
83.5
50M
9.4G
83.7
45.2M
7.4G
Table 10: Ablation study of DTN on relative positional embedding (RPE) on ImageNet. Ablation (a) denotes that investigate the effect of RPE in eq.6 of DTN on models without RPE such as
VIT and PVT. Ablation (b) indicates that DTN can still improve vision transformers with carcfully designed RPE.
Ablation
Method
Model
Top- 1 (%)
Model
Top-I (9)
Ablation
Method
Model
Top-I I (8)
LN + MHSA
ViT-S*
80.6
PVT-Tuy
75.1
LN
Swin-S
83.0
IN MISA Fqn.16
VIT-S"
80.8
PVT:
75.4
DTN
Swin.S
83.5
DTN w/o RPE + MHSA
VIT-S"
81.4
PVT-Timy
75.9
(b)
LN DTN
LeVIT-128S
76.7
DTN MISA (aurs)
VIT.S"
81.7
PVT:
76.3
LeVIT 1288
77.3
Performance of DTN on larger models. we employ DTN on some large-scale models, including Swin-S, PVT-L, and PVTv2-B3 (Wang et al. 2021a). Note that these models either have large sizes (> 45M) or attain outstanding top-1 accuracy (> 82.5%) on ImageNet. As shown in Table | our DTN achieves consistent gains on top of these large models. For example, DTN improves the plain Swin-S and PVT-L by 0.5% and 0.6% top-1 accuracy, respectively. For the improved version of the PVT model, e.g.. PVTvv-B33 WWang et al., (021a), we also observe the performance gain (+0.55 top-1 accuracy).
How RPE (i.c. R in (qn.(6) affects the performance of DTN. We clarify that our DTN is normalization component that helps a variety of transformer models (ViT, PVT. Swin, TiT-ViT, relative
some of the models positional encoding. To separate the effect of RPE in Egn. from DTN, we conduct two ablation studies on Imagenet shown in Table I0 (a) We put RPE in Eqn. 6 into the self-attention module of ViT and PVT and use LN to normalize the tokens. Note that neither ViT nor PVT uses RPE in their implementation. Hence, ablation (a) can evaluate how RPE in Eqn. affects the performance of the original model. We also investigate the effect of RPE in Eqn.66) by removing it from DTN; (b) We plug DTN into vision transformers using carefully-designe
BigBird, Reformer, etc.) learn better token representation,
RPE such as Le Le eet al., (022)) and SWin (Liu et al. 202T), which demonstrates that DTN can still improve the models with RPE. Results. From ablation (a) Table [O] we can see that a naive RPE used in Eqn. has a marginal improvement when directly put into the MHSA module. However, the performance boosts a lot when such simple RPE is used in our DTN as did (q.. We also see that removing RPE in Eqn.(6) has limited influence on the performance of DTTN Moreover, our DTN can still improve those models with carefully-dssigne RPE as shown by ablation (b) in Table 10 demonstrating the versatility of DTN as normalization technique.
Results on downstream task. We assess the generalization of ourDTN on detectiion/segmenaaiin track using the COC02017 dataset [Lin et al. (2014). We train our model the union of 80k training images and 35k validation images and report the performance on the mini-val 5k images. Mask R-CNN and RetinaNet are used the base detection/segmeetttionn framework. The standard COCO metrics of Average Precision (AP) for bounding box detection (APbb) and instance segmen- tation (APm) is used to evaluate our methods. We use MMDetection training framework with PVT models as basic backbones and all the hyper-parameterss are the same as Wang et al. (202Ib). Table I shows the detection and segmentation results. The results show that compared with vanilla LN, our DTN block can consistently improve the performance. For example. our DTN with PVT-Tiny is 1.4 AP higher in detection and 1.2 AP higher in segmentation than LN. To sum up. these experiments demonstrate the generalization ability of our DTN in dense tasks that require local context.
The effect of DTN on the token direction and magnitude. In our DTN, we have an interesting observation about the learning of direction and magnitude. We perform PCA visualization of tokens before and after normalization. The projected tokens in first two principle components are visualized in Fig Results are obtained on the first layer of trained ViT-S with LN and DTN, respectively. It can be seen that the PCA projected tokens after DTN are closer to each other than LN. It implies that DTN encourages tokens to learn a less diverse direction than LN since tokens normalized by DTN have already presented diverse magnitudes. As we know, learning diverse magnitudes (1 Dimension) could be easier than learning diverse directions (C-1 Dimension). Hence, our DTN can reduce the
17

---

Published as a conference paper at ICLR 2022
Table 11:
Object detection performance on COCO val2017 using Mask R-CNN and RetinaNet.
DTN can consistently improve both AP and box AP by a large margin.
RetinaNet Ix
Mask R-CNN
Backbone
Norm.
AP
AP7s
APs
APM
APr.
Ap
AP
AP
Apm
AP
AP
PVT--Tiny
LN
36.7
56.9
38.9
22.6
38.8
50.0
36.7
39.3
35.1
56.7
37.3
PVT-Tiny
DIN
38.0
58.7
40.4
41.2
50.9
38.1
36.3
38.7
PVT-Small
LN
40.4
61.3
43.0
25.0
42.9
55.7
40.4
43.8
37.8
60.1
40.3
PVT-Small"
DTN
41.8
62.9
44.6
26.4
45.3
56.0
41.7
64.5
44.3
39.0
61.2
41.8
Head
Head
Head
Head
Head
Head
Before LN
0
0
0
Figure 9: Visualization of token directions using PCA for LN and DTN. The projected tokens in the first two principle component visualized before and after LN (rows 1-2), and before and after DTN (rows 3-4). We see PCA projected tokens after DTN are closer to each other than LN. implies that DTN encourages tokens to learn less diverse direction than LN. Results obtained on the first layer of trained ViT-S with LN and DTN, respectively.
Table 12: Performance of the proposed DTN evaluated on transformers with local context already induced. DTN can consistently outperform LN with various vision transformers.
Model
Method
FLOPs
Params
Top-I (%)
4.5G
29M
81.2
Swin-T
29M
IN
8.7G
30M
83.0
Swin-S
DTN
9.4G
50M
83.5
LN
305M
7.8M
76.6
LeViT-128S
DTN
320M
7.8M
77.3
IN
6.1G
21.5M
81.7
12T-ViT-14
DIN
6.4G
21.5M
82.4
optimization difficulty in learning diverse token representations. In the experiment, we also find that VIT models with DTN can converge much faster than LN. Understanding the role of token magnitude and direction self-attention modules would meaningful future research direction.
DTN is also effective on those models with local context already induced. Our DTN nor- malization technique can seamlessly applied to transformer models with local context already en- coded. For example, replace LN with our DTN on T2T-ViT (Yuan et al. (021), LeViT (Graham et al. 202T). and Swin. These models introduce positional information, convolutions, or window attention to induce inductive bias. As shown in Table 12 DTN consistently outperform LN on these models. For example, DTN improves plain Swin-T and Swin-S with LN by 0.7% and 0.5% top-1 accuracy, respectively.
18


Reviews:
[{"review_id": "_jGAmvmHYfc", "reviewer": "Reviewer_PWvF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well written and organized.\n2. The performance of popular vision transformer models gain with the proposed DTN, no matter with the ViT model or with the stage-wise model PVT and Swin Transformer;\n3. The proposed DTN can also be combined with other sparse attention modules such as BigBird or Reformer to boost the performance;\n4. The visualization results in Figure 2 confirm that DTN can boost the model to have a relatively small attention distance in early layers.\n\nCons:\n1. As shown in Table 3, the performance of ViT with BN drops a lot. Can you give some more explanations? As BN has proven to be more effective than LN as shown in many CNN networks, I am wondering why it is less effective than LN in vision transformers. The reason that tokens represent different semantics seems not convincing, since contextual information is also very important in vision tasks.\n2. It seems that the initialization of DTN depends on the image size, as it uses position embeddings. As demonstrated in CPVT, such learned or fixed position embedding may not be so effective in testing on variant image sizes, i.e., training with image size 224 and test with image size 384, which is necessary for downstream tasks. As the proposed DTN is also integrated into Swin and PVT, can the authors provide the performance comparison between the origin Swin/PVT and the DTN integrated versions on downstream tasks, such as detection/segmentation?\n3. All experiments are conducted upon models with performance less than 82.5 top-1 accuracy on ImageNet, which makes the utilization of DTN on more powerful models unclear. The authors should consider more experimental results of DTN on other models like Swin-B or PVTv2-B3.\n\n\nAdditional:\n\n1. The notion of \\mu \\ sigma in Figure 4 seems to be put at the wrong place, which makes readers confusing.\n\n2. Just for curiosity, if we initialize the DTN modules with the $lambda$ learned in Figure 4, fix them and train the model from scratch, would that lead to better performance?\n", "summary_of_the_paper": "This paper studies properties in layer normalization and instance normalization, and states that the two normalization operations have their own drawbacks in vision transformers: LN suffers from lacking inductive bias while IN may be affected by the different semantics within the tokens. Accordingly, this paper proposes a new normalization method called DTN and considers both inter- and intra- token normalization into vision transformers. According to the experiments in the paper, the proposed DTN normalization boosts the performance of various vision transformer models on classification with ImageNet dataset as well as robustness with IMAGENET-C and IMAGENET-R.", "summary_of_the_review": "The paper's analysis seems reasonable on some given models. However, it is not clear in the paper whether this finding is still valid for larger models, or whether it can handle inputs with different sizes. I am willing to raise my score if the authors can address my concerns in their response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W2Z7XlUdfu", "reviewer": "Reviewer_j3Et", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+1) Both the motivation and idea are clear. The analysis is reasonable.\n\n+2) The ablation study shows that DTN can improve the performance on some small/middle-scale models.\n\n\n\n- Q1: Can authors evaluate the training or inference throughput? Flops sometimes cannot reflect model speed.\n\n- Q2: DTN only is evaluated with small models (e.g. swin-tiny, vit-small, vit-base, etc.). It may be more robust if DTN can improve performance on some larger models (e.g. ViT-L, Swin-B, Swin-L, PVT-L, etc) or down-stream tasks (detection, segmentation, recognition, etc.).\n\n- Q3: I notice that the relative positional embedding is used in DTN. However, vanilla ViTs use the learnable positional embedding. Does the positional embedding keep same in ablation studies?\n\n- Q4: The motivation is to balance LN and IN in token normalization. It is straightforward to integrate output of LN and output of IN with a balance weight. However, P^h = softmax(R*a^h) is closely related to positional offsets. Therefore I am afraid the improvements mainly come from positional information instead of normalization paradigms. In other words, ViTs can introduce a better positional embedding but does not change LN to do the same thing.", "summary_of_the_paper": "The paper first analyzes the limitation of LN in Transformers and then proposes DTN to capture both long-range dependencies and local positional context. DTN is a unified version that balances LN and IN. Extensive experiments show the effectiveness of the proposed DTN with some small/middle-scale Transformers on the ImageNet.", "summary_of_the_review": "The paper clearly presents the problem and methods. The experiments can be improved to further support the conclusions. Therefore, I rate to 'marginally above the acceptance threshold' now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SLhXeSRyVSp", "reviewer": "Reviewer_AKdH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* The motivation of this work is clear and solid. Normalization plays an essential role in both CNN and Transformer-based models, but it does not attract too much attention for current ViT models. This work analyses some properties of token normalization. In Fig 2 and Fig3 (b), it shows us the importance of token statistics diversity. \n\n* The proposed method is technically sound. Adoptingrelative positional embedding to significantly reduce the extra learnable parameters is quite interesting and effective. \n\n* The paper is clearly written and easy to follow.\n\nWeakness:\n\n* The novelty of dynamic normalization is limited. The technology of dynamic networks has been widely adopted for architecture, activation functions, and normalization, especially for CNN models. The authors should discuss some related works [1].\n\n* The experimental results are not sufficient to support this work. One key motivation of this work is to easily induce inductive bias such as local context, which is more important in some downstream vision tasks such as detection and segmentation. So I think the single result in imagenet classification task is not enough. The author also claims that DTN is easy to be plugged into Swin and PVT, so it would be appreciated to see the experiments on more downstream tasks. \n\n* Minors: in Eq (3), the sigma sum over $\\mu_c^{in}$ should be T, not C.\n\n[1] Luo P, Zhanglin P, Wenqi S, et al. Differentiable dynamic normalization for learning deep representation[C]//International Conference on Machine Learning. PMLR, 2019: 4203-4211.", "summary_of_the_paper": "This paper introduces a dynamic normalization, named Dynamic Token Normalization (DTN), to replace the vanilla layer norm in ViT. It learns to normalize tokens in both intra-token and inter-token manners, enabling Transformers to capture both the global contextual information and the local positional context. Experimental results show that DTN can improve some Vision Transformers in ImageNet Classification and Long ListOps tasks. ", "summary_of_the_review": "Overall, I like the motivation and some technology in this work, but my major concern is the insufficient experiments. Thus I stand in \" marginally below the acceptance threshold\" now and waiting for the author's response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3vpnqdPa47B", "reviewer": "Reviewer_g5rX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The norm component is some important ingredient of ViTs and it is worth further investigation and bringing in our attentions\n2. An extensive study of different token norm methods for ViTs\n3. The writing is clear and easy to understand \n4. The results of DTN is generally good across different benchmarks\n\n**Weaknesses**\n1. Motivation is not clear and strong sufficiently:\n> + Why is the token magnitude so critical for model performance? The objective of norm itself is to restrict the training data (e.g., tokens in ViTs) to some specific range which has been shown to be helpful for model optimization. For learning capacity, the normalised vectors of high-dimension should be well expressive, and hence there is no high necessity to further explore the magnitude dimension (a single dimension).\n>+ In Figure 1, I have these concerns:\n>>- If one considers that (a) LN reduces the token magnitude difference for the red box case, then (b) IN and (c) DTN would increase the difference, rather than *preserve*. \n>>- From (a) it is seen that different heads actually present different patterns. There is no interpretation and insights that why all heads need to be similar in maintaining the token magnitude, and the current bahaviour of LN is not as good or desired. \n>>- Using norm to impose inductive bias (e.g. local context) is some unusual and implicit. How does this compare with existing local window attention e.g. [a], [c], [d], [e].\n\n2. Unclear and inaccurate content\n>+ Global vs. Local attention: It is unreasonble that when the model can model global context (attention), it will be hard to capture the local context. Local context is just part of global context. At least this is not accurate statement.\n>+ With conventional wisdom, norm is usually not considered as the decisive component for choosing between global context and local context. Instead, it is the scope of tokens in computing the attention scores for given a token, i.e., what tokens are used for pairwise attention learning. So this authors need to be further justified this.\n>+ Figure 2: how is the mean attention distance computed? In particular, what is the center of attention?\n\n\n3. Experiments\n>+ What positional embedding is used for baseline such as ViTs, PVTs, and Swin-T? Given that DTN uses the relative positional embedding (RPE) at each block, for a fair comparison, RPE should be applied to baseline as well. This also helps to separate the effect of RPE from the proposed DTN in analysis. Besides, RPE is shown to be beneficial for ViTs in some works [b].\n>+ How many iterations run for each experiment? In some cases, the margin of DTN over LN/BN is not big, so the variation of different runs may become more important.\n>+ As the authors consider DTN as a component that is able to better learn local context, except comparing different normalization designs, more other alternatives (e.g. [a], [c], [d], [e]) that help with local context learning should be considered in comparison. While some of these works are very new, but the authors should at least include some necessary evaluations on this aspect.\n\n**References** \n\n[a] Stand-Alone Self-Attention in Vision Models. NeurIPS 2019.\n\n[b] LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference. arXiv 2021. \n\n[c] Twins: Revisiting the Design of Spatial Attention in Vision Transformers. NeurIPS 2021\n\n[d] Transformer in Transformer. NeurIPS 2021\n\n[e] Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet. ICCV 2021\n\n", "summary_of_the_paper": "This work presents a token normalization method in replacement of Layer Norm (LN) and Instance Norm (IN) for vision transformers (ViTs). The motivation is that the authors find that the common normalization used in most existing ViTs, LN, will reduce the difference in token magnitude, and this may lead to failure of capturing positional context and other inductive bias. Hence, they propose a Dynamic Token Normalization (DTN) component by combining LN and relative positional embedding based transformation. DTN can be plug in varying ViTs e.g., ViT, Swin, PVT. The experiments are done on ImageNet, ImageNet-C, ImageNet-R, and ListOps in supervised learning setting, as well as self-supervised pre-training. ", "summary_of_the_review": "It is an interesting point for research in ViTs. The motivation is not sufficiently clear and solid. Some of the claims are not precise as stated above. In the experiments some important details and comparisons are missing, while a variety of experiments have been included including self-supervised learning. \n\nOverall, in the current form this submission is not strong enough for acceptance. I pretty much vote a rejection rate. However, more information from the authors is needed to make a final recommendation. \n\n\n******* Post-rebuttal Update *********\n\nThe authors have well resolved all of my concerns with additional interpretation and experiments.\nOverall, this work is novel in taking the normalization aspect for resolving the limitations of existing ViTs in exploring local context, and showing consistent gain for SOTA ViTs, based on interesting observations and solid design. Therefore, I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A new method for dynamic token normalization in ViTs (both within and across tokens) is introduced in the paper. As noted by the reviewers, the proposed method is technically sound, with a clear and solid motivation. The main raised concerns included the lack of experiments using larger models, unclear reason for the accuracy gains, and lack of experiments on other tasks beyond classification, such as detection and segmentation. The authors’ response was strong, clarifying other questions and providing additional experiments, for example, showing the effectiveness of the method on object detection, and when applied to larger models or architectures that explicitly model local context. Two reviewers recommend borderline rejection, but they did not participate in the discussion nor updated their reviews after the author response. The AC considers that their concerns were adequately addressed by the rebuttal, and agrees with the other two reviewers that the paper passes the acceptance bar of ICLR. The authors should carefully proofread the paper for the final version.
Decision: Accept (Poster)