Dual PatchNorm
Authors: Manoj Kumar
Mostafa Dehghani
Neil Houlsby
Venue: Accepted by TMLR ()

Abstract: We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments on image classification and contrastive learning, incorporating this trivial modification, often leads to improved accuracy over well-tuned vanilla Vision Transformers and never hurts.

Published in Transactions on Machine Learning Research (05/2023)
Dual PatchNorm
Manoj Kumar
muchcoader@google.com
Mostafa Dehghani
dehghoni@@google.com
Neil Houlsby
neilhoulsby wwwgllsby@gggge.coo
Google Research, Brain Team
Reviewed on OpenReview:
https: // // openreview. net/ forum? id= jgMque6@hw
Abstract
We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual Patch- Norm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments on image classification, con- trastive learning, semantic segmentation and transfer on downstream classification datasets, incorporating this trivial modification, often leads improved accuracy over well-tuned vanilla Vision Transformers and never hurts.
1 Introduction
Layer Normalization (Ba et al. 2016) is key to Translormer's success in achieving both stable training and high performance across a range of tasks. Such normalization also crucial Vision Transformers
(ViT) (Dosovitskiy et al. 2020 Touvron et al. 2021 which closely follow the standard recipe of the original Transformer model.
Following the "pre-I."" strategy in Baevski Auli (019) and Xiong et al. (2020) ViTs place NayerNorms before the self-attention layer and MLP layer in each Transformer block. We explore the following question: Can we improve ViT models with a different LayerNorm ordering? First, across five ViT architectures on ImageNet-1k 1k (Russakovsky et al. 2015), we demonstrate that an exhaustive search of LayerNorm placements between the components of a Transformer block does not improve classification accuracy. This indicates that the Pre-LN strategy in ViT is close to optimal. Our observation also applies other alternate LayerNorm placements: NormFormer (Shleifer et al., (021] and Sub-LN (Wang al. 2022), which in isolation, do not improve over strong ViT classification models.
Second, make an intriguing observation: placing additional LayerNorms before and after the slandard ViT-projection layer, which we call Dual PatchNorm (DPN), can improve significantly over well tuned vanilla VIT baselines. Our experiments on image classification across three different datasets with varying number of examples and contrastive learning, demonstrate the efficacy of DPN. Interestingly, our qualitative experiments show that the LayerNorm scale parameters upweight. the pixels at the center and corners of each patch.
. up patch_size [0], [atch_size einops rearrange
x, "b (ht hp) (st up)
(ht wt) (hp up c)", hp=hp, up=up)
Dense (nane= (x)
name- " "denee") (x)
Dual PatchNorm consists of 2 line change to the standard ViT-projection layer.

---

Published in Transactions on Machine Learning Research (05/2023)
2 Related Work
Kim et al. 2021] add LayerNorm after the patch-embedding and show that this improves the robustness of ViT against corruptions on small-scale datasets. Xiao et al. (2021) replace the standard Transformer convolutions with batch normalizations and show
stem with a small number of stacked stride-two
that this improves the sensitivity to optimization hyperparamcters and final accuracy. Xu al. (2019)
analyze LayerNorm and show that the derivatives of mean and variance have a greater contribution linal performance opposed to forward normalization. Beyer et al. (2022a) consider Image-LN and Patch-LN as alternative strategies to efficiently train a single model for different patch sizes. [Wang et al. (2022) add
extra LayerNorms before the final dense projection in the self-attention block and the non-lincarity in the MLP block, with a different. initialization strategy. [Shleieer et. al. (2021) propose extra LayerNorms after the final dense projection the self-attention block instead with a LayerNorm after the non-linearity in the MLP block. Unlike previous work, we show that LayerNorms before and after the embodding layer provide consistent improvements classification and contrastive learning tasks. An orthogonal line of work {Liu et al. 2021 d' d'Ascoli et al. 2021 Wang et al. 2021) involves incorporating convolutional inductive biases to Vision Transformers. Here, we exclusively and extensively study LayerNorm placements of vanilla ViT.
3
Background
3.1 Patch Embedding Layer in Vision Transformer
Vision Transformers (Dosovitskiy et al. 2020 consist a patch embedding layer (PE) followed by a stack of Transformer blocks. The PE layer first rearranges the image I E R #xx'x3 into a sequence of patches 2p E R Ft xp² where P denotes the patch size. It then projects each patch independently with dense projection to constitute sequence of "visual tokens' X₃ E R FT P controls the tradc-off between granularity of the visual tokens and the computational cost in the subsequent Transformer layers.
3.2 Layer Normalization
Given a sequence patches x E R NxD, LayerNorm as applied in ViTs consist of two operations:
X x p(x)
X
(1)
(xx)
= 7x +
(2)
where p(x: RRR.No (o()) E RM .Y RD, ,8 RD.
normalizes each patch E RD of the sequence to have mean and unit standard deviation. Then, Eq 2 applies learnable shifts and scales 3 and which are shared across all patches.
First, Eq.
4
Methods
4.1 Alternate LayerNorm placements:
Following Baevski & Auli 2019 and Xiong et al. (2020) ViTs incorporate LayerNorm before every self- attention and MLP layer, commonly known as the pre-LN strategy. For each of the self-attention and MLP layer, we evaluate 3 strategies: place LayerNorm before (pre-LN), after (post-LNN, before and after ppse+toot-LNN) leading nine different combinations.
4.2 Dual PatchNorm
Instead of adding LayerNorms to the Transformer block, we also propose apply LayerNorms in the stem alone, both before and after the patch embedding layer. In partieular, replace
2

---

Published in Transactions on Machine Learning Research (05/2023)
x= PF(()
(3)
with
LN(PE(LN(x)))
(4)
and keep the rest of the architecture fixed. We call this Dual PatchNorm (DPN).
5
Experiments on ImageNet Classification
5.1 Setup
We adopt the standard formulation of Vision Transformers (Sec. 3.1 which has shown broad applicability across a number of vision tasks. We train ViT architectures (with and without DPN) in supervised fashion on 3 different datasets with varying number of examples: ImageNet-1k (1M), ImageNet-21k (21M) and JFT (4B) (Zhai et al. 2022a).. In our experiments, we apply DPN directly on top of the baseline ViT recipes without additional hyperparamter tuning. We split the ImageNet train set into a train and validation split, and the validation split to arrive at the final DPN recipe.
ImageNet 1k: We train 5 architectures: Ti/16, S/16, S/32, B/16 and B/32 using the AugReg (Steiner et al. 2022) recipe for 93000 steps with a batch size of 4096 and report the accuracy on the official ImageNet validation split as is standard practice. The AugReg recipe provides the optimal mixup regularization (Zhang
et al. 2017) and RandAugment (Cubuk et al. 2020 for each ViT backbone. Further, we evaluate S/16
baseline (S/16+) with additional extensive hyperparameter tuning on ImageNet (Beyer et al.
we also apply DPN on top of the base and small DeiT variants (Touvron et al.) 2021). ). Our full set of hyperparameters are available in Appendix and Appendix D
ImageNett 21k: We adopt a similar setup as in ImageNet 1k. We report ImageNet 25 shot accuracies in two training regimes: 93K and 930K steps.
JFT: We evaluate the ImageNet 25 shot accuracies of 3 variants (B/32, B/16 and L/16) on 2 training regimes: (220K and 1.1M steps) with a batch size of 4096. Tn this setup, we do not use any additional data augmentation or mixup regularization.
On ImageNet- 1k, we report the 95% confidence interval across atleast 3 independent runs. On ImageNet-211 and JFT, because of expensive training runs, we train each model once and report the mean 25 shot accuracy with 95% confideece interval across 3 random seeds.
5.2 DPN versus alternate LayerNorm placements
Each Transformer block in ViT consists of a self-attention (SA) and MLP layer. Following the pre-LN strategy (Xiong et al., 2020), LN inserted before both the SA and MLP layers. We first. show that the default pre-LN strategy in ViT models is close to optimal by evaluating alternate LN placements on ImageNet-Ik. We then contrast this with the performance of NormFormer, Sub-LN and DPN.
For each SA and MLP layer, evaluate three LN placements: Pre, Post and Pre+Post, that leads to nine total LN placement configurations. Additionally, we evaluate the LayerNorm placements in NormFormer (Shleifer et al. 2021h and Sub LayerNorm (Wang et al., 2022) which add additional LayerNorms within each of the self- attention and MLP layers in the transformer block. Figure [ shows that none of the placements outperform the default Pre-LN strategy significantly, indicating that the default pre-LN strategy is close optimal. NormFormer provides some improvements on ViT models with a patch size of 32. DPN on the other-hand provides consistent improvements across all architectures.
3

---

Published in Transactions on Machine Learning Research (05/2023)
2.0
Dual PatchNorm
NormFomee
1.5
+
Sub LayerNom
Other Placement
Accuracy Gain
1.0
+
0.5
+
0.0
-0.5
-1.
S/16
B/16
B/32
Ti/16
S/32
Figure 1: The plot displays the accuracy gains of different LayerNorm placement strategics over the default pre-LN strategy. Each blue point (Other LN placement) corresponds to different LN placement in the Transformer block. None of the placements outperform the default. Pre-LN strategy on TmageNet-1k 1k [Russakovsky et al. 2015 architectures.
Applying DPN (black cross) provides consistent improvements across all 5
Arch
Base
DPN
Arch
Base
DPN
VIT AugReg
93K Steps
S/32
72.1 i 0.07
74.0 0.009
Ti/16
72.5
73.9
Ti/16
52.2 : 0.07
53.6 1 0.07
B/32
76.2 0.06
S/32
54.1 = 0.03
56. + 0.03
S/16
78.6
79.7 0.2
B/32
60.9 1: 0.03
63. 1: 0.03
S/16+
79.7 : 0.09
80.2 0.03
S/16
6.13
65.0 + 0.06
B/16
80.4 0.06
81.1 0.09
B/16
70.8 + 0.09
72.0 + 0.03
DeiT
930K Steps
S/16
80.1 0.00
80.4 0.06
Ti/16
6.00
61.2 +
B/16
0.08
82.0 0.05
S/32
63.8
65. 0.112
B/32
73.1 7..8
AugReg
+ 384 x 384 Finetune
S/16
72.5 1.1.1
72.5 1.1.1
B/32
79.0 0.00
80.0 + 0.03
B/16
78.0
7.04
B/16
82.2 0.03
82.8 + 0.00
Table 1: Left: ImageNet-lk validation accuracies of five ViT architectures with and without dual patch norm after 93000 steps. Right: We train ViT models ImageNet-211 in two training regimes: 93k and 930k steps with batch size of 4096. The table shows their ImageNet 25 shot accuracies with and without Dual PatchNorm
5.3 Comparison to ViT
In Table I left, DPN improved the accuracy of B/16, the best VIT model by 0.7 while S/32 obtains the maximum accuracy gain of 1.9. The average gain across all architecture is 1.4. On top of DeïT-S and 0.3 and 0.2 respectively. Further, we finetune B/16 and B/32 models with and without DPN on high resolution ImageNet (384 x 384) for 5000 steps with a batch-size of 512 (See Appendix D for the full hyperparameter setting). Applying DPN improves high-res, finetuned B/16 and B/32 by 0.6 and 1.0 respectively.
DeiT-B, DPN provides an improvement
4

---

Published in Transactions on Machine Learning Research (05/2023)
DPN improves all architectures trained ImageNet-21k (Table 1 Right) and JFT (Table 2 on shorter training regimes with average gains of 1.7 and 0.8 respectively. On longer training regimes, DPN improves the accuracy of the best-performing architectures on JFT and ImageNet-21k by 0.5 and 0.4 respectively.
In three cases, Ti/16 and S/32 with ImageNet-2111 and B/16 with JFT, DPN matches or leads to marginally worse results than the baseline. Nevertheless, across a large fraction of ViT models, simply employing DPN out-of-the-box on top of well-tuned ViT baselines lead to significant improvements.
5.4 Finetuning on ImageNet with DPN
We finetune four models trained on JFT-4B with two resolutions on ImageNet-lk: (B/32, B/16) x (220K, 1.1M) steps on resolutions 224 x 224 and 384 x 384. On B/32 we observe consistent improvement across all configurations. With L/16, DPN outperforms the baseline on 3 out of 4 configurations.
Arch
Base
DPN
Arch
Resolution
Steps
Base
DPN
220K steps
B/32
224
220K
77.6
78.3 + 0.00
B/32
63.8 0.03
65.2 1: 0.03
B/32
384
220K
3..3
81.6 1: 0.00
B/16
72.4 0.07
B/32
224
1.1M
0.1
81.3 = 0.00
L/16
7
77.9 = 0.006
B/32
384
1.1M
83.8 - 0.03
84.1 + 0.00
1.1M steps
L/16
224
220K
84.9
85.3 = 0.03
B/32
70.7 0.1
71.1 0.09
L/16
384
220K
86.7 0.03
87.0 + 0.00
B/16
76.9 = 0.03
76.6 0.03
L/16
224
1.1M
86.7 1 0.03
87.1 1: 0.00
L/16
80.9 +.08
81.4 0.06
0.06
L/16
384
1.1M
88.2 + 0.00
88.3 = 0.06
Table 2: Left: We train 3 ViT models on JFT-4B two training regimes: 200K and 1.1M steps with a batch size 4096. The table displays their ImageNet 25 shot accuracies with and without DPN. Right: Corresponding full finetuneing results on ImageNet-1k.
6 Experiments on Downstream Tasks
6.1 Finetuning on VTAB
We finetune Smageeee--reerrinned B/16 and B/32 with and without DPN on the Visual Task Adaption benchmark (VTAB) (Zhai al. 2019). VTAB consists of 19 datasets: 7 Natural, 4 Specialized and Structured Natural consist of datasets with natural images captured with standard cameras, Specialized has images captured with specialized equipment and Structured require scene comprehension. We use the VTAB training protocol which defines a standard train split of 800 examples and a validation split of 200 examples per dataset. We perform a lightweight sweep across 3 learning rates on each dataset and use the mean validation accuracy across seeds to pick the best model. Appendix B references the standard VTAB finetuning configuration. We then report the corresponding mean test score across 3 seeds in Table 3 In Table 3 accuracies within 95% confidence interval are not boldod.
On Natural which has datasets closest to the source dataset ImageNet, B/32 and B/16 with DPN significantly outperform the baseline on out of 7 and 6 out of 7 datasets respectively. Sun397 (Xiao et al. 2010 is the only dataset where applying DPN performs worse In Appendix F we additionally show that DPN helps when B/16 is trained from scratch on Sun397. Applying DPN on Structured improves accuracy on 4 out 8 datasets and remains neutral on 2 on both B/16 and B/32. On Specialized, DPN improves on out datasots, and is neutral on 2. To conclude, DPN offers the biggest improvements, when finctuned on Natural On Structured and Specialized. DPN is a lightweight alternative, that can help or least not hurt majority of datasets.
5

---

Published in Transactions on Machine Learning Research (05/2023)
Flowers 102
Retinopathy
Calttehh111
CIFAR-101
Camelyon
EuroSAT
Resise45
Sun397
SVVS
DTD
Pets
B/32
87.1
53.7
56.0
83.9
87.2
32.0
76.8
77.9
94.8
78.2
71.2
DPN
87.7
58.1
60.7
86.4
88.0
35.4
80.3
78.5
95.0
81.6
70.3
B/16
86.1
35.5
60.1
90.8
90.9
33.9
76.7
81.3
95.9
81.2
74.7
DPN
86.6
51.4
63.1
91.3
92.1
32.5
78.3
80.6
95.8
83.5
73.3
Clevr- Count
KITTI-Dist
&NORB--AZI Azim
&NORB- Elev
Clevr-Dist Dist.
dSpr-L.cc
dSpr-Ori
DMLab
B/32
58.3
52.6
39.2
71.3
59.8
73.6
20.7
47.2
DPN
62.5
55.5
40.7
60.8
61.6
73.4
20.9
34.4
B/16
65.2
59.8
39.7
72.1
61.9
81.3
18.9
50.4
DPN
73.7
48.3
41.0
72.4
63.0
80.6
21.6
36.2
Table 3: We evaluate DPN VTAB Zhai et al. 2019
When finetuned on Natural B/32 and B/16
with DPN significantly outperform the baseline 7 out of and 6 out of 7 datasets respectively. On
Structured DPN improves both B/16 and B/32 on 4 out of 8 datasets and remains neutral on 2. On Specialized, DPN improves on 1 out of 4 datascts, and is neutral on 2.
6.2 Contrastive Learning
We apply DPN on image-text contrastive learning (Radford et al. h.21h. Each minibatch consists of a set of image and text pairs. We train a text and image encoder to map an image to its correct text over all other texts minibatch. Specifically, we adopt LiT (Zhai al. 2022b,, where we initialize and freeze the image encoder from pretrained checkpoint and train the text encoder from scratch. To evaluate zero-shot TmageNet accuracy, represent each ImageNet class by its text label, which the text encoder maps into class embedding. For given image embedding, the prediction is the class corresponding to the nearest class embedding.
We evalute frozen image encoders: 2 architectures (B/32 and I./16) trained with 2 schedules (220K and 1.1M steps). We resue standard hyperparameters and train only the text encoder using a contrastive loss for 55000 steps with batch-size 16384. T shows that on B/32, DPN improves over the baselines on both the setups while on L/16 DPN provides improvement when the image encoder trained with shorter training schedules.
6.3 Semantic Segmentation
We finetune ImageNet-pretrained B/16 with and without DPN on the ADE-20K 512 512 Zhou et al. 2019) semantic segmentation task. Following (ttuddl al. (2021), a single dense layer maps the ViT features into per-patch output logits. A bilinear upsampling layer then transforms the output distribution into the final high resolution 512 x 512 semantic segmentation output. We finetune the entire ViT backbone with standard

---

Published in Transactions on Machine Learning Research (05/2023)
Arch
Steps
Base
DPN
B/32
220K
61.9 0.11
63.0 0.09
B/32
1.1M
0.04
68.0 0.009
T/16
220K
75.0
75.4 0.00
L/16
1.1M
7..7
78.7 0.1
Table 4: Zero Shot ImageNet accuracy on the LiT ZZai et al.
2022b contrastive learning setup.
Fraction of Train Data
1/16
1/8
1/4
1/2
1
B/16 +DPN
27.3
32.6 + 0.09
36.9 + 0.13
0.1
0.08
28.0 0.21
33.7 0.11
38.0
00.1
41.9
= 0.09
46.1 - 0.11
Table 5: We finetune ImageNet pretrained B/16 models with and without DPN on the ADE20K Semantic Segmentation task, when varying fraction of ADE20K training data is available. The table reports the mean IoU across ten random seeds. Applying DPN improves loU across all settings.
per-pixcl cross-cntropy loss. Appendix G specifies the full set of finctuning hyperparameters. Table 5 reports the mean mIOU across 10 random seeds and different fractions of training data. The improvement in ToU is consistent across all setups.
7 Ablations
Is normalizing both the inputs and outputs of the embedding layer optimal? In Eq DPN applies LN to both the inputs and outputs to the embedding layer. We assess three alternate strategies: Pre, Post and Post PosEmb (Radford et al. 2021). Pre applics LayerNorm only to the inputs, Post only to the outputs and Post PosEmb to the oulputs after being summed with positional embeddings.
Table 6 displays the accuracy gains with two alternate strategies: Pre is unstable on B/32 leading to a significant drop in accuracy. Additionally, Pre obtains minor drops accuracy on S/32 and Ti/16. Post and Post PosEmb achieve worse performance on smaller models B/32, S/32 and Ti/ /6. Our experiments show that applying LayerNorm to both inputs and outputs of the embedding layer is necessary to obtain consistent improvements in accuracy across all VIT variants.
B/16
S/16
B/32
S/32
Ti/16
Pre
-0.1
0.0
-2.6
-0.2
-0.3
Post
0.0
-0.2
-0.5
-0.7
-1.1
Post PosEmb
0.0
-0.1
-0.4
-0.9
-1.1
Only learnable
-0.8
-0.9
-1.2
-1.6
-1.6
RMSNorm
0.0
-0.1
-0.4
-0.5
-1.7
No learnable
-0.5
0.0
-0.2
-0.1
-0.1
Table 6: Ablations of various components of DPN. Pre: LayerNorm only to the inputs of the embedding layer. Post: LayerNorm only to the outputs of the embedding layer. No learnable: Per-patch normal- ization without learnable LayerNorm parameters. Only learnable: Learnable scales and shifts without standardization.
Normalization vs Learnable Parameters: As seen in Sec. 3.2 LayerNorm constitutes a normalization operation followed by learnable scales and shifts. We also ablate the effect of each of these operations DPN.

---

Published in Transactions on Machine Learning Research (05/2023)
Applying only learnable scales and shifts without normalization leads to significant decrease in accuracy across all architectures. (See: Only learnable in Table 6 Additionally, removing the learnable parameters leads to unstable training on B/16 (No learnable in Table 6 Finally, removing the centering and bias parameters as done in RMSNorm (Zhang & Sennrich 2019 reduces the accuracy B/32, S/32 and Ti/16. We conclude that while both normalization and learnable paramelers contribute to the success of DPN, normalization has a higher impact.
8 Analysis
8.1
Gradient Norm Scale
Base DPN
1.0
0.8
Gradient Norm
0.6
Gradient Norm
0.8
Mm المبيم المة
0.6
0.4
شحمة معمة
0.4
0.2
0.2
0.0
stem
Block: 1
Block: 2-
Block: 3
Block: 4
Block: 7
Block: 8-
Block: 9
Block: 10.
Block: 11
Base
Block:
Block:
Block:
0.0
DPN
20000
40000 60000
80000
Number of Steps
(a)
Figure Gradient Norms with and without DPN in B/16. Left: Gradient Norm Depth. Right:
Gradient Norm of the embedding layer number of steps.
We report per-layer gradient norms with and without DPN on B/16. Fig. 2 (Leftt) plots the mean gradient norm of the last 1000 training steps as function of depth with and without DPN. Interestingly, the gradient norm of the base ViT patch embedding (black) is disproportionately large compared to the other layers. Applying DPN (red), on the other hand, scales down the gradient norm the embedding layer. Fig. g (Right) additionally shows that the gradient norm of the embedding layer is reduced not only before convergence but also throughout the course of training. This property is consistent across VIT architectures of different sizes (Appendix
8.2 Visualizing Scale Parameters
Note that the first LayerNorm in Eq. 4 is applied directly on patches, that is, to raw pixels. Thus, the learnable parameters (biases and scales) of the first. LayerNorm can be visualized directly in pixel space. Fig. shows the scales of our smallest model and largest model which are: Ti/16 trained on ImageNet for 90000 steps and L/16 trained JFT for 1.1M steps respectively. Since the absolute magnitude of the scale parameters vary across the R, G and B channel, we visualize the scale separately for each channel. Interestingly, for both models the scale parameter increases the weight of the pixels in the center of the patch and at the corners.
9 Conclusion
We propose a simple modification vanilla ViT models and show its efficacy on classification, contrastive learning, semantic segmentation and transfer to small classification datasets.
8

---

Published in Transactions on Machine Learning Research (05/2023)
TV/6: R
TV/6: G
TV/16:
-1.4
1.6
1.2
3
1.2
3
1.4
1.1
5
5
1.0
7-
1.0
7
1.2
0.9
0.8
1.0
0.8
0.7
11-
11
11
0.8
0.6
13
13
13
0.6
0.5
15
15
0.6
15
13
3
11
(b)
(c)
L/16:
2.0
1
1..4
1
1.4
3
1.2
1.8
1.2
5
1.6
1.0
7
1.0
7-
9-
9-
1.4
9-
0.8
0.8
11-
11
1.2
11
0.6
13
0.6
13
13
1.0
0.4
15
15
3
5
7
11 13
15
3
9
11
13
15
1
3
7
9
11
13
15
(d)
(e)
(f)
Figure 3: Visualization scale parameters of the first LayerNorm. Bottom: L/16 trained on JFT-4B
Top: Ti/16 trained on ImageNet 1k.
References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. arXiv:1607.06450 2016.
Layer normalization.
arXiv preprint
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. ICLR, 2019.
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. arXiv preprint 22Xi::222.00011. 2022a.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit basclines for imagenet- 1k. arXiv preprinl ar xrXii:2205.00 01500000, 2022b.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision.
https:://githbb..oo com/
google-researchDnn vision 2022c.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reducod scarch space. Advances in Neural Information Processing Systems, 33: 18613-18624, 2020.
Mostafa Dehghani, Alexcy Gritsenko, Amurag Arnab, Matthias Minderer, and Yi Tay. Scenie: A jax library for computer vision research and beyond. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 21393 21398, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Rep- resentations, 2020.

---

Published in Transactions on Machine Learning Research (05/2023)
Stéphane d'Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference Machine Learning, pp. 2286 2296. PMLR, 2021.
Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang, Dong Gu Lee, Wonseok Jeong, and Sang Woo Kim. Improved robustness of vision transformer via prelayernorm in patch embedding. arXiv preprint arXiv:211 11.22111104440 1.084 13,
2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IBEE/CVF international conference on computer vision, pp.
2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Alex Rogozhnikov. Einops:
Clear and reliable tensor manipulations with cinstein-like notation. In In- lernational Conference on Learning Representations, 2022. URL htttps///operreviieennefffoourue oapKSVM2bc
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3))211-22,, 2015.
Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization. arxiv preprint arXiv:2110.09.0 09456, 2021.
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lueas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Mochine Learning Research, 2022. URL https://openrevier http::///opooorrooooooceorrroooo
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. Tn Proceedings of the IFFE/CVF international conference on computer vision, pp. 7262- 7272, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347 10357. PMLR, 2021.
Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. Foundation transformers. arXiv preprint ar arXi::2210.06; 064222 2022.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVE international conference computer vision, pp. 568-578, 578, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pottern recognition, pp. 3485-3492. IEEE, 2010.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in Neural Information Processing Systems, 34:30392 30400, 2021.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International PMLR, 2020.
Conference Machine Learning, pp.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Published in Transactions on Machine Learning Research (05/2023)
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of repre- sentation learning with the visual task adaptation benchmark. arXiv preprint (xX0::9100004677, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. Tn CVPR, 2022a.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with lockod-image text tuning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 18123 18133, 2022b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Newral Information Processing Systems, 32, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint orXiv:1710. 0710.0941.. 09412, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela. Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k datasct. International Journal of Computer Vision, 127:302 321, 2019.
A Initial Project Idea
We arrived at the Dual PatchNorm solution because of another project that explored adding whitened (decorrelated) patches to ViT. Our initial prototype had LayerNorm right after the decorrelated patches, to ensure that they are of an appropriate scale. This lead to improvements across multiple benchmarks, suggesting that whitened patches can improve image classification. We later found out via ablations, that just LayerNorm sufficient the inputs and adding whitened patches on their own could degrade performance. Our paper highlights the need for rigorous ablations of complicated algorithms to arrive at simpler solutions more effective.
which can be equally
B Code
We perform all our experiments in the big-vision (Beyer et al. 2022c) and Scenic (Dehghani et al. 2022) library. Since the first LayerNorm of DPN is directly applied pixels, we replace the first convolution with patchify operation implemented with the cinops (Rogozhnikov 2022 library and dense projection.
c ViT AugReg: Training Configurations
import big_vision confign. common bvcc from big_vision configs. common_feushie import get_feushot__l import nl_collections mlc
RANDAUG DEF "onne"
'lighti': "aanaaug (2
'light2': 'randaug (2,10)
'mediumi':
'randaug (2)
'medium2': 'randaug (2,15) 15)
'strongg's 'randaug (2)
strrong': 'randaug (2 20)
MIXUP_DES
17
'none' dict (p=0.0, fold_ ind_in=)o)e),
18
'lighti's dict (p=0.00000, fold. hold_in-Nons
18
'light2': diet (p=0.2, fold [id_iinNone))
11

---

Published in Transactions on Machine Learning Research (05/2023)
mediuml's : dict(p=0.2 fold_in-None
21
meddum2': diet(p=0.5 fold_in-Mone),
sstrong': diet (p=0.,, fold_in-None),
23
"strong2: dict 8 fold_in-None),
24
26
28
27
def get__cott cet_coofig((i (arg=None);
28
Config for training."" *"
29
arg bvec. parse, parse_argg arg (arg, variant='B/32 runlocal=False, aug="))
config mlc. Configgiet)) 30
31
config pP_modules L'ops_genoral', 'ops_inage" ']
config. cait_head_bias = -6.9
variant 'B/16'
aug_netting
arg aug {
37
'T1/16':
'lightl',
38
19/32"
'mediuml"
39
19/16"
'medium2
40
1B/32':
'medium2
41
'B/16"
'medium2'
42
'L/16':
"medium2"
43
[variant]
44
45
config input = dict
46
config. data dict(
47
name imagenet2012
48
tplit=
50
config. bitch 4096
config. capput.ccche_ru True
config. input. shuffle shuffle_buffer buffer size 250_000
Pp_common
range (-1, 1)
'Jonehot (1000, key="(1b1}", rey_ ("ingee"" "labels")'
57
58
50
60
config. config.iaput.r
61
"decode_jpeg_ jateee_peeaaddd..dee_eee
ott.:pppseed.ttnetttooooooooo _nneeptiioonnroppppppp (224) [flip_irl'
62
RANDAUG_DES [aug_setting]
63
pp_common format (ibl='label" )
64
)
65
PP_eva eval ddecode resize small (256) centrall_rrop (224) common
66
config. input prefetch = 8
config.
catggsuun_csssssn classes 1000
config.
losss loss 'sigsoid_xen _entt
totsl_epochs 300
config. soafig.ckpt_aae
72
1000
Nodel section
75
config.nodel_ng wwn.ddde_naee 'vit'
76
config. config.nodel model dict<
77
varlant=varient,
78
rep_eize-True, pool_type='tok tok dropout=0.1, stoch_depth=0.1
70
80
81
52
stem_ln-'dpm')
84
# Optinizer section
config. grad_clip_nor 1.0
config.. optax_name 'scale by bd_ad
config. optax dict (mu, (mu_atype="_/"e bfloati6')
12

---

Published in Transactions on Machine Learning Research (05/2023)
89
config. 0.001
90
config. 0.0001
91
config. seed
92
dict (warmup_steps-10000 decay_types cosine')
93
94
config.aixu MIXUP [aug_setting]
9G
96
# Eval section
97
def get get_eval (split, d inageeet21111
98
return dict(
90
typa='classificationn
data=dict (pameraaattt split=split), pp_in=pp_eval format labl=")abel'),
loss_name soss_amme-cofig...e loss, 2500,, fache_final-mot arg. runloca,,
config. evals
config evals. train get_ ('train [:2X] ')
config evals. minival evalsaniiiiall get_eval ('train [99%:] !)
100
config. val get_eval validation')
110
return config
AugReg Recipe: B/16.
For smaller models (S/32, Ti/16 and S/16), as per the AugReg recipe, we switch off' stochastic depth and dropout. For S/32, also set representation size be false.
ViT AugReg: High Res Finetuning
import ml nl_coolections
def get__coniie config (runlocal=False):
*""Config for adaptation on imagenet.
config Confignfi)
config. loss "sigmoid_xens xentt
config.num nun_classee 1000
10
5000
Doffgg-ppmoddut
d'ops_general', 'ops_inage" ']
config. seed =
config input (
config input data dict(
name= imagenet0022',
split='train (:99%)
18
config. inggg.inut..bbccc.. www.bbbtthhaaa..e config. shuffle.errf..e...rrrr.. ssze 50_000 not runlocal else 100 config. wwwpt.accheeee....ee.eaaa.aaaaee variant 'B/32
if runlocal
20
21
cache_rav rau True
22
23
24
PP_ common
25
paalue_range (-1, 1)
28
'onehot (1000, keb="' key_roeult="labels")) "Ximag""mage" "labels")!
27
config. input. [
ddecode_jpeg.ee
httede_pp__aa___pppttoooooom inwepniionnoroooe (384) [flip_lr]]
pp_common format (ibl='label)) label!)
PP_eva eval decode resize_ssntt small (418) central_crop (384) pp_common
35
config- log_ config. ckpt_ateps 1000
www.ttrrinnnsteeee steps
36
13

---

Published in Transactions on Machine Learning Research (05/2023)
38
config. maili_.odde_aaag 'vit'
39
config. www.moode__itt init 'low_ /eourre/ppath
40
config.moder dict (variant-variant, pool_type=/tot 'tak', sten_ 1ora__nn'ooo 'pn rree
41
42
config wadeg_odde__looed dict dict (doont__ooode ("eea h'head/kernel', 'head/bias '])
43
44
# Optinizer section
46
Soofig.optax_aane 'big_ wtw.vviioooooetuun......m momentun_hp
46
config. grad_ clip__ooo = 1.0
47
config.vd None
48
config. lr 0.03
49
config schedule = dict(
50
warmup_ varmup_step==500, decsy_type- cosine'',,
Eval section
def get_eval (split, dataget=' Ietaaet='inagennee Inagenet2012>>>>
58
diet
57
"yp=icclassiiicationn"
58
data=dict daameedate,,ee split=split),
50
pp_fn=pp_eval (aol
60
loss_ loss_name-corrig.lees log_ 1og_staps=2500, cache_final runlocal,
61
62
68
64
evals = (
65
config evals. train = get_eval ( ('train [:28] !)
config wwwls.ainiiee...........a..aeeee minival get_eval ( ('train [99X:] !)
evals. val = get_eval ('validation')
return config
High Resolution Finetuning
E VTAB Finetuneing
from co.collectoon import ConfigDict
def get_confii(): ()
*." Config for adaptation VTAB. "**
config ConfigDict()
config loss 'sigmoid_xent"
config nun_ classes config total_steps steps 2500
11
config-pp_modules
o'os_ggnerrl', 'aps_inage', 'proj. vtab. wwt.tttaapppppppp...pppppppppppe op_
13
13 config. seed
14
config. input dict
15
config. input data data dict( name='1, split=' [:800] )
16
17
18
19
config. imput batch_aits 512
20
config. shufft buffer size 50_000
21
False
config.iaput
config. bartig.oog.trainngg.ttt 10
coffiglog_eval steps 100
coptstaa 1000
27
config.ckpp..timeou
28
config-prefeteet device
14

---

Published in Transactions on Machine Learning Research (05/2023)
31
Nodel. config.model semmmodde__aae name 'vit'
32
33
stem_in in 'dpn'
34
variant 'B/32'
35
38
config.aode model_ init model_inite [variant] [atem_in]
37
seffgg.ndee dict(
38
varlant=variant.
39
tep_size=True,
40
pool_type='tok
41
stem stem_ln=stem__n1 modiig-nodelload dict (dont_load=[ lotadldddddaadd/keeeel 'head/bias '))
# Optinizer section
config. optax_name "big_vision momentum_hp config. grad_clip_norn 1.0 config.
47
None
48
config.1r 1r 0.0003 config ckpt_timeen
48
50
config schedule diet
51
varmup_steps steuup_step==200,
52
decay_type- cosine',
53
54
5G
return config
High Resolution Finetuning
F SUN397: Train from scratch
On Sun397, applying DPN improves VIT models trained from scratch. We first search for optimal hyperparameter setting across 3 learning rates: le-3, 3e-4, le-4, 2 weight decays: 0.03, 0.1 and two dropout values: 0.0, 0.1. We then searched across 3 mixup values 0.0, 0.2 and 0.5 and 4 randaugment distortion magnitudes 0, 5, 10 and 15. We train the final config for 600 epochs.
Base
DPN
Base
DPN
41.4
47.5
45.6
51.8
Augmentation
48.3
50.7
Augmentation
58.7
63.0
Train Longer
52.5
56.0
Train Longer
60.8
66.3
Table 7: Sun train from scratch. Left: B/32 and Right: B/16
G Semantic Segmentation Hyperparameter
get (
*""Returns the base experiment configuration for Segmentation on ADE20k.
config colceelections ConfigDict config. eewerrieete.iient.. name "linear "Linear_decodes dertaarddeedder_eeeee seerrddeeddrr_ess__eeee0000
# Dataset.
config. dataset_namt poofii.dataset_ume
lrenseg_datasee
ml_collections cta.leectioos.Corrigpitttttt ()
config-dataset_co dataset_confige name = 'ade20k'
config.
config.dataset_com dataset_confige satasettoorigg.usscrrre....eaa
False
config. config.dataset_ dataset_confige train_ train_data_pet 100
mean_atd 485 0.456, 0.406]. 229 0.224, 0. 225]
common
"Itandardize mean_std keep ("inputs", ")
data_key=* ")
config. dataset_ ceatasst_cooffigg.p..tte
17
immegg style, resize (ing_ (2048, (2048, 512), ratio_ Paeio_ranne-(O 50.5, 2.0))
15

---

Published in Transactions on Machine Learning Research (05/2023)
cat_max=0.75 Lgaore__abel=0)
20
Isqueeze (dte
21
22
512
23
24
common
25
config. conigg..bttaet.cooiggg....... =
26
27
laqueeze (data_ (araa
сommon
28
# Nodel.
version, patch splitt config.mode mo_collections. ConfigDict config.moder hidden size ('Ti': 192,
1s: 384,
:B: 768,
'L' 1024, "H': 1280) [version]
38
37
config. model. patches ml_collectione ConfigDict()
38
config.mode model. patches.pptceeeeeeee config. model. seceel.uum_eeee num_heads {'T1'' 3, 's": 6, 'B': 12. "L' 16, 'H': 16) [version]
[int (patch), int (patch)]
39
40
config.mode mip_din ('T1': 768.
41
19": 1536.
42
:B: 3072,
43
'L': 4096,
44
'H': 5120) [version]
46
config model. nun layers ('T1': 12,
46
:s 12,
:B: 12,
LL: 24,
::: 32) [version] 0.0
moneigg..oel..dgooutt...tm 0.0
camiig_.dda..crrpouu.........._ 0.0
cati:g_adbllatcccastic....p. 0.1
cortie_aodel_uyppe_e===== '
cettggaadda.pppnnneeepppaaaaaaaa 'bilinear'
55
56
coniig..odal.poollngg=====
57
False
58
contig.prettrained..pa..
50
config..pretraiieed.maee "dpn_ b16
60
config. (32, 32) 512 / 16
61
= 'learned'
62
config.:odell.peereet Falss
63
config. True
64
config.nodel.auxilla..
-1
out_aath___ttt_aaee norm False
conelg. www.el..see.atccooor. False config.nodel dpn True
# Trainer.
config. trainer_ait 'segmentation_tranl trainer
71
oval False config ovalle_eval config.vindors vindos_stride 341
72
78
74
76
# Optinizer.
78
config. optimizer 'adamy'
77
config.veit veight_decay = 0.01
78
config fonfig.freeze. backbone False
79
config. laakeriseyeriieeeeec. = 0.
shofig.skip_ecee Ba_p___ecle_aa____________ and_aaad_______aaaaaaaaaaaaaaaaa sias_regularizationn - True
config.optinas catiegoppniizzeccttiii - milcollections.co ConfigDict ()
config. batch_size config-batch_size 16
config. nun www.ttrannnn...mmnn epochs 128 config... wtw_naaxa__dedeeoeeeee None
16

---

Published in Transactions on Machine Learning Research (05/2023)
config raaaaa.rbalaacnng.aaea config.clase. config.rng_see
None
87
88
88
90
# Learning rate.
91
20210 // config. config.batch. batch_size
92
config.total_stepp config. www..aaaiiing...oc.....
contig.ateps..er.epoo
98
config.1r_conff1
colcollections ConfigDict((
94
'compound'
96
cortig.:t..caiig..faaccc.. factors cconstatt polynomial linear_waraup
98
cottig..E.coorttee.aauu.... =
97
coffiggli.conttgs..ddae...p
config. totps_steps
98
0.00003
= 0.
config. berrig...cconfig..poo. 0.9
return config
Semantic Segmentation Config
H Gradient Norm Scale
0.8
1.2-
0.7
Gredient Nerm
Gredent Norm
1.0
0.5
0.4
0.6
0.2
0.3
0.4
0.4
0.2
0.2
0.0
Block
Black
Blacks 2
Block 3
Blocke
Black 5
Block
Block
Block
Dlocke 9
Block: 1o
Blode: 12
Block Block 1
Blocks
Block
Blucke 5
Black 6
Block 7
Block: 8
Block 9
Blook
Block
Block: 1
Blocks 2
Blacks 3
Dlockc
Black
Block
Block
Block
Blocke
Block: 10
Blode: 11
(a)
(b)
Figure 4: Gradient Norm vs Depth. Left: B/32. Center: S/32 Right: S/16
17


Reviews:
[{"review_id": "rwq5rp7nXH", "reviewer": "Reviewer_pmPK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) DPN is simple and effective.\n\nWeaknesses:\n\n1) Given the simplicity of DPN, I think the authors should conduct extensive experiments to validate its effectiveness, such that the contributions of this paper will be sufficient to be published. However, the following results are absent. Personally, I'll lean toward accepting this paper if at least two of them are provided, and the results are solid.\n- Results on top of DeiT (Training data-efficient image transformers & distillation through attention, ICML'21), a widely-used, state-of-the-art pipeline for training ViTs efficiently. Providing the results with DeiT III (ECCV'22) would be more impressive.\n- Results on multi-stage ViTs (e.g., Swin or PVT) or ConvNets (e.g., ConvNeXt).\n- Results on downstream tasks (e.g., detection and segmentation on COCO).\n\n2) In addition, this paper may lack some necessary insights. I'm pretty curious about why DPN yields signification improvements. It seems that the current paper does not answer this question. The authors may consider visualizing the scale of gradients in ViTs.\n\n3) In Tables 2 and 3, the gains of DPN are not significant for relatively large models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a dual PatchNorm (DPN) technique to facilitate the effective training of vision Transformers. In specific, two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. Results on ImageNet and JFT demonstrate the effectiveness of DPN.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above.", "broader_impact_concerns": "I have no concerns about the broader impacts and the ethical implications.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GB0dTCWqRH", "reviewer": "Reviewer_oWPk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides a light-weigh, easy to implement extension for transformer based vision models showing some performance improvement, complementing the toolbox of techniques to design and train transformer based models. The paper is well written and structured.\n\nMay major criticism is that there is no motivation / argument for the specific DPN beside the empirical results. Maybe the authors could emphasize the cases in which they improved performance and lay out some argument / rational why DPN improves performance. Do the authors see any restrictions of their approach, e.g. considering other tasks (NLP)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Dual Patch Norm (DPN). A light weight extension to common transformer based vision models that adds LayerNorm layers before and after patch encoding. The work provides some empirical evidence that DPN improves classification performance on image classification and contrastive learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Considering the claims of the paper, the work is already in a good state. Addressing my raised concerns would strengthen the work.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aupNMh8I3L", "reviewer": "Reviewer_tHeX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper is clearly written and straightforward. It shows pseudocode and reports gains in many cases\n* the gains are across different architecture sizes \n\nWeaknesses: \n\n* it only presents results on imagenet (and a non-public propriety dataset) and for testing on the training classes (apart from the non-reproducible JFT finetuning experiment). No transfer experiments are shown, eg to other smaller datasets with classes beyond the ones in the training set, and no experiments for ViTs beyond classification\n* This is an empirical study with no effort to try to explore or understand where the gains are coming from (beyond a visualization of the scale parameter on the first layer)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a simple modification to the normalization strategy used for the patch embedding layer in Vision Transformers (ViTs). \nThe proposed recipe, DualPatchNorm or DPN, is simple: Layer Normalization layers (LayerNorm) are applied before and after the patch embedding layer. The authors show that this modification helps in many scenarios for supervised classification on imagenet and contrastive learning", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* **Transfer learning experiments:** Using the learned models for transfer experiments would show if the added robustness/performance gains that the DPN offers persists for generalization to other datasets (eg fine-tuning or after freezing the model and learning classifiers) \n\n* **Tables with hyper parameters**: To be more reproducible, Tables with all model and optimization hyperparameters used for the experiments (on Imagenet1k and 21k) are needed ( e.g. something similar to Tab11 in the latest arxiv version of the MAE paper [He et al 2022])\n\n* Clarify \"zero-shot ImageNet accuracy\" for the contrastive learning experiments - does this mean 1-NN retrieval? Please add more info on the experimental setup\n\n* The visualization of the scale parameter on the first layer, that is not really explored more deeply - why are the corner patches so important? Why is there such variance across color channels?\n\n* Experiments on other tasks: It would be great to see if this also transfers to ViTs for other tasks, eg object detection or semantic segmentation, where ViT are now used.\n\n[He et al 2022] Masked Autoencoders Are Scalable Vision Learners\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a dual PatchNorm to train vision transformers. It is a simple but effective method, i.e., inserting layer normalization layers before and after the patch embedding layer.  Extensive experiments are conducted to investigate the method's effectiveness. 

In the rebuttal phase, the authors add more explanations and investigate the dual PatchNorm on more tasks. All the reviewers satisfy the responses and recommend accepting this manuscript.  The claim in this paper is supported by convincing experiments and many individuals will be interested in it.

Overall, I recommend accepting this paper.
Decision: Accept (poster)