All Tokens Matter: Token Labeling for Training Better Vision Transformers
Authors: Zihang Jiang
Qibin Hou
Li Yuan
Zhou Daquan
Yujun Shi
Xiaojie Jin
Anran Wang
Jiashi Feng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Image classification
neural network
vision transformer
semantic segmentation

Abstract: In this paper, we present token labeling---a new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViTs that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to compute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specific supervision generated by a machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous models (250M+) reaching 86%. We also show that token labeling can clearly improve the generalization capability of the pretrained models on downstream tasks with dense prediction, such as semantic segmentation.  Our code and model are publicly
available at https://github.com/zihangJiang/TokenLabeling.

All Tokens Matter: Token Labeling for Training Better Vision Transformers
Zihang Jiang'
Qibin 2
Li Yuan
Daquan Zhou'
Yujun Shi
Xiaojie Jin
Anran Wang!
Jiashi Feng*
National University Singapore
2 Nankai University
Peking University
"yteDDance
(jzh0103, andrewhoux, ylustcnus, zhoudaquan21, shi shiyujun1 0166) 0gmaii com xjjin0731 @gmail. com, comm
anran. wwwgwwaeeddneeeeeeeeomooo.o com, jshfeng com
Abstract
In this paper, we present token labeling new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViT's that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to com- pute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specine supervision gen- erated by machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous mod- els (250M+) reaching 86%. also show that token labeling can clearly improve the generalization capability of the pretrained models downstream tasks with dense prediction, such as semantic segmentation. Our code and model are publicly available at https: /ttp::////ttttta com/ hwa.a//aininnnnnookeeaaaaaeeeeee
1 Introduction
Transformers 339 have achieved great performance for almost the natural language processing
(NLP) tasks over the past years 14 24 Motivated by such success, recently, many researchers attempt to build transformer models for vision tasks, and their encouraging results have shown the great potential of transformer based models for image classification 66 [5| 25 36 40 46), especially the strong benefits of the self-attention mechanism in building long-range dependencies between pairs of input tokens.
Despite the importance of gathering long-range dependencies, recent work on local data augmenta- tion [57] has demonstrated that well modeling and leveraging local information for image classifica- tion would avoid biasing the model towards skewed and non-generalizaabe patterns and substantially
Work done as an intern at ByteDance AT Lab. CCoresponniig author. Part of this work was done research fellow at NUS.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

86
LAVIT
GJIT-3660384 (KD)
T2T-VIT
85
CrossviT@4880
ConviT
Top-1 Accuraty 196
CVT-2183844
BoTNet
DelT (KD)
BoTNet-S1
DeepVIT
DeepVIT
83
T2T-VIT
ConVil
DeIT
82
Swin-T
LambdaNet-152
PVT
DeiT (KD)
VIT
81
Bo SoNNNt-S1-59
Swin-T
Lambdallet-152
90
CxT (184)
CrossViT
PVT
20
80
100
CaIT
Model Size (M)
Figure 1: Comparison between the proposed LV-ViT and other recent works based vision trans- formers, including T2T- VIT [46), Con [2]. BoTNet [3I, Vie 59
DeiT [3), VIT [[3], Swin Transformer [25]. LambdaNet M1. CvT [43, CrossViT (6. PVT 40, CaiT [37] Note that
we only show models whose model sizes are under 100M. As can be seen, our LV-ViT achieves the best results using the least amount of learnable parameters. The default test resolution is 224 224 unless specified after @
improve the model performance. However, recent vision transformers normally utilize class tokens that aggregate global information predict the output class while neglecting the role of other patch tokens that encode rich information on their respective local image patches.
In this paper, we present a new training objective for vision transformers, termed token labeling, that takes advantage both the patch tokens and the class tokens. Our method takes a K --dimensional score map generated by machine annotator supervision to supervise all the tokens in a dense manner, where K is the number of categories for the target dataset. this way, each patch token is explicitly associated with an individual location-specific supervision indicating the existence of the target objects inside the corresponding image patch, so as to improve the object grounding and recognition capabilities of vision transformers with negligible computation overhead. To the best of our knowledge, this the first work demonstrating that dense supervision is beneficial to vision transformers in image classification.
According to our experiments, utilizing the proposed token labeling objective can clearly boost the performance of vision transformers. As shown in Figure T our model, named LV-ViT, with 56M parameters, yields 85.4% top-1 accuracy on ImageNet [I3], behaving better than all the other transformer-bast models having more than 100M parameters. When the model size is scaled up to 150M, the result can be further improved to 86.4%. In addition, we have empirically found that the pretrained models with token labeling are also beneficial to downstream tasks with dense prediction, such as semantic segmentation.
2 Related Work
Transformers refer to the models that entirely rely on the self-attention mechanism build global dependencies. which are originally designed for natural language processing tasks. Due to their strong capability of capturing spatial information, transformers have also been successfully applied to variety of vision problems, including low-level vision tasks like image enhancement 7 45 as well as more challenging tasks such as image classification 2 15], object detection [ 55| 61].
segmentation 7 33 and image generation (28). Some works also extend transformers for vidco
and 3D point cloud processing [50] 53 60].
Vision Transformer (ViT) is one of the earlier attempts that achieved tfate-of-the-arr performance on ImageNet classification, using pure transformers as basic building blocks. However, ViTs need pretraining very large datasets, such as ImageNet-222 22k and JFT-300M. and huge computation resources to achieve comparable performance to ResNet [88] with a similar model size trained on ImageNet. Later, DeiT [36] manages to tackle the data-inefficiency problem by simply adjusting the network architecture and adding an additional token along with the class token for Knowledge Distillation [21 47 to improve model performance.

---

Token Labeling
Class
Outpat Tokens
Dense score map
A Sequence of T'Trnsformer Encoders
Patch Tokens
Patch Embedding of Flattened Patches
Class T Token
Input Image
Figure 2: Pipeline of training vision transformers with token labeling. Other than utilizing the class token (pink rectangle), we also take advantage of the output patch tokens (orange rounded rectangle) by assigning each patch token an individual location-specific prediction generated by machine annotator [3] as supervision (see the part in the red dash rectangle). Our proposed token labeling method can be treated
auxiliary objective provide each patch token the local details that aid vision transformers to more accurately locate and recognize the target objects. Note that the traditional vision transformer training does not include the red dash rectangle part.
46 also attempt to introduce the local dependency into vision transformers by modifying the patch embedding block or the transformer block or both, leading to significant performance gains. Moreover, there are also some works 20 25 40 adopting pyramid structure to reduce the overall computation while maintaining the model's 's ability to capture low-level
Some recent works 16
features.
Unlike most aforementioned works that design new transformer blocks or transformer architectures, attempt improve vision transformers by studying the role of patch tokens that embed rich local information inside image patches. We show that by slightly tuning the structure of vision transformers and employing the proposed token labeling objective, can achieve strong baselines for transformer models at different model size levels.
Token Labeling Method
In this section, we first briefly review structure of the vision transformer 15 and then describe the proposed training objective-toker labeling.
3.1 Revisiting Vision Transformer
first decomposes fixed-size input image into a sequence of small patches. Each small patch is mapped to a feature vector, or called a token, by projection with linear layer. Then, all the tokens combined with an additional learnable class token for classification score prediction are sent into stack of transformer blocks for feature encoding.
typical vision transformer 15
In loss computing, the class token from the output tokens of the last transformer block is usually selected and sent into a linear layer for the classification score prediction. Mathematically, given an image denote the output of the last transformer block as [X X ..... X*), where N is the total number of patch tokens, and X and X ..... .....N correspond to the class token and the patch tokens, respectively. The classification loss for image can be written as
Lass H(Xcle cle)
where H(,-) is the softmax cross-entropy loss and yo a is the class label.

---

Figure 3: Comparison between CutMix 48 (Left) and our proposed MixToken (Right). CutMix is operated on the input images. This results in patches containing mixed regions from the two images (see the patches enclosed by red bounding boxes). Differently, Mix To- ken targets mixing tokens after patch em- bedding. This enables each token after patch embedding have clean content as shown the right part of this figure. The detailed ad- vantage of MixToken can be found in Sec. 4.2
3.2
Token Labeling
The above classification problem only adopts an image-level label as supervision whereas neglects the rich information embedded in each image patch. In this subsection, present new training objective-token labeling- -tht takes advantage of the complementary information between the patch tokens and the class tokens.
Token Labeling: Different from the classification loss formulated in Eqn. (T) that measures the distance between the single class token (representing the whole input image) and the corresponding image-level label, token labeling emphasizes the importance of all output tokens and advocates that each output token should be associated with an individual location-specd method, the ground truth for an input image involves not only a single K -i-imensiooal vector yol but also a K x N matrix or called K K-dimensional score map as represented by "y.....""" y,,, where N the number of the output patch tokens.
label. Therefore, in our
Specifically, we leverage a dense score map for each training image and use the cross- cross-entropy loss between each output patch token and the corresponding aligned label the dense score map as an auxiliary loss at the training phase. Figure provides an intuitive interpretation. Given the output patch tokens X ..., XN and the corresponding labels [y' "...... J. the token labeling objective can be defined as
Lu Σ H(X)y))
(2)
Recall that H is the cross-entropy loss. Therefore, the total loss function can be written as
Tatotal H(xds yrla) B- Tus-
(3)
N
H(xds yl ) + E
H(X
(4)
where is paree--prrametr to balance the two terms. In our experiment, we empirically set it to 0.5.
Our token labeling offers the following advantages. First of all, unlike knowledge distillation methods that require a teacher model to generate supervision labels online, token labeling a cheap operation. The dense score map can be generated by pretrained model in advance (c.g.. EfficientNet [34] or NFNet (3)). During training, we only need to crop the score map and perform interpolation make it aligned with the cropped image in the spatial coordinate. Thus, the additional computations are negligible. Second, rather than utilizing single label vector as supervision as done most classification models and the ReLabel strategy 44), we also harness score maps to supervise the models in dense manner and thereby the label for each patch token provides location-speciff information, which can aid the training models to easily discover the target objects and improve recognition accuracy. Last but not the least, as dense supervision is adopted in training, we found that the pretrained models with token labeling benefit downstream tasks with dense prediction, like semantic segmentation.
Advantages:

---

3.3 Token Labeling with MixToken Token
While training vision transformer, previous studies [36] 46 have shown that augmentation methods, like MixUp [52] and CutMix [48), can effectively boost the performance and robustness of the models. However, vision transformers rely on patch-based tokenization to map each imput image to a sequence of tokens and our token labeling strategy also operates on patch-based token labels. If we apply CutMix directly on the raw image, some of the resulting patches may contain content from two images, leading to mixed regions within a small patch as shown in Figure When performing token labeling, it is difficult to assign each output token a clean and correct label. Taking this situation into account, we rethink the CutMix augmentation method and present MixToken, which can be viewed as modified version of CutMix operating on the tokens after patch embedding as illustrated in the right part of Figure
To be specific, for two images denoted as 41. 1e and their corresponding token labels Y = [v.. vi as well as Y2= = [v2- .... 3/2 J. we first feed the two images into the patch embedding module to tokenize each as a sequence of tokens, resulting in 7' = [i...., f and T22 = [t ..., t2 Then, we produce a new sequence of tokens by applying Mix using binary mask M as follows:
ř= о M T2 (1-M),
(5)
where is element-wine multiplication. We use the same way to generate the mask M as in 48 For the corresponding token labels, we also mix them using the same mask M:
Y W
M).
The label for the class token can be written as
ملر = My + (1 - M My
(7)
where M is the average of all element values of M.
4 Experiments
4.1 Experiment Setup
We evaluate our method on the ImageNet 13 dataset. All experiments are built and conducted upon PyTorch [29] and the timm 422 library. We follow the standard training schedule and train
models on the ImageNet dataset for 300 epochs. Besides normal augmentations like CutOut [57] and RandAug [10), we also explore the effect of applying MixUp [52) and CutMix [48] together with proposed token labeling. Empirically, we have found that using MixUp together with token labeling brings no benefit to the performance, and thus do not apply it in our experiments.
For optimization, by default, we use the AdamW optimizer (2 with a linear learning rate scaling strategy Ir 10 batch_size and 10-2 weight decay rate. For Dropout regularization, we 640
observe that for small models, using Dropout hurts the performance. This has also been observed in a few other works related to training vision transformers [36] 37 46). As a result, we do not apply Dropout (32) and use Stochastic Depth (23 instead. More details on hyper-parameters and finetuning can be found in our supplementary materials.
We use the NFNet-F6 3 trained on ImageNet with an 86.3% Top-1 accuracy as the machine annotator to generate dense score maps for the ImageNet dataset, yielding 1000-dimensional score map for each image for training. The score map generation procedure is similar to [49], but we limit our experiment setting by training all models from scratch on ImageNet without extra data support, such as JFT-300M and ImageNet-22K. This is different from the original ReLabel paper 49). in which the EfficientNet-L22 model pretrained on JFT-300M is used. The input resolution for NFNet-F6 is 576 x 576, and the dimension of the corresponding output score map for each image L E IR 18811x10000 During training, the target labels for the tokens are generated by applying RolAlign 17 on the corresponding score map. In practice, we only store the top-5 score maps for each position in half-precision to save space as storing the entire score maps for all the images results in 2TB storage. In our experiment, we only need 10GB of storage to store all the score maps.
5

---

Table 1: Performance of the proposed LV-ViT with different model sizes. Here, 'depth' denotes the number of transformer blocks used in different models. By default, the test resolution is set to 224 x 224 except the last one which is 288 x 288.
Name
Depth Embed dim.
MLP Ratio #Heaas #Params Throughpul (im/s) Test size
Top-1 Acc. (%)
LV-V
12
240
3,0
4
8.5M
2032.6
224
79.1
LV-ViT-S
16
384
3,0
6
26M
1018.2
224
83.3
LV-VIT-M
20
512
3.0
8
56M
668.9
224
84.1
LV-VIT-L
768
3.0
12
150M
204.8
288
85.3
4.2
Ablation Analysis
The default settings of the proposed LV-VIT are given in Table [ where both token labeling and MixToken are used. A slight architecture modification VIT [15 that we replace the patch embedding module with a 4-layer convolution to better tokenize the input image and integrate local information. Detailed ablation about patch embedding can be found in our supplementary materials. As can be seen, our LV-ViT-T with only 8.5M parameters can already achieve a top-1 accuracy of 79.1% on ImageNet. Increasing the embedding dimension and network depth can further boost the performance. More experiments compared to other methods can be found in Sec. [..] In the following ablation experiments, we will set our LV-V as baseline and show the advantages of the proposed token labeling and MixTokkn methods.
Model Settings:
MixToken: We use MixToken a substitution for CutMix while applying token labeling. Our experiments show that MixToken performs better than CutMix for token-based transformer models. As shown in Table 2 when training with the original ImageNet labels, using MixToken 0.1% higher than using CutMix. When using the ReLabel supervision, we can also see an advantage of 0.2% over the CutMix baseline. Combining with our token labeling, the performance can be further raised to 83.3%.
Table Ablation on the proposed MixToken and token labeling augmentations. We also
Table 3: Ablation on different widely-used data aug- mentations. We have empirically found our proposed MixToken performs even better than the combination of MixUp and CutMix in vision transformers.
show results with either the ImageNet hard
label and the ReLabel [49 as supervision.
Aug. Method
Supervision
Top-1 Acc.
MixToken MixUp
CutOut
RandAug Top-1 Acc.
Mix Token
Token labeling
83.3
X
83.3
MixToken
ReLabel
x
x
CutMix
ReLabel
82.8
/
V
V
/
83.1
Mixtoken
ImageNet Label
82.5
X
X
v 83.0
CutMix
ImageNet Label
82.4
X
X
82.8
Data Augmentation: Here, we study the compatibility of MixToken with other augmentation techniques, such as MixUp [52], CutOut (57) and RandAug [TO] The ablation results are shown in Table We can see when all the four augmentation methods are used, top-1 accuracy of 83.1 % is achieved. Interestingly, when the MixUp augmentation is removed, the performance can be improved to 83.3%. This may be explained as, using MixTokea and MixUp at the same time would bring too much noise in the label, and consequently cause confusion of the model. Moreover, the CutOut augmentation, which randomly erases some parts of the image, is also effective and removing it brings a performance drop of 0.3%. Similarly, the RandAug augmentation also contributes to the performance and using it brings improvement of 0.5%.
To show the importance involving all tokens in our token labeling method, we attempt to randomly drop some tokens and use the remaining ones for computing the token labeling loss. The percentage of the remaining tokens is denoted as Token Participation Rate. As shown in Figure h (Left), we conduct experiments on two models: LV-ViT-S and LV-ViT-M. As can be seen, using only 20% of the tokens to compute the token labeling loss decreases the performance (-0.5% for LV-ViT-S and -0.4% for LV-ViT-M). Involving more tokens for loss computation consistently leads to better performance. Since involving all tokens brings negligible computation cost and gives the best performance, we always set the token participation rate as 100% in the following experiments.
All Tokens Matter:

---

84.5
84.1
InaegeeNe Top-1 Aoc. (%6)
84.0
84.0
Ly-II--TTpp Top-1 Ace. (%)
03.2
63.7
83.5
63.3
EfficwentNet-
83.1
83.1
82.8
EfficientNot-B
83.0
82.9
EmtclestNat BS
ElticeeatNet
02.8
82.5
Model
EmfeeentNet
LV-VEE-S EX-VVT-M
ResNeSt2698
Baseline (82.4)
NFNet-Po P6
82.0
Token Participation Rate
Annotator Top-1 Acc. (%)
Figure 4: Left: LV-VIT ImageNet Top-1 Accuracy w.t.t. the token participation rate while applying token labeling. Token participation rate indicates the percentage of patch tokens involved in computing the token labeling loss. This experiment reflects that all tokens matter for vision transformers. Right: LV-ViT-S ImageNet Top-1 Accuracy w.r.t. different annotator models. The point size indicates the parameter number of the annotator model. Clearly, our token labeling objective robust to different annotator models.
Table 4: Comparison of token labeling (TL), knowledge distillation (KD) based method and ReLabel method based on utilized tokens, DeiT-S/LV-viT-S Top-1 accuracy on ImageNet validation set and training time single V100 GPU node.
Method
Online KD
Online TL.
TL
ReLabel
Vanilla
Tokens Utilized
2
All
All
DeiT-S Acc. (%)
81.2
81.8
81.0
80.4
79.9
LV-VIT-S Acc. (%)
83.0
83.5
83.3
82.8
82.4
Training Time (8x V100)
63 hrs
63 hrs
45 hrs
41 hrs
Online Token Labeling: Unlike the online knowledge distillation method which generates labels by a teacher model online, our token labeling approach utilizes the dense label map generated in advance and directly applies the corresponding augmentation methods, such as random crop, the label map to obtain token-level labels. To directly compare with the online knowledge distillation based method and validate the effectiveness of token-level supervision, we further conduct experiments on the online version of our token labeling method, which generates token-level labels online during training. Following DeiT [36), we use RegNetY- 16GF [30] as the online teacher model. Results in terms of SeiT-S/LV-ViT-S Top-1 accuracy and training time for our token labeling, online knowledge distillation, and ReLabel 449 are listed in Table with number of utilized tokens also included for clear comparison. As can be seen, for both online and offline cases, using token-level supervision can improve the overall performance with only negligible additional training cost. Meanwhile, compared to the vanilla training baseline, our proposed offline token labeling brings almost no additional training cost, and boosts the overall performance of VIT- by 0.9%, which well demonstrates its efficiency and effectiveness.
Robustness to Different Annotators: To evaluate the robustness of our token labeling method, use different pretrained CNNs, including EfficientNet-B3
B5,,,,6BB33 17,88 (34), NFNet-F6 (3) and ResNes1269 [5 as annotator models to provide dense supervision. Results are shown the right part of Figure 4 We can see that, even if we use an annotator with relatively lower performance, such as EfficientNet-33 whose Top-1 accuracy is 81.6%, it can still provide multi-label location-specific supervision and help improve the performance of our LV- VI-V model. Meanwhile, annotator models with better performance can provide more accurate supervision, bringing even better performance, as stronger annotator models can generate better token-level labels. The largest annotator NFNet-F6 [3]. which has the best performance of 86.3%, allows us to achieve the best result for LV-ViT-S, which 83.3%. In addition, we also attempt to use better model, EfficientNet-L.2 pretrained on JFT-300M as described in 49) which has 88. 2% Top-I ImageNet accuracy, as our annotator. The performance of LV-ViT-S can be further improved to 83.5%. However, to fairly compare with the models without
7

---

extra training data, we only report results based on dense supervision produced by NFNet-F6 [ that uses only ImageNet training data.
84
Tokec Lab-ling
Tolee: Labeling wib TC
Tokor Labeling
Deaggolee Top Moc. 196
84.2
IRLAANNNt Sogo ANS 1h0
I1.3
wi TL.
WF TL
62
81.8
84.0
$1.0
11.9
83.3
$0.1
82.4
Deft-S (2N0
Deit-d (1:N4)
121-338 121-39 (29N0)
1221-24 (54M)
LIAIES (26M) LVATE- (HEMIL-WWEEL (150N))
Figure Performance of the proposed token labeling objective on three different vision transformers: DeiT 36 (Left), ViT-ViT [46] (Middle), and LV-ViT (Right). Our method has consistent improvement on all 7 different VIT models.
To further evaluate the robustness of our token labeling, we train different transformer-basee networks, including DeiT [36), TiT-ViT 3 and our model LV-VIT, with the proposed training objective. Results are shown in Figure 5 It can be found that, all the models trained with token labeling consistently outperform their vanilla counterparts, demonstrating the robustness of token labeling with respect to different variants of patch-based vision transformers. Meanwhile, for different scales of the models, the improvement also consistent. Interestingly, we observe larger improvements for larger models. These indicate that our proposed token labeling method is widely applicable to large range patch-based vision transformer variants.
Robustness to Different VIT Variants:
Beyond Vision Transformers: We further explore the performance of token labeling on other CNN-based and MLP-based models. Results are shown in Table Besides our re- c--imleemenationn with more data augmentation and regularization techniques, we also provide the results from the original papers. It can be found that for both MLP-based and CCN-based models, our token labeling objective can also improve the performance over strong baselines by providing location-specific dense supervision.
Table 5: Performance of the proposed token labeling objective on representative CNN-based (ResNeSt) and MLP-based (Mixer-MLP) models. Our method has consistent improvement all different models. Here + indicates results reported in original papers.
Model
Mixer-S/16
335
Mixer-B/I6
Mixer-L/16
35
ResNeSt-50
S1
Token Labeling
X
X
V
X
Parameters
18M
18M
18M
59M
59M
59M
207M
207M
207M
27M
27M
27M
Top-1 Acc. (%)
73.8
75.6
76.1
76.4
78.3
79.5
71.6
77.7
80.1
81.1*
80.9
81.5
4.3 Comparison to Other Methods
We compare our proposed model LV-ViT with other state-of-the-arn methods in Table 6 For small- sized models, when the test resolution is set to 224 x 224, we achieve an 83.3% accuracy on ImageNet with only 26M parameters, which 3.4% higher than the strong baseline DeiT-S [66.. For medium- sized models, when the test resolution is set to 384 x 384 we achieve the performance of 85.4%, the same as Ca6 [37, but with much less computational cost and parameters. Note that both DeiT and CaiT use knowledge distillation to improve their models, which introduce much more computations in training. However, we do not require any extra computations in training and only have to compute and store the dense score maps in advance. For large-sized models, our LV- with a test resolution of 448 x 448 achieves an 86.2% top-1 accuracy, which is comparable to C-6 (37] but with far fewer FLOPs and parameters.

---

Table Top-1 aceuracy comparison with other methods on ImageNet [13 and ImageNet Real [2].
All models are trained without external data. With the same computation and parameter constraint, our model consistently outperforms other CNN-based and transformer-base counterparts. The
results of CNNs and ViT are referenced from 37
Network
Params
FLOPs
Train size
Test size
Top- (op-1(%)
Real Top-1 (%)
EfficientNet-Bs
30M
9.9B
456
456
83.6
88.3
EfficientNNet-B B7
66M
37.0B
600
600
84.3
Fix-EificientteetBB
87M
89.5B
672
800
85.7
90.0
NFNet-F3 3
255M
114.8B
320
416
85.7
89,4
NFNet-F4
316M
215.3B
384
512
85.9
89.4
NFNet-F5
377M
289.8B
416
544
86.0
89.2
ViT-B/16
55.4B
224
384
77.9
83.6
ViT-L/16 [5
307M
190.7B
224
384
76.5
82.2
T2T-ViT- 14 44
22M
5.2B
224
224
81.5
T2T-ViT- VEVVI-114134 46
22M
17.1B
224
384
83.3
CiossViT 6
45M
56.6B
224
480
84.1
Transformers
Swin-B 25
88M
47.0B
221
384
TNT-B 16
66M
14.1B
221
221
82.8
ViePViT-S
159
27M
6.2B
221
224
82.3
DeepViT-L
[59
55M
12.5B
224
224
83.1
DeiT-S 36
22M
4.6B
224
224
85.7
Distilled DeiT-S 36
22M
4.6B
224
224
81.2
86.8
DeiT-B [3
86M
17.5B
224
224
81.8
86.7
DeiT-B1384 36
86M
55.4B
224
384
83.1
87.7
Distilled DeiT-B
87M
17.5B
224
224
83.4
88.3
BoTNNct-S1-128
31
79.1M
19.3B
256
256
84.2
BoTNNe-S1-128-38.
31
79.1M
45.8B
256
384
84.7
CaiT-S361384 37
68M
48.0B
221
384
85.4
89.8
CaiT-M36 371
271M
53.7B
221
224
85.1
89.3
CaiT-M361448
271M
247.8B
224
86.3
IV-VIT-S
26M
6.6B
224
224
83.3
88.1
Ours LV- VTT
LV-VIT-$1884
26M
22.2B
224
384
84.4
88.9
LV-VIT-M
56M
16.0B
224
224
84.1
88.4
LV-VIT-M1384
56M
42.2B
224
LV-VIT-L
150M
59.0B
288
288
85.3
89.3
LV-ViT-L
150M
157.2B
288
448
85.9
89.7
LV-VIT-L1448
150M
157.2B
448
448
86.2
89,9
LV-VT-L11512
151M
214.8B
448
512
86.4
90.1
4.4 Semantic Segmentation ADE20K
It has been shown in [19] that different training techniques for pretrained models have different impacts on downstream tasks with dense prediction, like semantic segmentation. To demonstrate the advantage of the proposed token labeling objective tasks with dense prediction, apply pretrained LV-Vi ViT with token labeling to the semantic segmentation task.
Similar to previous work [25], we run experiments on the widely-used ADE20K 588 dataset. ADE20K contains 25K images in total, including 20K images training, images for validation and 3K images for test, and covering 150 different foreground categories. We take both FCN [26] and UperNet [4] as our segmentation frameworks and use the mmssg toolbox to implement. During use the AdamW optimizer with initial learning rate of 6c-5 and a weight decay of 0.01. We also use a linear learning schedule with minimum learning rate of 5e-6. All models are trained on 8 GPUs and with batch size of 16 (i.e., images on each GPU). The imput resolution is set to 512 x 512. In inference, a mulli-scal test with interpolation rates of [0.75, 1.0, 1.25, 1.5, 1.75] is used. As suggested by (58]. report results in terms of both mean intersection-over-umie
training, following (25),
(mloU) and the average pixel accuracy (Pixel Acc.).
In Table 7 we test the performance of token labeling on both FCN and UperNet frameworks. The FCN framework has a light convolutional head and can directly reflect the performance of the pretrained models in terms of transferable capability. As can be pretrained models with token

---

Table 7: Transfer performance of the proposed LV-ViT in semantic segmentation. We take two classic methods, FCN and UperNet, as segmentation architectures and show both single-scale (SS) and multi-scale (MS) results on the validation set.
Method
Token Labeling Model Size
mloU (SS)
P. Acc. (SS)
mloU (MS) P. Acc. (MS)
LV-VIT-S FCN
X
30M
46.1
81.9
47.3
82.6
LV-VIT-S FCN
/ 30M
47.2
82.4
48.4
83.0
LV-ViT-S UperNet
X
44M
46.5
82.1
47.6
82.7
LV-VIT-S UperNet
/
44M
47.9
82.6
48.6
83.1
labeling perform better than those without token labeling. This indicates token labeling is indeed beneficial to semantic segmentation.
We also compare our segmentation results with previous state-of-the-arr segmentation methods in Table Without pretraining on large-scale datasets such ImageNet-22K, our LV-ViT-M with the UperNet segmentation architecture achieves an mloU score of 50.6 with only 77M parameters. This result is much better than the previous CNN-based and transformer-basee models. Furthermore, using our LV-ViT-L as the pretrained model yields a better result of 51.8 in terms of mloU. As far as we know, this is the best result reported on ADE20K with no pretraining on ImageNet-22K, or other large-scale datasets.
Table 8: Comparison with previous work on ADE20K validation set. As far as we know, our LV-
ViT-L + UperNet achieves the best result on ADE20K with only ImageNet- IK as training data in pretraining. Prettrained on ImageNet-2266
Backbone
Segmentation Architecture
Model Size
mloU (MS)
Pixel Acc. (MS)
ResNet-269
PSPNet 54
44.9
81.7
CNNs
ResNet-101
UperNet 45
86M
44.9
ResNet-101
Strip Pooling
45.6
82.1
ResNeSt200
DeepLabV3+
88M
48.4
DeiT-S
UperNet
52M
44.0
LV-VT Transformers
ViT-Large
SETR [56
308M
50.3
83.5
Swin-T
UperNet
60M
46.1
Swin-S
UperNet
81M
49.3
Swin-B B
25
UperNet
121M
49.7
Swin-B
25
UperNet
121M
51.6
LV-VIT-S
FCN
30M
48.4
83.0
LV-VIT-S
UperNet
44M
48.6
83.1
LV-VIT-M
UperNet
77M
50.6
83.5
LV-VIT-L
UperNet
209M
51.8
84.1
Conclusions and Discussion
In this paper, we introduce new token labeling method to help improve the performance of vision transformers. We also analyze the effectiveness and robustness of our token labeling with respect different annotators and different variants patch- based vision transformers. By applying token labeling, our proposed LV-VTT achieves 84.4% Top-1 accuracy with only 26M parameters and 86.4% Top-1 accuracy with 150M parameters on ImageNet-1K K benchmark.
Despite the effectiveness, token labeling has a limitation of requiring a pretrained model as the machine annotator. Fortunately, the machine annotating procedure can be done in advance to avoid introducing extra computational cost in training. This makes our method quite different from knowledge distillation methods that rely on online teaching. For users with limited machine resources on hand, our token labeling provides promising training technique to improve the performance of vision transformers.
10

---

References
[1] Irwan Bello. Lambdanctworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08600 2021.
[2] Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint MXXx:20006001159, 2020.
[3] Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. high-peerfrrancc large-scale image recognition without normalization. arXiv preprint arXiv:2/02.006/71 2021.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14065 2020.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint arXiv:2005. /2872, 2020.
[6] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103. 14899, 2021.
[7] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwci Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint 20X0:2012.00364, 2020.
[8] Liang-Chieh Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. decoder- with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018
[9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewao Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMI.R, 2020.
[10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[11] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:2011.09094 2020.
[12] Stéphane d' Ascoli, Hugo Touvron, Matthe Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.10697, 2021.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[14] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 18x.001810.04805, 2018.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale, arXiv preprint arXiv:2010.1 11999, 2020.
[16] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.001 /2, 2021.
[17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770- 778777, 2016.
[19] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 558-567, 2019.
[20] Byeongho Heo, Sangdoo Yun. Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint 203.022103.16302, 2021.
11

---

[21] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 50300330 2015.
[22] Qibin Hou, Li Zhang, Ming-Ming Cheng, and Jiashi Feng. Strip pooling: Rethinking spatial pooling for scene parsing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4003-4012 2020.
[23] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer, 2016.
(24] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
[25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. arXiv preprint 1X3x:2103,14033 2021.
[26] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[27] Ilya Loshchilov and Frank Hutter. arXiv: 2017.
Decoupled weight decay regularization. arXiv preprint
[28] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin 2018.
Tran. Image transformer. arXiv preprint
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep lcarning library. In Advances in neural information processing systems, pages 8026-8037, 2019.
[30] Ilija Radosavovic, Raj Prateek Kosaraju. Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436, 2020.
[31] Aravind Srinivas. Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101. 1100, 2021.
[32] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting The journal of machine learning research, 15(1):1929- 19589,999988211191111111 2014.
[33] Zhiqing Sun, Shengeao Cao, Yiming Yang, and Kris Kitani. Rethinking transformer-based set prediction for object detection. arXiv preprint arXiv:2011. 2020.
[34] Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1905. 1946, 2019.
[35] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner. Jessica Yung. Daniel Keysers. Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint 2 2021.
[36] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint 1rXiv:2012.1287.. 2020.
[37] Hugo Touvron, Mathieu Cord, Alexandre Sablayrolles. Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint 1rXiv:2103.172333 2021.
[38] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy, arxiv preprint arXiv: 100.::/90060022333 2019.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30:5998-6000 2017.
[40] Wenhai Wang, Enze Xie, Xiang Li, Deng- Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint 122 2011.
12

---

[41] Yuqing Wang. Zhaoliang Xu. Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint arXiv:2011.14503, 2020.
[42]
Ross Wightman.
Pytorch image models.
https://githuu..com//m
bytorch- image-models 2019.
[43] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 158088 2021.
[44] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418-434, 2018.
[45] Fuzhi Yang. Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In Pruceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pages 5791-5800, 2020.
[46] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986 2021.
[47] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng, Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3903-3911, 2020.
[48] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Culmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 6023 6032, 2019.
[49] Sangdoo Yun, Scong Joon Oh, Bycongho Heo, Dongyoon Han, Junsuk Choc, and Sanghyuk Chun. Re-labeling imagenet: from single multi-labels, from global to localized labels. arXiv preprint arXiv:2 000022 2021.
[50] Yanhong Zeng. Jianlong Fu, and Hongyang Chao. Learning joint spatial- temporal transformations for video inpainting. In European Conference on Computer Vision, pages 228-543. Springer, 2020.
[51] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955 2020.
[52] Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
[53] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. arXiv preprint arXiv:2012.09164 2020.
[54] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia, Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881 2881-2890,22 2017.
[55] Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:2011.09311 2020.
[56] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang. Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to--eqqeece perspective with transformers. arXiv preprint 20 15840, 2020.
[57] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[58] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, (33333333-332, 2019.
[59] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 11886, 2021.
[60] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 8739-8748 8748, 2018
[61] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li. Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to and-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
13


Reviews:
[{"review_id": "eWXLrobBi1z", "reviewer": "Reviewer_kkHf", "summary": "This paper presents a novel approach for training vision transformer. Instead of using the image-level label as the supervision, the proposed approach uses pixel-level labels as supervision, and use the cutmix-like image augmentation (MixToken) to generate pixel-level labels. The proposed approach can be regarded as the combination of cutmix [46] and relabeling [47]. Experimental results implied that proposed scheme improves the training quality. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The main drawback of this paper is that the novelty is limited. The proposed MixToken and Token Labeling can be viewed as a modification of cutmix [46] and relabel [47]. The modification leads to $0.5$ gain (Table 2), which is not significant. \n\nQuality and clarity: this paper is clearly written, and the reader can easily understand this paper. The experiments are also very strong. \n\nSignificance:  Experimental results show that the proposed schemes are helpful. Unfortunately, this paper is more like a straightforward application of cutmix and relabel with slight modifications. This reduces the technical signification.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of this paper is the limited novelty. The proposed MixToken and Token Labeling can be viewed as a modification of cutmix [46] and relabel [47]. The modification leads to $0.5$ gain (Table 2), which is not significant. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}, {"review_id": "NCHMyFdjrnP", "reviewer": "Reviewer_mqsb", "summary": "This paper introduces token labelling, a new training objective for vision transformers. The core idea is to supervise each of the patches of the last layer, instead of simply discarding them as is usually done. The proposed approach significantly improves performance of various architectures on ImageNet.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like the idea presented in this paper, which is simple and yields good results. Various ablation convincingly demonstrate the effectiveness and wide applicability of the token labelling technique. However, I am not sure I understood the claims made by the authors that this approach is more efficient computationally than distillation, as it requires annotating each one of the patches. However, I still think this paper deserves publication.\n\nStrengths: \n- The proposed approach reaches state-of-the art, both in classification and segmentation\n- Extensive ablations effectively demonstrate the benefit of token labelling\n- Although this is not mentioned in the main text (likely due to last minute rush), a section in the appendix shows that token labelling also benefits MLP-mixer architectures. If the paper is accepted, I believe this section should go in the main text, as it is important to show the wide applicability of this method (perhaps replacing the ablation over the annotator which is less crucial)\n\nWeaknesses:\n- Storing the score maps is costly. This is acknowledged by the authors, but they seem to suggest nonetheless that token labelling is computationally cheaper than distillation. I do not understand this, see questions below.\n- It would be nice to include latency or throughput measurements to give an idea of the speed of the models considered, and check that fetching the score maps from memory doesn’t slow down training.\n\nComments:\n- “Unlike knowledge distillation methods that require a teacher model to generate supervision labels online, token labeling is a cheap operation. The dense score map can be generated by a pretrained model in advance “. I do not understand this sentence : token labelling involves a forward pass of an annotator in the same way as distillation does, what is the difference ? The distillation tokens can also be stored in advance, or am I missing something ?\n- I am confused as to why the score maps are 18x18x1000 rather than just 18x18. Does this mean the score maps are soft labels ? Couldn’t it be possible to store simply 18x18 maps of correct labels, and supervise with hard labels ? Since Touvron et al. find that hard label distillation is better than soft label distillation for DeiT, perhaps the same could hold true for token labelling ?\n- In the comparison table, I think it would be fair to add a comparison with the distilled DeiT and CaiT models, since token labelling is rather similar in spirit to distillation. Ideally, it would even be nice to have, among the ablations, a direct comparison of token labelling vs token distillation in the same training setting.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6cUmBZYskp7", "reviewer": "Reviewer_tac1", "summary": "This paper presents a new strategy to more effectively train vision transformers by introducing token-wise supervision. A token labeling loss, MixToken data augmentation and several modifications on ViT architectures are proposed. Extensive experiments are conducted to show the effectiveness of the proposed method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed token labeling method is well motivated and extensively validated. The proposed method can largely improve the performance of vision transformers while maintaining the relatively simple architecture (only the patch embedding layer is replaced by a small CNN). The modifications on the original ViT may also be useful in future research and applications.  Extensive experiments are conducted to show the effectiveness of the proposed method. The new designs on the network archietcture are clearly verified. It is interesting to see that the token labeling method also works well on DeiT, T2T-ViT, Mixer and CNNs. However, I still have some concerns:\n\n- The proposed token labeling method is very similar to hard knowledge distillation and the method is closely related to ReLabel strategy [47]. Using the knowledge distillation method to vision transformers is also studied in DeiT [34]. Although I agree with the authors that the proposed method is more efficient, the technical novelty of the proposed method is limited. Besides, I think it would be better to provide the results of online hard/soft knowledge distillation as references.\n\n- I think it is necessary to provide a more detailed analysis of the differences between the distillation method proposed in DeiT and the proposed token labeling. According to Figure 5 (left), the top-1 accuracy of the DeiT-B model with token labeling is 83.1%, while [34] reports that DeiT-B with hard distillation (300 epochs) can achieve 83.4% top-1 accuracy. It seems that the simple hard distillation method in DeiT can outperform token labeling. However, the performance gap may come from different teacher models or the online/offline distillation strategy. To support the motivation of token labeling, it is critical to show the proposed token-wise supervision is more effective than global supervision like DeiT-B. I think the authors can provide the results of online token labeling or use NFNet-F6 as the teacher model for global distillation to directly compare with the distillation method in DeiT. I suspect token-wise supervision may be more useful for downstream dense prediction tasks. This paper can be much stronger if the advantages of token labeling (compared to global distillation in DeiT) are clearly demonstrated. \n\n- Although I agree the results are impressive, I still think emphasizing the performance under a certain level of #Param may not be very reasonable. With similar parameters, models using larger input images will always lead to better #Param/accuracy trade-offs. I think FLOP is a better metric to measure the computational complexity and the throughput/latency are even more realistic metrics. It would be better to report the FLOPs/throughput/latency for the main results and comparisons.\n\nOverall, I think this is a practical method with several technical innovations and good results. However, the proposed token labeling method may not be clearly verified. As its current state, I would like to rate this paper as Marginally below the acceptance threshold. I will be happy to upgrade my rating if my second concern is properly addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and potential negative societal impact of this paper have been discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new framework to train vision transformers with token level supervision. While there were some debate over the novelty of the proposed technique in the context of previous methods (MixToken, CutMix, relabel), it was agreed that the proposed method is efficient and performant, and the extensive empirical studies in this paper could benefit the community. The authors also did well by providing convincing results on extra baselines requested by the reviewers in the rebuttal. 
Decision: Accept (Poster)