Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new strategy to more effectively train vision transformers by introducing token-wise supervision. A token labeling loss, MixToken data augmentation and several modifications on ViT architectures are proposed. Extensive experiments are conducted to show the effectiveness of the proposed method. 
main_review: The proposed token labeling method is well motivated and extensively validated. The proposed method can largely improve the performance of vision transformers while maintaining the relatively simple architecture (only the patch embedding layer is replaced by a small CNN). The modifications on the original ViT may also be useful in future research and applications.  Extensive experiments are conducted to show the effectiveness of the proposed method. The new designs on the network archietcture are clearly verified. It is interesting to see that the token labeling method also works well on DeiT, T2T-ViT, Mixer and CNNs. However, I still have some concerns:

- The proposed token labeling method is very similar to hard knowledge distillation and the method is closely related to ReLabel strategy [47]. Using the knowledge distillation method to vision transformers is also studied in DeiT [34]. Although I agree with the authors that the proposed method is more efficient, the technical novelty of the proposed method is limited. Besides, I think it would be better to provide the results of online hard/soft knowledge distillation as references.

- I think it is necessary to provide a more detailed analysis of the differences between the distillation method proposed in DeiT and the proposed token labeling. According to Figure 5 (left), the top-1 accuracy of the DeiT-B model with token labeling is 83.1%, while [34] reports that DeiT-B with hard distillation (300 epochs) can achieve 83.4% top-1 accuracy. It seems that the simple hard distillation method in DeiT can outperform token labeling. However, the performance gap may come from different teacher models or the online/offline distillation strategy. To support the motivation of token labeling, it is critical to show the proposed token-wise supervision is more effective than global supervision like DeiT-B. I think the authors can provide the results of online token labeling or use NFNet-F6 as the teacher model for global distillation to directly compare with the distillation method in DeiT. I suspect token-wise supervision may be more useful for downstream dense prediction tasks. This paper can be much stronger if the advantages of token labeling (compared to global distillation in DeiT) are clearly demonstrated. 

- Although I agree the results are impressive, I still think emphasizing the performance under a certain level of #Param may not be very reasonable. With similar parameters, models using larger input images will always lead to better #Param/accuracy trade-offs. I think FLOP is a better metric to measure the computational complexity and the throughput/latency are even more realistic metrics. It would be better to report the FLOPs/throughput/latency for the main results and comparisons.

Overall, I think this is a practical method with several technical innovations and good results. However, the proposed token labeling method may not be clearly verified. As its current state, I would like to rate this paper as Marginally below the acceptance threshold. I will be happy to upgrade my rating if my second concern is properly addressed.
limitations_and_societal_impact: The limitations and potential negative societal impact of this paper have been discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a novel approach for training vision transformer. Instead of using the image-level label as the supervision, the proposed approach uses pixel-level labels as supervision, and use the cutmix-like image augmentation (MixToken) to generate pixel-level labels. The proposed approach can be regarded as the combination of cutmix [46] and relabeling [47]. Experimental results implied that proposed scheme improves the training quality. 
main_review: Originality: The main drawback of this paper is that the novelty is limited. The proposed MixToken and Token Labeling can be viewed as a modification of cutmix [46] and relabel [47]. The modification leads to $0.5$ gain (Table 2), which is not significant. 

Quality and clarity: this paper is clearly written, and the reader can easily understand this paper. The experiments are also very strong. 

Significance:  Experimental results show that the proposed schemes are helpful. Unfortunately, this paper is more like a straightforward application of cutmix and relabel with slight modifications. This reduces the technical signification.
limitations_and_societal_impact: The main limitation of this paper is the limited novelty. The proposed MixToken and Token Labeling can be viewed as a modification of cutmix [46] and relabel [47]. The modification leads to $0.5$ gain (Table 2), which is not significant.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces token labelling, a new training objective for vision transformers. The core idea is to supervise each of the patches of the last layer, instead of simply discarding them as is usually done. The proposed approach significantly improves performance of various architectures on ImageNet.
main_review: Overall, I like the idea presented in this paper, which is simple and yields good results. Various ablation convincingly demonstrate the effectiveness and wide applicability of the token labelling technique. However, I am not sure I understood the claims made by the authors that this approach is more efficient computationally than distillation, as it requires annotating each one of the patches. However, I still think this paper deserves publication.

Strengths: 
- The proposed approach reaches state-of-the art, both in classification and segmentation
- Extensive ablations effectively demonstrate the benefit of token labelling
- Although this is not mentioned in the main text (likely due to last minute rush), a section in the appendix shows that token labelling also benefits MLP-mixer architectures. If the paper is accepted, I believe this section should go in the main text, as it is important to show the wide applicability of this method (perhaps replacing the ablation over the annotator which is less crucial)

Weaknesses:
- Storing the score maps is costly. This is acknowledged by the authors, but they seem to suggest nonetheless that token labelling is computationally cheaper than distillation. I do not understand this, see questions below.
- It would be nice to include latency or throughput measurements to give an idea of the speed of the models considered, and check that fetching the score maps from memory doesn’t slow down training.

Comments:
- “Unlike knowledge distillation methods that require a teacher model to generate supervision labels online, token labeling is a cheap operation. The dense score map can be generated by a pretrained model in advance “. I do not understand this sentence : token labelling involves a forward pass of an annotator in the same way as distillation does, what is the difference ? The distillation tokens can also be stored in advance, or am I missing something ?
- I am confused as to why the score maps are 18x18x1000 rather than just 18x18. Does this mean the score maps are soft labels ? Couldn’t it be possible to store simply 18x18 maps of correct labels, and supervise with hard labels ? Since Touvron et al. find that hard label distillation is better than soft label distillation for DeiT, perhaps the same could hold true for token labelling ?
- In the comparison table, I think it would be fair to add a comparison with the distilled DeiT and CaiT models, since token labelling is rather similar in spirit to distillation. Ideally, it would even be nice to have, among the ablations, a direct comparison of token labelling vs token distillation in the same training setting.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper proposed a new framework to train vision transformers with token level supervision. While there were some debate over the novelty of the proposed technique in the context of previous methods (MixToken, CutMix, relabel), it was agreed that the proposed method is efficient and performant, and the extensive empirical studies in this paper could benefit the community. The authors also did well by providing convincing results on extra baselines requested by the reviewers in the rebuttal. 
Decision: Accept (Poster)