Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The article presents an experimental evaluation of the vision transformers for human pose estimation tasks. The experimental analysis focuses on transformer’s simplicity in model structure, scalability in model size, flexibility in training, and transferability of knowledge (distillation) from bigger models to smaller ones. The experimental evaluations are carried out on single MS COCO dataset by exploring various strategies such as attention types, image resolutions, pre-trained weights and finetuning. The largest models gives state-of-the-art accuracy on the MS COCO dataset and the authors have mentioned that the code and models will be released for other researchers. The evaluation on other datasets (CrowdPose, OCHuman, MPII, AI Challanger) is provided in supplementary material.
strengths_and_weaknesses: Strength:

The idea is good and is inspired by the lack of baseline evaluations of vision transformers for solving the pose estimation problem. The author has clarified that the paper does not claim algorithm superiority rather extensive evaluation on the MS COCO dataset to justify the benefits of vision transformers in solving pose estimation.  

The paper has justified the usefulness of detailed experimental analysis by focusing on various aspect of vision transformer such as its structure, size, training process and knowledge distillation. 

Experimental evaluation using well-known benchmarked human pose dataset of MS COCO is carried out. The performance of the proposed approach is compared to the some recent transformer and/or attention driven approaches.

Ablation study involving the benefit of pre-training data, influence of input resolution, impact of attention types, influence of partially finetuning and multi-task learning.

The article is well-written and easy to follow.

Weakness:

The proposed idea is an application of vision transformers to human pose estimation thus there is a question mark on the novelty. 
The model’s computation complexity is presented as training memory. However, the other parameters such as GFLOPs and trainable parameters would have given a better reflection.  

It is unclear how the proposed knowledge token is different form the visual tokens. 

Although COCO + AI challenger dataset volume is half of ImageNet-1K (Table 2) but cropped person is used for training which is much cleaner dataset in comparison to ImageNet-1K. This could have influenced the performance gap. Any justification?

In table 4:  Full+window training memory (28,594) is less than full only training memory (36,141). Any reason?

In table 4: (16,12) window size is compared to (8,8) in the rest of the experiments except full. Any justification on selecting 16,12 why not 16,16?

How the multi-task training carried out in Table 6. Other than pose what other tasks that learned during multi-task learning. 

questions: 1. Novelty of the approach

2. What is the difference between knowledge tokens and visual tokens? Which one is more important and why?

3. Model complexity should have included GFLOPS and trainable params on top of memory footprint.

4. What is the accuracy of COCO + AI challenger dataset  (Table 4) without using cropped person

5. In table 4:  Full+window training memory (28,594) is less than full only training memory (36,141). Any reason?

6. How the multi-task training carried out in Table 6?
limitations: The section 5 header says "Limitation" but the content does not discuss any limitations. However, it does mention about social impact of machine learning biases and personal privacy. It also mentions about the carbon footprint from data centers using this large-scale model training. 

The paper should explore the training process and time taken to train such large models and how these models could be improved in near future.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper applied the vision transformer to the task of single human pose estimation. The model uses plain and non-hierarchical vision transformers to extract features from the input image. Then a very simple decoder is added to regress heatmaps for human pose estimation. The method with a large version model achieved 80.9 AP on the COCO test-dev set. The authors also proposed a knowledge distillation method to transfer the knowledge from larger ViT models to smaller models.
strengths_and_weaknesses: Strengths 
1) The paper is well-written and easy to read.
2) The model achieved state-of-the-art performance on the MSCOCO human pose estimation test-dev set.
3) The paper showed that with the powerful encoding ability of vision transformers, a very simple decoder (a bilinear upsample and one layer convolution) can achieve a very good result.
4) The authors also proposed a knowledge distillation method to transfer the knowledge from larger ViT models to smaller models, which improved the AP from 75.8 to 76.6 with little computation overhead.
5)  The authors did a lot of detailed ablation studies on the influences of partially fine-tuning, attention type, input resolution and pre-training data.
Weaknesses
1) Although the experiments of this paper are solid, the novelty of this paper is limited as this is an application of ViT to human pose estimations.
2) The claimed advantages of structural simplicity, model size scalability, training paradigm flexibility, and knowledge transferability can also be achieved by previous deep learning models. The vision transformer itself is not structurally simpler than CNN, stacking ViT without introducing other modules doesn't mean this model has a simpler structure compared to CNNs. Model size of other models can also be easily controlled by stacking. The training paradigm flexibility is very common in CNNs for humane pose estimation. Instead of using different decoders, particular output channels from different datasets can be masked during training due to the flexibility of heatmaps.
3) In lines 76-77, the authors claimed this model can be extended from a single pose dataset to multiple pose datasets. This can be confusing as in human pose estimation, multi-pose datasets normally refer to images with more than one pose to predict. It looks like the authors are describing extending the method to multi-dataset training.
4) The paper particular discussed "multi-task training". As far as I can tell, it's multi-dataset instead of multi-task training. The task is human pose estimation with different numbers of joints.
questions: As in the weaknesses, there are some details to be clarified by the authors.
limitations: Yes. The authors described the limitation of not fully exploring more advanced technologies.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a simple transformer based architecture for human pose estimation in images. The model is composed of a visual backbone (ViT) along with a simple decoder to obtain the heat maps providing the joint locations. The authors conduct multiple ablations to study the importance of various components (input resolution, number of parameters, ability to perform multi-task training etc). This result in a state of the art model on the COCO Keypoint Detection benchmark while maintaining a fast inference speed thanks to the ViT architecture which is modern hardware friendly.
strengths_and_weaknesses: **Strengths**

- The paper is clearly written and well executed
- The method is simple to understand and performs well. Such strong baseline papers are important to the field as they can be the starting point for even better methods as it is easier to build on top of such approaches.
- I enjoyed reading the ablations study which is well conducted and properly shed lights on the important components of the approach

**Weaknesses**

- One could argue that there is not much novelty in the paper, as it is mainly a well executed strong baselines. However I still feel the paper in itself is a good enough contribution in itself. 
- I am not sure to understand the motivation behind the learned token for transferability from large models to smaller models (see Questions below).
questions: **Question/clarifications**

- What is the intuition behind the "token-based distillation"? Since it is trained to only work for a specific set of parameters, I am not sure to understand why adding it to a smaller model would suddenly distill knowledge from the big model to the small. In Table 7, the gains are not very clear (I am not super familiar with the type of variance one may encounter on the COCO keypoint detection benchmark, can the authors confirm whether or not those results are significant?). It would be great if the authors could comment on that point. If this is inspired by past work, it should be cited.

- What is layer-wise decay? How useful is that? Can you cite the paper for this?

**Minor typos**

- L151: double period at the end of the last sentence
limitations: Limitations and societal impact are adequately discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new baseline for pose estimation with vision transformer architecture. It obtains state-of-the-art performance on the MS COCO dataset even without the usage of elaborate structural designs or complex frameworks. Further, it has simplicity, scalability, flexibility and transferability. 
strengths_and_weaknesses: Pros:

(1)	A new baseline for pose estimation with vision transformer architecture is introduced. 

(2)	SOTA results on MS COCO keypoint. 

(3)	Comprehensive experiments show its simplicity, scalability, flexibility and transferability. 

Cons:

This manuscript introduces a new baseline architecture, which achieves superior performance on the MS COCO keypoint dataset. I think it is a strong paper. However, there are still some questions.

In Table8, compared to HRFormer, ViTPose has more parameters but with a faster speed (e.g., ViTPose-B vs. HRFormer-B), which should be explained in detail. The authors are also encouraged to employ a smaller model (e.g., ViTPose-S or stack smaller layers) with comparable parameters to previous methods to show the superiority of this baseline. Besides, such a method in this manuscript should be a very simple baseline, and it is easy to think and implement, what is the reason for it surpassing the previous transformer-based method? It should be discussed. 

questions: As listed in Weaknesses, the following questions need to be solved. 

(1)	Introduce a smaller model for comparison with previous methods.

(2)	More explanations about this method and experimental results. 

limitations: There exist some biased training data (e.g., gender, race, age) and personal privacy problems, as listed in this manuscript. This might be a general problem in the pose estimation community. These might be alleviated by masking face or using synthetic datasets.
review_title: null

Meta-review: metareview: This submission received positive reviews. After rebuttal and discussions, all the reviewers feel positive about this submission with raised concerns addressed. After checking all the reviews and rebuttal, the AC stands on the reviewers' side and believe the current work is suitable for publication in this venue. The authors shall revise the manuscript according to the suggestions from the reviewers in the camera-ready submissions. 
confidence: Certain
Decision: Accept