Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a bottom-up multi-person pose estimation method based on transformer model. In this method, a new loss is used to encourage a self-attention map to put its attention on the region of an instance via the given ground-truth segmentation mask. Besides, the attention scores between estimated keypoint-locations are used to assign the estimation to correct instance. 

Experiments are applied on COCO keypoint dataset. The results shows that the proposed method outperforms or is comparable with some competitors such as OpenPose and Person Lab on keypoint detection. When compared with the existing methods which explicitly estimate instance segmentation mask, the proposed method performs worse.
main_review: * Strengths: 
    * The idea is clear and reasonable. Self-attention module computes attention for every point in the feature map. It is reasonable to restrict the region where a self-attention module puts its attention by using some prior knowledge. The authors proposed two methods to restrict the attention region, one is a loss applied on attention map  which is interesting, and the other is attention-score based keypoint grouping. The visualized attention maps show that the proposed loss works well.
    * The ablation study on Tab.1 shows the effectiveness of the proposed method.
* Weaknesses: 
    * The idea is lack of novelty. The restrict attention region is reasonable but not novel, since the core idea is still instance segmentation.
    * The performance is not promising when compared with some classic methods.
    * As for instance segmentation, the authors argue that the worse performance is due to lower resolution. I think it is necessary to set up a fair experiment, otherwise the comparison is not very meaningful.
    * In 3.2, only loss is shown, but it is not clear how the supervised self-attention contributes to accuracy.

summary_of_the_review: The idea is clear and reasonable, but lack of novelty and promising performance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Bottom-up methods tackle the problem of multi-person detection, pose estimation, and segmentation by localizing human keypoints and then grouping them into person instances, which inevitably bring up the question of what features to use for grouping and how to efficiently group keypoints and break down corner cases. In this work, the authors propose to use Transformer to exploit associative information between keypoints. Initial experiments show that naive self-attention has non-zero attention scores between keypoints from different human instances, and even high attention scores in crowded scenarios. Such self-attention layers make it hard to directly associate keypoints of the same instance from the self-attention scores, and subsequently requires careful hyperparameter tuning and inevitably introduce errors in hard examples. Therefore, the authors propose a instance mask loss to supervise self-attention with instance segmentation masks.

Another advantage of such framework is to unify the task of keypoint detection and instance segmentation. The segmentation mask can be easily generated from the self-attention scores and the network learns a representation that is consistent with both the keypoint locations and the instance masks.
main_review: **Strengths**

The idea to incorporate self-attention models to extract semantic features for keypoints association is a novel and elegant approach to address this problem. The efficacy of this approach is also verified by the experimental results. For large objects, the neighboring keypoints could be several hundred pixels away and Personlab [1] adopt recurrent offset refinement to refine the inaccurate regressed long-range offsets. In contrast, the proposed model benefit from the self-attention and can easily group keypoints with large offsets.

Another contribution of this work is to unify keypoint detection, grouping and segmentation. The instance mask helps to supervise the association between keypoints of a single instance and the predicted keypoint detection would help produce better masks from a bottom-up perspective. We can see the two tasks are highly-correlated and the proposed approach seems to be a reasonable and novel design in this direction.

**Concerns**

Supervising self-attention with instance mask is a natural idea to assist keypoint grouping but there are some concerns related to the choice of supervision that are not addressed well. The visualization of the association reference in Figure 1 show that the supervised self-attention helps to discriminate keypoints from the same instance and from different instances. However, we could also see that the naive self-attention produce self-attention scores with good characteristics that start by looking at the context around the instance and then shift to focus on keypoints of the same instance. Meanwhile, the supervised self-attention seems to only focus on associating keypoints of the same instance, which is a direct result of the mean squared error (MSE) of the instance mask loss. It seems that the self-attention behavior is completely changed by the instance mask loss and subsequently the final keypoint heatmap. It is good to produce a instance segmentation mask at certain layer but it is unclear how will this affect the keypoint features and the keypoint detection. Will the instance mask loss make the keypoint features less discriminative and hence less accurate keypoint localization? There is no ablation study experiments on the design of the instance mask loss to address such concerns so the advantage of such loss is doubtful beyond the visualization results. Ablation study in A.3 show that there is no clear differences between supervising different layers of the self-attention model, which I don't understand since the self-attention scores focus on very different regions before and after the layer that is supervised.

Experimental results are limited and fail to show the benefits of self-attention over regression. Most quantitative results are only roughly comparable with previous methods even with different refinements. The issue of Transformer models performing worse on smaller objects (e.g., DETR [2]) is also left unaddressed. While the proposed model can elegantly unify the task of keypoint detection and instance segmentation, the advantage of such approach is not supported by the experiments. The experiments cannot show the benefits of sharing the features for both keypoint detection and instance segmentation and there are no known superior aspects of the proposed approach in terms of number of parameters, MACs, or post-processing time.
summary_of_the_review: Overall, I think this paper presents a novel and interesting idea. However, the current experimental results are relatively weak and cannot show the strength of the method, which limits the contribution. There are also some concerns with the self-attention supervision that are not explained well and not supported by the ablation study.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes explicit supervision of transformer attention for the purposes of bottom-up multi-person pose estimation. The network (TransPose [Yang et al. 2021]) passes an image through a resnet followed by several tranformer layers to produce per-pixel heatmaps localizing the joints of all people in the given image. The attention of the transformer layers is across pixel space, so the authors propose supervising the resulting distribution so that the attention score is higher between keypoints belonging to the same person. This is done by sampling the attention masks for keypoints and supervising them with MSE to match the annotated segmentation mask of that person. At test time, detected keypoints can be grouped into invidual people with a simple heuristic parsing of relative attention scores.
main_review: I like that the proposed approach takes advantage of an existing property of the TransPose architecture which is already doing the work of explicitly associating pixels with each other in each of the transformer layers, adding a little extra supervision is straightforward enough and can always be incorporated as dense transformer architecture design matures. Overall, the paper is written clearly and sufficient information is provided to reimplement the method.

However I have some concerns:

- Comparisons are left out to more recent bottom-up human pose work, notably missing is the HRNet family of work such as HigherHRNet (Cheng et al CVPR 2020) and the more recent DEKR (Geng et al CVPR 2021) both of which set a higher bar for bottom-up results and also do not rely on single-person pose refinement. Beating the state-of-the-art is certainly not a requirement here, but it is important to include other recent methods in comparisons.

- It seems to me that the transformer layers introduce serious compute overhead (this is alluded to in the concluding discussion), the paper does not offer much information in terms of the cost of this approach compared to other work. It would be helpful to get some comparisons (e.g. images/sec on fixed hardware, memory requirements, etc).

- A related point is that attention across pixel space requires that a large stride is used to operate at a reasonably low resolution. This is a difficult trade-off in bottom-up human pose where metrics are so sensitive to small localization jitter. It is unclear what good strategies to overcome this might be with this style of architecture.

- Another caveat is the requirement for additional instance segmentation mask supervision which other methods do not rely on. Were any alternatives considered so that this method could be used on datasets where these annotations are unavailable?

- One question I had is how restricting the attention in this way affects accuracy of the predicted heatmaps. From Fig 6 it seems as though it hurts performance somewhat. It would be interesting to know the effect here (for example, by comparing with and without the attention loss and evaluating using oracle assignment)

summary_of_the_review: The authors propose an nice, simple strategy to associate keypoints in multi-person pose estimation, but I don't know that a compelling enough case is made that this method offers advantages given the restrictions in terms of additional annotation requirements + compute overhead while falling short of and excluding more recent methods in performance comparisons.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to incorporate the instance segmentation task into the human pose estimation task (serving as attention layer loss for transformers). This supervision is novel and serves as a good alternative for AE/PAF, etc. They also propose two decoding methods (part-based / body-based). (with accuracy/time trade off. )
main_review: strengths: as far as I know it is the 1st paper to do mid-layer supervision and use instance mask to supervise the transformers attention.

weakness: although they've done several ablation studies, there are not enough
(1) what if the mean of the keypoint is replaced with the mean of vector inside bbox (this seems a more reasonable alternative)
(2) what if this is a multitask transformer, decoded to both instance-segmentation mask and keypoint estimation. Will the end-supervision better be than intermediate supervision? 
(3) the final results are not compelling, even worse than PersonLab (2018)
(4) what is the speed and complexity increment  after adding this intermediate loss / for inference (compared with original transformers)
(5) the paragraph above Fig 5 seems not reasonable to me. This makes the method more like a top-down method instead of bottom-up, making this methods less compelling. 
(6) Fig 4 need more description to be  better understood.

summary_of_the_review: Please refer to the Main Review of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a bottom-up 2D multi-person pose estimation system. The proposed system takes a single RGB image and predicts 2D Gaussian heatmaps of human keypoints, used to localize human keypoints. The main contribution of this paper is that they introduced a supervised self-attention (SSA), which supervises self-attention maps with human segmentation masks. The self-attention maps are computed in Transformer encoder modules, which represents how much each pixel pays attention to all pixels in feature maps. They showed that without SSA, naive self-attention maps tend to have activations on wrong positions.
main_review: * Strong points.

S1. Clear motivation.
The proposed SSA has a clear motivation. Supervising self-attention maps with human segmentation can guide the self-attention map to focus on correct pixels, which can lead to more accurate pose estimation.

S2. Well-written manuscript.
Overall manuscript and figures successfully deliver their main ideas.

* Weak points.

W1. Lack of numerical comparison between naive self-attention and SSA.
Figure 1 and 6 show qualitative comparison between naive self-attention and the proposed SSA. However, no numerical comparison (i.e., AP on test or validation set) is not reported. This makes me hard to be convinced SSA is clearly more effective than naive self-attention.

W2. Weak experimental results compared to recent state-of-the-art methods.
Table 1 and 2 show comparison between the proposed method and state-of-the-art methods. Compared to other bottom-up methods, the proposed method fails to outperform them. Importantly, most of the bottom-up comparison targets are not recent ones. Recent bottom-up methods, such as [1] and [2], achieve far better results than the proposed one.
[1] achieves AP 66.5 and 68.4 on validation and test-dev set, respectively, without multi-scale testing. [1] and [2] are representative bottom-up methods; however, they are not cited.

[1] Cheng, Bowen, et al. "Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation." CVPR. 2020.
[2] Geng, Zigang, et al. "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression." CVPR. 2021.

W3. Unfair comparison using the refinement.
The refinement introduced in Section 3.1 seems used only for the proposed method and not used for other bottom-up methods. The refinement is a kind of post-processing using off-the-shelf well-performing 2D human pose estimator, which is not a novel point of this paper. Applying the refinement only to the proposed method and comparing with other is unfair.

W4. Lack of qualitative results.
No qualitative 2D human pose results are provided. 

W5. Is human segmentation the best choice?
Although the human segmentation provides pixel-level annotations, it does not have human articulation information. As human keypoints are connected in the kinematic chain, I guess supervising the attention map with Gaussian heatmap or one-hot matrix could provide better results than human segmentation map since the heatmap and one-hot matrix has activation for each keypoint in channel dimension.

summary_of_the_review: Although the paper has a clear motivation and is well-written, lack of experimental demonstration and weak experimental results are weak points.
review_title: null

Meta-review: This paper proposes a bottom-up multi-person pose estimation method using a Transformer model. There is consensus among the reviewers that this paper is not ready for acceptance/publication. Although some reviewers find the proposed idea interesting (some find it lacking novelty though), all the reviewers agree that the quantitative experimental results are not promising. Some reviewers explicitly criticized lacking empirical accuracy compared to state-of-the-arts. The authors provided additional details and results in the rebuttal, but they were not sufficient to change the opinions of the reviewers.

We recommend rejecting the paper.
Decision: Reject