Attend to Who You Are: Supervising Self-Attention for Keypoint Detection and Instance-Aware Association
Authors: Sen Yang
Zhicheng Wang
Ze Chen
Yanjie Li
Shoukui Zhang
Zhibin Quan
Shu-Tao Xia
Yiping Bao
Erjin Zhou
Wankou Yang
Venue: ICLR 2022 Submitted (2022)
Keywords: human pose estimation
bottom-up
self-attention
transformer
instance segmentation

Abstract: Bottom-up multi-person pose estimation models need to detect keypoints and learn associative information between keypoints. 
We argue that these problems can be entirely solved by the Transformer model. Specifically, the self-attention in Transformer measures the pairwise dependencies between locations, which can play a role in providing association information for keypoints grouping.
However, the naive attention patterns are still not subjectively controlled, so there is no guarantee that the keypoints will always attend to the instances to which they belong.
To address it we propose a novel approach of multi-person keypoint detection and instance association using instance masks to supervise self-attention. By supervising self-attention to be instance-aware, we can assign the detected keypoints to the correct human instances based on the pairwise attention scores, without using pre-defined offset vector fields or embedding like CNN-based bottom-up models. An additional benefit of our method is that the instance segmentation results of any number of people can be directly obtained from the supervised attention matrix, thereby simplifying the pixel assignment pipeline.
The experiments on the COCO multi-person keypoint detection challenge and person instance segmentation task demonstrate the effectiveness and simplicity of the proposed method.

Under review as a conference paper at ICLR 2022
ATTEND TO WHO YOU ARE: SUPERVISING SELF- ATTENTION FOR KEYPOINT DETECTION AND INSTANCE-AWARE ASSOCIATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
This paper presents a new method to solve keypoint detection and instance associ- ation by using Transformer. For bottom-up multi-person pose estimation models, they need to detect keypoints and learn associative information between keypoints. We argue that these problems can be entirely solved by Transformer. Specifically, the self-attention in Transformer measures dependencies between any pair of lo- cations, which can provide association information for keypoints grouping. How- ever, the naive attention patterns are still not subjectively controlled, so there is no guarantee that the keypoints will always attend to the instances to which they belong. To address it we propose novel approach of supervising self-attention for multi-person keypoint detection and instance association. By using instance masks to supervise self-attention to be instance-aware, keypoints to their instances based on the pairwise attention scores, without using pre-defined offset vector fields embedding like CNN-based bottom- up models. An additional benefit of our method is that the instance segmentation results of any number of people be directly obtained from the supervised attention matrix, thereby simplifying the pixel assignment pipeline. The experiments on the COCO multi-person keypoint detection challenge and person instance segmentation task demonstrate the effectiveness and simplicity of the proposed method, and show a promising way to control self-attention behavior for specific purposes.
assign the detected
INTRODUCTION
Multi-person pose estimation approaches usually can be classified into two schemes: top-down or bottom-up. Unlike the top-down scheme that converts the pipeline into two independent tasks detection and single-pose estimation, the bottom-up scheme is confronted with more challenging problems. An unknown number of persons with any scale, posture, or occlusion condition may ap- pear at any location of the input image. The bottom-up approaches need to detect all body joints first. and group them into instances second. the typical systems such as DeeperCut (Insafutdinov et al.) OpenPose (Cao et al.) (2017), Associative Embedding (Newell et al. (017). PersonLab (Pa- pandreou et al., 2018), PifPaf (Kreiss et al. 2019) and CenterNet (Zhou et al. 2019). keypoint heterogeneous learning targets. This requires the model to learn the keypoint heatmaps encoding position information and the human knowledge guided signals encoding association information such as part hypotheses, part affinity fields, asso- ciative embeddings offset vector fields.
detection and grouping are usually regarded
In this paper we explore whether we can exploit the instance semantic clues implicitly used by the model to group the detected keypoints into individual instances. Our key intuition is that, when the model predicts a location of specific keypoint, it may know the human instance region this keypoint belongs to, which means that the model has implicitly associated related joints together. For example, when an elbow is recognized, the model may learn its strong spatial dependencies in its adjacent wrist shoulder but weak dependencies in the joints of other persons. Therefore, if we can read out such information learned and encoded the model. the detected keypoints can be correctly grouped into instances, without the help of the human pre-defined associative signals.

---

Under review as a conference paper at ICLR 2022
Association Reference
Layer 1
Layer
Layer
Layer
Layer
Layer
Example
Naive Self-attention
Supervised Self-attention
Example B
Naive Self-atlention
Supervised
Self-attention
Figure 1: We choose two examples to show differences between the naive and supervised self- attention patterns. The association reference for the naive self-attention is the average of all layers.
We argue that self-attention model (Vaswani et al. 2017) can meet this requirement because it can provide image-specific pairwise similarities between any pair of image positions without distance limitation, and the resulting attention patterns show object-related semantics. Hence, attempt to use self-attention mechanism to perform multi-person pose estimation. But instead of following the top-down strategy with the single person region the input, feed the Transformer model the high-resolution input images with the presence of multiple persons, and expect it to output the heatmaps encoding multi-person keypoint locations. Our initial results show that 1) the heatmaps outputted by Transformer can also accurately respond to multiple persons' keypoints at multiple candidate locations; 2) the attention scores between the detected keypoint locations tend be higher within the same person but lower across different persons. Based on these findings, we introduce an attention-base based parsing algorithm to group the detected keypoints into different human instances.
Unfortunately, the naive self-attention does not always show desirable properties. In many cases, detected keypoint also probably have relatively higher attention scores with those belonging to different person instances. This will definitely lead to wrong associations and implausible human poses. To address this issue, we propose a novel method that leverages a loss function to explic- itly supervise the attention area of each person instance by the mask of the instance. The results show that supervising self-attentions in such a way can achieve the expected instance-discriminatie characteristics without affecting the standard forward propagation of Transformer. Such charac- teristics guarantee the effectiveness and accuracy of the attention-base grouping algorithm. The results the COCO keypoint detection challenge show that our models with limited refinement can achieve comparable performances compared with the highly optimized bottom-up pose estima- tion systems (Cao et al. 2017 Newell al. 2017 Papandreou al. 2018). Meanwhile, also can easily obtain the person instance masks by sampling the corresponding attention areas, thereby avoiding an extra pixel assignment or grouping algorithm.
1 KEY CONTRIBUTIONS
Using self-attention to unify keypoint detection, grouping and human mask prediction. We use Transformer solve the challenging multi-person keypoint detection, grouping and mask prediction in unified way. We realize that the self-attention shows instance-related semantics, which can be served as the association information in a bottom-up fashion. We further use instance masks to supervise the self-attention It ensures that each keypoint is assigned to the correct human instance according to the attention scores, making it easy to obtain the instance masks as well.
Supervising self-attention "for your need". A common practice of using Transformer models is to use task-specific signals to supervise the final output of transformer based models, such as class labels, object box coordinates, keypoint positions semantic masks. In this method, key novelty to use some type of constraint terms to control the behaviors of self-attention. The results show that under supervision the self-attention achieve instance-aware characteristics for multi-person
2

---

Under review as a conference paper at ICLR 2022
pose estimation and mask prediction, without destroying the standard forward of Transformer. This demonstrates that using appropriate guidance signals makes self-attention controllable and help the model learning, which is also applicable to other vision tasks such instance segmentation (Wang et al. 2021 and object detection (Carion et al. 2020).
METHOD
2.1 PROBLEM SETTING
Given RGB image / of size x H x W, the goal of 2D multi-person pose estimation to estimate all persons" keypoints locations: S {xf y/ y/ 1, 2..,, N; k 1, ..., KJ. where N is the number persons in this image and is the number defined keypoint types.
We follow the bottom-up strategy. First, the model detects all the candidate locations for each type of keypoints in an image: C C1 C2 U...UUU where C₁ = [ 3/ ..,, N₂. represents the k-th type of keypoint set with N& detected candidates. Second, heuristic decoding algorithm g groups all candidates into M skeletons based on the association information A. which determines unique person ID) for each keypoint location. We formulate this process g( Us), A) m E {1, (1,2...,MM. M).
Next, we present the model architecture and show how to use self-attention as the association infor- mation A. We analyze the problems when using the naive self-attention as the grouping reference. We propose to supervise self-attention via instance masks for keypoints grouping- We present two types of grouping algorithm from the body-first and part-first views. Finally, we describe how we obtain the person instance masks and how we use the obtained masks refine the results.
2.2 NETWORK ARCHITECTURE AND NAIVE SELF-ATTENTION
Architecture. We use a simple architecture combination that includes ResNet (He et al. 2016)) and Transformer encoder /Vaswani al.) 2017), like the design of TransPose (Yang al.] (021). The downsampled feature maps of ResNet with r stride are flattened to a sequence of L. x d size and sent to Transformer where L are used to upsample the Transformer output into the target keypoint heatmap size x x 4
W. Several transposed convolutions and a 1 convolution W.
Heatmap loss. To observe what patterns the self-attentions layers spontaneously learn, we first only leverage the mean square error (MSE) loss between the predicted heatmap H, and the groundtruth heatmap HŞ to train the model:
K
1
Cheatmap
M k=1
H
H
(I)
K
where M is a mask that masks out the crowd areas and small size person segments in the whole image. After the model is trained only by heatmap loss, the keypoint detection results show the trained model can accurately localize keypoints of multiple persons.
Issues in naive self-attention. We obtain the keypoint locations from heatmaps and further visu- alize the attention areas of these locations. As revealed by the examples shown in Figure using the naive self-attention matrices as the association reference poses several challenges: 1) There are multiple attention layers in Transformer, each of which shows distinct characteristics. Selecting which attention layers as the association reference and how to process the raw attention require very thoughtful fusion and post-processing strategy. 2) Although most of the sampled keypoint lo- cations show local attention areas, especially for the people they belong to, some keypoints may still produce relatively high attention scores for the parts of other people at longer distance. It is almost impossible determine perfect attention threshold for all situations. which makes keypoint group- ing highly dependent on specific experimental observations. As a consequence, the attention- -ased grouping cannot ensure the correctness of the keypoint assignment, leading to inferior performance.
3

---

Under review as a conference paper at ICLR 2022
Porson Alserionn
Person GT Mask M
Abeetion Matt:
Burrpted Neypoet
Supervie
Boy
Parsen My
Sampling
Superviee
Redivpa
Keyacett llesraape
Saet-
Sus
FFN
Sell
Tramsposed
BapenviAl
Figure 2: Model overview. The model architecture consists of three parts: a regular ResNet, regular Transformer encoder, and several transposed convolutional layers. Two types loss function are leveraged to supervise the model training. The final output of the model is supervised by the groundtruth keypoint heatmaps. One of the immediate self-attention layers is sparsely supervised by the instance masks. In particular, we sample the rows of the attention matrix of the chosen attention layer according to the visible keypoint locations of each human instance, reshape them into 2D-like maps, and then use the mask of each instance to supervise the average map. In this figure, we only show a few keypoints cach instance for simplicity.
2.3 SUPERVISING SHLF-ATTENTION BY INSTANCE MASKS
To address the aforementioned challenges of using the naive self-attention for keypoints grouping, we Supervise Self-Attention (SSA) to be what we expect. Ideally, the expected attention pattern should be that each keypoint location only attends to the person instance it belongs to. The value distribution (0 or 1) in a person instance mask provides an ideal guidance signal to supervise the pairwise keypoints's 's locations to have lower higher attention scores. Then we propose sparse sampling method based on the instance keypoint locations to supervise the specific attention matrix generated by the self-attention computation in Transformer, illustrated in Figure
Instance mask loss. We suppose that the p-th person's keypoints groudtruth locations are (x/h w/ ) k=11 k=1 where op E {0, is visibility flag, i.c., op 0: not labeled, oa 1: la- beled. We take out the immediate attention matrix A Softmax( OK
LL of the specific vd
layer in Transformer to leverage the supervision. We first reshape the attention matrix A into tensor A of (h x w) x (h x w) size, where h = H/r, w = W/r. Then we transform the keypoint coordinates into the coordinate system of the downsampled feature maps. And then take out the corresponding rows of the attention matrix specified by these locations. So we can obtain the re- shaped attention map each keypoint location: A Ajint( / /r), (nt(x///)) i, For person instance, we sample and average the attention maps based on its visible keypoint locations to estimate the mean attention map. We name as person attention map Api
Thhen
K
A,
Alint(yy/r)
Li=1
Assuming the groundtruth mask of the p-th person in the image is Mn. also the MSE loss function to supervise the attention matrix sparsely. Since the self-attention scores have been normal- ized by the softmax function, we need to rescale the Ap by dividing its maximum value that the rescaled Ap is closer to the value range of the annotated mask. Note that the size of Ap is while the grountruth instance mask is constructed to be H 4 x size. So we use r/4 times bilinear interpolation to resize the Ap to have the same size as the instance mask. We formulate the instance mask loss as:
Lmask = MSE(bilinear(AAp (Ap/ max (Aax(AA)): M,)
[bilinear (Aph maxxaxxx) (Ap)) - Mp 11 (3)
'Q. K E Lxd are queries and keys. For simplicity consider there is only one head. For multihead self attention, the attention matrix A is the average of all heads' attention matrices.
4

---

Under review as a conference paper at ICLR 2022
Objective. So the overall objective for training the model is:
Liroin Lheatmap + B Lmosk
(4)
where a and / are two coefficients to balance two types of learning. In the standard self-attention computation of Transformer, the attention matrix is computed by the inner products of queries and keys. Its gradient weighted sum of values. By introducing the instance mask loss supervise the self-attention, the gradient learning direction for the supervised attention matrix has two sources: the implicit gradient signal from keypoint heatmaps learning and the explicit similarity constraint from instance mask learning. Choosing approximate values of a and 3 is critical for training the model well. We set a= = 1 = 0.01 to balance both heatmap learning and mask learning.
procpppppagttinn information is entirely derived from the subsequent attention
2.4 KEYPOINTS GROUPING
When the well-trained model makes a single forward pass for a given im- age, can decode the multi-person human poses and masks from the outputted keypoint heatmaps and the supervised attention matrix in the immediate attention layer. We first conduct non-maximum suppression in a 7x7 local window on the key- point heatmaps and obtain all local maximum locations whose scores exceed the threshold t. We put all these candidates into a queue and decode them into skeletons using the attention-based algorithm. Us- ing the self-attention similarity ma- trix with quadratic complexity in- evitably brings redundant computation.
Morgan
Vertex Keypoint
Found keypoint in skeleton
Unmatched candidate keypoint
POLO
Edge Pairwise attention
High attention score
Low attention score
Figure 3: Self-Attention based Grouping. When the founded keypoints in a skeleton induce a stronger attention attraction to an unmatched keypoint, this candidate will be assigned to this skeleton. The blue edges (thick) have totally higher attention scores than the green edges (slim).
However, in part, this also makes minimal assumptions about where the keypoints of the instances may appear and the number of persons in the image. Next we present the self-attention based algorithms from the body-first and part-first views.
Body-first view. This view aims to decode each person skeleton one-by-one from the queue. As- suming we have the sorted all types of candidate keypoints by descending order of score a single queue, we pop out the first keypoint (maybe any keypoint type) to seed a new skeleton S, and then greedily find the best matched adjacent candidate keypoint from the queue.
For the seeded S with the initial keypoint, we find the other keypoints along the search path ac- cording to defined human skeleton kinematic tree. When looking for a certain type of joint, the founded joints (denoted as the set Sy) of this skeleton S induce basin of attraction to "attract" the joint that most likely belongs to it, as illustrated in Figure 3 For a certain unmatched point Pa (I. y; s) in the candidate set C& of the keypoint type k. between the current found keypoints and pe as the metric to measure the attraction from this skele- tor Attraction(p.- (p.. | a'"') s' A[y,, v''] Thus the candidate point with the highest score Attraction considered to belong to the current skeleton S:
the mean attention scores
(5) We repeat the process above and record all the matched keypoints until all keypoints of this skeleton have been found. Then we need to decode the next skeleton. We pop the first unmatched keypoint seed new skeleton S' again. We follow the previous steps to find keypoints belonging this instance. Note if the Auraction(p" Sy) is smaller than a threshold À (empirically set to 0.0025), this type of keypoint in this skeleton to be empty fellonillingg. It is also worth noting that we also consider the keypoints that have already been claimed by a previous skeleton S, but only when Attraction(p.. S)) Attraction(2) (pe, S), we assign the matched Pa to the current skeleton S
_ argmaxp.ec, s Attraction Sy).
2To ohtain the correct coordinate (int(æ/r), int(y//)), use (3r, 2) to omit the downsampling factor and rounding operation for simplicity.
5

---

Under review as a conference paper at ICLR 2022
Table I: Results the COCO validation set. (res101, 11, i640) represents that ResNet-101; the output stride is 16; the input resolution is 640x640, Refinement# represents only refining the keypoints without filling. Refinement#2 represents refining the keypoints with filling.
Method
AP
AP
AP
APM
APr
AR
OpenPose Cao et 2017
58.4
81.5
62.6
54,4
65.1
OpenPose Refinement (Cáo al. 2017
61.0
84.9
67.5
56.3
69.3
PersonLab (res 101, s16, i601) Papandreuu al. 2018
53.2
76.0
56.3
38.6
73.1
57,0
PersonLab (res 101, s16, i801) ( apundrrou et a
2018
60.0
82.1
64.3
49.7
74.6
64.1
PersonL ab (0es s16, i1401)
Papandreou el al.
2018
65.6
85.9
71.4
61.1
72.8
70.1
Ours (res 101, s16, 16-40)
50.4
78.5
53.1
41.6
62.8
56.9
Ours (res s16, i644))
50.7
77.7
53.6
41.1
64.2
56.9
Ours (res152, s16, i6-40) Refinement#1
58.7
81.1
62.9
54.0
66.0
63.9
Ours (5cs s16, 16400)
Refinement#2
65.3
85.8
71.3
59.1
74.4
Ours (res101, s16, 1800)
51.6
79.7
55.1
44.6
61.2
57.9
Ours (res101, s16, 1800)
Refinement#1
59.3
82.1
63.7
56.4
63.6
64.6
Ours (res 101, s16, 1800) Refinement12
66.4
86.1
72.6
61.1
74.0
71.2
Part-first view. This view aims to decode all human skeletons part-by-part, Given all candidates for each keypoint type, initialize multiple skeleton seeds {S', S ..... Smm) with the most cas- ily detected keypoints such as nose. Then we follow a fixed order to connect the candidate parts to the current skeletons. These skeletons points. Like the body-first view, we also use the mean attention attraction Attraction(pe- St) from the found keypoints in the skeletons as the metric to assign the candidate parts (Figure 3 But in the part-first view, compute the pairwise distance matrix between the candidate parts and exist- ing skeletons, and then we use the Hungarian algorithm (Kuhn) 1955) to solve this bipartite graph matching problem. Note,
be as multiple clusters consisting of found key-
Attraction(p... S*) that represents matching in the solution is lower than threshold X, we use this corresponding candidate part to start a new skeleton seed. We repeat the process above until all types of candidate parts have been assigned. This part-first grouping al- gorithm can achieve the optimal solution for assigning local parts to the skeletons although it cannot guarantee the global optimal assignment. We choose the part-first grouping as the default. And we compare both algorithms on the performance, complexity and runtime in Appendix A.5
2.5 MASK PREDICTION
The instance masks are easy to obtain after the detected keypoints have been grouped into skele- tons. To produce the instance segmentation results, we sample the visible keypoint locations {(it Uhh at
of the m-th instance from the supervised self-attention matrix: Am Eg 6(05 >000)00)--- Alot. (e
K
km
Then we achieve the estimated instance mask: Mm max(A...
Am
Σx 6169 20)
where threshold (0.4 by default) determine the mask region. When we obtain the initial skeletons and masks for all person instances, the joints of person may fall in multiple incomplete skeletons, but their corresponding segments (sampled attention areas) may overlap. Thus we further perform non-maximum suppression to merge instances if the Intersection-over-Maa (IoM) of two masks exceeds 0.3, where Max denotes the maximum area between two masks.
3 EXPERIMENTS
Dataset. We evaluate our method on the COCO keypoint dectection challenge (Lin et al. 2014 and on the instance segmentation of the COCO person category.
Model setup. We follow the model architecture design TransPose (Yang et al. 202 to predict the keypoint heatmaps. The setup is built on top of pre-existing ResNet and Transformer Encoder. We use the Imagenet pre-trained ResNet- 101 or ResNet-151 as the backbone whose final classifica- tion layer is replaced by output stride of ResNet backbone is 32 but we increase the feature map resolution of its final stage (C5 stage) by adding the dilation and removing the stride, i.e., the downsampling ratio of ResNet
convolution reduce the channels from 2048 to (192). The normal

---

Under review as a conference paper at ICLR 2022
is 16. We use a regular Transformer with 6 encoder layers with single attention head for each layer. The hidden dimension of FFN is 384. See more training and inference details in Appendix A.
3.1 RESULTS ON COCO KEYPOINT DETECTION AND PERSON INSTANCE SEGMENTATION
The standard evaluation metric for COCO keypoint localization is the object keypoint similarity (OKS) and the mean average precision (AP) over 10 thresholds (0.5,0.55....0.95) is regarded as the performance metric. We train our models on COCO train2017 set, and evaluate the model on the va12017 and test-dev2017 sets, as shown in Table I and Table 2 We mainly compare with the typical bottom-up models that have similar pipelines to our method: OpenPose (Cao et al., 2017). PersonLab (Papandreou ct al. 2018), and AE (Newell ct al.. 2017). Following the works (Cao al, 2017 Newell et al. 2017). we also refine the grouped skeletons using a single pose estimator. We Mannsoose--AAS (Yang et al. 2021) that has very similar architecture to our model and has only 6M parameters. We apply the single pose estimator to each single scaled person region achieved by the box containing the person mask. Note that the refinement results highly dependent on the effect of the grouping and mask prediction, and we only update the keypoint estimates where the predictions of the two models almost the same. The concrete update rule is whether the keypoint similarity (KS) metrid computing between two keypoints exceeds 0.75, indicating that the distance between two predicted locations is already very small.
adopt the COCO pretrained
Good
65.6
Good 78.8
Jit.
10.6
J1t. 18.0
11t.
11.1
Inv. 3.7
Inv.
2.6
Inw 3.8
Iev. 2.9
Mice: 7.1
M155 5.1
Miss 12.3
Mis5 5..6
Swap: 1.9
1.3
(a) OpenPose
(b) TransPose-R
(c) Ours (BU)
(d) Ours (BU+Refine)
Figure 4: Localization errors analysis on COCO validation set.
Analysis. We further analyze the differences between pure bottom-up results and the refined ones through the benchmarking and error diagnosis tool (Ronchi & Peronal 2017). We compare our methods with the typical OpenPose model and the Transformer-based model. The yielded localiza- tion error bars (Figure reveal the weaknesses and strengths of our model: (1) Jitter error: The heatmap localization precision under the existence of multi-person is still not as accurate as the lo- calization precision of single pose estimate; (2) Missing error: Since our algorithm does not ensure that the coordinate of every keypoint in a detected pose has been predicted, if the GT coordinate of a keypoint is annotated, zero-filling coordinates will seriously pull down the calculated OKS value. Thus, for the evaluation, it is necessary to produce complete predictions. When we further use the single pose estimator to fill the missing joints with zero scores in the initially grouped skeletons, it achieves about 7 AP gains (Table and reduces the missing error (shown in Figure Ard); (3) Inversion ervor: Forcing diverse keypoint types in an individual instance to have higher query-key similarity may make it difficult for the model to distinguish different keypoint types, especially the left and right inversion; (4) Swap error: We notice that our pure bottom-up model has fewer swap errors (1.2%, shown in Figure 4(c)). which represents less confusion between semantically similar parts of different instances. It indicates that compared with OpenPose model, our attention-based grouping strategy performs relatively better in assigning parts to their corresponding instances. We show the qualitative human poses and instance segmentation results in Appendix A.6
Person instance segmentation. We evaluate the instance segmentation results on COCO val split (person category only). We compare our method with PersonLab (Papandreou et al. 2018). In Table we report the results with a maximum of 20 person proposals due to the convention of the COCO person keypoint evaluation protocol. The results on the mean average precision (AP) show that our model still has a gap in the segmentation performance compared with PersonLab. We argue that this is mainly because we conduct the mask learning on low-resolution attention maps that have
We consider the per-keypoint standard deviation and object scale as the standard OKS metric does.

---

Under review as a conference paper at ICLR 2022
Table 2: Results on the COCO lest-dev2017 set. The methods marked with use the multi-scale inference settings. Our result is achieved based on ResNet-101 model with 16 output stride and 8002 input resolution.
Method
AP
APe.s
APR.7
AP
AP
AR
ARu.
ARa.75
ARMr
AR,
Top down
G-RMI (Papandroou et al
2017
64.9
85.5
71.3
62.3
70.0
69.7
88.7
75,5
64,4
77.1
Mask-RCNN (Har kle 2017
63.1
87.3
68.7
57.8
71.4
SimpleBaseline Alao é 2018
73.7
91.9
81.1
70.3
80.8
79.0
HRNet (Sun
Bottom-up
OpenPise (Cad et 2017
61.8
84.9
67.5
57.1
68.2
AF (Newell Pil
62.8
84.6
69.2
57.5
70.6
AE Newelll el al 2077
65.5
86.8
72.3
60.6
72.6
70.2
89.5
76.0
64.6
78.1
PersonLab (Pspandreoe
2018
66.5
58.0
72.6
62.4
72.3
71.0
90.3
76.6
66.1
77.7
Personl.ab (Famandroou
2018
68.7
89.0
75.4
64.1
75.5
75.4
92.7
81.2
69.7
83.0
SPM Nie
66.9
88.5
72.9
62.6
73.1
Highe/AKiNor Caaa 0
2020
70.5
89.3
77.2
66.6
75.8
DEKR" (Geng
71.0
89.2
78.0
67.1
76.9
76.7
93.2
83.0
71.5
83,9
Ours (SSA)
65.0
86.2
72.2
60.1
71.8
70.1
88.9
76.2
64.2
78.2
Table Instance segmentation results (person class only) obtained with 20 proposals per image on the COCO validation set.
Methud
AF
AR..k.
ARR
Penurllah (mo1101 enieen ieporst
13.8
56.0
16.8
76
45.9
59.1
15.6
37.
58.1
3.0
51.4
63.0
Oura (mad 52, stdde116 iepu-164ll
30.
43.5
16.9
03
26.5
590
12.9
29.
46.1
1.0
36.1
68.5
Ours (nx101, strde116, impu-9000)
22.0
453
188
09
27.7
55.
1322
308
32.0
1.8
41.1
66.9
been downsampled 16 times w.r.t. the 640 or 8002 input resolution while the reported Personl the 14012 input resolution. As shown in Table 3
result is based on 8 times downsampling our model performs worse on small and medium scales but achieves comparable or even superior performance on large scale persons even if PersonLab uses larger resolution.
3.2 COMPARISON BETWEEN NAIVE SELF-ATTENTION AND SUPERVISED SELF-ATTENTION
To study the differences in model learning when trained with and without supervising self-attention, we compare their convergences in the heatmap loss and instance mask loss, since the overfitting on COCO train data is usually not an issue. As illustrated in Figure compared with training the naive self-attention model, supervising self-attention achieves better fitting effect in the mask learning, while achieving an acceptable sacrifice on the fitting of heatmap learning. It is worth noting that the instance mask training loss curve of the naive eelf-attention model drops slightly, which suggests that the spontaneously formed attention pattern has tendency to instance- awareness. To quantitatively evaluate the performance of using naive self-attention patterns for keypoint grouping, we average the attentions from all transformer layers as the association reference (shown Figure When we use the totally same conditions (including model configuration, training testing settings and grouping algorithm) of the supervised self-attention model based on (res152, s16, i640), we achieve 29 99.0AP on COCO validation set, which is far from the 50.7AP result achieved by supervising self-attention.
4 RELATED WORK
Transformer. Transformer Vaswani al., 2017 has shown very powerful visual relation mod- eling capability in various computer vision tasks, such as image classification (Dosovisskiy et al.) 2020 Touvron et al. 22000), object detection (Carion et al. 2020 Zhu et al.) 2020), semantic seg- mentation (Zheng et al. 2222), tracking (Sun al., 20200) Meinhardt et al. (021). human pose estimation (Lin et al. 2021 Li et al. 2021aa at Yang et al.] 2021 ei al. 2021b Stoffl et al. 202T) and etc. The common practice of these methods to the task-specific supervision signals such as class labels, object box coordinates, keypoint positions or semantic masks to supervise the fi- nal output of transformer-basee models. They may visualize the attention maps to understand the
8

---

Under review as a conference paper at ICLR 2022
Heatmmpp loss comparison w/ and w/o Instance mask supervision
Mask loss comparison w/ and wio instance mask supervision
2.38
Heatmoo wwosotraintoo 1052
0,0812
w/ supervising self-attention
supervising selt attention
Instarce mhack 1o3s
2.175
w/ supervising self-attention supervising attention
8.139
0,3110
0.125
0.20
0.755
0.2330
800
1000
293
420
900
1030
Iteratiion (X100)
Iteration [x100)
Figure 5: The convergences on the heatmap loss and instance mask loss when trained with and without supervising self-attention.
model but few works directly use it as an explicit function in the inference process. Our work gives a successful example of explicitly using and supervising attention for a specific purpose.
Human Pose Estimation & Instance segmentation. Multi-person pose estimation methods are usually classified into two categories: top-down (TD) or bottom-up (BU), TD models first detect persons, and then estimate single pose for each person, such as G-RMI (Papandreou et al., 2017), Mask-RCNN (He et al. 2017). CPN (Chen et al.) 2018), SimpleBaseline (Xiao et al. 2018), 3. and HRNet (Sun et al. 2019). BU models need to detect the existence of various types of keypoints any position and scale. And matching keypoints into instances requires the model to learn dense association signals pre-defined by human knowledge. OpenPose (Cao et al., (017) proposes part affinity field (PAF) to measure the association between keypoints by computing the integral along the connecting line. Associative Embedding (Newell et al. 2017) abstracts an embedding as the human 'tag' ID to measure the association. PersonLab (Papandreou et al. 2018) constructs mid- range offset as the geometric embedding to group keypoints into instances. In addition, single-stage methods (Zhou et al. 2019; Nie et al., 2019) also regress offset field to assign keypoints to their centers. Compared with them, we use Transformer to capture the intra-dependencies within person and enterddependences across different persons. And we explicitly exploit the intrinsic property of self-attention mechanism to solve the association problem, rather than regressing highly abstracted offiset fields or embeddings. The generic instance segmentation methods also can be categorized into top-down and bottom-up schemes. Top-down approaches predict the instance masks based on the object proposals, such as FCIS (Li al.) 2017) and Mask-RCNN (He et al.) 2017). Bottom-up approaches mainly cluster the semantic segmentation results to obtain instance segmentation using an embedding space a discriminative loss measure the pixel association like (Newell et al., 2017 De Brabandere et al. 2017: Fathi et al. 2017). Compared with them, our method uses self- attention to measure the association and estimates instance masks based on instance keypoints.
DISCUSSION AND FUTURE WORKS
This paper presents
method solve keypoint detection and instance association by using Transformer. We supervise the inherent characteristics of self-attention - the feature similarity be- tween any pair of positions to solve the grouping problem of the keypoints or pixels. Unlike a typical CNN-based bottom-up model, it no longer requires a pre-defined vector field or embed- ding as the associative reference, thus reducing the model redundancy and simplifying the pipeline. We demonstrate the effectiveness and simplicity of the proposed method on the challenging COCO keypoint detection and person instance segmentation tasks.
The current approach also brings limitations and challenges. Due to the quadratic complexity of the standard Transformer, the model still struggles in simultaneously scaling up the Transformer capacity and the resolution of the input image. The selection of loss criteria, model architecture, and training procedures can be further optimized. In addition, the reliance on the instance mask annotations also can be removed in future works, such by imposing high and low attention con- straints only on the pairs of keypoint locations. While, the current approach still has not yet beaten the CNN-based bottom-up state-of-the-ar counterparts developed with many sophisticated designs, we believe it is promising to exploit or supervise self- attention to solve the detection and association problems in multi-person pose estimation, and other tasks or applications.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
The Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh, Realtime multi- 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7291-7299, 2017.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV, pp. 213-229, Cham, 2020.
Yilun Chen, Zhicheng Wang, Yuxiang Peng. Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7103-7112, 2018.
Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang. and Lei Zhang. Higherhr- net: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5386-5395, 2020.
Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with discriminative loss function. arXiv preprint 70x1::1708..25511.222
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /929, 2020.
Alireza Fathi, Zhigniew Wojna, Vivek Rathod, Peng Wang, Hyun Song, Sergio Guadarrama, and Kevin P Murphy. Semantic instance segmentation via deep metric learning. arXiv preprint 1rXiv:1703.10271 2017.
Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. Bottom-up human pose estimation via disentangled keypoint regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14676-14686 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schiele. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer Vision, pp. 34-50. Springer, 2016.
Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Pifpaf: Composite fields for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 11977-11986, 2019.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83 1955.
Ke Li, Shijie Wang, Xiang Zhang, Yifan Xu, Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1944-1953, 2021a.
Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021b.
Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance- aware semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2359-2367, 2017.
10

---

Under review as a conference paper at ICLR 2022
Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1954-1963, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tim Mcinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi-object tracking with transformers, arXiv preprint arXi::2101.02700 2021.
Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In Advances Neural Information Processing Systems, volume 30, 2017.
Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan. Single-stage multi-person pose machines. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6951-6960, 2019.
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bre- gler, and Kevin Murphy. Towards accurate multi-person pose estimation in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4903-4911, 2017.
George Papandreou, Tyler Zhu, Liang-Chied Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. Personlab: Person pose estimation and instance segmentation with bottom-up, part- based, geometric embedding model. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 269-286, 2018
Matteo Ruggero Ronchi and Pietro Perona. Benchmarking and error diagnosis in multi-instance pose estimation. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 369-378, 2017.
Lucas Stoffl, Maxime Vidal, and Alexander Mathis. End-to-end trainable multi-instance pose esti- mation with transformers. arXiv preprint 11X1:2210..22115, 2021.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703, 2019.
Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple-object tracking with transformer. arXiv preprint arXiv:2012.154600 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. pp. 5998-6008, 2017.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng. Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8741-8750, 2021.
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pp. 466-481, 2018.
Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Keypoint localization via trans- former. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
11

---

Under review as a conference paper at ICLR 2022
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from eegueenceto-sqquenee perspective with transformers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 6881-6890, 2021.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. arXiv:1904.07850. 2019.
Objects as points.
arXiv preprint
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 TRAINING DETAILS
In the training phase, we use data augmentation with random scale factor between 0.75 and 1.5, random flip with probability 0.5, random rotation with #30 degrees, and random translate with +-40 pixels along the horizontal and vertical directions. The imput size is 640 or 8002 and thus the input sequence length of Transformer is 1600 or 2500, We use the Post-Norm Transformer architecture and ReLU activation function in FFN. We supervise the 4-th self-attention layer by default (the total layer depth is 6). By convention, we use 2 transposed convolution layers to upsample the Transformer output size to 160x 160 x00x200. The standard deviation for the Gaussian kernel in the generated heatmaps is set to 2. We use Adam optimizer to train the model. 'he model is distributed across 8 Tesla V100 GPUs with total batchsize of 16 8. The initial learning rate is set to batcheise x 0.0001, and decays 10 times at the 150-th and 200-th epochs respectively, with total of 240 training epochs.
8
A.2 INFERENCE DETAILS
The threshold score f for obtaining candidate keypoint from heatmaps is set to 0.0025. The final person masks are achieved by bilinear interploating the estimated Mn. the original image size. The skeleton kinematic tree used in the body-first grouping is defined as a graph structure: the vertices are all types of keypoints that are denoted as the numbers from
16 by the order defined by COCO dataset; the edges are defined as [(0. 1), (0, 2), (0, 3), (0, 4), (3, 5), (4, 6), (5, 7), (5, 11), (6, 8), (6, 12). (7, 9), (8, 10), (11, 13), (13, 15), (12, 14), (14, 16), (5, 6). (15, 16), (13, 14), (11, 12)].
A.3 ABLATION ON WHICH ATTENTION LAYERS SHOULD SUPERVISED
Supervising the self-ate iiffaattention matrix in different Transformer layer depths may have different effects on the heatmap and mask learning. To study such effects, we train smaller proxy model to compare their differences in the fitting of heatmap and instance mask loss on small subset (1/5) split of the COCO train set. The model configurations are: ResNet-50 based, 5762 input resolution and 5 transformer layers with d 160 and 320 hidden dimensions in FFN. Note that using a smaller model and small-scale training data inevitably reduces the overall performances of the model, but only aim to find the relative differences in supervising at different Transformer layers. As illustrated in Figure do not observe significant differences in both heatmap loss and instance mask loss when leveraging the mask supervision in different Transformer layers. We further evaluate all these models on the COCO validation set. As shown in Table 4 supervising one of the last three attention layers achieves better performance compared with supervising the first two layers. Especially, supervising the penultimate third-to-last layer shows better performance. This suggests that leveraging the instance mask loss in this layer depth is a better trade-off between heatmap learning and mask learning.
Heatmap comparison supervised layers
comparison for different supervised layers
0.05
6.0014
layer-1
layer-1
layer-2
training loss
layer-2
Heatmap training
layer-3
layer-3
0.0010
layer-4 4
layer-
0.0008
layer
layer
0.02
2.0004
0.0002
0.08
a2o
Iteration (x100)
Iteration (x100)
Figure 6: The convergences on the heatmap loss and mask loss when supervising the aelf-attention in different layer depths.
13

---

Under review as a conference paper at ICLR 2022
Supervised layer
AP
APa.5
APo.75
AMar
APL
AR
AP (with Refinement)
1-th
32.3
60.9
29.8
23.0
45.5
38.8
52.1
2-th
33.7
62.1
31.7
22.9
48.8
40.0
54.6
3-th
34.1
63.0
31.4
23.4
49.0
40.4
54.6
4-th
34.1
63.0
32,0
23.3
49.0
40.2
54.7
5-th
33.9
62.7
31,4
23.5
48.5
40.5
54.7
Table 4: Comparisons for different supervised layers on COCO validation set when using small
proxy model.
A.4 WILL AN INDEPENDENT SELF-ATTENTION HEAD BE BETTER THAN A SHARED ONE TO LEVERAGE THE INSTANCE MASK LOSS?
Intuitively, using independent self-attention head may be helpful to reduce the effect of intro- ducing an intermediate instance mask loss on the standard Transformer forward. Thus we try to mitigate the negative effect the heatmap localization by using independent self-attention head to leverage the mask supervision. This design will need to insert an extra self-attention layer to the transformer intermediate output, as shown in Figure [ However, by comparing the conver- gence of the training losses, we find no obvious difference in the heatmap loss fitting hetween using shared self-attention attention and independent self self-attention, while, the independent self-attention performs relatively better in fitting the instance mask loss.
When we test their performances on COCO validation set, we find both designs achieve similar performances, as shown in Table Such results indicate that using an independent layer to the in- termediate loss bring little gain, and introducing an intermediate instance mask loss may generate weak effect the prediction of keypoint heatmaps. We conjecture that the existence of the residual path parallel to the supervised self self-ateention layer may also adaptively reduce the effect the in- stance mask loss on the subsequent transformer layers, since only leverage the sparse constraints to the self-attention matrix certain transformer layer.
Shared self-attention
Independent self-attention
Supentieed Selt-ltention
IN
FFN
+
Sel-Attertion
LN
PFN
LN
instance Mosk loss
Suparvised Self-Attrtion
Instance Mask loss
Figure 7: The architecture designs for supervising shared self-attention and independent self- attention.
Heatmap loss comparison with different supervised method
Mask loss comparison with different supervised methed attention
3.004
Heatmap training (665
independent self-attentio
in independent self attention
0.33
800000
3.000
0.22
08042
0.33
3000
260
100
606
800
1000
200
+20
c0o
000
1000
Iteration (X100)
Iteration
(c100)
Figure 8: The convergences on the heatmap loss and mask loss when trained with supervising shared self-attention and independent self-attention.
14

---

Under review as a conference paper at ICLR 2022
Supervision type
AP
APa.5
APo.75
AP
APL
AR
ARo.s
ARa.75
ARM
ARg
Shared
50.7
77.7
53.5
41.0
64.2
56.9
80.0
59.9
43.3
75.7
Independent
50.7
77.0
53.6
40.9
64.6
56.7
79.7
59.4
42.9
75.9
Table 5: Results COCO validation when using shared self-attention and independent self- attention designs.
A.5 RUNTIME AND COMPLEXITY ANALYSIS
We take the ResNet-101 101 based model as the exemplar to test two types of grouping algorithm. We the total 5000 images from COCO validation For each image, we the model forward on a single GPU and the grouping algorithm on the CPU where the grouping runtime is far less than the model forward. In Table 6 we provide a controlled study to compare their differences in the model performance, theoretical complexity for per part assignment, runtime for the whole inference pipeline (keypoint detection, grouping and instance segmentation). Note that the complexity is theoretical analysis based on the assumption that there are N existing skeletons and N candidates for a certain part type. We report the performances and runtime for the pure bottom-up result and the ones with refinement.
Theoretical complexity for (BU) per part assignment
Grouping Algorithm
Runtime (BU)
(BU+Refine)
Runtime (BU+Refine)
Part-first view Body-first view
( ONN)
50.4
8.45 img/see
65.3
5.69 img/sec
49.7
8.94 img/see
64.8
5.72 img/sec
Table 6: Comparison between the body-fine first and part-first grouping algorithm
In Table 7 we compare our models with the mainstream bottom-up models, in terms of the number of model parameters and computational complexity of the model forward pass. The results of Hour- glass (Newell et al.] 2017), PersonLab (Papandreou et al. 2018), and HigherHRRNet (Cheng et al. 2020) are taken from the HigherNet paper (Cheng al. 2020 We see that compared with them, our models have fewer parameters and less computational complexity in the model forward pass.
Model
Input Resolution
#Param
FLOPs
Hourglass Newell et al. 2017)
512
277.8M
206.9G
LersonLab (aapandreou et 2018
1401
68.7M
405.5G
HigherHRNNt (Cheng et al.
2020
640
63.8M
154.3G
DEKR (Geng et al. 202
640
65.7M
141.5G
Ours ToasNet10--Transfnnmn
640
45.0M
102.3G
Ours
640
60.6M
132.7G
Ours RResNec101+Traasfomme
800
45.0M
159.8G
Table 7: Comparisons on the number of model parameters and model forward complexity.
A.6 VISUALIZATION FOR HUMAN SKELETONS, INSTANCE MASKS AND KEYPOINT ATTENTION AREAS.
In Figure we visualize the qualitative results predicted by our pure bottom model based on ResNet- 152 and 5T22 input resolution. Our model still can perform relatively well even in some hard cases, such as occluded persons and crowded scene with the existence of a large number of people (>30) (shown in the 4-th row in Figure We also can see that the attention areas of the sampled keypoints belonging to a specific person can accurately and reasonably attend to the target person and not attend to the areas excluding the person.
'NVIDIA Tesla V100 GPU and Intel(R) Xeon(R) Gold 6130 CPU 2. 1.
15

---

Under review as a conference paper at ICLR 2022
Figure 9: Qualitative visualization results predicted by our pure bottom up model. For each image, we show the original image plotted with human poses and masks. And, for each image, we also show the learned attention areas from the views of sampled keypoints, each location which has been annotated by white color pentagram. Redder areas mean higher attention scores.
16


Reviews:
[{"review_id": "t5v-AAySvD", "reviewer": "Reviewer_PtRJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe idea to incorporate self-attention models to extract semantic features for keypoints association is a novel and elegant approach to address this problem. The efficacy of this approach is also verified by the experimental results. For large objects, the neighboring keypoints could be several hundred pixels away and Personlab [1] adopt recurrent offset refinement to refine the inaccurate regressed long-range offsets. In contrast, the proposed model benefit from the self-attention and can easily group keypoints with large offsets.\n\nAnother contribution of this work is to unify keypoint detection, grouping and segmentation. The instance mask helps to supervise the association between keypoints of a single instance and the predicted keypoint detection would help produce better masks from a bottom-up perspective. We can see the two tasks are highly-correlated and the proposed approach seems to be a reasonable and novel design in this direction.\n\n**Concerns**\n\nSupervising self-attention with instance mask is a natural idea to assist keypoint grouping but there are some concerns related to the choice of supervision that are not addressed well. The visualization of the association reference in Figure 1 show that the supervised self-attention helps to discriminate keypoints from the same instance and from different instances. However, we could also see that the naive self-attention produce self-attention scores with good characteristics that start by looking at the context around the instance and then shift to focus on keypoints of the same instance. Meanwhile, the supervised self-attention seems to only focus on associating keypoints of the same instance, which is a direct result of the mean squared error (MSE) of the instance mask loss. It seems that the self-attention behavior is completely changed by the instance mask loss and subsequently the final keypoint heatmap. It is good to produce a instance segmentation mask at certain layer but it is unclear how will this affect the keypoint features and the keypoint detection. Will the instance mask loss make the keypoint features less discriminative and hence less accurate keypoint localization? There is no ablation study experiments on the design of the instance mask loss to address such concerns so the advantage of such loss is doubtful beyond the visualization results. Ablation study in A.3 show that there is no clear differences between supervising different layers of the self-attention model, which I don't understand since the self-attention scores focus on very different regions before and after the layer that is supervised.\n\nExperimental results are limited and fail to show the benefits of self-attention over regression. Most quantitative results are only roughly comparable with previous methods even with different refinements. The issue of Transformer models performing worse on smaller objects (e.g., DETR [2]) is also left unaddressed. While the proposed model can elegantly unify the task of keypoint detection and instance segmentation, the advantage of such approach is not supported by the experiments. The experiments cannot show the benefits of sharing the features for both keypoint detection and instance segmentation and there are no known superior aspects of the proposed approach in terms of number of parameters, MACs, or post-processing time.", "summary_of_the_paper": "Bottom-up methods tackle the problem of multi-person detection, pose estimation, and segmentation by localizing human keypoints and then grouping them into person instances, which inevitably bring up the question of what features to use for grouping and how to efficiently group keypoints and break down corner cases. In this work, the authors propose to use Transformer to exploit associative information between keypoints. Initial experiments show that naive self-attention has non-zero attention scores between keypoints from different human instances, and even high attention scores in crowded scenarios. Such self-attention layers make it hard to directly associate keypoints of the same instance from the self-attention scores, and subsequently requires careful hyperparameter tuning and inevitably introduce errors in hard examples. Therefore, the authors propose a instance mask loss to supervise self-attention with instance segmentation masks.\n\nAnother advantage of such framework is to unify the task of keypoint detection and instance segmentation. The segmentation mask can be easily generated from the self-attention scores and the network learns a representation that is consistent with both the keypoint locations and the instance masks.", "summary_of_the_review": "Overall, I think this paper presents a novel and interesting idea. However, the current experimental results are relatively weak and cannot show the strength of the method, which limits the contribution. There are also some concerns with the self-attention supervision that are not explained well and not supported by the ablation study.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "s-LD3PLA2Mm", "reviewer": "Reviewer_mi8h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengths: \n    * The idea is clear and reasonable. Self-attention module computes attention for every point in the feature map. It is reasonable to restrict the region where a self-attention module puts its attention by using some prior knowledge. The authors proposed two methods to restrict the attention region, one is a loss applied on attention map  which is interesting, and the other is attention-score based keypoint grouping. The visualized attention maps show that the proposed loss works well.\n    * The ablation study on Tab.1 shows the effectiveness of the proposed method.\n* Weaknesses: \n    * The idea is lack of novelty. The restrict attention region is reasonable but not novel, since the core idea is still instance segmentation.\n    * The performance is not promising when compared with some classic methods.\n    * As for instance segmentation, the authors argue that the worse performance is due to lower resolution. I think it is necessary to set up a fair experiment, otherwise the comparison is not very meaningful.\n    * In 3.2, only loss is shown, but it is not clear how the supervised self-attention contributes to accuracy.\n", "summary_of_the_paper": "This paper proposes a bottom-up multi-person pose estimation method based on transformer model. In this method, a new loss is used to encourage a self-attention map to put its attention on the region of an instance via the given ground-truth segmentation mask. Besides, the attention scores between estimated keypoint-locations are used to assign the estimation to correct instance. \n\nExperiments are applied on COCO keypoint dataset. The results shows that the proposed method outperforms or is comparable with some competitors such as OpenPose and Person Lab on keypoint detection. When compared with the existing methods which explicitly estimate instance segmentation mask, the proposed method performs worse.", "summary_of_the_review": "The idea is clear and reasonable, but lack of novelty and promising performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aLDq9V_Kmnh", "reviewer": "Reviewer_cESx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths: as far as I know it is the 1st paper to do mid-layer supervision and use instance mask to supervise the transformers attention.\n\nweakness: although they've done several ablation studies, there are not enough\n(1) what if the mean of the keypoint is replaced with the mean of vector inside bbox (this seems a more reasonable alternative)\n(2) what if this is a multitask transformer, decoded to both instance-segmentation mask and keypoint estimation. Will the end-supervision better be than intermediate supervision? \n(3) the final results are not compelling, even worse than PersonLab (2018)\n(4) what is the speed and complexity increment  after adding this intermediate loss / for inference (compared with original transformers)\n(5) the paragraph above Fig 5 seems not reasonable to me. This makes the method more like a top-down method instead of bottom-up, making this methods less compelling. \n(6) Fig 4 need more description to be  better understood.\n", "summary_of_the_paper": "The paper proposes to incorporate the instance segmentation task into the human pose estimation task (serving as attention layer loss for transformers). This supervision is novel and serves as a good alternative for AE/PAF, etc. They also propose two decoding methods (part-based / body-based). (with accuracy/time trade off. )", "summary_of_the_review": "Please refer to the Main Review of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SJiNeElYYl9", "reviewer": "Reviewer_YSvp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strong points.\n\nS1. Clear motivation.\nThe proposed SSA has a clear motivation. Supervising self-attention maps with human segmentation can guide the self-attention map to focus on correct pixels, which can lead to more accurate pose estimation.\n\nS2. Well-written manuscript.\nOverall manuscript and figures successfully deliver their main ideas.\n\n* Weak points.\n\nW1. Lack of numerical comparison between naive self-attention and SSA.\nFigure 1 and 6 show qualitative comparison between naive self-attention and the proposed SSA. However, no numerical comparison (i.e., AP on test or validation set) is not reported. This makes me hard to be convinced SSA is clearly more effective than naive self-attention.\n\nW2. Weak experimental results compared to recent state-of-the-art methods.\nTable 1 and 2 show comparison between the proposed method and state-of-the-art methods. Compared to other bottom-up methods, the proposed method fails to outperform them. Importantly, most of the bottom-up comparison targets are not recent ones. Recent bottom-up methods, such as [1] and [2], achieve far better results than the proposed one.\n[1] achieves AP 66.5 and 68.4 on validation and test-dev set, respectively, without multi-scale testing. [1] and [2] are representative bottom-up methods; however, they are not cited.\n\n[1] Cheng, Bowen, et al. \"Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation.\" CVPR. 2020.\n[2] Geng, Zigang, et al. \"Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression.\" CVPR. 2021.\n\nW3. Unfair comparison using the refinement.\nThe refinement introduced in Section 3.1 seems used only for the proposed method and not used for other bottom-up methods. The refinement is a kind of post-processing using off-the-shelf well-performing 2D human pose estimator, which is not a novel point of this paper. Applying the refinement only to the proposed method and comparing with other is unfair.\n\nW4. Lack of qualitative results.\nNo qualitative 2D human pose results are provided. \n\nW5. Is human segmentation the best choice?\nAlthough the human segmentation provides pixel-level annotations, it does not have human articulation information. As human keypoints are connected in the kinematic chain, I guess supervising the attention map with Gaussian heatmap or one-hot matrix could provide better results than human segmentation map since the heatmap and one-hot matrix has activation for each keypoint in channel dimension.\n", "summary_of_the_paper": "This paper presents a bottom-up 2D multi-person pose estimation system. The proposed system takes a single RGB image and predicts 2D Gaussian heatmaps of human keypoints, used to localize human keypoints. The main contribution of this paper is that they introduced a supervised self-attention (SSA), which supervises self-attention maps with human segmentation masks. The self-attention maps are computed in Transformer encoder modules, which represents how much each pixel pays attention to all pixels in feature maps. They showed that without SSA, naive self-attention maps tend to have activations on wrong positions.", "summary_of_the_review": "Although the paper has a clear motivation and is well-written, lack of experimental demonstration and weak experimental results are weak points.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LJhHXEAgqWd", "reviewer": "Reviewer_oDDC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like that the proposed approach takes advantage of an existing property of the TransPose architecture which is already doing the work of explicitly associating pixels with each other in each of the transformer layers, adding a little extra supervision is straightforward enough and can always be incorporated as dense transformer architecture design matures. Overall, the paper is written clearly and sufficient information is provided to reimplement the method.\n\nHowever I have some concerns:\n\n- Comparisons are left out to more recent bottom-up human pose work, notably missing is the HRNet family of work such as HigherHRNet (Cheng et al CVPR 2020) and the more recent DEKR (Geng et al CVPR 2021) both of which set a higher bar for bottom-up results and also do not rely on single-person pose refinement. Beating the state-of-the-art is certainly not a requirement here, but it is important to include other recent methods in comparisons.\n\n- It seems to me that the transformer layers introduce serious compute overhead (this is alluded to in the concluding discussion), the paper does not offer much information in terms of the cost of this approach compared to other work. It would be helpful to get some comparisons (e.g. images/sec on fixed hardware, memory requirements, etc).\n\n- A related point is that attention across pixel space requires that a large stride is used to operate at a reasonably low resolution. This is a difficult trade-off in bottom-up human pose where metrics are so sensitive to small localization jitter. It is unclear what good strategies to overcome this might be with this style of architecture.\n\n- Another caveat is the requirement for additional instance segmentation mask supervision which other methods do not rely on. Were any alternatives considered so that this method could be used on datasets where these annotations are unavailable?\n\n- One question I had is how restricting the attention in this way affects accuracy of the predicted heatmaps. From Fig 6 it seems as though it hurts performance somewhat. It would be interesting to know the effect here (for example, by comparing with and without the attention loss and evaluating using oracle assignment)\n", "summary_of_the_paper": "This work proposes explicit supervision of transformer attention for the purposes of bottom-up multi-person pose estimation. The network (TransPose [Yang et al. 2021]) passes an image through a resnet followed by several tranformer layers to produce per-pixel heatmaps localizing the joints of all people in the given image. The attention of the transformer layers is across pixel space, so the authors propose supervising the resulting distribution so that the attention score is higher between keypoints belonging to the same person. This is done by sampling the attention masks for keypoints and supervising them with MSE to match the annotated segmentation mask of that person. At test time, detected keypoints can be grouped into invidual people with a simple heuristic parsing of relative attention scores.", "summary_of_the_review": "The authors propose an nice, simple strategy to associate keypoints in multi-person pose estimation, but I don't know that a compelling enough case is made that this method offers advantages given the restrictions in terms of additional annotation requirements + compute overhead while falling short of and excluding more recent methods in performance comparisons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a bottom-up multi-person pose estimation method using a Transformer model. There is consensus among the reviewers that this paper is not ready for acceptance/publication. Although some reviewers find the proposed idea interesting (some find it lacking novelty though), all the reviewers agree that the quantitative experimental results are not promising. Some reviewers explicitly criticized lacking empirical accuracy compared to state-of-the-arts. The authors provided additional details and results in the rebuttal, but they were not sufficient to change the opinions of the reviewers.

We recommend rejecting the paper.
Decision: Reject