Heatmap Distribution Matching for Human Pose Estimation
Authors: Haoxuan Qu
Li Xu
Yujun Cai
Lin Geng Foo
Jun Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep learning
Human Pose Estimation
Distribution Matching

Abstract: For tackling the task of 2D human pose estimation, the great majority of the recent methods regard this task as a heatmap estimation problem, and optimize the heatmap prediction using the Gaussian-smoothed heatmap as the optimization objective and using the pixel-wise loss (e.g. MSE) as the loss function. In this paper, we show that optimizing the heatmap prediction in such a way, the model performance of body joint localization, which is the intrinsic objective of this task, may not be consistently improved during the optimization process of the heatmap prediction. To address this problem, from a novel perspective, we propose to formulate the optimization of the heatmap prediction as a distribution matching problem between the predicted heatmap and the dot annotation of the body joint directly. By doing so, our proposed method does not need to construct the Gaussian-smoothed heatmap and can achieve a more consistent model performance improvement during the optimization of the heatmap prediction. We show the effectiveness of our proposed method through extensive experiments on the COCO dataset and the MPII dataset.

Heatmap Distribution Matching for Human Pose Estimation
Haoxuan Qu
Li SUTD
SUTD
Singapore
Singapore
haoxuan_ caciaaanguummmiiil..m sutd.
1i www.xuummmmll..mmmm sutd edu
Yujun Cai NTU
Lin Geng Foo SUTD Singapore
Singapore yujun0010e. ntu atw..ee...tuuuuuuu.......
Singeng_foo@mynnnn huttyyaaalluuuu....mmm sutd, edu. sg
Jun Liu
SUTD
Singapore
jun_liu@sutd.eer sdu.
Abstract
For tackling the task of 2D human pose estimation, the great majority of the re- cent methods regard this task as a heatmap estimation problem, and optimize the heatmap prediction using the Gaussian-smoodle objective and using the pixel-wise loss (e.g. MSE) as the loss function. In this paper, we show that optimizing the heatmap prediction in such way, the model performance of body joint localization. which is the intrinsic objective of this task, may not be consistently improved during the optimization process of the heatmap prediction. To address this problem, from novel perspective, we propose formulate the optimization of the heatmap prediction as distribution match- ing problem between the predicted heatmap and the dot annotation of the body joint directly. By doing so, our proposed method does not need to construct the Gaussian-smootler heatmap and can achieve more consistent model performance improvement during the optimization of the heatmap prediction. We show the effectiveness of proposed method through extensive experiments on the COCO dataset and the MPII dataset.
heatmap the optimization
Introduction
2D human pose estimation aims to locate body joints of person in given RGB image. It is relevant to a variety of applications, such as action recognition (34), human-machine interaction [40]. and sign language understanding [19] For tackling the task of 2D human pose estimation, most of the recent methods [29] 20 33
[7 37] are heatmap-based, i.c., they regard 2D human pose estimation as a heatmap estimation problem. Specifically, for each body joint, these methods generally estimate grid-like heatmap, on which each pixel value represents the probability that this pixel contains the body joint. Compared the methods [30 2 32 B
26
14
39
35
[5]
that directly regress the coordinates of body joints (i.e. coordinate regression- baased methods), the caatmap-based methods
"Corresponinng Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ground-truth dot annotation coordinate
argmax coordinate prediction
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0.25
0.5
0.25
0.15
0.2
0.15
0.23
10.90
0.25
0
10.80
0
0.8
0
0
0
0
0
0.25
0.5
0.25
0
0
0.15
0.2
0.15
0
0.25
0.5
0.25
0
(a) GT dot-annotated heatmap
(b) GT Gaussian- smoothed heatmap
(c) predicted heatmap
(d) predicted heatmap #2 (pisel wise w.nt
(pixel went the GT Gaussian-smoothed heatmap- - 0.44)
the GT Geussian-mmoohhe
heatmap - 0.20)
Figure 1: Illustration of heatmaps. Although the pixel-wise MSE loss calculated between the predicted heatmap #2 and the Gaussian-smoothed heatmap is smaller than the loss calculated between the predicted heatmap #1 and the Gaussian- heatmap, the predicted heatmap #2 localizes the body joint wrongly, whereas the predicted heatmap #1 localizes the body joint correctly.
demonstrate a more robust performance since they maintain the spatial structure of the input image throughout the encoding and decoding process (7].
During the training process of the heatmap-based methods, an important step is the optimization of the heatmap prediction. This optimization can be done naively via constructing dot-annotated heatmap for each body joint as shown in Fig. [a), and then measuring the difference (i.e., conducting pixel-wise comparison) between the predicted heatmap and the constructed ground-truth (GT) dot- annotated heatmap. However, such a dot-annotated heatmap is sparse, as it has the same zero value for all pixels except the pixel representing the dot annotation of the body joint. Because of this, optimizing the heatmap prediction in such naive way can lead to hard training process and suboptimal model performance (28). To address this problem, most heatmap-baut based methods
37 adopt a strategy to construct Gaussian Gaussian-smoottee smoothed heatmaps, where pixels near the dot annotation have larger pixel values than pixels far from the dot annotation. Specifically, they construct the Gaussian-smoother heatmap via smoothing the dot annotation of the body joint through Gaussian distribution shown in Fig. [b), instead of only setting the pixel representing the dot annotation to be one.
29 20 33 26 [4.
39 35
15
17
While easing the training process, constructing the GT Gaussian-smoothed heatmap still brings problems into the model training process. Firstly, for constructing the Gaussian-smoothel heatmap, we need choose a proper standard deviation of the Gaussian distributions. However, the proper standard deviations of the Gaussian distributions (i.c., the standard deviations that can lead to an optimal performance) often vary across different types of body joints, different body postures, and different body sizes [17]. Hence, the standard deviations of the Gaussian distributions often need to be carefully chosen, which non-trivial. Secondly, during the process of optimizing the heatmap prediction by minimizing the pixel-wise loss (e.g. MSE) between the predicted heatmap and the moussiian-smoother heatmap, the model performance of body joint localization may not be consistently improved. As shown in Fig. [ although compared to the loss calculated between the predicted heatmap #1 and the Gaussian-smoothee heatmap, the pixel-wise MSE loss calculated between the predicted heatmap #2 and the Gaussian-mmoothed heatmap is smaller, the predicted heatmap #2 localizes the body joint wrongly, whereas the predicted heatmap #1 localizes the body joint correctly.
As result, optimizing the heatmap prediction using the dot-annotated heatmap and the Gaussian- smoothed heatmap the ground-truth both have their respective problems. Hence, in this work, we aim to tackle their respective problems, and propose to optimize the heatmap prediction directly via minimizing the difference between the predicted heatmap and the dot annotation. By doing so, we can optimize the model directly towards accurately localizing the dot annotation of the body joint, which is the intrinsic objective of 2D human pose estimation, instead of optimizing the model heatmap. However,
indirectly towards either the dot-annotated heatmap or the Gaussian-smootte as the number of pixels in the predicted heatmap and the number of entries representing the dot

---

annotation are different, we cannot measure the difference between the predicted heatmap and the dot annotation trivially by measuring their entry-wise difference. To handle this problem. inspired by the fact that we can measure the difference between two distributions via measuring their Earth Mover's Distance even if they have different numbers of entries, in this paper, we propose to first formulate the optimization of the heatmap prediction as a distribution matching problem. Specifically, we construct two distributions respectively from the predicted heatmap and the dot annotation. After that, we optimize the heatmap prediction via minimizing the distribution difference based on the Earth Mover's 's Distance. Using such a novel method to optimize the heatmap prediction directly from the dot annotation, we do not need to construct the the issues of the binary dot-annotated heatmap. Thus, our method achieves superior performance.
heatmap, well as avoiding
Our proposed method is simple yet effective, which can be easily applied to various off fff-the-heelf 2D human pose estimation models by replacing their original loss function with our proposed loss function measuring the distribution difference between the predicted heatmap and the dot annotation. We experiment our proposed method on multiple models and our method achieves a consistent model performance improvement.
The contributions of our work are summarized follows. We analyze (in Sec. that the performance of the human pose estimation model may not be consistently improved during the process of minimizing the pixel-wise loss between the predicted heatmap and the GT cquusian-smoothee heatmap. 2) From a novel perspective, we formulate the optimization of the heatmap prediction as a distribution matching problem between the predicted heatmap and the GT dot annotation directly, which bypasses the step of constructing the Gaussian-mootthed heatmap and achieves consistent model performance improvement. 3) Our proposed method achieves state-of-the-an performance on the evaluated benchmarks.
2 Related Work
2D Human Pose Estimation. Due to the wide range of applications, the task of 2D human pose estimation has received lots of attention [30 2
32
13 21 29 20 33 26 14] 4 39 35 15| I7
37 27 7 g). DeepPose 33) made the first attempt of applying deep neural networks into the task of 2D human pose estimation via directly regressing the coordinates of body joints. This type of often show inferior performances compared
coordinate regression-bass
methods (30 2
32
[3
to the heatmap- -based methods (29 20
15 17 37.. as the heatmap-based methods can preserve the spatial structure of the input image throughout the encoding and decoding
33
26
14
4
39 35
process 77. Hence, recently, the great majority of the state-of tatte-of-the-ase art methods (29 20 B3] 26
14
regard 2D human pose estimation as a heatmap estimation problem instead of the coordinate regression problem. Among the heatmap-based methods, Tompson et al.
39
35 15
17 37
29 proposed to apply Markov Random Field (MRF) into the task of 2D human pose estimation. After that, "hourglass" network, with conv-deconv architecture, was proposed by Newell et al. (20). Xiao et al. [33] proposed a baseline method to predict the heatmap via adding several deconvolutional layers to a backbone network. Later to maintain high- nigh-resolution representations throughout the heatmap estimation process, HRNet was proposed by Sun et al. [26]. Yuan et al.
37 further proposed HRFormer to learn the high- nigh-resolution representations utilizing transformer- based architecture. Besides the above heatmap-boo based methods that use the Gaussian -smooohhed
heatmap the optimization objective, there are also some methods (2 7 g that combine the idea of healmap and coordinate regression by taking the expectation of the predicted heatmap as the predicted coordinates.
Here in this work, different from previous works, our method bypasses both the step of regression and the step of constructing the Saussian-smoothee heatmap as the optimization objective. Instead, from a novel perspective, we propose to formulate the optimization of the heatmap prediction as distribution matching problem by minimizing the distribution difference between the predicted heatmap and the dot annotation.
Distribution Matching. The idea of distribution matching has been studied in various tasks 24 38 23 such as image retrieval [24], tracking (25), few-shot learning [88], and long-tail recognition 23 In this work, from novel perspective, we design a new distribution matching scheme to optimize the heatmap prediction with the help of sub-pixel resolutions for 2D human pose estimation.
25
3I
3

---

3 Method
In 2D human pose estimation, optimizing the heatmap prediction using the dot-annotated heatmap and the Gaussian-smoother heatmap as the ground-truth both have their respective problems. Hence, in this work, we aim to handle their respective problems, and optimize the heatmap prediction directly with the dot annotation of the body joint. To achieve this goal, we propose to formulate the optimization of the heatmap prediction as a distribution matching problem, and minimize the difference between the distribution constructed from the predicted heatmap and that constructed from the dot annotation based on the Earth Mover's Distance.
Below, we first give a brief review of the Earth Mover's ' Distance, and then discuss how we formulate the heatmap optimization process as distribution matching problem. After that, we introduce how we construct the loss function measuring the distribution difference.
3.1 Revisiting Earth Mover's Distance
The Earth Mover's ' Distance is technique used for measuring the difference between two probability distributions, which can be understood as the optimal cost needed transport the mass from one distribution to another. Specifically, for calculating the Earth Mover's s Distance, we regard the source distribution as set of (NV suppliers S ($1, SN) )" T where Sn, represents the total units of mass that the n-th supplier has, and we regard the target distribution as a set of (M) demanders D = (d1,..., di,...,d)) ) where dm represents the total units of mass that the m-th demander requires. Besides, we also denote C' E R as the cost function between the source and target distributions, where Cn.m represents the cost for transporting a unit of mass from the n-th supplier the m- th demander. Then we aim to find a least-cost transportation plan from the set of possible plans P = tp E R% N pl = S & p = D} to transport all mass from the N suppliers to the M demanders, where represents vector of ones. The Earth Mover's Distance Ec(S, D) denotes the cost of the least-cost transportation plan, which can be formulated as: (C,p)
20
Es D)
min pEP
where ( represents the Frobenius dot product.
However, optimizing Eq. T directly is computationally expensive. Hence, to reduce the computational cost especially when handling large-scale problems, Cuturi 5 proposed regularized formulation of the Earth Mover's Distance F. D) as:
[CC.D)
1 x (())
( D) = (C,p (09) where p0g arg min PEP
(2)
where À > O, and h())
N
m== Cr,m log On Optimizing Eq. 2 is computationally
M
m=
can be optimized with matrix scaling through the Sinkhorn cheaper than optimizing Eq. since Eq. 2 algorithm.
3.2 Problem Formulation
Below, we formulate the optimization of the heatmap prediction as a distribution matching problem by constructing the suppliers S, the demanders D, and the cost function C.
The suppliers S. For body joint, let Hpred E R@ denote its corresponding predicted heatmap with height и and width W. To formulate the suppliers S to represent Hpreds we first localize a set of H x W suppliers corresponding to the H x W pixels of Hpred. Then, for determining the units of mass each supplier stores, as the suppliers cannot hold negative units of mass, we construct S based on non-nate non-nagetive formulation of H Hpred- Besides, by the suppliers S to be the same as the total units of mass required by the demanders D. To meet these requirements, derive S by first passing H Hpred through a relu activation function and then normalizing it as:
also constrain the total units of mass stored
S relu(Hprea)
[ 11
The demanders D. As for the demanders, we aim to construct w.r.t. each body joint to represent its corresponding GT dot annotation. To achieve this goal, for each body joint, naive formulation of

---

D is to identify the pixel containing the GT dot annotation (i.e., the upper left pixel in Fig. 2aa) and construct a single demander (i.e., the yellow dot in Fig. aa) the center of this pixel. However, this naive formulation can result in suboptimal model performance, as the demander formulated in this way can noticeably different from what the GT dot annotation might suggest. Generally, the predicted heatmaps outputted by most of the existing heatmap-bass
methods (29 20] 33 26
[4|4 39 35 [S 17 (37] have a lower resolution compared to the input image. For example, for the method HRNet [26], when the size of the input image is 384 288, the size of the predicted heatmap 96 x 72 only. Due to such resolution gap, as shown in Fig. aa, there can exist a non-negligible distance between the location of the demander and the location of the dot annotation, which can affect the performance of the pose estimation model.
To address this problem, we aim to make the formulated demanders D a more accurate repre- sentation of the dot annotation by overcoming the resolution gap. To achieve this, inspired by the fact that the more accurate location informa- tion an object can be deduced using sub-pixel resolutions [1). we propose to formulate the demanders with the following four steps. (1) As illustrated in the upper left of Fig. 22 first split the pixel containing the dot annotation into four "sub-pixels" (four squares separated from each other by dashed lines). (2) After that, identify the "sub-pixel" that the dot annota- tion lies in (i.e., the green "sub-pixel" shown in Fig. b)). (3) Then, shown by the four yellow dots in Fig. 2]) we localize a set of four demanders the centers of both the pixel con- taining the identified "sub-pixel" (i.e., the upper left pixel in Fig. b)), and the three pixels adja- cent to the identified "sub-pixel" (i.e., the three blue pixels in Fig. b)). (4) Finally, we determine the units of mass each demander requires in the following way so that the dot annotation can be precisely represented using the set of the four demanders. Specifically, denote the 2D Euclidean distance between the centers of two adjacent pixels as g (which is also the pixel size), the coordinates of the GT dot annotation as (ædot Bdot). ), and the coordinates of the i-th demander as (I,, yi). where i E 11,2,3, 43. Then we construct the demanders D as:
GT dot annotation location
demander
sub-pixel contains the dot annotation
pixel adjacent the sub pixel containing the dot annotation
(b)
Figure 2: Illustration of (a) the naive formulation of the demanders D, and (b) formulation of the demanders D that involves the idea of sub-pixel resolution.
D= (d1, d2, d3, da), where d; (g |xot x.() x (g - -Vdoo y:l)
g²
where d, denotes the units of mass the i-th demander requires. By designing d, in this way, can achieve the following two properties. (1) Among the four demanders, via assigning more mass to the demanders nearer the dot annotation and less mass the demanders farther from the dot annotation, we can accurately derive the coordinates of the dot annotation as (I Ydot) = (d, 21 d2 222 + d3 X D3 + d4 x I4, y1 d2 1/2 dg ya da y4). Hence, the set of demanders D constructed in this way can represent the GT dot annotation accurately, regardless of how many times the resolution of the heatmap is lower than the resolution of the input image. (2) Besides, as di + d2 + d3 + da = 1, the total units of mass required by the demanders D are also constrained be the same the total units of mass stored by the suppliers S.
The cost function C. To measure the distribution difference between the H x W suppliers (S) con- structed from the predicted heatmap and the 4 demanders (D) constructed from the GT dot annotation via calculating their Earth Mover's Distance, we also need to formulate a cost function C' E R
NWW x4
Denote the coordinates of the n-th supplier as (Is.. Vs. ). where n {1. H x W}. and the coordinates of the m-th demander as (Id.,, Da.,), where m E {1, 2, 3, 4}. We here simply formulate Cn,m, the cost per unit transported from the n-th supplier to the m-th demander,
the 1.2 distance between the n-th supplier and the m-th demander, i.e., Cn.m = v (a.... - x x.,,)² )2 + (ya., - ys.,)² ). Using such a cost function, our method can optimize the model directly towards accurately localizing the dot annotation of the body joint via minimizing the distribution difference between the suppliers S and the demanders D.
5

---

3.3 Loss Function
Above we formulate the optimization of the heatmap prediction as distribution matching problem for single body joint. In this section, we introduce how we construct the loss function following such a formulation. Note that in 2D human pose estimation, we need locate multiple body joints. Therefore, to construct the loss function, we first construct the suppliers, the demanders, and the cost function for each body joint respectively. After that, we calculate the loss value corresponding to each body joint via measuring the distribution difference (calculating the Farth Mover's Distance) between its corresponding demanders and suppliers. Specifically, denote K the total number of body joints and La the loss term calculated w.r.t. the k-th body joint. We then formulate the loss function as:
K
Mateking
(S Lk. where Lx = E (St DF)
(5)
where St D= and O respectively denote the constructed suppliers, demanders, and cost function for the k-th body joint. The corresponding Earth Mover's Distance (*) is calculated using Eq.
3.4 Training and Testing
Our proposed method can be flexibly applied on various off-the-shelf 2D human pose estimation models. During training, we optimize the heatmap prediction via minimizing the loss function in Eq. 5 During testing, for each body joint, we first select a square of four adjacent pixels with the largest sum of pixel values from the predicted heatmap, and then normalize the sum of these four pixel values to The normalized pixel values of these four pixels can be seen as the units of mass each of them requires. Hence, in the same way as how we get the coordinates of the GT dot annotation from the demanders in Sec. 3.2 we can get the predicted coordinates of this joint. Specifically, denote the coordinates of the i-th selected pixel as (Ii, Mi). and the normalized pixel value of this pixel as dl, where E {1, 3, 4]. We can derive the predicted coordinates of this joint as (I pred Upred) (d1 x + d2 22 + d3 23 d1 x 21, d1 y1 d2 82 d3 x ys + d1 y1).
4 Analysis
(37 optimize the heatmap prediction via minimizing the pixel-wise MSE loss between the predicted heatmap and the Gaussian- smoothed heatmap. Below we do some analysis about this type of methods.
Most of the heatmap-based methods 229 20 33 26
35
[5]
17
We denote K the number of joints per input image T. Then we denote Haot = [H) .... HK }. HGau = [Hfdaa...., lK and Hpred = (Hfrred..... .... (K respectively the corresponding K GT dot-annotated heatmaps, GT Gaussian-smootte the input image T. We denote Daot {(7, Haoo)} the joint distribution of the input image and the corresponding K dot-annotated heatmaps, and DGau = ((7. H the joint distribu- tion of the input image and the corresponding K Gaussian-smoother
heatmaps, and predicted heatmaps of
heatmaps. Besides, we denote IMSE (a, b) ||a b1l the pixel-wise MSE loss, and & the model parameters where ( Haot)] as the expected risk calculated between the predicted heatmaps and the GT dot-annotated heatmaps. l s ( /), HGm))] as the expected risk calculated heatmaps.
((I)= = Hrred. After that, we denote R(Daot, o, (MSE) = EJ (.JHaaa)DDDu
and R(DGauw Φb, JASSE) = Ed Ha..)>DP... [ between the predicted heatmaps and the Gaussian-smoothhe
Theorem 1. The relationship between R(Daot, 0, (MS)) and R(D 0, (SE))) can be written as
R(Daot Φ, (.EE)) =R(DGaus (.MMSE) + 2 x R. H..Henn)DGGn..... (( (Hpeed: HGau)]
(6)
[ ((HHHao)) (Hpred, Haot!
C
where C = L[ H HGan
Hato [ is a constant.
The proof of Theorem is provided in the supplementary. As shown in Theorem I when we minimize the wise loss between Hpred and HGau- while R(DGaus Φ, (MSE) de- creases, the second term 2 E(I, (Hpred. HGau)) can increase and the third term 2 x E H...Ha)))Daa... [ (Hpred, Haot)] cannot be guaranteed to increase or decrease. Because of this,

---

Table 1: The improvement of AP on COCO validation set when our proposed method is applied to
various baselines.
Method
Venue
Backbone
Input
AP
AP
APIS
APM AP
Hourglass
BCCV 2016
8-Stuge Hourglass
256 192
66.9
CPN
3
CVPR 2018
ResNet-50
256 x 192
69.4
CPN 1
CVPR 2018
ResNet-50
384 x 288
71.6
DarkPose 39
CVPR 2020
HRNct-W32
x 288
76.6
82.8
72.7
83.9
81.5
UDP 10
CVPR 2020
MIRNet-W32.
384 x 288
77.8
91.7
84.5
74.2
84.3
82.4
UDP 10
CVPR 2020
HRNet-W48
384 x 288
77.8
92.0
84.3
74.2
84.5
82.5
TokenPose
ICCV 2021
TokenPose-
256 x 192
75.8
90.3
82.5
72.3
82.7
80.9
Removing Bias
ICCV 2021
ResNet-112
384 288
74.4
Removing Bias 7
ICCV 2021
HRNei-W32
256 x 192
75.8
Simple Baseline
ECCV 2018
ResNet-152
354 x 288
75.0
90.8
82.1
67.8
78.3
80.0
Ours
ResNet-152
384 x 288
76.7(11.7))
83.6
69.7
80.0
$1.3
22
EVPR2001)
HRNct-W32
381 288
76.7
83.6
73.2
83.2
81.6
Ours
HRNet- W32
384 x 288
78.2(11.55
92.2
84.5
74.3
84.7
82.5
HRNeet 26
CVPR 2019
HRNer W48
384 x 288
77.1
91.8
83.8
73.5
83.5
81.8
+ Ours
HRNet- W48
384 x 288
78.8(11.7)
92.5
85.1
75.0
85.3
83.1
HRFormer
NIPS 2021
HRFommer-Ban
384 x 288
92.2
84.8
74.3
84.6
82.6
Ours
HiRFormer-Bare
384 x 288
78.9(10.9)
92.6
85.4
75.3
85.3
83.3
R(Daot: o, (SEE) cannot be guaranteed to decrease, and the model performance of body joint local- ization may not be consistently improved during such optimization heatmap prediction. A more intuitive analysis is follows. In the optimization process of most of the heatmap-basee methods, since all the H W pixels from the predicted heatmap contribute to the overall pixel-wise loss, learning fit the other pixels better instead of the pixel representing the dot annotation can also lead to a smaller overall loss, as shown in Fig. [ Hence, when minimizing the overall pixel-wise loss, the model performance of body joint localization cannot be guaranteed to improve consistently. Besides, during training. the GT Gaussian-smoother heatmap is constructed via using the Gaussian blob. Therefore, during testing, the predicted heatmap often has relatively large area of pixels with large values around the dot annotation, as shown in Fig. 3 which can make it difficult to accurately locate the body joint.
Differently, our method optimizes the heatmap prediction from a novel perspective via minimizing the difference between the distribution constructed from the predicted heatmap (i.e., the suppliers S), and the distribution constructed from the dot annotation (i.e., the demanders D). During training, since formulate the cost function between each pair of supplier and demander their L. distance, minimizing the distribution difference based on such a cost function can aggregate the pixel values in the predicted heatmap towards the dot annotation. Hence, our method can help achieve a more consistent model performance improvement when minimizing the distribution difference. Besides, during testing, equipped with such mechanism of aggregating the predicted pixel values towards the dot annotation, our method can achieve more compact body joint localization as shown in Fig.
5 Experiments
To evaluate the effectiveness of our proposed method, we conduct experiments on the COCO dataset [16] and the MPII Human Pose dataset [T]. Besides, to the generality of our method, we apply to various backbones, c.g., ResNet B3]. HRNet [26], and HRFormer [37].
5.1 COCO Keypoint Detection
Dataset & evaluation metric. The COCO dataset [16] contains more than 200k images and 250k person instances, which are annotated with 17 body joints. This dataset has three subsets including COCO training set, COCO validation set, and COCO test-dev set, which have 57k, 5k and 20k images, respectively. We conduct experiments on this dataset via first training the model on the train2017 set, and then evaluating the model on the val2017 set and test-dev2017 set. Following standard average precision (AP) calculated based Object Keypoint Similarity
33 26 37) (OKS) to evaluate model performance.
Implementation details. We apply our method to various baselines including Simple Baseline RK
HRNet 26), and HRForrer (37], with their respective backbones including ResNet-152. HRNet- W32, HRNet- W48, and HRFormer-Bars For these baselines, we follow their original lcarning and
7

---

Table 2: The improvement of AP on COCO test-dev set when our proposed method is applied to
various baselines.
Method
Venue
Backbone
Input
AP
AP
AP
AP
AP
G-RMI
CVPR 2017
ResNet-
353 26r
64.9
85.5
71.3
62.3
70.0
69.7
Mask-RCNN 9
ICCV 2017
ReaNet-50-FPN
63.1
68.7
71.4
6MPE
ICCV 2017
PyraNetB66
320 256
72.3
89.2
79.1
68.0
78.6
CFN 11
ICCV 2017
72.6
86.1
69.7
78.3
64.1
CPN
CVPR 2018
ResNet-Inception ResNet-Inception
384 288
72.1
91.4
80.0
68.7
77.2
78.5
CPN(enscmble) R
CVPR 2018
384 288
73.0
91.7
80.9
69.5
78.1
79.0
Integral Regression
ResNet-100
Pusefix [8
CVPR 2019
ResNet 152
384 288
73.6
90.8
81.0
70.3
79.8
79.0
DarkPose
81.1
[DP[10]
CVPR 2020
HRNet W48
384 288
76.5
92.7
84.0
73.0
82.4
81.6
TokenPose
ICCV
TokenPose-1
80.8
Removing Blas
ICCV 2021
HRNet- W48
384 288
76.1
81.0
Simple Kaseling
ECCV 2018
ResNer: 132
384 283
73X8
91.7
81.2
70.3
80.D
79.1
Ours
ResNet-155
384 258
75.3(11.55
92.6
83.1
71.7
81.1
80.3
HRRNet 26
CVPR 2019
HRNe: W32
384 288
74.9
92.5
82.8
71.3
80.9
80.1
Ours
TRNNs-W332
384
76.7(11.8)
92.6
HRNNet26
CVPR 2019
HRNet W48
384 288
75.5
92.5
83.3
71.9
81.5
80.5
+ Ours
HRNes- W48
384 288
77.2(11.7)
93.0
84.4
73.4
83.3
82.0
NIPS 2021
81,2
Ours
HRForeer Base
384 288
77.2(1100)
93.1
84.7
73.8
83.0
82.1
optimization configurations for model training. To calculate the Earth Mover's Distance using the Sinkhorn algorithm, we set the Sinkhorn entropic regularization parameter to and the number of Sinkhorn iterations to 1000 in our experiments.
Results. Tab. [ and Tab. 2 we report results on the COCO validation and test-dev sets. We observe that after applying our method various baselines, a significant performance enhancement achieved, which shows effectiveness of our proposed method. Moreover, we compare our method with other state-of-the-ar 2D human pose estimation methods. Our method achieves superior performance compared to these methods, further demonstrating the effectiveness our method.
5.2 MPII Human Pose Estimation
Dataset & evaluation metric. The MPII dataset [] contains around 25K images and more than 40k person instances, which are annotated with 16 body joints. We adopt the standard train/val split in [I] to build the MPII training set and validation set, and conduct all the experiments on this dataset via first training the model on the MPII training
and then evaluating it the MPII validation set. Following (66), we use the head-normalized probability of correct keypoint (PCKh) score as the evaluation metric this dataset and report the PCKh@0. score.
Implementation details. On the MPII dataset, we also apply our method to various methods as our baselines, including Simple Baseline 333 and HRNet (26). with their respective backbones including ResNet-152, 152, HRNet-W- W32, and HRNeI-W4S3 We follow the original learning and optimization configurations for model training for both Simple Baseline (33 and HRNet et [26]. Besides, same as the experiments on the COCO dataset, we also set the Sinkhorn entropic regularization parameter to I and the number of Sinkhorn iterations 1000 in experiments the MPII dataset.
we report the results on the MPII validation set. As shown, applying our proposed method on various baselines results in consistent performance improvement, which demonstrates the effectiveness of our proposed method.
Results on the MPII validation set. Tab. 3
5.3
Ablation Studies
We conduct ablation studies on the COCO validation set via applying our proposed method on HRNet-W48 (26).
Impact of involving the idea of sub-pixels in formulating D. In our proposed method, we formulate the demanders D by involving the idea of sub-pixel resolution. To investigate the impact of such way, we compare our proposed method (sub-pixel demanders
formulating the demanders formulation) with a variant (naive demanders formulation). This variant still formulates the suppliers and the cost function C in the same way, but formulates the demanders /) naively as single demander at the center of the pixel containing the dot annotation, shown in Fig. a))
8

---

Table 3: The improvement of AP on MPII validation set when our proposed method is applied to
various baselines.
Method
Venue
Backhone
Ingut size
Mein
Hest
Sho
Fib
Win
Hip
Kne
Ank
Integral
Regression223
UDP[00]
CVPR 2020
HRNcI-W32
256 256
943.4
97.4
96.0
91.0
86.5
89.1
86.6
83.3
DarkPose
59
CVPR 2020
HRNet-W32
256 256
90.6
97.2
95.9
91.2
86.7
89.7
86.7
84.0
TokenPose
TokenPPse 15
ICCV 2021
TokenPose-1,113
206 256
90.1
97.2
95.8
90.7
85.
89.2
86.2
82.3
TokenPos:
ICCV 2021
TokenPose--LDD
256 256
90.2
97.1
95.9
90.4
86.0
89.3
87.1
82.5
Removing
ICCV
ResNet-152
Removing Bins
ICCV 2021
HRNei-W32
256 256
90.6
Simple Baseline
ICCV 2018
ResNer-152
256
97.0
95.9
90.0
85.0
89.2
85.3
81.3
ResNes-152
90.311077
97.0
89.2
83.1
2RNNi(26
CVPR 2019
HRNet. W:32
256 256
90.4
97.1
95.9
90.7
86.
89.4
86.9
83.2
90.9(1955
97.3
87.4
84.1
HRNe:(25
CVRR 2019
HRNer-War
256 x 256
90.3
96.9
96.0
90.9
86.2
89.6
87.1
83.5
+ Ours
HRNet-W-48
256 256
90.9X19.4)
97.1
96.3
91.2
87.0
90.2
87.5
84,2
Table 4: Evaluation the effectiveness of for- Table 5: Evaluation on the number of Sinkhorn mulating the demanders D involving the idea of iterations. sub-pixel resolution.
Method
AP
AP"
AP
AP"
AP
AR
AP
Apm
Biseline(HRNae W48)
77.1
91.8
83.8
73.5
83.5
81.8
HisclleeHHRNNe. Way)
demanders formolation
779
1000 Sinkhorn iterations
Sinkhorn Herations
78.8
92.5
85.1
75.0
85.3
83.1
demanders
78.8
1500 Sinkhorn lterations
78.7
92.4
85.2
74.8
85.4
83.0
As shown in Tab. our proposed method consistently outperforms this variant, which shows effectiveness of our sub-pixel demanders formulation.
Impact of the number of Sinkhorn iter-
ations. For measuring the Earth Mover's Distance utilizing the Sinkhorn algorithm, we need to set the number of Sinkhorn iterations, which we set 1000 in our experiments. We evaluate other choices of the number of Sinkhorn iterations in Tab. 5 As shown, all variants outperform the baseline method, and after the number of Sinkhorn iterations becomes larger than 1000, the model performance becomes stabi- lized. Hence, set the number of Sinkhorn iterations be 1000 in all our experiments,
Body
shoulder
Baseline [HRNett-148)] recul:
Ours
result:
Figure 3: Qualitative results of our method formulating the optimization of the heatmap prediction as a distri- bution matching problem and the baseline method (26) using the maanssiin-smoothen heatmap as the optimiza- tion objective. As shown, our method localizes body joints much more compactly.
Qualitative results. Some qualitative re- sults are shown Fig. 3 As shown, via formulating the cost function as the L2 dis- tance, our proposed method can aggregate the pixel values in the predicted heatmap to- wards the dot annotation, and thus localize body joints much more compactly than the baseline method (26 relying on the Gaussian-smooote smoothed heatmap. This demonstrates that our method can effectively optimize the model the direction of accurately localizing the body joint.
Conclusion
In this paper, from novel perspective, we formulate the optimization of the heatmap prediction as distribution matching problem between the predicted heatmap and the annotation via calculating their Earth Mover's Distance. Our proposed method is simple yet effective, and can be easily applied to various 2D human pose estimation models. Our method achieves superior performance on the COCO dataset and the MPII dataset.

---

Acknowledgments and Disclosure of Funding
This project is supported by the Ministry of Education, Singapore, under the SUTD Kickstarter Initiative Project (SKI 2021 02_06), National Research Foundation Singapore under its Al Singapore Programme (AISG-100E-2000--665)) and SUTD Startup Research Grant.
References
[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. Proceedings the IEEE Conference on computer Vision and Puttern Recognition, pages 3686-3693, 2014.
[2] Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4733-4742, 2016.
[3] Yilun Chen, Zhicheng Wang. Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and pallern recognition, pages 7103-7112, 2018.
[4] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5386-5395, 2020.
[5] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[6] Hao-Shu Fang, Shuqin Xie, Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In Proceedings of the IEEE international conference computer vision, pages 2334 2343, 2017.
[7] Kerui Gu, Linlin Yang, and Angela Yao. Removing the bias of integral pose regression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 10667- 11076, 2021.
[8] Kerui Gu, Linlin Yang. and Angela Yao. Dive deeper into integral pose regression. In International Conference on Learning Representations, 2022.
[9] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[10] Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang. The devil is in the details: Delving into unbiased data processing for human pose estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5700-5709, 2020.
(11) Shaoli Huang, Mingming Gong, and Dacheng Tao. A coarse-fine fine network for keypoint lo- calization. In Proceedings of the IEEE international conference on computer vision, pages 3028 2228-303,, 2017.
[12] Siamak Khademi, Ahmad Darudi, and Zahra Abbasi. A sub pixel resolution method. arXiv preprint 1rXiv:1211.2222 2012.
[13] Jiefeng Li, Siyuan Bian, Ailing Zeng. Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log-likellng likeliood estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11025-11034 2021.
[14] Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun. Rethinking on multi-stage networks for human pose estimation. arXiv preprint arXiv: 1XX1:0901000044. 2019.
[15] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11313-11322, 2021.
10

---

[16] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[17] Zhengxiong Luo, Zhicheng Wang, Yan Huang, Liang Wang, Tieniu Tan, and Erjin Zhou. Rethinking the heatmap regression for bottom-up human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13264-13273,
2021.
[18] Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Posefix: Model-agnostic general human pose refinement network. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1773-7781, 2019.
[19] Amit Moryossef, loannis Tsochantaridis, Roee Aharoni, Sarah Ebling, and Srini Narayanan. Real-time sign language detection using human pose estimation. In European Conference on Computer Vision, pages 237-248. Springer, 2020.
[20] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pages 483-499, Springer, 2016.
[21] Aiden Nibali, Zhen He, Stuart Morgan, and Luke Prendergast, Numerical coordinate regression with convolutional neural networks. arXiv preprint arXiv: arXiv:/801.07230 1801.07372, 2018.
[22] George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and Kevin Murphy. Towards accurate multi-person pose estimation the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4903-
4911, 2017.
[23] Hanyu Peng, Mingming Sun, and Ping Li. Optimal transport for long-tailed, recognition with learnable cost matrix. In International Conference Learning Representations, 2022.
[24] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance metric for image retrieval. International journal of computer vision, 40(2):99-122) 121, 2000.
[25] Samuel Schulter, Paul Vernaza, Wongun Choi, and Manmohan Chandraker. Deep network flow for multi-object tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6951 2017.
[26] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5693-5703 2019.
[27] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In Proceedings of the European conference on computer vision (ECCV), pages 529-545, 2018.
[28] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-22 18-2826, 2016.
[29] Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of convolutional network and graphical model for human pose estimation. Advances in neural information processing systems, 27, 2014.
[30] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1653-1660, 2014.
[31] Boyu Wang, Huidong Liu, Dimitris Samaras, and Minh Hoai Nguyen. Distribution matching for crowd counting. Advances in Neural Information Processing Systems, 33:1595-1607, 2020.
[32] Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, and Stephen Lin. Point-set anchors for object detection, instance segmentation and pose estimation. European Conference on Computer Vision, pages 527-544. Springer, 2020.

---

[33] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pages 466 481, 2018.
[34] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-basee action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.
[35] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Towards explainable human pose estimation by transformer. arXiv e-prints, pages arXiv-2012, 2020.
[36] Wei Yang. Shuang Li, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang. Learning feature pyramids for human pose estimation. In proceedings of the IEEE international conference on computer vision, pages 1-2900 2017.
[37] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense prediction. 2021.
[38] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's distance and structured classifiers. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 12203-12213, 2020.
[39] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution- aware coordinate representation for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7093-7102, 2020.
[40] Yizhai Zhang, Kuo Chen, Jingang Yi, and Liu Liu. Pose estimation in physical human-machine interactions with application to bicycle riding. In 2014 IEEE/RS. International Conference on IEEE, 2014.
Intelligent Robots and Systems, pages
[41] Lu Zhou, Yingying Chen, Yunze Gao, Jinqiao Wang, and Hanqing Lu. Occlusion-aware siamese network for human pose estimation. European Conference on Computer Vision, pages 396-412. Springer, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
12

---

(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "swkU4QS2u1F", "reviewer": "Reviewer_wVS3", "summary": "This paper presents a method to estimate 2d locations of joints for human pose estimation task. Authors present a technique loss function to reduce the trade off between dot notation and heatwap notation to estimate the joint locations. For this purpose authors propose using earth mover's distance as the loss function. The paper proposes the construction of supplier and demander values as well as the cost function associated. The final minimization is obtained by using the sink horn's iteration method. Authors also present an analysis of why minimize the Gaussian heatmap with l2 loss is not optimal. Experiments on mpii and coco datasets are provided", "questions": "Point (2) from the weakness section. If the objective is to decrease the area of localization it is important to observe a naive approach first.\nSeveral, paper have shown that to work for the cash of facial landmark localization.\n\nPoint (3) please provide failure cases in the experiment section. The paper does not talk about any limitations of the proposed approach\n.\n\nPoint (4) Earth movers distance can be difficult to train from scratch. This limits the paper to only be applied to already existing pre trained networks.", "limitations": "The authors have not talked about any limitation of the paper. Conceptual or societal\n\nSuggestions are already included is the previous sections", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1. The paper is well rewritten and easy to understand with almost no typos \n2. Although not very complicated paper the paper present an easy way to improve the quality of joint localization\n3. The paper presents an analysis demonstrating the limitation of the methods.\n\nWeakness\n1. Several other paper have shown the limitation of heatmap regression and proposed solution to tackle those limitation. For ex, integral pose regression shows that it achieves similar numbers on mpii dataset, which the authors have not included in the table, however the paper does show companion with pose regression for coco dataset. Similarly (7) also shows similar performance on mpii dataset.\n\n2. It will be interesting to observe how does this method compare to a heatmap regression based method where a multistep straining is applied where with each step of training the standard deviation is decreased\n\n3. It will be good to see the failure cases of this method.\n\n4. were the networks in the experiment initialized from already available pretrained weights? If that in the case it will be interesting to observe how the networks train when trained from scratch. Do the networks converge withEMD when trained from scratch. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kKYD44TMxJJ", "reviewer": "Reviewer_aExd", "summary": "1. The paper analyzes that the commonly used minimizing pixel-wise loss cannot guarantee that the performance of pose estimation increase.\n2. The paper proposes using Earth Mover’s Distance to match the distributions between GT dot heatmap and predicted heatmap. The experiment shows that this loss can improve the performance by 0.9-1.8% AP.\n", "questions": "1. Why baseline dot-annotated heatmap (MSE loss) doesn't work, but proposed method does. Is it because the MSE loss (gradient) is too sparse? What role does Heatmap normalization play here? The authors may be able to provide more thorough analysis on this problem.\n\n2. The authors can compare in more detail with UDP and Darkpose, which fix some issues in native Gaussian Heatmap.\n\n3. Compared with baseline, does the running time increase? This helps to assess the practicality of the method.\n", "limitations": "This paper does not provide limitations and potential negative societal impact. Please refer to the Questions section on limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The heatmap is a basic and important tool in human pose estimation. The paper analyzes the problem of commonly used pixel-wise heatmap loss, and formulate the optimization of the heatmap prediction as a distribution matching problem by Earth Mover’s Distance. It is novel and technically sound.\n\n2. It seems that the proposed loss is model-agnostic that can be applied in most pose estimator and improve the performance.\n\n**Weaknesses:**\n\nThe paper lacks the complexity analysis of the proposed method, such as the training and inference time.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bCcXNkALcem", "reviewer": "Reviewer_ggaN", "summary": "This paper proposes using EMD to measure the difference between the GT and the predicted heatmaps. Compared with L2 distance, EMD avoids the choice of standard deviation and is more consistent with the final objective of pose estimation. The proposed loss function can largely improve the performance of pose-estimation methods.", "questions": "See the questions above.", "limitations": "The idea is elegant and effective. My only concern is about the training time. However, given its marvelous performance, a longer training time is also acceptable to some degree. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well written and easy to follow. The proposed method is simple yet can largely improve the performance of pose-estimation methods.\n\nQuestions:\n1. According to Line144, the authors seem to use Sinkhorn algorithm to solve the EMD. Is this process differentiable? Could you please supplement more details about how the gradients are passed backward? \n\n2. According to Eq 5, the EMD is respectively calculated for each keypoint.  How long does it take to solve the EMD for one keypoint? And it seems that the calculation of EMD also cannot be paralleled among a mini-batch. Will the proposed method largely extend the training time?\n\n3. It seems that EMD can also be applied to the heatmaps with multiple keypoints, which means that the proposed functions can also be applied for muli-human pose estimation. Have the author tried this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "VDl_iC17e32", "reviewer": "Reviewer_1LPW", "summary": "Pose estimation performance may not be consistently improved when optimizing the heatmap prediction by minimizing the mean-squared error(MSE). The paper proposes to utilize the Sinkhorn Distance as the optimizing target.", "questions": "Please refer to Weaknesses", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The paper points out and analyzes the misalignment between the training loss and the final performance.\n1. Sinkhorn Distance is proposed to replace the MSE loss to achieve a more consistent performance improvement during training.\n1. The proposed method is reported to have notable performance improvements on COCO and MPII.\n\n### Weaknesses\n\n#### Novelty & Contribution\n\n1. Formulating the heatmap optimization as a distribution matching problem is not a novel perspective. Previous methods represent the GT coordinates in either Categorial or Gaussian distribution and optimize the predictions towards the GT distribution by minimizing MSE.\n1. Proposing to replace MSE with Sinkhorn Distance does not have enough contribution.\n\n#### Presentation\n\n1. There are inconsistent presentations. In my perspective, the limitation to address lies in the pixel-wise loss (e.g. MSE), which is claimed in the Introduction(line45-52).\n   + However, in Abstract(line4-7), it's claimed that using the Gaussian-smoothed heatmap as the optimization objective is the limitation of previous methods\n   + The paper's target should not be 'bypassing the step of constructing the GT heatmap', but to alleviate or solve the limitation of loss. \n   + Missing analysis on why the proposed method can alleviate or solve the limitation of loss. \n2. It's misleading to claim bypassing the step of constructing the GT heatmap. The demander $D^k$ is just another type of GT heatmap and still needs the construction step. The naïve demander formulation is the same as the dot-annotated heatmap.\n\n#### Experiments\n\n1. Missing important reports and comparisons on the training time between the baseline and the proposed method. As the Sinkhorn Distance takes 1000 iterations and each iteration has calculations on $(H_{hm} * W_{hm}) \\times (H_{hm} * W_{hm})$ dim matrixes, it's presumed to be time-consuming. \n2. Missing comparisons between the MSE loss and the proposed loss with the same type of target heatmap", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept