Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method to estimate 2d locations of joints for human pose estimation task. Authors present a technique loss function to reduce the trade off between dot notation and heatwap notation to estimate the joint locations. For this purpose authors propose using earth mover's distance as the loss function. The paper proposes the construction of supplier and demander values as well as the cost function associated. The final minimization is obtained by using the sink horn's iteration method. Authors also present an analysis of why minimize the Gaussian heatmap with l2 loss is not optimal. Experiments on mpii and coco datasets are provided
strengths_and_weaknesses: Strengths 
1. The paper is well rewritten and easy to understand with almost no typos 
2. Although not very complicated paper the paper present an easy way to improve the quality of joint localization
3. The paper presents an analysis demonstrating the limitation of the methods.

Weakness
1. Several other paper have shown the limitation of heatmap regression and proposed solution to tackle those limitation. For ex, integral pose regression shows that it achieves similar numbers on mpii dataset, which the authors have not included in the table, however the paper does show companion with pose regression for coco dataset. Similarly (7) also shows similar performance on mpii dataset.

2. It will be interesting to observe how does this method compare to a heatmap regression based method where a multistep straining is applied where with each step of training the standard deviation is decreased

3. It will be good to see the failure cases of this method.

4. were the networks in the experiment initialized from already available pretrained weights? If that in the case it will be interesting to observe how the networks train when trained from scratch. Do the networks converge withEMD when trained from scratch. 


questions: Point (2) from the weakness section. If the objective is to decrease the area of localization it is important to observe a naive approach first.
Several, paper have shown that to work for the cash of facial landmark localization.

Point (3) please provide failure cases in the experiment section. The paper does not talk about any limitations of the proposed approach
.

Point (4) Earth movers distance can be difficult to train from scratch. This limits the paper to only be applied to already existing pre trained networks.
limitations: The authors have not talked about any limitation of the paper. Conceptual or societal

Suggestions are already included is the previous sections
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Pose estimation performance may not be consistently improved when optimizing the heatmap prediction by minimizing the mean-squared error(MSE). The paper proposes to utilize the Sinkhorn Distance as the optimizing target.
strengths_and_weaknesses: ### Strengths

1. The paper points out and analyzes the misalignment between the training loss and the final performance.
1. Sinkhorn Distance is proposed to replace the MSE loss to achieve a more consistent performance improvement during training.
1. The proposed method is reported to have notable performance improvements on COCO and MPII.

### Weaknesses

#### Novelty & Contribution

1. Formulating the heatmap optimization as a distribution matching problem is not a novel perspective. Previous methods represent the GT coordinates in either Categorial or Gaussian distribution and optimize the predictions towards the GT distribution by minimizing MSE.
1. Proposing to replace MSE with Sinkhorn Distance does not have enough contribution.

#### Presentation

1. There are inconsistent presentations. In my perspective, the limitation to address lies in the pixel-wise loss (e.g. MSE), which is claimed in the Introduction(line45-52).
   + However, in Abstract(line4-7), it's claimed that using the Gaussian-smoothed heatmap as the optimization objective is the limitation of previous methods
   + The paper's target should not be 'bypassing the step of constructing the GT heatmap', but to alleviate or solve the limitation of loss. 
   + Missing analysis on why the proposed method can alleviate or solve the limitation of loss. 
2. It's misleading to claim bypassing the step of constructing the GT heatmap. The demander $D^k$ is just another type of GT heatmap and still needs the construction step. The naïve demander formulation is the same as the dot-annotated heatmap.

#### Experiments

1. Missing important reports and comparisons on the training time between the baseline and the proposed method. As the Sinkhorn Distance takes 1000 iterations and each iteration has calculations on $(H_{hm} * W_{hm}) \times (H_{hm} * W_{hm})$ dim matrixes, it's presumed to be time-consuming. 
2. Missing comparisons between the MSE loss and the proposed loss with the same type of target heatmap
questions: Please refer to Weaknesses
limitations: None
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 1. The paper analyzes that the commonly used minimizing pixel-wise loss cannot guarantee that the performance of pose estimation increase.
2. The paper proposes using Earth Mover’s Distance to match the distributions between GT dot heatmap and predicted heatmap. The experiment shows that this loss can improve the performance by 0.9-1.8% AP.

strengths_and_weaknesses: **Strengths:**

1. The heatmap is a basic and important tool in human pose estimation. The paper analyzes the problem of commonly used pixel-wise heatmap loss, and formulate the optimization of the heatmap prediction as a distribution matching problem by Earth Mover’s Distance. It is novel and technically sound.

2. It seems that the proposed loss is model-agnostic that can be applied in most pose estimator and improve the performance.

**Weaknesses:**

The paper lacks the complexity analysis of the proposed method, such as the training and inference time.

questions: 1. Why baseline dot-annotated heatmap (MSE loss) doesn't work, but proposed method does. Is it because the MSE loss (gradient) is too sparse? What role does Heatmap normalization play here? The authors may be able to provide more thorough analysis on this problem.

2. The authors can compare in more detail with UDP and Darkpose, which fix some issues in native Gaussian Heatmap.

3. Compared with baseline, does the running time increase? This helps to assess the practicality of the method.

limitations: This paper does not provide limitations and potential negative societal impact. Please refer to the Questions section on limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes using EMD to measure the difference between the GT and the predicted heatmaps. Compared with L2 distance, EMD avoids the choice of standard deviation and is more consistent with the final objective of pose estimation. The proposed loss function can largely improve the performance of pose-estimation methods.
strengths_and_weaknesses: Strengths:
The paper is well written and easy to follow. The proposed method is simple yet can largely improve the performance of pose-estimation methods.

Questions:
1. According to Line144, the authors seem to use Sinkhorn algorithm to solve the EMD. Is this process differentiable? Could you please supplement more details about how the gradients are passed backward? 

2. According to Eq 5, the EMD is respectively calculated for each keypoint.  How long does it take to solve the EMD for one keypoint? And it seems that the calculation of EMD also cannot be paralleled among a mini-batch. Will the proposed method largely extend the training time?

3. It seems that EMD can also be applied to the heatmaps with multiple keypoints, which means that the proposed functions can also be applied for muli-human pose estimation. Have the author tried this?
questions: See the questions above.
limitations: The idea is elegant and effective. My only concern is about the training time. However, given its marvelous performance, a longer training time is also acceptable to some degree.
review_title: null

Meta-review: metareview: This paper proposes to use earth mover distance to measure the loss function between a predicted heatmap and ground truth heatmap. It initially received mixed reviews. After rebuttal and discussion, all reviewers converged to acceptance of the paper. Reviewers believe this paper is novel and achieved significant practical performance across several models. AC follows the consensus and recommends acceptance of the paper.
confidence: Certain
Decision: Accept