Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposed a new method/mechanism for training the soft-argmax network. They found the conventional training approach(Eq. 4) has training difficulty, and they proposed a sampling-based training approach to alleviating the training issue. They evaluate the proposed method on various tasks and benchmarks and illustrate promising results compared to conventional training approach and its variants.
main_review: Pros: Overall, this work is well written and organized. The motivation is reasonable and the idea is technically soundable. The comparisons with baseline methods on various tasks and benchmarks show consistent improvement. 
Cons:
- In L106-107, the author claims the conventional method suffered the training difficulty since of the high variance. I think this claim is not self-evident and the related experimental result is needed. 
- While this work has been validated in many tasks with the baseline method, I am still wondering how can the proposed method improve the state-of-the-art in at least one task, such as the COCO keypoint. I think it always is good to see a new method that could push the edge forward.


limitations_and_societal_impact: No
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel sampling-argmax for regression based approach for localizing the target positions, as an alternative module for soft-argmax which is widely used for regression based localization. The authors report extensive experiments on several applications to demonstrate its effectiveness.
main_review: The paper is well written and easy to follow. The idea is also simple and interesting. But I have some concerns as follows.
1. Make the motivation clearer. I know for error of expectation, to minimize the distance to be 0, the probability masses could be redistributed to different locations, or being allocated to the optimal position. But for expectation of errors, since all distances are positive, the optimal solution should be allocating all mass to the optimal position. I think the authors could elaborate the intuition to make the paper more readable.
2. The authors could show the prediction distribution of sampling-argmax in experiments to verify the motivation that the proposed approach encourages unimodality.
3. The experiments are conducted on several applications, keypoint estimation, cloud point estimation, etc. The largest performance gain comes in COCO keypoint benchmark where the proposed sampling-argmax improves SimplePose by 5.3% mAP. However, the SimplePose is a 2018 work and current SOTA could reaches ~80% mAP. Is the proposed approach able to improve some new approaches, e.g. PoseFix (CVPR 2019), EvoPose2D (NeurIPS 2020). At least the authors should cite them. Moreover, on some other datasets, such as MTFL, KeypointNet, OCT, and Human3.6M, the improvement is quite marginal and less than 1%. The authors might try to see if the proposed approach could improve some newer approaches.
4. Typos and other minor issues:
- For Eq. (3), it might be better to use \bold{y} to represent a matrix/vector
- In line 2 of  Eq. (5), why the LHS could be equal to RHS with a logarithm operator?
limitations_and_societal_impact: This paper does not address the limitations and societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes sampling-argmax, a method to do detection-based localization that imposes implicit constraints on the shape of the probability maps. The idea is to use a continuous mixture distribution, with learnable mixture weights, and fixed continuous base distributions (uniform, triangular, or gaussian). The mixure weights are learned through Gumbel-Softmax. Experiments on a variety of localization tasks show that in many cases, there exists at least one version of sampling-argmax that is better than the baseline methods.
main_review: To the best of my knowledge, using a continuous mixture distribution to model the probability maps for localization has not been done before, and the experimental results are convincing that this idea is better than soft-argmax and some other baselines. The paper is well written and easy to follow.

Some comments:
- I wonder how a fully continuous distribution would compare against sampling-argmax. The learnable parameters would be the parameters for the continuous distribution (instead of the weights), so as long as the distribution is reparameterizable, obtaining unbiased gradients wouldn’t be an issue. At test time, you would just need to find the partition that contains the continuous sample. 
- Consider moving lines 206-213 to related work, because it breaks the flow in its current location. 
- Add “<higher/lower> is better” to captions for tables for readability.


Update:
Considering all the other reviews and the authors' response, I decided to increase my score to a 7.

limitations_and_societal_impact: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper presents the sampling-argmax operation to construct a probability map for regressing coordinates with deep neural networks. The proposed operation gives higher probabilities to positions close to the ground truth. As a result, the proposed operation as part of the loss function minimizes the expectation of the error. In addition, non-differentiable and differentiable schemes are presented. The approach is evaluated for 2D/3D human pose estimation, object keypoint estimation from point clouds, facial localization and retina segmentation. The proposed approach shows promising results compared to the prior work.
main_review: Paper strengths: 

+ The formulation of Eq. (4) is a novel improvement compared to the soft-argmax.

+ The paper is well-written, the related work is complete and the method easy to follow.

+ The proposed sampling approach is well-justified with the comparison of Eq. 5 to the REINFORCE convergence problems. Moreover, the Gumbel-softmax idea makes the sampling process differentiable. 

+ The results support the functionality of the proposed operation.

+ In total, the paper makes a solid contribution with the sampling-argmax and differentiable sampling. The approach can be employed for different regression problems. 


Paper weaknesses and Improvements: 

- The results are overall convincing. Nevertheless, there are some cases where the proposed idea does not make a big difference compared to soft-argmax, e.g. Table 4. These cases should have been discussed. In total, more discussion would be helpful to the interpretation of the results in the experimental part.

- Result presentation: The tables contain abbreviations that are not explained in the caption. It's difficult to follow the results of all tables. 


Overall: 

The paper presents a novel approach that works well for a variety of localization problems. The proposed approach can effectively replace soft-argmax. This is a clear message out of the proposed work. 


After rebuttal comments:

The authors sufficiently addressed the questions of all reviews. In two reviews, there were points some concerns which are answered in the rebuttal. The final version should integrate all rebuttal comments. Overall, the paper has enough merit to be accepted.
limitations_and_societal_impact: No limitations and potential negative societal impact.
review_title: null

Meta-review: This work proposes a new technique for differentiable sampling, allowing to encode shape constraints about the distribution. One of the motivations is to improve training performance compared to the usual Gumbel softmax. 

The reviewers are unanimous that this is a good submission, and that it should be accepted.
Decision: Accept (Poster)