Localization with Sampling-Argmax
Authors: Jiefeng Li
Tong Chen
Ruiqi Shi
Yujing Lou
Yong-Lu Li
Cewu Lu
Venue: NeurIPS 2021 Poster (2021)
Keywords: soft-argmax
probability map
localization

Abstract: Soft-argmax operation is commonly adopted in detection-based methods to localize the target position in a differentiable manner. However, training the neural network with soft-argmax makes the shape of the probability map unconstrained. Consequently, the model lacks pixel-wise supervision through the map during training, leading to performance degradation. In this work, we propose sampling-argmax, a differentiable training method that imposes implicit constraints to the shape of the probability map by minimizing the expectation of the localization error. To approximate the expectation, we introduce a continuous formulation of the output distribution and develop a differentiable sampling process. The expectation can be approximated by calculating the average error of all samples drawn from the output distribution. We show that sampling-argmax can seamlessly replace the conventional soft-argmax operation on various localization tasks. Comprehensive experiments demonstrate the effectiveness and flexibility of the proposed method. Code is available at https://github.com/Jeff-sjtu/sampling-argmax

Localization with Sampling-Argmax
Jiefeng Li
Tong Chen
Ruiqi Shi
Yujing Lou Yong-Lu Li Cewu Lu
Shanghai Jiao Tong University
(ljf_likit, chentongto22 gzfoxie, louyujing, yonglu_li, _il lucewu)dsjtu.edd
Abstract
Soft-argmax operation is commonly adopted in detection- methods to localize the target position in differentiable manner. However, training the neural network with soft-argmax makes the shape of the probability map unconstrained. Conse- quently, the model lacks pixel-wise supervision through the map during training, leading to performance degradation. In this work, we propose sampling-argman, differentiable training method that imposes implicit constraints to the shape of the probability map by minimizing the expectation of the localization error. To approximate the expectation, we introduce a continuous formulation of the output distribution and develop differentiable sampling process. The expectation can be approximated by calculating the average error of all samples drawn from the output distribution. We show that cmplinngargmaa can seamlessly replace the conventional soft-argmax operation on various localization tasks. Comprehensive experiments demonstrate the effectiveness and flexibility of the proposed method. Code is available at https://github.coo/// http///eeffsjuussmmliini
Introduction
Localizing the target position from the input is fundamental task in the field of computer vision. Common approaches to localization can be divided into two categories: regression- detection-based. Detection-based methods show superiority over regression-based methods and demonstrate impressive performance on a wide variety of tasks [41, 34, 40, 7, 15,9, 9, 17, 12, 32, 18, 31]. Probability maps (also referred to heat maps) are predicted in detection-basee methods indicate the likelihood of the target position. The position with the highest probability is retrieved from the probability map with the argmax operation. However, the argmax operation is not differentiable and suffers from quantization eтror. For accurate localization and end end-to-end learning, soft-argmat [4, 3] proposed as an approximation of argmax. It has found a wide range of applications in human pose estimation [34, 21, 22, 35], facial landmark localization [9, 20, 1]. stereo matching [41, 13, 2] and object keypoint estimation [31].
and
Nevertheless, the mechanism of training networks with soft-argmax is rarely studied. The conven- tional training strategy is minimize the error between the output coordinate from soft-argmax and the ground truth position. However, this strategy is deficient since only provides constraints to the expectation of the probability map, not its shape. As shown Figure 1, these two maps have the same mean values, but the bottom one is more concentrated. In well-calibrated probability maps, positions that locate closer to the ground truth have higher probabilities. Reliable confidence scores of localization results could be provided, which is essential in unconstrained real-world applications and downstream tasks. Besides, imposing constraints on the probability map can provide supervised pixel-wise gradients and facilitate the learning process.
Prior work [28] attempts shape the probability map by introducing hand-crafted regularizations. The variance regularization encourages the variance of the probability map to get close to the pre- defined variance. The Gaussian regularization forces the probability map to resemble Gaussian distribution. We argue that these variants are overconstrained. The hand-crafted constraints are not
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

always correct in different cases, For example, the underlying shape of the probability map is not necessarily Gaussian, and the underlying variance might change as the input changes. Imposing the model to learn fixed-variance Gaussian distribution might degrade the model performance.
novel training method to obtain well-calibrated probability maps and improve the localization accuracy. To constrain the shape of the map, we replace the objective function of minimizing "the error
In this work, present compling-argwann
probability
of the expectation" with minimizing "the expectation of the error". In this way, the network is encouraged to generate higher probabilities around the ground truth position.
meen
natural way to estimate the expectation is by calculating the probability-weiighed sum of the errors at all grid positions. However, we find that the gradient has high variance, and the choose to
probabilit
model is hard to train. To address this issue, approximate the expectation by sampling. The expectation of the error is calculated as the mean error of all samples. Therefore, the sampling process should be differentiable for end-to-end learning.
mean
Figure 1: Top: an unconstrained probability map. Bottom: well- calibrated probability map. These two maps have different shapes but a same mean value.
In our work, we show that the likelihood of the target position can be modelled in the continuous space with mixture dis- tribution. Samples can be drawn from the mixture distribution by three steps: generate categorical weights of the mixture distribution from the probability map; ii) draw samples from sub-distributions: iii) obtain sample by the category wweghted sum. The benefit of using mixture distribution that differentiable sampling from arbitrary continuous distributions can be resolved by differentiable sampling from categorical distributions, which is less challenging and can be addressed by off-the-shelf discrete sampling methods.
compling-argmann is simple and effective. With out-of-the-bee settings, it can be integrated into methods that using soft-argmax operation. To study its effectiveness, conduct experiments variety of localization tasks. Quantitative results demonstrate the superiority of sampling-argmax against soft-argman and its variants. In summary, the contributions of this work threefold:
We propose sampling campling-rgmma argmax for improving detection- based localization methods. By minimizing "the expectation of the error", the network generates well-calibrated probability maps and obtains higher localization accuracy.
We show the output likelihood can be formulated as a mixture distribution and develop a differentiable sampling pipeline.
Comprehensive experiments show that sampling-argmax is effective and can be flexibly generalized to different localization tasks.
2
Preliminary
Given a learned discrete probability map æ, the value Ty, indicates the probability of the predicted target appearing at yı. A direct way to localize the target is taking the position with the maximum likelihood. However, this approach is non-differentiable, and the output is discrete, which impeees end-to-end training and brings quantization errors. Soft-argmax is elegant approximation to address these issues:
Σ Ty.
Notice that is normalized distribution and the soft-argmax operation calculates the probability- weighted sum, which is equivalent to taking the expectation of the probability map म. A conventional way to train the model with the soft-argmax operation is minimizing the distance between the expectation and the ground truth:
C = d(y E,ll)) vo d(yr Σ Ny,
(2)
where y denotes the ground truth position and d(.,)) denotes the distance function, e.g. l distance. We refer to this objective function as "the error of the expectation".

---

3 Method
The conventional detection- based method with soft-argmax only supervises the expectation of the probability map. The shape of the distribution remains unconstrained. In well-calibrated probability maps, the positions closer the ground truth should have higher probabilities. To this end, we proposed new objective function that optimizes "the expectation of the error" instead of "the error of the expectation". In particular, the objective function is formulated as:
L y [d(ve- v)]
The learned distribution tends to allocate high probabilities around the ground truth to minimize the entire loss. this way, the shape of the probability map implicitly constrained.
Discrete Distribution. The probability map predicted by the neural network is discrete. Similar to the soft-argmax operation, the expectation of error can be approximated by calculating the probability-weighter sum of the errors at all grid positions:
L = BE [d( y)1 ~ Σ nt d(v,,
(4)
This approximation treats the distribution of the target position as a discrete distribution. The target only appears at the grid positions, i.c. at position y with the probability Ty.
However, because the underlying target lies in a continuous space, modelling the distribution as discrete distribution is not accurate. The probability map has limited resolution due to the computation complexity. Besides, we find the model is slow to converge by training with Equation 4. When training with Equation 4, the model only obtains 30.9 mAP on COCO Keypoint, while conventional soft-argmax obtains 64.5 mAP, For analysis, we derive the gradient from the loss function to the model parameters 0 under the discrete approximation:
V&L = VeEyId(Ve 8)l
= Σ d(yte y1 V =
Σ d(yte 3e) V log Tyy
(5)
=Eyld(ye y) Vo log my)-
Notice that the form of the gradient is similar to the score function estimator (SF), which is alterna- tively called the REINFORCE estimator [36]. SF estimator is known to have very high variance and slow converge. Therefore, using the discrete approximation for training is not a good solution. This challenge prompts us to explore a better approximation to calculate the expectation of the error.
In the following parts, present sampling argmmax to estimate the expectation of the error by sampling. We first develop continuous approximation to the distribution of the target position (Section 3.1). Then we propose differentiable sampling method (Section 3.2).
3.1 Continuous Mixture Distribution
differentiable process is necessary to estimate the expectation by sampling. However, since the underlying probability density functions can vary among different input images, it is challenging to draw samples from arbitrary distributions differentiably. this work, we present a unified method by formulating the target distribution mixture distribution.
Let p(y) denotes the underlying density function of the target position, which is defined within the boundary of the input image, i.e. y E [0, W). As illustrated in Figure 2(a), the interval [0, W] can be divided into n subintervals. The density function can be partitioned into shapes the subintervals. We could use regular shape (rectangles, triangles, Gaussian functions) in subintervals to form the entire function (as illustrated Figure 2(b-c)).
Formally, given a finite set of probability density functions (f1(), f2(y).... , fn(y)) and weights {wi, w2.... Wwn such that w; 2 0 and] Σ wa 1, the mixture density function p(y) formulated as a sum:
p(v) = Σ wi w.l(
(6)
3

---

Σ
fi(y)
f((v)
(c)
Figure 2: Representing the continuous distribution as a mixture distribution. (a) The original probability density function be viewed as the sum of n sub-functions. Each sub-function be replaced by standard density functions with proper weights to approximate the original function. (b) Approximate the original function by replacing the sub-functions with uniform distribution. (c) Approximate the original function by replacing the sub-function with the triangular distribution, which is equivalent the linear interpolation of the discrete weights.
Here, we can leverage the discrete probability map π to represent the mixture weights, i.e. w;; = Ty. In the context of signal processing, the original function can be perfectly reconstructed if the sample rate (the distance between two adjacent grid points) satisfies the Nyquist-Shannon sampling theorem. However, in our case, the sub-function (v() must be probability density function, i.e. it has the non-negative values, and its integral over the entire space is equal to 1. Therefore, with these restrictions, the original function p(y) cannot be perfectly reconstructed. For approximation, study three different types of standard density functions below.
Uniform Basis. For the uniform basis, the sub-function fi(y) a uniform distribution centred the position y::
E Im 2.3 otherwise,
fl(y) = 6,
where e is the distance between two adjacent grid points.
Triangular Basis. For the triangular basis, the sub-function fz(y) is a triangular distribution:
(y - y1) yy 34 0,
y E [y; 3i), E Dc y: + c). otherwise.
[1()) =
(8)
For y, there exist grid points y/ and yo+1 that satisfy E [yt, W++1]. Therefore, we have p(y) wifi (y)) + (i++1fet11)) Wi+1-w; (y ye) wi which is the linear interpolation of w; and W;+1 1- In 22
other words, using triangular bases is equivalent to the linear interpolation of the discrete probability map.
Gaussian Basis. For the Gaussian basis, f1(y) is the Gaussian function:
1
1
32)
fi())
exp
ov 2T
where a denotes the standard deviation. We set a = e by default in the experiments.
3.2 Differentiable Sampling
In this part, we present how draw a sample from the mixture distribution. We first study the non-differentiable process and then present the differentiable approximation.
Mon-differentiable Process. As illustrated in Figure 3(a), the non-differentiable sampling process can be divided into two steps: i) determine which sub-distribution the sample comes from; ii) draw sample from the selected sub-distribution. In the first step, the sub-distribution can selected by drawing random variable from categorical distribution. The categorical distribution is indicated

---

proboballty
E
sample from
step 1
categorical distribution
slop
Gumbel-sofftma
0
0
0
0
0
0000
0.01
0.02
0.1
0.3
0.45
0.08
0:05
0.01
0.01
i1
fs(u)
0x
fa(y)
In fn(1))
in
fz(u)
De
(u(
Dn Sa(U)
slep 2
sample from the selected sub-distribution
step 2
slep
y
Σ
weighted sum
(a) non-difeerentaabe sampling
(b) differentiable sampling
Figure 3: Illustration the sampling process. (a) The non-differentiable process: i) select a sub-distribution by categorical sampling: ii) draw samples from the selected sub-distribution. (b) The differentiable process: i) approximate the categorical sampled weights by Gumbel-softmax; ii) draw samples from all sub-distribution; iii) add all samples together with the sampled weights. Reparameterization allows gradients to flow from the sample to the probability map.
by the predicted probability map The sub-distribution fi(y) is chosen with the probability Tye There are a number of methods to draw samples from the categorical distribution. Here, we introduce the Gumbel-Max trick [6, 24]:
one www.hhhtthhha.o.. log mi).
(10)
where 91 ...
9n are i.dd samples drawn from Gumbel(O, 1), and the sample is a one-hot vector with the value 1 in the maximum categorical column.
In the second step, sampling from the standard basis function easy to implement. This step independent of the predicted probability map æ. Therefore, the key to differentiable sampling from the mixture distribution to make the first step differentiable.
Differentiable Process. The differentiable sampling process consists of three steps. In the first step, we adopt the Gumbel-softmax [11] operation to sample the categorical weight from probability map. Gumbel-softmax is continuous and differentiable approximation of the Gumbel-Max trick. can obtain an (n 1)-dimensional simplex E A:
exp ((gi + log TT)/T)
(11)
E exp ((g log T*)/T)²
where स ...,...... " and क denotes the sampled weight of the sub-distribution f.(y). As the softmax temperature T approaches O, the simplex á becomes one-hot, and its distribution becomes identical to the categorical distribution म.
In the second step, we draw sample ÿ from every sub-distribution fi()). Note that the sampled weight is not completely one-hot. Therefore, obtain the final sample Y in the third step by adding all samples together with the sampled weight A:
(12)
This process is illustrated in Figure 3(b). With the reparameterizztion trick, the sample Ý is computed as a deterministic function of the probability map and the independent random variables. The randomness of the sampling process is transferred to the variable gl, 99n- We denote the sampling process Y s(T, e), where -- } follows the multivariate Gumbel(O, 1) distribution.
5

---

The gradient from the expected error to the model parameters 0 is derived as:
Od Os dm
VoByy,(y y)] VolE, S(T, e))]
E
(13)
Os On 00
As we see, the gradient of the continuous sampling process is easy to compute via backpropagation. Therefore, can relax the objective function by calculating the average error of the samples drawn from the mixture distribution. The objective function written as:
L = Fxrpp() [d(ye,))) v)
1 N
Σ d Vx)
N Σ s(y,,
(14)
where N, denotes the number of samples. In the testing phase, no randomness is introduced, and sampling-argmax degrades to soft-argmax.
While the sampling process differentiable, the sample Y does not follow the original mixture distribution p(y) for non-zero temperature. For small temperatures, the distribution of Y close to p(y). but the variance of the gradients is large. There is a tradeoff between small temperatures and large temperatures. In our experiments, we start at a high temperature and anneal small temperature.
4 Related Work
Variants of Soft-Argun Soft-Argmax. Nibali et al. [28] introduced hand-crafted regularization to constrain the shape of the probability map.
Variance Regularization. Variance regularization is to control the variance of the probability map. pushes the variance of the probability map close to the target variance of:
L [ar of 12.
(15)
where the target variance of is hyperparameter and the variance of the probability map Var(T) is approximated in a discrete manner, i.c. Var( Σ, Ty. T. -x Tyy Ty 12.
Distribution Regularization. Distribution regularization is to impose strict regularization on the appearance of the heatmap to directly encourage certain shape. Specifically, [28] forces the probability map to resemble a Gaussian distribution by minimizing the Jensen- censen-Shannon divergence between æ and target discrete Gaussian distribution:
L = Js (WI((())))
(16)
Unlike them, our objective function does not set pre-defined hyperparameters for the shape of the map, which makes it general and flexible in applying various applications.
Other works [12, 15] study how to localize target with soft-argmax in different situations. Joung et al. [12] proposed sinusoidal soft-argmax for cylindrical probabilities map. Lee et al. [15] proposed kernel soft-argmax to make the results less susceptible to multi-modal probability map. Our work compatible with these methods by applying the sinusoidal function to the grid positions or multiplying the Gaussian kernel before obtaining the probability map.
Differentiable Sampling. Differentiable sampling for discrete random variable has been studied for long time. Maddison al. [23] and Jang et al. [11] concurrently proposed the idea of using a softmax of Gumbel as relaxation for differentiable sampling from discrete distributions. Kočiský et
al. [14] relaxed the discrete sampling by drawing symbols from logistic-normal distribution rather than drawing from softmax. In this work, unlike previous methods that study discrete distributions, we focus on continuous distributions. We propose relaxation of continuous sampling by formulating the target distribution mixture distribution.
5 Experiments
We validate the benefits of the proposed aampling-argmax with experiments on variety of local- ization tasks, including human pose estimation, retina segmentation and object keypoint estimation.

---

Additional experiments on facial landmark localization are provided in appendix. Sampling-argmman is compared with the conventional soft-argmax and the variants that using additional auxiliary loss [28]. Training details of all tasks are provided in the supplemental material.
5.1 2D Human Pose Estimation from RGB
We first evaluate the proposed sampling-argnan in 2D human pose estimation. In 2D human pose estimation, the probability map is a typical representation to localize body keypoints. The experiments conducted the large-scale in-the-wild 2D human pose benchmark COCO Keypoint [19]. Significant progress has been achieved this field [37, 33, 27, 25]. We adopt the standard model SimplePose [37] for experiments. We follow the standard metric of COCO Keypoint and use mAP over 10 OKS (object keypoint similarity) thresholds for evaluation.
As shown in Table 1, the proposed sampling-argmax significantly outperforms the soft-argmax operation and its variants. Soft, Soft w/V.R. and Soft w/D.R correspond to conventional soft- argmax, soft-argmax with variance regularization and distribution regularization, respectively. Samp, Uni., Tri. and Gau. correspond to sampling-argman
with uniform, triangular and Gaussian basis, respectively. The triangular basis brings 5.3 mAP improvement (relative 8.2%) to the original soft- argmax operation. Besides, we find the auxiliary losses degrade the model performance in COCO Keypoint.
Table I: Quantitative results on COCO Keypoint.
Soft
Soft w/ V.R.
Soft w/ D.R.
Samp. Uni.
Samp. Tri.
Samp. Gau.
mAP
64.5
60.6
55.6
68.2
69.8
68.3
MAP@0.51
84.7
81.5
77.8
87.2
87.9
87.3
mAP@0.751
70.9
65.7
60.8
75.0
76.2
75.2
Number of Samples. In our method, the differentiable sampling process is utilized approximate the expectation of the error. As the number of samples increases, the approximation will be closer to the underlying expectation. To study how the number of samples affects the final results, we compare the performance of the models that trained with different numbers of samples. In Table 2, we report the results with N₂ = {1, 5, 10, 30, 50]. It shows that large number of samples might improve the performance but not necessary. Training the model with only one sample can still obtain high performance while saving computation resources.
Table 2: Comparison of different sample numbers.
N₂
1
10
50
Samp. Uni. 67.8
67.8
67.9
68.2
68.1
Samp. Tri. 69.7
69.7
69.6
69.8
69.8
Samp. Gau.
68.1
68.1
68.2
68.3
68.3
Correlation with Prediction Correctness. For a well-calibrated probability map, the shape of the map could reflect the uncertainty of the regression output. When encountering challenging cases, the probability map would have large variance, resulting in lower peak value. In other words, the peak value establishes the correlation with the prediction correctness. To demonstrate the probability map trained with sampling-argma argmax is better-calibrated, we calculate the Pearson correlation coefficient between the peak value and the prediction correctness. The correctness is represented by the OKS between the predicted pose and the ground-trus -tuth pose. Table 3 compares the correlation with prediction correctness among different methods. shows that sampling-argmas has a much stronger correlation to the correctness than other methods. Compared to the soft-max operation, sampling-argman with the triangular bases brings 85.4% relative improvement. It demonstrates that training with campling-argman can obtain a more reliable probability map, which is essential to real-world applications and downstream tasks.
Table 3: Correlation testing.
Method
Corr. t
Soft
0.233
Soft w/ V.R.
0.158
Soft w/ D.R.
0.082
Samp. Uni. 0.394
Samp. Tri. 0.432
Samp. Gau. 0.423
7

---

5.2 3D Human Pose Estimation from RGB
We further evaluate the proposed sampling- argmaxx on Human3.6N [10], an indoor benchmark for 3D human pose estimation. The 3D probability map is adopted to represent the likelihoods for joints in the discrete 3D space. We adopt the model architecture of prior work [34]. Following previous methods [29, 34, 26, 16], MPJPF and PA-MPJPE [5] are used as the evaluation metrics. Comparisons with baselines are shown in Table The proposed sampling-argmax provides consistent performance improvements. Different from the experiments on COCO Keypoint, the variance regularization provides performance improvements in Human3.6M.
Table 4: Quantitative results on Human3.6M.
Soft
Soft w/ V.R.
Soft w/ D.R.
Samp. Uni
Samp. Tri.
Samp. Gau,
MPJPE 4
50.4
49.7 39.2
51.9 41.4
49.6
49.5
50.9
PA-MPJPE +
39.5
39.1
39.1
39.0
5.3 Retina Segmentation from OCT
Using optical coherence tomography (OCT) obtain 3D retina images widely used in the clinic. A major goal of analyzing retinal OCT images is retinal layer segmentation. Previous work [7] proposes a regression method to regress the boundary and obtain the sub-pixel surface positions. One-dimensional probability maps are leveraged to model the position distribution of the surface in each column. In the testing phase, the soft- coft-argmax method is used infer the final surface positions. The entire surface can be reconstructed by connecting the surface positions in all columns.
The experiments are conducted on the multiple sclerosis and healthy controls dataset (MSHC) [8]. Mean absolute distance (MAD) and standard deviation (Std. Dev.) are used as evaluation met- rics. Quantitative results are reported in Table 5. It shows that sampling-argman achieve superior performance to other methods, while the auxiliary losses also provide performance improvements.
Table 5: Quantitative results on MSHC dataset.
Soft
Soft w/ V.R.
Soft w/ D.R.
Samp. Uni
Samp. Tri.
Samp. Gau.
MAD 4
3.08
0.743
0.746 0.108
0.735 0.101
0.744
0.740
Std. Dev. +
0.281
0.114
0.100
0.104
5.4 Supervised Object Keypoint Estimation from Point Clouds
Detecting aligned 3D object keypoints from point clouds has a wide range of applications on object tracking, shape retrieval and robotics. Probability maps adopted to localize the semantic keypoints. Different from the RGB input, the probability map indicates the pointwise score of the input point cloud, not the grid position of image. The distances between the adjacent point-pairs are different. Besides, point clouds are unordered, and each point has a different number neighbours. Therefore, is hard to directly apply the uniform bases or linear interpolation. which requires constant adjacent distance. Fortunately, the Gaussian basis can be adopted. In the experiment, we set the standard deviation a of the Gaussian bases to 0.01, which the average adjacent point distance in the input point clouds. PointNet++ [30] is adopted as the backbone network. The experiments are conducted on the large-scale object keypoint dataset KeypointNet [39]. The percentage of correct keypoints (PCK) [38] is adopted for evaluation. The error distance threshold is set to 0.01.
Table 6 shows the quantitative results 16 categories. It shows that the proposed sampling aumpling-argmax argmax is also effective on the non-grid input data. Table 6 also compare the results of sampling arggaxx with different numbers of samples. It is seen that N, 30 leads to the best average performance.
5.5 Unsupervised Object Keypoint Estimation from Point Clouds
We then evaluate the proposed method on object keypoint estimation in the context of unsupervised learning. The autoencoder framework is adopted to estimate the keypoint in an unsupervised manner.
8

---

Table 6: Quantitative results of supervised learning on KeypointNet dataset, reported as PCK (higher better).
Bat.
Bed
Bot.
Cap
Cha.
Kni.
Lap.
Mug
Avg
Soft Soft w/ V.R. Soft w/ D.R.
64.9
43.6
44.0
53.9
8.3
40.2
37.2
45.5
43.8
46.6
40.8
23.9
27.7
53.9
32.6
38.2
64.1
41.6
39.2
53.2
12.5
38.3
37.7
44.5
3.7
39.8
52.8
44.0
24,9
25.6
54,4
30.7
37.9
63.2
42.7
43.9
55.8
16.7
42.2
38.6
43.2
4.9
42.4
48.9
41.9
26.8
28.2
54.0
30.3
39.0
Samp. Gau. (N,
43.0
412
53.6
6.2
43.4
38.7
42.5
45.4
50.6
43.5
26.3
33.3
39.3
Samp. Gau. (N. 5)
65.1
42.4
43.8
54.7
12.5
43.2
37.1
44.6
1.9
45.4
46.6
44.7
29.7
26.7
54.6
31.4
39.0
Samp. Gau. (N, 10)
64.0
45.5
41.7
58.6
20.8
40.9
37.0
434
3.7
45.7
48.3
46.4
18.2
34,4
53.5
32.3
39.7
Samp. Gan. (N, 50)
47.5
58.4
44.6
The encoder first estimates the 3D keypoints, and the decoder reconstructs the object point clouds from the estimated keypoints. We follow the srate-of-the-aa method [31] that generates 3D keypoints with the soft-arge aoft-argmax operation for differentiable and end-to-end learning. The soft-argmax is replaced with sampling-argmax where the Gaussian bases with the standard deviation a 0.01 are used.
The experiments are conducted on KeypointNet [39]. Unlike supervised learning, the semantic of each predicted keypoint is unknown in unsupervised methods. Therefore, the PCK metric is not applicable. For evaluation, we adopt the dual alignment score (DAS) following the previous method [31]. Table reports the performance comparison with other methods.
Table 7: Quantitative results of unsupervised learning on KeypointNet dataset, reported DAS (higher is better).
Air.
Bat.
Bed
Bot.
Cup
Car
Cha.
Gui.
Hel.
Kni.
Lap.
Mot.
Mug
Ska.
Tab.
Ves.
Avg
Soft Soft V.R.
69.1
56.2
58.0
45.4
59.1
70.2
76.8
34.1
55.7
50.0
91.5
53.4
52.2
65.7
72.5
35,8
59.1
57.4
82.4
62.3
Soft w/ D.R.
47.9
35.5
47.3
46.1
58.3
65.5
60.9
35.3
47.6
69.3
64.1
55.0
45,9
44.2
57.6
28.8
50.6
Samp. Gau. (N, = 1)
73.9
53.8
63.5
43,9
67.0
69.3
77.7
46.6
59.1
55.9
87.8
59.0
67.0
66.2
80.3
36.4
62.9
Samp. Gau. (N, 5)
73.1
54.0
61.9
48.4
64.4
67.0
81.1
50.7
55.2
50.1
87.5
58.2
58.9
65.9
77.9
41.2
62.2
Samp. Gau. (N, 10)
73.9
58.8
61.7
46.2
60.9
68.6
72.0
53.6
56.5
48.1
91.6
59.8
68.8
65.8
83.5
34,9
62.8
Samp. Gau. (N, 30)
71.2
56.7
60.0
51.0
58.4
64.1
83.8
47.6
61.8
47.8
91.3
55.5
68.5
70.6
81.7
37.5
63.0
5.6 Discussion
Although the variants of soft-argmax can bring improvements in some cases, they need laborious tuning of parameters, such as the weight of the regularization term and the variance of the target distribution. The best parameters for different tasks are different. Besides, the best parameters for variance regularization and distribution regularization is also different, which increases the effort needed for the process of parameters tunning. In our experiment, we tune the loss weight ranging from 0.1 10 and the variance ranging from I for each task. After laborious tuning, the performances of these variants are still not consistent across different tasks and they are inferior to the performance of our method, while our method is think our method is effective and general to different cases.
and free from parameters tuning. Therefore, we
In addition to probability maps and provide more reliable confidence scores. COCO Keypoint uses the mAP metric to evaluate multi- cullti-pesson pose estimation. Thus reliable confidence scores could also improve the performance. In other datasets, the metric only reflects the localization performance and ignore the importance of confidence scores. In many real-world applications and downstream tasks, reliable confidence score is very important and necessary.
accurate localization performance, sampling-argmax can predict well-calibrated
6 Conclusion
In this paper, we propose compling-argmant an operation for improving the detection- basedd local- ization. Sampling Sampling-argmax argmax implicitly imposes shape constraints to the predicted probability map by optimizing "the expectation of error". With the continuous formulation and differentiable sampling, sampling-argmat can seamlessly replace the conventional soft-argmax operation. We show that sampling-argnan effective and flexible by conducting comprehensive experiments on various localization tasks.

---

Funding Funding in direct support of this work: the National key R&D Program of China, No. 2017YFA0700800, National Natural Science Foundation of China under Grants 61772332 and Shanghai Qi Zhi Institute, SHEITC (018-RG7N-00046),
References
[1] Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. Attention-driven cropping for very high resolution facial landmark detection. In CVPR, 2020.
[2] Shivam Duggal, Shenlong Wang, Wei-Chiu Ma, Rui Hu, and Raquel Urtasun. Deeppruner: Learning efficient stereo matching via differentiable patchmatch. In ICCV, 2019.
[3] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In ICRA, 2016.
[4] Ross Goroshin, Michaël Mathieu, and Yann LeCun. Learning to linearize under uncertainty. In NeurIPS, 2015.
[5] John C Gower. Generalized procrustes analysis. Psychometrika, 1975.
[6] Emil Julius Gumbel. Staristical theory of extreme values and some practical applications: series of lectures, volume 33. US Government Printing Office, 1954.
[7] Yufan He, Aaron Carass, Yihao Liu, Bruno M Jedynak, Sharon D Solomon, Shiv Saidha, Peter A Calabresi, and Jerry L Prince. Fully convolutional boundary regression for retina segmentation. In MICCAI, 2019.
[8] Yufan He, Aaron Carass, Sharon D Solomon, Shiv Saidha, Peter A Calabresi, and Jerry L Prince. Retinal layer parcellation of optical coherence tomography images: Data resource for multiple sclerosis and healthy controls. Data in brief, 2019.
[9] Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher Pal, and Jan Kautz. Improving landmark localization with semi-supervised learning. In CVPR, 2018.
[10] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchiseseu. Human3, 6.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments. TPAMI, 2014.
[11] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax In ICLR, 2017.
[12] Sunghun Joung, Seungryong Kim, Hanjae Kim, Minsu Kim, Ig-Jae Kim, Junghyun Cho, and Kwanghoon Sohn. Cylindrical convolutional networks for joint object detection and viewpoint estimation. In CVPR, 2020.
[13] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression, In ICCV, 2017.
[14] Tomáš Kočiský, Gábor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and Karl Moritz Hermann. Semantic parsing with semi-supervised sequential autoencoders. In EMNLP, 2016.
[15] Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham. Sfnet: Learning object-aware semantic correspondence. In CVPR, 2019.
[16] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log-likelihood estimation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11025-11044 2021.
[17] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Effi- cient crowded scenes pose estimation and new benchmark. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 10863-10872, 2019.
10

---

[18] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3383-3393, 2021.
[19] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[20] Yinglu Liu, Hao Shen, Yue Si, Xiaobo Wang, Xiangyu Zhu, Hailin Shi, Zhibin Hong. Hanqi Guo, Ziyuan Guo, Yanqin Chen, al. Grand challenge of 106-point facial landmark localization. In ICMEW, 2019.
[21] Diogo C Luvizon, David Picard, and Hedi Tabia. 2d/3d pose estimation and action recognition using multitask deep learning. In CVPR, 2018.
[22] Diogo C Luvizon, Hedi Tabia, and David Picard. Human pose regression by combining indirect part detection and contextual information. Computers & Graphicx, 2019.
[23] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017.
[24] Chris J Maddison, Daniel Tarlow, and Tom Minka. A sampling. In NewrIPS, 2014.
[25] William McNally, Kanav Vats, Alexander Wong, and John McPhee. Evopose2d: Pushing the boundaries of 2d human pose estimation using neurocvolution. arXiv preprint arXiv:2011.08446 2020.
[26] Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Camera distance-aware top-down approach for 3D multi multi-person pose estimation from single rgb image. ICCV, 2019.
[27] Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Posefix: Model-agnostic general human pose refinement network. In CVPR, 2019.
[28] Aiden Nibali, Zhen He, Stuart Morgan, and Luke Prendergast. Numerical coordinate regression with convolutional neural networks. arXiv preprint
2018.
[29] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. Coarse-to- fine volumetric prediction for single-image 3D human pose. In CVPR, 2017.
[30] Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point in a metric space. In NeurIPS, 2017.
[31] Ruoxi Shi, Zhengrong Xue, Yang You, and Cewu Lu, Skeleton merger: an unsupervised aligned keypoint detector. In CVPR, 2021.
[32] Riceardo Spezialetti, Federico Stella, Marlon Marcon, Luciano Silva, Samuele Salti, and Luigi Di Stefano. Learning to orient surfaces by self-supervised spherical cnns. In NeurlPS, 2020.
[33] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
[34] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018.
[35] Can Wang, Jiefeng Li, Wentao Liu, Chen Qian, and Cewu Lu. Hmor: Hierarchical multi-person ordinal relations for monocular multi-person 3d pose estimation. In European Conference on Computer Vision, pages 242-259. Springer, 2020.
[36] Ronald J Williams. Simple statistical fralient algorithms for connectionist reinforce- ment learning, Machine learning, 1992.
[37] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV, 2018

---

[38] Li Yi, Hao Su, Xingwen Guo, and Leonidas J Guibas. Synespecenn: Synchronized spectral enn for 3d shape segmentation. CVPR, 2017.
[39] Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, Liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. Keypointnet: A large-scale 3d keypoint dataset aggregated from numerous human annotations. In CVPR, 2020.
[40] Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleopera- tion. In ICRA, 2018.
[41] Chao Zhou, Hong Zhang, Xiaoyong Shen, and Jiaya Jia. Unsupervised learning of stereo matching. In ICCV, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We claim that the proposed sampling-argee -argmax can help the model obtain well-calibrated probability maps and improve the localization accuracy. our experiments, we validate the localization accuracy of sampling-argman across six tasks. We also demonstrate that the probability maps well-calibrated by conducting correlation testing.
(b) Did you describe the limitations of your work? [Yes] Please see Section ??.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [Yes] Our code is attached in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training details are claborated in the supplemental material. (c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Error bars are not reported because it would be too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The required training resources (number of GPUs) are elaborated in the supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please Section 5.
(b) Did you mention the license of the assets? [Yes] Please see the supplemental material.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'le re using/curating? [Yes] All data we used is publicly available. Please see the supplemental material.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Please see the supplemental material.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
12

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "aSObUfYS7S", "reviewer": "Reviewer_mj3d", "summary": "The paper proposes a sampling-based soft argmax formulation that is differentiable. The method focuses on the fact that the conventional softmax algorithm focuses on having the expectation of a heatmap to be in a desired location, which may not be what one desires. Instead, the paper proposes to consider the expected error to the desired location, should one sample according to the heatmap. To implement this idea, the paper proposes a two-stage sampling strategy that utilized the Gumbel softmax and Kernel Density Estimation. The method is shown to outperform a naive softmax strategy in various tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strategy proposed in the paper seems effective and can be used as a drop-in replacement over existing soft argmax implementations. The motivation behind the paper is also interesting albeit simple. It does point out an overlooked problem with conventional soft argmax formulations. However, the paper in its current form has critical issues that need to be resolved before it could be considered for publication.\n\n1. Why sampling?\n\nAlthough the authors motivate that the variance of the formulation in Eq 5 would be large, this is never demonstrated. REINFORCE in some cases does not suffer from the variance issue and can be trained without too much trouble, for example as shown by [a]. Thus, to have the sampling-based scheme justified, either theoretical or empirical analysis should be provided. The paper provided neither.\n\n[a] Michal et al., \"DISK: Learning local features with policy gradient\", NeurIPS 20\n\n2. Isn't 3.1 Kernel Density Estimation (KDE)?\n\nThe formulation shown in 3.1 seems to be KDE, also known as Parzen windowing. This formulation is simply a sum of kernel functions, which all are friendly to backpropagation. Also, the formulations shown here are cases when these kernel bandwidths are set to behave as interpolation kernels. As such, I also find it hard to get convinced that approximating this with sampling is a better idea than simply using the analytical gradient.\n\nIn fact, the temperature used in Eq. 11 can be incorporated into the analytical solution as kernel bandwidths, which would effectively behave the same, but less randomly and as if one took infinite samples. As shown by the results in Table 2, more samples seem to lead to a better outcome, thus the analytical solutions seem like they would be better.\n\n3. Experiments required to answer 1 and 2\n\nThus, simply put, the following baselines are missing to demonstrate the effectiveness of the method.\n- Method trained with Eq. 5\n- Method trained by directly using the KDE formulations instead of sampling\n\n4. Is it truly better than regularization?\n\nAnother natural question for the method is whether the method is truly better than regularization. For example, `Soft w/ V.R` seems to perform quite well in most of the experiments, almost as good as the proposed method (except Table 6). The differences are small enough to require multiple runs. While the paper argues in the checklist that multiple runs would be costly, at this point where the performance difference is around 1% difference relative, it is hard to see if the proposed method brings significant advantage without it. Moreover, considering that the regularization parameters can affect the performance quite a lot, the gain from the proposed method is not convincing.\n\nAn addition to this is when considering results in Table 1. In [b], a heatmap-based supervision method is employed, which would be somewhat similar to `Soft w/ D.R.`, in the sense it is enforcing a certain distribution. In [b], the method reports 73.7 mAP, which is significantly higher than the 69.8 mAP of the proposed method. \n\n[b] Li et al., \"Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation\", AAAI 20\n\n5. Presentation\n\nThe quality of the presentation needs improvement. The paper does not read well in some cases, with quite a few grammatical errors. For example line 1 \"the detection-based method\"; line 7 it is unclear at this point what error the paper is talking about; line 16 \"from the input is\"; line 61 categorical weights is unclear at this point since mixture distributions could be anything; line 62 the same goes for the term \"sub-distribution\"; line 62 it is unclear what category means at this point; line 103 \"we find the model is slow to converge\". This is not an exhaustive list.\n\n==== Post Rebuttal Update ====\n\nAfter the rebuttal and the discussions, I am now convinced that the method greatly outperforms the closed-form solution without sampling (I would love to try this for myself as I find this gap still very surprising!). However, I am still concerned about the discussion on KDE, and the lack of significance tests, which make me hesitant to cross the acceptance threshold. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the paper lays out the limitations of using KDE and mention normalizing flows, but I am not sure how this could be done. For example, this would require that the method is aware of the distributions that can occur, while the aim of the method is to be used as a generic soft argmax algorithm. To me, it seems like a contradiction here.\n\nWhile I do not believe it is a major issue, in the Broader Impact section, the paper discusses reducing the computational burden, but the proposed method itself would require more computation than a typical soft argmax. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QFo3NW2E0V", "reviewer": "Reviewer_CcdV", "summary": "This paper proposes a different loss function for training localisation models in computer vision. Currently, many keypoint-localisaiton tasks use the soft-argmax to predict the location of a keypoint from heatmaps / probability maps predicted by the network. The authors note that the soft-argmax only encourages the mean of the predicted heatmap to match the ground truth (\"error of the expectation\"). The authors then propose a sampling-argmax which minimises the \"expectation of the error\" by computing the probability-weighted sum of errors at all locations. As this method has high variance and trains poorly, the authors introduce additional structure into the predicted probability maps by representing them as a linear combination of basis functions, and show how this can be sampled from in a differentiable manner.\n\nThe authors present strong results on multiple keypoint localisation tasks where they outperform the soft argmax, and other regularisation methods for heatmaps [21]. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors have two main contributions in their loss: representing the predicted probabilty map as a linear combination of basis functions, and then sampling from this to compute the expectation of the proposed loss (Eq 4). However, the effects of these two contributions are not really disentangled from each other.\n\nIt is possible to use a linear combination of basis functions for the probability map, and the standard soft-argmax as the loss function (ie no stochasticity in the loss). This ablation is not performed at all, and I believe it would strengthen the paper considerably. Without it, it is not clear to me if the \"sampling\" part of the contribution is necessary at all.\n\nAnd it is also possible to use the sampling-based loss without the basis functions. The authors mention on Page 3 that \"we find the model is slow to converge by training with Equation 4\", but do not include actual experimental results on it, which I believe would be useful to the reader.\n\nTable 2 shows that using more samples in the loss improves final accuracy, in line with what I would expect. However, the computational cost of using more samples is not detailed at all, and is an important detail.\n\nI have some further comments about the experiments:\n\n- In general, the tables do not have informative captions. And it is not immediately clear what \"Soft\", \"Soft w/ V.R \" etc are as these abbreviations are not described anywhere.\n- A strength of the paper is that the authors have experimented on multiple different keypoint localisation tasks. However, the resutls are not all necessarily consistent across them, and there is no real explanation for this\n  - The sampling-argmax gives large improvements on COCO keypoints, but not on Human 3.6M. Why is this?\n  - In Table 5, the soft-argmax baseline performs very badly. Why is this? Is this an error with the implementation rather than the method?\n  - Why does Soft-argmax with distributional regularization do so badly in Table 7, unlike all the other experiments?\n- The Brier Score and Expected Calibration Error are more standard metrics for comparing the calibration of a network. I think it would be better to also report these in Table 3, unless the authors have reasons for not doing so.\n\n\nThere are a few writing issues in the paper too:\n\n- Line 1. \"Soft-argmax operation is commonly adopted ...\" -> \"The soft-argmax operation ...\"\n- Line 77. \"Preliminary\" --> \"Preliminaries\"\n\n\nMy main concern about the paper is that the effects of the main contributions (basis functions and sampling to compute the expectation) are not disentangled from each other. I am willing to increase my rating if this is addressed in the rebuttal.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. Limitations are included in the main paper, and societal impact in the supplementary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Mah700dDDJa", "reviewer": "Reviewer_N85X", "summary": "The paper proposes a method for learning better calibrated probability distributions for detection (i.e., localisation and classification) tasks. Rather than directly apply the maximum likelihood principle to learn the parameters of the model that maximizes the probability of the ground-truth detection, the approach introduces a differentiable sampling operation with mixture kernels to better control the \"shape\" of the learned probability distribution. This replaces the argmax delta function approximation and is compared against other regularisation mechanisms such as variance regularisation (L2) and distribution regularisation (Jensen-Shannon divergence). While I have some concerns about the theoretical motivation for the proposed approach (see below), the empirical results are compelling for the tasks considered in the experiments (human pose estimation, retina segmentation and facial landmark localisation), noting that these are all structured prediction tasks in some sense.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are several grammatical errors but overall the paper is straightforward to understand. My main concern is the theoretical positioning of the work and the discussion around the term \"shape\". The maximum likelihood principle for learning model parameters is well-established and theoretically sound, and any deviation from it needs to be justified clearly and carefully, e.g., insufficient data and the need for model priors (maximum a posteriori estimation), poor model calibration or approximation due to improve tractability. While the paper introduces a nice differentiable approach that improves on some experimental benchmarks, insights relating to the theoretical justification for this remains unclear.\n\nIn particular, for a typical single object detection task the probability distribution appears to be unimodal and highly peaked---the object appears at a single (well-defined) location within the image. Here standard regularisation techniques appear to handle the small ambiguity in the neighbourhood of the mode quite well. For a multimodal scenario the situation is different and a model that learns to match a single mode may not perform well. The tasks explored experimentally in this paper are all structured prediction tasks in that they predict on multiple output variables simultaneously, where exploring multiple modes may be what is leading to the improved empirical results.\n\nIn short, the paper argues for constraining the \"shape\" of the learned probability distribution in deep learning architectures and provides a differentiable sampling approach for doing so. However, it does not make clear under what scenarios this is the right thing to do compared with standard maximum likelihood. A discussion in the rebuttal and indication of how this will be addressed in the paper will help me to refine my recommendation for the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I cannot see any negative social impact of this work. Minor technical limitations are mentioned but the bigger question raised above is not addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GLItwvN6LNu", "reviewer": "Reviewer_K5Z7", "summary": "The authors propose a novel Sampling-Argmax operator, that they demonstrate empirically out-performs the traditional Soft-Argmax operator on a number of localization tasks. Firstly, the authors propose a modified loss which encourages probability maps to have high likelihood near the ground truth location (i.e. the loss adds sensitivity to pixel location), while not overly constraining the predicted distribution. Next, the authors acknowledge that this loss has high gradient variance, and so propose a sampling method to stabilize training and reduce grad variance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A few minor points first:\nFor some reason I found the notation in (3) confusing, while the written out discrete distribution in equation (4) was immediately obvious. In particular, the fact that the network predictions \\pi_{y_i} are not present in the equation makes it rather difficult to parse.\nLine 107: To make the paper more self contained, I would have appreciated an explanation for why the SF estimator has high variance rather than a citation.\nFigure 1 seems superfluous. I struggled to figure out what the insightful point was until I realized that all this is trying to convey is that 2 distributions can have the same mean. This could have been done with one sentence.\n\\pi_{y_i} is a normalized distribution. It’s probably obvious to the authors how the network predictions are normalized, but in the interest of being more self contained, I would state this exactly (i.e. there is a exp()/sum(exp()) in there).\nIt’s not necessarily an issue, but the quantitative results are effectively all ablations. For example, on COCO Keypoint, the authors do not show the performance of other methods (e.g. SoTA large-scale detection). I think these numbers are important to calibrate the reader and decide if the improvements are statistically significant compared to other methods.\n\nHowever, the paper is well written. The problem is well scoped and the solution seems reasonable and empirically justified. While the technical contribution feels a little light (IMO) - e.g. there’s a few tricks being borrowed from pretty standard papers at this point (e.g. gumbel softmax), the method as presented does seem novel (from a quick literature review) and I can’t think of any major reason NOT to accept this paper. Furthermore, soft-argmax has a pretty loyal following in the community (ironically, I’m not one of them), especially those who work in compute-constrained environments, and I think this paper would be of interest to that community.\n\nOne other minor point is that I think soft-argmax is not well cited. There are two concurrent works that first presented the idea: Goroshin et al., Learning to Linearize Under Uncertainty and Finn et al., Deep Spatial Autoencoders for Visuomotor Learning. These should be added.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No obvious limitations that I can think of and that the authors haven't already addressed (e.g. that their mixture distribution might not represent the true underlying distribution).\n\nThis method presents a loss function that can be used on a wide range of localization tasks in different domains and with different input modalities. In that sense, it’s impact could be large as it benefits multiple topics.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received mixed ratings, with one reviewer recommending acceptance and three rejection. The paper was thoroughly discussed by the reviewers but the authors' feedback did not convince the negative reviewers to recommend acceptance. In particular, the reviewers' main concerns include the motivation behind the proposed sampling strategy and its theoretical justification, the lack of comparison with some baselines (although the authors provided some of these comparisons in their feedback, the reviewers remained unconvinced), and the significance of the improvements obtained in some experiments. Based on this, we therefore believe that this paper is not ready for acceptance to NeurIPS but encourage the authors to revise it based on the feedback and resubmit to a future venue.
Decision: Accept (Poster)