DIVE DEEPER INTO INTEGRAL POSE REGRESSION
Year: 2021

Abstract:
Integral pose regression combines the use of an implicit heatmap with end-to-end training for human body (Sun et al., 2018; Nibali et al., 2018) and hand (Iqbal et al., 2018) pose estimation. Unlike detection-based heatmap methods, which decode final joint positions from the heatmap with a non-differentiable argmax operation, integral regression methods apply a differentiable expectation operation. The differentiable decoding allows for end-to-end training directly from ground-truth coordinates, though this slows down the learning process. It also leads to curious differences in performance, i.e. integral regression is competitive or better than detection-based methods on “hard” samples but perform worse on the “easy” samples (Gu et al., 2021). We do a deep dive on the inference and back-propagation of integral pose regression to better understand the causes behind the performance and training differences. For inference, we give theoretical support that expectation is better than argmax operation, but it only takes place in hard cases in practice as activation region of heatmaps shrink in easy cases. We then experimentally show that the shrinkage of activation regression is one of the main causes of its inferior performance. For back-propagation, we theoretically and empirically analyze the gradients to explain the slow training speeds. Our analysis based on expectation operation and backward propagation gives insight to understand IPR and we use experiments to demonstrate the capability of IPR to surpass the performance of detection.

heading: 1 INTRODUCTION
text: 2D human pose estimation aims to detect the image coordinates of the body and or the hand. In the last few years, detection-based methods (Newell et al., 2016; Xiao et al., 2018; Sun et al., 2019; Li et al., 2019) and integral pose regression (IPR) methods (Sun et al., 2018; Iqbal et al., 2018) have emerged as two common paradigms for performing human pose estimation. Both detection-based and IPR methods learn a heatmap representing pixel-wise likelihoods of the joint positions; the heatmap is learned explicitly for detection-based methods, but remains implicit or “latent” for IPR. To decode the heatmaps to joint coordinates, detection methods use an argmax, while IPR methods take the expected value. As the expectation operation is differentiable, IPR has the benefit of being end-to-end learnable, even though detection methods seem more competitive accuracy-wise (COC, 2020).
Gu et al. (2021)’s recent work showed a curious performance difference between IPR and detection methods. With a nuanced split over the evaluation set, IPR outperforms detection when test samples are “harder”, i.e. , with fewer keypoints present in the scene, under higher occlusion, and lower bounding box resolutions. Given that both detection and IPR methods work with fully convolutional feed-forward architectures, this naturally begs the question - what are the reasons behind this performance difference? Why is it that IPR, which otherwise seems to lag behind detection methods in performance (COC, 2020), can excel on these hard cases? This serves as the motivation for our closer study and analysis of integral pose regression.
Detection and IPR methods differ in two aspects (see Fig. 1 for an overview) in the forwards and backwards pass. In the forwards pass, the heatmap is decoded with an argmax for detection versus a softmax normalization and expectation for IPR. In the backwards pass, detection methods are supervised with an explicitly defined Gaussian heatmap centered on the ground truth joint, while IPR is supervised by the joint coordinates directly. Through detailed theoretical analysis and experimentation on the decoding and back-propagation process, we make following findings and contributions:
1. We propose a unified model of the heatmap to interpret and compare detection and integral regression methods. We verify the model experimentally and show that as samples progress from hard to easy, the region of activation on the heatmap decreases for both detection and integral regression methods.
2. We demonstrate experimentally that degenerately small regions of activation degrades the accuracy of both detection and integral regression methods.
3. Integral regression methods, by decoding the heatmap with an expectation operation, should result in a lower expected EPE than detection methods that decode with an argmax operation. In practice, this can be observed only for hard samples due to the shrinkage of the active region on the heatmap.
4. Direct supervision with the joint coordinates in IPR, although end-to-end, suffers from gradient vanishing and provides less spatial cues for learning the heatmap than the explicit heatmap supervision of detection methods. As a result, the training of IPR is more inefficient and slower to converge than detection methods.
In conclusion, our findings serve as insight to understand IPR which has a better decoding method theoretically and shows the potential to surpass the performance of detection by our experiments.

heading: 2 RELATED WORK
text: Since the concept of “heatmap” is proposed in (Tompson et al., 2014), detection-based methods have been top performers for human pose. Existing detection-based works emphasize to extract high-quality multi-resolution features. Xiao et al. (2018) proposed to adapt ResNet with deconvolution layers while Hourglass (Newell et al., 2016) and Cascaded Pyramid Network (CPN) (Li et al., 2019) introduced cascaded network architectures with coarse-to-fine design paradigm. HighResolution Network (HRNet) (Sun et al., 2019) follows the coarse-to-fine paradigm and further improves the performance by adapting more dense connection among different resolution representations. IPR (Sun et al., 2018) introduced a competitive regression-based framework, but exceeded by more advanced detection-based works (Xiao et al., 2018; Sun et al., 2019).
Numerical regression-based methods, which directly regress the joint coordinates, are commonly used in facial land-mark detection (Feng et al., 2018; Zhu et al., 2020). These methods however are not accurate for human pose because they fail to capture spatial knowledge. As a follow-up, IPR tries to merge the spatial knowledge from the heatmap implicitly, hence the “latent” heatmap. IPR methods are especially preferred in hand pose estimation (Spurr et al., 2020; Yang et al., 2021) but are still less common for the human pose estimation (Sun et al., 2018; Nibali et al., 2018).
Recently, two parallel lines of work have developed to further understand the behaviour of detection and IPR methods respectively. For detection-based methods, Huang et al. (2020) revealed a heatmap bias caused by the inconsistency of coordinate system transformations and solved it by redesigning the transformations during data processing. Moreover, as the predicted heatmaps during inference may violate the Gaussian assumption and worsen the performance, Huang et al. (2020) proposed to find the optimal Gaussian location based on the prediction and Zhang et al. (2020) directly modulated the predicted Gaussian distribution for decoding.
For IPR methods, Nibali et al. (2018) experimentally compared different heatmap regularizers, heatmap normalization schemes and loss functions. They find using Jensen-Shannon regularization on the heatmap with a softmax normalization and L1 loss achieves the best performance for IPR. More recently, Gu et al. (2021) performed a systematic comparison of detection vs. IPR methods with a common backbone and discovered the performance advantage of IPR on the “hard” samples. Furthermore, Gu et al. (2021) revealed a bias that arises from taking the expectation after the softmax normalization; they propose a compensation scheme to mitigate the bias which in turn improves the overall performance of IPR, making them competitive with detection methods.
The work (Zhang et al., 2020; Huang et al., 2020) focus on the explicit heatmaps of detection-based methods and investigate the bias under the Gaussian assumption. However, this is not applicable for IPR methods without the Gaussian assumption. Like (Nibali et al., 2018; Gu et al., 2021), our work also investigates the implicit heatmaps of IPR; the previous works focus on experimental exploration, while we place more emphasis on theoretical analysis to pinpoint underlying causes.

heading: 3 PRELIMINARIES ON HUMAN POSE ESTIMATION
text: In this work, we target the more commonly used ‘top-down’ form of pose estimation in which a person detector already provides a cropped image I of the person. For simplicity, we focus our discussion on one given joint of the K total joints in the body. The pose estimation model outputs a heatmap Ĥ ∈ RM×N where M and N are the dimensions of the spatial heatmap. Typically, M and N are scaled down by a factor of 4 from the original input dimensions of I (Xiao et al., 2018; Sun et al., 2019). The heatmap Ĥ represents a (discrete) spatial likelihood P (J|I), where J ∈ R1×2 is the 2D coordinates of that joint. In practice, all K heatmaps are predicted simultaneously by the same network, where each joint is one channel. In both detection and IPR methods, the coordinates Ĵ are decoded from Ĥ; where the two methods differ is the manner of decoding (see Sec. 3.1) and the form of supervision applied (see Sec. 3.2).

heading: 3.1 HEATMAP DECODING: MAX VERSUS EXPECTED VALUE
text: Detection Methods apply an argmax on Ĥ indexed by p to estimate the joint coordinates Ĵde:
Ĵde = arg max p Ĥ(p), (1)
Taking an argmax can be interpreted as taking a maximum likelihood on the heatmap Ĥ, assuming that Ĥ is proportional to the likelihood. In practice, the final Ĵde value is determined as a linear combination of the highest and second-highest response on Ĥ as a way to account for quantization effects in the discrete heatmap (Newell et al., 2016). A more recent work, DARK (Zhang et al., 2020) approximates the true prediction by a Taylor series evaluated at the maximum activation of the heatmap and shows this to be more accurate.
Integral Pose Regression applies an expectation operation on Ĥ to give a mean estimate of the joint coordinates. To do so, the heatmap must first be normalized to sum up to 1; the most common and effective (Nibali et al., 2018) approach is to apply a softmax normalization. Afterwards, the predicted joint Ĵre with x and y components Ĵx and Ĵy1 is determined by taking the expectation on the normalized heatmap H̃ with elements h̃p at location p:
Ĵre = [ Ĵx Ĵy ] = ∑ p∈Ω p · h̃p where h̃p = eβĥp∑
(p′)∈Ω eβĥp′
. (2)
Here, Ω is the domain of the heatmap and β is a scaling parameter used in the softmax normalization. Note that softmax normalization assigns a non-zero value to all pixels in H̃, even if it was originally zero in Ĥ. These values also contribute to the expected value, resulting in a center-biased estimated joint coordinate Ĵre (Gu et al., 2021). The smaller the β is, the stronger the bias is. Although Gu
1For clarity, we drop the subscript ‘re’, as we refer to the individual components for IPR only.
et al. (2021) propose a compensation scheme, for the purposes of our analysis we will assume that β is sufficiently large that the impact of the bias is negligible.

heading: 3.2 SUPERVISION: EXPLICIT HEATMAP VS. GROUND TRUTH COORDINATES
text: Detection Methods are learned by providing supervision on the heatmap. The ground truth H is given as a circular Gaussian, with a mean centered at the ground truth joint coordinate (see Fig. 1). The loss applied is a pixel-wise MSE between the ground truth hp and the predicted ĥp:
Lde = ||H− Ĥ||22 = ∑ p∈Ω (hp − ĥp)2, (3)
where Ω is the domain of the heatmap. As the loss is defined in terms of the heatmap and not the actual variable of interest, i.e. the predicted joints, detection based methods are not end-to-end in their learning and this is often cited as a drawback (Sun et al., 2018; Zhang et al., 2020).
Integral Pose Regression defines a loss based on the difference between the prediction Ĵre and ground truth joint location J. The L1 loss empirically performs better than the L2 (Sun et al., 2018):
Lre = ‖Jgt − Ĵre‖1 = (|Ĵx − Jx|+ |Ĵy − Jy|). (4) IPR are end-to-end methods because they provide supervision at the joint level. As the estimated H is learned only implicitly, some works refer to the heatmap as “latent” (Iqbal et al., 2018).

heading: 3.3 PERFORMANCE DIFFERENCES
text: The performance of detection versus integral regression methods on human pose estimation has been compared by Sun et al. (2018) and Gu et al. (2021). The former found that integral regression is either as competitive or better for 2D pose estimation, though results are not fully conclusive, as Sun et al. (2018) used a different backbone for integral regression than detection.
The more standardized and detailed comparison of Gu et al. (2021) used a common backbone and partitioned the data based on three factors of variation dominating current human pose estimation benchmarks (Ruggero Ronchi & Perona, 2017): (1) the number of joints present in the scene, (2) the percentage of occlusion and (3) the largest dimension of the bounding box input. The “hard” samples are those with either 1-5 joints present or > 50% occlusion or 32-64 pixel input; integral regression methods had, on average, 15% lower EPE than detection methods. The “easy” samples are defined by 11-17 joints, <10% occlusion and >128px input; detection methods were marginally better. The remaining cases with factors in between were considered to be “medium”, where performance of both methods are roughly equivalent.

heading: 4 ANALYSIS ON HEATMAP DECODING

heading: 4.1 LOCALIZED HEATMAP MODEL
text: To analyze the performance differences between detection and IPR methods, we make the following assumptions on the heatmap itself. First and foremost, we assume that a well-trained network will produce a heatmap ĥp with significant or large values in a localized support region around Ĵ. We denote the region of support as Φ and assume that outside of Φ, the heatmap activation is approximately zero (see Fig. 2 (a)).
Specifically, we model that the normalized heatmap2 H̃ is modelled by some density distribution P . We make no assumption on the form of P , but do assume that it fully captures the support for the joint location within Φ and that Φ is centered on the expected value of P on location µ = (µx, µy) with an area of (2s+ 1, 2s+ 1), i.e. ,
µ = ∑ p∈Ω p · h̃p = ∑ p∈Φ(µ,s) p · h̃p = [ µx+s∑ i=µx−s µy+s∑ j=µy−s i · h̃ij , µx+s∑ i=µx−s µy+s∑ j=µy−s j · h̃ij ]ᵀ . (5)
2For purpose of discussion, we may consider normalizing the heatmap for detection methods as per Eq. 2, which will not affect the outcome since arg max(H̃)=arg max(Ĥ).
Experimental Verification of Localized Heatmap Model The assumption of localized activations is reasonable for detection methods as their heatmaps are learned explicitly to match a (localized) ground truth Gaussian. We empirically verify this model by statistically tallying the activations in the normalized heatmap for both detection and integral regression methods based on the easy/medium/hard splits of Sec. 3.3.
Specifically, we train both a detection and an IPR network for human pose estimation. For both networks, we use a Simple Baseline (SBL) (Xiao et al., 2018) architecture with a ResNet50 (He et al., 2016) backbone. The two networks differ only in their manner of decoding and form of supervision as outlined in Section 3. We then apply the networks on the standard human pose estimation benchmark MSCOCO (Lin et al., 2014) using the same easy, medium and hard splits as Gu et al. (2021). For each produced heatmap of 64×48, we estimate an activation sum A by summing the normalized heatmap activations within a 2s+ 1 square around the ground truth coordinate, (Jx, Jy):
A(s) = Jx+s∑ i=Jx−s Jy+s∑ j=Jy−s h̃ij . (6)
The larger A is, the more localized the activations within the region around the ground truth joint, with a maximally achievable A of 1. Plotting A versus s (see Fig. 2 (b)), we observe that a region of 19x19, i.e. s=9 is sufficient to contain 95% of all non-zero activiations for both detection and regression values. Unsurprisingly, the containing region is smaller for easy samples, i.e. activations on the heatmap are more localized and gets progressively larger for the medium and hard samples, as the network becomes more uncertain in determining the joint location (see larger plot in Appendix A.1).

heading: 4.2 EXTREME LOCALIZATION AS A CAUSE FOR PERFORMANCE DIFFERENCES
text: Expected End-Point-Error (EPE) For detection methods that decode the heatmap with an argmax, we assume that each position p has some probability w(µ, s) of being the maximum activation, i.e. arg max(P)Φ(µ,s) ∼ w(µ, s). Here w(µ, s) represents a radially symmetric distribution centered on µ with all non-zero support contained within Φ. It follows that the expected EPE for one sample for detection methods can be defined as:
Ede(µ, s) = ∑
p∈Φ(µ,s)
w(p)||Jgt − p||2, (7)
where Jgt denotes the ground truth coordinates. For integral regression methods that decode the heatmap with an expectation, the estimated joint coordinate aligns with the center of Φ by definition, i.e. Ĵ=µ, which leads to the following expected EPE for a single sample:
Ere(µ) = ||Jgt − µ||2. (8)
It can be verified (see proof in Appendix A.2) that the Ede ≥ Ere. This result suggests that for some fixed (µ, s), the expected EPE of regression should always be better than detection. However, the results of Gu et al. (2021) as summarized in Sec. 3.3 clearly show this not to be the case. This inconsistency can be better understood with the following two experiments.
Experimental Verification: IPR Methods Are “More” Localized than Detection The plot in Fig. 2 (b) indicates that regression methods seem to be more localized than detection methods. In particular, for smaller s, A is smaller for detection methods than integral regression methods. To verify, we compare the resulting heatmaps with an idealized Gaussian heatmap of varying standard deviations or σ, and plot the σ which gives the lowest Pearson Chi-square statistic, i.e. the highest similarity (see Fig. 2 (c)). The detailed similarities with different standard deviations can be seen in Appendix A.3.
As expected, the optimal σ decreases as the samples progress from hard to easy; this result is in line with Fig. 2 (b) and shows that heatmaps are more localized for easy samples. However, we also observe that the optimal σ for regression methods are much smaller for the medium and easy cases. We speculate that this difference arises due to detection methods being trained with a Gaussian heatmap of σ = 2 (Xiao et al., 2018; Sun et al., 2019). Integral regression methods, on the other hand, have no such explicit supervision and therefore no restriction on the extent of localization.
Experimental Verification: Extremely Localized Heatmaps Degrade Performance Is it possible that smaller hypothetical σ, i.e. more localized heatmaps cause poor performance? We verify this for detection methods by applying ground truth Gaussian heatmaps of varying σ for training the networks as per Sec. 4.1. We evaluate network performance using Average Precision (AP). A higher AP corresponds to more accurately located joints. Fig. 2 (d) confirms that σ= 2 appears to be optimal. A smaller σ = 1 degrades the performance slightly, and there is a significant drop in the extreme case when σ= 0.5. To verify the impact of σ on regression methods, we add a distribution prior to the heatmap to prevent the shrink of the support region Φ. Specifically, we apply a Kullback-Leibler Divergence (KLDiv) loss between the non-normalized heatmap Ĥ and a Gaussian heatmap with varying σ. From Table 1, we see that the regression baseline is worse than detection baseline. However, when we add a prior on the heatmap distribution to encourage a sufficiently large Φ, i.e. (+σ=1, 2, 3), performance improves. At the optimal +σ=2, IPR becomes competitive with the detection baseline with the optimal and even exceeds. Most importantly, extreme values σ causes performance to degenerate, especially in the case of the extremely small +σ = 0.5. We also demonstrate the performance and localization extent of easy/medium and hard cases by AP and activation sum A(s= 2). After adding priors (e.g. , +σ = 2), A(s= 2)e/m decreases from 0.65 to 0.54, indicating the heatmaps are less localized, and performance APe/m is significantly improved. We provide the details of training of combining the two losses and magnitudes of two gradients for better understanding in A.4. We also provide detailed heatmap shrinkage change in A.5.

heading: 5 ANALYSIS OF SUPERVISION AND LEARNING
text: The difference in heatmap decoding not only influences the final coordinates given same assumed heatmap, but also determines the the gradients that arise from the loss. The gradients in turn influence the learning and thereby generation of the heatmaps themselves. In this section, we explore the gradients of heatmaps with respect to loss function for detection and regression based methods and pinpoint the specific gradient components which slow down the learning of integral regression.

heading: 5.1 HEATMAP GRADIENTS
text: Detection Methods The gradient of Lde (see Eq. 3) with respect to each pixel in the estimated heatmap ĥp is straightforward:
∂Lde ∂ĥp = 2(ĥp − hp). (9)
The gradient in Eq. 9 features the predicted heatmap value ĥp and the ground truth heatmap value hp and provides explicit supervision at every pixel. False positives and false negatives are penalized, by reducing wrong high likelihoods and raising incorrect low likelihoods respectively. Specifically, for each position p in the heatmap, if the predicted ĥp is smaller than the ground truth hp, the gradient is negative and proportionally increases the value in the next iteration and vice versa for predicted ĥp larger than the ground truth.
Integral Regression The gradient of the loss Lre (see Eq. 4) with respect to ĥp can be estimated based on the chain rule (see detailed derivation in Appendix A.6) as
∇p := ∂Lre ∂ĥp = ∂Lre ∂Ĵ ∂Ĵ ∂h̃p ∂h̃p ∂ĥp = β h̃p︸︷︷︸ ∇1 (value factor) (s(Ĵx − Jx)(i− Ĵx) + s(Ĵy − Jy)(j − Ĵy))︸ ︷︷ ︸ ∇2 (location factor) . (10)
In Eq. 10, the gradient is into two terms of interest: ∇1 and∇2 which we name the value factor and location factor respectively. The nature of the value and location factors give strong indications to the learning process for integral regression methods:
1. The gradient ∇p is proportional to the normalized predicted heatmap value h̃p as given in the value factor. This factor makes it prone to gradient vanishing wherever locations p have small heatmap predictions.
2. the location factor is a linear combination of i and j, which presents a linear plane where far away points like corners are more likely to have large magnitudes.
Though these two points do not prevent the loss from decreasing during learning, they slow down the speed of training. We further elaborate on this in the following section.

heading: 5.2 A DETAILED ANALYSIS OF THE IPR HEATMAP
text: We now conduct a detailed analysis of the gradients presented in Eq. 10 and show the typical cases that arise during learning. When referring to Ω, the domain of the heatmap, we use a coordinate system with the origin at the upper-left corner and increasing coordinates moving to the lower right corner (see Fig. 2 (a)). For simplicity of discussion, we assume that the ground truth joint Jgt is located somewhere in the lower-right quadrant of Ω, though the analysis holds for Jgt in all other quadrants of Ω analogously. The visualizations of the four scenarios can be viewed in the left panel of Fig. 3.
We base our analysis on the assumption that larger magnitude of ∇p will have larger influence on the weights of the network and effect greater change at the location p in the next iteration, i.e.
hn+1p ≈ hnp − γ∇np, (11)
where γ represents a step size or learning rate (see sketch of proof in Appendix A.7) and n denotes the update iteration. As the value and location factors have different contributions on the gradient from Eq. 10 and these are in turn affected by the size s and location µ of Φ, we can define four characteristic cases of Φ and outline how learning is affected.
(1) s is large, Φ has uniformly random values, e.g. a randomly initialized heatmap at the start of training. As h̃p is in a similar range for all p, the value factor ∇1 (see Eq. 10) of all pixels are approximately similar. As such, the distinction between one pixel’s gradient versus another is determined by the location factor ∇2. Given that the ground truth coordinates (Jx, Jy) is in the lower-right quadrant, as long as (Jx, Jy) are both greater than the predicted coordinate (Ĵx, Ĵy),∇p has a gradation that increases progressively towards the bottom right corner. Generally speaking, the location factor pushes the activations of the heatmap towards the corner of the correct quadrant as corners own the largest gradients in the resultant linear plane of location factor∇2. (2) s is small, µ is far from the ground truth J, e.g. when µ is in the upper left quadrant of Ω while the ground truth Jgt is at the bottom right. A typical scenario of this case is if the heatmap activates around the left ankle for the ground truth right ankle. In this case, for the pixels outside of Φ, the value factor approaches zero, i.e. h̃p −→ 0 and pushes∇p towards zero and the pixels receive
very limited gradient updates. For pixels in Φ, the values decrease gradually until the value factor outside of Φ no longer dominates, i.e. they reach the same scale, at which point the heatmap returns to case (1).
(3) s is small, µ is in the corner of the same quadrant as the ground truth Jgt. Initially, this case is similar with case (2). In the process of all the elements becoming the same scale, which also means the activations moves towards the diagonal side, the prediction approaches the ground truth coordinate.
(4) s is small, µ close to ground truth Jgt. This scenario occurs when the model is reasonably trained and can roughly localize the joint. As the gradient at the ground truth pixel Jgt is always negative or zero, i.e. ,
∇(Jx,Jy) = βh̃(Jx,Jy)(−s(Ĵx − Jx)(Ĵx − Jx)− s(Ĵy − Jy)(Ĵy − Jy)). (12) This non-positive value guides the network to predict a large heatmap response at the ground truth location, i.e. a large h̃(Jx,Jy); this in turn increases the gradient ∇(Jx,Jy). This property makes the network tend to predict few extraordinary large pixels, thus shrinking the support region Φ and leads to a very small σ in Sec. 4.1.
However, for detection based methods, the ground truth heatmaps have already explicitly point out whether this pixel should be activated or not in the heatmap. Combined with its effective pixel-wise loss, the network learns faster than regression based methods. We use experiments to validate in the following section.

heading: 5.3 EXPERIMENTAL VERIFICATION
text: Idealized Sample We start by considering the case of one sample for a single joint and visualize the progression of a (64, 48) heatmap as it gets updated by Eq. 11. Note that while ∇np can be calculated mathematically by Eq. 10, however we use autograd in Pytorch to obtain the gradients which we verify to be equivalent. We manually initialize the four cases from Sec. 5.2 by randomized initialization, Gaussian circle centered at some upper left point, linear plane at lower right quadrant, and Gaussian circle centered at ground truth location with σ=2. The progression of the differently initialized heatmaps can be observed in Fig. 3.
Mathematically, it is clear from Eq. 2 that the resulting joint coordinate location from IPR can align with the ground truth joint even if the underlying heatmap does not follow a localized heatmap model. This is also illustrated in the first three cases of Sec. 5.2. However, we posit that during realworld training, over the thousands of training samples observed, over the many epochs, the network can only consistently lower the loss if the heatmap H̃ is learned to represent P (J|I). As such, the activation region Φ will be correctly localized over the corresponding semantic region in the image. Instead, what slows down convergence is the learning of Φ, or rather, the lack of direct guidance to yield a correct Φ as per detection methods. For the same four cases, we visualize the updated heatmap of Eq. 11 from the gradients defined by the detection loss in Eq. 9. Unlike regression,
each case leads consistently to the same localized Φ centered on the ground truth joint coordinate, regardless of the initialization.
Real World Samples The four cases listed in Sec. 5.2 are not so clearly observable in real-world training as the output is based on the optimization results of batch-wise training on an actual network. To verify the results, we compare the training progression for a detection and IPR network under the experimental setting described in Sec. 4.1. Generally, we compare the training speeds of both detection and regression in Fig. 4 (b), which shows the slower convergence of regression method.
We visualize for one sample from the MSCOCO validation set the heatmap at epochs 1 and 10 to compare the progression of training for detection and integral regression. Fig. 4 (a) shows the heatmaps of the “left eye” and “right ankle” joints. For regression, after one epoch, the activations are widespread over a quarter to half the heatmap. This aligns roughly to a mix of Case 1 and 3 where activations are in the correct quadrant but not yet localized to a small region of support. After 10 epochs of training, predictions for the left eye (which is actually occluded, making it a “hard” sample) are localized with several activations of approximately the same value (all shown as red). For the right ankle (which is clearly present and is an “easy” sample), the network has predicts heatmap values that have collapsed spatially and dominate on one or two pixel locations.
In contrast, the detection method presents a well-localized heatmap after only one epoch of training. While there is some activation on the left ankle in epoch 1, this is unlikely to cause errors for the argmax decoding. After 10 epochs, all activations are centered around the correct right ankle.

heading: 6 CONCLUSION
text: In this paper, we dive deeper into the integral pose regression method. We explore the difference between detection and regression based methods based on the heatmap activations and gradients. Specifically, we find that the shrinkage of heatmaps of regression method is the main cause of lower accuracy. Furthermore, we investigate the gradients of IPR by giving theoretical evidence, toy problem and real world illustrations to show that the learning of IPR suffers from implicit supervisory signal. As a result, we propose a simple method based on our analysis to improve the performance of integral methods.

heading: A APPENDIX
text: A.1 EXTENSION OF FIG. 2 (B)
We provide a extended plot of Fig. 2 (b), which includes both detection and regression on easy, medium, and hard cases in Fig. 5. We also provide the datapoint in Table. 2.
A.2 PROOF FOR EQ. 7>EQ. 8 IN SEC. 4.2
In this subsection, we give detailed proof for Eq. 7>Eq. 8, which means∑ p∈Φ(µ,s) w(p)||Jgt − p||2 > ||Jgt − µ||2,where ∑ p∈Φ(µ,s) w(p) = 1. (13)
In this way, this equation can be rewritten as∑ p∈Φ(µ,s) w(p)||Jgt − p||2 > ∑ p∈Φ(µ,s) w(p)||Jgt − µ||2. (14)
As the distribution of Φ(µ, s) is radially symmetric, One sufficient but unnecessary proof is that for any symmetric pair {p1(µx − a, µy − b),p2(µx + a, µy + b)}, it satisfies
√ (x− µx − a)2 + (y − µy − b)2 + √ (x− µx + a)2 + (y − µy + b)2 > 2 √ (x− µx)2 + (y − µy)2.
(15)
According to Triangle Inequality in normed vector space, we arrive at ||u||2 + ||v||2 > ||u + v||2. (16)
When we set u = (x−µx−a, y−µy− b) and v = (x−µx +a, y−µy + b), Eq. 14 is established.
A.3 PEARSON CHI-SQUARE VALUE ON DIFFERENT CASES
We report the detailed Pearson Chi-square value between heatmaps of both detection and regression in different scenarios and Gaussian templates with different σs in Table. 3, 4, 5. We set the threshold of A(s) > 0.8 to show that the support region can represent the whole heatmaps. Therefore, for easy and medium case, s=4; while for hard cases, s=8.
A.4 DETAILS OF COMBINING PRIOR LOSS AND INTEGRAL LOSS
In this subsection, we will demonstrate the influence of weights of prior loss and integral loss, and the magnitudes of two gradients regarding the shared heatmaps during the training.
Let assume wp and wi are weights of the prior and integral losses respectively. We consider the ratio wpLp : wi · λLi where λ=10−2 to set the two approximately equal magnitudes at the initial epoch. From Table 8, we can see that the AP heavily depends on the value of σ but for some fixed σ (within each row), the ratio of loss weightings has less impact and results are stable within 1 − 2%, with larger influence only at the extremely settings of 100 : 1 or 1 : 100. We also provide the ratio of gradient magnitudes in Table 7 to help explain why the weightings do not have much impact. Table 7 gives the mean ratio gpgi where gp and gi are the gradient magnitudes of the prior loss and integral loss respectively over the course of training. A column of 2/1/0/-1/-2 means 102/101/100/10−1/10−2 ratio in gradients over the epochs. The table shows that regardless of the weighting, the gradient ratio decreases over the course of training (i.e. , dropoff with each column). Apart from an extremely high weighting of the prior, i.e. 100:1 in column 1, the prior influences training only in the initial 10 epochs. After epoch 10, the gradient ratio is around the same scale or smaller.
A.5 DETAILS OF HEATMAP SHRINKAGE CHANGE
In order to check whether the added prior influences the spread of the heatmaps, we investigate the shrinkage change of heatmaps by separating the samples. We separate the samples based on easy/medium vs. hard cases where the former generally produces localized heatmaps while the latter has dispersed heatmaps as per Fig. 5; the alternative is to separate by thresholding the activation sum in a fixed area (Eq. 6). We consider activation sumA(1) > 0.34 of the integral regression baseline as
localized, otherwise as dispersed (threshold 0.34 selected based on Table 2). We tally the activation sums for the localized vs. dispersed heatmaps without the prior vs. adding prior with different σs. Activation sums A(s)e/m, A(s)h, A(s)l, A(s)h, and Average Precision APe/m, APh, APl, APh indicates the extent of localization and performance of the two conditions. An increase in activation sum indicates further heatmap localization; a decrease indicates dispersion. The ”No prior” row is the baseline integral pose regression.
From Table 6, both separation presents the same trend. For localized cases, forcing σ to be degenerately small or too large (σ= 0.5 or σ= 4) increases/decreases the activation sum over the baseline, thereby confirming the effect of the prior as intended. The change in AP follows, i.e. decrease when σ incompatible from being too small or too large. For dispersed cases, the prior shrinks the activation sum as expected. Like the localized case, there is an optimal range for which improvement in AP is better; degenerate σ will decrease the AP from the baseline.
A.6 DERIVATION OF EQ. 10
Eq. 10 is obtained by the chain rule. Therefore, we list the value of each term as follows
∂Lre
∂Ĵ = [ s(Ĵx − Jx) s(Ĵy − Jy) ] , ∂Ĵ ∂h̃p = (i+ j), ∂h̃p ∂ĥq = { βh̃q(1− h̃q), q = p −h̃ph̃q, q 6= p , (17)
where q = (u, v) can be any pixel in Ω. Combining the terms, gradients at p can be calculated by
∂Lre ∂ĥp = β((Ĵx − Jx)i+ (Ĵy − Jy)j)h̃p(1− h̃p)− ∑ q∈Ω, q6=p β((Ĵx − Jx)u+ (Ĵy − Jy)v)h̃qh̃p. (18)
Eq. 10 can be obtained by rearranging the terms of Eq. 18.
A.7 DERIVATION OF EQ. 11
Let suppose Hn = g(wn|I) where Hn is the heatmap of given image I at the n-th iteration, g(wn) is the function of generating the heatmap, i.e. , backbone network, with weights at the n-th iteration. According to taylor expansion, we arrive at
g(wn+1) = g(wn) + g
′ (wn)
1! (wn+1 −wn) +
g ′′ (wn)
2! (wn+1 −wn)2 + . . . , (19)
where wn+1−wn=−α∇g(w)=−α ∂L∂Hg ′ (wn) according to gradient descent, and (wn+1−wn)i is negligible compared with (wn+1−wn) when i>1. Therefore, Eq. 19 can be rewritten as
Hn+1 = g(wn+1) ≈ g(wn) + g ′ (wn)(wn+1 −wn) = Hn − α
∂L ∂H ||g ′ (wn)||2. (20)
Therefore, for each pixel hp in H, the update rule is
hn+1p ≈ hnp − γ∇np, (21)
where γ represents α‖g′(wn)‖, hnp and ∇np denote heatmap and gradient value located at p in the heatmap at nth iteration. h0p denotes the original heatmap.
A.8 VISUALIZATIONS OF REAL APPLICATIONS

Review summaries:
This paper investigates the performance inconsistency of integral pose regression (IPR) methods on 'easy' pose samples and 'hard' pose samples (v.s. argmax methods). The authors find that the heatmap shrinkage caused by the IPR methods leads to the lower accuracy on 'easy' samples than argmax based methods. They give some toy examples and intuitively demonstrate how the shrinkage of heatmaps is caused by the gradient of the IPR method. They further propose a heatmap distribution prior loss to mitigate the shrinkage.

Heatmap detection-based method and integral regression-based method are widely used in the pose estimation task. The paper gave a detailed comparison and theoretical analysis between these two methods. A unified model is proposed to compare these two methods. Comprehensive experiments are conducted to support the theoretical analysis. 

This paper is an in-depth study of the Integral Pose Regression method that is very common for Keypoint Localization. Integral Pose Regression has been adopted by many methods, and it is probably one of the two main paradigms for Keypoint Localization, along with Heatmap Regression. Many works have compared the two approaches, and in some cases (e.g., Gu et al, 2021) they provide a more detailed empirical comparison, identifying the strong and weak points of each approach. This paper has a similar goal, but combines theoretical investigation with empirical results to highlight insights about the two methods.