Dive Deeper Into Integral Pose Regression

1 INTRODUCTION . 2D human pose estimation aims to detect the image coordinates of the body and or the hand . In the last few years , detection-based methods ( Newell et al. , 2016 ; Xiao et al. , 2018 ; Sun et al. , 2019 ; Li et al. , 2019 ) and integral pose regression ( IPR ) methods ( Sun et al. , 2018 ; Iqbal et al. , 2018 ) have emerged as two common paradigms for performing human pose estimation . Both detection-based and IPR methods learn a heatmap representing pixel-wise likelihoods of the joint positions ; the heatmap is learned explicitly for detection-based methods , but remains implicit or “ latent ” for IPR . To decode the heatmaps to joint coordinates , detection methods use an argmax , while IPR methods take the expected value . As the expectation operation is differentiable , IPR has the benefit of being end-to-end learnable , even though detection methods seem more competitive accuracy-wise ( COC , 2020 ) . Gu et al . ( 2021 ) ’ s recent work showed a curious performance difference between IPR and detection methods . With a nuanced split over the evaluation set , IPR outperforms detection when test samples are “ harder ” , i.e . , with fewer keypoints present in the scene , under higher occlusion , and lower bounding box resolutions . Given that both detection and IPR methods work with fully convolutional feed-forward architectures , this naturally begs the question - what are the reasons behind this performance difference ? Why is it that IPR , which otherwise seems to lag behind detection methods in performance ( COC , 2020 ) , can excel on these hard cases ? This serves as the motivation for our closer study and analysis of integral pose regression . Detection and IPR methods differ in two aspects ( see Fig . 1 for an overview ) in the forwards and backwards pass . In the forwards pass , the heatmap is decoded with an argmax for detection versus a softmax normalization and expectation for IPR . In the backwards pass , detection methods are supervised with an explicitly defined Gaussian heatmap centered on the ground truth joint , while IPR is supervised by the joint coordinates directly . Through detailed theoretical analysis and experimentation on the decoding and back-propagation process , we make following findings and contributions : 1 . We propose a unified model of the heatmap to interpret and compare detection and integral regression methods . We verify the model experimentally and show that as samples progress from hard to easy , the region of activation on the heatmap decreases for both detection and integral regression methods . 2 . We demonstrate experimentally that degenerately small regions of activation degrades the accuracy of both detection and integral regression methods . 3 . Integral regression methods , by decoding the heatmap with an expectation operation , should result in a lower expected EPE than detection methods that decode with an argmax operation . In practice , this can be observed only for hard samples due to the shrinkage of the active region on the heatmap . 4 . Direct supervision with the joint coordinates in IPR , although end-to-end , suffers from gradient vanishing and provides less spatial cues for learning the heatmap than the explicit heatmap supervision of detection methods . As a result , the training of IPR is more inefficient and slower to converge than detection methods . In conclusion , our findings serve as insight to understand IPR which has a better decoding method theoretically and shows the potential to surpass the performance of detection by our experiments . 2 RELATED WORK . Since the concept of “ heatmap ” is proposed in ( Tompson et al. , 2014 ) , detection-based methods have been top performers for human pose . Existing detection-based works emphasize to extract high-quality multi-resolution features . Xiao et al . ( 2018 ) proposed to adapt ResNet with deconvolution layers while Hourglass ( Newell et al. , 2016 ) and Cascaded Pyramid Network ( CPN ) ( Li et al. , 2019 ) introduced cascaded network architectures with coarse-to-fine design paradigm . HighResolution Network ( HRNet ) ( Sun et al. , 2019 ) follows the coarse-to-fine paradigm and further improves the performance by adapting more dense connection among different resolution representations . IPR ( Sun et al. , 2018 ) introduced a competitive regression-based framework , but exceeded by more advanced detection-based works ( Xiao et al. , 2018 ; Sun et al. , 2019 ) . Numerical regression-based methods , which directly regress the joint coordinates , are commonly used in facial land-mark detection ( Feng et al. , 2018 ; Zhu et al. , 2020 ) . These methods however are not accurate for human pose because they fail to capture spatial knowledge . As a follow-up , IPR tries to merge the spatial knowledge from the heatmap implicitly , hence the “ latent ” heatmap . IPR methods are especially preferred in hand pose estimation ( Spurr et al. , 2020 ; Yang et al. , 2021 ) but are still less common for the human pose estimation ( Sun et al. , 2018 ; Nibali et al. , 2018 ) . Recently , two parallel lines of work have developed to further understand the behaviour of detection and IPR methods respectively . For detection-based methods , Huang et al . ( 2020 ) revealed a heatmap bias caused by the inconsistency of coordinate system transformations and solved it by redesigning the transformations during data processing . Moreover , as the predicted heatmaps during inference may violate the Gaussian assumption and worsen the performance , Huang et al . ( 2020 ) proposed to find the optimal Gaussian location based on the prediction and Zhang et al . ( 2020 ) directly modulated the predicted Gaussian distribution for decoding . For IPR methods , Nibali et al . ( 2018 ) experimentally compared different heatmap regularizers , heatmap normalization schemes and loss functions . They find using Jensen-Shannon regularization on the heatmap with a softmax normalization and L1 loss achieves the best performance for IPR . More recently , Gu et al . ( 2021 ) performed a systematic comparison of detection vs. IPR methods with a common backbone and discovered the performance advantage of IPR on the “ hard ” samples . Furthermore , Gu et al . ( 2021 ) revealed a bias that arises from taking the expectation after the softmax normalization ; they propose a compensation scheme to mitigate the bias which in turn improves the overall performance of IPR , making them competitive with detection methods . The work ( Zhang et al. , 2020 ; Huang et al. , 2020 ) focus on the explicit heatmaps of detection-based methods and investigate the bias under the Gaussian assumption . However , this is not applicable for IPR methods without the Gaussian assumption . Like ( Nibali et al. , 2018 ; Gu et al. , 2021 ) , our work also investigates the implicit heatmaps of IPR ; the previous works focus on experimental exploration , while we place more emphasis on theoretical analysis to pinpoint underlying causes . 3 PRELIMINARIES ON HUMAN POSE ESTIMATION . In this work , we target the more commonly used ‘ top-down ’ form of pose estimation in which a person detector already provides a cropped image I of the person . For simplicity , we focus our discussion on one given joint of the K total joints in the body . The pose estimation model outputs a heatmap Ĥ ∈ RM×N where M and N are the dimensions of the spatial heatmap . Typically , M and N are scaled down by a factor of 4 from the original input dimensions of I ( Xiao et al. , 2018 ; Sun et al. , 2019 ) . The heatmap Ĥ represents a ( discrete ) spatial likelihood P ( J|I ) , where J ∈ R1×2 is the 2D coordinates of that joint . In practice , all K heatmaps are predicted simultaneously by the same network , where each joint is one channel . In both detection and IPR methods , the coordinates Ĵ are decoded from Ĥ ; where the two methods differ is the manner of decoding ( see Sec . 3.1 ) and the form of supervision applied ( see Sec . 3.2 ) . 3.1 HEATMAP DECODING : MAX VERSUS EXPECTED VALUE . Detection Methods apply an argmax on Ĥ indexed by p to estimate the joint coordinates Ĵde : Ĵde = arg max p Ĥ ( p ) , ( 1 ) Taking an argmax can be interpreted as taking a maximum likelihood on the heatmap Ĥ , assuming that Ĥ is proportional to the likelihood . In practice , the final Ĵde value is determined as a linear combination of the highest and second-highest response on Ĥ as a way to account for quantization effects in the discrete heatmap ( Newell et al. , 2016 ) . A more recent work , DARK ( Zhang et al. , 2020 ) approximates the true prediction by a Taylor series evaluated at the maximum activation of the heatmap and shows this to be more accurate . Integral Pose Regression applies an expectation operation on Ĥ to give a mean estimate of the joint coordinates . To do so , the heatmap must first be normalized to sum up to 1 ; the most common and effective ( Nibali et al. , 2018 ) approach is to apply a softmax normalization . Afterwards , the predicted joint Ĵre with x and y components Ĵx and Ĵy1 is determined by taking the expectation on the normalized heatmap H̃ with elements h̃p at location p : Ĵre = [ Ĵx Ĵy ] = ∑ p∈Ω p · h̃p where h̃p = eβĥp∑ ( p′ ) ∈Ω eβĥp′ . ( 2 ) Here , Ω is the domain of the heatmap and β is a scaling parameter used in the softmax normalization . Note that softmax normalization assigns a non-zero value to all pixels in H̃ , even if it was originally zero in Ĥ . These values also contribute to the expected value , resulting in a center-biased estimated joint coordinate Ĵre ( Gu et al. , 2021 ) . The smaller the β is , the stronger the bias is . Although Gu 1For clarity , we drop the subscript ‘ re ’ , as we refer to the individual components for IPR only . et al . ( 2021 ) propose a compensation scheme , for the purposes of our analysis we will assume that β is sufficiently large that the impact of the bias is negligible . 3.2 SUPERVISION : EXPLICIT HEATMAP VS . GROUND TRUTH COORDINATES . Detection Methods are learned by providing supervision on the heatmap . The ground truth H is given as a circular Gaussian , with a mean centered at the ground truth joint coordinate ( see Fig . 1 ) . The loss applied is a pixel-wise MSE between the ground truth hp and the predicted ĥp : Lde = ||H− Ĥ||22 = ∑ p∈Ω ( hp − ĥp ) 2 , ( 3 ) where Ω is the domain of the heatmap . As the loss is defined in terms of the heatmap and not the actual variable of interest , i.e . the predicted joints , detection based methods are not end-to-end in their learning and this is often cited as a drawback ( Sun et al. , 2018 ; Zhang et al. , 2020 ) . Integral Pose Regression defines a loss based on the difference between the prediction Ĵre and ground truth joint location J . The L1 loss empirically performs better than the L2 ( Sun et al. , 2018 ) : Lre = ‖Jgt − Ĵre‖1 = ( |Ĵx − Jx|+ |Ĵy − Jy| ) . ( 4 ) IPR are end-to-end methods because they provide supervision at the joint level . As the estimated H is learned only implicitly , some works refer to the heatmap as “ latent ” ( Iqbal et al. , 2018 ) .

Summary: This paper investigates the performance inconsistency of integral pose regression (IPR) methods on 'easy' pose samples and 'hard' pose samples (v.s. argmax methods). The authors find that the heatmap shrinkage caused by the IPR methods leads to the lower accuracy on 'easy' samples than argmax based methods. They give some toy examples and intuitively demonstrate how the shrinkage of heatmaps is caused by the gradient of the IPR method. They further propose a heatmap distribution prior loss to mitigate the shrinkage.