Dive Deeper Into Integral Pose Regression

1 INTRODUCTION . 2D human pose estimation aims to detect the image coordinates of the body and or the hand . In the last few years , detection-based methods ( Newell et al. , 2016 ; Xiao et al. , 2018 ; Sun et al. , 2019 ; Li et al. , 2019 ) and integral pose regression ( IPR ) methods ( Sun et al. , 2018 ; Iqbal et al. , 2018 ) have emerged as two common paradigms for performing human pose estimation . Both detection-based and IPR methods learn a heatmap representing pixel-wise likelihoods of the joint positions ; the heatmap is learned explicitly for detection-based methods , but remains implicit or “ latent ” for IPR . To decode the heatmaps to joint coordinates , detection methods use an argmax , while IPR methods take the expected value . As the expectation operation is differentiable , IPR has the benefit of being end-to-end learnable , even though detection methods seem more competitive accuracy-wise ( COC , 2020 ) . Gu et al . ( 2021 ) ’ s recent work showed a curious performance difference between IPR and detection methods . With a nuanced split over the evaluation set , IPR outperforms detection when test samples are “ harder ” , i.e . , with fewer keypoints present in the scene , under higher occlusion , and lower bounding box resolutions . Given that both detection and IPR methods work with fully convolutional feed-forward architectures , this naturally begs the question - what are the reasons behind this performance difference ? Why is it that IPR , which otherwise seems to lag behind detection methods in performance ( COC , 2020 ) , can excel on these hard cases ? This serves as the motivation for our closer study and analysis of integral pose regression . Detection and IPR methods differ in two aspects ( see Fig . 1 for an overview ) in the forwards and backwards pass . In the forwards pass , the heatmap is decoded with an argmax for detection versus a softmax normalization and expectation for IPR . In the backwards pass , detection methods are supervised with an explicitly defined Gaussian heatmap centered on the ground truth joint , while IPR is supervised by the joint coordinates directly . Through detailed theoretical analysis and experimentation on the decoding and back-propagation process , we make following findings and contributions : 1 . We propose a unified model of the heatmap to interpret and compare detection and integral regression methods . We verify the model experimentally and show that as samples progress from hard to easy , the region of activation on the heatmap decreases for both detection and integral regression methods . 2 . We demonstrate experimentally that degenerately small regions of activation degrades the accuracy of both detection and integral regression methods . 3 . Integral regression methods , by decoding the heatmap with an expectation operation , should result in a lower expected EPE than detection methods that decode with an argmax operation . In practice , this can be observed only for hard samples due to the shrinkage of the active region on the heatmap . 4 . Direct supervision with the joint coordinates in IPR , although end-to-end , suffers from gradient vanishing and provides less spatial cues for learning the heatmap than the explicit heatmap supervision of detection methods . As a result , the training of IPR is more inefficient and slower to converge than detection methods . In conclusion , our findings serve as insight to understand IPR which has a better decoding method theoretically and shows the potential to surpass the performance of detection by our experiments . 2 RELATED WORK . Since the concept of “ heatmap ” is proposed in ( Tompson et al. , 2014 ) , detection-based methods have been top performers for human pose . Existing detection-based works emphasize to extract high-quality multi-resolution features . Xiao et al . ( 2018 ) proposed to adapt ResNet with deconvolution layers while Hourglass ( Newell et al. , 2016 ) and Cascaded Pyramid Network ( CPN ) ( Li et al. , 2019 ) introduced cascaded network architectures with coarse-to-fine design paradigm . HighResolution Network ( HRNet ) ( Sun et al. , 2019 ) follows the coarse-to-fine paradigm and further improves the performance by adapting more dense connection among different resolution representations . IPR ( Sun et al. , 2018 ) introduced a competitive regression-based framework , but exceeded by more advanced detection-based works ( Xiao et al. , 2018 ; Sun et al. , 2019 ) . Numerical regression-based methods , which directly regress the joint coordinates , are commonly used in facial land-mark detection ( Feng et al. , 2018 ; Zhu et al. , 2020 ) . These methods however are not accurate for human pose because they fail to capture spatial knowledge . As a follow-up , IPR tries to merge the spatial knowledge from the heatmap implicitly , hence the “ latent ” heatmap . IPR methods are especially preferred in hand pose estimation ( Spurr et al. , 2020 ; Yang et al. , 2021 ) but are still less common for the human pose estimation ( Sun et al. , 2018 ; Nibali et al. , 2018 ) . Recently , two parallel lines of work have developed to further understand the behaviour of detection and IPR methods respectively . For detection-based methods , Huang et al . ( 2020 ) revealed a heatmap bias caused by the inconsistency of coordinate system transformations and solved it by redesigning the transformations during data processing . Moreover , as the predicted heatmaps during inference may violate the Gaussian assumption and worsen the performance , Huang et al . ( 2020 ) proposed to find the optimal Gaussian location based on the prediction and Zhang et al . ( 2020 ) directly modulated the predicted Gaussian distribution for decoding . For IPR methods , Nibali et al . ( 2018 ) experimentally compared different heatmap regularizers , heatmap normalization schemes and loss functions . They find using Jensen-Shannon regularization on the heatmap with a softmax normalization and L1 loss achieves the best performance for IPR . More recently , Gu et al . ( 2021 ) performed a systematic comparison of detection vs. IPR methods with a common backbone and discovered the performance advantage of IPR on the “ hard ” samples . Furthermore , Gu et al . ( 2021 ) revealed a bias that arises from taking the expectation after the softmax normalization ; they propose a compensation scheme to mitigate the bias which in turn improves the overall performance of IPR , making them competitive with detection methods . The work ( Zhang et al. , 2020 ; Huang et al. , 2020 ) focus on the explicit heatmaps of detection-based methods and investigate the bias under the Gaussian assumption . However , this is not applicable for IPR methods without the Gaussian assumption . Like ( Nibali et al. , 2018 ; Gu et al. , 2021 ) , our work also investigates the implicit heatmaps of IPR ; the previous works focus on experimental exploration , while we place more emphasis on theoretical analysis to pinpoint underlying causes . 3 PRELIMINARIES ON HUMAN POSE ESTIMATION . In this work , we target the more commonly used ‘ top-down ’ form of pose estimation in which a person detector already provides a cropped image I of the person . For simplicity , we focus our discussion on one given joint of the K total joints in the body . The pose estimation model outputs a heatmap Ĥ ∈ RM×N where M and N are the dimensions of the spatial heatmap . Typically , M and N are scaled down by a factor of 4 from the original input dimensions of I ( Xiao et al. , 2018 ; Sun et al. , 2019 ) . The heatmap Ĥ represents a ( discrete ) spatial likelihood P ( J|I ) , where J ∈ R1×2 is the 2D coordinates of that joint . In practice , all K heatmaps are predicted simultaneously by the same network , where each joint is one channel . In both detection and IPR methods , the coordinates Ĵ are decoded from Ĥ ; where the two methods differ is the manner of decoding ( see Sec . 3.1 ) and the form of supervision applied ( see Sec . 3.2 ) . 3.1 HEATMAP DECODING : MAX VERSUS EXPECTED VALUE . Detection Methods apply an argmax on Ĥ indexed by p to estimate the joint coordinates Ĵde : Ĵde = arg max p Ĥ ( p ) , ( 1 ) Taking an argmax can be interpreted as taking a maximum likelihood on the heatmap Ĥ , assuming that Ĥ is proportional to the likelihood . In practice , the final Ĵde value is determined as a linear combination of the highest and second-highest response on Ĥ as a way to account for quantization effects in the discrete heatmap ( Newell et al. , 2016 ) . A more recent work , DARK ( Zhang et al. , 2020 ) approximates the true prediction by a Taylor series evaluated at the maximum activation of the heatmap and shows this to be more accurate . Integral Pose Regression applies an expectation operation on Ĥ to give a mean estimate of the joint coordinates . To do so , the heatmap must first be normalized to sum up to 1 ; the most common and effective ( Nibali et al. , 2018 ) approach is to apply a softmax normalization . Afterwards , the predicted joint Ĵre with x and y components Ĵx and Ĵy1 is determined by taking the expectation on the normalized heatmap H̃ with elements h̃p at location p : Ĵre = [ Ĵx Ĵy ] = ∑ p∈Ω p · h̃p where h̃p = eβĥp∑ ( p′ ) ∈Ω eβĥp′ . ( 2 ) Here , Ω is the domain of the heatmap and β is a scaling parameter used in the softmax normalization . Note that softmax normalization assigns a non-zero value to all pixels in H̃ , even if it was originally zero in Ĥ . These values also contribute to the expected value , resulting in a center-biased estimated joint coordinate Ĵre ( Gu et al. , 2021 ) . The smaller the β is , the stronger the bias is . Although Gu 1For clarity , we drop the subscript ‘ re ’ , as we refer to the individual components for IPR only . et al . ( 2021 ) propose a compensation scheme , for the purposes of our analysis we will assume that β is sufficiently large that the impact of the bias is negligible . 3.2 SUPERVISION : EXPLICIT HEATMAP VS . GROUND TRUTH COORDINATES . Detection Methods are learned by providing supervision on the heatmap . The ground truth H is given as a circular Gaussian , with a mean centered at the ground truth joint coordinate ( see Fig . 1 ) . The loss applied is a pixel-wise MSE between the ground truth hp and the predicted ĥp : Lde = ||H− Ĥ||22 = ∑ p∈Ω ( hp − ĥp ) 2 , ( 3 ) where Ω is the domain of the heatmap . As the loss is defined in terms of the heatmap and not the actual variable of interest , i.e . the predicted joints , detection based methods are not end-to-end in their learning and this is often cited as a drawback ( Sun et al. , 2018 ; Zhang et al. , 2020 ) . Integral Pose Regression defines a loss based on the difference between the prediction Ĵre and ground truth joint location J . The L1 loss empirically performs better than the L2 ( Sun et al. , 2018 ) : Lre = ‖Jgt − Ĵre‖1 = ( |Ĵx − Jx|+ |Ĵy − Jy| ) . ( 4 ) IPR are end-to-end methods because they provide supervision at the joint level . As the estimated H is learned only implicitly , some works refer to the heatmap as “ latent ” ( Iqbal et al. , 2018 ) .

Summary: Heatmap detection-based method and integral regression-based method are widely used in the pose estimation task. The paper gave a detailed comparison and theoretical analysis between these two methods. A unified model is proposed to compare these two methods. Comprehensive experiments are conducted to support the theoretical analysis.