Continuous Heatmap Regression for Pose Estimation via Implicit Neural Representation

Authors: shengxiang  hu
huaijiang  sun
dong  wei
xiaoning  sun
jin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 10:56:04

Abstract: Heatmap regression has dominated human pose estimation due to its superior performance and strong generalization. To meet the requirements of traditional explicit neural networks for output form, existing heatmap-based methods discretize the originally continuous heatmap representation into 2D pixel arrays, which leads to performance degradation due to the introduction of quantization errors. This problem is significantly exacerbated as the size of the input image decreases, which makes heatmap-based methods not much better than coordinate regression on low-resolution images. In this paper, we propose a novel neural representation for human pose estimation called NerPE to achieve continuous heatmap regression. Given any position within the image range, NerPE regresses the corresponding confidence scores for body joints according to the surrounding image features, which guarantees continuity in space and confidence during training. Thanks to the decoupling from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution during inference without retraining, which easily achieves sub-pixel localization precision. To reduce the computational cost, we design progressive coordinate decoding to cooperate with continuous heatmap regression, in which localization no longer requires the complete generation of high-resolution heatmaps. The code is available at https://github.com/hushengxiang/NerPE.

Main paper: 
 introduction: 
 Human pose estimation (HPE) is a fundamental task in the field of computer vision, which is widely used in various human-centered applications [41, 29, 37, 9, 55, 53] . In multi-person pose estimation, the top-down framework is a mainstream two-stage pipeline, in which the person areas are firstly cropped out and then the body joints within them are located. According to the way to describe the positions of body joints, pose estimation methods can be further divided into coordinate regression [44, 43, 20, 22, 28] and heatmap regression [31, 48, 42, 24, 52] . Due to better performance brought by spatial encoding of heatmap representation, heatmap regression has received more attention than coordinate regression. Recently, most heatmap-based methods [11, 51, 50] focus on the design of network structure but ignore the importance of heatmap generation. Only a few works [26, 23] have noticed the irrationality in the standard heatmap representation widely used in existing methods.In the top-down framework, the ground-truth coordinates of body joints in the original image are mapped to the input plane of a keypoint detector, through the same affine transformation applied to the cropped image patch. To supervise the output of traditional explicit neural networks, the current heatmap generation strategy yields 2D pixel arrays to reflect the spatial distribution of body joints, which means that the originally continuous heatmap representation needs to be discretized as shown in Fig. 1 . Specifically, the center of the Gaussian kernel is placed at body joints to calculate the confidence scores on the grid points as the ground-truth heatmaps. Although this discrete heatmap representation has achieved great success, it introduces quantization errors since body joints after affine transformation may fall anywhere within the image range rather than just at these fixed grid points. For the output of explicit neural networks, the information loss caused by spatial sampling is difficult to compensate through post-processing operations [31, 54] . With the reduction of the input resolution, this problem will be further exacerbated, which makes the performance of heatmap-based methods degraded even worse than some methods based on coordinate regression. Considering that the Gaussian function used for heatmap generation is inherently continuous, it is a natural idea to learn a continuous heatmap representation to replace fixed spatial sampling for HPE.Implicit neural representations (INRs) are proposed to parameterize a variety of continuous signals [30, 8, 7, 47, 39, 46] in computer vision. Unlike explicit neural networks that output specific structures (e.g., mesh, sequence), INRs map an index to its corresponding value, enabling not only continuous representations but also greater flexibility in use. Obviously, by combining INR and heatmap regression, the pose estimation model is able to learn continuous confidence scores for all body joints at any position. Compared to the discrete heatmap representation with a fixed resolution, the introduction of INR avoids the invariant spatial sampling caused by discretization during training and can yield the predicted heatmaps at arbitrary resolution during testing.In this paper, to improve the performance of heatmap-based HPE, we abandon the main culprit of quantization errors, namely discrete heatmap representation. Instead, we propose NerPE to achieve continuous heatmap regression through a novel neural representation. Specifically, we generalize the upsampling factor in discrete sub-pixel convolution [38] to infinity to obtain a continuous upsampling function with respect to 2D coordinate. NerPE learns its continuous heatmap representation at a series of queried positions that are randomly sampled within the image range. For any position, the target likelihood at it is calculated from the target 2D pose and its absolute coordinate, and the estimated likelihood at it is inferred from the local feature vector and its relative coordinate. Since confidence scores are learned at all positions within the image range during training, our method achieves better performance than discrete heatmap-based methods that focus only on fixed grid points. As the resolution of the input image decreases, the superiority of continuous heatmap representation over existing methods becomes more prominent.Limited by the explicit neural representation of existing methods, the heatmap resolution for a given image cannot be changed once the network structure is determined. In contrast, another benefit that INR brings to heatmap regression is that the heatmap resolution is no longer correlated with the image resolution. This means that our method can flexibly generate the predicted heatmaps with different resolutions based on accuracy requirements without retraining. To speed up the inference when high-resolution heatmaps are required, we design a progressive coordinate decoding method. Thanks to the decoupling of INR and spatial resolution, NerPE enables high-precision localization without the need to calculate the complete heatmaps. Specifically, low-resolution heatmaps are first output to determine the approximate locations of body joints. Subsequently, the area near the maximal activation is further retrieved in an iterative manner. Notably, our method can be easily integrated into most heatmap-based methods. The contributions are summarized as follows:• We propose NerPE to avoid the introduction of quantization errors during training and to output the predicted heatmaps at arbitrary resolution during inference. To our knowledge, we are the first to apply implicit neural representations to human pose estimation.• We design a progressive coordinate decoding method to derive the coordinates of body joints from continuous heatmap representation, in which our method achieves high-precision localization with low computational cost through coarse-to-fine retrieval.• We conduct extensive experiments on three pose estimation benchmarks: COCO [25] , MPII [1] , and CrowdPose [21] . The results show that NerPE significantly enhances existing heatmap-based methods and obtains superior performance on low-resolution input images.2 Related Work 
human pose estimation: 
 2D human pose estimation (HPE) aims to locate a series of anatomical keypoints to represent the human pose in the input image. Currently, coordinate regression [3, 22, 28] and heatmap regression [17, 45, 12] are the two main pose estimation paradigms, both of which have received widespread attention. Coordinate regression, also known as direct regression, relies on deep neural networks to explore the mapping between the input image and the coordinates of body joints. To be clear at first, the cause of quantization errors is discretization, not the introduction of heatmap representations this behavior itself. As far as we know, existing heatmap-based methods [48, 42, 24, 52] all belong to discrete heatmap regression, using 2D pixel arrays to describe the spatial distribution of body joints. There are a few works that attempt to compensate for the damage to performance caused by quantization errors. In [31] , the coordinates of body joints are empirically determined as the position of moving a 0.25 pixel from the maximal to second maximal activation. DARK [54] implements Taylor series approximation for heatmap activation to locate body joints based on distribution information. Although these post-processing operations achieve sub-pixel localization precision, subjective assumptions make the inference not so reliable, especially in low-resolution cases. Instead of remedying the shortcomings of discrete heatmap representation, we propose continuous heatmap regression to preserve the continuity of ground-truth heatmaps.In response to the need to model continuous signals, implicit neural representations (INRs) aim to learn a neural function that predicts the corresponding value according to a given index. Encouraged by the success in 3D reconstruction [14, 32, 4] and generation [6, 33, 49] , INR has been extended to some other tasks, including super-resolution [8, 10, 13] , image generation [40, 5, 2] , and video compression [7, 15, 27] . In the paper, to avoid the damage of discretization to heatmap regression, we approximate the continuous Gaussian function by multi-layer perceptrons (MLPs). Given a coordinate within the image range, NerPE is designed to regress the confidence scores of all body joints at that position, in which the heatmap representation is free from the constraints of 2D pixel arrays [11, 51, 50] . As a result, NerPE has the ability to output the predicted heatmaps at arbitrary resolution during inference. In terms of eliminating quantization errors, offset prediction [34, 19] has the same goal as our work, and it aims to perform unbiased estimation for the coordinates of body joints. However, the performance of offset prediction is still affected by the resolution of the distance field. There is complementarity between the two techniques in HPE: offset prediction helps achieve more advanced decoding, and INR makes the distance field continuous. In order to highlight the superiority of INR, we only regress confidence scores in this paper.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of this work and future research in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method: 
 In this section, we first discuss the hazards of discrete heatmap representation and the limitations of existing post-processing operations. Then, we introduce a novel Neural representation for human Pose Estimation (NerPE) and illustrate the superiority of continuous heatmap regression.Given an input image I ∈ R 3×H I ×W I , heatmap-based models output 2D pixel arrays H of size K × H I 4 × W I 4 to reflect the spatial distribution of K body joints. To conform the output form of explicit neural networks, the continuous Gaussian function is discretized in the heatmap generation process, which leads to the introduction of quantization errors and the reduction of positioning accuracy. Currently, there are two post-processing operations [31, 54] that are widely used to mitigate the negative effects of discretization.The standard coordinate decoding method [31] is designed entirely according to experience. Based on the analysis of the model's performance, the position p at which the maximal activation H(c m ) moves a 0.25 pixel towards the second maximum H(c s ) is taken as the final coordinate:EQUATION)The distribution-aware decoding method [54] assumes that the predicted heatmaps still conform to the Gaussian distribution. The Taylor series expansion to the quadratic term is implemented at the position c m of the maximal activation as:EQUATIONwhere H ′ (•) and H ′′ (•) respectively denote the first and second order derivatives of the Gaussian distribution close to the predicted heatmaps. The position p of each body joint is determined by the Gaussian meanµ = c m -(H ′′ (c m )) -1 H ′ (c m ).To achieve sub-pixel localization precision against quantization errors, the above post-processing operations impose subjective assumptions in coordinate decoding. Compared with the limited improvements brought by the above passive compensations, the most direct way to solve quantization errors is to abandon the explicit neural network and learn a continuous heatmap representation.From a structural and functional perspective, general pose estimation models consist of an encoder and a decoder (also called "head" in some works). To reduce the computational cost and increase the receptive field, the image features are downsampled in the encoder. Correspondingly, the decoder increases the output resolution by including upsampling layers such as interpolation, deconvolution or sub-pixel convolution [38] . However, once the network structure is determined, the decoder in existing methods makes the model only output the predicted heatmaps with a fixed resolution. Starting from sub-pixel convolution, we derive its continuous version via implicit neural representation (INR), known as a continuous upsampling function.Using sub-pixel convolution as the final layer, the image features Z of size C × H Z × W Z are mapped into higher resolution heatmaps H of size K × rH Z × rW Z with an upscaling factor r:EQUATIONwhere PS denotes a PixelShuffle operation used to reshape each element along the feature dimension into small 2D pixel arrays (called a cell) corresponding to body joints. According to the spatial prior, the weight W and bias b in Eq. ( 3) are further expressed as:EQUATIONwhere the order of concatenation corresponds to the order of expansion in the PixelShuffle operation. These sub-weights W i,j and sub-biases b i,j can be viewed as the outputs of functions f W (i, j) and f b (i, j) with a 2D index as argument. We use z * to refer to each element in the image features Z, called a local feature vector. The confidence scores at (i, j) relative to z * is given by:EQUATIONIn order to ensure the consistency in description of sub-pixel convolution with different upscaling factors, we normalize the 2D index array composed of integers in {1, 2, . . . , r} into [-1, 1]:EQUATIONAs r increases, the spatial sampling of [-1, 1] × [-1, 1] becomes denser. When r approaches ∞, the discrete grid points become a continuous region covering all resolutions. Therefore, we obtain the expression for the continuous version of sub-pixel convolution H z * (c rel ) = f θ (z * , c rel ), and further replace the relative coordinate c rel with the absolute coordinate c abs to get:EQUATION)where c z * is the center coordinate of cell H z * , and s cell is the window size of cell H z * . Starting from sub-pixel convolution, the conclusion drawn is similar to the local implicit image function in [8] except for the additional normalization step. In terms of network architecture, we only replace the last few layers in existing heatmap-based models with our continuous upsampling function, which fundamentally solves the problem of discretization in training.Obviously, the discrete heatmap representation is a subcase of the proposed NerPE, where only several fixed positions (i.e., grid points) within the image range are trained and predicted. Thanks to the decoupling of heatmap representation and spatial resolution, NerPE can comprehensively fit Gaussian or Laplace functions. The design of our method involves three aspects: continuous heatmap generation, uniform position sampling, and progressive coordinate decoding.Continuous heatmap generation. Since the output of the network is no longer 2D pixel arrays like existing methods, we need to calculate the ground-truth confidence scores for those positions being queried. In our method, the continuous heatmap representations based on Gaussian and Laplace distributions respectively are expressed relative to c abs as: H gt gau (c abs ) = e -(c abs -c gt ) 2where c gt is the ground-truth coordinates of body joints. Since the queried position can be anywhere in the image range, the ground-truth heatmaps are continuous in space and confidence. The loss function of continuous heatmap regression is:Uniform position sampling. For heatmap learning based on INR, the selection of queried positions for training is critical to the model performance. On the one hand, in order to fully take care of the entire heatmap plane, the sampling of queried positions during training should be evenly distributed over the image range. Thus, we divide each cell uniformly into √ N Z × √ N Z regions and perform random sampling within them, as shown in Fig. 2(b) . In each training sample, N Z × H Z × W Z positions are picked out and their ground-truth confidence scores are calculated for supervision.On the other hand, INR acts as a parameterized continuous function, which expects that spatially close positions should have similar confidence scores. Inside each cell, this is easy to implement for deep neural networks. However, at the junctions between cells, the difference of local feature vectors z * and the mutation of relative coordinates c rel make the network tend to produce discontinuous predictions. To solve this problem, we adopt the local ensemble in [8] to perform bilinear interpolation, where the sampling range is expanded to achieve overlapping.Progressive coordinate decoding. The design of heatmap representation has been well discussed above, and more importantly, it ultimately serves keypoint localization. In order to decode the coordinates of body joints through argmax, a straightforward way is to arrange the candidate points within the cells at a specific density, so that NerPE can output the predicted heatmaps with the corresponding upsampling factor r ≥ 1:EQUATIONwhereH (n)cell is the distribution of body joints in the n-th cell of the input image, and C r×r is a relative coordinate matrix to indicate the candidate points. When the heatmap resolution is high enough, argmax is sufficient to deal with quantization errors without extra post-processing operations.In existing methods, the likelihood of body joints at each position is derived simultaneously by an explicit neural network, which means there is no choice but to output the entire predicted heatmaps. In contrast, NerPE can estimate confidence scores based on 2D coordinates in a serial manner thanks to the decoupling of INR from spatial resolution. We propose progressive coordinate decoding to reduce the computational cost by evading the complete generation of predicted heatmaps when the heatmap resolution is high, as shown in Fig. 2(d) . Specifically, we first generate the low-resolution heatmaps to estimate the approximate locations of body joints. Then, the area near the maximal activation is iteratively sub-divided for coarse-to-fine retrieval. This decoding method only needs to calculate additional t × K × (r + p o ) × (r + p o ) positions to achieve an equivalent resolution of r t H Z × r t W Z , where t is the number of iterations and p o is the number of pixels overlapped.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [No] Justification: Although we do not provide open access to our code during review, we will release it and include its URL in the abstract if the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we compare the performance of NerPE and discrete heatmap-based methods on input images of different resolutions. The experimental results show the superiority of continuous heatmap regression and the hazards of discrete heatmap representation, especially in the case of low resolution.Network architecture. We adopt ResNet [16] , HRNet [42] or TokenPose [24] as the backbone network, and resize the extracted image features to 8 × 8. The focus of this work is to provide a new perspective on heatmap regression through implicit neural representation (INR) rather than pursuing extreme performance. For clarity, we use a pure MLP structure as a decoder to implement continuous heatmap regression. Queried position embeddings, derived from 2D coordinates and their sinusoidal encodings, are concatenated with local feature vectors and then fed into the decoder. In each cell corresponding to a local feature vector, we collect N Z = 64 queried positions as training subjects. For continuous heatmap generation, σ in the Gaussian function is set to 0.06, which is close in proportion to the discrete heatmap representation. When not explicitly stated, the size of the predicted heatmaps is 256 × 256 by default. NerPE no longer needs those empirical operations to align flipped heatmaps [48] and shift decoded coordinates [31] .Optimization. In the main experimental results, the training settings of NerPE is consistent with the comparison methods [48, 42, 24] based on discrete heatmap regression. We use the Adam optimizer [18] for training, in which the learning rate is initialized to 1e -3 and decreased to 1e -4 and 1e -5. The data augmentation used includes random rotation, random scale, image flipping, and half body cropping. All our experiments are conducted on an open-source machine learning, PyTorch [35] . Evaluation on MPII. We compare our NerPE with representative discrete heatmap-based methods [42, 54, 23] on the MPII dataset, as shown in Table 3 . At the input size of 128 × 128, our method achieves better performance based on the same backbone HRNet-W32. The higher scores obtained on PCKh@0.1 indicate that NerPE's positioning of body joints is closer to the ground truth.Evaluation on CrowdPose. To evaluate the performance in crowded scenes, we test NerPE on the CrowdPose dataset (see Table 4 ), in which YoloV3 [36] is adopted as the human detector. At the input size of 256 × 192, our method achieves superior performance with 66.9 AP. Thanks to the learned continuous heatmap representation, NerPE delivers performance gains on AP 75 , a more stringent metric. For the low-resolution case, the experimental results show that NerPE brings an improvement of 6.1 AP to HRNet and further expands its lead over discrete heatmap-based methods. Progressive coordinate decoding. In the proposed NerPE, the heatmap resolution is not dependent on the input resolution with the help of INR. Therefore, our method can flexibly increase the heatmap resolution to reduce quantization errors during inference. The standard NerPE directly outputs the complete predicted heatmaps for keypoint localization, which will bring a large amount of calculation when the heatmap resolution is high. To solve this problem, we propose the progressive coordinate decoding method and denote the corresponding version as NerPE-p. Specifically, NerPE-p first yields coarse heatmaps of size 32 × 32. Then, we iteratively divide the area near the maximal activation into 4 × 4 and set p o to 2. Given inputs of size 128 × 128 from MPII, the comparison results of NerPE and NerPE-p are given in Fig. 3 . As the heatmap resolution increases, the positioning accuracy is indeed improved but has a diminishing marginal effect. Compared with the standard NerPE, the use of our progressive coordinate decoding trades negligible performance degradation for a large reduction in computational cost.Sample selection of INR. In NerPE, we let the INR-related network learn the continuous heatmap representation covering the entire image through uniform position sampling. We perform ablations on the sampling modes (w/ and w/o uniform) and evaluate the impact of two hyper-parameters: the division of cells H Z × W Z (see Table 5 ) and the number of samples per cell N Z (see Table 6 ).The experimental results based on ResNet-50 at the input size of 128 × 128 show that the model performance decreases when uniform position sampling is not used. For the division of cells, if it is too sparse (e.g., 2 × 2), each local feature vector needs to be responsible for the prediction of a larger area, which is challenging for the network to fit the continuous heatmap representation. Considering both performance and computational cost, 8 × 8 is a better setting for NerPE. Furthermore, as the number of samples per cell increases, our method achieves better performance. The reason is that, on the one hand, querying more positions makes each gradient descent more robust, On the other hand, cells are decomposed into smaller regions to make the sampling more uniform.Study of heatmap generation. In the Gaussian function, σ is used as a hyper-parameter to control the scale of activation peaks. The difference between continuous and discrete heatmap representations has been discussed, as shown in Fig. 1 . Due to the existence of discretization in existing methods, σ is commonly set to an integer to facilitate the generation of the pixel-based Gaussian kernel (formally named standard biased encoding in [54] ). In contrast, NerPE uses continuous coordinates rather than discrete indices to describe the heatmap plane, and the setting of σ is more flexible under our continuous heatmap representation. The same conclusion goes for b in the Laplacian function. We explore the influence of scale parameters σ and b on keypoint localization, as shown in Table 7 . The proposed NerPE achieves better performance when σ = 0.06 and b = 0.09 respectively.In order to more intuitively show the superiority of continuous heatmap representation, we visualize the output of NerPE at different heatmap resolutions, as shown in Fig. 4 . Thanks to the decoupling of INR from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution without changing the structure and retraining the network. In addition, we visualize the results of using Gaussian and Laplace functions for supervision in continuous heatmap generation in Fig. 5 .  
conclusion: 
 In this paper, to solve the quantization error issue plaguing heatmap regression, we propose an implicit neural representation method NerPE for 2D human pose estimation. According to the extracted image features, NerPE trains a simple MLP-based decoder to fit the Gaussian or Laplace functions at a series of queried positions, which makes the learned heatmap representation continuous in space and confidence. During inference, the decoupling from spatial resolution enables NerPE to output the predicted heatmaps at arbitrary resolution. As a result, our continuous heatmap regression achieves better performance than existing methods using discrete heatmap representation, especially in the case of low resolution. Last but not least, inspired by the flexibility of implicit neural representation, we design a progressive coordinate decoding method to speed up inference by avoiding the complete generation of predicted heatmaps when the desired heatmap resolution is quite high.Limitations and future work. The goal of this work is to explore the feasibility of using implicit neural representations (INRs) to achieve continuous heatmap regression for 2D human pose estimation.To highlight the superiority of NerPE over discrete heatmap representation, our INR-based decoder is designed to be as simple as possible. In future work, we will conduct in-depth research on the network structure to better utilize the characteristics of INRs. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The experimental settings are detailed in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not report error bars or other statistical significance information.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not have the risks mentioned above.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The datasets used in the paper are publicly available and properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: As an implicit neural representation (INR) method, NerPE needs the correct coordinates to query the confidence scores of body joints during training and testing, but traditional coordinate transformation cannot meet our accuracy requirements. We give our data pre-processing and post-processing below.A.1 Coordinate Transformation in Data Pre-processingIn the proposed NerPE, image pixels are expected to be located at the center of their corresponding regions, which is not hold after affine transformation due to the implementation at the code level. In fact, for each region, the interpolation result of its upper left corner in the original image is used as its RGB value, as shown in Here, we discuss in detail the local ensemble used in NerPE, and perform ablation on it as shown in Fig. A3 . For the difference of local feature vectors z * and the mutation of relative coordinates c rel at the junctions between cells, the local ensemble uses bilinear interpolation to ensure that the confidence scores output by the network is continuous. This process is formulated as:H(c abs ) = t∈{00,01,10,11} S t S • f θ z * t , c abs -c z * t s cell /2 ,where z * t refers to the four local feature vectors that are closest to the queried position. The predicted confidence scores are weighted based on the surrounded areas S t and their sum S = t S t . The use of local ensemble means that the sampling of queried positions is no longer limited to the interior of each cell, but the sampling range is expanded to twice the original to achieve overlapping. It can be found in Fig. A3 that the predicted heatmaps show discontinuity after removing the local ensemble. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML)..(9) 


Decision: This paper proposes to use implicit representations to address heatmap quantization error for keypoint localization. It received 2 BAs and 2 WAs after rebuttal, which all the reviewers agreed to accept. 


The initial concerns focused on clarification, visualization, and especially insufficient experiments (resolution, backbone, and efficiency). Moreover, some important related works (like heatmap & offset, non-standard Gaussian heatmap, and regression) are missing.

After going through the paper, the review, and the response, the AC values the key idea, finds most concerns are addressed, and recommends the acceptance of the paper.

The AC also recognizes the weaknesses (insufficient experiments and missing related works) and thus urges the authors to further revise the paper.

Reviews: review: paper_summary: paper_summary: This paper mainly studies the quantization problem of discrete representation of heatmap, especially in the case of low resolution. It proposes to use a continuous Implicit Neural Representation (coordinate-input MLP) to coarse-to-fine query to generate heatmap at arbitrary resolution, which achieves better performance than the current common heatmap method at lower resolution.
strength_weakness: strength_weakness: I will give the boardline first, and then want to see how the author responds to questions about method details, clarification of experimental settings, insufficient experiments, etc. before making a decision.

[Method]

- For training (Fig. 2), why not consider adding additional supervision in the area near the ground truth or even progressive coordinate decoding to refine the heatmap prediction near the ground truth? Will not this give better results?

- So does the final 2D coordinate input into the INR MLP use positional embedding (L312)?

- The bias of confidence estimation comes not only from discretization but also from heuristic confidence estimation [a]. From the gap (confidence) between AP (accuracy + confidence) and AR (accuracy) in Tab. 1, we could see that confidence calibration is still not good enough, and this limitation should be mentioned in the Sec. 5 Limitation & Future Work as well.

[Experiments]

- Does the backbone of this method use COCO/heatmap pre-trained or ImageNet checkpoint? Please describe in detail.

- I don't understand the meaning of the OR/IR column (Tab. 1). Why is HRNet's OR/IR constant at 1/4 and does not change with the input size?

- I don't seem to find GFLOPs, AP and AR results comparing different heatmap resolutions of the heatmap-based baseline method under a fixed input size? They may need to be added in Tab. 1.

- In addition to 256x192, it is also necessary to conduct experiments on a higher resolution 384x288 standard setting to test the applicability of the method's solving quantization error. It seems that the improvement of the method at high resolution is minor.

- How are the overhead savings and performance improvements on the newer heatmap-based method ViTPose [48]? Because ViTPose is powerful, and improving it will make this work more widely applicable.

- There are also some methods that use the heatmap + offset prediction method [b], which also needs to be discussed.

- If I understand correctly, I guess that progressive coordinate decoding may perform similarly to a fully generated heatmap in easier cases, and may only find local minima in more difficult cases (such as multi-peak, non-Gaussian non-uniform heatmaps). If this is the case, then splitting the test by difficulty (such as number of joints, human size, etc.) [c] may be more clear.

- Since the values ​​don't match, are the settings of Tabs. 5 & 6 ablation studies (82.65) different from Tab. 3 (87.7)?

- If the division in Tab. 5 is 16x16 or even higher resolution, the result should be improved further, though the cost gradually approaches the heatmap-based method? Could you add more reports in these regions (not necessarily done during the rebuttal)?

- The author mentioned that the difference in local feature vectors at the cell junction will make the predicted heatmap not smooth, so they proposed to use bilinear interpolation of local features (Ls202-207). Could we perform an ablation study to see the difference in results before and after and visualize the heatmap difference? I think we may also further consider the idea of ​​sparse convolution to fuse neighborhoods.

- It is recommended that the ablation study in Tab. 7 be performed more standardized on COCO 128x128 (not necessarily made during the rebuttal period) because of CrowdPose biases towards crowded situations. The 64x64 input resolution is too small and does not seem to have much application in practice.

- Lack of heatmap visualization and comparisons with other methods

References:

[a] On the calibration of human pose estimation. ICML, 2024.

[b] Towards accurate multi-person pose estimation in the wild. CVPR, 2017.

[c] Benchmarking and error diagnosis in multi-instance pose estimation. ICCV, 2017.
questions: questions: Please see the weaknesses.
limitations: limitations: The authors state their limitations as 1. the decoder structure not studied in depth, 2. discrete heatmaps still required to generate and considered as not elegant. Besides, some limitations mentioned in the weaknesses are suggested to include as well.
paper_summary: paper_summary: The authors tackle 2D human pose estimation through a continuous heatmap representation. Specifically, instead of representing the heatmap as a grid of values, they use an implicit neural representation, where the coarse feature vector along with queried coordinates are fed to an MLP, which predicts the heatmap value for that requested position. This can in theory provide infinite resolution for the heatmap, allowing more accurate localization of its peak, compared to the usual discretized heatmaps that can lead to quantization errors due to the low resolution. The model is trained with randomized query points with Gaussian target. For test time, the authors propose a progressive refinement scheme where the highest-value area in the heatmap is successively queried at finer levels.
Experiments are performed on COCO, MPII and CrowdPose with improvements over baselines, especially for evaluations with stricter thresholds.
strength_weakness: strength_weakness: Some important alternative techniques from the literature are not discussed, and this impacts the conclusions.
First, soft-argmax (integral regression) [1] was introduced specifically to enable continuous output that is not limited by the coarse downsampled grid. This would be an important baseline decoding method to use here. (There have been several extensions, including [2])
Second, offset regression such as the short-range offsets learned in PersonLab [3] would be another simple formulation allowing continuous output not limited by the grid.

The statement around L305 is not correct, the radius of the Gaussian does not have to be an integer for the gridded heatmap. There is no connection between these two, it is neither easier nor more appropriate to use integer-valued standard deviations for a Gaussian on a grid (alternatively, clarification is needed).

[1] Sun et al., Integral Human Pose Regression, ECCV 2018
[2] Gu et al., Bias-Compensated Integral Regression for Human Pose Estimation, PAMI 2023
[3] Papandreou et al. PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model, ECCV 2018
questions: questions: Why is the proposed method not tested also on resolution 384x288? This would allow better comparability in Table 2. Further, why is the gap performance advantage significantly smaller on COCO test-dev compared to COCO val?

Were other alternatives considered for the implicit function, such as learning a distance field?
As for the progressive refinement strategy, wouldn't it be useful to base the exploration on the gradient of heatmap value wrt query location? This seems cheaply differentiable and could give a more direct path towards the peak.

Since the target is always a Gaussian, how does the network output at test time look? Are less certain predictions wider in distribution?

As the queries are given as coordinates to the MLP, would it be possible to query truncated joints outside the heatmap bounds?
limitations: limitations: The authors provide an adequate "Limitations" section in the paper.
paper_summary: paper_summary: This paper proposes a new approach to predict continuous heatmap for keypoint localization. The proposed method adopts a MLP that receives position coordinates and corresponding feature and output the confidence score of this position. Then this method can query candidate points and select the point with maximum score as final results. Experiments on several keypoint localization benchmarks demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: My major concern is about the decoding process of the proposed method. It seems that this method should evaluate a lots of points to select the best results, which will introduce much inference computational cost. The introduced progressive coordinate decoding can reduce the computational cost but can avoidably increase the decoding time due to multi-round decoding process. This paper should give a detailed efficiency comparison of existed methods such as heatmap (SimpleBaseline, HRNet, SimCC, ViTPose), regression (RLE), including GFLOPs, inference time (ms), GPU memory consumption under the same setting (all above methods release their codes, so it is not impossible to conduct such comparisons). These comparisons are necessary because it seems that the advantages of the proposed method is not obvious.
questions: questions: In Weakness
limitations: limitations: This paper discusses its limitation in Section 5.
paper_summary: paper_summary: This paper introduces a new heatmap representation for 2D human pose estimation. Prior approaches use a quantized representation of heatmaps, where a confidence score is assigned to each pixel. In addition to being dependent on the image resolution, this representation does not match the usual ground truth coordinates, which are continuous when working in image crops.

This paper proposes an implicit neural representation to address this limitation of prior works. Specifically, given 2D coordinates (which do not necessarily correspond to the center of a pixel), the introduced neural network outputs a confidence score for each keypoint. At inference, the coordinates of each keypoint are recovered using the maximum confidence value. The main contributions of this paper are the following: (1) The authors introduce NerPE, a novel implicit representation of heatmaps for continuous human pose estimation; (2) A progressive coordinate decoding method is introduced to recover 2D coordinates from continuous heatmaps.
strength_weakness: strength_weakness: - The Related Work section lacks discussions and comparisons with other approaches attempting to address the quantization error in heatmaps, such as [42,a,b,c,d,e,f,g]. The authors say that "As far as we know, existing heatmap-based methods all belong to discrete heatmap regression" (L104), but for instance, [b] predicts a float location using an offset within the original heatmap. Similar approaches using PixelShuffle operation were proposed [f]. In general, the introduction presents the problem of heatmap quantization as if prior works ignored it, but plenty of works attempted to address this problem.

- I am unsure if having a continuous representation of heatmaps is useful. The ground truth is given in pixel coordinates on the full image. I agree that increasing the heatmap resolution until we reach the original image resolution is helpful, but is it useful to have a higher resolution than the original image? In the end, the ground truth is given in full pixel coordinates.

- Some information important for reproducibility is missing. For instance, what are the training and validation datasets and the number of epochs? Even if the authors say they used "standard settings" I believe this is not enough to reproduce the results.

- No information is provided regarding the material used for training and testing or the running times. This is particularly concerning as the method has a high computational cost. The authors propose an alternative decoding strategy that seems much less expensive, but comparing it with other methods and heatmap representations would be necessary.




[a] Bulat, A., Sanchez, E., & Tzimiropoulos, G. (2021). Subpixel heatmap regression for facial landmark localization. BMVC

[b] Lan, X., Hu, Q., & Cheng, J. (2021). Revisiting quantization error in face alignment. ICCV

[c] Yu, B., & Tao, D. (2021). Heatmap regression via randomized rounding. PAMI

[d] Papandreou, G., Zhu, T., Kanazawa, N., Toshev, A., Tompson, J., Bregler, C., & Murphy, K. (2017). Towards accurate multi-person pose estimation in the wild. CVPR

[e] Luvizon, D. C., Picard, D., & Tabia, H. (2018). 2d/3d pose estimation and action recognition using multitask deep learning. CVPR

[f] Wang, H., Liu, J., Tang, J., & Wu, G. (2023). Lightweight Super-Resolution Head for Human Pose Estimation. MM

[g] Li, J., Bian, S., Zeng, A., Wang, C., Pang, B., Liu, W., & Lu, C. (2021). Human pose regression with residual log-likelihood estimation. ICCV
questions: questions: (1) "The meaning of the confidence scores output by current heatmap-based models has not been theoretically proven." (L43). What does this really mean? Is the NerPE representation "theoretically proven"?

(2) In Equation (5), to my understanding, $f_W(i,j)$ and $f_b(i,j)$ are floats, and $z^*$ would be of dimension $C$? But isn't $H_{z^*}$ supposed to be a float since it is a confidence value?

(3) "We are surprised to find that the conclusion drawn is similar to the local implicit image function in [8] except for the additional normalization step." (L179). Why is that surprising? Isn't the process a bit similar?

(4) Could we consider that the proposed method consists of doing sub-heatmaps until we reach a desired degree of precision?

(5) The model is trained to output Gaussian or Laplacian heatmaps. However, at inference, there is no guarantee that the confidence distribution follows the same distribution. I think that this is a good point, as this brings flexibility. However, couldn't that raise issues for finding the argmax? For instance, the model could potentially "hesitate" between 2 different places when placing the center of the heatmap (the predicted distribution would look like a mixture of Gaussians, for instance). Did you observe such cases in practice, and could other strategies be explored instead of choosing the argmax?


Side remark: Typo L15 "desgin"
limitations: limitations: Limitations of this work are briefly discussed in Section 5:

- The limited experiments on network architecture and index embedding are not a problem, as this work aimed to focus on the heatmap representation.

- The discussion on the necessity of predicting discrete heatmaps for computing an argmax is interesting, but it would have been more valuable if the authors had provided potential improvements and strategies on that point.

The potential negative societal impacts are not discussed but are rather limited. The authors could consider mentioning the potential misuse of pose estimation for intrusive surveillance or military applications. The environmental cost could also be discussed, especially since this method is more computationally expensive than prior works.

Comments: title: Conditional Borderline
Reviewer-author discussions
comment: We sincerely thank the reviewer for the kind reply and willingness to improve the score. We will supplement the experiments related to offset prediction and more visualization of non-standard Gaussian heatmaps in the next/final revision of our paper, as suggested by the reviewer.

> Can you explain why sparse convolution is not considered, which has been explored in 3D reconstruction and generation?

If we understand correctly, in 3D reconstruction and generation, continuous 3D data (i.e. point clouds) needs to be voxelized into a grid structure to meet the requirements of sparse convolution. This process is similar to heatmap discretization in human pose estimation, which is exactly what we want to avoid. Back to our proposed NerPE, during training, the 2D coordinates as one of the inputs are randomly sampled in the image plane, so they exist in the form of a set in space and cannot be processed by convolution operations. This is why implicit neural representations used to model continuous signals are mostly implemented by MLP even when the task is image-related.
I carefully read the reviews of other reviewers and the author's careful rebuttal. I sincerely thank the author for addressing most of my concerns carefully, especially the experiments on GFLOPs, inference latency of FPS, different input and cell resolutions, SOTA backbones, etc., which help to have a more comprehensive understanding of the proposed method. The idea of ​​INR is very straightforward, especially achieving good performance in low-resolution experiments. So if the author can promise in addition to the rebuttal,

1. to add comparison and combination experiments with the "offset" method later (see W9 and other reviews),

2. to add some more visualization of non-standard Gaussian heatmaps, such as the difference between the rendered and baseline heatmaps for medium and hard cases [c] (Ws 10 & 15),

3. Open source is encouraged to facilitate reproduction for the community.

then I will consider to raise my rating to **borderline** accept (**conditional**).

In addition, there are some follow-up questions:

**W13:** Can you explain why sparse convolution is not considered, which has been explored in 3D reconstruction and generation?
Thanks to the reviewer for further discussion on the concerns. First of all, we need to clarify again that the division of cells (i.e., grid resolution) in NerPE is a hyperparameter independent of the input resolution. We distinguish the division of cells and heatmap resolution from both theoretical and experimental aspects to show the differences between NerPE and the above-mentioned methods as follows:

We are pleased that the reviewer agrees that the compared methods are still affected by heatmap resolution even if they output continuous values through additional operations. Due to information loss caused by heatmap discretization, the performance deteriorates as the resolution decreases. In contrast, the division of cells plays a different role than the heatmap resolution. For local implicit neural representations, if the division is too coarse, it is challenging for each local feature vector to be responsible for regressing a larger area in the image plane. If the division is too fine, the importance of the image information will overshadow the coordinate information because the confidence scores in each cell change marginally. To illustrate this, we add the case where the division of cells $H_Z \times W_Z$ is 64×64 based on Tab. 5, as shown in the following table. The results show that as $H_Z \times W_Z$ increases, the performance of NerPE first increases and then decreases, which is inconsistent with the trend of the impact of heatmap resolution.

|     | 2×2 | 4×4 | 8×8 | 64×64 |
| --- | --- | --- | --- | --- |
| w\o uniform | 75.28 | 80.83 | 82.54 | 82.17 |
| w\ uniform | 76.03 | 81.42 | 82.65 | 82.25 |

In summary, although resolution is related to the performance of both NerPE and the above-mentioned methods, the principles and impacts are different. Furthermore, thanks to implicit neural representations, NerPE achieves decoupling from the output resolution, which can also be applied to techniques such as PersonLab to learn continuous distance fields.
> For images with different resolutions and different backbone networks, we resize the encoded feature map to 8×8 (L233), which means that each local feature vector is responsible for a 64-equally divided area on the 2D plane, independent of the resolution.

Based on the ablation in Table 5, NerPE's performance also depends on the grid resolution, so my point is that this aspect is not so different from the above-mentioned methods which can output continuous values but have worse performance on coarser grids.
We thank the reviewer for the positive feedback. We are glad that the reviewer thinks our work is worthy of acceptance.

> I think it is important to add the information from Table A3 in the final version; I expected the running times to be larger given the inference process.

We will report the analysis of efficiency in our paper. There are two reasons why the inference time is faster than expected: (1) In both NerPE and NerPE-p, the bakcbone network is executed only once in the forward pass. (2) The INR-related part is implemented by a simple MLP, and queries for multiple positions are performed in parallel on the GPU. Therefore, the inference time of our method is close to that of our baseline.
Thanks a lot to the authors for this rebuttal, which addressed all my questions and concerns. I still think that this paper should be accepted.

I think it is important to add the information from Table A3 in the final version; I expected the running times to be larger given the inference process.
Thanks to Reviewer k3b9 for the discussion on the rebuttal.



Dear Reviewer xx7e, KNNC and rvLc,

Would you please look at the rebuttal, discuss with the authors and finalize your score after discussion?

Thanks,
You AC
Thank you for your kind reply!

> Overall, I believe it would be important to discuss both of these prior lines of works in the paper. Even if they aren't doing the same thing, the motivation is similar enough.

We will mention the soft-argmax approaches and PersonLab in Related Work to show the difference in solving the quantization error problem.

> I would expect that NerPE's performance is similarly impacted if the underlying feature map has lower resolution (even though the implicit heatmap is continuous). Is it not so?

For images with different resolutions and different backbone networks, we resize the encoded feature map $Z$ to 8×8 (L233), which means that each local feature vector $z^*$ is responsible for a 64-equally divided area on the 2D plane, independent of the resolution.

If you have further questions, we are happy to discuss them.
Thank you for the detailed reply to my review.

> The original intention of these methods to construct a heatmap-like form is to narrow the gap between coordinate regression and heatmap regression, rather than to overcome the limitations of the coarse downsampled grid.

The issue of quantization errors (coarseness of the grid) is an explicit motivation for soft-argmax approaches [a] and [b], so there is considerable motivational overlap.

[a] Nibali et al., Numerical Coordinate Regression with Convolutional Neural Networks. arxiv:1801.07372, 2018
[b] Sun et al., Integral Human Pose Regression. ECCV, 2018

> Although PersonLab can achieve sub-pixel positioning with the help of offset prediction, its performance still depends on the quality of heatmap regression.

This may be true, however the offsets do enable a continuous output without quantization errors (not bound to grid points).

> When the output resolution of the heatmap and offset field is not high enough, the performance of PersonLab is also unsatisfactory.

I would expect that NerPE's performance is similarly impacted if the underlying feature map has lower resolution (even though the implicit heatmap is continuous). Is it not so?

Overall, I believe it would be important to discuss both of these prior lines of works in the paper. Even if they aren't doing the same thing, the motivation is similar enough.

> We report the results of NerPE on the COCO test-dev set with the input resolution of 384x288 as follows

Thank you, this result is convincing.