Title: Continuous Heatmap Regression for Pose Estimation via Implicit Neural Representation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Human pose estimation

continuous heatmap regression

implicit neural representation
Abstract: Heatmap regression has dominated human pose estimation due to its superior performance and strong generalization. To meet the requirements of traditional explicit neural networks for output form, existing heatmap-based methods discretize the originally continuous heatmap representation into 2D pixel arrays, which leads to performance degradation due to the introduction of quantization errors. This problem is significantly exacerbated as the size of the input image decreases, which makes heatmap-based methods not much better than coordinate regression on low-resolution images. In this paper, we propose a novel neural representation for human pose estimation called NerPE to achieve continuous heatmap regression. Given any position within the image range, NerPE regresses the corresponding confidence scores for body joints according to the surrounding image features, which guarantees continuity in space and confidence during training. Thanks to the decoupling from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution during inference without retraining, which easily achieves sub-pixel localization precision. To reduce the computational cost, we design progressive coordinate decoding to cooperate with continuous heatmap regression, in which localization no longer requires the complete generation of high-resolution heatmaps. The code is available at https://github.com/hushengxiang/NerPE.

Continuous Heatmap Regression for Pose Estimation via Implicit Neural Representation

Abstract

Heatmap regression has dominated human pose estimation due to its superior performance and strong generalization. To meet the requirements of traditional explicit neural networks for output form, existing heatmap-based methods discretize the originally continuous heatmap representation into 2D pixel arrays, which leads to performance degradation due to the introduction of quantization errors. This problem is significantly exacerbated as the size of the input image decreases, which makes heatmap-based methods not much better than coordinate regression on low-resolution images. In this paper, we propose a novel neural representation for human pose estimation called NerPE to achieve continuous heatmap regression. Given any position within the image range, NerPE regresses the corresponding confidence scores for body joints according to the surrounding image features, which guarantees continuity in space and confidence during training. Thanks to the decoupling from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution during inference without retraining, which easily achieves sub-pixel localization precision. To reduce the computational cost, we design progressive coordinate decoding to cooperate with continuous heatmap regression, in which localization no longer requires the complete generation of high-resolution heatmaps. The code is available at https://github.com/hushengxiang/NerPE.

1 Introduction

Human pose estimation (HPE) is a fundamental task in the field of computer vision, which is widely used in various human-centered applications [41, 29, 37, 9, 55, 53]. In multi-person pose estimation, the top-down framework is a mainstream two-stage pipeline, in which the person areas are firstly cropped out and then the body joints within them are located. According to the way to describe the positions of body joints, pose estimation methods can be further divided into coordinate regression [44, 43, 20, 22, 28] and heatmap regression [31, 48, 42, 24, 52]. Due to better performance brought by spatial encoding of heatmap representation, heatmap regression has received more attention than coordinate regression. Recently, most heatmap-based methods [11, 51, 50] focus on the design of network structure but ignore the importance of heatmap generation. Only a few works [26, 23] have noticed the irrationality in the standard heatmap representation widely used in existing methods.

In the top-down framework, the ground-truth coordinates of body joints in the original image are mapped to the input plane of a keypoint detector, through the same affine transformation applied to the cropped image patch. To supervise the output of traditional explicit neural networks, the current heatmap generation strategy yields 2D pixel arrays to reflect the spatial distribution of body joints, which means that the originally continuous heatmap representation needs to be discretized as shown

Discrete heatmap representation at different resolutions

Continuous heatmap representation

Input image

Implicit neural function

Image features

Query position

Shoulder(l)

likelihood

)

,

(

y

x

)

,

(

y

x

low resolution

High resolution

in Fig. 1. Specifically, the center of the Gaussian kernel is placed at body joints to calculate the confidence scores on the grid points as the ground-truth heatmaps. Although this discrete heatmap representation has achieved great success, it introduces quantization errors since body joints after affine transformation may fall anywhere within the image range rather than just at these fixed grid points. For the output of explicit neural networks, the information loss caused by spatial sampling is difficult to compensate through post-processing operations [31, 54]. With the reduction of the input resolution, this problem will be further exacerbated, which makes the performance of heatmap-based methods degraded even worse than some methods based on coordinate regression. Considering that the Gaussian function used for heatmap generation is inherently continuous, it is a natural idea to learn a continuous heatmap representation to replace fixed spatial sampling for HPE.

Implicit neural representations (INRs) are proposed to parameterize a variety of continuous signals [30, 8, 7, 47, 39, 46] in computer vision. Unlike explicit neural networks that output specific structures ( e.g. , mesh, sequence), INRs map an index to its corresponding value, enabling not only continuous representations but also greater flexibility in use. Obviously, by combining INR and heatmap regression, the pose estimation model is able to learn continuous confidence scores for all body joints at any position. Compared to the discrete heatmap representation with a fixed resolution, the introduction of INR avoids the invariant spatial sampling caused by discretization during training and can yield the predicted heatmaps at arbitrary resolution during testing.

In this paper, to improve the performance of heatmap-based HPE, we abandon the main culprit of quantization errors, namely discrete heatmap representation. Instead, we propose NerPE to achieve continuous heatmap regression through a novel neural representation. Specifically, we generalize the upsampling factor in discrete sub-pixel convolution [38] to infinity to obtain a continuous upsampling function with respect to 2D coordinate. NerPE learns its continuous heatmap representation at a series of queried positions that are randomly sampled within the image range. For any position, the target likelihood at it is calculated from the target 2D pose and its absolute coordinate, and the estimated likelihood at it is inferred from the local feature vector and its relative coordinate. Since confidence scores are learned at all positions within the image range during training, our method achieves better performance than discrete heatmap-based methods that focus only on fixed grid points. As the resolution of the input image decreases, the superiority of continuous heatmap representation over existing methods becomes more prominent.

Limited by the explicit neural representation of existing methods, the heatmap resolution for a given image cannot be changed once the network structure is determined. In contrast, another benefit that INR brings to heatmap regression is that the heatmap resolution is no longer correlated with the image resolution. This means that our method can flexibly generate the predicted heatmaps with different resolutions based on accuracy requirements without retraining. To speed up the inference when high-resolution heatmaps are required, we design a progressive coordinate decoding method. Thanks to the decoupling of INR and spatial resolution, NerPE enables high-precision localization without the need to calculate the complete heatmaps. Specifically, low-resolution heatmaps are first output to determine the approximate locations of body joints. Subsequently, the area near the maximal activation is further retrieved in an iterative manner. Notably, our method can be easily integrated into most heatmap-based methods. The contributions are summarized as follows:

We propose NerPE to avoid the introduction of quantization errors during training and to output the predicted heatmaps at arbitrary resolution during inference. To our knowledge, we are the first to apply implicit neural representations to human pose estimation.

We design a progressive coordinate decoding method to derive the coordinates of body joints from continuous heatmap representation, in which our method achieves high-precision localization with low computational cost through coarse-to-fine retrieval.

We conduct extensive experiments on three pose estimation benchmarks: COCO [25], MPII [1], and CrowdPose [21]. The results show that NerPE significantly enhances existing heatmap-based methods and obtains superior performance on low-resolution input images.

2 Related Work

2.1 Human Pose Estimation

2D human pose estimation (HPE) aims to locate a series of anatomical keypoints to represent the human pose in the input image. Currently, coordinate regression [3, 22, 28] and heatmap regression [17, 45, 12] are the two main pose estimation paradigms, both of which have received widespread attention. Coordinate regression, also known as direct regression, relies on deep neural networks to explore the mapping between the input image and the coordinates of body joints. The high degree of nonlinearity is difficult for optimization, which makes coordinate regression not perform well enough. As for heatmap regression, HPE is converted into a combination of multi-label keypoint classification and coordinate decoding post-processing. Benefiting from dense prediction, heatmap-based methods not only easily exploit visual cues around the target position, but also take the ambiguity of keypoint localization into account. This is why heatmap-based methods are generally superior to those based on coordinate regression. Although heatmap-based methods have made steady progress, quantization errors are still a troubling problem, which is what our work is dedicated to solving.

2.2 Discrete Heatmap Regression

To be clear at first, the cause of quantization errors is discretization, not the introduction of heatmap representations this behavior itself. As far as we know, existing heatmap-based methods [48, 42, 24, 52] all belong to discrete heatmap regression, using 2D pixel arrays to describe the spatial distribution of body joints. There are a few works that attempt to compensate for the damage to performance caused by quantization errors. In [31], the coordinates of body joints are empirically determined as the position of moving a 0.25 pixel from the maximal to second maximal activation. DARK [54] implements Taylor series approximation for heatmap activation to locate body joints based on distribution information. Although these post-processing operations achieve sub-pixel localization precision, subjective assumptions make the inference not so reliable, especially in low-resolution cases. Instead of remedying the shortcomings of discrete heatmap representation, we propose continuous heatmap regression to preserve the continuity of ground-truth heatmaps.

2.3 Implicit Neural Representation

In response to the need to model continuous signals, implicit neural representations (INRs) aim to learn a neural function that predicts the corresponding value according to a given index. Encouraged by the success in 3D reconstruction [14, 32, 4] and generation [6, 33, 49], INR has been extended to some other tasks, including super-resolution [8, 10, 13], image generation [40, 5, 2], and video compression [7, 15, 27]. In the paper, to avoid the damage of discretization to heatmap regression, we approximate the continuous Gaussian function by multi-layer perceptrons (MLPs). Given a coordinate within the image range, NerPE is designed to regress the confidence scores of all body joints at that position, in which the heatmap representation is free from the constraints of 2D pixel arrays [11, 51, 50]. As a result, NerPE has the ability to output the predicted heatmaps at arbitrary resolution during inference. In terms of eliminating quantization errors, offset prediction [34, 19] has the same goal as our work, and it aims to perform unbiased estimation for the coordinates of body joints. However, the performance of offset prediction is still affected by the resolution of the distance field. There is complementarity between the two techniques in HPE: offset prediction helps achieve more advanced decoding, and INR makes the distance field continuous. In order to highlight the superiority of INR, we only regress confidence scores in this paper.

3 Proposed Method

In this section, we first discuss the hazards of discrete heatmap representation and the limitations of existing post-processing operations. Then, we introduce a novel Ne ural r epresentation for human P ose E stimation (NerPE) and illustrate the superiority of continuous heatmap regression.

3.1 Preliminary of Heatmap Representation

Given an input image I ∈ R 3 × H I × W I , heatmap-based models output 2D pixel arrays H of size K × H I 4 × W I 4 to reflect the spatial distribution of K body joints. To conform the output form of explicit neural networks, the continuous Gaussian function is discretized in the heatmap generation process, which leads to the introduction of quantization errors and the reduction of positioning accuracy. Currently, there are two post-processing operations [31, 54] that are widely used to mitigate the negative effects of discretization.

The standard coordinate decoding method [31] is designed entirely according to experience. Based on the analysis of the model's performance, the position p at which the maximal activation H ( c m ) moves a 0.25 pixel towards the second maximum H ( c s ) is taken as the final coordinate:

The distribution-aware decoding method [54] assumes that the predicted heatmaps still conform to the Gaussian distribution. The Taylor series expansion to the quadratic term is implemented at the position c m of the maximal activation as:

where H ′ ( · ) and H ′′ ( · ) respectively denote the first and second order derivatives of the Gaussian distribution close to the predicted heatmaps. The position p of each body joint is determined by the Gaussian mean µ = c m -( H ′′ ( c m )) -1 H ′ ( c m ) .

To achieve sub-pixel localization precision against quantization errors, the above post-processing operations impose subjective assumptions in coordinate decoding. Compared with the limited improvements brought by the above passive compensations, the most direct way to solve quantization errors is to abandon the explicit neural network and learn a continuous heatmap representation.

3.2 Reformulation in a Continuous Form

From a structural and functional perspective, general pose estimation models consist of an encoder and a decoder (also called 'head' in some works). To reduce the computational cost and increase the receptive field, the image features are downsampled in the encoder. Correspondingly, the decoder increases the output resolution by including upsampling layers such as interpolation, deconvolution or sub-pixel convolution [38]. However, once the network structure is determined, the decoder in existing methods makes the model only output the predicted heatmaps with a fixed resolution. Starting from sub-pixel convolution, we derive its continuous version via implicit neural representation (INR), known as a continuous upsampling function.

Using sub-pixel convolution as the final layer, the image features Z of size C × H Z × W Z are mapped into higher resolution heatmaps H of size K × rH Z × rW Z with an upscaling factor r :

where PS denotes a PixelShuffle operation used to reshape each element along the feature dimension into small 2D pixel arrays (called a cell) corresponding to body joints. According to the spatial prior, the weight W and bias b in Eq. (3) are further expressed as:

where the order of concatenation corresponds to the order of expansion in the PixelShuffle operation. These sub-weights W i,j and sub-biases b i,j can be viewed as the outputs of functions f W ( i, j ) and

Image

Encoder

(a) random sampling

(b) uniform sampling

Selection of query positions

OR

MLP-based

Decoder

(c) standard coordinate decoding

(d) progressive coordinate decoding

Keypoint localization

OR

argmax

in the training phase

in the testing phase

argmax

gt confidence

gt coord

f b ( i, j ) with a 2D index as argument. We use z ∗ to refer to each element in the image features Z , called a local feature vector. The confidence scores at ( i, j ) relative to z ∗ is given by:

In order to ensure the consistency in description of sub-pixel convolution with different upscaling factors, we normalize the 2D index array composed of integers in { 1 , 2 , . . . , r } into [ -1 , 1] :

As r increases, the spatial sampling of [ -1 , 1] × [ -1 , 1] becomes denser. When r approaches ∞ , the discrete grid points become a continuous region covering all resolutions. Therefore, we obtain the expression for the continuous version of sub-pixel convolution H z ∗ ( c rel ) = f θ ( z ∗ , c rel ) , and further replace the relative coordinate c rel with the absolute coordinate c abs to get:

where c z ∗ is the center coordinate of cell H z ∗ , and s cell is the window size of cell H z ∗ . Starting from sub-pixel convolution, the conclusion drawn is similar to the local implicit image function in [8] except for the additional normalization step. In terms of network architecture, we only replace the last few layers in existing heatmap-based models with our continuous upsampling function, which fundamentally solves the problem of discretization in training.

3.3 Learning Continuous Heatmap Regression

Obviously, the discrete heatmap representation is a subcase of the proposed NerPE, where only several fixed positions ( i.e. , grid points) within the image range are trained and predicted. Thanks to the decoupling of heatmap representation and spatial resolution, NerPE can comprehensively fit Gaussian or Laplace functions. The design of our method involves three aspects: continuous heatmap generation, uniform position sampling, and progressive coordinate decoding.

Continuous heatmap generation. Since the output of the network is no longer 2D pixel arrays like existing methods, we need to calculate the ground-truth confidence scores for those positions being queried. In our method, the continuous heatmap representations based on Gaussian and Laplace distributions respectively are expressed relative to c abs as:

where c gt is the ground-truth coordinates of body joints. Since the queried position can be anywhere in the image range, the ground-truth heatmaps are continuous in space and confidence. The loss function of continuous heatmap regression is:

Uniform position sampling. For heatmap learning based on INR, the selection of queried positions for training is critical to the model performance. On the one hand, in order to fully take care of the entire heatmap plane, the sampling of queried positions during training should be evenly distributed over the image range. Thus, we divide each cell uniformly into √ N Z × √ N Z regions and perform random sampling within them, as shown in Fig. 2(b). In each training sample, N Z × H Z × W Z positions are picked out and their ground-truth confidence scores are calculated for supervision.

On the other hand, INR acts as a parameterized continuous function, which expects that spatially close positions should have similar confidence scores. Inside each cell, this is easy to implement for deep neural networks. However, at the junctions between cells, the difference of local feature vectors z ∗ and the mutation of relative coordinates c rel make the network tend to produce discontinuous predictions. To solve this problem, we adopt the local ensemble in [8] to perform bilinear interpolation, where the sampling range is expanded to achieve overlapping.

Progressive coordinate decoding. The design of heatmap representation has been well discussed above, and more importantly, it ultimately serves keypoint localization. In order to decode the coordinates of body joints through argmax , a straightforward way is to arrange the candidate points within the cells at a specific density, so that NerPE can output the predicted heatmaps with the corresponding upsampling factor r ≥ 1 :

where H ( n ) cell is the distribution of body joints in the n -th cell of the input image, and C r × r is a relative coordinate matrix to indicate the candidate points. When the heatmap resolution is high enough, argmax is sufficient to deal with quantization errors without extra post-processing operations.

In existing methods, the likelihood of body joints at each position is derived simultaneously by an explicit neural network, which means there is no choice but to output the entire predicted heatmaps. In contrast, NerPE can estimate confidence scores based on 2D coordinates in a serial manner thanks to the decoupling of INR from spatial resolution. We propose progressive coordinate decoding to reduce the computational cost by evading the complete generation of predicted heatmaps when the heatmap resolution is high, as shown in Fig. 2(d). Specifically, we first generate the low-resolution heatmaps to estimate the approximate locations of body joints. Then, the area near the maximal activation is iteratively sub-divided for coarse-to-fine retrieval. This decoding method only needs to calculate additional t × K × ( r + p o ) × ( r + p o ) positions to achieve an equivalent resolution of r t H Z × r t W Z , where t is the number of iterations and p o is the number of pixels overlapped.

4 Experiments

In this section, we compare the performance of NerPE and discrete heatmap-based methods on input images of different resolutions. The experimental results show the superiority of continuous heatmap regression and the hazards of discrete heatmap representation, especially in the case of low resolution.

4.1 Implementation Details

Network architecture. We adopt ResNet [16], HRNet [42] or TokenPose [24] as the backbone network, and resize the extracted image features to 8 × 8. The focus of this work is to provide a new perspective on heatmap regression through implicit neural representation (INR) rather than pursuing extreme performance. For clarity, we use a pure MLP structure as a decoder to implement continuous heatmap regression. Queried position embeddings, derived from 2D coordinates and their sinusoidal encodings, are concatenated with local feature vectors and then fed into the decoder. In each cell corresponding to a local feature vector, we collect N Z = 64 queried positions as training subjects. For continuous heatmap generation, σ in the Gaussian function is set to 0.06, which is close in proportion to the discrete heatmap representation. When not explicitly stated, the size of the predicted heatmaps is 256 × 256 by default. NerPE no longer needs those empirical operations to align flipped heatmaps [48] and shift decoded coordinates [31].

Optimization. In the main experimental results, the training settings of NerPE is consistent with the comparison methods [48, 42, 24] based on discrete heatmap regression. We use the Adam optimizer [18] for training, in which the learning rate is initialized to 1 e -3 and decreased to 1 e -4 and 1 e -5 . The data augmentation used includes random rotation, random scale, image flipping, and half body cropping. All our experiments are conducted on an open-source machine learning, PyTorch [35].

4.2 Main Experimental results

Evaluation on COCO. To evaluate the value of continuous heatmap representation for human pose estimation (HPE), we perform NerPE with three backbones [16, 42, 24] at three input resolutions on the COCO validation set, as shown in Table 1. The experimental results show that our method achieves better performance, and the superiority over discrete heatmap-based methods increases as the input resolution decreases. Thanks to INR's modeling of continuous signals, NerPE can still output fine and smooth heatmaps of body joints even with a low-resolution image as input. In contrast, the discrete heatmap representation resorts to manually designed decoding methods [31, 54] to achieve sub-pixel accuracy, which suffers from significant performance degradation when quantization errors are too large. The comparison results on the COCO test-dev set are given in Table 2. These suggest that the accuracy and flexibility of the model are greatly improved by only replacing the last few layers in the existing heatmap-based methods with the INR-based MLP.

Evaluation on CrowdPose. To evaluate the performance in crowded scenes, we test NerPE on the CrowdPose dataset (see Table 4), in which YoloV3 [36] is adopted as the human detector. At the input size of 256 × 192, our method achieves superior performance with 66.9 AP. Thanks to the learned continuous heatmap representation, NerPE delivers performance gains on AP 75 , a more stringent metric. For the low-resolution case, the experimental results show that NerPE brings an improvement of 6.1 AP to HRNet and further expands its lead over discrete heatmap-based methods.

Progressive coordinate decoding. In the proposed NerPE, the heatmap resolution is not dependent on the input resolution with the help of INR. Therefore, our method can flexibly increase the heatmap resolution to reduce quantization errors during inference. The standard NerPE directly outputs the complete predicted heatmaps for keypoint localization, which will bring a large amount of calculation when the heatmap resolution is high. To solve this problem, we propose

86.6

87.2

87.3

19.4

33.2

34.3

86.6

87.5

87.5

19.4

33.5

34.8

32×32

128×128

512×512

PCKh

NerPE-p

NerPE

86.6

87.2

87.3

19.4

33.2

34.3

86.6

87.5

87.5

19.4

33.5

34.8

32×32

128×128

512×512

PCKh

NerPE-p

NerPE

86.6

87.2

87.3

19.4

33.2

34.3

86.6

87.5

87.5

19.4

33.5

34.8

32×32

128×128

512×512

PCKh

NerPE-p

NerPE

3.5

4.2

4.8

7.0

20.8

32×32

64×64

128×128

256×256

512×512

GFLOPS

NerPE-p

NerPE

3.5

4.2

4.8

7.0

20.8

32×32

64×64

128×128

256×256

512×512

GFLOPS

NerPE-p

NerPE

the progressive coordinate decoding method and denote the corresponding version as NerPE-p. Specifically, NerPE-p first yields coarse heatmaps of size 32 × 32. Then, we iteratively divide the area near the maximal activation into 4 × 4 and set p o to 2. Given inputs of size 128 × 128 from MPII, the comparison results of NerPE and NerPE-p are given in Fig. 3. As the heatmap resolution increases, the positioning accuracy is indeed improved but has a diminishing marginal effect. Compared with the standard NerPE, the use of our progressive coordinate decoding trades negligible performance degradation for a large reduction in computational cost.

4.3 Ablation Study

Sample selection of INR. In NerPE, we let the INR-related network learn the continuous heatmap representation covering the entire image through uniform position sampling. We perform ablations on the sampling modes (w/ and w/o uniform) and evaluate the impact of two hyper-parameters: the division of cells H Z × W Z (see Table 5) and the number of samples per cell N Z (see Table 6). The experimental results based on ResNet-50 at the input size of 128 × 128 show that the model

performance decreases when uniform position sampling is not used. For the division of cells, if it is too sparse ( e.g. , 2 × 2), each local feature vector needs to be responsible for the prediction of a larger area, which is challenging for the network to fit the continuous heatmap representation. Considering both performance and computational cost, 8 × 8 is a better setting for NerPE. Furthermore, as the number of samples per cell increases, our method achieves better performance. The reason is that, on the one hand, querying more positions makes each gradient descent more robust, On the other hand, cells are decomposed into smaller regions to make the sampling more uniform.

Study of heatmap generation. In the Gaussian function, σ is used as a hyper-parameter to control the scale of activation peaks. The difference between continuous and discrete heatmap representations has been discussed, as shown in Fig. 1. Due to the existence of discretization in existing methods, σ is commonly set to an integer to facilitate the generation of the pixel-based Gaussian kernel (formally named standard biased encoding in [54]). In contrast, NerPE uses continuous coordinates rather than discrete indices to describe the heatmap plane, and the setting of σ is more flexible under our continuous heatmap representation. The same conclusion goes for b in the Laplacian function. We explore the influence of scale parameters σ and b on keypoint localization, as shown in Table 7. The proposed NerPE achieves better performance when σ = 0 . 06 and b = 0 . 09 respectively.

4.4 Visualization

In order to more intuitively show the superiority of continuous heatmap representation, we visualize the output of NerPE at different heatmap resolutions, as shown in Fig. 4. Thanks to the decoupling of INR from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution without changing the structure and retraining the network. In addition, we visualize the results of using Gaussian and Laplace functions for supervision in continuous heatmap generation in Fig. 5.

32

×

32

64

×

64

Input

128

×

128

256

×

256

Sho.(l)

Elb.(r)

Kne.(l)

Head

Laplace

A525360

18.00-

eiR

5 Conclusion

In this paper, to solve the quantization error issue plaguing heatmap regression, we propose an implicit neural representation method NerPE for 2D human pose estimation. According to the extracted image features, NerPE trains a simple MLP-based decoder to fit the Gaussian or Laplace functions at a series of queried positions, which makes the learned heatmap representation continuous in space and confidence. During inference, the decoupling from spatial resolution enables NerPE to output the predicted heatmaps at arbitrary resolution. As a result, our continuous heatmap regression achieves better performance than existing methods using discrete heatmap representation, especially in the case of low resolution. Last but not least, inspired by the flexibility of implicit neural representation, we design a progressive coordinate decoding method to speed up inference by avoiding the complete generation of predicted heatmaps when the desired heatmap resolution is quite high.

Limitations and future work. The goal of this work is to explore the feasibility of using implicit neural representations (INRs) to achieve continuous heatmap regression for 2D human pose estimation. To highlight the superiority of NerPE over discrete heatmap representation, our INR-based decoder is designed to be as simple as possible. In future work, we will conduct in-depth research on the network structure to better utilize the characteristics of INRs.

References

Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR , pages 3686-3693, 2014.

Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool. Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation. In CVPR , pages 3981-3990, 2022.

Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In CVPR , pages 4733-4742, 2016.

Rohan Chabra, Jan E. Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In ECCV , pages 608-625, 2020.

Lucy Chai, Michaël Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. In ECCV , pages 170-188, 2022.

Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR , pages 5799-5809, 2021.

Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. In NeurIPS , volume 34, pages 21557-21568, 2021.

Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In CVPR , pages 8628-8638, 2021.

Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution for skeleton-based action recognition. In ICCV , pages 13359-13368, 2021.

Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In CVPR , pages 2047-2057, 2022.

Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S. Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In CVPR , pages 5386-5395, 2020.

Runyang Feng, Yixing Gao, Tze Ho Elden Tse, Xueqing Ma, and Hyung Jin Chang. Diffpose: Spatiotemporal diffusion model for video-based human pose estimation. In ICCV , pages 14861-14872, 2023.

Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In CVPR , pages 10021-10030, 2023.

Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In CVPR , pages 4857-4866, 2020.

Sharath Girish, Abhinav Shrivastava, and Kamal Gupta. Shacira: Scalable hash-grid compression for implicit neural representations. In ICCV , pages 17513-17524, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang. The devil is in the details: Delving into unbiased data processing for human pose estimation. In CVPR , pages 5700-5709, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Pifpaf: Composite fields for human pose estimation. In CVPR , pages 11977-11986, 2019.

Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log-likelihood estimation. In ICCV , pages 11025-11034, 2021.

Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. In CVPR , pages 10863-10872, 2019.

Ke Li, Shijie Wang, Xiang Zhang, Yifan Xu, Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. In CVPR , pages 1944-1953, 2021.

Yanjie Li, Sen Yang, Peidong Liu, Shoukui Zhang, Yunxiao Wang, Zhicheng Wang, Wankou Yang, and Shu-Tao Xia. Simcc: A simple coordinate classification perspective for human pose estimation. In ECCV , pages 89-106, 2022.

Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. In ICCV , pages 11313-11322, 2021.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755, 2014.

Zhengxiong Luo, Zhicheng Wang, Yan Huang, Liang Wang, Tieniu Tan, and Erjin Zhou. Rethinking the heatmap regression for bottom-up human pose estimation. In CVPR , pages 13264-13273, 2021.

Shishira R. Maiya, Sharath Girish, Max Ehrlich, Hanyu Wang, Kwot Sin Lee, Patrick Poirson, Pengxiang Wu, Chen Wang, and Abhinav Shrivastava. Nirvana: Neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling. In CVPR , pages 14378-14387, 2023.

Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, Zhibin Wang, and Anton van den Hengel. Poseur: Direct human pose regression with transformers. In ECCV , pages 72-88, 2022.

Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. Pose-guided feature alignment for occluded person re-identification. In ICCV , pages 542-551, 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , pages 405-421, 2020.

Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV , pages 483-499, 2016.

Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR , pages 3504-3515, 2020.

Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai. A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis. In NeurIPS , volume 34, pages 20002-20013, 2021.

George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In ECCV , pages 269-286, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , volume 32, 2019.

Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 , 2018.

Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Skeleton-based action recognition with directed graph neural networks. In CVPR , pages 7912-7921, 2019.

Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR , pages 1874-1883, 2016.

J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In CVPR , pages 20875-20886, 2023.

Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In CVPR , pages 10753-10764, 2021.

Chi Su, Jianing Li, Shiliang Zhang, Junliang Xing, Wen Gao, and Qi Tian. Pose-driven deep convolutional model for person re-identification. In ICCV , pages 3960-3969, 2017.

Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR , pages 5693-5703, Jun. 2019.

Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV , pages 529-545, 2018.

Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR , pages 1653-1660, 2014.

Jian Wang, Xiang Long, Yuan Gao, Errui Ding, and Shilei Wen. Graph-pcnn: Two stage human pose estimation with graph pose refinement. In ECCV , pages 492-508, 2020.

Dong Wei, Huaijiang Sun, Bin Li, Xiaoning Sun, Shengxiang Hu, Weiqing Li, and Jianfeng Lu. NeRM: Learning neural representations for high-framerate human motion synthesis. In ICLR , 2024.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Learning deep time-index models for time series forecasting. In ICML , pages 37217-37237, 2023.

Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV , pages 466-481, 2018.

Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai. Generative occupancy fields for 3d surface-aware image synthesis. In NeurIPS , volume 34, pages 20683-20695, 2021.

Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. In NeurIPS , volume 35, pages 38571-38584, 2022.

Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Keypoint localization via transformer. In ICCV , pages 11802-11812, 2021.

Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense prediction. In NeurIPS , volume 34, pages 7281-7293, 2021.

Dan Zeng, Yuhang Huang, Qian Bao, Junjie Zhang, Chi Su, and Wu Liu. Neural architecture search for joint human parsing and pose estimation. In ICCV , pages 11385-11394, 2021.

Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-aware coordinate representation for human pose estimation. In CVPR , pages 7093-7102, 2020.

Ziwei Zhang, Chi Su, Liang Zheng, and Xiaodong Xie. Correlating edge, pose with parsing. In CVPR , pages 8900-8909, 2020.

A Supplementary Implementation Details

As an implicit neural representation (INR) method, NerPE needs the correct coordinates to query the confidence scores of body joints during training and testing, but traditional coordinate transformation cannot meet our accuracy requirements. We give our data pre-processing and post-processing below.

A.1 Coordinate Transformation in Data Pre-processing

In the proposed NerPE, image pixels are expected to be located at the center of their corresponding regions, which is not hold after affine transformation due to the implementation at the code level. In fact, for each region, the interpolation result of its upper left corner in the original image is used as its RGB value, as shown in Fig. A1. As a result, there is a spatial offset between the cropped image and its coordinate system O o -X o -Y o . To solve this issue, we translate the 2D coordinates of body joints to an unbiased target coordinate system O u -X u -Y u to achieve alignment. Specifically, what needs to be done is to add 0.5 to the coordinates of body joints in O o -X o -Y o .

affine

transformation

pixel

coord

pixel

coord

Original image

Input image

X

A.2 Coordinate Decoding in Data Post-processing

The schematic diagram of coordinate decoding in NerPE is shown in Fig. A2. To determine the positions of body joints during inference, first the argmax operation is performed on the predicted heatmaps H to obtain a series of 0-based integral indices. Then, NerPE calculates the corresponding coordinates of these positions in O u -X u -Y u . Finally, these coordinates are transferred to O o -X o -Y o for mapping back to the original image. The entire process is formulated as p = ( argmax ( H ) + 0 . 5) · s -0 . 5 , where s represents the ratio of input resolution to heatmap resolution.

as the heatmap resolution increases

affine

transformation

argmax

Original image

X

stride>1

stride=1

stride<1

B Additional Visualization and Analysis

Here, we discuss in detail the local ensemble used in NerPE, and perform ablation on it as shown in Fig. A3. For the difference of local feature vectors z ∗ and the mutation of relative coordinates

Sho. (l)

Sho. (r)

Elb. (l)

Elb. (r)

Wri. (l)

Wri. (r)

Hip (l)

Hip (r)

Kne. (l)

Kne. (r)

Ank. (l)

Ank. (r)

Head

Neck

Input image

with LE

without LE

without LE

with LE

lar0152536073

08.00-

2

eiFc

c rel at the junctions between cells, the local ensemble uses bilinear interpolation to ensure that the confidence scores output by the network is continuous. This process is formulated as:

where z ∗ t refers to the four local feature vectors that are closest to the queried position. The predicted confidence scores are weighted based on the surrounded areas S t and their sum S = ∑ t S t . The use of local ensemble means that the sampling of queried positions is no longer limited to the interior of each cell, but the sampling range is expanded to twice the original to achieve overlapping. It can be found in Fig. A3 that the predicted heatmaps show discontinuity after removing the local ensemble.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's contributions and scope are stated in the abstract and introduction. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of this work and future research in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a thorough description of the proposed method in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Although we do not provide open access to our code during review, we will release it and include its URL in the abstract if the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings are detailed in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not report error bars or other statistical significance information.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the detailed efficiency comparison in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper fully conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not have the risks mentioned above.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets used in the paper are publicly available and properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: We will make the code publicly available upon acceptance of the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new heatmap representation for 2D human pose estimation. Prior approaches use a quantized representation of heatmaps, where a confidence score is assigned to each pixel. In addition to being dependent on the image resolution, this representation does not match the usual ground truth coordinates, which are continuous when working in image crops.\n\nThis paper proposes an implicit neural representation to address this limitation of prior works. Specifically, given 2D coordinates (which do not necessarily correspond to the center of a pixel), the introduced neural network outputs a confidence score for each keypoint. At inference, the coordinates of each keypoint are recovered using the maximum confidence value. The main contributions of this paper are the following: (1) The authors introduce NerPE, a novel implicit representation of heatmaps for continuous human pose estimation; (2) A progressive coordinate decoding method is introduced to recover 2D coordinates from continuous heatmaps.\n\nSTRENGTHS:\n- The approach is novel; to my knowledge, no prior work has used implicit representations for heatmaps.\n\n- The overall writing is good, and the paper reads nicely. The weakness of prior works addressed in this paper is clearly identified, and the proposed approach is well-justified.\n\n- The proposed approach yields consistent improvement over the SOTA methods, and the ablation study gives a good idea of the impact of choice designs.\n\nWEAKNESSES:\n- The Related Work section lacks discussions and comparisons with other approaches attempting to address the quantization error in heatmaps, such as [42,a,b,c,d,e,f,g]. The authors say that \"As far as we know, existing heatmap-based methods all belong to discrete heatmap regression\" (L104), but for instance, [b] predicts a float location using an offset within the original heatmap. Similar approaches using PixelShuffle operation were proposed [f]. In general, the introduction presents the problem of heatmap quantization as if prior works ignored it, but plenty of works attempted to address this problem.\n\n- I am unsure if having a continuous representation of heatmaps is useful. The ground truth is given in pixel coordinates on the full image. I agree that increasing the heatmap resolution until we reach the original image resolution is helpful, but is it useful to have a higher resolution than the original image? In the end, the ground truth is given in full pixel coordinates.\n\n- Some information important for reproducibility is missing. For instance, what are the training and validation datasets and the number of epochs? Even if the authors say they used \"standard settings\" I believe this is not enough to reproduce the results.\n\n- No information is provided regarding the material used for training and testing or the running times. This is particularly concerning as the method has a high computational cost. The authors propose an alternative decoding strategy that seems much less expensive, but comparing it with other methods and heatmap representations would be necessary.\n\n\n\n\n[a] Bulat, A., Sanchez, E., & Tzimiropoulos, G. (2021). Subpixel heatmap regression for facial landmark localization. BMVC\n\n[b] Lan, X., Hu, Q., & Cheng, J. (2021). Revisiting quantization error in face alignment. ICCV\n\n[c] Yu, B., & Tao, D. (2021). Heatmap regression via randomized rounding. PAMI\n\n[d] Papandreou, G., Zhu, T., Kanazawa, N., Toshev, A., Tompson, J., Bregler, C., & Murphy, K. (2017). Towards accurate multi-person pose estimation in the wild. CVPR\n\n[e] Luvizon, D. C., Picard, D., & Tabia, H. (2018). 2d/3d pose estimation and action recognition using multitask deep learning. CVPR\n\n[f] Wang, H., Liu, J., Tang, J., & Wu, G. (2023). Lightweight Super-Resolution Head for Human Pose Estimation. MM\n\n[g] Li, J., Bian, S., Zeng, A., Wang, C., Pang, B., Liu, W., & Lu, C. (2021). Human pose regression with residual log-likelihood estimation. ICCV\n\nQUESTIONS:\n(1) \"The meaning of the confidence scores output by current heatmap-based models has not been theoretically proven.\" (L43). What does this really mean? Is the NerPE representation \"theoretically proven\"?\n\n(2) In Equation (5), to my understanding, $f_W(i,j)$ and $f_b(i,j)$ are floats, and $z^*$ would be of dimension $C$? But isn't $H_{z^*}$ supposed to be a float since it is a confidence value?\n\n(3) \"We are surprised to find that the conclusion drawn is similar to the local implicit image function in [8] except for the additional normalization step.\" (L179). Why is that surprising? Isn't the process a bit similar?\n\n(4) Could we consider that the proposed method consists of doing sub-heatmaps until we reach a desired degree of precision?\n\n(5) The model is trained to output Gaussian or Laplacian heatmaps. However, at inference, there is no guarantee that the confidence distribution follows the same distribution. I think that this is a good point, as this brings flexibility. However, couldn't that raise issues for finding the argmax? For instance, the model could potentially \"hesitate\" between 2 different places when placing the center of the heatmap (the predicted distribution would look like a mixture of Gaussians, for instance). Did you observe such cases in practice, and could other strategies be explored instead of choosing the argmax?\n\n\nSide remark: Typo L15 \"desgin\"\n\nLIMITATIONS:\nLimitations of this work are briefly discussed in Section 5:\n\n- The limited experiments on network architecture and index embedding are not a problem, as this work aimed to focus on the heatmap representation.\n\n- The discussion on the necessity of predicting discrete heatmaps for computing an argmax is interesting, but it would have been more valuable if the authors had provided potential improvements and strategies on that point.\n\nThe potential negative societal impacts are not discussed but are rather limited. The authors could consider mentioning the potential misuse of pose estimation for intrusive surveillance or military applications. The environmental cost could also be discussed, especially since this method is more computationally expensive than prior works.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

comments: SUMMARY:
This paper proposes a new approach to predict continuous heatmap for keypoint localization. The proposed method adopts a MLP that receives position coordinates and corresponding feature and output the confidence score of this position. Then this method can query candidate points and select the point with maximum score as final results. Experiments on several keypoint localization benchmarks demonstrate the effectiveness of the proposed method.

STRENGTHS:
1.	The idea of introducing implicit neural representation to perform keypoint localization is new and can addresses the quantization error of existing heatmap-based method.
2.	This paper is well-written and easy to understand.

WEAKNESSES:
My major concern is about the decoding process of the proposed method. It seems that this method should evaluate a lots of points to select the best results, which will introduce much inference computational cost. The introduced progressive coordinate decoding can reduce the computational cost but can avoidably increase the decoding time due to multi-round decoding process. This paper should give a detailed efficiency comparison of existed methods such as heatmap (SimpleBaseline, HRNet, SimCC, ViTPose), regression (RLE), including GFLOPs, inference time (ms), GPU memory consumption under the same setting (all above methods release their codes, so it is not impossible to conduct such comparisons). These comparisons are necessary because it seems that the advantages of the proposed method is not obvious.

QUESTIONS:
In Weakness

LIMITATIONS:
This paper discusses its limitation in Section 5.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle 2D human pose estimation through a continuous heatmap representation. Specifically, instead of representing the heatmap as a grid of values, they use an implicit neural representation, where the coarse feature vector along with queried coordinates are fed to an MLP, which predicts the heatmap value for that requested position. This can in theory provide infinite resolution for the heatmap, allowing more accurate localization of its peak, compared to the usual discretized heatmaps that can lead to quantization errors due to the low resolution. The model is trained with randomized query points with Gaussian target. For test time, the authors propose a progressive refinement scheme where the highest-value area in the heatmap is successively queried at finer levels.\nExperiments are performed on COCO, MPII and CrowdPose with improvements over baselines, especially for evaluations with stricter thresholds.\n\nSTRENGTHS:\nThis is a creative and well-motivated approach. Whereas prior works often used heuristics such as moving the output point towards the second-highest heatmap pixel by a factor of 0.25 etc., this work is a more principled solution to producing arbitrary-resolution heatmaps.\n\nThe experiments are extensive, with three different backbones and three different datasets. The most convincing of these is the MPII experiment with the strict threshold, showing a clear improvement over comparable baselines. This is exactly the setting where one would expect to see the improvement. The other settings and dataset also see some quantitative improvement, which substantiates the claims of the paper.\n\nThere is little overhead in terms of parameter count and computational cost, and the technique can be applied to a wide range of approaches giving the potential for wide impact.\n\nThe writing has high quality and the structure is clear.\n\nWEAKNESSES:\nSome important alternative techniques from the literature are not discussed, and this impacts the conclusions.\nFirst, soft-argmax (integral regression) [1] was introduced specifically to enable continuous output that is not limited by the coarse downsampled grid. This would be an important baseline decoding method to use here. (There have been several extensions, including [2])\nSecond, offset regression such as the short-range offsets learned in PersonLab [3] would be another simple formulation allowing continuous output not limited by the grid.\n\nThe statement around L305 is not correct, the radius of the Gaussian does not have to be an integer for the gridded heatmap. There is no connection between these two, it is neither easier nor more appropriate to use integer-valued standard deviations for a Gaussian on a grid (alternatively, clarification is needed).\n\n[1] Sun et al., Integral Human Pose Regression, ECCV 2018\n[2] Gu et al., Bias-Compensated Integral Regression for Human Pose Estimation, PAMI 2023\n[3] Papandreou et al. PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model, ECCV 2018\n\nQUESTIONS:\nWhy is the proposed method not tested also on resolution 384x288? This would allow better comparability in Table 2. Further, why is the gap performance advantage significantly smaller on COCO test-dev compared to COCO val?\n\nWere other alternatives considered for the implicit function, such as learning a distance field?\nAs for the progressive refinement strategy, wouldn't it be useful to base the exploration on the gradient of heatmap value wrt query location? This seems cheaply differentiable and could give a more direct path towards the peak.\n\nSince the target is always a Gaussian, how does the network output at test time look? Are less certain predictions wider in distribution?\n\nAs the queries are given as coordinates to the MLP, would it be possible to query truncated joints outside the heatmap bounds?\n\nLIMITATIONS:\nThe authors provide an adequate \"Limitations\" section in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly studies the quantization problem of discrete representation of heatmap, especially in the case of low resolution. It proposes to use a continuous Implicit Neural Representation (coordinate-input MLP) to coarse-to-fine query to generate heatmap at arbitrary resolution, which achieves better performance than the current common heatmap method at lower resolution.\n\nSTRENGTHS:\n- Heatmap quantization error has always been an important research topic, esp. at not high resolution.\n\n- This paper is well-motivated and written in a simple and easy-to-understand way.\n\nWEAKNESSES:\nI will give the boardline first, and then want to see how the author responds to questions about method details, clarification of experimental settings, insufficient experiments, etc. before making a decision.\n\n[Method]\n\n- For training (Fig. 2), why not consider adding additional supervision in the area near the ground truth or even progressive coordinate decoding to refine the heatmap prediction near the ground truth? Will not this give better results?\n\n- So does the final 2D coordinate input into the INR MLP use positional embedding (L312)?\n\n- The bias of confidence estimation comes not only from discretization but also from heuristic confidence estimation [a]. From the gap (confidence) between AP (accuracy + confidence) and AR (accuracy) in Tab. 1, we could see that confidence calibration is still not good enough, and this limitation should be mentioned in the Sec. 5 Limitation & Future Work as well.\n\n[Experiments]\n\n- Does the backbone of this method use COCO/heatmap pre-trained or ImageNet checkpoint? Please describe in detail.\n\n- I don't understand the meaning of the OR/IR column (Tab. 1). Why is HRNet's OR/IR constant at 1/4 and does not change with the input size?\n\n- I don't seem to find GFLOPs, AP and AR results comparing different heatmap resolutions of the heatmap-based baseline method under a fixed input size? They may need to be added in Tab. 1.\n\n- In addition to 256x192, it is also necessary to conduct experiments on a higher resolution 384x288 standard setting to test the applicability of the method's solving quantization error. It seems that the improvement of the method at high resolution is minor.\n\n- How are the overhead savings and performance improvements on the newer heatmap-based method ViTPose [48]? Because ViTPose is powerful, and improving it will make this work more widely applicable.\n\n- There are also some methods that use the heatmap + offset prediction method [b], which also needs to be discussed.\n\n- If I understand correctly, I guess that progressive coordinate decoding may perform similarly to a fully generated heatmap in easier cases, and may only find local minima in more difficult cases (such as multi-peak, non-Gaussian non-uniform heatmaps). If this is the case, then splitting the test by difficulty (such as number of joints, human size, etc.) [c] may be more clear.\n\n- Since the values ​​don't match, are the settings of Tabs. 5 & 6 ablation studies (82.65) different from Tab. 3 (87.7)?\n\n- If the division in Tab. 5 is 16x16 or even higher resolution, the result should be improved further, though the cost gradually approaches the heatmap-based method? Could you add more reports in these regions (not necessarily done during the rebuttal)?\n\n- The author mentioned that the difference in local feature vectors at the cell junction will make the predicted heatmap not smooth, so they proposed to use bilinear interpolation of local features (Ls202-207). Could we perform an ablation study to see the difference in results before and after and visualize the heatmap difference? I think we may also further consider the idea of ​​sparse convolution to fuse neighborhoods.\n\n- It is recommended that the ablation study in Tab. 7 be performed more standardized on COCO 128x128 (not necessarily made during the rebuttal period) because of CrowdPose biases towards crowded situations. The 64x64 input resolution is too small and does not seem to have much application in practice.\n\n- Lack of heatmap visualization and comparisons with other methods\n\nReferences:\n\n[a] On the calibration of human pose estimation. ICML, 2024.\n\n[b] Towards accurate multi-person pose estimation in the wild. CVPR, 2017.\n\n[c] Benchmarking and error diagnosis in multi-instance pose estimation. ICCV, 2017.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nThe authors state their limitations as 1. the decoder structure not studied in depth, 2. discrete heatmaps still required to generate and considered as not elegant. Besides, some limitations mentioned in the weaknesses are suggested to include as well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}