Continuous Heatmap Regression for Pose Estimation via Implicit Neural Representation
Authors: Shengxiang Hu
Huaijiang Sun
Dong Wei
Xiaoning Sun
Jin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Human pose estimation
continuous heatmap regression
implicit neural representation
Primary area: machine_vision

Abstract: Heatmap regression has dominated human pose estimation due to its superior performance and strong generalization. To meet the requirements of traditional explicit neural networks for output form, existing heatmap-based methods discretize the originally continuous heatmap representation into 2D pixel arrays, which leads to performance degradation due to the introduction of quantization errors. This problem is significantly exacerbated as the size of the input image decreases, which makes heatmap-based methods not much better than coordinate regression on low-resolution images. In this paper, we propose a novel neural representation for human pose estimation called NerPE to achieve continuous heatmap regression. Given any position within the image range, NerPE regresses the corresponding confidence scores for body joints according to the surrounding image features, which guarantees continuity in space and confidence during training. Thanks to the decoupling from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution during inference without retraining, which easily achieves sub-pixel localization precision. To reduce the computational cost, we design progressive coordinate decoding to cooperate with continuous heatmap regression, in which localization no longer requires the complete generation of high-resolution heatmaps. The code is available at https://github.com/hushengxiang/NerPE.

Continuous Heatmap Regression for Pose Estimation via Implicit Neural Representation
Shengxiang Hu', Huaijiang Sun'; Dong Wei', Xiaoning Sun', Jin Jiin Wang"
Nanjing University of Science and Technology, Nanjing, China Nantong University, Nantong, China
(hushengxiang, jiang) wew.juust.uuu... edu.cn
Abstract
Heatmap regression has dominated human pose estimation due to its superior performance and strong generalization. To meet the requirements of traditional explicit neural networks for output form, existing heatmap-basee based methods discretize the originally continuous heatmap representation into 2D pixel arrays, which leads to performance degradation due to the introduction of quantization errors. This problem is significantly exacerbated the size of the input image decreases, which makes heatmap-basee methods not much better than coordinate regression on low-resolution images. In this paper, we propose novel neural representation for human pose estimation called NerPE to achieve continuous heatmap regression. Given any position within the image range, NerPE regresses the corresponding confidence scores for body joints according to the surrounding image features, which guarantees continuity in space and confidence during training. Thanks the decoupling from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution during inference without retraining, which easily achieves sub-pixel localization precision. To reduce the computational cost, we design progressive coordinate decoding to cooperate with continuous heatmap regression, in which localization no longer requires the complete generation of high-resolution heatmaps. The code is available at http://gghbbbb.commmuuuueugggggg
Introduction
Human pose estimation (HPE) a fundamental task in the field of computer vision, which is widely used in various human-centered applications [41, 29, 37, 9, 55, 53]. In multi-person pose estimation, the top-down framework is a mainstream two-stage pipeline, in which the person areas are firstly cropped out and then the body joints within them are located. According to the way to describe the positions of body joints, pose estimation methods can be further divided into coordinate regression [44, 43, 20, 22, 28] and heatmap regression [31, 48, 42, 24, 52]. Due to better performance brought by spatial encoding heatmap representation, heatmap regression has received more attention than coordinate regression. Recently, most heatmap- based methods [11, 51, 50] focus on the design of network structure but ignore the importance of heatmap generation. Only a few works [26, 23] have noticed the irrationality in the standard heatmap representation widely used in existing methods.
In the top-down framework, the ground-truti coordinates of body joints in the original image are mapped to the input plane of a keypoint detector, through the same affine transformation applied to the cropped image patch. To supervise the output of traditional explicit neural networks, the current heatmap generation strategy yields 2D pixel arrays to reflect the spatial distribution of body joints, which means that the originally continuous heatmap representation needs to be discretized as shown
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Diowte teotnop oppetentation of d'feeest masldaas
Continune heatrop ippertatttion
lce ruzeat on
Shapk Marune
Topat rags
Figure 1: Comparison of discrete and continuous heatmap representations. In heatmap- based methods, the Gaussian function is discretized to satisfy the form of 2D pixel arrays. As the resolution decreases, the impact of quantization errors on positioning accuracy increases significantly. In contrast, NerPE can regress confidence scores at any position via implicit neural representation.
in Fig. 1. Specifically, the center of the Gaussian kernel is placed at body joints to calculate the confidence scores on the grid points as the ground-truth heatmaps. Although this discrete heatmap representation has achieved great success, it introduces quantization errors since body joints after affine transformation may fall anywhere within the image range rather than just at these fixed grid points. For the output of explicit neural networks, the information loss caused by spatial sampling is difficult compensate through post-processing operations [31, 54]. With the reduction of the input resolution, this problem will be further exacerbated, which makes the performance of heatmap-based methods degraded even worse than some methods based on coordinate regression. Considering that the Gaussian function used for heatmap generation is inherently continuous, it is a natural idea to learn continuous heatmap representation to replace fixed spatial sampling for HPE.
Implicit neural representations (INRs) are proposed to parameterize a variety of continuous signals [30, 8, 7, 47, 39, 46] in computer vision. Unlike explicit neural networks that output specific structures (e.g., mesh sequence), INRs map an index to its corresponding value, enabling not only continuous representations but also greater flexibility in use. Obviously, by combining INR and heatmap regression, the pose estimation model is able to learn continuous confidence scores for all body joints at any position. Compared the discrete heatmap representation with fixed resolution, the introduction of INR avoids the invariant spatial sampling caused by discretization during training and can yield the predicted heatmaps at arbitrary resolution during testing.
In this paper, to improve the performance of heatmap-based HPE, we abandon the main culprit of quantization errors, namely discrete heatmap representation. Instead, we propose NerPE achieve continuous heatmap regression through a novel neural representation. Specifically, we generalize the upsampling factor in discrete sub-pixel convolution [38] to infinity to obtain continuous upsampling function with respect to 2D coordinate. NerPE learns its continuous heatmap representation at series of queried positions that are randomly sampled within the image range. For any position, the target likelihood at it is calculated from the target 2D pose and its absolute coordinate, and the estimated likelihood at it inferred from the local feature vector and its relative coordinate. Since confidence scores are learned at all positions within the image range during training, our method achieves better performance than discrete heatmap- based methods that focus only on fixed grid points. As the resolution of the input image decreases, the superiority of continuous heatmap representation over existing methods becomes more prominent.
Limited by the explicit neural representation of existing methods, the heatmap resolution for a given image cannot be changed once the network structure is determined. In contrast, another benefit that INR brings to heatmap regression is that the heatmap resolution is no longer correlated with the image resolution. This means that our method can flexibly generate the predicted heatmaps with different resolutions based on accuracy requirements without retraining. To speed up the inference when high-resolution heatmaps are required, we design progressive coordinate decoding method. Thanks to the decoupling of INR and spatial resolution, NerPE enables high-precise precision localization without the need to calculate the complete heatmaps. Specifically, low-resolution heatmaps are first output to determine the approximate locations of body joints. Subsequently, the area near the maximal activation is further retrieved an iterative manner. Notably, our method can be easily integrated into most heatmap-based methods. The contributions summarized as follows:

---

We propose NerPE to avoid the introduction of quantization errors during training and to output the predicted heatmaps at arbitrary resolution during inference. To our knowledge, we are the first to apply implicit neural representations to human pose estimation.
We design a progressive coordinate decoding method to derive the coordinates of body joints from continuous heatmap representation. in which our method achieves high-precision localization with low computational cost through coarse-to-fine retrieval.
We conduct extensive experiments on three pose estimation benchmarks: COCO [25], MPII [1], and CrowdPose [21]. The results show that NerPE significantly enhances existing heatmap-based methods and obtains superior performance on low-resolution input images.
2 Related Work
2.1 Human Pose Estimation
2D human pose estimation (HPE) aims to locate a series of anatomical keypoints to represent the human pose in the input image. Currently, coordinate regression [3, 22, 28] and heatmap regression [17, 45, 12] are the two main pose estimation paradigms, both of which have received widespread attention. Coordinate regression, also known as direct regression, relies on deep neural networks to explore the mapping between the input image and the coordinates of body joints. The high degree of nonlinearity difficult for optimization, which makes coordinate regression not perform well enough. As for heatmap regression, HPE is converted into a combination of multi-label keypoint classification and coordinate decoding post-processing. Benefiting from dense prediction, heatmap-based methods not only casily exploit visual cues around the target position, but also take the ambiguity of keypoint localization into account. This is why heatmap-basee methods are generally superior to those based on coordinate regression. Although heatmap- based methods have made steady progress, quantization errors are still a troubling problem, which is what our work is dedicated to solving.
2.2 Discrete Heatmap Regression
To be clear at first, the cause of quantization errors is discretization, not the introduction of heatmap representations this behavior itself. As far as we know, existing nautmap-based methods [48, 42, 24, 52] all belong to discrete heatmap regression, using 2D pixel arrays to describe the spatial distribution of body joints. There are a few works that attempt to compensate for the damage to performance caused by quantization errors. In [31], the coordinates of body joints are empirically determined as the position of moving a 0.25 pixel from the maximal to second maximal activation. DARK [54] implements Taylor series approximation for heatmap activation to locate body joints based on distribution information. Although these post-processing operations achieve sub-pixel localization precision, subjective assumptions make the inference so reliable, especially in low-resolution cases. Instead of remedying the shortcomings of discrete heatmap representation, we propose continuous heatmap regression preserve the continuity of ground-truth heatmaps.
2.3 Implicit Neural Representation
In response to the need to model continuous signals, implicit neural representations (INRs) aim to learn neural function that predicts the corresponding value according a given index. Encouraged by the success in 3D reconstruction [14, 32, 4] and generation [6. 33, 49], INR has been extended to some other tasks, including super-resolution [8, 10, 13], image generation [40, 5, 2], and video compression [7, 15, 27]. In the paper, to avoid the damage of discretization to heatmap regression, we approximate the continuous Gaussian function by multi-layer perceptrons (MLPs). Given coordinate within the image range, NerPE is designed to regress the confidence scores of all body joints at that position, in which the heatmap representation is free from the constraints of 2D pixel arrays [11, 51, 50]. As result, NerPE has the ability to output the predicted heatmaps at arbitrary resolution during inference. In terms of eliminating quantization errors, offset prediction [34, 19] has the same goal as our work, and it aims to perform unbiased estimation for the coordinates of body joints. However, the performance of offset prediction still affected by the resolution of the distance field. There is complementarity between the two techniques in HPE: offset prediction helps achieve more advanced decoding, and INR makes the distance field continuous. In order to highlight the superiority of INR, only regress confidence scores in this paper.
3

---

3
Proposed Method
In this section, we first discuss the hazards of discrete heatmap representation and the limitations of existing post-processing operations. Then, we introduce novel Neural representation for human Pose Estimation (NerPE) and illustrate the superiority of continuous heatmap regression.
3.1 Preliminary of Heatmap Representation
Given an input image R3xHixWi, heatmap-based models output 2D pixel arrays H of size K Wx reflect the spatial distribution of K body joints. To conform the output form of explicit neural networks, the continuous Gaussian function is discretized in the heatmap generation process, which leads the introduction of quantization errors and the reduction of positioning accuracy. Currently, there are two post-processe processins operations [31, 54] that are widely used to mitigate the negative effects of discretization.
The standard coordinate decoding method [31] is designed entirely according to experience. Based on the analysis of the model's performance, the position p at which the maximal activation H (cm) moves 0.25 pixel towards the second maximum H (c,) is taken as the final coordinate:
C2 Cm
p = Cy + 0.25
|[c,
C. Cmll2
The distribution- aware decoding method [54] assumes that the predicted heatmaps still conform to the Gaussian distribution. The Taylor series expansion to the quadratic term is implemented at the position Cm of the maximal activation as:
H (p)) (cm) H' (u cm) 5(u cm H (cm)(u- Cm))
where H²(-) and (.) respectively denote the first and second order derivatives of the Gaussian distribution close the predicted heatmaps. The position p of each body joint is determined by the Gaussian mean = Cm
(cm)))11f1
To achieve sub-pixel localization precision against quantization errors, the above post-processing operations impose subjective assumptions in coordinate decoding. Compared with the limited improvements brought by the above passive compensations, the most direct way solve quantization errors is to abandon the explicit neural network and learn a continuous heatmap representation.
3.2 Reformulation in a Continuous Form
From a structural and functional perspective, general pose estimation models consist of an encoder and decoder (also called "head" in some works). To reduce the computational cost and increase the receptive field, the image features are downsampled in the encoder. Correspondingly, the decoder increases the output resolution by including upsampling layers such as interpolation, deconvolution or sub-pixel convolution [38]. However, once the network structure is determined, the decoder in existing methods makes the model only output the predicted heatmaps with fixed resolution. Starting from sub-pixel convolution, we derive its continuous version via implicit neural representation (INR), known as a continuous upsampling function.
Using sub-pixel convolution as the final layer, the image features of size C x Hz x Wz are mapped into higher resolution heatmaps of size K x rHz x rW2 with an upscaling factor
H PS(W b),
where PS denotes PixelShuffle operation used reshape each element along the feature dimension into small 2D pixel arrays (called cell) corresponding to body joints. According to the spatial prior, the weight W and bias b in Eq. (3) are further expressed as:
Wi,1
Wi.r
Ti,1
b
W = Concat(
:
),
Concat(
(4)
W
Wr.
b
by.r
where the order of concatenation corresponds to the order of expansion in the PixelShuffle operation. These sub-weights Wey and sub-biases bi, can be viewed as the outputs of functions fw (i.j) and

---

Selection query positions
training
MLP cosec
Tragee
random sampling
(b) uniform sampling
gl confidaacc
Keypoint localization
Wy
OR
testing phose
(4) Mandard coordimate decoding
(d) progresarve coordinate decoding
Figure 2: Overview of NerPE. The network structure consists of general image encoder and an MLPPbased decoder. During training, we use random or uniform sampling to pick queried positions, and calculate their confidence scores via continuous heatmap generation. During testing, we can obtain the predicted heatmaps at arbitrary resolution by standard and progressive coordinate decoding.
j)( with a 2D index as argument, We use =" to refer to each element in the image features Z, called local feature vector. The confidence scores at (i.j) relative to is given by:
Hg. (i.j) = fw(1, * + fo j) - fo(z", (i.j)).
(5)
In order to ensure the consistency in description of sub- sub-pixel convolution with different upscaling factors, normalize the 2D index array composed of integers in (1,2,...,r)) into [-1, 1]:
o
0
0
j'
1114
0
(6)
As increases, the spatial sampling of discrete grid points become a continuous region covering all resolutions. Therefore, we obtain the expression for the continuous version of sub-pixel convolution Hz- (Cra)) = fo(=", Crat), and further replace the relative coordinate Gret with the absolute coordinate Cabs to get:
becomes denser. When r approaches co, the
Cabs Cz'
H (Cabs) = fo ==,
Scont1
where C.. is the center coordinate of cell H.,, and Scetl is the window size of cell H.. Starting from sub-pixel convolution, the conclusion drawn is similar to the local implicit image function in [8] except for the additional normalization step. In terms of network architecture, only replace the last few layers in existing heatmap-bas basen models with our continuous upsampling function, which fundamentally solves the problem of discretization in training.
3.3 Learning Continuous Heatmap Regression
Obviously, the discrete heatmap representation is a subcase of the proposed NerPE, where only several fixed positions (i.e., grid points) within the image range are trained and predicted. Thanks the decoupling of heatmap representation and spatial resolution, NerPE can comprehensively fit Gaussian or Laplace functions. The design of our method involves three aspects: continuous heatmap generation, uniform position sampling, and progressive coordinate decoding.
Continuous heatmap generation. Since the output of the network is no longer 2D pixel arrays like existing methods, we need calculate the ground-truth confidence scores for those positions being queried. In our method, the continuous heatmap representations based on Gaussian and Laplace distributions respectively are expressed relative Cubs as:
? and 1
Hot (Cabe) = gau
(8)
where Cgt is the ground-truth coordinates of body joints. Since the queried position can be anywhere the image range, the ground-truth heatmaps are continuous space and confidence. The loss function of continuous heatmap regression is:
L |H a (Cube)
(9)
5

---

Uniform position sampling. For heatmap learning based on INR, the selection of queried positions for training is critical to the model performance. On the one hand, in order fully take care of the entire heatmap plane, the sampling of queried positions during training should be evenly distributed over the image range. Thus, we divide each cell uniformly into vNz x vNz regions and perform random sampling within them, as shown in Fig. 2(b). In each training sample, Nz x Hz x Wz positions are picked out and their ground-truth confidence scores are calculated for supervision.
On the other hand, INR acts as a parameterized continuous function, which expects that spatially close positions should have similar confidence scores. Inside each cell, this easy to implement for deep neural networks. However, at the junctions between cells, the difference of local feature vectors 2 and the mutation of relative coordinates Crel make the network tend to produce discontinuous predictions. To solve this problem, we adopt the local ensemble in [8] to perform bilinear interpolation, where the sampling range is expanded to achieve overlapping.
Progressive coordinate decoding. The design of heatmap representation has been well discussed above, and more importantly, it ultimately serves keypoint localization. In order decode the coordinates of body joints through argmax, a straightforward way is to arrange the candidate points within the cells at specific density, so that NerPE can output the predicted heatmaps with the corresponding upsampling factor 2 1:
H = (o( Crxr),
(10)
where Hind is the distribution of body joints in the n-th cell of the input image, and Cyxr is relative coordinate matrix to indicate the candidate points. When the heatmap resolution is high enough, argmax is sufficient to deal with quantization errors without extra post-processing operations.
In existing methods, the likelihood of body joints at each position is derived simultaneously by an explicit neural network, which means there is choice but to output the entire predicted heatmaps. In contrast, NerPE can estimate confidence scores based on 2D coordinates in a serial manner thanks to the decoupling of INR from spatial resolution. We propose progressive coordinate decoding to reduce the computational cost by evading the complete generation of predicted heatmaps when the heatmap resolution is high, as shown in Fig. 2(d). Specifically, heatmaps to estimate the approximate locations of body joints. Then, the area near the maximal activation is iteratively sub-divided for coarse-to-fine to-fine retrieval. This decoding method only needs to calculate additional x K x (r + Po) x (r + Po) positions to achieve an equivalent resolution of r' Hz 2 x r'Z,, where is the number of iterations and po is the number of pixels overlapped.
first generate the resouusol
4 Experiments
In this section, we compare the performance of NerPE and discrete heatmap-based methods input images of different resolutions. The experimental results show the superiority of continuous heatmap regression and the hazards of discrete heatmap representation, especially in the case of low resolution.
4.1 Implementation Details
Network architecture. We adopt ResNet [16], HRNet [42] or TokenPose [24] as the backbone network, and resize the extracted image features to 8 8. The focus of this work is to provide a new perspective on heatmap regression through implicit neural representation (INR) rather than pursuing extreme performance. For clarity, we use a pure MLP structure as a decoder to implement continuous heatmap regression. Queried position embeddings, derived from 2D coordinates and their sinusoidal encodings, are concatenated with local feature vectors and then fed into the decoder. In each cell corresponding to a local feature vector, we collect Nz = 64 queried positions as training subjects. For continuous heatmap generation, in the Gaussian function is set to 0.06, which is close in proportion to the discrete heatmap representation. When not explicitly stated, the size of the predicted heatmaps is 256 256 by default. NerPE no longer needs those empirical operations to align flipped heatmaps [48] and shift decoded coordinates [31].
Optimization. In the main experimental results, the training settings of NerPE is consistent with the comparison methods [48, 42, 24) based on discrete heatmap regression. We use the Adam optimizer [18] for training, in which the learning rate is initialized to le - 3 and decreased to le 4 and le - 5. The data augmentation used includes random rotation, random scale, image flipping, and half body open-source machine learning, Py Torch [35].
cropping. All our experiments are conducted

---

Table 1: Comparisons on the COCO validation set. We report the performance of existing discrete methods and continuous NerPE different input resolutions. OR/IR: the ratio of output resolution to input resolution. SimBa: SimpleBaseline. The best results are marked in bold.
Input size
Method
OR/R
Params
AP
AR
SimBa [48]
1/4
34.0M
34.4
43.7
64 x 64
SimCC [23]
3/1
24.7M
39.3
48.4
ResNet- 50 [16]
Ours
4/1
29.6M
40.8 (11.5)
49.5 (11.1)
SimBa [48]
1/4
34.0M
60.3
67.6
128 128
SimCC [23]
3/1
25.0M
62.6
69.5
Ours
2/1
29.0M
63.3 (10.7)
70.1 (10.6)
SimBa [48]
174
34.0M
70.4
76.3
256 x 192
SimCC [23]
2/1
25.7M
70.8
76,8
Ours
1/1
28.4M
71.0 (10.2)
77.0 (10.2)
HRNet [42]
1/4
63.6M
48.5
57.8
HRNe: W48 [42]
64 x 64
SimCC [23]
3/1
63.7M
59.7
67.5
Ours
4/1
63.9M
62.5 (12.8)
70.0 (12.5)
HRNet [42]
1/4
63.6M
68.9
75.3
128 128
SimCC [23]
2/1
64.1M
72.0
77.9
Ours
2/1
64.4M
73.1 (11.1)
78.8 (10.9)
HRNet [42]
1/4
63.6M
75.1
80.4
256 x 192
SimCC [23]
2/1
66.3M
75.9
81.2
Ours
1/1
65.0M
76.1 (10.2)
81.3 (10.1)
DARK [54]
1/4
4.9M
57.1
64.8
TokenPose-s [24]
64 64
SimCC [23]
4.9M
62.8
70.1
Ours
4/1
5.4M
64.4 (11.6)
71.6 (11.5)
DARK [54]
1/4
5.2M
65.4
71.6
128 x 128
SimCC [23]
5.1M
70.4
76.4
Ours
2/1
5.5M
71.8 (t1.4)
77.7 (11.3)
DARK [54]
1/4
6.6M
72.5
78.0
256 x 192
SimCC [23]
5.5M
73.6
78.9
Ours
1/1
6.5M
73.9 (10.3)
79.1 (10.2)
Table 2: Comparisons on the COCO test-dev set. indicates the ground-trute tound confidence scores of body joints calculated by the Laplace function in continuous heatmap generation.
Method
Backbone
Input size
Params
AP
AP 50
AP75 75
AP M
APr.
AR
SimBa [48]
ResNet-152
384 x 288
68 68.6M
73.7
91.9
81.1
70.3
80.0
79.0
HRNet [42]
HRNet-W- W48
384 288
63.6M
75.5
92.5
83.3
71.9
81.5
80.5
TokenPose [24]
HRNet- W48
256 x 192
27.5M
75.1
92.1
82.5
71.7
81.1
80.2
SimCC [23]
HRNet- W48
384 288
76.0
92.4
83.5
72.5
81.9
81.1
Ourst
HRNet- W48
256 192
65.0M
75.6
92.5
83.4
72.0
81.6
80.6
Ours
HRNet-Wer W48
384 x 288
65.0M
76.2
92.6
83.6
72.8
82.0
81.2
Main Experimental results
To evaluate the value of continuous heatmap representation for human pose estimation (HPE), we perform NerPE with three backbones [16, 42, 24] at three input resolutions on the COCO validation set, as shown in Table 1. The experimental results show that our method achieves better performance, and the superiority over discrete heatmap-basee based methods increases the input resolution decreases. Thanks to INR's modeling of continuous signals, NerPE can still output fine and smooth heatmaps of body joints even with low-resolution image input. contrast, the discrete heatmap representation resorts to manually designed decoding methods [31, 54] to achieve sub-pixel accuracy, which suffers from significant performance degradation when quantization errors are too large. The comparison results on the COCO test-dev set are given in Table 2. These suggest that the accuracy and flexibility the model are greatly improved by only replacing the last few layers in the existing heatmap-baat based methods with the INR-based MLP.
Evaluation on COCO.

---

Table 3: Comparisons on the MPII dataset. The input resolution is 128 x 128 and the backbone is
HRNet-W32. As more stringent metric, PCKh @0.1 higher requirements for localization.
Method
Hea
Sho
Elb
Wri
Hip
Kne
Ank
PCKh@0.5
PCKh@0.1
HRNet [42]
96.6
94.7
87.3
81.7
86.4
82.3
78.0
87.3
26.5
DARK [54]
96.6
94.5
87.7
82.2
87.2
82.8
78.4
87.6
29.6
SimCC [23]
96.6
94.6
87.5
81.3
86.8
82.5
78.2
87.4
32.6
Ours
96.6
94.7
87.6
81.7
87.5
82.7
78.4
87.6
33.9
Ours
96.7
94.8
87.7
81.7
87.6
82.8
78.5
87.7
34.6
Table Comparisons on the CrowdPose dataset. For the same backbone HRNet- W32, the impact of heatmap representation is given in the standard (256 x 192) and low-resolution (64 x 64) cases.
Input size
Method
Continuity
AP
AP
AP75
APE
AP M
APr H
HRNet [42]
42.4
69.6
45.5
51.2
43.1
31.8
64 x 64
SimCC [23]
X
46.5
70.9
50.0
56.0
47.5
34.7
Ours
V
47.4
71.3
51.6
56.6
48.3
35.5
HRNet [42]
X
66.4
81.1
71.5
74.0
67.4
55.6
256 x
SimCC [23]
66.7
82.1
72.0
74.1
67.8
56.2
Ours
V
66.9
82.1
72.6
74.2
68.0
56.4
Evaluation on MPII. We compare our NerPE with representative discrete heatmap-based methods [42, 54, 23] on the MPII dataset, as shown in Table 3. the input size of 128 x 128, our method achieves better performance based on the same backbone HRNet-W33 The higher scores obtained on PCKh@0.1 indicate that NerPE's positioning of body joints closer to the ground truth.
Evaluation on CrowdPose. To evaluate the performance in crowded scenes, we test NerPE on the CrowdPose dataset (see Table 4), in which YoloV3 [36] is adopted the human detector. At the input size of 256 192, our method achieves superior performance with 66.9 AP. Thanks to the learned continuous heatmap representation, NerPE delivers performance gains on AP7s, more stringent metric. For the low-resolution case, the experimental results show that NerPE brings an improvement of 6. AP to HRNet and further expands its lead over discrete heatmap- -eatmap-based methods.
Progressive coordinate decoding. In the proposed NerPE, the heatmap resolution is not dependent on the input resolution with the help of INR. Therefore, our method can flexibly increase the heatmap resolution to reduce quantization errors during inference. The standard NerPE directly outputs the complete predicted heatmaps for keypoint localization, which will bring large amount of calculation when the heatmap resolution is high. To solve this problem, we propose the progressive coordinate decoding method and denote the corresponding version as NerPE-p. Specifically, NerPE-p first yields coarse heatmaps of size 32 x 32. Then, we iteratively divide the area near the maximal activation into 4 and set po to Given inputs of size 128 x 128 from MPII, comparison results of NerPE and NerPE-p are given in Fig. 3. As the heatmap resolution increases, has a diminishing marginal effect. Compared with
GALOPS
28.9
10.10
115110
150-25
Figure 3: Comparison of computational cost (left) and accuracy (right) different heatmap resolutions.
positioning accuracy is indeed improved the standard NerPE, the of our progressive coordinate decoding trades negligible performance degradation for a large reduction computational cost.
4.3 A Alation Study
Sample selection of INR. In NerPE, we let the INR-related network learn the continuous heatmap representation covering the entire image through uniform position sampling. We perform ablations on the sampling modes (w/ and w/o uniform) and evaluate the impact of two byper-parameter the division of cells Hz Wz (see Table 5) and the number of samples per cell Nz (see Table 6). The experimental results based on ResNet-50 at the input size of 128 x 128 show that the model

---

Table 5: Ablation study on different divisions of cells. The number of samples per cell set to 64 on MPII (PCKh@0.5) using ResNet-50.
Table 6: Ablation study on different number of samples per cell. The division of cells set to 8 x on MPII (PCKh@0.5), using ResNet-50.
division of cells Hz x Wz
num sample per cell Nz
Sampling
Sampling
2x2
4x4
8x8
16
64
w/o uniform
75.28
80.83
82.54
w/o uniform
80.98
82.13
82.54
w/ uniform
76.03
81.42
82.65
w/ uniform
81.45
82.49
82.65
Table 7: Ablation study on scale parameters for continuous heatmap generation. Experiments are performed on CrowdPose with input resolutions of 128 x 128. The backbone is HRNet- -32ettW33..
Scale parameter
AP
APso
AP
APg
APM M
APи
0.08
59.1
78.1
64.5
68,9
60.4
46.5
0.06
60.1
79.1
65.4
70.0
61.3
47.6
0.04
59.5
79.1
64.6
69.1
60.6
47.2
0.12
57.7
77.6
63.7
67.6
58.9
45.2
b
0.09
58.8
78.5
64.6
68.4
60.1
46.5
0.06
58.4
78.5
64.1
67.8
59.7
46.4
performance decreases when uniform position sampling not used. For the division of cells, if too sparse (e.g., 2 2), each local feature vector needs to be responsible for the prediction of larger area, which is challenging for the network to fit the continuous heatmap representation. Considering both performance and computational cost, x 8 is a better setting for NerPE. Furthermore, the number of samples per cell increases, our method achieves better performance. The reason is that, on the one hand, querying more positions makes each gradient descent more robust, On the other hand, cells are decomposed into smaller regions to make the sampling more uniform.
Study of heatmap generation. In the Gaussian function, is used as hyper-yppnee parameter to control the scale activation peaks. The difference between continuous and discrete heatmap representations has been discussed, shown in Fig. 1. Due to the existence of discretization in existing methods, commonly set to an integer to facilitate the generation the pixel-based Gaussian kernel (formally named standard biased encoding in [54]). In contrast, NerPE than discrete indices to describe the heatmap plane, and the setting of a is more flexible under our continuous heatmap representation. The same conclusion goes for b in the Laplacian function. We explore the influence scale parameters a and on keypoint localization, as shown in Table 7. The proposed NerPE achieves better performance when 0.06 and b 0.09 respectively.
continuous coordinates rather
4.4 Visualization
In order to more intuitively show the superiority of continuous heatmap representation, visualize the output of NerPE at different heatmap resolutions, as shown Fig. 4. Thanks to the decoupling of INR from spatial resolution, NerPE output the predicted heatmaps arbitrary resolution without changing the structure and retraining the network. In addition, we visualize the results of using Gaussian and Laplace functions for supervision in continuous heatmap generation in Fig. 5.
Gaussian
Laplace
Input
32 32
64 x 64
128 128
256 x 256
Sho.Il)
Elb.fr]
Kne..0)
Head
Figure 4: The predicted heatmap of knee(r) output by NerPE at different heatmap resolutions.
Figure 5: The output of NerPE supervised by different heatmap generation functions.

---

5 Conclusion
In this paper, to solve the quantization error issue plaguing heatmap regression, we propose an implicit neural representation method NerPE for 2D human pose estimation. According to the extracted image features, NerPE trains a simple series of queried positions, which makes the learned heatmap representation continuous in space and confidence. During inference, the decoupling from spatial resolution enables NerPE to output the predicted heatmaps arbitrary resolution. As result, our continuous heatmap regression achieves better performance than existing methods using discrete heatmap representation, especially in the case of low resolution. Last but not least, inspired by the flexibility of implicit neural representation, we design a progressive coordinate decoding method to speed up inference by avoiding the complete generation of predicted heatmaps when the desired heatmap resolution is quite high.
decoder to fit the Gaussian or Laplace functions at
Limitations and future work. The goal of this work is to explore the feasibility of using implicit neural representations (INRs) to achieve continuous heatmap regression for 2D human pose estimation. To highlight the superiority of NerPE over discrete heatmap representation, our INR-based decoder is designed to be as simple possible. In future work, will conduct in-depth research on the network structure to better utilize the characteristics INRs.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 62176125, Grant 61772272 and Grant 62406143, and in part by the Natural Science Foundation of Jiangsu Province under Grant BK20241468.
References
[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, pages (1886-6693.00 2014.
[2] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Lue Van Gool. Pix2nerf: Unsupervised conditional p-gan for single image neural radiance fields translation. In CVPR, pages 3981-3990, 2022.
[3] Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik, Human pose estimation with iterative error feedback. In CVPR, pages
2016.
[4] Rohan Chabra, Jan E. Lenssen, Eddy 1lg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors detailed 3d reconstruction. In ECCV, pages 608 000,-22,222 2020.
[5] Lucy Chai, Michaël Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. In ECCV, pages 170-188, 2022.
[6] Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, pages 5799-5809, 2021.
[7] Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. In NewrlPS, volume 34, pages 21557 -1568, 2021.
[8] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In CVPR, pages 8628 8638,66888880 2021.
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution
sicleton-based action recognition. In ICCV, pages 13359-13368, 2021.
[10] Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning vidco implicit neural representation for continuous space-time super-resolution. In CVPR, pages 2047-2057, 2022.
[11] Bowen Cheng, Bin Xiao, Jingdong Wang. Honghui Shi, Thomas S. Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up up human pose estimation. In CVPR, pages 2020.
10

---

[12] Runyang Feng. Yixing Gao, Tze Ho Elden Tse. Xueqing Ma. and Hyung Jin Chang. Diffpose: Spa- tiotemporal diffusion model for video-based human pose estimation. In ICCV, pages 14861-14872. 2023.
[13] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In CVPR, pages 10021-10030, 2023.
[14] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. CVPR. pages 4857-4866, 2020.
[15] Sharath Girish, Abhinav Shrivastava, and Kamal Gupta. Shacira: Scalable hash-grid compression for implicit neural representations. In ICCV, pages 17513- 115224, 2023.
[16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. pages 770-778, 2016.
[17] Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang. The devil is in the details: Delving into unbiased data processing for human pose estimation. In CVPR, pages 5700-5709, 2020.
[18] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[19] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi, Pifpaf: Composite fields for human pose estimation. In CVPR, pages 11977-11986, 2019.
[20] Jiefeng Li, Siyuan Bian, Ailing Zeng. Can Wang, Bo Pang. Wentao Liu, and Cewu Lu. Human pose regression with residual likeliikdd estimation. ICCV, pages 11025-1 1034, 2021.
[21] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu, Crowdpose: Efficient crowded scenes pose estimation and new benchmark. In CVPR, pages 10862-10872, 2019.
[22] Ke Li. Shijie Wang, Xiang Zhang. Yifan Xu. Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. CVPR, pages 1944-1953, 2021.
[23] Yanjie Li, Sen Yang, Peidong Liu, Shoukui Zhang, Yunxiao Wang, Zhicheng Wang, Wankou Yang. and Shu-Tao Xia. Simee: A simple coordinate classification perspective for human pose estimation. In ECCV, pages 89-106, 2022.
[24] Yanjie Li, Shoukui Zhang, Zhicheng Wang. Sen Yang. Wankou Yang. Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. In ICCV, pages 11313- 11322, 2021.
[25] Tsung- Yi Lin, Michacl Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, pages 740-755, 2014.
[26] Zhengxiong Luo., Zhicheng Wang, Yan Huang, Liang Wang. Tieniu Tan. and Erjin Zhou. Rethinking the heatmap regression for bottom-up human pose estimation. In CVPR, pages 13264 13273, 2021.
[27] Shishira R. Maiya, Sharath Girish, Max Ehrlich. Hanyu Wang, Kwot Sin Lee, Patrick Poirson, Pengxiang Wu, Chen Wang, and Abhinav Shrivastava. Nirvana: Neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling. In CVPR, pages 14378-14387, 2023.
[28] Weian Mao, Yongtao Ge. Chunhua Shen, Zhi Tian, Xinlong Wang, Zhibin Wang, and Anton van den Hengel. Poseur: Direct human pose regression with transformers. In ECCV, pages 72-88, 2022.
[29] Jiaxu Miao. Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. Pose-guided feature alignment for occluded person re-identification. In ICCV, pages 542-551, 2019.
[30] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-421, 2020.
[31] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, pages
2016.
[32] Michael Niemeyer, Lars Mcscheder, Michael Oechsle, and Andreas Geiger. Diffcrentiable volumctric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, pages 3504-3515, 2020.
[33] Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai. A shading guided generative implicit model for shape-aoe aacprrccuurate 3d aw-ae image synthesis. In NewrIPS, volume 34, pages 2021.
11

---

[34] George Papandreou, Tyler Zhu, Liang-Chiel Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. Personlab: Person pose estimation and instance segmentation with bottom-up, part-based, geometric embedding model. ECCV. pages 269-286, 2018.
[35]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, volume 32, 2019.
[36] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 1rXiv:884.002777, 2018.
[37] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Skeleton-based action recognition with directed graph neural networks. In CVPR, pages 7912-7921, 2019.
[38] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, pages 1874-1883, 2016.
[39] Ryan Shue, Fric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein, 3d neural field generation using triplane diffusion. In CVPR, pages 20875-20886, 2023.
[40] Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In CVPR, pages 10753 10764, 2021.
[41] Chi Su, Jianing Li, Shiliang Zhang. Junliang Xing, Wen Gao, and Qi Tian. Pose-driven deep convolutional model for person re-identification. In ICCV, pages 3960-3969, 2017.
[42] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, pages 5693-5703, Jun. 2019.
[43] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, pages 529 545, 2018.
[44] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR, pages 1653-1660, 2014.
[45] Jian Wang. Xiang Long, Yuan Gao, Errui Ding, and Shilei Wen. Graph-penn: Two stage human pose estimation with graph pose refinement. In ECCV, pages 492-508. 2020.
[46] Dong Wei, Huaijiang Sun, Bin Li, Xiaoning Sun, Shengxiang Hu, Weiqing Li, and Jianfeng Lu. NeRM: Learning neural representations for high-framerate human motion synthesis. In ICLR, 2024.
[47] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Learning deep time-index models for time series forecasting. In ICML, pages 37217-37237, 2023.
[48] Bin Xiao, Haiping Wu, and Yichen Wei. Simple haselines for human pose estimation and tracking. In ECCV, pages 466-481, 1,, 2018.
[49] Xudong Xu, Xingang Pan, Dahua Lin, and Dai. Generative occupancy fields for surface-aware image synthesis. In NewrIPS, volume 34, pages 20683-20695, 2021.
[50] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer basclines for human pose estimation. In NeurlPS, volume 35, pages 38571-38584, 2022.
[51] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Keypoint localization via transformer. In ICCV, pages 1180--18122 2021.
[52] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang. Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense prediction. NeurlPS, volume 34, pages 7281-7293, 2021.
[53] Dan Zeng, Yuhang Huang, Qian Bao, Junjie Zhang, Chi Su, and Wu Liu. Neural architecture search for joint human parsing and pose estimation. In ICCV, pages 11385-11394, 2021.
[54] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye. and Ce Zhu. Distribution-aware coordinate representation for human pose estimation. In CVPR, pages 7093-7102, 2020.
[55] Ziwei Zhang, Chi Su, Liang Zheng, and Xiaodong Xie. Correlating edge, pose with parsing. In CVPR, pages 3900-8909, 2020.
12

---

A
Supplementary Implementation Details
As an implicit neural representation (INR) method, NerPE needs the correct coordinates query the confidence scores of body joints during training and testing, but traditional coordinate transformation cannot meet our accuracy requirements. We give our data pre- pre-processing and post- processog below.
A.1 Coordinate Transformation in Data Pre-processing
In the proposed NerPE, image pixels are expected to be located at the center of their corresponding regions, which is not hold after affine transformation due the implementation the code level. In fact, for each region, the interpolation result of its upper left corner in the original image is used as its RGB value, shown in Fig. Al. As result, there is spatial offset between the cropped image and its coordinate system Oo-Xo-Yo. To solve this issue, we translate the 2D coordinates of body joints an unbiased target coordinate system Ou-X,-Y, achieve alignment. Specifically, what needs be done is to add 0.5 to the coordinates of body joints in Oo-Xo-Y.
Transformation
Original image
Input image
Figure Al: Pre- processing of NerPE. Due to differences in the affine transformations performed on pixels and coordinates, they are misaligned in the input image after standard data transformation. Therefore, need to map the coordinates into -Y, to achieve alignment.
A.2 Coordinate Decoding in Data Post-processing
The schematic diagram of coordinate decoding in NerPE is shown in Fig. A2. To determine the positions of body joints during inference, first the argmax operation is performed on the predicted heatmaps // to obtain series of 0-based integral indices. Then, NerPE calculates the corresponding coordinates of these positions in D,. Finally, these coordinates are transferred to 0 for mapping back the original image. The entire process is formulated as p (argmax( H) + 0.5) - 0.5. where represents the ratio of input resolution to heatmap resolution.
the Treat oommatton
orgmax
stride 1-
Atride =1
stride <1
Original Image
the Aww.naaanaaggoooo resolution mnrrases
Figure A2: Post-processing of NerPE. Since the affine transformation is established between the cropped image and the original image, we need convert the 0- O-based integral indices calculated by argmax into the coordinates in O,.X,-Y...
B Additional Visualization and Analysis
Here, we discuss in detail the local ensemble used NerPE, and perform ablation on as shown in Fig, A3. For the difference of local feature vectors 2 and the mutation of relative coordinates
13

---

with
tithee LE
Figure A3: Visualization of qualitative ablation on local ensemble. The prediction of activation peaks the key to heatmap-based pose estimation. When the activation peaks appear at the junction between cells, the confidence scores show obvious discontinuity without local ensemble (LE).
Cral at the junetions between cells, the local ensemble uses bilinear interpolation to ensure that confidence scores output by the network is continuous. This process is formulated as:
St 2 Cabs Cz2 Scell /2
(Cabs)) = E
fo
S
(00.01.100.11)
where of refers to the four local feature vectors that are closest to the queried position. The predicted confidence scores weighted based the surrounded areas S; and their sum S Σ, St. The use of local ensemble means that the sampling of queried positions is no longer limited to the interior of each cell, but the sampling range is expanded to twice the original achieve overlapping. It can be found in Fig. A3 that the predicted heatmaps show discontinuity after removing the local ensemble.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions and scope are stated in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work and future research in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide thorough description of the proposed method in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: Although we do not provide open access to our code during review, we will release and include URL in the abstract if the paper accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
Policyy)) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: halliii/ggiee//eeee
/ /nips...
ssiisnin swissssliccyyyyyc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental settings detailed in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: The paper does not report error bars or other statistical significance information.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the detailed efficiency comparison in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tp://eeeu..eeeeeeeeeee http://wwrrs..cccpbiiieeeeeeeeee htte/publcc/eeeee cew.iiiscciiiiiieeee
ee ines?
Answer: [Yes]
Justification:
The paper fully conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not have the risks mentioned above.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The datasets used in the paper are publicly available and properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will make the code publicly available upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "SpIqaRf2Ei", "reviewer": "Reviewer_xx7e", "summary": "This paper mainly studies the quantization problem of discrete representation of heatmap, especially in the case of low resolution. It proposes to use a continuous Implicit Neural Representation (coordinate-input MLP) to coarse-to-fine query to generate heatmap at arbitrary resolution, which achieves better performance than the current common heatmap method at lower resolution.", "questions": "Please see the weaknesses.", "limitations": "The authors state their limitations as 1. the decoder structure not studied in depth, 2. discrete heatmaps still required to generate and considered as not elegant. Besides, some limitations mentioned in the weaknesses are suggested to include as well.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Heatmap quantization error has always been an important research topic, esp. at not high resolution.\n\n- This paper is well-motivated and written in a simple and easy-to-understand way.", "weaknesses": "I will give the boardline first, and then want to see how the author responds to questions about method details, clarification of experimental settings, insufficient experiments, etc. before making a decision.\n\n[Method]\n\n- For training (Fig. 2), why not consider adding additional supervision in the area near the ground truth or even progressive coordinate decoding to refine the heatmap prediction near the ground truth? Will not this give better results?\n\n- So does the final 2D coordinate input into the INR MLP use positional embedding (L312)?\n\n- The bias of confidence estimation comes not only from discretization but also from heuristic confidence estimation [a]. From the gap (confidence) between AP (accuracy + confidence) and AR (accuracy) in Tab. 1, we could see that confidence calibration is still not good enough, and this limitation should be mentioned in the Sec. 5 Limitation & Future Work as well.\n\n[Experiments]\n\n- Does the backbone of this method use COCO/heatmap pre-trained or ImageNet checkpoint? Please describe in detail.\n\n- I don't understand the meaning of the OR/IR column (Tab. 1). Why is HRNet's OR/IR constant at 1/4 and does not change with the input size?\n\n- I don't seem to find GFLOPs, AP and AR results comparing different heatmap resolutions of the heatmap-based baseline method under a fixed input size? They may need to be added in Tab. 1.\n\n- In addition to 256x192, it is also necessary to conduct experiments on a higher resolution 384x288 standard setting to test the applicability of the method's solving quantization error. It seems that the improvement of the method at high resolution is minor.\n\n- How are the overhead savings and performance improvements on the newer heatmap-based method ViTPose [48]? Because ViTPose is powerful, and improving it will make this work more widely applicable.\n\n- There are also some methods that use the heatmap + offset prediction method [b], which also needs to be discussed.\n\n- If I understand correctly, I guess that progressive coordinate decoding may perform similarly to a fully generated heatmap in easier cases, and may only find local minima in more difficult cases (such as multi-peak, non-Gaussian non-uniform heatmaps). If this is the case, then splitting the test by difficulty (such as number of joints, human size, etc.) [c] may be more clear.\n\n- Since the values ​​don't match, are the settings of Tabs. 5 & 6 ablation studies (82.65) different from Tab. 3 (87.7)?\n\n- If the division in Tab. 5 is 16x16 or even higher resolution, the result should be improved further, though the cost gradually approaches the heatmap-based method? Could you add more reports in these regions (not necessarily done during the rebuttal)?\n\n- The author mentioned that the difference in local feature vectors at the cell junction will make the predicted heatmap not smooth, so they proposed to use bilinear interpolation of local features (Ls202-207). Could we perform an ablation study to see the difference in results before and after and visualize the heatmap difference? I think we may also further consider the idea of ​​sparse convolution to fuse neighborhoods.\n\n- It is recommended that the ablation study in Tab. 7 be performed more standardized on COCO 128x128 (not necessarily made during the rebuttal period) because of CrowdPose biases towards crowded situations. The 64x64 input resolution is too small and does not seem to have much application in practice.\n\n- Lack of heatmap visualization and comparisons with other methods\n\nReferences:\n\n[a] On the calibration of human pose estimation. ICML, 2024.\n\n[b] Towards accurate multi-person pose estimation in the wild. CVPR, 2017.\n\n[c] Benchmarking and error diagnosis in multi-instance pose estimation. ICCV, 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Below we address all the concerns.\n\n***W1: For training (Fig. 2), why not consider adding additional supervision ...***\n\nTheoretically, additional supervision in the area near the ground truth can improve the performance of NerPE, but it also applies to discrete heatmap regression, which does not highlight the advantage of continuous representation. As for progressive coordinate decoding, it is only performed during inference.\n\n***W2: So does the final 2D coordinate input into the INR MLP use positional embedding (L312) ?***\n\nWe use a common high-frequency function $\\gamma(x)=(\\cdots,\\sin (2^{i} \\pi x),\\cos(2^{i} \\pi x),\\cdots)$, to encode coordinates as sinusoidal embeddings. The relevant description will be added to the paper.\n\n***W3: The bias of confidence estimation comes not only from discretization but also ...***\n\nSince NerPE is designed for quantization errors, the performance gains from confidence calibration over existing methods increase as the input resolution decreases. Our continuous representation does not conflict with the correction for heuristic confidence estimation. We will mention this in  Limitation & Future Work.\n\n***W4: Does the backbone of this method use COCO/heatmap pre-trained or ImageNet checkpoint?***\n\nNerPE has the same backbone and loss function as its baseline. Therefore, the training process is consistent with its baseline, where the model is initialized with the pre-trained weights from ImageNet.\n\n***W5: I don't understand the meaning of the OR/IR ...***\n\nOR is the output resolution and IR is the input resolution, which is explained in the caption of Tab. 1. Since HRNet belongs to explicit neural representation, once its network structure is determined, OR/IR becomes a constant.\n\n***W6: I don't seem to find GFLOPs, AP and AR results ...***\n\nWe report the efficiency and performance in **Tab. A3 of the attached PDF**. The high amount of computation is the disadvantage of implicit neural representations (INR) when generating high-resolution signals. The results show that we reduce the GFLOPs from 73.6 to 9.5 by progressive coordinate decoding.\n\n***W7 & W8: It is also necessary to conduct experiments on a higher resolution ... & How are the overhead savings and performance improvements on ViTPose ?***\n\nQuantization errors caused by discretization decrease as the input resolution increases. Naturally, NerPE achieves greater improvements at low resolution than at high resolution. When the input resolution is 388×284, the accuracy of NerPE with HRNet and ViTPose as the backbone on the COCO test-dev set is as follows.\n\n| Backbone | AP  | AP$_{50}$ | AP$_{75}$ | AR  |\n| --- | --- | --- | --- | --- |\n| HRNet-W48 | 76.2 | 92.6 | 83.6 | 81.2 |\n| ViTPose-B | 76.8 | 92.6 | 84.3 | 81.8 |\n\n***W9: There are also some methods that use the heatmap + offset prediction method ...***\n\nThe heatmap + offset prediction method can indeed achieve sub-pixel positioning accuracy but it still depends on the quality of heatmap regression. When the resolution of heatmaps and offset fields is not high enough, its performance is also unsatisfactory. Therefore, our method not only does not form a competitive relationship with it but can even be combined with it.\n\n***W10: If I understand correctly, I guess that progressive coordinate decoding ...***\n\nI agree. The intention of progressive coordinate decoding is to find a simple way to accelerate inference at the cost of slight performance degradation, as reported in Fig. 3. Indeed, thanks to decoupling from heatmap resolution, there are some better potential implementations for coordinate decoding, as the reviewer mentioned.\n\n***W11: Since the values ​​don't match, are the settings of Tabs. 5 & 6 ablation studies (82.65) different from Tab. 3 (87.7)?***\n\nThe backbone of NerPE in Tab.3 is HRNet-W32 and in Tabs. 5 & 6 is ResNet-50, which is given in the description of the corresponding experiments but may not be eye-catching enough. We will highlight it in the caption.\n\n***W12: If the division in Tab. 5 is 16x16 or even higher resolution, the result should be ...***\n\nThe results to divide the cells into 16×16 and 64×64 are shown below. Combined with Tab. 3, we find that 16×16 is indeed a better choice than 8×8. When it continues to increase to 64×64, the positioning accuracy decreases. This is because the over-fine division causes the degradation of NerPE. Since the confidence score in each region changes marginally, INR is more focused on image features than relative coordinates.\n\n|     | 16×16 | 64×64 |\n| --- | --- | --- |\n| w\\o uniform | 83.13 | 82.17 |\n| uniform | 83.07 | 82.25 |\n\n***W13: The author mentioned that the difference in local feature vectors at the cell junction will ...***\n\nThe local ensemble is quantitatively proven to be effective for local INR in its original paper. In **Fig. A1 of the attached PDF**, we supplement the qualitative ablation of local resemble by visualization. Sparse convolution is not suitable for INR, because the positions queried during training do not spatially meet the form of 2D pixel arrays.\n\n***W14: It is recommended that the ablation study in Tab. 7 be performed more standardized on COCO 128x128 ...***\n\nCrowdPose has higher requirements for heatmap generation. The $\\sigma$ of Gaussian distribution should not be set too large, otherwise it will be difficult to distinguish the same kind of joints from different instances when they are close. Therefore, CrowdPose is more worthy of study to determine the scale parameters $\\sigma$ and $b$. Thanks for the reviewer's suggestion, we will perform ablation with input size of 128x128.\n\n***W15: Lack of heatmap visualization and comparisons with other methods.***\n\nCompared with the existing methods with fixed output sizes, NerPE can generate the predicted heatmaps at arbitrary resolutions. Without retraining, visualization of NerPE's outputs at different resolutions is given in **Fig. A2 of the attached PDF**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DC4BsE8122", "reviewer": "Reviewer_k3b9", "summary": "The authors tackle 2D human pose estimation through a continuous heatmap representation. Specifically, instead of representing the heatmap as a grid of values, they use an implicit neural representation, where the coarse feature vector along with queried coordinates are fed to an MLP, which predicts the heatmap value for that requested position. This can in theory provide infinite resolution for the heatmap, allowing more accurate localization of its peak, compared to the usual discretized heatmaps that can lead to quantization errors due to the low resolution. The model is trained with randomized query points with Gaussian target. For test time, the authors propose a progressive refinement scheme where the highest-value area in the heatmap is successively queried at finer levels.\nExperiments are performed on COCO, MPII and CrowdPose with improvements over baselines, especially for evaluations with stricter thresholds.", "questions": "Why is the proposed method not tested also on resolution 384x288? This would allow better comparability in Table 2. Further, why is the gap performance advantage significantly smaller on COCO test-dev compared to COCO val?\n\nWere other alternatives considered for the implicit function, such as learning a distance field?\nAs for the progressive refinement strategy, wouldn't it be useful to base the exploration on the gradient of heatmap value wrt query location? This seems cheaply differentiable and could give a more direct path towards the peak.\n\nSince the target is always a Gaussian, how does the network output at test time look? Are less certain predictions wider in distribution?\n\nAs the queries are given as coordinates to the MLP, would it be possible to query truncated joints outside the heatmap bounds?", "limitations": "The authors provide an adequate \"Limitations\" section in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a creative and well-motivated approach. Whereas prior works often used heuristics such as moving the output point towards the second-highest heatmap pixel by a factor of 0.25 etc., this work is a more principled solution to producing arbitrary-resolution heatmaps.\n\nThe experiments are extensive, with three different backbones and three different datasets. The most convincing of these is the MPII experiment with the strict threshold, showing a clear improvement over comparable baselines. This is exactly the setting where one would expect to see the improvement. The other settings and dataset also see some quantitative improvement, which substantiates the claims of the paper.\n\nThere is little overhead in terms of parameter count and computational cost, and the technique can be applied to a wide range of approaches giving the potential for wide impact.\n\nThe writing has high quality and the structure is clear.", "weaknesses": "Some important alternative techniques from the literature are not discussed, and this impacts the conclusions.\nFirst, soft-argmax (integral regression) [1] was introduced specifically to enable continuous output that is not limited by the coarse downsampled grid. This would be an important baseline decoding method to use here. (There have been several extensions, including [2])\nSecond, offset regression such as the short-range offsets learned in PersonLab [3] would be another simple formulation allowing continuous output not limited by the grid.\n\nThe statement around L305 is not correct, the radius of the Gaussian does not have to be an integer for the gridded heatmap. There is no connection between these two, it is neither easier nor more appropriate to use integer-valued standard deviations for a Gaussian on a grid (alternatively, clarification is needed).\n\n[1] Sun et al., Integral Human Pose Regression, ECCV 2018\n[2] Gu et al., Bias-Compensated Integral Regression for Human Pose Estimation, PAMI 2023\n[3] Papandreou et al. PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model, ECCV 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. Below we address all the concerns.\n\n***W1: Some important alternative techniques from the literature are not discussed, and this impacts the conclusions. First, ...***\n\nHere we analyze and compare our proposed method HerPE with the alternative techniques mentioned by the reviewer. As for Integral and its extension methods, from the perspective of supervision signals, they are essentially classified as coordinate regression, which is why they are not troubled by quantization errors. The original intention of these methods to construct a heatmap-like form is to narrow the gap between coordinate regression and heatmap regression, rather than to overcome the limitations of the coarse downsampled grid. In contrast, our work aims to propose a more advanced heatmap representation scheme to improve the performance of most heatmap-based methods. PersonLab consists of heatmap regression and offset prediction. Although PersonLab can achieve sub-pixel positioning with the help of offset prediction, its performance still depends on the quality of heatmap regression. When the output resolution of the heatmap and offset field is not high enough, the performance of PersonLab is also unsatisfactory. Therefore, our continuous heatmap representation also applies to PersonLab.\n\n***W2: The statement around L305 is not correct, the radius of the Gaussian does not have to be an integer for the gridded heatmap ...***\n\nThere is a mismatch between theory and practice. Most studies on heatmap-based human pose estimation only describe heatmap generation at a conceptual level using words or formulas, without discussing implementation in their papers. According to the official open source code, placing a fixed Gaussian kernel on the heatmap plane (i.e., integer Gaussian radius) is a standard operation in many influential works (such as SimpleBaseline, HRNet, HRFormer, ViTPose). To eliminate ambiguity, we will emphasize this detail in our paper and state that not all heatmap-based methods have this problem.\n\n***Q1: Why is the proposed method not tested also on resolution 384x288? This would allow better comparability in Table 2. Further, ...***\n\nWe report the results of NerPE on the COCO test-dev set with the input resolution of 384x288 as follows, to supplement Tab. 2. The reason for the different performance gap between NerPE and the comparison methods is that the input resolution is the same in Tab. 1 and different in Tab. 2.\n\n| Backbone | AP  | AP$_{50}$ | AP$_{75}$ | AP$_{M}$ | AP$_{L}$ | AR  |\n| --- | --- | --- | --- | --- | --- | --- |\n| HRNet-W48 | 76.2 | 92.6 | 83.6 | 72.8 | 82.0 | 81.2 |\n\n***Q2: Were other alternatives considered for the implicit function, such as learning a distance field? As for the progressive refinement strategy, ...***\n\nThere is no competition between our work and technologies such as distance fields. The purpose for us introducing implicit neural representations in human pose estimation is to achieve the transition from discrete to continuous. We use heatmap representation as the initial object to demonstrate the benefits of continuity, and the achievements can be generalized to other types of values ​​with 2D pixel arrays, also applicable to the distance field. In fact, the reviewer’s suggestion to use the gradient of heatmap value is to combine heatmap regression and offset prediction from the perspective of implicit neural representations.\n\n***Q3: Since the target is always a Gaussian, how does the network output at test time look? Are less certain predictions wider in distribution ?***\n\nThe trained NerPE is able to represent continuous heatmap distributions. The most direct way to obtain the coordinates of body joints is to query the confidence scores on the grid points according to the required resolution. Without retraining, visualization of the network output at different resolutions is given in **Fig. A2 of the attached PDF**.\n\n***Q4: As the queries are given as coordinates to the MLP, would it be possible to query truncated joints outside the heatmap bounds ?***\n\nUnfortunately, in the current experimental setup, it is not possible to localize body joints that are outside the heatmap bounds due to differences in data distribution. If supervision information on truncated joints is provided for training, it is theoretically feasible to query whether there are body joints at a certain position outside the image range.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SeNeEozZEr", "reviewer": "Reviewer_KNNC", "summary": "This paper proposes a new approach to predict continuous heatmap for keypoint localization. The proposed method adopts a MLP that receives position coordinates and corresponding feature and output the confidence score of this position. Then this method can query candidate points and select the point with maximum score as final results. Experiments on several keypoint localization benchmarks demonstrate the effectiveness of the proposed method.", "questions": "In Weakness", "limitations": "This paper discusses its limitation in Section 5.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of introducing implicit neural representation to perform keypoint localization is new and can addresses the quantization error of existing heatmap-based method.\n2.\tThis paper is well-written and easy to understand.", "weaknesses": "My major concern is about the decoding process of the proposed method. It seems that this method should evaluate a lots of points to select the best results, which will introduce much inference computational cost. The introduced progressive coordinate decoding can reduce the computational cost but can avoidably increase the decoding time due to multi-round decoding process. This paper should give a detailed efficiency comparison of existed methods such as heatmap (SimpleBaseline, HRNet, SimCC, ViTPose), regression (RLE), including GFLOPs, inference time (ms), GPU memory consumption under the same setting (all above methods release their codes, so it is not impossible to conduct such comparisons). These comparisons are necessary because it seems that the advantages of the proposed method is not obvious.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful review. Below we address all the concerns.\n\n***Response to weaknesses:***\n\n***Efficiency:*** We compare the efficiency and performance of NerPE and existing methods as shown in **Tab. A3 of the attached PDF**. Although implicit neural representations help us realize continuous representation, it has the disadvantage of a large amount of computation when generating high-resolution signals, just like its application in other computer vision fields. In Tab. A3, NerPE-p significantly reduces the GFLOPs to an acceptable level (from 73.6 to 9.5) by our progressive coordinate decoding that is designed according to the characteristics of human pose estimation.\n\n***Performance:*** NerPE has a positive impact on the performance of the baseline and can be generalized to most heatmap-based methods with minor structural changes. We need to be clear that the quantization error problem worsens as the input resolution decreases. The introduction of implicit neural representations in NerPE significantly improves the performance of the baseline on low-resolution images (e.g., 64×64), as shown in Tabs. 1 & 4. In Tab. 3, the same conclusion is also illustrated by the large lead of NerPE in PCKh\\@0.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WLTmwfKKgm", "reviewer": "Reviewer_rvLc", "summary": "This paper introduces a new heatmap representation for 2D human pose estimation. Prior approaches use a quantized representation of heatmaps, where a confidence score is assigned to each pixel. In addition to being dependent on the image resolution, this representation does not match the usual ground truth coordinates, which are continuous when working in image crops.\n\nThis paper proposes an implicit neural representation to address this limitation of prior works. Specifically, given 2D coordinates (which do not necessarily correspond to the center of a pixel), the introduced neural network outputs a confidence score for each keypoint. At inference, the coordinates of each keypoint are recovered using the maximum confidence value. The main contributions of this paper are the following: (1) The authors introduce NerPE, a novel implicit representation of heatmaps for continuous human pose estimation; (2) A progressive coordinate decoding method is introduced to recover 2D coordinates from continuous heatmaps.", "questions": "(1) \"The meaning of the confidence scores output by current heatmap-based models has not been theoretically proven.\" (L43). What does this really mean? Is the NerPE representation \"theoretically proven\"?\n\n(2) In Equation (5), to my understanding, $f_W(i,j)$ and $f_b(i,j)$ are floats, and $z^*$ would be of dimension $C$? But isn't $H_{z^*}$ supposed to be a float since it is a confidence value?\n\n(3) \"We are surprised to find that the conclusion drawn is similar to the local implicit image function in [8] except for the additional normalization step.\" (L179). Why is that surprising? Isn't the process a bit similar?\n\n(4) Could we consider that the proposed method consists of doing sub-heatmaps until we reach a desired degree of precision?\n\n(5) The model is trained to output Gaussian or Laplacian heatmaps. However, at inference, there is no guarantee that the confidence distribution follows the same distribution. I think that this is a good point, as this brings flexibility. However, couldn't that raise issues for finding the argmax? For instance, the model could potentially \"hesitate\" between 2 different places when placing the center of the heatmap (the predicted distribution would look like a mixture of Gaussians, for instance). Did you observe such cases in practice, and could other strategies be explored instead of choosing the argmax?\n\n\nSide remark: Typo L15 \"desgin\"", "limitations": "Limitations of this work are briefly discussed in Section 5:\n\n- The limited experiments on network architecture and index embedding are not a problem, as this work aimed to focus on the heatmap representation.\n\n- The discussion on the necessity of predicting discrete heatmaps for computing an argmax is interesting, but it would have been more valuable if the authors had provided potential improvements and strategies on that point.\n\nThe potential negative societal impacts are not discussed but are rather limited. The authors could consider mentioning the potential misuse of pose estimation for intrusive surveillance or military applications. The environmental cost could also be discussed, especially since this method is more computationally expensive than prior works.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach is novel; to my knowledge, no prior work has used implicit representations for heatmaps.\n\n- The overall writing is good, and the paper reads nicely. The weakness of prior works addressed in this paper is clearly identified, and the proposed approach is well-justified.\n\n- The proposed approach yields consistent improvement over the SOTA methods, and the ablation study gives a good idea of the impact of choice designs.", "weaknesses": "- The Related Work section lacks discussions and comparisons with other approaches attempting to address the quantization error in heatmaps, such as [42,a,b,c,d,e,f,g]. The authors say that \"As far as we know, existing heatmap-based methods all belong to discrete heatmap regression\" (L104), but for instance, [b] predicts a float location using an offset within the original heatmap. Similar approaches using PixelShuffle operation were proposed [f]. In general, the introduction presents the problem of heatmap quantization as if prior works ignored it, but plenty of works attempted to address this problem.\n\n- I am unsure if having a continuous representation of heatmaps is useful. The ground truth is given in pixel coordinates on the full image. I agree that increasing the heatmap resolution until we reach the original image resolution is helpful, but is it useful to have a higher resolution than the original image? In the end, the ground truth is given in full pixel coordinates.\n\n- Some information important for reproducibility is missing. For instance, what are the training and validation datasets and the number of epochs? Even if the authors say they used \"standard settings\" I believe this is not enough to reproduce the results.\n\n- No information is provided regarding the material used for training and testing or the running times. This is particularly concerning as the method has a high computational cost. The authors propose an alternative decoding strategy that seems much less expensive, but comparing it with other methods and heatmap representations would be necessary.\n\n\n\n\n[a] Bulat, A., Sanchez, E., & Tzimiropoulos, G. (2021). Subpixel heatmap regression for facial landmark localization. BMVC\n\n[b] Lan, X., Hu, Q., & Cheng, J. (2021). Revisiting quantization error in face alignment. ICCV\n\n[c] Yu, B., & Tao, D. (2021). Heatmap regression via randomized rounding. PAMI\n\n[d] Papandreou, G., Zhu, T., Kanazawa, N., Toshev, A., Tompson, J., Bregler, C., & Murphy, K. (2017). Towards accurate multi-person pose estimation in the wild. CVPR\n\n[e] Luvizon, D. C., Picard, D., & Tabia, H. (2018). 2d/3d pose estimation and action recognition using multitask deep learning. CVPR\n\n[f] Wang, H., Liu, J., Tang, J., & Wu, G. (2023). Lightweight Super-Resolution Head for Human Pose Estimation. MM\n\n[g] Li, J., Bian, S., Zeng, A., Wang, C., Pang, B., Liu, W., & Lu, C. (2021). Human pose regression with residual log-likelihood estimation. ICCV", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review. Below we address all the concerns.\n\n***W1: The Related Work section lacks discussions and comparisons with other approaches attempting to address the quantization error ...***\n\nThe quantization error problem caused by heatmap discretization is a well-known fact. In the Introduction, what we want to express is that the existing works are committed to overcoming quantization errors (as a result), while we hope to solve heatmap discretization (as a cause). Although some methods achieve sub-pixel positioning accuracy with the help of techniques such as offset prediction, their performance still depends on the quality of heatmap regression. We do not deny the achievements of current works on the quantization error problem, but provide a new perspective to solve it inspired by implicit neural representations. We will correct the text description to avoid this misunderstanding in the Introduction and distinguish our method from the works mentioned by the reviewer in Related Work.\n\n***W2: I am unsure if having a continuous representation of heatmaps is useful. The ground truth is given in pixel ...***\n\nIn the top-down paradigm, the image patches containing the person instances in the original image are scaled and rotated before being fed into the pose estimation network. As a result, the original pixel-level annotation becomes the floating-point ground truth after affine transformation. Therefore, in coordinate decoding, even if the heatmap resolution is equal to the input resolution, the coordinates calculated by argmax are still lossy.\n\n***W3: Some information important for reproducibility is missing. For instance, what are the training and validation datasets ...***\n\nCOCO, MPII, and CrowdPose are standard human pose estimation datasets with fixed settings in terms of train\\val division, data augmentation, etc. The random rotation and scaling factors are [[-45°, 45°], [0.65, 1.35]] in COCO, [[-30°, 30°], [0.75, 1.25]] in MPII, and [[-45°, 45°], [0.65, 1.35]] in CrowdPose. \"standard settings\" means that the training process of NerPE is consistent with the corresponding baseline, where the learning rate is initialized to 0.001 and decayed twice by a factor of 0.1. The learning schedules with SimpleBaseline, HRNet, and TokenPose as the backbone are [90th, 120th, 140th] epoch, [170th, 200th, 210th] epoch, and [200th, 260th, 300th] epoch.\n\n***W4: No information is provided regarding the material used for training and testing or the running times ...***\n\nWe provide a comprehensive analysis of the efficiency and performance in **Tab. A3 of the attached PDF**, where we supplement GFLOPs and inference time to better compare NerPE with existing methods. As shown in Tab. A3, our progressive coordinate decoding effectively solves the shortcoming of implicit neural representations that generate a lot of computation when outputting high-resolution signals.\n\n***Q1: \"The meaning of the confidence scores output by current heatmap-based models has not been theoretically proven.\" ...***\n\nExisting methods regress heatmaps in the form of 2D pixel arrays. To generate discrete ground truth heatmaps, many influential works (such as HRNet, ViTPose) place a fixed Gaussian kernel on the grid points to which body joints belong. Therefore, the ground truth heatmaps have a series of fixed discrete values while the heatmaps predicted by the neural network have continuous values, which are not equivalent. In contrast, our generated ground truth heatmaps are continuous, so there is no such problem.\n\n***Q2: In Equation (5), to my understanding,*** $f_{W}(i,j)$ ***and*** $f_{b}(i,j)$ ***are floats, and*** $z^{*}$ ***would be of dimension*** $C$ ***? But ...***\n\nIn Eq. 5, $f_{W}(i,j)$ and $f_{b}(i,j)$ are rewritten from $W_{i,j}$ and $b_{i,j}$, where the mapping $f$ represents the selection in the weight libraries $W$ and $b$ according to an index, i.e., $f:(i,j)\\mapsto W_{i,j},b_{i,j}$. The local feature vector $z^* \\in \\mathbb{R}^{C \\times 1 \\times 1}$ is a element in the image features $Z \\in \\mathbb{R}^{C \\times H_{Z} \\times W_{Z}}$, and $H_{z^*}$ denotes the cell in the heatmaps $H$ corresponding to $z^*$. So, $H_{z^*}$ is a float tensor instead of a float.\n\n***Q3: \"We are surprised to find that the conclusion ...\" (L179). Why is that surprising? Isn't the process a bit similar?***\n\nIn [Ref1], the local version of implicit neural representation is proposed based on intuition, and at least the persuasive derivation process is not provided in its paper. In our work, we derive a similar conclusion from sub-pixel convolution, which is exciting because we find a reasonable explanation for the local implicit neural representation.\n\n***Q4: Could we consider that the proposed method consists of doing sub-heatmaps until we reach a desired degree of precision ?***\n\nYes. Our method has two ways to reach the desired degree of precision. The standard NerPE directly outputs complete predicted heatmaps at the resolution corresponding to the required precision. The variant NerPE-p achieves equivalent resolution through coordinate decoding to iteratively refine the local parts of heatmaps, as the reviewer said.\n\n***Q5: The model is trained to output Gaussian or Laplacian heatmaps. However, at inference, there is no guarantee that ...***\n\nAs the reviewer said, in existing heatmap regression research, the distribution of predicted heatmaps is generally not guaranteed to conform to the Gaussian or Laplace distribution. The reason is that the loss function is the average of the differences of each heatmap pixel, so our method is not immune even if our heatmap representation is continuous. To solve this problem, it may be a simple and effective solution to add predictions of mean and variance to assist argmax.\n\n[Ref1] Learning continuous image representation with local implicit image function. CVPR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their time and their constructive feedback. We appreciate their assessment of our work NerPE as a "well-motivated" approach for "an important research topic" (xx7e), “a more principled solution to producing arbitrary-resolution heatmaps” (k3b9), a "new idea introduces implicit neural representations" (KNNC), and a "well-justified" approach to "address the weakness of prior works" (rvLc). All reviewers seemed to agree that our work is "good writing" (rvLc), "easy-to-understand" (xx7e, KNNC), and "clearly structured" (k3b9).

Inspired by the reviewers’ helpful comments, we will incorporate the following changes into the next/final revision of our paper:

-  We test the performance of NerPE at a higher resolution (384x288) and with another backbone (ViTPose) on the COCO test-dev set (see Tab. A1 of the attached PDF).
  
- We report more about the division of cells in NerPE (see Tab. A2 of the attached PDF).
  
-  We compare the efficiency and performance of NerPE and existing methods (see Tab. A3 of the attached PDF).
  
-  We visualize the ablation of local resemble in NerPE (see Fig. A1 of the attached PDF).
  
-  We show the heatmaps output by NerPE at different resolutions (see Fig. A2 of the attached PDF).

Meta-review: This paper proposes to use implicit representations to address heatmap quantization error for keypoint localization. It received 2 BAs and 2 WAs after rebuttal, which all the reviewers agreed to accept. 


The initial concerns focused on clarification, visualization, and especially insufficient experiments (resolution, backbone, and efficiency). Moreover, some important related works (like heatmap & offset, non-standard Gaussian heatmap, and regression) are missing.

After going through the paper, the review, and the response, the AC values the key idea, finds most concerns are addressed, and recommends the acceptance of the paper.

The AC also recognizes the weaknesses (insufficient experiments and missing related works) and thus urges the authors to further revise the paper.
Decision: Accept (poster)