APP: Adaptive Pose Pooling for 3D Human Pose Estimation from Videos

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-21 03:44:54

Abstract: Current advancements in 3D human pose estimation have attained notable success by converting 2D poses into their 3D counterparts. However, this approach is inherently influenced by the errors introduced by 2D pose detectors and overlooks the intrinsic spatial information embedded within RGB images. To address these challenges, we introduce a versatile module called Adaptive Pose Pooling (APP), compatible with many existing 2D-to-3D lifting models. The APP module includes three novel sub-modules: Pose-Aware Offsets Generation (PAOG), Pose-Aware Sampling (PAS), and Spatial Temporal Information Fusion (STIF). First, we extract latent features of the multi-frame lifting model. Then, a 2D pose detector is utilized to extract multi-level feature maps from the image. After that, PAOG generates offsets according to featuremaps. PAS uses offsets to sample featuremaps. Then, STIF can fuse PAS sampling features and latent features. This innovative design allows the APP module to simultaneously capture spatial and temporal information. We conduct comprehensive experiments on two widely used datasets: Human3.6M and MPI-INF-3DHP. Meanwhile, we employ various lifting models to demonstrate the efficacy of the APP module. Our results show that the proposed APP module consistently enhances the performance of lifting models, achieving state-of-the-art results. Significantly, our module achieves these performance boosts without necessitating alterations to the architecture of the lifting model.

Main paper: 
 related work: 
 Transformers [42] initially dominated the domain of natural language processing. Owing to its expressive performance, it has gained significant attention in computer vision as well [10] . Therefore, Transformer-based models have been increasingly applied across various domains, including 3D human pose estimation. Pose-Former [49] stands out as the pioneering model to integrate Transformers into 3D human pose estimation. Building upon this foundation, PoseFormerV2 [48] represents an advancement, introducing a frequency domain approach to enhance model robustness against fast motion changes. Furthermore, MHFormer [22] addresses the challenge of depth ambiguity by learning the representations of multiple hypothesis poses. In the quest for improved efficiency and accuracy, novel architectural modifications have emerged. StridedTrans [21] utilizes fully connected layers in the Transformer encoder with stride convolutions. It reduces computational complexity and progressively shortens sequence lengths, thus enhancing pose estimation accuracy for the central frame. Meanwhile, MixSTE [45] focuses on modeling strong intra-frame correspondences between individual joints, thereby facilitating the learning of spatio-temporal correlations. P-STMO [37] adopts a strategy of occlusion masking to enable the model to learn more resilient pose representations. Additionally, it leverages a large volume of unlabeled data for self-supervised pretraining, thereby augmenting generalization capabilities. HoT [23] achieves this by implementing feature compression and restoration techniques, rendering it compatible with various lifting models while reducing computational overhead.Advancements in attention mechanisms have also been pivotal. HDFormer [4] , for instance, integrates self-attention and higherorder attention mechanisms to construct a hierarchical attention module aimed at mitigating complexities and handling heavily occluded scenarios. STCFormer [41] decomposes correlation learning into spatial and temporal dimensions, thereby reducing model complexity. Moreover, MotionBERT [51] emerges as a dual-stream spatio-temporal Transformer capable of concurrently capturing spatio-temporal information, showcasing state-of-the-art performance across multiple downstream tasks. Similarly, MotionAG-Former [27] introduces a hybrid architecture composed of multiple dual-stream modules, effectively capturing both spatial and temporal features of keypoints in tandem. KTPFormer [33] tries to overcome the weakness in existing transformer-based methods by proposing two plug-and-play attention modules, namely Kinematics Prior Attention (KPA) and Trajectory Prior Attention (TPA). These modules take advantage of the structure of the human body and motion trajectory information.For the 2D-to-3D lifting stage, given a sequence of 2D human poses 𝒑 ∈ R 𝑇 ×𝐽 ×𝐶 𝑖𝑛 , the goal is to estimate the corresponding 3D human poses p ∈ R 𝑇 ×𝐽 ×𝐶 𝑜𝑢𝑡 , where 𝑇 is the sequence length of 2D pose frames, 𝐽 is the number of keypoints of the 2D human pose, 𝐶 𝑖𝑛 and 𝐶 𝑜𝑢𝑡 are the dimension of the 2D and 3D human pose keypoints, respectively.These advancements collectively underscore the profound impact and ongoing evolution of Transformer-based approaches in advancing the field of 3D human pose estimation. Unlike our proposed method, which integrates image features into estimating 3D human poses, none of the existing transformer-based multi-frame methods take advantage of such information. This presents a unique opportunity for our approach to bridge this gap and potentially unlock new avenues for improving the accuracy and robustness of 3D human pose estimation methods. 
approach 3.1 framework overview: 
 Figure 2 illustrates the overview of our proposed method. Given input images 𝑰 ∈ R 𝑡 ×3×𝐻 ×𝑊 , where 𝑡 is the number of image frames and 𝐻 , 𝑊 are the height and width of the image, note that 𝑡 ≪ 𝑇 . We first use a 2D pose detector to extract the multi-level feature maps denoted by {𝑯 𝑖 } 𝑆 𝑠=1 , where 𝑆 is the number of scales. 𝑯 1 is a heatmap to get the 2D pose 𝑝. Then pose features 𝑷 ∈ R 𝑇 ×𝐽 ×𝑑 𝑷 is calculated in the multi-frame lifting models without regression head, where 𝑑 𝑷 is the dimension of pose features. This process is formulated as:EQUATION𝑀𝐿𝑀 (•) denotes multi-frame lifting models without regression head; the backbone is a 2D pose detector. In Figure 2 , the values of averaged image features F ∈ R 𝑇 × (𝑘 2 ×𝐽 ) ×𝑑 and hidden features 𝑿 ∈ R 𝑇 ×𝐽 ×𝑑 are updated by each layer of the APP module, where 𝑘 is the window size.To get the initialized offsets 𝚫 0 ∈ R 𝑇 × (𝑘 2 ×𝐽 ) ×𝐶 𝑖𝑛 , we repeat the values by taking each point in the 2D human poses 𝒑 as a center and generating a 𝑘 × 𝑘 window around it. Then the 𝑝 is resized to R 𝑇 × (𝑘 2 ×𝐽 ) ×𝐶 𝑖𝑛 . Now, the 2D poses 𝑝 data is a 2D vector, which means the 𝑥 and 𝑦 coordinates of the human keypoints.We denote F𝑙 -1 and 𝑿 𝑙 -1 as the last layer outputs of the APP module. As for the first layer of APP, the offsets 𝚫 0 are used to extract the initialized F 0 and 𝑿 0 :EQUATIONWhere 𝜃 𝑓 is a group of 2D convolution operators to project multilevel feature maps into the same dimension 𝑑, the implementation of 𝐺𝑟𝑖𝑑𝑆𝑎𝑚𝑝𝑙𝑒 is consistent with nn.functional.grid_sample in PyTorch, 𝐴𝑣𝑒𝑟𝑎𝑔𝑒𝑃𝑜𝑜𝑙 is to get the mean value of each 𝑘 ×𝑘 window.Our proposed APP module consists of 𝐿 stacked layers. In detail, the calculation process of each layer APP module is defined as follows:EQUATIONIn the last layer of the APP module, a simple regression head consisting of two fully connected layers estimates the final 3D human pose given by 𝑿 𝑙 . 𝑝 as input to generate learned offsets Δ 𝑙 , which is defined as follows:EQUATIONWhere ⊗ denotes element-wise multiplication, 𝜃 𝑙 𝑠 and 𝜃 𝑙 𝑤 generate offset points based on the image center and their corresponding weights. Note that they are both fully connected layers.𝑝 generated by PAOG and the previous averaged image features F𝑙 -1 , PAS also utilizes the multi-level feature maps {𝑯 } 𝑆 𝑠=1 ∈ R 𝑡 ×𝐶 𝑠 ×𝐻 𝑠 ×𝑊 𝑠 extracted from the 2D pose detector. We follow Deformable-DETR [52] before sampling features from the multi-level feature maps 𝑭 . They are projected into the same dimension 𝑑 as in Eq. ( 2). The sampling process is then defined as:EQUATION3.2.3 Feature Updating. At the 𝑙 layer of the APP module, we update the averaged image features F𝑙 and hidden features 𝑿 𝑙 as follows:𝑿 𝑙 = 𝜃 𝑙 𝑘 ( F𝑙 ), F𝑙 = 𝛼 F𝑙 -1 + (1 -𝛼) F𝑙 (6)Where 𝜃 𝑙 𝑘 (•) represents a convolution operation with a kernel size of 𝑘 2 , whose size is the same as the previously mentioned 𝑘 × 𝑘 window, this operation allows the PAS to learn the weights of 𝑘 2 features within the window to calculate 𝑿 𝑙 given by F𝑙 . 𝛼 is a parameter controlling the magnitude of the update for F𝑙 .Multi-Head Cross Attention. MHCA enables the APP module to harness the temporal features extracted by the multi-frame lifting models while interacting with the spatial features extracted by PAS. We first combine the pose feature 𝑷 extracted with positional embeddings 𝑃𝐸, initialized to zero, and updated during training. We take 𝑿 𝑙 as 𝑸, 𝑷 as 𝑲 , and 𝑽 , then we perform as follows:EQUATIONWhere 𝑾 𝑸 ∈ R 𝑑 ×𝑑 , 𝑾 𝑲 ∈ R 𝑑 𝑷 ×𝑑 , and 𝑾 𝑽 ∈ R 𝑑 𝑷 ×𝑑 , 𝑗 ( 𝑗 = 1, ..., ℎ) represents the 𝑗-th attention head, and ℎ is the number of attention heads.3.2.5 Feed Forward Network. Like many monocular 3D HPE transformerbased methods, we apply an FFN at the end to finish computing the current layer. We use GeLU [15] as the activation function, as outlined in Eq. (8) .EQUATION3.2.6 Loss Function. The loss function of our proposed APP module is the same as [27, 51] . In addition to the widely used MPJPE loss, we also incorporate the Mean Per-Joint Velocity Error (MPJVE) loss and the normalized MPJPE [36] loss. Let L 𝑝 , L 𝑣 , and L 𝑠 denote these three losses. L 𝑝 calculates the 𝐿 2 distance between estimated 3D human poses p and ground truth 𝒈, while L 𝑣 minimizes the difference between Δ p and Δ𝒈. L 𝑝 and L 𝑣 are thus given by:EQUATIONWhere Δ p𝑡 = p𝑡 -p𝑡 -1 , Δ𝒈 𝑡 = 𝒈 𝑡 -𝒈 𝑡 -1 . For L 𝑠 , estimated 3D poses p is scaled according to ground truth 𝒈. Then, we calculate MPJPE loss between scaled p and 𝒈. The total loss is formulated as follows:EQUATION)where 𝛾 𝑝 , 𝛾 𝑣 , and 𝛾 𝑠 are weights for L 𝑝 , L 𝑣 , and L 𝑠 , respectively.  
implementation details: 
 We adopt Transformer [42] , MotionAGFormer-B [27] , MixSTE [45] ,MotionBERT [51] , and HoT [23] as the foundation models. These models' dimensions 𝑑 𝑷 are 128, 256, 512, and 128, respectively. For a fair comparison, we use pretrained lifting models provided by the authors.YOLOv3 [35] is used to extract bounding boxes of humans in the images, and then we crop the images to 192 × 256 according to the bounding boxes. However, in [47] , the 3D ground truth human pose is directly used to extract bounding boxes, which is not feasible in real-world scenarios. The lengths of 2D human pose sequences 𝑇 on Human3.6M and MPI-INF-3DHP are set to 243 and 81, respectively, and images are uniformly sampled into sequence lengths 𝑡 = 𝑇 9 for a tradeoff between performance and efficiency. Additionally, 𝑡 must be divisible by 𝑇 . We extract feature maps from images using HRNet-w32 [39] with four resolutions. It is chosen as the pose detector,EQUATION, 𝑊 32 }. The parameters of our proposed APP module are mainly determined by the number of layers 𝐿, the number of attention heads ℎ, the kernel size 𝑘, the dimension 𝑑 of the APP module, and the dimension of hidden feature 𝑭 𝑝 used in the selected lifting model. Also, we employ DropPath [20] with the probability 𝑝. For Hu-man3.6M and MPI-INF-3DHP, 𝐿, ℎ, 𝑘, 𝛼, 𝑝, and 𝑑 are set to 6, 8, 3, 0.9, 0.2, and 128, respectively. We train the APP module using the Adam optimizer [19] . 2D ground truth human pose is sent to the model for MPI-INF-3DHP. In Eq. ( 9), the weights of each part of the loss are set to 1.0, 20.0, and 0.5, respectively. All experiments are conducted on one NVIDIA RTX 3090 GPU.We conducted comparative evaluations with recent state-of-the-art (SOTA) models on the Human3.6M dataset. The results, summarized in Table 1 , underscore the superiority of our proposed APP module when integrated with MotionAGFormer-B, attaining SOTA performance. Specifically, in comparison with lifting models MixSTE [45] and HDFormer [4] , which also utilize HRNet-detected 2D human pose as input, our approach achieves notable improvements of 3.6mm (from 39.8mm to 36.2mm) and 4.1mm (from 40.3mm to 36.2mm) in MPJPE, and 1.1mm (from 30.6mm to 29.5mm) and 2.2mm (from 31.7mm to 29.5mm) in terms of P-MPJPE, respectively. What's more, our proposed method surpasses CA-PF-HRNet-48 [47] 3.6mm (from 39.8mm to 36.2mm) in MPJPE, and 3.2mm (from 32.7mm to 29.5mm).Although our method exhibits a slightly higher MPJPE of 1.3mm compared to MotionBERT [51] when ground truth 2D human pose is employed as input, it surpasses MotionAGFormer-B by 0.3mm. These findings collectively attest to the efficacy and competitiveness of our proposed approach in advancing the state-of-the-art in monocular 3D human pose estimation. In evaluating the MPI-INF-3DHP dataset, we aligned the sequence length𝑇 to 81 frames to maintain consistency with the experimental setup of MotionAGFormer-B [27] . As detailed in Table 2 , our method demonstrates incremental enhancements across various evaluation metrics. Specifically, we observe marginal improvements in the AUC and MPJPE upon KTPFormer [33] , with gains of 3.6 (from 85.9 to 89.5) and 4mm (from 16.7mm to 12.7mm), respectively. Moreover, the PCK of our method is the same as KTPFormer [33] because the threshold of PCK is set to 150mm. When comparing our method with the baseline model MotionAGFormer-B [27] , there are huge improvements for AUC (from 84.2 to 89.5) and MPJPE (from 18.2mm to 12.7mm). These findings underscore the efficacy and robustness of our proposed method in enhancing monocular 3D human pose estimation performance on the MPI-INF-3DHP dataset. We conduct extensive experiments to substantiate the effectiveness of our proposed APP module. As shown in Table 4 , our experimental results manifest a consistent enhancement in the performance of the lifting models across diverse configurations of 2D pose detectors and multiple-frame lifting models, affirming the efficacy of the APP module. Transformer [42] is a simple spatial-temporal vanilla transformer with 12 layers. Notably, the APP module contributes to performance improvements while maintaining small parameters, accounting for approximately 37% (from 9.62M to 13.2M), 17% (from 33.78M to 39.54M), 14% (from 42.47M to 48.23M), 30% (from 11.72M to 15.3M), 26% (from 16.35M to 20.66M) parameters of Transformer [42] , MixSTE [45] , MotionBERT [51] , MotionAGFormer-B [27] , HoT [23] , respectively. When it comes to MACs, the gains are 34% (from 46.16G to 70.71G), 16% (from 147.6G to 174.85G), 12% (from 185.6G to 211.77G), 28% (from 64.78G to 90.14G), 34% (from 33.48G to 50.8G).All ablation studies are conducted on the Human3.6M dataset, with MotionAGFormer-B [27] chosen as the baseline model and 2D human body pose detected using HRNet [39] . Initially, we conducted experiments on parameter selections for the APP module. The results of Table 5 presents indicate that our proposed method achieves the best performance when 𝐿, 𝑘, 𝛼, 𝑝, and 𝑑 are selected as 6, 3, 0.9, 0.2, and 128, respectively. Then, we conducted ablation experiments on the submodules of our proposed APP module. In Table 6 , the first row presents partial components of three submodules: MHCA, PAOG, and PAS. The table shows that MHCA has the most significant impact on the APP model's performance. However, the other three components also enhance the model's performance to varying degrees. Finally, we conducted ablaton experiments on image frames 𝑡 setting. Although the MACs (24.83G) of the APP module are not affected by the number of frames, yet image frames 𝑡 need more computation in training stage because we need multi-level featuremaps extracted by 2D pose detector. Furthermore, when 𝑡 ≥ 81, the training time becomes very long. Therefore, 𝑡 = 27 = 𝑇 9 is tradeoff between accuracy and efficiency. In inference stage, this problem no longer appears, owing to that 2D pose detector is used to estimate 2D pose for every frame. To validate the generalization ability of the model, we selected the MotionBERT [51] (finetune version) and MotionAGFormer-B [27] as foundation models, which were trained on 2D poses extracted by SH [30] on Human3.6M dataset, for comparison with our proposed APP module. We chose two video segments from the internet to visually analyze the APP module. For the visual analysis in Figures 3 and 4 , MotionAGFormer-B was selected as the foundation model. 4.6.3 Analysis of In-the-wild Videos. We selected six frames from two videos to compare the predicted 3D human poses by two lifting models, MotionBERT [51] and MotionAGFormer-B [27] , with those predicted by our proposed model. The differences have been circled in orange.  
conclusion: 
 Presenting APP, short for Adaptive Pose Pooling, a flexible module crafted to mesh with various multi-frame lifting models. This versatile module utilizes the hidden features extracted by lifting models with the feature maps obtained from the 2D pose detector. APP excels in simultaneously extracting spatial and temporal features through this collaborative approach module, and our visualization demonstrates this. Extensive experimentation highlights the reliability of our module, showcasing its ability to enhance the performance metrics of lifting models while consistently achieving state-of-the-art results across diverse scenarios. Notably, APP module presents remarkable adaptability, maintaining its effectiveness even when paired with different 2D poses. A noteworthy feature is its capacity to enhance performance without requiring structural changes to the existing lifting model architecture.  
: 
 Our method addresses the issue of underutilizing image features by leveraging multi-level feature maps extracted by a pretrained 2D pose detector. Furthermore, while single-frame models utilize image features, they lack temporal information. However, the APP module addresses this by extracting temporal and spatial information simultaneously.categorized into two fundamental classes based on the number of viewpoints employed: multi-view and single-view. Multi-view approaches leverage information from multiple camera perspectives, as explored in studies such as [6, 14, 17, 34, 46] . They mitigate depth ambiguity and reach better results than monocular 3D HPE methods. However, their practical applicability is hindered by the necessity of employing multiple cameras.In contrast, single-view models offer a more applicable alternative and can be subdivided into single-stage and two-stage methodologies. Single-stage methods [3, 18, 25, 31, 40, 50] directly infer 3D human pose from input images or video frames without intermediate steps. While two-stage like early methods [26, 32] adopt a sequential approach. As shown in Figure 1 (a), off-the-shelf 2D pose detectors [5, 30, 39] extract 2D human poses from input images or video frames. After that, they can obtain corresponding 3D human poses based on these 2D estimations, so this stage is often called the lifting stage. Models in the lifting stage can further be categorized into single-frame or multi-frame variants. Single-frame models [7, 47] infer 3D human poses from individual images, while multi-frame models process sequences of 2D human poses. The latter one contains two categories: many-to-one and many-to-many methods. Many-to-one models estimating the 3D human pose of a central frame using multiple 2D human poses [21, 22, 48, 49] , and many-to-many, also called seq2seq models inferring every frame of a 3D human pose from corresponding 2D human poses [4, 27, 33, 37, 45, 51] . They try to address different problems, ranging from spatio-temporal information exploration to enhancing model robustness.In this paper, we focus on monocular 3D HPE. It uses only one camera to obtain images of humans. Therefore, it introduces ambiguity because of the lack of depth information, which is an ill-posed task. Besides, occlusion, including self-occlusion and occlusion by other objects and persons, needs to be drawn to researchers' attention. Hence, single-view models sometimes fail to get accurate and robust 3D human poses owing to depth ambiguity and occlusion. Many monocular 3D HPE models suffer from them because the information of the input images fails to be fully used. HEMlets Pose [50] and Context-Aware PoseFormer (CA-PF) [47] try to handle this. HEMlets Pose explicitly lets the model learn relative depth among different keypoints in 2D human poses. In comparison, CA-PF adopts Deformable Attention [52] to extract features of feature maps obtained by the 2D pose detector. However, these methods [47, 50] are single-frame models that cannot capture temporal dynamics. Recently, multi-frame lifting models have reached significant improvement. Why not use both image and temporal features? This work proposes a well-designed Adaptive Pose Pooling (APP) module. Serving as a plug-and-play component that can be integrated with many existing lifting models. The APP module facilitates extracting temporal and spatial information from multi-frame 2D human pose data and feature maps. Figure 1 illustrates the distinction between previous methods and our proposed method. Previous methods freeze the 2D pose detector and only take detected 2D poses as the model input to estimate corresponding 3D poses. Our method uses feature maps extracted by a 2D pose detector and takes multi-frame lifting models' calculated pose features. Therefore, the APP module inherently can capture spatiotemporal features. Do we really need to use all the features of the feature maps? Features near 2D poses should be more important than features at other positions. To enhance the model's flexibility, the model should adjust the sampling points during training. In the meantime, getting spatial and temporal features and interacting among features is also important for the model.Our proposed APP module contains three novel submodules: Pose-Aware Offsets Generation (PAOG), Pose-Aware Sampling (PAS), and Spatial Temporal Information Fusion (STIF). To address the problem of insufficient utilization of spatial information in images, we propose PAOG and PAS. PAOG leverages 2D human poses as indices to learn offsets during training. PAS adaptively extracts features from feature maps generated by the 2D human pose detector. Though single-frame models use image features, they suffer from lacking temporal features. We proposed STIF to solve this problem by fusing image and temporal features. We conduct extensive experiments on two popular datasets: Human3.6M and MPI-INF-3DHP. Our proposed method achieves a new state-of-theart performance, with 1.3mm (from 37.5mm to 36.2mm) and 1.1mm (from 30.6mm to 29.5mm) improvement in terms of Mean Per-Joint Position Error (MPJPE) and P-MPJPE (Procrustes-MPJPE) on Hu-man3.6M. There are 4.6 (from 85.9 to 89.5) and 4mm (from 16.7mm to 12.7mm) improvements for Area Under Curve (AUC) and MPJPE on MPI-INF-3DHP.Compared to current methods for monocular 3D HPE, our contributions are summarized as follows:• We propose the APP module, enabling multi-frame lifting models to utilize information from the previous stage. This module is a plug-and-play solution compatible with most multi-frame lifting models and mitigates the problem of excessive reliance on 2D human pose outputs.  
 
 Appendix:  


Decision: The paper present an original Adaptive Pose Pooling model to estimate 3D human poses from videos, which is clearly presented and extensively validated.  The plug-and-play module has excellent generalization capabilities and represent a solid contribution to the community. 
The rebuttal addressed the reviewers concerns and all reviewers agreed on accepting this paper.

Reviews: review: paper_summary: paper_summary: his paper proposes Adaptive Pose Pooling model to estimate 3D human poses from videos.
The model uses 2D features and multi-level feature maps as inputs to obtain spatial and temporal information. Experiments on two datasets demonstrate the effectiveness of the proposed model.
strength_weakness: strength_weakness: The extensive qualitative comparisons on the Human3.6M dataset demonstrate the effectiveness of the proposed model.
limitations: limitations: Why in Table 2 the performance of MotionBERT is better than the proposed model?
paper_summary: paper_summary: The paper presents a flexible plug-and-play module, adaptive pose pooling, for human pose estimation from videos. The method proposes a pose-aware offsets generation module and a pose-aware sampling module, enhancing the model with features from the previous stage of lifting. With the help of a spatio-temporal information fusion module, the method efficiently captures both temporal and spatial features. Experimental results demonstrate the superiority of the method in the HPE task, achieving state-of-the-art performance on two datasets when combined with MotionBERT and MotionAGFormer-B.
strength_weakness: strength_weakness: 1. Sophisticated method design: The method embraces a pose-aware sampling module and a pose-aware offsets generation module to learn both temporal and spatial features of human pose from videos. It is well-designed and elegant.
2. Effective experimental results: The experimental results show the effectiveness of the method. The paper provides a detailed comparison with state-of-the-art methods on two datasets, demonstrating superior performance on Human3.6M and MPI-INF-3DHP.
3. Great generalization of the plug-and-play module: The plug-and-play module has excellent generalization capabilities, potentially bringing strong features for future work in the field of human pose estimation.
limitations: limitations: 1. Qualitative visualization: In Figure 5, the visualizations for MotionBERT and APP with MotionBERT are not clear. They look the same and are hard to distinguish. Additionally, a column showing the ground truth visualization should be included.
2. Experimental design: Currently, the results mainly include the baseline method. However, the focus should be on how the plug-and-play module boosts the performance of the baseline method. Table 4 only contains a subset of baselines.
3. Paper presentation: The tables currently occupy too much space, making the paper hard to read. To address this, Table 1 can be merged with Table 4 into a single table. The results for different categories can be placed in the supplementary materials. The results on different metrics should be presented in a single-column table, which would be better for demonstration.
paper_summary: paper_summary: This paper presents the Adaptive Pose Pooling (APP) module, designed to enhance 3D human pose estimation in videos by integrating latent features from both 2D pose detector and 2D pose lifting model within a multi-frame framework. The efficacy of the APP module is substantiated through comprehensive experiments on the Human3.6M and MPI-INF-3DHP datasets, demonstrating notable accuracy improvements compared to existing methods.
strength_weakness: strength_weakness: 1. The simple motivation and straightforward approach make the paper easy to follow.
2. The proposed approach outperforms the state-of-the-art method on benchmark datasets.
3. Some nice visualizations in the experimental sections.
limitations: limitations: 1. The paper does not adequately explain why incorporating image features enhances model performance. It appears that these features might be used to refine the estimates from the 2D pose detector for more stable 3D pose predictions.
2. It would be beneficial to explore the impact of using latent features from APP as both the query and value in the Multi-Head Cross Attention (MHCA) setup. What outcomes would this alteration yield?
3. Table 2 should specify the 2D pose detector used by each method for clarity and fair comparison. Most referenced methods utilize CPN or SH detectors, whereas this study employs HRN. Notably, the APP and MotionAGFormer-B demonstrate similar results on the Human3.6M dataset under the same conditions.
4. In Table 6, it is unclear what serves as the query for MHCA in the first configuration. Does it directly employ the hidden features from the 2D pose detector as the key? Additionally, comparing the fifth and last items shows that the influence of APP may not be as significant as expected. To improve readability and facilitate better comparison, assigning serial numbers to each experimental configuration in the table is recommended.