Joint-Motion Mutual Learning for Pose Estimation in Video

Authors: haipeng  chen
yifang  yin
runyang  feng
yingying  jiao
ziqi  yang
zhenguang  liu
sihao  hu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 05:26:02

Abstract: Human pose estimation in videos has long been a compelling yet challenging task within the realm of computer vision. Nevertheless, this task remains difficult because of the complex video scenes, such as video defocus and self-occlusion. Recent methods strive to integrate multi-frame visual features generated by a backbone network for pose estimation. However, they often ignore the useful joint information encoded in the initial heatmap, which is a by-product of the backbone generation. Comparatively, methods that attempt to refine the initial heatmap fail to consider any spatio-temporal motion features. As a result, the performance of existing methods for pose estimation falls short due to the lack of ability to leverage both local joint (heatmap) information and global motion (feature) dynamics.
 
 To address this problem, we propose a novel joint-motion mutual learning framework for pose estimation, which effectively concentrates on both local joint dependency and global pixel-level motion dynamics. Specifically, we introduce a context-aware joint learner that adaptively leverages initial heatmaps and motion flows to retrieve robust local joint features. Given that local joint features and global motion flows are complementary, we further propose a progressive joint-motion mutual learning that synergistically exchanges information and interactively learns between joint features and motion flows to improve the capability of the model. More importantly, to capture more diverse joint and motion cues, we theoretically analyze and propose an information orthogonality objective to avoid learning redundant information from multi-cues. Empirical experiments show our method outperforms prior arts on three challenging benchmarks.

Main paper: 
 introduction: 
 Accurately estimating human pose in videos by localizing the anatomical position of human joints (e.g., ankle, shoulder) serves as a fundamental cornerstone for higher-level tasks such as action recognition [62] , motion transfer [36, 58] , and motion prediction [7, 47] . It also paves the way for modern human-machine interaction applications, including robotics [32] , augmented reality (AR) [19] , and smart home [46] , etc.Figure 1 : Heatmap-based methods such as DCPose refine heatmaps to perform pose estimation but lack the ability to incorporate spatio-temporal features. Conversely, featurebased methods like TDMI employ feature difference to extract the valuable features but neglect local joint dependency present in heatmaps. These methods may encounter estimation difficulties in scenarios involving video defocus and self-occlusion. In comparison, our approach models the jointmotion information with a novel mutual learning framework, which grasps more meaningful and complementary representations, delivering a more robust result.Fueled by the power of deep learning [9, 44, 65, 66, 71] and the accessibility of large-scale datasets [1, 10, 23] , human pose estimation in static images has garnered extensive attention and achieved notable breakthroughs. Pioneering approaches propose pictorial structure models [43, 54] for pose estimation. Recently, deep convolutional neural models [33, 52] , Transformer models [61, 67] , and diffusion models [40] have drastically pushed forward the frontier of pose estimation in static images. Unfortunately, existing methods often encounter suboptimal performance when directly applied to human pose estimation in videos. This mainly stems from the fact that videos are usually accompanied with challenging frames due to frequent pose occlusion and video blurring.Existing methods for video-based human pose estimation can be broadly classified into heatmap-based and feature-based approaches. These methods are distinguished by their emphasis on refining either heatmaps or visual features extracted from the backbone network. One line of works, i.e., heatmap-based methods [5, 34] , utilize unidirectional or bidirectional neighboring heatmaps of consecutive frames to refine the heatmap of the keyframe. Another line of works, i.e., feature-based approaches [14, 35] , focus on extracting the most useful visual features for human pose estimation.Upon scrutinizing and experimenting with the released implementations of existing methods [14, 34] , we observe that they suffer from degraded performance in challenging scenes like video defocus and self-occlusion: (1) For example, in Figure 1 , the heatmap-based method DCPose [34] fails to locate the left hand in a defocused frame and the right foot in a self-occlusion frame. The reason is that DCPose estimates the joint by merging the initial heatmaps extracted from consecutive frames and ignores the cues contained in the spatio-temporal motion feature. (2) As shown in Figure 1 , the feature-based method TDMI [14] demonstrates inaccuracies in estimating joint positions within challenging frames. These include the misalignment of the left hand in a defocused frame and the drifting right foot in a self-occlusion frame. We think that TDMI employs joint (heatmap)-agnostic visual features to estimate human pose, overlooking the implicit position information and joint dependency contained in initial heatmaps. In addition, visual features fail to capture pixel-level motion dynamics and are not robust enough for challenging scenarios. These observations inspire us to ask: How to effectively aggregate local joint dependency and global pixel-level motion dynamics to obtain a comprehensive representation?To answer the question, we propose a novel framework, termed Joint-Motion Mutual Learning for Pose Estimation (JM-Pose), which comprises two key components. Firstly, to enable the model to learn joint (heatmap)-relevant features, we introduce a contextaware joint learner to extract local joint feature. Specifically, guided by initial heatmaps, we employ modulated deformable operations to capture local joint context features. Secondly, we propose a progressive joint motion mutual learning to explore the cooperative effect of the two representations in pose estimation. This alleviates the limitation of existing methods that only exploit single-modal representations. The progressive joint motion mutual learning dynamically exchanges and refines information between local joint feature and global motion flow to capture an informative jointmotion representation. We conduct further theoretical analysis and arrive at an information orthogonality objective. Minimizing the objective endows our model with the ability to fully mine the diverse joint feature and motion flow, enhancing the performance of pose estimation. We perform comprehensive experiments on three widely used benchmark datasets. Empirical results demonstrate that our method consistently outperforms state-of-the-art approaches.Contribution. In summary, this paper makes three original contributions:• To the best of our knowledge, we are the first to present a joint-motion mutual learning framework for human pose estimation, which collaboratively amalgamates local joint feature and global pixel-level motion flow. • We propose a novel context-aware joint learner guided by heatmaps to retrieve refined local joint-level information from the motion flow. We also learn a comprehensive jointmotion representation by progressive joint-motion mutual learning with information orthogonality objective, which effectively captures the diverse knowledge from complementary features. • Extensive experiments on three widely used benchmark datasets show that our method achieves a new state-of-theart performance. 
related work 2.1 image-based human pose estimation: 
 Before the surge of deep learning, traditional approaches [49, 56, 63] address the human pose estimation task by designing a probabilistic graphical structure or a pictorial structure to model the relationship of human joints. With the advent of deep learning such as CNN [29] , Transformer [53] , diffusion model [21] and the large-scale datasets such as PoseTrack [1, 10, 23] , human pose estimation in static images has been extensively studied and achieved significant breakthroughs [2, 31, 57] . There are two mainstream paradigms: bottom-up and top-down. Bottom-up methods [27, 28, 38] are also known as part-based methods. They first detect all human joints in the given images and then assemble them into individual skeletons.[6] employs a dual branch structure and utilizes part affinity fields to represent pairwise confidence maps between different body parts. [37] assigns an identification tag to each detected part that indicates to which individual it belongs. Although bottom-up methods have obtained promising results, their human part detectors are fragile when the people in the image are far away from the camera position. Contrary to the above method, top-down methods [12, 13, 48] first detect all human bounding boxes in the given images and then perform pose estimation on each individual. [48] proposes a novel framework, named HRNet, which preserves high-resolution features through the multiple stages for precise human pose estimation. [8] proposes a feature pyramid network to detect simple joints and a fine-grained network to identify hard joints which incorporate all the hierarchical representations of the previous network.Applying image-level approaches to video-level pose estimation results in degraded performance due to the neglect of temporal cues inherent in videos. Current methods for video-level human pose estimation can be classified into heatmap-based methods and feature-based methods, depending on whether the approach focuses on refining either heatmaps or visual features extracted from the backbone network.The heatmap-based methods could employ optical flow fields, temporal models (i.e., RNN and Transformer), or heatmap optimization to detect human pose. To capture temporal dynamics in the video, (i) [45, 69] compute dense optical flow between consecutive frames for human pose estimation. However, the computation of optical flow is expensive and largely depends on the quality of the video, and challenging cases such as blurring and character occlusion are very common in videos. (ii) [2] employs an LSTM module to utilize the similarities and temporal relationships between frames, achieving promising results. However, RNNs have limitations in focusing on global temporal information, leading to the development of Transformer-based approaches. [16] propose using a Transformer to model spatiotemporal representation with self-feature refinement and cross-frame temporal learning. (iii) [5, 34] employ unidirectional or bidirectional neighboring heatmaps of consecutive frames to refine the heatmap of the keyframe.The feature-based method tends to fuse meaningful features to perform pose estimation. (i) [35] proposes feature alignments between supporting frames and the keyframe and utilizes the aligned feature to estimate human pose. (ii) [14] employs feature differences and disentanglement to obtain useful visual features to detect human poses. However, visual features contains much task-irrelevant information, such as background and nearby-person. We employ pixel-level optical flow to capture global motion dynamic, which is more robust to challenging scenes. 
method: 
 Preliminaries. Presented with a video, our task is to estimate human poses in each frame. Technically, all human bounding boxes in each frame are identified with the object detector proposed in [39] . These bounding boxes are then enlarged by 25% to crop the identical human across consecutive frames. Subsequently, for person 𝑘, we acquire the cropped video clip I 𝑘 𝑡 =< 𝐼 𝑘 𝑡 -𝛿 , . . . , 𝐼 𝑘 𝑡 , . . . , 𝐼 𝑘 𝑡 +𝛿 > centered on the key frame 𝐼 𝑘 𝑡 (where 𝛿 represents the predefined time span), and aim to estimate the pose of person 𝑘 in frame 𝐼 𝑘 𝑡 . For simplicity, we set 𝛿 = 1 and omit the superscript 𝑘.Method overview. An overview of our approach, JM-Pose, is outlined in Figure 2 . JM-Pose consists of two key components: a Context-aware Joint Learner (CJL, Subsec. 3.1) and a progressive Joint-Motion Mutual learning module (JMML, Subsec. 3.2). Formally, we first extract initial heatmaps Ĥ𝑡 and pixel-level motion flow 𝑀 𝑡 from the input sequence I 𝑡 , and feed them into the context-aware joint learner. The CJL utilizes modulated deformable operations to retrieve the local joint feature 𝐽 𝑡 from motion flow. Subsequently, the progressive joint-motion mutual learning module accepts local joint feature 𝐽 𝑡 and global motion flow 𝑀 𝑡 as inputs, and dynamically exchanges information between them. This yields a more informative joint-motion representation 𝑆 𝑡 . Finally, 𝑆 𝑡 is fed into a detection head to generate the human pose estimation Ĥ𝑡 . In what follows, we will elaborate on the two key components in detail.In order to better leverage local joint-related contexts, we propose to explicitly retrieve the local joint feature from the consecutive frames with the guidance of readily available initial heatmaps. There are two key steps: feature extraction and deformable feature retrieval.Feature extraction. Specifically, given the video clip I 𝑡 , we first employ a backbone [48] network to generate initial heatmaps H =< 𝐻 𝑡 -1 , 𝐻 𝑡 , 𝐻 𝑡 +1 >, which encode possible position information of the joints in each frame. Then, these heatmaps are aggregated to yield the sequence-level representations for the current frame Ĥ𝑡 . In parallel to the heatmap extraction, we also employ RAFT [50] to extract dense motion flow 𝑀 𝑡 , which captures the spatial variation of joint pixel positions over time, i.e., the dynamic information of the pose, as follows:EQUATIONwhere ⊕ represents the concatenation operation and Θ(•, •) represents the RAFT estimator. Deformable feature retrieval. Given Ĥ𝑡 and 𝑀 𝑡 , we further retrieve context-aware joint cues from motion flow 𝑀 𝑡 guided by the initial heatmap Ĥ𝑡 .Instead of fusing 𝑀 𝑡 and Ĥ𝑡 with regular convolution blocks, we employ modulated deformable convolution to extract the contextaware joint feature 𝐽 𝑡 , sampling from motion flow by adaptively learning the offsets and weights of the deformable convolution. Initially, we concatenate the motion flow 𝑀 𝑡 and the heatmap Ĥ𝑡 to generate the fused features 𝑅 𝑡 . Subsequently, given the fused features 𝑅 𝑡 , we compute the kernel sampling offsets 𝐽𝑂 𝑡 and modulated weights 𝐽𝑊 𝑡 by: The context-aware joint learner is designed to extract the local joint-level feature 𝐽 𝑡 from motion flow 𝑀 𝑡 using modulated deformable operations guided by initial heatmap Ĥ𝑡 . Joint-motion mutual learning further refines local joint feature 𝐽 𝑡 and global motion flow 𝑀 𝑡 using their knowledge to complement each other. An information orthogonality objective L 𝐼𝑂 is adopted to improve the diversity of learned 𝐽 𝑡 and 𝑀 𝑡 , which is conditioned on initial heatmap Ĥ𝑡 . The final 𝐿 𝑡ℎ representation is aggregated and fed to the detection head to obtain the final heatmap Ĥ𝑡 for pose estimation. Finally, we employ a heatmap loss L 𝐻 to measure the discrepancy between the ground truth H 𝑡 and the detected heatmaps Ĥ𝑡 .EQUATIONEQUATIONEQUATIONwhere the kernel offset 𝐽𝑂 𝑡 reflects the pixel movement association field, and the modulated weight 𝐽𝑊 𝑡 represents the scalar matrix for the convolution kernel. The network parameters of the two processes for 𝐽𝑂 𝑡 and 𝐽𝑊 𝑡 are learned independently. Finally, we retrieve the context-aware joint feature 𝐽 𝑡 via a modulated deformable convolution [72] as follows:EQUATIONContext-aware joint learner outputs the joint feature 𝐽 𝑡 that encodes local joint dependency and the motion flow 𝑀 𝑡 that encodes global pixel-level motion dynamics. Simply merging these two features may not fully exploit their individual strengths for accurate pose estimation, since joint feature and motion flow essentially have the potential to guide each other. For instance, in scenes with video defocus (poor video quality), local joint feature can supply motion flow with position information for hard-to-detect joints.Conversely, in scenarios of pose occlusion, the occluded joints could be inferred based on global motion dynamics. Given that motion flow and joint feature are complementary, we propose a progressive L-layer joint-motion mutual learning (JMML) paradigm. This process involves a joint-motion interaction block and an information orthogonality objective, aimed at effectively enhancing the expressive capabilities of each feature by densely exchanging valuable information between them. Joint-motion interaction block (JMIB). Here, we elaborate on the 𝑖 𝑡ℎ layer joint-motion interaction block in JMML. As shown in Figure 3 , with 𝐽 𝑖 -1 𝑡 and 𝑀 𝑖 -1 𝑡 produced by 𝑖 -1 𝑡ℎ JMML, 𝑖 𝑡ℎ joint-motion interaction block applies two convolution blocks on them, obtaining Ĵ𝑖-1 𝑡 and M𝑖-1𝑡 . Subsequently, to leverage the complementarity of local joint feature and global motion flow, we propose a joint-motion cross-attention mechanism within the interaction block, which allows joints and motion features to act as mutual guides. Specifically, we generate a joint query matrix 𝑄 𝑗 based on Ĵ𝑖-1 𝑡 while also generating a motion key matrix 𝐾 𝑚 and a motion value matrix 𝑉 𝑚 based on M𝑖-1 𝑡 using independent convolution layers. Similarly, we employ M𝑖-1 𝑡 to generate a motion query matrix 𝑄 𝑚 while also employing Ĵ𝑖-1 𝑡 to generate a joint key matrix 𝐾 𝑗 and a joint value matrix 𝑉 𝑗 . Then, we compute feature compatibility between corresponding query-key pairs using a cross attention mechanism:EQUATIONEQUATIONM′ 𝑖 -1 respectively. The above process can be described as :EQUATIONEQUATIONEQUATIONwhere Φ(•) denotes the operations of channel chunk, ⊕, ⊗, and 𝜎 (•) reveal concatenate, spatial-wise multiplication, and the sigmoid activation function, respectively. Information orthogonality objective. After generating the complementary joint feature 𝐽 𝑖 𝑡 and motion flow 𝑀 𝑖 𝑡 , we further conduct theoretical analysis and propose an information orthogonality objective inspired by mutual information theory [20] . This objective ensures that both joint feature and motion flow provide diverse cues by minimizing redundant information introduced by the mutual learning block.Mutual information measures the information dependency between random variables, which can be formalized as:EQUATIONwhere 𝑥 1 and 𝑥 2 denote the two random variables, 𝑝 (𝑥 1 , 𝑥 2 ) represents the joint probability distribution between 𝑥 1 and 𝑥 2 . 𝑝 (𝑥 1 ) and 𝑝 (𝑥 2 ) represent the marginals of 𝑥 1 and 𝑥 2 , respectively.To ensure the diversity of features extracted from the mutual learning (𝐽 𝑖 𝑡 and 𝑀 𝑖 𝑡 ) and information gain of the initial heatmap Ĥ𝑡 , the proposed information orthogonality loss is given by:EQUATIONAs L 𝑖 𝐼𝑂 in equation 12 is intractable to estimate analytically, we seek to simplify it as depicted in Figure 3 . The information orthogonality loss L 𝑖 𝐼𝑂 can be stated as : (EQUATIONThe first term compresses the relevancy between joint feature 𝐽 𝑖 𝑡 and motion flow 𝑀 𝑖 𝑡 , directing both to focus on more diverse cues. The second term allows the joint feature 𝐽 𝑖 𝑡 to contain more location and dependency information related to the heatmap Ĥ𝑡 . The third term constrains the context-aware joint learner to extract joint feature 𝐽 𝑡 from the motion flow 𝑀 𝑡 . These terms are illustrated in Figure 3 and can be estimated by existing mutual information estimators [35, 51] . In our experiments, we employ the Variational Self-Distillation [51] to estimate each term.Heatmap estimation. Ultimately, we derive the joint feature 𝐽 𝐿 𝑡 and motion flow 𝑀 𝐿 𝑡 through the 𝐿 𝑡ℎ joint-motion mutual learning. We aggregate these two features to obtain a comprehensive representation 𝑆 𝑡 as follows:EQUATIONwhere Υ(•) denotes a convolution block and ⊕ is the concatenation operation. 𝑆 𝑡 is subsequently fed into a detection head to generate the pose heatmap Ĥ𝑡 . The detection head consists of a set of 3 × 3 convolution layers. Through mutual and adaptive refinement of local joint dependency and global pixel-level motion flow, we can obtain complementary cues between them, thereby generating more tailored information that contributes to accurate pose estimation.We adopt a standard heatmap loss L 𝐻 to supervise the final pose estimation:EQUATIONwhere H 𝑡 and Ĥ𝑡 represent the ground truth heatmap and the prediction heatmap, respectively. We also employ the proposed information orthogonality loss to supervise the joint-motion mutual learning as mentioned in Sec. 3.2. The total loss function L 𝑇 𝑜𝑡𝑎𝑙 is given by:EQUATION)where 𝛼 is a trade-off parameter to balance the two loss terms. 
experiments: 
 In this section, we conduct comprehensive experiments on three widely used benchmark datasets for human pose estimation in video. Our overarching goal is to answer the following research questions:• RQ1: How does our method compare with the state-of-theart human pose estimation approaches on quantitative results?• RQ2: How is the proposed method compared against existing human pose estimation methods in challenging scenarios? • RQ3: How does the proposed method compare to the stateof-the-art video human pose estimation methods on visual results? • RQ4: How much do various components of JM-Pose contribute to its overall performance?Next, we first introduce the experimental settings, followed by answering the above research questions one by one.Datasets. We adopt three widely-used pose estimation datasets, namely PoseTrack2017 [23] , PoseTrack2018 [1] , and PoseTrack21 [10] , to evaluate the proposed method.PoseTrack is a large-scale public dataset designed for evaluating human pose estimation methods in video and involves the challenging case of complex movements of highly occluded people in crowded scenes. Specifically, PoseTrack2017 comprises 300 video sequences and 80,144 pose annotations, with 250 video clips allocated for training and the remaining used for validation (split following the official protocol). PoseTrack2018 increases the number of videos and pose annotations, including 593 video clips for training and 170 video clips for validation (with 153,615 pose labels). Each annotated person in both datasets has 15 keypoints positions and a flag indicating joint visibility. Moreover, the training samples provide dense labels in the center 30 frames, and validation videos are additionally labeled every four frames. The Posetrack21 further extends PoseTrack2018 with a focus on small subjects and those in the occluded scenes, encompassing 177,164 pose labels.Evaluation metric. The average precision (AP) is commonly adopted to evaluate the performance for human pose estimation. We further average the AP of all joints (mAP) to derive and compare the final performance.Implementation details. We use PyTorch to implement our method. The input image size is fixed to 384 × 288. For initial heatmap extraction, we leverage the HRNet-W48 model pre-trained on the COCO dataset as our backbone. During the training process, we incorporate several data augmentations including random rotation [-45 • , 45 • ], random scale [0.65, 1.35], random truncation, and flipping. The predefined time span 𝛿 is set to 2. In joint-motion mutual learning, 𝐿 is set to 4. We utilize the AdamW optimizer with an initial learning rate of 1𝑒 -4 (decays to 1𝑒 -5, 1𝑒 -6, 1𝑒 -7 at the 5 𝑡ℎ , 10 𝑡ℎ , 𝑎𝑛𝑑 15 𝑡ℎ epochs, respectively). Our model is trained on 2 Nvidia Geforce RTX 4090 GPUs and terminated within 20 epochs.State-of-the-art Methods (RQ1)Results on the PoseTrack2017 dataset. We first evaluate the performance of our model on the PoseTrack2017 validation dataset. Specifically, we compare 22 methods, and their performance are shown in Table 1 . The proposed JM-Pose consistently outperforms existing approaches, achieving an mAP of 86.4. In comparison to the widely-used backbone network HRNet, our JM-Pose improves the mAP by 9.1 points. Moreover, when compared to the state-ofthe-art method TDMI, JM-Pose delivers a 0.7 mAP gain. Notably, we observe an encouraging improvement for the challenging joints Results on the PoseTrack21 dataset. Table 3 shows that our JM-Pose achieves the best mAP performance on the PoseTrack21 dataset. Experimental results of the first four methods are officially provided by the dataset [10] . [14] offers the quantitive results of three methods (i.e., DCPose [34] , FAMI-Pose [35] , TDMI [14] ). We observe that the previous approach TDMI has delivered a commendable result. In comparison to TDMI, our JM-Pose improves the performance with an mAP of 84.0. Additionally, a notable observation from Table 3 is that JM-Pose achieves an mAP of 82.5 for the wrist joint and 78.5 for the ankle joint.We conduct experiments to evaluate the performance of our method and existing methods on complex and challenging scenes. We select a more difficult subset (Challenging-PoseTrack) accroding to [30] from the PoseTrack dataset [23] . Challenging-PoseTrack consists of 806 frames and contains many challenging scenes, such as pose occlusions, tangleed and crowded person, etc. Seven methods are evaluated in this subset and the quantitative results are presented in Table 4 . The heatmap-based method DCPose tends to fail in these complex scenes, possibly due to the lack of the ability to capture spatio-temporal features, achieving an mAP of 48. Looking at the visuals would be more instructive. We visualize the results for scenes with rapid motion or pose occlusions attest to the robustness of our method. Specifically, we present in Figure 4 the visual comparisons of our JM-Pose against state-of-the-art methods.From Figure 4 , we can observe that JM-Pose displays more accurate pose estimation given the heavily crowded and challenging scenarios. DCPose only leverages temporal heatmap information from neighboring frames to refine the initial pose heatmap, resulting in suboptimal performance. TDMI solely refines and disentangles backbone features yet ignores the important effect of joint cues. For example, in the Rapid-Motion scene (c), the right foot and right knee (pink line) of the woman wearing black in the TDMI red box are to the left of the true keypoint location, while our method identified the correct location. Similarly, in the pose occlusions scene (c), the right knee (pink line) of the woman wearing white pants in the TDMI red box is located to the right of the true keypoint location. At the same time, TDMI did not estimate the right hip of the boy (in the lower right corner of the image), while our method identified the correct location. More visual results of our method are demonstrated in Figure 5 and supplementary material. Through the principled designs of context-aware joint learner and joint-motion mutual learning for mining complementary and informative representations, our JM-Pose is capable of detecting accurate human poses for challenging scenes.To better investigate the contribution of JM-Pose under different settings, we conduct a series of ablation studies, including contextaware joint learner (CJL) and joint-motion mutual learning (JMML). These experiments are conducted on the PoseTrack2017 validation dataset.Study on components of JM-Pose. We conduct experiments to investigate the effect of each component in our JM-Pose and present the empirical result in Table 5 . TDMI serve as a baseline which refines visual features for human pose estimation. However, it neglects local joint dependency present in heatmaps. In addition, JM-Pose* refers to using visual feature differences to extract motion features (from backbone network), which replaces the global pixellevel motion flow in JM-Pose and the mAP decreases from 86.4 to 85.9. This performance drop (↓ 0.5) highlights that our global motion flow could capture pixel-level motion dynamics and is helpful for human pose estimation. In case w/o CJL, we remove the contextaware joint learner (CJL), and the initial heatmap Ĥ𝑡 and motion flow 𝑀 𝑡 are fed into the JMML. We can observe that mAP falls from 86.4 to 85.6. This decline in performance upon removal of the context-aware joint learner suggests the efficacy of the joint-level feature clues encoded by the context-aware joint learner. Moreover, in case w/o JMML, we remove joint-motion mutual learning (JMML), and the local joint feature 𝐽 𝑡 is fed into the detection head. We can observe that mAP falls from 86.4 to 84.6 mAP. This decline in performance upon removal of the joint-motion mutual learning suggests the importance of the mutually aggregate the joint feature and motion flow.Study on components of JMML. Finally, we also examine the contributions of joint-motion mutual learning (JMML) under different settings and report the results in Table 6 . (a) For the first setting, we remove the joint-motion interaction block (JMIB) module, which performs mutual learning between joint feature and motion flow. It should be noted that after removing JMIB, our proposed information orthogonality objective (L 𝐼𝑂 ) will constrain the joint feature and motion flow in the context-aware joint learner to generate diverse joint feature. The mAP results drop from 86.4 to 85.0. This performance deterioration on top of JMIB corroborates the importance of excavating the complementary joint-motion cues in guiding more accurate estimations of the joint locations. (b) For the next setting, we remove the information orthogonality objective (L 𝐼𝑂 ) from JMML, which minimizes redundant information and differentiates the context-aware joint feature and global motion flow. We observe the performance drops from 86.4 mAP to 85.8 mAP. This demonstrates the importance of our L 𝐼𝑂 objective in introducing meaningful and diverse joint feature and motion flow to enhance human pose estimation.ConclusionIn this work, we explore human pose estimation from the perspective of effectively integrating context-aware local joint feature and global pixel-level motion flow. We propose a novel joint-motion mutual learning framework termed JM-Pose. We propose a contextaware joint learner guided by heatmaps to retrieve local joint-level information from the motion flow. A particular highlight of our method is the progressive joint-motion mutual learning to exchange information between joint feature and motion flow, improving the performance of pose estimation. Theoretically, we further propose an information orthogonality objective for effective feature diversity supervision of joint-motion interactive blocks. Empirical evaluation on three large-scale datasets shows that our method achieves notable improvements, showcasing the ability of our method to detect human poses in complex and challenging scenes. 
 
 Appendix:  


Decision: Reviewers pointed out that the observations and the ideas of leveraging the complementarity of local joint feature and global motion flow are novel. The designed context-aware joint learner (CJL) and progressive joint-motion mutual learning (JMML) scheme are interesting.
This work also provides extensive experimental comparisons with state-of-the-arts on benchmarking datasets. All the reviewers thus recommend acceptance. AC agrees with the reviewers.

Reviews: review: paper_summary: paper_summary: This paper aims to improve human pose estimation in videos, based on the hypothesis that local information (about human joint positions) and global information (about motion dynamics) are complementary, and both are crucial to accurate estimation results, particularly in challenging scenarios.
A joint-motion mutual learning framework is proposed to combine useful information in joint detection heatmaps ("local") with dense optical flows ("global"), which utilizes modulated deformable convolutions and stacked cross-attention blocks.
Experimental results demonstrate its superiority over previous methods that only leverage either "local" or "global" features.
strength_weakness: strength_weakness: **Novelty and clarity**:
The observations and the ideas of leveraging the complementarity of local joint feature and global motion flow are somewhat novel.
The comparison between heatmap-based methods and feature-based methods in Figure 1 clearly presents the motivation and the philosophy.

**Significant results**:
Quantitative results presented in Table 1-4 show that the proposed method significantly outperforms previous state-of-the-art methods in various scenarios by a large margin. The visual comparisons also demonstrate the weaknesses of previous methods and the strengths of the proposed one.
limitations: limitations: **Insufficient evidence**:
Though the evaluation results show that the proposed method outperforms previous studies notably, it has not yet been proved that these components (more specifically, JMIB blocks) work as expected, i.e., "progressively exchanging valuable information between joint feature and motion flow, hence enhancing both branches orthogonally and comprehensively".

More evidence is required to show that these $M_t^i$ and $J_t^i$ are indeed progressively improved, and a stack of $L$ consecutive blocks is essential for this progressive process, rather than just enlarging model capacity.

For example, the performances of training detection heads on intermediate blocks $l=1,...,L-1$ could be reported to show a progressive refinement process.
Additionally, is the $L_{IO}^i$ value (or other metrics) progressively decreasing when $i$ increases from $1$ to $L$?
Visualizations of $M_t^i$ and $J_t^i$ may also be helpful if they are meaningful.

**Comparison fairness**:
Moreover, the information about model capacity (e.g., parameter count, inference speed, flops) is missing when comparing to other methods. Since the proposed method involves new operations like deformable convolution and cross-attention, such information is required to ensure a fair comparison.
paper_summary: paper_summary: This paper proposes a joint-motion mutual learning framework for pose estimation, which effectively concentrates on both local joint dependency and global pixel-level motion dynamics. It introduces a context-aware joint learner that adaptively leverages initial heatmaps and motion flow to retrieve robust local joint feature. Empirical experiments show this method outperforms prior arts on three challenging benchmarks.
strength_weakness: strength_weakness: - The paper proposes a novel method by integrating both heatmap-based and feature-based paradigms to address the challenges (e.g. video defocus and self-occlusion) in complex video scenes for video pose estimation.
- Extensive analysis and visualization of existing paradigms in Fig.1.
- Anonymous codes are provided.
limitations: limitations: - Its contributions, e.g. CJL and JMML, are limited to several slight technical modifications and combinations to the well-established technology in video domain (Motion-Appearance Interaction via transformer, gated weight and deformable operations).
- It seems that no component is specifically designed for pose prediction or joint. It lacks significant distinction from appearance-motion mutual learning frameworks designed for other video tasks.

Minor Limitation
- The local joint dependency means the temporally local information among consecutive frames or spatially local information within each frame, stating this information in the Introduction for better readability. What are the differences between the proposed method and the common Image (spatial)/Optical Flow (temporal) based methods?
- Adding several multi-modality fusion methods (Appearance-Motion) of video tasks in Related work can help to understand the differences from other methods.
- There should be more detailed labels in Fig.2, such as JO and JW. The related information should be available in the figures without reading corresponded contents.
- RAFT can generate high-quality flow map, while you can use PWCNet or FlowNet to verify the model's robustness and generalization ability with low-quality optical flow.
paper_summary: paper_summary: This paper proposes a novel joint-motion mutual learning framework for human pose estimation in videos, which addresses the limitations of existing methods that fail to leverage both local joint information and global motion dynamics. The framework introduces a context-aware joint learner that utilizes initial heatmaps and motion flow to extract robust joint features. A progressive joint-motion mutual learning mechanism is further proposed to enhance the model's performance by synergistically exchanging information between joint features and motion flow. An information orthogonality objective is introduced to prevent learning redundant information. Experimental results show the proposed method outperforms previous works on challenging benchmarks.
strength_weakness: strength_weakness: This article proposes a new method for joint training and attitude estimation using motion features in videos. Although the idea of using motion features is not novel, the author fully considers the modeling relationship between motion and joints, and uses deformable conv to achieve effective feature fusion between joints and motion, thus obtaining good experimental results.
limitations: limitations: In my opinion, there are no significant flaws or shortcomings in this paper. However, due to the reliance of the paper on motion+heatmap, a classic solution in video pose estimation, its novelty is undoubtedly limited. The hypothesis and framework of motion+joint advocated by the author may seem reasonable, but there are still some rough details, such as the symmetrical model design of JMML, which integrates motion and joint features as the same type of features, abandoning the relationship modeling between joints and motion constructed using DCN earlier. I believe that JMML should have a better architecture design.
paper_summary: paper_summary: This paper proposed a joint-motion mutual learning framework for video pose estimation. Specifically, a context-aware joint learner is introduced to learn joint features between initial heatmaps and motion flow. In addition, a progressive joint-motion mutual learning structure is developed to fuse local join features and global motion flow. Experimental results show that the proposed method can outperform the state-of-the-art on several benchmark datasets.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. The proposed context-aware joint learner (CJL) and progressive joint-motion mutual learning (JMML) sound interesting.
3. The paper provides sufficient experimental comparisons with state-of-the-art methods on several benchmark datasets.
limitations: limitations: 1. My major concern is the contribution of this paper. To me, the proposed model is more like a multi-modal feature fusion framework, and the core idea is how to fuse intra-frame spatial $\hat{H}_t$ and inter-frame temporal feature $M_t$, i.e., motion flow, for video pose estimation. So what is the key insight of this proposed model? What is the difference between the proposed model and existing works?

2. Could authors elaborate more on the definitions of “local joint dependency” and “global pixel-level motion dynamics”? How to define the concepts of “local” and “global” here? What is “global pixel-level motion dynamics”? I mean the motion flow is just calculated by two consecutive frames? Why is it global?

3. Figure 2(b) CJL block shows that the joint feature $\hat{H}_t$ and the motion flow $M_t$ are concatenated and then fed into two separate convolutional layers for producing the offset and convolution weights of the deformable convolution. I wonder why the motion flow should be modulated by using such a deformable convolution. What is the rationale behind this?

4. If the motion flow $M_t$ should be modulated, then the $J_t$ will be regarded as another form of motion cues. Therefore, the inputs of JMML are from the same modality, but it means no joint features are fed into the JMML. I wonder how to perform “joint-motion” mutual learning in this setting.

5. The notations in Figure 1 are confusing. For example, $H_{t-1}$ denotes the extracted feature of $I_{t-\delta}$, so as the $H_{t+1}$ and the $I_{t+\delta}$. As mentioned in the paper, here $\delta = 1$. But the figure shows that the $\delta$ has suddenly disappeared. In addition, $M_t^r$ and $M_t^l$ are not well-explained and not mentioned in the main text, maybe the superscript $r$ and $l$ refer to “right” and “left” motion flow, respectively?

6. For the JMML, Sec 3.2, could authors elaborate more on why the information orthogonality objective should be achieved in the multi-modal feature fusion? The motivation for introducing such information orthogonality objective is not clear to me.