Personalized motion kernel learning for human pose estimation

International Journal of Intelligent SystemsVolume 37, Issue 9 p. 5859-5879 RESEARCH ARTICLE Personalized motion kernel learning for human pose estimation Xue Wang, Xue Wang College of Computer Science and Technology, Jilin University, Changchun, China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, ChinaSearch for more papers by this authorRunyang Feng, Runyang Feng orcid.org/0000-0002-4912-6646 School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, ChinaSearch for more papers by this authorHaoming Chen, Corresponding Author Haoming Chen [email protected] School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, China Correspondence Haoming Chen and Zhenguang Liu, School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou 310018, China. Email: [email protected] and [email protected]Search for more papers by this authorRoger Zimmermann, Roger Zimmermann School of Computing, National University of Singapore, Singapore, SingaporeSearch for more papers by this authorZhenguang Liu, Corresponding Author Zhenguang Liu [email protected] School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, China Correspondence Haoming Chen and Zhenguang Liu, School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou 310018, China. Email: [email protected] and [email protected]Search for more papers by this authorHengchang Liu, Hengchang Liu Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, ChinaSearch for more papers by this author Xue Wang, Xue Wang College of Computer Science and Technology, Jilin University, Changchun, China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, ChinaSearch for more papers by this authorRunyang Feng, Runyang Feng orcid.org/0000-0002-4912-6646 School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, ChinaSearch for more papers by this authorHaoming Chen, Corresponding Author Haoming Chen [email protected] School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, China Correspondence Haoming Chen and Zhenguang Liu, School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou 310018, China. Email: [email protected] and [email protected]Search for more papers by this authorRoger Zimmermann, Roger Zimmermann School of Computing, National University of Singapore, Singapore, SingaporeSearch for more papers by this authorZhenguang Liu, Corresponding Author Zhenguang Liu [email protected] School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, China Correspondence Haoming Chen and Zhenguang Liu, School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou 310018, China. Email: [email protected] and [email protected]Search for more papers by this authorHengchang Liu, Hengchang Liu Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, ChinaSearch for more papers by this author First published: 12 January 2022 https://doi.org/10.1002/int.22817Citations: 1Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinkedInRedditWechat Abstract Estimating human poses from a video is at the foundation of many visual intelligent systems. Various convolutional neural networks have been proposed, achieving state-of-the-art performance on different image datasets. However, most existing approaches are image based, which deliver unreliable estimations on videos since they fail to model temporal consistency across video frames. Recently, another line of work leverages temporal cues for multi-frame person pose estimation, yet still in an instance-unaware fashion, disregarding the specific traits of different instances (persons) or different joints. In this paper, we propose a novel approach to learn specific keypoint motion representations for each person, termed Personalized Motion-Aware Network (PMAN). In the PMAN, we devise three components: (i) an Instance-Sensitive Extractor that adaptively computes the spatial features according to human physical characteristics; (ii) a Keypoint Motion Encoder that separately generates convolution kernels with fine-grained keypoint motion encoding; (iii) a Motion Driven Decoder that parses multi-frame spatial features of the same person to provide precise human pose estimations. Extensive experiments on PoseTrack2017 and PoseTrack2018 datasets demonstrate that our approach greatly improves the performance of multi-frame human pose estimation. It is worth mentioning that our approach surpasses the state-of-the-art method by +1.7 mAP and achieves 82.9 mAP on PoseTrack2017 dataset. REFERENCES 1Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C. DeepFlow: large displacement optical flow with deep matching. In: Proceedings of the IEEE International Conference on Computer Vision; 2013: 1385-1392. 2Fang HS, Xie S, Tai YW, Lu C. RMPE: regional multi-person pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision; 2017: 2334-2343. 3Cao Z, Simon T, Wei SE, Sheikh Y. Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017: 7291-7299. 4Sun K, Xiao B, Liu D, Wang J. Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 5693-5703. 5Xiao B, Wu H, Wei Y. Simple baselines for human pose estimation and tracking. In: Proceedings of the European Conference on Computer Vision (ECCV); 2018: 466-481. 6Wei SE, Ramakrishna V, Kanade T, Sheikh Y. Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016: 4724-4732. 7Newell A, Yang K, Deng J. Stacked hourglass networks for human pose estimation. In: European Conference on Computer Vision. Springer; 2016: 483-499. 8Li C, Lee GH. From synthetic to real: unsupervised domain adaptation for animal pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 1482-1491. 9Wang M, Hong R, Yuan XT, Yan S, Chua TS. Movie2comics: towards a lively video content presentation. IEEE Trans Multimedia. 2012; 14(3): 858-870. 10Yang X, Wang M, Tao D. Person re-identification with metric learning using privileged information. IEEE Trans Image Process. 2017; 27(2): 791-805. 11Yang X, Wang M, Hong R, Tian Q, Rui Y. Enhancing person re-identification in a self-trained subspace. ACM Trans Multim Comput Commun Appl. 2017; 13(3): 1-23. 12Jin S, Liu W, Ouyang W, Qian C. Multi-person articulated tracking with spatial and temporal embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 5664-5673. 13Guo H, Tang T, Luo G, Chen R, Lu Y, Wen L. Multi-domain pose network for multi-person pose estimation and tracking. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops; 2018. 14Bao Q, Liu W, Cheng Y, Zhou B, Mei T. Pose-guided tracking-by-detection: robust multi-person pose tracking. IEEE Trans Multimedia. 2020; 23: 161-175. 15Raaj Y, Idrees H, Hidalgo G, Sheikh Y. Efficient online multi-person 2d pose tracking with recurrent spatio-temporal affinity fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 4620-4628. 16Ning G, Liu P, Fan X, Zhang C. A top-down approach to articulated human pose estimation and tracking. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops; 2018. 17Yang X, Du X, Wang M. Learning to match on graph for fashion compatibility modeling. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2020: 287-294. 18Wang M, Hong R, Li G, Zha ZJ, Yan S, Chua TS. Event driven web video summarization by tag localization and key-shot identification. IEEE Trans Multimedia. 2012; 14(4): 975-985. 19Pfister T, Charles J, Zisserman A. Flowing ConvNets for human pose estimation in videos. In: Proceedings of the IEEE International Conference on Computer Vision; 2015: 1913-1921. 20Song J, Wang L, Van Gool L, Hilliges O. Thin-slicing network: a deep structured model for pose estimation in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017: 4220-4229. 21Luo Y, Ren J, Wang Z, et al. LSTM pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018: 5207-5215. 22Wang M, Tighe J, Modolo D. Combining detection and tracking for human pose estimation in videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020: 11088-11096. 23Liu Z, Chen H, Feng R, et al. Deep dual consecutive network for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 525-534. 24Geng Z, Sun K, Xiao B, Zhang Z, Wang J. Bottom-up human pose estimation via disentangled keypoint regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 14676-14686. 25Luo Z, Wang Z, Huang Y, Wang L, Tan T, Zhou E. Rethinking the heatmap regression for bottom-up human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 13264-13273. 26Wei F, Sun X, Li H, Wang J, Lin S. Point-set anchors for object detection, instance segmentation and pose estimation. In: European Conference on Computer Vision. Springer; 2020: 527-544. 27Jin S, Liu W, Xie E, et al. Differentiable hierarchical graph grouping for multi-person pose estimation. In: European Conference on Computer Vision. Springer; 2020: 718-734. 28Li J, Su W, Wang Z. Simple pose: rethinking and improving a bottom-up approach for multi-person pose estimation. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2020: 11354-11361. 29Nie X, Feng J, Xing J, Yan S. Pose partition networks for multi-person pose estimation. In: Proceedings of the European Conference on Computer Vision (ECCV); 2018: 684-699. 30Nie X, Li Y, Luo L, Zhang N, Feng J. Dynamic kernel distillation for efficient pose estimation in videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019: 6942-6950. 31Zhang F, Zhu X, Ye M. Fast human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 3517-3526. 32Moon G, Chang JY, Lee KM. PoseFix: model-agnostic general human pose refinement network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 7773-7781. 33Newell A, Huang Z, Deng J. Associative embedding: end-to-end learning for joint detection and grouping. In Proceedings of the 31st International Conference on Neural Information Processing Systems; 2017: 2274-2284. 34Kreiss S, Bertoni L, Alahi A. PifPaf: composite fields for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 11977-11986. 35Cheng B, Xiao B, Wang J, Shi H, Huang TS, Zhang L. HigherHRNet: scale-aware representation learning for bottom-up human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020: 5386-5395. 36Qiao S, Chen LC, Yuille A. Detectors: detecting objects with recursive feature pyramid and switchable atrous convolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 10213-10224. 37Cai Z, Vasconcelos N. Cascade R-CNN: delving into high quality object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018: 6154-6162. 38Toshev A, Szegedy C. DeepPose: human pose estimation via deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014: 1653-1660. 39Yu C, Xiao B, Gao C, et al. Lite-HRNet: a lightweight high-resolution network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 10440-10450. 40Liu X, Liu H, Lin Y. Video frame interpolation via optical flow estimation with image inpainting. Int J Intell Syst. 2020; 35(12): 2087-2102. 41Huang YF, Shih LP, Tsai CH, Shen GT. Describing video scenarios using deep learning techniques. Int J Intell Syst. 2021; 36(6): 2465-2490. 42Liu Z, Su P, Wu S, et al. Motion prediction using trajectory cues. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2021: 13299-13308. 43Zhang D, Shah M. Human pose estimation in videos. In: Proceedings of the IEEE International Conference on Computer Vision; 2015: 2012-2020. 44Zhang D, Guo G, Huang D, Han J. PoseFlow: a deep motion representation for understanding human behaviors in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018: 6762-6770. 45Artacho B, Savakis A. UniPose: unified human pose estimation in single images and videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020: 7035-7044. 46Gkioxari G, Toshev A, Jaitly N. Chained predictions using convolutional neural networks. In: European Conference on Computer Vision. Springer; 2016: 728-743. 47Yang Y, Ren Z, Li H, Zhou C, Wang X, Hua G. Learning dynamics via graph neural networks for human pose estimation and tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 8074-8084. 48Yu D, Su K, Sun J, Wang C. Multi-person pose estimation for pose tracking with enhanced cascaded pyramid network. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops; 2018. 49Zhou C, Ren Z, Hua G. Temporal keypoint matching and refinement network for pose estimation and tracking. In: European Conference on Computer Vision. Springer; 2020: 680-695. 50Bertasius G, Feichtenhofer C, Tran D, Shi J, Torresani L. Learning temporal pose estimation from sparsely-labeled videos. In: Advances in Neural Information Processing Systems; 2019: 3027-3038. 51Zhang Y, Wang Y, Camps O, Sznaier M. Key frame proposal network for efficient pose estimation in videos. In: European Conference on Computer Vision. Springer; 2020: 609-625. 52Jia X, De Brabandere B, Tuytelaars T, Gool LV. Dynamic filter networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems; 2016: 667-675. 53Zamora Esquivel J, Cruz Vargas A, Lopez Meyer P, Tickoo O. Adaptive convolutional kernels. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops; 2019. 54Chen Y, Dai X, Liu M, Chen D, Yuan L, Liu Z. Dynamic convolution: attention over convolution kernels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020: 11030-11039. 55Zhou J, Jampani V, Pi Z, Liu Q, Yang MH. Decoupled dynamic filter networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 6647-6656. 56Zhang R, Tang S, Zhang Y, Li J, Yan S. Scale-adaptive convolutions for scene parsing. In: Proceedings of the IEEE International Conference on Computer Vision; 2017: 2031-2039. 57Tian Z, Shen C, Chen H. Conditional convolutions for instance segmentation. In: Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK, August 23-28, 2020 (ECCV 2020), Part I 16. Springer; 2020: 282-298. 58Pang Y, Zhang L, Zhao X, Lu H. Hierarchical dynamic filtering network for RGB-D salient object detection. In: Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK, August 23-28, 2020 (ECCV 2020), Part XXV 16. Springer; 2020: 235-252. 59Zhang J, Xie Y, Xia Y, Shen C. DoDNet: learning to segment multi-organ and tumors from multiple partially labeled datasets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021: 1195-1204. 60Tabernik D, Kristan M, Leonardis A. Spatially-adaptive filter units for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018: 9388-9396. 61Tabernik D, Kristan M, Leonardis A. Spatially-adaptive filter units for compact and efficient deep neural networks. Int J Comput Vis. 2020; 128(8): 2049-2067. 62Dai J, Qi H, Xiong Y, et al. Deformable convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision; 2017: 764-773. 63Zhu X, Hu H, Lin S, Dai J. Deformable ConvNets v2: more deformable, better results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 9308-9316. 64Ma N, Zhang X, Huang J, Sun J. WeightNet: revisiting the design space of weight networks. In: Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK, August 23-28, 2020 (ECCV 2020), Part XV 16. Springer; 2020: 776-792. 65Yang B, Bender G, Le QV, Ngiam J. CondConv: conditionally parameterized convolutions for efficient inference. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems; 2019: 1307-1318. 66Wang J, Chen K, Xu R, Liu Z, Loy CC, Lin D. Carafe: content-aware reassembly of features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019: 3007-3016. 67Wang X, Zhang R, Kong T, Li L, Shen C. SOLOv2: Dynamic and Fast Instance Segmentation. In: Advances in Neural Information Processing Systems, 2020; 33. 68Liu Z, Wu S, Jin S, et al. Towards natural and accurate future motion prediction of humans and animals. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019: 10004-10012. 69Andriluka M, Iqbal U, Insafutdinov E, et al. PoseTrack: a benchmark for human pose estimation and tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018: 5167-5176. 70Girdhar R, Gkioxari G, Torresani L, Paluri M, Tran D. Detect-and-track: efficient pose estimation in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018: 350-359. 71Xiu Y, Li J, Wang H, Fang Y, Lu C. Pose flow: efficient online pose tracking. arXiv preprint arXiv:1802.00977. 2018. 72Doering A, Iqbal U, Gall J. Joint flow: temporal flow fields for multi person tracking. arXiv preprint arXiv:1805.04596. 2018. 73Zhang J, Zhu Z, Zou W, et al. FastPose: towards real-time pose estimation and tracking via scale-normalized multi-task networks. arXiv preprint arXiv:1908.05593. 2019. 74Hwang J, Lee J, Park S, Kwak N. Pose estimator and tracker using temporal flow maps for limbs. In: 2019 International Joint Conference on Neural Networks (IJCNN). IEEE; 2019: 1-8. 75Snower M, Kadav A, Lai F, Graf HP. 15 keypoints is all you need. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020: 6738-6748. Citing Literature Volume37, Issue9September 2022Pages 5859-5879 ReferencesRelatedInformation