## 5 Conclusions  
In this study, we propose a unified framework for recognizing hand actions and hand poses from first-person RGB videos. The framework effectively predicts the hand key point information and object category information for a single frame, and, at the same time, it is able to output the hand action categories in consecutive video frames. Specifically, HPEM estimates the hand key points in each image frame to obtain hand spatial information. Each video frame is first subjected to global feature extraction by FlexiViT and then fed to multiple morphing encoders to capture the relative positions of elements in the time-series data. Our hand pose estimator employs a dual residual structure to introduce a Spatially Detailed Feature Extraction Part, which uses MLP to predict hand key point coordinates and object categories. The hand pose results and object categories' information--along with the corresponding frame-level features--are then concatenated to fully exploit spatial information from each RGB frame. The Action Recognition Module (ARM) aggregates feature across consecutive frames to reconstruct the hand's motion trajectory, thereby capturing temporal dynamics over time. Finally, an action classifier determines the action class based on the fused spatiotemporal representations derived from the entire video sequence. We assess our method using two first-person hand action datasets, with experimental results demonstrating its strong performance on both the FPHA and H2O datasets. We focus on using the transformer architecture to address the problem of a unified framework for pose estimation and action recognition, including the introduction of Flexivit into our approach during the initial feature extraction phase. However, when it comes to multi-task joint learning as well as transformer architectures, the computational complexity of the model will be higher and face more computational pressure, whereas the sparse attention mechanism aims to significantly reduce the computational complexity and storage requirements by selectively focusing on key parts of the sequence to improve the efficiency and performance of the model when dealing with long sequential data, which we leave as future work. Conceptualization, J.Y., X.W. and Q.G. ; methodology, J.Y., J.L. and H.P. ; software, J.Y., Y.C. and X.W. ; validation, H.P., Q.G., Y.C. and X.W. ; formal analysis, J.Y., X.W. and Q.G. ; investigation, J.Y., X.W. and Q.G. ; resources, J.Y., X.W., Y.C. and Q.G. ; data curation, J.Y., Q.G. and J.L. ; writing--original draft preparation, J.Y. ; writing--review and editing, Y.C., Q.G., H.P., J.L. and X.W. ; visualization, J.Y. ; supervision, Q.G. and X.W. ; project administration, J.Y., X.W. and Q.G. ; funding acquisition, Q.G. and X.W. All authors have read and agreed to the published version of the manuscript. This research was supported by the Natural Science Foundation of China (No. 62072362, 12101479), Natural Science Basis Research Plan in Shaanxi Province of China (Nos. 2021)Q-660 and 2024 JC-YBMS-531), Shaanxi Provincial Innovation Capacity Support Programme Project (No. 2024 ZC-KJX-034), and Xi'an Major Scientific and Technological Achievements Transformation Industrialization Project (No. 23 CGZHCYH0008). Data Availability Statement: Here, we provide links to the two datasets used in our experiments. FPHA: [[https://guiggh.github.io/publications/first-person-hands/](https://guiggh.github.io/publications/first-person-hands/)]([https://guiggh.github.io/publications/first-person-hands/](https://guiggh.github.io/publications/first-person-hands/)) (accessed on 11 September 2024); H2O: [[https://taeinkwon.com/projects/h2o/](https://taeinkwon.com/projects/h2o/)]([https://taeinkwon.com/projects/h2o/](https://taeinkwon.com/projects/h2o/)) (accessed on 4 July 2024). **Acknowledgments:** We thank the editor and the anonymous reviewers for their valuable suggestions to improve the quality of this study.