## 1 Introduction  
Hand-object interaction tasks have a wide range of application areas, from virtual reality (VR) and augmented reality (AR) to human-computer interaction. Also, recognizing hand movements from first-person RGB videos has received significant attention from researchers due to the rich visual information provided by RGB videos. On the one hand, the inherent complexity of hand articulation, combined with frequent occlusions (either self-occlusions between fingers or occlusions by objects), severely impedes the capture of complete hand information during the feature extraction stage. On the other hand, compounding this difficulty, hand movements within complex RGB video scenes constitute inherently continuous temporal sequences. Effectively comprehending spatiotemporal action demands the precise capture of nuanced inter-frame action coherence and subtle motion dynamics, necessitating robust temporal modeling capabilities. Notably, there is a high correlation between 3D hand pose estimation and hand movement recognition [1], yet there are relatively few joint studies on the two. To address this issue, recent studies have started to explore frameworks that combine hand pose estimation with action recognition [2, 3, 4, 5]for more accurate action understanding. However, the task still faces many challenges. [PERSON] et al. [3] proposed an end-to-end deep learning model, H+O, which, for the first time, modeled hand pose estimation, object pose and interaction state understanding in a unified way, emphasizing that the interaction between the two is crucial for action understanding in a first-person perspective. Building upon dual-task frameworks, [PERSON] et al. [5] proposed a multi-order multi-stream feature analysis method using weakly supervised network learning to learn gesture and multi-order motion information from the intermediate feature maps of the video, which is capable of learning gesture recognition and 3D hand pose estimation, even when only pose labels are available. Crucially, their paradigm capitalizes on the mutual reinforcement between hand pose semantics and action contexts, thereby partially resolving occlusion-induced ambiguities. In parallel, [PERSON] et al. [2] extended [PERSON] et al. 's [3] foundation via a topology-aware graph convolutional network (TA-GCN), explicitly modeling holistic hand-object interdependencies. By integrating gesture kinematics, object-state dynamics, and their bidirectional relational constraints, their approach significantly advances interaction modeling precision. Despite these methodological innovations, temporal coherence across sequential frames remains inadequately optimized, limiting contextual continuity in video analysis. Consequently, extracting accurate hand motion semantics from first-person RGB footage persists as a formidable challenge. In this study, we present a framework for recognizing hand movements and hand gestures from first-person RGB videos. The framework consists of two main modules: the Hand Pose Estimation Module and the Action Recognition Module, as shown in Figure 1. In the Hand Pose Estimation Module, individual video frames are first passed through a feature extractor and then passed into a series of transformer encoders. We construct a dual residual network structure and use a multi-layer perceptron to predict the hand pose results and object categories for each frame. High-level semantic features and low-level spatial details are jointly modelled via two residual paths. The capture of fine-grained spatial information is enhanced using a spatial detail feature extraction component while retaining global feature consistency. The hand pose results, object categories, and feature information of each frame are concatenated for the subsequent action recognition task. In the Action Recognition Module, the vectors of each frame are aggregated by means of a multi-layer transformer encoder.