## 3 Methodology
### Hand Pose Estimation Module  
To accurately capture the spatial positioning of the hand in an RGB video, the task of predicting hand key point coordinates and object category information for each frame from a given first-person RGB video is implemented. We designed the lightweight Hand Pose Estimation Module based on transformer and dual residual network structure. The module is described in detail in this subsection. Due to a large amount of redundant information between neighboring frames of the video, the direct computation cost of the transformer-based approach is high. If the whole video is directly fed into the transformer-based model for processing, it will not only bring high computational overhead but may also cause memory overflow problems when processing multiple frames at the same time. To mitigate this, we divide the original video sequence \(\upsilon\) into j consecutive subsets, each of which \(\mathrm{set}=\{\mathrm{set}_{i}\}_{i=1}^{\mathrm{j}}\) is processed as an independent input to the network. This approach reduces the computational complexity while ensuring temporal continuity, effectively preserves the ability to perceive dynamic gesture changes, and, thus, improves the overall inference efficiency. Meanwhile, from the perspective of model training efficiency and performance, this approach helps to control the amount of data in each batch of inputs and achieve stable training and efficient inference  
Figure 2: Schematic of the overall framework. Our framework comprises two main modules: the Hand Pose Estimation Module (HPEM) and the Action Recognition Module (ARM). For the input video \(\upsilon\), the spatial position of the hand in a single video frame is captured by HPEM. The spatial information is obtained by estimating the key points of the hand in each frame using a dual residual network structure and multi-layer perceptron. The ARM captures the temporal information of the hand between video frames by aggregating the information of each frame through a multi-layer Transformer encoder and ultimately outputs the class of hand actions in consecutive video frames. under limited hardware resources. For each input frame \(t\in\text{set}\subset\text{R}^{W\times H\times 3}\), the pre-trained FlexiViT [36] architecture is used as our backbone network to extract feature information from each frame, where W represents the width and H represents the height of the input image. Specifically, using 16 \(\times\) 16 image block segmentation, each image block is mapped to an embedding space of dimension 192 by linear projection, thus forming an embedding vector with semantic information for subsequent feature learning. A 12-layer transformer encoder structure is used, with each layer containing 3 parallel attention heads. While maintaining a low computational complexity, multi-layer image features are gradually extracted to enhance the feature representation capability. In order to efficiently model the temporal dependence in the input video sequence, we generate the position information of the input sequence by means of fixed sine/cosine position encoding [32]. This enables the model to more effectively capture the relative positions of elements within the time-series data. Each frame of the visual feature sequence obtained through the FlexiViT backbone and its corresponding position encoding is passed to the transformer Encoder Block for processing to obtain the global context features \(f_{glo}(t)\). The transformer Encoder Block is one of the core components of the network responsible for modelling long-range dependencies and mining contextual associations between features. Also, Layer Normalization is applied prior to the attention and feed-forward operations to stabilize the training process. A transformer Encoder Block is composed of six stacked encoder layers, each comprising a multi-head deformable self-attention layer with eight heads and a feed-forward network with a dimension of 2048.