## 3 Methodology
### Hand Pose Estimation Module
A transformer Encoder Block significantly enhances its ability to capture spatiotemporal dependencies and further enriches the feature representation. In order to further strengthen the ability to perceive the local structure of the hand and the region around the key point, more accurate spatial details are extracted on the basis of preserving the original features, and the problems of gradient vanishing and gradient explosion in deep network training are effectively mitigated. We designed a hand pose estimator based on a dual residual network structure, including two parts: a Spatially Detailed Feature extraction part and an estimation part. For the Spatially Detailed Feature Extraction Part, multi-scale parallel convolution operation and enhanced parallel attention operation are used. The standard convolution is able to expand the sensory field and extract global features, and the multiscale parallel depth-separable convolution operation captures different levels of image information. Here, \(3\times 3\), \(5\times 5\), and \(7\times 7\) sized convolution kernels are used and then the results are fused. In addition, to improve the model's attention to the key features of the hand, a parallel structure is used here to fuse different types of attention mechanisms, including channel attention and pixel attention. Among them, the pixel attention focuses on the accurate extraction of hand position-related information, while the channel attention can efficiently extract global information while adjusting the feature channel dimensions. Specifically, we use BatchNorm for normalization to splice the results of these two attention mechanisms along the channel dimension. An MLP is applied to project the concatenated features back to the original input dimensionality. We concatenate the Spatially Detailed Features \(f_{det}(t)\) obtained from the above operation with the global context features \(f_{glo}(t)\) by residual concatenation to obtain \(f^{\prime}(t)\)  
\[f^{\prime}(t)=\Big{\|}\ f_{det}(t)+f_{glo}(t) \tag{1}\]  
where \(\|\) is the concatenation operation. Finally, a multi-layer perceptron is used to accurately predict the coordinates of the hand key points in each frame from the features \(f^{\prime}(t)\):  
\[H_{t}=\left(H^{pre},H^{pre}_{dep}\right)=\text{FC}\ \big{[}\ MLP\left(f^{\prime}(t) \right)\big{]} \tag{2}\]where \(\rm H_{t}\) is the concatenation of the \(\rm H^{pre}\) and \(\rm H^{pre}_{dep}\), \(\rm H^{pre}\) denotes the joint coordinates in the 2D image, \(\rm H^{pre}_{dep}\) denotes the joint depth to the camera. The FC layer maps the features to the new space, and, similarly, we predict the object class from the feature \(f\ ^{\prime}(t)\) and fuse the object category prediction as auxiliary information into the action recognition module. Finally, inspired by [37], we concatenate the hand pose estimates, object category predictions, and frame-wise visual features \(f\ ^{\prime}(t)\) to construct the input representation for the action recognition module. Figure 3 shows the importance of object category information for disambiguation. This fusion strategy preserves information integrity while maintaining computational efficiency, explicitly bridging high-level semantic cues with low-level visual details, thus improving discriminative capability for complex hand-object interactions. Specifically, for a frame \(\rm t\in set\subset R^{W\times H\times 3}\) with \(\rm set=\{set_{i}\}_{i=1}^{i}\), \(\rm F_{HPEM}(t)\) is the result of HPEM calculation:  
\[F_{HPEM}(t)=\big{\|}\ H_{t}+O_{t}+f\ ^{\prime}(t) \tag{3}\]  
where \(\big{\|}\) is the concatenation operation; \(\rm O_{t}\) is the probability distribution of object classification.