###### Abstract  
The advancement of deep learning techniques has significantly propelled the development of the continuous sign language recognition (cSLR) task. However, the spatial feature extraction of sign language videos in the RGB space tends to focus on the overall image information while neglecting the perception of traits at different granularities, such as eye gaze and lip shape, which are more detailed, or posture and gestures, which are more macroscopic. Exploring the efficient fusion of visual information of different granularities is crucial for accurate sign language recognition. In addition, applying a vanilla Transformer to sequence modeling in cSLR exhibits weak performance because specific video frames could interfere with the attention mechanism. These limitations constrain the capability to understand potential semantic characteristics. We introduce a feature fusion method for integrating visual features of disparate granularities and refine the metric of attention to enhance the Transformer's comprehension of video content. Specifically, we extract CNN feature maps with varying receptive fields and employ a self-attention mechanism to fuse feature maps of different granularities, thereby obtaining multi-scale spatial features of the sign language framework. As for video modeling, we first analyze why the vanilla Transformer failed in cSLR and observe that the magnitude of the feature vectors of video frames could interfere with the distribution of attention weights. Therefore, we utilize the Euclidean distance among vectors to measure the attention weights instead of scaled-dot to enhance dynamic temporal modeling capabilities. Finally, we integrate the two components to construct the model _MSF-ET_ (Multi-Scaled feature Fusion-Euclidean Transformer) for cSLR and train the model end-to-end. We perform experiments on large-scale cSLR benchmarks--PHOENIX-2014 and Chinese Sign Language (CSL)--to validate the effectiveness. continuous sign language recognition; multi-scaled feature fusion; self-attention; Transformer +
Footnote †: journal:  
subtle facial expressions, lip movements, and eye gaze [7; 8]. Thus, the representation of semantic features in sign language videos is multi-scale, making it crucial to comprehensively extract and integrate spatial features of different granularities. On the other hand, the lack of boundary annotations in prevalent datasets, which only provide gloss word annotations, poses a significant challenge to continuous sign language recognition. Therefore, cSLR is regarded as a weakly supervised learning technique [9]. Additionally, aligning extended sign videos with glosses adds to the complexity [10]. Hence, it can be observed that multi-scale perception of spatial features in continuous sign language gestures and temporal modeling of video sequences are crucial for improving recognition accuracy. Currently, the predominant research approach involves extracting image features using CNNs, followed by temporal modeling with LSTM or 1D-CNN to perceive the semantic information in sign language videos [11; 12]. Three-dimensional (3D)-CNNs have also been used to integrate temporal and spatial features for continuous sign language recognition [6; 13; 14; 15; 16]. At the same time, research teams have achieved end-to-end learning for sign language recognition and translation using the Transformer model [17; 18]. Moreover, hidden Markov models and iterative EM algorithms were also applied in earlier studies to enhance sequence modeling capabilities [19; 20]. Although previous research has improved cSLR accuracy, there are still limitations. First, in terms of spatial feature extraction, current studies mainly use convolutional neural networks (CNNs) to convert RGB or depth information into feature maps [10], without considering multi-level and multi-granular details and non-manual semantic information, such as facial expressions and body postures in sign language, which have not been effectively integrated into existing recognition frameworks.