## 2 Related Work  
### Continuous Sign Language Recognition  
Continuous sign language recognition (cSLR) aims at converting a sign language video into a gloss sequence. The challenge lies in the lack of explicit boundaries in videos for  
Figure 1: Currently, most video spatial representation methods for cSLR extract features by pre-trained CNN backbone networks ((**left**) in the figure). Although the approach can extract high-level semantic information, it lacks perception of details, such as mouth shape and gaze, which are important for understanding sign language. We propose a multi-scale feature fusion method based on self-attention mechanism ((**right**) in figure), which enables more comprehensive extraction of semantic information.  
various glosses, which is a weakly supervised sequence labeling problem [10; 26]. Deep learning advances cSLR by focusing on spatial representation and temporal modeling of videos. TWo-dimensional convolutional neural networks (2D-CNNs) are employed for perceiving the static information in video frames. Several state-of-the-art pre-trained CNNs, such as VGGs [27], ResNets [28], and EfficientNets [29] are introduced in cSLR, which strengthens the recognition and understanding of frame semantics [30; 31; 32]. [PERSON] et al. integrate the 2D-CNN with temporal convolution to learning the spatial-temporal information [3]. Similarly, [PERSON] et al. propose a fully convolutional network (FCN) for cSLR that extracts frame features and models the feature sequence with 1D-CNN [12]. Furthermore, to comprehend movement information, 3D-CNNs are utilized to learn motion characteristics, providing a more comprehensive understanding of dynamic actions in sign language videos [33; 34].  
Sequence modeling in cSLR is to learn the mapping between sign language videos and gloss sequences. The hidden Markov model (HMM) was adopted in earlier research to predict the glosses through state transitions. [PERSON] et al. adopted HMM in combination with CNNs to achieve sign language recognition [4; 19]. With the emergence of recurrent neural networks, improved models such as LSTM and GRU, more suitable for sequence processing, were applied to this task. [PERSON] et al. introduced the bi-directional LSTM to model the sign sequences [3]. In addition, as the input video sequence is considerably longer than the corresponding glosses, the connectionist temporal classification (CTC) [35] approach was adopted for sequence alignment. Recent studies demonstrated that CTC effectively aligns unsegmented data by dynamically adjusting to varying sequence lengths, making it a powerful solution for handling lengthy and complex video sequences [13; 36].  
There exist some endeavors aimed at enhancing the performance. [PERSON] et al. designed a position-aware temporal convolution to capture local consistent semantics within temporally neighboring frames [9]. This method addresses the issue of CNNs being agnostic to similarity or dissimilarity. [PERSON] et al. proposed a spatial-temporal multi-cue (STMC) network to learn implicit visual grammars underlying the collaboration of different visual cues [7]. [PERSON] et al. introduced the iterative training that uses the alignment proposal generated by recognition model as strong supervisory inference to tune the feature extraction module [37]. Additionally, the research applied RGB images and optical flow to capture both appearance and motion information, achieving competitive performance.