## 3 Methodology
### CTC Loss and CTC Decoder  
The continuous sign language videos are unsegmented so that there are not the obvious boundaries between the glosses. Therefore, we introduce the connectionist temporal  
Figure 5: The detail about self-attention with Euclidean distance and local window. We assume that the window size is 5. Therefore, every frame interacts with others by attention mechanism in the window centered on itself.  
classification (CTC) to align the sequences. CTC aims to construct the map from frame sequence to gloss sequence even though these two sequences have different and various lengths, and it is applied for speech recognition first.  
In our model, given a sequence \(\mathbf{S}=\{s_{i}\}_{i=1}^{T}\) generated from video by spatial-temporal modules, CTC endeavors to map it to the sign gloss sequence \(\mathbf{G}=\{g_{j}\}_{j=1}^{L}\). The objective of CTC is to maximize the sum of probabilities of all possible alignment paths between the input and target sequence. We introduce the special token \(<BLANK>\) to model the transitions meticulously and eliminate the ambiguity caused by repeated glosses. In addition, we define the alignment path of the source sequence as \(\pi=\{\pi_{t}\}_{t=1}^{T}\), where label \(\pi_{t}\) denotes one of the gloss vocabulary included blank token. The probability of alignment path \(\pi\) given the input sequence is represented as follows:  
\[p(\pi|\mathbf{S})=\prod_{t=1}^{T}p(\pi_{t}|\mathbf{S})=\prod_{t=1}^{T}y_{t,\pi_ {t}} \tag{10}\]  
We calculate the conditional probability of gloss sequence \(\mathbf{G}\) as the sum of probabilities of all paths which can be mapped to \(\mathbf{G}\):  
\[p(\mathbf{G}|\mathbf{S})=\sum_{\pi\in\mathcal{B}^{-1}(\mathbf{G})}p(\pi| \mathbf{S}) \tag{11}\]  
where \(\mathcal{B}\) denotes the mapping operation that removes all \(<BLANK>\) tokens and repeated words in the alignment path, and \(\mathcal{B}^{-1}(G)=\{\pi|\mathcal{B}(\pi)=G\}\) is the inverse operation of \(\mathcal{B}\). Eventually, CTC loss, which is as the objective function of the model, is defined as follows:  
\[\mathcal{L}_{CTC}=-log\ p(\mathbf{G}|\mathbf{S})=-log[\sum_{\pi\in\mathcal{B} ^{-1}(\mathbf{G})}p(\pi|\mathbf{S})] \tag{12}\]  
The entire training process of MSF-EF is shown in Algorithm 1. During the training stage, we instruct the model by reducing the CTC loss gradually, and the CTC decoder is applied for target sequence prediction and inference. The CTC decoder is utilized to obtain the target sequence with the highest cumulative probability by the beam search algorithm.  
```
0: Sign language videos \(V\in\mathbb{R}^{T\times 3\times 224\times 224}\)
0: Sequence of glosses \(\mathbf{G}\) corresponding to the video \(V\)
1: set \(i=0\);
2:while\(i<max\_epoch\)do
3:for\((V,\mathbf{G})\) in Dataset \(\mathbb{D}\)do
4: Initialize MSF-EF model parameters \(\theta\)
5:\(m_{1},m_{2}=f_{SE}(V)\) //Obtain spatial features;
6:\(m_{1}\leftarrow[<cls>,flatten(m_{1})]\), \(m_{2}\leftarrow[<cls>,flatten(m_{2})]\)
7:\(v_{1}=f_{trans\_enc}(m_{1})\), \(v_{2}=f_{trans\_enc}(m_{2})\) // Multi-granularity features processing;
8:\(c\gets Concat([v_{1},v_{2}])\) // Feature fusion;
9:\(\mathbf{S}=f_{temp}(c)\) // Temporal sequence modeling;
10:\(\mathcal{L}=-logp(\mathbf{G}|\mathbf{S})=-log[\sum_{\pi\in\mathcal{B}^{-1}( \mathbf{G})}p(\pi|\mathbf{S})]\) // CTC loss calculation;
11:\(\theta^{{}^{\prime}}=\theta-\eta\
abla_{\theta}\mathcal{L}\) // Gradient backward;
12:\(\theta\leftarrow\theta^{{}^{\prime}}\) // Parameter update.
13:endfor
14:\(i=i+1\)
15:endwhile
```  
**Algorithm 1** Multi-scaled feature fusion-Euclidean Transformer (MSF-ET) for continuous sign language recognition.