# Experimenting the Automatic Recognition of Non-Conventionalized Units in Sign Language
## 2 Materials and Methods
### Redefining Continuous Sign Language Recognition  
#### 2.2.1 Formalization  
In order to formalize the general problem of SLR, let:  
* \(X=[f_{1},\dots,f_{T}]\) a SL video sequence of \(T\) frames. * \(\mathcal{R}\) an intermediate representation of \(X\), often called _features_. * \(\mathcal{M}\) a learning and prediction model. * \(Y\) the element(s) of interest from \(X\), that are to be _recognized_. * \(\widehat{Y}\) an estimation of \(Y\). * \(\mathcal{G}=\{g^{(1)},\dots,g^{(\mathrm{G})}\}\) a dictionary of \(G\) lexical sign glosses. The process of SLR can be seen as a function, or model, using \(\mathcal{R}\) and \(\mathcal{M}\) to estimate \(Y\):  
\[X\xrightarrow{\mathcal{R}\mathcal{M}}\widehat{Y} \tag{2}\]  
The performance of such a model is then evaluated through a function \(\mathcal{P}\), which measures the discrepancy between \(Y\) and \(\widehat{Y}\), the objective being of course that \(\widehat{Y}\) is as close as possible to \(Y\):  
\[\mathcal{P}(Y,\widehat{Y}). \tag{3}\]  
Obviously, the performance is always evaluated on videos unseen during training of both \(\mathcal{R}\) and \(\mathcal{M}\). A crucial setting is the choice of signer-dependency: a signer-independent setting, in which tested signers are excluded from training videos, which makes learning a much harder task than a signer-dependent training, but also drastically increases the generalizability of the trained model. The different categories of SLR rely on the form and content of \(X\) and \(Y\). Within each category, different options can be considered for \(\mathcal{R}\), \(\mathcal{M}\) and \(\mathcal{P}\). It is important to note that \(\mathcal{R}\) and \(\mathcal{M}\), which is the representation of data and the learning-prediction model, are usually chosen in conjunction. Some learning architectures are indeed better adapted to some representations than others. Also, \(\mathcal{R}\) and \(\mathcal{M}\) are sometimes one and the same, for instance in the case of CNNs. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multirow{2}{*}{**FLS**} & \multicolumn{3}{c}{**PLS**} & \multicolumn{3}{c}{**NLS**} \\ \cline{3-8}  & & **DS** & **PTS** & **FBuoy** & **NS** & **FS** \\ \hline Non blank frames & 205530 & 60794 & 23045 & 14359 & 3830 & 1941 \\ \% & 66.4\% & 19.7\% & 7.5\% & 4.6\% & 1.2\% & 0.6\% \\ Cumulative \% & 66.4\% & 86.1\% & 93.6\% & 98.2\% & 99.4\% & 100.0\% \\ \hline Manual units & 24565 & 3606 & 3651 & 589 & 155 & 118 \\ \% & 75.2\% & 11.0\% & 11.2\% & 1.7\% & 0.5\% & 0.4\% \\ Cumulative \% & 75.2\% & 86.2\% & 97.4\% & 99.1\% & 99.6\% & 100.0\% \\ \hline Avg. frames/unit & 8.4 & 16.8 & 6.3 & 24.4 & 24.7 & 16.4 \\ Avg. duration (ms) & 335 & 674 & 252 & 975 & 988 & 658 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Frame count and sign count (manual unit) statistics for the main annotation categories of Dicta-Sign–LSF–v2. Case of CLexSR  
The common acceptation of CSLR, which we refer to as CLexSR, corresponds to the recognition of the lexical sign glosses within the input video sequence \(X\). Let us assume that \(X\) contains \(N\) consecutive lexical signs (\(N\geq 1\)). We assume \(\widehat{N}\) lexical signs are recognized, such that:  
\[\left\{ \begin{array}{rcl}Y_{\text{CLexSR}}&=&\left[g_{1}\quad\cdots\quad g_{N }\right],g_{i}\in\mathcal{G}\\ \widehat{Y}_{\text{CLexSR}}&=&\left[\widehat{g}_{1}\quad\cdots\quad\widehat{g} _{\widehat{N}}\right],\widehat{g}_{i}\in\mathcal{G}.\end{array} \right. \tag{4}\]  
Note than in general, \(N\
eq\widehat{N}\), so \(Y_{\text{CLexSR}}\) and \(\widehat{Y}_{\text{CLexSR}}\) have different lengths. Then, the usual sequence-wise recognition performance \(\mathcal{P}_{\text{CLexSR}}\) is usually defined as the WER, also referred to as Levenshtein Distance, applied to the expected sequences of lexical sign glosses (_cf._ Equation (1)). Our Proposed Approach of General CSLR  
Generally speaking, we propose interpreting CSLR as the continuous recognition of several linguistic descriptors.