# Experimenting the Automatic Recognition of Non-Conventionalized Units in Sign Language
## 3 Results and Discussion
### Quantitative Assessment
#### 3.1.4. Signer-Independence and Task-Independence  
Because we are considering both the problem of signer-independence and that of task-independence, four cases are to be analyzed. We only consider tasks 1 to 8, as task 9 of Dicta-Sign-LSF-v2 corresponds to isolated signs. **Signer-dependent and task-dependent (SD-TD):**: we randomly pick 60% of the videos for training, 20% for validation and 20% for testing. Some signers and tasks are shared across the three sets. **Signer-independent and task-dependent (SI-TD):**: we randomly pick 10 signers for training, 3 signers for validation and 3 signers for testing. All tasks are shared across the three sets. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & & & \multicolumn{3}{c}{**Frame-Wise**} & \multicolumn{3}{c}{**Unit-Wise**} \\ \cline{3-10} \multirow{3}{*}{**Body and Face**} & \multicolumn{2}{c}{**Hands**} & \multirow{3}{*}{**Acc**} & \multirow{3}{*}{**F1 (\(\mathbf{r_{W}}\))} & \multirow{3}{*}{**F1\({}_{\mathbf{w}}^{\star}(\mathbf{t_{w}}=\mathbf{12})\)**} & \multirow{3}{*}{**F1\({}_{\mathbf{pr}}^{\star}(\mathbf{0},\mathbf{0})\)**} & \multirow{3}{*}{**F1\({}_{\mathbf{pr}}^{\star}(\mathbf{0},\mathbf{0})\)**} & \multirow{3}{*}{**Ipr**} \\ \cline{3-10}  & & **OP** & & & & & & \\ \hline \multirow{8}{*}{2D} & \multirow{8}{*}{Raw} & \multirow{8}{*}{\(\checkmark\)} & 0.97 & 0.23 (0.23/0.21) & 0.28 (0.24/0.30) & 0.24 (0.19/0.33) & 0.21 \\  & & & \(\checkmark\) & & & Not converged & \multicolumn{1}{c}{\(\checkmark\)} & \multicolumn{1}{c}{\(\checkmark\)} \\ \cline{3-10}  & & & \(\checkmark\) & 0.97 & 0.14 (0.22/0.10) & 0.22 (0.26/0.13) & 0.22 (0.26/0.18) & 0.18 \\ \cline{2-10}  & & & \(\checkmark\) & & & & & \\ \cline{2-10}  & & & \(\checkmark\) & & 0.97 & 0.14 (0.22/0.10) & 0.31 (0.35/0.29) & 0.21 (0.22/0.20) & 0.12 \\  & & & \(\checkmark\) & & 0.97 & 0.30 (0.40/0.24) & 0.46 (0.38/0.59) & **0.45** (0.37/0.59) & 0.29 \\  & & & \(\checkmark\) & & 0.97 & 0.14 (0.17/0.12) & **0.51** (0.41/0.67) & 0.28 (0.21/0.41) & 0.17 \\  & & & \(\checkmark\) & & 0.97 & **0.32** (0.41/0.27) & 0.48 (0.42/0.58) & 0.43 (0.35/0.59) & 0.32 \\ \cline{2-10}  & & & \(\checkmark\) & & & & Not converged & \multicolumn{1}{c}{\(\checkmark\)} & \multicolumn{1}{c}{\(\checkmark\)} \\ \cline{3-10}  & & \(\checkmark\) & & 0.97 & 0.04 (0.10/0.03) & 0.17 (0.24/0.13) & 0.17 (0.24/0.13) & 0.12 \\  & & & \(\checkmark\) & & & & Not converged & \multicolumn{1}{c}{\(\checkmark\)} & \multicolumn{1}{c}{\(\checkmark\)} \\ \cline{3-10}  & & & \(\checkmark\) & & & & Not converged & \multicolumn{1}{c}{\(\checkmark\)} & \multicolumn{1}{c}{\(\checkmark\)} \\ \cline{3-10}  & & & \(\checkmark\) & 0.96 & 0.15 (0.16/0.13) & 0.30 (0.32/0.28) & 0.22 (0.22/0.21) & 0.13 \\ \cline{2-10}  & & & & 0.96 & 0.11 (0.13/0.10) & 0.37 (0.28/0.50) & 0.24 (0.19/0.35) & 0.12 \\  & & & \(\checkmark\) & & 0.96 & 0.27 (0.26/0.29) & 0.48 (0.35/0.72) & 0.44 (0.32/0.67) & 0.31 \\  & & & \(\checkmark\) & & 0.97 & 0.09 (0.13/0.06) & 0.43 (0.42/0.43) & 0.24 (0.23/0.28) & 0.12 \\  & & & \(\checkmark\) & & 0.97 & 0.31 (0.41/0.26) & 0.46 (0.4  
**Signer-dependent and task-independent (SD-TI):**: we randomly pick five tasks for training, two tasks for validation and one task for testing. All signers are shared across the three sets. **Signer-independent and task-independent (SI-TI):**: we randomly pick eight signers for training, four signers for validation and four signers for testing; three tasks for training, three tasks for validation and two tasks for testing. This roughly corresponds to a 55%-27%-18% training-validation-testing split in terms of video count. Notably in this setting, a fraction of the videos has to be left out--videos that correspond to signers in the training set, and tasks in the other sets, _etc._ In the end, it is thus expected that the amount of training data is more likely to be a limiting factor than for the three previously described configurations. The results--averaged out values from seven repeats--are summarized in Table 10, using the same performance metrics as before.