# Experimenting the Automatic Recognition of Non-Conventionalized Units in Sign Language
## 2 Materials and Methods
### Redefining Continuous Sign Language Recognition
#### 2.2.2 Relevant Metrics
These formula can actually be generalized to non-binary values, with the following definitions:  
\[P^{m}=\frac{\sum_{t}\mathds{1}(d_{t}^{m}=\widehat{d}_{t}^{m}\text{ and }d_{t}^{m}\
eq 0)}{\sum_{t}\mathds{1}(\widehat{d}_{t}^{m}\
eq 0)} \tag{11}\]  
\[R^{m}=\frac{\sum_{t}\mathds{1}(d_{t}^{m}=\widehat{d}_{t}^{m}\text{ and }d_{t}^{m}\
eq 0)}{ \sum_{t}\mathds{1}(d_{t}^{m}\
eq 0)} \tag{12}\]  
The F1-score, defined as the harmonic mean of precision and recall, is then used as a trade-off metric for binary classification:  
\[\text{F1}^{m}=2\left((P^{m})^{-1}+(R^{m})^{-1}\right)^{-1}. \tag{13}\]  
One advantage of F1-score is that the minimum of the two performance values is emphasized. Unit-Wise  
Although accurate temporal localization is aimed for, frame-wise performance metrics may not be perfectly informative. Indeed, because the start and end of each unit can be quite subjective, even a good recognition model can get poor frame-wise Acc, P, R, F1 _etc._, especially if the units are short, like in the case of PTSs (_cf._ Table 4). Unit-level metrics are then needed to get a better perspective on a system performance. Let \(U_{G}\) be the set of all ground-truth annotated units and \(U_{D}\) that of all detected units. The notion of precision and recall for categorical values in a temporal sequence format can then be extended to units. True and false positives and negatives are counted with respect to two points of view: either analyzing each annotated unit--and deciding whether it is sufficiently close to any detected unit (or each detected unit) and deciding whether it is sufficiently close to any annotated unit, i.e., precision matches each unit of the detected list to one of the units in the ground truth list, whereas recall matches each unit of the ground truth to one of the units in the detection list. Modified versions of precision and recall are defined as follows:  
\[P^{\star}=\frac{\text{\# of correctly detected units w.r.t. }U_{D}}{\text{\# of detected units}}=\frac{1}{|U_{D}|}\sum_{u_{d}\in U_{D}}\text{IsCorrectlyPredicted}(u_{d},U_{G}) \tag{14}\]  
\[R^{\star}=\frac{\text{\# of correctly detected units w.r.t. }U_{G}}{\text{\# of annotated units}}=\frac{1}{|U_{G}|}\sum_{u_{g}\in U_{G}}\text{IsCorrectlyPredicted}(u_{g},U_{D}) \tag{15}\]  
where IsCorrectlyPredicted is a counting function (values are 0 or 1). The F1-score is defined as in Equation (13). Then, we propose two ways of counting correct predictions:  
1. _Counting units within a certain temporal window \(t_{w}\): \(P^{\star}_{w}(t_{w})\), \(R^{\star}_{w}(t_{w})\) and \(FI^{\star}_{w}(t_{w})\)_ First, we propose a rather straightforward counting function that consists of positively counting a unit \(u_{d}\in U_{D}\) if and only if there exists a unit of the same class in \(U_{G}\), within a certain margin (_temporal window_) \(t_{w}\)--respectively, a unit \(u_{g}\in U_{G}\) is counted positively if and only if there exists a unit of the same class in \(U_{D}\), within a certain margin \(t_{w}\). In this configuration, precision, recall and F1-score are named \(P_{w}^{\star}(t_{w})\), \(R_{w}^{\star}(t_{w})\) and \(\mathrm{F1}_{w}^{\star}(t_{w})\). 2. _Counting units with thresholds \(\bar{I}_{p}\) and \(\bar{I}_{r}\) on their normalized temporal intersection: \(P_{pr}^{\star}(\bar{I}_{p},\bar{I}_{r})\), \(R_{pr}^{\star}(\bar{I}_{p},\bar{I}_{r})\) and \(\mathrm{F1}_{pr}^{\star}(\bar{I}_{p},\bar{I}_{r})\)._ The authors of [46] proposed and applied a similar but refined set of metrics, adapted for human action recognition and localization, both in space and time. Because our data are only labeled in time, we set aside the space metrics, although they would definitely be useful with adapted annotations. In this setting, \(P_{pr}^{\star}(\bar{I}_{p},\bar{I}_{r})\) and \(R_{pr}^{\star}(\bar{I}_{p},\bar{I}_{r})\) are calculated by finding the _best matching units_.