# Continuous Recognition of Teachers' Hand Signals for Students with Attention Deficits
## 2 Proposed Method
The _Pointing to left_ and the _Pointing to right_ signals should meet in Equation (1) and as indicated in Figure 3. Let \(X\) and \(Y\) represent the horizontal and the vertical coordinates in a video frame, respectively. The coordinates of the wrist and the shoulder are denoted as (\(X_{wrist}\), \(Y_{wrist}\)) and (\(X_{shoulder}\), \(Y_{shoulder}\)), respectively. According to the vertical coordinates of the wrist and the shoulder, \(R\) is defined in Equation (1). \[R=\left\{ \begin{array}{ll}\frac{D}{L},&\text{if }Y_{wrist}\geq Y_{shoulder}\\ \frac{D_{x}}{L},&\text{otherwise}\end{array} \right. \tag{1}\]  
In Equation (1) and Figure 3, \(L\) represents the Euclidean distance between the center of two shoulder landmarks and the center of two hip landmarks, \(D\) indicates the Euclidean distance of shoulder and wrist landmarks, and \(D_{x}\) denotes the absolute difference between the horizontal coordinates of shoulder and wrist landmarks. The essential criterion for pointing signals is \(R>Tln_{R}\), which is an adjustable threshold and is empirically set to 0.5 in our algorithm. \begin{table}
\begin{tabular}{c c c c c c} \hline
**Number** & **Name** & **Number** & **Name** & **Number** & **Name** \\ \hline
8 & Right ear & 19 & Left index & 30 & Right heel \\
9 & Mouth left & 20 & Right index & 31 & Left foot index \\
10 & Mouth right & 21 & Left thumb & 32 & Right foot index \\ \hline \end{tabular}
\end{table} 
Table 1: _Cont._  
Figure 1: Landmarks of MediaPipe BlazePose model [35]. Figure 2: Examples of the three kinds of hand signals. (**a**) Pointing to left. (**b**) Pointing to right. (**c**) Non-pointing. signal can be triggered. \(Th_{t}\) is empirically set to 1.5 s in our method to avoid the unintended motion of the arms. If the recognized signals are triggered by both hands, the later status is given precedence over the previous one. In other words, if the recognized signals of both hands happen in the same frame, the result of the right hand has higher priority. However, the recognized pointing signal of _Pointing to left_ or _Pointing to right_ has higher priority than _Non-pointing_ in the same frame. Figure 5a,b depict the signal time series of \(\frac{D}{L}\) and \(\frac{D_{t}}{L}\), respectively. In the figures, the horizontal axis represents the frame number, the red broken lines indicate the threshold of \(Th_{R}\) (0.5), and the yellow background shows the selection of \(\frac{D}{L}\) or \(\frac{D_{t}}{L}\) signal according to whether the vertical coordinate of the wrist is greater than or equal to that of the shoulder. Finally, the signals with the yellow background will be decided as pointing signals by the essential criterion of Equation (1). Figure 5c depicts the combined \(\frac{D}{L}\) and \(\frac{D_{t}}{L}\) signals, in which the blue line comes from \(\frac{D}{L}\) and the green line comes from \(\frac{D}{L}\). The purple and the orange backgrounds indicate the durations of the _Pointing to right_ and the _Pointing to left_ signals, respectively, the blue background represents the duration of the _Non-pointing_ signal, and the slashed background is the lasting duration of signal switching. Nine video frames captured in the video sequence are marked as No. 1 to No. 9. These frames can be mapped to the triggered frame numbers of recognized signals in the combined signal diagram accordingly.