# Recognizing Teachers' Hand Gestures for Effective Non-Verbal Interaction
## 3 Proposed Framework
### Hand Gesture Recognition  
In this study, we use CNN to learn gesture patterns. CNN has the characteristics of local connectivity and weight sharing, so that the closer the pixels are, the greater the influence. The calculation of the convolutional layer is shown in Formula (5) [46].  
\[\text{conv}=\sigma(imgMat*W+b) \tag{5}\]  
where \(\sigma\) is the activation function, \(imgMat\) is the matrix of grayscale images, \(*\) is the convolution operation, \(W\) is the convolution kernel, and \(b\) is the bias.  
It is worth noting that in Figure 4, we use two different convolution kernels to perform convolution operations. As a consequence, we obtain two exemplified images. However, in a practical setting, we employ multiple convolution kernels to extract image features for further computation. Additional image presentations of convolution kernel operations are omitted due to space limitation.  
Figure 4: Extracted image features of convolution kernel operations.  
Figure 3: MCT process and results demonstration.  
We use Hu's invariant for gesture shape matching. Hu's invariant has the characteristics of rotation, translation, and scale invariance. The 2D \(p\), \(q\) order statistics of the function \(f(x,y)\) can be defined as Formula (6) [47], where \(p\), \(q\) are non-negative integers.  
\[M_{pq}=\int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}x^{p}y^{q}f(x,y)\mathrm{ d}x\mathrm{d}y \tag{6}\]  
When we consider the centroid point \((x_{\mathrm{c}},y_{\mathrm{c}})\), Equation (6) becomes Equation (7):  
\[\mu_{pq}=\int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}(x-x_{\mathrm{c}})^{p }(y-y_{\mathrm{c}})^{q}f(x,y)\mathrm{d}x\mathrm{d}y \tag{7}\]  
where \(x_{\mathrm{c}}=\frac{M_{10}}{M_{00}}\) and \(y_{\mathrm{c}}=\frac{M_{01}}{M_{00}}\).  
The translation invariance is obtained from the results of the above statistics, and the scale invariance is calculated by Formula (8) [48].  
\[\eta_{pq}=\frac{\mu_{pq}}{\mu_{00}^{\gamma}} \tag{8}\]  
where \(\gamma=\frac{p+q}{2}+1\) is a normalization factor.  
In order to satisfy the input of the deep convolutional neural network, we choose the logarithm of the absolute value of the seven Hu's invariants as the eigenvalue. They can be calculated by Formulas (9)-(15) [49], respectively.  
\[\varphi_{1}=\eta_{20}+\eta_{02} \tag{9}\]  
\[\varphi_{2}=(\eta_{20}-\eta_{02})^{2}+4\eta_{11}^{2} \tag{10}\]  
\[\varphi_{3}=(\eta_{30}-3\eta_{12})^{2}+(\eta_{03}-3\eta_{21}) \tag{11}\]  
\[\varphi_{4}=(\eta_{30}+\eta_{12})^{2}+(\eta_{03}+\eta_{21})^{2} \tag{12}\]  
\[\begin{split}\varphi_{5}=(\eta_{30}-\eta_{12})(\eta_{30}+\eta_{ 12})\Big{[}(\eta_{30}+\eta_{12})^{2}-3(\eta_{21}+\eta_{03})^{2}\Big{]}\\ +(3\eta_{21}-\eta_{03})(\eta_{21}+\eta_{03})\Big{[}3(\eta_{21}+ \eta_{30})^{2}-(\eta_{21}+\eta_{03})^{2}\Big{]}\end{split} \tag{13}\]  
\[\begin{split}\varphi_{6}=(\eta_{20}-\eta_{02})\Big{[}(\eta_{30} +\eta_{12})^{2}-(\eta_{21}+\eta_{03})^{2}\Big{]}+4\eta_{11}(\eta_{30}+\eta_{ 21})(\eta_{21}+\eta_{03})\end{split} \tag{14}\]  
\[\begin{split}\varphi_{7}=(3\eta_{21}-\eta_{03})(\eta_{30}+\eta_{ 12})\Big{[}(\eta_{30}+\eta_{12})^{2}-3(\eta_{03}+\eta_{12})^{2}\Big{]}\\ +(3\eta_{21}-\eta_{03})(\eta_{03}+\eta_{12})\Big{[}3(\eta_{30}+ \eta_{12})^{2}-(\eta_{03}+\eta_{21})^{2}\Big{]}\end{split} \tag{15}\]