# Sign Language Recognition Using Two-Stream Convolutional Neural Networks with Wi-Fi Signals
## 2 Materials and Methods
### Data Preprocessing  
The amplitude and phase can be obtained from the 3D matrix of the raw CSI. Their size is \(200\times 30\times 3\). The amplitude and phase of an antenna can be obtained from Equations (3) and (4):  
\[\left\|H(f_{k})\right\|=\sqrt{\left(Re\right)^{2}+\left(Im\right)^{2}} \tag{3}\]  
\[\left\|H(f_{k})\right\|=\arctan(Im/Re) \tag{4}\]  
Note that we directly get the angular degree value of the phase without unwrapping the phase to eliminate the phase shift like SignFi method [1]. We combined the amplitude and phase of each gesture and reshaped it into a combination matrix with a size of \(200\times 60\times 3\) as the input data of the spatial-stream network. The input of the motion-stream network was the difference (size of \(199\times 60\times 3\)) of the above combination matrix, which was a concatenation of amplitude difference and phase difference. The difference comes from two consecutive instances and describes the changes in amplitude and phase. It indicates the change of the gesture corresponding to the salient area of movement. Then, two types of modality data, namely combination matrix and their difference matrix, were preprocessed by SVD to remove redundant and irrelevant noise.  
Figure 2 shows the combination matrix and difference matrix before and after SVD preprocessing of sign language \"GO\" in the home and laboratory environments. Each picture in Figure 2 represents a 3D matrix. Figure 2a,b,e,f are the combination matrices with a size of \(200\times 60\times 3\), and Figure 2c,d,g,h are the difference matrices of a size of \(199\times 60\times 3\). The Y axis represents the first dimension, and the X axis represents the second dimension. The RGB color is the third dimension representing three antenna signals. On the X axis, the first half (0-29) is the amplitude information and the second half (30-59) is the phase information.  
From Figure 2, we observed: (1) Combination matrices are more colorful than difference matrices. The color channels correspond to the three antenna signals. The richer the color, the greater the diversity of the signal, and the more information it contains. Thus, the combination matrices contain more information than the difference matrices. (2) The same user performs the same gesture, and the difference between the home and laboratory environment results in different CSI data, especially in the combination matrices, as shown in Figure 2a,b,e,f. In other words, the amplitude and phase of CSI are easily affected by the environment. However, the difference matrices are less affected by the environment. (3) We perform SVD preprocessing on the amplitude and phase of CSI data, respectively. In order to strike a balance between data feature integrity and noise elimination, SVD only selects the top 20 of the 30 singular value rankings, namely SVD_20. From Figure 2c,d,g,h, we can know that the matrix signal becomes smoother after performing SVD.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \begin{tabular}{c} **Number** \\ **of Users** \\ \end{tabular} & \begin{tabular}{c} **Data Groups** \\ **Categories** \\ \end{tabular} & \
\begin{tabular}{c} **Number of Gesture** \\ **Gesture Instances** \\ \end{tabular} &
\begin{tabular}{c} **Number of Instances of** \\ **Each Gesture per User** \\ \end{tabular} \\ \hline
1 & Home276 & 276 & 2760 & 10 \\ \hline
1 & Lab276 & 276 & 5520 & 20 \\ \hline
1 & Lab + Home276 & 276 & 8280 & 20 + 10 \\ \hline
5 & Lab150 & 150 & 7500 & 10 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the SignFi dataset.