# Sign Language Recognition Using Two-Stream Convolutional Neural Networks with Wi-Fi Signals
## 3 Experimental Results
### SignFi Dataset Evaluation  
We quantify the performance of dual-output two-stream network on the benchmark SignFi dataset. The first evaluation is to show the effect of the attention mechanism. We use ResNet50 in our model with and without attention mechanism, respectively. The combination modality of SVD preprocessing on all data groups is used as input. The evaluation results are shown in Table 2. We can observe that the attention mechanism can indeed improve accuracy. The next evaluation is to test the Home276 data group. We use RenNet18 and ResNet50 models in our two stream CNNs to understand which is more suitable for CSI data in sign language recognition tasks. The results in Table 3 show that, compared with other methods and single stream CNNs, the dual-output dual-stream network with ResNet50 and SVD obtains competitive results. In Table 3 we can also know that SVD preprocessing and difference matrices also improves the accuracy. The evaluation of Lab276 data group is shown in Table 4. SVD still improves the accuracy of our dual-output two-stream network. However, the best recognition accuracy (96.79%) of our method is lower than SignFi method (98.91%) and HOS-Re method (98.26%). The laboratory environment is likely to be more complicated than the home environment. The SignFi method uses the unwrapping transform to preprocess the CSI phase data, while the HOS-Re method extracts the third-order cumulant feature that can reduce signal noise. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Data Groups** & **Accuracy without Attention** & **Accuracy with Attention** \\ \hline Home276 & 99.06\% & 99.13\% \\ \hline Lab276 & 96.52\% & 96.79\% \\ \hline Lab + Home276 & 97.04\% & 97.08\% \\ \hline Lab150 & 95.81\% & 95.88\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation of attention mechanism. In order to verify the generalization of our proposed network, we also mixed the Home276 group and Lab276 group together. Examples of the mixed group are randomly divided into training data and test data at a ratio of 8:2. The accuracies reported in Table 5 clearly show that our proposed method with ResNet50 (97.08%) is superior to other methods (94.81%, 96.34%) even without SVD preprocessing. SVD still improves performance a bit. Table 6 shows the comparison results of Lab150 data group. In our proposed method, the accuracy of using SVD preprocessing is also higher than that of not using SVD preprocessing. However, our best result has an accuracy of 95.88%, which is lower than the HOS-Re method (96.23%), but higher than the SignFi method (86.66%). The accuracy difference between our method and the HOS-Re method is less than 1%. Therefore, our method has similar performance to HOS-Re and is significantly better than SignFi method in this evaluation. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Network Model** & **Modality** & **Preprocessing** & **Accuracy** \\ \hline SignFi [1] (9-layer CNN) & combination & no & 95.72\% \\ \hline SignFi [1] (9-layer CNN) & combination & Phase Shift & 98.01\% \\ \hline HOS-Re [3] (SVM) & combination & no & 97.84\% \\ \hline Single-stream ResNet18 & combination & no & 95.99\% \\ \hline Single-stream ResNet18 & combination & SVD\_20 & 95.54\% \\ \hline Single-stream ResNet50 & combination & no & 96.47\% \\ \hline Single-stream ResNet50 & combination & SVD\_20 & 96.30\% \\ \hline Dual-output two-stream with ResNet18 & combination + difference & no & 96.32\% \\ \hline Dual-output two-stream with ResNet18 & combination + difference & SVD\_20 & 96.54\% \\ \hline Dual-output two-stream with ResNet50 & combination + difference & no & 96.63\% \\ \hline Dual-output two-stream with ResNet50 & combination + difference & SVD\_20 & 96.79\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparisons of dual-output two-stream network with other methods on Lab276 group.