# Unsupervised Pre-Training for Voice Activation
## 3 Results and Discussion
### Results  
In this section, we present only the test metrics in order to not clutter the description. For the hyperparameters' choice, see Appendix A. In order to get baseline metrics, we ran experiments on full datasets with both log-Mel filter banks and wav2 vec features. The best results of these runs are presented in Table 3 for the English dataset. We got slightly better results than in [32]. This can be explained by the following reasons:  
* The Google Speech Commands dataset [14] was extended since its publication,
* we used a more extensive hyperparameter search. We made the following conclusions from the results:
* wav2 vec audio features give a competitive result for the voice activation problem with very simple downstream models such as the feedforward neural network, \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **res8-Narrow** & **res8** & **res15-Narrow** & **res15** & **res26-Narrow** & **res26** & **ff** \\ \hline conv/first FC & 171 & 405 & 171 & 405 & 171 & 405 & 10K \\ residual blocks & 19.5K & 109K & 39K & 219K & 78K & 437K & - \\ conv/second FC & - & - & 3K & 18.2K & - & - & 8K \\ softmax & 228 & 540 & 228 & 540 & 228 & 540 & 75K \\
**Total** & 19.K & 110K & 42.6K & 238K & 78.4K & 438K & 93K \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of parameters in the architectures used in the experiments for the case of the Google Speech Commands dataset [14] and the log-Mel filter banks. The names of the residual networks follow [32]. FC stands for the fully-connected layer. conv stands for the convolutional layer. * the profit of unsupervised pre-training vanishes as the model gets more sophisticated and deep. We repeated the same experiments for the Russian dataset and got similar results (Table 4): ff and ResNet8-narrow as the simplest models got better results with wav2 vec audio features. However, the overall best result was still with log-Mel filter banks: 97.22%. The best result of wav2 vec runs was 96.62%, which was worse, but still very competitive. It is worth noting that wav2 vec model was trained on the Librespeech dataset [39], which contains only English audio books. It is promising that using this model, it was possible to get good accuracy both on the Russian and Lithuanian datasets (see Table 5). Moreover, we got better results on the Lithuanian dataset using wav2 vec than using log-Mel filter banks (90.77% vs. 89.23%). Next, we ran experiments with a small amount of training data. In order to do that, we limited the number of training samples per keyword by 3, 5, 7, 10, and 20 for all the datasets. Note that the limit of 20 is effectively the same as using the whole dataset for the Lithuanian language. The size of the validation and test sets remained the same in order to get reliable and comparable results. We used random search with all the models and report the test accuracy of the best runs. The motivation of these experiments goes as follows. First of all, the authors of [22] reported that they got state-of-the-art results in automatic speech recognition with unsupervised pre-training in the case when limited training data were available. Secondly, our first set of experiments showed that wav2 vec audio features are superior when the machine learning model is simple. Simpler models tend to perform better when a dataset is small. Therefore, it might be profitable to use unsupervised pre-trained audio features in this scenario. The results of these experiments are summarized in Table 6. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Architecture** & **Log-Mel Filter Banks** & **wav2 vec** \\ \hline ff & 89.15\% & **94.87\%** \\ res8-narrow & 94.70\% & **95.81\%** \\ res8 & **96.72\%** & 96.26\% \\ res15-narrow & **94.53\%** & 94.24\% \\ res15 & **97.22\%** & 96.03\% \\ res26-narrow & 95.42\% & **96.62\%** \\ res26 & **95.81\%** & 95.03\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Test accuracy on the full Russian dataset.