# Voice Activation for Low-Resource Languages
## 3 Experiments
### Methods
#### 3.4.3 Exemplar-Like Pre-Training  
The kind of pre-training discussed in this subsection is based on the pre-training suggested in [35] for image classification. We propose to adapt this method for a voice activation problem. The authors of [35] proposed to learn (in unsupervised manner) features that are invariant to some predefined transformations in the following way:  
* Select \(n\) \"interesting\" (seed) samples. * Generate \(k-1\) samples from each of the samples using the augmentations (so we have a data set of \(nk\) samples): the source sample and \(k-1\) augmented samples form a surrogate class. * Train a classifier to distinguish \(n\) surrogate classes. The resulting classifier can be fine-tuned to the downstream task. \"Interesting\" samples means that the chosen sample should contain information similar to the samples of the downstream task. For example, the authors of [35] chose patches of images with a considerable gradient so these patches contain objects or parts of objects. Carefully chosen augmentations should ensure that the resulting classifier are invariant for some transformations. For example, in [35] scaling, translation and rotation are used so the classifier does not depend on the position of the classified object, just on the object itself. We use two kinds of augmentations: the common for the Google Speech Commands data set [38] described in Section 3.3 and SpecAugment [44]. The latter consists of a combination of the following transformations (for the detailed explanation see [44]):  
* Time wrapping. * Frequency masking. * Time masking. \begin{table}
\begin{tabular}{c c} \hline \hline
**Architecture** & **Accuracy** \\ \hline res8 & 96.29\% \\ res15 & 95.45\% \\ res26-narrow & 95.93\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Test accuracy for models pre-trained on the Google Speech Commands data set [38]. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Architecture** & **Baseline Accuracy** & **Method Accuracy** \\ \hline res8 & 78.46\% & **89.23\%** \\ res15 & 89.23\% & **90.77\%** \\ res26-narrow & 83.08\% & **89.23\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The test accuracy after the fine-tuning on the target data set. The best result for each architecture is highlighted. The severity of the transformations are controlled via a set of hyperparameters. The values used in our experiments are presented in Table 6.  
The level of SpecAugment transformations is a hyperparameter in all the grid searches in this subsection. We conduct two sets of experiments using this kind of pre-training. First, we run our best baseline classifier (res8 trained on wav2 vec features) on the 20 h of lectures in Lithuanian. More specifically, we apply our classifier to one-second patches of each audio file with 100 ms shift between the patches. After that, we select the top \(K\) patches by the model score for each of the classes. We make sure that no two patches are closer than 10 s apart (so that we do not have too similar samples). Those patches are the seed samples for the first set of experiments. We try several values of \(K\), so the numbers of seed samples are 100, 1000 and 2000. After that, we conduct a grid search for several architectures. The resulting metrics of the pre-training are presented in Table 7 (only models with testing accuracy equal or higher than 90% were left). The weights of these models are used for initialization (except for the last linear layer) for fine-tuning on the target Lithuanian data set. The choice of model is one of the hyperparameters in the grid search. The results are presented in Table 8.  
The second set of experiments is the same as the first, except for the choice of the seed samples. There, we choose to use 200 audio files from the training set of the target data set as a source of samples for the surrogate classes. We hypothesize that pre-training on the samples from the target domain will improve the quality of the resulting classifier.