# Voice Activation for Low-Resource Languages
## 4 Discussion
We hope that adding the data set as unknown samples will help the classifier to learn more patterns, but there is no success. Making English commands an additional target helps the resulting classifier, but the results on only Lithuanian commands are horrible. We find out that it is because the Lithuanian part of the data set is small, compared to the Google Speech Commands part, so it is favorable for the classifier to focus on the English part. After that, we scale up the Lithuanian part by simply repeating the samples several times. The resulting metrics are better than the log-mel filterbanks baseline in all models and better than the wav2 vec baseline in all models except res8. The accuracy is not the best metric for assessing the detection quality of a voice activation system because it does not reflect the balance between the two different types of errors [1]:  
* False alarms. * False rejects. Many researchers and practitioners use a false alarm rate (the rate of false detections over the samples without a keyword) and false reject rate (the rate of skipping the detection over the samples with a keyword). Still, the researchers that use the Google Speech Commands data set [38] (and collected with the same principles Lithuanian dataset [24]) use the test accuracy as a final metric (see, for example, [15; 46; 47]) because it is easier to compare in the case of many keywords, and because a relatively big and diverse speech data set is required in order to get a reasonable estimate for a false alarm rate. Still, it is possible to compute the false alarm rate and the false reject rate, even for a multiclass models. The obvious method is to compute these metrics for each keyword, but for our experiments it is not very practical: each keyword has a handful of test samples. Additionally, comparing two models by 15 pairs of false alarm rates and false reject rates is not a trivial task. Because of that, we use the same method as proposed in scikit-learn tutorials [48]: we convert this multiclass problem to a binary classification. For a sample with a target label \(t\) out of \(n\) classes with predicted probabilities \(p_{1},\ldots,p_{n}\) we consider \(n-1\) negative samples with predicted probabilities \(p_{1},\ldots,p_{t-1},p_{t},\ldots,p_{n}\) and one positive sample with a predicted probability \(p_{t}\). After that, we can compute false alarm rate and false reject rate. The values of these metrics for different values of classifying thresholds are presented in Figure 5 for the best models for baseline methods and joint training (Section 3.4.5). As you can see, the model with the highest quality (trained with joint training on both data sets) is almost everywhere better than all the competitors, which supports our findings. It is interesting to see that, though the wav2 vec model is better than the log-mel filterbanks one in terms of accuracy, it is significantly worse in operating points, with a low false alarm rate. The equal error rate (EER, the minimal value that both false alarm rate and false reject rate can accept at the same time for a some value of threshold) also shows that the best log-mel filterbanks model (\(\text{ERR}=5.68\%\)) can be better than a wav2 vec (\(\text{ERR}=5.84\%\)) for some scenarios. The best model from the proposed joint training method has an equal error rate 4.5%.