# Unsupervised Pre-Training for Voice Activation
## 3 Results and Discussion
### Model  
We used two types of audio features and two types of neural network architectures in our experiments. We used either log-Mel filter banks or pre-trained audio features from the wav2 vec model [22]. The log-Mel filter banks features are often chosen for building voice activation or speech recognition systems [1,32]. We used the kaldi[33] implementation of feature computation with the following parameters: frame width--25 ms, frame shift--10 ms, number of bins--80. Thus, we got a \(98\times 80\) feature matrix by computing log-Mel filter banks on one second samples from the datasets. The method torchaudio.compliance.kaldi.fbank can be used in PyTorch [34] to reproduce this computation. In the case of wav2 vec audio features, we uses the pre-trained model from [[https://github.com/pytorch/fairseq/blob/master/examples/wav2](https://github.com/pytorch/fairseq/blob/master/examples/wav2) vec/README.md#wav2 vec]([https://github.com/pytorch/fairseq/blob/master/examples/wav2](https://github.com/pytorch/fairseq/blob/master/examples/wav2) vec/README.md#wav2 vec). We got a \(98\times 512\) feature matrix as an input for the neural network. We used the following neural network architectures: a three-layer fully-connected neural network and residual neural networks (ResNets) as described in [32]. Our fully-connected neural network consisted of the following blocks:  
* fully-connected layer of size 128,
* rectified linear unit (ReLU) as an activation function [35],
* fully-connected layer of size 64,
* ReLU,* flattening of a \(T\times 64\) matrix in a \(64T\) vector, where \(T\) is the number of frames in a sample (98 in all our experiments),
* fully-connected layer of size \(C\), where \(C\) is the number of classes to discriminate,
* softmax layer. This neural network architecture is presented on Figure 1.  
The ResNets that we used in our experiments were based on [36] and repeat the solutions found in [32]. The authors of [36] proposed that it may be easier to learn the residual \(H(x)=F(x)+x\) instead of the true mapping \(F(x)\), since it is empirically difficult to learn the identity mapping for F when the model has an unnecessary depth. In ResNets, residuals are expressed via connections between layers (see Figure 2), where the input of some layer is added to the output of some downstream layer. The architectures that we used from [32] consisted of the following blocks:  
* bias-free \(3\times 3\) convolutional layer,
* optional average pooling layer (e.g., \(4\times 3\) layer in res8),
* several residual blocks consisting of repeated \(3\times 3\) convolutions, ReLUs, and batch normalization layers [37] (see Figure 2b),
* \(3\times 3\) convolutional layer,
* batch normalization layer,
* average pooling layer
* fully-connected layer of size \(C\), where \(C\) is the number of classes to discriminate,
* softmax-layer. All the layers were zero-padded. For some variants, dilated convolutions were applied to increase the receptive field of the model. The parameters of all used ResNet architectures can be seen in Table 1.  
Figure 1: The scheme of the feedforward neural network used in our experiments. The shape of the resulting tensor is specified after each block in square brackets. FC stands for the fully-connected layer. \(T\) is the number of frames in a sample (98 in all our experiments). \(B\) is the number of channels in the audio features (80 for log-Mel filter banks, 512 for wav2 vec features). \(C\) is the number of classes to discriminate. The number of trainable parameters in the architectures used are reported in Table 2. More details about the residual architectures can be found in [32].