# Whispered Speech Recognition Based on Audio Data Augmentation and Inverse Filtering
## 5 Materials and Methods
### Automatic Speech Recognition Systems
#### 5.3.2 CNN-Based ASR System  
Even though Convolutional Neural Networks were originally designed for image recognition tasks, recent studies demonstrated promising performance in speech recognition, particularly for isolated and control words [36].  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Name/Value** \\ \hline Feature vector & MFCC \\ Window & Hamming \\ Pre-emphasis & 0.97 \\ Window size/frame shift & 24/8 ms \\ Cepstrum & Magnitude \\ Number of channels & 26 \\ Number of coefficients (static + \(\Delta\) + \(\Delta\Delta\)) & 39 (13 + 13 + 13) \\ Normalization & CMS \\ \hline \hline \end{tabular}
\end{table}
Table 2: Configuration parameters in the feature extraction procedure.  
Figure 6: The topology of HMM models.  
A typical CNN model architecture consists of the following components:  
* Input layer,
* Convolution layer with an activation function,
* Pooling layer, and
* Fully Connected Layer  
The most important parameters for the CNN model are summarized in Table 3.  
In order to normalize input data (set between 0 and 1), the following transformation was applied:  
\[norm(x)=\frac{x-x_{min}}{x_{max}-x_{min}} \tag{6}\]  
This is necessary because deep learning algorithms work more properly when the input data are normalized. Various algorithms are available for training neural networks, and the SGD algorithm and its variants are commonly used for training CNN models [37].  
The speaker-independent mode was analyzed using 90% training and 10% validation split. The remaining speaker was reserved for testing employing a leave-one-speaker-out cross-validation approach. To ensure reliable evaluation of the performance, the experiments were repeated 10 times, and the average accuracy was calculated for each speaker.  
After training both models, the training histories were obtained, in which training and validation losses were plotted against the epochs. Figure 7 shows the training history of the model. As shown, the training process was completed correctly, as the validation loss decreased along with the training loss among the epochs.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Name/Value** \\ \hline Number of convolution layers & 2 \\ Number of filters & 32 and 64 \\ Activation function & ReLU \\ Pooling layer & Max \\ Number of epochs & 10 \\ Optimizer & ADAM \\ Loss function & Categorical cross-entropy \\ Metric & Accuracy \\ \hline \hline \end{tabular}
\end{table}
Table 3: The parameters of the CNN model.  
Figure 7: Training history of model: blue—training loss; orange—validation loss.  
The speech signal is analyzed on a short-time basis, where it is divided into overlapping frames, and a feature vector is computed for each frame. The window size is typically between 20 and 30 ms, with consecutive frames shifted between 10 and 15 ms. Consequently, utterances of varying durations have different numbers of feature vectors. To mitigate this limitation, two approaches are considered:  
1. The window size is proportional to the frame period, \(w_{s}=Kf_{p}\), where \(K\) is a constant for all utterances (Figure 8).
2. The window length is fixed, but \(K\) is dynamically changed.  
Research studies, such as the one presented in [38], have shown better results in the recognition of isolated words using the first approach. The best results in recognizing words from the Whi-Spe database (in the speaker-dependent case) were achieved using 18 windows per utterance [39]. Therefore, in our experiments, segmentation into 18 overlapping windows and K = 3 was employed. Speech parameterization involved using 39 MFCC dynamic feature vectors per frame (13 static + \(\Delta\) + \(\Delta\Delta\)), with cepstral mean subtraction (CMS) applied. Ultimately, each utterance was represented by a vector of 702 coefficients (18 \(\times\) 39). The ASR recognizer was developed using Python software (version 3.9.12), utilizing the Scikit-learn package.