# Whispered Speech Recognition Based on Audio Data Augmentation and Inverse Filtering
###### Abstract  
Modern Automatic Speech Recognition (ASR) systems are primarily designed to recognize normal speech. Due to a considerable acoustic mismatch between normal speech and whisper, ASR systems suffer from a significant loss of performance in whisper recognition. Creating large databases of whispered speech is expensive and time-consuming, so research studies explore the synthetic generation using pre-existing normal or whispered speech databases. The impact of standard audio data augmentation techniques on the accuracy of isolated-word recognizers based on Hidden Markov Models (HMM) and Convolutional Neural Networks (CNN) is examined in this research study. Furthermore, the study explores the potential of inverse filtering as an augmentation strategy for producing pseudo-whisper speech. The Whi-Spe speech database, containing recordings in normal and whisper phonation, is utilized for data augmentation, while the internally recorded speech database, developed specifically for this study, is employed for testing purposes. Experimental results demonstrate statistically significant improvement in performance when employing data augmentation strategies and inverse filtering. artificial neural networks; audio databases; automatic speech recognition; convolutional neural network; hidden Markov models; inverse filtering; whispered speech +
Footnote †: journal: Medical  
when faced with speech that differs from the one used in training, typically consisting of normal speech recorded under controlled conditions in a laboratory setting. Recognizing such unconventional speech, including changes in vocal effort, variations in speaker emotional states and accents, and adverse conditions such as background noise, reverberation, and loudness, presents a challenging task for researchers in the field. Speech can be classified into five modes based on the level of effort: whispered, soft, normally phonated (normal), loud, and shouted speech [4]. Whisper is the most unique mode due to the absence of glottal vibrations and noise excitation of the vocal tract. People tend to whisper or lower their voices for various reasons. Firstly, whispering is used in situations where normal speech is not allowed or deemed inappropriate, such as in a theater or reading room. Secondly, it is employed to convey confidential information that uninvolved individuals should not hear. Lastly, whispering is utilized in criminal activities to conceal one's identity. Apart from intentional whispering, it can also occur as a result of health issues (laryngitis or rhinitis) [5]. Whispered speech poses the greatest challenge to the ASR system's performance compared to other speech modes. The notable acoustic mismatch between normal and whispered speech plays a key role in such performance degradation. To improve the robustness, an extensive speech database containing different speech modalities and variability should be developed, especially for modern deep-learning-based ASR systems [6]. However, this procedure is expensive and time-consuming. To address these constraints and enhance the generalization capability of ASR systems, data augmentation (DA) methods have been utilized. A review of the augmentation techniques used in the literature for speech recognition tasks is provided in Table 1.  
The application of data augmentation in audio classification tasks is discussed in [16]. Study [17] demonstrates the synthesis of speech waveforms based on Continuous Wavelet Transform (CWT), which produces natural-sounding synthetic speech. Recent advancements in whispered speech recognition involve pseudo-whispered speech conversion through the following three steps [18]:  
1. Removing the glottal contribution via spectral subtraction;
2. Shifting the first formant using frequency warping;