Speech Enhancement with a Generative Adversarial Network

Who has not been in a phone call distorted by background noises like traffic or wind? An algorithm able to denoise a distorted speech signal is of interest in many everyday situations. We have implemented a state-of-the-art deep learning algorithm for speech enhancement, a conditional generative adversarial net inspired by Pascual et al. (2017). The algorithm learns a mapping from noisy to clean speech through a two-player game between a generator and a discriminator. This approach is interesting because of two things: it enhances end-to-end and constructs the loss function in an untraditional way. It is hard to capture the quality and intelligibility of a noisy signal with a traditional loss function. Here, the loss function is learned based on competition between the generator and discriminator; the discriminator learns a loss for the generator’s enhancement to be accurate. Initially, the aim of a generative adversarial network (GAN) was to learn to generate samples from a training distribution. The generator receives latent random noise as input and maps to the wanted distribution. The latent noise makes the output of the algorithm stochastic. In the speech enhancement setting, noisy speech is used as a conditional variable in both generator and discriminator the goal to learn an accurate mapping from noisy to clean speech. If the mapping is accurate, it is not of importance whether or not it is stochastic. Inspired by similar approaches in the image-to-image setting, we have compared the enhancement results for a conditional generative adversarial net with and without latent noise. The algorithm was trained with speech signals from 220 different speakers from a Norwegian speech database and 99 different noise signals from two noise corpora with environmental noise recordings. The training files were constructed at speech-to-noise ratios 0, 10 and 15 dB. The test set contains unseen speech and noise signals, combined at the same ratios of SNR, in addition to the unseen ratio 5 dB. Assessment of the performance of the generative adversarial network was evaluated objectively by use of the ITU-T standard Perceptual Evaluation of Speech Quality (PESQ) and the Short-Time Objective Intelligibility (STOI). There have also been some subjective reviews on the enhanced files from the student. The proposed setup without latent noise perform comparable to the original setup with latent noise, but the scores obtained in terms of PESQ and STOI are slightly lower on average. Both implementations achieve improvements in PESQ similar to other implementations that are using a GAN framework for speech enhancement. The STOI scores decline a little after enhancement, but that might be partly because the STOI scores of the noisy test files were high to begin with. In general, the enhanced speech signals have a reduced noise level. Some of the enhanced signals have high-frequency artifacts and a degree of speech distortion. The training progress is unstable. Early stopping could have been implemented to ensure that the final model is the best one of the different versions developed during training. Pascual et al. (2019) seems to have found solutions to both unstable gradients and high-frequency artifacts, but this article was not published before late April and was un-