Abstract: This paper describes a general, scalable, end-to-end framework that uses the generative adversarial network (GAN) objective to enable robust speech recognition. Encoders trained with the proposed approach enjoy improved invariance by learning to map noisy audio to the same embedding space as that of clean audio. Unlike previous methods, the new framework does not rely on domain expertise or simplifying assumptions as are often needed in signal processing, and directly encourages robustness in a data-driven way. We show the new approach improves simulated far-field speech recognition of vanilla sequence-to-sequence models without specialized front-ends or preprocessing.

Introduction: Automatic speech recognition (ASR) is becoming increasingly more integral in our day-to-day lives enabling virtual assistants and smart speakers like Siri, Google Now, Cortana, Amazon Echo, Google Home, Apple HomePod, Microsoft Invoke, Baidu Duer and many more. While recent breakthroughs have tremendously improved ASR performance BIBREF0 , BIBREF1 these models still suffer considerable degradation from reasonable variations in reverberations, ambient noise, accents and Lombard reflexes that humans have little or no issue recognizing. Most of these problems can be mitigated by training the models on a large volume of data that exemplify these effects. However, in the case of non-stationary processes, such as accents, accurate data augmentation is most likely infeasible, and in general, collecting high quality datasets can be expensive and time-consuming. Past robust ASR literature has considered hand-engineered front-ends and data-driven approaches in an attempt to increase the value of relatively parsimonious data with desired effects BIBREF2 , BIBREF3 . While these techniques are quite effective in their respective operating regimes, they do not generalize well to other modalities in practice due to the aforementioned reasons. Namely, it is difficult to model anything beyond reverberation and background noise from the first principles. Existing techniques do not directly induce invariance for ASR or are not scalable. And, due to the sequential nature of speech, alignments are needed to compare two different utterances of the same text. In this work, we employ the generative adversarial network (GAN) framework BIBREF4 to increase the robustness of seq-to-seq models BIBREF5 in a scalable, end-to-end fashion. The encoder component is treated as the generator of GAN and is trained to produce indistinguishable embeddings between noisy and clean audio samples. Because no restricting assumptions are made, this new robust training approach can in theory learn to induce robustness without alignment or complicated inference pipeline and even where augmentation is not possible. We also experiment with encoder distance objective to explicitly restrict the embedding space and demonstrate that achieving invariance at the hidden representation level is a promising direction for robust ASR. The rest of the paper is organized as follows. Related work is documented in Section "RELATED WORK" . Section "ROBUST ASR" defines our notations and details the robust ASR GAN. Section "EXPERIMENTAL SETUP" explains the experimental setup. Section "RESULTS" shows results on the Wall Street Journal (WSJ) dataset with simulated far-field effects. Finishing thoughts are found in Section "CONCLUSION" .

Question: Are there experiments with real data?
Answer: Yes. They show results on the Wall Street Journal Corpus, which consists of recordings of real speech.