Abstract: Despite the strong modeling power of neural network acoustic models, speech enhancement has been shown to deliver additional word error rate improvements if multi-channel data is available. However, there has been a longstanding debate whether enhancement should also be carried out on the ASR training data. In an extensive experimental evaluation on the acoustically very challenging CHiME-5 dinner party data we show that: (i) cleaning up the training data can lead to substantial error rate reductions, and (ii) enhancement in training is advisable as long as enhancement in test is at least as strong as in training. This approach stands in contrast and delivers larger gains than the common strategy reported in the literature to augment the training database with additional artificially degraded speech. Together with an acoustic model topology consisting of initial CNN layers followed by factorized TDNN layers we achieve with 41.6 % and 43.2 % WER on the DEV and EVAL test sets, respectively, a new single-system state-of-the-art result on the CHiME-5 data. This is a 8 % relative improvement compared to the best word error rate published so far for a speech recognizer without system combination.

Introduction: Neural networks have outperformed earlier GMM based acoustic models in terms of modeling power and increased robustness to acoustic distortions. Despite that, speech enhancement has been shown to deliver additional WER improvements, if multi-channel data is available. This is due to their ability to exploit spatial information, which is reflected by phase differences of microphone channels in the STFT domain. This information is not accessible by the ASR system, at least not if it operates on the common log mel spectral or cepstral feature sets. Also, dereverberation algorithms have been shown to consistently improve ASR results, since the temporal dispersion of the signal caused by reverberation is difficult to capture by an ASR acoustic model BIBREF0. However, there has been a long debate whether it is advisable to apply speech enhancement on data used for ASR training, because it is generally agreed upon that the recognizer should be exposed to as much acoustic variability as possible during training, as long as this variability matches the test scenario BIBREF1, BIBREF2, BIBREF3. Multi-channel speech enhancement, such as acoustic BF or source separation, would not only reduce the acoustic variability, it would also result in a reduction of the amount of training data by a factor of $M$, where $M$ is the number of microphones BIBREF4. Previous studies have shown the benefit of training an ASR on matching enhanced speech BIBREF5, BIBREF6 or on jointly training the enhancement and the acoustic model BIBREF7. Alternatively, the training data is often artificially increased by adding even more degraded speech to it. For instance, Ko et al. BIBREF8 found that adding simulated reverberated speech improves accuracy significantly on several large vocabulary tasks. Similarly, Manohar et al. BIBREF9 improved the WER of the baseline CHiME-5 system by relative 5.5% by augmenting the training data with approx. 160hrs of simulated reverberated speech. However, not only can the generation of new training data be costly and time consuming, the training process itself is also prolonged if the amount of data is increased. In this contribution we advocate for the opposite approach. Although we still believe in the argument that ASR training should see sufficient variability, instead of adding degraded speech to the training data, we clean up the training data. We make, however, sure that the remaining acoustic variability is at least as large as on the test data. By applying a beamformer to the multi-channel input, we even reduce the amount of training data significantly. Consequently, this leads to cheaper and faster acoustic model training. We perform experiments using data from the CHiME-5 challenge which focuses on distant multi-microphone conversational ASR in real home environments BIBREF10. The CHiME-5 data is heavily degraded by reverberation and overlapped speech. As much as 23% of the time more than one speaker is active at the same time BIBREF11. The challenge's baseline system poor performance (about 80% WER) is an indication that ASR training did not work well. Recently, GSS enhancement on the test data was shown to significantly improve the performance of an acoustic model, which had been trained with a large amount of unprocessed and simulated noisy data BIBREF12. GSS is a spatial mixture model based blind source separation approach which exploits the annotation given in the CHiME-5 database for initialization and, in this way, avoids the frequency permutation problem BIBREF13. We conjectured that cleaning up the training data would enable a more effective acoustic model training for the CHiME-5 scenario. We have therefore experimented with enhancement algorithms of various strengths, from relatively simple beamforming over single-array GSS to a quite sophisticated multi-array GSS approach, and tested all combinations of training and test data enhancement methods. Furthermore, compared to the initial GSS approach in BIBREF13, we describe here some modifications, which led to improved performance. We also propose an improved neural acoustic modeling structure compared to the CHiME-5 baseline system described in BIBREF9. It consists of initial CNN layers followed by TDNN-F layers, instead of a homogeneous TDNN-F architecture. Using a single acoustic model trained with 308hrs of training data, which resulted after applying multi-array GSS data cleaning and a three-fold speed perturbation, we achieved a WER of 41.6% on the development (DEV) and 43.2% on the evaluation (EVAL) test set of CHiME-5, if the test data is also enhanced with multi-array GSS. This compares very favorably with the recently published top-line in BIBREF12, where the single-system best result, i.e., the WER without system combination, was 45.1% and 47.3% on DEV and EVAL, respectively, using an augmented training data set of 4500hrs total. The rest of this paper is structured as follows. Section SECREF2 describes the CHiME-5 corpus, Section SECREF3 briefly presents the guided source separation enhancement method, Section SECREF4 shows the ASR experiments and the results, followed by a discussion in Section SECREF5. Finally, the paper is concluded in Section SECREF6.

Question: How much is error rate reduced by cleaning up training data?
Answer: In case of singe model the WER was better by 10.%  (6.4%) and in case of multi model it was 3.5% ( 4.1%)