Reference-free and Confidence-independent Binary Quality Estimation for Automatic Speech Recognition
Authors: Hamed Zamani, Jose G. C. de Souza, Matteo Negri, Marco Turchi, Daniele Falavigna (University of Tehran, Università di Trento, Fondazione Bruno Kessler, 2015)
Topic: Multimodal

Abstract
We address the problem of assigning binary quality labels to automatically transcribed utterances when neither reference transcripts nor information about the decoding process are accessible. Our quality estimation models are evaluated in a large vocabulary continuous speech recognition setting (the transcription of English TED talks). In this setting, we apply different learning algorithms and strategies and measure performance in two testing conditions characterized by different distributions of “good” and “bad” instances. The positive results of our experiments pave the way towards the use of binary estimators of ASR output quality in a number of application scenarios.

Introduction
Accurate and cost-effective methods to estimate ASR output quality are becoming a critical need for a variety of applications, such as the large vocabulary continuous speech recognition systems used to transcribe audio recordings from different sources (e.g. YouTube videos, TV programs, corporate meetings), or the dialogue systems for human-machine interaction. For obvious efficiency reasons, in some of these application scenarios, ASR output quality cannot be determined by means of standard reference-based methods. Indeed, besides the fact that reference transcripts are not always available, quality indicators should often be computed at run-time to ensure quick response. This motivates research towards alternative “reference-free” solutions. To cope with this problem, word-level confidence estimates have been used in the past either to measure how an ASR system is certain about the quality of its hypotheses (Wessel et al., 1998; Evermann and Woodland, 2000; Mangu, 2000; Xu et al., 2010, inter alia) or to automatically detect ASR errors (Seigel, 2013; Seigel and Woodland, 2014; Tam et al., 2014). The reliance on confidence information and the emphasis on the word/sub-word level mark the major differences between such prior works and our research, which aims to give an objective assessment of ASR output quality: i) at the whole utterance level and ii) without the constraint of having access to the system’s decoding process. This information, in fact, is not always accessible, as in the case of the increasingly large amount of captioned audio/video recordings that can be found on the Web. This advocates for the development of “confidence-independent” quality estimation methods. These problems have been addressed by Negri et al. (2014), who proposed the task of predicting the word error rate (WER) of an automatically transcribed utterance.1 Results indicate that even with a relatively small set of black-box features (i.e. agnostic about systems’ internal decoding strategies), the predictions closely approximate the true WER scores calculated over reference transcripts. Experiments, however, are limited to a regression problem and further developments either disregard its natural extension to binary classification (Jalalvand et al., 2015), or address it without the same exhaustiveness of this work (C. de Souza et al., 2015). The automatic assignment of explicit good/bad labels has several practical applications. For instance, instead of leaving to the user the burden of interpreting scores in the [0, 1] interval, easilyinterpretable binary quality predictions would help in tasks like: i) deciding if an utterance in a dialogue application has been correctly recognized, ii) deciding if an automatic transcription is good enough for the corresponding audio recording or needs manual revision (e.g. in subtitling applications), iii) selecting training data for acoustic modelling based on active learning, and iv) retrieving audio data with a desired quality for subsequent processing in media monitoring applications. To support these applications, we extend ASR quality estimation to the binary classification setting and compare different strategies. All of them significantly outperform the trivial approach based on thresholding predicted regression scores (our first contribution). The best solution, a stacking method that effectively exploits the complementarity of different models, achieves impressive accuracy results (our second contribution).

Conclusion
We investigated the problem of assigning informative and unambiguous binary quality labels (good/bad) to automatically transcribed utterances. Aiming at an application-oriented approach, we developed a reference-free and confidence-independent method, which has been evaluated in different settings. Our experiments on English TED talks’ transcriptions from the IWSLT campaign show that our best stacking models can successfully combine the complementarity of different strategies. With a balanced accuracy ranging from 86.33% to 88.40%, the full-fledged classifiers that combine black-box and glass-box (i.e. confidence-based) features bring the problem close to its solution. With results in the range 75.90%-78.47%, our reference-free and confidence-independent models provide a reliable solution to meet the demand of cost-effective methods to estimate the quality of the output of unknown ASR systems.