Abstract: The standard approach to mitigate errors made by an automatic speech recognition system is to use confidence scores associated with each predicted word. In the simplest case, these scores are word posterior probabilities whilst more complex schemes utilise bi-directional recurrent neural network (BiRNN) models. A number of upstream and downstream applications, however, rely on confidence scores assigned not only to 1-best hypotheses but to all words found in confusion networks or lattices. These include but are not limited to speaker adaptation, semi-supervised training and information retrieval. Although word posteriors could be used in those applications as confidence scores, they are known to have reliability issues. To make improved confidence scores more generally available, this paper shows how BiRNNs can be extended from 1-best sequences to confusion network and lattice structures. Experiments are conducted using one of the Cambridge University submissions to the IARPA OpenKWS 2016 competition. The results show that confusion network and lattice-based BiRNNs can provide a significant improvement in confidence estimation.

Introduction:  Recent years have seen an increased usage of spoken language technology in applications ranging from speech transcription BIBREF0 to personal assistants BIBREF1 . The quality of these applications heavily depends on the accuracy of the underlying automatic speech recognition (ASR) system yielding 1-best hypotheses and how well ASR errors are mitigated. The standard approach to ASR error mitigation is confidence scores BIBREF2 , BIBREF3 . A low confidence can give a signal to downstream applications about the high uncertainty of the ASR in its prediction and measures can be taken to mitigate the risk of making a wrong decision. However, confidence scores can also be used in upstream applications such as speaker adaptation BIBREF4 and semi-supervised training BIBREF5 , BIBREF6 to reflect uncertainty among multiple possible alternative hypotheses. Downstream applications, such as machine translation and information retrieval, could similarly benefit from using multiple hypotheses. A range of confidence scores has been proposed in the literature BIBREF3 . In the simplest case, confidence scores are posterior probabilities that can be derived using approaches such as confusion networks BIBREF7 , BIBREF8 . These posteriors typically significantly over-estimate confidence BIBREF8 . Therefore, a number of approaches have been proposed to rectify this problem. These range from simple piece-wise linear mappings given by decision trees BIBREF8 to more complex sequence models such as conditional random fields BIBREF9 , and to neural networks BIBREF10 , BIBREF11 , BIBREF12 . Though improvements over posterior probabilities on 1-best hypotheses were reported, the impact of these approaches on all hypotheses available within confusion networks and lattices has not been investigated. Extending confidence estimation to confusion network and lattice structures can be straightforward for some approaches, such as decision trees, and challenging for others, such as recurrent forms of neural networks. The previous work on encoding graph structures into neural networks BIBREF13 has mostly focused on embedding lattices into a fixed dimensional vector representation BIBREF14 , BIBREF15 . This paper examines a particular example of extending a bi-directional recurrent neural network (BiRNN) BIBREF16 to confusion network and lattice structures. This requires specifying how BiRNN states are propagated in the forward and backward directions, how to merge a variable number of BiRNN states, and how target confidence values are assigned to confusion network and lattice arcs. The paper shows that the state propagation in the forward and backward directions has close links to the standard forward-backward algorithm BIBREF17 . This paper proposes several approaches for merging BiRNN states, including an attention mechanism BIBREF18 . Finally, it describes a Levenshtein algorithm for assigning targets to confusion networks and an approximate solution for lattices. Combined these make it possible to assign confidence scores to every word hypothesised by the ASR, not just from a single extracted hypothesis. The rest of this paper is organised as follows. Section "Bi-Directional Recurrent Neural Network" describes the use of bi-directional recurrent neural networks for confidence estimation in 1-best hypotheses. Section "Confusion Network and Lattice Extensions" describes the extension to confusion network and lattice structures. Experimental results are presented in Section "Experiments" . The conclusions drawn from this work are given in Section "Conclusions" .

Question: What is a confusion network or lattice?
Answer: graph-like structures where arcs connect nodes representing multiple hypothesized words, thus allowing multiple incoming arcs unlike 1-best sequences