# Semantic Features Based N-Best Rescoring Methods for Automatic Speech Recognition
## 1 Introduction  
Language modeling, which learns the probability distribution of a given word and captures the extent to which a sequence of words can be considered well formed, is an important part of automatic speech recognition (ASR) tasks. It provides a context to distinguish between words and phrases that are pronounced in a similar way. This is because n-gram language models use the assumption that the probability of the \(i\)-th word \(w_{i}\) given the context history of the preceding \(i-1\) words can be approximated by the probability of observing it in the context history of the preceding \(n-1\) words. Generally, n-gram language models are used in the first-pass decoding process in ASR tasks as only a small number of historical words are needed. Along with the 1-best hypotheses, which could be the final outputs of the system, lattices and n-bests hypotheses can be generated by the first pass decoding process. Other language models like neural network language models are usually used to rescore lattices or n-best hypotheses [1; 2]. Here is an example of some of the hypotheses of the utterance named \"446c040q\" of a Wall Street Journal (WSJ) dataset:* 446c040q-1 THERE ARE INDICATIONS THAT SALES ARE SLOWING DOWN but CONSUMER CREIDIT SEARCH UPWARD IN DECEMBER
* 446c040q-2 THERE ARE INDICATIONS THAT SALES ARE SLOWING DOWN but CONSUMER CREIDIT SURGED UPWARD IN DECEMBER
* 446c040q-3 THERE ARE INDICATIONS THAT SALES ARE SLOWING DOWN but CONSUMER CREIDIT SIR <UNK> UPWARD IN DECEMBER  
Even without listening to the original speech, people who can speak English and have common knowledge can easily point out that the second hypothesis is the most possible one as the word \"surge\" fits the topic of the sentence more and only with \"surge\" does the sentence make sense. Inspired by this, we propose to use a score to measure whether a word is abrupt in the context according to semantic information, and use the score to rescore the hypotheses to promote the performance of ASR systems. Most of the work of using semantic features is based on neural network language models. Neural network language models can often offer excellent results, but they have a severe requirement of machine and time for training, especially when the training set or vocabulary set is very large. In addition, neural language models lack flexibility. A small modification on the vocabulary list could lead to retraining the whole network. Thus, in this work, we offer a different option by using the semantic features without neural networks. Our task has some similarity with ASR error detection. ASR error detection focuses on distinguishing whether the words in ASR results are wrong by using both decoder based and non-decoder based features. The methods are usually tested on the classification error rate of classifying words to right and wrong classes. Generally speaking, the 1-best hypothesis (hypothesis with the highest score of the corresponding utterance) is not always the hypothesis which leads to the lowest WERs. Therefore, an effective hypotheses selection can be of great help. Thus, in this work we do not focus on differentiating whether a certain word is right or wrong, but propose a score to evaluate on what degree the word is appropriate in the hypothesis sentence by semantic features. In this work, we use the latent semantic marginal (LSM) [3] and a variation of log bi-linear language model [4] to form a sentence coordination score. In addition to that, we propose a word-sentence topic similarity score and a word-discourse probability score based on the theory of [PERSON]'s text generation hypotheses [5]. All the four scores are designed to be light-weight, i.e., they do not need parameters other than topic models or word embedding models. We use the four scores in the n-best rescoring process and show the effectiveness of each score on improving ASR systems. As shown by the previous example, there are many common words in hypotheses lists.