# Semantic Features Based N-Best Rescoring Methods for Automatic Speech Recognition
## 3 Topic Model and Word Embeddings
### Word Embedding  
Word embeddings are distributed representations of words in a vector space, which help learning algorithms to achieve better performance in natural language processing tasks by grouping similar words. Many different approaches have been investigated to generate the word embeddings of word given some training text. In this paper, we use the widely used GloVe algorithm [20] to generate word embeddings. GloVe is based on the co-occurrences of words in a window. Let X be the word-word co-occurrence matrix, of which entry \(x_{i,j}\) denote the count of \(i\)-th and \(j\)-th word co-occurrence in a window. Let \(x_{i}=\sum_{j}x_{i,j}\) denote the number of times any word appears in the context of the \(i\)-th word. The GloVe can be regarded as a weighted least square regression with the cost function as follows:  
\[J=\sum_{i,j=1}^{V}f(X_{i,j})(w_{i}^{\mathrm{T}}\bar{w}_{j}+b_{i}+\bar{b}_{j}- logX_{i,j})^{2}, \tag{1}\]  
where \(V\) is the size of vocabulary, \(w\in\mathbb{R}^{d}\) and \(\bar{w}\in\mathbb{R}^{d}\) are word vectors, \(b\) and \(\bar{b}\) are biases. As \(X\) is symmetric, \(W\) and \(\bar{W}\) are equivalent and differ only as a result of their random initializations. The sum \(W+\bar{W}\) is the final output of the algorithm. The weighted function should have the following properties:  
1. \(f(0)=0\). If f is viewed as a continuous function, it should obey that \(\lim_{x\to 0}f(x)log^{2}x\) is finite. 2. \(f(x)\) should be non-decreasing in case that rare co-occurrences are overweighted. 3. \(f(x)\) should be relatively small for large value of \(x\), in case that frequent co-occurrences are overweighted. In practice, \(f(x)\) are usually chosen as follows:  
\[f(x)= \begin{cases}(x/x_{max})^{\alpha},&x>x_{max}\\ 1,&\text{otherwise}\end{cases}  \tag{2}\]  
Figure 1 shows the model architecture of GloVe. The input is a one-hot representation of a word. The word embedding matrices serve as weight matrices in the model and thus the output of the model is a vector of inner products of word vectors. The embedding matrices are updated by the gradient of the loss function introduced before in this section. The text generation process is modelled as a random walk in the word vector space by [PERSON] et al. [2016]. Given that the last word produced was \(w\), the probability that the next word is \(w^{\prime}\) is assumed to be given by \(h(|v_{w}-v_{w^{\prime}}|^{2})\) for a suitable \(h\). [PERSON] et al. [2016, 2016] further proposed a latent discourse vector in the random walk model, which has a clear semantic interpretation and has been proven useful in subsequent work, e.g., in their work of understanding structure of word embeddings for polysemous words. Figure 1: The model architecture of GloVe. The input is a one-hot representation of a word. The word embedding matrices serve as weight matrices in the model and thus the output of the model is a vector of inner products of word vectors. In their model, the text generation process is driven by the discourse vector \(c_{t}\in\mathbb{R}^{d}\), which is a vector to express what is being talked about. Each word has a continuous representation \(w_{t}\in\mathbb{R}^{d}\) and the probability of a word is emitted at time \(t\) given the discourse vector is modelled with a log-linear word production model by [PERSON] [21]  
\[P[w\text{ emitted at time }t|c_{t}]\propto exp(c_{t}^{T}w_{t}). \tag{3}\]  
The discourse vector \(c_{t}\) does a slow random walk, and \(c_{t+1}\) can be obtained by adding a small random vector from \(c_{t}\). Therefore, nearby words are generated under similar discourse. With the assumption that the discourse vector \(c_{t}\) does not change much when modelling the word emitting procedure, it can be simplified to replace all the \(c_{t}\)'s in the sentence \(s\) with a sentence discourse \(c_{s}\). It was shown in the paper of [PERSON] et al. [5,21] that the maximum a posteriori estimation of \(c_{s}\) is the average of word embeddings for all the words in the sentence.