# A Nested Chinese Restaurant Topic Model for Short Texts with Document Embeddings
## 3 Model and Inference
### Overview  
Topic modeling methods sample topics through a joint probability distribution constructed by a generation process. We chose the conventional method LDA as an example. The LDA method designs three variables: text \(d\), word \(w\), and topic \(z\). Then model constructs a generation process among these variables. It suggests that for each word \(w\) in each text \(d\), a topic \(z\) is sampled from text \(d\) according to a multi-nominal distribution, and word \(w\) is sampled from the topic \(z\) according to another multi-nominal distribution. Then the joint probability distribution \(P(d,z,w)\) is define by two multi-nominal distributions \(P(z|d)\) and \(P(w|z)\), where \(P(d,z,w)=P(w|z)P(z|d)\). The two variables \(w\) and \(d\) are already known and variable \(z\) is the latent variable. So we can sample \(z\) according to the Gibbs sampling method [31]. To deal with short texts, self-aggregated topic models design a new latent variable \(l\) as the long document. The state-of-the-art model PTM proposes a generation process among four variables: short text \(d\), word \(w\), topic \(z\), and long document \(l\). Firstly, for each short text \(d\), sample a long document \(l\) according to a multi-nominal distribution. Then for each word \(w\) of short text \(d\), sample a topic \(z\) from a long document \(l\) according to a multi-nominal distribution. Finally, sample the word \(w\) from topic \(z\) according to another multi-nominal distribution. So, this generation process constructs a joint probability distribution \(P(d,w,z,l)\) according to three multi-nominal distributions \(P(w|z)\), \(P(z|l)\) and \(P(l|d)\), where \(P(d,w,z,l)=P(w|z)P(z|l)P(l|d)\). Every short text shares a same multinomial distribution \(P(l|d)\). So if the number of long documents is less than the number of short texts, short texts will be aggregated into long documents. \begin{table}
\begin{tabular}{c c} \hline \hline
**Strategy** & **Models** \\ \hline \multirow{2}{*}{Models with auxiliary data} & Incorporating metadata [7; 8; 9; 24; 25; 26] \\  & Incorporating a long document set [10; 11; 12; 13; 14; 29; 30] \\ \hline \multirow{3}{*}{Models without auxiliary data} & Limiting topics correspond to each text [16; 17; 18] \\  & Using global word co-occurrence information [19; 20] \\ \cline{1-1}  & Aggregating short texts automatically [21; 22] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Classification of topic models for short texts. However, the aggregation according to a multi-nominal distribution cannot avoid aggregating semantically irrelevant short texts into one long document. So we propose a nested Chinese restaurant process to incorporate document embeddings instead of using a multi-nominal distribution. As the Chinese restaurant process is a specific sampling method of the Dirichlet process. So firstly, we use the Dirichlet process (DP) to describe the generalized mathematical representation of the generation process of our model. Then we will introduce details of the nested Chinese restaurant process in our model. The generation process is as follows:  
1. Sample \(G_{0}\sim DP(\alpha,H)\)
2. Sample \(G\sim DP(\delta,G_{0})\)
3. For each topic \(z\)  
Sample \(\eta_{z}\sim Dir(\gamma)\)
4. For each latent long document \(l\)  
Sample \(\theta_{l}\sim Dir(\beta)\)
5. For each short text \(i\)  
1. Sample a long document \(l\sim G\)
2. For each word \(w_{i,l}\) in short text \(i_{l}\)  
1. Sample a topic \(z\sim Multi(\theta_{l})\)
2. Sample the word \(w_{i,l}\sim Multi(\eta_{z})\)  
The Dirichlet process is a stochastic process that samples distributions. In our generation process, we first sample a distribution \(G_{0}\) from a Dirichlet process \(DP(\alpha,H)\), where \(\alpha\) is the prior parameter and \(H\) is the prior distribution. Then using \(G_{0}\) as another prior distribution, we sample a distribution \(G\) from a Dirichlet process \(DP(\delta,G_{0})\), where \(\delta\) is the prior parameter. Therefore, the expectation of \(G\) is \(G_{0}\).