# Supervised Dynamic Correlated Topic Model for Classifying Categorical Time Series
## 2 Supervised Dynamic Correlated Topic Model
### Approximate Inference  
Given the \(d\)th document \(\mathcal{D}_{d}\) and the associated response \(\mathbf{y}_{d}\) for \(d=1,2,\ldots,M\), the posterior distribution of the latent variables \((\mathbf{\delta}_{d,1:T},\mathbf{Z}_{d,1:T})\) is \[p(\delta_{d,1:T},\mathbf{Z}_{d,1:T}|\mathcal{D}_{d},\mathbf{y}_{d},\mathbf{\Phi}, \mathbf{\Sigma},\mathbf{\beta},\mathbf{\eta}) = \frac{p(\delta_{d},\mathbf{Z}_{d},\mathcal{D}_{d},\mathbf{y}_{d}\mid\mathbf{ \Phi},\mathbf{\Sigma},\mathbf{\beta},\mathbf{\eta})}{p(\mathcal{D}_{d},\mathbf{y}_{d}\mid\mathbf{ \Phi},\mathbf{\Sigma},\mathbf{\beta},\mathbf{\eta})}, \tag{3}\]  
for \(d=1,2,\ldots,M\). The numerator term in Equation (3), \(p(\delta_{d},\mathbf{Z}_{d},\mathcal{D}_{d},\mathbf{y}_{d}\mid\mathbf{\Phi},\mathbf{\Sigma}, \mathbf{\beta},\mathbf{\eta})\), corresponds to the joint distribution of the latent variables \((\delta_{d,1:T},\mathbf{Z}_{d,1:T})\) and the observed document-response pair \((\mathcal{D}_{d},\mathbf{y}_{d})\) and is given by  
\[p(\delta_{d},\mathbf{Z}_{d},\mathcal{D}_{d},\mathbf{y}_{d}\mid\mathbf{\Phi}, \mathbf{\Sigma},\mathbf{\beta},\mathbf{\eta}) = \prod_{t=1}^{T}\left(p(\mathbf{w}_{d,t}|\mathbf{Z}_{d,t},\mathbf{\beta})\times p (\mathbf{Z}_{d,t}|\delta_{d,t})\times\right.\] \[\left.p(\delta_{d,t}\mid\delta_{d,t-1},\mathbf{\Phi},\mathbf{\Sigma}) \right)\times p(\mathbf{y}_{d}\mid\mathbf{Z}_{d,1:T},\mathbf{\eta}),\]  
for \(d=1,2,\ldots,M\). This joint distribution of the latent variables and the document-response pair can be factorized due to the sDCTM framework. The denominator term in Equation (3), i.e., \(p(\mathcal{D}_{d},\mathbf{y}_{d}\mid\mathbf{\Phi},\mathbf{\Sigma},\mathbf{\beta},\mathbf{\eta})\), represents the joint distribution of the \(d\)th document-response pair \((\mathcal{D}_{d},\mathbf{y}_{d})\) and is given by  
\[p(\mathcal{D}_{d},\mathbf{y}_{d}\mid\mathbf{\Phi},\mathbf{\Sigma},\mathbf{\beta },\mathbf{\eta}) = \int\Bigg{(}\sum_{\mathbf{Z}_{d,1:T}}\Big{(}\prod_{t=1}^{T}p(\mathbf{w}_{ d,t}|\mathbf{Z}_{d,t},\mathbf{\beta})\times p(\mathbf{Z}_{d,t}|\delta_{d,t})\] \[p(\delta_{d,t}\mid\delta_{d,t-1},\mathbf{\Phi},\mathbf{\Sigma})\Big{)} \times p(\mathbf{y}_{d}\mid\mathbf{Z}_{d,1:T},\mathbf{\eta})\Bigg{)}d\delta_{d,1:T},\]  
for \(d=1,2,\ldots,M\). Computing the posterior distribution in Equation (3) is not analytically tractable. Hence, we use variational methods that consider a simple family of distributions over the latent variables, indexed by free variational parameters. The variational parameters are estimated to minimize the Kullback-Leibler (KL) divergence [23] between the variational distribution and the true posterior distribution. The latent variables in the sDCTM framework include the per-word topic structure \(\delta_{d,t}\) and the per-word topic assignment \(\mathbf{Z}_{d,t}\) for \(t=1,2,\ldots T\) and \(d=1,2,\ldots M\). We define the approximate variational posterior for the \(d\)th document as  
\[q(\delta_{d,1:T},\mathbf{Z}_{d,1:T})=q(\delta_{d,1},\delta_{d,2},\ldots,\delta_{d,T}\mid\delta_{d,1},\delta_{d,2},\ldots,\delta_{d,T},\hat{\sigma}^{2})\times \prod_{t=1}^{T}q(\mathbf{Z}_{d,t}\mid\gamma_{d,t}), \tag{4}\]  
for \(d=1,2,\ldots,M\), where \(\mathbf{Z}_{d,t}\mid\gamma_{d,t}\) is \(Mult(1,\gamma_{d,t})\), and we obtain \(q(\delta_{d,1:T}\mid\delta_{d,1:T})\) using a dynamic model with Gaussian \"variational observations\" \(\{\delta_{d,1}\), \(\delta_{d,2},\ldots,\delta_{d,T}\}\). This dynamic model is defined using a variational Kalman filter, where the variational parameters \(\hat{\delta}_{d,1:T}\) are treated as observations. Using the variational distribution, we form a variational state space model as follows.