# Knowledge Interpolated Conditional Variational Auto-Encoder for Knowledge Grounded Dialogues
## 4 Knowledge Interpolated Conditional Variational Auto-Encoder (KIV)
### Response Decoder  
Upon obtaining \(\mathbf{z}_{c}\) and \(\mathbf{z}_{k}\), an interpolation network produces the interpolated latent variable \(\mathbf{z}\) as described in Equation (2). The decoder, a GRU-based recurrent neural network \(f^{dec}\), maintains a hidden state \(\mathbf{h}^{dec}_{t}\) at each step. The initial hidden state \(\mathbf{h}^{dec}_{0}\) is configured by the latent variable \(\mathbf{z}\) through a MLP: \(\mathbf{h}^{dec}_{0}=\text{MLP}(\mathbf{z})\). To fully harness the context \(\mathbf{c}\), knowledge \(\mathbf{k}\), and latent information inherent in \(\mathbf{z}\), we introduce a mixture-of-decoders mechanism to output the probability of a response. This mechanism draws inspiration from  
Figure 2: The detailed architecture of our proposed Knowledge Interpolated Conditional Variational Auto-Encoder. Two latent variables \(\mathbf{z}_{k},\mathbf{z}_{c}\), corresponding to context and knowledge, are smoothly fused by (**a**) deterministic and (**b**) probabilistic interpolation methods.  
for its flexibility in approximating the Dirichlet distribution, and it can effectively capture correlations between components of probability vectors [50]. Each response is generated by sampling an interpolation weight from this Logistic-Normal distribution:  
\[\begin{split}& p(\lambda|\mu^{\lambda},\sigma^{\lambda})\sim \mathcal{LN}(\mu^{\lambda},\sigma^{\lambda})\\ &=\frac{1}{\sigma^{\lambda}\sqrt{2\pi}}\frac{1}{\lambda(1- \lambda)}\exp\left(-\frac{\left(\text{logit}(\lambda)-\mu^{\lambda}\right)^{2 }}{2(\sigma^{\lambda})^{2}}\right),\end{split} \tag{10}\]  
where \(\mu^{\lambda}\) and \(\sigma^{\lambda}\) are parameters of the Logistic Normal distribution. To parameterize \(p(\lambda|\mu^{\lambda},\sigma^{\lambda})\), we obtain the posterior and priori latent variables by reparameterizing \(\mu^{\lambda},\sigma^{\lambda}\) as follows:  
\[\begin{split}\left[\mu^{\lambda}_{pst},\sigma^{\lambda}_{pst} \right]&=f^{\lambda}_{pst}([\mathbf{c},\mathbf{k},\mathbf{y}]),\\ \left[\mu^{\lambda}_{pri},\sigma^{\lambda}_{pri}\right]& =f^{\lambda}_{pri}([\mathbf{c},\mathbf{k}]),\\ p(\lambda^{pst}|\mathbf{z}_{\mathbf{k}},\mathbf{z}_{\mathbf{c}})& \sim\mathcal{LN}(\mu^{\lambda}_{pst},\sigma^{\lambda}_{pst}),\\ p(\lambda^{pri}|\mathbf{z}_{\mathbf{k}},\mathbf{z}_{\mathbf{c}})& \sim\mathcal{LN}(\mu^{\lambda}_{pri},\sigma^{\lambda}_{pri}). \end{split} \tag{11}\]  
The posterior and prior network \(f^{\lambda}_{pst},f^{\lambda}_{pst}\) are MLP networks with tanh activation functions. To obtain a sample from the reparameterized Logistic-Normal distribution, we first draw a sample from the Normal distribution \(\mathcal{N}(\mu^{\lambda},\sigma^{\lambda})\) and then apply the logistic function to transform the sample into the Logistic-Normal distribution space. The KL-divergence between the posterior and prior weight can be computed using the closed-form formula for the Gaussian distribution [50]. By injecting the probabilistic interpolation weight \(\lambda\), the posterior distribution during training can be factorized as: \(q_{\phi}(\mathbf{z},\lambda|\mathbf{y},\mathbf{c},\mathbf{k})=q_{\phi}(\mathbf{z}|\mathbf{y},\mathbf{c}, \mathbf{k})q_{\phi}(\lambda|\mathbf{y},\mathbf{c},\mathbf{k})\). The prior distribution \(p_{\theta}(\mathbf{z}|\mathbf{c},\mathbf{k})\) can be decomposed in the same way. Injecting the factorized distribution in Equation (3), the final objective can be rewritten as:  
\[\mathcal{L}^{p}_{\text{KIV}}=-\text{KL}(q_{\phi}(\lambda|\mathbf{c},\mathbf{k},\mathbf{y })||p_{\theta}(\lambda|\mathbf{c},\mathbf{k}))+\mathcal{L}_{\text{KIV}}. \tag{12}\]  
The KIV model with the above objective is denoted as \(\text{KIV}_{\text{p}}\), as shown in Figure 1c.