## 4 Model
### Variational Lower Bound
\begin{split}\mathcal{L}(\theta,\phi;\mathbf{O})=&-D _{KL}(q_{\phi}(\mathbf{Z^{\mathcal{E}}}\mid\mathbf{H})\parallel p_{\theta}( \mathbf{Z^{\mathcal{E}}}))\\ &+\mathbb{E}_{q_{\phi}(\mathbf{Z^{E}}\mid\mathbf{H})}\Big{[}\log p _{\theta}(\mathbf{H}\mid\mathbf{Z^{\mathcal{E}}})\Big{]}\\ &-D_{KL}(q_{\phi}(\mathbf{Z^{\mathcal{R}}}\mid\mathbf{R})\parallel p _{\theta}(\mathbf{Z^{\mathcal{R}}}))\\ &+\mathbb{E}_{q_{\phi}(\mathbf{Z^{R}}\mid\mathbf{R})}\Big{[}\log p _{\theta}(\mathbf{R}\mid\mathbf{Z^{\mathcal{R}}})\Big{]}\\ &-D_{KL}(q_{\phi}(\mathbf{Z^{\mathcal{E}}}\mid\mathbf{T})\parallel p _{\theta}(\mathbf{Z^{\mathcal{E}}}))\\ &+\mathbb{E}_{q_{\phi}(\mathbf{Z^{\mathcal{E}}}\mid\mathbf{T})} \Big{[}\log p_{\theta}(\mathbf{T}\mid\mathbf{Z^{\mathcal{E}}})\Big{]}\end{split}  \tag{7}\]  
In Equation (7), the conditional probabilities \(q(\mathbf{Z^{\mathcal{E}}}\mid\mathbf{H})\), \(q(\mathbf{Z^{\mathcal{R}}}\mid\mathbf{R})\) and \(q(\mathbf{Z^{\mathcal{E}}}\mid\mathbf{T})\) can be regarded as probabilistic encoders to embed real data into latent space. Similarly, the conditional probabilities \(p(\mathbf{H}\mid\mathbf{Z^{\mathcal{E}}})\), \(p(\mathbf{R}\mid\mathbf{Z^{\mathcal{R}}})\) and \(p(\mathbf{T}\mid\mathbf{Z^{\mathcal{E}}})\) can be regarded as probabilistic decoders, producing corresponding data from latent vector representations. To approximate the real distributions of KG components, we assume that the prior distributions and the variational posterior distributions are Gaussian distributions. \[ \begin{split} p(\mathbf{Z_{i}}^{\mathcal{E}})&= \mathcal{N}(\mathbf{0},\mathbf{I})\\ p(\mathbf{Z_{j}}^{\mathcal{R}})&=\mathcal{N}( \mathbf{0},\mathbf{I})\\ q_{\phi}(\mathbf{Z_{h}}^{\mathcal{E}}\mid\mathbf{H})& =\mathcal{N}(\overline{\mathbf{E}},\sigma_{\overline{\mathbf{c}}}^{2} \cdot\mathbf{I})\\ q_{\phi}(\mathbf{Z_{r}}^{\mathcal{R}}\mid\mathbf{R})& =\mathcal{N}(\overline{\mathbf{R}},\sigma_{\overline{\mathbf{c}}}^{2} \cdot\mathbf{I})\\ q_{\phi}(\mathbf{Z_{t}}^{\mathcal{E}}\mid\mathbf{T})& =\mathcal{N}(\overline{\mathbf{E}},\sigma_{\overline{\mathbf{c}}}^{2}\cdot \mathbf{I})\end{split}  \tag{8}\]  
Assuming priors and variational posteriors to be Gaussian distributions, the \(D_{KL}\) terms in Equation (7) can be formed computationally. In addition, we adopt the Monte Carlo gradient estimator to deal with the \(\mathbb{E}_{q_{\phi}}\) terms:  
\[ \begin{split}\mathcal{L}(\theta,\phi;\mathbf{O})&= \frac{1}{L}\sum_{i=0}^{L}\sum_{(h_{i},r_{i};t_{i})\in\mathcal{O}}^{W}(\log p_{ \theta}(t_{i}\mid\mathbf{Z_{t_{i}}^{\mathcal{E}})}\\ &+\log p_{\theta}(h_{i}\mid\mathbf{Z_{t_{i}}^{\mathcal{E}})}+\log p _{\theta}(r_{i}\mid\mathbf{Z_{r_{i}}^{\mathcal{R}}}))\\ &+\frac{1}{M}\sum_{e_{i}\in\mathcal{E}}^{M}\sum_{d=0}^{D}(\mu_{e_{ i},d}^{2}+\sigma_{e_{i},d}^{2}-\log\sigma_{e_{i},d}^{2}-1)\\ &+\frac{1}{N}\sum_{r_{i}\in\mathcal{R}}^{N}\sum_{d=0}^{D}(\mu_{r_{ i},d}^{2}+\sigma_{r_{i},d}^{2}-\log\sigma_{r_{i},d}^{2}-1),\end{split}  \tag{9}\]where \(D\) is the output dimension of latent variables, \(L\) is the sampling number in the Monte Carlo estimator, and \(M\), \(N\), and \(W\) are the number of entities, relations, and triples. We also adopt the reparameterization trick mentioned in the VAE section to generate samples. \[ \begin{split}&\mathbf{z}_{h_{i}}^{\mathcal{E}}=\overline{\mathbf{h_{i}}}+ \sigma_{h_{i}}^{2}\odot\epsilon,\text{ with }h_{i}\in\mathbf{H},\epsilon\sim\mathcal{N}( \mathbf{0},\mathbf{I})\\ &\mathbf{z}_{r_{i}}^{\mathcal{R}}=\overline{\mathbf{r_{i}}}+\sigma_{ r_{i}}^{2}\odot\epsilon,\text{ with }r_{i}\in\mathcal{R},\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ &\mathbf{z}_{i}^{\mathcal{E}}=\overline{\mathbf{t_{i}}}+\sigma_{t_{i }}^{2}\odot\epsilon,\text{ with }t_{i}\in\mathbf{T},\epsilon\sim\mathcal{N}( \mathbf{0},\mathbf{I})\end{split}  \tag{10}\]