## 4 Model
### Variational Lower Bound  
For a KG represented as \(\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{O})\), the embeddings of KG components can be represented as \(\mathbf{Z}^{\mathcal{E}},\mathbf{Z}^{\mathcal{R}}\) in latent spaces. To embed both entities and relations in the same semantic space, we first define the log-likelihood of \(\mathcal{O}\), notated as the set of triples in KG, as:  
\[\log p(\mathbf{O}) =\log p(\mathbf{H},\mathbf{R},\mathbf{T}) \tag{2}\] \[=\log p(\mathbf{H})+\log p(\mathbf{R})+\log p(\mathbf{T})\]  
where \(\mathbf{O}\in\mathbb{R}^{W\times 3}\) and \(\mathbf{H}\), \(\mathbf{R}\), and \(\mathbf{T}\) are components in \(\mathbf{O}\). The log-likelihood of KG components, represented as \(\log p(\mathbf{H})\), \(\log p(\mathbf{R})\), and \(\log p(\mathbf{T})\), can be derived using the Bayesian algorithm:  
\[\log p(\mathbf{H}) =\log\sum_{i=0}^{D}\left\{p_{\theta}(\mathbf{H}\mid\mathbf{Z}_{i }^{\mathcal{E}})\cdot p_{\theta}(\mathbf{Z}_{i}^{\mathcal{E}})\right\} \tag{3}\] \[=\log\sum_{i=0}^{D}\left\{p_{\theta}(\mathbf{Z}_{i}^{\mathcal{E}} \mid\mathbf{H})\cdot p_{\theta}(\mathbf{H})\right\}\] \[=\log\sum_{i=0}^{D}\left\{p_{\theta}(\mathbf{Z}_{i}^{\mathcal{E}} \mid\mathbf{H})\cdot\frac{p_{\theta}(\mathbf{Z}_{i}^{\mathcal{E}},\mathbf{H}) \cdot q_{\phi}(\mathbf{Z}_{i}^{\mathcal{E}}\mid\mathbf{H})}{q_{\phi}( \mathbf{Z}_{i}^{\mathcal{E}}\mid\mathbf{H})\cdot p_{\theta}(\mathbf{Z}_{i}^{ \mathcal{E}}\mid\mathbf{H})}\right\}\] \[=q_{\phi}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})\cdot\log\frac{q _{\phi}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})}{p_{\theta}(\mathbf{Z}^{ \mathcal{E}}\mid\mathbf{H})}\] \[\qquad+q_{\phi}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})\cdot\log \frac{p_{\theta}(\mathbf{Z}^{\mathcal{E}},\mathbf{H})}{q_{\phi}(\mathbf{Z}^{ \mathcal{E}}\mid\mathbf{H})}\] \[=D_{KL}(q_{\phi}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})\parallel p _{\theta}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H}))+\mathcal{L}(\theta,\phi; \mathcal{E})\] \[\geq\mathcal{L}(\theta,\phi;\mathbf{H})\]  
The conditional probability \(q_{\phi}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})\) is the variational posterior to approximate the true posterior \(p(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})\), where the parameter \(\phi\) is estimated in the inference model. In Equation (3), the second RHS term \(\mathcal{L}(\theta,\phi;\mathcal{E})\) is called the evidence lower bound (ELBO) on the marginal likelihood of the variables \(\mathcal{E}\):  
\[\mathcal{L}(\theta,\phi;\mathbf{H}) =\mathbb{R}_{q_{\phi}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})} \Big{[}-\log q_{\phi}(\mathbf{Z}^{\mathcal{E}}\mid\mathbf{H})+\log p_{\theta}( \mathbf{H},\mathbf{Z}^{\mathcal{E}})\Big{]} \tag{4}\] \[=\log p(\mathbf{H})+\log p(\mathbf{R})+\log p(\mathbf{T})\]0 \[=\log p(\mathbf{H})+\log p(\mathbf{R})+\log p(\mathbf{T})\]1  
where the \(D_{KL}\) term denotes the Kullback-Leibler divergence, a measure of how one probability distribution is different from a second. Respectively, we have:\[ \begin{split}\mathcal{L}(\theta,\phi;\mathbf{R})&=-D_{KL}(q_{ \phi}(\mathbf{Z^{R}}\mid\mathbf{R})\parallel p_{\theta}(\mathbf{Z^{R}}))\\ &\quad+\mathbb{E}_{q_{\phi}(\mathbf{Z^{R}}\mid\mathbf{R})}\Big{[} \log p_{\theta}(\mathbf{R}\mid\mathbf{Z^{R}})\Big{]}\\ \mathcal{L}(\theta,\phi;\mathbf{T})&=-D_{KL}(q_{ \phi}(\mathbf{Z^{E}}\mid\mathbf{T})\parallel p_{\theta}(\mathbf{Z^{E}}))\\ &\quad+\mathbb{E}_{q_{\phi}(\mathbf{Z^{E}}\mid\mathbf{T})}\Big{[} \log p_{\theta}(\mathbf{T}\mid\mathbf{Z^{E}})\Big{]}\end{split}  \tag{5}\]  
Substituting Equations (3)-(5) into Equation (2), the variational lower bound of \(\log\mathcal{O}\) can be represented with the parameters \(\theta\) and \(\phi\):  
\[ \begin{split}\log p(\mathbf{O})&=\log p_{\theta}( \mathbf{H})+\log p_{\theta}(\mathbf{R})+\log p_{\theta}(\mathbf{T})\\ &\geq\mathcal{L}(\theta,\phi;\mathbf{H})+\mathbf{L}(\theta,\phi; \mathcal{R})+\mathcal{L}(\theta,\phi;\mathbf{T})\\ &=\mathcal{L}(\theta,\phi;\mathbf{O})\end{split}  \tag{6}\]  
where  
\[