## 4 Model
### Learning  
To optimize the parameters in Equation (9), we apply two neural networks in VAE: (1) An inference model \(f_{\phi}\) with parameter \(\phi\) to map observation data into latent vector spaces. (2) A generative model \(g_{\theta}\) with parameter \(\theta\) to produce random variables from latent embeddings.  
Inference model \(f_{\phi}\). To encode KG components to Gaussian embeddings, we apply two fully-connected layers to map the entities and relations to the means and log-variances in their resulting Gaussian embeddings. One of the benefits of encoding log-variance instead of variance is that it enables us to avoiding using activation functions, since the variance \(\sigma^{2}\) must be a positive number.  
\[\begin{split}&(\overline{\mathbf{h_{i}}},\log\sigma_{\mathbf{h_{ i}}}^{2})\quad=f_{\phi_{1}}(\mathbf{h_{i}})\\ &(\overline{\mathbf{r_{i}}},\log\sigma_{\mathbf{r_{i}}}^{2})\quad =f_{\phi_{2}}(\mathbf{r_{i}})\\ &(\overline{\mathbf{t_{i}}},\log\sigma_{\mathbf{t_{i}}}^{2})\quad =f_{\phi_{3}}(\mathbf{t_{i}})\end{split} \tag{11}\]  
where \(\phi=[\phi_{1},\phi_{2},\phi_{3}]\) and \(\mu\) and \(\log\sigma^{2}\) are the means and log-variances of learned Gaussian embeddings of KG components:  
\[\begin{split}& q_{\phi}(\mathbf{z}_{\mathbf{h_{i}}}^{\mathcal{E}} \mid\mathbf{h_{i}})\quad=\mathcal{N}(\overline{\mathbf{h_{i}}},\sigma_{ \mathbf{h_{i}}}^{2}\cdot\mathbf{I})\\ & q_{\phi}(\mathbf{z}_{\mathbf{h_{i}}}^{\mathcal{E}}\mid\mathbf{r_{i} })\quad=\mathcal{N}(\overline{\mathbf{r_{i}}},\sigma_{\mathbf{h_{i}}}^{2} \cdot\mathbf{I})\\ & q_{\phi}(\mathbf{z}_{\mathbf{t_{i}}}^{\mathcal{E}}\mid\mathbf{t_{i} })\quad=\mathcal{N}(\overline{\mathbf{t_{i}}},\sigma_{\mathbf{t_{i}}}^{2} \cdot\mathbf{I})\end{split} \tag{12}\]  
We apply the reparameterization trick mentioned in Equation (10) to obtain the deterministic variables \(\mathbf{\mathrm{Z}}_{h}^{\mathcal{E}}\), \(\mathbf{\mathrm{Z}}_{r}^{\mathcal{R}}\), and \(\mathbf{\mathrm{Z}}_{t}^{\mathcal{E}}\), transformed from latent random variables, with a noise term \(\epsilon\) from \(\mathcal{N}(\mathbf{0},\mathbf{I})\), which benefit from gradient propagation between the inference model and the generative model. We compute the loss of the inference model by measuring the KL divergence between those conditional probabilities and \(\mathcal{N}(\mathbf{0},\mathbf{I})\).  
Generative model \(g_{\theta}\). The generative model decodes from deterministic values to random variables. For example, given resulting embeddings \(\mathbf{\mathrm{Z}}^{\mathcal{E}}\) and \(\mathbf{\mathrm{Z}}^{\mathcal{R}}\) from a KG represented as \(\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{O})\), our goal is to reconstruct random variables for each triple \((\mathbf{h_{i}},\mathbf{r_{i}},\mathbf{t_{i}})\in\mathcal{O}\), where:  
\[p_{\theta}(\mathbf{h_{i}},\mathbf{r_{i}},\mathbf{t_{i}}\mid\mathbf{z}_{\mathbf{h_{ i}}}^{\mathcal{E}},\mathbf{\mathrm{Z}}_{t_{i}}^{\mathcal{R}},\mathbf{z}_{\mathbf{t_{i}}}^{ \mathcal{E}})=g_{\theta}(\mathbf{z}_{\mathbf{h_{i}}}^{\mathcal{E}},\mathbf{z}_{\mathbf{ t_{i}}}^{\mathcal{R}},\mathbf{z}_{\mathbf{t_{i}}}^{\mathcal{E}}) \tag{13}\]  
The random distributions of those components can be defined as:  
\[\begin{split}& p_{\theta_{1}}(\mathbf{h_{i}}\mid\mathbf{z}_{\mathbf{h_{ i}}}^{\mathcal{E}})\quad=\mathcal{N}(\overline{\mathbf{z_{h_{i}}}},\sigma_{ \mathbf{z_{h_{i}}}}^{2}\cdot\mathbf{I})\\ & p_{\theta_{2}}(\mathbf{r_{i}}\mid\mathbf{z}_{\mathbf{t_{i}}}^{ \mathcal{R}})\quad=\mathcal{N}(\overline{\mathbf{z_{r_{i}}}},\sigma_{\mathbf{z_{h_{ i}}}}^{2}\cdot\mathbf{I})\\ & p_{\theta_{3}}(\mathbf{t_{i}}\mid\mathbf{z}_{\mathbf{t_{i}}}^{ \mathcal{E}})\quad=\mathcal{N}(\overline{\mathbf{z_{t_{i}}}},\sigma_{\mathbf{z_{h_{ i}}}}^{2}\cdot\mathbf{I})\end{split} \tag{14}\]  
where \(\theta=[\theta_{1},\theta_{2},\theta_{3}]\), and the reconstruction loss of the generative model can be measured based on the binary cross entropy (BCE) between the generative variables and the real data.