# An Approach to Knowledge Base Completion
## 5 Measuring Plausibility by Embedding Committee  
The product of experts (PoE) proposed by [PERSON] et al. (2017) is adopted as a committee machine to combine knowledge graph embeddings. When all experts are probabilistic models, PoE models their overall probability distribution by combining their outputs. While each expert considers a particular aspect of a target task, PoE manages the task comprehensively. As a result, it produces a better distribution than individual experts. In this paper, the plausibility of a triple \(\mathbf{x}\) is represented as its probability, where \(\mathbf{P}(\mathbf{x})\), the probability of \(\mathbf{x}\), is estimated by  
\[\mathbf{P}(\mathbf{x}|\theta_{1},...,\theta_{n})=\frac{\prod_{m}f_{m}( \mathbf{x}|\theta_{m})}{\sum_{\mathbf{x}\in\Lambda}\prod_{m}f_{m}(\mathbf{a}| \theta_{m})}. \tag{1}\]  
Here, \(\theta_{m}\) is the parameter of the \(m\)-th individual embedding, \(f_{m}(\mathbf{x}|\theta_{m})\) is the score of \(\mathbf{x}\) by the \(m\)-th embedding, and \(\mathbf{A}\) is all possible candidate triples. \(\mathbf{P}(\mathbf{x})\) is basically a product of all outputs by individual embeddings. Thus, the more plausible \(\mathbf{x}\) is, the higher \(\mathbf{P}(\mathbf{x})\) it has. Four knowledge graph embeddings of TransE ([PERSON] and [PERSON], 2016), TransR ([PERSON] et al., 2017), DistMult ([PERSON] et al., 2017), and ComplEx ([PERSON] et al., 2017) are adopted as members of a committee to determine the final plausibility of candidate triples. As shown in Table 3, these four embeddings have diverse tendencies for ranking triples. These embeddings output a real value as a score for a triple, but each expert should be a probabilistic model in PoE. Thus, the sigmoid function \(\sigma\) is used to covert the score to a probability. That is, the score functions of the embeddings are  
\[f_{transE} = \sigma(-|\mathbf{h}+\mathbf{r}-\mathbf{t}|_{2}^{2}), \tag{2}\] \[f_{transR} = \sigma(-|\mathbf{M}_{r}\mathbf{h}+\mathbf{r}-\mathbf{M}_{r}\mathbf{ t}|_{2}^{2}),\] (3) \[f_{distMult} = \sigma(\mathbf{h}^{\top}diag(\mathbf{r})\mathbf{t}),\] (4) \[f_{compplex} = \sigma(Re(\mathbf{h}^{\top}diag(\mathbf{r})\mathbf{\bar{t}})). \tag{5}\]  
TransE and TransR have a distance-based score function, which implies that the smaller the score of a triple is, the more plausible the triple is. Thus, in Equation (2) and (3), a negative score is used in the sigmoid. Note that the parameter \(\theta_{m}\) of an embedding in Equation (1) is determined to maximize the performance the embedding without considering other committee members. Therefore, in order to rank the candidate triples optimally by the committee, all \(\theta_{m}\)'s should be fine-tuned by considering neighboring embeddings. To fine-tune the parameters, the negative log-likelihood loss is used, which is defined as  
\[L(\mathbf{x})=-\log\mathbf{P}(\mathbf{x}|\theta_{1},...,\theta_{n}). \tag{6}\]  
The derivative of \(L(\mathbf{x})\) with respect to \(\theta_{m}\) is  
\[\frac{\partial L}{\partial\theta_{m}} = \frac{\partial\log\mathbf{P}(\mathbf{x}|\theta_{1},...,\theta_{n} )}{\partial\theta_{m}} \tag{7}\] \[= \frac{\partial\log f_{m}(\mathbf{x}|\theta_{m})}{\partial\theta_{ m}}-\frac{\partial\log\sum_{\mathbf{e}\mathbf{c}\mathbf{c}}\prod_{m}f_{m}( \mathbf{c}|\theta_{m})}{\partial\theta_{m}}.\] \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **TransE** & **TransR** & **DistMult** & **ComplEx** \\ \hline
**TransE** & - & 0.5904 & 0.4940 & 0.4946 \\
**TransR** & 0.5904 & - & 0.4916 & 0.4926 \\
**DistMulti** & 0.4940 & 0.4916 & - & 0.5317 \\
**ComplEx** & 0.4946 & 0.4926 & 0.5317 & - \\ \hline \hline \end{tabular}
\end{table} 
Table 3: [PERSON]’s rank correlation coefficients among the knowledge graph embeddings. Since the second term of Equation (7) is intractable, it is approximated through a negative sampling [37]. In this paper, negative samples are generated by swapping the head or tail entity of training triples.