Learning New Facts From Knowledge Bases With Neural Tensor Networks and
      Semantic Word Vectors
Authors: Danqi Chen, Richard Socher, Christopher Manning, Andrew Y. Ng
ICLR 2013

Abstract: Knowledge bases provide applications with the benefit of easily accessible, systematic relational knowledge but often suffer in practice from their incompleteness and lack of knowledge of new entities and relations. Much work has focused on building or extending them by finding patterns in large unannotated text corpora. In contrast, here we mainly aim to complete a knowledge base by predicting additional true relationships between entities, based on generalizations that can be discerned in the given knowledgebase. We introduce a neural tensor network (NTN) model which predicts new relationship entries that can be added to the database. This model can be improved by initializing entity representations with word vectors learned in an unsupervised fashion from text, and when doing this, existing relations can even be queried for entities that were not present in the database. Our model generalizes and outperforms existing models for this problem, and can classify unseen relationships in WordNet with an accuracy of 82.8%.

arXiv:1301.3618v2  [cs.CL]  16 Mar 2013
Learning New Facts From Knowledge Bases With
Neural T ensor Networks and Semantic W ord V ectors
Danqi Chen, Richard Socher , Christopher D. Manning, Andrew Y . Ng
Computer Science Department, Stanford University, Stanford, CA 94305, USA
{danqi,manning,ang}@stanford.edu, richard@socher.org
Abstract
Knowledge bases provide applications with the beneﬁt of eas ily accessible, sys-
tematic relational knowledge but often suffer in practice from their incompleteness
and lack of knowledge of new entities and relations. Much wor k has focused on
building or extending them by ﬁnding patterns in large unann otated text corpora.
In contrast, here we mainly aim to complete a knowledge base by predicting addi-
tional true relationships between entities, based on generalizations that can be dis-
cerned in the given knowledgebase. We introduce a neural tensor network (NTN)
model which predicts new relationship entries that can be ad ded to the database.
This model can be improved by initializing entity represent ations with word vec-
tors learned in an unsupervised fashion from text, and when d oing this, existing
relations can even be queried for entities that were not pres ent in the database.
Our model generalizes and outperforms existing models for this problem, and can
classify unseen relationships in WordNet with an accuracy of 75.8%.
1 Introduction
Ontologies and knowledge bases such as WordNet [1] or Yago [2 ] are extremely useful resources
for query expansion [3], coreference resolution [4], question answering (Siri), information retrieval
(Google Knowledge Graph), or generally providing inferenc e over structured knowledge to users.
Much work has focused on extending existing knowledge bases[5, 6, 2] using patterns or classiﬁers
applied to large corpora.
We introduce a model that can accurately learn to add additional facts to a database using only that
database. This is achieved by representing each entity (i.e., each object or individual) in the database
by a vector that can capture facts and their certainty about t hat entity. Each relation is deﬁned by
the parameters of a novel neural tensor network which can explicitly relate two entity vectors and is
more powerful than a standard neural network layer.
Furthermore, our model allows us to ask whether even entitie s that were not in the database are
in certain relationships by simply using distributional wo rd vectors. These vectors are learned by a
neural network model [7] using unsupervised text corpora such as Wikipedia. They capture syntactic
and semantic information and allow us to extend the database without any manually designed rules
or additional parsing of other textual resources.
The model outperforms previously introduced related model s such as that of Bordes et al. [8]. We
evaluate on a heldout set of relationships in WordNet. The accuracy for predicting unseen relations
is 75.8%. We also evaluate in terms of ranking. For WordNet, t here are 38,696 different entities
and we use 11 relationship types. On average for each left ent ity there are 100 correct entities in a
speciﬁc relationship. For instance, dog has many hundreds of hyponyms such as puppy, barker or
dachshund. In 20.9% of the relationship triplets, the model ranks the c orrect test entity in the top
100 out of 38,696 possible entities.
1
2 Related W ork
There is a vast amount of work extending knowledge bases usin g external corpora [5, 6, 2], among
many others. In contrast, little work has been done in extens ions based purely on the knowledge
base itself. The work closest to ours is that by Bordes et al. [ 9]. We implement their approach and
compare to it directly. Our model outperforms it by a signiﬁc ant margin in terms of both accuracy
and ranking. Both models can beneﬁt from initialization with unsupervised word vectors.
Another related approach is that by Sutskever et al. [10] who use tensor factorization and Bayesian
clustering for learning relational structures. Instead of clustering the entities in a nonparametric
Bayesian framework we rely purely on learned entity vectors . Their computation of the truth of a
relation can be seen as a special case of our proposed model. Instead of using MCMC for inference,
we use standard backpropagation which is modiﬁed for the Neu ral Tensor Network. Lastly, we do
not require multiple embeddings for each entity. Instead, we consider the subunits (space separated
words) of entity names. This allows more statistical strength to be shared among entities.
Many methods that use knowledge bases as features such as [3, 4] could beneﬁt from a method
that maps the provided information into vector representat ions. We learn to modify unsupervised
word representations via grounding in world knowledge. This essentially allows us to analyze word
embeddings and query them for speciﬁc relations. Furthermo re, the resulting vectors could be used
in other tasks such as NER [7] or relation classiﬁcation in natural language [11].
Lastly, Ranzato et al. [12] introduced a factored 3-way Restricted Boltzmann Machine which is also
parameterized by a tensor.
3 Neural T ensor Networks
In this section we describe the full neural tensor network. We begin by describing the representation
of entities and continue with the model that learns entity relationships.
We compare using both randomly initialized word vectors and pre-trained 100-dimensional word
vectors from the unsupervised model of Collobert and Weston [13, 7]. Using free Wikipedia text,
this model learns word vectors by predicting how likely it is for each word to occur in its context.
The model uses both local context in the window around each wo rd and global document context.
Similar to other local co-occurrence based vector space mod els, the resulting word vectors cap-
ture distributional syntactic and semantic information. F or further details and evaluations of these
embeddings, see [14, 13, 15].
For cases where the entity name has multiple words, we simply average the word vectors.
The Neural Tensor Network (NTN) replaces the standard linea r layer with a bilinear layer that di-
rectly relates the two entity vectors. Lete1, e2 ∈ Rd be the vector representations of the two entities.
We can compute a score of how plausible they are in a certain relationship R by the following NTN-
based function:
g(e1, R, e2) =UT f
(
eT
1 W [1:k]
R e2 + VR
[
e1
e2
]
+ bR
)
, (1)
where f = tanhis a standard nonlinearity. We deﬁne W [1:k] ∈ Rd× d× k as a tensor and the bilinear
tensor product results in a vector h ∈ Rk, where each entry is computed by one slice of the tensor:
hi = eT
1 W [i]e2. (2)
The remaining parameters for relation R are the standard form of a neural network: VR ∈ Rk× 2d
and U ∈ Rk, bR ∈ Rk.
The main advantage of this model is that it can directly relate the two inputs instead of only implicitly
through the nonlinearity. The bilinear model for truth valu es in [10] becomes a special case of this
model with VR = 0, bR = 0, k= 1, f= identity.
In order to train the parameters W, U, V, E, b, we minimize the following contrastive max-margin
objective:
J(W, U, V, E, b) =
N∑
i=1
C∑
c=1
max(0,1 − g(e(i)
1 , R(i), e(i)
2 ) +g(e(i)
1 , R(i), ec)), (3)
2
where N is the number of training triplets and we score the correct re lation triplets higher than a
corrupted one in which one of the entities was replaced with arandom entity. For each correct triplet
we sample C random corrupted entities.
The model is trained by taking gradients with respect to the ﬁ ve sets of parameters and using mini-
batched L-BFGS.
4 Experiments
In our experiments, we follow the data settings of WordNet in [9]. There are a total of 38,696
different entities and 11 relations. We use 112,581 triplet s for training, 2,609 for the development
set and 10,544 for ﬁnal testing.
The WordNet relationships we consider are has instance, type of, member meronym, member
holonym, part of, has part, subordinate instance of, domain region, synset domain region, similar
to, domain topic.
We compare our model with two models in Bordes et al. [9, 8], wh ich have the same goal as ours.
The model of [9] has the following scoring function:
g(e1, R, e2) =∥WR,lefte1 − WR,righte2∥1, (4)
where WR,left, WR,right ∈ Rd× d. The model of [8] also maps each relation type to an embedding
eR ∈ Rd and scores the relationships by:
g(e1, R, e2) =−(W1e1 ⊗ Wrel,1eR + b1) · (W2e2 ⊗ Wrel,2eR + b2), (5)
where W1, Wrel,1, W2, Wrel,2 ∈ Rd× d, b1, b2 ∈ Rd× 1. In the comparisons below, we call these
two models the similarity model and the Hadamard model respectively. While our function scores
correct triplets highly, these two models score correct tri plets lower. All models are trained in a
contrastive max-margin objective functions.
Our goal is to predict “correct” relations (e1, R, e2) in the testing data. We can compute a score for
each triplet (e1, R, e2). We can consider either just a classiﬁcation accuracy result as to whether the
relation holds, or look at a ranking of e2, for considering relative conﬁdence in particular relatio ns
holding. We use a different evaluation set from Bordes et al. [9] because it has became apparent to
us and them that there were issues of overlap between their training and testing sets which impacted
the quality and interpretability of their evaluation.
Ranking
For each triplet (e1, R, e2), we compute the score g(e1, R, e) for all other entities in the knowledge
base e ∈ E. We then sort values by decreasing order and report the rank of the correct entity e2.
For WordNet the total number of entities is|E| = 38,696. Some of the questions relating to triplets
are of the form “A is a type of ?” or “A has instance ?” Since thes e have multiple correct answers,
we report the percentage of times that e2 is ranked in the top 100 of the list (recall @ 100). The
higher this number, the more often the speciﬁc correct test entity has likely been correctly estimated.
After cross-validation of the hyperparameters of both mode ls on the development fold, our neural
tensor net obtains a ranking recall score of 20.9% while the similarity model achieves 10.6%, and the
Hadamard model achieves only 7.4%. The best performance of the NTN with random initialization
instead of the semantic vectors drops to 16.9% and the simila rity model and the Hadamard model
only achieve 5.7% and 7.1%.
Classiﬁcation
In this experiment, we ask the model whether any arbitrary triplet of entities and relations is true or
not. With the help of the large vocabulary of semantic word ve ctors, we can query whether certain
WordNet relationships hold or not even for entities that were not originally in WordNet.
We use the development fold to ﬁnd a thresholdTR for each relation such that if f(e1, R, e2) ≥ TR,
the relation (e1, R, e2) holds, otherwise it is considered false. In order to create n egative examples,
3
we randomly switch entities and relations from correct test ing triplets, resulting in a total of 2 ×
10,544 triplets. The ﬁnal accuracy is based on how many of of triplets are classiﬁed correctly.
The Neural Tensor Network achieves an accuracy of 75.8% withsemantically initialized entity vec-
tors and 70.0% with randomly initialized ones. In compariso n, the similarity based model only
achieve 66.7% and 51.6%, the Hadamard model achieve 71.9% and 68.2% with the same setup. All
models improve in performance if entities are represented a s an average of their word vectors but
we will leave experimentation with this setup to future work.
5 Conclusion
We introduced a new model based on Neural Tensor Networks. Unlike previous models for predict-
ing relationships purely using entity representations in k nowledge bases, our model allows direct
interaction of entity vectors via a tensor. This architectu re allows for much better performance in
terms of both ranking correct answers out of tens of thousands of possible ones and predicting unseen
relationships between entities. It enables the extension o f databases even without external textual
resources but can also beneﬁt from unsupervised large corpo ra even without manually designed
extraction rules.
References
[1] G.A. Miller. WordNet: A Lexical Database for English. Communications of the ACM, 1995.
[2] F. M. Suchanek, G. Kasneci, and G. Weikum. Yago: a core of semantic knowledge. In Proceedings of the
16th international conference on W orld Wide W eb, 2007.
[3] J. Graupmann, R. Schenkel, and G. Weikum. The SphereSear ch engine for uniﬁed ranked retrieval of
heterogeneous XML and web documents. In Proceedings of the 31st international conference on V ery
large data bases, VLDB, 2005.
[4] V . Ng and C. Cardie. Improving machine learning approaches to coreference resolution. In ACL, 2002.
[5] R. Snow, D. Jurafsky, and A. Y . Ng. Learning syntactic pat terns for automatic hypernym discovery. In
NIPS, 2005.
[6] A. Fader, S. Soderland, and O. Etzioni. Identifying rela tions for open information extraction. In EMNLP,
2011.
[7] J. Turian, L. Ratinov, and Y . Bengio. Word representatio ns: a simple and general method for semi-
supervised learning. In Proceedings of ACL, pages 384–394, 2010.
[8] A. Bordes, X. Glorot, J. Weston, and Y . Bengio. Joint Lear ning of Words and Meaning Representations
for Open-Text Semantic Parsing. AISTATS, 2012.
[9] A. Bordes, J. Weston, R. Collobert, and Y . Bengio. Learning structured embeddings of knowledge bases.
In AAAI, 2011.
[10] I. Sutskever, R. Salakhutdinov, and J. B. Tenenbaum. Mo delling relational data using Bayesian clustered
tensor factorization. In NIPS, 2009.
[11] R. Socher, B. Huval, C. D. Manning, and A. Y . Ng. Semantic Compositionality Through Recursive
Matrix-Vector Spaces. In EMNLP, 2012.
[12] M. Ranzato and A. Krizhevsky G. E. Hinton. Factored 3-Wa y Restricted Boltzmann Machines For Mod-
eling Natural Images. AISTATS, 2010.
[13] R. Collobert and J. Weston. A uniﬁed architecture for na tural language processing: deep neural networks
with multitask learning. In ICML, 2008.
[14] Y . Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neur al probabilistic language model. J. Mach.
Learn. Res., 3, March 2003.
[15] E. H. Huang, R. Socher, C. D. Manning, and A. Y . Ng. Improv ing Word Representations via Global
Context and Multiple Word Prototypes. In ACL, 2012.
4


Peer review:
[Review]:
- A brief summary of the paper's contributions, in the context of prior work.

This paper proposes a new energy function (or scoring function) for ranking pairs of entities and their relationship type. The energy function is based on a so-called Neural Tensor Network, which essentially introduces a bilinear term in the computation of the hidden layer input activations of a single hidden layer neural network. A favorable comparison with the energy-function proposed in Bordes et al. 2011 is presented.

- An assessment of novelty and quality.

This work follows fairly closely the work of Border et al. 2011, with the main difference being the choice of the energy/scoring function. This is an advantage in terms of the interpretability of the results: this paper clearly demonstrates that the proposed energy function is better, since everything else (the training objective, the evaluation procedure) is the same. This is however a disadvantage in terms of novelty as this makes this work somewhat incremental.

Bordes et al. 2011 also proposed an improved version of their model, using kernel density estimation, which is not used here. However, I suppose that the proposed model in this paper could also be similarly improved.

More importantly, Bordes and collaborators have more recently looked at another type of energy function, in 'Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing' (AISTATS 2012), which also involves bilinear terms and is thus similar (but not the same) as the proposed energy function here. In fact, the Bordes et al. 2012 energy function seems to outperform the 2011 one (without KDE), hence I would argue that the former would have been a better baseline for comparisons.

- A list of pros and cons (reasons to accept/reject).

Pros: Clear demonstration of the superiority of the proposed energy function over that of Bordes et al. 2011.

Cons: No comparison with the more recent energy function of Bordes et al. 2012, which has some similarities to the proposed Neural Tensor Networks.

Since this was submitted to the workshop track, I would be inclined to have this paper accepted still. This is clearly work in progress (the submitted paper is only 4 pages long), and I think this line of work should be encouraged. However, I would suggest the authors also perform a comparison with the scoring function of Bordes et al. 2012 in future work, using their current protocol (which is nicely setup so as to thoroughly compare energy functions).