Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Neural networks for NLP:
Can structured knowledge help?
Christophe Gravier
Riken AIPTokyo

July 16th, 2019

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

1Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Who am I
Acknowledgement
I am very thankful to the French Embassy for the travel grant I wasoffered !
Associate Professor with habilitation
Université Jean Monnet in Saint-Etienne
Laboratoire Hubert Curien, UMR CNRS 5516
Director of development and innovation at Télécom Saint-Etienne(U. Jean Monnet and affiliated to Institut Mines Télécom)

Research interests (NLP) :Learning representations: improve embeddings, differents embeddings
spaces such as Hamming or Wasserstein.
Downstream tasks: Question answering/generation, chatbots, free texts/ structured taxonomies alignments, text generation.
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

2Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Where I am coming fromChristophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

3Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Where I am coming fromChristophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

4Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Where I am coming fromChristophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

5Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Where I am coming fromChristophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

6Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

People involved in this research
Julien Tissier+ 3rd year PhD student+ Currently at Riken AIP Tokyo+ Co-supervised by Pr. Amaury Habrard, U. Jean Monnet, LaHC

Hady Elsahar+ PhD student (defended on the 5th July)+ Now at Naver Labs Grenoble+ Co-supervised by Pr. Frédérique Laforest, INSA Lyon, Lab. LIRIS
European WDAqua Pierre&Marie Curie project, strong collaborations
with U. of Southampton, especially Lucie-Aimee Kaffee and Pavlos
Vougiouklis (and supervisors Elena Simperl, Jonathon Hare)
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

7

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

General modern NLP pipeline

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

8Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Where are we know ? (GLUE [Wang et al. 2018])

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

9Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

All we need ...

Self-supervised learning from massive unlabelled text corpora
Attention (and residual connections) is all you need (ELMO [Peters
et al. 2018], BERT [Devlin et al. 2018])
And Super GLUE [Wang et al. 2019])

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

10Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

All we need ...

Self-supervised learning from massive unlabelled text corpora
Attention (and residual connections) is all you need (ELMO [Peters
et al. 2018], BERT [Devlin et al. 2018])
And Super GLUE [Wang et al. 2019])
But. . . “Do Supervised Distributional Methods Really Learn Lexical
Inference Relations” [Levy et al. 2015] ?

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

10Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Wait ... (Figure from [Wang et al. 2019])

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

11

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Coincidence ? I think not.
Most tasks present over-human performance ! But ...
- No improvement on Natural Language inference (WNLI) [Levesque,Davis, and Morgenstern 2012]- Subpar for Recognizing Textual Entailment (RTE) [Dagan, Glickman,
and Magnini 2005; Bar-Haim et al. 2006; Giampiccolo et al. 2007;
Bentivogli et al. 2009]
Example from RTE (Does A entails hypothesis B ?):A: Time Warner is the worlds largest media and internet company
B: Time Warner is the worlds largest company
Example from WNLI (Co-reference resolution):
The trophy does not fit in the brown suitcase because it is too
small. What is too small ? A: The trophy, B: The suitcase
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

12Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Some unsolved problems (we are saved)

Some NLP models that are not “solved” [Devlin 2019]
Models minimizing total training cost vs. accuracy on modern hardware
Models that are very parameter efficient (e.g., for mobile)
Models that represent knowledge/context in latent space
Models that represent structured data (e.g., knowledge graph)
Models that jointly represent vision and language

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

13

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

What we are gonna discuss today

How to combine the modern neural based
NLP pipeline with external, structured (orsemi-structured) knowledge and how muchdoes it help ?

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

14

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Table of contents

1 Preambule
2 Semi-structured knowledge for learning representations

Dict2Vec
3 Text generation from structured knowledge

Question Generation from Knowledge Graphs
Generating summaries from Knowledge Graphs
4 Conclusion

Digression: Binarization of word embeddings

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

15

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Plain old dictionaries

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

16

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Word2vec
Slide a window across a corpora
Move closer vectors of words within the same window

Image taken from : https://mubaris.com/posts/word2vec/
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

17

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

word2vec
Limitations– Words within the same window are not always related– Synonymy, meronymy, etc. happen rarely inside a window

Solution: add information contained in dictionaries+ Strong semantic information+ Weak supervision using higher-level and/or noisier input fromsubject matter experts [Ratner et al. 2017], here linguists.
+ Move closer related words

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

18

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Crawling online dictionaries

Crawl dictionary webpage of each word
Parse HTML to extract the definitions
Use 4 different dictionaries (Oxford,
Cambridge, Collins, dictionary.com) →Definitions of 200k words
For each word, all senses areconcatenated (no disambiguation)
POS, etymology, synonymy ... are nottaken into accountChristophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

19

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Strong and Weak pairs

car : a road vehicle, typically with four wheels, powered by an internal
combustion engine and able to carry a small number of people.
vehicle : a thing used for transporting people or goods, especially on
land, such as a car, lorry or cart.
wheel : a circular object that revolves on an axle and is fixed below avehicle or other object to enable it to move easily over the ground.
Strong pair: mutual inclusion (car – vehicle)
Weak pair: one-sided inclusion (car – wheel), (wheel – vehicle)

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

20

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Objective function
Positive Sampling: move closer words forming either a strong or a weak pair
X
X
Jpos (wt ) = βs`(vt · vi ) + βw`(vt · vj )(1)
wi ∈Strong (wt )wj ∈Weak (wt )Controlled Negative Sampling: prevent related words to be moved further
X
Jneg (wt ) =`(−vt · vi )(2)
wi ∈F (wt )wi ∈S/ trong (wt )wi ∈W/ eak (wt )Global Objective Function
J=

C X
n
X

`(vt · vt+c ) + Jpos (wt ) + Jneg (wt )(3)

t=1 c=−nChristophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

21

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Word semantic similarity evaluation
word1

word2

tiger
book
computerplane
telephone
stock
stock
fertility
Arafat
law

cat
paper
keyboard
car
communication
phone
egg
egg
Jacksonlawyersimilarity (mean)
7.35
7.46
7.62
5.77
7.50
1.62
1.81
6.69
2.50
8.38

Table: Example of WS353 word similarity dataset.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

22

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Word semantic similarity evaluation
Trained on Wikipedia (July 2017), 3 corpora sizes
Compared to word2vec [Mikolov et al. 2013] (w2v) andfasttext [Bojanowski et al. 2017] (FT)
50M

MC-30
RW
SimVerb
WS353
YP-130

200M

Full

w2v

FT

our

w2v

FT

our

w2v

FT

our

69.7
37.5
16.5
66.0
45.8

72.2
44.2
17.9
65.7
41.5

84.0
47.5
36.3
73.8
66.6

74.2
37.7
18.3
69.4
44.9

79.5
47.5
20.6
70.1
50.9

85.4
46.7
37.7
76.2
61.6

80.9
40.7
18.6
70.5
50.2

83.1
46.4
22.2
72.9
53.3

86.0
48.2
38.4
75.6
64.6

Table: Spearman’s rank correlation scores for several similarity datasets.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

23

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Using WordNet pairs for positive sampling

Replace strong pairs with WordNet pairs
Dictionaries are better than WordNet for positive samplingNo pairs
With WordNet pairs
With dictionary pairs

50M

200M

full

45.3
56.4
56.9

47.1
56.6
56.9

48.8
55.9
57.1

Table: Weighted average semantic similarity score of dict2vec vectors whentrained with WordNet or dictionary pairs.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

24

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Using dictionaries for retrofitting
Retrofitting: moving closer related words after training as
in [Faruqui et al. 2014]
Dictionaries are also better than WordNet for retrofitting
Retrofitted w2v and FT are worse than the basic dict2vec vectors

50M

No retrofit
RWordNet
Rdictionary

200M

Full

w2v

FT

our

w2v

FT

our

w2v

FT

our

45.3
47.4
47.9

46.7
47.6
48.9

56.9
58.2
58.2

47.1
48.8
49.4

50.3
50.4
52.4

56.9
58.1
58.7

48.8
50.7
51.2

50.8
50.3
52.9

57.1
58.3
59.2

Table: Weighted average semantic similarity score of raw vectors and after
retrofitting with WordNet or dictionary pairs.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

25

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Conference reviewer“Comparison is not fair, word2vec and Fasttext never have a chance to
see the dictionary data. You should at least try to add the dictionary
content to the wikpiedia dump ?” (Reviewer2)

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

26

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Dict2Vec

Recap

Dictionaries contain important semantic information
Positive sampling allows to incorporate additional information
during learning (dictionaries or WordNet)
If retrofitting is desirable in your application, use dictionary overWordnet.
Clamping domain corpora to a Wikipedia dump for training may
be a very cheap boost for domain-specific downstream tasks.
You don’t have to think big to make something count.
Entire source code for training and downloading pretrained vectors:
https://github.com/tca19/dict2vec

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

27

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Table of contents

1 Preambule
2 Semi-structured knowledge for learning representations

Dict2Vec
3 Text generation from structured knowledge

Question Generation from Knowledge Graphs
Generating summaries from Knowledge Graphs
4 Conclusion

Digression: Binarization of word embeddings

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

28

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Question Generation from Knowledge Graphs

From [Serban et al. 2016]:
Given a Knowledge base triple[Subject, Predicate, Object]
Generate a question that asking
about the object of the triple
Feed-forward
sequence-to-sequence decoderarchitecture

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

29

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Training
Trained using datasets aligning questions (free text) and triples.
Most popular dataset: SimpleQuestions dataset [Bordes et al. 2015](Freebase KB).

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

30

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Training
Trained using datasets aligning questions (free text) and triples.
Most popular dataset: SimpleQuestions dataset [Bordes et al. 2015](Freebase KB).
Examples:(fb:fires creek, fb:containedby, fb:nantahala national forest)
Which forest is fires creek in?
(fb:jimmy neutron, fb:fictional character occupation, fb:inventor)
What does jimmy neutron do?
(fb:kimchi, fb:incompatible with dietary restrictions, fb:veganism)
What dietary restriction is incompatible with kimchi ?
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

30

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Wait. . .

We usually want to answer questions not the contrary!
Question Generation has been shown to improve the performance offactoid Question Answering Systems either by dual training or byaugmenting existing training datasets [Dong et al. 2017; Reddy et al.
2017]

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

31Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Problems
Problem 1: Coverage and bias.
75% of Freebase predicates are not in SimpleQuestions.
Problem 2: Predicates sets size have usually a monotonic increase.
Extreme case: 2500+ Wikidata predicates over the last 3 years.

How to generalize of QA systems beyond training datasets ?
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

32

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

What happens if we do nothing
Generate the question given the input :Unseen predicate: [videogame publisher]unseen entities ([Tokyo Xtreme Racer], [SEGA]).
Note: unseen predicate means we do not have [videogame publisher] in our
predicate vocabulary – we did not even saw any instances of that class in the
training set.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

33

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Expected generated questions

what is the name of the topic of the film tokyo xtreme racer?
what is the videogame of the computer videogame tokyo xtreme racer ?
what is the developer of the computer videogame tokyo xtreme racer ?
what is the genre of the album tokyo xtreme racer ?
what is the name of the tokyo xtreme racer ?
which country is known for playing by tokyo xtreme racer ?
what kinds of videogame is tokyo xtreme racer ?

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

34

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Explanations

Explanation #1 : The encoder fails to encode the new unseen IDs intomeaningful vectors. The decoder is not able to pair those vectors towords.

⇒ Garbage in, garbage out: it generates rubbish.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

35

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Explanations
Explanation #2 : The rare word problem.
The following words are rare in the training set so they have been
generated with low probability in the decoder language model – it isunlikely that they will be used to generate questions :videogamecompany
releasedgame

So instead the decoder generates:
What is the genre of the album tokyo xtreme racer ?
Note : Remember the main semantic content in Freebase. . .
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

36

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Textuals contexts
Intuitively, a human who is given the task to write a question on a fact
offered by a KB, would read natural language sentences where the entity
or the predicate of the fact occur, and build up questions that are alignedwith what he reads from both a lexical and grammatical standpoint.
Intuition
Providing models with textual contexts that have potentially highoverlap with the target Question – they can provide semantic andvocabulary richness to the decoder.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

37

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

New inputs
We start with a new set of inputs for the encoder (triple + textual
contexts) :Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

38

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Collecting textual contexts

T-REx : A Large Scale
Alignment of Natural
Language with Knowledge
Base Triples [LREC2018]
hadyelsahar.github.io/t-rex

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

39

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Textual contexts from T-Rex
Examples :Find them: plain string matching and then replace subject by [S] andobject by [O] to keep track of the direction.
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

40

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Zero-shot QG from triples: neural architecture– Facts embeddings are initialized using Trans-E– Textual encoders words embeddings using Glove.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

41

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Dealing with out-of-vocabulary and rare words
Copy action mechanism [Luong et al. 2014] using POS.
– Each word in every input
textual context is replaced by aspecial token containing a
combination of its context id(e.g. C1) and its POS tag (e.g.
NOUN).
– Then, update the question
with the corresponding tag if
possible

Inference regime
At each time step the network draw a token from the vocabulary or aplaceholder – the latter being replaced with their original words from the
textual contexts as a final step.
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

42

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Baselines
SELECT: baseline from [Serban et al. 2016].
Unseen predicate ? Pick the question from the training set that
corresponds to the triple having the same answer type.
Unseen sub-types or obj-types ? Pick the fact that have the same answer
predicate.
R-TRANSE: Same as above but we select the fact in the training set
with the closest cosine similarity (concatenation of TransE embeddings).
IR: A historically IR-based solution using TD-IDF and LSA to pick up
questions to ouput from the training set.
Encoder-Decoder: Encoder-Decoder model with a single placeholder(best model in [Serban et al. 2016]). Same embeddings initialization as
our.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

43

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Results for unseen predicatesStandard metrics from the machine translation task.
In this case we measure how close each generated question is close to its“ground truth” question.
Model
Unseen Predicates

BLEU-1

BLEU-2

BLEU-3

BLEU-4

ROUGELMETEOR

SELECTIR
IRCOPY
R-TRANSE
R-TRANSECOPY
Encoder-Decoder

46.81 ± 2.12
48.43 ± 1.64
48.22 ± 1.84
49.09 ± 1.69
49.0 ± 1.76
58.92 ± 2.05

38.62 ± 1.78
39.13 ± 1.34
38.82 ± 1.5
40.75 ± 1.42
40.63 ± 1.48
47.7 ± 1.62

31.26 ± 1.9
31.4 ± 1.66
31.01 ± 1.72
33.4 ± 1.7
33.28 ± 1.74
38.18 ± 1.86

23.66 ± 2.22
23.59 ± 2.36
23.12 ± 2.24
25.97 ± 2.22
25.87 ± 2.23
28.71 ± 2.35

52.04 ± 1.43
52.88 ± 1.24
52.72 ± 1.26
54.07 ± 1.31
54.09 ± 1.35
59.12 ± 1.16

27.11 ± 0.74
27.34 ± 0.55
27.24 ± 0.57
28.13 ± 0.54
28.12 ± 0.57
34.28 ± 0.54

Our-Model
Our-Modelcopy

60.8 ± 1.52
62.44 ± 1.85

49.8 ± 1.37
50.62 ± 1.46

40.32 ± 1.92
40.82 ± 1.77

30.76 ± 2.7
31.1 ± 2.46

60.07 ± 0.9
61.23 ± 1.2

35.34 ± 0.43
36.24 ± 0.65

Evaluation results of our model and all other baselines for the unseen
predicate evaluation setup.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

44

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Results for unseen entity types

Obj-Types

Sub-Types

Model

BLEU-4

ROUGEL

R-TRANSE
Encoder-Decoder

32.41 ± 1.74
42.14 ± 2.05

59.27 ± 0.92
68.95 ± 0.86

Our-Model
Our-Modelcopy

42.13 ± 1.88
42.2 ± 2.0

69.35 ± 0.9
69.37 ± 1.0

R-TRANSE
Encoder-Decoder

30.59 ± 1.3
37.79 ± 2.65

57.37 ± 1.17
65.69 ± 2.25

Our-Model
Our-Modelcopy

37.78 ± 2.02
38.02 ± 1.9

65.51 ± 1.56
66.24 ± 1.38

Automatic evaluation of our model against selected baselines for unseen
sub-types and obj-types.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

45

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

I’m BLEU – what ?
BLEU suffers from many limitations [Novikova et al. 2017] – as all
automatic evaluation metrics.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

46

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

I’m BLEU – what ?
BLEU suffers from many limitations [Novikova et al. 2017] – as all
automatic evaluation metrics.
Given this question : “What kind of film is kill bill vol. 2?”, which is
the most similar question :A (Bleu: 71) – What is the name of the film kill bill vol. 2 ?
B (Bleu: 55) – Which genre is kill bill vol. 2 in?

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

46

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

I’m BLEU – what ?
BLEU suffers from many limitations [Novikova et al. 2017] – as all
automatic evaluation metrics.
Given this question : “What kind of film is kill bill vol. 2?”, which is
the most similar question :A (Bleu: 71) – What is the name of the film kill bill vol. 2 ?
B (Bleu: 55) – Which genre is kill bill vol. 2 in?
Human evaluation :– Four annotators– 100 facts randomly chosen– Evaluation for each 4 models

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

46

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Question Generation from Knowledge Graphs

Human evaluation
Predicate identification: Generated question contains the given
predicate in the fact (yes/no).
Naturalness: Following [Ngonga Ngomo et al. 2013] we measure thecomprehensibility and readability of the generated questions on a 1–5
Likert scale.

Model% Pred. Identified

Encoder-Decoder
Our-Model (No Copy)
Our-Modelcopy (Types context)
Our-Modelcopy (All contexts)

6
6
37
46

Natural.
3.14
2.72
3.21
2.61

Results of human evaluation on % of predicates identified andnaturalness 0-5.
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

47

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Table of contents

1 Preambule
2 Semi-structured knowledge for learning representations

Dict2Vec
3 Text generation from structured knowledge

Question Generation from Knowledge Graphs
Generating summaries from Knowledge Graphs
4 Conclusion

Digression: Binarization of word embeddings

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

48

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Wikipedia placeholder
When an entity lacks its Wikipedia page, Wikidata is used to generate
an article placeholder as content (think underserved languages).
Display Wikidata triples in Wikipedia in a tabular way – withmultilingual interlinked data from Wikidata.
Currently deployed on 14 under-resourced Wikipedias (e.g. HaitianCreole, Urdu, Gujarati)

Access to knowledge !
Can we go further ?
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

49

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Wikipedia placeholder

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

50Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

From ArticlePlaceholder to features
Given the following text :“La Nigragora pigogarolo (Calocitta colliei) estas rimarkinda longvosta
pigogarolo de la familio ()” (extract from Esperanto Wikipedia
https://eo.wikipedia.org/wiki/Nigragor%C4%9Da pigogarolo)

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

51

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

From ArticlePlaceholder to features
Given the following text :“La Nigragora pigogarolo (Calocitta colliei) estas rimarkinda longvosta
pigogarolo de la familio ()” (extract from Esperanto Wikipedia
https://eo.wikipedia.org/wiki/Nigragor%C4%9Da pigogarolo)
We leverage existing Wikidata triples :[S] [taksonomia nomo] [Calocitta][S] [supera taksono] [Pigogarolo]...

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

51

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

From ArticlePlaceholder to features
Given the following text :“La Nigragora pigogarolo (Calocitta colliei) estas rimarkinda longvosta
pigogarolo de la familio ()” (extract from Esperanto Wikipedia
https://eo.wikipedia.org/wiki/Nigragor%C4%9Da pigogarolo)
We leverage existing Wikidata triples :[S] [taksonomia nomo] [Calocitta][S] [supera taksono] [Pigogarolo]...
Then we replace the entity or predicate in the original sentence, but we
use a property placeholder mechanism to tackle rare and oov words :La Nigragora pigogarolo ( [[P225]] colliei ) estas rimarkinda longvosta[[P171]] de la familio. . .
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

51

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Neural Wikipedian architecture
Same kind of mechanisms involved (without POS copy action and textual
contexts that we discovered later) than in the previous Question
Generation task.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

52

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Neural Wikipedian results

Standard automatic evaluation metrics results on test set of Wikipedia
article summaries.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

53

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Human evalaution
Readers

Editors– 27 participants (Wikipedia users) for
each of the 2 studied languages(Esperanto and Arabic)– Fluency: 0–6 score for text
understandabilty and grammaticallycorrect– Appropriateness : Does the summary“feel” like a Wikipedia article ?

Editors were asked to edit the article
starting from our summary and thecorresponding triples (2-3 sentences).
How much of the text was reused ?
The more, the best the summary.
Measured using Greedy String-Tiling(GST) – detects whole bock moves
unlike Levenstein distance.

Experimental sentences set made of ourgenerated sentences + sentences from
Wikipedia itself and news feed.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

54

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Readers evaluation

Mean

SD

Part of Wikipedia

ArabicAppropriatenessOur model
Wikipedia
News

4.7
4.6
5.3

1.2
0.9
0.4

77%0.77
74%0.74
35%0.35

Esperanto

Fluency

Our model
Wikipedia
News

4.5
4.9
4.2

1.5
1.2
1.2

69%0.6984%0.84
52%0.52

Results for fluency and appropriateness as assessed by Wikipedian in the
two underserved languages.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

55

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Editors evaluation

Over 60 sampled evaluated summaries, 78% Whole or Partiallyreused for Arabic, 94% for Esperanto !
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

56

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Generating summaries from Knowledge Graphs

Table of contents

1 Preambule
2 Semi-structured knowledge for learning representations

Dict2Vec
3 Text generation from structured knowledge

Question Generation from Knowledge Graphs
Generating summaries from Knowledge Graphs
4 Conclusion

Digression: Binarization of word embeddings

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

57

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Conclusion

Pipeline is here for a long time.
(Semi) structured knowledge can help in improving each stage.
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

58

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Open issues / research directions
Models that are very parameter efficient (e.g., for mobile) : do
we really need R300 ? Can we (even) better compress model weights?
Models that represent knowledge/context in latent space
Labeling data: we may have all we need. Further improvement may
comes from collecting the correct data but it is expensive anderror-prone :we may further revisit pseudo-labeling and self-training tasks.
Further solution for zero and few-shots settings.

Revise model practicability: academic datasets are (“sometimes”)
biased. Some task we tag as solved may not be solved in practice(noisy data, more classes, user-generated contents, . . . ).
Actually anything that drive us from mimicking to real understanding(inc. common sense).
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

59

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Questions ?
Tissier et al. (2019) – AAAI 2019 (pp. tbd)
Near-lossless Binarization of Word Embeddings
ElSahar et al. (2018) – NAACL-HLT 2018 (pp. 218–228)
Zero-Shot Question Generation from Knowledge Graphs for Unseen Predicates andEntity Types
Vougiouklis et al. (2018) – J. Web Semantics (52–53, pp. 1–15)
Neural Wikipedian: Generating Textual Summaries from Knowledge Base Triples
Kaffee et al. (2018) – NAACL-HLT 2018 (pp. 640–645)
Learning to Generate Wikipedia Summaries for Underserved Languages from Wikidata
Kaffee et al. (2018) – ESWC 2018 (pp. 319–334)
Mind the (Language) Gap: Generation of Multilingual Wikipedia Summaries from
Wikidata for ArticlePlaceholders
ElSahar et al. (2018) – LREC 2018 (pp. 3448 – 3452)
T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples
Tissier et al. (2017) – EMNLP 2017 (pp. 254 – 263)
Dict2vec : Learning Word Embeddings using Lexical Dictionaries
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

60

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

References I
Bar-Haim, Roy et al. (2006). “The second pascal recognising textual
entailment challenge”. In: Proceedings of the second PASCAL
challenges workshop on recognising textual entailment. Vol. 6. 1.
Venice, pp. 6–4.
Bentivogli, Luisa et al. (2009). “The Fifth PASCAL Recognizing Textual
Entailment Challenge.”. In: TAC.
Bojanowski, Piotr et al. (2017). “Enriching word vectors with subword
information”. In: Transactions of the Association for Computational
Linguistics 5, pp. 135–146.
Bordes, Antoine et al. (2015). “Large-scale simple question answeringwith memory networks”. In: arXiv preprint arXiv:1506.02075.
Dagan, Ido, Oren Glickman, and Bernardo Magnini (2005). “ThePASCAL recognising textual entailment challenge”. In: Machine
Learning Challenges Workshop. Springer, pp. 177–190.
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

61

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

References II
Devlin, Jacob (2019). BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding.
https://nlp.stanford.edu/seminar/details/jdevlin.pdf.
[Online; accessed 19-May-2019].
Devlin, Jacob et al. (2018). “BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding”. In: arXiv preprint
arXiv:1810.04805.
Dong, Li et al. (2017). “Learning to paraphrase for question answering”.
In: arXiv preprint arXiv:1708.06022.
Faruqui, Manaal et al. (2014). “Retrofitting word vectors to semantic
lexicons”. In: arXiv preprint arXiv:1411.4166.
Giampiccolo, Danilo et al. (2007). “The third pascal recognizing textual
entailment challenge”. In: Proceedings of the ACL-PASCAL workshop
on textual entailment and paraphrasing. Association for Computational
Linguistics, pp. 1–9.
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

62

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

References III
Levesque, Hector, Ernest Davis, and Leora Morgenstern (2012). “Thewinograd schema challenge”. In: Thirteenth International Conference
on the Principles of Knowledge Representation and Reasoning.
Levy, Omer et al. (2015). “Do supervised distributional methods reallylearn lexical inference relations?”. In: Proceedings of the 2015
Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies,
pp. 970–976.
Luong, Minh-Thang et al. (2014). “Addressing the rare word problem inneural machine translation”. In: arXiv preprint arXiv:1410.8206.
Mikolov, Tomas et al. (2013). “Efficient estimation of word
representations in vector space”. In: arXiv preprint arXiv:1301.3781.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

63

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

References IV

Ngonga Ngomo, Axel-Cyrille et al. (2013). “Sorry, i don’t speak
SPARQL: translating SPARQL queries into natural language”. In:
Proceedings of the 22nd international conference on World Wide Web.
ACM, pp. 977–988.
Novikova, Jekaterina et al. (2017). “Why we need new evaluation metrics
for NLG”. In: arXiv preprint arXiv:1707.06875.
Peters, Matthew E. et al. (2018). “Deep contextualized word
representations”. In: Proc. of NAACL.
Ratner, Alexander et al. (2017). “Snorkel: Rapid training data creation
with weak supervision”. In: Proceedings of the VLDB Endowment
11.3, pp. 269–282.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

64Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

References V
Reddy, Sathish et al. (2017). “Generating natural language
question-answer pairs from a knowledge graph using a rnn basedquestion generation model”. In: Proceedings of the 15th Conferenceof the European Chapter of the Association for Computational
Linguistics: Volume 1, Long Papers, pp. 376–385.
Serban, Iulian Vlad et al. (2016). “Generating factoid questions withrecurrent neural networks: The 30m factoid question-answer corpus”.
In: arXiv preprint arXiv:1603.06807.
Wang, Alex et al. (2018). “Glue: A multi-task benchmark and analysis
platform for natural language understanding”. In: arXiv preprint
arXiv:1804.07461.
Wang, Alex et al. (2019). “SuperGLUE: A Stickier Benchmark for
General-Purpose Language Understanding Systems”. In: arXiv preprint
arXiv:1905.00537.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

65Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Table of contents

1 Preambule
2 Semi-structured knowledge for learning representations

Dict2Vec
3 Text generation from structured knowledge

Question Generation from Knowledge Graphs
Generating summaries from Knowledge Graphs
4 Conclusion

Digression: Binarization of word embeddings

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

66

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

RememberSome NLP models that are not “solved” [Devlin 2019]
Models minimizing total training cost vs. accuracy on modern hardware
Models that are very parameter efficient (e.g., for mobile)
Models that represent knowledge/context in latent space
Models that represent structured data (e.g., knowledge graph)
Models that jointly represent vision and language

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

67

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Limitations of real-valued vectors

Require a lot of space for storage
Millions of words in vocabulary
300 dimensions per vector
float usually requires 32 bits
⇒ 1.2 GB to store 1M vectors

Not suitable for embedded devices
limited memory
low computing power

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

68Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Binary representations
Solution: associate a m-bits vector to each word, i.e. embed real-value
dist
vectors into a Hamming hypercube of arbitrary dimensions (Rm ,−−→ Bn ).
Faster vector operations: Considerations– Cosine similarity requires O(n) additions and multiplications+ Binary similarity requires a XOR and a popcount() operations thatcan be vectorized using CPU intrinsics (order of magnitude faster).
Small memory size
– 32 bits×300 dimensions = 9600 bits per real-valued vectors+ 128 or 256 bits per binary vectors+ 16.1 MB to store 1M vectors of 128 bits (vs. 1.2 GB)+ Computation can be done locally; no need of sending data to acomputing server
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

69

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Binary representations: Post-embedding scheme
Solution: associate a m-bits vector to each word, i.e. embed real-value
dist
vectors into a Hamming hypercube of arbitrary dimensions (Rm ,−−→ Bn ).
Binarize instead of training binary vectors
Many pre-trained are already available
Some are specific (subwords, dictionaries...). Need a general
method that works for all types of architecture
Preserve semantic and syntactic information
Why not a naive binarization ?
Binary vectors size need to be in adequacy with CPU registers
size (64, 128 or 256 bits) to benefit from hardware optimizations
Pre-trained vectors have usually 300 dimensions
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

70Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Auto-encoder architecture
Use pre-trained vectors as input (x ∈ Rm )Latent representation Φ(x) is binary (Bn )Reconstruct a real-valued vector ŷ ∈ Rm given Φ(x) ∈ Bn
How to back-propagate in the encoder, given the
non-differentiability of the Heaviside function ?

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

71

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Auto-encoder architecture
Let W ∈ Rn×m , c ∈ Rm and xi ∈ Rm
Encoding(
1
Heaviside function h(x) =0

if x ≥ 0otherwisebi = Φ(xi ) = h(W · xiT )+ Can construct binary vectors of arbitrary size
Decodingŷi = tanh(W T · Φ(xi ) + c)
Pre-trained embeddings have been cropped to be in [-1, 1] range

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

72

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Objective Function

Reconstruction loss: minimize the distance between xi and ŷim`rec (xi ) =1 X(xik − ŷik )2
m

(4)

k=0

Regularization loss: minimize the correlation between each binary dimension`reg =

1||W T W − I ||22(5)`rec (xi ) + λreg `reg

(6)

Global Objective Function
L=

X
xi ∈X

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

73

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Word semantic similarity
1 ,v2 )
BinarySimilarity (v1 , v2 ) = n−HammingDistance(vndict2vec (2.3M), fasttext (1M), GloVe (400k)
Best average results achieved with 256 bits (37.5 times smaller)

dict2vec
raw
MEN
bin
RW
bin
SimLexbin
SimVerb
bin
WS353
bin

74.6
50.5
45.2
41.7
72.5
-

64

128

fasttext
256

512

66.1

71.3

70.3

71.3

36.5

42.0

45.6

45.6

32.0

38.1

44.8

42.9

25.3

36.6

38.4

35.5

63.7

71.6

69.6

66.6

raw
80.7
53.8
44.1
35.6
69.7
-

64

128

GloVe
256

512

57.9

72.0

75.9

76.3

36.8

44.7

52.7

52.7

25.1

38.0

44.6

43.0

19.2

26.7

33.7

35.1

50.3

69.1

70.0

70.3

raw
73.7
41.2
37.1
22.7
60.9
-

64

128

256

512

46.1

63.3

69.4

72.7

25.1

34.3

40.7

40.2

20.5

31.4

37.2

36.8

7.8

18.7

22.9

23.0

30.1

44.9

56.6

60.3

Table: Spearman’s rank correlation scores on semantic similarity datasets forbinary vectors (bin) of 64, 128, 256 and 512 bits and original vectors (raw ).
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

74

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Text classification
Use the fasttext text classification model
Initialize the network with the binary vectors
256/512 bits vectors are on par/slightly outperforms real vectors

dict2vec
raw
AG-Newsbin
Amazon Fullbin
DBpediabinYahoo Ans
bin

89.0
47.5
97.6
68.1
-

64

128

fasttext
256

512

85.3

85.9

87.7

87.8

39.9

43.9

46.8

47.7

94.1

96.1

97.0

97.3

60.7

63.8

66.0

66.8

raw
86.9
49.0
95.0
67.2
-

64

128

GloVe
256

512

84.5

85.9

87.3

87.7

39.0

43.9

47.9

49.8

91.7

95.1

96.6

97.3

60.4

63.9

66.4

67.8

raw
89.5
47.1
97.2
68.1
-

64

128

256

512

84.0

87.2

88.5

88.5

37.4

42.6

46.7

47.8

90.9

95.0

96.8

97.2

57.5

62.5

66.4

66.1

Table: Document classification accuracies for binary vectors (bin) of 64, 128,
256 and 512 bits and original vectors (raw ).

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

75

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Word semantic similarity and document classification
Reconstruct 300-dimensional vectors from binary codes (W , c)
Spearman’s rank correlation score and document classification
Sometimes reconstructed vectors are better than original (GloVe)

dict2vec
raw
SimLex

rec

45.2
72.5
-

Amazon Fullrec
Yahoo Ans
rec

47.5
68.1
-

rec
WS353

64

128

fasttext
256

512

30.4

37.9

42.4

39.3

61.4

69.0

67.4

58.8

40.1

44.0

46.8

46.2

60.8

63.8

65.9

66.0

raw
44.1
69.7
49.0
67.2
-

64

128

GloVe
256

512

19.2

30.0

40.5

34.0

36.5

53.6

64.0

53.6

39.1

43.8

48.1

48.5

60.0

62.9

66.3

66.8

raw
37.1
60.9
47.1
68.1
-

64

128

256

512

19.6

19.1

34.2

38.2

26.5

42.2

56.5

62.0

39.8

45.3

47.1

47.3

58.4

64.3

66.7

67.0

Table: Word similarity scores and text classification accuracies vectors
reconstructed (rec) from binary codes and original vectors scores (raw ).

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

76

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

K-nearest neighbors search

Find the k vectors with highest cosine/binary similarity

Execution time (ms)128-bit

256-bit

512-bit

Real-valued

Top 1
Top 10
Loading vectors + Top 10

2.87 (x34)2.89 (x34)
213 (x110)

3.23 (x30)3.25 (x30)
310 (x75)

4.28 (x23)4.29 (x23)500 (x47)

97.89
98.08
23500

Table: Execution time (in ms) to run a top-K query on binary and real-valued
vectors.

Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

77

Preambule

Semi-structured knowledge for learning representations

Text generation from structured knowledge

Conclusion

References

Digression: Binarization of word embeddings

Recap
Simple architecture to binarize any real-valued word embeddings
Optimization trick: Optimize decoder weights, apply transpose to
encoder to avoid non-differentiability of Heaviside() function
Regularization trick: minimize correlation between binary dimension

Reduce the vector size by 37.5 with only 2% performance loss
Top-k query 30 times faster than with real-valued vectors
Open questions / research
Quality trade-off w.r.t. other approximate kNN techniques in Rn ?
The solution is agnostic to text classification and even NLP. How
does it generalize to other application domains ?
Embed representations built in other spaces than Rn ?
Can we do ... even more compact binary codes ?
https://github.com/tca19/near-lossless-binarization
Christophe Gravier Riken AIP Tokyo
Neural networks for NLP:Can structured knowledge help?

78