June 5, 2019.
NAACL-HLT 2019

Learning to Describe Unknown Phrases
with Local and Global Contexts,

25 // 5,25/ 5,// // /5/
555 / /

1// 1/ // /5

2

52 / /

New/unfamiliar phrases avoid us reading smoothly… I was in 3rd grade during
the sonic boom tour; one…

sonic boom |ˈsänik bo͞om| noun
a loud explosive noise caused by theshock wave from an aircraft traveling
faster than the speed of sound.

2How do human infer the meanings?
STEP1: Search web to find other usages“sonic boom”… that a sonic boom happens …
… the sonic boom kick-off concert …
… planes cause sonic boom when …
… albums are “sonic boom” and …
… for the sonic boom bonus disc …

STEP2: Classify the usages… that a sonic boom happens …
… the sonic boom kick-off concert …
… planes cause sonic boom when …
… albums are “sonic boom” and …
… for the sonic boom bonus disc …

Related to natural
phenomenon?
Related tomusic?

3

How do human infer the meanings?
STEP1: Search web to find other usages“sonic boom”… that a sonic boom happens …
… the sonic boom kick-off concert …
… planes cause sonic boom when …
… albums are “sonic boom” and …
… for the sonic boom bonus disc …

STEP2: Classify the usages… that a sonic boom happens …
… the sonic boom kick-off concert …
… planes cause sonic boom when …
… albums are “sonic boom” and …
… for the sonic boom bonus disc …

STEP3: Interpret them
… I was in 3rd grade during thesonic boom tour; one…

An albumby KISS?

4RQ: Can machines help us infer the meanings?
STEP1: Map the usages to vector space… that a sonic boom happens …
… the sonic boom kick-off concert …
… planes cause sonic boom when …
… albums are “sonic boom” and …
… for the sonic boom bonus disc …

typhoonbandsonic_boom
thriller

STEP2: Describe the target phrase reading the context

Input:… I was in 3rd grade during the sonic_boom tour; one…

Output: Album by KISS.
5

Contributions
1. Context-aware description generation task
– Large-scale dataset: 989k entries– Multi-domain: name of places, movies, persons, …
– Various expressions: phrases as well as words

2. Description generation model which– achieves SOTA performances on 4 datasets– combines local & global context dynamically… the sonic_boom tour …

typhoonsonic_boom

bandthriller

6

Task: Context-aware description generationl WPDD: Wikipedia Phrase Description Dataset

Phrase: sonic boom
This tour is the first official
tour to support the band'sContext: latest studio effort,
2009's [TRG].

Description: album by Kiss
Can we make this dataset automatically?

7

Idea: Combine Wikipedia & Wikidata
Article: Kiss Sonic Boom Over Europe… This tour is the first official tour to support theband's latest studio effort, 2009's Sonic Boom. …
Item: Sonic Boom (Q908201)
Description: album by Kiss
Instance of: album
Genre:
hard rock

Extracted Data

Phrase: sonic boom
This tour is the first official tour to support theContext: band's latest studio effort, 2009's [TRG].

Description: album by Kiss

8

Contributions
1. Context-aware description generation task
– Large-scale: 989k entries– Multi-domain: name of places, movies, persons, …
– Various expressions: phrases as well as words

2. Description generation model which– achieves SOTA performances on 4 datasets– combines local & global context dynamically… the sonic_boom tour …

typhoonsonic_boom

bandthriller

9Baseline: Definition-generation [Noraset+ 17]
l Seed-feeding to condition LSTM language modelliraEmbeddingof the word
to be defined

LSTM

ℎ$ℎ"

ℎ#

thebasicunit

Definition of the word “lira” in a dictionary

The input embedding loses its influenceas the definition becomes longer

10Baseline: Definition-generation [Noraset+ 17]

l Seed-feeding + Gate function
liraℎ%&"GATE

Controls theinfluence of
the word
embeddingℎ′%&"the11Baseline: Definition-generation [Noraset+ 17]

l Seed-feeding + Gate function
liraℎ%GATE

Controls theinfluence of
the word
embeddingℎ′%&"ℎ′%thebasic
12

Baseline: Definition-generation [Noraset+ 17]

l Seed-feeding + Gate function
liraℎ%("GATE

Controls theinfluence of
the word
embeddingtheℎ′%ℎ′%("basicunit
13Baseline: Definition-generation [Noraset+ 17]

l Seed-feeding + Gate functionliraThe input
information isfixed regardlessof contextℎ%("GATEℎ′%ℎ′%("basicunit14

Proposed: Local & Global Context-aware model
Global context

sonic

CBOWboom

Character info. [Noraset+ 17]
sonic booms o n i c _ b o o mCNN

Description decoder)′%

)%
…

LSTM

GATE)%("LSTM

movingGATE…
15

Proposed: Local & Global Context-aware model
Local context [Ni+ 18]…

causedby

[TRG]

orby…

…

Global context

sonic

sonic boomboom…

sonic boom

Attention

CBOW

Character info. [Noraset+ 17]
s o n i c _ b o o m

CNNDescription decoder)′%)%…

LSTMGATE)%("LSTM

movingGATE…
16

Proposed: Local & Global Context-aware model
Local context [Ni+ 18]…

caused…

Attention

by

[TRG]

orby

…
…

sonic boom

Global context

sonicsonic boom

boom

CBOW

Character info. [Noraset+ 17]
s o n i c _ b o o m

CNN

Global cont.: general meaning in large corpus

Local cont.: specific usage in a sentence17Experiments: Can the model utilize both contexts?
Datasetl WPDD (Proposed)l WordNet [Noraset+ AAAI17]l Oxford Dictionary [Gadetsky+ ACL18]l Urban Dictionary [Ni+ IJCNLP17]
Methodsl LOG-CaD (Proposed): Local & Global Context-aware Desc. Gen.
l Global: Uses word embedding as context [Noraset+ 17]
l Local Uses local context as context [Ni+ 17]
Evaluationl Automatic (BLEU) & Manual (5-levels, 3 En. speakers) eval.

18

Result: SOTA on four datasets
100% of the words
have global context(i.e., word embeddings)
1.6M

WordNetOnly 27% of the words
have global context
OxfordUrban

WikipediaB8

24.10

15.05

6.05

44.77

1B9

22.34

17.90

9.03

52.94

3 BCBE :

24.79

18.53

10.55

53.85

Local cont. is essential if global context is unavailable

19

Example0ACG L :B A

after being enlarged by publisher danielo'neill it was reportedly one of the largest
and most prosperous newspapers in theunited states.

BA#BA

2

Ain 1967 he returned to belfast where he met
fellow belfast artist daniel o'neill.

B:

Output #1

B8

american musician

1B9

american publisher

3 BCBE :american writer

4A9

american journalist

Output #2Sameamerican musician outputs forbritish musician different
contexts!
british musician
Irish artist

20

Result: Manual evaluation
Annotated Scores:
1: completely wrong
2: correct topic w/ wrong info.
3: correct but incomplete
4: small details are missing
5: correct

9B
1B9

2.7

3 BCBE :

3.0
G B

0ACG
BAI

1B9
3 BCBE :9the regulating body for parimutuel keirin and auto racecompetitions in japan .
a type of sports equipment
type of motor racing+E9B:#21

How do the models utilize local context?
Global context
is unreliable

Global context
is reliable

BLEU

60

Proposed

50
4030

Global+6.2~20+11.9

Local

20~30

30~40

40~

Unknown words in target phrase [%]Local context provides more gain
if global context is unreliable22How do the models utilize global context?

BLEU

Local context is
extremely short50

Proposed48
46

Local context
is long+1.7

44 ~10

10~15

Local
15~20

+0.4
20~25

25~

Length of local context (#words)

Global context is essential if thelocal context is not informative

23

Summaryl Research Question– Can machines help us describe unknown phrases?

l Answer to RQ– Yes. In average, the generated descriptions arecorrect but incompletel Contributions– Large, multi-domain phrase description dataset– SOTA model that captures local & global context

l Future work
– Use multiple local contexts to describe phrases
24