Human Language Technology:
Applica7on to Informa7on AccessLesson 4
Deep learning for NLP:
Word Representa7on Learning
October 20, 2016

EPFL Doctoral Course EE-724
Nikolaos Pappas
Idiap Research Ins7tute, Mar7gny

Outline of the talk

1. Introduc7on and Mo7va7on
2. Neural Networks - The basics
3. Word Representa7on Learning
4. Summary and Beyond Words

Nikolaos Pappas

2 /59

Deep learning•

Machine Learning boils down to minimizing an objec7ve
func7on to increase task performance•

mostly relies on human-craYed features•e.g. topic, syntax, grammar, polarity➡ Representa)on Learning: a[empts to learn

automa7cally good features or representa7ons➡ Deep Learning: machine learning algorithms based on

mul7ple levels of representa7on or abstrac7on
Nikolaos Pappas

3 /59Key point: Learning mul7ple
levels of representa7on

Nikolaos Pappas

4 /59

Mo7va7on for exploring
deep learning: Why care?
• Human craYed features are 7me-consuming, rigid,

and oYen incomplete• Learned features are easy to adapt and learn• Deep Learning provides a very ﬂexible, uniﬁed, andlearnable framework that can handle a variety of
input, such as vision, speech, and language.
• unsupervised from raw input (e.g. text)• supervised with labels by humans (e.g. sen7ment)
Nikolaos Pappas

5 /59

Mo7va7on for exploring
deep learning: Why now?
• What enabled deep learning techniques to startoutperforming other machine learningtechniques since Hinton et al. 2006?
• Larger amounts of data

• Faster computers and mul7core cpu and gpu• New models, algorithms and improvements

over “older” methods (speech, vision andlanguage)
Nikolaos Pappas

6 /59

Deep learning for speech:
Phoneme detec7on• The ﬁrst breakthrough resultsof “deep learning” on large
datasets by Dahl et al. 2010

• -30% reduc7on of error• Most recently on speechsynthesis Oord et al. 2016

Nikolaos Pappas

7 /59

Deep learning for vision:
Object detec7on• Popular topic for DL• Breakthrough on ImageNet
by Krizhevsky et al. 2012
• -21% and -51% error
reduc7on at top 1 and 5

Nikolaos Pappas

8 /59

Deep learning for language:
Ongoing•

Signiﬁcant improvements in recent years across diﬀerent
levels (phonology, morphology, syntax, seman7cs) andapplica7ons in NLP
Machine transla)on (most notable)• Ques)on answering••Sen)ment classiﬁca)on•

Summariza)on

S7ll a lot of work to be done… e.g. metrics(beyond “basic” recogni7on - a[en7on, reasoning, planning)
Nikolaos Pappas

9 /59

A[en7on mechanism fordeep learning•

Operates on input or intermediate sequence•

Chooses “where to look” or learns to assign a relevanceto each input posi7on — essen7ally parametric pooling

Nikolaos Pappas

10 /59Deep learning for language:
Machine Transla7on•

Reached the state-of-the-art in one year: Bahdanau et al.
2014, Jean et al. 2014, Gulcehre et al. 2015

Nikolaos Pappas

11 /59Outline of the talk
1. Neural Networks• Basics: perceptron, logis7c regression• Learning the parameters• Advanced models: spa7al andtemporal / sequen7al
2. Word Representa7on Learning• Seman7c similarity• Tradi7onal and recent approaches• Intrinsic and extrinsic evalua7on
3. Summary and Beyond
Nikolaos Pappas

12 /59

Introduc7on to neural networks•

Biologically inspired from
how the human brain works•

Seems to have a genericlearning algorithm•

Neurons ac7vate in
response to inputs andproduce excite otherneurons

Nikolaos Pappas

13 /59

Ar7ﬁcial neuron or Perceptron

Nikolaos Pappas

14 /59What can a perceptron do?
•

Solve linearly separable problems•… but not non-linearly separable ones.

Nikolaos Pappas

15 /59

From logis7c regression toneural networks

Nikolaos Pappas

16 /59

A neural network: several logis7c
regressions at the same 7me•

Apply several regressions to
obtain a vector of outputs•

The values of the outputs
are ini7ally unknown•

Nikolaos Pappas

No need to specify
ahead of 7me whatvalues the logis7c
regressions are trying
to predict
17 /59

A neural network: several logis7c
regressions at the same 7me•

The intermediate variables
are learned directly basedon the training objec7ve•

This makes them do a good
job at predic7ng the target
for the next layer•

Result: able to model nonlineari7es in the data!

Nikolaos Pappas

18 /59

A neural network: extension to
mul7ple layers

Nikolaos Pappas

19 /59

A neural network: Matrix
nota7on for a layer

Nikolaos Pappas

20 /59Several ac7va7on func7ons tochoose from

Nikolaos Pappas

21 /59

Learning parameters using
gradient descend•

Given training dataﬁndandthat minimizes loss with respect to these parameters•

Compute gradient with respect to parameters and makesmall step towards the direc7on of the nega7ve gradient

Nikolaos Pappas

22 /59Going large scale: Stochas7c
gradient descent (SGD)•

Approximate the gradient using a mini-batch ofexamples instead of en7re training set•

Online SGD when mini batch size is one•

Most commonly used when compared to GD

Nikolaos Pappas

23 /59

Learning parameters using
gradient descend•

Several out-of-the-box
strategies for decaying
learning rate of an
objec7ve func7on:•

Select the best
according tovalida7on set
performance

Nikolaos Pappas

24 /59Training neural networks with
arbitrary layers: Backpropaga7on•

We s7ll minimize the objec7ve func7on but this 7me we“backpropagate” the errors to all the hidden layers•

Chain rule: If y = f(u) and u = g(x), i.e. y=f(g(x)), then:•Useful basic deriva7ves:

Nikolaos Pappas

Typically, backprop
computation is
implemented in
popular libraries:
Theano, Torch,
Tensorflow
25 /59Training neural networks with
arbitrary layers: Backpropaga7on

Nikolaos Pappas

26 /59Advanced neural networks•

Essen7ally, now we have all the basic “ingredients” weneed to build deep neural networks•

More layers more non-linear the ﬁnal projec7on•

Augmenta7on with new proper7es➡ Advanced neural networks are able to deal with diﬀerent

arrangements of the input•Spa)al: convolu7onal networks•

Sequen)al: recurrent networks
Nikolaos Pappas

27 /59

Spa7al Modeling: Convolu7onal
neural networks•

Fully connected network to input pixels is not eﬃcient•

Inspired by the organiza7on of the animal visual cortex• assumes that the inputs are images• connects each neuron to a local region

Nikolaos Pappas

28 /59Sequence modeling: Recurrent
neural networks
Tradi7onal networks can’t model sequence informa7on• lack of informa7on persistence• Recursion: Mul7ple copies of the same network where
each one passes on informa7on to its successor•* Diagram from Christopher Olah’s blog.

Nikolaos Pappas

29 /59Sequence modeling: Gated
recurrent networks
Long-short term memory nets are able to learn longterm dependencies: Hochreiter and Schmidhuber 1997
• Gated RNN by Cho et al 2014 combines the forget andinput gates into a single “update gate.”•* Diagram from Christopher Olah’s blog.

Nikolaos Pappas

30 /59Sequence modeling: Neural Turing
Machines or Memory Networks•

Combina7on of recurrent network with external
memory bank: Graves et al. 2014, Weston et.al 2014

* Diagram from Christopher Olah’s blog.

Nikolaos Pappas

31 /59Sequence modeling: Recurrent
neural networks are ﬂexible* Diagram from Karpathy’s Stanford CS231n course.

• Vanilla nns • Image
cap7oning• Sen7ment
classiﬁca7on• Topic detec7on
Nikolaos Pappas• Machinetransla7on• Summariza7on• Speech recogni7on• Video classiﬁca7on

32 /59Outline of the talk
1. Neural Networks• Basics: perceptron, logis7c regression• Learning the parameters• Advanced models: spa7al andtemporal / sequen7al
2. Word Representa7on Learning• Seman7c similarity• Tradi7onal and recent approaches• Intrinsic and extrinsic evalua7on
3. Summary and Beyond* image from Lebret's thesis (2016).

Nikolaos Pappas

33 /59Seman7c similarity: How similar
are two linguis7c items?
Word level
screwdriver —?—> wrenchvery similar
screwdriver —?—> hammerli[le similar
screwdriver —?—> technician
relatedscrewdriver —?—> fruit
unrelated• Sentence level
The boss ﬁred the worker
The supervisor let the employee go very similar
The boss reprimanded the worker li[le similar
The boss promoted the worker
relatedThe boss went for jogging today
unrelated•

Nikolaos Pappas

34 /59Seman7c similarity: How similar
are two linguis7c items?
• Deﬁned in many levels• words, word senses or concepts, phrases,
paragraphs, documents• Similarity is a speciﬁc type of relatedness• related: topically or via rela7on
heart vs surgeon
wheel vs bike• similar: synonyms and hyponyms
doctor vs surgeon
bike vs bicycle
Nikolaos Pappas

35 /59Seman7c similarity: Numerousa[empts to answer that

*Image from D. Jurgens’ NAACL 2016 tutorial.

Nikolaos Pappas

36 /59Seman7c similarity: Numerousa[empts to answer that

Nikolaos Pappas

37 /59Seman7c similarity: Why do wehave so many methods?
•

New resources or methods• new datasets reveal weakness in previous methods• state-of-the-art is moving target

Task-speciﬁc similarity func7ons• Performance in new tasks not sa7sfactory•➡ Seman7c similarity is not the end-task

Pick the one which yields best results• Need for methods to quickly adapt similarity•

Nikolaos Pappas

38 /59

Two main sources formeasuring similarity

Massive text corpora
Nikolaos Pappas

Seman)c resources andknowledge bases
39 /59

How to represent seman7cs?
Vector space models•

Explicit: each dimension denotes
speciﬁc linguis7c items• interpretable dimensions• high dimensionality•

Con)nuous: dimensions are not
7ed to explicit concepts• enable comparison between
represented linguis7c items• low dimensionality
Nikolaos Pappas

40 /59

How to compare two linguis7c
items in the vector space•

Cosine of the angle θ between A and B:
A

•

θ
B

Explicit models have a serious sparsity problem due to theirdiscrete or “k-hot” vector representa7ons
france = [0, 0, 0, 1, 0, 0]
england = [0, 1, 0, 0, 0, 0]
france is near spain = [1, 0, 0, 1, 1, 1]• cos(france, england) = 0.0• cos(france, france is near spain) = 0.57
Nikolaos Pappas

41 /59

Learning word vector
representa7ons from text•

Limita7ons of knowledge-based methods• out-of-context despite validity of resources• most lack of evalua7on on prac7cal tasks•

What if we do not know anything about words?
Follow the distribu7onal hypothesis:“You shall know a word by the company it keeps”, Firth 1957
ﬁnancial ins)tu)on

The value of the central bank increased by 10%.
She oYen goes to the bank to withdraw cash.
She went to the river bank to have picnic with her child.
Nikolaos Pappas

geographical term
42 /59

Simple approach: Compute a wordin-context co-occurence matrix•

Matrix of counts between words and contexts

words
•

context

document

Limita)ons of this method:• all words have equal importance (imbalance)• vectors are very high dimensional (storage issue)• infrequent words have overly sparse vectors (makesubsequent models less robust)
Nikolaos Pappas

43 /59

The most standard approach:
Dimensionality Reduc7on•

Perform singular value decomposi7on (SVD) of the word
co-occurence matrix that we saw previously•

typically, U*Σ is used as the vector space*Image from D. Jurgens’ NAACL 2016 tutorial.

Nikolaos Pappas

44 /59

The most standard approach:
Dimensionality Reduc7on•

Syntac7cally and seman7cally related words cluster together*Plots from Rohde et al. 2005

Nikolaos Pappas

45 /59Dimensionality reduc7on withHellinger PCA• Perform PCA with Hellinger distance on the word co-occurence matrix: Lebret and Collobert 2014
• Well suited for discrete probability distribu7ons (P, Q)• Neural approaches are 7me-consuming (tuning, data)• instead compute word vectors eﬃciently with PCA• ﬁne-tuning them on speciﬁc tasks! Be[er than neural2• Limita)ons: hard to add new words, not scalable O(mn )h[ps://github.com/rlebret/hpca
Nikolaos Pappas

46 /59Dimensionality reduc7on withweighted least squares•

Glove vectors by Pennington et al 2014. Factorizes the log ofthe co-occurence matrix:•

Fast training, scalable to huge corpora but s7ll hard toincorporate new words•

Much be[er results than neural embedding, however underequivalent tuning it is not the case: Levy and Goldberg 2015
h[p://nlp.stanford.edu/projects/glove/
Nikolaos Pappas

47 /59Dimensionality reduc7on withneural networks• The main idea is to directly learn low-dimensional word

representa7ons from data

• Learning representa7ons: Rumelhart et al 1986
• Neural probabilis7c language model: Bengio et al 2003
• NLP (almost) from scratch: Collobert and Weston 2008
• Recent methods are faster and more simple• Con7nuous Bag-Of-Words (CBOW)• Skip-gram with Nega7ve Sampling (SGNS)•

word2vec toolkit: Mikolov et al. 2013
Nikolaos Pappas

48 /59

word2vec: Skip-gram with
nega7ve sampling (SGNS)•

Given the middle word predict surrounding ones in a ﬁxed
window of words (maximize log likelihood)

Nikolaos Pappas

49 /59

word2vec: Skip-gram with
nega7ve sampling (SGNS)•

How is the P(wt|h) probability implemented?

Denominator is very ineﬃcient for big vocabulary!
• Instead it uses a more scalable objec7ve, logQθ is abinary logis7c regression of word w and history h:•

Nikolaos Pappas

50 /59

word2vec: Con7nuous Bag-Of-Words
with nega7ve sampling (CBOW)•

More eﬃcient but the ordering informa7on of the words
does not inﬂuence the projec7on•

Factorizes a PMI word-context
matrix: Levy and Goldberg 2014
• builds upon exis7ng
methods (new decomp.)•

improvements on a variety
of intrinsic tasks such as
relatedness, categoriza7on
and analogy: Baroni et al2014, Schnabel et al 2015
Nikolaos Pappas

51 /59

word2vec: Learns meaningfullinear rela7onships of words•

Word vector dimensions capture several meaningful rela7ons
between words: present—past tense, singular—plural, male—
female, capital—country•

Analogy between words can be eﬃciently computed using basicarithme7c opera7ons between vectors (+, -)king - man + woman ≈ queen

Nikolaos Pappas

52 /59

Learning word representa7ons
from text: Recap•

Most methods are *similar* to SVD over PMI matrix however
word2vec has the edge over alterna7ves• scales well on massive text corpora and new words• yields top results in most tasks•

On extrinsic tasks it is essen7al to ﬁne-tune (for bea7ng BOW)

➡ Several extensions

dependency-based embeddings: Levy and Goldberg 2014
• retroﬁ[ed-to-lexicons embeddings: Faruqui et al. 2014
• sense-aware embeddings: Li and Jurafsky 2015
• visually-grounded embeddings: Lazaridou et al. 2015
• mul7lingual embeddings: Gouws et al 2015•

Nikolaos Pappas

53 /59

Open problems in seman7c
similarity research
Irregular language
can i watch 4od bbc iplayer etc with 10GB useage
allowence?

••

Mul7-word expressions
We need to sort out the problem
We need to sort the problem out•

Syntax and punctua7ons
Man bites dog | Dog bites man
A woman: without her, man is nothing.
Nikolaos Pappas

54 /59

Open problems in seman7c
similarity research• Variable-size input

Prius
A fuel-eﬃcient hybrid car
An automobile powered by both an internal combus7on (…)• Ambiguity when lacking context

The boss ﬁred his worker.
• Subjec7vity versus objec7vity

This was a good day. | This was a bad day.
• Out-of-vocabulary words: slang, hash-tags, neologisms
Nikolaos Pappas

55 /59Beyond words••

Word vectors are also useful for building seman7c vectors ofphrases, sentences and documents•

input or output space for several prac7cal tasks•

basis for mul7lingual or mul7modal transfer (via alignment)•interpretability: do we care about what each word vector
dimension means? It depends. We may need to compromise.

Next course:•

learning representa7ons of word sequences•more details on sequence models
Nikolaos Pappas

56 /59References• Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeﬀ Dean. “Distributed representa7ons of words

and phrases and their composi7onality.” In NIPS 2013.
• Jeﬀrey Pennington, Richard Socher, and Christopher D. Manning. “Glove: Global Vectors for Word
Representa7on.” In EMNLP, 2014.
• Remi Lebret, Ronan Collobert. “Word Embeddings through Hellinger PCA.” In EACL, 2014
• Quoc V. Le, and Tomas Mikolov. “Distributed Representa7ons of Sentences and Documents.” In ICML, 2014.
• Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. “Retroﬁ~ng word
vectors to seman7c lexicons.”, In ACL 2014.
• Omer Levy and Yoav Goldberg. “Dependency-Based Word Embeddings.” In ACL 2014.
• Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. "Evalua7on methods for unsupervised
word embeddings." In EMNLP, 2015.
• Omer Levy, Yoav Goldberg, and Ido Dagan. “Improving distribu7onal similarity with lessons learned from wordembeddings.” TACL, 2015.
• Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. “Problems With Evalua7on of Word
Embeddings Using Word Similarity Tasks.” In RepEval 2016.
• Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. “Sparse overcomplete word vector
representa7ons.” ACL 2015.
• Yoav Goldberg. “A primer on neural network models for natural language processing” arXiv preprint:
1510.00726, 2015.
• Ian Goodfellow, Aaron Courville, and Joshua Bengio. “Deep learning”. Book in prepara7on for MIT Press., 2015.

Nikolaos Pappas

57 /59Resources (1/2)➡ Online courses

Coursera course on “Neural networks for machine learning” by Geoﬀrey Hintonh[ps://www.coursera.org/learn/neural-networks• Coursera course on “Machine learning” by Andrew Ng
h[ps://www.coursera.org/learn/machine-learning• Stanford CS224d “Deep learning for NLP” by Richard Socherh[p://cs224d.stanford.edu/•➡ Conference tutorials

Richard Socher and Christopher Manning, “Deep learning for NLP”, EMNLP 2013
tutorial.
h[p://nlp.stanford.edu/courses/NAACL2013/• David Jurgens and Mohammad Taher Pilehvar, “Seman7c Similarity Fron7ers: From
Concepts to Documents”, EMNLP 2015 tutorial.
h[p://www.emnlp2015.org/tutorials.html#t1• Mitesh M Kharpa, Sarath Chandar, “Mul7lingual and Mul7modal Language
Processing”, NAACL 2016 tutorial.
h[p://naacl.org/naacl-hlt-2016/t2.html•

Nikolaos Pappas

58 /59Resources (2/2)➡ Deep learning toolkits• Theano h[p://deeplearning.net/soYware/theano• Torch h[p://www.torch.ch/• Tensorﬂow h[p://www.tensorﬂow.org/• Keras h[p://keras.io/➡ Pre-trained word vectors and codes• Word2vec toolkit and vectors
h[ps://code.google.com/p/word2vec/• GloVe code and vectors
h[p://nlp.stanford.edu/projects/glove/• Hellinger PCA
h[ps://github.com/rlebret/hpca• Online word vector evalua7onh[p://wordvectors.org/
Nikolaos Pappas

59 /59