Basics and (Modern) Methods for
Natural Language Processing
Digital Product School of UnternehmerTUM
June 5, 2018

Nikolaos Pappas
Natural Language Understanding Group
Idiap Research Ins>tute, Mar>gny

Outline of the talk
1. Introduc>on and Mo>va>on
2. Word Representa>on Learning• Seman>c similarity• Tradi>onal and recent approaches• Intrinsic and extrinsic evalua>on
3. Word Sequence Modeling• Essen>als: Encoders, ARen>on, DL tricks• Text Classiﬁca>on• Machine Transla>on
4. Conclusion and Discussion
Nikolaos Pappas

2 /111Natural Language Processing• NLP is a ﬁeld at the intersec>on of AI and linguis>cs• Linguis>cs (structure of language, brain mappings,

language learning)• Computa>onal Linguis>cs (comp. modelsof language, tools for studying language)

Artificial
NLP Linguistics
Intelligence• Goals✓ Process large amounts of natural text
✓ Give computers the ability to “understand”

language to perform useful tasks➡Intrinsic tasks: parsing, language modeling, etc➡Extrinsic tasks: speech recogni>on, transla>on, etc
Nikolaos Pappas

3 /111Levels of processing➡ Lexical level

Speech: phone>c analysis• Vision: character recogni>on➡ Morphological & Syntac8c levels• Word structure (forms, inﬂec>ons)• Sentence structure (grammar, syntax)➡ Seman8c & Discourse levels• Word and sentence meanings• Broad context, co-reference•➡ The ul>mate goal of a system however is to be abletranslate, assist, retrieve, classify, communicate
Nikolaos Pappas

4 /111

Intrinsic tasks: Text segmenta>on& Morphology••

Tokeniza>on (split text into • Stemming (reduc>on of
meaningful segments)
word forms to stems)

Punctua>on predic>on•

Lemma>za>on (reduc>on ofword forms to base form &intended POS and meaning )Morphemes: smallest linguistic pieces with a grammatical function (inflectional: fish-> fishes, derivational: fish ->fishery, compounding: sky + scraper -> skyscraper)

Nikolaos Pappas

5 /111

Intrinsic tasks: Syntax & Grammar•

Part-of-speech tagging(POS tags sequence)•Cons>tuency parsing (nested
phrasal structures)•

Dependency parsing (rolespeciﬁc structures)•

Language modeling (wordsequence)

DT: determiner, NN: noun, singular, VBD: verb past tense, NP: noun phrase, VP: Verb phrase, ATT: attributive,
SBJ: nominal subject, TMP: temporal modifier, PC: prepositional complement

Nikolaos Pappas

6 /111

Intrinsic tasks: Seman>cs & Discourse•

Lexical seman>cs•

Named en>ty recogni>on•

Textual entailment(direc>onal rela>on
between text fragments)•

Coreference Resolu>on (ﬁndexpressions referring to thesame en>ty in a text)

Nikolaos Pappas

7 /111Extrinsic Tasks•

Machine transla>on•Ques>on answering•

Nikolaos Pappas

8 /111Extrinsic Tasks•

Sen>ment analysis•

Summariza>on•

Dialogue agents / Chatbots• Topic recogni>on• Search and retrieval• and more•

Nikolaos Pappas

9 /111

What is special about natural
language?
•

Formal languages are sta8c, explicit and non-ambiguous• Deﬁned as mathema>cal abstrac>ons (alphabet, rules)• One can explicitly enumerate all well-formed words•

Natural languages are dynamic, implicit and ambiguous• Exist in the real world and is spoken by its users• Grammar is discovered through empirical inves>ga>on“Cathrine and John gave flowers to Mary.
She said “thanks” and put them in a vase.”“I need to talk to you asap.” (abbreviations)“Did you download the app?” (neologisms)
Nikolaos Pappas

10 /111What is natural language?
• A naturally evolved system used by humans to express thoughts for(i) communica>ng with one another, (ii) learning from previous(iii) experiences and achieving their goals• Essen>ally, it is a discrete / symbolic / categorical signaling system✓ Symbols are invariant across signals (audio, visual)✓ Concise and grounded on shared knowledge (entails ambigui>es)“Did you watch the finals? Our goalkeeper was useless!”✓ Unlimited expressive power (implies ﬂexible interpreta>on rules

i.e. meaning cannot be exclusively expressed in the surface form)“All politicians lie.”|
Nikolaos Pappas

11 /111What are the main challenges inlanguage `understanding’?
•

Symbolic encodings require large vocabularies• Sparsity issues for machine learning• Scaling issues in real-world sejngs•

Brain encodings appear to be a con>nuous paRernof ac>va>ons (distributed across neurons)➡ Con>nuous encodings provides a cogni>velyplausible way to encode thoughts
Buchweitz et al. 2009•

Challenges• How to learn con>nuous encodings that generalize well?
• Can we encode very complex thoughts in a single con>nuous encoding?
• Can we create and reason over thoughts to solve any NLP task?
• How to transfer knowledge from one domain, task, language to another?
Nikolaos Pappas

12 /111What is Deep Learning?
• Machine Learning boils down to minimizing an objec>ve func>on

to increase task performance• Mostly relies on human-craoed features• Tasks involve regression, classiﬁca>on,
structured predic>on, representa>on learning➡ Representa8on Learning: Learn good features or representa>ons➡ Deep Learning: Machine learning algorithms basedon mul>ple levels of representa>on or abstrac>on✓ Biologically inspired from how the human brain works✓ Neurons ac>vate to certain inputs and excite other neurons✓ Can handle a variety of input, such as vision, speech, and language
Nikolaos Pappas

13 /111

Deep Learning: Why thisdecade?
• What enabled deep learning techniques to startoutperforming other machine learningtechniques since Hinton et al. 2006?
• Larger amounts of data

• Faster computers and mul>core cpu and gpu• New models, algorithms and improvements

over “older” methods (speech, vision andlanguage)
Nikolaos Pappas

14 /111

Deep Learning for speech:
Phoneme detec>on• The ﬁrst breakthrough resultsof “deep learning” on large
datasets by Dahl et al. 2010

• -30% reduc>on of error• Most recently on speechsynthesis Oord et al. 2016

Nikolaos Pappas

15 /111

Deep Learning for vision:
Object detec>on• Popular topic for DL• Breakthrough on ImageNet
by Krizhevsky et al. 2012
• -21% and -51% error
reduc>on at top 1 and 5

Nikolaos Pappas

16 /111

Deep Learning for language:
Ongoing•

Signiﬁcant improvements in recent years across diﬀerent
levels (phonology, morphology, syntax, seman>cs) andapplica>ons in NLP
Machine transla8on (most notable)• Ques8on answering••

Sen8ment classiﬁca8on•

Summariza8on
S>ll a lot of work to be done…
(beyond supervised and “basic” recogni>on)
Nikolaos Pappas

17 /111

Deep Learning for language:
Machine Transla>on•

Reached the state-of-the-art in one year: Bahdanau et al.
2014, Jean et al. 2014, Gulcehre et al. 2015

Nikolaos Pappas

18 /111Neural network components forlanguage `understanding’Distributed Representa8ons (word/subword units)• Ability to represent `meaning’ eﬃciently• Abstrac8on & Composi8on (word sequences)• Ability to compose complex `meanings’ from simpler ones
• AKen8on Mechanism• Ability to focus on/collect what is `relevant’ (input, memory)• Memory Mechanism• Ability to store/retrieve important previous informa>on/knowledge• Reasoning Mechanism• Ability to reason with what is `relevant’• Learning Mechanism• Ability to learn from past experience•…

Nikolaos Pappas

19 /111

Outline of the talk
1. Introduc>on and Mo>va>on• Basics: Perceptron, NNs, SGD
2. Word Representa>on Learning• Seman>c similarity• Tradi>onal and recent approaches• Intrinsic and extrinsic evalua>on
3. Word Sequence Modeling• Essen>als: Encoders, ARen>on, DL tricks• Text Classiﬁca>on• Machine Transla>on
4. Conclusion and Discussion
Nikolaos Pappas

20 /111

Basics: Perceptron

Nikolaos Pappas

21 /111

Basics: What can a perceptron
do?
•

Solve linearly separable problems•… but not non-linearly separable ones.

Nikolaos Pappas

22 /111

Basics: From logis>c regression
to neural networks

Nikolaos Pappas

23 /111

Basics: Neural network•

Apply several regressions to
obtain a vector of outputs•

The values of the outputs
are ini>ally unknown•

Nikolaos Pappas

No need to specify
ahead of >me whatvalues the logis>c
regressions are trying
to predict
24 /111

Basics: Neural network•

The intermediate variables
are learned directly basedon the training objec>ve•

This makes them do a good
job at predic>ng the target
for the next layer•

Result: able to model nonlineari>es in the data!

Nikolaos Pappas

25 /111

Basics: Neural network withmul>ple layers

Nikolaos Pappas

26 /111

Basics: Learning model
parameters with gradient descend•

Given training dataﬁndandthat minimizes loss with respect to these parameters•

Compute gradient with respect to parameters and makesmall step towards the direc>on of the nega>ve gradient• Apply chain-rule for nested func>ons e.g. y=f(g(x))

Nikolaos Pappas

27 /111

Basics: Stochas>c gradientdescent (SGD)•

Approximate the gradient using a mini-batch ofexamples instead of en>re training set•

Online SGD when mini batch size is one•

Most commonly used when compared to GD

Nikolaos Pappas

28 /111

Basics: Choosing a Stochas>c
Op>miza>on Algorithm•

Several out-of-the-box
strategies for decaying
learning rate of an
objec>ve func>on:•Select the best
according tovalida>on set
performance

Nikolaos Pappas

29 /111

Training neural networks with
arbitrary layers: Backpropaga>on•

We s>ll minimize the objec>ve func>on but this >me we“backpropagate” the errors to all the hidden layers•

Chain rule: If y = f(u) and u = g(x), i.e. y=f(g(x)), then:•Useful basic deriva>ves:•

Nikolaos Pappas

Typically, backprop
computation is
implemented in
popular libraries:
Theano, Torch,
Tensorflow
30 /111

Basics: The end•

Essen>ally, now we have all the basic “ingredients” we need tobuild deep neural networks•

However, we will also need
➡ Ability to learn from diﬀerent inputs (spa>al, sequen>al,

con>nuous vs discrete)➡ Overcome op>miza>on diﬃcul>es (exploding/vanishinggradient, informa>on ﬂow, convergence)➡ Avoid overﬁjng / Regulariza>on (dropout, L2 norm)➡ and other…
Nikolaos Pappas

31 /111

Outline of the talk
1. Introduc>on and Mo>va>on
2. Word Representa>on Learning• Seman>c similarity• Tradi>onal and recent approaches• Intrinsic and extrinsic evalua>on
3. Word Sequence Modeling• Essen>als: Encoders, ARen>on, DL tricks• Text Classiﬁca>on• Machine Transla>on* Figure from Lebret’s thesis, EPFL

4. Conclusion and Discussion
Nikolaos Pappas

32 /111

Seman>c similarity: How similar
are two linguis>c items?
Word level
screwdriver —?—> wrenchvery similar
screwdriver —?—> hammer
liRle similar
screwdriver —?—> technician
relatedscrewdriver —?—> fruit
unrelated• Sentence level
The boss ﬁred the worker
The supervisor let the employee go very similar
The boss reprimanded the worker liRle similar
The boss promoted the worker
relatedThe boss went for jogging today
unrelated•

Nikolaos Pappas

33 /111

Seman>c similarity: How similar
are two linguis>c items?
• Deﬁned in many levels• Words, word senses or concepts, phrases,
paragraphs, documents• Similarity is a speciﬁc type of relatedness• Related: topically or via rela>onheart vs surgeon
wheel vs bike• Similar: synonyms and hyponyms
doctor vs surgeon
bike vs bicycle
Nikolaos Pappas

34 /111

Seman>c similarity: Numerous
aRempts to answer that

*Image from D. Jurgens’ NAACL 2016 tutorial.

Nikolaos Pappas

35 /111

Seman>c similarity: Numerous
aRempts to answer that

Nikolaos Pappas

36 /111

Seman>c similarity: Why do wehave so many methods?
•

New resources or methods• Datasets reveal weakness in previous methods• State-of-the-art is moving target

Task-speciﬁc similarity func>ons• Performance in new tasks not sa>sfactory•➡ Seman>c similarity is not the end-task

Pick the one which yields best results• Need for methods to quickly adapt similarity•

Nikolaos Pappas

37 /111

Two main sources formeasuring similarity

Massive text corpora
Nikolaos Pappas

Seman8c resources andknowledge bases
38 /111

How to Represent Word
`Meaning’?
• Discrete: each dimension

denotes a speciﬁc linguis>c item
• Interpretable dimensions• High dimensionality

dog = [0, 0, 0, 1, 0, 0]
cat = [0, 1, 0, 0, 0, 0]
sim(dog, cat) = 0.0• Con8nuous: dimensions are not>ed to explicit concepts• Enable comparison between
represented linguis>c items• Low dimensionality
Nikolaos Pappas

39 /111

How to compare two linguis>c
items in the vector space•

Cosine of the angle θ between A and B:
A

•

θ
B

Explicit models have a serious sparsity problem due to theirdiscrete or “k-hot” vector representa>ons
france = [0, 0, 0, 1, 0, 0]
england = [0, 1, 0, 0, 0, 0]
france is near spain = [1, 0, 0, 1, 1, 1]• cos(france, england) = 0.0• cos(france, france is near spain) = 0.57
Nikolaos Pappas

40 /111

Learning Word Representa>ons
From Text
Limita>ons of knowledge-based methods• Out-of-context despite validity of resources• Most lack of evalua>on on prac>cal tasks• What if we do not know anything about words?
•-

Follow the distribu>onal hypothesis (unsupervised): “Youshall know a word by the company it keeps”, Firth 1957
ﬁnancial ins8tu8on

The value of the central bank increased by 10%.
She ooen goes to the bank to withdraw cash.
She went to the river bank to have picnic with her child.
Nikolaos Pappas

geographical term
41 /111

Simple approach: Compute a wordin-context co-occurence matrix•

Matrix of counts between words and contexts

words
•

context

document

Limita8ons• All words have equal importance (imbalance)• Vectors are very high dimensional (storage issue)• Infrequent words have overly sparse vectors (makesubsequent models less robust)
Nikolaos Pappas

42 /111

The most standard approach:
Dimensionality Reduc>on•

Perform singular value decomposi>on (SVD) of the word
co-occurence matrix that we saw previously•

Typically, U*Σ is used as the vector space*Image from D. Jurgens’ NAACL 2016 tutorial.

Nikolaos Pappas

43 /111

The most standard approach:
Dimensionality Reduc>on•

Syntac>cally and seman>cally related words cluster together•*Plots from Rohde et al. 2005

Nikolaos Pappas

44 /111Dimensionality reduc>on withHellinger PCA• Perform PCA with Hellinger distance on the word co-occurence matrix: Lebret and Collobert 2014
• Well suited for discrete probability distribu>ons (P, Q)• Neural approaches are >me-consuming (tuning, data)• Instead compute word vectors eﬃciently with PCA• Fine-tuning them on tasks; beRer than neural• Limita8ons: hard to add new words, not scalable O(mn2)hRps://github.com/rlebret/hpca
Nikolaos Pappas

45 /111Dimensionality reduc>on withweighted least squares•

Glove vectors by Pennington et al 2014. Factorizes the log ofthe co-occurence matrix:•

Fast training, scalable to huge corpora but s>ll hard toincorporate new words•

Much beRer results than neural embedding, however underequivalent tuning it is not the case: Levy and Goldberg 2015
hRp://nlp.stanford.edu/projects/glove/
Nikolaos Pappas

46 /111Dimensionality reduc>on withneural networks• The main idea is to directly learn low-dimensional word

representa>ons from data

• Learning representa>ons: Rumelhart et al 1986
• Neural probabilis>c language model: Bengio et al 2003
• NLP (almost) from scratch: Collobert and Weston 2008
• Recent methods are faster and more simple• Con>nuous Bag-Of-Words (CBOW)• Skip-gram with Nega>ve Sampling (SGNS)•

word2vec toolkit: Mikolov et al. 2013
Nikolaos Pappas

47 /111

word2vec: Skip-gram withnega>ve sampling (SGNS)•

Given the middle word predict surrounding ones in a ﬁxed
window of words (maximize log likelihood)

Nikolaos Pappas

48 /111

word2vec: Skip-gram withnega>ve sampling (SGNS)•

How is the P(wt|h) probability computed?

Denominator is very costly for big vocabulary!
• Instead it uses a more scalable objec>ve, logQθ is abinary logis>c regression of word w and history h:•

Nikolaos Pappas

49 /111

word2vec: Con>nuous Bag-Of-Words
with nega>ve sampling (CBOW)•

More eﬃcient but the ordering informa>on of the words
does not inﬂuence the projec>on•

Factorizes a PMI word-context
matrix: Levy and Goldberg 2014
• Builds upon exis>ng
methods (new decomp.)•

Improvements on a variety
of intrinsic tasks such as
relatedness, categoriza>on
and analogy: Baroni et al2014, Schnabel et al 2015
Nikolaos Pappas

50 /111

Distributed representa>ons:
Encoded proper>es•

Encodes general-purpose rela>ons between words: present—
past tense, singular—plural, male—female, capital—country•

Analogy between words can be eﬃciently computed using basicarithme>c opera>ons between vectors (+, -)king - man + woman ≈ queen

Nikolaos Pappas

51 /111

Summary: Learning word
representa>ons
Neural versus count-based methods• neural ones implicitly do SVD over a PMI matrix• similar to count-based when using the same tricks• Neural methods appear to have the edge (word2vec)• eﬃcient and scalable objec>ve + toolkit• intui>ve formula>on (=predict words in context)➡ Several extensions• Dependency-based embeddings: Levy and Goldberg 2014
• RetroﬁRed-to-lexicons embeddings: Faruqui et al. 2014
• Sense-aware embeddings: Li and Jurafsky 2015
• Visually-grounded embeddings: Lazaridou et al. 2015
• Mul>lingual embeddings: Gouws et al 2015•

Nikolaos Pappas

52 /111

Summary: Learning word
representa>ons
How can we beneﬁt from them?
• study linguis>c proper>es of words• inject general knowledge on downstream tasks• transfer knowledge across languages or modali>es• representa>ons of word sequences

Nikolaos Pappas

53 /111

Outline of the talk
1. Introduc>on and Mo>va>on
2. Word Representa>on Learning• Seman>c similarity• Tradi>onal and recent approaches• Intrinsic and extrinsic evalua>on
3. Word Sequence Modeling• Essen>als: RNNs, ARen>on, DL tricks• Text Classiﬁca>on• Machine Transla>on* Figure from Colah’s blog, 2015.

4. Conclusion and Discussion
Nikolaos Pappas

54 /111

Language Modeling•

Computes the joint probability of a sequence of words by employing
the chain rule (“How likely is a text”):
p(w1,w2,…,wt) = p(w1)p(w2|w1)p(w3|w2,w1)… p(wt|wt-1,wt-2,…)•

Given the observed text how likely is the new uRerance?
p(wt|wt-1,…,w1)•

Evalua>on

Hence, we can compare orderings (transla>on)p(he likes apples) > p(apples likes he)or word choice (speech recogni>on)p(he likes apples) > p(he licks apples)➡ Exact decomposi>on allows to learn complex distribu>ons➡ Many NLP tasks can be structured as (condi>onal) language model
Nikolaos Pappas

55 /111

Language Modeling:
Markov Models•

N-gram models: history of observed words is approximated withjust the previous n words (Markov model):• hard to capture long-term dependencies (bounded memory)• does not leverage word seman>cs and rela>onships•

Neural n-gram models: embed the same ﬁxed n-gram history in acon>nuous space (s>ll Markov model)• captures beRer correla>ons + smaller memory footprint•

trained with MLE
Nikolaos Pappas

56 /111

Language Modeling:
Recurrent Neural Networks (RNN)•

With RNN LMs we drop the ﬁxed n-gram history andcompress the en>re history in a ﬁxed length vector• long range correla>ons are captured — in theory• can represent unbounded dependencies• but, they are hard to learn (vanishing gradient)

Nikolaos Pappas

57 /111

Language Modeling: Deep RNNs•

Increasing the size of the hidden layer results in aquadra>c increase in the model size and computa>on•

Stacking mul>ple RNNs increases the memory capacity
and representa8onal ability with linear scaling•

We can also increase depth in the >me dimension

Nikolaos Pappas

58 /111

Scaling: Large Vocabularies•

Much of the computa>onal cost comes from the classiﬁca>onlayer because its parameters depend on the size of the vocabulary:•

Several solu>ons exist•

Short-lists: use most frequent words + n-gram LM for the rest•

Local short-lists: subsets of vocabulary speciﬁc to data segments•

Gradient approxima>ons: use Noise Contras>ve Es>ma>on(NCE) i.e. learning a binary classiﬁer to dis>nguish between data
samples from k samples from a noise distribu>on:

Nikolaos Pappas

59 /111

Scaling: Large Vocabularies•

Changing the input granularity and model text at themorpheme or character level• Much smaller soomax but longer dependencies• It captures morphological proper>es of words• Byte-Pair Encoding method is most common forneural MT (Sennrich et al 2015)

Nikolaos Pappas

60 /111

Long Short Term Memory (LSTM)•

Long-short term memory nets are able to learn longterm dependencies: Hochreiter and Schmidhuber 1997
Simple RNN:

* Figure from Colah’s blog, 2015.

Nikolaos Pappas

61 /111

Long Short Term Memory (LSTM)•

Long-short term memory nets are able to learn longterm dependencies: Hochreiter and Schmidhuber 1997
• Ability to remove or add informa>on to the cellstate regulated by “gates” (avoids grad. vanishing)•* Figure from Colah’s blog, 2015.

Nikolaos Pappas

62 /111

Gated Recurrent Unit (GRU)•

Gated RNN by Chung et al, 2014 combines the forget
and input gates into a single “update gate”• keep memories to capture long-term dependencies• allow error messages to ﬂow at diﬀerent strengths* Figure from Colah’s blog, 2015.

zt: update gate — rt: reset gate — ht: regular RNN update
Nikolaos Pappas

63 /111

Deep Bidirec>onal Models• Here RNN but it applies to LSTMs and GRUs too(Irsoy and Cardie, 2014)

Nikolaos Pappas

64 /111

Convolu>onal Neural Network(CNN)
Typically good for images• Convolu>onal ﬁlter(s) is (are) applied
every k words:••

Similar to Recursive NNs but without
constraining to gramma>cal phrases
only, as Socher et al., 2011
• no need for a parser (!)• less linguis>cally mo>vated ?

(Collobert et al., 2011)(Kim, 2014)
Nikolaos Pappas

65 /111

Hierarchical Models• Word-level and sentence-level abstrac>ons(Tang et al., 2015)
Nikolaos Pappas

66 /111

ARen>on Mechanism:
Machine Transla>on• Can we compress all the needed informa>on in thelast encoder state? Idea: use all the hidden states!
• length propor>onal to sentence length• weighted average of all hidden states• Learns to assign a relevance to each input

posi>on given current encoder state and theprevious decoder state• soo bilingual alignment model(Bahdanau et al., 2015)
Nikolaos Pappas

67 /111

ARen>on Mechanism:
Machine Transla>on(Bahdanau et al., 2015)
Nikolaos Pappas

68 /111

ARen>on Mechanism:
Document Classiﬁca>on
Operates on input word
sequence or intermediate
hidden states• Learns to focus on relevant
parts of the input with respect
to each target label• soo summariza>on model• Can be applied at mul>ple
language levels (Yang et al,
2016)•(Pappas and Popescu-Belis, 2014 & 2017)
Nikolaos Pappas

69 /111

Hierarchical aRen>on networks•

Very similar hierarchicalstructure as Tang et al., 2015
except average pooling• aRen>on mechanism at theword and document levels(Yang et al., 2016)
Nikolaos Pappas

70 /111

ARen>on Mechanism:
Sen>ment Classiﬁca>on(Yang et al, 2016)

Nikolaos Pappas

71 /111

Memory mechanism: Neural Turing
Machines or Memory Networks•

Combina>on of recurrent network with external
memory bank: Graves et al. 2014, Weston et.al 2014

* Diagram from Christopher Olah’s blog.

Nikolaos Pappas

72 /111

Residual connec>ons•

Residual learning allows informa>on to
ﬂow more easily by adding the input of alayer F(x) to its output i.e. F(x) + x•

It’s typically used for making connec>ons
from one layer to another•

This improves the training and avoids
the vanishing gradient problem•

Layer is ignored if not beneﬁcial
Nikolaos Pappas

73 /111Other DL tricks• Dropout units at random: is used as a regulariza>on

method to avoid overﬁjng• It allows to over-parameterize a network and s>ll
generalize well
• Layer output normaliza8on: stabilizes the training processof a model, especially useful for self-aRen>on architecture• Hyper-parameter op8miza8on: tuning well may have veryhuge impact in performance• and more (e.g. ini>aliza>on, label smoothing, weight decay)
Nikolaos Pappas

74 /111

Pujng everything together:
Flexible modeling• Sen>ment classiﬁca>on• Topic detec>on• Spam detec>on• Named En>ty Recogni>on• Machine transla>on• Summariza>on• Image cap>oning• Conversa>onal agents• Ques>on answering• Paraphrase detec>on• Rela>on Extrac>on✓ Mul>ple levels of abstrac>on (deep, hierarchical)✓ End-to-end training with stochas8c gradient descent✓ Good basis for mul>-task learning / transfer learning
Nikolaos Pappas

75 /111

Outline of the talk
1. Introduc>on and Mo>va>on
2. Word Representa>on Learning• Seman>c similarity• Tradi>onal and recent approaches• Intrinsic and extrinsic evalua>on
3. Word Sequence Modeling• Essen>als: RNNs, ARen>on, DL tricks• Text Classiﬁca>on• Machine Transla>on* Figure from Colah’s blog, 2015.

4. Conclusion and Discussion
Nikolaos Pappas

76 /111

Paragraph vectors forDocument Classiﬁca>on

Learning vectors of paragraphs inspired by word2vec• trained without supervision on a large corpus• preferably similar domain as the target• Two methods: with or without word ordering•(Le et al., 2014)
Nikolaos Pappas

77 /111

Paragraph vectors forDocument Classiﬁca>on
Learned paragraph vectors + logis>c regression• Outperformed previous method on sentence-level anddocument-level sen>ment classiﬁca>on•(Le et al., 2014)
Nikolaos Pappas

78 /111

Convolu>onal neural network
for Document Classiﬁca>on
Used mul>ple ﬁlter widths• Dropout regulariza>on (randomly dropping por>on of
hidden units during back-propaga>on)•(Kim et al., 2014)
Nikolaos Pappas

79 /111

Convolu>onal neural network
for Document Classiﬁca>on•

Not all baseline methods used drop-out though(Kim et al., 2014)
Nikolaos Pappas

80 /111Modeling and Summarizing Documentswith a Convolu>onal Network•

Similar to Kim et al, 2014 however diﬀerent• K-max pooling instead of max pooling• Two layers of convolu>ons(Denil et al., 2014)
Nikolaos Pappas

81 /111Modeling and Summarizing Documentswith a Convolu>onal Network(Denil et al., 2014)
Nikolaos Pappas

82 /111Modeling and Summarizing Documentswith a Convolu>onal Network(Denil et al., 2014)
Nikolaos Pappas

83 /111Modeling and Summarizing Documentswith a Convolu>onal Network(Denil et al., 2014)
Nikolaos Pappas

84 /111

Gated recurrent neural network
for Document Classiﬁca>on(Tang et al., 2015)
Nikolaos Pappas

85 /111

Gated recurrent neural network
for Document Classiﬁca>on(Tang et al., 2015)
Nikolaos Pappas

86 /111

Hierarchical aRen>on networks
for Document Classiﬁca>on•Very similar hierarchicalstructure as Tang et al., 2015
except average pooling• aRen>on mechanism at theword and document levels(Yang et al., 2016)
Nikolaos Pappas

87 /111

Hierarchical aRen>on networks
for Document Classiﬁca>on(Yang et al., 2016)
Nikolaos Pappas

88 /111

Outline of the talk
1. Introduc>on and Mo>va>on
2. Word Representa>on Learning• Seman>c similarity• Tradi>onal and recent approaches• Intrinsic and extrinsic evalua>on
3. Word Sequence Modeling• Essen>als: RNNs, ARen>on, DL tricks• Text Classiﬁca>on• Machine Transla>on* Figure from Colah’s blog, 2015.

4. Conclusion and Discussion
Nikolaos Pappas

89 /111

RNN encoder-decoder for
Machine Transla>on
GRU as hidden layer• Maximize the log likelihoodof the target sequence
given the source sequence:••

WMT 2014 (EN→FR)(Cho et al., 2014)
Nikolaos Pappas

90 /111Sequence to sequence learningfor Machine Transla>on• LSTM hidden layers instead of GRU• 4 layers deep instead of shallow encoder-decoder(Sutskever et al., 2014)
Nikolaos Pappas

91 /111Sequence to sequence learningfor Machine Transla>on•

WMT 2014 (EN→FR)•

PCA projec>on of the hidden state of the last encoder layer(Sutskever et al., 2014)
Nikolaos Pappas

92 /111

Jointly learning to align andtranslate for Machine Transla>on
Limita8on: can we compress all
the needed informa>on in thelast encoder state?
• Idea: use all the hidden statesof the encoder• length propor>onal to thatof the sentence!
• compute a weighted averageof all the hidden states•(Bahdanau et al., 2015)
Nikolaos Pappas

93 /111

Jointly learning to align andtranslate for Machine Transla>on•

WMT 2014 (EN→FR)(Bahdanau et al., 2015)
Nikolaos Pappas

94 /111

Eﬀec>ve approaches toaRen>on-based NMT• Global and local aRen>on• Input-feeding approach• Stacked LSTM instead of single-layer(Luong et al., 2015)
Nikolaos Pappas

95 /111

Mul>-source NMT• Train p(e|f, g) model

directly on trilingual data
• Use it to decode e givenany (f, g) pair
• Take local-aRen>on NMT
model and concatenate
context from mul>plesources(Zoph and Knight, 2016)
Nikolaos Pappas

96 /111

Mul>-source NMT• Mul>-source training improves over individual

French English and German English pairs• Best: basic concatena>on with aRen>on(Zoph and Knight, 2016)
Nikolaos Pappas

97 /111

Mul>-target NMT• Mul>-task learning framework for mul>ple target

language transla>on• Op>miza>on for one to many model(Dong et al., 2015)
Nikolaos Pappas

98 /111

Mul>-target NMT• Improves over NMT

and moses baselines
over WMT 2013 test
• but also on larger
datasets• Faster and beRer
convergence inmul>ple languagetransla>on(Dong et al., 2015)
Nikolaos Pappas

99 /111

Mul>-way, Mul>lingual NMT••

Encoder-decoder model withmul>ple encoders and decoders
shared across language pairs• share knowledge through auniversal space• good for low-resource langs
ARen>on is pair speciﬁc, hence
expensive O(L^2)• instead share aRen>on acrossall pairs!

Figure: n_th encoder and m_th decoder at >mestep t / φ makes encoder & decoder states compa>blewith the aRen>on mechanism / f_adp makes context vector compa>ble with the decoder→ all these transforma>ons to support diﬀerent types of encoders/decoders for diﬀerent languages!
(Firat et al., 2016)
Nikolaos Pappas

100/111

Mul>-way, Mul>lingual NMT
Consistent improvements for lowresource languages• the lower the training data thebigger the improvement• In large-scale transla8on improves
only transla8on to English• hypothesis: EN appears always assource or target language for all
pairs → beRer decoder ?
•(Firat et al., 2016)
Nikolaos Pappas

101/111Google’s Neural MachineTransla>on System• An encoder, a decoder and an aRen>on network• 8-layer deep with residual connec>ons• Reﬁnement with Reinforcement Learning• Sub-word units…and more

(Wu et al., 2016)
Nikolaos Pappas

102/111Google’s Neural MachineTransla>on System•

EN->FR training took 6 days on 96GPUS !!!! and 3 more days for reﬁnement...

(Wu et al., 2016)
Nikolaos Pappas

103/111

Convolu>onal Encoder-Decoder• Outperformed GNMT and was more

eﬃcient in terms of speed, but• Lacks long-term memory• Requires me>culous ini>aliza>on
schemes and careful normaliza>on• Requires posi>onal embeddings• Requires more depth (15 layers)(Gehring et al., 2017)
Nikolaos Pappas

104/111

Self-ARen>on or Transformer
Networks•

Encode/Decode input w.o. using CNNs or LSTMs• Lower training cost but lack long-term memory• Stacked self-aRen>on with mul>ple heads•

To capture sequence informa>on it uses
posi>onal embeddings (sinusoids)

Nikolaos Pappas

105/111

Self-ARen>on or Transformer
Networks
Head 1

Head 2

Nikolaos Pappas

106/111

Best of both worlds: LSTMs &Transformer Networks•

Transformer network uses techniquesthat other models do not use• Combine strengths of both worlds• CNNs fall behind in BLEU andconvergence speed(Chen & Firat & Bapna et al., 2018)
Nikolaos Pappas

107/111

Outline of the talk
1. Introduc>on and Mo>va>on
2. Word Representa>on Learning• Seman>c similarity• Tradi>onal and recent approaches• Intrinsic and extrinsic evalua>on
3. Word Sequence Modeling• Essen>als: RNNs, ARen>on, DL tricks• Text Classiﬁca>on• Machine Transla>on
4. Conclusion and Discussion
Nikolaos Pappas

108/111

Conclusion•

Deep learning for NLP has ﬂourished the last couple of years➡ ARen>on mechanism became popular even outside NLP➡ Companies heavily use NLP e.g. machine transla>on•

Although aRen>on-based models can get us far➡ Linguis>c structure can get us even further by constraining
models and crea>ng useful induc>ve biases➡ Inspec>ng and analyzing the learned structures can help us
gain insights about language➡ There is s>ll a lot of work to be done on weakly-supervisedand unsupervised learning

Nikolaos Pappas

109/111

Future Challenges• Transferring knowledge across domains, languages

and diﬀerent outputs• Contextualizing word representa>ons• Generalizing on unseen examples• Learning from very few examples• Summariza>on / Entailment / Reasoning
Nikolaos Pappas

110/111

Discussion / Coding Session• Learning word embeddings

hRps://www.tensorﬂow.org/tutorials/word2vec
hRps://nlp.stanford.edu/projects/glove/• Classiﬁca>on using pre-trained word embeddings

hRps://blog.keras.io/using-pre-trained-wordembeddings-in-a-keras-model.html➡ https://www.tensorflow.org/install/➡ https://keras.io/#installation➡ https://keras.io/backend/
Nikolaos Pappas

111/111