CS11-747 Neural Networks for NLP

Sentence andContextualised Word
Representations
Graham NeubigSite
https://phontron.com/class/nn4nlp2019/(w/ slides by Antonis Anastasopoulos)

Sentence Representations•

We can create a vector or sequence of vectors
from a sentence
this is an example
this is an example
Obligatory Quote!
“You can’t cram the meaning of a whole %&!$ing
sentence into a single $&!*ing vector!”— Ray Mooney

Goal for Today•

Briefly Introduce tasks, datasets and methods•

Introduce different training objectives•

Talk about multitask/transfer learningTasks Using Sentence
Representations

Where would we need/use
Sentence Representations?
•Sentence Classification•

Paraphrase Identification•

Semantic Similarity•

Entailment•

RetrievalSentence Classification•

Classify sentences according to various traits•

Topic, sentiment, subjectivity/objectivity, etc.

I hate this movie

very good
good
neutralbadvery bad

I love this movie

very good
good
neutralbadvery badParaphrase Identification(Dolan and Brockett 2005)•Identify whether A and B mean the same thing

Charles O. Prince, 53, was named as Mr. Weill’s successor.
Mr. Weill’s longtime confidant, Charles O. Prince, 53, wasnamed as his successor.
•Note: exactly the same thing is too restrictive, souse a loose sense of similarity

Semantic Similarity/Relatedness(Marelli et al. 2014)•

Do two sentences mean something similar?

•Like paraphrase identification, but with shades of gray.

Textual Entailment(Dagan et al. 2006, Marelli et al. 2014)•

Entailment: if A is true, then B is true (c.f. paraphrase,
where opposite is also true)••

Contradiction: if A is true, then B is not true••

The woman bought a sandwich for lunch→ The woman bought lunch

The woman bought a sandwich for lunch→ The woman did not buy a sandwich

Neutral: cannot say either of the above•

The woman bought a sandwich for lunch→ The woman bought a sandwich for dinnerModel for Sentence Pair
Processing•

Calculate vector representation•Feed vector representation into classifier
this is an example
classifier

yes/nothis is another example
How do we get such a representation?

Multi-task LearningOverview

Types of Learning•

Multi-task learning is a general term for training on
multiple tasks•

Transfer learning is a type of multi-task learningwhere we only really care about one of the tasks•

Domain adaptation is a type of transfer learning,where the output is the same, but we want tohandle different topics or genres, etc.

Plethora of Tasks in NLP••In NLP, there are a plethora of tasks, each requiring
different varieties of data•

Only text: e.g. language modeling•

Naturally occurring data: e.g. machinetranslation•

Hand-labeled data: e.g. most analysis tasks

And each in many languages, many domains!

Rule of Thumb 1:
Multitask to Increase Data•

Perform multi-tasking when one of your two tasks has
many fewer data•

General domain → specific domain(e.g. web text → medical text)•

High-resourced language → low-resourcedlanguage(e.g. English → Telugu)•

Plain text → labeled text
(e.g. LM -> parser)

Rule of Thumb 2:•

Perform multi-tasking when your tasks are related•

e.g. predicting eye gaze and summarization(Klerke et al. 2016)

Standard Multi-task
Learning•

Train representations to do well on multiple tasks atonceLMthis is an example

EncoderTagging•

In general, as simple as randomly choosing minibatch from oneof multiple tasks•

Many many examples, starting with Collobert and Weston (2011)Pre-training•

First train on one task, then train on anotherthis is an example

Encoder

Translation

Initialize
this is an example

Encoder

Tagging•

Widely used in word embeddings (Turian et al. 2010)•Also pre-training sentence encoders or contextualized
word representations (Dai et al. 2015, Melamud et al.
2016)

Thinking about Multi-tasking,and Pre-trained Representations•

Many methods have names like SkipThought, ParaNMT,CoVe, ELMo, BERT along with pre-trained models•

These often refer to a combination of••Model: The underlying neural network architecture•

Training Objective: What objective is used to pretrain•

Data: What data the authors chose to use to train themodel

Remember that these are often conflated (and don'tneed to be)!

End-to-end vs. Pre-training••

For any model, we can always use an end-to-end
training objective•

Problem: paucity of training data•

Problem: weak feedback from end of sentence
only for text classification, etc.

Often better to pre-train sentence embeddings onother task, then use or fine tune on target task

Training Sentence
Representations

General Model Overview
I

hatethismovie

lookuplookuplookuplookupscoressome complicated function
to extract combination
featuresprobs
softmax

Language Model Transfer(Dai and Le 2015)•Model: LSTM•

Objective: Language modeling objective•

Data: Classification data itself, or Amazon
reviews•

Downstream: On text classification, initialize
weights and continue training

Unidirectional Training + Transformer(OpenAI GPT)
(Radford et al. 2018)•

Model: Masked self-attention•

Objective: Predict the next word left->right•

Data: BooksCorpus

Downstream: Some task fine-tuning, other tasks
additional multi-sentence training

Auto-encoder Transfer(Dai and Le 2015)•

Model: LSTM•

Objective: From single sentence vector, reconstruct the sentence•

Data: Classification data itself, or Amazonreviews•Downstream: On text classification, initialize
weights and continue training

Context Prediction Transfer(Skip-thought Vectors)(Kiros et al. 2015)••

Model: LSTM•

Objective: Predict the surrounding sentences•

Data: Books, important because of context

Downstream Usage: Train logistic regression on [|u-v|; u*v] (component-wise)Paraphrase ID Transfer(Wieting et al. 2015)•

Model: Try many different ones•

Objective: Predict whether two phrases areparaphrases or not from•

Data: Paraphrase database (http://paraphrase.org), created from bilingual data
Downstream Usage: Sentence similarity,classification, etc.

••

Result: Interestingly, LSTMs work well on indomain data, but word averaging generalizes
betterLarge Scale Paraphrase Data
(ParaNMT-50MT)(Wieting and Gimpel 2018)•

Automatic construction of large paraphrase DB•

Get large parallel corpus (English-Czech)•

Translate the Czech side using a SOTA NMT system•

Get automated score and annotate a sample•

Corpus is huge but includes noise, 50M sentences(about 30M are high quality)•Trained representations work quite well and generalize

Entailment Transfer(InferSent)(Conneau et al. 2017)•Previous objectives use no human labels, but whatif:•

Objective: supervised training for a task such asentailment learn generalizable embeddings?
•Task is more difficult and requires capturing
nuance → yes?, or data is much smaller → no?

•

Model: Bi-LSTM + max pooling•

Data: Stanford NLI, MultiNLI•

Results: Tends to be better than unsupervised
objectives such as SkipThought

Contextualized Word
Representations

Contextualized Word
Representations•Instead of one vector per sentence, one vector perword!
this is an example
classifier

this is another example

How to train this representation?

yes/noCentral Word Prediction Objective(context2vec)(Melamud et al. 2016)•

Model: Bi-directional
LSTM•

Objective: Predict theword given context•

Data: 2B word ukWaC
corpus•Downstream: use vectors
for sentence completion,word sense
disambiguation, etc.

Machine Translation Objective(CoVe)(McMann et al. 2017)•

Model: Multi-layer bi-directional LSTM•

Objective: Train attentional encoder-decoder•

Data: 7M English-German sentence pairs

Downstream: Use bi-attention network over sentencepairs for classification

Bi-directional Language Modeling Objective(ELMo)(Peters et al. 2018)•

Model: Multi-layer bi-directional LSTM•

Objective: Predict the next word left->right, next
word right->left independently•

Data: 1B word benchmark LM dataset

Downstream: Finetune the weights of the linear
combination of layers on the downstream task

Masked Word Prediction(BERT)(Devlin et al. 2018)•

Model: Multi-layer self-attention. Input sentenceor pair, w/ [CLS] token, subword representation•

Objective: Masked word prediction + nextsentence prediction•

Data: BooksCorpus + English Wikipedia

Masked Word Prediction(Devlin et al. 2018)
1. predict a masked word••

80%: substitute input word with [MASK]•

10%: substitute input word with random
word•

10%: no change

Like context2vec, but better suited for
multi-layer self attention

Consecutive Sentence Prediction(Devlin et al. 2018)

1. classify two sentences as consecutive ornot:•

50% of training data (from OpenBooks) is"consecutive"

Using BERT
with pre-training/finetuning•

Use the pre-trained model as the first “layer” of the final
model, then train on the desired task

Using BERT
for Representations•

Use the pre-trained model to obtain contextualised word
representations for the input[visualization from The Illustrated BERT: https://jalammar.github.io/illustrated-bert/]

Which Method is Better?

Which Model?
•Not very extensive comparison...

•Wieting et al. (2015) find that simple word
averaging is more robust out-of-domain•

Devlin et al. (2018) compare unidirectional and bidirectional transformer, but no comparison to LSTM
like ELMo (for performance reasons?)

Which Training Objective?
•Not very extensive comparison...

•Zhang and Bowman (2018) control for training
data, and find that bi-directional LM seems better
than MT encoder•

Devlin et al. (2018) find next-sentence prediction
objective good compliment to LM objective

Which Data?
•

Not very extensive comparison...

•Zhang and Bowman (2018) find that more data isprobably better, but results preliminary.

•Data with context is probably essential.

Questions?