Vision-Language Pretraining:
Current Trends and the Future
Aishwarya Agrawal & Damien Teney & Aida Nematzadeh

ACL
22 May 2022

Please check our website for our final slide deck:
https://vlp-tutorial-acl2022.github.io/2

Why Multimodal Pretraining?
The ability to ground language to vision–multimodal pretraining–
is a fundamental aspect of both language & vision.

Q: What are the people waiting for?
A: bus3[Images are taken from https://vision-explorer.allenai.org/visual_question ]Why Multimodal Pretraining?
Train once, use multiple times. Multimodal features are usefulacross a range of multimodal tasks and applications.

Q: What are the people waiting for?
A: bus4[Images are taken from https://vision-explorer.allenai.org/visual_question ]Goal: Give an overview of ingredients needed for working on
multimodal problems (particularly vision and language). Also,
discuss some of the open problems.
Plan for today:● Before the Pretraining Era (Aishwarya,● Vision-Language Pretraining (Aida,● Beyond Statistical Learning (Damien,min)min)min)

5

Vision-Language Pretraining
Aida Nematzadeh6Success of Pretraining in NLP

predict the next word

Performance gain is due to architecture innovations & larger
data. [Peters et al., ; Howard & Ruder, ; Devlin et al., ; Radford et al., ; Raffel et al., ; Rae et al.
]

7[Images are from http://jalammar.github.io/illustrated-bert/]

Similar Models for Multimodal Pretraining?
“The scenic routethrough mountain
ranges includes
these unbelievably
colouredmountains.

ViLBERTpredict the next word

Other objectives?

Dataset: image-text pairs wherea given text describes its image.
Model: attention mechanisms
over both image and text;
preprocessing images to “visualtokens”.
Objective: loss functions speciﬁcto the image modality andimage-text pairs.
8[Images are taken from http://jalammar.github.io/illustrated-bert/ , https://ai.googleblog.com///conceptual-captions-new-dataset-and.html, https://visualqa.org/]Multimodal Pretraining: How it Started
VideoBert
Sun et al. ICCVBERTDevlin et al. NAACL(arxiv)

VilBERTLu et al. NeurIPS.

.

LXMertTan et al. EMNLP

CBTSun et al. Arxiv

UNITER
Chen et al. ECCV.

.

.

Models share the same “backbone” with slight differences in loss
design, preprocessing, etc.
They achieve the state-of-the-art results in a range of tasks.
9Multimodal Transformers (Joint Encoders)
Image
Modelling Loss

Language
Modelling Loss

Image-Language
Matching Loss

MULTIMODAL TRANSFORMER

A

dog

runs

inthe

<MASK>

Thanks Lisa Anne Hendricks for sharing slides on MMT.

10

Multimodal Transformers (Joint Encoders)
Image
Modelling Loss

Language
Modelling Loss

Image-Language
Matching Loss

MULTIMODAL TRANSFORMER

A

dog

runs

inBERTthe<MASK>

11Multimodal Transformers (Joint Encoders)
Image
Modelling Loss

Language
Modelling Loss

Image-Language
Matching Loss

MULTIMODAL TRANSFORMER

A

dog

visual “word”:
bounding boxrunsinthe<MASK>VOLTA: a general framework (withEnglish and multilingual MMT). 12

What Contributes to these Models’ Success?
Are results due to advances in the architecture or large pretraining
datasets? Architecture
Are the “adopted” losses from language models good enough? No, weneed better image modeling losses.
Is the cross-talk between modalities (via attention) important? Yes,
multimodal attention is important.
What makes a good pretraining dataset? The level of noise andlanguage matter.
13

Evaluation: Zero-Shot Image Retrieval
Zero-shot image retrieval directly evaluates the goodness of pretrained
representations.
Image Retrieval (IR)“Grey haired man in blackand yellow tie.”

14

Typical Loss Functions
Language/image modeling: masked language/region modeling
Image–language matching: binary classiﬁcation or contrastive
formulation[Taken from ALIGN]
15Are All Losses Needed?

[Hendricks et al. TACL 2021]

R@ZeroshotFlickr

No LanguageModeling LossNo Image
Modeling Loss

All Loses.

41.1

40.7

With the right
hyper-parameters,
image modeling loss is
not needed.
Vision-and-Language
orVision-for-Language?
[Frank et al,]16

Different Attention Mechanisms

Each modality
attends to bothmodalities.

Each modality attends
only to the othermodality (twoasymmetricattentions).

[Hendricks et al. TACL 2021]

Only one modality(e.g., language)attends to the othermodality (e.g., image).

multimodal attention
17Different Attention Mechanisms

Similarperformance

R@Zeroshot
Flickr[Hendricks et al. TACL 2021]

mergedcoattention

asymmetric(language)

asymmetric(image)modality
speciﬁc.

41.9

.

.

.

18

Different Attention Mechanisms

R@Zeroshot
Flickrmergedcoattention

.

41.9

>

[Hendricks et al. TACL 2021]

asymmetric(language)

asymmetric(image)

.

.

19

Multimodal Attention > Depth / Size
multimodal layers &[Hendricks et al. TACL 2021]

attention heads

R@

coattention

asymmetric(language)

asymmetric(image)coattention
withmultimodal
layer

Zeroshot
Flickr

41.9

.

.

.

coattention
withattention
heads.

Depth and number of parameters alone are not enough.
20

What Contributes to these Models’ Success?
Are results due to advances in the architecture or large pretraining
datasets? Architecture
Are the “adopted” losses from language models good enough? No, weneed better image modeling losses.
Is the cross-talk between modalities (via attention) important? Yes,
multimodal attention is important.
What makes a good pretraining dataset? The level of noise andlanguage matter.
21

Pretraining Datasets
MSCOCOMSCOCO/OI
Narratives

VisualGenomeConceptualCaptions

SBU Captions“The two people arewalking down thebeach.”“In this image we can see a bridgeand sea. In the background, wecan see trees and the sky. We
can see so many people on thebridge. At the bottom of theimage, we can see two people.
We can see stairs in the rightbottom of the image …”

small round yellow
frisbee, man has caston his arm, concrete
trail path in the park,
man wearing black
sunglasses“The scenic route
through mountain
ranges includes these
unbelievably
coloured mountains.

“King Arthur’sbeheading rock right on thesidewalk in the
middle of town”.

manually annotatedfrom “the wild”

Noisier image-text correspondence but larger

22

Dataset Considerations: Size

23

Dataset Considerations: Language
COCO style caption: “Single blackdog sitting on the grass”
Narratives style caption: “The dogis black and brown. The collar is red.
… The dog is on the grass. ...”
Genome style caption: “Black dog”

24

Dataset Considerations: Noise“Single black dog sitting on thegrass”“A person takes a dog on a walknear the river.”25

Image Retrieval: Language Similarity Mattersmore images
equal # of images

Performance is not directly
correlated with the numberof images.
Language similarity inpretraining & test (measuredby perplexity) explains thedifference in the results.

sameimages

pretraining datasets

Hendricks et al. “Data, Architecture, or Losses: What Contributes
Most to Multimodal Transformer Success?” TACL 2021.
26

Image Retrieval: Noise Mattersmore images

pretraining datasets

SBU is larger than Open
Images and has lower
perplexity, but is still worse.
However, SBU has more noise,meaning the language doesnot always describe theimage content.
Hendricks et al. “Data, Architecture, or Losses: What Contributes
Most to Multimodal Transformer Success?” TACL 2021.
27

Image Retrieval: Dataset Sampling Matters

Combining datasets does lead
to better results, but how wesample from combined
datasets matter.

ConceptualCaptions

Combined

Combined:
Sample Evenly
from Datasets

MSCOCO is a good dataset for
pretraining; sampling method
which weights MSCOCO
images higher does better.
Hendricks et al. “Data, Architecture, or Losses: What Contributes
Most to Multimodal Transformer Success?” TACL 2021.
28

“Best” Dataset is Task Dependent

Best datasets are different for IR (Conceptual Captions isbest) and VQA (VG is best)29

What Contributes to these Models’ Success?
Are results due to advances in the architecture or large pretraining
datasets? Architecture
Are the “adopted” losses from language models good enough? No, weneed better image modeling losses.
Is the cross-talk between modalities (via attention) important? Yes,
multimodal attention is important.
What makes a good pretraining dataset? The level of noise andlanguage matter.

We have released our pretrained models!

30

Language Encoders

Joint Encoders
Image Modelling
Loss

Language
Modelling Loss

Image-language
Matching LossLANGUAGE ENCODERLANGUAGE ENCODER

MULTIMODAL TRANSFORMER
MASKA

dog

runs

inthe

<MASK>

dog

MASK

thein

EncoderDecoders

A

dog

MASK

dogruns

in

MASk

in

the

the

LANGUAGE DECODER

Dual Encoders

Image-language
Matching Loss
SOS

A

dog

runs

in

LANGUAGE ENCODER

IMAGE ENCODER

MULTIMODAL ENCODER

A

dogruns

inthe<MASK>
captionregionV

31

Language Encoders

Joint Encoders
Image Modelling
Loss

LanguageModelling Loss

Image-language
Matching LossLANGUAGE ENCODERLANGUAGE ENCODER

MULTIMODAL TRANSFORMER
MASKA

dog

runs

inthe

<MASK>

dog

MASKthein

EncoderDecoders

A

dog

MASKdogruns

inMASkinthethe

LANGUAGE DECODER

Dual Encoders

Image-language
Matching Loss
SOS

A

dog

runs

in

LANGUAGE ENCODER

IMAGE ENCODER

MULTIMODAL ENCODER

A

dog

runs

inthe

<MASK>
captionregionV

32

Dual Encoders
Two separate encoders for image and language modalities; nocross-talk between the two. [Weston et al., ; Frome et al., ; Kiros et al., ]Very successful for retrieval tasks [Chowdhury et al.,; Miech, Alayrac, et al.

]

Image-language Matching LossLANGUAGE ENCODER

IMAGE ENCODER

Adogrunsinthe<MASK>

33

Recent Large-Scale Dual Encoders
CLIP [Radford et al,]and ALIGN [Jia et al,][Radford et al, 2021; Jia et al, 2021]: Larger models & datasets

How to collect large-scale datasets?

34

Pretraining Datasets: Refresher
MSCOCOMSCOCO/OI
Narratives

VisualGenomeConceptual
Captions

SBU Captions“The two people arewalking down thebeach.”“In this image we can see a bridgeand sea. In the background, wecan see trees and the sky. We
can see so many people on thebridge. At the bottom of theimage, we can see two people.
We can see stairs in the rightbottom of the image …”

small round yellow
frisbee, man has caston his arm, concrete
trail path in the park,
man wearing black
sunglasses“The scenic route
through mountain
ranges includes these
unbelievably
coloured mountains.

“King Arthur’sbeheading rock right on thesidewalk in themiddle of town”.

manually annotatedfrom “the wild”

Noisier image-text correspondence but larger

35

Recent Large-Scale Dual Encoders
CLIP [Radford et al,]and ALIGN [Jia et al,][Radford et al, 2021; Jia et al, 2021]: Larger models & datasets

How to collect large-scale datasets?
● ALIGN removes any ﬁltering to increase the size ( . B) → noisier.
○ The same pipeline as Conceptual Captions (CC).
● CLIP uses heuristics to clean the data (M).
Tradeoff between data size & noise: CC ( M) > ALIGN ( M/ M) on
MSCOCO retrieval. [Jia et al, ]36

Dataset Considerations: Size37

Recent Large-Scale Dual Encoders
CLIP [Radford et al,]and ALIGN [Jia et al,][Radford et al, 2021; Jia et al, 2021]: Larger models & datasets

Use similar contrastive losses; ALIGN uses label smoothing thatcan be helpful with dataset noise.
Perform zero-shot image classiﬁcation as a image-text retrieval
task.

38

Qualitative Examples from ALIGN
Image retrieval with image +/- text queries

39Qualitative Examples from ALIGN
Image retrieval with ﬁne-grained queries.

40

Dual Encoders Vs. Multimodal Transformers
Dual encoders are easier to scale since they can reuse
image/language features across pairs.
But they are not sample efﬁcient.
UNITERsame data

MMTBOW-DE

same performance

CLIP

ALIGN
BOW-DE: [Miech & Alayrac et al. CVPR
MMT: [Hendricks et al. TACL]UNITER: [Chen et al. ECCV]
CLIP: [Radford et al. Arxiv]
ALIGN: [Jia et al. Arxiv]]

41

Language Encoders

Joint Encoders
Image Modelling
Loss

Language
Modelling Loss

Image-language
Matching LossLANGUAGE ENCODERLANGUAGE ENCODER

MULTIMODAL TRANSFORMER
MASKA

dogruns

inthe<MASK>dog

MASKthein

EncoderDecoders

A

dog

MASKdogruns

inMASkinthethe

LANGUAGE DECODER

Dual Encoders

Image-language
Matching Loss
SOS

A

dogrunsin

LANGUAGE ENCODER

IMAGE ENCODER

MULTIMODAL ENCODER

A

dogruns

inthe<MASK>
captionregionV42

Encoder-Decoders

Language
Modeling Loss

An image captioning setup:● Replace the image encoder with amultimodal one● Virtex, VL-BART(T ), SimVLM [Desai &Johnson,; Cho et al,; Wang et al,]Uses language as supervision for visionor multimodal pretraining.

Adogrunsin

LANGUAGE DECODER

SOS

A

dogrunsin

MULTIMODAL
ENCODER
IMAGE ENCODERcaption

Requires less images than the imageclassiﬁcation setting.

theregion

V43

Encoder-Decoders
An image captioning setup:● Replace the image encoder with amultimodal one● Virtex, VL-BART(T ), SimVLM [Desai &Johnson,; Cho et al,; Wanget al,]Language
Modeling LossA

dogruns

inthe

LANGUAGE DECODER

SOSA

dogrunsin

MULTIMODAL
ENCODER
IMAGE ENCODER

Uses language as supervision for vision ormultimodal pretraining.
Check out LNLS workshop on Thursday!

captionregion

V44

VL-BART (VL-T5)[Cho et al, 2021]
Uniﬁes tasks as text generation (w/ task-speciﬁc preﬁxes).
● Parameters for each task are separately optimized.
Builds on a pretrained language model (BART or T ).
Trains on ~ . M image–text pairs.

45

SimVLM [Wang et al, 2022]
Uniﬁes tasks as text generation.
Removes object detection supervision.
Trains on large-scale noisy image-text data (ALIGN).

46

Combining Frozen (Pretrained) Models
Given the cost of pretraining large models, can we reuse andcombine existing vision and/or language models?
● Frozen [Tsimpoukelli et al,]● MAGMA [Eichenberg et al,]
● Flamingo [Alayrac et al,]But, need to map different feature spaces (e.g., image features
to the language ones) → Train adapter layers
48Combining Frozen (Pretrained) Models
Flamingo [Alayrac et al,]49

Moving Towards Generative Models
Pros.
● Uniﬁed modeling of vision–language tasks● Better out-of-distribution generalization in VQA [Agrawal et al,]Cons.
● How to accurately evaluate the generated text? [Agrawal et al,]
● Inheriting existing biases of pretrained language (and vision)models.
50

Qualitative Examples from Flamingo
Language model bias can result in hallucinations.

51

Qualitative Examples from Flamingo
Mistakes in spatial understanding but correct language use.

[Examples from JB Alayrac]52

Language Encoders

Joint Encoders
Image Modelling
Loss

Language
Modelling Loss

Image-language
Matching LossLANGUAGE ENCODERLANGUAGE ENCODER

MULTIMODAL TRANSFORMER
MASKA

dog

runs

inthe

<MASK>

dog

MASKthein

EncoderDecoders

Adog

MASKdogrunsinMASkinthethe

LANGUAGE DECODER

Dual Encoders

Image-language
Matching LossSOSA

dog

runs

in

LANGUAGE ENCODER

IMAGE ENCODER

MULTIMODAL ENCODER

A

dog

runs

inthe

<MASK>
captionregionV53

Language Encoders
A language modeling setup:● Vokenization: map eachlanguage token to a visual
token (voken) [Tan & Bansal,]

Uses vision as supervision for
language pretraining.

Masked Language
Modeling

Voken Classiﬁcation

runs

vokens: image ids

A

LANGUAGE ENCODER

MASKdog

MASKinthe

LANGUAGE ENCODER

MASKdog

MASkinthe54Summary of Different Approaches
How to evaluate pretrained models?
● Use task-speciﬁc heads for each downstream task (e.g.,ViLBERT, LXMERT, UNITER, OSCAR, VinVL).
● Treat all downstream tasks as language generation with notask-speciﬁc head (e.g., VL-T , VL-BART, SimVLM).

55

Summary of Different Approaches
How are the features used (other than vision-language tasks)?
● In vision tasks (e.g., VirTex, CLIP, ALIGN)● In language tasks, including multilingual data (e.g.,Vokenization, M P, VL-T , SimVLM)

56

Towards a Better Evaluation of Pretrained Models
Performance after ﬁne-tuning depends on the the size ofﬁne-tuning data and other experimental set-up [Yogatama et al.,].

Recent work has shifted focus to few- and zero-shot evaluation.
Other approaches● Evaluate for out-of-distribution generalization (transfer)● Probe for certain capabilities (e.g., verb understanding)See my talk on evaluation if you are interested!

57

Answering Questions from Blind People

Q: What are the people waiting for?
A: bus

Q: What is this?
A:euros.

VizWiz is a benchmark curated
from visually-impaired users.

Evaluate in a Transfer Setting [Agrawal et al,]Fine-tune a multimodal transformer on one dataset (VQAv ),test on another one (VizWiz): we observe ~26 drop in accuracy.

Q: What are the people waiting for?
A: bus

Q: What is this?
A:euros.

Winoground: Visio-Linguistic Compositionality[Thrush et al,]60

What Image Retrieval Tests
Order images with respect to their match to a sentence.

<<A person is riding a horse.
SubjectVerbObjectDoes not require ﬁne-grained multimodal understanding.

What SVO-Probes Tests [Hendricks et al., Findings of ACL]

A person is riding a horse❌

✔

Correctly classify both the positive & negative examples.
We have released our dataset! 🎉🎉Do MMTs Have Fine-grained Verb Understanding?
A woman lying with a dog❌

✔
ChanceWhy?

Do MMTs Have Fine-grained Verb Understanding?
A animal lays in the grass❌

✔
ChanceWhy?

Do MMTs Have Fine-grained Verb Understanding?
A woman jogs on the beach❌

✔
ChanceWhy?

Do MMTs Have Fine-grained Verb Understanding?

Overall MMT
performance. -lots of room for
improvement!

ChanceWhy?

To build stronger models, we need tobetter evaluate them first.
Thanks for listening!

Questions?
67