Learning Transferable Visual Models From
Natural Language Supervision
Liwei Che
03/27/2023

Background
Although deep learning has revolutionized computer vision, current
approaches have several major problems:• Typical vision datasets are labor intensive and costly to create while
teaching only a narrow set of visual concepts (Expensive);• Standard vision models are good at one task and one task only, andrequire significant effort to adapt to a new task (Hard to transfer);• Models that perform well on benchmarks have disappointingly poor
performance on stress tests (Bad at generalization).

2Contribution
CLIP (Contrastive Language–Image Pre-training) mitigates the gap
between source domain and other domains, benchmarking
performance and wild test with a cheap and computational efficientway. --- Amazing zero-shot performance!
1. Collected 400 million image-text pairs from Internet as training
dataset2. Get the logits via the cosine similarities between the image-text
pairs (metric learning)3

Zero-shot Performance of CLIP

4

CLIP - Overview

5

CLIP – Approach: Natural Language Supervision
Learning perception from supervision contained in natural language,a.k.a, which text is paired with the image input.
Strengths:
1. Needless of crowd-sourced labeling for image classification.
2. Not only learn a representation but also connects thatrepresentation to language which enables flexible zero-shottransfer.

6

CLIP – Approach: Creating a Sufficiently Large Dataset
Instead of using the existing well-labeled datasets, the authors
collected over 400 million data pair that are available publicly on theinternet. Make it a dataset named WebImageText (WIT).
To attempt to cover as broad a set of visual concepts as possible, wesearch for (image, text) pairs as part of the construction process
whose text includes one of a set of 500,000 queries. Class balance
the results by including up to 20,000 (image, text) pairs per query.

7

CLIP – Approach: Selecting an Efficient Pre-Training Method
Training Task: Given N image-text pairs, CLIP is trained to
predict which of the N × N possible (image, text) pairings
across a batch actually occurred.

Method: Learn a multi-modal embedding space by jointly
training an image encoder and text encoder tomaximize the cosine similarity of the image and textembeddings of the N real pairs in the batch

while minimizing the cosine similarity of the embeddings ofthe 𝑁 2 − 𝑁 incorrect pairings.

8

CLIP – Approach: Selecting an Efficient Pre-Training Method• Pseudo code9

CLIP – Approach: Choosing and Scaling a Model
Architectures for the image encoder:
1. ResNet-50
2. Vision Transformer
Architectures for the text encoder:
Basic Transformer
CLIP’s performance to be less sensitive to the capacity of the text
encoder.

10

CLIP – Approach: Training
Image encoder:
ResNet-50x4, ResNet-50x16, ResNet-50x64, ResNet-101, ViT-B/32, ViT-L/14.
Epochs: 32
Optimizer: AdamW
Initialization: Grid searchMini-batch: 32768 (the larger the better for contrastive learning)
Training time:
ResNet-50x64: 592 V100 GPUs for 18 days
ViT-L/14: 256 V100 GPUs for 12 days

11

CLIP – Experiment: Zero-shot Transfer
Prompt Engineering:
1. Mismatch between the name and theimage.
2. Polysemy
3. the prompt template “A photo of a {label}.”improves the baseline of only using thelabel text.

12

CLIP – Experiment: Zero-shot Transfer

13

CLIP – Experiment: Zero-shot Transfer

14

CLIP – Experiment: Representation Learning

15

Zero-shot CLIP is much more robust to distribution shift thanstandard ImageNet models.

16

17

CLIP - Limitations
1. CLIP struggles on more abstract or systematic tasks such ascounting the number of objects in an image and on more
complex tasks such as predicting how close the nearest car is ina photo.
2. CLIP also still has poor generalization to images not covered in its
pre-training dataset.
3. CLIP’s zero-shot classifiers can be sensitive to wording orphrasing and sometimes require trial and error “promptengineering” to perform well.

18

Broader impacts
1 . Potential social impacts, such as privacy violations and biases indecision-making.
2. Future application on a wide range of tasks.

Reflection
1. Nature Language Supervision is a powerful high-dimensional supervision
signal, which can reduce ambiguity and be easily transferable.
2. Larger queries set will further enhance the model performance.
3. CLIP transforms the classification task into retrieval task, which makes it beefficient.

19Thanks for listening
Any Questions?