Flamingo: a Visual Language🦩Model for Few-Shot Learning
CSE 587 - Presentation

Pouria Mahdavinia, March 27

Task• Few shot in-context learning for Vision-Language tasks• Classi cation• Captioning• Visual question answeringfi• Visual dialogue

Task

Motivation
GPT3• Large-scale generative LMs -> Good few shot learners• Only work with text data

Motivation
CLIPCan not generate text -> Not good for open-endedtasks (captioning or VQA)

Challenges of Visual LMs
1. Combining pre-trained large-scale LMs (trained only on text), with Vision encoders• High computation cost of training from scratch• Fusing the visual feature to embedded text
2. Accommodating both Image/Video input with arbitrary length in a computationallye cient manner
3. Need for huge dataset• The size of image-text pair datasets like CLIP and ALIGN might not be enoughffi

for good few-shot learning performanceApproachFlamingo architecture overview

Approach
Vision encoder• F6 Normalizer-Free ResNet (NFNet) -> Computation e ciency• Contrastive pre-training similar to CLIP
• Deployed BERT as text-encoder, and NFNet for vision encoder• Simplify the CLIP, by using global average pooling instead of global attention
pooling• Trained on ALIGN (1.8 billion image-text pair), and LTIP (312 million image-ffitext pair) using accumulation combination strategy

Approach
Perceiver Resampler: from varying-size large feature maps to few visual tokens

Approach
Gated XATTN-Dense layers

Why Gated? Keeping LM
intact at initialization ->Improve stability andperformanceApproachInterleaved visual data and text support
Selective cross attention
Masked cross attention

Cute pics of my pets!

My puppy sitting in thegrass.

K=V=[X]

Q00000000111 11111111222 2222222<BOS> Cute pics of my pets!<EOC><image>My puppy sitting in the grass. <EOC><image>My cat looking very dignified.<EOC>

Perceiver
Resampler

Perceiver
Resampler

VisionEncoder

VisionEncoder

Image 1

Image 2

tokenization
My cat looking verydignified.

Input webpage<BOS>Cute pics of my pets!<EOC><image>My puppy sitting in the grass.<EOC><image> My cat looking very dignified.<EOC>

Processed text: <image> tags are inserted and special tokens are added

Datasets

This is animage of aflamingo.

A kiddoing akickflip.

Welcometo mywebsite!

This is a
picture ofmy dog.

This is a
picture ofmy cat.

Image-Text Pairs dataset

Video-Text Pairs dataset

Multi-Modal Massive Web (M3W) dataset[N=1, T=1, H, W, C][N=1, T>1, H, W, C][N>1, T=1, H, W, C]

1. MultiModel Massive Web (M3W)• Collected from 43 million webpages• Extract rst ve images, and randomly sample 256 Token subsequence• 185M images, and 182 GB of text
2. Image/Video-Text pairs data

• ALIGN (1.8B image-text pairs) + LTIP (312M image-text pairs, betterfifiquality) + VTP (27M short videos, around 22 seconds each)Training strategies• Training objective: Weighted sum on di erent datasets minimizing the empiricalnegative log likelihood• Optimizer -> AdamW• Learning rate schedule: Linear warmup, and then at LRflff• Mixing weights: M3W -> 1 , LTIP -> 0.2 , ALIGN -> 0.2 , VTP -> 0.03

Task adaptation with few-shot in-context learning
Multimodal promptFlamingo models

Evaluation
Overview of Flamingo performanceAblation studies

Limitation and Future work
• Performance gap on classi cation task comparing to contrastive models such as CLIP,
it would be nice future work to bridge this gap (I.e. Calibrate the prompt selection)• Inheriting the weakness of casual (auto-regressive) pre-trained LM (Replacing it withmore expressive bidirectional models)• Hallucinations and ungrounded guesses in open-ended visual question answeringfi• Adding additional modalities such as audio for improving the performance