VL‐BERT：Pretraining of Generic Visual‐
Linguistic Representations
Speaker: Gong, Qiqi

Basica Info.
• Conf.:ICLR 2020
• Affiliation: USTC (中科大), MSRA

Background Knowledges• V+L Tasks:• Visual contents• Language semantics• Cross‐modal alignments• Pretraining：• Pre‐training for Visual task: Pretrain classifier on ImageNet (since 2009)• Pre‐training for Linguistic task: Pretrain transformer on large scale corpus(语料库) for masked language modeling (MLM) (since 2017)• How to pretrain on V+L task?

Background Knowledges•*Introduction to MLM:
•Special tokens in NLP:•[CLS]: Begining of a sentence•[SEP]: Separation of sentences•[MASK]: Masking words with a probability•Aim: To predict masked words from given words•e.g.:
Input: My dog is [Mask]
Output: My dog is hairy

Background Knowledges• Downstream V&L Tasks:• Visual Question Answering (VQA):• Input:<Question, Answer([Mask]), Image>• Dataset: VQA 2.0• Train: 83k images, 444k questions• Val: 41k images, 214k questions• Pick corresponding answer from 3129 answers (BUTD,2018论文第9页)• Visual Commensense Reasoning (VCR):• Input: <Question,Answering,Image>• Dataset: VCR• Train: 80k images, 213k questions• Val: 10k images, 27k questions• Pick the right answer out of four ones and provide rationale explanation

Background Knowledges• Downstream Tasks:• Refe r r i n g E x p re s s i o n C o m p re h e n s i o n ( V i s u a lGrounding)• Input:<Query, Image>• Localize the object with given referring expression• Dataset: RefCOCO+• 141k expressions for 50k referred objects in 20k images• Split into 4 sets: training, val, two sets (one with multiplepeople, one with multiple objects)

Background Knowledges• Downstream Tasks:• Other Tasks
Image‐Text Retrieval(Dataset: Flicker30K)

Natural Language
Visual Reasoningfor Real (NVLR2)

Visual Entailment(Dataset: SNLI‐VE)

Basic Knowledges• Questions Conclusion:• How to design a model fusing both modalities?
• How to design a upstream V&L task?
• How to fine‐tune pretrained V&L network to adapt to downstream tasks?

Introduction• Motivation• Previous pratice is to combine base networks for CV and NLP respectively in atask‐spefic way• Overfitting when data for target is scarce (稀少的)• Benefit little from pre‐training• There lacks pre‐trained generic feature representations for V&L tasks• (Private Idea) Success of ViT

Introduction• Contribution• A single‐stream model for pretraining V&L representations• Pre‐train on both visual‐linguistic and text‐only datasets• Masking operation for raw images

VL‐BERT• Revisit BERT[bə(r)t] Model• Multi‐head Atten. ‐> Residual Connec. ‐> Feed‐forward ‐> Residual Connec.
• Characteristics:• Irrelevant to the order of input seq. (i.e. permutation(out) == Net(permutation(input)))• Position of an element is encoded in its own embedding features by sequence positionalembeddingVL‐BERT• Revisit BERT Model• Pretraining:• Masked language modeling (MLM)<==>
Incurred Loss:• Next Sentence Prediction• Two special elements: [CLS] [SEP]Loss:

VL‐BERT• Model Architectures• Take both visual and linguistic• Visual：RoIs generated(annotated) by Faster‐RCNN• Linguistic: Subwords from input sentences• Special elements: For disambiguating different input formats

• Model Architectures:• Input = Token Embedding + Segment Embedding + Sequence
Position Embedding + Visual Feature Embedding• Model Architectures:• Input = Token Embedding + Segment Embedding + Sequence
Position Embedding + Visual Feature Embedding• Token Embedding:• Language: Embedded with Word‐Piece Embeddings with 30,000 vocabulary• Vision: Special token [IMG]• Model Architectures:• Input = Token Embedding + Segment Embedding + Sequence
Position Embedding + Visual Feature Embedding• Segment Embedding:• Three Types: A for Question, B for Answer (widely, another sentence ifnecessary, C for images)• Represented with 768‐dim• Model Architectures:• Input = Token Embedding + Segment Embedding + Sequence
Position Embedding + Visual Feature Embedding• Sequence Position Embedding:• Max length: 512• Indicating order in the input sequence• e.g. Black cat is ... and white cat is ...

• Model Architectures:• Input = Token Embedding + Segment Embedding + Sequence
Position Embedding + Visual Feature Embedding• Visual Feature Embedding:• Applying Fast RCNN to extract visual feature• Non‐visual elements: extract features on whole image• Concatenate visual feature and geomerty (position in image) embedding(2048‐dim from 4‐dim)

VL‐BERT• Pre‐training• 1. V&L Dataset: Conceptual Captions; 2. Text only (overcome overfitting on
short sentences): BooksCorpus, English Wikipedia• Task1: Masked Language Modeling with Visual Clues• Masked a word randomly and predict what it is according to visual clues• e.g. Kitten is drinking from [MASK]• Loss: CE Loss (feed output feature into a classifier over whole vocabulary)VL‐BERT• Pre‐training• Task2: Masked RoI Classification with Linguistic Clues• Masked a RoI with 0• Set label predicted by Faster‐RCNN as GT• Loss: CE Loss
A kitten is drinking from a bottle

VL‐BERT• Fine‐tuning• Provide proper formatted input and output• Provide appropriate segment embeddings• Design task‐specific loss function• End‐to‐End Training• Output:• For sentence‐image‐relation level prediction: output feature from [CLS] is used
• Word‐level or RoI‐level

Fine Tuning• Visual Commonsense Reasoning (VCR)• Question ‐> Answering + Reasoning• Decompose into two steps: Q‐>A + QA‐>R (comcat. Q&A); RoIs: GT (annotate.)• Output from [CLS] ==> Softmax• Two Loss: Correctness of Answers + RoI Classification (based on language)• !SOTA!

Fine Tuning• Visual Question Answering• Generate/choose answers of question according to image• Input format:<Question, Answer([MASK]), Image>• Output from [MASK] ==> Softmax ==> Answer

Fine Tuning• Referring Expression Comprehension• Localize referred object in the expression (a NL phrase)• Input format:<Query, Image>• Train & Eval: GT regions+ Detected regions

Experiments• Pretraining Detail:• Two models: BASE and LARGE (added params., see BERT)• Faster RCNN + ResNet101 (at most 100 RoIs, at least 10 RoIs)• Conclusion: 一般没他好，比他好的更复杂Experiments

Experiments

Experiments• Ablation