E2E-VLP

ACL 2021

End-to-End Vision-Language Pre-training Enhanced by Visual Learning

Mengxue

Embedding&Pre-Training Task in BERT

预训练→训练→测试→ 上游任务：Mask Language Modelling & Next Sentence Prediction→ 下游任务：句子对分类任务、单句子分类任务、问答任务、单句标注任务

VLP-Visual&language pretraining

VL-BERT: PRE-TRAINING OF GENERIC VISUALLINGUISTIC REPRESENTATIONS:
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai (ICLR 2020)▪ MLM: Masked Language Modeling▪ MOC: Masked Object Classification▪ MSG: Masked Sentence Generation▪ ITM : Image Text Matching

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision:
Wonjae Kim, Bokyung Son, Ildoo Kim (ICML 2021)

Overall framwork

Input Representations▪ Sentence Embeddings▪ Image Representations
To improve the generalization of the image representation, we learn from pixels to represent an image instead ofusing bounding boxes. The pixel features are learned by a CNN visual backbone such as ResNet.

A HW length of d-dimensional vector

Cross-modal Encoder Pre-training▪ Input as:▪ Pretraining tasks:▪ Masked Language Modeling (MLM)
We randomly mask 15% tokens in the text and themodel is asked to predict these masked words withthe output text and visual representations.

▪ Image-Text Matching (ITM)
We randomly sample 50% mismatched image-text
pairs and 50% matched pairs, and train an classifier topredict whether an image and a sentence match eachother on the representation of token [CLS] in the lastencoder layer

True or False

FC
Transformer Encoder

Visual-enhanced Decoder▪ Motivation
Due to that the CNN feature map has no object level
semantics, it is difficult to directly align the cross-modal
semantics between CNN feature map and the language
embeddings.

▪ Transformer decoder▪ object detection
understanding the fine-grained object information within image▪ image-caption generation
guide the learning of visual features regarding the textual semantics

Enhanced by Object Detection--DETR▪ In encoder side
combine both the visual representation and language
embedding as input and reuse the Transformer encoder forcross-modal fusion.

▪ In decoder side
▪ Input: learned positional embeddings▪ Detects the N objects in parallel at each decoder layer▪ Box coordinate regression▪ Class category prediction▪ Object attribute prediction task for VG Dataset

Enhanced by Image Captioning

Joint Training
We pre-train E2E-VLP with all the encoder and decoder pre-training tasks ：▪ Masked Language Modeling▪ Image-Text Matching▪ Object Detection▪ Image-to-Text Generation

jointly by minimizing the four loss functions as:Experiments-Pre-training▪ Dataset▪ We pre-train our E2E-VLP on two in-domain image-text datasets: MS-COCO (Lin et al.,2014) and Visual Genome (Krishna et al., 2017).
▪ Object detection and image caption annotations in MS-COCO▪ Object detection, region description annotations in Visual Genome.

▪ Implementation Details▪ Pre-train E2E-VLP model with a total batch size of 32 for 200 epochs on 8 V100 GPUs.

Experiments-Downstream Tasks▪ VQA▪ NLVR2▪ Image Caption▪ Image-Text Retrieval

Experiments-Downstream Tasks▪ VQA▪ NLVR2▪ Image Caption▪ Image-Text Retrieval

Ablation study

Inference Efficiency

Importance of Visual Learning

Impact of Input Image Size

Architecture Selection
Object Detection with Paired Text

Thank you