Vision-LanguageLearning
Hongguang Zhu

visual commonsense reasoning

Natural Language for Visual ReasoningCVPR

https://github.com/phellonchen/awesome-Vision-and-Language-Pre-training

Application：BAIDU

https://m6.aliyun.com/?spm=a2c6h.12873639.article-detail.5.5de01e6bbb97gt#/
https://wenxin.baidu.com/

Topic List
● Before CVPR (2021.11）● ALBEF （NIPS21）Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Salesforce Research● CVPR：(exploration)● Vision-Language Pre-Training with Triple Contrastive Learning
UTA, Amazon● Multi-modal Alignment using Representation Codebook
UTA, Amazon● An Empirical Study of Training End-to-End Vision-and-Language Transformers
UCLA,Microsoft● After CVPR (2022.4-6)-> AGI, Foundational model● OpenAI: Flamingo● Google: LIMoE● Sensetime: Uni-Perceiver-MoE● Allen Institute for AI： UNIFIED-IO
CVPR Trend：● key word：CLIP，Pretrain，Video, Captioning -> more
● emerging: Sign Language (4)， vision-language Navigation (10+)● declining: Cooking Recipes(1)Part I: Before CVPR

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Salesforce ResearchALBEF:● Three key limitations● V/T embeddings reside in their own spaces●

multimodal encoders learn to model their interactions -> challenging●requires bounding box annotations during pre-training●

high resolution images during inference● object detector is both annotation-expensive and compute-expensive● IT datasets are collected from the web and are inherently noisy● pre-training objectives (MLM) may overfit to the noisy text●

degrade the model's generalization performance.

● intermediate image-text contrastive (ITC) loss
● align the image features and the text features● easier for the multimodal encoder to perform cross-modal learning● unimodal encoders to better understand the semantic meaning ofimages and texts● learns a common low-dimensional space to embed images and texts● to find more informative samples through our contrastive hard negative mining.

1. ITC
default:GT one-hot

softmax-normalized similarity

cross-entropy = KD loss

MoD -> pseudo target label(1-α) ones(V,T) + α(V,T_m)

2. Momentum Distillation

ITC
3. MIMLMITMAblation

finetune：ITM-one negativeMoD
queue:65536ALIGN:98.37

ITC+ITM (k) >ITC

more 128 forward time

text assignment pre-training

three-way classification, pretrain 1 epochPart II: CVPR

Vision-Language Pre-Training with Triple Contrastive Learning
UTA, Amazon● cross-modal alignment (CMA):maximizing the mutual information (MI)● fail to ensure that similar inputs from the same modality stay close by● global information -> localized and structural information (max local MI of local regions and global summary)● triple contrastive learning (TCL)● leveraging both cross-modal and intra-modal self-supervision.

I:two views �1 , �2 �+ = �
two encoder: raw, EMA
�2

�1��+ALBEF

TCL

better intra-modal representation: uniformly distributed text

momentum coefficient: m• pooling：local features16*16->GAP->16 patchs• use last layer

Multi-modal Alignment using Representation Codebook
UTA, Amazon● Image and text typically reside in different regions of the feature space● directly aligning them at instance level is challenging●features are still evolving during training.

● align cluster representation (codeword/ learnable prototypes)● a dictionary of cluster centers (codebook)● optimal transport problem● teacher-student distillation paradigm

momentum update

codebook learning: optimal transport

P: student predict
T: teacher

4000<TCLAn Empirical Study of Training End-to-End Vision-and-Language Transformers
UCLA,Microsoft● systematically investigate how to design and pre-train transformer-based VL model● VIT -> vital role than language transformer● the preformance of VIT on ImageNet classification is not a good indicator on VL● in multimodal fusion cross attention > self attention● encoder-only > encoder-decoder for VQA and zero-shot ITR● MIM is not a critical pre-training objective for VLP

tasks:VQA
NLVR
VEITR

MLM+ITM.

no decoder and no parameter shared

NCE
Masked Image Modeling -> masked patch classification• MSE feature, predict object label• ALBEF dont use,
• ViLT: maksed patch regression is not helpul

use VQVAE model in DALLE
predict the discrete tokens

● Explorations without VLP● initialize the bottom layers with specific pre-trained vision and text encoders, and randomly initialize the top layers.
● T: V-> CLIP-ViT-224/32 V: T-> RoBERTaswin without VLP is already comparable to some VLP model in VQA

RoBERTa -> most robust performace

DEBERTa and BEiT achieve better classification performance but not suitable for VL

the difference between BERT and RoBERTa seems to be diminished>embed-onlyuse a larger learning rate for the randomly initialized parametersthan parameters initialized with pre-trained models

increasing the image resolution during finetuning can
improve the model performance by a large margin

Mmerged = 12 and Mco = 6

MIM drop•
the conflicts between different objectives.
•image patches can be noisy

ALBEF has specially-designed objectives for retrieval
but it use CLIP_V (300M VLP)->no ITC

Part III: After CVPR

Flamingo (2022.4) : Analogical learning
Multimodal generative modeling● Unifying strong single-modal models● Interleave CA and SA (frozen) -> initialization->specific gating (stability/performance)● Supporting both images and videos● Native addition visual token sequence - > memory limitation.
● Local 2D priors (inductive bias) - >improve efficiency but not suitable for text.
● -> Perceiver-based architecture (fixed number : a hundred tokens)● Mixture of dataset：M3W + ALIGN + …= 3B● Frozen encoder ：V：CLIP， T：BERT decoder

Interleaved promptinterleaved visual and textual data

MoE (2022.6): multi-task Generalist Models -> Multi-architecture (task-interference)

Sensetime:Uni-Perceiver-MoE

Google: LiMoE
Only contrastive pretrain
Selected (5) and total (64) experts

UNIFIED-IO:A Unified Model For Vision, Language, And Multi-model Tasks

https://unified-io.allenai.org/

over 80 diverse datasets+pixel (refer expression)
inpaint/synthesis+Depth estimation+NLP tasksembedding align2017 and before

VQAfusion task->more dataset

2022： AGI？
Foundation modelcross attention
2018:SCAN

Transformer + Pretrain
2019.9 vilBERT, UNITER

2020.12 OSCAR,VinVL
stronger region feature

IT retriveal/IT matching
rely modal encoder

2021.7 ALBEF
encoder-decoder

2021.2 ViLT (ICML)
Grid/Patch feature

2021.2 CLIP
More data 400M

multi-task --> Generalist Models
saleforce, facebook -> encoder-decoder
OpenAI-> analogical learning
Google (pathways)
Alibaba (OFA)-> MoE
Sensetime
allenai-> multi data encoder - transformer - multi data decoderhttps://github.com/phellonchen/awesome-Vision-and-Language-Pre-training