NeurIPS 2022?

MILAN: Masked Image Pretraining onLanguage Assisted Representation

MengxuePreview▪ Fully-supervised → Self-supervised▪ Reconstruction based self-supervised pretraining▪ Masked Data Modeling▪ NLP → BERT
▪ MLM (Masked Language Modeling)▪ V-L → VL-BERT/ViLBERT▪ MLM▪ MIM (Masked Image Modeling)▪ ……

▪ CV → MAE
▪ MIM

BERT

Preview▪ Fully-supervised → Self-supervised▪ Reconstruction based self-supervised pretraining▪ Masked Data Modeling▪ NLP → BERT
▪ MLM (Masked Language Modeling)▪ V-L → VL-BERT/ViLBERT▪ MLM▪ MIM (Masked Image Modeling)▪ ……

▪ CV → MAE▪ MIM

VL-BERT

Preview▪ Fully-supervised → Self-supervised▪ Reconstruction based self-supervised pretraining▪ Masked Data Modeling▪ NLP → BERT
▪ MLM (Masked Language Modeling)▪ V-L → VL-BERT/ViLBERT▪ MLM▪ MIM (Masked Image Modeling)▪ ……

▪ CV → MAE▪ MIMMAE

Introduction▪ In this work, we analyze three highly correlated aspects inMAE(masked autoencoders):▪ the reconstruction target▪ the decoder design▪ the mask sampling strategy▪ We propose a new approach called MILAN, which performs
masked image pretraining on language assisted representations.

Methodology—Difference in MAE & MILAN
Predict Target / Sampling strategy / Decoder Design

MAEMILAN

Methodology—Difference in MAE & MILAN
Predict Target / Sampling strategy / Decoder Design▪ Predict Target▪ MAE: raw pixels▪ MILAN: latent representations obtained with language guidanceMethodology

Reconstruction target: language assisted representation▪ Why use CLIP feature as targets?
→ The learned representations are better clustered for different categories

Methodology—Difference in MAE & MILAN
Predict Target / Sampling strategy / Decoder Design▪ Decoder Design▪ MAE: normal encoder-decoder, both update▪ MILAN: prompting decoder, does not update the encoder (more efficient)Methodology

Decoder design: prompting decoder

Using the default 75% masking ratio, our prompting decoder reduces the computation cost by 20%
compared to MAE [25].

Methodology—Difference in MAE & MILAN
Predict Target / Sampling strategy / Decoder Design▪ Sampling strategy▪ MAE: uniform sampling▪ MILAN: mask sampling (more adapted to patches’discriminativeness)

Methodology

Masking strategy: semantic aware sampling

Because the class token from the last layer of
the CLIP image encoder is used to align with thetext embedding from the text encoder

Sclass reflects how much information one image
patch contributes to the output features of theCLIP image encoder.

Experiments▪ We pretrain the ViT-B/16 and ViT-L/16 models using MILAN
method on ImageNet-1K dataset for 400 epochs using PyTorchframework on A100 machines.

▪ We use the ViT-B/16 CLIP image encoder obtained from OpenAI’s
paper [43] to produce the reconstruction targets when pretraining
both ViT-B/16 and ViT-L/16 models.

Experiments

Classification on ImageNet-1K

Experiments

Downstream tasks▪ Object detection and instance segmentation on COCO
▪ Semantic segmentation on ADE20K

Experiments

Ablation study

Experiments

Visualization

Limitation▪ Similar to [3, 9, 33] which rely on external datasets to train their image tokenizers,

the reconstruction target in MILAN is obtained from the CLIP model which alsorequires an extra image-text dataset. Training the CLIP model, if it is notamortized for many downstream tasks, is considered an extra training step.

▪ Moreover, we recognize that our improvements on ViT-L is not as significant as
those on ViT-B. This may be because we employ the ViT-B version of the CLIP
image encoder to produce the reconstruction targets for training both ViT-B andViT-L for the sake of computational efficiency.

Thanks