MIMIC: Masked Image Modeling with Image Correspondences
Authors: name: Kalyani Marathe; name: Mahtab Bigverdi; name: Nishat Khan; name: Tuhin Kundu; name: Aniruddha Kembhavi; name: Linda G. Shapiro; name: Ranjay Krishna
Published: 2023-06-27 00:40:12

Abstract: Many pixelwise dense prediction tasks-depth estimation and semantic
segmentation in computer vision today rely on pretrained image representations.
Therefore, curating effective pretraining datasets is vital. Unfortunately, the
effective pretraining datasets are those with multi-view scenes and have only
been curated using annotated 3D meshes, point clouds, and camera parameters
from simulated environments. We propose a dataset-curation mechanism that does
not require any annotations. We mine two datasets: MIMIC-1M with 1.3M and
MIMIC-3M with 3.1M multi-view image pairs from open-sourced video datasets and
from synthetic 3D environments. We train multiple self-supervised models with
different masked image modeling objectives to showcase the following findings:
Representations trained on MIMIC-3M outperform those mined using annotations on
multiple downstream tasks, including depth estimation, semantic segmentation,
surface normals, and pose estimation. They also outperform representations that
are frozen and when downstream training data is limited to few-shot. Larger
dataset (MIMIC-3M) significantly improves performance, which is promising since
our curation method can arbitrarily scale to produce even larger datasets.
MIMIC code, dataset, and pretrained models are open-sourced at
https://github.com/RAIVNLab/MIMIC.