# Multi-View Masked Autoencoder for General Image Representation
## 1 Introduction
We note that the proposed contrastive learning-based MIM method can strengthen MIM by learning global patterns of an image with contrastive loss, which is in contrast to the existing pixel-level approaches that only learn local representations of images. In more detail, we adopt an asymmetric encoder-decoder architecture using ViT [22] blocks. The ViT makes the model focus on important features of an instance. We visualized maps of the attention of our pre-trained ViT encoder as shown in Figure 2, thus taking the average of the ViT heads following the work of [27]. CL is used to capture global information and learn discriminative representation by contrasting negative samples while pulling positive samples. By generating two augmented views via masking, with the  
Figure 1: Overview of autoencoder architecture. Given input, encoder compresses the input into low-dimensional latent representation, and it reconstructs the original data with decoder. Autoencoder aims to make input \(X\) and reconstructed output \(\tilde{X}\) similar. encoder, we compress them into latent representations, which are used for contrastive loss. While learning holistic information from contrastive loss, reconstruction loss helps the decoder to learn local representation by predicting patches from the masked image. We conducted experiments to prove the effectiveness of our work. The proposed method is a pre-training method, and we only used a pre-trained ViT encoder during fine-tuning. Our ViT encoder pre-trained with the proposed method exceeds previous work, thereby showing 84.3% ImageNet-1K classification top-1 accuracy. Though showing lower performance, but comparable compared to other CL-based methods in linear probing, our work shows an impressive performance gain compared to MIM methods by achieving a 76.7% accuracy. We also evaluated the transfer learning on object detection and segmentation. We recorded a 51.3% AP\({}^{box}\) and a 45.6% AP\({}^{mask}\) on COCO, as well as a 50.2% mIOU on ADE20K, which yielded the best and second best performance outcomes, respectively, compared to previous studies. Through ablation studies, we demonstrate that utilizing CL for MIM helps the model learn better representation. Our contributions are summarized as follows:  
* We propose a simple framework exploiting contrastive learning for MIM to learn rich and holistic representations. The model learns discriminative representation by contrasting two augmented views while reconstructing original signals from the corrupted ones. * A high masking ratio works as strong augmentation. Without additional augmentation like color distortion, blur, etc., our model shows better performance than previous CL-based methods by only using masking and random cropping. * Experimental results prove that our work is effective, thus outperforming previous MIM methods in ImageNet-1K classification, linear probing, and other downstream tasks like object detection and instance segmentation. The rest of this paper is structured as follows. Section 2 introduces related works. In Section 3, we give an overview and details of our framework. Then, we show the experimental results and analysis in Section 4. Finally, Section 5 concludes the paper.