# Multi-View Masked Autoencoder for General Image Representation
## 1 Introduction  
Deep learning, which has been revolutionized over the past decade, has recently faced data-hungry problems due to the rapid growth of hardware and resources [1, 2, 3]. Self-supervised learning, which learns meaningful data representations from unlabeled data [4], has emerged as an alternative to supervised learning resulting from the inefficiency of labeling in terms of time and labor [5, 6, 7]. Masked autoencoding [8] is a method that learns representations by removing part of the input and predicting the masked part. Autoencoder [9, 10] architecture is used for masked autoencoding, thus compressing high-dimensional data into a latent representation with an encoder and reconstructing the original data with a decoder, as shown in Figure 1. It has been successful in NLP as a method of self-supervised pre-training. The approach of learning representation by reconstructing images from corrupted images is not new; the idea was already proposed before 2017 [11, 12]. The idea was buried after the emergence of contrastive learning, since it has shown promising results on downstream tasks [13, 14, 15]. Upon witnessing the success of masked autoencoding in NLP fields [16, 17, 18], many works tried to apply masked autoencoding to vision, but they lagged behind due to the following reasons: (1) In vision, convolutional network architecture has been dominant [19], whereindicators like mask token [17] or positional embedding [20] are inapplicable. (2) With only a few neighboring pixels, missing parts of an image can be successfully predicted without a deep understanding of an image [21]. However, when predicting a missing part/token, complex language understanding should be investigated. In other words, the masked autoencoding in the vision field might not demand fully understanding the image, which results in capturing less useful features. Due to these differences between the two modalities, masked autoencoding has been limitedly applied in the vision field until the advent of the vision transformer (ViT) [22]. Motivated by the success of masked language modeling (MLM) in language understanding, masked image modeling (MIM), following the idea of MLM, learns rich and holistic representations by reconstructing masked original information (e.g., pixel and representation) from unmasked information. MIM has gained much importance recently by showing state-of-the-art performance [23; 24; 2] not only in ImageNet classification, but also in other downstream tasks like object detection and semantic segmentation. Before MIM, contrastive learning (CL), which learns meaningful representation by using similarities and differences between image representations, was a dominant method in self-supervised learning [4]. By learning embedding space in a way that contrasts each other so that positive samples are located close and negative samples are far away, CL learns to discriminate instances using features of the entire image [25]. Contrary to CL, MIM does not learn instance discriminativeness, since it only considers relationships between patches or pixels through the image reconstruction task [26]. Therefore, although MIM methods exceed the performance of CL methods in fine-tuning, they are shown to be less effective in linear separability. In this work, we propose a simple yet effective framework, thus adopting multi-view autoencoder architecture and utilizing contrastive learning for MIM to overcome the gap between CL and MIM. Different from conventional autoencoder architectures are shown in Figure 1; we add an additional branch to learn common information from two different augmented views by contrasting different images originating from the same image. We call this architecture a multi-view autoencoder. CL learns instance discriminative representations to result in better performance in linear probing, while MIM shows better performance in fine-tuning settings.